This notebook is part of https://github.com/AudioSceneDescriptionFormat/splines, see also https://splines.readthedocs.io/.

[back to overview](bezier.ipynb)

# Non-Uniform Bézier Splines

Very commonly,
Bézier splines are used with a parameter range of $0 \le t \le 1$,
which has also been used to derive the basis polynomials and basis matrices in
[the notebook about De Casteljau's algorithm](bezier-de-casteljau.ipynb).

The parameter range can be re-scaled to any desired parameter range,
but since the shape of a Bézier curve is fully defined by its control polygon,
this will not change the shape of the curve,
but only its speed, and therefore its tangent vectors.

To derive equations for non-uniform tangent vectors,
let us quickly re-implement De Casteljau's algorithm:

```python
def lerp(one, two, t):
    return (1 - t) * one + t * two
```

```python
def de_casteljau(points, t):
    while len(points) > 1:
        points = [lerp(a, b, t) for a, b in zip(points, points[1:])]
    return points[0]
```

```python
import sympy as sp
sp.init_printing()
```

We'll also use our trusty SymPy tools from [utility.py](utility.py):

```python
from utility import NamedExpression
```

In this notebook we are only looking at cubic Bézier splines.
More specifically, we are looking at the fifth spline segment,
from $\boldsymbol{x}_4$ to $\boldsymbol{x}_5$
within a parameter range from $t_4$ to $t_5$,
but later we can easily generalize this.

```python
control_points = sp.symbols('xbm4 xtildebm4^(+) xtildebm5^(-) xbm5')
control_points
```

```python
t, t4, t5 = sp.symbols('t t4 t5')
```

As before,
we are using De Casteljau's algorithm,
but this time we are re-scaling the parameter range using the transformation
$t \to \frac{t - t_i}{t_{i+1} - t_i}$:

```python
p4 = NamedExpression(
    'pbm4',
    de_casteljau(control_points, (t - t4) / (t5 - t4)))
```

## Tangent Vectors

As always,
the tangent vectors can be obtained by means of the first derivative:

```python
pd4 = p4.diff(t)
```

```python
pd4.evaluated_at(t, t4)
```

This expression for the outgoing tangent vector at
$\boldsymbol{x}_4$
can be generalized to

\begin{equation*}
\boldsymbol{\dot{x}}_i^{(+)} =
\frac
{3 \left(\boldsymbol{\tilde{x}^{(+)}}_i - \boldsymbol{x}_i\right)}
{\Delta_i},
\end{equation*}

where $\Delta_i = t_{i+1} - t_i$.

Similarly,
the incoming tangent vector at $\boldsymbol{x}_5$ ...

```python
pd4.evaluated_at(t, t5)
```

... can be generalized to

\begin{equation*}
\boldsymbol{\dot{x}}_i^{(-)} =
\frac
{3 \left(\boldsymbol{x}_i - \boldsymbol{\tilde{x}^{(-)}}_i\right)}
{\Delta_{i-1}}.
\end{equation*}

This is similar to the
[uniform case](bezier-de-casteljau.ipynb#Cubic-Tangent-Vectors),
the tangent vectors are just divided by the parameter interval.

## Control Points From Tangent Vectors

If the tangent vectors are given in the first place
(i.e. when a non-uniform [Hermite spline](hermite.ipynb) is given),
the cubic Bézier control points can be calculated like this:

\begin{align*}
\boldsymbol{\tilde{x}^{(+)}}_i &=
\boldsymbol{x}_i + \frac{\Delta_i \boldsymbol{\dot{x}}_i^{(+)}}{3} \\
\boldsymbol{\tilde{x}^{(-)}}_i &=
\boldsymbol{x}_i - \frac{\Delta_{i-1} \boldsymbol{\dot{x}}_i^{(-)}}{3}
\end{align*}


```python
from bateman import bateman_full
from sympy import symbols
from scipy.optimize import brentq
import sympy as sp
import numpy as np
import matplotlib.pyplot as plt
sp.init_printing()
%matplotlib inline
```

```python
args = t, x0, y0, z0, p, q = symbols('t x0 y0 z0 p q', real=True)
analytic = bateman_full([x0, y0, z0], [p, q, 0], t, exp=sp.exp)
```

```python
# sp.solve(analytic[1] - analytic[2], t)  # raises NotImplementedError
```

```python
fx, fy, fz = [sp.lambdify(args, analytic[idx]) for idx in range(3)]
xy = sp.lambdify(args, analytic[0] - analytic[1])
xz = sp.lambdify(args, analytic[0] - analytic[2])
yz = sp.lambdify(args, analytic[1] - analytic[2])
params = 1, 0, 0, 7, 3
def fxy(x):
    return xy(x, *params)

def fxz(x):
    return xz(x, *params)

def fyz(x):
    return yz(x, *params)

f = [fxy, fxz, fyz]
```

```python
zeros = xy0, xz0, yz0 = [brentq(func, *bounds) for func, bounds in zip(f, [(.05, .15), (.15, .3), (.3, .4)])]
```

```python
x = np.linspace(0, .5)
y = np.array(bateman_full([1, 0, 0], [7, 3, 0], x, exp=np.exp)).T
fig, ax = plt.subplots(1, 1, figsize=(14, 8))
ax.vlines(zeros, 0, 1, transform=ax.get_xaxis_transform(), linestyle='--')
ax.plot(zeros, [func(x, *params) for func, x in zip([fx, fx, fy], zeros)], 'or', markerfacecolor='None', markersize=15)
ax.plot(x, y)
print(zeros)
```

```python

```

```python
iv2 = [1, .1, 0]
params2 = iv2 + [7, 3]
xy2 = sp.lambdify(args, analytic[0] - 3*analytic[1])
xz2 = sp.lambdify(args, analytic[0] - 3*analytic[2])
yz2 = sp.lambdify(args, 3*analytic[1] - analytic[2])

def fxy2(x):
    return xy2(x, *params2)

def fxz2(x):
    return xz2(x, *params2)

def fyz2(x):
    return yz2(x, *params2)

f2 = [fxy2, fxz2, fyz2]
zeros2 = xy02, xz02, yz02 = [brentq(func, *bounds) for func, bounds in zip(f2, [(.0, .1), (.05, .15), (.4, .8)])]
x2 = np.linspace(0, .8)
y2 = np.array(bateman_full(iv2, [7, 3, 0], x2, exp=np.exp)).T
fig2, ax2 = plt.subplots(1, 1, figsize=(14, 8))
ax2.vlines(zeros2, 0, 1, transform=ax2.get_xaxis_transform(), linestyle='--')
ax2.plot(zeros2, [func(x, *params2) for func, x in zip([fx, fx, fz], zeros2)], 'or', markerfacecolor='None', markersize=15)
ax2.plot(x2, y2)
print(zeros2)
```

```python

```


# Linear SVC

### Primal Formulation

$$
\begin{align}
&\min_{\mathbf{w}, b} && \frac{1}{2}\mathbf{w}^{T}\mathbf{w} + C \sum_{i=1}^{m}\xi_{i}\\
&\text{subj. to} && y_{i}(\mathbf{w}^{T}\mathbf{x}_{i} + b) \ge 1 - \xi_{i}, \quad \xi_{i} \ge 0
\end{align}
$$

Make the substitution $\xi_{i} = \max\Big(0, 1 - y_{i}(\mathbf{w}^{T}\mathbf{x}_{i} + b)\Big)$.  Note that this choice of $\xi_{i}$ satisfies the constraints of the primal problem.  The new primal problem, which is unconstrained, becomes:

$$
\begin{align}
&\min_{\mathbf{w}, b} && \frac{1}{2}\mathbf{w}^{T}\mathbf{w} + C \sum_{i=1}^{m}\max\Big(0, 1 - y_{i}(\mathbf{w}^{T}\mathbf{x}_{i} + b)\Big)
\end{align}
$$

where the first term seeks to create a wide margin and second term is the hinge loss which will be positive for misclassified instances.  Setting C very large results in a hard margin classifier.

```python
import numpy as np
import numpy.linalg as LA
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.svm import LinearSVC
import pandas as pd
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
#Load the data
iris = datasets.load_iris()

#convert to dataframe
df = pd.DataFrame(iris.data, columns = ['Sepal Length', 'Sepal Width', 'Petal Length', 'Petal Width'])
df.insert(4,'Target',iris.target)
# target = [0,1,2] corresponds to Setosa, Versicolour, and Virginica, resp.
df.head()
```

      Sepal Length
      Sepal Width
      Petal Length
      Petal Width
      Target

      0
      5.1
      3.5
      1.4
      0.2
      0

      1
      4.9
      3.0
      1.4
      0.2
      0

      2
      4.7
      3.2
      1.3
      0.2
      0

      3
      4.6
      3.1
      1.5
      0.2
      0

      4
      5.0
      3.6
      1.4
      0.2
      0

```python
#Extract features and labels
X =  df[['Petal Length', 'Petal Width']].values
y = (df[['Target']].values == 2).astype(np.float64).squeeze()

#split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

#Scale the data
scalerX = StandardScaler().fit(X)
X_train_scaled = scalerX.transform(X_train)
X_test_scaled = scalerX.transform(X_test)
```

## sklearn Linear SVC

```python
#sklearn linear SVC
c = 1
lsvc = LinearSVC(C=c, loss='hinge')
lsvc.fit(X_train_scaled,y_train)
coefs = lsvc.coef_.flatten()
intercept = lsvc.intercept_
theta_lsvc_sklearn = np.concatenate((coefs, intercept), axis=0)
print("Linear SVC coefficients (sklearn):")
print(theta_lsvc_sklearn)
```

    Linear SVC coefficients (sklearn):
    [ 1.30870506  2.24455751 -2.07046507]

```python
#hinge loss
def hinge_loss(theta,X,y):
    loss, grad = 0,0
    for (x_,y_) in zip(X,y):
        v = y_*np.dot(x_,theta)
        loss += max(0,1-v)
        grad += 0 if v > 1 else -y_*x_
    return loss,grad
```

## scratch Linear SVC

```python
y_train_lsvc = (np.copy(y_train)).reshape(-1,1)
y_train_lsvc[y_train == 0] = -1

X_aug = np.concatenate( (X_train_scaled, np.ones((len(X_train_scaled),1))), axis=1)

#solve via GD
epsilon = 1e-7
delta = np.inf
alpha = 1e-3
theta0 = np.zeros((X_aug.shape[1]))
theta = []
theta.append(theta0)
J = []
J.append(np.inf)
k = 0

while delta > epsilon:
    #compute penalty and gradient of penalty
    loss, grad = hinge_loss(theta[k],X_aug,y_train_lsvc)

    #obj. function
    J.append(1/2 * theta[k].T @ theta[k] + c * loss)

    #check convergence
    delta = np.abs(J[k] - J[k-1])

    #gradient descent
    theta.append(theta[k] - alpha * (theta[k] + c * grad))

    if k%500 == 0:
        print('iteration: ', k, ' - Obj. fxn: ', J[k], ' - delta: ', delta)

    k+=1

theta_scratch = theta[k]
print("linear SVC coefficients:")
print("scratch coefficients: ", theta_scratch)
print("sklearn coefficients: ", theta_lsvc_sklearn)
```

    iteration:  0  - Obj. fxn:  inf  - delta:  [inf]
    iteration:  500  - Obj. fxn:  [19.26587964]  - delta:  [0.00137765]
    iteration:  1000  - Obj. fxn:  [19.16660528]  - delta:  [0.00133149]
    iteration:  1500  - Obj. fxn:  [19.15894032]  - delta:  [9.49930698e-05]
    iteration:  2000  - Obj. fxn:  [19.15719128]  - delta:  [0.00131263]
    iteration:  2500  - Obj. fxn:  [19.15792374]  - delta:  [0.0013141]
    iteration:  3000  - Obj. fxn:  [19.15556252]  - delta:  [0.00130937]
    iteration:  3500  - Obj. fxn:  [19.15738402]  - delta:  [0.0017284]
    iteration:  4000  - Obj. fxn:  [19.15667442]  - delta:  [0.0013116]
    iteration:  4500  - Obj. fxn:  [19.15585559]  - delta:  [0.00391263]
    iteration:  5000  - Obj. fxn:  [19.15600714]  - delta:  [0.00010572]
    iteration:  5500  - Obj. fxn:  [19.15617989]  - delta:  [0.0013106]
    iteration:  6000  - Obj. fxn:  [19.15770777]  - delta:  [0.00131366]
    iteration:  6500  - Obj. fxn:  [19.15662061]  - delta:  [0.00131149]
    iteration:  7000  - Obj. fxn:  [19.15801011]  - delta:  [0.00131427]
    iteration:  7500  - Obj. fxn:  [19.15957745]  - delta:  [0.00384211]
    iteration:  8000  - Obj. fxn:  [19.15598829]  - delta:  [7.10089177e-05]
    linear SVC coefficients:
    scratch coefficients:  [ 1.32078479  2.23333678 -2.06909415]
    sklearn coefficients:  [ 1.30870506  2.24455751 -2.07046507]

```python
#Compare classification error
loss_sklearn,_ = hinge_loss(theta_lsvc_sklearn,X_aug,y_train_lsvc)
print("hinge loss sklearn: ", loss_sklearn)

loss_scratch,_ = hinge_loss(theta_scratch,X_aug,y_train_lsvc)
print("hinge loss scratch: ", loss_scratch)
```

    hinge loss sklearn:  [13.63618496]
    hinge loss scratch:  [13.65142924]

```python
#Plot the data and the decision boundary

#create feature data for plotting
x_dec_bnd = np.linspace(-2,2,100).reshape(-1,1)
#Linear SVC classification boundary from sklearn
y_lsvc_sklearn = (theta_lsvc_sklearn[2] * np.ones((100,1)) + theta_lsvc_sklearn[0] * x_dec_bnd) / -theta_lsvc_sklearn[1]
#Linear SVC classification boundary from scratch
y_scratch = (theta_scratch[2] * np.ones((100,1)) + theta_scratch[0] * x_dec_bnd) / -theta_scratch[1]

y_1 = np.where(y_test==1)[0] #training data, iris-virginica
y_0 = np.where(y_test==0)[0] #training data, not iris-virginica
plt.plot(X_test_scaled[y_0,0],X_test_scaled[y_0,1],'bo',label="not iris-virginica")
plt.plot(X_test_scaled[y_1,0],X_test_scaled[y_1,1],'k+',label="iris-virginica")

plt.plot(x_dec_bnd,y_lsvc_sklearn,'g',label="lsvc")
plt.plot(x_dec_bnd,y_scratch,'m',label="lsvc scratch")
plt.xlabel('petal length')
plt.ylabel('petal width')
plt.title('Comparison of Classifiers')
plt.xlim((-0.5,2))
plt.ylim((-1,2))
plt.legend()
```


# Lecture 25: Beta-Gamma (bank-post office), order statistics, conditional expectation, two envelope paradox

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Connecting the Gamma and Beta Distributions

Say you have to visit both the bank and the post office today. What can we say about the total times you have to wait in the lines?

Let $X \sim \operatorname{Gamma}(a, \lambda)$ be the total time you wait in line at the bank, given that there are $a$ people in line in front of you, and the waiting times are i.i.d $\operatorname{Expo}(\lambda)$; recall the analogies of geometric $\rightarrow$ negative binomial, and of exponential $\rightarrow$ gamma. The waiting time in line at the bank for everyone individually is $\operatorname{Expo}(\lambda)$, and as the $a+1^{th}$ person, your time in line is sum of those $a$ $\operatorname{Expo}(\lambda)$ times.

Similarly, let $Y \sim \operatorname{Gamma}(b, \lambda)$ be the total time you wait in line at the post office, given that there are $b$ people in line in front of you.

Assume that $X, Y$ are independent.

### Questions

1. What is the distribution of $T = X + Y$?
1. Given $T = X + Y$ and $W = \frac{X}{X+Y}$, what is the joint distribution?
1. Are $T, W$ independent?

### What is the distribution of $T$?

We immediately know that the total time you spend waiting in the lines is

\begin{align}
  T &= X + Y \\
    &\sim \operatorname{Gamma}(a+b, \lambda)
\end{align}

### What is the distribution of  $T,W$?

Let $\lambda = 1$, to make the calculation simpler. We do not lose any generality, since we can scale by $\lambda$ later.

So we are looking the joint PDF of $T,W$

\begin{align}
  \text{joint PDF } f_{T,W}(t,w) &= f_{X,Y}(x,y) \, \left| \frac{\partial(x,y)}{\partial(t,w)} \right| \\
  &= \frac{1}{\Gamma(a) \Gamma(b)} \, x^a \, e^{-x} \, y^b \, e^{-y} \, \frac{1}{xy} \, \left| \frac{\partial(x,y)}{\partial(t,w)} \right| \\\\
  \\
  \text{for the Jacobian, let } x + y &= t \\
  \frac{x}{x+y} &= w \\
  \\
  \Rightarrow x &= tw \\
  \\
  1 - \frac{x}{x+y} &= 1 - w \\
  \frac{x + y - x}{t} &= 1 - w \\
  \\
  \Rightarrow y &= t(1-w) \\\\
  \\
  \left| \frac{\partial(x,y)}{\partial(t,w)} \right| &=
    \begin{bmatrix}
      \frac{\partial x}{\partial t} & \frac{\partial x}{\partial w} \\
      \frac{\partial y}{\partial t} & \frac{\partial y}{\partial w}
    \end{bmatrix} \\
    &=
    \begin{bmatrix}
      w & t \\
      1-w & -t
    \end{bmatrix} \\
    &= -tw - t(1-w) \\
    &= -t \\\\
    \\
    \text{returning to PDF } f_{T,W}(t,w) &=  \frac{1}{\Gamma(a) \Gamma(b)} \, x^a \, e^{-x} \, y^b \, e^{-y} \, \frac{1}{xy} \, \left| \frac{\partial(x,y)}{\partial(t,w)} \right| \\
    &= \frac{1}{\Gamma(a) \Gamma(b)} \, (tw)^a \, e^{-(tw)} \, (t(1-w))^b \, e^{-t(1-w)} \, \frac{1}{tw \, t(1-w)} \, t \\
    &= \frac{1}{\Gamma(a) \Gamma(b)} \, w^{a-1} \, (1-w)^{b-1} \,\, t^{a+b} \, e^{-t} \, \frac{1}{t} \, c &\quad \text{ where } c \text{ is the normalizing constant for } T \\
    &= \frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \, w^{a-1} \, (1-w)^{b-1} \,\, \frac{t^{a+b} \, e^{-t} \, \frac{1}{t}}{\Gamma(a+b)} &\quad \text{ multiplying by } 1
\end{align}

Since we are able to successfully derive $f_{T,W}(t,w)$ in terms of $T \sim \operatorname{Gamma}(a,b)$ and $W \sim \operatorname{Beta}(a,b)$, this means we have also answered the third question: _$T,W$ are independent_.

### Unexpected Discovery: Normalizing Constant for Beta

Now say we are interested in finding the marginal PDF for $W$

\begin{align}
  f_{W}(w) &= \int_{-\infty}^{\infty} f_{T,W}(t,w) dt \\
  &= \int_{-\infty}^{\infty} \frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \, w^{a-1} \, (1-w)^{b-1} \,\, \frac{t^{a+b} \, e^{-t} \, \frac{1}{t}}{\Gamma(a+b)} \, dt \\
  &= \frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \, w^{a-1} \, (1-w)^{b-1} \, \int_{-\infty}^{\infty} \frac{t^{a+b} \, e^{-t} \, \frac{1}{t}}{\Gamma(a+b)} \, dt\\
  &= \frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \, w^{a-1} \, (1-w)^{b-1}
\end{align}

But notice that since marginal PDF $f_{W}(w)$ must integrate to 1, then $\frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)}$ is the normalizing constant for the Beta distribution! If this were not true, then $f_{W}(w)$ could not be a valid PDF.

## Example Usage: Finding $\mathbb{E}(W), W \sim \operatorname{Beta}(a,b)$

There are two ways you could find $\mathbb{E}(W)$.

You could use LOTUS, where you would simply do:

\begin{align}
  \mathbb{E}(W) &= \int \frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \, w^{a-1} \, (1-w)^{b-1} \, w \, dw \\
  &= \int \frac{\Gamma(a+b)}{\Gamma(a) \Gamma(b)} \, w^{a} \, (1-w)^{b-1} \, dw \\
\end{align}

... and would not be so hard to handle, since that also is a $\operatorname{Beta}$.

Or, since we are continuing on the topic of $W = X + Y$, we have:

\begin{align}
  \mathbb{E}(W) &= \mathbb{E}\left( \frac{X}{X+Y} \right) \\
  &= \frac{\mathbb{E}(X)}{\mathbb{E}(X+Y)} \quad \text{ which is true, under certain conditions}
\end{align}

So why is $\mathbb{E}\left( \frac{X}{X+Y} \right) = \frac{\mathbb{E}(X)}{\mathbb{E}(X+Y)}$?

Facts

1. since $T$ is independent of $W$, $\frac{X}{X+Y}$ is independent of $X+Y$
2. since independence implies they are uncorrelated, $\frac{X}{X+Y}$ and $X+Y$ are therefore _uncorrelated_
3. by definition, uncorrelated means \begin{align}
  \mathbb{E}(AB) - \mathbb{E}(A) \, \mathbb{E}(B) &= 0 \\
  \mathbb{E}(AB) &= \mathbb{E}(A) \, \mathbb{E}(B) \\\\
  \\
  \mathbb{E} \left( \frac{X}{X+Y} \, (X+Y) \right) &= \mathbb{E}(\frac{X}{X+Y}) \, \mathbb{E}(X+Y) \\
  \mathbb{E}(X) &= \mathbb{E}(\frac{X}{X+Y}) \, \mathbb{E}(X+Y) \\
  \Rightarrow \mathbb{E}\left( \frac{X}{X+Y} \right) &= \frac{\mathbb{E}(X)}{\mathbb{E}(X+Y)} \\\\
  \\
  \therefore \mathbb{E}(W) &= \mathbb{E} \left( \frac{X}{X+Y} \right) \\
  &= \frac{\mathbb{E}(X)}{\mathbb{E}(X+Y)} \\
  &= \frac{a}{a+b}
\end{align}

## Order Statistics

Let $X_1, X_2, \dots , X_n$ be i.i.d. The _order statistics_ are $X_{(1)} \le X_{(2)}\le \dots \le X_{(n)}$, where

\begin{align}
  X_{(1)} &= min(X_1, X_2, \dots , X_n) \\
  X_{(n)} &= max(X_1, X_2, \dots , X_n) \\
  \\
  \text{if } n \text{ is odd, } &\text{ median is } X_{( \frac{n+1}{2} )}
\end{align}

Other statistics inlucde quartiles, etc.

* Order statistis are difficult to work with, because they are dependent; knowing $X_{(1)}$ gives you information about $X_{(n)}$, for example
* In the discrete case, things are tricky since you need to consider what to do in case of ties

### Distribution of Order Statistics

Let $X_1, X_2, \dots , X_n$ be i.i.d. with PDF $f$ and CDF $F$. Find the CDF and PDF of _marginal_ $X_{(j)}$ (we focus in only on $j$).

#### CDF: $P(X_{(j)} \le x)$

Looking at the image above...

\begin{align}
  \text{marginal CDF } P(X_{(j)} \le ) &= P(\text{at least } j \text{ of } X_i \le x) \\
  &= \sum_{k=j}^n \binom{n}{k} \, F(x)^k \, \left( 1-F(x) \right)^{n-k} \\
\end{align}

#### PDF: $f_{X_{(j)}}(x)$

Rather than taking the derivative of the CDF (and avoiding working with tedious sums), let's once again look at an image and think about this...

Imagine a tiny interval about $x$ which we call $dx$. If we multiply the PDF by a infinitesimally small interval, we can calculate the probability that the order statistic of interest $j$ is in this tiny interval.

* pick one of the $n$ statistics to land inside of $dx$
* the probability that an order statistic lands inside of the $dx$ area is $f(x)dx$
* there are $j-1$ to the left of $dx$
* the remaining $n-j$ are to the right of $dx$

\begin{align}
  f_{X_{(j)}}(x) \, dx &= n \, \binom{n-1}{j-1} \left( f(x)dx \right) \, F(x)^{j-1} \, \left( 1-F(x) \right)^{n-j} \\
  \\
  \text{marginal PDF } f_{X_{(j)}}(x) &= n \, \binom{n-1}{j-1} \,  F(x)^{j-1} \, \left( 1-F(x) \right)^{n-j} \, f(x) \\
\end{align}

## Example: $\mathbb{E}|U_1 - U_2|$

Let $U_1, U_2, \dots , U_n$ be i.i.d. $\operatorname{Unif}(0,1)$.

Then the corresponding marginal PDF $f_{U_{(j)}}(x)$ is

\begin{align}
  f_{U_{(j)}}(x) &= n \, \binom{n-1}{j-1} \, x^{j-1} \, (1-x)^{n-j} \quad \text{for } 0 \le x \le 1 \\
  \\
  \Rightarrow U_{(j)} &\sim \operatorname{Beta}(j, n-j+1)
\end{align}

Recall an earlier discussion of $\mathbb{E}|U_1 - U_2| = \mathbb{E}\left( max(U_1,U_2) \right) - \mathbb{E}\left( min(U_1,U_2) \right)$

But since

\begin{align}
  max(U_1,U_2) &= U_2 \, \text{ so } n = 2, j = 2 \\
  \mathbb{E}\left( max(U_1,U_2) \right) &= \mathbb{E} \left(\operatorname{Beta}(2, 2-2+1) \right) \\
  &= \mathbb{E} \left( \operatorname{Beta}(2, 1) \right) \\
  &= \frac{2}{2+1} \\
  &= \frac{2}{3} \\
  \\
  min(U_1,U_2) &= U_1 \, \text{ so } n = 2, j = 1 \\
  \mathbb{E}\left( min(U_1,U_2) \right) &= \mathbb{E} \left(\operatorname{Beta}(1, 2-1+1) \right) \\
  &=  \mathbb{E} \left( \operatorname{Beta}(1,2) \right) \\
  &= \frac{1}{2+1} \\
  &= \frac{1}{3} \\
  \\
  \Rightarrow \mathbb{E}|U_1 - U_2| &= \frac{2}{3} - \frac{1}{3} \\
  &= \boxed{\frac{1}{3}}
\end{align}

## Conditional Expection

If you understand conditional probability, then you can extend that to _conditional expectation_.

\begin{align}
  \text{consider } \mathbb{E}(X|A) &\text{where } A \text{ is an event } \\\\
  \\
  \mathbb{E}(X) &= \mathbb{E}(X|A)P(A) + \mathbb{E}(X|A^{\complement})P(A^{\complement}) \\
  \\
  \mathbb{E}(X) &= \sum_{x} x \, P(X=x) &\text{where you expand } P(X=x) \text{ with LOTP }
\end{align}

We will go more into this next time...

## Two-envelope Paradox

Now consider this paradox before we leave off.

There are two envelopes with cash inside them. You do not know how much is inside, only that one envelope has twice as much as the other.

Let's say you open up one of the envelopes and find $100 inside.

_Should you switch?_

Well, the other envelope could contain either 50 or it could contain 200. The mean of those two amounts is $125, so wouldn't that mean you should switch?

But then again, it doesn't matter that the envelope you opened contained 100: it could have been any amount $n$. So the other envelope could hold $\frac{n}{2}$ or $2n$, the average being $\frac{5n}{4}$, so you should switch. But then the same argument applies, so you should switch back. But then the same argument applies, so you should switch again...? And again...? And again...? Ad infinitum, ad nauseum.

To be continued.

----

View [Lecture 25: Order Statistics and Conditional Expectation | Statistics 110](http://bit.ly/2oRCNDa) on YouTube.


# Support Vector Machines

    Authors: Jesús Cid Sueiro (jcid@tsc.uc3m.es)
             Jerónimo Arenas García (jarenas@tsc.uc3m.es)

This notebook is a compilation of material taken from several sources:

- The [sklearn documentation](href = http://scikit-learn.org/stable/modules/svm.html>)
- A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)
- [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)

    Notebook version: 1.0 (Oct 28, 2015)
                      1.1 (Oct 27, 2016)
                      2.0 (Nov 2, 2017)
                      2.1 (Oct 20, 2018)

    Changes:
    v.1.0 - First version
    v.1.1 - Typo correction and illustrative figures for linear SVM
    v.2.0 - Compatibility with Python 3 (backcompatible with Python 2.7)
    v.2.1 - Minor corrections on the notation
    v.2.2 - Minor equation errors. Reformatted hyperlinks. Restoring broken visualization of images in some Jupyter versions.

```python
from __future__ import print_function

# To visualize plots in the notebook
%matplotlib inline

# Imported libraries
#import csv
#import random
#import matplotlib
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
#import pylab

import numpy as np
#from sklearn.preprocessing import PolynomialFeatures
from sklearn import svm
from sklearn.datasets.samples_generator import make_blobs
from sklearn.datasets.samples_generator import make_circles

from ipywidgets import interact
```

## 1. Introduction

  [Source: [sklearn documentation](http://scikit-learn.org/stable/modules/svm.html)  ]

Support vector machines (SVMs) are a set of supervised learning methods used for classification, regression and outliers detection.

The advantages of support vector machines are:

- Effective in high dimensional spaces.
- Still effective in cases where number of dimensions is greater than the number of samples.
- Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
- Versatile: different Kernel functions can be specified for the decision function.

The disadvantages of support vector machines include:

- SVMs do not directly provide probability estimates.

## 2. Motivating Support Vector Machines

  [Source: A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)]

Support Vector Machines (SVMs) are a kind of ***discriminative*** classifiers: that is, they draw a boundary between clusters of data without making any explicit assumption about the probability model underlying the data generation process.

Let's show a quick example of support vector classification. First we need to create a dataset:

```python
X, y = make_blobs(n_samples=50, centers=2, random_state=0, cluster_std=0.60)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

A discriminative classifier attempts to draw a line between the two sets of data. Immediately we see an inconvenience: such problem is ill-posed! For example, we could come up with several possibilities which perfectly discriminate between the classes in this example:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')

for m, b in [(1, 0.65), (0.5, 1.6), (-0.2, 2.9)]:
    plt.plot(xfit, m * xfit + b, '-k')

plt.xlim(-1, 3.5);
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

These are three very different separators which perfectly discriminate between these samples. Depending on which you choose, a new data point will be classified almost entirely differently! How can we improve on this?

Support Vector Machines (SVM) select the boundary decision maximizing the ***margin***. The margin of a classifier is defined as twice the maximum signed distance between the decision boundary and the training data. By *signed* we mean that the distance to misclassified samples is counted negatively. Thus, if the classification problem is "separable" (i.e. if there exist a decision boundary with zero errors in the training set), the SVM will choose the zero-error decision boundary that is "as far as possible" from the training data.

In summary, what an SVM does is to not only draw a line, but consider the "sample free" region about the line. Here's an example of what it might look like:

```python
xfit = np.linspace(-1, 3.5)
plt.scatter(X[:,0], X[:,1], c=y, s=50, cmap='copper')

for m, b, d in [(1, 0.65, 0.33), (0.5, 1.6, 0.55), (-0.2, 2.9, 0.2)]:
    yfit = m * xfit + b
    plt.plot(xfit, yfit, '-k')
    plt.fill_between(xfit, yfit-d, yfit+d, edgecolor='none',
                     color='#AAAAAA', alpha=0.4)

plt.xlim(-1, 3.5)
plt.xlabel("$x_0$", fontsize=14)
plt.ylabel("$x_1$", fontsize=14)
plt.axis('equal')
plt.show()
```

Notice here that if we want to maximize this width, the middle fit is clearly the best. This is the intuition of the SVM, which optimizes a linear discriminant model in conjunction with a margin representing the perpendicular distance between the datasets.

## 3. Linear SVM

  [Source: adapted from [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)]

In order to present the SVM in a formal way, consider a training dataset $\mathcal{D} = \left\{ (\mathbf{x}^{(k)}, y^{(k)}) \mid \mathbf{x}^{(k)}\in \Re^N,\, y^{(k)} \in \{-1,1\}, k=0,\ldots, {K-1}\right\}$, where the binary symmetric label $y^{(k)}\in \{-1,1\}$ indicates the class to which the point $\mathbf{x}^{(k)}$ belongs. Each $\mathbf{x}^{(k)}$ is a $p$-dimensional real vector. We want to find the maximum-margin hyperplane that divides the points having $y^{(k)}=1$ from those having $y^{(k)}=-1$.

Any hyperplane can be written as the set of points $\mathbf{x}$ satisfying

$$
\mathbf{w}^\intercal \mathbf{x} - b=0,
$$

where ${\mathbf{w}}$ denotes the (not necessarily normalized) normal vector to the hyperplane. The parameter $\tfrac{b}{\|\mathbf{w}\|}$ determines the offset of the hyperplane from the origin along the normal vector ${\mathbf{w}}$.

If the training data are linearly separable, we can select two parallel hyperplanes in a way that they separate the data and there are no points between them, and then try to maximize their distance. The region bounded by them is called "the margin". These hyperplanes can be described by the equations

$$\mathbf{w}^\intercal \mathbf{x} - b=1$$
and
$$\mathbf{w}^\intercal \mathbf{x} - b=-1.$$

Note that the two equations above can represent any two parallel hyperplanes in $\Re^N$. Essentially, the direction of vector $\mathbf{w}$ determines the orientation of the hyperplanes, whereas parameter $b$ and the norm of $\mathbf{w}$ can be used to select their exact location.

To compute the distance between the hyperplanes, we can obtain the projection of vector ${\mathbf x}_1 - {\mathbf x}_2$, where ${\mathbf x}_1$ and ${\mathbf x}_2$ are points from each of the hyperplanes, onto a unitary vector orthonormal to the hyperplanes:

$$\text{Distance between hyperplanes} = \left[\frac{\mathbf{w}}{\|\mathbf{w}\|}\right]^\top ({\mathbf x}_1 - {\mathbf x}_2) = \frac{\mathbf{w}^\top {\mathbf x}_1 - \mathbf{w}^\top {\mathbf x}_2}{\|\mathbf{w}\|} = \frac{2}{\|\mathbf{w}\|}.$$

Therefore, to maximize the distance between the planes we want to minimize $\|\mathbf{w}\|$.

As we also have to prevent data points from falling into the margin, we add the following constraints: for each $k$ either

\begin{align}
\mathbf{w}^\top \mathbf{x}^{(k)} - b &\ge +1, \qquad\text{ if } y^{(k)}=1, \qquad \text{or} \\
\mathbf{w}^\top \mathbf{x}^{(k)} - b &\le -1, \qquad\text{ if } y^{(k)}=-1.
\end{align}

This can be rewritten as:

$$
y^{(k)}(\mathbf{w}^\top \mathbf{x}^{(k)} - b) \ge 1, \quad \text{ for all } 1 \le k \le K.
$$

We can put this together to get the optimization problem:

$$
(\mathbf{w}^*,b^*) = \arg\min_{(\mathbf{w},b)} \|\mathbf{w}\| \\
\text{subject to: }
y^{(k)}(\mathbf{w}^\top \mathbf{x}^{(k)} - b) \ge 1, \, \text{ for any } k = 0, \dots, {K-1}
$$

This optimization problem is difficult to solve because it depends on $\|\mathbf{w}\|$, the norm of $\mathbf{w}$, which involves a square root. Fortunately it is possible to alter the minimization objective $\|\mathbf{w}\|$ by substituting it with $\tfrac{1}{2}\|\mathbf{w}\|^2$ (the factor of $\frac{1}{2}$ being used for mathematical convenience) without changing the solution (the minimum of the original and the modified equation have the same $\mathbf{w}$ and $b$):

$$
(\mathbf{w}^*,b^*) = \arg\min_{(\mathbf{w},b)} \frac{1}{2}\|\mathbf{w}\|^2 \\
\text{subject to: }
y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) \ge 1, \, \text{ for any } k = 0, \dots, {K-1}
$$

This is a particular case of a *quadratic programming* problem.

### 3.1. Primal form

The optimization problem stated in the preceding section can be solved by means of a generalization of the Lagrange method of multipliers for inequality constraints, using the so called Karush–Kuhn–Tucker (KKT) multipliers $\boldsymbol{\alpha}$. According to it, the constrained problem can be expressed as

$$(\mathbf{w}^*,b^*, \boldsymbol{\alpha}^*) = \arg\min_{\mathbf{w},b } \max_{\boldsymbol{\alpha}\geq 0 } \left\{ \frac{1}{2}\|\mathbf{w}\|^2 - \sum_{k=0}^{K-1}{\alpha^{(k)}\left[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b)-1\right]} \right\}
$$

that is, we look for a *saddle point*.

A key result in convex optimization theory is that, for the kind of optimization problems discussed here (see [here](http://www.onmyphd.com/?p=kkt.karush.kuhn.tucker&ckattempt=1), for instance), the *max* and *min* operators are interchangeable, so that
$$
(\mathbf{w}^*,b^*, \boldsymbol{\alpha}^*) = \arg\max_{\boldsymbol{\alpha}\geq 0 } \min_{\mathbf{w},b } \left\{ \frac{1}{2}\|\mathbf{w}\|^2 - \sum_{k=0}^{K-1}{\alpha^{(k)}\left[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b)-1\right]} \right\}
$$

Note that the inner minimization problem is now quadratic in $\mathbf{w}$ and, thus, the minimum can be found by differentiation:
$$
\mathbf{w}^* = \sum_{k=0}^{K-1}{\alpha^{(k)} y^{(k)}\mathbf{x}^{(k)}}.
$$

#### 3.1.1. Support Vectors

In view of the optimization problem, we can check that all the points which can be separated as $y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) - 1 > 0 $ do not matter since we must set the corresponding $\alpha^{(k)}$ to zero. Therefore, only a few $\alpha^{(k)}$ will be greater than zero. The corresponding $\mathbf{x}^{(k)}$ are known as `support vectors`.

It can be seen that the optimum parameter vector $\mathbf{w}^\ast$ can be expressed in terms of the support vectors only:

$$
\mathbf{w}^* = \sum_{k\in {\cal{S}}_{SV}}{\alpha^{(k)} y^{(k)}\mathbf{x}^{(k)}}.
$$

where ${\cal{S}}_{SV}$ is the set of indexes associated to support vectors.

#### 3.1.2. The computation of $b$

Support vectors lie on the margin and satisfy $y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) = 1$. From this condition, we can obtain the value of $b$, since for any support vector:

$$\mathbf{w}^\intercal\mathbf{x}^{(k)} - b = \frac{1}{y^{(k)}} = y^{(k)} \iff b = \mathbf{w}^\intercal\mathbf{x}^{(k)} - y^{(k)}
$$

This estimate of $b$, the centerpoint of the division, depends only on a single pair $y^{(k)}$ and $x^{(k)}$.  We may get a more robust estimate of the center by averaging over all of the $N_{SV}$ support vectors, if we believe the population mean is a good estimate of the midpoint, so in practice, $b$ is often computed as:

$$
b = \frac{1}{N_{SV}} \sum_{\mathbf{x}^{(k)}\in {\cal{S}}_{SV}}{(\mathbf{w}^\intercal\mathbf{x}^{(k)} - y^{(k)})}
$$

### 3.2. Dual form

Writing the classification rule in its unconstrained dual form reveals that the *maximum-margin hyperplane* and therefore the classification task is only a function of the *support vectors*, the subset of the training data that lie on the margin.

Using the fact that $\|\mathbf{w}\|^2 = \mathbf{w}^\intercal \mathbf{w}$ and substituting $\mathbf{w} = \sum_{k=0}^{K-1}{\alpha^{(k)} y^{(k)}\mathbf{x}^{(k)}}$, we obtain

\begin{align}
(b^*, \boldsymbol{\alpha}^*)
   &= \arg\max_{\boldsymbol{\alpha}\geq 0 }
   \min_b \left\{
   \sum_{k=0}^{K-1}\alpha^{(k)} -
   \frac{1}{2}
   \sum_{k=0}^{K-1} \sum_{j=0}^{K-1} {\alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} (\mathbf{x}^{(k)})^\intercal\mathbf{x}^{(j)}}
   + b \sum_{k=0}^{K-1}\alpha^{(k)}y^{(k)}
\right\}
\end{align}

Note that, if $\sum_{k=0}^{K-1}\alpha^{(k)}y^{(k)} \neq 0$ the optimal value  of $b$ is $+\infty$ of $-\infty$, and

\begin{align}
\min_b \left\{
\sum_{k=0}^{K-1}\alpha^{(k)} -
\frac{1}{2}
\sum_{k=0}^{K-1} \sum_{j=0}^{K-1} {\alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} (\mathbf{x}^{(k)})^\intercal\mathbf{x}^{(j)}}
+ b \sum_{k=0}^{K-1}\alpha^{(k)}y^{(k)}
\right\} = -\infty.
\end{align}

Therefore, any $\boldsymbol{\alpha}$ satifying $\sum_{k=0}^{K-1}\alpha^{(k)}y^{(k)} \neq 0$ is suboptimal, so that the optimal multipliers must satisfy the condition  $\sum_{k=0}^{K-1}\alpha^{(k)}y^{(k)} = 0$.

Summarizing, the dual formulation of the optimization problem is

$$
\boldsymbol{\alpha}^* = \arg\max_{\boldsymbol{\alpha}\geq 0} \sum_{k=0}^{K-1} \alpha^{(k)} -
       \frac12 \sum_{k,j} \alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} k(\mathbf{x}^{(k)}, \mathbf{x}^{(j)})  \\
\text{subject to: } \qquad \sum_{k=0}^{K-1} \alpha^{(k)} y^{(k)} = 0.
$$

where the *kernel* $k(\cdot)$ is defined by $k(\mathbf{x}^{(k)},\mathbf{x}^{(j)})=(\mathbf{x}^{(k)})^\intercal\mathbf{x}^{(j)}$.

Many implementations of the SVM use this dual formulation. They proceed in three steps:

1. Solve the dual problem to obtain $\boldsymbol{\alpha}^*$. Usually, only a small number of $\alpha^{*(k)}$ are nonzero. The corresponding values of ${\bf x}^{(k)}$ are called the *support vectors*.
2. Compute $\mathbf{w}^* = \sum_{k=0}^{K-1}{\alpha^{*(k)} y^{(k)}\mathbf{x}^{(k)}}$
3. Compute $b^* = \frac{1}{N_{SV}} \sum_{\alpha^{*(k)}\neq 0}{(\mathbf{w}^{*\intercal}\mathbf{x}^{(k)} - y^{(k)})}
$

## 4. Fitting a Support Vector Machine

  [Source: A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)]

Now we'll fit a Support Vector Machine Classifier to these points.

```python
clf = svm.SVC(kernel='linear')
clf.fit(X, y)
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto', kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

To better visualize what's happening here, let's create a quick convenience function that will plot SVM decision boundaries for us:

```python
def plot_svc_decision_function(clf, ax=None):
    """Plot the decision function for a 2D SVC"""
    if ax is None:
        ax = plt.gca()
    x = np.linspace(plt.xlim()[0], plt.xlim()[1], 30)
    y = np.linspace(plt.ylim()[0], plt.ylim()[1], 30)
    Y, X = np.meshgrid(y, x)
    P = np.zeros_like(X)
    for i, xi in enumerate(x):
        for j, yj in enumerate(y):
            P[i, j] = clf.decision_function(np.array([xi, yj]).reshape(1,-1))
    # plot the margins
    ax.contour(X, Y, P, colors='k',
               levels=[-1, 0, 1], alpha=0.5,
               linestyles=['--', '-', '--'])

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);
```

Notice that the dashed lines touch a couple of the points: these points are the *support vectors*. In scikit-learn, these are stored in the ``support_vectors_`` attribute of the classifier:

```python
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, marker='s');
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)

```

Let's use IPython's interact functionality to explore how the distribution of points affects the support vectors and the discriminative fit. (This is only available in IPython 2.0+, and will not work in a static view)

```python
def plot_svm(N=10):
    X, y = make_blobs(n_samples=200, centers=2,
                      random_state=0, cluster_std=0.60)
    X = X[:N]
    y = y[:N]
    clf = svm.SVC(kernel='linear')
    clf.fit(X, y)
    plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
    plt.xlim(-1, 4)
    plt.ylim(-1, 6)
    plot_svc_decision_function(clf, plt.gca())
    plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
                s=200, facecolors='none')

interact(plot_svm, N=[10, 200], kernel='linear')
```

    interactive(children=(Dropdown(description='N', options=(10, 200), value=10), Output()), _dom_classes=('widget…

    <function __main__.plot_svm(N=10)>

Notice the unique thing about SVM is that only the support vectors matter: that is, if you moved any of the other points without letting them cross the decision boundaries, they would have no effect on the classification results!

## 5. Non-separable problems.

  [Source: adapted from [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)]

In 1995, Corinna Cortes and Vladimir N. Vapnik suggested a modified maximum margin idea that allows for mislabeled examples. If there exists no hyperplane that can split the `positive` and `negative` samples, the `Soft Margin`
 method will choose a hyperplane that splits the examples as cleanly as possible, while still maximizing the distance to the nearest cleanly split examples. The method introduces non-negative slack variables, $\xi^{(k)}$, which measure the degree of misclassification of the data $\mathbf{x}^{(k)}$

$$y^{(k)}(\mathbf{w}^\intercal\mathbf{x}^{(k)} - b) \ge 1 - \xi^{(k)} \quad k=0,\ldots, K-1.
$$

The objective function is then increased by a function which penalizes non-zero $\xi^{(k)}$, and the optimization becomes a trade off between a large margin and a small error penalty. If the penalty function is linear, the optimization problem becomes:

$$(\mathbf{w}^*,\mathbf{\xi}^*, b^*) = \arg\min_{\mathbf{w},\mathbf{\xi}, b } \left\{\frac{1}{2} \|\mathbf{w}\|^2 + C \sum_{k=0}^{K-1} \xi^{(k)} \right\} \\
\text{subject to: } \quad y^{(k)}(\mathbf{w}^\intercal\mathbf{x}^{(k)} - b) \ge 1 - \xi^{(k)}, \quad \xi^{(k)} \ge 0, \quad  k=0,\ldots, K-1.
$$

This constraint along with the objective of minimizing $\|\mathbf{w}\|$ can be solved using KKT multipliers as done above. One then has to solve the following problem:

$$
\arg\min_{\mathbf{w}, \mathbf{\xi}, b } \max_{\boldsymbol{\alpha}, \boldsymbol{\beta} }
\left\{ \frac{1}{2}\|\mathbf{w}\|^2
+ C \sum_{k=0}^{K-1} \xi^{(k)}
- \sum_{k=0}^{K-1} {\alpha^{(k)}\left[y^{(k)}(\mathbf{w}^\intercal \mathbf{x}^{(k)} - b) -1 + \xi^{(k)}\right]}
- \sum_{k=0}^{K-1} \beta^{(k)} \xi^{(k)} \right \}\\
\text{subject to: } \quad
\alpha^{(k)}, \beta^{(k)} \ge 0.
$$

A similar analysis to that in the separable case can be applied to show that the dual formulation of the optimization problem is

$$
        \boldsymbol{\alpha}^* = \arg\max_{0 \leq \alpha^{(k)} \leq C} \sum_{k=0}^{K-1} \alpha^{(k)} -
       \frac12 \sum_{k,j} \alpha^{(k)} \alpha^{(j)} y^{(k)} y^{(j)} k(\mathbf{x}^{(k)}, \mathbf{x}^{(j)})  \\
\text{subject to: } \qquad \sum_{k=0}^{K-1} \alpha^{(k)} y^{(k)} = 0.
$$
Note that the only difference with the separable case is given by the constraints $\alpha^{(k)} \leq C$.

## 6. Nonlinear classification

  [Source: adapted from [Wikipedia](https://en.wikipedia.org/wiki/Support_vector_machine)]

The original optimal hyperplane algorithm proposed by Vapnik in 1963 was a linear classifier. However, in 1992, Bernhard E. Boser, Isabelle M. Guyon and Vladimir N. Vapnik suggested a way to create nonlinear classifiers by applying the *kernel trick* to maximum-margin hyperplanes. The resulting algorithm is formally similar, except that every dot product is replaced by a nonlinear kernel function. This allows the algorithm to fit the maximum-margin hyperplane in a transformed feature space. The transformation may be nonlinear and the transformed space high dimensional; thus though the classifier is a hyperplane in the high-dimensional feature space, it may be nonlinear in the original input space.

The kernel is related to the transform $\phi(\mathbf{x})$ by the equation $k(\mathbf{x}, \mathbf{x}') = \phi(\mathbf{x})^\intercal \phi(\mathbf{x}')$. However, note that we do not need to explicitly compute $\phi(\mathbf{x})$, as long as we can express all necessary calculations in terms of the kernel function only, as it is the case for the optimization problem in the dual case.

The predictions of the SVM classifier can also be expressed in terms of kernels only, so that we never need to explicitely compute $\phi(\mathbf{x})$.

$$\begin{align}\hat y({\mathbf{x}}) & = {\mathbf {w^\ast}}^\intercal \phi(\mathbf{x}) - b^\ast \\ \\
& = \left[\sum_{k \in {\cal{S}}_{SV}} \alpha^{(k)^*} y^{(k)} \phi(\mathbf{x}^{(k)})\right]^\intercal {\phi(\mathbf{x})} - b^\ast  \\ \\
& = - b^\ast + \sum_{k \in {\cal{S}}_{SV}} \alpha^{(k)^*} y^{(k)} k(\mathbf{x}^{(k)}, {\mathbf{x}})
\end{align}$$

Some common kernels include:

* **Gaussian**: $k(\mathbf{x},\mathbf{x}')=\exp(-\gamma \|\mathbf{x} - \mathbf{x}'\|^2)$, for $\gamma > 0$. Sometimes parametrized using $\gamma=\dfrac{1}{2 \sigma^2}$. This is by far the most widely used kernel.
* Polynomial (homogeneous): $k(\mathbf{x},\mathbf{x}')=(\mathbf{x}^\intercal \mathbf{x}')^d$
* Polynomial (inhomogeneous): $k(\mathbf{x},\mathbf{x}') = (\mathbf{x}^\intercal \mathbf{x}' + 1)^d$
* Hyperbolic tangent: $k(\mathbf{x},\mathbf{x}') = \tanh(\kappa \mathbf{x}^\intercal \mathbf{x}'+c)$, for some (not every) $\kappa > 0$ and $c < 0$.

### 6.1. Example.

  [Source: A [notebook](https://github.com/jakevdp/sklearn_pycon2015/blob/master/notebooks/03.1-Classification-SVMs.ipynb) by [Jake Vanderplas](https://github.com/jakevdp>)]

Where SVM gets incredibly exciting is when it is used in conjunction with *kernels*.
To motivate the need for kernels, let's look at some data which is not linearly separable:

```python
X, y = make_circles(100, factor=.1, noise=.1)

clf = svm.SVC(kernel='linear').fit(X, y)
plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf);
```

Clearly, no linear discrimination will ever separate these data.
One way we can adjust this is to apply a **kernel**, which is some functional transformation of the input data.

For example, one simple model we could use is a **radial basis function**

```python
r = np.exp(-(X[:, 0] ** 2 + X[:, 1] ** 2))
```

If we plot this along with our data, we can see the effect of it:

```python
def plot_3D(elev=30, azim=30):
    ax = plt.subplot(projection='3d')
    ax.scatter3D(X[:, 0], X[:, 1], r, c=y, s=50, cmap='spring')
    ax.view_init(elev=elev, azim=azim)
    ax.set_xlabel('x')
    ax.set_ylabel('y')
    ax.set_zlabel('r')

interact(plot_3D, elev=[-90, 90], azip=(-180, 180));
```

    interactive(children=(Dropdown(description='elev', options=(-90, 90), value=-90), IntSlider(value=30, descript…

We can see that with this additional dimension, the data becomes trivially linearly separable!
This is a relatively simple kernel; SVM has a more sophisticated version of this kernel built-in to the process. This is accomplished by using the Gaussian kernel (``kernel='rbf'``), short for *radial basis function*:

```python
clf = svm.SVC(kernel='rbf', C=10)
clf.fit(X, y)

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='copper')
plot_svc_decision_function(clf)
plt.scatter(clf.support_vectors_[:, 0], clf.support_vectors_[:, 1],
            s=200, facecolors='none');
```

Here there are effectively $N$ basis functions: one centered at each point! Through a clever mathematical trick, this computation proceeds very efficiently using the "Kernel Trick", without actually constructing the matrix of kernel evaluations.

**Exercise**: Apply the linear SVM and the SVM with Gaussian kernel to the discrimination of classes `Versicolor` and `Virginica` in the Iris Dataset, using atributes $x_0$ and $x_1$ only. Plot the corresponding decision boundaries and the support vectors.

## 7. Hyperparameters

Note that the SVM formulation has several free parameters (hyperparameters) that must be selected out of the optimization problem:

- The free parameter $C$ used to solve non-separable problems.
- The kernel parameters (for instance, parameter $\gamma$ from the Gaussian kernel.

These parameters are usually adjusted using a cross-validation procedure.