$\newcommand{\xv}{\mathbf{x}}
 \newcommand{\wv}{\mathbf{w}}
 \newcommand{\vv}{\mathbf{v}}
 \newcommand{\yv}{\mathbf{y}}
 \newcommand{\zv}{\mathbf{z}}
 \newcommand{\tv}{\mathbf{t}}
 \newcommand{\Chi}{\mathcal{X}}
 \newcommand{\R}{\rm I\!R}
 \newcommand{\sign}{\text{sign}}
 \newcommand{\Tm}{\mathbf{T}}
 \newcommand{\Xm}{\mathbf{X}}
 \newcommand{\Zm}{\mathbf{Z}}
 \newcommand{\Wm}{\mathbf{W}}
 \newcommand{\Ym}{\mathbf{Y}}
 \newcommand{\I}{\mathbf{I}}
 \newcommand{\muv}{\boldsymbol\mu}
 \newcommand{\Sigmav}{\boldsymbol\Sigma}
$

# Support Vector Machines (SVM)

What is a good classifier? There can be multiple hyperplanes that divides the data in two classes. SVMs, largin margin classifiers, choose the hyperplane that has a large margine.

The idea of support vector machines, or the large margin linear classifiers, first suggested in 1963 by Vapnik, et al. In 1992, [Boser, et al.](https://dl.acm.org/citation.cfm?id=130401), extended this to nonlinear classification and [Cortes, et al.](http://image.diku.dk/imagecanon/material/cortes_vapnik95.pdf), further examines non-separable data in 1995. After this, lots of large margin algorithms have been developed.

Let us take a look at the classifier choice with a largin margin and talk about other issues.

## What is the best classifier?

## Large Margin Classifier

Margin is the distance or room between the hyperplane and the closest data samples from each class.
The margin gives the classifer flexibility to changes of data or variant future data. Thus, having the largest margin can be a good choice.

# Geometric Margin

The margin, $d$, can be calculated by the linear projection with a unit vector in the direction of $\wv$, $\hat{\wv}$.

Let us say that there is a point $\zv$ on a line,
$$
\wv^\top \zv + b = 0.
$$

Also, we assume that $\zv$ is orthogoal to a data point $\xv$:
$$
\zv = \xv + d \hat{\wv}.
$$

Plugging in this to the hyperplane equation $\yv = \wv^\top \xv + b$,

$$
\begin{align*}
\wv^\top \zv + b &= \wv^\top (\xv + d \hat{\wv}) + b \\
0 &= \wv^\top \xv + d \wv^\top \hat{\wv} + b \\
0 &= y + d \wv^\top \frac{\wv}{\Vert \wv \Vert} = y + d \frac{\Vert \wv \Vert^2}{\Vert \wv \Vert}\\
d &= -\frac{y}{\Vert \wv \Vert}
\end{align*}
$$

Now, let us check the sign of the label. We assume that the data point $\xv$ here was from negative samples.
That is, the $\yv$ is negative.
Looking back our perceptron, we use our label to unify the objective function:

$$
t_i y_i = \tv_i (\wv^\top \xv_i + b) \gt 0.
$$

If we rescaling the weights and bias, we can get the following canonical hyperplane with the minimum distance to be 1:

$$
\min_{n=1, \dots, N} t_n (\wv^\top \xv + b) = 1.
$$

Under this assumption, we get the margin that equals to

$$d = \frac{1}{\Vert \wv \Vert}.$$

Remember that we want *maximize* this margine $\frac{1}{\Vert \wv \Vert}$, so we can invert this to *minimize*  $\Vert \wv \Vert$.

## The Linear SVM

The objective function can be written as:

$$
\begin{equation*}
\begin{aligned}
& \underset{\wv, b}{\text{minimize}}
& & \frac{1}{2} \Vert \wv \Vert^2 \\
& \text{subject to}
& & t_i (\wv^\top \xv_i + b) \ge 1, \; i = 1, \ldots, N.
\end{aligned}
\end{equation*}
$$

The Lagrangian is

$$
L = \frac{\Vert \wv \Vert^2 }{2} - \sum_i^N \lambda_i \big( t_i (\wv^\top \xv_i + b) - 1 \big).
$$

Let us find the optimal solution with gradient!
With positive Lagrangian multipliers $\lambda = [\lambda_1, \ldots, \lambda_N]$ and the constraint,
we can find the two additional gradient against $\lambda$ and $\wv$ and set to to zero to obtain the optimal conditions. That is,

$$
\begin{cases}
\frac{\partial L}{\partial \wv} = 0, \\
\frac{\partial L}{\partial b} = 0, \\
t_i (\wv^\top \xv_i + b) - 1 \ge 0, \\
\lambda \ge 0, \\
\lambda_i \big(t_i (\wv^\top \xv_i + b) - 1\big) = 0.
\end{cases}
$$

This is called **Karush–Kuhn–Tucker (KKT)** conditions (See http://www.svms.org/kkt/ and http://mat.gsia.cmu.edu/classes/QUANT/NOTES/chap4/node6.html).

Going through few steps for the derivaiton,
$$
\begin{align*}
\frac{\partial L}{\partial \wv} &= \wv - \sum_i^N \lambda_i t_i \xv_i = 0\\
\wv &= \sum_i^N \lambda_i t_i \xv_i,
\end{align*}
$$
and
$$
\begin{align*}
\frac{\partial L}{\partial b} &= - \sum_i^N \lambda_i t_i = 0\\
    \sum_i^N \lambda_i t_i &= 0.
\end{align*}
$$

Rewriting and summarizing the KKT conditions,

$$
\begin{align*}
&\wv = \sum_i^N \lambda_i t_i \xv_i, \\
&\sum_i^N \lambda_i t_i = 0, \\
&t_i (\wv^\top \xv_i + b) - 1 \ge 0, \\
\\
&\lambda \ge 0, \\
\\
&\lambda_i \big(t_i (\wv^\top \xv_i + b) - 1\big) = 0.
&\end{align*}
$$

# Dual Problem

For duality convention, let us change the notation $\lambda$ to $\alpha$:

$$
\begin{equation}
    \wv = \sum_i^N \alpha_i t_i \xv_i. \label{eq_w} \tag{1}
\end{equation}
$$

Since there is no solution for $b$, use the KKT condition and find it:

$$
\begin{align}
\lambda_i \big(t_i (\wv^\top \xv_i + b) - 1\big) &= 0 \\
t_i (\wv^\top \xv_i + b) &= 1 \\
\wv^\top \xv_i + b = t_i \\
\\
b = t_i - \wv^\top \xv_i. \label{eq_b} \tag{2}
\end{align}
$$

With the KKT conditions,
$$
\alpha_i > 0 \\
\text{and}\\
t_i (\wv^\top \xv + b) = 1,
$$
the $\xv_i$ are the data samples that lie on the re-weighted margin *1*.

The corresponding $\xv_i$ are called **support vectors**, that support the hyperplane construction.

Now, using the KKT conditions, replace $\wv$ and $b$ in the objective function $L$ with $\eqref{eq_w}$ and $\eqref{eq_b}$,

$$
\begin{align*}
L &= \frac{\Vert \wv \Vert^2 }{2} - \sum_i^N \alpha_i \big( t_i (\wv^\top \xv_i + b) - 1 \big) \\
  &= \frac{1}{2} \Big( \sum_i^N \alpha_i t_i \xv_i \Big)^\top \Big( \sum_i^N \alpha_i t_i \xv_i \Big)
     - \sum_i^N \alpha_i t_i \Big( \sum_j^N \alpha_j t_j \xv_j \Big)^\top \xv_i
     - b \sum_i^N \alpha_i t_i
     + \sum_i^N \alpha_i \\
  &= \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j  \xv_i^\top \xv_j
     - \sum_i^N \sum_j^N  \alpha_i \alpha_j t_i t_j \xv_i^\top  \xv_j
     - b \sum_i^N \alpha_i t_i
     + \sum_i^N \alpha_i \\
  &= - \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j  \xv_i^\top \xv_j + \sum_i^N \alpha_i.
\end{align*}
$$

Let us rewrite the optimization problem in duality:

$$
\begin{equation*}
\begin{aligned}
& \underset{\alpha}{\text{maximize}}
& & \sum_i^N \alpha_i - \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j  \xv_i^\top \xv_j \\
& \text{subject to}
& & \sum_i^N \alpha_i t_i = 0, \\
& & &\alpha_i \ge 0.
\end{aligned}
\end{equation*}
$$

# Soft Margin SVM

So far, we constrained the data is clearly separable:

$$
t_i (\wv^\top \xv_i + b) \ge 1.
$$

To relax this constraints, we can define additional slack variables $\xi_i \ge 0$:

$$
t_i (\wv^\top \xv_i + b) \ge 1 - \xi_i.
$$

With the slack variables, the objective function is

$$
\begin{equation*}
\begin{aligned}
& \underset{\wv, b, \xi}{\text{minimize}}
& & \frac{1}{2} \Vert \wv \Vert^2 + C \sum_i \xi_i \\
& \text{subject to}
& & t_i (\wv^\top \xv_i + b) \ge 1 - \xi_i,\\
& & & \xi_i \ge 0 \quad \; i = 1, \ldots, N.
\end{aligned}
\end{equation*}
$$

From the Lagrangin,

$$
L = \frac{\Vert \wv \Vert^2 }{2} - \sum_i^N \alpha_i \big( t_i (\wv^\top \xv_i + b) - 1 + \xi_i \big) - \sum_i^N \beta_i \xi_i + C \sum_i^N \xi_i,
$$

we get the same saddle point equations as in $\eqref{eq_w}$ and $\eqref{eq_b}$ along with the following:

$$
\frac{\partial L}{\partial \xi_i} = C - \alpha_i - \beta_i = 0.
$$

This leads us to dual problem of

$$
\begin{equation*}
\begin{aligned}
& \underset{\alpha}{\text{maximize}}
& & \sum_i^N \alpha_i - \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j  \xv_i^\top \xv_j \\
& \text{subject to}
& & \sum_i^N \alpha_i t_i = 0, \\
& & &\alpha_i \ge 0, \\
& & &\beta_i \ge 0, \\
& & &C - \alpha_i - \beta_i = 0.
\end{aligned}
\end{equation*}
$$

Using the last saddle point equaiton, we can replace $\beta_i$ inequality with the
$$
0 \le \alpha_i \le C.
$$

$$
\begin{equation*}
\begin{aligned}
& \underset{\alpha}{\text{maximize}}
& & \sum_i^N \alpha_i - \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j  \xv_i^\top \xv_j \\
& \text{subject to}
& & \sum_i^N \alpha_i t_i = 0, \\
& & &0 \le \alpha_i \le C.
\end{aligned}
\end{equation*}
$$

# Kernels

For the data that cannot be classified with a linear classifier, we used a non-linear function mapping from $\xv$ to $\phi(\xv)$. We can use this nonlinear mapping for the SVM as well:

$$
\begin{equation*}
\begin{aligned}
& \underset{\wv, b, \xi}{\text{minimize}}
& & \frac{1}{2} \Vert \wv \Vert^2 + C \sum_i \xi_i \\
& \text{subject to}
& & t_i (\wv^\top \phi(\xv_i) + b) \ge 1 - \xi_i,\\
& & & \xi_i \ge 0 \quad \; i = 1, \ldots, N.
\end{aligned}
\end{equation*}
$$

In duality, we can write the optimization problem as

$$
\begin{equation*} \
\begin{aligned}
& \underset{\alpha}{\text{maximize}}
& & \sum_i^N \alpha_i - \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j  \phi(\xv_i)^\top \phi(\xv_j) \\
& \text{subject to}
& & \sum_i^N \alpha_i t_i = 0, \\
& & &0 \le \alpha_i \le C.
\end{aligned}
\end{equation*}
$$

In nonlinear feature space $\Phi$, the inner product $\phi^\top \phi$ compute the euclidean distance between betweeen two feature vectors. We define this distance, or similarity measure as a kernel $k$:

$$
k(\xv, \vv) = \phi(\xv)^\top \phi(\vv).
$$

With this kernel notation, the dual problem can be written as

$$
\begin{equation*}
\begin{aligned}
& \underset{\alpha}{\text{maximize}}
& & \sum_i^N \alpha_i - \frac{1}{2} \sum_i^N \sum_j^N \alpha_i \alpha_j t_i t_j k(\xv_i, \xv_j) \\
& \text{subject to}
& & \sum_i^N \alpha_i t_i = 0, \\
& & &0 \le \alpha_i \le C.
\end{aligned}
\end{equation*}
$$

Here are summary of well-known stard kernels.
- Linear Kernel
$$ k(\xv, \yv) = \xv^\top \yv $$
- Polynomial Kernel
$$ k(\xv, \yv) = (\xv^\top \yv + 1)^d $$
- Gaussian Kernel
$$ k(\xv, \yv) = \exp(-\gamma \Vert \xv - \yv \Vert^2) $$

# Then, How do we solve this?

So far, we set up the optimization problems. How do we need to solve or program?
Any of optimization tools can be used to solve this, but there are some fast optimization tools for the SVM including the sequential minimal optimization (SMO) and libsvm.

Scikit-learn uses liblinear for the primal and libsvm for the dual problem.

Now, let us implement svm and try to classify the circles data. You can use any libsvm wrapper like scikit-learn, but for exercise, let us use gradient descent for learning.

The derivative of the objective function w.r.t. $\alpha$ is

$$
\frac{\partial L}{\partial \alpha_i} = 1 - t_i \sum_j^N \alpha_j t_j k(\xv_i, \xv_j) - \sum_j^N t_i.
$$

From

$$
\wv = \sum_i^N \alpha_i t_i \xv_i,
$$

$$
\wv^\top \xv_i = \sum_j^N \alpha_j t_j \xv_j^\top \xv_i = \sum_j^N \alpha_j t_j k(\xv_j, \xv_i).
$$
Thus,
$$
b = t_i - \wv^\top \xv_i =
t_i - \sum_j^N \alpha_j t_j k(\xv_j, \xv_i),
$$

we can make a prediction using the $\wv$, $b$, and the kernel evaluations.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
from sklearn.datasets import make_circles

X, T = make_circles(n_samples=800, noise=0.07, factor=0.4)
T[T==0] = -1

plt.figure(figsize=(8, 8))
plt.scatter(X[:, 0], X[:, 1], marker='o', c=T)
plt.title("Circles")
```

```python
# linear kernel
def linear_kernel(x, y):
    return x @ y

def polynomial_kernel(x, y, d=3):
    return (x @ y +1) ** d

def gaussian_kernel(x, y, r=1.):
    d = (x - y)
    return np.exp(-r * (d @ d))
```

```python
from functools import partial

kernelf = partial(linear_kernel)

###### build a kernel matrix (optional)
def build_Kmat(X, Xn=X, kernelf=linear_kernel):
    N = X.shape[0]
    Nn = Xn.shape[0]

    K = # TODO: finish this line to create a kernel matrix
    return K
```

```python
K = build_Kmat(X, X, kernelf)
T.shape, K[0, :].shape
```

    ((800,), (800,))

```python
def train(X, T, K, n_epochs=1, learning_rate=0.01, C=10):

    N = X.shape[0]

    #alpha = np.random.rand(N)
    alpha = np.zeros(N)
    T_sum = np.sum(T)
    for n in range(n_epochs):
        for i in range(N):
            grad = # TODO: fill this gradient
            alpha += learning_rate * grad

            alpha[alpha <= 0] = 0
            alpha[alpha >= C] = C
    b = # TODO: Finish this b
    return alpha, b

```

```python
# predict
#
# Parameters
# ===========
# Xtest    test input
# alpha    SVM learned parameters
# X        Support Vectors
# T        Target for SVs
# K        kernel matrix if exists
def predict(Xtest, alpha, b, **params):
    # support vectors
    i_sv = np.where(np.logical_and(alpha<C, alpha>0))[0]

    K = params.pop('K', None)
    T = params.pop('T', None)
    if T is None:
        raise ValueError("predict: None support vector targets!")

    if K is None:
        kernelf = params.pop('kernelf', linear_kernel)
        X = params.pop('X', None)
        if X is None:
            raise ValueError("predict: None support vectors!")
        print("rebuilding kernel matrix")
        K = build_Kmat(X, Xtest, kernelf)

    y = # TP
    return y

```

```python
# train
alpha, b = train(X, T, K)

# prediction
y = predict(X, alpha, b, K=K, T=T)

# plot
plt.plot(T)
plt.plot(np.sign(y))
plt.show()

print("Accuracy: ", 100 * np.sum(T == np.sign(y)) / N, "%")
```

```python

x = np.linspace(-1.5, 1.5, 100)
y = np.linspace(-1.5, 1.5, 100)

xs, ys = np.meshgrid(x, y)

Xt = np.vstack((xs.flat, ys.flat)).T

Y = predict(Xt, alpha, b, X=X, T=T)
classes = np.sign(Y)

zs = classes.reshape(xs.shape)

plt.figure(figsize=(6,6))
plt.contourf(xs, ys, zs.reshape(xs.shape), alpha=0.3)
plt.title("Decision Boundary")

plt.scatter(X[:, 0], X[:, 1], marker='o', c=T+3)
```

```python
# nonlinear now
gausskf = partial(gaussian_kernel, r=0.1)
K = build_Kmat(X, X, gausskf)
```

```python
alpha, b = train(X, T, K)
```

```python
# predict
y = predict(X, alpha, b, K=Kg, T=T)
print("Accuracy: ", 100 * np.sum(T == np.sign(y)) / N, "%")
```

    Accuracy:  100.0 %

```python

x = np.linspace(-1.5, 1.5, 100)
y = np.linspace(-1.5, 1.5, 100)

xs, ys = np.meshgrid(x, y)

Xt = np.vstack((xs.flat, ys.flat)).T

Y = predict(Xt, alpha, b, X=X, T=T, kernelf=gausskf)
classes = np.sign(Y)

zs = classes.reshape(xs.shape)

plt.figure(figsize=(6,6))
plt.contourf(xs, ys, zs.reshape(xs.shape), alpha=0.3)
plt.title("Decision Boundary")

plt.scatter(X[:, 0], X[:, 1], marker='o', c=T+3)
```


```python
import numpy as np
import sympy as sy
sy.init_printing()
```

```python
n = sy.Symbol('n')
```

$$
  \left[
    \begin{array}{rrr}
      a_{n+2} \\
      a_{n+1}
    \end{array}
  \right]
  = \left[
    \begin{array}{rrr}
      2 & 8 \\
      1 & 0 \\
    \end{array}
  \right]
  \left[
    \begin{array}{rrr}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
$$
を変形すると
$$
  \left[
    \begin{array}{}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
  = \left[
    \begin{array}{}
      2 & 8 \\
      1 & 0 \\
    \end{array}
  \right] ^ {n-1}
  \left[
    \begin{array}{}
      a_{2} \\
      a_{1}
    \end{array}
  \right]
$$

```python
A = sy.Matrix([[2, 8], [1, 0]])
A
```

$$\left[\begin{matrix}2 & 8\\1 & 0\end{matrix}\right]$$

$$
  \left[
    \begin{array}{rrr}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
= A^{n-1} \left[
    \begin{array}{}
      a_{2} \\
      a_{1}
    \end{array}
  \right]
$$

```python
A ** (n-1) * sy.Matrix([3, 1])
```

$$\left[\begin{matrix}- \frac{\left(-2\right)^{n - 1}}{3} + \frac{10 \cdot 4^{n - 1}}{3}\\\frac{\left(-2\right)^{n - 1}}{6} + \frac{5 \cdot 4^{n - 1}}{6}\end{matrix}\right]$$

このように求めることができた(2行目が `n` の一般項)

でも、これでは面白くないし、人力では厳しいので対角化して解いてみることにする

```python
# スペクトル分解して、固有値と固有ベクトルを求める
# 固有値, 重複度, 固有ベクトル(重複度分)
evs = A.eigenvects()
evs
```

$$\left [ \left ( -2, \quad 1, \quad \left [ \left[\begin{matrix}-2\\1\end{matrix}\right]\right ]\right ), \quad \left ( 4, \quad 1, \quad \left [ \left[\begin{matrix}4\\1\end{matrix}\right]\right ]\right )\right ]$$

```python
P = sy.Matrix([ev[-1][0].values() for ev in evs]).T
P
```

$$\left[\begin{matrix}-2 & 4\\1 & 1\end{matrix}\right]$$

```python
P ** -1
```

$$\left[\begin{matrix}- \frac{1}{6} & \frac{2}{3}\\\frac{1}{6} & \frac{1}{3}\end{matrix}\right]$$

```python
# 6倍したもの
P ** -1 * 6
```

$$\left[\begin{matrix}-1 & 4\\1 & 2\end{matrix}\right]$$

$$ D = P^{-1} A P $$

```python
D = P ** (-1) * A * P
D
```

$$\left[\begin{matrix}-2 & 0\\0 & 4\end{matrix}\right]$$

D を用いて

$$ A^{n-1} = P D^{n-1} P^{-1} $$

のように表現できる

```python
# これを計算すると同様の結果を得られる
(P) * (D ** (n-1)) * (P ** -1) * sy.Matrix([3, 1])
```

$$\left[\begin{matrix}- \frac{\left(-2\right)^{n - 1}}{3} + \frac{10 \cdot 4^{n - 1}}{3}\\\frac{\left(-2\right)^{n - 1}}{6} + \frac{5 \cdot 4^{n - 1}}{6}\end{matrix}\right]$$

ちなみに
$$
  \left[
    \begin{array}{rrr}
      a_{n+2} \\
      a_{n+1}
    \end{array}
  \right]= A
  \left[
    \begin{array}{rrr}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
$$

を変形して

$$
  \left[
    \begin{array}{rrr}
      a_{n+2} \\
      a_{n+1}
    \end{array}
  \right] = P D P^{-1}
  \left[
    \begin{array}{rrr}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
$$

$$ P^{-1}
  \left[
    \begin{array}{rrr}
      a_{n+2} \\
      a_{n+1}
    \end{array}
  \right]
= D P^{-1}
  \left[
    \begin{array}{rrr}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
$$

$$
  \left[
    \begin{array}{}
      -1 & 4 \\
      1 & 2 \\
    \end{array}
  \right]
  \left[
    \begin{array}{rrr}
      a_{n+2} \\
      a_{n+1}
    \end{array}
  \right]
  = \left[
    \begin{array}{}
      -2 & 0 \\
      0 & 4 \\
    \end{array}
  \right]
  \left[
    \begin{array}{}
      -1 & 4 \\
      1 & 2 \\
    \end{array}
  \right]
  \left[
    \begin{array}{rrr}
      a_{n+1} \\
      a_{n}
    \end{array}
  \right]
$$
(両辺を6倍してある)

漸化式の途中式と一致しているのがわかる
$$
\begin{eqnarray}
\left\{
\begin{array}{l}
  a_{n+2} - 4a_{n+1} = -2(a_{n+1} - 4a_n) \\
  a_{n+2} + 2a_{n+1} =  4(a_{n+1} + 2a_n)
\end{array}
\right.
\end{eqnarray}
$$

```python

```


# Part 1 - Scalars and Vectors

For the questions below it is not sufficient to simply provide answer to the questions, but you must solve the problems and show your work using python (the NumPy library will help a lot!) Translate the vectors and matrices into their appropriate python  representations and use numpy or functions that you write yourself to demonstrate the result or property.

## 1.1 Create a two-dimensional vector and plot it on a graph

```

```

## 1.2 Create a three-dimensional vecor and plot it on a graph

```

```

## 1.3 Scale the vectors you created in 1.1 by $5$, $\pi$, and $-e$ and plot all four vectors (original + 3 scaled vectors) on a graph. What do you notice about these vectors?

```
from math import e, pi
print(e)
print(pi)
```

    2.718281828459045
    3.141592653589793

```

```

## 1.4 Graph vectors $\vec{a}$ and $\vec{b}$ and plot them on a graph

\begin{align}
\vec{a} = \begin{bmatrix} 5 \\ 7 \end{bmatrix}
\qquad
\vec{b} = \begin{bmatrix} 3 \\4 \end{bmatrix}
\end{align}

```

```

## 1.5 find $\vec{a} - \vec{b}$ and plot the result on the same graph as $\vec{a}$ and $\vec{b}$. Is there a relationship between vectors $\vec{a} \thinspace, \vec{b} \thinspace \text{and} \thinspace \vec{a-b}$

```

```

## 1.6 Find $c \cdot d$

\begin{align}
\vec{c} = \begin{bmatrix}7 & 22 & 4 & 16\end{bmatrix}
\qquad
\vec{d} = \begin{bmatrix}12 & 6 & 2 & 9\end{bmatrix}
\end{align}

```

```

##  1.7 Find $e \times f$

\begin{align}
\vec{e} = \begin{bmatrix} 5 \\ 7 \\ 2 \end{bmatrix}
\qquad
\vec{f} = \begin{bmatrix} 3 \\4 \\ 6 \end{bmatrix}
\end{align}

```

```

## 1.8 Find $||g||$ and then find $||h||$. Which is longer?

\begin{align}
\vec{g} = \begin{bmatrix} 1 \\ 1 \\ 1 \\ 8 \end{bmatrix}
\qquad
\vec{h} = \begin{bmatrix} 3 \\3 \\ 3 \\ 3 \end{bmatrix}
\end{align}

```

```

# Part 2 - Matrices

## 2.1 What are the dimensions of the following matrices? Which of the following can be multiplied together? See if you can find all of the different legal combinations.
\begin{align}
A = \begin{bmatrix}
1 & 2 \\
3 & 4 \\
5 & 6
\end{bmatrix}
\qquad
B = \begin{bmatrix}
2 & 4 & 6 \\
\end{bmatrix}
\qquad
C = \begin{bmatrix}
9 & 6 & 3 \\
4 & 7 & 11
\end{bmatrix}
\qquad
D = \begin{bmatrix}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 0 & 1
\end{bmatrix}
\qquad
E = \begin{bmatrix}
1 & 3 \\
5 & 7
\end{bmatrix}
\end{align}

```

```

## 2.2 Find the following products: CD, AE, and BA. What are the dimensions of the resulting matrices? How does that relate to the dimensions of their factor matrices?

```

```

## 2.3  Find $F^{T}$. How are the numbers along the main diagonal (top left to bottom right) of the original matrix and its transpose related? What are the dimensions of $F$? What are the dimensions of $F^{T}$?

\begin{align}
F =
\begin{bmatrix}
20 & 19 & 18 & 17 \\
16 & 15 & 14 & 13 \\
12 & 11 & 10 & 9 \\
8 & 7 & 6 & 5 \\
4 & 3 & 2 & 1
\end{bmatrix}
\end{align}

```

```

# Part 3 - Square Matrices

## 3.1 Find $IG$ (be sure to show your work) 😃

You don't have to do anything crazy complicated here to show your work, just create the G matrix as specified below, and a corresponding 2x2 Identity matrix and then multiply them together to show the result. You don't need to write LaTeX or anything like that (unless you want to).

\begin{align}
G=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\end{align}

```

```

## 3.2 Find $|H|$ and then find $|J|$.

\begin{align}
H=
\begin{bmatrix}
12 & 11 \\
7 & 10
\end{bmatrix}
\qquad
J=
\begin{bmatrix}
0 & 1 & 2 \\
7 & 10 & 4 \\
3 & 2 & 0
\end{bmatrix}
\end{align}

```

```

## 3.3 Find $H^{-1}$ and then find $J^{-1}$

```

```

## 3.4 Find $HH^{-1}$ and then find $J^{-1}J$. Is $HH^{-1} == J^{-1}J$? Why or Why not?

Please ignore Python rounding errors. If necessary, format your output so that it rounds to 5 significant digits (the fifth decimal place).

# Stretch Goals:

A reminder that these challenges are optional. If you finish your work quickly we welcome you to work on them. If there are other activities that you feel like will help your understanding of the above topics more, feel free to work on that. Topics from the Stretch Goals sections will never end up on Sprint Challenges. You don't have to do these in order, you don't have to do all of them.

- Write a function that can calculate the dot product of any two vectors of equal length that are passed to it.
- Write a function that can calculate the norm of any vector
- Prove to yourself again that the vectors in 1.9 are orthogonal by graphing them.
- Research how to plot a 3d graph with animations so that you can make the graph rotate (this will be easier in a local notebook than in google colab)
- Create and plot a matrix on a 2d graph.
- Create and plot a matrix on a 3d graph.
- Plot two vectors that are not collinear on a 2d graph. Calculate the determinant of the 2x2 matrix that these vectors form. How does this determinant relate to the graphical interpretation of the vectors?


# Statistical Data Modeling

Some or most of you have probably taken some undergraduate- or graduate-level statistics courses. Unfortunately, the curricula for most introductory statisics courses are mostly focused on conducting **statistical hypothesis tests** as the primary means for interest: t-tests, chi-squared tests, analysis of variance, etc. Such tests seek to esimate whether groups or effects are "statistically significant", a concept that is poorly understood, and hence often misused, by most practioners. Even when interpreted correctly, statistical significance is a questionable goal for statistical inference, as it is of limited utility.

A far more powerful approach to statistical analysis involves building flexible **models** with the overarching aim of *estimating* quantities of interest. This section of the tutorial illustrates how to use Python to build statistical models of low to moderate difficulty from scratch, and use them to extract estimates and associated measures of uncertainty.

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt

# Set some Pandas options
pd.set_option('display.notebook_repr_html', False)
pd.set_option('display.max_columns', 20)
pd.set_option('display.max_rows', 25)
```

Estimation
==========

An recurring statistical problem is finding estimates of the relevant parameters that correspond to the distribution that best represents our data.

In **parametric** inference, we specify *a priori* a suitable distribution, then choose the parameters that best fit the data.

* e.g. $\mu$ and $\sigma^2$ in the case of the normal distribution

```python
x = np.array([ 1.00201077,  1.58251956,  0.94515919,  6.48778002,  1.47764604,
        5.18847071,  4.21988095,  2.85971522,  3.40044437,  3.74907745,
        1.18065796,  3.74748775,  3.27328568,  3.19374927,  8.0726155 ,
        0.90326139,  2.34460034,  2.14199217,  3.27446744,  3.58872357,
        1.20611533,  2.16594393,  5.56610242,  4.66479977,  2.3573932 ])
_ = plt.hist(x, bins=8)
```

### Fitting data to probability distributions

We start with the problem of finding values for the parameters that provide the best fit between the model and the data, called point estimates. First, we need to define what we mean by ‘best fit’. There are two commonly used criteria:

* **Method of moments** chooses the parameters so that the sample moments (typically the sample mean and variance) match the theoretical moments of our chosen distribution.
* **Maximum likelihood** chooses the parameters to maximize the likelihood, which measures how likely it is to observe our given sample.

### Discrete Random Variables

$$X = \{0,1\}$$

$$Y = \{\ldots,-2,-1,0,1,2,\ldots\}$$

**Probability Mass Function**:

For discrete $X$,

$$Pr(X=x) = f(x|\theta)$$

***e.g. Poisson distribution***

The Poisson distribution models unbounded counts:

$$Pr(X=x)=\frac{e^{-\lambda}\lambda^x}{x!}$$

* $X=\{0,1,2,\ldots\}$
* $\lambda > 0$

$$E(X) = \text{Var}(X) = \lambda$$

### Continuous Random Variables

$$X \in [0,1]$$

$$Y \in (-\infty, \infty)$$

**Probability Density Function**:

For continuous $X$,

$$Pr(x \le X \le x + dx) = f(x|\theta)dx \, \text{ as } \, dx \rightarrow 0$$

***e.g. normal distribution***

$$f(x) = \frac{1}{\sqrt{2\pi\sigma^2}}\exp\left[-\frac{(x-\mu)^2}{2\sigma^2}\right]$$

* $X \in \mathbf{R}$
* $\mu \in \mathbf{R}$
* $\sigma>0$

$$\begin{align}E(X) &= \mu \cr
\text{Var}(X) &= \sigma^2 \end{align}$$

### Example: Nashville Precipitation

The dataset `nashville_precip.txt` contains [NOAA precipitation data for Nashville measured since 1871](http://bit.ly/nasvhville_precip_data). The gamma distribution is often a good fit to aggregated rainfall data, and will be our candidate distribution in this case.

```python
precip = pd.read_table("data/nashville_precip.txt", index_col=0, na_values='NA', delim_whitespace=True)
precip.head()
```

           Jan   Feb   Mar    Apr   May   Jun   Jul   Aug   Sep   Oct   Nov   Dec
    Year
    1871  2.76  4.58  5.01   4.13  3.30  2.98  1.58  2.36  0.95  1.31  2.13  1.65
    1872  2.32  2.11  3.14   5.91  3.09  5.17  6.10  1.65  4.50  1.58  2.25  2.38
    1873  2.96  7.14  4.11   3.59  6.31  4.20  4.63  2.36  1.81  4.28  4.36  5.94
    1874  5.22  9.23  5.36  11.84  1.49  2.87  2.65  3.52  3.12  2.63  6.12  4.19
    1875  6.15  3.06  8.14   4.22  1.73  5.63  8.12  1.60  3.79  1.25  5.46  4.30

```python
_ = precip.hist(sharex=True, sharey=True, grid=False)
plt.tight_layout()
```

The first step is recognixing what sort of distribution to fit our data to. A couple of observations:

1. The data are skewed, with a longer tail to the right than to the left
2. The data are positive-valued, since they are measuring rainfall
3. The data are continuous

There are a few possible choices, but one suitable alternative is the **gamma distribution**:

$$x \sim \text{Gamma}(\alpha, \beta) = \frac{\beta^{\alpha}x^{\alpha-1}e^{-\beta x}}{\Gamma(\alpha)}$$

The ***method of moments*** simply assigns the empirical mean and variance to their theoretical counterparts, so that we can solve for the parameters.

So, for the gamma distribution, the mean and variance are:

$$ \hat{\mu} = \bar{X} = \alpha \beta $$
$$ \hat{\sigma}^2 = S^2 = \alpha \beta^2 $$

So, if we solve for these parameters, we can use a gamma distribution to describe our data:

$$ \alpha = \frac{\bar{X}^2}{S^2}, \, \beta = \frac{S^2}{\bar{X}} $$

Let's deal with the missing value in the October data. Given what we are trying to do, it is most sensible to fill in the missing value with the average of the available values.

```python
precip.fillna(value={'Oct': precip.Oct.mean()}, inplace=True)
```

            Jan    Feb   Mar    Apr    May   Jun   Jul   Aug    Sep   Oct   Nov  \
    Year
    1871   2.76   4.58  5.01   4.13   3.30  2.98  1.58  2.36   0.95  1.31  2.13
    1872   2.32   2.11  3.14   5.91   3.09  5.17  6.10  1.65   4.50  1.58  2.25
    1873   2.96   7.14  4.11   3.59   6.31  4.20  4.63  2.36   1.81  4.28  4.36
    1874   5.22   9.23  5.36  11.84   1.49  2.87  2.65  3.52   3.12  2.63  6.12
    1875   6.15   3.06  8.14   4.22   1.73  5.63  8.12  1.60   3.79  1.25  5.46
    1876   6.41   2.22  5.28   3.62   3.40  5.65  7.15  5.77   2.52  2.68  1.26
    1877   4.05   1.06  4.98   9.47   1.25  6.02  3.25  4.16   5.40  2.61  4.93
    1878   3.34   2.10  3.48   6.88   2.33  3.28  9.43  5.02   1.28  2.17  3.20
    1879   6.32   3.13  3.81   2.88   2.88  2.50  8.47  4.62   5.18  2.90  5.85
    1880   3.74  12.37  8.16   5.26   4.13  3.97  5.69  2.22   5.39  7.24  5.77
    1881   3.54   5.48  2.79   5.12   3.67  3.70  0.86  1.81   6.57  4.80  4.89
    1882  14.51   8.61  9.38   3.59   7.38  2.54  4.06  5.54   1.61  1.11  3.60
    ...     ...    ...   ...    ...    ...   ...   ...   ...    ...   ...   ...
    2000   3.52   3.75  3.34   6.23   7.66  1.74  2.25  1.95   1.90  0.26  6.39
    2001   3.21   8.54  2.73   2.42   5.54  4.47  2.77  4.07   1.79  4.61  5.09
    2002   4.93   1.99  9.40   4.31   3.98  3.76  5.64  3.13   6.29  4.48  2.91
    2003   1.59   8.47  2.30   4.69  10.73  7.08  2.87  3.88   8.70  1.80  4.17
    2004   3.60   5.77  4.81   6.69   6.90  3.39  3.19  4.24   4.55  4.90  5.21
    2005   4.42   3.84  3.90   6.93   1.03  2.70  2.39  6.89   1.44  0.02  3.29
    2006   6.57   2.69  2.90   4.14   4.95  2.19  2.64  5.20   4.00  2.98  4.05
    2007   3.32   1.84  2.26   2.75   3.30  2.37  1.47  1.38   1.99  4.95  6.20
    2008   4.76   2.53  5.56   7.20   5.54  2.21  4.32  1.67   0.88  5.03  1.75
    2009   4.59   2.85  2.92   4.13   8.45  4.53  6.03  2.14  11.08  6.49  0.67
    2010   4.13   2.77  3.52   3.48  16.43  4.96  5.86  6.99   1.17  2.49  5.41
    2011   2.31   5.54  4.59   7.51   4.38  5.04  3.46  1.78   6.20  0.93  6.15

           Dec
    Year
    1871  1.65
    1872  2.38
    1873  5.94
    1874  4.19
    1875  4.30
    1876  0.95
    1877  2.49
    1878  6.04
    1879  9.15
    1880  3.32
    1881  4.85
    1882  1.52
    ...    ...
    2000  3.44
    2001  3.32
    2002  5.81
    2003  3.19
    2004  5.93
    2005  2.46
    2006  3.41
    2007  3.83
    2008  6.72
    2009  3.99
    2010  1.87
    2011  4.25

    [141 rows x 12 columns]

Now, let's calculate the sample moments of interest, the means and variances by month:

```python
precip_mean = precip.mean()
precip_mean
```

    Jan    4.523688
    Feb    4.097801
    Mar    4.977589
    Apr    4.204468
    May    4.325674
    Jun    3.873475
    Jul    3.895461
    Aug    3.367305
    Sep    3.377660
    Oct    2.610500
    Nov    3.685887
    Dec    4.176241
    dtype: float64

```python
precip_var = precip.var()
precip_var
```

    Jan    6.928862
    Feb    5.516660
    Mar    5.365444
    Apr    4.117096
    May    5.306409
    Jun    5.033206
    Jul    3.777012
    Aug    3.779876
    Sep    4.940099
    Oct    2.741659
    Nov    3.679274
    Dec    5.418022
    dtype: float64

We then use these moments to estimate $\alpha$ and $\beta$ for each month:

```python
alpha_mom = precip_mean ** 2 / precip_var
beta_mom = precip_var / precip_mean
```

```python
alpha_mom, beta_mom
```

    (Jan    2.953407
     Feb    3.043866
     Mar    4.617770
     Apr    4.293694
     May    3.526199
     Jun    2.980965
     Jul    4.017624
     Aug    2.999766
     Sep    2.309383
     Oct    2.485616
     Nov    3.692511
     Dec    3.219070
     dtype: float64, Jan    1.531684
     Feb    1.346249
     Mar    1.077920
     Apr    0.979219
     May    1.226724
     Jun    1.299403
     Jul    0.969593
     Aug    1.122522
     Sep    1.462581
     Oct    1.050243
     Nov    0.998206
     Dec    1.297344
     dtype: float64)

We can use the `gamma.pdf` function in `scipy.stats.distributions` to plot the ditribtuions implied by the calculated alphas and betas. For example, here is January:

```python
from scipy.stats.distributions import gamma

precip.Jan.hist(normed=True, bins=20)
plt.plot(np.linspace(0, 10), gamma.pdf(np.linspace(0, 10), alpha_mom[0], beta_mom[0]))
```

Looping over all months, we can create a grid of plots for the distribution of rainfall, using the gamma distribution:

```python
axs = precip.hist(normed=True, figsize=(12, 8), sharex=True, sharey=True, bins=15, grid=False)

for ax in axs.ravel():

    # Get month
    m = ax.get_title()

    # Plot fitted distribution
    x = np.linspace(*ax.get_xlim())
    ax.plot(x, gamma.pdf(x, alpha_mom[m], beta_mom[m]))

    # Annotate with parameter estimates
    label = 'alpha = {0:.2f}\nbeta = {1:.2f}'.format(alpha_mom[m], beta_mom[m])
    ax.annotate(label, xy=(10, 0.2))

plt.tight_layout()
```

Maximum Likelihood
==================

**Maximum likelihood** (ML) fitting is usually more work than the method of moments, but it is preferred as the resulting estimator is known to have good theoretical properties.

There is a ton of theory regarding ML. We will restrict ourselves to the mechanics here.

Say we have some data $y = y_1,y_2,\ldots,y_n$ that is distributed according to some distribution:

$$Pr(Y_i=y_i | \theta)$$

Here, for example, is a **Poisson distribution** that describes the distribution of some discrete variables, typically *counts*:

```python
y = np.random.poisson(5, size=100)
plt.hist(y, bins=12, normed=True)
plt.xlabel('y'); plt.ylabel('Pr(y)')
```

The product $\prod_{i=1}^n Pr(y_i | \theta)$ gives us a measure of how **likely** it is to observe values $y_1,\ldots,y_n$ given the parameters $\theta$. Maximum likelihood fitting consists of choosing the appropriate function $l= Pr(Y|\theta)$ to maximize for a given set of observations. We call this function the *likelihood function*, because it is a measure of how likely the observations are if the model is true.

> Given these data, how likely is this model?

In the above model, the data were drawn from a Poisson distribution with parameter $\lambda =5$.

$$L(y|\lambda=5) = \frac{e^{-5} 5^y}{y!}$$

So, for any given value of $y$, we can calculate its likelihood:

```python
poisson_like = lambda x, lam: np.exp(-lam) * (lam**x) / (np.arange(x)+1).prod()

lam = 6
value = 10
poisson_like(value, lam)
```

    0.041303093412337726

```python
np.sum(poisson_like(yi, lam) for yi in y)
```

    11.338402687045475

```python
lam = 8
np.sum(poisson_like(yi, lam) for yi in y)
```

    7.6625623857972949

We can plot the likelihood function for any value of the parameter(s):

```python
lambdas = np.linspace(0,15)
x = 5
plt.plot(lambdas, [poisson_like(x, l) for l in lambdas])
plt.xlabel('$\lambda$')
plt.ylabel('L($\lambda$|x={0})'.format(x))
```

How is the likelihood function different than the probability distribution function (PDF)? The likelihood is a function of the parameter(s) *given the data*, whereas the PDF returns the probability of data given a particular parameter value. Here is the PDF of the Poisson for $\lambda=5$.

```python
lam = 5
xvals = np.arange(15)
plt.bar(xvals, [poisson_like(x, lam) for x in xvals])
plt.xlabel('x')
plt.ylabel('Pr(X|$\lambda$=5)')
```

Why are we interested in the likelihood function?

A reasonable estimate of the true, unknown value for the parameter is one which **maximizes the likelihood function**. So, inference is reduced to an optimization problem.

Going back to the rainfall data, if we are using a gamma distribution we need to maximize:

$$\begin{align}l(\alpha,\beta) &= \sum_{i=1}^n \log[\beta^{\alpha} x^{\alpha-1} e^{-x/\beta}\Gamma(\alpha)^{-1}] \cr
&= n[(\alpha-1)\overline{\log(x)} - \bar{x}\beta + \alpha\log(\beta) - \log\Gamma(\alpha)]\end{align}$$

(*Its usually easier to work in the log scale*)

where $n = 2012 − 1871 = 141$ and the bar indicates an average over all *i*. We choose $\alpha$ and $\beta$ to maximize $l(\alpha,\beta)$.

Notice $l$ is infinite if any $x$ is zero. We do not have any zeros, but we do have an NA value for one of the October data, which we dealt with above.

### Finding the MLE

To find the maximum of any function, we typically take the *derivative* with respect to the variable to be maximized, set it to zero and solve for that variable.

$$\frac{\partial l(\alpha,\beta)}{\partial \beta} = n\left(\frac{\alpha}{\beta} - \bar{x}\right) = 0$$

Which can be solved as $\beta = \alpha/\bar{x}$. However, plugging this into the derivative with respect to $\alpha$ yields:

$$\frac{\partial l(\alpha,\beta)}{\partial \alpha} = \log(\alpha) + \overline{\log(x)} - \log(\bar{x}) - \frac{\Gamma(\alpha)'}{\Gamma(\alpha)} = 0$$

This has no closed form solution. We must use ***numerical optimization***!

Numerical optimization alogarithms take an initial "guess" at the solution, and iteratively improve the guess until it gets "close enough" to the answer.

Here, we will use Newton-Raphson algorithm:

$$x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}$$

Which is available to us via SciPy:

```python
from scipy.optimize import newton
```

Here is a graphical example of how Newtone-Raphson converges on a solution, using an arbitrary function:

```python
# some function
func = lambda x: 3./(1 + 400*np.exp(-2*x)) - 1
xvals = np.linspace(0, 6)
plt.plot(xvals, func(xvals))
plt.text(5.3, 2.1, '$f(x)$', fontsize=16)
# zero line
plt.plot([0,6], [0,0], 'k-')
# value at step n
plt.plot([4,4], [0,func(4)], 'k:')
plt.text(4, -.2, '$x_n$', fontsize=16)
# tangent line
tanline = lambda x: -0.858 + 0.626*x
plt.plot(xvals, tanline(xvals), 'r--')
# point at step n+1
xprime = 0.858/0.626
plt.plot([xprime, xprime], [tanline(xprime), func(xprime)], 'k:')
plt.text(xprime+.1, -.2, '$x_{n+1}$', fontsize=16)
```

To apply the Newton-Raphson algorithm, we need a function that returns a vector containing the **first and second derivatives** of the function with respect to the variable of interest. In our case, this is:

```python
from scipy.special import psi, polygamma

dlgamma = lambda m, log_mean, mean_log: np.log(m) - psi(m) - log_mean + mean_log
dl2gamma = lambda m, *args: 1./m - polygamma(1, m)
```

where `log_mean` and `mean_log` are $\log{\bar{x}}$ and $\overline{\log(x)}$, respectively. `psi` and `polygamma` are complex functions of the Gamma function that result when you take first and second derivatives of that function.

```python
# Calculate statistics
log_mean = precip.mean().apply(np.log)
mean_log = precip.apply(np.log).mean()
```

Time to optimize!

```python
# Alpha MLE for December
alpha_mle = newton(dlgamma, 2, dl2gamma, args=(log_mean[-1], mean_log[-1]))
alpha_mle
```

    3.5189679152399647

And now plug this back into the solution for beta:

$$ \beta  = \frac{\alpha}{\bar{X}} $$

```python
beta_mle = alpha_mle/precip.mean()[-1]
beta_mle
```

    0.84261607548413797

We can compare the fit of the estimates derived from MLE to those from the method of moments:

```python
dec = precip.Dec
dec.hist(normed=True, bins=10, grid=False)
x = np.linspace(0, dec.max())
plt.plot(x, gamma.pdf(x, alpha_mom[-1], beta_mom[-1]), 'm-')
plt.plot(x, gamma.pdf(x, alpha_mle, beta_mle), 'r--')
```

For some common distributions, SciPy includes methods for fitting via MLE:

```python
from scipy.stats import gamma

gamma.fit(precip.Dec)
```

    (2.2427517753152308, 0.65494604470188622, 1.570073932063466)

This fit is not directly comparable to our estimates, however, because SciPy's `gamma.fit` method fits an odd 3-parameter version of the gamma distribution.

### Example: truncated distribution

Suppose that we observe $Y$ truncated below at $a$ (where $a$ is known). If $X$ is the distribution of our observation, then:

$$ P(X \le x) = P(Y \le x|Y \gt a) = \frac{P(a \lt Y \le x)}{P(Y \gt a)}$$

(so, $Y$ is the original variable and $X$ is the truncated variable)

Then X has the density:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

Suppose $Y \sim N(\mu, \sigma^2)$ and $x_1,\ldots,x_n$ are independent observations of $X$. We can use maximum likelihood to find $\mu$ and $\sigma$.

First, we can simulate a truncated distribution using a `while` statement to eliminate samples that are outside the support of the truncated distribution.

```python
x = np.random.normal(size=10000)
a = -1
x_small = x < a
while x_small.sum():
    x[x_small] = np.random.normal(size=x_small.sum())
    x_small = x < a

_ = plt.hist(x, bins=100)
```

We can construct a log likelihood for this function using the conditional form:

$$f_X(x) = \frac{f_Y (x)}{1−F_Y (a)} \, \text{for} \, x \gt a$$

```python
from scipy.stats.distributions import norm

trunc_norm = lambda theta, a, x: -(np.log(norm.pdf(x, theta[0], theta[1])) -
                                      np.log(1 - norm.cdf(a, theta[0], theta[1]))).sum()
```

For this example, we will use another optimization algorithm, the **Nelder-Mead simplex algorithm**. It has a couple of advantages:

- it does not require derivatives
- it can optimize (minimize) a vector of parameters

SciPy implements this algorithm in its `fmin` function:

```python
from scipy.optimize import fmin

fmin(trunc_norm, np.array([1,2]), args=(-1, x))
```

    Optimization terminated successfully.
             Current function value: 11077.807912
             Iterations: 44
             Function evaluations: 82

    array([ 0.02244612,  0.99710875])

In general, simulating data is a terrific way of testing your model before using it with real data.

### Kernel density estimates

In some instances, we may not be interested in the parameters of a particular distribution of data, but just a smoothed representation of the data at hand. In this case, we can estimate the disribution *non-parametrically* (i.e. making no assumptions about the form of the underlying distribution) using kernel density estimation.

```python
# Some random data
y = np.random.random(15) * 10
y
```

    array([ 0.62402604,  1.06204307,  7.27542973,  5.91978919,  9.95408103,
            9.51695422,  9.54608936,  4.3288413 ,  4.15275767,  2.47654605,
            3.74949496,  4.9779126 ,  0.68937206,  8.82221055,  2.87731832])

```python
x = np.linspace(0, 10, 100)
# Smoothing parameter
s = 0.4
# Calculate the kernels
kernels = np.transpose([norm.pdf(x, yi, s) for yi in y])
plt.plot(x, kernels, 'k:')
plt.plot(x, kernels.sum(1))
plt.plot(y, np.zeros(len(y)), 'ro', ms=10)
```

SciPy implements a Gaussian KDE that automatically chooses an appropriate bandwidth. Let's create a bi-modal distribution of data that is not easily summarized by a parametric distribution:

```python
# Create a bi-modal distribution with a mixture of Normals.
x1 = np.random.normal(0, 3, 50)
x2 = np.random.normal(4, 1, 50)

# Append by row
x = np.r_[x1, x2]
```

```python
plt.hist(x, bins=8, normed=True)
```

```python
from scipy.stats import kde

density = kde.gaussian_kde(x)
xgrid = np.linspace(x.min(), x.max(), 100)
plt.hist(x, bins=8, normed=True)
plt.plot(xgrid, density(xgrid), 'r-')
```

### Exercise: Cervical dystonia analysis

Recall the cervical dystonia database, which is a clinical trial of botulinum toxin type B (BotB) for patients with cervical dystonia from nine U.S. sites. The response variable is measurements on the Toronto Western Spasmodic Torticollis Rating Scale (TWSTRS), measuring severity, pain, and disability of cervical dystonia (high scores mean more impairment). One way to check the efficacy of the treatment is to compare the distribution of TWSTRS for control and treatment patients at the end of the study.

Use the method of moments or MLE to calculate the mean and variance of TWSTRS at week 16 for one of the treatments and the control group. Assume that the distribution of the `twstrs` variable is normal:

$$f(x \mid \mu, \sigma^2) = \sqrt{\frac{1}{2\pi\sigma^2}} \exp\left\{ -\frac{1}{2} \frac{(x-\mu)^2}{\sigma^2} \right\}$$

```python
cdystonia = pd.read_csv("data/cdystonia.csv")
cdystonia[cdystonia.obs==6].hist(column='twstrs', by=cdystonia.treat, bins=8)
```

```python
# Write your answer here
```

## Regression models

A general, primary goal of many statistical data analysis tasks is to relate the influence of one variable on another. For example, we may wish to know how different medical interventions influence the incidence or duration of disease, or perhaps a how baseball player's performance varies as a function of age.

```python
x = np.array([2.2, 4.3, 5.1, 5.8, 6.4, 8.0])
y = np.array([0.4, 10.1, 14.0, 10.9, 15.4, 18.5])
plt.plot(x,y,'ro')
```

We can build a model to characterize the relationship between $X$ and $Y$, recognizing that additional factors other than $X$ (the ones we have measured or are interested in) may influence the response variable $Y$.

$y_i = f(x_i) + \epsilon_i$

where $f$ is some function, for example a linear function:

$y_i = \beta_0 + \beta_1 x_i + \epsilon_i$

and $\epsilon_i$ accounts for the difference between the observed response $y_i$ and its prediction from the model $\hat{y_i} = \beta_0 + \beta_1 x_i$. This is sometimes referred to as **process uncertainty**.

We would like to select $\beta_0, \beta_1$ so that the difference between the predictions and the observations is zero, but this is not usually possible. Instead, we choose a reasonable criterion: ***the smallest sum of the squared differences between $\hat{y}$ and $y$***.

$$R^2 = \sum_i (y_i - [\beta_0 + \beta_1 x_i])^2 = \sum_i \epsilon_i^2 $$

Squaring serves two purposes: (1) to prevent positive and negative values from cancelling each other out and (2) to strongly penalize large deviations. Whether the latter is a good thing or not depends on the goals of the analysis.

In other words, we will select the parameters that minimize the squared error of the model.

```python
ss = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x) ** 2)
```

```python
ss([0,1],x,y)
```

    333.35000000000002

```python
b0,b1 = fmin(ss, [0,1], args=(x,y))
b0,b1
```

    Optimization terminated successfully.
             Current function value: 21.375000
             Iterations: 79
             Function evaluations: 153

    (-4.3500136038870876, 3.0000002915386412)

```python
plt.plot(x, y, 'ro')
plt.plot([0,10], [b0, b0+b1*10])
```

```python
plt.plot(x, y, 'ro')
plt.plot([0,10], [b0, b0+b1*10])
for xi, yi in zip(x,y):
    plt.plot([xi]*2, [yi, b0+b1*xi], 'k:')
plt.xlim(2, 9); plt.ylim(0, 20)
```

Minimizing the sum of squares is not the only criterion we can use; it is just a very popular (and successful) one. For example, we can try to minimize the sum of absolute differences:

```python
sabs = lambda theta, x, y: np.sum(np.abs(y - theta[0] - theta[1]*x))
b0,b1 = fmin(sabs, [0,1], args=(x,y))
print b0,b1
plt.plot(x, y, 'ro')
plt.plot([0,10], [b0, b0+b1*10])
```

We are not restricted to a straight-line regression model; we can represent a curved relationship between our variables by introducing **polynomial** terms. For example, a cubic model:

$y_i = \beta_0 + \beta_1 x_i + \beta_2 x_i^2 + \epsilon_i$

```python
ss2 = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2)) ** 2)
b0,b1,b2 = fmin(ss2, [1,1,-1], args=(x,y))
print b0,b1,b2
plt.plot(x, y, 'ro')
xvals = np.linspace(0, 10, 100)
plt.plot(xvals, b0 + b1*xvals + b2*(xvals**2))
```

Although polynomial model characterizes a nonlinear relationship, it is a linear problem in terms of estimation. That is, the regression model $f(y | x)$ is linear in the parameters.

For some data, it may be reasonable to consider polynomials of order>2. For example, consider the relationship between the number of home runs a baseball player hits and the number of runs batted in (RBI) they accumulate; clearly, the relationship is positive, but we may not expect a linear relationship.

```python
ss3 = lambda theta, x, y: np.sum((y - theta[0] - theta[1]*x - theta[2]*(x**2)
                                  - theta[3]*(x**3)) ** 2)

bb = pd.read_csv("data/baseball.csv", index_col=0)
plt.plot(bb.hr, bb.rbi, 'r.')
b0,b1,b2,b3 = fmin(ss3, [0,1,-1,0], args=(bb.hr, bb.rbi))
xvals = np.arange(40)
plt.plot(xvals, b0 + b1*xvals + b2*(xvals**2) + b3*(xvals**3))
```

Of course, we need not fit least squares models by hand. The `statsmodels` package implements least squares models that allow for model fitting in a single line:

```python
import statsmodels.api as sm

straight_line = sm.OLS(y, sm.add_constant(x)).fit()
straight_line.summary()
```

    /usr/local/lib/python2.7/site-packages/statsmodels/stats/stattools.py:72: UserWarning: omni_normtest is not valid with less than 8 observations; 6 samples were given.
      "samples were given." % int(n))

OLS Regression Results

  Dep. Variable:	            y	          R-squared:         	    0.891

  Model:	                   OLS	         Adj. R-squared:    	    0.864

  Method:	             Least Squares	    F-statistic:       	    32.67

  Date:	             Wed, 18 Jun 2014	   Prob (F-statistic):	  0.00463

  Time:	                 20:56:03	       Log-Likelihood:    	  -12.325

  No. Observations:	           6	        AIC:               	    28.65

  Df Residuals:	               4	        BIC:               	    28.23

  Df Model:	                   1

  Covariance Type:	      nonrobust

    	       coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  const	    -4.3500	     2.937	    -1.481	  0.213	   -12.505     3.805

  x1	        3.0000	     0.525	     5.716	  0.005	     1.543     4.457

  Omnibus:	          nan	   Durbin-Watson:     	    2.387

  Prob(Omnibus):	    nan	   Jarque-Bera (JB):  	    0.570

  Skew:	           0.359	   Prob(JB):          	    0.752

  Kurtosis:	       1.671	   Cond. No.          	     17.9

```python
from statsmodels.formula.api import ols as OLS

data = pd.DataFrame(dict(x=x, y=y))
cubic_fit = OLS('y ~ x + I(x**2)', data).fit()

cubic_fit.summary()
```

OLS Regression Results

  Dep. Variable:	            y	          R-squared:         	    0.929

  Model:	                   OLS	         Adj. R-squared:    	    0.881

  Method:	             Least Squares	    F-statistic:       	    19.50

  Date:	             Wed, 18 Jun 2014	   Prob (F-statistic):	  0.0191

  Time:	                 20:56:03	       Log-Likelihood:    	  -11.056

  No. Observations:	           6	        AIC:               	    28.11

  Df Residuals:	               3	        BIC:               	    27.49

  Df Model:	                   2

  Covariance Type:	      nonrobust

      	         coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  Intercept	   -11.0748	     6.013	    -1.842	  0.163	   -30.211     8.062

  x	             6.0577	     2.482	     2.441	  0.092	    -1.840    13.955

  I(x ** 2)	    -0.3027	     0.241	    -1.257	  0.298	    -1.069     0.464

  Omnibus:	          nan	   Durbin-Watson:     	    2.711

  Prob(Omnibus):	    nan	   Jarque-Bera (JB):  	    0.655

  Skew:	          -0.809	   Prob(JB):          	    0.721

  Kurtosis:	       2.961	   Cond. No.          	     270.

### Exercise: Polynomial function

Write a function that specified a polynomial of arbitrary degree.

```python
# Write your answer here
```

## Model Selection

How do we choose among competing models for a given dataset? More parameters are not necessarily better, from the standpoint of model fit. For example, fitting a 9-th order polynomial to the sample data from the above example certainly results in an overfit.

```python
def calc_poly(params, data):
        x = np.c_[[data**i for i in range(len(params))]]
        return np.dot(params, x)

ssp = lambda theta, x, y: np.sum((y - calc_poly(theta, x)) ** 2)
betas = fmin(ssp, np.zeros(10), args=(x,y), maxiter=1e6)
plt.plot(x, y, 'ro')
xvals = np.linspace(0, max(x), 100)
plt.plot(xvals, calc_poly(betas, xvals))
```

One approach is to use an information-theoretic criterion to select the most appropriate model. For example **Akaike's Information Criterion (AIC)** balances the fit of the model (in terms of the likelihood) with the number of parameters required to achieve that fit. We can easily calculate AIC as:

$$AIC = n \log(\hat{\sigma}^2) + 2p$$

where $p$ is the number of parameters in the model and $\hat{\sigma}^2 = RSS/(n-p-1)$.

Notice that as the number of parameters increase, the residual sum of squares goes down, but the second term (a penalty) increases.

To apply AIC to model selection, we choose the model that has the **lowest** AIC value.

```python
n = len(x)

aic = lambda rss, p, n: n * np.log(rss/(n-p-1)) + 2*p

RSS1 = ss(fmin(ss, [0,1], args=(x,y)), x, y)
RSS2 = ss2(fmin(ss2, [1,1,-1], args=(x,y)), x, y)

print aic(RSS1, 2, n), aic(RSS2, 3, n)
```

    Optimization terminated successfully.
             Current function value: 21.375000
             Iterations: 79
             Function evaluations: 153
    Optimization terminated successfully.
             Current function value: 14.001110
             Iterations: 198
             Function evaluations: 372
    15.7816583572 17.6759368019

Hence, we would select the 2-parameter (linear) model.

## Logistic Regression

Fitting a line to the relationship between two variables using the least squares approach is sensible when the variable we are trying to predict is continuous, but what about when the data are dichotomous?

- male/female
- pass/fail
- died/survived

Let's consider the problem of predicting survival in the Titanic disaster, based on our available information. For example, lets say that we want to predict survival as a function of the fare paid for the journey.

```python
titanic = pd.read_excel("data/titanic.xls", "titanic")
titanic.name
```

    0                      Allen, Miss. Elisabeth Walton
    1                     Allison, Master. Hudson Trevor
    2                       Allison, Miss. Helen Loraine
    3               Allison, Mr. Hudson Joshua Creighton
    4    Allison, Mrs. Hudson J C (Bessie Waldo Daniels)
    5                                Anderson, Mr. Harry
    6                  Andrews, Miss. Kornelia Theodosia
    7                             Andrews, Mr. Thomas Jr
    8      Appleton, Mrs. Edward Dale (Charlotte Lamson)
    9                            Artagaveytia, Mr. Ramon
    ...
    1298                  Wittevrongel, Mr. Camille
    1299                        Yasbeck, Mr. Antoni
    1300    Yasbeck, Mrs. Antoni (Selini Alexander)
    1301                       Youseff, Mr. Gerious
    1302                          Yousif, Mr. Wazli
    1303                      Yousseff, Mr. Gerious
    1304                       Zabour, Miss. Hileni
    1305                      Zabour, Miss. Thamine
    1306                  Zakarian, Mr. Mapriededer
    1307                        Zakarian, Mr. Ortin
    1308                         Zimmerman, Mr. Leo
    Name: name, Length: 1309, dtype: object

```python
jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
plt.yticks([0,1])
plt.ylabel("survived")
plt.xlabel("log(fare)")
```

I have added random jitter on the y-axis to help visualize the density of the points, and have plotted fare on the log scale.

Clearly, fitting a line through this data makes little sense, for several reasons. First, for most values of the predictor variable, the line would predict values that are not zero or one. Second, it would seem odd to choose least squares (or similar) as a criterion for selecting the best line.

```python
x = np.log(titanic.fare[titanic.fare>0])
y = titanic.survived[titanic.fare>0]
betas_titanic = fmin(ss, [1,1], args=(x,y))
```

    Optimization terminated successfully.
             Current function value: 277.621917
             Iterations: 55
             Function evaluations: 103

```python
jitter = np.random.normal(scale=0.02, size=len(titanic))
plt.scatter(np.log(titanic.fare), titanic.survived + jitter, alpha=0.3)
plt.yticks([0,1])
plt.ylabel("survived")
plt.xlabel("log(fare)")
plt.plot([0,7], [betas_titanic[0], betas_titanic[0] + betas_titanic[1]*7.])
```

If we look at this data, we can see that for most values of `fare`, there are some individuals that survived and some that did not. However, notice that the cloud of points is denser on the "survived" (y=1) side for larger values of fare than on the "died" (y=0) side.

### Stochastic model

Rather than model the binary outcome explicitly, it makes sense instead to model the *probability* of death or survival in a **stochastic** model. Probabilities are measured on a continuous [0,1] scale, which may be more amenable for prediction using a regression line. We need to consider a different probability model for this exerciese however; let's consider the **Bernoulli** distribution as a generative model for our data:

$$f(y|p) = p^{y} (1-p)^{1-y}$$

where $y = \{0,1\}$ and $p \in [0,1]$. So, this model predicts whether $y$ is zero or one as a function of the probability $p$. Notice that when $y=1$, the $1-p$ term disappears, and when $y=0$, the $p$ term disappears.

So, the model we want to fit should look something like this:

$$p_i = \beta_0 + \beta_1 x_i + \epsilon_i$$

However, since $p$ is constrained to be between zero and one, it is easy to see where a linear (or polynomial) model might predict values outside of this range. We can modify this model sligtly by using a **link function** to transform the probability to have an unbounded range on a new scale. Specifically, we can use a **logit transformation** as our link function:

$$\text{logit}(p) = \log\left[\frac{p}{1-p}\right] = x$$

Here's a plot of $p/(1-p)$

```python
logit = lambda p: np.log(p/(1.-p))
unit_interval = np.linspace(0,1)
plt.plot(unit_interval/(1-unit_interval), unit_interval)
```

And here's the logit function:

```python
plt.plot(logit(unit_interval), unit_interval)
```

The inverse of the logit transformation is:

$$p = \frac{1}{1 + \exp(-x)}$$

So, now our model is:

$$\text{logit}(p_i) = \beta_0 + \beta_1 x_i + \epsilon_i$$

We can fit this model using maximum likelihood. Our likelihood, again based on the Bernoulli model is:

$$L(y|p) = \prod_{i=1}^n p_i^{y_i} (1-p_i)^{1-y_i}$$

which, on the log scale is:

$$l(y|p) = \sum_{i=1}^n y_i \log(p_i) + (1-y_i)\log(1-p_i)$$

We can easily implement this in Python, keeping in mind that `fmin` minimizes, rather than maximizes functions:

```python
invlogit = lambda x: 1. / (1 + np.exp(-x))

def logistic_like(theta, x, y):
    p = invlogit(theta[0] + theta[1] * x)
    # Return negative of log-likelihood
    return -np.sum(y * np.log(p) + (1-y) * np.log(1 - p))
```

Remove null values from variables

```python
x, y = titanic[titanic.fare.notnull()][['fare', 'survived']].values.T
```

... and fit the model.

```python
b0,b1 = fmin(logistic_like, [0.5,0], args=(x,y))
b0, b1
```

    Optimization terminated successfully.
             Current function value: 827.015955
             Iterations: 47
             Function evaluations: 93

    (-0.88238984528338194, 0.012452067664164127)

```python
jitter = np.random.normal(scale=0.01, size=len(x))
plt.plot(x, y+jitter, 'r.', alpha=0.3)
plt.yticks([0,.25,.5,.75,1])
xvals = np.linspace(0, 600)
plt.plot(xvals, invlogit(b0+b1*xvals))
```

As with our least squares model, we can easily fit logistic regression models in `statsmodels`, in this case using the `GLM` (generalized linear model) class with a binomial error distribution specified.

```python
logistic = sm.GLM(y, sm.add_constant(x), family=sm.families.Binomial()).fit()
logistic.summary()
```

Generalized Linear Model Regression Results

  Dep. Variable:	          y	          No. Observations:  	    1308

  Model:	                 GLM	         Df Residuals:      	    1306

  Model Family:	       Binomial	       Df Model:          	       1

  Link Function:	        logit	        Scale:             	    1.0

  Method:	               IRLS	         Log-Likelihood:    	  -827.02

  Date:	           Wed, 18 Jun 2014	   Deviance:          	   1654.0

  Time:	               20:56:06	       Pearson chi2:      	 1.33e+03

  No. Iterations:	         6

    	       coef	     std err	      t	      P>|t|	 [95.0% Conf. Int.]

  const	    -0.8824	     0.076	   -11.684	  0.000	    -1.030    -0.734

  x1	        0.0125	     0.002	     7.762	  0.000	     0.009     0.016

### Exercise: multivariate logistic regression

Which other variables might be relevant for predicting the probability of surviving the Titanic? Generalize the model likelihood to include 2 or 3 other covariates from the dataset.

```python
# Write your answer here
```

## Bootstrapping

Parametric inference can be **non-robust**:

* inaccurate if parametric assumptions are violated
* if we rely on asymptotic results, we may not achieve an acceptable level of accuracy

Parmetric inference can be **difficult**:

* derivation of sampling distribution may not be possible

An alternative is to estimate the sampling distribution of a statistic *empirically* without making assumptions about the form of the population.

We have seen this already with the kernel density estimate.

### Non-parametric Bootstrap

The bootstrap is a resampling method discovered by [Brad Efron](http://www.jstor.org/discover/10.2307/2958830?uid=3739568&uid=2&uid=4&uid=3739256&sid=21102342537691) that allows one to approximate the true sampling distribution of a dataset, and thereby obtain estimates of the mean and variance of the distribution.

Bootstrap sample:

$$S_1^* = \{x_{11}^*, x_{12}^*, \ldots, x_{1n}^*\}$$

$S_i^*$ is a sample of size $n$, **with** replacement.

In Python, we have already seen the NumPy function `permutation` that can be used in conjunction with Pandas' `take` method to generate a random sample of some data without replacement:

```python
np.random.permutation(titanic.name)[:5]
```

    array([u'Meek, Mrs. Thomas (Annie Louise Rowley)',
           u'Thorneycroft, Mr. Percival', u'Williams, Mr. Leslie',
           u'Graham, Mr. George Edward', u'Petroff, Mr. Nedelio'], dtype=object)

Similarly, we can use the `random.randint` method to generate a sample *with* replacement, which we can use when bootstrapping.

```python
random_ind = np.random.randint(0, len(titanic), 5)
titanic.name[random_ind]
```

    41      Brown, Mrs. James Joseph (Margaret Tobin)
    1061              Nilsson, Miss. Helmina Josefina
    937                  Klasen, Miss. Gertrud Emilia
    426                            Hale, Mr. Reginald
    831                Goodwin, Mr. Charles Frederick
    Name: name, dtype: object

We regard S as an "estimate" of population P

> population : sample :: sample : bootstrap sample

The idea is to generate replicate bootstrap samples:

$$S^* = \{S_1^*, S_2^*, \ldots, S_R^*\}$$

Compute statistic $t$ (estimate) for each bootstrap sample:

$$T_i^* = t(S^*)$$

```python
n = 10
R = 1000
# Original sample (n=10)
x = np.random.normal(size=n)
# 1000 bootstrap samples of size 10
s = [x[np.random.randint(0,n,n)].mean() for i in range(R)]
_ = plt.hist(s, bins=30)
```

### Bootstrap Estimates

From our bootstrapped samples, we can extract *estimates* of the expectation and its variance:

$$\bar{T}^* = \hat{E}(T^*) = \frac{\sum_i T_i^*}{R}$$

$$\hat{\text{Var}}(T^*) = \frac{\sum_i (T_i^* - \bar{T}^*)^2}{R-1}$$

```python
boot_mean = np.sum(s)/R
boot_mean
```

    0.087385394806476724

```python
boot_var = ((np.array(s) - boot_mean) ** 2).sum() / (R-1)
boot_var
```

    0.10590407752057245

Since we have estimated the expectation of the bootstrapped statistics, we can estimate the **bias** of T:

$$\hat{B}^* = \bar{T}^* - T$$

```python
boot_mean - np.mean(x)
```

    -0.00528084680355842

### Bootstrap error

There are two sources of error in bootstrap estimates:

1. **Sampling error** from the selection of $S$.
2. **Bootstrap error** from failing to enumerate all possible bootstrap samples.

For the sake of accuracy, it is prudent to choose at least R=1000

### Bootstrap Percentile Intervals

An attractive feature of bootstrap statistics is the ease with which you can obtain an estimate of *uncertainty* for a given statistic. We simply use the empirical quantiles of the bootstrapped statistics to obtain percentiles corresponding to a confidence interval of interest.

This employs the *ordered* bootstrap replicates:

$$T_{(1)}^*, T_{(2)}^*, \ldots, T_{(R)}^*$$

Simply extract the $100(\alpha/2)$ and $100(1-\alpha/2)$ percentiles:

$$T_{[(R+1)\alpha/2]}^* \lt \theta \lt T_{[(R+1)(1-\alpha/2)]}^*$$

```python
s_sorted = np.sort(s)
s_sorted[:10]
```

    array([-0.82890714, -0.77634577, -0.76588512, -0.76230089, -0.75578488,
           -0.73850118, -0.72869116, -0.72862786, -0.72840095, -0.71831374])

```python
s_sorted[-10:]
```

    array([ 0.81823418,  0.86179331,  0.92314175,  0.93496722,  0.9358216 ,
            1.02058937,  1.03085586,  1.03121927,  1.22699691,  1.3599996 ])

```python
alpha = 0.05
s_sorted[[(R+1)*alpha/2, (R+1)*(1-alpha/2)]]
```

    array([-0.5684053 ,  0.68682205])

### Exercise: Cervical dystonia bootstrap estimates

Use bootstrapping to estimate the mean of one of the treatment groups, and calculate percentile intervals for the mean.

```python
# Write your answer here
```


good python code examples:
https://www.inovex.de/de/blog/support-vector-machines-guide/

# Support-Vector-Machines = Maximum Margin Classification

```python
import numpy as np
from IPython.display import Image
import matplotlib.pyplot as plt
```

```python
# seperating line
b = 10
w1 = -1.5
w2 = -1
# as vector and the normal to the separating line
w = np.array([w1, w2])
w_norm = w/np.sqrt(np.dot(w, w))
this_is_the_lenth = np.dot(w_norm, w_norm)
# class 1
cl1 = np.array([np.random.normal(loc=1, scale=1.1, size=5),
          np.random.normal(loc=0, scale = 1.2, size = 5)]).T
# support vectors class 1:
cl1_s = np.array([np.array([5, 7]), (b+4) + w1 * np.array([5, 7])]).T

# class 2
cl2 = np.array([np.random.normal(loc=11.5, scale=2., size=7),
                np.random.normal(loc=4, scale=0.6, size=7)]).T
# support vectors class 2:
cl2_s = np.array([np.array([3, 5]), (b-4) + w1 * np.array([3, 5])]).T

fig = plt.figure()
ax = plt.axes()
plt.xlim(-3, 15)
plt.ylim(-5, 10)
plt.gca().set_aspect('equal', adjustable='box')
x = np.linspace(-5, 15, 100)
y = b + w1 * x
ax.plot(x, b + w1 * x)
ax.plot(x, (b-4) + w1 * x, color='red')
ax.plot(x, (b+4) + w1 * x, color='red')
ax.scatter(cl1[:, 0], cl1[:, 1], color='blue', alpha=0.5)
ax.scatter(cl2[:, 0], cl2[:, 1], color='red', alpha=0.5)
ax.scatter(cl1_s[:, 0], cl1_s[:, 1], color='black', alpha=0.7)
ax.scatter(cl2_s[:, 0], cl2_s[:, 1], color='blue', alpha=0.6)
#ax.scatter(7, b + w[0] * 7, color='black')
ax.scatter(41/13, b + w1 *(41/13), color = 'black')
ax.plot([41/13, cl1_s[0,0]],[b + w1 * (41/13), cl1_s[0,1]], color='black')
plt.annotate(text='d',xy=(45/13, 6.5))

plt.arrow(-2.5, -2.5, w[0] * -2, w[1] * -2, head_width=0.5)
plt.annotate(text='w', xy=(-1,-0.5))
# plt.scatter(x, y, color='red', alpha = 0.45)
# print(f'{b + w1 * x + w2 * y}')
```

Im 2-dimensionalen Raum haben wir die Variablen x, y. Alle Punkte auf einer Geraden (separating plane) bekommt man:
\begin{equation*}
a + b\cdot x = c\cdot y
\end{equation*}

Trivialerweise ist $c=1$ meist gültig. Dies entspricht exakt unserer Geraden-Gleichung aus der Linearen Regression.

Umgekehrt kann man die Gerade auch als die Menge aller Punkte definieren, die folgender Bedingung genügen:

\begin{equation*}
a + b\cdot x + c\cdot y = 0
\end{equation*}

Im trivialen Fall wäre $c=-1$

Für alle Punkte unterhalb der Gerade gilt:
\begin{equation*}
a + b\cdot x + c\cdot y < 0
\end{equation*}
und für alle Punkte oberhalb der Geraden gilt:
\begin{equation*}
a + b\cdot x + c\cdot y > 0
\end{equation*}

Für die Klassification gilt, dass die Klassen immer mit $-1$ und $1$ benannt werden. Ein neuer Punkt $(x, y)$ wird klassifiziert:
\begin{equation*}
\hat{\text{pred}} = \text{sign}(a + b\cdot x + c\cdot y)
\end{equation*}

### Aenderung der Schreibweise:
In der Literatur zu Support Vector Machines werden die Koeffizienten anders benannt als in der klassischen Linearen Regression. Anstatt $a$ für den intercept wird $b$ verwendet. Anstatt $b$ für den Steigungskoeffizienten wird $\omega$ verwendet:
\begin{equation*}
\hat{\text{pred}} = \text{sign}(b + \omega_1\cdot x_i + \omega_2\cdot x_2)
\end{equation*}
und in Vektorschreibweise (exakt identisch zur Multivariaten Linearen Regression - mit Ausnahme der Bennenung der Koeffizienten):
\begin{equation*}
\hat{\text{pred}} = \text{sign}(b + \pmb{\omega} \pmb{x})
\end{equation*}

Es kann gezeigt werden, dass der Vektor  $\pmb{\omega}$  senkrecht auf die Trennungs-Ebene (in der Graphik oben die mittlere Gerade) steht. Dies habe ich als Pfeil eingezeichnet.

Nun soll eine Ebene gefunden werden, deren __margin maximal__ ist; als margin wird der Abstand zu den nähesten Daten-Punkten bezeichnet. Die Datenpunkte, die auf den margins zu liegen kommen heissen __support vectors__. Die Enfernung dieser Punkte von der Trennlinie kann angegeben werden:

\begin{equation*}
d= y_i(\pmb{\omega}\pmb{x}_i + b);\quad \quad y_i \in \{-1, 1\},
\end{equation*}

hier ändert $y_i$ das Vorzeichen so, dass die Distanz immer positiv ist.

Für alle Punkte $\mathbf{x_i}$, die genau die Distanz 1 haben gilt:

\begin{eqnarray*}
1 =& y_i(\pmb{\omega}\pmb{x}_i + b)\\
\frac{1}{\left\Vert \pmb{\omega}\right\Vert} =& \left|\frac{\pmb{\omega}\pmb{x}_i}{\left\Vert\pmb{\omega}\right\Vert} + \frac{b}{\left\Vert\pmb{\omega}\right\Vert}\right|
\end{eqnarray*}

$\|\pmb{\omega}\|$ ist die Länge des Vektors $\pmb{\omega}$, die sogenannte Vecktor-Norm. Teile ich einen Vektor durch seine Vektor-Norm, dann hat er genau die Länge $1$. Weil alle Punkte auf dem margin dieselbe Distanz zur Trennlinie haben und weil der margin symmetrisch ist muss

\begin{equation*}
p = \frac{2}{\|\pmb{\omega}\|}
\end{equation*}
maximiert werden, um den margin möglichst gross zu machen. Das bedeutet, dass $\|\pmb{\omega}\|$ möglichst klein sein muss.

Der Abstand der beide support-vectors ist also indirekt proportional zur Länge des Vectors $\omega$. Als Minimierungs-Problem ergibt sich:
\begin{equation*}
(\omega^*, b^*) = \text{argmin} \frac{1}{2}\|\pmb{\omega}\|
\end{equation*}
Maximieren von $\frac{2}{\|\pmb{\omega}\|}$ ist gleichbedeutend mit Minimieren von $\frac{\|\pmb{\omega}\|}{2}$.
Damit haben wir die Formulierung unseres maximum-margin classifiers:

Finde $\pmb{\omega}$ und $b$ so, dass:
 1. $\frac{1}{2}\pmb{\omega}^{T}\pmb{\omega}$ minimiert wird, für alle
 2. $(\mathbf{x}_i, y_i), y_i(\pmb{\omega}\pmb{x}_i + b) \geq 1$

Weil die Vektor-Norm sich als die Quadratwurzel der aufsummierten quadrierten Vectorelemente berechnet:

\begin{equation*}
\|\pmb{\omega}\| = \sqrt{\omega_1^2 + \omega_2^2 + \ldots \omega_d^2}
\end{equation*}
kann man auch schreiben:
\begin{equation*}
\|\pmb{\omega}\| = \sqrt{\pmb{\omega}^{T}\pmb{\omega}}
\end{equation*}
Da die Quadratwurzel eine monotone Transformation ist, kann sie in unserem Minimierungs-Problem einfach weggelassen werden. Somit ergibt sich obige 1. Bedinung.

Will man das lösen, ergibt sich ein quadratische Optimierungsproblem, welches mit Lagrange Multiplikatoren gelöst werden kann.

Hierbei ergibt sich die characteristische Gleichung (dual form) zu:
  \begin{equation*} \text{max}_{\alpha}\sum_i\alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i\alpha_j y_i y_j \mathbf{x}_i^{T} \cdot \mathbf{x}_j \end{equation*}

Für das Verständnis der nachfolgenden Erklärungen zu kernels, ist der Ausdruck $\mathbf{x}_i^{T} \cdot \mathbf{x}_j$ entscheidend. Hierbei wird das Skalarprodukt (dot product) eines jeden Daten-Punktes $\pmb{x}_i$ mit jedem anderen Daten-Punkt $\pmb{x}_j$ berechnet. Für normierte Vektoren $\mathbf{x}_i$ ist das Skalarprodukt ein Ähnlichkeitsmass (cosine-similarity, correlation-coefficient).

### Problem:
Der Ausdruck

\begin{equation*}
\mathbf{x}_i^{T} \cdot \mathbf{x}_j
\end{equation*}

in obiger Gleichung zeigt, dass Support-Vecktor-Machines quadratisch mit der Anzahl der Trainings-Punkte skalieren. SVMs sind also für BigData nicht zu gebrauchen.

## Kernel-Trick

```python
from sklearn.datasets import make_circles
import matplotlib.pyplot as plt
import numpy as np
```

```python
x, y = make_circles(n_samples = (70, 30), random_state=42, factor=0.8)
```

```python
plt.figure(figsize = (6,4))
plt.scatter(x[y==1, 0], x[y==1, 1], color='red', alpha = 0.45)
plt.scatter(x[y==0, 0], x[y==0, 1], color='blue', alpha = 0.45)
plt.title('inseparable datasets')
plt.xlabel('1st dimension')
plt.ylabel('2nd dimension')
```

In the above Figure, the two data-sets (red dots and blue dots) are not linearily separable. There is no plane/line that can separate the two classes.
### project the data into 3 dimensions

\begin{equation*}
\Phi(\mathbf{x})\rightarrow \mathbf{x}_1^{2}, \mathbf{x}_2^{2},\sqrt{2}\mathbf{x}_1\mathbf{x}_2\text{,}
\end{equation*}
i.e. there is a transformation that projects the x- and y-coordinates to new x- and y-coordinates and to an additional z-coordinate.

```python
# this is the projection into the 3rd coordinate (z-coordinate)
x3 = np.sqrt(2) * x[:, 0] * x[:, 1]
x3 = x3.reshape((-1, 1))
# in the first column of x we have the x-coordinates, in the second column the y-coordinate
mapped_x = np.concatenate((x ** 2, x3), axis=1)
# mapped_x has three coordinates right now:
mapped_x.shape
```

    (100, 3)

```python
fig = plt.figure(figsize = (8,6))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(mapped_x[y==1, 0], mapped_x[y==1, 1], mapped_x[y==1, 2], color='red', alpha = 0.45)
ax.scatter(mapped_x[y==0, 0], mapped_x[y==0, 1], mapped_x[y==0, 2], color='blue', alpha = 0.45)
#plt.title('inseparable datasets')
ax.set_xlabel('1st dimension')
ax.set_ylabel('2nd dimension')
ax.set_zlabel('3rd dimension')
```

As can be seen, the additional 3rd dimension makes the problem far easier to solve. Now, a plane can easily separate the two classes.

### just another projection into 3D

\begin{equation*}
\Phi(\mathbf{x})\rightarrow \mathbf{x}_i, \mathbf{x}_2, \sum_i^n \exp\left(-\frac{\|\mathbf{x}- \mathbf{x}_i\|^2}{2\sigma^2}\right)
\end{equation*}

__BUT__ as we will see later, there exists no kernel-trick for this projection. The projection is helpfull to make the two classes separable but other projections allow for more efficient solutions.

```python
# not 100 % correct though
# https://stats.stackexchange.com/questions/63881/use-gaussian-rbf-kernel-for-mapping-of-2d-data-to-3d

from scipy.spatial.distance import pdist, squareform
from scipy import exp
pairwise_d = squareform(pdist(x, 'euclidean'))
projection = exp(-pairwise_d **2 / (2 ** 2))
new_dim = np.sum(projection, axis=1)
```

    /home/martin/miniconda3/lib/python3.6/site-packages/ipykernel_launcher.py:7: DeprecationWarning: scipy.exp is deprecated and will be removed in SciPy 2.0.0, use numpy.exp instead
      import sys

```python
fig = plt.figure(figsize = (8,6))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(x[y==1, 0], x[y==1, 1], new_dim[y==1], color='red', alpha = 0.45)
ax.scatter(x[y==0, 0], x[y==0, 1], new_dim[y==0], color='blue', alpha = 0.45)
#plt.title('inseparable datasets')
ax.set_xlabel('1st dimension')
ax.set_ylabel('2nd dimension')
ax.set_zlabel('3rd dimension')
```

### Der kernel-Trick besteht genau darin, dass die Projektion in einen höher-dimensionalen Raum nicht vorgenommen werden muss

Wir erinnern die charakteristische Formel des maximal-margin classifiers:

\begin{equation*}\text{max}_{\alpha}\sum_i\alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i\alpha_j y_i y_j \mathbf{x}_i^{T} \cdot \mathbf{x}_j\end{equation*}

Anstatt mit den urspr. Daten-Punkten $\mathbf{x}_i$ und $\mathbf{x}_j$ muss diese Gleichung jetzt für die Projektionen der Daten vorgenommen werden:

### unsere erste Projektion:

$\Phi(\mathbf{x})\rightarrow \mathbf{x}_1^{2}, \mathbf{x}_2^{2},\sqrt{2}\mathbf{x}_1\mathbf{x}_2$

einsetzen in das Skalarprodukt (dot-product) für 2 Daten-Punkte $\mathbf{x}_i$ und $\mathbf{x}_j$ (wie in der characteristischen Gleichung oben):
\begin{eqnarray*}
\langle\Phi(\mathbf{x}_i),\Phi(\mathbf{x}_j)\rangle =&\\
\langle[x_{i1}^{2},x_{i2}^{2}, \sqrt{2}x_{i1}x_{i2}],[x_{j1}^{2},x_{j2}^{2}, \sqrt{2}x_{j1}x_{j2}]\rangle =& \\
x_{i1}^{2}x_{j1}^{2} + x_{i2}^{2}x_{j2}^{2} +2 x_{i1}x_{i2}x_{j1}x_{j2}
\end{eqnarray*}

### Trick: Es gibt eine einfachere Rechenvorschrift, die zum selben Ergebnis kommt. Das ist der Kernel-Trick

\begin{eqnarray*}
\langle\Phi(\mathbf{x}_i),\Phi(\mathbf{x}_j)\rangle = \langle\mathbf{x}_i,\mathbf{x}_j\rangle^{2} =&\\
\langle[x_{i1}, x_{i2}],[x_{j1}, x_{j2}]\rangle^{2} =& \\
(x_{i1}x_{j1} + x_{i2}x_{j2})^{2} =& \\
x_{i1}^{2}x_{j1}^{2} + x_{i2}^{2}x_{j2}^{2} + 2 x_{i1}x_{i2}x_{j1}x_{j2}
\end{eqnarray*}

Man kann jetzt also schreiben:

\begin{equation*}
k(\mathbf{x_i}, \mathbf{x_j}) = \langle\mathbf{x}_i,\mathbf{x}_j\rangle^{2}
\end{equation*}

Für unsere charackteristische Gleichung könnten wir nun schreiben:

\begin{equation*}
\text{max}_{\alpha}\sum_i\alpha_i - \frac{1}{2}\sum_{i,j}\alpha_i\alpha_j y_i y_j k(\mathbf{x_i}, \mathbf{x_j})
\end{equation*}

__Unser maximal-margin Classifier könnte also das Problem der 2 Ringe lösen, ohne explizit die Projektione in 3D vornehmen zu müssen__.

Unterschied:

1.Methode:
* Projektion in höher-dimensionalen Raum
* Multiplikation in höher-dimensionalen Raum

2.Method:
* Multiplikation im Ausgangs-Raum
* Quadratur des Ergebnisses

## What are kernels exactly?

Basically, kernels are similarity measures outside the usual Euclidean space.

We can interpret kernels in two different ways:

Either
 * we project the data into higher dimensional spaces and compute Euclidean distances (similarities)

Or

 * we use a different distance measure (other then Euclidean distance) in the original space that correponds to Euclidean distance in the projected dimension

For example, take the __radial basis function kernel__:

\begin{equation*}
K(X_1, X_2) = \exp\left(-\frac{\|X_1 - X_2\|^2}{2\sigma^2}\right)
\end{equation*}

$\left\Vert X_1 - X_2\right\Vert$ is the Euclidean distance between the points $X_1$ and $X_2$ or between two vectors $\mathbf{x}_1$ and $\mathbf{x}_2$.

If this distance is $0$, i.e. the two cases are actually identical, then the similarity - as measured by the rbf-kernel - is $1$:

```python
Image('../images/rbf_kernel.png')
```

```python
import warnings
warnings.filterwarnings('ignore')
from sklearn.neighbors import KernelDensity
import numpy as np
import matplotlib.pyplot as plt
def format_func(x, loc):
    if x == 0:
        return '0'
    elif x == 1:
        return '1'
    elif x == -1:
        return '-1'
    else:
        return '%i' % x
X_plot = np.linspace(-6, 6, 1000)[:, None]
X_src = np.zeros((1, 1))

fig, ax = plt.subplots(1, 1, sharex=True, sharey=True)
fig.subplots_adjust(left=0.05, right=0.95, hspace=0.05, wspace=0.05)
log_dens = KernelDensity(kernel='gaussian').fit(X_src).score_samples(X_plot)
ax.plot(X_plot[:, 0], np.exp(log_dens)/np.max(np.exp(log_dens)), '-k')#, fc='#AAAAFF')
ax.text(-2.6, 0.95, 'gaussian')

ax.xaxis.set_major_formatter(plt.FuncFormatter(format_func))
ax.xaxis.set_major_locator(plt.MultipleLocator(1))
#ax.yaxis.set_major_locator()

ax.set_ylim(0, 1.05)
ax.set_xlim(-2.9, 2.9)
plt.ylabel('similarity of a distance of 0')
density_at_1_5 = np.exp(KernelDensity(kernel='gaussian').fit(X_src).score_samples(np.array([1.5]).reshape((-1,1))))/np.max(np.exp(log_dens))
ax.plot([1.5] *2, [0, density_at_1_5],
        'b--', lw=2, label='')
ax.plot([1.5, -3], [density_at_1_5] * 2, 'b--', lw=2, label='')
ax.plot([0] * 2, [0, 1], 'r--', lw=2, label='')
ax.plot([0, -3], [1] * 2, 'r--', lw=2, label='')
```

# Simple Kernel-Regression
Here, Kernels just define a neighborhood by similarity in kernel-space. For a certain point the values of its neighbors are weighted by similarity and then averaged.

The Euclidean distance on the x-axis is transformed into a similarity-measure on the y-axis.

```python
# f = plt.figure(figsize=(5, 5), dpi=100)
y = np.load('/home/martin/python/fhnw_lecture/scripts/regression_y.pickle.npy')
X = np.load('/home/martin/python/fhnw_lecture/scripts/regression_X.pickle.npy')

fig = plt.figure(figsize=(8,8), constrained_layout=True)
plt.title(label='kernel regression', fontdict={'fontsize':20})
gs = fig.add_gridspec(nrows=3, ncols=3, figure=fig)
#fig.subplots_adjust( left=0.05, right=0.95, wspace=0.05)
#ax1 = fig.add_subplot(gs[:-1, :])
plt.subplot(gs[:-1,:])

#ax1.plot(X[:,0], y, 'ro')
plt.plot(X[:,0], y, 'ro')

for i,j in zip(X[2:7], y[2:7]):
    plt.plot([i] * 2, [-1, j], 'r--', lw=2, label='')
#ax1.set_ylim([np.min(y)-5, np.max(y) +5])
plt.ylim([np.min(y)-5, np.max(y) +5])
#plt.get_yaxis().set_visible(True)
plt.xlim([np.min(X)-2, np.max(X) +2])
#axes = plt.gca()

ax2 = fig.add_subplot(gs[-1:, :])
ax2.set_xlim([np.min(X)-2, np.max(X) +2])
X_plot = np.linspace(1, 9, 1000)[:, None]
X_src = np.array([[4.8]])
log_dens = KernelDensity(kernel='gaussian').fit(X_src).score_samples(X_plot)
ax2.plot(X_plot[:, 0], np.exp(log_dens)/np.max(np.exp(log_dens)), '-k')#, fc='#AAAAFF')

densities = np.exp(KernelDensity(kernel='gaussian').fit(X_src).score_samples(X[2:7, ]))
for i, d in zip(X[2:7,], densities):
    ax2.plot([i] * 2, [d/np.max(np.exp(log_dens)), 1.5], 'r--', lw=2, label='')
    ax2.plot([-5, i], [d/np.max(np.exp(log_dens))] * 2, 'b--', lw=2, label='')

# make vertikal line at 5.0
ax2.plot([4.8] *2, [0, 1.5], 'k--', lw=1)
ax2.set_ylabel('densities (or similarities)')
#plt.tight_layout()

```

The prediction takes part at the center $x$ of the kernel. The similiarities to this center-point $x$ are evaluated for all points $x_i$ in its vicinity and the values $y_i$ are weighted by the respective similarity. Points closer to the center $x$ receive higher weights and have more influence on the predicted value.

In order to make the prediction independent of the exact weights we have to divide by the sum of the weights.

\begin{equation}
\hat{m}(x) = \frac{\sum_i^n K_h(x - x_i)y_i}{\sum_i^n K_h(x - x_i)}
\end{equation}

Here, $h$ is the bandwidth or the scale of the kernel. In the context of a gaussian kernel one could also say the __variance__.

```python
fig = plt.figure(figsize=(8,8), constrained_layout=True)
plt.title(label='kernel regression', fontdict={'fontsize':20})
gs = fig.add_gridspec(nrows=3, ncols=3, figure=fig)
#fig.subplots_adjust( left=0.05, right=0.95, wspace=0.05)
#ax1 = fig.add_subplot(gs[:-1, :])
plt.subplot(gs[:-1,:])

plt.plot(X[:,0], y, 'ro')

for i,j in zip(X[2:7], y[2:7]):
    plt.plot([i] * 2, [0, j], 'r--', lw=2, label='')
plt.ylim([np.min(y)-5, np.max(y) +5])
#ax1.get_yaxis().set_visible(True)
#axes = plt.gca()
x = np.arange(1, 12, 0.05).reshape((-1, 1))
y_kr = []
for x_i in x:
    densities = np.exp(KernelDensity(kernel='gaussian').fit(np.array([[x_i[0]]])).score_samples(X))
    y_kr.append(np.sum(densities.reshape((-1, 1)) * y)/sum(densities))
plt.plot(X[:,0], y, 'ro', x[:,0], np.array(y_kr).reshape((-1,)))
plt.xlim([np.min(X)-2, np.max(X) +2])

# make vertikal line at 8.3
densities = np.exp(KernelDensity(kernel='gaussian').fit(np.array([[8.3]])).score_samples(X))
pred_kr = np.sum(densities.reshape((-1, 1)) * y)/sum(densities)
plt.plot([8.3] * 2, [-1, pred_kr], 'k--', lw=1)
plt.plot(8.3, pred_kr, 'bo')

# make vertikal line at 4.8
densities = np.exp(KernelDensity(kernel='gaussian').fit(np.array([[4.8]])).score_samples(X))
pred_kr = np.sum(densities.reshape((-1, 1)) * y)/sum(densities)
plt.plot([4.8] * 2, [-1, pred_kr], 'k--', lw=1)
plt.plot(4.8, pred_kr, 'bo')

ax2 = fig.add_subplot(gs[-1:, :])
ax2.set_xlim([np.min(X)-2, np.max(X) +2])

X_plot = np.linspace(4.3, 12.3, 1000)[:, None]
X_src = np.array([[8.3]])
log_dens = KernelDensity(kernel='gaussian').fit(X_src).score_samples(X_plot)
ax2.plot(X_plot[:, 0], np.exp(log_dens)/np.max(np.exp(log_dens)), 'k--', lw=1)#, fc='#AAAAFF')

X_plot = np.linspace(0.8, 8.8, 1000)[:, None]
X_src = np.array([[4.8]])
log_dens = KernelDensity(kernel='gaussian').fit(X_src).score_samples(X_plot)
ax2.plot(X_plot[:, 0], np.exp(log_dens)/np.max(np.exp(log_dens)), '-k')#, fc='#AAAAFF')

densities = np.exp(KernelDensity(kernel='gaussian').fit(X_src).score_samples(X[2:7, ]))
for i, d in zip(X[2:7,], densities):
    ax2.plot([i] * 2, [d/np.max(np.exp(log_dens)), 1.5], 'r--', lw=2, label='')
    ax2.plot([-5, i], [d/np.max(np.exp(log_dens))] * 2, 'b--', lw=2, label='')

# make vertikal line at 5.0
ax2.plot([4.8] *2, [0, 1.5], 'k--', lw=1)
# make vertikal line at 8.0
ax2.plot([8.3] *2, [0, 1.5], 'k--', lw=1)

ax2.set_ylabel('densities (or similarities)')
plt.tight_layout()

```

```python
x = np.arange(1, 12, 0.05).reshape((-1, 1))
y_kr = []
for x_i in x:
    densities = np.exp(KernelDensity(kernel='gaussian').fit(np.array([[x_i[0]]])).score_samples(X))
    y_kr.append(np.sum(densities.reshape((-1, 1)) * y)/sum(densities))

f = plt.figure(figsize=(5, 5), dpi=100)
plt.title(label='kernel regression', fontdict={'fontsize':20})
axes = f.add_subplot(111)
axes.plot(X[:,0], y, 'ro', x[:,0], np.array(y_kr).reshape((-1,)))
#axes = plt.gca()
axes.set_ylim([np.min(y)-5, np.max(y) +5])
```

## How is the similarity-measure related to the support-vector-machine?

Remember that in the characteristic equation we had:

$$\mathbf{x}_i^{T} \cdot \mathbf{x}_j$$

This is the dot-product between two vectors. The dot-product for standardized vectors happens to coincide with the cosine-similarity that is identical to the correlation-coefficient $r$ (both are measures for similarity or nearness).

With the kernel trick, we can now apply many different similarity-measures that hopefully help to distinguish our classes better than euclidean distance can do.

Or, put in another way: we can either project our data into higher spaces and compute our conventional similarity measures in those spaces __or__: we use other similarity-measures that approximate the former approach.

```python
## One-Class SVM
## Finde die kleinst mögliche Kugel (hyper sphere), die alle Datenpunkte umschliesst.
## Angewendet auf neue Datenpunkte:
##     - finde die Distanz zum Mittelpunkt: ist diese Grösser als der Radius der hyper sphere, dann ist der Datenpunkt nicht in der Klasse
##     - finde die Distanz zur Begrenuzungsebene: pos Distanz -> in-class; neg Distanz -> out-class
```


# Dimensionality Reduction

## The Problem

There is an interesting tradeoff between model performance and a feature's dimensionality:

>*If the amount of available training data is fixed, then overfitting occurs if we keep adding dimensions. On the other hand, if we keep adding dimensions, the amount of **training data needs to grow exponentially fast to maintain the same coverage** and to avoid overfitting* ([Computer Vision for Dummies](http://www.visiondummy.com/2014/04/curse-dimensionality-affect-classification/)).

### Multi-Collinearity

In many cases, there is a high degree of correlation between many of the features in a dataset. For instance, suppose that you

## Sparsity

- High dimensionality increases the sparsity of your features (**what NLP techniques have we used that illustrate this point?**)
- The density of the training samples decreases when dimensionality increases:
- **Distance measures (Euclidean, for instance) start losing their effectiveness**, because there isn't much difference between the max and min distances in higher dimensions.
- Many models that rely upon **assumptions of Gaussian distributions** (like OLS linear regression), Gaussian mixture models, Gaussian processes, etc. become less and less effective since their distributions become flatter and "fatter tailed".

What is the amount of data needed to maintain **20% coverage** of the feature space? For 1 dimension, it is **20% of the entire population's dataset**. For a dimensionality of $D$:

$$
X^{D} = .20
$$
$$
(X^{D})^{\frac{1}{D}} = .20^{\frac{1}{D}}
$$
$$
X = \sqrt[D]{.20}
$$
You can approximate this as
```python
def coverage_requirement(requirement, D):
    return requirement ** (1 / D)

x = []
y = []
for d in range(1,20):
    y.append(coverage_requirement(0.10, d))
    x.append(d)

import matplotlib.pyplot as plt

plt.plot(x,y)
plt.xlabel("Number of Dimensions")
plt.ylabel("Appromximate % of Population Dataset")
plt.title("% of Dataset Needed to Maintain 10% Coverage of Feature Space")
plt.show()
```

```python
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer
reviews = pd.read_csv("mcdonalds-yelp-sentiment.csv", encoding='latin-1')
reviews = open("poor_amazon_toy_reviews.txt", encoding='latin-1')

#text = reviews["review"].values
text = reviews.readlines()

vectorizer = CountVectorizer(ngram_range=(3,3), min_df=0.01, max_df=0.75, max_features=200)
# tokenize and build vocab
vectorizer.fit(text)
vector = vectorizer.transform(text)
features = vector.toarray()
features_df = pd.DataFrame(features, columns=vectorizer.get_feature_names())

correlations = features_df.corr()
correlations_stacked = correlations.stack().reset_index()
#set column names
correlations_stacked.columns = ['Bi-Gram 1','Bi-Gram 2','Correlation']
correlations_stacked = correlations_stacked[correlations_stacked["Correlation"] < 1]
correlations_stacked = correlations_stacked.sort_values(by=['Correlation'], ascending=False)
correlations_stacked.head()
```

```python
# visualize the correlations (install seaborn first)!

```

# Principle Component Analysis

If you have an original matrix $Z$, you can decompose this matrix into two smaller matrices $X$ and $Q$.

## Important Points:

- Multiplying a vector by a matrix typically changes the direction of the vector. For instance:

Lazy Programmer-
        Tutorial to PCA

However, there are eigenvalues λ and eigenvectors $v$ such that

$$
\sum_{X}v = \lambda v
$$

Multiplying the eigenvectors $v$ with the eigenvalue $\lambda$ does not change the direction of the eigenvector.

Multiplying the eigenvector $v$ by the covariance matrix $\sum_{X}$ also does not change the direction of the eigenvector.

If our data $X$ is of shape $N \times D$, it turns out that we have $D$ eigenvalues and $D$ eigenvectors. This means we can arrange the eigenvalues $\lambda$ in decreasing order so that

$$
\lambda_3 > \lambda_2 > \lambda_5
$$

In this case, $\lambda_3$ is the largest eigenvalue, followed by $\lambda_2$, and then $\lambda_5$. Then, we can arrange

We can also rearrange the eigenvectors the same: $v_3$ will be the first column, $v_2$ will be the second column, and $v_5$ will be the third column.

We'll end up with two matrices $V$ and $\Lambda$:

Lazy Programmer-
        Tutorial to PCA

```python
# what is the shape of our features?
features.shape
```

```python
from sklearn.decomposition import PCA

pca = PCA(n_components=4)

Z = pca.fit_transform(features)

# what is the shape of Z?
Z.shape
```

```python
# what will happen if we take the correlation matrix and covariance matrix of our new reduced features?
import numpy as np
covariances = pd.DataFrame(np.cov(Z.transpose()))
plt.rcParams["figure.figsize"] = (5,5)
sns.heatmap(covariances)
```

```python
# train the model to reduce the dimensions down to 2
pca = PCA(n_components=2)
Z_two_dimensions = pca.fit_transform(features)
Z_two_dimensions
```

```python
import matplotlib.pyplot as plt

plt.scatter(Z_two_dimensions[:,0], Z_two_dimensions[:, 1])
reduced_features_df = pd.DataFrame(Z_two_dimensions, columns=["x1", "x2"])
reduced_features_df["text"] = text
```

# Singular Value Decomposition

Given an input matrix $A$, we want to try to represent it instead as three smaller matrices $U$, $\sum$, and $V$. Instead of **$n$ original terms**, we want to represent each document as **$r$ concepts** (other referred to as **latent dimensions**, or **latent factors**):

        Mining of Massive Datasets - Dimensionality Reduction: Singular Value Decomposition by Leskovec, Rajaraman, and Ullman (Stanford University)

Here, **$A$ is your matrix of word vectors** - you could use any of the word vectorization techniques we have learned so far, include one-hot encoding, word count, TF-IDF.

- $\sum$ will be a **diagonal matrix** with values that are positive and sorted in decreasing order. Its value indicate the **variance (information encoded on that new dimension)**- therefore, the higher the value, the stronger that dimension is in capturing data from A, the original features. For our purposes, we can think of the rank of this $\sum$ matrix as the number of desired dimensions. Instance, if we want to reduce $A$ from shape $1020 x 300$ to $1020 x 10$, we will want to reduce the rank of $\sum$ from 300 to 10.

- $U^T U = I$ and $V^T V = I$

## Measuring the Quality of the Reconstruction

A popular metric used for measuring the quality of the reconstruction is the [Frobenius Norm](https://en.wikipedia.org/wiki/Matrix_norm#Frobenius_norm). When you explain your methodology for reducing dimensions, usually managers / stakeholders will want to some way to compare multiple dimensionality techniques' ability to quantify its ability to retain information but trim dimensions:

$$
\begin{equation}
||A_{old}-A_{new}||_{F} = \sqrt{\sum_{ij}{(A^{old}_{ij}- A^{new}_{ij}}})^2
\end{equation}
$$

## Heuristic Step for How Many Dimensions to Keep

1. Sum the $\sum$ matrix's diagonal values:
$$
\begin{equation}
\sum_{i}^{m}\sigma_{i}
\end{equation}
$$

2. Define your threshold of "information" (variance) $\alpha$ to keep: usually 80% to 90%.

3. Define your cutoff point $C$: $$
\begin{equation}
C = \sum_{i}^{m}\sigma_{i} \alpha
\end{equation}
$$

4. Beginning with your largest singular value, sum your singular values $\sigma_{i}$ until it is greater than C. Retain only those dimensions.

        Mining of Massive Datasets - Dimensionality Reduction: Singular Value Decomposition by Leskovec, Rajaraman, and Ullman (Stanford University)

```python
# create sample data

import numpy as np
import matplotlib.pyplot as plt
from scipy.linalg import svd
x = np.linspace(1,20, 20) # create the first dimension
x = np.concatenate((x,x))
y = x + np.random.normal(0,1, 40) # create the second dimension
z = x + np.random.normal(0,2, 40) # create the third dimension
a = x + np.random.normal(0,4, 40) # create the fourth dimension
plt.scatter(x,y) # plot just the first two dimensions
plt.show()
```

```python
# create matrix
A = np.stack([x,y,z,a]).T
```

```python
# perform SVD
D = 1
U, s, V = svd(A)
print(f"s is {s}\n")
print(f"U is {U}\n")
print(f"V is {V}")
```

```python
# Frobenius norm

s[D:] = 0
S = np.zeros((A.shape[0], A.shape[1]))
S[:A.shape[1], :A.shape[1]] = np.diag(s)
A_reconstructed = U.dot(S.dot(V))
np.sum((A_reconstructed - A) ** 2) ** (1/2) # Frobenius norm
# reconstruct matrix
U.dot(S)
```

# GLOVE

Global vectors for word presentation:

        GloVe: Global Vectors for Word Representation

```python
# import glove embeddings into a word2vec format that is consumable by Gensim
from gensim.scripts.glove2word2vec import glove2word2vec
glove_input_file = 'glove.6B.100d.txt'
word2vec_output_file = 'glove.6B.100d.txt.word2vec'
glove2word2vec(glove_input_file, word2vec_output_file)
```

    (400000, 100)

```python
from gensim.models import KeyedVectors
# load the Stanford GloVe model
filename = 'glove.6B.100d.txt.word2vec'
model = KeyedVectors.load_word2vec_format(filename, binary=False)
# calculate: (king - man) + woman = ?
result = model.most_similar(positive=['woman', 'king'], negative=['man'], topn=1)
print(result)
```

    [('queen', 0.7698541283607483)]

```python
words = ["woman", "king", "man", "queen", "puppy", "kitten", "cat",
         "quarterback", "football", "stadium", "touchdown",
         "dog", "government", "tax", "federal", "judicial", "elections",
         "avocado", "tomato", "pear", "championship", "playoffs"]

vectors = [model.wv[word] for word in words]

import pandas as pd
vector_df = pd.DataFrame(vectors)
vector_df["word"] = words
vector_df.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      91
      92
      93
      94
      95
      96
      97
      98
      99
      word

      0
      0.59368
      0.44825
      0.59320
      0.074134
      0.111410
      1.27930
      0.16656
      0.240700
      0.39045
      0.32766
      ...
      -0.22008
      0.824010
      -0.092207
      -0.459410
      0.465710
      -0.56018
      -0.54648
      0.151620
      -0.30754
      woman

      1
      -0.32307
      -0.87616
      0.21977
      0.252680
      0.229760
      0.73880
      -0.37954
      -0.353070
      -0.84369
      -1.11130
      ...
      -0.71320
      0.089623
      0.413940
      -1.336300
      -0.619150
      -0.33089
      -0.52881
      0.164830
      -0.98878
      king

      2
      0.37293
      0.38503
      0.71086
      -0.659110
      -0.001013
      0.92715
      0.27615
      -0.056203
      -0.24294
      0.24632
      ...
      -0.37501
      0.457420
      0.045975
      -0.246750
      0.045656
      -0.38302
      -0.93711
      0.039138
      -0.53911
      man

      3
      -0.50045
      -0.70826
      0.55388
      0.673000
      0.224860
      0.60281
      -0.26194
      0.738720
      -0.65383
      -0.21606
      ...
      -0.70584
      0.430120
      0.510510
      -0.803300
      -0.665720
      -0.63717
      -0.36032
      0.133470
      -0.56075
      queen

      4
      0.69548
      0.66222
      0.24528
      -1.130400
      -0.768530
      0.10149
      0.36038
      0.048728
      0.13055
      -0.16981
      ...
      0.34369
      0.015695
      0.099343
      -0.004544
      -0.736170
      -0.10185
      0.43978
      -0.003745
      -0.48888
      puppy

5 rows × 101 columns

```python
%matplotlib inline
from sklearn.manifold import TSNE
from sklearn.decomposition import PCA
from sklearn.decomposition import TruncatedSVD
import matplotlib.pyplot as plt

dimension_model = PCA(n_components=2)
reduced_vectors = dimension_model.fit_transform(vectors)

for i, vector in enumerate(reduced_vectors):
    x = vector[0]
    y = vector[1]
    plt.plot(x,y, 'bo')
    plt.text(x * (1 + 0.01), y * (1 + 0.01) , words[i], fontsize=12)

```

# Clustering Text

```python
from sklearn.cluster import KMeans

kmeans = KMeans(n_clusters=4)
cluster_assignments = kmeans.fit_predict(reduced_vectors)
for cluster_assignment, word in zip(cluster_assignments, words):
    print(f"{word} assigned to cluster {cluster_assignment}")

```

    woman assigned to cluster 1
    king assigned to cluster 1
    man assigned to cluster 1
    queen assigned to cluster 1
    puppy assigned to cluster 3
    kitten assigned to cluster 3
    cat assigned to cluster 3
    quarterback assigned to cluster 2
    football assigned to cluster 2
    stadium assigned to cluster 2
    touchdown assigned to cluster 2
    dog assigned to cluster 3
    government assigned to cluster 0
    tax assigned to cluster 0
    federal assigned to cluster 0
    judicial assigned to cluster 0
    elections assigned to cluster 0
    avocado assigned to cluster 3
    tomato assigned to cluster 3
    pear assigned to cluster 3
    championship assigned to cluster 2
    playoffs assigned to cluster 2

```python
color_map = {
    0: "r",
    1: "b",
    2: "g",
    3: "y"
}

plt.rcParams["figure.figsize"] = (10,10)

for i, vector in enumerate(reduced_vectors):
    x = vector[0]
    y = vector[1]
    plt.plot(x,y, 'bo', c=color_map[cluster_assignments[i]])
    plt.text(x * (1 + 0.01), y * (1 + 0.01) , words[i], fontsize=12)
```

```python

```

week7/Dimensionality Reduction and Clustering.ipynb