$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\vx}{\mathbf{x}}
\newcommand{\vy}{\mathbf{y}}
\newcommand{\vt}{\mathbf{t}}
\newcommand{\vb}{\mathbf{b}}
\newcommand{\vw}{\mathbf{w}}
$$

```python
%pylab inline
import numpy as np
import seaborn as sns
import pandas as pd
from Lec08 import *
```

    Populating the interactive namespace from numpy and matplotlib

# EECS 445:  Machine Learning
## Lecture 10:  Bias-Variance Tradeoff, Cross Validation, ML Advice
* Instructor:  **Jacob Abernethy**
* Date:  October 10, 2016

## Announcements

- I'm your new lecturer (for about 6 weeks)!
- Course website: https://eecs445-f16.github.io/
- HW3 out later today, due **Saturday 10/22, 5pm**
- We'll release solutions early Sunday, no late submissions after soln's released!
- Midterm exam is **Monday 10/24** in lecture
- We will release a "topic list" and practice exam early next week
- Key point: if you really understand the HW problems, you'll do fine on the exams

## Comments on Recent Piazza discussions

- We are happy to hear your feedback! But please use [Course Survey #2](https://piazza.com/class/issarttijnz3la?cid=185)
- Anonymous Piazza discussions aren't always helpful, and don't reflect overall student needs (Fully-anonymous posting now disallowed).
- The course staff is working very hard, and are investing a lot more time than previous semesters
- Struggling students need to find an OH to get help! If you can't find a time to attend an OH, tell us!
- We will approve all *Late Drop* requests for those who feel they can't catch up.

## Comments on the Mathematical nature of ML

- We know that students who haven't taken a serious Linear Algebra course, as well as a Probability/Stat course, are finding the mathematical aspects to be challenging. We are working to change course prereqs for future semesters.
- ML may not seem like a mathy topic, but **it certainly is**
- This course is near the frontlines of research, and there aren't yet books on the topic that work for EECS445. (But PRML and MLAPP are pretty good...)
- You can't understand the full nature of these algorithmic tools without having a strong grasp of the math concepts underlying them
- It may be painful now, but we're trying to put you all in the elite category of computer scientists who actually know ML

## Review of SVM

```python
plot_svc();
```

- **Separating Hyperplanes**

    - **Idea:** divide the vector space $\mathbb{R}^d$ where $d$ is the number of features into 2 "decision regions" with a $\mathbb{R}^{d - 1}$  subspace (a hyperplane).
        - Eg. Logistic Regression

    - As with other linear classifiers, classification could be achieved by
        $$
        y = \text{sign}(\vw^T\vx + b)
        $$
        **Note:** We may use $\vx$ and $\phi(\vx)$ interchangeably to denote features.

- **(Functional) Margin**
    - The distance from a separating hyperplane to the *closest* datapoint of *any* class.
        $$
        \rho
        = \rho(\vw, b)
        = \min_{i = 1, ..., n} \frac{| \vw^T\vx_i + b |}{\| \vw \|}
        $$
        where $\mathbf{x}_i$ is the $i$th datapoint from the training set.

### Finding the Max-Margin Hyperplane

- For dataset $\{\vx_i, t_i \}_{i=1}^n $, maximum margin separating hyperplane is the solution of
    $$
    \begin{split}
    \underset{\vw, b}{\text{maximize}} \quad & \min_{i = 1, ..., n} \frac{| \vw^T\vx_i + b |}{\| \vw \|}\\
    \text{subject to} \quad & t_i(\vw^T \vx_i + b) > 0  \quad \forall i \\
    \end{split}
    $$
    of which the constraint ensures every training data is correctly classified
    -  Note that $t_i \in \{+1, -1\}$ is the label of $i$th training data

- This problem guarantees optimal hyperplane, but the solution $\vw$ and $b$ is **not** unique :
    - we could scale both $\vw$ and $b$ by arbitrary scalar without affecting $\mathbb{H} = \{\vx : \vw^T\vx + b = 0\}$
    - we have infinite sets of solutions

### Restatement of Optimization Problem

- Simplifying further, we have
    $$
    \begin{split}
    \underset{\vw, b}{\text{maximize}} \quad & \frac{1}{\| \vw \|}\\
    \text{subject to} \quad & t_i(\vw^T \vx_i + b) = 1 \text{ for some } i\\
     \quad & t_i(\vw^T \vx_i + b) > 1  \text{ for other } i\\
    \end{split}
    \Longrightarrow
    \begin{split}
    \underset{\vw, b}{\text{minimize}} \quad & \frac{1}{2}{\| \vw \|}^2\\\
    \text{subject to} \quad & t_i(\vw^T \vx_i + b) \geq 1 \quad \forall i \\
    \quad
    \end{split}
    $$

### Optimal Soft-Margin Hyperplane (OSMH)

- To deal with non-linearly separable case, we could introduce slack variables:
    $$
    \begin{split}
    \underset{\vw, b}{\text{min}} \quad & \frac{1}{2}{\| \vw \|}^2\\\
    \text{s.t.} \quad & t_i(\vw^T \vx_i + b) \geq 1 \; \forall i \\
    & \\
    \end{split}
    \;
    \Longrightarrow
    \;
    \begin{split}
    \underset{\vw, b, \xi}{\text{min}} \quad & \frac{1}{2}{\| \vw \|}^2 + \frac{C}{n} \sum \nolimits_{i = 1}^n \xi_i\\
    \text{s.t.} \quad & t_i(\vw^T\vx_i + b) \geq 1 - \xi_i \; \forall i\\
     \quad & \xi_i \geq 0 \; \forall i\\
    \end{split}
    $$

- New term $\frac{C}{n} \sum_{i = 1}^n \xi_i$ penalizes errors and accounts for the influence of outliers through a constant $C \geq 0$ ($C=\infty$ would lead us back to the hard margin case) and $\mathbf{\xi} = [\xi_1, ..., \xi_n]$ are the "slack" variables.

- **Motivation:**
    - The **objective function** ensures margin is large *and* the margin violations are small
    - The **first set of constraints** ensures classifier is doing well
        * similar to the prev. max-margin constraint, except we now allow for slack
    - The **second set of constraints** ensure slack variables are non-negative.
        - keeps the optimization problem from *"diverging"*

### OSMH has *Dual* Formulation

- The previous objective function is referred to as the *Primal*
    - With $N$ datapoints in $d$ dimensions, the Primal optimizes over $d + 1$ variables ($\vw, b$).
- But the *Dual* of this optimization problem has $N$ variables, one $\alpha_i$ for each example $i$!

$$
    \begin{split}
    \underset{\alpha, \beta}{\text{maximize}} \quad &  -\frac12 \sum \nolimits_{i,j = 1}^n \alpha_i \alpha_j t_i t_j \vx_i^T \vx_j + \sum \nolimits_{i = 1}^n \alpha_i\\
    \text{subject to} \quad & 0 \leq \alpha_i \leq C/n \quad \forall i\ \\
    \quad & \sum \nolimits_{i=1}^n \alpha_i t_i = 0
    \end{split}
$$

- Often the Dual problem is easier to solve.
- Once you solve the dual problem for $\alpha^*_1, \ldots, \alpha^*_N$, you get a primal solution as well!

$$
\vw^* = \sum \nolimits_{i=1}^n \alpha_i^* t_i \vx_i \quad \text{and} \quad b^* = t_i - {\vw^*}^T\vx_i \; (\text{ for any } i)
$$

- Note: Generally we can't solve these by hand, one uses optimization packages (such as a QP solver)

# Statistical Inference
## Loss Functions & Bias-Variance Decomposition

### Estimators

- ML Algorithms can in general be thought of as "estimators."
> **Estimator:** A statistic (a function of data) that is used to infer the value of an unknown parameter in a statistical model.

- Suppose there is a fixed parameter $f$ that needs to be estimated. An estimator of $f$ is a function that maps the sample space to a set of sample estimates, denoted $\hat{f}$.

### Noise

- For most problems in Machine Learning, the relationship is functional but noisy.

- Mathematically, $y = f(x) + \epsilon$
    - $\epsilon$ is noise with mean $0$ variance $\sigma^2$

### Mathematical Viewpoint

- Let the training set be $D = \{\mathbf{x}_1, ..., \mathbf{x}_n\}, \mathbf{x}_i \in \mathbb{R}^d$.
- **Goal:** Find $\hat{f}$ that minimizes some **Loss function**, $L(y, \hat{f})$, which measures how good predictions are for **both**
 - Points in $D$ (the **sample**), and,
 - Points ***out of sample*** (outside $D$).
- Cannot minimize both perfectly because the relationship between $y$ and $\mathbf{x}$ is noisy.
 - ***Irreducible error***.

### Loss Functions

There are many loss functions, each with their own use cases and interpretations.

- **Quadratic Loss:**  $L(y,\hat{f}) = (y-\hat{f})^2$
- **Absolute Loss:**  $L(y,\hat{f}) = |y-\hat{f}|$

Classification-only loss functions:
- **Sigmoid Loss:**  $L(y,\hat{f}) = \mathrm{sigmoid}(-y\hat{f})$
- **Zero-One Loss:**  $L(y,\hat{f}) = \mathbb{I}(y \neq \hat{f})$
- **Hinge Loss:**  $L(y,\hat{f}) = \max(0, 1-y\hat{f})$
- **Logistic Loss:**  $L(y,\hat{f}) = \log[ 1 + \exp(-y\hat{f})]$
- **Exponential Loss:**  $L(y,\hat{f}) = \exp[ -y \hat{f} ]$

### Choosing a Loss Function

Different loss functions answer the following questions differently:

- How should we treat **outliers**?
- How **"correct"** do we need to be?
    - Do we want a **margin** of safety?
- What is our notion of **distance**?  What are we predicting?
    - Real-world measurements?
    - Probabilities?

### Quadratic Loss (aka Square Loss)

 - Commonly used for regression
 - Heavily influenced by outliers

$$
L(y, \hat{f}) = (y - \hat{f})^2
$$

```python
x = np.linspace(-1, 1, 100);
plt.plot(x, x**2)
plt.xlabel("$y-\hat{f}$", size=18);
```

### Absolute Loss

- Commonly used for regression.
- Robust to outliers.

$$
L(y, \hat{f}) = |y - \hat{f}|
$$

### Absolute Loss:  Plot

```python
x = np.linspace(-1, 1, 100);
plt.plot(x, np.abs(x));
plt.xlabel("$y-\hat{f}$", size=18);
plt.ylabel("$|y-\hat{f}|$", size=18);
```

### 0-1 Loss

 - Used for classification.
 - Not convex!
  - Not practical since optimization problems become intractable!
  - "Surrogate Loss functions" that are convex and differentiable can be used instead.

$$
L(y, \hat{f}) = \mathbb{I}(y \neq \hat{f})
$$

### Sigmoid Loss

 - Differentiable but non-convex! Can be used for classification.

$$L(y,\hat{f}) = \mathrm{sigmoid}(-y\hat{f})$$

```python
x = np.linspace(-6, 6, 100);
plt.plot(x, 1/(1 + np.exp(-x)));
plt.xlabel("$-y\hat{f}$", size=18);
plt.ylabel("$\sigma(-y\hat{f})$", size=18);
```

### Logistic Loss

 - Used in Logistic regression.
 - Influenced by outliers.
 - Provides well calibrated probabilities (can be interpreted as confidence levels).

$$L(y,\hat{f}) = \log[ 1 + \exp(-y\hat{f})]$$

```python
x = np.linspace(-6, 6, 100);
plt.plot(x, np.log2(1 + np.exp(-x)));
plt.xlabel("$y\hat{f}$", size=18);
plt.ylabel("$\log(1 + \exp(-y\hat{f}))$", size=18);
```

### Hinge Loss

 - Used in SVMs.
 - Robust to outliers.
 - Doesn't provide well calibrated probabilities.
$$L(y,\hat{f}) = \max(0, 1-y\hat{f})$$

```python
x = np.linspace(-6, 6, 100);
plt.plot(x, np.where(x < 1, 1 - x, 0));
plt.xlabel("$y\hat{f}$", size=18); plt.ylabel("$\max(0,1-y\hat{f})$", size=18);
```

### Exponential Loss

- Used for Boosting.
- Very susceptible to outliers.

$$L(y,\hat{f}) = \exp(-y\hat{f})$$

```python
x = np.linspace(-3, 3, 100);
plt.plot(x, np.exp(-x));
plt.xlabel("$y\hat{f}$", size=18);
plt.ylabel("$\exp(-y\hat{f})$", size=18);
```

### Loss Functions:  Comparison

```python
# adapted from http://scikit-learn.org/stable/auto_examples/linear_model/plot_sgd_loss_functions.html
def plot_loss_functions():
    xmin, xmax = -4, 4
    xx = np.linspace(xmin, xmax, 100)
    plt.plot(xx, xx ** 2, 'm-',
             label="Quadratic loss")
    plt.plot([xmin, 0, 0, xmax], [1, 1, 0, 0], 'k-',
             label="Zero-one loss")
    plt.plot(xx, 1/(1 + np.exp(xx)), 'b-',
             label="Sigmoid loss")
    plt.plot(xx, np.where(xx < 1, 1 - xx, 0), 'g-',
             label="Hinge loss")
    plt.plot(xx, np.log2(1 + np.exp(-xx)), 'r-',
             label="Log loss")
    plt.plot(xx, np.exp(-xx), 'c-',
             label="Exponential loss")
    plt.ylim((0, 8))
    plt.legend(loc="best")
    plt.xlabel(r"Decision function $f(x)$")
    plt.ylabel("$L(y, f)$")
```

```python
# Demonstrate some loss functions
plot_loss_functions()
```

## Break time!

### Risk

**Risk** is the expected loss or error.
- Calculated differently for Bayesian vs. Frequentist Statistics

For now, assume **quadratic loss** $L(y,\hat{f}) = (y-\hat{f})^2$
- Associated risk is $R(\hat{f}) = E_y[L(y, \hat{f})] = E_y[(y-\hat{f})^2]$

### Bias-Variance Decomposition

- Can decompose the expected loss into a **bias** term and **variance** term.
- Depending on samples, learning process can give different results
    - ML vs MAP vs Posterior Mean, etc..
- We want to learn a model with
    - Small bias (how well a model fits the data on average)
    - Small variance (how stable a model is w.r.t. data samples)

### Bias-Variance Decomposition

$$
\begin{align}
\mathbb{E}[(y - \hat{f})^2]
&= \mathbb{E}[y^2 - 2 \cdot y \cdot \hat{f} + {\hat{f}}^2] \\
&= \mathbb{E}[y^2] - \mathbb{E}[2 \cdot y \cdot \hat{f}] + \mathbb{E}[{\hat{f}}^2] \\
&= \mathrm{Var}[y] + {\mathbb{E}[y]}^2 - \mathbb{E}[2 \cdot y \cdot \hat{f}] +
   \mathrm{Var}[\hat{f}] + {\mathbb{E}[{\hat{f}}]}^2
\end{align}
$$

since $Var[X] = \mathbb{E}[{X}^2] - {\mathbb{E}[X]}^2 \implies \mathbb{E}[X^2] = Var[X] + {\mathbb{E}[X]}^2$

### Bias-Variance Decomposition

$$\begin{align} \mathbb{E}[y] &= \mathbb{E}[f + \epsilon] \\
               &= \mathbb{E}[f] + \mathbb{E}[\epsilon] & \text{ (linearity of expectations)}\\
               &= \mathbb{E}[f] + 0 &\text{(zero-mean noise)}\\
               &= f & \text{ (} f \text{ is determinstic)}\end{align}$$

### Bias-Variance Decomposition

$$\begin{align} Var[y] &= \mathbb{E}[(y - \mathbb{E}[y])^2] \\
                      &= \mathbb{E}[(y - f)^2] \\
                      &= \mathbb{E}[(f + \epsilon - f)^2] \\
                      &= \mathbb{E}[\epsilon^2] \equiv \sigma^2 \end{align}$$

### Bias-Variance Decomposition

We just showed that:
- $\mathbb{E}[y] = f$
- $\mathrm{Var}[y] = \mathbb{E}[\epsilon^2] = \sigma^2$

Therefore,

$$
\begin{align}
\mathbb{E}[(y - \hat{f})^2]
&= Var[y] + {\mathbb{E}[y]}^2 - \mathbb{E}[2 \cdot y \cdot \hat{f} + Var[\hat{f}] + {\mathbb{E}[{\hat{f}}]}^2 \\
&= \sigma^2 + f^2 - \mathbb{E}[2 \cdot y \cdot \hat{f}] + Var[\hat{f}] + {\mathbb{E}[{\hat{f}}]}^2
\end{align}
$$

### Bias-Variance Decomposition

- Note $y$ is random ***only*** in $\epsilon$ (again, $f$ is deterministic).
- Also, $\epsilon$ is ***independent*** from $\hat{f}$.

$\begin{align}\mathbb{E}[2 \cdot y \cdot \hat{f}]
                      &= \mathbb{E}[2 \cdot y \cdot \hat{f}]\\
                      &= \mathbb{E}[2 \cdot y] \cdot \mathbb{E}[\hat{f}] & \text{ (by independence) }\\
                      &= 2 \cdot \mathbb{E}[y] \cdot \mathbb{E}[\hat{f}] \\
                      &= 2 \cdot f \cdot \mathbb{E}[\hat{f}] \end{align}$

Thus, we now have $\mathbb{E}[(y - \hat{f})^2] = \sigma^2 + f^2 - 2 \cdot f \cdot \mathbb{E}[\hat{f}] + Var[\hat{f}] + {\mathbb{E}[{\hat{f}}]}^2$

### Bias-Variance Decomposition

$\mathbb{E}[(y - \hat{f})^2] = \sigma^2 + Var[\hat{f}] + f^2 - 2 \cdot f \cdot \mathbb{E}[\hat{f}] + {\mathbb{E}[{\hat{f}}]}^2$

Now, $f^2 - 2 \cdot f \cdot \mathbb{E}[\hat{f}] + \mathbb{E}[\hat{f}]^2 = (f - \mathbb{E}[\hat{f}])^2$

$\implies \mathbb{E}[(y - \hat{f})^2] = \sigma^2 + Var[\hat{f}] + (f - \mathbb{E}[\hat{f}])^2$

$\begin{align} \text{Finally, } \mathbb{E}[f - \hat{f}]
                        &= \mathbb{E}[f] - \mathbb{E}[\hat{f}] \text{ (linearity of expectations)} \\
                        &= f - \mathbb{E}[\hat{f}] \end{align}$

So,
$$\mathbb{E}[(y - \hat{f})^2] = \underbrace{{\sigma^2}}_\text{irreducible error} + \underbrace{{\text{Var}[\hat{f}]}}_\text{Variance} + \underbrace{{\mathbb{E}[f - \mathbb{E}[\hat{f}]]}^2}_{\text{Bias}^2}$$

### Bias-Variance Decomposition

We have
$$\mathbb{E}[(y - \hat{f})^2] = \underbrace{{\sigma^2}}_\text{irreducible error} + \underbrace{{\text{Var}[\hat{f}]}}_\text{Variance} + \underbrace{{\mathbb{E}[f - \mathbb{E}_S[\hat{f}]]}^2}_{\text{Bias}^2}$$

### Bias and Variance Formulae

Bias of an estimator, $B(\hat{f}) = \mathbb{E}[\hat{f}] - f$

Variance of an estimator, $Var(\hat{f}) = \mathbb{E}[(\hat{f} - \mathbb{E}[\hat{f}])^2]$

### An example to explain Bias/Variance and illustrate the tradeoff

- Consider estimating a sinusoidal function.

(Example that follows is inspired by Yaser Abu-Mostafa's CS 156 Lecture titled "Bias-Variance Tradeoff"

```python
import pylab as pl

RANGEXS = np.linspace(0., 2., 300)
TRUEYS = np.sin(np.pi * RANGEXS)

def plot_fit(x, y, p, show,color='k'):
    xfit = RANGEXS
    yfit = np.polyval(p, xfit)
    if show:
        axes = pl.gca()
        axes.set_xlim([min(RANGEXS),max(RANGEXS)])
        axes.set_ylim([-2.5,2.5])
        pl.scatter(x, y, facecolors='none', edgecolors=color)
        pl.plot(xfit, yfit,color=color)
        pl.hold('on')
        pl.xlabel('x')
        pl.ylabel('y')
```

```python
def calc_errors(p):
    x = RANGEXS
    errs = []
    for i in x:
        errs.append(abs(np.polyval(p, i) - np.sin(np.pi * i)) ** 2)
    return errs
```

```python
def calculate_bias_variance(poly_coeffs, input_values_x, true_values_y):
    # poly_coeffs: a list of polynomial coefficient vectors
    # input_values_x: the range of xvals we will see
    # true_values_y: the true labels/targes for y

    # First we calculate the mean polynomial, and compute the predictions for this mean poly
    mean_coeffs = np.mean(poly_coeffs, axis=0)
    mean_predicted_poly = np.poly1d(mean_coeffs)
    mean_predictions_y = np.polyval(mean_predicted_poly, input_values_x)

    # Then we calculate the error of this mean poly
    bias_errors_across_x = (mean_predictions_y - true_values_y) ** 2

    # To consider the variance errors, we need to look at every output of the coefficients
    variance_errors = []
    for coeff in poly_coeffs:
        predicted_poly = np.poly1d(coeff)
        predictions_y = np.polyval(predicted_poly, input_values_x)
        # Variance error is the average squared error between the predicted values of y
        # and the *average* predicted value of y
        variance_error = (mean_predictions_y - predictions_y)**2
        variance_errors.append(variance_error)

    variance_errors_across_x = np.mean(np.array(variance_errors),axis=0)

    return bias_errors_across_x, variance_errors_across_x
```

```python
from matplotlib.pylab import cm
def polyfit_sin(degree=0, iterations=100, num_points=5, show=True):
    total = 0
    l = []
    coeffs = []
    errs = [0] * len(RANGEXS)
    colors=cm.rainbow(np.linspace(0,1,iterations))
    for i in range(iterations):
        np.random.seed()
        x = np.random.choice(RANGEXS,size=num_points) # Pick random points from the sinusoid
        y = np.sin(np.pi * x)
        p = np.polyfit(x, y, degree)
        y_poly = [np.polyval(p, x_i) for x_i in x]
        plot_fit(x, y, p, show,color=colors[i])
        total += sum(abs(y_poly - y) ** 2) # calculate Squared Error (Squared Error)
        coeffs.append(p)
        errs = np.add(calc_errors(p), errs)
    return total / iterations, errs / iterations, np.mean(coeffs, axis = 0), coeffs
```

```python
def plot_bias_and_variance(biases,variances,range_xs,true_ys,mean_predicted_ys):
    pl.plot(range_xs, mean_predicted_ys, c='k')
    axes = pl.gca()
    axes.set_xlim([min(range_xs),max(range_xs)])
    axes.set_ylim([-3,3])
    pl.hold('on')
    pl.plot(range_xs, true_ys,c='b')
    pl.errorbar(range_xs, mean_predicted_ys, yerr = biases, c='y', ls="None", zorder=0,alpha=1)
    pl.errorbar(range_xs, mean_predicted_ys, yerr = variances, c='r', ls="None", zorder=0,alpha=0.1)
    pl.xlabel('x')
    pl.ylabel('y')
```

## Let's return to fitting polynomials

* Here we generate some samples $x,y$, with $y = \sin(2\pi x)$
* We then fit a *degree-0* polynomial (i.e. a constant function) to the samples

```python
# polyfit_sin() generates 5 samples of the form (x,y) where y=sin(2*pi*x)
# then it tries to fit a degree=0 polynomial (i.e. a constant func.) to the data
# Ignore return values for now, we will return to these later
_, _, _, _ = polyfit_sin(degree=0, iterations=1, num_points=5, show=True)
```

## We can do this over many datasets

* Let's sample a number of datasets
* How does the fitted polynomial change for different datasets?

```python
# Estimate two points of sin(pi * x) with a constant 5 times
_, _, _, _ = polyfit_sin(0, 5)
```

## What about over lots more datasets?

```python
# Estimate two points of sin(pi * x) with a constant 100 times
_, _, _, _ = polyfit_sin(0, 25)
```

```python
MSE, errs, mean_coeffs, coeffs_list = polyfit_sin(0, 100,num_points = 3,show=False)
biases, variances = calculate_bias_variance(coeffs_list,RANGEXS,TRUEYS)
plot_bias_and_variance(biases,variances,RANGEXS,TRUEYS,np.polyval(np.poly1d(mean_coeffs), RANGEXS))
```

* Decomposition: $\mathbb{E}[(y - \hat{f})^2] = \underbrace{{\sigma^2}}_\text{irreducible error} + \underbrace{{\text{Var}[\hat{f}]}}_\text{Variance} + \underbrace{{\mathbb{E}[f - \mathbb{E}_S[\hat{f}]]}^2}_{\text{Bias}^2}$
* Blue curve: true $f$
* Black curve: $\hat f$, average predicted values of $y$
* Yellow is error due to **Bias**, Red/Pink is error due to **Variance**

## Bias vs. Variance
* We can calculate how much error we suffered due to bias and due to variance

```python
poly_degree = 0
results_list = []
MSE, errs, mean_coeffs, coeffs_list = polyfit_sin(
    poly_degree, 500,num_points = 5,show=False)
biases, variances = calculate_bias_variance(coeffs_list,RANGEXS,TRUEYS)
sns.barplot(x='type', y='error',hue='poly_degree', data=pd.DataFrame([
    {'error':np.mean(biases), 'type':'bias','poly_degree':0},
    {'error':np.mean(variances), 'type':'variance','poly_degree':0}]))
```

## Let's now fit degree=3 polynomials

* Let's sample a dataset of 5 points and fit a cubic poly

```python
MSE, _, _, _ = polyfit_sin(degree=3, iterations=1)
```

## Let's now fit degree=3 polynomials

* What does this look like over 5 different datasets?

```python
_, _, _, _ = polyfit_sin(degree=3,iterations=5,num_points=5,show=True)
```

## Let's now fit degree=3 polynomials

* What does this look like over 50 different datasets?

```python
# Estimate two points of sin(pi * x) with a line 50 times
_, _, _, _ = polyfit_sin(degree=3, iterations=50)
```

```python
MSE, errs, mean_coeffs, coeffs_list = polyfit_sin(3,500,show=False)
biases, variances = calculate_bias_variance(coeffs_list,RANGEXS,TRUEYS)
plot_bias_and_variance(biases,variances,RANGEXS,TRUEYS,np.polyval(np.poly1d(mean_coeffs), RANGEXS))
```

$$\mathbb{E}[(y - \hat{f})^2] = \underbrace{{\sigma^2}}_\text{irreducible error} + \underbrace{{\text{Var}[\hat{f}]}}_\text{Variance} + \underbrace{{\mathbb{E}[f - \mathbb{E}_S[\hat{f}]]}^2}_{\text{Bias}^2}$$
* Blue curve: true $f$
* Black curve: $\hat f$, average *prediction* (of the value of $y$)
* Yellow is error due to **Bias**, Red/Pink is error due to **Variance**

```python
results_list = []
for poly_degree in [0,1,3]:
    MSE, errs, mean_coeffs, coeffs_list = polyfit_sin(poly_degree,500,num_points=5,show=False)
    biases, variances = calculate_bias_variance(coeffs_list,RANGEXS,TRUEYS)
    results_list.append({'error':np.mean(biases),
                         'type':'bias', 'poly_degree':poly_degree})
    results_list.append({'error':np.mean(variances),
                         'type':'variance', 'poly_degree':poly_degree})

sns.barplot(x='type', y='error',hue='poly_degree',data=pd.DataFrame(results_list))
```

### Bias Variance Tradeoff

#### Central problem in supervised learning.

Ideally, one wants to choose a model that both accurately captures the regularities in its training data, but also generalizes well to unseen data. Unfortunately, it is typically impossible to do both simultaneously.

- High Variance:
 - Model represents the training set well.
 - Overfit to noise or unrepresentative training data.
 - Poor generalization performance

- High Bias:
 - Simplistic models.
 - Fail to capture regularities in the data.
 - May give better generalization performance.

### Interpretations of Bias
 - Captures the errors caused by the simplifying assumptions of a model.
 - Captures the average errors of a model across different training sets.

### Interpretations of Variance
 - Captures how much a learning method moves around the mean.
 - How different can one expect the hypotheses of a given model to be?
 - How sensitive is an estimator to different training sets?

### Complexity of Model

- Simple models generally have high bias and complex models generally have low bias.
- Simple models generally have low variance andcomplex models generally have high variance.

- Underfitting / Overfitting
 - High variance is associated with overfitting.
 - High bias is associated with underfitting.

### Training set size

- Decreasing the training set size
 - Helps with a high bias algorithm:
  - Will in general not help in improving performance.
  - Can attain the same performance with smaller training samples however.
  - Additional advantage of increases in speed.

- Increase the training set size
 - Decreases Variance by reducing overfitting.

### Number of features
- Increasing the number of features.
 - Decreases bias at the expense of increasing the variance.

- Decreasing the number of features.
 - Dimensionality reduction can decrease variance by reducing over-fitting.

### Features

Many techniques for engineering and selecting features (Feature Engineering and Feature Extraction)
 - PCA, Isomap, Kernel PCA, Autoencoders, Latent sematic analysis, Nonlinear dimensionality reduction, Multidimensional Scaling

### Features

The importance of features
 > "Coming up with features is difficult, time-consuming, requires expert knowledge.  Applied machine learning is basically feature engineering"
 - Andrew Ng

 > "... some machine learning projects succeed and some fail.  What makes the difference? Easily the most important factor is the features used."
 - Pedro Domingo

### Regularization (Changing $\lambda$ or $C$)

Regularization is designed to impose simplicity by adding a penalty term that depends on the charactistics of the parameters.

- Decrease Regularization.
 - Reduces bias (allows the model to be more complex).

- Increase Regularization.
 - Reduces variance by reducing overfitting (again, regularization imposes "simplicity.")

### Ideal bias and variance?

- All is not lost. Bias and Variance can both be lowered through some methods:
 - Ex: Boosting (learning from weak classifiers).

- The sweet spot for a model is the level of complexity at which the increase in bias is equivalent to the reduction in variance.

# Model Selection

### Model Selection

- ML Algorithms generally have a lot of parameters that must be chosen. A natural question is then "How do we choose them?"
 - Examples: Penalty for margin violation (C), Polynomial Degree in polynomial fitting

### Model Selection

- Simple Idea:
 - Construct models $M_i, i = 1, ..., n$.
 - Train each of the models to get a hypothesis $h_i, i = 1, ..., n$.
 - Choose the best.
- Does this work? No! Overfitting. This brings us to **cross validation**.

### Hold-Out Cross Validation

(1) Randomly split the training data $D$ into $D_{train}$ and $D_{val}$, say 70% of the data and 30% of the data respectively.

(2) Train each model $M_i$ on $D_{train}$ only, each time getting a hypothesis $h_i$.

(3) Select and output hypothesis $h_i$ that had the smallest error on the held out validation set.

Disadvantages:
 - Waste some sizable amount of data (30\% in the above scenario) so that less training examples are available.
 - Using only some data for training and other data for validation.

### K-Fold Cross Validation (Step 1)

Randomly split the training data $D$ into $K$ ***disjoint*** subsets of $N/K$ training samples each.
 - Let these subsets be denoted $D_1, ..., D_K$.

### K-Fold Cross Validation (Step 2)

For each model $M_i$, we evaluate the model as follows:
 - Train the model $M_i$ on $D \setminus D_k$ (all of the subsets except subset $D_k$) to get hypothesis $h_i(k)$.
 - Test the hypothesis $h_i(k)$ on $D_k$ to get the error (or loss) $\epsilon_i(k)$.
 - Estimated generalization error for model $M_i$ is then given by $e^g_i = \frac{1}{K} \sum \limits_{k = 1}^K \epsilon_i (k)$

### K-Fold Cross Validation (Step 3)

Pick the model $M_i^*$ with the lowest estimated generalization error $e^{g*}_i$ and retrain the model on the entire training set, thus giving the final hypothesis $h^*$ that is output.

### Three Way Data Splits

- If model selection and true error estimates are to be computed simaltaneously, the data needs to be divided into three disjoin sets.

- Training set: A set of examples used for learning
- Validation set: A set of examples used to tune the hyperparameters of a classifier.
- Test Set: A set of examples used *** only *** to assess the performance of a fully-trained model.

### Procedure Outline

1. Divide the available data into training, validation and test set
2. Select a model (and hyperparameters)
3. Train the model using the training set
4. Evaluate the model using the validation set
5. Repeat steps 2 through 4 using different models (and hyperparameters)
6. Select the best model (and hyperparameter) and train it using data from the training and validation set
7. Assess this final model using the test set

### How to choose hyperparameters?

Cross Validation is only useful if we have some number of models. This often means constructing models each with a different combination of hyperparameters.

### Random Search
 - Just choose each hyperparameter randomly (possibly within some range for each.)
 - Pro: Easy to implement. Viable for models with a small number of hyperparameters and/or low dimensional data.
 - Con: Very inefficient for models with a large number of hyperparameters or high dimensional data (curse of dimensionality.)

### Grid Search / Parameter Sweep
 - Choose a subset for each of the parameters.
  - Discretize real valued parameters with step sizes as necessary.
 - Output the model with the best cross validation performance.
 - Pro: "Embarassingly Parallel" (Can be easily parallelized)
 - Con: Again, curse of dimensionality poses problems.

### Bayesian Optimization

- Assumes that there is a smooth but noisy relation that acts as a mapping from hyperparameters to the objective function.

- Gather observations in such a manner as to evaluate the machine learning model the least number of times while revealing as much information as possible about the mapping and, in particular, the location of the optimum.

- Exploration vs. Exploitation problem.

### Learning Curves
Provide a visualization for diagnostics such as:
- Bias / variance
- Convergence

```python
# Image from Andrew Ng's Stanford CS229 lecture titled "Advice for applying machine learning"
from IPython.display import Image
Image(filename='images/HighVariance.png', width=800, height=600)

# Testing error still decreasing as the training set size increases. Suggests increasing the training set size.
# Large gap Between Training and Test Error.
```

```python
# Image from Andrew Ng's Stanford CS229 lecture titled "Advice for applying machine learning"
from IPython.display import Image
Image(filename='images/HighBias.png', width=800, height=600)

# Training error is unacceptably high.
# Small gap between training error and testing error.
```

### Convergence

- Approach 1:
 - Measure gradient of the learning curve.
 - As learning curve gradient approaches 0, the model has been trained. Choose threshold to stop training.

- Approach 2:
 - Measure change in the model parameters each iteration of the algorithm.
 - One can assume that training is complete when the change in model parameters is below some threshold.

### Diagnostics related to Convergence (1)
- Convergence too slow?
 - Try using Newton's method.
 - Larger step size.
  - Note that too large of a step size could also lead to slow convergence (but the learning curves in general will then suggest instability if "oscillations" are occuring.)
 - Decrease batch size if using a batch based optimization algorithm.

### Diagnostics related to Convergence (2)

- Are the learning curves stable? If not:
 - Switch to a batch style optimization algorithm if not already using one (like minibatch gradient descent / gradient descent).
 - Increase batch sizes if already using one.
- Some algorithms always ensure a decrease or increase in the objective function each iterations. Ensure that this is the case if the optimization algorithm being used provides such guarantees.

### Ablative Analysis

- Similar to the idea of cross validation, except for components of a system.

- Example: Simple Logisitic Regression on spam classification gives 94% performance.
 - 95% with spell correction
 - 96% with top 100 most commonly used words removed
 - 98% with extra sender and receiver information
 - 99% overall performance


 ILI285 - Computación Científica I / INF285 - Computación Científica

 Least Squares

 [[S]cientific [C]omputing [T]eam](#acknowledgements)

 Version: 1.24

## Table of Contents
* [Introduction](#intro)
* [QR Factorization](#qr)
* [Examples](#ex)
* [Inconsistents Systems](#in)
* [A Survey of Models](#sm)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg as spla
%matplotlib inline
from sklearn import datasets
import ipywidgets as widgets
import matplotlib as mpl
mpl.rcParams['font.size'] = 14
mpl.rcParams['axes.labelsize'] = 20
mpl.rcParams['xtick.labelsize'] = 14
mpl.rcParams['ytick.labelsize'] = 14
# from scipy.interpolate import CubicSpline # Starting scipy 0.19.0
M=8
```

## Introduction

We have learned about square linear system of equations.
However, How can we solve a non-square system? (More equations that unknowns!) Well, we need to find a least squares approximation.

# QR Factorization

## Gram-Schmidt Orthogonalization

This algorithm orthogonalize a set of input vectors, returning an **orthogonal set** that spans the same column space.
We will only consider now that the input set of vectors are **linearly independent**.

Let $A=[A_1\, ...., A_n]$ a matrix with linearly independent column vectors $\in\mathbb{R}^m$ and $n \le m$.

We know the following for the orthogonal set:

\begin{align*}
    q_i^*\,q_i & = \|q_i\|_2^2= 1\\
    q_i^*\,q_j & = 0, \, \text{ for } i\neq j
\end{align*}

Then the Gram-Schmidt orthogonalization finds the $q_i$ and $r_{ij}$ from the following set of equations and considering the previous constraints:
\begin{align*}
    A_1 &= r_{11}\,q_1\\
    r_{11} &= \|A_1\|_2\\
    q_1 &= \dfrac{y_1}{r_{11}}\\
    A_2 &= r_{12}\,q_1+r_{22}\,q_2\\
    r_{12} &= q_1^*\,A_2\\
    r_{22} &= \|A_2-r_{12}\,q_1\|\\
    q_2 &= \dfrac{A_2-r_{12}\,q_1}{r_{22}}\\
    \vdots &=  \vdots\\
    A_j &= \sum_{i=1}^j r_{ij}\,q_i\\
    r_{ij} &= q_i^*\,A_j, \, \text{ for } i<j\\
    r_{jj} &= \|A_j-\sum_{i=1}^{j-1} r_{ij}\,q_i\|\\
    q_j &= \dfrac{A_j-\sum_{i=1}^{j-1} r_{ij}\,q_i}{r_jj}\\
    \vdots &=  \vdots\\
    A_n &= \sum_{i=1}^n r_{in}\,q_i\\
    r_{in} &= q_i^*\,A_n, \, \text{ for } i<n\\
    r_{nn} &= \|A_n-\sum_{i=1}^{n-1} r_{in}\,q_i\|\\
    q_n &= \dfrac{A_n-\sum_{i=1}^{n-1} r_{in}\,q_i}{r_{nn}}
\end{align*}

Thus, we obtain the QR decomposition as follows:

\begin{equation}
                 A_{m\times n} = Q_{m\times n}R_{n\times n}\\
\end{equation}

Where $Q$ is a matrix of vectors $q_{n}$, and $R$ is an upper-triangular matrix, with the coefficients $r_{ij}$:

This is known as the **Reduced QR Factorization**.

_**[IMPORTANT]** What is then a **full** QR decomposition?_

```python
# Inputs:
# A: A set of linearly independent columns
# type_factorization: reduced or full
# type_gram_schmidt: classic or modified
def QR(A, type_factorization = 'reduced', type_gram_schmidt='classic'):
    A.astype('float')
    if type_factorization == 'reduced':
        Q = np.zeros(A.shape)
        R = np.zeros((A.shape[1],A.shape[1]))
    elif type_factorization == 'full':
        Q = np.zeros((A.shape[0],A.shape[0]))
        R = np.zeros(A.shape)
    for j in np.arange(A.shape[1]):
        y = A[:,j]
        for i in np.arange(j):
            if type_gram_schmidt == 'classic':
                R[i,j] = np.dot(Q[:,i],A[:,j])
            elif type_gram_schmidt == 'modified':
                R[i,j] = np.dot(Q[:,i],y)
            y=y-R[i,j]*Q[:,i]
        R[j,j] = np.linalg.norm(y)
        Q[:,j] = y/np.linalg.norm(R[j,j])
    # The following lines must be completed by you!
    #if type_factorization == 'full':
        # (1) We need to add 0's to the R matrix so it is of the same shape as the matrix A,
        # fortunately this was already done!
        # (2) We need to add orthogonal vectors to Q so it is square,
        # how do we do this?
    return Q,R
```

```python
A = np.array([[1,-4],[2,3],[2,2]])
Qa, Ra = QR(A, type_factorization ='reduced', type_gram_schmidt='classic')
print(np.dot(Qa,Ra))
print(Qa)
print(Ra)
```

This method let us resolve a system of equations. However, exists a **Full QR Factorization**, creating the next system:

\begin{equation}
                 A_{m\times n} = Q_{m\times m}R_{m\times n}\\
\end{equation}

Q is a square orthogonal matrix, adding $m-n$ columns and R grows adding $m-n$ zero rows.

#### Theorem
A square matrix $Q$ is orthogonal if $Q^*\, = Q^{-1}$

## Examples

### Normal vs Modified Gram-Schmidt

```python
d = 1e-10
A = np.array([[1,1,1],[d,0,0],[0,d,0],[0,0,d]])

Q1,R1 = QR(A, type_gram_schmidt = 'classic')
Q2,R2 = QR(A, type_gram_schmidt = 'modified')

# Are truly orthogonal the Q's?
print(Q1)
print(Q2)
# Do we recover A?
print(np.dot(Q1,R1))
print(np.dot(Q2,R2))
```

## Inconsistent Systems

There is cases where the number of equations is greater than variables. Many times, those systems don't have an exact solution (inconsistent system). Then, in this case we needs an approximation closest to the data. Based in orthogonality, the shortest distance from a point to plane. The orthogonal distance represents the error which would be minimum.

\begin{equation}
                 b - A\,x = \vec{r}\\
                 b - A\,x \perp \{A\,x | x \in \mathcal{R}\}
\end{equation}

The idea is that $\vec{r}$ would be closest to zero. We need to apply orthogonality to find the vector that satisfied this condition.

\begin{equation}
                 (Ax)^*\,(b-A─\overline{x})=0 \hspace{1cm} \text{for all } x \in \mathcal{R^n}\\
                 x^*\, A^*\,(b-A─\overline{x})=0 \hspace{1cm} \text{for all } x \in \mathcal{R^n}\\
                 A^*\,(b-A─\overline{x})=0 \\
                 A^*\,A\overline{x}= A^*\,b \\
\end{equation}

This last equation gives us a new square $n\times n$ matrix, which let us resolve the equation system.
This linear system of equations is known as the **Normal Equations**.

```python
def least_squares(A,b):
    Q,R = QR(A,type_gram_schmidt='modified')
    return spla.solve_triangular(R,np.dot(Q.T,b))

def solve_model(M):
    A=M['A']
    b=M['b']
    M['x_bar']=least_squares(A,b)
    return M

def create_model(data, type_model='linear'):
    if type_model == 'linear': # f(x)=a0+a1*x
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = data[:,1]
    if type_model == 'parabollic': # f(x)=a0+a1*x+a_2*x^2
        A = np.ones((data.shape[0],3))
        A[:,1] = data[:,0]
        A[:,2] = data[:,0]**2
        b = data[:,1]
    if type_model == 'exponential': #f(x)=a0 \exp(a1*x) = \exp(\log(a0)+a1*x) -> log(f(x))=log(a0)+a1*x = A0+a1+x (it is linear now!)
        A = np.ones((data.shape[0],2))
        A[:,1] = data[:,0]
        b = np.log(data[:,1])
    M = {'A':A,
         'b':b,
         'type_model':type_model}
    M=solve_model(M)
    return M

def evaluate_model(M,x):
    x_bar=M['x_bar']
    if M['type_model'] == 'linear':
        return x_bar[0] + x_bar[1]*x
    if M['type_model'] == 'parabollic':
        return x_bar[0] + x_bar[1]*x + x_bar[2]*x**2
    if M['type_model'] == 'exponential':
        return np.exp(x_bar[0]+x_bar[1]*x)
```

# Adjusting some models

```python
def generate_data(type_of_data='linear'):
    n=40
    np.random.seed(0)
    x = np.linspace(0,10,n)
    y = np.random.rand(n)
    x = np.concatenate((x,x,y),axis=0)
    n = 3*n
    if type_of_data=='linear':
        y = x+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='parabollic':
        y = 4*x**2+0.1*x*np.random.normal(0,1,n)+1.5
    elif type_of_data=='exponential':
        y = np.exp(x+0.1*np.random.normal(0,1,n)+1.5)
    elif type_of_data=='sinusoidal':
        y = np.sin(2*np.pi*x/10)+0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='random':
        y = 0.1*np.random.normal(0,1,n)+1.5
    elif type_of_data=='boston house-prices':
        x,y=datasets.load_boston(return_X_y=True)
        x=x[:,5]
    elif type_of_data=='diabetes':
        x,y=datasets.load_diabetes(return_X_y=True)
        x=x[:,2]
    data = np.stack((x, y)).T
    return data
```

```python
def looking_at_data(type_of_data='diabetes'):
    data=generate_data(type_of_data)
    Ml = create_model(data, type_model='linear')
    Mp = create_model(data, type_model='parabollic')
    Me = create_model(data, type_model='exponential')
    xx=np.linspace(np.min(data[:,0])-0.1,np.max(data[:,0])+0.1,1000)
    yyl=evaluate_model(Ml,xx)
    yyp=evaluate_model(Mp,xx)
    yye=evaluate_model(Me,xx)

    error_l=data[:,1]-evaluate_model(Ml,data[:,0])
    error_p=data[:,1]-evaluate_model(Mp,data[:,0])
    error_e=data[:,1]-evaluate_model(Me,data[:,0])

    plt.figure(figsize=(2*M,M))

    plt.subplot(1, 2, 1)
    plt.plot(xx,yyl,'k-',linewidth=5,label='linear model')
    plt.plot(xx,yyp,'y--',linewidth=20,label='parabollic model')
    plt.plot(xx,yye,'g-',linewidth=5,label='exponential model')
    plt.plot(data[:,0],data[:,1],'.b',markersize=20,label='original data',alpha=0.3)
    plt.grid(True)
    plt.xlabel(r'$x$')
    plt.ylabel(r'$y$')
    plt.legend(loc='best')

    plt.subplot(1, 2, 2)
    plt.title('What does this histogram tell us?')
    three_errors=np.vstack((error_l, error_p, error_e)).T
    plt.hist(three_errors, bins=20,
             label=['linear','parabollic','exponential'],
             color=['k','y','g'], alpha=0.5)
    plt.legend(loc='best')
    plt.grid(True)

    plt.show()
widgets.interact(looking_at_data,type_of_data=['linear','parabollic','exponential','sinusoidal','random','boston house-prices','diabetes'])
```

## References

### Numpy Least Squares
http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.lstsq.html

### Numpy QR Factorization
http://docs.scipy.org/doc/numpy/reference/generated/numpy.linalg.qr.html

# Acknowledgements
* _Material created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistans: Laura Bermeo, Alvaro Salinas, Axel Símonsen and Martín Villanueva. DI UTFSM. April 2016._
* _Material updated by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) DI UTFSM. June 2017.

```python

```


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

# Projection Methods

The concept of
projection is key to developing an intuition about conditional
probability.  We
already have a natural intuition of projection from looking at
the shadows of
objects on a sunny day. As we will see, this simple idea
consolidates many
abstract ideas in optimization and mathematics.  Consider
[Figure](#fig:probability_001) where we want to find a point along the blue
line
(namely, $\mathbf{x}$) that is closest to the black square (namely,
$\mathbf{y}$). In other words, we want to inflate the gray circle until it just
touches the black line. Recall that the circle boundary is the set of points for
which

$$
\sqrt{(\mathbf{y}-\mathbf{x})^T(\mathbf{y}-\mathbf{x})}
=\|\mathbf{y}-\mathbf{x} \| = \epsilon
$$

 for some value of $\epsilon$. So we want a point $\mathbf{x}$ along
the line
that satisfies this for the smallest $\epsilon$.  Then, that point
will be the
closest point on the black line to the black square.
It may be obvious from the
diagram, but the closest point on the line
occurs where the line segment from
the black square to the black line is
perpedicular to the line. At this point,
the gray circle just touches the black
line. This is illustrated below in
[Figure](#fig:probability_002).

Given the point $\mathbf{y}$ (black
square) we want to find the $\mathbf{x}$ along the line that is closest to it.
The gray circle is the locus of points within a fixed distance from
$\mathbf{y}$.

**Programming Tip.**

[Figure](#fig:probability_001) uses
the `matplotlib.patches` module.  This
module contains primitive shapes like
circles, ellipses, and rectangles that
can be assembled into complex graphics.
After importing a particular shape, you
can apply that shape to an existing axis
using the `add_patch` method.  The
patches themselves can by styled using the
usual formatting keywords like
`color` and `alpha`.

The closest point on the line occurs when
the line is tangent to the circle. When this happens, the black line and the
line (minimum distance) are perpedicular.

 Now that we
can see what's going on, we can construct the the solution
analytically. We can
represent an arbitrary point along the black line as:

$$
\mathbf{x}=\alpha\mathbf{v}
$$

 where $\alpha\in\mathbb{R}$ slides the point up and down the line with

$$
\mathbf{v} = \left[ 1,1 \right]^T
$$

 Formally, $\mathbf{v}$ is the *subspace* onto which we want to
*project*
$\mathbf{y}$. At the closest point, the vector between
$\mathbf{y}$ and
$\mathbf{x}$ (the *error* vector above) is
perpedicular to the line. This means
that

$$
(\mathbf{y}-\mathbf{x} )^T \mathbf{v} = 0
$$

 and by substituting and working out the terms, we obtain

$$
\alpha = \frac{\mathbf{y}^T\mathbf{v}}{ \|\mathbf{v} \|^2}
$$

 The *error* is the distance between $\alpha\mathbf{v}$ and $
\mathbf{y}$.  This
is a right triangle, and we can use the Pythagorean
theorem to compute the
squared length of this error as

$$
\epsilon^2 = \|( \mathbf{y}-\mathbf{x} )\|^2 = \|\mathbf{y}\|^2 - \alpha^2
\|\mathbf{v}\|^2 = \|\mathbf{y}\|^2 -
\frac{\|\mathbf{y}^T\mathbf{v}\|^2}{\|\mathbf{v}\|^2}
$$

 where $ \|\mathbf{v}\|^2 = \mathbf{v}^T \mathbf{v} $. Note that since
$\epsilon^2 \ge 0 $, this also shows that

$$
\| \mathbf{y}^T\mathbf{v}\| \le \|\mathbf{y}\|  \|\mathbf{v}\|
$$

 which is the famous and useful Cauchy-Schwarz inequality which we
will exploit
later. Finally, we can assemble all of this into the *projection*
operator

$$
\mathbf{P}_v = \frac{1}{\|\mathbf{v}\|^2 } \mathbf{v v}^T
$$

 With this operator, we can take any $\mathbf{y}$ and find the closest
point on
$\mathbf{v}$ by doing

$$
\mathbf{P}_v \mathbf{y} = \mathbf{v} \left( \frac{  \mathbf{v}^T \mathbf{y}
}{\|\mathbf{v}\|^2} \right)
$$

 where we recognize the term in parenthesis as the $\alpha$ we
computed earlier.
It's called an *operator* because it takes a vector
($\mathbf{y}$) and produces
another vector ($\alpha\mathbf{v}$). Thus,
projection unifies geometry and
optimization.

## Weighted distance

We can easily extend this projection
operator to cases where the measure of
distance between $\mathbf{y}$ and the
subspace $\mathbf{v}$ is weighted. We can
accommodate these weighted distances
by re-writing the projection operator as

$$
\begin{equation}
\mathbf{P}_v=\mathbf{v}\frac{\mathbf{v}^T\mathbf{Q}^T}{\mathbf{v}^T\mathbf{Q v}}
\end{equation}
\label{eq:weightedProj} \tag{1}
$$

 where $\mathbf{Q}$ is positive definite matrix.  In the previous
case, we
started with a point $\mathbf{y}$ and inflated a circle centered at
$\mathbf{y}$
until it just touched the line defined by $\mathbf{v}$ and this
point was
closest point on the line to $\mathbf{y}$. The same thing happens
in the general
case with a weighted distance except now we inflate an
ellipse, not a circle,
until the ellipse touches the line.

In the weighted
case, the closest point on the line is tangent to the ellipse and is still
perpedicular in the sense of the weighted distance.

Note that the
error vector ($\mathbf{y}-\alpha\mathbf{v}$) in [Figure](#fig:probability_003)
is still perpendicular to the line (subspace
$\mathbf{v}$), but in the space of
the weighted distance.  The difference
between the first projection (with the
uniform circular distance) and the
general case (with the elliptical weighted
distance) is the inner product
between the two cases.  For example, in the first
case we have $\mathbf{y}^T
\mathbf{v}$ and in the weighted case we have
$\mathbf{y}^T \mathbf{Q}^T
\mathbf{v}$. To move from the uniform circular case
to the weighted ellipsoidal
case, all we had to do was change all of the vector
inner products.  Before we
finish, we need a formal property  of projections:

$$
\mathbf{P}_v \mathbf{P}_v = \mathbf{P}_v
$$

 known as the *idempotent* property which basically says that once we
have
projected onto a subspace, subsequent projections leave us in the
same subspace.
You can verify this by computing Equation [1](#eq:weightedProj).

Thus,
projection ties a minimization problem (closest point to a line) to an
algebraic
concept (inner product). It turns out that these same geometric ideas
from
linear algebra [[strang2006linear]](#strang2006linear) can be translated to the
conditional
expectation. How this works is the subject of our next  section.


```julia
using CSV
using PyPlot
using Statistics

ENV["COLUMNS"] = 1000 # change number of columns dispalyed
```

    1000

# CHE 599: Introduction to Data Science for Engineers

Fall 2019, MWF 10am-10:50am in Owen Hall 106

Instructor: Cory Simon [[simonensemble.github.io](https://simonensemble.github.io/)]

> "We are drowning in information but starved for knowledge." -John Naisbitt

> "The ability to take data — to be able to understand it, to process it, to extract value from it, to visualize it, to communicate it — that’s going to be a hugely important skill in the next decades." - Hal Varian, chief economist at Google and UC Berkeley professor of information sciences, business, and economics [source](https://datascience.berkeley.edu/about/what-is-data-science/)

... I'm an engineer, and I don't want to become a data scientist!

* Data science will play an increasingly important role in the chemical engineering discipline.

See ["Data Science for Chemical Engineers"](https://www.aiche.org/resources/publications/cep/2017/february/data-science-chemical-engineers) in AIChE's magazine.

* In our research, most of us will collect data from which we aim to extract useful insights, infer, and predict.

# What is data science?

"Data Science is about drawing useful conclusions from large and diverse data sets through exploration, prediction, and inference." [source](https://www.inferentialthinking.com/chapters/01/what-is-data-science.html)

## exploration

identifying patterns in data, summarizing data, and communicating data (descritive statistics, data visualization)

```julia
df = CSV.read("wine.csv")
first(df, 5)
```

	wine_class	alcohol	malic_acid	ash	alcalinity_of_ash	magnesium	total_phenols	flavanoids	nonflavanoid_phenols	proanthocyanins	color_intensity	hue	OD280_to_OD315_of_diluted_wines	proline

	Int64	Float64	Float64	Float64	Float64	Int64	Float64	Float64	Float64	Float64	Float64	Float64	Float64	Int64

5 rows × 14 columns

1	1	14.23	1.71	2.43	15.6	127	2.8	3.06	0.28	2.29	5.64	1.04	3.92	1065

2	1	13.2	1.78	2.14	11.2	100	2.65	2.76	0.26	1.28	4.38	1.05	3.4	1050

3	1	13.16	2.36	2.67	18.6	101	2.8	3.24	0.3	2.81	5.68	1.03	3.17	1185

4	1	14.37	1.95	2.5	16.8	113	3.85	3.49	0.24	2.18	7.8	0.86	3.45	1480

5	1	13.24	2.59	2.87	21.0	118	2.8	2.69	0.39	1.82	4.32	1.04	2.93	735

we will learn to make publication-quality, effective, and communicative data visualizations using the free and open-source [matplotlib](https://matplotlib.org/).

## unsupervised learning
algorithms to find patterns in data or group data *without* labels

e.g. principal component analysis, clustering

latent space of wines (color = type of wine)

## inference

make conclusions about a population from a random sample; quantify uncertainty about these conclusions.

* What is the average salary in San Francisco? (learn bootstrap, confidence intervals, maximum likelihood)

```julia
df = CSV.read("sf_salaries.csv")[!, [:total_compensation]]

println("# samples: ", size(df, 1))

first(df, 5)
```

    # samples: 500

	total_compensation

	Float64

5 rows × 1 columns

1	2.15741e5

2	1.83135e5

3	1.80116e5

4	27262.1

5	17792.3

```julia
figure()
hist(df[!, :total_compensation])
axvline(mean(df[!, :total_compensation]), color="y")
xlabel("salary")
ylabel("# employees")
```

* Is maternal smoking associated with lower birth weight? (learn permutation tests, p-value)

```julia
df = CSV.read("baby.csv")[!, [:birth_weight, :maternal_smoker]]
println("# samples: ", size(df, 1))
first(df, 5) # "class" attribute = label (kidney disease = 1, no kidney disease = 0)
```

    # samples: 1174

	birth_weight	maternal_smoker

	Int64	Bool

5 rows × 2 columns

1	120	0

2	113	0

3	128	1

4	108	1

5	136	0

```julia
figure()
hist(df[df[!, :maternal_smoker], :birth_weight], alpha=0.5, label="maternal smoker")
hist(df[.! df[!, :maternal_smoker], :birth_weight], alpha=0.5, label="not maternal smoker")
xlabel("birth weight")
ylabel("# babies")
legend()
```

## prediction
use data to make predictions (supervised learning: regression, classification)

* Based on blood test, does this patient have kidney disease? [classification]

```julia
df = CSV.read("kidney_disease.csv")
first(df, 5)
```

	Age	Blood Pressure	Specific Gravity	Albumin	Sugar	Red Blood Cells	Pus Cell	Pus Cell clumps	Bacteria	Blood Glucose	Blood Urea	Serum Creatinine	Sodium	Potassium	Hemoglobin	Packed Cell Volume	White Blood Cell Count	Red Blood Cell Count	Hypertension	Diabetes Mellitus	Coronary Artery Disease	Appetite	Pedal Edema	Anemia	Class

	Int64	Int64	Float64	Int64	Int64	String	String	String	String	Int64	Int64	Float64	Int64	Float64	Float64	Int64	Int64	Float64	String	String	String	String	String	String	Int64

5 rows × 25 columns

1	48	70	1.005	4	0	normal	abnormal	present	notpresent	117	56	3.8	111	2.5	11.2	32	6700	3.9	yes	no	no	poor	yes	yes	1

2	53	90	1.02	2	0	abnormal	abnormal	present	notpresent	70	107	7.2	114	3.7	9.5	29	12100	3.7	yes	yes	no	poor	no	yes	1

3	63	70	1.01	3	0	abnormal	abnormal	present	notpresent	380	60	2.7	131	4.2	10.8	32	4500	3.8	yes	yes	no	poor	yes	no	1

4	68	80	1.01	3	2	normal	abnormal	present	present	157	90	4.1	130	6.4	5.6	16	11000	2.6	yes	yes	yes	poor	yes	no	1

5	61	80	1.015	2	0	abnormal	abnormal	notpresent	notpresent	173	148	3.9	135	5.2	7.7	24	9200	3.2	yes	yes	yes	poor	yes	yes	1

*observation*: data shows that e.g. blood glucose and white blood cell count are biomarkers for kidney disease

```julia
figure()
scatter(df[!, Symbol("White Blood Cell Count")], df[!, Symbol("Blood Glucose")], c=df[!, Symbol("Class")])
xlabel("white blood cell count")
ylabel("blood glucose")
```

*idea*: train model on data we have, then use the model to predict the probability that a *new* patient has kidney disease on the basis of measurements of blood glucose, white blood cell count, etc.

* Given a nanoporous material and associated descriptors (density, pore size, void fraction), what is its adsorptive selectivity? [regression]

```julia
df = CSV.read("materials_data.csv")
first(df, 10)
```

	crystal_density	pore_size	void_fraction	selectivity

	Float64	Float64	Float64	Float64

10 rows × 4 columns

1	1659.26	6.90917	0.04596	7.16895

2	1783.07	4.96374	0.00746	6.43933

3	1071.12	10.0352	0.09914	9.44159

4	1875.9	4.96285	0.00996	7.53202

5	1002.19	7.02873	0.06094	11.6728

6	886.401	13.2699	0.23626	7.02738

7	1103.91	6.90622	0.05276	12.6155

8	1364.57	6.96292	0.12	3.09369

9	885.986	13.2751	0.23594	6.9777

10	577.278	8.38842	0.24708	4.09324

*observation*: data shows that pore size and selectivity are correlated

```julia
figure()
scatter(df[!, :pore_size], df[!, :selectivity])
xlabel("pore size")
ylabel("selectivity")
```

*idea*: train model on data we have, then use the model to predict the adsorptive selectivity of a new material (whose adsorption properties are unknown) on the basis of its crystal density, pore size, and void fraction.

# my journey into machine learning

* 1/2 PhD Mathematics at the University of British Columbia

* foundation in machine learning from course at UC Berkeley by [Jitendra Malik](https://en.wikipedia.org/wiki/Jitendra_Malik) and [Alexei Efros](https://en.wikipedia.org/wiki/Alexei_A._Efros) during my PhD

* data science intern at [Stitch Fix](https://algorithms-tour.stitchfix.com) during my PhD

* postdoc in computational genomics at Altius Institute for Biomedical Sciences
    * e.g., convolutional neural networks can predict transcription factor binding to DNA sequences; see DeepBind [DOI: 10.1038/nbt.3300](https://www.nature.com/articles/nbt.3300/)

* current research focus: applying machine learning to materials discovery
    * using a random forest of [decision trees](http://www.r2d3.us/visual-intro-to-machine-learning-part-1/) to predict Xe/Kr selectivity of nanoporous materials [DOI: 10.1021/acs.chemmater.5b01475](https://pubs.acs.org/doi/abs/10.1021/acs.chemmater.5b01475)
    * apply facial recognition algorithm to group together porous molecules with similarly shaped cavities [DOI: 10.1021/acscentsci.8b00638](https://pubs.acs.org/doi/10.1021/acscentsci.8b00638)
    * NSF award for machine-learning latent representations of porous materials [here](https://www.nsf.gov/awardsearch/showAward?AWD_ID=1920945)

# unsupervised learning applied to materials discovery

> ❝Eigencages: Learning a latent space of porous cage molecules❞ A. Sturluson, M. T. Huynh, A. H. York, C. Simon. *ACS Central Science*. (2018)

### porous organic cage molecules

a set of 74 experimentally synthesized porous organic cage molecules. each harbors a nano-sized cavity that selectively adsorbs gas molecules.

the shape of the cavity is a primary determinant of the adsorptive selectivity.

goal: automatically group together cages that have similar cavity shapes via learning "latent vector representations" of the cages

### viewing materials as high-dimensional vectors

* take 3D computational scans of the porosity of each cage $\rightarrow$ a set of 3D images

* pixel $i$ takes on value $0$ or $1$ for void space or not (i.e. overlapping an atom).

* flatten the image, and we have a high-dimenionsional vector

    $g \times g \times g$ image $\implies$ cage $i$ is a vector $\mathbf{c}_i \in \mathbb{R}^{g^3}$

### a matrix that encapsulates all of the data

* stack the flattened cage images into a matrix $\mathbf{A}$ (with the average cage image $\bar{\mathbf{c}}$ subtracted)

\begin{equation}
    \mathbf{A} = \begin{bmatrix}
        \text{---} \hspace{-0.2cm} & \mathbf{c}_1^\intercal - \bar{\mathbf{c}}^\intercal & \hspace{-0.2cm} \text{---} \\
        & \vdots & \\
        \text{---} \hspace{-0.2cm} & \mathbf{c}_{74}^\intercal - \bar{\mathbf{c}}^\intercal  & \hspace{-0.2cm} \text{---} \\
    \end{bmatrix}
\end{equation}

* this matrix is full rank. i.e., we need 74 basis vectors (= cage images) to form all cage images

### a data compression problem

* The goal is to find a low-rank approximation of the matrix $\mathbf{A}$ (data compression).
* The idea is to express each cage as a combination of much fewer than 74 "basis" cage images.
* In the process of learning a low-rank approximation of $\mathbf{A}$, we learn the salient "ingredients" of cage images.

* the singular value decomposition (SVD) provides a convenient way to find the optimal low-rank approximation: zero out the smallest singular values $\sigma_i$.
* the columns of orthogonal matrix $\mathbf{V}$ are the orthogonal "ingredients" of cage images, ranked by importance.

### expressing a cage as a combination of ingredients

idea: each cage image is expressed as a linear combination of eigencages (ingredients) $\mathbf{v}_i$. If we impose that only two ingredients can be used:
\begin{equation}
\mathbf{c}_i = \bar{\mathbf{c}} + \alpha_{i} \mathbf{v}_1 + \beta_{i}  \mathbf{v}_2.
\end{equation}

* the $\mathbf{v}_1$ and $\mathbf{v}_2$ are the two most important ingredients learned from SVD
* the 2D latent vector representation of cage $i$ is $(\alpha_i, \beta_i)$

### the ingredients of cage images (eigencages)

### the latent representation of the cages

think of this as a visualization of the resulting 2D representations of the cages.

cages with similar cavity shapes congregate nearby in this "latent" cage space.

this was "learned" by the dimensionality reduction algorithm without any labels on the structures, e.g. "these cages are donut-shaped"


# Low-Level Programmierung von maschinellem Lernen: Implementation einer Support Vector Machine ohne Hilfe von externen Bibliotheken und anschließendes Training mit Hilfe eines künstlichen Datensatzes

von Thomas Battenfeld

Quelle:  John C. Platt - A Fast Algorithm for Training Support Vector Machines

Eine Support Vector Machine (SVM) teilt einen n-dimensionalen Raum mit Hilfe einer n-1-dimensionalen Hyperebene. Diese Entscheidungsfunktion kann folgendermaßen beschrieben werden:

$$u= w \cdot x - b,$$

wobei\
$\ \ \ \ \ \ -$ ${w}$ den Normalenvektor zu der Hyperebene,\
$\ \ \ \ \ \ -$ ${x}$ den Eingabevektor und\
$\ \ \ \ \ \ -$ $b$ einen skalaren Schwellwert darstellt.

Eine SVM soll eine Hyperebene finden, die zwei Klassen trennt, d.h. ${w} \cdot {x} - b = 0$.

Für Instanzen der Klasse ''$-1$'' soll $w \cdot x - b \leq -1$ gelten, während für Instanzen der Klasse ''$1$'' $w \cdot x - b \geq 1$ gelten soll.
Dazu soll der Abstand $a$ der Hyperebene zu den beiden Klassen möglichst groß sein. Dieser Abstand kann als

$$a = \frac{1}{||{w}||_2}$$

formuliert werden. Die Maximierung dieses Abstands kann durch das folgende Optimierungsproblem erfolgen:

\begin{equation}
    \begin{array}{lll}
        \min &\frac{1}{2} ||w||^2 & \\
        \mbox{s.t.} & y_i(w \cdot x_i - b) \geq 1 &\forall i,
    \end{array}
\end{equation}

wobei $x_i$ den $i$-ten Trainingsdatensatz und $y_i$ das Label des $i$-ten Trainingsdatensatzes darstellt.
Da die Daten häufig nicht linear trennbar sind, wurden von Cortes & Vapnik 1995 Regularisierungsparameter sowie ein Strafterm eingeführt:

\begin{equation}
    \begin{array}{lll}
        \min &\frac{1}{2} ||w||^2 + C \sum^N_{i=1} \xi_i &\\
        \mbox{s.t.} & y_i(w \cdot x_i - b) \geq 1 - \xi_i &\forall i, \\
    \end{array}
\end{equation}

wobei $\xi_i$ Schlupfvariablen sind, die Datenpunkten innerhalb des Abstands $a$ zulassen. $C$ stellt den korrespondierenden Strafterm dar.
Dieses Optimierungsproblem kann in einer dualen Form und mit Hilfe von Lagrange-Multiplikatoren formuliert werden:

\begin{equation}
    \begin{array}{ll}
        \min& \frac{1}{2} \sum^N_{i=1} \sum^N_{j=1} y_i y_j K(x_i, x_j) \alpha_i \alpha_j -  \sum^N_{i=1} \alpha_i \\
        \mbox{s.t.} & 0 \leq \alpha_i \leq C \  \ \ \ \ \ \ \ \ \forall i, \\
        & \sum^N_{i=1} y_i \alpha_i = 0,
    \end{array}
\end{equation}

wobei \
$\ \ \ \ \ \ -$ $N$ die Anzahl der Trainingsdatensätze ist, \
$\ \ \ \ \ \ -$ $K$ eine Kernelfunktion ist und \
$\ \ \ \ \ \ -$ $\alpha$ Lagrange-Multiplikatoren darstellen.

Mit Hilfe der Lagrange-Multiplikatoren ist das Auffinden der lokalen Maxima und Minima einer Funktion (welche Nebenbedingungen unterliegt) möglich.
Des Weiteren unterliegt das Optimierungsproblem den folgenden Karush-Kuhn-Tucker Bedingungen:

\begin{equation}
    \begin{array}{lcl}
        \alpha_i = 0 & \Leftrightarrow & y_i u_i \geq 1, \\
        0 < \alpha_i < C & \Leftrightarrow & y_i u_i = 1, \\
        \alpha_i = C & \Leftrightarrow & y_i u_i \leq 1.
    \end{array}
\end{equation}

Karush-Kuhn-Tucker Bedingungen stellen die notwendigen Voraussetzungen dafür dar, dass eine Lösung in der nicht linearen Programmierung optimal sein kann.
Die korrespondierende Entscheidungsfunktion zu dem oben genannten dualen Problem wird folgendermaßen definiert:

$$u = \sum^N_{j=1}y_j \alpha_j K(x_j,x)-b.$$

# Support Vector Machines mit Sequential Minimal Optimization nach John C. Platt "A Fast Algorithm for Training Support Vector Machines" (1998)

Eine Möglichkeit dieses duale Problem zu lösen stellt die Sequential Minimal Optimization (SMO) dar. Diese Heuristik wurde im Jahr 1988 vom John C. Platt in dem Paper  "A Fast Algorithm for Training Support Vector Machines" vorgestellt und geht wie folgt vor:

$\ \ \ \ \ \ -$ Finde einen Lagrange-Multiplikator $\alpha _{1}$, der gegen die Karush-Kuhn-Tucker Bedingungen verstößt. \
$\ \ \ \ \ \ -$ Wähle einen zweiten Lagrange-Multiplikator $\alpha _{2}$ und optimiere für das Multiplikatoren-Paar $(\alpha _{1},\alpha _{2})$. \
$\ \ \ \ \ \ -$ Wiederhole die Schritte 1 und 2 bis das Optimierungsproblem konvergiert.

Der SMO Algorithmus ist im Folgenden nach dem Pseudo-Code aus dem Paper von John C. Platt in Python implementiert:

```python
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
import random
import math
sns.set()
%matplotlib inline
```

```python
class SVMwithSMO:
    # Quelle: https://www.semanticscholar.org/paper/Sequential-Minimal-Optimization%3A-A-Fast-Algorithm-Platt/53fcc056f79e04daf11eb798a7238e93699665aa
    def __init__(self, X, y, kernel, C=1000, b = 0, tol = 0.001, eps = 0.001):
        self.X = X                         # Features
        self.y = y                         # Class labels
        self.kernel = kernel               # Kernel
        self.C = C                         # Regularization
        self.b = b                         # Threshold
        self.tol = tol                     # Error tolerance
        self.eps = eps                     # Alpha tolerance
        self.m = len(self.X)               # Size of training data
        if self.kernel == linear_kernel:   # w vector
            self.w = 0
        else:
            self.w = 'Only SVM with linear kernels can return w vector'
        self.alphas = np.zeros(len(X))     # Lagrange multipliers
        self.errors = self.decision_function(self.y, self.alphas, self.kernel, self.X, self.X, self.b) - self.y  # Eq. below 16

    # Quelle: https://www.semanticscholar.org/paper/Sequential-Minimal-Optimization%3A-A-Fast-Algorithm-Platt/53fcc056f79e04daf11eb798a7238e93699665aa
    def decision_function(self, y, alphas, kernel, X_j, X, b):
        # Eq. 10
        return (y * alphas) @ kernel(X_j, X) - b

    # Quelle: https://www.semanticscholar.org/paper/Sequential-Minimal-Optimization%3A-A-Fast-Algorithm-Platt/53fcc056f79e04daf11eb798a7238e93699665aa
    def dual_function(self, y, kernel, X, alphas):
        # Eq. 11
        n = X.shape[0]
        return  0.5 * sum([y[i] * y[j] * kernel(X[i], X[j]) * alphas[i] * alphas[j] for i in range(n) for j in range(n)]) - np.sum(alphas)

    # Quelle: https://www.semanticscholar.org/paper/Sequential-Minimal-Optimization%3A-A-Fast-Algorithm-Platt/53fcc056f79e04daf11eb798a7238e93699665aa
    def takeStep(self, i1, i2):
        if i1 == i2:
            return 0

        alph1 = self.alphas[i1]
        y1 = self.y[i1]
        E1 = self.errors[i1]
        alph2 = self.alphas[i2]
        y2 = self.y[i2]
        E2 = self.errors[i2]
        s = y1 * y2

        # Compute L, H via equations (13) and (14)
        if (y1 != y2):
            L = max(0, alph2 - alph1)
            H = min(self.C, self.C + alph2 - alph1)
        elif (y1 == y2):
            L = max(0, alph1 + alph2 - self.C)
            H = min(self.C, alph1 + alph2)

        if (L == H):
            return 0

        # Compute kernel & 2nd derivative eta
        k11 = self.kernel(self.X[i1], self.X[i1])
        k12 = self.kernel(self.X[i1], self.X[i2])
        k22 = self.kernel(self.X[i2], self.X[i2])

        # Eq. 15
        eta = k11 + k22 - 2 * k12

        #Eq. 16
        if (eta > 0):
            a2 = alph2 + ((y2 * (E1 - E2)) / eta)
            #Eq. 17
            if (a2 > H):
                a2 = H
            elif ((a2 > L) & (a2 < H)):
                a2 = a2
            elif (a2 < L):
                a2 = L
        else:
            #Eq. 19
            temp = self.alphas
            temp[i2] = L
            Lobj = self.dual_function(self.y, self.kernel, self.X, temp)
            temp[i2] = H
            Hobj = self.dual_function(self.y, self.kernel, self.X, temp)
            if (Lobj < Hobj - self.eps):
                a2 = L
            elif (Lobj > Hobj + self.eps):
                a2 = H
            else:
                a2 = alph2

        if (np.abs(a2 - alph2) < self.eps * (a2 + alph2 + self.eps)):
            return 0

        # Eq. 18
        a1 = alph1 + s * (alph2 - a2)

        # Update threshold to reflect change in Lagrange multipliers
        # Eq. 20, 21, 12
        if 0 < a1 and a1 < self.C:
            self.b = E1 + y1 * (a1 - alph1) * k11 + y2 * (a2 - alph2) * k12 + self.b
        elif 0 < a2 and a2 < self.C:
            self.b = E2 + y1 * (a1 - alph1) * k12 + y2 * (a2 - alph2) * k22 + self.b
        else: # SMO chooses the threshold to be halfway in between b1 and b2.
            self.b = 0.5*((E1 + y1 * (a1 - alph1) * k11 + y2 * (a2 - alph2) * k12 + self.b) + (E2 + y1 * (a1 - alph1) * k12 + y2 * (a2 - alph2) * k22 + self.b))

        # Update weight vector to reflect change in a1 & a2, if SVM is linear
        if self.kernel == linear_kernel:
            # Eq. 22
            self.w = self.w + y1 * (a1 - alph1) * self.X[i1] + y2 * (a2 - alph2) * self.X[i2]

        # Update error cache using new Lagrange multipliers
        # Eq. below 16
        temp = self.alphas
        temp[i1] = a1
        temp[i2] = a2
        self.errors = self.decision_function(self.y, temp, self.kernel, self.X, self.X, self.b) - self.y

        # Store a1 in the alpha array
        self.alphas[i1] = a1
        # Store a2 in the alpha array
        self.alphas[i2] = a2

        return 1

    # Quelle: https://www.semanticscholar.org/paper/Sequential-Minimal-Optimization%3A-A-Fast-Algorithm-Platt/53fcc056f79e04daf11eb798a7238e93699665aa
    def examineExample(self, i2):
        y2 = self.y[i2]
        alph2 = self.alphas[i2]
        E2 = self.errors[i2] # E2 = SVM output on point[i2] – y2 (check in error cache)
        r2 = E2 * y2

        # Proceed if error is within specified tolerance (tol)
        if ((r2 < -self.tol and alph2 < self.C) or (r2 > self.tol and alph2 > 0)):
            if len(self.alphas[(self.alphas != 0) & (self.alphas != self.C)]) > 1:
                # i1 = result of second choice heuristic (section 2.2
                if self.errors[i2] > 0:
                    i1 = np.argmin(self.errors)
                elif self.errors[i2] <= 0:
                    i1 = np.argmax(self.errors)
                if self.takeStep(i1,i2):
                    return 1

            # loop over all non-zero and non-C alpha, starting at a random point
            for i1 in np.roll(np.where((self.alphas != 0) & (self.alphas != self.C))[0], np.random.choice(np.arange(self.m))):
                if self.takeStep(i1, i2):
                    return 1

            # loop over all possible i1, starting at a random point
            for i1 in np.roll(np.arange(self.m), np.random.choice(np.arange(self.m))):
                if self.takeStep(i1, i2):
                    return 1
        return 0

    # Quelle: https://www.semanticscholar.org/paper/Sequential-Minimal-Optimization%3A-A-Fast-Algorithm-Platt/53fcc056f79e04daf11eb798a7238e93699665aa
    def mainRoutine(self, verbose = False):
        if verbose:
            print('Training...')

        numChanged = 0
        examineAll = 1

        while(numChanged > 0) or (examineAll):
            numChanged = 0
            if examineAll:
                # loop I over all training examples
                for i in range(self.m):
                    numChanged += self.examineExample(i)
            else:
                # loop I over examples where alpha is not 0 & not C
                for i in np.where((self.alphas != 0) & (self.alphas != self.C))[0]:
                    numChanged += self.examineExample(i)
            if examineAll == 1:
                examineAll = 0
            elif numChanged == 0:
                examineAll = 1
        if verbose:
            print('Trained!')

    def fit(self):
        self.mainRoutine()

    def predict(self, X, toClass = True):
        if toClass:
            def toClass(x):
                if x < 0:
                    return -1
                elif x > 0:
                    return 1
            return  np.array([toClass(xi) for xi in self.decision_function(self.y, self.alphas, self.kernel, self.X, np.array(X), self.b)])
        else:
            return self.decision_function(self.y, self.alphas, self.kernel, self.X, np.array(X), self.b)

    # In Anlehnung an: https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html
    def plot(self, ax = None):
        if ax is None:
            fig, ax = plt.subplots()

        data = self.X

        x = np.linspace(data[:,0].min(), data[:,0].max(), 100)
        y = np.linspace(data[:,1].min(), data[:,1].max(), 100)
        Y, X = np.meshgrid(y, x)
        xy = np.vstack([X.ravel(), Y.ravel()]).T
        grid = self.decision_function(self.y, self.alphas, self.kernel, self.X, xy, self.b).reshape(X.shape)

        ax.contour(X, Y, grid, colors='k',levels=[-1, 0, 1], alpha=0.5, linestyles=['--', '-', '--'])
        ax.scatter(data[:, 0], data[:, 1], c=self.y, s=50, cmap='autumn')
        ax.set_xlabel('x[0]')
        ax.set_ylabel('x[1]')
        ax.set_title('SMO SVM')
        plt.show()
```

Wie eingangs erwähnt, wird bei SMO Kernels genutzt. Im Folgenden werden ein linearer und ein polynomialer Kernel implementiert:

```python
# Quelle: https://www.guru99.com/kernel-methods-machine-learning.html
# Quelle: https://www.di.ens.fr/~mallat/papiers/svmtutorial.pdf

def linear_kernel(x, y, b = 1):
    return (np.dot(x, y.T)) + b

def polynomial_kernel(x, y, p = 3):
	return (np.dot(x, y.T)) ** p
```

# Support Vector Machines mit Stochastic Gradient Descent

Quelle: Vivek Srikumar - Support	Vector	Machines Training with Stochastic Gradient Descent.

Als Alternativ zur SMO kann das primale Problem mit Hilfe von Stochastic Gradient Descent (SGD) bearbeitet werden. Dabei werden die Schlupfvariablen $\xi_i$ der Zielfunktion durch eine Verlustfunktion ersetzt, beispielsweise der Hinge-Funktion:

\begin{equation}
    \begin{array}{lll}
        \min &J = \frac{1}{2} ||w||^2 + C \sum_{i=1}^N \max (0, y_i w^Tx_i). &\\
    \end{array}
\end{equation}

Da diese Funktion konvex ist und keine Nebenbedingungen gelten, kann ein Gradientenverfahren angewandt werden. Der  stochastische Ansatz des SGD geht wie folgt vor:

1. Initialisiere  $w = 0$.
2. Für die Epoche $e = 1 ... E$ tue: \
    2.1 Wähle einen zufälligen Datensatz $(x, y)$ aus dem Trainingsset, wobei $x$ die Eigenschaften darstellt und $y$ die Zielvariable (bzw. das Label) ist. \
    2.2. Berechne den Gradienten der Zielfunktion für diesen Datensatz. \
    2.3. Update von $w$: $w^E \leftarrow w^{e-1} - \eta \nabla J(w^{e-1})$, wobei die Learning Rate $\eta$ die Größe des Updates bestimmt.
3. Gib das finale $w$ zurück.

Da die Hinge Verlustfunktion keine differenzierbare Funktion ist, werden Sub-Gradienten eingesetzt:

$$\nabla J =
\begin{cases}
\begin{array}{ll}
        w & \mbox{wenn} \max (0, y_i w^T x_i) = 0, \\
        w - C y_i x_i & \mbox{ansonsten}.
    \end{array}
\end{cases}
$$

Es ist klar, dass $w$ sowohl stark von der gewählten Anzahl der Epochen als auch von der Learning Rate abhängt. Im Folgendem ist ein solch eher rudimentärer SGD in Anlehnung an Vivek Srikumar  in Python implementiert:

```python
class SVMwithSGD:
    def __init__(self, X, y, C=1000, epoch = 1000000, learning_rate = 0.0001):
        self.X = X                         # Features
        self.y = y                         # Class labels
        self.C = C                         # Regularization
        self.epoch = epoch                 # Epochen
        self.learning_rate = learning_rate # Epochen Learning Rate

    def decision_function(self, w, X):
        X = np.array(X)
        a = np.empty(X.shape[0])
        for x in range(X.shape[0]):
            a[x] = np.dot(w.T, X[x])
        return a

    # Quelle: https://svivek.com/teaching/machine-learning/lectures/slides/svm/svm-sgd.pdf
    def primal_function(self, w, C, y, X):
        return 0.5 * w.T * w + C * max(0, y_i * (np.dot(w.T, x_i)))

    # Quelle: https://svivek.com/teaching/machine-learning/lectures/slides/svm/svm-sgd.pdf
    def subgradient(self, w, x_i, y_i):
        if max(0, 1 - y_i * (np.dot(w.T, x_i))) == 0:
            return  w
        else:
            return w - self.C * y_i * x_i

    # Quelle: https://svivek.com/teaching/machine-learning/lectures/slides/svm/svm-sgd.pdf
    def sgd(self):
        self.w = np.array([0,0])
        for e in range(self.epoch):
            i = random.randrange(0, self.X.shape[0], 1)
            x_i = self.X[i]
            y_i = self.y[i]

            self.w = self.w - self.learning_rate * self.subgradient(self.w, x_i, y_i)

    def fit(self):
        self.sgd()

    def predict(self, X, toClass = True):
        if toClass:
            def toClass(x):
                if x < 0:
                    return -1
                elif x > 0:
                    return 1
            return  np.array([toClass(xi) for xi in self.decision_function(self.w, X)])
        else:
            return self.decision_function(self.w, X)

    # In Anlehnung an: https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html
    def plot(self, ax = None):
        if ax is None:
            fig, ax = plt.subplots()

        data = self.X

        x = np.linspace(data[:,0].min(), data[:,0].max(), 100)
        y = np.linspace(data[:,1].min(), data[:,1].max(), 100)
        Y, X = np.meshgrid(y, x)
        xy = np.vstack([X.ravel(), Y.ravel()]).T
        grid = self.decision_function(self.w, xy).reshape(X.shape)

        ax.contour(X, Y, grid, colors='k',levels=[-1, 0, 1], alpha=0.5, linestyles=['--', '-', '--'])
        ax.scatter(data[:, 0], data[:, 1], c=self.y, s=50, cmap='autumn')
        ax.set_xlabel('x[0]')
        ax.set_ylabel('x[1]')
        ax.set_title('SGD SVM')
        plt.show()
```

# Anwendung und Vergleich der Hypereben der Algorithmen

Es folgt der Vergleich von SMO und SGD SVMs. Für diesen Vergleich wird zusätzlich eine SVM von scikit-learn genutzt. Mit Hilfe der scikit-learn Bibliothek werden darüber hinaus künstliche Datensets erzeugt.

Zunächst werden die drei SVMs mit einem linearen Kernel trainiert, die Hyperebene geplottet und die Vorhersage für unbekannte Datenpunkte ausgegeben. Anschließend werden Datenpunkte innerhalb von des Abstandes hinzufügt, die SVMs erneut trainiert und die veränderte Hyperebene geplottet. Das Gleiche erfolgt für ein geringeres $C$.

Abschließend werden die selben Schritte für eine SMO und eine scikit-learn SVM mit polynomialen Kernel durchgeführt.

## Scikit-learn SVM

```python
from sklearn.datasets.samples_generator import make_blobs, make_moons
from sklearn.preprocessing import StandardScaler
from sklearn.svm import SVC
```

```python
# In Anlehnung an: https://jakevdp.github.io/PythonDataScienceHandbook/05.07-support-vector-machines.html
def plot_svc_decision_function(model, X, y_label, ax=None, plot_support=False):
    if ax is None:
        fig, ax = plt.subplots()

    data = X

    x = np.linspace(data[:,0].min(), data[:,0].max(), 100)
    y = np.linspace(data[:,1].min(), data[:,1].max(), 100)
    Y, X = np.meshgrid(y, x)
    xy = np.vstack([X.ravel(), Y.ravel()]).T

    grid = model.decision_function(xy).reshape(X.shape)

    # plot decision boundary and margins
    ax.contour(X, Y, grid, colors='k',levels=[-1, 0, 1], alpha=0.5, linestyles=['--', '-', '--'])

    ax.scatter(data[:, 0], data[:, 1], c=y_label, s=50, cmap='autumn')
    ax.set_xlabel('x[0]')
    ax.set_ylabel('x[1]')
    ax.set_title('SKL SVM')
    plt.show()
```

## Generierung von Daten
Um die jeweiligen SVMs anwenden zu können, werden Daten benötigt. Hier wird ein künstliches Datenset mit Hilfe von scikit-learn erzeugt:

```python
X, y = make_blobs(n_samples = 100,
                  centers = 2,
                  random_state = 1,
                  cluster_std = 1.5,
                  n_features = 2)

X = StandardScaler().fit_transform(X)
y[y == 0] = -1

plt.scatter(X[:, 0], X[:, 1], c=y, s=50, cmap='autumn')
plt.show()
```

## SVMs mit linearen Kernel

```python
C = 1000

smo_model_lin = SVMwithSMO(X, y, linear_kernel, C = C)
smo_model_lin.fit()

sgd_model = SVMwithSGD(X, y, C = C)
sgd_model.fit()

skl_model_lin = SVC(kernel = 'linear', C = C)
skl_model_lin.fit(X, y)

print("SMO, SGD und SKL SVM trainiert.")
```

    SMO, SGD und SKL SVM trainiert.

```python
smo_model_lin.plot()
sgd_model.plot()
plot_svc_decision_function(skl_model_lin, X, y)
```

Alle drei SVMs teilen die Daten durch die geforderte Hyperebene. Allerdings ist zu erkennen, dass die Hyperebenen sich unterscheiden (bspw. variiert die jeweilige Steigung ). Als Gütekriterium kann der Abstand $a$ der Hyperebene der jeweiligen SVMs genutzt werden.

### Parameter der SVMs

```python
print('SMO w: ' + str(smo_model_lin.w))
print('SMO m: ' + str(-smo_model_lin.w[0] / smo_model_lin.w[1]))
print('SMO b: ' + str(smo_model_lin.b))
print('SMO a: ' + str(1 / (math.sqrt(sum(i*i for i in smo_model_lin.w)))))
print("")

print('SGD w: ' + str(sgd_model.w))
print('SGD m: ' + str(-sgd_model.w[0] / sgd_model.w[1]))
print('SGD a: ' + str(1 / (math.sqrt(sum(i*i for i in sgd_model.w)))))
print("")

print('SKL w: ' + str(skl_model_lin.coef_[0]))
print('SKL m: ' + str (- skl_model_lin.coef_[0][0] / skl_model_lin.coef_[0][1]))
print('SKL b: ' + str(skl_model_lin.intercept_[0] / skl_model_lin.coef_[0][0]))
print('SKL a: ' + str(1 / (math.sqrt(sum(i*i for i in skl_model_lin.coef_[0])))))
```

    SMO w: [-0.95400309 -0.98478231]
    SMO m: -0.9687451511711329
    SMO b: -0.14352891116197117
    SMO a: 0.7293408444567968

    SGD w: [-1.23939045 -0.97179895]
    SGD m: -1.2753568496394265
    SGD a: 0.6349389515607933

    SKL w: [-0.95353701 -0.9843012 ]
    SKL m: -0.9687451511711322
    SKL b: -0.1504491039231443
    SKL a: 0.7296973386443764

Wie schon bei der grafischen Betrachtung der Hyperebene aufgefallen ist, gleichen sich diese relativ stark. Vor allem bei der SMO und SKL SVM ist das der Fall. Der Abstand $a$ der SMO und der SKL SVM sind fast gleich groß, aber deutlich größer als der Abstand der SGD SVM. Auch die Steigung $m$ der SMO und SKL SVM gleichen sich stark.

### Vorhersage von unbekannten Datenpunkten

```python
toPredict = [[-0.5,-0.5],[0.5,0.5]]

plt.scatter(np.append(X, toPredict, axis= 0)[:, 0], np.append(X, toPredict, axis= 0)[:, 1], c=np.append(y, [0,0], axis= 0), s=50, cmap='autumn')
plt.show()
```

```python
print("SMO Vorhersagen: " + str(smo_model_lin.predict(toPredict, toClass = False)))
print("SGD Vorhersagen: " + str(sgd_model.predict(toPredict, toClass = False)))
print("SKL Vorhersagen: " + str(skl_model_lin.decision_function(toPredict)))
```

    SMO Vorhersagen: [ 1.11292161 -0.82586379]
    SGD Vorhersagen: [ 1.1055947 -1.1055947]
    SKL Vorhersagen: [ 1.11237789 -0.82546031]

```python
print("SMO Vorhersagen: " + str(smo_model_lin.predict(toPredict)))
print("SGD Vorhersagen: " + str(sgd_model.predict(toPredict)))
print("SKL Vorhersagen: " + str(skl_model_lin.predict(toPredict)))
```

    SMO Vorhersagen: [ 1 -1]
    SGD Vorhersagen: [ 1 -1]
    SKL Vorhersagen: [ 1 -1]

Trotz der unterschiedlichen Parameter der SVMs klassifizieren diese korrekt.

### Hinzufügen von neuen Datenpunkten innerhalb von $w$

```python
X_outlier = np.append(X, [[-0.1, -0.1], [0, 0.5]], axis=0)
y_outlier = np.append(y, [1, -1], axis = 0)

plt.scatter(X_outlier[:, 0], X_outlier[:, 1], c=y_outlier, s=50, cmap='autumn')
plt.show()
```

```python
C = 1000

smo_model_lin_o = SVMwithSMO(X_outlier, y_outlier, linear_kernel, C = C)
smo_model_lin_o.fit()

sgd_model_o = SVMwithSGD(X_outlier, y_outlier, C = C)
sgd_model_o.fit()

skl_model_lin_o = SVC(kernel = 'linear', C = C)
skl_model_lin_o.fit(X_outlier, y_outlier)

print("SMO, SGD und SKL SVM trainiert.")
```

    SMO, SGD und SKL SVM trainiert.

```python
smo_model_lin_o.plot()
sgd_model_o.plot()
plot_svc_decision_function(skl_model_lin_o, X_outlier, y_outlier)
```

Aufgrund des hohen Strafterms $C$ verändert sich die Hyperebene der SMO und SKL SVM korrekt. Die SGD SVM allerdings benötigt höchstwahrscheinlich mehrere Iterationen oder ein besser gesteuerten Learning Rate, um eine ähnlich veränderte Hyperebene vorzeigen zu können.

### Datenset mit neuen Datenpunken und kleinerem C

```python
C = 1

smo_model_lin_o = SVMwithSMO(X_outlier, y_outlier, linear_kernel, C = C)
smo_model_lin_o.fit()

sgd_model_o = SVMwithSGD(X_outlier, y_outlier, C = C)
sgd_model_o.fit()

skl_model_lin_o = SVC(kernel = 'linear', C = C)
skl_model_lin_o.fit(X_outlier, y_outlier)

print("SMO, SGD und SKL SVM trainiert.")
```

    SMO, SGD und SKL SVM trainiert.

```python
smo_model_lin_o.plot()
sgd_model_o.plot()
plot_svc_decision_function(skl_model_lin_o, X_outlier, y_outlier)
```

Auch hier zeigt sich, dass die SMO und SKL SVM relativ ähnlich handeln. Eine Verringerung des Starfterms bewirkt, das die hinzugefügten Datenpunkte innerhalb von $a$ liegen. Bei der SGD SVM bewirkt die Verringerung des Starfterms ein deutlich größeres $a$.

## SVMs mit polynomialen Kernel

Nun werden SMO und SKL SVMs mit polynomialen Kerneln eingesetzt. Aufgrund der rudimenteren Implementation der SGD SVM ist es nicht möglich einen Kernel zu benutzen.

```python
c = 1000

smo_model_pol = SVMwithSMO(X, y, polynomial_kernel, C = C)
smo_model_pol.fit()

skl_model_pol = SVC(kernel = 'poly', C = C)
skl_model_pol.fit(X, y)

print("SMO und SKL SVM trainiert.")
```

    SMO und SKL SVM trainiert.

```python
smo_model_pol.plot()
plot_svc_decision_function(skl_model_pol, X, y)
```

Wieder verhalten sich die beiden SVMs relativ ähnlich. Allerdings ist aufgrund der polynomialen Kernel keine Auswertung der oben aufgeführten Parameter möglich. Visuell erscheinen beide Hyperebene korrekt.

### Vorhersage von unbekannten Datenpunkten

```python
plt.scatter(np.append(X, toPredict, axis= 0)[:, 0], np.append(X, toPredict, axis= 0)[:, 1], c=np.append(y, [0,0], axis= 0), s=50, cmap='autumn')
plt.show()
```

```python
print("SMO Vorhersagen: " + str(smo_model_pol.predict(toPredict, toClass = False)))
print("SKL Vorhersagen: " + str(skl_model_pol.decision_function(toPredict)))
```

    SMO Vorhersagen: [ 1.0974263  -0.17722815]
    SKL Vorhersagen: [ 0.46449965 -0.22365957]

```python
print("SMO Vorhersagen: " + str(smo_model_pol.predict(toPredict)))
print("SKL Vorhersagen: " + str(skl_model_pol.predict(toPredict)))
```

    SMO Vorhersagen: [ 1 -1]
    SKL Vorhersagen: [ 1 -1]

Beide SVMs sagen die Datenpunkte korrekt voraus.

### Hinzufügen von neuen Datenpunkten

```python
# X_outlier = np.append(X, [[-0.1, -0.1], [0.1, 0.1]], axis=0)
# y_outlier = np.append(y, [1, -1], axis = 0)

plt.scatter(X_outlier[:, 0], X_outlier[:, 1], c=y_outlier, s=50, cmap='autumn')
plt.show()
```

```python
C = 1000

smo_model_pol_o_c  = SVMwithSMO(X_outlier, y_outlier, polynomial_kernel, C = C)
smo_model_pol_o_c.fit()

skl_model_pol_o_c = SVC(kernel = 'poly', C = C, gamma = "auto")
skl_model_pol_o_c.fit(X_outlier, y_outlier)

print("SMO und SKL SVM trainiert.")
```

    SMO und SKL SVM trainiert.

```python
smo_model_pol_o_c.plot()
plot_svc_decision_function(skl_model_pol_o_c, X_outlier, y_outlier)
```

Es erfolgt eine korrekte Veränderung der Hyperebene aufgrund des hohen Strafterms.

### Datenset mit neuen Punken und kleinerem C

```python
C = 1

smo_model_pol_o = SVMwithSMO(X_outlier, y_outlier, polynomial_kernel, C = C)
smo_model_pol_o.fit()

skl_model_pol_o = SVC(kernel = 'poly', C = C, gamma = "auto")
skl_model_pol_o.fit(X_outlier, y_outlier)

print("SMO, SGD und SKL SVM trainiert.")
```

    SMO, SGD und SKL SVM trainiert.

```python
smo_model_pol_o.plot()
plot_svc_decision_function(skl_model_pol_o, X_outlier, y_outlier)
```

Aufgrund des geringen Strafterms können mehrere Datenpunkte innerhalb des Abstands der Hyperebene liegen.

## SMO SVM mit polynomial Kernel $p = 9$

```python
X, y = make_moons(n_samples=100, shuffle=True, noise=0.09, random_state=42)

X = StandardScaler().fit_transform(X)
y[y == 0] = -1

def polynomial_kernel_5(x, y, p = 9):
	return (np.dot(x, y.T)) ** p

smo_model_pol = SVMwithSMO(X, y, polynomial_kernel_5, C = 1000)
smo_model_pol.fit()
smo_model_pol.plot()
```

Diese Grafik illustriert wie mächtig SVMs sein können. Häufig sind Daten nicht linear trennbar und von einer "normalen, linearen" SVM nicht bearbeitbar. Das ändert sich durch den Einsatz von Kerneln. In der Grafik sieht man, dass mit Hilfe eines polynomial Kernel eine relativ starke und korrekte Trennung erfolgen kann. Diese ist bei weitem noch nicht perfekt, könnte allerdings mit anderen Kerneln verbessert werden, wie bspw. einem RBF Kernel.


```python
%matplotlib inline
```

********************************************************************************
2D Robot SLAM - Example
********************************************************************************
Goals of this script:

- apply the UKF for performing 2D SLAM (Simultaneous Localization And Mapping).

- discover a computationally alternative way for performing UKF inspired from
  :cite:`HuangA2013`. This alternative leads to computational speed improvement
  when only a part of the state is involved in a propagation or in update step.

- augment the state when a new landmark is observed in a UKF derivative-free
  way.

*We assume the reader is already familiar with the approach described in the
tutorial.*

This script considers the 2D robot SLAM problem where the robot is equipped with
wheel odometry and observes unknown landmark measurements. The robot state is
propagated through the odometry model and landmark observations are used in the
UKF measurement step. Landmarks are static and we assume no error coming from
data association. We reproduce the simulations that are described in
:cite:`huangObservability2010` , :cite:`HuangA2013`.

Import
==============================================================================

```python
import ukfm
from ukfm import SLAM2D as MODEL
import numpy as np
import matplotlib
ukfm.set_matplotlib_config()
```

Model and Simulation
==============================================================================
This script uses the :meth:`~ukfm.SLAM2D` model that requires sequence time
and odometry frequency.

```python
# sequence time (s)
T = 2500
# odometry frequency (Hz)
odo_freq = 1
# create the model
model = MODEL(T, odo_freq)
```

The trajectory of the robot consists in turning at constant speed.

```python
# true speed of robot (m/s)
v = 0.25
# true angular velocity (rad/s)
gyro = 1.5 / 180 * np.pi
# odometry noise standard deviation
odo_std = np.array([0.05*v/np.sqrt(2),     # speed (v/m)
                    0.05*v*np.sqrt(2)*2])  # angular speed (rad/s)
```

When simulating data, we generate a map. The map is defined as landmarks
constantly spaced on a circle with slightly higher radius than the radius of
the robot trajectory.

```python
# simulate true trajectory and noisy input
states, omegas, ldks = model.simu_f(odo_std, v, gyro)
# observation noise standard deviation (m)
obs_std = 0.1
# plot the map
model.plot_traj(states, ldks)
```

The state and the input contain the following variables:

.. highlight:: python
.. code-block:: python

    states[n].Rot      # orientation (matrix)
    states[n].p        # robot position
    states[n].p_l      # landmark positions
    omegas[n].gyro     # robot angular velocity
    omegas[n].v        # robot speed

Landmark positions are a 2D array where we get the k-th landmark as
``states[n].p_l[k]``. The number of landmarks in the state starts from zero
and increases when the robot observes a new landmark.

We compute noisy landmark measurements based on the true states.

```python
ys = model.simu_h(states, obs_std, ldks)
```

A measurement contains the observations of all visible landmarks as:

.. highlight:: python
.. code-block:: python

   y_n = ys[n]    # measurement at timestamp n
   y_n_k = y_n[k] # k-th observed landmark at instant n, where y_n_k[2] is the
                  # landmark index (-1 if the landmark is not observed)

Filter Design and Initialization
------------------------------------------------------------------------------
We embed the robot state in $SO(2) \times \mathbb{R}^2$ and each
landmark position in $\mathbb{R}^2$, such that:

- the retraction $\varphi(.,.)$ is the $SO(2)$ exponential for
  orientation, and the vector addition for positions.

- the inverse retraction $\varphi^{-1}_.(.)$ is the $SO(2)$
  logarithm for orientation and the vector subtraction for positions.

Remaining parameter setting is standard.

```python
# propagation noise covariance matrix
Q = np.diag(odo_std**2)
# measurement noise covariance matrix
R = obs_std**2*np.eye(2)
# sigma point parameters
alpha = np.array([1e-3, 1e-3, 1e-3, 1e-3, 1e-3])
# initial uncertainty matrix
P0 = np.zeros((3, 3))  # The state is perfectly initialized without
```

Regarding implementation, we use the Jacobian UKF (:meth:`~ukfm.JUKF`) that
spares time when only a part of the space is involved in a propagation or
update step.

**How it works ?** Consider the propagation of the covariance in an extended
Kalman filter as

\begin{align}\mathbf{P}_{n+1} = \mathbf{F} \mathbf{P}_{n} \mathbf{F}^T +
  \mathbf{G} \mathbf{Q} \mathbf{G}^T,\end{align}

where the robot state uncertainty is put in the first indices of the
covariance matrix $\mathbf{P}_{n}$. As landmarks are statics, the
Jacobian take the forms

\begin{align}\mathbf{F} = \begin{bmatrix}  \mathbf{F}^R & \mathbf{0} \\
  \mathbf{0}  & \mathbf{I}  \end{bmatrix}, \mathbf{G} = \begin{bmatrix}
  \mathbf{G}^R & \mathbf{0} \\ \mathbf{0}  & \mathbf{0}  \end{bmatrix}.\end{align}

The JUKF allows to compute $\mathbf{F}^R$ and $\mathbf{G}^R$ by
only using the required sigma points. Here it corresponds to the sigma points
of the robot state. This requires to set the reduced retraction ``red_phi``
and inverse retraction ``red_phi_inv`` that compute the required  subpart of
the full retraction $\varphi(.,.)$ and inverse retraction
$\varphi^{-1}_.(.)$, and to define corresponding indices ``red_idx`` in
$\mathbf{P}_n$.

Similarly for the observation of a landmark, e.g. the first landmark, the
observation matrix has the form

\begin{align}\mathbf{H} = \begin{bmatrix}  \mathbf{H}^1 & \mathbf{0} \end{bmatrix}.\end{align}

The JUKF computes $\mathbf{H}^1$ by only using the required sigma points
of the robot state and the observed landmark. This requires to set another
function ``up_phi`` using during update to compute a subpart of the retraction
$\varphi(.,.)$, as corresponding indices ``up_idx`` in
$\mathbf{P}_n$.

Finally, we require to define a new function $z(.,.)$ to augment the
state such that

\begin{align}\boldsymbol{\chi}_n^{\mathrm{aug}} = z(\boldsymbol{\chi}_{n}, \mathbf{y}_n),\end{align}

where $\boldsymbol{\chi}_n^{\mathrm{aug}}$ is the augmented state and
the $\mathbf{y}_n$ the measurement used to augment the state. Here this
measurement is a landmark observation. To make the augmentation efficient we
need to compute sigma points for only the state involved in $z(.,.)$,
``aug_phi`` is thus only a subpart of $\varphi(.,.)$ and ``aug_inv_phi``
is the inverse retraction of $\varphi(.,.)^{-1}$ corresponding to the
novel part of the state only.

```python
# reduced weights during propagation
red_idxs = np.array([0, 1, 2])  # indices corresponding to the robot state in P
# weights during update
aug_idxs = np.array([0, 1, 2])  # indices corresponding to the robot state in P

state0 = model.STATE(Rot=states[0].Rot, p=states[0].p, p_l=np.zeros((0, 2)))

ukf = ukfm.JUKF(state0=state0, P0=P0, f=model.f, h=model.h, Q=Q, phi=model.phi,
                alpha=alpha,
                red_phi=model.red_phi,
                red_phi_inv=model.red_phi_inv,
                red_idxs=red_idxs,
                up_phi=model.up_phi,
                up_idxs=np.arange(5),  # it will changes during the sequence
                aug_z=model.aug_z,
                aug_phi=model.aug_phi,
                aug_phi_inv=model.aug_phi_inv,
                aug_idxs=aug_idxs,
                aug_q=2)

# set variables for recording estimates along the full trajectory
ukf_states = [states[0]]
ukf_Ps = [P0]

# indices of already observed landmarks
ukf_lmk = np.array([])
```

Filtering
==============================================================================
The UKF proceeds as a standard Kalman filter with a for loop.

```python
for n in range(1, model.N):
    # propagation
    ukf.propagation(omegas[n-1], model.dt)
    y_n = ys[n]
    # observed landmarks
    idxs = np.where(y_n[:, 2] >= 0)[0]
    # update each landmark already in the filter
    p_ls = ukf.state.p_l
    for idx0 in idxs:
        idx = np.where(ukf_lmk == y_n[idx0, 2])[0]
        if idx.shape[0] == 0:
            continue
        # indices of the robot and observed landmark in P
        up_idxs = np.hstack([0, 1, 2, 3+2*idx, 4+2*idx])
        ukf.state.p_l = np.squeeze(p_ls[idx])
        # compute observability matrices and residual
        ukf.H_num(np.squeeze(y_n[idx0, :2]), up_idxs, R)
    ukf.state.p_l = p_ls
    # update only if some landmarks have been observed
    if ukf.H.shape[0] > 0:
        ukf.state_update()
    # augment the state with new landmark
    for idx0 in idxs:
        idx = np.where(ukf_lmk == y_n[idx0, 2])[0]
        if not idx.shape[0] == 0:
            continue
        # augment the landmark state
        ukf_lmk = np.hstack([ukf_lmk, int(y_n[idx0, 2])])
        # indices of the new landmark
        idx = ukf_lmk.shape[0]-1
        # new landmark position
        p_l = np.expand_dims(ukf.state.p + ukf.state.Rot.dot(y_n[idx0, :2]), 0)
        p_ls = np.vstack([ukf.state.p_l, p_l])
        ukf.state.p_l = p_l
        # get Jacobian and then covariance following [2]
        R_n = obs_std ** 2 * np.eye(2)
        ukf.aug(y_n[idx0, :2], aug_idxs, R)
        ukf.state.p_l = p_ls
    # save estimates
    ukf_states.append(ukf.state)
    ukf_Ps.append(ukf.P)
```

Results
------------------------------------------------------------------------------
We plot the trajectory, the position of the landmarks and the estimated
trajectory in the same plot, the attitude error, the position error, and their
confidence interval.

```python
model.plot_results(ukf_states, ukf_Ps, states, ldks)
```

We note the $3\sigma$ confidence interval decreases along time.

Conclusion
==============================================================================
This script shows how the UKF on parallelizable manifolds can be used for 2D
SLAM. By leveraging numerical Jacobian inference, one obtains a
computationally more efficient filter. The UKF works for this example, but
consistency issues happear at the end of the trajectory.

You can now:

- consider non-linear range and bearing measurement.

- benchmark the UKF with different retractions and compare it to the extended
  Kalman filter and the invariant extended Kalman filter of
  :cite:`barrauInvariant2017`.