$$ \LaTeX \text{ command declarations here.}
\newcommand{\N}{\mathcal{N}}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\norm}[1]{\|#1\|_2}
\newcommand{\d}{\mathop{}\!\mathrm{d}}
\newcommand{\qed}{\qquad \mathbf{Q.E.D.}}
\newcommand{\vx}{\mathbf{x}}
\newcommand{\vy}{\mathbf{y}}
\newcommand{\vt}{\mathbf{t}}
\newcommand{\vb}{\mathbf{b}}
\newcommand{\vw}{\mathbf{w}}
$$

```python
%matplotlib inline
from Lec07 import *
```

# EECS 445: Introduction to Machine Learning
## Lecture 06:  Logistic Regression
* Instructor:  **Jacob Abernethy** and **Jia Deng**
* Date:  September 26, 2016

*Lecture Exposition Credit:*  Benjamin Bray, Saket Dewangan

## Outline

- Concept of Classification
- Logistic Regression
    - Intuition, Motivation
    - Newton's Method

## Reading List

- Suggested:
    - **[MLAPP]**, Chapter 8: Logistic Regression

> In this lecture, we will move from regression to classification.Unlike of predicting some value for data in regression, we predict what category data belongs to in classification. And we will introduce logistic regression. In logistic regression, we will show how to find the optimal coefficients $\vw$ using Newton's method.

## Review: Supervised Learning

- Goal
    - Given data $X$ in feature sapce and the labels $Y$
    - Learn to predict $Y$ from $X$
- Labels could be discrete or continuous
    - Discrete-valued labels:  Classification
    - Continuous-valued labels:  Regression

<center>  </center>

## Classification Problem

### Classification Problem: Basics

- Given an input vector $\vx$, assign it to one of $K$ distinct classes $C_k$, where $k = 1,\dots,K$.

- The case $K=2$ is **Binary Classification**
    - Label $t=1$ means $x \in C_1$
    - Label $t=0$ means $x \in C_2$ (or sometimes $t=-1$)

- **Training:**  Learn a classifier $y(\vx)$ from data,
    $$ \text{Training Data} \quad \{ (\vx_1, t_1), \dots, (\vx_N, t_N) \} \implies \text{Classifier} \ y(\vx) $$

- **Prediction:** Predict labels of new data,
    $$ \text{New Data} \quad \{ (\vx^{new}_1, t^{new}_1), \dots, (\vx^{new}_m, t^{new}_m) \} \stackrel{h}{\implies} \{ y(\vx^{new}_1), \dots, y(x^{new}_m) \} $$

- **Performance Evaluation:** Evaluate learned classifier on test data,
    $$ \text{Test Data} \quad \{ (\vx^{test}_1, t^{test}_1), \dots, (\vx^{test}_m, t^{test}_m) \} \stackrel{y}{\implies} \{ y(\vx^{test}_1), \dots, y(\vx^{test}_m) \} \implies \text{Error Estimate} $$
    - To estimate **classification error**, we could use e.g. *zero-one loss*:
        $$
        E = \frac{1}{m} \sum_{j=1}^m \mathbb{1} [ y(\vx^{test}_j) \neq t^{test}_j) ]
        $$
        i.e. number of misclassified data.

### Classification Problems:  Strategies

- **Nearest-Neighbors:**  Given query data $\vx$, find closest training points and do a majority vote.

- **Discriminant Functions:**  Learn a function $y(\vx)$ mapping $\vx$ to some class $C_k$.

- **Probabilistic Model:**  Learn the distributions $P(C_k | \vx)$
    - *Discriminative Models* directly model $P(C_k | \vx)$ and learn parameters from the training set.
    - *Generative Models* learn class-conditional densities $P(\vx | C_k)$ and priors $P(C_k)$

## Logistic Regression
> - Logistic Regression is a technique for **classification**!

> - We will focus on *binary* classification

### Logistic Regression: Preliminary—Logistic Sigmoid Function

- The **logistic sigmoid function** is
    $$
    \sigma(a)
    = \frac{1}{1 + \exp(-a)}
    = \frac{\exp(a)}{1 + \exp(a)}
    $$

- Sigmoid function $\sigma(a)$ maps $(-\infty, +\infty) \to (0,1)$
<center>  </center>

### Logistic Regression:  Why use Logistic Sigmoid Function?
- Prediction is picking the larger one of $P(y=1 | \vx)$ and $P(y=0 | \vx)$.

- This can be implemented by evaluating **log odds**
    $$
    a = \ln \frac{P(y=1 | \vx)}{P(y=0  | \vx)}
    $$

- So the prediction is
    $$
    y =
    \left\{\begin{matrix}
    1& a\geq 0\\
    0& a< 0
    \end{matrix}\right.
    $$

- Since $P(y=1 | \vx) + P(y=0  | \vx)=1$, we could solve for
    $$
    P(y=1 | \vx) = \frac{\exp(a)}{1+\exp(a)} = \sigma(a)
    $$
    *Logistic Function* appears!

- A heuristic choice for log odds is a separating **hyper plane** $a = \vw^T \phi(\vx)$. So the criterion becomes
    $$
    \boxed{
    y =
    \left\{\begin{matrix}
    1& \vw^T \phi(\vx)\geq 0, \quad \text{i.e.} \quad \sigma(\vw^T\phi(\vx)\geq 0) \geq 0.5\\
    0& \vw^T \phi(\vx)< 0 , \quad \text{i.e.} \quad \sigma(\vw^T\phi(\vx)\geq 0) < 0.5
    \end{matrix}\right.}
    $$

- In this case, $P(y=1 | \vx) = \sigma(\vw^T \phi(\vx))$ and $P(y=0 | \vx) = 1-\sigma(\vw^T \phi(\vx))$.

### Logistic Regression: Underlying Model

- We already have
    $$
    \begin{align}
    P(y=1 | \vx, \vw) &= \sigma(\vw^T \phi(\vx)) \\
    P(y=0 | \vx, \vw) &= 1-\sigma(\vw^T \phi(\vx))
    \end{align}
    $$

- So we could model **class posterior** using Bernoulli random variable
    $$
    y | \vx ,\vw \sim \mathrm{Bernoulli}( \sigma(\vw^T \phi(\vx)) )
    $$

- We can obtain the best paramter $\vw$ by maximizing the likelihood of the training data.(Later)

- Logistic regression is simpest discriminative model that is **linear**
in the parameters.

### Logistic Regression: Example

```python
plt.figure(figsize=(10,6)); plot_linear_boundary();
```

- We could clearly see the linear boundary corresponding to $\vw^T \phi(\vx)$

### Logistic Regression:  Likelihood

- We saw before that the **likelihood** for each binary label is:
    $$
    \begin{align}
    P(y = 1 | \vx,\vw) &= \sigma(\vw^T \phi(\vx)) \\
    P(y = 0 | \vx,\vw) &= 1 - \sigma(\vw^T \phi(\vx))
    \end{align}
    $$
- With a clever trick, this
    $$
    P(y | x,w) = \sigma(\vw^T \phi(\vx))^y \cdot (1 - \sigma(\vw^T \phi(\vx)))^{1-y}
    $$

- For a data set $\{(\vx_n, t_n) \}_{n=1}^N$ where $t_n \in \{ 0,1 \}$, the **likelihood function** is
    $$
    P(\vy = \vt| \mathcal{X}, \vw) = \prod_{n=1}^N P(y=t_n | \vx_n, \vw) =\prod_{n=1}^N \sigma(\vw^T \phi(\vx_n))^{t_n} [1 - \sigma(\vw^T \phi(\vx_n))] ^{1-t_n}
    $$
    - where $\mathcal{X} = \{\vx_n \}_{n=1}^N$

- The optimal $\vw$ can be obtained by maximizing this likelihood.

- Maximum likelihood estimate $\vw_{ML}$ makse sense because $\vw_{ML}$ is the coefficient that are most likely to produce $\{t_n \}_{n=1}^N$ given $\mathcal{X}$.

- Define **negative log-likelihood** as the **loss**
    $$
    E(\vw) \triangleq -\ln P(\vy = \vt| \mathcal{X}, \vw)
    $$
- Maximizing **likelihood** is equivalent to minimizing **loss** $E(\vw)$

### Logistic Regression:  Gradient of Loss

- Loss function $E(\vw)$ can be transformed:
    $$
    \begin{align}
    E(\vw)
    &= -\ln P(\vy = \vt| \mathcal{X}, \vw) \\
    &= -\ln \prod \nolimits_{n=1}^N \sigma(\vw^T \phi(\vx_n))^{t_n} [1 - \sigma(\vw^T \phi(\vx_n))] ^{1-t_n} \\
    &= -\sum \nolimits_{n=1}^N \left[ t_n \ln \sigma(\vw^T \phi(\vx_n)) + (1-t_n) \ln(1-\sigma(\vw^T \phi(\vx_n))) \right] \\
    &= -\sum \nolimits_{n=1}^N \left[ t_n \ln \frac{\exp(\vw^T\phi(\vx_n))}{1+\exp(\vw^T\phi(\vx_n))} + (1-t_n) \ln(\frac{1}{1+\exp(\vw^T\phi(\vx_n))}) \right] \\
    &= -\sum \nolimits_{n=1}^N \left[ t_n \ln \frac{1}{1+\exp(-\vw^T\phi(\vx_n))} + (1-t_n) \ln(\frac{1}{1+\exp(\vw^T\phi(\vx_n))}) \right] \\
    &= \boxed{\sum \nolimits_{n=1}^N \left[ t_n \ln (1+\exp(-\vw^T\phi(\vx_n))) + (1-t_n) \ln(1+\exp(\vw^T\phi(\vx_n))) \right] }\\
    \end{align}
    $$

- Gradient of loss $\nabla_\vw E(\vw)$
    $$
    \begin{align}
    \nabla_\vw E(\vw)
    &= \nabla_\vw \sum \nolimits_{n=1}^N \left[ t_n \ln (1+\exp(-\vw^T\phi(\vx_n))) + (1-t_n) \ln(1+\exp(\vw^T\phi(\vx_n))) \right] \\
    &= \sum \nolimits_{n=1}^N \left[-t_n \frac{\exp(-\vw^T\phi(\vx_n))}{1+\exp(-\vw^T\phi(\vx_n))} \phi(\vx_n)+ (1-t_n) \frac{\exp(\vw^T\phi(\vx_n))}{1+\exp(\vw^T\phi(\vx_n))} \phi(\vx_n) \right] \\
    &= \sum \nolimits_{n=1}^N \left[-t_n (1-\sigma(\vw^T\phi(\vx_n)))+ (1-t_n) \sigma(\vw^T\phi(\vx_n)) \right] \phi(\vx_n) \\
    &= \sum \nolimits_{n=1}^N \left[\sigma(\vw^T\phi(\vx_n)) - t_n \right] \phi(\vx_n) \\
    &= \boxed{ \Phi^T \left( \sigma(\Phi \vw) - \vt \right) }
    \end{align}
    $$
    of which
    $$
    \Phi = \begin{bmatrix} - & \phi(\vx_1)^T & -\\  & \vdots & \\  - & \phi(\vx_N)^T & - \end{bmatrix}_{N \times M}
    \qquad
    \sigma(\Phi \vw)=\begin{bmatrix}
    \sigma(\vw^T\phi(\vx_1))\\
    \vdots\\
    \sigma(\vw^T\phi(\vx_N))
    \end{bmatrix}_{N \times 1}
    \qquad
    \vt = \begin{bmatrix}
    t_1\\
    \vdots\\
    t_N
    \end{bmatrix}_{N \times 1}
    $$

- With the gradient of loss, we could perform *gradient descent* to find $\vw_{ML}$.

- But we will use a new method by finding roots of first order derivative!

> Remark
> - Note that this gradient resembles the gradient in linear regression with least squares (Check Lecture 4)
    $$
    \begin{align}
    \text{Logistic Regression} \quad & \nabla_\vw E(\vw) = \Phi^T \left( \sigma(\Phi \vw) - \vt \right) \\
    \text{Linear Regression} \quad & \nabla_\vw E(\vw) = \Phi^T \left( \Phi \vw - \vt \right)
    \end{align}
    $$

### Newton's Method:  Overview

- First let's consider one dimension case.

- **Goal:** Finding *root* of a general function $f(x)$, i.e. solve for $x$ such that $$f(x)=0$$

- **Newton's Method:** Repeat until convergence:
    $$
    x_{n+1} = x_n - \frac{f(x_n)}{f'(x_n)}
    $$

### Newton's Method:  Geometric Intuition

- Find the roots of $f(x)$ by following its **tangent lines**. The tangent line of $f(x)$ at $x_n$ has equation
    $$
    \ell(x) = f(x_n) + (x-x_n) f'(x_n)
    $$
- Set next iterate $x_{n+1}$ to be **root** of tangent line:
    $$
    \begin{gather}
    f(x_n) + (x-x_n) f'(x_n) = 0 \\
    \implies \boxed{ x_{n+1}= x_n - \frac{f(x_n)}{f'(x_n)} }
    \end{gather}
    $$

<center>  </center>

### Newton's Method: Example

```python
def fn(x):
    return (x-2)**3;
def d1(x):
    return 3*(x-2)**2;
newton_example(fn,d1)
```

> Remark
> - Here is how Newton's method works
>     - Given some initial point $x_1$, we first find the tangent line $\ell_1(x)$ of $f(x)$ at $x_1$.
>     - Let $x_2$ denote the root of $\ell_1(x)$, i.e. $\ell_1(x_2)=0$
>     - Find the tangent line $\ell_2(x)$ of $f(x)$ at $x_2$.
>     - Let $x_3$ denote the root of $\ell_2(x)$, i.e. $\ell_1(x_3)=0$

### Newton's Method: Finding Stationary Point

- We have shown how to use Newton's method to find the root for $f(x)$
    $$
    x_{n+1}= x_n - \frac{f(x_n)}{f'(x_n)}
    $$

- **Note** that Stationary point of $f(x)$ is equivalent to root of $f'(x)$

- So, we could find stationary point of $f(x)$ by finding root of $f'(x)$ using Newton's method.

- The iteration steps becomes
    $$
    x_{n+1}= x_n - \frac{f'(x_n)}{f''(x_n)}
    $$

- For *multi-dimension* case, this iteration turns into
    $$
    \vx_{n+1}= \vx_n - \left(\nabla^2 f(\vx_n)\right)^{-1} \nabla_\vx f(\vx_n)
    $$
    of which $\nabla^2 f(\vx_n)$ is **Hessian matrix** which is the *second order derivative*
    $$
    \nabla^2 f = \begin{bmatrix}
    \frac{\partial f}{\partial x_1\partial x_1} & \cdots & \frac{\partial f}{\partial x_1\partial x_n}\\
    \vdots & \ddots & \vdots\\
    \frac{\partial f}{\partial x_n\partial x_1} & \cdots & \frac{\partial f}{\partial x_n\partial x_n}
    \end{bmatrix}
    $$

### Logistic Regression: Applying Newton's Method

- Back to logistic regression!

- Recall our goal to minimize $E(\vw)$ and we already have its gradient
    $$
    \nabla_\vw E(\vw) = \sum \nolimits_{n=1}^N \left[\sigma(\vw^T\phi(\vx_n)) - t_n \right] \phi(\vx_n) = \Phi^T \left( \sigma(\Phi \vw) - \vt \right)
    $$

- To minimize of $E(\vw)$, we could use Newton's method to find its *stationary point*!

- To use Newton's method, we need the *Hessian matrix*.

### Logistic Regression: Hessian Matrix
$$
\begin{align}
\nabla^2 E(\vw)
&= \nabla_\vw \nabla_\vw E(\vw) \\
&= \nabla_\vw \sum \nolimits_{n=1}^N \left[\sigma(\vw^T\phi(\vx_n)) - t_n \right] \phi(\vx_n) \\
&= \sum \nolimits_{n=1}^N \nabla_\vw \sigma(\vw^T\phi(\vx_n))  \phi(\vx_n) \\
&= \sum \nolimits_{n=1}^N \nabla_\vw  \frac{1}{1 + \exp(-\vw^T \phi(\vx_n))}  \phi(\vx_n) \\
&= \sum \nolimits_{n=1}^N \phi(\vx_n) \frac{\exp(-\vw^T \phi(\vx_n))}{(1 + \exp(-\vw^T \phi(\vx_n)))^2}  \phi(\vx_n)^T \\
&= \sum \nolimits_{n=1}^N \phi(\vx_n) \frac{1}{1 + \exp(-\vw^T \phi(\vx_n))} \frac{\exp(-\vw^T \phi(\vx_n))}{1 + \exp(-\vw^T \phi(\vx_n))}   \phi(\vx_n)^T \\
&= \sum \nolimits_{n=1}^N \phi(\vx_n) [\sigma(\vw^T \phi(\vx_n)) \cdot ( 1 - \sigma(\vw^T \phi(\vx_n)) )]  \phi(\vx_n)^T \\
&= \sum \nolimits_{n=1}^N \phi(\vx_n) r_n(\vw) \phi(\vx_n)^T
\end{align}
$$
- of which $r_n(\vw) = \sigma(\vw^T \phi(\vx_n)) \cdot ( 1 - \sigma(\vw^T \phi(\vx_n)) )$

$$
\begin{align}
H_\vw E(\vw)
&= \sum \nolimits_{n=1}^N \phi(\vx_n) r_n(\vw) \phi(\vx_n)^T \\
&= \begin{bmatrix}
| &  & | \\
\phi(\vx_1) & \cdots & \phi(\vx_N)\\
| &  & |
\end{bmatrix}
\begin{bmatrix}
r_1(\vw) &  & \\
 & \ddots & \\
 &  & r_N(\vw)
\end{bmatrix}
\begin{bmatrix}
- & \phi(\vx_1)^T & -\\
 & \vdots & \\
- & \phi(\vx_N)^T & -
\end{bmatrix} \\
&= \boxed{\Phi^T R(\vw) \Phi}
\end{align}
$$

- of which
    $$
    R(\vw) = \begin{bmatrix}
    r_1(\vw) &  &  & \\
     & r_2(\vw) &  & \\
     &  & \ddots & \\
     &  &  & r_N(\vw)
    \end{bmatrix}
    $$

### Logistic Regression: Applying Newton's Method

- We already have
    $$
    \begin{gather}
    \nabla_\vw E(\vw) = \Phi^T \left( \sigma(\Phi \vw) - \vt \right) \\
    H_\vw E(\vw) = \Phi^T R(\vw) \Phi
    \end{gather}
    $$

- So the iteration step is
    $$
    \begin{align}
    \vw_{n+1}
    &= \vw_n - \left(H_\vw E(\vw_n)\right)^{-1} \nabla_\vw f(\vw_n) \\
    &= \boxed{\vw_n - \left(\Phi^T R(\vw_n) \Phi \right)^{-1} \Phi^T \left( \sigma(\Phi \vw_n) - \vt \right)}
    \end{align}
    $$

- Repeat until convergence and we could get maximum likelihood estimate $\vw_{ML}$ which minimizes the loss function $E(\vw)$ and maximizes likelihood function $ P(\vy = \vt| \mathcal{X}, \vw)$

### Logistic Regression: Do we have closed-form solution?

- Recall for **ordinary least squares** and **regularized least squares**, we have closed-form solution:

|               | Ordinary Least Squares  |  Regularized Least Squares |
| ------------- | :-------------: | :-------------: |
| **Derivate of Loss Function** | $\Phi^T\Phi \vec{w} - \Phi^T \vec{t}$ | $(\Phi^T \Phi + \lambda I)\vec{w} - \Phi^T \vec{t}$ |
| **Closed-form Solution**      | $(\Phi^T \Phi)^{-1} \Phi^T \vec{t}$   | $(\Phi^T \Phi + \lambda I)^{-1} \Phi^T \vec{t}$     |

- They are obtained by finding the closed-form root of derivative of loss function。

- For logistic regression, we have
    $$
    \begin{gather}
    \nabla_\vw E(\vw) = 0 \\
    \Downarrow \\
    \Phi^T \left( \sigma(\Phi \vw) - \vt \right) = 0
    \end{gather}
    $$

- Existence of sigmoid function makes $\nabla_\vw E(\vw)$ **nonlinear** and no closed-form solution exists.

- So we must **iterate**!

### Appendix: Multi-class Classification using Logistic Regression

- We have seen sigmoid function enables us to do binary classification with logistic regression.

- What if we want have multiple classes?

- We will resort to **softmax** aka **normalized exponential** function

- **Softmax Function**
    $$
    p_k = \frac{\exp(q_k)}{\sum_j \exp(q_j)}
    $$
    Given any real numbers $q_1, \ldots, q_n$, we can generate a distribution on them using softmax function.

- Recall in binary case, we have
    $$
    P(y = 1 | \vx,\vw) = \sigma(\vw^T \phi(\vx))
    $$

- For K-class classification, we define $\mathcal{W} = {\vw}_{k=1}^K$.

- The probablity data $\vx$ belongs to class $j$ is
    $$
    P(y = j | \vx,\mathcal{W}) = \frac{\exp(\vw_j^T \phi(\vw))}{\sum_{k=1}^{K} \exp(\vw_k^T \phi(\vw))}
    $$

- We classify using
    $$
    y = \underset{j \in \{1,\dots, K\}}{\arg \max} P(y = j | \vx,\mathcal{W})
    $$

- Similarly, $\mathcal{W} = {\vw}_{k=1}^K$ is learned by maximizing likelihood function.

- For details, please refer to [this](http://ufldl.stanford.edu/wiki/index.php/Softmax_Regression)


# Lista 1 - Computação Científica II 2019.2
> Gil Miranda

> Repo: https://github.com/mirandagil/university-courses/tree/master/comp-cientifica-II-2019-2

> GitHub: @mirandagil

`last update: 10/10/2018`

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sp
sp.init_printing(use_unicode = True)
```

---
#### 1.2.1

$$
\begin{align}
e^{at} &= e^{(\mu + i\omega)t}\\
&= e^{\mu t + i\omega t}\\
&= e^{\mu t} \cdot e^{i\omega t}\\
&= e^{\mu t} \cdot( cos(\omega t) + i sen(\omega t))\\
&= e^{\mu t}cos(\omega t) + i\, e^{\mu t}sen(\omega t)
\end{align}
$$
Então, a parte real e imaginária de $e^{at}$ com $a = \mu + i\,\omega$:
$$
Re(e^{at}) = e^{\mu t}cos(\omega t)\\
Im(e^{at}) = i\, e^{\mu t}sen(\omega t)
$$

---
#### 1.2.2

a) $\mu \in \mathbb{R}, \mu < 0 \, \, \& \, \,  \omega = 0$

Assim teremos $e^{\mu t}cos(0 t) = e^{\mu t}$, que com $\mu < 0$ é uma exponencial decrescente

b) $\mu \in \mathbb{R}, \mu > 0 \, \, \& \, \, \omega = 0$

Assim teremos $e^{\mu t}cos(0 t) = e^{\mu t}$, que com $\mu > 0$ é uma exponencial crescente

c) $\mu = 0, \omega \in \mathbb{R}$

Assim teremos $e^{0 t}cos(\omega t) = 1\cdot cos(\omega t)$, que com é uma senoidal com amplitude constante 1.

d) $\mu > 0, \omega \in \mathbb{R}, |\mu| << |\omega|$

Assim teremos $e^{\mu t}cos(\omega t)$, que com $| \mu| << |\omega|$ é uma senoidal com amplitude crescente.

d) $\mu < 0, \omega \in \mathbb{R}, |\mu| << |\omega|$

Assim teremos $e^{\mu t}cos(\omega t)$, que com $|\mu| << |\omega|$ é uma senoidal com amplitude crescente.

```python
def q_1_func(mu,w,t):
    return np.e**(mu*t)*np.cos(w*t)
```

```python
t = np.linspace(-20,20, 400)
m_a = -0.5
w_a = 0

m_b = 0.5
w_b = 0

m_c = 0
w_c = -.5

m_d = 0.05
w_d = 3

m_e = -0.05
w_e = 3

fig, ((ax1,ax2, ax3), (ax4,ax5, ax6)) = plt.subplots(nrows=2, ncols=3, figsize=(16,10))
fig.delaxes(ax6)
fig.suptitle('Exemplos para exercício 1.2.2')
names = ['(a)', '(b)', '(c)', '(d)', '(e)']
for i,j in zip([ax1, ax2, ax3, ax4, ax5],names):
    i.set_title('Exemplo para ' + j)

ax1.plot(q_1_func(m_a, w_a, t))
ax2.plot(q_1_func(m_b, w_b, t))
ax3.plot(q_1_func(m_c, w_c, t))
ax4.plot(q_1_func(m_d, w_d, t))
ax5.plot(q_1_func(m_e, w_e, t))
plt.show()
```

---

#### 2.1.5

Temos o sistema
$$
\begin{align}
        \begin{pmatrix}m_1 & 0 \\ 0 & m_2\end{pmatrix}\frac{\mathrm{d}^2 x}{\mathrm{d}t^2} + \begin{pmatrix}d_{v1} + d_{v_2} & -d_{v2} \\ -d_{v2} & d_{v2}\end{pmatrix}\frac{\mathrm{d} x}{\mathrm{d}t} + \begin{pmatrix}k_1 + k_2 & -k_{2} \\ -k_{2} & k_{2}\end{pmatrix}x = \begin{pmatrix}0 \\ \hat{F}_2 sin(\omega t) \end{pmatrix}
\end{align}
$$
Fazendo uma substituição: $u = \frac{\mathrm{d} x}{\mathrm{d}t}$, o que nos dá: $\frac{\mathrm{d} u}{\mathrm{d}t} = \frac{\mathrm{d}^2 x}{\mathrm{d}t^2}$
$$
\begin{align}
        \begin{pmatrix}m_1 & 0 \\ 0 & m_2\end{pmatrix}\frac{\mathrm{d} u}{\mathrm{d}t} + \begin{pmatrix}d_{v1} + d_{v_2} & -d_{v2} \\ -d_{v2} & d_{v2}\end{pmatrix}u + \begin{pmatrix}k_1 + k_2 & -k_{2} \\ -k_{2} & k_{2}\end{pmatrix}x = \begin{pmatrix}0 \\ \hat{F}_2 sin(\omega t) \end{pmatrix}
\end{align}
$$
Isolando $\frac{\mathrm{d} u}{\mathrm{d}t}$
$$
\begin{align}
        \begin{pmatrix}m_1 & 0 \\ 0 & m_2\end{pmatrix}\frac{\mathrm{d} u}{\mathrm{d}t} = \begin{pmatrix}0 \\ \hat{F}_2 sin(\omega t) \end{pmatrix} - \begin{pmatrix}d_{v1} + d_{v_2} & -d_{v2} \\ -d_{v2} & d_{v2}\end{pmatrix}u - \begin{pmatrix}k_1 + k_2 & -k_{2} \\ -k_{2} & k_{2}\end{pmatrix}x
\end{align}
$$
Como não há informações sobre $m_1$ e $m_2$, não podemos ter certeza de que a matriz multiplicando $\frac{\mathrm{d} u}{\mathrm{d}t}$ é invertível.

Reescrevendo o sistema de primeira ordem:
$$
\begin{align}
    \begin{cases}
    \frac{\mathrm{d} x}{\mathrm{d}t} = u\\
    \begin{pmatrix}m_1 & 0 \\ 0 & m_2\end{pmatrix}\frac{\mathrm{d} u}{\mathrm{d}t} = \begin{pmatrix}0 \\ \hat{F}_2 sin(\omega t) \end{pmatrix} - \begin{pmatrix}d_{v_1} + d_{v_2} & -d_{v_2} \\ -d_{v_2} & d_{v_2}\end{pmatrix}u - \begin{pmatrix}k_1 + k_2 & -k_{2} \\ -k_{2} & k_{2}\end{pmatrix}x
    \end{cases}
\end{align}
$$
E podemos escrever o sistema na forma vetorial, seja:
$$
\begin{align}
    \begin{cases}
    \begin{pmatrix}m_1 & 0 \\ 0 & m_2\end{pmatrix} = M\\
    \begin{pmatrix}0 \\ \hat{F}_2 sin(\omega t) \end{pmatrix} = F\\
    \begin{pmatrix}d_{v_1} + d_{v_2} & -d_{v_2} \\ -d_{v_2} & d_{v_2}\end{pmatrix} = D\\
    \begin{pmatrix}k_1 + k_2 & -k_{2} \\ -k_{2} & k_{2}\end{pmatrix} = K
     \end{cases}
\end{align}
$$
Então o sistema se reduz a
$$
\begin{align}
\begin{pmatrix}1 & 0 \\ 0 & M\end{pmatrix}\frac{\mathrm{d}}{\mathrm{d}t}\begin{pmatrix}x \\ u\end{pmatrix} = F + \begin{pmatrix}0 & 1 \\ -K & -D\end{pmatrix}\begin{pmatrix}x \\ u\end{pmatrix}
\end{align}
$$

---
#### 2.1.7

Dado o sistema:
$$
\begin{cases}
\frac{\mathrm{d}x}{\mathrm{d}t} = f(x,y)\\
0 = g(x,y)\\
x(0) = x_0
\end{cases}
$$
Precisamos transformar em um sistema de equações diferenciais, para isso falta uma expressão para $\frac{\mathrm{d}y}{\mathrm{d}t}$, portanto olhando para a derivada de $g(x,y)$, sabemos que $g(x,y) = 0$ é constante, portanto sua derivada será também $0$
$$
\begin{align}
0 &= \frac{\partial g}{\partial x}\frac{\mathrm{d}x}{\mathrm{d}t} + \frac{\partial g}{\partial y} \frac{\mathrm{d}y}{\mathrm{d}t}
\end{align}
$$
Aqui aparece o termo que queremos $\frac{\mathrm{d}y}{\mathrm{d}t}$, como por hipotese a matriz $\frac{\partial g}{\partial y}$ é não singular, então ela é invertível. Podemos então isolar o termo
$$
\begin{align}
\frac{\partial g}{\partial y} \frac{\mathrm{d}y}{\mathrm{d}t} &= -\frac{\partial g}{\partial x}\frac{\mathrm{d}x}{\mathrm{d}t}\\
\frac{\mathrm{d}y}{\mathrm{d}t} &= -\frac{\partial g}{\partial y}^{-1} \cdot \frac{\partial g}{\partial x}\frac{\mathrm{d}x}{\mathrm{d}t}\\
\end{align}
$$

E com estas informações podemos montar o sistema:

$$
\begin{cases}
\frac{\mathrm{d}x}{\mathrm{d}t} = f(x,y)\\
\frac{\mathrm{d}y}{\mathrm{d}t} = -\left(\frac{\partial g}{\partial y}\right)^{-1} \cdot \frac{\partial g}{\partial x}\frac{\mathrm{d}x}{\mathrm{d}t}\\
x(0) = x_0
\end{cases}
$$
Que é um sistema de equações diferenciais

---
#### 2.2.1

$$
A =
\begin{pmatrix}
-2 & 1 \\
-4 & 3
\end{pmatrix}
$$
Encontrando o determinante de $|A - \lambda I|$
$$
|A - \lambda I| =
\begin{pmatrix}
-2 -\lambda & 1 \\
-4 & 3 -\lambda
\end{pmatrix}
$$

$$
\begin{align}
det |A - \lambda I| &= (-2 -\lambda)(3 -\lambda) + 4\\
&= -6 + 2\lambda -3\lambda + \lambda^2 +4\\
&= \lambda^2 -\lambda -2
\end{align}
$$

Encontrando os autovalores como raízes do polinômio característico
$$
\begin{align}
\Delta &= 1 + 8 = 9\\
\lambda_1 &= \frac{1 + 3}{2}\\
&= 2\\
\lambda_2 &= \frac{1 -3}{2}\\
&= -1\\
\end{align}
$$

Encontrando os autovetores referentes a cada autovalor \\
$$
Av = \lambda v\\
(A - \lambda I)v = 0
$$

$\lambda_1 = 2$
$$
\begin{align}
\begin{pmatrix}
-2 -2 & 1 \\
-4 & 3 -2
\end{pmatrix} = \begin{pmatrix}
-4 & 1 \\
-4 & 1
\end{pmatrix}\\
\begin{pmatrix}
-4 & 1 \\
-4 & 1
\end{pmatrix} \begin{pmatrix}x \\ y\end{pmatrix} = \begin{pmatrix}0 \\ 0\end{pmatrix}
\end{align}
$$
$$
\begin{cases}
-4x + y = 0\\
-4x + y = 0
\end{cases}
-4x = -y\\
x = \frac{1}{4}y\\
v = \begin{pmatrix} \frac{1}{4} \\ 1 \end{pmatrix}
$$

Para normalizar, fazemos:
$$
\begin{align}
\lVert v \rVert &= \sqrt{\left(\frac{1}{4}\right)^2 + \left(1\right)^2}\\
&= \frac{\sqrt{(17)}}{4}
\end{align}
$$
Logo, o autovetor referente ao autovalor $\lambda_1 = 2$ será

$$
\begin{align}
v_1 &= \frac{4}{\sqrt{17}}\begin{pmatrix} \frac{1}{4} \\ 1 \end{pmatrix}\\
&= \begin{pmatrix} \frac{1}{\sqrt{17}} \\ \frac{4}{\sqrt{17}} \end{pmatrix}
\end{align}
$$

$\lambda_2 = -1$
$$
\begin{align}
\begin{pmatrix}
-2 +1 & 1 \\
-4 & 3 +1
\end{pmatrix} = \begin{pmatrix}
-1 & 1 \\
-4 & 4
\end{pmatrix}\\
\begin{pmatrix}
-1 & 1 \\
-4 & 4
\end{pmatrix} \begin{pmatrix}x \\ y\end{pmatrix} = \begin{pmatrix}0 \\ 0\end{pmatrix}
\end{align}
$$
$$
\begin{cases}
-x + y = 0\\
-4x + 4y = 0
\end{cases}\\
x = y\\
v = \begin{pmatrix} 1 \\ 1 \end{pmatrix}
$$

Para normalizar, fazemos:
$$
\begin{align}
\lVert v \rVert &= \sqrt{\left(1\right)^2 + \left(1\right)^2}\\
&= \sqrt{2}
\end{align}
$$
Logo, o autovetor referente ao autovalor $\lambda_2 = -1$ será

$$
\begin{align}
v_2 &= \frac{1}{\sqrt{2}}\begin{pmatrix} 1 \\ 1 \end{pmatrix}\\
&= \begin{pmatrix} \frac{1}{\sqrt{2}} \\ \frac{1}{\sqrt{2}} \end{pmatrix}
\end{align}
$$

```python
A = np.array([[-2,1],[-4,3]])
eigs_A = np.linalg.eig(A)
eigs_A
Am = sp.Matrix([[-2,1],[-4,3]])
norma = Am.eigenvects()[1][2][0].norm()
Am.eigenvects()[1][2][0]/norma
```

---
#### 2.2.2

Vamos calcular $e^{At}$, para isso primeiro fazemos a diagonalização da matriz A, já que ela é diagonalizavel e assim torna-se mais fácil a operação sendo feita na decomposição em autovalores.

```python
t = sp.symbols('t', real = True, constant = True) ## Define uma constante no SymPy

A = sp.Matrix([[-2,1],[-4,3]]) ## Define a Matriz a ser trabalhada

diag_A = sp.Matrix([[2,0],[0,-1]]) ## Criando a matriz diagonal com os autovalores de A

## Criando a matriz da base de autovetores
eigens = A.eigenvects()
eig1_norma = eigens[1][2][0].norm()
eig2_norma = eigens[0][2][0].norm()
eig1_vect = eigens[1][2][0]/eig1_norma
eig2_vect = eigens[0][2][0]/eig2_norma
S = sp.Matrix([])
S = S.row_join(eig1_vect)
S = S.row_join(eig2_vect) ## Matriz da base de autovetores

inv_S = S.inv() ## Inversa da Matriz da base

e_to_diag = sp.exp(diag_A*t) ## Aplicando exponencial a cada entrada da matriz diagonal
```

```python
## Print da Matriz A e sua diagonal correspondente
A, diag_A
```

```python
## Print da Matriz S e sua inversa
S, inv_S
```

Como $A = S\Lambda S^{-1}$, temos $e^{At} = Se^{\Lambda t}S^{-1}$.

```python
e_to_A = S * e_to_diag * inv_S
e_to_A
```

---
#### 2.2.3

Sabemos que a solução para a equação diferencial $ \dot{u} = Au$ é na forma
$$
\begin{align}
u &= e^{At} C
\end{align}
$$
Como $u_0 = \begin{pmatrix} 1 \\ 1 \end{pmatrix}$, então
$$
\begin{align}
u(0) &= e^{A0} C\\
&= C\\
\therefore\\
C &= u_0
\end{align}
$$
Então a solução da equação diferencial será:
$$
\begin{align}
u &= \left[\begin{matrix}- \frac{e^{2 t}}{3} + \frac{4 e^{- t}}{3} & \frac{e^{2 t}}{3} - \frac{e^{- t}}{3}\\- \frac{4 e^{2 t}}{3} + \frac{4 e^{- t}}{3} & \frac{4 e^{2 t}}{3} - \frac{e^{- t}}{3}\end{matrix}\right] \begin{pmatrix} 1 \\ 1 \end{pmatrix}\\
u &= \left[\begin{matrix}e^{- t}\\e^{- t}\end{matrix}\right]
\end{align}$$

```python
u0 = sp.Matrix([1,1])
sol = e_to_A*u0 ## Solução da Equação Diferencial
sol
```

---
#### 2.2.6

```python
A = sp.Matrix([[-1,0,0],[1,-1,0],[0,1,-1]])
eig_A = A.eigenvects()
eig_A
```

Portanto a Matriz $A$ tem autovalor $\lambda = -1$ com multiplicidade $3$. E possui autovetor $v = \begin{pmatrix} 0 \\ 0 \\ 1 \end{pmatrix}$.

Como os autovetores são múltiplos, são linearmente dependentes, o que impede que formemos uma base de autovetores para o espaço, logo a matriz A é defeituosa.

```python
B = sp.Matrix([[0,1,1],[0,0,1],[0,0,0]])
eig_B = B.eigenvects(B)
eig_B
```

Portanto a Matriz $A$ tem autovalor $\lambda = 0$ com multiplicidade $3$. E possui autovetor $v = \begin{pmatrix} 1 \\ 0 \\ 0 \end{pmatrix}$.

Como os autovetores são múltiplos, são linearmente dependentes, o que impede que formemos uma base de autovetores para o espaço, logo a matriz B é defeituosa. Além do mais, uma matriz nilpotente é aquela que $B^k = 0$, para algum $k \in \mathbb{N}$, vamos verificar que B satisfaz

```python
B, B**2, B**3
```

Então a matriz $B$ é nilpotente com índice 3

---
#### 2.2.7

Calculando $e^{Bt}$, a matrix é defeituosa então iremos expandir em séries de Taylor para a exponencial
$$
\begin{align}
e^{At} &= I + At + \frac{A^2 t^2}{2!} + \frac{A^3 t^3}{3!} + \dots\\
\end{align}
$$

Agora vamos calcular $e^{Bt}$, novamente precisamos expandir em Série de Taylor, mas como a Matriz $B$ é nilpotente com indice 3 utilizaremos até o termo de grau 2 da série, já que todo o resto será uma matriz de zeros.

$$
\begin{align}
e^{Bt} &= I + Bt + \frac{B^2 t^2}{2!} + \frac{B^3 t^3}{3!} + \dots\\
&= I + Bt + \frac{B^2 t^2}{2!} + 0\\
&= I + Bt + \frac{B^2 t^2}{2!}
\end{align}
$$

```python
I = sp.eye(3) ## Cria uma matriz identidade
e_to_B = I + B*2 + (B**2)*(2**2)/2
e_to_B
```

```python

```


```python
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
#  Copyright (c) 2020. Reda Bouadjenek, Deakin University                      +
#     Email:  reda.bouadjenek@deakin.edu.au                                    +
#                                                                              +
#  Licensed under the Apache License, Version 2.0 (the "License");             +
#   you may not use this file except in compliance with the License.           +
#    You may obtain a copy of the License at:                                  +
#                                                                              +
#                 http://www.apache.org/licenses/LICENSE-2.0                   +
#                                                                              +
#    Unless required by applicable law or agreed to in writing, software       +
#    distributed under the License is distributed on an "AS IS" BASIS,         +
#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.  +
#    See the License for the specific language governing permissions and       +
#    limitations under the License.                                            +
#                                                                              +
# ++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
from gurobipy import *
import matplotlib.pyplot as plt

```

# Optimizing expected F1-Score using Linear Programming

## Installation

The algorithm described in this Notebook requires [Gurobi](https://www.gurobi.com/) to be installed in your Python environement. Please follow [these instructions](https://www.gurobi.com/documentation/9.1/quickstart_mac/software_installation_guid.html).
Once Gurobi is installed, you need to add it to your own Python environment. Doing so requires you to install the gurobipy module. The steps for doing this depend on your
platform. On Linux, you will need to open a terminal window, change your current directory to
the Gurobi <installdir> (the directory that contains the file setup.py), and issue the following
command:
```
python setup.py install
```

## 1- Motivation
We will argue that (expected) F1-Score of a substring is the only standard criteria that balances all of substring desiderata and is hence objective we should optimize.

To proceed with the formal derivation, we adopt the Boolean relevance framework in information retrieval and thus assume that a token $j$ has a ground truth relevance assessment $B(j)$ available at evaluation time.

Because selecting a substring implies a Boolean selection model (a substring either contains or do not contain tokens) and we have a probabilistic estimate of relevance $P(j)$ for each token, we propose to evaluate
expected variants of standard precision, recall, and F1-score of these substrings.

However, as standard for both precision and recall, we note that precision and recall alone can be trivially optimized by undesired solutions.  That is, the substring that selects all tokens would trivially maximize (expected) recall.  Similarly, the substring that selects the highest probability singleton token would maximize expected precision.  This leaves expected F1-score as the only of these three objectives commonly used in Boolean information retrieval that does not have an undesired solution.

## 2- Deriving Expected F1-Score (EF1)

To formally define expected F1-Score, we first begin with definitions of expected precision and recall.  Given a substring $S_{sub}$  of a string $S$ ($|S|=m$), the precision of $S_{sub}$ is defined as follows:
\begin{equation}
   P(S_{sub}) = \dfrac{\sum_{j \in S_{sub}} B(j)}{|S_{sub}|} = \dfrac{\sum_{j=1}^m B(j)I(j)}{\sum_{j=1}^m I(j)}
\end{equation}

where the two variables  $I(j) \in \{0,1\}$ and $B(j) \in \{0,1\}$ are associated with each token $j$ of $S$:

- $I(j)$ is an indicator referring to whether a token $j$ is part of $S_{sub}$ ($\textrm{true}\!=\!1$);
- $B(j)$ is a Boolean random variable indicating the (ground truth) relevance of a tokan $j$ ($\textrm{relevant}\!=\!1$).

Given that $B(j)$ is a Boolean random variable, we can take the expectation of $P(S_{sub})$ leading to the following definition of expected precision $EP(S_{sub})$:

\begin{equation}
EP(S_{sub})=\mathbb{E_{P}}\left[\dfrac{{ \sum_{j=1}^{m}}B(j)I(j)}{{ \sum_{j=1}^{m}}I(j)}\right]=\dfrac{{ \sum_{j=1}^{m}}\mathbb{E_{S}}[B(j)]I(j)}{{ \sum_{j=1}^{m}}I(j)}=\dfrac{{ \sum_{j=1}^{m}}P(j)I(j)}{{ \sum_{j=1}^{m}}I(j)}
\end{equation}

where the variable $P(j) \in [0,1]$ is the probability relevance of a token $j$.

Similarly the recall of a substring $R(S_{sub})$ is defined as:

\begin{equation}
   R(S_{sub}) = \dfrac{\sum_{j \in S_{sub}} B(j)}{\sum_{j \in S} B(j)} = \dfrac{\sum_{j=1}^m B(j)I(j)}{ \sum_{j=1}^m B(j)}
\end{equation}

Taking a 1st order Taylor expansion, we have the following expectation approximation
$\mathbb{E}(X/Y)\approx \mathbb{E}(X)/ \mathbb{E}(Y)$ for two dependent random variables $X$ and $Y$. Hence, we can now define an approximated expected recall as follows:

\begin{equation}
   ER(S_{sub})=\mathbb{E_{P}}\left[\dfrac{{ \sum_{j=1}^{m}}B(j)I(j)}{\sum_{j=1}^m B(j)}\right]\approx\dfrac{{ \sum_{j=1}^{m}}\mathbb{E_{P}}[B(j)]I(j)}{{ \sum_{j=1}^{m}}\mathbb{E_{P}}[B(j)]}=\dfrac{{ \sum_{j=1}^{m}}P(j)I(j)}{{ \sum_{j=1}^{m}}P(j)}
\end{equation}

Finally, we define the approximated expected F1-Scor (EF1) using the expected precision and the approximated expected recall as follows:
\begin{align}
    EF1(E)  \approx \dfrac{2\times EP\times ER}{EP+ER} = \dfrac{2\times\sum_{j=1}^m P(j)I(j)}{\sum_{j=1}^m I(j) + \sum_{j=1}^m P(j)}
    \label{eq:EF1}
\end{align}

## 3- Fractional MILP Formulation

We begin by reformulating the Expected F1-score (EF1) objective to prepare for further optimization steps by replacing the global sum of scores of all tokens with a constant $C = \sum_{j=1}^m P(j)$:
\begin{equation}
\begin{aligned}
    EF1 &= \dfrac{2\times \sum_{j=1}^m P(j)I(j)}{\sum_{j=1}^m I(j) + \sum_{j=1}^m P(j)} = \dfrac{2 \times \sum_{j=1}^m P(j)I(j)}{\sum_{j=1}^m I(j) + C}
\end{aligned}
\end{equation}

In order to obtain the substring with the obtimal Expected F1-score, we define the following fractional MILP:

\begin{equation}
\begin{aligned}
& \underset{I(j)}{\text{maximize}}
& & \dfrac{\sum_{j=1}^m P(j)I(j)}{\sum_{j=1}^m I(j) + C} \\
& s.t
& & \text{# Interval selection contraints.} \\
& & & 1 <=  s <= e <= m \\
& & & I(j) =
   \begin{cases}
     1, & \text{if $(s \leqslant j \leqslant e)$}  \\
     0, & \text{otherwise}
  \end{cases}
\end{aligned}
\end{equation}

## 4- Transformation to a MILP

While there are no direct solvers for fractional MILPs, we can transform the former equation into a pure MILP form for which we have efficient and optimal solvers.  To do this, we use the Charnes-Cooper method and Glover linearization method with big-M constraints, where auxiliary variables $w(j)$ and $u$ are  introduced*. Here, $w(j)$ is defined as $w(j)=I(j)\times u$ with $u$ defined as follows:
\begin{equation}
u = \dfrac{1}{\sum_{j=1}^m I(j) + C}
\end{equation}

Then, the EF1 optimization problem is able to be transformed into the following MILP problem:
\begin{equation}
\begin{aligned}
& \underset{w,u,I,s,e}{\text{maximize}}
& & \sum_{j=1}^m P(j)w(j) \\
& s.t
& & \sum_{j=1}^m w(j) + uC = 1 \\
& & & w(j) \leqslant u, \quad w(j) \leqslant M\times I(j)  \\
& & & w(j) \geqslant u - M\times [1-I(j)] \\
& & & u > 0,  \quad I(j) \in \{0, 1\}, \quad w(j) \geqslant 0 \\
& & & \text{# Interval selection contraints.} \\
& & & 1 <=  s <= e <= m \\
& & & I(j) =
   \begin{cases}
     1, & \text{if $(s \leqslant j \leqslant e)$}  \\
     0, & \text{otherwise}
  \end{cases} \label{eq:cons1}
\end{aligned}
\end{equation}

*https://optimization.mccormick.northwestern.edu/index.php/Mixed-integer_linear_fractional_programming_(MILFP)

## 5- Algorithm implementation

```python
def select_LP_optimal_subsequence(probabilities):
    """
    This function optimizes the expected f1-score to choose the optimal subsequence
    of probabilities.
    :param probabilities: the sequence of probabilities.
    :return: the start index and the end index of the optimal subsequence.
    """
    C = sum(probabilities)
    M = 1e4  # Value for Big M.
    dict_probabilities = dict({j: probabilities[j] for j in range(len(probabilities))})
    ###################
    # Define the model.
    ###################
    model = Model('EF1-Score')
    model.setParam( 'OutputFlag', False) # make Gurobi silent.
    # element variable
    w = model.addVars(dict_probabilities.keys(), lb=0.0, ub=1.0, vtype='C', name='w')
    I = model.addVars(dict_probabilities.keys(), lb=0.0, ub=1.0, vtype='B', name='I')
    u = model.addVar(lb=0, vtype='C', name='u')

    # Index variables
    Is = model.addVars(dict_probabilities.keys(), vtype='B', name='Is')
    Ie = model.addVars(dict_probabilities.keys(), vtype='B', name='Ie')

    s = model.addVar(lb=0, ub=len(dict_probabilities) - 1, vtype='I', name='s')
    e = model.addVar(lb=0, ub=len(dict_probabilities) - 1, vtype='I', name='e')

    # Objective function
    model.setObjective(w.prod(dict_probabilities), GRB.MAXIMIZE)

    # Constraints

    model.addConstr(quicksum(w[j] for j in range(len(dict_probabilities))) + u * C == 1.0, 'sum(w)+uC=1')
    model.addConstr(s <= e, 's<=e')
    for j in dict_probabilities:
        model.addConstr(w[j] <= u, f"w[{j}]<=u")
        model.addConstr(w[j] <= M * I[j], f"w[{j}]<=M*I({j})")
        model.addConstr(w[j] >= u - M * (1 - I[j]), f"w[{j}]>=u-M*[1-I({j})]")
        model.addConstr(w[j] >= 0.0, f"w[{j}]>=0.0")

        # Interval selection constraints
        model.addGenConstrAnd(I[j], [Is[j], Ie[j]], f"selection_constraint-{j}")

        model.addGenConstrIndicator(Is[j], True, s <= j)
        model.addGenConstrIndicator(Ie[j], True, e >= j)

        model.addGenConstrIndicator(Is[j], False, s >= j + 1)
        model.addGenConstrIndicator(Ie[j], False, e <= j - 1)

    model.optimize()
    # model.write("out.lp")
    optimal_value = 2 * model.getObjective().getValue() # optimal ef1 score
    return int(round(s.X)), int(round(e.X)), optimal_value

```

## 6- Example

```python
text = ['A', 'Research', 'Fellow', 'position', 'is', 'available', 'in', 'Machine', 'Learning',
        'at', 'the', 'School', 'of', 'Computer', 'Science', 'of',
        'the', 'University', 'of', 'Birmingham', 'UK', '.',
        'The', 'appointment', 'is', 'in', 'connection', 'with', 'the', 'EPSRC', 'Fellowship', '.'
        ]

# question: in which university is the research position available?

probabilities = [0.5, 0.001, 0.001, 0.8, 0.005, 0.01, 0.005, 0.01, 0.005,
                 0.001, 0.91, 0.5, 0.01, 0.099, 0.099, 0.05,
                 0.0001, 0.5, 0.9, 0.99, 0.8, 0.7,
                 0.03, 0.1, 0.01, 0.001, 0.1, 0.01, 0.001, 0.1, 0.99, 0.0099]

```

```python
x = list(j for j in range(len(probabilities)))
plt.figure(figsize=(20, 3))
plt.bar(x, probabilities)  # A bar chart
plt.xlabel('tokens')
plt.ylabel('Probability')
plt.xticks(x, text, rotation='45')
plt.show()

```

```python
s,e,_ = select_LP_optimal_subsequence(probabilities)
```

    Academic license - for non-commercial use only - expires 2021-12-16
    Using license file /home/reda/gurobi.lic

```python
for i in range(s,e+1):
    print(text[i], end =" ")
```

    University of Birmingham UK .


+ This notebook is part of lecture 8 *Solving Ax=b, row reduced form* in the OCW MIT course 18.06 by Prof Gilbert Strang [1]
+ Created by me, Dr Juan H Klopper
    + Head of Acute Care Surgery
    + Groote Schuur Hospital
    + University Cape Town
    + Email me with your thoughts, comments, suggestions and corrections

Linear Algebra OCW MIT18.06 IPython notebook [2] study notes by Dr Juan H Klopper is licensed under a Creative Commons Attribution-NonCommercial 4.0 International License.

+ [1] OCW MIT 18.06
+ [2] Fernando Pérez, Brian E. Granger, IPython: A System for Interactive Scientific Computing, Computing in Science and Engineering, vol. 9, no. 3, pp. 21-29, May/June 2007, doi:10.1109/MCSE.2007.53. URL: http://ipython.org

```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
#import numpy as np
from sympy import init_printing, Matrix, symbols
#import matplotlib.pyplot as plt
#import seaborn as sns
#from IPython.display import Image
from warnings import filterwarnings

init_printing(use_latex = 'mathjax')
%matplotlib inline
filterwarnings('ignore')
```

# Solving A**x**=**b**

* We now want to solve the set of linear equations in the previous lecture (I_8) for any right-hand side **b**
$$ { x }_{ 1 }+2{ x }_{ 2 }+2{ x }_{ 3 }+2{ x }_{ 4 }={ b }_{ 1 }\\ 2{ x }_{ 1 }+4{ x }_{ 2 }+6{ x }_{ 3 }+8{ x }_{ 4 }={ b }_{ 2 }\\ 3{ x }_{ 1 }+6{ x }_{ 2 }+8{ x }_{ 3 }+10{ x }_{ 4 }={ b }_{ 3 }\\ \begin{bmatrix} 1 & 2 & 2 & 2 \\ 2 & 4 & 6 & 8 \\ 3 & 6 & 8 & 10 \end{bmatrix}\begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ { x }_{ 3 } \\ { x }_{ 4 } \end{bmatrix}=\begin{bmatrix} { b }_{ 1 } \\ { b }_{ 2 } \\ { b }_{ 3 } \end{bmatrix}\\ { x }_{ 1 }\begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix}+{ x }_{ 2 }\begin{bmatrix} 2 \\ 4 \\ 6 \end{bmatrix}+{ x }_{ 3 }\begin{bmatrix} 2 \\ 6 \\ 8 \end{bmatrix}+{ x }_{ 4 }\begin{bmatrix} 2 \\ 8 \\ 10 \end{bmatrix}=\begin{bmatrix} { b }_{ 1 } \\ { b }_{ 2 } \\ { b }_{ 3 } \end{bmatrix} $$

* Since the third row is the addition of rows 1 and 2, the only solution in the right-hand side must be
$$ {b}_{3} = {b}_{1} + {b}_{2} $$

```python
A = Matrix([[1, 2, 2, 2], [2, 4, 6, 8], [3, 6, 8, 10]])
A
```

$$\left[\begin{matrix}1 & 2 & 2 & 2\\2 & 4 & 6 & 8\\3 & 6 & 8 & 10\end{matrix}\right]$$

```python
A.nullspace()
```

$$\begin{bmatrix}\left[\begin{matrix}-2\\1\\0\\0\end{matrix}\right], & \left[\begin{matrix}2\\0\\-2\\1\end{matrix}\right]\end{bmatrix}$$

### So, for which right-hand sides is this set solvable?

* A**x**=**b** is solvable (exactly) when **b** is in the column space of A
    * **b** must be a linear combination of the columns

## Finding the complete solution to the set

### The particular solution: **x**particular

* To find **x**particular
    * Set all free variable to zero (in this example case that would be x2 = x4 = 0, since these are **columns without pivots**)
    * Solve A**x**=**b** for the pivot variables

* So, let's see that in action and calculate **x**particular for
$$ \overline { b } =\begin{bmatrix} 1 \\ 5 \\ 6 \end{bmatrix} $$

```python
A_augm = Matrix([[1, 2, 2, 2, 1], [2, 4, 6, 8, 5], [3, 6, 8, 10, 6]])
A_augm
```

$$\left[\begin{matrix}1 & 2 & 2 & 2 & 1\\2 & 4 & 6 & 8 & 5\\3 & 6 & 8 & 10 & 6\end{matrix}\right]$$

```python
A_augm.rref() # The video example is not solved to reduced row echelon form
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 2 & 0 & -2 & -2\\0 & 0 & 1 & 2 & \frac{3}{2}\\0 & 0 & 0 & 0 & 0\end{matrix}\right], & \begin{bmatrix}0, & 2\end{bmatrix}\end{pmatrix}$$

* So we are left with a new matrix (called A1 below)

```python
A1 = Matrix([[1, 0, -2] ,[0, 1, (3 / 2)] ,[0, 0, 0]]) # The video example is not solved to reduced row echelon form
A1
```

$$\left[\begin{matrix}1 & 0 & -2\\0 & 1 & 1.5\\0 & 0 & 0\end{matrix}\right]$$

* Thus we have for xparticular we have:
$$ { \overline { x }  }_{ P }=\begin{bmatrix} -2 \\ 0 \\ \frac { 3 }{ 2 }  \\ 0 \end{bmatrix} $$

### The nullspace solution: **x**nullspace

```python
A.nullspace()
```

$$\begin{bmatrix}\left[\begin{matrix}-2\\1\\0\\0\end{matrix}\right], & \left[\begin{matrix}2\\0\\-2\\1\end{matrix}\right]\end{bmatrix}$$

### The full set of solutions

* **x** = **x**particular + **x**nullspace
* This is so because:
$$ A{ \overline { x }  }_{ P }=b\\ A{ \overline { x }  }_{ N }=0\\ \therefore \quad A\left( { \overline { x }  }_{ P }+{ \overline { x }  }_{ N } \right) =b $$

* With the final solution:
$$ \overline { x } ={ \overline { x }  }_{ P }+{ \overline { x }  }_{ N }=\begin{bmatrix} -2 \\ 0 \\ \frac { 3 }{ 2 }  \\ 0 \end{bmatrix}+{ c }_{ 1 }\begin{bmatrix} -2 \\ 1 \\ 0 \\ 0 \end{bmatrix}+{ c }_{ 2 }\begin{bmatrix} -2 \\ 0 \\ -2 \\ 0 \end{bmatrix} $$

## Rank

* For any *m* × *n* matrix we have a rank (number of pivots)
* We cannot have more pivots than rows, ∴ *r* ≤ *m* and *r* ≤ *n*

### The case of full column rank, i.e. *r* = *n*

* This implies that ther are no free variables and the nullspace only has the zero vector (it is a subspace and MUST contain the zero vector)
* Thus **x** = **x**particular ONLY (if it exists)

* Consider this example with 2 columns
* One column is not a linear combination of the other and the rank will be two
* With only 2 unknowns and a rank of two, the nullspace wil only contain the zero vector

```python
A = Matrix([[1, 3], [2, 1], [6, 1], [5, 1]])
A
```

$$\left[\begin{matrix}1 & 3\\2 & 1\\6 & 1\\5 & 1\end{matrix}\right]$$

```python
A.nullspace()
```

$$\begin{bmatrix}\end{bmatrix}$$

```python
A.rref() # Just to show the reduced row echelon form
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 0\\0 & 1\\0 & 0\\0 & 0\end{matrix}\right], & \begin{bmatrix}0, & 1\end{bmatrix}\end{pmatrix}$$

### The case of full row rank, i.e. *r* = *m*

* Every row has a pivot
* For which **b** will the the set be solvable
    * For ALL **b**
* How many free variable?
    * We are left with *n* - *r* (or *n* - *m*) free variables

```python
A2 = A.transpose() # Creating a new matrix, which is the transpose of A above (just as an example)
A2
```

$$\left[\begin{matrix}1 & 2 & 6 & 5\\3 & 1 & 1 & 1\end{matrix}\right]$$

```python
A2.nullspace() # Showing the two nullspace solutions
```

$$\begin{bmatrix}\left[\begin{matrix}\frac{4}{5}\\- \frac{17}{5}\\1\\0\end{matrix}\right], & \left[\begin{matrix}\frac{3}{5}\\- \frac{14}{5}\\0\\1\end{matrix}\right]\end{bmatrix}$$

```python
A2.rref()
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 0 & - \frac{4}{5} & - \frac{3}{5}\\0 & 1 & \frac{17}{5} & \frac{14}{5}\end{matrix}\right], & \begin{bmatrix}0, & 1\end{bmatrix}\end{pmatrix}$$

### The case of full (row and column) rank, i.e. *r* = *m* = *n*

* Let's do an example

```python
A = Matrix([[1, 2], [3, 1]])
A
```

$$\left[\begin{matrix}1 & 2\\3 & 1\end{matrix}\right]$$

```python
A.nullspace() # Only the zero vector
```

$$\begin{bmatrix}\end{bmatrix}$$

```python
A.rref()
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 0\\0 & 1\end{matrix}\right], & \begin{bmatrix}0, & 1\end{bmatrix}\end{pmatrix}$$

* This is by definition an invertible matrix

* Let's look at an non-invertible matrix
* For this 2 × 2 matrix we will have one free variable (no pivot)

```python
A = Matrix([[2, 4], [3, 6]])
A
```

$$\left[\begin{matrix}2 & 4\\3 & 6\end{matrix}\right]$$

```python
A.nullspace()
```

$$\begin{bmatrix}\left[\begin{matrix}-2\\1\end{matrix}\right]\end{bmatrix}$$

```python
A.rref()
```

$$\begin{pmatrix}\left[\begin{matrix}1 & 2\\0 & 0\end{matrix}\right], & \begin{bmatrix}0\end{bmatrix}\end{pmatrix}$$

```python

```


# The scikit-learn library

Scikit-learn is probably the most used Machine Learning Library in Python. It is built on NumPy, SciPy, and matplotlib. The library offers a simple and efficient tools for data mining and data analysis. Scikit-learn offers a consistent API across different model and applications and hence is one of the best tools in Python for shallow learning algorithms.

## Modeling pipeline in scikit learn

* Loading The Data
* Training And Test Data
* Preprocessing The Data
* Create the Model
* Model Fitting
* Prediction
* Evaluate the Model's Performance
* Tune the Model

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import sklearn
```

**Add the `src` directory as one where we can import modules**

```python
import os
import sys

# add the 'src' directory as one where we can import modules
src_dir = os.path.abspath(os.path.join(os.getcwd(), os.pardir, 'src'))
sys.path.append(src_dir)
print(src_dir)
```

    /Users/jokr/Documents/workshops/python-workshop-zmk/src

```python
import helper_funcs as hf
```

## Loading The Data

Your data needs to be numeric and stored as NumPy arrays or SciPy sparse matrices. Other types that are convertible to numeric arrays, such as Pandas `DataFrame`, are also acceptable.

**Data for a Machine Learning model may look like this...**

```python
import numpy as np
X = np.random.random((10,5))
X[X < 0.3] = 0
print("Features array: {}\n".format(X.shape), X)
y = np.array(['M','M','F','F','F','M','M','F','F','F'])
print("Labels: {}\n".format(y.shape),y)
```

    Features array: (10, 5)
     [[0.91431617 0.36503539 0.60889711 0.30390331 0.34302656]
     [0.         0.83855127 0.69179491 0.         0.83479284]
     [0.4647658  0.57504806 0.37522482 0.         0.50130494]
     [0.62417367 0.         0.91645639 0.49021619 0.79609754]
     [0.8008854  0.         0.92919578 0.         0.62488513]
     [0.         0.         0.57372286 0.90971717 0.        ]
     [0.         0.63597323 0.79656206 0.         0.52509708]
     [0.55826234 0.         0.68597127 0.         0.        ]
     [0.         0.61487236 0.         0.49052894 0.8676565 ]
     [0.3419629  0.         0.86239635 0.         0.        ]]
    Labels: (10,)
     ['M' 'M' 'F' 'F' 'F' 'M' 'M' 'F' 'F' 'F']

Further scikit-learn provides a rich [dataset loading utilities](http://scikit-learn.org/stable/datasets/index.html). It comes with easy to load toy datasets, sample images and sample generators.

**Toy datasets:**

Boston house-prices dataset (regression)

    load_boston()

Iris dataset (classification)

    load_iris()

Diabetes dataset (regression)

    load_diabetes()

Digits dataset (classification)

    load_digits()

Linnerud dataset (multivariate regression)

    load_linnerud()

Wine dataset (classification)

    load_wine()

Breast cancer wisconsin dataset (classification)

    load_breast_cancer()

```python
from sklearn.datasets import load_boston
ds = load_boston()
```

```python
print(ds.DESCR)
```

    .. _boston_dataset:

    Boston house prices dataset
    ---------------------------

    **Data Set Characteristics:**

        :Number of Instances: 506

        :Number of Attributes: 13 numeric/categorical predictive. Median Value (attribute 14) is usually the target.

        :Attribute Information (in order):
            - CRIM     per capita crime rate by town
            - ZN       proportion of residential land zoned for lots over 25,000 sq.ft.
            - INDUS    proportion of non-retail business acres per town
            - CHAS     Charles River dummy variable (= 1 if tract bounds river; 0 otherwise)
            - NOX      nitric oxides concentration (parts per 10 million)
            - RM       average number of rooms per dwelling
            - AGE      proportion of owner-occupied units built prior to 1940
            - DIS      weighted distances to five Boston employment centres
            - RAD      index of accessibility to radial highways
            - TAX      full-value property-tax rate per $10,000
            - PTRATIO  pupil-teacher ratio by town
            - B        1000(Bk - 0.63)^2 where Bk is the proportion of blacks by town
            - LSTAT    % lower status of the population
            - MEDV     Median value of owner-occupied homes in $1000's

        :Missing Attribute Values: None

        :Creator: Harrison, D. and Rubinfeld, D.L.

    This is a copy of UCI ML housing dataset.
    https://archive.ics.uci.edu/ml/machine-learning-databases/housing/

    This dataset was taken from the StatLib library which is maintained at Carnegie Mellon University.

    The Boston house-price data of Harrison, D. and Rubinfeld, D.L. 'Hedonic
    prices and the demand for clean air', J. Environ. Economics & Management,
    vol.5, 81-102, 1978.   Used in Belsley, Kuh & Welsch, 'Regression diagnostics
    ...', Wiley, 1980.   N.B. Various transformations are used in the table on
    pages 244-261 of the latter.

    The Boston house-price data has been used in many machine learning papers that address regression
    problems.

    .. topic:: References

       - Belsley, Kuh & Welsch, 'Regression diagnostics: Identifying Influential Data and Sources of Collinearity', Wiley, 1980. 244-261.
       - Quinlan,R. (1993). Combining Instance-Based and Model-Based Learning. In Proceedings on the Tenth International Conference of Machine Learning, 236-243, University of Massachusetts, Amherst. Morgan Kaufmann.

```python
print("Features array: {}\n".format(ds.data.shape), ds.data)
print("Labels: {}\n".format(ds.target.shape),y)
```

    Features array: (506, 13)
     [[6.3200e-03 1.8000e+01 2.3100e+00 ... 1.5300e+01 3.9690e+02 4.9800e+00]
     [2.7310e-02 0.0000e+00 7.0700e+00 ... 1.7800e+01 3.9690e+02 9.1400e+00]
     [2.7290e-02 0.0000e+00 7.0700e+00 ... 1.7800e+01 3.9283e+02 4.0300e+00]
     ...
     [6.0760e-02 0.0000e+00 1.1930e+01 ... 2.1000e+01 3.9690e+02 5.6400e+00]
     [1.0959e-01 0.0000e+00 1.1930e+01 ... 2.1000e+01 3.9345e+02 6.4800e+00]
     [4.7410e-02 0.0000e+00 1.1930e+01 ... 2.1000e+01 3.9690e+02 7.8800e+00]]
    Labels: (506,)
     ['M' 'M' 'F' 'F' 'F' 'M' 'M' 'F' 'F' 'F']

**Sample generators:**

```python
from sklearn.datasets import make_classification
X, y = make_classification(n_samples=25, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           random_state=33)
```

```python
print("Features array: {}\n".format(X.shape), X)
print("Labels: {}\n".format(y.shape),y)
```

    Features array: (25, 2)
     [[ 0.84119533 -0.59034282]
     [ 0.30073529  1.32948683]
     [-0.35158507  0.72574001]
     [ 0.7203474  -0.57451647]
     [-1.91833439  2.08364911]
     [ 0.9030502   1.23783064]
     [-0.15302041  0.91097803]
     [ 1.44607025 -1.28411479]
     [-1.61288264 -0.64351396]
     [-0.9272653   0.24159894]
     [ 1.48322117  0.03325729]
     [-1.33804029  0.57162232]
     [-1.69935283  1.3004686 ]
     [-0.72134521  0.3439439 ]
     [ 0.2942302  -0.16300197]
     [-0.0526431   1.76285091]
     [-1.17874604 -0.62427607]
     [ 0.49636809  0.57330408]
     [ 0.93851305  0.34952606]
     [-0.16227224  0.42789673]
     [ 0.43508675 -1.94956553]
     [ 2.56176732 -1.68777497]
     [ 0.83080315  0.23559069]
     [ 1.19118673 -0.55582212]
     [-0.26748919  0.38544401]]
    Labels: (25,)
     [0 1 0 0 1 0 1 0 1 1 0 1 1 1 0 1 1 0 0 1 0 0 0 0 1]

```python
fig, ax = plt.subplots()
ax.scatter(X[:, 0], X[:, 1], marker='o', c=y, s=25, edgecolor='k');
```

## Training And Test Data

```python
X, y = make_classification(n_samples=1000, n_features=2, n_redundant=0,
                           n_informative=2, n_clusters_per_class=1,
                           weights=[0.096],
                           random_state=33)
fig, ax = plt.subplots()
ax.scatter(X[:, 0], X[:, 1], marker='o', c=y, s=25, edgecolor='k');
```

```python
np.bincount(y)
```

    array([100, 900])

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.4,
                                                    shuffle=True, random_state=42)
```

```python
print("Train set size:" , X_train.shape, "\nTest set size:", X_test.shape)
```

    Train set size: (600, 2)
    Test set size: (400, 2)

```python
print("Train label size:" , y_train.shape, "\nTest label size:", y_test.shape)
```

    Train label size: (600,)
    Test label size: (400,)

```python
pd.Series(y_train).value_counts()
```

    1    542
    0     58
    dtype: int64

```python
pd.Series(y_test).value_counts()
```

    1    358
    0     42
    dtype: int64

This `stratify` parameter makes a split so that the proportion of values in the sample produced will be the same as the proportion of values provided to parameter `stratify`.

```python
X_train, X_test, y_train, y_test = train_test_split(X, y, stratify=y,
                                                    test_size=0.4, random_state=42)

print(pd.Series(y_train).value_counts())
print(pd.Series(y_test).value_counts())
```

    1    540
    0     60
    dtype: int64
    1    360
    0     40
    dtype: int64

## Preprocessing The Data

In general, __learning algorithms benefit from standardization of the data set__. The `sklearn.preprocessing` package provides several utility functions and transformer classes to change raw feature vectors.

```python
from sklearn import preprocessing
```

```python
X_train = np.array([[ 1., -1.,  2.],
                    [ 2.,  0.,  0.],
                    [ 0.,  1., -1.]])
```

```python
scaler = preprocessing.StandardScaler()
scaler
```

    StandardScaler(copy=True, with_mean=True, with_std=True)

```python
scaler.fit(X_train)
```

    StandardScaler(copy=True, with_mean=True, with_std=True)

```python
X_train_scaled = scaler.transform(X_train)
X_train_scaled
```

    array([[ 0.        , -1.22474487,  1.33630621],
           [ 1.22474487,  0.        , -0.26726124],
           [-1.22474487,  1.22474487, -1.06904497]])

```python
print("Mean: ",np.mean(X_train_scaled, axis=0))
print("\nStandard deviation: ", np.std(X_train_scaled, axis=0))
```

    Mean:  [0. 0. 0.]

    Standard deviation:  [1. 1. 1.]

An alternative standardization is __aling features to lie between a given minimum and maximum value, often between zero and one, or so that the maximum absolute value of each feature is scaled to unit size__

```python
X_train
```

    array([[ 1., -1.,  2.],
           [ 2.,  0.,  0.],
           [ 0.,  1., -1.]])

```python
min_max_scaler = preprocessing.MinMaxScaler()
```

```python
min_max_scaler.fit(X_train)
```

    MinMaxScaler(copy=True, feature_range=(0, 1))

```python
X_train_scaled = min_max_scaler.transform(X_train)
X_train_scaled
```

    array([[0.5       , 0.        , 1.        ],
           [1.        , 0.5       , 0.33333333],
           [0.        , 1.        , 0.        ]])

```python
print("Min: ", np.min(X_train_scaled, axis=0))
print("\nMax: ", np.max(X_train_scaled, axis=0))
```

    Min:  [0. 0. 0.]

    Max:  [1. 1. 1.]

###  Encoding categorical features

Often features are not given as continuous values but categorical. For example a person could have features `"male"`, `"female"` or `"high"`, `"low"`, `"medium"`. Such features can be efficiently coded as integers, such as `0`, `1` (_nominal_)  or `2`, `0`, `1` (_ordinal_).

```python
X_train = np.array([['Male', 1, 0.76], ['Female', 3, 0.22], ['Female', 2, 0.57]])
X_train
```

    array([['Male', '1', '0.76'],
           ['Female', '3', '0.22'],
           ['Female', '2', '0.57']], dtype='<U6')

```python
X_train[:,0]
```

    array(['Male', 'Female', 'Female'], dtype='<U6')

#### Label encoding

```python
le = preprocessing.LabelEncoder()
le.fit(X_train[:,0])
```

    LabelEncoder()

```python
list(le.classes_)
```

    ['Female', 'Male']

```python
le.transform(X_train[:,0])
```

    array([1, 0, 0])

```python
X_encoded = np.copy(X_train)
X_encoded[:,0] = le.transform(X_train[:,0])
X_encoded = X_encoded.astype(float)
X_encoded
```

    array([[1.  , 1.  , 0.76],
           [0.  , 3.  , 0.22],
           [0.  , 2.  , 0.57]])

```python
le.inverse_transform(X_encoded[:,0].astype(int))
```

    array(['Male', 'Female', 'Female'], dtype='<U6')

#### One-Hot encoding

```python
enc = preprocessing.OneHotEncoder(categories='auto', )
enc.fit(X_train[:,0].reshape(-1,1))
```

    OneHotEncoder(categorical_features=None, categories='auto',
           dtype=<class 'numpy.float64'>, handle_unknown='error',
           n_values=None, sparse=True)

```python
X_train[:,0]
```

    array(['Male', 'Female', 'Female'], dtype='<U6')

```python
ohe = enc.transform(X_train[:,0].reshape(-1,1)).toarray()
ohe
```

    array([[0., 1.],
           [1., 0.],
           [1., 0.]])

### Imputation of missing values

Many real world datasets contain missing values, often encoded as blanks, `NaN`s or other placeholders. A basic strategy to use incomplete datasets is to discard entire rows and/or columns containing missing values. Another strategy is to impute the missing values, i.e., to infer them from the known part of the data.

In scikit-learn the `SimpleImputer` class provides basic strategies for imputing missing values, either using __the mean__, __the median__ or __the most frequent value__ of the row or column in which the missing values are located.

```python
from sklearn.impute import SimpleImputer
```

```python
X_train = np.array([[1, 2], [np.nan, 3], [7, 6]])
X_train
```

    array([[ 1.,  2.],
           [nan,  3.],
           [ 7.,  6.]])

```python
imp = SimpleImputer(missing_values=np.nan, strategy='mean')
imp.fit(X_train)
```

    SimpleImputer(copy=True, fill_value=None, missing_values=nan, strategy='mean',
           verbose=0)

```python
np.nanmean(X_train, axis=0)
```

    array([4.        , 3.66666667])

```python
imp.transform(X_train)
```

    array([[1., 2.],
           [4., 3.],
           [7., 6.]])

## Create the Model

Scikit-learn ships with many different supervised and unsupervised models as well as some basic neural network models; among those are:

* ### Linear models
  * Ordinary Least Squares
  * Polynomial regression
  * Ridge Regression
  * Lasso
  * Elastic Net
  * Logistic regression

* ### Support Vector Machines

* ### Nearest Neighbors

* ### Gaussian Processes

* ### Decision Trees

* ### Ensemble methods
  * Random Forest
  * Gradient Boosting

Visit the [scikit-learn documentation website](http://scikit-learn.org/stable/index.html) for a comprehensive list of avaiake models and techniques.

### Loading a toy data set

For  the sake of this tutorial we load the _wine dataset_ provided by scikit learn.

```python
from sklearn.datasets import load_wine
```

```python
ds = load_wine()
print(ds.DESCR)
```

    .. _wine_dataset:

    Wine recognition dataset
    ------------------------

    **Data Set Characteristics:**

        :Number of Instances: 178 (50 in each of three classes)
        :Number of Attributes: 13 numeric, predictive attributes and the class
        :Attribute Information:
     		- Alcohol
     		- Malic acid
     		- Ash
    		- Alcalinity of ash
     		- Magnesium
    		- Total phenols
     		- Flavanoids
     		- Nonflavanoid phenols
     		- Proanthocyanins
    		- Color intensity
     		- Hue
     		- OD280/OD315 of diluted wines
     		- Proline

        - class:
                - class_0
                - class_1
                - class_2

        :Summary Statistics:

        ============================= ==== ===== ======= =====
                                       Min   Max   Mean     SD
        ============================= ==== ===== ======= =====
        Alcohol:                      11.0  14.8    13.0   0.8
        Malic Acid:                   0.74  5.80    2.34  1.12
        Ash:                          1.36  3.23    2.36  0.27
        Alcalinity of Ash:            10.6  30.0    19.5   3.3
        Magnesium:                    70.0 162.0    99.7  14.3
        Total Phenols:                0.98  3.88    2.29  0.63
        Flavanoids:                   0.34  5.08    2.03  1.00
        Nonflavanoid Phenols:         0.13  0.66    0.36  0.12
        Proanthocyanins:              0.41  3.58    1.59  0.57
        Colour Intensity:              1.3  13.0     5.1   2.3
        Hue:                          0.48  1.71    0.96  0.23
        OD280/OD315 of diluted wines: 1.27  4.00    2.61  0.71
        Proline:                       278  1680     746   315
        ============================= ==== ===== ======= =====

        :Missing Attribute Values: None
        :Class Distribution: class_0 (59), class_1 (71), class_2 (48)
        :Creator: R.A. Fisher
        :Donor: Michael Marshall (MARSHALL%PLU@io.arc.nasa.gov)
        :Date: July, 1988

    This is a copy of UCI ML Wine recognition datasets.
    https://archive.ics.uci.edu/ml/machine-learning-databases/wine/wine.data

    The data is the results of a chemical analysis of wines grown in the same
    region in Italy by three different cultivators. There are thirteen different
    measurements taken for different constituents found in the three types of
    wine.

    Original Owners:

    Forina, M. et al, PARVUS -
    An Extendible Package for Data Exploration, Classification and Correlation.
    Institute of Pharmaceutical and Food Analysis and Technologies,
    Via Brigata Salerno, 16147 Genoa, Italy.

    Citation:

    Lichman, M. (2013). UCI Machine Learning Repository
    [http://archive.ics.uci.edu/ml]. Irvine, CA: University of California,
    School of Information and Computer Science.

    .. topic:: References

      (1) S. Aeberhard, D. Coomans and O. de Vel,
      Comparison of Classifiers in High Dimensional Settings,
      Tech. Rep. no. 92-02, (1992), Dept. of Computer Science and Dept. of
      Mathematics and Statistics, James Cook University of North Queensland.
      (Also submitted to Technometrics).

      The data was used with many others for comparing various
      classifiers. The classes are separable, though only RDA
      has achieved 100% correct classification.
      (RDA : 100%, QDA 99.4%, LDA 98.9%, 1NN 96.1% (z-transformed data))
      (All results using the leave-one-out technique)

      (2) S. Aeberhard, D. Coomans and O. de Vel,
      "THE CLASSIFICATION PERFORMANCE OF RDA"
      Tech. Rep. no. 92-01, (1992), Dept. of Computer Science and Dept. of
      Mathematics and Statistics, James Cook University of North Queensland.
      (Also submitted to Journal of Chemometrics).

```python
ds.data
```

    array([[1.423e+01, 1.710e+00, 2.430e+00, ..., 1.040e+00, 3.920e+00,
            1.065e+03],
           [1.320e+01, 1.780e+00, 2.140e+00, ..., 1.050e+00, 3.400e+00,
            1.050e+03],
           [1.316e+01, 2.360e+00, 2.670e+00, ..., 1.030e+00, 3.170e+00,
            1.185e+03],
           ...,
           [1.327e+01, 4.280e+00, 2.260e+00, ..., 5.900e-01, 1.560e+00,
            8.350e+02],
           [1.317e+01, 2.590e+00, 2.370e+00, ..., 6.000e-01, 1.620e+00,
            8.400e+02],
           [1.413e+01, 4.100e+00, 2.740e+00, ..., 6.100e-01, 1.600e+00,
            5.600e+02]])

```python
ds.target
```

    array([0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1,
           1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2, 2,
           2, 2])

### Subsetting the data set to two classes

```python
y = ds.target[ds.target < 2]
y.shape
```

    (130,)

```python
X = ds.data[ds.target < 2]
X.shape
```

    (130, 13)

### Train-test split

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.3,
                                                    random_state=42)
```

```python
print(X_train.shape)
print(X_test.shape)
print(y_train.shape)
print(y_test.shape)
```

    (91, 13)
    (39, 13)
    (91,)
    (39,)

### Creation of a model instance

For the purpose of this tutorial we introduce [**Logistic regression**](https://en.wikipedia.org/wiki/Logistic_regression), also known as **logit regression**, or **logit model**, a probabilistic linear model for dichotomous data. The response variable is a binary variable (nominal variable), which means the variable has two categories or two values; Class 1 vs. Class 2, True vs. False, or $1$ vs. $0$, or success vs. failure, with the probabilities of $\pi$ and $1-\pi$, respectively.

The output of a logistic regression is a probability $(\pi)$, thus a value between $0$ and $1$. Moreover, this output is a linear function of known covariates $x_i$, which is just another word for the features in our data set.
$$\pi =\beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k$$

However, the right term of the equation can take any real value, whereas the left term of the equation is a probability, on the scale $0$ to $1$. In order to transform the scale of the data (right term) into a probability between $0$ and $1$ we apply a so-called **link function**.

For the logistic regression model this link function is the [**logit function**](https://en.wikipedia.org/wiki/Logit). The logit function maps probabilities from the range $(0, 1)$ to the entire real number range $(-\infty, \infty)$. It is written as

$$\eta = logit(\pi)\text{,}$$

where $\pi$ is the probability.

To understand the logit we first introduce the [**odds ratio**](https://en.wikipedia.org/wiki/Odds_ratio) or in short **odds**. The odds (o) can be written as

$$o = \frac{\pi}{1-\pi}\text{,}$$

where $\pi$ is the probability that an event occurs. If the probability of an event is a $0.5$, the odds are one-to-one or even $\left(\frac{0.5}{1-0.5}=1\right)$. We further define the or **log-odds**, which is the logarithm of the odds:
$$\eta = logit(\pi)= log \left( \frac{\pi}{1-\pi}\right)$$

This logarithmic function has the effect of removing the floor restriction, thus the function, the [**logit function**](https://en.wikipedia.org/wiki/Logit), our link function, transforms values in the range $0$ to $1$ to values over the entire real number range $(-\infty, \infty)$. If the probability is $1/2$ the odds are even and the logit is zero. Negative logits represent probabilities below one half and positive logits correspond to probabilities above one half.

The inverse form of the logit function is also called the [logistic function](https://en.wikipedia.org/wiki/Logistic_function), sometimes simply abbreviated as [**sigmoid function**](https://en.wikipedia.org/wiki/Sigmoid_function) due to its characteristic S-shape. Is allows us to go back from logits to probabilities.

$$\pi =logit^{-1}(\eta)= \frac{e^{\eta}}{1+e^{\eta}}=\frac{1}{1+e^{-\eta}}=\frac{1}{1+e^{-\beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k}}$$

The logistic function for the interval $[-6,6]$ is shown below. For values of $\eta$ in the range from $-\infty$ to $\infty$ $\pi$ is in the range of $0$ to $1$.

```python
def logit(interval=[-6,6]):
    x = np.linspace(interval[0], interval[1])
    y = 1 / (1 + np.exp(-x))
    return (x,y)

fig, ax = plt.subplots(figsize=(8,4))
ax.plot(logit()[0], logit()[1])
ax.axvline(0, color="k", linestyle="dashed", linewidth=0.5)
ax.axhline(0.5, color="k", linestyle="dashed", linewidth=0.5)
ax.set_title("The logistic function", size=18);
```

The logit function maps probabilities to values over the entire real number range. Thus, the probability of an event/outcome/success to be true $(y=1)$, given the set of predictors $x_i$, which is our data, is written as

$$logit(P(y=1|x_i))= \beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k\text{,}$$
For a matter of simplification we express the inverse of the function above as

$$\phi(\eta) = \frac{1}{1+e^{-\eta}}\text{,}$$

where $\eta$ is the linear combination of coefficients $(\beta_i)$ and predictor variables $(x_i)$, calculated as $\eta = \beta_0+ \beta_1x_1+ \beta_2x_2+ ... +\beta_kx_k$.

The parameters $(\beta_i)$ of the logit model are estimated by the [**method of maximum likelihood**](https://en.wikipedia.org/wiki/Maximum_likelihood_estimation). However, there is no closed-form solution, so the maximum likelihood estimates are obtained by using iterative algorithms such as [gradient descent](https://en.wikipedia.org/wiki/Gradient_descent), among others.

The output of the sigmoid function is interpreted as the probability of a particular observation belonging to class 1. It is written as  $\phi(\eta)=P(y=1|x_i,\beta_i)$, the probability of success $(y=1)$ given the predictor variables $x_i$ parameterized by the coefficients $\beta_i$. For example, if we compute $\phi(\eta)=0.65$ for a particular observation, this means that the chance that this observation belongs to class 1 is 65%. Similarly, the probability that this observation belongs to class 2 is calculated as $\phi(\eta)=P(y=0|x_i,\beta_i)= 1 - P(y=1|x_i,\beta_i)=1-0.65=0.35$ or 35%. For class assignment the predicted probability is then converted into a binary outcome via a unit step function:

$$
\hat y =
\begin{cases}
1,  & \text{if $\phi(\eta) \ge$ 0.5} \\
0, & \text{otherwise}
\end{cases}
$$

```python
from sklearn import linear_model
logistic = linear_model.LogisticRegression(solver='lbfgs', max_iter=5000)
logistic
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=5000, multi_class='warn',
              n_jobs=None, penalty='l2', random_state=None, solver='lbfgs',
              tol=0.0001, verbose=0, warm_start=False)

## Model Fitting

```python
logistic.fit(X_train, y_train,)
```

    LogisticRegression(C=1.0, class_weight=None, dual=False, fit_intercept=True,
              intercept_scaling=1, max_iter=5000, multi_class='warn',
              n_jobs=None, penalty='l2', random_state=None, solver='lbfgs',
              tol=0.0001, verbose=0, warm_start=False)

## Prediction

### In-sample prediction

```python
logistic.predict(X_train)
```

    array([0, 0, 1, 1, 0, 1, 1, 0, 0, 1, 1, 1, 0, 1, 1, 0, 0, 1, 1, 0, 0, 0,
           0, 1, 0, 1, 0, 0, 1, 1, 1, 0, 0, 1, 0, 0, 0, 1, 1, 0, 1, 1, 1, 0,
           0, 1, 0, 1, 1, 1, 1, 1, 1, 0, 0, 1, 0, 1, 1, 0, 1, 0, 1, 1, 1, 1,
           0, 0, 1, 1, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1,
           0, 1, 1])

### Out of sample prediction

```python
y_pred = logistic.predict(X_test)
y_pred
```

    array([0, 0, 0, 0, 1, 0, 0, 1, 1, 0, 0, 0, 1, 0, 1, 1, 0, 1, 1, 1, 0, 0,
           1, 0, 1, 0, 1, 0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1])

```python
logistic.predict_proba(X_test)[:5]
```

    array([[9.99468532e-01, 5.31467720e-04],
           [9.87549463e-01, 1.24505369e-02],
           [9.98521420e-01, 1.47858024e-03],
           [9.99999623e-01, 3.77247327e-07],
           [7.94044520e-06, 9.99992060e-01]])

##  Evaluate the Model's Performance

In scikit-learn there are 3 different APIs for evaluating the quality of a model’s predictions:

* __Estimator score method__: Estimators have a score method providing a default evaluation criterion for the problem they are designed to solve.
* __Scoring parameter__: Model-evaluation tools using cross-validation rely on an internal scoring strategy.
* __Metric functions__: The metrics module implements functions assessing prediction error for specific purposes. Functions ending with `_score` return a value to maximize, the higher the better, and Functions ending with `_error` or `_loss` return a value to minimize, the lower the better.

### Estimator score method

Returns the mean accuracy on the given test data and labels.

$$\text{accuracy}(y, \hat y) = \frac{1}{n_{\text{samples}}} \sum_{i=0}^{n_{\text{samples}}-1}\mathbb1(\hat y_i = y_i)$$

```python
logistic.score(X_test, y_test)
```

    0.9230769230769231

```python
np.sum(y_pred == y_test)/len(y_test)
```

    0.9230769230769231

### Metric functions

```python
from sklearn.metrics import accuracy_score
accuracy_score(y_pred, y_test)
```

    0.9230769230769231

Note that there are many more [model metrics](http://scikit-learn.org/stable/modules/model_evaluation.html) available in scikit-learn.

```python
dir(sklearn.metrics)[11:]
```

    ['accuracy_score',
     'adjusted_mutual_info_score',
     'adjusted_rand_score',
     'auc',
     'average_precision_score',
     'balanced_accuracy_score',
     'base',
     'brier_score_loss',
     'calinski_harabaz_score',
     'check_scoring',
     'classification',
     'classification_report',
     'cluster',
     'cohen_kappa_score',
     'completeness_score',
     'confusion_matrix',
     'consensus_score',
     'coverage_error',
     'davies_bouldin_score',
     'euclidean_distances',
     'explained_variance_score',
     'f1_score',
     'fbeta_score',
     'fowlkes_mallows_score',
     'get_scorer',
     'hamming_loss',
     'hinge_loss',
     'homogeneity_completeness_v_measure',
     'homogeneity_score',
     'jaccard_similarity_score',
     'label_ranking_average_precision_score',
     'label_ranking_loss',
     'log_loss',
     'make_scorer',
     'matthews_corrcoef',
     'mean_absolute_error',
     'mean_squared_error',
     'mean_squared_log_error',
     'median_absolute_error',
     'mutual_info_score',
     'normalized_mutual_info_score',
     'pairwise',
     'pairwise_distances',
     'pairwise_distances_argmin',
     'pairwise_distances_argmin_min',
     'pairwise_distances_chunked',
     'pairwise_fast',
     'pairwise_kernels',
     'precision_recall_curve',
     'precision_recall_fscore_support',
     'precision_score',
     'r2_score',
     'ranking',
     'recall_score',
     'regression',
     'roc_auc_score',
     'roc_curve',
     'scorer',
     'silhouette_samples',
     'silhouette_score',
     'v_measure_score',
     'zero_one_loss']

## Tune the Model

Be aware that there are two types of parameters:
* __model parameters__ and
* __hyperparameters__.

Models parameters (i.e. regression coefficients) are learned from the data. Hyperparameters however, are parameters whose values are set before the learning process begins. Different model training algorithms require different hyperparameters, some simple algorithms  require none.

The optimal hyperparameter configuration for a particular modeling task is unknown. Hence, we apply different techniques, such as grid search, random search or Baysian optimization to approximate the best hyperparamter configuration (referred to as [hyperparameter optimization](https://en.wikipedia.org/wiki/Hyperparameter_optimization).

During the process of model optimization we want to avoid [overfitting](https://en.wikipedia.org/wiki/Overfitting). Hence, in machine learning a technique referred to as [k-fold cross-validation][1] is applied. Cross-validation is a process for reliably estimating the performance of a method for building a model by training and evaluating your model multiple times using the same method.

#### K-fold cross-validation (Source: [Wikipedia][1])

[1]: https://en.wikipedia.org/wiki/Cross-validation_(statistics)

### Model creation

For the purpose of this tutor ail we apply [support vector machines]() (SVMs) for classification of the subseted _wine data set_ (see above). An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible.

The advantages of support vector machines are:

* Effective in high dimensional spaces.
* Still effective in cases where number of dimensions is greater than the number of samples.
* Uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.
* Versatile: different Kernel functions can be specified for the decision function.

The disadvantages of support vector machines include:

* If the number of features is much greater than the number of samples, avoid over-fitting in choosing Kernel functions and regularization term is crucial.
* SVMs do not directly provide probability estimates.
* SVM do not scale well.

The mathematics of the algorithm are beyond the scope of this tutorial, however if you are interested we suggest to dive into the [scikit-learn documentation](http://scikit-learn.org/stable/modules/svm.html) or watch the informative [video](https://www.youtube.com/watch?v=-Z4aojJ-pdg) by Brandon Rohrer.

```python
from sklearn import svm
svc = svm.SVC()
svc
```

    SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto_deprecated',
      kernel='rbf', max_iter=-1, probability=False, random_state=None,
      shrinking=True, tol=0.001, verbose=False)

### Hyperparameter optimization using `GridSearchCV`

* `C`: Penalty parameter `C` of the error term. The parameter allows one to trade off training error vs. model complexity. Hence, the `C` parameter trades off misclassification of training examples against simplicity of the decision surface. A low `C` makes the decision surface smooth, while a high `C` aims at classifying all training examples correctly by giving the model freedom to select more samples as support vectors.

* `kernel`:  Specifies the kernel type to be used in the algorithm. It must be one of `linear`, `poly`, `rbf`, `sigmoid`, `precomputed` or a callable.

* `gamma`: Kernel coefficient for `rbf`, `poly` and `sigmoid`. Intuitively, the `gamma` parameter defines how far the influence of a single training example reaches, with low values meaning _far_ and high values meaning _close_. The `gamma` parameters can be seen as the inverse of the radius of influence of samples selected by the model as support vectors.

```python
from sklearn.model_selection import GridSearchCV

param_grid = [{'C': [0.1, 1, 10, 100],
               'gamma': [0.01, 0.001, 0.0001],
               'kernel': ['linear', 'rbf']}]

clf = GridSearchCV(svc, param_grid, cv=5, verbose=1, return_train_score=True)
```

### Model fitting

```python
clf.fit(X_train, y_train)
```

    Fitting 5 folds for each of 24 candidates, totalling 120 fits

    [Parallel(n_jobs=1)]: Using backend SequentialBackend with 1 concurrent workers.
    [Parallel(n_jobs=1)]: Done 120 out of 120 | elapsed:    2.3s finished
    /Users/jokr/miniconda3/envs/python-charite/lib/python3.6/site-packages/sklearn/model_selection/_search.py:841: DeprecationWarning: The default of the `iid` parameter will change from True to False in version 0.22 and will be removed in 0.24. This will change numeric results when test-set sizes are unequal.
      DeprecationWarning)

    GridSearchCV(cv=5, error_score='raise-deprecating',
           estimator=SVC(C=1.0, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma='auto_deprecated',
      kernel='rbf', max_iter=-1, probability=False, random_state=None,
      shrinking=True, tol=0.001, verbose=False),
           fit_params=None, iid='warn', n_jobs=None,
           param_grid=[{'C': [0.1, 1, 10, 100], 'gamma': [0.01, 0.001, 0.0001], 'kernel': ['linear', 'rbf']}],
           pre_dispatch='2*n_jobs', refit=True, return_train_score=True,
           scoring=None, verbose=1)

### Cross validation results

```python
pd.DataFrame(clf.cv_results_ )
```

<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>mean_fit_time</th>
      <th>std_fit_time</th>
      <th>mean_score_time</th>
      <th>std_score_time</th>
      <th>param_C</th>
      <th>param_gamma</th>
      <th>param_kernel</th>
      <th>params</th>
      <th>split0_test_score</th>
      <th>split1_test_score</th>
      <th>...</th>
      <th>mean_test_score</th>
      <th>std_test_score</th>
      <th>rank_test_score</th>
      <th>split0_train_score</th>
      <th>split1_train_score</th>
      <th>split2_train_score</th>
      <th>split3_train_score</th>
      <th>split4_train_score</th>
      <th>mean_train_score</th>
      <th>std_train_score</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>0.008750</td>
      <td>0.003003</td>
      <td>0.000372</td>
      <td>0.000057</td>
      <td>0.1</td>
      <td>0.01</td>
      <td>linear</td>
      <td>{'C': 0.1, 'gamma': 0.01, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>0.986111</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>0.986486</td>
      <td>0.989040</td>
      <td>0.005481</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.000474</td>
      <td>0.000084</td>
      <td>0.000270</td>
      <td>0.000047</td>
      <td>0.1</td>
      <td>0.01</td>
      <td>rbf</td>
      <td>{'C': 0.1, 'gamma': 0.01, 'kernel': 'rbf'}</td>
      <td>0.578947</td>
      <td>0.578947</td>
      <td>...</td>
      <td>0.571429</td>
      <td>0.013268</td>
      <td>24</td>
      <td>0.569444</td>
      <td>0.569444</td>
      <td>0.575342</td>
      <td>0.575342</td>
      <td>0.567568</td>
      <td>0.571428</td>
      <td>0.003269</td>
    </tr>
    <tr>
      <th>2</th>
      <td>0.005933</td>
      <td>0.003563</td>
      <td>0.000298</td>
      <td>0.000092</td>
      <td>0.1</td>
      <td>0.001</td>
      <td>linear</td>
      <td>{'C': 0.1, 'gamma': 0.001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>0.986111</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>0.986486</td>
      <td>0.989040</td>
      <td>0.005481</td>
    </tr>
    <tr>
      <th>3</th>
      <td>0.000396</td>
      <td>0.000013</td>
      <td>0.000236</td>
      <td>0.000028</td>
      <td>0.1</td>
      <td>0.001</td>
      <td>rbf</td>
      <td>{'C': 0.1, 'gamma': 0.001, 'kernel': 'rbf'}</td>
      <td>0.684211</td>
      <td>0.578947</td>
      <td>...</td>
      <td>0.615385</td>
      <td>0.046533</td>
      <td>23</td>
      <td>0.652778</td>
      <td>0.694444</td>
      <td>0.589041</td>
      <td>0.575342</td>
      <td>0.648649</td>
      <td>0.632051</td>
      <td>0.043963</td>
    </tr>
    <tr>
      <th>4</th>
      <td>0.003781</td>
      <td>0.001395</td>
      <td>0.000205</td>
      <td>0.000011</td>
      <td>0.1</td>
      <td>0.0001</td>
      <td>linear</td>
      <td>{'C': 0.1, 'gamma': 0.0001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>0.986111</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>0.986486</td>
      <td>0.989040</td>
      <td>0.005481</td>
    </tr>
    <tr>
      <th>5</th>
      <td>0.000375</td>
      <td>0.000008</td>
      <td>0.000220</td>
      <td>0.000009</td>
      <td>0.1</td>
      <td>0.0001</td>
      <td>rbf</td>
      <td>{'C': 0.1, 'gamma': 0.0001, 'kernel': 'rbf'}</td>
      <td>0.894737</td>
      <td>0.894737</td>
      <td>...</td>
      <td>0.934066</td>
      <td>0.053529</td>
      <td>14</td>
      <td>0.972222</td>
      <td>0.944444</td>
      <td>0.945205</td>
      <td>0.958904</td>
      <td>0.959459</td>
      <td>0.956047</td>
      <td>0.010331</td>
    </tr>
    <tr>
      <th>6</th>
      <td>0.016273</td>
      <td>0.005843</td>
      <td>0.000256</td>
      <td>0.000065</td>
      <td>1</td>
      <td>0.01</td>
      <td>linear</td>
      <td>{'C': 1, 'gamma': 0.01, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>0.986111</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>0.986486</td>
      <td>0.989040</td>
      <td>0.005481</td>
    </tr>
    <tr>
      <th>7</th>
      <td>0.000452</td>
      <td>0.000014</td>
      <td>0.000224</td>
      <td>0.000005</td>
      <td>1</td>
      <td>0.01</td>
      <td>rbf</td>
      <td>{'C': 1, 'gamma': 0.01, 'kernel': 'rbf'}</td>
      <td>0.789474</td>
      <td>0.684211</td>
      <td>...</td>
      <td>0.747253</td>
      <td>0.068756</td>
      <td>22</td>
      <td>0.986111</td>
      <td>1.000000</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>1.000000</td>
      <td>0.991743</td>
      <td>0.006742</td>
    </tr>
    <tr>
      <th>8</th>
      <td>0.018126</td>
      <td>0.004601</td>
      <td>0.000330</td>
      <td>0.000057</td>
      <td>1</td>
      <td>0.001</td>
      <td>linear</td>
      <td>{'C': 1, 'gamma': 0.001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>0.986111</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>0.986486</td>
      <td>0.989040</td>
      <td>0.005481</td>
    </tr>
    <tr>
      <th>9</th>
      <td>0.000436</td>
      <td>0.000015</td>
      <td>0.000214</td>
      <td>0.000003</td>
      <td>1</td>
      <td>0.001</td>
      <td>rbf</td>
      <td>{'C': 1, 'gamma': 0.001, 'kernel': 'rbf'}</td>
      <td>0.842105</td>
      <td>0.947368</td>
      <td>...</td>
      <td>0.934066</td>
      <td>0.063420</td>
      <td>14</td>
      <td>0.986111</td>
      <td>0.986111</td>
      <td>0.958904</td>
      <td>0.958904</td>
      <td>0.986486</td>
      <td>0.975303</td>
      <td>0.013391</td>
    </tr>
    <tr>
      <th>10</th>
      <td>0.014878</td>
      <td>0.004028</td>
      <td>0.000205</td>
      <td>0.000005</td>
      <td>1</td>
      <td>0.0001</td>
      <td>linear</td>
      <td>{'C': 1, 'gamma': 0.0001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>0.986111</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>0.986486</td>
      <td>0.989040</td>
      <td>0.005481</td>
    </tr>
    <tr>
      <th>11</th>
      <td>0.000392</td>
      <td>0.000112</td>
      <td>0.000226</td>
      <td>0.000050</td>
      <td>1</td>
      <td>0.0001</td>
      <td>rbf</td>
      <td>{'C': 1, 'gamma': 0.0001, 'kernel': 'rbf'}</td>
      <td>0.894737</td>
      <td>0.947368</td>
      <td>...</td>
      <td>0.956044</td>
      <td>0.040076</td>
      <td>13</td>
      <td>0.972222</td>
      <td>0.958333</td>
      <td>0.945205</td>
      <td>0.945205</td>
      <td>0.972973</td>
      <td>0.958788</td>
      <td>0.012255</td>
    </tr>
    <tr>
      <th>12</th>
      <td>0.061374</td>
      <td>0.028955</td>
      <td>0.000252</td>
      <td>0.000054</td>
      <td>10</td>
      <td>0.01</td>
      <td>linear</td>
      <td>{'C': 10, 'gamma': 0.01, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>13</th>
      <td>0.000458</td>
      <td>0.000016</td>
      <td>0.000234</td>
      <td>0.000026</td>
      <td>10</td>
      <td>0.01</td>
      <td>rbf</td>
      <td>{'C': 10, 'gamma': 0.01, 'kernel': 'rbf'}</td>
      <td>0.789474</td>
      <td>0.736842</td>
      <td>...</td>
      <td>0.769231</td>
      <td>0.111854</td>
      <td>20</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>14</th>
      <td>0.062221</td>
      <td>0.029089</td>
      <td>0.000261</td>
      <td>0.000041</td>
      <td>10</td>
      <td>0.001</td>
      <td>linear</td>
      <td>{'C': 10, 'gamma': 0.001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>15</th>
      <td>0.000541</td>
      <td>0.000068</td>
      <td>0.000236</td>
      <td>0.000026</td>
      <td>10</td>
      <td>0.001</td>
      <td>rbf</td>
      <td>{'C': 10, 'gamma': 0.001, 'kernel': 'rbf'}</td>
      <td>0.842105</td>
      <td>0.894737</td>
      <td>...</td>
      <td>0.923077</td>
      <td>0.064709</td>
      <td>17</td>
      <td>0.986111</td>
      <td>1.000000</td>
      <td>0.986301</td>
      <td>0.986301</td>
      <td>1.000000</td>
      <td>0.991743</td>
      <td>0.006742</td>
    </tr>
    <tr>
      <th>16</th>
      <td>0.063120</td>
      <td>0.029306</td>
      <td>0.000253</td>
      <td>0.000029</td>
      <td>10</td>
      <td>0.0001</td>
      <td>linear</td>
      <td>{'C': 10, 'gamma': 0.0001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>17</th>
      <td>0.000329</td>
      <td>0.000012</td>
      <td>0.000202</td>
      <td>0.000004</td>
      <td>10</td>
      <td>0.0001</td>
      <td>rbf</td>
      <td>{'C': 10, 'gamma': 0.0001, 'kernel': 'rbf'}</td>
      <td>0.789474</td>
      <td>0.947368</td>
      <td>...</td>
      <td>0.934066</td>
      <td>0.078303</td>
      <td>14</td>
      <td>0.986111</td>
      <td>0.972222</td>
      <td>0.958904</td>
      <td>0.958904</td>
      <td>0.972973</td>
      <td>0.969823</td>
      <td>0.010192</td>
    </tr>
    <tr>
      <th>18</th>
      <td>0.061695</td>
      <td>0.029112</td>
      <td>0.000238</td>
      <td>0.000027</td>
      <td>100</td>
      <td>0.01</td>
      <td>linear</td>
      <td>{'C': 100, 'gamma': 0.01, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>19</th>
      <td>0.000466</td>
      <td>0.000034</td>
      <td>0.000239</td>
      <td>0.000030</td>
      <td>100</td>
      <td>0.01</td>
      <td>rbf</td>
      <td>{'C': 100, 'gamma': 0.01, 'kernel': 'rbf'}</td>
      <td>0.789474</td>
      <td>0.736842</td>
      <td>...</td>
      <td>0.769231</td>
      <td>0.111854</td>
      <td>20</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>20</th>
      <td>0.061475</td>
      <td>0.028800</td>
      <td>0.000235</td>
      <td>0.000022</td>
      <td>100</td>
      <td>0.001</td>
      <td>linear</td>
      <td>{'C': 100, 'gamma': 0.001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>21</th>
      <td>0.000448</td>
      <td>0.000018</td>
      <td>0.000218</td>
      <td>0.000008</td>
      <td>100</td>
      <td>0.001</td>
      <td>rbf</td>
      <td>{'C': 100, 'gamma': 0.001, 'kernel': 'rbf'}</td>
      <td>0.789474</td>
      <td>0.894737</td>
      <td>...</td>
      <td>0.901099</td>
      <td>0.087199</td>
      <td>19</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>22</th>
      <td>0.061837</td>
      <td>0.029083</td>
      <td>0.000249</td>
      <td>0.000022</td>
      <td>100</td>
      <td>0.0001</td>
      <td>linear</td>
      <td>{'C': 100, 'gamma': 0.0001, 'kernel': 'linear'}</td>
      <td>0.947368</td>
      <td>1.000000</td>
      <td>...</td>
      <td>0.989011</td>
      <td>0.021392</td>
      <td>1</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>1.000000</td>
      <td>0.000000</td>
    </tr>
    <tr>
      <th>23</th>
      <td>0.000346</td>
      <td>0.000039</td>
      <td>0.000206</td>
      <td>0.000008</td>
      <td>100</td>
      <td>0.0001</td>
      <td>rbf</td>
      <td>{'C': 100, 'gamma': 0.0001, 'kernel': 'rbf'}</td>
      <td>0.789474</td>
      <td>0.894737</td>
      <td>...</td>
      <td>0.912088</td>
      <td>0.080153</td>
      <td>18</td>
      <td>0.986111</td>
      <td>0.986111</td>
      <td>0.972603</td>
      <td>0.972603</td>
      <td>0.986486</td>
      <td>0.980783</td>
      <td>0.006680</td>
    </tr>
  </tbody>
</table>
<p>24 rows × 23 columns</p>
</div>

`best_params_`: Parameter setting that gave the best results on the hold out data.

```python
clf.best_params_
```

    {'C': 0.1, 'gamma': 0.01, 'kernel': 'linear'}

`best_estimator_`: Estimator that was chosen by the search, i.e. estimator which gave highest score (or smallest loss if specified) on the left out data.

```python
clf.best_estimator_
```

    SVC(C=0.1, cache_size=200, class_weight=None, coef0=0.0,
      decision_function_shape='ovr', degree=3, gamma=0.01, kernel='linear',
      max_iter=-1, probability=False, random_state=None, shrinking=True,
      tol=0.001, verbose=False)

`best_score_`: Mean cross-validated score of the best_estimator

```python
clf.best_score_
```

    0.989010989010989

### Model preditction

```python
y_pred = clf.best_estimator_.predict(X_test)
y_pred
```

    array([0, 0, 0, 0, 1, 0, 1, 1, 1, 0, 1, 0, 1, 1, 1, 1, 0, 1, 1, 1, 0, 0,
           1, 0, 1, 0, 1, 1, 0, 0, 0, 0, 0, 0, 1, 1, 1, 0, 1])

### Model evaluation

**Accuracy**

```python
from sklearn.metrics import accuracy_score
accuracy_score(y_pred, y_test)
```

    0.9743589743589743

**Confusion matrix**

```python
from sklearn.metrics import confusion_matrix
cnf_matrix = confusion_matrix(y_test, y_pred)
cnf_matrix
```

    array([[19,  1],
           [ 0, 19]])

```python
# Plot non-normalized confusion matrix
hf.plot_confusion_matrix(cnf_matrix, classes=["class 0", "class 1"],
                      title='Confusion matrix, without normalization')
```

**Selected classification metrics**

\begin{equation}
\text{precision} = \frac{TP}{TP+FP}
\end{equation}

\begin{equation}
\text{recall} = \frac{TP}{TP+FN}
\end{equation}

\begin{equation}
\text{F1-score} = 2 \times \frac{\text{precision} \times \text{recall}}{\text{precision}+\text{recall}}
\end{equation}

```python
from sklearn.metrics import classification_report
print(classification_report(y_test, y_pred))
```

                  precision    recall  f1-score   support

               0       1.00      0.95      0.97        20
               1       0.95      1.00      0.97        19

       micro avg       0.97      0.97      0.97        39
       macro avg       0.97      0.97      0.97        39
    weighted avg       0.98      0.97      0.97        39