```python
### First imports and default parameters

%matplotlib inline

import warnings
warnings.filterwarnings("ignore")

import numpy as np
import matplotlib
import matplotlib.pyplot as plt

# Overwritting matplotlib default linestyle of negative contours
matplotlib.rcParams['contour.negative_linestyle'] = 'solid'

SEED = 42
```

# 1. First steps with unsupervised anomaly detection algorithms

The goal of this section is to get familiar with different unsupervised anomaly detection approaches and algorithms. In order to visualise the output of the different algorithms we consider a toy data set consisting in a two-dimensional Gaussian mixture.

### Generating the data set

```python
from utils import GaussianMixture

n_samples = 500
n_features = 2
weight_1 = 0.5
weight_2 = 0.5
mean_1 = np.zeros(n_features)
mean_2 = -1 * np.ones(n_features)
cov_1 = np.array([[2., 2.,], [2., 4.]])
cov_2 = 2 * np.identity(n_features)
weights = np.array([weight_1, weight_2])
means = np.array([mean_1, mean_2])
covars = np.array([cov_1, cov_2])

gm = GaussianMixture(weights, means, covars, random_state=SEED)
X = gm.sample(n_samples)
```

### Plot the samples and levels set of the density

```python
X_range = np.zeros((n_features, 2))
X_range[:, 0] = np.min(X, axis=0)
X_range[:, 1] = np.max(X, axis=0)

h = 0.1  # step size of the mesh
x_min, x_max = X_range[0, 0] - 0.1, X_range[0, 1] + 0.1
y_min, y_max = X_range[1, 0] - 0.1, X_range[1, 1] + 0.1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

grid = np.c_[xx.ravel(), yy.ravel()]
Z = gm.density(grid)
Z = Z.reshape(xx.shape)

plt.figure()
plt.contour(xx, yy, Z, 10, cmap=plt.cm.Blues_r)
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.show()
```

The goal is to estimate a Minimum volume set with mass at least 0.95. We know that under regularity assumption of the underlying distribution a Minimum volume set is a density level set of the density $f$.
$$
\{x, f(x) \geq F_f^{-1}(1-\alpha) \}
$$
where $F_f^{-1}(1-\alpha)$ is the quantile of order $1-\alpha$ of the distribution of the random variable $f(X)$.

1. Draw a sample $\mathcal{S}$ from the previous Gaussian mixture of size $n = 1,000,000$.

2. Using the sample $\mathcal{S}$ and the true density $f$ given by `gm.density` compute the empirical quantile of order $1-\alpha$ for $\alpha = 0.95$ and $\alpha = 0.99$. You can use for instance the `scipy.stats.mstats.mquantiles` function.

3. Plot the corresponding Minimum Volume set in a figure like the previous one, using `plt.contour` and replacing the number of levels (10) by the threshold you computed: `levels=threshold`.

4. Emphasize the outliers (for instance use a different color) in the previous plot where an outlier is a sample outside the Minimum Volume set.

```python
alpha_set = 0.95

from scipy.stats.mstats import mquantiles

n_quantile = 1000000
Xq = gm.sample(n_quantile)
density_q = gm.density(Xq)
tau = mquantiles(density_q, 1 - alpha_set)

X_outliers = X[gm.density(X) < tau]
```

```python
plt.figure()
c_0 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={tau[0]: str(alpha_set)})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.scatter(X_outliers[:, 0], X_outliers[:, 1], color='red', s=4.)
plt.show()
```

## 1.1. Density estimation

We are going to use the plug-in approach to estimate the Minimum Volume set with mass at least 0.95. The plug-in approach means that we replace the density $f$ by an estimate $\hat f$.

1. Equation of the corresponding Minimum Volume set estimate.

2. Using a Gaussian kernel, compute a kernel density estimator of $f$. To stick with `sklearn` you can use `sklearn.neighbors.kde.KernelDensity` with the default bandwidth.

3. Compute the empirical quantile $\widehat F_{\hat f}^{-1}(1-\alpha)$ from the original sample for $\alpha = 0.95$ (still using the function `mquantiles`).

4. Add the corresponding Minimum Volume set estimate to the previous plot to visualize the difference between the true MV set and the estimated one.

```python
from sklearn.neighbors.kde import KernelDensity

# Estimate density with a Gaussian kernel density estimator
kde = KernelDensity(kernel='gaussian')
kde = kde.fit(X)
```

```python
kde_X = kde.score_samples(X)
```

```python
tau_kde = mquantiles(kde_X, 1 - alpha_set)
```

```python
Z_kde = kde.score_samples(grid)
Z_kde = Z_kde.reshape(xx.shape)

plt.figure()
c_0 = plt.contour(xx, yy, Z_kde, levels=tau_kde, colors='red', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={tau_kde[0]: str(alpha_set)})
c_1 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=2, label='True')
plt.clabel(c_1, inline=1, fontsize=15, fmt={tau[0]: 'True'})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.legend()
plt.show()
```

### Bandwidth selection with cross validation

We used the default bandwidth.

1. Use cross validation to learn the bandwidth from the data. You may use `sklearn.model_selection.GridSearchCV` to perform a 3-fold cross validation.

2. Plot the Minimum Volume set estimate obtained with the learnt bandwidth.

```python
from sklearn.model_selection import GridSearchCV
```

```python
grid_cv = GridSearchCV(KernelDensity(kernel='gaussian'), {'bandwidth': np.linspace(0.1, 2.0, 30)}, cv=3)
grid_cv.fit(X)
grid_cv.best_params_
```

    {'bandwidth': 0.7551724137931034}

```python
kde_best = grid_cv.best_estimator_

kde_best_X = kde_best.score_samples(X)
tau_kde_best = mquantiles(kde_best_X, 1 - alpha_set)
```

```python
Z_kde_best = kde_best.score_samples(grid)
Z_kde_best = Z_kde_best.reshape(xx.shape)

plt.figure()
c_0 = plt.contour(xx, yy, Z_kde_best, levels=tau_kde_best, colors='red', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={tau_kde_best[0]: str(alpha_set)})
c_1 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=2, label='True')
plt.clabel(c_1, inline=1, fontsize=15, fmt={tau[0]: 'True'})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.legend()
plt.show()
```

## 1.2. One-Class SVM

The One-Class SVM separates the data from the origin by solving the following optimization problem.

\begin{equation}
\min_{\boldsymbol{w},\boldsymbol{\xi},\rho} \quad \frac{1}{2}\Vert \boldsymbol{w} \Vert^2 - \rho + \frac{1}{\nu n}\sum_{i=1}^n\xi_i\\
\text{s.t.} \quad \langle \boldsymbol{w}, \Phi(x_i) \rangle \geqslant \rho - \xi_i, \quad \xi_i \geqslant 0 \quad \quad 1 \leqslant i \leqslant n
\end{equation}

where the parameter $\nu \in (0,1)$ has to be set by the user and controls the proportion of outliers.

It is in general solved via the dual given by
\begin{equation}
\min_{\boldsymbol{\alpha}} \quad \frac{1}{2}\sum_{i,j} k(x_i, x_j)\\
\text{s.t.} \quad 0 \leqslant \alpha_i \leqslant \frac{1}{\nu n},\quad \sum_{i}\alpha_i = 1 \quad \quad 1 \leqslant i \leqslant n
\end{equation}

1. Show that when using a Gaussian kernel, all data in the feature space lies on the same hypersphere. Show that data are always linearly separable from the origin. Hint: what's the formula of $k(x, x')$ in terms of the feature map $\Phi$.

```python
from sklearn.svm import OneClassSVM
```

Under mild assumptions the following inequality holds true for all sample size $n$:

$$ \frac{\text{Outliers}}{n} \leq \nu \leq \frac{\text{Support Vectors}}{n} $$.

Futhermore the two fractions on the left-hand side and right-hand converge almost surely towards $\nu$.

1. If we want to estimate a Minimum Volume set with mass at least $\alpha$, how can we set $\nu$?

2. Use `sklearn.svm.OneClassSVM` with the default parameters (except the parameter $\nu$ set according to the previous question) to learn a Minimum Volume set with mass at least $0.95$.

3. Plot the Minimum Volume set estimate.

```python
nu = 0.05
ocsvm = OneClassSVM(kernel='rbf', gamma=0.05, nu=nu)
#ocsvm = OneClassSVM(kernel='rbf', gamma=50., nu=nu)
#ocsvm = OneClassSVM(kernel='rbf', nu=nu)
ocsvm.fit(X)
```

    OneClassSVM(cache_size=200, coef0=0.0, degree=3, gamma=0.05, kernel='rbf',
          max_iter=-1, nu=0.05, random_state=None, shrinking=True, tol=0.001,
          verbose=False)

```python
X_outliers = X[ocsvm.predict(X) == -1]
```

```python
Z_ocsvm = ocsvm.decision_function(grid)
Z_ocsvm = Z_ocsvm.reshape(xx.shape)

plt.figure()
c_0 = plt.contour(xx, yy, Z_ocsvm, levels=[0], colors='red', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={0: str(alpha_set)})
c_1 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=2, label='True')
plt.clabel(c_1, inline=1, fontsize=15, fmt={tau[0]: 'True'})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.scatter(X_outliers[:, 0], X_outliers[:, 1], s=4., color='red')
plt.legend()
plt.show()
```

1. What do you notice? The kernel used by the One-Class SVM is given by $$ k(x, x') = \exp(-\gamma\Vert x - x' \Vert^2). $$

2. What is the default gamma used when we trained the One-Class SVM? Do we need to increase or decrease its value?

```python
ocsvm?
```

### Support vectors - Outliers

1. Check that we indeed have $$ \frac{\text{Outliers}}{n} \leq \nu \leq \frac{\text{Support Vectors}}{n} $$.

2. Use `gamma=10`. Is the inequality always satisfied? Can you guess why?

```python
X_SV = X[ocsvm.support_]
n_SV = len(X_SV)
n_outliers = len(X_outliers)

print('{0:.2f} <= {1:.2f} <= {2:.2f}?'.format(1./n_samples*n_outliers, nu, 1./n_samples*n_SV))
```

    0.10 <= 0.05 <= 0.17?

Only the support vectors are involved in the decision function of the One-Class SVM.

1. Plot the level sets of the One-Class SVM decision function as we dit for the true density.
2. Emphasize the Support vectors.

```python
plt.figure()
plt.contourf(xx, yy, Z_ocsvm, 10, cmap=plt.cm.Blues_r)
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.scatter(X_SV[:, 0], X_SV[:, 1], s=20., color='orange')
plt.show()
```

1. Concerning the support vectors, what is the main difference between the OneClass SVM and a Kernel Density estimator when using a Gaussian Kernel?

2. When is it particularly advantageous to use the OneClass SVM?

### What if we change the value of $\nu$?

1. Set $\nu = 0.4$. How can we estimate a Minimum Volume set with mass at least $0.95$. Hint: Use the `decision_function`.

```python
nu = 0.4
ocsvm = OneClassSVM(kernel='rbf', gamma=0.5, nu=nu)
ocsvm.fit(X)
```

    OneClassSVM(cache_size=200, coef0=0.0, degree=3, gamma=0.5, kernel='rbf',
          max_iter=-1, nu=0.4, random_state=None, shrinking=True, tol=0.001,
          verbose=False)

```python
Z_ocsvm = ocsvm.decision_function(grid)
Z_ocsvm = Z_ocsvm.reshape(xx.shape)

ocsvm_X = ocsvm.decision_function(X)
tau_ocsvm = mquantiles(ocsvm_X, 1 - alpha_set)

plt.figure()
c_0 = plt.contour(xx, yy, Z_ocsvm, levels=tau_ocsvm, colors='red', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={tau_ocsvm[0]: str(alpha_set)})
c_1 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=2, label='True')
plt.clabel(c_1, inline=1, fontsize=15, fmt={tau[0]: 'True'})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.legend()
plt.show()
```

### Equivalence with SVDD

Support Vector Data Description consists in finding the ball of minimum volume containing an arbitrary proportion of data. This is obtained by minimizing the following optimization problem.

\begin{equation}
\min_{R, \boldsymbol{\xi},\boldsymbol{c}}  \quad R^2 + \frac{1}{\nu n}\sum_{i=1}^{n}\xi_i\\
    \text{s.t.} \quad \Vert \boldsymbol{x}_i - \boldsymbol{c}\Vert^2 \leqslant R^2 + \xi_i, \quad \xi_i \geqslant 0 \quad \quad 1\leqslant i \leqslant n\\
\end{equation}

It is in general solved via the dual given by
\begin{equation}
\min_{\boldsymbol{\alpha}} \quad \sum_{i,j} k(x_i, x_j) + \sum_{i}\alpha_i k(x_i,x_i)\\
\text{s.t.} \quad 0 \leqslant \alpha_i \leqslant \frac{1}{\nu n},\quad \sum_{i}\alpha_i = 1 \quad \quad 1 \leqslant i \leqslant n
\end{equation}

1. When are SVDD and the One-Class SVM equivalent?

## 1.3. Isolation Forest

1. Run Isolation with `n_estimators=1` , i.e., only one tree will be built.

2. Plot the contour of the set with the $95\%$ most normal observations. Hint: use the `decision_function` and the `contamination` parameter. Compare it with the true Minimum Volume set with mass at least $0.95$.

3. Increase `n_estimators` to 50, 100 and then 500.

4. What does the `max_sample` parameter change?

```python
from sklearn.ensemble import IsolationForest
```

```python
# iforest = IsolationForest(n_estimators=1, contamination=0.05)
iforest = IsolationForest(n_estimators=100, contamination=0.05)
iforest = iforest.fit(X)
```

```python
Z_iforest = iforest.decision_function(grid)
Z_iforest = Z_iforest.reshape(xx.shape)

plt.figure()
c_0 = plt.contour(xx, yy, Z_iforest, levels=[iforest.threshold_], colors='red', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={iforest.threshold_: str(alpha_set)})
c_1 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=2, label='True')
plt.clabel(c_1, inline=1, fontsize=15, fmt={tau[0]: 'True'})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.legend()
plt.show()
```

## 1.4. Local Outlier Factor

k-NN based anomaly detection algorithm. Available on the dev version of Scikit-Learn.

## 1.5. Unsupervised as supervised using SVM

Let $P$ be a distribution with finite support $C$. Let $\mu$ be the uniform distribution over the support $C$: $\mu(A) = \lambda(A)/\lambda(C)$. We now consider the joint distribution $Q$ of $(X, Y) \in \mathbb{R}^d \times \{-1, +1\}$ given by the class conditionals distributions:

$$
X \vert Y= +1 \sim P \quad \text{and} \quad X \vert Y=-1 \sim \mu
$$
and a priori class probabilities $p = \mathbb{P}(Y=1)$.

We are as before interested in finding a given density level set $\{ f \geq \tau \}$ where $f$ is the density of $P$ with respect to the Lebesgue measure $\lambda$.

The marginal distribution of $X$, $P_X$, is given by
$$
P_X = p P + (1-p) Q.
$$

Assuming that $P$ has a density $f$. It can be shown that

$$\eta(x) = P(Y=1\vert X=x) = \frac{pf(x)}{pf(x) + (1-p)/\lambda(C)}$$

The optimal classification set is given by $\{ \eta \geq 1/2 \}$

1. Show that if $p=1/(1 + \lambda(C)\tau)$ then $\lambda(\{\eta \geq 1/2 \} \Delta \{f \geq \tau \} ) = 0$

Thus solving a binary classification problem assigning weights $p$ and $1-p$ to the samples can lead to a density level set.

### Generate uniform data in the range of X

```python
### Generating artificial second class
rng = np.random.RandomState(SEED)

U = np.zeros((n_samples, n_features))
for i in range(n_features):
    U [:, i] = rng.uniform(X_range[i, 0], X_range[i, 1], n_samples)

vol = np.prod(X_range[:, 1] - X_range[:, 0])
```

```python
X_bin = np.r_[X, U]
y_bin = np.r_[np.ones(n_samples), np.zeros(n_samples)].astype(int)
```

```python
from sklearn.svm import SVC
# SVM parameters

C = 1.
lambda_reg = 0.00005
gamma = 0.01

p = 1. / (1. + vol * tau[0])

weight_1 = p * 1. / (lambda_reg * n_samples)
weight_0 = (1 - p) * 1. / (lambda_reg * n_samples)
class_weight = {1: weight_1, 0: weight_0}

clf = SVC(C=C, kernel='rbf', gamma=gamma, class_weight=class_weight)
clf = clf.fit(X_bin, y_bin)

Z_svm = clf.decision_function(grid)
Z_svm = Z_svm.reshape(xx.shape)

plt.figure()
c_0 = plt.contour(xx, yy, Z_svm, levels=[0], colors='red', linewidths=3)
plt.clabel(c_0, inline=1, fontsize=15, fmt={0: str(alpha_set)})
c_1 = plt.contour(xx, yy, Z, levels=tau, colors='green', linewidths=2, label='True')
plt.clabel(c_1, inline=1, fontsize=15, fmt={tau[0]: 'True'})
plt.scatter(X[:, 0], X[:, 1], s=1.)
plt.legend()
plt.show()
```

```python

```

# 2. Anomaly scoring and Mass Volume curve

Usually unsupervised anomaly detection algorithms returns a scoring function $s$ such that the smaller $s(x)$ the more abnormal $x$ is. To assess the performance of such a scoring function we can use the Mass Volume curve. The MV curve of a scoring function $s$ is given for all $\alpha \in (0,1)$ by
$$
MV_s(\alpha) = \lambda\{x, s(x) \geq F_s^{-1}(1-\alpha) \}
$$
where $F_s^{-1}(1-\alpha)$ is the quantile of order $1-\alpha$ of the distribution of the random variable $s(X)$.

Under regularity assumptions, the MV curve is also given by the parametric curve

$$
(\mathbb{P}(s(X) \geq t), \lambda(x, s(x) \geq t))
$$

Assume that the random variable $X$ with distribution $P$ has a density $f$ with respect to the Lebesgue measure. Furthermore assume that for all $\alpha \in (0,1)$ the Minimum Volume set optimization problem
$$
\min_{\Omega \in \mathcal{M}} \lambda(\Omega) \quad \text{ s.t } \quad P(\Omega) \geq \alpha .
$$
has a unique solution given by
$$
\Omega^*_{\alpha} = \{x, f(x) \geq F_f^{-1}(1-\alpha) \}.
$$

1. Show that for all $\alpha \in (0,1)$, $MV_s(\alpha) - MV_f(\alpha) \leq \lambda(\Omega^*_{\alpha} \Delta \{x, s(x) \geq F_s^{-1}(1-\alpha) \})$.

2. Show that for all $\alpha \in (0,1)$, $MV_f(\alpha) \leq MV_s(\alpha)$. Hint: use the following property: for all $\alpha \in (0,1)$, $\mathbb{P}(s(X) < F_s^{-1}(1-\alpha)) \leq 1-\alpha$.

3. In what sense a scoring function minimizing the area under the Mass Volume curve is optimal?

## 2.1. Using Mass Volume curve for performance evaluation

The lower $MV_s$ the better the scoring function $s$. The goal is to compare the performance of scoring functions given by the One-Class SVM for different values of $\gamma$ using the Mass Volume curve. To easily plot the Mass Volume we are going to use the ROC curve function given by `sklearn`. The idea is to use a uniform sample as the positive class and to consider the Gaussian mixture data set as the negative class. To plot the MV curve we want to plot the volume against the mass $\alpha$. This will be given by $1-TPR$ against $1-FPR$.

1. Using `sklearn.model_selection.ShuffleSplit` split the data set into a training set and a test set.

2. Generate `n_sim=100000` uniform random variables in the hypercube enclosing the data using `numpy.random.uniform`.

3. Build a training set consisting in the concatenation of the training set of the Gaussian mixture data set (obtained in 1.) and the uniform sample.

4. Similarly, build the test set by concatenating the test set of the Gaussian mixture data set (obtained in 1.) and the uniform sample.

```python
from sklearn.model_selection import ShuffleSplit

X_range = np.zeros((n_features, 2))
X_range[:, 0] = np.min(X, axis=0)
X_range[:, 1] = np.max(X, axis=0)

n_sim = 100000
U = np.zeros((n_sim, n_features))
for i in range(n_features):
    U [:, i] = rng.uniform(X_range[i, 0], X_range[i, 1], n_sim)

### Training on half the data
ss = ShuffleSplit(n_splits=1, test_size=0.5, random_state=SEED)

ss = ss.split(X)
train, test = ss.next()

X_train = X[train]
X_test = X[test]
X_train_all = np.r_[U, X_train]
y_train_all = np.r_[np.zeros(n_sim), np.ones(len(train))]
X_test_all = np.r_[U, X_test]
y_test_all = np.r_[np.zeros(n_sim), np.ones(len(test))]
```

1. Compute the Mass Volume curve, estimated on the training set, of the scoring function obtained with the One-Class SVM for $\gamma=0.05$ and $\gamma=10$. Compute the area under the Mass Volume curve for each value of $\gamma$.

2. Compute the Mass Volume curve estimated on the test set for each value of $\gamma$. Compute the area under the Mass Volume curve for each value of $\gamma$.

```python
from sklearn.metrics import roc_curve, auc

colors = ['red', 'blue']

plt.figure(figsize=(12, 6))

for g, gamma in enumerate([0.05, 10]):
    ocsvm = OneClassSVM(gamma=gamma, nu=0.05)
    ### Train algorithm (fit), compute decision function on train and test, compute ROC and AUC, draw ROC
    ocsvm.fit(X_train)
    ocsvm_train_all = ocsvm.decision_function(X_train_all)
    ocsvm_test_all = ocsvm.decision_function(X_test_all)
    fpr_train_, tpr_train_, _ = roc_curve(y_train_all, -ocsvm_train_all, pos_label=0)
    ocsvm_auc_train = auc(1 - fpr_train_, 1 - tpr_train_)
    fpr_test_, tpr_test_, _ = roc_curve(y_test_all, -ocsvm_test_all, pos_label=0)
    ocsvm_auc_test = auc(1 - fpr_test_, 1 - tpr_test_)

    plt.subplot(1, 2, 1)
    plt.title('Performance on training set')
    plt.plot(1 - fpr_train_, 1 - tpr_train_, color=colors[g], label= 'Gamma: {0:.2f} - AMV: {1:.3f}'.format(gamma, ocsvm_auc_train))

    plt.subplot(1, 2, 2)
    plt.title('Performance on test set')
    plt.plot(1 - fpr_test_, 1 - tpr_test_, color=colors[g], label= 'Gamma: {0:.2f} - AMV: {1:.3f}'.format(gamma, ocsvm_auc_test))

plt.subplot(1, 2, 1)
plt.legend(loc=0)
plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))

plt.subplot(1, 2, 2)
plt.legend(loc=0)
plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))

plt.show()
```

# 3. Performance evaluation on real data sets

## 3.1. Shuttle data set

```python
from sklearn.datasets import fetch_mldata

dataset = fetch_mldata('shuttle')
X = dataset.data
y = dataset.target

### Usual setup when using this data set for anomaly detection
# Instances with label 4 are removed
# Normal data: label 1
ind = (y != 4)
X = X[ind, :]
y = y[ind]
y = (y == 1).astype(int)
```

```python
n_samples, n_features = X.shape
n_samples, n_features
```

    (85849, 9)

1. What's the proportion of anomalies in the data set (label 0)?
2. Split the data set into a training a test set using `StratifiedShuffleSplit` to preserve the proportion of each class in the training and test sets.
3. Compare the performance of the OneClass SVM and Isolation Forest using the ROC curve (computed with the true labels) on the train set and the set set:
    - train the algorithm on the training set (without the labels), compute the ROC curve on the training set using the labels.
    - train the algorithm on the training set (without the labels), compute the ROC curve on the test set using the labels
4. Compare the performance of the OneClass SVM and Isolation Forest using the Mass Volume curve on the training and test sets

```python
'Percentage of anomalies: {0}'.format(1 - np.mean(y))
```

    'Percentage of anomalies: 0.0716956516675'

```python
from sklearn.model_selection import StratifiedShuffleSplit

sss = StratifiedShuffleSplit(n_splits=1, test_size=0.5, random_state=SEED)

train, test = sss.split(X, y).next()
X_train, y_train = X[train], y[train]
X_test, y_test = X[test], y[test]
```

```python
print('Percentage of anomalies in train : {0}'.format(1 - np.mean(y_train)))
print('Percentage of anomalies in test : {0}'.format(1 - np.mean(y_test)))
```

    Percentage of anomalies in train : 0.0716848383189
    Percentage of anomalies in test : 0.0717064647641

### Performance on training set and test set - ROC curve - AUC

```python
### !!!! This can take a few minutes. You might consider running it with Isolation Forest only at first

algorithms = [OneClassSVM(), IsolationForest()]
name = ['Isolation Forest', 'Ocsvm']
colors = ['green', 'red']

plt.figure(figsize=(12, 6))

for a, algo in enumerate(algorithms):
    algo.fit(X_train)
    algo_train = algo.decision_function(X_train)
    algo_test = algo.decision_function(X_test)
    fpr_train_, tpr_train_, _ = roc_curve(y_train, -algo_train, pos_label=0)
    algo_auc_train = auc(fpr_train_, tpr_train_)
    fpr_test_, tpr_test_, _ = roc_curve(y_test, -algo_test, pos_label=0)
    algo_auc_test = auc(fpr_test_, tpr_test_)

    plt.subplot(1, 2, 1)
    plt.title('Performance on training set')
    plt.plot(fpr_train_, tpr_train_, color=colors[a], label= '{0} - AUC: {1:.3f}'.format(name[a], algo_auc_train))

    plt.subplot(1, 2, 2)
    plt.title('Performance on test set')
    plt.plot(fpr_test_, tpr_test_, color=colors[a], label= '{0} - AUC: {1:.3f}'.format(name[a], algo_auc_test))

plt.subplot(1, 2, 1)
plt.legend(loc=0)
plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))

plt.subplot(1, 2, 2)
plt.legend(loc=0)
plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))

plt.show()
```

### Performance - MV curve - AMV

```python
X_range = np.zeros((n_features, 2))
X_range[:, 0] = np.min(X, axis=0)
X_range[:, 1] = np.max(X, axis=0)

n_sim = 100000
U = np.zeros((n_sim, n_features))
for i in range(n_features):
    U [:, i] = rng.uniform(X_range[i, 0], X_range[i, 1], n_sim)

### We add uniform data to compute volumes
X_train_all = np.r_[U, X_train]
X_test_all = np.r_[U, X_test]

# Assigning same label to all data (normal and anomalies): the anomalies are the uniform samples
y_train_all = np.r_[np.zeros(n_sim), np.ones(len(X_train))]
y_test_all = np.r_[np.zeros(n_sim), np.ones(len(X_test))]
```

```python
### !!!! This can take a few minutes. You might consider running it with Isolation Forest only at first

algorithms = [OneClassSVM(), IsolationForest()]
name = ['Isolation Forest', 'Ocsvm']
colors = ['green', 'red']
# algorithms = [IsolationForest()]
# name = ['Isolation Forest']
# colors = ['green']

plt.figure(figsize=(12, 6))

for a, algo in enumerate(algorithms):
    ### Train algorithm (fit), compute decision function on train and test, compute ROC and AUC, draw ROC
    algo.fit(X_train)
    algo_train_all = algo.decision_function(X_train_all)
    algo_test_all = algo.decision_function(X_test_all)
    fpr_train_, tpr_train_, _ = roc_curve(y_train_all, -algo_train_all, pos_label=0)
    algo_auc_train = auc(1 - fpr_train_, 1 - tpr_train_)
    fpr_test_, tpr_test_, _ = roc_curve(y_test_all, -algo_test_all, pos_label=0)
    algo_auc_test = auc(1 - fpr_test_, 1 - tpr_test_)

    plt.subplot(1, 2, 1)
    plt.title('Performance on training set')
    plt.plot(1 - fpr_train_, 1 - tpr_train_, color=colors[a], label= '{0} - AMV: {1:.3f}'.format(name[a], algo_auc_train))

    plt.subplot(1, 2, 2)
    plt.title('Performance on test set')
    plt.plot(1 - fpr_test_, 1 - tpr_test_, color=colors[a], label= '{0} - AUC: {1:.3f}'.format(name[a], algo_auc_test))

plt.subplot(1, 2, 1)
plt.legend(loc=0)
plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))

plt.subplot(1, 2, 2)
plt.legend(loc=0)
plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))

plt.show()
```

## 3.2. Http data set - Network intrusion detection

1. Same questions with the http data set

```python
from sklearn.datasets import fetch_kddcup99

dataset = fetch_kddcup99(subset='http', shuffle=True,
                         percent10=True, random_state=SEED)
X = dataset['data']
X = X.astype('float')
y = dataset['target']
y = (y == 'normal.').astype(int)
```

```python
n_samples, n_features = X.shape
```

### Novelty detection

Now we only train the model on normal data. For instance let's train the model on half the normal data.

1. Split the data set into a set with the normal data and a set with the anomalies.

2. Randomly split the normal data set into a two data sets: a training set and a test set.

3. Build a test set from the normal test set and the anomalies.

4. Train the OneClass SVM and Isolation Forest on the normal training set and compute the ROC curve on the test set.

```python
X_normal = X[y == 1]
X_ano = X[y != 1]
y_normal = y[y == 1]
y_ano = y[y != 1]
```

```python
from sklearn.model_selection import ShuffleSplit

ss = ShuffleSplit(n_splits=1, test_size=0.5, random_state=SEED)

ss = ss.split(X_normal, y_normal)
train_normal, test_normal = ss.next()
```

```python
X_train = X_normal[train_normal]
X_test = np.r_[X_normal[test_normal], X_ano]
y_test = np.r_[y_normal[test_normal], y_ano]
```

#### ROC - AUC on test set

```python
### !!!! This can take a few minutes. You might consider running it with Isolation Forest only at first

algorithms = [OneClassSVM(), IsolationForest()]
name = ['Isolation Forest', 'Ocsvm']
colors = ['green', 'red']
# algorithms = [IsolationForest()]
# name = ['Isolation Forest']
# colors = ['green']

plt.figure(figsize=(12, 6))

for a, algo in enumerate(algorithms):
    ### Train algorithm (fit), compute decision function on train and test, compute ROC and AUC, draw ROC
    algo.fit(X_train)
    algo_test = algo.decision_function(X_test)
    fpr_test_, tpr_test_, _ = roc_curve(y_test, -algo_test, pos_label=0)
    algo_auc_test = auc(fpr_test_, tpr_test_)

    plt.subplot(1, 2, 2)
    plt.title('Performance on test set')
    plt.plot(fpr_test_, tpr_test_, color=colors[a], label= '{0} - AUC: {1:.3f}'.format(name[a], algo_auc_test))

plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))
plt.legend(loc=0)

plt.show()
```

#### MV curve on test set

1. Plot the MV curve estimated on the test set for the OneClass SVM and Isolation Forest.

```python
X_range = np.zeros((n_features, 2))
X_range[:, 0] = np.min(X, axis=0)
X_range[:, 1] = np.max(X, axis=0)

n_sim = 100000
U = np.zeros((n_sim, n_features))
for i in range(n_features):
    U [:, i] = rng.uniform(X_range[i, 0], X_range[i, 1], n_sim)

### We add uniform data to compute volumes
X_test_all = np.r_[U, X_test]

# Assigning same label to all data (normal and anomalies): the anomalies are the uniform samples
y_test_all = np.r_[np.zeros(n_sim), np.ones(len(X_test))]
```

```python
### !!!! This can take a few minutes. You might consider running it with Isolation Forest only at first

algorithms = [OneClassSVM(), IsolationForest()]
name = ['Isolation Forest', 'Ocsvm']
colors = ['green', 'red']
# algorithms = [IsolationForest()]
# name = ['Isolation Forest']
# colors = ['green']

plt.figure(figsize=(12, 6))

for a, algo in enumerate(algorithms):
    ### Train algorithm (fit), compute decision function on train and test, compute ROC and AUC, draw ROC
    algo.fit(X_train)
    algo_test_all = algo.decision_function(X_test_all)
    fpr_test_all_, tpr_test_all_, _ = roc_curve(y_test_all, -algo_test_all, pos_label=0)
    algo_auc_test = auc(1 - fpr_test_all_, 1 - tpr_test_all_)

    plt.subplot(1, 2, 2)
    plt.title('Performance on test set')
    plt.plot(1 - fpr_test_all_, 1 - tpr_test_all_, color=colors[a], label= '{0} - AUC: {1:.3f}'.format(name[a], algo_auc_test))

plt.xlim((-0.05, 1.05))
plt.ylim((-0.05, 1.05))
plt.legend(loc=0)

plt.show()
```

# 4. Optional

1. Show that a density level set $\{f \geq \tau \}$ is a Minimum volume set with mass $\alpha = \mathbb{P}(f(X) \geq \tau)$.

2. Show that if the density has not flat parts, the minimum volume set optimization problem has a solution.

# 5. Overfitting for Minimum volume set estimation

Let $\lambda$ denotes the Lebesgue measure of $\mathbb{R}^d$ and let $\mathcal{M}$ be the class of all measurable sets. A Minimum Volume set with mass at least $\alpha$ is the solution of the following optimization problem:
$$
\min_{\Omega \in \mathcal{M}} \lambda(\Omega) \quad \text{ s.t } \quad P(\Omega) \geq \alpha .
$$
We do not know the distribution $P$ but only have a sample $X_1, \dots, X_n$ drawn from this distribution.

1. What is the solution for all $\alpha \in (0,1)$ of the empirical optimization problem if we solve the following empirical version of the Minimum Volume set optimization problem:
$$
\min_{\Omega \in \mathcal{M}} \lambda(\Omega) \quad \text{ s.t } \quad \widehat P(\Omega) \geq \alpha
$$
where for all $\Omega$, $\widehat P(\Omega) = \frac{1}{n}\sum_{i=1}^n \mathbb{I}\{X_i \in \Omega \}$.

# 6. Illustration of IsolationForest on Digits data set

```python
from sklearn import datasets
digits = datasets.load_digits()
```

The digits data set consits in images (8 x 8) of digits.

```python
images = digits.images
labels = digits.target
images.shape
```

    (1797, 8, 8)

```python
i = 0

plt.figure()
plt.title('{0}'.format(labels[i]))
plt.axis('off')
plt.imshow(images[i], cmap=plt.cm.gray_r, interpolation='nearest')
plt.show()
```

To use the images as a training set we need to flatten the images.

```python
n_samples = len(digits.images)
data = digits.images.reshape((n_samples, -1))
```

```python
data.shape
```

    (1797, 64)

```python
X = data
y = digits.target
```

We are going to focus on digit 5.

```python
X_5 = X[y == 5]
```

1. Use IsolationForest to find the top 5% most abnormal images.
2. Plot them using `plt.imshow(outlier.reshape(8, 8)), cmap=plt.cm.gray_r, interpolation='nearest')`

```python
from sklearn.ensemble import IsolationForest
iforest = IsolationForest(contamination=0.05)
iforest = iforest.fit(X_5)
iforest_X = iforest.decision_function(X_5)
X_outliers = X_5[iforest.predict(X_5)==-1]
```

```python
for i in range(len(X_outliers)):
    plt.subplot(2, int(len(X_outliers) / 2.) + 1, i + 1)
    plt.axis('off')
    plt.imshow(X_outliers[i].reshape((8, 8)), cmap=plt.cm.gray_r, interpolation='nearest')
plt.show()
```

```python

```

```python

```


# 11 Ordinary differential equations – Integrators
More about integrators to solve ODEs.

* Euler's Rule (and it's shortcomings)
* Runge-Kutta methods

## Euler's Rule

From the last lecture: Given the $n$-dimensional vectors from the ODE standard form

$$
\frac{d\mathbf{y}}{dt} = \mathbf{f}(t, \mathbf{y})
$$
the **Euler rule** amounts to

\begin{align}
\mathbf{f}(t, \mathbf{y}) = \frac{d\mathbf{y}(t)}{dt} &\approx \frac{\mathbf{y}(t_{n+1}) - \mathbf{y}(t_n)}{h}\\
\mathbf{y}_{n+1} &\approx \mathbf{y}_n + h \mathbf{f}(t_n, \mathbf{y}_n) \quad \text{with} \quad \mathbf{y}_n := \mathbf{y}(t_n)
\end{align}

Euler's rule cannot obtain high precision without very small $h$ but this leads to higher round-off error accumulation.

### Euler's Rule applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
y^{(0)}_1 &= y^{(0)}_0 + h y^{(1)}_0 = x_0 + v_0 h\\
y^{(1)}_1 &= y^{(1)}_0 + h \left(-\frac{k}{m}y^{(0)}_0\right) = v_0 + \frac{-k x_0}{m} h
\end{align}

Compare to the equations of movement under a constant force (*ballistic motion*):

$$
x = x_0 + v_0 h + \frac{1}{2} a h^2, \quad v = v_0 + a h
$$

In Euler, the position does not even contain the $h^2$ term, i.e., the acceleration only contributes *indirectly* via the velocity of the *next* step. (Remember: Euler's error is $\mathcal{O}(h^2)$!)

### Euler's Rule: Application to 3 oscillators (numerical)
`git pull` new resources.

Put the notebook `11-ODE_integrators-students.ipynb` *and* the module `integrators.py` into the same directory, for instance a work directory `~/PHY494/11_ODEs`.

Open [integrators.py](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/11-ODEs/integrators.py) to have a look at it: it contains code to integrate Newton's equations of motions for different forces and with different integrators (based on the work done in the last lecture).

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
%matplotlib inline
matplotlib.style.use('ggplot')

import integrators
```

```python
t_max = 20
t_harm, y_harm = integrators.integrate_newton(t_max=t_max)
t_anharm, y_anharm = integrators.integrate_newton(t_max=t_max,
                                                  force=integrators.F_anharmonic)
t_power, y_power = integrators.integrate_newton(t_max=t_max,
                                                force=integrators.F_power)
```

```python
plt.plot(t_harm, y_harm[:, 0], label=r"harmonic $\frac{k}{2} x^2$")
plt.plot(t_anharm, y_anharm[:, 0], label=r"anharmonic $\frac{k}{2} x^2(1 - \frac{2}{3} \alpha x)$")
plt.plot(t_power, y_power[:, 0], label=r"power $\frac{k}{6} x^6$")
plt.legend(loc="best")
```

#### Phase-space portrait
Plot position against momentum ($x, p$):

```python
m = 1
x = y_harm[:, 0]
p = m * y_harm[:, 1]
ax = plt.subplot(1,1,1)
ax.plot(x, p)
ax.set_aspect(1)
```

```python
t, y = integrators.integrate_newton(t_max=40,
                                    force=integrators.F_harmonic,
                                    h=0.01)
t2, y2 = integrators.integrate_newton(t_max=40, h=0.001,
                                      force=integrators.F_harmonic)
```

```python
ax = plt.subplot(1,1,1)
ax.plot(y[:, 0], m * y[:, 1], 'b-', label=r"$h=0.01$")
ax.plot(y2[:, 0], m * y2[:, 1], 'r--', label="$h=0.001$")
ax.set_xlabel("position $x$")
ax.set_ylabel("momentum $p$")
ax.set_aspect(1)
ax.legend(loc="best")
```

#### Energy
Energy is conserved: stringent test for the integrator!

$$
E = T_\text{kin} + U\\
\frac{dE}{dt} = 0
$$
with $T_\text{kin} = \frac{1}{2} m v^2$.

Calculate and plot the three energy terms:

```python
KE = 0.5 * m * y_harm[:, 1]**2
PE = integrators.U_harmonic(y_harm[:, 0])
energy = KE + PE
plt.plot(t_harm, KE, 'r-', t_harm, PE, 'b-', t_harm, energy, 'k--')
```

Harmonic oscillator with larger timestep $h=0.01$

```python
m = 1
t, y = integrators.integrate_newton(t_max=500, force=integrators.F_harmonic, h=0.01, mass=m)
x, v = y.T
KE = 1/2 * m * v**2
PE = integrators.U_harmonic(x)
energy = KE + PE
```

```python
ax = plt.subplot(1,1,1)
ax.plot(t, KE, 'r-', t, PE, 'b-', t, energy, 'k--')
ax.set_xlim(0, 20)
ax.set_ylim(0, 0.7)
ax.set_xlabel("time")
ax.set_ylabel("energy")
ax.set_title("Euler: harmonic oscillator")
```

Note how energy conservation becomes worse when $h$ is increased.

For longer time scales, the energy just keeps on increasing — clearly, we are not solving the equations of motions in a physically meaningful way.

```python
ax.set_ylim(0, 80)
ax.set_xlim(0,500)
ax.figure
```

##### Quantify energy drift
Measure the relative departure from energy conservation

$$
\epsilon_E(t) = \log_{10}\left|\frac{E(t) - E(t=0)}{E(t=0)}\right|
$$

(By using the decadic logarithm, $-\epsilon_E$ is a direct measure of the precision.)

###### Harmonic oscillator
Analyze energy drift for the last harmonic oscillator example ($h=0.01$):

```python
energy_precision = np.log10(energy/energy[0] - 1)
plt.plot(t, energy_precision)
plt.ylabel("log(relative error in $E$)")
plt.xlabel("time")
```

As we already know, with such a large time step, energy conservation is very bad with the *Euler* integrator: The relative (!) error quickly becomes 100% (after $t=200$).

##### Code re-use: energy analysis as a function
The module `integrators` contains the function `integrators.analyze_energies(t, y, U)` to automate the task of analysing the energy. You *must* provide the appropriate potential energy function `U`.

##### Problem: Make `analyze_energies()` work and analyze the power law oscillator
The code is *incomplete*.
1. Open `integrators.py` in your editor.
2. Complete the `kinetic_energy()` function (remember: $T_\text{kin} = \frac{1}{2} m v^2$)

Then `reload()` the module:

```python
# when working on the integrator module you need to reload() whenever you make changes
from importlib import reload   # for Python 3
reload(integrators);
```

and try it out for power law oscillator with a large time step $h = 0.01$. Can you find a value of $h$ that will give you energy precision to 4th decimal over 40 time units?

###### Energy stability analysis of the 6-th power oscillator
$x^6$ potential with $h=0.01$:

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.01)
```

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

Trying 100 times smaller $h$ (takes 100 times longer!!)

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    h=0.0001)
integrators.analyze_energies(t, y, integrators.U_power)
```

*Reducing $h$ further does not seem to help and takes order of magnitude longer to run*. **We need a better integrator!**

## Runge-Kutta methods
Explicit RK methods propagate with the general form

$$
y_{n+1} = y_n + h \sum_{i=1}^s b_i k_i, \quad t_{n+1} = t_n + h
$$

where $s$ is the number of *stages* and the coefficients $a_{ij}$, $b_i$ and $c_i$ define the method.

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + c_2 h, y_n + h(a_{21} k_1))\\
k_3 &= f(t_n + c_3 h, y_n + h(a_{31} k_1 + a_{32} k_2))\\
 \vdots& \\
k_s &= f(t_n + c_s h, y_n + h(a_{s1} k_1 + a_{s2} k_2 + \cdots + a_{s,s-1} k_{s-1})).
\end{align}

Basic idea: improve estimate for the slope by using weighted averages of slopes over the interval.

### RK4
General purpose ODE solver.

$$
y_{n+1} = y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big), \quad t_{n+1} = t_n + h
$$

and

\begin{align}
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

`rk4` can obtain $\mathcal{O}(h^4)$ precision.

### RK2
`rk2` is simpler to derive and understand than `rk4` (which we will use more often).

Start with the exact solution

$$
\frac{dy}{dt} = f(t, y) \quad\text{and}\quad y(t) = \int f(t, y)dt\\
y_{n+1} = y_n + \int_{t_n}^{t_{n+1}} f(t, y)dt
$$

Expand $f(t, y)$ around *midpoint* $t_{n+1/2} = t + \frac{h}{2}$ of integration interval

\begin{gather}
f(t, y) \approx f(t_{n+1/2}, y_{n+1/2}) + (t - t_{n+1/2})\,  f^{(1)}(t_{n+1/2}) +
   \mathcal{O}(h^2)
\end{gather}

\begin{align}
\int_{t_n}^{t_{n+1}} f(t, y) &\approx h\, f(t_{n+1/2}, y_{n+1/2}) + \mathcal{O}(h^3)\\
y_{n+1} &= y_n + h\, f(t_{n+1/2}, y_{n+1/2})
\end{align}

I.e., use **slope at midpoint**! (The $\mathcal{O}(h^2)$ term in the integral is zero because $[(h/2)^2 - (h/2)^2] = 0$.)

But we don't have $y_{n+1/2}$ so we use Euler's rule to estimate it to $\mathcal{O}(h^2)$:

$$
y_{n+1/2} \approx y_n + \frac{1}{2} h\, f(t_n, y_n)
$$

and insert into $y_{n+1} = y_n + h\, f(t_{n+1/2}, y_{n+1/2})$ (so it is still $\mathcal{O}(h^3)$ accurate).

Altogether: **RK2** algorithm

$$
\mathbf{y}_{n+1} = \mathbf{y}_n + h \mathbf{k}_2, \quad t_{n+1} = t_n + h
$$

and

\begin{align}
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

with $\mathcal{O}(h^3)$ accuracy.

#### RK2 applied to the 1D harmonic oscillator (theory)

\begin{alignat}{3}
f^{(0)}(t, \mathbf{y}) &= y^{(1)},
    &\quad y^{(0)}_0 &= x_0,\\
f^{(1)}(t, \mathbf{y}) &=   - \frac{k}{m} y^{(0)},
    &\quad y^{(1)}_0 &= v_0.
\end{alignat}

with $k=1$; $x_0 = 0$ and $v_0 = +1$.

First step from $t=0$ to $t=h$:

\begin{align}
  y^{(0)}_{1} &=
    y^{(0)} _{0} + hf^{(0)} \Big(\frac{h}{2}, y^{(0)}_{0} + hk_{1} \Big)\\
    &= x_{0} + h\left[v_{0} + \frac{h}{2} \left(-\frac{k}{m} x_{0}\right)  \right]\\
    &= x_0 + v_0 h + \frac{1}{2} \left(-\frac{k}{m} x_{0}\right) h^2\\
  y^{(1)}_{1} &= y^{(1)}_{0} +
                h f^{(1)}\left(\frac{h}{2},
                     \mathbf{y}_{0} + \frac{h}{2} \mathbf{f}(0, \mathbf{y}_{0})\right) \\
             &= v_{0} + \frac{h}{m} \left[-k\left(x_{0} - \frac{h}{2} k x_{0}\right)\right]
\end{align}

Note that `rk2` contains a $h^2$ term in the position update.

#### RK2 applied in practice
Let's implement `rk2`

\begin{align}
\mathbf{y}_{n+1} &= \mathbf{y}_n + h \mathbf{k}_2\\
\mathbf{k}_1 &= \mathbf{f}(t_n, \mathbf{y}_n)\\
\mathbf{k}_2 &= \mathbf{f}(t_n + \frac{1}{2} h, \mathbf{y}_n + \frac{1}{2} h \mathbf{k}_1)
\end{align}

in our `integrators.py`.

```python
from importlib import reload
reload(integrators);
```

##### RK2 for the power potential

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk2, h=0.01)
```

###### phase-space portrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

#### RK4 applied in practice
Implement `rk4` in `integrator.py`:

\begin{align}
y_{n+1} &= y_n + \frac{h}{6}\Big(k_1 + 2 k_2 + 2 k_3 + k_4\Big)\\
k_1 &= f(t_n, y_n)\\
k_2 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_1)\\
k_3 &= f(t_n + \frac{1}{2} h, y_n + \frac{1}{2} h k_2)\\
k_4 &= f(t_n + h, y_n + h k_3)
\end{align}

```python
reload(integrators)
```

and test the 6th power potential again. What level of energy precision can you obtain over 40 time units?

##### RK4 for the 6th power potential
Note that we can easily run `rk4` with a $h=0.01$!

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.01)
```

###### phase-space protrait

```python
plt.plot(y[:, 0], m*y[:, 1])
```

###### energy drift

```python
integrators.analyze_energies(t, y, integrators.U_power)
```

You should be able to get 8 - 11 decimals precision from `rk4` with a smaller $h$.

##### High precision RK4 for the 6th power potential
Try a 10 times smaller step $h=0.001$

```python
t, y = integrators.integrate_newton(t_max=40, force=integrators.F_power,
                                    integrator=integrators.rk4, h=0.001)
integrators.analyze_energies(t, y, integrators.U_power)
```

This looks like stable integration with an error of not worse than $10^{-12}$: *Looking very good!* (Especially when you compare it to *Euler* with $h=0.001$!)

It looks as if the main source of error comes from the force evaluation because the error spikes when the potential energy is dominant.


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of Passive Electrical Networks

The Laplace transform is a well-established tool for the analysis of differential equations including initial values. [Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described mathematically by linear ordinary differential equations (ODEs) with constant coefficients. The Laplace transform provides an elegant way of analyzing such networks.

### Complex Impedances and Equivalent Networks

The concept of complex impedances is used to analyze passive electrical networks in the Laplace domain. Let's first take a look at the ODEs describing the relation between voltage $u(t)$ and current $i(t)$ for linear passive elements. They are summarized in the second column of the following table

| Element | $\quad \qquad \qquad \quad \quad$ Temporal Domain $\qquad \qquad \qquad \quad \quad$ | $\qquad \qquad$ Laplace Domain $\qquad \qquad$ | Impedance $Z(s)$ |
|:---:|:---:|:---:|:---:|
|  | $u(t) = R \cdot i(t)$ | $U(s) = R \cdot I(s)$ | $R$ |
|  | $\begin{matrix} u(t) = L \frac{d}{dt} i(t) \\ i(t) = \frac{1}{L} \int_{0}^{t} u(\tau) d\tau + i(0+) \epsilon(t) \end{matrix}$ | $\begin{matrix} U(s) = s L I(s) - L i(0+) \\ I(s) = \frac{1}{s L} U(s) + \frac{1}{s} i(0+) \end{matrix}$ | $s L$ |
|  | $\begin{matrix} u(t) = \frac{1}{C} \int_{0}^{t} i(\tau) d\tau + u(0+) \epsilon(t) \\ i(t) = C \frac{d}{dt} u(t) \end{matrix}$ | $\begin{matrix} U(s) = \frac{1}{s C} I(s) + \frac{1}{s} u(0+) \\ I(s) = s C U(s) - C u(0+) \end{matrix}$ | $\frac{1}{s C}$

It was assumed that the voltage $u(t)=0$ and current $i(t)=0$ for $t < 0$, hence that both are causal signals. The initial values $u(0+)$ and $i(0+)$ denote the right-sided limit values for $t=0$. For instance $u(0+) = \lim_{\epsilon \to 0} u(0 + \epsilon)$. They initial values represent the energy stored in the capacitors and inductors at time instant $t=0$, respectively. The analysis of a passive electrical network is perfomed by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) resulting in an ODE, describing for instance the relation between input and output voltage. This ODE has to be solved explicitly. See for instance the [previous example](../systems_time_domain/network_analysis.ipynb).

The time-domain relations can be transformed into the Laplace domain by applying the [differentiation and integration theorem](table_theorems_transforms.ipynb#Properties-and-Theorems) of the Laplace transform. The results are summarized in the third column. The differentiation and integration are now represented by linear operations. Kirchhoff's circuit laws can be applied straighforward to the transformed quantities using the transformed relations. This is due to the fact that the Laplace transform is a linear operation. The result is an algebraic equation that can be solved straightforward with respect to the desired quantities.

When the initial values $i(0+)$ or $i(0+)$ are zero, the elements can be characterized in the Laplace domain by their [complex impedances](https://en.wikipedia.org/wiki/Electrical_impedance). The complex impedance $Z(s)$ is defined as follows

\begin{equation}
Z(s) = \frac{U(s)}{I(s)}
\end{equation}

Complex impedances can be used to represent a passive electrical network in the Laplace domain. The analysis of an electrical network in the Laplace domain is illustrated in the next section. Note that similar considerations also apply to mechanical systems and other problems that can be described by ODEs.

### Example: Second-Order Low-Pass Filter

The second-order low-pass filter from the [previous example](../systems_time_domain/network_analysis.ipynb) is analyzed using the Laplace transform. First the step response for zero initial values is computed followed by an analysis including initial values in a second stage.

#### Output signal for zero initial values

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. Consequently, the initial values can be discarded. The equivalent network in the Laplace domain is derived by transforming the input $x(t)$ and output $y(t)$, and introducing the complex impedances from above table for its elements.

Applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) with the complex impedances of the network elements yields the output signal $Y(s)$ in relation to the input $X(s)$ in the Laplace domain as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot X(s)
\end{equation}

This relation is defined in `SymPy` for subsequent evaluation.

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t, R, L, C = sym.symbols('t R L C', positive=True)
X = sym.Function('X')(s)

Y = 1/(L*C*s**2 + R*C*s + 1) * X
Y
```

The response $y(t)$ of the network for a [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) at its input, is computed by setting the input to $x(t) = \epsilon(t)$. The Laplace transform $Y(s)$ of the output signal is given as

\begin{equation}
Y(s) = \frac{1}{LC s^2 + RC s + 1} \cdot \frac{1}{s}
\end{equation}

The output signal $y(t)$ is computed by inverse Laplace transform of $Y(s)$ for the normalized values $L = .5$, $R = 1$, $C = .4$.

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}

y = sym.inverse_laplace_transform(Y.subs(RLC).subs(X, 1/s), s, t)
y
```

The result is simplified for the sake of readability

```python
y = y.simplify()
y
```

```python
sym.plot(y, (t, 0, 5), xlabel='$t$', ylabel='$y(t)$');
```

The computation of the output signal $y(t)$ did not require the solution of the underlying ODE as in the [previous example](../systems_time_domain/network_analysis.ipynb). Based on the equivalent network in the Laplace domain, only the computation of an inverse Laplace transform was required. Above result is equal to the [solution of the ODE for an Heaviside signal at the input](../systems_time_domain/network_analysis.ipynb#Step-Response).

#### Output signal including initial values

Now the analysis is performed for non-zero initial values. As initial values, the normalized voltage $u_\text{C}(0+) = -1$ at the capacitor and the normalized current $i_\text{L}(0+) = 0$ at the inductor is assumed. The network is analyzed again using Kirchhoff's circuit laws, but now the initial values are not discarded. Using the Laplace domain representation of the network elements from above table, this results in

\begin{align}
Y(s) &= \underbrace{\frac{1}{L C s^2 + R C s + 1} \cdot X(s)}_{Y_\text{ext}(s)} \\
&+ \underbrace{\frac{R C + L C s}{L C s^2 + R C s + 1} \cdot y(0+) + \frac{L}{L C s^2 + R C s + 1} \cdot i_\text{L}(0+)}_{Y_\text{int}(s)}
\end{align}

where the fact has been used that the initial voltage $u_\text{C}(0+)$ at the capacitor is equal to the inital value of the output $y(0+)$. The index for the current $i_\text{L}(t)$ at the inductor is discarded for brevity. The terms have been sorted with respect to their dependence on the input $X(s)$, and the initial values $y(0+)$ and $i(0+)$. The part of the output signal which depends only on the input is termed as *external* part $Y_\text{ext}(s)$. The parts of the output signal which depend only on the initial values are termed as *internal* parts $Y_\text{int}(s)$. The output signal is given as superposition of both contributions

\begin{equation}
y(t) = y_\text{ext}(t) + y_\text{int}(t)
\end{equation}

where $y_\text{ext}(t) = \mathcal{L}^{-1} \{ Y_\text{ext}(s) \}$ and $y_\text{int}(t) = \mathcal{L}^{-1} \{ Y_\text{int}(s) \}$.

The external part of the output signal has already been computed in the previous section

```python
yext = y
yext.simplify()
```

The Laplace transform of the internal part $Y_\text{int}(s)$ is defined for evaluation

```python
i0, y0 = sym.symbols('i0 y0', real=True)

Yint = (R*C + L*C*s) / (L*C*s**2 + R*C*s + 1) * y0 + L / (L*C*s**2 + R*C*s + 1) * i0
Yint
```

Now the inverse Laplace transform is computed for the initial values $y(0+)$ and $i(0+)$, and the specific values of $R$, $L$ and $C$ given above

```python
yint = sym.inverse_laplace_transform(Yint.subs(RLC).subs(i0, 0).subs(y0, -1), s, t)
yint
```

The output signal $y(t)$ is given as superposition of the external and internal contribution

```python
y = yext + yint
y.simplify()
```

The internal $y_\text{int}(t)$ (green line) and external $y_\text{ext}(t)$ (blue line) part, as well as the output signal $y(t)$ (red line) is plotted for illustration

```python
p1 = sym.plot(yext, (t, 0, 5), line_color='b', xlabel='$t$', ylabel='$y(t)$', show=False)
p2 = sym.plot(yint, (t, 0, 5), line_color='g', show=False)
p3 = sym.plot(y, (t, 0, 5), line_color='r', show=False)
p1.extend(p2)
p1.extend(p3)
p1.show()
```

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


```python
%%html
<!--Script block to left align Markdown Tables-->
<style>
  table {margin-left: 0 !important;}
</style>
```

##### Notes

Leave script block above in place to left justify the table.
This problem can also be used as laboratory exercise in `matplotlib` lesson.
Dependencies: `matplotlib` and `math`; could also be solved using `numpy` and/or `pandas`

```python

```

```python

```

# Problem XX

Graphing Functions Special Functions

Consider the two functions listed below:

\begin{equation}
f(x) = e^{-\alpha x}
\label{eqn:fofx}
\end{equation}

\begin{equation}
g(x) = \gamma sin(\beta  x)
\label{eqn:gofx}
\end{equation}

Prepare a plot of the two functions on the same graph.

Use the values in Table below for $\alpha$, $\beta$, and $\gamma$.

|Parameter|Value|
|:---|---:|
|$\alpha$|0.50|
|$\beta$|3.00|
|$\gamma$|$\frac{\pi}{2}$|

The plot should have $x$ values ranging from $0$ to $10$ (inclusive) in sufficiently small increments to see curvature in the two functions as well as to identify the number and approximate locations of intersections. In this problem, intersections are locations in the $x-y$ plane where the two "curves" cross one another of the two plots.

```python
# By-hand evaluate f(x) for x=1, alpha = 1/2
```

```python
# By-hand evaluate g(x) for x=3.14, beta = 1/2, gamma = 2
```

```python
# Define the first function f(x,alpha), test the function using your by hand answer
def f(x,alpha):
    import math
    f = math.exp(-1.0*alpha*x)
    return f

f(1,0.5)
```

    0.6065306597126334

```python
# Define the second function g(x,beta,gamma), test the function using your by hand answer
def g(x,beta,gamma):
    import math
    f = gamma*math.sin(beta*x)
    return f

g(3.14,0.5,2.0)
```

    1.9999993658636692

```python
# Built a list for x that ranges from 0 to 10, inclusive, with adjustable step sizes for plotting later on
howMany = 10
scale = 10.0/howMany
xvector = []
for i in range(0,howMany+1):
    xvector.append(scale*i)
#xvector # activate to display
```

```python
# Build a plotting function that plots both functions on the same chart
alpha = 0.5
beta = 0.5
gamma = 2.0
yf = []
yg = []
for i in range(0,howMany+1):
    yf.append(f(xvector[i],alpha))
    yg.append(g(xvector[i],beta,gamma))

def plot2lines(list11,list21,list12,list22,strx,stry,strtitle): # plot list1 on x, list2 on y, xlabel, ylabel, title
    from matplotlib import pyplot as plt # import the plotting library from matplotlibplt.show()
    plt.plot( list11, list21, color ='green', marker ='o', linestyle ='none' , label = "Observed" ) # create a line chart, years on x-axis, gdp on y-axis
    plt.plot( list12, list22, color ='red', marker ='o', linestyle ='solid' , label = "Model") # create a line chart, years on x-axis, gdp on y-axis
    plt.legend()
    plt.title(strtitle)# add a title
    plt.ylabel(stry)# add a label to the x and y-axes
    plt.xlabel(strx)
    plt.show() # display the plot
    return #null return

plot2lines(xvector,yf,xvector,yg,'x-value','y-value','plot of f and g')
```

```python

```

```python

```


# Problem Set 106

Complete the problems described below.

Submit your work via email. A special email address has been setup for this purpose, and you can attach whatever files are appropriate to any mail sent to the address below. So I can tell which files below to whom, please **title all submitted files as**:

    LASTNAME firstname.py

Submit this file to the class file submission email:

    SUBMITT.6g9n8dakvpw0bkft@u.box.com

```python
from decodes.core import *
from decodes.io.jupyter_out import JupyterOut
out = JupyterOut.two_pi( )
from math import *
```

## Lines and Planes in Cartesian Space

#### 106.01

We may plot a point $P$ on a given line if provided three quantities: an origin Point $\vec{p_{0}}$, a Vec direction $\vec{v}$, and a parameter along the line at which to find the desired Point $t$. In vector notation, this may be written as follows. In each of the scenarios below, using just the wits that your mother gave you (no coding), find the value of $P$

\begin{align}
P = \vec{p_{0}} + t\vec{v}
\end{align}

Note that in the code shown below, for scenario x, `p_origin_x` is the origin Point $\vec{p_{0}}$, `vec_x` is the Vec $\vec{v}$, and `t_x` is the required parameter.

```python
p_origin_a, vec_a, t_a = Point(), Vec(1,0), 1
# construct a new point here using coordinates corresponding to a point that results from the evaluation of the line above
pa = ???

p_origin_b, vec_b, t_b = Point(), Vec(1,1), 0.5
# construct a new point here using coordinates corresponding to a point that results from the evaluation of the line above
pb = ???

p_origin_c, vec_c, t_c = Point(20,20), Vec(0,-2), 1
# construct a new point here using coordinates corresponding to a point that results from the evaluation of the line above
pc = ???

p_origin_d, vec_d, t_d = Point(0,0,10), Vec(0,0,10), -2
# construct a new point here using coordinates corresponding to a point that results from the evaluation of the line above
pd = ???
```

## Lines and Planes in Decod.es

#### 106.02

In the space provided, author a function that converts any given vertically-oriented Decod.es Plane (that is, any plane that is perpendicular to the x-y plane) to a Decod.es CS. Instead of requiring a guide Vec, as demonstrated in the text, take advantage of the fact that the Plane is known to be vertical. Your function should successfully run so long as this condition holds.

```python
pt_rand = Point.random(Interval(-3,3))
vec_rand = Vec.random(Interval(-1,1),True)

pln_vert = Plane(pt_rand,vec_rand)

#### START YOUR WORK HERE ####
def to_cs(pln):
    ???
#### END YOUR WORK HERE ####

cs = to_cs(pln_vert)
print(cs)
```

#### 106.03

In our discussion of Planes, we present the Decod.es Circle as a special type of Plane which adds a single member, `rad` that defines the radius of the circle that lies on the underlying plane. Here we follow in this pattern in the construction of a Rectangle class as a special type of CS.

In the space provided, define a new class Rectangle as an extension of CS. Rectangle should be initialzed with arguments as shown below, and go on to add just two members to its parent: `w` as the span of the rectangle along the x-axis of the underlying CS, and `h` as the span along the y-axis.

Four properties should be defined, `pa`,`pa`,`pc`,and `pd`, that describe the corner points of the rectangle in world space. A single method should be added that returns the four Segments along the boundary of the Rectangle. This method will be called as shown below.

```python
"""
Rectangle Class
Inherits all properties of the Plane class, and also happens to require a Plane
in its constructor
"""
class Rectangle(CS):
    def __init__(self,cs,width,height):
        super(Rectangle,self).__init__(cs.origin, cs.x_axis, cs.y_axis)
#### START YOUR WORK HERE ####

    ???

#### END YOUR WORK HERE ####

rect = Rectangle(CS(),1.5,1.5)

for seg in rect.to_segs():
    out.put(seg)
out.draw()
out.clear()
```

## Methods of Projection

#### 106.04

Since Lines are infinite, Points may be properly projected at any position along them. This does not hold true for Rays and Segments, which each offer modificaitons of the generic infinite-line projection routine as to ensure that projected Points fall within the valid ranges of the parameter $t$ for each of these finite types: $0\rightarrow\infty$ for Rays, and $0\rightarrow1$ for Segments. Here we imagine a different scenario, and this question asks for the invention a new type of LinearEntity with a different range of valid values of $t$.

In the space provided, author two functions. Each function should handle the projection of Points to a new type of LinearEntity: the ***AntiRay*** and the ***AntiSegment***. These new imaginary types hold a valid range of $t$ values that are the inverse of their more conservative cousins: $0\rightarrow-\infty$ for Rays, and the discontinious range of $-\infty\rightarrow0$ , $1\rightarrow\infty$ for Segments.

```python
"""
AntiRay Projection
"""

def near(self,proj_pt):
    line_result = self.to_line().near(proj_pt)
    ### add your work here ###
```

```python
"""
AntiSegment Projection
"""

def near(self,proj_pt):
    line_result = self.to_line().near(proj_pt)
    ### add your work here ###
```