# Spam Filtering Using Gaussian Naive Bayes
---
Use the `spambase.data` file and the Gaussian Naive Bayes algorithm to build a simple spam detection algorithm. You can get accuracies around 80% with this simple classifier that is quickly trained.

```python
%matplotlib inline
import pandas as pd
import sklearn, sklearn.model_selection, sklearn.linear_model, sklearn.metrics
import numpy as np
import matplotlib.pyplot as plt
import itertools
```

## Read the data

---

We need to read in the data from the spambase.data which is in CSV format. Pandas is perfect for this. If you look at the spambase.data file, you'll notice that there are no column headers and that the identification for spam vs. non-spam is found in column 57. So make sure you set the `header` argument to `None` and the `index_col` argument to `57`.

```python
data = pd.read_csv('~/data/spam.data/spambase.data', header=None, index_col=57)
```

## Split the data

---

Now we want to split the data into training and testing sets. Scikit-learn has a great function for this: `sklearn.model_selection.train_test_split()`. The first parameter is the data to split, `data` in my case. We need to make sure to tell it to split the training and testing set in half by passing `0.5` to the `test_size` argument. Also, set the `random_state` argument to `np.random.RandomState()` so that the data is shuffled. (This is important since `spambase.data` is sorted by class).

```python
X_train, X_test = sklearn.model_selection.train_test_split(data, test_size=0.5, random_state = np.random.RandomState())
```

## Get the prior probabilities

---

Note that the `spambase.data` file has approximately 40% of the emails categorized as spam, so our split should reflect that. Since the `train_test_split()` function returned our split training and testing sets as pandas dataframes and since we set the index column to be the class of the instance, when we say `X_train.loc[1]` we are selecting all of the rows in the training set that have class 1. That is, select all the spam instances. Similarly, we can select all the non-spam (or ham) instances by doing `X_train.loc[0]`.

```python
prob_spam_train = len(X_train.loc[1].index)/len(X_train.index)
prob_spam_train
```

    0.39304347826086955

```python
prob_ham_train = len(X_train.loc[0].index) / len(X_train.index)
prob_ham_train
```

    0.6069565217391304

## Get the means and the standard deviations over the columns

---

In order to calculate the probability (which we simulate with the probability density function), we need to have the mean and standard deviation of each column for a given class (spam or ham). This is made quite easy by using the pandas [`describe()` function](http://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.describe.html). The result of `describe()` is a dataframe and to get the column means or standard deviations, we merely need to index the row labeled `mean` or `std` respectively.

```python
train_spam_col_means = X_train.loc[1].describe().loc['mean']
```

```python
train_spam_col_stds = X_train.loc[1].describe().loc['std']
```

```python
train_ham_col_means = X_train.loc[0].describe().loc['mean']
```

```python
train_ham_col_stds = X_train.loc[0].describe().loc['std']
```

## Define a function to calculate the sum of the log of the probabilities

---

This function returns the result of the following equation:

$$\log{P(\textit{class})} + \sum_i \log{P(x_i \mid \textit{class})}$$

Where

$$P(x_i \mid c_j) = N(x_i ; \mu_{i,c_j} , \sigma_{i, c_j})$$

and

$$N(x; \mu , \sigma) = \frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}$$

Keep in mind, though, that we will be taking the natural log of this equation. So given the log rules we will have

\begin{align}
    \log{N(x; \mu , \sigma)} &= \log{\Bigg(\frac{1}{\sqrt{2\pi}\sigma}e^{-\frac{(x-\mu)^2}{2\sigma^2}}\Bigg)} \\
    &= \log{\Bigg(\frac{1}{\sqrt{2\pi}\sigma}\Bigg)} + \log{\Bigg(e^{-\frac{(x-\mu)^2}{2\sigma^2}}\Bigg)} \\
    &= \log{\Bigg(\frac{1}{\sqrt{2\pi}}\Bigg)} + \log{\Bigg(\frac{1}{\sigma}\Bigg)} - \frac{(x-\mu)^2}{2\sigma^2} \\
    &= -\log{\bigg(\sqrt{2\pi}\bigg)} - \log{(\sigma)} - \frac{(x-\mu)^2}{2\sigma^2}
\end{align}

Also, since $-\log{\big(\sqrt{2\pi}\big)}$ is a constant, including it won't affect our results. So we only need to calculate the following $$-\sum_i \log{(\sigma_{i,c_j})} + \frac{(x_i-\mu_{i,c_j})^2}{2\sigma_{i, c_j}^2}$$

Again, Pandas makes this easy. It'll be the case that `x`, `mus`, and `sigmas` are all pandas dataframes with 1 row and 57 columns, so we can add, subtract, multiply, eponentiate, log, or divide on an item-by-item basis. So `x - mus` is also a 1 x 57 dataframe, for example. Additionally, the pandas `sum()` function ignores `NaN` values by default, so there isn't anything we need to do to avoid them (i.e. when the std is very large).

```python
def sum_of_log_probs(P_class, x, mus, sigmas):
    return np.log(P_class) - (np.log(sigmas) + ((x - mus)**2 / (2 * sigmas**2))).sum()
```

## Given an instance $\mathbf{\overrightarrow{x}}$, predict its class

---

Determine a class of an instance. Calculate the following

$$class_{NB}\big(\overrightarrow{\mathbf{x}}\big) = \underset{\textit{class} \in \{0, 1\}}{\mathrm{argmax}} \Big(\log{P(\textit{class})} + \sum_i \log{P(x_i \text{ | } \textit{class})} \Big)$$

where $P(x_i \mid c_j)$ is as given previously.

```python
def predict(x):
    p_spam = sum_of_log_probs(prob_spam_train, x, train_spam_col_means, train_spam_col_stds)
    p_ham = sum_of_log_probs(prob_ham_train, x, train_ham_col_means, train_ham_col_stds)
    return 1 if p_spam > p_ham else 0
```

## Classify all instances in the test set

---

Walk over the test set predicting each of the instances. Here's another time when pandas makes this nice. The `pandas.DataFrame.iterrows()` function returns a tuple `(index, row dataframe)`. So we have the actual value, which is stored in the index column since that's what was set by the `index_col` argument of the `pandas.read_csv()` function.

*Note: `numpy.log()` will return a runtime warning if the value it gets is equal to 0.0. It is fine for us to ignore this.*

```python
NB_y_pred = X_test.apply(predict, axis=1)
```

    /usr/local/var/pyenv/versions/3.6.0/envs/mlcv360/lib/python3.6/site-packages/ipykernel/__main__.py:2: RuntimeWarning: divide by zero encountered in log
      from ipykernel import kernelapp as app

```python
NB_confusion_matrix = sklearn.metrics.confusion_matrix(y_true=X_test.index.values, y_pred=NB_y_pred)
NB_accuracy = sklearn.metrics.accuracy_score(y_true=X_test.index.values, y_pred=NB_y_pred)
NB_precision = sklearn.metrics.precision_score(y_true=X_test.index.values, y_pred=NB_y_pred)
NB_recall = sklearn.metrics.recall_score(y_true=X_test.index.values, y_pred=NB_y_pred)
print(' accuracy: {:.5f}\n'
      'precision: {:.5f}\n'
      '   recall: {:.5f}'.format(NB_accuracy, NB_precision, NB_recall))
```

     accuracy: 0.82225
    precision: 0.70695
       recall: 0.93949

```python
def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    plt.figure(figsize=(7,7))
    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title, fontsize=24)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]

    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, cm[i, j],
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('Actual label', fontsize=14)
    plt.xlabel('Predicted label', fontsize=14)
    plt.style.use('seaborn-dark')
```

## Plot the confusion matrix

```python
plot_confusion_matrix(NB_confusion_matrix, classes=('ham', 'spam'), title='Confusion matrix')
```

## Part 2: Using a library to run a logistic regressor

---

This is simply a matter of running sklearn's LogisticRegression() and the various sklearn metrics functions.

```python
logistic_regressor = sklearn.linear_model.LogisticRegression().fit(X_train, X_train.index.values)
LR_y_pred = logistic_regressor.predict(X_test)
```

```python
LR_accuracy = sklearn.metrics.accuracy_score(y_true=X_test.index.values, y_pred=LR_y_pred)
LR_precision = sklearn.metrics.precision_score(y_true=X_test.index.values, y_pred=LR_y_pred)
LR_recall = sklearn.metrics.recall_score(y_true=X_test.index.values, y_pred=LR_y_pred)
print(' accuracy: {:.4f}\n'
      'precision: {:.4f}\n'
      '   recall: {:.4f}'.format(LR_accuracy, LR_precision, LR_recall))
```

     accuracy: 0.9218
    precision: 0.9147
       recall: 0.8845

```python
LR_confusion_matrix = sklearn.metrics.confusion_matrix(y_true=X_test.index.values, y_pred=LR_y_pred)
```

```python
plot_confusion_matrix(LR_confusion_matrix, classes=('ham', 'spam'), title='Confusion Matrix')
```

```python

```


**Exercise set 7**
==============

> The goal of this exercise is to run PCA and PLSR on a real
data set in order to show how these methods can be used in practice.
We are considering data that are given by
[Platikanov et al.](https://doi.org/10.1016/j.watres.2012.10.040) and we are aiming to
reproduce some of the results found in their work. You may find it useful to
inspect this article when interpreting your results.

**Exercise 7.1**

In this exercise, we will consider two data sets: one set contains data for bottled mineral water
(we will call this "data set 1"),
and the other set contains data for tap water and bottled water (we will call this "data set 2").

The different water samples have been blind-tasted
by trained panelists, who have rated the different water samples
according to their liking. Also, the chemical compositions of the
samples have been determined. All measured quantities are given
in table 1.

Our end goal is to predict the ratings (the "mean liking" in
table 1) of the water samples, given
measurements of the chemical composition. We will define this rating as the
$y$-variable we are going to predict.

|Variable| Unit|
|:-------|----:|
 |Conductivity                       | ($\mu$S/cm)  |
 |TDS (total dissolved solids)       | (mg/L)       |
 |Cl$^{-}$                            | (mg/L)      |
 |SO$_4^{2-}$                         | (mg/L)      |
 |NO$_3^-$                            | (mg/L)      |
 |HCO$_3^-$                           | (mg/L)      |
 |Ca$^{2+}$                           | (mg/L)      |
 |Mg$^{2+}$                           | (mg/L)      |
 |Na$^{+}$                            | (mg/L)      |
 |K$^{+}$                             | (mg/L)      |
 |pH                                  |             |
 |Si                                  | (mg/L)      |
 |Cl$_2$                              | (mg/L)      |
 |Mean liking                         |             |
 | **Table 1:** *Data columns present in the data sets: [Data/table1.csv](Data/table1.csv) and  [Data/table2.csv](Data/table2.csv)* |

**(a)** Begin by exploring the raw data. In the [original article](https://doi.org/10.1016/j.watres.2012.10.040),
      you can find correlation heat maps (see Fig. $1$ in the aforementioned article).
      Create such correlation maps yourself (for data set 1 & 2)
      and compare them with Fig. $1$ in the article. Does the mean liking
      seem to be correlated with some of the variables?

```python
# Your code here
```

**Your answer to question 7.1(a):** *Double click here*

**(b)** We will further explore the raw data with PCA. Perform a principal
      component analysis for data set 1 and for data set 2. How much of the variance is
      explained by the first two principal components? Also, plot the scores
      and the loadings for principal component 1 and 2, and for principal component 1 and 3.
      Does any of the variables seem to be correlated?
      Which variables seem to influence the mean liking most
      for the two data sets?

```python
# Your code here
```

**Your answer to question 7.1(b):** *Double click here*

**(c)** Let us start the modeling by creating a linear least-squares model:

* (i) Create linear least-squares models for the two data sets in which you use all available data in the fitting. Calculate $R^2$ and the root mean squared error (RMSE) of your models. Comment on the values you have obtained.

* (ii)  Evaluate the root mean squared error of cross-validation (RMSECV) for your two models. Compare the RMSECV values with the previously obtained RMSE values.

* (iii) Using the two models you have created, what variables seem to be most important for predicting a high mean liking? Here, you can inspect the regression coefficients (assuming that you have scaled the variables). If you were to create a new brand of bottled water, what chemical components would you focus on to maximize the mean liking?

```python
# Your code here
```

**Your answer to question 7.1(c):** *Double click here*

**(d)** We will now consider partial least-squares regression (PLSR) models.
Before we do the actual modeling, let us repeat the fundamental
equations for PLSR. The model itself is based on the following two
equations:
\begin{equation}
         \begin{split}
         \mathbf{X} &= \mathbf{T} \mathbf{P}^\top, \\
         \mathbf{Y} &= \mathbf{U} \mathbf{Q}^\top, \\
         \end{split}
         \tag{1}
      \end{equation}
      where $\mathbf{T}$ is the $x$-scores, $\mathbf{P}$ is the $x$-loadings,
      $\mathbf{U}$ is the $y$-scores, and $\mathbf{Q}$ is the $y$-loadings.
      The linear relation between $\mathbf{X}$ and $\mathbf{Y}$ is in this
      case given by,
      \begin{equation}
          \mathbf{Y} = \mathbf{T} \mathbf{Q}^\top.
          \label{eq:plsrreg} \tag{2}
      \end{equation}

When we wish to *predict* new $\mathbf{Y}$-values from new
      $\mathbf{X}$-values, we need to calculate new $x$-scores. This is
      done by introducing an additional matrix, $\mathbf{R}$, so that
      the following is satisfied:
      \begin{equation}
            \mathbf{T} = \mathbf{X} \mathbf{R}.
          \label{eq:plsrscores} \tag{3}
      \end{equation}
      Here, we can think of the $\mathbf{R}$ as a matrix we can use to invert
      the relation $\mathbf{X} = \mathbf{T} \mathbf{P}^\top$. We can then predict
      new $\mathbf{Y}$-values, by combining Eq. \eqref{eq:plsrreg}
      and Eq. \eqref{eq:plsrscores}:
      \begin{equation}
        \mathbf{Y} = \mathbf{T} \mathbf{Q}^\top =
                     \mathbf{X} \mathbf{R} \mathbf{Q}^\top =
                     \mathbf{X} \mathbf{B}_\text{PLS},
                     \tag{4}
      \end{equation}
      where the regression coefficients $\mathbf{B}_\text{PLS}$ are given by:
      \begin{equation}
        \mathbf{B}_\text{PLS}  = \mathbf{R} \mathbf{Q}^\top. \tag{5}
      \end{equation}

In `sklearn` we can run PLSR by using `PLSRegression`
which is found in the module `sklearn.cross_decomposition`.
After running the regression, we can access the matrices given above with the python code below.

```python
from sklearn.preprocessing import scale
from sklearn.cross_decomposition import PLSRegression
X = scale(X)
y = scale(y)
plsr = PLSRegression(n_components=2)
plsr.fit(X, y)
R = plsr.x_rotations_
B = plsr.coef_
Q = plsr.y_loadings_
```

* (i) Create PLSR models for the two data sets in which you use all available data in the fitting. Use
  only two components when you create the model, that is, set `PLSRegression(n_components=2)` when
  you set up the models. Calculate $R^2$ and the root mean squared error (RMSE) of your models.
  Comment on the values you have obtained, and compare them with the corresponding values from your
  linear least-squares models.

* (ii) Calculate RMSECV for your two PLSR models. Compare the RMSECV values with the previously obtained
  RMSE values. Would you say that your PLSR models perform better or worse than the least-squares
  models?

* (iii) Plot the $x$- and $y$-loadings for the two components. These loadings are available as
  `Q = plsr.y_loadings_`. What variables seem to be most important for predicting the mean liking?

* (iv) Inspect the weights (the $\mathbf{R}$ matrix) for PLS component 1 and PLS component 2.
  Compare your results to the results given by
  [Platikanov et al.](https://doi.org/10.1016/j.watres.2012.10.040) in Fig. $6$.

* (v) Based on the PLSR results: If you were to create
  a new brand of bottled water, what chemical components
  would you focus on to maximize the mean liking?

```python
# Your code here
```

**Your answer to question 7.1(d):** *Double click here*


# 整数规划 Integer Programming

xyfJASON

## 1 引入 `0-1` 变量的场景

### 1.1 多选一约束

例如有如下两个约束：

$$
5x_1+4x_2\leqslant 24\quad\text{or}\quad 7x_1+3x_2\leqslant 45
$$

满足其中一个约束即可。为了统一起来，引入 0-1 变量：

$$
y=\begin{cases}0&选第一个约束\\1&选第二个约束\end{cases}
$$

那么约束条件可改写为：

$$
\begin{cases}5x_1+4x_2\leqslant 24+yM\\7x_1+3x_2\leqslant 45+(1-y)M\end{cases}
$$

其中 $M$ 是充分大的正常数。

### 1.2 多选多约束

例如有如下三个约束，要求满足至少两个约束：

$$
2x+3y\leqslant 100,\quad x+y\leqslant 50,\quad x+2y\leqslant 80
$$

那么引入三个 0-1 变量 $z_1,z_2,z_3$ 分别表示选第 $1,2,3$ 个约束，则约束条件可改写为：

$$
\begin{cases}
2x+3y\leqslant 100+(1-z_1)M\\
x+y\leqslant 50+(1-z_2)M\\
x+2y\leqslant 80+(1-z_3)M\\
z_1+z_2+z_3=2
\end{cases}
$$

其中 $M$ 是充分大的正常数。

### 1.3 分段线性函数

例如目标函数是：

$$
z=\begin{cases}
3+4x&0\leqslant x< 2\\
15-2x&2\leqslant x<3\\
6+x&3\leqslant x<7
\end{cases}
$$

其通用的一种建模技巧如下：

设 $n$ 段线性函数 $f(x)$ 分点为 $b_1<b_2<\cdots<b_{n+1}$，引入变量 $w_k$ 和 0-1 变量 $z_k$ 满足：

$$
\begin{cases}
w_1\leqslant z_1,\,w_2\leqslant z_1+z_2,\,\cdots,\,w_n\leqslant z_{n-1}+z_n,\,w_{n+1}\leqslant z_n\\
z_1+z_2+\cdots+z_n=1,\quad z_k\in\{0,1\}\\
w_1+w_2+\cdots+w_{n+1}=1,\quad w_k\geqslant 0
\end{cases}
$$

那么 $x$ 和 $f(x)$ 就可以表示如下：

$$
\begin{align}
x&=\sum_{k=1}^{n+1}w_kb_k\\
f(x)&=\sum_{k=1}^{n+1}w_kf(b_k)
\end{align}
$$

### 1.4 固定费用问题

设目标函数中存在这样的项：

$$
P=\begin{cases}k+cx&x>0\\0&x=0\end{cases}
$$

那么引入 0-1 变量 $y=[x>0]$，则添加如下条件：

$$
y\varepsilon\leqslant x\leqslant yM
$$

其中 $\varepsilon$ 为充分小正常数，$M$ 为充分大正常数。现在目标函数中 $P$ 这一项只写作 $k+cx$ 即可。

## 2 算法

**非线性整数规划**没有通用的算法，事实上**非线性规划**都没有通用解法。

下列方法中，蒙特卡洛法可用于求解非线性整数规划，其余算法仅针对**线性整数规划**。

本文默认最小化目标函数，不等式约束条件均为小于等于。

### 2.1 分支定界法——整数规划（纯/混合）

忽略整数条件，单纯形法得到解作为原问题的上界。

如果当前解恰是整数解，则问题解决；否则，选取一个不满足整数条件的变量 $x_i$，设当前解中它的值为小数 $x_i^*$，则分别添加约束条件 $x\leqslant \lfloor x_i^*\rfloor$ 和 $x\geqslant \lceil x_i^*\rceil$ 得到两个新问题，分别单纯形法求解（分支）。

对于一个新问题而言，如果它是整数解，则更新原问题的下界；如果它不是整数解，若它比当前下界更优，则更新原问题的上界（定界）；否则裁剪该分支（剪枝）。然后选取一个未被裁剪的分支，重复上述步骤，直至上下界相等。

### 2.2 隐枚举法——0-1整数规划

对暴力枚举的优化，在得到一个可行解之后，容易知道更优的解必须小于该可行解，这提供了一个强力的剪枝。

### 2.3 匈牙利算法——指派问题（0-1整数规划特例）

指派问题，可以看作 0-1 规划问题求解，也可以看作二分图匹配问题，使用匈牙利算法求解。

不过 `scipy` 已经提供了指派问题的求解函数：`scipy.optimize.linear_sum_assignment`，所以我们不必实现。

Documentation: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.linear_sum_assignment.html

### 2.4 蒙特卡洛法——非线性整数规划

蒙特卡洛法就是随机取样法，随机从可行域中取点、代入、更新答案，取点充分多后能够接近最优解。

## 3 代码模板

分支定界法和隐枚举法的模板见同文件夹下的 `programming.py` 模块。

## 4 例题

### 4.1 例一

求解指派问题，已知指派矩阵如下：

$$
\begin{bmatrix}
3&8&2&10&3\\8&7&2&9&7\\6&4&2&7&5\\8&4&2&3&5\\9&10&6&9&10
\end{bmatrix}
$$

指派问题自然可以直接用 `linear_sum_assignment` 求解，但也可以化为如下整数规划问题：

$$
\min\quad
\begin{align}
&3x_{11}+8x_{12}+2x_{13}+10x_{14}+3x_{15}\\
+&8x_{21}+7x_{22}+2x_{23}+9x_{24}+7x_{25}\\
+&6x_{31}+4x_{32}+2x_{33}+7x_{34}+5x_{35}\\
+&8x_{41}+4x_{42}+2x_{43}+3x_{44}+5x_{45}\\
+&9x_{51}+10x_{52}+6x_{53}+9x_{54}+10x_{55}
\end{align}\\
\text{s.t.}\begin{cases}
\sum\limits_{j=1}^5x_{ij}=1&\forall i\in\{1,2,3,4,5\}\\
\sum\limits_{i=1}^5x_{ij}=1&\forall j\in\{1,2,3,4,5\}\quad\quad\quad\\
x_{ij}\in\{0,1\}&\forall i,j\in\{1,2,3,4,5\}
\end{cases}
$$

编写代码如下：

```python
import numpy as np
from scipy.optimize import linear_sum_assignment
from programming import branch_bound, implicit_enumerate

c = np.array([[3, 8, 2, 10, 3],
              [8, 7, 2, 9, 7],
              [6, 4, 2, 7, 5],
              [8, 4, 2, 3, 5],
              [9, 10, 6, 9, 10]])
A_eq = np.vstack((
    np.concatenate(([1] * 5, [0] * 20)),
    np.concatenate(([0] * 5, [1] * 5, [0] * 15)),
    np.concatenate(([0] * 10, [1] * 5, [0] * 10)),
    np.concatenate(([0] * 15, [1] * 5, [0] * 5)),
    np.concatenate(([0] * 20, [1] * 5)),
    np.array([1, 0, 0, 0, 0] * 5),
    np.array([0, 1, 0, 0, 0] * 5),
    np.array([0, 0, 1, 0, 0] * 5),
    np.array([0, 0, 0, 1, 0] * 5),
    np.array([0, 0, 0, 0, 1] * 5)
))
b_eq = np.array([1] * 10)
bounds = [(0, 1)] * 25
is_int = np.array([True] * 25)

row_id, col_id = linear_sum_assignment(c)
print(c[row_id, col_id].sum())  # 21
print('-'*30)

res = branch_bound(c=c.flatten(), A_eq=A_eq, b_eq=b_eq, bounds=bounds, is_int=is_int)
print(res)  # 21
print('-'*30)

res = implicit_enumerate(c=c.flatten(), A_eq=A_eq, b_eq=b_eq, known_solution=np.diag(np.ones(5)).flatten())
print(res)  # 21
```

    21
    ------------------------------
         con: array([0., 0., 0., 0., 0., 0., 0., 0., 0., 0.])
         fun: 21.0
     message: 'Optimization terminated successfully.'
         nit: 35
       slack: array([], dtype=float64)
      status: 0
     success: True
           x: array([0., 0., 0., 0., 1., 0., 0., 1., 0., 0., 0., 1., 0., 0., 0., 0., 0.,
           0., 1., 0., 1., 0., 0., 0., 0.])
    ------------------------------

    /Users/jason/Desktop/Mathematical-Modeling-with-Python/Mathematical Programming/programming.py:25: OptimizeWarning: A_eq does not appear to be of full row rank. To improve performance, check the problem formulation for redundant equality constraints.
      res = linprog(c, A_ub, b_ub, A_eq, b_eq, cur, method)

    (array([0, 0, 0, 0, 1, 0, 0, 1, 0, 0, 0, 1, 0, 0, 0, 0, 0, 0, 1, 0, 1, 0,
           0, 0, 0]), 21)

### 4.2 例二

求解：

$$
\begin{align}
&\min z=-3x_1-2x_2-x_1\\
&\text{s.t.}\begin{cases}x_1+x_2+x_3\leqslant 7\\
4x_1+2x_2+x_3=12\\
x_1,x_2\geqslant 0\\
x_3\in\{0,1\}\end{cases}
\end{align}
$$

这是一个混合整数规划问题，可以用分支定界法求解：

```python
c = np.array([-3, -2, -1])
A_ub = np.array([[1, 1, 1]])
b_ub = np.array([7])
A_eq = np.array([[4, 2, 1]])
b_eq = np.array([12])
bounds = [(0, None), (0, None), (0, 1)]
is_int = np.array([False, False, True])

res = branch_bound(c, A_ub, b_ub, A_eq, b_eq, bounds=bounds, is_int=is_int)
print(res)  # -12
```

         con: array([0.])
         fun: -12.0
     message: 'Optimization terminated successfully.'
         nit: 2
       slack: array([1.])
      status: 0
     success: True
           x: array([0., 6., 0.])

### 4.3 例三

求解：

$$
\begin{align}
&\max z=x_1+x_1x_2-x_3\\
&\text{s.t.}\begin{cases}
-2x_1+3x_2+x_3\leqslant 3\\x_j=0\text{ or } 1,\,j=1,2,3\end{cases}
\end{align}
$$

这是一个非线性的 $0-1$ 规划问题，但是只需要做变换 $y=x_1x_2$，就可以转换成线性的 $0-1$ 规划问题：

$$
\begin{align}
&\max z=x_1+y-x_3\\
&\text{s.t.}\begin{cases}
-2x_1+3x_2+x_3\leqslant 3\\
x_1+x_2-1\leqslant y\leqslant x_1\\
x_1+x_2-1\leqslant y\leqslant x_2\\
x_j=0\text{ or }1\\
y=0\text{ or }1
\end{cases}
\end{align}
$$

可以用分支定界法或隐枚举法求解：

```python
c = np.array([-1, 0, 1, -1])
A_ub = np.array([[-2, 3, 1, 0],
                 [1, 1, 0, -1],
                 [-1, 0, 0, 1],
                 [0, -1, 0, 1]])
b_ub = np.array([3, 1, 0, 0])
bounds = [(0, 1), (0, 1), (0, 1), (0, 1)]
is_int = np.array([True, True, True, True])

res = branch_bound(c, A_ub=A_ub, b_ub=b_ub, bounds=bounds, is_int=is_int)
print(res)  # -2
print('-'*30)

res = implicit_enumerate(c, A_ub=A_ub, b_ub=b_ub, known_solution=np.array([0, 0, 0, 0]))
print(res)  # -2
```

         con: array([], dtype=float64)
         fun: -2.0
     message: 'Optimization terminated successfully.'
         nit: 3
       slack: array([2., 0., 0., 0.])
      status: 0
     success: True
           x: array([1., 1., 0., 1.])
    ------------------------------
    (array([1, 1, 0, 1]), -2)


```python
from sklearn.datasets import make_circles
from sklearn.datasets import load_digits
from mpl_toolkits.mplot3d import Axes3D
from ipywidgets import interact, interactive, fixed, interact_manual, FloatSlider, IntSlider

import matplotlib.pyplot as plt
from sklearn.linear_model import Perceptron
import numpy as np
import pandas as pd

from itertools import product

from sklearn import datasets
from sklearn.tree import DecisionTreeClassifier
from sklearn.neighbors import KNeighborsClassifier
from sklearn.svm import SVC
from sklearn.ensemble import VotingClassifier
```

## Question 3) Kernel Perceptrons
Previously, you've used perceptrons for binary classification problems. We've already seen in this assignment that we can apply kernels to PCA, but the potential of kernels is limitless. In this question, we'll kernalize perceptrons to express richer decision boundaries than the simple decision boundaries you've seen thus far.

### Question 3a)
Run the following code to visualize our dataset, and answer the following questions.
* **Is the data linearly separable?**

```python
X, y = make_circles(n_samples=400, factor=.3, noise=.12)
plt.scatter(X[:,0], X[:,1], c=y)
```

Let's try to fit a perceptron to this data. Fill in the following code to fit a Perceptron classifier `clf` to X and y.

```python
# START TODO
clf = ...
# END TODO

h = .02  # step size in the mesh
# create a mesh to plot in
x_min, x_max = X[:, 0].min() - 1, X[:, 0].max() + 1
y_min, y_max = X[:, 1].min() - 1, X[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

# Plot the decision boundary. For that, we will assign a color to each
# point in the mesh [x_min, m_max]x[y_min, y_max].
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

# Put the result into a color plot
Z = Z.reshape(xx.shape)
plt.scatter(X[:,0], X[:,1], c=y)
plt.contour(xx, yy, Z, cmap=plt.cm.Paired)
"Perceptron accuracy on original data: " + str(clf.score(X,y))
```

### Question 3b)
Now let's try lifting the data to a higher dimensional feature space so that we can classify this with a Perceptron better. Let's apply the following feature map:
\begin{equation*}
\phi (x) = [x_1, x_2, x_1^2 + x_2^2]
\end{equation*}
Fill in the following code to create a data matrix `phi` that contains our augmented data, and visualize phi.
**Hints:**
* Check out np.column_stack https://numpy.org/doc/stable/reference/generated/numpy.column_stack.html#numpy.column_stack

```python
# START TODO
phi = ...
# END TODO

fig = plt.figure(figsize=(11,8))
ax = fig.add_subplot(111, projection='3d')
ax.scatter(phi[:,0],phi[:,1],phi[:,2], c=y)

```

Answer the following questions.
* **Is the data separable now?**

Now let's try using the perceptron on this augmented dataset, and see how it performs. This is a little janky, but you can use the sliders to control the view of the 3d plot.

```python
def viz_perceptron(x_rotate, y_rotate):
    # START TODO

    clf = ...

    # START TODO

    print ("Perceptron accuracy on lifted data: " + str(clf.score(phi, y)))

    z = lambda x,y: (-clf.intercept_[0]-clf.coef_[0][0]*x -clf.coef_[0][1]*y) / clf.coef_[0][2]

    tmp = np.linspace(-5,5,30)
    x_surf,y_surf = np.meshgrid(tmp,tmp)

    fig = plt.figure(figsize=(11,8))
    ax = fig.add_subplot(111, projection='3d')
    ax.scatter(phi[:,0],phi[:,1],phi[:,2], c=y)
    ax.plot_surface(x_surf, y_surf, z(x_surf,y_surf))
    ax.view_init(y_rotate, x_rotate)

    plt.show()

x_widget = IntSlider(min=-100, max=100)
y_widget = IntSlider(min=-100, max=100)
interact(viz_perceptron,x_rotate=x_widget,y_rotate=y_widget)

```

We can now see that lifting our features can be very advantageous to making a perceptron work. At this point this should scream kernels to you. Whenever we want to work in high dimensional spaces, a kernel would certainly help. So how do we kernelize the perceptron?

### Question 3c)
In order to kernelize perceptrons, we need to think about about how perceptrons actually work in the first place. Fill in the following code for a homemade `MyPerceptron` class that trains on training data for `num_iterations`.

```python
for i in range(len(y)):
    if y[i] == 0:

        y[i] = -1

class MyPerceptron:
    def __init__(self, num_iterations = 5):
        self.num_iterations = num_iterations

    def predict(self, x):
        # START TODO

        # END TODO
    def fit(self, X, y):
        self.w = np.zeros(X.shape[1])

        for i in range(self.num_iterations):
            for x_i, y_i in zip(X, y):
                # START TODO

                # END TODO
    def score(self, X, y):
        # START TODO

        # END TODO
        accuracy = correct / X.shape[0]
        return accuracy
```

Confirm that this works by testing classification accuracy on the original dataset, it should be around 50%.

```python
clf = MyPerceptron(10)
X_with_ones = np.column_stack([np.ones(X.shape[0]), X[:,0],X[:,1]])
clf.fit(X_with_ones, y)
print("Accuracy of MyPerceptron: ", clf.score(X_with_ones,y))

```

Let's look at where the kernel perceptron algorithm comes from.
*Note: for more information visit the wikipedia page as well: https://en.wikipedia.org/wiki/Kernel_perceptron*
* **To proceed with kernalizing this, let's first write out our update step.** \begin{equation} w + y_i * x_i \end{equation}
* **Notice that w is a linear combination of our data points. w starts at zero, and only changes by getting some portion of all our data points.** (If this doesn't make sense to you read the notes once more).
* **This means that we can write our w as follows.** \begin{equation} w = \sum_{i}^{n} \alpha_i y_i \vec{x_i}  \end{equation}
* **Now let's write our actual prediction formula for a given datapoint x.**
\begin{equation} sign(w^T x) = sign((\sum_{i}^{n} \alpha_i y_i \vec x_i)^T \vec x) = sign(\sum_{i}^{n} \alpha_i y_i (\vec x_i \cdot \vec x))\end{equation}
* **Notice now that we can see an inner product in our formula for the prediction. We can replace that with a kernel function.**
\begin{equation} sign(w^T x) = sign((\sum_{i}^{n} \alpha_i y_i \vec x_i)^T \vec x) = sign(\sum_{i}^{n} \alpha_i y_i k(\vec x_i, \vec x))\end{equation}

* **You can see how the kernel function has worked its way into the prediction function. To perform our update step, we are no longer working with the weights `w`. We are now working with the weights $\alpha$ instead, so we can perform this update step:** \begin{equation} a = a + 1 \end{equation} if a point is misclassified.

Now that we understand the update step and predict formula, let's implement this regression by filling  out `MyKernelPerceptron`.

```python
class MyKernelPerceptron:
    def __init__(self, kernel_function, num_iterations = 5):
        self.num_iterations = num_iterations
        self.k = kernel_function
    def predict(self, y, X, x):
        # START TODO

        # END TODO
    def fit(self, X, y):
        self.a = np.zeros(X.shape[0])
        for iteration in range(self.num_iterations):

            for i, x_i, y_i in zip(range(X.shape[0]),X, y):
                y_hat = self.predict(y, X, x_i)
                if y_hat != y_i:
                    # START TODO

                    # END TODO
    def score(self, X, y):
        # START TODO

        # END TODO
        accuracy = correct / X.shape[0]
        return accuracy
```

Now complete the code for the `rbf_kernel` implementation with a $\gamma$ = 3.

```python
def linear_kernel(x, z):
    return x.T@z
def rbf_kernel(x,z):
    # START TODO

    # END TODO

clf = MyKernelPerceptron(rbf_kernel, 3)

clf.fit(X_with_ones, y)
print("Accuracy of MyKernelPerceptron with rbf kernel", clf.score(X_with_ones,y))

clf = MyKernelPerceptron(linear_kernel, 3)

clf.fit(X_with_ones, y)
print("Accuracy of MyKernelPerceptron with linear kernel", clf.score(X_with_ones,y))
```

Answer the following questions.
* **Which perceptron performed better?**
* **Which was faster to run, MyKernelPerceptron or MyPerceptron?**
* **When would it be preferable to use MyKernelPerceptron on the original data over MyPerceptron on lifted data?**

```python

```


# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Direct Form Structures

The output signal $y[k] = \mathcal{H} \{ x[k] \}$ of a recursive linear-time invariant (LTI) system is given by

\begin{equation}
y[k] = \frac{1}{a_0} \left( \sum_{m=0}^{M} b_m \; x[k-m] - \sum_{n=1}^{N} a_n \; y[k-n] \right)
\end{equation}

where $a_n$ and $b_m$ denote constant coefficients and $N$ the order. Note that systems with $M > N$ are in general not stable. The computational realization of above equation requires additions, multiplications, the actual and past samples of the input signal $x[k]$, and the past samples of the output signal $y[k]$. Technically this can be realized by

* adders
* multipliers, and
* unit delays or storage elements.

These can be arranged in different topologies. A certain class of structures, which is introduced in the following, is known as *direct form structures*. Other known forms are for instance [cascaded sections](cascaded_structures.ipynb), parallel sections, lattice structures and state-space structures.

For the following it is assumed that $a_0 = 1$. This can be achieved for instance by dividing the remaining coefficients by $a_0$.

### Direct Form I

The [direct form I](https://en.wikipedia.org/wiki/Digital_filter#Direct_Form_I) is derived by rearranging above equation for $a_0 = 1$

\begin{equation}
y[k] = \sum_{m=0}^{M} b_m \; x[k-m] + \sum_{n=1}^{N} - a_n \; y[k-n]
\end{equation}

It is now evident that we can realize the recursive filter by a superposition of a non-recursive and a recursive part. With the elements given above, this results in the following block-diagram

This representation is not canonical since $N + M$ unit delays are required to realize a system of order $N$. A benefit of the direct form I is that there is essentially only one summation point which has to be taken care of when considering quantized variables and overflow. The output signal $y[k]$ for the direct form I is computed by realizing above equation.

The block diagram of the direct form I can be interpreted as the cascade of two systems. Denoting the signal in between both as $w[k]$ and discarding initial values we get

\begin{align}
w[k] &= \sum_{m=0}^{M} b_m \; x[k-m] = h_1[k] * x[k] \\
y[k] &= w[k] + \sum_{n=1}^{N} - a_n \; w[k-n] = h_2[k] * w[k] = h_2[k] * h_1[k] * x[k]
\end{align}

where $h_1[k] = [b_0, b_1, \dots, b_M]$ denotes the impulse response of the non-recursive part and $h_2[k] = [1, -a_1, \dots, -a_N]$ for the recursive part. From the last equality of the second equation and the commutativity of the convolution it becomes clear that the order of the cascade can be exchanged.

### Direct Form II

The [direct form II](https://en.wikipedia.org/wiki/Digital_filter#Direct_Form_II) is yielded by exchanging the two systems in above block diagram and noticing that there are two parallel columns of delays which can be combined, since they are redundant. For $N=M$ it is given as

Other cases with $N \neq M$ can be considered for by setting coefficients to zero. This form is a canonical structure since it only requires $N$ unit delays for a recursive filter of order $N$. The output signal $y[k]$ for the direct form II is computed by the following equations

\begin{align}
w[k] &= x[k] + \sum_{n=1}^{N} - a_n \; w[k-n] \\
y[k] &= \sum_{m=0}^{M} b_m \; w[k-m]
\end{align}

The samples $w[k-m]$ are termed *state* (variables) of a digital filter.

### Transposed Direct Form II

The block diagrams above can be interpreted as linear [signal flow graphs](https://en.wikipedia.org/wiki/Signal-flow_graph). The theory of these graphs provides useful transformations into different forms which preserve the overall transfer function. Of special interest is the *transposition* or *reversal* of a graph which can be achieved by

* exchanging in- and output,
* exchanging signal split and summation points, and
* reversing the directions of the signal flows.

Applying this procedure to the direct form II shown above for $N=M$ yields the transposed direct form II

The output signal of the transposed direct form II is given as

\begin{equation}
y[k] = b_0 x[k] + \sum_{m=1}^{M} b_m x[k-n] - \sum_{n=1}^{N} a_n y[k-n]
\end{equation}

Using the signal before the $n$-th delay unit as internal state $w_n[k]$ we can reformulate this into a set of difference equations for computation of the output signal

\begin{align}
w_n[k] &=
\begin{cases}
w_{n+1}[k-1] - a_n y[k] + b_n x[k] & \text{for } n=0,1,\dots,N-1 \\
-a_N y[k]  + b_N x[k] & \text{for } n=N
\end{cases}\\
y[k] &= w_1[k-1] + b_0 x[k]
\end{align}

#### Example - Computation of output signal

The following example illustrates the computation of the impulse response $h[k]$ of a 2nd-order recursive system using the transposed direct form II as realized by `scipy.signal.lfilter`.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

p = 0.90*np.exp(-1j*np.pi/4)
a = np.poly([p, np.conj(p)])  # denominator coefficients
b = [1, 0, 0]  # numerator coefficients
N = 40  # number of computed samples

# generate input signal (= Dirac impulse)
k = np.arange(N)
x = np.where(k==0, 1.0, 0.0)

# filter signal using transposed direct form II
h = sig.lfilter(b, a, x)

# plot output signal
plt.figure(figsize=(8, 4))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
plt.axis([-1, N, -1.5, 1.5])
plt.grid()
```

**Exercise**

* Compute the step-response $h_\epsilon[k] = \mathcal{H} \{ \epsilon[k] \}$ of the filter by modifying above example.

Solution: The step response can be computed by changing the input signal to

```python
x = np.where(k>=0, 1.0, 0.0)
```

or by cumulative summation of the impulse response due to the relation $h_\epsilon[k] = \sum_{\kappa =0}^k h[k]$ of the step response to the impulse response

```python
he = np.cumsum(h)
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2018*.


# When To Stop Fuzzing

In the past chapters, we have discussed several fuzzing techniques.  Knowing _what_ to do is important, but it is also important to know when to _stop_ doing things.  In this chapter, we will learn when to _stop fuzzing_ – and use a prominent example for this purpose: The *Enigma* machine that was used in the second world war by the navy of Nazi Germany to encrypt communications, and how Alan Turing and I.J. Good used _fuzzing techniques_ to crack ciphers for the Naval Enigma machine.

Turing did not only develop the foundations of computer science, the Turing machine. Together with his assistant I.J. Good, he also invented estimators of the probability of an event occuring that has never previously occured. We show how the Good-Turing estimator can be used to quantify the *residual risk* of a fuzzing campaign that finds no vulnerabilities. Meaning, we show how it estimates the probability of discovering a vulnerability when no vulnerability has been observed before throughout the fuzzing campaign.

We discuss means to speed up [coverage-based fuzzers](Coverage.ipynb) and introduce a range of estimation and extrapolation methodologies to assess and extrapolate fuzzing progress and residual risk.

**Prerequisites**

* _The chapter on [Coverage](Coverage.ipynb) discusses how to use coverage information for an executed test input to guide a coverage-based mutational greybox fuzzer_.
* Some knowledge of statistics is helpful.

```python
import fuzzingbook_utils
```

```python
import Fuzzer
import Coverage
```

## The Enigma Machine

It is autumn in the year of 1938. Turing has just finished his PhD at Princeton University demonstrating the limits of computation and laying the foundation for the theory of computer science. Nazi Germany is rearming. It has reoccupied the Rhineland and annexed Austria against the treaty of Versailles. It has just annexed the Sudetenland in Czechoslovakia and begins preparations to take over the rest of Czechoslovakia despite an agreement just signed in Munich.

Meanwhile, the British intelligence is building up their capability to break encrypted messages used by the Germans to communicate military and naval information. The Germans are using [Enigma machines](https://en.wikipedia.org/wiki/Enigma_machine) for encryption.  Enigma machines use a series of electro-mechanical rotor cipher machines to protect military communication.  Here is a picture of an Enigma machine:

By the time Turing joined the British Bletchley park, the Polish intelligence reverse engineered the logical structure of the Enigma machine and built a decryption machine called *Bomba* (perhaps because of the ticking noise they made). A bomba simulates six Enigma machines simultaneously and tries different decryption keys until the code is broken. The Polish bomba might have been the very _first fuzzer_.

Turing took it upon himself to crack ciphers of the Naval Enigma machine, which were notoriously hard to crack. The Naval Enigma used, as part of its encryption key, a three letter sequence called *trigram*. These trigrams were selected from a book, called *Kenngruppenbuch*, which contained all trigrams in a random order.

### The Kenngruppenbuch

Let's start with the Kenngruppenbuch (K-Book).

We are going to use the following Python functions.
* `shuffle(elements)` - shuffle *elements* and put items in random order.
* `choice(elements, p=weights)` - choose an item from *elements* at random. An element with twice the *weight* is twice as likely to be chosen.
* `log(a)` - returns the natural logarithm of a.
* `a ** b` - is the a to the power of b (a.k.a. [power operator](https://docs.python.org/3/reference/expressions.html#the-power-operator))

```python
import string
import numpy
```

```python
from numpy.random import choice
from numpy.random import shuffle
from numpy import log
```

We start with creating the set of trigrams:

```python
letters = list(string.ascii_letters[26:])  # upper-case characters
trigrams = [str(a + b + c) for a in letters for b in letters for c in letters]
shuffle(trigrams)
```

```python
trigrams[:10]
```

    ['ENN', 'FQG', 'GQU', 'MJG', 'RHW', 'ZEJ', 'AFX', 'ELU', 'CGD', 'FIC']

These now go into the Kenngruppenbuch.  However, it was observed that some trigrams were more likely chosen than others. For instance, trigrams at the top-left corner of any page, or trigrams on the first or last few pages were more likely than one somewhere in the middle of the book or page.  We reflect this difference in distribution by assigning a _probability_ to each trigram, using Benford's law as introduced in [Probabilistic Fuzzing](ProbabilisticGrammarFuzzer.ipynb).

Recall, that Benford's law assigns the $i$-th digit the probability $\log_{10}\left(1 + \frac{1}{i}\right)$ where the base 10 is chosen because there are 10 digits $i\in [0,9]$. However, Benford's law works for an arbitrary number of "digits". Hence, we assign the $i$-th trigram the probability $\log_b\left(1 + \frac{1}{i}\right)$ where the base $b$ is the number of all possible trigrams $b=26^3$.

```python
k_book = {}  # Kenngruppenbuch

for i in range(1, len(trigrams) + 1):
    trigram = trigrams[i - 1]
    # choose weights according to Benford's law
    k_book[trigram] = log(1 + 1 / i) / log(26**3 + 1)
```

Here's a random trigram from the Kenngruppenbuch:

```python
random_trigram = choice(list(k_book.keys()), p=list(k_book.values()))
random_trigram
```

    'FQG'

And this is its probability:

```python
k_book[random_trigram]
```

    0.04148257970673253

### Fuzzing the Enigma

In the following, we introduce an extremely simplified implementation of the Naval Enigma based on the trigrams from the K-book. Of course, the encryption mechanism of the actual Enigma machine is much more sophisticated and worthy of a much more detailed investigation. We encourage the interested reader to follow up with further reading listed in the Background section.

The personell at Bletchley Park can only check whether an encoded message is encoded with a (guessed) trigram.
Our implementation `naval_enigma()` takes a `message` and a `key` (i.e., the guessed trigram).  If the given key matches the (previously computed) key for the message, `naval_enigma()` returns `True`.

```python
from Fuzzer import RandomFuzzer
from Fuzzer import Runner
```

```python
class EnigmaMachine(Runner):
    def __init__(self, k_book):
        self.k_book = k_book
        self.reset()

    def reset(self):
        """Resets the key register"""
        self.msg2key = {}

    def internal_msg2key(self, message):
        """Internal helper method.
           Returns the trigram for an encoded message."""
        if not message in self.msg2key:
            # Simulating how an officer chooses a key from the Kenngruppenbuch to encode the message.
            self.msg2key[message] = choice(list(self.k_book.keys()), p=list(self.k_book.values()))
        trigram = self.msg2key[message]
        return trigram

    def naval_enigma(self, message, key):
        """Returns true if 'message' is encoded with 'key'"""
        if key == self.internal_msg2key(message):
            return True
        else:
            return False
```

To "fuzz" the `naval_enigma()`, our job will be to come up with a key that matches a given (encrypted) message.  Since the keys only have three characters, we have a good chance to achieve this in much less than a seconds.  (Of course, longer keys will be much harder to find via random fuzzing.)

```python
class EnigmaMachine(EnigmaMachine):
    def run(self, tri):
        """PASS if cur_msg is encoded with trigram tri"""
        if self.naval_enigma(self.cur_msg, tri):
            outcome = self.PASS
        else:
            outcome = self.FAIL

        return (tri, outcome)
```

Now we can use the `EnigmaMachine` to check whether a certain message is encoded with a certain trigram.

```python
enigma = EnigmaMachine(k_book)
enigma.cur_msg = "BrEaK mE. L0Lzz"
enigma.run("AAA")
```

    ('AAA', 'FAIL')

The simplest way to crack an encoded message is by brute forcing. Suppose, at Bletchley park they would try random trigrams until a message is broken.

```python
class BletchleyPark(object):
    def __init__(self, enigma):
        self.enigma = enigma
        self.enigma.reset()
        self.enigma_fuzzer = RandomFuzzer(
            min_length=3,
            max_length=3,
            char_start=65,
            char_range=26)

    def break_message(self, message):
        """Returning the trigram for an encoded message"""
        self.enigma.cur_msg = message
        while True:
            (trigram, outcome) = self.enigma_fuzzer.run(self.enigma)
            if outcome == self.enigma.PASS:
                break
        return trigram
```

How long does it take Bletchley park to find the key using this brute forcing approach?

```python
from Timer import Timer
```

```python
enigma = EnigmaMachine(k_book)
bletchley = BletchleyPark(enigma)

with Timer() as t:
    trigram = bletchley.break_message("BrEaK mE. L0Lzz")
```

Here's the key for the current message:

```python
trigram
```

    'BRB'

And no, this did not take long:

```python
'%f seconds' % t.elapsed_time()
```

    '0.669883 seconds'

```python
'Bletchley cracks about %d messages per second' % (1/t.elapsed_time())
```

    'Bletchley cracks about 1 messages per second'

### Turing's Observations
Okay, lets crack a few messages and count the number of times each trigram is observed.

```python
from collections import defaultdict
```

```python
n = 100  # messages to crack
```

```python
observed = defaultdict(int)
for msg in range(0, n):
    trigram = bletchley.break_message(msg)
    observed[trigram] += 1

# list of trigrams that have been observed
counts = [k for k, v in observed.items() if int(v) > 0]

t_trigrams = len(k_book)
o_trigrams = len(counts)
```

```python
"After cracking %d messages, we observed %d out of %d trigrams." % (
    n, o_trigrams, t_trigrams)
```

    'After cracking 100 messages, we observed 83 out of 17576 trigrams.'

```python
singletons = len([k for k, v in observed.items() if int(v) == 1])
```

```python
"From the %d observed trigrams, %d were observed only once." % (
    o_trigrams, singletons)
```

    'From the 83 observed trigrams, 72 were observed only once.'

Given a sample of previously used entries, Turing wanted to _estimate the likelihood_ that the current unknown entry was one that had been previously used, and further, to estimate the probability distribution over the previously used entries. This lead to the development of the estimators of the missing mass and estimates of the true probability mass of the set of items occuring in the sample. Good worked with Turing during the war and, with Turing’s permission, published the analysis of the bias of these estimators in 1953.

Suppose, after finding the keys for n=100 messages, we have observed the trigram "ABC" exactly $X_\text{ABC}=10$ times. What is the probability $p_\text{ABC}$ that "ABC" is the key for the next message? Empirically, we would estimate $\hat p_\text{ABC}=\frac{X_\text{ABC}}{n}=0.1$. We can derive the empirical estimates for all other trigrams that we have observed.  However, it becomes quickly evident that the complete probability mass is distributed over the *observed* trigrams. This leaves no mass for *unobserved* trigrams, i.e., the probability of discovering a new trigram. This is called the missing probability mass or the discovery probability.

Turing and Good derived an estimate of the *discovery probability* $p_0$, i.e., the probability to discover an unobserved trigram, as the number $f_1$ of trigrams observed exactly once divided by the total number $n$ of messages cracked:
$$
p_0 = \frac{f_1}{n}
$$
where $f_1$ is the number of singletons and $n$ is the number of cracked messages.

Lets explore this idea for a bit. We'll extend `BletchleyPark` to crack `n` messages and record the number of trigrams observed as the number of cracked messages increases.

```python
class BletchleyPark(BletchleyPark):

    def break_message(self, message):
        """Returning the trigram for an encoded message"""
        # For the following experiment, we want to make it practical
        #   to break a large number of messages. So, we remove the
        #   loop and just return the trigram for a message.
        #
        # enigma.cur_msg = message
        # while True:
        #     (trigram, outcome) = self.enigma_fuzzer.run(self.enigma)
        #     if outcome == self.enigma.PASS:
        #         break
        trigram = enigma.internal_msg2key(message)
        return trigram

    def break_n_messages(self, n):
        """Returns how often each trigram has been observed,
           and #trigrams discovered for each message."""
        observed = defaultdict(int)
        timeseries = [0] * n

        # Crack n messages and record #trigrams observed as #messages increases
        cur_observed = 0
        for cur_msg in range(0, n):
            trigram = self.break_message(cur_msg)

            observed[trigram] += 1
            if (observed[trigram] == 1):
                cur_observed += 1
            timeseries[cur_msg] = cur_observed

        return (observed, timeseries)
```

Let's crack 2000 messages and compute the GT-estimate.

```python
n = 2000        # messages to crack
```

```python
bletchley = BletchleyPark(enigma)
(observed, timeseries) = bletchley.break_n_messages(n)
```

Let us determine the Good-Turing estimate of the probability that the next trigram has not been observed before:

```python
singletons = len([k for k, v in observed.items() if int(v) == 1])
gt = singletons / n
gt
```

    0.4025

We can verify the Good-Turing estimate empirically and compute the empirically determined probability that the next trigram has not been observed before. To do this, we repeat the following experiment repeats=1000 times, reporting the average: If the next message is a new trigram, return 1, otherwise return 0. Note that here, we do not record the newly discovered trigrams as observed.

```python
repeats = 1000  # experiment repetitions
```

```python
newly_discovered = 0
for cur_msg in range(n, n + repeats):
    trigram = bletchley.break_message(cur_msg)
    if(observed[trigram] == 0):
        newly_discovered += 1

newly_discovered / repeats
```

    0.427

Looks pretty accurate, huh? The difference between estimates is reasonably small, probably below 0.03. However, the Good-Turing estimate did not nearly require as much computational resources as the empirical estimate. Unlike the empirical estimate, the Good-Turing estimate can be computed during the campaign. Unlike the empirical estimate, the Good-Turing estimate requires no additional, redundant repetitions.

In fact, the Good-Turing (GT) estimator often performs close to the best estimator for arbitrary distributions ([Try it here!](#Kenngruppenbuch)). Of course, the concept of *discovery* is not limited to trigrams. The GT estimator is also used in the study of natural languages to estimate the likelihood that we haven't ever heard or read the word we next encounter. The GT estimator is used in ecology to estimate the likelihood of discovering a new, unseen species in our quest to catalog all _species_ on earth. Later, we will see how it can be used to estimate the probability to discover a vulnerability when none has been observed, yet (i.e., residual risk).

Alan Turing was interested in the _complement_ $(1-GT)$ which gives the proportion of _all_ messages for which the Brits have already observed the trigram needed for decryption. For this reason, the complement is also called sample coverage. The *sample coverage* quantifies how much we know about decryption of all messages given the few messages we have already decrypted.

The probability that the next message can be decrypted with a previously discovered trigram is:

```python
1 - gt
```

    0.5974999999999999

The *inverse* of the GT-estimate (1/GT) is a _maximum likelihood estimate_ of the expected number of messages that we can decrypt with previously observed trigrams before having to find a new trigram to decrypt the message.  In our setting, the number of messages for which we can expect to reuse previous trigrams before having to discover a new trigram is:

```python
1 / gt
```

    2.484472049689441

But why is GT so accurate? Intuitively, despite a large sampling effort (i.e., cracking $n$ messages), there are still $f_1$ trigrams that have been observed only once. We could say that such "singletons" are very rare trigrams. Hence, the probability that the next messages is encoded with such a rare but observed trigram gives a good upper bound on the probability that the next message is encoded with an evidently much rarer, unobserved trigram. Since Turing's observation 80 years ago, an entire statistical theory has been developed around the hypothesis that rare, observed "species" are good predictors of unobserved species.

Let's have a look at the distribution of rare trigrams.

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
```

```python
frequencies = [v for k, v in observed.items() if int(v) > 0]
frequencies.sort(reverse=True)
# Uncomment to see how often each discovered trigram has been observed
# print(frequencies)

# frequency of rare trigrams
plt.figure(num=None, figsize=(12, 4), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 2, 1)
plt.hist(frequencies, range=[1, 21], bins=numpy.arange(1, 21) - 0.5)
plt.xticks(range(1, 21))
plt.xlabel('# of occurances (e.g., 1 represents singleton trigrams)')
plt.ylabel('Frequency of occurances')
plt.title('Figure 1. Frequency of Rare Trigrams')

# trigram discovery over time
plt.subplot(1, 2, 2)
plt.plot(timeseries)
plt.xlabel('# of messages cracked')
plt.ylabel('# of trigrams discovered')
plt.title('Figure 2. Trigram Discovery Over Time');
```

```python
# Statistics for most and least often observed trigrams
singletons = len([v for k, v in observed.items() if int(v) == 1])
total = len(frequencies)

print("%3d of %3d trigrams (%.3f%%) have been observed   1 time (i.e., are singleton trigrams)."
      % (singletons, total, singletons * 100 / total))

print("%3d of %3d trigrams ( %.3f%%) have been observed %d times."
      % (1, total, 1 / total, frequencies[0]))
```

    805 of 995 trigrams (80.905%) have been observed   1 time (i.e., are singleton trigrams).
      1 of 995 trigrams ( 0.001%) have been observed 180 times.

The *majority of trigrams* have been observed only once, as we can see in Figure 1 (left). In other words, a the majority of observed trigrams are "rare" singletons. In Figure 2 (right), we can see that discovery is in full swing. The trajectory seems almost linear. However, since there is a finite number of trigrams (26^3 = 17,576) trigram discovery will slow down and eventually approach an asymptote (the total number of trigrams).

### Boosting the Performance of BletchleyPark
Some trigrams have been observed very often. We call these "abundant" trigrams.

```python
print("Trigram : Frequency")
for trigram in sorted(observed, key=observed.get, reverse=True):
    if observed[trigram] > 10:
        print("    %s : %d" % (trigram, observed[trigram]))
```

    Trigram : Frequency
        ENN : 180
        FQG : 72
        GQU : 55
        RHW : 49
        MJG : 46
        ZEJ : 33
        AFX : 30
        FIC : 29
        ELU : 23
        ODQ : 21
        CGD : 18
        KSJ : 17
        IJZ : 15
        UUJ : 15
        VSQ : 12
        ZBD : 12
        NBE : 12
        SNP : 11

We'll speed up the code breaking by _trying the abundant trigrams first_.

First, we'll find out how many messages can be cracked by the existing brute forcing strategy at Bledgley park, given a maximum number of attempts. We'll also track the number of messages cracked over time (`timeseries`).

```python
class BletchleyPark(BletchleyPark):

    def __init__(self, enigma):
        super().__init__(enigma)
        self.cur_attempts = 0
        self.cur_observed = 0
        self.observed = defaultdict(int)
        self.timeseries = [None] * max_attempts * 2

    def break_message(self, message):
        """Returns the trigram for an encoded message, and
           track #trigrams observed as #attempts increases."""
        self.enigma.cur_msg = message
        while True:
            self.cur_attempts += 1                                 # NEW
            (trigram, outcome) = self.enigma_fuzzer.run(self.enigma)
            self.timeseries[self.cur_attempts] = self.cur_observed # NEW
            if outcome == self.enigma.PASS:
                break
        return trigram

    def break_max_attempts(self, max_attempts):
        """Returns #messages successfully cracked after a given #attempts."""
        cur_msg  = 0
        n_messages = 0

        while True:
            trigram = self.break_message(cur_msg)

            # stop when reaching max_attempts
            if self.cur_attempts >= max_attempts:
                break

            # update observed trigrams
            n_messages += 1
            self.observed[trigram] += 1
            if (self.observed[trigram] == 1):
                self.cur_observed += 1
                self.timeseries[self.cur_attempts] = self.cur_observed
            cur_msg += 1
        return n_messages
```

`original` is the number of messages cracked by the bruteforcing strategy, given 100k attempts.  Can we beat this?

```python
max_attempts = 100000
```

```python
bletchley = BletchleyPark(enigma)
original = bletchley.break_max_attempts(max_attempts)
original
```

    7

Now, we'll create a boosting strategy by trying trigrams first that we have previously observed most often.

```python
class BoostedBletchleyPark(BletchleyPark):

    def break_message(self, message):
        """Returns the trigram for an encoded message, and
           track #trigrams observed as #attempts increases."""
        self.enigma.cur_msg = message

        # boost cracking by trying observed trigrams first
        for trigram in sorted(self.prior, key=self.prior.get, reverse=True):
            self.cur_attempts += 1
            (_, outcome) = self.enigma.run(trigram)
            self.timeseries[self.cur_attempts] = self.cur_observed
            if outcome == self.enigma.PASS:
                return trigram

        # else fall back to normal cracking
        return super().break_message(message)
```

`boosted` is the number of messages cracked by the boosted strategy.

```python
boostedBletchley = BoostedBletchleyPark(enigma)
boostedBletchley.prior = observed
boosted = boostedBletchley.break_max_attempts(max_attempts)
boosted
```

    16

We see that the boosted technique cracks substantially more messages. It is worthwhile to record how often each trigram is being used as key and try them in the order of their occurence.

***Try it***. *For practical reasons, we use a large number of previous observations as prior (`boostedBletchley.prior = observed`). You can try to  change the code such that the strategy uses the trigram frequencies (`self.observed`) observed **during** the campaign itself to boost the campaign. You will need to increase `max_attempts` and wait for a long while.*

Let's compare the number of trigrams discovered over time.

```python
# print plots
line_old, = plt.plot(bletchley.timeseries, label="Bruteforce Strategy")
line_new, = plt.plot(boostedBletchley.timeseries, label="Boosted Strategy")
plt.legend(handles=[line_old, line_new])
plt.xlabel('# of cracking attempts')
plt.ylabel('# of trigrams discovered')
plt.title('Trigram Discovery Over Time');
```

We see that the boosted fuzzer is constantly superior over the random fuzzer.

## Estimating the Probability of Path Discovery

<!-- ## Residual Risk: Probability of Failure after an Unsuccessful Fuzzing Campaign -->
<!-- Residual risk is not formally defined in this section, so I made the title a bit more generic -- AZ -->

So, what does Turing's observation for the Naval Enigma have to do with fuzzing _arbitrary_ programs? Turing's assistant I.J. Good extended and published Turing's work on the estimation procedures in Biometrica, a journal for theoretical biostatistics that still exists today. Good did not talk about trigrams. Instead, he calls them "species". Hence, the GT estimator is presented to estimate how likely it is to discover a new species, given an existing sample of individuals (each of which belongs to exactly one species).

Now, we can associate program inputs to species, as well. For instance, we could define the path that is exercised by an input as that input's species. This would allow us to _estimate the probability that fuzzing discovers a new path._ Later, we will see how this discovery probability estimate also estimates the likelihood of discovering a vulnerability when we have not seen one, yet (residual risk).

Let's do this. We identify the species for an input by computing a hash-id over the set of statements exercised by that input. In the [Coverage](Coverage.ipynb) chapter, we have learned about the [Coverage class](Coverage.ipynb#A-Coverage-Class) which collects coverage information for an executed Python function. As an example, the function [`cgi_decode()`](Coverage.ipynb#A-CGI-Decoder) was introduced. The function `cgi_decode()` takes a string encoded for a website URL and decodes it back to its original form.

Here's what `cgi_decode()` does and how coverage is computed.

```python
from Coverage import Coverage, cgi_decode
```

```python
encoded = "Hello%2c+world%21"
with Coverage() as cov:
    decoded = cgi_decode(encoded)
```

```python
decoded
```

    'Hello, world!'

```python
print(cov.coverage());
```

    {('cgi_decode', 25), ('__exit__', 80), ('cgi_decode', 24), ('cgi_decode', 13), ('cgi_decode', 23), ('cgi_decode', 12), ('cgi_decode', 33), ('cgi_decode', 22), ('cgi_decode', 11), ('cgi_decode', 32), ('cgi_decode', 21), ('cgi_decode', 10), ('cgi_decode', 31), ('cgi_decode', 20), ('cgi_decode', 19), ('cgi_decode', 18), ('cgi_decode', 17), ('cgi_decode', 27), ('cgi_decode', 16), ('cgi_decode', 26)}

### Trace Coverage
First, we will introduce the concept of execution traces, which are a coarse abstraction of the execution path taken by an input. Compared to the definition of path, a trace ignores the sequence in which statements are exercised or how often each statement is exercised.

* `pickle.dumps()` - serializes an object by producing a byte array from all the information in the object
* `hashlib.md5()` - produces a 128-bit hash value from a byte array

```python
import pickle
import hashlib
```

```python
def getTraceHash(cov):
    pickledCov = pickle.dumps(cov.coverage())
    hashedCov = hashlib.md5(pickledCov).hexdigest()
    return hashedCov
```

Remember our model for the Naval Enigma machine? Each message must be decrypted using exactly one trigram while multiple messages may be decrypted by the same trigram. Similarly, we need each input to yield exactly one trace hash while multiple inputs can yield the same trace hash.

Let's see whether this is true for our `getTraceHash()` function.

```python
inp1 = "a+b"
inp2 = "a+b+c"
inp3 = "abc"

with Coverage() as cov1:
    cgi_decode(inp1)
with Coverage() as cov2:
    cgi_decode(inp2)
with Coverage() as cov3:
    cgi_decode(inp3)
```

The inputs `inp1` and `inp2` execute the same statements:

```python
inp1, inp2
```

    ('a+b', 'a+b+c')

```python
cov1.coverage() - cov2.coverage()
```

    set()

The difference between both coverage sets is empty. Hence, the trace hashes should be the same:

```python
getTraceHash(cov1)
```

    '7ea57fa32002a357d62c81a40c363f0e'

```python
getTraceHash(cov2)
```

    '7ea57fa32002a357d62c81a40c363f0e'

```python
assert getTraceHash(cov1) == getTraceHash(cov2)
```

In contrast, the inputs `inp1` and `inp3` execute _different_ statements:

```python
inp1, inp3
```

    ('a+b', 'abc')

```python
cov1.coverage() - cov3.coverage()
```

    {('cgi_decode', 21)}

Hence, the trace hashes should be different, too:

```python
getTraceHash(cov1)
```

    '7ea57fa32002a357d62c81a40c363f0e'

```python
getTraceHash(cov3)
```

    '54a1a3b094f620070507fb0e3ad23a70'

```python
assert getTraceHash(cov1) != getTraceHash(cov3)
```

### Measuring Trace Coverage over Time
In order to measure trace coverage for a `function` executing a `population` of fuzz inputs, we slightly adapt the `population_coverage()` function from the [Chapter on Coverage](Coverage.ipynb#Coverage-of-Basic-Fuzzing).

```python
def population_trace_coverage(population, function):
    cumulative_coverage = []
    all_coverage = set()
    cumulative_singletons = []
    cumulative_doubletons = []
    singletons = set()
    doubletons = set()

    for s in population:
        with Coverage() as cov:
            try:
                function(s)
            except BaseException:
                pass
        cur_coverage = set([getTraceHash(cov)])

        # singletons and doubletons -- we will need them later
        doubletons -= cur_coverage
        doubletons |= singletons & cur_coverage
        singletons -= cur_coverage
        singletons |= cur_coverage - (cur_coverage & all_coverage)
        cumulative_singletons.append(len(singletons))
        cumulative_doubletons.append(len(doubletons))

        # all and cumulative coverage
        all_coverage |= cur_coverage
        cumulative_coverage.append(len(all_coverage))

    return all_coverage, cumulative_coverage, cumulative_singletons, cumulative_doubletons
```

Let's see whether our new function really contains coverage information only for *two* traces given our three inputs for `cgi_decode`.

```python
all_coverage = population_trace_coverage([inp1, inp2, inp3], cgi_decode)[0]
assert len(all_coverage) == 2
```

Unfortunately, the `cgi_decode()` function is too simple. Instead, we will use the original Python [HTMLParser](https://docs.python.org/3/library/html.parser.html) as our test subject.

```python
from Fuzzer import RandomFuzzer
from Coverage import population_coverage
from html.parser import HTMLParser
```

```python
trials = 50000  # number of random inputs generated
```

Let's run a random fuzzer for $n=50000$ times and plot trace coverage over time.

```python
# create wrapper function
def my_parser(inp):
    parser = HTMLParser()  # resets the HTMLParser object for every fuzz input
    parser.feed(inp)
```

```python
# create random fuzzer
fuzzer = RandomFuzzer(min_length=1, max_length=100,
                      char_start=32, char_range=94)

# create population of fuzz inputs
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())

# execute and measure trace coverage
trace_timeseries = population_trace_coverage(population, my_parser)[1]

# execute and measure code coverage
code_timeseries = population_coverage(population, my_parser)[1]

# plot trace coverage over time
plt.figure(num=None, figsize=(12, 4), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 2, 1)
plt.plot(trace_timeseries)
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised')
plt.title('Trace Coverage Over Time')

# plot code coverage over time
plt.subplot(1, 2, 2)
plt.plot(code_timeseries)
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements covered')
plt.title('Code Coverage Over Time');
```

Above, we can see trace coverage (left) and code coverage (right) over time. Here are our observations.
1. **Trace coverage is more robust**. There are less sudden jumps in the graph compared to code coverage.
2. **Trace coverage is more fine grained.** There more traces than statements covered at the end (y-axis)
3. **Trace coverage grows more steadily**. Code coverage exercise more than half the statements with the first input that it exercises after 50k inputs. Instead, the number of traces covered grows slowly and steadily since each input can yield only one execution trace.

It is for this reason that one of the most prominent and successful fuzzers today, american fuzzy lop (AFL), uses a similar *measure of progress* (a hash computed over the branches exercised by the input).

### Evaluating the Discovery Probability Estimate

Let's find out how the Good-Turing estimator performs as estimate of discovery probability when we are fuzzing to discover execution traces rather than trigrams.

To measure the empirical probability, we execute the same population of inputs (n=50000) and measure in regular intervals (measurement=100 intervals). During each measurement, we repeat the following experiment repeats=500 times, reporting the average: If the next input yields a new trace, return 1, otherwise return 0. Note that during these repetitions, we do not record the newly discovered traces as observed.

```python
repeats = 500      # experiment repetitions
measurements = 100  # experiment measurements
```

```python
emp_timeseries = []
all_coverage = set()
step = int(trials / measurements)

for i in range(0, trials, step):
    if i - step >= 0:
        for j in range(step):
            inp = population[i - j]
            with Coverage() as cov:
                try:
                    my_parser(inp)
                except BaseException:
                    pass
            all_coverage |= set([getTraceHash(cov)])

    discoveries = 0
    for _ in range(repeats):
        inp = fuzzer.fuzz()
        with Coverage() as cov:
            try:
                my_parser(inp)
            except BaseException:
                pass
        if getTraceHash(cov) not in all_coverage:
            discoveries += 1
    emp_timeseries.append(discoveries / repeats)
```

Now, we compute the Good-Turing estimate over time.

```python
gt_timeseries = []
singleton_timeseries = population_trace_coverage(population, my_parser)[2]
for i in range(1, trials + 1, step):
    gt_timeseries.append(singleton_timeseries[i - 1] / i)
```

Let's go ahead and plot both time series.

```python
line_emp, = plt.semilogy(emp_timeseries, label="Empirical")
line_gt, = plt.semilogy(gt_timeseries, label="Good-Turing")
plt.legend(handles=[line_emp, line_gt])
plt.xticks(range(0, measurements + 1, int(measurements / 5)),
           range(0, trials + 1, int(trials / 5)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('discovery probability')
plt.title('Discovery Probability Over Time');
```

Again, the Good-Turing estimate appears to be *highly accurate*. In fact, the empirical estimator has a much lower precision as indicated by the large swings. You can try and increase the number of repetitions (repeats) to get more precision for the empirical estimates, however, at the cost of waiting much longer.

### Discovery Probability Quantifies Residual Risk

Alright. You have gotten a hold of a couple of powerful machines and used them to fuzz a software system for several months without finding any vulnerabilities. Is the system vulnerable?

Well, who knows? We cannot say for sure; there is always some residual risk. Testing is not verification. Maybe the next test input that is generated reveals a vulnerability.

Let's say *residual risk* is the probability that the next test input reveals a vulnerability that has not been found, yet. Böhme \cite{stads} has shown that the Good-Turing estimate of the discovery probability is also an estimate of the maxmimum residual risk.

**Proof sketch (Residual Risk)**. Here is a proof sketch that shows that an estimator of discovery probability for an arbitrary definition of species gives an upper bound on the probability to discover a vulnerability when none has been found: Suppose, for each "old" species A (here, execution trace), we derive two "new" species: Some inputs belonging to A expose a vulnerability while others belonging to A do not. We know that _only_ species that do not expose a vulnerability have been discovered. Hence, _all_ species exposing a vulnerability and _some_ species that do not expose a vulnerability remain undiscovered. Hence, the probability to discover a new species gives an upper bound on the probability to discover (a species that exposes) a vulnerability. **QED**.

An estimate of the discovery probability is useful in many other ways.

1. **Discovery probability**. We can estimate, at any point during the fuzzing campaign, the probability that the next input belongs to a previously unseen species (here, that it yields a new execution trace, i.e., exercises a new set of statements).
2. **Complement of discovery probability**. We can estimate the proportion of *all* inputs the fuzzer can generate for which we have already seen the species (here, execution traces). In some sense, this allows us to quantify the *progress of the fuzzing campaign towards completion*: If the probability to discovery a new species is too low, we might as well abort the campaign.
3. **Inverse of discovery probability**. We can predict the number of test inputs needed, so that we can expect the discovery of a new species (here, execution trace).

## How Do We Know When to Stop Fuzzing?

In fuzzing, we have measures of progress such as [code coverage](Coverage.ipynb) or [grammar coverage](GrammarCoverageFuzzer.ipynb). Suppose, we are interested in covering all statements in the program. The _percentage_ of statements that have already been covered quantifies how "far" we are from completing the fuzzing campaign. However, sometimes we know only the _number_ of species $S(n)$ (here, statements) that have been discovered after generating $n$ fuzz inputs. The percentage $S(n)/S$ can only be computed if we know the _total number_ of species $S$. Even then, not all species may be feasible.

### A Success Estimator

If we do not _know_ the total number of species, then let's at least _estimate_ it: As we have seen before, species discovery slows down over time. In the beginning, many new species are discovered. Later, many inputs need to be generated before discovering the next species. In fact, given enough time, the fuzzing campaign approaches an _asymptote_. It is this asymptote that we can estimate.

In 1984, Anne Chao, a well-known theoretical bio-statistician, has developed an estimator $\hat S$ which estimates the asymptotic total number of species $S$:
\begin{align}
\hat S_\text{Chao1} = \begin{cases}
S(n) + \frac{f_1^2}{2f_2} & \text{if $f_2>0$}\\
S(n) + \frac{f_1(f_1-1)}{2} & \text{otherwise}
\end{cases}
\end{align}
* where $f_1$ and $f_2$ is the number of singleton and doubleton species, respectively (that have been observed exactly once or twice, resp.), and
* where $S(n)$ is the number of species that have been discovered after generating $n$ fuzz inputs.

So, how does Chao's estimate perform? To investigate this, we generate trials=400000 fuzz inputs using a fuzzer setting that allows us to see an asymptote in a few seconds. We measure trace coverage coverage. After half-way into our fuzzing campaign (trials/2=100000), we generate Chao's estimate $\hat S$ of the asymptotic total number of species. Then, we run the remainer of the campaign to see the "empirical" asymptote.

```python
trials = 400000
fuzzer = RandomFuzzer(min_length=2, max_length=4,
                      char_start=32, char_range=32)
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())

_, trace_ts, f1_ts, f2_ts = population_trace_coverage(population, my_parser)
```

```python
time = int(trials / 2)
time
```

    200000

```python
f1 = f1_ts[time]
f2 = f2_ts[time]
Sn = trace_ts[time]
if f2 > 0:
    hat_S = Sn + f1 * f1 / (2 * f2)
else:
    hat_S = Sn + f1 * (f1 - 1) / 2
```

After executing `time` fuzz inputs (half of all), we have covered these many traces:

```python
time
```

    200000

```python
Sn
```

    66

We can estimate there are this many traces in total:

```python
hat_S
```

    84.0

Hence, we have achieved this percentage of the estimate:

```python
100 * Sn / hat_S
```

    78.57142857142857

After executing `trials` fuzz inputs, we have covered these many traces:

```python
trials
```

    400000

```python
trace_ts[trials - 1]
```

    70

The accuracy of Chao's estimator is quite reasonable. It isn't always accurate -- particularly at the beginning of a fuzzing campaign when the [discovery probability](WhenIsEnough.ipynb#Measuring-Trace-Coverage-over-Time) is still very high. Nevertheless, it demonstrates the main benefit of reporting a percentage to assess the progress of a fuzzing campaign towards completion.

***Try it***. *Try setting and `trials` to 1 million and `time` to `int(trials / 4)`.*

### Extrapolating Fuzzing Success
<!--  ## Cost-Benefit Analysis: Extrapolating the Number of Species Discovered -->

Suppose you have run the fuzzer for a week, which generated $n$ fuzz inputs and discovered $S(n)$ species (here, covered $S(n)$ execution traces). Instead, of running the fuzzer for another week, you would like to *predict* how many more species you would discover. In 2003, Anne Chao and her team developed an extrapolation methodology to do just that. We are interested in the number $S(n+m^*)$ of species discovered if $m^*$ more fuzz inputs were generated:

\begin{align}
\hat S(n + m^*) = S(n) + \hat f_0 \left[1-\left(1-\frac{f_1}{n\hat f_0 + f_1}\right)^{m^*}\right]
\end{align}
* where $\hat f_0=\hat S - S(n)$ is an estimate of the number $f_0$ of undiscovered species, and
* where $f_1$ the number of singleton species, i.e., those we have observed exactly once.

The number $f_1$ of singletons, we can just keep track of during the fuzzing campaign itself. The estimate of the number $\hat f_0$ of undiscovered species, we can simply derive using Chao's estimate $\hat S$ and the number of observed species $S(n)$.

Let's see how Chao's extrapolator performs by comparing the predicted number of species to the empirical number of species.

```python
prediction_ts = [None] * time
f0 = hat_S - Sn

for m in range(trials - time):
    prediction_ts.append(Sn + f0 * (1 - (1 - f1 / (time * f0 + f1)) ** m))
```

```python
plt.figure(num=None, figsize=(12, 3), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 3, 1)
plt.plot(trace_ts, color='white')
plt.plot(trace_ts[:time])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised')

plt.subplot(1, 3, 2)
line_cur, = plt.plot(trace_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised')

plt.subplot(1, 3, 3)
line_emp, = plt.plot(trace_ts, color='grey', label="Actual progress")
line_cur, = plt.plot(trace_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_emp, line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised');
```

The prediction from Chao's extrapolator looks quite accurate. We make a prediction at $time=trials/4$. Despite an extrapolation by 3 times (i.e., at trials), we can see that the predicted value (black, dashed line) closely matches the empirical value (grey, solid line).

***Try it***. Again, try setting and `trials` to 1 million and `time` to `int(trials / 4)`.

## Lessons Learned

* One can measure the _progress_ of a fuzzing campaign (as species over time, i.e., $S(n)$).
* One can measure the _effectiveness_ of a fuzzing campaign (as asymptotic total number of species $S$).
* One can estimate the _effectiveness_ of a fuzzing campaign using the Chao1-estimator $\hat S$.
* One can extrapolate the _progress_ of a fuzzing campaign, $\hat S(n+m^*)$.
* One can estimate the _residual risk_ (i.e., the probability that a bug exists that has not been found) using the Good-Turing estimator $GT$ of the species discovery probability.

## Next Steps

This chapter is the last in the book!  If you want to continue reading, have a look at the [Appendices](99_Appendices.ipynb).  Otherwise, _make use of what you have learned and go and create great fuzzers and test generators!_

## Background

* A **statistical framework for fuzzing**, inspired from ecology. Marcel Böhme. [STADS: Software Testing as Species Discovery](https://mboehme.github.io/paper/TOSEM18.pdf). ACM TOSEM 27(2):1--52
* Estimating the **discovery probability**: I.J. Good. 1953. [The population frequencies of species and the
estimation of population parameters](https://www.jstor.org/stable/2333344). Biometrika 40:237–264.
* Estimating the **asymptotic total number of species** when each input can belong to exactly one species: Anne Chao. 1984. [Nonparametric estimation of the number of classes in a population](https://www.jstor.org/stable/4615964). Scandinavian Journal of Statistics 11:265–270
* Estimating the **asymptotic total number of species** when each input can belong to one or more species: Anne Chao. 1987. [Estimating the population size for capture-recapture data with unequal catchability](https://www.jstor.org/stable/2531532). Biometrics 43:783–791
* **Extrapolating** the number of discovered species: Tsung-Jen Shen, Anne Chao, and Chih-Feng Lin. 2003. [Predicting the Number of New Species in Further Taxonomic Sampling](http://chao.stat.nthu.edu.tw/wordpress/paper/2003_Ecology_84_P798.pdf). Ecology 84, 3 (2003), 798–804.

## Exercises
I.J. Good and Alan Turing developed an estimator for the case where each input belongs to exactly one species. For instance, each input yields exactly one execution trace (see function [`getTraceHash`](#Trace-Coverage)). However, this is not true in general. For instance, each input exercises multiple statements and branches in the source code. Generally, each input can belong to one *or more* species.

In this extended model, the underlying statistics are quite different. Yet, all estimators that we have discussed in this chapter turn out to be almost identical to those for the simple, single-species model. For instance, the Good-Turing estimator $C$ is defined as
$$C=\frac{Q_1}{n}$$
where $Q_1$ is the number of singleton species and $n$ is the number of generated test cases.
Throughout the fuzzing campaign, we record for each species the *incidence frequency*, i.e., the number of inputs that belong to that species. Again, we define a species $i$ as *singleton species* if we have seen exactly one input that belongs to species $i$.

### Exercise 1: Estimate and Evaluate the Discovery Probability for Statement Coverage

In this exercise, we create a Good-Turing estimator for the simple fuzzer.

#### Part 1: Population Coverage

Implement a function `population_stmt_coverage()` as in [the section on estimating discovery probability](#Estimating-the-Discovery-Probability) that monitors the number of singletons and doubletons over time, i.e., as the number $i$ of test inputs increases.

```python
from Coverage import population_coverage, Coverage
...
```

**Solution.** Here we go:

```python
def population_stmt_coverage(population, function):
    cumulative_coverage = []
    all_coverage = set()
    cumulative_singletons = []
    cumulative_doubletons = []
    singletons = set()
    doubletons = set()

    for s in population:
        with Coverage() as cov:
            try:
                function(s)
            except BaseException:
                pass
        cur_coverage = cov.coverage()

        # singletons and doubletons
        doubletons -= cur_coverage
        doubletons |= singletons & cur_coverage
        singletons -= cur_coverage
        singletons |= cur_coverage - (cur_coverage & all_coverage)
        cumulative_singletons.append(len(singletons))
        cumulative_doubletons.append(len(doubletons))

        # all and cumulative coverage
        all_coverage |= cur_coverage
        cumulative_coverage.append(len(all_coverage))

    return all_coverage, cumulative_coverage, cumulative_singletons, cumulative_doubletons
```

#### Part 2: Population

Use the random `fuzzer(min_length=1, max_length=1000, char_start=0, char_range=255)` from [the chapter on Fuzzers](Fuzzer.ipynb) to generate a population of $n=10000$ fuzz inputs.

```python
from Fuzzer import RandomFuzzer
from html.parser import HTMLParser
...
```

    Ellipsis

**Solution.**  This is fairly straightforward:

```python
trials = 2000  # increase to 10000 for better convergences. Will take a while..
```

We create a wrapper function...

```python
def my_parser(inp):
    parser = HTMLParser()  # resets the HTMLParser object for every fuzz input
    parser.feed(inp)
```

... and a random fuzzer:

```python
fuzzer = RandomFuzzer(min_length=1, max_length=1000,
                      char_start=0, char_range=255)
```

We fill the population:

```python
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())
```

#### Part 3: Estimating Probabilities

Execute the generated inputs on the Python HTML parser (`from html.parser import HTMLParser`) and estimate the probability that the next input covers a previously uncovered statement (i.e., the discovery probability) using the Good-Turing estimator.

**Solution.** Here we go:

```python
measurements = 100  # experiment measurements
step = int(trials / measurements)

gt_timeseries = []
singleton_timeseries = population_stmt_coverage(population, my_parser)[2]
for i in range(1, trials + 1, step):
    gt_timeseries.append(singleton_timeseries[i - 1] / i)
```

#### Part 4: Empirical Evaluation

Empirically evaluate the accuracy of the Good-Turing estimator (using $10000$ repetitions) of the probability to cover new statements using the experimental procedure at the end of [the section on estimating discovery probability](#Estimating-the-Discovery-Probability).

**Solution.** This is as above:

```python
# increase to 10000 for better precision (less variance). Will take a while..
repeats = 100
```

```python
emp_timeseries = []
all_coverage = set()
for i in range(0, trials, step):
    if i - step >= 0:
        for j in range(step):
            inp = population[i - j]
            with Coverage() as cov:
                try:
                    my_parser(inp)
                except BaseException:
                    pass
            all_coverage |= cov.coverage()

    discoveries = 0
    for _ in range(repeats):
        inp = fuzzer.fuzz()
        with Coverage() as cov:
            try:
                my_parser(inp)
            except BaseException:
                pass
        # If intersection not empty, a new stmt was (dis)covered
        if cov.coverage() - all_coverage:
            discoveries += 1
    emp_timeseries.append(discoveries / repeats)
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
line_emp, = plt.semilogy(emp_timeseries, label="Empirical")
line_gt, = plt.semilogy(gt_timeseries, label="Good-Turing")
plt.legend(handles=[line_emp, line_gt])
plt.xticks(range(0, measurements + 1, int(measurements / 5)),
           range(0, trials + 1, int(trials / 5)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('discovery probability')
plt.title('Discovery Probability Over Time');
```

### Exercise 2: Extrapolate and Evaluate Statement Coverage

In this exercise, we use Chao's extrapolation method to estimate the success of fuzzing.

#### Part 1: Create Population

Use the random `fuzzer(min_length=1, max_length=1000, char_start=0, char_range=255)` to generate a population of $n=400000$ fuzz inputs.

**Solution.** Here we go:

```python
trials = 400  # Use 400000 for actual solution.  This takes a while!
```

```python
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())

_, stmt_ts, Q1_ts, Q2_ts = population_stmt_coverage(population, my_parser)
```

#### Part 2: Compute Estimate

Compute an estimate of the total number of statements $\hat S$ after $n/4=100000$ fuzz inputs were generated. In the extended model, $\hat S$ is computed as
\begin{align}
\hat S_\text{Chao1} = \begin{cases}
S(n) + \frac{Q_1^2}{2Q_2} & \text{if $Q_2>0$}\\
S(n) + \frac{Q_1(Q_1-1)}{2} & \text{otherwise}
\end{cases}
\end{align}
  * where $Q_1$ and $Q_2$ is the number of singleton and doubleton statements, respectively (i.e., statements that have been exercised by exactly one or two fuzz inputs, resp.), and
  * where $S(n)$ is the number of statements that have been (dis)covered after generating $n$ fuzz inputs.

**Solution.** Here we go:

```python
time = int(trials / 4)
Q1 = Q1_ts[time]
Q2 = Q2_ts[time]
Sn = stmt_ts[time]

if Q2 > 0:
    hat_S = Sn + Q1 * Q1 / (2 * Q2)
else:
    hat_S = Sn + Q1 * (Q1 - 1) / 2

print("After executing %d fuzz inputs, we have covered %d **(%.1f %%)** statements.\n" % (time, Sn, 100 * Sn / hat_S) +
      "After executing %d fuzz inputs, we estimate there are %d statements in total.\n" % (time, hat_S) +
      "After executing %d fuzz inputs, we have covered %d statements." % (trials, stmt_ts[trials - 1]))
```

    After executing 100 fuzz inputs, we have covered 126 **(63.6 %)** statements.
    After executing 100 fuzz inputs, we estimate there are 198 statements in total.
    After executing 400 fuzz inputs, we have covered 171 statements.

#### Part 3: Compute and Evaluate Extrapolator

Compute and evaluate Chao's extrapolator by comparing the predicted number of statements to the empirical number of statements.

**Solution.** Here's our solution:

```python
prediction_ts = [None] * time
Q0 = hat_S - Sn

for m in range(trials - time):
    prediction_ts.append(Sn + Q0 * (1 - (1 - Q1 / (time * Q0 + Q1)) ** m))
```

```python
plt.figure(num=None, figsize=(12, 3), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 3, 1)
plt.plot(stmt_ts, color='white')
plt.plot(stmt_ts[:time])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements exercised')

plt.subplot(1, 3, 2)
line_cur, = plt.plot(stmt_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements exercised')

plt.subplot(1, 3, 3)
line_emp, = plt.plot(stmt_ts, color='grey', label="Actual progress")
line_cur, = plt.plot(stmt_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_emp, line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements exercised');
```