# Tensorflow 1.X. Tutorial

Credit: This tutorial is heavily based on a tutorial used in Berkeley's CS 294-112 course

This tutorial will provide a brief overview of the core concepts and functionality of Tensorflow. This tutorial will cover the following:

1. Creating a neural network
2. Performing computations
3. Training a neural network

```python
import tensorflow as tf
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.cm as cm
import matplotlib.patches as mpatches
```

Check the Tensorflow version

```python
tf.__version__

```

    '1.15.3'

```python
def tf_reset():
    try:
        sess.close()
    except:
        pass
    tf.reset_default_graph()
    return tf.Session()
```

## 1. Creating a Neural Network

We create neural network from the scratch here. Below code describes how to create neural network using tensorflow. For given input placeholder, $x$, the output of neural network is computed as follows:

\begin{align}
h1 &= \sigma(x * W0 + b0) \\
h2 &= \sigma(h1 * W1 + b1) \\
out &= h2 * W2 + b2
\end{align}
where $W$ is weight matrix for each hidden layer, $b$ is bias, and $\sigma$ is activation function.

```python
tf.reset_default_graph()

# number of input features
num_features = 5

# number of output actions
num_outputs = 1

# numer of neurons in hidden layer
num_hidden = 32

# create the placeholders
input_ph = tf.placeholder(tf.float32, [None, num_features])

weights_0 = tf.Variable(tf.random_normal([num_features,num_hidden]))
bias_0 = tf.Variable(tf.random_normal([num_hidden]))
weights_1 = tf.Variable(tf.random_normal([num_hidden,num_hidden]))
bias_1 = tf.Variable(tf.random_normal([num_hidden]))
weights_2 = tf.Variable(tf.random_normal([num_hidden,num_outputs]))
bias_2 = tf.Variable(tf.random_normal([num_outputs]))

hidden_layer1 = tf.nn.relu(tf.matmul(input_ph,weights_0)+bias_0)
hidden_layer2 = tf.nn.relu(tf.matmul(hidden_layer1,weights_1)+bias_1)
output_layer = tf.matmul(hidden_layer2,weights_2)+bias_2

```

## 2. Performing computations

Tensorflow starts with creating session for 1.X version.

```python
sess = tf.Session()
```

We can these execute commands by calling sess.run()

```python
a = tf.constant(1.0)
b = tf.constant(2.0)
c = a + b

c_run = sess.run(c)
print(c)
print('c = {}'.format(c_run))
```

    Tensor("add_3:0", shape=(), dtype=float32)
    c = 3.0

If we are trying to run an operation with variables, we must first initalize the variables

```python
init_op = tf.global_variables_initializer()
sess.run(init_op)
```

If we want to run commands with placeholders for inputting data, we specify an addition "feed_dict" term.

```python
numbers = np.random.randn(10,5)

res = sess.run(output_layer, feed_dict={input_ph: numbers})

print(res)
```

    [[ -0.53196734]
     [-37.963474  ]
     [-21.387754  ]
     [-18.46625   ]
     [-36.519405  ]
     [  3.5763748 ]
     [-39.331776  ]
     [-27.67065   ]
     [-58.13487   ]
     [ -9.681015  ]]

## 3. Training a neural network

In this section, we'll train a neural network on a simple regression problem. First of all, let's create data for a 1-dimensional regression problem:

```python
# generate the data
inputs = np.linspace(-2*np.pi, 2*np.pi, 10000)[:, None]
outputs = np.cos(inputs) + 0.05 * np.random.normal(size=[len(inputs),1])

plt.scatter(inputs[:, 0], outputs[:, 0], s=0.1, color='k', marker='o')
```

The below code creates the inputs, variables, neural network operations, mean-squared-error loss, gradient descent optimizer, and runs the optimizer using minibatches of the data.

```python
tf.reset_default_graph()

def create_model():
    # create inputs
    input_ph = tf.placeholder(dtype=tf.float32, shape=[None, 1])
    output_ph = tf.placeholder(dtype=tf.float32, shape=[None, 1])

    num_hidden=32
    num_outputs=1
    num_inputs = input_ph.shape[1].value
    weights_0 = tf.Variable(tf.random_normal([num_inputs,num_hidden]))
    bias_0 = tf.Variable(tf.random_normal([num_hidden]))
    weights_1 = tf.Variable(tf.random_normal([num_hidden,num_hidden]))
    bias_1 = tf.Variable(tf.random_normal([num_hidden]))
    weights_2 = tf.Variable(tf.random_normal([num_hidden,num_outputs]))
    bias_2 = tf.Variable(tf.random_normal([num_outputs]))

    hidden_layer1 = tf.nn.relu(tf.matmul(input_ph,weights_0)+bias_0)
    hidden_layer2 = tf.nn.relu(tf.matmul(hidden_layer1,weights_1)+bias_1)
    output_layer = tf.matmul(hidden_layer2,weights_2)+bias_2

    return input_ph, output_ph, output_layer

input_ph, output_ph, output_pred = create_model()

# create a tensorflow session
sess = tf.Session()

# create loss
mse = tf.reduce_mean(0.5 * tf.square(output_pred - output_ph))

# create optimizer
opt = tf.train.AdamOptimizer().minimize(mse)

# initialize variables
sess.run(tf.global_variables_initializer())

# create saver to save model variables
saver = tf.train.Saver()

# run training
batch_size = 100
for training_step in range(10000):
    # get a random subset of the training data
    indices = np.random.randint(low=0, high=len(inputs), size=batch_size)
    input_batch = inputs[indices]
    output_batch = outputs[indices]

    # run the optimizer and get the mse
    _, mse_run = sess.run([opt, mse], feed_dict={input_ph: input_batch, output_ph: output_batch})

    # print the mse every so often
    if training_step % 1000 == 0:
        print('{0:04d} mse: {1:.3f}'.format(training_step, mse_run))
        saver.save(sess, '/tmp/model.ckpt')
```

    0000 mse: 3723.336
    1000 mse: 0.211
    2000 mse: 0.011
    3000 mse: 0.005
    4000 mse: 0.005
    5000 mse: 0.003
    6000 mse: 0.004
    7000 mse: 0.002
    8000 mse: 0.003
    9000 mse: 0.003

Now that the neural network is trained, we can use it to make predictions:

```python
sess = tf_reset()

# create the model
input_ph, output_ph, output_pred = create_model()

# restore the saved model
saver = tf.train.Saver()
saver.restore(sess, "/tmp/model.ckpt")

output_pred_run = sess.run(output_pred, feed_dict={input_ph: inputs})

plt.scatter(inputs[:, 0], outputs[:, 0], c='k', marker='o', s=0.1)
plt.scatter(inputs[:, 0], output_pred_run[:, 0], c='r', marker='o', s=0.1)
```

Now you know the basics of Tensorflow!

```python

```


```
import numpy as np
import pandas as pd
import dimod
```

### Solving a QUBO with Simulated Annealing Sampler
\begin{equation}
H_{1}^{QUBO}=-4.4x_{1}^2+0.6x_{2}^2-2x_{3}^2+2.8x_{1}x_{2}-0.8x_{2}x_{3}+2.4
\end{equation}

```
linear = {0: -4.4, 1: 0.6, 2: -2}
quadratic = {(0,1): 2.8, (1,2):-0.8}
offset = 2.4

bqm_qubo = dimod.BinaryQuadraticModel(linear,quadratic,offset,dimod.Vartype.BINARY)
print(bqm_qubo)
print('\n',bqm_qubo.to_numpy_matrix().astype(float))
```

    BinaryQuadraticModel({0: -4.4, 1: 0.6, 2: -2.0}, {(0, 1): 2.8, (1, 2): -0.8}, 2.4, 'BINARY')

     [[-4.4  2.8  0. ]
     [ 0.   0.6 -0.8]
     [ 0.   0.  -2. ]]

```
sampleset = dimod.SimulatedAnnealingSampler().sample(bqm_qubo,num_reads=10,num_sweeps=1000)
print(sampleset)
```

       0  1  2 energy num_oc.
    0  1  0  1   -4.0       1
    1  1  0  1   -4.0       1
    2  1  0  1   -4.0       1
    3  1  0  1   -4.0       1
    4  1  0  1   -4.0       1
    5  1  0  1   -4.0       1
    6  1  0  1   -4.0       1
    7  1  0  1   -4.0       1
    8  1  0  1   -4.0       1
    9  1  0  1   -4.0       1
    ['BINARY', 10 rows, 10 samples, 3 variables]

# Sampleset Properties

```
######  First sample with lowest energy  ##########
print(sampleset.first)

print('Energy of the first sample: ',sampleset.first.energy)   # energy of 1st sample
print('Occurences of first sample: ',sampleset.first.num_occurrences)
```

    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Energy of the first sample:  -4.000000000000001
    Occurences of first sample:  1

```
#####  Info of smpleset in numpy rec-array   ########

sampleset.record
```

    rec.array([([1, 0, 1], -4., 1), ([1, 0, 1], -4., 1), ([1, 0, 1], -4., 1),
               ([1, 0, 1], -4., 1), ([1, 0, 1], -4., 1), ([1, 0, 1], -4., 1),
               ([1, 0, 1], -4., 1), ([1, 0, 1], -4., 1), ([1, 0, 1], -4., 1),
               ([1, 0, 1], -4., 1)],
              dtype=[('sample', 'i1', (3,)), ('energy', '<f8'), ('num_occurrences', '<i8')])

```
#####  Index of the variables associated to the probllem   ####

sampleset.variables
```

    Variables([0, 1, 2])

```
#####  Variable type of tthe sampleset  #####
sampleset.vartype
```

    <Vartype.BINARY: frozenset({0, 1})>

# Sampleset methods

```
####   Sampleset to pandas dataframe   #########

print(sampleset.to_pandas_dataframe())
```

       0  1  2  energy  num_occurrences
    0  1  0  1    -4.0                1
    1  1  0  1    -4.0                1
    2  1  0  1    -4.0                1
    3  1  0  1    -4.0                1
    4  1  0  1    -4.0                1
    5  1  0  1    -4.0                1
    6  1  0  1    -4.0                1
    7  1  0  1    -4.0                1
    8  1  0  1    -4.0                1
    9  1  0  1    -4.0                1

```
##### Iterating over samples  #######
for sample in sampleset.data():
    print(sample)
```

    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)
    Sample(sample={0: 1, 1: 0, 2: 1}, energy=-4.000000000000001, num_occurrences=1)

```
##### Iterating over samples  #######
for sample in sampleset.samples():
    print(sample)
```

    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}
    {0: 1, 1: 0, 2: 1}

```
#####   changing variable type to spin  ######

sampleset1 = sampleset.change_vartype(dimod.Vartype.SPIN)

print(sampleset1.to_pandas_dataframe())
```

       0  1  2  energy  num_occurrences
    0  1 -1  1    -4.0                1
    1  1 -1  1    -4.0                1
    2  1 -1  1    -4.0                1
    3  1 -1  1    -4.0                1
    4  1 -1  1    -4.0                1
    5  1 -1  1    -4.0                1
    6  1 -1  1    -4.0                1
    7  1 -1  1    -4.0                1
    8  1 -1  1    -4.0                1
    9  1 -1  1    -4.0                1

```
####   Aggregating all the samples   ###########
print(sampleset.aggregate())
```

       0  1  2 energy num_oc.
    0 +1 -1 +1   -4.0      10
    ['SPIN', 1 rows, 10 samples, 3 variables]

```
####   Sample with lowest energy   ######
print(sampleset.lowest())
```

       0  1  2 energy num_oc.
    0 +1 -1 +1   -4.0       1
    1 +1 -1 +1   -4.0       1
    2 +1 -1 +1   -4.0       1
    3 +1 -1 +1   -4.0       1
    4 +1 -1 +1   -4.0       1
    5 +1 -1 +1   -4.0       1
    6 +1 -1 +1   -4.0       1
    7 +1 -1 +1   -4.0       1
    8 +1 -1 +1   -4.0       1
    9 +1 -1 +1   -4.0       1
    ['SPIN', 10 rows, 10 samples, 3 variables]

#### Slicing a sampleset

```
#####   sampleset.slice()  #####

sampleset2 = sampleset.slice(4)
print(sampleset2)
```

       0  1  2 energy num_oc.
    0 +1 -1 +1   -4.0       1
    1 +1 -1 +1   -4.0       1
    2 +1 -1 +1   -4.0       1
    3 +1 -1 +1   -4.0       1
    ['SPIN', 4 rows, 4 samples, 3 variables]

```
#####    sampleset.truncate()  ######

sampleset3 = sampleset.truncate(3)
print(sampleset3)
```

       0  1  2 energy num_oc.
    0 +1 -1 +1   -4.0       1
    1 +1 -1 +1   -4.0       1
    2 +1 -1 +1   -4.0       1
    ['SPIN', 3 rows, 3 samples, 3 variables]

## Creating custom samples

```
sampleset4 = dimod.SampleSet.from_samples([{'a': -1, 'b': +1},
                                           {'a': +1, 'b': +1},
                                           {'a': +1, 'b':-1}],dimod.BINARY,
                                          energy=[-1.0, 1.0,-2.0])

print(sampleset4)
```

       a  b energy num_oc.
    2  1  0   -2.0       1
    0  0  1   -1.0       1
    1  1  1    1.0       1
    ['BINARY', 3 rows, 3 samples, 2 variables]

```
#####   Adding new variable to sampleset    ####

sampleset5 = sampleset4.append_variables({'c':-1})
print(sampleset5)
```

       a  b  c energy num_oc.
    2  1  0  0   -2.0       1
    0  0  1  0   -1.0       1
    1  1  1  0    1.0       1
    ['BINARY', 3 rows, 3 samples, 3 variables]

```
####   concatenating sampleset  ############
print(sampleset)

print('\n\n',sampleset5)

sampleset6 = dimod.concatenate(sampleset,sampleset5)
print('\n\n',sampleset6)
```

```

```

D'wave tutorials/3.Introduction to dimod Samples.ipynb


```julia
using CausalityTools, DynamicalSystemsBase, Plots, Distributions, Measures
```

# Tutorial: predictive asymmetry test

### Defining a dynamical system and generating time series

Defining a set of unidirectionally coupled logistic maps

```julia
function eom_logistic2(dx, x, p, n)
    c_xy, r₁, r₂, σ = (p...,)
    ξ = rand()
    x, y = x[1], x[2]
    f_xy = (y +  (c_xy*(x + σ*ξ)/2) ) / (1 + (c_xy/2)*(1+σ))

    dx[1] = r₁ * x * (1 - x)
    dx[2] = r₂ * (f_xy) * (1 - f_xy)
    return
end
```

    eom_logistic2 (generic function with 1 method)

Create a function that returns an instance of a `DynamicalSystems.DiscreteDynamicalSystem`.

```julia
function logistic2(u₀, c_xy, r₁, r₂, σ)
    p = [c_xy, r₁, r₂, σ]
    DiscreteDynamicalSystem(eom_logistic2_unidir, u₀, p)
end

logistic2(;u₀ = rand(2), c_xy = 0.1, r₁ = 3.78, r₂ = 3.66, σ = 0.05) =
    logistic2(u₀, c_xy, r₁, r₂, σ)
```

    logistic2 (generic function with 2 methods)

Initialise a system

```julia
system = logistic2(c_xy = 0.5, r₁ = 3.7, r₂ = 3.8)
```

    2-dimensional discrete dynamical system
     state:       [0.230387, 0.220331]
     e.o.m.:      eom_logistic2_unidir
     in-place?    true
     jacobian:    ForwardDiff
     parameters:  [0.5, 3.7, 3.8, 0.05]

Generate orbit (time series).

```julia
npts = 300
orbit = trajectory(system, 300, Ttr = 1000); x, y = columns(orbit);
```

Plot it

```julia
p = plot(xlabel = "Time step", ylabel = "Value",
    tickfont = font(12), guidefont = font(12),
    left_margin = 8mm, bottom_margin = 8mm, legendfont = font(12))
plot!(p, [columns(orbit)...], label = ["x", "y"])
```

Okay, so we have some time series

## PredictiveAsymmetryTest

Let's first investigate the documentation for the test

```julia
?PredictiveAsymmetryTest
```

    search: PredictiveAsymmetryTest predictive_asymmetry

```
PredictiveAsymmetryTest(predictive_test::CausalityTest)
```

The parameters for a predictive asymmetry causality test [1].

## Mandatory keywords

  * **`predictive_test`**: An instance of a predictive causality test that explicitly    uses prediction lags (e.g. [`VisitationFrequencyTest`](@ref) or    [`TransferOperatorGridTest`](@ref)).

## About the prediction lags

The prediction lags in the predictive causality test must consist of `n` negative  integers and `n` positive integers that are symmetric around zero.

In other words, negative lags  must exactly match the positive lags but with opposite  sign. The zero lag can be included, but will be ignored, so it is possible to give  ranges too.

## Examples

```julia
test_visitfreq = VisitationFrequencyTest(ηs = [-5, -4, -2, -1, 0, 1, 2, 4, 5])
test_transferoperator = TransferOperatorGridTest(ηs = -3:3)

# Note that `predictive_test` is a *mandatory* keyword.
PredictiveAsymmetryTest(predictive_test = test_visitfreq)
PredictiveAsymmetryTest(predictive_test = test_transferoperator)
```

## References

1. Diego, David, Kristian Agasøster Haaga, Jo Brendryen, and Bjarte Hannisdal.   A simple test for causal asymmetry in complex systems. In prep.

## Transfer entropy test

Okay, so we first need to define a transfer entropy test to use for the prediction. For this, we have several choices, but for the demonstration, we'll stick to the `TransferOperatorGridTest`, which estimates probabilities from an approximation to the transfer operator (Diego et al., 2019).

```julia
?TransferOperatorGridTest
```

    search: TransferOperatorGridTest TransferOperatorGrid

```
TransferOperatorGridTest(k::Int = 1, l::Int = 1, m::Int = 1, n::Int = 1,
    τ::Int = 1, b = 2, estimator::TransferOperatorGrid = TransferOperatorGrid(),
    binning_summary_statistic::Function = StatsBase.mean,
    binning::RectangularBinning, ηs)
```

The parameters for a transfer entropy test using the `TransferOperatorGrid` estimator [1].

## Mandatory keyword arguments

  * **`binning::RectangularBinning`**: An instance of a [`RectangularBinning`](@ref) that dictates    how the delay embedding is discretized.
  * **`ηs`**: The prediction lags (that gos into the $T_{f}$ component of the embedding).

## Optional keyword arguments

  * **`k::Int`**: The dimension of the $T_{f}$ component of the embedding.
  * **`l::Int`**: The dimension of the $T_{pp}$ component of the embedding.
  * **`m::Int`**: The dimension of the $S_{pp}$ component of the embedding.
  * **`n::Int`**: The dimension of the $C_{pp}$ component of the embedding.
  * **`τ::Int`**: The embedding lag. Default is `τ = 1`.
  * **`b`**: Base of the logarithm. The default (`b = 2`) gives the TE in bits.
  * **`estimator::TransferOperatorGrid`**: A `TransferOperatorGrid` estimator instance.
  * **`binning_summary_statistic::Function`**: A summary statistic to summarise the    transfer entropy values if multiple binnings are provided.

## Estimation of the invariant measure

With a `TransferOperatorGridTest`, the first step is to compute an approximation to the  [transfer operator](@ref transfer_operator_rectangular) over the partition elements of a rectangular [discretization](@ref discretization)  of an appropriate [delay reconstruction](@ref custom_delay_reconstruction) of the time  series to be analysed. Transfer entropy is then computed from the [invariant  distribution](@ref invariant_measure_rectangular) arising from the transfer operator.

## About the delay reconstruction for transfer entropy analysis

Denote the time series for the source process $S$ as $S(t)$, and the time series for  the target process $T$ as $T(t)$, and $C_i(t)$ as the time series for any conditional  processes $C_i$ that also may influence $T$. To compute (conditional) TE, we need a  generalised embedding [3, 4] incorporating all of these processes.

For convenience, define the state vectors

$$
\begin{align}
T_f^{(k)} &= \{(T(t+\eta_k), \ldots, T(t+\eta_2), T(t+\eta_1))\}, \label{eq:Tf} \\
T_{pp}^{(l)} &= \{ (T(t), T(t-\tau_1), T(t-\tau_2), \ldots, T(t - \tau_{l - 1})) \\
S_{pp}^{(m)} &= \{(S(t), S(t-\tau_1), S(t-\tau_2), \ldots, S(t-\tau_{m - 1}))\},\\
C_{pp}^{(n)} &= \{ (C_1(t), C_1(t-\tau_1), \ldots,  C_2(t), C_2(t-\tau_1) \},
\end{align}
$$

where the state vectors $T_f^{(k)}$ contain $k$ future values of the target  variable, $T_{pp}^{(l)}$ contain $l$ present and past values of the target  variable, $S_{pp}^{(m)}$ contain $m$ present and past values of the source  variable, $C_{pp}^{(n)}$ contain a total of $n$ present and past values of any  conditional variable(s).  Combining all variables, we have the generalised embedding

$$
\begin{align}
\mathbb{E} = (T_f^{(k)}, T_{pp}^{(l)}, S_{pp}^{(m)}, C_{pp}^{(n)})
\end{align}
$$

with a total embedding dimension of $k + l + m + n$.  Here, $\tau$ indicates the [embedding](@ref custom_delay_reconstruction) lag  (in the `VisitationFrequencyTest`, we set $\tau_1 = \tau_2 = \tau_3 = \ldots$), and $\eta$ indicates the prediction lag (the lag of the influence the source  has on the target).

Hence, in the generalised embedding, only $T_f$ depends on the prediction lag $\eta$,  which is to be determined by the analyst. For transfer entropy analysis, $\eta$ is chosen  to be some positive integer, while for  [predictive asymmetry analysis](@ref predictive_asymmetry),  symmetric negative and positive $\eta$s are used for computing $\mathbb{A}$.

In terms of this generalised embedding, transfer entropy from a source variable $S$ to a  target variable $T$ with conditioning on variable(s) $C$ is thus defined as

$$
\begin{align}
TE_{S \rightarrow T|C} = \int_{\mathbb{E}} P(T_f, T_{pp}, S_{pp}, C_{pp}) \log_{2}{\left( \frac{P(T_f | T_{pp}, S_{pp}, C_{pp})}{P(T_f | T_{pp}, C_{pp})}\right)}
\end{align}
$$

Without conditioning, we have

$$
\begin{align}
TE_{S \rightarrow T} = \int_{\mathbb{E}} P(T_f, T_{pp}, S_{pp}) \log_{2}{\left(\frac{P(T_f | T_{pp}, S_{pp})}{P(T_f | T_{pp})}\right)}
\end{align}
$$

## Low-level estimator

This test uses the `TransferOperatorGrid` estimator on the following low-level method  under the hood.

  * [`transferentropy(::Any, ::TEVars, ::RectangularBinning, ::TransferEntropyEstimator)`](@ref)

In this estimator, the mapping between variables of the  [generalised embedding](@ref custom_delay_reconstruction) and the marginals during  transfer entropy computation is controlled using a [`TEVars`](@ref)  instance. It is *highly* recommended that you check the documentation for this  method, because it describes the transfer entropy estimation procedure in detail.

## Notes:

  * Use `causality(source, target, params::TransferOperatorGridTest)` for regular    transfer entropy analysis. This method uses only the `k`, `l`, `m` and ignores `n`    when constructing the delay reconstruction.
  * Use `causality(source, target, cond, params::TransferOperatorGridTest)` for conditional    transfer entropy analysis. This method uses the `k`, `l`, `m` *and* `n` when constructing    the delay reconstruction.

## Example

```julia
# Prediction lags
ηs = 1:10
binning = RectangularBinning(10)

# Use defaults, binning and prediction lags are required.
# Note that `binning` and `ηs` are *mandatory* keyword arguments.
TransferOperatorGridTest(binning = binning, ηs = ηs)

# The other keywords can also be adjusted
TransferOperatorGridTest(k = 1, l = 2, binning = binning, ηs = ηs)
```

## References

1. Diego, David, Kristian Agasøster Haaga, and Bjarte Hannisdal. "Transfer entropy computation   using the Perron-Frobenius operator." Physical Review E 99.4 (2019): 042212.  [https://journals.aps.org/pre/abstract/10.1103/PhysRevE.99.042212](https://journals.aps.org/pre/abstract/10.1103/PhysRevE.99.042212)

Now we have the information we need!

#### Define test

```julia
# Maximum prediction
η_max = 5

# Embedding parameters (3-dimensional embedding)
k, l, m = 1, 1, 1

# Partition
n_subdivisions = floor(Int, npts^(1/(k + l + m + 1)))
bin_scheme = RectangularBinning(n_subdivisions)

#Transfer entropy test with base-2 logarithm,
te_test = TransferOperatorGridTest(binning = bin_scheme,
    ηs = -η_max:η_max, b = 2, k = k, l = l, m = m)

# Predictive asymmetry test based on the transfer entropy test
test = PredictiveAsymmetryTest(predictive_test = te_test)
```

    PredictiveAsymmetryTest{TransferOperatorGridTest}(predictive_test = TransferOperatorGridTest(k = 1, l = 1, m = 1, n = 1, τ = 1, b = 2, estimator = TransferOperatorGrid(), binning_summary_statistic = mean, binning = RectangularBinning(4), ηs = -5:5))

#### Apply test to time series

```julia
influence_xtoy = causality(x, y, test)
```

    5-element Array{Float64,1}:
     0.11898049344502759
     0.29581876296632004
     0.4681055617841716
     0.6348198945975616
     0.8164238890248452

```julia
influence_ytox = causality(y, x, test)
```

    5-element Array{Float64,1}:
     -0.06654332504469362
     -0.13137874169150177
     -0.19910704334137552
     -0.31574168707525985
     -0.4170290756567008

Plot the predictive asymmetries against the prediction lags.

```julia
plot(size = (600, 300), bottom_margin = 8mm, left_margin = 8mm,
    tickfont = font(12), guidefont = font(12), legendfont = font(12),
    xlabel = "Prediction lag", ylabel = "Predictive asymmetry (bits)",
    legend = :topleft)
plot!(1:η_max, influence_xtoy, c = :black, label = "x to y")
scatter!(1:η_max, influence_xtoy, c = :black, label = "")
plot!(1:η_max, influence_ytox, c = :red, label = "y to x")
scatter!(1:η_max, influence_ytox, c = :red, label = "")
hline!([0], ls = :dash, lc = :grey, label = "")
```

We find a unidirectional relationship from $X \to Y$, which is the ground truth!

```julia

```


### 수치적 최적화(numerical optimization)
* 현재 위치 $x_k$가 최적점인지 판단하는 알고리즘
    - 기울기 필요조건
        - 미분값이 0 또는 모든변수에 대한 그레디언트 벡터값이 0
        - 이차도함수의 값이 양수 또는 헤시안벡터가 양의정부호

* 어떤 위치 $x_k$를 시도한 뒤, 다음 번에 시도할 위치 $x_{k+1}$을 찾는 알고리즘
    - 최대경사법(Steepest Gradient Descent)
    $$
\begin{align}
x_{k+1} = x_{k} - \mu \nabla f(x_k) = x_{k} - \mu g(x_k)
\tag{5.1.11}
\end{align}
$$

#### 최대경사법
- 특징
    - 특정 점에서 기울기가 음수면 앞으로 진행, 양수면 뒤로 진행하여 점점 낮은 위치로 진행
    - 최적점에 도달하면 $\nabla f(x_k)$값이 0이므로 더이상 이동X

- 문제: 진동문제

- 해결
    - 2차도함수/ 헤시안 행렬 이용
    - 모멘텀 방법

#### 뉴턴방법
$$
\begin{align}
{x}_{n+1} = {x}_n - [{H}f({x}_n)]^{-1} \nabla f({x}_n)
\tag{5.1.12}
\end{align}
$$
- 장점
    - 스텝사이즈 필요 없음(헤시안 함수)
    - 빠르게 수렴
    - 지그재그 현상 최소화

- 단점
    - 해시안 함수, 그레디언트 벡터 모두 알아야함
    - 함수모양에 따라 수렴 안할수도 있다

- 해결
    - 준뉴턴문제
        - 현재 점 주변의 몇몇 점에서 함수의 값 구하고 이를 이용해 2차 도함수 값 계산
        - ex) BFGS, CG

#### scipy를 이용한 최적화

```python
import scipy.optimize as optimize
# 목적함수 정의
def f1(x):
    return (x-2) ** 2+2

x0= 0 #초기값
result= optimize.minimize(f1,x0)
result
```

          fun: 2.0
     hess_inv: array([[0.5]])
          jac: array([0.])
      message: 'Optimization terminated successfully.'
         nfev: 9
          nit: 2
         njev: 3
       status: 0
      success: True
            x: array([1.99999999])

- success: True # 최적값을 찾아내는데 성공
-  x: array([1.99999999]) # x=2일때의 값이 최적점이다
-  fun: 2.0 # 최적값에서 높이(f(X))가 2
- hess_inv: array([[0.5]]) #에시안벡터 역행렬이 양수니 볼록함수
- jac: array([0.]) # 자코비안(그레디언트) 벡터의 값이 0이므로 최저값
- message: 'Optimization terminated successfully.' #메세지(성공)
- nit: 2 #iteration 2번(두번 점프 시도)해서 최적점에 도달
- njev: 3 #자코비안 계산횟수: x0와 두번 시도한 점에서 기울기 구해서
- nfev: 9 #각 포인트마다 수치미분해야해서 3*3번 시도
    - 이 숫자를 줄여서 3으로 하려면 함수를 넣어야한다. 함수를 선언해준 후 도함수를 구하여 'jac='으로 넣어주면 된다

```python
def f1(x):
    return (x-2) ** 2+2

def f1p(x):
    return 2*(x-2)

result=optimize.minimize(f1,x0, jac=f1p)
result
```

          fun: 2.0
     hess_inv: array([[0.5]])
          jac: array([0.])
      message: 'Optimization terminated successfully.'
         nfev: 3
          nit: 2
         njev: 3
       status: 0
      success: True
            x: array([2.])

#### 다변후 함수의 최적화
- x를 스칼라가 아니라 벡터를 받을 수 있또록 array로 정의한다

```python
# 2차원 목적함수 최적화
def f2(x):
    return (1 - x[0])**2 + 100.0 * (x[1] - x[0]**2)**2

x0 = (0, 0)
result = optimize.minimize(f2, x0)
print(result)
```

          fun: 2.8440316101421118e-11
     hess_inv: array([[0.49482454, 0.98957634],
           [0.98957634, 1.98394215]])
          jac: array([ 3.98673709e-06, -2.84416559e-06])
      message: 'Optimization terminated successfully.'
         nfev: 96
          nit: 19
         njev: 24
       status: 0
      success: True
            x: array([0.99999467, 0.99998932])

```python
import sympy
from scipy.misc import derivative
import scipy.optimize as optimize
# jac에 이차도함수 넣어서 최적화 하기
x,y = sympy.symbols('x y')
f2=(1 - x)**2 + 100.0 * (y - x**2)**2
sympy.simplify(sympy.diff(f2,x))
```

$\displaystyle 400.0 x \left(x^{2} - y\right) + 2 x - 2$

```python
sympy.diff(f2,y)
```

$\displaystyle - 200.0 x^{2} + 200.0 y$

```python
x0 = (0, 0)
def f2(x):
    return (1 - x[0])**2 + 100.0 * (x[1] - x[0]**2)**2
#f21에 각각 미분한거 더한함수로 도함수 만든 후 jac에 넣어서 최적화하기
def f21(x):
    return np.array((400*x[0]*(x[0]**2-x[1])+2*x[0]-2,-200*x[0]**2+200*x[1]))
result= optimize.minimize(f2,x0,jac=f21)
result

```

          fun: 7.717288542643934e-13
     hess_inv: array([[0.49480256, 0.98953879],
           [0.98953879, 1.98387918]])
          jac: array([ 3.92841203e-06, -2.83120875e-06])
      message: 'Optimization terminated successfully.'
         nfev: 24
          nit: 19
         njev: 24
       status: 0
      success: True
            x: array([0.99999913, 0.99999825])

### 제한조건이 있는 최적화 문제
#### 1. 등식제한조건이 있는 최적화문제
    - 라그랑주 승수법 사용

```python
def f1array(x):
    return x[0]**2+x[1]**2

def eq_constraint(x):
    return x[0]+x[1]-1

optimize.fmin_slsqp(f1array, np.array([1,1]), eqcons= [eq_constraint])
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 0.5000000000000002
                Iterations: 2
                Function evaluations: 8
                Gradient evaluations: 2

    array([0.5, 0.5])

```python
#x[0]=0.5, x[1]=0.5일때 최저점이다
```

```python
def f2array(x):
    return x[0]+x[1]

def eq_const(x):
    return x[0]**2+x[1]**2-1

optimize.fmin_slsqp(f2array, np.array([3,3]), eqcons=[eq_const])
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 1.4142135623732228
                Iterations: 6
                Function evaluations: 25
                Gradient evaluations: 6

    array([0.70710677, 0.7071068 ])

```python
1/2**0.5
```

    0.7071067811865475

#### 2. 부등식 제한조건이 있는 최적화 문제

```python
def f2(x):
    return np.sqrt((x[0]-4)**2 + (x[1] -2) ** 2)

# 제한조건 상수
k=1
def ieq_constraint(x):
    return np.atleast_1d(k- np.sum(np.abs(x)))

optimize.fmin_slsqp(f2,np.array([0,0]), ieqcons= [ieq_constraint])
```

    Optimization terminated successfully.    (Exit mode 0)
                Current function value: 3.605551280732028
                Iterations: 11
                Function evaluations: 77
                Gradient evaluations: 11

    array([9.99999981e-01, 1.89941792e-08])

```python
from functools import partial
import matplotlib as mpl
import matplotlib.pylab as plt

def f2plt(x1, x2):
    return np.sqrt((x1 - 4) ** 2 + (x2 - 2) ** 2)

def f2(x):
    return np.sqrt((x[0] - 4) ** 2 + (x[1] - 2) ** 2)

def ieq_constraint2(x, k):
    return np.atleast_1d(k - np.sum(np.abs(x)))

def plot_opt(k):
    x1 = np.linspace(-2, 5, 100)
    x2 = np.linspace(-1.5, 3, 100)
    X1, X2 = np.meshgrid(x1, x2)
    Y = f2plt(X1, X2)

    plt.contour(X1, X2, Y, colors="gray",
                levels=np.arange(0.5, 5, 0.5) * np.sqrt(2))

    ax = plt.gca()
    x12 = np.linspace(-k, 0, 10)
    x13 = np.linspace(0, k, 10)
    ax.fill_between(x12, x12 + k, -k - x12, color='g', alpha=0.5)
    ax.fill_between(x13, x13 - k, k - x13, color='g', alpha=0.5)

    # 최적점 위치     x_sol = sp.optimize.fmin_slsqp(f2, np.array([0, 0]),                                     ieqc ons=[partial(ieq_constraint2, k=k)], iprint=0)    plt.plot(x_sol[0], x_sol[1], 'ro', ms=20)

    plt.xlim(-2, 5)
    plt.ylim(-1.5, 3)
    plt.xticks(np.linspace(-2, 5, 8))
    plt.yticks(np.linspace(-1, 3, 5))
    plt.xlabel("$x_1$")
    plt.ylabel("$x_2$")
    plt. title("$|x_1| + |x_2| \leq {}$ 제한조건을 가지는 최적화 문제" .format(k))
    plt.show()
plot_opt(1)

```

```python
plot_opt(2)
```

```python
plot_opt(3)
```

```python
plot_opt(4)
```

```python
plot_opt(6)
```

```python
def ieq_constraint(k):
    return lambda x: np.atleast_1d(k - np.sum(np.abs(x)))

opt=[]
k_range = np.linspace(0.1, 10, 100)
for k in k_range:
    opt += [
         sp.optimize.fmin_slsqp(
             f2, np.array([0, 0]), ieqcons=[ieq_constraint(k)], iprint=0
         )]
plt.plot(k_range, np.array(opt)[:, 0], 'g-',label="x1")
plt.plot(k_range, np.array(opt)[:, 1], 'r-',label="x2")
plt.legend()
plt.show()

```

```python
#경고 무시
import warnings
warnings.simplefilter('ignore')

# 자주 사용하는 패키지를 임포트
import matplotlib as mpl
import matplotlib.pylab as plt
from mpl_toolkits.mplot3d import Axes3D
import seaborn as sns
import numpy as np
import scipy as sp
import pandas as pd
import statsmodels.api as sm
import sklearn as sk

# matplotlib 설정
mpl.use('Agg')

# seaborn 설정
sns.set()
sns.set_style("whitegrid")
sns.set_color_codes()
```

```python

```

```python

```


# Clase Nro. 6: Álgebra Lineal

> El álgebra lineal es una rama de las matemáticas que estudia conceptos tales como vectores, matrices, espacio dual, sistemas de ecuaciones lineales y en su enfoque de manera más formal, espacios vectoriales y sus transformaciones lineales

np.linalg: paquete de álgebra lineal en NumPy

- Funciones básicas
    - vectores
    - operaciones con vectores
    - norma de un vector
    - operaciones con matrices
    - inversa de una matriz
    - determinante
- Resolución de sistemas

_Con el manejo básico de arrays en Python con NumPy, es hora de pasar a operaciones más interesantes como son las propias del Álgebra Lineal._

_Los productos escalares y las inversiones de matrices están por todas partes en los programas científicos e ingenieriles, así que vamos a estudiar cómo se realizan en Python._

Como sabemos, las operaciones del álgebra lineal aparecen con mucha frecuencia a la hora de resolver sistemas de ecuaciones en derivadas parciales y en general al linealizar problemas de todo tipo, y suele ser necesario resolver sistemas con un número enorme de ecuaciones e incógnitas. Gracias a los arrays de NumPy podemos abordar este tipo de cálculos en Python, ya que todas las funciones están escritas en C o Fortran y tenemos la opción de usar bibliotecas optimizadas al límite.

El paquete de álgebra lineal en NumPy se llama `linalg`, así que importando NumPy con la convención habitual podemos acceder a él escribiendo `np.linalg`.

**Pero ¿Porqué álgebra lineal?**

Si entendemos álgebra lineal podremos desarrollar una mejor intuición para el aprendizaje automático y algoritmos. Además, también seremos capaces de desarrollar algoritmos desde cero y hacer variaciones de ellos.

**¿Qué es un vector?**

Un vector tiene tanto magnitud como dirección. Utilizamos vectores para describir, por ejemplo, la velocidad de objetos en movimiento.

Por dirección, se refiere a dónde en el espacio apunta la “flecha”, y la magnitud te dice qué tan lejos debes ir en esa dirección. Si solo tienes magnitud, pero no dirección, entonces estás hablando de escalares. Una vez que le das al escalar alguna dirección, se convierte en un vector.

Un vector se representa con una letra minúscula y una flecha arriba, apuntando a la derecha

**Suma de vectores**

Para sumar los vectores (x₁,y₁) y (x₂,y₂), sumamos los componentes correspondientes de cada vector: (x₁+x₂,y₁+y₂)

```python

```

**Multiplicar un vector por un escalar**

```python

```

**Producto punto o escalar**

Esto suena raro, pero no es más que la multiplicación de vectores cuyo resultado es un escalar. Para calcular el producto escalar de dos vectores, primero debemos multiplicar los elementos correspondientes y luego sumar los términos del producto.

En la siguiente fórmula lo vemos mucho más fácil

```python

```

**Norma vectorial**

La norma es solo otro término para la magnitud de un vector y se denota con dos lineas dobles (||) en cada lado. Se define como una raíz cuadrada de la suma de cuadrados para cada componente de un vector

Pasos:
- Elevar al cuadrado cada componente
- Suma todos los cuadrados
- Toma la raíz cuadrada

Trabajemos ahora con la formula:

```python

```

**Vector unitario**

Los vectores unitarios son aquellos cuya magnitud es exactamente 1 unidad. Son muy útiles por diversas razones. Específicamente, los vectores unitarios [0,1] y [1,0] juntos pueden formar cualquier otro vector.

Un vector unitario se denota con mayor frecuencia con un símbolo de sombrero (^) y se determina calculando la norma y luego dividiendo cada componente del vector con la norma.

Suena complejo, pero vamos a hacer un ejercicio para poder ver que es más complejo de lo que imaginamos

```python

```

## Operaciones con matrices

 ¿Qué es una matriz? Una matriz es simplemente un arreglo rectangular de números

En data science las usamos un montón, así no nos demos cuenta. Es muy importante de diferenciar un vector de una matriz. En pocas palabras un vector es una sola columna (atributo) en su conjunto de datos y una matriz es una colección de todas las columnas

```python

```

**Definición de matrices**

$A = \begin{equation}
\begin{bmatrix}
1 & 2 & 3\\
4 & 5 & 6\\
\end{bmatrix}
\end{equation}$

```python

```

$B = \begin{equation}
\begin{bmatrix}
5 & 5 & 5\\
5 & 5 & 5\\
\end{bmatrix}
\end{equation}$

```python

```

$v = \begin{equation}
\begin{bmatrix}
6\\
7\\
8
\end{bmatrix}
\end{equation}$

```python

```

**Suma de matrices**

```python

```

**Resta de matrices**

```python

```

**Multiplicación por un escalar**

```python

```

**Matriz transpuesta**

```python

```

**Matriz identidad**

```python

```

```python
#help(np.linalg)
```

Recordemos que si queremos usar una función de un paquete pero no queremos escribir la "ruta" completa cada vez, podemos usar la sintaxis `from package import func`:

```python
from numpy.linalg import norm, det
norm
```

El producto matricial usual (no el que se hace elemento a elemento, sino el del álgebra lineal) se calcula con la misma función que el producto matriz-vector y el producto escalar vector-vector: con la función `dot`, que **no** está en el paquete `linalg` sino directamente en `numpy` y no hace falta importarlo.

```python
np.dot
```

Una consideración importante a tener en cuenta es que en NumPy no hace falta ser estricto a la hora de manejar vectores como si fueran matrices columna, siempre que la operación sea consistente. Un vector es una matriz con una sola dimensión: por eso si calculamos su traspuesta no funciona.

```python
M = np.array([
    [1, 2],
    [3, 4]
])
v = np.array([1, -1])
```

v

```python
M.T
```

```python
v.T
```

```python
v.reshape(2,1).T
```

```python
u = np.dot(M, v)
u
```

**Multiplicación entre matrices**

```python

```

```python
# producto matricial

```

**Determinante de una matriz**

```python
# calcula el determinante de una matriz

```

**Inversa de una matriz**

Para calcular la matriz inversa en Python, use la función linalg.inv () del módulo numpy.

linalg.inv(x)

El parámetro x de la función es una matriz cuadrada invertible M definida con la función array() de numpy.

La función genera la matriz inversa M-1 de la matriz M.

> ¿Qué es la matriz inversa? La matriz inversa M-1 de una matriz cuadrada es una matriz tal que el producto M · M-1 es igual a una matriz de identidad I.

Encuentre la matriz inversa M-1 de la siguiente matriz invertible M.

```python
m=np.array([[3,4,-1],[2,0,1],[1,3,-2]])
```

```python
# Calcula la matriz inversa con la función linalg.inv().

```

La matriz de salida inversa es también un objeto array(). Se puede leer como una lista anidada. Los elementos de la matriz inversa son números reales.

Verificación. El producto de la matriz M para la matriz M-1 es una matriz de identidad.

```python
# verificación
```

**Otro ejemplo**
Encontrar el determinante y la matriz inversa de D.

$ D = \begin{equation}
\begin{bmatrix}
1 & 2\\
3 & 4\\
\end{bmatrix}
\end{equation}$

```python
D = np.array([[1, 2], [3, 4]])
D
```

```python
# cálculo del determinante

```

```python
# y el inverso

```

**Solución de ecuaciones con matrices**

1. Resolver el siguiente sistema de ecuaciones

$\begin{eqnarray}
3x + y  = 9 \\
x + 2y = 8 \\
\end{eqnarray}$

```python

```

### Ejercicios

1- Hallar el producto de estas dos matrices y su determinante:

$$\begin{pmatrix} 1 & 0 & 0 \\ 2 & 1 & 1 \\ -1 & 0 & 1 \end{pmatrix} \begin{pmatrix} 2 & 3 & -1 \\ 0 & -2 & 1 \\ 0 & 0 & 3 \end{pmatrix}$$

```python
from numpy.linalg import det
```

```python

```

```python

```

```python

```

3- Resuelva el siguiente sistema de ecuaciones

$\begin{eqnarray}
x + 2y  = 3 \\
3x + 4y = 5 \\
\end{eqnarray}$

```python

```

4- Un ingeniereo escribe dos ecuaciones que describen el circuito, como sigue:

$\begin{gather}
300I_{1} + 500(I_{1}-I_{2})-20 = 0\\
200I_{2} + 500(I_{2}-I_{1}) +10 = 0\\
\end{gather}$

Coloque las dos ecuaciones en forma estándar y resuelva las dos ecuaciones:

```python

```

5- Resolver el siguiente sistema de ecuaciones

```python

```

3- Resolver el siguiente sistema:

$$ \begin{pmatrix} 2 & 0 & 0 \\ -1 & 1 & 0 \\ 3 & 2 & -1 \end{pmatrix} \begin{pmatrix} 1 & 1 & 1 \\ 0 & 1 & 2 \\ 0 & 0 & 1 \end{pmatrix} \begin{pmatrix} x \\ y \\ z \end{pmatrix} = \begin{pmatrix} -1 \\ 3 \\ 0 \end{pmatrix} $$

```python

```

    array([[ 2,  2,  2],
           [-1,  0,  1],
           [ 3,  5,  6]])

```python

```

    array([ 0.5, -4.5,  3.5])

```python

```

    True


# When To Stop Fuzzing

In the past chapters, we have discussed several fuzzing techniques.  Knowing _what_ to do is important, but it is also important to know when to _stop_ doing things.  In this chapter, we will learn when to _stop fuzzing_ – and use a prominent example for this purpose: The *Enigma* machine that was used in the second world war by the navy of Nazi Germany to encrypt communications, and how Alan Turing and I.J. Good used _fuzzing techniques_ to crack ciphers for the Naval Enigma machine.

Turing did not only develop the foundations of computer science, the Turing machine. Together with his assistant I.J. Good, he also invented estimators of the probability of an event occuring that has never previously occured. We show how the Good-Turing estimator can be used to quantify the *residual risk* of a fuzzing campaign that finds no vulnerabilities. Meaning, we show how it estimates the probability of discovering a vulnerability when no vulnerability has been observed before throughout the fuzzing campaign.

We discuss means to speed up [coverage-based fuzzers](Coverage.ipynb) and introduce a range of estimation and extrapolation methodologies to assess and extrapolate fuzzing progress and residual risk.

**Prerequisites**

* _The chapter on [Coverage](Coverage.ipynb) discusses how to use coverage information for an executed test input to guide a coverage-based mutational greybox fuzzer_.
* Some knowledge of statistics is helpful.

```python
import bookutils
```

```python
import Fuzzer
import Coverage
```

## The Enigma Machine

It is autumn in the year of 1938. Turing has just finished his PhD at Princeton University demonstrating the limits of computation and laying the foundation for the theory of computer science. Nazi Germany is rearming. It has reoccupied the Rhineland and annexed Austria against the treaty of Versailles. It has just annexed the Sudetenland in Czechoslovakia and begins preparations to take over the rest of Czechoslovakia despite an agreement just signed in Munich.

Meanwhile, the British intelligence is building up their capability to break encrypted messages used by the Germans to communicate military and naval information. The Germans are using [Enigma machines](https://en.wikipedia.org/wiki/Enigma_machine) for encryption.  Enigma machines use a series of electro-mechanical rotor cipher machines to protect military communication.  Here is a picture of an Enigma machine:

By the time Turing joined the British Bletchley park, the Polish intelligence reverse engineered the logical structure of the Enigma machine and built a decryption machine called *Bomba* (perhaps because of the ticking noise they made). A bomba simulates six Enigma machines simultaneously and tries different decryption keys until the code is broken. The Polish bomba might have been the very _first fuzzer_.

Turing took it upon himself to crack ciphers of the Naval Enigma machine, which were notoriously hard to crack. The Naval Enigma used, as part of its encryption key, a three letter sequence called *trigram*. These trigrams were selected from a book, called *Kenngruppenbuch*, which contained all trigrams in a random order.

### The Kenngruppenbuch

Let's start with the Kenngruppenbuch (K-Book).

We are going to use the following Python functions.
* `random.shuffle(elements)` - shuffle *elements* and put items in random order.
* `random.choices(elements, weights)` - choose an item from *elements* at random. An element with twice the *weight* is twice as likely to be chosen.
* `log(a)` - returns the natural logarithm of a.
* `a ** b` - means `a` to the power of `b` (a.k.a. [power operator](https://docs.python.org/3/reference/expressions.html#the-power-operator))

```python
import string
```

```python
import numpy
from numpy import log
```

```python
import random
```

We start with creating the set of trigrams:

```python
letters = list(string.ascii_letters[26:])  # upper-case characters
trigrams = [str(a + b + c) for a in letters for b in letters for c in letters]
```

```python
random.shuffle(trigrams)
```

```python
trigrams[:10]
```

    ['TJK', 'NWV', 'LBM', 'AZC', 'GZP', 'ADE', 'DNO', 'OQL', 'FGK', 'IPT']

These now go into the Kenngruppenbuch.  However, it was observed that some trigrams were more likely chosen than others. For instance, trigrams at the top-left corner of any page, or trigrams on the first or last few pages were more likely than one somewhere in the middle of the book or page.  We reflect this difference in distribution by assigning a _probability_ to each trigram, using Benford's law as introduced in [Probabilistic Fuzzing](ProbabilisticGrammarFuzzer.ipynb).

Recall, that Benford's law assigns the $i$-th digit the probability $\log_{10}\left(1 + \frac{1}{i}\right)$ where the base 10 is chosen because there are 10 digits $i\in [0,9]$. However, Benford's law works for an arbitrary number of "digits". Hence, we assign the $i$-th trigram the probability $\log_b\left(1 + \frac{1}{i}\right)$ where the base $b$ is the number of all possible trigrams $b=26^3$.

```python
k_book = {}  # Kenngruppenbuch

for i in range(1, len(trigrams) + 1):
    trigram = trigrams[i - 1]
    # choose weights according to Benford's law
    k_book[trigram] = log(1 + 1 / i) / log(26**3 + 1)
```

Here's a random trigram from the Kenngruppenbuch:

```python
random_trigram = random.choices(list(k_book.keys()), weights=list(k_book.values()))[0]
random_trigram
```

    'PSK'

And this is its probability:

```python
k_book[random_trigram]
```

    0.0008284144853894445

### Fuzzing the Enigma

In the following, we introduce an extremely simplified implementation of the Naval Enigma based on the trigrams from the K-book. Of course, the encryption mechanism of the actual Enigma machine is much more sophisticated and worthy of a much more detailed investigation. We encourage the interested reader to follow up with further reading listed in the Background section.

The personell at Bletchley Park can only check whether an encoded message is encoded with a (guessed) trigram.
Our implementation `naval_enigma()` takes a `message` and a `key` (i.e., the guessed trigram).  If the given key matches the (previously computed) key for the message, `naval_enigma()` returns `True`.

```python
from Fuzzer import RandomFuzzer
from Fuzzer import Runner
```

```python
class EnigmaMachine(Runner):
    def __init__(self, k_book):
        self.k_book = k_book
        self.reset()

    def reset(self):
        """Resets the key register"""
        self.msg2key = {}

    def internal_msg2key(self, message):
        """Internal helper method.
           Returns the trigram for an encoded message."""
        if message not in self.msg2key:
            # Simulating how an officer chooses a key from the Kenngruppenbuch
            # to encode the message.
            self.msg2key[message] = \
                random.choices(list(self.k_book.keys()),
                               weights=list(self.k_book.values()))[0]
        trigram = self.msg2key[message]
        return trigram

    def naval_enigma(self, message, key):
        """Returns true if 'message' is encoded with 'key'"""
        if key == self.internal_msg2key(message):
            return True
        else:
            return False
```

To "fuzz" the `naval_enigma()`, our job will be to come up with a key that matches a given (encrypted) message.  Since the keys only have three characters, we have a good chance to achieve this in much less than a second.  (Of course, longer keys will be much harder to find via random fuzzing.)

```python
class EnigmaMachine(EnigmaMachine):
    def run(self, tri):
        """PASS if cur_msg is encoded with trigram tri"""
        if self.naval_enigma(self.cur_msg, tri):
            outcome = self.PASS
        else:
            outcome = self.FAIL

        return (tri, outcome)
```

Now we can use the `EnigmaMachine` to check whether a certain message is encoded with a certain trigram.

```python
enigma = EnigmaMachine(k_book)
enigma.cur_msg = "BrEaK mE. L0Lzz"
enigma.run("AAA")
```

    ('AAA', 'FAIL')

The simplest way to crack an encoded message is by brute forcing. Suppose, at Bletchley park they would try random trigrams until a message is broken.

```python
class BletchleyPark(object):
    def __init__(self, enigma):
        self.enigma = enigma
        self.enigma.reset()
        self.enigma_fuzzer = RandomFuzzer(
            min_length=3,
            max_length=3,
            char_start=65,
            char_range=26)

    def break_message(self, message):
        """Returning the trigram for an encoded message"""
        self.enigma.cur_msg = message
        while True:
            (trigram, outcome) = self.enigma_fuzzer.run(self.enigma)
            if outcome == self.enigma.PASS:
                break
        return trigram
```

How long does it take Bletchley park to find the key using this brute forcing approach?

```python
from Timer import Timer
```

```python
enigma = EnigmaMachine(k_book)
bletchley = BletchleyPark(enigma)

with Timer() as t:
    trigram = bletchley.break_message("BrEaK mE. L0Lzz")
```

Here's the key for the current message:

```python
trigram
```

    'XQC'

And no, this did not take long:

```python
'%f seconds' % t.elapsed_time()
```

    '0.295669 seconds'

```python
'Bletchley cracks about %d messages per second' % (1/t.elapsed_time())
```

    'Bletchley cracks about 3 messages per second'

### Turing's Observations
Okay, lets crack a few messages and count the number of times each trigram is observed.

```python
from collections import defaultdict
```

```python
n = 100  # messages to crack
```

```python
observed = defaultdict(int)
for msg in range(0, n):
    trigram = bletchley.break_message(msg)
    observed[trigram] += 1

# list of trigrams that have been observed
counts = [k for k, v in observed.items() if int(v) > 0]

t_trigrams = len(k_book)
o_trigrams = len(counts)
```

```python
"After cracking %d messages, we observed %d out of %d trigrams." % (
    n, o_trigrams, t_trigrams)
```

    'After cracking 100 messages, we observed 72 out of 17576 trigrams.'

```python
singletons = len([k for k, v in observed.items() if int(v) == 1])
```

```python
"From the %d observed trigrams, %d were observed only once." % (
    o_trigrams, singletons)
```

    'From the 72 observed trigrams, 63 were observed only once.'

Given a sample of previously used entries, Turing wanted to _estimate the likelihood_ that the current unknown entry was one that had been previously used, and further, to estimate the probability distribution over the previously used entries. This lead to the development of the estimators of the missing mass and estimates of the true probability mass of the set of items occuring in the sample. Good worked with Turing during the war and, with Turing’s permission, published the analysis of the bias of these estimators in 1953.

Suppose, after finding the keys for n=100 messages, we have observed the trigram "ABC" exactly $X_\text{ABC}=10$ times. What is the probability $p_\text{ABC}$ that "ABC" is the key for the next message? Empirically, we would estimate $\hat p_\text{ABC}=\frac{X_\text{ABC}}{n}=0.1$. We can derive the empirical estimates for all other trigrams that we have observed.  However, it becomes quickly evident that the complete probability mass is distributed over the *observed* trigrams. This leaves no mass for *unobserved* trigrams, i.e., the probability of discovering a new trigram. This is called the missing probability mass or the discovery probability.

Turing and Good derived an estimate of the *discovery probability* $p_0$, i.e., the probability to discover an unobserved trigram, as the number $f_1$ of trigrams observed exactly once divided by the total number $n$ of messages cracked:
$$
p_0 = \frac{f_1}{n}
$$
where $f_1$ is the number of singletons and $n$ is the number of cracked messages.

Lets explore this idea for a bit. We'll extend `BletchleyPark` to crack `n` messages and record the number of trigrams observed as the number of cracked messages increases.

```python
class BletchleyPark(BletchleyPark):
    def break_message(self, message):
        """Returning the trigram for an encoded message"""
        # For the following experiment, we want to make it practical
        #   to break a large number of messages. So, we remove the
        #   loop and just return the trigram for a message.
        #
        # enigma.cur_msg = message
        # while True:
        #     (trigram, outcome) = self.enigma_fuzzer.run(self.enigma)
        #     if outcome == self.enigma.PASS:
        #         break
        trigram = enigma.internal_msg2key(message)
        return trigram

    def break_n_messages(self, n):
        """Returns how often each trigram has been observed,
           and #trigrams discovered for each message."""
        observed = defaultdict(int)
        timeseries = [0] * n

        # Crack n messages and record #trigrams observed as #messages increases
        cur_observed = 0
        for cur_msg in range(0, n):
            trigram = self.break_message(cur_msg)

            observed[trigram] += 1
            if (observed[trigram] == 1):
                cur_observed += 1
            timeseries[cur_msg] = cur_observed

        return (observed, timeseries)
```

Let's crack 2000 messages and compute the GT-estimate.

```python
n = 2000        # messages to crack
```

```python
bletchley = BletchleyPark(enigma)
(observed, timeseries) = bletchley.break_n_messages(n)
```

Let us determine the Good-Turing estimate of the probability that the next trigram has not been observed before:

```python
singletons = len([k for k, v in observed.items() if int(v) == 1])
gt = singletons / n
gt
```

    0.401

We can verify the Good-Turing estimate empirically and compute the empirically determined probability that the next trigram has not been observed before. To do this, we repeat the following experiment `repeats=1000` times, reporting the average: If the next message is a new trigram, return 1, otherwise return 0. Note that here, we do not record the newly discovered trigrams as observed.

```python
repeats = 1000  # experiment repetitions
```

```python
newly_discovered = 0
for cur_msg in range(n, n + repeats):
    trigram = bletchley.break_message(cur_msg)
    if(observed[trigram] == 0):
        newly_discovered += 1

newly_discovered / repeats
```

    0.412

Looks pretty accurate, huh? The difference between estimates is reasonably small, probably below 0.03. However, the Good-Turing estimate did not nearly require as much computational resources as the empirical estimate. Unlike the empirical estimate, the Good-Turing estimate can be computed during the campaign. Unlike the empirical estimate, the Good-Turing estimate requires no additional, redundant repetitions.

In fact, the Good-Turing (GT) estimator often performs close to the best estimator for arbitrary distributions ([Try it here!](#Kenngruppenbuch)). Of course, the concept of *discovery* is not limited to trigrams. The GT estimator is also used in the study of natural languages to estimate the likelihood that we haven't ever heard or read the word we next encounter. The GT estimator is used in ecology to estimate the likelihood of discovering a new, unseen species in our quest to catalog all _species_ on earth. Later, we will see how it can be used to estimate the probability to discover a vulnerability when none has been observed, yet (i.e., residual risk).

Alan Turing was interested in the _complement_ $(1-GT)$ which gives the proportion of _all_ messages for which the Brits have already observed the trigram needed for decryption. For this reason, the complement is also called sample coverage. The *sample coverage* quantifies how much we know about decryption of all messages given the few messages we have already decrypted.

The probability that the next message can be decrypted with a previously discovered trigram is:

```python
1 - gt
```

    0.599

The *inverse* of the GT-estimate (1/GT) is a _maximum likelihood estimate_ of the expected number of messages that we can decrypt with previously observed trigrams before having to find a new trigram to decrypt the message.  In our setting, the number of messages for which we can expect to reuse previous trigrams before having to discover a new trigram is:

```python
1 / gt
```

    2.4937655860349124

But why is GT so accurate? Intuitively, despite a large sampling effort (i.e., cracking $n$ messages), there are still $f_1$ trigrams that have been observed only once. We could say that such "singletons" are very rare trigrams. Hence, the probability that the next messages is encoded with such a rare but observed trigram gives a good upper bound on the probability that the next message is encoded with an evidently much rarer, unobserved trigram. Since Turing's observation 80 years ago, an entire statistical theory has been developed around the hypothesis that rare, observed "species" are good predictors of unobserved species.

Let's have a look at the distribution of rare trigrams.

```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
```

```python
frequencies = [v for k, v in observed.items() if int(v) > 0]
frequencies.sort(reverse=True)
# Uncomment to see how often each discovered trigram has been observed
# print(frequencies)

# frequency of rare trigrams
plt.figure(num=None, figsize=(12, 4), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 2, 1)
plt.hist(frequencies, range=[1, 21], bins=numpy.arange(1, 21) - 0.5)
plt.xticks(range(1, 21))
plt.xlabel('# of occurances (e.g., 1 represents singleton trigrams)')
plt.ylabel('Frequency of occurances')
plt.title('Figure 1. Frequency of Rare Trigrams')

# trigram discovery over time
plt.subplot(1, 2, 2)
plt.plot(timeseries)
plt.xlabel('# of messages cracked')
plt.ylabel('# of trigrams discovered')
plt.title('Figure 2. Trigram Discovery Over Time');
```

```python
# Statistics for most and least often observed trigrams
singletons = len([v for k, v in observed.items() if int(v) == 1])
total = len(frequencies)

print("%3d of %3d trigrams (%.3f%%) have been observed   1 time (i.e., are singleton trigrams)."
      % (singletons, total, singletons * 100 / total))

print("%3d of %3d trigrams ( %.3f%%) have been observed %d times."
      % (1, total, 1 / total, frequencies[0]))
```

    802 of 1009 trigrams (79.485%) have been observed   1 time (i.e., are singleton trigrams).
      1 of 1009 trigrams ( 0.001%) have been observed 152 times.

The *majority of trigrams* have been observed only once, as we can see in Figure 1 (left). In other words, the majority of observed trigrams are "rare" singletons. In Figure 2 (right), we can see that discovery is in full swing. The trajectory seems almost linear. However, since there is a finite number of trigrams (26^3 = 17,576) trigram discovery will slow down and eventually approach an asymptote (the total number of trigrams).

### Boosting the Performance of BletchleyPark
Some trigrams have been observed very often. We call these "abundant" trigrams.

```python
print("Trigram : Frequency")
for trigram in sorted(observed, key=observed.get, reverse=True):
    if observed[trigram] > 10:
        print("    %s : %d" % (trigram, observed[trigram]))
```

    Trigram : Frequency
        TJK : 152
        LBM : 69
        NWV : 64
        AZC : 43
        GZP : 41
        ADE : 37
        DNO : 27
        OQL : 26
        TCO : 20
        BDA : 19
        ARO : 18
        IPT : 16
        FGK : 16
        MSV : 15
        ONO : 15
        EOR : 13
        JTV : 11
        IBT : 11
        PWN : 11

We'll speed up the code breaking by _trying the abundant trigrams first_.

First, we'll find out how many messages can be cracked by the existing brute forcing strategy at Bledgley park, given a maximum number of attempts. We'll also track the number of messages cracked over time (`timeseries`).

```python
class BletchleyPark(BletchleyPark):
    def __init__(self, enigma):
        super().__init__(enigma)
        self.cur_attempts = 0
        self.cur_observed = 0
        self.observed = defaultdict(int)
        self.timeseries = [None] * max_attempts * 2

    def break_message(self, message):
        """Returns the trigram for an encoded message, and
           track #trigrams observed as #attempts increases."""
        self.enigma.cur_msg = message
        while True:
            self.cur_attempts += 1                                 # NEW
            (trigram, outcome) = self.enigma_fuzzer.run(self.enigma)
            self.timeseries[self.cur_attempts] = self.cur_observed # NEW
            if outcome == self.enigma.PASS:
                break
        return trigram

    def break_max_attempts(self, max_attempts):
        """Returns #messages successfully cracked after a given #attempts."""
        cur_msg = 0
        n_messages = 0

        while True:
            trigram = self.break_message(cur_msg)

            # stop when reaching max_attempts
            if self.cur_attempts >= max_attempts:
                break

            # update observed trigrams
            n_messages += 1
            self.observed[trigram] += 1
            if (self.observed[trigram] == 1):
                self.cur_observed += 1
                self.timeseries[self.cur_attempts] = self.cur_observed
            cur_msg += 1
        return n_messages
```

`original` is the number of messages cracked by the bruteforcing strategy, given 100k attempts.  Can we beat this?

```python
max_attempts = 100000
```

```python
bletchley = BletchleyPark(enigma)
original = bletchley.break_max_attempts(max_attempts)
original
```

    3

Now, we'll create a boosting strategy by trying trigrams first that we have previously observed most often.

```python
class BoostedBletchleyPark(BletchleyPark):
    def break_message(self, message):
        """Returns the trigram for an encoded message, and
           track #trigrams observed as #attempts increases."""
        self.enigma.cur_msg = message

        # boost cracking by trying observed trigrams first
        for trigram in sorted(self.prior, key=self.prior.get, reverse=True):
            self.cur_attempts += 1
            (_, outcome) = self.enigma.run(trigram)
            self.timeseries[self.cur_attempts] = self.cur_observed
            if outcome == self.enigma.PASS:
                return trigram

        # else fall back to normal cracking
        return super().break_message(message)
```

`boosted` is the number of messages cracked by the boosted strategy.

```python
boostedBletchley = BoostedBletchleyPark(enigma)
boostedBletchley.prior = observed
boosted = boostedBletchley.break_max_attempts(max_attempts)
boosted
```

    23

We see that the boosted technique cracks substantially more messages. It is worthwhile to record how often each trigram is being used as key and try them in the order of their occurence.

***Try it***. *For practical reasons, we use a large number of previous observations as prior (`boostedBletchley.prior = observed`). You can try to  change the code such that the strategy uses the trigram frequencies (`self.observed`) observed **during** the campaign itself to boost the campaign. You will need to increase `max_attempts` and wait for a long while.*

Let's compare the number of trigrams discovered over time.

```python
# print plots
line_old, = plt.plot(bletchley.timeseries, label="Bruteforce Strategy")
line_new, = plt.plot(boostedBletchley.timeseries, label="Boosted Strategy")
plt.legend(handles=[line_old, line_new])
plt.xlabel('# of cracking attempts')
plt.ylabel('# of trigrams discovered')
plt.title('Trigram Discovery Over Time');
```

We see that the boosted fuzzer is constantly superior over the random fuzzer.

## Estimating the Probability of Path Discovery

<!-- ## Residual Risk: Probability of Failure after an Unsuccessful Fuzzing Campaign -->
<!-- Residual risk is not formally defined in this section, so I made the title a bit more generic -- AZ -->

So, what does Turing's observation for the Naval Enigma have to do with fuzzing _arbitrary_ programs? Turing's assistant I.J. Good extended and published Turing's work on the estimation procedures in Biometrica, a journal for theoretical biostatistics that still exists today. Good did not talk about trigrams. Instead, he calls them "species". Hence, the GT estimator is presented to estimate how likely it is to discover a new species, given an existing sample of individuals (each of which belongs to exactly one species).

Now, we can associate program inputs to species, as well. For instance, we could define the path that is exercised by an input as that input's species. This would allow us to _estimate the probability that fuzzing discovers a new path._ Later, we will see how this discovery probability estimate also estimates the likelihood of discovering a vulnerability when we have not seen one, yet (residual risk).

Let's do this. We identify the species for an input by computing a hash-id over the set of statements exercised by that input. In the [Coverage](Coverage.ipynb) chapter, we have learned about the [Coverage class](Coverage.ipynb#A-Coverage-Class) which collects coverage information for an executed Python function. As an example, the function [`cgi_decode()`](Coverage.ipynb#A-CGI-Decoder) was introduced. The function `cgi_decode()` takes a string encoded for a website URL and decodes it back to its original form.

Here's what `cgi_decode()` does and how coverage is computed.

```python
from Coverage import Coverage, cgi_decode
```

```python
encoded = "Hello%2c+world%21"
with Coverage() as cov:
    decoded = cgi_decode(encoded)
```

```python
decoded
```

    'Hello, world!'

```python
print(cov.coverage());
```

    {('cgi_decode', 10), ('cgi_decode', 19), ('cgi_decode', 16), ('cgi_decode', 22), ('cgi_decode', 25), ('cgi_decode', 31), ('cgi_decode', 12), ('cgi_decode', 9), ('cgi_decode', 15), ('cgi_decode', 18), ('cgi_decode', 24), ('cgi_decode', 21), ('cgi_decode', 30), ('cgi_decode', 8), ('cgi_decode', 11), ('cgi_decode', 17), ('cgi_decode', 20), ('cgi_decode', 26), ('cgi_decode', 23), ('cgi_decode', 32), ('__exit__', 79)}

### Trace Coverage
First, we will introduce the concept of execution traces, which are a coarse abstraction of the execution path taken by an input. Compared to the definition of path, a trace ignores the sequence in which statements are exercised or how often each statement is exercised.

* `pickle.dumps()` - serializes an object by producing a byte array from all the information in the object
* `hashlib.md5()` - produces a 128-bit hash value from a byte array

```python
import pickle
import hashlib
```

```python
def getTraceHash(cov):
    pickledCov = pickle.dumps(cov.coverage())
    hashedCov = hashlib.md5(pickledCov).hexdigest()
    return hashedCov
```

Remember our model for the Naval Enigma machine? Each message must be decrypted using exactly one trigram while multiple messages may be decrypted by the same trigram. Similarly, we need each input to yield exactly one trace hash while multiple inputs can yield the same trace hash.

Let's see whether this is true for our `getTraceHash()` function.

```python
inp1 = "a+b"
inp2 = "a+b+c"
inp3 = "abc"

with Coverage() as cov1:
    cgi_decode(inp1)
with Coverage() as cov2:
    cgi_decode(inp2)
with Coverage() as cov3:
    cgi_decode(inp3)
```

The inputs `inp1` and `inp2` execute the same statements:

```python
inp1, inp2
```

    ('a+b', 'a+b+c')

```python
cov1.coverage() - cov2.coverage()
```

    set()

The difference between both coverage sets is empty. Hence, the trace hashes should be the same:

```python
getTraceHash(cov1)
```

    'cc9ca13e7b89eebade1883fa6ad03a42'

```python
getTraceHash(cov2)
```

    'cc9ca13e7b89eebade1883fa6ad03a42'

```python
assert getTraceHash(cov1) == getTraceHash(cov2)
```

In contrast, the inputs `inp1` and `inp3` execute _different_ statements:

```python
inp1, inp3
```

    ('a+b', 'abc')

```python
cov1.coverage() - cov3.coverage()
```

    {('cgi_decode', 20)}

Hence, the trace hashes should be different, too:

```python
getTraceHash(cov1)
```

    'cc9ca13e7b89eebade1883fa6ad03a42'

```python
getTraceHash(cov3)
```

    '752221c11cc3bafc35add968deeaf935'

```python
assert getTraceHash(cov1) != getTraceHash(cov3)
```

### Measuring Trace Coverage over Time
In order to measure trace coverage for a `function` executing a `population` of fuzz inputs, we slightly adapt the `population_coverage()` function from the [Chapter on Coverage](Coverage.ipynb#Coverage-of-Basic-Fuzzing).

```python
def population_trace_coverage(population, function):
    cumulative_coverage = []
    all_coverage = set()
    cumulative_singletons = []
    cumulative_doubletons = []
    singletons = set()
    doubletons = set()

    for s in population:
        with Coverage() as cov:
            try:
                function(s)
            except BaseException:
                pass
        cur_coverage = set([getTraceHash(cov)])

        # singletons and doubletons -- we will need them later
        doubletons -= cur_coverage
        doubletons |= singletons & cur_coverage
        singletons -= cur_coverage
        singletons |= cur_coverage - (cur_coverage & all_coverage)
        cumulative_singletons.append(len(singletons))
        cumulative_doubletons.append(len(doubletons))

        # all and cumulative coverage
        all_coverage |= cur_coverage
        cumulative_coverage.append(len(all_coverage))

    return all_coverage, cumulative_coverage, cumulative_singletons, cumulative_doubletons
```

Let's see whether our new function really contains coverage information only for *two* traces given our three inputs for `cgi_decode`.

```python
all_coverage = population_trace_coverage([inp1, inp2, inp3], cgi_decode)[0]
assert len(all_coverage) == 2
```

Unfortunately, the `cgi_decode()` function is too simple. Instead, we will use the original Python [HTMLParser](https://docs.python.org/3/library/html.parser.html) as our test subject.

```python
from Fuzzer import RandomFuzzer
from Coverage import population_coverage
from html.parser import HTMLParser
```

```python
trials = 50000  # number of random inputs generated
```

Let's run a random fuzzer for $n=50000$ times and plot trace coverage over time.

```python
# create wrapper function
def my_parser(inp):
    parser = HTMLParser()  # resets the HTMLParser object for every fuzz input
    parser.feed(inp)
```

```python
# create random fuzzer
fuzzer = RandomFuzzer(min_length=1, max_length=100,
                      char_start=32, char_range=94)

# create population of fuzz inputs
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())

# execute and measure trace coverage
trace_timeseries = population_trace_coverage(population, my_parser)[1]

# execute and measure code coverage
code_timeseries = population_coverage(population, my_parser)[1]

# plot trace coverage over time
plt.figure(num=None, figsize=(12, 4), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 2, 1)
plt.plot(trace_timeseries)
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised')
plt.title('Trace Coverage Over Time')

# plot code coverage over time
plt.subplot(1, 2, 2)
plt.plot(code_timeseries)
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements covered')
plt.title('Code Coverage Over Time');
```

Above, we can see trace coverage (left) and code coverage (right) over time. Here are our observations.
1. **Trace coverage is more robust**. There are less sudden jumps in the graph compared to code coverage.
2. **Trace coverage is more fine grained.** There are more traces than statements covered in the end (y-axis).
3. **Trace coverage grows more steadily**. Code coverage exercises more than half the statements it has exercised after 50k inputs with the first input. Instead, the number of traces covered grows slowly and steadily since each input can yield only one execution trace.

It is for this reason that one of the most prominent and successful fuzzers today, american fuzzy lop (AFL), uses a similar *measure of progress* (a hash computed over the branches exercised by the input).

### Evaluating the Discovery Probability Estimate

Let's find out how the Good-Turing estimator performs as estimate of discovery probability when we are fuzzing to discover execution traces rather than trigrams.

To measure the empirical probability, we execute the same population of inputs (n=50000) and measure in regular intervals (`measurements=100` intervals). During each measurement, we repeat the following experiment `repeats=500` times, reporting the average: If the next input yields a new trace, return 1, otherwise return 0. Note that during these repetitions, we do not record the newly discovered traces as observed.

```python
repeats = 500      # experiment repetitions
measurements = 100  # experiment measurements
```

```python
emp_timeseries = []
all_coverage = set()
step = int(trials / measurements)

for i in range(0, trials, step):
    if i - step >= 0:
        for j in range(step):
            inp = population[i - j]
            with Coverage() as cov:
                try:
                    my_parser(inp)
                except BaseException:
                    pass
            all_coverage |= set([getTraceHash(cov)])

    discoveries = 0
    for _ in range(repeats):
        inp = fuzzer.fuzz()
        with Coverage() as cov:
            try:
                my_parser(inp)
            except BaseException:
                pass
        if getTraceHash(cov) not in all_coverage:
            discoveries += 1
    emp_timeseries.append(discoveries / repeats)
```

Now, we compute the Good-Turing estimate over time.

```python
gt_timeseries = []
singleton_timeseries = population_trace_coverage(population, my_parser)[2]
for i in range(1, trials + 1, step):
    gt_timeseries.append(singleton_timeseries[i - 1] / i)
```

Let's go ahead and plot both time series.

```python
line_emp, = plt.semilogy(emp_timeseries, label="Empirical")
line_gt, = plt.semilogy(gt_timeseries, label="Good-Turing")
plt.legend(handles=[line_emp, line_gt])
plt.xticks(range(0, measurements + 1, int(measurements / 5)),
           range(0, trials + 1, int(trials / 5)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('discovery probability')
plt.title('Discovery Probability Over Time');
```

Again, the Good-Turing estimate appears to be *highly accurate*. In fact, the empirical estimator has a much lower precision as indicated by the large swings. You can try and increase the number of repetitions (`repeats`) to get more precision for the empirical estimates, however, at the cost of waiting much longer.

### Discovery Probability Quantifies Residual Risk

Alright. You have gotten a hold of a couple of powerful machines and used them to fuzz a software system for several months without finding any vulnerabilities. Is the system vulnerable?

Well, who knows? We cannot say for sure; there is always some residual risk. Testing is not verification. Maybe the next test input that is generated reveals a vulnerability.

Let's say *residual risk* is the probability that the next test input reveals a vulnerability that has not been found, yet. Böhme \cite{Boehme2018stads} has shown that the Good-Turing estimate of the discovery probability is also an estimate of the maximum residual risk.

**Proof sketch (Residual Risk)**. Here is a proof sketch that shows that an estimator of discovery probability for an arbitrary definition of species gives an upper bound on the probability to discover a vulnerability when none has been found: Suppose, for each "old" species A (here, execution trace), we derive two "new" species: Some inputs belonging to A expose a vulnerability while others belonging to A do not. We know that _only_ species that do not expose a vulnerability have been discovered. Hence, _all_ species exposing a vulnerability and _some_ species that do not expose a vulnerability remain undiscovered. Hence, the probability to discover a new species gives an upper bound on the probability to discover (a species that exposes) a vulnerability. **QED**.

An estimate of the discovery probability is useful in many other ways.

1. **Discovery probability**. We can estimate, at any point during the fuzzing campaign, the probability that the next input belongs to a previously unseen species (here, that it yields a new execution trace, i.e., exercises a new set of statements).
2. **Complement of discovery probability**. We can estimate the proportion of *all* inputs the fuzzer can generate for which we have already seen the species (here, execution traces). In some sense, this allows us to quantify the *progress of the fuzzing campaign towards completion*: If the probability to discovery a new species is too low, we might as well abort the campaign.
3. **Inverse of discovery probability**. We can predict the number of test inputs needed, so that we can expect the discovery of a new species (here, execution trace).

## How Do We Know When to Stop Fuzzing?

In fuzzing, we have measures of progress such as [code coverage](Coverage.ipynb) or [grammar coverage](GrammarCoverageFuzzer.ipynb). Suppose, we are interested in covering all statements in the program. The _percentage_ of statements that have already been covered quantifies how "far" we are from completing the fuzzing campaign. However, sometimes we know only the _number_ of species $S(n)$ (here, statements) that have been discovered after generating $n$ fuzz inputs. The percentage $S(n)/S$ can only be computed if we know the _total number_ of species $S$. Even then, not all species may be feasible.

### A Success Estimator

If we do not _know_ the total number of species, then let's at least _estimate_ it: As we have seen before, species discovery slows down over time. In the beginning, many new species are discovered. Later, many inputs need to be generated before discovering the next species. In fact, given enough time, the fuzzing campaign approaches an _asymptote_. It is this asymptote that we can estimate.

In 1984, Anne Chao, a well-known theoretical bio-statistician, has developed an estimator $\hat S$ which estimates the asymptotic total number of species $S$:
\begin{align}
\hat S_\text{Chao1} = \begin{cases}
S(n) + \frac{f_1^2}{2f_2} & \text{if $f_2>0$}\\
S(n) + \frac{f_1(f_1-1)}{2} & \text{otherwise}
\end{cases}
\end{align}
* where $f_1$ and $f_2$ is the number of singleton and doubleton species, respectively (that have been observed exactly once or twice, resp.), and
* where $S(n)$ is the number of species that have been discovered after generating $n$ fuzz inputs.

So, how does Chao's estimate perform? To investigate this, we generate `trials=400000` fuzz inputs using a fuzzer setting that allows us to see an asymptote in a few seconds: We measure trace coverage. After half-way into our fuzzing campaign (`trials`/2=100000), we generate Chao's estimate $\hat S$ of the asymptotic total number of species. Then, we run the remainer of the campaign to see the "empirical" asymptote.

```python
trials = 400000
fuzzer = RandomFuzzer(min_length=2, max_length=4,
                      char_start=32, char_range=32)
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())

_, trace_ts, f1_ts, f2_ts = population_trace_coverage(population, my_parser)
```

```python
time = int(trials / 2)
time
```

    200000

```python
f1 = f1_ts[time]
f2 = f2_ts[time]
Sn = trace_ts[time]
if f2 > 0:
    hat_S = Sn + f1 * f1 / (2 * f2)
else:
    hat_S = Sn + f1 * (f1 - 1) / 2
```

After executing `time` fuzz inputs (half of all), we have covered this many traces:

```python
time
```

    200000

```python
Sn
```

    66

We can estimate there are this many traces in total:

```python
hat_S
```

    73.2

Hence, we have achieved this percentage of the estimate:

```python
100 * Sn / hat_S
```

    90.1639344262295

After executing `trials` fuzz inputs, we have covered this many traces:

```python
trials
```

    400000

```python
trace_ts[trials - 1]
```

    72

The accuracy of Chao's estimator is quite reasonable. It isn't always accurate -- particularly at the beginning of a fuzzing campaign when the [discovery probability](WhenIsEnough.ipynb#Measuring-Trace-Coverage-over-Time) is still very high. Nevertheless, it demonstrates the main benefit of reporting a percentage to assess the progress of a fuzzing campaign towards completion.

***Try it***. *Try setting `trials` to 1 million and `time` to `int(trials / 4)`.*

### Extrapolating Fuzzing Success
<!--  ## Cost-Benefit Analysis: Extrapolating the Number of Species Discovered -->

Suppose you have run the fuzzer for a week, which generated $n$ fuzz inputs and discovered $S(n)$ species (here, covered $S(n)$ execution traces). Instead, of running the fuzzer for another week, you would like to *predict* how many more species you would discover. In 2003, Anne Chao and her team developed an extrapolation methodology to do just that. We are interested in the number $S(n+m^*)$ of species discovered if $m^*$ more fuzz inputs were generated:

\begin{align}
\hat S(n + m^*) = S(n) + \hat f_0 \left[1-\left(1-\frac{f_1}{n\hat f_0 + f_1}\right)^{m^*}\right]
\end{align}
* where $\hat f_0=\hat S - S(n)$ is an estimate of the number $f_0$ of undiscovered species, and
* where $f_1$ is the number of singleton species, i.e., those we have observed exactly once.

The number $f_1$ of singletons, we can just keep track of during the fuzzing campaign itself. The estimate of the number $\hat f_0$ of undiscovered species, we can simply derive using Chao's estimate $\hat S$ and the number of observed species $S(n)$.

Let's see how Chao's extrapolator performs by comparing the predicted number of species to the empirical number of species.

```python
prediction_ts = [None] * time
f0 = hat_S - Sn

for m in range(trials - time):
    assert (time * f0 + f1) != 0 , 'time:%s f0:%s f1:%s' % (time, f0,f1)
    prediction_ts.append(Sn + f0 * (1 - (1 - f1 / (time * f0 + f1)) ** m))
```

```python
plt.figure(num=None, figsize=(12, 3), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 3, 1)
plt.plot(trace_ts, color='white')
plt.plot(trace_ts[:time])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised')

plt.subplot(1, 3, 2)
line_cur, = plt.plot(trace_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised')

plt.subplot(1, 3, 3)
line_emp, = plt.plot(trace_ts, color='grey', label="Actual progress")
line_cur, = plt.plot(trace_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_emp, line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of traces exercised');
```

The prediction from Chao's extrapolator looks quite accurate. We make a prediction at `time=trials/4`. Despite an extrapolation by 3 times (i.e., at trials), we can see that the predicted value (black, dashed line) closely matches the empirical value (grey, solid line).

***Try it***. Again, try setting `trials` to 1 million and `time` to `int(trials / 4)`.

## Lessons Learned

* One can measure the _progress_ of a fuzzing campaign (as species over time, i.e., $S(n)$).
* One can measure the _effectiveness_ of a fuzzing campaign (as asymptotic total number of species $S$).
* One can estimate the _effectiveness_ of a fuzzing campaign using the Chao1-estimator $\hat S$.
* One can extrapolate the _progress_ of a fuzzing campaign, $\hat S(n+m^*)$.
* One can estimate the _residual risk_ (i.e., the probability that a bug exists that has not been found) using the Good-Turing estimator $GT$ of the species discovery probability.

## Next Steps

This chapter is the last in the book!  If you want to continue reading, have a look at the [Appendices](99_Appendices.ipynb).  Otherwise, _make use of what you have learned and go and create great fuzzers and test generators!_

## Background

* A **statistical framework for fuzzing**, inspired from ecology. Marcel Böhme. [STADS: Software Testing as Species Discovery](https://mboehme.github.io/paper/TOSEM18.pdf). ACM TOSEM 27(2):1--52
* Estimating the **discovery probability**: I.J. Good. 1953. [The population frequencies of species and the
estimation of population parameters](https://www.jstor.org/stable/2333344). Biometrika 40:237–264.
* Estimating the **asymptotic total number of species** when each input can belong to exactly one species: Anne Chao. 1984. [Nonparametric estimation of the number of classes in a population](https://www.jstor.org/stable/4615964). Scandinavian Journal of Statistics 11:265–270
* Estimating the **asymptotic total number of species** when each input can belong to one or more species: Anne Chao. 1987. [Estimating the population size for capture-recapture data with unequal catchability](https://www.jstor.org/stable/2531532). Biometrics 43:783–791
* **Extrapolating** the number of discovered species: Tsung-Jen Shen, Anne Chao, and Chih-Feng Lin. 2003. [Predicting the Number of New Species in Further Taxonomic Sampling](http://chao.stat.nthu.edu.tw/wordpress/paper/2003_Ecology_84_P798.pdf). Ecology 84, 3 (2003), 798–804.

## Exercises
I.J. Good and Alan Turing developed an estimator for the case where each input belongs to exactly one species. For instance, each input yields exactly one execution trace (see function [`getTraceHash`](#Trace-Coverage)). However, this is not true in general. For instance, each input exercises multiple statements and branches in the source code. Generally, each input can belong to one *or more* species.

In this extended model, the underlying statistics are quite different. Yet, all estimators that we have discussed in this chapter turn out to be almost identical to those for the simple, single-species model. For instance, the Good-Turing estimator $C$ is defined as
$$C=\frac{Q_1}{n}$$
where $Q_1$ is the number of singleton species and $n$ is the number of generated test cases.
Throughout the fuzzing campaign, we record for each species the *incidence frequency*, i.e., the number of inputs that belong to that species. Again, we define a species $i$ as *singleton species* if we have seen exactly one input that belongs to species $i$.

### Exercise 1: Estimate and Evaluate the Discovery Probability for Statement Coverage

In this exercise, we create a Good-Turing estimator for the simple fuzzer.

#### Part 1: Population Coverage

Implement a function `population_stmt_coverage()` as in [the section on estimating discovery probability](#Estimating-the-Discovery-Probability) that monitors the number of singletons and doubletons over time, i.e., as the number $i$ of test inputs increases.

```python
from Coverage import population_coverage, Coverage
...
```

**Solution.** Here we go:

```python
def population_stmt_coverage(population, function):
    cumulative_coverage = []
    all_coverage = set()
    cumulative_singletons = []
    cumulative_doubletons = []
    singletons = set()
    doubletons = set()

    for s in population:
        with Coverage() as cov:
            try:
                function(s)
            except BaseException:
                pass
        cur_coverage = cov.coverage()

        # singletons and doubletons
        doubletons -= cur_coverage
        doubletons |= singletons & cur_coverage
        singletons -= cur_coverage
        singletons |= cur_coverage - (cur_coverage & all_coverage)
        cumulative_singletons.append(len(singletons))
        cumulative_doubletons.append(len(doubletons))

        # all and cumulative coverage
        all_coverage |= cur_coverage
        cumulative_coverage.append(len(all_coverage))

    return all_coverage, cumulative_coverage, cumulative_singletons, cumulative_doubletons
```

#### Part 2: Population

Use the random `fuzzer(min_length=1, max_length=1000, char_start=0, char_range=255)` from [the chapter on Fuzzers](Fuzzer.ipynb) to generate a population of $n=10000$ fuzz inputs.

```python
from Fuzzer import RandomFuzzer
from html.parser import HTMLParser
...
```

    Ellipsis

**Solution.**  This is fairly straightforward:

```python
trials = 2000  # increase to 10000 for better convergences. Will take a while..
```

We create a wrapper function...

```python
def my_parser(inp):
    parser = HTMLParser()  # resets the HTMLParser object for every fuzz input
    parser.feed(inp)
```

... and a random fuzzer:

```python
fuzzer = RandomFuzzer(min_length=1, max_length=1000,
                      char_start=0, char_range=255)
```

We fill the population:

```python
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())
```

#### Part 3: Estimating Probabilities

Execute the generated inputs on the Python HTML parser (`from html.parser import HTMLParser`) and estimate the probability that the next input covers a previously uncovered statement (i.e., the discovery probability) using the Good-Turing estimator.

**Solution.** Here we go:

```python
measurements = 100  # experiment measurements
step = int(trials / measurements)

gt_timeseries = []
singleton_timeseries = population_stmt_coverage(population, my_parser)[2]
for i in range(1, trials + 1, step):
    gt_timeseries.append(singleton_timeseries[i - 1] / i)
```

#### Part 4: Empirical Evaluation

Empirically evaluate the accuracy of the Good-Turing estimator (using $10000$ repetitions) of the probability to cover new statements using the experimental procedure at the end of [the section on estimating discovery probability](#Estimating-the-Discovery-Probability).

**Solution.** This is as above:

```python
# increase to 10000 for better precision (less variance). Will take a while..
repeats = 100
```

```python
emp_timeseries = []
all_coverage = set()
for i in range(0, trials, step):
    if i - step >= 0:
        for j in range(step):
            inp = population[i - j]
            with Coverage() as cov:
                try:
                    my_parser(inp)
                except BaseException:
                    pass
            all_coverage |= cov.coverage()

    discoveries = 0
    for _ in range(repeats):
        inp = fuzzer.fuzz()
        with Coverage() as cov:
            try:
                my_parser(inp)
            except BaseException:
                pass
        # If intersection not empty, a new stmt was (dis)covered
        if cov.coverage() - all_coverage:
            discoveries += 1
    emp_timeseries.append(discoveries / repeats)
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
line_emp, = plt.semilogy(emp_timeseries, label="Empirical")
line_gt, = plt.semilogy(gt_timeseries, label="Good-Turing")
plt.legend(handles=[line_emp, line_gt])
plt.xticks(range(0, measurements + 1, int(measurements / 5)),
           range(0, trials + 1, int(trials / 5)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('discovery probability')
plt.title('Discovery Probability Over Time');
```

### Exercise 2: Extrapolate and Evaluate Statement Coverage

In this exercise, we use Chao's extrapolation method to estimate the success of fuzzing.

#### Part 1: Create Population

Use the random `fuzzer(min_length=1, max_length=1000, char_start=0, char_range=255)` to generate a population of $n=400000$ fuzz inputs.

**Solution.** Here we go:

```python
trials = 400  # Use 400000 for actual solution.  This takes a while!
```

```python
population = []
for i in range(trials):
    population.append(fuzzer.fuzz())

_, stmt_ts, Q1_ts, Q2_ts = population_stmt_coverage(population, my_parser)
```

#### Part 2: Compute Estimate

Compute an estimate of the total number of statements $\hat S$ after $n/4=100000$ fuzz inputs were generated. In the extended model, $\hat S$ is computed as
\begin{align}
\hat S_\text{Chao1} = \begin{cases}
S(n) + \frac{Q_1^2}{2Q_2} & \text{if $Q_2>0$}\\
S(n) + \frac{Q_1(Q_1-1)}{2} & \text{otherwise}
\end{cases}
\end{align}
  * where $Q_1$ and $Q_2$ is the number of singleton and doubleton statements, respectively (i.e., statements that have been exercised by exactly one or two fuzz inputs, resp.), and
  * where $S(n)$ is the number of statements that have been (dis)covered after generating $n$ fuzz inputs.

**Solution.** Here we go:

```python
time = int(trials / 4)
Q1 = Q1_ts[time]
Q2 = Q2_ts[time]
Sn = stmt_ts[time]

if Q2 > 0:
    hat_S = Sn + Q1 * Q1 / (2 * Q2)
else:
    hat_S = Sn + Q1 * (Q1 - 1) / 2

print("After executing %d fuzz inputs, we have covered %d **(%.1f %%)** statements.\n" % (time, Sn, 100 * Sn / hat_S) +
      "After executing %d fuzz inputs, we estimate there are %d statements in total.\n" % (time, hat_S) +
      "After executing %d fuzz inputs, we have covered %d statements." % (trials, stmt_ts[trials - 1]))
```

    After executing 100 fuzz inputs, we have covered 132 **(70.6 %)** statements.
    After executing 100 fuzz inputs, we estimate there are 187 statements in total.
    After executing 400 fuzz inputs, we have covered 155 statements.

#### Part 3: Compute and Evaluate Extrapolator

Compute and evaluate Chao's extrapolator by comparing the predicted number of statements to the empirical number of statements.

**Solution.** Here's our solution:

```python
prediction_ts = [None] * time
Q0 = hat_S - Sn

for m in range(trials - time):
    prediction_ts.append(Sn + Q0 * (1 - (1 - Q1 / (time * Q0 + Q1)) ** m))
```

```python
plt.figure(num=None, figsize=(12, 3), dpi=80, facecolor='w', edgecolor='k')
plt.subplot(1, 3, 1)
plt.plot(stmt_ts, color='white')
plt.plot(stmt_ts[:time])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements exercised')

plt.subplot(1, 3, 2)
line_cur, = plt.plot(stmt_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements exercised')

plt.subplot(1, 3, 3)
line_emp, = plt.plot(stmt_ts, color='grey', label="Actual progress")
line_cur, = plt.plot(stmt_ts[:time], label="Ongoing fuzzing campaign")
line_pred, = plt.plot(prediction_ts, linestyle='--',
                      color='black', label="Predicted progress")
plt.legend(handles=[line_emp, line_cur, line_pred])
plt.xticks(range(0, trials + 1, int(time)))
plt.xlabel('# of fuzz inputs')
plt.ylabel('# of statements exercised');
```