# Problems in numerical earthquake modeling

Last updated Sept 9th, 2021.

There are many many interesting problems to work on in the world of earthquake modeling and simulation. This page is intended to explain some of the interesting topics to software and numerics experts outside the field and is intended to be continually updated. Currently, it is mostly unfinished with some sections completely blank and it is too light on literature review.

At the moment, the tooling is a major limitation for the field. Improving both the algorithms and the software would be a very productive contribution.

In particular, boundary integral equation (BIE) methods are very promising for a set of problems where the object under study is itself a boundary! So, let's focus on boundary integral (and element) methods. At the moment, the standard methodology is to use constant basis function boundary elements for [which](okada) [analytical](tdes) [solutions](qdyn) have been derived. Generally, these elements are used to represent the fault in either a fullspace or halfspace.  There are some efforts to move beyond these simple methods but those efforts have not yet panned out. The main barrier is lack of software and knowledge of appropriate algorithms. I think QBX-based integration combined with FMM or H-matrix acceleration would be very powerful.

## Target capabilities
So, what would that hypothetical BIE software look like? Let's start by just listing the types of applications that would be nice to tackle, starting with "absolutely critical" down to "nice to have".

* linear elasticity
* complex realistic geometries
    * faults
    * topography
    * material interfaces
    * fault roughness
* large models (large enough that $O(n^2)$ methods won't work)
* high performance
* earthquake rupture modeling
* linear and nonlinear viscoelastic behavior
* elastic-plastic behavior
* seismic waves

This list is probably biased to my personal preferences and the things I've worked on so far, but also covers a huge range of problems. Solving even two or three of these bullet points would be a huge contribution!

## A starting point!

As an initial application of QBX/FMM methodology, I would propose working on a 2D antiplane earthquake rupture problem. A useful starting reference would be "Quasi‐dynamic versus fully dynamic simulations of earthquakes and aseismic slip with and without enhanced coseismic weakening" by Thomas and others. I'd especially recommend sections 2.1 and 2.2.

The basic model that has been used in the past here is that of a horizontally infinite half-space free surface above a out-of-plane infinite strike slip fault. Bear with me if these words aren't making sense. In this "antiplane shear" setting, the Navier equations of elasticity reduce to the Poisson equation. Combining this with fault frictional ODEs into a DAE system, we get an exciting but fairly simple time dependent model of earthquakes.

Let's dig more into the details of the geometry. There are two main surfaces:
- The free surface. We assume that the atmosphere applies negligible force on the surface of the Earth. As a result, the traction on the surface is zero. Or in Poisson-speak, the potential gradient is zero. The infinite free surface, is of course problematic in a numerical setting and must be made finite.
  - One approach is to just truncate the flat surface. However that introduces some problems because the domain is no longer closed.
  - Another approach is to adopt a more realistic geometry and simply use an Earth-sized circle for the free surface. That also introduces problems because the length-scale of the Earth is much much larger than that of most fault systems.
  - A third approach is to truncate the domain with some physically-justifiable far-field vertical and basal boundary conditions - basically replace the half-space with a box. This is probably the best approach in many situations, but it does require being very careful about those far-field boundary conditions. Also, doing this in a naive way will introduce some numerically annoying corners from the corners of the box. Smoothing out those corners makes the numerics easier!
- The fault. Here, we will be updating some ODEs in order to compute the slip rate. Slip on a fault is an earthquake science term for crack displacement. See below in the "Background" section for an explanation of how to deal with cracks in a BIE approach to elasticity. Ultimately, the jump in displacement requires using a hypersingular integral equation.

```python
"""
A simple analytical solution for constant slip on a fault underneath a free surface.
See "Earthquake and Volcano Deformation", Segall chapter 2.
"""
import matplotlib.pyplot as plt
%config InlineBackend.figure_format='retina'

N = 100
xs = np.linspace(-1, 1, N)
ys = np.linspace(-2, 0, N)
X, Y = np.meshgrid(xs, ys)

# We need to account for the fault tip and it's image across the free surface
# in order to impose zero traction on the free surface.
fault_term = np.arctan((Y - 1) / X)
image_term = np.arctan((Y + 1) / X)
u3 = -1 / (2 * np.pi) * (fault_term - image_term)

plt.plot([-10, 10], [0,0], linewidth = 3)
plt.plot([0, 0], [0, -1], 'r-', linewidth = 3)
plt.contourf(X, Y, u3, levels = np.linspace(-0.5, 0.5, 11))
plt.axis('equal')
plt.xlim([-1, 1])
plt.ylim([-1, 0])
plt.xlabel(r'$x$')
plt.ylabel(r'$y$')
cbar = plt.colorbar()
cbar.set_label('displacement')
plt.title('displacement resulting from unit slip on a fault beneath a free surface')
plt.show()
```

For many methods, the fault introduces a lot of difficulties because of the need to integrate a hypersingular kernel. QBX-based integration has none of those problems. However, there is a critical problem, *What do we do about the fault-surface intersection?* This seems closely related with corner-handling in parametrized curves. However, it actually seems a bit easier because there is no singular physical behavior.

Let's return to those ODEs that we are integrating on the fault. [I wrote an accessible introduction to a common form of fault friction here](https://tbenthompson.com/post/block_slider/) This type of *rate and state friction* is a common framework for frictional earthquake modeling. For more info, "Laboratory-Derived Friction Laws and Their Application to Seismic Faulting" by Chris Marone is a useful review that covers a lot of the study of fault friction. However, going into too much depth is probably not necessary initially.

Ultimately, in pseudocode, an earthquake simulation will often look something like this, alternating between solving the PDEs for elasticity and solving the fault-frictional equations:
```
while not done:
    slip += velocity * dt
    fault_traction = elastic_solver(slip)
    velocity = friction_solver(fault_traction)
```

#### Test problems
I don't know of any analytical solutions for this type of earthquake simulation, but there are several useful testing avenues:
- The elastic solver can be thoroughly tested. There are a wide variety of idealized elasticity solutions.
  - The analytical solutions for the displacement from a constant amount of slip on a line segment, triangle or rectangular are available. These are called "dislocation solutions" and are common building blocks for boudnary element methods. For line segments, see sections 2.1 and 2.2 of "Earthquake and Volcano Deformation" by Paul Segall. In 3D with constant slip on a triangle, see [the software here](https://github.com/tbenthompson/cutde) and the paper referenced there.
  - The solution for slip on a line segment in a 2D model is particularly useful. I replicated [it using QBX-based BIE methods here](https://tbenthompson.com/book/c1qbx/part3_topo_solver.html). One specific solution that is very useful is exactly
  - Another useful solution is that of a compressed sphere. Solutions for this problem and other similar "elementary elasticity" problems can be found in "Theory of Elasticity" by Timoshenko and Goodier. That book is a treasure trove of useful analytical test problems.
- Several extensions of the elastic solver can be tested in special cases, for example, if the deeper regions are visco-elastic instead of elastic.
- There is a project, "The SCEC Sequences of Earthquakes and Aseismic Slip Project" that is a community to benchmark earthquake simulation software solutions against each other. While that doesn't prove that any particular piece of software is correct, it does help to identify errors.
- I've been working on an implementation of body force handling which would enable *method of manufactured solutions* testing.

## Example past applications

## A couple potential future applications

The space of possibilities is huge, but I think the most exciting potential research is in geometrically realistic modeling of actual fault systems. Simulating the interaction of many faults at once would be a powerful demonstration of methodological capabilities and also have real scientific value. It would also provide a foundation for connecting a lot more empirical and observational evidence into theoretical models.

Another area of personal interest is evaluating Lyapunov time scales for chaotic behavior in an earthquake system as a function of different parameters and physics. This would be foundational work in the study of whether earthquakes are predictable or not.

# Notes and more details

## Linear elasticity

Let's focus on elasticity as it appears in a BIE method. If you need a more basic introduction to linear elasticity, there's a lot out there. Some terms to search are "elastostatics", "elastodynamics", "solid mechanics". (Link to some books and documents http://www.cns.gatech.edu/~predrag/GTcourses/PHYS-4421-04/lautrup/2.8/solids.pdf)

For someone coming from an applied math or numerics background, relating the concepts here to the Poisson equation can be helpful. We can extend the concept of a scalar potential into the analogous vector displacement. And extend the vector potential gradient into the tensor stress.

### Differential form

In differential form, the equations for (static, isotropic, homogeneous) linear elasticity are:

\begin{align}
\nabla \cdot \sigma &= f\\
\epsilon &= (\nabla u) + (\nabla u)^T\\
\sigma &= \lambda \delta_{ij} \epsilon_{kk} + 2\mu\epsilon_{ij}
\end{align}

where $\sigma$ is the stress tensor, $f$ is the vector body force, $\epsilon$ is the strain tensor, $u$ is the displacement vector, $\lambda$ is the Lamé parameter, $\mu$ is the shear modulus and $\delta_{ij}$ is the Kronecker delta. The first equation is a differential statement of Newton's law. The second and third statements together are a statement of infinitesimal strain linear elasticity. These equations are sometimes called the Navier equations or the Navier-Cauchy equations (not to be mistaken for the Navier-Stokes equations!!).

### Integral form

The Somigliana identity is an integral representation of the displacements in an elastic body in terms of the displacements and stresses on the boundary of that body. We can write the Somigliana identity cleanly by adopting index notation:

\begin{equation}
    u_{k}(\mathbf{x}) + \int_{S} T^{*}_{kj}(\mathbf{x},\mathbf{y}) u_j(\mathbf{y}) d\mathbf{y} = \int_{S} U^*_{kj}(\mathbf{x},\mathbf{y}) t_j(\mathbf{y}) d\mathbf{y} ~~~~~ \forall \mathbf{x} \in V
    \label{somigliana}
\end{equation}

where $S = \partial V$ (the boundary of $V$), $u_k(\textbf{x})$ is $k$-th component the displacement field at $\mathbf{x}$, $t_k(\textbf{x})$ is $k$-th component of the traction field (note: $t_k = \sigma_{kj} n_j$ -- the traction is "stress applied to a surface") and

\begin{equation}
    U^*_{kj}(\mathbf{x},\mathbf{y}) = \frac{1}{16\pi\mu(1 - \nu)r}\big[(3 - 4\nu)\delta_{kj} + r_{,k}r_{,j}\big]
\end{equation}
\begin{equation}
    T^*_{kj}(\mathbf{x},\mathbf{y}) = \frac{-1}{8\pi(1-\nu)r^2}\big[ \{(1-2\nu)\delta_{kj} + 3r_{,k}r_{,j} \}\frac{\partial r}{\partial \mathbf{n}} - (1 - 2\nu)\{n_jr_{,k} - n_kr_{,j}\}\big]
\end{equation}

are the fundamental elastic and traction Green's functions of elasticity,
with $\delta_{ij}$ is the Kronecker delta, $\mu$ as the elastic shear modulus, $\nu$ as the poisson ratio, $\mathbf{n}$ is the normal vector to $S$ at $y$, $r = \|\mathbf{x} - \mathbf{y}\|$ and $r_{,i} = \partial r/\partial x_i$.

If traction, $t$, is known on the surface, we have Neumann problem. If displacement $u$ is known, we have a Dirichlet problem.

### Cracks

So far, we've ignored cracks or faults in the elastic body. However, for earthquake science, these are the most important surfaces in the problem. The standard approach to treat cracks is to separate the crack into two infinitesimally separated surfaces, $C^+$ and $C^-$. Then, introducing the displacement discontinuity $\Delta u = u^+ - u^-$, enforcing force balance across the crack, $t^+ + t^- = 0$, the $U^*$ integrals cancel each other out and we get:
\begin{equation}
    \begin{split}
        u_{k}(\mathbf{x}) = -\int_{F} T^{*}_{kj}(\mathbf{x},\mathbf{y}) \Delta u_j(\mathbf{y}) d\mathbf{y}
    \end{split}
    \label{somigliana_crack}
\end{equation}
where $F$ is the crack surface. Note that the $U^*$ integral over $F$ on the right hand side has dropped out to the force balance assumption.

For folks who've seen some potential theory, this is starting to look a lot like the double layer potential.

### Traction BIE

Critically, the traction on the fault surface does not appear in equation (\ref{somigliana_crack}). Solving for traction given fault slip is a critical step in many earthquake simulation problems. To solve this issue, another integral equation can be derived from the Somigliana identity by taking gradients and applying the elastic constitutive equations:
\begin{equation}
    \sigma_{lk}(\mathbf{x}) - \int_{S} A^*_{lkj}(\mathbf{x},\mathbf{y}) t_j(\mathbf{y}) d\mathbf{y} = -\int_{S} H^{*}_{lkj}(\mathbf{x},\mathbf{y}) u_j(\mathbf{y}) d\mathbf{y} ~~~~~ \forall \mathbf{x} \in V
    \label{tractionbie}
\end{equation}
with
\begin{equation}
    A^*_{lkj}(\mathbf{x},\mathbf{y}) = \frac{1}{8\pi(1 - \nu)r^2}\big[(1 - 2\nu)\{\delta_{lj}r_{,k} + \delta{kj}r_{,l} - \delta_{lk}r_{,j}\} + 3r_{,l}r_{,k}r_{,j}\big]
\end{equation}
\begin{equation}
\begin{split}
    H^*_{lkj}(\mathbf{x},\mathbf{y}) = \frac{\mu}{4\pi(1 \nu)r^3}\big[&3\frac{\partial r}{\partial \mathbf{n}}(\{1 - 2\nu\}\delta_{lk}r_{,j} + \nu(\delta_{kj}r_{,l} + \delta_{lj}r_{,k}) - 5r_{,l}r_{,k}r_{,j})\\
    &+ (1 - 2\nu)(3n_jr_{,l}r_{,k} + n_k\delta{lj} + n_l\delta_{kj})\\
    &+ 3\nu(n_lr_{,k}r_{,j} + n_kr_{,l}r_{,j})\\
    &- (1 - 4\nu)n_m\delta_{lk} \big]
\end{split}
\end{equation}

This equation is often called the traction boundary integral equation, because in contrast to equation (\ref{somigliana}), given the boundary conditions, we calculate the traction at any interior point instead of the displacement.

Just like the displacement boundary integral equation, we can treat cracks as two infinitesimally separated surfaces with a jump in displacement and balanced tractions. The resulting integral equation is:

\begin{equation}
    \begin{split}
        \sigma_{lk}(\mathbf{x}) = \int_{F} H^{*}_{lkj}(\mathbf{x},\mathbf{y}) \Delta u_j(\mathbf{y}) d\mathbf{y}~~~~~ \forall \mathbf{x} \in V
    \end{split}
    \label{tractionbie_crack}
\end{equation}

While the fault traction still does not appear within any surface integrals, the point $\mathbf{x}$ can be chosen to lie on the surface $F$, giving us the stress and traction on the fault surface. As a result, in contrast to the displacement boundary integral equation, this integral equation can be used to solve for fault stress and traction and will be a fundamental component of earthquake simulation methods.

### Displacement discontinuity method

We can solve Somigliana's identity as described above directly for displacement and traction is at least one or the other is known at any point on $S$. However, it's often easier to use a double layer potential approach to represent the solution in terms of an only-sometimes-meaningful vector density $\phi(\mathbf{y})$. Given $u$ for some subset $S_u$ and $t$ for some subset $S_t$, we solve:

\begin{equation}
u_{k}(\mathbf{x}) = -\int_{F} T^{*}_{kj}(\mathbf{x},\mathbf{y}) \phi(\mathbf{y}) d\mathbf{y}
\end{equation}
\begin{equation}
t_{k}(\mathbf{x}) = \int_{F} H^{*}_{lkj}(\mathbf{x},\mathbf{y}) n_l \phi(\mathbf{y}) d\mathbf{y}
\end{equation}

Given that these equations are exactly the ones that we derived above for a crack, $\phi$ matches exactly with the displacement discontinuity (aka "slip") $\Delta u$ when $S$ is a crack surface. For any other surface, we can interpret $\phi$ as a sort of fictitious slip or fictitious displacement discontinuity. Then, if we want to compute displacement or traction, we can compute the result using the above integrals of $\phi$. This approach is called the "Displacement discontinuity method". It is exactly analogous to solving a Poisson or Laplace equation problem using a double layer potential.

## Complex geometries, large models and high performance

There's a lot of data on the Earth. Put a cool 3D figure here.

### Fault roughness

Fault roughness is a particularly fascinating topic because it introduces a small scale geometrical nonlinearity into the model.

## Quasidynamic rupture modeling

###
TODO: Put a link to youtube video here

Quasidynamic earthquake simulation is a powerful tool for investigating the frictional behavior of faults over many earthquake cycles without having to invest the numerical resources required for fully dynamic rupture modeling \citep{rice1993spatio, liu2005aseismic, thomas2014quasi}.

The quasidynamic approximation is a first order approximation of inertial wave effects with a "radiation damping" term. The quasidynamic shear stress on the fault surface is
\begin{equation}
    \tau_{\mathrm{qd}} = \tau_{\mathrm{static}} - \frac{\mu}{2c_s} V
\end{equation}
where $\mu$ is the shear modulus, $c_s$ is the shear wave speed, and $V$ is the local fault slip velocity. The advantage of this approximation is that the shear stress can be calculated using static elastic numerical methods and then adjusted by the slip velocity.

To complete the system, we need to a friction law that relates shear stress to slip velocity. A common framework is rate-state friction where the strength of friction is related to both the rate of slip and a state variable the evolves during fault slip. The state variable evolution law can take various forms. Here, we present the aging law. Then, rate-state friction takes the form
\begin{align}
    \tau_{\mathrm{qd}} = a\sigma_n \sinh^{-1}\left(\frac{V}{2V_0}e^{\Psi / a}\right)\label{ratestate}\\
    \frac{d\Psi}{dt} = \frac{bV_0}{D_c}\left(e^{(f_0 - \Psi) / b} - \frac{V}{V_0}\right)\label{stateevolution}
\end{align}
where $\sigma_n$ is the normal stress, $\Psi$ is the state variable, $f_0$ is a the friction coefficient at a steady state slip velocity of $V_0$, $a$ and $b$ are dimensionless parameters determining the strength of velocity and state changes respectively on the evolution of friction and $D_c$ is the state evolution length scale. In our implementation, fault slip is always parallel to the shear stress vector and can be in any direction on the fault plane.

Several approaches have been used for quasidynamic earthquake cycle simulation. Fourier domain convolution methods are extremely efficient for planar faults with a uniform discretization \citep{rice1993spatio, lapusta2000elastodynamic}. However, these methods break down in the face of any non-planarity. Boundary element methods using rectangular or triangular dislocations \citep{liu2005aseismic, segall2012slow} have the aforementioned stress singularity issues, especially on nonplanar faults. Both Fourier and dislocation approaches struggle with rheologies beyond linear elasticity. In comparison, finite difference methods or finite element methods can successfully model a much wider range of rheologies including non-uniform material properties \citep{erickson2014efficient} or viscoplasticity \citep{allison2018earthquake}. However, finite difference methods still have difficulty with complex nonplanar fault geometries. In comparison, our boundary element methods can model arbitrary nonplanar fault geometries.

We build a quasidynamic earthquake cycle simulator on top of Tectosaur. We track the current slip deficit and state variable at every degree of freedom on the fault surface. Then, at each time step, we calculate the traction on the fault surface from the slip deficit field using Tectosaur to solve the static elastic equations. The friction equations are then solved for the current slip velocity using Equation (\ref{ratestate}). The state derivatives are calculated using Equation (\ref{stateevolution}). Finally, having both the slip deficit derivatives and state derivatives, we integrate in time. The algorithm is flexible to the particular time integration method. A popular method has been to use a time step dependent on the fastest slip velocity on the fault \citep{lapusta2009three}. We follow \citet{erickson2014efficient} in using an adaptive Runge-Kutta algorithm.

## Viscoelastic behavior and elasto-plastic behavior

The Earth is not purely elastic and there are many settings.

DEFINE A MAXWELL RHEOLOGY

DEFINE A SIMPLE PLASTIC RHEOLOGY

Change the fundamental material behavior seems like it would require a new set of basic equations, but given the mild perturbation away from elasticity, it's also possible to push the extra physics into the "body force" term.

## Seismic waves

## Fully dynamic rupture modeling

```python

```


```python
# ML_in_Finance-Kalman_Filters
# Version: 1.0 (28.4.2020)
# License: MIT
# Email: paul@thalesians.co.uk and matthew.dixon@iit.edu
# Notes: tested on Mac OS X running Python 3.6.9 with the following packages:
# scipy=1.4.1, numpy=1.18.1, matplotlib=3.1.3, pandas=1.0.3
# Citation: Please cite the following reference if this notebook is used for research purposes:
# Dixon M.F., Halperin I. and P. Bilokon, Machine Learning in Finance: From Theory to Practice, Springer Graduate Textbook Series, 2020.
```

```python
import numpy as np
import scipy.stats
import matplotlib.pyplot as plt
import pandas as pd
import pymc3 as pm
import theano.tensor as tt

import warnings
warnings.filterwarnings("ignore")

%matplotlib inline
```

# Overview
The purpose of this notebook is to demonstrate the application of particle filtering to stochastic volatility modeling. The insight gained from Chapter 2, namely a **sequential** or **iterative** application of Bayes's theorem, referred to as "sequential Bayesian updates", is the foundation of real-time **Bayesian filtering**. Kalman filtering is well known example of Bayesian filtering and we begin by reviewing it here.

## Kalman filtering

### Question: the autoregressive moving average $\text{ARMA}(p, q)$ model

The **autoregressive moving average $\text{ARMA}(p, q)$** model can be written as
$$y_t = \phi_1 y_{t-1} + \ldots + \phi_p y_{t-p} + \eta_t + \theta_1 \eta_{t-1} + \ldots + \theta_q \eta_{t-q},$$
where $\eta_t \sim \mathcal{N}(0, \sigma^2)$, and includes as special casses all $\text{AR}(p)$ and $\text{MA}(q)$ models. Such models are often fitted to financial time series. Suppose that we would like to filter this time series using a Kalman filter. Write down suitable process and observation models.

### Solution

Set $m := \max(p, q + 1)$, $\phi_i := 0$ for $i > p$, $\theta_i := 0$ for $i > q$. Then we obtain our process model as
$$\mathbf{X}_t = \mathbf{F}_t \mathbf{X}_{t-1} + \mathbf{a}_t + \mathbf{W}_t \mathbf{w}_t,$$
and the observation model as
$$\mathbf{Y}_t = \mathbf{H}_t \mathbf{X}_t + \mathbf{b}_t + \mathbf{V}_t \mathbf{v}_t,$$
where
\begin{equation*}
\mathbf{X}_t =
\begin{pmatrix}
y_t \\
\phi_2 y_{t-1} + \ldots + \phi_p y_{t-m+1} + \theta_1 \eta_t + \ldots + \theta_{m-1} \eta_{t-m+2} \\
\phi_3 y_{t-1} + \ldots + \phi_p y_{t-m+2} + \theta_2 \eta_t + \ldots + \theta_{m-1} \eta_{t-m+3} \\
\vdots \\
\phi_m y_{t-1} + \theta_{m-1} \eta_t
\end{pmatrix}
\in \mathbb{R}^{m \times 1},
\end{equation*}
\begin{equation*}
\mathbf{F} =
\begin{pmatrix}
\phi_1     & 1      & 0      & \cdots & 0      \\
\phi_2     & 0      & 1      &        & 0      \\
\vdots     & \vdots & \ddots & \ddots & \vdots \\
\phi_{m-1} & 0      & 0      &        & 1      \\
\phi_m     & 0      & 0      & \cdots & 0
\end{pmatrix}
\in \mathbb{R}^{m \times m},
\end{equation*}
\begin{equation*}
\mathbf{W} =
\begin{pmatrix}
1            &
\theta_1     &
\cdots       &
\theta_{m-1}
\end{pmatrix}^{\intercal}
\in \mathbb{R}^{m \times 1},
\end{equation*}
\begin{equation*}
w_t = \eta_t, \quad Q_t = \sigma^2, \quad
\mathbf{H} =
\begin{pmatrix}
1 & 0 & \ldots & 0
\end{pmatrix}
\in 1 \times m, \quad b_t = 0, \quad V_t = 0.
\end{equation*}

If $y_t$ is stationary, then $\mathbf{X}_t \sim \mathcal{N}(\mathbf{0}, \mathbf{P})$ with $\mathbf{P}$ given by the equation $\mathbf{P} = \mathbf{F} \mathbf{P} \mathbf{F}^{\intercal} + \sigma^2 \mathbf{W} \mathbf{W}^{\intercal}$, so we can set the initial state and error covariance to $\mathbf{0}$ and $\mathbf{P}$, respectively. For a detailed discussion of applying the Kalman filter in this particular case, see **[dJP00]**, **[ZW06]**. There are other ways to express $\text{ARMA}(p, q)$ as a state-space model and filter it. See **[BD87]**, **[BJR94]**, **[Ham94]**, **[Har89]**, and **[Pea80]** for details.

### Question: the Ornstein-Uhlenbeck process

Consider the one-dimensional Ornstein-Uhlenbeck (OU) process, the stationary Gauss-Markov process given by the SDE
$$dX_t = \theta(\mu - X_t) \, dt + \sigma \, dW_t,$$
where $X_t \in \mathbb{R}$, $X_0 = x_0$, and $\theta > 0$, $\mu$, and $\sigma > 0$ are constants. Formulate the Kalman process model for this process.

### Solution

The solution to the SDE is given by
$$X_t = x_0 e^{-\theta t} + \mu (1 - e^{-\theta t}) + \int_0^t \sigma e^{-\theta (t - u)} \, dW_u.$$

An Ito integral, $\int_s^t f(u) \, dW_u$, of a deterministic integrand, $f(u)$, is a Gaussian random variable with mean 0 and variance $\int_0^t f^2(u) \, du$. In our case, $f(u) = \sigma e^{-\theta (t - u)}$, and $\int_0^t f^2(u) \, du = \frac{\sigma^2}{2\theta} (1 - e^{-2\theta t})$.

Since this Markov process is homogeneous, its transition density depends only upon the time difference. Setting, for $s \leq t$, $h_k := t - s$ as the time interval between the time ticks $k - 1$ and $k$, we obtain a discretised process model
$$X_k = F_k X_{k-1} + a_k + w_k,$$
with $F_k = e^{-\theta h_k}$, $a_k = \mu(1 - e^{-\theta h_k})$, $w_k \sim \left(0, \frac{\sigma^2}{2 \theta}(1 - e^{-2\theta h_k})\right)$.

As a further exercise, consider extending this to a multivariate OU process.

## Particle filtering

Kalman filter maintains its state as moments of the multivatiate normal (Gaussian) distribution, $\mathcal{N}(\mathbf{m}, \mathbf{P})$. This works when the state is truly Gaussian, or when the true distribution can be closely approximated by the Gaussian. What if the distribution is, for example, bimodal?

```python
xs = np.linspace(-8., 12., 100)
plt.plot(xs, \
    .2 * scipy.stats.norm.pdf(xs, loc=-3., scale=np.sqrt(2.)) + \
    .8 * scipy.stats.norm.pdf(xs, loc=3., scale=np.sqrt(5.)));
```

Arguably the simplest way to approximate more or less any distribution, including bimodal, is by data points sampled from that distribution. We call those data points **particles**.

```python
particles = []
for i in range(5000):
    if scipy.stats.uniform.rvs() <= .2:
        particles.append(scipy.stats.norm.rvs(loc=-3., scale=np.sqrt(2.)))
    else:
        particles.append(scipy.stats.norm.rvs(loc=3., scale=np.sqrt(5.)))
```

```python
xs = np.linspace(-8., 12., 100)
plt.plot(xs, \
    .2 * scipy.stats.norm.pdf(xs, loc=-3., scale=np.sqrt(2.)) + \
    .8 * scipy.stats.norm.pdf(xs, loc=3., scale=np.sqrt(5.)))
plt.hist(particles, density=True, bins=100);
plt.plot(particles, [0 for p in particles], '+', markersize=10);
```

The more particles we have, the more closely can we approximate the target distribution. The approximate, empirical distribution is then given by the histogram. The particles don't have to be univariate, as in our example. They may be multivariate if we are approximating a multivariate distribution. Also, in our example the particles all have the same weight. More generally, we may consider weighted particles, whose weights are not all the same.

This gives rise to the family of algorithms known as **particle filtering** algorithms **[GSS93]**, **[Kit93]**. One of the most common of them is the **sequential importance resampling (SIR)** algorithm:

#### Sequential importance resampling (SIR)

1. **Initialisation step**: At time $t = 0$, draw $M$ i.i.d. samples (called **particles**) from the initial distribution $\tau_0$. Also, initialise $M$ normalised (to 1) weights to an identical value $\frac{1}{M}$. For $i = 1, 2, \ldots, M$, the samples will be denoted $\hat{\textbf{x}}_{0 \,|\, 0}^{(i)}$ and the normalised weights $\lambda_0^{(i)}$.
1. **Recursive step**: At time $t = 1, 2, 3, \ldots$, let $(\hat{\textbf{x}}_{t-1 \,|\, t-1}^{(i)})_{i=1,\ldots,M}$ be the particles generated at time $t - 1$.
 1. **Importance sampling**: For $i = 1, \ldots M$, sample $\hat{\mathbf{x}}_{t \,|\, t-1}^{(i)}$ from the Markov transition kernel $\tau_t(\cdot \,|\, \hat{\textbf{x}}_{t-1 \,|\, t-1}^{(i)})$. For $i = 1, \ldots, M$, use the observation density to compute the non-normalised weights
 $$\omega_t^{(i)} := \lambda_{t-1}^{(i)} \cdot p(\mathbf{y}_t \,|\, \hat{\mathbf{x}}_{t \,|\, t-1}^{(i)})$$
 and the values of the normalised weights before resampling ("br")
 $$^{\text{br}}\lambda_t^{(i)} := \frac{\omega_t^{(i)}}{\sum_{k=1}^M \omega_t^{(k)}}.$$
 1. **Resampling** (or **selection**): For $i = 1, \ldots, M$, use an appropriate resampling algorithm (such as **multinomial resampling** — see below) to sample $\mathbf{x}_{t \,|\, t}^{(i)}$ from the mixture
 $$\sum_{k=1}^M {^{\text{br}}} \lambda_t^{(k)} \delta(\mathbf{x}_t - \mathbf{x}_{t \,|\, t-1}^{(k)}),$$
 where $\delta(\cdot)$ denotes the Dirac delta generalised function, and set the normalised weights after resampling, $\lambda_t^{(i)}$, appropriately (for most common resampling algorithms this means $\lambda_t^{(i)} := \frac{1}{M}$). $\blacksquare$

In some sense, SIR resembles genetic algorithms. Based on the likelihoods $p(\mathbf{y}_t \,|\, \hat{\mathbf{x}}_{t \,|\, t-1}^{(i)})$, we increase the weights of the more "successful" particles, allowing them to "thrive" at the resampling step.

The resampling step was introduced to avoid the degeneration of the particles, with all the weight concentrating on a single point.

The most common resampling scheme is the so-called **multinomial resampling**:

#### Multinomial resampling

Notice that we are working with the *normalised* weights computed before reampling, ${^{\text{br}}}\lambda_t^{(1)}, {^{\text{br}}}\lambda_t^{(2)}, \ldots, {^{\text{br}}}\lambda_t^{(M)}$.

1. For $i = 1, 2, \ldots, M$, compute the cumulative sums
$${^{\text{br}}}\Lambda_t^{(i)} = \sum_{k=1}^i {^{\text{br}}}\lambda_t^{(k)},$$
so that, by construction, ${^{\text{br}}}\Lambda_t^{(M)} = 1$.
1. Generate $M$ random samples from $\mathcal{U}(0, 1)$, $u_1, u_2, \ldots, u_M$.
1. For each $i = 1, \ldots, M$, choose the particle $\hat{\mathbf{x}}_{t \,|\, t}^{(i)} = \hat{\mathbf{x}}_{t \,|\, t-1}^{(j)}$ with $j \in \{1, 2, \ldots, M - 1\}$ such that $u_i \in \left[{^{\text{br}}}\Lambda_t^{(j)}, {^{\text{br}}}\Lambda_t^{(j+1)}\right]$.

Thus we end up with $M$ new particles (**children**), $\mathbf{x}_{t \,|\, t}^{(1)}, \ldots, \mathbf{x}_{t \,|\, t}^{(M)}$ sampled from the existing set $\mathbf{x}_{t \,|\, t-1}^{(1)}, \ldots, \mathbf{x}_{t \,|\, t-1}^{(M)}$, so that some of the existing particles may disappear, while others may appear multiple times. For each $i = 1, \ldots, M$ the number of times $\mathbf{x}_{t \,|\, t-1}^{(i)}$ appears in the resampled set of particles is known as the particle's **replication factor**, $N_t^{(i)}$.

Set the normalised weights after resampling: $\lambda_t^{(i)} := \frac{1}{M}$.

We could view this algorithm as the sampling of the replication factors $N_t^{(1)}, \ldots N_t^{(M)}$ from the multinomial distribution with probabilities ${^{\text{br}}}\lambda_t^{(1)}, \ldots, {^{\text{br}}}\lambda_t^{(M)}$, respectively. Hence the name of the method. $\blacksquare$

## An application: stochastic volatility models

**Stochastic Volatility (SV)** models have been studied extensively in the literature, often as applications of **particle filtering** and **Markov chain Monte Carlo (MCMC)**.

In particular, Pitt, Malik, and Doucet apply the particle filter to the **stochastic volatility with leverage and jumps (SVLJ)** **[MP09]**, **[MP11a]**, **[MP11b]**, **[PMD14]**. The model has the general form of Taylor's **[Tay82]** with two modifications. For $t \in \mathbb{N}^0$, let $y_t$ denote the log-return on an asset and $x_t$ denote the log-variance of that return. Then
\begin{align}
y_t &= \epsilon_t e^{x_t/2} + J_t \varpi_t, \label{eq:svlj-log-return} \\
x_{t+1} &= \mu (1 - \phi) + \phi x_t + \sigma_v \eta_t, \label{eq:svlj-log-variance}
\end{align}
where $\mu$ is the mean log-variance, $\phi$ the persistence parameter, $\sigma_v$ the volatility of log-variance.

The first modification to Taylor's model is the introduction of correlation between $\epsilon_t$ and $\eta_t$:
\begin{equation*}
\left(
 \begin{array}{c}
   \epsilon_t \\
   \eta_t \\
 \end{array}
\right)
\sim \mathcal{N}(0, \Sigma),
\quad
\Sigma =
\left(
  \begin{array}{cc}
    1 & \rho \\
    \rho & 1 \\
  \end{array}
\right).
\end{equation*}

The correlation $\rho$ is the **leverage** parameter. In general, $\rho < 0$, due to the stylised fact known as the **leverage effect** **[Bla76]**.

The second change is the introduction of jumps. $J_t \in \{0, 1\}$ is a Bernoulli counter with intensity $p$ (thus $p$ is the jump intensity parameter), $\varpi_t \sim \mathcal{N}(0, \sigma_J^2)$ determines the jump size (thus $\sigma_J$ is the jump volatility parameter).

We obtain a **stochastic volatility with leverage (SVL)**, but no jumps, if we delete the $J_t \varpi_t$ term or, equivalently, set $p$ to zero. Taylor's original model is a special case of SVLJ with $p = 0$, $\rho = 0$.

### Particle filter for SVLJ

This, then, leads to the following adaptation of SIR, developed by Doucet, Malik, and Pitt, for this special case with nonadditive, correlated noises. The initial distribution of $x_0$ is taken to be $\mathcal{N}\left(0, \sigma_v^2 / (1 - \phi^2)\right)$.
1. **Initialisation step**: At time $t = 0$, draw $M$ i.i.d. particles from the initial distribution $\mathcal{N}(0, \sigma_v^2 / (1 - \phi^2))$. Also, initialise $M$ normalised (to 1) weights to an identical value of $\frac{1}{M}$. For $i = 1, 2, \ldots, M$, the samples will be denoted $\hat{x}^{(i)}_{0 \ | \  0}$ and the normalised weights $\lambda^{(i)}_0$.
1. **Recursive step**: At time $t \in \mathbb{N}^*$, let $(\hat{x}^{(i)}_{t-1 \ | \  t-1})_{i = 1, \ldots, M}$ be the particles generated at time $t - 1$.
    1. **Importance sampling**:
        - First,
            - For $i = 1, \ldots, M$, sample $\hat{\epsilon}^{(i)}_{t-1}$ from $p(\epsilon_{t-1} \ | \  x_{t-1} = \hat{x}^{(i)}_{t-1 \ | \  t-1}, y_{t-1})$. (If no $y_{t-1}$ is available, as at $t = 1$, sample from $p(\epsilon_{t-1} \ | \  x_{t-1} = \hat{x}^{(i)}_{t-1 \ | \  t-1})$).
            - For $i = 1, \ldots, M$, sample $\hat{x}^{(i)}_{t \ | \  t-1}$ from $p(x_t \ | \  x_{t-1} = \hat{x}^{(i)}_{t-1 \ | \  t-1}, y_{t-1}, \hat{\epsilon}^{(i)}_{t-1})$.
        - For $i = 1, \ldots, M$, compute the non-normalised weights
            \begin{equation}
            \omega^{(i)}_t := \lambda^{(i)}_{t-1} \cdot p_{\gamma_t}(y_{t} \ | \  \hat{x}^{(i)}_{t \ | \  t-1}),
            \end{equation}
            using the observation density
            \begin{gather*}
            p(y_t \ | \  \hat{x}^{(i)}_{t \ | \  t-1}, p, \sigma_J^2) = (1 - p) \left[ \left( 2\pi e^{\hat{x}^{(i)}_{t \ | \  t-1}} \right)^{-1/2} \exp\left(-y_t^2 / (2 e^{\hat{x}^{(i)}_{t \ | \  t-1}})\right) \right] + \\
            p \left[ \left( 2\pi ( e^{\hat{x}^{(i)}_{t \ | \  t-1}} + \sigma_J^2 ) \right)^{-1/2} \exp\left(-y_t^2 / (2 e^{\hat{x}^{(i)}_{t \ | \  t-1}} + 2\sigma_J^2)\right) \right],
            \end{gather*}
            and the values of the normalised weights before resampling (`br')
            \begin{equation*}
            {^\text{br}\lambda^{(i)}_t} := \frac{\omega^{(i)}_t}{\sum_{k=1}^M \omega^{(k)}_t}.
            \end{equation*}
    1. **Resampling** (or **selection**): For $i = 1, \ldots, M$, use an appropriate resampling algorithm (such as multinomial resampling) sample $\hat{x}^{(i)}_{t \ | \  t}$ from the mixture
        \begin{equation*}
        \sum_{k=1}^M {^\text{br}\lambda^{(k)}_t} \delta(x_t - \hat{x}^{(k)}_{t \ | \  t-1}),
        \end{equation*}
        where $\delta(\cdot)$ denotes the Dirac delta generalised function, and set the normalised weights after resampling, $\lambda^{(i)}_t$, according to the resampling algorithm.

### Calibrating stochastic filters: the frequentist approach

We have seen in the example of the stochastic volatility model with leverage and jumps (SVLJ) that the state-space model may be parameterised by a parameter vector, $\mathbf{\theta} \in \mathbb{R}^{d_{\theta}}$, $d_{\theta} \in \mathbb{N}^*$. In that particular case,
\begin{equation*}
\mathbf{\theta} = \left(
           \begin{array}{c}
             \mu \\
             \phi \\
             \sigma_{\eta}^2 \\
             \rho \\
             \sigma_J^2 \\
             p \\
           \end{array}
         \right).
\end{equation*}

We may not know the true value of this parameter. How do we estimate it? In other words, how do we **calibrate** the model, given a time series of either historical or generated observations, $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_T$, $T \in \mathbb{T} = \mathbb{N}^*$.

The frequentist approach relies on the (joint) probability density function of the observations, which depends on the parameters, $p(\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_T \,|\, \mathbf{\theta})$. We can regard this as a function of $\mathbf{\theta}$ with $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_T$ fixed, $p(\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_T \,|\, \mathbf{\theta}) =: \mathcal{L}(\mathbf{\theta})$ — the **likelihood function**.

It is sometimes referred to as **marginal** likelihood, since the hidden states, $\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_T$, are margined out. We seek a **maximum likelihood estimator (MLE)**, $\hat{\mathbf{\theta}}_{ML}$, the value of $\mathbf{\theta}$ that maximises the likelihood function.

Each evaluation of the objective function, $\mathcal{L}(\mathbf{\theta})$, constitutes a run the stochastic filter over the observations $\mathbf{y}_1, \mathbf{y}_2, \ldots, \mathbf{y}_T$. By the chain rule (i), and since we are dealing with a Markov chain (ii),
\begin{equation*}
p(\mathbf{y}_1, \ldots, \mathbf{y}_T) \overset{\text{(i)}}{=} \prod_{t=1}^T p(\mathbf{y}_t \ | \  \mathbf{y}_0, \ldots, \mathbf{y}_{t-1}) \overset{\text{(ii)}}{=} \prod_{t=1}^T \int p(\mathbf{y}_t \ | \  \mathbf{x}_t) p(\mathbf{x}_t \ | \  \mathbf{y}_0, \ldots, \mathbf{y}_{t-1}) \, d\mathbf{x}_t.
\end{equation*}
Here we have omitted the dependence of all the probability densities on $\mathbf{\theta}$, e.g. we should have really written $p(\mathbf{y}_1, \ldots, \mathbf{y}_T; \mathbf{\theta})$.

For the particle filter, we can estimate the log-likelihood function from the non-normalised weights:
\begin{equation*}
p(\mathbf{y}_1, \ldots, \mathbf{y}_T) = \prod_{t=1}^T \int p(\mathbf{y}_t \ | \  \mathbf{x}_t) p(\mathbf{x}_t \ | \  \mathbf{y}_0, \ldots, \mathbf{y}_{t-1}) \, d\mathbf{x}_t \approx \prod_{t=1}^T \left( \frac{1}{M} \sum_{k=1}^M \omega^{(k)}_t \right),
\end{equation*}
whence
\begin{equation}
\ln(\mathcal{L}(\mathbf{\theta})) = \ln \left\{ \prod_{t=1}^T \left( \frac{1}{M} \sum_{k=1}^M \omega^{(k)}_t \right) \right\} = \sum_{t=1}^T \ln\left( \frac{1}{M} \sum_{k=1}^M \omega^{(k)}_t \right).
\label{eq:particle-filter-log-likelihood}
\end{equation}

This was first proposed by Kitagawa **[Kit93]**, **[Kit96]** for the purposes of approximating $\hat{\mathbf{\theta}}_{ML}$.

In most practical applications one needs to resort to numerical methods, perhaps quasi-Newton methods, such as **Broyden-Fletcher-Goldfarb-Shanno (BFGS)** **[GMW82]**, to find $\hat{\mathbf{\theta}}_{ML}$.

Malik, Pitt, and Doucet **[PMD14]** point out the practical difficulties which result when using the above as an objective function in an optimiser. In the resampling (or selection) step of the particle filter, we are sampling from a discontinuous empirical distribution function. Therefore $\ln(\mathcal{L}(\mathbf{\theta}))$ will not be continuous as a function of $\mathbf{\theta}$. To remedy this, they rely on an alternative, continuous, resampling procedure described in. A quasi-Newton method is then used to find $\hat{\mathbf{\theta}}_{ML}$ for the parameters $\mathbf{\theta} = (\mu, \phi, \sigma_v^2, \rho, p, \sigma_J^2)^{\intercal}$ of the SVLJ model.

Kalman filters can also be calibrated using a similar maximum likelihood approach.

### Calibrating stochastic filters: the Bayesian approach

Let us briefly discuss how filtering methods relate to **Markov chain Monte Carlo methods (MCMC)** --- a vast subject in its own right, therefore our discussion will be cursory at best. The technique takes its origin from **[MRRT53]**.

Proceeding along the lines of **[KSC98]**, Meyer and Yu **[MY00]**, **[Yu05]** demonstrate how MCMC techniques can be used to estimate the parameters of the SVL model. They calibrate the parameters to the time series of observations of daily mean-adjusted log-returns, $y_1, \ldots, y_T$. They obtain the joint prior density
\begin{equation*}
p(\mathbf{\theta}, x_0, \ldots, x_T) = p(\mathbf{\theta}) p(x_0 \ | \  \mathbf{\theta}) \prod_{t=1}^T p(x_t \ | \  x_{t-1}, \mathbf{\theta})
\end{equation*}
by successive conditioning. Here $\mathbf{\theta} = (\mu, \phi, \sigma_v^2, \rho)^{\intercal}$ is, as before, the vector of the model's parameters. They assume prior independence of the parameters and choose the same priors as in **[KSC98]** for $\mu$, $\phi$, and $\sigma_v^2$, and a uniform prior for $\rho$. The observation model and the conditional independence assumption give the likelihood
\begin{equation*}
p(y_1, \ldots, y_T \ | \  \mathbf{\theta}, x_0, \ldots, x_T) = \prod_{t=1}^T p(y_t \ | \  x_t),
\end{equation*}
and the joint posterior distribution of the **unobservables** (the parameters $\mathbf{\theta}$ and the hidden states $x_0, \ldots, x_T$; in the Bayesian perspective these are treated identically and estimated in a similar manner) follows from Bayes's theorem; for the SVL model, this posterior satisfies
\begin{equation*}
p(\mathbf{\theta}, x_0, \ldots, x_T \ | \  y_1, \ldots, y_T) \propto p(\mu) p(\phi) p(\sigma_v^2) p(\rho) \prod_{t=1}^T p(x_{t+1} \ | \  x_t, \mu, \phi, \sigma_v^2) \prod_{t=1}^T p(y_t \ | \  x_{t+1}, x_t, \mu, \phi, \sigma_v^2, \rho),
\end{equation*}
where $p(\mu)$, $p(\phi)$, $p(\sigma_v^2)$, $p(\rho)$ are the appropriately chosen priors,
\begin{align*}
x_{t+1} \ | \  x_t, \mu, \phi, \sigma_v^2 &\sim \mathcal{N}\left(\mu (1 - \phi) + \phi x_t, \sigma_v^2\right), \\
y_t \ | \  x_{t+1}, x_t, \mu, \phi, \sigma_v^2, \rho &\sim \mathcal{N}\left(\frac{\rho}{\sigma_v} e^{x_t/2} \left( x_{t+1} - \mu (1 - \phi) - \phi x_t \right), e^{x_t} (1 - \rho^2)\right).
\end{align*}

Meyer and Yu use the software package BUGS (an acronym for Bayesian inference Using Gibbs Sampling) **[SH96], [Lun00]** to represent the resulting Bayesian model as a **directed acyclic graph (DAG)**, where the nodes are either constants (denoted by rectangles), stochastic nodes (variables that are given a distribution, denoted by ellipses), or deterministic nodes (logical functions of other nodes); the arrows either indicate stochastic dependence (solid arrows) or logical functions (hollow arrows). This graph helps visualise the conditional (in)dependence assumptions and is used by BUGS to construct full univariate conditional posterior distributions for all unobservables. It then uses Markov chain Monte Carlo algorithms to sample from these distributions.

The algorithm based on the original work  **[Met53]** is now known as the **Metropolis algorithm**. It has been generalised by Hastings (1930-2016) to obtain the **Metropolis-Hastings algorithm** **[Has70]** 1970} and further by Green to obtain what is known as the **Metropolis-Hastings-Green algorithm** **[Gre95]**. A popular algorithm based on a special case of the Metropolis-Hastings algorithm, known as the **Gibbs sampler**, was developed by the brothers Geman **[Gem84]** and, independently, Tanner and Wong **[TW87]** Sometimes the Gibbs sampler is referred to as **data augmentation** following this paper. It was further popularised by Gelfand and Smith **[GS90]**. Gibbs sampling and related algorithms **[Gil92], [Rit92]** are used by BUGS to sample from the univariate conditional posterior distributions for all unobservables.

As a result we perform Bayesian estimation --- obtain estimates of the **distributions** of the parameters $\mu$, $\phi$, $\sigma_v^2$, $\rho$ --- rather than frequentist estimation, where a single value of the parameters vector, which maximises the likelihood, $\hat{\mathbf{\theta}}_{ML}$, is produced. Stochastic filtering, sometimes in combination with MCMC, can be used for both frequentist and Bayesian parameter estimation **[Che03]**. Filtering methods that update estimates of the parameters online, while processing observations in real-time, are referred to as **adaptive filtering**. **[Say08]**, **[Rey13]**, **[Cri13]**, **[Nae15]**

```python
df1 = pd.read_csv('../data/dataset-1.csv')
y1 = df1['daily_log_return'].values
```

```python
df2 = pd.read_csv('../data/dataset-2.csv')
y2 = df2['daily_log_return'].values
```

To use Dataset 1:

```python
# log_returns = y1
```

To use Dataset 2:

```python
log_returns = y2
```

#### Implementation of the stochastic volatility model with leverage in PyMC3

```python
class StochasticVolatilityProcess(pm.distributions.distribution.Continuous):
    def __init__(self, mu=None, phi=None, sigmav=None,
                 *args, **kwargs):
        super(StochasticVolatilityProcess, self).__init__(*args, **kwargs)
        self.alpha = pm.Deterministic('alpha', (1. - phi) * mu)
        self.phi = phi
        self.sigmav = sigmav
        self.init = pm.Normal.dist(mu=mu, sd=sigmav)
        self.mean = tt.as_tensor_variable(0.)

    def logp(self, x):
        alpha = self.alpha
        phi = self.phi
        sigmav = self.sigmav
        init = self.init

        x_im1 = x[:-1]
        x_i = x[1:]

        innov_like = pm.Normal.dist(mu=alpha + phi * x_im1, sd=sigmav).logp(x_i)
        return init.logp(x[0]) + tt.sum(innov_like)

    def _repr_latex_(self, name=None, dist=None):
        if dist is None:
            dist = self
        mu = dist.mu
        sd = dist.sd
        name = r'\text{%s}' % name
        return r'${} \sim \text{{StochasticVolatilityProcess}}(\mathit{{mu}}={},~\mathit{{sd}}={})$'.format(name,
                get_variable_name(mu), get_variable_name(sd))
```

```python
def make_stochastic_volatility_model(model, log_returns):
    mu = pm.Normal(name='mu', mu=0., sd=np.sqrt(25.))
    phistar = pm.Beta(name='phistar', alpha=20., beta=1.5)
    recsigmav2 = pm.Gamma(name='recsigmav2', alpha=2.5, beta=.25)
    beta = pm.math.exp(pm.Deterministic('beta', .5 * mu))
    phi = pm.Deterministic('phi', 2. * phistar - 1.)
    sigmav = pm.Deterministic('sigmav', pm.math.sqrt(1. / recsigmav2))
    x = StochasticVolatilityProcess('x', mu, phi, sigmav, shape=len(log_returns) + 1)
    y = pm.Normal(name='y', mu=0., sd=pm.math.sqrt(pm.math.exp(x[1:])), observed=log_returns)
```

```python
def make_stochastic_volatility_model_with_leverage(model, log_returns):
    mu = pm.Normal(name='mu', mu=0., sd=np.sqrt(25.))
    phistar = pm.Beta(name='phistar', alpha=20., beta=1.5)
    recsigmav2 = pm.Gamma(name='recsigmav2', alpha=2.5, beta=.25)
    beta = pm.math.exp(pm.Deterministic('beta', .5 * mu))
    phi = pm.Deterministic('phi', 2. * phistar - 1.)
    sigmav = pm.Deterministic('sigmav', pm.math.sqrt(1. / recsigmav2))
    rho = pm.Uniform(name='rho', lower=-1., upper=1.)
    x = StochasticVolatilityProcess('x', mu, phi, sigmav, shape=len(log_returns) + 2)
    y = pm.Normal(name='y',
                  mu=rho / sigmav * pm.math.exp(.5 * x[1:-1]) * (x[2:] - (1. - phi) * mu - phi * x[1:-1]),
                  sd=pm.math.sqrt(pm.math.exp(x[1:-1]) * (1 - rho * rho)),
                  observed=log_returns)
```

#### Playing with the model

Let's apply the stochastic volatility model, no leverage:

```python
with pm.Model() as model:
    print('Making model...')
    make_stochastic_volatility_model(model, log_returns)
    print('Sampling...')
    trace = pm.sample(tune=10, draws=100, chains=1)
    print('Producing trace plot...')
    pm.traceplot(trace);
```

Let's apply the stochastic volatility model with leverage:

```python
with pm.Model() as model:
    print('Making model...')
    make_stochastic_volatility_model_with_leverage(model, log_returns)
    print('Sampling...')
    trace = pm.sample(tune=1000, draws=10000, nuts_kwargs=dict(target_accept=.9), chains=1)
    print('Producing trace plot...')
    pm.traceplot(trace);
```

## Further reading

For more information on filtering we refer the reader to **[Hay01]**, **[May79]**, **[Sim06]**, **[BC09]**, **[Sar13]**.

For more information on MCMC, the reader may wish to consult **[Gey11]**.

## Bibliography

**[BC09]** Alan Bain and Dan Crisan. *Fundamentals of Stochastic Filtering (Stochastic Modelling and Applied Probability Book 60).* Springer, 2009.

**[BD87]** Peter J. Brockwell and Richard A. Davis. *Time Series: Theory and Methods.* Springer-Verlag, 1987.

**[Bil16]** Paul Bilokon. Bayesian methods for solving estimation and forecasting problems in the high-frequency trading environment. MSc thesis, University of Oxford, 2016.

**[BJR94]** George E. P. Box, Gwilym M. Jenkins, and Gregory C. Reinsel. *Time Series Analysis.* Prentice-Hall, 1994.

**[Bla76]** Fisher Black, 1976. Studies of Stock Price Volatility Changes. *Proceedings of the Business and Economics Section of the American Statistical Association*, 177-181.

**[dJP00]** Piet de Jong and Jeremy Penzer. The ARIMA model in state space form. Research Report 40, Department of Statistics, London School of Economics, Houghton Street, London, WC2A 2AE, UK, August 2000. http://www.lse.ac.uk/statistics/documents/researchreport40.pdf.

**[DK00]** James Durbin and Siem Jan Koopman. Time series analysis of non-Gaussian observations based on state space models from both classical and Bayesian perspectives (with discussion). *Journal of the Royal Statistical Society, Series B*, 62:3-56, 2000.

**[Gey11]** Charles J. Geyer. *Handbook of Markov Chain Monte Carlo*, chapter Introduction to Markov Chain Monte Carlo, pages 3-49. Handbooks of Modern Statistical Methods. CRC Press, 2011.

**[GG84]** Stuart Jay Geman and Donald Geman. Stochastic relaxation, Gibbs distributions, and the Bayesian restoration of images. *IEEE Transactions on Pattern Analysis and Machine Intelligence*, 6:721-741, 1984.

**[GMW82]** Philip E. Gill, Walter Murray and Margaret H. Wright. *Practical Optimization*. Academic Press, 1982.

**[GSS93]** N.J. Gordon, D.J. Salmond, and A.F.M. Smith. Novel approach to nonlinear/non-Gaussian Bayesian state estimation. In *IEE Proceedings F (Radar and Signal Processing)*, 1993.

**[Ham94]** James Douglas Hamilton. *Time Series Analysis.* Princeton University Press, 1994.

**[Har89]** Andrew C. Harvey. *Forecasting, structural time series models and the Kalman filter.* Cambridge
University Press, 1989.

**[Hay01]** Simon Haykin, editor. *Kalman Filtering and Neural Networks.* John Wiley and Sons, Inc., 2001.

**[HRS94]** Andrew C. Harvey, Esther Ruiz, and Neil Shephard. Multivariate stochastic variance models. *Review of Economic Studies*, 61:247-264, 1994.

**[Kal60]** Rudolf Emil Kalman. A new approach to linear filtering and prediction problems. *Transactions of the ASME - Journal of Basic Engineering, Series D*, 82:35-45, 1960.

**[Kit93]** Genshiro Kitagawa. A Monte Carlo filtering and smoothing method for non-Gaussian nonlinear state space models. In *Proceedings of the 2nd U.S.-Japan Joint Seminar on Statistical Time Series Analysis*, pages 110-131, 1993.

**[Kit96]** Genshiro Kitagawa. Monte Carlo filter and smoother for non-Gaussian nonlinear state space models. *Journal of Computational and Graphical Statistics*, 5(1):1-25, 1996.

**[KSC98]** Sangjoon Kim, Neil Shephard, and Siddhartha Chib. Stochastic volatility: Likelihood inference and comparison with ARCH models. *The Review of Economic Studies*, 65(3):361-393, July 1998.

**[May79]** Peter S. Maybeck. *Stochastic Models, Estimation and Control, Volume 1.* Academic Press, 1979.

**[MP09]** Sheheryar Malik and Michael K. Pitt. Modelling stochastic volatility with leverage and jumps: A simulated maximum likelihood approach via particle filtering. Warwick Economic Research Papers 897, The University of Warwick, Department of Economics, Coventry CV4 7AL, April 2009.

**[MP11a]** Sheheryar Malik and Michael K. Pitt. Modelling stochastic volatility with leverage and jumps: A simulated maximum likelihood approach via particle filtering. document de travail 318, Banque de France Eurosysteme, February 2011.

**[MP11b]** Sheheryar Malik and Michael K. Pitt. Particle filters for continuous likelihood evaluation and maximisation. *Journal of Econometrics*, 165:190–209, 2011.

**[MRRT53]** Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, Augusta H. Teller, and Edward Teller. Equation of state calculations by fast computing machines. *Journal of Chemical Physics*, 21, 1953.

**[MY00]** Renate Meyer and Jun Yu. BUGS for a Bayesian analysis of stochastic volatility models. *Econometrics Journal*, 3:198-215, 2000.

**[Pea80]** J. G. Pearlman. An algorithm for the exact likelihood of a high-order autoregressive-moving average
process. *Biometrika*, 67(1):232–233, 1980.

**[PMD14]** Michael K. Pitt, Sheheryar Malik, and Adnaud Doucet. Simulated likelhood inference for stochastic volatility models using continuous particle filtering. *Annals of the Institute of Statistical Mathematics*, 66:527–552, 2014.

**[Sar13]** Simo Sarkka. *Bayesian filtering and smoothing (Institute of Mathematical Statistics Textbooks).* Cambridge University Press, 2013.

**[Sim06]** Dan Simon. *Optimal State Estimation: Kalman, H-infinity, and Nonlinear approaches.* Wiley, 2006.

**[SP97]** Neil Shephard and Michael K. Pitt. Likelihood analysis of non-Gaussian measurement time series. *Biometrika*, 84:653-667, 1997.

**[Tay82]** Stephen J. Taylor. *Time Series Analysis: Theory and Practice*, chapter Financial returns modelled by
the product of two stochastic processes, a study of daily sugar prices, pages 203–226. North-Holland,
1982.

**[Yu05]** Jun Yu. On leverage in a statistical volatility model. *Journal of Econometrics*, 127:165-178, 2005.

**[ZW06]** Eric Zivot and Jiahui Wang. *Modeling Financial Time Series with S-PLUS.* Springer, 2006.

**[Met53]** Equation of State Calculations by Fast Computing Machines, Nicholas Metropolis, Arianna W. Rosenbluth, Marshall N. Rosenbluth, and Augusta H. Teller, J. Chem. Phys. 21, 1087 (1953)

**[Has70]** W.K. Hastings, Monte Carlo sampling methods using Markov chains and their applications, Biometrika, Volume 57, Issue 1, April 1970, Pages 97–109

**[Gre95]** Green, Reversible jump Markov chain Monte Carlo computation and Bayesian model determination, Biometrika, Volume 82, Issue 4, December 1995, Pages 711–732

**[Gem84]**  S. Geman and D. Geman, "Stochastic Relaxation, Gibbs Distributions, and the Bayesian Restoration of Images," in IEEE Transactions on Pattern Analysis and Machine Intelligence, vol. PAMI-6, no. 6, pp. 721-741, Nov. 1984

**[TW87]** Tanner, Martin A., and Wing Hung Wong. “The Calculation of Posterior Distributions by Data Augmentation.” Journal of the American Statistical Association, vol. 82, no. 398, 1987, pp. 528–540.

**[Gil92]** Gilks, W.R. and Wild, P. (1992), Adaptive Rejection Sampling for Gibbs Sampling. Journal of the Royal Statistical Society: Series C (Applied Statistics), 41: 337-348

**[Rit92]** Chopra, Navin, Lakonishok, Josef and Ritter, Jay, (1992), Measuring abnormal performance: Do stocks overreact?, Journal of Financial Economics, 31, issue 2, p. 235-268

**[GS90]** Gelfand, Alan E., and Adrian F. M. Smith. “Sampling-Based Approaches to Calculating Marginal Densities.” Journal of the American Statistical Association, vol. 85, no. 410, 1990, pp. 398–409

**[SH96]** Spiegelhalter, D.J., Best, N.G., Carlin, B.P. and Van Der Linde, A. (2002), Bayesian measures of model complexity and fit. Journal of the Royal Statistical Society: Series B (Statistical Methodology), 64: 583-639.,

```python

```


#  CS109B Data Science 2: Advanced Topics in Data Science

## Lab 4 - Bayesian Analysis

**Harvard University**

**Spring 2020**

**Instructors:** Mark Glickman, Pavlos Protopapas, and Chris Tanner

**Lab Instructors:** Chris Tanner and Eleni Angelaki Kaxiras

**Content:** Eleni Angelaki Kaxiras

---

```python
## RUN THIS CELL TO PROPERLY HIGHLIGHT THE EXERCISES
import requests
from IPython.core.display import HTML
styles = requests.get("https://raw.githubusercontent.com/Harvard-IACS/2019-CS109B/master/content/styles/cs109.css").text
HTML(styles)
```

```python
import pymc3 as pm
from pymc3 import summary
```

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as stats
import pandas as pd
%matplotlib inline

import warnings
warnings.filterwarnings('ignore')
```

```python
print('Running on PyMC3 v{}'.format(pm.__version__))
```

    Running on PyMC3 v3.8

```javascript
%%javascript
IPython.OutputArea.auto_scroll_threshold = 20000;
```

    <IPython.core.display.Javascript object>

## Learning Objectives

By the end of this lab, you should be able to:
* Understand how probability distributions work.
* Apply Bayes Rule in calculating probabilities.
* Understand how to apply Bayesian analysis using PyMC3
* Avoid getting fired when talking to your Bayesian employer.

**This lab corresponds to Lectures 6, 7, and 8, and maps to Homework 3.**

## Table of Contents

1. The Bayesian Way of Thinking or Is this a Fair Coin?
2. [Intro to `pyMC3`](#pymc3).
3. [Bayesian Linear Regression](#blr).
4. [Try this at Home: Example on Mining Disasters](#no4).

## 1. The Bayesian way of Thinking

```
Here is my state of knowledge about the situation. Here is some data, I am now going to revise my state of knowledge.
```

Table Exercise: Discuss the statement above with your table mates and make sure everyone understands what it means and what constitutes Bayesian way of thinking. Finally, count the Bayesians among you.

### A. Bayes Rule

\begin{equation}
\label{eq:bayes}
P(A|\textbf{B}) = \frac{P(\textbf{B} |A) P(A) }{P(\textbf{B})}
\end{equation}

$P(A|\textbf{B})$ is the **posterior** distribution, prob(hypothesis | data)

$P(\textbf{B} |A)$ is the **likelihood** function, how probable is my data **B** for different values of the parameters

$P(A)$ is the marginal probability to observe the data, called the **prior**, this captures our belief about the data before observing it.

$P(\textbf{B})$ is the marginal distribution (sometimes called marginal likelihood)

Table Exercise: Solve the Monty Hall Paradox using Bayes Rule.

You are invited to play a game. There are 3 doors behind **one** of which are the keys to a brand new red Tesla. There is a goat behind each of the other two.

You are asked to pick one door, and let's say you pick **Door1**. The host knows where the keys are. Of the two remaining closed doors, he will always open the door that has a goat behind it. He'll say "I will do you a favor and open **Door2**". So he opens Door2 inside which there is, of course, a goat. He now asks you, do you want to open the initial Door you chose or change to **Door3**? Generally, in this game, when you are presented with this choice should you swap the doors?

**Initial Steps:**
- Start by defining the `events` of this probabilities game. One definition is:

    - $A_i$: car is behind door $i$

    - $B_i$ host opens door $i$

$i\in[1,2,3]$

- In more math terms, the question is: is the probability that the price is behind **Door 1** higher than the probability that the price is behind **Door2**, given that an event **has occured**?

### B. Bayes Rule written with Probability Distributions

We have data that we believe come from an underlying distribution of unknown parameters. If we find those parameters, we know everything about the process that generated this data and we can make inferences (create new data).

\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

#### But what is $\theta \;$?

$\theta$ is an unknown yet fixed set of parameters. In Bayesian inference we express our belief about what $\theta$ might be and instead of trying to guess $\theta$ exactly, we look for its **probability distribution**. What that means is that we are looking for the **parameters** of that distribution. For example, for a Poisson distribution our $\theta$ is only $\lambda$. In a normal distribution, our $\theta$ is often just $\mu$ and $\sigma$.

### C. A review of Common Probability Distributions

#### Discrete Distributions

The random variable has a **probability mass function (pmf)** which measures the probability that our random variable will take a specific value $y$, denoted $P(Y=y)$.

- **Bernoulli** (binary outcome, success has probability $\theta$, $one$ trial):
$
P(Y=k) =  \theta^k(1-\theta)^{1-k}
$

- **Binomial** (binary outcome, success has probability $\theta$, $n$ trials):
\begin{equation}
P(Y=k) =  {{n}\choose{k}} \cdot \theta^k(1-\theta)^{n-k}
\end{equation}

*Note*: Binomial(1,$p$) = Bernouli($p$)

- **Negative Binomial**

- **Poisson** (counts independent events occurring at a rate)
\begin{equation}
P\left( Y=y|\lambda \right) = \frac{{e^{ - \lambda } \lambda ^y }}{{y!}}
\end{equation}
y = 0,1,2,...

- **Discrete Uniform**

- **Categorical, or Multinulli** (random variables can take any of K possible categories, each having its own probability; this is a generalization of the Bernoulli distribution for a discrete variable with more than two possible outcomes, such as the roll of a die)

- **Dirichlet-multinomial** (a generalization of the beta distribution for many variables)

#### Continuous Distributions

The random variable has a **probability density function (pdf)**.
- **Uniform** (variable equally likely to be near each value in interval $(a,b)$)
\begin{equation}
P(X = x) = \frac{1}{b - a}
\end{equation}
anywhere within the interval $(a, b)$, and zero elsewhere.

- **Normal** (a.k.a. Gaussian)
\begin{equation}
X \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

    A Normal distribution can be parameterized either in terms of precision $\tau$ or standard deviation ($\sigma^{2}$. The link between the two is given by
\begin{equation}
\tau = \frac{1}{\sigma^{2}}
\end{equation}
 - Mean $\mu$
 - Variance $\frac{1}{\tau}$ or $\sigma^{2}$
 - Parameters: `mu: float`, `sigma: float` or `tau: float`

- **Beta** (variable ($\theta$) taking on values in the interval $[0,1]$, and parametrized by two positive parameters, $\alpha$ and $\beta$ that control the shape of the distribution.

*Note:*Beta is a good distribution to use for priors (beliefs) because its range is $[0,1]$ which is the natural range for a probability and because we can model a wide range of functions by changing the $\alpha$ and $\beta$ parameters.

\begin{equation}
\label{eq:beta}
P(\theta) = \frac{1}{B(\alpha, \beta)} {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1} \propto {\theta}^{\alpha - 1} (1 - \theta)^{\beta - 1}
\end{equation}

where the normalisation constant, $B$, is a beta function of $\alpha$ and $\beta$,

\begin{equation}
B(\alpha, \beta) = \int_{t=0}^1 t^{\alpha - 1} (1 - t)^{\beta - 1} dt.
\end{equation}

- **Exponential**

- **Gamma**

 #### Code Resources:
 - Statistical Distributions in numpy/scipy: [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html)
 - Statistical Distributions in pyMC3: [distributions in PyMC3](https://docs.pymc.io/api/distributions.html) (we will see those below).

Exercise: Plot a Discrete variable

Change the value of $\mu$ in the Poisson PMF and see how the plot changes. Remember that the y-axis in a discrete probability distribution shows the probability of the random variable having a specific value in the x-axis.

\begin{equation}
P\left( X=k \right) = \frac{{e^{ - \mu } \mu ^k }}{{k!}}
\end{equation}

**stats.poisson.pmf(x, mu)** $\mu$(mu) is our $\theta$ in this case.

```python
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 30)
for m in [0.5, 3, 8]:
    pmf = stats.poisson.pmf(x, m)
    plt.plot(x, pmf, 'o', alpha=0.5, label='$\mu$ = {}'.format(m))
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability', fontsize=12)
plt.legend(loc=1)
plt.ylim=(-0.1)
plt.show()
```

```python
# same for binomial
plt.style.use('seaborn-darkgrid')
x = np.arange(0, 22)
ns = [10, 17]
ps = [0.5, 0.7]
for n, p in zip(ns, ps):
    pmf = stats.binom.pmf(x, n, p)
    plt.plot(x, pmf, 'o', alpha=0.5, label='n = {}, p = {}'.format(n, p))
plt.xlabel('x', fontsize=14)
plt.ylabel('f(x)', fontsize=14)
plt.legend(loc=1)
plt.show()
```

```python
# discrete uniform
plt.style.use('seaborn-darkgrid')
ls = [0]
us = [3] # watch out, this number can only be integer!
for l, u in zip(ls, us):
    x = np.arange(l, u+1)
    pmf = [1.0 / (u - l + 1)] * len(x)
    plt.plot(x, pmf, '-o', label='lower = {}, upper = {}'.format(l, u))
plt.xlabel('x', fontsize=12)
plt.ylabel('probability P(x)', fontsize=12)
plt.legend(loc=1)
plt.show()
```

Exercise: Plot a continuous variable

Change the value of $\mu$ in the Uniform PDF and see how the plot changes.

Remember that the y-axis in a continuous probability distribution does not shows the actual probability of the random variable having a specific value in the x-axis because that probability is zero!. Instead, to see the probability that the variable is within a small margin we look at the integral below the curve of the PDF.

The uniform is often used as a noninformative prior.

```
Uniform - numpy.random.uniform(a=0.0, b=1.0, size)
```

$\alpha$ and $\beta$ are our parameters. `size` is how many tries to perform.
Our $\theta$ is basically the combination of the parameters a,b. We can also call it
\begin{equation}
\mu = (a+b)/2
\end{equation}

```python
from scipy.stats import uniform

r = uniform.rvs(size=1000)
plt.plot(r, uniform.pdf(r),'r-', lw=5, alpha=0.6, label='uniform pdf')
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.ylabel(r'probability density')
plt.xlabel(f'random variable')
plt.legend(loc='best', frameon=False)
plt.show()
```

```python
from scipy.stats import beta

alphas = [0.5, 1.5, 3.0]
betas = [0.5, 1.5, 3.0]
x = np.linspace(0, 1, 1000)
colors = ['red', 'green', 'blue']

fig, ax = plt.subplots(figsize=(8, 5))

for a, b, colors in zip(alphas, betas, colors):
    dist = beta(a, b)
    plt.plot(x, dist.pdf(x), c=colors,
             label=f'a={a}, b={b}')

ax.set_ylim(0, 3)

ax.set_xlabel(r'$\theta$')
ax.set_ylabel(r'$p(\theta|\alpha,\beta)$')
ax.set_title('Beta Distribution')

ax.legend(loc='best')
fig.show();
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.]
sigmas = [0.4, 1., 2., 0.4]
for mu, sigma in zip(mus, sigmas):
    pdf = stats.norm.pdf(x, mu, sigma)
    plt.plot(x, pdf, label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

```python
plt.style.use('seaborn-darkgrid')
x = np.linspace(-5, 5, 1000)
mus = [0., 0., 0., -2.] # mean
sigmas = [0.4, 1., 2., 0.4] # std
for mu, sigma in zip(mus, sigmas):
      plt.plot(x, uniform.pdf(x, mu, sigma), lw=5, alpha=0.4, \
               label=r'$\mu$ = '+ f'{mu},' + r'$\sigma$ = ' + f'{sigma}')
plt.xlabel('random variable', fontsize=12)
plt.ylabel('probability density', fontsize=12)
plt.legend(loc=1)
plt.show()
```

### D. Is this a Fair Coin?

We do not want to promote gambling but let's say you visit the casino in **Monte Carlo**. You want to test your theory that casinos are dubious places where coins have been manipulated to have a larger probability for tails. So you will try to estimate how fair a coin is based on 100 flips.

You begin by flipping the coin. You get either Heads ($H$) or Tails ($T$) as our observed data and want to see if your posterior probabilities change as you obtain more data, that is, more coin flips. A nice way to visualize this is to plot the posterior probabilities as we observe more flips (data).

We will be using Bayes rule. $\textbf{D}$ is our data.

\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{D}) = \frac{P(\textbf{D} |\theta) P(\theta) }{P(\textbf{D})}
\end{equation}

In the case of a coin toss when we observe $k$ heads in $n$ tosses:
\begin{equation}
\label{eq:bayes}
P(\theta|\textbf{k}) = Beta(\alpha + \textbf{k}, \beta + n - \textbf{k})
\end{equation}

we can say that $\alpha$ and $\beta$ play the roles of a "prior number of heads" and "prior number of tails".

```python
# play with the priors - here we manually set them but we could be sampling from a separate Beta
trials = np.array([0, 1, 3, 5, 10, 15, 20, 100, 200, 300])
heads = np.array([0, 1, 2, 4, 8, 10, 10, 50, 180, 150])
x = np.linspace(0, 1, 100)

# for simplicity we set a,b=1

plt.figure(figsize=(10,8))
for k, N in enumerate(trials):
    sx = plt.subplot(len(trials)/2, 2, k+1)
    posterior = stats.beta.pdf(x, 1 + heads[k], 1 + trials[k] - heads[k])
    plt.plot(x, posterior, alpha = 0.5, label=f'{trials[k]} tosses\n {heads[k]} heads');
    plt.fill_between(x, 0, posterior, color="#348ABD", alpha=0.4)
    plt.legend(loc='upper left', fontsize=10)
    plt.legend()
    plt.autoscale(tight=True)

plt.suptitle("Posterior probabilities for coin flips", fontsize=15);
plt.tight_layout()
plt.subplots_adjust(top=0.88)
```

 [Top](#top)

## 2. Introduction to `pyMC3`

PyMC3 is a Python library for programming Bayesian analysis, and more specifically, data creation, model definition, model fitting, and posterior analysis. It uses the concept of a `model` which contains assigned parametric statistical distributions to unknown quantities in the model. Within models we define random variables and their distributions. A distribution requires at least a `name` argument, and other `parameters` that define it. You may also use the `logp()` method in the model to build the model log-likelihood function. We define and fit the model.

PyMC3 includes a comprehensive set of pre-defined statistical distributions that can be used as model building blocks. Although they are not meant to be used outside of a `model`, you can invoke them by using the prefix `pm`, as in `pm.Normal`.

#### Markov Chain Monte Carlo (MCMC) Simulations

PyMC3 uses the **No-U-Turn Sampler (NUTS)** and the **Random Walk Metropolis**, two Markov chain Monte Carlo (MCMC) algorithms for sampling in posterior space. Monte Carlo gets into the name because when we sample in posterior space, we choose our next move via a pseudo-random process. NUTS is a sophisticated algorithm that can handle a large number of unknown (albeit continuous) variables.

```python
with pm.Model() as model:
    z = pm.Normal('z', mu=0., sigma=5.)
    x = pm.Normal('x', mu=z, sigma=1., observed=5.)
print(x.logp({'z': 2.5}))
print(z.random(10, 100)[:10])
```

    -4.043938533204672
    [-5.52605326  1.20648857  5.8620595   3.04417449 -1.10438114 -9.88886501
     -8.86744907 -6.41620148 -6.57866998 -2.76540963]

**References**:

- *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55* [(https://doi.org/10.7717/peerj-cs.55)](https://doi.org/10.7717/peerj-cs.55)
- [Distributions in PyMC3](https://docs.pymc.io/api/distributions.html)
- [More Details on Distributions](https://docs.pymc.io/developer_guide.html)

Information about PyMC3 functions including descriptions of distributions, sampling methods, and other functions, is available via the `help` command.

```python
#help(pm.Poisson)
```

 [Top](#top)

## 3. Bayesian Linear Regression

Let's say we want to predict outcomes Y as normally distributed observations with an expected value $mu$ that is a linear function of two predictor variables, $\bf{x}_1$ and $\bf{x}_2$.

\begin{equation}
\mu = \alpha + \beta_1 \bf{x}_1 + \beta_2 x_2
\end{equation}

\begin{equation}
Y \sim  \mathcal{N}(\mu,\,\sigma^{2})
\end{equation}

where $\sigma^2$ represents the measurement error.

In this example, we will use $\sigma^2 = 10$

We also choose the parameters as normal distributions:

\begin{eqnarray}
\alpha \sim  \mathcal{N}(0,\,10) \\
\beta_i \sim  \mathcal{N}(0,\,10) \\
\sigma^2 \sim  |\mathcal{N}(0,\,10)|
\end{eqnarray}

We will artificially create the data to predict on. We will then see if our model predicts them correctly.

```python
# Initialize random number generator
np.random.seed(123)

# True parameter values
alpha, sigma = 1, 1
beta = [1, 2.5]

# Size of dataset
size = 100

# Predictor variable
X1 = np.linspace(0, 1, size)
X2 = np.linspace(0,.2, size)

# Simulate outcome variable
Y = alpha + beta[0]*X1 + beta[1]*X2 + np.random.randn(size)*sigma

fig, ax = plt.subplots(1,2, figsize=(10,6), sharex=True)
ax[0].scatter(X1,Y)
ax[1].scatter(X2,Y)
ax[0].set_xlabel(r'$x_1$', fontsize=14)
ax[0].set_ylabel(r'$Y$', fontsize=14)
ax[1].set_xlabel(r'$x_2$', fontsize=14)
ax[1].set_ylabel(r'$Y$', fontsize=14)
```

```python
from pymc3 import Model, Normal, HalfNormal

basic_model = Model()

with basic_model:

    # Priors for unknown model parameters, specifically create stochastic random variables
    # with Normal prior distributions for the regression coefficients,
    # and a half-normal distribution for the standard deviation of the observations, σ.
    alpha = Normal('alpha', mu=0, sd=10)
    beta = Normal('beta', mu=0, sd=10, shape=2)
    sigma = HalfNormal('sigma', sd=1)

    # Expected value of outcome - posterior
    mu = alpha + beta[0]*X1 + beta[1]*X2

    # Likelihood (sampling distribution) of observations
    Y_obs = Normal('Y_obs', mu=mu, sd=sigma, observed=Y)
```

```python
# model fitting with sampling
from pymc3 import NUTS, sample, find_MAP
from scipy import optimize

with basic_model:

    # obtain starting values via MAP
    start = find_MAP(fmin=optimize.fmin_powell)

    # instantiate sampler
    step = NUTS(scaling=start)

    # draw 2000 posterior samples
    trace = sample(2000, step, start=start)
```

    logp = -164.5:   5%|▌         | 270/5000 [00:00<00:02, 1788.68it/s]

    Optimization terminated successfully.
             Current function value: 164.496957
             Iterations: 6
             Function evaluations: 271

    logp = -164.5:   5%|▌         | 271/5000 [00:01<00:31, 148.10it/s]
    Sequential sampling (2 chains in 1 job)
    NUTS: [sigma, beta, alpha]
    Sampling chain 0, 0 divergences: 100%|██████████| 2500/2500 [00:19<00:00, 129.34it/s]
    Sampling chain 1, 0 divergences: 100%|██████████| 2500/2500 [00:18<00:00, 138.27it/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
from pymc3 import traceplot

traceplot(trace);
```

```python
results = pm.summary(trace,
                     var_names=['alpha', 'beta', 'sigma'])
results
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      alpha
      1.009
      0.227
      0.581
      1.423
      0.005
      0.003
      2331.0
      2331.0
      2328.0
      2028.0
      1.00

      beta[0]
      1.542
      2.003
      -2.008
      5.408
      0.089
      0.072
      506.0
      385.0
      514.0
      580.0
      1.01

      beta[1]
      -0.046
      9.831
      -17.396
      18.375
      0.433
      0.338
      516.0
      424.0
      521.0
      582.0
      1.00

      sigma
      1.148
      0.084
      1.000
      1.310
      0.001
      0.001
      3807.0
      3776.0
      3831.0
      2622.0
      1.00

This linear regression example is from the original paper on PyMC3: *Salvatier J, Wiecki TV, Fonnesbeck C. 2016. Probabilistic programming in Python using PyMC3. PeerJ Computer Science 2:e55 https://doi.org/10.7717/peerj-cs.55*

 [Top](#top)

## 4. Try this at Home: Example on Mining Disasters
We will go over the classical `mining disasters from 1851 to 1962` dataset.

This example is from the [pyMC3 Docs](https://docs.pymc.io/notebooks/getting_started.html).

```python
import pandas as pd
disaster_data = pd.Series([4, 5, 4, 0, 1, 4, 3, 4, 0, 6, 3, 3, 4, 0, 2, 6,
                           3, 3, 5, 4, 5, 3, 1, 4, 4, 1, 5, 5, 3, 4, 2, 5,
                           2, 2, 3, 4, 2, 1, 3, np.nan, 2, 1, 1, 1, 1, 3, 0, 0,
                           1, 0, 1, 1, 0, 0, 3, 1, 0, 3, 2, 2, 0, 1, 1, 1,
                           0, 1, 0, 1, 0, 0, 0, 2, 1, 0, 0, 0, 1, 1, 0, 2,
                           3, 3, 1, np.nan, 2, 1, 1, 1, 1, 2, 4, 2, 0, 0, 1, 4,
                           0, 0, 0, 1, 0, 0, 0, 0, 0, 1, 0, 0, 1, 0, 1])
fontsize = 12
years = np.arange(1851, 1962)
plt.figure(figsize=(10,5))
#plt.scatter(years, disaster_data);
plt.bar(years, disaster_data)
plt.ylabel('Disaster count', size=fontsize)
plt.xlabel('Year', size=fontsize);
plt.title('Was there a Turning Point in Mining disasters from 1851 to 1962?', size=15);
```

#### Building the model

**Step1:** We choose the probability model for our experiment. Occurrences of disasters in the time series is thought to follow a **Poisson** process with a large **rate** parameter in the early part of the time series, and from one with a smaller **rate** in the later part. We are interested in locating the change point in the series, which perhaps is related to changes in mining safety regulations.

```
disasters = pm.Poisson('disasters', rate, observed=disaster_data)
```

We have two rates, `early_rate` if $t<=s$, and `late_rate` if $t>s$, where $s$ is the year the switch was made (a.k.a. the `switchpoint`).

**Step2:** Choose a prior distributions of the two rates, what we believe the rates were before we observed the data, and the switchpoint. We choose Exponential.
```
early_rate = pm.Exponential('early_rate', 1)
```

The parameters of this model are:

**Note:** Watch for missing values. Missing values are handled transparently by passing a MaskedArray or a pandas.DataFrame. Behind the scenes, another random variable, disasters.missing_values is created to model the missing values. If you pass a np.array with missing values you will get an error.

```python
with pm.Model() as disaster_model:

    # discrete
    switchpoint = pm.DiscreteUniform('switchpoint', lower=years.min(), upper=years.max(), testval=1900)

    # Priors for pre- and post-switch rates number of disasters
    early_rate = pm.Exponential('early_rate', 1)
    late_rate = pm.Exponential('late_rate', 1)

    # our theta - allocate appropriate Poisson rates to years before and after current
    # switch is an `if` statement in puMC3
    rate = pm.math.switch(switchpoint >= years, early_rate, late_rate)

    # our observed data as a likelihood function of the `rate` parameters
    # shows how we think our data is distributed
    disasters = pm.Poisson('disasters', rate, observed=disaster_data)
```

#### Model Fitting

```python
# there are defaults but we can also more explicitly set the sampling algorithms
with disaster_model:

    # for continuous variables
    step1 = pm.NUTS([early_rate, late_rate])

    # for discrete variables
    step2 = pm.Metropolis([switchpoint, disasters.missing_values[0]] )

    trace = pm.sample(10000, step=[step1, step2])
    # try different number of samples
    #trace = pm.sample(5000, step=[step1, step2])
```

    Sequential sampling (2 chains in 1 job)
    CompoundStep
    >NUTS: [late_rate, early_rate]
    >CompoundStep
    >>Metropolis: [disasters_missing]
    >>Metropolis: [switchpoint]
    Sampling chain 0, 0 divergences: 100%|██████████| 10500/10500 [00:21<00:00, 489.31it/s]
    Sampling chain 1, 0 divergences: 100%|██████████| 10500/10500 [00:20<00:00, 512.37it/s]
    The number of effective samples is smaller than 10% for some parameters.

#### Posterior Analysis

On the left side plots we notice that our early rate is between 2.5 and 3.5 disasters a year. In the late period it seems to be between 0.6 and 1.2 so definitely lower.

The right side plots show the samples we drew to come to our conclusion.

```python
pm.traceplot(trace, ['early_rate', 'late_rate', 'switchpoint'], figsize=(20,10));
```

```python
results = pm.summary(trace,
                     var_names=['early_rate', 'late_rate', 'switchpoint'])
results
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      early_rate
      3.085
      0.291
      2.538
      3.620
      0.004
      0.003
      6294.0
      6294.0
      6235.0
      10243.0
      1.0

      late_rate
      0.929
      0.119
      0.708
      1.146
      0.001
      0.001
      8104.0
      8104.0
      8056.0
      11769.0
      1.0

      switchpoint
      1889.915
      2.418
      1886.000
      1894.000
      0.096
      0.068
      637.0
      637.0
      658.0
      778.0
      1.0


```python
import os
import bopt
import tqdm
import torch
import numpy as np

from pyDOE2.doe_lhs import lhs
from matplotlib import pyplot as plt
from statsmodels.stats.multitest import multipletests
from scipy.stats import median_absolute_deviation, wilcoxon

%load_ext autoreload
%autoreload 2
```

##### save images and set to use the libertine font (required for ACM)

```python
save_images = False
```

```python
if save_images:
    # configure matplotlib to use latex so we're able to use the libertine font
    plt.rcParams['text.usetex'] = True
    plt.rcParams['text.latex.preamble'] = [r'\usepackage{libertine}',
                                           r'\usepackage[libertine]{newtxmath}',
                                           r'\usepackage{sfmath}',
                                           r'\usepackage[T1]{fontenc}'
                                          ]

    import matplotlib
    matplotlib.rcParams['pdf.fonttype'] = 42
    matplotlib.rcParams['ps.fonttype'] = 42
    matplotlib.rcParams['backend'] = 'pdf'

else:
    plt.rcParams['text.usetex'] = False
```

```python
start_run = 1
end_run = 51
problems = [('Branin', {}),
            ('Eggholder', {}),
            ('GoldsteinPrice', {}),
            ('SixHumpCamel',{} ),
            ('Shekel', {}),
            ('Ackley', {'d': 5}),
            ('Hartmann6', {}),
            ('Michalewicz', {'d': 10}),
            ('Rosenbrock', {'d': 10}),
            ('StyblinskiTang', {'d': 10}),
            ('push4', {}),
            ('push8', {}),
            ]

mean_names = ['MeanZero',
              'MeanMedian',
              'MeanMin',
              'MeanMax',
              'MeanLinearCV',
              'MeanQuadraticCV',
              'MeanRandomForrest',
              'MeanRBFCV',
              ]

acq_names = ['EI']
```

# Evaluate the prediction error of the GP + mean function
For each of the synthetic functions + push4 + push8:
- Select $N = 1000$ LHS locations
- for each of the 51 optimisation runs
    - Evaluate the LHS locations on the function $\mathbf{f} = \{ f_1, f_2, \dots, f_N\}$
    - Train a model using the first 100 training points (i.e. those chosen during optimisation)
    - Predict the values $\{ \hat{f}_1, \hat{f}_2, \dots, \hat{f}_N\}$ of the LHS locations
- save the LHS locations and their function values for each problem instance

NRMSE for each problem instance:
\begin{equation}
NRMSE = \frac{\sqrt{ \frac{1}{N} \sum_{i=1}^N ( f_i - \hat{f_i} )^2 }} { f_{max} - f_{min}},
\end{equation}
where $f_{max}$ and $f_{min}$ are the largest and smallest values of the test locations $\mathbf{f}$ respectively.

#### Replace "False" with "True" on the line below to rerun the experiments

```python
if False:
    N_TEST = 1000
    acq_name = 'EI'
    n_train = 100

    transform_function = getattr(bopt.transforms, 'Transform_Standardize')
    ls_bounds = [1e-6, 1e6]
    out_bounds = [1e-6, 1e6]

    total = 51 * len(problems) * len(mean_names)

    with tqdm.notebook.tqdm(total=total) as pbar:

        for problem_name, problem_params in problems:
            # load basic version of the problem to get its dimension and bounds
            f_class = getattr(bopt.test_problems, problem_name)
            f = f_class(**problem_params)
            f = bopt.util.TorchProblem(bopt.util.UniformProblem(f))
            problem_bounds = torch.stack((f.lb, f.ub))

            # result storage
            r = {'Xtest': torch.tensor(lhs(f.dim, N_TEST)),
                 'Ftest': torch.zeros((51, N_TEST)),
                 'preds': {mean_name: torch.zeros((51, N_TEST)) for mean_name in mean_names}
                 }

            for i, run_no in enumerate(range(start_run, end_run + 1)):
                # ---------------- load problem instance data
                data_path = bopt.util.generate_data_filename(problem_name, run_no,
                                                             problem_params, data_dir='data')
                data = torch.load(data_path)
                if 'problem_params' in data:
                    problem_params.update(data['problem_params'])

                # instantiate problem instance and its bounds
                f_class = getattr(bopt.test_problems, problem_name)
                f = f_class(**problem_params)
                f = bopt.util.TorchProblem(bopt.util.UniformProblem(f))
                problem_bounds = torch.stack((f.lb, f.ub))

                # evaluate the LHS test points for this problem instance
                r['Ftest'][i, :] = f(r['Xtest'])

                # ------------ mean function evaluation
                for mean_name in mean_names:
                    # load the training data
                    fn = bopt.util.generate_save_filename(problem_name, mean_name,
                                                          acq_name, run_no,
                                                          problem_params=problem_params)
                    data = torch.load(fn)
                    Ytr = data['Ytr'][n_train:, ]
                    Xtr = data['Xtr'][n_train:, ]

                    T_out = transform_function(Ytr)
                    train_y = T_out.scale_mean(Ytr)
                    train_x = Xtr

                    # instantiate the mean function and train the model
                    mean_func = getattr(bopt.mean_functions, mean_name)
                    mf = mean_func(train_x, train_y)

                    model, likelihood = bopt.gp.create_and_fit_GP(train_x, train_y, ls_bounds,
                                                                  out_bounds, mf, n_restarts=2)

                    # predict the test data
                    model.eval()
                    with torch.no_grad():
                        observed_pred = likelihood(
                            model(r['Xtest']),
                            noise=model.likelihood.noise.mean().expand(N_TEST)
                        )

                        mu = observed_pred.mean
                        mu = T_out.unscale_mean(mu)  # unscale mean

                    # store the result
                    r['preds'][mean_name][i, :] = mu

                    print(problem_name,
                          'instance', run_no,
                          mean_name,
                          'RMSE:', torch.mean(torch.pow(mu - r['Ftest'][i, :], 2)).item())

                    pbar.update()
                print()

            save_path = f'rmse_results/RMSE_{problem_name}.pt'
            torch.save(obj=r, f=save_path)
```

# Boxplot results

```python
for i, (problem_name, problem_params) in enumerate(problems):
    fn = f'rmse_results/RMSE_{problem_name}.pt'

    if not os.path.exists(fn):
        continue

    data = torch.load(fn)
    Ftest = data['Ftest'].numpy()
    preds = {key: value.numpy() for key, value in data['preds'].items()}

    boxplot_data = []
    boxplot_labels = []
    for j, mean_function in enumerate(preds):
        MSE = np.mean(np.square(Ftest - preds[mean_function]), axis=1)
        RMSE = np.sqrt(MSE)

        NRMSE = RMSE / (np.max(Ftest, axis=1) - np.min(Ftest, axis=1))

        boxplot_data.append(NRMSE)
        boxplot_labels.append(mean_function[4:])

    fig, ax = plt.subplots(1, 1, figsize=(10, 3))
    ax.boxplot(boxplot_data, labels=boxplot_labels)
    ax.set_title(problem_name, fontsize=15)
    ax.set_xlabel('Mean functions', fontsize=15)
    ax.set_ylabel('Normalised RMSE', fontsize=15)
    ax.semilogy()
    plt.show()
```

# create table data (with stats tests)

```python
td = {}

for problem_name, problem_params in problems:

    # ---------- load
    fn = f'rmse_results/RMSE_{problem_name}.pt'

    if not os.path.exists(fn):
        continue

    data = torch.load(fn)
    Ftest = data['Ftest'].numpy()
    preds = {key: value.numpy() for key, value in data['preds'].items()}

    sigmaF = np.std(Ftest, axis=1)
    maxF = np.max(Ftest, axis=1)
    minF = np.min(Ftest, axis=1)

    # ---------- process
    err = np.zeros((len(mean_names), 51))
    for i, mean_name in enumerate(mean_names):
        MSE = np.mean(np.square(Ftest - preds[mean_name]), axis=1)
        RMSE = np.sqrt(MSE)

        NRMSE = RMSE / (maxF - minF)

        err[i, :] = NRMSE

    median_err = np.median(err, axis=1)
    mad_err = median_absolute_deviation(err, axis=1)

    # idx of best mean function
    best_idx = np.argmin(median_err)

    # mask of methods equivlent to the best
    stats_equal_to_best_mask = np.zeros(len(mean_names), dtype='bool')
    stats_equal_to_best_mask[best_idx] = True

    # perform wilcoxon signed rank test between best and all other methods
    p_values = []
    for i, mean_name in enumerate(mean_names):
        if i == best_idx:
            continue

        # a ValueError will be thrown if the runs are all identical,
        # therefore we can assign a p-value of 0 as they are identical
        try:
            _, p_value = wilcoxon(err[best_idx, :], err[i, :])
            p_values.append(p_value)

        except ValueError:
            p_values.append(0)

    # calculate the Holm-Bonferroni correction
    reject_hyp, pvals_corrected, _, _ = multipletests(p_values, alpha=0.05,
                                                      method='holm')

    for reject, mean_name in zip(reject_hyp, [m for m in mean_names
                                   if m != mean_names[best_idx]]):
        # if we can't reject the hypothesis that a technique is
        # statistically equivalent to the best method
        if not reject:
            idx = np.where(np.array(mean_names) == mean_name)[0][0]
            stats_equal_to_best_mask[idx] = True

    td[problem_name] = {'medians': median_err,
                        'mads': mad_err,
                        'best_mask': stats_equal_to_best_mask,
                        'best_idx': best_idx,
                        'err': err}
```

```python
# labels for the table heading
mean_labels = []
for j, mean_name in enumerate(mean_names):
    mean_name = mean_name[4:]

    if mean_name == 'LinearCV':
        mean_name = 'Lin'

    if mean_name == 'QuadraticCV':
        mean_name = 'Quad'

    if mean_name == 'RandomForrest':
        mean_name = 'RF'

    if mean_name == 'RBFCV':
        mean_name = 'RBF'

    if mean_name == 'Zero':
        mean_name = 'Arith'

    if mean_name == 'Median':
        mean_name = 'Med'

    mean_labels.append(mean_name)
```

```python
s = ''
for j, mean_name in enumerate(mean_labels):
    s += f' & {mean_name:s}'
s += ' \\\\\n\\toprule'
print(s)

for problem_name, _ in problems:
    s = f'{problem_name:s} '

    for i, mean_name in enumerate(mean_names):
        s += ' & '

        # if the best method
        if i == td[problem_name]['best_idx']:
            s += r'\best '

        # if stats equiv
        elif td[problem_name]['best_mask'][i]:
            s += r'\statsimilar '

        s += f"{td[problem_name]['medians'][i]:0.3f}"
    s += r' \\'
    print(s)
```

     & Arith & Med & Min & Max & Lin & Quad & RF & RBF \\
    \toprule
    Branin  & 0.271 & 0.271 & 0.271 & 0.263 & 0.270 & 0.263 & \statsimilar 0.121 & \best 0.114 \\
    Eggholder  & 0.217 & \statsimilar 0.165 & 0.216 & 0.177 & \best 0.161 & \statsimilar 0.163 & 0.198 & \statsimilar 0.166 \\
    GoldsteinPrice  & 0.136 & 0.136 & 0.136 & 0.136 & 0.136 & 0.136 & \best 0.069 & 0.136 \\
    SixHumpCamel  & 0.232 & 0.231 & 0.231 & 0.223 & 0.215 & 0.216 & \best 0.142 & 0.228 \\
    Shekel  & 0.265 & 0.215 & 0.494 & \statsimilar 0.058 & 0.083 & 0.094 & 0.142 & \best 0.053 \\
    Ackley  & 0.663 & 0.716 & 0.316 & 0.899 & \best 0.107 & 0.378 & 0.334 & \statsimilar 0.117 \\
    Hartmann6  & 0.222 & 0.394 & 0.184 & 0.154 & 0.140 & 0.185 & 0.195 & \best 0.097 \\
    Michalewicz  & 0.271 & 0.311 & 0.417 & 0.316 & 0.205 & \statsimilar 0.193 & 0.360 & \best 0.190 \\
    Rosenbrock  & 0.362 & 0.368 & 0.371 & 0.270 & 0.257 & \best 0.151 & 0.355 & \statsimilar 0.157 \\
    StyblinskiTang  & 0.325 & 0.347 & 0.360 & \best 0.151 & 0.267 & \statsimilar 0.199 & 0.220 & \statsimilar 0.168 \\
    push4  & 0.202 & 0.200 & 0.256 & 0.254 & \statsimilar 0.153 & \statsimilar 0.152 & 0.222 & \best 0.151 \\
    push8  & 0.205 & 0.202 & 0.238 & 0.168 & 0.136 & \statsimilar 0.129 & 0.191 & \best 0.125 \\