# 18 PDEs: Waves

(See *Computational Physics* Ch 21 and *Computational Modeling* Ch 6.5.)

## Background: waves on a string

Assume a 1D string of length $L$ with mass density per unit length $\rho$ along the $x$ direction. It is held under constant tension $T$ (force per unit length). Ignore frictional forces and the tension is so high that we can ignore sagging due to gravity.

### 1D wave equation
The string is displaced in the $y$ direction from its rest position, i.e., the displacement $y(x, t)$ is a function of space $x$ and time $t$.

For small relative displacements $y(x, t)/L \ll 1$ and therefore small slopes $\partial y/\partial x$ we can describe $y(x, t)$ with a *linear* equation of motion:

Newton's second law applied to short elements of the string with length $\Delta x$ and mass $\Delta m = \rho \Delta x$: the left hand side contains the *restoring force* that opposes the displacement, the right hand side is the acceleration of the string element:

\begin{align}
\sum F_{y}(x) &= \Delta m\, a(x, t)\\
T \sin\theta(x+\Delta x) - T \sin\theta(x) &= \rho \Delta x \frac{\partial^2 y(x, t)}{\partial t^2}
\end{align}

The angle $\theta$ measures by how much the string is bent away from the resting configuration.

Because we assume small relative displacements, the angles are small ($\theta \ll 1$) and we can make the small angle approximation

$$
\sin\theta \approx \tan\theta = \frac{\partial y}{\partial x}
$$

and hence

\begin{align}
T \left.\frac{\partial y}{\partial x}\right|_{x+\Delta x} - T \left.\frac{\partial y}{\partial x}\right|_{x}  &= \rho \Delta x \frac{\partial^2 y(x, t)}{\partial t^2}\\
\frac{T \left.\frac{\partial y}{\partial x}\right|_{x+\Delta x} - T \left.\frac{\partial y}{\partial x}\right|_{x}}{\Delta x} &= \rho \frac{\partial^2 y}{\partial t^2}
\end{align}

or in the limit $\Delta x \rightarrow 0$ a linear hyperbolic PDE results:

\begin{gather}
\frac{\partial^2 y(x, t)}{\partial x^2} = \frac{1}{c^2} \frac{\partial^2 y(x, t)}{\partial t^2}, \quad c = \sqrt{\frac{T}{\rho}}
\end{gather}

where $c$ has the dimension of a velocity. This is the (linear) **wave equation**.

### General solution: waves

General solutions are propagating waves:

If $f(x)$ is a solution at $t=0$ then

$$
y_{\mp}(x, t) = f(x \mp ct)
$$

are also solutions at later $t > 0$.

Because of linearity, any linear combination is also a solution, so the most general solution contains both right and left propagating waves

$$
y(x, t) = A f(x - ct) + B g(x + ct)
$$

(If $f$ and/or $g$ are present depends on the initial conditions.)

In three dimensions the wave equation is

$$
\boldsymbol{\nabla}^2 y(\mathbf{x}, t) - \frac{1}{c^2} \frac{\partial^2 y(\mathbf{x}, t)}{\partial t^2} = 0\
$$

### Boundary and initial conditions

* The boundary conditions could be that the ends are fixed

  $$y(0, t) = y(L, t) = 0$$

* The *initial condition* is a shape for the string, e.g., a Gaussian at the center

  $$
  y(x, t=0) = g(x) = y_0 \frac{1}{\sqrt{2\pi\sigma}} \exp\left[-\frac{(x - x_0)^2}{2\sigma^2}\right]
  $$

  at time 0.
* Because the wave equation is *second order in time* we need a second initial condition, for instance, the string is released from rest:

  $$
  \frac{\partial y(x, t=0)}{\partial t} = 0
  $$

  (The derivative, i.e., the initial displacement velocity is provided.)

### Analytical solution
Solve (as always) with *separation of variables*.

$$
y(x, t) = X(x) T(t)
$$

and this yields the general solution (with boundary conditions of fixed string ends and initial condition of zero velocity) as a superposition of normal modes

$$
y(x, t) = \sum_{n=0}^{+\infty} B_n \sin k_n x\, \cos\omega_n t,
\quad \omega_n = ck_n,\ k_n = n \frac{2\pi}{L} = n k_0.
$$

(The angular frequency $\omega$ and the wave vector $k$ are determined from the boundary conditions.)

The coefficients $B_n$ are obtained from the initial shape:

$$
y(x, t=0) = \sum_{n=0}^{+\infty} B_n \sin n k_0 x = g(x)
$$

In principle one can use the fact that $\int_0^L dx \sin m k_0 x \, \sin n k_0 x = \pi \delta_{mn}$ (orthogonality) to calculate the coefficients:

\begin{align}
\int_0^L dx \sin m k_0 x \sum_{n=0}^{+\infty} B_n \sin n k_0 x &= \int_0^L dx \sin(m k_0 x) \, g(x)\\
\pi \sum_{n=0}^{+\infty} B_n \delta_{mn} &= \dots \\
B_m &= \pi^{-1} \dots
\end{align}

(but the analytical solution is ugly and I cannot be bothered to put it down here.)

## Numerical solution

1. discretize wave equation
2. time stepping: leap frog algorithm (iterate)

Use the central difference approximation for the second order derivatives:

\begin{align}
\frac{\partial^2 y}{\partial t^2} &\approx \frac{y(x, t+\Delta t) + y(x, t-\Delta t) - 2y(x, t)}{\Delta t ^2} = \frac{y_{i, j+1} + y_{i, j-1} - 2y_{i,j}}{\Delta t^2}\\
\frac{\partial^2 y}{\partial x^2} &\approx \frac{y(x+\Delta x, t) + y(x-\Delta x, t) - 2y(x, t)}{\Delta x ^2} = \frac{y_{i+1, j} + y_{i-1, j} - 2y_{i,j}}{\Delta x^2}
\end{align}

and substitute into the wave equation to yield the *discretized* wave equation:

$$
\frac{y_{i+1, j} + y_{i-1, j} - 2y_{i,j}}{\Delta x^2} = \frac{1}{c^2} \frac{y_{i, j+1} + y_{i, j-1} - 2y_{i,j}}{\Delta t^2}
$$

Re-arrange so that the future terms $j+1$ can be calculated from the present $j$ and past $j-1$ terms:

$$
y_{i,j+1} = 2(1 - \beta^2)y_{i,j} - y_{i, j-1} + \beta^2 (y_{i+1,j} + y_{i-1,j}), \quad
\beta := \frac{c}{\Delta x/\Delta t}
$$

This is the time stepping algorithm for the wave equation.

## Numerical implementation

```python
# if you have plotting problems, try
# %matplotlib inline
%matplotlib notebook

import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

plt.style.use('ggplot')
```

### Student version

```python
L = 0.5  # m
Nx = 50
Nt = 100

Dx = L/Nx
Dt = 1e-4  # s

rho = 1.5e-2   # kg/m
tension = 150  # N

c = np.sqrt(tension/rho)

# TODO: calculate beta
beta = c*Dt/Dx
beta2 = beta**2

print("c = {0} m/s".format(c))
print("Dx = {0} m,  Dt = {1} s, Dx/Dt = {2} m/s".format(Dx, Dt, Dx/Dt))
print("beta = {}".format(beta))

X = np.linspace(0, L, Nx+1)  # need N+1!

def gaussian(x, y0=0.05, x0=L/2, sigma=0.1*L):
    return y0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2/(2*sigma**2))

# displacements at j-1, j, j+1
y0 = np.zeros_like(X)
y1 = np.zeros_like(y0)
y2 = np.zeros_like(y0)

# save array
y_t = np.zeros((Nt+1, Nx+1))

# boundary conditions
y0[0] = y0[-1] = y1[0] = y1[-1] = 0
y2[:] = y0

# initial conditions: velocity 0, i.e. no difference between y0 and y1
y0[1:-1] = y1[1:-1] = gaussian(X)[1:-1]

# save initial
t_index = 0
y_t[t_index, :] = y0
t_index += 1
y_t[t_index, :] = y1

for jt in range(2, Nt):
    y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
    y0[:], y1[:] = y1, y2

    t_index += 1
    y_t[t_index, :] = y2
    print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

### Fancy version
Package as a function and can use `step` to only save every `step` time steps.

```python
def wave(L=0.5, Nx=50, Dt=1e-4, Nt=100, step=1,
        rho=1.5e-2, tension=150.):

    Dx = L/Nx

    #rho = 1.5e-2   # kg/m
    #tension = 150  # N

    c = np.sqrt(tension/rho)

    beta = c*Dt/Dx
    beta2 = beta**2

    print("c = {0} m/s".format(c))
    print("Dx = {0} m,  Dt = {1} s, Dx/Dt = {2} m/s".format(Dx, Dt, Dx/Dt))
    print("beta = {}".format(beta))

    X = np.linspace(0, L, Nx+1)  # need N+1!

    def gaussian(x, y0=0.05, x0=L/2, sigma=0.1*L):
        return y0/np.sqrt(2*np.pi*sigma) * np.exp(-(x-x0)**2/(2*sigma**2))

    # displacements at j-1, j, j+1
    y0 = np.zeros_like(X)
    y1 = np.zeros_like(y0)
    y2 = np.zeros_like(y0)

    # save array
    y_t = np.zeros((int(np.ceil(Nt/step)) + 1, Nx+1))

    # boundary conditions
    y0[0] = y0[-1] = y1[0] = y1[-1] = 0
    y2[:] = y0

    # initial conditions: velocity 0, i.e. no difference between y0 and y1
    y0[1:-1] = y1[1:-1] = gaussian(X)[1:-1]

    # save initial
    t_index = 0
    y_t[t_index, :] = y0
    if step == 1:
        t_index += 1
        y_t[t_index, :] = y1

    for jt in range(2, Nt):
        y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
        y0[:], y1[:] = y1, y2

        if jt % step == 0 or jt == Nt-1:
            t_index += 1
            y_t[t_index, :] = y2
            print("Iteration {0:5d}".format(jt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

    return y_t, X, Dx, Dt, step

```

```python
y_t, X, Dx, Dt, step = wave()
```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

### 1D plot
Plot the output in the save array `y_t`. Vary the time steps that you look at with `y_t[start:end]`.

We indicate time by color changing.

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis(i) for i in np.linspace(0, 1, len(y_t))])
ax.plot(X, y_t[:40].T, alpha=0.5);
```

    <IPython.core.display.Javascript object>

### 1D Animation
For 1D animation to work in a Jupyter notebook, use

```python
%matplotlib notebook
```

If no animations are visible, restart kernel and execute the `%matplotlib notebook` cell as the very first one in the notebook.

We use `matplotlib.animation` to look at movies of our solution:

```python
import matplotlib.animation as animation
```

Generate one full period:

```python
y_t, X, Dx, Dt, step = wave(Nt=100)
```

    c = 100.0 m/s
    Dx = 0.01 m,  Dt = 0.0001 s, Dx/Dt = 100.0 m/s
    beta = 1.0
    Completed    99 iterations: t=0.0099 s

The `update_wave()` function simply re-draws our image for every `frame`.

```python
y_limits = 1.05*y_t.min(), 1.05*y_t.max()

fig1 = plt.figure(figsize=(5,5))
ax = fig1.add_subplot(111)
ax.set_aspect(1)

def update_wave(frame, data):
    global ax, Dt, y_limits
    ax.clear()
    ax.set_xlabel("x (m)")
    ax.set_ylabel("y (m)")
    ax.plot(X, data[frame])
    ax.set_ylim(y_limits)
    ax.text(0.1, 0.9, "t = {0:3.1f} ms".format(frame*Dt*1e3), transform=ax.transAxes)

wave_anim = animation.FuncAnimation(fig1, update_wave, frames=len(y_t), fargs=(y_t,),
                                    interval=30, blit=True, repeat_delay=100)

```

    <IPython.core.display.Javascript object>

If you have ffmpeg installed then you can export to a MP4 movie file:

```python
wave_anim.save("string.mp4", fps=30, dpi=300)
```

The whole video can be incoroporated into an html page (uses the HTML5 `<video>` tag).

```python
 with open("string.html", "w") as html:
    html.write(wave_anim.to_html5_video())
```

```python

```

### 3D plot
(Uses functions from previous lessons.)

```python
def plot_y(y_t, Dt, Dx, step=1):
    T, X = np.meshgrid(range(y_t.shape[0]), range(y_t.shape[1]))
    Y = y_t.T[X, T]    # intepret index 0 as "t" and index 1 as "x", but plot x along axis 1 and t along axis 2
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X*Dx, T*Dt*step, Y)
    ax.set_ylabel(r"time $t$ (s)")
    ax.set_xlabel(r"position $x$ (m)")
    ax.set_zlabel(r"displacement $y$ (m)")
    fig.tight_layout()
    return ax

def plot_surf(y_t, Dt, Dx, step=1, filename=None, offset=-1, zlabel=r'displacement',
             elevation=40, azimuth=-20, cmap=plt.cm.coolwarm):
    """Plot y_t as a 3D plot with contour plot underneath.

    Arguments
    ---------
    y_t : 2D array
          displacement y(t, x)
    filename : string or None, optional (default: None)
          If `None` then show the figure and return the axes object.
          If a string is given (like "contour.png") it will only plot
          to the filename and close the figure but return the filename.
    offset : float, optional (default: 20)
          position the 2D contour plot by offset along the Z direction
          under the minimum Z value
    zlabel : string, optional
          label for the Z axis and color scale bar
    elevation : float, optional
          choose elevation for initial viewpoint
    azimuth : float, optional
          chooze azimuth angle for initial viewpoint
    """

    t = np.arange(y_t.shape[0], dtype=int)
    x = np.arange(y_t.shape[1], dtype=int)
    T, X = np.meshgrid(t, x)
    Y = y_t.T[X, T]

    fig = plt.figure()
    ax = fig.add_subplot(111, projection='3d')
    surf = ax.plot_surface(X*Dx, T*Dt*step, Y, cmap=cmap, rstride=1, cstride=1, alpha=1)
    cset = ax.contourf(X*Dx, T*Dt*step, Y, 20, zdir='z', offset=offset+Y.min(), cmap=cmap)

    ax.set_xlabel('x')
    ax.set_ylabel('t')
    ax.set_zlabel(zlabel)
    ax.set_zlim(offset + Y.min(), Y.max())

    ax.view_init(elev=elevation, azim=azimuth)

    cb = fig.colorbar(surf, shrink=0.5, aspect=5)
    cb.set_label(zlabel)

    if filename:
        fig.savefig(filename)
        plt.close(fig)
        return filename
    else:
        return ax
```

```python
plot_y(y_t, Dt, Dx)
```

    <IPython.core.display.Javascript object>

    <matplotlib.axes._subplots.Axes3DSubplot at 0x11a44d780>

```python
plot_surf(y_t, Dt, Dx, offset=-0.04, cmap=plt.cm.coolwarm)
```

    <IPython.core.display.Javascript object>

    <matplotlib.axes._subplots.Axes3DSubplot at 0x11c27b390>

## von Neumann stability analysis: Courant condition

Assume that the solutions of the discretized equation can be written as normal modes

$$
y_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

The time stepping algorith is stable if

$$
|\xi(k)| < 1
$$

Insert normal modes into the discretized equation

$$
y_{i,j+1} = 2(1 - \beta^2)y_{i,j} - y_{i, j-1} + \beta^2 (y_{i+1,j} + y_{i-1,j}), \quad
\beta := \frac{c}{\Delta x/\Delta t}
$$

and simplify (use $1-\cos x = 2\sin^2\frac{x}{2}$):

$$
\xi^2 - 2(1-2\beta^2 s^2)\xi + 1 = 0, \quad s=\sin(k\Delta x/2)
$$

The characteristic equation has roots

$$
\xi_{\pm} = 1 - 2\beta^2 s^2 \pm \sqrt{(1-2\beta^2 s^2)^2 - 1}.
$$

It has one root for

$$
\left|1-2\beta^2 s^2\right| = 1,
$$

i.e., for

$$
\beta s = 1
$$

We have two real roots for

$$
\left|1-2\beta^2 s^2\right| < 1 \\
\beta s > 1
$$

but one of the roots is always $|\xi| > 1$ and hence these solutions will diverge and not be stable.

For

$$
\left|1-2\beta^2 s^2\right| ≥ 1 \\
\beta s ≤ 1
$$

the roots will be *complex conjugates of each other*

$$
\xi_\pm = 1 - 2\beta^2s^2 \pm i\sqrt{1-(1-2\beta^2s^2)^2}
$$

and the *magnitude*

$$
|\xi_{\pm}|^2 =  (1 - 2\beta^2s^2)^2 - (1-(1-2\beta^2s^2)^2) = 1
$$

is unity: Thus the solutions will not grow and will be *stable* for

$$
\beta s ≤ 1\\
\frac{c}{\frac{\Delta x}{\Delta t}} \sin\frac{k \Delta x}{2} ≤ 1
$$

Assuming the "worst case" for the $\sin$ factor (namely, 1), the **condition for stability** is

$$
c ≤ \frac{\Delta x}{\Delta t}
$$

or

$$
\beta ≤ 1.
$$

This is also known as the **Courant condition**. When written as

$$
\Delta t ≤ \frac{\Delta x}{c}
$$

it means that the time step $\Delta t$ (for a given $\Delta x$) must be *smaller than the time that the  wave takes to travel one grid step*.

## Simplified simulation
The above example used real numbers for a cello string. Below is bare-bones where we just set $\beta$.

```python
L = 10.0
Nx = 100
Nt = 200
step = 1

Dx = L/Nx

beta2 = 1.0

X = np.linspace(0, L, Nx+1)  # need N+1!

def gaussian(x):
    return np.exp(-(x-5)**2)

# displacements at j-1, j, j+1
y0 = np.zeros_like(X)
y1 = np.zeros_like(y0)
y2 = np.zeros_like(y0)

# save array
y_t = np.zeros((int(np.ceil(Nt/step)) + 1, Nx+1))

# boundary conditions
y0[0] = y0[-1] = y1[0] = y1[-1] = 0
y2[:] = y0

# initial conditions: velocity 0, i.e. no difference between y0 and y1
y0[:] = y1[:] = gaussian(X)

# save initial
t_index = 0
y_t[t_index, :] = y0
if step == 1:
    t_index += 1
    y_t[t_index, :] = y1

for jt in range(2, Nt):
    y2[1:-1] = 2*(1-beta2)*y1[1:-1] - y0[1:-1] + beta2*(y1[2:] + y1[:-2])
    y0[:], y1[:] = y1, y2

    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        y_t[t_index, :] = y2
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))

```

    Completed   199 iterations: t=199 s

```python
%matplotlib inline
```

```python
ax = plt.subplot(111)
ax.set_prop_cycle("color", [plt.cm.viridis_r(i) for i in np.linspace(0, 1, len(y_t))])
ax.plot(X, y_t.T);
```


# Base - Reinforcement

- author: Marcelo de Gomensoro Malheiros
- revision: 2020-09
- license: MIT (attribution is not required but greatly appreciated)

## Reinforcement model
- diffusion rate is based on the scale $s$
- depends on parameters threshold $t$ and width $w$

\begin{align}
\frac{\partial{c}}{\partial{t}} &= \gamma (t - w - c) (t - c) (t + w - c) + s \nabla^2 c
\\
\gamma &= \frac{3 \sqrt{3}}{2 w^2}
\end{align}

```python
from scipy import ndimage

def reinforcement_model(mc, kc, lc, dc, dt, t, w, g, wrap):

    if wrap: ndimage.convolve(mc, kc, output=lc, mode='wrap')
    else:    ndimage.convolve(mc, kc, output=lc, mode='reflect')

    global c
    c = mc + ((t - w - mc) * (t - mc) * (t + w - mc) * g + dc * lc) * dt
```

## Simulation

Parameter list, with default values shown:
- `threshold=1` - threshold parameter
- `width=1` - width parameter
- `scale=1` - overall scale of the pattern
- `speed=100` - percent of default time step
- `start=0` - start iteration counter
- `stop=1000` - final iteration counter
- `time=None` - simulation time (in milliseconds), used instead of `stop` and taking into account `speed`
- `use_c=False` - use previous values for C?
- `wrap=True` - use toroidal domain if true, non-flux boundary otherwise
- `seed=1` - random seed
- `ini_c=0` - initial constant values for C
- `var_c=2` - added randomness magnitude for C
- `shape=40` - domain dimension (single integer for square, or tuple)
- `axis=False` - show domain size?
- `cmap='inferno'` - Matplotlib colormap used
- `first=False` - show initial state?
- `info=False` - monitor concentration intervals (each 100 iterations, may be changed)
- `limit=None` - plot images with given dimension
- `show='c'` - reagents to show (can be the empty string)
- `size=2` - image output size
- `snap=4` - how many captures are shown (can also be a list of iteration counts)
- `detail=None` - show 1-D plot at given domain row (A in orange and B in blue)
- `extent=(-0.2, 2.2)` - upper and lower values for 1-D plot
- `func=None` - auxiliary function
- `out=None` - output file name (PDF/PNG/JPG/...)
- `dpi=100` - resolution for output
- `interpolation='bilinear'` - used when converting a matrix to an image
- `detect=False` - detect constant-valued and stable pattern states, besides numerical problems (each 100 iterations, may be changed)
- `model=reinforcement_model` - model function used

Global variables set after the simulation is run:
- `c` - final concentrations for C
- `sim` - information about the simulation (can be used as an object or a dict)

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

# reinforcement: cubic autocatalytic reinforcement model - v6.3

kernel_c = np.array([[1, 4, 1], [4, -20, 4], [1, 4, 1]]) / 6

class Bunch(dict):
    def __init__(self, dictionary):
        dict.__init__(self, dictionary)
        self.__dict__.update(dictionary)

def reinforcement(threshold=1, width=1, scale=1, speed=100, start=0, stop=1000, time=None, use_c=False, wrap=True,
                  seed=1, ini_c=0, var_c=2, shape=40,
                  axis=False, cmap='inferno', first=False, info=False, limit=None, show='c', size=2, snap=4,
                  detail=None, extent=(-0.2, 2.2), func=None, out=None, dpi=100, interpolation='bilinear',
                  detect=False, model=reinforcement_model):

    # simulation init

    diff_c = scale
    delta_t = 0.01 * speed / 100
    if time: stop = int(time * 100 / speed) + start

    global sim, c
    np.random.seed(seed)
    if type(shape) == int: shape = (shape, shape)
    if not use_c:
        c = np.full(shape, ini_c, dtype=float)
        if var_c != 0: c += np.random.random_sample(shape) * var_c
    lap_c = np.empty_like(c)
    is_nan = is_stable = is_uniform = last_c = False
    if info:
        high_c = - float('inf')
        low_c = float('inf')
        if info is True: info = 100
    if detect is True: detect = 100

    # plotting helper functions

    def draw(matrix, row):
        if axis: axes[row, col].axis('on')
        axes[row, col].imshow(matrix, cmap=cmap, interpolation=interpolation)
        axes[row, col].set_anchor('N')
        if limit:
            axes[row, col].set_xbound(0, limit[1] - 1)
            axes[row, col].set_ybound(0, limit[0] - 1)

    def plot():
        axes[0, col].set_title(iteration)
        row = 0
        if detail:
            if 'c' in show:
                t = c.copy()
                t[detail - 1,:] = t[detail + 1,:] = c.min()
                draw(t, row); row += 1
        else:
            if 'c' in show: draw(c, row); row += 1

        if detail:
            axes[row, col].axis('on')
            axes[row, col].get_xaxis().set_visible(False)
            axes[row, col].grid()
            axes[row, col].plot((threshold - width,) * shape[1], color='orange', linestyle='--')
            axes[row, col].plot((threshold,        ) * shape[1], color='orange', linestyle='--')
            axes[row, col].plot((threshold + width,) * shape[1], color='orange', linestyle='--')
            axes[row, col].plot(c[detail], color='blue')
            axes[row, col].set_anchor('N')
            axes[row, col].set_ybound(extent[0], extent[1])

    # plotting init

    axes = ax = ay = col = fig = rows = 0
    if 'c' in show: rows += 1
    if detail: rows += 1
    if type(snap) == int:
        if snap > 100: print("too many captures, check 'snap' parameter"); return
        if first: snap = np.linspace(start, stop, snap, dtype=int)
        else: snap = np.linspace(start, stop, snap + 1, dtype=int)[1:]
    cols = len(snap)
    if show:
        fig, axes = plt.subplots(rows, cols, squeeze=False, figsize=(cols * size, rows * size))
        for ay in axes:
            for ax in ay: ax.axis('off')
    if first and show:
        iteration = start
        plot()
        col += 1
    if type(limit) == int: limit = (limit, limit)

    # simulation loop

    gamma = 3 * np.sqrt(3) / (2 * width * width)

    for iteration in range(start + 1, stop + 1):
        if func: func(iteration, seed)

        if detect and iteration % detect == 0: last_c = c.copy()
        if c.shape != shape:
            shape = c.shape
            lap_c = np.empty_like(c)

        model(c, kernel_c, lap_c, diff_c, delta_t, threshold, width, gamma, wrap)

        if info and iteration % info == 0:
            high_c = max(c.max(), high_c)
            low_c = min(c.min(), low_c)
        if detect and iteration % detect == 0:
            if c.ptp() < 0.001: is_uniform = True
            elif np.isnan(np.sum(c)): is_nan = True
            elif type(last_c) != bool and np.allclose(c, last_c, atol=0.00001, rtol=0): is_stable = True
            last_c = c.copy()

        if is_stable or iteration in snap:
            if show: plot()
            col += 1
        if is_stable or is_uniform or is_nan: break

    # finalization

    if info:
        min_c, max_c = c.min(), c.max()
        print('C [{:.2f}, {:.2f}] <{:.2f}, {:.2f}>'.format(min_c, max_c, low_c, high_c), end='  ')
        if is_stable: print('stability of C at {}'.format(iteration))
        elif is_uniform: print('uniformity of C at {}'.format(iteration))
        elif is_nan: print('NaN found in C at {}'.format(iteration))
        else: print()

    if col == 0 or not show: plt.close()
    else:
        plt.show()
        if out: fig.savefig(out, bbox_inches='tight', dpi=dpi)

    del axes, ax, ay, col, cols, draw, fig, gamma, last_c, lap_c, plot, rows
    sim = Bunch(locals())
```

## Simple validation

```python
reinforcement()
```

## Usage examples

```python
# change in the initial state, compensated by the threshold and width parameters, yields the same pattern
reinforcement(ini_c=2, var_c=1, threshold=2.5, width=0.5, scale=1, stop=1000)
```

```python
# show C and 1D detail view (dashed lines indicate the t-w, t and t+w levels)
# show initial state (iteration 0)
reinforcement(detail=25, first=True, snap=5, stop=2000, scale=1.2)
```

```python
# set non-zero 'start' iteration count
# 'time' adjusts number of iterations accordingly to speed
reinforcement(scale=0.6, stop=2000)
reinforcement(scale=0.6, start=1000, time=2000, speed=50)
```

```python
# skip output, but still stop at a stable pattern
reinforcement(detect=True, info=True, scale=0.1, show='')
```

    C [0.00, 2.00] <0.00, 2.00>  stability of C at 600

```python
# stop prematurely when concentrations turn constant
reinforcement(detect=True, threshold=2)
print('is uniform?', sim.is_uniform)
```

    is uniform? True

```python
# fields available for simulation object
for i in sorted(sim):
    print(i + '=' + repr(sim[i]))
```

    axis=False
    cmap='inferno'
    delta_t=0.01
    detail=None
    detect=100
    diff_c=1
    dpi=100
    extent=(-0.2, 2.2)
    first=False
    func=None
    info=False
    ini_c=0
    interpolation='bilinear'
    is_nan=False
    is_stable=False
    is_uniform=True
    iteration=200
    limit=None
    model=<function reinforcement_model at 0x7ff2594009e0>
    out=None
    scale=1
    seed=1
    shape=(40, 40)
    show='c'
    size=2
    snap=array([ 250,  500,  750, 1000])
    speed=100
    start=0
    stop=1000
    threshold=2
    time=None
    use_c=False
    var_c=2
    width=1
    wrap=True


# Fault Tree with qubovert

*qubovert* must be pip installed.

We will encode the Fault Tree problem shown below. The inputs are `e0`, `e1`, `e2`, and `e3`, which are boolean variables; for example, `e0 == 1` if Error 0 occurs, otherwise it is 0. The output is `p0`, which is a boolean indicating top level failure; it is 1 if a top level failure occurs, otherwise it is 0. The goal of this analysis is to find the minimum number of errors (`e0`, `e1`, `e2`, `e3`) that must be 1 in order for `p0` to be 1.

The Fault Tree image above can be summarized by the following:

\begin{align*}
z0 &= e0 \text{ OR }  e3\\
z1 &= e2 \text{ OR } z0\\
z2 &= e0 \text{ OR } e1\\
p0 &= z1 \text{ AND } z2
\end{align*}

We will solve this problem in two ways:

1. Solving with the `qubovert.sat` library
1. Solving with `qubovert.PCBO`

## Solving with the `qubovert.sat` library

Let's create the variables and use the `qubovert.sat` libary to encode `p0`.

```python
from qubovert import boolean_var
from qubovert.sat import OR, AND

e0, e1 = boolean_var('e0'), boolean_var('e1')
e2, e3 = boolean_var('e2'), boolean_var('e3')

z0 = OR(e0, e3)
z1 = OR(z0, e2)
z2 = OR(e0, e1)
p0 = AND(z1, z2)

print(p0)
```

    {('e0',): 1, ('e0', 'e1', 'e3'): -1, ('e1', 'e3'): 1, ('e0', 'e1', 'e3', 'e2'): 1, ('e1', 'e3', 'e2'): -1, ('e0', 'e1', 'e2'): -1, ('e1', 'e2'): 1}

We want to find the minimum number of errors that will lead to a top level error. Thus, we want to minimize `H`.

```python
H = e0 + e1 + e2 + e3

print(H)
```

    {('e0',): 1, ('e1',): 1, ('e2',): 1, ('e3',): 1}

We now subject `H` to the constraint that `p0 == 1`, or equivalently `1 - p0 == 0`, where we notice that `1 - p0` is bounded below by 0 and above by 1. We will add a penalty `lam` to the PCBO to enforce this constraint. For now, let's make it a symbol that we can tune later.

```python
#!pip install sympy
from sympy import Symbol

lam = Symbol("lam", positive=True)

H.add_constraint_eq_zero(1 - p0, lam=lam, bounds=(0, 1))

print("PCBO:\n", H, "\n")
print("Constraints:\n", H.constraints)
```

    PCBO:
     {('e0',): 1 - lam, ('e1',): 1, ('e2',): 1, ('e3',): 1, ('e0', 'e1', 'e3'): lam, ('e1', 'e3'): -lam, ('e0', 'e1', 'e3', 'e2'): -lam, ('e1', 'e3', 'e2'): lam, ('e0', 'e1', 'e2'): lam, ('e1', 'e2'): -lam, (): lam}

    Constraints:
     {'eq': [{('e0',): -1, ('e0', 'e1', 'e3'): 1, ('e1', 'e3'): -1, ('e0', 'e1', 'e3', 'e2'): -1, ('e1', 'e3', 'e2'): 1, ('e0', 'e1', 'e2'): 1, ('e1', 'e2'): -1, (): 1}]}

Notice there is one equality constraint coming from the requirement that `1 == p0`. The `H.is_solution_valid` function will take in a proposed solution to the problem and ensure that this constraint is satisfied.

The `'eq'` key of the constraints dictionary indicates that the quantity equals zero, and the `'lt'` key of the constraints dictionary indicates that the quantity is less than zero. Other possible keys are `'le'`, `'gt'`, and `'ge'`. See the docstrings for `PCBO.add_constraint_eq_zero`, `PCBO.add_constraint_lt_zero`, `PCBO.add_constraint_le_zero`, `PCBO.add_constraint_gt_zero`, and `PCBO.add_constraint_ge_zero` for info.

For testing purposes, let's solve this bruteforce to make sure everything is working.

```python
solutions = H.solve_bruteforce(all_solutions=True)
print(solutions)
```

    [{'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0}]

Notice that there is one unique solution that minimizes the objective function and obeys all the constraints. If just the error `e0` occurs, then a top level failure will occur.

```python
solution = solutions[0]
print("Minimum number of failures that leads to a top level failure:", H.value(solution))
print("p0 =", p0.value(solution))
```

    Minimum number of failures that leads to a top level failure: 1
    p0 = 1

Now let's solve this problem with a generic QUBO solver. Notice that the degree of problem is more than two, making `H` not a natural Quadratic Unconstrained Boolean Optimization Problem (QUBO).

```python
H.degree
```

    4

We can convert it to a QUBO (note that there are some options for the reduction from PUBO to QUBO, see the `H.to_qubo` method for details). Ancilla bits will need to be added, and bit labels are mapped to integers.

```python
Q = H.to_qubo()
print("num PUBO variables", H.num_binary_variables)
print("num QUBO variables", Q.num_binary_variables)
print()
print(Q)
```

    num PUBO variables 4
    num QUBO variables 6

    {(0,): 1 - lam, (1,): 1, (2,): 1, (3,): 1, (4,): 9*lam + 9, (1, 3): 2*lam + 3, (1, 4): -6*lam - 6, (3, 4): -6*lam - 6, (0, 4): lam, (5,): 6*lam + 6, (0, 2): 2*lam + 2, (0, 5): -4*lam - 4, (2, 5): -4*lam - 4, (4, 5): -lam, (2, 4): lam, (1, 5): lam, (1, 2): -lam, (): lam}

For testing purposes, let's solve this with bruteforce to see what the proper value of $\lambda$ should be to enforce the constraints. Notice how we remap the QUBO solution to the PCBO solution with `H.convert_solution(x)`.

```python
for l in (1, 2, 3):
    Q_temp = Q.subs({lam: l})
    solutions = Q_temp.solve_bruteforce(all_solutions=True)
    solutions = [H.convert_solution(x) for x in solutions]
    print('lam', l)
    for s in solutions:
        print("\t", s, "is", "valid" if H.is_solution_valid(s) else "invalid")
    print()
```

    lam 1
    	 {'e0': 0, 'e1': 0, 'e2': 0, 'e3': 0} is invalid
    	 {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0} is valid

    lam 2
    	 {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0} is valid

    lam 3
    	 {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0} is valid

We see that $\lambda = 2$ is sufficient to enforce the constraints. So let's update our QUBO.

```python
Q_good = Q.subs({lam: 2})
```

Now let's solve the QUBO with D'Wave's simulated annealer.

```python
#!pip install dwave-neal
from neal import SimulatedAnnealingSampler

sampler = SimulatedAnnealingSampler()
```

Note that their software package takes in a specific form for QUBOs, namely, the keys of the dictionary must be two element tuples. This form can be accessed from `Q` and `Q_good` with `Q.Q` or `Q_good.Q`.

```python
qubo_sample = sampler.sample_qubo(Q_good.Q, num_reads=100)
print("objective function:", qubo_sample.first.energy + Q_good.offset, "\n")

qubo_solution = qubo_sample.first.sample
print("qubo solution:", qubo_solution, "\n")

solution = H.convert_solution(qubo_solution)
print("pcbo solution:", solution)
print("objective function:", H.value(solution), "\n")

print("The solution is", "valid" if H.is_solution_valid(solution) else "invalid")
```

    objective function: 1.0

    qubo solution: {0: 1, 1: 0, 2: 0, 3: 0, 4: 0, 5: 0}

    pcbo solution: {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0}
    objective function: 1

    The solution is valid

This matches the result of `H.solve_bruteforce()`. Recall that the objective function is equal to the minimum number of failures that will lead to a top level failure.

Now we'll solve an QUSO formulation of our problem. Again we'll take $\lambda = 2$.

```python
L = H.to_quso().subs({lam: 2})
# note that we cannot do H.subs({lam: 2}).to_quso()!! This is because H.subs({lam: 2})
# creates a new PCBO object, and it's mapping from variables labels to integers may be
# different than H's mapping. For example, try H.mapping == H.subs({lam: 2}).mapping a
# few times. They will often be different.
print("num PUBO variables", H.num_binary_variables)
print("num QUSO variables", L.num_binary_variables)
print()
print(L)
```

    num PUBO variables 4
    num QUSO variables 6

    {(0,): 1.5, (): 14.25, (1,): 2.25, (2,): 1.0, (3,): 2.25, (4,): -5.0, (1, 3): 1.75, (1, 4): -4.5, (3, 4): -4.5, (0, 4): 0.5, (5,): -3.0, (0, 2): 1.5, (0, 5): -3.0, (2, 5): -3.0, (4, 5): -0.5, (2, 4): 0.5, (1, 5): 0.5, (1, 2): -0.5}

Similar to their QUBO solver, D'Wave's QUSO solver accepts a specific form for QUSO models, namely a linear term dictionary and a quadratic term dictionary. These can be accessed with `L.h` and `L.J`.

```python
quso_sample = sampler.sample_ising(L.h, L.J, num_reads=100)
print("objective function:", quso_sample.first.energy + L.offset, "\n")

quso_solution = quso_sample.first.sample
print("quso solution:", quso_solution, "\n")

solution = H.convert_solution(quso_solution)
print("pcbo solution:", solution)
print("objective function:", H.value(solution), "\n")

print("The solution is", "valid" if H.is_solution_valid(solution) else "invalid")
```

    objective function: 1.0

    quso solution: {0: -1, 1: 1, 2: 1, 3: 1, 4: 1, 5: 1}

    pcbo solution: {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0}
    objective function: 1

    The solution is valid

Again this matches the result of `H.solve_bruteforce()`.

## Solving with `qubovert.PCBO`

We want to find the minimum number of errors that will lead to a top level error. Thus, we want to minimize `H`. We could make `H` the same way that we did in the previous section, by creating variables with ``qubovert.boolean_var``, but instead for illistration we will make it as a dictionary instead.

```python
from qubovert import PCBO

H = PCBO(
    {(x,): 1 for x in ('e0', 'e1', 'e2', 'e3')}
)

print(H)
```

    {('e0',): 1, ('e1',): 1, ('e2',): 1, ('e3',): 1}

Now let's enforce the constraints. We will need to enforce the constraints with a penalty factor $\lambda$. Let's create a symbol here that we can tune later.

```python
#!pip install sympy
from sympy import Symbol

lam = Symbol("lam", positive=True)
```

Now let's enforce the constraints (reproduced here for reference):
\begin{align*}
z0 &= e0 \text{ OR }  e3\\
z1 &= e2 \text{ OR } z0\\
z2 &= e0 \text{ OR } e1
\end{align*}

```python
H.add_constraint_eq_OR(
    'z0', 'e0', 'e3', lam=lam
).add_constraint_eq_OR(
    'z1', 'e2', 'z0', lam=lam
).add_constraint_eq_OR(
    'z2', 'e0', 'e1', lam=lam
)

print(H)
```

    {('e0',): 2*lam + 1, ('e1',): lam + 1, ('e2',): lam + 1, ('e3',): lam + 1, ('z0',): 2*lam, ('e0', 'e3'): lam, ('e0', 'z0'): -2*lam, ('z0', 'e3'): -2*lam, ('z1',): lam, ('z0', 'e2'): lam, ('e2', 'z1'): -2*lam, ('z0', 'z1'): -2*lam, ('z2',): lam, ('e0', 'e1'): lam, ('e0', 'z2'): -2*lam, ('e1', 'z2'): -2*lam}

Finally, we want to make $p0 = z1 \text{ AND } z2$ energetically favorable. We can do this with the following.

```python
H.add_constraint_AND('z1', 'z2', lam=lam)
print("PCBO:\n", H, "\n")
print("Constraints:\n", H.constraints)
```

    PCBO:
     {('e0',): 2*lam + 1, ('e1',): lam + 1, ('e2',): lam + 1, ('e3',): lam + 1, ('z0',): 2*lam, ('e0', 'e3'): lam, ('e0', 'z0'): -2*lam, ('z0', 'e3'): -2*lam, ('z1',): lam, ('z0', 'e2'): lam, ('e2', 'z1'): -2*lam, ('z0', 'z1'): -2*lam, ('z2',): lam, ('e0', 'e1'): lam, ('e0', 'z2'): -2*lam, ('e1', 'z2'): -2*lam, ('z2', 'z1'): -lam, (): lam}

    Constraints:
     {'eq': [{('z0',): 1, ('e0',): 1, ('e3',): 1, ('e0', 'e3'): 1, ('e0', 'z0'): -2, ('z0', 'e3'): -2}, {('z1',): 1, ('e2',): 1, ('z0',): 1, ('z0', 'e2'): 1, ('e2', 'z1'): -2, ('z0', 'z1'): -2}, {('z2',): 1, ('e0',): 1, ('e1',): 1, ('e0', 'e1'): 1, ('e0', 'z2'): -2, ('e1', 'z2'): -2}, {('z2', 'z1'): -1, (): 1}]}

The `H.is_solution_valid` function will take in a proposed solution to the problem and ensure that these constraints are satisfied.

For testing purposes, let's solve this bruteforce to make sure everything is working.

```python
solutions = H.solve_bruteforce(all_solutions=True)
print(solutions)
```

    [{'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 1, 'z1': 1, 'z2': 1}]

Notice that there is one unique solution that minimizes the objective function and obeys all the constraints. If just the error `e0` occurs, then a top level failure will occur.

```python
solution = solutions[0]
print("Minimum number of failures that leads to a top level failure:", H.value(solution))
print("p0 =", H.value(solution))
```

    Minimum number of failures that leads to a top level failure: 1
    p0 = 1

Now let's solve this problem with a generic QUBO solver. Notice that the degree of problem is two, making `H` a natural Quadratic Unconstrained Boolean Optimization Problem (QUBO).

```python
H.degree
```

    2

```python
Q = H.to_qubo()
print(Q)
```

    {(0,): 2*lam + 1, (1,): lam + 1, (2,): lam + 1, (3,): lam + 1, (4,): 2*lam, (0, 3): lam, (0, 4): -2*lam, (3, 4): -2*lam, (5,): lam, (2, 4): lam, (2, 5): -2*lam, (4, 5): -2*lam, (6,): lam, (0, 1): lam, (0, 6): -2*lam, (1, 6): -2*lam, (5, 6): -lam, (): lam}

For testing purposes, let's solve this with bruteforce to see what the proper value of $\lambda$ should be to enforce the constraints. Notice how we remap the QUBO solution to the PCBO solution with `H.convert_solution(x)`.

```python
for l in (1, 2, 3):
    Q_temp = Q.subs({lam: l})
    solutions = Q_temp.solve_bruteforce(all_solutions=True)
    solutions = [H.convert_solution(x) for x in solutions]
    print('lam', l)
    for s in solutions:
        print("\t", s, "is", "valid" if H.is_solution_valid(s) else "invalid")
    print()
```

    lam 1
    	 {'e0': 0, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 0, 'z1': 0, 'z2': 0} is invalid
    	 {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 1, 'z1': 1, 'z2': 1} is valid

    lam 2
    	 {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 1, 'z1': 1, 'z2': 1} is valid

    lam 3
    	 {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 1, 'z1': 1, 'z2': 1} is valid

We see that $\lambda = 2$ is sufficient to enforce the constraints. So let's update our QUBO.

```python
Q_good = Q.subs({lam: 2})
```

Now let's solve the QUBO with D'Wave's simulated annealer.

```python
#!pip install dwave-neal
from neal import SimulatedAnnealingSampler

sampler = SimulatedAnnealingSampler()
```

Note that their software package takes in a specific form for QUBOs, namely, the keys of the dictionary must be two element tuples. This form can be accessed from `Q` and `Q_good` with `Q.Q` or `Q_good.Q`.

```python
qubo_sample = sampler.sample_qubo(Q_good.Q, num_reads=100)
print("objective function:", qubo_sample.first.energy + Q_good.offset, "\n")

qubo_solution = qubo_sample.first.sample
print("qubo solution:", qubo_solution, "\n")

solution = H.convert_solution(qubo_solution)
print("pcbo solution:", solution)
print("objective function:", H.value(solution), "\n")

print("The solution is", "valid" if H.is_solution_valid(solution) else "invalid")
```

    objective function: 1.0

    qubo solution: {0: 1, 1: 0, 2: 0, 3: 0, 4: 1, 5: 1, 6: 1}

    pcbo solution: {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 1, 'z1': 1, 'z2': 1}
    objective function: 1

    The solution is valid

This matches the result of `H.solve_bruteforce()`. Recall that the objective function is equal to the minimum number of failures that will lead to a top level failure.

Now we'll solve an QUSO formulation of our problem. Again we'll take $\lambda = 2$.

```python
L = H.to_quso().subs({lam: 2})
# note that we cannot do H.subs({lam: 2}).to_quso()!! This is because H.subs({lam: 2})
# creates a new PCBO object, and it's mapping from variables labels to integers may be
# different than H's mapping. For example, try H.mapping == H.subs({lam: 2}).mapping a
# few times. They will often be different.
print("num QUSO variables", L.num_binary_variables)
print()
print(L)
```

    num QUSO variables 7

    {(0,): -1.5, (): 8.0, (1,): -1.0, (2,): -1.0, (3,): -1.0, (0, 3): 0.5, (0, 4): -1.0, (3, 4): -1.0, (2, 4): 0.5, (4,): 0.5, (2, 5): -1.0, (4, 5): -1.0, (5,): 1.5, (0, 1): 0.5, (0, 6): -1.0, (1, 6): -1.0, (6,): 1.5, (5, 6): -0.5}

Similar to their QUBO solver, D'Wave's QUSO solver accepts a specific form for QUSO models, namely a linear term dictionary and a quadratic term dictionary. These can be accessed with `L.h` and `L.J`.

```python
quso_sample = sampler.sample_ising(L.h, L.J, num_reads=100)
print("objective function:", quso_sample.first.energy + L.offset, "\n")

quso_solution = quso_sample.first.sample
print("quso solution:", quso_solution, "\n")

solution = H.convert_solution(quso_solution)
print("pcbo solution:", solution)
print("objective function:", H.value(solution), "\n")

print("The solution is", "valid" if H.is_solution_valid(solution) else "invalid")
```

    objective function: 1.0

    quso solution: {0: -1, 1: 1, 2: 1, 3: 1, 4: -1, 5: -1, 6: -1}

    pcbo solution: {'e0': 1, 'e1': 0, 'e2': 0, 'e3': 0, 'z0': 1, 'z1': 1, 'z2': 1}
    objective function: 1

    The solution is valid

Again this matches the result of `H.solve_bruteforce()`.


# Comparison of asymptotically chisquare-distributed test statistics

This notebook is released under the MIT license. See LICENSE.

For GoF tests, we often use a test statistic that is asymptotically $\chi^2$ distributed.

We compute the test statistic on binned data with $k$ bins with Poisson-distributed counts $n_i$ for which we have estimates of the expected counts $\nu_i$, typically obtained from a fitted model with $m$ free parameters. If the data is truely sampled from the model (the $H_0$ hypothesis), the test statistic $X$ is asymptotically distributed as $\chi^2(k - m)$.

Small p-values $P = \int_{X}^\infty \chi^2(X'; k - m) \, \text{d}X'$ can be used as evidence against the hypothesis $H_0$.

Note: the asymptotic limit is assumed to be taken while keeping the binning constant. If the binning is adaptive to the sample size so that $\nu_i$ remains constant, $X$ will *not* approach the $\chi^2(k - m)$ distribution.

There are candidates for an asymptotically $\chi^2$-distributed test statistic.

* Pearson's test statistic
$$
X_P = \sum_i \frac{(n_i - \nu_i)^2}{\nu_i}
$$

* Pearson's test statistic with $\nu_i$ replaced by bin-wise estimate $n_i$
$$
X_N = \sum_i \frac{(n_i - \nu_i)^2}{n_i}
$$

* Likelihood ratio to saturated model (see e.g. introduction of https://doi.org/10.1088/1748-0221/4/10/P10009)
$$
X_{L} = 2\sum_i \Big( n_i \ln\frac{n_i}{\nu_i} - n_i + \nu_i \Big)
$$
The statistic is equal to $-2\ln(L/L_\text{saturated})$, with $L = \prod_i P_\text{poisson}(n_i; \nu_i)$ and $L_\text{saturated} = \prod_i P_\text{poisson}(n_i; n_i)$, the likelihood for the saturated model with $\nu_i = n_i$. The saturated model has the largest possible likelihood given the data.

* Transform-based test statistic
$$
X_T = \sum_i z_i^2 \quad\text{with}\quad z_i = q_\text{norm}\bigg(\sum_{k=0}^{n_i} P_\text{poisson}(k; v_i)\bigg)
$$
where $q_\text{norm}$ is the quantile of the standard normal distribution. The double-transform converts a Poisson distributed count $k$ into a variable $z$ that has a standard normal distribution.

```python
from sympy import *

n = symbols("n", integer=True, positive=True)
v = symbols("v", real=True, positive=True)

def poisson(n, v):
    return v ** n * exp(-v) / factorial(n)

poisson(n, v)
```

$\displaystyle \frac{v^{n} e^{- v}}{n!}$

```python
X_L = simplify(- 2 * (log(poisson(n, v)) - log(poisson(n, n)))); X_L
```

$\displaystyle 2 n \log{\left(n \right)} - 2 n \log{\left(v \right)} - 2 n + 2 v$

Given this choice, it is fair to ask which test statistic performs best. The test statistics may approach the $\chi^2(k - m)$ distribution with different speeds.

We check this empirically with toy simulations. For each toy experiment, we use a fixed expected value $\nu_i = \mu$ and draw $k$ Poisson-distributed numbers $n_i$ from $\mu$, corresponding to $k$ bins in the toy experiment. We then compute the test statistics $X$ and its p-value, assuming it is $\chi^2(k)$ distributed ($m = 0$ here). This is repeated many times to get a histogram of p-values. The distribution is uniform if the test statistic $X$ is indeed $\chi^2(k)$ distributed. Deviations from uniformity indicate that the test statistic has not reached the asymptotic limit yet.

We then quantify the agreement of the p-value distribution with a flat distribution with the reduced $\chi^2$ value.

```python
import numpy as np
import numba as nb
from scipy.stats import chi2, poisson, norm
import matplotlib.pyplot as plt
from pyik.mplext import plot_hist
```

```python
# numba currently does not support scipy, so we cannot access
# scipy.stats.norm.ppf and scipy.stats.poisson.cdf in a JIT'ed
# function. As a workaround, we wrap special functions from
# scipy to implement the needed functions here.

from numba.extending import get_cython_function_address
import ctypes

def get(name, narg):
    addr = get_cython_function_address("scipy.special.cython_special", name)
    functype = ctypes.CFUNCTYPE(ctypes.c_double, *([ctypes.c_double] * narg))
    return functype(addr)

erfinv = get("erfinv", 1)
gammaincc = get("gammaincc", 2)

@nb.vectorize('float64(float64)')
def norm_ppf(p):
    return np.sqrt(2) * erfinv(2 * p - 1)

@nb.vectorize('float64(intp, float64)')
def poisson_cdf(k, m):
    return gammaincc(k + 1, m)

# check implementations
m = np.linspace(0.1, 3, 20)[:,np.newaxis]
k = np.arange(10)
got = poisson_cdf(k, m)
expected = poisson.cdf(k, m)
np.testing.assert_allclose(got, expected)

p = np.linspace(0, 1, 10)
got = norm_ppf(p)
expected = norm.ppf(p)
np.testing.assert_allclose(got, expected)
```

```python
@nb.njit
def xn(n, v):
    r = 0.0
    k = 0
    for ni in n:
        if ni > 0:
            k += 1
            r += (ni - v) ** 2 / ni
    return r, k

@nb.njit
def xp(n, v):
    return np.sum((n - v) ** 2 / v), len(n)

@nb.njit
def xl(n, v):
    r = 0.0
    for ni in n:
        if ni > 0:
            r += ni * np.log(ni / v) + v - ni
        else:
            r += v
    return 2 * r, len(n)

@nb.njit
def xt(n, v):
    p = poisson_cdf(n, v)
    z = norm_ppf(p)
    return np.sum(z ** 2), len(n)

@nb.njit(parallel=True)
def run(n, mu, nmc):
    args = []
    for i in range(len(n)):
        for j in range(len(mu)):
            for imc in range(nmc):
                args.append((i, j, imc))
    results = np.empty((4, len(n), len(mu), nmc, 2))
    for m in nb.prange(len(args)):
        i, j, imc = args[m]
        ni = n[i]
        mui = mu[j]
        np.random.seed(imc)
        x = np.random.poisson(mui, size=ni)
        rp = xp(x, mui)
        rn = xn(x, mui)
        rl = xl(x, mui)
        rt = xt(x, mui)
        results[0, i, j, imc, 0] = rp[0]
        results[0, i, j, imc, 1] = rp[1]
        results[1, i, j, imc, 0] = rn[0]
        results[1, i, j, imc, 1] = rn[1]
        results[2, i, j, imc, 0] = rl[0]
        results[2, i, j, imc, 1] = rl[1]
        results[3, i, j, imc, 0] = rt[0]
        results[3, i, j, imc, 1] = rt[1]
    return results

mu = np.geomspace(1e-1, 1e3, 17)
n = 1, 3, 10, 30, 100, 1000
nmc = 10000

result = run(n, mu, nmc)
```

```python
def reduced_chi2(r):
    p = 1 - chi2.cdf(*np.transpose(r))
    bins = 20
    xe = np.linspace(0, 1, bins + 1)
    n = np.histogram(p, bins=xe)[0]
    v = len(p) / bins
    return np.sum((n - v) ** 2 / v) / bins

matrix = np.empty((len(result), len(n), len(mu)))
for k in range(len(result)):
    for i, ni in enumerate(n):
        for j, mui in enumerate(mu):
            matrix[k, i, j] = reduced_chi2(result[k, i, j])
```

```python
def plot(r, **kwargs):
    p = 1 - chi2.cdf(*np.transpose(r))
    bins = 20
    xe = np.linspace(0, 1, bins + 1)
    n = np.histogram(p, bins=xe)[0]
    plot_hist(xe, n, **kwargs)

for i, ni in enumerate(n):
    for j, mui in enumerate(mu):
        if j % 4 == 0:  # draw only every forth value of mu
            fig, ax = plt.subplots(1, 4, figsize=(15, 5))
            plt.suptitle(f"k = {ni} mu = {mui:.2f}")
            for k, t in enumerate("PNLT"):
                plt.sca(ax[k])
                plot(result[k, i, j])
                plt.title(f"$X_{t}$ ($\chi^2/n_\mathrm{{dof}} = ${matrix[k, i, j]:.2g})")
```

We can by eye that $X_P$ gives the best results. Its p-value distribution converges to a uniform distribution for lower values of $\mu$ and $n$ than the other two test statistics. We summarize this by plotting the reduced $\chi^2$ for flatness as a function of $\mu$ and $n$.

```python
fig, ax = plt.subplots(1, len(result), figsize=(15, 5), sharex=True, sharey=True)
for i, ni in enumerate(n):
    for axi, matrixi in zip(ax, matrix):
        axi.plot(mu, matrixi[i],
                 label=f"$k = {ni:.1f}$")

for axi, t in zip(ax, "PNLT"):
    axi.set_title(f"$X_{t}$")
    axi.axhline(1, ls=":", color="0.5", zorder=0)
    axi.set_xlabel("$\mu$")
ax[0].set_ylabel("$\chi^2 / n_\mathrm{dof}$")
ax[1].legend()
plt.loglog();
```

```python
for i, ni in enumerate(n):
    plt.figure()
    plt.title(f"k = {ni}")
    for j, t in enumerate("PNLT"):
        plt.plot(mu, matrix[j, i] / matrix[0, i],
                 label=f"$X_{t}$")
    plt.loglog()
    plt.xlabel("$\mu$")
    plt.ylabel("red. $\chi^2$ ratio")
    plt.legend()
```

The classic test statistics $X_P$ perform best overall, it converges to the asymptotic $\chi^2$ distribution for smaller values of $\mu$ and $k$ and is also the simplest to compute.

The statistic $X_L$ shows a curious behavior, it converges well around $k=10$, on par with $X_P$, but the convergence gets worse again for $k > 10$.

The study confirms that none of the test statistics is doing well if the expected count per bin is smaller than about 10. If the original binned data contains bins with small counts, it is recommended to compute the distribution of the test statistic via parametric bootstrapping from the fitted model instead of using the asymptotic $\chi^2$ distribution. Parametric bootstrapping in this case simplifies to drawing new samples for each data bin from a Poisson distribution whose expectation is equal to the expected count per bin predicted by the model. For many such samples one computes the test statistic to obtain a distribution. The p-value is then given by the fraction of sample values that are higher than the original value.


```python
%matplotlib inline
import matplotlib
import seaborn as sns
sns.set()
matplotlib.rcParams['figure.dpi'] = 144
```

# Introduction to Hypothesis Testing

## Statistics vs. Probability

Probability is the study of how (abstract) random variables and distributions behave.  Statistics is the study of how to interpret data while making assumptions about the underlying distributions.  Probability is a rigorous branch of mathematics, involving precise theorems and proofs.  (Applied) statistics is a much more squishy subject that requires imperfect assumptions about real-world phenomena. It is therefore more subject to errors in human judgment and argumentation.

```python
import numpy as np
import scipy as sp
from scipy import stats
import matplotlib.pylab as plt

def plot_hist_dist(rvs, dist, title=None, label='', mean=None, confidence_interval=None, ax=None):
    ax = ax if ax else plt.gca()
    _, bins, _ = ax.hist(rvs, bins=50, alpha=.6, density=True, label=(label + ' rvs').strip(), color='blue')
    xmin, xmax = bins.min(), bins.max()
    xpoints = np.arange(xmin, xmax, (xmax - xmin) / 100)
    ax.plot(xpoints, dist.pdf(xpoints), label=(label+' pdf').strip(), color='black')

    if mean is not None:
        ax.plot([mean, mean], plt.ylim(), label='mean', color='purple')

    if confidence_interval:
        ymid = np.sum(plt.ylim()) / 2.
        plt.text(mean, ymid, 'CI', ha='center', va='bottom')
        plt.annotate("", xy=(confidence_interval[0], ymid), xycoords='data',
                     xytext=(confidence_interval[1], ymid), textcoords='data',
                     arrowprops=dict(arrowstyle="|-|", lw=2, color='r'))

    if title:
        ax.set_title(title)

    ax.legend()

def plot_hist_dist_discrete(rvs, dist, title=None, label='', ax=None):
    ax = ax if ax else plt.gca()
    uniques = np.unique(rvs)
    mids = (uniques[1:] + uniques[:-1]) / 2.
    bins = np.hstack([[uniques[0]-.5], mids, [uniques[-1] + .5]])
    plt.hist(rvs, bins=bins, density=True, label=(label + ' rvs').strip(), alpha=.6, color='blue')
    if title:
        ax.set_title(title)
    plt.plot(uniques, dist.pmf(uniques), label=(label + ' pmf').strip(), color='black')
    ax.legend()
```

## Statistical Inference

The most common applications of statistics are to estimate parameters that describe a population, process, or phenomenon under some model and to test hypotheses (usually about whether a model or model parameters accurately describe a population, process, or phenomenon).

Examples of parameter estimation are:
- Estimating the mean income of American construction workers
- Estimating the variance in daily energy consumption during a summer in New York City
- Estimating the correlation coefficient between height and average points per game for power forwards in the NBA

Examples of hypothesis testing are:
- Testing if there is a significant difference between the mean income of unionized and non-union construction workers
- Testing if the variance in energy consumption is significantly larger than the surge capacity of the electrical grid can accommodate
- Testing if the correlation coefficient between height and points per NBA game is significantly different from zero

Let's examine the example of income of union and non-union construction workers in more depth.

## Estimating Mean

The first step in testing whether the mean income of unionized and non-union workers is significantly different is to make measurements. In statistics we call a collection of measurements a **sample**. There are many sampling methods and collecting an unbiased and representative sample is often a difficult task. Sampling theory is beyond the scope of this lecture.

Let's say you have a sample of data (e.g. individual unionized worker incomes) $X_1, \ldots, X_n$.  We estimate the population mean, $\mu$, as

$$ \overline X = \frac{1}{n} \sum_{k=1}^n X_k. $$

This is implemented by the function `np.mean()` or the `.mean()` method of a NumPy array.

```python
def plot_empirical_dist(dist, size=100):
    X = dist.rvs(size=size)
    mean, var = dist.stats('mv')
    rvs_mean = X.mean()

    plt.hist(X, bins=20, density=True)
    ymin, ymax = plt.ylim()
    ymid = (ymin + ymax) / 2.
    plt.plot([mean]*2, [ymin, ymax], label='true mean')
    plt.plot([rvs_mean]*2, [ymin, ymax], label='rvs mean')
    plt.text(mean, ymid, 'std', ha='center', va='bottom')
    plt.annotate("", xy=(mean-np.sqrt(var), ymid), xycoords='data',
                 xytext=(mean+np.sqrt(var), ymid), textcoords='data',
                 arrowprops=dict(arrowstyle="|-|", connectionstyle="arc3", lw=2, color='k'))
    plt.title(dist.dist.name)
    plt.legend()

dists = (
    sp.stats.expon(scale=1/2.),
    sp.stats.beta(a=2., b=4.),
    sp.stats.norm(loc=2., scale=4.),
    sp.stats.chi2(df=4.)
)

for k, dist in enumerate(dists):
    plt.subplot(2,2,k+1)
    plot_empirical_dist(dist)
plt.tight_layout()
```

Let's create a probability distribution that we can use to generate samples of incomes for union construction workers.

```python
union_incomes = sp.stats.lognorm(1, loc=20, scale=10)
plot_empirical_dist(union_incomes, size=100)
plt.plot(np.arange(20, 100, .01), union_incomes.pdf(np.arange(20, 100, .01)), 'k--')
plt.xlim([10, 100]);
```

## Standard Error of Mean

If we run the example code for estimating the mean, we will get different samples and different sample means.

As we have seen, an estimate is noisy and is not always equal to its expected value.  What is the uncertainty of our sample mean?  More appropriately, what is the standard deviation of the sampling distribution of the sample mean given our sample size $n$?

Let's take our mean estimator $\overline X$ for the sample $X_1, \ldots, X_n$.  The Central Limit Theorem tells us that as $n$ increases, our estimate of $\overline X$ starts to look like the normal distribution:

$$ \overline X \sim N\left(\mu, \frac{\sigma^2}{n} \right) $$

```python
def plot_sample_mean_dist(dist, sample_size, n_samples=100):
    sample_means = [dist.rvs(size=sample_size).mean() for i in range(n_samples)]
    mean, var = dist.stats('mv')

    plt.hist(sample_means, bins=20, density=True, label='Sample means')
    ymin, ymax = plt.ylim()
    ymid = (ymin + ymax) / 2.
    plt.plot(np.linspace(min(sample_means),
                         max(sample_means), 1000),
             sp.stats.norm.pdf(np.linspace(min(sample_means),
                                           max(sample_means), 1000),
                               loc=mean, scale=np.sqrt(var/sample_size)), 'k--',
            label = 'Central limit theorem')
    plt.title("Sampling distribution of sample mean given {} population".format(dist.dist.name))
    plt.legend()

plot_sample_mean_dist(union_incomes, 10)
```

The standard deviation of the sampling distribution of the sample mean (i.e. the normal distribution given by the Central Limit Theorem) is called the **standard error** or **standard error of the sample mean**. As we can see from the result of the Central Limit Theorem

$$ s_{\overline X} = \sqrt{\frac{\sigma^2}{n}} $$

Given an observation and an underlying distribution, the **z-score** measures the number of standard deviations the observation is above the mean.  For the case of the the observation $\overline X$, we expect the z-score
$$ z = \frac{\overline X - \mu}{\sigma / \sqrt{n}} $$
to be distributed as a standard normal (with mean zero and standard deviation one).

**Gotchas:**
- Standard deviation ($\sigma$ in our case) is not standard error ($\sigma/\sqrt{n}$)!  Don't confuse the two.
- What is the difference between the standard deviation and the standard error?

```python
_, var = union_incomes.stats()
n=100
sampling_hist_std = np.std([union_incomes.rvs(size=n).mean() for i in range(1000)])

print("Standard error in central limit theorem: {}".format(np.sqrt(var / n)))
print("Standard error estimated from repeated sampling: {}".format(sampling_hist_std))
```

## Hypothesis Testing

Perhaps we think unionized workers are paid more than their non-union counterparts. We can test this hypothesis in the following conventional framework:

1. **Define a null hypothesis, $H_0$.** The null hypothesis is what we assume to be true at the outset. For example, we might naively believe that unions don't affect construction worker pay. If we know that non-union workers make on average \\$32K/yr, then our null hypothesis is that union construction workers mean income is \\$32K/yr. Formally, $H_0$ is $\mu=32$.

2. **Define an alternative hypothesis, $H_a$ or $H_1$.** The alternative hypothesis is a particular negation of the null hypothesis. In this case, the possibilities are $\mu > 32$, $\mu < 32$, or $\mu \ne 32$. Because we think unionized workers are paid more than non-union workers, we will choose $\mu > 32$.

3. **Choose a threshold of significance, $\alpha$.** Assuming that the null hypothesis is true, we should be unlikely to observe samples with mean incomes much higher than \\$32K/yr. The significance level is a probability threshold at which we decide that we could not have observed such a large deviation from the null hypothesis by random chance alone, and that therefore the null hypothesis is false. We'll choose $\alpha = 0.05$. This is a conventional, _though totally arbitrary_, choice. You should choose an $\alpha$ based on your _tolerance for error_.

We've established that for a sufficiently large sample, the mean income of a sample of union workers is itself a normally distributed random variable with a mean equal to $\mu$ and variance equal to the square of the standard error. Therefore we can calculate the probability of observing a particular sample mean _or larger_

$$ \mathbb{P}(\overline X \ge \overline x) = 1 - \int_{-\infty}^{\overline x} n(y \mid \mu, \frac{\sigma^2}{n})dy = 1 - N\left(\frac{\overline x - \mu}{\sigma/\sqrt{n}}\right)$$

or equivalently

$$ 1 - \int_{-\infty}^{z} n(x \mid 0, 1)dx = 1 - N(z), \textrm{ where } z = \frac{\overline x - \mu}{\sigma/\sqrt{n}}. $$

**Question:** How would we calculate the probability of observing a particular sample mean _or less_?

```python
_, var = union_incomes.stats()
h0 = 32

n=100
sample_mean = union_incomes.rvs(n).mean()
x = np.linspace(h0 - 4 * np.sqrt(var / n), h0 + 4 * np.sqrt(var / n), 1000)
clt = sp.stats.norm(loc=h0, scale=np.sqrt(var / n))

plt.figure(figsize=[16, 4])

plt.subplot(121)
plt.plot(x, clt.pdf(x), 'k--', label = 'Null hypothesis sampling distribution')
plt.fill_between(x[x>sample_mean], 0, clt.pdf(x)[x>sample_mean], label = 'p-value')
plt.text(x[-250], clt.pdf(x).max() / 2, 'p-value: {:.3}'.format(1 - clt.cdf(sample_mean)))
plt.legend(loc=2)

plt.subplot(122)
plt.plot(x, clt.cdf(x), 'k--', label = 'Null hypothesis sampling distribution')
plt.axvline(sample_mean, label='Sample mean')
plt.text(sample_mean-4.5, .5, 'p-value: {:.3}'.format(1 - clt.cdf(sample_mean)))
plt.legend(loc=2);
```

The probability of observing $\overline X$ *or a result that is more extreme* is called the **p-value**.  When we observe a sample with a p-value less than $\alpha$, we take it as evidence to reject the null hypothesis.  What does more extreme mean?  Here are two common definitions:
1. **One-sided test:** If we are testing whether $\overline X$ is statistically significantly greater than $\mu$, the p-value would be given by
$$1-N\left(\overline X \mid \mu, \sigma^2\right) = 1-N\left(z \mid 0, 1\right)$$
where $N(x \mid \mu,\sigma^2)$ is the normal CDF with mean $\mu$ and standard deviation $\sigma$.  If we are testing whether $\overline X$ is statistically significantly less than zero, the p-value would be $N\left(\overline X \mid \mu, \sigma^2\right)$.  As usual, we will probably use $\sigma = \hat \sigma$, the estimated standard deviation.
1. **Two-sided test:** We are testing whether $\overline X$ is different (either greater or smaller) than $\mu$ and if this is statistically significant.  The $p$-value is given by $2 \cdot N\left(-\left|\overline X\right|  \mid  \mu, \sigma^2\right)$.

## Confidence Intervals and Probabilities

We may decide to reject the null hypothesis, $H_0: \mu=32$. In that case, what is $\mu$? Our alternative hypothesis doesn't say!

We define the $z$-$\sigma$ **confidence interval** or $z$-standard-deviation **confidence interval** (where $z \ge 0$) to be the interval
$$ [\overline X - zs, \overline X + zs] \,.$$

If we assume the mean estimate is normally distributed (due to the Central Limit Theorem), then we can use the statistics of the normal distribution to compute the probability that the mean estimate falls within the $z$-$\sigma$ confidence interval.  If $n(x\mid\mu,\sigma)$ is the normal PDF with mean $\mu$ and standard deviation $\sigma$, then the probability that the mean falls within the $z$-$\sigma$ confidence interval is

$$ \int_{\overline X - zs}^{\overline X + zs} n(x \mid \overline X, s)dx = \int_{- z}^{z} n(x \mid 0, 1)dx = N(z) - N(-z) $$
where $N$ is the cumulative (standard) normal distribution.  We usually choose $z$ to be 2 (for a ~95% confidence interval) or 3 (giving a ~99% confidence interval).

```python
# Confidence intervals at various rates

N = 1000

norm = sp.stats.norm()

zs = (1, 2, 2.5, 3)

for k, z in enumerate(zs):
    plt.subplot(2,2,k+1)
    plot_hist_dist(norm.rvs(size=N), norm)
    plt.plot([-z, -z], [0, norm.pdf(-z)], 'r', label='CI')
    plt.plot([z, z], [0, norm.pdf(z)], 'r')
    prob = norm.cdf(z) - norm.cdf(-z)
    plt.title('{} sigma CI: {:.4f}'.format(z, prob))
    plt.legend()

plt.tight_layout()

plt.figure()
z=np.arange(0, 3., .1-1e-9)
plt.plot(z, norm.cdf(z) - norm.cdf(-z), color='black')
plt.xlabel('z-score')
plt.ylabel('probability captured in CI');
```

## Hypothesis Testing Two Means

In our hypothesis testing example, we are interested in whether unionized construction workers make more money than their non-union counterparts. We assert a null hypothesis that they do not, and therefore union workers share the average income of non-union workers: \$32K/yr. Where did this number come from? We "magically" knew it (i.e. we chose it for the example). Sometimes this will be the case -- previous studies will have established good estimates of population parameters that we can then safely assume to be accurate. However, more often we will need to sample both populations that we are comparing.

We may collect a sample of incomes, $X_1, \ldots, X_m$, from union workers and calculate its mean, $\overline X$, and a sample of incomes from non-union workers, $Y_1, \ldots, Y_n$, and calculate its mean, $\overline Y$.  From above, we know that

$$ \overline X \longrightarrow N\left(\mu_X, \frac{\sigma^2_X}{m} \right) \qquad \overline Y \longrightarrow N\left(\mu_Y, \frac{\sigma^2_Y}{n} \right)\,. $$

Because we know that the sum of two normal distributions is also normal, we know that

$$ \overline X - \overline Y \longrightarrow N\left(\mu_X - \mu_Y, \frac{\sigma^2_X}{m} + \frac{\sigma^2_Y}{n} \right)\,. $$

We can use the above to compute a z-score for the difference $\overline X - \overline Y$.

In this framework we would revise our null hypothesis

$$ H_0: \mu=32 \Rightarrow H_0: \mu_x - \mu_y = 0 $$

**Questions:**
1. How do we hypothesis test if $\overline X > \overline Y$ for a specific p-value?
1. How do the formulas change if we model the underlying variables $X$ and $Y$ as Bernoulli trials?

## Estimating Variance

Let's say you have a collection of data $X_1, \ldots, X_n$.  To estimate the variance, one might think the right answer is

$$ \frac{1}{n} \sum_{k=1}^n (X_k - \overline X)^2 $$

where $\overline{X}$ is the mean estimator or the *empirical mean*.  This turns out to be **biased**.  That is, that number is expected to be smaller than the true variance.  Mathematically, if we let $\mu$ be the true mean and $\sigma^2$ be the true variance, then
$$
\begin{align}
    \mathbb{E}\left[ \frac{1}{n}\sum_{k=1}^n \left(X_k-\overline{X}\right)^2 \right]
    &= \mathbb{E}\bigg[ \frac{1}{n}\sum_{k=1}^n \big((X_k-\mu)-(\overline{X}-\mu)\big)^2 \bigg] \\[4pt]
    &= \mathbb{E}\bigg[ \frac{1}{n}\sum_{k=1}^n (X_k-\mu)^2 -
                              2(\overline{X}-\mu)(X_k-\mu) +
                              (\overline{X}-\mu)^2 \bigg] \\[4pt]
    &= \mathbb{E}\bigg[ \frac{1}{n}\sum_{k=1}^n (X_k-\mu)^2 - (\overline{X}-\mu)^2 \bigg] \\[4pt]
    &= \sigma^2 - \mathbb{E}\left[ \left(\overline{X}-\mu\right)^2 \right] \\
    &< \sigma^2\,
\end{align}
$$

Mathematically, we can see that
$$
\begin{align}
    \mathbb{E}\left[ \left(\overline{X}-\mu\right)^2 \right]
    &= \mathbb{E}\left[ \left(\frac{1}{n}\sum_{k=1}^n (X_k-\mu)\right)^2 \right] \\[4pt]
    &= \frac{1}{n^2} \sum_{k=1}^n \mathbb{E}\left[ \left(X_k-\mu\right)^2 \right] \\
    &= \frac{\sigma^2}{n}
\end{align}\,
$$
Combining this with the above, we see that
$$
\begin{align}
    \frac{1}{n-1} \mathbb{E}\left[ \sum_{k=1}^n \left(X_k-\overline{X}\right)^2 \right]
    &= \frac{n}{n-1} \cdot \mathbb{E}\left[ \frac{1}{n} \sum_{k=1}^n \left(X_k-\overline{X}\right)^2 \right] \\[4pt]
    &= \frac{n}{n-1} \cdot \sigma^2 \cdot \left( 1 - \frac{1}{n} \right) \\[4pt]
    &= \sigma^2 \,
\end{align}
$$

Therefore, the **unbiased estimator** of the variance is
$$ \hat\sigma^2 = \frac{1}{n-1} \sum_{k=1}^n (X_k - \overline X)^2\, $$

Both estimators are implemented by `np.var`.  For the former, set the optional parameter `ddof=0`, the default.  For the latter, set `ddof=1`.  Nominally, the unbiased estimator is assuming a single degree of freedom.

```python
dist = sp.stats.norm()
X = dist.rvs(size=[40, 10000])
_, var = dist.stats('mv')
rvs_var, rvs_var_unbiased  = np.var(X, axis=0), np.var(X, axis=0, ddof=1)

plt.hist(rvs_var, label='biased variance', bins=40, alpha=.5)
plt.hist(rvs_var_unbiased, label='unbiased variance', bins=40, alpha=.5)
plt.legend()
plt.xlabel('Calculated variance')
plt.ylabel('Count')
plt.title("Distribution of biased and unbiased variances")

print("True variance to unbiased variance", np.abs(rvs_var_unbiased.mean() - var))
print("True variance to biased variance", np.abs(rvs_var.mean() - var))
```

### Student t-Distribution

In real life, we usually do not know $\sigma$ so we often use $\hat\sigma$, the square root of the estimate for the variance $\sigma^2$.  Because $\hat\sigma$ is now a function of the data, even if $\overline X$ were a normal random variable, the distribution of the z-score

$$ z = \frac{\overline X - \mu}{\hat\sigma / \sqrt{n}} $$

is no longer exactly normal but is a [**Student t**-distribution](https://en.wikipedia.org/wiki/Student%27s_t-distribution) with degrees of freedom $\nu = n - 1$.  The PDF is given by

$$ p(x) = \frac{\Gamma(\frac{\nu+1}{2})} {\sqrt{\nu\pi}\,\Gamma(\frac{\nu}{2})} \left(1+\frac{x^2}{\nu} \right)^{\!-\frac{\nu+1}{2}} \,$$

Note: $\Gamma(w)$ used in the PDF is the [gamma function](https://en.wikipedia.org/wiki/Gamma_function).

The distribution has the following statistics:
- The mean is $\mathbb{E}[X] = 0.$
- The variance is $\mbox{Var}[X] = \frac{\nu}{\nu-2}$.

Fortunately as $\nu \to \infty$ (or $n \to \infty$), this approaches the standard normal distribution

$$ \overline X \longrightarrow N\left(\mu, \frac{\sigma^2}{n} \right)\,$$

where $\mu$ and $\sigma$ are the mean and the standard deviation of each of the $X_k$.  So we expect that the **standard error** - the standard deviation of this normal distribution - becomes
$$ s = \frac{\sigma}{\sqrt{n}}\,. $$

### Large n Assumption

The assumption that the distribution of $\overline X$ is normal is only valid in the limit of large $n$.  This is because
1. Central Limit Theorem only ensures that $\overline X$ becomes normal for large $n$ and
1. The use of $\hat\sigma$ rather than $\sigma$ ensures that we are approaching the Student t-distribution, which is only approximately normal if $n$ is large.

What is the boundary for "large"?  Conventional wisdom puts it between 20 and 50. Again, this convention is _totally arbitrary_, and you should decide for yourself based on your _tolerance for error_.

## Multiple Hypothesis Testing

Sometimes, instead of testing for just one hypothesis, we end up testing for multiple hypotheses.  This can be very dangerous and is generally something to be avoided.  For example, assume that we are rejecting null hypotheses at 5% p-values.  That is, we reject null hypotheses if, assuming they were true, we would expect to see data this extreme less than 5% of the time.  Then, if we test 20 independent null hypotheses, we'd expect to see sample data that falls outside of a 5% p-value at least once. Hence, we'd expect to (incorrectly) reject 1 null hypothesis even in the event that all 20 null hypotheses were true.

There are corrections for this (for example, the [Bonferroni correction](https://en.wikipedia.org/wiki/Bonferroni_correction)).

*Copyright © 2019 [Pragmatic Institute](https://www.pragmaticmarketing.com/data-science). This content is licensed solely for personal use. Redistribution or publication of this material is strictly prohibited.*