# Spectral Analysis of Deterministic Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Short-Time Fourier Transform

The discrete Fourier transform (DFT) is not very well suited for the analysis of instationary signals when applied to the entire signal at once. Furthermore practical signals, for instance, an antenna signal, cannot be analyzed in an on-line manner by the DFT. This motivates to split a long signal into segments and compute the DFT on these segments. This transform is known as the [short-time Fourier transform](https://en.wikipedia.org/wiki/Short-time_Fourier_transform) (STFT).

The STFT $X[\mu, n]$ of a signal $x[k]$ is defined as

\begin{equation}
X[\mu, n] = \sum_{k = n}^{n+N-1} x[k] \, w[k-n] \;  \mathrm{e}^{\,-\mathrm{j}\,\frac{2 \pi}{N} k \mu}
\end{equation}

where $w[k]$ denotes a window function of length $N$ which is normalized by $\sum_{k=0}^{N-1} w[k] = 1$. Starting from $k=n$, the signal $x[k]$ is windowed by $w[k]$ to a segment of length $N$. This windowed segment is then transformed by a DFT of length $N$.

The STFT has many applications in digital signal processing, for instance, in the spectral analysis of signals or the processing of instationary signals. The resulting spectrum $X[\mu, n]$ depends on the frequency index $\mu$ and the time index $n$. The spectral domain is therefore also termed as [time-frequency domain](https://en.wikipedia.org/wiki/Time%E2%80%93frequency_representation) and techniques using the STFT as time-frequency processing.

The properties of the STFT depend on

* the length $N$ of the segments,
* the overlap between the segments, and
* the window function $w[k]$.

The size $N$ of the segments and the window function influence the spectral and temporal resolution of the STFT. The time index $n$ of the STFT can be increased by an arbitrary step size. The step size determines the overlap between two consecutive STFTs. For instance, the spectra $X[\mu, n]$ and $X[\mu, n+1]$ have $N-1$ overlapping samples. The overlap is sometimes given as percentage of the segment length $N$.

## The Spectrogram

The magnitude $|X[\mu, n]|$ of the STFT is known as the [spectrogram](https://en.wikipedia.org/wiki/Spectrogram) of a signal. It is frequently used to analyze signals in the time-frequency domain, for instance by a [spectrum analyzer](https://en.wikipedia.org/wiki/Spectrum_analyzer).

**Example**

The following example computes the spectrogram of a chirp/sweep signal.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 8192  # length of the signal
L = 256  # length of one segment
overlap = 128 # overlap between segments

# generate signal
k = np.arange(N)
x = sig.chirp(k, 0, N, .5)

# compute and plot spectrogram
plt.figure(figsize = (10, 5))
plt.specgram(x, NFFT=L, Fs=2, noverlap=overlap, sides='onesided')
plt.xlabel(r'$n$')
plt.ylabel(r'$\Omega$ / $\pi$')
cb = plt.colorbar()
cb.set_label(r'$|X[\Omega,n]|$ in dB')
plt.autoscale(tight=True)
```

**Exercise**

* Can you characterize the signal from its spectrogram? How would it sound like?
* Change the segment length `L` and the overlap `overlap` between segments. Rerun the example. What changes in the spectrogram?

The signal consists of single frequencies that increase with time. A higher segment length leads to a narrower main lobe and thus better selection of the current frequency. With a higher overlap, the spectrum has a higher temporal resolution.

**Example**

The following example computes the spectrogram of a speech signal containing a short sentence. The source signal is loaded using the [`soundfile`](http://pysoundfile.readthedocs.io/en/latest/) module for Python.

```python
import soundfile as sf

L = 1024  # length of one segment
overlap = 512  # overlap between segments

# read speech signal from file
x, fs = sf.read('../data/speech_8k.wav')
x = x/np.max(np.abs(x))

# compute and plot spectrogram
plt.figure(figsize = (10, 5))
plt.specgram(x, NFFT=L, Fs=fs, noverlap=overlap, sides='onesided')
plt.xlabel(r'$n$ in s')
plt.ylabel(r'$f$ in Hz')
cb = plt.colorbar()
cb.set_label(r'$|X[f,n]|$ in dB')
plt.autoscale(tight=True)
plt.ylim([0, 2000]);
```

**Exercise**

* A speech signal exhibits a specific spectral structure. What insights can you gain from the spectrum above?
* Change the segment length and overlap of the STFT. How does the spectrogram change?

In case of a vowel a speech signal is periodic due to the underlying speech production mechanism. A periodic signal has a line spectrum consisting of a fundamental frequency and various harmonics with multiple of the fundamental frequency. The [harmonic structure of the vowels](https://en.wikipedia.org/wiki/Formant) is visible in the spectrogram.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


# Physics 256
## Monte Carlo Integration

## Last Time

- Simpson's rule
- Improper and divergent integrals
- `scipy.integrate`

## Today
- Error scaling for high dimensional quadrature
- Integration using random numbers

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('notebook');
%config InlineBackend.figure_format = 'retina'
colors = ["#2078B5", "#FF7F0F", "#2CA12C", "#D72827", "#9467BE", "#8C574B",
            "#E478C2", "#808080", "#BCBE20", "#17BED0", "#AEC8E9", "#FFBC79",
            "#98E08B", "#FF9896", "#C6B1D6", "#C59D94", "#F8B7D3", "#C8C8C8",
           "#DCDC8E", "#9EDAE6"]
```

## Beyond One-Dimension

Until now we have performed numerical integrals by approximating them as a disccrete sum over equally sized panels with width:

\begin{equation}
\Delta x = \frac{b-a}{N}.
\end{equation}

### Rectangular

\begin{equation}
\int_a^b f(x) dx = \frac{b-a}{N} \sum_{j=0}^{N-1} f\left(a + \frac{j(b-a)}{N}\right) + \mathrm{O}\left(\frac{1}{N}\right).
\end{equation}

### Trapezoidal

\begin{equation}
\int_a^b f(x) dx = \frac{b-a}{N}\left [\frac{1}{2}f(a) +  \sum_{j=1}^{N-1} f\left(a + \frac{j(b-a)}{N}\right) +\frac{1}{2}f(b)\right] + \mathrm{O}\left(\frac{1}{N^2}\right).
\end{equation}

### Simpson

\begin{equation}
\int_a^b f(x) dx = \frac{b-a}{3N}\left \{ f(a) +  \sum_{j=1}^{N-1}\left[3-(-1)^j\right] f\left(a + \frac{j(b-a)}{N}\right) +f(b)\right\} + \mathrm{O}\left(\frac{1}{N^4}\right).
\end{equation}

We now ask how these methods scale in higher dimensions, i.e.:

\begin{equation}
I = \int dx_1 \int dx_2 \cdots \int dx_d\, f(\vec{x}).
\end{equation}

We will need to break up $d$-dimensional space into a grid with $N$ panels in each dimension which will form $M = N^d$ total points at which we must evaulate our function.  If we say a quadrature method is $k^{th}$ order of the error scales as $N^{-k}$ then using this method in $d$ dimensions will lead to an error:

\begin{equation}
\mathrm{Error} \sim \frac{1}{N^k} = \frac{1}{M^{k/d}}.
\end{equation}

This is bad!  An order $k$-scheme in 1-dimension is an order $k/d$ scheme in $d$-dimensions.

## Integrals as Averages

Consider a $d$-dimensional region of space $\Omega$ and a function $f: \mathbb{R}^d \to \mathbb{R}$.  We are interested in the integral:

\begin{equation}
\int_{\Omega} d\vec{x}\, f(\vec{x})
\end{equation}

We can re-write the integral as an average of the function over the region $\Omega$ with the uniform probability:

\begin{equation}
P(\vec{x}) = \frac{1}{\int_\Omega d\vec{x}}
\end{equation}

which gives:

\begin{equation}
\langle f \rangle_\Omega =  \int_{\Omega} d\vec{x}\, P(\vec{x}) f(\vec{x}).
\end{equation}

If we select $M$ points at random from the uniform distribution $\vec{x}_i \in \mathcal{U}_\Omega$ then we can approximate the average as:

\begin{equation}
I = \langle f \rangle_\Omega \approx \frac{1}{M} \sum_{i=0}^{M-1} f(\vec{x}_i).
\end{equation}

It is just the *average* of the function at a discrete set of points selected uniformly in the region.

Programming challenge

Use Monte Carlo integration to aproximate the integral of the $2$-dimensional function $f(\vec{x})$ on the unit square.

\begin{equation}
I = \int_{-1/2}^{1/2} dx \int_{-1/2}^{1/2} dx \mathrm{e}^{-x^2 -y^2 -x^2 y^2}
\end{equation}

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure(figsize=(12,8))
ax = fig.gca(projection='3d')
X, Y = np.meshgrid(np.linspace(-2, 2, 100), np.linspace(-2, 2, 100))
f = np.exp(-X**2 - Y**2 - X**2*Y**2)
surf = ax.plot_surface(X, Y, f, rstride=2, cstride=2, cmap='bwr', linewidth=0, antialiased=True)

ax.set_xlabel('x',labelpad=14)
ax.set_ylabel('y',labelpad=14)
ax.set_zlabel('z',labelpad=14)

[t.label.set_fontsize(14) for caxis in [ax.xaxis, ax.yaxis, ax.zaxis] for t in caxis.get_major_ticks()];
```

```python
def f(x,y):
    return np.exp(-x**2 - y**2 - x**2*y**2)

# Number of MC points
M = 10000

# initialize the integral
I = 0

for i in range(M):
    x = -0.5 + np.random.random()
    y = -0.5 + np.random.random()
    I += f(x,y)

I /= M

print('I = ',I)
```

    I =  0.84446714582

## Computing the Error

The error in computing the integral is now stochastic (as opposed to systematic).  How can we estimate it?

Suppose we repeat the process of performing the integral many times, i.e. we generate many sets of $M$ points.  The error will be related to the variance of the resulting values for the average.

\begin{align*}
\sigma_I^2 &= \langle I^2\rangle - \langle I\rangle^2 \newline
&= \left\langle \left(\frac{1}{M} \sum_{i=0}^{M-1} f_i\right)^2\right\rangle - \left( \left\langle \frac{1}{M} \sum_{i=0}^{M-1} f_i\right\rangle\right)^2 \newline
&= \frac{1}{M^2}\left[\sum_{i=0}^{M-1} \sum_{j=0}^{M-1}\langle f_i f_j \rangle - \sum_{i=0}^{M-1} \sum_{j=0}^{M-1} \langle f_i \rangle\langle f_j \rangle\right] \newline
&= \frac{1}{M^2}\left[\sum_{i=0}^{M-1} \langle f_i^2 \rangle + \sum_{i\ne j}\langle f_i f_j \rangle - \sum_{i=0}^{M-1} \sum_{j=0}^{M-1} \langle f_i \rangle\langle f_j \rangle\right] \newline
\end{align*}

Now, for completely uncorrelated measurements:

\begin{equation}
\langle f_i f_j \rangle = \langle f_i \rangle \langle f_j \rangle
\end{equation}

and we are left with:

\begin{equation}
\sigma_I^2 = \frac{1}{M^2} \sum_{i=0}^{M-1} \left( \langle f_i^2 \rangle - \langle f_i \rangle^2\right).
\end{equation}

However, $\langle \cdots \rangle$ manes that we are sampling the function of a large number of *sets* of numbers $\left\{\vec{x}_i\right\}$ so:

\begin{equation}
\langle f_i \rangle = \langle f \rangle
\end{equation}

and we are left with:

\begin{equation}
\sigma_I^2 =  \frac{1}{M} \left(\langle f_i^2 \rangle - \langle f_i \rangle^2 \right) = \frac{\sigma_f^2}{M}.
\end{equation}

This is just a result of the [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) and shows that the error in our Monte Carlo integration technicle is:

\begin{equation}
\mathrm{O}\left(\frac{1}{M^{1/2}}\right).
\end{equation}

Recall that we found for $d$-dimensions, the trapezoidal method had an $\mathrm{O}(M^{-2/d})$ error, thus Monte Carlo beats trapezoidal for $d>4$.

Let's check the error scaling for our previous integral.

```python
def Monte_Carlo_integration(f,M):
    r = -0.5 + np.random.random([M,2])
    return np.average(f(r[:,0],r[:,1]))

exact = 0.846008

# we consider different numbers of MC points and many trials
M = np.array(range(1,100,1))
trials = 1000

I = np.zeros([trials,M.size])

# for each trial and number of points, compute the itegral
for trial in range(trials):
    for j,cM in enumerate(M):
        I[trial,j] = Monte_Carlo_integration(f,cM)

# compute the average and standard error in the integral
aveI = np.average(I,axis=0)
errI = np.std(I,axis=0)/np.sqrt(trials)

# the error in the integral
error = np.abs(aveI - exact)
```

```python
from scipy.optimize import curve_fit

def fit_func(x,*a):
    return a[0]/x**a[1]

# fit to extract the form of the error
a1,a1_cov = curve_fit(fit_func,M,error,sigma=errI,p0=(1,1))
plt.plot(M,fit_func(M,*a1), color=colors[1], zorder=0, label=r'$M^{-%2.1f}$'%a1[1])

# compare with the calculated value
plt.errorbar(M,error,yerr=errI, marker='o', capsize=0, elinewidth=1, mec=colors[0],
             mfc=colors[0], ms=6, linestyle='None', ecolor=colors[0], label='Monte Carlo Integration')

plt.ylabel('Error')
plt.xlabel('Number MC Points')
plt.legend(loc='upper right')
```

```python

```


#### Jupyter notebooks

This is a [Jupyter](http://jupyter.org/) notebook using Python.  You can install Jupyter locally to edit and interact with this notebook.

# Differential Equations

We now consider the problem of computing $y(t)$ from an *ordinary differential equation* (ODE)

$$ y'(t) = f(t,y) $$

and initial condition $y(0)$.
In the following, $y$ may be a scalar or vector value.
For convenience of notation, we may suppress the time dependence by writing $y$ instead of $y(t)$.

### Second Order Equations

A second order system such as

$$ y'' = f(t, y, y') $$

may always be converted to a first order system by introducing a new variable

$$ \begin{bmatrix} y_0 \\ y_1 \end{bmatrix}' = \begin{bmatrix} y_1 \\ f(t, y_0, y_1) \end{bmatrix} . $$
Therefore, without loss of generality, we will focus on first order systems.

### Implicit formulations

We have chosen the explicit representation $y' = f(t,y)$, but it is more general to write $h(t,y,y') = 0$.
If $\partial h/\partial y'$ is singular, then this describes a *differential algebraic equation* (DAE).  DAE are more challenging to solve and beyond the scope of this course.

## Linear Equations

If $f(y,t)$ is a linear function of $y$ then we have a linear ODE
$$ y' = A(t) y + \text{source}(t) . $$
If $A(t)$ is independent of $t$ and $\text{source}(t) = 0$ then we have a linear, constant, autonomous ODE.

When $y$ is a scalar then $A = a$ is a scalar and the solution is
$$ y = y(0) e^{at} . $$

#### Question
What qualitative dynamics does this imply for
* $a > 0$?
* $a < 0$?
* $a$ imaginary?

#### Question
* What if $A$ is diagonal?
* What if $A = X \Lambda X^{-1}$?

#### Matrix exponential
The general solution can be written in terms of the matrix exponential.
$$ y(t) = e^{At} y(0) . $$
The matrix exponential is defined by its Taylor series
$$ e^A = 1 + A + \frac{A^2}{2} + \frac{A^3}{3!} + \dotsb $$
and there are many [practical ways to compute it](http://www.cs.cornell.edu/cv/ResearchPDF/19ways+.pdf).

#### Question
Suppose that the diagonalization $A = X \Lambda X^{-1}$ exists and derive a finite expression for the matrix exponential using the scalar `exp` function.

## Forward Euler Method

The simplest method for solving $y'(t) = f(t,y)$ is
to use numerical differentiation to write
$$ y' \approx \frac{y(h) - y(0)}{h} $$
which yields the solution estimate
$$ \tilde y(h) = y(0) + h f(0, y(0)) $$
where $h$ is the step size.

Let's try this on a scalar problem
$$ y' = -k (y - \cos t) $$
where $k$ is a parameter controlling the rate at which the solution $u(t)$ is pulled toward the curve $\cos t$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('seaborn')

def ode_euler(f, y0, tfinal=1, h=0.1):
    y = np.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        tnext = min(t+h, tfinal)
        h = tnext - t
        y += h * f(t, y)
        t = tnext
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

tests = []

class cosine:
    def __init__(self, k=5):
        self.k = k
    def __repr__(self):
        return 'cosine(k={:d})'.format(self.k)
    def f(self, t, y):
        return -self.k * (y - np.cos(t))
    def y(self, t, y0):
        k2p1 = self.k**2+1
        return (y0 - self.k**2/k2p1) * np.exp(-self.k*t) + self.k*(np.sin(t) + self.k*np.cos(t))/k2p1

tests.append(cosine(k=2))
tests.append(cosine(k=5))
```

```python
y0 = np.array([.2])
for test in tests:
    thist, yhist = ode_euler(test.f, y0, h=.3, tfinal=10)
    plt.plot(thist, yhist, '.', label=repr(test)+' Forward Euler')
    plt.plot(thist, test.y(thist, y0), label=repr(test)+' exact')
plt.plot(thist, np.cos(thist), label='cos')
plt.legend(loc='upper right');
```

#### Question

* What happens when $h$ is increased?
* What if $k$ is increased?
* What if the final time is increased?

#### Example: Linear system

Now we consider linear systems

$$ y' = A y $$

which have an exact solution $y(t) = e^{At} y(0)$ in terms of the matrix exponential.

```python
def expm(A):
    """Compute the matrix exponential"""
    L, X = np.linalg.eig(A)
    return X @ np.diag(np.exp(L)) @ np.linalg.inv(X)

class linear:
    def __init__(self, A):
        self.A = A.copy()
    def f(self, t, y):
        return self.A @ y
    def y(self, t, y0):
        return [np.real_if_close(expm(self.A*s) @ y0) for s in t]

test = linear(np.array([[0, 1],[-1, 0]]))
y0 = np.array([.5, 0])
thist, yhist = ode_euler(test.f, y0, h=.2, tfinal=10)
plt.figure()
plt.plot(thist, yhist, '.', label='Euler')
plt.plot(thist, test.y(thist, y0), label='exact')
plt.legend(loc='upper right');
```

#### Questions
* Does shrinking $h$ make this more accurate?
* What if `tfinal` is extended?
* What are the eigenvalues of $A$?

#### Runge-Kutta 4
Let's try a different method.  (We'll look back at where this comes from later.)

```python
def ode_rk4(f, y0, tfinal=1, h=0.1):
    y = np.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        k1 = f(t, y)
        k2 = f(t+h/2, y + k1*h/2)
        k3 = f(t+h/2, y + k2*h/2)
        k4 = f(t+h, y + k3*h)
        y += h/6 * (k1 + 2*k2 + 2*k3 + k4)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

thist, yhist = ode_rk4(test.f, y0, h=.2, tfinal=10)
plt.figure()
plt.plot(thist, yhist, '.', label=repr(test)+' RK4')
plt.plot(thist, test.y(thist, y0), label=repr(test)+' exact');
```

### Linear Stability Analysis

Why did Euler diverge (even if slowly) while RK4 solved this problem accurately?
And why do both methods diverge if the step size is too large?
We can understand the convergence of methods by analyzing the test problem
$$ y' = \lambda y $$
for different values of $\lambda$ in the complex plane.
One step of the Euler method with step size $h$ maps
$$ y \to y + h \lambda y = \underbrace{(1 + h \lambda)}_{R(h \lambda)} y $$
where we have introduced the complex-valued function $R(z)$.

* When does this map cause solutions to "blow up" and when is it stable?

```python
def plot_stability(x, y, Rz, label):
    plt.figure()
    levels = np.linspace(0, 2, 21)
    C = plt.contourf(xx, yy, np.abs(Rz), levels, cmap=plt.cm.coolwarm)
    cbar = plt.colorbar(C, ticks=np.linspace(0, 2, 5))
    plt.axvline(x=0, linewidth=1, color='grey')
    plt.axhline(y=0, linewidth=1, color='grey')
    plt.contour(xx, yy, np.abs(Rz), [.5, 1, 1.5], colors='k')
    plt.title(label)

x = np.linspace(-2,2)
xx, yy = np.meshgrid(x, x)
zz = xx + 1j*yy
R = 1 + zz
plot_stability(xx, yy, R, 'Forward Euler')
```

Evidently the forward Euler method is stable if $z = h\lambda$ is in the unit circle centered at $z=-1$, but not stable otherwise.

#### Implicit methods
Recall that forward Euler is the step
$$ \tilde y(h) = y(0) + h f(0, y(0)) . $$
This can be evaluated **explicitly**; all the terms on the right hand side are known so the approximation $\tilde y(h)$ is computed merely by evaluating the right hand side.
Let's consider an alternative, **backward Euler** (or "implicit Euler"),
$$ \tilde y(h) = y(0) + h f(h, \tilde y(h)) . $$
This is a (generally) nonlinear equation for $\tilde y(h)$.
For the test equation $y' = \lambda y$, the backward Euler method is
$$ \tilde y(h) = y(0) + h \lambda \tilde y(h) $$
or
$$ \tilde y(h) = \underbrace{\frac{1}{1 - h \lambda}}_{R(h\lambda)} y(0) . $$

```python
plot_stability(xx, yy, 1/(1-zz), 'Backward Euler')
```

Evidently backward Euler is stable in the entire left half plane (where the exact solution is also stable) and also in some significant portions of the right half plane.  Let's test it on the oscillator problem.

```python
def solve_newtonfd(f, x0):
    def fdjacobian(x):
        J = np.eye(len(x),len(x))
        base = f(x)
        for col in J.T:
            col[:] = (f(x + 1e-8*col) - base) / 1e-8
        return J
    x = x0.copy()
    while True:
        res = f(x)
        if np.linalg.norm(res) < 1e-6:
            return x
        x -= np.linalg.solve(fdjacobian(x), res)

def ode_beuler(f, y0, tfinal=1, h=0.1):
    y = np.array(y0)
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        # Solve x = y + h f(x)
        def residual(ytilde):
            return ytilde - (y + h * f(t+h, ytilde))
        y = solve_newtonfd(residual, y)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

y0 = np.array([.5, 0])
thist, yhist = ode_beuler(test.f, y0, h=.1, tfinal=40)
plt.plot(thist, yhist, '.')
plt.plot(thist, test.y(thist, y0))
plt.title('Backward Euler');
```

```python
y0 = np.array([.2])
for tst in tests:
    thist, yhist = ode_beuler(tst.f, y0, h=1, tfinal=10)
    plt.plot(thist, yhist, '.', label=repr(tst)+' Backward Euler')
    plt.plot(thist, tst.y(thist, y0), label=repr(tst)+' exact')
plt.plot(thist, np.cos(thist), label='cos')
plt.legend(loc='upper right');
```

#### Observations
* We need to solve a system of equations on each time step.
* The cost to solve the linear system is $O(n^3)$ for a system of $n$ equations.
* The Jacobian matrix is $n\times n$, so requires $n^2$ storage when everything else requires $O(n)$.
* I used finite differencing to compute the Jacobian, but many problems have efficient ways to compute an exact Jacobian (it just takes some programming).
* Backward Euler is stable (as predicted by theory), but not very accurate.
* What happens when you make the time step smaller or larger?

### Midpoint Method

What if instead of evaluating the function at the end of the time step, we evaluated in the middle of the time step using the average of the endpoint values.  After all, something similar improved accuracy for our numerical integration...

$$ \tilde y(h) = y(0) + h f\left(\frac h 2, \frac{\tilde y(h) + y(0)}{2} \right) $$

```python
def ode_midpoint(f, y0, tfinal=1, h=0.1):
    y = y0.copy()
    t = 0
    thist = [t]
    yhist = [y0]
    while t < tfinal:
        h = min(h, tfinal - t)
        # Solve x = y + h f(x)
        def residual(x):
            return x - h * f(t+h/2, (x + y)/2) - y
        y = solve_newtonfd(residual, y)
        t += h
        thist.append(t)
        yhist.append(y.copy())
    return np.array(thist), np.array(yhist)

test = linear(np.array([[0, 1],[-1, 0]]))
y0 =
thist, yhist = ode_midpoint(test.f, y0, h=.5, tfinal=50)
plt.plot(thist, yhist, '.')
plt.plot(thist, test.y(thist, y0))
plt.title('Midpoint');
```

```python
y0 = np.array([.2])
for test in [cosine(k=1000)]:
    thist, yhist = ode_rk4(test.f, y0, h=.001, tfinal=2)
    plt.plot(thist, yhist, '.', label=repr(test)+' Midpoint')
    plt.plot(thist, test.y(thist, y0), label=repr(test)+' exact')
plt.legend(loc='upper right')
plt.title('Cosine');
```

#### Observations/Questions

* This appears to be quite accurate.
* We still have to solve a system of equations on each time step.
* What happens when the step size changes?
* What is the stability function $R(z)$ for the midpoint rule?

### Stability function

When applied to the test problem $y' = \lambda y$, the midpoint method is
\begin{align}
\tilde y(h) &= y(0) + h \lambda \frac{\tilde y(h) + y(0)}{2} \\
(1 - h \lambda/2) y(h) &= (1 + h\lambda/2) y(0) \\
y(h) &= \underbrace{\frac{1 + h\lambda/2}{1 - h \lambda/2}}_{R(h \lambda} y(0)
\end{align}
so the stability function is
$$ R(z) = \frac{1 + z/2}{1 - z/2} .$$

```python
plot_stability(xx, yy, (1+zz)/(1-zz), 'Midpoint')
```

### Runge-Kutta Methods

All of the methods we have seen thus far can be represented as Runge-Kutta methods, which can be arranged as a series of $s$ "stage" equations (possibly coupled) and a completion formula.

$$\begin{split}
Y_i = y(0) + h \sum_j a_{ij} f(t+c_j h, Y_j) \\
y(h) = y(0) + h \sum_j b_j f(t+c_j h, Y_j)
\end{split}$$

where $c$ is a vector of *abscissa*, $A$ is a table of coefficients, and $b$ is a vector of completion weights.
These coefficients are typically expressed in a Butcher Table
$$ \left[ \begin{array}{c|cc}
c_0 & a_{00} & a_{01} \\
c_1 & a_{10} & a_{11} \\
\hline
& b_0 & b_1
\end{array} \right] . $$

If the matrix $A$ is strictly lower triangular, then the method is **explicit** (does not require solving equations).  We have seen forward Euler

$$ \left[ \begin{array}{c|cc}
0 & 0 & 0 \\
1 & 1 & 0 \\
\hline
& 1 & 0
\end{array} \right] ,$$
backward Euler
$$ \left[ \begin{array}{c|c}
1 & 1 \\
\hline
& 1
\end{array} \right] ,$$
and Midpoint
$$ \left[ \begin{array}{c|c}
\frac 1 2 & \frac 1 2 \\
\hline
& 1
\end{array} \right], $$
and that explicit method we called RK4
$$ \left[ \begin{array}{c|cccc}
0 & 0 & 0 & 0 & 0 \\
\frac 1 2 & \frac 1 2 & 0 & 0 & 0 \\
\frac 1 2 & 0 & \frac 1 2 & 0 & 0 \\
1 & 0 & 0 & 1 & 0 \\
\hline
& \frac 1 6 & \frac 1 3 & \frac 1 3 & \frac 1 6
\end{array} \right] . $$

```python
def Rstability(A, b, z):
    s = len(b)
    def R(z):
        return 1 + z * (b @ np.linalg.solve(np.eye(s) - z*A, np.ones(s)))
    f = np.vectorize(R)
    return f(z)

def plot_rkstability(A, b, name):
    x = np.linspace(-4,4)
    xx, yy = np.meshgrid(x, x)
    zz = xx + 1j*yy
    R = Rstability(A, b, zz)
    plot_stability(xx, yy, R, name)

A_rk4 = np.array([[0,0,0,0],
                    [.5,0,0,0],
                    [0,.5,0,0],
                    [0,0,1,0]])
b_rk4 = np.array([1/6, 1/3, 1/3, 1/6])

plot_rkstability(A_rk4, b_rk4, 'RK4')
```

```python
plot_rkstability(np.eye(1)/2, [1], 'RK Midpoint')
```

## Ballistics

The velocity of a particle in the $x$-$z$ plane is the time derivative of its position,
$$ \begin{bmatrix} v_x \\ v_z \end{bmatrix} = \begin{bmatrix} p_x \\ p_z \end{bmatrix}' . $$
The acceleration (derivative of velocity) depends on the force applied,
$$ \begin{bmatrix} v_x \\ v_z \end{bmatrix}' = \frac 1 m \begin{bmatrix} F_x \\ F_z \end{bmatrix} $$
where $m$ is the mass of the particle.
The gravitational force is
$$ \mathbf F^{\text{grav}} = m \begin{bmatrix} 0 \\ -g \end{bmatrix} $$
where $g = 9.8\ \text{meter}/\text{second}^2 $.
The drag force is often approximated as
$$ F^{\text{drag}} = \frac 1 2 \rho v^2 c_d A $$
where $\rho$ is the density of the fluid (assume $1\ \text{kilogram}/\text{meter}^3$),
$v =\sqrt{v_x^2  + v_z^2}$ is the velocity, $c_d$ is the drag coefficient (assume 0.2 for this projectile), and $A = \pi r^2$  is the cross-sectional area of the projectile (assume the radius is $r = .05\ \text{meter}$).
The direction of the drag force is opposite the velocity, thus the vector drag is
$$ \mathbf F^{\text{drag}}(v_x, v_z) = - \frac{\pi}{2} v c_d r^2 \begin{bmatrix} v_x \\ v_z \end{bmatrix} . $$
This produces the system of equations

$$ \begin{bmatrix} p_x \\ p_z \\ v_x \\ v_z \end{bmatrix}' =
\begin{bmatrix} v_x \\ v_z \\ \frac 1 m F_x^{\text{drag}}  \\ \frac 1 m F_z^{\text{drag}} - 9.8 \end{bmatrix} . $$

For the ballistics computation, you should solve this differential equation with an $m=8$ kilogram projectile and initial condition of a velocity  $v_0 = 300\ \text{meter}/\text{second}$ at angle $\theta$ above the horizontal, i.e.,
$$ \begin{bmatrix} 0 \\ 0 \\ v_0 \cos \theta \\ v_0 \sin \theta \end{bmatrix} . $$

The homework question is to compute the angle needed to hit a target on a distant slope of angle $\phi = 40^\circ = 40 \pi / 180$.  For that, you should adapt one of the ODE solvers to determine where the projectile crosses the plane of the target slope and adjust the angle (either by hand or using a rootfinder) to hit the target.  Don't aim too high or your explosive will detonate on the other side of the mountain!  Make sure you use a time step that you are confident is accurate enough.

Your code should print the angle $\theta$ and the time at which your projectile hits the slope.

### Chemical reactions

The [Oregonator](http://www.scholarpedia.org/article/Oregonator) mechanism in chemical kinetics describes an oscillatory chemical system.  It consists of three species with concentrations $\mathbf x = [x_0,x_1,x_2]^T$ (scaled units) and the evolution equations
$$ \mathbf {x'} = \begin{bmatrix} 77.27 \big(x_1 + x_0 (1 - 8.375\cdot 10^{-6} x_0 - x_1) \big) \\
                    \frac{1}{77.27} \big(x_2 - (1 + x_0) x_1 \big) \\
                    0.161 (x_0 - x_2)
                    \end{bmatrix} . $$
Starting with the initial conditions $\mathbf x_0 = [1, 2, 3]^T$, this produces the time evolution below.

This calculation was performed using 287 steps of an adaptive implicit time integrator, where the time step size was adapted to preserve accuracy.  The time step size in shown in the first panel and the other three are the evolution of concentrations over time.

#### 1. How to measure accuracy?

A differential equation solver produces a time history of chemical concentrations
$$ x(t), \quad 0 \le t \le T_{\text{final}} .$$
We would like a measure of accuracy that does not explicitly depend on the solver or time steps that are used.  If a different method produces the evolution $\tilde x(t)$, then we would like to be able to measure the difference as
$$ \lVert \phi(\tilde x) - \phi(x) \rVert . $$
The function $\phi(\cdot)$ could return the solution at a particular time, it could measure the time between key events, or could be something else.
Choose a metric and explain why you chose it.

#### 2. Implicit versus Explicit

Is an implicit method better than an explicit method for solving the Oregonator?
Solve the Oregonator problem using an explicit method and using an implicit method (you can adapt code from the notebook or implement a method that is not in the notebook).
The Oregonator system has no known analytic solution, so you'll have to numerically compute a reference solution.
Do this using a method you believe to be highly accurate and explain why you believe it is very accurate.
(A common justification would be to refine the time step and compare using the metric $\phi$ above.)

Different methods (implicit versus explicit, Euler versus RK4, etc.) have different costs per time step.  Choose a way to measure cost and explain your choice and any shortcomings that it may have.

Compare the cost (using the metric you chose above) and accuracy (using $\phi$) of the implicit method to the cost and accuracy of the explicit method.  (This is often plotted as *accuracy versus cost* with a log-log scale.)  Explain what factors might go into your decision.

#### 3. Eigenvalues

Calculate the eigenvalues of the Jacobian matrix at different times.  (You can compute it analytically or using the `fdjacobian` code in this notebook.)  Do you notice any qualitative differences?  For example, are there any times when there are eigenvalues with positive real part?

#### 4. Generalization

Realistic chemical mechanisms can have hundreds or thousands of species (the Oregonator has only three).  Explain how the choice of methods might change when solving a large reaction mechanism.

```python

```


# 神经网络解偏微分方程

## 一. 介绍
偏微分方程数值解法按是否进行网格划分可以分为两类：
1. 基于网格的方法包括有限差分、有限元、有限体等方法,这些方法需要用网格划分偏微分方程的定义域，网格越细得到的解越精确，然而越细的网格需要更高的计算代价和更大的存储空间。
2. 无网格主要是蒙特卡洛方法，使用Feynman-Kac公式将偏微分方程中的待求函数表示成随机过程中随机变量的数学期望，再通过蒙特卡洛求期望的办法求解偏微分方程。蒙特卡洛法的收敛性依赖于空间中采样的点的个数，采样的点越多，我们能得到越精确的解。但采样点越多，计算时间越长、存储空间越大，在高维情况下需要很大的计算量。

最近基于神经网络的方法求解偏微分方程得到学术界关注。使用神经网络来表示偏微分方程中的待求函数，通过学习神经网络的参数来求得偏微分方程的近似解。

## 二、DGM方法
神经网络就是一个函数族$\{u_\theta\}$，DGM认为偏微分方程的解函数$f$可以用函数族中的某个函数近似$u_\theta^*$。考虑如下形式的偏微分方程：

$$\begin{align}
& \frac{\partial u(t, x) }{\partial t} + F[u(t, x)] = 0 \ (t, x)\in [0, T]\times  \Omega\\
& u(0, x)  = u_0(x), \ x\in \Omega \\
& u(t, x) = g(t, x), \ (t, x)\in [0, T]\times \partial \Omega
\end{align}$$

求解偏微分方程，除偏微分方程形式外，还需要知道初始条件$ \ u_0(x)$和边界条件$ \ u(t, x), \ x\in \partial \Omega$。定义神经网络**期望损失函数**(expected loss function)如下

$$\begin{aligned}
J(\theta) = & \mathbb{E}_{t, x \sim {U_{[0, T]\times \Omega}}}[||\frac{\partial u_{\theta}(t, x) }{\partial t} + F[u_{\theta}(t, x)]||^{2}] \\
&+ \mathbb{E}_{t, x \sim {U_{[0, T]\times \partial\Omega}}}[|| u_\theta(t, x) - g(t, x)||^2] \\
&+ \mathbb{E}_{t, x \sim {U_{\{0\}\times \Omega}}}[|| u_\theta(t, x) - u_0(x)||^2]
\end{aligned}$$

最小化优化函数相当于让$u_{\theta}(t, x)$尽量满足偏微分方程、初始条件和边界条件。在具体优化时，不能直接对期望损失函数进行优化，只能在初始条件、边界条件函数和在整个定义域空间采样作为训练集构造**经验损失函数**(empirical loss function)。

## 三、参考资料
[1] [https://zhuanlan.zhihu.com/p/359328643](https://zhuanlan.zhihu.com/p/359328643)


```python
import numpy
from matplotlib import pyplot
%matplotlib inline
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 14
```

## Question 1.a)

```python
def hw4_q1_func(n_grids, x_begin, x_end, dx, D, t_grids, which_eq):
    # set parameters
    L = x_end - x_begin
    N_x = int(L/dx) + 1 # number of spatial steps
    x = numpy.linspace(x_begin, x_end, num=N_x)
    f_history = [x.copy()]
    if which_eq == 'q(a) & (b)' :
        # equation for question 1(a) and 2(b)
        for t in t_grids:
            for n in n_grids:
                f = numpy.zeros(N_x)
                for nn in range(1, n+1):
                    eigen_value = (2 * nn - 1) * numpy.pi / (2 * L)
                    f = f + (-1) ** (nn + 1) / (2 * nn - 1) * \
                        numpy.cos(eigen_value * x) * \
                        numpy.exp(-D * t * (eigen_value ** 2))
                f = f * 4 / numpy.pi
                f_history.append(f)
        return f_history
    elif which_eq == 'q(c) & (d)' :
        # equation for question 1(c) and 2(d)
        for t in t_grids:
            for n in n_grids:
                f = numpy.zeros(N_x)
                for nn in range(1, n+1):
                    eigen_value = (2 * nn - 1) * numpy.pi / (2 * L)
                    f = f + (1 / ((2 * nn -1) ** 2)) * \
                        numpy.cos(eigen_value * x) * \
                        numpy.exp(-D * t * (eigen_value ** 2))
                f = f * 8 / (numpy.pi ** 2)
                f_history.append(f)
        return f_history
```

```python
# set parameters
n_grids = [1, 2, 5, 100, 1000]
x_begin, x_end, D, dx = [0.0, 1.0, 1.0, 0.01]
t_grids = [0.0]
f = hw4_q1_func(n_grids, x_begin, x_end, dx, D, t_grids, 'q(a) & (b)')
```

```python
# plot
pyplot.figure(figsize=(8.0, 3.0))
pyplot.title('Initial condition value under \ndifferent series terms')
pyplot.xlabel('x')
pyplot.ylabel(r'$ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $')
pyplot.grid()
pyplot.plot(f[0], f[1], label=r'$ \sum_{n=1}^{1}$', color='C0', linestyle='-', linewidth=2)
pyplot.plot(f[0], f[2], label=r'$ \sum_{n=1}^{2}$', color='C1', linestyle='--', linewidth=2)
pyplot.plot(f[0], f[3], label=r'$ \sum_{n=1}^{5}$', color='C2', linestyle=':', linewidth=2)
pyplot.plot(f[0], f[4], label=r'$ \sum_{n=1}^{100}$', color='C3', linestyle='-.', linewidth=2)
pyplot.plot(f[0], f[5], label=r'$ \sum_{n=1}^{1000}$', color='C4', linestyle='-', linewidth=2)
pyplot.legend(loc='center left', bbox_to_anchor=(1, 0.5));
```

### Peformance of Different Number of Series Terms

As one should know from initial condition, the $T^{(0)}(x)$ is a consant, which means the value of $ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $ would be euqal to 1.

Therefore, by looking through all curves above, with the number of series terms increasing, the value is more closer to constant 1.

```python
# set parameters
n_grids = [1, 2, 5, 100, 1000]
x_begin, x_end, D, dx = [-10.0, 10.0, 1.0, 0.05]
t_grids = [0.0]
f = hw4_q1_func(n_grids, x_begin, x_end, dx, D, t_grids, 'q(a) & (b)')
```

```python
# plot
pyplot.figure(figsize=(8.0, 3.0))
pyplot.title('Initial condition value under \ndifferent series terms')
pyplot.xlabel('x')
pyplot.ylabel(r'$ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $')
pyplot.grid()
pyplot.plot(f[0], f[1], label=r'$ \sum_{n=1}^{1}$', color='C0', linestyle='-', linewidth=2)
pyplot.plot(f[0], f[2], label=r'$ \sum_{n=1}^{2}$', color='C1', linestyle='--', linewidth=2)
pyplot.plot(f[0], f[3], label=r'$ \sum_{n=1}^{5}$', color='C2', linestyle=':', linewidth=2)
pyplot.plot(f[0], f[4], label=r'$ \sum_{n=1}^{100}$', color='C3', linestyle='-.', linewidth=2)
pyplot.plot(f[0], f[5], label=r'$ \sum_{n=1}^{1000}$', color='C4', linestyle='-', linewidth=2)
pyplot.legend(loc='center left', bbox_to_anchor=(1, 0.5));
```

The period is the suumation of different amplitude and period cosine function. With the number of series increasing, the value of each point is converging to the 1.0, which is the ideal value.

## Question 1.b)

```python
# set parameters
n_grids = [100]
x_begin, x_end, D, dx = [0.0, 1.0, 1.0, 0.01]
t_grids = [0.1, 0.5, 1.0, 10.0]
f = hw4_q1_func(n_grids, x_begin, x_end, dx, D, t_grids, 'q(a) & (b)')
```

```python
# plot
pyplot.figure(figsize=(8.0, 3.0))
pyplot.title('Difference between initial and wall temperature \
\n decrease with time alone with the L at \nspatial resolution: dx = {}'.format(dx))
pyplot.xlabel('x')
pyplot.ylabel(r'$ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $')
pyplot.grid()
pyplot.plot(f[0], f[1], label=r'$t = 0.1$', color='C0', linestyle='-', linewidth=2)
pyplot.plot(f[0], f[2], label=r'$t = 0.5$', color='C1', linestyle='--', linewidth=2)
pyplot.plot(f[0], f[3], label=r'$t = 1.0$', color='C2', linestyle=':', linewidth=2)
pyplot.plot(f[0], f[4], label=r'$t = 10.0$', color='C3', linestyle='-.', linewidth=2)
pyplot.legend(loc='center left', bbox_to_anchor=(1, 0.5));
```

### The Curve Varies with Time

As one should know from initial condition, the $T^{(0)}(x) = T^{(0)} + \frac{T_L - T^{(0)}}{L}x $, which means the value of $ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $ would be simplified:

$$
\begin{equation}
\frac{T(x,0) - T_L}{T^{(0)} - T_L} = \frac{T^{(0)}}{T^{(0)} - T_L} - \frac{1}{L}x
\end{equation}
$$

which is a slope that is equal to $-\frac{1}{L}$ line, starting from $\frac{T^{(0)}}{T^{(0)} - T_L}$, which indicates the line at $t = 0.1$ is about correct.

Also, with time flows, the exponential terms will eventually goes close to zero, which indicates the temperature in the space is eventually equal to the wall temperature.

## Question 1.c)

```python
# set parameters
n_grids = [1, 2, 5, 10, 100, 1000]
x_begin, x_end, D, dx = [0.0, 1.0, 1.0, 0.01]
t_grids = [0.0]
f = hw4_q1_func(n_grids, x_begin, x_end, dx, D, t_grids, 'q(c) & (d)')
```

```python
# plot
pyplot.figure(figsize=(5.0, 5.0))
pyplot.title('Linearly varying initial condition value under \
different series terms at \nspatial resolution: dx = {}, time t = 0.0'.format(dx))
pyplot.xlabel('x')
pyplot.ylabel(r'$ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $')
pyplot.grid()
pyplot.plot(f[0], f[1], label=r'$ \sum_{n=1}^{1}$', color='C0', linestyle='-', linewidth=2)
pyplot.plot(f[0], f[2], label=r'$ \sum_{n=1}^{2}$', color='C1', linestyle='--', linewidth=2)
pyplot.plot(f[0], f[3], label=r'$ \sum_{n=1}^{5}$', color='C2', linestyle=':', linewidth=2)
pyplot.plot(f[0], f[4], label=r'$ \sum_{n=1}^{10}$', color='C3', linestyle='-.', linewidth=2)
pyplot.plot(f[0], f[5], label=r'$ \sum_{n=1}^{100}$', color='C4', linestyle='-', linewidth=2)
pyplot.plot(f[0], f[6], label=r'$ \sum_{n=1}^{1000}$', color='C5', linestyle='--', linewidth=2)
pyplot.legend(loc='center left', bbox_to_anchor=(1.3, 0.5));
```

### Peformance of Different Number of Series Terms

As one should know from initial condition, the $T^{(0)}(x) = T^{(0)} + \frac{T_L - T^{(0)}}{L}x $, which means the value of $ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $ would be simplified:

$$
\begin{equation}
\frac{T(x,0) - T_L}{T^{(0)} - T_L} = \frac{T^{(0)}}{T^{(0)} - T_L} - \frac{1}{L}x
\end{equation}
$$

which is a slope that is equal to $-\frac{1}{L}$ line, in this case L = 1, starting from $\frac{T^{(0)}}{T^{(0)} - T_L}$.

Therefore, if one could look through the curves, with the number of series terms increasing, the curve is closer to the ideal line.

## Question 1.d)

```python
# set parameters
n_grids = [100]
x_begin, x_end, D, dx = [0.0, 1.0, 1.0, 0.01]
t_grids = [0.1, 0.5, 1.0, 10.0]
f = hw4_q1_func(n_grids, x_begin, x_end, dx, D, t_grids, 'q(c) & (d)')
```

```python
# plot
pyplot.figure(figsize=(8.0, 3.0))
pyplot.title('Difference between initial and wall temperature \
\n decrease with time alone with the L at \nspatial resolution: dx = {}'.format(dx))
pyplot.xlabel('x')
pyplot.ylabel(r'$ \frac{T(x,0) - T_L}{T^{(0)} - T_L} $')
pyplot.grid()
pyplot.plot(f[0], f[1], label=r'$t = 0.1$', color='C0', linestyle='-', linewidth=2)
pyplot.plot(f[0], f[2], label=r'$t = 0.5$', color='C1', linestyle='--', linewidth=2)
pyplot.plot(f[0], f[3], label=r'$t = 1.0$', color='C2', linestyle=':', linewidth=2)
pyplot.plot(f[0], f[4], label=r'$t = 10.0$', color='C3', linestyle='-.', linewidth=2)
pyplot.legend(loc='center left', bbox_to_anchor=(1.0, 0.5));
```

### The Curve Varies with Time

Similar with Q1.b, with time flows, the exponential terms will eventually goes close to zero, which indicates the temperature in the space is eventually equal to the wall temperature.


# An Introduction to Bayesian Statistical Analysis

Before we jump in to model-building and using MCMC to do wonderful things, it is useful to understand a few of the theoretical underpinnings of the Bayesian statistical paradigm. A little theory (and I do mean a *little*) goes a long way towards being able to apply the methods correctly and effectively.

## What *is* Bayesian Statistical Analysis?

Though many of you will have taken a statistics course or two during your undergraduate (or graduate) education, most of those who have will likely not have had a course in *Bayesian* statistics. Most introductory courses, particularly for non-statisticians, still do not cover Bayesian methods at all, except perhaps to derive Bayes' formula as a trivial rearrangement of the definition of conditional probability. Even today, Bayesian courses are typically tacked onto the curriculum, rather than being integrated into the program.

In fact, Bayesian statistics is not just a particular method, or even a class of methods; it is an **entirely different paradigm for doing statistical analysis.**

> Practical methods for making inferences from data using probability models for quantities we observe and about which we wish to learn.
*-- Gelman et al. 2013*

A Bayesian model is described by parameters, and uncertainty in those parameters is described using probability distributions.

All conclusions from Bayesian statistical procedures are stated in terms of *probability statements*

This confers several benefits for scientific research, including:

- ease of interpretation
- clear summarization of uncertainty
- ability incorporate uncertainty in parent parameters
- easy to calculate summary statistics
- lego-like model-buliding

## Bayesian vs Frequentist Statistics: What's the difference?

Any statistical paradigm, Bayesian or otherwise, involves at least the following:

1. Some **unknown quantities** about which we are interested in learning or testing. We call these *parameters*.
2. Some **data** which have been observed, and hopefully contain information about (1).
3. One or more **models** that relate the data to the parameters, and is the instrument that is used to learn.

### The Frequentist World View

- The data that have been observed are considered **random**, because they are realizations of random processes, and hence will vary each time one goes to observe the system.
- Model parameters are considered **fixed**. The parameters' values are unknown, but they are fixed, and so we *condition* on them.

In mathematical notation, this implies a (very) general model of the following form:

\\[f(y | \theta)\\]

Here, the model \\(f\\) accepts data values \\(y\\) as an argument, conditional on particular values of \\(\theta\\).

Frequentist inference typically involves deriving **estimators** for the unknown parameters. Estimators are formulae that return estimates for particular estimands, as a function of data. They are selected based on some chosen optimality criterion, such as *unbiasedness*, *variance minimization*, or *efficiency*.

> For example, lets say that we have collected some data on the prevalence of encrusting byrazoans among kelp in coastal Nova Scotia. Our sample includes \\(n\\) sampled kelp plants, \\(y\\) of them being encrusted with bryazoans. A frequentist estimator of the prevalence \\(p\\) is:

>

> \\[\hat{p} = \frac{y}{n}\\]
>

> Why this particular function? Because it can be shown to be unbiased and minimum-variance.

It is important to note that new estimators need to be derived for every estimand that is introduced.

### The Bayesian World View

- Data are considered **fixed**. They used to be random, but once they were written into your lab notebook/spreadsheet/IPython notebook they do not change.
- Model parameters themselves may not be random, but Bayesians use probability distribtutions to describe their uncertainty in parameter values, and are therefore treated as **random**. In some cases, it is useful to consider parameters as having been sampled from probability distributions.

This implies the following form:

\\[p(\theta | y)\\]

This formulation used to be referred to as ***inverse probability***, because it infers from observations to parameters, or from effects to causes.

Bayesians do not seek new estimators for every estimation problem they encounter. There is only one estimator for Bayesian inference: **Bayes' Formula**.

# Computational Methods in Bayesian Analysis

The process of conducting Bayesian inference can be broken down into three general steps (Gelman *et al.* 2013):

### Step 1: Specify a probability model

As was noted above, Bayesian statistics involves using probability models to solve problems. So, the first task is to *completely specify* the model in terms of probability distributions. This includes everything: unknown parameters, data, covariates, missing data, predictions. All must be assigned some probability density.

This step involves making choices.

- what is the form of the sampling distribution of the data?
- what form best describes our uncertainty in the unknown parameters?

### Step 2: Calculate a posterior distribution

The mathematical form \\(p(\theta | y)\\) that we associated with the Bayesian approach is referred to as a **posterior distribution**.

> posterior /pos·ter·i·or/ (pos-tēr´e-er) later in time; subsequent.

Why posterior? Because it tells us what we know about the unknown \\(\theta\\) *after* having observed \\(y\\).

This posterior distribution is formulated as a function of the probability model that was specified in Step 1. Usually, we can write it down but we cannot calculate it analytically. In fact, the difficulty inherent in calculating the posterior distribution for most models of interest is perhaps the major contributing factor for the lack of widespread adoption of Bayesian methods for data analysis. Various strategies for doing so comprise this tutorial.

**But**, once the posterior distribution is calculated, you get a lot for free:

- point estimates
- credible intervals
- quantiles
- predictions

### Step 3: Check your model

Though frequently ignored in practice, it is critical that the model and its outputs be assessed before using the outputs for inference. Models are specified based on assumptions that are largely unverifiable, so the least we can do is examine the output in detail, relative to the specified model and the data that were used to fit the model.

Specifically, we must ask:

- does the model fit data?
- are the conclusions reasonable?
- are the outputs sensitive to changes in model structure?

## Example: binomial calculation

Binomial model is suitable for data that are generated from a sequence of exchangeable Bernoulli trials. These data can be summarized by $y$, the number of times the event of interest occurs, and $n$, the total number of trials. The model parameter is the expected proportion of trials that an event occurs.

$$p(Y|\theta) = \frac{n!}{y! (n-y)!} \theta^{y} (1-\theta)^{n-y}$$

where $y \in \{0, 1, \ldots, n\}$ and $p \in [0, 1]$.

To perform Bayesian inference, we require the specification of a prior distribution. A reasonable choice is a uniform prior on [0,1] which has two implications:

1. makes all probability values equally probable *a priori*
2. makes calculation of the posterior easy

The second task in performing Bayesian inference is, given a fully-specified model, to calculate a posterior distribution. As we have specified the model, we can calculate a posterior distribution up to a proportionality constant (that is, a probability distribution that is **unnormalized**):

$$P(\theta | n, y) \propto P(y | n, \theta) P(\theta) = \theta^y (1-\theta)^{n-y}$$

We can present different posterior distributions as a function of different realized data.

We can also calculate posterior estimates for $\theta$ by maximizing the unnormalized posterior using optimization.

### Exercise: posterior estimation

Write a function that returns posterior estimates of a binomial sampling model using a uniform prior on the unknown probability. Plot the posterior densities for each of the following datasets:

1. n=5, y=3
2. n=20, y=12
3. n=100, y=60
4. n=750, y=450

what type of distribution do these plots look like?

```R
n = c(5, 20, 100, 750)
y = c(3, 12, 60, 450)
```

```R
# Write your answer here
```

## Estimation, not hypothesis testing

Instead of testing, a more informative and effective approach for inference is based on **estimation** (be it frequentist or Bayesian). That is, rather than testing whether two groups are different, we instead pursue an estimate of *how different* they are, which is fundamentally more informative.

Additionally, we include an estimate of **uncertainty** associated with that difference which includes uncertainty due to our lack of knowledge of the model parameters (*epistemic uncertainty*) and uncertainty due to the inherent stochasticity of the system (*aleatory uncertainty*).

## One Group

Before we compare two groups using Bayesian analysis, let's start with an even simpler scenario: statistical inference for one group.

For this we will use Gelman et al.'s (2007) now-famous radon dataset. In this dataset the amount of the radioactive gas radon has been measured among different households in all counties of several states. Radon gas is known to be the highest cause of lung cancer in non-smokers. It is believed to be more strongly present in households containing a basement and to differ in amount present among types of soil.

>  the US EPA has set an action level of 4 pCi/L. At or above this level of radon, the EPA recommends you take corrective measures to reduce your exposure to radon gas.

Let's import the dataset:

```R
RANDOM_SEED = 20090425
```

```R
radon = read.csv('../data/radon.csv')
head(radon)
```

A data.frame: 6 × 30

	X	idnum	state	state2	stfips	zip	region	typebldg	floor	room	⋯	pcterr	adjwt	dupflag	zipflag	cntyfips	county	fips	Uppm	county_code	log_radon

	<int>	<dbl>	<fct>	<fct>	<dbl>	<int>	<dbl>	<dbl>	<dbl>	<dbl>	⋯	<dbl>	<dbl>	<dbl>	<dbl>	<dbl>	<fct>	<dbl>	<dbl>	<int>	<dbl>

1	0	5081	MN	MN	27	55735	5	1	1	3	⋯	 9.7	1146.4992	1	0	1	AITKIN	27001	0.502054	0	0.83290912

2	1	5082	MN	MN	27	55748	5	1	0	4	⋯	14.5	 471.3662	0	0	1	AITKIN	27001	0.502054	0	0.83290912

3	2	5083	MN	MN	27	55748	5	1	0	4	⋯	 9.6	 433.3167	0	0	1	AITKIN	27001	0.502054	0	1.09861229

4	3	5084	MN	MN	27	56469	5	1	0	4	⋯	24.3	 461.6237	0	0	1	AITKIN	27001	0.502054	0	0.09531018

5	4	5085	MN	MN	27	55011	3	1	0	4	⋯	13.8	 433.3167	0	0	3	ANOKA 	27003	0.428565	1	1.16315081

6	5	5086	MN	MN	27	55014	3	1	0	4	⋯	12.8	 471.3662	0	0	3	ANOKA 	27003	0.428565	1	0.95551145

Let's focus on the (log) radon levels measured in a single county (Hennepin).

Suppose we are interested in:

- whether the mean log-radon value is greater than 4 pCi/L in Hennepin county
- the probability that any randomly-chosen household in Hennepin county has a reading of greater than 4

```R
hennepin_radon = radon$log_radon[radon$county=="HENNEPIN"]
hist(hennepin_radon,prob=TRUE,c='lightblue', main="")
lines(density(hennepin_radon), xlab="log_radon")
```

> ## Software
> Today there is an array of software choices for Bayesians, including both open source software (*e.g.*, Stan, PyMC, JAGS, emcee) and commercial (*e.g.*, SAS, Stata). These examples can be replicated in any of these environments.

### The model

Recall that the first step in Bayesian inference is specifying a **full probability model** for the problem.

This consists of:

- a likelihood function(s) for the observations
- priors for all unknown quantities

The measurements look approximately normal, so let's start by assuming a normal distribution as the sampling distribution (likelihood) for the data.

$$y_i \sim N(\mu, \sigma^2)$$

(don't worry, we can evaluate this assumption)

This implies that we have 2 unknowns in the model; the mean and standard deviation of the distribution.

#### Prior choice

How do we choose distributions to use as priors for these parameters?

There are several considerations:

- discrete vs continuous values
- the support of the variable
- the available prior information

While there may likely be prior information about the distribution of radon values, we will assume no prior knowledge, and specify a **diffuse** prior for each parameter.

Since the mean can take any real value (since it is on the log scale), we will use another normal distribution here, and specify a large variance to allow the possibility of very large or very small values:

$$\mu \sim N(0, 10^2)$$

For the standard deviation, we know that the true value must be positive (no negative variances!). I will choose a uniform prior bounded from below at zero and from above at a value that is sure to be higher than any plausible value the true standard deviation (on the log scale) could take.

$$\sigma \sim U(0, 10)$$

We can encode these in an R model, using the RStan package, as follows:

```R
library(rethinking)
```

    Loading required package: rstan

    Loading required package: StanHeaders

    Loading required package: ggplot2

    rstan (Version 2.19.2, GitRev: 2e1f913d3ca3)

    For execution on a local, multicore CPU with excess RAM we recommend calling
    options(mc.cores = parallel::detectCores()).
    To avoid recompilation of unchanged Stan programs, we recommend calling
    rstan_options(auto_write = TRUE)

    Loading required package: parallel

    Loading required package: dagitty

    rethinking (Version 1.95)

    Attaching package: ‘rethinking’

    The following object is masked from ‘package:stats’:

        rstudent

```R
radon_model <- alist(
    y ~ normal(mu, sigma),
    mu ~ normal( 0 , 5),
    sigma ~ uniform(0, 5)
)
```

Before we go ahead and estimate the model paramters from the data, its a good idea to perform a **prior predictive check**. This involves sampling from the model before data are incorporated, and gives you an idea of the range of observations that would be considered reasonable within the scope of the modeling assumptions (including choice of priors). If the simnulations generate too many extreme observations relative to our expectations based on domain knowledge, then it can be an indication of problems with model formulation.

```R
prior_sample = ulam(radon_model)
```

Now, we will fit the model using **Markov chain Monte Carlo (MCMC)**, which will be covered in detail in an upcoming section. This will draw samples from the posterior distribution (which cannot be calculated exactly).

```R
samples = ulam(radon_model, data=list(y=hennepin_radon))
```

    SAMPLING FOR MODEL '226d4aba67bd114d98757bdf442bb2b7' NOW (CHAIN 1).
    Chain 1:
    Chain 1: Gradient evaluation took 2e-05 seconds
    Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.2 seconds.
    Chain 1: Adjust your expectations accordingly!
    Chain 1:
    Chain 1:
    Chain 1: Iteration:   1 / 1000 [  0%]  (Warmup)
    Chain 1: Iteration: 100 / 1000 [ 10%]  (Warmup)
    Chain 1: Iteration: 200 / 1000 [ 20%]  (Warmup)
    Chain 1: Iteration: 300 / 1000 [ 30%]  (Warmup)
    Chain 1: Iteration: 400 / 1000 [ 40%]  (Warmup)
    Chain 1: Iteration: 500 / 1000 [ 50%]  (Warmup)
    Chain 1: Iteration: 501 / 1000 [ 50%]  (Sampling)
    Chain 1: Iteration: 600 / 1000 [ 60%]  (Sampling)
    Chain 1: Iteration: 700 / 1000 [ 70%]  (Sampling)
    Chain 1: Iteration: 800 / 1000 [ 80%]  (Sampling)
    Chain 1: Iteration: 900 / 1000 [ 90%]  (Sampling)
    Chain 1: Iteration: 1000 / 1000 [100%]  (Sampling)
    Chain 1:
    Chain 1:  Elapsed Time: 0.012781 seconds (Warm-up)
    Chain 1:                0.011223 seconds (Sampling)
    Chain 1:                0.024004 seconds (Total)
    Chain 1:

```R
pairs(samples)
```

```R
sum(extract(samples@stanfit)$mu>log(4))/length(extract(samples@stanfit)$mu)
```

0.178

```R
sum(extract(samples@stanfit)$mu<log(4))/length(extract(samples@stanfit)$mu)
```

0.822

The plot shows the posterior distribution of $\mu$.

The output

    82% < 1.38629 < 18%

an estimate of the 95% posterior **credible interval**, informing us that the probability of $\mu$ being less than log(4) is 82% and the corresponding probability of being greater than log(4) is 18%.

> The posterior probability that the mean level of household radon in Henneprin County is greater than 4 pCi/L is 0.18.

### Prediction

What is the probability that a given household has a log-radon measurement larger than one? To answer this, we make use of the **posterior predictive distribution**.

$$p(z |y) = \int_{\theta} p(z |\theta) p(\theta | y) d\theta$$

where here $z$ is the predicted value and y is the data used to fit the model.

We can estimate this from the posterior samples of the parameters in the model.

```R
mus = extract(samples@stanfit)$mu
sigmas = extract(samples@stanfit)$sigma
```

```R
radon_samples = rnorm(length(mus), mus,sigmas)
```

```R
mean((radon_samples>log(4)))
```

0.458

> The posterior probability that a randomly-selected household in Henneprin County contains radon levels in excess of 4 pCi/L is about 0.46.

### Model checking

But, ***how do we know this model is any good?***

Its important to check the fit of the model, to see if its assumptions are reasonable. One way to do this is to perform **posterior predictive checks**. This involves generating simulated data using the model that you built, and comparing that data to the observed data.

One can choose a particular statistic to compare, such as tail probabilities or quartiles, but here it is useful to compare them graphically.

We already have these simulations from the previous exercise!

```R
hist(hennepin_radon,prob=TRUE,col='#00008b50', main="", xlab="log_radon")
lines(density(hennepin_radon), col='#00008b', lwd=3)
hist(radon_samples,prob=TRUE,col='#ff8c0050', add=TRUE)
lines(density(radon_samples), col='#ff8c00', lwd=3)
legend(2, .6, legend=c("simulated", "observed"),
       col=c("#ff8c00", "#00008b"), lty=1:2, cex=0.8)
```

### Prior sensitivity

Its also important to check the sensitivity of your choice of priors to the resulting inference.

Here is the same model, but with drastically different (though still uninformative) priors specified:

```R
prior_sensitivity <- alist(
    y ~ normal(mu, sigma),
    mu ~ normal(0 , 500),
    sigma ~ uniform(0, 500)
)
```

```R
sensitivity_samples = ulam(prior_sensitivity, data=list(y=hennepin_radon))
```

    SAMPLING FOR MODEL '46492a00e263301e43b35891c380be01' NOW (CHAIN 1).
    Chain 1:
    Chain 1: Gradient evaluation took 1.3e-05 seconds
    Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.13 seconds.
    Chain 1: Adjust your expectations accordingly!
    Chain 1:
    Chain 1:
    Chain 1: Iteration:   1 / 1000 [  0%]  (Warmup)
    Chain 1: Iteration: 100 / 1000 [ 10%]  (Warmup)
    Chain 1: Iteration: 200 / 1000 [ 20%]  (Warmup)
    Chain 1: Iteration: 300 / 1000 [ 30%]  (Warmup)
    Chain 1: Iteration: 400 / 1000 [ 40%]  (Warmup)
    Chain 1: Iteration: 500 / 1000 [ 50%]  (Warmup)
    Chain 1: Iteration: 501 / 1000 [ 50%]  (Sampling)
    Chain 1: Iteration: 600 / 1000 [ 60%]  (Sampling)
    Chain 1: Iteration: 700 / 1000 [ 70%]  (Sampling)
    Chain 1: Iteration: 800 / 1000 [ 80%]  (Sampling)
    Chain 1: Iteration: 900 / 1000 [ 90%]  (Sampling)
    Chain 1: Iteration: 1000 / 1000 [100%]  (Sampling)
    Chain 1:
    Chain 1:  Elapsed Time: 0.011046 seconds (Warm-up)
    Chain 1:                0.008688 seconds (Sampling)
    Chain 1:                0.019734 seconds (Total)
    Chain 1:

```R
plot_posterior = function(x,title,xlab){
    hist(x,prob=TRUE,col='#00008b50', main=title, xlab=xlab,ylim=c(0,max(density(x)$y)))
    lines(density(x), col='#00008b', lwd=3)
}
```

```R
plot_posterior(extract(sensitivity_samples@stanfit)$mu,"Sensitivity samples","log_radon")
abline(v=log(4),lwd=3,col='red')
```

Here is the original model for comparison:

```R
plot_posterior(extract(samples@stanfit)$mu,"samples","log_radon")
abline(v=log(4),lwd=3,col='red')
```

For some (typically frequentist) people this may seem like a radical departure that totally undermines the analysis. For rational (typically Bayesian) people, this is just fine; we get pretty much the same answer with wildly different priors.

## Two Groups with Continiuous Outcome

To illustrate how this Bayesian estimation approach works in practice, we will use a fictitious example from Kruschke (2012) concerning the evaluation of a clinical trial for drug evaluation. The trial aims to evaluate the efficacy of a "smart drug" that is supposed to increase intelligence by comparing IQ scores of individuals in a treatment arm (those receiving the drug) to those in a control arm (those recieving a placebo). There are 47 individuals and 42 individuals in the treatment and control arms, respectively.

```R
drug = data.frame('iq'=c(101,100,102,104,102,97,105,105,98,101,100,123,105,103,100,95,102,106,
        109,102,82,102,100,102,102,101,102,102,103,103,97,97,103,101,97,104,
        96,103,124,101,101,100,101,101,104,100,101),'group'='drug')

placebo = data.frame('iq'=c(99,101,100,101,102,100,97,101,104,101,102,102,100,105,88,101,100,
           104,100,100,100,101,102,103,97,101,101,100,101,99,101,100,100,
           101,100,99,101,100,102,99,100,99),'group'='placebo')

trial_data = rbind(drug,placebo)
```

```R
par(mfrow=c(1,2))
hist(drug$iq,col='#00008b50',main='drug')
hist(placebo$iq,col='#00008b50',main='placebo')
```

Since there appear to be extreme ("outlier") values in the data, we will choose a Student-t distribution to describe the distributions of the scores in each group. This sampling distribution adds **robustness** to the analysis, as a T distribution is less sensitive to outlier observations, relative to a normal distribution.

The three-parameter Student-t distribution allows for the specification of a mean $\mu$, a precision (inverse-variance) $\lambda$ and a degrees-of-freedom parameter $\nu$:

$$f(x|\mu,\lambda,\nu) = \frac{\Gamma(\frac{\nu + 1}{2})}{\Gamma(\frac{\nu}{2})} \left(\frac{\lambda}{\pi\nu}\right)^{\frac{1}{2}} \left[1+\frac{\lambda(x-\mu)^2}{\nu}\right]^{-\frac{\nu+1}{2}}$$

the degrees-of-freedom parameter essentially specifies the "normality" of the data, since larger values of $\nu$ make the distribution converge to a normal distribution, while small values (close to zero) result in heavier tails.

Thus, the likelihood functions of our model are specified as follows:

$$\begin{align}
y^{(drug)}_i &\sim T(\nu, \mu_1, \sigma_1) \\
y^{(placebo)}_i &\sim T(\nu, \mu_2, \sigma_2)
\end{align}$$

As a simplifying assumption, we will assume that the degree of normality $\nu$ is the same for both groups.

### Prior choice

Chosing priors is sometimes (nearly always?) tricky. However the world-leading Gelman lab and the associated Stan team have some [good guidance](https://github.com/stan-dev/stan/wiki/Prior-Choice-Recommendations).

Since the means are real-valued, we will apply normal priors. Since we know something about the population distribution of IQ values, we will center the priors at 100, and use a standard deviation that is more than wide enough to account for plausible deviations from this population mean:

$$\mu_k \sim N(100, 10^2)$$

Similarly, we will use a uniform prior for the standard deviations, with an upper bound of 20. For the degrees-of-freedom parameter $\nu$, we will use an **exponential** distribution with a mean of 30; this allocates high prior probability over the regions of the parameter that describe the range from normal to heavy-tailed data under the Student-T distribution.

```R
plot_posterior(rexp(10000,1/29),"Exponential(1/29)","")
```

```R
1/29
```

0.0344827586206897

```R
drug_model <- alist(
    mu_0 ~ normal(100, 10),
    mu_1 ~ normal(100, 10),
    sigma_0 ~ exponential(.5),
    sigma_1 ~ exponential(.5),
    nu ~ exponential(0.034),
    drug_y ~ student_t(nu,mu_1,sigma_1**-2),
    placebo_y ~ student_t(nu,mu_0,sigma_0**-2)
)
```

```R
drug_trace = ulam(drug_model, data=list(drug_y=drug$iq,placebo_y=placebo$iq))
```

    SAMPLING FOR MODEL '7b25d4779f5253db19999d9ce570c5d5' NOW (CHAIN 1).
    Chain 1:
    Chain 1: Gradient evaluation took 2.6e-05 seconds
    Chain 1: 1000 transitions using 10 leapfrog steps per transition would take 0.26 seconds.
    Chain 1: Adjust your expectations accordingly!
    Chain 1:
    Chain 1:
    Chain 1: Iteration:   1 / 1000 [  0%]  (Warmup)
    Chain 1: Iteration: 100 / 1000 [ 10%]  (Warmup)
    Chain 1: Iteration: 200 / 1000 [ 20%]  (Warmup)
    Chain 1: Iteration: 300 / 1000 [ 30%]  (Warmup)
    Chain 1: Iteration: 400 / 1000 [ 40%]  (Warmup)
    Chain 1: Iteration: 500 / 1000 [ 50%]  (Warmup)
    Chain 1: Iteration: 501 / 1000 [ 50%]  (Sampling)
    Chain 1: Iteration: 600 / 1000 [ 60%]  (Sampling)
    Chain 1: Iteration: 700 / 1000 [ 70%]  (Sampling)
    Chain 1: Iteration: 800 / 1000 [ 80%]  (Sampling)
    Chain 1: Iteration: 900 / 1000 [ 90%]  (Sampling)
    Chain 1: Iteration: 1000 / 1000 [100%]  (Sampling)
    Chain 1:
    Chain 1:  Elapsed Time: 0.064216 seconds (Warm-up)
    Chain 1:                0.031958 seconds (Sampling)
    Chain 1:                0.096174 seconds (Total)
    Chain 1:

Now that the model is fully sampled, we can turn our attention to tracking the posterior quantities of interest. Namely, we can calculate the difference in means between the drug and placebo groups.

As a joint measure of the groups, we will also estimate the "effect size", which is the difference in means scaled by the pooled estimates of standard deviation. This quantity can be harder to interpret, since it is no longer in the same units as our data, but it is a function of all four estimated parameters.

```R
sqrt(5)
```

2.23606797749979

```R
diff_of_means = extract(drug_trace@stanfit)$mu_1-extract(drug_trace@stanfit)$mu_0
effect_size = diff_of_means/sqrt((extract(drug_trace@stanfit)$sigma_1**2+extract(drug_trace@stanfit)$sigma_0**2)/2)
```

```R
options(repr.plot.width=10, repr.plot.height=5)
par(mfrow=c(1,2))
plot_posterior(diff_of_means,"Difference of means","")
plot_posterior(effect_size,"Effect size","")
```

```R
sum(diff_of_means>0)/length(diff_of_means)
```

0.988

> The posterior probability that the mean IQ of subjects in the treatment group is greater than that of the control group is 0.99.

## Exercise: Two Groups with Binary Outcome

Now that we have seen how to generalize normally-distributed data to another distribution, try our hand with another data type. Binary outcomes are common in biological research:

- survival/death
- true/false
- presence/absence
- positive/negative

In practice, binary outcomes are encoded as ones (for event occurrences) and zeros (for non-occurrence). A single binary variable is distributed as a **Bernoulli** random variable:

$$f(x \mid p) = p^{x} (1-p)^{1-x}$$

In terms of inference, we are typically interested in whether $p$ is larger or smaller in one group relative to another.

To demonstrate the comparison of two groups with binary outcomes using Bayesian inference, we will use a sample coral reef dataset. Data on 102 reef transects relating to herbivory and coral-algal phase shifts were collected from three reefs on the Great Barrier Reef by the Australian Institute of Marine Science. Of interest is the relationship between a phase shift being present (PS) and the presence of various functional groups of herbivorous fish.

Results from the original study can be found in [Cheal et al. 2010](https://link.springer.com/article/10.1007/s00338-010-0661-y)

```R
psdata = read.csv('../data/psdata.csv')
head(psdata)
```

A data.frame: 6 × 12

	reef	visit	site	acaneam	acanalgae	acandeter	scaream	sigalgae	total_herbs	coral	algae	PS

	<fct>	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<int>	<dbl>	<dbl>	<int>

1	FITZROY ISLAND REEF	3	1	0	0	 3	13	12	28	32.74940	0.000000	0

2	FITZROY ISLAND REEF	3	2	2	7	 3	29	 9	50	33.20631	0.137931	0

3	FITZROY ISLAND REEF	3	3	1	1	 3	11	 7	23	25.95713	0.000000	0

4	FITZROY ISLAND REEF	4	1	1	2	 1	24	 4	32	40.04990	0.000000	0

5	FITZROY ISLAND REEF	4	2	1	1	 0	 9	16	27	36.83819	0.000000	0

6	FITZROY ISLAND REEF	4	3	3	1	12	26	11	53	31.95169	0.100000	0

To demonstrate binary data analysis, we will try to estimate the difference in the probability of a phase-shift for sites having less than the average number of total herbivores, relative to sites with more than the average number.

```R
High_herbs = ((psdata$total_herbs>mean(psdata$total_herbs))*1)
```

Fit a model that evaluates the association of a phase-shift with the probability of having greater or fewer than the average number of herbivorous fish per transect.

```R
# Write your answer here
```

---

# References

Gelman, Andrew, John B. Carlin, Hal S. Stern, David B. Dunson, Aki Vehtari, and Donald B. Rubin. 2013. Bayesian Data Analysis, Third Edition. CRC Press.

Pilon, Cam-Davidson. [Probabilistic Programming and Bayesian Methods for Hackers](http://camdavidsonpilon.github.io/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/)


## Tutorial 3: Poisson Equation (Advanced)

In this tutorial, we will try to validate an advanced implementation of the problem described by the previous tutorial, which should be looked at first. The main objective is to check some properties of iterative methods applied to the Poisson problem, specifically those described in the chapter 4.2 of $\mathrm{LeVeque^{[1]}}$.

$\mathrm{^{[1]}}$ __LeVeque, R.J., 2007. Finite difference methods for ordinary and partial differential equations: steady-state and time-dependent problems (Vol. 98). Siam.__

The spectral radius $\rho$ of the iteration matrix is a value that measures how much the error decays after one iteration for the worst scenario, which happens for the lowest frequencies of the solution. Given an initial error at the nth step $e^{[n]}$, the error after one iteration $e^{[n + 1]}$ is bounded by $\rho\ e^{[n]}$. From section 4.2 of the book "Finite Difference Methods for Ordinary and Partial Differential Equations", we can get the formula of the spectral radius $\rho$ in the case of some common implicit methods:

$$\textrm{Jacobi:}\ \rho_h = \cos{(\pi h)} = 1 - \frac{\pi^2 h^2}{2} + O(h^4)$$

$$\textrm{Gauss-Seidel:}\ \rho_h = 1 - \pi^2 h^2 + O(h^4)$$

$$\textrm{SOR using}\ \omega_{opt} = \frac{2}{1 + \sin{(\pi h)}}\textrm{:}\ \rho_h = 1 - 2 \pi h + O(h^2)$$

Since $e^{[n + 1]} = \rho\ e^{[n]}$, given an error reduction factor $\epsilon$ such that we want $e^{[n]} \leq \epsilon\ e^{[0]}$, then the required number of iterations $k$ to guarantee that precision respects the equation:

$$\rho^k \leq \epsilon \implies k \geq \log_{\rho} \epsilon = \frac{\log \epsilon}{\log \rho}$$

```python
from devito import Grid
from math import log, ceil, cos, pi

shape = (50, 50)   # Grid size in number of points
extent = (1., 1.)  # Grid domain limits -> [0,1] x [0,1]

epsilon = 0.1
space_order = 2

grid = Grid(shape = shape, extent = extent) # The spatial grid
hx, hy = grid.spacing # The calculated spatial steps (dx and dy)

assert abs(hx - hy) < 0.0001, \
    "Spatial spacing must be equal in both dimensions for analysis (%e, %e)" % (hx, hy)

h = hx # This could be h = hy or h = (hx + hy) / 2

rho = cos(pi * h) # The spectral radius of the iterative matrix
k = ceil(log(epsilon, rho)) # The required number of iterations

print("Grid spacing is %f" % h)
print("Spectral radius of the iteration matrix is %f for the Jacobi method" % rho)
print("Number of iterations to guarantee precision for the worst frequence is %d" % k)
```

    Grid spacing is 0.020408
    Spectral radius of the iteration matrix is 0.997945 for the Jacobi method
    Number of iterations to guarantee precision for the worst frequence is 1120

One of the main disadvantages of using the Jacobi method is that is has very slow convergence rate for the lowest frequency of the solution. For the problem above $\rho \approx 0.998$, which means that the upper bound for the error decreases by only 0.2% every iteration.

Also note that, some of the time, depending on the initial guess, the number of iterations executed is not enough to guarantee the precision. A very common residual criteria is to stop whenever the relative error is below a limit $\epsilon$:

$$\mathrm{stop\ if\ } \frac{||p_{n+1}-p_n||_2}{||p_n||_2} < \epsilon$$.

But if that condition is used, there is no guarantee that the number $k$ of required iterations will be executed. This relation depends on the value of the condition number $\kappa_2$ of the eigenvector matrix using norm-2, presented in the section 4.2.1 of LeVeque. This value measures how the evolution of the residual compares with the evolution of the error.

Let's now build an Operator that will solve the same problem as the last tutorial, but being able to choose between Jacobi, Gauss-Seidel or SOR and using the relative residual as the stop condition, instead of the absolute one. Let's also not save all the iterations, but use alternating buffers to save memory, so that we can vary $h$ into a wider range, without having to use swap memory. Let's also enable parallelism with OpenMP and enable aggressive optimizations to make our tests run faster.

```python
from devito import configuration

configuration['loop-splitting'] = False
configuration['language'] = 'openmp'
```

```python
# Choose your method
#method = 'jacobi'
#method = 'gauss'
method = 'sor'
```

```python
from devito import Dimension, ConditionalDimension, Eq, Function, Inc, Operator, TimeFunction, solve
from numpy import int32
from sympy import sin as Sin

# The symbolic representation of the Poisson function
p = TimeFunction(name = 'p', grid = grid, space_order = space_order)

# The symbolic representation of the Poisson source/forcing term
f = Function(name = 'f', grid = grid, space_order = space_order)

# The symbolic representation of the absolute residual ||p(n + Δn) - p(n)||
residual = Function(name = 'residual', shape = (1,), dimensions = (Dimension(name = 'residual_dim'),))

# The symbolic representation of the normalizing term ||p(n + Δn)||
pnorm = Function(name = 'pnorm', shape = (1,), dimensions = (Dimension(name = 'pnorm_dim'),))

# The variable that will store which iteration is currently being executed
niter = Function(name = 'niter', shape = (1,), dimensions = (Dimension(name = 'niter_dim'),),
                 dtype = int32)

n = grid.stepping_dim  # The dimension of iteration
x, y = grid.dimensions # The spatial dimensions

# Temporary dimensions used to separate the residual calculation from the PDE equation
# Note: Since we use a Dimension instead of DefaultDimension, then we have to later set
#       xt_M and yt_M manually when calling the operator
xtemp = Dimension(name = 'xt')
ytemp = Dimension(name = 'yt')

xy_to_xtyt = [(x, xtemp), (y, ytemp)] # Used to convert an equation written in (x,y) to (xtemp, ytemp)

# Epsilon is now a variable so that it can be dynamically changed
epsilon = Function(name = 'epsilon', shape = (1,), dimensions = (Dimension(name = 'epsilon_dim'),))

cn = ConditionalDimension(name = 'cn', parent = n.parent,
                          condition = (residual[0] >= pnorm[0] * epsilon[0]), brk = True)

# The poisson PDE
pde = Eq(p.laplace, f, subdomain = grid.interior)

# One iteration of the Jacobi method
jacobiEq = solve(pde, p)

# One iteration of the Gauss-Seidel method
# Note: Gauss-Seidel equation is the same as the Jacobi one, but changing p(n) to p(n + Δn),
#       thus forcing one iteration to operate using only p(n + Δn), which is a single array
gaussEq = jacobiEq.subs(n, n + n.spacing)

if method == 'jacobi':
    iterEq = Eq(p.forward, jacobiEq)
elif method == 'gauss':
    iterEq = Eq(p.forward, gaussEq)
elif method == 'sor':
    # This is the equation for the optimal omega
    #omega = 2 / (1 + pi * Sin(grid.spacing_symbols[0])) # Exact omega
    omega = 2 - 2 * pi * grid.spacing_symbols[0] # Approximated omega

    # One iteration of the SOR method
    sorEq = omega * gaussEq + (1 - omega) * p.forward

    iterEq = Eq(p.forward, sorEq)

# Equation to calculate ||p(n + Δn) - p(n)||
resEq = [Eq(residual[0], 0, implicit_dims = [n]),
         Inc(residual[0], (p.forward - p).subs(xy_to_xtyt) ** 2)]

# Equation to calculate ||p(n + Δn)||
pnormEq = [Eq(pnorm[0], 0, implicit_dims = [n]),
           Inc(pnorm[0], p.subs(xy_to_xtyt) ** 2)]

# Equation to count, so we can know in what iteration it stopped
cntEq = Eq(niter[0], cn)

# Dirichlet boundary conditions
boundary = [Eq(p[n + 1, 0, y], 0.)]
boundary += [Eq(p[n + 1, x.symbolic_max, y], 0.)]
boundary += [Eq(p[n + 1, x, 0], 0.)]
boundary += [Eq(p[n + 1, x, y.symbolic_max], 0.)]

if method == 'jacobi':
    eqs = [iterEq] + boundary + resEq + pnormEq + [cntEq]
else:
    # Save last iteration so we can calculate the residual
    saveEq = Eq(p.forward, p)

    eqs = [saveEq, iterEq] + boundary + resEq + pnormEq + [cntEq]

op = Operator(eqs)
```

To validate the execution of the developed implicit method, we will execute it using various values for the spatial step $h = h_x = h_y$. This will be done to check the approximately linear evolution of the relative error ratio $r_h$ when reducing the value of the spatial step $h$ linearly, for low values of $h \rightarrow 0$.

Here is a demonstration for the two properties using the Gauss-Seidel spectral radius. The demonstration is almost identical for the Jacobi and SOR, the latter having a decay of $\alpha$ instead of $\alpha^2$:

$$r_h = \frac{e^{[n + 1]} - e^{[n]}}{e^{[n]}} = \rho - 1 \approx - \pi^2 h^2\ \textrm{when}\ h \rightarrow 0 \implies \frac{r_h}{r_{\frac{h}{\alpha}}} \approx \frac{- \pi^2 h^2}{-\pi^2 (\frac{h}{\alpha})^2} = \alpha^2 \implies r_{\frac{h}{\alpha}} \approx \frac{r_h}{\alpha^2} \implies$$

$$ \implies  \textrm{The relative error when dividing the spacing by a factor}\ \alpha \textrm{ decays by a factor of } \alpha^2$$

$$k_h = \frac{\log \epsilon}{\log \rho_h} \implies \frac{k_{\frac{h}{\alpha}}}{k_h} = \frac{\log (\rho_h)}{\log (\rho_{\frac{h}{\alpha}})} \approx \frac{\log (1 - \pi^2 h^2)}{\log (1 - \pi^2 (\frac{h}{\alpha})^2)} \approx \frac{- \pi^2 h^2}{-\pi^2 (\frac{h}{\alpha})^2} = \alpha^2 \implies k_{\frac{h}{\alpha}} \approx k_h \alpha^2 \implies$$

$$ \implies \textrm{The number of required iterations increases approximately by } \alpha^2 \textrm{ when the number of points increases by a factor of}\ \alpha$$

__Note__: The taylor expansion for $\log (1 - h)$ around $h = 0$ is $\log (1 - h) = -h - \frac{h^2}{2} - \frac{h^3}{3} - O(h^4)$

```python
# If "exec_all" is true, then execute exactly n% of the required k iterations
if method == 'sor':
    exec_all = True
else:
    exec_all = False
npercent = 1

epsilon_ = 0.0000001
nmax = 10000 # This is the maximum iterations when exec_all = False
```

```python
from warnings import warn

msg = "This cell usually takes minutes to properly execute. Try reducing 'ns' for a faster execution."
warn(msg, RuntimeWarning)

from numpy import cos, inf, round, linspace, meshgrid, sin
from numpy.linalg import norm

# Useful: h[i] = extent[i] / (shape[i] - 1) => shape[i] = extent[i] / h[i] + 1

extent = 1.0

# Test parameters
h0 = 0.25 # First spatial step
alpha = 2 # Linear factor to decrease h
ns = 8    # Number of spacings to be calculated

H = [h0 * alpha ** -i for i in range(ns)]                   # List of spacings to be tested
shapes = [int(round(extent / H[i]) + 1) for i in range(ns)] # List of shapes for the grid

if exec_all:
    epsilon.data[0] = 0.0
else:
    epsilon.data[0] = epsilon_

hs = []     # List of actual spacings used
niters = [] # List of performed iterations
rs = []     # List of relative errors
ks = []     # List of required number of iterations

# Auxiliary function to calculate the absolute error in an iteration
def absolute_error(p, it, sol):
    return norm(sol - p.data[it, :, :], ord = 2)

extent = (extent, extent)
for shape in shapes:
    shape = (shape, shape)
    grid = Grid(shape = shape, extent = extent)
    h = grid.spacing[0]

    pnorm.data[0] = 0
    residual.data[0] = 0
    niter.data[0] = 0

    p = TimeFunction(name = 'p', grid = grid, space_order = space_order)
    p.data[:] = 0 # Initial data is null instead of random for consistent results if executing more than once

    residuals = Function(name = 'residuals', shape = grid.shape, dimensions = [xtemp, ytemp], space_order = space_order)

    f = Function(name = 'f', grid = grid)
    H = linspace(0., 1., shape[0], endpoint = True)
    X, Y = meshgrid(H, H, sparse = True)

    f.data[:] = 4 * pi ** 2 * sin(2 * pi * X) * (2 * cos(2 * pi * Y) - 1)
    solution = sin(2 * pi * X) * (1 - cos(2 * pi * Y))

    if method == "jacobi":
        rho = 1 - pi ** 2 * h ** 2 / 2
    elif method == "gauss":
        rho = 1 - pi ** 2 * h ** 2
    elif method == "sor":
        rho = 1 - 2 * pi * h
    if rho < 0:
        rho = epsilon_
    k = ceil(log(epsilon_) / log(rho))

    if exec_all:
        nmax = ceil(k * npercent)

    # Note: h_x and h_y must be manually updated since they are present in the equations of the operator
    op(time_M = nmax, p = p, f = f, h_x = h, h_y = h, xt_M = p.shape[1] - 1, yt_M = p.shape[2] - 1)
    last_iter = niter.data[0]

    En = absolute_error(p, last_iter - 1, solution)
    rh = abs(absolute_error(p, last_iter, solution) - En) / En

    hs.append(h)
    niters.append(last_iter)
    rs.append(rh)
    ks.append(k)
```

    <ipython-input-6-ad7f82d9663b>:4: RuntimeWarning: This cell usually takes minutes to properly execute. Try reducing 'ns' for a faster execution.
      warn(msg, RuntimeWarning)
    Operator `Kernel` ran in 0.08 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.01 s
    Operator `Kernel` ran in 0.07 s
    Operator `Kernel` ran in 0.41 s
    Operator `Kernel` ran in 3.81 s
    Operator `Kernel` ran in 29.79 s

```python
from numpy import array, exp, full, log, polyfit
import matplotlib.pyplot as plt

# Cast some of the lists to numpy array
rs = array(rs)
niters = array(niters)

ratios = rs[:-1] / rs[1:]

# Plot the relative error ratio
# Note: This plot should approach "alpha ** 2" for Jacobi and Gauss-Seidel and "alpha" for SOR
plt.figure()
plt.title("Blue Plot -> Orange Plot")
plt.grid()
plt.xlabel("Test Number")
plt.ylabel("Relative Error Ratio")
X = range(1, ns)
plt.xticks(X)
plt.plot(X, ratios)
if method == 'sor':
    plt.plot(X, full(ns-1, alpha))
else:
    plt.plot(X, full(ns-1, alpha ** 2))

# Plot the number of required iterations
# Note: This plot should be approximately line (when exec_all = False), especially for smaller h
plt.figure()
plt.title("Blue Plot -> Straight Line = Orange Plot")
plt.grid()
plt.xlabel("Test Number")
plt.ylabel("Iterations Executed")
plt.yscale('log')
X = range(ns)
plt.xticks(X)
plt.plot(X, niters)
a, b = polyfit(X[2:], log(niters[2:]), 1)
plt.plot(X, exp(a * X + b))

None
```

If you try to plot the last approximated solution, even though the basic "form" of the solution already shows up, you will see that it is not even close to the real solution, especially for the Jacobi and Gauss-Seidel methods. That happens because, as $h \rightarrow 0$, the number of iterations required for the method to converge increases by the order of $h^2$. This means that when $h = 0.001$, for example, the number of required iterations to guarantee precision is on the order of $\frac{1}{h^2} = 10^6$. The convergence is very slow in this case, so $10^6$ is much higher than the actual number of iterations executed when using the relative error stop criteria.

Even though the real solution is still very badly approximated for small $h$, the properties deduced in this notebook are independent of the current iteration index and also independent of the actual distance between the real and approximated solution.

```python
from numpy import max, min

if method != "sor":
    print("The absolute error of the last approximated solution is %f, which is still very large\n" % En)
    print("The maximum and minimum value for the last approximated solution are %f and %f." % (max(p.data[0,:]),
                                                                                min(p.data[0,:])))
    print("These values should converge to ±2, but are still far away\n")

    print("Even though %d iterations should be done for guaranteed convergence, only %d were executed" % (ks[-1],
                                                                                niter.data[0]))
```

__Warning__: Be aware that we are plotting large data when using the code below, which will be the last approximated solution with the smallest $h$. As you can see, the solution has the basic "form", but the actual values are still far from the real solution.