# Infection Modeling

The [Numberphile video](https://www.youtube.com/watch?v=k6nLfCbAzgo) on the corona curve inspired me to try solve these ODEs in Python and make it interactive.

The original video shows the SIR model, a ["compartmental model"](https://en.wikipedia.org/wiki/Compartmental_models_in_epidemiology) for epidemology, deterministically modeling:

- **S**usceptible to the disease
- **I**nfectious with the disease
- **R**emoved from the disease and immune

This is done by solving these three ordinary differential equations (ODEs) and luckily SciPy has just the tools for this. The change of susceptible population is dependent on the transmission coefficient $\beta$, the number of infected and suscptible people $(I, S)$, as well as the population size $N$:

$
\frac{dS}{dt} = - \frac{\beta I S}{N},
$

the change of infected people is given by the influx of the infected people from the susceptible group from before and the transfer of infected people multiplied by the recovery factor $\gamma$:

$
\frac{dI}{dt} = \frac{\beta I S}{N}- \gamma I,
$

the change of recovered people is simply given by the influx of infected people modified by the recovery factor from before:

$
\frac{dR}{dt} = \gamma I,
$

We can see that eventually all people will have gone through the infected group to the recovered group. This model simply depends on the assumption that the population N never changes (steady state).

```
import numpy as np
from scipy.integrate import solve_ivp

import matplotlib.pyplot as plt
from ipywidgets import interact, interactive, FloatSlider
import ipywidgets as widgets

%matplotlib inline
```

These following start values show a simple percentage ($N=100$) and the values for the novel Corona virus that causes COVID-19.

Additionally, we'll define some time we want to analyze and the sampling of time for our simulation. Coarser is faster but finer is more stable.

```
N = 100

transmisision_rate = 3.2
recovery_rate = 0.23
waning_rate = 0.00

max_time = 5e1
dt = 1e-2
```

Then we need some initially infected people from our population.

```
# Population stats

def SIR_init(N=1, infected_ratio=0.01):
    """ Initialize S, I, R
    Initial Susceptible, Infected and Removed Population

    N (int): number of people in population
    infected_ratio (float): ratio of initially infected people
    """
    I = (infected_ratio * N)
    S = ((1 - infected_ratio) * N)
    R = 0
    return S, I, R

S_ini, I_ini, R_ini = SIR_init(N)
print(SIR_init(N))
```

    (99.0, 1.0, 0)

# Solve Differential Equations

We use a slightly modified version of the SIR formulation. Influenza and influenza-like viruses (possibly the corona virus?) can cause a loss of immunity. That means the SIRS model, which has a rate that changes from recovered to susceptible, is possibly better suited to model this virus. That changes the aforementioned ODEs to:

$
\begin{align}
& \frac{dS}{dt} = - \frac{\beta I S}{N} + \xi R \\[6pt]
& \frac{dI}{dt} = \frac{\beta I S}{N}- \gamma I \\[6pt]
& \frac{dR}{dt} = \gamma I - \xi R
\end{align}
$

adding a change to $\frac{dR}{dt}$ of $R$ modified by the waning ratio $\xi$ that is added to the susceptible population $S$. As long as we keep $\xi=0$ the SIRS model is equivalent to the SIR model.

```

def dSIR_dt(t, SIR, N, transmisision_rate, recovery_rate, waning_rate=0):
#def dSIR_dt(t, S, I, R, N, transmisision_rate, recovery_rate, waning_rate=0):
    S, I, R = SIR

    infected = transmisision_rate * S * I / N
    removed = recovery_rate * I
    waned = waning_rate * R

    S_new = - infected + waned
    I_new = infected - removed
    R_new = removed - waned
    return (S_new, I_new, R_new)

```

The Scipy package provides us with several ODE solvers for "Initial Value Provlems", which is our kind of problem. These solvers are neatly packed in [solve_ivp](https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.solve_ivp.html#scipy.integrate.solve_ivp).

This function can be used to portray all the data we want in the entire time we simulate. However, it can solve specific points as well without providing that dense solution. Pretty neat:

```
def solve_SIR(N, max_time, dt, transmisision_rate, recovery_rate, waning_rate):
    t = np.arange(0, max_time, dt)
    SIR = solve_ivp(dSIR_dt,
                    [0, max_time],
                    SIR_init(N),
                    args=(N, transmisision_rate, recovery_rate, waning_rate),
                    dense_output=True)
    return t, SIR.sol(t)
```

# Interactive Plot

A nice big chunk of code to plot three lines. But now we can interactively explore the data and how we can flatten the curve.

Social Distancing reduces the transmission coefficient, try it out!

```
t, SIR = solve_SIR(N, max_time, dt, transmisision_rate, recovery_rate, waning_rate)

fig, ax = plt.subplots(figsize=(10,5))
S_plot, = ax.plot(t, SIR[0],  label='Susceptible')
I_plot, = ax.plot(t, SIR[1],  label='Infectious')
R_plot, = ax.plot(t, SIR[2],  label='Removed')
_ = ax.legend(loc='best')
_ = ax.set_xlabel('Time')
_ = ax.set_ylabel('Population')
_ = ax.set_title('Deterministic SIR(S) model')
r_o = plt.text(40, 0, f'$R_O$={transmisision_rate/recovery_rate:.2f}', fontsize=10)
plt.close()

def plot_SIR(transmission=3.2, recovery=.23, wane=.05):
    t, SIR = solve_SIR(N, max_time, dt, transmission, recovery, wane)
    S_plot.set_ydata(SIR[0])
    I_plot.set_ydata(SIR[1])
    R_plot.set_ydata(SIR[2])
    r_o.set_text(f'$R_O$={transmission/recovery:.2f}')
    fig.canvas.draw()
    display(fig)

style = {'description_width': 'initial'}
interactive_plot = interact(plot_SIR,
         transmission=FloatSlider(value=3.2, min=0, max=5, step=1e-2, continuous_update=False, description="Transmission Rate", style=style),
         recovery=FloatSlider(value=.23, min=0, max=1, step=1e-2, continuous_update=False, description="Recovery Rate", style=style),
         wane=FloatSlider(value=0, min=0, max=1, step=1e-2, continuous_update=False, description="Immunity Loss", style=style))

```

    interactive(children=(FloatSlider(value=3.2, continuous_update=False, description='Transmission Rate', max=5.0…

```

```


```python
from resources.workspace import *
%matplotlib inline
```

## Dynamical systems
are systems (sets of equations) whose variables evolve in time (the equations contains time derivatives). As a branch of mathematics, its theory is mainly concerned with understanding the behaviour of solutions (trajectories) of the systems.

## Chaos
is also known as the butterfly effect: "a buttefly that flaps its wings in Brazil can 'cause' a hurricane in Texas".

As opposed to the opinions of Descartes/Newton/Laplace, chaos effectively means that even in a deterministic (non-stochastic) universe, we can only predict "so far" into the future. This will be illustrated below using two toy-model dynamical systems made by Edward Lorenz.

---
## The Lorenz (1963) attractor

The [Lorenz-63 dynamical system](resources/DA_intro.pdf#page=22) can be derived as an extreme simplification of *Rayleigh-Bénard convection*: fluid circulation in a shallow layer of fluid uniformly heated (cooled) from below (above).
This produces the following 3 *coupled* ordinary differential equations (ODE):

$$
\begin{aligned}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{aligned}
$$

where the "dot" represents the time derivative, $\frac{d}{dt}$. The state vector is $\mathbf{x} = (x,y,z)$, and the parameters are typically set to

```python
SIGMA = 10.0
BETA  = 8/3
RHO   = 28.0
```

The ODEs can be coded as follows

```python
def dxdt(xyz, t0, sigma, beta, rho):
    """Compute the time-derivative of the Lorenz-63 system."""
    x, y, z = xyz
    return [
        sigma * (y - x),
        x * (rho - z) - y,
        x * y - beta * z
    ]
```

#### Numerical integration to compute the trajectories

Below is a function to numerically **integrate** the ODEs and **plot** the solutions.

```python
from scipy.integrate import odeint # integrator

output_63 = [None]

@interact(         sigma=(0.,50), beta=(0.,5), rho=(0.,50), N=(0,50), eps=(0.01,1), T=(0.,30))
def animate_lorenz(sigma=SIGMA,   beta=BETA,   rho=RHO    , N=2,      eps=0.01,     T=1.0):

    # Initial conditions: perturbations around some "proto" state
    seed(1)
    x0_proto = array([-6.1, 1.2, 32.5])
    x0 = x0_proto + eps*randn((N, 3))

    # Compute trajectories
    tt = linspace(0, T, int(100*T)+1)             # Time sequence for trajectory
    dd = lambda x,t: dxdt(x,t, sigma,beta,rho)    # Define dxdt(x,t) with fixed params.
    xx = array([odeint(dd, xn, tt) for xn in x0]) # Integrate

    # PLOTTING
    ax = plt.figure(figsize=(10,5)).add_subplot(111, projection='3d')
    ax.axis('off')
    colors = plt.cm.jet(linspace(0,1,N))
    for i in range(N):
        ax.plot(*(xx[i,:,:].T),'-'  ,c=colors[i])
        #ax.scatter3D(*xx[i,0 ,:],s=20,c=colors[i],marker='<')
        ax.scatter3D(*xx[i,-1,:],s=40,c=colors[i])

    output_63[0] = xx
```

**Exc 4.2**:
* Move `T` (use your arrow keys). What does it control?
* Set `T` to something small; move the sliders for `N` and `eps`. What do they control?
* Visually investigate the system's (i.e. the trajectories') sensititivy to initial conditions by moving `T`, `N` and `eps`. Very roughtly, estimate its predictability (i.e. how far into the future one can forecasts for a fixed `eps` and a fixed skill level)?

### Averages

**Exc 4.8*:** Slide `N` and `T` to their upper bounds. Execute the code cell below. It computes the average location of  the $i$-th component of the state in two ways.  Do you think the histograms actually approximate the same distribution? If so, then the system is called [ergodic](https://en.wikipedia.org/wiki/Ergodic_theory#Ergodic_theorems). In that case, does it matter if one computes statistics (over the system dynamics) by using several short experiment runs or one long run?

```python
xx = output_63[0][:,:,0] # state component index 0 (must be 0,1,2)

plt.hist(xx[:,-1]   ,normed=1,label="ensemble dist.",alpha=1.0) # -1: last time
plt.hist(xx[-1,:]   ,normed=1,label="temporal dist.",alpha=0.5) # -1: last ensemble member
#plt.hist(xx.ravel(),normed=1,label="total    distribution",alpha=0.5)
plt.legend();
```

---

## The "Lorenz-95" model

The Lorenz-96 system
is a "1D" model, designed to simulate atmospheric convection.  Each state variable $\mathbf{x}_i$ can be considered some atmospheric quantity at grid point at a fixed lattitude of the earth.  The system
is given by the coupled set of ODEs,
$$
\frac{d \mathbf{x}_i}{dt} = (\mathbf{x}_{i+1} − \mathbf{x}_{i-2}) \mathbf{x}_{i-1} − \mathbf{x}_i + F
\, ,
\quad \quad i \in \{1,\ldots,m\}
\, ,
$$
where the subscript indices apply periodically.

This model is not derived from physics but has similar characterisics, such as

 there is external forcing, determined by a parameter $F$;

 there is internal dissipation, emulated by the linear term;

 there is energy-conserving advection, emulated by quadratic terms.

[Further description](resources/DA_intro.pdf#page=23).

**Exc 4.10:** Show that the "total energy" $\sum_{i=1}^{m} \mathbf{x}_i^2$ is preserved by the quadratic terms in the ODE.

```python
show_answer("Hint: Lorenz energy")
```

Hint: what's its time-derivative?

```python
show_answer("Lorenz energy")
```

\begin{align}
\frac{d}{dt}
\sum_i
x_i^2
&=
2 \sum_i
x_i \dot{x}_i
\end{align}

Next, insert the quadratic terms from the ODE,
$
\dot x_i = (x_{i+1} − x_{i-2}) x_{i-1}
\, .
$

Finally, apply the periodicity of the indices.

The model is animated below.

```python
# For all i, any n: s(x,n) := x[i+n], circularly.
def s(x,n):
    return np.roll(x,-n)

output_95 = [None]

def animate_lorenz_95(m=40,Force=8.0,eps=0.01,T=0):
    # Initial conditions: perturbations
    x0 = zeros(m)
    x0[0] = eps

    def dxdt(x,t):
        return (s(x,1)-s(x,-2))*s(x,-1) - x + Force

    tt = linspace(0, T, int(40*T)+1)
    xx = odeint(lambda x,t: dxdt(x,t), x0, tt)
    output_95[0] = xx

    plt.figure(figsize=(7,4))

    # Plot last only
    #plt.plot(xx[-1],'b')

    # Plot multiple
    Lag = 8
    colors = plt.cm.cubehelix(0.1+0.6*linspace(0,1,Lag))
    for k in range(Lag,0,-1):
        plt.plot(xx[max(0,len(xx)-k)],c=colors[Lag-k])

    plt.ylim(-10,20)
    plt.show()

interact(animate_lorenz_95,eps=(0.01,3,0.1),T=(0.05,40,0.05),Force=(0,40,1),m=(5,60,1));
```

    Widget Javascript not detected.  It may not be installed or enabled properly.

**Exc 4.12:** Under which settings of the force `F` is the system chaotic?

---
## Error/perturbation dynamics

**Exc 4.14*:** Suppose $x(t)$ and $z(t)$ are "twins": they evolve according to the same law $f$:

$$\frac{dx}{dt} = f(x) \\ \frac{dz}{dt} = f(z) \, .$$

* a) Define the "error": $\varepsilon(t) = x(t) - z(t)$.
Suppose $z(0)$ is close to $x(0)$.
Let $F = \frac{df}{dx}(x(t))$.
Show that the error evolves according to the ordinary differential equation (ODE)
$$\frac{d \varepsilon}{dt} \approx F \varepsilon \, .$$
* b) Show that the error grows exponentially: $\varepsilon(t) = \varepsilon(0) e^{F t} $.
* c)
 * 1) Suppose $F<1$.
 What happens to the error?
 What does this mean for predictability?
 * 2) Now suppose $F>1$.
 Given that all observations are uncertain (i.e. $R_t>0$, if only ever so slightly),
 can we ever hope to estimate $x(t)$ with 0 uncertainty?
* d) Consider the ODE derived above.
How might we change it in order to model (i.e. emulate) a saturation of the error at some level?
Can you solve this equation?
* e) Now suppose $z(t)$ evolves according to $\frac{dz}{dt} = g(z)$, with $g \neq f$.
What is now the differential equation governing the evolution of the error, $\varepsilon$?

```python
show_answer("error evolution")
```

(a). $\frac{d \varepsilon}{dt} = \frac{d (x-z)}{dt}
= \frac{dx}{dt} - \frac{dz}{dt} = f(x) - f(z) \approx f(x) - [f(x) - \frac{df}{dx}\varepsilon ] = F \varepsilon$

(b). Differentiate $e^{F t}$.

(c).

(1). Dissipates to 0.

(2). No.
  A balance is always reached between
  the uncertainty reduction $(1-K)$ and growth $F^2$.

  Also recall the asymptotic value of $P_k$ computed from
  the previous tutorial.

(d). link

(e). $\frac{d \varepsilon}{dt} \approx F \varepsilon + (f-g)$

**Exc 4.16*:** Recall the Lorenz-63 system. What is its doubling time (i.e. estimate how long does it take for two trajectories to grow twice as far apart as they were to start with) ?
*Hint: Set `N=50, eps=0.01, T=1,` and compute the spread of the particles now as compared to how they started*

```python
xx = output_63[0][:,-1] # Ensemble of particles at the end of integration
### compute your answer here ###
```

```python
show_answer("doubling time")
```

xx   = output_63[0][:,-1]      # Ensemble of particles at the end of integration
v    = np.var(xx, axis=0)      # Variance (spread^2) of final ensemble
v    = mean(v)                 # homogenize
d    = sqrt(v)                 # std. dev.
eps  = [FILL IN SLIDER VALUE]  # initial spread
T    = [FILL IN SLIDER VALUE]  # integration time
rate = log(d/eps)/T            # assuming d = eps*exp(rate*T)
print("Doubling time (approx):",log(2)/rate)

The answer actually depends on where in "phase space" the particles started.
To get a universal answer one must average these experiments for many different initial conditions.

---
## In summary:
Prediction (forecasting) with these systems is challenging because they are chaotic: small errors grow exponentially.

Conversely: chaos means that there is a limit to how far into the future we can make predictions (skillfully).

It is therefore crucial to minimize the intial error as much as possible. This is a task for DA.

### Next: [Ensemble [Monte-Carlo] approach](T5 - Ensemble [Monte-Carlo] approach.ipynb)

tutorials/T4 - Dynamical systems, chaos, Lorenz.ipynb


# Sampling

In this article, I will discuss some sampling methods: **Inverse Samling**, **Reject-Accept Sampling**, **Important Sampling**, **Markov Chain Monte Carol(MCMC)-MH Algorithm** and **Gibbs Sampling**.

*Why do we need sampling?* Sampling is the key issue of Monte Carol method, a better sampling approach can help us obtain a better result in term of accuracy and time in practise. For example, density estimation is hot topic in statistics all the time.(**Why？** Because we can calculate expectation, variation and other property related to pdf once we know some information on pdf.) The easies way to get a pdf of random variable $X$ is to inverse CDF.

```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
%matplotlib inline

plt.rcParams['figure.figsize']=(10,8)
```

## 1. Inverse sampling

The idea behind inverse sampling is very ituitive: to transform uniform samples into samples from a different distribution. That is, by somehow drawing from an uniform distribution(because CDF is range from 0 to 1), we make it possible to draw from the other distribution. The procedure of inverse sampling is illustrated as:

The assumption of inverse sampling is that **CDF must be invertiable!**

The algorithm of inverse sampling is:

1. get a uniform sample $\mu$ from **U** (0,1)
2. obtain the sample $x$ through $x=F^{-1}(\mu)$ where F is the CDF distribution we desire.
3. repeat.

**Why does inverse sampling work?**

To be note that:

$$
            \begin{equation}
            F^{-1}(\mu)= \text{ smallest }x \text{ such that } F(x)\geq \mu
            \end{equation}
$$

What's the ditribution does the random variable $y=F^{-1}(\mu)$ follow?

The CDF of y is $p(y\leq x)$. Since the CDF is monotonic, we can obtain without loss of generity:

$$
    \begin{equation}
    p(y\leq x)=p(F(y)\leq F(x))=p(\mu\leq F(x))=F(x)
    \end{equation}
$$

Thus we get the CDF and hence the pdf that we want to sample from

**Limitation of Inverse Sampling**

Not all the pdf has analytical CDF, for example, gaussian ditribution. In addition, for some complex distributions, the inverse CDF may be complicated and it is hard to do inverse sampling.

## 2. Rejection Sampling

### 2.1 Basic Rejection Sampling

The basic idea is come up with von Neumann. **If you have a function you are trying to sample from, whose functional form is well known, basically accept the sample by generating a uniform random number at any $x$ and accepting it if the value is below the value of the function at that $x$.** The procedure of basic sampling is illustrated as:

The procedure of basic rejection sampling:

1. Draw $x$ uniformly from $[x_{min}, x_{max}]$.
2. Draw $y$ uniformly from $[0, y_{max}]$.
3. If $y<f(x)$, accept. Else, reject.
4. Repeat.

**The intuitive explaination:** This works as more samples will be accepted in the regions of $x$-space where the function $f$ is higher: indeed they will be accepted in the ratio of the height of the function at any given $x$ to $y_{max}$.  **From the perspective of probability interpretation,**the accept-to-total ratio reflects the probability mass in each x silver.

### Code of Basic Rejction Sampling

```python
## target function

f = lambda x: np.exp(-x)
#f = lambda x x**2

## domain limits
xmin = 0 # lower bound of feasiable domain
xmax = 10 #upper bound of feasiable domain

## range limit for y
ymax=1
#ymax = 100

N = 10000 #the total of samples
accept =0 #count the total number of acceptance
samples = np.zeros(N)
count = 0

while (accept <N):

    x = np.random.uniform(xmin,xmax)
    y = np.random.uniform(0, ymax)

    if y<f(x):
        samples[accept]=x
        accept += 1
    count +=1

print('Count: ', count, ', Accepted: ', accept)

hinfo = np.histogram(samples, 30)
plt.hist(samples, bins=30, label='Smaples')
print(hinfo[0][0])
xvals = np.linspace(xmin, xmax, 10000)
plt.plot(xvals, hinfo[0][0]*f(xvals), 'r', label='f(x)')
plt.grid('on')
plt.legend()
```

### 2.2 Modified Reject Sampling

The basic rejection sampling has determinstic limitation(**low acceptance rate**):

For basic rejection sampling, we need know the supremum of function we want to sample from. In practice, it is hard to evaluate the function on interest domain and find a tight bound for this function. Furthermore, even if you find a tight bound for this function, you may abserve that the accept rate is very low, epescially in low density regions.

In order to overcome low acceptance rate while preserving the simplicity of rejection sampling, **proposal density** $g(x)$ is introduced. This proposal density $g(x)$ is commom density function $f^{'}(x)$ with a scaling factor $M$. (**Why? Because it is impossible to find a density function always above another density function. The integration of density function is always equal to 1. $\int p(x) dx=1$**) The reason we introduce proposal density function is to increase acceptance rate, but this is not the only way to do so.

The proposal density must has the following charicteristics:

1. $g(x)$ is easy to sample and calculate pdf
2. the range of M is $[0, \infty]$ so that $Mg(x)>f(x)$ in your entire interest domain.
3. ideally g(x) will be somewhat close to $f(x)$ so that you will sample more in high density regions and much less in low density region.

Obviously, the optimal value of $M$ is the supremum over your domain of interest of $\frac{f}{g}$. At that position x, the acceptance rate is  1. Ideally, the value of $M$ should be as close as to 1, since the acceptance rate is $\frac{1}{M}$. In other words, proposl density function $g(x)$ should be as indentical as $f(x)$.

The proportion of samples from $g(x)$ that are accept at each position $x$ and then average over $x$:

$$
\begin{equation}
\int dxg(x)\text{prop}(x)=\int dxg(x)\frac{f(x)}{Mg(x)}=\frac{1}{M}\int dxf(x)=\frac{1}{M}
\end{equation}
$$

**The procedure of modified rejection sampling is:**

1. Draw $x$ from your proposal distribution $g(x)$.
2. Draw $y$ uniformly from $[0, 1]$.
3. If $y<\frac{f(x)}{Mg(x)}$, accept the sample, otherwhile, reject the sample.
4. Repeat the procedure.

## The code of modified rejection sampling

```python
p = lambda x: np.exp(-x)  # our distribution
g = lambda x: 1/(x+1)  # our proposal pdf (we're thus choosing M to be 1)
invCDFg = lambda x: np.log(x +1) # generates our proposal using inverse sampling

# domain limits
xmin = 0 # the lower limit of our domain
xmax = 10 # the upper limit of our domain

# range limits for inverse sampling
umin = invCDFg(xmin)
umax = invCDFg(xmax)

N = 10000 # the total of samples we wish to generate
accepted = 0 # the number of accepted samples
samples = np.zeros(N)
count = 0 # the total count of proposals

# generation loop
while (accepted < N):

    # Sample from g using inverse sampling
    u = np.random.uniform(umin, umax)
    xproposal = np.exp(u) - 1

    # pick a uniform number on [0, 1)
    y = np.random.uniform(0,1)

    # Do the accept/reject comparison
    if y < p(xproposal)/g(xproposal):
        samples[accepted] = xproposal
        accepted += 1

    count +=1

print("Count", count, "Accepted", accepted)

# get the histogram info
hinfo = np.histogram(samples,50)

# plot the histogram
plt.hist(samples,bins=50, label=u'Samples');

# plot our (normalized) function
xvals=np.linspace(xmin, xmax, 1000)
plt.plot(xvals, hinfo[0][0]*p(xvals), 'r', label=u'p(x)')
plt.plot(xvals, hinfo[0][0]*g(xvals), 'k', label=u'g(x)')

# turn on the legend
plt.legend()
```

## 3. Important Sampling

**Different from reject sampling, import sampling does not has any rejection action. To replace with rejection action, important sampling adopt the approach of weighted sample.** In detail, we want to draw sample from $h(x)$, where a function whose integral or expectation we desire, is large. In the case of expecation, it would indeed be even better to draw more samples where $h(x)f(x)$ is large, where $f(x)$ is the pdf we are calculating the integral with respect to. I will show how importan sampling work below:

**Why improtant?** Often, in the computation of an expectation or other integral, the integrand has a very small value on a dominant fraction of the whole integration volume. If the points are chosen evenly in the integration volume, the small minority of the points close to the ‘peak’ give the dominant contribution to the integral.

**(Example: Expectation)**

$$
\begin{equation}
\mathbb{E}_{f}[h]=\int_{V}f(x)h(x)dx
\end{equation}
$$

Choose a distribution $g(x)$, which is close to the function $f(x)$, but which is simple enough so that it is possible to generate random $x$-values from this distribution. The integral can now be re-written as:

$$
\begin{equation}
\mathbb{E}_{f}[h]=\int h(x)g(x)\frac{f(x)}{g(x)} dx
\end{equation}
$$

Therefore if we choose random number $x_{i}$ from distribution $g(x)$, we obtain:

$$
\begin{equation}
\mathbb{E}_{f}[h(x)]=\lim_{N\rightarrow \infty}\frac{1}{N}\sum_{x_{i}\sim g(\cdot)}h(x_{i})\frac{f(x_{i})}{g(x_{i})}
\end{equation}
$$

Let $w(x_{i})=\frac{f(x_{i})}{g(x_{i})}$, the formulation can be rewritten:

$$
\begin{equation}
\mathbb{E}_{f}[h(x)]=\lim_{N\rightarrow \infty}\frac{1}{N}\sum_{x_{i}\sim g(\cdot)}h(x_{i})\omega(x_{i})
\end{equation}
$$

Now the variance(error) of monte carol is that:

$$
\begin{equation}
\widetilde{V}=\frac{V_{f}[h(x)]}{N}
\end{equation}
$$

where $N$ is the sample size.

With the important sampling this formula has now changed to

$$
\begin{equation}
\widetilde{V}=\frac{V_{g}[\omega(x)h(x)]}{N}
\end{equation}
$$

Our goal is to minimize the $V_{g}[\omega(x)h(x)]$.

As a somewhat absurd notion, this variance should be set to zero, if

$$
\begin{equation}
\omega(x)h(x)=C\Rightarrow f(x)h(x)=Cg(x)
\end{equation}
$$

which leads to (since $g(x)$ is density thus we need normalization):

$$
\begin{equation}
g(x) = \frac{f(x)h(x)}{\int f(x)h(x)dx}=\frac{f(x)h(x)}{\mathbb{E}_{f}[h(x)]}
\end{equation}
$$

Actually, the expection is what we expect to estimate. Let's ignore the denominator, this formula tell us that to achieve low variance, we must have $g(x)$ large where the product $f(x)h(x)$ is large. Didirectly, maximizing the latter in some fashion was our original intuition.

Or from another perspective, $\frac{f(x)}{g(x)}$ ought to be large where $h(x)$ is large. This means that, as we say earlier, choose more samples near the peak.

In detail,  We have a $f$ that we might or might not know. We have a pdf $g$ which we choose to be higher than $f$ at the points where hh has peaks. Now what we are left to do is to sample from $g$, and this will give us an oversampling at the place hh has peaks, and thus we must correct this there by multiplying by weights $w=\frac{f}{g}<1$ in thse places.

Be careful to choose $g(x)$ appropriately, it should have thicker tails than $f$, or the ratio $\frac{f}{g} will be too big and count contribute too much in the tails. All of these considerations may be seen in the diagram below:

Another way of seeing this whole thing is that we will draw the sample from a proposal distribution and re-weight the integral appropriately so that the expectation with respect to the correct distribution is used. And since $\frac{f}{g}$ is flatter than $f$, the variance of $h\times\frac{f}{g}$ is smaller that the variance of $h\times f$ and therefore the error will be smaller for all N.

### Code of Important Sampling ( Example: $\int_{0}^{\pi}sin(x)xdx$ )

```python
from scipy import stats
from scipy.stats import norm

mu = 2;
sig = .7;

f = lambda x: np.sin(x)*x
infun = lambda x: np.sin(x)-x*np.cos(x)
p = lambda x: (1/np.sqrt(2*np.pi*sig**2))*np.exp(-(x-mu)**2/(2.0*sig**2))
normfun = lambda x: norm.cdf(x-mu, scale=sig)

# Range of integration
xmax = np.pi
xmin = 0

# Number of draws
N = 1000

#Just Want to plot the function
x = np.linspace(xmin,xmax,1000)
plt.figure(figsize=(18,8))
plt.subplot(1,2,1)
plt.plot(x, f(x), 'b', label='Original $x\sin(x)$')
plt.plot(x, p(x), 'r', label='Important Sampling Function:')
plt.plot(x, np.ones(1000)/np.pi, 'k')
xis = mu + sig*np.random.randn(N,1)
plt.plot(xis, 1/(np.pi*p(xis)), '.', alpha=0.1)
plt.xlim([0, np.pi])
plt.ylim([0, 2])
plt.xlabel('x')
plt.legend()

## VANILLA MONTE CAROL
Ivmc = np.zeros(1000)
for k in np.arange(0, 1000):
    x = np.random.uniform(low=xmin, high=xmax, size=N)
    Ivmc[k] = (xmax-xmin)*np.mean(f(x))

print('Mean basic MC estimate:', np.mean(Ivmc))
print('Standard deviation of our estimates:', np.std(Ivmc))

## IMPORTANCE SAMPLING, choose gaussian so it is
## similar to the original functions

Iis = np.zeros(1000)
for k in np.arange(0, 1000):
    xis = mu + sig * np.random.randn(N,1)
    xis = xis[(xis<xmax) & (xis>xmin)]

    # normalization for gaussian from 0 to pi
    normal = normfun(np.pi)-normfun(0)
    Iis[k] = np.mean(f(xis)/p(xis))*normal

print('Mean important sampling MC estimate:', np.mean(Iis))
print('Standard deviation of our estimates:', np.std(Iis))

plt.subplot(1,2,2)
plt.hist(Iis, 30, histtype='step', label='Importance Sampling')
plt.hist(Ivmc, 30, color='r', histtype='step', label='Vanilla')
plt.grid('on')
plt.legend()
```

## 4. Markov Chain Monte Carol(MCMC)

### 4.1 What's the Markov Chain

**([Definition](https://en.wikipedia.org/wiki/Markov_chain):)** A Markov Chain is a stochastic process that satisfies the Markov property. In other words, a sequence of random variable taking value in state space is called a Markov Chain if the probability of the next step only depends on the current state(from [Lecture 9](https://am207.github.io/2017/wiki/markov.html)).

Using the notation of transition probabilities to define the probability of going from state $x$ ot state $y$ as $T(x\vert y)$, we can write this mathematically:

$$
\begin{equation}
T(x_{n+1}\vert x_{n},...,x_{1})=T(x_{n+1}\vert x_{n})
\end{equation}
$$

### 4.2 Some Markov Chains

**Homogenous Markov Chain:** A chain is homogeneous at step tt if the transition probabilities are independent of tt. Thus the evolution of the Markov chain only depends on the previous state with a fixed transition matrix.

**Irreducible Markov Chain:** Every state is accessible in a finite number of steps from another state. That is, there are no absorbing states. In other words, one eventually gets everywhere in the chain.(*example:* Consider as an example surfing the web. We do want to reach all parts of the web so we dont want to be trapped into an subset.)

**Recurrent:** States visited repeatedly are recurrent: positive recurrent if time-to-return is bounded and null recurrent otherwise. Harris recurrent if all states are visited infinitely often as $t\rightarrow \infty$.

**Aperiodic:** There are no deterministic loops. This would be bad in our web example as well as we would be stuck in a loop at some pages.

### 4.3 Stationary Markov Chain

**(Formal Definition of Stationarity):** A staionary Markov Chain produces the same marginal distribution when multiplied by the transition matrix. That is

$$
\begin{equation}
sT = s \text{ or }\sum_{i}s_{i}T_{ij}=s_{j}
\end{equation}
$$

In the case of continuous state space, which are the ones we encounter in sampling, if the transition kernel T is defined so that

$$
\begin{equation}
\int dx_{i}s(x_{i})T(x_{i+1}\vert x_{i})=s(x_{i+1})
\end{equation}
$$

then

$$
\begin{equation}
\int dxs(x)T(y\vert x)=s(y)
\end{equation}
$$

**Ergodicity**: Aperiodic, irreducible, positive Harris recurrent markov chains are ergodic, that is, in the limit of infinite (many) steps, the marginal distribution of the chain is the same(the probability of all state remain stable, nonzero and is independent with the initial position). This means that if we take largely spaced about samples from a stationary markov chain, we can draw independent samples.

$$
\begin{equation}
\int g(x)f(x)dx=\frac{1}{N}\sum_{j=B+1}^{B+N}g(x_{j})
\end{equation}
$$

Here B is called the burin (which comes from the approach to stationarity after a while) and T is called the thinning (which comes from ergodicity). So we have this “ergodic” law of large numbers.

([摘抄慕课网笔记](https://mooc.guokr.com/note/15627/))在满足一定条件的情况下，马尔可夫过程将收敛至一个均衡。这是一个统计均衡，在每种状态下的概率是固定不变的，但事物将依旧在各个状态间转移。

马尔可夫过程收敛到均衡的四个条件：

一、可能的状态数量是有限的。

二、转移概率固定不变。

三、从任意一个状态能够变到任意其他一个状态。有可能不是从状态A直接变到状态C，而是先变到状态B再变到C，但只要有路径从状态A变成状态C就行。

四、过程不是简单循环。比如不能是从全A变到全B，然后又自动从全B变到全A。

马尔可夫收敛定理（Markov Convergence Theorem）：如果满足上述四个条件，一个马尔科夫过程将收敛到一个均衡状态，且此均衡唯一。

**只要转移概率不变，那么初始状态、历史过程、中途干预都不重要，最后必将达到那个唯一的均衡。换句话说，马尔科夫链最后达到的均衡与初始状态，转移过程以及中途干预无关。**

```python
import scipy.stats as st

def target(lik, prior, n, h, theta):
    if theta < 0 or theta > 1:
        return 0
    else:
        return lik(n, theta).pmf(h)*prior.pdf(theta)

def mh_coin(niters, n, h, theta, lik, prior, sigma):
    samples = [theta]
    while len(samples) < niters:
        theta_p = theta + st.norm(0, sigma).rvs()
        rho = min(1, target(lik, prior, n, h, theta_p)/target(lik, prior, n, h, theta ))
        u = np.random.uniform()
        if u < rho:
            theta = theta_p
        samples.append(theta)
    return samples

n = 100
h = 61
lik = st.binom
prior = st.beta(10, 10)
sigma = 0.05
niters = 100

sampless = [mh_coin(niters, n, h, theta, lik, prior, sigma) for theta in np.arange(0.1, 1, 0.2)]

# Convergence of multiple chains

for samples in sampless:
    plt.plot(samples, '-o')
plt.xlim([0, niters])
plt.ylim([0, 1]);
```

### 4.3 Markov Chain and Monte Carol Methods

A irreducible (goes everywhere) and aperiodic (no cycles) markov chain will converge to a stationary markov chain. It is the marginal distribution of this chain that we want to sample from, and which we do in metropolis (and for that matter, in simulated annealing).

As we can see above, to find stationary distribution, we need to solve an eigenvector proble. This can be hard.

However, A sufficient, but not necessary, condition to ensure that s(x)s(x) is the desired stationary distribution is the already seen reversibility condition, also known as detailed balance:

$$
\begin{equation}
\int dxs(x)t(y\vert x)=s(y)\int dxT(x\vert y)
\end{equation}
$$

which gives back us back the stationarity condition from above.

Thus we want to design us samplers which satisfy detail balance.

### 4.4 Metropolis Hasting Algorithm(MH Algorithm)

**(why do we need Metropolis Hasting Algorithm?)** We've learnt how to do the inverse transform and how to use rejection sampling with a majority function. So why do we use these methods to sample a ditribution? ** inefficient as dimensions increased.** In other words, dimension curse. **How do we understand this point?**

In generally, we want to calculate the expectation of distribution as sample average, however, as dimension of space increased, majorizing in multiple dimensions can have us spending a lot of time in tail dimension because you leave more and more space out.  If inverse tranform and reject sampling methods are adopted, then it will boost inefficient.

In multiple dimensions, volumns get smaller and smaller, that's the curse of dimension. This concept can be shown as:

where the centre-partitions combination to an integral goes from 1/3rd to 1/27th. Now suppose the mode of the distibution is contained in this partition: then its contribution to the integral is going down with dimensions.

As the centre volume decreases, the outer volume increases, but this is in distribution tails, so we dont get much of a contribution from there either:

It is the neighborhood between these extremes, called the typical set which our sampler must explore well. And to get a good rejection sampling majorizer for this becomes hard.

**The idea of MH Algorithm**

1. Use a proposal distribution to propose s step
2. Then we calculate the pdf at that step, and compare it to the one at that previous step.
3. If the probability increased we accept. If the probability decreased, we accept the some of time, based on the ratio of the new probability to the old one.
4. We accumulate our samplees, as we now trying to sample a distribution.

** MH Algorithm**

1. initialize $x^{(0)}$.
2. Draw $\mu\sim U(0,1)$.
3. Draw propose $x^{*}\sim q(x^{*}\vert x)$.
4. If $\mu<\min(1, \frac{s(x^{*})q(x\vert x^{*})}{s(x)q(x^{*}\vert x)})$, $x^{(i+1)}=x^{*}$. Else, $x^{(i+1)}=x$
5. back step 2 for loop.

**Why does MH algorithm work?**

To prove if MH algorithm work is to prove MH algorithm satisfied detail balance condition.

Transition Kernel $K(x\rightarrow x^{*})$ includes the joint density of the following:

  1. Propose $x^{*}$ from the $q(x^{*}\vert x)$.
  2. the accept $x^{*}$ with the ratio $\alpha(x^{*}, x)=\min(1,\frac{s(x^{*})q(x\vert x^{*})}{s(x)q(x^{*}\vert x)})$

To verify the detail balance condition:

$$
\begin{aligned}
s(x^{*})q(x^{*}\vert x)\alpha(x^{*},x) &= s(x^{*})q(x^{*}\vert x)\min(1,\frac{s(x^{*})q(x\vert x^{*})}{s(x)q(x^{*}\vert x)}) \\
&= \min(s(x)q(x^{*}\vert x), s(x^{*})q(x\vert x^{*})) \\
&= s(x)q(x\vert x^{*})\min(1,\frac{s(x)q(x^{*}\vert x)}{s(x^{*})q(x\vert x^{*})}) \\
&= s(x)q(x\vert x^{*})\alpha(x,x^{*})
\end{aligned}
$$

### Code of Metropolis Hasting Algorithm

```python
def metropolis_hastings(p,q, qdraw, nsamp, xinit):
    samples=np.empty(nsamp)
    x_prev = xinit
    accepted=0
    for i in range(nsamp):
        x_star = qdraw(x_prev)
        p_star = p(x_star)
        p_prev = p(x_prev)
        pdfratio = p_star/p_prev
        proposalratio = q(x_prev, x_star)/q(x_star, x_prev)
        if np.random.uniform() < min(1, pdfratio*proposalratio):
            samples[i] = x_star
            x_prev = x_star
            accepted +=1
        else:#we always get a sample
            samples[i]= x_prev

    return samples, accepted

# target function
f = lambda x: 0.554*x*np.exp(-(x/1.9)**2)

x = np.linspace(0,10,100)
plt.plot(x, f(x), 'r')
plt.grid('on')
plt.title('The target function')
```

```python
from scipy.stats import gamma

t=10.0

def gammapdf(x_new, x_old):
    return gamma.pdf(x_new, x_old*t, scale=1/t)

def gammadraw(x_old):
    return gamma.rvs(x_old*t,scale=1/t)

x_init = np.random.uniform()
samps, acc = metropolis_hastings(f, gammapdf, gammadraw, 100000, x_init)

# plot our sample histogram
plt.hist(samps,bins=100, alpha=0.4, label=u'MCMC distribution', normed=True)
somesamps=samps[0::20000]
for i,s in enumerate(somesamps):
    xs=np.linspace(s-3, s+3, 100)
    plt.plot(xs, gamma.pdf(xs,s*t,scale=1/t),'k', lw=1)
xx= np.linspace(0,10,100)
plt.plot(xx, f(xx), 'r', label=u'True distribution')
plt.legend()
plt.xlim([0,10])
plt.show()
print("starting point was ", x_init)
```

## 5. Gibbs Sampling

### 5.1 The idea of Gibbs Sampling

The idea of gibbs as a markov chain in which the transition matrix can be obtained as the kernel of the an integral fixed point equation by sampling alternatively from two conditionals.

Gibbs determined the energy states of gases at equilibrium by cycling through all the particles, drawing from each one of them conditionally given the enerygy levels of the others, taking the time average.

Now, suppose you have a density function of two variables $(x,y)$. You wish to sample from this density.

The definition of the X marginal is

$$
\begin{equation}
f_{X}(x)=\int f_{XY}(x,y)dy
\end{equation}
$$

We reformulate the above formula:

$$
\begin{equation}
f_{X}(x)=\int f_{XY}(x,y)dy=\int f(x\vert y)f(y)dy=\int f(x\vert y)dy\int f(y\vert x^{'})f(x^{'})dx^{'}
\end{equation}
$$

Thus

$$
\begin{equation}
f(x)=\int h(x,x^{'})f(x^{'})dx^{'}
\end{equation}
$$

where

$$
\begin{equation}
h(x,x^{'})=\int f(x\vert y)f(y\vert x^{'})dy
\end{equation}
$$

Now consider an iterative scheme in which the “transition kernel” $h(x,x′)$ is used to create a proposal for metropolis-hastings moves. This looks like:

$$
\begin{equation}
f(x_{t})=\int h(x_{t},x_{t-1})f(x_{t-1})dx_{t-1}
\end{equation}
$$

which is the equation of stationary distribution.

The big idea, then, here, as in the case of markov chains, is that the above equation can be thought of as a fixed-point integral equation, and thus we can think of an iterative process which at first does not satisy this condition but then does as time goes on and we reach stationarity.

Similarly here, if we draw $y$, from the conditional $f(y|x′)$ and then $x$ again from $f(x|y)$ we will be able to get the marginal distribution of $x$. Symmetrically we can get the marginal for $y$.

Now, if I can draw from the $x$ marginal, and the $y|x$ conditional, i can draw from the $x,y$ joint, and I am done.

#### 5.2 Gibbs Sampling Algorithm

- Given a string sampling $(x_{1},y_{1},z_{1})^{T}$.
- You want to sample: ${(x_{2},y_{2},z_{2})^{T}, (x_{3},y_{3},z_{3})^{T},...,(x_{N},y_{N},z_{N})^{T}}\sim P(x,y,z)$.
- Then the algorithm goes:

$$
\begin{aligned}
x_{2}&\sim P(x\vert y_{1},z_{1}) \\
y_{2}&\sim P(y\vert x_{2},z_{1}) \\
z_{2}&\sim P(z\vert y_{2},x_{2}) \\
&... \\
x_{3}&\sim P(x\vert y_{2},z_{2}) \\
y_{3}&\sim P(y\vert x_{3},z_{2}) \\
z_{3}&\sim P(z\vert x_{3},y_{3}) \\
\end{aligned}
$$

#### 5.3 Gibbs and MH Algorithm

Gibbs is the extension of MH Algorithm in high dimension and also is a special case of MH Algorithm without rejection. I will show you the proof.

Look at the M-H acceptance ratio:

- Let **x**=$x_{1},...,x_{D}$.
- When sampling $k$th component, $q_{k}(x^{*}\vert x)=\pi(x_{k}^{*}\vert x_{-k})$.
- When sampling $k$th component, $x^{*}_{-k}=x_{-k}$.

$$
\begin{equation}
\frac{\pi(x^{*})q(x\vert x^{*})}{\pi(x)q(x^{*}\vert x)}= \frac{\pi(x^{*})q_{k}(x\vert x^{*}_{-k})}{\pi(x)q(x^{*}_{k}\vert x_{-k})} = \frac{\pi(x^{*}\vert x^{*}_{-k})q_{k}(x\vert x^{*}_{-k})}{\pi(x_{k}\vert x_{-k})q(x^{*}_{k}\vert x_{-k})}=1
\end{equation}
$$

### Code of Gibbs Sampling

$$
\begin{equation}
f(x,y)=x^{2}\text{exp}(-xy^{2}-y^{2}-2*y-4*x)
\end{equation}
$$

```python
func = lambda x,y: x**2*np.exp( -x*y**2 - y**2 + 2*y - 4*x )

numgridpoints=400
x = np.linspace(0,2,numgridpoints)
y = np.linspace(-1,2.5,numgridpoints)
xx,yy = np.meshgrid(x,y)
zz = np.zeros((numgridpoints,numgridpoints))
for i in np.arange(0,numgridpoints):
    for j in np.arange(0,numgridpoints):
        zz[i,j]=func(xx[i,j],yy[i,j])

plt.contourf(xx,yy,zz)
```

```python

```


## Central Limit Theorem

The [central limit theorem](https://en.wikipedia.org/wiki/Central_limit_theorem) is a key result in probability theory that helps explain why normal, or Gaussian, distributions are so omnipresent. The setup is that you have distributions for $N$ random variables $x_i$ and you want to know the distribution of $q = \sum_{i=1}^{N} x_i$. Think of each $x_i$ as coming from it's own distribution like in the figure below. For instance, $x_1$ might be the weight of spoons, $x_2$ the weight of forks, $x_3$ the weight of bowls, ..., and $x_N$ of plates in your kitchen. Then $q$ would represent the total weight when you have one of each of those objects. The distribution of weights for each object might be weird because you have some mix-and-match set of silverware from your parents, grandparents, IKEA, and the thrift shop. The *central limit theorem* says that if you have enough objects (i.e. $N$ is large), then $q$ has a normal (Gaussian) distribution.

Moreover, the central limit theorem states that the mean value of $q$ is given by

\begin{equation}
\mu_{q} = \sum_{i=1}^{N} \mu_{x_i}
\end{equation}

and the variance (standard deviation squared) is given by

\begin{equation}
\sigma_{q}^{2} = \sum_{i=1}^{N} \sigma^2_{x_i}
\end{equation}

*if you are having problems with the math displaing, click [here](http://nbviewer.jupyter.org/github/cranmer/intro-exp-phys-II/blob/master/Central-Limit-Theorem.ipynb?flush_cache=true)*

The mean probably isn't surprising because $q$ is just a sum and the integral the defines the mean just distributes across each term. Also, the equation for the variance is the same as the propagation of errors formula we use when we add different measurements together. However, that propoagation of errors formula is derived from the Central Limit Theorem, not vice versa.

### This is a collaborative project

In this repository there is a folder called `distributions` with several python files. The idea is that each student will create one of these python files and we will use GitHub to collect them. Each of these files has a python class that describes a probability distribution.  Each of these classes will define:
   * `x_min, x_max, f_max` - used for the accept/reject Monte Carlo sampling algorithm
   * `pdf()` - the probability density function
   * `mean()` - the mean of the pdf
   * `std()` - the standard deviation of the pdf

In addition, each of these python classes inherits from `BaseDistribution` which knows how to run the accept/reject algorithm using the information above ([see inside](distributions/base_distribution.py) ). In order to generate `n_samples` from the pdf, you simply call `dist.rvs(n_samples)`, where `dist` is an instance of one of these python classes.

**Naming Convention**: Name your file `Dist_<netid>.py` and your distribution the same way (without the `.py`). If you want to contribute more than one distribution, you can can add `Dist_<netid>_<index>.py`, where `<index>` is 1,2,3,...

Here's an example:

```python
!cat distributions/Dist_kc90.py
```

    import numpy as np
    from .base_distribution import BaseDistribution

    class Dist_kc90(BaseDistribution):
    	def __init__(self):
    		self.f_max = 1
    		self.x_min = -1
    		self.x_max = 1

    	def pdf(self, x):
    		"""This is your PDF"""
    		return np.abs(x)

    	def mean(self):
    		"""This is the mean of the PDF"""
    		return 0.

    	def std(self):
    		"""This is the standard deviation of the pdf"""
    		return np.sqrt(0.5)

## Example usage of the distributions

Ok, now let's use them. So far there are only two distributions, but there will be more soon.

```python
%pylab inline --no-import-all
from scipy.stats import norm #will use this for plotting
```

    Populating the interactive namespace from numpy and matplotlib

```python
# import all our distributions
import distributions
```

```python
# some funny python to make a list of all the distributions for convenience
all_distributions_dict = dict([(name, cls) for name, cls in distributions.__dict__.items() if isinstance(cls, type)])
all_distributions_list = [(cls) for name, cls in distributions.__dict__.items() if isinstance(cls, type)]
# print out their names
all_distributions_dict.keys()
```

    dict_keys(['Dist_rdm445', 'Dist_pr1392', 'Dist_pw1091', 'Dist_omr234', 'Dist_ap5312', 'Dist_os852', 'Dist_phh250', 'Dist_kc90', 'Dist_fh828', 'Dist_pme240', 'Dist_knd286', 'Dist_yz4244', 'Dist_mm7253', 'Dist_cas955', 'Dist_lwp226', 'Dist_lac683', 'Dist_fay221', 'Dist_jnt299'])

```python
len(all_distributions_dict.keys())
```

    18

```python
## Do tests
ok_distributions_list=[]
problems=[]
for i, cls in enumerate(all_distributions_list):
    #print(cls)
    try:
        dist = cls()
        N_test = 100000
        #print('will try to generate for %s' %(cls.__name__))
        if dist.pdf(dist.x_min + .3*(dist.x_max-dist.x_min)) < 1E-3:
            print("may have a problem")
            continue

        rvs = dist.rvs(N_test)
        if np.abs(np.mean(rvs) - dist.mean()) > 5*np.std(rvs)/np.sqrt(N_test):
            print("means don't match for %s: %f vs. %f" %(cls.__name__,
                                                          np.mean(rvs), dist.mean()))
            continue

        elif np.abs(np.std(rvs) - dist.std()) > 5*np.std(rvs)/np.sqrt(np.sqrt(1.*N_test)):
            print("std devs. don't match for %s: %f vs. %f" %(cls.__name__,
                                                          np.std(rvs), dist.std()))
            continue
        elif np.abs(np.std(rvs) - dist.std()) / dist.std() > 0.1:
            print("std devs. don't match for %s: %f vs. %f" %(cls.__name__,
                                                          np.std(rvs), dist.std()))
            continue

        elif np.sum(dist.pdf(np.linspace(dist.x_min,dist.x_max,100))<0) > 0:
            print("pdf was negative in some places")
            continue

        else:
            print("%s passes tests, adding it" %(cls.__name__))
            ok_distributions_list.append(cls)
    except:
        print("%s has errors, does't work" %(cls.__name__))
        continue

print("list of ok distributions:",[i.__name__ for i in ok_distributions_list])

```

    Dist_os852 passes tests, adding it
    means don't match for Dist_pr1392: 0.732852 vs. 0.000000
    Dist_pw1091 passes tests, adding it
    means don't match for Dist_ap5312: 0.721174 vs. 0.800000
    Dist_kc90 passes tests, adding it
    std devs. don't match for Dist_fh828: 0.217878 vs. 0.181690
    means don't match for Dist_yz4244: 0.833356 vs. 0.166667
    means don't match for Dist_mm7253: 0.097384 vs. 0.119000
    std devs. don't match for Dist_jnt299: 0.040570 vs. 0.001646
    Dist_rdm445 passes tests, adding it
    Dist_phh250 passes tests, adding it
    std devs. don't match for Dist_pme240: 0.095205 vs. 0.040377
    Dist_knd286 passes tests, adding it
    means don't match for Dist_omr234: 0.665732 vs. 1.000000
    means don't match for Dist_cas955: 0.621786 vs. 0.000000
    means don't match for Dist_lwp226: 0.332858 vs. 2.500000
    Dist_lac683 passes tests, adding it
    means don't match for Dist_fay221: 0.499351 vs. 1.500000
    list of ok distributions: ['Dist_os852', 'Dist_pw1091', 'Dist_kc90', 'Dist_rdm445', 'Dist_phh250', 'Dist_knd286', 'Dist_lac683']

```python
problems = [x for x in all_distributions_list if x not in ok_distributions_list]
[i.__name__ for i in problems]
```

    ['Dist_pr1392',
     'Dist_ap5312',
     'Dist_fh828',
     'Dist_yz4244',
     'Dist_mm7253',
     'Dist_jnt299',
     'Dist_pme240',
     'Dist_omr234',
     'Dist_cas955',
     'Dist_lwp226',
     'Dist_fay221']

```python
# how many samples for plots?
n_samples = 100000
```

```python
# Here's an example of usage
dist = distributions.Dist_kc90()
rvs = dist.rvs(n_samples)
counts, bins, edges = plt.hist(rvs, bins=50, normed=True, alpha =0.2)
y = []
for bin in bins:
    y.append(dist.pdf(bin))
plt.plot(bins, y, c='r', lw=2)
```

```python
dist.std()
```

    0.70710678118654757

## Let's inspect all the distributions we have

Here we will loop over the different distributions and make a plot like the one above

```python
fig = plt.figure(figsize=2*plt.figaspect(len(ok_distributions_list)))
for i, cls in enumerate(ok_distributions_list):
    dist = cls()
    rvs = dist.rvs(10000)
    ax = fig.add_subplot(len(ok_distributions_list),1,i+1)
    counts, bins, patches = ax.hist(rvs, bins=50, normed=True, alpha=0.2)
    # depending on how you code up your pdf() function, numpy might not
    # be able to "broadcast" (apply the function for each element of a numpy array)
    # but this should always wrok
    y = []
    for bin in bins:
        y.append(dist.pdf(bin))

    # ok, now plot it
    plt.plot(bins, y, c='r', lw=2)

    # and let's check if the distribution is ok
    print("%s: std from samples = %.2f, std from dist = %.2f" %(cls.__name__,np.std(dist.rvs(n_samples)), dist.std()))
    if np.abs(np.std(dist.rvs(n_samples)) - dist.std()) / dist.std() > 0.1:
        print("looks like a problem with this distribution: ", cls)
```

## Demonstration of the Central Limit Theorem

ok, let's use one of the distributions to demonstrate the central limit theorem. We will use the same distribution $N$ times.

First let's make a little helper function.

```python
def do_convolution(dist, N):
    n_samples = 10000
    q = np.zeros(n_samples)
    var_q = 0.
    mean_q = 0.

    for i in range(N):
        q = q+dist.rvs(n_samples)
        var_q = var_q + dist.std()**2
        mean_q = mean_q + dist.mean()

    std_q = np.sqrt( var_q )

    counts, bins, patches = plt.hist(q,bins=50, normed=True, alpha=.2)
    plt.plot(bins, norm.pdf(bins,loc=mean_q, scale=std_q), lw=2, c='r')
```

Now let's use it for $N=\{2,4,32\}$

```python
dist = distributions.Dist_kc90()
do_convolution(dist,2)
```

```python
do_convolution(dist,4)
```

```python
do_convolution(dist,32)
```

*Gorgeous!*

## Now let's do the same thing randomly using different distributions

To do this we will use `np.random.choice` to randomly choose from our list. Here's an example

```python
np.random.choice(['a','b','c','d'], 10)
```

    array(['d', 'c', 'd', 'c', 'a', 'a', 'a', 'c', 'b', 'a'],
          dtype='<U1')

Now let's make a variation on the helper function above

```python
def do_random_convolution(list_of_distributions, N):
    n_samples = 10000
    q = np.zeros(n_samples)
    var_q = 0.
    mean_q = 0.

    for dist_class in np.random.choice(list_of_distributions,N):
        dist = dist_class()
        print(dist_class.__name__, dist.std())
        q = q+dist.rvs(n_samples)
        var_q = var_q + dist.std()**2
        mean_q = mean_q + dist.mean()

    std_q = np.sqrt( var_q )

    counts, bins, patches = plt.hist(q,bins=50, normed=True, alpha=.2)
    plt.plot(bins, norm.pdf(bins,loc=mean_q, scale=std_q), lw=2, c='r')
```

```python
do_random_convolution(ok_distributions_list,2)
```

```python
do_random_convolution(ok_distributions_list,4)
```

```python
do_random_convolution(ok_distributions_list,4)
```

```python
do_random_convolution(ok_distributions_list,32)
```

```python

```

# Part 2 of project

## a) preliminaries
From either master or the branch you used to submit your distribution, update so that you have the current version of `cranmer/intro-exp-phys-II`.

You can either do this in GitHub desktop by finding the button near the top left or by typing this:
```
git fetch cranmer master
git merge cranmer/master
```

Now Create a new branch called "part2"

## b) execute the notebook above

## c) Make a $\chi^2$ function

Below is a copy of the `do_random_convolution` function with a new name. Modify it so taht it returns the chi-square quantity that says how closely the observed distribution matches the prediction from the Central Limit theorem.

YOu might want to check out the [chi-square-of-distribution](chi-square-of-distribution.ipynb) notebook

```python
def do_random_convolution_with_chi2(list_of_distributions, N):
    n_samples = 100000
    q = np.zeros(n_samples)
    var_q = 0.
    mean_q = 0.

    for dist_class in np.random.choice(list_of_distributions,N):
        dist = dist_class()
        q = q+dist.rvs(n_samples)
        var_q = var_q + dist.std()**2
        mean_q = mean_q + dist.mean()

    std_q = np.sqrt( var_q )

    counts, bins, patches = plt.hist(q,bins=50, normed=True, alpha=.2)
    plt.plot(bins, norm.pdf(bins,loc=mean_q, scale=std_q), lw=2, c='r')
```

## d) Make a plot

Plot the $\chi^2$ quantity vs. N for N=2,4,8,16,32

## e) push a new version of this notebook to GitHub

Execute the notebook, get your plots, save it, commit the changes to the `part2` branch, and then push (sync) to GitHub.com.  When you are done, make a new pull request.

```python

```

```python

```

```python

```


We consider solving a system of linear equations by using Gaussian elimination. Given a nonsingular square matrix $A \in \mathbb{R}^{n \times n}$, and a column vector $\vec{b} = \mathbb{R}^{n}$, find $\vec{x} \in \mathbb{R}^{n}$ such that
$$A\vec{\mathstrut x} = \vec{\mathstrut b}$$

# Backward and forward substitution
Assume $A$ is an upper triangular matrix, denoted as $U$, consider to solve:

$$U\vec{x} = \left[
\begin{array}{cccc}
u_{11} & u_{12} & \cdots & u_{1n} \\
       & u_{22} & \cdots & u_{2n} \\
       &        & \ddots & \vdots \\
       &        &        & u_{nn}
\end{array}\right] \cdot
\left[
\begin{array}{c}
x_1 \\
x_2 \\
\vdots \\
x_n
\end{array} \right] = \left[
\begin{array}{c}
b_1 \\
b_2 \\
\vdots \\
b_n
\end{array} \right] = \vec{b}$$

$Algorithm$: **Backward substitution**

    for j = n to 1
      if u_{jj} = 0, then display('Error: Singular Matrix');
      x_j = b_j /u_{jj};
      for i = 1 to j − 1
        b_i = b_i − u_{ij} x_j ;
      end
    end

And for lower triangular matrix, denoted as $L$, consider to solve:

$$L\vec{x} = \left[
\begin{array}{cccc}
l_{11} &        &        &        \\
l_{21} & l_{22} &        &        \\
\vdots & \vdots & \ddots &        \\
l_{n1} & l_{n2} & \cdots & l_{nn}
\end{array}\right] \cdot
\left[
\begin{array}{c}
x_1 \\
x_2 \\
\vdots \\
x_n
\end{array} \right] = \left[
\begin{array}{c}
b_1 \\
b_2 \\
\vdots \\
b_n
\end{array} \right] = \vec{b}$$

$Algorithm$: **Forward substitution**

    for j = 1 to n
      if u_{jj} = 0, then display('Error: Singular Matrix');
      x_j = b_j /u_{jj};
      for i = j + 1 to n
        b_i = b_i − u_{ij} x_j ;
      end
    end

# Simple examples of Gaussian elimination
Let's look at an example

$$A = \left[
\begin{array}{ccc}
2 & 2 & 2 \\
4 & 3 & 2 \\
4 & 6 & 4
\end{array}\right]$$

We apply Gaussian elimination by multiplying elementary matrices to the left of $A$, we have

$$\begin{align}
M_1 A = \left[
\begin{array}{ccc}
1 & 0 & 0 \\
-2 & 1 & 0 \\
-2 & 0 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
2 & 2 & 2 \\
4 & 3 & 2 \\
4 & 6 & 4
\end{array}\right] &= \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 2 & 0
\end{array}\right] \\
M_2 A = \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 2 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 2 & 0
\end{array}\right] &= \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{array}\right]
\end{align}$$

which is upper triangular, so that now we have:

$$M_2 M_1 A = \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 2 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
1 & 0 & 0 \\
-2 & 1 & 0 \\
-2 & 0 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
2 & 2 & 2 \\
4 & 3 & 2 \\
4 & 6 & 4
\end{array}\right] = \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{array}\right] = U$$

$i.e.$

$$\begin{align}
A = M_{1}^{-1} M_{2}^{-1} U &= \left[
\begin{array}{ccc}
1 & 0 & 0 \\
-2 & 1 & 0 \\
-2 & 0 & 1
\end{array}\right]^{-1} \cdot \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & 2 & 1
\end{array}\right]^{-1} \cdot \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{array}\right] \\
&= \left[
\begin{array}{ccc}
1 & 0 & 0 \\
2 & 1 & 0 \\
2 & 0 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
1 & 0 & 0 \\
0 & 1 & 0 \\
0 & -2 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{array}\right] \\
&= \left[
\begin{array}{ccc}
1 & 0 & 0 \\
2 & 1 & 0 \\
2 & -2 & 1
\end{array}\right] \cdot \left[
\begin{array}{ccc}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{array}\right] = LU
\end{align}$$

where $L$ is *lower triangular* and $U$ is *upper triangular*. We call $LU$ the **LU factorization** of $A$.

# Elementary matrices
This in a different elementary matrics with the following structure

$$M_{k} = \left[
\begin{array}{ccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & m_{k+1, k} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{n,k} & 0 & \cdots & 1
\end{array}\right] $$

where the only nontrivial entries are located below the $(k, k)$ diagonal on column $k$.

$Conclusion$
1. Gaussian elimination
$$M_{k} \vec{a} = \left[
\begin{array}{ccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & -a_{k+1}/a_{k} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & -a_{n}/a_{k} & 0 & \cdots & 1
\end{array}\right] \cdot \left[
\begin{array}{c}
a_1 \\
a_2 \\
\vdots \\
a_{k} \\
a_{k+1} \\
\vdots \\
a_n
\end{array} \right] = \left[
\begin{array}{c}
a_1 \\
a_2 \\
\vdots \\
a_{k} \\
0 \\
\vdots \\
0
\end{array} \right] $$

2. Inverse
$$M_{k}^{-1} = \left[
\begin{array}{ccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & m_{k+1, k} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{n,k} & 0 & \cdots & 1
\end{array}\right]^{-1} =\;\; \left[
\begin{array}{ccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & -m_{k+1, k} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & -m_{n,k} & 0 & \cdots & 1
\end{array}\right]$$

3. Production
$$\left[
\begin{array}{ccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & m_{k+1, k} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{n,k} & 0 & \cdots & 1
\end{array}\right] \cdot \left[
\begin{array}{ccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & m_{j+1, j} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{n,j} & 0 & \cdots & 1
\end{array}\right] \\ = \left[
\begin{array}{ccccccccccc}
1 & \cdots & 0 & 0 & 0 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 1 & 0 & 0 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & 1 & 0 & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{k+1, k} & 1 & \cdots & 0 & 0 & 0 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{j-1, k} & 0 & \cdots & 1 & 0 & 0 & \cdots & 0 \\
0 & \cdots & 0 & m_{j, k} & 0 & \cdots & 0 & 1 & 0 & \cdots & 0 \\
0 & \cdots & 0 & m_{j+1, k} & 0 & \ddots & 0 & m_{j+1, j} & 1 & \cdots & 0 \\
\vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \cdots & 0 & m_{j+1, n} & 0 & \ddots & 0 & m_{j+1, n} & 0 & \cdots & 0 \\
\end{array}\right]$$

# LU factorization
For a general invertible matrix $A = [a_{i,j}]_{n \times n}$,

$$\begin{align}
M_1A = \left[
\begin{array}{ccccc}
1 & & & & \\
-a_{21}/a_{11} & 1 & & &  \\
-a_{31}/a_{11} & 0 & 1 & & \\
\vdots & \vdots & \vdots & \ddots & \\
-a_{n1}/a_{11} & 0 & 0 & \cdots & 1
\end{array}\right] \cdot \left[
\begin{array}{ccccc}
a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
a_{21} & a_{22} & a_{23} & \cdots & a_{2n} \\
a_{31} & a_{32} & a_{33} & \cdots & a_{3n} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
a_{n1} & a_{n2} & a_{n3} & \cdots & a_{nn} \\
\end{array}\right] = \left[
\begin{array}{ccccc}
a_{11} & a_{12} & a_{13} & \cdots & a_{1n} \\
0 & \tilde{a}_{22} & \tilde{a}_{23} & \cdots & \tilde{a}_{2n} \\
0 & \tilde{a}_{32} & \tilde{a}_{33} & \cdots & \tilde{a}_{3n} \\
\vdots & \vdots & \vdots & \ddots & \vdots \\
0 & \tilde{a}_{n2} & \tilde{a}_{n3} & \cdots & \tilde{a}_{nn} \\
\end{array}\right]
\end{align}$$

Keep multiply these $M$ Lower matrix, finally we can see that

$$M_{n-1}M_{n-2}\cdots M_2M_1A = \left[
\begin{array}{cccc}
u_{11} & u_{12} & \cdots & u_{1n} \\
       & u_{22} & \cdots & u_{2n} \\
       &        & \ddots & \vdots \\
       &        &        & u_{nn}
\end{array}\right] = U$$

$i.e.$

$$A = \left(\prod _{i = 1}^{n-1} M_{i}^{-1} \right) U = LU = \left[
\begin{array}{cccc}
   1   &        &        &        \\
l_{21} &    1   &        &        \\
\vdots & \vdots & \ddots &        \\
l_{n1} & l_{n2} & \cdots &   1
\end{array}\right] \cdot \left[
\begin{array}{cccc}
u_{11} & u_{12} & \cdots & u_{1n} \\
       & u_{22} & \cdots & u_{2n} \\
       &        & \ddots & \vdots \\
       &        &        & u_{nn}
\end{array}\right] $$

where $L$ is a lower triangular matrix with $1$ on its diagonals, and $U$ is an upper normal triangular matrix.

$Algorithm$: **LU factorization**

    for k = 1, . . . , n − 1         % loop over the first n − 1 columns
      if a_{kk} = 0 then stop
        for i = k + 1, . . . , n     % loop over Row k + 1 through Row n
          a_ {ik} ← a_ {ik}/a_{kk}   % store the multiplier, which gives L, NOT included in the flops
          for j = k + 1, . . . , n   % update other entries of that row
            a_{ij} ← a_{ij} − a_ {ik} a_{kj}
          end
        end
    end

To evaluate the c*omputational expense* of an algorithm we can analyze the number of floating point operation (flops). One operation of addition, subtraction, multiplication, division, etc, is called a **floating point operation**. For LU factorization, we have

$$\begin{align}
\text{flops} &= \sum_{k = 1} ^{n - 1} \sum_{i = k + 1} ^{n} \sum_{j = k + 1} ^{n} 2 \\
&= 2 \sum_{k = 1} ^{n} (n - k)^2 = 2\sum_{l = 1} ^{n - 1} l^2 \\
&= 2\left[n^3 - 1 - \frac{3n(n - 1)} {2} - (n - 1) \right]\\
&\approx \frac{2} {3} n^3
\end{align}$$

It is a big number, when we are dealing with a huge matrix. But with certain special structures we can modify the algorithm to make it more effective. Especially the **banded matrix**, where the nonzero entries of a matrix $A_{n \times n}$ only appear in the main $m$ off diagonals.

$Algorithm$: **LU factorization for banded matrix**

    for k = 1, . . . , n − 1                     % loop over the first n − 1 columns
      if a_{kk} = 0 then `stop`
        for i = k + 1, . . . , min{k + m, n}     % loop over Row k + 1 to the last nonzero entry
          a_ {ik} ← a_ {ik}/a_{kk}               % store the multiplier, which gives L, NOT included in the flops
          for j = k + 1, . . . , min{k + m, n}   % update other entries of that row
            a_{ij} ← a_{ij} − a_ {ik} a_{kj}
          end
        end
    end

$$\begin{align}
\text{flops} &= \sum_{k = 1} ^{n - 1} \sum_{i = k + 1} ^{k + m} \sum_{j = k + 1} ^{k + m} 2 \\
&= 2 \sum_{k = 1} ^{n} m^2 \\
&= 2m^2 (n - 1)
\end{align}$$
***
So how do we use this factorization? Actually we can use it for solving linear equations

$$A\vec{x} = \vec{b} \Leftrightarrow
\begin{cases}
L\vec{\mathstrut y} = \vec{\mathstrut b} \\
U\vec{\mathstrut x} = \vec{\mathstrut y}
\end{cases}
$$

>**e.g.** Solve
>
>$$A\vec{x} = \left[
\begin{array}{ccc}
2 & 2 & 2 \\
4 & 3 & 2 \\
4 & 6 & 4
\end{array}\right] \cdot \left[
\begin{array}{c}
x_1 \\
x_2 \\
x_3
\end{array}\right] = \left[
\begin{array}{c}
2 \\
3 \\
2
\end{array}\right]$$
>
>First we factorization matrix $A$ like
>
>$$\begin{align}
A\vec{x} &= LU\vec{x} = \vec{b} \\
&= \left( \begin{bmatrix}
1 & 0 & 0 \\
2 & 1 & 0 \\
2 & -2 & 1
\end{bmatrix} \cdot
\begin{bmatrix}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{bmatrix}\right ) \cdot \left[
\begin{array}{c}
x_1 \\
x_2 \\
x_3
\end{array}\right] = \left[
\begin{array}{c}
2 \\
3 \\
2
\end{array}\right]
\end{align}$$
>
>$i.e.$
>
>$$L\vec{y} = \begin{bmatrix}
1 & 0 & 0 \\
2 & 1 & 0 \\
2 & -2 & 1
\end{bmatrix} \cdot \left[
\begin{array}{c}
y_1 \\
y_2 \\
y_3
\end{array}\right] = \left[
\begin{array}{c}
2 \\
3 \\
2
\end{array}\right] \\ \Downarrow \\
\vec{y} = \left[
\begin{array}{c}
y_1 \\
y_2 \\
y_3
\end{array}\right] = \left[
\begin{array}{c}
2 \\
-1 \\
-4
\end{array}\right]$$
>
>Then we solve
>$$U\vec{x} = \begin{bmatrix}
2 & 2 & 2 \\
0 & -1 & -2 \\
0 & 0 & -4
\end{bmatrix} \cdot \left[
\begin{array}{c}
x_1 \\
x_2 \\
x_3
\end{array}\right] = \left[
\begin{array}{c}
2 \\
-1 \\
-4
\end{array}\right] \\ \Downarrow \\
\vec{x} = \left[
\begin{array}{c}
x_1 \\
x_2 \\
x_3
\end{array}\right] = \left[
\begin{array}{c}
1 \\
-1 \\
1
\end{array}\right]$$

# Gaussian elimination with pivoting
**Pivot**, the $a_{kk}$. And the elimination without pivoting may fail when meeting situation like the following:

$$A = \begin{bmatrix}
0 & 1 \\
1 & 1
\end{bmatrix}$$

First we define the **permutation matrix**, obtained by exchanging any two rows of the identity matrix. An example can show how it works.

$$PMP = \begin{bmatrix}
1 & & & & \\
 & & & 1 & \\
 & & 1 & & \\
 & 1 & & & \\
 & & & & 1
\end{bmatrix}\begin{bmatrix}
1 & & & & \\
m_{21} & 1 & & & \\
m_{31} & & 1 & & \\
m_{41} & & & 1 & \\
m_{51} & & & & 1
\end{bmatrix}\begin{bmatrix}
1 & & & & \\
 & & & 1 & \\
 & & 1 & & \\
 & 1 & & & \\
 & & & & 1
\end{bmatrix} = \begin{bmatrix}
1 & & & & \\
m_{41} & 1 & & & \\
m_{31} & & 1 & & \\
m_{21} & & & 1 & \\
m_{51} & & & & 1
\end{bmatrix}$$

Another property of the permutation matrix is that for any permutation matrix $P$, it holds that $P^2 = I$, $i.e.$, $P
^{−1} = P$.

Now we can get down to the Gaussian elimination with pivoting. Before we eliminate column $k$, we first need to exchange the largest entry among all $a_{ik}, i = k, k+1, \dots, n$ to the entry $a_{kk}$, meaning that change the row with the largest entry with row $k$.

So that in general, this process can be represented as

$$M_{n-1}P_{n-1}M_{n-2}P_{n-2}\cdots M_{2}P_{2}M_{1}P_{1}A = U$$

$Conclusion$

For any nonsingular matrix $A$, there exists a permutation matrix $P = P_{n-1}P_{n-2}\dots P_2P_1$ such that $PA = LU$.

$Proof$

We've already got that $A = P_{1}^{-1}M_{1}^{-1}P_{2}^{-1}M_{2}^{-1}\dots P_{n-2}^{-1}M_{n-2}^{-1}P_{n - 1}^{-1}M_{n-1}^{-1} = P_{1}L_{1}P_{2}L_{2}\dots P_{n-2}L_{n-2}P_{n-1}L_{n-1}$. The $L$ is the inverse of $M$, also a lower triangular matrix. So that now we have

$\begin{align}
A &= P_{1}L_{1}P_{2}L_{2}\dots P_{n-2}L_{n-2}P_{n-1}L_{n-1}\\
P_{n-1}P_{n-2}\dots P_2P_1A &= (P_{n-1}P_{n-2}\dots P_2) L_{1}P_{2}L_{2}\dots P_{n-2}L_{n-2}P_{n-1}L_{n-1} \\
&= (P_{n-1}P_{n-2}\dots P_2 L_{1}P_2P_3\dots P_{n-2}P_{n-1}) (P_{n-1}P_{n-2}\dots P_2)P_2 L_2 \dots P_{n-2}L_{n-2}P_{n-1}L_{n-1} \\
& = \tilde{L}_1 (P_{n-1}P_{n-2}\dots P_3 L_{2}P_3\dots P_{n-2}P_{n-1}) (P_{n-1}P_{n-2}\dots P_3)P_3 L_3 \dots P_{n-2}L_{n-2}P_{n-1}L_{n-1} \\
&\cdots \\
&= \tilde{L}_1\tilde{L}_2\cdots \tilde{L}_{n-1} L_n U
\end{align}$

To summary,

$$\begin{cases}
\tilde{L}_1 &= P_{n-1}P_{n-2}\dots P_{2}L_{1}P_{2}\dots P_{n-2}P_{n-1} \\
\tilde{L}_2 &= P_{n-1}P_{n-2}\dots P_{3}L_{2}P_{3}\dots P_{n-2}P_{n-1} \\
\vdots &\vdots \\
\tilde{L}_{n-2} &=  P_{n-1}L_{n-2}P_{n-1} \\
\tilde{L}_{n-1} &= L_{n-1} \\
P &= P_{n-1} P_{n-2} \dots P_2 P_1 \\
L &= \tilde{L}_1 \tilde{L}_2 \cdots \tilde{L}_{n-2} \tilde{L}_{n-1}
\end{cases}$$

$Algorithm$: **LU factorization using pivoting**

    for k = 1, . . . , n − 1
      find the row number i^∗, where |a_{i^∗k}| = max_{k≤i≤n} |a_{ik}|
      exchange the two rows A(k,:) ↔ A(i^∗,:)
      p(k) = i^∗
      for i = k + 1, . . . , n
        a_{ik} ← a_{ik}/a{kk}
        for j = k + 1, . . . , n
          a_{ij} ← a_{ij} − a_{ik} a_{kj}
        end
      end
    end

So now to solve a linear system $A\vec{x} = \vec{b}$, we can solve $PA\vec{x} = LU\vec{x} = P\vec{b}$, so actually what's needed is permutation matrix $P$, which can be obtained during the process of pivoting.

# Stability of Gaussian elimination
For matrix

$$A = \begin{bmatrix}
10^{-20} & 1 \\
1 & 1
\end{bmatrix}$$

Easily we can find that $L = \begin{bmatrix}
1 & 0 \\
10^{20} & 1
\end{bmatrix}$ and $U = \begin{bmatrix}
10^{-20} & 1 \\
0 & 1 - 10^{20}
\end{bmatrix}$. But without pivoting, in computer, we will see that $\tilde{U} = \begin{bmatrix}
10^{-20} & 1 \\
0 & - 10^{20}
\end{bmatrix}$. And the truth is that

$$\tilde{L}\tilde{U} =  \begin{bmatrix}
1 & 0 \\
10^{20} & 1
\end{bmatrix} \cdot \begin{bmatrix}
10^{-20} & 1 \\
0 & - 10^{20}
\end{bmatrix} = \begin{bmatrix}
10^{-20} & 1 \\
1 & 0
\end{bmatrix}$$

So the first process should be

$$P_1A = \begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}\begin{bmatrix}
10^{-20} & 1 \\
1 & 1
\end{bmatrix} = \begin{bmatrix}
1 & 1 \\
10^{-20} & 1
\end{bmatrix}$$

Then

$$M_1P_1A = \begin{bmatrix}
1 & 0 \\
-10^{-20} & 1
\end{bmatrix}\begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}\begin{bmatrix}
10^{-20} & 1 \\
1 & 1
\end{bmatrix} = \begin{bmatrix}
1 & 1 \\
0 & 1-10^{-20}
\end{bmatrix} = \begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}$$

Now, $\tilde{U} = \begin{bmatrix}
1 & 1 \\
0 & 1
\end{bmatrix}$, $\tilde{L} = \begin{bmatrix}
1 & 0 \\
10^{-20} & 1
\end{bmatrix}$, $P = \begin{bmatrix}
0 & 1 \\
1 & 0
\end{bmatrix}$, and we can find that

$$\tilde{L}\tilde{U} = \begin{bmatrix}
1 & 1 \\
10^{-20} & 1+10^{-20}
\end{bmatrix}$$

which is the true *LU factorization* of the nearby matrix to $PA$. And since been pivoted, generally the lower trianuglar entries in the $L$ matrix in the Gaussian elimination with pivoting are always at most $1$ in absolute value.

$Theorem$ 1

Let $A = LU$ be computed by Gaussian elimination, with or without partial pivoting, on a computer with machine epsilon $\epsilon_{\text{machine}}$ . Denote the computed $L$ and $U$ matrices by $\tilde{L}$ and $\tilde{U}$, respectively. Then we have

$$\tilde{L}\tilde{U} = A + \delta A$$

for a certain $\delta A$ which satisfies $\displaystyle \frac{\left\| \delta A \right\|} {\left\| L \right\| \left\| U \right\|} = O(\epsilon_{\text{machine}})$

This help us to estimate the relative difference between $\tilde{L}\tilde{U}$ and $A$:

$$\frac{\left\| \delta A \right\|} {\left\| A \right\|} = \frac{\left\| L \right\| \left\| U \right\|} {\left\| A \right\|}O(\epsilon_{\text{machine}})$$

$i.e.$

The stability ($\tilde{f}(x) = f(\tilde{x})$) of the Gaussian elimination depends on the norms of $L$ and $U$. After pivoting, all entries in $L$ is equal or less than $1$, so that the norm of $L$ is not large, then the stability will be determined by $\displaystyle \frac{\left\| U \right\|} {\left\| A \right\|}$

Now we define $\rho = \displaystyle \frac{max_{ij} u_{ij}} {max_{ij} a_{ij}}$, the **growth factor** of the LU factorization. Because $\left\| U \right\| = O( \rho \left\| A \right\| )$.

$Theorem$ 2

Let $PA = LU$ be the LU factorization computed by Gaussian elimination **with partial pivoting** on a **computer** with machine epsilon $\epsilon_{\text{machine}}$. Denote the computed $P$, $L$ and $U$ matrices by $\tilde{P}$, $\tilde{L}$ and $\tilde{U}$, respectively. Then we have

$$\tilde{L}\tilde{U} = \tilde{P}A + \delta A$$

for a certain $\delta A$, which now satisfies $\displaystyle \frac{\left\|U \right\|} {\left\|A\right\|} = O(\rho \epsilon_{\text{machine}})$

A typical size for the growth factor $\rho$ for an $n \times n$ matrix is $O(\sqrt{n})$. And the worst case is that $\rho = 2^{n-1}$

# Cholesky Factorization
If an $n$ by $n$ matrix $A$ is symmetric positive definite, $i.e.$, for all $n\text{-dimensional}$ column vector $\vec{x} \neq \vec{0}$, $x^{\mathrm{T}} Ax > 0$, then we can write the matrix $A$ as

$$\begin{align}
A &= \begin{bmatrix}
a_{11} & \vec{v}_1^{\mathrm{T}} \\
\vec{v}_1 & K_1
\end{bmatrix} \\
&= \begin{bmatrix}
\sqrt{a_{11}} & \vec{0} \\
\frac{\vec{v}_1}{\sqrt{a_{11}}} & I
\end{bmatrix}
\begin{bmatrix}
1 & \vec{0} \\
\vec{0} & K_1 - \frac{\vec{v}_1\vec{v}_1^{\mathrm{T}}}{a_{11}}
\end{bmatrix}
\begin{bmatrix}
\sqrt{a_{11}} & \frac{\vec{v}_1^{\mathrm{T}}}{\sqrt{a_{11}}} \\
\vec{0} & I
\end{bmatrix} \\
&= L_1 A_1 L_1^{\mathrm{T}}
\end{align}$$

where $L_1$ is lower triangular and $A_1$ is also symmetric positive definite. Continuing the process, we have a series of $A_i$ that $A_i = L_{i+1}A_{i+1}L_{i+1}^{\mathrm{T}}$, where $L_i$ are all lower triangular and $A_i$ are all symmetric positive definite, with $A_{n+1} = 1$. Therefore

$$A = L_1 A_1 L_1^{\mathrm{T}} = L_1 L_2 A_2 L_2^{\mathrm{T}} L_1^{\mathrm{T}} = \cdots = LL^{\mathrm{T}}$$

here
$$\begin{align}L &= L_1 L_2 \cdots L_n \\
&= \begin{bmatrix}
\sqrt{a_{11}} & \vec{0} \\
\frac{\vec{v}_1}{\sqrt{a_{11}}} & I
\end{bmatrix}
\left[
\begin{array}{cc}
1 & \vec{0}\\
\vec{0} & \begin{array}{cc}
 {\sqrt{a_{22}}} & \vec{0} \\
 \frac{\vec{v}_2} {\sqrt{a_{22}}} & I
\end{array}
\end{array}
\right] \cdots
\begin{bmatrix}
1 & & & \\
 & \ddots & & \\
 & & 1 & \\
 & & & \sqrt{a_{22}}
\end{bmatrix} \\
&= \left[
\begin{array}{cc}
\sqrt{a_{11}} & \vec{0} \\
\frac{\vec{v}_1} {\sqrt{a_{11}}} & \begin{array}{cc}
\sqrt{a_{22}} & \vec{0} \\
\frac{\vec{v}_2} {\sqrt{a_{22}}} & \begin{array}{cc}
\ddots & \vdots \\
\cdots & \sqrt{a_{nn}}
\end{array}
\end{array}
\end{array}
\right]
\end{align}$$

which is still a lower triangular. We call this the **Cholesky factorization** of the matrix $A$.

$Algorithm$: **Cholesky Factorization**

```
for k = 1, . . . , n                         % loop over columns
    for i = k + 1, . . . , n                 % update each row behind row k
        A(i, k + 1 : n) = A(i, k + 1 : n) − a_{ik}/a_{kk} A(k, k + 1 : n)
    end
    A(k : n, k) = A(k : n, k)/√(a_{kk})
end
```

With this algorithm, $L$ is saved as the lower triangular part of the result matrix $A$. And actually the update of the upper part is not necessary, so here's the efficient version

$Algorithm$: **Cholesky Factorization Efficient Version**

```
for k = 1, . . . , n                         % loop over columns
    for i = k + 1, . . . , n                 % update each row behind row k
        A(i, k + 1 : i) = A(i, k + 1 : i) − a_{ik}/a_{kk} (A(k + 1 : i, k))^T
    end
    A(k : n, k) = A(k : n, k)/√(a_{kk})
end
```

At last, we claim that The Cholesky factorization of symmetric positive definite matrices is **always stable**.

Computational/Intro to Numerical Computing/Note_Chap05_Gaussian elimination, LU factorization, Cholesky factorization.ipynb


# Lecture 11: Poisson Distribution and Approximation

## Stat 110, Prof. Joe Blitzstein, Harvard University

----

## Sympathetic Magic, or _[Confusing the map for the territory](http://nobeliefs.com/MapandTerritory.htm)_

### Don't mistake a random variable for its distribution

As a case in point, say we have two random variables $X$ and $Y$.

\begin{align}
  P(X + Y) \ne P(X=x) + P(Y=y)
\end{align}

Claiming equality is clearly nonsense, since $X + Y$ is itself a random variable, and it should be recognized that $P(X=x) + P(Y=y)$ can exceed 1.

### Think of them as houses & blueprints

Metaphorically, the distribution is the _blueprint_, and the random variable is the _house_.

----

## Poisson Distribution

### Description

The single most often used real-world discrete model. Applicable when there are a _large number of trials_ with a _small probability of success in each trial_.

Note that the probability of success in each trial does _not_ have to be the same in each trial (unlike $\operatorname{Bin}(n,p)$).

The Poisson distribution expresses the probability of a given number of events occurring in a fixed interval of time and/or space, if these events occur with a known average _rate_ and independently of the time since the last event. The Poisson distribution can also be used for the number of events in other specified intervals such as distance, area or volume.

Examples might be counting:

- e-mails coming into your mailbox in an hour
- buses following the same route arriving at a bus-stop in a day
- natural phenomena, like earthquakes or meteor sightings, etc., in a month
- chips in a chocolate-chip cookie (area/volume)

### Notation

$X \sim \operatorname{Pois}(\lambda)$

### Parameters

$\lambda$ - the average number of events per interval (rate), where $\lambda \gt 0$

```python
import matplotlib
import numpy as np
import matplotlib.pyplot as plt

from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.stats import poisson

%matplotlib inline

plt.xkcd()
_, ax = plt.subplots(figsize=(12,8))

# seme Poisson parameters
lambda_values = [1, 4, 8, 16]

# colorblind-safe, divergent color scheme
colors = ['#018571', '#80cdc1', '#dfc27d', '#a6611a']

for i,l in enumerate(lambda_values):
    x = np.arange(poisson.ppf(0.01, l), poisson.ppf(0.99, l))
    pmf = poisson.pmf(x, l)
    ax.plot(x, pmf, 'o', color=colors[i], ms=8, label=r'$\lambda={}$'.format(l))
    ax.vlines(x, 0, pmf, lw=2, color=colors[i], alpha=0.3)

# legend styling
legend = ax.legend()
for label in legend.get_texts():
    label.set_fontsize('large')
for label in legend.get_lines():
    label.set_linewidth(1.5)

# y-axis
ax.set_ylim([0.0, 0.4])
ax.set_ylabel(r'$P(X=k)$')

# x-axis
ax.set_xlim([0, 26])
ax.set_xlabel(r'total # of occurrences $k$ under $\lambda$')

# x-axis tick formatting
majorLocator = MultipleLocator(5)
majorFormatter = FormatStrFormatter('%d')
minorLocator = MultipleLocator(1)
ax.xaxis.set_major_locator(majorLocator)
ax.xaxis.set_major_formatter(majorFormatter)
ax.xaxis.set_minor_locator(minorLocator)

ax.grid(color='grey', linestyle='-', linewidth=0.3)

plt.suptitle(r'Poisson PMF: $P(X = k) = e^{-\lambda} \frac{\lambda^k}{k!}$')

plt.show()
```

### Probability mass function

\begin{align}
  P(X = k) = e^{-\lambda} \frac{\lambda^k}{k!} ~~ \text{, where } k \in \{0,1,2,\dots,\}
\end{align}

Checking the validity of this PMF it is easy to see that it is always non-negative.

Furthermore,

\begin{align}
  \sum_{k=0}^{\infty} P(X = k) &= \sum_{k=0}^{\infty} e^{-\lambda} \frac{\lambda^k}{k!} \\
  &= e^{-\lambda} \sum_{k=0}^{\infty} \frac{\lambda^k}{k!} & &\text{since } e^{-\lambda} \text{ is constant} \\
  &= e^{-\lambda} e^{\lambda} & &\text{recall Taylor series for } e^{\lambda} \\
  &= 1 ~~~~ \blacksquare
\end{align}

### Expected value

\begin{align}
  \mathbb{E}(X) &= \sum_{k=0}^{\infty} k ~~ e^{-\lambda} \frac{\lambda^k}{k!} \\
  &= e^{-\lambda} \sum_{k=1}^{\infty} k \frac{\lambda^k}{k!} \\
  &= e^{-\lambda} \sum_{k=1}^{\infty} \frac{\lambda^k}{(k-1)!} \\
  &= \lambda e^{-\lambda} \sum_{k=1}^{\infty} \frac{\lambda^{k-1}}{(k-1)!} \\
  &= \lambda e^{-\lambda} e^{\lambda} \\
  &= \lambda ~~~~ \blacksquare
\end{align}

----

## Poisson Paradigm (Approximate)

Suppose we have:

1. a lot of events $A_1, A_2, \dots, A_n$, with $P(A_j) = p$
1. number of trials $n$ is very large
1. $p$ is very small

The events could be _independent_ (knowing that $A_1$ occurred has no bearing whatsoever on $A_2$.

They could even be _weakly dependent_ ($A_1$ occurring has some level of affect on the likelihood of $A_2$).

Either way, the expected number of events $A_j$ occurring is approximately $\operatorname{Pois}(\lambda)$, and by Linearity

\begin{align}
  \mathbb{E}(A) &= \lambda = \sum_{j=1}^n p_j
\end{align}

### Relating the Binomial distribution to the Poisson

This example will relate the Binomial distribution to the Poisson distribution.

Suppose we have $X \sim \operatorname{Bin}(n,p)$, and let $n \rightarrow \infty$, $p \rightarrow 0$.

We will hold $\lambda = np$ constant, which will let us see what happens to the Binomial distribution when the number of trails approaches $\infty$ while the probabability of success $p$ gets very small.

Specifically let's see what happens to the Binomial PMF under these conditions.

\begin{align}
  P(X = k) &= \binom{n}{k} p^k q^{n-k} ~~~~ \text{where k is fixed} \\
  &= \underbrace{\frac{n (n-1) \dots (n-k+1)}{k!} \left(\frac{\lambda}{n}\right)^k}_{\text{n terms cancel out}} ~~ \underbrace{\left(1 - \frac{\lambda}{n}\right)^n}_{\text{continuous compounding and e}} ~~ \underbrace{\left(1 - \frac{\lambda}{n}\right)^{-k}}_{\text{goes to 1}}   \\
  \\
  &\rightarrow \frac{\lambda^k}{k!} e^{-\lambda} ~~ \text{which is the Poisson PMF at k}
\end{align}

### Thought-experiment: counting raindrops

Let's say we want to count the number of raindrops that fall on a certain area on the ground.

We could divide this area with a grid, with each grid cell being very, very tiny.

Therefore:

- the number grid cells is very, very large
- the probability of a rain drop landing in an arbitrary cell is very, very small

_What distribution could we use to model this situation?_

#### How about $X \sim \operatorname{Bin}(n,p)$?
Considering the Binomial distribution, we would have to make certain assumptions:

1. the event of a raindrop hitting a cell is independent of other events
1. the probability $p$ of a raindrop hitting an arbitrary cell is identical.

While it might be OK to make these assumptions, but one **big** stumbling block in using the Binomial is that we would end up calculating factorials of a very large $n$, which even for a computer would be problematic.

#### How about $X \sim \operatorname{Pois}(\lambda)$?

The Poisson distribution is a better model approximation than the Binomial, since

1. no factorials involved, so much simpler than the Binomial
1. we can deal with the case where more than one raindrop lands in a cell
1. we don't need to make any assumption on $p$ being identical

#### Example: Graph comparing $X \sim \operatorname{Bin}(n,p)$ vs $X \sim \operatorname{Pois}(\lambda = np)$ where $n = 1e^{6}$ and $p = 1e^{-5}$

```python
from scipy.stats import binom

plt.xkcd()

fig = plt.figure(figsize=(12,8))

# Binomial parameters
n = 1e6
p = 1e-5

x = np.arange(binom.ppf(0.01, n, p), binom.ppf(0.99, n, p))

# plot Bin(n,np)
binom_pmf = binom.pmf(x, n, p)
ax1 = fig.add_subplot(211)
ax1.plot(x, binom_pmf, 'o', ms=8, color='#f46d43', label=r'$X \sim Bin(n,p)$')
ax1.vlines(x, 0, binom_pmf, lw=2, color='#f46d43', alpha=0.3)
ax1.set_title(r'$P(x=k) = \binom{n}{k} p^k (1-p)^{n-k}$')
ax1.legend(loc='upper right')

ax1.grid(color='grey', linestyle='-', linewidth=0.3)

# y-axis
ax1.set_ylim([0.0, 0.15])

# plot Pois(np)
pois_pmf = poisson.pmf(x, n*p)
ax2 = fig.add_subplot(212)
ax2.plot(x, pois_pmf, 'o', ms=8, color='#4575b4', label=r'$X \sim Pois(\lambda = np)$')
ax2.vlines(x, 0, pois_pmf, lw=2, color='#4575b4', alpha=0.3)
ax2.set_title(r'$P(X = k) = e^{-\lambda} \frac{\lambda^k}{k!}$')
ax2.legend(loc='upper right')

ax2.grid(color='grey', linestyle='-', linewidth=0.3)

# y-axis
ax2.set_ylim([0.0, 0.15])

fig.tight_layout()

st = fig.suptitle(r'Approximating $X \sim Bin(n,p)$ with $X \sim Pois(\lambda)$, where $n=1e^{6}, p=1e^{-5}$', fontsize=18)
st.set_y(1.035)
```

----

## Birthdays, revisited: triple matches

Suppose we have $n$ people, and we want to know the _approximate probability_ that there are 3 people among the $n$ who share the same birthday.

Doing this the way we did in Lecture 3 is possible, but it would get very messy.

Let's set up the problem first.

- there are $\binom{n}{3}$ triplets
- we used indicator r.v. $I_{ijk} \text{, where } i < j < k$
- $\Rightarrow \mathbb{E}(\# \text{ triple matches}) = \binom{n}{3} \frac{1}{365^2}$

But we are interested in the _approximate probability_ of triple matches, so now let's use the Poisson.

_Why?_

1. we expect the total number of trials $\binom{n}{3}$ to be very large
1. the probability $P(I_{ijk})$ is very small
1. events are _weakly dependent_, since if persons $1$ and $2$ are already known to share a birthday, then  $I_{123} \text{, } I_{124}$ are not completely independent

We claim that these circumstances are _approximately_ $\operatorname{Pois}(\lambda)$ with $\lambda = \mathbb{E}(\# \text{ triple matches})$.

Let $X = \text{# triple matches}$

\begin{align}
  P(X \ge 1) &= 1 - P(X=0) \\
  &\approx 1 - e^{-\lambda} \frac{\lambda^0}{0!} \\
  &= 1 - e^{-\lambda} ~~~~ \text{where you just plug in } \lambda = \binom{n}{3} \frac{1}{365^2} ~~~~ \blacksquare
\end{align}

----

View [Lecture 11: The Poisson distribution | Statistics 110](http://bit.ly/2Pn99BO) on YouTube.