# Capítulo 14: Forward-Backward Sweep Adaptado

### Exemplos numéricos do capítulo

Para ilustrar o método, vamos propor exemplos e seus códigos aqui descritos.

### Importando Bibliotecas

```python
import numpy as np
from matplotlib import pyplot as plt
from scipy.optimize import root_scalar
```

## Um estado com pontos finais fixos

Considere o seguinte exemplo:

\begin{align*}
    &\min_u \frac{1}{2}\int_0^1 x_1(t)^2 + u(t)^2 dt \\
    \text{sujeito a }&x_1 '(t) = x_2(t), x_1(0) = 1, \\
    &x_2 '(t) = -ru(t), x_2(0) = 0, x_2(1) = s, \\
    &r > 0, s < 0
\end{align*}

### Passo 1: Escrever condições necessárias

Vejamos que o Hamiltoniano é

$$
H = \frac{1}{2}(x_1^2 + u^2) + \lambda_1x_2 - r\lambda_2 u
$$

Assim, as equações adjuntas são
$$
\lambda_1 ' = -H_{x_1}  = -x_1
$$
$$
\lambda_2 ' = -H_{x_2}  = - \lambda_1
$$
com condição de transversalidade $\lambda_1(1) = 0$. A condição de otimalidade é dada por

$$
0 = H_u = u - r\lambda_2 \implies u^* = r \lambda_2
$$

```python
diff_state = lambda t, x, u, r: np.array([
    x[1],
    -r*u
])

diff_adjoint = lambda t, x, u, lambda_, r: np.array([
    -x[0],
    -lambda_[0]
])

update_u = lambda t, x, lambda_, r: r*lambda_[:,1]
```

### Passo 2: Algoritmo de integração

Aqui desenvolvemos os algoritmos de forward e de backward para a integração segundo o método de Runge-Kutta.

```python
def forward(g, t, x, u, r, h):

    for i in range(len(t)-1):
        k1 = g(t[i],x[i],u[i], r)
        k2 = g(t[i]+h/2,x[i] + 0.5*h*k1, 0.5*(u[i] + u[i+1]), r)
        k3 = g(t[i]+h/2,x[i] + 0.5*h*k2, 0.5*(u[i] + u[i+1]), r)
        k4 = g(t[i]+h,x[i] + h*k3, u[i+1], r)
        x[i+1] = x[i] + (h/6)*(k1 + 2*k2 + 2*k3 + k4)
    return x

def backward(dadj, t, x, u, lambda_, r, h):

    for i in range(len(t)-1,0,-1):
        k1 = dadj(t[i],x[i],u[i],lambda_[i], r)
        k2 = dadj(t[i]-h/2,0.5*(x[i] + x[i-1]), 0.5*(u[i] + u[i-1]),lambda_[i] - 0.5*h*k1, r)
        k3 = dadj(t[i]-h/2,0.5*(x[i] + x[i-1]), 0.5*(u[i] + u[i-1]),lambda_[i] - 0.5*h*k2, r)
        k4 = dadj(t[i]-h,x[i-1], u[i-1], lambda_[i] - h*k3, r)
        lambda_[i-1] = lambda_[i] - (h/6)*(k1 + 2*k2 + 2*k3 + k4)
    return lambda_
```

### Passo 3: Método completo de controle ótimo

O código a seguir constrói o método forward-backward a partir de um chute $\theta$. Com esse valor, faremos $\lambda_2(t_1) = \theta$. A partir desse cálculo, obteremos um valor do aproximado do estado final $x_2(1)$, supondo que a convergência ocorra. O valor `h` é o passo do método de Runge-Kutta e `tol` é a tolerância do erro para o algoritmo. Eles estão pré-determinados nesse caso.

```python
def optimal_control(theta, r, h = 0.001, tol = 0.0001):

    condition = -1

    N = int(np.round(1/h))
    t = np.linspace(0,1,N+1)
    u = np.random.normal(scale=0.01, size=N+1)
    x = np.zeros(shape = (N+1, 2))
    lambda_ = np.zeros(shape = (N+1, 2))

    x[0] = np.array([1,0])
    lambda_[-1] = np.array([0,theta])

    while condition < 0:

        u_old = u.copy()
        x_old = x.copy()
        lambda_old = lambda_.copy()

        x = forward(diff_state, t, x, u, r, h)
        lambda_ = backward(diff_adjoint, t, x, u, lambda_, r, h)

        u = 0.5*(update_u(t, x, lambda_, r) + u)

        cond1 = tol*np.sum(abs(u)) - np.sum(abs(u_old - u))
        cond2 = min(tol*np.sum(abs(x), axis=0) - np.sum(abs(x_old - x), axis=0))
        cond3 = min(tol*np.sum(abs(lambda_), axis=0) - np.sum(abs(lambda_old - lambda_), axis=0))
        condition = min(cond1, cond2, cond3)

    return t,x,u,lambda_
```

### Passo 4: Método secante

Por fim, precisamos encontrar o valor de $\theta$ correto. Para isso, vamos escrever a função $V(\theta) = \tilde{x}_{21} - s$, onde $\tilde{x}_{21}$ é uma aproximação para $x_{21}$ a partir de $\theta$. Queremos encontrar os zeros dessa função. Para isso usaremos o método descrito no capítulo e escrito na biblioteca `scipy`.

```python
V = lambda theta, r, s: optimal_control(theta, r)[1][-1][1] - s

theta = lambda r, s: root_scalar(f = V,
                                 args = (r, s),
                                 method = 'secant',
                                 x0 = -1,
                                 x1 = 1,
                                 )
```

## Aplicando as funções ao exemplo

Vamos fazer algumas exeperimentações. Por fim, vamos testar o código com valores de $r$ e $s$.

```python
r = 1
s = -1
t, x, u, lambda_ = optimal_control(theta(r, s).root, r)
```

Primeiro vamos verificar o valor $x_2(1)$. De fato não é o valor exato, mas é bem próximo do desejado.

```python
x[-1,1]
```

    -1.0000000608421535

```python
fig, ax = plt.subplots(3,1, sharex = True, figsize = (12, 8))

ax[0].plot(t, x[:,0])
ax[1].plot(t, x[:,1])
ax[2].plot(t, u)

ax[0].set_ylabel(r'$x_0$')
ax[1].set_ylabel(r'$x_1$')
ax[2].set_ylabel(r'$u$')

plt.show()
```

Variar o método para encontrar a raíz pode ser interessante.

## Termos de payoff não lineares

Considere o seguinte exemplo:

\begin{align*}
    \min_u \frac{1}{4} x_2(1)^4 + &\frac{1}{2}\int_0^1 x_1(t)^2 + u(t)^2 dt \\
    \text{sujeito a }&x_1 '(t) = x_2(t), x_1(0) = 1, \\
    &x_2 '(t) = - ru(t), x_2(0) = 0, \\
    r > 0,
\end{align*}

Note que esse exemplo é muito próximo ao anterior, com diferença sendo: não fixamos $x_2(1)$, mas queremos minimizar uma função dele. Como $\phi(s) = \frac{1}{4}s^4, \phi '(s) = s^3$. Assim definimos
$$
V(\theta) = (\tilde{x}_{21})^3 - \theta
$$

O que precisamos mudar do código que já fizemos até então? Precisamos apenas mudar a função $V$! De resto, tudo será equivalente.

```python
V = lambda theta, r: (optimal_control(theta, r)[1][-1][1])**3 - theta

theta = lambda r: root_scalar(f = V,
                              args = (r,),
                              method = 'secant',
                              x0 = -1,
                              x1 = 1,
                              )
```

## Aplicando as funções ao exemplo

Vamos fazer algumas exeperimentações. Por fim, vamos testar o código com valores de $r$.

```python
r = 1
t, x, u, lambda_ = optimal_control(theta(r).root, r)
```

```python
fig, ax = plt.subplots(3,1, sharex = True, figsize = (12, 8))

ax[0].plot(t, x[:,0])
ax[1].plot(t, x[:,1])
ax[2].plot(t, u)

ax[0].set_ylabel(r'$x_0$')
ax[1].set_ylabel(r'$x_1$')
ax[2].set_ylabel(r'$u$')

plt.show()
```

## Shots Múltiplos

Considere o seguinte exemplo

\begin{align}
    &\min_u \frac{1}{2}\int_0^1 x_1(t)^2 + u(t)^2 dt \\
    \text{sujeito a }&x_1'(t) = x_2(t), x_1(0) = 1, x_1(1) = 0, \\
    &x_2'(t) = -ru(t), x_2(0) = 0, x_2(1) = s, \\
    &r > 0, s < 0.
\end{align}

Temos o mesmo exemplo, agora com $x_1$ com tempo final fixo. Nem $\lambda_1$, nem $\lambda_2$ tem valor conhecido em $1$ e, por isso, precisamos fazer chutes para ambos. Vamos fazer algumas mudanças para que tenhamos dois chutes.

```python
def optimal_control(theta1, theta2, r, h = 0.001, tol = 0.0001):

    condition = -1

    N = int(np.round(1/h))
    t = np.linspace(0,1,N+1)
    u = np.random.normal(scale=0.01, size=N+1)
    x = np.zeros(shape = (N+1, 2))
    lambda_ = np.zeros(shape = (N+1, 2))

    x[0] = np.array([1,0])
    lambda_[-1] = np.array([theta1, theta2])

    while condition < 0:

        u_old = u.copy()
        x_old = x.copy()
        lambda_old = lambda_.copy()

        x = forward(diff_state, t, x, u, r, h)
        lambda_ = backward(diff_adjoint, t, x, u, lambda_, r, h)

        u = 0.5*(update_u(t, x, lambda_, r) + u)

        cond1 = tol*np.sum(abs(u)) - np.sum(abs(u_old - u))
        cond2 = min(tol*np.sum(abs(x), axis=0) - np.sum(abs(x_old - x), axis=0))
        cond3 = min(tol*np.sum(abs(lambda_), axis=0) - np.sum(abs(lambda_old - lambda_), axis=0))
        condition = min(cond1, cond2, cond3)

    return t,x,u,lambda_
```

Vamos montar agora as duas funções para encontrar as raízes.

```python
# V1 = \tilde{x}_{11} - 0
V1 = lambda theta1, r, s: optimal_control(theta1, f_theta2(theta1, r,s).root, r)[1][-1][0]
# V2 = \tilde{x}_{21} - s
V2 = lambda theta2, theta1, r, s: optimal_control(theta1, theta2, r)[1][-1][1] - s

# encontra a raíz theta1.
f_theta1 = lambda r, s: root_scalar(f = V1,
                                     args = (r, s),
                                     method = 'secant',
                                     x0 = -1,
                                     x1 = 1,
                                     )
# encontra a raíz theta2, dado theta1
f_theta2 = lambda theta1, r, s: root_scalar(f = V2,
                                     args = (theta1, r, s),
                                     method = 'secant',
                                     x0 = -1,
                                     x1 = 1,
                                     )
```

## Aplicando as funções ao exemplo

Vamos fazer algumas exeperimentações. Por fim, vamos testar o código com valores de $r$ e $s$.

```python
r = 1
s = -1

theta1 = f_theta1(r,s).root
theta2 = f_theta2(theta1,r,s).root
t,x,u,_ = optimal_control(theta1, theta2, r)
```

```python
fig, ax = plt.subplots(3,1, sharex = True, figsize = (12, 8))

ax[0].plot(t, x[:,0])
ax[1].plot(t, x[:,1])
ax[2].plot(t, u)

ax[0].set_ylabel(r'$x_0$')
ax[1].set_ylabel(r'$x_1$')
ax[2].set_ylabel(r'$u$')

plt.show()
```

### Este é o fim do notebook


## Logistic Population Differential Equations

In this notebook we will model non-linear logistic population growth [1]

```python
## Library
import numpy as np
import math
import matplotlib.pyplot as plt
```

## Logistic Population Equation
The logistic growth differential equation is a non-linear first order differential equation
\begin{equation}
\frac{d p}{dt}=\alpha_{Birth} p-\alpha_{Death} p^2
\end{equation}
for $a\leq t \leq b$ years, where $\alpha_{Birth}$ is the birth rate and $\alpha_{Death}$ is the death rate, with the initial condition
$$ p(a)=A. $$
As the equation is non-linear we must numverically approximate the solution.
In this simulation we set the parameters of the different equations:
*  as $\alpha_{Birth}=0.3$ and $\alpha_{Death}=0.01$, with the inital condition $p(0)=3$, over the time period $0\leq t \leq 20$.

```python
alpha_Birth=0.3
alpha_Death=0.01
```

## Discrete Domain
To numerically approximate the solution we discretise the continuous domain using a step size, $h=1$,
which gives
\begin{equation}
t_i=0+i h,
\end{equation}
for $i=0,...20$.
The figure below illustrate the discrete domain.

```python
h=1
a=0
b=20
t=np.arange(a,b+h/2,h)
fig = plt.figure(1,figsize=(5,4))
ax = fig.add_subplot(1,1,1)
ax.plot(t,0*t,'o:',color='k')
ax.set_xlabel('Time (yrs)')
plt.title('Discrete Time Domain')
plt.show()
```

```python
N=int((b-a)/h) # Number of Steps
P=np.zeros(N+1) # Numerical Solution P
P[0]=3

```

We numerically approximate the differential equation using Eulers method to give the difference equation
\begin{equation}
\frac{P_{i+1}-P_i}{h}=\alpha_{Birth}P_i-\alpha_{Death}P_i^2
\end{equation}
re-arranging
\begin{equation}
\color{red}{P_{i+1}}=\color{green}{P_i}+h\big(\alpha_{Birth}\color{green}{P_i}-\alpha_{Death}\color{green}{P_i^2}\big)
\end{equation}

where $h$ is the stepsize and where $P_i$ is the numerical approximation of $p(t_i)$ at time $t_i$, for $i=0,...,10$

```python
for i in range (0,N):
    P[i+1]=P[i]+h*(alpha_Birth*P[i]-alpha_Death*P[i]*P[i])

```

## Results
The plot bellow shows the Numerical solution $P_i$ of the Logistic Population as a function of time $t$.

```python
## Plotting Figure
fig = plt.figure(1,figsize=(4,4))

ax = fig.add_subplot(1,1,1)

ax.plot(t,P,'o:',color='red',label='Population')
ax.legend()

ax.set_xlabel('Time (yrs)')
ax.set_ylabel('Population (billions)')
#ax.legend(loc='best')
ax.set_title('Numerical Solution, P')

plt.tight_layout()
plt.subplots_adjust(top=0.85)
plt.show()

```

# References

[1] Stover, Christopher and Weisstein, Eric W. "Population Growth." From MathWorld--A Wolfram Web Resource. http://mathworld.wolfram.com/PopulationGrowth.html

[2]  Vito Volterra.  Fluctuations in the abundance of a species considered mathematically. Nature,118:558–560,1926.

[3] Alfred J Lotka.   Analytical note on certain rhythmic  relations inorganic  systems.Proceedings of the National Academy of Sciences,6(7):410–415,1920.

```python

```

04_Logisitic Population Differential Equations.ipynb


# Chapter 4
`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`

______

## The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been using this simple idea in every example thus far.

### The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear by examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for almost *any distribution*, minus some important cases we will encounter later.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size = 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to its parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```python
%matplotlib inline
import numpy as np
from IPython.core.pylabtools import figsize
import matplotlib.pyplot as plt

figsize( 12.5, 5 )

sample_size = 100000
expected_value = lambda_ = 4.5
poi = np.random.poisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average),
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend();
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This is simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

The above formulae is interpretable as a distance away from the true value (on average), for some $N$. (We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

By computing the above many, $N_y$, times (remember, it is random), and averaging them:

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \rightarrow E[ Y_k ] = E\;\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \right]$$

Finally, taking the square root:

$$ \sqrt{\frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k} \approx D(N) $$

```python
figsize( 12.5, 4)

N_Y = 250 #use this many to approximate D(N)
N_array = np.arange( 1000, 50000, 2500 ) #use this many samples in the approx. to the variance.
D_N_results = np.zeros( len( N_array ) )

lambda_ = 4.5
expected_value = lambda_ #for X ~ Poi(lambda) , E[ X ] = lambda

def D_N( n ):
    """
    This function approx. D_n, the average variance of using n samples.
    """
    Z = poi( lambda_, (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for i,n in enumerate(N_array):
    D_N_results[i] =  D_N(n)

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 2, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " );
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of convergence to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 5, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 5 ) =  \frac{1}{N}\sum_{i=1}^N \mathbb{1}_{z > 5 }(Z_i) $$

```python
N = 10000
print( np.mean( [ np.random.exponential( 0.5 ) > 5 for i in range(N) ] ) )
```

    0.0001

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give us *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```python
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = np.random.randint
norm = np.random.normal

#generate some artificial population numbers
population = pop_generator(100, 1500, n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height,
                                        population[i] ).mean()

#located the counties with the apparently most extreme average heights.
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population size vs. recorded average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1);
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error results from the calculated average of smaller populations not being a good reflection of the true expected value of the population (which in truth should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 1500, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```python
print("Population sizes of 10 'shortest' counties: ")
print(population[ np.argsort( average_across_county )[:10] ], '\n')
print("Population sizes of 10 'tallest' counties: ")
print(population[ np.argsort( -average_across_county )[:10] ])
```

    Population sizes of 10 'shortest' counties:
    [109 152 144 132 347 219 127 185 171 447]

    Population sizes of 10 'tallest' counties:
    [135 100 108 164 191 109 158 153 187 139]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```python
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);
```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How to order Reddit submissions

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, called submissions, for people to comment on. Redditors can vote up or down on each submission (called upvotes and downvotes). Reddit, by default, will sort submissions to a given subreddit by Hot, that is, the submissions that have the most upvotes recently.

How would you determine which submissions are the best? There are a number of ways to achieve this:

1. *Popularity*: A submission is considered good if it has many upvotes. A problem with this model is that a submission with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the submission is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of submission. Depending on when a submission is posted, the website may be experiencing high or low traffic. The difference method will bias the *Top* submissions to be the those made during high traffic periods, which have accumulated more upvotes than submissions that were not so graced, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the submission. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter-example is, if we use per second, a 1 second old submission with 1 upvote would be better than a 100 second old submission with 99 upvotes. One can avoid this by only considering at least t second old submission. But what is a good t value? Does this mean no submission younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old submissions).
3. *Ratio*: Rank submissions by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new submissions who score well can be considered Top just as likely as older submissions, provided they have many upvotes to total votes. The problem here is that a submission with a single upvote (ratio = 1.0) will beat a submission with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter submission is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former submission, with a single upvote, is in fact a better submission than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former submission might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this submission a upvote, versus a downvote"). So the 999 upvote/1 downvote submission probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the submission with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is to look at the historical distribution of upvote ratios. This can be accomplished by scraping Reddit's submissions and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of submissions have very few votes, hence there will be many submissions with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use submissions with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of submissions available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards submissions of these two subreddits are very different: visitors are likely friendly and affectionate in the former, and would therefore upvote submissions more, compared to the latter, where submissions are likely to be controversial and disagreed upon. Therefore not all submissions are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `top_showerthoughts_submissions.py` will scrape the best posts from the `showerthoughts` community on Reddit. This is a text-only community so the title of each post *is* the post. Below is the top post as well as some other sample posts:

```python
#adding a number to the end of the %run call will get the ith top post.
%run top_showerthoughts_submissions.py 2

print("Post contents: \n")
print(top_post)
```

```python
"""
contents: an array of the text from the last 100 top submissions to a subreddit
votes: a 2d numpy array of upvotes, downvotes for each submission.
"""
n_submissions = len(votes)
submissions = np.random.randint( n_submissions, size=4)
print("Some Submissions (out of %d total) \n-----------"%n_submissions)
for i in submissions:
    print('"' + contents[i] + '"')
    print("upvotes/downvotes: ",votes[i,:], "\n")
```

    Some Submissions (out of 98 total)
    -----------
    "Rappers from the 90's used guns when they had beef rappers today use Twitter."
    upvotes/downvotes:  [32  3]

    "All polls are biased towards people who are willing to take polls"
    upvotes/downvotes:  [1918  101]

    "Taco Bell should give customers an extra tortilla so they can make a burrito out of all the stuff that spilled out of the other burritos they ate."
    upvotes/downvotes:  [79 17]

    "There should be an /r/alanismorissette where it's just examples of people using "ironic" incorrectly"
    upvotes/downvotes:  [33  6]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular submission's upvote/downvote pair.

```python
import pymc3 as pm

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    """
    This function accepts the number of upvotes and downvotes a particular submission recieved,
    and the number of posterior samples to return to the user. Assumes a uniform prior.
    """
    N = upvotes + downvotes
    with pm.Model() as model:
        upvote_ratio = pm.Uniform("upvote_ratio", 0, 1)
        observations = pm.Binomial( "obs",  N, upvote_ratio, observed=upvotes)

        trace = pm.sample(samples, step=pm.Metropolis())

    burned_trace = trace[int(samples/4):]
    return burned_trace["upvote_ratio"]

```

Below are the resulting posterior distributions.

```python
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(submissions)):
    j = submissions[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 10, normed = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 10, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different submissions");
```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the submissions from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean is a bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```python
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(submissions)):
    j = submissions[i]
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")
plt.title("Posterior distributions of upvote ratios on different submissions");
order = np.argsort( -np.array( lower_limits ) )
print(order, lower_limits)
```

The best submissions, according to our procedure, are the submissions that are *most-likely* to score a high percentage of upvotes. Visually those are the submissions with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best.  When using the lower-bound of the 95% credible interval, we believe with high certainty that the 'true upvote ratio' is at the very least equal to this value (or greater), thereby ensuring that the best submissions are still on top. Under this ordering, we impose the following very natural properties:

1. given two submissions with the same observed upvote ratio, we will assign the submission with more votes as better (since we are more confident it has a higher ratio).
2. given two submissions with the same number of votes, we still assign the submission with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every submission takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```python
votes
```

```python
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print("Approximate lower bounds:")
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print(lb)
print("\n")
print("Top 40 Sorted according to approximate lower bounds:")
print("\n")
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print(votes[i,0], votes[i,1], contents[i])
    print("-------------")
```

    Approximate lower bounds:
    [ 0.93349005  0.9532194   0.94149718  0.90859764  0.88705356  0.8558795
      0.85644927  0.93752679  0.95767101  0.91131012  0.910073    0.915999
      0.9140058   0.83276025  0.87593961  0.87436674  0.92830849  0.90642832
      0.89187973  0.89950891  0.91295322  0.78607629  0.90250203  0.79950031
      0.85219422  0.83703439  0.7619808   0.81301134  0.7313114   0.79137561
      0.82701445  0.85542404  0.82309334  0.75211374  0.82934814  0.82674958
      0.80933194  0.87448152  0.85350205  0.75460106  0.82934814  0.74417233
      0.79924258  0.8189683   0.75460106  0.90744016  0.83838023  0.78802791
      0.78400654  0.64638659  0.62047936  0.76137738  0.81365241  0.83838023
      0.78457533  0.84980627  0.79249393  0.69020315  0.69593922  0.70758151
      0.70268831  0.91620627  0.73346864  0.86382644  0.80877728  0.72708753
      0.79822085  0.68333632  0.81699014  0.65100453  0.79809005  0.74702492
      0.77318569  0.83221179  0.66500492  0.68134548  0.7249286   0.59412132
      0.58191312  0.73142963  0.73142963  0.66251028  0.87152685  0.74107856
      0.60935684  0.87152685  0.77484517  0.88783675  0.81814153  0.54569789
      0.6122496   0.75613569  0.53511973  0.74556767  0.81814153  0.85773646
      0.6122496   0.64814153]

    Top 40 Sorted according to approximate lower bounds:

    596 18 Someone should develop an AI specifically for reading Terms & Conditions and flagging dubious parts.
    -------------
    2360 98 Porn is the only industry where it is not only acceptable but standard to separate people based on race, sex and sexual preference.
    -------------
    1918 101 All polls are biased towards people who are willing to take polls
    -------------
    948 50 They should charge less for drinks in the drive-thru because you can't refill them.
    -------------
    3740 239 When I was in elementary school and going through the DARE program, I was positive a gang of older kids was going to corner me and force me to smoke pot. Then I became an adult and realized nobody is giving free drugs to somebody that doesn't want them.
    -------------
    166 7 "Noted" is the professional way of saying "K".
    -------------
    29 0 Rewatching Mr. Bean, I've realised that the character is an eccentric genius and not a blithering idiot.
    -------------
    289 18 You've been doing weird cameos in your friends' dreams since kindergarten.
    -------------
    269 17 At some point every parent has stopped wiping their child's butt and hoped for the best.
    -------------
    121 6 Is it really fair to say a person over 85 has heart failure? Technically, that heart has done exceptionally well.
    -------------
    535 40 It's surreal to think that the sun and moon and stars we gaze up at are the same objects that have been observed for millenia, by everyone in the history of humanity from cavemen to Aristotle to Jesus to George Washington.
    -------------
    527 40 I wonder if America's internet is censored in a similar way that North Korea's is, but we have no idea of it happening.
    -------------
    1510 131 Kenny's family is poor because they're always paying for his funeral.
    -------------
    43 1 If I was as careful with my whole paycheck as I am with my last $20 I'd be a whole lot better off
    -------------
    162 10 Black hair ties are probably the most popular bracelets in the world.
    -------------
    107 6 The best answer to the interview question "What is your greatest weakness?" is "interviews".
    -------------
    127 8 Surfing the internet without ads feels like a summer evening without mosquitoes
    -------------
    159 12 I wonder if Superman ever put a pair of glasses on Lois Lane's dog, and she was like "what's this Clark? Did you get me a new dog?"
    -------------
    21 0 Sitting on a cold toilet seat or a warm toilet seat both suck for different reasons.
    -------------
    1414 157 My life is really like Rihanna's song, "just work work work work work" and the rest of it I can't really understand.
    -------------
    222 22 I'm honestly slightly concerned how often Reddit commenters make me laugh compared to my real life friends.
    -------------
    52 3 The world must have been a spookier place altogether when candles and gas lamps were the only sources of light at night besides the moon and the stars.
    -------------
    194 19 I have not been thankful enough in the last few years that the Black Eyed Peas are no longer ever on the radio
    -------------
    18 0 Living on the coast is having the window seat of the land you live on.
    -------------
    18 0 Binoculars are like walkie talkies for the deaf.
    -------------
    28 1 Now that I am a parent of multiple children I have realized that my parents were lying through their teeth when they said they didn't have a favorite.
    -------------
    16 0 I sneer at people who read tabloids, but every time I look someone up on Wikipedia the first thing I look for is what controversies they've been involved in.
    -------------
    1559 233 Kid's menus at restaurants should be smaller portions of the same adult dishes at lower prices and not the junk food that they usually offer.
    -------------
    1426 213 Eventually once all phones are waterproof we'll be able to push people into pools again
    -------------
    61 5 Myspace is so outdated that jokes about it being outdated has become outdated
    -------------
    52 4 As a kid, seeing someone step on a banana peel and not slip was a disappointment.
    -------------
    90 9 Yahoo!® is the RadioShack® of the Internet.
    -------------
    34 2 People who "tell it like it is" rarely do so to say something nice
    -------------
    39 3 Closing your eyes after turning off your alarm is a very dangerous game.
    -------------
    39 3 Your known 'first word' is the first word your parents heard you speak. In reality, it may have been a completely different word you said when you were alone.
    -------------
    87 10 "Smells Like Teen Spirit" is as old to listeners of today as "Yellow Submarine" was to listeners of 1991.
    -------------
    239 36 if an ocean didnt stop immigrants from coming to America what makes us think a wall will?
    -------------
    22 1 The phonebook was the biggest invasion of privacy that everyone was oddly ok with.
    -------------
    57 6 I'm actually the most productive when I procrastinate because I'm doing everything I possibly can to avoid the main task at hand.
    -------------
    57 6 You will never feel how long time is until you have allergies and snot slowly dripping out of your nostrils, while sitting in a classroom with no tissues.
    -------------

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```python
r_order = order[::-1][-40:]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order], capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30].replace("\n",""), ordered_contents) );
```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Extension to Starred rating systems

The above procedure works well for upvote-downvotes schemes, but what about systems that use star ratings, e.g. 5 star rating systems. Similar problems apply with simply taking the average: an item with two perfect ratings would beat an item with thousands of perfect ratings, but a single sub-perfect rating.

We can consider the upvote-downvote problem above as binary: 0 is a downvote, 1 if an upvote. A $N$-star rating system can be seen as a more continuous version of above, and we can set $n$ stars rewarded is equivalent to rewarding $\frac{n}{N}$. For example, in a 5-star system, a 2 star rating corresponds to 0.4. A perfect rating is a 1. We can use the same formula as before, but with $a,b$ defined differently:

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where

\begin{align}
& a = 1 + S \\\\
& b = 1 + N - S \\\\
\end{align}

where $N$ is the number of users who rated, and $S$ is the sum of all the ratings, under the equivalence scheme mentioned above.

##### Example: Counting Github stars

What is the average number of stars a Github repository has? How would you calculate this? There are over 6 million respositories, so there is more than enough data to invoke the Law of Large numbers. Let's start pulling some data. TODO

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting submissions formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead use a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally accurate?

```python
## Enter code here
import scipy.stats as stats
exp = stats.expon( scale=4 )
N = 1e5
X = exp.rvs( int(N) )
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by their percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

In August 2013, [a popular post](http://bpodgursky.wordpress.com/2013/08/21/average-income-per-programming-language/) on the average income per programmer of different languages was trending. Here's the summary chart: (reproduced without permission, cause when you lie with stats, you gunna get the hammer). What do you notice about the extremes?

------

#### Average household income by programming language

<table >
 <tr><td>Language</td><td>Average Household Income ($)</td><td>Data Points</td></tr>
 <tr><td>Puppet</td><td>87,589.29</td><td>112</td></tr>
 <tr><td>Haskell</td><td>89,973.82</td><td>191</td></tr>
 <tr><td>PHP</td><td>94,031.19</td><td>978</td></tr>
 <tr><td>CoffeeScript</td><td>94,890.80</td><td>435</td></tr>
 <tr><td>VimL</td><td>94,967.11</td><td>532</td></tr>
 <tr><td>Shell</td><td>96,930.54</td><td>979</td></tr>
 <tr><td>Lua</td><td>96,930.69</td><td>101</td></tr>
 <tr><td>Erlang</td><td>97,306.55</td><td>168</td></tr>
 <tr><td>Clojure</td><td>97,500.00</td><td>269</td></tr>
 <tr><td>Python</td><td>97,578.87</td><td>2314</td></tr>
 <tr><td>JavaScript</td><td>97,598.75</td><td>3443</td></tr>
 <tr><td>Emacs Lisp</td><td>97,774.65</td><td>355</td></tr>
 <tr><td>C#</td><td>97,823.31</td><td>665</td></tr>
 <tr><td>Ruby</td><td>98,238.74</td><td>3242</td></tr>
 <tr><td>C++</td><td>99,147.93</td><td>845</td></tr>
 <tr><td>CSS</td><td>99,881.40</td><td>527</td></tr>
 <tr><td>Perl</td><td>100,295.45</td><td>990</td></tr>
 <tr><td>C</td><td>100,766.51</td><td>2120</td></tr>
 <tr><td>Go</td><td>101,158.01</td><td>231</td></tr>
 <tr><td>Scala</td><td>101,460.91</td><td>243</td></tr>
 <tr><td>ColdFusion</td><td>101,536.70</td><td>109</td></tr>
 <tr><td>Objective-C</td><td>101,801.60</td><td>562</td></tr>
 <tr><td>Groovy</td><td>102,650.86</td><td>116</td></tr>
 <tr><td>Java</td><td>103,179.39</td><td>1402</td></tr>
 <tr><td>XSLT</td><td>106,199.19</td><td>123</td></tr>
 <tr><td>ActionScript</td><td>108,119.47</td><td>113</td></tr>
</table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. [Web](http://www.sloansportsconference.com/wp-content/uploads/2013/Going%20for%20Three%20Predicting%20the%20Likelihood%20of%20Field%20Goal%20Success%20with%20Logistic%20Regression.pdf). 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

<style>
    img{
        max-width:800px}
</style>


# MAE 6286: week 3 class practice 1

## Make things work first

The first stage of code and software development is alway to ***make it work***. So before you go home and read/learn the beauty of Python in [lesson 3 notebook](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/5a894f2c7c525a0a4bd5809bb78ba72ca616c40f/lessons/01_phugoid/01_03_PhugoidFullModel.ipynb), let's write some code that may not look elegant but can do the job, which is to solve the full phugoid motion with Euler method.

Once you finish the practice in this notebook, you then can go back to the lesson 3 notebook again and learn how to improve your code here with:

* `function`
* slicing with a `numpy` array
* array arithmetic

You can also how to make the figures nicer with more functions from `pyplot`.

### Full phugoid motion model

Here's the equation of full phugoid motion.

\begin{equation}
\left\{
\begin{aligned}
&v' = -g\sin{\theta} - \frac{C_D}{C_L}\frac{g}{v_t^2}v^2 \\
&\theta' = -\frac{g}{v}\cos{\theta} + \frac{g}{v_t^2}v \\
&x' = v\cos{\theta} \\
&y' = v\sin{\theta}
\end{aligned}
\right.
\end{equation}

And initial condition (i.e., the values of $v$, $\theta$, $x$, and $y$ at time $t=0$):

\begin{equation}
\left\{
\begin{aligned}
&v(0) = v_0 \\
&\theta(0) = \theta_0 \\
&x(0) = x_0 \\
&y(0) = y_0
\end{aligned}
\right.
\end{equation}

### Euler method

To solve the system of ODE of the full phugoid motion with Euler method, we re-write the eqautions:

\begin{equation}
\mathbf{u}^{n+1} = \mathbf{u}^n + \Delta t \cdot f(\mathbf{u^n})
\end{equation}

where

\begin{equation}
\mathbf{u}^n =
\left[
\begin{array}{c}
v^n \\
\theta^n \\
x^n \\
y^n
\end{array}
\right]
\text{, }
\mathbf{u}^{n+1} =
\left[
\begin{array}{c}
v^{n+1} \\
\theta^{n+1} \\
x^{n+1} \\
y^{n+1}
\end{array}
\right]
\text{, and }
f(\mathbf{u}^n) =
\left[
\begin{array}{c}
-g\sin{\theta^n} - \frac{C_D}{C_L}\frac{g}{v_t^2}\left(v^n\right)^2 \\
-\frac{g}{v^n}\cos{\theta^n} + \frac{g}{v_t^2}v^n \\
v^n\cos{\theta^n} \\
v^n\sin{\theta^n}
\end{array}
\right]
\end{equation}

### Import packages

In this practice, we'll use the `numpy` package and the sub-package `pyplot` from `matplotlib`.

```python
import numpy
from matplotlib import pyplot
```

## Part A. Implement a full phugoid motion solver with Euler method

### Set up constants

There are several constants we need to provide to the full phugoid motion model: gravity acceleration, drag coefficient, lift coefficient, and trim velocity.

In the code cell below, complete the code by assigning the following values to the corresponding variables:
* gravity acceleration: 9.81 m/s^2
* drag coefficient: 1.0 / 40.0
* lift coefficient: 1.0
* trim velocity: 30 m/s

```python
# gravity acceleration; unit: m/s^2
g

# drag coefficient; dimensionless
CD

# drag coefficient; dimensionless
CL

# trim velocity; unit: m / s
vt
```

Now if you look carefully into the equations, you should find that the equations don't really use the trim velocity $v_t$ at all. Instead, it uses the square of the trim velocity, $v_t^2$. So let's define another constant for the square of the trim velocity, and we won't have to calculate the square everytime.

**Hint**: do not assigne a scalar value directly to the variable; try to use the `vt` variable defined above.

```python
# the square of the trim velocity; do not assign value directly;
vt2
```

Now we need to determine the values for the initial conditions, i.e., $v_0$, $\theta_0$, $x_0$, and $y_0$. In this practice, we use:
* $v_0$ is equal to the trim velocity (unit: m/s)
* $\theta_0 = 0.0$ (unit: rad)
* $x_0 = 0.0$ (unit: m)
* $y_0 = 1000.0$ (unit: m)

```python
# velocity at t = 0s; unit: m / s
v0

# angle between trajectory and horizontal at t = 0s; unit: rad
theta0

# x-coordinate at t = 0s; unit: m
x0

# y-coordinate at t = 0s; unit: m
y0
```

We need to determine how long our phugoid motion lasts, that is, the final time $T$. In this practice, we want the final time $T$ to be $100$ (in seconds). And we also want to know the varaible values at every $0.1$ second, i.e., $\Delta t=0.1$. In other words, at the end of the phugoid motion calculation, we should have the values of $v$, $\theta$, $x$, and $y$ at $t=0,\ 0.1,\ 0.2,\ 0.3,\ \dots,\ 99.8,\ 99.9,\ 100$.

```python
# final T
T

# time step
dt

# the number of time steps from t=0 to t=100
N
```

Next, we want to have an array to hold all variable values at all time points. We know there will be $N$ time points on the timeline, and we know at each time point we have $4$ values (i.e., $v$, $\theta$, $x$, & $y$). This implies that we can use a $N$ by $4$ 2D array to hold the values. When we create the array, we don't have to worry about the values in the array, because we will update the array with correct values in Euler method. So let's simply use [`numpy.zeros`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html) to create such an array.

```python
# an array to hold variables at each time point
u
```

When we try to access a value in the 2D array `u` above, we need two indices: the first index represents the time point, and the second index represents which variable. For example, if we want to access the $x$ coordinate at the seventh time point (i.e., $t=6\Delta t$), we can use `u[6, 2]` to get the value. Note in Python, indices start from $0$.

Now, assign the initial conditions to the corresponding indices in the array `u`.

```python
# assign v0 to the correct index in u
u[?, ?]

# assign theta0 to the correct index in u
u

# assign x0 to the correct index in u
u

# assign y0 to the correct index in u
u
```

Now let's update the values at $n=1,\ 2,\ 3,\ \dots,\ N-1$, which represent the time points at $t=0.1,\ 0.2,\ \dots,\ T$. We use Euler method here.

```python
# iteration through Euler method
# the loop should loop through n=0, 1, 2, ..., N-2
for n in range(?):

    # extract v at the n-th time point from the array u
    v_n =

    # extract theta at the n-th time point from the array u
    theta_n =

    # extract x at the n-th time point from the array u
    x_n =

    # extract y at the n-th time point from the array u
    y_n =

    # f(u^n) in the right-hand-side
    f_n = numpy.array([
        ?,
        ?,
        ?,
        ?
    ])

    # update values at n+1 by Euler method: u^{n+1} = u^{n} + dt * f(u^{n})
    u[n+1] =
```

We should now have values at all time steps. Let's plot the $x$ and $y$ coordinates of the flight! Here we use slicing techniques to do the plotting. But the main purpose of this practice is to understand the logic of Euler method and to learn how to write a ODE solver from scratch, so I'll do the slicing and plotting for you. After executing the next cell, you should be able to see the plot, and the plot should be exactly the same as the one in lesson 3 notebook.

```python
pyplot.plot(u[:, 2], u[:, 3]);
```

## Part B. Calculate the order of convergence

We know the numerical error of Euler method is propotional to the size of $\Delta t$. That implies if we have a solution calculated with $\Delta t$, and now we calculate a new solution with $\Delta t_{new}=\frac{1}{2}\Delta t$, we should see the error reduces by a factor of 2.

But to calculate the error, we need to know tho analytical solution, which may not be known to us. So we also learned how to calculate the order of convergence, $p$, with anther method:

\begin{equation}
p = \frac{log_{10}\frac{f_3-f_2}{f_2-f_1}}{log_{10}r}
\end{equation}

where $f_1$, $f_2$, and $f_3$ represent the solutions with the smallest, medium, and largest $\Delta t$. And $r$ is the ratio between the $\Delta t$. That is, $\Delta t_{smallest}=\frac{1}{r}\Delta t_{medium}$, and $\Delta t_{medium}=\frac{1}{r}\Delta t_{largest}$. And

\begin{equation}
\left\{
\begin{aligned}
f_3-f_2=\sum_{n=0}^{N_3-1}\Delta t_3\left(abs(v_3^n-v_2^{r\cdot n})+abs(\theta_3^n-\theta_2^{r\cdot n})+abs(x_3^n-x_2^{r\cdot n})+abs(y_3^n-y_2^{r\cdot n})\right) \\
f_2-f_1=\sum_{n=0}^{N_2-1}\Delta t_2\left(abs(v_2^n-v_1^{r\cdot n})+abs(\theta_2^n-\theta_1^{r\cdot n})+abs(x_2^n-x_1^{r\cdot n})+abs(y_2^n-y_1^{r\cdot n})\right) \\
\end{aligned}
\right.
\end{equation}

Now use the logic developed in Part A to calculate the order of convergence, $p$, by completing the following code.

```python
# ratio
r =

# time steps
dt_coarsest = 0.004
dt_medium =
dt_finest =

# N for different time step sizes
N_coarsest =
N_medium =
N_finest =

# initial u arrays holding variables to all time points
u_coarsest =
u_medium =
u_finest =

# assign initial conditions to correct locations in u_coarsest, u_medium, and u_finest
# <-- COMPLETE YOU CODE HERE -->

# iteration through Euler method for dt_coarsest
# n = 0, 1, 2, ..., N_coarsest-2
# <-- COMPLETE YOU CODE HERE -->

# iteration through Euler method for dt_medium
# n = 0, 1, 2, ..., N_medium-2
# <-- COMPLETE YOU CODE HERE -->

# iteration through Euler method for dt_finest
# n = 0, 1, 2, ..., N_finest-2
# <-- COMPLETE YOU CODE HERE -->
```

We now have $f_1$, $f_2$, and $f_3$, which are the `u_finest`, `u_medium`, and `u_coarsest` in our code. Let try to calculate $f_3-f_2$ and $f_2-f_1$.

```python
# initialize the variable f32 with zero to represent f3-f2
f32 =
```

```python
# loop through n=0, 1, ..., N_coarsest-1 to sum up the difference in solutions
for n in range(N_coarsest):

    # add the absolute value between v with largest and medium dt at time point n
    f32 = f32 + numpy.abs(u_coarsest[n, 0] - u_medium[2*n, 0])

    # <-- complete the code to also add the absolute difference in theta, x, and y -->
```

```python
# Now multiply the f32 with coarsest dt
f32 = f32 * dt_coarsest
```

```python
# initialize the variable f21 with zero to represent f2-f1
f21 =
```

```python
# loop through n=0, 1, ..., N_medium-1 to sum up the difference in solutions

# <-- complete the code following the same loging in calculating f32 -->

```

```python
# Now multiply the f32 with medium dt

# <-- complete the code following the same loging in calculating f32 -->
```

```python
# calculate p = log(f32/f21) / log(r); you can use numpy.log10
p =
```

```python
# after printing out the p, you should get a value around 1.0139
print(p)
```


# Monte Carlo Calculation of π

 Christina C. Lee

Category: Numerics

## Monte Carlo- Random Numbers to Improve Calculations

When one hears "Monte Carlo", most people might think of something like this:

 My Mom and I touring Monte Carlo, Monaco

Monte Carlo, Monaco: known for extremely large amounts of money, car racing, no income taxes,and copious gambling.

In addition to Monaco, Europe, Las Vegas decided to host a Monte Carlo-themed casino as well.  So during the Manhattan project, when the best minds in the United States were camped out in the New Mexican desert, they had plenty of inspiration from Las Vegas, and plenty of difficult problems to work on in the form of quantifying the inner workings of nuclei.  Enrico Fermi first played with these ideas, but Stanislaw Ulam invented the modern Monte Carlo Markov Chain later.

At the same time, these scientists now had computers at their disposal.  John von Neumann programmed Ulam's algorithm onto ENIAC (Electronic Numerical Integrator and Computer), the very first electronic, general purpose computer, even though it did still run on vacuum tubes.

That still doesn't answer, why do random numbers actually help us solve problems?

Imagine you are visiting a new city for the first time (maybe Monte Carlo). You only have a day or two, and you want to really get to know the city.  You have two options for your visit

* Hit the tourist sites you researched online
* Wander around.  Try and communicate with the locals.  Find an out-of-the-way restaurant and sample food not tamed for foreigners.  Watch people interact.  Get lost.

Both are legitimate ways to see the city.  But depending on what you want, you might choose a different option.  The same goes for exploring physics problems.  Sometimes you want to go in and study just everything you knew about beforehand, but sometimes you need to wander around, not hit everything, but get a better feeling for what everything might be like.

## Buffon's Needle: Calculation of π
Even back in the 18th century, Georges-Louis Leclerc, Comte de Buffon posed a problem in geometric probability.  Nowdays, we use a slightly different version of that problem to calculate π and illustrate Monte Carlo simulations.

Suppose we have a square dartboard, and someone with really bad, completely random aim, even though he/she always at least hits inside the dartboard.  We then inscribe a circle inside that dartboard. After an infinity number of hits, what is the ratio between hits in the circle, and hits in the square?

\begin{equation}
    f= \frac{N_{circle}}{N_{square}} =\frac{\text{Area of circle}}{\text{Area of square}} =\frac{\pi r^2}{4 r^2}= \frac{\pi}{4}
\end{equation}
\begin{equation}
    \pi = 4 f
\end{equation}

## Onto the Code!

```julia
using Statistics
using Plots
gr()
```

    Plots.GRBackend()

We will generate our random numbers on the unit interval.  Thus the radius in our circumstance is $.5$.

 Write a function `incircle(r2)` such that if `r2` is in the circle, it returns true, else, it returns false.  We will use this with the julia function `filter`.  Assume `r2` is the radius squared, and already centered around the middle of the unit circle

```julia
function incircle(r2)
    if r2<.25
        return true
    else
        return false
    end
end
```

    incircle (generic function with 1 method)

```julia
#The number of darts we will throw at the board.  We will see how accurate different numbers are
N=[10,25,50,75,100,250,500,750,1000,2500,5000,7500,10000];
# We will perform each number multiple times in order to calculate error bars
M=15;
```

```julia
πapprox=zeros(Float64,length(N),M);

for ii in 1:length(N)
    for jj in 1:M

        #popular our array with random numbers on the unit interval
        X=rand(N[ii],2)

        #calculate their radius squared
        R2=(X[:,1].-0.5).^2.0.+(X[:,2].-0.5).^2

        # 4*number in circle / total number
        πapprox[ii,jj]=4.0*length(filter(incircle,R2))/N[ii];

    end
end
```

```julia
# Get our averages and standard deviations
πave=mean(πapprox,dims=2);
πstd=std(πapprox,dims=2);
```

## Analysis

So that was a nice, short little piece of code.  Lets plot it now to see means.

```julia
plot(N,π*ones(length(N)),xscale=:log10);

for j in 1:M
    scatter!(N,πapprox[:,j]);
end

scatter!(N,πave,yerr=πstd)
plot!(xlabel="Number of Darts",ylabel="π Estimate",
        title="Monte Carlo Estimate of π",legend=:false)
```

When we have fewer numbers of points, our estimates vary much more wildly, and much further from 3.1415926 .
But, at least, the guesses from our different runs all seem equally distributed around the correct value, so it seems we have no systematic error.

As we get up to $10^4$, our estimate starts getting much more accurate and consistent.

```julia
plot(N,πstd,xscale=:log10)

plot!(xlabel="N points"
    ,ylabel="standard deviation"
    ,title="Dependence of Monte Carlo Error on Number of Points")
```

So what we guessed in the first plot about dispersion in estimate, we quantify here in this plot.  When we only have 10 darts, the guesses vary by up to .3, but when we get down to 1,000 trials, we are starting to be consistent to .0002

```julia
plot(N,πave,xscale=:log10,label="π Guess")
plot!(N,π*ones(length(N)),label="π")

plot!(xlabel="N steps"
    ,ylabel="Average of 15 runs"
    ,title="Overall Averages")
```

Now lets just make a graphical representation of what we've been doing this whole time.  Plot our points on unit square, and color the ones inside a circle a different color.

```julia
X=rand(1000);
Y=rand(1000);
R2=(X.-0.5).^2.0.+(Y.-0.5).^2;
Xc=[];
Yc=[]
for ii in 1:length(X)
    if R2[ii]<.25
        push!(Xc,X[ii]);
        push!(Yc,Y[ii]);
    end
end

```

```julia
scatter(X,Y)
scatter!(Xc,Yc)
plot!(aspect_ratio=1,xlabel="X",ylabel="Y",legend=:false,
        title="Dartboard")
```

That's all folks!
Now here's a picture of some pie to congratulate you on calculating π.

By Scott Bauer, USDA ARS - This image was released by the Agricultural Research Service, the research agency of the United States Department of Agriculture, with the ID K7252-47 (next).This tag does not indicate the copyright status of the attached work. A normal copyright tag is still required. See Commons:Licensing for more information.English | français | македонски | +/−, Public Domain, https://commons.wikimedia.org/w/index.php?curid=264106

@article{Markov,
author = {A.  Markov , A},
year = {2006},
month = {12},
pages = {591 - 600},
title = {An Example of Statistical Investigation of the Text Eugene Onegin Concerning the Connection of Samples in Chains},
volume = {19},
journal = {Science in Context},
doi = {10.1017/S0269889706001074}
}

```julia

```


# Characterization of Discrete Systems

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Difference Equations

The relation between input $x[k]$ and output $y[k] = \mathcal{H} \{ x[k] \}$ of a discrete system can be described by relating actual and past samples of the input and output signal to each other. When restricting the relation to a linear superposition of these, this results in a [linear difference equation](https://en.wikipedia.org/wiki/Linear_difference_equation). The relation of difference equations to [linear time-invariant (LTI) systems](https://en.wikipedia.org/wiki/LTI_system_theory) is discussed in the following.

### Linear Time-Invariant Systems

The in-/output relation of a system in terms of a linear difference equation with constant coefficients $a_n, b_m \in \mathbb{C}$ reads

\begin{equation}
\sum_{n = 0}^{N} a_n \; y[k - n] = \sum_{m = 0}^{M} b_m \; x[k -m]
\end{equation}

where the order of the difference equation is given by $N$. LTI systems are an important class of systems. In order to show that linear difference equations with constant coefficients represent LTI systems, the linearity and time-invariance property has to be checked. A system is linear when the superposition principle holds

\begin{equation}
\mathcal{H} \{ A x_1[k] + B x_2[k] \} = A \cdot \mathcal{H} \{ x_1[k] \} + B \cdot \mathcal{H} \{ x_2[k] \}
\end{equation}

for arbitraty signals $x_1[k], x_2[k] \in \mathbb{C}$ and coefficients $A, B \in \mathbb{C}$. Above condition is fulfilled by a linear difference equation, since the [summation](https://en.wikipedia.org/wiki/Summation) on left- and right-hand side is a linear operation. A system is time-invariant when

\begin{equation}
y[k - \kappa] = \mathcal{H} \{ x[k - \kappa] \}
\end{equation}

with $y[k] = \mathcal{H} \{ x[k] \}$ holds for $\kappa \in \mathbb{Z}$. By substitution of $k$ by $k -\kappa$ in the linear difference equation above it is straightforward to show that the time-invariance is fulfilled.

### Computation of the Output Signal

The solution of above difference equation is derived by extracting $y[k]$ from the sum on the left-hand-side and rearranging terms. This results in

\begin{equation}
y[k] = \frac{1}{a_0} \left( \sum_{m=0}^{M} b_m \; x[k-m] - \sum_{n=1}^{N} a_n \; y[k-n] \right)
\end{equation}

for $a_0 \neq 0$. The output signal is given by a weighted superposition of the actual and past samples of the input signal $x[k]$ and past samples of the output signal $y[k]$. The second sum is known as *recursive* part since it constitutes a feedback of past output samples. The dependency on the input samples (the first sum) is known as *non-recursive* part. For a unique solution of the difference equation, $N$ initial conditions are required. Under the assumption that the input signal of the system $x[k] = 0$ for $k < 0$ these are given by $y[0], y[-1], \dots, y[-N + 1]$. Other cases can be handled accordingly.

It can be concluded that linear difference equations with constant coefficients represent LTI systems. The coefficients together with the initial conditions characterize an LTI system fully. However, not all LTI systems can be described by linear difference equations of finite order $N$. Idealized systems may result in a linear difference equation of infinite order, for instance.

### Recursive and Non-Recursive Systems

Above analysis of the solution of a linear difference equation revealed two contributions to the output signal $y[k]$: a non-recursive part which depends only on the current and past input samples $x[k]$ and a recursive part which depends on past output samples $y[k]$. A system is said to be

* non-recursive if its output depends only on the input signal $x[k]$. This implies that $a_n = 0$ for $n > 0$.
* recursive if its output depends on the input signal $x[k]$ and past output samples $y[k]$. This implies that at least one $a_n \neq 0$ for $n > 0$.

The classification into non-recursive and recursive systems has far reaching implications on the theory and practical realization of recursive systems. The feedback of past output samples in recursive systems for instance, may lead to instabilities and numerical issues. The former are discussed in detail later.

### Examples

Some examples for discrete systems defined by their input/output relation formulated in terms of a linear difference equation are given in the following.

#### Delay

The output signal $y[k] = \mathcal{H} \{ x[k] \}$ of a discrete system which delays the input $x[k]$ by $\kappa \in \mathbb{N}$ samples is given as

\begin{equation}
y[k] = x[k - \kappa]
\end{equation}

The order of the system is $N=0$. The coefficients of the linear difference equation read

\begin{align}
a_0 &= 1 \\
b_m &= \begin{cases}
1 & \text{for } m=\kappa \\
0 & \text{otherwise}
\end{cases}
\end{align}

The system is non-recursive as can be deduced from its coefficients.

The output signal $y[k]$ for a cosine signal at the input $x[k] = \cos[\Omega k]$ is computed by solving the difference equation for zero initial conditions and $\kappa = 5$. The Python module [`scipy.signal`](http://docs.scipy.org/doc/scipy/reference/signal.html) offers various functionality for continuous and discrete systems. The function [`scipy.signal.filter`](http://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.lfilter.html#scipy.signal.lfilter) implements above formula for the computation of the output signal $y[k]$ of a discrete LTI system given its coefficients and initial conditions. The coefficients $a_n$ and $b_m$ of the system are stored in the vectors `a` and `b`.

```python
%matplotlib inline
import numpy as np
from scipy import signal
import matplotlib.pyplot as plt

a = 1.0
b = [0.0, 0.0, 0.0, 0.0, 1.0]

k = np.arange(30)
x = np.cos(2*np.pi/15 * k)
y = signal.lfilter(b, a, x)
```

The input $x[k]$ and output signal $y[k]$ are plotted for illustration.

```python
plt.figure(figsize=(10,4))
plt.subplot(121)
plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel(r'$x[k]$')
plt.ylim([-1.1, 1.1])

plt.subplot(122)
plt.stem(k, y)
plt.xlabel('$k$')
plt.ylabel('$y[k]$')
plt.ylim([-1.1, 1.1])
plt.tight_layout()
```

**Exercise**

* Change the delay $\kappa$ of the system and rerun the example.

#### Moving Average

The [moving average](https://en.wikipedia.org/wiki/Moving_average) of a time-series is frequently used to smooth out short-term fluctuations and to work out longer-term trends. A system which calculates the moving average of its input signal $x[k]$ is represented by

\begin{equation}
y[k] = \frac{1}{N} \sum_{\kappa = 0}^{N-1} x[k - \kappa]
\end{equation}

The order of the system is $N=0$. The coefficients of the linear difference equation read

\begin{align}
a_0 &= 1 \\
b_m &= \begin{cases}
\frac{1}{N} & \text{for } 0 \leq m < N \\
0 & \text{otherwise}
\end{cases}
\end{align}

The system is non-recursive as can be deduced from its coefficients. The output of the system for a cosine signal at the input superimposed with noise is computed as illustration of the functionality of the moving average.

```python
np.random.seed(seed=0)

N = 10
a = [1.0]
b = 1/N * np.ones(N)

k = np.arange(0, 60)
x = np.cos(2*np.pi/30 * k) + .2 * np.random.normal(size=(len(k)))
y = signal.lfilter(b, a, x)

plt.figure(figsize=(10,4))
plt.subplot(121)
plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel(r'$x[k]$')
plt.ylim([-1.5, 1.5])

plt.subplot(122)
plt.stem(k, y)
plt.xlabel('$k$')
plt.ylabel('$y[k]$')
plt.ylim([-1.5, 1.5])
plt.tight_layout()
```

**Exercise**

* Change the length `N` of the moving average. How does the output of the system change?

#### Second-Order System

The recursive LTI system with the following linear difference equation is given

\begin{equation}
y[k] - y[k-1] + \frac{1}{2} y[k-2] = x[k]
\end{equation}

The order of the system is $N=2$. The coefficients of the system can be derived from its difference equation as $a_0 = 1$, $a_1 = -1$, $a_2 = \frac{1}{2}$ and $b_0 = 1$. The output signal for a rectangular signal $\text{rect}_{20}[k]$ at the input $x[k]$ is computed and plotted for zero initial conditions.

```python
def rect(k, N):
    return np.where((k>=0) & (k<N), 1.0, 0.0)

a = [1.0, -1.0, 1/2]
b = [1.0]

k = np.arange(0, 40)
x = rect(k, 20)
y = signal.lfilter(b, a, x)

plt.figure(figsize=(10,4))
plt.subplot(121)
plt.stem(k, x)
plt.xlabel('$k$')
plt.ylabel(r'$x[k]$')
plt.ylim([-0.7, 2.6])

plt.subplot(122)
plt.stem(k, y)
plt.xlabel('$k$')
plt.ylabel('$y[k]$')
plt.ylim([-0.7, 2.6])
plt.tight_layout()
```

### Copyright

  To the extent possible under law,
  the person who associated CC0
  with this work has waived all copyright and related or neighboring
  rights to this work.