# Model Project

In the following project we will analyse the basic solow-model as well as the extended R&D based growth model.

Import packages and set magics:

```python
from __future__ import print_function
import numpy as np
import sympy as sm
import pandas as pd
import matplotlib.pyplot as plt
import ipywidgets as widgets
from ipywidgets import interact, interactive, fixed, interact_manual
from scipy import optimize

sm.init_printing(use_unicode=True)
```

# Model setup

We consider the **basic Solow-model** as presented by 'Whitta-Jacobsen and Birch Sørensen'. The complete basic model looks as follows:

1. $Y_t = BK_t^\alpha L_t^{1-\alpha}$
* $r_t = \alpha B\left(\frac{K_t}{L_t}\right)^{\alpha-1}$
* $w_t = (1-\alpha)B\left(\frac{K_t}{L_t}\right)^\alpha$
* $S_t = s Y_t$
* $K_{t+1} - K_t = S_t - \delta K_t$
* $L_{t+1} = (1+n)L_t$

Where $K_t$ is capital, $L_t$ is labor and $Y_t$ is output

We note that the production function has constant return to scale, since:

$ (\gamma K_t)^\alpha (\gamma L_t)^{(1-\alpha)} = \gamma^\alpha \gamma^{(1-\alpha)} K_t^\alpha L_t^{(1-\alpha)} = \gamma Y_t $

Production function also has positive decreasing returns to both capital and labor since

$ \frac{\delta Y}{\delta K} = r_t > 0 $, and $\frac{\delta^2 Y}{\delta K^2} = r_t' < 0$

$ \frac{\delta Y}{\delta L} = w_t > 0 $, and $\frac{\delta^2 Y}{\delta L^2} = w_t' < 0$

Defining per capita capital and per capita labor:

$ \big(\frac{Y_t}{L_t}\big) = y_t $

$ \big(\frac{K_t}{L_t}\big) = k_t $

Which yields the following output per capita function:

7. $ y_t = Bk_t^\alpha $

By inserting the savings behavior (4) into the capital equation (5), dividing both sides with $L_{t+1}$ from (6), and subsituting in the expression for $y_t$ (7) we arrive at the transition equation:

$ k_{t+1} = \frac{1}{1+n} \left[sBk_t^\alpha + (1-\delta) k_t \right] $

By subtracting $k_t$ from both sides, we obtain the so-called Solow-equation:

8. $ k_{t+1} - k_t = \frac{1}{1+n} \left[sBk_t^\alpha - (n+\delta) k_t \right] $

# Steady state

## Analytical solution

We start by solving the steady state analytically. For this purpose we use the sumpy function, and solve for optimal level of capital from the Solow equation (8).

$$ k_{t+1} - k_t = \frac{1}{1+n} \left[sBk_t^\alpha - (n+\delta) k_t \right] $$

We note that the steady state is charaterized by no change in capital per capita, which means that $k_{t+1} = k_t = k^*$. Hence the solow equation for steady state simplyfies into:

$ 0 = \frac{1}{1+n} \left[sBk^{*\alpha} - (n+\delta) k^* \right] $

$ sBk^{*\alpha} = (n+\delta) k^* $

We can now solve for the steady state level of capital per capita with the sympy function. First we define the symbols used for the sympy solution:

```python
Y = sm.symbols("Y_t")         # Output
y = sm.symbols("y_t")         # Output per capita
K = sm.symbols("K_t")         # Capital
k = sm.symbols("k_t")         # Capital per capita
s = sm.symbols("s")           # Savings rate
n = sm.symbols("n")           # Population growth
B = sm.symbols("B")           # Total factor productivity
L = sm.symbols("L_t")         # Labor
alpha = sm.symbols("alpha")   # Capital intensity
delta = sm.symbols("delta")   # Depreciation rate
kstar = sm.symbols("k^*")     # Steady state for capital per capita
ystar = sm.symbols("y^*")     # Steady state for output per capita
```

We then define the steady state equation. We do this with the sm.Eq function inserting the transition equation where  $k_{t+1} = k_t = k^*$:

```python
steadystate = sm.Eq(kstar, (s*B*kstar**alpha+(1-delta)*kstar)/(1+n))
```

and solve it using the Sympy function sm.solve:

```python
steadystate_k = sm.solve(steadystate,kstar)[0]
sm.Eq(kstar, steadystate_k)
```

We now have en expression for the steady state level of capital per capita. From (7) we know that the steady state level of output per capita kan be found as:

```python
steadystate_y = sm.Eq(ystar, B*steadystate_k**alpha)
steadystate_y
```

We now have expressions for both capital and output per capita.

## Numerical solution and interactive visualization

We solve for the real numbers of the steady state by finding the root of the equation $ sBk^{*\alpha} = (n+\delta) k^* $. We do this by using the biseciton method. The bisection method is away of finding the root(s) of some function. What the bisect function does is to go through the function of interest, and investigate every bit of the function partially. The steps can be described in the following way:

1. First we define a function f for which the product of the function of the lower bound in the interval a, and the upper bound b, is greater than zero. Is this fulfilled, it means that there cannot be any roots in this particular interval since the function does not cross the x axis.
* Next step is to tell the function to discard all found values where this is the case.
* We now create a for-loop in which we calculate a number of midpoints by the average of 'low' and 'high'.
* For as long as the productof f(low) and f(midpoint) is >0 we keep replacing the low with themidpoint value. We do this until the product is negative, meaning there is root in the interval. When this is the case we instead replace the high value with midpoint, which means that the for loop converges towards the true value of the root.
* We then define our function we need to find the roots for ($ sBk^{*\alpha} = (n+\delta) k^* $) and call the bisect function we just created for some interval 0.1-100. This returns the steady state level of capital per capita.

```python
def samesign(a, b):
        return a * b > 0

def bisect(func, low, high):

    'Find root of continuous function where f(low) and f(high) have opposite signs'

    assert not samesign(func(low), func(high))

    for i in range(100):
        midpoint = (low + high) / 2.0
        if samesign(func(low), func(midpoint)):
            low = midpoint
        else:
            high = midpoint

    return midpoint

def f(k, alpha=1/3, s=0.34, B=1, n=0.01, delta=0.03):
        return s*B*k**alpha - (n+delta)*k

k = bisect(f, 0.1, 100)
print('Optimal steady state level of capital is: k_star =', k)
```

    Optimal steady state level of capital is: k_star = 24.781545553092528

The same result can simply be obtained by calling the pre defined optimizer bisect function. We see below that this yields the exact same result. For simplicity in the code, we will from now on call the optimizer function (which does the exact same steps as described above) when finding roots.

```python
def bisect_ss(a,b, alpha=1/3, s=0.34, B=1, n=0.01, delta=0.03):

    """
    Input. Parameters of interest, and the interval [a,b]. This determines the interval on which the function evaluates

    Output: Steady state value of capital per capita.

    """
    result = optimize.bisect(lambda k: s*B*k**alpha - (n+delta)*k, a,b, full_output=False)
    return result

print(f'Steady state capital per capita for given parameter values is: {bisect_ss(1,100):.4f}')
```

    Steady state capital per capita for given parameter values is: 24.7815

We now add en interactive element to the plot below. The following code allows the reader to use sliders to change paramter values $\alpha, s, n, \delta$ and $T$. The plot is a combination of the classic solow diagram where the intersection between $sBk^{\alpha}$ and $(n+\delta)k$ denotes the steady state level of capital per capita; and then an output diagram, where the steady state output $y_t$ shows as a function of capital per capita $k_t$. By combining the classic solow diagram with an output diagram, we allow to change the paramters, observe the intersection of the blue and the black line, and then go vertically up on the red line to find the corresponding steady state level of output per capita.

First we define the function that creates the plot:

```python
def simulate_solow(k,y,alpha,s,B,n,delta,T):
    """
    Input: varibales and paramters of interest

    Output: A compbination of the solow diagram and an output diagram.

    """

    # Create lists for k, diagonal line and y
    k_list = [k]
    diagonal = [0]
    y_list = [y]

    # Create the diagonal line
    for t in range(1,T):
        line = (n + delta)*t
        diagonal.append(line)

    # Create the Solow movement
    for t in range(1,T):
        k_plus = s*B*t**alpha
        k_list.append(k_plus)

    # Create the output graph
    for t in range(1,T):
        y_plus = B*t**alpha
        y_list.append(y_plus)

    # Plot
    plt.figure(figsize=(5,5))
    plt.plot(k_list[:T], label='$sBk_t^{\u03B1}$', color = 'blue')
    plt.plot(diagonal[:T], label = '$(n+\delta)k_t$', color = 'black')
    plt.plot(y_list[:T], label='$y_t$', color = 'red')
    plt.xlim(0,T)
    plt.ylim(0,6)
    plt.xlabel('$k_t$')
    plt.grid(True)
    plt.legend()

    return plt.show()
```

And then we add the interactive sliders:

```python
widgets.interact(simulate_solow,
                k     = widgets.fixed(0),
                y     = widgets.fixed(0),
                alpha = widgets.FloatSlider(description = '$\u03B1$' , min = 0 ,    max = 0.9 , step = 0.05 , value = 1/3),
                delta = widgets.FloatSlider(description = '$\u03B4$' , min = 0 ,    max = 0.1 , step = 0.01 , value = 0.03),
                s     = widgets.FloatSlider(description = 's' ,        min = 0.01 , max = 0.8 , step = 0.05 , value = 0.34),
                n     = widgets.FloatSlider(description ='$n$',        min = 0.01 , max = 0.1 , step = 0.005, value = 0.01),
                B     = widgets.fixed(1),
                T     = widgets.IntSlider(description='$T$' ,          min = 1,     max = 250, step = 1,    value = 100))
```

    interactive(children=(FloatSlider(value=0.3333333333333333, description='$α$', max=0.9, step=0.05), FloatSlide…

    <function __main__.simulate_solow(k, y, alpha, s, B, n, delta, T)>

# Model extension: The R&D based growth model

The extension of the solow model we choose is the R&D based growth model where we add technology $A_t$ to the model. The full model now looks as follows:

1. $Y_t = K_t^\alpha (A_t L_{Y_t})^{1-\alpha}$
* $A_{t+1} - A_t = \rho A_t^{\phi} L_{A_t}^{\lambda}$
* $K_{t+1} = sY_t + (1-\delta) K_t$
* $L_{t+1} = (1+n)L_t$
* $L_{Y_t} + L_{A_t} = L_t$
* $L_{A_t} = s_R L_t$

From equation (2) we introduce the technology growth, $g_t$.

$ g_t \equiv \frac{A_{t+1}-A_t}{A_t} = \rho A_t^{\phi-1} L_{A_t}^\lambda = \rho A_t^{\phi-1} (s_R L_t)^\lambda$

Defining capital and labor per effective capita:

$ \big(\frac{Y_t}{A_t L_t}\big) = \tilde{y}_t $

$ \big(\frac{K_t}{A_t L_t}\big) = \tilde{k}_t $

Which yields the following output per effective capita function:

7. $ \tilde{y}_t = \tilde{k}_t^\alpha (1-s_R)^{1-\alpha} $

The transition equation becomes the following:

$ \tilde{k}_{t+1} = \frac{1}{(1+n)(1+g_t)} [s\tilde{k}_t^\alpha (1-s_R)^{1-\alpha} + (1-\delta) \tilde{k}_t]$

## Analytical solution

The approah to solving this model is exactly the same as the basic solow model above. We start by defining all sm.symbols:

```python
A = sm.symbols("A_t")                  # Technology
Ly = sm.symbols("L_{Y_t}")             # Labor input, (non-technology)
La = sm.symbols("L_{A_t}")             # Labor input, technology
sr = sm.symbols("s_R")                 # Labor share in research
g = sm.symbols("g_t")                  # Technology growth
phi = sm.symbols("phi")
rho = sm.symbols("rho")
lambd = sm.symbols("lambda")
ktilde = sm.symbols("ktilde_t")        # Capital per effective capita
ytilde = sm.symbols("ytilde_t")        # Output per effective capita
ktildestar = sm.symbols("ktilde^*")    # Steady state capital per effective capita
ytildestar = sm.symbols("ytilde^*")    # Steady state capital per effective capita
```

Assuming that $\tilde{k}_{t+1} = \tilde{k}_t = \tilde{k}^*$ allows us to set up the steady state equation from the transition equation:

```python
steadystatetilde = sm.Eq(ktildestar, (s*ktildestar**alpha*(1-sr)**(1-alpha)+(1-delta)*ktildestar)/((1+n)*(1+g)))
```

Solving for steady state capital per effective capita with sm.solve:

```python
steadystatetilde_k = sm.solve(steadystatetilde,ktildestar)[0]
sm.Eq(ktildestar, steadystatetilde_k)
```

... and for output per effective capita:

```python
steadystatetilde_y = sm.Eq(ytildestar, steadystatetilde_k**alpha*(1-sr)**(1-alpha))
steadystatetilde_y
```

## Numerical solution

```python

```

```python
def bisect_sstilde(a,b, alpha=1/3, s=0.34, sr=0.05 , n=0.01, g=0.02, delta=0.03):

    """
    Input. Parameters of interest, and the interval [a,b]. This determines the interval on which the function evaluates

    Output: Steady state value of capital per capita.

    """
    result_tilde = optimize.bisect(lambda ktilde: s*ktilde**alpha*(1-sr)**(1-alpha) - (delta+g+n+n*g)*ktilde, a,b, full_output=False)
    return result_tilde

print(f'Steady state capital per effective capita for given parameter values is: {bisect_sstilde(1,100):.4f}')
```

    Steady state capital per effective capita for given parameter values is: 12.7511

We now add the interactive plot, allowing us to change the same paramters as before, but now also the technology growth $g_t$.

First the plot function:

```python
def simulate_solowRD(ktilde,ytilde,alpha,s,sr,n,g,delta,T):
    """
    Input: varibales and paramters of interest

    Output: A compbination of the solow diagram and an output diagram.

    """

    # Create lists for k, diagonal line and y
    ktilde_list = [ktilde]
    diagonaltilde = [0]
    ytilde_list = [ytilde]

    # Create the diagonal line
    for t in range(1,T):
        linetilde = (delta+g+n+n*g)*t
        diagonaltilde.append(linetilde)

    # Create the Solow movement
    for t in range(1,T):
        ktilde_plus = s*t**alpha*(1-sr)**(1-alpha)
        ktilde_list.append(ktilde_plus)

    # Create the output graph
    for t in range(1,T):
        ytilde_plus = t**alpha*(1-sr)**(1-alpha)
        ytilde_list.append(ytilde_plus)

    # Plot
    plt.figure(figsize=(5,5))
    plt.plot(ktilde_list[:T], label='$s \~{k}_t^{\u03B1}(1-s_R)^{1-\u03B1}$', color = 'blue')
    plt.plot(diagonaltilde[:T], label = '$(\delta+g+n+ng) \~{k}_t$', color = 'black')
    plt.plot(ytilde_list[:T], label='$\~{y}_t$', color = 'red')
    plt.xlim(0,T)
    plt.ylim(0,6)
    plt.xlabel('$\~{k}_t$')
    plt.grid(True)
    plt.legend()

    return plt.show()
```

... And the interactive sliders:

```python
widgets.interact(simulate_solowRD,
                ktilde     = widgets.fixed(0),
                ytilde     = widgets.fixed(0),
                alpha = widgets.FloatSlider(description = '$\u03B1$' , min = 0 ,    max = 0.9 , step = 0.05 , value = 1/3),
                delta = widgets.FloatSlider(description = '$\u03B4$' , min = 0 ,    max = 0.1 , step = 0.01 , value = 0.03),
                s     = widgets.FloatSlider(description = 's' ,        min = 0.01 , max = 0.8 , step = 0.05 , value = 0.34),
                n     = widgets.FloatSlider(description ='$n$',        min = 0.01 , max = 0.1 , step = 0.005, value = 0.01),
                g     = widgets.FloatSlider(description ='$g$',        min = 0.01 , max = 0.1 , step = 0.005, value = 0.02),
                T     = widgets.IntSlider(description='$T$' ,          min = 1,     max = 250, step = 1,    value = 100),
                sr    = widgets.FloatSlider(description ='$s_R$',      min = 0.01 , max = 0.5 , step = 0.01, value = 0.05))
```

    interactive(children=(FloatSlider(value=0.3333333333333333, description='$α$', max=0.9, step=0.05), FloatSlide…

    <function __main__.simulate_solowRD(ktilde, ytilde, alpha, s, sr, n, g, delta, T)>

# Conclusion

We can conclude that we succesfully created an algorithm that solved the basic Solow model for steady state levels of per-capita capital and output. We also succesfully created interactive sliders, allowing the reader to change the paramters of the model as the reader wishes. The model is set up with examples of realistic paramter values, which can then easily be changed after the plot is created.
After analyzing the basic solow model, we looked into the extended R&D based growth model, including technology and technology growth. We can conclude that the method of finding steady state levels does not change much, but that the levels themselves in the extended model depend on both technology growth $g$ and also the share of labor used in Research and development $s_R$.

```python

```


#Chapter 4
______

##The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been used this simple idea in every example thus far.

###The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear be examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for almost *any distribution*, minus some important cases we will encounter later.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size = 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to it's parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```
%pylab inline

figsize( 12.5, 5 )
import pymc as mc

sample_size = 100000
expected_value = lambda_ = 4.5
poi = mc.rpoisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, size = sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average),
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend();
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

The above formulae is interpretable as a distance away from the true value (on average), for some $N$. (We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

By computing the above many, $N_y$, times (remember, it is random), and averaging them:

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \rightarrow E[ Y_k ] = E\;\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \right]$$

Finally, taking the square root:

$$ \sqrt{\frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k} \approx D(N) $$

```
figsize( 12.5, 4)

N_Y = 250 #use this many to approximate D(N)
N_array = np.arange( 1000, 50000, 2500 ) #use this many samples in the approx. to the variance.
D_N_results = np.zeros( len( N_array ) )

lambda_ = 4.5
expected_value = lambda_ #for X ~ Poi(lambda) , E[ X ] = lambda

def D_N( n ):
    """
    This function approx. D_n, the average variance of using n samples.
    """
    Z = poi( lambda_, size = (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for i,n in enumerate(N_array):
    D_N_results[i] =  D_N(n)

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 2, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " );
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of converge to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 10, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 10 ) =  \sum_{i=1}^N \mathbb{1}_{z > 10 }(Z_i) $$

```
import pymc as mc

N = 10000
print np.mean( [ mc.rexponential( 0.5 )>10 for i in range(N) ] )
```

    0.0058

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give use *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to the us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = mc.rdiscrete_uniform
norm = mc.rnormal

#generate some artificial population numbers
population = pop_generator(100, 1500, size = n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height**2,
                                        size=population[i] ).mean()

#located the counties with the apparently most extreme average heights.
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population size vs. recorded average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1)
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error results from the calculated average of smaller populations not being a good reflection of the true expected value of the population (which in truth should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 4000, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```
print "Population sizes of 10 'shortest' counties: "
print population[ np.argsort( average_across_county )[:10] ]
print
print "Population sizes of 10 'tallest' counties: "
print population[ np.argsort( -average_across_county )[:10] ]
```

    Population sizes of 10 'shortest' counties:
    [181 168 229 110 156 123 222 154 498 375]

    Population sizes of 10 'tallest' counties:
    [105 114 111 236 373 244 183 278 234 268]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);
```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How to order Reddit comments

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, and a very popular part of the site are the comments associated with each link. Redditors can vote up or down on each comment (called upvotes and downvotes). Reddit, by default, will sort comments by Top, that is, the best comments.

How would you determine which comments are the best? There are a number of ways to achieve this:

1. *Popularity*: A comment is considered good if it has many upvotes. A problem with this model is that a comment with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the comment is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of comments. Comments can be posted many hours after the original link submission. The difference method will bias the *Top* comments to be the oldest comments, which have accumulated more upvotes than newer comments, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the comment. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter example is, if we use per second, a 1 second old comment with 1 upvote would be better than a 100 second old comment with 99 upvotes. One can avoid this by only considering at least t second old comments. But what is a good t value? Does this mean no comment younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old comments).
3. *Ratio*: Rank comments by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new comments who score well can be considered Top just as likely as older comments, provided they have many upvotes to total votes. The problem here is that a comment with a single upvote (ratio = 1.0) will beat a comment with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter comment is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former comment, with a single upvote, is in fact a better comment than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former comment might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this comment a upvote, versus a downvote"). So the 999 upvote/1 downvote comment probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the comment with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is that look at the historical distribution of upvote ratios. This can be accomplished by scrapping Reddit's comments and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of comments have very few votes, hence there will be many comments with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use comments with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of comments available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards comments of these two subreddits are very different: visitors are likely friend and affectionate in the former, and would therefore upvote comments more, compared to the latter, where comments are likely to be controversial and disagreed upon. Therefore not all comments are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `comments_for_top_reddit_pic.py` will scrape the comments from the current top picture on Reddit. Below is the picture, and some comments:

```
from IPython.core.display import Image
#adding a number to the end of the %run call with get the ith top photo.
%run top_pic_comments.py 2

Image(top_post_url)

```

    Version 2.1.4 of praw is outdated. Version 2.1.5 was released 4 days ago.
    Title of submission:
    My friend is 7'5 ft tall, and he finally found a chair big enough for him.
    http://i.imgur.com/M7LjpG6.jpg

```
"""
contents: an array of the text from all comments on the pic
votes: a 2d numpy array of upvotes, downvotes for each comment.
"""
n_comments = len(contents )
comments = np.random.randint( n_comments, size=4)
print "Some Comments (out of %d total) \n-----------"%n_comments
for i in comments:
    print '"' + contents[i] + '"'
    print"upvotes/downvotes: ",votes[i,:]
    print
```

    Some Comments (out of 39 total)
    -----------
    "Weta could have saved a fortune by hiring this guy in The Hobbit, for the forced perspective scenes"
    upvotes/downvotes:  [50 14]

    "Hodor? "
    upvotes/downvotes:  [54 16]

    "Please provide a dollar bill or ruler for scale. He might be 2'7" and be sitting in doll furniture. "
    upvotes/downvotes:  [657 126]

    "The lack of replies for the OP is disturbing - I am starting to think this guy may not be his friend after all. Or he could have posted and just gone away from the computer without checking up on his post. Which made front page. But would any redditor really do that?

    Who is this /u/Triforcetrilogy? A guy with over 10k link karma, but only 200 comment karma. Who posts and doesn't talk?

    I am not sure what my point is, I guess I am just genuinely curious."
    upvotes/downvotes:  [10  3]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular comment's upvote/downvote pair.

```
import pymc as mc

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    """
    This function accepts the number of upvotes and downvotes a particular comment recieved,
    and the number of posterior samples to return to the user. Assumes a uniform prior.
    """
    N = upvotes + downvotes
    upvote_ratio = mc.Uniform( "upvote_ratio", 0, 1 )
    observations = mc.Binomial( "obs",  N, upvote_ratio, value = upvotes, observed = True)
    #do the fitting; first do a MAP as it is cheap and useful.
    map_ = mc.MAP([upvote_ratio, observations ]).fit()
    mcmc = mc.MCMC([upvote_ratio, observations ])
    mcmc.sample(samples, samples/4)
    return mcmc.trace("upvote_ratio")[:]

```

Below are the resulting posterior distributions.

```
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(comments)):
    j = comments[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 18, normed = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 18, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different comments");
```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the comments from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(comments)):
    j = comments[i]
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")
plt.title("Posterior distributions of upvote ratios on different comments");
order = argsort( -np.array( lower_limits ) )
print order, lower_limits
```

The best comments, according to our procedure, are the comments that are *most-likely* to score a high percentage of upvotes. Visually those are the comments with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best. That is, even in the worst case scenario, when we have severely overestimated the upvote ratio, we can be sure the best comments are still on top. Under this ordering, we impose the following very natural properties:

1. given two comments with the same observed upvote ratio, we will assign the comment with more votes as better (since we are more confident it has a higher ratio).
2. given two comments with the same number of votes, we still assign the comment with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every comment takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print "Approximate lower bounds:"
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print lb
print
print "Top 40 Sorted according to approximate lower bounds:"
print
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print  votes[i,0], votes[i,1], contents[i]
    print "-------------"
```

    Approximate lower bounds:
    [ 0.83557827  0.81654362  0.81502525  0.81295936  0.68825128  0.72597113
      0.77427338  0.81305042  0.74613833  0.85444435  0.79445943  0.82407077
      0.73404509  0.70913094  0.63630795  0.65082982  0.70597731  0.75075188
      0.68187341  0.54368293  0.56085485  0.60091591  0.55091897  0.60091591
      0.53055613  0.60091591  0.56085485  0.5543662   0.5997376   0.51184301
      0.51184301  0.45074913  0.53055613  0.3726793   0.43047887  0.43047887
      0.43047887  0.43047887  0.43047887]

    Top 40 Sorted according to approximate lower bounds:

    84 8 Why no NBA?
    -------------
    1000 172 i want to see more pics of this giant friend of yours
    -------------
    55 6 OP won't deliver :(
    -------------
    657 126 Please provide a dollar bill or ruler for scale. He might be 2'7" and be sitting in doll furniture.
    -------------
    371 68 For the love of God, post pics of him holding comically small things.
    -------------
    400 75 I'm just gonna say it. I want to see his penis.
    -------------
    85 12 AMA request, your friend
    -------------
    56 8 Does your friend have marfans syndrome, by any chance? I do, is why I ask.
    -------------
    36 5 That would make him one of the top 5 tallest people in the New World. He's tall, alright, but I'm calling shenanigans.
    -------------
    8 0 I expected a picture of him sitting on Lincoln's  lap at the Lincoln memorial.
    -------------
    73 16 You should post one of you two next to each other
    -------------
    17 2 I'm 4'10"...I wish we could stand next to each other...and have him hold really small things and I could hold really large things. Damn that's tall.
    -------------
    39 8 This guy should do an AMA i bet being 7'5 is not very fun, but i bet it has its perks
    -------------
    58 15 I think your friend should lift weights. Do you know how awesome he would look with huge muscles?
    -------------
    29 6 "7 foot 5 foot tall"...
    -------------
    50 14 Weta could have saved a fortune by hiring this guy in The Hobbit, for the forced perspective scenes
    -------------
    54 16 Hodor?
    -------------
    28 8 7'5ft is like saying $2.50 dollars
    -------------
    24 7 Serious question: how does someone who is not 7 foot 5 even get into that chair? I see no step, no tiny ladder.
    -------------
    4 0 So...you want me to write the theme tune, sing the theme tune...
    -------------
    4 0 I'm glad he's finally able to experience dangling his legs.
    -------------
    4 0 He looks 7'4"
    -------------
    12 3 I call bullshit, that guy is not 7'5. Just because O.P put it in the title does not make it true.
    -------------
    6 1 Bad Luck Brian:

    Finally finds a chair not small enough for him.

    Too Big
    -------------
    6 1 How big is his penis?
    -------------
    21 9 Your friend is cute, more pictures in general!
    -------------
    10 3 The lack of replies for the OP is disturbing - I am starting to think this guy may not be his friend after all. Or he could have posted and just gone away from the computer without checking up on his post. Which made front page. But would any redditor really do that?

    Who is this /u/Triforcetrilogy? A guy with over 10k link karma, but only 200 comment karma. Who posts and doesn't talk?

    I am not sure what my point is, I guess I am just genuinely curious.
    -------------
    25 12 hey, has he tried basketball?

    ^^^hahahaha ^^^haha ^^^ha ^^^hahaha ^^^ha
    -------------
    3 0 jebus - i thought i had it bad, but that guy is 6" taller than i am!
    -------------
    3 0 I have a friend that's 7'2", where can I buy this comically large chair?
    -------------
    5 1 Dear god. Put him in power armor and he would be almost the same height of a space marine, a rather short space marine.
    -------------
    5 1 Pretty sure this guy isn't 7ft 5. I looked up the tallest ppl in the world, got a list of everyone over 7ft with pictures. The Internet is fact. Gotta believe it.
    -------------
    4 1 Where is the photo analyst guy with height estimates when you need him?
    -------------
    2 0 I see red carts and Texas stars, is this at a HEB?
    -------------
    2 0 proof?
    -------------
    2 0 This guy is tall enough to make eight figures on an NBA bench.
    -------------
    2 0 He looks so satisfied
    -------------
    2 0 7'5 ft?
    -------------
    3 1 Those feet.
    -------------

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```
r_order = order[::-1][:40]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order],xuplims=True, capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30], ordered_contents) );
```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Extension to Starred rating systems

The above procedure works well for upvote-downvotes schemes, but what about systems that use star ratings, e.g. 5 star rating systems. Similar problems apply with simply taking the average: an item with two perfect ratings would beat an item with thousands of perfect ratings, but a single sub-perfect rating.

We can consider the upvote-downvote problem above as binary: 0 is a downvote, 1 if an upvote. A $N$-star rating system can be seen as a more continuous version of above, and we can set $n$ stars rewarded is equivalent to rewarding $\frac{n}{N}$. For example, in a 5-star system, a 2 star rating corresponds to 0.4. A perfect rating is a 1. We can use the same formula as before, but with $a,b$ defined differently:

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where

\begin{align}
& a = 1 + S \\\\
& b = 1 + N - S \\\\
\end{align}

where $N$ is the number of users who rated, and $S$ is the sum of all the ratings, under the equivilance scheme mentioned above.

##### Example: Counting Github stars

What is the average number of stars a Github repository has? How would you calculate this? There are over 6 million respositories, so there is more than enough data to invoke the Law of Large numbers. Let's start pulling some data. TODO

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting comments formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead using a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally accurate?

```
## Enter code here
import scipy.stats as stats
exp = stats.expon( scale=4 )
N = 1e5
X = exp.rvs( N )
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by their percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

In August 2013, [a popular post](http://bpodgursky.wordpress.com/2013/08/21/average-income-per-programming-language/) on the average income per programmer of different languages was trending. Here's the summary chart: (reproduced without permission, cause when you lie with stats, you gunna get the hammer). What do you notice about the extremes?

------

#### Average household income by programming language

<table >
<tr ><th></th></tr>
 <tr><td>Language</td><td>Average Household Income ($)</td><td>Data Points</td></tr>
 <tr><td>Puppet</td><td>87,589.29</td><td>112</td></tr>
 <tr><td>Haskell</td><td>89,973.82</td><td>191</td></tr>
 <tr><td>PHP</td><td>94,031.19</td><td>978</td></tr>
 <tr><td>CoffeeScript</td><td>94,890.80</td><td>435</td></tr>
 <tr><td>VimL</td><td>94,967.11</td><td>532</td></tr>
 <tr><td>Shell</td><td>96,930.54</td><td>979</td></tr>
 <tr><td>Lua</td><td>96,930.69</td><td>101</td></tr>
 <tr><td>Erlang</td><td>97,306.55</td><td>168</td></tr>
 <tr><td>Clojure</td><td>97,500.00</td><td>269</td></tr>
 <tr><td>Python</td><td>97,578.87</td><td>2314</td></tr>
 <tr><td>JavaScript</td><td>97,598.75</td><td>3443</td></tr>
 <tr><td>Emacs Lisp</td><td>97,774.65</td><td>355</td></tr>
 <tr><td>C#</td><td>97,823.31</td><td>665</td></tr>
 <tr><td>Ruby</td><td>98,238.74</td><td>3242</td></tr>
 <tr><td>C++</td><td>99,147.93</td><td>845</td></tr>
 <tr><td>CSS</td><td>99,881.40</td><td>527</td></tr>
 <tr><td>Perl</td><td>100,295.45</td><td>990</td></tr>
 <tr><td>C</td><td>100,766.51</td><td>2120</td></tr>
 <tr><td>Go</td><td>101,158.01</td><td>231</td></tr>
 <tr><td>Scala</td><td>101,460.91</td><td>243</td></tr>
 <tr><td>ColdFusion</td><td>101,536.70</td><td>109</td></tr>
 <tr><td>Objective-C</td><td>101,801.60</td><td>562</td></tr>
 <tr><td>Groovy</td><td>102,650.86</td><td>116</td></tr>
 <tr><td>Java</td><td>103,179.39</td><td>1402</td></tr>
 <tr><td>XSLT</td><td>106,199.19</td><td>123</td></tr>
 <tr><td>ActionScript</td><td>108,119.47</td><td>113</td></tr>
</table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. Web. 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

<style>
    img{
        max-width:800px}
</style>


# Init

```python
import numpy as np
import pylab as plt
```

# Bootstrap distribution of means

## Estimation of the mean

Suppose you have a set of random variables $x_i \in X$ from which you would like to compute the mean value $\mu(X)$.

```python
mu0  = np.random.uniform(low=1, high=2)
std0 = mu0 * np.random.uniform(low=0.7, high=0.9)

NX = 100
X  = np.random.normal(mu0, std0, size=100)
```

For normally distributed variables in the limit of large statistics, this is straight forward.
The mean value and standard deviation are computed by their estimators
\begin{align}
    \hat \mu(X) &= \frac{1}{N_X}\sum\limits_{i=1}^{N_X} x_i \, , &
    \hat \sigma(X) = \sqrt{\frac{1}{N_X-1}\sum\limits_{i=1}^{N_X} (x_i - \mu)^2 } \, .
\end{align}

```python
meanX = np.mean(X)
sdevX = np.std(X, ddof=1)

print("muEst = {mu} +/- {std}".format(mu=meanX, std=sdevX))
```

    muEst = 1.9876414742755415 +/- 1.6170022713736447

For 100 samples one could acutally expect to have a quite nice result for the mean:

```python
fig, ax = plt.subplots(dpi=300, figsize=(2,2))

xRange = np.linspace(meanX-3*sdevX, meanX+3*sdevX, 100)
normal = np.exp( - (xRange-meanX)**2 / 2 / sdevX**2 ) / np.sqrt(2*np.pi) / sdevX

ax.hist(X, normed=True, label="Hist")
ax.plot(xRange, normal, label="Normal")

ax.legend(loc="best", fontsize="xx-small")
ax.set_xlabel("$x$")
ax.set_ylabel("$P(x)$")

plt.show(fig)
```

We are now interested in the goodness of our estimatation for the mean value.
The uncertainty of this estimation is defined by the infinite statistics mean value compared to the estimated mean value for finite statistics. However, one usually does not have infinite statistics.
Also, this uncertainty, at least in the test case, will most likely be smaller than the standard deviation.

## Bootstrap

To answser this question, we generate Bootstrapp which mimic the generation of a larger ensemble by drawing from the inital distribution. For a more detailed description I refer to https://arxiv.org/abs/hep-lat/0506036.

1. We generate a new set of indices
    \begin{align}
        i &\in I_{n_B} \, , &
        I &= \mathrm{Uniform}(1, N_X) \, , &
        |I_{n_B}| &= N_{Size} \, .
    \end{align}
2. We compute the mean of the new distribution
    $$\hat\mu(X, I_{n_B}) = \frac{1}{|I_{n_B}|}\sum\limits _{i\in I_{n_B}} x_i\,.$$
3. We repeat this procedure to generate a set of means
    $$X_\mu = \{ \hat\mu(X, I_{1}),\hat\mu(X, I_{2} ) , \cdots , \hat\mu(X, I_{N_B}) \}\,.$$

From this new distribution, one can compute the mean of means and the standard deviation of means to estimate the uncertainty of the mean for the inital data
$$
    \mu(X) = \hat \mu (X_\mu) \pm \hat \sigma(X_\mu) \, .
$$

This procedute is implemented in the `bootstats` package

```python
import bootstats as boot

# Specify the data, must be two-dimensional
## the first dimension is a random variable
## the last dimension is the random data for each variable
data = np.array([X])
# The number of bootstrap samples (size of X_mu)
NSamples = 1000
# Binning of data (in case one wants to remove auto correlation)
NBinSize = 1
# The size of each bootstrap sample |I_nB|
NSize    = int(data.shape[-1]/NBinSize)

bs = boot.Bootstrapper(data, NSamples=NSamples, NSize=NSize, NBinSize=NBinSize)

meanMean = np.mean(bs.samples)
meanSdev = np.std(bs.samples, ddof=1)

print("mu_0 = {mu0:1.2f}, mu = {meanMean:1.2f} +/- {meanSdev:1.2f}".format(mu0=mu0, meanSdev=meanSdev, meanMean=meanMean))
```

    mu_0 = 1.81, mu = 1.99 +/- 0.16

Note that the mean of the bootstrap means and the standard deviation of the bootstrap means should converege to fixed values when increasing the sample size. In other words, the fluctuation of the mean of bootstrap means and the fluctuation of standard deviation of bootstrap means should be way smaller than the standard deviation of the bootstrap means.

```python
NSamplesRange = range(100, 2000, 100)

meanMeans = []
meanSdevs = []

for NSamples in NSamplesRange:
    bs = boot.Bootstrapper(data, NSamples=NSamples, NSize=NSize, NBinSize=NBinSize)

    meanMeans.append( abs(np.mean(bs.samples)-mu0)/mu0 )
    meanSdevs.append( np.std(bs.samples, ddof=1)/mu0 )

fig, ax = plt.subplots(dpi=300, figsize=(2,2))

ax.plot(NSamplesRange, meanMeans, label="$\hat\mu(X_\mu) - \mu_0$")
ax.plot(NSamplesRange, meanSdevs, label="$\hat\sigma(X_\mu)$")

ax.legend(loc="best", fontsize="xx-small")
ax.set_ylabel("$\Delta\mu \, ,  \sigma$ in [$\mu_0$]")
ax.set_xlabel("NSamples")

ax.set_ylim(0, max(meanMeans)*1.5)

plt.show(fig)
```

## Non-normal distributions

Note that bootstrapping makes no assumptions about the inital distribution

```python
mu0  = np.random.uniform(low=1, high=2)
std0 = mu0 * np.random.uniform(low=0.7, high=0.9)

NX = 100
X  = mu0 + np.random.uniform(low=-std0, high=+std0, size=100)
```

```python
data = np.array([X])
NSamples = 1000
NBinSize = 1
NSize    = int(data.shape[-1]/NBinSize)

bs = boot.Bootstrapper(data, NSamples=NSamples, NSize=NSize, NBinSize=NBinSize)

meanMean = np.mean(bs.samples)
meanSdev = np.std(bs.samples, ddof=1)
```

```python
fig, ax = plt.subplots(dpi=300, figsize=(2,2))

ax.hist(X, normed=True, label="Hist", alpha=0.7)
ax.axvline(mu0, label="$\mu_0$", color="red")

ax.axvline(meanMean, label="Bootstrap $\mu$", color="black")
ax.axvspan(meanMean-meanSdev, meanMean+meanSdev, color="gray", label="Bootstrap $\Delta \mu$", alpha=0.7 )

ax.legend(loc="lower left", fontsize="xx-small", frameon=True, framealpha=0.95)
ax.set_xlabel("$x$")
ax.set_ylabel("$P(x)$")

plt.show(fig)
```

```python

```


```python
!pip install sympy
```

    Requirement already satisfied: sympy in /Users/seanmcowen/anaconda/lib/python2.7/site-packages

```python
import sympy
x = sympy.Symbol("x")
```

```python
equation = x*2+1
print(equation)
```

    2*x + 1

```python
print(equation.subs(x,5))
```

    11

```python
equation1 = x*2+1
equation2 = x-1
print(equation1 + equation2)
```

    3*x

```python
print(sympy.solve(equation1-equation2))
```

    [-2]

```python
import sympy
p = sympy.Symbol("p")

demandEquation = 10-p
supplyEquation = p

prices = []
demandQ = []
supplyQ = []

for price in range(0,11):
    demandQ += [demandEquation.subs(p,price)]
    supplyQ += [supplyEquation.subs(p,price)]
    prices += [price]
print(prices)
print(demandQ)
print(supplyQ)
```

    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]
    [10, 9, 8, 7, 6, 5, 4, 3, 2, 1, 0]
    [0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10]

```python
EquilibriumP = sympy.solve(demandEquation-supplyEquation)
print(EquilibriumP)
EquilibriumP = EquilibriumP[0]
print(EquilibriumP)
```

    [5]
    5

```python
EquilibriumQ = demandEquation.subs(p,EquilibriumP)
print(EquilibriumQ)
```

    5

```python
import matplotlib.pyplot as plt
plt.plot(demandQ,prices)
plt.plot(supplyQ,prices)
plt.plot(EquilibriumQ,EquilibriumP, 'ro')

plt.xlabel("Quantity")
plt.ylabel("Price")
plt.show()
```

```python
import matplotlib.pyplot as plt
plt.plot(demandQ,prices)
plt.plot(supplyQ,prices)
unitsDemanded = demandEquation.subs(p,8)
unitsSupplied = supplyEquation.subs(p,8)
plt.plot(unitsDemanded,8, 'ro')
plt.plot(unitsSupplied,8, 'ro')
plt.plot([unitsDemanded,unitsSupplied],[8,8],":")

plt.xlabel("Supply and Demand")
plt.ylabel("Price")
plt.show()

surplus = unitsSupplied-unitsDemanded

print("The surplus is equal to "+str(surplus)+" units.")
```

```python
import matplotlib.pyplot as plt
plt.plot(demandQ,prices)
plt.plot(supplyQ,prices)
unitsDemanded = demandEquation.subs(p,2)
unitsSupplied = supplyEquation.subs(p,2)
plt.plot(unitsDemanded,2, 'ro')
plt.plot(unitsSupplied,2, 'ro')
plt.plot([unitsDemanded,unitsSupplied],[2,2],":")

plt.xlabel("Supply and Demand")
plt.ylabel("Price")
plt.show()

shortage = unitsDemanded-unitsSupplied

print("The shortage is equal to "+str(shortage)+" units.")
```


# Exercise: beam bending with the collocation method

The differential equation for beam bending
$$
(EI w'')'' - q = 0
$$
can be integrated analytically by specifying the linear loading as $q = q_0\frac{x}{L}$ and the boundary conditions
$$
w(0) = 0\\
w(L) = 0\\
w''(0)=0\\
w''(L) = 0
$$
to yield the deflection solution ("Biegelinie"):
$$
w(x) = \frac{q_0 L^4}{360 EI} \left[ 3\left(\frac{x}{L}\right)^5 - 10 \left(\frac{x}{L}\right)^3 + 7 \left(\frac{x}{L}\right)\right]
$$
We now seek to approximate this solution numerically using the collocation method.

```python
import numpy as np #numerical methods
import sympy as sp #symbolic operations
import matplotlib.pyplot as plt #plotting
sp.init_printing(use_latex='mathjax') #makes sympy output look nice

#Some plot settings
plt.style.use('seaborn-deep')
plt.rcParams['lines.linewidth']= 2.0
plt.rcParams['lines.color']= 'black'
plt.rcParams['legend.frameon']=True
plt.rcParams['font.family'] = 'serif'
plt.rcParams['legend.fontsize']=14
plt.rcParams['font.size'] = 14
plt.rcParams['axes.spines.right'] = False
plt.rcParams['axes.spines.top'] = False
plt.rcParams['axes.spines.left'] = True
plt.rcParams['axes.spines.bottom'] = True
plt.rcParams['axes.axisbelow'] = True
plt.rcParams['figure.figsize'] = (8, 6)
```

```python

#Defining the geometric an material characteristics as symbolic quantities
L,q,EI,x = sp.symbols('L q_0 EI x')
```

```python
#Analytical solution to deflection
def deflection_analytical():
    a = x/L
    f = q*L**4/(360 * EI)
    return f*(3*a**5 - 10*a**3 + 7*a)
```

```python
deflection_analytical() #check definition
```

$$\frac{L^{4} q_{0}}{360 EI} \left(\frac{7 x}{L} - \frac{10 x^{3}}{L^{3}} + \frac{3 x^{5}}{L^{5}}\right)$$

Now, let's plot the analytical solution. For that purpose, we use some Python magic ("lambdify"). We sample the analytical solution for $x \in [0,L]$ at 100 points and plot the dimensionless deflection over the dimensionless length.

```python
lam_x = sp.lambdify(x, deflection_analytical(), modules=['numpy'])
#For the variable x the function deflection_analytical() will obtain something

x_vals = np.linspace(0, 1, 100)*L #This something is x from 0 to L
analytical = lam_x(x_vals) #We calculate the solution by passing x = [0,...,L] to deflection_analytical

plt.plot(x_vals/L, analytical/(L**4*q)*EI)
plt.xlabel('$x / L$')
plt.ylabel('$w / L^4 q_0  EI^{-1}$')
```

## Trigonometric Ansatz

Let's try the approximation
$$
    \tilde{w} = a_1 \sin \left(\pi \frac{x}{L}\right) + a_2 \sin \left(2\pi\frac{x}{L}\right)
$$

```python
a1, a2 = sp.symbols('a_1 a_2')#Defining the free values as new symbols
```

```python
def deflection_ansatz():
    return a1*sp.sin(sp.pi/L*x) + a2*sp.sin(2*sp.pi/L*x) #defining the approximate solution with the unknown coefficients
```

Now we substitute this solution into the fourth-order ODE for beam bending by differentiating it twice and adding the distributed loading:
$$
EI w^\text{IV} - q_0 \frac{x}{L} = 0 \text{ with } EI = \text{const.}
$$

```python
ODE = EI * deflection_ansatz().diff(x,4) - q * (x/L)
ODE
```

$$\frac{\pi^{4} EI}{L^{4}} \left(a_{1} \sin{\left (\frac{\pi x}{L} \right )} + 16 a_{2} \sin{\left (\frac{2 \pi}{L} x \right )}\right) - \frac{q_{0} x}{L}$$

Now we evaluate the Ansatz we made at two collocation points distributed along the beam: $x_1 = L/2$ and $x_2 = 3L/4$. This creates two equations for two unknowns:

```python
collocation_conditions = (ODE.subs(x,L/4), ODE.subs(x,3*L/4))
```

```python
collocation_conditions
```

$$\left ( \frac{\pi^{4} EI}{L^{4}} \left(\frac{\sqrt{2} a_{1}}{2} + 16 a_{2}\right) - \frac{q_{0}}{4}, \quad \frac{\pi^{4} EI}{L^{4}} \left(\frac{\sqrt{2} a_{1}}{2} - 16 a_{2}\right) - \frac{3 q_{0}}{4}\right )$$

Now we solve these equations for our values $a_1$ and $a_2$ by demanding that the residuals vanish at the collocation points:

```python
coefficients = sp.solve(collocation_conditions,a1,a2)
```

```python
coefficients
```

$$\left \{ a_{1} : \frac{\sqrt{2} L^{4} q_{0}}{2 \pi^{4} EI}, \quad a_{2} : - \frac{L^{4} q_{0}}{64 \pi^{4} EI}\right \}$$

Now we're ready to plot the result and compare it to the analytical solution.

```python
#We first substite the now known Ansatz free values into our Ansatz
z = sp.symbols('z')
w_numerical = deflection_ansatz().subs([(a1,coefficients[a1]),(a2,coefficients[a2]),(x,z*L)])
#We also made the coordinate dimensionless (x/L --> z) because of sympy problems

lam_x_num = sp.lambdify(z, w_numerical, modules=['numpy'])
#For the variable x the expression w_numerical will be given something
z_vals = np.linspace(0, 1,100) #This something is z from 0 to 1
numerical = lam_x_num(z_vals) #We calculate the solution by passing x = [0,...,L] to deflection_analytical
plt.plot(x_vals/L, analytical/(L**4*q)*EI)
plt.plot(z_vals, numerical/(L**4*q)*EI)
plt.xlabel('$x\ /\ L$')
plt.ylabel('$w\ /\ L^4 q_0  EI^{-1}$')
plt.tight_layout()
```

```python
print("Maximum absolute error in deflection: ", np.max(np.abs(analytical/(L**4*q)*EI - numerical/(L**4*q)*EI)))
```

    Maximum absolute error in deflection:  0.000751049425655786

We can also plot and compare the bending moment. Let's first find the analytical expression by symbolically differentiating the deflection expression twice to obtain $M(x) = -EI w''(x)$:

```python
#analytical bending moment
moment_analytical = -deflection_analytical().diff(x,2)*EI
#numerical bending moment
moment_numerical = -deflection_ansatz().subs([(a1,coefficients[a1]),(a2,coefficients[a2])]).diff(x,2)*EI

#create lambdas for plotting along dimensionless length z
lam_x_analyt = sp.lambdify(z, moment_analytical.subs(x,z*L), modules=['numpy'])
lam_x_num = sp.lambdify(z, moment_numerical.subs(x,z*L), modules=['numpy'])
z_vals = np.linspace(0, 1,100)
analytical = lam_x_analyt(z_vals)
numerical = lam_x_num(z_vals)

#plot
plt.plot(x_vals/L, analytical/(L**2*q),label='analytical')
plt.plot(z_vals, numerical/(L**2*q),label='numerical')
plt.xlabel('$x\ /\ L$')
plt.ylabel('$M\ /\ L^2 q_0$')
plt.legend()
plt.tight_layout()
plt.savefig('beam_collocation_moment.pdf')
```

```python
print("Maximum absolute error in bending moment: ", np.max(np.abs(analytical/(L**2*q) - numerical/(L**2*q))))
```

    Maximum absolute error in bending moment:  0.00915175157530091

*Tasks*:

- What happens if you chose the symmetrical collocation points $L/4$ and $3L/4$?
- What happens if you move them further outside or inside?
- How does the solution develop when you choose the Ansatz $\tilde{w} = a_1 \left(\frac{x}{L}\right)^4 + a_2 \left(\frac{x}{L}\right)^6$? Why is the result so bad? (Hint: Ansatz requirements)
- How does the solution develop, if you add another member of the same shape, e.g. $a_3 \sin\left(\pi\frac{x}{L}\right)  $ but add another collocation point? (Hint: Ansatz requirements)


# PageRank Algorithm

This notebook implements the PageRank algorithm, prepared as a homework in BLG202E - Numerical Methods in CE class at ITU, Spring 2020.

```python
!pip install mechanize
```

    Defaulting to user installation because normal site-packages is not writeable
    Requirement already satisfied: mechanize in /home/marif/.local/lib/python3.6/site-packages (0.4.5)
    Requirement already satisfied: html5lib>=0.999999999 in /home/marif/.local/lib/python3.6/site-packages (from mechanize) (1.0.1)
    Requirement already satisfied: six>=1.9 in /home/marif/.local/lib/python3.6/site-packages (from html5lib>=0.999999999->mechanize) (1.14.0)
    Requirement already satisfied: webencodings in /home/marif/.local/lib/python3.6/site-packages (from html5lib>=0.999999999->mechanize) (0.5.1)

```python
import numpy as np #For arrays and linear algebra routines
from mechanize import Browser #For getting the names of the papers from arXiv in part 3 results
                              #Not necessary, but to use it, above cell must be run to install mechanize library
```

```python
eps = 1e-10
```

# Motivating Example: Academic Ranking System

Our motivating example uses the paper citation network of Arxiv High Energy Physics Theory category. We will rank the papers by their influence. Data is represented as a directed graph, and we have the list of edges in a text file. We will start by loading the text file that contains the directed graph into a NumPy array.

```python
array = np.genfromtxt("cit-HepTh.txt", dtype=int)
```

Let's see the first 10 edges.

```python
print(array[0:10])
```

    [[   1001 9304045]
     [   1001 9308122]
     [   1001 9309097]
     [   1001 9311042]
     [   1001 9401139]
     [   1001 9404151]
     [   1001 9407087]
     [   1001 9408099]
     [   1001 9501030]
     [   1001 9503124]]

We will create a transition matrix from the list that contains links between the nodes of the graph. A transition matrix shows transitions from nodes to other nodes and the probability that transition happens.

These weights or normalization factors are calculated by taking $\frac{1}{\textrm{(no of outgoing links from that node)}}$. Following code will define a function with parameter **arr**: multidimensional array of edges, and returns **labels**: array consisting of distinct nodes and **weights**: a dictionary that maps these nodes to their weights.

```python
def createWeights(arr):
    labels, counts = np.unique(arr[:,0], return_counts=True) ##Finds the unique entries in first column, returns their values
                                                              #and their counts to calculate weights
    weight_dict = dict()
    for index, label in enumerate(labels):                   ##Creates a dictionary that holds the normalization factor,
        weight_dict[label] = 1 / counts[index]               # 1/(number of outgoing links) for every paper ID in labels

    return labels, weight_dict
```

```python
labels, weights = createWeights(array)
```

Following code will initialize transition matrix, finds the cells that represent transitions using a for loop that examines each node and links from that node, and fills in the cell with the weight specified in weights dictionary created above.

```python
def createTransitionMatrix(arr, labels, weight_dict):
    transitionMatrix = np.zeros((labels.size,labels.size))

    for i, label in enumerate(labels):
        links_from_label = arr[np.nonzero(label == arr[:,0]),1][0]
      #  if links_from_label.shape == (0,):                          ##For dangling nodes, nodes that have no outgoing links
      #      weight_dict[label] = 1 / labels.size
      #      links_to_label = labels
        transitionMatrix[np.searchsorted(labels, links_from_label),i] = weight_dict[label]

    return transitionMatrix
```

```python
rankMatrix = createTransitionMatrix(array, labels, weights)
```

We expect our transition matrix to be column-stochastic, and we also expect it to be a sparse matrix it is mostly populated by zeros. Operations on sparse matrices can be done by some faster methods, so sparsity is an advantage in speed.

Now we will define and run two functions to test these attributes of our matrix. Results will show that our expectations hold.

```python
def checkStochastic(matrix):
    eps = 1e-10
    print("Sums of columns that do not add to 1 in a reasonable error margin {} will be shown.".format(eps))
    for i in matrix.sum(0):
        if not (i - 1 < eps):
            print(i)
    print("Calculation finished, average value of sums of all columns is {}.".format(np.mean(matrix.sum(0))))

def checkSparsity(matrix):
    zeros = np.count_nonzero(matrix==0)
    elements = matrix.size
    sparse_rate = np.divide(zeros,elements)
    print("There are {} elements in total, {} of them are zero.".format(elements, zeros))
    print("Sparsity rate of this matrix is %{}".format(sparse_rate*100))
    return sparse_rate
```

```python
checkStochastic(rankMatrix)
sparsity = checkSparsity(rankMatrix)
```

    Sums of columns that do not add to 1 in a reasonable error margin 1e-10 will be shown.
    Calculation finished, average value of sums of all columns is 0.9958169865008099.
    There are 627953481 elements in total, 627601470 of them are zero.
    Sparsity rate of this matrix is %99.94394314059069

---
Now that we have the required matrix, we can solve the equation
\begin{equation}
A x = x
\end{equation}
where A is the matrix, and x is the result vector that contains the rank. We will solve this by **power method**, by repeatedly multiplying an arbitrary vector* by our matrix until the difference in resulting vectors of two iterations is smaller than epsilon.

*While any arbitrary vector should work, it is better practice to use an all ones vector normalized by the size of itself, so initially every rank is equal and vector sums up to one. We will follow this practice.

```python
def solveRank(rankMatrix):
    eps = 1e-7
    v0 = np.ones(rankMatrix.shape[0]) / rankMatrix.shape[0]
#   v0 = np.random.random(rankMatrix.shape[0] / rankMatrix.shape[0])

    counter = 1
    while True:
        v = np.dot(rankMatrix, v0)
        v = v / np.linalg.norm(v)
        if (np.mean(np.abs(np.subtract(v,v0))) < 2*eps):
            break

#        print("Error: {}".format(np.mean(np.abs(np.subtract(v,v0)))))   ##Uncomment this line to print error in each step
                                                                          #If this function is taking too long, printing the error may be a good idea for debugging
        counter += 1
        v0 = v

    print("Appropriate vector found in {} iterations, final difference between two iteration result vectors was less than {}.".format(counter,eps))
    return v
```

```python
final = solveRank(rankMatrix)
```

    Appropriate vector found in 80 iterations, final difference between two iteration result vectors was less than 1e-07.

Finally, we will rank our papers according to the resulting vector, and show the first 10 papers.

```python
def rankPagesDescending(labels, final):
    return labels[final.argsort()][::-1]
```

```python
rankPagesDescending(labels, final)[0:10]
```

    array([9201015, 9207016, 9206003,  209015, 9205071, 9202067, 9201047,
           9205038, 9202018, 9205006])

Using the mechanize library, we can collect the information on these papers.

```python
ranking = rankPagesDescending(labels, final)
br = Browser()

for index,paper_id in enumerate(ranking[0:10]):
    str_id = str(paper_id)
    page_url = "https://arxiv.org/abs/hep-th/"
    while(len(str_id) < 7):
        str_id = '0' + str_id
    page_url += str_id
    br.open(page_url)
    paper_title = br.title()[17:]
    print("{}. paper ID is {}.".format(index + 1, paper_id))
    print("Name of the paper is: {}".format(paper_title))
    print(page_url)
```

    1. paper ID is 9201015.
    Name of the paper is: An Algorithm to Generate Classical Solutions for String Effective Action
    https://arxiv.org/abs/hep-th/9201015
    2. paper ID is 9207016.
    Name of the paper is: Noncompact Symmetries in String Theory
    https://arxiv.org/abs/hep-th/9207016
    3. paper ID is 9206003.
    Name of the paper is: From Form Factors to Correlation Functions: The Ising Model
    https://arxiv.org/abs/hep-th/9206003
    4. paper ID is 209015.
    Name of the paper is: Advances in String Theory in Curved Space Times
    https://arxiv.org/abs/hep-th/0209015
    5. paper ID is 9205071.
    Name of the paper is: Novel Symmetry of Non-Einsteinian Gravity in Two Dimensions
    https://arxiv.org/abs/hep-th/9205071
    6. paper ID is 9202067.
    Name of the paper is: Stringy Domain Walls and Other Stringy Topological Defects
    https://arxiv.org/abs/hep-th/9202067
    7. paper ID is 9201047.
    Name of the paper is: Duality-Invariant Gaugino Condensation and One-Loop Corrected Kahler Potentials in String Theory
    https://arxiv.org/abs/hep-th/9201047
    8. paper ID is 9205038.
    Name of the paper is: Recent Developments in Classical and Quantum Theories of Connections, Including General Relativity
    https://arxiv.org/abs/hep-th/9205038
    9. paper ID is 9202018.
    Name of the paper is: Jones Polynomials for Intersecting Knots as Physical States of Quantum Gravity
    https://arxiv.org/abs/hep-th/9202018
    10. paper ID is 9205006.
    Name of the paper is: Stabilized Quantum Gravity: Stochastic Interpretation and Numerical Simulation
    https://arxiv.org/abs/hep-th/9205006

In some edge cases, the transition matrix above may not abide by the conditions algorithm requires. To account for those situations, there is a **damping factor** defined for the algorithm, which takes the weighted average of our transition matrix with a matrix of all ones. While transition matrix represents probabilities of going from one node to another, adding these damping factor will give us a chance to randomly go from any node to another.

```python
def addDamping(transitionMatrix, labels, p = 0.15):
    if(p < 0 or p > 1):
        print("Please try again with a damping factor in interval [0,1].")
        return None
    rankMatrix = (1 - p) * transitionMatrix + p * ((1/labels.size) * np.ones((labels.size,labels.size)))
    return rankMatrix
```

```python
rankMatrix = addDamping(rankMatrix, labels)
checkStochastic(rankMatrix)
sparsity = checkSparsity(rankMatrix)
```

    Sums of columns that do not add to 1 in a reasonable error margin 1e-10 will be shown.
    Calculation finished, average value of sums of all columns is 0.9964444385253646.
    There are 627953481 elements in total, 0 of them are zero.
    Sparsity rate of this matrix is %0.0

We see that while stochastic structure of the matrix holds, it is no longer a sparse matrix after adding damping factor. This is because now every node has the probability to go to any other node, even if this is a very small probability. So we do not have any 0 cells in our matrix anymore.

Now, we will solve out matrix again.

```python
final = solveRank(rankMatrix)
ranking = rankPagesDescending(labels, final)
br = Browser()

for index,paper_id in enumerate(ranking[0:10]):
    str_id = str(paper_id)
    page_url = "https://arxiv.org/abs/hep-th/"
    while(len(str_id) < 7):
        str_id = '0' + str_id
    page_url += str_id
    br.open(page_url)
    paper_title = br.title()[17:]
    print("{}. paper ID is {}.".format(index + 1, paper_id))
    print("Name of the paper is: {}".format(paper_title))
    print(page_url)
```

    Appropriate vector found in 30 iterations, final difference between two iteration result vectors was less than 1e-07.
    1. paper ID is 9201015.
    Name of the paper is: An Algorithm to Generate Classical Solutions for String Effective Action
    https://arxiv.org/abs/hep-th/9201015
    2. paper ID is 9207016.
    Name of the paper is: Noncompact Symmetries in String Theory
    https://arxiv.org/abs/hep-th/9207016
    3. paper ID is 9205071.
    Name of the paper is: Novel Symmetry of Non-Einsteinian Gravity in Two Dimensions
    https://arxiv.org/abs/hep-th/9205071
    4. paper ID is 209015.
    Name of the paper is: Advances in String Theory in Curved Space Times
    https://arxiv.org/abs/hep-th/0209015
    5. paper ID is 9202067.
    Name of the paper is: Stringy Domain Walls and Other Stringy Topological Defects
    https://arxiv.org/abs/hep-th/9202067
    6. paper ID is 9201047.
    Name of the paper is: Duality-Invariant Gaugino Condensation and One-Loop Corrected Kahler Potentials in String Theory
    https://arxiv.org/abs/hep-th/9201047
    7. paper ID is 9202018.
    Name of the paper is: Jones Polynomials for Intersecting Knots as Physical States of Quantum Gravity
    https://arxiv.org/abs/hep-th/9202018
    8. paper ID is 9205006.
    Name of the paper is: Stabilized Quantum Gravity: Stochastic Interpretation and Numerical Simulation
    https://arxiv.org/abs/hep-th/9205006
    9. paper ID is 9205038.
    Name of the paper is: Recent Developments in Classical and Quantum Theories of Connections, Including General Relativity
    https://arxiv.org/abs/hep-th/9205038
    10. paper ID is 9206048.
    Name of the paper is: Conformally Exact Results for SL(2,R)\times SO(1,1)^{d-2}/SO(1,1) Coset Models
    https://arxiv.org/abs/hep-th/9206048

---

Since our dataset is very big, it is not easy to say if our algorithm correctly chooses the rankings. So we will test the same functions on a smaller dataset below, and we will compare the results with our expectation that will be computed by hand mathematically.

```python
def calculateRankVector(data, damping=False):
    labels, weights = createWeights(data)
    rankMatrix = createTransitionMatrix(data, labels, weights)
    if damping:
        rankMatrix = addDamping(rankMatrix, labels)
    resultVector = solveRank(rankMatrix)
    return resultVector
```

```python
filename = "testset.txt" #Change the filename to test it with another file
data = np.genfromtxt(filename, dtype = str)
resultVector = calculateRankVector(data, damping=True)
ranking = rankPagesDescending(labels, resultVector)
print(ranking)
print(resultVector.reshape(resultVector.size,1))
```

    Appropriate vector found in 20 iterations, final difference between two iteration result vectors was less than 1e-07.
    ['A' 'D' 'C' 'B']
    [[0.63395272]
     [0.34267715]
     [0.444879  ]
     [0.53175087]]


## Impulse Control Algorithm for HFT Market Making

For my homie Pontus <3

### Algorithmic design
The algorithm seeks to optimize the bid ask spread + hedging decisions for each time $t<T$ such that the net Profit and Loss of trading the spread is maximized at the end of the trading session time $T$. We also want to control the algorithm for excessive risk taking during the trading session. Thus, we impose two penalties: (1) running inventory penalty for trading too large exposures during the trading session, and (2) terminal inventory penalty i.e. holding non-zero inventory at the end of the trading day.  Mathematically, the maximization of stochastic system subject to the penalties is equivalent to maximizing the following:

\begin{align}
    V(t, X_t, q_t, S_t) = \underset{\delta^b, \delta^a}{max} \ \mathbb{E} \left( X_T + q_T(S_T - \alpha q_T) - \phi\int_{t}^{T}q_s^2 ds - \sum_{\tau < T}\xi_{\tau}\right)
\end{align}

where

- $X_t$ is the market maker's cash at time $t$
- $q_t$ is the market maker's inventory at time $t$
- $S_t$ is the mid-price of the asset at time $t$
- $\xi_{\tau}$ is average cost of crossing one contract at (stopping) time $\tau$
- $\alpha > 0$ is terminal inventory penalty parameter
- $\phi > 0$ is running inventory penalty parameter

### Solving the control problem (HJB-QVI)
The problem above may be solved as a HJB-QVI problem where we decide between (1) market making, (2) sending market order to buy one contract or (3) sending market order to sell one contract. Applying standard Dynamic Programming Principle to $V(t, X_t, q_T, S_t)$ tells us that $V$ must satisfy the following HJB-QVI:

\begin{align}
   0 =  max \left( \frac{\partial V}{\partial t} + \underset{\delta^b, \delta^a}{max} \left[ \mathcal{L}V(t) -\phi q_t^2 \right] ; \mathcal{S}V-V;\mathcal{B}V-V\right)
\end{align}

where the operators $\mathcal{S}$ and $\mathcal{B}$ act on $V(t, X_t, q_T, S_t)$ such that

\begin{align}
    \mathcal{S}V(t) &= V(t, X_t - \xi, q_T - 1, S_t) \\
    \mathcal{B}V(t) &= V(t, X_t - \xi, q_T + 1, S_t)
\end{align}
In other words, they represent (in abstract sense) the cost of crossing the spread.

### Numerical computation
The problem with the above HJB-QVI is that it is difficult (for me the author at least hehe...hmm) to find a closed form solution. This is not a problem, if we are willing to get our hands dirty (read: use numerical schemes!). Consequently, we will use a hammer called finite differences to nail the problem down. We first solve the HJB part as if there was no QVI conditions at all. Then, after we have obtained HJB PDE solution we look the grid again and satisfy the QVIs.

So, looking at the HJB part

\begin{align}
   \frac{\partial V}{\partial t} + \underset{\delta^b, \delta^a}{max} \left[ \mathcal{L}V(t) -\phi q_t^2 \right]
\end{align}

we see that the time derivative may be discretized using a backward Euler finite difference scheme. Then solving for
$V(t-dt)$ yields:

\begin{align}
    V(t_{i-1}) = V(t_{i})\Delta t + \underset{\delta^b, \delta^a}{max} \left[ \mathcal{L}V(t_{i}) -\phi q_t^2 \right] \Delta t
\end{align}

After we have computed $V(t_{i})$ for each $i = 0, 1, ..., N$ over the inventory time grid $(t,q)$ we solve the QVI part by taking maximums between the three options:

\begin{align}
    V(t, X_t, q_t, S_t) = max \{ V(t_{i}); \mathcal{S}V(t_{i}) ;\mathcal{B}V(t_{i}) \}
\end{align}

Finally, we need to assume some functional form (educated guess!) for the function $V$. To make our life easy, we use the following linear form which is quite common in the literature:

\begin{align}
    V(t, X_t, q_t, S_t) &= X_t + q_t(S_t - \alpha q_t) + v(t, q_t) \\
    v(T, q_T) &= -\alpha q_T^2
\end{align}

### Numerical examples

Let us consider in which kind of behavior the model would result if we programmed a computer to implement it. In the following we will solve the model for 5 minute trading period. We would then repeat the 5 minute cycle until the end of the trading horizon.

```python
import numpy as np
import matplotlib.pyplot as pp;

from src.hjb_solvers import (
    MM_Model_Parameters,
    AS3P_Finite_Difference_Solver,
)

```

#### Model parameters:
Below we have assumed some set of parameters. In real life you need to do some data science hocus pocus to come up with $\mathbb{P}$-measured set of parameters for the model. For the decay parameter $\kappa$ the reader is adviced to refer the original paper by Avellaneda and Stoikov (2008).

```python

lambda_m = 50  # Number of market order arrivals per minute on the bid
lambda_p = 50  # Number of market order arrivals per minute on the ask
kappa_m = 100  # Decay parameter for "fill rate" on the bid
kappa_p = 100  # Decay parameter for "fill rate" on the ask
delta = 0
phi = 0.000001 # running inventory penalty parameter (try changing me!)
alpha = 0.0001 # terminal inventory penalty parameter (try changing me!)
q_min = -25    # largest allowed short position
q_max = 25     # largest allowed long position
cost = 0.005   # trading cost per market order
rebate = 0.0025

# We solve the model for 5 minute chunks. T
T = 5 # minutes
n = 5*60 # one step per second
```

```python
parameters = MM_Model_Parameters(lambda_m, lambda_p ,kappa_m, kappa_p, delta,
                                   phi, alpha, q_min, q_max, T, cost, rebate)

impulses, model = AS3P_Finite_Difference_Solver.solve(parameters, N_steps=n)
```

### Value function $v$

```python
# Plot the value function
Y = model.q_grid
X = model.t_grid
X, Y = np.meshgrid(X,Y)
f = pp.figure(figsize=[5, 4]);
pp3d = pp.axes(projection="3d", elev=20, azim=50);
pp3d.set_title("Value function");
pp3d.set_xlabel("Minute");
pp3d.set_ylabel("Inventory");
pp3d.set_zlabel("Value");
pp3d.plot_surface(X, Y, model.h, cmap='magma');
```

### Impulse regions i.e. when do we send market orders??

```python
 # Plot the impulse regions
from matplotlib import colors
import matplotlib.patches as mpatches
mycolors = ['white', 'blue', 'red']
cmap = colors.ListedColormap(mycolors)
f, ax = pp.subplots(figsize=[4, 4])
ax.imshow(impulses, cmap=cmap, aspect='auto')
ax.set_xticks([0, 0.5*n, n])
ax.set_xticklabels([0, int(0.5*n), n],fontsize=8);
ax.set_yticks(np.arange(0, len(model.q_grid), 2));
ax.set_yticklabels(model.q_grid[::2],fontsize=8);
ax.set_ylabel('Inventory')
ax.set_xlabel('Second')
```

In the above graph the red region is where we "hedge" by sending sell market orders while blue region is where we send buy market orders.

### How do we market make then?

```python
# Plot the ask spread for continuation region
f, ax = pp.subplots(figsize=[5, 4])
for q in range(3, -4, -1):
    ax.plot(model.l_p[q], label=f'q={q}')
ax.set_title("Ask to mid spread")
pp.legend()

# Plot the bid spread for continuation region
fig, ax = pp.subplots(figsize=[5, 4])
for q in range(3, -4, -1):
    ax.plot(model.l_m[q], label=f'q={q}')
ax.set_title("Bid to mid spread")
pp.legend()

```

```python

```