# Chapter 4
`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`

______

## The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been using this simple idea in every example thus far.

### The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear by examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for almost *any distribution*, minus some important cases we will encounter later.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size = 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to its parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```python
%matplotlib inline
import numpy as np
from IPython.core.pylabtools import figsize
import matplotlib.pyplot as plt

figsize( 12.5, 5 )

sample_size = 100000
expected_value = lambda_ = 4.5
poi = np.random.poisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average),
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend();
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This is simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

The above formulae is interpretable as a distance away from the true value (on average), for some $N$. (We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

By computing the above many, $N_y$, times (remember, it is random), and averaging them:

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \rightarrow E[ Y_k ] = E\;\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \right]$$

Finally, taking the square root:

$$ \sqrt{\frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k} \approx D(N) $$

```python
figsize( 12.5, 4)

N_Y = 250 #use this many to approximate D(N)
N_array = np.arange( 1000, 50000, 2500 ) #use this many samples in the approx. to the variance.
D_N_results = np.zeros( len( N_array ) )

lambda_ = 4.5
expected_value = lambda_ #for X ~ Poi(lambda) , E[ X ] = lambda

def D_N( n ):
    """
    This function approx. D_n, the average variance of using n samples.
    """
    Z = poi( lambda_, (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for i,n in enumerate(N_array):
    D_N_results[i] =  D_N(n)

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 2, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " );
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of convergence to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 5, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 5 ) =  \frac{1}{N}\sum_{i=1}^N \mathbb{1}_{z > 5 }(Z_i) $$

```python
N = 10000
print( np.mean( [ np.random.exponential( 0.5 ) > 5 for i in range(N) ] ) )
```

    0.0001

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give us *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```python
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = np.random.randint
norm = np.random.normal

#generate some artificial population numbers
population = pop_generator(100, 1500, n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height,
                                        population[i] ).mean()

#located the counties with the apparently most extreme average heights.
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population size vs. recorded average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1);
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error results from the calculated average of smaller populations not being a good reflection of the true expected value of the population (which in truth should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 1500, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```python
print("Population sizes of 10 'shortest' counties: ")
print(population[ np.argsort( average_across_county )[:10] ], '\n')
print("Population sizes of 10 'tallest' counties: ")
print(population[ np.argsort( -average_across_county )[:10] ])
```

    Population sizes of 10 'shortest' counties:
    [140 174 100 101 220 171 244 105 227 171]

    Population sizes of 10 'tallest' counties:
    [101 110 142 113 111 160 115 656 357 199]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```python
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);
```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How to order Reddit submissions

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, called submissions, for people to comment on. Redditors can vote up or down on each submission (called upvotes and downvotes). Reddit, by default, will sort submissions to a given subreddit by Hot, that is, the submissions that have the most upvotes recently.

How would you determine which submissions are the best? There are a number of ways to achieve this:

1. *Popularity*: A submission is considered good if it has many upvotes. A problem with this model is that a submission with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the submission is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of submission. Depending on when a submission is posted, the website may be experiencing high or low traffic. The difference method will bias the *Top* submissions to be the those made during high traffic periods, which have accumulated more upvotes than submissions that were not so graced, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the submission. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter-example is, if we use per second, a 1 second old submission with 1 upvote would be better than a 100 second old submission with 99 upvotes. One can avoid this by only considering at least t second old submission. But what is a good t value? Does this mean no submission younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old submissions).
3. *Ratio*: Rank submissions by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new submissions who score well can be considered Top just as likely as older submissions, provided they have many upvotes to total votes. The problem here is that a submission with a single upvote (ratio = 1.0) will beat a submission with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter submission is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former submission, with a single upvote, is in fact a better submission than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former submission might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this submission a upvote, versus a downvote"). So the 999 upvote/1 downvote submission probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the submission with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is to look at the historical distribution of upvote ratios. This can be accomplished by scraping Reddit's submissions and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of submissions have very few votes, hence there will be many submissions with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use submissions with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of submissions available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards submissions of these two subreddits are very different: visitors are likely friendly and affectionate in the former, and would therefore upvote submissions more, compared to the latter, where submissions are likely to be controversial and disagreed upon. Therefore not all submissions are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `top_showerthoughts_submissions.py` will scrape the best posts from the `showerthoughts` community on Reddit. This is a text-only community so the title of each post *is* the post. Below is the top post as well as some other sample posts:

```python
reddit = praw.Reddit(client_id="3tGpc7FJeHhOag", client_secret="SSBymfcpf00fRV-JLIQTvjTwWd4X8A", user_agent="test")
subreddit  = reddit.subreddit("showerthoughts")
top_submissions = subreddit.top(limit=100)
top_submission = next(top_submissions)
print(top_submission.title)
print(top_submission.upvote_ratio)
```

    Whoever created the tradition of not seeing the bride in the wedding dress beforehand saved countless husbands everywhere from hours of dress shopping and will forever be a hero to all men.
    0.96

```python
#adding a number to the end of the %run call will get the ith top post.
%run top_showerthoughts_submissions.py 2

print("Post contents: \n")
print(top_post)
```

    Post contents:

    We laugh at dogs getting excited when they hear a bark on TV, but if TV was a nonstop stream of unintelligible noises and then someone suddenly spoke to you in your language, you'd be pretty fucking startled too.

```python
"""
contents: an array of the text from the last 100 top submissions to a subreddit
votes: a 2d numpy array of upvotes, downvotes for each submission.
"""
#print(votes)
print(votes.shape)
n_submissions = len(votes)
submissions = np.random.randint( n_submissions, size=4)
print("Some Submissions (out of %d total) \n-----------"%n_submissions)
for i in submissions:
    print('"' + contents[i] + '"')
    print("upvotes/downvotes: ",votes[i,:], "\n")
```

    (98, 2)
    Some Submissions (out of 98 total)
    -----------
    "If elevators hadn't been invented, all the CEOs and important people would have their offices on the first floor as a sign of status."
    upvotes/downvotes:  [150845   7939]

    "Apparently, a lemon is not naturally occurring and is a hybrid developed by cross breeding a bitter orange and a citron. Life never gave us lemons; we invented them all by ourselves."
    upvotes/downvotes:  [133301  10033]

    "Nobody cares if their deodorant is 24 hour or 48 hour, they are still going to apply it once a day"
    upvotes/downvotes:  [108269   8149]

    "Smoking kills you, but the actual “act” of smoking is a very healthy practice - leave a stressful environment, go outside for five minutes and take deep breaths."
    upvotes/downvotes:  [123836  16887]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular submission's upvote/downvote pair.

```python
import pymc3 as pm

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    """
    This function accepts the number of upvotes and downvotes a particular submission recieved,
    and the number of posterior samples to return to the user. Assumes a uniform prior.
    """
    N = upvotes + downvotes
    with pm.Model() as model:
        upvote_ratio = pm.Uniform("upvote_ratio", 0, 1)
        observations = pm.Binomial( "obs",  N, upvote_ratio, observed=upvotes)

        trace = pm.sample(samples, step=pm.Metropolis())

    burned_trace = trace[int(samples/4):]
    return burned_trace["upvote_ratio"]

```

Below are the resulting posterior distributions.

```python
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(submissions)):
    j = submissions[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 10, density = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 10, density = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different submissions");
```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the submissions from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean is a bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```python
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(submissions)):
    j = submissions[i]
    plt.hist( posteriors[i], bins = 20, density = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, density = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")
plt.title("Posterior distributions of upvote ratios on different submissions");
order = np.argsort( -np.array( lower_limits ) )
print(order, lower_limits)
```

The best submissions, according to our procedure, are the submissions that are *most-likely* to score a high percentage of upvotes. Visually those are the submissions with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best.  When using the lower-bound of the 95% credible interval, we believe with high certainty that the 'true upvote ratio' is at the very least equal to this value (or greater), thereby ensuring that the best submissions are still on top. Under this ordering, we impose the following very natural properties:

1. given two submissions with the same observed upvote ratio, we will assign the submission with more votes as better (since we are more confident it has a higher ratio).
2. given two submissions with the same number of votes, we still assign the submission with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every submission takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```python
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print("Approximate lower bounds:")
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print(lb)
print("\n")
print("Top 40 Sorted according to approximate lower bounds:")
print("\n")
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print(votes[i,0], votes[i,1], contents[i])
    print("-------------")
```

    Approximate lower bounds:
    [0.96933753 0.93907338 0.95920114 0.91891673 0.9490931  0.91889453
     0.92895017 0.90884659 0.91889197 0.95916043 0.94907383 0.93898424
     0.89875857 0.90879399 0.92889779 0.87867424 0.94902535 0.92888465
     0.91882347 0.9389398  0.94901392 0.92887485 0.90876456 0.88868133
     0.90875869 0.92886959 0.89871685 0.91880661 0.91879415 0.91879188
     0.89869476 0.8886536  0.95908063 0.81850281 0.95908242 0.93890543
     0.95907501 0.88863762 0.95906682 0.92882116 0.90871204 0.94896687
     0.92881978 0.89866495 0.91875154 0.83848356 0.89865174 0.93887756
     0.91874332 0.89864019 0.87856354 0.93886385 0.88859832 0.93886058
     0.89863058 0.94893942 0.9086671  0.90866807 0.94893144 0.93884454
     0.93884612 0.91871004 0.91870965 0.90865793 0.87853176 0.89860543
     0.8986088  0.9590174  0.94891813 0.84844537 0.8785201  0.92876367
     0.86848838 0.92876096 0.92875541 0.84842727 0.88854318 0.89858964
     0.9489116  0.92874834 0.92875039 0.89858347 0.82838958 0.92874585
     0.92874105 0.90862445 0.91867462 0.89857674 0.94890055 0.87849525
     0.93881043 0.81836607 0.8985679  0.92873191 0.94889201 0.90860991
     0.91866641 0.93880078]

    Top 40 Sorted according to approximate lower bounds:

    178697 5527 When you're a kid, you don't realize you're also watching your mom and dad grow up.
    -------------
    159005 6625 Being able to tolerate the sound of your own voice in a video is probably the highest form of self acceptance.
    -------------
    143578 5982 Pavlov probably thought about feeding his dogs every time someone rang a bell.
    -------------
    120298 5012 Lamps in Videogames use real electricity.
    -------------
    120838 5035 Security at every level of an airport is absolutely ridiculous. Until you get to the baggage claim. Then it’s just like take whatever bag you want.
    -------------
    119469 4978 Night before a day off is more satisfying than the actual day off.
    -------------
    116575 4857 The internet both almost killed off the postal service with email and then made it more needed than ever with online delivery
    -------------
    106147 4423 Your future self is talking shit about you
    -------------
    150845 7939 If elevators hadn't been invented, all the CEOs and important people would have their offices on the first floor as a sign of status.
    -------------
    144764 7619 11 babies have been born in Antarctica, and none of them died as infants, meaning Antarctica has the lowest infant mortality rate at 0%
    -------------
    131382 6915 There is probably at least one white guy, adopted and raised in China, who speaks English with a Chinese accent and strangers just assume he’s a total asshole
    -------------
    128474 6762 It kinda makes sense that the target audience for fidget spinners lost interest in them so quickly
    -------------
    117131 6165 Biting your tongue while eating is a perfect example of how you can still screw up, even with decades of experience.
    -------------
    110658 5824 Watching a graduation ceremony is like sitting through a movie thats entirely end credits
    -------------
    108667 5719 It must suck to be an air conditioner repairman. You spend your day working in buildings that have no air conditioning. When it's fixed and finally cool, you leave.
    -------------
    107341 5650 The first person who inhaled helium must have been so relieved when the effects wore off
    -------------
    104584 5504 Replying "k" in morse "-.-", has the same passive aggressive tone
    -------------
    102588 5399 When medication says "do not operate heavy machinery" they're probably mainly referring to cars, but my mind always goes to forklift.
    -------------
    101537 5344 Before radios, tvs, and smartphones people never knew there were so many dumb people out there. They just thought it was that one guy in town.
    -------------
    170744 10899 Dads probably bond with dogs so much because, in our society, men don’t get shown a lot of affection but dogs give tons of affection regardless
    -------------
    142045 9067 Most people know so little that if they were transported 200 years into the past, they wouldn't be able to invent anything any quicker.
    -------------
    130188 8310 It would be cool if after you died you could see the top 5 times you almost died.
    -------------
    122710 7833 Coffee makes you hyper, but coffee shops are designed for people to chill, whereas alcohol is a depressant but bars and clubs are designed for people to be energetic.
    -------------
    115438 7368 Thermometers are speedometers for atoms
    -------------
    113906 7271 Tinder is the opposite of porn site advertisements. There actually are a ton of hot singles in my area, but none of them are interested in me.
    -------------
    113062 7217 Superman’s most unappreciated power is his extraordinary fine motor skill. He has almost unlimited strength and can literally move mountains, yet is also able to shake someone’s hand without inadvertently ripping off their arm.
    -------------
    109715 7003 The Porn sites in the Star Wars Universe must have sooooo many categories.
    -------------
    109774 7007 On a galactic scale, amber and pearl are far rarer than gold and diamonds because they require life.
    -------------
    103648 6616 If people from the Star Wars universe ever came to Earth, the fact that it has deserts, rain forests, prairies, tundras AND oceans would probably be a real mind fuck for them.
    -------------
    102282 6529 Your thoughts could be in such a personalized coded shorthand that even if someone could read your mind they wouldn’t understand what the hell you’re thinking.
    -------------
    150637 11338 Being able to do well in high school without having to put in much effort is actually a big disadvantage later in life.
    -------------
    136674 10287 Somebody at google was just like "yea, just have someone drive down every road on fucking earth".
    -------------
    133301 10033 Apparently, a lemon is not naturally occurring and is a hybrid developed by cross breeding a bitter orange and a citron. Life never gave us lemons; we invented them all by ourselves.
    -------------
    131345 9886 Avatar, the highest grossing film of all time, seems to have made zero impact on culture. No one references the movie and no one really talks about it since it was released 10 years ago.
    -------------
    129833 9772 If we look back, Pornhub has been one of the most innovative video sharing websites. They have had permanent dark mode for years, they popularized the preview setting where you can watch the video without starting it, and still are one of the few to have like/dislike percentage on display
    -------------
    120207 9048 The spiders that live in Buckingham Palace now are probably the descendants of the spiders that lived there during Victorian times - it's a whole parallel royal family but with spiders
    -------------
    119928 9027 If Eminem’s Mom wanted to she could probably make a good amount of money selling her own spaghetti sauce
    -------------
    108892 8196 Placing hand sanitizers in elevators would probably increase there usage simply because people have nothing else to do.
    -------------
    108269 8149 Nobody cares if their deodorant is 24 hour or 48 hour, they are still going to apply it once a day
    -------------
    108102 8137 Sunny D tastes like someone made a bet that they could make orange juice without oranges
    -------------

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```python
r_order = order[::-1][-40:]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order], capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30].replace("\n",""), ordered_contents) );
```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Extension to Starred rating systems

The above procedure works well for upvote-downvotes schemes, but what about systems that use star ratings, e.g. 5 star rating systems. Similar problems apply with simply taking the average: an item with two perfect ratings would beat an item with thousands of perfect ratings, but a single sub-perfect rating.

We can consider the upvote-downvote problem above as binary: 0 is a downvote, 1 if an upvote. A $N$-star rating system can be seen as a more continuous version of above, and we can set $n$ stars rewarded is equivalent to rewarding $\frac{n}{N}$. For example, in a 5-star system, a 2 star rating corresponds to 0.4. A perfect rating is a 1. We can use the same formula as before, but with $a,b$ defined differently:

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where

\begin{align}
& a = 1 + S \\\\
& b = 1 + N - S \\\\
\end{align}

where $N$ is the number of users who rated, and $S$ is the sum of all the ratings, under the equivalence scheme mentioned above.

##### Example: Counting Github stars

What is the average number of stars a Github repository has? How would you calculate this? There are over 6 million respositories, so there is more than enough data to invoke the Law of Large numbers. Let's start pulling some data. TODO

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting submissions formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead use a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally accurate?

```python
## Enter code here
import scipy.stats as stats
import numpy as np
exp = stats.expon( scale=4 )
N = 1e5
X = exp.rvs( int(N) )
plt.hist(X)
print(X.shape)
Y = np.cos(X)
print(Y)
print(np.mean(Y))

X_lt1 = X[X < 1]
print(X_lt1.shape)
Y_lt1 = np.cos(X_lt1)
print(np.mean(Y_lt1))
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by their percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

In August 2013, [a popular post](http://bpodgursky.wordpress.com/2013/08/21/average-income-per-programming-language/) on the average income per programmer of different languages was trending. Here's the summary chart: (reproduced without permission, cause when you lie with stats, you gunna get the hammer). What do you notice about the extremes?

------

#### Average household income by programming language

<table >
 <tr><td>Language</td><td>Average Household Income ($)</td><td>Data Points</td></tr>
 <tr><td>Puppet</td><td>87,589.29</td><td>112</td></tr>
 <tr><td>Haskell</td><td>89,973.82</td><td>191</td></tr>
 <tr><td>PHP</td><td>94,031.19</td><td>978</td></tr>
 <tr><td>CoffeeScript</td><td>94,890.80</td><td>435</td></tr>
 <tr><td>VimL</td><td>94,967.11</td><td>532</td></tr>
 <tr><td>Shell</td><td>96,930.54</td><td>979</td></tr>
 <tr><td>Lua</td><td>96,930.69</td><td>101</td></tr>
 <tr><td>Erlang</td><td>97,306.55</td><td>168</td></tr>
 <tr><td>Clojure</td><td>97,500.00</td><td>269</td></tr>
 <tr><td>Python</td><td>97,578.87</td><td>2314</td></tr>
 <tr><td>JavaScript</td><td>97,598.75</td><td>3443</td></tr>
 <tr><td>Emacs Lisp</td><td>97,774.65</td><td>355</td></tr>
 <tr><td>C#</td><td>97,823.31</td><td>665</td></tr>
 <tr><td>Ruby</td><td>98,238.74</td><td>3242</td></tr>
 <tr><td>C++</td><td>99,147.93</td><td>845</td></tr>
 <tr><td>CSS</td><td>99,881.40</td><td>527</td></tr>
 <tr><td>Perl</td><td>100,295.45</td><td>990</td></tr>
 <tr><td>C</td><td>100,766.51</td><td>2120</td></tr>
 <tr><td>Go</td><td>101,158.01</td><td>231</td></tr>
 <tr><td>Scala</td><td>101,460.91</td><td>243</td></tr>
 <tr><td>ColdFusion</td><td>101,536.70</td><td>109</td></tr>
 <tr><td>Objective-C</td><td>101,801.60</td><td>562</td></tr>
 <tr><td>Groovy</td><td>102,650.86</td><td>116</td></tr>
 <tr><td>Java</td><td>103,179.39</td><td>1402</td></tr>
 <tr><td>XSLT</td><td>106,199.19</td><td>123</td></tr>
 <tr><td>ActionScript</td><td>108,119.47</td><td>113</td></tr>
</table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. [Web](http://www.sloansportsconference.com/wp-content/uploads/2013/Going%20for%20Three%20Predicting%20the%20Likelihood%20of%20Field%20Goal%20Success%20with%20Logistic%20Regression.pdf). 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunss.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsx.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunsi.otf');
    }
    @font-face {
        font-family: "Computer Modern";
        font-weight: bold;
        font-style: oblique;
        src: url('http://9dbb143991406a7c655e-aa5fcb0a5a4ec34cff238a2d56ca4144.r56.cf5.rackcdn.com/cmunso.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

<style>
    img{
        max-width:800px}
</style>


```python
import numpy as np
import numpy.random as nr
import sympy as sy
import IPython.display as disp

sy.init_printing()

```

# 선형 연립 방정식
Systems of Linear Equations

미지수가 3개인 선형 연립 방정식을 생각해 보자.
Let's think about a system of linear equations with three unknowns.

```python
n = 3

x = np.array(sy.symbols(f'x:{n}'))
x

```

세 미지수를 모두 결정하려면, 보통 세개의 서로 선형 독립인 방정식이 필요하다.
To decide all three unknowns, usually we need three linearly independent equations.

```python
a = np.array(sy.symbols(
    f'a:{n}(:{n})'
)).reshape((n, n)).tolist()

b = sy.symbols(f'b:{n}')

```

```python
eqs = []
for coefs, const in zip(a, b):
    lhs = sum([aij * xj for aij, xj in zip(coefs, x)])
    eq = sy.Eq(lhs, const)
    eqs.append(eq)
    disp.display(eq)

```

행렬 형태로 정리해 보자
Let's rewrite in the matrix form

```python
matA = sy.Matrix(a)
vecB = sy.Matrix(b)
vecX = sy.Matrix(x)

eq_mat = sy.Eq(matA * vecX, vecB)

eq_mat

```

여기서 계수 행렬과 상수 벡터만 생각해 보자.
Here, let's just think about the coefficient matrix and constant vector.

```python
matAb = matA.col_insert(n, vecB)
matAb

```

## 가우스 소거법
Gauss Elimination

임의의 행렬을 생각해 보자.
Let's think about a matrix of random values.

```python
A = nr.randint(-100, 100, size=(n, n)) * 1.0
b = nr.randint(-100, 100, size=(n, 1)) * 1.0
Ab = np.hstack((A, b))
Ab

```

우선 첫 행의 첫 열 원소에 pivot 이라는 이름을 준다.
First, let's designate the first element of the first row as pivot.

```python
p = 0
pivot = Ab[p, p]

```

두번째 행 첫 열 원소를 pivot 으로 나눈 비를 계산한다.

Divide the element at the first column of the second row with pivot

```python
i = p + 1
ratio = Ab[i, p] / pivot
ratio

```

첫 행에 이 비를 곱한 후 둘째 행에서 뺀다.
Multiply the first row with this ratio and subtract from the second row.

```python
Ab[i, :]  = Ab[i, :] - ratio * Ab[p, :]
Ab

```

세번째 행에 대해서도 반복한다.

Let's repeat for the third row.

세번째 행 첫 열 원소를 pivot 으로 나눈 비를 계산한다.

Divide the element at the first column of the third row with pivot.

```python
i += 1
ratio = Ab[i, p] / pivot
ratio

```

첫 행에 이 비를 곱한 후 세째 행에서 뺀다.
Multiply the first row with this ratio and subtract from the third row.

```python
Ab[i, :]  = Ab[i, :] - ratio * Ab[p, :]
Ab

```

이제 p 에 1을 더하고 반복하자.
Now let's add 1 to `p` and repeat.

```python
p += 1
pivot = Ab[p, p]

```

`p+1` 행 `p` 열 원소를 `pivot` 으로 나눈 비를 계산한다.

Divide the element at the `p`th column of the `p+1`th row with pivot

```python
i = p + 1
ratio = Ab[i, p] / pivot
ratio

```

`p` 행에 이 비를 곱한 후 `p+1` 행에서 뺀다.

Multiply the `p`th row with this ratio and subtract from the `p+1`th row.

```python
Ab[i, :]  = Ab[i, :] - ratio * Ab[p, :]
Ab

```

이런 식으로 왼쪽 위로부터 오른쪽 아래로의 주대각선 아래 원소를 모두 0으로 만든다.
This way, make all elements below main diagonal, from the left upper corner to the right lower direction, zero.

## 후진대입법
Backward substitution

주대각선 아래가 모두 0이라면 아래와 같이 생각해 볼 수 있다.
If all elements below the main diagonal are zeros, we may think as follows.

```python
alpha = np.array(sy.symbols(
    f'alpha:{n}(:{n})'
)).reshape((n, n)).tolist()

beta = sy.symbols(f'beta:{n}')

```

```python
eqs2 = []
for p in range(n):

    lhs_list = []
    for i in range(p, n):
        lhs_list.append(alpha[p][i]*x[i])

    eq = sy.Eq(sum(lhs_list), beta[p])
    eqs2.append(eq)

for eq in eqs2:
    disp.display(eq)

```

맨 마지막 행에서 마지막 미지수를 알 수 있다.

From the last row, we can find the last unknown.

```python
sol = sy.Matrix([None] * n)

```

```python
sol_n_1 = sy.solve(eqs2[-1], x[-1])
sol[-1] = sol_n_1[0]
disp.display(sol)

```

그 하나 앞 미지수는 마지막에서 두번째 방정식에서 구할 수 있다.
We can find the second last unknown from the second last equation.

```python
eqs2[-2].subs(x[-1], sol[-1])

```

```python
sol_n_2 = sy.solve(eqs2[-2].subs(x[-1], sol[-1]), x[-2])
sol[-2] = sol_n_2[0]
disp.display(sol)

```

반복하면 모든 해를 구할 수 있다.
We can find all solutions this way.

`numpy.linalg` 의 `solve()` 함수를 이용할 수도 있다.
We can use `solve()` of `numpy.linalg`.

```python
import numpy.linalg as nl

x_sol = nl.solve(A, b)
x_sol

```

```python
assert 1e-7 > nl.norm(np.matrix(A) * np.matrix(x_sol) - b)

```

## 연습 문제
Exercise

위 방법을 적용 가능한 공학 문제 사례를 설명하고 `numpy.linalg.solve()`로 해를 구해 보시오. 이렇게 구한 해가 맞는지 어떻게 확인할 수 있는가?

Describe an engineering problem that we can apply the method above and find the solution using `numpy.linalg.solve()`. How can we verify if the solution is correct?

## Final Bell
마지막 종

```python
# stackoverfow.com/a/24634221
import os
os.system("printf '\a'");

```

```python

```


© 2018 Suzy Beeler and Vahe Galstyan. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT)

This exercise was generated from a Jupyter notebook. You can download the notebook [here](bacterial_growth.ipynb).
___

# Objective

In this tutorial, we will discuss how we can describe bacterial growth (i.e. exponential growth) mathematically. Given this mathematical model of how bacteria grow, we will then analyze images to determine the growth rate of a bacterial colony. For now, we will determine the growth rate value that best describes our data by fitting "by eye." Later in the bootcamp, we will learn more rigorous ways to determine which parameter value(s) best describe our data.

# Mathematically describing bacterial growth

Provided a nutrient-rich environment, bacteria will continually multiply, resulting in exponential growth which we can describe mathematically. Let's assume that the growth rate of bacteria is $k$. If $N(t)$ is the number of bacteria at time $t$, after a time increment $\Delta t$, the number of bacteria will be

\begin{align}
N(t+\Delta t) = N(t) + k \, N(t) \Delta t.
\end{align}

Taking the $\Delta t \rightarrow 0$ limit, we obtain the time evolution equation for N(t)

\begin{align}
\frac{N(t+\Delta t) - N(t)}{\Delta t} \approx \frac{\mathrm{d} N(t)}{\mathrm{d} t} = k N(t).
\end{align}

Knowing the initial number of bacteria, $N_0$, the solution to the above ordinary differential equation will be

\begin{align}
N(t) = N_0 e^{k t}.
\end{align}

The doubling time of bacteria, $\tau$, is also of interest. It's defined as the time it takes for a bacterium to multiply, i.e.

\begin{align}
N(t+\tau) = 2 N(t).
\end{align}

Substituting the solution for $N(t)$, we find

\begin{align}
N_0 e^{k(t+\tau)} &= 2N_0 e^{k t} \\
e^{k \tau} &= 2 \\
\tau &= \frac{\ln(2)}{k}.
\end{align}

Now that we have a mathematical description for how bacteria grow, we can analyze microscopy images of a bacterial colony over time to estimate the growth rate $k$ and the doubling time $\tau$.

# Analysis of bacterial colony images

Whenever we write code, we first need to import all the relevant packages that we will be using, as done below.

```python
# import the necessary modules
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# for pretty plots
import seaborn as sns
rc={'lines.linewidth': 2, 'axes.labelsize': 14, 'axes.titlesize': 14, \
    'xtick.labelsize' : 14, 'ytick.labelsize' : 14}
sns.set(rc=rc)

# for bulk acquisition of filenames
import glob

# for importing bacterial images
import skimage.io
```

Now, let's use the microscopy images of a growing bacterial colony to estimate the rate of bacterial growth. In the `data/` folder you can see microscopy images taken in *phase contrast* and *fluorescence* methods in 5 minute intervals. The phase images are named `ecoli_phase_xx.tif`, and the fluorescence ones are named `ecoli_TRITC_xx.tif`. Let's begin by taking a look at one of the fluorescence images.

```python
# read an example phase image
im = skimage.io.imread('./data/ecoli_growth/ecoli_TRITC_05.tif')

# take a look at its contents
im
```

    array([[205, 205, 205, ..., 205, 205, 206],
           [204, 204, 205, ..., 204, 205, 205],
           [204, 204, 205, ..., 204, 203, 203],
           ...,
           [205, 204, 204, ..., 203, 203, 203],
           [205, 204, 204, ..., 205, 205, 205],
           [205, 204, 204, ..., 205, 205, 205]], dtype=uint16)

Here we see that the image is just an array of integers. You might not be used to thinking of pictures in this way, but microscopy images in fact give us *quantitative* information, with each pixel in a 2D array being assigned a numerical value to describe the intensity, with larger numbers corresponding to brighter pixels. If we want to actually see the image (and not just the numerical values), we can call `plt.imshow()` on our image.

```python
# show the image, with a nice color map
plt.imshow(im, cmap = plt.cm.viridis)

# Remove the grid that is shown by default
plt.grid(False)
```

Let's hover over the pixels and get a sense of what values the background and bacterial pixels have.

```python
# allows us to interact with the image once it's displayed
%matplotlib notebook

plt.imshow(im, cmap=plt.cm.viridis)
plt.grid(False)
```

    <IPython.core.display.Javascript object>

    <IPython.core.display.Javascript object>

It seems like the background pixels have values around `205`, while the bacterial pixels have higher values of `210-220`. To see this more concretely, let's remember that this image contains *quantitative* data that we can plot. Below, let's plot a histogram of the pixel intensities.

```python
%matplotlib inline

# generate the image histogram
counts, bins = skimage.exposure.histogram(im)

# plot it as a bar plot
plt.bar(bins, counts)
plt.yscale('log')

# add appropriate labels.
plt.xlabel('pixel value')
plt.ylabel('number of observations')
```

We can more clearly see the two population of pixel values -- those that are dim, corresponding to the background and those that are bright, corresponding to bacteria. If we want the computer to be able to distinguish between bacteria and background (a process known as **segmenting** the image), we can use a **thresholding** approach, where we assign pixels above a certain value to `1` (indicating the pixel corresponds to bacteria) and pixels below a certain value to `0` (indicating the pixel corresponds to background). From the histogram above, a pixel intensity of `208` seems like a good threshold for distinguishing between these two populations of pixels.

```python
# Threshold value
threshold = 208

# Apply the threshold and show the segmented image
plt.imshow(im > threshold, cmap = plt.cm.viridis)
plt.grid(False)
```

Looks like thresholding did a decent job in isolating the bacterial cells! Let's follow the same procedure for an image at a later frame to make sure our threshold value work well for multiple images.

```python
# Load the image
im = skimage.io.imread('./data/ecoli_growth/ecoli_TRITC_18.tif')

# Show the image
plt.imshow(im, cmap = plt.cm.viridis)
plt.grid(False)
```

```python
# Apply the threshold and show the segmented image
plt.imshow(im > threshold, cmap = plt.cm.viridis)
plt.grid(False)
```

This looks pretty good. Now that we've shown that our selected threshold works well for both early and late images in the sequence, we can now apply this thresholding approach to all of the images. To do this, we will be using the `glob.glob()` function, which will grab all the images with a given name (like `ecoli_TRITC_xx.tif`).

```python
# Extract the names of all phase images
im_names = np.sort(glob.glob('./data/ecoli_growth/ecoli_TRITC_*.tif'))

# Take a look at some
im_names[0:5]
```

    array(['./data/ecoli_growth/ecoli_TRITC_00.tif',
           './data/ecoli_growth/ecoli_TRITC_01.tif',
           './data/ecoli_growth/ecoli_TRITC_02.tif',
           './data/ecoli_growth/ecoli_TRITC_03.tif',
           './data/ecoli_growth/ecoli_TRITC_04.tif'], dtype='<U38')

```python
# number of frames
n_frames = len(im_names)

# an array to store the bacterial areas
area_tot = np.zeros(n_frames)

# extract the bacterial areas
for n in range(n_frames):

    # read the frame
    im = skimage.io.imread(im_names[n])

    # apply the threshold
    im_thresh = im > 208

    # save the area in our array
    area_tot[n] = np.sum(im_thresh)

# time resolution in minutes
dt = 5

# measurement times
times = dt * np.arange(n_frames)

# plot the time dependence of the bacterial area
plt.plot(times, area_tot, '.')
plt.xlabel('Time (min)')
plt.ylabel('Total area (pixels)')
plt.show()
```

This in fact looks like an exponential growth curve! Since we are interested in the growth rate $\lambda$, let's rescale the areas so that in the first frame the area is normalized to `1`.

```python
# area normalization
area_tot_norm = area_tot / area_tot[0]

# plot the normalized area
plt.plot(times, area_tot_norm, '.')
plt.xlabel('Time (min)')
plt.ylabel('Normalized area')
plt.show()
```

We need to find the value of the growth rate $\lambda$ which can best explain the curve above. To that end, let's introduce a fitting error function defined as

\begin{align}
\mathrm{Err}(\lambda) = \sum_{i=1}^{N} \left(f_i- \hat{f}_i(\lambda) \right)^2,
\end{align}

where the iteration takes place over each frame. Here $f_i$ represents the normalized area at the $i^{\text{th}}$ frame, and $\hat{f}_i(\lambda) = \exp(\lambda \, t_i)$ is the theoretical prediction of the fractional growth for a given value of the growth rate. Let's write a function that evaluates the error for a given choice of $\lambda$.

```python
def err(norm_area, times, rate):
    """
    Returns the sum of the errors for the theory curve (given a the provided rate)
    versus the provided data (norm_area)
    """

    theory = np.exp(rate*times)

    return np.sum((theory - norm_area)**2)
```

To determine the growth rate of our bacterial colony, we should choose the value of $\lambda$ that minimizes the fitting error. Let's plot the error for different values of $\lambda$ and identify the optimal value.

```python
# number of points to plot
n_points = 100

# range of k's to test
ks = np.linspace(0.01, 0.05, n_points)

# initilize an array of error
errors = np.zeros(n_points)

# for each k, call our err funtion to determine the amount of error
for i in range(n_points):
    errors[i] = err(area_tot_norm, times, ks[i])

# plot the erros over the values of lambda
plt.plot(ks, errors, '.')
plt.ylabel('Error')
plt.xlabel('Growth rate $\lambda$ (1/min)')
plt.yscale('log')
plt.show()
```

The minimum seems to be around $0.03$, but we can find the exact value as shown below.

```python
# index of the optimal k
ind_optimal = np.where(errors == np.min(errors))

# optimal k
k_fit = ks[ind_optimal]

# show the optimal fit
print(k_fit)
```

    [0.0310101]

The optimal fit is $k \approx 0.03$ min$^{-1}$. It corresponds to a doubling time of $\tau \approx 23$ minutes, which is consistent with what we see with *E. coli* grown in the lab. In the final step, let's add the best growth curve on top of the experimental one to visually see the goodness of fit.

```python
# plot the normalized area
plt.plot(times, area_tot_norm, '.')
plt.plot(times, np.exp(k_fit*times))
plt.xlabel('Time (min)')
plt.ylabel('Normalized area')
plt.legend(['data','fit'])
plt.show()
```

```python

```


# Optimality conditions

Now we will move to studying constrained optimizaton problems i.e., the full problem
$$
\begin{align} \
\min \quad &f(x)\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_k(x) = 0\text{ for all }k=1,\ldots,K\\
&x\in \mathbb R^n.
\end{align}
$$

In order to identify which points are optimal, we want to define similar conditions as there are for unconstrained problems through the gradient:

>If $x$ is a  local optimum to function $f$, then $\nabla f(x)=0$.

## Feasible descent directions

Let $S\subset \mathbb R^n$ ($S\neq \emptyset$ closed) and $x^*\in S$.
**Definition:** The set
$$ D = \{d\in \mathbb R^n: d\neq0,x^*+\alpha d\in S \text{ for all } \alpha\in (0,\delta) \text{ for some } \delta>0\}$$
is called the cone of feasible directions of $S$ in $x^*$.

**Definition:** The set
$$ F = \{d\in \mathbb R^n: f(x^*+\alpha d)<f(x^*)\text{ for all } \alpha\in (0,\delta) \text{ for some } \delta>0\}$$
is called the cone of descent directions.

**Definition:** The set $F\cap D$ is called the cone of feasible descent directions.

**(Obvious) Theorem:** Consider an optimization problem
$$
\begin{align}
\min &\  f(x)\\
\text{s.t. }&\ x\in S
\end{align}
$$
and let $x^*\in S$. Now if $x^*$ is a local minimizer **then** the set of feasible descent directions $F\cap D$ is empty.

Since, if $\nabla f(x)d<0$, **then** $d$ is a descent direction, the following theorem follows easily.

**Theorem:** Consider an optimization problem
$$
\begin{align}
\min &\  f(x)\\
\text{s.t. }&\ x\in S
\end{align}
$$
and let $x^*\in S$. Now, if $x^*$ is a local minimizer, then $\{d\in B(0,1):\nabla f(x^*)d<0 \}\cap D$ is empty.

## KKT conditions

Unfortunately, the set $D$ is not easily explicitly modelled. Thus, we need to develop  methods for explicitly defining the set $D$ or even better the set $\{d\in B(0,1):\nabla f(x^*)d<0 \}\cap D$. This is done through the KKT conditions:

**Theorem (Kuhn-Tucker Necessary Conditions)** Let $x^**$ be a local minimum for problem
$$
$$
\begin{align} \
\min \quad &f(x)\\
\text{s.t.} \quad & g_j(x) \geq 0\text{ for all }j=1,\ldots,J\\
& h_k(x) = 0\text{ for all }k=1,\ldots,K\\
&x\in \mathbb R^n.
\end{align}
$$
$$

and assume that $x^*$ is regular. Then there exists unique Lagrance multiplier vectors $\lambda^* = (\lambda^*_1,\ldots,\lambda_J^*)$ and $\mu^*=(\mu_1^*,\ldots,\mu_K^*)$ such that
$$
\begin{align}
&\nabla_xL(x,\lambda,\mu) = 0\\
&\mu_j^*\geq0,\text{ for all }j=1,\ldots,J\\
&\mu_j^*=0,\text{for all }j\in A(x^*),
\end{align}
$$
where $$L(x,\lambda,\mu) = f(x)+\sum_{j=1}^J\lambda_jh_j(x) + \sum_{k=1}^K\mu_kg_k(x)$$ and $A(x^*)$ is the set of active constraints at $x^*$. If in addition $f$, $h$ and $g$ are twice continuously differentiable, it holds that
$$
yH_{x}L(x^*,\lambda^*,\mu^*)y\geq0, \text{ for all }y\in V(x^*),
$$
where
$$
V(x^*) = \{y:\nabla h_j(x^*)'y=0, \text{ for all }j=1,\ldots,J, \text{ and }\nabla g_k(x^*)'y=0, \text{ for all }j\in A(x^*).
$$

**Example (page 285, Bertsekas: Nonlinear Programming)** Consider the optimizaiton problem
$$
\begin{align}
\min &\qquad \frac12 (x_1^2+x^2_2+x^2_3)\\
\text{s.t}&\qquad x_1+x_2+x_3\geq 0.
\end{align}
$$
Let us verify the Kuhn-Tucker necessary conditions for the local optimum $x^*=(-1,-1,-1)$.

```python
def f(x):
    return 0.5*sum([i**2 for i in x])
def g(x):
    return 3-sum(x)
def h(x):
    return 0*sum(x)
```

```python
import numpy as np
import ad
def grad_x_L(x,lambda_,mu,f,g,h):
    return ad.gh(f)[0](x)+lambda_*np.array(ad.gh(h)[0](x))+mu*np.array(ad.gh(g)[0](x))
```

```python
import ad
mu = 1
lambda_ = 10 #Does not play a role. Think why?
x_opt = [-1,-1,-1]

print grad_x_L(x_opt,lambda_,mu,f,g,h)

print g(x_opt)
```

    [-2. -2. -2.]
    6


# Números naturales. Inducción. Recursividad.

## Naturales

En `python` podemos utilizar `isinstance` para determinar si un objeto es un entero

```python
isinstance(4,int)
```

    True

```python
isinstance([3,4],int)
```

    False

Podemos con esta función definir otra que detecte si un objeto es un número es natural

```python
def isnatural(n):
    if not isinstance(n,int):
        return false
    return n>=0
```

```python
isnatural(3)
```

    True

```python
isnatural(-3)
```

    False

La funciones sucesor y predecesor quedarían como sigue

```python
sucesor = lambda x: x+1
```

```python
sucesor(2)
```

    3

```python
def prec(n):
    if not isnatural(n):
        raise TypeError("El argumento debe ser un número natural")
    if n==0:
        raise ValueError("El 0 no tiene predecesor")
    return n-1
```

```python
prec(1)
```

    0

Con esto podemos definir una suma

```python
def suma(m,n):
    if n == 0:
        return m
    return sucesor(suma(m,prec(n)))
```

```python
suma(2,3)
```

    5

## Sucesiones, inducción

Con la librería `sympy` podemos hacer cálculo simbólico. En particular, podemos calcular el valor de algunas sumas con parámetros. Una ramificación de `sympy`, `diofant`, permite resolver algunos de los problemas que tenía `sympy` con `rsolve`.

```python
from diofant import *
```

Cuando vamos a utilizar un símbolo tenemos que declararlo primero. Para el ejemplo que sigue vamos a utilizar `n` como entero e `i` como contador

```python
n, i = symbols("n, i", integer = True)
```

### Algunas sumatorias

Calculemos por ejemplo $\sum_{i=1}^n i$

```python
s = Sum(i,(i,1,n))
```

```python
s
```

\begin{equation}\sum_{i=1}^{n} i\end{equation}

Si queremos calcular el "valor" de esta sumatoria, podemos utilizar el método `doit`

```python
s.doit()
```

\begin{equation}\frac{n^{2}}{2} + \frac{n}{2}\end{equation}

```python
pprint(_)
```

     2
    n    n
    ── + ─
    2    2

```python
summation(i,(i,1,n))
```

\begin{equation}\frac{n^{2}}{2} + \frac{n}{2}\end{equation}

Y una suma de potencias

```python
s=Sum(i**30,(i,1,n))
```

```python
s.doit()
```

\begin{equation}\frac{n^{31}}{31} + \frac{n^{30}}{2} + \frac{5 n^{29}}{2} - \frac{203 n^{27}}{6} + \frac{1131 n^{25}}{2} - \frac{16965 n^{23}}{2} + \frac{216775 n^{21}}{2} - \frac{2304485 n^{19}}{2} + \frac{19959975 n^{17}}{2} - \frac{137514723 n^{15}}{2} + \frac{731482225 n^{13}}{2} - \frac{31795091601 n^{11}}{22} + \frac{8053785025 n^{9}}{2} - \frac{102818379585 n^{7}}{14} + \frac{15626519181 n^{5}}{2} - \frac{23749461029 n^{3}}{6} + \frac{8615841276005 n}{14322}\end{equation}

```python
a = Symbol("a")
```

```python
Sum(a**i,(i,1,n)).doit()
```

\begin{equation}\begin{cases} n & \text{for}\: a = 1 \\\frac{a - a^{n + 1}}{- a + 1} & \text{otherwise} \end{cases}\end{equation}

### Inducción

Veamos cómo podemos utilizar `sympy` para hacer algunos ejemplos de inducción

Por ejemplo, veamos que para todo $6\mid 7^n-1$ para todo $n\in \mathbb N$. Empezamos definiendo una función que nos devuelva $7^n-1$

```python
f = lambda n: 7**n -1
```

Primero veamos qué valor tiene en el 0

```python
f(0)
```

\begin{equation}0\end{equation}

Si por hipótesis de inducción $f(n)$ es un múltiplo de 6, entonces para probar que $f(n+1)$ también lo es, bastará demostrar que la diferencia $f(n+1)-f(n)$ es un múltiplo de 6

```python
simplify(f(n+1)-f(n))
```

\begin{equation}6 \cdot 7^{n}\end{equation}

```python
f = lambda n: 7**(2*n)+16*n-1
```

```python
f(0)
```

\begin{equation}0\end{equation}

```python
simplify((f(n+1)-f(n)) % 64)
```

\begin{equation}16 \operatorname{Mod}{\left (3 \cdot 7^{2 n} + 1,4 \right )}\end{equation}

Por tanto, si $3\times 7^{2n}+1$ es un múltiplo de $4$, habremos acabado

```python
g = lambda n: 3*7**(2*n)+1
```

```python
g(0)
```

\begin{equation}4\end{equation}

```python
simplify((g(n+1)-g(n))%4)
```

\begin{equation}0\end{equation}

### Recursividad e iteración

Veamos cómo las funciones recursivas se pueden acelerar usando memorización de los pasos anteriores

```python
fib = lambda n: n if n<2 else fib(n-2)+fib(n-1)
```

```python
fib(10)
```

\begin{equation}55\end{equation}

```python
fibonacci(10)
```

\begin{equation}55\end{equation}

```python
import time

```

```python
start=time.time()
fib(30)
time.time()-start

```

\begin{equation}0.41254711151123047\end{equation}

```python
start=time.time()
fibonacci(30)
time.time()-start
```

\begin{equation}0.0004341602325439453\end{equation}

Un posible solución a este problema es ir almacenando los resultados previos de cálculo, lo cual es conocido como [memoization](http://stackoverflow.com/questions/1988804/what-is-memoization-and-how-can-i-use-it-in-python)

```python
import functools
```

```python
@functools.lru_cache(maxsize=None)
def fibo(num):
    if num < 2:
        return num
    else:
        return fibo(num-1) + fibo(num-2)
```

```python
start=time.time()
fibo(30)
time.time()-start
```

\begin{equation}0.0001010894775390625\end{equation}

## Resolución de ecuaciones recursivas

Las ecuaciones en recurrencia se pueden resolver con el comando `rsolve` de `sympy`

Empecemos con un ejemplo:
- $a(0)=0$,
- $a(1)=1$,
- $a(n+2)=5a(n+1)-6a(n)$.

Empezamos declarando `a` como función

```python
a = Function('a')
```

```python
rsolve(a(n+2)-5*a(n+1)+6*a(n),a(n))
```

\begin{equation}2^{n} C_{0} + 3^{n} C_{1}\end{equation}

```python
rsolve(a(n+2)-5*a(n+1)+6*a(n),a(n), {a(0):0,a(1):1})
```

\begin{equation}- 2^{n} + 3^{n}\end{equation}

Vamos a resolverlo mediante el polinomio característico

```python
x=Symbol("x")
solve(x**2-5*x+6)
```

\begin{equation}\left [ 2, \quad 3\right ]\end{equation}

Y por tanto la solución general será de la forma $a_n=u 2^n + v 3^n$, para ciertas constantes $u$ y $v$ que tenemos que encontrar a partir de las condiciones iniciales

Imponemos por tanto que $a_0=0=u+v$ y que $a_1=1=2u+3v$

```python
u,v = symbols("u,v")
solve([u+v,2*u+3*v-1])
```

\begin{equation}\left \{ u : -1, \quad v : 1\right \}\end{equation}

Por lo que $a_n=-2^n+3^n$, como habíamos visto arriba

Veamos un ejemplo de una que no sea homogénea

```python
rsolve(a(n+2)-5*a(n+1)+6*a(n)-n,a(n))
```

\begin{equation}2^{n} C_{0} + 3^{n} C_{1} + \frac{n}{2} + \frac{3}{4}\end{equation}

Y ahora otro ejemplo no lineal

```python
f=a(n)-(n-1)*(a(n-1)+a(n-2))
```

```python
rsolve(f,a(n),{a(0):1})
```

\begin{equation}n!\end{equation}

```python
simplify(factorial(n)-(n-1)*(factorial(n-1)+factorial(n-2)))
```

\begin{equation}0\end{equation}

Por último, veamos algunos ejemplos de la relación de problemas

```python
from diofant import *
n=Symbol("n", integer = True)
a=Function('a')
rsolve(a(n)-2*a(n-1)-n,a(n),[1])
```

\begin{equation}3 \cdot 2^{n} - n - 2\end{equation}

Comprobamos la solución

```python
f = lambda n : 3*2**n-n-2
```

```python
f(0)
```

\begin{equation}1\end{equation}

```python
simplify(f(n)-2*f(n-1)-n)
```

\begin{equation}0\end{equation}

Veamos ahora un ejemplo con término no lineal

```python
from diofant import *
n=Symbol("n", integer = True)
x=Function('x')
rsolve(x(n)-2*x(n-1)-3**n,x(n),[1])
```

\begin{equation}- 2 \cdot 2^{n} + 3^{n + 1}\end{equation}

```python
f = lambdify(n,_)
```

```python
f(1)
```

\begin{equation}5\end{equation}

```python
f(n)-2*f(n-1)
```

\begin{equation}- 2 \cdot 2^{n} + 4 \cdot 2^{n - 1} - 2 \cdot 3^{n} + 3^{n + 1}\end{equation}

```python
expand(_)
```

\begin{equation}3^{n}\end{equation}

Otro ejemplo más

```python
rsolve(x(n)-2*x(n-1)-n-2**n,x(n),[0])
```

\begin{equation}2^{n} n + 2 \cdot 2^{n} - n - 2\end{equation}

```python
f = lambda n: n*2**n
```

```python
f(n)-2*f(n-1)
```

\begin{equation}2^{n} n - 2 \cdot 2^{n - 1} \left(n - 1\right)\end{equation}

```python
expand(_)
```

\begin{equation}2^{n}\end{equation}

Los ejemplos anteriores y el siguiente dieron lugar a este [informe de error](https://github.com/diofant/diofant/issues/451)

```python
rsolve(x(n)-2*x(n-1)-3**n*(n+1),x(n),[0])
```

\begin{equation}3 \cdot 2^{n} + 3^{n + 1} \left(n - 1\right)\end{equation}

```python
f = lambda n:3*2**n-3*3**n+3*n*3**n
```

```python
f(0)
```

\begin{equation}0\end{equation}

```python
f(1)
```

\begin{equation}6\end{equation}

```python
f(2)
```

\begin{equation}39\end{equation}

```python
f(n)-2*f(n-1)-(n+1)*3**n
```

\begin{equation}3 \cdot 2^{n} - 6 \cdot 2^{n - 1} + 3 \cdot 3^{n} n - 3^{n} \left(n + 1\right) - 3 \cdot 3^{n} - 2 \cdot 3^{n - 1} \left(3 n - 3\right) + 6 \cdot 3^{n - 1}\end{equation}

```python
expand(_)
```

\begin{equation}0\end{equation}

Resolvamos ahora $x_0=0$, $x_1=1$, $x_n=3x_{n-1}-2x_{n-2}+2^n+2n$.

```python
rsolve(x(n)-3*x(n-1)+2*x(n-2)-2**n-2*n,x(n),[0,1])
```

\begin{equation}3 \cdot 2^{n} + n \left(2^{n + 1} - n - 5\right) - 3\end{equation}

Hagámoslo ahora a mano. La solución tendrá un polinomio característico asociado $(x-2)(x-1)(x-1)^2(x-2)=(x-2)^2(x-1)^3$. Por tanto la solución será de la forma $x_n=r2^n+sn2^n+t 1^n+u n 1^n + vn^2 1^n$.

```python
def f(n):
    if n==0:
        return 0
    if n==1:
        return 1
    return 3*f(n-1)-2*f(n-2)+2**n+2*n
```

```python
[f(i) for i in range(5)]
```

\begin{equation}\left [ 0, \quad 1, \quad 11, \quad 45, \quad 137\right ]\end{equation}

```python
from diofant import *
```

```python
r,s,t,u,v = symbols("r,s,t,u,v")
n=Symbol("n", integer=True)
eq = lambda n:r*2**n+s*n*2**n+t+u*n+v*n**2
solve([eq(0)-f(0),eq(1)-f(1),eq(2)-f(2),eq(3)-f(3), eq(4)-f(4)],[r,s,t,u,v])
```

\begin{equation}\left \{ r : 3, \quad s : 2, \quad t : -3, \quad u : -5, \quad v : -1\right \}\end{equation}

```python
eq(n).subs(_)
```

\begin{equation}2 \cdot 2^{n} n + 3 \cdot 2^{n} - n^{2} - 5 n - 3\end{equation}

```python
f = lambdify(n,_)
```

```python
simplify(f(n)-3*f(n-1)+2*f(n-2)-2**n-2*n)
```

\begin{equation}0\end{equation}


# Evidence calculation for EFT expansions

## The toy model

Here we continue to explore aspects of Bayesian statistical analysis using toy models for effective field theories (EFTs), namely Taylor series of some specified functions. In this notebook we are exploring the evidence for how many coefficients in the EFT expansion are determined by the given data.

Let's first review the function we are using as a toy model, taken from [*Bayesian parameter estimation for effective field theories*](https://arxiv.org/abs/1511.03618):

$$
g(x) = \left(\frac12 + \tan\left(\frac{\pi}{2}x\right)\right)^2
$$

represents the true, underlying theory.  It has a Taylor expansion

$$
g(x) = 0.25 + 1.57x + 2.47x^2 + 1.29 x^3 + \cdots
$$

Our model for an EFT for this "theory" is

$$
g_{\rm th}(x) \equiv \sum_{i=0}^k a_i x^i \;.
$$

In mini-project I, our general task was to fit 1, 2, 3, ... of the parameters $a_i$ and to analyze the results.

$% Some LaTeX definitions we'll use.
\newcommand{\pr}{{p}}  %\newcommand{\pr}{\textrm{p}}
\newcommand{\abar}{\bar a}
\newcommand{\avec}{{\bf a}}
\newcommand{\kmax}{k_{\rm max}}
$

## The statistical model (recap)

Here we are given data with simple error bars, which imply that the probability for any *single* data point is a normal distribution about the true value. That is,

$$
y_i \sim \mathcal{N}(y_M(x_i;\theta), \varepsilon_i)
$$

or, in other words,

$$
\pr(x_i\mid y_i, \theta) = \frac{1}{\sqrt{2\pi\varepsilon_i^2}} \exp\left(\frac{-\left[y_i - y_M(x_i;\theta)\right]^2}{2\varepsilon_i^2}\right)
$$

where $\varepsilon_i$ are the (known) measurement errors indicated by the error bars.

Assuming all the points are independent, we can find the full likelihood by multiplying the individual likelihoods together:

$$
\pr(D\mid\theta) = \prod_{i=1}^N \pr(x_i,y_i | \theta)
$$

For convenience and numerical accuracy, this is usually expressed in terms of the log-likelihood:

$$
\log \pr(D\mid\theta) = -\frac{1}{2}\sum_{i=1}^N\left(\log(2\pi\varepsilon_i^2) + \frac{\left[y_i - y_M(x_i;\theta)\right]^2}{\varepsilon_i^2}\right)
$$

We consider two priors for the coefficients.  The first is a Gaussian prior that encodes naturalness through the parameter $\abar$:

$$
\pr(\avec\mid \abar, I) = \left(\frac{1}{\sqrt{2\pi}\abar}\right)^{k+1} \exp{\left(-\frac{\avec^2}{2\abar^2}\right)}
$$

with $\abar$ taken to be fixed (at $\abar_{\rm fix} = 5$ usually).  That is, the prior pdf for $\abar$ is

$$
\pr(\abar) = \delta(\abar - \abar_{\rm fix}) \;.
$$

(In more recent work, we have used a conjugate prior for $\abar$ that simplifies the calculations.)

The second is an uninformative uniform prior that we take to be a constant (cutting it off only at very high values, which
may not even be needed).
Given likelihood and prior, the posterior pdf by Bayes' Theorem is

$$
\pr(\avec\mid D, k, \kmax, I) = \frac{\pr(D\mid \avec, k, \kmax, I)\; \pr(\avec\mid I)}{\pr(D \mid k, \kmax, I)}
$$

We have focused previously on calculating this posterior to find distributions for the coefficients $\theta = \{a_0, a_1, \cdots, a_k\}$.
Furthermore, up to now we have ignored the denominator, which is the *evidence*, because we didn't need to calculate it independently.  Now we will calculate it.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import seaborn; seaborn.set("talk") # for plot formatting
import scipy.stats as stats
from scipy import linalg

from cycler import cycler
from matplotlib.cm import get_cmap

```

## The Data and the true result

Let's start by defining the exact function and the data for the toy model.

```python
def g_fun(x):
    """
    Toy function to model an EFT expansion.  It has a Taylor expansion about
    x=0 with a radius of convergence of 1.
    """
    return (0.5 + np.tan(np.pi * x / 2.))**2

def y_model(x_pts, theta, orders=None):
    """
    Returns the evaluation of the theoretical model at all x values in the
    numpy array x_pts, using orders coefficients from theta (defaults to all).
    """
    if orders is None:   # allow for not using the full theta vector
        orders = len(theta)

    return np.array( [ np.sum(
                               [theta[i] * x**i for i in range(orders)]
                             ) for x in x_pts ] )

```

```python
theta_true = np.array([0.25, 1.5707963, 2.4674011, 1.2919282, 4.0587121,
                       1.275082, 5.67486677])

# Generate data points as described in the paper; remember these are relative
#  errors, so multiply the percent by the data at each x.
x_max = 1./np.pi   # we'll eventually test sensitivity to x_max
x_data_pts = np.linspace(x_max/10., x_max, 10) # don't start at x=0
eta = 0.05         # specified relative uncertainty is 5%

# Here we generate new (different) data points with every run
y_data_pts = g_fun(x_data_pts) * \
                  (1. + stats.norm.rvs(0., eta, size=len(x_data_pts)) )
# *** The following is the exact data with errors from the paper ***
y_data_pts = np.array([0.31694, 0.33844, 0.42142, 0.57709, 0.56218, \
                        0.68851, 0.73625, 0.87270, 1.0015, 1.0684])

dy_data = eta * y_data_pts

# Always make a figure to check your data!
fig = plt.figure(figsize=(8,6))
ax = fig.add_subplot(1,1,1)
ax.errorbar(x_data_pts, y_data_pts, dy_data, fmt='o')
ax.set_xlabel(r'x')
ax.set_ylabel(r'g(x)')
ax.set_xlim(0, 0.5)
ax.set_ylim(0, 1.5)

x_pts_all = np.arange(0., 1., .01)
ax.plot(x_pts_all, g_fun(x_pts_all), color='red', alpha=0.5, label='exact')
ax.set_title('Toy function, data, and first terms in expansion')

n_dim = 3
colors = ['b', 'g', 'c', 'm', 'k']
for order in range(n_dim):
    ax.plot(x_pts_all, y_model(x_pts_all, theta_true[:n_dim], order+1),
            label=f'order {order:d}', color=colors[order], alpha=0.8)

ax.legend()
fig.tight_layout()
```

## Evidence calculation

Now we seek to reproduce and understand Figure 8 in the paper [*Bayesian parameter estimation for effective field theories*](https://arxiv.org/abs/1511.03618), which shows that the evidence for the model expansion up to order $\kmax$ *saturates* (i.e., increases up to a maximum and then flattens out close to that value).  This is in contrast to the more typical expectation from evidence calculations that lead to a definite peak.

The evidence can be expressed by marginalization as an integral over *all possible* $\avec$. (The notation with $k$ and $\kmax$ is for consistency with the paper; for our purposes today consider this as the evidence for an expansion up to order $k$.)

$$
\begin{align}
\pr(D \mid k \leq \kmax, \kmax, I) &= \int d\abar \int d\avec \, \pr(D \mid \avec, k=\kmax, \kmax, I) \;
   \pr(\avec\mid\abar, I)\; \pr(\abar\mid I)
\end{align}
$$

*If you don't see how this equations comes about, please ask!*

The first term in the integrand is the likelihood, which we saw above is a multivariate Gaussian and, in the present case with independent points, it is very simple, just the product of one-dimensional Gaussians.  If we take the case of a Gaussian prior for $\avec$ and the fixed (delta function) prior for $\abar$, we can do the $\abar$ integral for free and the remaining integral for the evidence can be done analytically.

### Evidence using linear algebra and Gaussian integrals

If we write the multivariate Gaussians in the evidence in matrix form, we can use the basic formula for integration:

$$
\int e^{-\frac12 x^T A x + B^T x}\, d^nx = \sqrt{\det (2\pi A^{-1})} \; e^{\frac12 B^T A^{-1} B}
$$

where $x$ and $B$ are n-dimensional vectors and $A$ is an $n\times n$ matrix, with $n$ the number of data points.  The $x_i$ integrations are from $-\infty$ to $+\infty$.

### Evidence using conjugate prior

The usefulness of a conjugate prior is in carrying out a Bayesian update without having to do any calculation.  Recall yet again how Bayes theorem tells us how to update (the information $I$ will be implicit in the following):

$$
\pr(\theta\mid D) = \frac{\pr(D\mid\theta)\; \pr(\theta)}{\pr(D)}
$$

If $\pr(\theta)$ is a conjugate prior to the likelihood, the updating consists solely of changing the parameters that specify the prior pdf.

The most complete table of conjugate priors out in the wild seems to be the Wikipedia webpage [Conjugate Prior](https://en.wikipedia.org/wiki/Conjugate_prior#Table_of_conjugate_distributions).  Take a look!

```python
def make_matrices(x_pts, y_pts, dy_pts, k_order, a_bar):
    """
    Construct and return the matrices we'll need to calculate the evidence.
    We have only one observable for now, so d is omitted.
    """
    m = k_order + 1   # number of coefficients is one more than the order

    A_mat = np.array( [[x**i for x in x_pts] for i in range(m)] ).T
    Sigma_mat = np.diag(dy_pts**2)
    Vp_mat = a_bar**2 * np.eye(m)
    y_vec = y_pts

    return A_mat, Sigma_mat, Vp_mat, y_vec

def gaussian_norm(cov_mat):
    """Return the normalization factor for Gaussians.
       You can decide whether to use a covariance matrix or its inverse."""
    return 1. / np.sqrt(linalg.det(2. * np.pi * cov_mat))

```

```python
# step through the orders
k_max = 10
k_orders = range(k_max)

evidence = np.zeros(k_max)
for k_order in k_orders:
    a_bar = 5.
    A_mat, Sigma_mat, Vp_mat, y_vec = make_matrices(x_data_pts, y_data_pts,
                                                    dy_data, k_order, a_bar)

    Sigma_mat_inv = linalg.inv(Sigma_mat)

    Lambda_mat = A_mat.T @ Sigma_mat_inv @ A_mat + linalg.inv(Vp_mat)
    Lambda_mat_inv = linalg.inv(Lambda_mat)

    Vp_mat_inv = linalg.inv(Vp_mat)

    a_hat = Lambda_mat_inv @ A_mat.T @ Sigma_mat_inv @ y_vec

    chisq_min = (y_vec - A_mat @ a_hat).T @ Sigma_mat_inv @ \
                (y_vec - A_mat @ a_hat)

    evidence[k_order] = np.sqrt(linalg.det(2.*np.pi*Lambda_mat_inv)) \
                        * gaussian_norm(Sigma_mat) * np.exp(-chisq_min / 2.) \
                        * gaussian_norm(Vp_mat) \
                        * np.exp(- a_hat.T @ Vp_mat_inv @ a_hat / 2.)

fig, ax = plt.subplots(figsize=(8,6))
ax.plot(k_orders[1:], evidence[1:], color='blue', marker='o',
        linestyle='solid', linewidth=1, markersize=12)
ax.set_title('Evidence [Fig. 8 in J Phys. G 43, 074001]')
ax.set_xlabel(r'$k$')
ax.set_ylabel(r'$p(D1_{5\%} \mid k, k_{\rm max}=k)$')

fig.tight_layout()
```

## Things to try:
* What do you expect to happen if you increase the range of data (set by `x_max` at the upper end)?
* What do you expect to happen if you change (first decrease, then increase) the relative error at each point?
* What happens if you comment the definition of `y_data_pts` that uses the exact data from the paper and instead generate the noise randomly.  Does the pattern of the evidence change?  Does the magnitude of the evidence change?

## Notes
* The simple expression for $\hat a$ here, which minimizes $\chi^2$ (or, equivalently, maximizes the likelihood), analytically reproduces the results we worked hard for earlier to get by sampling.  The point of the latter exercise was to illustrate in a checkable problem how to do sampling, not because it was required in this case.

```python

```