# Chapter 4
`Original content created by Cam Davidson-Pilon`

`Ported to Python 3 and PyMC3 by Max Margenot (@clean_utensils) and Thomas Wiecki (@twiecki) at Quantopian (@quantopian)`

______

## The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been using this simple idea in every example thus far.

### The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear by examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for almost *any distribution*, minus some important cases we will encounter later.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size = 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to its parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```python
%matplotlib inline
import numpy as np
from IPython.core.pylabtools import figsize
import matplotlib.pyplot as plt

figsize( 12.5, 5 )

sample_size = 100000
expected_value = lambda_ = 4.5
poi = np.random.poisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average),
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend();
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This is simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

The above formulae is interpretable as a distance away from the true value (on average), for some $N$. (We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

By computing the above many, $N_y$, times (remember, it is random), and averaging them:

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \rightarrow E[ Y_k ] = E\;\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \right]$$

Finally, taking the square root:

$$ \sqrt{\frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k} \approx D(N) $$

```python
figsize( 12.5, 4)

N_Y = 250 #use this many to approximate D(N)
N_array = np.arange( 1000, 50000, 2500 ) #use this many samples in the approx. to the variance.
D_N_results = np.zeros( len( N_array ) )

lambda_ = 4.5
expected_value = lambda_ #for X ~ Poi(lambda) , E[ X ] = lambda

def D_N( n ):
    """
    This function approx. D_n, the average variance of using n samples.
    """
    Z = poi( lambda_, (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for i,n in enumerate(N_array):
    D_N_results[i] =  D_N(n)

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 2, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " );
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of convergence to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 5, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 5 ) =  \frac{1}{N}\sum_{i=1}^N \mathbb{1}_{z > 5 }(Z_i) $$

```python
N = 10000
print( np.mean( [ np.random.exponential( 0.5 ) > 2.0 for i in range(N) ] ) )
```

    0.0193

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give us *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```python
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = np.random.randint
norm = np.random.normal

#generate some artificial population numbers
population = pop_generator(100, 1500, n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height,
                                        population[i] ).mean()

#located the counties with the apparently most extreme average heights.
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population size vs. recorded average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1);
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error results from the calculated average of smaller populations not being a good reflection of the true expected value of the population (which in truth should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 1500, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```python
print("Population sizes of 10 'shortest' counties: ")
print(population[ np.argsort( average_across_county )[:10] ], '\n')
print("Population sizes of 10 'tallest' counties: ")
print(population[ np.argsort( -average_across_county )[:10] ])
```

    Population sizes of 10 'shortest' counties:
    [119 123 116 118 103 100 253 171 458 101]

    Population sizes of 10 'tallest' counties:
    [164 116 218 130 129 131 224 154 179 142]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```python
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);
```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How to order Reddit submissions

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, called submissions, for people to comment on. Redditors can vote up or down on each submission (called upvotes and downvotes). Reddit, by default, will sort submissions to a given subreddit by Hot, that is, the submissions that have the most upvotes recently.

How would you determine which submissions are the best? There are a number of ways to achieve this:

1. *Popularity*: A submission is considered good if it has many upvotes. A problem with this model is that a submission with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the submission is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of submission. Depending on when a submission is posted, the website may be experiencing high or low traffic. The difference method will bias the *Top* submissions to be the those made during high traffic periods, which have accumulated more upvotes than submissions that were not so graced, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the submission. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter-example is, if we use per second, a 1 second old submission with 1 upvote would be better than a 100 second old submission with 99 upvotes. One can avoid this by only considering at least t second old submission. But what is a good t value? Does this mean no submission younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old submissions).
3. *Ratio*: Rank submissions by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new submissions who score well can be considered Top just as likely as older submissions, provided they have many upvotes to total votes. The problem here is that a submission with a single upvote (ratio = 1.0) will beat a submission with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter submission is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former submission, with a single upvote, is in fact a better submission than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former submission might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this submission a upvote, versus a downvote"). So the 999 upvote/1 downvote submission probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the submission with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is to look at the historical distribution of upvote ratios. This can be accomplished by scraping Reddit's submissions and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of submissions have very few votes, hence there will be many submissions with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use submissions with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of submissions available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards submissions of these two subreddits are very different: visitors are likely friendly and affectionate in the former, and would therefore upvote submissions more, compared to the latter, where submissions are likely to be controversial and disagreed upon. Therefore not all submissions are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `top_showerthoughts_submissions.py` will scrape the best posts from the `showerthoughts` community on Reddit. This is a text-only community so the title of each post *is* the post. Below is the top post as well as some other sample posts:

```python
#adding a number to the end of the %run call will get the ith top post.
%run top_showerthoughts_submissions.py 2

print("Post contents: \n")
print(top_post)
```

    here
    0
    1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    17
    18
    19
    20
    21
    22
    23
    24
    25
    26
    27
    28
    29
    30
    31
    32
    33
    34
    35
    36
    37
    38
    39
    40
    41
    42
    43
    44
    45
    46
    47
    48
    49
    50
    51
    52
    53
    54
    55
    56
    57
    58
    59
    60
    61
    62
    63
    64
    65
    66
    67
    68
    69
    70
    71
    72
    73
    74
    75
    76
    77
    78
    79
    80
    81
    82
    83
    84
    85
    86
    87
    88
    89
    90
    91
    92
    93
    94
    95
    96
    97
    Post contents:

    Finding food in the forest is easy, the problem is finding poison is 20 times easier

    <Figure size 900x468 with 0 Axes>

```python
"""
contents: an array of the text from the last 100 top submissions to a subreddit
votes: a 2d numpy array of upvotes, downvotes for each submission.
"""
n_submissions = len(votes)
submissions = np.random.randint( n_submissions, size=4)
print("Some Submissions (out of %d total) \n-----------"%n_submissions)
for i in submissions:
    print('"' + contents[i] + '"')
    print("upvotes/downvotes: ",votes[i,:], "\n")
```

    Some Submissions (out of 1 total)
    -----------
    "We teach small children naps are good, but it takes 1-2 decades to understand how true that is."
    upvotes/downvotes:  [37  1]

    "We teach small children naps are good, but it takes 1-2 decades to understand how true that is."
    upvotes/downvotes:  [37  1]

    "We teach small children naps are good, but it takes 1-2 decades to understand how true that is."
    upvotes/downvotes:  [37  1]

    "We teach small children naps are good, but it takes 1-2 decades to understand how true that is."
    upvotes/downvotes:  [37  1]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular submission's upvote/downvote pair.

```python
import pymc3 as pm

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    """
    This function accepts the number of upvotes and downvotes a particular submission recieved,
    and the number of posterior samples to return to the user. Assumes a uniform prior.
    """
    N = upvotes + downvotes
    with pm.Model() as model:
        upvote_ratio = pm.Uniform("upvote_ratio", 0, 1)
        observations = pm.Binomial( "obs",  N, upvote_ratio, observed=upvotes)

        trace = pm.sample(samples, step=pm.Metropolis())

    burned_trace = trace[int(samples/4):]
    return burned_trace["upvote_ratio"]

```

Below are the resulting posterior distributions.

```python
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(submissions)):
    j = submissions[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 10, normed = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 10, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different submissions");
```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the submissions from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean is a bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```python
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(submissions)):
    j = submissions[i]
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")
plt.title("Posterior distributions of upvote ratios on different submissions");
order = np.argsort( -np.array( lower_limits ) )
print(order, lower_limits)
```

The best submissions, according to our procedure, are the submissions that are *most-likely* to score a high percentage of upvotes. Visually those are the submissions with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best.  When using the lower-bound of the 95% credible interval, we believe with high certainty that the 'true upvote ratio' is at the very least equal to this value (or greater), thereby ensuring that the best submissions are still on top. Under this ordering, we impose the following very natural properties:

1. given two submissions with the same observed upvote ratio, we will assign the submission with more votes as better (since we are more confident it has a higher ratio).
2. given two submissions with the same number of votes, we still assign the submission with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every submission takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```python
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print("Approximate lower bounds:")
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print(lb)
print("\n")
print("Top 40 Sorted according to approximate lower bounds:")
print("\n")
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print(votes[i,0], votes[i,1], contents[i])
    print("-------------")
```

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```python
r_order = order[::-1][-40:]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order], capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30].replace("\n",""), ordered_contents) );
```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Extension to Starred rating systems

The above procedure works well for upvote-downvotes schemes, but what about systems that use star ratings, e.g. 5 star rating systems. Similar problems apply with simply taking the average: an item with two perfect ratings would beat an item with thousands of perfect ratings, but a single sub-perfect rating.

We can consider the upvote-downvote problem above as binary: 0 is a downvote, 1 if an upvote. A $N$-star rating system can be seen as a more continuous version of above, and we can set $n$ stars rewarded is equivalent to rewarding $\frac{n}{N}$. For example, in a 5-star system, a 2 star rating corresponds to 0.4. A perfect rating is a 1. We can use the same formula as before, but with $a,b$ defined differently:

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where

\begin{align}
& a = 1 + S \\\\
& b = 1 + N - S \\\\
\end{align}

where $N$ is the number of users who rated, and $S$ is the sum of all the ratings, under the equivalence scheme mentioned above.

##### Example: Counting Github stars

What is the average number of stars a Github repository has? How would you calculate this? There are over 6 million respositories, so there is more than enough data to invoke the Law of Large numbers. Let's start pulling some data. TODO

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting submissions formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead use a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally accurate?

```python
## Enter code here
import scipy.stats as stats
exp = stats.expon( scale=4 )
N = 1e5
X = exp.rvs( int(N) )
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by their percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

In August 2013, [a popular post](http://bpodgursky.wordpress.com/2013/08/21/average-income-per-programming-language/) on the average income per programmer of different languages was trending. Here's the summary chart: (reproduced without permission, cause when you lie with stats, you gunna get the hammer). What do you notice about the extremes?

------

#### Average household income by programming language

<table >
 <tr><td>Language</td><td>Average Household Income ($)</td><td>Data Points</td></tr>
 <tr><td>Puppet</td><td>87,589.29</td><td>112</td></tr>
 <tr><td>Haskell</td><td>89,973.82</td><td>191</td></tr>
 <tr><td>PHP</td><td>94,031.19</td><td>978</td></tr>
 <tr><td>CoffeeScript</td><td>94,890.80</td><td>435</td></tr>
 <tr><td>VimL</td><td>94,967.11</td><td>532</td></tr>
 <tr><td>Shell</td><td>96,930.54</td><td>979</td></tr>
 <tr><td>Lua</td><td>96,930.69</td><td>101</td></tr>
 <tr><td>Erlang</td><td>97,306.55</td><td>168</td></tr>
 <tr><td>Clojure</td><td>97,500.00</td><td>269</td></tr>
 <tr><td>Python</td><td>97,578.87</td><td>2314</td></tr>
 <tr><td>JavaScript</td><td>97,598.75</td><td>3443</td></tr>
 <tr><td>Emacs Lisp</td><td>97,774.65</td><td>355</td></tr>
 <tr><td>C#</td><td>97,823.31</td><td>665</td></tr>
 <tr><td>Ruby</td><td>98,238.74</td><td>3242</td></tr>
 <tr><td>C++</td><td>99,147.93</td><td>845</td></tr>
 <tr><td>CSS</td><td>99,881.40</td><td>527</td></tr>
 <tr><td>Perl</td><td>100,295.45</td><td>990</td></tr>
 <tr><td>C</td><td>100,766.51</td><td>2120</td></tr>
 <tr><td>Go</td><td>101,158.01</td><td>231</td></tr>
 <tr><td>Scala</td><td>101,460.91</td><td>243</td></tr>
 <tr><td>ColdFusion</td><td>101,536.70</td><td>109</td></tr>
 <tr><td>Objective-C</td><td>101,801.60</td><td>562</td></tr>
 <tr><td>Groovy</td><td>102,650.86</td><td>116</td></tr>
 <tr><td>Java</td><td>103,179.39</td><td>1402</td></tr>
 <tr><td>XSLT</td><td>106,199.19</td><td>123</td></tr>
 <tr><td>ActionScript</td><td>108,119.47</td><td>113</td></tr>
</table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. [Web](http://www.sloansportsconference.com/wp-content/uploads/2013/Going%20for%20Three%20Predicting%20the%20Likelihood%20of%20Field%20Goal%20Success%20with%20Logistic%20Regression.pdf). 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```python
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    img{
        max-width:800px}
</style>


# Computational Astrophysics
## Root Finding 01

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the techniques to find the roots of a function.

---

It is often needed to find the root of a function, i.e., the values of a variable
$x$ for which $f(x) = 0$. In general, $x$ could be a scalar or a vector quantitiy and $f$ could be a single function or could represent a system of equations. It is also possible that $f$ explicitly depends on $x$ or that $f$ depends implicitly on $x$.

---
## Newton-Raphson Method

There are a variety of ways to find the roots of an equation, but the mos popular is the **Newton-Raphson** method. To describe this method, consider the expansion of a function $f(x)$ in a Taylor's series about a root $x_r$,

\begin{equation}
f(x_r) = f(x) + (x_r - x) f'(x) +  \mathcal{O}( (x_r - x)^2) = 0 .
\end{equation}

The value $x_r$ can be seen as the value for the root at the $n$-th step of an iterative procedure. Hence, The $n+1$-th step in this iteration, up to first order, will be given by

\begin{equation}
f(x_{n+1}) = f(x_n) + \underbrace{(x_{n+1} - x_{n})}_{\delta x}
f'(x_n) \approx 0 ,
\end{equation}

from which we obtain

\begin{equation}
x_{n+1} =  x_n - \frac{f(x_n)}{f'(x_n)} .
\end{equation}

The implementation of this iterative procedure will stop when the
fractional change between iteration $n$ and $n+1$ is smaller than the floating point accuracy (machine's epsilon),

\begin{equation}
\left| \frac{[f(x_{n+1}) - f(x_{n})]}{f({x_n})} \right| < \epsilon .
\end{equation}

The convergence of the Newton-Raphson Method is second order if $f(x)$ is well
behaved and if the initial value $x_0$ is a good guess for the root.

Important Note: This method needs to know the analytical form of the first derivative $f'(x)$. If this is not the case, we will need to evaluate the derivative numerically, which gives the so-called **secant method** which is described below.

---
### Secant Method

As stated above, the secant method is similar to the Newton-Raphson method, but the first derivative $f'(x_n)$ is evaluated numerically. Usually, it is done with a backward difference, giving  the iterative procedure

\begin{equation}
x_{n+1} = x_{n} - f(x_n)\frac{x_n - x_{n-1}}{f(x_n) - f(x_{n-1})}\,\,.
\end{equation}

Note that this iterative procedure needs two points to start, or  to have an initial guess for the first derivative, i.e. $f'(x_0)$.  Due to the inclusion of the numerical procedure to find the first derivative, it is clear that the secant method will converge less rapidly than the Newton-Raphson method.

#### Example. Roots of a point-sampled function using Newton-Raphson ( secant ) Method

We will read a data set from a .txt file and numerically find the root of the function using the Newton-Raphson (or secant) method. We will use a piecewise quadratic interpolation to model the function.

We load the data and, before applying  the root finding method, we note that the root can be approximately localized graphically.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x_data = data[:,0]
f_data = data[:,1]

plt.figure(figsize=(7,5))
plt.plot(x_data, f_data, color='red')
plt.title('')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.grid()
plt.show()
```

 The root is somewhere between $x=0$ and $x=2$. Thus, we can restrict our attention to this interval:

```python
plt.figure(figsize=(7,5))
plt.plot(x_data, f_data, color='red')
plt.scatter(x_data, f_data, color='black')
plt.title('')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.xlim(0,2)
plt.ylim(-10,10)
plt.grid()
plt.show()
```

This zoom of the plot shows that the root is inside the interval $1<x<1.75$. Therefore, we will search for three points in this interval in order to interpolate a quadratic polynomial piecewise and apply the root finding method.

```python
# Find the initial and final points of the restricted interval of
# three points to interpolate.
for i in range(len(x_data)-1):
    if x_data[i]>=1:
        a=i
        break

# The interval will include three points:
# x[a], x[a+1], x[a+2]
# Here we define the arrays that enter as arguments
# for the interpolated polynomial
x_int =[x_data[a], x_data[a+1], x_data[a+2]]
f_int =[f_data[a], f_data[a+1], f_data[a+2]]

print('a = ', a, ': x_a = ', x_data[a], ': f(x_a) = ', f_data[a])

```

    a =  30 : x_a =  1.12245 : f(x_a) =  2.44474

Now, we will use a quadratic polynomial interpolated piecewise in this restricted interval to find the root using the Newton-Raphson (or secant) method.

```python
# 2-order Interpolated Polynomial
def p2(x):
    pol = (((x-x_int[1])*(x-x_int[2]))/((x_int[0]-x_int[1])*(x_int[0]-x_int[2])))*f_int[0] +\
          (((x-x_int[0])*(x-x_int[2]))/((x_int[1]-x_int[0])*(x_int[1]-x_int[2])))*f_int[1] +\
          (((x-x_int[0])*(x-x_int[1]))/((x_int[2]-x_int[0])*(x_int[2]-x_int[1])))*f_int[2]
    return pol

# Newton-Raphson Method. We begin with the value at point x[a]
root0 = x_data[a]-1e-5
root = x_data[a]

while np.abs(p2(root))>1e-10:
    new_root = root - p2(root)*(root - root0)/(p2(root) - p2(root0))
    root0, root = root, new_root

print(f'The root is x_r = {root:.8f}')
print(f'for which the interpolated function gives f(x_r) =  {p2(root):.8e}')

```

    The root is x_r = 1.32429525
    for which the interpolated function gives f(x_r) =  7.09501902e-16

We can also visualise the location of the root:

```python
plt.figure(figsize=(7,5))
plt.plot(x_data, f_data, color='red')
plt.scatter(x_data, f_data, color='black')
plt.scatter(root, p2(root), color='blue', marker='*',s=150)
plt.vlines(root, -7, p2(root), linestyle='dashed')
plt.hlines(p2(root),0, root, linestyle='dashed')
plt.title('Root')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.xlim(0.5,2)
plt.ylim(-7,7)
plt.grid()
plt.show()
```

---
## Bisection Method

The bisection method is based on the intermediate-value theorem, which states that a continuous function $f(x)$ has at least one root in the interval $[x_a,x_b]$ if $f(x_a)$ and
$f(x_b)$ have opposite signs.

Hence, the bisection method is implemented as

1. Pick the initial values of $x_a$ and $x_b$ so that $f(x_a)$ and $f(x_b)$ have opposite signs.

2. Compute the midpoint $x_c = \frac{x_a+x_b}{2}$ and $f(x_c)$.

3. If $f(x_c) = 0$ or
  $\left| \frac{f(x_c) - f(x_a)}{f(x_a)} \right| < \epsilon$ or $\left| \frac{f(x_c) - f(x_b)}{f(x_b)} \right| <
  \epsilon$, then he root has been found.

  If not:
    * If $f(x_a)$ and $f(x_c)$ have opposite signs, then they
      bracket a root. Go to (2) with $x_a=x_a$, $x_b=x_c$.

    * If $f(x_c)$ and $f(x_b)$ have opposite signs, then they
      bracket a root. Got to (2) with $x_a=x_c$, $x_b=x_b$.

Although the bisection method is very effective and robust, it is generally not as fast as Newton's Method because it requires more iterations to find the root.

#### Example. Roots of a point-sampled function using the Bisection Method

We will read a data set from a .txt file and numerically find the root of the function using the bisection method. We will use a piecewise quadratic interpolation to model the function.

We load the data and, before applying  the root finding method, we note that the root can be approximately localized graphically.

```python
import numpy as np
import matplotlib.pyplot as plt

# Reading the data
data = np.loadtxt('data_points.txt', comments='#', delimiter=',')
x_data = data[:,0]
f_data = data[:,1]

plt.figure(figsize=(7,5))
plt.plot(x_data, f_data, color='red')
plt.title('')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.grid()
plt.show()
```

 The root is somewhere between $x=0$ and $x=2$. Thus, we can restrict our attention to this interval:

```python
plt.figure(figsize=(7,5))
plt.plot(x_data, f_data, color='red')
plt.scatter(x_data, f_data, color='black')
plt.title('')
plt.xlabel(r'$x$')
plt.ylabel(r'$f(x)$')
plt.xlim(0,2)
plt.ylim(-10,10)
plt.grid()
plt.show()
```

This zoom of the plot shows that the root is inside the interval $1<x<1.75$. Therefore, we will search for three points in this interval in order to interpolate a quadratic polynomial piecewise and apply the root finding method.

```python
# Find the initial and final points of the restricted interval of
# three pints to interpolate.
for i in range(len(x_data)-1):
    if x_data[i]>=1:
        a=i
        break

# The interval will include three points:
# x[a], x[a+1], x[a+2]
# Here we define the arrays that enter as arguments
# for the interpolated polynomial
x_int =[x_data[a], x_data[a+1], x_data[a+2]]
f_int =[f_data[a], f_data[a+1], f_data[a+2]]

print('a = ', a, ': x_a = ', x_data[a], ': f(x_a) = ', f_data[a])

```

    a =  30 : x_a =  1.12245 : f(x_a) =  2.44474

Now, we will use a quadratic polynomial interpolated piecewise in this restricted interval to find the root using the Newton-Raphson (or secant) method.

```python
# 2-order Interpolated Polynomial
def p2(x):
    pol = (((x-x_int[1])*(x-x_int[2]))/((x_int[0]-x_int[1])*(x_int[0]-x_int[2])))*f_int[0] +\
          (((x-x_int[0])*(x-x_int[2]))/((x_int[1]-x_int[0])*(x_int[1]-x_int[2])))*f_int[1] +\
          (((x-x_int[0])*(x-x_int[1]))/((x_int[2]-x_int[0])*(x_int[2]-x_int[1])))*f_int[2]
    return pol

# Bisection Method.
# We begin with the values at the points x_data[a] and x_data[a+2]
x_a = x_data[a]
x_b = x_data[a+2]
x_c = (x_a + x_b)/2.

while np.abs(p2(x_c)) > 1e-10:
    x_c = (x_a + x_b)/2.
    if p2(x_a)*p2(x_c) < 0:
        x_b = x_c
    else:
        x_a = x_c

root = x_c

print(f'The root is x_r = {root:.8f}')
print(f'for which the interpolated function gives f(x_r) =  {p2(root):.8e}')

```

    The root is x_r = 1.32429525
    for which the interpolated function gives f(x_r) =  -2.53866234e-11

```python

```


# Exercise 2
* Due: 21.11. at noon
* Max points: 8

## General rules
* Send the source code or the written notes of your answers as an attachment to markus.hartikainen@jyu.fi before the due time
* You will get feedback about your answers a week from the due date
* From the second week on, the exercises will be given on the previous Wednesday's lecture

## Exercises

1. **(2 points)** Assume that that you are buying and selling electricity. The more electricity you buy, the better price you get per kwH and the buying prize of electricity follows function $f(x) = 1-0.01x$, where $0\leq x\leq50$ is the amount of electricity you buy. On the other hand, the price that you get from selling electricity follows function $g(x) = 2-0.01x^2$ with $x$ again the amount of electricity that you sell. Formulate an optimization problem that maximizes the profit.
2. **(2 points)** Use bisection search to optimize problem
$$
\begin{align}
\min & (1-x)^2+x\\
\text{s.t. }&x\in [0,2].
\end{align}
$$
3. **(2 points)** Use golden section search to optimize the problem from above exercise.
4. **(2 points)** Use differentiation to optimize the optimization problem of exercise 1 and to verify the answer of optimization problem 2.


```python
from sympy import *
```

Consider Fibonacci numbers $F_n$: $F_0 = 0, F_1 = 1, F_{n+2} = F_n + F_{n+1}$.

```python
def F(n):
    if n == 0:
        return 0
    if n <= 2:
        return 1
    return F(n-1) + F(n-2)

[F(n) for n in range(11)]
```

    [0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55]

Denote $\Phi_{n+1} := \begin{bmatrix} F_{n+1} \\ F_n \end{bmatrix}$

Then $\Phi_{n+2} = \begin{bmatrix} F_{n+2} \\ F_{n+1} \end{bmatrix} = \begin{bmatrix} F_{n+1} + F_n \\ F_{n+1} \end{bmatrix} = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix} \cdot \begin{bmatrix} F_{n+1} \\ F_n \end{bmatrix} =\begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix} \cdot \Phi_{n+1}$

So $\Phi_{n+1} = \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix} \cdot \Phi_n$

Denote $A := \begin{bmatrix} 1 & 1 \\ 1 & 0 \end{bmatrix}$

Then $\Phi_{n+1} = A \cdot \Phi_n = A^n \cdot \Phi_1 = A^n \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix}$

Notice, that $A^n$ has form $\begin{bmatrix} p + q & q \\ q & p \end{bmatrix}$ for all natural $n$

Let's check it by induction

```python
p = IndexedBase('p')
q = IndexedBase('q')
m, n = symbols('m, n')
```

```python
def A(n):
    return Matrix([[p[n] + q[n], q[n]], [q[n], p[n]]])
```

```python
A(n)
```

$\displaystyle \left[\begin{matrix}{p}_{n} + {q}_{n} & {q}_{n}\\{q}_{n} & {p}_{n}\end{matrix}\right]$

Base case:

```python
A1 = A(1).subs({p[1]: 0, q[1]: 1}); A1
```

$\displaystyle \left[\begin{matrix}1 & 1\\1 & 0\end{matrix}\right]$

Induction step:

If $A^n = \begin{bmatrix} p_i + q_i & q_i \\ q_i & p_i \end{bmatrix}$ for all natural $i \leqslant n$ \
then $A^n \cdot A = A^{n+1}$ with substitution $p_{n+1} = q_n, q_{n+1} = p_n + q_n$

```python
A(n) * A1
```

$\displaystyle \left[\begin{matrix}{p}_{n} + 2 {q}_{n} & {p}_{n} + {q}_{n}\\{p}_{n} + {q}_{n} & {q}_{n}\end{matrix}\right]$

```python
A(n+1).subs({p[n+1]: q[n], q[n+1]: p[n] + q[n]})
```

$\displaystyle \left[\begin{matrix}{p}_{n} + 2 {q}_{n} & {p}_{n} + {q}_{n}\\{p}_{n} + {q}_{n} & {q}_{n}\end{matrix}\right]$

Now we can get $\Phi_{n+1} = \begin{bmatrix} F_{n+1} \\ F_n \end{bmatrix} = A^n \cdot \Phi_1 = A^n \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix} = \begin{bmatrix} p_n + q_n & q_n \\ q_n & p_n \end{bmatrix} \cdot \begin{bmatrix} 1 \\ 0 \end{bmatrix}$

So $F_n = q_n = p_{n+1}, \quad F_{n+1} = p_n + q_n = q_{n+1}$

Let's compute $A^{m+n}$ by coefficients of $A^m$ and $A^n$

```python
Amn = A(n) * A(m); Amn
```

$\displaystyle \left[\begin{matrix}\left({p}_{m} + {q}_{m}\right) \left({p}_{n} + {q}_{n}\right) + {q}_{m} {q}_{n} & \left({p}_{n} + {q}_{n}\right) {q}_{m} + {p}_{m} {q}_{n}\\\left({p}_{m} + {q}_{m}\right) {q}_{n} + {p}_{n} {q}_{m} & {p}_{m} {p}_{n} + {q}_{m} {q}_{n}\end{matrix}\right]$

```python
p_mn = Amn[1, 1]; p_mn
```

$\displaystyle {p}_{m} {p}_{n} + {q}_{m} {q}_{n}$

```python
q_mn = Amn[0, 1]; q_mn
```

$\displaystyle \left({p}_{n} + {q}_{n}\right) {q}_{m} + {p}_{m} {q}_{n}$

Now we can use these formulas to fast compute of Fibonacci numbers (see **pmtests.lesson1.Fibonacci.fast()** method):

$\begin{cases}
p_{m+n} = p_m p_n + q_m q_n \\
q_{m+n} = \left( p_n + q_n \right) q_m + p_m q_n
\end{cases}$

Let's check its

```python
pn = [0] # pn[i] = p_mn[i+1]
qn = [1] # qn[i] = q_mn[i+1]

for i in range(1, 10):
    pn.append(p_mn.subs({m: i, n: 1}))
    qn.append(q_mn.subs({m: i, n: 1}))
    for j in range(i):
        pn[i] = pn[i].subs({p[j+1]: pn[j], q[j+1]: qn[j]})
        qn[i] = qn[i].subs({p[j+1]: pn[j], q[j+1]: qn[j]})

pn, qn
```

    ([0, 1, 1, 2, 3, 5, 8, 13, 21, 34], [1, 1, 2, 3, 5, 8, 13, 21, 34, 55])

```python
display(A1**2)
Amn.subs({m: 1, n: 1, p[1]: pn[0], q[1]: qn[0]})
```

$\displaystyle \left[\begin{matrix}2 & 1\\1 & 1\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}2 & 1\\1 & 1\end{matrix}\right]$

```python
display(A1**3)
Amn.subs({m: 2, n: 1,
          p[1]: pn[0], q[1]: qn[0],
          p[2]: pn[1], q[2]: qn[1]})
```

$\displaystyle \left[\begin{matrix}3 & 2\\2 & 1\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}3 & 2\\2 & 1\end{matrix}\right]$

```python
display(A1**4)
Amn.subs({m: 2, n: 2,
          p[1]: pn[0], q[1]: qn[0],
          p[2]: pn[1], q[2]: qn[1]})
```

$\displaystyle \left[\begin{matrix}5 & 3\\3 & 2\end{matrix}\right]$

$\displaystyle \left[\begin{matrix}5 & 3\\3 & 2\end{matrix}\right]$


Construir o gráfico e encontrar o foco e uma equação da diretriz.

9. $y = \frac{x^2}{16}$

Arrumando a equação

$x^2 = 16y$

$2p = 16$, logo

$p =  8$

Calculando o foco

$F = \frac{p}{2}$

$F = \frac{8}{2}$

$F =  4$

$F(0,4)$

Calculando a diretriz

$D = -\frac{p}{2}$

$D : y = -4$

$V(0,0)$

$F(0,4)$

Gráfico da parábola

```python
from sympy import *
from sympy.plotting import plot_implicit
x, y = symbols("x y")
plot_implicit(Eq((x-0)**2, 16*(y+0)), (x,-10,10), (y,-10,10),
title=u'Gráfico da parábola', xlabel='x', ylabel='y');
```

Problemas Propostos. Pag. 172 - 175/09.ipynb


# Logistic Regression with Fast Gradient Descent

### Author: Juan Solorio

-----

# Overview
In this exercise, I will implement a first version of my own own fast gradient algorithm to solve the $l^2_2$-regularized logistic regression problem. I will be building on the stocastic gradient descent algorithm and adding a backtracking function for the gradient descent algorithm.

## Objectives
- Mathematically define _Objective Function_ for Logistic Regression ($F(\beta)$)
    - Compute gradiant $\nabla F$
- Create functions for algorithm:
    - Objective function
    - Gradient function
    - Fast Gradient Descent funtion & gradient descent function and compare
    - Backtracking Line Search function
    - Check misclassification error

- Observe (plot) the convergence in terms of the function values and the gradients
    and tune for the optimal hyperparameters for step-size ($\eta$) and normalization ($\lambda$)
- Compare to _sklearn_

## Environment Setup - *Importing Libraries*

```python
# needed libraries
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
from sklearn.model_selection import train_test_split
from sklearn import preprocessing
from sklearn.linear_model import LogisticRegression
import time

%matplotlib inline
plt.style.use('ggplot')
```

# Background and Theory

***logistic Regression*** is a type of _Supervised Machine Learning_ algorithm with the purpose to be used in ***binary classification***, as such this algorithm is confine to make predictions between {0,1} or {-1,1}. Given that we are trying to make this binary prediction of our target variable, we are dealing with a model trying to get variables where the probabilities are bounded on both ends (they must be between 0 and 1).

We start with the traditional definition of the _Linear Regression_ formula:
$$
y = b_0*x + b_1
$$
> y - target value
$b_0$ - slope
x - predictor variable
$b_1$ - intercept

However, given linear regression gives _unbounded_ solutions, we must _restrict_ the function by wrapping it around or passing it through a **link function**. We start by taking our linear function and defining it in terms of the statistical ***Odds*** of the given event occuring:
> $P(Y=1 | X=x)$, read as the probability of $Y=1$ given that $X$ is already value $x$.
$Odds=\frac{P}{1-P}$, the _Odds_ of something happening is the probability of it _happening_ devided by it _not happening_ .

We then transform our linear model and Odds of occurance into our _logistic regression_ :
$$
Ln(\frac{P}{1-P}) = b_0*x + b_1 \\
\Rightarrow \frac{P}{1-P} = e^{b_0*x + b_1} \\
\Rightarrow P = e^{b_0*x + b_1} * (1-P) \\
\Rightarrow e^{b_0*x + b_1} = P + P*e^{b_0*x + b_1}  \\
\Rightarrow P = \frac{e^{b_0*x + b_1}}{1+e^{b_0*x + b_1}} \\
$$

As we can observe this final result is what is known as passing our linear regression through the ***Sigmoid function***:
>a mathematical function having a characteristic "S"-shaped curve or **sigmoid curve**
$$
S(x) = \frac{1}{1+e^{-x}}
$$

## Logistic Regression
We can generalize the *Logistic Regression* ***objective function*** as:
$$
F(\beta) = \min_{\beta}F(\beta) = \frac{1}{n}\sum_{i=1}^n log(1 + exp^{-y_ix^T_i\beta}) + \lambda\|\beta\|^2_2
$$

or equivalently

\begin{equation}
\min_{\beta}F(\beta) = \frac{1}{n}\sum_{i=1}^n log(1 + exp^{-y_i\sum^d_{j=1}x^T_{i,j}\beta_j}) + \lambda\sum_{j=1}^d\beta_j^2
\end{equation}

Using the machine learning convention for the labels that is $y_i\ \in$ {−1, +1}

Assuming $d=1$ and $n=1$, we can compute the $\nabla F$ as follows:

\begin{equation}
F(\beta) = log(1 + exp^{-yx\beta}) + \lambda\beta^2
\end{equation}
\begin{equation}
\Rightarrow \frac{\partial F}{\partial \beta} = \frac{(-yx)exp^{-yxb}}{(1 + exp^{-yx\beta})} + 2\lambda\beta
\end{equation}
\begin{equation}
\Rightarrow \nabla F(\beta) = 2\lambda \beta - \frac{yx}{(1 + exp^{-yx\beta})* exp^{yx\beta}}
\end{equation}
\begin{equation}
\Rightarrow \nabla F(\beta) = 2\lambda \beta - \frac{yx}{exp^{yx\beta} + 1}
\end{equation}

We then generalize for $d>1$ and $n>1$ as:
$$
\nabla F(\beta) = -\frac{1}{n}\sum_{i=1}^n \frac{yx}{exp^{yx\beta} + 1} + 2\lambda \beta
$$

# Algorithm Function Definition

* Objective function for _Logistic Regression_ $F(\beta)$:
>$$
F(\beta) = \min_{\beta}F(\beta) = \frac{1}{n}\sum_{i=1}^n log(1 + exp^{-y_ix^T_i\beta}) + \lambda\|\beta\|^2_2
$$

```python
def obj_fx(beta,lamda,X,y):
    """
    Linear regression with Ridge penalty L1:
    F(b) = 1/n sum(log(1 + exp(-y*x dotproduct b))) + lamda*norm(b)^2

    Parameters
    ----------
    beta : arr
        array of values for weights
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data

    Returns
    -------
    int
        computation of the logistic regression objective function

    """

    # dot product can be accomplish by 'numpy_arrayA @ numpy_arrayB' or 'numpy_arrayA.dot(numpy_arrayB)'
    return 1/len(y) * np.sum(np.log(1 + np.exp(-y*X.dot(beta)))) + lamda * np.linalg.norm(beta)**2

def obj_fx_alt(b, lamda,X,y):
    h = X @ b * y
    return (1/len(y)) * sum(np.log(1 + np.exp(-h))) + (lamda * np.linalg.norm(b)**2)
```

* Gradient of the logistic regression objective function - $\nabla F$
> $$\nabla F(\beta) = -\frac{1}{n}\sum_{i=1}^n \frac{yx}{exp^{yx\beta} + 1} + 2\lambda \beta$$

```python
def gradient_fx(beta,lamda,X,y):
    """
    Computes gradient of the Linear regression with Ridge penalty L1 function:
    grad F(b) = -1/n sum(yx / (exp(y*x dotproduct b)+1)) + 2*lamda*b

    Parameters
    ----------
    beta : arr
        array of values for weights
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data

    Returns
    -------
    int
        computation of the gradient of Logistic regression objective function

    """
    yx = y[:, np.newaxis]*X
    denom = 1+np.exp(-yx.dot(beta))
    grad = 1/len(y)*np.sum(-yx*np.exp(-yx.dot(beta[:, np.newaxis]))/denom[:, np.newaxis], axis=0) + 2*lamda*beta
    return grad

def gradient_fx_alt(b, lamda,X,y):
    h = X @ b * y
    return 2*lamda*b - (1/len(y) * (X.T @ (y / (np.exp(h) + 1))))
```

* Backtracking algorithm
> **initialize** $\eta_t = \eta_{t-1}$ if $t \geq 1$
$\eta_0 = \frac{1}{L_{est}}$
**Iterate** $\eta_t = \gamma \eta_t$
_Until condition met_

```python
def backtracking(beta, lamda, X, y, eta=1, alpha=0.5, gamma=0.8, maxiter=100):
    """
    Computes backtracking linear search algorithm:
    F(B_t − η∇F(B_t)) ≤ F(B_t) + αη||∇F(B_t)||^2

    Parameters
    ----------
    beta : arr
        array of values for weights
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data
    eta: int
        integer value for step size parameter
    alpha: int
        integer value for stepsize change
    gamma: int
        integer value

    Returns
    -------
    int
        computation of the change in eta

    """

    grad_beta = gradient_fx(beta, lamda, X, y)
    norm_grad_beta = np.linalg.norm(grad_beta)
    found_eta = 0
    num_iters = 0
    while found_eta == 0 and num_iters < maxiter:
        if obj_fx(beta - eta * grad_beta, lamda, X,y) < obj_fx(beta, lamda, X, y) - alpha * eta * norm_grad_beta ** 2:
            found_eta = 1
        elif num_iters == maxiter:
            raise ('Max number of iterations of backtracking line search reached')
        else:
            eta *= betaparam
            num_iters += 1
    return eta

```

* Gradient Descent Algorith:
>`Gradient Descent algorithm with fixed constant step-size
__input__  step-size $\eta$
__initialization__ $\beta_0 = 0$
__repeat for__ t = 0, 1, 2, . . .
    $\beta_{t+1} = \beta_t − \eta \nabla F(\beta_t)$
__until__ the stopping criterion $\|\nabla F\| \leq \epsilon$ is satisfied.

```python
def gradient_descent(beta_init, eta_init ,lamda,X,y,epsilon=0.005):
    """
    Computes gradient descent with a fixed step size eta and stopping condition norm gradient F < epislon

    Parameters
    ----------
    beta_init : arr
        array of values for weights as starting point
    eta_init : int
        interger value for step size parameter
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data
    epsilon : int
        interger value for stopping parameter condition, defaul = 0.005

    Returns
    -------
     beta_vals: Matrix
         Estimated betas at each iteration, with the most recent values in the last row
    """
    # setting initial value of betas
    beta = beta_init
    eta = eta_init
    # gradient calculation for starting values
    gradient = gradient_fx(beta,lamda,X,y)
    # list to save beta values
    beta_vals = [beta_init]
    # loop for stopping criterion epsilon
    while np.linalg.norm(gradient) > epsilon:
        # updating values for eta, beta and gradient
        eta = backtracking_1(beta, lamda, eta=eta, X=X, y=y)
        beta = beta - eta*gradient
        gradient = gradient_fx(beta,lamda,X,y)

        # appending values
        beta_vals.append(beta)
    return np.array(beta_vals)

```

* Fast Gradient Descent Algorithm:
> **input** step-size $\eta_0$, target accuracy $\epsilon$
**initialization** $\beta_0 = 0$, $\theta_0 = 0$
**repeat** for $t = 0, 1, 2, . . .$
**Find** $\eta_t$ with backtracking rule
    $\beta_{t+1} = \theta_t - \eta_t \nabla F(\eta_t)$
    $\theta_{t+1} = \beta_{t+1} + \frac{t}{t+3}(\beta_{t+1} - \beta_t)$
**until** the stopping criterion $||\nabla F|| \leq \epsilon$.

```python
def fast_gradient_algo(beta_init, theta_init, eta_init,lamda, X,y,epsilon=0.005):
    """
    Computes Fast gradient descent with a initial conditions beta_init, theta_init, fixed step size eta
    and stopping condition norm gradient F < epislon

    Parameters
    ----------
    beta_init : arr
        array of values for weights as starting point
    eta_init : int
        interger value for step size parameter
    theta_init : int
        interger value for updating beta parameter
    lamda : int
        interger value for normalization parameter
    X : arr
        array of features from data
    y : arr
        array of target values from data
    epsilon : int
        interger value for stopping parameter condition, defaul = 0.005

    Returns
    -------
     beta_vals: Matrix
         Estimated betas at each iteration, with the most recent values in the last row
    """
    # setting initial value of betas
    beta = beta_init
    eta = eta_init
    theta = theta_init
    # gradient calculation for starting values
    grad = gradient_fx(theta,lamda,X,y)
    # list to save beta values
    beta_vals = [beta_init]
    num_iters = 0
    # loop for stopping criterion epsilon
    while np.linalg.norm(grad) > epsilon:
        # updating values for eta, beta and gradient
        eta = backtracking_1(beta, lamda, eta=eta, X=X, y=y)
        beta_new = theta - eta*grad
        theta = beta_new + num_iters/(num_iters+3)*(beta_new-beta)

        # appending values
        beta = beta_new
        beta_vals.append(beta)

        grad = gradient_fx(theta,lamda,X,y)

        num_iters += 1

    return np.array(beta_vals)

```

# Implementation

For testing our _logistic Regression_ algorithm, we'll be using the ___Spam___ dataset from `'https://web.stanford.edu/~hastie/ElemStatLearn/datasets/spam.data'` for out features and `'https://web.stanford.edu/~hastie/ElemStatLearn/datasets/spam.traintest'` for our target variable, and we'll drop the samples with `NA` values.

```python
# Load the data
spam = pd.read_table("https://web.stanford.edu/~hastie/ElemStatLearn/datasets/spam.data",sep=" ", header=None)
spam['indicator'] = pd.read_table("https://web.stanford.edu/~hastie/ElemStatLearn/datasets/spam.traintest",header=None)
spam = spam.dropna()

spam.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      49
      50
      51
      52
      53
      54
      55
      56
      57
      indicator

      0
      0.00
      0.64
      0.64
      0.0
      0.32
      0.00
      0.00
      0.00
      0.00
      0.00
      ...
      0.000
      0.0
      0.778
      0.000
      0.000
      3.756
      61
      278
      1
      1

      1
      0.21
      0.28
      0.50
      0.0
      0.14
      0.28
      0.21
      0.07
      0.00
      0.94
      ...
      0.132
      0.0
      0.372
      0.180
      0.048
      5.114
      101
      1028
      1
      0

      2
      0.06
      0.00
      0.71
      0.0
      1.23
      0.19
      0.19
      0.12
      0.64
      0.25
      ...
      0.143
      0.0
      0.276
      0.184
      0.010
      9.821
      485
      2259
      1
      1

      3
      0.00
      0.00
      0.00
      0.0
      0.63
      0.00
      0.31
      0.63
      0.31
      0.63
      ...
      0.137
      0.0
      0.137
      0.000
      0.000
      3.537
      40
      191
      1
      0

      4
      0.00
      0.00
      0.00
      0.0
      0.63
      0.00
      0.31
      0.63
      0.31
      0.63
      ...
      0.135
      0.0
      0.135
      0.000
      0.000
      3.537
      40
      191
      1
      0

5 rows × 59 columns

```python

```

```python
# Create our X matrix with the predictors and y vector with the response
X = spam.drop("indicator", axis=1)
y = spam.indicator
y.replace(0,-1,inplace=True)
# Divide the data into training and test sets.
X_train, X_test, y_train, y_test = train_test_split(X, y,random_state=0)
# Standardize the data
scaler = preprocessing.StandardScaler().fit(X_train)
X_train = scaler.transform(X_train)
X_test = scaler.transform(X_test)
scaler = preprocessing.StandardScaler().fit(y_train.values.reshape(-1, 1))
# Keep track of the number of samples and dimension of each sample
n_train = len(y_train)
n_test = len(y_test)

```

```python

```

```python
# Variable to load into functions
lamda = 0.5
L = (np.max(np.linalg.eig(1/len(y_train)*X_train.T@X_train)[0])) + lamda
eta_init = 1/L
d = X_train.shape[1]
beta_init = np.zeros(d)
theta_init = np.zeros(d)
maxiter = 1000

```

```python
# store starting time
begin = time.time()
print(gradient_fx(beta=beta_init,lamda=lamda,X=X_train,y=y_train))
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")

# store starting time
begin = time.time()
print(computegrad(b=beta_init, lamda=lamda,X=X_train,y=y_train))
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")

```

    [ 0.02032379  0.0027978  -0.00584371  0.00202893 -0.00126049  0.00671368
     -0.00065947  0.00121773 -0.01000378  0.00207672 -0.00820628 -0.00858195
     -0.00094936 -0.01933799  0.0020736  -0.01503039 -0.00170232  0.00161148
      0.00071245 -0.01061828  0.00350524  0.00459303  0.00163439 -0.00363655
     -0.00080175 -0.00234983  0.00323827 -0.00892796  0.01102586 -0.00145684
     -0.00163501  0.00535138 -0.00677281  0.005918   -0.00181521  0.00446524
      0.00861823  0.00627413  0.00205909  0.00170645  0.00752408  0.00109195
      0.00468198 -0.00098964  0.00613565 -0.0003137  -0.00225585 -0.00293253
     -0.00110998 -0.00093037  0.01690183  0.00284367  0.00756908  0.01537267
      0.00131323  0.00384906  0.00171852  0.00559917]
    Total runtime of the program is 1.0137851238250732
    [ 0.02032379  0.0027978  -0.00584371  0.00202893 -0.00126049  0.00671368
     -0.00065947  0.00121773 -0.01000378  0.00207672 -0.00820628 -0.00858195
     -0.00094936 -0.01933799  0.0020736  -0.01503039 -0.00170232  0.00161148
      0.00071245 -0.01061828  0.00350524  0.00459303  0.00163439 -0.00363655
     -0.00080175 -0.00234983  0.00323827 -0.00892796  0.01102586 -0.00145684
     -0.00163501  0.00535138 -0.00677281  0.005918   -0.00181521  0.00446524
      0.00861823  0.00627413  0.00205909  0.00170645  0.00752408  0.00109195
      0.00468198 -0.00098964  0.00613565 -0.0003137  -0.00225585 -0.00293253
     -0.00110998 -0.00093037  0.01690183  0.00284367  0.00756908  0.01537267
      0.00131323  0.00384906  0.00171852  0.00559917]
    Total runtime of the program is 1.010300874710083

```python
# store starting time
begin = time.time()
print(obj_fx(beta=beta_init,lamda=lamda,X=X_train,y=y_train))
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")

# store starting time
begin = time.time()
print(Fx_logisticReg(b=beta_init, lamda=lamda,X=X_train,y=y_train))
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")

```

    0.6931471805599453
    Total runtime of the program is 1.0062980651855469
    0.6931471805599352
    Total runtime of the program is 1.008281946182251

```python
# Initializing gradient decent and fast algorithm grad descent
begin = time.time()
graddesc = gradient_descent(beta_init, eta_init ,lamda,X=X_train,y=y_train,epsilon=0.005)
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")

# store starting time
begin = time.time()
fastAlgo = fast_gradient_algo(beta_init, theta_init, eta_init,lamda, X=X_train,y=y_train,epsilon=0.005)
time.sleep(1)
# store end time
end = time.time()
# total time taken
print(f"Total runtime of the program is {end - begin}")

```

    Total runtime of the program is 1.54715895652771
    Total runtime of the program is 1.277160406112671

```python
graddesc.shape, fastAlgo.shape
```

    ((14, 58), (7, 58))

```python
def objective_plot(betas_gd, betas_fg, lambduh, x=X_train, y=y_train):
    num_points_gd = np.size(betas_gd, 0)
    objs_gd = np.zeros(num_points_gd)
    num_points_fg = np.size(betas_fg, 0)
    objs_fg = np.zeros(num_points_fg)
    obj_fx(betas_fg[0, :], lambduh, X=x, y=y)
    for i in range(num_points_gd):
        objs_gd[i] = obj_fx(betas_gd[i, :], lambduh, X=x, y=y)
    for i in range(num_points_fg):
        objs_fg[i] = obj_fx(betas_fg[i, :], lambduh, X=x, y=y)
    fig, ax = plt.subplots()
    ax.plot(range(1, num_points_gd + 1), objs_gd,label='gradient descent')
    ax.plot(range(1, num_points_fg + 1), objs_fg,label='fast gradient')
    plt.xlabel('Iteration')
    plt.ylabel('Objective value')
    plt.title('Objective value vs. iteration when lambda='+str(lambduh))
    ax.legend(loc='upper right')
    plt.show()

objective_plot(graddesc, fastAlgo, lamda)

```

```python

```

1-Python/2 - Logistic Regression w Fast Gradient Descent.ipynb


Make sure you remove `raise NotImplementedError()` and fill in any place that says `# YOUR CODE HERE`, as well as your `NAME`, `ID`, and `LAB_SECTION` below:

```python
NAME = "19201063"
ID = "FAHMID BIN KIBRIA"
LAB_SECTION = "02"
```

---

# Part 1: Differentiation: Forward, Backward, And Central
---

## Task 1: Differentiation

We have already learnt about *forward differentiation*, *backward diferentiation* and *central differentiation*. In this part of the assignment we will write methods to calculate this values and check how they perform.

The equations are as follows,

\begin{align}
\text{forward differentiation}, f^\prime(x) \simeq \frac{f(x+h)-f(x)}{h} \tag{4.6} \\
\text{backward differentiation}, f^\prime(x) \simeq \frac{f(x)-f(x-h)}{h} \tag{4.7} \\
\text{central differentiation}, f^\prime(x) \simeq \frac{f(x+h)-f(x-h)}{2h} \tag{4.8}
\end{align}

## Importing libraries

```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from numpy.polynomial import Polynomial
```

Here, `forward_diff(f, h, x)`, `backward_diff(f, h, x)`, and `central_diff(f, h, x)` calculates the *forward differentiation*, *backward differentiation* and *central differentiation* respectively. finally the `error(f, f_prime, h, x)` method calculates the different values for various $h$ and returns the errors.

Later we will run some code to test out performance. The first one is done for you.

```python
def forward_diff(f, h, x):
    return (f(x+h) - f(x)) / h
```

```python
def backward_diff(f, h, x):
    # --------------------------------------------
    # YOUR CODE HERE
    return ( f(x)  - f(x-h) ) / h
    # --------------------------------------------
```

```python
def central_diff(f, h, x):
    # --------------------------------------------
    # YOUR CODE HERE
    return ( f(x+h) - f(x-h) ) / ( 2 * h )
    # --------------------------------------------
```

```python
def error(f, f_prime, h, x):
    Y_correct = f_prime(x)
    f_error = np.array([])
    b_error = np.array([])
    c_error = np.array([])

    for h_i in h:
        # for different values of h (h_i)
        # calculate the error at the point x for  (i) forward method
        #                                         (ii) backward method
        #                                         (ii) central method
        # the first one is done for you
        f_error_h_i = forward_diff(f, h_i, x) - Y_correct
        f_error = np.append(f_error, f_error_h_i)

        # --------------------------------------------
        # YOUR CODE HERE
        b_error_h_i = backward_diff(f, h_i, x) - Y_correct
        b_error = np.append(b_error, b_error_h_i)

        c_error_h_i = central_diff(f, h_i, x) - Y_correct
        c_error = np.append(c_error, c_error_h_i)
        # --------------------------------------------
        pass

    return f_error, b_error, c_error
```

## Plot1
Polynomial and Actual Derivative Function

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')

p = Polynomial([2.0, 1.0, -6.0, -2.0, 2.5, 1.0])
data = p.linspace(domain=[-2.4, 1.5])
ax.plot(data[0], data[1], label='Function')

p_prime = p.deriv(1)
data2 = p_prime.linspace(domain=[-2.4, 1.5])
ax.plot(data2[0], data2[1], label='Derivative')

ax.legend()
```

```python
h = 1
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

x = np.linspace(-2.0, 1.3, 50, endpoint=True)
y = forward_diff(p, h, x)
bx.plot(x, y, label='Forward; h=1')
y = backward_diff(p, h, x)
bx.plot(x, y, label='Backward; h=1')
y = central_diff(p, h, x)
bx.plot(x, y, label='Central; h=1')

data2 = p_prime.linspace(domain=[-2.0, 1.3])
bx.plot(data2[0], data2[1], label='actual')

bx.legend()

```

```python
h = 0.1
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

x = np.linspace(-2.2, 1.3, 50, endpoint=True)
y = forward_diff(p, h, x)
bx.plot(x, y, label='Forward; h=0.1')
y = backward_diff(p, h, x)
bx.plot(x, y, label='Backward; h=0.1')
y = central_diff(p, h, x)
bx.plot(x, y, label='Central; h=0.1')

data2 = p_prime.linspace(domain=[-2.2, 1.3])
bx.plot(data2[0], data2[1], label='actual')

bx.legend()

```

```python
h = 0.01
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

x = np.linspace(-2.2, 1.3, 50, endpoint=True)
y = forward_diff(p, h, x)
bx.plot(x, y, label='Forward; h=0.01')
y = backward_diff(p, h, x)
bx.plot(x, y, label='Backward; h=0.01')
y = central_diff(p, h, x)
bx.plot(x, y, label='Central; h=0.01')

data2 = p_prime.linspace(domain=[-2.2, 1.3])
bx.plot(data2[0], data2[1], label='actual')

bx.legend()

```

```python
fig, bx = plt.subplots()
bx.axhline(y=0, color='k')

h = np.array([1., 0.55, 0.3, .17, 0.1, 0.055, 0.03, 0.017, 0.01])
err = error(p, p_prime, h, 2.0)

bx.plot(h, err[0], label='Forward')
bx.plot(h, err[1], label='Backward')
bx.plot(h, err[2], label='Central')
bx.legend()

```

## Part 2: Richardson Extrapolation
---

## Instructions

Today's assignment is to:
1. Implement Richardson Extrapolation method using Python

## Richardson Extrapolation:
We used central difference method to calculate derivatives of functions last task. In this task we will use Richardson extrapolation to get a more accurate result.
Let,
$$ D_h = \frac{f(x_1+h) -f(x_1-h)}{2h}\tag{5.1}$$

General Taylor Series formula:
$$ f(x) = f(x_1) + f'(x_1)(x - x_1) + \frac{f''(x_1)}{2}(x - x_1)^2+... $$
Using Taylor's theorem to expand we get,

\begin{align}
f(x_1+h) &= f(x_1) + f^{\prime}(x_1)h + \frac{f^{\prime \prime}(x_1)}{2}h^2 + \frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + \frac{f^{(4)}(x_1)}{4!}h^4 + \frac{f^{(5)}(x_1)}{5!}h^5 + O(h^6)\tag{5.2} \\
f(x_1-h) &= f(x_1) - f^{\prime}(x_1)h + \frac{f^{\prime \prime}(x_1)}{2}h^2 - \frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + \frac{f^{(4)}(x_1)}{4!}h^4 - \frac{f^{(5)}(x_1)}{5!}h^5 + O(h^6)\tag{5.3}
\end{align}
Subtracting $5.3$ from $5.2$ we get,
$$ f(x_1+h) - f(x_1-h) = 2f^{\prime}(x_1)h + 2\frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + 2\frac{f^{(5)}(x_1)}{5!}h^5 + O(h^7)\tag{5.4}$$
So,
\begin{align}
D_h &= \frac{f(x_1+h) - f(x_1-h)}{2h} \\
&= \frac{1}{2h} \left( 2f^{\prime}(x_1)h + 2\frac{f^{\prime \prime \prime}(x_1)}{3!}h^3 + 2\frac{f^{(5)}(x_1)}{5!}h^5 + O(h^7) \right) \\
&= f^{\prime}(x_1) + \frac{f^{\prime \prime \prime}(x_1)}{6}h^2 + \frac{f^{(5)}(x_1)}{120}h^4 + O(h^6) \tag{5.5}
\end{align}
We get our derivative $f'(x)$ plus some error terms of order $>= 2$ Now, we want to bring our error order down to 4.

If we use $h, \text{and} \frac{h}{2}$ as step size in $5.5$, we get,
\begin{align}
D_h &= f^{\prime}(x_1) + f^{\prime \prime \prime}(x_1)\frac{h^2}{6} + f^{(5)}(x_1) \frac{h^4}{120} + O(h^6) \tag{5.6} \\
D_{h/2} &= f^{\prime}(x_1) + f^{\prime \prime \prime}(x_1)\frac{h^2}{2^2 . 6} + f^{(5)}(x_1) \frac{h^4}{2^4 . 120} + O(h^6) \tag{5.7}
\end{align}
Multiplying $5.7$ by $4$ and subtracting from $5.6$ we get,
\begin{align}
D_h - 4D_{h/2} &= -3f^{\prime}(x) + f^{(5)}(x_1) \frac{h^4}{160} + O(h^6)\\
\Longrightarrow D^{(1)}_h = \frac{4D_{h/2} - D_h}{3} &= f^{\prime}(x) - f^{(5)}(x_1) \frac{h^4}{480} + O(h^6) \tag{5.8}
\end{align}
Let's calculate the derivative using $5.8$

### 1. Let's create a function named `dh(f, h, x)`
function `dh(f, h, x)` takes three parameters as input: a function `f`, a value `h`, and a set of values `x`. It returns the derivatives of the function at each elements of array `x` using the Central Difference method. This calculates equation $(5.1)$.

```python
def dh(f, h, x):
    '''
    Input:
        f: np.polynomial.Polynonimial type data.
        h: floating point data.
        x: np.array type data.
    Output:
        return np.array type data of slope at each point x.
    '''
    # --------------------------------------------
    # YOUR CODE HERE
    return ( f(x+h)-f(x-h) ) / ( 2 * h )
    # --------------------------------------------
```

### 2. Let's create another funtion `dh1(f, h, x)`.
`dh1(f, h, x)` takes the same type of values as `dh(f, h, x)` as input. It calculates the derivative using previously defined `dh(f, h, x)` function and using equation $5.8$ and returns the values.

```python
def dh1(f, h, x):
    '''
    Input:
        f: np.polynomial.Polynonimial type data.
        h: floating point data.
        x: np.array type data.
    Output:
        return np.array type data of slope at each point x.
    '''
    # --------------------------------------------
    # YOUR CODE HERE
    return ( 4 * dh(f,h/2,x) - dh(f,h,x) ) / 3
    # --------------------------------------------
```

### 3. Now let's create the `error(f, hs, x_i)` function
The `error(f, hs, x_i)` function takes a function `f` as input. It also takes a list of different values of h as `hs` and a specific value as `x_i` as input. It calculates the derivatives as point `x_i` using both functions described in **B** and **C**, i.e. `dh` and `dh1`.

```python
def error(f, hs, x_i):  #Using the functions we wrote dh() my c_diff  and dh1() which is my first order c diff, we find the error through appending their diffrences with Y_actual ny f(x)
    '''
    Input:
        f  : np.polynomial.Polynonimial type data.
        hs : np.array type data. list of h.
        x_i: floating point data. single value of x.
    Output:
        return two np.array type data of errors by two methods..
    '''
    f_prime = f.deriv(1)   #first order derivitive f^1(x)
    Y_actual = f_prime(x_i)

    diff_error = []
    diff2_error = []

    for h in hs: #where h is my loop counter iterating through hs
        # for each values of hs calculate the error using both methods
        # and append those values into diff_error and diff2_error list.

        # --------------------------------------------
        # YOUR CODE HERE
        error1 =  Y_actual - dh(f,h,x_i)
        error2 = Y_actual - dh1(f,h,x_i)

        diff_error.append(error1)
        diff2_error.append(error2)
        # --------------------------------------------

    print(pd.DataFrame({"h": hs, "Diff": diff_error, "Diff2": diff2_error}))

    return diff_error, diff2_error
```

### 4. Finally let's run some tests
function to draw the actual function.

```python
def draw_graph(f, ax, domain=[-10, 10], label=None):
    data = f.linspace(domain=domain)
    ax.plot(data[0], data[1], label='Function')
```

### Draw the polynomial and it's actual derivative function

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')

p = Polynomial([2.0, 1.0, -6.0, -2.0, 2.5, 1.0])
p_prime = p.deriv(1)
draw_graph(p, ax, [-2.4, 1.5], 'Function')
draw_graph(p_prime, ax, [-2.4, 1.5], 'Derivative')

ax.legend()
```

### Draw the actual derivative and richardson derivative using `h=1` and `h=0.1` as step size.

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')

draw_graph(p_prime, ax, [-2.4, 1.5], 'actual')

h = 1
x = np.linspace(-2.4, 1.5, 50, endpoint=True)
y = dh1(p, h, x)
ax.plot(x, y, label='Richardson; h=1')

h = 0.1
x = np.linspace(-2.4, 1.5, 50, endpoint=True)
y = dh1(p, h, x)
ax.plot(x, y, label='Richardson; h=0.1')

ax.legend()

```

### Draw error-vs-h cuve

```python
fig, ax = plt.subplots()
ax.axhline(y=0, color='k')
hs = np.array([1., 0.55, 0.3, .17, 0.1, 0.055, 0.03, 0.017, 0.01])
e1, e2 = error(p, hs, 2.0)
ax.plot(hs, e1, label='e1')
ax.plot(hs, e2, label='e2')

ax.legend()

```

```python

```