#Chapter 4
______

##The greatest theorem never told

This chapter focuses on an idea that is always bouncing around our minds, but is rarely made explicit outside books devoted to statistics. In fact, we've been used this simple idea in every example thus far.

###The Law of Large Numbers

Let $Z_i$ be $N$ independent samples from some probability distribution. According to *the Law of Large numbers*,  so long as the expected value $E[Z]$ is finite, the following holds,

$$\frac{1}{N} \sum_{i=1}^N Z_i \rightarrow E[ Z ],  \;\;\; N \rightarrow \infty.$$

In words:

>   The average of a sequence of random variables from the same distribution converges to the expected value of that distribution.

This may seem like a boring result, but it will be the most useful tool you use.

### Intuition

If the above Law is somewhat surprising,  it can be made more clear be examining a simple example.

Consider a random variable $Z$ that can take only two values, $c_1$ and $c_2$. Suppose we have a large number of samples of $Z$, denoting a specific sample $Z_i$. The Law says that we can approximate the expected value of $Z$ by averaging over all samples. Consider the average:

$$ \frac{1}{N} \sum_{i=1}^N \;Z_i $$

By construction, $Z_i$ can only take on $c_1$ or $c_2$, hence we can partition the sum over these two values:

\begin{align}
\frac{1}{N} \sum_{i=1}^N \;Z_i
& =\frac{1}{N} \big(  \sum_{ Z_i = c_1}c_1 + \sum_{Z_i=c_2}c_2 \big) \\\\[5pt]
& = c_1 \sum_{ Z_i = c_1}\frac{1}{N} + c_2 \sum_{ Z_i = c_2}\frac{1}{N} \\\\[5pt]
& = c_1 \times \text{ (approximate frequency of $c_1$) } \\\\
& \;\;\;\;\;\;\;\;\; + c_2 \times \text{ (approximate frequency of $c_2$) } \\\\[5pt]
& \approx c_1 \times P(Z = c_1) + c_2 \times P(Z = c_2 ) \\\\[5pt]
& = E[Z]
\end{align}

Equality holds in the limit, but we can get closer and closer by using more and more samples in the average. This Law holds for almost *any distribution*, minus some important cases we will encounter later.

##### Example
____

Below is a diagram of the Law of Large numbers in action for three different sequences of Poisson random variables.

 We sample `sample_size = 100000` Poisson random variables with parameter $\lambda = 4.5$. (Recall the expected value of a Poisson random variable is equal to it's parameter.) We calculate the average for the first $n$ samples, for $n=1$ to `sample_size`.

```
%pylab inline

figsize( 12.5, 5 )
import pymc as pm

sample_size = 100000
expected_value = lambda_ = 4.5
poi = pm.rpoisson
N_samples = range(1,sample_size,100)

for k in range(3):

    samples = poi( lambda_, size = sample_size )

    partial_average = [ samples[:i].mean() for i in N_samples ]

    plt.plot( N_samples, partial_average, lw=1.5,label="average \
of  $n$ samples; seq. %d"%k)

plt.plot( N_samples, expected_value*np.ones_like( partial_average),
    ls = "--", label = "true expected value", c = "k" )

plt.ylim( 4.35, 4.65)
plt.title( "Convergence of the average of \n random variables to its \
expected value" )
plt.ylabel( "average of $n$ samples" )
plt.xlabel( "# of samples, $n$")
plt.legend();
```

Looking at the above plot, it is clear that when the sample size is small, there is greater variation in the average (compare how *jagged and jumpy* the average is initially, then *smooths* out). All three paths *approach* the value 4.5, but just flirt with it as $N$ gets large. Mathematicians and statistician have another name for *flirting*: convergence.

Another very relevant question we can ask is *how quickly am I converging to the expected value?* Let's plot something new. For a specific $N$, let's do the above trials thousands of times and compute how far away we are from the true expected value, on average. But wait — *compute on average*? This simply the law of large numbers again! For example, we are interested in, for a specific $N$, the quantity:

$$D(N) = \sqrt{ \;E\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \;\;\right] \;\;}$$

The above formulae is interpretable as a distance away from the true value (on average), for some $N$. (We take the square root so the dimensions of the above quantity and our random variables are the same). As the above is an expected value, it can be approximated using the law of large numbers: instead of averaging $Z_i$, we calculate the following multiple times and average them:

$$ Y_k = \left( \;\frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \; \right)^2 $$

By computing the above many, $N_y$, times (remember, it is random), and averaging them:

$$ \frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k \rightarrow E[ Y_k ] = E\;\left[\;\; \left( \frac{1}{N}\sum_{i=1}^NZ_i  - 4.5 \;\right)^2 \right]$$

Finally, taking the square root:

$$ \sqrt{\frac{1}{N_Y} \sum_{k=1}^{N_Y} Y_k} \approx D(N) $$

```
figsize( 12.5, 4)

N_Y = 250 #use this many to approximate D(N)
N_array = np.arange( 1000, 50000, 2500 ) #use this many samples in the approx. to the variance.
D_N_results = np.zeros( len( N_array ) )

lambda_ = 4.5
expected_value = lambda_ #for X ~ Poi(lambda) , E[ X ] = lambda

def D_N( n ):
    """
    This function approx. D_n, the average variance of using n samples.
    """
    Z = poi( lambda_, size = (n, N_Y) )
    average_Z = Z.mean(axis=0)
    return np.sqrt( (  (average_Z - expected_value)**2  ).mean() )

for i,n in enumerate(N_array):
    D_N_results[i] =  D_N(n)

plt.xlabel( "$N$" )
plt.ylabel( "expected squared-distance from true value" )
plt.plot(N_array, D_N_results, lw = 3,
            label="expected distance between\n\
expected value and \naverage of $N$ random variables.")
plt.plot( N_array, np.sqrt(expected_value)/np.sqrt(N_array), lw = 2, ls = "--",
        label = r"$\frac{\sqrt{\lambda}}{\sqrt{N}}$" )
plt.legend()
plt.title( "How 'fast' is the sample average converging? " );
```

As expected, the expected distance between our sample average and the actual expected value shrinks as $N$ grows large. But also notice that the *rate* of convergence decreases, that is, we need only 10 000 additional samples to move from 0.020 to 0.015, a difference of 0.005, but *20 000* more samples to again decrease from 0.015  to 0.010, again only a 0.005 decrease.

It turns out we can measure this rate of convergence. Above I have plotted a second line, the function $\sqrt{\lambda}/\sqrt{N}$. This was not chosen arbitrarily. In most cases, given a sequence of random variable distributed like $Z$, the rate of converge to $E[Z]$ of the Law of Large Numbers is

$$ \frac{ \sqrt{ \; Var(Z) \; } }{\sqrt{N} }$$

This is useful to know: for a given large $N$, we know (on average) how far away we are from the estimate. On the other hand, in a Bayesian setting, this can seem like a useless result: Bayesian analysis is OK with uncertainty so what's the *statistical* point of adding extra precise digits? Though drawing samples can be so computationally cheap that having a *larger* $N$ is fine too.

### How do we compute $Var(Z)$ though?

The variance is simply another expected value that can be approximated! Consider the following, once we have the expected value (by using the Law of Large Numbers to estimate it, denote it $\mu$), we can estimate the variance:

$$ \frac{1}{N}\sum_{i=1}^N \;(Z_i - \mu)^2 \rightarrow E[ \;( Z - \mu)^2 \;] = Var( Z )$$

### Expected values and probabilities
There is an even less explicit relationship between expected value and estimating probabilities. Define the *indicator function*

$$\mathbb{1}_A(x) =
\begin{cases} 1 &  x \in A \\\\
              0 &  else
\end{cases}
$$
Then, by the law of large numbers, if we have many samples $X_i$, we can estimate the probability of an event $A$, denoted $P(A)$, by:

$$ \frac{1}{N} \sum_{i=1}^N \mathbb{1}_A(X_i) \rightarrow E[\mathbb{1}_A(X)] =  P(A) $$

Again, this is fairly obvious after a moments thought: the indicator function is only 1 if the event occurs, so we are summing only the times the event occurs and dividing by the total number of trials  (consider how we usually approximate probabilities using frequencies). For example, suppose we wish to estimate the probability that a $Z \sim Exp(.5)$ is greater than 10, and we have many samples from a $Exp(.5)$ distribution.

$$ P( Z > 10 ) =  \sum_{i=1}^N \mathbb{1}_{z > 10 }(Z_i) $$

```
import pymc as pm

N = 10000
print np.mean( [ pm.rexponential( 0.5 )>10 for i in range(N) ] )
```

    0.0073

### What does this all have to do with Bayesian statistics?

*Point estimates*, to be introduced in the next chapter, in Bayesian inference are computed using expected values. In more analytical Bayesian inference, we would have been required to evaluate complicated expected values represented as multi-dimensional integrals. No longer. If we can sample from the posterior distribution directly, we simply need to evaluate averages. Much easier. If accuracy is a priority, plots like the ones above show how fast you are converging. And if further accuracy is  desired, just take more samples from the posterior.

When is enough enough? When can you stop drawing samples from the posterior? That is the practitioners decision, and also dependent on the variance of the samples (recall from above a high variance means the average will converge slower).

We also should understand when the Law of Large Numbers fails. As the name implies, and comparing the graphs above for small $N$, the Law is only true for large sample sizes. Without this, the asymptotic result is not reliable. Knowing in what situations the Law fails can give use *confidence in how unconfident we should be*. The next section deals with this issue.

## The Disorder of Small Numbers

The Law of Large Numbers is only valid as $N$ gets *infinitely* large: never truly attainable.  While the law is a powerful tool, it is foolhardy to apply it liberally. Our next example illustrates this.

##### Example: Aggregated geographic data

Often data comes in aggregated form. For instance, data may be grouped by state, county, or city level. Of course, the population numbers vary per geographic area. If the data is an average of some characteristic of each the geographic areas, we must be conscious of the Law of Large Numbers and how it can *fail* for areas with small populations.

We will observe this on a toy dataset. Suppose there are five thousand counties in our dataset. Furthermore,  population number in each state are uniformly distributed between 100 and 1500. The way the population numbers are generated is irrelevant to the discussion, so we do not justify this. We are interested in measuring the average height of individuals per county. Unbeknownst to the us, height does **not** vary across county, and each individual, regardless of the county he or she is currently living in, has the same distribution of what their height may be:

$$ \text{height} \sim \text{Normal}(150, 15 ) $$

We aggregate the individuals at the county level, so we only have data for the *average in the county*. What might our dataset look like?

```
figsize( 12.5, 4)
std_height = 15
mean_height = 150

n_counties = 5000
pop_generator = pm.rdiscrete_uniform
norm = pm.rnormal

#generate some artificial population numbers
population = pop_generator(100, 1500, size = n_counties )

average_across_county = np.zeros( n_counties )
for i in range( n_counties ):
    #generate some individuals and take the mean
    average_across_county[i] = norm(mean_height, 1./std_height**2,
                                        size=population[i] ).mean()

#located the counties with the apparently most extreme average heights.
i_min = np.argmin( average_across_county )
i_max = np.argmax( average_across_county )

#plot population size vs. recorded average
plt.scatter( population, average_across_county, alpha = 0.5, c="#7A68A6")
plt.scatter( [ population[i_min], population[i_max] ],
           [average_across_county[i_min], average_across_county[i_max] ],
           s = 60, marker = "o", facecolors = "none",
           edgecolors = "#A60628", linewidths = 1.5,
            label="extreme heights")

plt.xlim( 100, 1500 )
plt.title( "Average height vs. County Population")
plt.xlabel("County Population")
plt.ylabel("Average height in county")
plt.plot( [100, 1500], [150, 150], color = "k", label = "true expected \
height", ls="--" )
plt.legend(scatterpoints = 1)
```

What do we observe? *Without accounting for population sizes* we run the risk of making an enormous inference error: if we ignored population size, we would say that the county with the shortest and tallest individuals have been correctly circled. But this inference is wrong for the following reason. These two counties do *not* necessarily have the most extreme heights. The error results from the calculated average of smaller populations not being a good reflection of the true expected value of the population (which in truth should be $\mu =150$). The sample size/population size/$N$, whatever you wish to call it,  is simply too small to invoke the Law of Large Numbers effectively.

We provide more damning evidence against this inference. Recall the population numbers were uniformly distributed over 100 to 1500. Our intuition should tell us that the counties with the most extreme population heights should also be uniformly spread over 100 to 4000, and certainly independent of the county's population. Not so. Below are the population sizes of the counties with the most extreme heights.

```
print "Population sizes of 10 'shortest' counties: "
print population[ np.argsort( average_across_county )[:10] ]
print
print "Population sizes of 10 'tallest' counties: "
print population[ np.argsort( -average_across_county )[:10] ]
```

    Population sizes of 10 'shortest' counties:
    [120 186 150 167 108 178 257 156 164 164]

    Population sizes of 10 'tallest' counties:
    [100 121 125 232 198 378 177 120 113 184]

Not at all uniform over 100 to 1500. This is an absolute failure of the Law of Large Numbers.

##### Example:  Kaggle's *U.S. Census Return Rate Challenge*

Below is data from the 2010 US census, which partitions populations beyond counties to the level of block groups (which are aggregates of city blocks or equivalents). The dataset is from a Kaggle machine learning competition some colleagues and I participated in. The objective was to predict the census letter mail-back rate of a group block, measured between 0 and 100, using census variables (median income, number of females in the block-group, number of trailer parks, average number of children etc.). Below we plot the census mail-back rate versus block group population:

```
figsize( 12.5, 6.5 )
data = np.genfromtxt( "./data/census_data.csv", skip_header=1,
                        delimiter= ",")
plt.scatter( data[:,1], data[:,0], alpha = 0.5, c="#7A68A6")
plt.title("Census mail-back rate vs Population")
plt.ylabel("Mail-back rate")
plt.xlabel("population of block-group")
plt.xlim(-100, 15e3 )
plt.ylim( -5, 105)

i_min = np.argmin(  data[:,0] )
i_max = np.argmax(  data[:,0] )

plt.scatter( [ data[i_min,1], data[i_max, 1] ],
             [ data[i_min,0],  data[i_max,0] ],
             s = 60, marker = "o", facecolors = "none",
             edgecolors = "#A60628", linewidths = 1.5,
             label="most extreme points")

plt.legend(scatterpoints = 1);
```

The above is a classic phenomenon in statistics. I say *classic* referring to the "shape" of the scatter plot above. It follows a classic triangular form, that tightens as we increase the sample size (as the Law of Large Numbers becomes more exact).

I am perhaps overstressing the point and maybe I should have titled the book *"You don't have big data problems!"*, but here again is an example of the trouble with *small datasets*, not big ones. Simply, small datasets cannot be processed using the Law of Large Numbers. Compare with applying the Law without hassle to big datasets (ex. big data). I mentioned earlier that paradoxically big data prediction problems are solved by relatively simple algorithms. The paradox is partially resolved by understanding that the Law of Large Numbers creates solutions that are *stable*, i.e. adding or subtracting a few data points will not affect the solution much. On the other hand, adding or removing data points to a small dataset can create very different results.

For further reading on the hidden dangers of the Law of Large Numbers, I would highly recommend the excellent manuscript [The Most Dangerous Equation](http://nsm.uh.edu/~dgraur/niv/TheMostDangerousEquation.pdf).

##### Example: How to order Reddit comments

You may have disagreed with the original statement that the Law of Large numbers is known to everyone, but only implicitly in our subconscious decision making. Consider ratings on online products: how often do you trust an average 5-star rating if there is only 1 reviewer? 2 reviewers? 3 reviewers? We implicitly understand that with such few reviewers that the average rating is **not** a good reflection of the true value of the product.

This has created flaws in how we sort items, and more generally, how we compare items. Many people have realized that sorting online search results by their rating, whether the objects be books, videos, or online comments, return poor results. Often the seemingly top videos or comments have perfect ratings only from a few enthusiastic fans, and truly more quality videos or comments are hidden in later pages with *falsely-substandard* ratings of around 4.8. How can we correct this?

Consider the popular site Reddit (I purposefully did not link to the website as you would never come back). The site hosts links to stories or images, and a very popular part of the site are the comments associated with each link. Redditors can vote up or down on each comment (called upvotes and downvotes). Reddit, by default, will sort comments by Top, that is, the best comments.

How would you determine which comments are the best? There are a number of ways to achieve this:

1. *Popularity*: A comment is considered good if it has many upvotes. A problem with this model is that a comment with hundreds of upvotes, but thousands of downvotes. While being very *popular*, the comment is likely more controversial than best.
2. *Difference*: Using the *difference* of upvotes and downvotes. This solves the above problem, but fails when we consider the temporal nature of comments. Comments can be posted many hours after the original link submission. The difference method will bias the *Top* comments to be the oldest comments, which have accumulated more upvotes than newer comments, but are not necessarily the best.
3. *Time adjusted*:  Consider using Difference divided by the age of the comment. This creates a *rate*, something like *difference per second*, or *per minute*. An immediate counter example is, if we use per second, a 1 second old comment with 1 upvote would be better than a 100 second old comment with 99 upvotes. One can avoid this by only considering at least t second old comments. But what is a good t value? Does this mean no comment younger than t is good? We end up comparing unstable quantities with stable quantities (young vs. old comments).
3. *Ratio*: Rank comments by the ratio of upvotes to total number of votes (upvotes plus downvotes). This solves the temporal issue, such that new comments who score well can be considered Top just as likely as older comments, provided they have many upvotes to total votes. The problem here is that a comment with a single upvote (ratio = 1.0) will beat a comment with 999 upvotes and 1 downvote (ratio = 0.999), but clearly the latter comment is *more likely* to be better.

I used the phrase *more likely* for good reason. It is possible that the former comment, with a single upvote, is in fact a better comment than the later with 999 upvotes. The hesitation to agree with this is because we have not seen the other 999 potential votes the former comment might get. Perhaps it will achieve an additional 999 upvotes and 0 downvotes and be considered better than the latter, though not likely.

What we really want is an estimate of the *true upvote ratio*. Note that the true upvote ratio is not the same as the observed upvote ratio: the true upvote ratio is hidden, and we only observe upvotes vs. downvotes (one can think of the true upvote ratio as "what is the underlying probability someone gives this comment a upvote, versus a downvote"). So the 999 upvote/1 downvote comment probably has a true upvote ratio close to 1, which we can assert with confidence thanks to the Law of Large Numbers, but on the other hand we are much less certain about the true upvote ratio of the comment with only a single upvote. Sounds like a Bayesian problem to me.

One way to determine a prior on the upvote ratio is that look at the historical distribution of upvote ratios. This can be accomplished by scrapping Reddit's comments and determining a distribution. There are a few problems with this technique though:

1. Skewed data:  The vast majority of comments have very few votes, hence there will be many comments with ratios near the extremes (see the "triangular plot" in the above Kaggle dataset), effectively skewing our distribution to the extremes. One could try to only use comments with votes greater than some threshold. Again, problems are encountered. There is a tradeoff between number of comments available to use and a higher threshold with associated ratio precision.
2. Biased data: Reddit is composed of different subpages, called subreddits. Two examples are *r/aww*, which posts pics of cute animals, and *r/politics*. It is very likely that the user behaviour towards comments of these two subreddits are very different: visitors are likely friend and affectionate in the former, and would therefore upvote comments more, compared to the latter, where comments are likely to be controversial and disagreed upon. Therefore not all comments are the same.

In light of these, I think it is better to use a `Uniform` prior.

With our prior in place, we can find the posterior of the true upvote ratio. The Python script `comments_for_top_reddit_pic.py` will scrape the comments from the current top picture on Reddit. Below is the picture, and some comments:

```
from IPython.core.display import Image
#adding a number to the end of the %run call with get the ith top photo.
%run top_pic_comments.py 2

Image(top_post_url)
```

    Title of submission:
    Happy halloween filthy animals.
    http://i.imgur.com/Ecery9G.jpg

```
"""
contents: an array of the text from all comments on the pic
votes: a 2d numpy array of upvotes, downvotes for each comment.
"""
n_comments = len(contents )
comments = np.random.randint( n_comments, size=4)
print "Some Comments (out of %d total) \n-----------"%n_comments
for i in comments:
    print '"' + contents[i] + '"'
    print"upvotes/downvotes: ",votes[i,:]
    print
```

    Some Comments (out of 117 total)
    -----------
    "Dat iron mark...this is some award winning work right here"
    upvotes/downvotes:  [43  6]

    "Harry: [talking to Kate as she drags Kevin upstairs] Don't worry about me. I spoke with your husband already, and don't worry about your home. It's in good hands."
    upvotes/downvotes:  [1 0]

    "You win. "
    upvotes/downvotes:  [15  3]

    "Brilliant. "
    upvotes/downvotes:  [8 0]

 For a given true upvote ratio $p$ and $N$ votes, the number of upvotes will look like a Binomial random variable with parameters $p$ and $N$. (This is because of the equivalence between upvote ratio and probability of upvoting versus downvoting, out of $N$ possible votes/trials). We create a function that performs Bayesian inference on $p$, for a particular comment's upvote/downvote pair.

```
import pymc as pm

def posterior_upvote_ratio( upvotes, downvotes, samples = 20000):
    """
    This function accepts the number of upvotes and downvotes a particular comment recieved,
    and the number of posterior samples to return to the user. Assumes a uniform prior.
    """
    N = upvotes + downvotes
    upvote_ratio = pm.Uniform( "upvote_ratio", 0, 1 )
    observations = pm.Binomial( "obs",  N, upvote_ratio, value = upvotes, observed = True)
    #do the fitting; first do a MAP as it is cheap and useful.
    map_ = pm.MAP([upvote_ratio, observations ]).fit()
    mcmc = pm.MCMC([upvote_ratio, observations ])
    mcmc.sample(samples, samples/4)
    return mcmc.trace("upvote_ratio")[:]

```

Below are the resulting posterior distributions.

```
figsize( 11., 8)
posteriors = []
colours = ["#348ABD", "#A60628", "#7A68A6", "#467821", "#CF4457"]
for i in range(len(comments)):
    j = comments[i]
    posteriors.append( posterior_upvote_ratio( votes[j, 0], votes[j,1] ) )
    plt.hist( posteriors[i], bins = 18, normed = True, alpha = .9,
            histtype="step",color = colours[i%5], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 18, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )

plt.legend(loc="upper left")
plt.xlim( 0, 1)
plt.title("Posterior distributions of upvote ratios on different comments");
```

Some distributions are very tight, others have very long tails (relatively speaking), expressing our uncertainty with what the true upvote ratio might be.

### Sorting!

We have been ignoring the goal of this exercise: how do we sort the comments from *best to worst*? Of course, we cannot sort distributions, we must sort scalar numbers. There are many ways to distill a distribution down to a scalar: expressing the distribution through its expected value, or mean, is one way. Choosing the mean bad choice though. This is because the mean does not take into account the uncertainty of distributions.

I  suggest using the *95% least plausible value*, defined as the value such that there is only a 5% chance the true parameter is lower (think of the lower bound on the 95% credible region). Below are the posterior distributions with the 95% least-plausible value plotted:

```
N = posteriors[0].shape[0]
lower_limits = []

for i in range(len(comments)):
    j = comments[i]
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .9,
            histtype="step",color = colours[i], lw = 3,
            label = '(%d up:%d down)\n%s...'%(votes[j, 0], votes[j,1], contents[j][:50]) )
    plt.hist( posteriors[i], bins = 20, normed = True, alpha = .2,
            histtype="stepfilled",color = colours[i], lw = 3, )
    v = np.sort( posteriors[i] )[ int(0.05*N) ]
    #plt.vlines( v, 0, 15 , color = "k", alpha = 1, linewidths=3 )
    plt.vlines( v, 0, 10 , color = colours[i], linestyles = "--",  linewidths=3  )
    lower_limits.append(v)
    plt.legend(loc="upper left")

plt.legend(loc="upper left")
plt.title("Posterior distributions of upvote ratios on different comments");
order = argsort( -np.array( lower_limits ) )
print order, lower_limits
```

The best comments, according to our procedure, are the comments that are *most-likely* to score a high percentage of upvotes. Visually those are the comments with the 95% least plausible value close to 1.

Why is sorting based on this quantity a good idea? By ordering by the 95% least plausible value, we are being the most conservative with what we think is best. That is, even in the worst case scenario, when we have severely overestimated the upvote ratio, we can be sure the best comments are still on top. Under this ordering, we impose the following very natural properties:

1. given two comments with the same observed upvote ratio, we will assign the comment with more votes as better (since we are more confident it has a higher ratio).
2. given two comments with the same number of votes, we still assign the comment with more upvotes as *better*.

### But this is too slow for real-time!

I agree, computing the posterior of every comment takes a long time, and by the time you have computed it, likely the data has changed. I delay the mathematics to the appendix, but I suggest using the following formula to compute the lower bound very fast.

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where
\begin{align}
& a = 1 + u \\\\
& b = 1 + d \\\\
\end{align}

$u$ is the number of upvotes, and $d$ is the number of downvotes. The formula is a shortcut in Bayesian inference, which will be further explained in Chapter 6 when we discuss priors in more detail.

```
def intervals(u,d):
    a = 1. + u
    b = 1. + d
    mu = a/(a+b)
    std_err = 1.65*np.sqrt( (a*b)/( (a+b)**2*(a+b+1.) ) )
    return ( mu, std_err )

print "Approximate lower bounds:"
posterior_mean, std_err  = intervals(votes[:,0],votes[:,1])
lb = posterior_mean - std_err
print lb
print
print "Top 40 Sorted according to approximate lower bounds:"
print
order = np.argsort( -lb )
ordered_contents = []
for i in order[:40]:
    ordered_contents.append( contents[i] )
    print  votes[i,0], votes[i,1], contents[i]
    print "-------------"
```

    Approximate lower bounds:
    [ 0.84053422  0.80282754  0.81932714  0.84117301  0.84126989  0.80968254
      0.78400654  0.79853422  0.83882784  0.78547082  0.68548406  0.7609129
      0.80556899  0.7908931   0.75075188  0.75075188  0.65597619  0.6931046
      0.6931046   0.60100251  0.63905875  0.6931046   0.6530083   0.6931046
      0.635       0.68704735  0.47201974  0.60091591  0.47201974  0.60100251
      0.53055613  0.53055613  0.60091591  0.53055613  0.60091591  0.43047887
      0.43047887  0.53055613  0.43047887  0.43047887  0.43047887  0.43047887
      0.43047887  0.53055613  0.43047887  0.43047887  0.27        0.53055613
      0.43047887  0.43047887  0.45074913  0.45093892  0.3726793   0.3726793
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.43047887  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.13104878  0.27775794  0.27775794  0.27775794  0.43047887  0.27775794
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.13104878
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.27775794  0.27775794  0.27775794  0.43047887  0.27775794  0.27775794
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.27775794  0.27775794  0.27775794  0.27775794  0.27775794  0.27775794
      0.27775794  0.27775794  0.27775794]

    Top 40 Sorted according to approximate lower bounds:

    205 28 "SUCK BRICK KID!"
    -------------
    362 54 "Why'd you take your shoes off?"
     "Why are you dressed like a chicken?"
    -------------
    856 140 There really is only one thing a bartender could say to you two:

    > You guys give up, or ya thirsty for more?
    -------------
    127 16 You guys fucking win any contest in my book.  Amazingly done.
    -------------
    160 25 Are you scared Marv? Are you afraid?
    -------------
    125 20 Marv: Kids are scared of the dark.

    Harry: You're afraid of the dark, too, Marv.
    -------------
    11 0 So awesome.

    "SMASH his face with an iron"
    -------------
    95 15 Really weird because Home Alone 2 was on TV tonight. It's not even the Christmas season yet but I am pumped for classic movies like this to come back on.

    Whenever my dad will hurt himself he will do the voice of Marv when he's in the basement of the apartment building in HA2 and go "Haaaawwwy..."
    -------------
    145 26 The wet bandits!
    -------------
    23 2 You guys actually resemble the actors, too.
    -------------
    72 12 You know what I'll give you Snakes... I'll give you to the count of ten, to get your ugly, yella, no-good kester, off my property before I pump your guts full of lead!

    One..
    Two..
    TEN!

    *unleashes like a billion rounds while laughing like a psycho*

    And keep the change ya filthy animal.
    -------------
    43 6 Dat iron mark...this is some award winning work right here
    -------------
    29 4 Keep the change.
    -------------
    8 0 right in the shnauz!
    -------------
    8 0 Brilliant.
    -------------
    6 0 We're being scammed by a kindy-garner!

    Well done boys. Well done. i wish I was there to be your kevin.
    -------------
    6 0 A legitimate scary costume from that movie would've been the old man who salted the sidewalk. He freaked me the hell out a as a kid. Granted it wouldn't be very topical but you get the idea.
    -------------
    6 0 That's the sound of a tool chest coming down the stairs....adjust nose
    -------------
    6 0 http://i.imgur.com/kPuetee.jpg
    -------------
    10 1 Marry Christmas *tiwnkle*Smile*
    -------------
    17 3 macaulay culkin is now the same age as daniel stern was when he was in home alone.
    -------------
    15 3 You win.
    -------------
    5 0 Hey guys I'm callin' the cops!
    -------------
    14 3 Snakes, Snakes. I don't know no Snakes.
    -------------
    11 2 Excellent detail work!
    -------------
    7 1 Upvote from me you big horse's ass.
    -------------
    7 1 The Wet Bandits!
    -------------
    4 0 Haha, brings back great childhood memories!
    -------------
    4 0 And a happy new year!!
    -------------
    4 0 Im upvoting because of my childhood appreciates you.
    -------------
    3 0 Shut up, Marv! Geez
    -------------
    3 0 Bald guy needs more of a shit-eating grin displaying a little *gold* crown.

    EDIT: Now we're talkin': http://whiggles.landofwhimsy.com/hdcaptures/homealone3.jpg
    -------------
    3 0 This gave me a nostalgia rush
    -------------
    3 0  What's the brand mark on his hand from? I must not have remembered that part.
    -------------
    3 0 Gotta agree with most people here. Awesome costumes!
    -------------
    3 0 So much fucking win.
    -------------
    6 2 I have such a strong holiday association of Christmas for this movie that I'm having trouble processing that it's a Halloween costume. But the fact that your costume makes me think of Christmas means there must be an incredible likeness.
    -------------
    6 2 This leads me to believe Michael Rapaport would be excellent in a remake.
    -------------
    7 3 I wonder if the youngins are going to know who you guys are?
    -------------
    4 1 A, I'm not that lucky. Two, we use smoke detectors and D, we live on the most boring street in the whole United States of America, where nothing even remotely dangerous will ever happen. Period.
    -------------

We can view the ordering visually by plotting the posterior mean and bounds, and sorting by the lower bound. In the plot below, notice that the left error-bar is sorted (as we suggested this is the best way to determine an ordering), so the means, indicated by dots, do not follow any strong pattern.

```
r_order
```

    array([ 77,  66,  46, 116,  69,  82,  81,  80,  79,  78,  76,  75,  74,
            73,  72,  71,  67,  68,  84,  65,  64,  63,  62,  61,  59, 115,
            57,  56,  55,  54,  83,  58,  85, 100,  86, 113, 112, 111, 110, 109])

```
r_order = order[::-1][-40:]
plt.errorbar( posterior_mean[r_order], np.arange( len(r_order) ),
               xerr=std_err[r_order],xuplims=True, capsize=0, fmt="o",
                color = "#7A68A6")
plt.xlim( 0.3, 1)
plt.yticks( np.arange( len(r_order)-1,-1,-1 ), map( lambda x: x[:30].replace("\n",""), ordered_contents) );
```

In the graphic above, you can see why sorting by mean would be sub-optimal.

### Extension to Starred rating systems

The above procedure works well for upvote-downvotes schemes, but what about systems that use star ratings, e.g. 5 star rating systems. Similar problems apply with simply taking the average: an item with two perfect ratings would beat an item with thousands of perfect ratings, but a single sub-perfect rating.

We can consider the upvote-downvote problem above as binary: 0 is a downvote, 1 if an upvote. A $N$-star rating system can be seen as a more continuous version of above, and we can set $n$ stars rewarded is equivalent to rewarding $\frac{n}{N}$. For example, in a 5-star system, a 2 star rating corresponds to 0.4. A perfect rating is a 1. We can use the same formula as before, but with $a,b$ defined differently:

$$ \frac{a}{a + b} - 1.65\sqrt{ \frac{ab}{ (a+b)^2(a + b +1 ) } }$$

where

\begin{align}
& a = 1 + S \\\\
& b = 1 + N - S \\\\
\end{align}

where $N$ is the number of users who rated, and $S$ is the sum of all the ratings, under the equivilance scheme mentioned above.

##### Example: Counting Github stars

What is the average number of stars a Github repository has? How would you calculate this? There are over 6 million respositories, so there is more than enough data to invoke the Law of Large numbers. Let's start pulling some data. TODO

### Conclusion

While the Law of Large Numbers is cool, it is only true so much as its name implies: with large sample sizes only. We have seen how our inference can be affected by not considering *how the data is shaped*.

1. By (cheaply) drawing many samples from the posterior distributions, we can ensure that the Law of Large Number applies as we approximate expected values (which we will do in the next chapter).

2. Bayesian inference understands that with small sample sizes, we can observe wild randomness. Our posterior distribution will reflect this by being more spread rather than tightly concentrated. Thus, our inference should be correctable.

3. There are major implications of not considering the sample size, and trying to sort objects that are unstable leads to pathological orderings. The method provided above solves this problem.

### Appendix

##### Derivation of sorting comments formula

Basically what we are doing is using a Beta prior (with parameters $a=1, b=1$, which is a uniform distribution), and using a Binomial likelihood with observations $u, N = u+d$. This means our posterior is a Beta distribution with parameters $a' = 1 + u, b' = 1 + (N - u) = 1+d$. We then need to find the value, $x$, such that 0.05 probability is less than $x$. This is usually done by inverting the CDF ([Cumulative Distribution Function](http://en.wikipedia.org/wiki/Cumulative_Distribution_Function)), but the CDF of the beta, for integer parameters, is known but is a large sum [3].

We instead using a Normal approximation. The mean of the Beta is $\mu = a'/(a'+b')$ and the variance is

$$\sigma^2 = \frac{a'b'}{ (a' + b')^2(a'+b'+1) }$$

Hence we solve the following equation for $x$ and have an approximate lower bound.

$$ 0.05 = \Phi\left( \frac{(x - \mu)}{\sigma}\right) $$

$\Phi$ being the [cumulative distribution for the normal distribution](http://en.wikipedia.org/wiki/Normal_distribution#Cumulative_distribution)

##### Exercises

1\. How would you estimate the quantity $E\left[ \cos{X} \right]$, where $X \sim \text{Exp}(4)$? What about $E\left[ \cos{X} | X \lt 1\right]$, i.e. the expected value *given* we know $X$ is less than 1? Would you need more samples than the original samples size to be equally accurate?

```
## Enter code here
import scipy.stats as stats
exp = stats.expon( scale=4 )
N = 1e5
X = exp.rvs( N )
## ...
```

2\. The following table was located in the paper "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression" [2]. The table ranks football field-goal kickers by their percent of non-misses. What mistake have the researchers made?

-----

####  Kicker Careers Ranked by Make Percentage
<table><tbody><tr><th>Rank </th><th>Kicker </th><th>Make % </th><th>Number  of Kicks</th></tr><tr><td>1 </td><td>Garrett Hartley </td><td>87.7 </td><td>57</td></tr><tr><td>2</td><td> Matt Stover </td><td>86.8 </td><td>335</td></tr><tr><td>3 </td><td>Robbie Gould </td><td>86.2 </td><td>224</td></tr><tr><td>4 </td><td>Rob Bironas </td><td>86.1 </td><td>223</td></tr><tr><td>5</td><td> Shayne Graham </td><td>85.4 </td><td>254</td></tr><tr><td>… </td><td>… </td><td>…</td><td> </td></tr><tr><td>51</td><td> Dave Rayner </td><td>72.2 </td><td>90</td></tr><tr><td>52</td><td> Nick Novak </td><td>71.9 </td><td>64</td></tr><tr><td>53 </td><td>Tim Seder </td><td>71.0 </td><td>62</td></tr><tr><td>54 </td><td>Jose Cortez </td><td>70.7</td><td> 75</td></tr><tr><td>55 </td><td>Wade Richey </td><td>66.1</td><td> 56</td></tr></tbody></table>

In August 2013, [a popular post](http://bpodgursky.wordpress.com/2013/08/21/average-income-per-programming-language/) on the average income per programmer of different languages was trending. Here's the summary chart: (reproduced without permission, cause when you lie with stats, you gunna get the hammer). What do you notice about the extremes?

------

#### Average household income by programming language

<table >
<tr ><th></th></tr>
 <tr><td>Language</td><td>Average Household Income ($)</td><td>Data Points</td></tr>
 <tr><td>Puppet</td><td>87,589.29</td><td>112</td></tr>
 <tr><td>Haskell</td><td>89,973.82</td><td>191</td></tr>
 <tr><td>PHP</td><td>94,031.19</td><td>978</td></tr>
 <tr><td>CoffeeScript</td><td>94,890.80</td><td>435</td></tr>
 <tr><td>VimL</td><td>94,967.11</td><td>532</td></tr>
 <tr><td>Shell</td><td>96,930.54</td><td>979</td></tr>
 <tr><td>Lua</td><td>96,930.69</td><td>101</td></tr>
 <tr><td>Erlang</td><td>97,306.55</td><td>168</td></tr>
 <tr><td>Clojure</td><td>97,500.00</td><td>269</td></tr>
 <tr><td>Python</td><td>97,578.87</td><td>2314</td></tr>
 <tr><td>JavaScript</td><td>97,598.75</td><td>3443</td></tr>
 <tr><td>Emacs Lisp</td><td>97,774.65</td><td>355</td></tr>
 <tr><td>C#</td><td>97,823.31</td><td>665</td></tr>
 <tr><td>Ruby</td><td>98,238.74</td><td>3242</td></tr>
 <tr><td>C++</td><td>99,147.93</td><td>845</td></tr>
 <tr><td>CSS</td><td>99,881.40</td><td>527</td></tr>
 <tr><td>Perl</td><td>100,295.45</td><td>990</td></tr>
 <tr><td>C</td><td>100,766.51</td><td>2120</td></tr>
 <tr><td>Go</td><td>101,158.01</td><td>231</td></tr>
 <tr><td>Scala</td><td>101,460.91</td><td>243</td></tr>
 <tr><td>ColdFusion</td><td>101,536.70</td><td>109</td></tr>
 <tr><td>Objective-C</td><td>101,801.60</td><td>562</td></tr>
 <tr><td>Groovy</td><td>102,650.86</td><td>116</td></tr>
 <tr><td>Java</td><td>103,179.39</td><td>1402</td></tr>
 <tr><td>XSLT</td><td>106,199.19</td><td>123</td></tr>
 <tr><td>ActionScript</td><td>108,119.47</td><td>113</td></tr>
</table>

### References

1. Wainer, Howard. *The Most Dangerous Equation*. American Scientist, Volume 95.
2. Clarck, Torin K., Aaron W. Johnson, and Alexander J. Stimpson. "Going for Three: Predicting the Likelihood of Field Goal Success with Logistic Regression." (2013): n. page. Web. 20 Feb. 2013.
3. http://en.wikipedia.org/wiki/Beta_function#Incomplete_beta_function

```
from IPython.core.display import HTML
def css_styling():
    styles = open("../styles/custom.css", "r").read()
    return HTML(styles)
css_styling()
```

<style>
    @font-face {
        font-family: "Computer Modern";
        src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
    }
    div.cell{
        width:800px;
        margin-left:16% !important;
        margin-right:auto;
    }
    h1 {
        font-family: Helvetica, serif;
    }
    h4{
        margin-top:12px;
        margin-bottom: 3px;
       }
    div.text_cell_render{
        font-family: Computer Modern, "Helvetica Neue", Arial, Helvetica, Geneva, sans-serif;
        line-height: 145%;
        font-size: 130%;
        width:800px;
        margin-left:auto;
        margin-right:auto;
    }
    .CodeMirror{
            font-family: "Source Code Pro", source-code-pro,Consolas, monospace;
    }
    .prompt{
        display: None;
    }
    .text_cell_render h5 {
        font-weight: 300;
        font-size: 22pt;
        color: #4057A1;
        font-style: italic;
        margin-bottom: .5em;
        margin-top: 0.5em;
        display: block;
    }

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

<style>
    img{
        max-width:800px}
</style>


# Non Linear Regression Analysis

If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression since linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example in python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Although linear regression can do a great job at modeling some datasets, it cannot be used for all datasets. First recall how linear regression, models a dataset. It models the linear relationship between a dependent variable y and the independent variables x. It has a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##We can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

Non-linear regression is a method to model the non-linear relationship between the independent variables $x$ and the dependent variable $y$. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).  For example:

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and so on. For example: $$ y = \log(x)$$

We can have a function that's even more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##We can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

As we can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##We can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##We can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y,'green')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying the logarithmic map from the input $x$ to the output $y$. It is one of the simplest form of **log()**: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be a polynomial representation of the $x$ values. In general form it would be written as\
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y,'green')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y,'green')
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2021-12-11 01:19:01 URL:https://cf-courses-data.s3.us.cloud-object-storage.appdomain.cloud/IBMDeveloperSkillsNetwork-ML0101EN-SkillsNetwork/labs/Module%202/data/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

### Plotting the Dataset

This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerates slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Independent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta\_1(X-\beta\_2)}}$$

$\beta\_1$: Controls the curve's steepness,

$\beta\_2$: Slides the curve on the x-axis.

### Building The Model

Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?

we can use **curve_fit** which uses non-linear least squares to fit our sigmoid function, to data. Optimize values for the parameters so that the sum of the squared residuals of sigmoid(xdata, \*popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.447531, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Can we calculate what is the accuracy of our model?

```python
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )
```

    Mean absolute error: 0.03
    Residual sum of squares (MSE): 0.00
    R2-score: 0.98


Solution to: [Day 5: Poisson Distribution I](https://www.hackerrank.com/challenges/s10-poisson-distribution-1/problem)

Table of Contents

- Table of Contents
- Notes
    - Poisson Experiment
    - Poisson Distribution
- Examples
    - Example 1
    - Example 2
    - Special Example
- Math Solution
- Solution
    - Imports
    - Constants
    - Input
    - Poisson Distribution
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Notes

## Poisson Experiment
A Poisson experiment is a statistical experiment that has the following properties:
- binary outcome of experiment: Success (s) or failure (f)
- average number of successes in specified region is known
- Probability that success occurs is proportional to size of region
- Probability that success occurs in extremely small region is virtually 0

## Poisson Distribution
A Poisson random variable is the number of successes that result from a Poisson experiment. The probability distribution of a Poisson random variable is called a Poisson distribution:

\begin{equation}
\large
P(k, \lambda) =
\frac
{\lambda ^{k} * e ^{-\lambda}}
{k!}
\end{equation}

Where:
- P(k, λ) is the Poisson probability to get exactly k successes when average success is λ.
- λ = average number of successes in region
- k = actual number of successes in region
- e = 2.71828

# Examples

## Example 1
Acme Realty company sells an average of 2 homes per day. What is the probability that exactly 3 homes will be sold tomorrow?

- $\lambda$ = 2
- k = 3

\begin{equation}
\large
P(3, 2) =
\frac
{2^{3} * e^{-2}}
{3!}
\end{equation}

\begin{equation}
\large
= 0.180
\end{equation}

## Example 2
Suppose the average number of lions seen by tourists on a one-day safari is 5. What is the probability that tourists will see fewer than 4 lions on the next one-day safari?

This requires a summation of Poisson distribution Mass Probability Function, from 1 to 3.

\begin{equation}
\large
= \sum_{k=1}^{3}\frac
{5^{k} * e^{-5}}
{k!}
\end{equation}

## Special Example
Consider some Poisson random variable, X. Let **E[X]** be the expectation of X. Find the value of E[X**2].

Let **Var(X)** be the variance of X. Recall that if a random variable has a Poisson distribution, then:

\begin{equation}
\large
E[X] = \lambda
\end{equation}

\begin{equation}
\large
Var(X) = \lambda
\end{equation}

The following is the property of expectation and variation for any random variable, X:

\begin{equation}
\large
Var(x) = E[X^{2}] - E[X]^{2}
\end{equation}

\begin{equation}
\large
E[X^{2}] = Var(X) + E[X]^{2}
\end{equation}

Thus,

\begin{equation}
\large
E[X^{2}] = \lambda + \lambda^{2}
\end{equation}

# Math Solution

A random variable, X, follows Poisson distribution with mean of 2.5. Find the probability with which the random variable X is equal to 5.

- labmda = 2.5
- k = 5

\begin{equation}
\large
P(5, 2.5) = \frac
{2.5^{5} * e ^{-2.5}}
{5!}
\end{equation}

\begin{equation}
\large
0.067
\end{equation}

# Solution

## Imports

```python
from typing import Tuple
```

## Constants

```python
E = 2.71828
```

## Input

```python
def get_input() -> Tuple[float, int]:
	"""Returns input for Day 5: Poisson Distribution I.

	Returns:
		Tuple[float, int]: X's mean and desired random variable value, respectively
	"""
	var_mean = float(input())
	desired_value = int(input())
	return (var_mean, desired_value)
```

## Poisson Distribution

```python
def factorial(num: int) -> int:
	"""Returns factorial of num

	Args:
		num (int): Number to factorial

	Returns:
		int: Factorial result
	"""
	output = 1
	for i in range(num, 0, -1):
		output *= i
	return output
```

```python
def calc_poisson_prob_mass(desired_value: int, val_mean: float) -> float:
	"""Returns mass probability for desired value in Poisson Distribution with val_mean.

	Args:
		desired_value (int): Desired value of random variable, X
		val_mean (float): mean value of X

	Returns:
		float: Returns probability of desired value in Poisson Dist.
	"""
	return val_mean ** desired_value * E ** -val_mean / factorial(desired_value)
```

```python
def format_scale(num: float) -> float:
	"""Returns number formatted to scale.

	Args:
		num (float): number to

	Returns:
		float: formatted number
	"""
	return f"{num:.3f}"
```

## Main

```python
def main():
	var_mean, desired_value = get_input()

	prob = calc_poisson_prob_mass(desired_value, var_mean)
	print(format_scale(prob))
```

```python
if __name__ == "__main__":
	main()
```

    2.5
    5
    0.067


```python
%pylab inline
import utils
import scipy.stats
from sklearn.preprocessing import PolynomialFeatures

try:
    from linear_model import BayesianLinearModel
except:
    msg = "'The 'BayesianLinearModel' module does not appear to be installed."
    raise Exception(msg)

# Use same random data (for reproducibility).
np.random.seed(42)
```

#Bayesian Linear Regression

**Authors**: Asher Bender
$$
\DeclareMathOperator{\absoluteVal}{abs}
\renewcommand{\det} [1]{{\begin{vmatrix}#1\end{vmatrix}}}
\newcommand{\brac}  [1]{{ \left( #1 \right) }}
\newcommand{\sbrac} [1]{{ \left[ #1 \right] }}
\newcommand{\cbrac} [1]{{ \left{ #1 \right} }}
\newcommand{\AutoMiddle}{{\;\vert\;}}
\newcommand{\prob}      [2][p]{{#1\!\brac{#2}}}
\newcommand{\condprob}  [2]{p\!\brac{#1\AutoMiddle#2}}
\newcommand{\observations}{{\mathcal{D}}}
\newcommand{\sinput}{{x}}
\newcommand{\vinput}{{\mathbf{x}}}
\newcommand{\minput}{{\mathbf{X}}}
\newcommand{\soutput}{{y}}
\newcommand{\voutput}{{\mathbf{y}}}
\newcommand{\moutput}{{\mathbf{Y}}}
\newcommand{\starget}{{t}}
\newcommand{\vtarget}{{\mathbf{t}}}
\newcommand{\mtarget}{{\mathbf{T}}}
\newcommand{\weight}{{w}}
\newcommand{\weights}{{\mathbf{\weight}}}
\newcommand{\params}{{\mathbf{\theta}}}
\newcommand{\design}{{\Phi}}
\newcommand{\sbasis}{{\phi}}
\newcommand{\vbasis}{{\mathbf{\phi}}}
\newcommand{\snoise}{{\epsilon}}
\newcommand{\vnoise}{{\mathbf{\snoise}}}
\newcommand{\sbasisfcn}[1]{\sbasis\!\brac{#1}}
\newcommand{\vbasisfcn}[1]{\vbasis\!\brac{#1}}
\newcommand{\variance}{{\sigma^2}}
\newcommand{\precision}{{\lambda}}
\newcommand{\normal}{{\mathcal{N}}}
\newcommand{\normalmean}{{\mathbf{\mu}}}
\newcommand{\normalvariance}{{\mathbf{V}}}
\newcommand{\normalprecision}{{\mathbf{S}}}
\newcommand{\normaldist}[2]{\normal\brac{#1, #2}}
\newcommand{\normalcond}[3]{\normal\brac{#1\AutoMiddle#2, #3}}
\newcommand{\gammashape}{{\alpha}}
\newcommand{\gammascale}{{\beta}}
\newcommand{\gammarate}{{\beta}}
\newcommand{\gammasym}{{\mathcal{G}}}
\newcommand{\gammadist}[2]{\gammasym\brac{#1, #2}}
\newcommand{\gammacond}[3]{\gammasym\brac{#1\AutoMiddle#2, #3}}
\newcommand{\igammadist}[2]{i\gammasym\brac{#1, #2}}
\newcommand{\igammacond}[3]{i\gammasym\brac{#1\AutoMiddle#2, #3}}
\newcommand{\normaligammadist}[4]{\normal\!i\gammasym\brac{#1, #2, #3, #4}}
\newcommand{\normaligammacond}[5]{\normal\!i\gammasym\brac{#1\AutoMiddle#2, #3, #4, #5}}
\newcommand{\normalgammadist}[4]{\normal\gammasym\brac{#1, #2, #3, #4}}
\newcommand{\normalgammacond}[5]{\normal\gammasym\brac{#1\AutoMiddle#2, #3, #4, #5}}
\newcommand{\gammafcn}[1]{\Gamma\!\brac{#1}}
\newcommand{\data}{\mathcal{D}}
\newcommand{\params}{\mathbf{\theta}}
$$

# Linear Model

Consider a linear model that produces target outputs, $\soutput$, by linearly  combining the model parameters, $\weight_i$, and the input variables, $\sinput_i$:

\begin{equation}
  \soutput\brac{\vinput, \weights} = \weight_0 + \weight_1\sinput_1 + \ldots +
                                     \weight_D\sinput_D.
\end{equation}

The linear model can be extended to produce outputs that are a linear combination of fixed nonlinear functions of the input variables. These functions are known as *basis* functions. The linear model can now be
expressed as

\begin{equation}
  \soutput\brac{\vinput, \weights} = \weight_0\sbasis_0\brac{x} +
                                     \weight_1\sbasis_1\brac{\sinput} +
                                     \ldots +
                                     \weight_M\sbasis_M\brac{\sinput}
\end{equation}

For a set of observations,

\begin{equation}
  \voutput = \design\weights
\end{equation}

where:
* $\voutput = \brac{\soutput_1, \ldots, \soutput_N}^T$ are the observed scalar outputs.
* $\minput = \brac{\vinput_1, \ldots, \vinput_N}^T$ are the $M$-dimensional inputs.
* $\weights = \brac{\weight_1, \ldots, \weight_M}^T$ is the vector of model parameters.
* $\design$ is the design matrix where the input data, $\minput$, has been passed through the vector of nonlinear basis functions $\vbasis = \brac{\sbasis_1, \ldots, \sbasis_D}$.

The dimensions involved in the linear model are shown below

\begin{equation}
   \begin{bmatrix}
     \soutput_1 \\
     \vdots     \\
     \soutput_N \\
 \end{bmatrix}
 =
 \begin{bmatrix}
     \sbasis_0\!\brac{\vinput_1} &  \ldots & \sbasis_D\!\brac{\vinput_1} \\
     \vdots                      &  \ddots & \vdots                      \\
     \sbasis_0\!\brac{\vinput_N} &  \ldots & \sbasis_D\!\brac{\vinput_N} \\
 \end{bmatrix}
 \begin{bmatrix}
     \weight_0 \\
     \vdots    \\
     \weight_D \\
 \end{bmatrix}
\end{equation}

## Example

The example shown below is created by sampling from the uniform distribution $U\brac{\vinput|-1, 1}$, substituting these values into the function $f\brac{\vinput} = -0.3 + 0.5\vinput$

```python
# Create hidden linear model.
w_true = [-0.3, 0.5]
polybasis = lambda x, p: PolynomialFeatures(p).fit_transform(x)
linear_model = lambda x, w=w_true: polybasis(x, len(w) - 1).dot(w).reshape(len(x), 1)

utils.plot(({'x': np.linspace(-1., 1.)[:, None], 'model': linear_model},))
```

## Least squares parameter estimation

Supposing the form of the model is *known* but the model parameters are *unknown*. Following from the example above, we assume the model takes the form

\begin{equation}
  f\brac{\vinput} = b + m\vinput
\end{equation}

Although the form of the model is known, the model parameters are treated as unknown. The task is to estimate the slope $m$ and intercept $b$ from the data. In this scenario we will also assume the data is perturbed by an unknown noise.

```python
# Make noisy observations of model.
N = 1000
noise = 0.2
X = np.random.uniform(-1.0, 1.0, size=(N, 1)).reshape((N, 1))
y = linear_model(X) + np.random.normal(scale=noise, size=(N, 1))

utils.plot({'x': X, 'y': y, 'linestyle': '', 'marker': '.', 'markersize': 2, 'color': 'k'})
```

More generally, the task is to estimate the model weights $\weights$ from the linear model:

\begin{equation}
  \vtarget = \design\weights + \vnoise
\end{equation}

The least-squares solution can be obtained by finding the parameters that minimise the squared error between the model and observed values. These parameters are given by:

\begin{equation}
  \weights_{ml} = (\design^T\design)^{-1}\design^T\vtarget
\end{equation}

```python
# Note that numpy provides a function for implementing the above equation.
# 'lstsq' returns the least-squares solution to a linear matrix equation.
w_ml = np.linalg.lstsq(polybasis(X, len(w_true) - 1), y)[0]
model = lambda x: linear_model(x, w=w_ml)

utils.plot(({'x': X, 'y': y, 'linestyle': '', 'marker': '.', 'markersize': 2, 'color': 'k'},
            {'x': X, 'model': model, 'color': 'r', 'linewidth': 2}))
```

```python
print "'True' coefficients:        {}".format(w_true)
print "Least-squares coefficients: {}".format(w_ml.squeeze())
```

# Bayesian Linear model

If the observations are perturbed by independent and identically distributed (i.i.d.) Gaussian noise with a zero mean and a precision of $\precision$, an observed output can be modelled as a Gaussian random variable where

\begin{equation}
  \starget = \weights^T \sbasisfcn{\vinput} + \vnoise,
\end{equation}

and

\begin{equation}
  \condprob{\starget}{\vinput, \weights, \precision} =
  \normalcond{\starget}{\weights^T \sbasisfcn{\vinput}}{\precision^{-1}}.
\end{equation}

Assuming Gaussian noise, the above equation states that the expected output value at a nominated input location, $\vinput$, is a function of the basis functions, $\vbasis$, and model parameters, $\weights$. The variance in observations is inversely proportional to the precision of the Gaussian, $\precision^{-1}$. Often these variables are unknown. In such cases, fitting a linear model to the data requires finding the parameters $\params = \brac{\weights, \precision}$ that optimally model the relationship between the observed input and output data.

## Parameter estimation

The Bayesian approach to estimating the model parameters, $\params$, is to place a distribution over these variables. For a Gaussian with an unknown mean and covariance, the normal-gamma distribution can be used.

\begin{equation}
  \prob{\weights, \precision} = \normalgammacond{\weights, \precision}
                                                {\normalmean}{\normalprecision}
                                                {\gammashape}{\gammarate}
\end{equation}

Since the normal-gamma distribution forms a conjugate prior, closed form expressions can be drived for the distribution updates where:

\begin{equation}
  \condprob{\weights, \precision}{\minput, \voutput} =
          \normalgammacond{\weights, \precision}
                          {\normalmean_N}{\normalprecision_N}
                          {\gammashape_N}{\gammarate_N}
\end{equation}

and the sufficient statistics of the update are given by:

\begin{align}
  \normalmean_N &= \normalprecision_N
                   \brac{\normalprecision^{-1}_0\normalmean_0 +
                         \design^T\voutput}
  \\
  \normalprecision^{-1}_N &= \normalprecision^{-1}_0 + \design^T\design
  \\
  \gammashape_N &= \gammashape_0 + \frac{n}{2}
\end{align}

\begin{equation}
  \gammarate_N = \gammarate_0 + \frac{k}{2}
                 \brac{\normalmean_0^T\normalprecision_0^{-1}\normalmean_0 +
                       \voutput^T\voutput -
                       \normalmean_N^T\normalprecision_N^{-1}\normalmean_N}
\end{equation}

These equations allow the distribution over the model parameters to be updated sequentially as data is observed.

## Example

This section replicates the examples shown in Figure 3.7 of [1] and Figure 7.11 of [2]. The example has been extended to the case where the precision is unknown. The example demonstrates sequential Bayesian updating using a simple two parameter model of the form $\voutput\brac{\vinput, \weights} = w_0 + w_1\vinput$. Since this model only has two parameters, the posterior can be visualised easily.

As in the previous [example](#ml_example), the input data is created by sampling from the uniform distribution $U\brac{\vinput|-1, 1}$. The input data is substituted into the function $f\brac{\vinput} = -0.3 + 0.5\vinput$ and Gaussian noise, with a standard deviation of 0.2, is added to the output.

### Initialisation
The distribution is initialised with an uninformative and improper prior where:

\begin{align}
  \normalmean_0 &= \mathbf{0}                     \\
  \normalprecision^{-1} &= \infty^{-1}\mathbf{I}  \\
  \gammashape_0 &= \frac{-D}{2}                   \\
  \gammarate_0 &= 0                               \\
\end{align}

Since the prior is improper, it is only defined for $D < N - 1$. In the 2D example, the model can only be updated using more than three observations.

```python
# Create Bayesian linear model.
basis = lambda x: polybasis(x, 1)
blm = BayesianLinearModel(basis=basis)

# Perform update.
blm.update(X[:4], y[:4])
utils.plot_update(X[:4], y[:4], linear_model, blm)
```

### 10 observations

```python
blm.update(X[4:10], y[4:10])
utils.plot_update(X[:10], y[:10], linear_model, blm)
```

### 100 observations

```python
blm.update(X[10:100], y[10:100])
utils.plot_update(X[:100], y[:100], linear_model, blm)
```

### 1000 observations

```python
blm.update(X[100:], y[100:])
utils.plot_update(X, y, linear_model, blm)
```

Note that as more observations are added, the posterior distribution collapses on the correct estimates of the model parameters. Since an uniformative prior was used, the maximum a posteriori estimates of the model weights are identical to the maximum likelihood solution. This can be confirmed by comparing the Bayesian solution to the maximum likelihood solution:

```python
print "'True' coefficients:        {}".format(w_true)
print "Least-squares coefficients: {}".format(w_ml.squeeze())
print "Bayesian coefficients:      {}".format(blm.location.squeeze())
```

##Model Selection

It is important to note that the term 'linear model' describes linearity in the model parameters. That is, the output is a linear combination of the model weights and the inputs (or design matrix produced by basis function expansion). For example, complex non-linear outputs can be constructed using high degree polynomial basis functions:

```python
N = 50
noise = 0.25
X = np.sort(np.random.uniform(0, 2*np.pi, N)).reshape((N, 1))
y = np.sin(X) + np.random.normal(scale=noise, size=(N, 1))

# Approximate Sin function with a 6-degree polynomial.
w_ml = np.linalg.lstsq(polybasis(X, 6), y)[0]
model = lambda x: linear_model(x, w=w_ml)

utils.plot(({'x': X, 'y': y, 'linestyle': '', 'marker': '.', 'markersize': 2, 'color': 'k'},
            {'x': X, 'model': model, 'color': 'r', 'linewidth': 2}))
```

If the model is known, the parameters can be estimated using [least-squares](#least_squares) or [Bayesian updates](#bayesian_updates). The [previous section](#equivalence) showed that under a (semi-conjugate) uninformative prior, these two methods produce the same estimates. As the complexity of the data increases, specifying a model may be difficult.

Bayesian methods have a distinct advantage over maximum-likelihood methods during model selection. The problem with using maximum likelihood for model selection is that it favours complex models. As the complexity of the model is increased, the extra degrees of freedom allow the model to more precisely fit the training data. Despite producing models which reflect the training data accurately, the models usually have poor generalisation and reflect noise in the data - these models are examples of 'over-fitting'.

Bayesian methods differ to maximum likelihood model selection in that they display a preference for simpler models. Bayesian model selection is performed by calculating the marginal likelihood also known as the model evidence. For a specified model, this is done by marginalising the model parameters and calculating the probability of the data:

\begin{equation}
   \condprob{\data}{m} = \int \condprob{\data}{\params}\condprob{\theta}{m}d\theta
\end{equation}

For the Bayesian linear model, the marginal likelihood is given by:

\begin{equation}
    \prob{\data} = \frac{1}{2\pi^\frac{D}{2}}
                   \frac{{\gammascale_0}^{\gammashape_0}}
                        {{\gammascale_N}^{\gammashape_N}}
                   \frac{\gammafcn{\gammashape_N}}
                        {\gammafcn{\gammashape_0}}
                   \frac{\det{\normalprecision_N}^\frac{1}{2}}
                        {\det{\normalprecision_0}^\frac{1}{2}}
\end{equation}

### Example

In this example, polynomials of increasing complexity are fit to the trigonometric data shown in the [previous figure](#model_selection). Trigonometric functions are complex non-linear functions which cannot be exactly represented using a linear model. Despite this, a polynomial of sufficient complexity can reasonably approximate the observed data. This leaves the open question: how many degrees are sufficient? Model selection can be used to answer this question.

```python
num_plots = 15
num_cols = 3
num_rows = np.ceil(float(num_plots)/num_cols)

# Interate through polynomial degrees and plot linear models.
MSE = np.zeros(num_plots)
ML = np.zeros(num_plots)
fig = plt.figure(figsize=(5 * num_cols, 5 * num_rows))
fig.subplots_adjust(hspace=0.6)
for p in range(0, num_plots):

    # Determine the maximum likelihood weights and evaluate the model.
    theta = polybasis(X, p)
    w_ml = np.linalg.lstsq(theta, y)[0]
    fQuery = theta.dot(w_ml)
    MSE[p] = np.mean((fQuery - y)**2)

    # Create Bayesian linear model.
    blm = BayesianLinearModel(basis=lambda x: polybasis(x, p))
    blm.update(X, y)
    ML[p] = blm.evidence()

    # Plot ML/Bayesian linear models.
    plt.subplot(num_rows, num_cols, p + 1)
    utils.plot(({'x': X, 'y': y, 'linestyle': '', 'marker': '.', 'markersize': 2, 'color': 'k'},
                {'x': X, 'y': fQuery, 'color': 'b', 'linestyle': '--', 'linewidth': 2},
                {'x': X, 'y': blm.predict(X), 'color': 'r', 'linewidth': 2}))
    plt.xlabel('x')
    plt.ylabel('f(x)')
    plt.title('Polynomial degrees: ' + str(p))
```

Again, note that if an uninformative prior is used, the Bayesian solution is identical to the maximum likelihood solution (least-squares). The plots above show that as the model complexity increases, the model provides a closer fit to the data.

The mean squared error decreases as the model complexity increases. This is shown in the figure below. This behaviour heavily biases the maximum liklihood method towards over-fitting. Note that the rate at which the mean square error decreases has plateaued by three degrees of freedom. Conversely, the log marginal likelihood, returned by the Bayesian solution, displays a distinct peak at three degrees of freedom. Beyond three degrees of freedom, model complexity is penalised more heavily than data-fit, leading to deminishing log marginal likelihood.

```python
utils.plot_model_fit(MSE, ML)
```

##Empirical Bayes

When, the model parameters cannot be analytically marginalised out, closed form [parameter estimation](#parameter_estimation) cannot be used. Instead, the posterior on the hyper-parameters can be approximated with a point estimate. This is done by maximising the marginal likelihood of the data with respect to the hyper-parameters. This is approach is called *empirical Bayes*, *type-II maximum likelihood* or the *evidence procedure*. Compared to the [fully Bayesian](#parameter_estimation) solution, empirical Bayes violates the principle that the prior should be nominated before any data are observed. A level of Bayesian inference is sacrificed for tractability.

###Example

Consider the following sigmoid function:

\begin{equation}
    f\!\left(x\right) = \frac{A}{1 + e^{B\left(-x + C\right)} - D}
\end{equation}

where:
- $A$ controls the output scale
- $B$ controls the rate/slope of the output
- $C$ controls $x$-offset of the output
- $D$ controls the $y$-offset of the output

The output of the sigmoid function is clearly a non-linear function of the input parameters. Closed form [parameter estimation](#parameter_estimation) can not be used to find values for the sigmoid parameters. Instead, empirical Bayes can be used.

```python
def sigmoid(x, params=(1., 1., 0., 0.)):
    scale, rate, xoffset, yoffset = params
    return np.abs(scale) / (1. + np.exp(np.abs(rate)*(-x + xoffset))) - yoffset

N = 50
noise = 0.1
X = np.sort(np.random.uniform(-6, 6, N)).reshape((N, 1))
y = sigmoid(X, (2, 2, 1, 0.5)) + np.random.normal(scale=noise, size=(N, 1))

utils.plot(({'x': X, 'y': y, 'linestyle': '', 'marker': '.', 'markersize': 5, 'color': 'k'},))
```

Maximising the marginal likelihood is seldom a convex problem. However, using convex optimisers to locate *locally* optimal hyper-parameters is common. Although these parameters may not be *globally* optimal, this practice is a computationally cheap solution for locating useful parameters. If it is clear local optima are affecting the solution, multi-start optimisation or more complicated methods of setting the hyper-parameters will be required.

In the following block, the method ``empirical_bayes`` uses a convex optimiser to maximise the marginal likelihood with respect to the sigmoid parameters.

```python
# Create Bayesian linear model.
blm = BayesianLinearModel(basis=sigmoid)

# Fit parameters using empirical Bayes.
x0 = np.array([1., 1., 0., 0.])
blm.empirical_bayes(x0, X, y)

M = 100
xs = np.linspace(-6, 6, M).reshape((M, 1))
mu, S = blm.predict(xs, variance=True)
utils.plot(({'x': xs, 'y': mu, 'color': 'r', 'linewidth': 2},
            {'x': xs, 'y': mu - S, 'color': 'r', 'linestyle': '--'},
            {'x': xs, 'y': mu + S, 'color': 'r', 'linestyle': '--'},
            {'x': X, 'y': y, 'linestyle': '', 'marker': '.', 'markersize': 5, 'color': 'k'},))
```