# Statistics and Probability

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import time
np.random.seed(seed=int(time.time()))

# Make plots larger
plt.rcParams['figure.figsize'] = (15, 9)
```

## Probability

The [probability](https://en.wikipedia.org/wiki/Probability) of a specified event is measure of the chance or likelihood that it will occur. Probability is quantified as a number between $0$ and $1$, where, loosely speaking, $0$ indicates impossibility and $1$ indicates certainty. The higher the probability of an event, the more likely it is that the event will occur. A simple example is the tossing of a fair (unbiased) coin. Since the coin is fair, the two outcomes ("heads" and "tails") are both equally probable; the probability of "heads" equals the probability of "tails"; and since no other outcomes are possible, the probability of either "heads" or "tails" is 1/2 (which could also be written as 0.5 or 50%).

One way to determine this is to do repeated trails and count the success rate over the total tries.

Another view would be subjective in nature, in other words an educated guess. An example of subjective probability is asking Boston Red Sox fans,  about the chances of the Red Sox winning the World Series. While there is no counting of trails, fans might still reply in actual percentage terms, based on their opinion of the strength of the team relative to its compeitiion.

One also may have a subjective assessment of something that can be tested via trails. Say somebody believes she is a 90% free throw shooter. Then as she shoots, that subjective assessment can become more objective by decreasing the assessment with each miss and increasing the assessment with each make until it becomes a objective estimate of her true free throw percentage.

* Probability is expressed in numbers between 0 and 1.
* Probability = 0 means the event never happens.
* Probability = 1 means it always happens.
* The total probability of all possible event always sums to 1.

## Events and Outcomes

In [probability theory](https://en.wikipedia.org/wiki/Probability_theory), an outcome is a possible result of an experiment.Each possible outcome of a particular experiment is unique, and different outcomes are mutually exclusive (only one outcome will occur on each trial of the experiment). All of the possible outcomes of an experiment form the elements of a sample space.

* The result of an experiment is called an [outcome](https://en.wikipedia.org/wiki/Outcome_(probability)).
* An event is any particular outcome or group of outcomes.
* A simple event is an event that cannot be broken down further.
* The sample space is the set of all possible simple events.

For the experiment where we flip a coin twice, the four possible outcomes that make up our sample space are (H, T), (T, H), (T, T) and (H, H), where "H" represents a "heads", and "T" represents a "tails". Outcomes should not be confused with events, which are sets (or informally, "groups") of outcomes. For comparison, we could define an event to occur when "at least one 'heads'" is flipped in the experiment - that is, when the outcome contains at least one 'heads'. This event would contains all outcomes in the sample space except the element (T, T).

** Sample Space **

* Coin Toss ={head,tail}
* Two coins S = {HH, HT, TH, TT}
* Inspecting a part ={good,bad}
* Rolling a die S ={1,2,3,4,5,6}

## Probability

The classical definition or interpretation of [probability](https://en.wikipedia.org/wiki/Probability_theory) is identified with the works of Jacob Bernoulli and Pierre-Simon Laplace. As stated in Laplace's Théorie analytique des probabilités,

_The probability of an event is the ratio of the number of cases favorable to it, to the number of all cases possible when nothing leads us to expect that any one of these cases should occur more than any other, which renders them, for us, equally possible._

For example, if the event is "occurrence of an even number when a die is rolled", the probability is given by $\tfrac{3}{6}=\tfrac{1}{2}$, since 3 faces out of the 6 have even numbers and each face has the same probability of appearing.

**Modern definition**

The modern definition starts with a [countable set](https://en.wikipedia.org/wiki/Countable_set) called the [sample space](https://en.wikipedia.org/wiki/Sample_space) , which relates to the set of all _possible outcomes_ in classical sense, denoted by$\Omega$. It is then assumed that for each element $x \in \Omega\,$, an intrinsic "probability" value $f(x)\,$ is attached, which satisfies the following properties:

$$f(x)\in[0,1]\mbox{ for all }x\in \Omega\,;$$

$$\sum_{x\in \Omega} f(x) = 1\,.$$

That is, the probability function $f(x$) lies between zero and one for every value of $x$ in the sample space $\Omega$, and the sum of $f(x$) over all values $x$ in the sample space $\Omega$ is equal to 1. An [event (probability theory](https://en.wikipedia.org/wiki/Event_(probability_theory)) is defined as any [subset](https://en.wikipedia.org/wiki/Subset) $E\,$ of the sample space$\Omega\,$. The _probability_ of the event$E\,$ is defined as

$$P(E)=\sum_{x\in E} f(x)\,.$$

So, the probability of the entire sample space is 1, and the probability of the null event is 0.

The function$f(x)\,$ mapping a point in the sample space to the "probability" value is called a [probability mass function](https://en.wikipedia.org/wiki/Probability_mass_function) abbreviated as _pmf_. The modern definition does not try to answer how probability mass functions are obtained; instead it builds a theory that assumes their existence.

**Probability:** the number of ways that an event can occur divided by the total number of possible outcomes.

#### What's the probability of getting heads in a fair coin flip?

```python
1/2.0
```

    0.5

#### What's the probability of getting an even number on a standard six-sided die?

```python
3/6.0
```

    0.5

#### What's the probability an 'ace' from a standard deck of cards?

A standard deck of 52 playing cards consists of four suits (hearts, spades, diamonds and clubs). Spades and clubs are black while hearts and diamonds are red. Each suit contains 13 cards, each of a different rank: an Ace (which in many games functions as both a low card and a high card),
cards numbered 2 through 10, a Jack, a Queen and a King.

```python
4/52.0
```

    0.07692307692307693

## Odds

The odds for an event is the ratio of the number of ways the event can occur compared to the number of ways it does not occur.

For example, using the same events as above, the odds for:
drawing a red card from a standard deck of cards is 1:1; and
drawing a club from that deck is 1:3.

#### What's the odds of a fair coin flip?

```python
1/1.0
```

    1.0

#### What's the odds of getting an even number on a standard six-sided die?

```python
3/3.0
```

    1.0

#### What's the odds of an 'ace' from a standard deck of cards

```python
4/48.0
```

    0.08333333333333333

## The Axioms of Probability

### First axiom - The probability of an event is a non-negative real number:
$$
P(E)\in\mathbb{R}, P(E)\geq 0 \qquad \forall E\in F
$$
where $F$ is the event space

### Second axiom -  unit measure:

The probability that some elementary event in the entire sample space will occur is 1.

$$
P(\Omega) = 1.
$$

### Third axiom - the assumption of $\sigma$-additivity:

Any countable sequence of disjoint (synonymous with mutually exclusive) events $E_1, E_2, ...$ satisfies

$$
P\left(\bigcup_{i = 1}^\infty E_i\right) = \sum_{i=1}^\infty P(E_i).
$$

The total probability of all possible event always sums to 1.

### Consequences of these axioms

The probability of the empty set:
$$
P(\varnothing)=0.
$$

Monotonicity
$$
\quad\text{if}\quad A\subseteq B\quad\text{then}\quad P(A)\leq P(B).
$$

The numeric bound between 0 and 1:

$$
0\leq P(E)\leq 1\qquad \forall E\in F.
$$

*Probability is expressed in numbers between 0 and 1.*

Probabilty of a certain event is 1:

$$
P(True) = 1
$$

Probability = 1 means it always happens.

Probabilty of an impossible event is 0:

$$
P(False) = 0
$$

Probability = 0 means the event never happens.

Probabilty of A or B:

$$
P(A \quad or \quad B) = P(A) + P(B) - P(A \quad and \quad B)
$$

or

$$
P(A \cup B) = P(A) + P(A) - P(A \cap B)
$$

Probabilty of not A:

$$
P(not  \quad A) = 1- P(A)
$$

#### If you pull a random card from a deck of playing cards, what is the probability it is a heart?

```python
13/52.0
```

    0.25

#### If you pull a random card from a deck of playing cards, what is the probability it is not a heart?

```python
1-(13/52.0)
```

    0.75

## Independent Events

Events A and B are independent events if the probability of Event B occurring is the same whether or not Event A occurs.

Are the following independent events?

* A fair coin is tossed two times. The two events are (1) first toss is a head and (2) second toss is a head.
* You draw a card from a deck two times without replacement. The two events are (1) first draw is a heart and (2) second toss is a draw is a heart.

## Probabilty of A or B

The probability of either A or B occurring (or both) is

$$
P(A \quad or \quad B) = P(A) + P(B) - P(A \quad and \quad B)
$$

#### Suppose we draw one card from a standard deck. What is the probability that we get a Queen or a King?

$$
P(King \quad or \quad Queen) = P(King) + P(Queen) - P(King \quad and \quad Queen)
$$

```python
4/52.0+4/52.0-0
```

    0.15384615384615385

## Conditional Probability

In probability theory, a [conditional probability](https://en.wikipedia.org/wiki/Conditional_probability) measures the probability of an event given that another event has occurred. That is,  "the conditional probability of A given B."

 the conditional probability of A given B is defined as the quotient of the probability of the joint of events A and B, and the probability of B:

$$
P(A|B) = \frac{P(A \cap B)}{P(B)}
$$

This may be visualized using a Venn diagram.

*$P(A \cap B)$*

### Corollary of Conditional Probability is The Chain Rule

If we multiply both sides by $P(B)$ then

$$
P(A|B) = \frac{P(A \cap B)}{P(B)}
$$

becomes

$$
P(A|B) P(B) = P(A \cap B)
$$

If Events A and B are not independent, then

$$
P(A \quad and \quad B) = P(A) · P(B | A)
$$

## Statistical independence

Events A and B are defined to be statistically independent if:

$$
\begin{align}
             P(A \cap B) &= P(A) P(B) \\
  \Leftrightarrow P(A|B) &= P(A) \\
  \Leftrightarrow P(B|A) &= P(B)
\end{align}
$$

That is, the occurrence of A does not affect the probability of B, and vice versa

Probabilty of A or B for independent events $P(A and B) is 0$:

$$
P(A \quad or \quad B) = P(A) + P(A)
$$

#### What is the probability that two cards drawn at random from a deck of playing cards will both be aces?

```python
(4/52.0)*(3/51.0)
```

    0.004524886877828055

#### If you pull 2 cards out of a deck, what is the probability that both are hearts?

If Events A and B are not independent, then

$$
P(A \quad and \quad B) = P(A) · P(B | A)
$$

$$
P(Heart \quad and \quad Heart) = P(Heart) · P(Heart | Heart)
$$

```python
(13/52.0)*(12/51.0)
```

    0.058823529411764705

## Bayes Rule

[Bayes' theorem](https://en.wikipedia.org/wiki/Bayes%27_theorem) (alternatively Bayes' law or Bayes' rule) describes the probability of an event, given prior events. That is, a conditional probability.

$$
P(A|B) = \frac{P(A)\, P(B | A)}{P(B)},
$$

where A and B are events.

* P(A) and P(B) are the independent probabilities of A and B.
* P(A | B), a conditional probability, is the probability of observing event A given that B is true.
* P(B | A), is the probability of observing event B given that A is true.

## Bayesian inference

[Bayesian inference](https://en.wikipedia.org/wiki/Bayesian_inference) is a method of statistical inference in which Bayes' theorem is used to update the probability for a hypothesis as evidence. Bayesian inference derives the posterior probability as a consequence of two antecedents, a prior probability and a "likelihood function" derived from a statistical model for the observed data.

Bayesian inference computes the posterior probability according to Bayes' theorem:

$$
P(H\mid E) = \frac{P(E\mid H) \cdot P(H)}{P(E)}
$$

where,

$P(H\mid E)$ the posterior probability, denotes a conditional probability of $\textstyle H$ (the hypothesis) whose probability may be affected by the evidence $\textstyle E$.

$\textstyle P(H)$, the prior probability, is an estimate of the probability that a hypothesis is true, before observing the current evidence.

$\textstyle P(E\mid H)$ is the probability of observing $\textstyle E$ given $\textstyle H$. It indicates the compatibility of the evidence with the given hypothesis.

$\textstyle P(E)$ is sometimes termed the marginal likelihood or "model evidence". This factor is the same for all possible hypotheses being considered.

Note that Bayes' rule can also be written as follows:

$$
P(H\mid E) = \frac{P(E\mid H)}{P(E)} \cdot P(H)
$$

where the factor $\textstyle \frac{P(E\mid H)}{P(E)}$ represents the impact of $E$ on the probability of $H$.

# Bayesian probability example

Suppose a certain disease has an incidence rate of 0.01% (that is, it afflicts 0.01% of the population).  A test has been devised to detect this disease.  The test does not produce false negatives (that is, anyone who has the disease will test positive for it), but the false positive rate is 1% (that is, about 1% of people who take the test will test positive, even though they do not have the disease).  Suppose a randomly selected person takes the test and tests positive.  What is the probability that this person actually has the disease?

Bayes theorem would ask the question, what is the probability of disease given a postive result, or $P(disease\mid positive))$.

What do we know?

$P(positive\mid disease)=1$ (i.e. The test does not produce false negatives.)
$P(disease)=0.0001$ (i.e.  1/10,000 have the disease)
$P(positive\mid no disease)=0.01$ (i.e. he false positive rate is 1%. This means 1% of people who take the test will test positive, even though they do not have the disease)

Bayes’ Theorem

$$
P(A|B) = \frac{P(A)\, P(B | A)}{P(B)},
$$

which can be rewritten as

$$
P(A|B) = \frac{P(A)\, P(B | A)}{P(A)P(B|A)+P(\bar{A})P(B|\bar{A})},
$$

which in our example is
$$
P(disease|positive) = \frac{P(disease)\, P(positive | disease)}{P(disease)P(positive|disease)+P(no \quad  disease)P(positive|no \quad disease)},
$$

plugging in the numbers gives

$$
P(disease|positive)= \frac{(0.0001)\, (1)}{(0.0001)(1)+(0.9999)(0.01)}, \approx 0.01
$$

So even though the test is 99% accurate, of all people who test positive, over 99% do not have the disease.

```python
(0.0001*1)/((0.0001*1)+(0.9999*0.01))
```

    0.009901970492127933

## Bayesians versus Frequentists

[Frequentist inference](https://en.wikipedia.org/wiki/Frequentist_inference) or frequentist statistics is a scheme for making statistical inference based on the frequency or proportion of the data. This effectively requires that conclusions should only be drawn with a set of repetitions.

Frequentists will only generate statistical inference given a large enough set of repetitions. In contrast, a Bayesian approach to inference does allow probabilities to be associated with unknown parameters.

*Count Von Count*
- from https://en.wikipedia.org/wiki/File:Count_von_Count_kneeling.png

While "probabilities" are involved in both approaches to inference, frequentist probability is essentially equivelent to counting. The Bayesian approach allows these estimates of probabilities to be based upon counting but also allows for subjective estimates (i.e. guesses) of prior probabilities.

Bayesian probability, also called evidential probability, or subjectivist probability, can be assigned to any statement whatsoever, even when no random process is involved. Evidential probabilities are considered to be degrees of belief, and a Bayesian can even use an un-informative prior (also called a non-informative or Jeffreys prior).

In Bayesian probability, the [Jeffreys prior](https://en.wikipedia.org/wiki/Jeffreys_prior), named after Sir Harold Jeffreys, is a non-informative (objective) prior distribution for a parameter space.  The crucial idea behind the Jeffreys prior is the Jeffreys posterior. This posterior aims to reflect as best as possible the information about the parameters brought by the data, in effect  "representing ignorance" about the prior. This is sometimes called the "principle of indifference." Jeffreys prior is proportional to the square root of the determinant of the Fisher information:

$$
p\left(\vec\theta\right) \propto \sqrt{\det \mathcal{I}\left(\vec\theta\right)}.\,
$$

It has the key feature that it is invariant under reparameterization of the parameter vector $\vec\theta.$

At its essence the Bayesian can be vague or subjective about an inital guess at a prior probability. and the the posterior probability be updated data point by data point. A Bayesian defines a "probability" in the same way that many non-statisticians do - namely an indication of the plausibility or belief of a proposition.

A Frequentist is someone that believes probabilities represent long run frequencies with which events occur; he or she will have a model (e.g. Guassian, uniform, etc.) of how the sample popluation was generated. The observed counts are considered a random sample the estimate the true parameters of the model.

It is important to note that most Frequentist methods have a Bayesian equivalent (that is, they give the same results) when there are enough repeated trails. They converge the the same result given enough data.

## Probability distributions

* We use probability distributions because they model data in real world.
* They allow us to calculate what to expect and therefore understand what is unusual.
* They also provide insight in to the process in which real world data may have been generated.
* Many machine learning algorithms have assumptions based on certain probability distributions.

_Cumulative distribution function_

A probability distribution Pr on the real line is determined by the probability of a scalar random variable X being in a half-open interval (-$\infty$, x], the probability distribution is completely characterized by its cumulative distribution function:

$$
 F(x) = \Pr[X \leq x] \quad \forall \quad x \in R .
$$

## Uniform Distribution

$$
X \equiv U[a,b]
$$

$$
 f(x) = \frac{1}{b-a} \quad for \quad a \lt x \lt b
$$

$$
 f(x) = 0 \quad for \quad a \leq x  \quad or  \quad \geq b
$$

$$
 F(x) = \frac{x-a}{b-a} \quad for \quad a \leq x \lt b
$$

$$
F(x) = 0 \quad for \quad x  \lt a  \quad
 F(x) = 1 \quad for \quad x  \geq b
$$

_Continuous Uniform Distribution_

In probability theory and statistics, the continuous uniform distribution or rectangular distribution is a family of symmetric probability distributions such that for each member of the family, all intervals of the same length on the distribution's support are equally probable.

- from [Uniform distribution (continuous  Wikipedia)](https://en.wikipedia.org/wiki/Uniform_distribution_(continuous))

_Discrete Uniform Distribution_

In probability theory and statistics, the discrete uniform distribution is a symmetric probability distribution whereby a finite number of values are equally likely to be observed; every one of n values has equal probability 1/n. Another way of saying "discrete uniform distribution" would be "a known, finite number of outcomes equally likely to happen".

- from [Uniform distribution (discrete)  Wikipedia)](https://en.wikipedia.org/wiki/Uniform_distribution_(discrete))

```python
#------------------------------------------------------------
# Define the distribution parameters to be plotted
W_values = [1.0, 3.0, 5.0]
linestyles = ['-', '--', ':']
mu = 0
x = np.linspace(-4, 4, 1000)

#------------------------------------------------------------
# plot the distributions
fig, ax = plt.subplots(figsize=(10, 5))

for W, ls in zip(W_values, linestyles):
    left = mu - 0.5 * W
    dist = stats.uniform(left, W)

    plt.plot(x, dist.pdf(x), ls=ls, c='black',
             label=r'$\mu=%i,\ W=%i$' % (mu, W))

plt.xlim(-4, 4)
plt.ylim(0, 1.2)

plt.xlabel('$x$')
plt.ylabel(r'$p(x|\mu, W)$')
plt.title('Uniform Distribution')

plt.legend()
plt.show()

# Adapted from http://www.astroml.org/book_figures/chapter3/fig_uniform_distribution.html

```

## Normal Distribution

In probability theory, the normal (or Gaussian) distribution is a very common continuous probability distribution. The normal distribution is remarkably useful because of the central limit theorem. In its most general form, under mild conditions, it states that averages of random variables independently drawn from independent distributions are normally distributed. Physical quantities that are expected to be the sum of many independent processes (such as measurement errors) often have distributions that are nearly normal.

- from [Normal Distribution - Wikipedia)](https://en.wikipedia.org/wiki/Normal_distribution)

$$
X \sim \quad N(\mu, \sigma^2)
$$

$$
 f(x) = \frac{1}{\sigma \sqrt {2\pi }} e^{-\frac{( x - \mu)^2}{2\sigma^2}} \quad
$$

Normal cumulative distribution function

_Properties of normal distribution_

- symmetrical, unimodal, and bell-shaped
- on average, the error component will equal zero, the error above and below the mean will cancel out
- Z-Score is a statistical measurement is (above/below) the mean of the data
- important characteristics about z scores:
  1. mean of z scores is 0
  2. standard deviation of a standardized variable is always 1
  3. the linear transformation does not change the _form_ of the distribution

The normal (or Gaussian) distribution was discovered in 1733 by Abraham de Moivre as an approximation to the binomial distribution when the number of trails is large.

- from [Abraham de Moivre - Wikipedia)](https://en.wikipedia.org/wiki/Abraham_de_Moivre)

The Gaussian distribution was derived in 1809 by Carl Friedrich Gauss.

- from [Carl Friedrich Gauss - Wikipedia)](https://en.wikipedia.org/wiki/Carl_Friedrich_Gauss)

Importance lies in the Central Limit Theorem, which states that the sum of a large number of independent random variables (binomial, Poisson, etc.) will approximate a normal distribution

## Central Limit Theorem

In probability theory, the central limit theorem (CLT) states that, given certain conditions, the arithmetic mean of a sufficiently large number of iterates of independent random variables, each with a well-defined expected value and well-defined variance, will be approximately normally distributed, regardless of the underlying distribution. The central limit theorem has a number of variants. In its common form, the random variables must be identically distributed.

- from [Central Limit Theorem - Wikipedia)](https://en.wikipedia.org/wiki/Central_limit_theorem)

The Central Limit Theorem tells us that when the sample size is large the average $\bar{Y}$ of a random sample follows a normal distribution centered at the population average $\mu_Y$ and with standard deviation equal to the population standard deviation $\sigma_Y$, divided by the square root of the sample size $N$.

This means that if we subtract a constant from a random variable, the mean of the new random variable shifts by that constant. If $X$ is a random variable with mean $\mu$ and $a$ is a constant, the mean of $X - a$ is $\mu-a$.

This property also holds for the spread, if $X$ is a random variable with mean $\mu$ and SD $\sigma$, and $a$ is a constant, then the mean and SD of $aX$ are $a \mu$ and $\|a\| \sigma$ respectively.
This implies that if we take many samples of size $N$ then the quantity

$$
\frac{\bar{Y} - \mu}{\sigma_Y/\sqrt{N}}
$$

is approximated with a normal distribution centered at 0 and with standard deviation 1.

### Central Limit Theorem in python

```python
x = np.arange(0, 7, 0.1)
dist = stats.uniform(1,5)
plt.plot(x, dist.pdf(x), lw=2)
```

```python
print ("Mean:", dist.mean())
print ("Std Dev:", dist.std())
```

    Mean: 3.5
    Std Dev: 1.44337567297

```python
# Take a random sample of size 33
sample = dist.rvs(33)
print ("Sample Mean:", np.mean(sample))
print ("Sample Std Dev:", np.std(sample))
plt.hist(sample, bins=10)
plt.show()
```

```python
# Take a random sample of size 3333
sample = dist.rvs(3333)
print ("Sample Mean:", np.mean(sample))
print ("Sample Std Dev:", np.std(sample))
plt.hist(sample, bins=10)
plt.show()
```

```python
means = []
devs = []
samples = 10
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
```

```python
means = []
devs = []
samples = 100
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
```

```python
means = []
devs = []
samples = 1000
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
```

```python
means = []
devs = []
samples = 10000
for i in range(samples):
    sample = dist.rvs(33)
    means.append(np.mean(sample))
    devs.append(np.std(sample))
plt.hist(means, bins=20)
plt.title("Sample Means")
plt.show()

print ("Mean of Means:", np.mean(means))
print ("SD of Means:", np.std(means))
```

```python
sns.distplot(dist.rvs(1000))
sns.plt.show()

```

```python
sns.distplot(means)
sns.plt.show()
```

## Empirical distribution functions (ECDF)

In statistics, an [empirical distribution function](https://en.wikipedia.org/wiki/Empirical_distribution_function) is the distribution function associated with the empirical measure of a sample. This cumulative distribution function is a step function that jumps up by 1/n at each of the n data points. Its value at any specified value of the measured variable is the fraction of observations of the measured variable that are less than or equal to the specified value.
The empirical distribution function is an estimate of the cumulative distribution function that generated the points in the sample. It converges with probability 1 to that underlying distribution, according to the Glivenko–Cantelli theorem. A number of results exist to quantify the rate of convergence of the empirical distribution function to the underlying cumulative distribution function.

```python
from statsmodels.distributions.empirical_distribution import ECDF

sample = np.random.uniform(0, 1, 10000)
ecdf = ECDF(sample)

x = np.linspace(min(sample), max(sample))
y = ecdf(x)
plt.step(x, y)
```

```python
mu, sigma = 5, 3 # mean and standard deviation
sample = np.random.normal(mu, sigma, 10000)
ecdf = ECDF(sample)

x = np.linspace(min(sample), max(sample))
y = ecdf(x)
plt.step(x, y)
```

## Covariance

[Covariance](https://en.wikipedia.org/wiki/Covariance) (How two variables depend on each other)  is a measure of the joint variability of two random variables. If the greater values of one variable mainly correspond with the greater values of the other variable, and the same holds for the lesser values, i.e., the variables tend to show similar behavior, the covariance is positive. In the opposite case, when the greater values of one variable mainly correspond to the lesser values of the other, i.e., the variables tend to show opposite behavior, the covariance is negative. The sign of the covariance therefore shows the tendency in the linear relationship between the variables. The magnitude of the covariance is not easy to interpret. The normalized version of the covariance, the [correlation coefficient](https://en.wikipedia.org/wiki/Pearson_correlation_coefficient), however, shows by its magnitude the strength of the linear relation.

The covariance between two jointly distributed real-valued random variable's $X$ and $Y$ with finite second moment's is defined as  the expected product of their deviations from their individual expected values:

$$
\operatorname{cov}(X,Y) = \operatorname{E}{\big[(X - \operatorname{E}[X])(Y - \operatorname{E}[Y])\big]}
$$

## Hypothesis testing

A hypothesis is proposed for the statistical relationship between the two data sets, and this is compared as an alternative to an idealized null hypothesis that proposes no relationship between two data sets. The comparison is deemed statistically significant if the relationship between the data sets would be an unlikely realization of the null hypothesis according to a threshold probability—the significance level. Hypothesis tests are used in determining what outcomes of a study would lead to a rejection of the null hypothesis for a pre-specified level of significance.

[Hypothesis tests](https://en.wikipedia.org/wiki/Statistical_hypothesis_testing) assume the thing you want to disprove, and then to look for evidence that the assumption is wrong. In this case, we assume that there is no difference between $\bar{x_1}$ and $\bar{x_2}$ (i.e. the mean of one distribution versus another) This is called the *null hypothesis* and is stated as

$$H_0: \bar{x_1} = \bar{x_2}$$

If $\bar{x_1}$ is very different from $\bar{x_2}$ we conclude that the null hypothesis is incorrect and that the evidence suggests there really is a difference between $\bar{x_1}$ and $\bar{x_2}$.

There are many hypothesis tests that can be used to test whether there is a difference between $\bar{x_1}$ and $\bar{x_2}$:

* Student’s T-Tests
* One-Sample T-Test
* Two-Sample T-Test
* Paired T-Test
* Wilcoxon Rank-Sum Test
* Analysis of Variance (ANOVA)
* Kruskal-Wallis Test

We will discuss these more in the module on hypothesis testing.

## P-value

To determine how big the difference between  $\bar{x_1}$ and  $\bar{x_2}$ must be before we would reject the null hypothesis, we calculate the probability of obtaining a value of $\bar{x_2}$ as large as we have calculated if the null hypothesis were true. This probability is known as the *P-value*.

In statistics, the p-value is a function of the observed sample results (a statistic) that is used for testing a statistical hypothesis. Before the test is performed, a threshold value is chosen, called the significance level of the test, traditionally 5% or 1% and denoted as $\alpha$.

If the p-value is equal to or smaller than the significance level ($\alpha$), it suggests that the observed data are inconsistent with the assumption that the null hypothesis is true and thus that hypothesis must be rejected (but this does not automatically mean the alternative hypothesis can be accepted as true). When the p-value is calculated correctly, such a test is guaranteed to control the Type I error rate to be no greater than $\alpha$.

from [P-value](https://en.wikipedia.org/wiki/P-value)

A is **p-value** is the probability of observing a test statistic equally or more extreme than the one you observed, assuming the hypothesis you are testing is true.

## Confidence intervals

In statistics, a confidence interval (CI) is a type of interval estimate of a population parameter. It provides an interval estimate for lower or upper confidence bounds. For $\beta_1$, usually referred to as a *confidence interval* and is typically +/-0.5% (a 99% confidence interval),+/-1% (a 98% confidence interval),+/-2.5% (a 95% confidence interval) or +/-5% (a 90% confidence interval). The lower and upper confidence bounds need not be equal, and they can be any number such that the confidence interval not exceed 100%.

## The t-distribution

In probability and statistics, Student's t-distribution (or simply the t-distribution) is any member of a family of continuous probability distributions that arises when estimating the mean of a normally distributed population in situations where the sample size is small and population standard deviation is unknown. Whereas a normal distribution describes a full population, t-distributions describe samples drawn from a full population; accordingly, the t-distribution for each sample size is different, and the larger the sample, the more the distribution resembles a normal distribution.
The t-distribution plays a role in a number of widely used statistical analyses, including the Student's t-test for assessing the statistical significance of the difference between two sample means, the construction of confidence intervals for the difference between two population means, and in linear regression analysis. The Student's t-distribution also arises in the Bayesian analysis of data from a normal family.

- from [The t-distribution - Wikipedia)](https://en.wikipedia.org/wiki/Student%27s_t-distribution)

When the CLT does not apply (i.e. as the number of samples is large), there is another option that does not rely on large samples When a the original population from which a random variable, say $Y$, is sampled is normally distributed with mean 0 then we can calculate the distribution of

number of variants. In its common form, the random variables must be identically distributed.

$$
\sqrt{N} \frac{\bar{Y}}{s_Y}
$$

Normal cumulative distribution function

## Which of distributions below are significantly different?

```python
# Plot two normal distributions
domain = np.arange(-22, 33, 0.1)
values = stats.norm(3.3, 5.5).pdf(domain)
plt.plot(domain, values, color='r', linewidth=2)
plt.fill_between(domain, 0, values, color='#ffb6c1', alpha=0.3)
values = stats.norm(4.4, 6.6).pdf(domain)
plt.plot(domain, values, color='b', linewidth=2)
plt.fill_between(domain, 0, values, color='#89cff0', alpha=0.3)
plt.ylabel("Probability")
plt.title("Normal Distributions")
plt.show()
```

```python
# Plot two normal distributions
domain = np.arange(1, 15, 0.1)
values = stats.norm(5.5, 1.1).pdf(domain)
plt.plot(domain, values, color='r', linewidth=2)
plt.fill_between(domain, 0, values, color='#ffb6c1', alpha=0.3)
values = stats.norm(9.9, 1.1).pdf(domain)
plt.plot(domain, values, color='b', linewidth=2)
plt.fill_between(domain, 0, values, color='#89cff0', alpha=0.3)
plt.ylabel("Probability")
plt.title("Normal Distributions")
plt.show()
```

## Statistical hypothesis tests

There are a few good statistical tests for hypothesis testing:
* [ANOVA](https://en.wikipedia.org/wiki/Analysis_of_variance)
* [Welch's t-test](https://en.wikipedia.org/wiki/Welch's_t-test)
* [Mann-Whitney test](https://en.wikipedia.org/wiki/Mann%E2%80%93Whitney_U_test)

Each test makes various assumptions:

* ANOVA assumes normal distributions and equal variances in the two data sets
* The Welch t-test assumes normal distributions but not necessarily equal variances, and accounts for small sample sizes better
* The Mann-Whitney test assumes nothing about the distributions but requires at least 20 data points in each set, and produces a weaker p-value

Typically you need to choose the most appropriate test. Tests that make more assumptions are more discriminating (stronger p-values) but can be misleading on data sets that don't satisfy the assumptions.

All of these tests are available in the `scipy` library, a stats library for python:
* [ANOVA](http://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.stats.f_oneway.html)
* [Welch's t-test](http://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.ttest_ind.html)
* [Mann-Whitney](http://docs.scipy.org/doc/scipy-0.17.0/reference/generated/scipy.stats.mannwhitneyu.html)

```python
# Generate some rnd_data_ and randomize

rnd_data_1 = []
rnd_data_1.extend(stats.norm(3.3, 5.5).rvs(3333))
np.random.shuffle(rnd_data_1)

rnd_data_2 = []
rnd_data_2.extend(stats.norm(4.4, 6.6).rvs(3333))
np.random.shuffle(rnd_data_2)

rnd_data_3 = []
rnd_data_3.extend(stats.norm(5.5, 1.1).rvs(3333))
np.random.shuffle(rnd_data_3)

rnd_data_4 = []
rnd_data_4.extend(stats.norm(9.9, 1.1).rvs(3333))
np.random.shuffle(rnd_data_4)

rnd_data_5 = []
rnd_data_5.extend(stats.norm(9.9, 1.1).rvs(3333))
np.random.shuffle(rnd_data_5)

# Make a rnd_data_ frame
rnd_data = pd.DataFrame()
rnd_data["A"] = rnd_data_1
rnd_data["B"] = rnd_data_2
rnd_data["C"] = rnd_data_3
rnd_data["D"] = rnd_data_4
rnd_data["E"] = rnd_data_5
rnd_data.head()
```

<div>
<style>
    .dataframe thead tr:only-child th {
        text-align: right;
    }

    .dataframe thead th {
        text-align: left;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>A</th>
      <th>B</th>
      <th>C</th>
      <th>D</th>
      <th>E</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>9.494269</td>
      <td>12.822614</td>
      <td>4.315905</td>
      <td>10.742011</td>
      <td>10.563410</td>
    </tr>
    <tr>
      <th>1</th>
      <td>0.007479</td>
      <td>7.672812</td>
      <td>4.648618</td>
      <td>11.277304</td>
      <td>11.986510</td>
    </tr>
    <tr>
      <th>2</th>
      <td>7.508426</td>
      <td>1.235893</td>
      <td>4.428931</td>
      <td>8.704675</td>
      <td>9.932005</td>
    </tr>
    <tr>
      <th>3</th>
      <td>6.952109</td>
      <td>-1.160678</td>
      <td>3.416166</td>
      <td>8.763451</td>
      <td>10.368277</td>
    </tr>
    <tr>
      <th>4</th>
      <td>10.939092</td>
      <td>6.085958</td>
      <td>4.606536</td>
      <td>9.837315</td>
      <td>10.366129</td>
    </tr>
  </tbody>
</table>
</div>

```python
rnd_data.hist()
```

```python
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["A"], rnd_data["B"])
print (pvalue)
print (statistic)
```

    3.41323836964e-07
    26.0532751647

```python
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["C"], rnd_data["D"])
print (pvalue)
print (statistic)
```

    0.0
    26604.3107172

```python
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["A"], rnd_data["A"])
print (pvalue)
print (statistic)
```

    nan
    -2.16273023326e-30

```python
# To use ANOVA
statistic, pvalue = stats.f_oneway(rnd_data["D"], rnd_data["E"])
print (pvalue)
print (statistic)
```

    0.41735786338
    0.657824869577

```python
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["A"], rnd_data["B"])
print (pvalue)
print (statistic)
```

    3.41323836965e-07
    -5.10424089995

```python
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["C"], rnd_data["D"])
print (pvalue)
print (statistic)
```

    0.0
    -163.108279119

```python
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["B"], rnd_data["B"])
print (pvalue)
print (statistic)
```

    1.0
    0.0

```python
# to use Welch
statistic, pvalue = stats.ttest_ind(rnd_data["D"], rnd_data["E"])
print (pvalue)
print (statistic)
```

    0.41735786338
    0.811064035436

```python
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["A"], rnd_data["B"])
print (pvalue)
print (statistic)
```

    7.78355515042e-07
    5177049.0

```python
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["C"], rnd_data["D"])
print (pvalue)
print (statistic)
```

    0.0
    28081.0

```python
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["C"], rnd_data["C"])
print (pvalue)
print (statistic)
```

    0.499997460956
    5554444.5

```python
# Use the Mann-Whitney test on our data
# look up the function in scipy from the link above
# stats.mannwhitneyu
statistic, pvalue = stats.mannwhitneyu(rnd_data["D"], rnd_data["E"])
print (pvalue)
print (statistic)
```

    0.138922092626
    5469191.0

```python
mw = stats.mannwhitneyu(rnd_data["A"], rnd_data["B"])
print (mw)
print (mw.statistic)
```

    MannwhitneyuResult(statistic=5177049.0, pvalue=7.7835551504176807e-07)
    5177049.0

## Inferential Statistical Tests

* Chi Square • compares observed frequencies to expected frequencies.
* t-Test • looks at differences between two groups on some variable of interest.
* Welch-Test • looks at differences between two groups on some variable of interest.
* Mann-Whitney test • looks at differences between two groups on some variable of interest.
* ANOVA • tests the significance of group differences between two or more groups. (Only determines that there is a difference between groups, but doesn’t tell which is different.)

##  One Way ANOVA Vs Two Way ANOVA

One way ANOVA takes only one factor (i.e. independent variable). Two way ANOVA assesses two factors concurrently.

|   | ONE WAY ANOVA    | TWO WAY ANOVA    |
|---|------------------|------------------|
|   | One way ANOVA is a hypothesis test.  | Two way ANOVA is a statistical technique assessing the interaction between factors. |
| Independent Variables  | One  | Two  |
|Number of Observations   | Need not to be same in each group. |  Need to be equal in each group.  |

## Resampling

* Sampling with and without replacement
* Bootstrap (using sampling with replacement)
* Jackknife (using subsets)
* Permutation resampling (switching labels)

In statistics, [resampling](https://en.wikipedia.org/wiki/Resampling_(statistics)) is any of a variety of methods for doing one of the following:

* Estimating the precision of sample statistics (medians, variances, percentiles) by using subsets of available data (jackknifing) or drawing randomly with replacement from a set of data points (bootstrapping)
* Exchanging labels on data points when performing significance tests (permutation tests, also called exact tests, randomization tests, or re-randomization tests)
* Validating models by using random subsets (bootstrapping, cross validation)

**Bootstrap**

Bootstrapping is a statistical method for estimating the sampling distribution of an estimator by sampling with replacement from the original sample, most often with the purpose of deriving robust estimates of standard errors and confidence intervals of a population parameter like a mean, median, proportion, odds ratio, correlation coefficient or regression coefficient.

**Jackknife**

Jackknifing, which is similar to bootstrapping, is used in statistical inference to estimate the bias and standard error (variance) of a statistic, when a random sample of observations is used to calculate it.

The basic idea behind the jackknife variance estimator lies in systematically recomputing the statistic estimate, leaving out one or more observations at a time from the sample set. From this new set of replicates of the statistic, an estimate for the bias and an estimate for the variance of the statistic can be calculated.

**Bootstrap**

~~~~
statistics = []
for i in bootstraps:
	sample = select_sample_with_replacement(data)
	stat = calculate_statistic(sample)
	statistics.append(stat)

~~~~

**Jackknife**

~~~~
statistics = []
for i in jackknifes:
	sample = select_sample_without_replacement(data)
	stat = calculate_statistic(sample)
	statistics.append(stat)

~~~~

Last update September 13, 2017

Prof. Nik Brown's Notes/B01. Statistics and Probability/NBB_Statistics_and_Probability.ipynb


```python
import sympy as sp
X=[1,2,3,4,5,6]
Y=[6,5,4,3,2,1]
```

```python

```

Badamy szybkość wunkcji generującej symboliczny wzór. Niektóre Państwa funkcje zostały zmodyfikowane, żeby nie sprawdzały wartości opcjonalnego argumentu oraz nie weryfikowały długości list.

```python
#Daniel
def interpolacja(X,Y,pt):
  x=sp.symbols('x')
  temp=sp.prod([x-el for el in X])
  tabelaTemp=[temp/(x-X[i]) for i in range(len(X))]
  wzor=sum([Y[i]*tabelaTemp[i]/(tabelaTemp[i].subs(x,X[i])) for i in range(len(X))])
  return wzor
%timeit interpolacja(X,Y,0)

```

    352 µs ± 2.03 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
# Anna Cabaj
from sympy import *
def lagrange2 (X,Y,x):
  x=symbols('x')
  f=symbols('f', cls=Function)
  f=0
  for k in range(len(Y)):
    lk=Y[k]
    for n in range(len(X)):
      if n==k:
        continue
      lk=lk*(x-X[n])/(X[k]-X[n])
    f=f+lk
  return f
%timeit lagrange2(X,Y,0)
```

    458 µs ± 26.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
# Aleksandra
def Lagringo (X,Y,x, symbol):
  w=0
  if symbol:
    sym = ''
  for i in range(len(X)):
    if i>0:
      if symbol:
        sym +='+'
    iloczyn = 1
    if symbol:
      sym += str(Y[i])
    for j in range (len(Y)):
      if i != j:
        iloczyn *= (x-X[j])/(X[i]-X[j])
        if symbol:
          sym += "*(x-"+ str(X[j])+ ")/("+ str(X[i])+ "-"+ str(X[j])+ ")"
    w += Y[i]*iloczyn
  if symbol:
    return w, sym
  else:
    return w

%timeit Lagringo(X,Y,0,True)
```

    64.3 µs ± 6.5 µs per loop (mean ± std. dev. of 7 runs, 10000 loops each)

```python
#Klaudia
def wielomianp(X,Y,x):
  h=symbols('h')
  f=Function('f')
  f=0
  for i in range(len(Y)):
    il=Y[i]
    for j in range(len(X)):
      if j==i:
        continue
      il=il*(h-X[j])/(X[i]-X[j])
    f=f+il
  return f

%timeit wielomianp(X,Y,0)
```

    517 µs ± 105 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
#Wiktoria
def Interpolacja_symbol(X,Y):
  x=symbols('x')
  n=0
  for i in range (len(X)):
    a = 1
    for j in range (len(X)):
      if j != i:
        a=a*((x-X[j]) /(X[i]-X[j]))
    n += a* Y[i]
  return n
%timeit Interpolacja_symbol(X,Y)
```

    489 µs ± 41.6 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
#Wojtek
from functools import reduce
from operator import mul
def interpolate(X, Y, x):
  a = symbols("x")
  L = lambda x_i: lambda x_j: (a-x_j)/(x_i-x_j)
  w = sum(y*reduce(mul, map(L(X[i]), X[:i]+X[i+1:])) for i,y in enumerate(Y))
  return w

%timeit interpolate(X,Y,0)
```

    463 µs ± 6.55 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
#Marek
def lagranż(a,b,c,d=0): #a,b
  x=symbols('x')
  value=Function('f')
  value=0
  for i in range(len(b)):
    value_1=b[i]
    for j in range(len(a)):
      if i==j: continue
      value_1=value_1*(x-a[j])/(a[i]-a[j])
    value=value+value_1
  return value

%timeit lagranż(X,Y,0)
```

    530 µs ± 51 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
#Szymon
def lagrange(X, Y, x):
  x_ = symbols('x')
  result = 0; lenght = len(X)
  for i in range(lenght):
    multi = 1
    for j in range(lenght):
      if j != i:
        multi *= (x_ - X[j])/(X[i] - X[j])
    result += Y[i] * multi
  return result

%timeit lagrange(X,Y,0)
```

    568 µs ± 64.4 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
#Anna Zgrzebna
def wiel_inter_Lagrange(X,Y,x):
    x1 = symbols('x')
    sum = 0
    n = len(X)
    for i in range (n):
        p = 1
        for j in range (n):
            if j != i:
                p = p* ((x1-X[j])/(X[i]-X[j]))
        sum = sum + Y[i]*p
    return sum

%timeit wiel_inter_Lagrange(X,Y,0)
```

    533 µs ± 51.2 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python
#Michał
def lagrange(x, X, Y):
  m = len(X)
  z = 0.0
  for i in range(m):
      p = 1.0
      for k in range(m):
        if i == k: continue
        p = p* ((x - X[k])/(X[i] - X[k]))
      z += Y[i] * p
  return z

x=symbols('x')
%timeit lagrange(x,X,Y)
```

    808 µs ± 101 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)

```python

```


Version 1.4

Last updated: 2022-01-27

Collaborated with: Ritvik Puri 1004811778

# Probabilistic ML: Assignment 1
- **Deadline**: Feb 6, at 23:59.
- **Submission**: You need to submit your solutions through Crowdmark, including all your derivations, plots, and your code. You can produce the files however you like (e.g. LATEX, Microsoft Word, etc), as long as it is readable. Points will be deducted if we have a hard time reading your solutions or understanding the structure of your code.
- **Collaboration policy**: After attempting the problems on an individual basis, you may discuss and work together on the assignment with up to two classmates. However, **you must write your own code and write up your own solutions individually and explicitly name any collaborators** at the top of the homework.

# Q1 - Decision Theory

One successful use of probabilistic models is for building spam filters, which take in an email and take different actions depending on the likelihood that it’s spam.

Imagine you are running an email service. You have a well-calibrated spam classifier that tells you the probability that a particular email is spam: $p(spam|email)$. You have three options for what to do with each email: You can show it to the user, put it in the spam folder, or delete it entirely.

Depending on whether or not the email really is spam, the user will suffer a different amount  of wasted time for the different actions we can take, $L$(action, spam):

Action   | Spam        | Not spam
-------- | ----------- | -----------
Show     | 10          | 0
Folder   | 1           | 50
Delete   | 0           | 200

## Q1.1
[3pts] Plot the expected wasted user time for each of the three possible actions, as a function of the probability of spam: $p(spam|email)$.

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
losses = [[10, 0], [1, 50],[0, 200]]
actions_names = ['Show', 'Folder', 'Delete']
num_actions = len(losses)
def expected_loss_of_action(prob_spam, action):
    # Return expected loss over a Bernoulli random variable
    # with mean prob_spam.
    # Losses are given by the table above.
    expected_loss = prob_spam * losses[action][0] + \
      (1-prob_spam) * losses[action][1]

    return expected_loss

prob_range = np.linspace(0., 1., num=500)

# Make plot
for action in range(num_actions):
    plt.plot(prob_range, expected_loss_of_action(prob_range, action), label=actions_names[action])

plt.xlabel('$p(spam|email)$')
plt.ylabel('Expected loss of action')
plt.legend()

```

## Q1.2
[2pts] Write a function that computes the optimal action given the probability of spam.

```python
def optimal_action(prob_spam):
    #Return best action given the probability of spam.
    #Hint: np.argmin might be helpful.
    losses = []
    for action in range(num_actions):
      losses.append(expected_loss_of_action(prob_spam, action))
    optimal = np.argmin(losses)
    return optimal
```

## Q1.3
[4pts] Plot the expected loss of the optimal action as a function of the probability of spam.

Color the line according to the optimal action for that probability of spam.

```python
prob_range = np.linspace(0., 1., num=500)
optimal_losses = []
optimal_actions = []
for p in prob_range:
    # Compute the optimal action and its expected loss for
    # probability of spam given by p.
    action = optimal_action(p)
    optimal_actions.append(action)
    # print(optimal_actions)
    optimal_losses.append(expected_loss_of_action(p, action))

plt.xlabel('p(spam|email)')
plt.ylabel('Expected loss of optimal action')
plt.plot(prob_range, optimal_losses) # Cannot change the colour
```

## Q1.4
[4pts] For exactly which range of the probabilities of an email being spam should we delete an email?

Find the exact answer by hand using algebra.

Your answer: \\
Let $p = p(spam|email)$. Then we seek the range of probabilities where

$$L(delete, spam) \leq L(show, spam), L(folder, spam) $$

That is,

\begin{align}
0 \cdot p + 200 \cdot (1-p) &\leq 10 \cdot p + 0 \cdot (1-p) \\
200-200p &\leq 10p \\
200 &\leq 210p\\
\frac{200}{210} &\leq p
\end{align}

and

\begin{align}
0 \cdot p + 200 \cdot (1-p) &\leq 1 \cdot p + 50 \cdot (1-p) \\
200 - 200p &\leq p + 50 - 50p \\
150 &\leq 151p \\
\frac{150}{151} &\leq p
\end{align}

But $\frac{150}{151} > \frac{200}{210}$, so we should delete an email in the range $[\frac{150}{151}, 1]$, so that the expected loss of deleting it is less then both expected losses of showing it and keeping it in a folder.

# Q2 - Naïve Bayes, A Generative Model

In this question, we'll fit a Naïve Bayes model to the MNIST digits dataset, and use this model for making predictions and generating new images from the same distribution. MNIST is a dataset of 28x28 black-and-white images of handwritten digits of 0 to 9. We represent each image by a vector $x^{(i)} \in \{0,1\}^{784}$, where 0 and 1 represent white and black pixels respectively. Each class label $c^{(i)}$ is a number between 0 and 9, which in the code is represented by a 10-dimensional one-hot vector.

The Naïve Bayes model parameterized by $\theta$ and $\pi$ defines the following joint probability of $x$ and $c$,
$$p(x,c|\theta,\pi) = p(c|\pi)p(x|c,\theta) = p(c|\pi)\prod_{j=1}^{784}p(x_j|c,\theta),$$
where $x_j | c,\theta \sim \operatorname{Bernoulli}(\theta_{jc})$ or in other words $p(x_j | c,\theta) = \theta_{jc}^{x_j}(1-\theta_{jc})^{1-x_j}$, and $c|\pi$ follows a simple categorical distribution, i.e. $p(c|\pi) = \pi_c$.

We begin by learning the parameters $\theta$ and $\pi$. The following code will download and prepare the training and test sets.

```python
import numpy as np
import os
import gzip
import struct
import array
import matplotlib.pyplot as plt
import matplotlib.image
from urllib.request import urlretrieve

def download(url, filename):
    if not os.path.exists('data'):
        os.makedirs('data')
    out_file = os.path.join('data', filename)
    if not os.path.isfile(out_file):
        urlretrieve(url, out_file)

def mnist():
    base_url = 'http://yann.lecun.com/exdb/mnist/'

    def parse_labels(filename):
        with gzip.open(filename, 'rb') as fh:
            magic, num_data = struct.unpack(">II", fh.read(8))
            return np.array(array.array("B", fh.read()), dtype=np.uint8)

    def parse_images(filename):
        with gzip.open(filename, 'rb') as fh:
            magic, num_data, rows, cols = struct.unpack(">IIII", fh.read(16))
            return np.array(array.array("B", fh.read()), dtype=np.uint8).reshape(num_data, rows, cols)

    for filename in ['train-images-idx3-ubyte.gz',
                     'train-labels-idx1-ubyte.gz',
                     't10k-images-idx3-ubyte.gz',
                     't10k-labels-idx1-ubyte.gz']:
        download(base_url + filename, filename)

    train_images = parse_images('data/train-images-idx3-ubyte.gz')
    train_labels = parse_labels('data/train-labels-idx1-ubyte.gz')
    test_images = parse_images('data/t10k-images-idx3-ubyte.gz')
    test_labels = parse_labels('data/t10k-labels-idx1-ubyte.gz')

    return train_images, train_labels, test_images[:1000], test_labels[:1000]

def load_mnist():
    partial_flatten = lambda x: np.reshape(x, (x.shape[0], np.prod(x.shape[1:])))
    one_hot = lambda x, k: np.array(x[:, None] == np.arange(k)[None, :], dtype=int)
    train_images, train_labels, test_images, test_labels = mnist()
    train_images = (partial_flatten(train_images) / 255.0 > .5).astype(float)
    test_images = (partial_flatten(test_images) / 255.0 > .5).astype(float)
    train_labels = one_hot(train_labels, 10)
    test_labels = one_hot(test_labels, 10)
    N_data = train_images.shape[0]

    return N_data, train_images, train_labels, test_images, test_labels
```

## Q2.1
[2pts] Derive the expression for the Maximum Likelihood Estimator (MLE) of $\theta$ and $\pi$.

Your answer: \\
For MLE of $\theta$: Begin by taking the log-likelihood function,

\begin{align}
L(\theta) &= \prod_{n=1}^N p(c^{(n)}|\pi)\prod_{j=1}^{784}p(x_j^{(n)}|c,\theta) \\
l(\theta) &= log(\prod_{n=1}^N p(c^{(n)}|\pi)\prod_{j=1}^{784}p(x_j^{(n)}|c,\theta)) \\
&= \sum_{n=1}^N log \pi_{c^{(n)}} + \sum_{j=1}^{784} log \theta_{jc^{(n)}}^{x_{j}^{(n)}}(1-\theta_{jc^{(n)}})^{1-x_j^{(n)}} \\
&= \sum_{n=1}^N log \pi_{c^{(n)}} + \sum_{j=1}^{784} x_j^{(n)} log \theta_{jc^{(n)}} + (1-x_j^{(n)}) log (1-\theta_{jc^{(n)}}) \\
&= \sum_{n=1}^N log \pi_{c^{(n)}} + \sum_{k=0}^9 \sum_{j=1}^{784} \mathbb{1}\{ c^{(i)}=k \} (x_j^{(n)} log \theta_{jk} + (1-x_j^{(n)}) log (1-\theta_{jk}))
\end{align}

Now taking the derivative wrt $\theta_{jk}$ and setting equal to $0$,

\begin{align}
0 &= \sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} ( \frac{x_j^{(n)}}{\theta_{jc^{(n)}}} + \frac{1-x_j^{(n)}}{1-\theta_{jc^{(n)}}}) \\
&= \sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} (x_j^{(n)}-x_j^{(n)}\theta_{jc^{(n)}} - \theta_{jc^{(n)}}+\theta_{jc^{(n)}}x_j^{(n)}) \\
&= \sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} (x_j^{(n)} - \theta_{jc^{(n)}}) \\
\implies \hat{\theta}_{jc} &= \frac{\sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} x_j^{(n)}}{\sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \}} , k=0,\dots,9, j=1,\dots,784
\end{align}

For MLE of $\pi$: By Naive-Bayes' assumption of independence,

$$p(c^{(n)}|\pi) = \prod_{k=0}^{9} \pi_k^{c_k^{(n)}}$$

Now taking the log-likelihood function,

\begin{align}
L(\pi) &= \prod_{n=1}^N p(c^{(n)}|\pi) = \prod_{n=1}^N \prod_{k=0}^{9} \pi_k^{c_k^{(n)}}\\
l(\pi) &= \sum_{n=1}^N \sum_{k=0}^9 c_k^{(n)} log \pi_k\\
&= \sum_{n=1}^N \sum_{k=1}^9 c_k^{(n)}log \pi_k + c_0^{(n)}log(1-\sum_{k=1}^9 \pi_k)
\end{align}

Taking the derivative wrt $\pi_k$ and setting equal to $0$,

\begin{align}
0 &= \sum_{n=1}^N \frac{c_k^{(n)}}{pi_k} - \frac{c_0^{(n)}}{1-\sum_{k=1}^9 pi_k} \\
\sum_{n=1}^N \frac{c_k^{(n)}}{\pi_k} &= \sum_{n=1}^{N} \frac{c_0^{(n)}}{\pi_0} \\
\implies \frac{\hat{\pi}_k}{\hat{\pi}_0} &= \sum_{n=1}^N \frac{c_k^{(n)}}{c_0^{(n)}}
\end{align}

and since $\sum_{k=0}^9 \hat{\pi}_k = 1 $ means that $\sum_{k=0}^9 \frac{\hat{\pi}_k}{\hat{\pi}_0} = \frac{1}{\hat{\pi}_0}$, then

\begin{align}
\hat{\pi}_0 = \sum_{n=1}^N \frac{c_0^{(n)}}{\sum_{k=0}^9 c_k^{(n)}} = \frac{\sum_{n=1}^N c_0^{(n)}}{N}
\end{align}

Without loss of generality, for $t = 0, \dots, 9$,

\begin{align}
\hat{\pi}_t = \frac{\sum_{n=1}^N c_t^{(n)}}{N}
\end{align}

## Q2.2
[4pts] Using the MLE for this data, many entries of $\theta$ will be estimated to be 0, which seems extreme. So we look for another estimation method.

Assume the prior distribution of $\theta$ is such that the entries are i.i.d. and drawn from $\operatorname{Beta}(2,2)$. Derive the Maximum A Posteriori (MAP) estimator for $\theta$ (it has a simple final form). You can return the MLE for $\pi$ in your implementation. From now on, we will work with this estimator.

Your answer: For MAP estimation of $\theta$ with $Beta(2,2)$ prior,

\begin{align}
p(\theta|x,c,\pi) \propto p(\theta) \cdot p(x,c| \theta,\pi)
\end{align}

\begin{align}
l(\theta) &= log( p(\theta) p(x,c| \theta, \pi)) = log(\theta(1-\theta)p(x,c| \theta, \pi))\\
&= log\theta + log(1-\theta) + log(\prod_{n=1}^N p(c^{(n)}|\pi)\prod_{j=1}^{784}p(x_j^{(n)}|c,\theta)) \\
\dots
&= log\theta + log(1-\theta) + \sum_{n=1}^N log \pi_{c^{(n)}} + \sum_{k=0}^9 \sum_{j=1}^{784} \mathbb{1}\{ c^{(i)}=k \} (x_j^{(n)} log \theta_{jk} + (1-x_j^{(n)}) log (1-\theta_{jk}))
\end{align}

Now taking the derivative wrt $\theta_{jk}$ and setting equal to $0$,

\begin{align}
0 &= \frac{1}{\theta_{jk}} - \frac{1}{1-\theta_{jk}} + \sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} ( \frac{x_j^{(n)}}{\theta_{jk}} + \frac{1-x_j^{(n)}}{1-\theta_{jk}}) \\
&= 1 - \theta_{jk} - \theta_{jk} + \sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} (x_j^{(n)} - \theta_{jk})\\
2\theta_{jk} + \sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} \theta_{jk} &= 1 + \sum_{n=1}^N  \mathbb{1}\{ c^{(i)}=k \} x_{j}^{(n)}\\
\implies \hat{\theta}_{jc} &= \frac{1+\sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \} x_j^{(n)}}{2+\sum_{n=1}^N \mathbb{1}\{ c^{(i)}=k \}} , k=0,\dots,9, j=1,\dots,784
\end{align}

```python
def train_map_estimator(train_images, train_labels):
    """ Inputs: train_images (N_samples x N_features), train_labels (N_samples x N_classes)
        Returns the MAP estimator theta_est (N_features x N_classes) and the MLE
        estimator pi_est (N_classes)"""

    # YOU NEED TO WRITE THIS PART
    labels = np.sum(train_labels, axis=0)
    pi_est = labels/train_labels.shape[0]
    theta_est = (1+np.dot(np.transpose(train_images), train_labels))/(2+labels)
    return theta_est, pi_est
```

## Q2.3
[5pts] Derive an expression for the class log-posterior $\log p(c|x,\theta,\pi)$ for a single image. Then, complete the implementation of the following functions. Recall that our prediction rule is to choose the class that maximizes the above posterior, and accuracy is defined as the fraction of samples that are correctly predicted.

Report the average log-posterior $\frac{1}{N}\sum_{i=1}^{N}\log p(c^{(i)}|x^{(i)},\hat{\theta},\hat{\pi})$ (where $N$ is the number of samples) on the training test, as well the training and test errors.

Your answer: Using Bayes' rule,

\begin{align}
log p(c|x, \theta, \pi) &= log \frac{p(c|\theta, \pi)p(x|c, \theta, \pi)}{\sum_{k=0}^9 p(c_k |\theta, \pi)p(x|c_k, \theta, \pi)} \\
&= log \frac{ \pi_c \prod_{j=1}^{784} \theta_{jc}^{x_j}(1-\theta_{jc}^{x_j})}{{\sum_{k=0}^9 \pi_{c_k} \prod_{j=1}^{784}\theta_{jc_k}^{x_j}(1-\theta_{jc_k}^{x_j})}} \\
&= log \pi_c + \sum_{j=1}^{784} x_jlog\theta_{jc} + (1-x_j)log(1-\theta_{jc}) - log \sum_{k=0}^{9} \pi_{c_k} \prod_{j=1}^{784} \theta_{jc_k}^{x_j}(1-\theta_{jc_k}^{x_j}) \\
&= log \pi_c + \sum_{j=1}^{784} x_jlog\theta_{jc} + (1-x_j)log(1-\theta_{jc}) - log(\sum_{k=0}^{9} exp(log \pi_{c_k} \prod_{j=1}^{784} \theta_{jc_k}^{x_j}(1-\theta_{jc_k}^{x_j})) \\
&= log \pi_c + \sum_{j=1}^{784} x_jlog\theta_{jc} + (1-x_j)log(1-\theta_{jc}) - log(\sum_{k=0}^{9} exp(log \pi_{c_k} + \sum_{j=1}^{784} \theta_{jc_k}{x_j} + (1-\theta_{jc_k})(1-x_j)))
\end{align}

The average log-likelihood for MAP is nan.
The training accuracy for MAP is 0.962.
The test accuracy for MAP is 0.958

```python
def log_likelihood(images, theta, pi):
    """ Inputs: images (N_samples x N_features), theta, pi
        Returns the matrix 'log_like' of loglikehoods over the input images where
        log_like[i,c] = log p (c |x^(i), theta, pi) using the estimators theta and pi.
        log_like is a matrix of (N_samples x N_classes)
    Note that log likelihood is not only for c^(i), it is for all possible c's."""

    # YOU NEED TO WRITE THIS PART
    num = np.log(pi) + images.dot(np.log(theta)) + (1-images).dot(np.log(1-theta))
    denom = -np.log(np.exp(np.log(pi) + (images.dot(theta) + (1-images).dot(1-theta))))
    log_like = (num - denom)
    return log_like

def accuracy(log_like, labels):
    """ Inputs: matrix of log likelihoods and 1-of-K labels (N_samples x N_classes)
    Returns the accuracy based on predictions from log likelihood values"""

    # YOU NEED TO WRITE THIS PART
    predictions = (log_like == log_like.max(axis=1, keepdims=1))
    accuracy = np.mean(predictions==labels)
    return accuracy

N_data, train_images, train_labels, test_images, test_labels = load_mnist()
theta_est, pi_est = train_map_estimator(train_images, train_labels)

loglike_train = log_likelihood(train_images, theta_est, pi_est)
avg_loglike = np.sum(loglike_train * train_labels) / N_data
train_accuracy = accuracy(loglike_train, train_labels)
loglike_test = log_likelihood(test_images, theta_est, pi_est)
test_accuracy = accuracy(loglike_test, test_labels)

print(f"Average log-likelihood for MAP is {avg_loglike:.3f}")
print(f"Training accuracy for MAP is {train_accuracy:.3f}")
print(f"Test accuracy for MAP is {test_accuracy:.3f}")
```

    Average log-likelihood for MAP is nan
    Training accuracy for MAP is 0.962
    Test accuracy for MAP is 0.958

    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:10: RuntimeWarning: overflow encountered in exp
      # Remove the CWD from sys.path while we load stuff.
    /usr/local/lib/python3.7/dist-packages/ipykernel_launcher.py:29: RuntimeWarning: invalid value encountered in multiply

## Q2.4
[2pts] In this model, is it always true that any two pixels $x_i$ and $x_j$ with $i \neq j$ are independent given $c$? How about after marginalizing over $c$? Explain your answers.

Your Answer: In this model, it is always true that any two different pixels $x_i$ and $x_j$ are independent, given $c$, because by the Naive-Bayes' assumption, all pixels should have been sampled independently. However, upon marginalizing over $c$, the Naive-Bayes' assumption no longer holds, thus the pixels may not be independent in this case.

## Q2.5
[4pts] Since we have a generative model for our data, we can do more than just prediction. Randomly sample and plot 10 images from the learned distribution over images using the MAP parameter estimates. (Hint: You first need to sample the class $c$, and then sample pixels conditioned on $c$.)

```python
def image_sampler(theta, pi, num_images):
    """ Inputs: parameters theta and pi, and number of images to sample
    Returns the sampled images (N_images x N_features)"""

    # YOU NEED TO WRITE THIS PART
    sampled_images = np.zeros((num_images, 784))
    for i in range(num_images):
      number = np.random.choice(a=10, p=pi)
      sampled_images[i] = np.random.binomial(1, theta[:,number]).reshape((1, 784))
    return sampled_images

def plot_images(images, ims_per_row=5, padding=5, digit_dimensions=(28, 28),
                cmap=matplotlib.cm.binary, vmin=0., vmax=1.):
    """Images should be a (N_images x pixels) matrix."""
    fig = plt.figure(1)
    fig.clf()
    ax = fig.add_subplot(111)

    N_images = images.shape[0]
    N_rows = np.int32(np.ceil(float(N_images) / ims_per_row))
    pad_value = vmin
    concat_images = np.full(((digit_dimensions[0] + padding) * N_rows + padding,
                             (digit_dimensions[1] + padding) * ims_per_row + padding), pad_value)
    for i in range(N_images):
        cur_image = np.reshape(images[i, :], digit_dimensions)
        row_ix = i // ims_per_row
        col_ix = i % ims_per_row
        row_start = padding + (padding + digit_dimensions[0]) * row_ix
        col_start = padding + (padding + digit_dimensions[1]) * col_ix
        concat_images[row_start: row_start + digit_dimensions[0],
                      col_start: col_start + digit_dimensions[1]] = cur_image
        cax = ax.matshow(concat_images, cmap=cmap, vmin=vmin, vmax=vmax)
        plt.xticks(np.array([]))
        plt.yticks(np.array([]))

    plt.plot()

sampled_images = image_sampler(theta_est, pi_est, 10)
plot_images(sampled_images)
```

## Q2.6
[4pts] One of the advantages of generative models is that they can handle missing data, or be used to answer different sorts of questions about the model. Assume we have only observed some pixels of the image. Let $x_E = \{x_p : \text{pixel $p$ is observed}\}$. Derive an expression for $p(x_j|x_E,\theta,\pi)$, the conditional probability of an unobserved pixel $j$ given the observed pixels and distribution parameters. (Hint: You have to marginalize over $c$.)

Your answer: We have by independence that,

\begin{align}
p(x_j | x_E, \pi, \theta) = \sum_c p(x_j | c, \theta) p(c | x_E, \pi, \theta)
\end{align}

because in this model having observed pixels does not provide new information.

## Q2.7
[4pts] We now reveal only a random 20% of the pixels to the model. For the first 20 images in the training set, plot the images when the unobserved pixels are left as white, as well as the same images when the unobserved pixels are filled with the marginal probability of each pixel being 1 given the observed pixels, i.e. the value of the unobserved pixel $j$ is $p(x_j = 1|x_E,\theta,\pi)$.

```python
def probabilistic_imputer(theta, pi, original_images, is_observed):
    """Inputs: parameters theta and pi, original_images (N_images x N_features),
        and is_observed which has the same shape as original_images, with a value
        1. in every observed entry and 0. in every unobserved entry.
    Returns the new images where unobserved pixels are replaced by their
    conditional probability"""

    # YOU NEED TO WRITE THIS PART
    N = len(is_observed)
    imputed_images = original_images
    for i in range(N):
      for j in range(N):
        if is_observed[i][j] == 0:
          imputed_images[i][j] = 1
    return imputed_images

num_features = train_images.shape[1]
is_observed = np.random.binomial(1, p=0.2, size=(20, num_features))
plot_images(train_images[:20] * is_observed)
```

```python
imputed_images = probabilistic_imputer(theta_est, pi_est, train_images[:20], is_observed)
plot_images(imputed_images)
```

# Q3: Regression [33 pts]

## 3.1 Manually Derived Linear Regression [10 pts]

Suppose that $X \in \mathbb{R}^{m \times n}$ with $n \ge m$ and $Y \in \mathbb{R}^n$, and that $Y \sim \mathcal{N}(X^\top\beta, \sigma^2I)$.

In this question you will derive the result that the maximum likelihood estimate $\hat{\beta}$ of $\beta$ is given

$$\hat{\beta} = (XX^\top)^{-1}XY$$

1. [1 pts] What happens if $n < m$?

Since we are estimating $m+1$ parameters (including intercept), then the residuals have degree of freedom $n-m+1$ in which case if $n < m$, we have either 0 or negative degree of freedom.

2. [2 pts] What are the expectation and covariance matrix of $\hat{\beta}$ for a given true value of $\beta$?

We have that

\begin{align}
\hat{\beta} &= \left( XX^T\right)^{-1}XY\\
&= \left( XX^T\right)^{-1}X\left( X^T\beta+\epsilon \right)\\
&= \left( XX^T\right)^{-1}XX^T\beta + \left( XX^T\right)^{-1}X\epsilon\\
&= \beta + \left( XX^T\right)^{-1}X\epsilon
\end{align}

Taking expectation of both sides yields $E[\hat{\beta}] = \beta$. Taking variance of both sides yields,

\begin{align}
Var\left( \hat{\beta}\right)
   &= V\left( \beta + \left( XX^T\right)^{-1}X\epsilon \right)\\
   &= V\left(\left( XX^T\right)^{-1}X\epsilon \right) \\
   &= \left( XX^T\right)^{-1}XV\left(\epsilon \right) X^T \left( XX^T\right)^{-1} \\
   &= \left( XX^T\right)^{-1}X\sigma^2I X^T \left( XX^T\right)^{-1} \\
   &= \sigma^2\left( XX^T\right)^{-1}XX^T \left( XX^T\right)^{-1} \\
   &= \sigma^2\left( XX^T\right)^{-1}
\end{align}

3. [2 pts] Show that maximizing the likelihood is equivalent to minimizing the squared error $\sum_{i=1}^n(y_i - x_i^T\beta)^2$. [Hint: Use $\sum_{i=1}^na_i^2 = a^\top a]$.

The sum of squared residuals can be written as,

\begin{align}
e^Te = \sum_{i=1}^n e_i^2 = \sum_{i=1}^n (y_i - x_i^T\beta)^2
\end{align}

Thus maximizing likelihood will be equivalent to minimizing squared error.

4. [2 pts] Write the squared error in vector notation and expand. [Hint: Use $\beta^\top X^\top Y = Y^\top X \beta$.]

Squared error is,

\begin{align}
e^Te &= (y-X\hat{\beta})^T(y-X\hat{\beta}) \\
&= y^Ty - \hat{\beta}^TX^Ty - y^TX\hat{\beta} + \hat{\beta}X^TX\hat{\beta} \\
&= y^Ty - 2\hat{\beta}^TX^Ty+\hat{\beta}^TX^TX\hat{\beta}
\end{align}

This gives the likelihood expression.

5. [3 pts] Use the likelihood expression to write the negative log-likelihood. Write the derivative of the negative log-likelihood with respect to $\beta$, set equal to zero, and solve to show the maximum likelihood estimate $\hat{\beta}$ as above.

Taking the derivative and setting equal to 0,

\begin{align}
0 = \frac{\partial e^Te}{\partial \hat{\beta}} &= -2X^Ty + 2X^TX\hat{\beta} \\
X^Ty &= X^TX\hat{\beta}
\end{align}

Thus, we conclude

\begin{align}
\hat{\beta} = (XX^\top)^{-1}XY
\end{align}

## 3.2 Toy Data [3 pts]

### Note: For this question, we will use [JAX](https://jax.readthedocs.io/en/latest/notebooks/quickstart.html). JAX is a machine learning framework with an API almost identical to numpy. JAX has the benefit of implementing automatic differentiation, which we will use later in this question.

For visualization purposes and to minimize computational resources, we will work with 1-dimensional toy data.

That is, $X \in \mathbb{R}^{m \times n}$ where $m = 1$. We will learn models for 3 target functions.

- `target_f1`, linear trend with constant noise
- `target_f2`, linear trend heteroskedastic noise
- `target_f3`, non-linear trend with heteroskedatic noise

```python
from jax import grad
from jax import lax
import jax.numpy as jnp
import jax.random as random
import jax.scipy.stats as stats
from jax.ops import index_update
import numpy as np
import matplotlib.pyplot as plt
from scipy.special import gamma as gammaf
from typing import List, Optional, Callable, Tuple
%matplotlib inline

key = random.PRNGKey(42)
key_x = random.PRNGKey(414)
```

    WARNING:absl:No GPU/TPU found, falling back to CPU. (Set TF_CPP_MIN_LOG_LEVEL=0 and rerun for more info.)

```python
def target_f1(x, sigma_true):
  """
  Linear trend with constant noise.
  """
  noise = random.normal(key, (x.shape[-1],))
  y = 2 * x + sigma_true * noise

  return y.flatten()

def target_f2(x, **kwargs):
  """
  Linear trend with heteroskedastic noise.
  """
  noise = random.normal(key, (x.shape[-1],))
  y = 2 * x + jnp.linalg.norm(x, axis=0) * 0.3 * noise

  return y.flatten()

def target_f3(x, **kwargs):
  """
  Non-linear trend with heteroskedastic noise.
  """
  noise = random.normal(key, (x.shape[-1],))
  y = 2 * x + 5 * jnp.sin(0.5 * x) + jnp.linalg.norm(x, axis=0) * 0.3 * noise

  return y.flatten()
```

1. [2 pts] Write a function which produces a batch of data $x \sim \text{Uniform}(0, 20)$ and $y = target_f(x)$.

```python
def generate_data(target_f: Callable, data_size: int, stdev=0.1):
  """
  Sample data from the target function <target_f> with <data_size> samples.
  """
  # Note: Use `key_x` (not `key`!), defined above, as a PRNG key to generate x

  ## hint: pass sigma_true as a keyword argument (not positional) to avoid errors

  x = np.random.uniform(0, 20, size = (1,data_size))
  y = target_f(x, sigma_true=True)

  return x, y
```

```python
m, n = 1, 200    # dim, data_size
for target_f in (target_f1, target_f2, target_f3):
  x, y = generate_data(target_f, n)
  try:
    assert x.shape == (m, n)
    assert y.shape == (n,)
    print(f"{target_f.__name__} passed")

  except AssertionError:
        print(f"{target_f.__name__} failed")
        print(f"Expected x to have shape {(m, n)}. Got: {x.shape}")
        print(f"Expected y to have shape {(n,)}. Got: {y.shape}")
```

    target_f1 passed
    target_f2 passed
    target_f3 passed

2. [1 pt] For all three targets, plot a $n = 1000$ sample of the data.

```python
fig, axs = plt.subplots(nrows=3, figsize=(10, 10), sharey=True)
m, n = 1, 1000

for i, target_f in enumerate((target_f1, target_f2, target_f3)):
  x, y = generate_data(target_f, n)
  axs[i].scatter(x.flatten(), y)
  axs[i].set_title(f"Target function: {target_f.__name__}")

# Plotting code, no need to edit it
fig.add_subplot(111, frameon=False)
plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False)
plt.xlabel("Sampled data")
plt.ylabel("Sampled targets")

fig.tight_layout()
plt.show()
```

## 3.3 Linear Regression with $\hat{\beta}$ MLE [4 pts]

1. [2 pts] Program the function that computes the maximum likelihood estimate given $X$ and $Y$. Use it to estimate $\hat{\beta}$ for a $n=1000$ sample from each target function.

```python
def beta_mle(X, Y):
  """
  Return the Beta MLE for the linear regression between X and Y.
  """
  beta = jnp.linalg.inv(X.dot(X.T)).dot(X.dot(Y))
  return beta
```

2. [2 pt] For each function, plot the linear regression model given by $Y \sim \mathcal{N}(X^\top\beta, \sigma^2 I)$ for $\sigma = 1$. This plot should have the line of best fit given by the maximum likelihood estimate, as well as a shaded region around the line corresponding to plus/minus one standard deviation (i.e. the fixed uncertainty $\sigma = 1.0$). Display 3
plots, one for each target function, showing the data sample and maximum likelihood estimate linear regression model fit to that data.

```python
m, n = 1, 1000

for target_f in [target_f1, target_f2, target_f3]:

  # get data from the target function
  x, y = generate_data(target_f, n)
  # fit the regression
  beta_estimate = beta_mle(x, y)

  plt.figure(figsize=(10, 6))
  plt.scatter(x.flatten(), y, label="Generated data")
  yhat = x.transpose() @ beta_estimate
  plt.plot(x.flatten(), yhat, label="Fitted model", color="red")

  error_up, error_down = yhat - 1, yhat + 1
  sorted_indices = lax.top_k(x.flatten(), x.shape[1])[1][::-1]
  plt.fill_between(x.flatten()[sorted_indices],
                  error_down[sorted_indices], error_up[sorted_indices],
                  facecolor='green', alpha=0.4, label="Uncertainty")

  plt.legend(loc="lower right")
  plt.show()
```

## 3.4 Log-likelihood of Data Under Model [6 pts]

1. [2 pts] Write code for the function that computes the likelihood of a scalar $x$ under the Gaussian distribution $\mathcal{N}(\mu, \sigma)$. This function should be able to broadcast over $x, \mu, \sigma$.  I.e. it should allow these to be arrays of the same shape, and return an array of likelihoods of the same shape, i.e. $x_i \sim \mathcal{N}(\mu_i, \sigma_i)$.

```python
def gaussian_log_likelihood(mean, stdev, x):
  """
  Compute the gaussian log-likelihood, supporting arguments that are vector or
  matrix valued.
  """
  # implement
  log_pdf = stats.norm.logpdf(x, mean, stdev)
  return log_pdf
```

```python
# CHECK YOUR SOLUTION IS RIGHT:

key1, key2 = random.split(key)
x = random.normal(key1)
mean = random.normal(key2)
stdev = random.uniform(key)
log_likelihood = gaussian_log_likelihood(mean, stdev, x)
assert log_likelihood.shape == ()
assert jnp.isclose(log_likelihood, stats.norm.logpdf(x, mean, stdev))

x = random.normal(key1, shape=(100,))
mean = random.normal(key2)
stdev = random.uniform(key)
log_likelihood = gaussian_log_likelihood(mean, stdev, x)
assert log_likelihood.shape == (100,)
assert all(jnp.isclose(implemented, standard, atol=1e-4, rtol=1e-4) for
           implemented, standard in zip(
               log_likelihood, stats.norm.logpdf(x, mean, stdev)))

x = random.normal(key1, shape=(10,))
mean = random.normal(key2, shape=(10,))
stdev = random.uniform(key, shape=(10,))
log_likelihood = gaussian_log_likelihood(mean, stdev, x)
assert log_likelihood.shape == (10,)

assert all(jnp.isclose(implemented, standard) for
           implemented, standard in zip(
               log_likelihood, stats.norm.logpdf(x, mean, stdev)))

print("All tests passed")
```

    All tests passed

2. [2 pts] Use your gaussian log-likelihood function to write a function that computes the negative log-likelihood of the target value $Y$ under the model $Y \sim \mathcal{N}(X^\top\beta, \sigma^2 I)$.

```python
def lr_model_nll(beta, x, y, stdev=1.0):
  """
  Return the negative log-likelihood of y given beta.
  """
  # you can ignore these two lines, they just make sure that scalars are treated as vectors
  if beta.ndim is 0:
    beta = jnp.array([beta])

  # implement
  return -gaussian_log_likelihood(jnp.dot(jnp.transpose(x), beta), stdev, y)*100
```

3. [1 pts] Use this function to compute and report the negative log-likelihood of a $n = 1000$ batch of data under the model with the maximum-likelihood estimate $\hat{\beta}$ and $\sigma \in \{0.1, 0.3, 1., 2.\}$ for each target function.

```python
n = 1000
for i, target_f in enumerate((target_f1, target_f2, target_f3)):
  fname = target_f.__name__
  print(f"------- target_f: {fname} -------")
  for sig_model in (0.1, 0.3, 1.0, 2.0):
    print(f"------- sigma: {sig_model} -------")

    # compute negative log likelihood
    x, y = generate_data(target_f, n)
    b_mle = beta_mle(x, y)
    nll = lr_model_nll(b_mle, x, y, stdev = sig_model)

    print(f"Average Negative Log-Likelihood: {nll.mean()}")
```

    ------- target_f: target_f1 -------
    ------- sigma: 0.1 -------
    Average Negative Log-Likelihood: 5069.90185546875
    ------- sigma: 0.3 -------
    Average Negative Log-Likelihood: 550.8822021484375
    ------- sigma: 1.0 -------
    Average Negative Log-Likelihood: 144.04067993164062
    ------- sigma: 2.0 -------
    Average Negative Log-Likelihood: 174.24594116210938
    ------- target_f: target_f2 -------
    ------- sigma: 0.1 -------
    Average Negative Log-Likelihood: 64145.5859375
    ------- sigma: 0.3 -------
    Average Negative Log-Likelihood: 6490.22412109375
    ------- sigma: 1.0 -------
    Average Negative Log-Likelihood: 640.1566162109375
    ------- sigma: 2.0 -------
    Average Negative Log-Likelihood: 305.81060791015625
    ------- target_f: target_f3 -------
    ------- sigma: 0.1 -------
    Average Negative Log-Likelihood: 114856.9765625
    ------- sigma: 0.3 -------
    Average Negative Log-Likelihood: 12414.5947265625
    ------- sigma: 1.0 -------
    Average Negative Log-Likelihood: 1263.2967529296875
    ------- sigma: 2.0 -------
    Average Negative Log-Likelihood: 468.02874755859375

4. [1 pts] For each target function, what is the best choice of $\sigma$?

YOUR ANSWER HERE: \\
In the case of target_f1, sigma=1.0 is the best choice as it minimizes the Average Negative Log-Likelihood.
For target_f2 and target_f3, the value of sigma that yields this is 2.0.

## 3.5 Automatic Differentiation and Maximizing Likelihood

In a previous question you derived the expression for the derivative of the negative loglikelihood with respect to $\beta$. We will use that to test the gradients produced by automatic
differentiation.

1. [3 pts] For a random value of $\beta$, $\sigma$, and $n = 100$ sample from a target function, use automatic differentiation to compute the derivative of the negative log-likelihood of the sampled data with respect to $\beta$. Test that this is equivalent to the hand-derived value.

```python
def compute_nll_grad(x, y, beta, sigma):
  """
  Compute the gradient of the negative log-likelihood (hand-derived)
  """
  # you can ignore these two lines, they just make sure that scalars are treated as vectors
  if beta.ndim is 0:
    beta = jnp.array([beta])

  temp1 = x / (sigma**2)
  temp2 = y - jnp.dot(jnp.transpose(x), beta)
  temp = jnp.dot((-temp1), temp2)
  return temp

def avg_lr_model_nll(beta, x, y, stdev):
  return lr_model_nll(beta, x, y, stdev).mean()

beta_test = random.normal(key)
sigma_test = random.uniform(key, minval=0, maxval=1)
x, y = generate_data(target_f1, 100)

# grad() uses automatic differentiation
ad_grad = grad(avg_lr_model_nll)(beta_test, x, y, sigma_test)
hand_derivative = compute_nll_grad(x, y, beta_test, sigma_test)

jnp.isclose(ad_grad, hand_derivative)
```

    DeviceArray([ True], dtype=bool)

### 3.5.1 Train Linear Regression Model with Gradient Descent [5 pts]

In this question we will compute gradients of negative log-likelihood with respect to $\beta$. We will use gradient descent to find $\beta$ that maximizes the likelihood.

1. [3 pts] Write a function `train_lin_reg` that accepts a target function and an initial estimate for $\beta$ and some hyperparameters for batch-size, model variance, learning rate, and number of iterations. Then, for each iteration:

- sample data from the target function
- compute gradients of negative log-likelihood with respect to $\beta$ using automatic differentiation
- update the estimate of $\beta$ with gradient descent with specified learning rate

and, after all iterations, returns the final estimate of $\beta$.

```python
def train_lin_reg(target_f: Callable, initial_beta,
                  model_variance: float, learning_rate: float, num_iter: int):
  """
  Train the linear regression model using the given parameters.
  """

  beta = initial_beta

  for i in range(num_iter):
    # compute gradients of nll wrt beta using andy
    nll_grad = grad(avg_lr_model_nll)(beta, x, y, model_variance**(1/2))
    # update beta estimate with gd using specified lr
    beta -= (nll_grad)*learning_rate

  if beta.ndim == 0:
    beta = jnp.array([beta])

  return beta
```

2. [2 pts] For each target function, start with an initial parameter $\beta$, learn an estimate for $\beta_{\text{learned}}$ by gradient descent. Then plot a $n = 1000$ sample of the data and the learned linear regression model with shaded region for uncertainty corresponding to plus/minus one standard deviation.

```python
for target_function in [target_f1, target_f2, target_f3]:

  # sample a random beta
  initial_beta = random.normal(key)
  # fit the linear regression
  fitted_beta = train_lin_reg(target_function, initial_beta, 1, 0.0001, 1000)

  # Plotting code (you don't need to edit it)
  plt.figure(figsize=(10, 6))
  x, y = generate_data(target_function, 1000, 1.0)
  plt.scatter(x.flatten(), y, label="Generated data")
  yhat = x.transpose() @ fitted_beta
  plt.plot(x.flatten(), yhat, label="Fitted model", color="red")

  error_up, error_down = yhat - 1, yhat + 1

  sorted_indices = lax.top_k(x.flatten(), x.shape[1])[1][::-1]
  plt.fill_between(x.flatten()[sorted_indices],
                  error_down[sorted_indices], error_up[sorted_indices],
                  facecolor='green', alpha=0.4, label="Uncertainty")

  plt.legend(loc="lower right")
  plt.show()
```

### 3.5.2 Non-linear regression with a Neural Network [5 pts]

In the previous questions we have considered a linear regression model

$$ Y \sim \mathcal{N}(X^\top \beta, \sigma^2 I) $$

This model specified the mean of the prediction distribution for each distribution by the product of that datapoint with our parameter.

Now, let us generalize this to consider a model where the mean of the predictive distribution
is a non-linear function of each datapoint. We will have our non-linear model be a simple
function called neural net with parameters $\theta$ (collection of weights and biases).

$$Y \sim \mathcal{N}(\text{neural_net}(X, \theta), \sigma^2)$$

The code for a fully-connected neural network (multi-layer perceptron) with one 10-dimensional hidden layer and a tanh nonlinearirty. You must write this yourself using only basic operations like matrix multiply and tanh, you may not use layers provided by a library. This network will output the mean vector, test that it outputs the correct shape for some random parameters.

```python
def neural_network(x, theta):
  """
  Compute the forward pass of a neural network with 1 hidden layer consisting
  of 10 hidden neurons and a tanh nonlinear activation.
  """
  n = x.shape[-1]
  input_to_hidden_weights = theta["input_hidden_w"]
  hidden_to_output_weights = theta["hidden_out_w"]
  input_to_hidden_bias = theta["input_hidden_b"]
  hidden_to_output_bias = theta["hidden_out_b"]

  outputs = jnp.tanh(x.transpose() @ input_to_hidden_weights +
                     input_to_hidden_bias)

  return outputs @ hidden_to_output_weights + hidden_to_output_bias
```

```python
n = 100
x, y = generate_data(target_f1, n)

theta = {
    "input_hidden_w": random.normal(key, shape=(1, 10)),
    "hidden_out_w": random.normal(key, shape=(10,)),
    "input_hidden_b": random.normal(key, shape=(10,)),
    "hidden_out_b": random.normal(key, shape=(1,))
}

mean = neural_network(x, theta)
assert mean.shape == (n,)
```

2. [1pts] Write the code that computes the negative log-likelihood for this model where the mean is given by the output of the neural network and $\sigma = 1.0$

```python
def nn_model_nll(theta, x, y, sigma=1.0):
  """
  Compute the negative log-likelihood for the model where the mean is the output
  of the neural network.
  """
  # get neural network output
  mean = neural_network(x, theta)
  stdev_arg = jnp.ones(y.flatten().shape) * sigma

  return -gaussian_log_likelihood(mean, stdev_arg, y)
```

3. [2 pts] Write a function `train_nn_reg` that accepts a target function and an initial estimate for $\theta$ and some hyperparameters for batch-size, model variance, learning rate, and number of iterations. Then, for each iteration:

- sample data from the target function
- compute gradients of negative log-likelihood with respect to $\theta$
- update the estimate of $\theta$ with gradient descent with specified learning rate

and, after all iterations, returns the final estimate of $\theta$.

```python
def avg_nn_model_nll(theta, x, y, sigma=1.0):
  return nn_model_nll(theta, x, y, sigma).mean()

def train_nn_reg(target_f: Callable, initial_theta,
                 model_variance: float = 1.0, learning_rate: float= 1e-3, num_iter: int=5000
                 ):
  """
  Train the linear regression model using the given parameters.
  """
  theta = initial_theta

  sigma = model_variance ** (1/2)

  log_frequency = int(jnp.ceil(0.1 * num_iter))

  model_nll_grad = grad(avg_nn_model_nll)

  for i in range(num_iter):

    # compute average negative log likelihood
    avg_neg_log_like = avg_nn_model_nll(theta, x, y, sigma)

    if i % log_frequency == 0:
      print(f"[Iteration {i + 1}] Loss: {avg_neg_log_like}")

    # compute gradient here
    # Hint: use the grad() function
    grad_theta = model_nll_grad(theta, x, y, sigma)

    theta["input_hidden_w"] -= learning_rate * grad_theta["input_hidden_w"]
    theta["hidden_out_w"] -= learning_rate * grad_theta["hidden_out_w"]
    theta["input_hidden_b"] -= learning_rate * grad_theta["input_hidden_b"]
    theta["hidden_out_b"] -= learning_rate * grad_theta["hidden_out_b"]

  return theta
```

4. [2pts] For each target function, start with an initialization of the network parameters, $\theta$, use your train function to minimize the negative log-likelihood and find an estimate for $\theta$ learned by gradient descent. Then plot a $n = 1000$ sample of the data and the learned regression model with shaded uncertainty bounds given by $\sigma = 1.0$.

```python

initial_theta = {
    "input_hidden_w": random.normal(key, shape=(1, 10)),
    "hidden_out_w": random.normal(key, shape=(10,)),
    "input_hidden_b": random.normal(key, shape=(10,)),
    "hidden_out_b": random.normal(key, shape=(1,))
}

for target_function in [target_f1, target_f2, target_f3]:

  # fit a neural network to your data
  fitted_params = fitted_params = train_nn_reg(target_function, initial_theta, 1.0, 1e-3, 1000)

  # Plotting code, you don't need to edit this
  plt.figure(figsize=(10, 6))
  x, y = generate_data(target_function, 1000, 1.0)
  plt.scatter(x.flatten(), y, label="Generated data")
  plot_x = np.linspace(0, 20, 1000)
  nn_preds = neural_network(plot_x.reshape(1, 1000), fitted_params)
  plt.plot(plot_x, nn_preds, label="Fitted model", color="red")

  error_up, error_down = nn_preds + 1, nn_preds - 1
  plt.fill_between(plot_x,
                  error_down, error_up,
                  facecolor='green', alpha=0.2, label="Uncertainty")

  plt.legend(loc="lower right")
  plt.show()
```