## 2.1 Introduction
### 2.1.1 Model Context
In PyMC3, we typically handle all the variables we want in our model within the context of the Model object.

```python
import pymc3 as pm

with pm.Model() as model:
    parameter = pm.Exponential("poisson_param", 1.0)
    data_generator = pm.Poisson("data_generator", parameter)
```

    /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters

This is an extra layer of convenience compared to PyMC. Any variables created within a given Model's context will be automatically assigned to that model. If you try to define a variable outside of the context of a model, you will get an error.

We can continue to work within the context of the same model by using with with the name of the model object that we have already created.

```python
with model:
    data_plus_one = data_generator + 1
```

We can examine the same variables outside of the model context once they have been defined, but to define more variables that the model will recognize they have to be within the context.

```python
parameter.tag.test_value
```

    array(0.69314718)

Each variable assigned to a model will be defined with its own name, the first string parameter (we will cover this further in the variables section). To create a different model object with the same name as one we have used previously, we need only run the first block of code again.

```python
with pm.Model() as model:
    theta = pm.Exponential("theta", 2.0)
    data_generator = pm.Poisson("data_generator", theta)
```

We can also define an entirely separate model. Note that we are free to name our models whatever we like, so if we do not want to overwrite an old model we need only make another.

```python
with pm.Model() as ab_testing:
    p_A = pm.Uniform("P(A)", 0, 1)
    p_B = pm.Uniform("P(B)", 0, 1)
```

You probably noticed that PyMC3 will often give you notifications about transformations when you add variables to your model. These transformations are done internally by PyMC3 to modify the space that the variable is sampled in (when we get to actually sampling the model). This is an internal feature which helps with the convergence of our samples to the posterior distribution and serves to improve the results.

### 2.1.2 PyMC3 Variables
All PyMC3 variables have an initial value (i.e. test value). Using the same variables from before:

```python
print("parameter.tag.test_value =", parameter.tag.test_value)
print("data_generator.tag.test_value =", data_generator.tag.test_value)
print("data_plus_one.tag.test_value =", data_plus_one.tag.test_value)
```

    parameter.tag.test_value = 0.6931471824645996
    data_generator.tag.test_value = 0
    data_plus_one.tag.test_value = 1

The *test_value* is used only for the model, as the starting point for sampling if no other start is specified. It will not change as a result of sampling. This initial state can be changed at variable creation by specifying a value for the testval parameter.

```python
with pm.Model() as model:
    parameter = pm.Exponential("poisson_param", 1.0, testval=0.5)

print("\nparameter.tag.test_value =", parameter.tag.test_value)
```

    parameter.tag.test_value = 0.5

This can be helpful if you are using a more unstable prior that may require a better starting point.

PyMC3 is concerned with two types of programming variables: stochastic and deterministic.

* *stochastic variables* are variables that are not deterministic, i.e., even if you knew all the values of the variables' parameters and components, it would still be random. Included in this category are instances of classes `Poisson`, `DiscreteUniform`, and `Exponential`.

* *deterministic variables* are variables that are not random if the variables' parameters and components were known. This might be confusing at first: a quick mental check is if I knew all of variable `foo`'s component variables, I could determine what `foo`'s value is.

```python
with pm.Model() as model:
    lambda_1 = pm.Exponential("lambda_1", 1.0)
    lambda_2 = pm.Exponential("lambda_2", 1.0)
    tau = pm.DiscreteUniform("tau", lower=0, upper=10)

new_deterministic_variable = lambda_1 + lambda_2
```

```python
import numpy as np

n_data_points = 5  # in CH1 we had ~70 data points
idx = np.arange(n_data_points)
with model:
    lambda_ = pm.math.switch(tau >= idx, lambda_1, lambda_2)
```

### 2.1.3 Including Observations in the Model
At this point, it may not look like it, but we have fully specified our priors. For example, we can ask and answer questions like "What does my prior distribution of λ1 look like?"

```python
%matplotlib inline
from IPython.core.pylabtools import figsize
import matplotlib.pyplot as plt
import scipy.stats as stats
figsize(12.5, 4)

samples = lambda_1.random(size=20000)
plt.hist(samples, bins=70, normed=True, histtype="stepfilled")
plt.title("Prior distribution for $\lambda_1$")
plt.xlim(0, 8);
```

To frame this in the notation of the first chapter, though this is a slight abuse of notation, we have specified P(A). Our next goal is to include data/evidence/observations X into our model.

PyMC3 stochastic variables have a keyword argument `observed`. The keyword `observed` has a very simple role: fix the variable's current value to be the given data, typically a NumPy array or pandas DataFrame. For example:

```python
data = np.array([10, 5])
with model:
    fixed_variable = pm.Poisson("fxd", 1, observed=data)
print("value: ", fixed_variable.tag.test_value)
```

    value:  [10  5]

This is how we include data into our models: initializing a stochastic variable to have a *fixed value*.

To complete our text message example, we fix the PyMC3 variable `observations` to the observed dataset.

```python
# We're using some fake data here
data = np.array([10, 25, 15, 20, 35])
with model:
    obs = pm.Poisson("obs", lambda_, observed=data)
print(obs.tag.test_value)
```

    [10 25 15 20 35]

## 2.2 Modeling Approaches
A good starting thought to Bayesian modeling is to think about how your data might have been generated. Position yourself in an omniscient position, and try to imagine how you would recreate the dataset.

In the last chapter we investigated text message data. We begin by asking how our observations may have been generated:

1. We started by thinking "what is the best random variable to describe this count data?" A Poisson random variable is a good candidate because it can represent count data. So we model the number of sms's received as sampled from a Poisson distribution.

2. Next, we think, "Ok, assuming sms's are Poisson-distributed, what do I need for the Poisson distribution?" Well, the Poisson distribution has a parameter $\lambda$.

3. Do we know $\lambda$? No. In fact, we have a suspicion that there are two $\lambda$ values, one for the earlier behaviour and one for the later behaviour. We don't know when the behaviour switches though, but call the switchpoint $\tau$.

4. What is a good distribution for the two $\lambda$s? The exponential is good, as it assigns probabilities to positive real numbers. Well the exponential distribution has a parameter too, call it $\alpha$.

5. Do we know what the parameter $\alpha$ might be? No. At this point, we could continue and assign a distribution to $\alpha$, but it's better to stop once we reach a set level of ignorance: whereas we have a prior belief about $\lambda$, ("it probably changes over time", "it's likely between 10 and 30", etc.), we don't really have any strong beliefs about $\alpha$. So it's best to stop here. What is a good value for $\alpha$ then? We think that the $\lambda$s are between 10-30, so if we set $\alpha$ really low (which corresponds to larger probability on high values) we are not reflecting our prior well. Similar, a too-high alpha misses our prior belief as well. A good idea for $\alpha$ as to reflect our belief is to set the value so that the mean of $\lambda$, given $\alpha$, is equal to our observed mean. This was shown in the last chapter.

6. We have no expert opinion of when $\tau$ might have occurred. So we will suppose $\tau$ is from a discrete uniform distribution over the entire timespan.

### 2.2.1 Same Story, Different Ending
Interestingly, we can create new datasets by retelling the story. For example, if we reverse the above steps, we can simulate a possible realization of the dataset.

1> Specify when the user's behaviour switches by sampling from DiscreteUniform(0,80):

```python
tau = np.random.randint(0, 80)
print(tau)
```

    58

2> Draw λ1 and λ2 from an Exp(α) distribution:

```python
alpha = 1./20.
lambda_1, lambda_2 = np.random.exponential(scale=1/alpha, size=2)
print(lambda_1, lambda_2)
```

    39.648140101666314 7.49579089954877

3> For days before τ, represent the user's received SMS count by sampling from Poi(λ1), and sample from Poi(λ2) for days after τ. For example:

```python
data = np.r_[stats.poisson.rvs(mu=lambda_1, size=tau), stats.poisson.rvs(mu=lambda_2, size = 80 - tau)]
```

4> Plot the artificial dataset:

```python
plt.bar(np.arange(80), data, color="#348ABD")
plt.bar(tau-1, data[tau - 1], color="r", label="user behaviour changed")
plt.xlabel("Time (days)")
plt.ylabel("count of text-msgs received")
plt.title("Artificial dataset")
plt.xlim(0, 80)
plt.legend();
```

It is okay that our fictional dataset does not look like our observed dataset: the probability is incredibly small it indeed would. PyMC3's engine is designed to find good parameters, λi,τ, that maximize this probability.

The ability to generate artificial dataset is an interesting side effect of our modeling, and we will see that this ability is a very important method of Bayesian inference. We produce a few more datasets below:

```python
def plot_artificial_sms_dataset():
    tau = stats.randint.rvs(0, 80)
    alpha = 1./20.
    lambda_1, lambda_2 = stats.expon.rvs(scale=1/alpha, size=2)
    data = np.r_[stats.poisson.rvs(mu=lambda_1, size=tau), stats.poisson.rvs(mu=lambda_2, size=80 - tau)]
    plt.bar(np.arange(80), data, color="#348ABD")
    plt.bar(tau - 1, data[tau-1], color="r", label="user behaviour changed")
    plt.xlim(0, 80);

figsize(12.5, 5)
plt.title("More example of artificial datasets")
for i in range(4):
    plt.subplot(4, 1, i+1)
    plot_artificial_sms_dataset()
```

### 2.2.3 A Simple Case
As this is a hacker book, we'll continue with the web-dev example. For the moment, we will focus on the analysis of site A only. Assume that there is some true $0 \lt p_A \lt 1$ probability that users who, upon shown site A, eventually purchase from the site. This is the true effectiveness of site A. Currently, this quantity is unknown to us.

Suppose site A was shown to $N$ people, and $n$ people purchased from the site. One might conclude hastily that $p_A = \frac{n}{N}$. Unfortunately, the observed frequency $\frac{n}{N}$ does not necessarily equal $p_A$ -- there is a difference between the observed frequency and the true frequency of an event. The true frequency can be interpreted as the probability of an event occurring. For example, the true frequency of rolling a 1 on a 6-sided die is $\frac{1}{6}$. Knowing the true frequency of events like:

* fraction of users who make purchases,
* frequency of social attributes,
* percent of internet users with cats etc.

are common requests we ask of Nature. Unfortunately, often Nature hides the true frequency from us and we must infer it from observed data.

The observed frequency is then the frequency we observe: say rolling the die 100 times you may observe 20 rolls of 1. The observed frequency, 0.2, differs from the true frequency, $\frac{1}{6}$. We can use Bayesian statistics to infer probable values of the true frequency using an appropriate prior and observed data.

With respect to our A/B example, we are interested in using what we know, $N$ (the total trials administered) and $n$ (the number of conversions), to estimate what $p_A$, the true frequency of buyers, might be.

To setup a Bayesian model, we need to assign prior distributions to our unknown quantities. A priori, what do we think $p_A$ might be? For this example, we have no strong conviction about $p_A$, so for now, let's assume $p_A$ is uniform over [0,1]:

```python
import pymc3 as pm

# The parameters are the bounds of the Uniform.
with pm.Model() as model:
    p = pm.Uniform('p', lower=0, upper=1)
```

Had we had stronger beliefs, we could have expressed them in the prior above.

For this example, consider $p_A = 0.05$, and $N = 1500$ users shown site A, and we will simulate whether the user made a purchase or not. To simulate this from $N$ trials, we will use a Bernoulli distribution: if  $X\ \sim \text{Ber}(p)$, then $X$ is 1 with probability $p$ and 0 with probability $1 - p$. Of course, in practice we do not know $p_A$, but we will use it here to simulate the data.

```python
# set constants
p_true = 0.05  # remember, this is unknown.
N = 1500

# sample N Bernoulli random variables from Ber(0.05).
# each random variable has a 0.05 chance of being a 1.
# this is the data-generation step
occurrences = stats.bernoulli.rvs(p_true, size=N)

print(occurrences) # Remember: Python treats True == 1, and False == 0
print(np.sum(occurrences))
```

    [0 0 0 ... 0 0 0]
    70

The observed frequency is:

```python
# Occurrences.mean is equal to n/N.
print("What is the observed frequency in Group A? %.4f" % np.mean(occurrences))
print("Does this equal the true frequency? %s" % (np.mean(occurrences) == p_true))
```

    What is the observed frequency in Group A? 0.0467
    Does this equal the true frequency? False

We combine the observations into the PyMC3 observed variable, and run our inference algorithm:

```python
#include the observations, which are Bernoulli
with model:
    obs = pm.Bernoulli("obs", p, observed=occurrences)
    # To be explained in chapter 3
    step = pm.Metropolis()
    trace = pm.sample(18000, step=step)
    burned_trace = trace[1000:]
```

    Multiprocess sampling (2 chains in 2 jobs)
    Metropolis: [p]
    Sampling 2 chains: 100%|██████████| 37000/37000 [00:07<00:00, 5138.47draws/s]
    The number of effective samples is smaller than 25% for some parameters.

We plot the posterior distribution of the unknown $p_A$ below:

```python
figsize(12.5, 4)
plt.title("Posterior distribution of $p_A$, the true effectiveness of site A")
plt.vlines(p_true, 0, 90, linestyle="--", label="true $p_A$ (unknown)")
plt.hist(burned_trace["p"], bins=25, histtype="stepfilled", normed=True)
plt.legend();
```

Our posterior distribution puts most weight near the true value of $p_A$, but also some weights in the tails. This is a measure of how uncertain we should be, given our observations. Try changing the number of observations, N, and observe how the posterior distribution changes.

### 2.2.4 A and B together
A similar analysis can be done for site B's response data to determine the analogous $p_B$. But what we are really interested in is the difference between $p_A$ and $p_B$. Let's infer $p_A$, $p_B$, and $\text{delta} = p_A - p_B$, all at once. We can do this using PyMC3's deterministic variables. (We'll assume for this exercise that $p_B = 0.04$, so $\text{delta} = 0.01$, $N_B = 750$ (significantly less than $N_A$) and we will simulate site B's data like we did for site A's data )

```python
figsize(12, 4)

#these two quantities are unknown to us.
true_p_A = 0.05
true_p_B = 0.04

#notice the unequal sample sizes -- no problem in Bayesian analysis.
N_A = 1500
N_B = 750

#generate some observations
observations_A = stats.bernoulli.rvs(true_p_A, size=N_A)
observations_B = stats.bernoulli.rvs(true_p_B, size=N_B)
print("Obs from Site A: ", observations_A[:30], "...")
print("Obs from Site B: ", observations_B[:30], "...")
```

    Obs from Site A:  [0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0] ...
    Obs from Site B:  [0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1 0 0 0 0 0 0 0 0 0 1] ...

```python
print(np.mean(observations_A))
print(np.mean(observations_B))
```

    0.05266666666666667
    0.04933333333333333

```python
# Set up the pymc3 model. Again assume Uniform priors for p_A and p_B.
with pm.Model() as model:
    p_A = pm.Uniform("p_A", 0, 1)
    p_B = pm.Uniform("p_B", 0, 1)

    # Define the deterministic delta function. This is our unknown of interest.
    delta = pm.Deterministic("delta", p_A - p_B)

    # Set of observations, in this case we have two observation datasets.
    obs_A = pm.Bernoulli("obs_A", p_A, observed=observations_A)
    obs_B = pm.Bernoulli("obs_B", p_B, observed=observations_B)

    # To be explained in chapter 3.
    step = pm.Metropolis()
    trace = pm.sample(20000, step=step)
    burned_trace=trace[1000:]
```

    Multiprocess sampling (2 chains in 2 jobs)
    CompoundStep
    >Metropolis: [p_B]
    >Metropolis: [p_A]
    Sampling 2 chains: 100%|██████████| 41000/41000 [00:15<00:00, 2648.46draws/s]
    The number of effective samples is smaller than 25% for some parameters.

Below we plot the posterior distributions for the three unknowns:

```python
p_A_samples = burned_trace["p_A"]
p_B_samples = burned_trace["p_B"]
delta_samples = burned_trace["delta"]
```

```python
figsize(12.5, 10)

#histogram of posteriors

ax = plt.subplot(311)

plt.xlim(0, .1)
plt.hist(p_A_samples, histtype='stepfilled', bins=25, alpha=0.85,
         label="posterior of $p_A$", color="#A60628", normed=True)
plt.vlines(true_p_A, 0, 80, linestyle="--", label="true $p_A$ (unknown)")
plt.legend(loc="upper right")
plt.title("Posterior distributions of $p_A$, $p_B$, and delta unknowns")

ax = plt.subplot(312)

plt.xlim(0, .1)
plt.hist(p_B_samples, histtype='stepfilled', bins=25, alpha=0.85,
         label="posterior of $p_B$", color="#467821", normed=True)
plt.vlines(true_p_B, 0, 80, linestyle="--", label="true $p_B$ (unknown)")
plt.legend(loc="upper right")

ax = plt.subplot(313)
plt.hist(delta_samples, histtype='stepfilled', bins=30, alpha=0.85,
         label="posterior of delta", color="#7A68A6", normed=True)
plt.vlines(true_p_A - true_p_B, 0, 60, linestyle="--",
           label="true delta (unknown)")
plt.vlines(0, 0, 60, color="black", alpha=0.2)
plt.legend(loc="upper right");
```

Notice that as a result of N_B < N_A, i.e. we have less data from site B, our posterior distribution of $p_B$ is fatter, implying we are less certain about the true value of $p_B$ than we are of $p_A$.

With respect to the posterior distribution of $\text{delta}$, we can see that the majority of the distribution is above $\text{delta}=0$, implying there site A's response is likely better than site B's response. The probability this inference is incorrect is easily computable:

```python
# Count the number of samples less than 0, i.e. the area under the curve
# before 0, represent the probability that site A is worse than site B.
print("Probability site A is WORSE than site B: %.3f" % \
    np.mean(delta_samples < 0))

print("Probability site A is BETTER than site B: %.3f" % \
    np.mean(delta_samples > 0))
```

    Probability site A is WORSE than site B: 0.376
    Probability site A is BETTER than site B: 0.624

If this probability is too high for comfortable decision-making, we can perform more trials on site B (as site B has less samples to begin with, each additional data point for site B contributes more inferential "power" than each additional data point for site A).

Try playing with the parameters true_p_A, true_p_B, N_A, and N_B, to see what the posterior of $\text{delta}$ looks like. Notice in all this, the difference in sample sizes between site A and site B was never mentioned: it naturally fits into Bayesian analysis.

I hope the readers feel this style of A/B testing is more natural than hypothesis testing, which has probably confused more than helped practitioners. Later in this book, we will see two extensions of this model: the first to help dynamically adjust for bad sites, and the second will improve the speed of this computation by reducing the analysis to a single equation.

### 2.2.6 The Binomial Distribution
The binomial distribution is one of the most popular distributions, mostly because of its simplicity and usefulness. Unlike the other distributions we have encountered thus far in the book, the binomial distribution has 2 parameters: $N$, a positive integer representing  $N$ trials or number of instances of potential events, and $p$, the probability of an event occurring in a single trial. Like the Poisson distribution, it is a discrete distribution, but unlike the Poisson distribution, it only weighs integers from $0$ to $N$. The mass distribution looks like:

$$P( X = k ) =  {{N}\choose{k}}  p^k(1-p)^{N-k}$$
If $X$ is a binomial random variable with parameters $p$ and $N$, denoted $X \sim \text{Bin}(N,p)$, then $X$ is the number of events that occurred in the $N$ trials (obviously $0 \le X \le N$). The larger $p$ is (while still remaining between 0 and 1), the more events are likely to occur. The expected value of a binomial is equal to $Np$. Below we plot the mass probability distribution for varying parameters.

```python
figsize(12.5, 4)

import scipy.stats as stats
binomial = stats.binom

parameters = [(10, .4), (10, .9)]
colors = ["#348ABD", "#A60628"]

for i in range(2):
    N, p = parameters[i]
    _x = np.arange(N + 1)
    plt.bar(_x - 0.5, binomial.pmf(_x, N, p), color=colors[i],
            edgecolor=colors[i],
            alpha=0.6,
            label="$N$: %d, $p$: %.1f" % (N, p),
            linewidth=3)

plt.legend(loc="upper left")
plt.xlim(0, 10.5)
plt.xlabel("$k$")
plt.ylabel("$P(X = k)$")
plt.title("Probability mass distributions of binomial random variables");
```

The special case when $N = 1$ corresponds to the Bernoulli distribution. There is another connection between Bernoulli and Binomial random variables. If we have $X_1, X_2, ... , X_N$ Bernoulli random variables with the same $p$, then $Z = X_1 + X_2 + ... + X_N \sim \text{Binomial}(N, p )$.

The expected value of a Bernoulli random variable is $p$. This can be seen by noting the more general Binomial random variable has expected value $Np$ and setting $N=1$.

### 2.2.7 Example: Cheating Among Students
We will use the binomial distribution to determine the frequency of students cheating during an exam. If we let $N$ be the total number of students who took the exam, and assuming each student is interviewed post-exam (answering without consequence), we will receive integer $X$ "Yes I did cheat" answers. We then find the posterior distribution of $p$, given $N$, some specified prior on $p$, and observed data $X$.

Suppose 100 students are being surveyed for cheating, and we wish to find $p$, the proportion of cheaters. There are a few ways we can model this in PyMC3. I'll demonstrate the most explicit way, and later show a simplified version. Both versions arrive at the same inference. In our data-generation model, we sample $p$, the true proportion of cheaters, from a prior. Since we are quite ignorant about $p$, we will assign it a $\text{Uniform}(0,1)$ prior.

```python
N = 100
with pm.Model() as model:
    p = pm.Uniform("freq_cheating", 0, 1)
```

Again, thinking of our data-generation model, we assign Bernoulli random variables to the 100 students: 1 implies they cheated and 0 implies they did not.

```python
with model:
    true_answers = pm.Bernoulli("truths", p, shape=N, testval=np.random.binomial(1, 0.5, N))
```

If we carry out the algorithm, the next step that occurs is the first coin-flip each student makes. This can be modeled again by sampling 100 Bernoulli random variables with $p=1/2$: denote a 1 as a Heads and 0 a Tails.

```python
with model:
    first_coin_flips = pm.Bernoulli("first_flips", 0.5, shape=N, testval=np.random.binomial(1, 0.5, N))
print(first_coin_flips.tag.test_value)
```

    [0 1 1 0 1 1 0 1 1 0 1 1 1 1 0 1 1 0 1 1 0 0 1 0 0 1 0 1 0 1 0 0 0 0 1 0 1
     0 0 0 0 1 1 1 1 1 0 1 0 1 0 0 1 0 0 1 0 0 0 0 1 0 1 1 1 1 0 0 0 0 0 1 0 1
     1 1 1 1 0 0 0 1 0 1 1 0 0 1 1 1 1 1 0 1 0 1 0 0 1 0]

Although not everyone flips a second time, we can still model the possible realization of second coin-flips:

```python
with model:
    second_coin_flips = pm.Bernoulli("second_flips", 0.5, shape=N, testval=np.random.binomial(1, 0.5, N))
```

Using these variables, we can return a possible realization of the observed proportion of "Yes" responses. We do this using a PyMC3 deterministic variable:

```python
import theano.tensor as tt
with model:
    val = first_coin_flips*true_answers + (1 - first_coin_flips)*second_coin_flips
    observed_proportion = pm.Deterministic("observed_proportion", tt.sum(val)/float(N))
```

The line fc*t_a + (1-fc)*sc contains the heart of the Privacy algorithm. Elements in this array are 1 if and only if i) the first toss is heads and the student cheated or ii) the first toss is tails, and the second is heads, and are 0 else. Finally, the last line sums this vector and divides by float(N), produces a proportion.

```python
observed_proportion.tag.test_value
```

    array(0.63999999)

Next we need a dataset. After performing our coin-flipped interviews the researchers received 35 "Yes" responses. To put this into a relative perspective, if there truly were no cheaters, we should expect to see on average 1/4 of all responses being a "Yes" (half chance of having first coin land Tails, and another half chance of having second coin land Heads), so about 25 responses in a cheat-free world. On the other hand, if all students cheated, we should expected to see approximately 3/4 of all responses be "Yes".

The researchers observe a Binomial random variable, with N = 100 and p = observed_proportion with value = 35:

```python
X = 35

with model:
    observations = pm.Binomial("obs", N, observed_proportion, observed=X)
```

Below we add all the variables of interest to a Model container and run our black-box algorithm over the model.

```python
with model:
    step = pm.Metropolis(vars=[p])
    trace = pm.sample(40000, step=step)
    burned_trace = trace[15000:]
```

    Multiprocess sampling (2 chains in 2 jobs)
    CompoundStep
    >Metropolis: [freq_cheating]
    >BinaryGibbsMetropolis: [truths, first_flips, second_flips]
    Sampling 2 chains: 100%|██████████| 81000/81000 [19:59<00:00, 67.53draws/s]
    The number of effective samples is smaller than 10% for some parameters.

```python
figsize(12.5, 3)
p_trace = burned_trace["freq_cheating"][15000:]
plt.hist(p_trace, histtype="stepfilled", normed=True, alpha=0.85, bins=30,
         label="posterior distribution", color="#348ABD")
plt.vlines([.05, .35], [0, 0], [5, 5], alpha=0.3)
plt.xlim(0, 1)
plt.legend();
```

With regards to the above plot, we are still pretty uncertain about what the true frequency of cheaters might be, but we have narrowed it down to a range between 0.05 to 0.35 (marked by the solid lines). This is pretty good, as a priori we had no idea how many students might have cheated (hence the uniform distribution for our prior). On the other hand, it is also pretty bad since there is a .3 length window the true value most likely lives in. Have we even gained anything, or are we still too uncertain about the true frequency?

I would argue, yes, we have discovered something. It is implausible, according to our posterior, that there are no cheaters, i.e. the posterior assigns low probability to $p=0$. Since we started with an uniform prior, treating all values of $p$ as equally plausible, but the data ruled out $p=0$ as a possibility, we can be confident that there were cheaters.

This kind of algorithm can be used to gather private information from users and be reasonably confident that the data, though noisy, is truthful.

### 2.2.8 Alternative PyMC3 Model
Given a value for $p$ (which from our god-like position we know), we can find the probability the student will answer yes:

$$\begin{align}
P(\text{"Yes"}) = P( \text{Heads on first coin} )P( \text{cheater} ) + P( \text{Tails on first coin} )P( \text{Heads on second coin} ) \\\\
 = \frac{1}{2}p + \frac{1}{2}\frac{1}{2}\\\\
 = \frac{p}{2} + \frac{1}{4}
\end{align}$$
Thus, knowing $p$ we know the probability a student will respond "Yes". In PyMC3, we can create a deterministic function to evaluate the probability of responding "Yes", given $p$:

```python
with pm.Model() as model:
    p = pm.Uniform("freq_cheating", 0, 1)
    p_skewed = pm.Deterministic("p_skewed", 0.5*p + 0.25)
```

I could have typed p_skewed  = 0.5*p + 0.25 instead for a one-liner, as the elementary operations of addition and scalar multiplication will implicitly create a deterministic variable, but I wanted to make the deterministic boilerplate explicit for clarity's sake.

If we know the probability of respondents saying "Yes", which is p_skewed, and we have $N=100$ students, the number of "Yes" responses is a binomial random variable with parameters N and p_skewed.

This is where we include our observed 35 "Yes" responses. In the declaration of the pm.Binomial, we include value = 35 and observed = True.

```python
with model:
    yes_responses = pm.Binomial("number_cheaters", 100, p_skewed, observed=35)
```

Below we add all the variables of interest to a Model container and run our black-box algorithm over the model.

```python
with model:
    step = pm.Metropolis()
    trace = pm.sample(25000, step=step)
    burned_trace = trace[2500:]
```

    Multiprocess sampling (2 chains in 2 jobs)
    Metropolis: [freq_cheating]
    Sampling 2 chains: 100%|██████████| 51000/51000 [00:13<00:00, 3913.76draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
figsize(12.5, 3)
p_trace = burned_trace["freq_cheating"]
plt.hist(p_trace, histtype="stepfilled", normed=True, alpha=0.85, bins=30,
         label="posterior distribution", color="#348ABD")
plt.vlines([.05, .35], [0, 0], [5, 5], alpha=0.2)
plt.xlim(0, 1)
plt.legend();
```

### 2.2.10 Example: Challenger Space Shuttle Disaster
On January 28, 1986, the twenty-fifth flight of the U.S. space shuttle program ended in disaster when one of the rocket boosters of the Shuttle Challenger exploded shortly after lift-off, killing all seven crew members. The presidential commission on the accident concluded that it was caused by the failure of an O-ring in a field joint on the rocket booster, and that this failure was due to a faulty design that made the O-ring unacceptably sensitive to a number of factors including outside temperature. Of the previous 24 flights, data were available on failures of O-rings on 23, (one was lost at sea), and these data were discussed on the evening preceding the Challenger launch, but unfortunately only the data corresponding to the 7 flights on which there was a damage incident were considered important and these were thought to show no obvious trend. The data are shown below:

```python
figsize(12.5, 3.5)
np.set_printoptions(precision=3, suppress=True)
challenger_data = np.genfromtxt("data/challenger_data.csv", skip_header=1,
                                usecols=[1, 2], missing_values="NA",
                                delimiter=",")
#drop the NA values
challenger_data = challenger_data[~np.isnan(challenger_data[:, 1])]

#plot it, as a function of tempature (the first column)
print("Temp (F), O-Ring failure?")
print(challenger_data)

plt.scatter(challenger_data[:, 0], challenger_data[:, 1], s=75, color="k",
            alpha=0.5)
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Fahrenheit)")
plt.title("Defects of the Space Shuttle O-Rings vs temperature");
```

It looks clear that the probability of damage incidents occurring increases as the outside temperature decreases. We are interested in modeling the probability here because it does not look like there is a strict cutoff point between temperature and a damage incident occurring. The best we can do is ask "At temperature $t$, what is the probability of a damage incident?". The goal of this example is to answer that question.

We need a function of temperature, call it $p(t)$, that is bounded between 0 and 1 (so as to model a probability) and changes from 1 to 0 as we increase temperature. There are actually many such functions, but the most popular choice is the logistic function.

$$p(t) = \frac{1}{ 1 + e^{ \;\beta t } } $$
In this model, $\beta$ is the variable we are uncertain about. Below is the function plotted for $\beta = 1, 3, -5$.

```python
figsize(12, 3)

def logistic(x, beta):
    return 1.0 / (1.0 + np.exp(beta * x))

x = np.linspace(-4, 4, 100)
plt.plot(x, logistic(x, 1), label=r"$\beta = 1$")
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$")
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$")
plt.legend();
```

But something is missing. In the plot of the logistic function, the probability changes only near zero, but in our data above the probability changes around 65 to 70. We need to add a bias term to our logistic function:

$$p(t) = \frac{1}{ 1 + e^{ \;\beta t + \alpha } } $$
Some plots are below, with differing $\alpha$.

```python
def logistic(x, beta, alpha=0):
    return 1.0 / (1.0 + np.exp(np.dot(beta, x) + alpha))

x = np.linspace(-4, 4, 100)

plt.plot(x, logistic(x, 1), label=r"$\beta = 1$", ls="--", lw=1)
plt.plot(x, logistic(x, 3), label=r"$\beta = 3$", ls="--", lw=1)
plt.plot(x, logistic(x, -5), label=r"$\beta = -5$", ls="--", lw=1)

plt.plot(x, logistic(x, 1, 1), label=r"$\beta = 1, \alpha = 1$",
         color="#348ABD")
plt.plot(x, logistic(x, 3, -2), label=r"$\beta = 3, \alpha = -2$",
         color="#A60628")
plt.plot(x, logistic(x, -5, 7), label=r"$\beta = -5, \alpha = 7$",
         color="#7A68A6")

plt.legend(loc="lower left");
```

Adding a constant term $\alpha$ amounts to shifting the curve left or right (hence why it is called a bias).

Let's start modeling this in PyMC3. The $\beta, \alpha$ parameters have no reason to be positive, bounded or relatively large, so they are best modeled by a Normal random variable, introduced next.

### 2.2.11 The Normal Distribution
A Normal random variable, denoted $X \sim N(\mu, 1/\tau)$, has a distribution with two parameters: the mean, $\mu$, and the precision, $\tau$. Those familiar with the Normal distribution already have probably seen $\sigma^2$ instead of $\tau^{-1}$. They are in fact reciprocals of each other. The change was motivated by simpler mathematical analysis and is an artifact of older Bayesian methods. Just remember: the smaller $\tau$, the larger the spread of the distribution (i.e. we are more uncertain); the larger $\tau$, the tighter the distribution (i.e. we are more certain). Regardless, $\tau$ is always positive.

The probability density function of a $N( \mu, 1/\tau)$ random variable is:

$$ f(x | \mu, \tau) = \sqrt{\frac{\tau}{2\pi}} \exp\left( -\frac{\tau}{2} (x-\mu)^2 \right) $$
We plot some different density functions below.

```python
import scipy.stats as stats

nor = stats.norm
x = np.linspace(-8, 7, 150)
mu = (-2, 0, 3)
tau = (.7, 1, 2.8)
colors = ["#348ABD", "#A60628", "#7A68A6"]
parameters = zip(mu, tau, colors)

for _mu, _tau, _color in parameters:
    plt.plot(x, nor.pdf(x, _mu, scale=1./_tau),
             label="$\mu = %d,\;\\tau = %.1f$" % (_mu, _tau), color=_color)
    plt.fill_between(x, nor.pdf(x, _mu, scale=1./_tau), color=_color,
                     alpha=.33)

plt.legend(loc="upper right")
plt.xlabel("$x$")
plt.ylabel("density function at $x$")
plt.title("Probability distribution of three different Normal random \
variables");
```

A Normal random variable can be take on any real number, but the variable is very likely to be relatively close to $\mu$. In fact, the expected value of a Normal is equal to its $\mu$ parameter:

$$ E[ X | \mu, \tau] = \mu$$
and its variance is equal to the inverse of $\tau$:

$$Var( X | \mu, \tau ) = \frac{1}{\tau}$$
Below we continue our modeling of the Challenger space craft:

```python
temperature = challenger_data[:, 0]
D = challenger_data[:, 1]  # defect or not?

#notice the`value` here. We explain why below.
with pm.Model() as model:
    beta = pm.Normal("beta", mu=0, tau=0.001, testval=0)
    alpha = pm.Normal("alpha", mu=0, tau=0.001, testval=0)
    p = pm.Deterministic("p", 1.0/(1. + tt.exp(beta*temperature + alpha)))
```

We have our probabilities, but how do we connect them to our observed data? A Bernoulli random variable with parameter $p$, denoted $\text{Ber}(p)$, is a random variable that takes value 1 with probability $p$, and 0 else. Thus, our model can look like:

$$ \text{Defect Incident, $D_i$} \sim \text{Ber}( \;p(t_i)\; ), \;\; i=1..N$$
where $p(t)$ is our logistic function and $t_i$ are the temperatures we have observations about. Notice in the above code we had to set the values of beta and alpha to 0. The reason for this is that if beta and alpha are very large, they make p equal to 1 or 0. Unfortunately, pm.Bernoulli does not like probabilities of exactly 0 or 1, though they are mathematically well-defined probabilities. So by setting the coefficient values to 0, we set the variable p to be a reasonable starting value. This has no effect on our results, nor does it mean we are including any additional information in our prior. It is simply a computational caveat in PyMC3.

```python
# connect the probabilities in `p` with our observations through a
# Bernoulli random variable.
with model:
    observed = pm.Bernoulli("bernoulli_obs", p, observed=D)

    # Mysterious code to be explained in Chapter 3
    start = pm.find_MAP()
    step = pm.Metropolis()
    trace = pm.sample(120000, step=step, start=start)
    burned_trace = trace[100000::2]
```

    /Library/Frameworks/Python.framework/Versions/3.6/lib/python3.6/site-packages/pymc3/tuning/starting.py:61: UserWarning: find_MAP should not be used to initialize the NUTS sampler, simply call pymc3.sample() and it will automatically initialize NUTS in a better way.
      warnings.warn('find_MAP should not be used to initialize the NUTS sampler, simply call pymc3.sample() and it will automatically initialize NUTS in a better way.')
    logp = -19.024, ||grad|| = 9.9071: 100%|██████████| 27/27 [00:00<00:00, 1179.34it/s]
    Multiprocess sampling (2 chains in 2 jobs)
    CompoundStep
    >Metropolis: [alpha]
    >Metropolis: [beta]
    Sampling 2 chains: 100%|██████████| 241000/241000 [01:55<00:00, 2094.53draws/s]
    The estimated number of effective samples is smaller than 200 for some parameters.

We have trained our model on the observed data, now we can sample values from the posterior. Let's look at the posterior distributions for $\alpha$ and $\beta$:

```python
alpha_samples = burned_trace["alpha"][:, None]  # best to make them 1d
beta_samples = burned_trace["beta"][:, None]

figsize(12.5, 6)

#histogram of the samples:
plt.subplot(211)
plt.title(r"Posterior distributions of the variables $\alpha, \beta$")
plt.hist(beta_samples, histtype='stepfilled', bins=35, alpha=0.85,
         label=r"posterior of $\beta$", color="#7A68A6", normed=True)
plt.legend()

plt.subplot(212)
plt.hist(alpha_samples, histtype='stepfilled', bins=35, alpha=0.85,
         label=r"posterior of $\alpha$", color="#A60628", normed=True)
plt.legend();
```

All samples of $\beta$ are greater than 0. If instead the posterior was centered around 0, we may suspect that $\beta = 0$, implying that temperature has no effect on the probability of defect.

Similarly, all $\alpha$ posterior values are negative and far away from 0, implying that it is correct to believe that $\alpha$ is significantly less than 0.

Regarding the spread of the data, we are very uncertain about what the true parameters might be (though considering the low sample size and the large overlap of defects-to-nondefects this behaviour is perhaps expected).

Next, let's look at the expected probability for a specific value of the temperature. That is, we average over all samples from the posterior to get a likely value for $p(t_i)$.

```python
t = np.linspace(temperature.min() - 5, temperature.max()+5, 50)[:, None]
p_t = logistic(t.T, beta_samples, alpha_samples)

mean_prob_t = p_t.mean(axis=0)
```

```python
figsize(12.5, 4)

plt.plot(t, mean_prob_t, lw=3, label="average posterior \nprobability \
of defect")
plt.plot(t, p_t[0, :], ls="--", label="realization from posterior")
plt.plot(t, p_t[-2, :], ls="--", label="realization from posterior")
plt.scatter(temperature, D, color="k", s=50, alpha=0.5)
plt.title("Posterior expected value of probability of defect; \
plus realizations")
plt.legend(loc="lower left")
plt.ylim(-0.1, 1.1)
plt.xlim(t.min(), t.max())
plt.ylabel("probability")
plt.xlabel("temperature");
```

Above we also plotted two possible realizations of what the actual underlying system might be. Both are equally likely as any other draw. The blue line is what occurs when we average all the 20000 possible dotted lines together.

An interesting question to ask is for what temperatures are we most uncertain about the defect-probability? Below we plot the expected value line and the associated 95% intervals for each temperature.

```python
from scipy.stats.mstats import mquantiles

# vectorized bottom and top 2.5% quantiles for "confidence interval"
qs = mquantiles(p_t, [0.025, 0.975], axis=0)
plt.fill_between(t[:, 0], *qs, alpha=0.7,
                 color="#7A68A6")

plt.plot(t[:, 0], qs[0], label="95% CI", color="#7A68A6", alpha=0.7)

plt.plot(t, mean_prob_t, lw=1, ls="--", color="k",
         label="average posterior \nprobability of defect")

plt.xlim(t.min(), t.max())
plt.ylim(-0.02, 1.02)
plt.legend(loc="lower left")
plt.scatter(temperature, D, color="k", s=50, alpha=0.5)
plt.xlabel("temp, $t$")

plt.ylabel("probability estimate")
plt.title("Posterior probability estimates given temp. $t$");
```

The 95% credible interval, or 95% CI, painted in purple, represents the interval, for each temperature, that contains 95% of the distribution. For example, at 65 degrees, we can be 95% sure that the probability of defect lies between 0.25 and 0.75.

More generally, we can see that as the temperature nears 60 degrees, the CI's spread out over [0,1] quickly. As we pass 70 degrees, the CI's tighten again. This can give us insight about how to proceed next: we should probably test more O-rings around 60-65 temperature to get a better estimate of probabilities in that range. Similarly, when reporting to scientists your estimates, you should be very cautious about simply telling them the expected probability, as we can see this does not reflect how wide the posterior distribution is.

### 2.2.12 What about the day of the Challenger disaster?
On the day of the Challenger disaster, the outside temperature was 31 degrees Fahrenheit. What is the posterior distribution of a defect occurring, given this temperature? The distribution is plotted below. It looks almost guaranteed that the Challenger was going to be subject to defective O-rings.

```python
figsize(12.5, 2.5)

prob_31 = logistic(31, beta_samples, alpha_samples)

plt.xlim(0.995, 1)
plt.hist(prob_31, bins=1000, normed=True, histtype='stepfilled')
plt.title("Posterior distribution of probability of defect, given $t = 31$")
plt.xlabel("probability of defect occurring in O-ring");
```

## 2.3 Is our model appropriate?
The skeptical reader will say "You deliberately chose the logistic function for $p(t)$ and the specific priors. Perhaps other functions or priors will give different results. How do I know I have chosen a good model?" This is absolutely true. To consider an extreme situation, what if I had chosen the function $p(t) = 1,\; \forall t$, which guarantees a defect always occurring: I would have again predicted disaster on January 28th. Yet this is clearly a poorly chosen model. On the other hand, if I did choose the logistic function for $p(t)$, but specified all my priors to be very tight around 0, likely we would have very different posterior distributions. How do we know our model is an expression of the data? This encourages us to measure the model's goodness of fit.

We can think: how can we test whether our model is a bad fit? An idea is to compare observed data (which if we recall is a fixed stochastic variable) with artificial dataset which we can simulate. The rationale is that if the simulated dataset does not appear similar, statistically, to the observed dataset, then likely our model is not accurately represented the observed data.

Previously in this Chapter, we simulated artificial dataset for the SMS example. To do this, we sampled values from the priors. We saw how varied the resulting datasets looked like, and rarely did they mimic our observed dataset. In the current example, we should sample from the posterior distributions to create very plausible datasets. Luckily, our Bayesian framework makes this very easy. We only need to create a new Stochastic variable, that is exactly the same as our variable that stored the observations, but minus the observations themselves. If you recall, our Stochastic variable that stored our observed data was:

`observed = pm.Bernoulli("bernoulli_obs", p, observed=D)`

Hence we create:

`simulated_data = pm.Bernoulli("simulation_data", p)`

Let's simulate 10 000:

```python
N = 10000
with pm.Model() as model:
    beta = pm.Normal("beta", mu=0, tau=0.001, testval=0)
    alpha = pm.Normal("alpha", mu=0, tau=0.001, testval=0)
    p = pm.Deterministic("p", 1.0/(1. + tt.exp(beta*temperature + alpha)))
    observed = pm.Bernoulli("bernoulli_obs", p, observed=D)

    simulated = pm.Bernoulli("bernoulli_sim", p, shape=p.tag.test_value.shape)
    step = pm.Metropolis(vars=[p])
    trace = pm.sample(N, step=step)
```

    Multiprocess sampling (2 chains in 2 jobs)
    CompoundStep
    >CompoundStep
    >>Metropolis: [beta]
    >>Metropolis: [alpha]
    >BinaryGibbsMetropolis: [bernoulli_sim]
    Sampling 2 chains: 100%|██████████| 21000/21000 [00:36<00:00, 579.87draws/s]
    The estimated number of effective samples is smaller than 200 for some parameters.

```python
figsize(12.5, 5)

simulations = trace["bernoulli_sim"]
print(simulations.shape)

plt.title("Simulated dataset using posterior parameters")
figsize(12.5, 6)
for i in range(4):
    ax = plt.subplot(4, 1, i+1)
    plt.scatter(temperature, simulations[1000*i, :], color="k",
                s=50, alpha=0.6)
```

Note that the above plots are different (if you can think of a cleaner way to present this, please send a pull request and answer here!).

We wish to assess how good our model is. "Good" is a subjective term of course, so results must be relative to other models.

We will be doing this graphically as well, which may seem like an even less objective method. The alternative is to use Bayesian p-values. These are still subjective, as the proper cutoff between good and bad is arbitrary. Gelman emphasises that the graphical tests are more illuminating [7] than p-value tests. We agree.

The following graphical test is a novel data-viz approach to logistic regression. The plots are called separation plots. For a suite of models we wish to compare, each model is plotted on an individual separation plot.

For each model, we calculate the proportion of times the posterior simulation proposed a value of 1 for a particular temperature, i.e. compute $P( \;\text{Defect} = 1 | t, \alpha, \beta )$ by averaging. This gives us the posterior probability of a defect at each data point in our dataset. For example, for the model we used above:

```python
posterior_probability = simulations.mean(axis=0)
print("posterior prob of defect | realized defect ")
for i in range(len(D)):
    print("%.2f                     |   %d" % (posterior_probability[i], D[i]))
```

    posterior prob of defect | realized defect
    0.41                     |   0
    0.25                     |   1
    0.28                     |   0
    0.32                     |   0
    0.37                     |   0
    0.19                     |   0
    0.16                     |   0
    0.25                     |   0
    0.77                     |   1
    0.55                     |   1
    0.25                     |   1
    0.08                     |   0
    0.37                     |   0
    0.85                     |   1
    0.37                     |   0
    0.12                     |   0
    0.25                     |   0
    0.06                     |   0
    0.11                     |   0
    0.07                     |   0
    0.12                     |   1
    0.11                     |   0
    0.73                     |   1

Next we sort each column by the posterior probabilities:

```python
ix = np.argsort(posterior_probability)
print("probb | defect ")
for i in range(len(D)):
    print("%.2f  |   %d" % (posterior_probability[ix[i]], D[ix[i]]))
```

    probb | defect
    0.06  |   0
    0.07  |   0
    0.08  |   0
    0.11  |   0
    0.11  |   0
    0.12  |   0
    0.12  |   1
    0.16  |   0
    0.19  |   0
    0.25  |   1
    0.25  |   0
    0.25  |   0
    0.25  |   1
    0.28  |   0
    0.32  |   0
    0.37  |   0
    0.37  |   0
    0.37  |   0
    0.41  |   0
    0.55  |   1
    0.73  |   1
    0.77  |   1
    0.85  |   1

We can present the above data better in a figure: I've wrapped this up into a `separation_plot` function.

```python
from separation_plot import separation_plot

figsize(11., 1.5)
separation_plot(posterior_probability, D)
```

The snaking-line is the sorted probabilities, blue bars denote defects, and empty space (or grey bars for the optimistic readers) denote non-defects. As the probability rises, we see more and more defects occur. On the right hand side, the plot suggests that as the posterior probability is large (line close to 1), then more defects are realized. This is good behaviour. Ideally, all the blue bars should be close to the right-hand side, and deviations from this reflect missed predictions.

The black vertical line is the expected number of defects we should observe, given this model. This allows the user to see how the total number of events predicted by the model compares to the actual number of events in the data.

It is much more informative to compare this to separation plots for other models. Below we compare our model (top) versus three others:

1. the perfect model, which predicts the posterior probability to be equal 1 if a defect did occur.
2. a completely random model, which predicts random probabilities regardless of temperature.
3. a constant model: where $P(D = 1 \; | \; t) = c, \;\; \forall t$. The best choice for $c$ is the observed frequency of defects, in this case 7/23.

```python
figsize(11., 1.25)

# Our temperature-dependent model
separation_plot(posterior_probability, D)
plt.title("Temperature-dependent model")

# Perfect model
# i.e. the probability of defect is equal to if a defect occurred or not.
p = D
separation_plot(p, D)
plt.title("Perfect model")

# random predictions
p = np.random.rand(23)
separation_plot(p, D)
plt.title("Random model")

# constant model
constant_prob = 7./23*np.ones(23)
separation_plot(constant_prob, D)
plt.title("Constant-prediction model");
```

In the random model, we can see that as the probability increases there is no clustering of defects to the right-hand side. Similarly for the constant model.

The perfect model, the probability line is not well shown, as it is stuck to the bottom and top of the figure. Of course the perfect model is only for demonstration, and we cannot infer any scientific inference from it.


# Deep Dive into Principal Component Analysis (PCA).

# Table of Contents

1. [Introduction.](#introduction)

2. [Principal Component Analysis (PCA) Intuition.](#intuition)

3. [PCA Mathemathical Formulation.](#math)

4. [Principal Component Analysis (PCA) in Practice.](#practice)

  4.1 [PCA for Dimension Reduction.](#dim_reduction)

  4.2 [PCA for Visualization and Better Insights.](#visualization)

  4.3 [PCA for Noise Filtering.](#filter_noise)

  4.4 [PCA as a Preprocessor for ML algorithms.](#pre_processor)

5. [Summary.](#summary)

## 1. Introduction

In this Notebook, we will explore Principal Component Analysis from various perspectives.

We will first give a short intuitive explanation for principal component analysis and why it makes sense. Then we will go deeper into the actual derivation of Principal Components using the principle of maximizing the total projected variances onto components. Once we have understood the theory and concept, we will dive deeper into the use cases and examples. We will consider four scenarios with examples.

  1. Dimension Reduction
  2. Visualization
  3. Noise Filtering
  4. As a pre-processor for ML algorithms.

In the end, we will summarize our discussion with various pointers to alternatives to PCA.

## 2. Principal Component Analysis Intuition.

##### 2.1. Why Dimensionality reduction make sense?

In real life, the data is generated from a fixed degree of freedom and when we collect data, there is noise added or sometimes we don't know the actual dimensions . These all mean that to represent a data, we use a lot of dimension then that was used by data generation process. However, since we don't always have information about the data generation, it would be nice to have a tool, which can remove the unnecessary dimensions.

Examples: Consider the black and white image of digits like in [MNIST Digit Database](https://en.wikipedia.org/wiki/MNIST_database). Each image is 28x28 == 784 pixels or 784 dimensions. However, most of the pixels are white and most of the digits are generated by a combination of vertical, horizontal and rotation stroke i.e data is generated by about 3 dimensions.

##### 2.2. What is Dimensionality Reduction

Dimensionality reduction is choosing a basis or mathematical representation within which you can describe most but not all of the variance within your data, thereby retaining the relevant information, while reducing the amount of information necessary to represent it. There are a variety of techniques for doing this including but not limited to PCA, [ICA](https://en.wikipedia.org/wiki/Independent_component_analysis), and [Matrix Feature Factorization](https://en.wikipedia.org/wiki/Matrix_decomposition). These will take existing data and reduce it to the most discriminative components. All of these allow you to represent most of the information in your dataset with fewer, more discriminative features.

##### 2.3. Why is Dimensionality Reduction Useful?

In terms of performance, having data of high dimensions is problematic because:

* It can mean high computational cost to perform learning and inference and

* It often leads to over-fitting when learning a model, which means that the model will perform well on the training data but poorly on test data.

Dimensionality reduction addresses both of these problems, while (hopefully) preserving most of the relevant information in the data needed to learn accurate, predictive models.

Also note that, in general visualization of lower dimension data and its interpretation are more straightforward and it could be used for getting better insights into the data.

##### 2.3 Intuition of PCA.

The intuition behind PCA is remarkably simple. What we ask is given a dataset, can we represent the dataset using fewer number of dimensions?

In the following picture, we can see that the original data spans in 3D but most of the information is contained in the 2D plain which is oriented around at 45 degree on x-y plane.
The second picture shows how the 3D information is represented in the 2D plain.

[Source [Matthias Scholz Thesis](http://phdthesis-bioinformatics-maxplanckinstitute-molecularplantphys.matthias-scholz.de/)] [License a Creative Commons - Attribution license]

##### 2.4 PCA transformation

In general, dataset will be spread across multiple axes in the n-dimensional space. It is natural that the spread (aka variance) will be different in the direction of those axes. The next question, we  ask is : Can we only pick the axes where the variances are high and remove the axes and data variances are low. Note that the maximum variance need not be in the direction of the original axes but in the direction which is the linear combination of the subset of the original axes. Principal component analysis (PCA) rotates the original data space such that the axes of the new coordinate system points to the direction of the highest variance of the data. User can choose a subset of the rotated axes thus resulting in the reduction of noise and also reduction in the data size.

See the following links for more details:

[nlpca](http://www.nlpca.org/pca_principal_component_analysis.html)

[wiki](https://en.wikipedia.org/wiki/Principal_component_analysis)

## 3. PCA Mathemathical formulation.

If we would like to approximate the real dataset with an approximate dataset, we would need to have some way to measure the discrepancy between real and approximate dataset. There are multiple ways to derive PCA, we will choose one of them i.e variance of the projected data onto the reduced rotated dimensional axes i.e principal component is maximized. Since having a maximum variances preserve as must information as possible as compare to minimum variances.

Before we move forward, let's define some of the terms:
##### 3.1 Convention Used.

| Symbol        | description                                                 |
|:--------------|:------------------------------------------------------------|
| $x_{i}$       | scalar is represented as lowercase with dimensional subscript $i$ |
| $\mathbf{x} $ | vector is represented as a bold lowercase                   |
| $\mathbf{X} $ | matrix is represented as bold uppercase                   |

##### 3.2. Definition of Statistical and Matrix Operations.

Lets defined some of terms related to Statistics and Matrix operations.

##### 3.2.1. Scalar Variable.

When we collect data each of the measurement is scalar. We could collect N samples of each scalar.

###### 3.2.2. Vector variable.

During data collection, we might measure various attributes, for example age, gender etc of the test subject.
To compactly represent M scalars for each subject i.e sample, vector is used, it is just a tuple of M scalars and uniquely describes a point in M dimensional space.

###### Represenation of a Sample using Vector.
Let the original input dataset be represented by M number of dimensions i.e each sample consist of M different 1D data point and together they form a sample data point column vector. $ \mathbf{x}  = (x_{1}, x_{2} \cdots x_{M})^{T} $.

Since we will be having N different data point, we would like to sub-script $ i $ for the i'th sample and the i'th data point vector is
$$ \mathbf{x_{i}}  = (x_{i1}, x_{i2} \cdots , x_{iM})^{T} $$

(**Note**: We use Transpose operation to compactly represent column vector as row vector.)

##### Representation of a Principal Component using Vector.

As we talked before that we will first project the standardized dataset on to K principal
components with $ K \in \{{1, 2, \cdots, M}\}$. Lets call $\mathbf{p_{j}}$ a M dimentional vector representing the direction of j'th principal
$$ \mathbf{p_{j}}  = (p_{j1}, p_{k2} \cdots , p_{jM})^{T} $$

There are a couple of points to be noted about $\mathbf{p_{j}}$

* each $\mathbf{p_{j}}$ lives in the space of M dimension i.e space of standardized or original data space i.e $dim(\mathbf{p_{j}}) == M$

* Since we are only interested in the direction we will assume i.e each of $p_{j}$ is [unit vector](https://en.wikipedia.org/wiki/Unit_vector) i.e $\|\mathbf{p_{j}}\|_{2}==1$ without loss of generality.

###### 3.2.3. Matrix Variable.

##### Representation of Whole Dataset.
As we noted in previous sub section that each subject or sample can be represented by one column vector. If we take N such measurement, we can compactly represent all those N measurements across M dimension using $ N \times M $ matrix.

$$ \mathbf{X} = ( \mathbf{x_{1}}^{T}, \mathbf{x_{2}}^{T}, \cdots  \mathbf{x_{i}}^{T} \cdots ,\mathbf{x_{n}}^{T} )^{T}
$$

###### Representation of K Principal Component Direction.

There are total K numbers of principal components i.e we have $\mathbf{p_{1}}, \mathbf{p_{2}}, \cdots , \mathbf{p_{k}}$ i.e we can compactly represent them as a $K X M $ matrix. $\mathbf{P} = ( \mathbf{p_{1}}^{T}, \mathbf{p_{2}}^{T}, \cdots  \mathbf{p_{i}}^{T} \cdots ,\mathbf{p_{k}}^{T} )^{T}$ with $ K \in \{{1, 2, \cdots, M}\}$.

##### 3.2.4. Standardization of Data Points.

To get standardized data, we will shift each of the dimensions by its mean. The advantage of this would be simpler math.

We would also like to divide it by dispersion i.e standard deviation because this will remove any bias associated with measuring units i.e measurement in inch vs cm etc.

it means new i'th sample is represented as $\mathbf{s_{i}}  = (s_{i1}, s_{i2} \cdots , s_{iM})^{T}$ i.e a column vector of standardized dimensions $s_{ij}$

Each of the i'th samples' j'th dimension is represented as follows

$$s_{ij} = \frac{x_{ij} - \overline{x_{j}}}{ \sigma_{x_{j}}}$$

Where
  * Average for j'th dimension $  \Rightarrow  \overline{x_{j}} = \frac{1}{N}\Sigma x_{ij} $

  * Standard Deviation for j'th dimension  $  \Rightarrow  \sigma_{x_{j}} = \sqrt[2]{\frac{1}{N-1}\sum_{i=1}^{N-1}(x_{ij} - \overline{x_{j}})^2}$

and new standardized data matrix of size $ N \times M $ becomes

$ \mathbf{S} = ( \mathbf{s_{1}}^{T}, \mathbf{s_{2}}^{T}, \cdots  \mathbf{s_{i}}^{T} \cdots ,\mathbf{s_{n}}^{T} )^{T}
$

###### 3.2.5. Variance of Scalar Variable.

  If we take N samples of a random variable $v$ with zero mean, then its [variance](https://en.wikipedia.org/wiki/Variance) is given by
$$variance(v) = \frac{1}{N}\sum_{i=1}^{N}v^2$$

##### 3.2.6. Length of a Vector.

M dimensional Vector is a point in M dimensional space and it has direction as well as magnitude. The idea of vector norm is motivated from the euclidean distance in our usual 3D space. If we have a point in 3D specified by vector $\mathbf{v} = (x, y, z)$ then its distance from origin or length is given by following expression

$$
\begin{equation}
\begin{split}
dist(\mathbf{v}, \mathbf{0}) &= norm_{2}(\mathbf{v}) \\
& = \|\mathbf{v}\|_{2} \\
& = \sqrt[2]{x^{2} + y^{2} + z^{2}} \\
& = \sqrt[2]{\mathbf{v}^{T}\mathbf{v}} && \text{from defn of dot_product)}
\end{split}
\end{equation}
$$

To generalize, above expression, if we have M dimensional vector $\mathbf{p_{k}}$ then its $norm_2$ is given as:

$$
$$
\begin{equation}
\begin{split}
dist(\mathbf{p_{k}}, \mathbf{0}) &= norm_{2}(\mathbf{p_{k}}) \\
& = \|\mathbf{p_{k}}\|_{2} \\
& = \sqrt[2]{ p_{k1}^{2} + p_{k2}^{2} + \cdots + p_{kj}^{2} + \cdots + p_{kM}^{2}} && \text{; $ p_{kj}\ \forall j \in \{{1, 2, \cdots, M}\} $ are the dimensions of vector $\mathbf{p_k}$}\\
& = \sqrt[2]{\mathbf{p_{k}}^{T}\mathbf{p_{k}}} && \text{;from defn of dot_product}
\end{split}
\end{equation}
$$
$$

###### 3.2.7. Vector Projection.
 [Vector projection](https://en.wikipedia.org/wiki/Vector_projection) is given by [dot product](https://en.wikipedia.org/wiki/Dot_product). i.e if we project vector $\mathbf{a}$ onto vector $\mathbf{b}$, then followings holds true:

$$ project\_\mathbf{a}\_onto\_\mathbf{b} = dot\_product(\mathbf{a}, \mathbf{b}) = \mathbf{a} \cdot \mathbf{b} = \mathbf{a}^{T} \mathbf{b} $$

 In our case, projection  on a M dimensional point $\mathbf{s_{i}}$ vector onto a principal component $\mathbf{p_{k}}$ is given by dot product as follows:

$$ project\_\mathbf{s_{i}}\_onto\_\mathbf{p_{k}} = dot\_product(\mathbf{s_{i}}, \mathbf{p_{k}}) = \mathbf{s_{i}} \cdot \mathbf{p_{k}} =  \mathbf{s_{i}}^{T} \mathbf{p_{k}}$$

##### 3.2.8. How to Represent a Scalar using Dot Product.

While deriving PCA, we will use this standard trick of representing a scalar as a dot product.
If a variable $v$ is scalar, then its square can be represented as dot product as follows:

$$ v^{2} = dot\_prodct(v, v) = v \cdot v = v^{T}v$$

i.e self dot product of scalar variable is just its multiplication with itself.

##### 3.2.9. Transpose of Product of Matrices.

 [Transpose of the product of Matrices $\mathbf{a}$ and $\mathbf{b}$](https://proofwiki.org/wiki/Transpose_of_Matrix_Product) is given by following:

$$
\begin{equation}
\begin{split}
(\mathbf{a}\mathbf{b})^{T} = \mathbf{b}^{T}\mathbf{a}^{T} && \text{  ; where $\mathbf{a}$ is $K X M$ and $\mathbf{b}$ is $M X K$ }
\end{split}
\end{equation}
$$

##### 3.2.10 Covariance of M Dimensional Vector.

If we collect N samples for one dimensional data, then we know that using mean and variance (aka sufficient statistics), we can get a good perspective on data. In short, using sufficient statistics i.e mean and variance allows us to have compression and remove the noise. The natural question is what to do when we have more than one dimensions to data i.e say M dimensions. Well how about characterizing it by M mean and M variances? This seems intuitively right, but here we are assuming that there is no co-relation or co-variance between any two dimensions. What if we want to account for this? The approach would be to compute variances between two dimensions and it is called co-variance. Since we have assumed without loss of generality that each dimension has zero mean. We can define covariance between them. Thus, with about $MxM$ parameters, we can represent our dataset.

We will assume the dataset as described in section 3.2.4 above. Where we collect N samples for each of M dimensions.

Let's also define $\mathbf{ss_{k}}$ as the k'th column of matrix $\mathbf{S}$ i.e $\mathbf{ss_{k}}$ represents all the samples we collected for the k'th dimensions and is defined as
$$ \mathbf{ss_{k}} = (s_{1k}, s_{2k}, \cdots ,s_{nk})^{T} $$

The [Co-variance Matrix](https://en.wikipedia.org/wiki/Covariance_matrix) is $M \times M$ matrix and is defined as:

$$\Sigma = (\Sigma_{ij}) \in \Re^{M \times M}$$

* Each $\Sigma_{ij}$ is a covariance of i'th and j'th dimensions and is given by

$$
\begin{equation}
\begin{split}
\Sigma_{ij} &= cov(\mathbf{ss_{i}}, \mathbf{ss_{j}}) \\
 &= \mathbb{E}(\mathbf{ss_{i}}, \mathbf{ss_{j}}) && \text{ ;from the definition of co-variance with zero means}\\
 &= \frac{1}{N} \sum_{m=1}^{N}(ss_{mi} \times {ss_{mj}})  && \text{ ;from the definition expectation}
\end{split}
\end{equation}
$$

Note that the above expression can be compactly represented in [co-variance matrix form](https://en.wikipedia.org/wiki/Covariance_matrix#Definition)
$$
\begin{equation}
\begin{split}
\Sigma &= \mathbb{E}(\mathbf{s}\mathbf{s}^{T}) && \text{ ;from definition of co-variance}\\
& = \frac{1}{N}\sum_{i=1}^{N}(\mathbf{s_{i}}\mathbf{s_{i}}^{T}) && \text{ ;from the definition expectation} \\
\end{split}
\end{equation}
$$

## 3.3 Derivation using Total Maximum Variances of Data Projection.

Now we are ready to derive the expression for all the principal components. Note that since, we would like to maximize the total projected variances. Total variances will be sum of the variances along each of the principal component. So to simplify our problem statement, we can find the expression for an arbitrary principal component i.e $\mathbf{p_{k}}$ and then generalize it for other principal components.

##### 3.3.1 Finding best Principal Component by Maximizing Projected Variances onto Principal Component.

We can write variance of the projection of M dimensional N points (i.e vectors) $\mathbf{s_{1}}, \mathbf{s_{1}}, \cdots , \mathbf{s_{n}}$ onto a arbitrary principal component $\mathbf{p_{k}} ,\ \ \ \forall k \in {\{1, 2, \cdots, M}\}$  as

$$
\begin{equation}
\begin{split}
  total\_projected\_variance\_on\_\mathbf{p_{k}} & = variance\_of\_projection\_of\_N\_points\_onto\_\mathbf{p_{k}} \\
  & = variance(projection(\mathbf{s_{1}}, \mathbf{p_{k}}), projection(\mathbf{s_{2}}, \mathbf{p_{k}}) \cdots, projection(\mathbf{s_{n}}, \mathbf{p_{k}}) ) \\
  & = variance(\mathbf{s_{1}}^{T}\mathbf{p_{k}}, \mathbf{s_{2}}^{T}\mathbf{p_{k}} \cdots,\mathbf{s_{n}}^{T}\mathbf{p_{k}} )) && \text{;see projection at 3.2.7} \\
  & = \frac{1}{N}\sum_{i=1}^{N}(\mathbf{s_{i}}^{T} \mathbf{p_{k}})^2 & & \text{;see variance at 3.2.5} \\
  & = \frac{1}{N} \sum_{i=1}^{N}(\mathbf{s_{i}}^{T} \mathbf{p_{k}})^{T}(\mathbf{s_{i}}^{T} \mathbf{p_{k}}) && \text{;Scalar and dot product representation (see 3.2.8 above)} \\
  & = \frac{1}{N} \sum_{i=1}^{N}(\mathbf{p_{k}}^{T}(\mathbf{s_{i}}^{T})^{T})(\mathbf{s_{i}}^{T} \mathbf{p_{k}}) && \text{;Transpose of matrix product (see 3.2.9 above)} \\
    & = \frac{1}{N} \sum_{i=1}^{N}(\mathbf{p_{k}}^{T}\mathbf{s_{i}})(\mathbf{s_{i}}^{T} \mathbf{p_{k}}) && \text{;Since transpose is an involution i.e self inverse i.e transpose of transpose is original vector} \\
  & = \frac{1}{N} \sum_{i=1}^{N}\mathbf{p_{k}}^{T}(\mathbf{s_{i}}\mathbf{s_{i}}^{T}) \mathbf{p_{k}} && \text{;Associativity of matrix product} \\
  & = \mathbf{p_{k}}^{T}(\frac{1}{N} \sum_{i=1}^{N}(\mathbf{s_{i}}\mathbf{s_{i}}^{T})) \mathbf{p_{k}} && \text{;Since summation doesn't involve k, we can bring terms containing k out of summation} \\
 & = \mathbf{p_{k}}^{T} \mathbf{\Sigma} \mathbf{p_{k}} && \text{;From the defn of covariance matrix as explained in 3.2.10}
\end{split}
\end{equation}
$$

As explained in previous sections, we would like to maximize the projected variances onto an arbitrary principal component. Above expression of total projected variances on an arbitrary principal component $\mathbf{p_{k}}$ can be cast as the constrained maximization problem aka as [Method of Lagrange Multipliers](https://en.wikipedia.org/wiki/Lagrange_multiplier).

What should be the constraints? The obvious constraint on vector $\mathbf{p_{k}}$ is that it has to be of constant length.

Why is it so? To understand it, lets assume that it is not constant and is variable. Since the expression of maximum projected variances involves product of  $\mathbf{p_{k}}$ and $\Sigma$,  the maximizing process will make length of $\mathbf{p_{k}}$ i.e $norm_2(\mathbf{p_{k}})$ (see 3.2.2  above), to be as maximum as possible which is infinity. And when $norm_2(\mathbf{p_{k}}) \rightarrow \infty $, we don't have a solution and hence it proves that $norm_2(\mathbf{p_{k}})$ must be constant to have a reasonable solution.

For mathematical convenience, we will choose $\mathbf{p_{k}}$'s length to be 1.

In compact notation, we have this numerical optimization formulation:

$$
\begin{equation}
\begin{split}
  & maximize(\mathbf{p_{k}}^{T} \mathbf{\Sigma} \mathbf{p_{k}}) \\
  & subject\ to\ \mathbf{p_{k}}^{T}\mathbf{p_{k}} = 1
\end{split}
\end{equation}
$$

**Note**: We use the definition of [unit vector](https://en.wikipedia.org/wiki/Unit_vector) i.e a vector whose norm is unity.

Using [Method of Lagrange Multipliers](https://en.wikipedia.org/wiki/Lagrange_multiplier), we can write above expression as

$$
 maximize(\mathbf{f}(\mathbf{p_{k}})) = maximize(\mathbf{p_{k}}^{T} \mathbf{\Sigma} \mathbf{p_{k}} + \lambda_{k}(1-\mathbf{p_{k}}^{T}\mathbf{p_{k}}))
$$
**Note**: $\lambda_{k}$ is positive scalar.

Now we must find, maximum of above expression, we know from calculus that a [function's maximum value can be found using the first and second derivatives](https://en.wikipedia.org/wiki/Derivative_test#First_derivative_test). What is our variable here? Since we want to find the unknown unit vector $\mathbf{p_{k}}$ it is our variable. (**Note**: even if magnitude is constant, direction is variable). Hence, we will differentiate it with $\mathbf{p_{k}}$.

Here we would advise user to look at the details of [Matrix Calculus](https://en.wikipedia.org/wiki/Matrix_calculus).

**First derivative is given by:**

$$
\begin{equation}
\begin{split}
\frac{\partial^1(\mathbf{f}(\mathbf{p_{k}}))}{\partial \mathbf{p_{k}}^1} &= \frac{\partial^1(\mathbf{p_{k}}^{T} \mathbf{\Sigma} \mathbf{p_{k}} + \lambda_{k}(1-\mathbf{p_{k}}^{T}\mathbf{p_{k}}))}{\partial \mathbf{p_{k}}^1} \\
& = 2\mathbf{\Sigma}\mathbf{p_{k}} - 2\lambda_{k}\mathbf{p_{k}}
\end{split}
\end{equation}
$$

The condition for critical points (i.e maximum or minimum) is that each component of first vector derivative should be zero i.e

$$2\mathbf{\Sigma}\mathbf{p_{k}} - 2\lambda_{k}\mathbf{p_{k}} = \mathbf{0}$$
$$\Rightarrow \mathbf{\Sigma}\mathbf{p_{k}} = \lambda_{k}\mathbf{p_{k}} $$

The above condition is saying that, if we have to maximize or minimize the projected variance onto $\mathbf{p_{k}}$, then
transformation done by the sample data's co-variance matrix must be equal to the scaling of the $\mathbf{p_{k}}$ and this is precisely the definition of [eigenvectors and eigenvalues](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors).

**Second derivative is given by:**
$$
\begin{equation}
\begin{split}
\frac{\partial^2(\mathbf{f}(\mathbf{p_{k}}))}{\partial \mathbf{p_{k}}^2} &= \frac{\partial^1 (\frac{\partial^1(\mathbf{p_{k}}^{T} \mathbf{\Sigma} \mathbf{p_{k}} + \lambda_{k}(1-\mathbf{p_{k}}^{T}\mathbf{p_{k}}))}{\partial \mathbf{p_{k}}^1})}{\partial \mathbf{p_{k}}^1} \\
& = \frac{\partial^1 (2\mathbf{\Sigma}\mathbf{p_{k}} - \lambda_{k}\mathbf{p_{k}})}{\partial \mathbf{p_{k}}^1} \\
& = 2\mathbf{\Sigma} - \lambda_{k}
\end{split}
\end{equation}
$$

The condition on a critical point (found by first derivative) to be maximum is that second vector derivative must be positive for each component. i.e

$$2\mathbf{\Sigma} - \lambda_{k} \succ \mathbf{0} \ \ \ \text{; Each element is greater than zero} $$
$$\Rightarrow \mathbf{\Sigma} \succ \frac{\mathbf{0} +  \lambda_{k}}{2}$$
$$\Rightarrow \mathbf{\Sigma} \succ \mathbf{0} \ \ \ \text{; Since $\lambda_{k}$ is positive}$$

The above condition is saying that co-variance matrix of data i.e $\mathbf{\Sigma}$ must be [positive definite](https://en.wikipedia.org/wiki/Positive-definite_matrix).

###### Additional Principal Components.

Note that in the previous sub-section, we didn't assume anything about $\mathbf{p_{k}}$. Hence, following the above procedure, we could find other principal components i.e $\mathbf{p_{1}}, \mathbf{p_{2}}, \cdots  \mathbf{p_{i}} \cdots ,\mathbf{p_{k}}$ with $ k <= M$. For  mathematical convenience, we choose all the [principal components to be orthogonal to each other](http://mathworld.wolfram.com/OrthogonalVectors.html). i.e

$$
\begin{equation}
dot\_product(\mathbf{p_{i}},\mathbf{p_{j}})  = \mathbf{p_{i}}\cdot\mathbf{p_{j}} = \mathbf{p_{i}}^{T}\mathbf{p_{j}} =
\begin{cases}
0 & \text{$ \forall \ i \neq j$}\\
1 & \text{$ \forall \ i = j$}
\end{cases}
\end{equation}
$$

Each of the $\mathbf{p_{i}}$ maximizes the projected variances onto the i'th principal component and can be computed using the eigenvalues and eigenvectors computation of the data samples' co-variance matrix.

Also, note that higher the eigenvalues, more is the total projected variances on the corresponding principal component.

###### Summary of Steps to Compute all the Principal Components.

* Collect $N$ samples of $M$ dimensional data.

* Standardize it to calculate $N \times M$ matrix $\mathbf{S}$ as in section 3.2.4 .

* Compute the $M \times M $ covariance matrix $\mathbf{\Sigma}$ as explained in section 3.2.6 .

* Compute the [eigenvector and eigenvalues](https://en.wikipedia.org/wiki/Eigenvalues_and_eigenvectors) of $\mathbf{\Sigma}$ using [SVD](https://en.wikipedia.org/wiki/Singular-value_decomposition).

* Order the eigenvector in decreasing order or eigenvalues.

* First vector is the first principal component vector, where the variances is maximum and so on.

* Using these techniques, you can compress the data set by using only first $k < M$ principal components.

## 4. Principal Component Analysis (PCA) in Practice.

We have seen, intuition and mathematical formulation of Principal Component Analysis. Now it's time to play around with real dataset. We will be using [Scikit-learn](http://scikit-learn.org/stable/) and [matplotlib](https://matplotlib.org/) to dive deep into examples.

We will look into following four use cases:

* [PCA for Dimension Reduction.](#dim_reduction)

* [PCA for Visualization and Better Insights.](#visualization)

* [PCA for Noise Filtering.](#filter_noise)

* [PCA as a Preprocessor for ML algorithms.](#pre_processor)

Let's load relevant library:

```python

# load high performance linalg lib
import numpy as np

# pandas util
import pandas as pd

# load PCA
from sklearn.decomposition import  PCA, KernelPCA

# load util for datasets
from sklearn import datasets

# load StandardScalar i.e shift by mean and scale by standard deviation
from sklearn.preprocessing import StandardScaler

# for visualization
import matplotlib.pyplot as plt
import seaborn as sns; sns.set()

# load util for datasets
from sklearn import datasets

```

We will use [Iris dataset](https://en.wikipedia.org/wiki/Iris_flower_data_set) for our initial experiments.

Let's first prepare dataset by standardizing:

```python
# load iris datasets
iris = datasets.load_iris()

# get X i.e input dataset
# get y i.e labelled prediction
X = iris.data
y = iris.target

# Standardization improves the performance of PCA.
S = StandardScaler().fit_transform(X)

# as discussed before lets standardized the data set

print("Features or Attributes of iris dataset = ")
print(iris["feature_names"])
print("\nSamples of input data X = \n")
print(X[:4,:])

print("\nAfter Standardization of Samples of input data X is S = \n")
print(S[:4,:])

```

**Note**: Standardization improves performance of PCA and if we don't standardized the data, output won't be that great and is illustrated by [this example](http://scikit-learn.org/stable/auto_examples/preprocessing/plot_scaling_importance.html#sphx-glr-auto-examples-preprocessing-plot-scaling-importance-py).

#### 4.1 PCA for Dimension Reduction (also achieving the data Compression).

As explained before that dimension reduction has many advantages, and we will illustrate of data compression using PCA.

```python

# create PCA transformer with 2 principal component
pca = PCA(n_components=2)

# fit input standardized dataset.
prcomp = pca.fit_transform(S)

print("First few dimensions of 2 principal components")
prcomp[:4,:]

```

**Note**: Principal components that we select are linear combinations of the input data columns"

Let's see, if our principal components contain enough information about the original dataset.

Recall that from our discussion in previous sections, we know that PCA maximizes the projected variances. So it's natural to ask that how much variances are contained withing first two principal components.

Apart from giving us principal components, PCA transformer also provides us with **explained_variance\_ratio\_**, which Scikit-learn provides so that we can know how much variances are captured by each of the principal components.

```python

print ("\nVariance captured by each principal component = ")
print(pca.explained_variance_ratio_)

print ("\nTotal Variance captured by principal components = ")
total_var = np.cumsum(pca.explained_variance_ratio_)[1]
print(total_var)

```

As we can see, that

* Iris dataset has four dimensions i.e 'sepal length (cm)', 'sepal width (cm)', 'petal length (cm)', 'petal width (cm)'. We can only visualize 3 dimensions. So lets reduce the dataset to lower dimensions. We will reduce it to 2 dimensions for our study purpose.

* By using 2 dimensions i.e principal component instead of 4 dimensions, we were able to save 50% of memory and thus achieving compression

* Since, 2 principal components captures more than 95% of total variance, only 5% information is lost with 50% of memory saving.

#### 4.2 PCA for Visualization and Better Insights.

Before we do apply ML algorithms, we would like to get better insights into datasets.
Note that original data generation process might be lower dimensions but after data collection it becomes higher dimensions. It is easier for human to visualize 3 dimensions or fewer. Since PCA reduces dimensions without much loss of information. Let's walk through an example of visualizing a dataset into lower dimensions.

Note that Iris dataset has 4 dimensions, we used PCA to reduce it to 2 dimensions, let's plot the data in the space spanned by first two principal components.

```python
targets = ['setosa', 'versicolor', 'virginica']

fig = plt.figure(figsize=(9, 9))
ax = fig.add_subplot(1, 1, 1)
ax.set_xlabel('First Principal Component', fontsize = 10)
ax.set_ylabel('Second Principal Component', fontsize = 10)
ax.set_title('Plotting data using two principal component', fontsize = 15)

plt_pr_df = pd.DataFrame(data=prcomp, columns=['pc_1', 'pc_2'])
plt_y_df = pd.DataFrame(data=y, columns=['y'])
plt_df = pd.concat([plt_pr_df, plt_y_df], axis=1)
ax.scatter(plt_df['pc_1'], plt_df['pc_2'],
            c=plt_df['y'], edgecolor='none', alpha=0.8,
            cmap=plt.cm.get_cmap('RdYlBu_r', 3))

#plt.colorbar();
ax.legend(targets)
plt.show()
```

From the above figure, it is clear that using PCA , we can clearly see three different classes. And thus PCA helps in data visualization and in exploratory analysis, we can get better insights into the data without running any classifier.

#### 4.3 PCA for Noise Filtering.

Since PCA select the dimensional axis with maximum total projected variance after optimally rotating the dataset, it follows naturally that if there is the noise in the input dataset and if we apply PCA on it, it will filter out the noise since variance would be much smaller for noise.

To bring point home, we will take one image and artificially introduce various noise and see if PCA can remove noise.

We will load one of the images that is shipped with Scikit-learn. Let's first write some of utilities which we will be using multiple times:

1. **Normalize Image**: Since image rendered utility in scikit-learn can only display images in the range of 0-1, normalize image utility will divide it with 255 (which is the maximum color levels for RGB format). In above process, introduced noises might make image values to be less than 0 or more than 1, we will also make sure that the range is between 0 and 1.

2. **Plot Image**: This utility plots image using matplotlib utilities.

3. **Add Guassian Noise**:
  Since, in real life, [Guassian noise](https://en.wikipedia.org/wiki/Gaussian_noise) is very common phenomenon, we will add a routine to add Guassian noise. This simulate the situation when we take a picture with noises i.e
    * There is over lighting like some flash or sun light, which is modelled by mean of color value.
    * And some noises like flickering which is introduced by variance of guassian distribution.

```python

def normalize_image(img):
    """
    Convert to floats instead of the default 8 bits integer coding. Dividing by
    255 is important so that plt.imshow behaves works well on float data (need to
    be in the range [0-1]). Basic idea is this.
    norm_image = np.array(image, dtype=np.float64) / 255
    """
    norm_img = np.array(img, dtype=np.float64) / 255

    w, h, d = original_shape = tuple(norm_img.shape)

    norm_img_1d = np.reshape(norm_img, (w * h* d, 1))
    norm_img_1d[norm_img_1d > 1.0] = 1
    norm_img_1d[norm_img_1d < 0.0] = 0
    norm_img = np.reshape(norm_img_1d, (w, h, d))
    return norm_img

def plot_color_image(img, title="Image"):
    """
    Utilty to plot image.
    """
    plt.figure(1)
    plt.clf()
    ax = plt.axes([0, 0, 1, 1])
    plt.axis('off')
    plt.title(title)
    plt.imshow(img)
    plt.show()

def add_guassian_noise(image):
    """
    Add guassian noise. This simulate the situation when we take a picture
    with noises
    1) there is over lighting like some flash or sun light, which is introduce by mean 255
    2) and some noise like flickering which is introduce by variance 100
    """
    out = image + np.random.normal(255, 100, image.shape)
    return out

```

Let's load the "china.jpg" image and add some Guassian noise. This simulate the situation when we take a picture with noises.

```python
# load china image
china_org = datasets.load_sample_image('china.jpg')

w, h, d = original_shape = tuple(china_org.shape)

#plot_color_image(normalize_image(china_org))

# print(china_org.shape)

# add guassian noise to image
china_noisy_guass = add_guassian_noise(china_org)

china_noisy = china_noisy_guass

# plot the noisy image
plot_color_image(normalize_image(china_noisy), "Image with Guassian noise added")

print(" Image dimensions = ", china_noisy.shape)
```

What this image is? Kind of very difficult to answer it. Now lets use PCA to do noise reduction.

Typically, a image is represented by 2D array of pixels, where each pixel is of RGB color and there are $width \times height$ pixels. Thus, total dimensions of this image is $427 \times 640 \times 3 = 819840 $ i.e image has about $800k$ dimensions

Let's apply linear PCA to this.

```python
china_norm = normalize_image(china_org)

# we will combine color and height dimensions so as to confirm to api of linear PCA and KernelPCA

china_noisy_2d = np.reshape(china_noisy, (w ,h * d))

```

```python
pca = PCA(0.9999)

pca = pca.fit(china_noisy_2d)

print("New dimensions of image = ", pca.n_components_)
print("Image reconstruction after linear PCA noise filtering")

# transform the noisy image using linear PCA
components = pca.transform(china_noisy_2d)
filtered = pca.inverse_transform(components)

# confirm image into orig 3D
china_filtered = np.reshape(filtered, (w , h, d))

plot_color_image(normalize_image(china_filtered), "Image with noise filtered by linear PCA")

```

We observe that:

1. Image's Dimensions reduces significantly from $800,000 $ to around $426$ thus resulting in big saving in data space storage in memory.

2. But image is almost same as noisy image in previous sections.

To understand second issue, we have to understand that image is actually highly non-linear dataset and hence dimensions reduction using linear PCA probably does not achieve the best result.

Is there any better way to reconstructing image? Since image is non-linear data, how about using [KernelPCA](https://en.wikipedia.org/wiki/Kernel_principal_component_analysis)?
KernelPCA performs PCA in a new space whose dimensions are nonlinear. Interested user are encouraged to look into details about KernelPCA in previous links.

Kernel PCA is very typically used for:

* de-noising
* novelty detection

```python

# create a kernel PCA transformer
# -with inverse_transform so that we can reduce the noise
# -gamma controls the strength of cohesiveness of two samples. higher gamma means not so much affected by noise.
# -we will remove zero eigenvector
# -we will let KernelPCA determine best number of dimensions

kpca = KernelPCA(kernel="rbf", fit_inverse_transform=True, gamma=1000, remove_zero_eig = True, n_components = None)

kpca = kpca.fit(china_noisy_2d)

kcomponents = kpca.transform(china_noisy_2d)

# do inverse transform so as to filter our noise.
kfiltered = kpca.inverse_transform(kcomponents)

# get the 3D image from 2D versoin
china_kfiltered = np.reshape(kfiltered, (w , h, d))

plot_color_image(normalize_image(china_kfiltered), "Image with noise filterd by Kernel PCA")
```

Here, we can see that this is a picture of trees with a tower. Much better than images with noise and image with noise reduction by linear PCA.

Now lets look at the original image.

```python
plot_color_image(normalize_image(china_org), "Original Image with 65k colors")

```

Thus, we saw that using Kernel PCA, we can remove noise from images.

#### 4.4 PCA as a Preprocessor for ML algorithms.

We have seen how PCA rotates the dataset so that the dimensional axis align with maximum variance i.e eigenvectors of the Standardized Co-variance Matrix. Doing that allows us, to remove the dimensional axis where the variance is very little and thus achieving

1) Data Compression and Dimensional Reduction.
2) Reducing and Filtering Noise.
3) Making dataset more suitable for applying Machine Learning Algorithms

Last point is very important, as we know that most of the machine learning algorithms will perform great, when the data set is more relevant. This property allows us to us PCA as the pre-processing steps to machine learning algorithms (i.e logistic regression, support vector machine etc) and thus achieving better performance.

We will not go over the examples in this notebook but interested user can look at following examples to get more details

* [Using PCA increase performance of faces detection using SVM classification](http://scikit-learn.org/stable/auto_examples/applications/plot_face_recognition.html#sphx-glr-auto-examples-applications-plot-face-recognition-py).

* [Using PCA to increase performance of K-means clustering](http://scikit-learn.org/stable/auto_examples/cluster/plot_kmeans_digits.html#sphx-glr-auto-examples-cluster-plot-kmeans-digits-py).

## 5. Summary.

Thus we have given a great details about PCA and various perspective.

* First, we looked at the PCA intuition.
* Second, we derived PCA from mathematical perspective and connected intuition with rigorous analyis.
* Then, we looked at various examples to illustrate various PCA applications like:
    * Dimension Reduction.
    * Visualization.
    * Noise Filtering.
    * PCA as pre-processor for better performance of ML algorithms.

We encourage user to try out various more examples and explore PCA to get deeper insights.
Interested users are encouraged to looked at other alternatives to PCA:

* [T-SNE for visualization](https://en.wikipedia.org/wiki/T-distributed_stochastic_neighbor_embedding).
* [Isomap non linear dimensional reduction](https://en.wikipedia.org/wiki/Isomap).
* [Laplacian Eigenmaps for dimensionality Reduction](https://en.wikipedia.org/wiki/Nonlinear_dimensionality_reduction#Laplacian_eigenmaps).


# Tutorial
## Preliminaries
Before following this tutorial we need to set up the tools and load the data. We need to import several packages, so before running this notebook you should create an environment (conda or virtualenv) with matplotlib, numpy, and scikit-image, and jupyter.

You can use the Anaconda Navigator to do this, see:

https://docs.continuum.io/anaconda/navigator/getting-started.html

Or from the terminal (command window), for example:

`conda create -n BIO399E jupyter matplotlib numpy scipy scikit-image`

and then activate it, e.g. on Mac/Linux:

`source activate BIO399E`

or on windows

`activate BIO399E`

or select the conda env here in Jupyter.

## Modules
First import the standard tools, numpy and matplotlib. These are very well documented packages, more info can be found here:

http://www.matplotlib.org

http://www.numpy.org

```python
import numpy as np
import matplotlib
np.__version__
```

## Numpy arrays
Lists are a simple way to store collections of data, but they are not very flexible. To deal with numerical data it is better is to use a package called numpy, which stores data in n-dimensional arrays. The simplest is a lot like a list, and we can make it from a list:

```python
days = [31,28,31,30,31,30,31,31,30,31,30,31]
adays = np.array(days)
print adays
```

We can also make an array from scratch and fill it with zeros, ones, random values, etc, and combine arrays to compute functions:

```python
a = np.zeros((12,))
x = np.ones((100,))
y = np.random.random((12,))

z = np.arange(1,13)
w = z*5 + 0.5

print x
print y
print z
print w
```

Now we have the days in a numpy array we can use functions from numpy to easily do the control:

```python
# Calculate mean using numpy
average_days = adays.mean()
print adays
print average_days
```

```python
# Calculate variance using numpy
var_days = adays.var()
print var_days

# Calculate variance step-by-step (same result)
devs = (adays-average_days)**2
print devs

sum_devs = devs.sum()
print sum_devs

var_days = sum_devs/len(days)
print var_days
```

### 2-dimensional arrays
Numpy can handle arrays of any number dimensions. For example for images we will use 2-dimensional arrays (in a later class). Here is how to make a 2-d array:

```python
# A 2-dimensional array with random values
twod = np.random.random((10,5))
print twod

# Means (variance, etc.) are computed along specified axes
print np.mean(twod, axis=1)
print np.mean(twod, axis=(0,1))
```

## Exercise 1
As we saw above Numpy has many functions that perform calculations on arrays, see here:

https://docs.scipy.org/doc/numpy-1.13.0/reference/arrays.ndarray.html#array-methods

One of these functions is to load data from a text file.

In this folder you will find 3 text files, 'fluo.csv', 'od.csv', and 't.csv' (time). These files contain comma separated lists of numbers corresponding to fluorescence and od measurements at times t.

1.1 Write code to load these files into 3 numpy arrays `fluo`, `od` and `t`:

```python

```

1.2 Now calculate the mean and variance of each data set. What else can you calculate with numpy that might be useful?

```python

```

## Matplotlib, making graphs
Matplotlib is a module that works well with numpy arrays, and can make many kinds of graphs, charts, heatmaps etc. The part of the module that does plotting is called pyplot, we import it like this with some magic to put plots in this notebook:

```python
import matplotlib.pyplot as plt
%matplotlib inline
```

For example, here is a simple plot of the fluorescence:

```python
# Simple plot
plt.plot(fluo)
# We can also choose the color, point shape etc.
plt.plot(fluo, 'g+')
```

Make use of the matplotlib documentation to do the following:

## Exercise 2
2.1 Make a plot for each data set, `fluo`, `od` against time `t`. Label the axes and give the plot a title.

```python

```

2.2 Do the same as above, but plot the log of the data:

```python

```

2.3 Plot `fluo` and `od` in the same plot. Try and make the axes different so that you can really see od:

```python

```

2.4 Plot `fluo/od` for all times:

```python

```

## Exercise 3
3.1 Plot histograms of `fluo` and `od`:

```python

```

3.2 Plot `fluo` against `od`:

```python

```

3.3 Calculate the correlation between `fluo` and `od`:

```python

```

## Data analysis, calculating gene expression

Here is a simple model for fluorescent gene expression from a single cell:

\begin{equation}
\frac{dF}{dt} = k(t) - \mu(t) F(t)
\end{equation}

where $\mu(t)$ is the growth rate. If we have $N \approx OD$ cells, then we measure the sum of their gene expression $I(t) = F(t)OD(t)$ and we can show that:

\begin{equation}
k(t) = \frac{1}{OD}\frac{dI}{dt}
\end{equation}

## Exercise 4 (Advanced)
Given the data above with $I(t)$ in the variable `fluo`, how would you calculate the gene expression rate at each time `t`?

```python

```