# Lambda School Data Science Module 132
## Sampling, Confidence Intervals, and Hypothesis Testing

## Prepare - examine other available hypothesis tests

If you had to pick a single hypothesis test in your toolbox, t-test would probably be the best choice - but the good news is you don't have to pick just one! Here's some of the others to be aware of:

```
import numpy as np
from scipy.stats import chisquare  # One-way chi square test

# Chi square can take any crosstab/table and test the independence of rows/cols
# The null hypothesis is that the rows/cols are independent -> low chi square
# The alternative is that there is a dependence -> high chi square
# Be aware! Chi square does *not* tell you direction/causation

ind_obs = np.array([[1, 1], [2, 2]]).T
print(ind_obs)
print(chisquare(ind_obs, axis=None))

dep_obs = np.array([[16, 18, 16, 14, 12, 12], [32, 24, 16, 28, 20, 24]]).T
print(dep_obs)
print(chisquare(dep_obs, axis=None))
```

    [[1 2]
     [1 2]]
    Power_divergenceResult(statistic=0.6666666666666666, pvalue=0.8810148425137847)
    [[16 32]
     [18 24]
     [16 16]
     [14 28]
     [12 20]
     [12 24]]
    Power_divergenceResult(statistic=23.31034482758621, pvalue=0.015975692534127565)

```
# Distribution tests:
# We often assume that something is normal, but it can be important to *check*

# For example, later on with predictive modeling, a typical assumption is that
# residuals (prediction errors) are normal - checking is a good diagnostic

from scipy.stats import normaltest
# Poisson models arrival times and is related to the binomial (coinflip)
sample = np.random.poisson(5, 1000)
print(normaltest(sample))  # Pretty clearly not normal
```

    NormaltestResult(statistic=32.74993223644408, pvalue=7.734683967546769e-08)

```
# Kruskal-Wallis H-test - compare the median rank between 2+ groups
# Can be applied to ranking decisions/outcomes/recommendations
# The underlying math comes from chi-square distribution, and is best for n>5
from scipy.stats import kruskal

x1 = [1, 3, 5, 7, 9]
y1 = [2, 4, 6, 8, 10]
print(kruskal(x1, y1))  # x1 is a little better, but not "significantly" so

x2 = [1, 1, 1]
y2 = [2, 2, 2]
z = [2, 2]  # Hey, a third group, and of different size!
print(kruskal(x2, y2, z))  # x clearly dominates so p is small
```

    KruskalResult(statistic=0.2727272727272734, pvalue=0.6015081344405895)
    KruskalResult(statistic=7.0, pvalue=0.0301973834223185)

And there's many more! `scipy.stats` is fairly comprehensive, though there are even more available if you delve into the extended world of statistics packages. As tests get increasingly obscure and specialized, the importance of knowing them by heart becomes small - but being able to look them up and figure them out when they *are* relevant is still important.

## Write functions

Here you could run any hypothesis test on a dataset for an interview. Like you might write a function to check if a column of integer datatype is normally distributed.

## T-test Assumptions

```
from scipy.stats import ttest_ind

?ttest_ind
```

<https://statistics.laerd.com/statistical-guides/independent-t-test-statistical-guide.php>

- Independence of means

Are the means of our voting data independent (do not affect the outcome of one another)?

The best way to increase thel likelihood of our means being independent is to randomly sample (which we did not do).

- "Homogeneity" of Variance?

Is the magnitude of the variance between the two roughly the same?

I think we're OK on this one for the voting data, although it probably could be better, one party was larger than the other.

If we suspect this to be a problem then we can use Welch's T-test

- "Dependent Variable" (sample means) are Distributed Normally

<https://stats.stackexchange.com/questions/9573/t-test-for-non-normal-when-n50>

Lots of statistical tests depend on normal distributions. We can test for normality using Scipy as was shown above.

This assumption is often assumed even if the assumption is a weak one. If you strongly suspect that things are not normally distributed, you can transform your data to get it looking more normal and then run your test. This problem typically goes away for large sample sizes (yay Central Limit Theorem) and is often why you don't hear it brought up. People declare the assumption to be satisfied either way.

## Central Limit Theorem

```
np.random.binomial(n=1, p=.5, size=30)
```

    array([1, 0, 1, 0, 1, 1, 1, 0, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0,
           1, 0, 1, 0, 0, 0, 1, 1])

```
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
N = 3000
sample_means = []
for x in range(0,N):
  coinflips = np.random.binomial(n=1, p=.5, size=100)
  one_sample = coinflips
  sample_means.append(coinflips.mean())

print(len(sample_means))
print(sample_means)
```

    3000
    [0.57, 0.54, 0.46, 0.48, 0.5, 0.51, 0.48, 0.49, 0.51, 0.49, 0.39, 0.51, 0.48, 0.55, 0.46, 0.48, 0.61, 0.48, 0.54, 0.56, 0.52, 0.5, 0.5, 0.45, 0.47, 0.52, 0.53, 0.49, 0.44, 0.54, 0.44, 0.47, 0.48, 0.54, 0.36, 0.49, 0.58, 0.38, 0.49, 0.55, 0.57, 0.56, 0.43, 0.49, 0.49, 0.49, 0.52, 0.48, 0.5, 0.51, 0.45, 0.49, 0.48, 0.5, 0.54, 0.52, 0.42, 0.5, 0.37, 0.47, 0.41, 0.52, 0.43, 0.51, 0.61, 0.51, 0.56, 0.41, 0.48, 0.5, 0.48, 0.46, 0.33, 0.46, 0.55, 0.54, 0.57, 0.49, 0.55, 0.45, 0.48, 0.48, 0.46, 0.54, 0.51, 0.57, 0.46, 0.52, 0.53, 0.54, 0.5, 0.47, 0.54, 0.47, 0.51, 0.49, 0.48, 0.4, 0.49, 0.6, 0.53, 0.48, 0.36, 0.34, 0.42, 0.45, 0.49, 0.45, 0.59, 0.6, 0.43, 0.46, 0.51, 0.53, 0.56, 0.48, 0.53, 0.5, 0.52, 0.5, 0.53, 0.47, 0.48, 0.48, 0.54, 0.36, 0.48, 0.53, 0.45, 0.57, 0.45, 0.48, 0.48, 0.5, 0.5, 0.49, 0.55, 0.51, 0.46, 0.57, 0.48, 0.47, 0.46, 0.47, 0.51, 0.49, 0.47, 0.51, 0.55, 0.53, 0.54, 0.48, 0.52, 0.46, 0.43, 0.57, 0.53, 0.57, 0.53, 0.5, 0.55, 0.46, 0.48, 0.44, 0.44, 0.56, 0.51, 0.46, 0.44, 0.5, 0.49, 0.57, 0.52, 0.55, 0.41, 0.42, 0.57, 0.43, 0.57, 0.55, 0.49, 0.45, 0.55, 0.42, 0.5, 0.51, 0.53, 0.47, 0.57, 0.55, 0.47, 0.44, 0.47, 0.53, 0.54, 0.48, 0.46, 0.49, 0.49, 0.39, 0.56, 0.5, 0.48, 0.47, 0.44, 0.52, 0.55, 0.42, 0.51, 0.54, 0.54, 0.52, 0.44, 0.51, 0.58, 0.51, 0.47, 0.54, 0.5, 0.53, 0.51, 0.51, 0.49, 0.48, 0.51, 0.44, 0.51, 0.42, 0.55, 0.37, 0.52, 0.53, 0.5, 0.45, 0.47, 0.43, 0.45, 0.52, 0.5, 0.51, 0.48, 0.46, 0.48, 0.5, 0.48, 0.46, 0.43, 0.41, 0.45, 0.46, 0.5, 0.43, 0.47, 0.48, 0.48, 0.57, 0.42, 0.47, 0.47, 0.45, 0.51, 0.53, 0.51, 0.52, 0.5, 0.44, 0.52, 0.44, 0.46, 0.5, 0.47, 0.4, 0.44, 0.5, 0.46, 0.52, 0.46, 0.55, 0.57, 0.55, 0.48, 0.45, 0.53, 0.47, 0.52, 0.45, 0.48, 0.44, 0.41, 0.38, 0.54, 0.44, 0.57, 0.45, 0.5, 0.57, 0.47, 0.54, 0.46, 0.44, 0.51, 0.52, 0.48, 0.55, 0.42, 0.48, 0.45, 0.46, 0.53, 0.44, 0.45, 0.48, 0.53, 0.49, 0.47, 0.51, 0.53, 0.58, 0.46, 0.51, 0.52, 0.49, 0.54, 0.52, 0.52, 0.49, 0.51, 0.51, 0.37, 0.39, 0.51, 0.5, 0.45, 0.47, 0.45, 0.56, 0.51, 0.51, 0.54, 0.51, 0.42, 0.41, 0.5, 0.55, 0.5, 0.47, 0.51, 0.51, 0.53, 0.43, 0.56, 0.51, 0.59, 0.41, 0.5, 0.52, 0.45, 0.61, 0.5, 0.39, 0.5, 0.52, 0.57, 0.4, 0.52, 0.49, 0.48, 0.45, 0.51, 0.57, 0.56, 0.52, 0.49, 0.58, 0.49, 0.5, 0.35, 0.53, 0.51, 0.53, 0.49, 0.55, 0.48, 0.45, 0.63, 0.46, 0.56, 0.4, 0.51, 0.47, 0.39, 0.49, 0.53, 0.49, 0.44, 0.47, 0.44, 0.51, 0.42, 0.55, 0.47, 0.48, 0.5, 0.54, 0.5, 0.5, 0.47, 0.52, 0.55, 0.58, 0.57, 0.46, 0.49, 0.48, 0.51, 0.44, 0.47, 0.45, 0.43, 0.48, 0.52, 0.43, 0.44, 0.52, 0.56, 0.51, 0.48, 0.42, 0.47, 0.56, 0.39, 0.57, 0.48, 0.47, 0.49, 0.41, 0.45, 0.58, 0.47, 0.49, 0.45, 0.47, 0.48, 0.47, 0.44, 0.53, 0.41, 0.57, 0.51, 0.5, 0.52, 0.42, 0.51, 0.48, 0.46, 0.44, 0.49, 0.47, 0.52, 0.52, 0.59, 0.5, 0.52, 0.55, 0.56, 0.51, 0.43, 0.51, 0.46, 0.51, 0.44, 0.51, 0.45, 0.5, 0.51, 0.45, 0.55, 0.55, 0.61, 0.48, 0.5, 0.51, 0.45, 0.54, 0.53, 0.53, 0.58, 0.46, 0.53, 0.58, 0.41, 0.53, 0.48, 0.53, 0.5, 0.54, 0.46, 0.5, 0.46, 0.53, 0.5, 0.54, 0.56, 0.42, 0.48, 0.47, 0.65, 0.52, 0.53, 0.51, 0.56, 0.51, 0.48, 0.58, 0.49, 0.47, 0.46, 0.47, 0.53, 0.5, 0.49, 0.55, 0.47, 0.49, 0.5, 0.51, 0.48, 0.57, 0.46, 0.5, 0.45, 0.44, 0.51, 0.47, 0.51, 0.5, 0.53, 0.43, 0.64, 0.52, 0.44, 0.54, 0.55, 0.47, 0.52, 0.51, 0.51, 0.55, 0.54, 0.58, 0.55, 0.51, 0.44, 0.64, 0.54, 0.49, 0.54, 0.48, 0.58, 0.52, 0.56, 0.44, 0.57, 0.53, 0.48, 0.39, 0.43, 0.38, 0.38, 0.44, 0.51, 0.53, 0.56, 0.53, 0.51, 0.46, 0.54, 0.48, 0.5, 0.43, 0.43, 0.44, 0.5, 0.49, 0.41, 0.44, 0.53, 0.46, 0.44, 0.55, 0.49, 0.52, 0.45, 0.42, 0.49, 0.42, 0.49, 0.56, 0.51, 0.47, 0.43, 0.55, 0.59, 0.46, 0.51, 0.59, 0.42, 0.44, 0.52, 0.4, 0.59, 0.43, 0.52, 0.56, 0.49, 0.5, 0.5, 0.51, 0.55, 0.54, 0.49, 0.45, 0.48, 0.5, 0.5, 0.54, 0.47, 0.5, 0.58, 0.49, 0.54, 0.44, 0.48, 0.47, 0.54, 0.56, 0.48, 0.43, 0.6, 0.51, 0.49, 0.42, 0.46, 0.58, 0.48, 0.46, 0.4, 0.51, 0.49, 0.53, 0.46, 0.46, 0.55, 0.45, 0.58, 0.51, 0.48, 0.43, 0.58, 0.49, 0.52, 0.51, 0.48, 0.46, 0.58, 0.51, 0.48, 0.4, 0.51, 0.54, 0.45, 0.53, 0.56, 0.51, 0.56, 0.51, 0.39, 0.44, 0.53, 0.55, 0.53, 0.56, 0.54, 0.57, 0.59, 0.59, 0.5, 0.44, 0.46, 0.51, 0.47, 0.53, 0.53, 0.5, 0.54, 0.51, 0.57, 0.51, 0.4, 0.47, 0.49, 0.53, 0.51, 0.47, 0.49, 0.42, 0.59, 0.43, 0.49, 0.54, 0.44, 0.51, 0.51, 0.54, 0.54, 0.51, 0.47, 0.51, 0.47, 0.6, 0.53, 0.48, 0.43, 0.54, 0.48, 0.52, 0.49, 0.47, 0.48, 0.49, 0.46, 0.48, 0.44, 0.45, 0.51, 0.5, 0.53, 0.52, 0.52, 0.54, 0.45, 0.47, 0.51, 0.52, 0.49, 0.48, 0.48, 0.45, 0.55, 0.47, 0.48, 0.56, 0.52, 0.52, 0.46, 0.48, 0.49, 0.45, 0.53, 0.53, 0.53, 0.55, 0.58, 0.5, 0.4, 0.49, 0.52, 0.45, 0.55, 0.44, 0.54, 0.44, 0.56, 0.47, 0.59, 0.5, 0.52, 0.49, 0.62, 0.52, 0.52, 0.49, 0.57, 0.47, 0.49, 0.6, 0.46, 0.44, 0.51, 0.45, 0.48, 0.49, 0.61, 0.59, 0.5, 0.48, 0.46, 0.5, 0.51, 0.53, 0.44, 0.48, 0.51, 0.54, 0.49, 0.57, 0.58, 0.55, 0.53, 0.43, 0.55, 0.49, 0.49, 0.44, 0.45, 0.49, 0.54, 0.41, 0.58, 0.61, 0.49, 0.48, 0.47, 0.46, 0.62, 0.52, 0.52, 0.5, 0.53, 0.59, 0.52, 0.47, 0.48, 0.51, 0.54, 0.5, 0.53, 0.51, 0.51, 0.52, 0.46, 0.43, 0.46, 0.53, 0.6, 0.54, 0.51, 0.49, 0.42, 0.5, 0.41, 0.44, 0.58, 0.46, 0.47, 0.55, 0.47, 0.47, 0.47, 0.5, 0.55, 0.46, 0.57, 0.55, 0.47, 0.43, 0.44, 0.5, 0.51, 0.46, 0.48, 0.56, 0.44, 0.46, 0.46, 0.55, 0.47, 0.62, 0.42, 0.49, 0.44, 0.6, 0.58, 0.49, 0.61, 0.5, 0.53, 0.53, 0.45, 0.56, 0.48, 0.43, 0.56, 0.44, 0.48, 0.52, 0.42, 0.45, 0.41, 0.52, 0.55, 0.48, 0.58, 0.5, 0.51, 0.44, 0.44, 0.5, 0.53, 0.54, 0.47, 0.56, 0.51, 0.5, 0.52, 0.5, 0.42, 0.54, 0.41, 0.46, 0.63, 0.55, 0.49, 0.53, 0.55, 0.48, 0.5, 0.42, 0.59, 0.5, 0.47, 0.47, 0.47, 0.43, 0.52, 0.46, 0.44, 0.43, 0.49, 0.55, 0.36, 0.53, 0.51, 0.53, 0.51, 0.48, 0.51, 0.58, 0.54, 0.46, 0.52, 0.49, 0.53, 0.43, 0.58, 0.53, 0.47, 0.46, 0.59, 0.5, 0.53, 0.52, 0.47, 0.59, 0.47, 0.44, 0.56, 0.54, 0.41, 0.5, 0.54, 0.38, 0.5, 0.51, 0.6, 0.5, 0.54, 0.49, 0.61, 0.53, 0.52, 0.39, 0.54, 0.51, 0.47, 0.48, 0.49, 0.49, 0.47, 0.46, 0.44, 0.46, 0.53, 0.53, 0.55, 0.52, 0.47, 0.45, 0.51, 0.53, 0.46, 0.56, 0.48, 0.52, 0.46, 0.56, 0.4, 0.52, 0.5, 0.39, 0.55, 0.45, 0.53, 0.46, 0.51, 0.49, 0.48, 0.56, 0.49, 0.59, 0.41, 0.59, 0.49, 0.44, 0.47, 0.56, 0.55, 0.54, 0.46, 0.42, 0.49, 0.51, 0.54, 0.45, 0.49, 0.49, 0.5, 0.51, 0.49, 0.45, 0.46, 0.48, 0.45, 0.57, 0.57, 0.56, 0.52, 0.54, 0.47, 0.52, 0.44, 0.46, 0.53, 0.43, 0.57, 0.39, 0.5, 0.53, 0.51, 0.55, 0.46, 0.41, 0.44, 0.46, 0.46, 0.49, 0.46, 0.48, 0.59, 0.54, 0.55, 0.47, 0.53, 0.5, 0.54, 0.6, 0.55, 0.56, 0.6, 0.47, 0.46, 0.6, 0.42, 0.53, 0.47, 0.52, 0.44, 0.47, 0.49, 0.44, 0.56, 0.5, 0.46, 0.46, 0.56, 0.58, 0.41, 0.57, 0.53, 0.44, 0.42, 0.5, 0.55, 0.47, 0.53, 0.44, 0.57, 0.48, 0.49, 0.53, 0.56, 0.46, 0.47, 0.42, 0.51, 0.56, 0.61, 0.45, 0.37, 0.53, 0.56, 0.48, 0.42, 0.43, 0.48, 0.52, 0.53, 0.52, 0.57, 0.43, 0.45, 0.53, 0.5, 0.58, 0.52, 0.5, 0.48, 0.43, 0.58, 0.57, 0.51, 0.53, 0.52, 0.41, 0.55, 0.51, 0.49, 0.59, 0.54, 0.45, 0.36, 0.55, 0.51, 0.51, 0.5, 0.45, 0.47, 0.45, 0.38, 0.52, 0.51, 0.53, 0.51, 0.6, 0.39, 0.46, 0.53, 0.55, 0.51, 0.52, 0.48, 0.48, 0.56, 0.5, 0.57, 0.54, 0.53, 0.48, 0.55, 0.56, 0.47, 0.52, 0.49, 0.47, 0.52, 0.44, 0.48, 0.5, 0.54, 0.5, 0.46, 0.43, 0.47, 0.51, 0.45, 0.4, 0.54, 0.51, 0.57, 0.47, 0.57, 0.53, 0.52, 0.46, 0.49, 0.59, 0.56, 0.55, 0.47, 0.48, 0.47, 0.51, 0.54, 0.46, 0.45, 0.49, 0.56, 0.49, 0.4, 0.45, 0.46, 0.53, 0.5, 0.44, 0.56, 0.45, 0.56, 0.45, 0.45, 0.5, 0.54, 0.5, 0.49, 0.5, 0.42, 0.5, 0.56, 0.47, 0.46, 0.45, 0.43, 0.46, 0.51, 0.45, 0.45, 0.51, 0.55, 0.53, 0.5, 0.45, 0.55, 0.46, 0.46, 0.55, 0.45, 0.43, 0.5, 0.44, 0.43, 0.49, 0.48, 0.59, 0.46, 0.44, 0.6, 0.5, 0.52, 0.48, 0.56, 0.46, 0.56, 0.44, 0.43, 0.52, 0.49, 0.48, 0.61, 0.43, 0.55, 0.47, 0.55, 0.49, 0.48, 0.59, 0.53, 0.51, 0.47, 0.34, 0.45, 0.47, 0.57, 0.53, 0.57, 0.51, 0.46, 0.46, 0.5, 0.55, 0.42, 0.49, 0.5, 0.52, 0.52, 0.4, 0.39, 0.55, 0.53, 0.44, 0.47, 0.44, 0.55, 0.5, 0.46, 0.54, 0.51, 0.53, 0.57, 0.43, 0.43, 0.5, 0.51, 0.57, 0.49, 0.49, 0.47, 0.53, 0.5, 0.42, 0.53, 0.52, 0.55, 0.49, 0.58, 0.48, 0.52, 0.44, 0.44, 0.45, 0.54, 0.45, 0.55, 0.56, 0.44, 0.54, 0.44, 0.48, 0.49, 0.57, 0.49, 0.49, 0.5, 0.58, 0.43, 0.45, 0.4, 0.48, 0.52, 0.54, 0.49, 0.54, 0.51, 0.49, 0.4, 0.45, 0.47, 0.52, 0.56, 0.46, 0.5, 0.55, 0.5, 0.57, 0.56, 0.44, 0.44, 0.47, 0.5, 0.55, 0.56, 0.45, 0.51, 0.53, 0.48, 0.59, 0.59, 0.5, 0.49, 0.42, 0.54, 0.36, 0.42, 0.53, 0.46, 0.46, 0.47, 0.52, 0.4, 0.44, 0.39, 0.52, 0.49, 0.59, 0.55, 0.5, 0.51, 0.49, 0.44, 0.47, 0.51, 0.49, 0.43, 0.48, 0.49, 0.46, 0.51, 0.47, 0.52, 0.44, 0.56, 0.55, 0.45, 0.51, 0.5, 0.53, 0.54, 0.59, 0.49, 0.48, 0.49, 0.54, 0.59, 0.52, 0.43, 0.45, 0.52, 0.5, 0.52, 0.52, 0.47, 0.57, 0.48, 0.48, 0.5, 0.47, 0.48, 0.4, 0.47, 0.43, 0.54, 0.52, 0.47, 0.53, 0.5, 0.47, 0.42, 0.56, 0.54, 0.54, 0.49, 0.51, 0.38, 0.51, 0.55, 0.53, 0.56, 0.51, 0.55, 0.58, 0.47, 0.52, 0.49, 0.5, 0.43, 0.48, 0.49, 0.46, 0.47, 0.56, 0.5, 0.47, 0.53, 0.52, 0.42, 0.48, 0.58, 0.43, 0.53, 0.53, 0.52, 0.49, 0.58, 0.52, 0.46, 0.5, 0.51, 0.53, 0.48, 0.52, 0.52, 0.48, 0.59, 0.5, 0.46, 0.48, 0.6, 0.5, 0.47, 0.49, 0.44, 0.52, 0.42, 0.44, 0.45, 0.41, 0.57, 0.48, 0.46, 0.5, 0.54, 0.55, 0.48, 0.6, 0.55, 0.44, 0.45, 0.42, 0.49, 0.5, 0.48, 0.45, 0.45, 0.43, 0.51, 0.55, 0.57, 0.4, 0.6, 0.46, 0.48, 0.49, 0.58, 0.42, 0.53, 0.5, 0.54, 0.55, 0.47, 0.46, 0.48, 0.53, 0.48, 0.43, 0.51, 0.46, 0.54, 0.56, 0.44, 0.48, 0.45, 0.51, 0.48, 0.45, 0.58, 0.51, 0.54, 0.51, 0.45, 0.47, 0.43, 0.54, 0.53, 0.53, 0.53, 0.46, 0.53, 0.5, 0.56, 0.55, 0.49, 0.44, 0.47, 0.56, 0.39, 0.54, 0.45, 0.48, 0.55, 0.47, 0.5, 0.51, 0.49, 0.49, 0.54, 0.36, 0.39, 0.56, 0.45, 0.47, 0.52, 0.52, 0.6, 0.57, 0.51, 0.42, 0.48, 0.49, 0.49, 0.52, 0.55, 0.39, 0.47, 0.5, 0.42, 0.52, 0.43, 0.46, 0.42, 0.51, 0.54, 0.5, 0.5, 0.61, 0.54, 0.53, 0.56, 0.49, 0.45, 0.56, 0.54, 0.59, 0.5, 0.55, 0.45, 0.52, 0.56, 0.48, 0.52, 0.43, 0.5, 0.52, 0.52, 0.52, 0.53, 0.5, 0.54, 0.49, 0.48, 0.64, 0.51, 0.53, 0.54, 0.57, 0.43, 0.48, 0.55, 0.51, 0.47, 0.52, 0.5, 0.59, 0.56, 0.52, 0.55, 0.42, 0.57, 0.46, 0.48, 0.39, 0.59, 0.45, 0.55, 0.43, 0.45, 0.44, 0.5, 0.55, 0.51, 0.47, 0.46, 0.56, 0.47, 0.5, 0.58, 0.45, 0.49, 0.51, 0.55, 0.5, 0.56, 0.51, 0.51, 0.54, 0.47, 0.48, 0.44, 0.48, 0.52, 0.47, 0.54, 0.48, 0.44, 0.45, 0.46, 0.53, 0.54, 0.52, 0.47, 0.52, 0.47, 0.49, 0.48, 0.54, 0.55, 0.6, 0.52, 0.5, 0.5, 0.47, 0.5, 0.44, 0.45, 0.39, 0.54, 0.48, 0.53, 0.46, 0.42, 0.41, 0.49, 0.44, 0.48, 0.6, 0.46, 0.5, 0.53, 0.57, 0.52, 0.43, 0.5, 0.59, 0.45, 0.56, 0.51, 0.46, 0.53, 0.4, 0.5, 0.53, 0.54, 0.5, 0.46, 0.55, 0.57, 0.49, 0.42, 0.44, 0.54, 0.64, 0.49, 0.54, 0.6, 0.5, 0.59, 0.46, 0.55, 0.54, 0.47, 0.48, 0.46, 0.49, 0.44, 0.52, 0.61, 0.47, 0.55, 0.53, 0.42, 0.49, 0.38, 0.44, 0.5, 0.5, 0.55, 0.53, 0.51, 0.47, 0.43, 0.45, 0.45, 0.48, 0.5, 0.56, 0.55, 0.55, 0.44, 0.45, 0.59, 0.53, 0.48, 0.44, 0.48, 0.48, 0.48, 0.49, 0.4, 0.56, 0.43, 0.43, 0.53, 0.42, 0.54, 0.52, 0.47, 0.65, 0.49, 0.54, 0.54, 0.59, 0.5, 0.54, 0.53, 0.61, 0.53, 0.56, 0.36, 0.43, 0.47, 0.49, 0.49, 0.48, 0.58, 0.54, 0.46, 0.49, 0.5, 0.43, 0.52, 0.52, 0.59, 0.36, 0.6, 0.43, 0.43, 0.48, 0.5, 0.43, 0.48, 0.51, 0.49, 0.4, 0.43, 0.48, 0.53, 0.54, 0.52, 0.49, 0.51, 0.45, 0.47, 0.55, 0.46, 0.55, 0.5, 0.47, 0.48, 0.53, 0.52, 0.5, 0.47, 0.49, 0.48, 0.53, 0.51, 0.45, 0.51, 0.48, 0.5, 0.54, 0.42, 0.49, 0.48, 0.48, 0.43, 0.53, 0.52, 0.43, 0.47, 0.56, 0.45, 0.49, 0.54, 0.44, 0.61, 0.48, 0.43, 0.43, 0.52, 0.5, 0.53, 0.46, 0.47, 0.57, 0.48, 0.5, 0.5, 0.41, 0.61, 0.5, 0.45, 0.54, 0.52, 0.45, 0.44, 0.59, 0.49, 0.43, 0.57, 0.52, 0.51, 0.44, 0.5, 0.45, 0.57, 0.54, 0.6, 0.47, 0.4, 0.49, 0.47, 0.47, 0.44, 0.48, 0.47, 0.48, 0.5, 0.49, 0.47, 0.55, 0.55, 0.54, 0.53, 0.48, 0.54, 0.42, 0.58, 0.44, 0.54, 0.53, 0.46, 0.45, 0.41, 0.56, 0.46, 0.44, 0.57, 0.59, 0.44, 0.51, 0.48, 0.44, 0.54, 0.44, 0.49, 0.5, 0.45, 0.44, 0.53, 0.54, 0.55, 0.51, 0.52, 0.47, 0.47, 0.41, 0.53, 0.53, 0.52, 0.55, 0.46, 0.5, 0.59, 0.39, 0.42, 0.53, 0.46, 0.49, 0.53, 0.52, 0.46, 0.47, 0.42, 0.58, 0.52, 0.45, 0.45, 0.51, 0.44, 0.54, 0.46, 0.54, 0.55, 0.48, 0.53, 0.56, 0.49, 0.58, 0.5, 0.57, 0.51, 0.52, 0.53, 0.55, 0.42, 0.5, 0.52, 0.52, 0.52, 0.54, 0.52, 0.48, 0.52, 0.46, 0.53, 0.5, 0.52, 0.45, 0.52, 0.51, 0.51, 0.3, 0.55, 0.52, 0.43, 0.48, 0.45, 0.51, 0.49, 0.52, 0.55, 0.46, 0.54, 0.49, 0.53, 0.51, 0.4, 0.47, 0.46, 0.44, 0.4, 0.5, 0.4, 0.49, 0.4, 0.56, 0.5, 0.53, 0.49, 0.57, 0.49, 0.45, 0.48, 0.63, 0.48, 0.51, 0.46, 0.55, 0.49, 0.5, 0.46, 0.46, 0.46, 0.41, 0.51, 0.5, 0.4, 0.46, 0.49, 0.47, 0.52, 0.44, 0.55, 0.54, 0.53, 0.39, 0.41, 0.4, 0.55, 0.6, 0.51, 0.46, 0.57, 0.51, 0.59, 0.47, 0.47, 0.5, 0.43, 0.52, 0.56, 0.48, 0.55, 0.48, 0.52, 0.46, 0.49, 0.49, 0.42, 0.5, 0.56, 0.47, 0.44, 0.45, 0.55, 0.59, 0.49, 0.54, 0.5, 0.53, 0.57, 0.45, 0.46, 0.49, 0.49, 0.54, 0.41, 0.53, 0.52, 0.51, 0.5, 0.51, 0.52, 0.43, 0.45, 0.5, 0.44, 0.48, 0.45, 0.48, 0.38, 0.58, 0.46, 0.41, 0.57, 0.51, 0.44, 0.58, 0.46, 0.49, 0.52, 0.46, 0.6, 0.5, 0.5, 0.4, 0.51, 0.46, 0.62, 0.48, 0.47, 0.44, 0.45, 0.47, 0.49, 0.53, 0.46, 0.6, 0.48, 0.48, 0.52, 0.53, 0.53, 0.48, 0.46, 0.45, 0.36, 0.43, 0.49, 0.48, 0.5, 0.45, 0.51, 0.43, 0.55, 0.48, 0.46, 0.46, 0.46, 0.45, 0.47, 0.51, 0.53, 0.38, 0.52, 0.48, 0.5, 0.42, 0.61, 0.44, 0.46, 0.49, 0.55, 0.45, 0.55, 0.45, 0.67, 0.46, 0.48, 0.54, 0.57, 0.42, 0.55, 0.5, 0.58, 0.49, 0.47, 0.55, 0.53, 0.48, 0.52, 0.53, 0.55, 0.44, 0.61, 0.54, 0.51, 0.53, 0.52, 0.5, 0.53, 0.45, 0.48, 0.6, 0.45, 0.52, 0.5, 0.42, 0.52, 0.49, 0.49, 0.41, 0.5, 0.43, 0.55, 0.49, 0.52, 0.51, 0.48, 0.39, 0.51, 0.53, 0.45, 0.45, 0.47, 0.55, 0.55, 0.53, 0.43, 0.5, 0.47, 0.5, 0.45, 0.46, 0.56, 0.53, 0.46, 0.43, 0.47, 0.46, 0.46, 0.53, 0.48, 0.49, 0.5, 0.53, 0.55, 0.52, 0.43, 0.5, 0.55, 0.49, 0.43, 0.44, 0.44, 0.56, 0.54, 0.5, 0.51, 0.48, 0.43, 0.57, 0.59, 0.49, 0.55, 0.55, 0.53, 0.54, 0.56, 0.52, 0.47, 0.55, 0.43, 0.53, 0.54, 0.47, 0.54, 0.49, 0.53, 0.47, 0.55, 0.48, 0.51, 0.54, 0.41, 0.58, 0.47, 0.45, 0.54, 0.54, 0.51, 0.42, 0.42, 0.52, 0.48, 0.45, 0.47, 0.58, 0.48, 0.55, 0.51, 0.49, 0.54, 0.49, 0.45, 0.41, 0.5, 0.6, 0.49, 0.55, 0.5, 0.53, 0.53, 0.54, 0.44, 0.44, 0.61, 0.49, 0.56, 0.45, 0.51, 0.53, 0.4, 0.49, 0.51, 0.49, 0.53, 0.52, 0.48, 0.53, 0.5, 0.42, 0.61, 0.45, 0.41, 0.47, 0.47, 0.49, 0.45, 0.44, 0.47, 0.44, 0.52, 0.55, 0.51, 0.59, 0.45, 0.48, 0.53, 0.51, 0.51, 0.51, 0.49, 0.6, 0.57, 0.38, 0.52, 0.39, 0.46, 0.54, 0.47, 0.51, 0.44, 0.53, 0.56, 0.52, 0.54, 0.55, 0.45, 0.57, 0.55, 0.49, 0.43, 0.46, 0.5, 0.47, 0.53, 0.47, 0.47, 0.49, 0.5, 0.48, 0.49, 0.48, 0.47, 0.5, 0.46, 0.51, 0.53, 0.48, 0.48, 0.61, 0.49, 0.59, 0.58, 0.57, 0.53, 0.53, 0.51, 0.43, 0.45, 0.51, 0.55, 0.51, 0.45, 0.44, 0.47, 0.57, 0.51, 0.51, 0.58, 0.5, 0.56, 0.45, 0.43, 0.48, 0.45, 0.48, 0.48, 0.5, 0.42, 0.69, 0.55, 0.45, 0.52, 0.5, 0.51, 0.5, 0.51, 0.51, 0.48, 0.47, 0.6, 0.52, 0.61, 0.43, 0.51, 0.5, 0.44, 0.52, 0.47, 0.55, 0.45, 0.59, 0.53, 0.44, 0.52, 0.44, 0.54, 0.5, 0.47, 0.48, 0.55, 0.53, 0.47, 0.48, 0.45, 0.44, 0.47, 0.48, 0.58, 0.49, 0.54, 0.46, 0.55, 0.48, 0.46, 0.5, 0.45, 0.43, 0.5, 0.57, 0.52, 0.56, 0.65, 0.56, 0.48, 0.54, 0.5, 0.48, 0.52, 0.55, 0.47, 0.52, 0.47, 0.54, 0.44, 0.42, 0.5, 0.52, 0.57, 0.46, 0.44, 0.48, 0.44, 0.53, 0.42, 0.45, 0.52, 0.49, 0.48, 0.6, 0.52, 0.53, 0.48, 0.49, 0.5, 0.4, 0.41, 0.57, 0.45, 0.53, 0.5, 0.53, 0.49, 0.48, 0.43, 0.59, 0.52, 0.57, 0.49, 0.47, 0.6, 0.49, 0.49, 0.48, 0.42, 0.47, 0.44, 0.54, 0.54, 0.52, 0.53, 0.63, 0.59, 0.47, 0.44, 0.42, 0.5, 0.55, 0.59, 0.45, 0.59, 0.55, 0.42, 0.52, 0.43, 0.44, 0.44, 0.53, 0.54, 0.41, 0.49, 0.43, 0.4, 0.46, 0.41, 0.44, 0.45, 0.47, 0.56, 0.54, 0.48, 0.5, 0.41, 0.47, 0.49, 0.47, 0.48, 0.46, 0.45, 0.52, 0.46, 0.58, 0.55, 0.55, 0.54, 0.53, 0.54, 0.47, 0.56, 0.58, 0.48, 0.56, 0.53, 0.35, 0.46, 0.44, 0.5, 0.5, 0.56, 0.48, 0.52, 0.49, 0.45, 0.52, 0.45, 0.46, 0.55, 0.52, 0.55, 0.53, 0.49, 0.54, 0.61, 0.58, 0.53, 0.5, 0.48, 0.48, 0.49, 0.53, 0.42, 0.52, 0.47, 0.55, 0.5, 0.42, 0.52, 0.53, 0.49, 0.46, 0.48, 0.51, 0.57, 0.47, 0.49, 0.56, 0.45, 0.47, 0.48, 0.62, 0.52, 0.43, 0.46, 0.58, 0.5, 0.58, 0.45, 0.53, 0.52, 0.43, 0.46, 0.59, 0.59, 0.54, 0.44, 0.47, 0.51, 0.56, 0.45, 0.48, 0.46, 0.48, 0.48, 0.54, 0.46, 0.55, 0.6, 0.48, 0.45, 0.49, 0.46, 0.43, 0.42, 0.44, 0.56, 0.54, 0.55, 0.49, 0.5, 0.55, 0.53, 0.51, 0.47, 0.58, 0.48, 0.39, 0.58, 0.46, 0.42, 0.54, 0.52, 0.49, 0.48, 0.55, 0.46, 0.52, 0.5, 0.5, 0.45, 0.48, 0.55, 0.55, 0.5, 0.49, 0.55, 0.56, 0.46, 0.43, 0.56, 0.47, 0.38, 0.43, 0.55, 0.56, 0.45, 0.53, 0.57, 0.51, 0.46, 0.48, 0.49, 0.55, 0.51, 0.52, 0.47, 0.48, 0.49, 0.57, 0.45, 0.58, 0.53, 0.54, 0.5, 0.42, 0.49, 0.47, 0.5, 0.55, 0.5, 0.47, 0.46, 0.52, 0.43, 0.56, 0.54, 0.56, 0.55, 0.56, 0.57, 0.55, 0.46, 0.45, 0.5, 0.44, 0.5, 0.5, 0.49, 0.51, 0.57, 0.56, 0.47, 0.47, 0.47, 0.49, 0.49, 0.49, 0.55, 0.46, 0.58, 0.63, 0.52, 0.52, 0.54, 0.51, 0.49, 0.52, 0.48, 0.58, 0.54, 0.56, 0.52, 0.46, 0.49, 0.39, 0.5, 0.51, 0.54, 0.45, 0.48, 0.46, 0.43, 0.6, 0.64, 0.48, 0.49, 0.47, 0.47, 0.55, 0.52, 0.54, 0.61, 0.48, 0.55, 0.49, 0.47, 0.52, 0.52, 0.5, 0.53, 0.53, 0.51, 0.44, 0.55, 0.48, 0.45, 0.52, 0.49, 0.49, 0.55, 0.48, 0.46, 0.42, 0.48, 0.54, 0.55, 0.56, 0.45, 0.46, 0.34, 0.47, 0.38, 0.54, 0.54, 0.46, 0.51, 0.46, 0.52, 0.39, 0.48, 0.54, 0.6, 0.51, 0.44, 0.51, 0.49, 0.44, 0.42, 0.52, 0.53, 0.45, 0.46, 0.57, 0.4, 0.49, 0.55, 0.46, 0.46, 0.5, 0.63, 0.43, 0.53, 0.52, 0.56, 0.5, 0.53, 0.53, 0.42, 0.42, 0.48, 0.51, 0.52, 0.55, 0.46, 0.52, 0.47, 0.5, 0.57, 0.57, 0.49, 0.46, 0.46, 0.55, 0.43, 0.49, 0.42, 0.58, 0.52, 0.51, 0.58, 0.45, 0.44, 0.48, 0.49, 0.52, 0.52, 0.57, 0.43, 0.57, 0.53, 0.57, 0.53, 0.44, 0.51, 0.5, 0.55, 0.36, 0.5, 0.47, 0.56, 0.47, 0.43, 0.52, 0.49, 0.48, 0.47, 0.47, 0.52, 0.52, 0.52, 0.48, 0.42, 0.49, 0.51, 0.5, 0.52, 0.54, 0.52, 0.56, 0.43, 0.38, 0.5, 0.47, 0.54, 0.52, 0.48, 0.54, 0.52, 0.51, 0.5, 0.5, 0.51, 0.43, 0.53, 0.48, 0.5, 0.47, 0.43, 0.49, 0.57, 0.45, 0.45, 0.58, 0.44, 0.42, 0.56, 0.5, 0.47, 0.51, 0.49, 0.58, 0.46, 0.54, 0.54, 0.44, 0.51, 0.49, 0.53, 0.52, 0.5, 0.5, 0.49, 0.42, 0.43, 0.41, 0.5, 0.43, 0.51, 0.52, 0.56, 0.5]

```
# Create dataframe with single coin flip
df = pd.DataFrame({'one-samp': one_sample})
df.head()
```

      one-samp

      0
      1

      1
      0

      2
      1

      3
      0

      4
      1

```
# Plot histogram to look at distribution of a single coin flip
df.hist()
```

```
# Plot histogram to look at distribution of all coin flips
ax = plt.hist(sample_means)
plt.title(f'Distribtion of {N} sample means \n (of 30 coinflips of each)');
```

What does the Central Limit Theorem State? That no matter the initial distribution of the population, the distribution of sample means taken will approximate a normal distribution as $n \rightarrow \infty$.

This has very important implications for hypothesis testing and is precisely the reason why the t-distribution begins to approximate the normal distribution as our sample size increases.

## Standard Error of the Mean

What does it mean to "estimate"? the Population mean?

```
import numpy as np
import pandas as pd

# Average Height
mu = 70
sigma = 3

lambda_heights = np.random.normal(mu, sigma, 2000)
print(len(lambda_heights))
lambda_heights
```

    2000

    array([66.90108928, 70.84717486, 72.37683043, ..., 65.71048083,
           66.71224677, 70.63992345])

```
import seaborn as sns

sns.distplot(lambda_heights)
plt.title('Distribution of Heights (in inches)');
```

```
print("Population Mean:", lambda_heights.mean())
print("Population Standard Deviation:", lambda_heights.std())
```

    Population Mean: 70.02424579436976
    Population Standard Deviation: 3.042489431974776

```
population = pd.DataFrame({'heights': lambda_heights})
print(population.shape)
population.head()
```

    (2000, 1)

      heights

      0
      66.901089

      1
      70.847175

      2
      72.376830

      3
      70.116083

      4
      70.751211

```
# Take a random sample and print sample mean
sample1 = population.sample(100)
print(sample1.shape)
sample1.head()
```

    (100, 1)

      heights

      890
      67.598192

      1619
      74.184065

      56
      68.994803

      1109
      72.621634

      568
      72.411452

```
print('Sample1 Mean #1:', sample1['heights'].mean())
```

    Sample1 Mean #1: 69.8108354823691

```
# Take a different random sample and print sample mean
```

```
sample2 = population.sample(100)
print(sample2.shape)
sample2.head()
```

    (100, 1)

      heights

      1680
      63.458686

      276
      67.978647

      665
      72.300605

      1375
      69.306754

      1695
      70.259486

```
print('Sample1 Mean #2:', sample2['heights'].mean())
```

    Sample1 Mean #2: 70.08812374010863

## Build and Interpret a Confidence Interval

```
coinflips_100 = np.random.binomial(n=1, p=0.5, size=100)

sample_std = np.std(coinflips_100)
print('Sample St Dev:', sample_std)
sample_size = len(coinflips_100)
print('Sample Size:', sample_size)
```

    Sample St Dev: 0.4998999899979995
    Sample Size: 100

```
standard_error = sample_std/np.sqrt(sample_size)
print(standard_error)
```

    0.04998999899979995

### What confidence level do we want our confidence interval to represent?

95% confidence Interval? 99% confidence interval?

```
import scipy.stats as stats
help(stats.t.ppf)
```

    Help on method ppf in module scipy.stats._distn_infrastructure:

    ppf(q, *args, **kwds) method of scipy.stats._continuous_distns.t_gen instance
        Percent point function (inverse of `cdf`) at q of the given RV.

        Parameters
        ----------
        q : array_like
            lower tail probability
        arg1, arg2, arg3,... : array_like
            The shape parameter(s) for the distribution (see docstring of the
            instance object for more information)
        loc : array_like, optional
            location parameter (default=0)
        scale : array_like, optional
            scale parameter (default=1)

        Returns
        -------
        x : array_like
            quantile corresponding to the lower tail probability q.

```
t = stats.t.ppf(0.975, sample_size-1)
t
```

    1.9842169515086827

```
sample_mean = coinflips_100.mean()
confidence_interval = (sample_mean - t* standard_error, sample_mean + t* standard_error)
margin_of_error = t* standard_error

print("Sample Mean: ", sample_mean)
print("Margin of Error: ", margin_of_error)
print("Confidence Interval: ", confidence_interval)
```

    Sample Mean:  0.51
    Margin of Error:  0.09919100342130516
    Confidence Interval:  (0.41080899657869485, 0.6091910034213052)

## Graphically Represent a Confidence Interval

```
import seaborn as sns

sns.kdeplot(coinflips_100);
plt.axvline(x=sample_mean, color='k')
plt.axvline(x=confidence_interval[0], color='r')
plt.axvline(x=confidence_interval[1], color='r');
```

## Relationship between Confidence Intervals and T-tests

Confidence Interval == Bounds of statistical significance for our t-test

A sample mean that falls inside of our confidence interval will "FAIL TO REJECT" our null hypothesis

A sample mean that falls outside of our confidence interval will "REJECT" our null hypothesis

```
from scipy.stats import t, ttest_1samp
```

```
import numpy as np

coinflip_means = []
for x in range(0,100):
  coinflips = np.random.binomial(n=1, p=.5, size=30)
  coinflip_means.append(coinflips.mean())

print(coinflip_means)
```

    [0.4, 0.4666666666666667, 0.4, 0.43333333333333335, 0.4666666666666667, 0.43333333333333335, 0.4, 0.5666666666666667, 0.5333333333333333, 0.4666666666666667, 0.4666666666666667, 0.6333333333333333, 0.5333333333333333, 0.6333333333333333, 0.6666666666666666, 0.5333333333333333, 0.5666666666666667, 0.4666666666666667, 0.5333333333333333, 0.5666666666666667, 0.4666666666666667, 0.5666666666666667, 0.4666666666666667, 0.6333333333333333, 0.4666666666666667, 0.5333333333333333, 0.4666666666666667, 0.43333333333333335, 0.5, 0.5, 0.6, 0.5666666666666667, 0.5333333333333333, 0.5333333333333333, 0.5333333333333333, 0.43333333333333335, 0.5333333333333333, 0.4, 0.4666666666666667, 0.43333333333333335, 0.5333333333333333, 0.5666666666666667, 0.4666666666666667, 0.6666666666666666, 0.36666666666666664, 0.4, 0.4, 0.36666666666666664, 0.6, 0.4, 0.5333333333333333, 0.36666666666666664, 0.5333333333333333, 0.5333333333333333, 0.3, 0.43333333333333335, 0.4666666666666667, 0.5333333333333333, 0.4666666666666667, 0.43333333333333335, 0.5333333333333333, 0.5, 0.5666666666666667, 0.4, 0.5333333333333333, 0.4666666666666667, 0.6333333333333333, 0.5333333333333333, 0.5666666666666667, 0.3, 0.6, 0.43333333333333335, 0.43333333333333335, 0.4666666666666667, 0.6, 0.5333333333333333, 0.6333333333333333, 0.3, 0.6, 0.43333333333333335, 0.5, 0.4666666666666667, 0.4666666666666667, 0.5333333333333333, 0.36666666666666664, 0.5, 0.43333333333333335, 0.5333333333333333, 0.5666666666666667, 0.6, 0.4666666666666667, 0.6666666666666666, 0.8, 0.5666666666666667, 0.5, 0.5333333333333333, 0.6333333333333333, 0.5, 0.5666666666666667, 0.43333333333333335]

```
np.mean(coinflip_means)
```

    0.5029999999999999

```
# 95% confidence interval
t_stat = stats.t.ppf(0.975, 99)
print('T Statistic: ', t_stat)

std_sample = np.std(coinflip_means)
std_err = std_sample/np.sqrt(len(coinflip_means))

CI = stats.t.interval(0.95, 99, loc=np.mean(coinflip_means), scale=std_err)
print('95% confidence interval: ', CI)
```

    T Statistic:  1.9842169515086827
    95% confidence interval:  (0.48575024138185613, 0.5202497586181436)

A null hypothesis that's just inside of our confidence interval == fail to reject

```
ttest_1samp(coinflip_means, 0.48575)
```

    Ttest_1sampResult(statistic=1.974298565903243, pvalue=0.051133955079730474)

A null hypothesis that's just outside of our confidence interval == reject

```
ttest_1samp(coinflip_means, 0.520249)
```

    Ttest_1sampResult(statistic=-1.9741841138124883, pvalue=0.051147166772916766)

```
def confidence_interval(data, confidence=0.95):
  """
  Calculate a confidence interval around a sample mean for given data.
  Using t-distribution and two-tailed test, default 95% confidence.

  Arguments:
    data - iterable (list or numpy array) of sample observations
    confidence - level of confidence for the interval

  Returns:
    tuple of (mean, lower bound, upper bound)
  """
  data = np.array(data)
  mean = np.mean(data)
  n = len(data)
  stderr = stats.sem(data)
  interval = stderr * stats.t.ppf((1 + confidence) / 2.0, n - 1)
  return (mean, mean - interval, mean + interval)
```

## Run a $\chi^{2}$ Test "by hand" (Using Numpy)

```
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/adult.csv', na_values=" ?")
print(df.shape)
df.head()
```

    (32561, 15)

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary

      0
      39
      State-gov
      77516
      Bachelors
      13
      Never-married
      Adm-clerical
      Not-in-family
      White
      Male
      2174
      0
      40
      United-States
      <=50K

      1
      50
      Self-emp-not-inc
      83311
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      13
      United-States
      <=50K

      2
      38
      Private
      215646
      HS-grad
      9
      Divorced
      Handlers-cleaners
      Not-in-family
      White
      Male
      0
      0
      40
      United-States
      <=50K

      3
      53
      Private
      234721
      11th
      7
      Married-civ-spouse
      Handlers-cleaners
      Husband
      Black
      Male
      0
      0
      40
      United-States
      <=50K

      4
      28
      Private
      338409
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      Black
      Female
      0
      0
      40
      Cuba
      <=50K

```
df.describe()
```

      age
      fnlwgt
      education-num
      capital-gain
      capital-loss
      hours-per-week

      count
      32561.000000
      3.256100e+04
      32561.000000
      32561.000000
      32561.000000
      32561.000000

      mean
      38.581647
      1.897784e+05
      10.080679
      1077.648844
      87.303830
      40.437456

      std
      13.640433
      1.055500e+05
      2.572720
      7385.292085
      402.960219
      12.347429

      min
      17.000000
      1.228500e+04
      1.000000
      0.000000
      0.000000
      1.000000

      25%
      28.000000
      1.178270e+05
      9.000000
      0.000000
      0.000000
      40.000000

      50%
      37.000000
      1.783560e+05
      10.000000
      0.000000
      0.000000
      40.000000

      75%
      48.000000
      2.370510e+05
      12.000000
      0.000000
      0.000000
      45.000000

      max
      90.000000
      1.484705e+06
      16.000000
      99999.000000
      4356.000000
      99.000000

```
df.describe(exclude='number')
```

      workclass
      education
      marital-status
      occupation
      relationship
      race
      sex
      country
      salary

      count
      30725
      32561
      32561
      30718
      32561
      32561
      32561
      31978
      32561

      unique
      8
      16
      7
      14
      6
      5
      2
      41
      2

      top
      Private
      HS-grad
      Married-civ-spouse
      Prof-specialty
      Husband
      White
      Male
      United-States
      <=50K

      freq
      22696
      10501
      14976
      4140
      13193
      27816
      21790
      29170
      24720

```
df['hours-per-week'].hist(bins=10);
```

```
cut_points = [0, 9, 19, 29, 39, 49, 500]
label_names = ['0-9', '10-19', '20-29', '30-39', '40-49', '50+']

df['hours_per_week_categories'] = pd.cut(df['hours-per-week'], cut_points, labels=label_names)
df.hours_per_week_categories.value_counts()
```

    40-49    18336
    50+       6462
    30-39     3667
    20-29     2392
    10-19     1246
    0-9        458
    Name: hours_per_week_categories, dtype: int64

```
df.sex.value_counts()
```

     Male      21790
     Female    10771
    Name: sex, dtype: int64

```
df = df.sort_values(by='hours_per_week_categories', ascending=True)
contingency_table = pd.crosstab(df['sex'], df['hours_per_week_categories'], margins=True)
contingency_table
```

      hours_per_week_categories
      0-9
      10-19
      20-29
      30-39
      40-49
      50+
      All

      sex

      Female
      235
      671
      1287
      1914
      5636
      1028
      10771

      Male
      223
      575
      1105
      1753
      12700
      5434
      21790

      All
      458
      1246
      2392
      3667
      18336
      6462
      32561

```
row_sums = contingency_table.iloc[0:2, 6].values
col_sums = contingency_table.iloc[2, 0:6].values

print(row_sums)
print(col_sums)
```

    [10771 21790]
    [  458  1246  2392  3667 18336  6462]

```
total = contingency_table.loc['All', 'All']
total
```

    32561

## Expected Value Calculation
\begin{align}
expected_{i,j} =\frac{(row_{i} \text{total})(column_{j} \text{total}) }{(\text{total observations})}
\end{align}

```
expected = []
for i in range(len(row_sums)):
  expected_row = []
  for column in col_sums:
    expected_val = column*row_sums[i]/total
    expected_row.append(expected_val)
  expected.append(expected_row)

expected = np.array(expected)
print(expected.shape)
print(expected)
```

    (2, 6)
    [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

```

```

## Chi-Squared Statistic with Numpy

\begin{align}
\chi^2 = \sum \frac{(observed_{i}-expected_{i})^2}{(expected_{i})}
\end{align}

For the $observed$ values we will just use a version of our contingency table without the margins as a numpy array. In this way, if our observed values array and our expected values array are the same shape, then we can subtract them and divide them directly which makes the calculations a lot cleaner. No for loops!

```
observed = pd.crosstab(df['sex'], df['hours_per_week_categories']).values
print(observed.shape)
observed
```

    (2, 6)

    array([[  235,   671,  1287,  1914,  5636,  1028],
           [  223,   575,  1105,  1753, 12700,  5434]])

```
chi_square = ((observed - expected)**2/(expected)).sum()
chi_square
```

    2287.190943926107

## Run a $\chi^{2}$ Test using Scipy

```
chi_square, p_value, dof, expected = stats.chi2_contingency(observed)
print(chi_square, p_value, dof, expected)
```

    2287.190943926107 0.0 5 [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

Null Hypothesis: Hours worked per week bins is **independent** of sex.

Due to a p-value of 0, we REJECT the null hypothesis that hours worked per week and sex are independent, and conclude that there is an association between hours worked per week and sex.

## Assignment - Build a confidence interval

A confidence interval refers to a neighborhood around some point estimate, the size of which is determined by the desired p-value. For instance, we might say that 52% of Americans prefer tacos to burritos, with a 95% confidence interval of +/- 5%.

52% (0.52) is the point estimate, and +/- 5% (the interval $[0.47, 0.57]$) is the confidence interval. "95% confidence" means a p-value $\leq 1 - 0.95 = 0.05$.

In this case, the confidence interval includes $0.5$ - which is the natural null hypothesis (that half of Americans prefer tacos and half burritos, thus there is no clear favorite). So in this case, we could use the confidence interval to report that we've failed to reject the null hypothesis.

But providing the full analysis with a confidence interval, including a graphical representation of it, can be a helpful and powerful way to tell your story. Done well, it is also more intuitive to a layperson than simply saying "fail to reject the null hypothesis" - it shows that in fact the data does *not* give a single clear result (the point estimate) but a whole range of possibilities.

How is a confidence interval built, and how should it be interpreted? It does *not* mean that 95% of the data lies in that interval - instead, the frequentist interpretation is "if we were to repeat this experiment 100 times, we would expect the average result to lie in this interval ~95 times."

For a 95% confidence interval and a normal(-ish) distribution, you can simply remember that +/-2 standard deviations contains 95% of the probability mass, and so the 95% confidence interval based on a given sample is centered at the mean (point estimate) and has a range of +/- 2 (or technically 1.96) standard deviations.

Different distributions/assumptions (90% confidence, 99% confidence) will require different math, but the overall process and interpretation (with a frequentist approach) will be the same.

Your assignment - using the data from the prior module ([congressional voting records](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records)):

### Confidence Intervals:
1. Generate and numerically represent a confidence interval
2. Graphically (with a plot) represent the confidence interval
3. Interpret the confidence interval - what does it tell you about the data and its distribution?

### Chi-squared tests:
4. Take a dataset that we have used in the past in class that has **categorical** variables. Pick two of those categorical variables and run a chi-squared tests on that data
  - By hand using Numpy
  - In a single line using Scipy

Stretch goals:

1. Write a summary of your findings, mixing prose and math/code/results. *Note* - yes, this is by definition a political topic. It is challenging but important to keep your writing voice *neutral* and stick to the facts of the data. Data science often involves considering controversial issues, so it's important to be sensitive about them (especially if you want to publish).
2. Apply the techniques you learned today to your project data or other data of your choice, and write/discuss your findings here.
3. Refactor your code so it is elegant, readable, and can be easily run for all issues.

```
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
```

    --2019-08-10 17:09:38--  https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
    Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252
    Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 18171 (18K) [application/x-httpd-php]
    Saving to: ‘house-votes-84.data’

    house-votes-84.data 100%[===================>]  17.75K  --.-KB/s    in 0.07s

    2019-08-10 17:09:39 (267 KB/s) - ‘house-votes-84.data’ saved [18171/18171]

```
import pandas as pd
# Making a list of missing values
missing_values = ["?"]
df = pd.read_csv('house-votes-84.data', na_values = missing_values, header=None)
print(df.shape)
df.head()
```

    (435, 17)

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      11
      12
      13
      14
      15
      16

      0
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      y
      NaN
      y
      y
      y
      n
      y

      1
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      n
      n
      y
      y
      y
      n
      NaN

      2
      democrat
      NaN
      y
      y
      NaN
      y
      y
      n
      n
      n
      n
      y
      n
      y
      y
      n
      n

      3
      democrat
      n
      y
      y
      n
      NaN
      y
      n
      n
      n
      n
      y
      n
      y
      n
      n
      y

      4
      democrat
      y
      y
      y
      n
      y
      y
      n
      n
      n
      n
      y
      NaN
      y
      y
      y
      y

```
columns = ['party', 'handicapped-infants', 'water-project-cost-sharing', 'adoption-of-the-budget-resolution',
           'physician-fee-freeze', 'el-salvador-aid', 'religious-groups-in-schools', 'anti-satellite-test-ban',
           'aid-to-nicaraguan-contras', 'mx-missile', 'immigration', 'synfuels-corporation-cutback', 'education-spending',
           'superfund-right-to-sue', 'crime', 'duty-free-exports', 'export-administration-act-south-africa']
```

```
df.columns = columns
df.head()
```

      party
      handicapped-infants
      water-project-cost-sharing
      adoption-of-the-budget-resolution
      physician-fee-freeze
      el-salvador-aid
      religious-groups-in-schools
      anti-satellite-test-ban
      aid-to-nicaraguan-contras
      mx-missile
      immigration
      synfuels-corporation-cutback
      education-spending
      superfund-right-to-sue
      crime
      duty-free-exports
      export-administration-act-south-africa

      0
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      y
      NaN
      y
      y
      y
      n
      y

      1
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      n
      n
      y
      y
      y
      n
      NaN

      2
      democrat
      NaN
      y
      y
      NaN
      y
      y
      n
      n
      n
      n
      y
      n
      y
      y
      n
      n

      3
      democrat
      n
      y
      y
      n
      NaN
      y
      n
      n
      n
      n
      y
      n
      y
      n
      n
      y

      4
      democrat
      y
      y
      y
      n
      y
      y
      n
      n
      n
      n
      y
      NaN
      y
      y
      y
      y

```
df.isna().sum()
```

    party                                       0
    handicapped-infants                        12
    water-project-cost-sharing                 48
    adoption-of-the-budget-resolution          11
    physician-fee-freeze                       11
    el-salvador-aid                            15
    religious-groups-in-schools                11
    anti-satellite-test-ban                    14
    aid-to-nicaraguan-contras                  15
    mx-missile                                 22
    immigration                                 7
    synfuels-corporation-cutback               21
    education-spending                         31
    superfund-right-to-sue                     25
    crime                                      17
    duty-free-exports                          28
    export-administration-act-south-africa    104
    dtype: int64

```
import matplotlib.pyplot as plt
from scipy.stats import ttest_ind, ttest_ind_from_stats, ttest_rel
import pandas as pd
import numpy as np

def convert_to_int(df):
  for feat in df.columns:
    df[feat].replace(to_replace=['n', 'y'], value=[0, 1], inplace=True)

def clean_data(df):
  for feat in df.columns:
    if df[feat].dtype != 'object':
      df[feat].fillna(df[feat].mode()[0], inplace=True)
```

```
convert_to_int(df)
clean_data(df)
df.isna().sum()
```

    party                                     0
    handicapped-infants                       0
    water-project-cost-sharing                0
    adoption-of-the-budget-resolution         0
    physician-fee-freeze                      0
    el-salvador-aid                           0
    religious-groups-in-schools               0
    anti-satellite-test-ban                   0
    aid-to-nicaraguan-contras                 0
    mx-missile                                0
    immigration                               0
    synfuels-corporation-cutback              0
    education-spending                        0
    superfund-right-to-sue                    0
    crime                                     0
    duty-free-exports                         0
    export-administration-act-south-africa    0
    dtype: int64

```
print(df.shape)
df.head()
```

    (435, 17)

      party
      handicapped-infants
      water-project-cost-sharing
      adoption-of-the-budget-resolution
      physician-fee-freeze
      el-salvador-aid
      religious-groups-in-schools
      anti-satellite-test-ban
      aid-to-nicaraguan-contras
      mx-missile
      immigration
      synfuels-corporation-cutback
      education-spending
      superfund-right-to-sue
      crime
      duty-free-exports
      export-administration-act-south-africa

      0
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      1.0

      1
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      0.0
      1.0

      2
      democrat
      0.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0

      3
      democrat
      0.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0

      4
      democrat
      1.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      1.0

# Confidence Interval

```
demo_sample_std = df[df['party'] == 'democrat']['handicapped-infants']
demo_handicapped_infants = pd.DataFrame(demo_sample_std)
demo_handicapped_infants.columns=['demo_handicapped_infants']
print(demo_handicapped_infants.shape)
demo_handicapped_infants.head()
```

    (267, 1)

      demo_handicapped_infants

      2
      0.0

      3
      0.0

      4
      1.0

      5
      0.0

      6
      0.0

```
sample_std = np.std(demo_handicapped_infants['demo_handicapped_infants'])
sample_size = len(demo_handicapped_infants)
print(sample_std)
print(sample_size)
```

    0.492847465156435
    267

```
standard_error = sample_std/np.sqrt(sample_size)
print(standard_error)
```

    0.030161775044324438

```
import scipy.stats as stats
t = stats.t.ppf(0.975, sample_size-1)
t
```

    1.9689223241409017

```
sample_mean = demo_handicapped_infants['demo_handicapped_infants'].mean()
confidence_interval = (sample_mean - t* standard_error, sample_mean + t* standard_error)
margin_of_error = t* standard_error
```

```
import seaborn as sns

sns.kdeplot(demo_handicapped_infants['demo_handicapped_infants']);
plt.axvline(x=sample_mean, color='k')
plt.axvline(x=confidence_interval[0], color='r')
plt.axvline(x=confidence_interval[1], color='r');
```

##Interpretation

How can we interpret this visualization? Well, this means that there was a pretty even split in how the Democrats voted on this issue. We can say with 95% confidence that the average democratic vote was .56 (with 0 being no and 1 being yes) or slightly in favor of a yes vote on this issue.

# Chi Squared Tests

Chi-squared tests:
Take a dataset that we have used in the past in class that has categorical variables. Pick two of those categorical variables and run a chi-squared tests on that data
By hand using Numpy
In a single line using Scipy

## Chi-Squared Statistic with Numpy

\begin{align}
\chi^2 = \sum \frac{(observed_{i}-expected_{i})^2}{(expected_{i})}
\end{align}

For the $observed$ values we will just use a version of our contingency table without the margins as a numpy array. In this way, if our observed values array and our expected values array are the same shape, then we can subtract them and divide them directly which makes the calculations a lot cleaner. No for loops!

```
import pandas as pd
entities = pd.read_csv('https://raw.githubusercontent.com/open-numbers/ddf--gapminder--systema_globalis/master/ddf--entities--geo--country.csv')
print(entities.shape)
entities.head()
```

    (273, 33)

      country
      alt_5
      alternative_1
      alternative_2
      alternative_3
      alternative_4_cdiac
      arb1
      arb2
      arb3
      arb4
      arb5
      arb6
      g77_and_oecd_countries
      gapminder_list
      god_id
      gwid
      income_groups
      is--country
      iso3166_1_alpha2
      iso3166_1_alpha3
      iso3166_1_numeric
      iso3166_2
      landlocked
      latitude
      longitude
      main_religion_2008
      name
      pandg
      un_state
      unicode_region_subtag
      upper_case_name
      world_4region
      world_6region

      0
      abkh
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      others
      Abkhazia
      GE-AB
      i0
      NaN
      True
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      Abkhazia
      NaN
      False
      NaN
      NaN
      europe
      europe_central_asia

      1
      abw
      NaN
      NaN
      NaN
      NaN
      Aruba
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      others
      Aruba
      AW
      i12
      high_income
      True
      AW
      ABW
      533.0
      NaN
      coastline
      12.50000
      -69.96667
      christian
      Aruba
      NaN
      False
      AW
      ARUBA
      americas
      america

      2
      afg
      NaN
      Islamic Republic of Afghanistan
      NaN
      NaN
      Afghanistan
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      g77
      Afghanistan
      AF
      i1
      low_income
      True
      AF
      AFG
      4.0
      NaN
      landlocked
      33.00000
      66.00000
      muslim
      Afghanistan
      AFGHANISTAN
      True
      AF
      AFGHANISTAN
      asia
      south_asia

      3
      ago
      NaN
      NaN
      NaN
      NaN
      Angola
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      g77
      Angola
      AO
      i7
      upper_middle_income
      True
      AO
      AGO
      24.0
      NaN
      coastline
      -12.50000
      18.50000
      christian
      Angola
      ANGOLA
      True
      AO
      ANGOLA
      africa
      sub_saharan_africa

      4
      aia
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      others
      Anguilla
      AI
      i8
      NaN
      True
      AI
      AIA
      660.0
      NaN
      coastline
      18.21667
      -63.05000
      christian
      Anguilla
      NaN
      False
      AI
      ANGUILLA
      americas
      america

```
# lets take a look at income_groups and world_6region

entities['income_groups'].isna().sum()
```

    59

```
entities['income_groups'].dropna(inplace=True)
```

```
entities['income_groups'].isna().sum()
```

    0

```
entities['world_6region'].isna().sum()
```

    13

```
entities['world_6region'].dropna(inplace=True)
```

```
entities['world_6region'].isna().sum()
```

    0

```
entities['income_groups'].value_counts().plot(kind='bar');
```

```
entities['world_6region'].value_counts().plot(kind='bar');
```

```
# Margins here is the subtotal for each row.
contingency_table = pd.crosstab(entities['income_groups'], entities['world_6region'], margins=True)
contingency_table
```

      world_6region
      america
      east_asia_pacific
      europe_central_asia
      middle_east_north_africa
      south_asia
      sub_saharan_africa
      All

      income_groups

      high_income
      17
      12
      37
      7
      0
      1
      74

      low_income
      1
      3
      2
      0
      3
      27
      36

      lower_middle_income
      7
      12
      6
      6
      4
      13
      48

      upper_middle_income
      18
      9
      13
      7
      1
      7
      55

      All
      43
      36
      58
      20
      8
      48
      213

```
row_sums = contingency_table.iloc[0:4, 6].values
col_sums = contingency_table.iloc[4, 0:6].values

print(row_sums)
print(col_sums)
```

    [74 36 48 55]
    [43 36 58 20  8 48]

```
total = contingency_table.loc['All', 'All']
total
```

    213

```
import numpy as np
expected = []
for i in range(len(row_sums)):
  expected_row = []
  for column in col_sums:
    expected_val = column*row_sums[i]/total
    expected_row.append(expected_val)
  expected.append(expected_row)

expected = np.array(expected)
print(expected.shape)
print(expected)
```

    (4, 6)
    [[14.93896714 12.50704225 20.15023474  6.94835681  2.77934272 16.67605634]
     [ 7.26760563  6.08450704  9.8028169   3.38028169  1.35211268  8.11267606]
     [ 9.69014085  8.11267606 13.07042254  4.50704225  1.8028169  10.81690141]
     [11.10328638  9.29577465 14.97652582  5.16431925  2.0657277  12.3943662 ]]

```
observed = pd.crosstab(entities['income_groups'], entities['world_6region']).values
print(observed.shape)
observed
```

    (4, 6)

    array([[17, 12, 37,  7,  0,  1],
           [ 1,  3,  2,  0,  3, 27],
           [ 7, 12,  6,  6,  4, 13],
           [18,  9, 13,  7,  1,  7]])

```
chi_square = ((observed - expected)**2/(expected)).sum()
chi_square
```

    112.60236492756655

## With Scipy

```
 import scipy.stats as stats
chi_square, p_value, dof, expected = stats.chi2_contingency(observed)
print(chi_square, p_value, dof, expected)
```

    112.60236492756655 5.094803393946877e-17 15 [[14.93896714 12.50704225 20.15023474  6.94835681  2.77934272 16.67605634]
     [ 7.26760563  6.08450704  9.8028169   3.38028169  1.35211268  8.11267606]
     [ 9.69014085  8.11267606 13.07042254  4.50704225  1.8028169  10.81690141]
     [11.10328638  9.29577465 14.97652582  5.16431925  2.0657277  12.3943662 ]]

## Resources

- [Interactive visualize the Chi-Squared test](https://homepage.divms.uiowa.edu/~mbognar/applets/chisq.html)
- [Calculation of Chi-Squared test statistic](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test)
- [Visualization of a confidence interval generated by R code](https://commons.wikimedia.org/wiki/File:Confidence-interval.svg)
- [Expected value of a squared standard normal](https://math.stackexchange.com/questions/264061/expected-value-calculation-for-squared-normal-distribution) (it's 1 - which is why the expected value of a Chi-Squared with $n$ degrees of freedom is $n$, as it's the sum of $n$ squared standard normals)


```python
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
from scipy import stats
plt.style.use('fivethirtyeight')
import pandas as pd
```

#  Descriptive Statistics

Before making any inferences from a dataset, it is crucial that we understand the features of datasets by 'describing' them. We can either describe them by graphing or calculating numerical summarisations.

We will refresh most common descriptive statistics in this chapter, if you are working with data on a daily basis, no concepts here should sound strange to you.

Furthermore, we will not engage in complex programming techniques, such as OOP and sophisticated data structures. All the programmes should be self-explanatory to most of audiences.

## Frequency Distribution/Histogram

Strictly speaking, frequency distribution and histogram are different descriptive tools, though they are delivering the largely identical information. Frequency distribution are usually presented in a table, for instance, rolling a dice $1000$ times might give a us frequency distribution table as following:

Rolling A Dice 1000 Times

    Sides
    Frequency

    1
    172

    2
    158

    3
    170

    4
    158

    5
    187

    6
    155

Or we can draw a frequency histogram, which simply converts the information of table into a graph.

```python
rollings = np.random.randint(1, 7, 1000)

fig, ax = plt.subplots(figsize=(9, 9))
n, bins, patches = ax.hist(rollings, bins = 6)
ax.set_title('Frequency Histogram of 1000 Times of Rolling a Dice', size = 19)
ax.set_xlim(0, 7)
ax.set_ylim(0, 400)
plt.show()
```

Next we try generating an array from a standard normal distribution $x\sim N(0, 1)$, then plot the histogram. And note that we add one parameter ```density=True``` in the ```hist``` method, it turns into a **relative frequency histogram**, because the $y$-axis represents relative frequencies.

```python
x = np.random.randn(1000)
fig, ax = plt.subplots(figsize=(9, 9))
n, bins, patches = ax.hist(x, bins = 50, density=True)
```

Careful audiences might have noticed that while we are plotting the histogram, we also have 3 variables returned ```n```, ```bins``` and ```patches```. ```n``` is (relative) frequency counts, ```bins``` is the location of each bin, ```patches``` is the list of patches objects in the plot.

The first charts below shows the same information as histogram, but with line plots rather than bars. But the second one is called *Empirical cumulative distribution**, it is empirical because it is not theoretically plotted, i.e. using a **cumulative distribution function (CDF)**.

```python
fig, ax = plt.subplots(nrows = 2, ncols = 1,figsize=(9, 9))
ax[0].plot(bins[:50], n)
ax[1].plot(np.cumsum(n))
plt.show()
```

#  Basic Numeric Descriptives

##  Measure Of Location

The most important **mean** we need is **arithmetic mean**, which is prevalent in all kinds of statistical techniques.
$$
\mu = \frac{1}{N}\sum_{i=1}^Nx_i\\
\bar{x} = \frac{1}{n}\sum_{i=1}^nx_i
$$
The former one is _population mean_, the latter is _sample mean_. The formulae appear the same, but with different indication. $N$ is the population size, imagine the number of all human beings on earth, on the other hand, $n$ is sample size, for instance a sample of $1000$ persons from UK.

And one tricky mean commonly used in finance is **geometric mean**, not so intuitive at first sight.
$$
g = \bigg(\prod_{i=1}^nx_i\bigg)^{1/n}
$$
If you can't make sense out of it, accept a fact that geometric mean is commonly used when calculating _compound growth rates_, such as portfolio return. For instance, a portfolio manager has annual return recorded as below

Portfolio Return

    Year
    Return

    2015
    36%

    2016
    23%

    2017
    -48%

    2018
    -30%

    2019
    15%

    2020
    31%

    Arithmetic Mean
    4.5%

    Geometric Mean
    -1.4%

The arithmetic mean of return is $4.45\%$.

```python
portfolio_return = np.array([0.36, 0.23, -0.48, -0.3, 0.15, 0.31])
np.mean(portfolio_return)
```

    0.045000000000000005

However the geometric mean of return is $-1.42\%$. Geometric mean ```sp.stats.mstats.gmean``` is more accurate measurement when considering the compound effect, i.e. the data are related to each other by a growth rate.

```python
sp.stats.mstats.gmean(portfolio_return + 1) - 1
```

    -0.014282599080668423

##  Measures Of Variability

The first measurement of variability is the **range** measuring distance from lowest to highest. We'll generate an array of standard normal distribution for demonstration.

```python
x = np.random.randn(50)
```

```python
rangeLS = x.max() - x.min()
rangeLS
```

    4.725332255653198

**Percentile** is also a common statistic concept, which could be best explained by an example. For instance, the GRE test result shows percentile besides your absolute score, say you have a percentile of $96\%$, it means your score is higher than $96\%$ of candidates. The special percentile of $75\%$ and $25\%$ are sometimes called the _third quartile_ and _first quartile_.

```python
q75, q25 = np.percentile(x, [75 ,25]) # IQR
q75 - q25
```

    1.1190705852500262

Before moving any further, we must clarify two statistical concepts: **population** and **sample**. For example, we want to know the variance of all human adults height, therefore we ought to measure all adults (population) on earth in order to calculate the variance, however the mission is impossible, instead we measure a smaller group of adults (sample) to make inferential statements about population.

Population and sample variance differ in degree of freedom, where $N$ is the population size, whereas $n$ is the sample size, $\mu$ is the population mean and $\bar{x}$ is the sample mean. The formulae of variances are

$$
\sigma^2 = \frac{\sum(x_i - \mu)^2}{N}\\
s^2 = \frac{\sum(x_i - \bar{x})^2}{n-1}
$$

The latter is an unbiased estimator of population variance, which is also the _sample variance_.

To illustrate the idea, let's pretend there are only $N = 1000$ people on earth, we can generate an array to represent the population height, only for demonstrative purpose. We generate a population by setting ```loc=170``` and ```scale=10``` with ```np.random.normal```, i.e. $X\sim N(170, 10)$, then calculate the population variance by using ```np.var()```.

```python
population_height = np.random.normal(170, 10, 1000)
np.var(population_height)
```

    108.33360191813833

Now suppose we know nothing about the population, but we can get a sample of 100 persons. By setting ```ddof=1``` in function, we actually mean the degree of freedom is $N-1$, used on sample estimators.

```python
sample_height = np.random.choice(population_height, size = 100)
np.var(sample_height, ddof = 1)
```

    98.1270651211727

Theoretically, we can have tremendous amount of samples, say $10000$ samples. Yes, I mean $10000$ samples, not the sample size, but this is just a thought experiment, will never be achieved in real world. Again, pure demonstrative purpose.

What we are doing next: generate $10000$ samples, calculate the sample variances, plot histogram. The vertical line is the mean of sampling distribution of variance estimates. We set standard deviation of $\sigma = 10$, the theoretical variance $\sigma^2=100$, therefore we see the point estimator is doing a fair well job.

```python
sample_height_array = []
for i in range(10000):
    sample_height = np.random.choice(population_height, size = 100)
    sample_height_array.append(np.var(sample_height, ddof=1))
fig, ax = plt.subplots(figsize=(9, 9))
n, bins, patches = ax.hist(sample_height_array, bins = 50)
ax.axvline(x=np.mean(sample_height_array), color = 'tomato')
ax.text(np.mean(sample_height_array)+1, np.max(n), r'$\mu_\sigma = {:.2f}$'.format(np.mean(sample_height_array)), size = 16)
ax.set_title('Sampling Distribution of Variance Estimates', size = 19)
plt.show()
```

I guess you have noticed the power of estimators now, if you have many samples, by using an unbiased sample estimator, you will get an accurate estimate of population parameters. Even if you have only one sample, you can still can estimate how possible you would be correct based on sampling distribution.

But we shall keep in mind that **standard deviation** is the most popular measurement of variability. The population/sample standard deviation is simply the square root of variances respectively.

$$
\sigma = \sqrt{\sigma^2}\\
s = \sqrt{s^2}
$$

Similar to variance function in NumPy.

```python
np.std(sample_height_array, ddof=1)
```

    15.50977277891788

##  Measures of Distribution Shape

**z-score** is defined as below, used for measuring how many standard deviations away from the mean.

\begin{equation}
z_i = \frac{x_i-\bar{x}}{s}
\end{equation}

Note that $z$ has subscript notation $i$ which means each observation has its own $z$-score.

Actually，measuring how many standard deviations away from the mean (or hypothesis) is the fundamental philosophy of frequentist statistics, it asks one important question: how far away from the mean is far-away? If it is far enough, very likely the mean we are looking at right now is not the 'real' mean of the random mechanism that generates the observation.

Here is the example of calculating $z$-score for an randomly generated array. So you can see for a standard normal distribution, it would be fairly hard to stray $2$ standard deviations away.

```python
x = np.random.randn(10)
z = (x - np.mean(x))/np.std(x)
np.round(z, 2)
```

    array([ 0.09, -0.59,  0.6 , -0.91,  1.06, -1.44,  0.78, -1.58,  0.69,
            1.3 ])

###  Chebyshev's Theorem

**Chebyshev's Theorem** is used as the last resort of deduction when we have absolute _no knowledge_ of a sample and its distribution. It guarantees minimum proportion of data that must be within $z$ standard deviation from the mean, where the $z$ is $z$-score. But the downside of the theorem is that it only addresses the symmetric distribution.

\begin{equation}
p \geq 1-\frac{1}{z^2}
\end{equation}

For instance, the average height of people in Helsinki is $174cm$, with a standard deviation of of $4cm$, so the question is how many people (percentage) are within $166cm$ and $182cm$? Note that this range is symmetrically $2$ standard derivations away from its mean. Thus according to Chebyshev, we know that

```python
z_l = (166 - 174)/4 # lower z-score
z_u = (182 - 174)/4 # upper z-score
```

Because it is a symmetric range, $z$-score on each sides are equal, we can calculate the probability as below and print the conclusion.

```python
p = 1 - 1/z_l**2
print('At least {0}% of people are within 168cm and 182cm in Helsinki.'.format(p*100))
```

    At least 75.0% of people are within 168cm and 182cm in Helsinki.

If you think of Chebyshev as a function, then the only variable is $z$-score. Now we define a Chebyshev function and plot it against $z$-score.

Because $z$-score means how many standard deviation away from the mean, from the graph we could conclude that no matter what types of distributions we are investigating, it is guaranteed that $\pm 2.5$ standard deviation range would cover at least $84\%$ of data.

```python
def chebyshev(z):
    return 1 - 1/z**2

chebyshev_array = []
for z in np.arange(1, 21, 0.5):
    chebyshev_array.append(chebyshev(z))

fig, ax = plt.subplots(figsize=(9, 9))
ax.plot(np.arange(1, 21, 0.5), chebyshev_array)
ax.scatter(2.5, chebyshev(2.5), s = 100, color = 'red', zorder = 3)
ax.text(2.5+.5, chebyshev(2.5), r'(2.5, {}%)'.format(chebyshev(2.5)*100))
ax.set_title("Chebyshev's Theorem")
ax.set_xlabel('z-score')
ax.set_ylabel('Probability')
plt.show()
```

However in practice we are more likely to deal with data of (semi)bell-shape distribution, they are regular and easier to make deduction. So **Empirical Rules** apply.

* 68% of data within 1 standard deviation of the mean
* 95% of data within 2 standard deviation of the mean
* 99.7% of data within 3 standard deviation of the mean

These empirical numbers are from normal distribution.

##  Measures of Association Between Two Variables

With similar notation, the **population** and **sample covariance** is defined as:
$$
\sigma_{xy} \frac{\sum(x_i-\mu_x)(y_i-\mu_y)}{N}\\
s_{xy} \frac{\sum(x_i-\bar{x})(y_i-\bar{y})}{n-1}
$$

Generate two random arrays, then evaluate a covariance matrix. The values on the diagonal presents the variance, the off-diagonal presents the covariance.

```python
x = np.random.randn(300)
y = np.random.rand(300)
np.cov(x, y)
```

    array([[ 0.88955858, -0.00649224],
           [-0.00649224,  0.07885574]])

**Correlation Coefficient** is normalized version of covariance, just like the standard deviation is the normalised variance, some statistics textbook calls it Pearson Product Moment Correlation Coefficient.

$$
\rho = \frac{\sigma_{xy}}{\sigma_x\sigma_y}\\
r_{xy}=\frac{s_{xy}}{s_xs_y}
$$

For the sake of everyone's sanity, not to cluster the lecture notes with repetitive codes, some of them were transported in to ```plot_material.py``` module. Take a look inside if you are curious.

What we did below was basically generating eight linear regression plot $Y = \beta_1+\beta_2X+u$ with different parameters. The correlation coefficient $\rho$ is displayed in the graph, therefore you could observe that the correlation coefficients are affected by size of parameters $\beta_2$ and scale of error term.

The reasons are:

The more significantly $\beta_2$ differs from $0$, the more significant linear relationship the model has, therefore the relatively larger correlation between $X$ and $Y$.

If the parameters $\beta_1$ and $\beta_2$ kept constant, the larger scale of the error term, the more dispersed the data are, therefore lower correlation as well.

```python
import plot_material
plot_material.reg_corr_plot()
```

We will dive much deeper into linear regression in our econometrics training sessions.

###  Different Types Of Correlation Coefficient

The correlation coefficient above is the **Pearson correlation**, which assumes a linear relationship between two variables. However, there are other correlation coefficients which also measures nonlinear correlation, such as **Spearman correlation** and **Kendall's $\tau$**.

```python
X = np.linspace(-10, 10, 200)
Y = 1/(1+np.exp(-X))
df_dict = {'X': X, 'Y': Y}
df = pd.DataFrame(df_dict)

df.plot(x ='X', y ='Y', kind='scatter', figsize=(16, 7)); plt.show()
```

```python
df.corr(method='pearson')
```

      X
      Y

      X
      1.000000
      0.936137

      Y
      0.936137
      1.000000

```python
print('Pearson coeffcient: {}'.format(sp.stats.stats.pearsonr(df['X'], df['Y'])[0]))
```

    Pearson coeffcient: 0.9361365508325603

```python
print('Pearson coeffcient: {}'.format(sp.stats.stats.spearmanr(df['X'], df['Y'])[0]))
```

    Pearson coeffcient: 1.0

```python
sp.stats.stats.kendalltau(X, Y)
print('Pearson coeffcient: {}'.format(sp.stats.stats.kendalltau(df['X'], df['Y'])[0]))
```

    Pearson coeffcient: 1.0

Notice the difference, that Pearson and Kendall produce a perfect correlation coefficient, it is because the latter two are rank tests.

Mathematics/Statistics/0.0 Basic_Statistics_With_Python/Chapter 1 - Descriptive Statistics.ipynb


    Run in Google Colab

    View source on GitHub

# Deep Learning

Recently deep learning has produced significant advancements on various machine learning tasks. To develop an appreciation of how deep learning has an effect on recommender systems, in this tutorial we look into example models from two broad categories. One models collaborative filtering or user-item interaction data, while the other models content associated with items (or users).

## 1. Setup

```
!pip install --quiet cornac==1.7.1
```

```
import os
import sys

import cornac
from cornac.data import Reader
from cornac.datasets import movielens, citeulike
from cornac.eval_methods import RatioSplit
from cornac.models import GMF, MLP, NeuMF, VAECF, CDL, ConvMF, WMF, CTR
from cornac.data import TextModality
from cornac.data.text import BaseTokenizer

%tensorflow_version 1.x
import tensorflow as tf

print(f"System version: {sys.version}")
print(f"Cornac version: {cornac.__version__}")
print(f"Tensorflow version: {tf.__version__}")

SEED = 42
VERBOSE = True
```

    TensorFlow 1.x selected.
    System version: 3.6.9 (default, Apr 18 2020, 01:56:04)
    [GCC 8.4.0]
    Cornac version: 1.7.1
    Tensorflow version: 1.15.2

## 2. Deep Learning for Collaborative Filtering

In this section, we study two frameworks of deep learning, and how they can be used to model user-item interactions.  Neural collaborative filtering is based on multi-layer perceptrons, while as suggested by its name, variational autoencoder for collaborative filtering is based on VAE.

### Neural Collaborative Filtering

Neural collaborative filtering consists of a family of models developed based on neural networks to tackle the problem of collaborative filtering based on implicit feedback.  The final model NeuMF is a combination of two components, namely Generalized Matrix Factorization (GMF) and Multi-Layer Perceptrons (MLP), which are also independent models respectively.

- GMF projects element-wise product between user and item embedding vectors through *Sigmoid* function to out the prediction:

where $\odot$ is element-wise product, $\mathbf{u}_i$ and $\mathbf{v}_j$ are user and item embeddings, $\mathbf{w}$ are the projection weights, and $\sigma$ is the *Sigmoid* function.

- MLP projects concatenation of user and item embeddings through multiple layers:

where $\phi$ is a non-linear activation function (could be *Tanh* as recommended by the authors).

- NeuMF is a fusion of GMF and MLP:

The models are learned by minimizing log-loss function:

Let's try the models all together!

```
GMF_FACTORS = 8  # @param
MLP_LAYERS = [32, 16, 8]  # @param
ACTIVATION = "tanh"  # @param ["tanh", "sigmoid", "relu", "leaky_relu"]
NEG_SAMPLES = 3  # @param
NUM_EPOCHS = 10  # @param
BATCH_SIZE = 256  # @param
LEARNING_RATE = 0.001  # @param

gmf = GMF(num_factors=GMF_FACTORS, num_epochs=NUM_EPOCHS, batch_size=BATCH_SIZE,
          num_neg=NEG_SAMPLES, lr=LEARNING_RATE, seed=SEED, verbose=VERBOSE)
mlp = MLP(layers=MLP_LAYERS, act_fn=ACTIVATION, num_epochs=NUM_EPOCHS, batch_size=BATCH_SIZE,
          num_neg=NEG_SAMPLES, lr=LEARNING_RATE, seed=SEED, verbose=VERBOSE)
neumf = NeuMF(num_factors=GMF_FACTORS, layers=MLP_LAYERS, act_fn=ACTIVATION, num_epochs=NUM_EPOCHS,
              num_neg=NEG_SAMPLES, batch_size=BATCH_SIZE, lr=LEARNING_RATE, seed=SEED, verbose=VERBOSE)

wmf = WMF(k=GMF_FACTORS, max_iter=200, learning_rate=0.001, seed=SEED, verbose=VERBOSE)

ml_100k = movielens.load_feedback(variant="100K", reader=Reader(bin_threshold=4.0))
ratio_split = RatioSplit(
  data=ml_100k, test_size=0.2, exclude_unknowns=True, seed=SEED, verbose=VERBOSE
)
ndcg_50 = cornac.metrics.NDCG(k=50)
rec_50 = cornac.metrics.Recall(k=50)

cornac.Experiment(
  eval_method=ratio_split,
  models=[gmf, mlp, neumf, wmf],
  metrics=[ndcg_50, rec_50],
).run()
```

    TEST:
    ...
          | NDCG@50 | Recall@50 | Train (s) | Test (s)
    ----- + ------- + --------- + --------- + --------
    GMF   |  0.1609 |    0.2778 |   43.2287 |   1.4973
    MLP   |  0.1630 |    0.2807 |   42.7741 |   1.5705
    NeuMF |  0.2412 |    0.4268 |   43.8981 |   1.7990
    WMF   |  0.1976 |    0.3753 |    8.4105 |   0.7220

### Variational Autoencoder for Collaborative Filtering (VAECF)

Variational Autoencoders (VAE) is a type of autoencoders which is a neural network with auto-associative mapping of inputs.  Normal autoencoders learns a determinisic latent representation of an input, while VAE learns a distribution of that representation.  VAECF model extends VAE for the collaborative filtering problem with implicit feedback data.

Let the data for user $i$ (e.g., history clicks) be a vector $\mathbf{x}_i \in \mathbb{N}^M$ where M is the number of items.  The generative model of VAECF can be described as follows:

- User latent representation is sampled from standard Gaussion prior: $\mathbf{u}_i \sim \mathcal{N}(0, \mathbf{I}_K)$

- $\mathbf{z}_i$ is transformed via a non-linear function $f_{\theta}$ to produce a probability distribution over items (normalized using *softmax*): $\pi(\mathbf{u}_i) \propto \exp \{ f_{\theta}(\mathbf{u}_i) \}$

- User data $\mathbf{x}_i$ is assumed to have been drawn from the multinomial distribution: $\mathbf{x}_i \sim \mathrm{Mult}(N_i, \pi(\mathbf{u}_i))$

The Multinomial (*mult*) log-likelihood for user $i$ conditioned on the latent representation is:

Other choices of likelihood including Bernoulli (*bern*), Gaussian (*gaus*), or Poisson (*pois*) can be used:

The encoder-decoder architecture is due to the use of amortized inference, a technique of variational inference, for learning the generative model (**decoder**) $f_{\theta}$.  Amortized inference learns an inference model (**encoder**) $g_{\phi} (\mathbf{x}_i) \equiv [\mu_{\phi}(\mathbf{x}_i), \sigma_{\phi}(\mathbf{x}_i)]$, which makes use of the observed data $\mathbf{x}_i$ to approximate the true intractable posterior $p(\mathbf{u}_i | \mathbf{x}_i)$ with a simpler variational distribution:

Learning the model parameters $\{\theta, \phi\}$ is done via maximizing the evidence lower bound (ELBO):

where $\beta$ is a hyper-parameter to control the strength of regularization introduced by enforcing small Kullback-Leibler (KL) divergence between posterior $q_{\phi} (\mathbf{u}_i | \mathbf{x}_i)$ and the prior $p(\mathbf{u}_i)$.

```
NUM_FACTORS = 25  # @param
AE_LAYERS = [100, 50]  # @param
ACTIVATION = "tanh"  # @param ["tanh", "sigmoid", "relu", "leaky_relu"]
LIKELIHOOD = "bern"  # @param ["bern", "mult", "gaus", "pois"]
NUM_EPOCHS = 600  # @param
BATCH_SIZE = 256  # @param
LEARNING_RATE = 0.001  # @param

vaecf = VAECF(k=NUM_FACTORS, autoencoder_structure=AE_LAYERS, act_fn=ACTIVATION,
              likelihood=LIKELIHOOD, n_epochs=NUM_EPOCHS, batch_size=BATCH_SIZE,
              learning_rate=LEARNING_RATE, seed=SEED, verbose=VERBOSE, use_gpu=True)

wmf = WMF(k=NUM_FACTORS, max_iter=200, learning_rate=0.001, seed=SEED, verbose=VERBOSE)

ml_100k = movielens.load_feedback(variant="100K", reader=Reader(bin_threshold=4.0))
ratio_split = RatioSplit(
  data=ml_100k, test_size=0.2, exclude_unknowns=True, seed=SEED, verbose=VERBOSE
)

cornac.Experiment(
  eval_method=ratio_split, models=[vaecf, wmf], metrics=[rec_50, ndcg_50],
).run()
```

    TEST:
    ...
          | NDCG@50 | Recall@50 | Train (s) | Test (s)
    ----- + ------- + --------- + --------- + --------
    VAECF |  0.2405 |    0.4250 |   29.1858 |   1.6399
    WMF   |  0.2045 |    0.3958 |    8.8830 |   0.7460

## 3. Deep Learning for Content-Based Filtering

A different way to use deep learning is to model the content of items (or users).  The key idea is to derive low-dimensional representations from this content, originally in the form of text, images, etc., and to use these representations to enhance recommender systems in a similar way to content-based methods.  In the following, we illustrate two examples based on autoencoder and convolutional neural networks respectively.

### Collaborative Deep Learning (CDL)

CDL makes use of Stacked-Denoising Autoencoders (SDAE) to model item text content, with the objective of improving upon Collaborative Topic Regression (CTR) model making use of topic model for the similar purpose.  CDL is a hierarchical Bayesian model, the generative process is as follows:

- For each user $i$
  - Draw latent vector $\mathbf{u}_i \sim \mathcal{N}(0, \lambda^{-1} \mathbf{I})$
- For each item $j$
  - Put its corrupted content $\mathbf{x}_j^{0}$ through SDAE
  - Draw clean content $\mathbf{x}_j \sim \mathcal{N}(\mathbf{x}_L, \lambda^{-1}\mathbf{I})$
    - User middle representation $\mathbf{x}_j^{L/2}$
  - Draw item offset $\mathbf{\epsilon}_j \sim \mathcal{N}(0, \lambda^{-1} \mathbf{I})$
  - Set item latent vector $\mathbf{v}_j$ to be: $\mathbf{v}_j = \mathbf{\epsilon}_j + \mathbf{x}_j^{L/2}$
- For each user-item pair $(i, j)$
  - Draw rating $r_{ij} \sim \mathcal{N}(\mathbf{u}_i^T \mathbf{v}_j, c_{ij}^{-1})$

The confidence parameter $c_{ij}$ is defined similar to that for CTR:

Learning CDL model is done via minimizing the loss function:

Let's do a comparison between CDL and CTR models.

```
VOCAB_SIZE = 2000  # @param
NUM_FACTORS = 50  # @param
AE_LAYERS = [500, 100]  # @param
ACTIVATION = "tanh"  # @param ["tanh", "sigmoid", "relu", "leaky_relu"]
A = 1.0  # @param
B = 0.01  # @param
LAMBDA_U = 0.1  # @param
LAMBDA_V = 0.1  # @param
LAMBDA_W = 0.1  # @param
LAMBDA_N = 10.0  # @param
CORRUPTION_RATE = 0.3  # @param
DROPOUT_RATE = 0.1  # @param
NUM_EPOCHS = 100  # @param
BATCH_SIZE = 100  # @param
LEARNING_RATE = 0.001  # @param

cdl = CDL(k=NUM_FACTORS, autoencoder_structure=AE_LAYERS, act_fn=ACTIVATION, a=A, b=B,
          lambda_u=LAMBDA_U, lambda_v=LAMBDA_V, lambda_w=LAMBDA_W, lambda_n=LAMBDA_N,
          corruption_rate=CORRUPTION_RATE, dropout_rate=DROPOUT_RATE, vocab_size=VOCAB_SIZE,
          max_iter=NUM_EPOCHS, batch_size=BATCH_SIZE, seed=SEED, verbose=VERBOSE)

ctr = CTR(k=NUM_FACTORS, max_iter=NUM_EPOCHS, a=A, b=B, lambda_u=LAMBDA_U, lambda_v=LAMBDA_V, seed=SEED, verbose=VERBOSE)

plots, movie_ids = movielens.load_plot()
feedback = movielens.load_feedback(variant="100k", reader=Reader(item_set=movie_ids, bin_threshold=4.0))
item_text_modality = TextModality(
  corpus=plots, ids=movie_ids, tokenizer=BaseTokenizer(sep="\t", stop_words="english"), max_vocab=VOCAB_SIZE,
)

ratio_split = RatioSplit(
  data=feedback,
  test_size=0.2,
  exclude_unknowns=True,
  item_text=item_text_modality,
  seed=SEED, verbose=VERBOSE,
)

cornac.Experiment(eval_method=ratio_split, models=[cdl, ctr], metrics=[rec_50, ndcg_50]).run()
```

    TEST:
    ...
        | NDCG@50 | Recall@50 | Train (s) | Test (s)
    --- + ------- + --------- + --------- + --------
    CDL |  0.2063 |    0.4066 |   25.0348 |   0.7186
    CTR |  0.1865 |    0.3343 |  117.5390 |   0.7346

### Convolutional Matrix Factorization (ConvMF)

ConvMF model integrates convolutional neural network (CNN) into probabilistic matrix factorization (PMF).  The model captures contextual information of item documents (e.g., reviews, abstracts, or synopses) and further enhances the rating prediction accuracy.  The generative process is as follows:

- For each user $i$
  - Draw latent vector $\mathbf{u}_i \sim \mathcal{N}(0, \lambda^{-1} \mathbf{I})$
- For each item $j$
  - Put its content $\mathbf{x}_j$ through CNN parameterized by $\mathbf{W}$
  - Draw item offset $\mathbf{\epsilon}_j \sim \mathcal{N}(0, \lambda^{-1} \mathbf{I})$
  - Set item latent vector $\mathbf{v}_j$ to be: $\mathbf{v}_j = \mathbf{\epsilon}_j + \mathrm{CNN}(\mathbf{W}, \mathbf{x}_j)$
- For each user-item pair $(i, j)$
  - Draw rating $r_{ij} \sim \mathcal{N}(\mathbf{u}_i^T \mathbf{v}_j, \sigma^2)$

Learning ConvMF model is done via minimizing the following negative log-likelihood function:

```
VOCAB_SIZE = 2000  # @param
NUM_FACTORS = 5  # @param
HIDDEN_DIM = 50  # @param
CNN_FILTER_SIZES = [2, 3, 4, 5]  # @param
CNN_NUM_FILTERS = 128  # @param
WORD_EMB_DIM = 32  # @param
SEQ_MAX_LENGTH = 50  # @param
DROPOUT_RATE = 0.2  # @param
LAMBDA_U = 0.1  # @param
LAMBDA_V = 0.1  # @param
NUM_EPOCHS = 5  # @param
CNN_EPOCHS = 3  # @param
CNN_BATCH_SIZE = 256  # @param
CNN_LEARNING_RATE = 0.001  # @param

convmf = ConvMF(k=NUM_FACTORS, n_epochs=NUM_EPOCHS, cnn_epochs=CNN_EPOCHS, cnn_bs=CNN_BATCH_SIZE,
                cnn_lr=CNN_LEARNING_RATE, emb_dim=WORD_EMB_DIM, max_len=SEQ_MAX_LENGTH,
                filter_sizes=CNN_FILTER_SIZES, num_filters=CNN_NUM_FILTERS, hidden_dim=HIDDEN_DIM,
                dropout_rate=DROPOUT_RATE, lambda_u=LAMBDA_U, lambda_v=LAMBDA_V,
                seed=SEED, verbose=VERBOSE)

plots, movie_ids = movielens.load_plot()
feedback = movielens.load_feedback(variant="100k", reader=Reader(item_set=movie_ids))
item_text_modality = TextModality(
  corpus=plots, ids=movie_ids, tokenizer=BaseTokenizer(sep="\t", stop_words="english"), max_vocab=VOCAB_SIZE,
)

ratio_split = RatioSplit(
  data=feedback,
  test_size=0.2,
  rating_threshold=4.0,
  exclude_unknowns=True,
  item_text=item_text_modality,
  seed=SEED,
  verbose=VERBOSE
)

rmse = cornac.metrics.RMSE()

cornac.Experiment(
  eval_method=ratio_split, models=[convmf], metrics=[rmse, rec_50], user_based=False
).run()
```

    TEST:
    ...
           |   RMSE | Recall@50 | Train (s) | Test (s)
    ------ + ------ + --------- + --------- + --------
    ConvMF | 0.9795 |    0.0075 |   12.0318 |   0.7525

## References

1.   He, X., Liao, L., Zhang, H., Nie, L., Hu, X., & Chua, T. S. (2017, April). Neural collaborative filtering. In Proceedings of the 26th international conference on world wide web (pp. 173-182).
2.   Liang, D., Krishnan, R. G., Hoffman, M. D., & Jebara, T. (2018, April). Variational autoencoders for collaborative filtering. In Proceedings of the 2018 World Wide Web Conference (pp. 689-698).
3.   Wang, H., Wang, N., & Yeung, D. Y. (2015, August). Collaborative deep learning for recommender systems. In Proceedings of the 21th ACM SIGKDD international conference on knowledge discovery and data mining (pp. 1235-1244).
4.   Kim, D., Park, C., Oh, J., Lee, S., & Yu, H. (2016, September). Convolutional matrix factorization for document context-aware recommendation. In Proceedings of the 10th ACM conference on recommender systems (pp. 233-240).
5.   Cornac - A Comparative Framework for Multimodal Recommender Systems (https://cornac.preferred.ai/)