# Lambda School Data Science Module 132
## Sampling, Confidence Intervals, and Hypothesis Testing

## Prepare - examine other available hypothesis tests

If you had to pick a single hypothesis test in your toolbox, t-test would probably be the best choice - but the good news is you don't have to pick just one! Here's some of the others to be aware of:

```
import numpy as np
from scipy.stats import chisquare  # One-way chi square test

# Chi square can take any crosstab/table and test the independence of rows/cols
# The null hypothesis is that the rows/cols are independent -> low chi square
# The alternative is that there is a dependence -> high chi square
# Be aware! Chi square does *not* tell you direction/causation

ind_obs = np.array([[1, 1], [2, 2]]).T
print(ind_obs)
print(chisquare(ind_obs, axis=None))

dep_obs = np.array([[16, 18, 16, 14, 12, 12], [32, 24, 16, 28, 20, 24]]).T
print(dep_obs)
print(chisquare(dep_obs, axis=None))
```

    [[1 2]
     [1 2]]
    Power_divergenceResult(statistic=0.6666666666666666, pvalue=0.8810148425137847)
    [[16 32]
     [18 24]
     [16 16]
     [14 28]
     [12 20]
     [12 24]]
    Power_divergenceResult(statistic=23.31034482758621, pvalue=0.015975692534127565)

```
# Distribution tests:
# We often assume that something is normal, but it can be important to *check*

# For example, later on with predictive modeling, a typical assumption is that
# residuals (prediction errors) are normal - checking is a good diagnostic

from scipy.stats import normaltest
# Poisson models arrival times and is related to the binomial (coinflip)
sample = np.random.poisson(5, 1000)
print(normaltest(sample))  # Pretty clearly not normal
```

    NormaltestResult(statistic=22.273646023922907, pvalue=1.4565963588062096e-05)

```
# Kruskal-Wallis H-test - compare the median rank between 2+ groups
# Can be applied to ranking decisions/outcomes/recommendations
# The underlying math comes from chi-square distribution, and is best for n>5
from scipy.stats import kruskal

x1 = [1, 3, 5, 7, 9]
y1 = [2, 4, 6, 8, 10]
print(kruskal(x1, y1))  # x1 is a little better, but not "significantly" so

x2 = [1, 1, 1]
y2 = [2, 2, 2]
z = [2, 2]  # Hey, a third group, and of different size!
print(kruskal(x2, y2, z))  # x clearly dominates
```

    KruskalResult(statistic=0.2727272727272734, pvalue=0.6015081344405895)
    KruskalResult(statistic=7.0, pvalue=0.0301973834223185)

And there's many more! `scipy.stats` is fairly comprehensive, though there are even more available if you delve into the extended world of statistics packages. As tests get increasingly obscure and specialized, the importance of knowing them by heart becomes small - but being able to look them up and figure them out when they *are* relevant is still important.

## T-test Assumptions

```
from scipy.stats import ttest_ind

?ttest_ind
```

<https://statistics.laerd.com/statistical-guides/independent-t-test-statistical-guide.php>

- Independence of means

Are the means of our voting data independent (do not affect the outcome of one another)?

The best way to increase thel likelihood of our means being independent is to randomly sample (which we did not do).

- "Homogeneity" of Variance?

Is the magnitude of the variance between the two roughly the same?

I think we're OK on this one for the voting data, although it probably could be better, one party was larger than the other.

If we suspect this to be a problem then we can use Welch's T-test

- "Dependent Variable" (sample means) are Distributed Normally

<https://stats.stackexchange.com/questions/9573/t-test-for-non-normal-when-n50>

Lots of statistical tests depend on normal distributions. We can test for normality using Scipy as was shown above.

This assumption is often assumed even if the assumption is a weak one. If you strongly suspect that things are not normally distributed, you can transform your data to get it looking more normal and then run your test. This problem typically goes away for large sample sizes (yay Central Limit Theorem) and is often why you don't hear it brought up. People declare the assumption to be satisfied either way.

## Central Limit Theorem

```
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd

N = 3000
sample_means = []
for x in range(0, N):
    coinflips = np.random.binomial(n=1, p=.5, size=100)
    one_sample = coinflips
    sample_means.append(coinflips.mean())

print(len(sample_means))
print(sample_means)
```

    3000
    [0.45, 0.52, 0.45, 0.51, 0.45, 0.58, 0.49, 0.49, 0.48, 0.4, 0.41, 0.49, 0.44, 0.49, 0.49, 0.49, 0.44, 0.45, 0.41, 0.48, 0.42, 0.49, 0.53, 0.41, 0.51, 0.57, 0.54, 0.54, 0.47, 0.56, 0.49, 0.54, 0.45, 0.55, 0.46, 0.51, 0.59, 0.51, 0.59, 0.56, 0.59, 0.47, 0.59, 0.41, 0.41, 0.46, 0.56, 0.49, 0.4, 0.44, 0.37, 0.51, 0.51, 0.53, 0.54, 0.4, 0.47, 0.39, 0.53, 0.51, 0.44, 0.55, 0.43, 0.55, 0.52, 0.44, 0.43, 0.45, 0.56, 0.54, 0.47, 0.46, 0.54, 0.53, 0.61, 0.52, 0.56, 0.5, 0.49, 0.55, 0.51, 0.5, 0.5, 0.52, 0.51, 0.46, 0.49, 0.42, 0.52, 0.46, 0.48, 0.43, 0.54, 0.47, 0.59, 0.43, 0.56, 0.47, 0.54, 0.5, 0.51, 0.59, 0.51, 0.44, 0.46, 0.5, 0.39, 0.47, 0.55, 0.49, 0.47, 0.53, 0.58, 0.53, 0.4, 0.53, 0.49, 0.47, 0.43, 0.4, 0.48, 0.64, 0.46, 0.49, 0.43, 0.58, 0.49, 0.54, 0.44, 0.46, 0.47, 0.53, 0.52, 0.52, 0.5, 0.5, 0.5, 0.46, 0.51, 0.55, 0.44, 0.52, 0.49, 0.45, 0.44, 0.44, 0.5, 0.48, 0.45, 0.49, 0.44, 0.46, 0.46, 0.54, 0.56, 0.57, 0.47, 0.45, 0.43, 0.53, 0.54, 0.51, 0.48, 0.46, 0.5, 0.52, 0.51, 0.47, 0.51, 0.51, 0.47, 0.45, 0.54, 0.59, 0.6, 0.44, 0.52, 0.48, 0.57, 0.45, 0.56, 0.49, 0.4, 0.56, 0.4, 0.47, 0.46, 0.46, 0.54, 0.49, 0.45, 0.51, 0.45, 0.56, 0.43, 0.58, 0.51, 0.45, 0.45, 0.48, 0.5, 0.61, 0.46, 0.49, 0.49, 0.42, 0.49, 0.48, 0.52, 0.51, 0.52, 0.53, 0.54, 0.43, 0.55, 0.48, 0.6, 0.52, 0.52, 0.46, 0.53, 0.53, 0.51, 0.47, 0.53, 0.42, 0.4, 0.52, 0.44, 0.56, 0.54, 0.49, 0.5, 0.46, 0.5, 0.44, 0.55, 0.58, 0.39, 0.51, 0.55, 0.53, 0.49, 0.48, 0.38, 0.55, 0.52, 0.56, 0.43, 0.49, 0.5, 0.58, 0.48, 0.54, 0.47, 0.48, 0.46, 0.5, 0.54, 0.49, 0.5, 0.53, 0.53, 0.49, 0.51, 0.54, 0.47, 0.54, 0.55, 0.48, 0.45, 0.5, 0.44, 0.57, 0.57, 0.48, 0.53, 0.5, 0.54, 0.48, 0.47, 0.47, 0.55, 0.46, 0.55, 0.39, 0.45, 0.4, 0.49, 0.52, 0.56, 0.5, 0.47, 0.44, 0.49, 0.49, 0.58, 0.53, 0.5, 0.41, 0.44, 0.45, 0.4, 0.58, 0.49, 0.49, 0.37, 0.59, 0.44, 0.5, 0.43, 0.48, 0.51, 0.47, 0.46, 0.54, 0.51, 0.51, 0.51, 0.63, 0.51, 0.47, 0.52, 0.53, 0.57, 0.49, 0.52, 0.58, 0.51, 0.51, 0.53, 0.53, 0.44, 0.52, 0.53, 0.44, 0.56, 0.51, 0.55, 0.45, 0.53, 0.52, 0.59, 0.45, 0.53, 0.46, 0.5, 0.41, 0.45, 0.48, 0.46, 0.47, 0.45, 0.45, 0.55, 0.51, 0.51, 0.54, 0.51, 0.46, 0.49, 0.42, 0.59, 0.51, 0.56, 0.37, 0.48, 0.49, 0.43, 0.43, 0.49, 0.46, 0.56, 0.55, 0.5, 0.52, 0.55, 0.53, 0.48, 0.51, 0.48, 0.44, 0.5, 0.53, 0.52, 0.37, 0.41, 0.51, 0.44, 0.49, 0.59, 0.61, 0.47, 0.6, 0.54, 0.52, 0.48, 0.45, 0.52, 0.51, 0.47, 0.58, 0.43, 0.48, 0.52, 0.35, 0.47, 0.47, 0.5, 0.47, 0.58, 0.52, 0.56, 0.45, 0.58, 0.48, 0.42, 0.4, 0.49, 0.6, 0.59, 0.49, 0.53, 0.61, 0.59, 0.54, 0.5, 0.53, 0.47, 0.44, 0.43, 0.61, 0.48, 0.53, 0.54, 0.51, 0.47, 0.53, 0.5, 0.61, 0.55, 0.53, 0.5, 0.6, 0.45, 0.5, 0.53, 0.44, 0.57, 0.57, 0.46, 0.54, 0.57, 0.47, 0.48, 0.53, 0.47, 0.48, 0.66, 0.5, 0.47, 0.47, 0.45, 0.5, 0.54, 0.57, 0.47, 0.56, 0.43, 0.53, 0.52, 0.55, 0.61, 0.51, 0.55, 0.45, 0.52, 0.49, 0.55, 0.55, 0.55, 0.47, 0.49, 0.45, 0.51, 0.47, 0.42, 0.58, 0.45, 0.45, 0.57, 0.48, 0.49, 0.43, 0.48, 0.5, 0.47, 0.58, 0.51, 0.58, 0.53, 0.44, 0.44, 0.51, 0.53, 0.45, 0.49, 0.52, 0.53, 0.43, 0.51, 0.47, 0.46, 0.51, 0.48, 0.45, 0.56, 0.48, 0.58, 0.48, 0.47, 0.51, 0.5, 0.57, 0.44, 0.44, 0.47, 0.43, 0.45, 0.53, 0.51, 0.51, 0.54, 0.5, 0.57, 0.44, 0.51, 0.46, 0.48, 0.48, 0.56, 0.46, 0.52, 0.5, 0.57, 0.52, 0.55, 0.47, 0.47, 0.54, 0.59, 0.54, 0.58, 0.49, 0.41, 0.55, 0.62, 0.55, 0.51, 0.5, 0.51, 0.5, 0.51, 0.52, 0.5, 0.53, 0.47, 0.36, 0.45, 0.5, 0.61, 0.61, 0.47, 0.36, 0.43, 0.5, 0.54, 0.5, 0.49, 0.5, 0.45, 0.48, 0.47, 0.56, 0.53, 0.58, 0.56, 0.53, 0.39, 0.53, 0.49, 0.47, 0.43, 0.49, 0.52, 0.52, 0.45, 0.57, 0.44, 0.54, 0.49, 0.54, 0.55, 0.57, 0.44, 0.47, 0.52, 0.44, 0.54, 0.53, 0.53, 0.45, 0.53, 0.41, 0.46, 0.5, 0.46, 0.48, 0.55, 0.53, 0.57, 0.52, 0.57, 0.57, 0.46, 0.46, 0.59, 0.5, 0.52, 0.62, 0.58, 0.52, 0.4, 0.47, 0.51, 0.53, 0.45, 0.49, 0.58, 0.56, 0.61, 0.58, 0.57, 0.54, 0.56, 0.45, 0.41, 0.52, 0.53, 0.54, 0.45, 0.42, 0.45, 0.54, 0.52, 0.48, 0.46, 0.56, 0.4, 0.44, 0.41, 0.51, 0.49, 0.44, 0.55, 0.52, 0.49, 0.51, 0.51, 0.61, 0.47, 0.38, 0.51, 0.6, 0.49, 0.46, 0.45, 0.55, 0.51, 0.49, 0.46, 0.39, 0.6, 0.56, 0.42, 0.46, 0.55, 0.61, 0.57, 0.57, 0.51, 0.47, 0.47, 0.51, 0.55, 0.44, 0.59, 0.56, 0.5, 0.61, 0.57, 0.45, 0.61, 0.47, 0.53, 0.52, 0.44, 0.56, 0.46, 0.53, 0.49, 0.47, 0.52, 0.41, 0.58, 0.51, 0.48, 0.46, 0.56, 0.36, 0.53, 0.57, 0.58, 0.48, 0.45, 0.41, 0.51, 0.46, 0.56, 0.52, 0.5, 0.51, 0.49, 0.52, 0.5, 0.43, 0.48, 0.53, 0.52, 0.48, 0.43, 0.48, 0.5, 0.44, 0.45, 0.47, 0.56, 0.51, 0.5, 0.45, 0.48, 0.51, 0.52, 0.47, 0.52, 0.52, 0.44, 0.53, 0.47, 0.38, 0.46, 0.46, 0.56, 0.57, 0.55, 0.49, 0.43, 0.54, 0.55, 0.5, 0.45, 0.47, 0.41, 0.64, 0.5, 0.49, 0.51, 0.58, 0.43, 0.53, 0.49, 0.52, 0.5, 0.51, 0.46, 0.53, 0.52, 0.48, 0.56, 0.57, 0.54, 0.5, 0.56, 0.53, 0.56, 0.5, 0.46, 0.52, 0.42, 0.51, 0.53, 0.47, 0.53, 0.49, 0.55, 0.47, 0.42, 0.48, 0.55, 0.46, 0.51, 0.43, 0.52, 0.56, 0.45, 0.54, 0.52, 0.53, 0.64, 0.54, 0.57, 0.53, 0.44, 0.47, 0.44, 0.51, 0.54, 0.52, 0.51, 0.53, 0.54, 0.53, 0.48, 0.55, 0.61, 0.44, 0.57, 0.51, 0.59, 0.41, 0.45, 0.48, 0.45, 0.48, 0.48, 0.52, 0.48, 0.49, 0.47, 0.5, 0.5, 0.47, 0.42, 0.46, 0.45, 0.48, 0.51, 0.44, 0.47, 0.42, 0.61, 0.41, 0.49, 0.47, 0.39, 0.58, 0.46, 0.46, 0.48, 0.6, 0.48, 0.52, 0.55, 0.51, 0.54, 0.5, 0.59, 0.53, 0.48, 0.45, 0.49, 0.45, 0.41, 0.5, 0.54, 0.5, 0.51, 0.52, 0.52, 0.58, 0.47, 0.55, 0.5, 0.49, 0.51, 0.52, 0.61, 0.54, 0.5, 0.5, 0.48, 0.49, 0.48, 0.5, 0.52, 0.55, 0.47, 0.5, 0.54, 0.46, 0.47, 0.5, 0.54, 0.49, 0.44, 0.61, 0.55, 0.42, 0.56, 0.5, 0.49, 0.49, 0.52, 0.43, 0.56, 0.43, 0.44, 0.52, 0.47, 0.58, 0.54, 0.43, 0.43, 0.43, 0.54, 0.58, 0.51, 0.53, 0.55, 0.48, 0.52, 0.47, 0.57, 0.59, 0.55, 0.43, 0.56, 0.49, 0.43, 0.49, 0.43, 0.42, 0.5, 0.6, 0.52, 0.44, 0.5, 0.53, 0.53, 0.51, 0.5, 0.53, 0.49, 0.55, 0.47, 0.53, 0.53, 0.48, 0.48, 0.51, 0.52, 0.46, 0.52, 0.55, 0.45, 0.46, 0.53, 0.51, 0.38, 0.55, 0.48, 0.43, 0.47, 0.49, 0.44, 0.51, 0.46, 0.55, 0.42, 0.48, 0.52, 0.48, 0.41, 0.51, 0.44, 0.52, 0.51, 0.47, 0.45, 0.52, 0.59, 0.47, 0.46, 0.54, 0.41, 0.5, 0.47, 0.5, 0.54, 0.52, 0.54, 0.47, 0.51, 0.48, 0.48, 0.48, 0.56, 0.54, 0.46, 0.47, 0.58, 0.48, 0.46, 0.47, 0.47, 0.39, 0.47, 0.46, 0.54, 0.52, 0.55, 0.46, 0.5, 0.53, 0.48, 0.48, 0.49, 0.48, 0.46, 0.43, 0.51, 0.56, 0.5, 0.42, 0.46, 0.55, 0.49, 0.52, 0.49, 0.51, 0.49, 0.44, 0.51, 0.44, 0.51, 0.5, 0.47, 0.45, 0.56, 0.58, 0.41, 0.57, 0.45, 0.54, 0.49, 0.51, 0.61, 0.69, 0.49, 0.46, 0.56, 0.44, 0.44, 0.54, 0.59, 0.52, 0.56, 0.51, 0.54, 0.51, 0.58, 0.56, 0.53, 0.5, 0.51, 0.47, 0.55, 0.48, 0.57, 0.43, 0.48, 0.43, 0.6, 0.43, 0.48, 0.46, 0.51, 0.5, 0.37, 0.53, 0.52, 0.56, 0.48, 0.49, 0.57, 0.49, 0.44, 0.44, 0.44, 0.52, 0.56, 0.45, 0.51, 0.53, 0.49, 0.52, 0.45, 0.49, 0.52, 0.48, 0.46, 0.54, 0.5, 0.52, 0.51, 0.47, 0.49, 0.53, 0.6, 0.45, 0.59, 0.5, 0.59, 0.47, 0.41, 0.49, 0.51, 0.55, 0.45, 0.4, 0.53, 0.43, 0.56, 0.4, 0.56, 0.51, 0.44, 0.43, 0.57, 0.52, 0.5, 0.53, 0.5, 0.51, 0.47, 0.58, 0.39, 0.51, 0.39, 0.5, 0.45, 0.48, 0.56, 0.42, 0.48, 0.5, 0.37, 0.59, 0.5, 0.47, 0.57, 0.51, 0.51, 0.59, 0.53, 0.51, 0.5, 0.48, 0.55, 0.47, 0.59, 0.49, 0.59, 0.54, 0.48, 0.53, 0.58, 0.49, 0.51, 0.57, 0.44, 0.45, 0.49, 0.51, 0.55, 0.57, 0.54, 0.47, 0.55, 0.43, 0.56, 0.42, 0.42, 0.51, 0.52, 0.4, 0.49, 0.43, 0.4, 0.52, 0.48, 0.51, 0.53, 0.42, 0.5, 0.48, 0.47, 0.48, 0.56, 0.51, 0.45, 0.42, 0.43, 0.57, 0.56, 0.56, 0.47, 0.55, 0.53, 0.55, 0.52, 0.46, 0.48, 0.5, 0.49, 0.53, 0.42, 0.49, 0.4, 0.54, 0.48, 0.44, 0.5, 0.5, 0.5, 0.47, 0.55, 0.5, 0.54, 0.44, 0.44, 0.5, 0.39, 0.48, 0.54, 0.49, 0.5, 0.52, 0.54, 0.54, 0.5, 0.49, 0.44, 0.51, 0.53, 0.46, 0.55, 0.5, 0.5, 0.51, 0.45, 0.46, 0.55, 0.47, 0.52, 0.53, 0.48, 0.47, 0.51, 0.5, 0.52, 0.57, 0.54, 0.54, 0.5, 0.51, 0.5, 0.46, 0.46, 0.61, 0.49, 0.44, 0.56, 0.47, 0.49, 0.47, 0.55, 0.48, 0.55, 0.46, 0.43, 0.51, 0.54, 0.5, 0.42, 0.5, 0.47, 0.49, 0.51, 0.54, 0.47, 0.52, 0.5, 0.59, 0.4, 0.42, 0.46, 0.46, 0.45, 0.52, 0.49, 0.46, 0.57, 0.54, 0.48, 0.49, 0.49, 0.53, 0.52, 0.53, 0.47, 0.54, 0.52, 0.53, 0.42, 0.59, 0.51, 0.44, 0.57, 0.54, 0.59, 0.44, 0.52, 0.51, 0.43, 0.51, 0.55, 0.59, 0.55, 0.42, 0.47, 0.51, 0.6, 0.58, 0.46, 0.43, 0.47, 0.48, 0.44, 0.52, 0.57, 0.59, 0.49, 0.55, 0.52, 0.47, 0.47, 0.44, 0.53, 0.5, 0.51, 0.51, 0.49, 0.59, 0.5, 0.49, 0.45, 0.53, 0.47, 0.54, 0.45, 0.49, 0.46, 0.5, 0.47, 0.56, 0.51, 0.42, 0.5, 0.45, 0.49, 0.51, 0.53, 0.53, 0.43, 0.51, 0.48, 0.49, 0.54, 0.48, 0.53, 0.51, 0.54, 0.55, 0.48, 0.58, 0.54, 0.52, 0.53, 0.5, 0.53, 0.44, 0.5, 0.49, 0.52, 0.56, 0.55, 0.43, 0.52, 0.54, 0.36, 0.48, 0.47, 0.53, 0.48, 0.53, 0.51, 0.49, 0.52, 0.58, 0.46, 0.48, 0.47, 0.43, 0.43, 0.56, 0.43, 0.46, 0.54, 0.49, 0.52, 0.52, 0.55, 0.52, 0.52, 0.45, 0.5, 0.45, 0.51, 0.54, 0.42, 0.45, 0.46, 0.44, 0.52, 0.48, 0.52, 0.53, 0.45, 0.47, 0.48, 0.53, 0.56, 0.53, 0.5, 0.52, 0.44, 0.53, 0.34, 0.37, 0.5, 0.44, 0.44, 0.44, 0.46, 0.5, 0.52, 0.54, 0.57, 0.51, 0.49, 0.49, 0.46, 0.54, 0.52, 0.52, 0.57, 0.45, 0.43, 0.55, 0.52, 0.53, 0.55, 0.49, 0.54, 0.5, 0.5, 0.43, 0.47, 0.57, 0.41, 0.41, 0.58, 0.49, 0.54, 0.43, 0.53, 0.52, 0.49, 0.45, 0.45, 0.47, 0.51, 0.58, 0.42, 0.49, 0.51, 0.45, 0.47, 0.55, 0.5, 0.48, 0.54, 0.47, 0.53, 0.44, 0.5, 0.51, 0.44, 0.5, 0.46, 0.46, 0.44, 0.49, 0.49, 0.48, 0.5, 0.44, 0.48, 0.47, 0.46, 0.46, 0.42, 0.5, 0.53, 0.48, 0.41, 0.5, 0.56, 0.44, 0.5, 0.46, 0.58, 0.52, 0.47, 0.54, 0.52, 0.55, 0.47, 0.52, 0.48, 0.51, 0.42, 0.55, 0.54, 0.4, 0.58, 0.53, 0.48, 0.46, 0.46, 0.54, 0.5, 0.45, 0.6, 0.54, 0.53, 0.57, 0.55, 0.52, 0.46, 0.53, 0.43, 0.49, 0.58, 0.46, 0.5, 0.37, 0.54, 0.52, 0.57, 0.61, 0.39, 0.54, 0.54, 0.48, 0.52, 0.46, 0.55, 0.48, 0.53, 0.45, 0.49, 0.52, 0.46, 0.5, 0.5, 0.45, 0.58, 0.5, 0.48, 0.44, 0.44, 0.51, 0.53, 0.59, 0.48, 0.51, 0.5, 0.52, 0.51, 0.57, 0.42, 0.44, 0.61, 0.51, 0.46, 0.45, 0.43, 0.61, 0.43, 0.58, 0.48, 0.51, 0.47, 0.47, 0.44, 0.52, 0.51, 0.47, 0.48, 0.52, 0.51, 0.53, 0.5, 0.51, 0.47, 0.45, 0.47, 0.53, 0.49, 0.52, 0.55, 0.48, 0.53, 0.51, 0.5, 0.58, 0.48, 0.44, 0.51, 0.52, 0.54, 0.6, 0.44, 0.53, 0.48, 0.49, 0.47, 0.44, 0.53, 0.5, 0.55, 0.52, 0.55, 0.46, 0.46, 0.54, 0.47, 0.51, 0.52, 0.48, 0.56, 0.55, 0.53, 0.51, 0.54, 0.55, 0.41, 0.44, 0.49, 0.55, 0.47, 0.48, 0.52, 0.45, 0.47, 0.56, 0.55, 0.51, 0.5, 0.48, 0.57, 0.54, 0.51, 0.55, 0.56, 0.54, 0.52, 0.46, 0.45, 0.49, 0.51, 0.49, 0.51, 0.49, 0.58, 0.56, 0.47, 0.48, 0.54, 0.52, 0.57, 0.56, 0.49, 0.52, 0.5, 0.55, 0.44, 0.48, 0.5, 0.49, 0.58, 0.53, 0.55, 0.49, 0.49, 0.52, 0.57, 0.58, 0.54, 0.46, 0.46, 0.56, 0.48, 0.52, 0.53, 0.49, 0.48, 0.54, 0.62, 0.45, 0.57, 0.51, 0.51, 0.56, 0.47, 0.57, 0.48, 0.46, 0.52, 0.54, 0.54, 0.56, 0.48, 0.5, 0.58, 0.51, 0.43, 0.56, 0.47, 0.47, 0.52, 0.52, 0.52, 0.48, 0.47, 0.48, 0.51, 0.48, 0.53, 0.45, 0.52, 0.53, 0.5, 0.46, 0.49, 0.5, 0.51, 0.52, 0.49, 0.45, 0.51, 0.47, 0.49, 0.59, 0.46, 0.53, 0.57, 0.52, 0.51, 0.56, 0.43, 0.49, 0.55, 0.41, 0.44, 0.54, 0.57, 0.62, 0.47, 0.46, 0.6, 0.49, 0.48, 0.53, 0.55, 0.57, 0.46, 0.47, 0.38, 0.38, 0.5, 0.51, 0.54, 0.52, 0.57, 0.47, 0.48, 0.49, 0.49, 0.5, 0.44, 0.34, 0.53, 0.47, 0.49, 0.54, 0.49, 0.49, 0.52, 0.42, 0.43, 0.49, 0.57, 0.52, 0.48, 0.48, 0.47, 0.5, 0.48, 0.61, 0.61, 0.54, 0.51, 0.64, 0.47, 0.53, 0.49, 0.45, 0.58, 0.57, 0.55, 0.43, 0.53, 0.55, 0.49, 0.47, 0.5, 0.43, 0.6, 0.53, 0.44, 0.51, 0.53, 0.53, 0.59, 0.55, 0.43, 0.51, 0.44, 0.42, 0.44, 0.39, 0.4, 0.47, 0.51, 0.46, 0.53, 0.53, 0.57, 0.56, 0.63, 0.52, 0.55, 0.4, 0.49, 0.56, 0.5, 0.41, 0.52, 0.46, 0.48, 0.46, 0.5, 0.44, 0.56, 0.53, 0.46, 0.6, 0.62, 0.5, 0.51, 0.51, 0.53, 0.52, 0.38, 0.5, 0.48, 0.57, 0.46, 0.51, 0.44, 0.49, 0.6, 0.51, 0.44, 0.52, 0.55, 0.51, 0.56, 0.52, 0.63, 0.59, 0.43, 0.53, 0.55, 0.49, 0.49, 0.52, 0.46, 0.57, 0.49, 0.39, 0.54, 0.51, 0.51, 0.43, 0.49, 0.52, 0.52, 0.58, 0.48, 0.44, 0.47, 0.42, 0.5, 0.53, 0.5, 0.46, 0.57, 0.49, 0.55, 0.56, 0.47, 0.54, 0.49, 0.49, 0.53, 0.45, 0.51, 0.5, 0.39, 0.43, 0.52, 0.61, 0.39, 0.4, 0.42, 0.55, 0.52, 0.52, 0.52, 0.5, 0.54, 0.45, 0.46, 0.45, 0.51, 0.54, 0.47, 0.48, 0.47, 0.45, 0.56, 0.52, 0.51, 0.52, 0.55, 0.42, 0.54, 0.53, 0.5, 0.53, 0.52, 0.52, 0.51, 0.56, 0.44, 0.49, 0.46, 0.46, 0.49, 0.56, 0.47, 0.44, 0.44, 0.52, 0.47, 0.55, 0.46, 0.54, 0.46, 0.57, 0.53, 0.53, 0.59, 0.45, 0.43, 0.42, 0.52, 0.41, 0.51, 0.52, 0.56, 0.51, 0.47, 0.55, 0.45, 0.51, 0.56, 0.53, 0.53, 0.45, 0.44, 0.48, 0.48, 0.44, 0.52, 0.49, 0.48, 0.49, 0.53, 0.51, 0.59, 0.5, 0.52, 0.43, 0.57, 0.59, 0.5, 0.5, 0.54, 0.47, 0.53, 0.5, 0.5, 0.41, 0.52, 0.47, 0.52, 0.43, 0.4, 0.49, 0.59, 0.48, 0.54, 0.46, 0.5, 0.5, 0.44, 0.58, 0.45, 0.51, 0.58, 0.52, 0.47, 0.42, 0.48, 0.44, 0.53, 0.56, 0.52, 0.53, 0.48, 0.46, 0.64, 0.56, 0.59, 0.59, 0.52, 0.46, 0.57, 0.41, 0.49, 0.46, 0.55, 0.49, 0.52, 0.53, 0.5, 0.49, 0.47, 0.51, 0.57, 0.56, 0.55, 0.44, 0.44, 0.59, 0.53, 0.56, 0.52, 0.55, 0.54, 0.55, 0.47, 0.52, 0.54, 0.54, 0.43, 0.5, 0.48, 0.55, 0.52, 0.48, 0.44, 0.43, 0.54, 0.5, 0.59, 0.45, 0.49, 0.55, 0.44, 0.56, 0.46, 0.61, 0.55, 0.48, 0.42, 0.49, 0.52, 0.49, 0.53, 0.53, 0.5, 0.55, 0.51, 0.51, 0.53, 0.46, 0.45, 0.44, 0.57, 0.5, 0.53, 0.56, 0.55, 0.44, 0.59, 0.38, 0.48, 0.49, 0.46, 0.44, 0.55, 0.5, 0.51, 0.47, 0.5, 0.5, 0.48, 0.51, 0.47, 0.5, 0.46, 0.42, 0.56, 0.4, 0.49, 0.49, 0.46, 0.5, 0.55, 0.53, 0.44, 0.5, 0.49, 0.49, 0.45, 0.46, 0.48, 0.48, 0.52, 0.59, 0.51, 0.56, 0.62, 0.44, 0.55, 0.53, 0.54, 0.48, 0.54, 0.52, 0.44, 0.56, 0.43, 0.47, 0.53, 0.42, 0.57, 0.5, 0.43, 0.47, 0.53, 0.39, 0.49, 0.48, 0.54, 0.45, 0.46, 0.51, 0.51, 0.57, 0.48, 0.52, 0.56, 0.53, 0.49, 0.5, 0.51, 0.48, 0.45, 0.44, 0.47, 0.48, 0.51, 0.51, 0.46, 0.56, 0.52, 0.48, 0.47, 0.44, 0.43, 0.52, 0.56, 0.44, 0.55, 0.44, 0.49, 0.61, 0.55, 0.51, 0.51, 0.51, 0.6, 0.54, 0.5, 0.48, 0.46, 0.51, 0.51, 0.68, 0.42, 0.56, 0.55, 0.49, 0.54, 0.53, 0.4, 0.4, 0.53, 0.48, 0.46, 0.49, 0.46, 0.49, 0.51, 0.52, 0.57, 0.57, 0.49, 0.51, 0.51, 0.47, 0.56, 0.51, 0.52, 0.5, 0.55, 0.62, 0.54, 0.59, 0.5, 0.66, 0.43, 0.41, 0.43, 0.46, 0.49, 0.52, 0.53, 0.48, 0.47, 0.39, 0.46, 0.5, 0.45, 0.45, 0.48, 0.57, 0.44, 0.55, 0.52, 0.49, 0.6, 0.54, 0.5, 0.49, 0.51, 0.53, 0.55, 0.47, 0.56, 0.52, 0.55, 0.46, 0.45, 0.57, 0.44, 0.49, 0.55, 0.53, 0.54, 0.53, 0.55, 0.45, 0.55, 0.56, 0.53, 0.53, 0.49, 0.49, 0.54, 0.57, 0.44, 0.54, 0.42, 0.51, 0.49, 0.4, 0.45, 0.49, 0.52, 0.47, 0.43, 0.49, 0.62, 0.47, 0.57, 0.51, 0.49, 0.45, 0.54, 0.46, 0.48, 0.41, 0.48, 0.49, 0.48, 0.48, 0.43, 0.44, 0.48, 0.56, 0.52, 0.47, 0.54, 0.51, 0.58, 0.48, 0.52, 0.46, 0.47, 0.48, 0.5, 0.53, 0.47, 0.52, 0.46, 0.48, 0.34, 0.54, 0.55, 0.48, 0.49, 0.5, 0.46, 0.53, 0.53, 0.56, 0.55, 0.44, 0.58, 0.52, 0.51, 0.45, 0.56, 0.5, 0.45, 0.53, 0.48, 0.55, 0.63, 0.54, 0.44, 0.55, 0.41, 0.52, 0.46, 0.49, 0.53, 0.5, 0.5, 0.37, 0.45, 0.53, 0.46, 0.49, 0.58, 0.65, 0.49, 0.51, 0.51, 0.48, 0.42, 0.48, 0.49, 0.41, 0.47, 0.52, 0.47, 0.48, 0.45, 0.52, 0.57, 0.55, 0.47, 0.51, 0.51, 0.51, 0.47, 0.49, 0.56, 0.52, 0.47, 0.42, 0.52, 0.48, 0.51, 0.54, 0.45, 0.47, 0.47, 0.58, 0.43, 0.38, 0.63, 0.62, 0.51, 0.5, 0.49, 0.52, 0.53, 0.54, 0.53, 0.44, 0.58, 0.53, 0.46, 0.55, 0.48, 0.54, 0.49, 0.47, 0.51, 0.49, 0.48, 0.48, 0.49, 0.52, 0.47, 0.51, 0.58, 0.48, 0.45, 0.51, 0.44, 0.59, 0.47, 0.6, 0.53, 0.55, 0.44, 0.54, 0.51, 0.49, 0.45, 0.43, 0.51, 0.54, 0.48, 0.49, 0.46, 0.56, 0.52, 0.45, 0.47, 0.51, 0.58, 0.55, 0.49, 0.49, 0.5, 0.49, 0.44, 0.51, 0.55, 0.46, 0.56, 0.51, 0.55, 0.52, 0.47, 0.48, 0.47, 0.53, 0.48, 0.52, 0.49, 0.56, 0.44, 0.54, 0.55, 0.55, 0.4, 0.48, 0.51, 0.53, 0.49, 0.48, 0.49, 0.49, 0.52, 0.55, 0.43, 0.42, 0.56, 0.4, 0.55, 0.54, 0.54, 0.59, 0.48, 0.51, 0.49, 0.49, 0.55, 0.48, 0.5, 0.42, 0.62, 0.57, 0.53, 0.42, 0.55, 0.47, 0.47, 0.57, 0.42, 0.49, 0.53, 0.55, 0.41, 0.46, 0.57, 0.57, 0.49, 0.57, 0.54, 0.44, 0.52, 0.48, 0.53, 0.48, 0.56, 0.55, 0.5, 0.45, 0.55, 0.47, 0.5, 0.49, 0.49, 0.48, 0.45, 0.37, 0.5, 0.52, 0.51, 0.46, 0.42, 0.5, 0.52, 0.53, 0.52, 0.48, 0.56, 0.45, 0.42, 0.53, 0.56, 0.48, 0.52, 0.42, 0.49, 0.5, 0.49, 0.53, 0.55, 0.57, 0.56, 0.46, 0.52, 0.54, 0.55, 0.5, 0.49, 0.6, 0.48, 0.44, 0.41, 0.48, 0.46, 0.52, 0.49, 0.51, 0.51, 0.56, 0.51, 0.45, 0.51, 0.47, 0.53, 0.54, 0.49, 0.48, 0.53, 0.42, 0.52, 0.5, 0.42, 0.64, 0.53, 0.52, 0.48, 0.44, 0.5, 0.51, 0.54, 0.5, 0.37, 0.52, 0.56, 0.51, 0.54, 0.56, 0.5, 0.52, 0.52, 0.57, 0.5, 0.52, 0.52, 0.47, 0.57, 0.48, 0.44, 0.45, 0.5, 0.47, 0.53, 0.43, 0.5, 0.62, 0.49, 0.51, 0.47, 0.49, 0.57, 0.51, 0.4, 0.57, 0.49, 0.43, 0.54, 0.39, 0.48, 0.53, 0.49, 0.57, 0.49, 0.52, 0.56, 0.48, 0.47, 0.46, 0.51, 0.51, 0.5, 0.61, 0.53, 0.51, 0.45, 0.48, 0.44, 0.59, 0.42, 0.52, 0.6, 0.51, 0.53, 0.5, 0.43, 0.58, 0.56, 0.57, 0.51, 0.53, 0.49, 0.53, 0.57, 0.54, 0.56, 0.44, 0.52, 0.45, 0.48, 0.49, 0.41, 0.46, 0.47, 0.6, 0.53, 0.47, 0.56, 0.52, 0.56, 0.49, 0.49, 0.52, 0.52, 0.49, 0.59, 0.43, 0.61, 0.53, 0.43, 0.55, 0.45, 0.54, 0.57, 0.49, 0.51, 0.49, 0.53, 0.59, 0.47, 0.45, 0.53, 0.41, 0.46, 0.48, 0.47, 0.53, 0.56, 0.56, 0.55, 0.53, 0.48, 0.45, 0.5, 0.54, 0.5, 0.52, 0.36, 0.47, 0.53, 0.41, 0.49, 0.45, 0.48, 0.49, 0.5, 0.54, 0.43, 0.43, 0.45, 0.48, 0.49, 0.49, 0.54, 0.41, 0.47, 0.56, 0.49, 0.55, 0.39, 0.57, 0.46, 0.47, 0.53, 0.46, 0.45, 0.59, 0.45, 0.48, 0.52, 0.46, 0.5, 0.5, 0.52, 0.51, 0.52, 0.55, 0.49, 0.53, 0.42, 0.58, 0.47, 0.47, 0.56, 0.44, 0.55, 0.39, 0.5, 0.42, 0.59, 0.46, 0.56, 0.52, 0.45, 0.56, 0.52, 0.6, 0.44, 0.49, 0.5, 0.52, 0.47, 0.54, 0.46, 0.56, 0.48, 0.54, 0.45, 0.48, 0.5, 0.54, 0.5, 0.48, 0.5, 0.41, 0.46, 0.54, 0.57, 0.54, 0.5, 0.4, 0.49, 0.5, 0.44, 0.48, 0.5, 0.5, 0.47, 0.5, 0.51, 0.46, 0.53, 0.52, 0.51, 0.48, 0.52, 0.57, 0.42, 0.55, 0.48, 0.5, 0.55, 0.45, 0.53, 0.44, 0.53, 0.39, 0.44, 0.58, 0.44, 0.44, 0.39, 0.47, 0.52, 0.5, 0.49, 0.46, 0.46, 0.52, 0.51, 0.47, 0.6, 0.44, 0.49, 0.56, 0.54, 0.6, 0.52, 0.51, 0.56, 0.47, 0.54, 0.54, 0.49, 0.49, 0.55, 0.51, 0.48, 0.49, 0.43, 0.38, 0.45, 0.6, 0.54, 0.55, 0.46, 0.43, 0.48, 0.46, 0.48, 0.52, 0.52, 0.54, 0.56, 0.45, 0.52, 0.43, 0.58, 0.57, 0.44, 0.51, 0.5, 0.48, 0.46, 0.52, 0.53, 0.44, 0.49, 0.43, 0.52, 0.5, 0.46, 0.51, 0.51, 0.41, 0.53, 0.44, 0.48, 0.55, 0.49, 0.57, 0.54, 0.56, 0.54, 0.5, 0.48, 0.51, 0.5, 0.44]

```
# Create dataframe with single coin flip

df = pd.DataFrame({'one-samp': one_sample})
df.head()
```

      one-samp

      0
      1

      1
      1

      2
      1

      3
      0

      4
      1

```
# Plot histogram to look at distribution of a single coin flip

df.hist();
```

```
# Plot histogram to look at distribution of all coin flips

ax = plt.hist(sample_means, bins=30)
plt.title(f'Distribution of {N} sample means \n (of 30 coinflips each)');
```

What does the Central Limit Theorem State? That no matter the initial distribution of the population, the distribution of sample means taken will approximate a normal distribution as $n \rightarrow \infty$.

This has very important implications for hypothesis testing and is precisely the reason why the t-distribution begins to approximate the normal distribution as our sample size increases.

## Standard Error of the Mean

What does it mean to "estimate"? the Population mean?

```
import numpy as np
import pandas as pd

# Average Height
mu = 70
sigma = 3

lambda_heights = np.random.normal(mu, sigma, 2000)
print(len(lambda_heights))
lambda_heights
```

    2000

    array([71.51294399, 66.75282069, 67.59159409, ..., 71.10525723,
           71.94880877, 69.13737422])

```
import seaborn as sns

sns.distplot(lambda_heights)
plt.title('Distribution of Heights (in inches)');
```

```
print("Population Mean:", lambda_heights.mean())
print("Population Standard Deviation:", lambda_heights.std())
```

    Population Mean: 70.0041524330643
    Population Standard Deviation: 3.0172901619538237

```
population = pd.DataFrame({'heights': lambda_heights})
print(population.shape)
population.head()
```

    (2000, 1)

      heights

      0
      71.512944

      1
      66.752821

      2
      67.591594

      3
      70.164178

      4
      76.079145

```
# Take a random sample and print sample mean

sample1 = population.sample(100)
print(sample1.shape)
sample1.head()
```

    (100, 1)

      heights

      100
      68.989625

      1217
      76.369801

      296
      70.072318

      687
      70.283488

      85
      71.055414

```
print('Sample Mean #1: ', sample1['heights'].mean())
```

    Sample Mean #1:  70.2468623482625

```
# Take a different random sample and print sample mean

sample2 = population.sample(100)
print(sample1.shape)
sample2.head()
```

    (100, 1)

      heights

      537
      74.441399

      1832
      67.811208

      1346
      70.763693

      1479
      71.480101

      229
      69.303825

```
print('Sample Mean #2: ', sample2['heights'].mean())
```

    Sample Mean #2:  70.25036871624675

## Build and Interpret a Confidence Interval

```
coinflips_100 = np.random.binomial(n=1, p=0.5, size=10000)

sample_std = np.std(coinflips_100)
print('Sample St Dev: ', sample_std)
sample_size = len(coinflips_100)
print('Sample Size: ', sample_size)
```

    Sample St Dev:  0.4999963899869677
    Sample Size:  10000

```
standard_error = sample_std/np.sqrt(sample_size)
print(standard_error)
```

    0.0049999638998696775

### What confidence level do we want our confidence interval to represent?

95% confidence Interval? 99% confidence interval?

```
import scipy.stats as stats
```

```
t = stats.t.ppf(0.975, sample_size-1)
t
```

    1.9602012636213575

```
sample_mean = coinflips_100.mean()
confidence_interval = (sample_mean - t*standard_error, sample_mean + t*standard_error)
margin_of_error = t*standard_error

print('Sample Mean: ', sample_mean)
print('Margin of Error: ', margin_of_error)
print('Confidence Interval: ', confidence_interval)
```

    Sample Mean:  0.4981
    Margin of Error:  0.009800935554585713
    Confidence Interval:  (0.48829906444541427, 0.5079009355545857)

## Graphically Represent a Confidence Interval

```
import seaborn as sns

sns.kdeplot(coinflips_100)
plt.axvline(x=sample_mean, color='k')
plt.axvline(x=confidence_interval[0], color='r')
plt.axvline(x=confidence_interval[1], color='r')
```

## Relationship between Confidence Intervals and T-tests

Confidence Interval == Bounds of statistical significance for our t-test

A sample mean that falls inside of our confidence interval will "FAIL TO REJECT" our null hypothesis

A sample mean that falls outside of our confidence interval will "REJECT" our null hypothesis

```
from scipy.stats import t, ttest_1samp
```

```
import numpy as np

coinflip_means = []
for x in range(0,100):
    coinflips = np.random.binomial(n=1, p=.5, size=30)
    coinflip_means.append(coinflips.mean())

print(coinflip_means)
```

    [0.6333333333333333, 0.4666666666666667, 0.43333333333333335, 0.5333333333333333, 0.6333333333333333, 0.5333333333333333, 0.5, 0.4666666666666667, 0.4666666666666667, 0.5, 0.4, 0.6, 0.5333333333333333, 0.5666666666666667, 0.3333333333333333, 0.5333333333333333, 0.4, 0.5, 0.4666666666666667, 0.5, 0.6333333333333333, 0.3333333333333333, 0.5666666666666667, 0.5666666666666667, 0.43333333333333335, 0.5333333333333333, 0.5, 0.6666666666666666, 0.5333333333333333, 0.5333333333333333, 0.6, 0.36666666666666664, 0.5333333333333333, 0.5333333333333333, 0.43333333333333335, 0.43333333333333335, 0.43333333333333335, 0.5666666666666667, 0.3, 0.43333333333333335, 0.43333333333333335, 0.43333333333333335, 0.5333333333333333, 0.4666666666666667, 0.4, 0.6333333333333333, 0.4, 0.5666666666666667, 0.5666666666666667, 0.5333333333333333, 0.5666666666666667, 0.5333333333333333, 0.4666666666666667, 0.5333333333333333, 0.5, 0.5666666666666667, 0.43333333333333335, 0.4666666666666667, 0.5, 0.43333333333333335, 0.5666666666666667, 0.5, 0.6, 0.6, 0.4666666666666667, 0.5, 0.36666666666666664, 0.4666666666666667, 0.5666666666666667, 0.5333333333333333, 0.4666666666666667, 0.43333333333333335, 0.5333333333333333, 0.5, 0.6333333333333333, 0.6333333333333333, 0.43333333333333335, 0.43333333333333335, 0.4, 0.43333333333333335, 0.6, 0.5666666666666667, 0.3, 0.6333333333333333, 0.36666666666666664, 0.5666666666666667, 0.43333333333333335, 0.43333333333333335, 0.4666666666666667, 0.43333333333333335, 0.6666666666666666, 0.4666666666666667, 0.6666666666666666, 0.4666666666666667, 0.6, 0.36666666666666664, 0.5333333333333333, 0.43333333333333335, 0.5666666666666667, 0.5]

```
np.mean(coinflip_means)
```

    0.5003333333333333

```
# 95% confidence interval

t_stat = stats.t.ppf(0.975, 99)
print('T statistic: ', t_stat)

std_sample = np.std(coinflip_means)
std_err = std_sample/np.sqrt(len(coinflip_means))

CI = stats.t.interval(0.95, 99, loc=np.mean(coinflip_means), scale=std_err)
print('95% confidence interval: ', CI)
```

    T statistic:  1.9842169515086827
    95% confidence interval:  (0.48379832435242015, 0.5168683423142464)

A null hypothesis that's just inside of our confidence interval == fail to reject

```
ttest_1samp(coinflip_means, 0.51686)
```

    Ttest_1sampResult(statistic=-1.973274871531969, pvalue=0.05125222761123825)

A null hypothesis that's just outside of our confidence interval == reject

```
ttest_1samp(coinflip_means, 0.52686)
```

    Ttest_1sampResult(statistic=-3.1672693480539342, pvalue=0.0020463163356837315)

```
def confidence_interval(data, confidence=0.95):
  """
  Calculate a confidence interval around a sample mean for given data.
  Using t-distribution and two-tailed test, default 95% confidence.

  Arguments:
    data - iterable (list or numpy array) of sample observations
    confidence - level of confidence for the interval

  Returns:
    tuple of (mean, lower bound, upper bound)
  """
  data = np.array(data)
  mean = np.mean(data)
  n = len(data)
  stderr = stats.sem(data)
  interval = stderr * stats.t.ppf((1 + confidence) / 2.0, n - 1)
  return (mean, mean - interval, mean + interval)
```

## Run a $\chi^{2}$ Test "by hand" (Using Numpy)

```
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/adult.csv', na_values=" ?")
print(df.shape)
df.head()
```

    (32561, 15)

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary

      0
      39
      State-gov
      77516
      Bachelors
      13
      Never-married
      Adm-clerical
      Not-in-family
      White
      Male
      2174
      0
      40
      United-States
      <=50K

      1
      50
      Self-emp-not-inc
      83311
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      13
      United-States
      <=50K

      2
      38
      Private
      215646
      HS-grad
      9
      Divorced
      Handlers-cleaners
      Not-in-family
      White
      Male
      0
      0
      40
      United-States
      <=50K

      3
      53
      Private
      234721
      11th
      7
      Married-civ-spouse
      Handlers-cleaners
      Husband
      Black
      Male
      0
      0
      40
      United-States
      <=50K

      4
      28
      Private
      338409
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      Black
      Female
      0
      0
      40
      Cuba
      <=50K

```
df.corr()
```

      age
      fnlwgt
      education-num
      capital-gain
      capital-loss
      hours-per-week

      age
      1.000000
      -0.076646
      0.036527
      0.077674
      0.057775
      0.068756

      fnlwgt
      -0.076646
      1.000000
      -0.043195
      0.000432
      -0.010252
      -0.018768

      education-num
      0.036527
      -0.043195
      1.000000
      0.122630
      0.079923
      0.148123

      capital-gain
      0.077674
      0.000432
      0.122630
      1.000000
      -0.031615
      0.078409

      capital-loss
      0.057775
      -0.010252
      0.079923
      -0.031615
      1.000000
      0.054256

      hours-per-week
      0.068756
      -0.018768
      0.148123
      0.078409
      0.054256
      1.000000

```
df['hours-per-week'].hist()
```

```
df.describe(exclude='number')
```

      workclass
      education
      marital-status
      occupation
      relationship
      race
      sex
      country
      salary

      count
      30725
      32561
      32561
      30718
      32561
      32561
      32561
      31978
      32561

      unique
      8
      16
      7
      14
      6
      5
      2
      41
      2

      top
      Private
      HS-grad
      Married-civ-spouse
      Prof-specialty
      Husband
      White
      Male
      United-States
      <=50K

      freq
      22696
      10501
      14976
      4140
      13193
      27816
      21790
      29170
      24720

```
cut_points = [0, 9, 19, 29, 39, 49, 500]
label_names = ['0-9', '10-19', '20-29', '30-39', '40-49', '50+']
df['hours_per_week_categories'] = pd.cut(df['hours-per-week'], cut_points, labels=label_names)
df.hours_per_week_categories.value_counts()
```

    40-49    18336
    50+       6462
    30-39     3667
    20-29     2392
    10-19     1246
    0-9        458
    Name: hours_per_week_categories, dtype: int64

```
df.sex.value_counts()
```

     Male      21790
     Female    10771
    Name: sex, dtype: int64

```
df = df.sort_values(by='hours_per_week_categories')
contingency_table = pd.crosstab(df['sex'], df['hours_per_week_categories'], margins=True)
contingency_table
```

      hours_per_week_categories
      0-9
      10-19
      20-29
      30-39
      40-49
      50+
      All

      sex

      Female
      235
      671
      1287
      1914
      5636
      1028
      10771

      Male
      223
      575
      1105
      1753
      12700
      5434
      21790

      All
      458
      1246
      2392
      3667
      18336
      6462
      32561

## Expected Value Calculation
\begin{align}
expected_{i,j} =\frac{(row_{i} \text{total})(column_{j} \text{total}) }{(\text{total observations})}
\end{align}

```
row_sums = contingency_table.iloc[0:2, 6].values
col_sums = contingency_table.iloc[2, 0:6].values

print(row_sums)
print(col_sums)
```

    [10771 21790]
    [  458  1246  2392  3667 18336  6462]

```
total = contingency_table.loc['All', 'All']
total
```

    32561

```
expected = []
for i in range(len(row_sums)):
    expected_row = []
    for column in col_sums:
        expected_val = column*row_sums[i]/total
        expected_row.append(expected_val)
    expected.append(expected_row)

expected = np.array(expected)
print(expected.shape)
print(expected)
```

    (2, 6)
    [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

```
contingency_table
```

      hours_per_week_categories
      0-9
      10-19
      20-29
      30-39
      40-49
      50+
      All

      sex

      Female
      235
      671
      1287
      1914
      5636
      1028
      10771

      Male
      223
      575
      1105
      1753
      12700
      5434
      21790

      All
      458
      1246
      2392
      3667
      18336
      6462
      32561

## Chi-Squared Statistic with Numpy

\begin{align}
\chi^2 = \sum \frac{(observed_{i}-expected_{i})^2}{(expected_{i})}
\end{align}

For the $observed$ values we will just use a version of our contingency table without the margins as a numpy array. In this way, if our observed values array and our expected values array are the same shape, then we can subtract them and divide them directly which makes the calculations a lot cleaner. No for loops!

```
observed = pd.crosstab(df['sex'], df['hours_per_week_categories']).values
print(observed.shape)
observed
```

    (2, 6)

    array([[  235,   671,  1287,  1914,  5636,  1028],
           [  223,   575,  1105,  1753, 12700,  5434]])

```
chi_square = ((observed - expected)**2/(expected)).sum()
chi_square
```

    2287.190943926107

## Run a $\chi^{2}$ Test using Scipy

```
chi_squared, p_value, dof, expected = stats.chi2_contingency(observed)
print(chi_squared, p_value, dof, expected)
```

    2287.190943926107 0.0 5 [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

Null Hypothesis: Hours worked per week bins is **independent** of sex.

Due to a p-value of 0, we REJECT the null hypothesis that hours worked per week and sex are independent, and conclude that there is an association between hours worked per week and sex.

## Assignment - Build a confidence interval

A confidence interval refers to a neighborhood around some point estimate, the size of which is determined by the desired p-value. For instance, we might say that 52% of Americans prefer tacos to burritos, with a 95% confidence interval of +/- 5%.

52% (0.52) is the point estimate, and +/- 5% (the interval $[0.47, 0.57]$) is the confidence interval. "95% confidence" means a p-value $\leq 1 - 0.95 = 0.05$.

In this case, the confidence interval includes $0.5$ - which is the natural null hypothesis (that half of Americans prefer tacos and half burritos, thus there is no clear favorite). So in this case, we could use the confidence interval to report that we've failed to reject the null hypothesis.

But providing the full analysis with a confidence interval, including a graphical representation of it, can be a helpful and powerful way to tell your story. Done well, it is also more intuitive to a layperson than simply saying "fail to reject the null hypothesis" - it shows that in fact the data does *not* give a single clear result (the point estimate) but a whole range of possibilities.

How is a confidence interval built, and how should it be interpreted? It does *not* mean that 95% of the data lies in that interval - instead, the frequentist interpretation is "if we were to repeat this experiment 100 times, we would expect the average result to lie in this interval ~95 times."

For a 95% confidence interval and a normal(-ish) distribution, you can simply remember that +/-2 standard deviations contains 95% of the probability mass, and so the 95% confidence interval based on a given sample is centered at the mean (point estimate) and has a range of +/- 2 (or technically 1.96) standard deviations.

Different distributions/assumptions (90% confidence, 99% confidence) will require different math, but the overall process and interpretation (with a frequentist approach) will be the same.

Your assignment - using the data from the prior module ([congressional voting records](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records)):

### Confidence Intervals:
1. Generate and numerically represent a confidence interval
2. Graphically (with a plot) represent the confidence interval
3. Interpret the confidence interval - what does it tell you about the data and its distribution?

### Chi-squared tests:
4. Take a dataset that we have used in the past in class that has **categorical** variables. Pick two of those categorical variables and run a chi-squared tests on that data
  - By hand using Numpy
  - In a single line using Scipy

Stretch goals:

1. Write a summary of your findings, mixing prose and math/code/results. *Note* - yes, this is by definition a political topic. It is challenging but important to keep your writing voice *neutral* and stick to the facts of the data. Data science often involves considering controversial issues, so it's important to be sensitive about them (especially if you want to publish).
2. Apply the techniques you learned today to your project data or other data of your choice, and write/discuss your findings here.
3. Refactor your code so it is elegant, readable, and can be easily run for all issues.

```
# TODO - your code!
```

## Resources

- [Interactive visualize the Chi-Squared test](https://homepage.divms.uiowa.edu/~mbognar/applets/chisq.html)
- [Calculation of Chi-Squared test statistic](https://en.wikipedia.org/wiki/Pearson%27s_chi-squared_test)
- [Visualization of a confidence interval generated by R code](https://commons.wikimedia.org/wiki/File:Confidence-interval.svg)
- [Expected value of a squared standard normal](https://math.stackexchange.com/questions/264061/expected-value-calculation-for-squared-normal-distribution) (it's 1 - which is why the expected value of a Chi-Squared with $n$ degrees of freedom is $n$, as it's the sum of $n$ squared standard normals)


```c++
#pragma cling add_include_path("../../include")
#pragma cling add_include_path("../feltor/inc") // Feltor path
#define THRUST_DEVICE_SYSTEM THRUST_DEVICE_SYSTEM_CPP
#include <iostream>
#include "dg/algorithm.h"
// include json and netcdf?
```

    In file included from input_line_8:2:
    In file included from ../feltor/inc/dg/algorithm.h:8:
    ../feltor/inc/dg/backend/config.h:20:9: warning: NOTE: Fast std::fma(a,b,c) not activated! Using a*b+c instead! [-W#pragma-messages]
    #pragma message( "NOTE: Fast std::fma(a,b,c) not activated! Using a*b+c instead!")
            ^
    In file included from input_line_8:2:
    In file included from ../feltor/inc/dg/algorithm.h:11:
    In file included from ../feltor/inc/dg/topology/split_and_join.h:4:
    In file included from ../feltor/inc/dg/backend/blas1_dispatch_shared.h:12:
    In file included from ../feltor/inc/dg/backend/blas1_serial.h:6:
    In file included from ../feltor/inc/dg/backend/exblas/exdot_serial.h:25:
    In file included from ../feltor/inc/dg/backend/exblas/accumulate.h:19:
    ../feltor/inc/dg/backend/exblas/config.h:31:9: warning: WARNING: Instruction set below SSE4.1! Deactivating vectorization!
          [-W#pragma-messages]
    #pragma message("WARNING: Instruction set below SSE4.1! Deactivating vectorization!")
            ^
    In file included from input_line_8:2:
    In file included from ../feltor/inc/dg/algorithm.h:11:
    In file included from ../feltor/inc/dg/topology/split_and_join.h:4:
    In file included from ../feltor/inc/dg/backend/blas1_dispatch_shared.h:12:
    In file included from ../feltor/inc/dg/backend/blas1_serial.h:6:
    In file included from ../feltor/inc/dg/backend/exblas/exdot_serial.h:25:
    ../feltor/inc/dg/backend/exblas/accumulate.h:93:43: warning: shifting a negative signed value is undefined [-Wshift-negative-value]
            carrybit = (s ? 1ll << KRX : -1ll << KRX);
                                         ~~~~ ^
    In file included from input_line_8:2:
    In file included from ../feltor/inc/dg/algorithm.h:11:
    In file included from ../feltor/inc/dg/topology/split_and_join.h:4:
    In file included from ../feltor/inc/dg/backend/blas1_dispatch_shared.h:12:
    In file included from ../feltor/inc/dg/backend/blas1_serial.h:6:
    In file included from ../feltor/inc/dg/backend/exblas/exdot_serial.h:26:
    ../feltor/inc/dg/backend/exblas/ExSUM.FPE.hpp:143:46: warning: unknown attribute 'optimize' ignored [-Wunknown-attributes]
    template<typename T, int N, typename TRAITS> UNROLL_ATTRIBUTE
                                                 ^
    ../feltor/inc/dg/backend/exblas/config.h:42:41: note: expanded from macro 'UNROLL_ATTRIBUTE'
    #define UNROLL_ATTRIBUTE __attribute__((optimize("unroll-loops")))
                                            ^
    In file included from input_line_8:2:
    In file included from ../feltor/inc/dg/algorithm.h:11:
    In file included from ../feltor/inc/dg/topology/split_and_join.h:4:
    In file included from ../feltor/inc/dg/backend/blas1_dispatch_shared.h:12:
    In file included from ../feltor/inc/dg/backend/blas1_serial.h:6:
    In file included from ../feltor/inc/dg/backend/exblas/exdot_serial.h:26:
    ../feltor/inc/dg/backend/exblas/ExSUM.FPE.hpp:189:46: warning: unknown attribute 'optimize' ignored [-Wunknown-attributes]
    template<typename T, int N, typename TRAITS> UNROLL_ATTRIBUTE
                                                 ^
    ../feltor/inc/dg/backend/exblas/config.h:42:41: note: expanded from macro 'UNROLL_ATTRIBUTE'
    #define UNROLL_ATTRIBUTE __attribute__((optimize("unroll-loops")))
                                            ^
    In file included from input_line_8:2:
    In file included from ../feltor/inc/dg/algorithm.h:11:
    In file included from ../feltor/inc/dg/topology/split_and_join.h:4:
    In file included from ../feltor/inc/dg/backend/blas1_dispatch_shared.h:12:
    In file included from ../feltor/inc/dg/backend/blas1_serial.h:6:
    In file included from ../feltor/inc/dg/backend/exblas/exdot_serial.h:26:
    ../feltor/inc/dg/backend/exblas/ExSUM.FPE.hpp:221:46: warning: unknown attribute 'optimize' ignored [-Wunknown-attributes]
    template<typename T, int N, typename TRAITS> UNROLL_ATTRIBUTE
                                                 ^
    ../feltor/inc/dg/backend/exblas/config.h:42:41: note: expanded from macro 'UNROLL_ATTRIBUTE'
    #define UNROLL_ATTRIBUTE __attribute__((optimize("unroll-loops")))
                                            ^

(sec:pdes)=
# Advanced Timesteppers

We now want to demonstrate how to use Feltor to solve partial differential equations.
We use the simple advection diffusion equation as a model equation
\begin{align}
    \frac{\partial \omega}{\partial t} &= -v\cdot \nabla\omega + D \Delta \omega \\
     -\Delta \phi &= \omega \\
     v_x &:= -\partial_y \phi \\
     v_y &:= \partial_x \phi
\end{align}

## Explicit stepping
As long as the diffusion coefficient is small enough to not influence the CFL condition we can compute everything explicitly. In Feltor we simply need a functor implementing the right hand side like we did in the
last chapter. We here repeat the building blocks that regard the timestepper:

```cpp
template<class Geometry, class Matrix, class Container>
struct Equations
{
    void operator()(double t, const Container& omega, Container& omegaDot)
    {
        // solve Poisson equation
        // implement advection term
        // implement diffusion term
    }
};
// Construct init condition
omega = myproject::initial_conditions(grid, js["init"] );

// Construct Equations
myproject::Equations<dg::x::CartesianGrid2d, dg::x::DMatrix,
    dg::x::DVec> rhs( grid, js);

// The timestepper
dg::Adaptive< dg::ERKStep< dg::x::DVec>> adapt(tableau, omega);

// The timeloop
dg::AdaptiveTimeloop<dg::x::DVec> timeloop( adapt, rhs,
                    dg::pid_control, dg::l2norm, rtol, atol);
for( unsigned u=1; u<=maxout; u++)
{

    timeloop.integrate( time, omega, u*deltaT, omega,
                          u < maxout ? dg::to::at_least : dg::to::exact);
    // ...
}
```

## Explicit advection - implicit diffusion
We want to split the PDE into two parts: $E(\phi, \omega) = -\vec v \cdot \nabla \omega$ and $I(\omega) = D\Delta\omega$ with $\phi = S(\omega) = \Delta^{-1}\omega$.
We intend to use a semi-implicit time integrator:
```cpp
template<class Geometry, class Matrix, class Container>
struct Explicit
{
    Explicit(...){}
    void operator()( double t, const Container& omega,
                    Container& k)
    {
        // Solve Phi = S(omega) with e.g. Multigrid
        // Compute k = E(phi,omega)
    }
    void implicit_part( double t, const Container& omega,
                       Container& k)
    {
        // Compute k = D Delta omega
    }
};
```
```{note}
There are several possibilities to partition the explicit, implicit and solver parts into functors (or lambdas in the main program). Generally, it is a good idea to keep all equation related functionality in one class. Since we cannot overload the `operator()` twice we reverted to a little trick, writing the `implicit_part` method that we later bind to the `operator()` of the Implicit class below.
```
```cpp
template<class Geometry, class Matrix, class Container>
struct Implicit
{
    Implicit( Explicit<...>& exp, ...): m_exp(exp) {...}
    void operator()( double t, const Container& omega, Container& k)
    {
        m_exp->implicit_part( t, omega, k);
    }
```
```{note}
For the solve method we here chose a PCG solver since the Laplace is self-adjoint. Note, how we used  a small lambda wrapper to compute the implicit left hand side.

Typically, the solver would also write some information about its performance to `std::cout` so that a user is kept informed about the status of the integration.
```
```cpp
    void operator()( double alpha, double t, Container& omega, const Container& rhs)
    {
        auto wrapper = [=]( const auto& x, auto& y){
            // x - a I (x,t)
            operator()( t, x, y); // calls the above operator
            dg::blas1::axpby( 1., x, -alpha, y);
        };
        dg::blas1::copy( rhs, omega); // use rhs as initial guess
        unsigned number = m_pcg.solve( wrapper, omega, rhs, 1., m_weights, m_eps_time);
    }
    private:
    Explicit<Geometry,Matrix,Container>& m_exp;
    dg::PCG< Container> m_pcg;
    Container m_weights;
    value_type m_eps_time;
};

// Construct equations
Explicit<...> ex( ...);
Implicit<...> im(ex, ...);

// The timestepper
dg::Adaptive< dg::ARKStep< dg::x::DVec>> adapt(tableau, omega);

// The timeloop
dg::AdaptiveTimeloop<dg::x::DVec> timeloop( adapt, std::tie( ex, im, im),
                    dg::pid_control, dg::l2norm, rtol, atol);
```

## Implicit advection-diffusion solver
In order to solve the entire system implicitly we have to write both equations and Solvers.
To make it more clear let us reformulate the structure of equations that we have
\begin{align}
    \dot \omega &= I(\omega, \phi)\\
     0 &= R(\omega, \phi)
\end{align}
with
\begin{align}
I( \omega, \phi) &= -\vec v \cdot \nabla \omega + D\Delta\omega \\
R( \omega, \phi) &= \Delta \phi + \omega
\end{align}
To implement an implicit timestepper we need to solve the equation (the mass matrix is the identity)
\begin{align}
\begin{cases}
    \omega - \alpha I(\omega, \phi) &= \omega^*  \\
    \omega+\Delta\phi &= 0
\end{cases}
\end{align}
We can solve this equation for $\omega$ by first solving
\begin{align}
    -\Delta\phi - \alpha I(-\Delta\phi, \phi) = \omega^*
\end{align}
for $\phi$ and then using $\omega = -\Delta\phi$.
```{note}
The solution for $\phi$ is the same for both variants but the solution for $\omega$ is not (numerically). This can be seen by setting $\alpha=0$. Then in the first version $\omega=\omega^*$ but in the second version $\omega=-\Delta\phi = \omega^* + \mathcal O(\epsilon)$, depends on how well the equation is solved. For this reason the accuracy of the implicit solver should be well higher than the accuracy of the timestepper.
```

Since these equations are non-linear the solver needs to be a non-linear solver. Our idea is to use a multigrid FAS solver. For this solver we need to implement both the operator as well as its inverse on multiple grids.

```cpp
template<class Geometry, class Matrix, class Container>
struct Equations //corresponds to I
{
    void rhs( double t, const Container& omega, const Container& phi, Container& omegaDot)
    {
    // compute: omegaDot = I(omega, phi, t)
        // implement advection term
        // implement diffusion term
    }
    void compute_omega( const Container& phi, Container& omega)
    {
        // compute -Delta phi
        dg::blas2::symv( m_lapM, phi, omega);
    }
};
template<class Geometry, class Matrix, class Container>
struct Implicit
{
    Implicit(...)
    {
        // constructed nested grids
        ...
        for ( unsigned u=0; u<stages; u++)
        {
            // construct Equations
            m_eqs[u] = ...
            m_imp[u] = [&, m_omega = m_weights[u]]
                ( const auto& phi, auto& f) mutable
            {
                // omega - a I(omega, phi)
                m_eqs[u].compute_omega( phi, m_omega );
                m_eqs[u].rhs( m_time, m_omega, phi, f);
                dg::blas1::axpby( 1., m_omega, -m_alpha, f);
            }
            m_inv_imp[u] = [acc = dg::AndersonAcceleration( ...),
                           &I = m_imp[u],
                           &m_weights = m_weights[u]]
                ( const auto& omS, auto& phi) mutable
            {
                // Solve Implicit( phi) = omS
                acc.solve( I, phi, omS, m_weights...);
            };
        }
    }

    void operator()(double t, double alpha, Container& omega, const Container& omS)
    {
        // Note how the operators and solvers hold these as references
        // so they know when they are updated
        m_time = t, m_alpha = alpha;

        // Solve the implicit equation
        nested_iterations( m_imp, m_phi, omS, m_inv_imp, m_nested);
        m_eqs[0].compute_omega( m_phi, omega);
    }
    private:
    // since we capture by reference we cannot easily copy this class
    Implicit( const Implicit&);
    Implicit( Implicit&&);
    double m_time, m_alpha;
    dg::NestedGrids< Geometry, Matrix, Container> m_nested;
    Container m_phi;
    std::vector<Container> m_weights;
    std::vector<Equations<Geometry, Matrix, Container>> m_eqs;
    std::vector< std::function<void( const Container&, Container&)>> m_imp, m_inv_imp;
};

// Construct init condition
omega = myproject::initial_conditions(grid, js["init"] );

// Construct Equations
myproject::Equations<dg::x::CartesianGrid2d, dg::x::DMatrix,
    dg::x::DVec> rhs( grid, js);

// The timestepper
dg::Adaptive< dg::DIRKStep< dg::x::DVec>> adapt(tableau, omega);

// The timeloop
dg::AdaptiveTimeloop<dg::x::DVec> timeloop( adapt, std::tie(imp,imp),
                    dg::pid_control, dg::l2norm, rtol, atol);
for( unsigned u=1; u<=maxout; u++)
{

    timeloop.integrate( time, omega, u*deltaT, omega,
                          u < maxout ? dg::to::at_least : dg::to::exact);
    // ...
}
```

## Implicit mass-matrix -  timestepper
 Consider the following general equation:
\begin{align}
    M(y,t)\frac{d y}{d t} &= F(y,t)
\end{align}
We intend to solve the resulting implicit equation with multigrid nested iteration. We thus have to implement and solve the following operators on multiply grids:
\begin{align}
     M(y,t) k &= F(y,t) \\
     M(y,t) ( y - y^* ) - \alpha F(y,t) &= 0
\end{align}
To simplify the implementation we restrict ourselves to solvers that
use the implicit solve only (i.e. Multistep, symplectic DIRK and SDIRK timesteppers).

```cpp
template<...>
struct Equations
{
    Equations(...){}
    void mass_matrix( double a, double t, const Container& y,
                     const Container& k,
                    double b, Container& result)
    {
        // Compute result = a M(y,t) k + b result
        //...
    }
    void rhs( double t, const Container& y,  Container& k)
    {
        // Compute k = F(y,t)
    }
};
template< ...>
class ImplicitSolver
{
    ImplicitSolver( ...){
        // Construct nested grids
        ...
        // Construct nested objects m_eqs
        ...
        // Construct nested Operators and Solvers:
        for( unsigend u=0; u<stages; u++)
        {
            m_imp[u] = [&ys = m_ys, &time = m_time, &alpha = m_alpha, k = m_weights[u]]
                ( const auto& y, auto& I) mutable
            {
                //compute M(y,t) ( y- y^*) - a F(y,t)
                m_eqs[u].rhs(  time, y, I);
                dg::blas1::axpby( 1., y, -1., ys, k);
                m_eqs[u].mass_matrix(  1., time, y, k, -alpha,  I);
            };
            m_inv_imp[u] = [acc = dg::AndersonAcceleration( ...),
                                 &I = m_implicit[u],
                                 &m_weights = m_weights[u]]
                ( const auto& zero, auto& y) mutable
            {
                // Solve Implicit( y) = 0
                acc.solve( I, y, zero, m_weights...);
            };
        }
    }
    void operator()( double t, const Container& y,  Container& k)
    {
        assert( false && "This should not be called");
    }
    void operator()( double alpha, double t, Container& y, const Container& ys)
    {
        // Note how the operators and solvers hold these as references
        // so they know when they are updated
        m_time = t, m_alpha = alpha;
        m_nested.project( ys, m_ys);
        // Solve the implicit equation
        nested_iterations( m_imp, y, 0., m_inv_imp, m_nested);
    }
    private:
    // since we capture by reference we cannot easily copy this class
    ImplicitSolver( const ImplicitSolver&);
    ImplicitSolver( ImplicitSolver&&);
    double m_time, m_alpha;
    dg::NestedGrids< ...> m_nested;
    std::vector<Container> m_weights, m_ys;
    std::vector<Equations<...>> m_eqs;
    std::vector< std::function<void( const Container&, Container&)>> m_imp, m_inv_imp;
};

// in main
ImplicitSolver<...> imp(...);
dg::ImplicitMultistep<Vector> multistep("BDF-3-3", y0);
multistep.init( std::tie( imp, imp), t0, y0, dt);
multistep.step( std::tie( imp, imp), t0, y0);
```

```c++

```


```python
!pip install pennylane
from IPython.display import clear_output
clear_output()
```

```python
# This cell is added by sphinx-gallery
# It can be customized to whatever you like
%matplotlib inline
```

Data-reuploading classifier
===========================
*Author: Shahnawaz Ahmed (shahnawaz.ahmed95@gmail.com)*

.. meta::
   :property="og:description": Implement a single-qubit universal quantum classifier using PennyLane.
   :property="og:image": https://pennylane.ai/qml/_images/universal_dnn1.png

.. related::

   tutorial_variational_classifier Variational quantum classifier
   tutorial_multiclass_classification Multiclass margin classifier
   tutorial_expressivity_fourier_series Quantum models as Fourier series

A single-qubit quantum circuit which can implement arbitrary unitary
operations can be used as a universal classifier much like a single
hidden-layered Neural Network. As surprising as it sounds,
`Pérez-Salinas et al. (2019) <https://arxiv.org/abs/1907.02085>`_
discuss this with their idea of 'data
reuploading'. It is possible to load a single qubit with arbitrary
dimensional data and then use it as a universal classifier.

In this example, we will implement this idea with Pennylane - a
python based tool for quantum machine learning, automatic
differentiation, and optimization of hybrid quantum-classical
computations.

Background
----------

We consider a simple classification problem and will train a
single-qubit variational quantum circuit to achieve this goal. The data
is generated as a set of random points in a plane $(x_1, x_2)$ and
labeled as 1 (blue) or 0 (red) depending on whether they lie inside or
outside a circle. The goal is to train a quantum circuit to predict the
label (red or blue) given an input point’s coordinate.

.. figure:: ../demonstrations/data_reuploading/universal_circles.png
   :scale: 65%
   :alt: circles

Transforming quantum states using unitary operations
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

A single-qubit quantum state is characterized by a two-dimensional state
vector and can be visualized as a point in the so-called Bloch sphere.
Instead of just being a 0 (up) or 1 (down), it can exist in a
superposition with say 30% chance of being in the $|0 \rangle$ and
70% chance of being in the $|1 \rangle$ state. This is represented
by a state vector $|\psi \rangle = 0.3|0 \rangle + 0.7|1 \rangle$ -
the probability "amplitude" of the quantum state. In general we can take
a vector $(\alpha, \beta)$ to represent the probabilities of a qubit
being in a particular state and visualize it on the Bloch sphere as an
arrow.

.. figure:: ../demonstrations/data_reuploading/universal_bloch.png
   :scale: 65%
   :alt: bloch

Data loading using unitaries
~~~~~~~~~~~~~~~~~~~~~~~~~~~~

In order to load data onto a single qubit, we use a unitary operation
$U(x_1, x_2, x_3)$ which is just a parameterized
matrix multiplication representing the rotation of the state vector in
the Bloch sphere. E.g., to load $(x_1, x_2)$ into the qubit, we
just start from some initial state vector, $|0 \rangle$,
apply the unitary operation $U(x_1, x_2, 0)$ and end up at a new
point on the Bloch sphere. Here we have padded 0 since our data is only
2D. Pérez-Salinas et al. (2019) discuss how to load a higher
dimensional data point ($[x_1, x_2, x_3, x_4, x_5, x_6]$) by
breaking it down in sets of three parameters
($U(x_1, x_2, x_3), U(x_4, x_5, x_6)$).

Model parameters with data re-uploading
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Once we load the data onto the quantum circuit, we want to have some
trainable nonlinear model similar to a neural network as well as a way of
learning the weights of the model from data. This is again done with
unitaries, $U(\theta_1, \theta_2, \theta_3)$, such that we load the
data first and then apply the weights to form a single layer
$L(\vec \theta, \vec x) = U(\vec \theta)U(\vec x)$. In principle,
this is just application of two matrix multiplications on an input
vector initialized to some value. In order to increase the number of
trainable parameters (similar to increasing neurons in a single layer of
a neural network), we can reapply this layer again and again with new
sets of weights,
$L(\vec \theta_1, \vec x) L(\vec \theta_2, , \vec x) ... L(\vec \theta_L, \vec x)$
for $L$ layers. The quantum circuit would look like the following:

.. figure:: ../demonstrations/data_reuploading/universal_layers.png
   :scale: 75%
   :alt: Layers

The cost function and "nonlinear collapse"
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

So far, we have only performed linear operations (matrix
multiplications) and we know that we need to have some nonlinear
squashing similar to activation functions in neural networks to really
make a universal classifier (Cybenko 1989). Here is where things gets a
bit quantum. After the application of the layers, we will end up at some
point on the Bloch sphere due to the sequence of unitaries implementing
rotations of the input. These are still just linear transformations of
the input state. Now, the output of the model should be a class label
which can be encoded as fixed vectors (Blue = $[1, 0]$, Red =
$[0, 1]$) on the Bloch sphere. We want to end up at either of them
after transforming our input state through alternate applications of
data layer and weights.

We can use the idea of the “collapse” of our quantum state into
one or other class. This happens when we measure the quantum state which
leads to its projection as either the state 0 or 1. We can compute the
fidelity (or closeness) of the output state to the class label making
the output state jump to either $| 0 \rangle$ or
$|1\rangle$. By repeating this process several times, we can
compute the probability or overlap of our output to both labels and
assign a class based on the label our output has a higher overlap. This
is much like having a set of output neurons and selecting the one which
has the highest value as the label.

We can encode the output label as a particular quantum state that we want
to end up in and use Pennylane to find the probability of ending up in that
state after running the circuit. We construct an observable corresponding to
the output label using the `Hermitian <https://pennylane.readthedocs.io/en/latest/code/ops/qubit.html#pennylane.ops.qubit.Hermitian>`_
operator. The expectation value of the observable gives the overlap or fidelity.
We can then define the cost function as the sum of the fidelities for all
the data points after passing through the circuit and optimize the parameters
$(\vec \theta)$ to minimize the cost.

\begin{align}\texttt{Cost} = \sum_{\texttt{data points}} (1 - \texttt{fidelity}(\psi_{\texttt{output}}(\vec x, \vec \theta), \psi_{\texttt{label}}))\end{align}

Now, we can use our favorite optimizer to maximize the sum of the
fidelities over all data points (or batches of datapoints) and find the
optimal weights for classification. Gradient-based optimizers such as
Adam (Kingma et. al., 2014) can be used if we have a good model of
the circuit and how noise might affect it. Or, we can use some
gradient-free method such as L-BFGS (Liu, Dong C., and Nocedal, J., 1989)
to evaluate the gradient and find the optimal weights where we can
treat the quantum circuit as a black-box and the gradients are computed
numerically using a fixed number of function evaluations and iterations.
The L-BFGS method can be used with the PyTorch interface for Pennylane.

Multiple qubits, entanglement and Deep Neural Networks
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

The Universal Approximation Theorem declares that a neural network with
two or more hidden layers can serve as a universal function approximator.
Recently, we have witnessed remarkable progress of learning algorithms using
Deep Neural Networks.

Pérez-Salinas et al. (2019) make a connection to Deep Neural Networks by
describing that in their approach the
“layers” $L_i(\vec \theta_i, \vec x )$ are analogous to the size
of the intermediate hidden layer of a neural network. And the concept of
deep (multiple layers of the neural network) relates to the number
of qubits. So, multiple qubits with entanglement between them could
provide some quantum advantage over classical neural networks. But here,
we will only implement a single qubit classifier.

.. figure:: ../demonstrations/data_reuploading/universal_dnn.png
   :scale: 35%
   :alt: DNN

"Talk is cheap. Show me the code." - Linus Torvalds
---------------------------------------------------

```python
import pennylane as qml
from pennylane import numpy as np
from pennylane.optimize import AdamOptimizer, GradientDescentOptimizer
qml.enable_tape()

import matplotlib.pyplot as plt

# Set a random seed
np.random.seed(42)

# Make a dataset of points inside and outside of a circle
def circle(samples, center=[0.0, 0.0], radius=np.sqrt(2 / np.pi)):
    """
    Generates a dataset of points with 1/0 labels inside a given radius.

    Args:
        samples (int): number of samples to generate
        center (tuple): center of the circle
        radius (float: radius of the circle

    Returns:
        Xvals (array[tuple]): coordinates of points
        yvals (array[int]): classification labels
    """
    Xvals, yvals = [], []

    for i in range(samples):
        x = 2 * (np.random.rand(2)) - 1
        y = 0
        if np.linalg.norm(x - center) < radius:
            y = 1
        Xvals.append(x)
        yvals.append(y)
    return np.array(Xvals), np.array(yvals)

def plot_data(x, y, fig=None, ax=None):
    """
    Plot data with red/blue values for a binary classification.

    Args:
        x (array[tuple]): array of data points as tuples
        y (array[int]): array of data points as tuples
    """
    if fig == None:
        fig, ax = plt.subplots(1, 1, figsize=(5, 5))
    reds = y == 0
    blues = y == 1
    ax.scatter(x[reds, 0], x[reds, 1], c="red", s=20, edgecolor="k")
    ax.scatter(x[blues, 0], x[blues, 1], c="blue", s=20, edgecolor="k")
    ax.set_xlabel("$x_1$")
    ax.set_ylabel("$x_2$")

Xdata, ydata = circle(500)
fig, ax = plt.subplots(1, 1, figsize=(4, 4))
plot_data(Xdata, ydata, fig=fig, ax=ax)
plt.show()

# Define output labels as quantum state vectors
def density_matrix(state):
    """Calculates the density matrix representation of a state.

    Args:
        state (array[complex]): array representing a quantum state vector

    Returns:
        dm: (array[complex]): array representing the density matrix
    """
    return state * np.conj(state).T

label_0 = [[1], [0]]
label_1 = [[0], [1]]
state_labels = [label_0, label_1]
```

```python
Xdata.shape, ydata.shape
```

    ((500, 2), (500,))

Simple classifier with data reloading and fidelity loss
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```python
dev = qml.device("default.qubit", wires=1)
# Install any pennylane-plugin to run on some particular backend

@qml.qnode(dev)
def qcircuit(params, x=None, y=None):
    """A variational quantum circuit representing the Universal classifier.

    Args:
        params (array[float]): array of parameters
        x (array[float]): single input vector
        y (array[float]): single output state density matrix

    Returns:
        float: fidelity between output state and input
    """
    for i in range(len(params[0])):
        qml.Rot(*(params[0][i]*x + params[1][i]), wires=0)
        #qml.Rot(*params[1][i], wires=0)
    return qml.expval(qml.Hermitian(y, wires=[0]))

def cost(params, x, y, state_labels=None):
    """Cost function to be minimized.

    Args:
        params (array[float]): array of parameters
        x (array[float]): 2-d array of input vectors
        y (array[float]): 1-d array of targets
        state_labels (array[float]): array of state representations for labels

    Returns:
        float: loss value to be minimized
    """
    # Compute prediction for each input in data batch
    loss = 0.0
    dm_labels = [density_matrix(s) for s in state_labels]
    for i in range(len(x)):
        f = qcircuit(params, x=x[i], y=dm_labels[y[i]])
        loss = loss + (1 - f) ** 2
    return loss / len(x)
```

Utility functions for testing and creating batches
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```python
def test(params, x, y, state_labels=None):
    """
    Tests on a given set of data.

    Args:
        params (array[float]): array of parameters
        x (array[float]): 2-d array of input vectors
        y (array[float]): 1-d array of targets
        state_labels (array[float]): 1-d array of state representations for labels

    Returns:
        predicted (array([int]): predicted labels for test data
        output_states (array[float]): output quantum states from the circuit
    """
    fidelity_values = []
    dm_labels = [density_matrix(s) for s in state_labels]
    predicted = []

    for i in range(len(x)):
        fidel_function = lambda y: qcircuit(params, x=x[i], y=y)
        fidelities = [fidel_function(dm) for dm in dm_labels]
        best_fidel = np.argmax(fidelities)

        predicted.append(best_fidel)
        fidelity_values.append(fidelities)

    return np.array(predicted), np.array(fidelity_values)

def accuracy_score(y_true, y_pred):
    """Accuracy score.

    Args:
        y_true (array[float]): 1-d array of targets
        y_predicted (array[float]): 1-d array of predictions
        state_labels (array[float]): 1-d array of state representations for labels

    Returns:
        score (float): the fraction of correctly classified samples
    """
    score = y_true == y_pred
    return score.sum() / len(y_true)

def iterate_minibatches(inputs, targets, batch_size):
    """
    A generator for batches of the input data

    Args:
        inputs (array[float]): input data
        targets (array[float]): targets

    Returns:
        inputs (array[float]): one batch of input data of length `batch_size`
        targets (array[float]): one batch of targets of length `batch_size`
    """
    for start_idx in range(0, inputs.shape[0] - batch_size + 1, batch_size):
        idxs = slice(start_idx, start_idx + batch_size)
        yield inputs[idxs], targets[idxs]
```

Train a quantum classifier on the circle dataset
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

```python
# Generate training and test data
num_training = 200
num_test = 2000

Xdata, y_train = circle(num_training)
X_train = np.hstack((Xdata, np.zeros((Xdata.shape[0], 1))))

Xtest, y_test = circle(num_test)
X_test = np.hstack((Xtest, np.zeros((Xtest.shape[0], 1))))

# Train using Adam optimizer and evaluate the classifier
num_layers = 10
learning_rate = 0.6
epochs = 10
batch_size = 32

opt = AdamOptimizer(learning_rate, beta1=0.9, beta2=0.999)

# initialize random weights
theta = np.random.uniform(size=(num_layers, 3))
w = np.random.uniform(size=(num_layers, 3))
params = [w, theta]

predicted_train, fidel_train = test(params, X_train, y_train, state_labels)
accuracy_train = accuracy_score(y_train, predicted_train)

predicted_test, fidel_test = test(params, X_test, y_test, state_labels)
accuracy_test = accuracy_score(y_test, predicted_test)

# save predictions with random weights for comparison
initial_predictions = predicted_test

loss = cost(params, X_test, y_test, state_labels)

print(
    "Epoch: {:2d} | Cost: {:3f} | Train accuracy: {:3f} | Test Accuracy: {:3f}".format(
        0, loss, accuracy_train, accuracy_test
    )
)

for it in range(epochs):
    for Xbatch, ybatch in iterate_minibatches(X_train, y_train, batch_size=batch_size):
        params = opt.step(lambda v: cost(v, Xbatch, ybatch, state_labels), params)

    predicted_train, fidel_train = test(params, X_train, y_train, state_labels)
    accuracy_train = accuracy_score(y_train, predicted_train)
    loss = cost(params, X_train, y_train, state_labels)

    predicted_test, fidel_test = test(params, X_test, y_test, state_labels)
    accuracy_test = accuracy_score(y_test, predicted_test)
    res = [it + 1, loss, accuracy_train, accuracy_test]
    print(
        "Epoch: {:2d} | Loss: {:3f} | Train accuracy: {:3f} | Test accuracy: {:3f}".format(
            *res
        )
    )
```

    Epoch:  0 | Cost: 0.397512 | Train accuracy: 0.440000 | Test Accuracy: 0.414000

    /usr/local/lib/python3.6/dist-packages/pennylane/tape/tapes/jacobian_tape.py:461: VisibleDeprecationWarning: Creating an ndarray from ragged nested sequences (which is a list-or-tuple of lists-or-tuples-or ndarrays with different lengths or shapes) is deprecated. If you meant to do this, you must specify 'dtype=object' when creating the ndarray
      params = np.array(params)

    Epoch:  1 | Loss: 0.214109 | Train accuracy: 0.660000 | Test accuracy: 0.605500
    Epoch:  2 | Loss: 0.192650 | Train accuracy: 0.670000 | Test accuracy: 0.691500
    Epoch:  3 | Loss: 0.126039 | Train accuracy: 0.810000 | Test accuracy: 0.808500
    Epoch:  4 | Loss: 0.093838 | Train accuracy: 0.870000 | Test accuracy: 0.829000
    Epoch:  5 | Loss: 0.076994 | Train accuracy: 0.940000 | Test accuracy: 0.874500
    Epoch:  6 | Loss: 0.079456 | Train accuracy: 0.920000 | Test accuracy: 0.866000
    Epoch:  7 | Loss: 0.060256 | Train accuracy: 0.960000 | Test accuracy: 0.890000
    Epoch:  8 | Loss: 0.059333 | Train accuracy: 0.955000 | Test accuracy: 0.893500
    Epoch:  9 | Loss: 0.064725 | Train accuracy: 0.940000 | Test accuracy: 0.896000
    Epoch: 10 | Loss: 0.063304 | Train accuracy: 0.935000 | Test accuracy: 0.875000

Results
~~~~~~~

```python
print(
    "Cost: {:3f} | Train accuracy {:3f} | Test Accuracy : {:3f}".format(
        loss, accuracy_train, accuracy_test
    )
)

print("Learned weights")
for i in range(num_layers):
    print("Layer {}: {}".format(i, params[i]))

fig, axes = plt.subplots(1, 3, figsize=(10, 3))
plot_data(X_test, initial_predictions, fig, axes[0])
plot_data(X_test, predicted_test, fig, axes[1])
plot_data(X_test, y_test, fig, axes[2])
axes[0].set_title("Predictions with random weights")
axes[1].set_title("Predictions after training")
axes[2].set_title("True test data")
plt.show()
```

References
----------
[1] Pérez-Salinas, Adrián, et al. “Data re-uploading for a universal
quantum classifier.” arXiv preprint arXiv:1907.02085 (2019).

[2] Kingma, Diederik P., and Ba, J. "Adam: A method for stochastic
optimization." arXiv preprint arXiv:1412.6980 (2014).

[3] Liu, Dong C., and Nocedal, J. "On the limited memory BFGS
method for large scale optimization." Mathematical programming
45.1-3 (1989): 503-528.