## Congressional Voting Assignment - Apply the t-test to real data

Your assignment is to determine which issues have "statistically significant" differences between political parties in this [1980s congressional voting data](https://archive.ics.uci.edu/ml/datasets/Congressional+Voting+Records). The data consists of 435 instances (one for each congressperson), a class (democrat or republican), and 16 binary attributes (yes or no for voting for or against certain issues). Be aware - there are missing values!

Your goals:

1. Load and clean the data (or determine the best method to drop observations when running tests)
2. Using hypothesis testing, find an issue that democrats support more than republicans with p < 0.01
3. Using hypothesis testing, find an issue that republicans support more than democrats with p < 0.01
4. Using hypothesis testing, find an issue where the difference between republicans and democrats has p > 0.1 (i.e. there may not be much of a difference)

Note that this data will involve *2 sample* t-tests, because you're comparing averages across two groups (republicans and democrats) rather than a single group against a null hypothesis.

```python
# Grab the file from UCI
!wget https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
```

    --2019-04-11 18:44:53--  https://archive.ics.uci.edu/ml/machine-learning-databases/voting-records/house-votes-84.data
    Resolving archive.ics.uci.edu (archive.ics.uci.edu)... 128.195.10.252
    Connecting to archive.ics.uci.edu (archive.ics.uci.edu)|128.195.10.252|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 18171 (18K) [application/x-httpd-php]
    Saving to: ‘house-votes-84.data’

    house-votes-84.data 100%[===================>]  17.75K  --.-KB/s    in 0.07s

    2019-04-11 18:44:53 (271 KB/s) - ‘house-votes-84.data’ saved [18171/18171]

```python
# Imports
import pandas as pd
import numpy as np
from scipy import stats
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
# Load Data
df = pd.read_csv('house-votes-84.data',
                 header=None,
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])
print(df.shape)
df.head()
```

    (435, 17)

      party
      handicapped-infants
      water-project
      budget
      physician-fee-freeze
      el-salvador-aid
      religious-groups
      anti-satellite-ban
      aid-to-contras
      mx-missile
      immigration
      synfuels
      education
      right-to-sue
      crime
      duty-free
      south-africa

      0
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      y
      ?
      y
      y
      y
      n
      y

      1
      republican
      n
      y
      n
      y
      y
      y
      n
      n
      n
      n
      n
      y
      y
      y
      n
      ?

      2
      democrat
      ?
      y
      y
      ?
      y
      y
      n
      n
      n
      n
      y
      n
      y
      y
      n
      n

      3
      democrat
      n
      y
      y
      n
      ?
      y
      n
      n
      n
      n
      y
      n
      y
      n
      n
      y

      4
      democrat
      y
      y
      y
      n
      y
      y
      n
      n
      n
      n
      y
      ?
      y
      y
      y
      y

```python
# Replace '?' with np.NaN, 'n' with 0, and 'y' with 1
df = df.replace({'?':np.NaN, 'n':0, 'y':1})
df.head()
```

      party
      handicapped-infants
      water-project
      budget
      physician-fee-freeze
      el-salvador-aid
      religious-groups
      anti-satellite-ban
      aid-to-contras
      mx-missile
      immigration
      synfuels
      education
      right-to-sue
      crime
      duty-free
      south-africa

      0
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      1.0
      NaN
      1.0
      1.0
      1.0
      0.0
      1.0

      1
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      0.0
      NaN

      2
      democrat
      NaN
      1.0
      1.0
      NaN
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0

      3
      democrat
      0.0
      1.0
      1.0
      0.0
      NaN
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0

      4
      democrat
      1.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      NaN
      1.0
      1.0
      1.0
      1.0

```python
# How many abstentions? (NaNs)
df.isnull().sum()
```

    party                     0
    handicapped-infants      12
    water-project            48
    budget                   11
    physician-fee-freeze     11
    el-salvador-aid          15
    religious-groups         11
    anti-satellite-ban       14
    aid-to-contras           15
    mx-missile               22
    immigration               7
    synfuels                 21
    education                31
    right-to-sue             25
    crime                    17
    duty-free                28
    south-africa            104
    dtype: int64

```python
# Create Republicans Dataframe
rep = df[df.party == "republican"]
print(rep.shape)
rep.head()
```

    (168, 17)

      party
      handicapped-infants
      water-project
      budget
      physician-fee-freeze
      el-salvador-aid
      religious-groups
      anti-satellite-ban
      aid-to-contras
      mx-missile
      immigration
      synfuels
      education
      right-to-sue
      crime
      duty-free
      south-africa

      0
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      1.0
      NaN
      1.0
      1.0
      1.0
      0.0
      1.0

      1
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      0.0
      NaN

      7
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      NaN
      1.0

      8
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      0.0
      1.0

      10
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      NaN
      NaN
      1.0
      1.0
      0.0
      0.0

```python
# Create Democrats Dataframe
dem = df[df.party == "democrat"]
print(dem.shape)
dem.head()
```

    (267, 17)

      party
      handicapped-infants
      water-project
      budget
      physician-fee-freeze
      el-salvador-aid
      religious-groups
      anti-satellite-ban
      aid-to-contras
      mx-missile
      immigration
      synfuels
      education
      right-to-sue
      crime
      duty-free
      south-africa

      2
      democrat
      NaN
      1.0
      1.0
      NaN
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0

      3
      democrat
      0.0
      1.0
      1.0
      0.0
      NaN
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0

      4
      democrat
      1.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      NaN
      1.0
      1.0
      1.0
      1.0

      5
      democrat
      0.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      1.0

      6
      democrat
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      0.0
      NaN
      1.0
      1.0
      1.0

# 1 Sample T-Tests

In a 1-sample T-test we are testing the mean of one sample against a null hypothesis of our choosing.

The null hypothesis that we designate depends on how we have encoded our data and the kind of questions that we want to test.

If I have encoded votes as 0 for no and 1 for yes, I want to test Democratic support for an issue, and I use a null hypothesis of 0, then I am comparing Democrat voting support against a null hypothesis of no Democrat support at all for a given issue.

If I use a null hypothesis of .5 then I am comparing the democrat voting support against a null hypothesis of democrats being neither in favor or against a particular issue.

If I use a null hypothesis of 1 then I am comparing the democrat voting support against a null hypothesis of all democrats being favor of a particular issue.

Lets use the 0 and .5 null-hypotheses to test the significance of those particular claims. They're all valid questions to be asking, they're just posing a slightly different question --testing something different.

```python
# Lets test this out on the handicapped-infants issue since it's the first one.
# I am just going to omit NaN values from my tests.
# Null Hypothesis that Democratic support is 0.
stats.ttest_1samp(dem['handicapped-infants'], 0, nan_policy='omit')
```

    Ttest_1sampResult(statistic=19.825711173357988, pvalue=1.0391992873567661e-53)

Given the results of the above test I would REJECT the null hypothesis that there is no Democrat support for the handicapped-infants bill at the 95% significance level.

In Layman's terms It would be a FALSE statment to declare that there is no democratic support for this bill. That's something that you might hear a political pundit declare, but you'll notice that they don't report their alpha value or p-value when they make such claims, they just spew them. --*Tell us how you really feel!*

```python
# Null Hypothesis that Democrats neither support not oppose the issue
stats.ttest_1samp(dem['handicapped-infants'], .5, nan_policy='omit')
```

    Ttest_1sampResult(statistic=3.431373087696574, pvalue=0.000699612317167372)

```python
# Look at vote counts by party and by issue
# It's very easy to perform some Interocular Traumatic Tests (IOT) on this data
# https://www.r-bloggers.com/inter-ocular-trauma-test/
# We can eyeball the outcomes of some of these before we perform any T-tests.
# But which of the differences is statistically significant?
dem.apply(pd.Series.value_counts).T
```

    /usr/local/lib/python3.6/dist-packages/pandas/core/indexes/api.py:87: RuntimeWarning: '<' not supported between instances of 'str' and 'float', sort order is undefined for incomparable objects
      result = result.union(other)
    /usr/local/lib/python3.6/dist-packages/pandas/core/indexes/api.py:87: RuntimeWarning: '<' not supported between instances of 'float' and 'str', sort order is undefined for incomparable objects
      result = result.union(other)

      democrat
      1.0
      0.0

      party
      267.0
      NaN
      NaN

      handicapped-infants
      NaN
      156.0
      102.0

      water-project
      NaN
      120.0
      119.0

      budget
      NaN
      231.0
      29.0

      physician-fee-freeze
      NaN
      14.0
      245.0

      el-salvador-aid
      NaN
      55.0
      200.0

      religious-groups
      NaN
      123.0
      135.0

      anti-satellite-ban
      NaN
      200.0
      59.0

      aid-to-contras
      NaN
      218.0
      45.0

      mx-missile
      NaN
      188.0
      60.0

      immigration
      NaN
      124.0
      139.0

      synfuels
      NaN
      129.0
      126.0

      education
      NaN
      36.0
      213.0

      right-to-sue
      NaN
      73.0
      179.0

      crime
      NaN
      90.0
      167.0

      duty-free
      NaN
      160.0
      91.0

      south-africa
      NaN
      173.0
      12.0

As we look at the above table we see our findings corroborated in the raw voting counts of democrats. They are definitely not all against the handicapped-infants bill hence the very high p-value. We also see that democrats on this issue are not evenly split but that there is a significant margin of support for the handicapped-infants bill. However this margin is not as extreme for our second hypothesis as for our first, and hence our secont 1-sample t-test has a lower t statistic and a higher p-value although it is still significant.

```python
# Null hypothesis that there is no democratic support for the bill.
stats.ttest_1samp(dem['physician-fee-freeze'], 0, nan_policy='omit')
```

    Ttest_1sampResult(statistic=3.8396428405331067, pvalue=0.0001550891349511097)

We see that even though this issue has the most extreme Democrat opposition, we cannot conclude that there is no Democrat support for this issue. We will again REJECT the null hypothesis that the mean of democratic support is 0 --or that there is no Democrat support for this issue.

```python
# Null hypothesis that there is even support for the bill among democrats.
stats.ttest_1samp(dem['physician-fee-freeze'], .5, nan_policy='omit')
```

    Ttest_1sampResult(statistic=-31.67705343439813, pvalue=6.796885728494356e-91)

Here we are again testing the null hypothesis of Democrats being neither for nor against the issue, but being split in their voting. This time we see a strong negative t-statistic and low p-value. The negative sign on this t-statistic suggests that democratic support is much further to the left of .5 (our null hypothesis of even yes/no voting in the party) in other words, this t-statistic says that no only are democrats not split on the issue, they are highly against this policy. Again this is very clear to us as we consult the output of raw vote counts for the democrats.

As a final test and example of 1-sample T-tests lets try and get something that's not statistically significant. Lets test our split-support hypothesis on the "synfuels" policy.

```python
stats.ttest_1samp(dem['synfuels'], .5, nan_policy='omit')
```

    Ttest_1sampResult(statistic=0.18751153526716705, pvalue=0.8514092819865365)

Here we FAIL TO REJECT the null hypothesis that democrats voting is even on this issue because our t-statistic is small resulting in a p-value greater than .05. So we would fail to reject this null hypothesis at the 95% level. Remember that we never "accept" the null hypothesis, we only "fail to reject" it. We're not claiming that the null hypothesis is true, we're just stating that our test doesn't have enough statistical power to show otherwise.

### We could do the same thing with the Republicans dataframe, but the result would be extremely similar.

```python
# Look at republican voting patterns
rep.apply(pd.Series.value_counts).T
```

    /usr/local/lib/python3.6/dist-packages/pandas/core/indexes/api.py:87: RuntimeWarning: '<' not supported between instances of 'float' and 'str', sort order is undefined for incomparable objects
      result = result.union(other)

      republican
      0.0
      1.0

      party
      168.0
      NaN
      NaN

      handicapped-infants
      NaN
      134.0
      31.0

      water-project
      NaN
      73.0
      75.0

      budget
      NaN
      142.0
      22.0

      physician-fee-freeze
      NaN
      2.0
      163.0

      el-salvador-aid
      NaN
      8.0
      157.0

      religious-groups
      NaN
      17.0
      149.0

      anti-satellite-ban
      NaN
      123.0
      39.0

      aid-to-contras
      NaN
      133.0
      24.0

      mx-missile
      NaN
      146.0
      19.0

      immigration
      NaN
      73.0
      92.0

      synfuels
      NaN
      138.0
      21.0

      education
      NaN
      20.0
      135.0

      right-to-sue
      NaN
      22.0
      136.0

      crime
      NaN
      3.0
      158.0

      duty-free
      NaN
      142.0
      14.0

      south-africa
      NaN
      50.0
      96.0

## What if we didn't split our dataframe up by Republicans and Democrats, then what would that be testing?

The contents of the overall dataframe that we're working with determines the GENERALIZABILITY of our results. If we're running tests on Democrat voting behavior then our hypothesis tests can only make claims about Democrat voting, they say nothing about Republican support or opposition for an issue, we would have to run those tests on the Republican dataframe.

But then what do T-tests on the entire dataframe of both Republicans and Democrats say? They're testing the same thing but generalized to all congresspeople and the results are not specific to one party or another.

```python
# Look at congressional voting patterns
df.apply(pd.Series.value_counts).T
```

    /usr/local/lib/python3.6/dist-packages/pandas/core/indexes/api.py:87: RuntimeWarning: '<' not supported between instances of 'float' and 'str', sort order is undefined for incomparable objects
      result = result.union(other)

      democrat
      republican
      0.0
      1.0

      party
      267.0
      168.0
      NaN
      NaN

      handicapped-infants
      NaN
      NaN
      236.0
      187.0

      water-project
      NaN
      NaN
      192.0
      195.0

      budget
      NaN
      NaN
      171.0
      253.0

      physician-fee-freeze
      NaN
      NaN
      247.0
      177.0

      el-salvador-aid
      NaN
      NaN
      208.0
      212.0

      religious-groups
      NaN
      NaN
      152.0
      272.0

      anti-satellite-ban
      NaN
      NaN
      182.0
      239.0

      aid-to-contras
      NaN
      NaN
      178.0
      242.0

      mx-missile
      NaN
      NaN
      206.0
      207.0

      immigration
      NaN
      NaN
      212.0
      216.0

      synfuels
      NaN
      NaN
      264.0
      150.0

      education
      NaN
      NaN
      233.0
      171.0

      right-to-sue
      NaN
      NaN
      201.0
      209.0

      crime
      NaN
      NaN
      170.0
      248.0

      duty-free
      NaN
      NaN
      233.0
      174.0

      south-africa
      NaN
      NaN
      62.0
      269.0

We'll give one example of this, but it's the same as the T-tests above, just with the context being congress as a whole rather than a specific party.

```python
stats.ttest_1samp(df['mx-missile'], .5, nan_policy='omit')
```

    Ttest_1sampResult(statistic=0.049147318718299554, pvalue=0.9608257103188511)

We FAIL TO REJECT the null hypothesis that congressional support for the 'mx-missile' policy is split (even) among congresspeople.

# 2-sample T-tests

Two-sample T-tests are very similar to 1-sample t-tests, except that instead of providing a raw value as a null hypothesis, we will be comparing the mean of a second sample as the alternate hypothesis.

```python
stats.ttest_ind(dem['mx-missile'], rep['mx-missile'],nan_policy='omit')
```

    Ttest_indResult(statistic=16.437503268542994, pvalue=5.03079265310811e-47)

## Don't make this mistake!

Notice that a 2-sample T-test does not give the same results as a 1-sample t-test where the null hypothesis is the mean of a second sample. The test below is representative of a t-test comparing the mean of the first sample to a null-hypothesis value of .115, but it is not representative of comparing the mean of the sample to the mean of a second sample. This is because passing in the mean as a single value does not account for the variance around the mean of the second sample, so the 1-sample t-test against the mean value of another sample has a much higher t-statistic and significance than a test where you pass in the full data of both samples.

You can avoid this mistake by making sure that you provide a raw null-hypothesis value (like 0, .5, 1) as the null-hypothesis value when performing a 1-sample t-test, and only two separate samples in their entirety to the function when performing a two-sample t-test.

```python
rep_mx_missile_mean = rep['mx-missile'].mean()
print(rep_mx_missile_mean)
```

    0.11515151515151516

```python
stats.ttest_1samp(dem['mx-missile'], rep_mx_missile_mean, nan_policy='omit')
```

    Ttest_1sampResult(statistic=23.593795768600376, pvalue=3.200954476509414e-65)

Also notice that the order in which you pass the two samples to the 2-sample test will reverse the direction of the t-statistic.

A t-statistic with a positive sign indicates that the first sample mean is higher than the second sample mean, and the significance level indicates whether or not the means are different at the 95% confidence level.

```python
print(dem['mx-missile'].mean())
print(rep['mx-missile'].mean())

stats.ttest_ind(dem['mx-missile'], rep['mx-missile'], nan_policy='omit')
```

    0.7580645161290323
    0.11515151515151516

    Ttest_indResult(statistic=16.437503268542994, pvalue=5.03079265310811e-47)

Here we have reversed the order of which sample we pass in first to demonstrate how `ttest_ind` might result in a negative t-statistic.

```python
print(dem['mx-missile'].mean())
print(rep['mx-missile'].mean())

stats.ttest_ind(rep['mx-missile'], dem['mx-missile'], nan_policy='omit')
```

    0.7580645161290323
    0.11515151515151516

    Ttest_indResult(statistic=-16.437503268542994, pvalue=5.03079265310811e-47)

Because of this, in order to ensure consistency of the signs of the t-statistic during 2-sample tests, I suggest sticking with passing in one party as the first argument and the other party as the second argument and sticking with that pattern throughout your testing so that you don't confuse yourself.

## Two-sample T-tests for Democrat Support, Republican Support, and no singnificant difference in support between parties:

## Significant Democrat Support

The outcome of this tests indicates that we should reject the null hypothesis that the mean of Democrat votes is equal to the mean of Republican votes for this issue. We would conclude due to the positive t-statistic here that Democrat support is significantly higher than republican support for this issue.

```python
stats.ttest_ind(dem['synfuels'], rep['synfuels'], nan_policy='omit')
```

    0.5058823529411764
    0.1320754716981132

    Ttest_indResult(statistic=8.293603989407588, pvalue=1.5759322301054064e-15)

## Significant Republican Support

The outcome of this tests indicates that we should reject the null hypothesis that the mean of democrat votes is equal to the mean of republican votes for this issue. We would conclude due to the positive t-statistic here that Republican support is significantly higher than Democrat support for this issue.

```python
print(dem['south-africa'].mean())
print(rep['south-africa'].mean())

stats.ttest_ind(rep['south-africa'], dem['south-africa'], nan_policy='omit')
```

    0.9351351351351351
    0.6575342465753424

    Ttest_indResult(statistic=-6.849454815841208, pvalue=3.652674361672226e-11)

## No Significant Difference in Support

Due to the insignificant p-value we would FAIL TO REJECT the null hypothesis that the means of Democrat support and Republican support are equal. This means that regardless of the sign on the t-statistic, we cannot make statistically significant claims about the difference between Republican and Democrat support of this issue.

```python
print(dem['water-project'].mean())
print(rep['water-project'].mean())

stats.ttest_ind(dem['water-project'], rep['water-project'], nan_policy='omit')
```

    0.502092050209205
    0.5067567567567568

    Ttest_indResult(statistic=-0.08896538137868286, pvalue=0.9291556823993485)

# $\chi^2$ Hypothesis Tests

Chi-square hypothesis tests can only be performed on categorical data to test independence (lack of association) between two variables.

You can tabulate the different cross-sections of your categorical data by creating a "contingency table" also known as a "cross-tab". If we see substantial differences between categories in the cross-tab, this might indicate a dependence between variables (possibly indicating some level of correlation although correlation and dependence are not perfectly synonymous),however, we must perform a chi-square test in order to test this hypothesis.

Correlation implies dependence (association), but dependence does not necessarily imply correlation. (Although it is correlated with correlation - you see what I did there?)

## First, a Numpy vs Scipy implementation

Lets demonstrate a chi-squared test on the "Adult" dataset (1994 census data) from UCI.

Lets compare gender and binned working hours per week (both are categorical).

<http://archive.ics.uci.edu/ml/datasets/adult>

```python
df = pd.read_csv('https://raw.githubusercontent.com/ryanleeallred/datasets/master/adult.csv')
print(df.shape)
df.head()
```

    (32561, 15)

      age
      workclass
      fnlwgt
      education
      education-num
      marital-status
      occupation
      relationship
      race
      sex
      capital-gain
      capital-loss
      hours-per-week
      country
      salary

      0
      39
      State-gov
      77516
      Bachelors
      13
      Never-married
      Adm-clerical
      Not-in-family
      White
      Male
      2174
      0
      40
      United-States
      <=50K

      1
      50
      Self-emp-not-inc
      83311
      Bachelors
      13
      Married-civ-spouse
      Exec-managerial
      Husband
      White
      Male
      0
      0
      13
      United-States
      <=50K

      2
      38
      Private
      215646
      HS-grad
      9
      Divorced
      Handlers-cleaners
      Not-in-family
      White
      Male
      0
      0
      40
      United-States
      <=50K

      3
      53
      Private
      234721
      11th
      7
      Married-civ-spouse
      Handlers-cleaners
      Husband
      Black
      Male
      0
      0
      40
      United-States
      <=50K

      4
      28
      Private
      338409
      Bachelors
      13
      Married-civ-spouse
      Prof-specialty
      Wife
      Black
      Female
      0
      0
      40
      Cuba
      <=50K

Notice that for chi-squared tests, I will not have to categorically encode my data since I will perform my calculations on the contingency table (cross-tab) and not on the raw dataframe.

```python
def process_hours(df):
  cut_points = [0,9,19,29,39,49,1000]
  label_names = ["0-9","10-19","20-29","30-39","40-49","50+"]
  df["hours_per_week_categories"] = pd.cut(df["hours-per-week"],
                                           cut_points,labels=label_names)
  return df

data = process_hours(df)
workhour_by_sex = data[['sex', 'hours_per_week_categories']]
workhour_by_sex.head()
```

      sex
      hours_per_week_categories

      0
      Male
      40-49

      1
      Male
      10-19

      2
      Male
      40-49

      3
      Male
      40-49

      4
      Female
      40-49

```python
workhour_by_sex['sex'].value_counts()
```

     Male      21790
     Female    10771
    Name: sex, dtype: int64

```python
workhour_by_sex['hours_per_week_categories'].value_counts()
```

    40-49    18336
    50+       6462
    30-39     3667
    20-29     2392
    10-19     1246
    0-9        458
    Name: hours_per_week_categories, dtype: int64

Before we calculate our contingency table, lets make very clear what our null and alternative hypotheses are in this situation.

$H_{0}$ : There is *no* statistically significant relationship between gender and working hours per week.

$H_{a}$ : There *is* a statistically significant relationship between gender and working hours per week.

```python
# Calculate our contingency table with margins
contingency_table = pd.crosstab(
    workhour_by_sex['sex'],
    workhour_by_sex['hours_per_week_categories'],
    margins = True)

contingency_table
```

      hours_per_week_categories
      0-9
      10-19
      20-29
      30-39
      40-49
      50+
      All

      sex

      Female
      235
      671
      1287
      1914
      5636
      1028
      10771

      Male
      223
      575
      1105
      1753
      12700
      5434
      21790

      All
      458
      1246
      2392
      3667
      18336
      6462
      32561

Using the contingency table with margins included will make our from-scratch implementation a little bit easier.

Our code would be more reusable if we calculated totals as we went instead of pulling them from the table, but I wanted to be really clear about what they represented, so we're going to grab the row and column totals from the margins of the contingency table directly.

### Visualizing preferences with a stacked bar-chart.

```python
#Assigns the frequency values
malecount = contingency_table.iloc[0][0:6].values
femalecount = contingency_table.iloc[1][0:6].values

#Plots the bar chart
fig = plt.figure(figsize=(10, 5))
sns.set(font_scale=1.8)
categories = ["0-9","10-19","20-29","30-39","40-49","50+"]
p1 = plt.bar(categories, malecount, 0.55, color='#d62728')
p2 = plt.bar(categories, femalecount, 0.55, bottom=malecount)
plt.legend((p2[0], p1[0]), ('Male', 'Female'))
plt.xlabel('Hours per Week Worked')
plt.ylabel('Count')
plt.show()
```

It's harder to eyeball if these variables might be dependent. I would look at the ratios between 30-39, 40-49 and 50+. As we do this we see that males working 40-49 hours per week experiences something like 5x jump from the amount working 30-39 whereas for women it's something like a 2.5x jump. Similarly for men we see a decrease to about 1/3 the size as we move from >40 to the >50 category, but for women the final category is about 1/5 the size. This suggests to me that gender and working hours are not independent. Hopefully this passes the smell test for you as it matches your intuition that has been trained from a lifetime of training data.

### Expected Value Calculation
\begin{align}
expected_{i,j} =\frac{(row_{i} \text{total})(column_{j} \text{total}) }{(\text{total observations})}
\end{align}

```python
# Get Row Sums
row_sums = contingency_table.iloc[0:2,6].values
print(row_sums)
```

    [10771 21790]

```python
# Get Column Sums
col_sums = contingency_table.iloc[2,0:6].values
print(col_sums)
```

    [  458  1246  2392  3667 18336  6462]

```python
# Calculate Expected Values for each cell
total = contingency_table.loc['All', 'All']
print("Total number of observations:", total)

expected = []
for i in range(len(row_sums)):
  expected_row = []
  for column in col_sums:
    expected_val = column*row_sums[i]/total
    expected_row.append(expected_val)
  expected.append(expected_row)

print(np.array(expected))
```

    Total number of observations: 32561
    [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

## Chi-Squared Statistic with Numpy

\begin{align}
\chi^2 = \sum \frac{(observed_{i}-expected_{i})^2}{(expected_{i})}
\end{align}

For the $observed$ values we will just use a version of our contingency table without the margins as a numpy array. In this way, if our observed values array and our expected values array are the same shape, then we can subtract them and divide them directly which makes the calculations a lot cleaner. No for loops!

```python
# Get contingency table without margins
contingency = pd.crosstab(workhour_by_sex['sex'],
                          workhour_by_sex['hours_per_week_categories'])
contingency = contingency.values

print(contingency.shape)
print(contingency)
```

    (2, 6)
    [[  235   671  1287  1914  5636  1028]
     [  223   575  1105  1753 12700  5434]]

```python
chi_squared = ((contingency - expected)**2/(expected)).sum()
print(f"Chi-Squared: {chi_squared}")
```

    Chi-Squared: 2287.190943926107

### Degrees of Freedom

\begin{align}
DoF = (\text{Number of Rows} -1)\times(\text{Number of Columns}-1)
\end{align}

```python
dof = (len(row_sums)-1)*(len(col_sums)-1)
print(f"Degrees of Freedom: {dof}")
```

    Degrees of Freedom: 5

```python
# Calculate the p-value from the chi_squared and dof
p_value = stats.chi2.sf(chi_squared, dof)
print(f"P-value: {p_value}")
```

    P-value: 5.759792112623893e-53

```python
chi_squared, p_value, dof, expected = stats.chi2_contingency(contingency)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}")
print("Expected: \n", np.array(expected))
```

    Chi-Squared: 2287.190943926107
    P-value: 0.0
    Degrees of Freedom: 5
    Expected:
     [[  151.50388502   412.16995793   791.26046497  1213.02346365
       6065.44811277  2137.59411566]
     [  306.49611498   833.83004207  1600.73953503  2453.97653635
      12270.55188723  4324.40588434]]

# Can we perform a Chi2 test on our congressional voting data?

Is it categorical? Then yes. Lets do it!

Are political party and voting behavior on the "budget" independent? Lets test it!

```python
# Load the data again to be safe:
df = pd.read_csv('house-votes-84.data',
                 header=None,
                 names=['party','handicapped-infants','water-project',
                          'budget','physician-fee-freeze', 'el-salvador-aid',
                          'religious-groups','anti-satellite-ban',
                          'aid-to-contras','mx-missile','immigration',
                          'synfuels', 'education', 'right-to-sue','crime','duty-free',
                          'south-africa'])
df = df.replace({'?':np.NaN, 'n':0, 'y':1})
print(df.shape)
df.head()
```

    (435, 17)

      party
      handicapped-infants
      water-project
      budget
      physician-fee-freeze
      el-salvador-aid
      religious-groups
      anti-satellite-ban
      aid-to-contras
      mx-missile
      immigration
      synfuels
      education
      right-to-sue
      crime
      duty-free
      south-africa

      0
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      1.0
      NaN
      1.0
      1.0
      1.0
      0.0
      1.0

      1
      republican
      0.0
      1.0
      0.0
      1.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      0.0
      1.0
      1.0
      1.0
      0.0
      NaN

      2
      democrat
      NaN
      1.0
      1.0
      NaN
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0

      3
      democrat
      0.0
      1.0
      1.0
      0.0
      NaN
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      0.0
      1.0
      0.0
      0.0
      1.0

      4
      democrat
      1.0
      1.0
      1.0
      0.0
      1.0
      1.0
      0.0
      0.0
      0.0
      0.0
      1.0
      NaN
      1.0
      1.0
      1.0
      1.0

```python
contingency_table =  pd.crosstab(df['party'], df['budget'])
contingency_table
```

      budget
      0.0
      1.0

      party

      democrat
      29
      231

      republican
      142
      22

```python
chi_squared, p_value, dof, expected = stats.chi2_contingency(contingency_table)

print(f"Chi-Squared: {chi_squared}")
print(f"P-value: {p_value}")
print(f"Degrees of Freedom: {dof}")
print("Expected: \n", np.array(expected))
```

    Chi-Squared: 234.65408769323486
    P-value: 5.759792112623893e-53
    Degrees of Freedom: 1
    Expected:
     [[104.85849057 155.14150943]
     [ 66.14150943  97.85849057]]

```python
stats.chi2.sf?
```

```python
# Calculate the p-value from the chi_squared and dof
p_value = stats.chi2.sf(chi_squared, dof)
print(f"P-value: {p_value}")
```

    P-value: 5.759792112623893e-53

We REJECT the null hypothesis that political party and voting on the budget are independent. They must be associated. Which hopefully also passes the smell test.


[From Briggs, Cochran *Calculus: Early Transcendentals*.]

```python

```

```python

```

# 3

### Constant Rule
If $c$ is a real number, then $\frac{d}{dx}(c) = 0$

### Power Rule
If $n$ is a postive integer, then $\frac{d}{dx}(x^{n}) = nx^{n-1}$

##### Definitions
$f(x) = x^{n} \implies f(x) - f(a) = x^{n} - a^{n}$

$x^{n} - a^{n} = (x - a)(x^{n-1} + x^{n-2}a + ... + xa^{n-2} + a^{n-1})$

$\lim_{x \rightarrow a} (x^{n-1} + x^{n-2}a + ... + xa^{n-2} + a^{n-1}) = a^{n-1} + a^{n-2} \cdot a + ... + a \cdot a^{n-2} + a^{n-1} = na^{n-1}$

##### Definition Examples
$x^{4} - a^{4} = (x - a)(x^{3} + x^{2}a + xa^{2} + a^{3})$

$\lim_{x \rightarrow a} (x^{3} + x^{2}a + xa^{2} + a^{3}) = a^{3} + a^{2} \cdot a + a \cdot a^{2} = a^{3} + a^{3} + a^{3} = 3n^{3}$

##### Proof
$$
\begin{align}
f'(a) &= \lim_{x \rightarrow a} \frac{f(x) - f(a)}{x - a} \\
&= \lim_{x \rightarrow a} \frac{x^{n} - a^{n}}{x - a} \\
&= \lim_{x \rightarrow a} \frac{(x - a)(x^{n-1} + x^{n-2}a + ... + xa^{n-2} + a^{n-1})}{x - a} \\
&= \lim_{x \rightarrow a} (x^{n-1} + x^{n-2}a + ... + xa^{n-2} + a^{n-1}) \\
&= a^{n-1} + a^{n-2} \cdot a + ... + a \cdot a^{n-2} + a^{n-1} \\
&= na^{a-1}
\end{align}
$$

### The derivative of $e^{x}$
The function $f(x) = e^{x}$ is differentiable for all real numbers $x$ and
$$\frac{d}{dx}(e^{x}) = e^{x}$$

##### Proof
$$
\begin{align}
\frac{d}{dx}(e^{x}) &= \lim_{h \rightarrow 0} \frac{e^{x + h} - e^{x}}{h} \\
&= \lim_{h \rightarrow 0} \frac{e^{x} \cdot e^{h} - e^{x}}{h} \\
&= \lim_{h \rightarrow 0} \frac{e^{x}(e^{h} - 1)}{h} \\
&= e^{x} \cdot \lim_{h \rightarrow 0} \frac{e^{h} - 1}{h} \\
&= e^{x} \cdot 1 = e^{x}
\end{align}
$$

### Product Rule
If $f$ and $g$ are differentiable at $x$, then
$$\frac{d}{dx}[f(x)g(x)] = f'(x)g(x) + f(x)g'(x)$$

##### Definitions
$Z: -f(x)g(x+h) + f(x)g(x + h) = 0$

$\lim_{h \rightarrow 0} f(x) = f(x)$

$\lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{h} = f'(x)$

$\lim_{h \rightarrow 0} g(x + h) = g(x)$

$\lim_{h \rightarrow 0} \frac{g(x + h) - g(x)}{h} = g'(x)$

##### Proof
$$
\begin{align}
\frac{d}{dx}[f(x)g(x)] &= \lim_{h \rightarrow 0} \frac{f(x + h)g(x + h) - f(x)g(x)}{h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x + h) - f(x)g(x) + Z}{h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x + h) - f(x)g(x) + [-f(x)g(x+h) + f(x)g(x + h)]}{h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x + h) - f(x)g(x+h) + f(x)g(x + h) - f(x)g(x)}{h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x + h) - f(x)g(x+h)}{h} + \lim_{h \rightarrow 0} \frac{f(x)g(x + h) - f(x)g(x)}{h} \\
&= \lim_{h \rightarrow 0} \bigg[\frac{f(x + h) - f(x)}{h} \cdot g(x + h)\bigg] + \lim_{h \rightarrow 0} \bigg[f(x) \cdot \frac{g(x + h) - g(x)}{h} \bigg] \\
&= f'(x) \cdot g(x) + f(x) \cdot g'(x)
\end{align}
$$

### Quotient Rule
If $f$ and $q$ are differentiable at $x$, then the derivative of $f/g$ at $x$ exists provided $g(x) \neq 0$ and

$$\frac{d}{dx}\bigg[\frac{f(x)}{g(x)}\bigg] = \frac{g(x)f'(x) - f(x)g'(x)}{[g(x)]^{2}}$$

##### Definitions
$q(x) = \frac{f(x)}{g(x)}$

$f(x) = g(x)q(x)$

##### Calculation
If $\frac{d}{dx}f/g$ exists,
$$
\begin{align}
f'(x) &= g'(x)q(x) + g(x)q'(x) \\
q'(x) &= \frac{f'(x) - g'(x)q(x)}{g(x)} \\
q'(x) &= \frac{f'(x) - g'(x)\frac{f(x)}{g(x)}}{g(x)} \\
&= \frac{g(x) \Big(f'(x) - g'(x)\frac{f(x)}{g(x)}\Big)}{g(x) \cdot g(x)} \\
&= \frac{g(x)f'(x) - f(x)g'(x)}{[g(x)]^{2}}
\end{align}
$$

##### Definitions
Let $F = f/g$ be the quotient of two functions that are differentiable at $x$

$Z: -f(x)g(x) + f(x)g(x) = 0$

$\lim_{h \rightarrow 0} f(x) = f(x)$

$\lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{h} = f'(x)$

$\lim_{h \rightarrow 0} g(x) = g(x)$

$\lim_{h \rightarrow 0} \frac{g(x) - g(x + h)}{h} = -g'(x)$

$\lim_{h \rightarrow 0} \frac{1}{g(x)g(x + h)} = \frac{1}{[g(x)]^{2}}$

##### Proof
$$
\begin{align}
\frac{d}{dx}[f(x)g(x)] &= \lim_{h \rightarrow 0} \frac{\frac{f(x + h)}{g(x + h)} - \frac{f(x)}{g(x)}}{h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)}{g(x + h) \cdot h} - \frac{f(x)}{g(x) \cdot h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x) - f(x)g(x + h)}{g(x)g(x + h) \cdot h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x) - f(x)g(x + h) + Z}{g(x)g(x + h) \cdot h} = \lim_{h \rightarrow 0} \frac{f(x + h)g(x) - f(x)g(x + h) + [-f(x)g(x) + f(x)g(x)]}{g(x)g(x + h) \cdot h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x) - f(x)g(x) + g(x)f(x) - g(x+h)f(x)}{g(x)g(x + h) \cdot h} \\
&= \lim_{h \rightarrow 0} \frac{f(x + h)g(x) - f(x)g(x)}{g(x)g(x + h) \cdot h} + \frac{g(x)f(x) - g(x + h)f(x)}{g(x)g(x + h) \cdot h} \\
&= \bigg[\lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{g(x)g(x + h) \cdot h} \cdot g(x) \bigg] + \bigg[\lim_{h \rightarrow 0} \frac{g(x) - g(x + h)}{g(x)g(x + h) \cdot h} \cdot f(x)\bigg] \\
&= \bigg[\lim_{h \rightarrow 0} \frac{f(x + h) - f(x)}{h} \cdot g(x) \cdot \frac{1}{g(x)g(x + h)} \bigg] + \bigg[\lim_{h \rightarrow 0} \frac{g(x) - g(x + h)}{h} \cdot f(x)  \cdot \frac{1}{g(x)g(x + h)} \bigg] \\
&= \bigg[f'(x) \cdot g(x) \cdot \frac{1}{[g(x)]^{2}} \bigg] + \bigg[-g'(x) \cdot f(x)  \cdot \frac{1}{[g(x)]^{2}} \bigg] \\
&= \frac{f'(x)g(x) - g'(x)f(x)}{[g(x)]^{2}}
\end{align}
$$

### The derivative of $e^{kx}$
For real numbers $k$,
$$\frac{d}{dx}(e^{kx}) = ke^{kx}$$

##### Example
$$
\begin{align}
\frac{d}{dx}(e^{2x}) &= \frac{d}{dx}(e^{x} \cdot e^{x}) \\
&= \frac{d}{dx}(e^{x}) \cdot e^{x} + e^{x} \cdot \frac{d}{dx}(e^{x}) \\
&= e^{x} \cdot e^{x} + e^{x} \cdot e^{x} = 2e^{2x}
\end{align}
$$

##### Proof by induction for positive integers $k$
Base case ($k = 1$)
$$
\begin{align}
\frac{d}{dx}e^{kx} &= \frac{d}{dx}e^{1 \cdot x} \\
&= \frac{d}{dx}e^{x} \\
&= e^{x}
\end{align}
$$

Inductive step (inductive hypothesis: assume true for $k = n$)
$$
\begin{align}
\frac{d}{dx}e^{nx} = ne^{nx}
\end{align}
$$

Prove true for $k = n + 1$: $\frac{d}{dx}e^{(n+1)x} = (n+1)e^{(n+1)x}$

$$
\begin{align}
\frac{d}{dx}e^{(n+1)x} &= \frac{d}{dx}e^{nx + x} \\
&= \frac{d}{dx}(e^{nx} \cdot e^{x}) \\
&= ne^{nx} \cdot e^{x} + e^{nx} \cdot e^{x} \\
&= e^{x}(ne^{nx} + e^{nx}) \\
&= e^{x} \cdot e^{nx}(n + 1) \\
&= e^{nx+x}(n+1) \\
&= e^{(n+1)x}(n+1) \\
&= (n+1)e^{(n+1)x}
\end{align}
$$

##### Proof by induction for negative integers $k$
Base case ($k = -1$)
$$
\begin{align}
\frac{d}{dx}e^{kx} &= \frac{d}{dx}e^{-1 \cdot x} \\
&= \frac{d}{dx}e^{-x} = \frac{d}{dx}\bigg(\frac{1}{e^{x}}\bigg) \\
&= \frac{o \cdot e^{x} - 1 \cdot e^{x}}{e^{x} \cdot e^{x}} = \frac{-e^{x}}{[e^{x}]^{2}} = \frac{-1}{e^{x}} = -e^{-x}
\end{align}
$$

Inductive step (inductive hypothesis: assume true for $k = -n$)
$$
\begin{align}
\frac{d}{dx}e^{-nx} = -ne^{-nx}
\end{align}
$$

Prove true for $k = -n - 1$: $\frac{d}{dx}e^{(-n - 1)x} = (-n - 1)e^{(-n - 1)x}$

$$
\begin{align}
\frac{d}{dx}e^{(-n - 1)x} &= \frac{d}{dx}e^{-nx - x} \\
&= \frac{d}{dx}(e^{-nx} \cdot e^{-x}) \\
&= \frac{d}{dx}\bigg(\frac{e^{-nx}}{e^{x}}\bigg) \\
&= \frac{-ne^{-nx} \cdot e^{x} - e^{-nx} \cdot e^{x}}{e^{x} \cdot e^{x}} = \frac{e^{x}(-ne^{-nx} - e^{-nx})}{e^{x} \cdot e^{x}} = \frac{-ne^{-nx} - e^{-nx}}{e^{x}} = \frac{e^{-nx}(-n - 1)}{e^{x}} \\
&= e^{-nx} \cdot e^{-x} (-n - 1) = e^{-nx-x}(-n - 1) = e^{(-n - 1)x}(-n - 1) = (-n - 1)e^{(-n - 1)x}
\end{align}
$$


```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams.update({'font.size': 20})
```

# Numerical solution to the KPZ Equation

We will use the standard mapping between the height in the KPZ equation with the free energy of a directed polymer at high temperature embedded in a 1+1 random potential.

We will consider a polymer with size $L = 2n$ bonds in a 1+1 random potential drawn from a gaussian distribution $\mathcal{N}(0,1)$. We need to consider all polymers that starts at (0,0) and end at (n,n), the polymer is only allowed to grow onto positive neighboring sites, so the "diagonal" d = x + y will be increased by one.

A polymer visiting a set $P$ of sites has an energy:
\begin{equation}
E_V (P) = \sum_{(x,y) \in P} \ V[x,y]
\end{equation}

The boltzmann weight is given by:
\begin{equation}
w_V (P) = e^{\frac{-E_V (P)}{T}}
\end{equation}

The partition function Z(V) is given by:
\begin{equation}
Z(V) = \sum_P w_V (P)
\end{equation}

Since the polymer can only increas on the "diagonal", we have a recurssion relation that will be used on our class for the partition function:
\begin{equation}
Z[x,y] = (Z[x-1,y] + Z[x,y-1]) e^{\frac{-V[x,y]}{T}}
\end{equation}

Where Z[x,y] is the partition function starting at (0,0) and ending on (x,y). So our final partition function will be given by:
\begin{equation}
Z(V) = Z[n,n]
\end{equation}

```python
class KPZ:
    def __init__(self,n,T):
        """
        Initialization of our class.
        Parameters:
        - n : Size of the polymer.
        - T : Temperature.
        """

        self.n = n + 1
        self.matrix = np.random.normal(size=(self.n,self.n))
        self.Z_matrix = np.zeros((self.n,self.n))
        self.T = T

    def Z(self,x,y):
        """
        Return the partition function for a polymer at point (x,y) at a heat bath T.
        """

        if x == 0 and y == 0:
            self.Z_matrix[x,y] = np.exp(-self.matrix[x,y])
        elif x == 0:
            self.Z_matrix[x,y] = (self.Z_matrix[x,y-1])*np.exp(-self.matrix[x,y])
        elif y == 0:
            self.Z_matrix[x,y] = (self.Z_matrix[x-1,y])*np.exp(-self.matrix[x,y])
        else:
            self.Z_matrix[x,y] = (self.Z_matrix[x-1,y] + self.Z_matrix[x,y-1])*np.exp(-self.matrix[x,y])

    def run(self):
        """
        Run the simulation and return the partition function Z(n,n).
        """
        for i in range(self.n):
            for j in range(self.n):
                self.Z(i,j)
        return self.Z_matrix[self.n-1,self.n-1]
```

### Simulation
Using the mapping between free energy an the height on the KPZ equation (H), we have:

\begin{equation}
H = log\left(\frac{Z(L,L)}{\bar{Z}}\right)
\end{equation}

\begin{equation}
t = \frac{2N}{T^4}
\end{equation}

So we can calculate $Z(V)$ doing various simulations for our disordered potential and then calculate the average for all samples $\bar{Z}$ and $P(H,t)$ is the histogram of values of H.

In this simulation we will have a fixed time ($t$) given by the above equation and verify the distribution $P(H,t)$.

```python
def direct_sampling(N,temperature,samples= 1000):
    H_ = np.zeros(samples)
    for i in range(samples):
        Sol = KPZ(N,T = temperature)
        H_[i] = Sol.run()
    t = 2*N/float(temperature**4.)
    return t, H_
```

```python
t, H = direct_sampling(N= 64,temperature= 6.72)
```

```python
H__ = H/np.mean(H)
```

```python
aux = np.log(H__)
```

```python
plt.hist(aux)
plt.show()
```

```python

```


# Region Estimation Using Active Learning GPs

This notebook demonstrates how CSAF can be used in a machine learning based analysis. Specifically, an active learning framework (modAL) uses CSAF as an oracle to sample the state space in a way that recovers a region as a Gaussian Process.

**Task**: Estimate the region of initial system states where a condition holds true by sampling the state space. In this case, discover what initial conditions yield a successful ground collision avoidance maneuver for the F-16 aircraft.

**Assumptions**:
* Naive sampling policies like uniform sampling is too inefficient for the dimensionality of the state space.
* The recovered region has a sufficiently low dimensional structure.
* The recovered region’s boundary is a Gaussian process f(x) with a known covariance function k(x, x’).

\begin{equation}
f(\mathbf x) \sim \mathcal G \mathcal P (m(\mathbf x), k(\mathbf x, \mathbf x’)), \quad \sum_{i=1}^{m}\sum_{j=i}^{m} \alpha_i \alpha_j k(\mathcal x_i, \mathcal x_j) \ge 0.
\end{equation}

```python
import sys

import numpy as np
import matplotlib.pyplot as plt

import tqdm

# CSAF Imports
import csaf.config as cconf
import csaf.system as csys
import run_parallel as rp  # for now, batch simulations is not in CSAF package

# ML Imports
from modAL.models import ActiveLearner
from sklearn.gaussian_process import GaussianProcessRegressor, GaussianProcessClassifier
from sklearn.gaussian_process.kernels import RBF
```

## CSAF System Creation

```python
# create a csaf configuration out of toml
my_conf = cconf.SystemConfig.from_toml("/csaf-system//f16_shield_config.toml")
```

```python
from IPython.display import Image

import pathlib

plot_fname = f"pub-sub-{my_conf.name}-plot.png"

# plot configuration pub/sub diagram as a file -- proj specicies a dot executbale and -Gdpi is a valid dot
# argument to change the image resolution
my_conf.plot_config(fname=pathlib.Path(plot_fname).resolve(), prog=["dot", "-Gdpi=400"])

# display written file to notebook
Image(plot_fname, height=600)
```

## Analysis Parameters

```python
## User Set Parameters

# bounds to sample
bounds = [(200, 1000),
          (np.deg2rad(2.1215), np.deg2rad(2.1215)),
          (0.0, 0.0),
          ((np.pi/2)*0.5, (np.pi/2)*0.5),
          (-np.pi, np.pi),
          (-np.pi/4, np.pi/4 ),
         (0.0, 0.0),
         (-0.5, 0.5),
         (0.0, 0.0),
         (0.0, 0.0),
         (0.0, 0.0),
         (500, 8000),
         (9, 9)]

# length scales for RBF kernel used
rbf_scales = (100, 1, 1, 1, 30, 1, 1, 1, 1, 1, 1, 1000, 1)

# initial sampling count
n_initial_sampling = 13 * 2
```

```python
## Extracted Parameters

# name of system configuration
run_name = my_conf.name

# center of plane to plot contours
x_center = np.array([b[1]/2 + b[0]/2 for b in bounds])

# name of state variables
names = my_conf.get_msg_setting("plant", "states", "msg").fields_no_header
```

## Helper Functions

```python
def gen_random_state(bounds):
    """create a uniformly distributed initial condition"""
    sample = np.random.rand(len(bounds))
    ranges = np.array([b[1] - b[0] for b in bounds])
    offset = np.array([- b[0] for b in bounds])
    return sample * ranges - offset

def gpr_scalar_field(cx_train, cy_train, x_center, bounds, indices, rbf_scales = rbf_scales):
    """produce a matrix of GPR scalar field values against the plane centered at x_center and spanning the
    natural basis vector indexed at indices
    """
    x_min, x_max = bounds[indices[0]]
    y_min, y_max = bounds[indices[1]]
    xx, yy = np.meshgrid(np.linspace(x_min, x_max, 50),
                             np.linspace(y_min, y_max, 50))

    kernel = 100*RBF(rbf_scales)
    clf = GaussianProcessClassifier(kernel=kernel)
    clf.fit(cx_train, cy_train)

    score = clf.score(cx_train, cy_train)
    dec = np.zeros((len(xx.ravel()), 13))
    for idx in range(0, 13):
        dec[:, idx] = x_center[idx]
    dec[:, indices[0]] = xx.ravel()
    dec[:, indices[1]] = yy.ravel()
    if hasattr(clf, "decision_function"):
            Z = clf.decision_function(dec)
    else:
            Z = clf.predict_proba(dec)[:, 1]
    Z = Z.reshape(xx.shape)
    return xx, yy, Z

## PLOTTING CODE

def plot_sample_plane(ax, names, xx, yy, zz, x_train=None, y_train=None):
    """given the output of gpr_scalar_field, create a contour plot"""
    cm = plt.cm.RdBu
    ax.contourf(xx, yy, zz, cmap=cm, alpha=.8)
    ax.set_xlabel(f"{names[0]}")
    ax.set_ylabel(f"{names[1]}")

def gpr_plot_field(planes, names, bounds, x_all, y_all, x_center):
    """given an array of planes defined by indices, create a multi-axis plot of them"""
    n_plots = len(planes)
    n_rows = int(np.floor(n_plots**0.5))
    n_cols =int(np.ceil(n_plots / n_rows))
    fig, ax = plt.subplots(figsize=(n_cols * 5, n_rows * 5), ncols=n_cols, nrows=n_rows)
    for idx, pl in enumerate(planes):
        ridx = idx % n_cols
        cidx = idx // n_cols
        plot_sample_plane(ax[ridx][cidx], (names[pl[0]], names[pl[1]]),
                          *gpr_scalar_field(x_all, y_all, x_center, bounds, indices=pl))
    plt.tight_layout()
    return fig, ax
```

## Create the Active Learner Oracle

```python
def ground_collision_condition(cname, outs):
        """ground collision premature termnation condition"""
        return cname == "plant" and outs["states"][11] <= 0.0

def validate_samples(x_samp, config, tspan=(0, 20.0)):
    """given an array of plant initial states,
    run batches of simulations to see if they meet the termination criteria"""
    # assign samples to the system plant
    init_states = [{"plant": xi} for xi in x_samp]
    n = len(init_states)
    # send jobs to the CSAF simulation workgroup
    res = rp.run_workgroup(n, config, init_states, tspan,
                            terminating_conditions=ground_collision_condition)
    # unzip a list, get a list of pass/fail and a list of initial conditions
    passed = [passed for passed,_, _ in res]
    initial_cond = [x0['plant'] for _,_,x0 in res]
    return tuple([passed, initial_cond])

```

## Uniform Sampling

```python
x_init = [gen_random_state(bounds) for _ in range(n_initial_sampling)]
y_train, x_train = validate_samples(x_init, my_conf)
y_train, x_train = np.array(y_train), np.array(x_train)
```

```python
fig, ax = gpr_plot_field(((0, 11), (0, 4), (4, 11), (4, 7)), names, bounds, x_train, y_train, x_center)
ax[0][0].set_title(f"F-16 ({run_name}) ROA Slices (initial sampling)")
plt.tight_layout()
plt.show()
```

## Active Learner Sampling

```python
def GP_regression_std(regressor, X):
    """GPR batch sampling policy (modified for parallel simulations)"""
    _, std = regressor.predict(X, return_std=True)
    query_idxs = std.argsort()[-16:][::-1]
    return query_idxs, X[query_idxs]

# kernel to use for active GP classification
kernel = RBF(rbf_scales)

# active learner
learner = ActiveLearner(
    estimator=GaussianProcessRegressor(kernel=kernel),
    query_strategy=GP_regression_std,
    X_training=x_train, y_training=y_train
)

# define query pool of samples
n_pool_samples = 1000
x_pool = np.array([gen_random_state(bounds) for _ in range(n_pool_samples)])
```

```python
xs_new = np.copy(x_train)
ys_new = np.copy(y_train)

for i in tqdm.tqdm(range(3)):
    # query for labels
    query_idxs, x_samp = learner.query(x_pool)
    x_pool = np.delete(x_pool, tuple(query_idxs), 0)

    # process them in the oracle
    y_train_i, x_train_i = validate_samples(x_samp, my_conf)
    y_train_i, x_train_i = np.array(y_train_i), np.array(x_train_i)
    xs_new = np.vstack((xs_new, x_train_i))
    ys_new = np.hstack((ys_new, y_train_i))

    # teach the active learner
    for xi, yi in zip(x_train_i, y_train_i):
        learner.teach(xi[np.newaxis, :], [yi])
```

```python
fig, ax = gpr_plot_field([(0, 11), (0, 4), (4, 11), (4, 7)], names, bounds, xs_new[:240], ys_new[:240], x_center)
ax[0][0].set_title(f"F-16 ({run_name}) ROA Slices")
plt.tight_layout()
plt.show()
```


# Introduction

## What is a Riemann Problem?

Bernhard Riemann studied [many problems](https://en.wikipedia.org/wiki/List_of_things_named_after_Bernhard_Riemann) in diverse mathematical fields, but for our purposes the term **Riemann problem** refers to a hyperbolic partial differential equation (PDE) together with special initial data:  a piecewise constant function with a single jump discontinuity separating two states.  For a hyperbolic system of $m$ PDEs, each of these states would be a vector with $m$ components. We will generally use $q(x,t)\in\mathbb{R}^m$ to denote the generic solution to a hyperbolic problem in on space dimension, and denote the Riemann initial data by $q_l$ for for the left state (i.e. $q(x,0) = q_l$ for $x<0$) and $q_r$ for the right state  (i.e. $q(x,0) = q_r$ for $x \geq 0$).

We will formally define what we mean by a hyperbolic system of PDEs shortly, but for now it suffices to know that these are the PDEs that naturally arise whenever we model wave propagation.  If we study small amplitude waves then the equations may be linear (e.g. acoustics or linear elasticity, modeling seismic waves for example).  These PDEs arise from linearizing nonlinear hyperbolic systems that must be used when we consider larger amplitude waves (e.g. the Euler equations of compressible gas dynamics, or nonlinear elastic wave equations).

## An example: the dam break problem

To get a feel for what a Riemann problem and it's solution looks like, consider a *dam break* problem in which a dam initially separates water with constant depth $h_l$ to the left of the dam from water with depth $h_r$ to the right of the dam.  Suppose the water is initially stationary, with velocity 0 on both sides of the dam.   Now suppose we remove the dam at time $t=0$.  What happens for $t>0$?

If the water to the left of the dam is deeper than the water to the right (i.e. $h_l > h_r$) then we expect water to flow from left to right in some manner.  In reality we might expect the flow to be somewhat turbulent and multidimensional, but suppose that we could idealize this with a one-dimensional model in which the velocity is purely horizontal and is constant throughout the depth of the fluid, so for $t>0$ the velocity varies only with $x$ and $t$, given by some function $u(x,t)$, and the depth is given by $h(x,t)$.

In [Shallow_water.ipynb](Shallow_water.ipynb) we will discuss the one-dimensional **shallow water equations** that can be used to model this idealized situation.

The animation below shows the solution to the shallow water equations in the situation just described.  The water is colored red for water that is initially to the left of the dam and blue for water that is initially to the right.  The stripes with different shading are to help you visualize the flow.  Think of the color as a dye that is simply carried along with the water as it flows.  In [Shallow_water_tracer.ipynb](Shallow_water_tracer.ipynb) we will see that we can in fact model this by adding another equation to the shallow water equations for the density of a tracer.

```python
%matplotlib inline
from exact_solvers import shallow_water
demo_plot = shallow_water.make_demo_plot_function
from ipywidgets import widgets, fixed
from ipywidgets import interact
```

```python
interact(demo_plot(h_l=3., h_r=1., u_l=0., u_r=0),
         t=widgets.FloatSlider(min=0., max=0.6, step=0.1,
                               value=0.), fig=fixed(0));
```

### Things to note about this animation

 - Initially the fluid velocity $u(x,0) \equiv 0$ at $t=0$.  For $t>0$, the water remains stationary far from away from the dam.  The region where the water is not stationary grows linearly outward from $x=0$ as $t$ increases.
 - The fluid is accelerated to the right everywhere as water starts to flow.
 - Water upstream from the dam (for $x<0$) starts to accelerate slowly as the depth of the water starts to fall, and both the depth and the velocity remain continuous functions of $x$ in this region for all time. The depth and velocity vary through a **rarefaction wave** that moves upstream.
 - The water downstream from the dam is stationary until a **shock wave** passes by, which instantaneously accelerates the fluid from $u_r=0$ to some constant velocity $u_m >0$.  At the same time the depth increases from $h_r$ to $h_m$. This *middle state* depth and velocity is the same as the depth and velocity that the red water upstream from the dam also reaches once it has passed through the rarefaction wave and been fully accelerated.
 - The shock wave moves at some constant velocity $s_r > u_m$.
 - The rarefaction wave spreads out as time evolves and the left and right edges move at constant velocities $\lambda_{l1}$ and $\lambda_{m1}$. We will see later that these speeds are eigenvalues of a certain Jacobian matrices.

A **shock wave** is a type of wave often seen when solving nonlinear hyperbolic equations.  The solution is discontinuous across a shock wave, which means that special mathematical techniques must be used to make sense of functions like this as solutions of a differential equation.  It also means that special algorithmic techniques must be used to compute accurate numerical solutions to such equations.  Simply replacing derivatives by finite differences typically does not work well.

In the context of water waves, this shock wave is often called a **hydraulic jump**.  In a real dam break problem the solution would not be discontinuous, but there might be turbulent bore that from far enough away looks like a discontinuity.  The nonlinear shallow water equations can be used to model such flows in the sense that the jump in depth and velocity and the propagation speed of the shock wave closely approximate the hydraulic jump and speed of the bore.

The nonlinear shallow water equations are usually solved in terms of the variables $q = [h, hu]$, where $h$ is the depth (mass per unit length) and $hu$ is the density of momentum (per unit length).  For flow on a flat bottom these are conserved quantities, and the hyperbolic problem in this case is called a **hyperbolic conservation law**.  For nonlinear hyperbolic equations conservation is the key to understanding the propagation of shock waves, as we will explore in several chapters.

## Why is the Riemann problem important?

Why study the simple case of piecewise constant initial data?  Here are some motivations:

 - As we have just seen, even this simple initial data can lead to complicated solutions.

 - However, the solution is simple enough that there is some hope we can determine it exactly, even for nonlinear systems of PDEs.

 - In the animation above we noted that the shock wave and the two edges of the rarefaction wave are all moving at *constant speeds*.  In fact if we plot contours of the depth (or the velocity) in the $x$-$t$ plane, we find that value of each variable is constant along any ray $x = at$ from the origin.  In other words $h(x,t) = H(x/t)$ and $u(x,t) = U(x/t)$ for some functions $H$ and $U$ of a single variable.  The solution is said to be a **similarity solution** and this structure greatly facilitates solving the Riemann problem.  This is true even for nonlinear hyperbolic problems provided certain conditions are satisfied.

 - Developing an understanding of how the Riemann solution behaves for all possible choices of left and right states for a hyperbolic problem is essential for understanding the behavior of more complicated solutions.

 - The development of mathematical theory of hyperbolic problems has relied extensively on the understanding that comes from studying Riemann problems.

 - Many robust numerical methods for approximating the solution with more general initial data use the Riemann solution as an essential building block.  The most basic method of this type is **Godunov's method**, described below. Numerical methods typically make use of *approximate Riemann solvers*, which are often computationally cheaper than an exact solver.

## The general Riemann problem for shallow water equations

The more general Riemann problem for the shallow water equations has arbitrary depths $h_l, h_r \geq 0$ and also arbitrary velocities $u_l$ and $u_r$ to the left and right of $x=0$.  The dam break problem considered above is a special case where the $u_l = u_r = 0$.

In the dam break problem considered above with $h_l > h_r$, the solution consists of a left-going rarefaction wave and a right-going shock wave.  If we instead have $h_l < h_r$ with $u_l=u_r=0$, then the water would flow to the left and the Riemann solution would have a left-going shock wave and a right-going rarefaction wave.

```python
interact(demo_plot(h_l=1., h_r=3., u_l=0., u_r=0),
         t=widgets.FloatSlider(min=0.0, max=0.6, step=0.1,
                               value=0.), fig=fixed(0));
```

By choosing non-zero velocities in the initial data, we can also obtain other combinations of shock and rarefaction waves. For example, if the depth is initially the same everywhere ($h_l = h_r$) while the velocities satisfy $u_l>0$ and $u_r<0$ (modeling two opposing streams of water colliding at $x=0$) then the Riemann solution consists of two shock waves. Note that the water is deeper between the shock waves, and in the symmetric case $u_r = -u_l$ the water is stationary ($u=0$) in the region between the shocks:

```python
interact(demo_plot(h_l=1., h_r=1., u_l=0.8, u_r=-0.8),
         t=widgets.FloatSlider(min=0., max=0.6, step=0.1,
                               value=0.), fig=fixed(0));
```

On the other hand, if $u_l<0$ and $u_r>0$ (so that the water is moving away from $x=0$ on both sides) then the solution consists of two rarefaction waves:

```python
interact(demo_plot(h_l=1., h_r=1., u_l=-0.8, u_r=0.8),
         t=widgets.FloatSlider(min=0.0, max=0.6, step=0.1,
                               value=0.), fig=fixed(0));
```

In [Shallow_water.ipynb](Shallow_water.ipynb) we will show how to solve the Riemann problem for arbitrary left and right states and in particular how to determine whether each wave is a shock or rarefaction wave.

## Goals of this book

The collection of Jupyter notebooks that make up this book will help you understand how to solve the Riemann problem for general hyperbolic PDE.  This will be accomplished by working through examples for several different systems of equations.  In the process you should also gain a better understanding of wave propagation phenomena in the context of several important applications.  [Expand this?]

## When is a PDE hyperbolic?

In this book we consider only first-order hyperbolic equations, meaning they involve only first derivatives in space and time. Many important hyperbolic PDE models can be written as a system of conservation laws:

\begin{align} \label{intro:multiDconslaw}
    q_t + \nabla \cdot F(q) & = 0.
\end{align}

Here $q(x,t)$ is a vector of conserved quantities (e.g. mass, momentum, energy) and each component of $F(q)$ is the corresponding flux.  In practice (and in most of this book) we focus on one-dimensional problems, for which (\ref{intro:multiDconslaw}) can be written

\begin{align} \label{intro:conslaw}
    q_t + f(q)_x & = 0.
\end{align}

If the flux in \eqref{intro:conslaw} is a linear function $f(q)= Aq$ for some matrix $A$, then we have

\begin{align} \label{intro:linconslaw}
    q_t + A q_x & = 0,
\end{align}

We say that \eqref{intro:linconslaw} is **hyperbolic** if $A$ is diagonalizable with real eigenvalues, and the nonlinear system \eqref{intro:conslaw} is hyperbolic at some state $q$ if the Jacobian matrix $f'(q)$ is diagonalizable with real eigenvalues.

A common property of these equations is that *information travels at finite speed* and they model *waves*.  As we will see in more detail later, the eigenvalues correspond to *wave speeds* while the eigenvectors tell us about the relation between the components of $q$ in each wave.

Some examples of physical problems that can be modeled with hyperbolic PDEs (and are included as examples in this book) are:

- Sound waves
- Surface water waves
- Traffic flow
- Gas dynamics

## Godunov's method

 - Add a brief description of Godunov's method and how it uses Riemann solutions.
 - Mention the need for approximate Riemann solvers.

## A linear example: acoustic waves

We end this introduction with a sample linear system. Acoustic waves in one dimension can be modeled by the system of PDEs

\begin{align} \label{intro:acoustics}
    p_t + K u_x & = 0 \\
    u_t + \frac{1}{\rho} p_x & = 0,
\end{align}

where $p$ represents the deviation from ambient pressure, $u$ is velocity, $K$ is the bulk modulus and $\rho$ is the density of the medium.  This first-order system is equivalent to the familiar wave equation $u_{tt} = c^2 u_{xx}$ with $c = \sqrt{K/\rho}$; recall that $c$ represents the speed of propagation of waves in the wave equation.  Coming back to the first-order form \eqref{intro:acoustics}, we can write it in the canonical form \eqref{intro:linconslaw} by setting

\begin{align}
    q & = \begin{bmatrix} p \\ u \end{bmatrix} &
    A & = \begin{bmatrix} 0 & K \\ 1/\rho & 0 \end{bmatrix}.
\end{align}

The eigenvalues of $A$ are $\pm \sqrt{K/\rho} = \pm c$; it is true in general that the speed of propagation of waves in \eqref{intro:linconslaw} is given by the eigenvalues of $A$, and this hints at the importance of the hyperbolicity condition (that $A$ be diagonalizable with real eigenvalues).

### The Riemann problem for acoustics
The Riemann problem consists of \eqref{intro:acoustics} together with initial data

\begin{align}
    (p_0,u_0) & = \begin{cases} (p_l, u_l) & \text{ for } x<0 \\
                                (p_r, u_r) & \text{ for } x>0. \end{cases}
\end{align}
The solution of the Riemann problem for this simple linear system consists of two jumps, one proportional to each of the eigenvectors of $A$, and each moving with velocity equal to the corresponding eigenvalue of $A$.  Between these two jumps lies a middle state different from the left and right states; we can think of solving the Riemann problem by finding the intersection (in the $p-u$ plane) of lines passing through the left and right states that are each parallel to the appropriate eigenvector of A.  This example is explored in more detail in [Acoustics.ipynb](Acoustics.ipynb).

[This webpage](http://www.clawpack.org/riemann_book/html/phase_plane_acoustics_small.html) shows the phase plane (the $p$-$u$ plane for acoustics) and the location of $q_m$ relative to $q_l$ and $q_r$.  You can drag the left and right states around to see how this changes the solution.  Also shown in the other two plots are the pressure $p(x,t)$ and velocity $u(x,t)$ at one particular time $t=1$ for this Riemann solution.

## Phase plane plot for the shallow water equations

The phase plane plot for the nonlinear shallow water equations is more interesting.  Since the eigenvectors of the Jacobian matrix vary with $q$, the curves connecting states separated by a single wave are not straight lines. There is also a distinction between the curves connecting states by a shock wave and those connecting states by a rarefaction wave.  This is discussed in much more detail in [Shallow_water.ipynb](Shallow_water.ipynb).

[This webpage](http://www.clawpack.org/riemann_book/html/phase_plane_shallow_water_small.html) provides an interactive view that allows you to see how the solution changes as you move the left and right states around in the phase plane.


# Poiseuille flow

$$
\renewcommand{\DdQq}[2]{{\mathrm D}_{#1}{\mathrm Q}_{#2}}
\renewcommand{\drondt}{\partial_t}
\renewcommand{\drondx}{\partial_x}
\renewcommand{\drondy}{\partial_y}
\renewcommand{\drondtt}{\partial_{tt}}
\renewcommand{\drondxx}{\partial_{xx}}
\renewcommand{\drondyy}{\partial_{yy}}
\renewcommand{\dx}{\Delta x}
\renewcommand{\dt}{\Delta t}
\renewcommand{\grandO}{{\mathcal O}}
\renewcommand{\density}[2]{\,f_{#1}^{#2}}
\renewcommand{\fk}[1]{\density{#1}{\vphantom{\star}}}
\renewcommand{\fks}[1]{\density{#1}{\star}}
\renewcommand{\moment}[2]{\,m_{#1}^{#2}}
\renewcommand{\mk}[1]{\moment{#1}{\vphantom{\star}}}
\renewcommand{\mke}[1]{\moment{#1}{e}}
\renewcommand{\mks}[1]{\moment{#1}{\star}}
$$

In this tutorial, we consider the classical $\DdQq{2}{9}$ to simulate a Poiseuille flow modeling by the Navier-Stokes equations.

```python
%matplotlib inline
```

## The $\DdQq{2}{9}$ **for Navier-Stokes**

The $\DdQq{2}{9}$ is defined by:

* a space step $\dx$ and a time step $\dt$ related to the scheme velocity $\lambda$ by the relation $\lambda=\dx/\dt$,

* nine velocities $\{(0,0), (\pm1,0), (0,\pm1), (\pm1, \pm1)\}$, identified in pylbm by the
numbers $0$ to $8$,

* nine polynomials used to build the moments

$$ \{1, \lambda X, \lambda Y, 3E-4, (9E^2-21E+8)/2, 3XE-5X, 3YE-5Y,X^2-Y^2, XY\},$$

where $E = X^2+Y^2$.

* three conserved moments $\rho$, $q_x$, and $q_y$,

* nine relaxation parameters (three are $0$ corresponding to conserved moments): $\{0,0,0,s_\mu,s_\mu,s_\eta,s_\eta,s_\eta,s_\eta\}$, where $s_\mu$ and $s_\eta$ are in $(0,2)$,

* equilibrium value of the non conserved moments
$$
\begin{aligned}\mke{3} &= -2\rho + 3(q_x^2+q_y^2)/(\rho_0\lambda^2), \\ \mke{4} &= \rho-3(q_x^2+q_y^2)/(\rho_0\lambda^2), \\ \mke{5} &= -q_x/\lambda, \\ \mke{6} &= -q_y/\lambda, \\ \mke{7} &= (q_x^2-q_y^2)/(\rho_0\lambda^2), \\ \mke{8} &= q_xq_y/(\rho_0\lambda^2),\end{aligned}
$$

where $\rho_0$ is a given scalar.

This scheme is consistant at second order with the following equations (taken $\rho_0=1$)
$$
\begin{gathered} \drondt\rho + \drondx q_x + \drondy q_y = 0,\\ \drondt q_x + \drondx (q_x^2+p) + \drondy (q_xq_y) = \mu \drondx (\drondx q_x + \drondy q_y ) + \eta (\drondxx+\drondyy)q_x, \\ \drondt q_y + \drondx (q_xq_y) + \drondy (q_y^2+p) = \mu \drondy (\drondx q_x + \drondy q_y ) + \eta (\drondxx+\drondyy)q_y,\end{gathered}
$$
with $p=\rho\lambda^2/3$.

## Build the simulation with pylbm
In the following, we build the dictionary of the simulation step by step.

### The geometry

The simulation is done on a rectangle of length $L$ and width $W$. We can use $L=W=1$.

We propose a dictionary that build the geometry of the domain. The labels of the bounds can be specified to different values for the moment.

```python
import numpy as np
import matplotlib.pyplot as plt

import pylbm

L, W = 1., 1.
dico_geom = {
    'box': {'x': [0, L],
            'y': [-.5*W, .5*W],
            'label':list(range(4))
           }
}
geom = pylbm.Geometry(dico_geom)
print(geom)
geom.visualize(viewlabel=True);
```

### The stencil

The stencil of the $\DdQq{2}{9}$ is composed by the nine following velocities in 2D:
$$
\begin{gathered}v_0=(0,0),\\ v_1=(1,0), \quad v_2=(0,1), \quad v_3=(-1,0), \quad v_4=(0,-1), \\ v_5=(1,1), \quad v_6=(-1,1), \quad v_7=(-1,-1), \quad v_8=(1,-1).\end{gathered}
$$

```python
dico_sten = {
    'dim': 2,
    'schemes': [
        {'velocities': list(range(9))}
    ],
}
sten = pylbm.Stencil(dico_sten)
print(sten)
sten.visualize();
```

### The domain

In order to build the domain of the simulation, the dictionary should contain the space step $\dx$ and the stencils of the velocities (one for each scheme).

```python
dico_dom = {
    'space_step': .1,
    'box': {'x': [0, L],
            'y': [-.5*W, .5*W],
            'label': list(range(4))
           },
    'schemes': [
        {'velocities': list(range(9))}
    ],
}
dom = pylbm.Domain(dico_dom)
print(dom)
dom.visualize(view_distance=True);
```

### The scheme

In pylbm, a simulation can be performed by using several coupled schemes. In this example, a single scheme is used and defined through a list of one single dictionary. This dictionary should contain:

* 'velocities': a list of the velocities
* 'conserved_moments': a list of the conserved moments as sympy variables
* 'polynomials': a list of the polynomials that define the moments
* 'equilibrium': a list of the equilibrium value of all the moments
* 'relaxation_parameters': a list of the relaxation parameters ($0$ for the conserved moments)
* 'init': a dictionary to initialize the conserved moments

(see the documentation for more details)

In order to fix the bulk ($\mu$) and the shear ($\eta$) viscosities, we impose
$$ s_\eta = \frac{2}{1+\eta d}, \qquad s_\mu = \frac{2}{1+\mu d}, \qquad d = \frac{6}{\lambda\rho_0\dx}.$$

The scheme velocity could be taken to $1$ and the inital value of $\rho$ to $\rho_0=1$, $q_x$ and $q_y$ to $0$.

In order to accelerate the simulation, we can use another generator. The default generator is Numpy (pure python). We can use for instance Cython that generates a more efficient code. This generator can be activated by using  'generator': pylbm.generator.CythonGenerator in the dictionary.

```python
import sympy as sp
X, Y, rho, qx, qy, LA = sp.symbols('X, Y, rho, qx, qy, LA')

# parameters
dx = 1./128  # spatial step
la = 1.      # velocity of the scheme
L = 1        # length of the domain
W = 1        # width of the domain
rhoo = 1.    # mean value of the density
mu   = 1.e-3 # shear viscosity
eta = 1.e-1 # bulk viscosity
# initialization
xmin, xmax, ymin, ymax = 0.0, L, -0.5*W, 0.5*W
dummy = 3.0/(la*rhoo*dx)
s_mu = 1.0/(0.5+mu*dummy)
s_eta = 1.0/(0.5+eta*dummy)
s_q = s_eta
s_es = s_mu
s  = [0.,0.,0.,s_mu,s_es,s_q,s_q,s_eta,s_eta]
dummy = 1./(LA**2*rhoo)
qx2 = dummy*qx**2
qy2 = dummy*qy**2
q2  = qx2+qy2
qxy = dummy*qx*qy

dico_sch = {
    'box': {'x': [xmin, xmax],
            'y': [ymin, ymax],
            'label':0
           },
    'space_step': dx,
    'scheme_velocity': la,
    'parameters': {LA: la},
    'schemes': [
        {
            'velocities': list(range(9)),
            'conserved_moments': [rho, qx, qy],
            'polynomials': [
                1, LA*X, LA*Y,
                3*(X**2+Y**2)-4,
                (9*(X**2+Y**2)**2-21*(X**2+Y**2)+8)/2,
                3*X*(X**2+Y**2)-5*X, 3*Y*(X**2+Y**2)-5*Y,
                X**2-Y**2, X*Y
            ],
            'relaxation_parameters': s,
            'equilibrium': [
                rho, qx, qy,
                -2*rho + 3*q2,
                rho-3*q2,
                -qx/LA, -qy/LA,
                qx2-qy2, qxy
            ],
        },
    ],
}
sch = pylbm.Scheme(dico_sch)
print(sch)
```

    +--------------------+
    | Scheme information |
    +--------------------+
        - spatial dimension: 2
        - number of schemes: 1
        - number of velocities: 9
        - conserved moments: [ρ, qx, qy]

        +----------+
        | Scheme 0 |
        +----------+
            - velocities
                (0: 0, 0)
                (1: 1, 0)
                (2: 0, 1)
                (3: -1, 0)
                (4: 0, -1)
                (5: 1, 1)
                (6: -1, 1)
                (7: -1, -1)
                (8: 1, -1)

            - polynomials

                ⎡                1                 ⎤
                ⎢                                  ⎥
                ⎢               LA⋅X               ⎥
                ⎢                                  ⎥
                ⎢               LA⋅Y               ⎥
                ⎢                                  ⎥
                ⎢            2      2              ⎥
                ⎢         3⋅X  + 3⋅Y  - 4          ⎥
                ⎢                                  ⎥
                ⎢                             2    ⎥
                ⎢      2       2     ⎛ 2    2⎞     ⎥
                ⎢  21⋅X    21⋅Y    9⋅⎝X  + Y ⎠     ⎥
                ⎢- ───── - ───── + ──────────── + 4⎥
                ⎢    2       2          2          ⎥
                ⎢                                  ⎥
                ⎢           ⎛ 2    2⎞              ⎥
                ⎢       3⋅X⋅⎝X  + Y ⎠ - 5⋅X        ⎥
                ⎢                                  ⎥
                ⎢           ⎛ 2    2⎞              ⎥
                ⎢       3⋅Y⋅⎝X  + Y ⎠ - 5⋅Y        ⎥
                ⎢                                  ⎥
                ⎢              2    2              ⎥
                ⎢             X  - Y               ⎥
                ⎢                                  ⎥
                ⎣               X⋅Y                ⎦

            - equilibria

                ⎡           ρ            ⎤
                ⎢                        ⎥
                ⎢           qx           ⎥
                ⎢                        ⎥
                ⎢           qy           ⎥
                ⎢                        ⎥
                ⎢             2         2⎥
                ⎢       3.0⋅qx    3.0⋅qy ⎥
                ⎢-2⋅ρ + ─────── + ───────⎥
                ⎢           2         2  ⎥
                ⎢         LA        LA   ⎥
                ⎢                        ⎥
                ⎢           2         2  ⎥
                ⎢     3.0⋅qx    3.0⋅qy   ⎥
                ⎢ ρ - ─────── - ───────  ⎥
                ⎢         2         2    ⎥
                ⎢       LA        LA     ⎥
                ⎢                        ⎥
                ⎢          -qx           ⎥
                ⎢          ────          ⎥
                ⎢           LA           ⎥
                ⎢                        ⎥
                ⎢          -qy           ⎥
                ⎢          ────          ⎥
                ⎢           LA           ⎥
                ⎢                        ⎥
                ⎢         2         2    ⎥
                ⎢   1.0⋅qx    1.0⋅qy     ⎥
                ⎢   ─────── - ───────    ⎥
                ⎢       2         2      ⎥
                ⎢     LA        LA       ⎥
                ⎢                        ⎥
                ⎢       1.0⋅qx⋅qy        ⎥
                ⎢       ─────────        ⎥
                ⎢            2           ⎥
                ⎣          LA            ⎦

            - relaxation parameters

                ⎡       0.0       ⎤
                ⎢                 ⎥
                ⎢       0.0       ⎥
                ⎢                 ⎥
                ⎢       0.0       ⎥
                ⎢                 ⎥
                ⎢1.13122171945701 ⎥
                ⎢                 ⎥
                ⎢1.13122171945701 ⎥
                ⎢                 ⎥
                ⎢0.025706940874036⎥
                ⎢                 ⎥
                ⎢0.025706940874036⎥
                ⎢                 ⎥
                ⎢0.025706940874036⎥
                ⎢                 ⎥
                ⎣0.025706940874036⎦

        - moments matrices

            ⎡1   1   1    1    1   1    1    1    1 ⎤
            ⎢                                       ⎥
            ⎢0   LA  0   -LA   0   LA  -LA  -LA  LA ⎥
            ⎢                                       ⎥
            ⎢0   0   LA   0   -LA  LA  LA   -LA  -LA⎥
            ⎢                                       ⎥
            ⎢-4  -1  -1  -1   -1   2    2    2    2 ⎥
            ⎢                                       ⎥
            ⎢4   -2  -2  -2   -2   1    1    1    1 ⎥
            ⎢                                       ⎥
            ⎢0   -2  0    2    0   1   -1   -1    1 ⎥
            ⎢                                       ⎥
            ⎢0   0   -2   0    2   1    1   -1   -1 ⎥
            ⎢                                       ⎥
            ⎢0   1   -1   1   -1   0    0    0    0 ⎥
            ⎢                                       ⎥
            ⎣0   0   0    0    0   1   -1    1   -1 ⎦

        - inverse of moments matrices

            ⎡1/9   0     0    -1/9    1/9     0      0     0     0  ⎤
            ⎢                                                       ⎥
            ⎢      1                                                ⎥
            ⎢1/9  ────   0    -1/36  -1/18  -1/6     0    1/4    0  ⎥
            ⎢     6⋅LA                                              ⎥
            ⎢                                                       ⎥
            ⎢            1                                          ⎥
            ⎢1/9   0    ────  -1/36  -1/18    0    -1/6   -1/4   0  ⎥
            ⎢           6⋅LA                                        ⎥
            ⎢                                                       ⎥
            ⎢     -1                                                ⎥
            ⎢1/9  ────   0    -1/36  -1/18   1/6     0    1/4    0  ⎥
            ⎢     6⋅LA                                              ⎥
            ⎢                                                       ⎥
            ⎢           -1                                          ⎥
            ⎢1/9   0    ────  -1/36  -1/18    0     1/6   -1/4   0  ⎥
            ⎢           6⋅LA                                        ⎥
            ⎢                                                       ⎥
            ⎢      1     1                                          ⎥
            ⎢1/9  ────  ────  1/18   1/36   1/12   1/12    0    1/4 ⎥
            ⎢     6⋅LA  6⋅LA                                        ⎥
            ⎢                                                       ⎥
            ⎢     -1     1                                          ⎥
            ⎢1/9  ────  ────  1/18   1/36   -1/12  1/12    0    -1/4⎥
            ⎢     6⋅LA  6⋅LA                                        ⎥
            ⎢                                                       ⎥
            ⎢     -1    -1                                          ⎥
            ⎢1/9  ────  ────  1/18   1/36   -1/12  -1/12   0    1/4 ⎥
            ⎢     6⋅LA  6⋅LA                                        ⎥
            ⎢                                                       ⎥
            ⎢      1    -1                                          ⎥
            ⎢1/9  ────  ────  1/18   1/36   1/12   -1/12   0    -1/4⎥
            ⎣     6⋅LA  6⋅LA                                        ⎦

### Run the simulation

For the simulation, we take

* The domain $x\in(0, L)$ and $y\in(-W/2,W/2)$, $L=2$, $W=1$,
* the viscosities $\mu=10^{-2}=\eta=10^{-2}$,
* the space step $\dx=1/128$, the scheme velocity $\lambda=1$,
* the mean density $\rho_0=1$.

Concerning the boundary conditions, we impose the velocity on all the edges by a bounce-back condition with a source term that reads
$$q_x(x, y) = \rho_0 v_{\text{max}} \Bigl( 1 - \frac{4y^2}{W^2} \Bigr), \qquad q_y(x, y) = 0,$$
with $v_{\text{max}}=0.1$.

We compute the solution for $t\in(0,50)$ and we plot several slices of the solution during the simulation.

This problem has an exact solution given by
$$ q_x = \rho_0 v_{\text{max}} \Bigl( 1 - \frac{4y^2}{W^2} \Bigr), \qquad q_y = 0, \qquad p = p_0 + K x, $$
where the pressure gradient $K$ reads
$$K = -\frac{8 v_{\text{max}} \eta}{W^2}.$$

We compute the exact and the numerical gradients of the pressure.

```python
X, Y, LA = sp.symbols('X, Y, LA')
rho, qx, qy = sp.symbols('rho, qx, qy')

def bc(f, m, x, y):
    m[qx] = rhoo * vmax * (1.-4.*y**2/W**2)
    m[qy] = 0.

def plot_coupe(sol):
    fig, ax1 = plt.subplots()
    ax2 = ax1.twinx()
    ax1.cla()
    ax2.cla()
    mx = int(sol.domain.shape_in[0]/2)
    my = int(sol.domain.shape_in[1]/2)
    x = sol.domain.x
    y = sol.domain.y
    u = sol.m[qx] / rhoo
    for i in [0,mx,-1]:
        ax1.plot(y+x[i], u[i, :], 'b')
    for j in [0,my,-1]:
        ax1.plot(x+y[j], u[:,j], 'b')
    ax1.set_ylabel('velocity', color='b')
    for tl in ax1.get_yticklabels():
        tl.set_color('b')
    ax1.set_ylim(-.5*rhoo*vmax, 1.5*rhoo*vmax)
    p = sol.m[rho][:,my] * la**2 / 3.0
    p -= np.average(p)
    ax2.plot(x, p, 'r')
    ax2.set_ylabel('pressure', color='r')
    for tl in ax2.get_yticklabels():
        tl.set_color('r')
    ax2.set_ylim(pressure_gradient*L, -pressure_gradient*L)
    plt.title('Poiseuille flow at t = {0:f}'.format(sol.t))
    plt.draw()
    plt.pause(1.e-3)

# parameters
dx = 1./16  # spatial step
la = 1.      # velocity of the scheme
Tf = 50      # final time of the simulation
L = 2        # length of the domain
W = 1        # width of the domain
vmax = 0.1   # maximal velocity obtained in the middle of the channel
rhoo = 1.    # mean value of the density
mu = 1.e-2   # bulk viscosity
eta = 1.e-2  # shear viscosity
pressure_gradient = - vmax * 8.0 / W**2 * eta
# initialization
xmin, xmax, ymin, ymax = 0.0, L, -0.5*W, 0.5*W
dummy = 3.0/(la*rhoo*dx)
s_mu = 1.0/(0.5+mu*dummy)
s_eta = 1.0/(0.5+eta*dummy)
s_q = s_eta
s_es = s_mu
s  = [0.,0.,0.,s_mu,s_es,s_q,s_q,s_eta,s_eta]
dummy = 1./(LA**2*rhoo)
qx2 = dummy*qx**2
qy2 = dummy*qy**2
q2  = qx2+qy2
qxy = dummy*qx*qy

dico = {
    'box': {'x': [xmin, xmax],
            'y': [ymin, ymax],
            'label': 0
           },
    'space_step': dx,
    'scheme_velocity': la,
    'parameters': {LA: la},
    'schemes': [
        {
            'velocities': list(range(9)),
            'conserved_moments': [rho, qx, qy],
            'polynomials': [
                1, LA*X, LA*Y,
                3*(X**2+Y**2)-4,
                (9*(X**2+Y**2)**2-21*(X**2+Y**2)+8)/2,
                3*X*(X**2+Y**2)-5*X, 3*Y*(X**2+Y**2)-5*Y,
                X**2-Y**2, X*Y
            ],
            'relaxation_parameters': s,
            'equilibrium': [
                rho, qx, qy,
                -2*rho + 3*q2,
                rho-3*q2,
                -qx/LA, -qy/LA,
                qx2-qy2, qxy
            ],
        },
    ],
    'init': {rho: rhoo,
             qx:0.,
             qy:0.
    },
    'boundary_conditions': {
        0: {'method': {0: pylbm.bc.BouzidiBounceBack}, 'value': bc}
    },
    'generator': 'cython',
}

sol = pylbm.Simulation(dico)
while (sol.t<Tf):
    sol.one_time_step()
plot_coupe(sol)
ny = int(sol.domain.shape_in[1]/2)
num_pressure_gradient = (sol.m[rho][-2,ny] - sol.m[rho][1,ny]) / (xmax-xmin) * la**2/ 3.0
print("Exact pressure gradient    : {0:10.3e}".format(pressure_gradient))
print("Numerical pressure gradient: {0:10.3e}".format(num_pressure_gradient))
```


```python
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import arviz as az
import pymc3 as pm

from scipy.special import expit as logistic
```

```python
az.style.use('arviz-darkgrid')
```

```python
%%HTML
<style>

.CodeMirror {
    width: 100vw;
}

.container {
    width: 99% !important;
}

.rendered_html {
  font-size:0.8em;
}
.rendered_html table, .rendered_html th, .rendered_html tr, .rendered_html td {
     font-size: 100%;
}

body {
  font-family: Ubuntu;
  background: #F0F0F0;
  background-color: #F0F0F0;
}

.reveal h1,
.reveal h2,
.reveal h3,
.reveal h4,
.reveal h5,
.reveal h6 {
  margin: 0 0 20px 0;
  color: #2a2eec;
  font-family: Ubuntu;
  line-height: 0.9em;
  letter-spacing: 0.02em;
  text-transform: none;
  text-shadow: none;
}

.reveal blockquote {
  display: block;
  position: relative;
  background: #fa7c17;
  border-radius: 15px;
  box-shadow: 0px 0px 2px rgba(0, 0, 0, 0.2);
  font-weight: bold;
}

</style>
```

```python
from traitlets.config.manager import BaseJSONConfigManager
path = "/home/osvaldo/anaconda3/etc/jupyter/nbconfig"
cm = BaseJSONConfigManager(config_dir=path)
cm.update("livereveal", {
              "theme": "serif",
              "transition": "zoom",
              "start_slideshow_at": "selected",
              "controls": "True",
              "progress": "False",
              "shortcut": "False"});
```

#####

Hierarchical models

Partially pooling data

## Aims

* Learn how to build hierarchical models
* Discuss the concept of shrinkage and regularization
* Discuss the implications of partially pooling data

## Hierarchical models

## Hierarchical Chemical Shifts

* Proteins are molecules formed by a variable number of 20 units in a variable order

* For each one of this units we can measure a "chemical shift" and we can also compute it using Quantum Chemistry methods

* As each one of the 20 units are chemically different there is variability between units

* As the chemical environment (the protein) modulates the value of a chemical shift there is variability within each unit

```python
cs_data = pd.read_csv('../datos/chemical_shifts_theo_exp.csv')
diff = cs_data.theo - cs_data.exp
idx = pd.Categorical(cs_data['aa']).codes
groups = len(np.unique(idx))

cs_data.tail()
```

      ID
      aa
      theo
      exp

      1771
      1KS9
      LYS
      55.79
      57.51

      1772
      1KS9
      ARG
      58.91
      59.02

      1773
      1KS9
      LYS
      59.49
      58.92

      1774
      1KS9
      GLU
      59.48
      58.36

      1775
      1KS9
      SER
      58.07
      60.55

```python
with pm.Model() as cs_nh:
    μ = pm.Normal('μ', mu=0, sd=10, shape=groups)
    σ = pm.HalfNormal('σ', sd=10, shape=groups)

    y = pm.Normal('y', mu=μ[idx], sd=σ[idx], observed=diff)

    trace_cs_nh = pm.sample(1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [σ, μ]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:03<00:00, 814.32draws/s]

```python
with pm.Model() as cs_h:
    # hyper_priors
    μ_mu = pm.Normal('μ_mu', mu=0, sd=10)
    μ_sd = pm.HalfNormal('μ_sd', 10)

    # priors
    μ = pm.Normal('μ', mu=μ_mu, sd=μ_sd, shape=groups)
    σ = pm.HalfNormal('σ', sd=10, shape=groups)

    y = pm.Normal('y', mu=μ[idx], sd=σ[idx], observed=diff)

    trace_cs_h = pm.sample(1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [σ, μ, μ_sd, μ_mu]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:04<00:00, 633.95draws/s]

```python
_, axes = az.plot_forest([trace_cs_nh, trace_cs_h], model_names=['non_hierarchical', 'hierarchical'],
              var_names='μ', combined=True, r_hat=False, ess=False, figsize=(10, 12),
              colors='cycle');
y_lims = axes[0].get_ylim()
axes[0].vlines(trace_cs_h['μ_mu'].mean(), *y_lims);
```

## Shrinkage and advantages of hierarchical models

* Each group informs the others and is informed by the others

* The estimation of each group is _pooled_ or _shrink_ toward the common mean

* The amount of shrinkage is not the same for every group
    * Typical groups are less affected
    * Groups with larger sample size are less affected

* Retain variation at the group level and among groups

* Hierarchical model is a form of regularization

##  Multilevel tadpoles

* We have 48 tanks filled with red frogs' tadpoles

* We want to model the probability of survival

* Conditions such as temperature, pH, light, etc vary slightly among tanks (but we are not explicitly taking them into account)

* We can think of each tank as a cluster or group

[statistical rethinking](https://xcelab.net/rm/statistical-rethinking/)

```python
d = pd.read_csv('../datos/reedfrogs.csv', sep=",")
d.head()
```

      density
      pred
      size
      surv
      propsurv

      0
      10
      no
      big
      9
      0.9

      1
      10
      no
      big
      10
      1.0

      2
      10
      no
      big
      7
      0.7

      3
      10
      no
      big
      10
      1.0

      4
      10
      no
      small
      9
      0.9

```python
with pm.Model() as model_tadpoles:
    # Hyperpriors
    μ = pm.Normal('μ', 0., 1.)
    σ = pm.HalfNormal('σ', 10.)
    # Prior
    a_tank = pm.Normal('a_tank', μ, σ, shape=d.shape[0])
    p = pm.Deterministic('p', pm.math.sigmoid(a_tank)) # a logistic transformation
    #likelihood
    surv = pm.Binomial('surv', n=d.density, p=p, observed=d.surv)

    trace_tadpoles = pm.sample(2000, tune=2000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [a_tank, σ, μ]
    Sampling 2 chains: 100%|██████████| 8000/8000 [00:06<00:00, 1145.26draws/s]

```python
_, ax = plt.subplots(1, 1, figsize=(12, 5))

ax.scatter(np.arange(0, 48), d.propsurv, color='C0')
ax.scatter(np.arange(0, 48), np.mean(trace_tadpoles['p'], 0), color='C1')
ax.hlines(logistic(np.median(trace_tadpoles['μ'], axis=0)), -1, 49, linestyles='--')

idx = d.density[d.density.diff() > 0].index
ax.vlines(idx + 0.5, -.05, 1.05, lw=.5)
for i, t in zip(np.linspace(0, 48, 7)[1::2], ('small', 'medium', 'large')):
    ax.text(i, 0, t, horizontalalignment='center')
ax.set_xlabel('tanks')
ax.set_ylabel('proportion survival')
ax.set_xlim(-1, 48)
ax.set_xticks([])
ax.set_ylim(-.05, 1.05)
ax.grid(False)
```

### Hierarchical coins

\begin{align}
\mu &\sim \operatorname{Beta}(\alpha, \beta) \\
\kappa &\sim \operatorname{Gamma}(s, r) \\
\theta &\sim \operatorname{Beta}(\alpha=\mu  \kappa, \beta=(1 - \mu)  \kappa) \\
y &\sim \operatorname{Bin}(n=1, p=\theta)
\end{align}

[Doing Bayesian Data Analysis](https://sites.google.com/site/doingbayesiandataanalysis/)

```python
N = [30, 30, 30]  # Number of trials for each coin
z = [3, 3, 3]  # Number of heads for each coin

# index vector
coins = np.repeat(np.arange(len(N)), N)
# list filled with  zeros (tails) and ones (heads)
data = []
for i in range(len(N)):
    data.extend(np.repeat([1, 0], [z[i], N[i]-z[i]]))
```

```python
with pm.Model() as modelo_j:
    # hiperparameters
    μ = pm.Beta('μ', 2., 2.)
    κ = pm.HalfNormal('κ', 10.)

    # priors
    θ = pm.Beta('θ', μ * κ, (1 - μ) * κ, shape=len(N))
    # likelihood
    y = pm.Bernoulli('y', p=θ[coins], observed=data)

    trace_j = pm.sample(2000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [θ, κ, μ]
    Sampling 2 chains: 100%|██████████| 5000/5000 [00:03<00:00, 1316.44draws/s]
    There was 1 divergence after tuning. Increase `target_accept` or reparameterize.

```python
az.plot_trace(trace_j);
```

```python
az.summary(trace_j)
```

      mean
      sd
      hpd_3%
      hpd_97%
      mcse_mean
      mcse_sd
      ess_mean
      ess_sd
      ess_bulk
      ess_tail
      r_hat

      μ
      0.188
      0.084
      0.054
      0.346
      0.002
      0.001
      2232.0
      2044.0
      2449.0
      2321.0
      1.0

      κ
      10.738
      6.293
      0.875
      22.173
      0.118
      0.083
      2858.0
      2858.0
      2384.0
      2169.0
      1.0

      θ[0]
      0.119
      0.054
      0.027
      0.221
      0.001
      0.001
      3036.0
      3021.0
      2847.0
      2176.0
      1.0

      θ[1]
      0.117
      0.053
      0.027
      0.213
      0.001
      0.001
      3576.0
      3576.0
      3280.0
      2323.0
      1.0

      θ[2]
      0.118
      0.053
      0.028
      0.217
      0.001
      0.001
      3473.0
      3473.0
      3289.0
      2237.0
      1.0

## Exercises

Repeat the hierarchical coins example, this time use the following data:

* z = [3,3,3]
* z = [18,3,18]

Record the value of the estimated means for $\theta$, so you can compare the 3 examples (including `z = [18,18,18]`)