# Confusion matrix

Part of a [Recidivism Case Study](https://allendowney.github.io/RecidivismCaseStudy/)

by [Allen Downey](https://allendowney.com)

[Attribution-NonCommercial 4.0 International (CC BY-NC 4.0)](https://creativecommons.org/licenses/by-nc/4.0/)

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
```

## Data

The authors of "Machine Bias" published their data and analysis in [this repository](https://github.com/propublica/compas-analysis).

The terms of use for the data [are here](https://www.propublica.org/datastore/terms).  In compliance with those terms, I am not redistributing the data.  The following cell downloads the data file we'll use directly from their repository.

```python
import os

if not os.path.exists('compas-scores-two-years.csv'):
    !wget https://github.com/propublica/compas-analysis/raw/master/compas-scores-two-years.csv
```

The following cell reads the data file:

```python
cp = pd.read_csv("compas-scores-two-years.csv")
cp.shape
```

    (7214, 53)

The dataset includes 7214 rows, one for each defendant, and 53 columns.

## Code

The functions from the previous notebooks are in a file called `utils.py`; the following cell downloads it if you don't already have it:

```python
import os

if not os.path.exists('utils.py'):
    !wget https://raw.githubusercontent.com/AllenDowney/RecidivismCaseStudy/master/utils.py
```

## Recreating the confusion matrix

An ideal test should have equal predictive value in all groups; that is, two people with the same risk score should have the same probability of recidivism, regardless of what group they are in.

An ideal test should also have the same error rates for all groups; that is, any two recidivists should have same probability of being mis-classified as low risk (FNR), and any two non-recidivists should have the same probability of being mis-classified as high risk (FPR).

But it is not possible to achieve both of these goals at the same time:

* If you design a test to achieve equal predictive value across groups with different prevalence, you will find that error rates differ. Specifically, false positive rates will be higher in groups with higher recividism.

* If you design a test to achieve equal error rates across groups, you will find that predictive values differ. Specifically, positive predictive value will be lower in groups with lower rates of recidivism.

In previous notebooks I asserted these claims without proof.  In this notebook I will prove them by showing:

* Given prevalence, PPV and NPV, we can compute the confusion matrix and the error rates.

* Given prevalence, FPR and FNR, we can compute the confusion matrix and the predictive values.

To see why, consider this: the confusion matrix has four numbers in it, but they have to add up to 100%, so if you provide any three (independent) metrics, I can compute the elements of the matrix and any other metrics.

We could do the algebra by hand, but it is easier (and probably more reliable) to use SymPy.

Here are the symbols we'll use:

```python
from sympy import symbols, Eq, solve

tn, fp, fn, tp = symbols('tn, fp, fn, tp')
ppv, npv, prev = symbols('ppv, npv, prev')
fpr, fnr = symbols('fpr, fnr')
```

And here are the equations that relate the metrics to the elements of the confusion matrix.

```python
eq1 = Eq(prev, tp+fn)
eq1
```

$\displaystyle prev = fn + tp$

```python
eq2 = Eq(fpr, fp / (fp+tn))
eq2
```

$\displaystyle fpr = \frac{fp}{fp + tn}$

```python
eq3 = Eq(fnr, fn / (fn+tp))
eq3
```

$\displaystyle fnr = \frac{fn}{fn + tp}$

```python
eq4 = Eq(ppv, tp / (tp+fp))
eq4
```

$\displaystyle ppv = \frac{tp}{fp + tp}$

```python
eq5 = Eq(npv, tn / (tn+fn))
eq5
```

$\displaystyle npv = \frac{tn}{fn + tn}$

```python
eq6 = Eq(tn+fp+fn+tp, 1)
eq6
```

$\displaystyle fn + fp + tn + tp = 1$

We can use `solve` to get expressions for the elements of the confusion matrix in terms of `ppv`, `npv`, and `prev`:

```python
soln1 = solve([eq1, eq2, eq3, eq4, eq5, eq6],
              [tp, fp, fn, tn, fpr, fnr])
soln1[0]
```

    (ppv*(npv + prev - 1)/(npv + ppv - 1),
     -(ppv - 1)*(npv + prev - 1)/(npv + ppv - 1),
     -(npv - 1)*(ppv - prev)/(npv + ppv - 1),
     npv*(ppv - prev)/(npv + ppv - 1),
     (ppv - 1)*(npv + prev - 1)/((prev - 1)*(npv + ppv - 1)),
     -(npv - 1)*(ppv - prev)/(prev*(npv + ppv - 1)))

We can also solve for the elements of the confusion matrix in terms of `fpr`, `fnr`, and `prev`:

```python
soln2 = solve([eq1, eq2, eq3, eq4, eq5, eq6],
              [tp, fp, fn, tn, ppv, npv])

soln2[0]
```

    (prev*(1 - fnr),
     fpr*(1 - prev),
     fnr*prev,
     (fpr - 1)*(prev - 1),
     prev*(fnr - 1)/(fnr*prev + fpr*prev - fpr - prev),
     (fpr - 1)*(prev - 1)/(fnr*prev + fpr*prev - fpr - prev + 1))

The following function takes `ppv`, `npv`, and `prev` and uses the expressions from `soln1` to compute a confusion matrix with those metrics.

```python
def constant_predictive_value(ppv, npv, prev):
    """Make a confusion matrix with given metrics.

    ppv: positive predictive value (0-100)
    npv: negative predictive value (0-100)
    prev: prevalence (0-100)

    returns: confusion matrix
    """
    ppv /= 100
    npv /= 100
    prev /= 100
    a = np.array((ppv*(npv + prev - 1)/(npv + ppv - 1),
                -(ppv - 1)*(npv + prev - 1)/(npv + ppv - 1),
                -(npv - 1)*(ppv - prev)/(npv + ppv - 1),
                npv*(ppv - prev)/(npv + ppv - 1)))

    return pd.DataFrame(a.reshape((2, 2)))
```

To test it, I'll use metrics from the confusion matrix for all defendants.

```python
from utils import make_matrix
from utils import compute_metrics

matrix_all = make_matrix(cp)
metrics_all = compute_metrics(matrix_all, 'All defendants')
metrics_all
```

      Percent

      All defendants

      FPR
      32.349230

      FNR
      37.403876

      PPV
      61.350618

      NPV
      68.796510

      Prevalence
      45.065151

I'll extract PPV, NPV, and prevalence.

```python
from utils import predictive_value
from utils import prevalence

ppv, npv = predictive_value(matrix_all)
prev = prevalence(matrix_all)
```

Given those values, we can reconstruct the confusion matrix and compute the metrics again.

```python
matrix = constant_predictive_value(ppv, npv, prev)
metrics_pred = compute_metrics(matrix)
metrics_pred
```

      Percent

      FPR
      32.349230

      FNR
      37.403876

      PPV
      61.350618

      NPV
      68.796510

      Prevalence
      45.065151

These "predicted metrics" are the same as the actual metrics, except for small errors due to floating-point approximation.

```python
metrics_pred - metrics_all
```

      Percent

      FPR
      0.000000e+00

      FNR
      0.000000e+00

      PPV
      0.000000e+00

      NPV
      -1.421085e-14

      Prevalence
      0.000000e+00

So that confirms that we can use PPV, NPV and prevalence to compute FPR and FNR.

Now let's see what happens if we hold PPV and NPV constant, and vary prevalence.

## The constant predictive value model

To show these effects more clearly, I'll sweep through a range of prevalences and plot the error rates we get if we hold predictive values constant.

```python
from utils import error_rates

def run_cpv_model(cp):
    """Run the constant predictive value model.

    cp: DataFrame of COMPAS data

    returns: DataFrame with a row for each prevalence and
             a column for each metric (FPR, FNR)
    """
    matrix_all = make_matrix(cp)

    ppv, npv = predictive_value(matrix_all)
    prevalences = np.linspace(35, 55, 11)

    pred_er = pd.DataFrame(columns=['fpr', 'fnr'])

    for prev in prevalences:
        m = constant_predictive_value(ppv, npv, prev)
        pred_er.loc[prev] = error_rates(m)

    return pred_er
```

```python
pred_er = run_cpv_model(cp)
pred_er
```

      fpr
      fnr

      35.0
      7.488034
      77.925569

      37.0
      11.795671
      68.118564

      39.0
      16.385776
      59.317406

      41.0
      21.287075
      51.374897

      43.0
      26.532324
      44.171227

      45.0
      32.159046
      37.607882

      47.0
      38.210427
      31.603121

      49.0
      44.736425
      26.088543

      51.0
      51.795158
      21.006482

      53.0
      59.454634
      16.307973

      55.0
      67.794953
      11.951173

The following figure shows the error rates we would expect from a test with equal predictive value for all groups, regardless of prevalence.

```python
from utils import decorate

def plot_cpv_model(pred_er):
    """Plot error rates with constant predictive values.

    pred_er: DataFrame of predicted error rates
    """
    pred_er['fpr'].plot(label='Predicted FPR', color='C2')
    pred_er['fnr'].plot(label='Predicted FNR', color='C4')
    decorate(xlabel='Prevalence', ylabel='Percent',
             title='Error rates, constant predictive value')
```

```python
plot_cpv_model(pred_er)
plt.legend();
```

As prevalence increases, FPR increases and FNR decreases.  Note the vertical scale: the difference in error rates between a low-prevalence group and a high-prevalence group is dramatic!

For the COMPAS test, the effect is not as extreme.  The following loop computes the actual error rates for white, black, and all defendants.

```python
black = (cp['race'] == 'African-American')
matrix_black = make_matrix(cp[black])
```

```python
white = (cp['race'] == 'Caucasian')
matrix_white = make_matrix(cp[white])
```

When prevalence is lower, the false positive rate is higher and the false negative rate lower.

```python
actual_er = pd.DataFrame(columns=['fpr', 'fnr'])

for m in (matrix_white, matrix_all, matrix_black):
    prev = prevalence(m)
    actual_er.loc[prev] = error_rates(m)

actual_er
```

      fpr
      fnr

      39.364303
      23.454301
      47.722567

      45.065151
      32.349230
      37.403876

      51.433983
      44.846797
      27.985271

The following figure shows the actual error rates compared to predictions from the constant predictive value (CPV) model.

The data points show error rates for white defendants (left), all defendants (middle), and black defendants (right).

```python
plot_cpv_model(pred_er)

actual_er['fpr'].plot(marker='o', linestyle='dotted', color='C2',
                      label='COMPAS FPR')

actual_er['fnr'].plot(marker='o', linestyle='dotted', color='C4',
                      label='COMPAS FNR')

plt.legend();
```

For the higher-prevalence group:

* The actual false positive rate is lower that what we would expect if the test had the same predictive value for all groups.

* The actual false negative rate is higher than expected.

For the lower-prevalence group:

* The actual false positive rate is higher than what we would expect if the test had the same predictive value for all groups.

* The actual false negative rate is lower than expected.

Relative to the CPV model, the COMPAS test is what I'll call "tempered", that is, less sensitive to variation in prevalence between groups.

## Constant error rate model

In the previous section we held predictive value constant and computed the effect on error rates.  In this section we'll go the other way: if we hold error rates constant for all groups, what effect does that have on predictive value?

The following function takes prevalence and error rates and returns a confusion matrix.

```python
def constant_error_rates(fpr, fnr, prev):
    """Make a confusion matrix with given metrics.

    fpr: false positive rate (0 - 100)
    fnr: false negative rate (0 - 100)
    prev: prevalence (0 - 100)

    returns: confusion matrix
    """
    prev /= 100
    fpr /= 100
    fnr /= 100
    a = [[prev*(1 - fnr),  fpr*(1 - prev)],
         [fnr*prev,       (fpr - 1)*(prev - 1)]]

    return pd.DataFrame(a)
```

To test it, we can extract metrics for all defendants:

```python
fpr, fnr = error_rates(matrix_all)
prev = prevalence(matrix_all)
```

And use them to compute the other metrics:

```python
matrix = constant_error_rates(fpr, fnr, prev)
metrics_pred = compute_metrics(matrix)
metrics_pred
```

      Percent

      FPR
      32.349230

      FNR
      37.403876

      PPV
      61.350618

      NPV
      68.796510

      Prevalence
      45.065151

These "predicted metrics" are the same as the actual metrics, except for small errors due to floating-point approximation.

```python
metrics_pred - metrics_all
```

      Percent

      FPR
      0.000000e+00

      FNR
      0.000000e+00

      PPV
      -7.105427e-15

      NPV
      1.421085e-14

      Prevalence
      0.000000e+00

Now we can see how predictive value depends on prevalence (with error rates held constant).

```python
def run_cer_model(cp):
    """Run the constant error rate model.

    cp: DataFrame of COMPAS data

    returns: DataFrame with a row for each prevalence and
             a column for each metric (PPV, NPV)
    """
    matrix_all = make_matrix(cp)

    fpr, fnr = error_rates(matrix_all)
    prevalences = np.linspace(35, 65, 11)

    pred_pv = pd.DataFrame(columns=['ppv', 'npv'])

    for prev in prevalences:
        m = constant_error_rates(fpr, fnr, prev)
        pred_pv.loc[prev] = predictive_value(m)

    return pred_pv
```

```python
pred_pv = run_cer_model(cp)
pred_pv
```

      ppv
      npv

      35.0
      51.026704
      77.058610

      38.0
      54.253815
      74.689735

      41.0
      57.350053
      72.243048

      44.0
      60.323222
      69.714652

      47.0
      63.180516
      67.100383

      50.0
      65.928580
      64.395791

      53.0
      68.573560
      61.596113

      56.0
      71.121148
      58.696245

      59.0
      73.576628
      55.690711

      62.0
      75.944906
      52.573628

      65.0
      78.230545
      49.338668

The following function plots results from the constant error rate (CER) model.

```python
def plot_cer_model(pred_pv):
    """Plot error rates with constant predictive values.

    pred_er: DataFrame of predicted error rates
    """
    pred_pv['ppv'].plot(label='Predicted PPV', color='C0')
    pred_pv['npv'].plot(label='Predicted NPV', color='C1')
    decorate(xlabel='Prevalence', ylabel='Percent',
             title='Predictive value, constant error rates')
```

```python
plot_cer_model(pred_pv)
plt.legend();
```

If we hold error rates constant and increase prevalence, PPV increases and NPV decreases.

For the COMPAS test, the effect is not as extreme.  The following loop computes the actual predictive values for the three groups.

```python
actual_pv = pd.DataFrame(columns=['ppv', 'npv'])

for m in (matrix_white, matrix_all, matrix_black):
    prev = prevalence(m)
    actual_pv.loc[prev] = predictive_value(m)

actual_pv
```

      ppv
      npv

      39.364303
      59.133489
      71.187500

      45.065151
      61.350618
      68.796510

      51.433983
      62.971481
      65.045992

The following figure shows the actual predictive values compared to predictions from the constant error rate (CER) model.

The data points show error rates for white defendants (left), all defendants (middle), and black defendants (right).

The following figure shows the constant error rate model again, along with actual predictive values for white defendants (left), all defendants (middle), and black defendants (right).

```python
plot_cer_model(pred_pv)

actual_pv['ppv'].plot(marker='o', linestyle='dotted', color='C0',
                      label='COMPAS PPV')

actual_pv['npv'].plot(marker='o', linestyle='dotted', color='C1',
                      label='COMPAS NPV')

plt.legend();
```

Again, the test is less sensitive to differences in prevalence between groups than we would expect from the constant error rate model.

## Constant predictive value model

If two groups have difference prevalence, a binary classifier can have the same predictive values (PPV and NPV) for both groups, or the same error rates (FPR and FNR), but not both at the same time.

* If we are given predictive values, we can compute error rates for any prevalence.  I call this is the constant predictive value (CPV) model.

* If we are given error rates, we can compute predictive values for any prevalence.  I call this the constant error rate (CER) model.

In the previous sections, we saw that COMPAS does not follow either model with respect to black and white defendants.  Instead it seems to be "tempered", which means that when we compare groups with lower or higher prevalence:

* Error rates do not vary as much as the CPV model predicts.

* Predictive values do not vary as much as the CER model predicts.

The algorithms COMPAS uses are kept secret, so we can't tell whether this behavior is deliberate.

But we can get a clue by using the same analysis to compare other groups.

We'll compare the model with actual error rates for female and male defendants.

```python
matrix_all = make_matrix(cp)
```

```python
male = (cp['sex'] == 'Male')
matrix_male = make_matrix(cp[male])
```

```python
female = (cp['sex'] == 'Female')
matrix_female = make_matrix(cp[female])
```

```python
actual_er = pd.DataFrame(columns=['fpr', 'fnr'])

for m in (matrix_female, matrix_all, matrix_male):
    prev = prevalence(m)
    actual_er.loc[prev] = error_rates(m)

actual_er
```

      fpr
      fnr

      35.698925
      32.107023
      39.156627

      45.065151
      32.349230
      37.403876

      47.310534
      32.420091
      37.086814

The following figure compares predictions from the CPV model to actual error rates for female defendants (on the left), all defendants (in the middle), and male defendants (on the right).

```python
plot_cpv_model(pred_er)

actual_er['fpr'].plot(marker='o', linestyle='dotted', color='C2',
                      label='COMPAS FPR')

actual_er['fnr'].plot(marker='o', linestyle='dotted', color='C4',
                      label='COMPAS FNR')

plt.legend();
```

Although male and female defendants have very difference prevalences (rates of recidivism), their error rates are almost the same.

This is the "natural" behavior we should expect from a test that does not depend on sex.

If we look again at the definition of FPR and FNR, using this diagram from [Wikipedia](https://en.wikipedia.org/wiki/Confusion_matrix)

We see that FNR depends only on people in the positive condition (people who will recidivate) and FPR depends only on people in the negative condition (people who will "survive" without being charged with another crime).

Neither error rate depends on how many people are in each condition, so if the test is sex blind, it should have the same error rates for both sexes, regardless of prevalence.

## Constant error rate model

The previous section shows the COMPAS has the same error rates for male and female defendants, even though they have substantially difference prevalences.

So we expected the test to have different predictive values for these groups.  To confirm that I'll compute the actual predictive values and compare them to the CER model.

The I'll use the CER model to compute PPV and NPV for a range of prevalences.

Here are the actual values for male and female defendants.

```python
actual_pv = pd.DataFrame(columns=['ppv', 'npv'])

for m in (matrix_female, matrix_all, matrix_male):
    prev = prevalence(m)
    actual_pv.loc[prev] = predictive_value(m)

actual_pv
```

      ppv
      npv

      35.698925
      51.269036
      75.746269

      45.065151
      61.350618
      68.796510

      47.310534
      63.536317
      66.989977

The following figure compares the actual values to the CER model.

```python
plot_cer_model(pred_pv)

actual_pv['ppv'].plot(marker='o', linestyle='dotted', color='C0',
                      label='COMPAS PPV')

actual_pv['npv'].plot(marker='o', linestyle='dotted', color='C1',
                      label='COMPAS NPV')

plt.legend();
```

```python

```


```python
import sympy as mp
```

```python
a, b, c, z, n = mp.symbols('a, b, c, z, n')
```

```python
mp.integrate(
    (c - z)**(n - 1) * z,
    (z, a, b)
)
```

$\displaystyle \begin{cases} \frac{\left(-1\right)^{n} a a^{n}}{n + 1} - \frac{\left(-1\right)^{n} b b^{n}}{n + 1} & \text{for}\: \left(c = 0 \wedge n = -1\right) \vee \left(c = 0 \wedge n = 0\right) \vee \left(c = 0 \wedge n = -1 \wedge n = 0\right) \vee c = 0 \\- \frac{a \log{\left(a - c \right)}}{a - c} + \frac{b \log{\left(b - c \right)}}{b - c} - \frac{c \log{\left(b - c \right)}}{b - c} - \frac{c}{b - c} + \frac{c \log{\left(a - c \right)}}{a - c} + \frac{c}{a - c} & \text{for}\: n = -1 \\a - b + c \log{\left(a - c \right)} - c \log{\left(b - c \right)} & \text{for}\: n = 0 \\- \frac{a^{2} n^{2} \left(- a + c\right)^{n}}{- 2 a n^{2} - 2 a n + 2 c n^{2} + 2 c n} - \frac{a^{2} n \left(- a + c\right)^{n}}{- 2 a n^{2} - 2 a n + 2 c n^{2} + 2 c n} - \frac{a n^{2} \left(- a + c\right)^{n}}{2 n^{2} + 2 n} + \frac{a n \left(- a + c\right)^{n}}{2 n^{2} + 2 n} + \frac{b^{2} n^{2} \left(- b + c\right)^{n}}{- 2 b n^{2} - 2 b n + 2 c n^{2} + 2 c n} + \frac{b^{2} n \left(- b + c\right)^{n}}{- 2 b n^{2} - 2 b n + 2 c n^{2} + 2 c n} + \frac{b n^{2} \left(- b + c\right)^{n}}{2 n^{2} + 2 n} - \frac{b n \left(- b + c\right)^{n}}{2 n^{2} + 2 n} + \frac{c^{2} n^{2} \left(- a + c\right)^{n}}{- 2 a n^{2} - 2 a n + 2 c n^{2} + 2 c n} - \frac{c^{2} n^{2} \left(- b + c\right)^{n}}{- 2 b n^{2} - 2 b n + 2 c n^{2} + 2 c n} + \frac{c^{2} n \left(- a + c\right)^{n}}{- 2 a n^{2} - 2 a n + 2 c n^{2} + 2 c n} - \frac{c^{2} n \left(- b + c\right)^{n}}{- 2 b n^{2} - 2 b n + 2 c n^{2} + 2 c n} - \frac{c n^{2} \left(- a + c\right)^{n}}{2 n^{2} + 2 n} + \frac{c n^{2} \left(- b + c\right)^{n}}{2 n^{2} + 2 n} - \frac{c n \left(- a + c\right)^{n}}{2 n^{2} + 2 n} + \frac{c n \left(- b + c\right)^{n}}{2 n^{2} + 2 n} + \frac{2 c \left(- a + c\right)^{n}}{2 n^{2} + 2 n} - \frac{2 c \left(- b + c\right)^{n}}{2 n^{2} + 2 n} & \text{otherwise} \end{cases}$

```python

```

```python

```

```python

```

```python

```


- - - -
# Mechpy Tutorials
a mechanical engineering toolbox

source code - https://github.com/nagordon/mechpy
documentation - https://nagordon.github.io/mechpy/web/

- - - -

Neal Gordon
2017-02-20

- - - -

## Solving a second order differential eqaution symbolically with python

```python
# setup
from sympy import Eq, pi
import sympy as sp
import matplotlib.pyplot as plt
from numpy import linspace
sp.init_printing(use_latex='mathjax')
get_ipython().magic('matplotlib inline') # inline plotting
```

```python
t,k,m,c = sp.symbols('t,k,m,c')
x = sp.Function('x') # (t)
```

```python
k_ = 1e3  # spring constant, kN/m
m_ = 50 # mass, Kg
c_ = 3  # damping coefficient
```

```python
ode = k*x(t) + m*c*x(t).diff(t,1) + m*x(t).diff(t,2)
Eq(ode)
```

$$c m \frac{d}{d t} x{\left (t \right )} + k x{\left (t \right )} + m \frac{d^{2}}{d t^{2}}  x{\left (t \right )} = 0$$

```python
ode_sol = sp.dsolve(ode)
ode_sol
```

$$x{\left (t \right )} = C_{1} e^{\frac{t}{2} \left(- c - \frac{1}{m} \sqrt{m \left(c^{2} m - 4 k\right)}\right)} + C_{2} e^{\frac{t}{2} \left(- c + \frac{1}{m} \sqrt{m \left(c^{2} m - 4 k\right)}\right)}$$

```python
x0 = 1.0
v0 = 0

bc1 = Eq(ode_sol.lhs.subs(x(t),x0), ode_sol.rhs.subs(t,0))
bc2 = Eq(ode_sol.lhs.subs(x(t),v0), ode_sol.rhs.diff(t).subs(t,0))

C_eq = {bc1,bc2}
C_eq
```

$$\left\{0 = C_{1} \left(- \frac{c}{2} - \frac{1}{2 m} \sqrt{m \left(c^{2} m - 4 k\right)}\right) + C_{2} \left(- \frac{c}{2} + \frac{1}{2 m} \sqrt{m \left(c^{2} m - 4 k\right)}\right), 1.0 = C_{1} + C_{2}\right\}$$

```python
known_params = {m,c,k,t}
const = ode_sol.free_symbols - known_params
const
```

$$\left\{C_{1}, C_{2}\right\}$$

```python
Csol = sp.solve(C_eq,const)
Csol
```

$$\left \{ C_{1} : - \frac{0.5 c m}{\sqrt{c^{2} m^{2} - 4.0 k m}} + 0.5, \quad C_{2} : \frac{0.5 c m}{\sqrt{c^{2} m^{2} - 4.0 k m}} + 0.5\right \}$$

```python
ode_sol = ode_sol.subs(Csol)
ode_sol
```

$$x{\left (t \right )} = \left(- \frac{0.5 c m}{\sqrt{c^{2} m^{2} - 4.0 k m}} + 0.5\right) e^{\frac{t}{2} \left(- c - \frac{1}{m} \sqrt{m \left(c^{2} m - 4 k\right)}\right)} + \left(\frac{0.5 c m}{\sqrt{c^{2} m^{2} - 4.0 k m}} + 0.5\right) e^{\frac{t}{2} \left(- c + \frac{1}{m} \sqrt{m \left(c^{2} m - 4 k\right)}\right)}$$

```python
ode_sol = ode_sol.subs({m:m_, c:c_, k:k_})
ode_sol
```

$$x{\left (t \right )} = \left(0.5 + 0.178017248729078 i\right) e^{\frac{t}{2} \left(-3 - 8.42614977317636 i\right)} + \left(0.5 - 0.178017248729078 i\right) e^{\frac{t}{2} \left(-3 + 8.42614977317636 i\right)}$$

```python
#sp.plot(ode_sol.rhs, (t,0,5)) ;

xfun = sp.lambdify(t,ode_sol.rhs, "numpy")
vfun = sp.lambdify(t,sp.diff(ode_sol.rhs), "numpy")

t = linspace(0,5,1000)

fig, ax1 = plt.subplots(figsize=(12,8))
ax2 = ax1.twinx()
ax1.plot(t,xfun(t),'b',label = r'$x (mm)$', linewidth=2.0)
ax2.plot(t,vfun(t),'g--',label = r'$\dot{x} (m/sec)$', linewidth=2.0)
ax2.legend(loc='lower right')
ax1.legend()
ax1.set_xlabel('time , sec')
ax1.set_ylabel('disp (mm)',color='b')
ax2.set_ylabel('velocity (m/s)',color='g')
plt.title('Mass-Spring System with $v_0=0.1%f' % (v0))
plt.grid()
plt.show()
```

## Solving a second order differential equation numerically

```python
from scipy.integrate import odeint
import matplotlib.pyplot as plt
from matplotlib.pyplot import plot,xlabel,ylabel,title,legend,figure,subplots
from numpy import cos, pi, arange, sqrt, pi, array
get_ipython().magic('matplotlib inline') # inline plotting
```

```python
def MassSpringDamper(state,t):
    '''
    k=spring constant, Newtons per metre
    m=mass, Kilograms
    c=dampign coefficient, Newton*second / meter

    for a mass, spring, damper
        xdd = -k*x/m -c*xd
    '''

    k=1e3  # spring constant, kN/m
    m=50 # mass, Kg
    c=3  # damping coefficient
    # unpack the state vector
    x,xd = state # displacement,x and velocity x'
    # compute acceleration xdd = x''
    xdd = -k*x/m -c*xd
    return [xd, xdd]

```

```python
x0 = 1.0
v0 = 0
state0 = [x0, v0]  #initial conditions [x0 , v0]  [m, m/sec]
ti = 0.0  # initial time
tf = 5.0  # final time
step = 0.001  # step
t = arange(ti, tf, step)
state = odeint(MassSpringDamper, state0, t)
x = array(state[:,[0]])
xd = array(state[:,[1]])
```

```python
# Plotting displacement and velocity
plt.rcParams['figure.figsize'] = (12, 8)
plt.rcParams['font.size'] = 14

fig, ax1 = plt.subplots()
ax2 = ax1.twinx()
ax1.plot(t,x,'b',label = r'$x (mm)$', linewidth=2.0)
ax2.plot(t,xd,'g--',label = r'$\dot{x} (m/sec)$', linewidth=2.0)
ax2.legend(loc='lower right')
ax1.legend()
ax1.set_xlabel('time , sec')
ax1.set_ylabel('disp (mm)',color='b')
ax2.set_ylabel('velocity (m/s)',color='g')
plt.title('Mass-Spring System with $v_0=%0.1f$' % (v0))
plt.grid()
plt.show()
```

```python

```


Lasers passively mode-locked by means of saturable absorbers have a tendency to Q-switched mode-locking (QML). Usually, this is not the desired mode of operation and has to be avoided by careful design. This is not always easy, especially not for the low pulse energies typically associatedd with very high pulse-repetition rates.

In the following, I summarize the dynamics of such a laser and show the criterion for stable mode locking (i.e. not QML).

# Laser Dynamics

The following differential equation describes the dynamics of a passively mode-locked (see, e.g., [C. Hönninger et al., J. Opt. Soc. Am. B/Vol. 16, No. 1, 1999] [honninger1999]):

$$\begin{align*}
\dot{P} &= \frac{g - l - q_P(S)}{T_R} P \\
\dot{g} &= -\frac{g}{\tau_L} + \frac{\eta \cdot P_P}{E_{sat,L}} - \frac{P \cdot g}{E_{sat,L}} \\
q_P(S) &= \frac{\Delta R}{S} \left(1 - e^{-S} \right)
\end{align*}$$,

where $S = \frac{E_P}{E_{sat,A}} = \frac{P \cdot T_R}{E_{sat,A}}$.

Let's set this up in sympy but let's - for the time being - ignore that $q_P$ is a function of $P(t)$:

[honninger1999]: http://www.ulp.ethz.ch/content/dam/ethz/special-interest/phys/quantum-electronics/ultrafast-laser-physics-dam/research/Overview2005/90.pdf

```
import sympy as sym
from IPython.display import display
sym.init_printing(use_latex='mathjax')

# Variables
P, g, PP, qP = sym.symbols('P g P_P q_P', real=True)
l = sym.symbols('l', real=True)
DR = sym.Symbol('\Delta R', real=True)
EsatL, tauL, TR, eta, EsatA = sym.symbols('E_{satL} tau_L T_R eta E_{satA}',
                                          real=True)

# Differential Equations
laserEq = [sym.Eq((g - l - qP) / TR * P),
           sym.Eq(-g / tauL + eta * PP / EsatL - P * g / EsatL)]
display(*laserEq)
```

$$\frac{P}{T_{R}} \left(g - l - q_{P}\right) = 0$$

$$- \frac{g}{\tau_{L}} - \frac{P g}{E_{{satL}}} + \frac{P_{P} \eta}{E_{{satL}}} = 0$$

# Steady-State

Next, we linearize these equations around their steady-state. The steady-state gain $g_{st}$ and intra-cavity power $P_{st}$ are:

```
PP0 = sym.Symbol('P_{P0}')
gst_, Pst_ = sym.solve(laserEq, (g, P))[1]

Pst_ = Pst_.subs(PP, PP0)

display(sym.Eq(sym.Symbol('g_{st}'), gst_),
        sym.Eq(sym.Symbol('P_{st}'), Pst_.subs(g, sym.Symbol('g_{st}'))))
```

$$g_{{st}} = l + q_{P}$$

$$P_{{st}} = \frac{1}{\tau_{L} \left(l + q_{P}\right)} \left(- E_{{satL}} l - E_{{satL}} q_{P} + P_{{P0}} \eta \tau_{L}\right)$$

To determine $q_P$, we multiply the $P_{st}$ equation by $g_{st}$ on both sides and substitute $g_{st}$:

```
qPEq = sym.Eq(P * gst_, Pst_ * g)
qPEq.subs(P, sym.Symbol('P_{st}'))
```

$$P_{{st}} \left(l + q_{P}\right) = \frac{g}{\tau_{L} \left(l + q_{P}\right)} \left(- E_{{satL}} l - E_{{satL}} q_{P} + P_{{P0}} \eta \tau_{L}\right)$$

Now we re-write both sides as functions of $S$ to obtain an equation for the steady-state $S$ parameter:

```
S = sym.symbols('S', real=True)
qPEq = qPEq.subs(qP, DR / S * (1 - sym.exp(-S))).subs(P, S * EsatA / TR)
sym.simplify(qPEq)
```

$$\frac{E_{{satA}}}{T_{R}} \left(S l e^{S} + \Delta R \left(e^{S} - 1\right)\right) e^{- S} = - \frac{g \left(E_{{satL}} \Delta R \left(e^{S} - 1\right) + S \left(E_{{satL}} l - P_{{P0}} \eta \tau_{L}\right) e^{S}\right)}{\tau_{L} \left(S l e^{S} + \Delta R \left(e^{S} - 1\right)\right)}$$

Unfortunately, it turns out that this equation is rather hard to solve symbolically:

```
sym.solve(qPEq, S)
```

# Linearization

Next, we rewrite the laser equations in terms of deviations from the steady-state:

$$\begin{align*}
P &= P_{st} + \delta P \\
g &= g_{st} + \delta g \\
P_P &= P_{P,0} + \delta P_P
\end{align*}$$

```
dP = sym.Symbol('\delta P', real=True)
dg = sym.Symbol('\delta g', real=True)
dPP = sym.Symbol('\delta P_P', real=True)
dqPdEP = sym.Symbol("q'_P", real=True)
Pst, gst = sym.symbols('P_{st} g_{st}', real=True)

deviationEq = ['', '']
deviationEq[0] = sym.Eq(sym.Symbol('\delta Pdot'), laserEq[0].lhs)
deviationEq[1] = sym.Eq(sym.Symbol('\delta gdot'), laserEq[1].lhs)

deviationEq = [eq.subs([(P, Pst + dP),
                        (g, gst + dg),
                        (PP, PP0 + dPP),
                        (qP, qP + dP * TR * dqPdEP)]) for eq in deviationEq]
display(*sym.simplify(deviationEq))
```

Next, we drop all higher order term. Unfortunately, sympy does not (yet?) support multivariate series expansion. Therefore, we solve the problem by introducing a helper variable $\delta x$ which is subsequently dropped.

```
dx = sym.Symbol('\delta x', real=True)
lin = [eq.rhs for eq in deviationEq]
lin = [expr.subs([(dg, dg * dx), (dP, dP * dx)]) for expr in lin]
lin = [expr.series(dx, x0=0, n=2).removeO().subs(dx, 1) for expr in lin]
lin = [sym.collect(sym.collect(eq, dP), dg) for eq in lin]
linEq = [sym.Eq(sym.Symbol('\delta Pdot'), lin[0]),
         sym.Eq(sym.Symbol('\delta gdot'), lin[1])]
linEq
```

There we are. However, this does not look linear. There seem to be unexpected constant terms. It turns out that these are zero due to the steady-state condition:

```
# Replace steady-state symbols with their corresponding expressions:
linEq[0] = sym.simplify(linEq[0].subs(gst, gst_))
linEq[1] = linEq[1].subs([(Pst, Pst_), (gst, gst_)])

# Rewrite in steady-state symbols for clarity:
linEq[1] = linEq[1].subs(PP0, sym.solve(sym.Eq(Pst, Pst_), PP0)[0])
linEq[1] = linEq[1].subs(l, sym.solve(sym.Eq(gst, gst_), l)[0])
linEq[1] = sym.simplify(linEq[1])

linEq
```

Now, we can rewrite this in handy matrix-notation in state-space form as follows:

```
A = sym.zeros(2, 2)
for i in range(2):
    for j in range(2):
        col = [dP, dg][j]
        A[i, j] = sym.collect(linEq[i].rhs.expand(), col, evaluate=False)[col]

B = sym.zeros(2, 1)
for i in range(2):
    try:
        B[i, 0] = sym.collect(linEq[i].rhs.expand(), dPP, evaluate=False)[dPP]
    except KeyError:
        B[i, 0] = 0

x = sym.MatrixSymbol('x', 2, 1)
u = sym.MatrixSymbol('u', 1, 1)

stateEq = sym.Eq(sym.MatrixSymbol('xdot', 2, 1), A * x + B * u)
stateEq
```

where $\vec{x} = \left[ \delta P, \delta g \right]^T$ and $\vec u = [\delta P_P]$. And to be complete, we add the output equation:

```
y = sym.MatrixSymbol('y', 1, 1)
Toc = sym.Symbol('T_{oc}', real=True)
C = sym.Matrix([[Toc, 0]])
outputEq = sym.Eq(y, C*x)
outputEq
```

where $\vec y$ is $\left[ P_{out} \right]$.

# Stability

It is now straightforward to discuss the stability of the system at its steady-state. The system is stable if the eigenvalue of $A$ all are in the left half of the complex plane. The eigenvalues of a 2 x 2 matrix

$$
\left( \begin{array}{ccc}
\alpha & \beta \\
-\gamma & -\epsilon
\end{array} \right)
$$

are:

```
alpha, beta, gamma, epsilon = sym.symbols('alpha beta gamma epsilon', real=True)
M = sym.Matrix([[alpha, beta], [-gamma, -epsilon]])
M.eigenvals()
```

In the case of solid-state lasers, the radicand is negative, leading to relaxation _oscillations_. If $\alpha - \epsilon < 0$, these oscillations are damped, and the laser is stably mode locking. This conditions translates to

```
A[0, 0] + A[1, 1] < 0
```

That's it for now. As soon as I understand limits and assumptions in sympy I may add the useful simplification for strongly saturated absorbers. The result will be:

$$ E_P^2 > E_{satL} E_{satA} \Delta R $$

Passively Mode-Locked Laser - Stability.ipynb


# Normalizing flows in PyTorch

One of the key in modern generative models is to find ways of optimizing the probability distribution of a given set of data. The recent idea of *Normalizing Flows* [[1](#reference1),[2](#reference)] addresses this problem and be able to rely on richer probability distributions. The main idea is to start from a simple probability distribution and approximate a complex multimodal density by *transforming* the simpler density through a sequence of invertible nonlinear transforms. To fully understand this blazing tool, we will see in this tutorial

1. The new [PyTorch distributions](#distribs) module and how to use it
2. How transforming a distribution is expressed as a [change of variables](#change) leading to a flow
3. How we can [chain multiple transforms](#chaining) leading to the overall framework of normalizing flows
4. Understanding the original [planar flow](#planar), its parameters and how to implement it
5. Defining [learnable flows](#learning) and performing optimization on a target density

### PyTorch distributions

In this tutorial, we are going to rely on the novel [PyTorch distributions module](https://pytorch.org/docs/stable/_modules/torch/distributions/), which is defined in `torch.distributions`. Most notably, we are going to rely both on the `Distribution` and `Transform` objects.

```python
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.distributions as distrib
import torch.distributions.transforms as transform
# Imports for plotting
import numpy as np
import matplotlib.pyplot as plt
from helper_plot import hdr_plot_style
hdr_plot_style()
# Define grids of points (for later plots)
x = np.linspace(-4, 4, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])
```

Inside this toolbox, we can already find some of the major probability distributions that we are used to deal with

```python
p = distrib.Normal(loc=0, scale=1)
p = distrib.Bernoulli(probs=torch.tensor([0.5]))
p = distrib.Beta(concentration1=torch.tensor([0.5]), concentration0=torch.tensor([0.5]))
p = distrib.Gamma(concentration=torch.tensor([1.0]), rate=torch.tensor([1.0]))
p = distrib.Pareto(alpha=torch.tensor([1.0]), scale=torch.tensor([1.0]))
```

The interesting aspect of these `Distribution` objects is that we can both obtain some samples from it through the `sample` (or `sample_n`) function, but we can also obtain the analytical density at any given point through the `log_prob` function

```python
# Based on a normal
n = distrib.Normal(0, 1)
# Obtain some samples
samples = n.sample((1000, ))
# Evaluate true density at given points
density = torch.exp(n.log_prob(torch.Tensor(x))).numpy()
# Plot both samples and density
fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(15, 4))
ax1.hist(samples, 50, alpha=0.8);
ax1.set_title('Empirical samples', fontsize=18);
ax2.plot(x, density); ax2.fill_between(x, density, 0, alpha=0.5)
ax2.set_title('True density', fontsize=18);
```

## Transforming distributions

### Change of variables and flow

In order to transform a probability distribution, we can perform a *change of variable*. As we are interested in probability distributions, we need to *scale* our transformed density so that the total probability still sums to one. This is directly measured with the determinant of our transform.

Let $\mathbf{z}\in\mathcal{R}^d$ be a random variable with distribution $q(\mathbf{z})$ and $f:\mathcal{R}^d\rightarrow\mathcal{R}^d$ an invertible smooth mapping (meaning that $f^{-1} = g$ and $g\circ f(\mathbf{z})=\mathbf{z}'$. We can use $f$ to transform $\mathbf{z}\sim q(\mathbf{z})$. The resulting random variable $\mathbf{z}'=f(\mathbf{z})$ has the following probability distribution

$$
q(\mathbf{z}')=q(\mathbf{z})\left|\text{ det}\frac{\delta f^{-1}}{\delta \mathbf{z}'}\right| = q(\mathbf{z})\left|\text{ det}\frac{\delta f}{\delta \mathbf{z}}\right|^{-1}
\tag{1}
$$

where the last equality is obtained through both the inverse function theorem [1] and the property of Jacobians of invertible functions. Therefore, we can transform probability distributions with this property.

Fortunately, this can be easily implemented in PyTorch with the `Transform` classes, that already defines some basic probability distribution transforms. For instance, if we define $\mathbf{z}\sim q_0(\mathbf{z})=\mathcal{N}(0, 1)$, we can apply the transform $\mathbf{z}'=exp(\mathbf{z})$ so that $\mathbf{z}'\sim q_1(\mathbf{z}')$

```python
q0 = distrib.Normal(0, 1)
exp_t = transform.ExpTransform()
q1 = distrib.TransformedDistribution(q0, exp_t)
samples_q0 = q0.sample((int(1e4),))
samples_q1 = q1.sample((int(1e4),))
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 4))
ax1.hist(samples_q0, 50, alpha=0.8);
ax1.set_title('$q_0 = \mathcal{N}(0,1)$', fontsize=18);
ax2.hist(samples_q1, 50, alpha=0.8, color='g');
ax2.set_title('$q_1=exp(q_0)$', fontsize=18);
```

But remember as the objects `q0` and `q1` are defined as `Distribution`, we can actually observe their true densities instead of just empirical samples

```python
hdr_plot_style()
x2 = np.linspace(-0.5, 7.5, 1000)
q0_density = torch.exp(q0.log_prob(torch.Tensor(x))).numpy()
q1_density = torch.exp(q1.log_prob(torch.Tensor(x2))).numpy()
fig, (ax1, ax2) = plt.subplots(1, 2, sharex=True, figsize=(15, 5))
ax1.plot(x, q0_density); ax1.fill_between(x, q0_density, 0, alpha=0.5)
ax1.set_title('$q_0 = \mathcal{N}(0,1)$', fontsize=18);
ax2.plot(x2, q1_density, color='g'); ax2.fill_between(x2, q1_density, 0, alpha=0.5, color='g')
ax2.set_title('$q_1=exp(q_0)$', fontsize=18);
fig.savefig('transform.pdf')
```

What we obtain here with `q1` is actually the `LogNormal` distribution. Interestingly, several distributions in the `torch.distributions` module are already defined based on `TransformedDistribution`. You can convince yourself of that by lurking in the code of the [`torch.distributions.LogNormal`](https://pytorch.org/docs/stable/_modules/torch/distributions/log_normal.html#LogNormal)

### Chaining transforms (normalizing flows)

Now, if we start with a random vector $\mathbf{z}_0$ with distribution $q_0$, we can apply a series of mappings $f_i$, $i \in 1,\cdots,k$ with $k\in\mathcal{N}^{+}$ and obtain a normalizing flow. Hence, if we apply $k$ normalizing flows, we obtain a chain of change of variables

$$
\mathbf{z}_k=f_k\circ f_{k-1}\circ...\circ f_1(\mathbf{z}_0)
\tag{2}
$$

Therefore the distribution of $\mathbf{z}_k\sim q_k(\mathbf{z}_k)$ will be given by

   $$
   \begin{align}
   q_k(\mathbf{z}_k) &= q_0(f_1^{-1} \circ f_{2}^{-1} \circ ... \circ f_k^{-1}(\mathbf{z}_k))\prod_{i=1}^k\left|\text{det}\frac{\delta f^{-1}_i}{\delta\mathbf{z}_{i}}\right|\\
   &= q_0(\mathbf{z_0})\prod_{i=1}^k\left|\text{det}\frac{\delta f_i}{\delta\mathbf{z}_{i-1}}\right|^{-1}
   \end{align}
   \tag{3}
   $$

where we compute the determinant of the Jacobian of each normalizing flow (as explained in the previous section). This series of transformations can transform a simple probability distribution (e.g. Gaussian) into a complicated multi-modal one. As usual, we will rely on log-probabilities to simplify the computation and obtain

$$
\text{log} q_K(\mathbf{z}_k) = \text{log} q_0(\mathbf{z}_0) - \sum_{i=1}^{k} \text{log} \left|\text{det}\frac{\delta f_i}{\delta\mathbf{z}_{i-1}}\right|
\tag{4}
$$

To be of practical use, however, we can consider only transformations whose determinants of Jacobians are easy to compute. Of course, we can perform any amount of combined transformations, and it also works with multivariate distributions. Here, this is demonstrated by transforming a `MultivariateNormal` successively with an `ExpTransform` and `AffineTransform`. (Note that the final distribution `q2` is defined as a `TransformedDistribution` directly with a *sequence* of transformations)

```python
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
# Define an affine transform
f1 = transform.ExpTransform()
q1 = distrib.TransformedDistribution(q0, f1)
# Define an additional transform
f2 = transform.AffineTransform(2, torch.Tensor([0.2, 1.5]))
# Here I define on purpose q2 as a sequence of transforms on q0
q2 = distrib.TransformedDistribution(q0, [f1, f2])
# Plot all these lads
fig, (ax1, ax2, ax3) = plt.subplots(1, 3, figsize=(15, 5))
ax1.hexbin(z[:,0], z[:,1], C=torch.exp(q0.log_prob(torch.Tensor(z))), cmap='rainbow')
ax1.set_title('$q_0 = \mathcal{N}(\mathbf{0},\mathbb{I})$', fontsize=18);
ax2.hexbin(z[:,0], z[:,1], C=torch.exp(q1.log_prob(torch.Tensor(z))), cmap='rainbow')
ax2.set_title('$q_1=exp(q_0)$', fontsize=18);
ax3.hexbin(z[:,0], z[:,1], C=torch.exp(q2.log_prob(torch.Tensor(z))), cmap='rainbow')
ax3.set_title('$q_2=Affine(exp(q_0))$', fontsize=18);
```

## Normalizing flows

Now, we are interested in normalizing flows as we could define our own flows. And, most importantly, we could optimize the parameters of these flow in order to fit complex and richer probability distributions. We will see how this plays out by trying to implement the *planar flow* proposed in the original paper by Rezende [1].

### Planar flow

 A planar normalizing flow is defined as a function of the form

   $$
   f(\mathbf{z})=\mathbf{z}+\mathbf{u}h(\mathbf{w}^T\mathbf{z}+b)
   \tag{5}
   $$

   where $\mathbf{u}\in\mathbb{R}^D$ and $\mathbf{w}\in\mathbb{R}^D$ are vectors (called here scale and weight), $b\in\mathbb{R}$ is a scalar (bias) and $h$ is an activation function. These transform functions are chosen depending on the fact that
1. the determinant of their Jacobian can be computed in linear time
2. the transformation is invertible (under usually mild conditions only)

As shown in the paper, for the planar flow, the determinant of the Jacobian can be computed in $O(D)$ time by relying on the matrix determinant lemma

$$
\psi(\mathbf{z})=h'(\mathbf{w}^T\mathbf{z}+b)\mathbf{w}
\tag{6}
$$

$$
\left|\text{det}\frac{\delta f}{\delta\mathbf{z}}\right| = \left|\text{det}\left(\mathbf{I}+\mathbf{u}\psi(\mathbf{z})^{T}\right)\right|=\left|1+\mathbf{u}^T\psi(\mathbf{z})\right|
\tag{7}
$$

Therefore, we have all definitions that we need to implement this flow as a `Transform` object. Note that here the non-linear activation function $h$ is selected as a $tanh$. Therefore the derivative $h'$ is $1-tanh(x)^2$

```python
class PlanarFlow(transform.Transform):

    def __init__(self, weight, scale, bias):
        super(PlanarFlow, self).__init__()
        self.bijective = False
        self.weight = weight
        self.scale = scale
        self.bias = bias

    def _call(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return f_z

    def log_abs_det_jacobian(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return abs_log_det
```

As before, we can witness the effect of this transform on a given `MultivariateNormal` distribution. You should note here that I am using the density estimation for `q0`, but only display empirical samples from `q1`.

```python
w = torch.Tensor([[3., 0]])
u = torch.Tensor([[2, 0]])
b = torch.Tensor([0])
q0 = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
flow_0 = PlanarFlow(w, u, b)
q1 = distrib.TransformedDistribution(q0, flow_0)
q1_samples = q1.sample((int(1e6), ))
# Plot this
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 5))
ax1.hexbin(z[:,0], z[:,1], C=torch.exp(q0.log_prob(torch.Tensor(z))), cmap='rainbow')
ax1.set_title('$q_0 = \mathcal{N}(\mathbf{0},\mathbb{I})$', fontsize=18);
ax2.hexbin(q1_samples[:,0], q1_samples[:,1], cmap='rainbow')
ax2.set_title('$q_1=planar(q_0)$', fontsize=18);
```

The reason for this is that the `PlanarFlow` is not invertible in all regions of the space. However, if we recall the mathematical reasoning of the previous section, we can see how the change of variables plays out if we are able to compute the determinant of the Jacobian of this transform.

```python
q0_density = torch.exp(q0.log_prob(torch.Tensor(z)))
# Apply our transform on coordinates
f_z = flow_0(torch.Tensor(z))
# Obtain our density
q1_density = q0_density.squeeze() / np.exp(flow_0.log_abs_det_jacobian(torch.Tensor(z)).squeeze())
# Plot this
fig, (ax1, ax2) = plt.subplots(1, 2, sharey=True, figsize=(15, 5))
ax1.hexbin(z[:,0], z[:,1], C=q0_density.numpy().squeeze(), cmap='rainbow')
ax1.set_title('$q_0 = \mathcal{N}(\mathbf{0},\mathbb{I})$', fontsize=18);
ax2.hexbin(f_z[:,0], f_z[:,1], C=q1_density.numpy().squeeze(), cmap='rainbow')
ax2.set_title('$q_1=planar(q_0)$', fontsize=18);
```

So we were able to "split" our distribution and transform a unimodal gaussian into a multimodal distribution ! Pretty neat

### Visualizing parameters effects

Here, we provide a little toy example so that you can play around with the parameters of the flow in order to get a better understanding of how it operates. As put forward by Rezende [1], this flow is related to the hyperplane defined by $\mathbf{w}^{T}\mathbf{z}+b=0$ and transforms the original density by applying a series of contractions and expansions in the direction perpendicular to this hyperplane.

```python
id_figure=1
plt.figure(figsize=(16, 18))
for i in np.arange(5):
    # Draw a random hyperplane
    w = torch.rand(1, 2) * 5
    b = torch.rand(1) * 5
    for j in np.arange(5):
        # Different effects of scaling factor u on the same hyperplane (row)
        u = torch.Tensor([[((j < 3) and (j / 2.0) or 0), ((j > 2) and ((j - 2) / 2.0) or 0)]])
        flow_0 = PlanarFlow(w, u, b)
        q1 = distrib.TransformedDistribution(q0, flow_0)
        q1_samples = q1.sample((int(1e6), ))
        plt.subplot(5,5,id_figure)
        plt.hexbin(q1_samples[:,0], q1_samples[:,1], cmap='rainbow')
        plt.title("u=(%.1f,%.1f)"%(u[0,0],u[0,1]) + " w=(%d,%d)"%(w[0,0],w[0,1]) + ", " + "b=%d"%b)
        plt.xlim([-3, 3])
        plt.ylim([-3, 3])
        id_figure += 1
```

## Optimizing normalizing flows

Now that we have this magnificent tool, we would like to apply this in order to learn richer distributions and perform *inference*. Now, we have to deal with the fact that the `Transform` object is not inherently parametric and cannot yet be optimized similarly to other modules.

To do so, we will start by defining our own `Flow` class which can be seen both as a `Transform` and also a `Module`that can be optmized

```python
class Flow(transform.Transform, nn.Module):

    def __init__(self):
        transform.Transform.__init__(self)
        nn.Module.__init__(self)

    # Init all parameters
    def init_parameters(self):
        for param in self.parameters():
            param.data.uniform_(-0.01, 0.01)

    # Hacky hash bypass
    def __hash__(self):
        return nn.Module.__hash__(self)
```

Thanks to this little trick, we can use the same `PlanarFlow` class as before, that we put back here just to show that the only change is that it now inherits from the `Flow` class (with the small added bonus that now parameters of this flow are also registered in the `Module` interface)

```python
class PlanarFlow(Flow):

    def __init__(self, dim):
        super(PlanarFlow, self).__init__()
        self.weight = nn.Parameter(torch.Tensor(1, dim))
        self.scale = nn.Parameter(torch.Tensor(1, dim))
        self.bias = nn.Parameter(torch.Tensor(1))
        self.init_parameters()

    def _call(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return f_z

    def log_abs_det_jacobian(self, z):

        ######################
        # YOUR CODE GOES HERE
        ######################

        return abs_log_det
```

Now let's say that we have a given complex density that we aim to model through normalizing flows, such as the following one

```python
def density_ring(z):
    z1, z2 = torch.chunk(z, chunks=2, dim=1)
    norm = torch.sqrt(z1 ** 2 + z2 ** 2)
    exp1 = torch.exp(-0.5 * ((z1 - 2) / 0.8) ** 2)
    exp2 = torch.exp(-0.5 * ((z1 + 2) / 0.8) ** 2)
    u = 0.5 * ((norm - 4) / 0.4) ** 2 - torch.log(exp1 + exp2)
    return torch.exp(-u)

# Plot it
x = np.linspace(-5, 5, 1000)
z = np.array(np.meshgrid(x, x)).transpose(1, 2, 0)
z = np.reshape(z, [z.shape[0] * z.shape[1], -1])
plt.hexbin(z[:,0], z[:,1], C=density_ring(torch.Tensor(z)).numpy().squeeze(), cmap='rainbow')
plt.title('Target density', fontsize=18);
```

Now to approximate such a complicated density, we will need to chain multiple planar flows and optimize their parameters to find a suitable approximation. We can do exactly that like in the following (you can see that we start by a simple normal density and perform 16 successive planar flows)

```python
# Main class for normalizing flow
class NormalizingFlow(nn.Module):

    def __init__(self, dim, flow_length, density):
        super().__init__()
        biject = []
        for f in range(flow_length):
            biject.append(PlanarFlow(dim))
        self.transforms = transform.ComposeTransform(biject)
        self.bijectors = nn.ModuleList(biject)
        self.base_density = density
        self.final_density = distrib.TransformedDistribution(density, self.transforms)
        self.log_det = []

    def forward(self, z):
        self.log_det = []
        # Applies series of flows
        for b in range(len(self.bijectors)):
            self.log_det.append(self.bijectors[b].log_abs_det_jacobian(z))
            z = self.bijectors[b](z)
        return z, self.log_det

# Create normalizing flow
flow = NormalizingFlow(dim=2, flow_length=16, density=distrib.MultivariateNormal(torch.zeros(2), torch.eye(2)))
```

Now the only missing ingredient is the loss function that is simply defined as follows

```python
def loss(density, zk, log_jacobians):

    ######################
    # YOUR CODE GOES HERE
    ######################

    return loss
```

We can now perform optimization as usual by defining an optimizer, the parameters it will act on and eventually a learning rate scheduler

```python
import torch.optim as optim
# Create optimizer algorithm
optimizer = optim.Adam(flow.parameters(), lr=2e-3)
# Add learning rate scheduler
scheduler = optim.lr_scheduler.ExponentialLR(optimizer, 0.9999)
```

And now we perform the loop by sampling a batch (here of 512) from the reference Normal distribution, and then evaluating our loss with respect to the density we want to approximate.

```python
ref_distrib = distrib.MultivariateNormal(torch.zeros(2), torch.eye(2))
id_figure=2
plt.figure(figsize=(16, 18))
plt.subplot(3,4,1)
plt.hexbin(z[:,0], z[:,1], C=density_ring(torch.Tensor(z)).numpy().squeeze(), cmap='rainbow')
plt.title('Target density', fontsize=15);
# Main optimization loop
for it in range(10001):
    # Draw a sample batch from Normal
    samples = ref_distrib.sample((512, ))
    # Evaluate flow of transforms
    zk, log_jacobians = flow(samples)
    # Evaluate loss and backprop
    optimizer.zero_grad()
    loss_v = loss(density_ring, zk, log_jacobians)
    loss_v.backward()
    optimizer.step()
    scheduler.step()
    if (it % 1000 == 0):
        print('Loss (it. %i) : %f'%(it, loss_v.item()))
        # Draw random samples
        samples = ref_distrib.sample((int(1e5), ))
        # Evaluate flow and plot
        zk, _ = flow(samples)
        zk = zk.detach().numpy()
        plt.subplot(3,4,id_figure)
        plt.hexbin(zk[:,0], zk[:,1], cmap='rainbow')
        plt.title('Iter.%i'%(it), fontsize=15);
        id_figure += 1
```

That concludes this tutorial ! In the next one we will see how to implement more complicated flows and how this can fit in a global inference framework

### References

[1] Rezende, Danilo Jimenez, and Shakir Mohamed. "Variational inference with normalizing flows." _arXiv preprint arXiv:1505.05770_ (2015). [link](http://arxiv.org/pdf/1505.05770)

[2] Kingma, Diederik P., Tim Salimans, and Max Welling. "Improving Variational Inference with Inverse Autoregressive Flow." _arXiv preprint arXiv:1606.04934_ (2016). [link](https://arxiv.org/abs/1606.04934)

[3] Germain, Mathieu, et al. "Made: masked autoencoder for distribution estimation." International Conference on Machine Learning. 2015.

### Inspirations and resources

https://blog.evjang.com/2018/01/nf1.html
https://github.com/ex4sperans/variational-inference-with-normalizing-flows
https://akosiorek.github.io/ml/2018/04/03/norm_flows.html
https://github.com/abdulfatir/normalizing-flows


# **Tutorial on Text as Data**: Text Representation

Source:  [https://github.com/d-insight/code-bank.git](https://github.com/d-insight/code-bank.git)
License: [MIT License](https://opensource.org/licenses/MIT). See open source [license](LICENSE) in the Code Bank repository.

---

## Overview

There are different ways to represent text data. In this notebook, we will dig deeper into the subject and look at different approaches to transform the text into its respective numerical representation.

## Part 1: Text representation with TF-IDF

In the previous notebook, we used a very simple solution to represent text data: by counting the most common words. One of the drawbacks of this approach is that all terms receive the same weights.

A better strategy is to weight words according to their occurrence in the document. The underline idea is that words that appear in every document probably do not bring any particular meaning to the topic.

To do that we normalize the term frequency of the world by the inverse of the document frequency. This might sound complicated at first. In this section, we will break down the TF-IDF algorithm step by step on a simple example.

> ☝scikit-learn provides a function to easily compute TF-IDF weights. At the end of our custom solution, we will compare the two solutions to make sure they return the same value.

Assume we are given the following reviews and we want to compute the TF-IDF weights.

```python
review_0 = "the pizza was very very good quality"
review_1 = "the pasta was very superior quality"
review_2 = "the gnocchi weren't that good"
```

Q1: First, using `scikit-learn`, compute the count term-vector matrix and save it into a DataFrame `df_count`.

```python
import pandas as pd
from sklearn.feature_extraction.text import CountVectorizer

cv = CountVectorizer(token_pattern="\S+")

reviews = [review_0, review_1, review_2]

count_cv = cv.fit_transform(reviews).toarray()

df_count = pd.DataFrame(count_cv, columns=cv.get_feature_names())
df_count
```

      gnocchi
      good
      pasta
      pizza
      quality
      superior
      that
      the
      very
      was
      weren't

      0
      0
      1
      0
      1
      1
      0
      0
      1
      2
      1
      0

      1
      0
      0
      1
      0
      1
      1
      0
      1
      1
      1
      0

      2
      1
      1
      0
      0
      0
      0
      1
      1
      0
      0
      1

Q2: Now, we need to compute the `idf` part. For each term, count the total number of documents that contain such a term (`document_freq`). In the literature, this is known as "document frequency".

```python
document_freq = (df_count != 0).sum()
document_freq
```

    gnocchi     1
    good        2
    pasta       1
    pizza       1
    quality     2
    superior    1
    that        1
    the         3
    very        2
    was         2
    weren't     1
    dtype: int64

Q3: Compute the `idf` part:

\begin{align}
idf(word_i) = ln(\frac{n+1}{document\_freq(word_i) + 1}) + 1
\end{align}

Example

\begin{align}
idf(very) = ln(\frac{3+1}{2 + 1}) + 1 = ln(4/3) + 1 \approx 1.29
\end{align}

```python
import numpy as np
n = df_count.shape[0]
idf = np.log((n + 1) / (document_freq + 1)) + 1
idf
```

    gnocchi     1.693147
    good        1.287682
    pasta       1.693147
    pizza       1.693147
    quality     1.287682
    superior    1.693147
    that        1.693147
    the         1.000000
    very        1.287682
    was         1.287682
    weren't     1.693147
    dtype: float64

Q4: Finnally, compute the TF-IDF (`tfidf`) weights

```python
tfidf = df_count * idf
tfidf
```

      gnocchi
      good
      pasta
      pizza
      quality
      superior
      that
      the
      very
      was
      weren't

      0
      0.000000
      1.287682
      0.000000
      1.693147
      1.287682
      0.000000
      0.000000
      1.0
      2.575364
      1.287682
      0.000000

      1
      0.000000
      0.000000
      1.693147
      0.000000
      1.287682
      1.693147
      0.000000
      1.0
      1.287682
      1.287682
      0.000000

      2
      1.693147
      1.287682
      0.000000
      0.000000
      0.000000
      0.000000
      1.693147
      1.0
      0.000000
      0.000000
      1.693147

Q5: Now we can compare our solution with scikit-learn and see if they are the same.

> ☝Note that by default scikit-learn normalize the output. To avoid that, we need to pass `norm=None`.

```python
from sklearn.feature_extraction.text import TfidfVectorizer

reviews = [review_0, review_1, review_2]

vectorizer = TfidfVectorizer(token_pattern="\S+", min_df=1, norm=None)

m = vectorizer.fit_transform(reviews).toarray()

tfidf_sklearn = pd.DataFrame(m, columns=vectorizer.get_feature_names())
tfidf_sklearn
```

      gnocchi
      good
      pasta
      pizza
      quality
      superior
      that
      the
      very
      was
      weren't

      0
      0.000000
      1.287682
      0.000000
      1.693147
      1.287682
      0.000000
      0.000000
      1.0
      2.575364
      1.287682
      0.000000

      1
      0.000000
      0.000000
      1.693147
      0.000000
      1.287682
      1.693147
      0.000000
      1.0
      1.287682
      1.287682
      0.000000

      2
      1.693147
      1.287682
      0.000000
      0.000000
      0.000000
      0.000000
      1.693147
      1.0
      0.000000
      0.000000
      1.693147

```python
tfidf_sklearn == tfidf
```

      gnocchi
      good
      pasta
      pizza
      quality
      superior
      that
      the
      very
      was
      weren't

      0
      True
      True
      True
      True
      True
      True
      True
      True
      True
      True
      True

      1
      True
      True
      True
      True
      True
      True
      True
      True
      True
      True
      True

      2
      True
      True
      True
      True
      True
      True
      True
      True
      True
      True
      True

Q6: Explain why the outputs of `CountVectorizer` and `TfidfVectorizer` are sparse matrix. What's the main advantage of using sparse representation?

**Answer**

Recall that the rows of the matrix represents the document (in this case reviews) whereas the columns are terms (i.e tokens/words). Clearly, not all documents contain all words of the vocabulary, and therefore most of the values are zero. Instead of returning the full matrix composed of many zeros, we return a special data structure (i.e a sparse matrix) containing only non-zero values plus the information regarding the position in the matrix. This way we can save a lot of memory.

Q7 (**homework**): Go back to Notebook 2, part 3 and compute again the topic, this time using `TfidfVectorizer` instead of `CountVectorizer`. Does the topics are better defined? (spoiler: **yes**).

## Part 2: Word2Vec

Word2vec is a two-layer neural networks to vectorize words. Its input is a text corpus and its output is a set of vectors, feature vectors that represent words in that corpus.

**Learning objective**: intuitively understand Word2Vec and be able to implement it

**Useful resources**:
 - [Word2vec paper: Distributed Representations of Words and Phrases and their Compositionality](https://papers.nips.cc/paper/5021-distributed-representations-of-words-and-phrases-and-their-compositionality.pdf)
 - [The Illustrated Word2vec](http://jalammar.github.io/illustrated-word2vec/)

Q1: By looking at the following image, can you explain the difference between the continuous-bag-of-words (CBOW) and skip-gram model architectures?

**Answer**

**CBOW**: given a context, predict the word

**Skip-gram**: given as input a single word, predict the context

Q2: We will now train a Word2Vec model using [Gensim](https://radimrehurek.com/gensim/). Gensim's Word2Vec expect as input a list of sentences where each sentence is a list of tokens as in the example below. Load `review_clean.csv` into a Pandas Dataframe `df`, transform it as a list of list of tokens and store it into a variable called `corpus`.

> ☝ For tokenization, you can simply use `string.split()`

```python
sentence_one = ["Resturant", "was", "ok", "."]
sentence_two = ["Restaurant", "was", "average"]

valid_gensim_input = [sentence_one, sentence_two]

print(valid_gensim_input)
```

    [['Resturant', 'was', 'ok', '.'], ['Restaurant', 'was', 'average']]

```python
df = pd.read_csv("./data/review_clean.csv")
df.head(2)
```

      text_clean
      text
      stars
      categories
      name
      address

      0
      Omar great !  diagnose correct cell phone issu...
      Omar is great!  He can diagnose and correct an...
      5.0
      Mobile Phones, Shopping, Telecommunications, M...
      T-Mobile
      16635 N Tatum Blvd, Ste 110

      1
      TERRIBLE ! ! ! went salon close proximity waxi...
      TERRIBLE!!! I went to this salon due to its cl...
      1.0
      Day Spas, Beauty & Spas, Nail Salons
      Pretty Nails
      1660 E Camelback Rd

```python
# Split into sentences
sentences_s = df['text_clean'].str.split(".").explode()

# Split each sentence into tokens
tokens_s = sentences_s.str.split()

# Concatenate all list togethers
corpus = list(tokens_s)
corpus[:2]
```

    [['Omar',
      'great',
      '!',
      'diagnose',
      'correct',
      'cell',
      'phone',
      'issue',
      '-',
      'professional',
      'patient',
      '-',
      'ideal',
      'buyers',
      '/',
      'customers',
      'limited',
      'technology',
      'knowledge'],
     ['Thank', 'Omar', 'help']]

Q3: Define and train the `word2vec` model. Set the embedding `size` to 300, train using the skip-gram model and set `min_count` to 5.

> ☝ Have a look at the [Gensim Word2Vec tutorial](https://radimrehurek.com/gensim/models/word2vec.html) for help

```python
from gensim.models import Word2Vec

w2v_model = Word2Vec(size=300, sg=1, min_count=5)

w2v_model.build_vocab(corpus, progress_per=10000)

w2v_model.train(corpus, total_examples=w2v_model.corpus_count, epochs=20, report_delay=1)
```

    (9089505, 11248120)

Q4: Once the model is trained we can extract the word embeddings and compare it. Execute the cell below to create a DataFrame that map selected words to their Word2Vec vector.

```python
list_of_words = ["pasta", "pizza", "spaghetti", "car", "insurance", "phone", "technology"]

dict_ = {}

for word in list_of_words:
    dict_[word] = [w2v_model.wv[word]]

df_vocab = pd.DataFrame(dict_).T.reset_index().rename(columns={0: 'vector', 'index': 'vocab'})
df_vocab
```

      vocab
      vector

      0
      pasta
      [0.16100411, -0.44462204, 0.09203769, 0.131131...

      1
      pizza
      [0.5975063, -0.54391915, 0.5388808, 0.19324102...

      2
      spaghetti
      [0.34720272, -0.29686236, 0.26158643, 0.001025...

      3
      car
      [0.3174404, -0.41663548, 0.5467024, 0.7372491,...

      4
      insurance
      [-0.38761958, -0.23813048, -0.011089393, 0.336...

      5
      phone
      [0.088869356, -0.2655629, 0.335964, 0.91946524...

      6
      technology
      [-0.09345005, -0.18767874, 0.06817056, 0.36209...

Q5: using Texthero `pca` and `scatterplot` functions, apply principal component analysis and visualize the vector space. Can you spot anything interesting?

```python
import texthero as hero

df_vocab['pca'] = hero.pca(df_vocab['vector'])
#hero.scatterplot(df_vocab, 'pca', color='vocab')

from matplotlib.pyplot import rcParams
rcParams['figure.figsize'] = 10, 8

import seaborn as sns; sns.set()
import seaborn
seaborn.scatterplot(x=df_vocab['pca'].str[0], y=df_vocab['pca'].str[1], hue=df_vocab['vocab']);

#hero.scatterplot(df, 'pca', color='stars');
```

## Part 3: Subwords and contextualized word embeddings

Word2Vec has been around for about a decade now. Word2Vec has some limitations, for instance it cannot handle words not present in the vocabulary used during training.

**Papers**:

- [Elmo: deep contextualized word representations](https://arxiv.org/pdf/1802.05365.pdf)
- [GloVe: Global Vectors for Word Representation](https://nlp.stanford.edu/pubs/glove.pdf)

**Resources and extra lectures**:

- [How is GloVe different from word2vec?](https://www.quora.com/How-is-GloVe-different-from-word2vec)
[ELMo vs. BERT vs. Word2vec vs. GloVe](https://www.quora.com/What-are-the-main-differences-between-the-word-embeddings-of-ELMo-BERT-Word2vec-and-GloVe)

Q1: What are the main differences between Word2Vec and FastText ?

**Answer**

- Word2vec treat words as the smallest unit to train on
- FastText treats each word as **composed of character ngrams**
- FastText generates better word embeddings for rare words
- FastText work with out–of–vocabulary (oov) words

Q2: What's the main difference between Word2vec/Glove and Elmo?

**Answer**

- Word2Vec (and Glove) word embeddings are context independent
- With Elmo, the same word in two different context will have different vectors
- Word2vec and Glove do not take into account word order in their training
- Once trained, we can discard model with Word2Vec and Glove but we need to keep it with Elmo.

We will now represent all review using Glove. To do so, we will make use of [Flair](https://github.com/flairNLP/flair/blob/master/resources/docs/TUTORIAL_3_WORD_EMBEDDING.md), a recently published python package that allow to easily represent any text using different embeddings.

Q3: Using `WordEmbeddings('glove')` represent the word `analytics`, `business` and `pizza`. Then compute the cosine distance between vectors. Which ones are closer?

```python
from scipy.spatial import distance

from flair.embeddings import WordEmbeddings
from flair.data import Sentence
glove_embedding = WordEmbeddings('glove')

analytics = Sentence('analytics')
glove_embedding.embed(analytics)
analytics_np = analytics[0].embedding.numpy()

business = Sentence('business')
glove_embedding.embed(business)
business_np = business[0].embedding.numpy()

pizza = Sentence('pizza')
glove_embedding.embed(pizza)
pizza_np = pizza[0].embedding.numpy()

print("distance analytics-business: ", distance.cosine(analytics_np, business_np))
print("distance analytics-pizza: ", distance.cosine(analytics_np, pizza_np))
```

    distance analytics-business:  0.6769274473190308
    distance analytics-pizza:  0.9892289051786065

Until now we represented only single words, but what if we want to represent documents? Here you are asked to represent 10'000 different reviews, and sort them with respect to their cosine distance to the first review.

Q4: First, define a function `embed_review` that given a review it returns the respective embed. For this part, you can use [Document Pool Embeddings](https://github.com/flairNLP/flair/blob/master/resources/docs/TUTORIAL_5_DOCUMENT_EMBEDDINGS.md) from Flair: `DocumentPoolEmbeddings([glove_embedding])`.
Test your function with the following review: "Restaurant was fabolous".

```python
from flair.embeddings import DocumentPoolEmbeddings

docpoolembed = DocumentPoolEmbeddings([glove_embedding])

def embed_review(review):
    s = Sentence(review)
    docpoolembed.embed(s)
    return s.embedding.numpy()

embed_review("Restaurant was fabolous")
```

    array([ 8.66933241e-02, -5.56396663e-01,  1.20346665e-01, -1.11973338e-01,
            1.04716666e-01,  3.58339958e-02,  4.94163364e-01,  4.51049991e-02,
            3.79222989e-01,  4.73299980e-01,  8.33370015e-02,  4.71833386e-02,
            3.73263329e-01, -2.21253335e-01,  2.71760017e-01,  3.98333259e-02,
            2.60616988e-01, -3.92656654e-01, -6.09496653e-01,  2.15656683e-01,
            3.42953324e-01,  1.41179338e-01,  9.83295068e-02, -5.61051071e-01,
            1.99090004e-01,  1.69324994e-01, -3.80899996e-01, -2.18599990e-01,
           -1.28083333e-01, -2.39233430e-02, -1.43140003e-01,  6.18003309e-01,
            2.43299320e-01,  5.81566632e-01,  6.24503382e-02,  2.65113324e-01,
           -5.40033318e-02,  1.77274346e-01,  2.21469685e-01, -4.31127310e-01,
            4.97676700e-01,  2.52333283e-02,  2.41698742e-01,  1.04423344e-01,
            5.22879660e-01,  2.40914330e-01, -2.45366693e-01, -1.21610999e-01,
            4.70633358e-01, -4.67902035e-01, -4.40006644e-01, -2.33179986e-01,
            1.93666723e-02,  5.97223341e-01, -7.86930025e-01, -1.58939993e+00,
           -4.39949661e-01, -1.84886649e-01,  4.01833326e-01,  3.98486704e-01,
           -1.67640015e-01,  5.07163346e-01,  8.07933286e-02,  7.87566602e-02,
            4.65440005e-01, -4.43812013e-01,  3.90460014e-01,  1.04683340e-01,
            9.49833319e-02,  1.43583313e-01, -5.01310349e-01,  2.47083351e-01,
           -2.21939925e-02,  2.21629992e-01,  8.97966623e-02,  3.05683643e-01,
            2.95543343e-01,  1.73456669e-01, -4.36293334e-01, -2.63754994e-01,
            1.26366302e-01,  6.09899946e-02, -2.18100667e-01,  3.79136652e-02,
           -4.67563361e-01, -4.22161013e-01, -3.01696688e-01, -3.03850025e-01,
           -5.28700054e-02,  5.39978326e-04,  3.06976676e-01, -3.80608678e-01,
            1.14586331e-01,  8.03600028e-02, -3.26580644e-01,  7.86633417e-02,
            1.99976657e-02, -3.71927023e-01,  4.55203295e-01,  3.51299942e-02],
          dtype=float32)

Q5: Apply to the `text` column your previously created method and store the result in a new column `embed`.

> ☝This operation might take a while, we use `tqdm` to show a progress bar.

```python
from tqdm import tqdm
tqdm.pandas()

df.reset_index(inplace=True)

df['embed'] = df['text_clean'].progress_apply(embed_review)
df['embed']
```

    /Library/Frameworks/Python.framework/Versions/3.7/lib/python3.7/site-packages/tqdm/std.py:668: FutureWarning: The Panel class is removed from pandas. Accessing it from the top-level namespace will also be removed in the next version
      from pandas import Panel
    100%|██████████| 10000/10000 [00:24<00:00, 405.18it/s]

    0       [-0.14973384, 0.105061956, 0.35441896, -0.0065...
    1       [0.0014385143, 0.14520112, 0.2159496, -0.34776...
    2       [-0.19049452, 0.20583253, 0.2609278, -0.240894...
    3       [-0.18839265, 0.33528236, 0.24974221, -0.26905...
    4       [-0.12573816, 0.31910342, 0.14125381, -0.14221...
                                  ...
    9995    [-0.22443968, 0.33964384, 0.3370281, -0.230378...
    9996    [-0.21270736, 0.21685344, 0.30420268, -0.26460...
    9997    [-0.108463414, 0.16915984, 0.37964147, -0.3303...
    9998    [-0.12619269, 0.06759783, 0.20094448, -0.12893...
    9999    [-0.03584042, 0.09587897, 0.33155766, -0.27944...
    Name: embed, Length: 10000, dtype: object

Q6: Sort the review by cosine distance w.r.t the first review and visualize on screen the first three reviews. Do they look similar?

> ☝ The function below compute the cosine similarity between a vector and every row of the given numpy matrix. You will need to use `np.array(list(df['embed']))` to transform the Pandas Series into a NumPy matrix.

```python
import scipy

def cosine_similarity(vector, matrix):
    """
    Compute the cosine distances between the vector and each row of the matrix.
    """

    print(matrix.shape)

    v = vector.reshape(1, -1)
    return scipy.spatial.distance.cdist(matrix, v, 'cosine').reshape(-1)
```

```python
first_review_vec = df.iloc[0]['embed']
df['cosine_similarity'] = cosine_similarity(first_review_vec, np.array(list(df['embed'])))

most_similar = df.sort_values('cosine_similarity')[:3]
for i, review in most_similar.iterrows():
    print(review['text'])
    print("\n")
```

    (10000, 100)
    Omar is great!  He can diagnose and correct any cell phone issue - professional and patient - ideal for buyers/customers with limited technology knowledge.  Thank you Omar for all your help.  Jana and Lorraine

    Just wanna quick thanks to the person repair my laptop, its working great and faster than ever before. Also, the information provided to me is the greatest thing and i know  how to keep up the machine up-to-date. Thanks again

    They person/company they sent out to provide a quote didn't even show up at the scheduled time.  No call or e-mail in advance.  Will never use this company if they can't even get that part right.  Embarrassed for them!

Q7: Look now at the farthest review. What's that about?

```python
df.sort_values('cosine_similarity')['text'].iloc[-1]
```

    '挺不错的 味道都挺好 食物卖相也不错 唯一遗憾蒸蛋没啦 跑来中国城吃饭 周围也蛮好停车的 点了一个面一个红油抄手 一个小菜 又送了一个木耳 酸酸辣辣的 很多地方的红油抄手都会偏甜 就感觉很不正宗 但是他们家的完全不会 所以觉得很好哦 然后面里的炒黄豆很好吃'


# Corollaries to the Axioms of Probability

Corollaries are results that can be proven from more fundamental theorems or properties. In this case, we are interested in what additional properties or relations we can develop for our probability measure, based on the Axioms of Probability.

Let $A \in \mathcal{F}$ and $B \in \mathcal{F}$.  Then the following
  properties of $P$ can be derived from the axioms and the
  mathematical structure of $\mathcal{F}$:

**Corollary 1.** Let $\overline{A}$ denote the complement of $A$; i.e., $\overline{A}$ contains every outcome in $S$ that is  not in $A$. Then

$$
P\left( \overline{A} \right) = 1 - P\left(A \right).
$$

*Proof:*

The proof uses Axioms II and III, as well as properties of sets. First, note that $A \cup \overline{A} = S$ and that $A$ and $\overline{A}$ are disjoint by definition. Then by Axioms II and III,
\begin{align*}
P(S) &= 1\\
P(A \cup \overline{A}) &=1 \\
P(A) + P(\overline{A}) &= 1 \\
P(A) &= 1 - P(\overline{A}).
\end{align*}

**Example**

A fair six-sided die is rolled two times and the top faces are recorded. What is the probability that neither roll is less than 3?

Let $E_i$ be the event that the outcome of roll $i$ is less than 3. Then we are asked to find $P\left(\overline{E_1} \cap \overline{E_2} \right)$.

By DeMorgan's rules,

$$
P\left(\overline{E_1} \cap \overline{E_2} \right) = P\left(\overline{E_1 \cup E_2} \right).
$$

We can apply Corollary 1 to get

$$
P\left(\overline{E_1 \cup E_2} \right) = 1 - P\left(E_1 \cup E_2 \right).
$$

But we already found the probability on the right-hand side to be 5/9 in {doc}`Axiomatic Probability<axiomatic-prob>`. Thus, probability we are looking for is $1-5/9=4/9$.

**Corollary 2.** $P(A)\le 1$

As previously noted, this restriction is not included in the axioms.

*Proof:*

By Corollary 1, we have

$$
P(A) = 1 - P(\overline{A}).
$$

By Axiom I, $P(\overline{A}) \ge 0$, so it must be that $P(A) \le 1$.

**Corollary 3.** $P(\emptyset)=0$

*Proof:*

By Corollary 1, we have

$$
P(\emptyset) = 1 - P(\overline{\emptyset}).
$$

But $\overline{\emptyset} =S$. Thus,
\begin{align}
P(\emptyset) &= 1 - P(S) \\
&= 1-1 \\
&=0
\end{align}

**Corollary 4.** If $A_1, A_2, \ldots, A_n$ are pairwise mutually
      exclusive, then

      \begin{eqnarray*}
        P\left(\bigcup_{k=1}^{n} A_k \right)= \sum_{k=1}^{n} P(A_k)
      \end{eqnarray*}

The proof is by induction and is omitted.

**Corollary 5.** $P(A \cup B) = P(A) +P(B) - P(A \cap B)$

*Proof*

The proof requires a bit of work with sets and applying Axiom III. It is based on the following Venn diagram for the event $A \cup B$:

<!-- -->

Note that the regions  $A \cap \overline{B}$, $A \cap B$, and $B \cap \overline{A}$ are disjoint.

In addition, note that we can write
\begin{align*}
A \cup B &= \biggl(A \cap \overline{B} \biggr) \cup  \biggl ( A \cap B \biggr)  \cup \biggl( B \cap \overline{A} \biggr) \\
A &=\biggl(A \cap \overline{B} \biggr) \cup  \biggl ( A \cap B \biggr), \mbox{ and} \\
 B &=    \biggl( B \cap \overline{A} \biggr) \cup \biggl ( A \cap B \biggr).  \\
\end{align*}

Applying Axiom III to each of these identities yields:
\begin{align*}
P(A \cup B) &= P\biggl(A \cap \overline{B} \biggr) + P\biggl ( A \cap B \biggr)+ P \biggl( B \cap \overline{A} \biggr) \\
P(A) &= P\biggl(A \cap \overline{B} \biggr) +  P\biggl ( A \cap B \biggr), \mbox{ and} \\
P(B) &=    P\biggl( B \cap \overline{A} \biggr) + P \biggl ( A \cap B \biggr).  \\
\end{align*}

We can write the last two equations as
\begin{align*}
P\biggl(A \cap \overline{B} \biggr) = P(A) -  P\biggl ( A \cap B \biggr), \mbox{ and} \\
P\biggl( B \cap \overline{A} \biggr) = P(B) - P \biggl ( A \cap B \biggr).
\end{align*}

Substituting into the remaining equation yields:
\begin{align*}
P(A \cup B) &= \biggl[ P(A) -  P\biggl ( A \cap B \biggr) \biggr] + P\biggl ( A \cap B \biggr)+ \biggl[P(B) - P \biggl ( A \cap B \biggr) \biggr],
\end{align*}
which simplifies to the desired result.

**Example**

In the United States, most high school students applying for college take a standardized achievement test called the SAT, which is administered by the College Board. The main test consists of a Verbal and Math part, each of which is scored on a scale from 200 to 800.  The following probability information is inferred from data found online[^sat_data].
* The probability of getting over a 600 on the Verbal part is 0.24.
* The probability of getting over a 600 on the Math part is 0.25.
* The probability of getting over a 600 on both the Math and Verbal parts is 0.16.

**(a)** If a college requires that a student get a score over 600 on at least one of the Math and Verbal parts of the SAT to be eligible for admission, what is the probability that a randomly chosen student will meet the college's SAT criterion for admission?

Let

$V$ = event that verbal score > 600
$M$ = event that Math score > 600

We are looking for $P\left(V \cup M \right)$. We can apply Corollary 5,
\begin{align*}
P\left( V \cup M \right) &= P(V) + P(M) - P(V \cap M) \\
 &= 0.24 + 0.25 -   0.16 \\
 &= 0.33
\end{align*}

About 1/3 of students who take the SAT will be meet the college's criterion.

[^sat_data]: Data on correlation  from https://eportfolios.macaulay.cuny.edu/liufall2013/files/2013/10/New_Perspectives.pdf .
Data on mean and variance is from https://blog.prepscholar.com/sat-standard-deviation .

(b) What is the probability that a randomly chosen student does not make over a 600 on either test?

This probability can be written as $P( \overline{V} \cap \overline{M})$.  It is the complement of the event in part (a):
\begin{align*}
P( \overline{V} \cap \overline{M}) &= P\left( \overline{V \cup M} \right) \\
&= 1- P\left( V \cup M \right) \\
&= 0.67
\end{align*}

(c) What is the probability that a randomly chosen student makes over a 600 on the Math part but a 600 or less on the Verbal part?

This probability can be written as $P(M \cap \overline{V})$. I will provide a purely mathematical answer, but it is helpful to draw a Venn diagram to visualize this scenario. Note that
\begin{align*}
P(M) &= P \left(M \cap V \right) + P\left(M \cap \overline{V} \right)\\
P\left(M \cap \overline{V} \right) &= P(M) - P \left(M \cap V \right) \\
&= 0.25 - 0.16 \\
&= 0.09
\end{align*}

**Example**

**(Take 2)** A fair six-sided die is rolled twice.  What is the probability that either of the rolls is a value less than 3?

As before, let $E_i$ be the event that the top face on roll $i$ is less than 3, for $i=1,2$.

Referring back to {doc}`../axiomatic-prob`, note that it is much easier to calculate the number of outcomes in $E_1 \cap E_2$ than to count the number of items in $E_1 \cup E_2$.  (Intersections are always no bigger than the smallest constituent set, where unions are no smaller than the largest of the  constituent sets.)

Here is the code that displays the intersection:

```python
from termcolor import colored
print("Outcomes in both E1 and E2 are in red:")
for j in range(1,7):
    for k in range(1,7):
        if j<3 and k<3:
            print(colored("("+str(j)+", "+str(k)+")   ", 'red'), end="")
        else:
            print("("+str(j)+", "+str(k)+")   ", end="")
    print()
```

    Outcomes in both E1 and E2 are in red:
    (1, 1)   (1, 2)   (1, 3)   (1, 4)   (1, 5)   (1, 6)
    (2, 1)   (2, 2)   (2, 3)   (2, 4)   (2, 5)   (2, 6)
    (3, 1)   (3, 2)   (3, 3)   (3, 4)   (3, 5)   (3, 6)
    (4, 1)   (4, 2)   (4, 3)   (4, 4)   (4, 5)   (4, 6)
    (5, 1)   (5, 2)   (5, 3)   (5, 4)   (5, 5)   (5, 6)
    (6, 1)   (6, 2)   (6, 3)   (6, 4)   (6, 5)   (6, 6)

We see that $| E_1 \cap E_2| = 4$, which means that $P(E_1 \cap E_2) = 4/36=1/9$.  Then we can calculate the desired probability as
\begin{align*}
P(E_1 \cup E_2) &= P(E_1) + P(E_2) - P(E_1 \cap E_2) \\
&= \frac{2}{6} + \frac{2}{6} - \frac{1}{9}\\
&= \frac{5}{9}
\end{align*}

**Corollary 6.**   If $A \subset B$, then $P(A) \le P(B)$.

*Proof*

It may be helpful to refer to the Venn diagram for some intuition:

Note that $ A \subset B$ implies that $A \cap B = A$. Then
\begin{align*}
P(B) &= P\left(B \cap A \right) + P \left(B \cap \overline{A} \right) \\
     &= P\left( A \right) + P \left(B \cap \overline{A} \right) \\
P\left( A \right) &= P(B) - P \left(B \cap \overline{A} \right)
\end{align*}
Since $P \left(B \cap \overline{A} \right) \ge 0$, $P(A) \le P(B)$.

**Corollary 7.**

\begin{eqnarray*}
P\left( \bigcup_{k=1}^{n} A_k \right) &=&
\sum_{k=1}^{n} P\left(A_j\right)
    -\sum_{j<k} P \left( A_j \cap A_k \right) + \cdots \\
    && +
    (-1)^{(n+1) } P\left(A_1 \cap A_2 \cap \cdots \cap A_n \right)
\end{eqnarray*}

Here the ellipses indicate that the pattern should be continued until exhaustion: add all single events, subtract off all intersections of pairs of events, add in all intersections of 3 events, ...}

The proof is by induction and is omitted.

Note that the set of Corollaries is not unique, and this set is not meant to be comprehensive. Rather, this represents some common tools that we will use in our work on probability.


# Physics 256
## Lecture 11 - Plot Types

```python
import style
style._set_css_style('../include/bootstrap.css')
```

## Last Time

### [Notebook Link: 10_PlotCustomization.ipynb](./10_PlotCustomization.ipynb)

- using LaTeX in plot labels
- meshgrid for multi-dimensional data sets
- plot customization

## Today

- plot types

## Plotting with Matplotlib

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib notebook
plt.style.use('../include/notebook.mplstyle');
%config InlineBackend.figure_format = 'svg'
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
```

## Plot Types

Matplotlib includes almost any imaginable type of chart or graph.  A good starting point to determine what is the best type of graph for your particular data is:

```python
from IPython.display import Image
Image(filename='../data/choosing_a_good_chart.png')
```

### Histograms

```python
# get 10000 gaussian distributed random numbers
h = np.random.randn(10000)

# the number of bins for our histogram
numBins = 100
plt.hist(h,numBins,edgecolor='w', density=True, alpha=0.5)

# let's add the theoretical value
x = np.arange(-5,5,0.01)
y = np.exp(-x**2/2)/np.sqrt(2.0*np.pi)
plt.plot(x,y,'-', color=colors[0])

# we can save our figure to disk
#plt.savefig('output/histogram.pdf')
```

### Errorbars

```python
# errorbars
x = np.arange(0, 4, 0.2)
y = np.exp(-x)
dy = 0.1 * np.abs(np.random.randn(len(y)))
dx = 0.1 * np.abs(np.random.randn(len(y)))
plt.errorbar(x,y,xerr=dx,yerr=dy,marker='o',markersize=8.0,capsize=0.0,elinewidth=1.0)
```

### Bar charts

```python
bar = {'A': 40, 'B': 70, 'C': 30, 'D': 85}
bar = sorted(bar.items())

keys = []
vals = []
for i,cbar in enumerate(bar):
    plt.bar(i, cbar[1], alpha=0.5,color=colors[i])
    keys.append(cbar[0])
    vals.append(cbar[1])

plt.xticks(np.arange(len(bar))+0.4,keys)
plt.yticks(vals)
```

### Pie charts

```python
# pie chart
plt.figure(4,figsize=(6,6));
aleast = {'Boston':84, 'New York':103, 'Tampa Bay':96, 'Toronto':67, 'Baltimore':54}
explode = [0.0, 0.0, 0.0, 0.0, 0.1]
plt.pie(list(aleast.values()), labels=list(aleast.keys()), explode=explode, autopct='%3.1f%%', colors=colors[:]);
```

### Scatter plots

Useful and effecient for large amounts of data points

```python
# scatter graph
x = np.random.randn(1000)
y = np.random.randn(1000)
size = 50*np.random.randn(1000)
col = np.random.rand(1000)
plt.scatter(x, y, s=size, c=col)
```

### Polar plots

```python
# polar plot
theta = np.linspace(0, 2.0*np.pi, 100)
plt.polar(3*theta, theta/5);
plt.polar(theta, np.cos(4*theta));
```

### Annotating plots

```python
# Produce a linear plot with an intercept

iL = np.linspace(0, 0.1, 100)
yint = 0.15
y = yint + 2.0*iL
plt.plot(iL,y,marker='None', linestyle='-')
plt.axis([0,0.09,0,0.4])
plt.xlabel(r'$1/L$')
plt.ylabel(r'$f(L)$')

# Our first label in data coordinates
plt.text(0.002,0.25,r'$f(0) = 0.15$', fontsize=18)

# The label in figure coordinates
plt.figtext(0.2,0.2,'intercept = 0.15', fontsize=18, color=colors[1])

# Add a label and an arrow
plt.annotate(r'$L \to \infty$',xy=(0, 0.15), xytext=(0.02, 0.08),\
             fontsize=30, color=colors[2],\
             arrowprops=dict(color=colors[2],shrink=0.1))
```

<div class="span alert alert-success">
<h2> Team Programming challenge </h2>
<h3> Determine a graphical solution for $x(a)$ (with at least 6 points) to the transcendental equation: </h3>
</div>
\begin{equation}
x = \tanh\left(\frac{x}{a} \right)
\end{equation}

```python
def trans(x,a):
    ''' A transcendental equationb x = tanh(x/a). '''

    # add equation
    return 0

# choose a range of a values and x values
# you might need to explore a bit with these
amin,amax = 0,1
xmin,xmax,Δx = 0,1,0.01
a = np.linspace(amin,amax,6)
x = np.arange(xmin,xmax,Δx)

# plot the LHS
plt.plot('-')

# Now plot the RHS for different values of a
for i,ca in enumerate(a):
    plt.plot('-', label=f'{ca:5.3f}')

plt.legend()
```

Visually identiry solutions using the notebook feature and add the values of x here

```python
sol = []
```

Plot the solution

```python
plt.plot()
plt.xlabel(r'$a$')
plt.ylabel(r'$x$')
```

```python
%load solutions/transcendental.py
```