# Přísnost známkování na TIMSS datech

Trends in International Mathematics and Science Study: https://en.wikipedia.org/wiki/Trends_in_International_Mathematics_and_Science_Study

Cílem je prozkoumat rozdíly v přísnosti známkování mezi školami na základě jejich různých charakteristik -- například kraj, velikost sídla apod. TIMSS data obsahují testové skóry a známky pro matematiku a přírodovědu (science), navíc lze dataset napojit s doplňujícími informacemi o škole.

Testové skóry jsou v datasetu s pěti plausible values - žáci zpravidla vyplňují pouze část z celé testové baterie, plausible values jsou nutné pro správné odhady směrodatných chyb. Metodologie týkající se plausible values v TIMSS a dalších mezinárodních vzdělávacích studiích je někde určitě popsaná důkladněji. Pro bodové odhady stačí vzít průměry plausible values. Pro odhadovaný model by možná bylo žádoucí pracovat s plausible values lepším způsobem, nejsem si jistý. Pravděpodobně by to ale neovlivnilo směr závěrů.

Odhady pro školy jsou založené na fixed effects modelu, který je v ekonometrii poměrně častý - zjednodušeně se dá přirovnat k lineárnímu modelu s odlišnými intercepts pro jednotlivé školy (nebo třídy). Nejsem si stoprocentně jistý, zdali je tento postup ideální pro analýzu rozdílů v přísnosti známkování, ale zatím jsem nevymyslel nic lepšího.

## Shrnutí

- Mezi skupinami jsou rozdíly, které jsou relativně stabilní pro danou školu nebo třídu (tzn. vysoká korelace efektů v matematice a přírodovědě - nemůže to ale být dáno tím, že někdy se bere stejná známka nebo stejný testovací skór pro oba předměty? Snad ne, v popisu dat jsem na nic podobného nenarazil - navíc stejná známka by mohla mít leckdy opačný efekt.)
- Rozdíly se nedaří vysvětlit dostupnými charakteristikami škol - velikost obce, lokalita, životní úroveň
- Je těžké posoudit význam velikosti efektů - většina škol se nachází zhruba +/- třetinu známkového stupně od průměrného hodnocení. Tedy mezi školami jsou rozdíly až do přibližně dvou třetin známkového stupně, což není zanedbatelný rozdíl.

## Co bych ještě chtěl dořešit?

- [ ] Plausible values: pro bodové odhady je jejich průměr v pohodě, jak je to ale s dalšími regresními odhady? Neměl bych v takovém případě uvažovat i variabilitu ztracenou zahozením plausible values? Byla by to další nahodilost v prediktorech v FE modelu?
  - Možná to vede k mírně odlišným výsledkům, ale asi to nemá zásadní vliv na hlavní zjištění - že rozdíly v přísnosti nejsou příliš velké a nelze je snadno vysvětlit školními charakteristikami.
  - Chci kromě FE modelu doplnit o nějaké deskriptivní grafy?
  - Nebylo by korektní zpracování PV prostě jen rozdělení každého studenta na 5 plausible studentů?
- [ ] Je FE model zde vhodný? Nebylo by lepší použít nějaký víceúrovňový?

## Příprava dat

```python
# nejake standardni importy
import os
import sys
import pyreadstat
import pandas as pd
import numpy as np
from statsmodels.stats.weightstats import DescrStatsW
from sklearn.linear_model import LinearRegression
import statsmodels.api as sm
import statsmodels.formula.api as smf
import seaborn as sns
#import matplotlib as mpl
#mpl.use('Agg')
import matplotlib.pyplot as plt
# aby grafy byly rozumně čitelné na obrazovce
plt.rcParams['figure.dpi'] = 90
plt.rcParams['axes.titlesize'] = 11
#plt.ioff()
```

```python
# cesta k datum
data_root = '/mnt/d/projects/idea/data'
```

```python
# načtení dat - obecně data k mezinárodním vzdělávacím studiím jsou ve formátu .sav pro SPSS
# v SPSS existuje i plugin pro korektní práci s plausible values, nicméně pro naše účely není potřeba
# - pro bodové odhady by mělo stačit zprůměrování plausible values; navíc nemám SPSS...
sg11, sg11_meta = pyreadstat.read_sav(f'{data_root}/TIMSS/2011/CZ/T11_dot_zak.sav')
sg15, sg15_meta = pyreadstat.read_sav(f'{data_root}/TIMSS/2015/CZ/ASGCZEM6.sav')
```

```python
# takto vypadjí raw data
sg11
```

      IDCNTRY
      IDBOOK
      IDSCHOOL
      IDCLASS
      IDSTUD
      IDGRADE
      ITBIRTHD
      ITBIRTHM
      ITBIRTHY
      ITSEX
      ...
      o07b
      o07c
      o07d
      o08a
      o08b
      o08c
      o08d
      o08e
      o08f
      o13

      0
      203.0
      8.0
      1.0
      102.0
      10201.0
      4.0
      23.0
      10.0
      2000.0
      2.0
      ...
      1.0
      1.0
      2.0
      4.0
      4.0
      3.0
      2.0
      1.0
      3.0
      4.0

      1
      203.0
      9.0
      1.0
      102.0
      10202.0
      4.0
      4.0
      5.0
      2001.0
      2.0
      ...
      2.0
      2.0
      2.0
      3.0
      1.0
      2.0
      1.0
      1.0
      2.0
      4.0

      2
      203.0
      10.0
      1.0
      102.0
      10203.0
      4.0
      14.0
      12.0
      1999.0
      2.0
      ...
      1.0
      2.0
      2.0
      4.0
      3.0
      1.0
      1.0
      3.0
      1.0
      4.0

      3
      203.0
      11.0
      1.0
      102.0
      10204.0
      4.0
      13.0
      2.0
      2001.0
      2.0
      ...
      2.0
      2.0
      2.0
      3.0
      3.0
      4.0
      2.0
      1.0
      2.0
      2.0

      4
      203.0
      12.0
      1.0
      102.0
      10205.0
      4.0
      19.0
      7.0
      2001.0
      2.0
      ...
      2.0
      2.0
      2.0
      4.0
      3.0
      3.0
      4.0
      1.0
      4.0
      4.0

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      4573
      203.0
      2.0
      2167.0
      216702.0
      21670221.0
      4.0
      28.0
      7.0
      2001.0
      2.0
      ...
      1.0
      1.0
      2.0
      2.0
      2.0
      1.0
      2.0
      1.0
      4.0
      3.0

      4574
      203.0
      3.0
      2167.0
      216702.0
      21670222.0
      4.0
      10.0
      9.0
      2000.0
      1.0
      ...
      2.0
      1.0
      2.0
      2.0
      2.0
      2.0
      2.0
      1.0
      2.0
      4.0

      4575
      203.0
      4.0
      2167.0
      216702.0
      21670223.0
      4.0
      6.0
      9.0
      2001.0
      1.0
      ...
      1.0
      1.0
      2.0
      3.0
      3.0
      2.0
      2.0
      1.0
      4.0
      3.0

      4576
      203.0
      5.0
      2167.0
      216702.0
      21670224.0
      4.0
      18.0
      10.0
      2000.0
      2.0
      ...
      2.0
      2.0
      2.0
      2.0
      3.0
      3.0
      2.0
      1.0
      3.0
      4.0

      4577
      203.0
      6.0
      2167.0
      216702.0
      21670225.0
      4.0
      16.0
      10.0
      2000.0
      1.0
      ...
      1.0
      1.0
      2.0
      2.0
      1.0
      2.0
      2.0
      1.0
      2.0
      3.0

4578 rows × 227 columns

```python
# vyber jen užitečné proměnné, vynech nesmyslné známky, napočítej průměry přes plausible values pro mat a sci

shared_cols = {
    'IDSCHOOL': 'school',
    'IDCLASS': 'class',
    'IDSTUD': 'student',
    'ITSEX': 'girl',
    'TOTWGT': 'weight',
    **{f'ASMMAT0{i}': f'mat_pv{i}' for i in range(1, 6)},
    **{f'ASSSCI0{i}': f'sci_pv{i}' for i in range(1, 6)}
}

cols11 = {**shared_cols, 'o02c': 'mat_grade', 'o02d': 'sci_grade' }
cols15 = {**shared_cols, 'ASXG20C': 'mat_grade', 'ASXG20D': 'sci_grade' }

sg11 = sg11[cols11.keys()].rename(columns=cols11)
sg15 = sg15[cols15.keys()].rename(columns=cols15)

for c in ['school', 'class', 'student', 'girl']:
    sg11[c] = pd.Series(sg11[c], dtype=np.int_)
    sg15[c] = pd.Series(sg15[c], dtype=np.int_)

sg11['girl'] = -sg11['girl'] + 2
sg15['girl'] = -sg15['girl'] + 2

for c in ['mat', 'sci']:
    sg11[c + '_grade'] = np.where(sg11[c + '_grade'] > 5, np.nan, sg11[c + '_grade'])
    sg15[c + '_grade'] = np.where(sg15[c + '_grade'] > 5, np.nan, sg15[c + '_grade'])
    sg11[c] = np.mean(sg11[[f'{c}_pv{i}' for i in range(1, 6)]], axis=1)
    sg15[c] = np.mean(sg15[[f'{c}_pv{i}' for i in range(1, 6)]], axis=1)
    sg11 = sg11.drop(columns=[f'{c}_pv{i}' for i in range(1, 6)])
    sg15 = sg15.drop(columns=[f'{c}_pv{i}' for i in range(1, 6)])

sg11['unit'] = 1
sg15['unit'] = 1
```

```python
# z dat vyberu jen co potřebuji a plausible values nahrazuji průměrem
sg11
```

      school
      class
      student
      girl
      weight
      mat_grade
      sci_grade
      mat
      sci
      unit

      0
      1
      102
      10201
      0
      13.344744
      1.0
      1.0
      614.667474
      646.389064
      1

      1
      1
      102
      10202
      0
      13.344744
      1.0
      1.0
      585.620028
      658.707538
      1

      2
      1
      102
      10203
      0
      13.344744
      3.0
      2.0
      474.949460
      458.692266
      1

      3
      1
      102
      10204
      0
      13.344744
      1.0
      1.0
      520.417550
      507.918698
      1

      4
      1
      102
      10205
      0
      13.344744
      3.0
      1.0
      489.474880
      557.399712
      1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      4573
      2167
      216702
      21670221
      0
      4.385031
      1.0
      1.0
      531.534766
      541.234242
      1

      4574
      2167
      216702
      21670222
      1
      4.385031
      1.0
      1.0
      644.117658
      628.935314
      1

      4575
      2167
      216702
      21670223
      1
      4.385031
      1.0
      1.0
      562.279782
      564.929678
      1

      4576
      2167
      216702
      21670224
      0
      4.385031
      2.0
      3.0
      502.262130
      540.098306
      1

      4577
      2167
      216702
      21670225
      1
      4.385031
      1.0
      1.0
      579.194018
      631.668776
      1

4578 rows × 10 columns

```python
cg15, cg15_meta = pyreadstat.read_sav(f'{data_root}/TIMSS/2015/CZ/ACGCZEM6.sav')
cg11, cg11_meta = pyreadstat.read_sav(f'{data_root}/TIMSS/2011/CZ/T11_dot_skolni.sav')

cg15['school'] = np.int_(cg15['IDSCHOOL'])
cg11['school'] = np.int_(cg11['IDSCHOOL'])
```

```python
cg11
```

      IDCNTRY
      IDSCHOOL
      ACBG01
      ACBG02
      ACBG03A
      ACBG03B
      ACBG04
      ACBG05A
      ACBG05B
      ACBG05C
      ...
      ACDGEAS
      ACBGDAS
      ACDGDAS
      ACDG03
      ACDGENS
      ACDG09
      ACDGCMP
      ACDG06HY
      ACDG06
      school

      0
      203.0
      1.0
      693.0
      74.0
      2.0
      2.0
      1.0
      1.0
      2.0
      2.0
      ...
      3.0
      9.45643
      2.0
      2.0
      4.0
      1.0
      1.0
      727.50
      5.0
      1

      1
      203.0
      2.0
      613.0
      66.0
      3.0
      3.0
      1.0
      1.0
      2.0
      1.0
      ...
      2.0
      9.14425
      2.0
      2.0
      1.0
      1.0
      2.0
      NaN
      5.0
      2

      2
      203.0
      3.0
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      ...
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      NaN
      5.0
      3

      3
      203.0
      4.0
      339.0
      33.0
      2.0
      1.0
      2.0
      1.0
      1.0
      3.0
      ...
      3.0
      10.17820
      1.0
      2.0
      4.0
      2.0
      1.0
      731.25
      5.0
      4

      4
      203.0
      5.0
      463.0
      60.0
      2.0
      2.0
      1.0
      1.0
      2.0
      2.0
      ...
      3.0
      8.86115
      2.0
      2.0
      4.0
      2.0
      2.0
      837.25
      5.0
      5

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      172
      203.0
      2012.0
      81.0
      8.0
      NaN
      NaN
      1.0
      6.0
      5.0
      3.0
      ...
      2.0
      9.45643
      2.0
      NaN
      4.0
      2.0
      1.0
      843.75
      5.0
      2012

      173
      203.0
      2100.0
      37.0
      8.0
      4.0
      2.0
      1.0
      6.0
      5.0
      2.0
      ...
      2.0
      11.81833
      1.0
      3.0
      3.0
      3.0
      1.0
      776.25
      5.0
      2100

      174
      203.0
      2119.0
      364.0
      37.0
      1.0
      3.0
      1.0
      5.0
      4.0
      2.0
      ...
      2.0
      9.45643
      2.0
      1.0
      4.0
      4.0
      1.0
      806.25
      5.0
      2119

      175
      203.0
      2154.0
      110.0
      26.0
      4.0
      1.0
      2.0
      4.0
      4.0
      3.0
      ...
      3.0
      11.81833
      1.0
      3.0
      4.0
      4.0
      1.0
      712.50
      5.0
      2154

      176
      203.0
      2167.0
      416.0
      52.0
      2.0
      4.0
      1.0
      2.0
      1.0
      2.0
      ...
      2.0
      9.14425
      2.0
      1.0
      2.0
      2.0
      1.0
      1144.25
      5.0
      2167

177 rows × 143 columns

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});
```

    <IPython.core.display.Javascript object>

## Fixed effects model

> Výběr modelu je trochu intuitivní -- bylo by lepší použít nějaký jiný přístup? Co nějaké multilevel modely (kterým ale nerozumím)? Stejně tak i k matematice a předpokladům přistupuju trochu pocitově.

Fixed effects model je v zásadě rychlejší metoda odhadu skupinových konstantních členů.

Předpokládejme následující model, kde pozorování jsou sdružena do skupin:

\begin{equation}\label{eqref:fe}
    y_{ij} = \alpha + \beta x_{ij} + \mu_i + \varepsilon_{ij}
\end{equation}

Kde $j = 1 \dots J_i$ jsou indexy v rámci skupin, $i = 1 \dots I$ jsou indexy skupin. $\mu_i$ je noise term pro jednotlivé skupiny (na který lze také nahlížet jako na odlišný intercept v rámci skupin; po celou dobu však předpokládáme shodný koeficient $\beta$ ve všech skupinách), $\varepsilon_{ij}$ noise term pro pozorování s obvyklými předpoklady (nezávislost, nestrannost). Použití běžné regrese vede k nekonzistentním výsledkům, neboť chyby uvnitř skupin jsou vzájemně korelované kvůli sdílenému $\mu_i$ členu.

Pokud do modelu doplníme indikátorové proměnné pro jednotlivé skupiny, pak získáme správný odhad ($\alpha$ bude dána váženým průměrem těchto skupinových koeficientů). Při velkém počtu skupin však může být takový odhad výpočetně náročný, alternativou je použití fixed effects modelu. Pokud rovnici (\ref{eqref:fe}) zprůměrujeme uvnitř skupin a následně i celkově:

\begin{eqnarray}
    y_{i\bullet} = \alpha + \beta x_{i\bullet} + \mu_i + \varepsilon_{i\bullet} \label{eqref:groupmeanfe}\\
    y_{\bullet\bullet} = \alpha + \beta x_{\bullet\bullet} + \mu_{\bullet} + \varepsilon_{\bullet\bullet} \label{eqref:grandmeanfe}
\end{eqnarray}

Kde $y_{i\bullet} = \frac{1}{J_i} \sum_{j = 1}^{J_i} y_{ij}$ a $y_{\bullet\bullet} = \frac{1}{N} \sum_{i = 1}^{I} \sum_{j = 1}^{J_i} y_{ij}$, analogicky pro $x_{i\bullet}$ a $x_{\bullet\bullet}$. Z předpokladů regrese dále vyplývá $\mu_{\bullet} = \varepsilon_{\bullet\bullet} = \varepsilon_{i\bullet} = 0$. Zkombinováním rovnic (\ref{eqref:fe}) - (\ref{eqref:groupmeanfe}) + (\ref{eqref:grandmeanfe}) dostáváme:

\begin{equation} \label{eqref:fullfe}
    (y_{ij} - y_{i\bullet} + y_{\bullet\bullet}) = \alpha + \beta \cdot (x_{ij} - x_{i\bullet} + x_{\bullet\bullet}) + \varepsilon_{ij}
\end{equation}

Z rovnice (\ref{eqref:fullfe}) můžeme obyčejnou regresí odhadnout koeficienty $\alpha$ a $\beta$, následně z rovnice (\ref{eqref:groupmeanfe}) můžeme dopočítat skupinové parametry $\mu_i$. Model lze snadno zobecnit na více prediktorů.

Skupinové parametry $\mu_i$ odpovídají rozdílům v přísnosti známkování v jednotlivých školách (nebo případně třídách). Tyto parametry pak můžeme porovnat s dalšími informacemi o školách, například s krajem, velikostí obce apod.

```python
def get_weighted_means(frame, weight_col='weight', cols=None):
    """
    Calculates weighted means of dataframe columns.

    :param frame: Input dataframe.
    :param weight_col: Column containing weights.
    :param cols: Columns to calculate the weighted means of. If not provided, use all non-weight columns.
    :return: Series with weighted means, keyed by column names.
    """
    res = pd.Series(dtype='float64')
    for c in cols or frame.columns[frame.columns != weight_col]:
        res[c] = (frame[c] * frame[weight_col]).sum() / frame[weight_col].sum()
    return res

def get_fixed_effects(df, group_col, grade_col, score_col, girl_col='girl', weight_col='weight'):
    df = df[[group_col, grade_col, score_col, girl_col, weight_col]].dropna().copy()

    # pomocí DescrStatsW získáme vážené průměry i směrodatné odchylky
    stats_grade = DescrStatsW(df[grade_col], weights=df[weight_col])
    stats_score = DescrStatsW(df[score_col], weights=df[weight_col])

    # převod na z-skóry, aby následně parametry bylo možné interpretovat ve směrodatných odchylkách
    # chci z-skóry i u známek? - asi ne, známky jsou zde snadněji interpretovatelné
    df[score_col] = (df[score_col] - stats_score.mean) / stats_score.std
    # df[grade_col] = -(df[grade_col] - stats_grade.mean) / stats_grade.std

    cols = [grade_col, score_col, girl_col]

    # c_means dataframe obsahuje skupinové průměry
    c_means = df.groupby(group_col).apply(get_weighted_means, weight_col=weight_col, cols=cols).reset_index()
    df = pd.merge(df, c_means.rename(columns={c: c + '_cm' for c in cols}))
    for c in cols:
        # _adj sloupce jsou proměnné pro odhad rovnice (4)
        df[c + '_adj'] = df[c] - df[c + '_cm'] + np.average(df[c], weights=df[weight_col])

    # WLS ze statsmodels je Weighted Least Squares
    model = sm.WLS(df[grade_col + '_adj'], sm.add_constant(df[[score_col + '_adj', girl_col + '_adj']]), weights=df[weight_col]).fit()

    # zpětným dosazením napočítáme \mu_i (označené jako fe pro fixed effects)
    c_means['fe'] = c_means[grade_col] - model.params['const'] \
        - model.params[score_col + '_adj'] * c_means[score_col] \
        - model.params[girl_col + '_adj'] * c_means[girl_col]
    c_means['count'] = df.groupby(group_col)[grade_col].count().values

    # a vrátíme dataframe s indexem skupiny, fixed effects a počtem žáků ve skupinách
    return c_means[[group_col, 'fe', 'count']]
```

## Základní popisné statistiky

```python
sg11
```

      school
      class
      student
      girl
      weight
      mat_grade
      sci_grade
      mat
      sci
      unit

      0
      1
      102
      10201
      0
      13.344744
      1.0
      1.0
      614.667474
      646.389064
      1

      1
      1
      102
      10202
      0
      13.344744
      1.0
      1.0
      585.620028
      658.707538
      1

      2
      1
      102
      10203
      0
      13.344744
      3.0
      2.0
      474.949460
      458.692266
      1

      3
      1
      102
      10204
      0
      13.344744
      1.0
      1.0
      520.417550
      507.918698
      1

      4
      1
      102
      10205
      0
      13.344744
      3.0
      1.0
      489.474880
      557.399712
      1

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      4573
      2167
      216702
      21670221
      0
      4.385031
      1.0
      1.0
      531.534766
      541.234242
      1

      4574
      2167
      216702
      21670222
      1
      4.385031
      1.0
      1.0
      644.117658
      628.935314
      1

      4575
      2167
      216702
      21670223
      1
      4.385031
      1.0
      1.0
      562.279782
      564.929678
      1

      4576
      2167
      216702
      21670224
      0
      4.385031
      2.0
      3.0
      502.262130
      540.098306
      1

      4577
      2167
      216702
      21670225
      1
      4.385031
      1.0
      1.0
      579.194018
      631.668776
      1

4578 rows × 10 columns

```python
fig, axes = plt.subplots(2, 2, figsize=(10, 6))
#sns.histplot(data=sg11, x='mat_grade', ax=axes[0][0])
bins = np.arange(6) + 0.5
axes[0][0].hist(sg11['mat_grade'], rwidth=0.5, weights=sg11['weight'], bins=bins)
axes[0][0].set_title('TIMSS 2011, mat_grade', fontsize=10)
axes[0][1].hist(sg11['sci_grade'], rwidth=0.5, weights=sg11['weight'], bins=bins)
axes[0][1].set_title('TIMSS 2011, sci_grade', fontsize=10)
axes[1][0].hist(sg15['mat_grade'], rwidth=0.5, weights=sg15['weight'], bins=bins)
axes[1][0].set_title('TIMSS 2015, mat_grade', fontsize=10)
axes[1][1].hist(sg15['sci_grade'], rwidth=0.5, weights=sg15['weight'], bins=bins)
axes[1][1].set_title('TIMSS 2015, sci_grade', fontsize=10)
```

```python
fig, axes = plt.subplots(2, 2, figsize=(10, 6))
axes[0][0].hist(sg11['mat'], bins=30, rwidth=0.9, weights=sg11['weight'])
axes[0][0].set_title('TIMSS 2011, mat score', fontsize=10)
axes[0][1].hist(sg11['sci'], bins=30, rwidth=0.9, weights=sg11['weight'])
axes[0][1].set_title('TIMSS 2011, sci score', fontsize=10)
axes[1][0].hist(sg15['mat'], bins=30, rwidth=0.9, weights=sg15['weight'])
axes[1][0].set_title('TIMSS 2015, mat score', fontsize=10)
axes[1][1].hist(sg15['sci'], bins=30, rwidth=0.9, weights=sg15['weight'])
axes[1][1].set_title('TIMSS 2015, sci score', fontsize=10)
```

```python
for yy in ['sg11', 'sg15']:
    for cc in ['mat', 'sci', 'mat_grade', 'sci_grade']:
        df = eval(yy)[[cc, 'weight']].dropna()
        dsw = DescrStatsW(df[cc], weights=df['weight'])
        print(f'{yy}, {cc}: mean={dsw.mean:.4g}, std={dsw.std:.4g}')
```

    sg11, mat: mean=510.8, std=66.65
    sg11, sci: mean=536.4, std=66.94
    sg11, mat_grade: mean=1.615, std=0.7578
    sg11, sci_grade: mean=1.51, std=0.7461
    sg15, mat: mean=528.1, std=66.65
    sg15, sci: mean=534.4, std=65.17
    sg15, mat_grade: mean=1.63, std=0.7836
    sg15, sci_grade: mean=1.477, std=0.7256

```python
fe15_mat = get_fixed_effects(sg15, 'school', 'mat_grade', 'mat')
fe15_sci = get_fixed_effects(sg15, 'school', 'sci_grade', 'sci')
fe11_mat = get_fixed_effects(sg11, 'school', 'mat_grade', 'mat')
fe11_sci = get_fixed_effects(sg11, 'school', 'sci_grade', 'sci')

fe15_mat_class = get_fixed_effects(sg15, 'class', 'mat_grade', 'mat')
fe15_sci_class = get_fixed_effects(sg15, 'class', 'sci_grade', 'sci')
fe11_mat_class = get_fixed_effects(sg11, 'class', 'mat_grade', 'mat')
fe11_sci_class = get_fixed_effects(sg11, 'class', 'sci_grade', 'sci')
```

    /mnt/d/projects/idea/code/venv/lib/python3.8/site-packages/statsmodels/tsa/tsatools.py:142: FutureWarning: In a future version of pandas all arguments of concat except for the argument 'objs' will be keyword-only
      x = pd.concat(x[::order], 1)

## Rozložení třídních / školních efektů

Skupinové efekty jsou vyjádřeny v prostoru známek - tzn. školní efekt 0.25 znamená, že žáci s určitým skórem dostávají o 0.25 stupně horší známku než by dostali v průměrné škole (tedy známkování zde je o 0.25 stupně přísnější).

```python
def hist_fe(fe, weights=False, title='', ax=None):
    stat = DescrStatsW(fe['fe'], weights=(fe['count'] if weights else None))
    ax = ax or plt.gca()
    ax.hist(fe['fe'], bins=20, rwidth=0.9, weights=(fe['count'] if weights else None))
    ax.set_title('{} (std dev = {:.3g})'.format(title, stat.std), fontsize=10)
    return ax

plt.rcParams['figure.figsize'] = 10, 2.5

for y in [15, 11]:
    for c in ['mat', 'sci']:
        for g in ['school', 'class']:
            fig, axes = plt.subplots(1, 2)
            for w in [0, 1]:
                fe = eval('fe{}_{}{}'.format(y, c, ('' if g == 'school' else '_class')))
                title = 'TIMSS, 20{}, {}, {} FE{}'.format(y, ('Maths' if c == 'mat' else 'Science'), g,
                    (', by #students' if w else ''))
                hist_fe(fe, w, title, ax=axes[w])
```

## Stabilita třídních / školních efektů

Data vykazují poměrně vysokou korelaci mezi skupinovými efekty v matematice a v přírodovědě. Skupiny se tedy pravděpodobně opravdu liší v přísnosti známkování, jen ji nedokážeme dobře vysvětlit pozorovanými atributy.

```python
fig, axes = plt.subplots(2, 2, figsize=(10, 7))
plt.subplots_adjust(hspace=0.3)

fe15 = pd.merge(fe15_mat.rename(columns={'fe': 'fe_mat'}), fe15_sci.rename(columns={'fe': 'fe_sci'}))
dsw = DescrStatsW(fe15[['fe_mat', 'fe_sci']], weights=fe15['count'])
sns.scatterplot(data=fe15, x='fe_mat', y='fe_sci', ax=axes[0][0])
axes[0][0].set_title(f'TIMSS 2015, školní efekty, corr = {dsw.corrcoef[0, 1]:.3g}')

fe15_class = pd.merge(fe15_mat_class.rename(columns={'fe': 'fe_mat'}), fe15_sci_class.rename(columns={'fe': 'fe_sci'}))
dsw = DescrStatsW(fe15_class[['fe_mat', 'fe_sci']], weights=fe15_class['count'])
sns.scatterplot(data=fe15_class, x='fe_mat', y='fe_sci', ax=axes[0][1])
axes[0][1].set_title(f'TIMSS 2015, třídní efekty, corr = {dsw.corrcoef[0, 1]:.3g}')

fe11 = pd.merge(fe11_mat.rename(columns={'fe': 'fe_mat'}), fe11_sci.rename(columns={'fe': 'fe_sci'}))
dsw = DescrStatsW(fe11[['fe_mat', 'fe_sci']], weights=fe11['count'])
sns.scatterplot(data=fe11, x='fe_mat', y='fe_sci', ax=axes[1][0])
axes[1][0].set_title(f'TIMSS 2011, školní efekty, corr = {dsw.corrcoef[0, 1]:.3g}')

fe11_class = pd.merge(fe11_mat_class.rename(columns={'fe': 'fe_mat'}), fe11_sci_class.rename(columns={'fe': 'fe_sci'}))
dsw = DescrStatsW(fe11_class[['fe_mat', 'fe_sci']], weights=fe11_class['count'])
sns.scatterplot(data=fe11_class, x='fe_mat', y='fe_sci', ax=axes[1][1])
axes[1][1].set_title(f'TIMSS 2011, třídní efekty, corr = {dsw.corrcoef[0, 1]:.3g}')
None
```

## Školní efekty v závislosti na sídle / charakteristikách školy

Některé grafy naznačují rozdíly podle charakteristik, avšak nikdy nelze tento efekt pozorovat v obou předmětech a u obou ročníků. Může se tedy jednat spíše o náhodný šum než o skutečný efekt. Některé školy pravděpodobně jsou přísnější, avšak nemusí se jednat o systematické tendence způsobené pozorovanými charakteristikami škol a obce.

```python
def compare_fe(cg, cg_meta, var, col, label, num_labels, left=0.16, title=None, fe_col='fe'):
    labels = [cg_meta.value_labels[label][x] for x in range(1, num_labels + 1)]
    cg[var] = pd.Categorical(cg[col].apply(lambda x: None if not np.isfinite(x)
        else cg_meta.value_labels[label][x]), labels, ordered=True)
    plt.rcParams['figure.figsize'] = 7, 2.5
    plt.rcParams['figure.subplot.left'] = left
    fig, ax = plt.subplots()
    sns.boxplot(y=var, x=fe_col, data=cg)
    if title is not None:
        ax.set_title(title)
    return ax

def compare_fe_all(cg, cg_meta, vars, cols, labels, num_labels, lefts, title):
    plots = []
    for v, c, l, nl, left in zip(vars, cols, labels, num_labels, lefts):
        ax = compare_fe(cg, cg_meta, v, c, l, nl, left=left)
        ax.set_title(title.format(v))
        plots.append(ax)
    return plots
```

```python
vars = ['population', 'area', 'poor', 'affluent']
cols = ['ACBG05A', 'ACBG05B', 'ACBG03A', 'ACBG03B']
labels = ['labels{}'.format(i) for i in [5, 6, 2, 3]]
num_labels15 = [7, 5, 4, 4]
num_labels11 = [6, 5, 4, 4]
lefts15 = [0.16, 0.33, 0.16, 0.16]
lefts11 = [0.24, 0.16, 0.16, 0.16]
```

```python
plots15_mat = compare_fe_all(pd.merge(cg15, fe15_mat), cg15_meta, vars, cols, labels, num_labels15, lefts15,
    title='TIMSS 2015, Maths, schools FE vs {}')
```

```python
plots15_sci = compare_fe_all(pd.merge(cg15, fe15_sci), cg15_meta, vars, cols, labels, num_labels15, lefts15,
    title='TIMSS 2015, Science, schools FE vs {}')
```

```python
plots11_mat = compare_fe_all(pd.merge(cg11, fe11_mat), cg11_meta, vars, cols, labels, num_labels11, lefts11,
    title='TIMSS 2011, Maths, schools FE vs {}')
```

```python
plots11_sci = compare_fe_all(pd.merge(cg11, fe11_sci), cg11_meta, vars, cols, labels, num_labels11, lefts11,
    title='TIMSS 2011, Science, schools FE vs {}')
```

```python
_, size11_mat = plt.subplots()
sns.scatterplot(x='ACBG01', y='fe', data=pd.merge(cg11, fe11_mat))
size11_mat.set(xlabel='size of school')
size11_mat.set_title('TIMSS 2011, Maths, schools FE vs size of school')
None
```

```python
_, size11_sci = plt.subplots()
sns.scatterplot(x='ACBG01', y='fe', data=pd.merge(cg11, fe11_sci))
size11_sci.set(xlabel='size of school')
size11_sci.set_title('TIMSS 2011, Science, schools FE vs size of school')
None
```

```python

```


```python
from __future__ import print_function
import sisl
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

$
\newcommand\HH{\mathbf{H}}
\newcommand\SO{\mathbf{S}}
\newcommand\Scat{\boldsymbol\Gamma}
\newcommand\ket[1]{|#1\rangle}
\newcommand\bra[1]{\langle#1|}
\newcommand\set[1]{\{#1\}}
\newcommand\eig{\epsilon}
$
In this example you will perform transport calculations by projecting scattering states onto molecular orbitals.

We will select Carbon chains as our electrodes and the C$_{60}$ fullerene as our molecule (it has nice symmetries).
The molecular projection method are created from a subset molecule space ($\set M$):
\begin{equation}
  \HH_{\set M} \ket{i} = \eig_i^{\set M}\SO_{\set M} \ket{i}
\end{equation}
with the projectors orthogonalized through the Löwdin transformation
\begin{equation}
  \ket{i'}=\SO^{1/2}\ket i
\end{equation}
Then the projection of the scattering states will read
\begin{equation}
   \tilde\Scat = \sum\ket{i'}\bra{i'}\Scat \sum \ket{j'}\bra{j'}
\end{equation}
which completes the basis transformation.

These projectors can be performed in TBtrans by defining the $\set M$ region and defining which scattering matrices should be projected onto.

From the above few equations it should be obvious that to create such a projection it is necessary to define a device region where the scattering matrices are *living* only on the projected region (i.e. the molecule).

---

Instead of manually setting up the C$_{60}$ molecule we find the coordinates on this web-page (http://www.nanotube.msu.edu/fullerene/fullerene-isomers.html). The coordinates are stored in the `C60.xyz` file in this directory. Note that when reading this geometry it does not know the orbital distance so we have to calculate it.

```python
C60 = sisl.Geometry.read('C60.xyz')
# Calculate the nearest neighbour distance
dist = C60.distance(R=5)
C60.atom.atom[0] = sisl.Atom(6, R=dist[0] + 0.01)
print(C60)
```

Now we have the molecule, all we need are some electrodes and the connection from the electrodes to the molecule.
This electrode is a 2x2 square lattice with transport along the $x$-direction.

```python
elec = sisl.Geometry([0] * 3, sisl.Atom(6, R=1.), [1, 1, 10]).tile(2, 0).tile(2, 1)
elec.set_nsc(a=3, b=1)
H_elec = sisl.Hamiltonian(elec)
H_elec.construct(([0.1, 1.1], [0., -1]))
H_elec.write('ELEC.nc')
```

Create the final device by making the electrode have 2 screening layers, then the C$_{60}$ and finally the right-electrode (equivalently setup to the left part).

```python
elec_x = elec.tile(3, 0)
# Translate to origo
C60 = C60.translate(-C60.center(what='xyz'))
C60 = C60.translate([-C60.xyz[:, 0].min(), 0, 0])
# Do trickery to make sure the coordinates are consecutive along x
C60.set_sc([C60.xyz[:, 0].max() + 1., 1., 1.])
device = elec_x.append(C60, 0).append(elec_x, 0)
```

The full device is now created and we simply need to create the electronic structure.

```python
H = sisl.Hamiltonian(device)
H.construct(([0.1, 1.1], [0., -1]))
# Correct the C_60 couplings to something different
idx_C60 = np.arange(len(elec_x), len(elec_x) + len(C60), dtype=np.int32)
for ia in idx_C60:
    idx = device.close(ia, R=[0.1, C60.maxR()])[1]
    # On-site is already 0, so don't bother doing anything there
    # Split idx into C60 couplings and chain couplings
    for i in idx:
        if i in idx_C60:
            H[ia, i] = -1.5
        else:
            # Coupling to chain
            # Since we are only looping atoms in C60
            # we have to also set the coupling into C60
            # (to assert Hermiticity)
            H[ia, i] = 0.1
            H[i, ia] = 0.1
H.write('DEVICE.nc')
```

Calculating projceted transmissions is verbose because multiple things is required to be defined. The following flags are required to be specified, careful read about each of them:
- TBT.Atoms.Device
- TBT.Projs (name the projection `C60`)
- TBT.Proj.<> (create a HOMO and a LUMO projection)
- TBT.Projs.T


# Dimensional Reduction
G. Richards (2016, 2018), based on materials from Ivezic, Connolly, Leighly, and VanderPlas

**Before class starts, please try to do the following:**

> find . -name “sdss_corrected_spectra.py” -print

> ./anaconda/lib/python2.7/site-packages/astroML/datasets/sdss_corrected_spectra.py

> emacs -nw ./anaconda/lib/python2.7/site-packages/astroML/datasets/sdss_corrected_spectra.py

> #DATA_URL = 'http://www.astro.washington.edu/users/vanderplas/spec4000.npz'

> DATA_URL = 'http://staff.washington.edu/jakevdp/spec4000.npz'

Just in case that doesn't work, I've put "spec4000.npz" in PHYS_T480_F18/data.  Copy this to your "astroML_data" directory.

## Curse of Dimensionality

You want to buy a car.  Right now--you don't want to wait.  But you are picky and have certain things that you would like it to have.  Each of those things has a probability between 0 and 1 of being on the the car dealer's lot.  You want a red car which has a probability of being on the lot of $p_{\rm red}$; you want good gas mileage, $p_{\rm gas}$; you want leather seats, $p_{\rm leather}$; and you want a sunroof, $p_{\rm sunroof}$.  The probability that the dealer has a car on the lot that meets all of those requirements is
$$p_{\rm red} \, p_{\rm gas} \, p_{\rm leather} \, p_{\rm sunroof},$$
or $p^n$ where $n$ is the number of features (assuming equal probability for each).

If the probability of each of these is 50%, then the probability of you driving off with your car of choice is only $0.5*0.5*0.5*0.5 = 0.0625$.  Not very good.  Imagine if you also wanted other things.  This is the [Curse of Dimensionality](https://en.wikipedia.org/wiki/Curse_of_dimensionality).

Let's illustrate the curse of dimensionality with two figures from [here.](https://medium.freecodecamp.org/the-curse-of-dimensionality-how-we-can-save-big-data-from-itself-d9fa0f872335)

In the first example we are trying to find which box hold some treasure, which gets harder and harder with more dimensions, despite there just being 5 boxes in each dimension:

In the next example we inscribe a circle in a square.  The area outside of the circle grows larger and larger as the number of dimensions increase:

Mathematically we can describe this as: the more dimensions that your data span, the more points needed to uniformly sample the space.

For $D$ dimensions with coordinates $[-1,1]$, the fraction of points in a unit hypersphere (with radius $r$, as illustrated above) is
$$f_D = \frac{V_D(r)}{(2r)^D} = \frac{\pi^{D/2}}{D2^{D-1}\Gamma(D/2)}$$
which goes to $0$ as $D$ goes to infinity!  Actually, as you can see from the plot below, it is effectively 0 much earlier than that!

```python
# Execute this cell
# from Andy Connolly
%matplotlib inline
import numpy as np
import scipy.special as sp
from matplotlib import pyplot as plt

def unitVolume(dimension, radius=1.):
    return 2*(radius**dimension *np.pi**(dimension/2.))/(dimension*sp.gamma(dimension/2.))

dim = np.linspace(1,100)

#------------------------------------------------------------
# Plot the results
fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(dim,unitVolume(dim)/2.**dim)

ax.set_yscale('log')
ax.set_xlabel('$Dimension$')
ax.set_ylabel('$Volume$')

plt.show()
```

Note that this works in the opposite direction too: let's say you want to find "rare" objects in 10 dimensions, where we'll define rare as <1% of the population.  Then you'll need to accept objects from 63% of the distribution in all 10 dimensions!  So are those really "rare" or are they just a particular 1% of the population?

```python
import numpy as np
p = 10**(np.log10(0.01)/10.0)
print p
```

    0.63095734448

N.B.  Dimensionality isn't just measuring $D$ parameters for $N$ objects.  It could be a spectrum with $D$ values or an image with $D$ pixels, etc.  In the book the examples used just happen to be spectra of galaxies from the SDSS project.  But we can insert the data of our choice instead.

For example: the SDSS comprises a sample of 357 million sources:
- each source has 448 measured attributes
- selecting just 30 (e.g., magnitude, size..) and normalizing the data range $-1$ to $1$

yields a probability of having one of the 357 million sources reside within a unit hypersphere of 1 in 1.4$\times 10^5$.

## Principal Component Analysis (PCA)

In [Principal Component Analysis (PCA)](https://en.wikipedia.org/wiki/Principal_component_analysis) we seek to take a data set like the one shown below and apply a transform to the data such that the new axes are aligned with the maximal variance of the data.  As can be seen in the Figure, this is basically just the same as doing regression by minimizing the square of the perpendicular distances to the new axes.  Note that we haven't made any changes to the data, we have just defined new axes.

```python
# Execute this cell
# Ivezic, Figure 7.2
# Author: Jake VanderPlas
# License: BSD
#   The figure produced by this code is published in the textbook
#   "Statistics, Data Mining, and Machine Learning in Astronomy" (2013)
#   For more information, see http://astroML.github.com
#   To report a bug or issue, use the following forum:
#    https://groups.google.com/forum/#!forum/astroml-general
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.patches import Ellipse

#------------------------------------------------------------
# Set parameters and draw the random sample
np.random.seed(42)
r = 0.9

sigma1 = 0.25
sigma2 = 0.08
rotation = np.pi / 6
s = np.sin(rotation)
c = np.cos(rotation)

X = np.random.normal(0, [sigma1, sigma2], size=(100, 2)).T
R = np.array([[c, -s],[s, c]])
X = np.dot(R, X) #Same data, now rotated by R matrix.

#------------------------------------------------------------
# Plot the diagram
fig = plt.figure(figsize=(5, 5), facecolor='w')
ax = plt.axes((0, 0, 1, 1), xticks=[], yticks=[], frameon=False)

# draw axes
ax.annotate(r'$x$', (-r, 0), (r, 0),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))
ax.annotate(r'$y$', (0, -r), (0, r),
            ha='center', va='center',
            arrowprops=dict(arrowstyle='<->', color='k', lw=1))

# draw rotated axes
ax.annotate(r'$x^\prime$', (-r * c, -r * s), (r * c, r * s),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))
ax.annotate(r'$y^\prime$', (r * s, -r * c), (-r * s, r * c),
            ha='center', va='center',
            arrowprops=dict(color='k', arrowstyle='<->', lw=1))

# scatter points
ax.scatter(X[0], X[1], s=25, lw=0, c='k', zorder=2)

# draw lines
vnorm = np.array([s, -c])
for v in (X.T):
    d = np.dot(v, vnorm)
    v1 = v - d * vnorm
    ax.plot([v[0], v1[0]], [v[1], v1[1]], '-k')

# draw ellipses
for sigma in (1, 2, 3):
    ax.add_patch(Ellipse((0, 0), 2 * sigma * sigma1, 2 * sigma * sigma2,
                         rotation * 180. / np.pi,
                         ec='k', fc='gray', alpha=0.2, zorder=1))

ax.set_xlim(-1, 1)
ax.set_ylim(-1, 1)

plt.show()
```

Note that the points are correlated along a particular direction which doesn't align with the initial choice of axes.  So, we should rotate our axes to align with this correlation.

We'll choose the rotation to maximize the ability to discriminate between the data points:
*   the first axis, or **principal component**, is direction of maximal variance
*   the second principal component is orthogonal to the first component and maximizes the residual variance
*   ...

PCA is a dimensional reduction process because we can generally account for nearly "all" of the variance in the data set with fewer than the original $K$ dimensions.  See more below.

We start with a data set $\{x_i\}$ which consists of $N$ objects for which we measure $K$ features.  We start by subtracting the mean for each feature in $\{x_i\}$ and write $X$ as a $N\times K$ matrix.

The covariance of this matrix is
$$C_X=\frac{1}{N-1}X^TX.$$

There are off-diagonal terms if there are correlations between the measurements (e.g., maybe two of the features are temperature dependent and the measurements were taken at the same time).

If $R$ is a projection of the data that is aligned with the maximal variance, then we have $Y= X R$ with covariance

$$ C_{Y} = R^T X^T X R = R^T C_X R.$$

$r_1$ is the first principal component of $R$, which can be derived using Langrange multipliers with the following cost function:

$$ \phi(r_1,\lambda_1) = r_1^TC_X r_1 - \lambda_1(r_1^Tr_1-1). $$

If we take derivative of $\phi(r_1,\lambda)$ with respect to $r_1$ and set it to 0, then we have

$$ C_Xr_1 - \lambda_1 r_1 = 0. $$

$\lambda_1$ (the largest eigenvalue of the matrix) is the root of the equation $\det(C_X -
\lambda_1 {\bf I})=0$ for which the eigenvalue is

$$ \lambda_1 =  r_1^T C_X r_1.$$

The columns of the full matrix, $R$ are the eigenvectors (known here as principal components).

The diagonal values of $C_Y$ are the variance contained within each component.

We aren't going to go through the linear algebra more than that here.  But it would be a good group project for someone.  See the end of 7.3.1 starting at the bottom on page 294 or go through [Karen Leighly's PCA lecture notes](http://seminar.ouml.org/lectures/principal-components-analysis/) if you want to walk through the math in more detail.

### Preparing data for PCA

   * Subtract the mean of each dimension (to "center" the data)
   * Divide by the variance in each dimension (to "whiten" the data)
   * (For spectra and images) normalize each row to yield an integral of unity.

Below is a typical call to the PCA algorithm.  Note that this is somewhat backwards.  We are starting with `X` and then we are making it higher dimensional--to create a mock high-$D$ data set.  Then we are applying PCA as a dimensionality reduction technique.

```python
#Example call from 7.3.2
import numpy as np
from sklearn.decomposition import PCA

X = np.random.normal(size=(100,3)) # 100 points in 3D
R = np.random.random((3,10)) # projection matrix
X = np.dot(X,R) # X is now 10-dim, with 3 intrinsic dims
pca = PCA(n_components=4) # n_components can be optionally set
pca.fit(X)
comp = pca.transform(X) # compute the subspace projection of X, 4 eigenvalues for each of the 100 samples
mean = pca.mean_ # length 10 mean of the data
components = pca.components_ # 4x10 matrix of components, multiply each by respective "comp" to reconstruct

#Reconstruction of object1
#Xreconstruct[0] = mean + [components][comp[0]]
```

To illustrate what is happening here is a PCA reconstruction of handwritten "3s" from [Hastie et al.](https://web.stanford.edu/~hastie/ElemStatLearn/) :

[Scikit-Learn's decomposition module](http://scikit-learn.org/stable/modules/classes.html#module-sklearn.decomposition) has a number of [PCA type implementations](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.PCA.html#sklearn.decomposition.PCA).
Let's work through an example using spectra of galaxies take during the Sloan Digital Sky Survey.  In this sample there are 4000 spectra with flux measurements in 1000 bins.  15 example spectra are shown below and our example will use half of the spectra chosen at random.

```python
%matplotlib inline
# Example from Andy Connolly
# See Ivezic, Figure 7.4
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA
from sklearn.decomposition import RandomizedPCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)
print(len(spectra), len(wavelengths))

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 2000 # We'll just look at 2000 random spectra
n_components = 5 # Do the fit with 5 components, which is the mean plus 4
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0) # Compute the mean spectrum, which is the first component
# spec_mean = spectra[:50].mean(0)

# use Randomized PCA for speed
#pca = RandomizedPCA(n_components - 1)
pca = PCA(n_components - 1,svd_solver='randomized')
pca.fit(spectra[ind])
pca_comp = np.vstack([spec_mean,pca.components_]) #Add the mean to the components
evals = pca.explained_variance_ratio_
print(evals)
```

    downloading PCA-processed SDSS spectra from http://staff.washington.edu/jakevdp/spec4000.npz to /home/dude/astroML_data
    Downloading http://staff.washington.edu/jakevdp/spec4000.npz
    [=========================================]  27.15Mb / 27.15Mb
    4000 1000
    [0.8893159  0.06058304 0.02481433 0.01012148]

```python
print(pca.explained_variance_ratio_)
```

    [0.8893159  0.06058304 0.02481433 0.01012148]

Now let's plot the components.  See also Ivezic, Figure 7.4.  The left hand panels are just the first 5 spectra for comparison with the first 5 PCA components, which are shown on the right.  They are ordered by the size of their eigenvalues.

```python
#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'PCA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel('wavelength (Angstroms)')
    ax.plot(wavelengths, pca_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel('wavelength (Angstroms)')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')
    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

Now let's make "scree" plots.  These plots tell us how much of the variance is explained as a function of the each eigenvector.  Our plot won't look much like Ivezic, Figure 7.5, so I've shown it below to explain where "scree" comes from.

```python
# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 5))

ax = fig.add_subplot(121)
ax.plot(np.arange(n_components-1), evals)
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("eigenvalue ")

ax = fig.add_subplot(122)
ax.plot(np.arange(n_components-1), evals.cumsum())
ax.set_xlabel("eigenvalue number")
ax.set_ylabel("cumulative eigenvalue")

plt.show()
```

How much of the variance is explained by the first two components?  How about all of the components?

```python
print("The first component explains {:.3f} of the variance in the data.".format(pca.explained_variance_ratio_[0]))
print("The second component explains {:.3f} of the variance in the data.".format(pca.explained_variance_ratio_[1]))
print("All components explain {:.3f} of the variance in the data.".format(sum(pca.explained_variance_ratio_)))
```

    The first component explains 0.889 of the variance in the data.
    The second component explains 0.061 of the variance in the data.
    All components explain 0.985 of the variance in the data.

This is why PCA enables dimensionality reduction.

How many components would we need to explain 99.5% of the variance?

```python
for num_feats in np.arange(1,20, dtype = int):
    pca = PCA(n_components=num_feats)
    pca.fit(spectra[ind])
    if (sum(pca.explained_variance_ratio_)>0.995):
        break
print("{:d} features are needed to explain 99.5% of the variance".format(num_feats))
```

    8 features are needed to explain 99.5% of the variance

Note that we would need 1000 components to encode *all* of the variance.

## Interpreting the PCA

- The output eigenvectors are ordered by their associated eigenvalues
- The eigenvalues reflect the variance within each eigenvector
- The sum of the eigenvalues is total variance of the system
- Projection of each spectrum onto the first few eigenspectra is a compression of the data

Once we have the eigenvectors, we can try to reconstruct an observed spectrum, ${x}(k)$, in the eigenvector basis, ${e}_i(k)$, as

$$ \begin{equation}
  {x}_i(k) = {\mu}(k) + \sum_j^R \theta_{ij} {e}_j(k).
\end{equation}
$$

That would give a full (perfect) reconstruction of the data since it uses all of the eigenvectors.  But if we truncate (i.e., $r<R$), then we will have reduced the dimensionality while still reconstructing the data with relatively little loss of information.

For example, we started with 4000x1000 floating point numbers.  If we can explain nearly all of the variance with 8 eigenvectors, then we have reduced the problem to 4000x8+8x1000 floating point numbers!

Execute the next cell to see how the reconstruction improves by adding more components.

```python
# Execute this cell
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#------------------------------------------------------------
# Compute PCA components

# Eigenvalues can be computed using PCA as in the commented code below:

#from sklearn.decomposition import PCA
#pca = PCA()
#pca.fit(spectra)
#evals = pca.explained_variance_ratio_
#evals_cs = evals.cumsum()

#  because the spectra have been reconstructed from masked values, this
#  is not exactly correct in this case: we'll use the values computed
#  in the file compute_sdss_pca.py
evals = data['evals'] ** 2
evals_cs = evals.cumsum()
evals_cs /= evals_cs[-1]
evecs = data['evecs']
spec_mean = spectra.mean(0)

#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 4, 8, 20]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

### Caveats I

PCA is a linear process, whereas the variations in the data may not be.  So it may not always be appropriate to use and/or may require a relatively large number of components to fully describe any non-linearity.

Note also that PCA can be very impractical for large data sets which exceed the memory per core as the computational requirement goes as $\mathscr{O}(D^3$) and the memory requirement goes as $\mathscr{O}(2D^2)$.

### Missing Data
We have assumed so far that there is no missing data (e.g., bad pixels in the spectrum, etc.).  But often the data set is incomplete.  Since PCA encodes the flux correlation with wavelength (or whatever parameters are in your data set), we can actually use it to determine missing values.

An example is shown below.  Here, black are the observed spectra.  Gray are the regions where we have no data.  Blue is the PCA reconstruction, including the regions where there are no data.  Awesome, isn't it?

```python
# Execute this cell
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import ticker

from astroML.datasets import fetch_sdss_corrected_spectra
from astroML.datasets import sdss_corrected_spectra

#------------------------------------------------------------
# Get spectra and eigenvectors used to reconstruct them
data = fetch_sdss_corrected_spectra()
spec = sdss_corrected_spectra.reconstruct_spectra(data)
lam = sdss_corrected_spectra.compute_wavelengths(data)
evecs = data['evecs']
mu = data['mu']
norms = data['norms']
mask = data['mask']

#------------------------------------------------------------
# plot the results
i_plot = ((lam > 5750) & (lam < 6350))
lam = lam[i_plot]

specnums = [20, 8, 9]
subplots = [311, 312, 313]

fig = plt.figure(figsize=(8, 10))
fig.subplots_adjust(hspace=0)

for subplot, i in zip(subplots, specnums):
    ax = fig.add_subplot(subplot)

    # compute eigen-coefficients
    spec_i_centered = spec[i] / norms[i] - mu
    coeffs = np.dot(spec_i_centered, evecs.T)

    # blank out masked regions
    spec_i = spec[i]
    mask_i = mask[i]
    spec_i[mask_i] = np.nan

    # plot the raw masked spectrum
    ax.plot(lam, spec_i[i_plot], '-', color='k', lw=2,
            label='True spectrum')

    # plot two levels of reconstruction
    for nev in [10]:
        if nev == 0:
            label = 'mean'
        else:
            label = 'N EV=%i' % nev
        spec_i_recons = norms[i] * (mu + np.dot(coeffs[:nev], evecs[:nev]))
        ax.plot(lam, spec_i_recons[i_plot], label=label)

    # plot shaded background in masked region
    ylim = ax.get_ylim()
    mask_shade = ylim[0] + mask[i][i_plot].astype(float) * ylim[1]
    plt.fill(np.concatenate([lam[:1], lam, lam[-1:]]),
             np.concatenate([[ylim[0]], mask_shade, [ylim[0]]]),
             lw=0, fc='k', alpha=0.2)

    ax.set_xlim(lam[0], lam[-1])
    ax.set_ylim(ylim)
    ax.yaxis.set_major_formatter(ticker.NullFormatter())

    if subplot == 311:
        ax.legend(loc=1, prop=dict(size=14))

    ax.set_xlabel('$\lambda\ (\AA)$')
    ax.set_ylabel('normalized flux')

plt.show()
```

The example that we have been using above is "spectral" PCA.  Some examples from the literature include:
- [Francis et al. 1992](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1992ApJ...398..476F&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf)
- [Connolly et al. 1995](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1995AJ....110.1071C&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf)
- [Yip et al. 2004](http://iopscience.iop.org/article/10.1086/425626/meta;jsessionid=31BB5F11B85D2BF4180834DC71BA0B85.c3.iopscience.cld.iop.org)

One can also do PCA on features that aren't ordered (as they were for the spectra).  E.g., if you have $D$ different parameters measured for your objects.  The classic example in astronomy is
[Boroson & Green 1992](http://articles.adsabs.harvard.edu/cgi-bin/nph-iarticle_query?1992ApJS...80..109B&data_type=PDF_HIGH&whole_paper=YES&type=PRINTER&filetype=.pdf).

### Caveats II

One of the things that I don't like about PCA is that the eigenvectors are defined relative to the mean.  So they can be positive or negative and they often don't look anything like the original data itself.  Whereas it is often the case that you might expect that the components would look like, well, the physical components.  For example, quasars are fundamentally galaxies.  So, part of their flux comes from the galaxy that they live in.  But PCA doesn't return any component that looks like a typical galaxy.

## Non-negative Matrix Factorization (NMF)

This is where [Non-negative Matrix Factorizaiton (NMF)](https://en.wikipedia.org/wiki/Non-negative_matrix_factorization) comes in.  Here we are treating the data as a linear sum of positive-definite components.

NMF assumes  any data matrix can be factored into two matrices, $W$ and $Y$, with
$$\begin{equation}
X=W Y,
\end{equation}
$$
where both $W$ and $Y$ are nonnegative.

So, $WY$ is an approximation of $X$. Minimizing the reconstruction error $|| (X - W Y)^2 ||$,
nonnegative bases can be derived through an iterative process.

Note, however, that the iterative process does not guarantee nonlocal minima (like $K$-means and EM), but using
random initialization and cross-validation can be used to find the global minimum.

An example from the literature is [Allen et al. 2008](http://arxiv.org/abs/0810.4231)

In Scikit-Learn the [NMF implementation](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.NMF.html) looks like:

```python
# Execute this cell
import numpy as np
from sklearn.decomposition import NMF
X = np.random.random((100,10)) # 100 points in 10-D
nmf = NMF(n_components=3)
nmf.fit(X)
proj = nmf.transform(X) # project to 3 dimension
comp = nmf.components_ # 3x10 array of components
err = nmf.reconstruction_err_ # how well 3 components capture the data
```

An example (and comparison to PCA) is given below.

```python
# Execute the next 2 cells
# Example from Figure 7.4
# Author: Jake VanderPlas
# License: BSD
%matplotlib inline

import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import NMF
from sklearn.decomposition import RandomizedPCA
from sklearn.decomposition import PCA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)
```

```python
#----------------------------------------------------------------------
# Compute PCA, and NMF components
def compute_PCA_NMF(n_components=5):
    spec_mean = spectra.mean(0)

    # PCA: use randomized PCA for speed
    #pca = RandomizedPCA(n_components - 1)
    pca = PCA(n_components - 1,svd_solver='randomized')
    pca.fit(spectra)
    pca_comp = np.vstack([spec_mean, pca.components_])

    # NMF requires all elements of the input to be greater than zero
    spectra[spectra < 0] = 0
    nmf = NMF(n_components)
    nmf.fit(spectra)
    nmf_comp = nmf.components_

    return pca_comp, nmf_comp

n_components = 5
decompositions = compute_PCA_NMF(n_components)

#----------------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10, 10))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = ['PCA components', 'NMF components']

for i, comp in enumerate(decompositions):
    for j in range(n_components):
        ax = fig.add_subplot(n_components, 3, 3 * j + 1 + i)

        ax.yaxis.set_major_formatter(plt.NullFormatter())
        ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
        if j < n_components - 1:
            ax.xaxis.set_major_formatter(plt.NullFormatter())
        else:
            ax.set_xlabel('wavelength (Angstroms)')

        ax.plot(wavelengths, comp[j], '-k', lw=1)

        # plot zero line
        xlim = [3000, 7999]
        ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
        ax.set_xlim(xlim)

        if j == 0:
            ax.set_title(titles[i])

        if titles[i].startswith('PCA') or titles[i].startswith('ICA'):
            if j == 0:
                label = 'mean'
            else:
                label = 'component %i' % j
        else:
            label = 'component %i' % (j + 1)

        ax.text(0.03, 0.94, label, transform=ax.transAxes,
                ha='left', va='top')

        for l in ax.get_xticklines() + ax.get_yticklines():
            l.set_markersize(2)

        # adjust y limits
        ylim = plt.ylim()
        dy = 0.05 * (ylim[1] - ylim[0])
        ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

plt.show()
```

## Independent Component Analysis (ICA)

For data where the components are statistically independent (or nearly so) [Independent Component Analysis (ICA)](https://en.wikipedia.org/wiki/Independent_component_analysis) has become a popular method for separating mixed components.  The classical example is the so-called "cocktail party" problem.  This is illustrated in the following figure from Hastie, Tibshirani, and Friedman (Figure 14.27 on page 497 in my copy, so they have clearly added some stuff!).  Think of the "source signals" as two voices at a party.  You are trying to concentrate on just one voice.  What you hear is something like the "measured signals" pattern.  You could run the data through PCA and that would do an excellent job of reconstructing the signal with reduced dimensionality, but it wouldn't actually isolate the different physical components (bottom-left panel).  ICA on the other hand can (bottom-right panel).

.](images/HastieFigure14_37.png)

[Hastie et al.](https://web.stanford.edu/~hastie/ElemStatLearn/): "ICA applied to multivariate data looks for a sequence of orthogonal projections such that the projected data look as far from Gaussian as possible. With pre-whitened data, this amounts to looking for
components that are as independent as possible."

In short you want to find components that are maximally non-Gaussian since the sum of 2 random variables will be more Gaussian than either of the components (remember the Central Limit Theorem).  Hastie et al. illustrate this as follows:

ICA is a good choice for a complex system with relatively indepent components.  For example a galaxy is roughly a linear combination of cool stars and hot stars, and a quasar is just a galaxy with others component from an accretion disk and emission line regions.  Ideally we want "eigenvectors" that are aligned with those physical traits/regions as opposed to mathematical constructs.

The basic call to the [FastICA algoirthm](http://scikit-learn.org/stable/modules/generated/sklearn.decomposition.FastICA.html) in Scikit-Learn looks like:

```python
# Execute this cell
import numpy as np
from sklearn.decomposition import FastICA
X = np.random.normal(size=(100,2)) # 100 objects in 2D
R = np.random.random((2,5)) # mixing matrix
X = np.dot(X,R) # Simulation of a 5D data space
ica = FastICA(2) # Now reproject to 2-D
ica.fit(X)
proj = ica.transform(X) # 100x2 projection of the data
comp = ica.components_ # 2x5 matrix of independent components
## sources = ica.sources_ # 100x2 matrix of sources
```

Execute the next 2 cells to produce a plot showing the ICA components.

```python
%matplotlib inline
#Example from Andy Connolly
import numpy as np
from matplotlib import pyplot as plt

from sklearn.decomposition import FastICA

from astroML.datasets import sdss_corrected_spectra
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Download data
data = sdss_corrected_spectra.fetch_sdss_corrected_spectra()
spectra = sdss_corrected_spectra.reconstruct_spectra(data)
wavelengths = sdss_corrected_spectra.compute_wavelengths(data)

#----------------------------------------------------------------------
# Compute PCA
np.random.seed(500)
nrows = 500
n_components = 5
ind = np.random.randint(spectra.shape[0], size=nrows)

spec_mean = spectra[ind].mean(0)
# spec_mean = spectra[:50].mean(0)

ica = FastICA(n_components - 1)
ica.fit(spectra[ind])
ica_comp = np.vstack([spec_mean,ica.components_]) #Add the mean to the components
```

```python
#Make plots
fig = plt.figure(figsize=(10, 8))
fig.subplots_adjust(left=0.05, right=0.95, wspace=0.05,
                    bottom=0.1, top=0.95, hspace=0.05)

titles = 'ICA components'

for j in range(n_components):

    # plot the components
    ax = fig.add_subplot(n_components, 2, 2*j+2)
    ax.yaxis.set_major_formatter(plt.NullFormatter())
    ax.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax.plot(wavelengths, ica_comp[j], '-k', lw=1)

    # plot zero line
    xlim = [3000, 7999]
    ax.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax.set_xlim(xlim)

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    # plot the first j spectra
    ax2 = fig.add_subplot(n_components, 2, 2*j+1)
    ax2.yaxis.set_major_formatter(plt.NullFormatter())
    ax2.xaxis.set_major_locator(plt.MultipleLocator(1000))
    if j < n_components - 1:
        ax2.xaxis.set_major_formatter(plt.NullFormatter())
    else:
        ax2.set_xlabel(r'wavelength ${\rm (\AA)}$')
    ax2.plot(wavelengths, spectra[j], '-k', lw=1)

    # plot zero line
    ax2.plot(xlim, [0, 0], '-', c='gray', lw=1)
    ax2.set_xlim(xlim)

    if j == 0:
        ax.set_title(titles, fontsize='medium')

    if j == 0:
        label = 'mean'
    else:
        label = 'component %i' % j

    # adjust y limits
    ylim = plt.ylim()
    dy = 0.05 * (ylim[1] - ylim[0])
    ax2.set_ylim(ylim[0] - dy, ylim[1] + 4 * dy)

    ax.text(0.02, 0.95, label, transform=ax.transAxes,
            ha='left', va='top', bbox=dict(ec='w', fc='w'),
            fontsize='small')

plt.show()
```

As with PCA and NMF, we can similarly do a reconstruction:

```python
# Execute this cell
#------------------------------------------------------------
# Find the coefficients of a particular spectrum
spec = spectra[1]
evecs = data['evecs']
coeff = np.dot(evecs, spec - spec_mean)

#------------------------------------------------------------
# Plot the sequence of reconstructions
fig = plt.figure(figsize=(8, 8))
fig.subplots_adjust(hspace=0)

for i, n in enumerate([0, 2, 4, 8]):
    ax = fig.add_subplot(411 + i)
    ax.plot(wavelengths, spec, '-', c='gray')
    ax.plot(wavelengths, spec_mean + np.dot(coeff[:n], evecs[:n]), '-k')

    if i < 3:
        ax.xaxis.set_major_formatter(plt.NullFormatter())

    ax.set_ylim(-2, 21)
    ax.set_ylabel('flux')

    if n == 0:
        text = "mean"
    elif n == 1:
        text = "mean + 1 component\n"
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]
    else:
        text = "mean + %i components\n" % n
        #text += r"$(\sigma^2_{tot} = %.2f)$" % evals_cs[n - 1]

    ax.text(0.01, 0.95, text, ha='left', va='top', transform=ax.transAxes)

fig.axes[-1].set_xlabel(r'${\rm wavelength\ (\AA)}$')
plt.show()
```

Ivezic, Figure 7.4 compares the components found by the PCA, ICA, and NMF algorithms.  Their differences and similarities are quite interesting.

If you think that I was pulling your leg about the cocktail problem, try it yourself!
Load the code instead of running it and see what effect changing some things has.

```python
%run code/plot_ica_blind_source_separation.py
```

Let's revisit the digits sample and see what PCA, NMF, and ICA do for it.

```python
## Execute this cell to load the digits sample
%matplotlib inline
import numpy as np
from sklearn.datasets import load_digits
from matplotlib import pyplot as plt
digits = load_digits()
grid_data = np.reshape(digits.data[0], (8,8)) #reshape to 8x8
plt.imshow(grid_data, interpolation = "nearest", cmap = "bone_r")
print(grid_data)
X = digits.data
y = digits.target
```

Do the PCA transform, projecting to 2 dimensions and plot the results.

```python
# PCA
from sklearn.decomposition import PCA
pca = PCA(n_components = 2)
pca.fit(X)
X_reduced = pca.transform(___)

plt.scatter(X_reduced[:,___], X_reduced[:,___], c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

Similarly for NMF and ICA

```python
# NMF
from sklearn.decomposition import ___
nmf = NMF(___)
nmf.___(___)
X_reduced = nmf.___(___)

plt.scatter(___, ___, c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

```python
# ICA
from sklearn.decomposition import ___
ica = FastICA(___)
ica.___(___)
X_reduced = ica.___(___)

plt.scatter(___, ___, c=y, cmap="nipy_spectral", edgecolor="None")
plt.colorbar()
```

Take a second to think about what ICA is doing.  What if you had digits from digital clocks instead of handwritten?

I wasn't going to introduce [Neural Networks](https://en.wikipedia.org/wiki/Artificial_neural_network) yet, but it is worth noting that Scikit-Learn's [`Bernoulli Restricted Boltzman Machine (RBM)`](http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.BernoulliRBM.html) is discussed in the [(unsupervised) neural network](http://scikit-learn.org/stable/modules/neural_networks_unsupervised.html) part of the User's Guide and is relevant here as the data input must be either binary or values between 0 and 1, which is the case that we have here.

We could think about doing dimensional reduction of the digits data set in another way.  There are 64 pixels in each of our images.  Presumably all of them aren't equally useful.  Let's figure out exactly which pixels are the most relevant.  We'll use Scikit-Learn's [`RandomForestRegressor`](http://scikit-learn.org/stable/modules/generated/sklearn.ensemble.RandomForestRegressor.html).  We won't get to regression until next week, but you don't need to understand the algorithm to do this, just look at the inputs and outputs.  Which pixels are the most important?  As a bonus see if you can plot digit images with those pixels highlighted.

```python
from sklearn.ensemble import RandomForestRegressor
RFreg = RandomForestRegressor()# Complete or leave blank as you see fit
RFreg.fit(X,y)# Do Fitting
importances = RFreg.feature_importances_# Determine "importances"
pixelorder = np.argsort(importances)[::-1] #Rank importances (highest to lowest)
print(pixelorder)

plt.figure()
plt.imshow(np.reshape(importances,(8,8)),interpolation="nearest")
plt.show()
```

```python

```


*This notebook contains course material from [CBE30338](https://jckantor.github.io/CBE30338)
by Jeffrey Kantor (jeff at nd.edu); the content is available [on Github](https://github.com/jckantor/CBE30338.git).
The text is released under the [CC-BY-NC-ND-4.0 license](https://creativecommons.org/licenses/by-nc-nd/4.0/legalcode),
and code is released under the [MIT license](https://opensource.org/licenses/MIT).*

< [Manometer Models and Dynamics](http://nbviewer.jupyter.org/github/jckantor/CBE30338/blob/master/notebooks/03.08-Manometer-Models-and-Dynamics.ipynb) | [Contents](toc.ipynb) | [PID Control](http://nbviewer.jupyter.org/github/jckantor/CBE30338/blob/master/notebooks/04.00-PID_Control.ipynb) >

# Modeling the Coronavirus COVID-19 Outbreak

## The SARS-CoV-2 Coronavirus

The disease now called [COVID-19](https://www.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6) is caused by the human coronavirus SARS-CoV-2. Human coronaviruses were first identified in the 1960's. There are currently four coronaviruses endemic to human populations around the world:

* 229E (alpha coronavirus)
* NL63 (alpha coronavirus)
* OC43 (beta coronavirus)
* HKU1 (beta coronavirus)

These four common coronaviruses typically cause an upper respiratory disease which, in a minority of cases, progresses to pneumonia. Most people will suffer from at least one of coronaviruses during their lifetimes which cause about a quarter of all common colds.

In recent decades, three coronaviruses that normally infect animals have evolved to infect humans. These include:

* MERS-CoV (the beta coronavirus that causes Middle East Respiratory Syndrome, or MERS)
* SARS-CoV (the beta coronavirus that causes severe acute respiratory syndrome, or SARS)
* SARS-CoV-2 (the novel coronavirus that causes coronavirus disease COVID-19)

The last of these, now called SARS-CoV-2, first appeared in December, 2019, at a seafood market in Wuhan (Hubei, China). The rapid spread of SARS-CoV-2 in Wuhan, and appearance in other locations around the globe, has resulted in the World Health Organization (WHO) declaring a global health emergency. Countries around the globe are mobilizing to track the virus and control new outbreaks.

At this stage, it is too early to know if these efforts will be successful in preventing COVID-19 from becoming a pandemic, and later an endemic disease with a global footprint. You can find the most recent data on the current outbreak at [Coronavirus COVID-19 Global Cases by Johns Hopkins CSSE](https://gisanddata.maps.arcgis.com/apps/opsdashboard/index.html#/bda7594740fd40299423467b48e9ecf6).

The purpose of this notebook is to demostrate the modeling of an infectious epidemic using the latest available data for COVID-19. The models and data used in this notebook have been extracted from a rapidly emerging and changing literature.

## References

Keeling, Matt J., and Pejman Rohani. Modeling Infectious Diseases in Humans and Animals. Princeton University Press, 2008. JSTOR, www.jstor.org/stable/j.ctvcm4gk0. Accessed 25 Feb. 2020.

Boldog, Péter, et al. "Risk Assessment of Novel Coronavirus COVID-19 Outbreaks Outside China." Journal of Clinical Medicine 9.2 (2020): 571. https://www.mdpi.com/2077-0383/9/2/571

## Model 1. SIR model for an infectious disease

The SIR model is simple compartment model for an infectious disease that describes key phenomena encountered in epidemiology. In the SIR model, a population is broken into three sub-groups corresponding to stages of the disease:

* **Susceptible.** The subpopulation susceptible to acquire the disease. For SARS-CoV-2, the assumption is that everyone who has not previously acquired the disease is susceptible to infection.
* **Infectious.** The subpopulation that has become infective.
* **Recovered.** The subpopulation that has recovered from infection, and presumed to be no longer susceiptible to the disease.

Neglecting demographic processes of birth and death from other causes, and assuming a negligible death rate due to infectious disease at issue, the progression of an epidemic can be modeled by rate processesl

$$\text{Susceptible}
\xrightarrow{\frac{\beta S I}{N}}
\text{Infectious}
\xrightarrow{\gamma I}
\text{Recovered} $$

The rate processes are modeled as follows.

* $\frac{\beta S I}{N}$ is the rate at which susecptible population encounters the infected population resulting in trasmission of the disease. $S$ is the size of the susceptible population. $\beta$ is a the model parameters with units of 1/day.
* $\gamma I$ is the rate at which infected population recovers and becomes resistent to further infection. $I$ is the size of the infective population.

A model for the spread of an infectious disease in a uniform population is given by the deterministic SIR equations

\begin{align*}
\frac{dS}{dt} & = -\frac{\beta S I}{N} \\
\frac{dI}{dt} & = \frac{\beta S I}{N} - \gamma I \\
\frac{dR}{dt} & = \gamma I
\end{align*}

The model becomes more transparent by working with population fractions rather than raw population counts. To this end, define

\begin{align}
s = \frac{S}{N} \qquad
i = \frac{I}{N} \qquad
r = \frac{R}{N}
\end{align}

After substitution, this results in a system of four equations.

\begin{align*}
\frac{ds}{dt} & = -\beta s i \\
\frac{di}{dt} & = \beta s i - \gamma i \\
\frac{dr}{dt} & = \gamma i
\end{align*}

where $s + i + r = 1$ is an invariant.

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# parameter values
R0 = 2.4
t_infective = 5.1 + 3.3

# initial number of infected and recovered individuals
i_initial = 1/20000
r_initial = 0.00
s_initial = 1 - i_initial - r_initial

gamma = 1/t_infective
beta = R0*gamma

# SIR model differential equations.
def deriv(x, t, beta, gamma):
    s, i, r = x
    dsdt = -beta * s * i
    didt = beta * s * i - gamma * i
    drdt =  gamma * i
    return [dsdt, didt, drdt]

t = np.linspace(0, 180, 2000)
x_initial = s_initial, i_initial, r_initial
soln = odeint(deriv, x_initial, t, args=(beta, gamma))
s, i, r = soln.T

# plot the data
fig = plt.figure(figsize=(12,6))
ax = [fig.add_subplot(221, axisbelow=True),
      fig.add_subplot(223),
      fig.add_subplot(122)]

ax[0].plot(t, s, lw=3, label='Susceptible')
ax[0].plot(t, i, lw=3, label='Infective')
ax[0].plot(t, r, lw=3, label='Recovered')
ax[0].set_xlabel('Time /days')
ax[0].set_ylabel('Fraction')

ax[1].plot(t, i, lw=3, label='Infective')
ax[1].set_xlabel('Time /days')
ax[1].set_ylabel('Fraction')

ax[2].plot(s, i, lw=3, label='trajectory')
ax[2].plot([1/R0, 1/R0], [0, 1], '--', lw=3, label='di/dt = 0 nullcline')
ax[2].set_aspect('equal')
ax[2].set_ylim(0, 1)
ax[2].set_xlim(0, 1)
ax[2].set_xlabel('Susceptible')
ax[2].set_ylabel('Infective')

for a in ax:
    a.grid(True)
    a.legend()

plt.tight_layout()
```

* The region from $i = 0, s \in [0, \frac{1}{R_0}]$ are all stable attractors. This is the region of **herd immunity**.

* The region from $i=0, s \in (\frac{1}{R_0}, 1]$ are all unstable. Any perturbation with $i > 0$ results in a disease outbreak.

* The size of the outbreak depends on the size of teh suscepitble population. The more susceptibles, the larger the outbeak.

## Model 2. SEIR model

The SEIR model extends the SIR model by adding an additional population compartment containing those individuals who have been exposed to the virus but not yet infective.

* **Exposed.** The subpopulation that has been exposed to the disease but not yet infective.

The compartment model can be diagrammed as follows.

$$\text{Susceptible}
\xrightarrow{\frac{\beta S I}{N}}
\text{Exposed}
\xrightarrow{\alpha E}
\text{Infectious}
\xrightarrow{\gamma I}
\text{Recovered} $$

The rate processes are modeled as follows.

* $\frac{\beta S I}{N}$ is the rate at which susecptible population encounters the infected population resulting in trasmission of the disease. $S$ is the size of the susceptible population. $\beta$ is a the model parameters with units of 1/day.
* $\alpha E$ is the rate at which exposed population becomes infective, where $E$ is the size of the exposed population. The average period of time in the exposed state is the incubation period of the disease, and equal to $\frac{1}{\alpha}$.
* $\gamma I$ is the rate at which infected population recovers and becomes resistent to further infection. $I$ is the size of the infective population.

An elementary model for the spread of an infectious disease in a uniform population is given by the deterministic SEIR equations}

After substitution, this results in a system of four equations.

\begin{align*}
\frac{ds}{dt} & = -\beta s i \\
\frac{de}{dt} & = \beta s i - \alpha e \\
\frac{di}{dt} & = \alpha e  - \gamma i \\
\frac{dr}{dt} & = \gamma i
\end{align*}

where $s + e + i + r = 1$ is an invariant.

```python
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# parameter values
R0 = 2.4
t_incubation = 5.1
t_infective = 3.3

# initial number of infected and recovered individuals
e_initial = 1/20000
i_initial = 0.00
r_initial = 0.00
s_initial = 1 - e_initial - i_initial - r_initial

alpha = 1/t_incubation
gamma = 1/t_infective
beta = R0*gamma

# SEIR model differential equations.
def deriv(x, t, alpha, beta, gamma):
    s, e, i, r = x
    dsdt = -beta * s * i
    dedt =  beta * s * i - alpha * e
    didt = alpha * e - gamma * i
    drdt =  gamma * i
    return [dsdt, dedt, didt, drdt]

t = np.linspace(0, 160, 160)
x_initial = s_initial, e_initial, i_initial, r_initial
soln = odeint(deriv, x_initial, t, args=(alpha, beta, gamma))
s, e, i, r = soln.T

# plot the data
fig = plt.figure(figsize=(12,6))
ax = [fig.add_subplot(221, axisbelow=True),
      fig.add_subplot(223),
      fig.add_subplot(122)]

ax[0].plot(t, s, lw=3, label='Susceptible')
ax[0].plot(t, e, lw=3, label='Exposed')
ax[0].plot(t, i, lw=3, label='Infective')
ax[0].plot(t, r, lw=3, label='Recovered')
ax[0].set_xlabel('Time /days')
ax[0].set_ylabel('Fraction')

ax[1].plot(t, i, lw=3, label='Infective')
ax[1].plot(t, e, lw=3, label='Exposed')
ax[1].set_xlabel('Time /days')
ax[1].set_ylabel('Fraction')

ax[2].plot(s, i, lw=3, label='trajectory')
ax[2].plot([1/R0, 1/R0], [0, 1], '--', lw=3, label='SIR nullcline')
ax[2].set_aspect('equal')
ax[2].set_ylim(0, 1)
ax[2].set_xlim(0, 1)
ax[2].set_xlabel('Susceptible')
ax[2].set_ylabel('Infective')

for a in ax:
    a.grid(True)
    a.legend()

plt.tight_layout()
```

* The addition of an exposed population compartment slows the outbreak, but doesn't appear to reduce the number of people ultimately infected by the disease.

* What are the campus policy implications of these results?

## Model 3. Mitigation and Social Distancing

* Pan, Jinhua, et al. "Effectiveness of control strategies for Coronavirus Disease 2019: a SEIR dynamic modeling study." medRxiv (2020). https://www.medrxiv.org/content/10.1101/2020.02.19.20025387v3.full.pdf

The lack of a vaccine reduces the options for controlling the COVID-19 outbreak. Current efforts are focused on 'social distancing' designed to reduce transmission of the virus from individuals in the infective state to susceptible individuals.

For the purposes of modeling, we introduce a control parameter $u$ indicating the effectiveness of these efforts. $u=0$ corresponds to no controls, $u=1$ corresponds to perfect isolation of infective individuals. The purpose of this model is to explore how a social distancing stragtegy affects the outcome of an epidemic.

* **Exposed.** The subpopulation that has been exposed to the disease but not yet infective.

The compartment model can be diagrammed as follows.

$$\text{Susceptible}
\xrightarrow{(1-u)\frac{\beta S I}{N}}
\text{Exposed}
\xrightarrow{\alpha E}
\text{Infectious}
\xrightarrow{\gamma I}
\text{Recovered} $$

The rate processes are modeled as follows.

* $(1-u)\frac{\beta S I}{N}$ is the rate at which susecptible population encounters the infected population resulting in trasmission of the disease. $u$ describes the effectiveness on any public health interventions to control transmission of the disease. $u=0$ means no effective public health interventions, $u=1$ means total elimination of disease transmission..

After substitution, this results in a system of four equations.

\begin{align*}
\frac{ds}{dt} & = -(1-u)\beta s i \\
\frac{de}{dt} & = (1-u)\beta s i - \alpha e \\
\frac{di}{dt} & = \alpha e  - \gamma i \\
\frac{dr}{dt} & = \gamma i
\end{align*}

where $s + e + i + r = 1$ is an invariant.

```python
#@title SEIR with Control { run: "auto" }
R0 = 2.2 #@param {type:"slider", min:0.8, max:6, step:0.1}
t_incubation = 5.1 #@param {type:"slider", min:2, max:8, step:0.1}
t_infective = 3.3 #@param {type:"slider", min:1, max:8, step:0.1}
u = 0.31 #@param {type:"slider", min:0, max:1, step:0.01}
import numpy as np
from scipy.integrate import odeint
import matplotlib.pyplot as plt

# parameter values

# initial number of infected and recovered individuals
e_initial = 1/20000
i_initial = 0.00
r_initial = 0.00
s_initial = 1 - e_initial - i_initial - r_initial

alpha = 1/t_incubation
gamma = 1/t_infective
beta = R0*gamma

# SEIR model differential equations.
def deriv(x, t, alpha, beta, gamma):
    s, e, i, r = x
    dsdt = -(1-u)*beta * s * i
    dedt =  (1-u)*beta * s * i - alpha * e
    didt = alpha * e - gamma * i
    drdt =  gamma * i
    return [dsdt, dedt, didt, drdt]

t = np.linspace(0, 365, 365)
x_initial = s_initial, e_initial, i_initial, r_initial
soln = odeint(deriv, x_initial, t, args=(alpha, beta, gamma))
s, e, i, r = soln.T

# plot the data
fig = plt.figure(figsize=(12,6))
ax = [fig.add_subplot(221, axisbelow=True),
      fig.add_subplot(223),
      fig.add_subplot(122)]

ax[0].plot(t, s, lw=3, label='Susceptible')
ax[0].plot(t, e, lw=3, label='Exposed')
ax[0].plot(t, i, lw=3, label='Infective')
ax[0].plot(t, r, lw=3, label='Recovered')
ax[0].set_xlabel('Time /days')
ax[0].set_ylabel('Fraction')

ax[1].plot(t, i, lw=3, label='Infective')
ax[1].plot(t, e, lw=3, label='Exposed')
ax[1].set_xlabel('Time /days')
ax[1].set_ylabel('Fraction')

ax[2].plot(s, i, lw=3, label='trajectory')
ax[2].plot([1/R0, 1/R0], [0, 1], '--', lw=3, label='SIR nullcline')
ax[2].set_aspect('equal')
ax[2].set_ylim(0, 1)
ax[2].set_xlim(0, 1)
ax[2].set_xlabel('Susceptible')
ax[2].set_ylabel('Infective')

for a in ax:
    a.grid(True)
    a.legend()

plt.tight_layout()
```

Social distancing has several beneficial effects:

* Slows down the progress of the epidemic.
* Seduces the fraction of the population infected at any point in time, thereby reducing strain on health care resources.
* Reduces the number of individuals ultimately experiencing the disease. For a disease with a non-zero mortaility, this saves lives.

## Model 4. Improving the fidelity of the model.

Boldog, et al.

```python
from scipy.integrate import odeint
import numpy as np
import matplotlib.pyplot as plt

u = 0.3
mu = 0
alpha = 1/5.1   # incubation period
R0 = 2.4
gamma = 3.3
beta = R0*gamma

N = 331000000

def SEIR(x, t):
    S, E1, E2, I1, I2, I3, R = x
    dS = -(1-u)*beta*S*(I1 + I2 + I3)/N
    dE1 = -dS - 2*alpha*E1
    dE2 = 2*alpha*E1 - 2*alpha*E2
    dI1 = 2*alpha*E2 - 3*gamma*I1 - mu*I1
    dI2 = 3*gamma*I1 - 3*gamma*I2 - mu*I2
    dI3 = 3*gamma*I2 - 3*gamma*I3 - mu*I3
    dR = 3*gamma*I3
    return [dS, dE1, dE2, dI1, dI2, dI3, dR]

IC = [N, 1, 0, 0, 0, 0, 0]
t = np.linspace(0, 180, 1000)

soln = odeint(SEIR, IC, t)

S = soln[:, 0]
E = soln[:, 1] + soln[:, 2]
I = soln[:, 3] + soln[:, 4] + soln[:, 5]
R = soln[:, 6]

plt.plot(t, S, label = 'Susceptable')
plt.plot(t, E, label = 'Exposed')
plt.plot(t, I, label = 'Infectious')
plt.plot(t, R, label = 'Recovered')
plt.ylim([1, 1.2*N])

plt.legend()
```

## Additional Modeling Opportunities. Transportation, power law kinetics, etc.

* Ziff, Robert M., and Anna L. Ziff. "Fractal kinetics of COVID-19 pandemic." medRxiv (2020). https://www.medrxiv.org/content/10.1101/2020.02.16.20023820v1

* Peng, Liangrong, et al. "Epidemic analysis of COVID-19 in China by dynamical modeling." arXiv preprint arXiv:2002.06563 (2020).  https://arxiv.org/abs/2002.06563

* Wu, Joseph T., Kathy Leung, and Gabriel M. Leung. "Nowcasting and forecasting the potential domestic and international spread of the 2019-nCoV outbreak originating in Wuhan, China: a modelling study." The Lancet (2020). https://www.thelancet.com/journals/lancet/article/PIIS0140-6736(20)30260-9/fulltext

* Tuite, Ashleigh R., and David N. Fisman. "Reporting, Epidemic Growth, and Reproduction Numbers for the 2019 Novel Coronavirus (2019-nCoV) Epidemic." Annals of Internal Medicine (2020). https://annals.org/aim/fullarticle/2760912

```python

```

< [Manometer Models and Dynamics](http://nbviewer.jupyter.org/github/jckantor/CBE30338/blob/master/notebooks/03.08-Manometer-Models-and-Dynamics.ipynb) | [Contents](toc.ipynb) | [PID Control](http://nbviewer.jupyter.org/github/jckantor/CBE30338/blob/master/notebooks/04.00-PID_Control.ipynb) >


# Applied Stochastic Processes HW03

**11510691 程远$\DeclareMathOperator*{\argmin}{argmin}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\EE}[2][\,\!]{\mathbb{E}_{#1}\left[#2\right]}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathrm{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}$星**

## Question 1

$\P{1}$

$\bspace$Let $N$ denote the number needed. And let $X_i$ denote the trials thus we have

$$\begin{align}
\EE{N} &= \EE{\EE{N \mid X_1}} \\
&= \EE{N \mid X_1 = H} \cdot P\CB{X_1 = H} + \EE{N \mid X_1 = T} \cdot P\CB{X_1 = T}
\end{align}$$

$\bspace$For $\EE{N \mid X_1 = H}$, since we already have a head toss, what we need is one tail toss, and that's nothing but a geometric variable, similar for $\EE{N \mid X_1 = T}$. Therefore,

$$\begin{align}
\EE{N} &= \EE{N \mid X_1 = H} \cdot P\CB{X_1 = H} + \EE{N \mid X_1 = T} \cdot P\CB{X_1 = T} \\
&= \P{1 + \ffrac{1} {1-p}} \cdot p + \P{1 + \ffrac{1} {p}} \cdot \P{1-p} \\
&= 1 + \ffrac{p} {1-p} + \ffrac{1-p} {p} = \ffrac{p} {1-p} + \ffrac{1} {p}
\end{align}$$

$\P{2}$

$\bspace$Now we let it be $M$ then:

$$\begin{align}
\EE{M} &= \EE{\EE{M \mid X_1}} \\
&= \EE{M \mid X_1 = H} \cdot P\CB{X_1 = H} + \EE{M \mid X_1 = T} \cdot P\CB{X_1 = T} \\
&= \P{1 + \EE{N}}\cdot P\CB{X_1 = H} + \P{1 + \ffrac{2} {p}}\cdot \P{1-p} \\
&= 2 + \ffrac{p^2} {1-p} + \ffrac{2\P{1-p}} {p} = \ffrac{p^2} {1-p} + \ffrac{2} {p}
\end{align}$$

$Remark$

>And we can guess the following statement, blahblahblah...

## Question 2

$\bspace$First we need to calculate the marginal distribution of $Y$

$$\begin{align}
f_Y\P{y} &= \int_{-\infty}^{\infty} f\P{x,y} \;\dd{x} \\
&= 4ye^{-y} \int_{y}^{\infty} xe^{-x} - y e^{-x} \;\dd{x}\\
&= 4ye^{-y}\P{\big.-\P{1+x}e^{-x}\big|_y^{\infty} - y \P{\big.-e^{-x}\big|_{y}^{\infty}}}\\
&= 4ye^{-2y}, \bspace y \geq 0
\end{align}$$

$\bspace$and $f_Y\P{y} = 0$ elsewhere. Thus we have

$$\begin{align}
\EE{X \mid Y = y} &= \int_{-\infty}^{\infty} x \cdot \ffrac{f\P{x,y}} {f_Y\P{y}} \; \dd{x}\\
&= \int_{y}^{\infty} x \cdot \P{x-y}e^{y-x} \;\dd{x} \\
&= e^{y} \P{\Big. -\P{2 + 2x + x^2}e^{-x} \Big|_{y}^{\infty}-y\P{-\big.\P{1+x}e^{-x}\big|_{y}^{\infty}}}\\
&= 2+y, \bspace y \geq 0
\end{align}$$

## Question 3

$\bspace$Since $X$ and $Y$ are non-negative independent $r.v.$s, $F_Z\P{z} = P\CB{X+Y \leq z} = 0$ and thus $f_Z\P{z} = 0$ for $z < 0$.

$\bspace$Now when $Z \geq 0$, we condition the probability on $Y$ which gives:

$$\begin{align}
F_Z\P{z} &= P\CB{Z \leq z} = P\CB{X + Y \leq z} \\
&= \int_{-\infty}^{\infty} P\CB{X + Y \leq z \mid Y = y} \cdot f_Y\P{y} \;\dd{y} \\
&= \int_{-\infty}^{\infty} P\CB{X \leq z-y}\cdot f_Y\P{y} \;\dd{y} \\
&\using{z \geq y \geq 0} \int_{0}^{z} F_X\P{z-y}\cdot f_Y\P{y} \;\dd{y} \\
\end{align}$$

$\bspace$take the derivative on $z$ we have

$$
\begin{align}
f_Z\P{z} &= F_Z'\P{z}\\
&= \int_{0}^{z} f_X\P{z-y}\cdot f_Y\P{y} \;\dd{y} + F_X\P{z-z}f_Y\P{z}\cdot 1-F_X\P{z-0}f_Y\P{0}\cdot 0 \\
&= \int_{0}^{z} f_X\P{z-y}\cdot f_Y\P{y} \;\dd{y}, \bspace z \geq 0
\end{align}$$

$\bspace$then combine these two parts we reach our final conclusion that

$$f_Z\P{z} = \begin{cases}
\d{\int_{0}^{z} f_X\P{z-y}\cdot f_Y\P{y} \;\dd{y}}, &\text{if }z \geq 0\\[0.6em]
0, &\text{if } z<0
\end{cases}$$

***
$\bspace$Now that $Y$ can be regard as a copy from $X$, the conclusion won't change when $Z < 0$ and if $Z \geq 0$, we have

$$F_Z\P{z} = P\CB{2X \leq z} = \int_{0}^{z/2} f_X\P{x}\;\dd{x}$$

$\bspace$thus $f_Z\P{z} = F_Z'\P{z} = \d{\int_{0}^{z/2} 0\;\dd{x}} + f_X\P{\ffrac{z} {2}}\cdot \ffrac{1} {2} - f_X\P{0}\cdot 0 = \ffrac{1} {2}\,f_X\P{\ffrac{z} {2}}$ and finally,

$$f_Z\P{z} = \begin{cases}
\ffrac{1} {2}\,f_X\P{\ffrac{z} {2}}, &\text{if }z \geq 0\\[0.6em]
0, &\text{if } z<0
\end{cases}$$

## Question 4

$$\EE{X \mid X > a} = \int_{-\infty}^{\infty} x \cdot \ffrac{P\CB{X = x, X > a}} {P\CB{X > a}}\;\dd{x} = \ffrac{1} {P\CB{X > a}}\int_{a}^{\infty} x \cdot f\P{x}\;\dd{x}$$

$\bspace$Now given $f_Y\P{y} = \begin{cases}
4e^{-4y}, &\text{if } y > 0\\
0, &\ow
\end{cases}$, we have

$$\EE{Y \mid Y > 2} = \ffrac{1} {1 - F_Y\P{2}} \int_{2}^{\infty} y \cdot 4e^{-4y}\;\dd{y} = \ffrac{1} {e^{-4\times 2}} \Big. \P{-\ffrac{1} {4} - y}e^{-4y} \Big|_{2}^{\infty} = 2.25$$

## Question 5

$\P{\text{a}}$

$\bspace$Let $X_i$ denote his $\texttt{i-}$th choice, $N$ the expected number of days.

$$\begin{align}
\EE{N} &= \EE{\EE{N \mid X_1}}\\
&= \EE{N \mid X_1 = 1} \cdot P\CB{X_1 = 1} + \EE{N \mid X_1 = 2} \cdot P\CB{X_1 = 2} + \EE{N \mid X_1 = 3} \cdot P\CB{X_1 = 3} \\
& = \P{2+\EE{N}} \cdot 0.5 + \P{3 + \EE{N}} \cdot 0.3 + 0 \cdot 0.2
\end{align}$$

$\bspace$Solve the equation above we can find that $\EE{N} = 9.5$ and thus the expected number of days until he reaches freedom is $10$.

$\P{\text{b}}$

$\bspace$In this problem we change the notation to $M$.

$$\overbrace{\begin{array}{ccc}
\begin{array}{c}
1 \\
\overbrace{\begin{array}{cc}
2 & 3 \\
3 & 0 \\
\downarrow & \downarrow \\
\ffrac{1} {6} & \ffrac{1} {6}
\end{array}}
\end{array} &
\begin{array}{c}
2 \\
\overbrace{\begin{array}{cc}
1 & 3 \\
3 & 0 \\
\downarrow & \downarrow \\
\ffrac{1} {6} & \ffrac{1} {6}
\end{array}}
\end{array} &
\begin{array}{c}
3 \\
\overbrace{\begin{array}{cc}
0 & 0 \\
0 & 0 \\
\downarrow & \downarrow \\
\ffrac{1} {6} & \ffrac{1} {6}
\end{array}}
\end{array}
\end{array}}$$

$\bspace$Thus, $\EE{M} = \ffrac{1} {6}\P{5+2+5+3} = 2.5$ and the expected number of days until he reaches freedom is $3$.

$\P{\text{c}}$

$$\begin{align}
\EE{N}^2 &= \EE{\EE{N^2 \mid X_1}}\\
&= \EE{N^2 \mid X_1 = 1} \cdot P\CB{X_1 = 1} + \EE{N^2 \mid X_1 = 2} \cdot P\CB{X_1 = 2} + \EE{N^2 \mid X_1 = 3} \cdot P\CB{X_1 = 3} \\
& = \EE{\P{2+N}^2} \cdot 0.5 + \EE{\P{3 + N}^2} \cdot 0.3 + 0 \cdot 0.2
\end{align}$$

$\bspace$Substitute $\EE{N}$ with $9.5$ and we can solve this, saying that $\EE{N^2} = 204$. Thus, $\Var{N} = \EE{N^2} - \P{\EE{N}}^2 = 113.75$, and the variance of the number of days until the prisoner reaches freedom is $114$.

$$\Var{M} = \ffrac{1} {6}\P{2.5^2+ 0.5^2 + 2.5^2 + 0.5^2 + 2.5^2 + 2.5^2} = 4.25$$

the variance of the number of days until the prisoner reaches freedom is $5$.

Probability and Statistics/Applied Random Process/HW/HW_03.ipynb