```python
from resources.workspace import *
```

$
% START OF MACRO DEF
% DO NOT EDIT IN INDIVIDUAL NOTEBOOKS, BUT IN macros.py
%
\newcommand{\Reals}{\mathbb{R}}
\newcommand{\Expect}[0]{\mathbb{E}}
\newcommand{\NormDist}{\mathcal{N}}
%
\newcommand{\DynMod}[0]{\mathscr{M}}
\newcommand{\ObsMod}[0]{\mathscr{H}}
%
\newcommand{\mat}[1]{{\mathbf{{#1}}}}
%\newcommand{\mat}[1]{{\pmb{\mathsf{#1}}}}
\newcommand{\bvec}[1]{{\mathbf{#1}}}
%
\newcommand{\trsign}{{\mathsf{T}}}
\newcommand{\tr}{^{\trsign}}
\newcommand{\tn}[1]{#1}
\newcommand{\ceq}[0]{\mathrel{≔}}
%
\newcommand{\I}[0]{\mat{I}}
\newcommand{\K}[0]{\mat{K}}
\newcommand{\bP}[0]{\mat{P}}
\newcommand{\bH}[0]{\mat{H}}
\newcommand{\bF}[0]{\mat{F}}
\newcommand{\R}[0]{\mat{R}}
\newcommand{\Q}[0]{\mat{Q}}
\newcommand{\B}[0]{\mat{B}}
\newcommand{\C}[0]{\mat{C}}
\newcommand{\Ri}[0]{\R^{-1}}
\newcommand{\Bi}[0]{\B^{-1}}
\newcommand{\X}[0]{\mat{X}}
\newcommand{\A}[0]{\mat{A}}
\newcommand{\Y}[0]{\mat{Y}}
\newcommand{\E}[0]{\mat{E}}
\newcommand{\U}[0]{\mat{U}}
\newcommand{\V}[0]{\mat{V}}
%
\newcommand{\x}[0]{\bvec{x}}
\newcommand{\y}[0]{\bvec{y}}
\newcommand{\z}[0]{\bvec{z}}
\newcommand{\q}[0]{\bvec{q}}
\newcommand{\br}[0]{\bvec{r}}
\newcommand{\bb}[0]{\bvec{b}}
%
\newcommand{\bx}[0]{\bvec{\bar{x}}}
\newcommand{\by}[0]{\bvec{\bar{y}}}
\newcommand{\barB}[0]{\mat{\bar{B}}}
\newcommand{\barP}[0]{\mat{\bar{P}}}
\newcommand{\barC}[0]{\mat{\bar{C}}}
\newcommand{\barK}[0]{\mat{\bar{K}}}
%
\newcommand{\D}[0]{\mat{D}}
\newcommand{\Dobs}[0]{\mat{D}_{\text{obs}}}
\newcommand{\Dmod}[0]{\mat{D}_{\text{obs}}}
%
\newcommand{\ones}[0]{\bvec{1}}
\newcommand{\AN}[0]{\big( \I_N - \ones \ones\tr / N \big)}
%
% END OF MACRO DEF
$
# The ensemble (Monte-Carlo) approach
is an approximate method for doing Bayesian inference. Instead of computing the full (gridvalues, or parameters, of the) posterior distributions, we instead try to generate ensembles from them.

An ensemble is an *iid* sample. I.e. a set of "members" ("particles", "realizations", or "sample points") that have been drawn ("sampled") independently from the same distribution. With the EnKF, these assumptions are generally tenous, but pragmatic.

Ensembles can be used to characterize uncertainty: either by reconstructing (estimating) the distribution from which it is assumed drawn, or by computing various *statistics* such as the mean, median, variance, covariance, skewness, confidence intervals, etc (any function of the ensemble can be seen as a "statistic"). This is illustrated by the code below.

```python
# Parameters
b   = 0
B   = 25
B12 = sqrt(B)

def true_pdf(x):
    return ss.norm.pdf(x,b,sqrt(B))

# Plot true pdf
xx = 3*linspace(-B12,B12,201)
fig, ax = plt.subplots()
ax.plot(xx,true_pdf(xx),label="True");

# Sample and plot ensemble
M = 1   # length of state vector
N = 100 # ensemble size
E = b + B12*randn((N,M))
ax.plot(E, zeros(N), '|k', alpha=0.3, ms=100)

# Plot histogram
nbins = max(10,N//30)
heights, bins, _ = ax.hist(E,density=1,bins=nbins,label="Histogram estimate")

# Plot parametric estimate
x_bar = np.mean(E)
B_bar = np.var(E)
ax.plot(xx,ss.norm.pdf(xx,x_bar,sqrt(B_bar)),label="Parametric estimate")

ax.legend();

# Uncomment AFTER Exc 4:
# dx = bins[1]-bins[0]
# c = 0.5/sqrt(2*pi*B)
# for height, x in zip(heights,bins):
#     ax.add_patch(mpl.patches.Rectangle((x,0),dx,c*height/true_pdf(x+dx/2),alpha=0.3))
# Also set
#  * N = 10**4
#  * nbins = 50
```

The plot demonstrates that the true distribution can be represented by a sample thereof (since we can almost reconstruct the Gaussian distribution by estimating the moments from the sample). However, there are other ways to reconstruct (estimate) a distribution from a sample. For example: a histogram.

**Exc 2:** Which approximation to the true pdf looks better: Histogram or the parametric?
Does one approximation actually start with more information? The EnKF takes advantage of this.

#### Exc 4*:
Use the method of `gaussian_kde` from `scipy.stats` to make a "continuous histogram" and plot it above.
`gaussian_kde`

```python
#show_answer("KDE")
```

**Exc 5*:** Suppose the histogram bars get normalized (divided) by the value of the pdf at their location.
How do you expect the resulting histogram to look?
Test your answer by uncommenting the block in the above code.

Being able to sample a Gaussian distribution is a building block of the EnKF.
In the previous example, we generated samples from a Gaussian distribution using the `randn` function.
However, that was just for a scalar (univariate) case, i.e. with `M=1`. We need to be able to sample a multivariate Gaussian distribution. That is the objective of the following exercise.

**Exc 6 (Multivariate Gaussian sampling):**
Suppose $\z$ is a standard Gaussian,
i.e. $p(\z) = \mathcal{N}(\z \mid \bvec{0},\I_M)$,
where $\I_M$ is the $M$-dimensional identity matrix.
Let $\x = \mat{L}\z + \bb$.
Recall [Exc 3.7](T3%20-%20Univariate%20Kalman%20filtering.ipynb#Exc-3.7:-The-forecast-step:),
which yields $p(\x) = \mathcal{N}(\x \mid \bb, \mat{L}^{}\mat{L}^T)$.

 * (a). $\z$ can be sampled using `randn((M,1))`. How (where) is `randn` defined?
 * (b). Consider the above definition of $\x$ and the code below.
 Complete it so as to generate a random realization of $\x$.
 Hint: matrix-vector multiplication can be done using the symbol `@`.

```python
M   = 3 # ndim
b   = 10*ones(M)
B   = diag(1+arange(M))
L   = np.linalg.cholesky(B) # B12
print("True mean and cov:")
print(b)
print(B)

### INSERT ANSWER (b) ###
```

```python
#show_answer('Gaussian sampling a')
```

```python
#show_answer('Gaussian sampling b')
```

 * (c). In the code cell below, sample $N = 100$ realizations of $\x$
 and collect them in an $M$-by-$N$ "ensemble matrix" $\E$.
   - Try to avoid `for` loops (the main thing to figure out is: how to add a (mean) vector to a matrix).
   - Run the cell and inspect the computed mean and covariance to see if they're close to the true values, printed in the cell above.

```python
N  = 100 # ensemble size

E = ### INSERT ANSWER (c) ###

# Use the code below to assess whether you got it right
x_bar = np.mean(E,axis=1)
B_bar = np.cov(E)

with printoptions(precision=1):
    print("Estimated mean:")
    print(x_bar)
    print("Estimated covariance:")
    print(B_bar)
plt.matshow(B_bar,cmap="Blues"); plt.grid('off'); plt.colorbar()
```

```python
#show_answer('Gaussian sampling c')
```

**Exc 8*:** How erroneous are the ensemble estimates on average?

```python
#show_answer('Average sampling error')
```

**Exc 10:** Above, we used numpy's (`np`) functions to compute the sample-estimated mean and covariance matrix,
$\bx$ and $\barB$,
from the ensemble matrix $\E$.
Now, instead, implement these estimators yourself:
$$\begin{align}\bx &\ceq \frac{1}{N}   \sum_{n=1}^N \x_n \, , \\
   \barB &\ceq \frac{1}{N-1} \sum_{n=1}^N (\x_n - \bx) (\x_n - \bx)^T \, . \end{align}$$

```python
# Don't use numpy's mean, cov
def estimate_mean_and_cov(E):
    M, N = E.shape

    ### INSERT ANSWER ###

    return x_bar, B_bar

x_bar, B_bar = estimate_mean_and_cov(E)
with printoptions(precision=1):
    print(x_bar)
    print(B_bar)
```

```python
#show_answer('ensemble moments')
```

**Exc 12:** Why is the normalization by $(N-1)$ for the covariance computation?

```python
#show_answer('Why (N-1)')
```

**Exc 14:** Like Matlab, Python (numpy) is quicker if you "vectorize" loops.
This is emminently possible with computations of ensemble moments.
Let $\X \ceq
\begin{bmatrix}
		\x_1 -\bx, & \ldots & \x_n -\bx, & \ldots & \x_N -\bx
	\end{bmatrix} \, .$
 * (a). Show that $\X = \E \AN$, where $\ones$ is the column vector of length $N$ with all elements equal to $1$.
 Hint: consider column $n$ of $\X$.
 * (b). Show that $\barB = \X \X^T /(N-1)$.
 * (c). Code up this, latest, formula for $\barB$ and insert it in `estimate_mean_and_cov(E)`

```python
#show_answer('ensemble moments vectorized')
```

**Exc 16:** The cross-covariance between two random vectors, $\bx$ and $\by$, is given by
$$\begin{align}
\barC_{\x,\y}
&\ceq \frac{1}{N-1} \sum_{n=1}^N
(\x_n - \bx) (\y_n - \by)^T \\\
&= \X \Y^T /(N-1)
\end{align}$$
where $\Y$ is, similar to $\X$, the matrix whose columns are $\y_n - \by$ for $n=1,\ldots,N$.
Note that this is simply the covariance formula, but for two different variables.
I.e. if $\Y = \X$, then $\barC_{\x,\y} = \barC_{\x}$ (which we have denoted $\barB$ in the above).

Implement the cross-covariance estimator in the code-cell below.

```python
def estimate_cross_cov(Ex,Ey):
    ### INSERT ANSWER ###
```

```python
#show_answer('estimate cross')
```

**Exc 18 (error notions)*:**
 * (a). What's the difference between error residual?
 * (b). What's the difference between error and bias?
 * (c). Show `MSE = RMSE^2 = Bias^2 + Var`

```python
#show_answer('errors')
```

### Next: [Writing your own EnKF](T8%20-%20Writing%20your%20own%20EnKF.ipynb)

notebooks/T7 - Ensemble [Monte-Carlo] approach.ipynb


```python
%matplotlib inline

import sys
sys.path.insert(0, '../python/casadi_f16')
import f16
import control
import numpy as np
import matplotlib.pyplot as plt
import scipy.linalg
import sympy

from analysis import loop_analysis, rlocus, bode

plt.rcParams['figure.figsize'] = (10, 10)
```

# Pitch-Rate CAS Design-Dhruv Jain

```python
p = f16.Parameters()
x0, u0 = f16.trim(x=f16.State(VT=550), p=p, phi_dot=0, theta_dot=0, psi_dot=0.349066, gam=0)
```

```python
def f_control(t, x):
    return f16.Control(
        thtl=u0.thtl,
        ail_cmd_deg=u0.ail_cmd_deg,
        elv_cmd_deg=u0.elv_cmd_deg + 1*np.sin(2*np.pi*2*t),
        rdr_cmd_deg=u0.rdr_cmd_deg)

f_control(0, x0)
```

    Control(thtl=DM(0.939641), ail_cmd_deg=DM(0.0832868), elv_cmd_deg=DM(-1.41359), rdr_cmd_deg=DM(-0.3512))

# A,B,C,D matrices

```python
ss = f16.StateSpace
ss = f16.linearize(x0, u0, p)
print("A= ",ss.A)
print("B= ",ss.B)
print("C= ",ss.C)
print("D= ",ss.D)
```

    A=  [[-1.00633464e-01 -2.20855448e+02  3.16848365e+01 -7.89498027e+00
      -3.11844580e+01  0.00000000e+00  4.29768610e-04 -8.59442671e+00
       1.81924240e-03  0.00000000e+00  0.00000000e+00  1.74909305e-04
       3.28097058e-01  7.20147501e-05 -1.61955145e-01  1.96611699e-04]
     [-5.51083668e-04 -1.15177247e+00  2.53266367e-04 -5.58711321e-02
       1.41691386e-02  0.00000000e+00 -3.92282765e-04  9.03393840e-01
      -1.00812602e-04  0.00000000e+00  0.00000000e+00  1.02191877e-05
      -1.53304645e-04  0.00000000e+00 -2.34025336e-03  0.00000000e+00]
     [-1.05585059e-04  1.24239855e-04 -3.52805546e-01  9.36199183e-03
      -2.37565513e-03  0.00000000e+00  2.50831071e-01  6.32908530e-06
      -9.60362120e-01  0.00000000e+00  0.00000000e+00 -1.89285193e-10
      -2.41616379e-07  3.23274943e-04  1.19266585e-07  8.82591907e-04]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       3.49367622e-01  0.00000000e+00  1.00000000e+00  4.10438538e-02
       6.65689383e-03  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00 -3.48764638e-01
       0.00000000e+00  0.00000000e+00  0.00000000e+00  1.60097727e-01
      -9.87101169e-01  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       1.45142305e-02  0.00000000e+00  0.00000000e+00  9.87954106e-01
       1.60236064e-01  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [-2.46725044e-04  5.78422089e-02 -7.17149411e+01  0.00000000e+00
       0.00000000e+00  0.00000000e+00 -3.49303895e+00 -4.31375506e-02
       1.78743793e+00  0.00000000e+00  0.00000000e+00 -4.32220398e-07
       0.00000000e+00 -8.20524519e-01  0.00000000e+00  1.51249863e-01]
     [ 9.30075577e-04  3.98296364e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  5.41358280e-02 -1.48074003e+00
      -1.48293685e-02  0.00000000e+00  0.00000000e+00 -2.58029797e-08
       0.00000000e+00  0.00000000e+00 -2.35589963e-01  0.00000000e+00]
     [ 4.28419694e-05 -6.18101073e-02  9.50452257e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00 -3.38823015e-01  1.16402106e-02
      -5.97428448e-01  0.00000000e+00  0.00000000e+00  1.16630091e-07
       0.00000000e+00 -2.87682151e-02  0.00000000e+00 -7.43371371e-02]
     [ 9.69364564e-01 -1.33234806e+02  2.23386906e+01 -5.61240681e+00
      -1.27405449e-02  1.35094535e+02  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [-2.45626428e-01 -5.25819663e+02  8.81084741e+01 -2.21366309e+01
       0.00000000e+00  5.33150510e+02  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 2.31646270e-05 -9.08961979e+01 -5.42436982e+02  1.34977903e+02
       5.33150510e+02  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
      -5.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00 -2.02020000e+01  0.00000000e+00  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00 -2.02020000e+01  0.00000000e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00  0.00000000e+00
       0.00000000e+00  0.00000000e+00  0.00000000e+00 -2.02020000e+01]]
    B=  [[   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [   0.       0.       0.       0.   ]
     [1086.9      0.       0.       0.   ]
     [   0.      20.202    0.       0.   ]
     [   0.       0.      20.202    0.   ]
     [   0.       0.       0.      20.202]]
    C=  [[1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1. 0.]
     [0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 1.]]
    D=  [[0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]
     [0. 0. 0. 0.]]

```python
res = f16.simulate(x0, f_control, p, 0, 20, 0.01)
plt.plot(res['t'], np.rad2deg(res['x'][:, f16.State().name_to_index('alpha')]))
plt.xlabel('t ,sec')
plt.ylabel(r'$\alpha$, deg')
plt.grid()
plt.title('angle of attack')
```

```python
res = f16.simulate(x0, f_control, p, 0, 20, 0.01)

plt.plot(res['x'][:, f16.State().name_to_index('p_E')], res['x'][:, f16.State().name_to_index('p_N')])
plt.axis('equal');
plt.xlabel('East, ft')
plt.ylabel('North, ft')
plt.grid()
plt.title('trajectory')
```

```python
def f_control(t, x):
    print(x)
    return f16.Control(
        thtl=u0.thtl,
        ail_cmd_deg=u0.ail_cmd_deg,
        elv_cmd_deg=u0.elv_cmd_deg + 1*np.sin(2*np.pi*2*t),
        rdr_cmd_deg=u0.rdr_cmd_deg)
```

```python
def select(n, i):
    D = np.zeros((1, n))
    D[0, i] = 1
    return control.ss([], [], [], D)
```

```python
ss = f16.linearize(x0, u0, p)
s = control.tf([1, 0], [0, 1])
G = -(180/np.pi)*ss.sub_system(x=['alpha', 'Q', 'elv_deg'],
                  u=['elv_cmd_deg'], y=['alpha', 'Q']).to_control()
sys3 = control.feedback(G, 0.2*10/(s+10)*select(2, 0))

```

# Transfer Function

```python
PI_c = 2.2 + 1*0.8/s
G0 = PI_c*sys3[1,0]
Gc = G0/(1+G0)
print('The Transfer Function is: ',Gc.minreal())
```

    The Transfer Function is:
             599.9 s^3 + 6932 s^2 + 9589 s + 2599
    ------------------------------------------------------
    s^5 + 32.83 s^4 + 879.6 s^3 + 7412 s^2 + 9707 s + 2599

```python
t,y = control.step_response(Gc,T=np.linspace(0,1,1000))
#plt.figure(figsize=(l))

plt.plot(t,y)
print(max(y))
for i in range(len(t)):
    if y[i]==max(y):
        print(t[i])
print('The PI control satisfy the given constraint')
```

# 10 Deg

```python
t,y = control.step_response(Gc,T=np.linspace(0,1,1000))
plt.plot(t,10*y)
```

# 100 Deg

```python
t,y = control.step_response(Gc,T=np.linspace(0,1,1000))
plt.plot(t,100*y)
```

```python
from analysis import rlocus
H = (10/(s+10))*select(2, 0)
plt.figure()
kalpha = 0.3
rlocus('alpha', control.minreal(H*G), kvect=np.linspace(0, 10, 1000), k=kalpha);
plt.plot([0, -2], [0, 2], '--')
plt.axis([-21, 0, -8, 8])
```

```python
plt.figure()
sys3 = control.feedback(G, kalpha*(10/(s+10))*select(2, 0))
rlocus('p', (s+3)/s*sys3[1, 0], kvect=np.linspace(0, 1, 1000), k=0.5)
plt.plot([0, -10], [0, 10*np.cos(0.707)], '--')
#plt.axis([-20, 0, -5, 5])

```

```python

```


#  AI Workshop - Część II
## Wykorzystanie konwolucyjnych sieci neuronowych do klasyfikacji cech twarzy

## Plan pracy:

####  >>> Zbudujemy konwolucyjną sieć neuronową
####  >>> Przeprowadzimy klasyfikację obrazów ze względu na wybraną cechę
####  >>> Podglądniemy, które z elementów obrazu zadecydowały o wyniku klasyfikacji

###  Do tego celu posłużymy się danymi pochodzącymi ze zbioru CelebA. Zbiór ten posiada ponad 200 tysięcy zdjęć twarzy znanych celebrytów.

### Uwaga: Miejsca w kodzie do uzupełnienia są oznaczone poprzez "____"

## Zaczniemy od zaimportowania niezbędnych bibliotek i ustawienia odpowiednich folderów
##### Wykorzystane funkcje zostaną wytłumaczone w dalszej części notatnika

```python
!pip install --user -r ../requirements.txt # pobieranie bibliotek z pliku requirements.txt
```

```python
# operacje związane ze ścieżkami
import os
from glob import glob

 #operacje na macierzach
import numpy as np

# operacje na danych tabelarycznych
import pandas as pd

# operacje na obrazach
from PIL import Image

# wizualizacja postępu wykonania kodu
from tqdm import tqdm

# tworzenie wykresów i wizualizacja danych
import matplotlib.pyplot as plt
import seaborn as sns
import matplotlib.cm as cm

# importy związane z budową i treningiem modelu
from sklearn.model_selection import train_test_split
from keras.layers import Dense, Input, Dropout
from keras.layers import Conv2D, Flatten, MaxPooling2D
from keras.models import Model
from keras.utils import plot_model
from keras.callbacks import EarlyStopping, ModelCheckpoint
from keras import activations

# weryfikacj jakości predykcji
from sklearn.metrics import classification_report, confusion_matrix

# wizualizacja map aktywacji klas (CAM)
from vis.visualization import visualize_cam, overlay
from vis.utils import utils

# importy do przykładu ze zdjęciem z internetu
from urllib.request import urlopen
from resizeimage import resizeimage

%matplotlib inline
```

#### Zawsze warto ustawić tak zwane ziarno - określa ono punkt początkowy dla algorytmu generacji liczb pseudolosowych. Dzięki temu przy każdorazowym uruchomieniu notatnika uzyskamy te same wyniki.
Podpowiedź: ziarno ustawiamy podając dowolną liczbę do funkcji np.random.seed()

```python
____
```

```python
# tworzymy foldery na zdjęcia i modele
dataset_dir = '../img_align_celeba'
model_dir = '../models'

os.makedirs(dataset_dir,exist_ok=True)
os.makedirs(model_dir,exist_ok=True)

labels_path = '../list_attr_celeba.txt'
landmarks_path = '../list_landmarks_align_celeba.txt'
```

## Pobranie i rozpakowanie danych
### Uwaga: Dane obrazowe zostały już przygotowane
*   Linux
Dane obrazowe:
```bash
!wget https://www.dropbox.com/s/lpmjzzk26nae9bh/img_align_celeba.zip -P ./..
!unzip -q -d ../img_align_celeba ../img_align_celeba.zip
```
Dane liczbowe:

```python
!wget https://www.dropbox.com/s/qpjuy9isvm19xsv/list_attr_celeba.txt -P ./..
```

* Windows
Pobierz pliki ze strony dropbox i wypakuj plik zip wybranym programem do obsługi archiwów.
https://www.dropbox.com/s/lpmjzzk26nae9bh/img_align_celeba.zip
https://www.dropbox.com/s/qpjuy9isvm19xsv/list_attr_celeba.txt
https://www.dropbox.com/s/lmc2ywmuk6c29tt/list_landmarks_align_celeba.txt

## Wczytanie i przygotowanie danych
#### Etykiety dotyczące cech twarzy

```python
attributes = []
with open(labels_path, 'r') as f:
    f.readline()
    attribute_names = ['fn']+f.readline().strip().split(' ')
    for i, line in enumerate(f):
        fields = line.strip().replace('  ', ' ').split(' ')
        img_name = fields[0]
        if int(img_name[:6]) != i + 1:
            raise ValueError('Parse error.')
        attr_vec = np.array([fields[0]]+[int(x) for x in fields[1:]])
        attributes.append(attr_vec)

attributes = np.array(attributes)
```

### Stworzenie tabeli z cechami i etykietami
#### 1 - cecha obecna na obrazie
#### -1 - brak cechy

```python
df = pd.DataFrame(data=attributes, columns=attribute_names)
df.head()
```

### Wyświetl listę wszystkich dostępnych cech

```python
for feature in df.columns[1:]:
    print(feature)
```

### Wybranie cechy badanej na warsztatach

```python
examined_feature = 'Bald'
```

### W celu przyspieszenia obliczeń wykorzystamy jedynie 10 000 losowo wybranych obrazów. Potrzebujemy zatem 10000 losowo wybranych indeksów, ale w taki sposób, żeby wśród wybranych przykładów nasza cecha pojawiła się w mniej więcej połowie przypadków.

#### W tym celu wykorzystamy funkcję df.groupby() która przyjmuje nazwę wybranej cechy (examined_feature) do pogrupowania rekordów według tej cechy

```python
df_balanced = ____(____) # pogrupuj rekordy według badanej cechy
n_samples = np.min([10000//2,df_balanced.size().min()])

print(n_samples) # wyświetlenie ilości przykładów zawierających naszą cechę
```

### Stworzenie tabeli tylko z wybranymi przykładami

#### Wybranie próbki dla każdej z wartości etykiet binarnych

```python
df_balanced = df_balanced.apply(lambda x: x.sample(n_samples))
```

#### Ustaw kolumnę 'fn' jako indeks za pomocą df.set_index() i pozostaw jedynie kolumnę z interesującą nas cechą

```python
df_balanced = df_balanced.____(____)[examined_feature]
```

#### Przekształcenie obiektu po grupowaniu z powrotem do tabeli i kolejny restart indeksu w celu uzyskania 'fn' jako osobnej kolumny

```python
df_balanced = (df_balanced.to_frame()[examined_feature]).reset_index()
```

#### Metoda  df.head(n=5) pozwala na wyświetlenie pierwszych 5 wierszy tabeli. Wyświetl 20 pierwszych wierszy.

```python
df_balanced.____(____) # wyświetl 20 pierwszych wierszy
```

#### Całkowita ilość wybranych obrazów:

```python
print('Mamy',str(len(df_balanced)),'obrazów')
```

### Przygotowanie etykiet

```python
labels = (df_balanced[examined_feature].values) == '1'
```

#### Sprawdź ilość przykładów zawierających naszą cechę oraz ilość przykładów bez tej cechy

```python
print(np.sum(labels==True),np.sum(labels==False))
```

### Wczytanie i przygotowanie obrazów

#### Razem z wczytaniem danych  (Image.open()) przygotujemy je do wykorzystania: przytniemy krawędzie (crop()) i zmniejszymy ich rozmiar (resize()), co znacznie przyspieszy trening modelu.
#### Zastosuj wylosowane indeksy w celu wczytania zbioru danych do pracy. Przytnij obraz wprowadzając wartości 9, 13, 169 i 205 (koordynaty pikseli) w odpowednie miejsce. Następnie zmniejsz obraz do rozmiaru 80x96

```python
images = np.array(
        [np.array(Image.open(os.path.join(dataset_dir,x['fn'])).
                  crop(box=(____)).resize((____)))
         for i,x in tqdm(df_balanced.iterrows())]).astype(np.uint8)
```

#### Żeby upewnić się, że wszystko poszło dobrze, wyświetl wymiary obrazów

```python
____
```

### Wyświetl przykładowe obrazy osób bez wybranej cechy oraz osób posiadających daną cechę
##### Ustaw wielkość figur na 10x10

```python
plt.figure(figsize=(____,____))
for i in range(1,10):
    plt.subplot(3,3,i)
    plt.title(examined_feature if labels[i]==1 else '~'+examined_feature)
    plt.axis('off')
    plt.imshow(np.squeeze(images[i]))
plt.show()
plt.figure(figsize=(____,____))
for i in range(1,10):
    plt.subplot(3,3,i)
    plt.title(examined_feature if labels[len(labels) - i]==1 else '~'+examined_feature)
    plt.axis('off')
    plt.imshow(np.squeeze(images[len(labels) - i]))
plt.show()
```

## Budowa modelu konwolucyjnej sieci neuronowej (CNN)

#### Na początku zdefiniujemy parametry określające model CNN i jego trening

```python
input_shape = images.shape[1:4] #wielkość obrazu wejściowego, wykorzystywana dla budowy grafu SN

""" Parametry określające trening modelu """
batch_size = 128 # liczba obrazów wykorzystywana do jednego kroku treningu SN
optimizer = 'adam' # funkcja odpowiadająca za redukcję błędu sieci
loss_fn = 'sparse_categorical_crossentropy' # funkcja służąca do obliczenia o ile różni się wartość oczekiwana od otrzymanej na wyjściu SN
epochs = 30 # maksymalna liczba epok, czyli ile razy sieć będzie widziała dane treningowe
model_path = os.path.join(model_dir,'celeb_cnn_v2_'+examined_feature+'.h5') # ustawienie ścieżki do pliku z modelem
```

### Na podstawie poniższego schematu i wskazówek zbuduj model konwolucyjnej sieci neuronowej (uzupełnij brakujące części kodu)

###  Ważne wskazówki :
*   Zwróć uwagę na kolory warstw na powyższym schemacie i nazwy w nawiasach
*   Brakujące warstwy uzupełniaj analogicznie do wpisanych już przykładów
*   Uzupełnij brakujące pola "____" w wyznaczonych miejscach
*   Funkcja aktywacji to 'relu' (za wyjątkiem ostatniej, gdzie jest 'softmax')
*   Dla każdej kolejnej warstwy konwolucyjnej liczbę filtrów zwiększ dwukrotnie w stosunku do poprzedniej
*   Dla każdej warstwy konwolucyjnej ustaw wielkość filtru 3x3
*   Dla każdej warstwy głosującej ustaw wielkość filtru 2x2
*   Dla każdej warstwy Dropout (z wyjątkiem ostatniej) 25% neuronów zostaje wyłączona

```python
inputs = Input(shape=input_shape, name='input')

x = Conv2D(64, (3, 3), activation='relu')(inputs)
x = MaxPooling2D(pool_size=(2, 2))(x)
x = Dropout(0.25)(x)

x = ____(____)(x)
x = ____(____)(x)
x = ____(____)(x)

x = ____(____)(x)
x = ____(____)(x)
x = ____(____)(x)

x = Flatten()(x)

x = ____(256, activation=____)(x)
x = Dropout(0.5)(x)
x = ____(2, activation='softmax')(x)

model = Model(inputs=inputs, outputs=x, name='CNN')
model.compile(optimizer=optimizer, loss=loss_fn, metrics=['acc'])
```

## Trening konwolucyjnej sieci neuronowej

### Podział danych na testowe i treningowe
#### Podział danych jest realizowany za pomocą funkcji train_test_split, która przyjmuje dane do podziału, ich etykiety oraz procent przykładów które będą przydzielone do zbioru testowego test_size

```python
X_train, X_test, y_train, y_test = ____(images.astype('float32')/255, labels, ____=0.2) # uzupełnij kod do podziału danych
```

#### Powyższa funkcja wyprodukowała 4 zmienne: X_train i X_test zawierające odpowiednio dane treningowe i testowe oraz y_train, y_test będące etykietami dla każdego z przykładów w zbiorach danych

### Standaryzacja danych
#### Dokończ standaryzowanie obrazów, w każdym z kanałów RGB (3), wykorzystując funkcje biblioteki numpy wyliczające średnią i odchylenie standardowe (np.mean(), np.std()) oraz poniższy wzór:
\begin{equation}
X = \frac{X - mean(X)}{std(X)}
\end{equation}

```python
X_mean = []
X_std = []
for i in range(3):
    # zapisanie średnich i odchyleń z danych - będą potrzebne później
    X_mean.append(np.mean(X_train[:,:,:,i]))
    X_std.append(np.std(X_train[:,:,:,i]))

    # uzupełnij poniższe wzory standaryzujące dane treningowe i testowe
    X_train[:,:,:,i] = (X_train[:,:,:,i] - ____(____))/____(____)
    X_test[:,:,:,i] = (X_test[:,:,:,i] - ____(____))/____(____)
```

#### Wprowadzenie parametrów treningowych i funkcji monitorujących przebieg treningu

```python
early_stop = EarlyStopping(restore_best_weights=True, patience=3) # zatrzyma uczenie jeśli nie będzie poprawy przez 3 epoki
checkpt = ModelCheckpoint(model_path, save_best_only=True, save_weights_only=True) # zapisze wagi dla najlepszej epoki
```

### Trening CNN
#### Dopisz w odpowiednich miejscach brakujące dane i etykiety (dane testowe jako walidacyjne)

```python
history = model.fit(X_train, ____,
                  epochs=epochs,
                  batch_size=batch_size,
                  validation_data=(____, y_test),
                  verbose=2,
                  callbacks = [early_stop, checkpt])
```

## Weryfikacja rezultatów
#### Wczytanie zapisanych wag z epoki, dla której wartość funkcji straty była najniższa

```python
model.load_weights(model_path)
```

#### Weryfikacja jakości predykcji

```python
y_pred = np.argmax(model.predict(X_test),axis=-1)
```

```python
print(confusion_matrix(y_test, y_pred)) # macierz błędów klasyfikacji
```

```python
print(classification_report(____, ____)) # uzupełnij raport
```

## Wizualizacja CAM

### CAMy to mapy cieplne wskazujące obszary obrazu, które przyczyniły się najbardziej do wyniku klasyfikacji

```python
# wymagane modyfikacje
model_vis = model
model_vis.layers[-1].activation = activations.linear
model_vis = utils.apply_modifications(model_vis)
```

```python
%matplotlib inline
plt.figure(figsize=(10,10))
for i in range(1,10):
    plt.subplot(3,3,i)
    plt.title(examined_feature if labels[i]==1 else '~'+examined_feature)
    grads = visualize_cam(model_vis, -1, filter_indices=int(labels[i]),
                          seed_input=images[i], backprop_modifier='guided')
    plt.imshow(overlay(cm.jet(grads)[:,:,:3], images[i]/255))
plt.show()

plt.figure(figsize=(10,10))
for i in range(1,10):
    plt.subplot(3,3,i)
    plt.title(examined_feature if labels[len(labels) - i]==1 else '~'+examined_feature)
    grads = visualize_cam(model_vis, -1, filter_indices=int(labels[[len(labels) - i]]),
                          seed_input=images[len(labels) - i], backprop_modifier='guided')
    plt.imshow(overlay(cm.jet(grads)[:,:,:3], images[len(labels) - i]/255))
plt.show()
```

### >>> Co można wywnioskować na podstawie uzyskanych wizualizacji CAMów?

## Sprawdzenie działania na zdjęciu spoza zbioru danych - przykład z internetu

```python
img = Image.open(urlopen('http://www.afternoondc.in/Thumbnails.aspx?Filename=2018319214726.jpg'))
img
```

#### Wyświetl rozmiar obrazu

```python

```

#### Za pomocą funkcji resizeimage.resize_cover zmniejsz obraz do rozmiaru 80x96

```python
img = ____(img, [____, ____])
img
```

### Przetworzenie obrazu do formy akceptowanej przez sieć neuronową

```python
img_float = np.array(img).astype(np.float32)
img_std = img_float.copy()
for i in range(3):
    img_std[:,:,i] = (img_std[:,:,i]-X_mean[i])/X_std[i]
img_std = np.expand_dims(img_std,0)

img_std.shape # wyświetlenie końcowych wymiarów obrazu (po przetworzeniu)
```

### Predykcja sieci neuronowej dla przygotowanego obrazu za pomocą predict()

```python
img_pred = model.____(____)
```

### Wizualizacja CAM dla obrazu

```python
grads = visualize_cam(model_vis, -1, filter_indices=1,
                      seed_input=np.squeeze(img_std), backprop_modifier='guided')
plt.imshow(overlay(cm.jet(grads)[:,:,:3], img_float/255))
plt.show()
```

## Twój przykład. Znajdź obraz na internecie i sprawdź odpowiedź sieci neuronowej :)

```python
from urllib.request import urlopen
from resizeimage import resizeimage

img = Image.open(urlopen(' tutaj link do obrazka '))
img
```

```python
img = ____(img, [____, ____]) # zmniejszenie rozmiaru resizeimage.resize_cover
img
```

```python
img_float = np.array(img).astype(np.float32)
img_std = img_float.copy()
for i in range(3):
    img_std[:,:,i] = (img_std[:,:,i]-X_mean[i])/X_std[i]
img_std = np.expand_dims(img_std,0)
img_std.shape
```

```python
img_pred = model.____(____) # predykcja
```

```python
grads = visualize_cam(model_vis, -1, filter_indices=1,
                      seed_input=np.squeeze(img_std), backprop_modifier='guided')
plt.imshow(overlay(cm.jet(grads)[:,:,:3], img_float/255))
plt.show()
```


```python
# Initialize OK
from client.api.notebook import Notebook
ok = Notebook('hw6.ok')
```

# Homework 6: Exploring fairness through Cook County’s property assessments

## Due Date: 11:59pm Monday, April 6

### Collaboration Policy

Data science is a collaborative activity. While you may talk with others about the homework, we ask that you **write your solutions individually**. If you do discuss the assignments with others please **include their names** in the collaborators cell below.

**Collaborators:** *write names here*

## Introduction

This assignment will continue from where we left off in in Homework 5. Recall that the linear model that you created failed to produce accurate estimates of the observed housing prices because the model was too simple. The goal of this homework is to guide you through the iterative process of specifying, fitting, and analyzing the performance of more complex linear models used to predict prices of houses in Ames, Iowa. Additionally, you will have the opportunity to choose your own features and create your own regression model!

By the end of this homework, you should feel comfortable:

1. Identifying informative variables through EDA
2. Feature engineering categorical variables
3. Using sklearn to build more complex linear models

Additionally, as a continuation of the last homework, we’ll explore the dynamics of the CCAO’s appraisal system with more depth as you continue developing your housing prediction model. Alongside our discussion on implicit bias, however, we’ll tackle another central facet of the CCAO’s work: transparency.

As you work through this assignment, consider the balance of power between the CCAO and its constituents - how might transparency redistribute this balance? And what are the limits of transparency as a solution for systemic inequity?

## Learning Outcomes

Through the completion of this homework, student will be able to:
* Understand the relationship between bias and fairness.
* Analyze the technical and performative functions of transparency initiatives.
* Recognize the social aspects of transparency in regard to the redistribution of power between different stakeholders.
* Weigh the effectiveness and limitations of transparency as a means of arriving at fair algorithmic systems in order to reimagine equitable practices in data science.

## Score Breakdown

*To be determined by course staff*

```python
import numpy as np

import pandas as pd
from pandas.api.types import CategoricalDtype

from sklearn.feature_extraction import DictVectorizer

%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
```

```python
# Plot settings
plt.rcParams['figure.figsize'] = (12, 9)
plt.rcParams['font.size'] = 12
```

# The Data (to be edited)

As a reminder, the [Ames dataset](http://jse.amstat.org/v19n3/decock.pdf) consists of 2930 records taken from the Ames, Iowa, Assessor’s Office describing houses sold in Ames from 2006 to 2010.  The data set has 23 nominal, 23 ordinal, 14 discrete, and 20 continuous variables (and 2 additional observation identifiers) --- 82 features in total.  An explanation of each variable can be found in the included `codebook.txt` file.  The information was used in computing assessed values for individual residential properties sold in Ames, Iowa from 2006 to 2010.

The raw data are split into training and test sets with 2000 and 930 observations, respectively. To save some time, we've used a slightly modified data cleaning pipeline from last week's assignment to prepare the training data. This data is stored in `ames_train_cleaned.csv`. It consists of 1998 observations and 83 features (we added TotalBathrooms from Homework 5).

```python
training_data = pd.read_csv("ames_train_cleaned.csv")
```

### Bias and Fairness

When the use of algorithms and statistical modeling has real-world consequences, we often refer to the idea of fairness as a measurement of how socially-responsible our work is. Does our algorithm make similar predictions across race and gender identities? Does our model adequately represent reality?

In the case of the Cook County Assessor’s Office, fair property tax rates are contingent on whether property values are assessed accurately - that they’re valued at what they’re worth, relative to properties with similar characteristics. This implies that having a more accurate model results in  fairer assessments. The goal of creating a property assessment model, then, is to be as accurate as possible, which is typically approached by attempting to minimize human-related biases in the data collection and modeling process.

In our previous examination of bias through Homework 1, however, we established that, because of human involvement and historical/institutional contexts, bias is impossible to eradicate. This adds a new dimension to our understanding of fairness in home assessments: Having “accurate” (and therefore fair) assessments requires us to constantly reflect on the decisions we make throughout the data lifecycle, as well as the contexts in which we’re working. Keep this relationship between bias and fairness in mind as we continue building our model!

## HCE: Question -1

Based on your work in the previous homework, what would you define as a fair property assessment? There isn’t one right answer! Share your thoughts in 1-2 sentences.

`TODO`: *Write your answer here*

## HCE: Question 0

Does removing human judgment from the assessment process make assessments fairer? In what ways? How might it be (still) unfair?

`TODO`: *Write your answer here*

# Part 4: More Feature Selection and Engineering

In this section, we identify two more features of the dataset that will increase our linear regression model's accuracy. Additionally, we will implement one-hot encoding so that we can include binary and categorical variables in our improved model.Before we make changes to our basic linear model, let’s go back to the data. In this section, we’ll identify two more features of the dataset that will increase our linear regression model's accuracy. Additionally, we will implement one-hot encoding so we can include binary and categorical variables in our improved model.

We’ll start by first diving into the hallmark of the CCAO’s existing model: mass appraisal.

### Mass Appraisal

A unique technique employed by the current assessor’s office is the idea of mass appraisal. Rather than assessing homes one by one, mass appraisal evaluates value by looking to the real estate market for local trends based on location and property characteristics. This differs from the classic system, where human evaluation was a more significant factor in evaluating housing prices. The CCAO’s website states that [“mass appraisal is a way to put fairness into the assessment system.”](https://www.cookcountyassessor.com/index.php/about-cook-county-assessors-office)

The dataset we’re currently working with is the same one used for mass appraisal. Let’s examine the column `Neighborhood Code` and see how the location of homes might influence the fairness of mass appraisal.

## HCE: Question 2

How does mass appraisal appeal to fairness? Consider how it might both benefit and hurt homeowners.

`TODO:` *Write your answer here*

## Question 1: Neighborhood vs Sale Price

First, let's take a look at the relationship between neighborhood and sale prices of the houses in our data set.

```python
fig, axs = plt.subplots(nrows=2)

sns.boxplot(
    x='Neighborhood',
    y='SalePrice',
    data=training_data.sort_values('Neighborhood'),
    ax=axs[0]
)

sns.countplot(
    x='Neighborhood',
    data=training_data.sort_values('Neighborhood'),
    ax=axs[1]
)

# Draw median price
axs[0].axhline(
    y=training_data['SalePrice'].median(),
    color='red',
    linestyle='dotted'
)

# Label the bars with counts
for patch in axs[1].patches:
    x = patch.get_bbox().get_points()[:, 0]
    y = patch.get_bbox().get_points()[1, 1]
    axs[1].annotate(f'{int(y)}', (x.mean(), y), ha='center', va='bottom')

# Format x-axes
axs[1].set_xticklabels(axs[1].xaxis.get_majorticklabels(), rotation=90)
axs[0].xaxis.set_visible(False)

# Narrow the gap between the plots
plt.subplots_adjust(hspace=0.01)
```

### Question 1a

Based on the plot above, what can be said about the relationship between the houses' sale prices and their neighborhoods?

*Write your answer here, replacing this text.*

### Question 1b

One way we can deal with the lack of data from some neighborhoods is to create a new feature that bins neighborhoods together.  Let's categorize our neighborhoods in a crude way: we'll take the top 3 neighborhoods measured by median `SalePrice` and identify them as "rich neighborhoods"; the other neighborhoods are not marked.

Write a function that returns list of the top n most pricy neighborhoods as measured by our choice of aggregating function.  For example, in the setup above, we would want to call `find_rich_neighborhoods(training_data, 3, np.median)` to find the top 3 neighborhoods measured by median `SalePrice`.

*The provided tests check that you answered correctly, so that future analyses are not corrupted by a mistake.*

```python
def find_rich_neighborhoods(data, n=3, metric=np.median):
    """
    Input:
      data (data frame): should contain at least a string-valued Neighborhood
        and a numeric SalePrice column
      n (int): the number of top values desired
      metric (function): function used for aggregating the data in each neighborhood.
        for example, np.median for median prices

    Output:
      a list of the top n richest neighborhoods as measured by the metric function
    """
    neighborhoods = ...
    return neighborhoods

rich_neighborhoods = find_rich_neighborhoods(training_data, 3, np.median)
rich_neighborhoods
```

```python
ok.grade("q1b");
```

### Question 1c

We now have a list of neighborhoods we've deemed as richer than others.  Let's use that information to make a new variable `in_rich_neighborhood`.  Write a function `add_rich_neighborhood` that adds an indicator variable which takes on the value 1 if the house is part of `rich_neighborhoods` and the value 0 otherwise.

**Hint:** [`pd.Series.astype`](https://pandas.pydata.org/pandas-docs/version/0.23.4/generated/pandas.Series.astype.html) may be useful for converting True/False values to integers.

*The provided tests check that you answered correctly, so that future analyses are not corrupted by a mistake.*

```python
def add_in_rich_neighborhood(data, neighborhoods):
    """
    Input:
      data (data frame): a data frame containing a 'Neighborhood' column with values
        found in the codebook
      neighborhoods (list of strings): strings should be the names of neighborhoods
        pre-identified as rich
    Output:
      data frame identical to the input with the addition of a binary
      in_rich_neighborhood column
    """
    data['in_rich_neighborhood'] = ...
    return data

rich_neighborhoods = find_rich_neighborhoods(training_data, 3, np.median)
training_data = add_in_rich_neighborhood(training_data, rich_neighborhoods)
```

```python
ok.grade("q1c");
```

Having identified rich neighborhoods, let’s now look toward the other end of the spectrum: lower-valued properties. According to the CCAO, their assessment system struggles with accurately predicting the values of properties that are worth less than 150k because they lack data for those properties. This ultimately diminishes the usefulness of mass appraisal for these properties.

## HCE: Question 2.5

In what situation does mass appraisal fail? How might mass appraisal work unfairly in this way?

`TODO:` *Write your answer here*

## Question 2: Floodplain

In 2019, the Cook County Assessor’s Office added the Federal Emergency Management Agency’s [floodplain data](https://msc.fema.gov/portal/home) to its assessment models. As described in their [Medium article](https://medium.com/@AssessorCook/why-and-how-floodplain-data-is-used-in-cook-county-property-assessments-6269d75189d7), “a floodplain is an area near a body of water that has a high risk of flooding.” A value of 0 indicates that a property is not on a floodplain, while a value of 1 indicates that a property is on a floodplain. There are, however, missing values in the dataset.

### Question 2a

Let's see if our data set has any missing values.  Create a Series object containing the counts of missing values in each of the columns of our data set, sorted from greatest to least.  The Series should be indexed by the variable names.  For example, `missing_counts['Fireplace_Qu']` should return 975.

**Hint:** [`pandas.DataFrame.isnull`](https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.isnull.html) may help here.

*The provided tests check that you answered correctly, so that future analyses are not corrupted by a mistake.*

```python
missing_counts = ...
missing_counts
```

```python
ok.grade("q2a");
```

It turns out that if we look at the codebook carefully, some of these "missing values" aren't missing at all! The Assessor's Office just used `NA` to denote a special value or that the information was truly not applicable for one reason or another.  One such example is the `Fireplace_Qu` variable.
```
FireplaceQu (Ordinal): Fireplace quality

       Ex	Excellent - Exceptional Masonry Fireplace
       Gd	Good - Masonry Fireplace in main level
       TA	Average - Prefabricated Fireplace in main living area or Masonry Fireplace inbasement
       Fa	Fair - Prefabricated Fireplace in basement
       Po	Poor - Ben Franklin Stove
       NA	No Fireplace
```

### Question 2b

An `NA` here actually means that the house had no fireplace to rate.  Let's fix this in our data set.  Write a function that replaces the missing values in `Fireplace_Qu` with `'No Fireplace'`.  In addition, it should replace each abbreviated condition with its full word.  For example, `'TA'` should be changed to `'Average'`.  Hint: the [DataFrame.replace](https://pandas.pydata.org/pandas-docs/stable/generated/pandas.DataFrame.replace.html) method may be useful here.

*The provided tests check that part of your answer is correct, but they are not fully comprehensive.*

```python
def fix_fireplace_qu(data):
    """
    Input:
      data (data frame): a data frame containing a Fireplace_Qu column.  Its values
                         should be limited to those found in the codebook
    Output:
      data frame identical to the input except with a refactored Fireplace_Qu column
    """
    ...
    return data

training_data = fix_fireplace_qu(training_data)
```

```python
ok.grade("q2b");
```

```python
training_data['Fireplace_Qu']
```

### An Important Note on One Hot Encoding

Unfortunately, simply fixing these missing values isn't sufficient for using `Fireplace_Qu` in our model.  Since `Fireplace_Qu` is a categorical variable, we will have to one-hot-encode the data using `DictVectorizer` from Lab 6. Note that we dropped the first one-hot-encoded column. For more information on categorical data in pandas, refer to this [link](https://pandas-docs.github.io/pandas-docs-travis/categorical.html).

```python
def ohe_fireplace_qu(data):
    """
    One-hot-encodes fireplace quality.  New columns are of the form Fireplace_Qu=QUALITY
    """
    vec_enc = DictVectorizer()
    vec_enc.fit(data[['Fireplace_Qu']].to_dict(orient='records'))
    fireplace_qu_data = vec_enc.transform(data[['Fireplace_Qu']].to_dict(orient='records')).toarray()
    fireplace_qu_cats = vec_enc.get_feature_names()
    fireplace_qu = pd.DataFrame(fireplace_qu_data, columns=fireplace_qu_cats)
    data = pd.concat([data, fireplace_qu], axis=1)
    data = data.drop(columns=fireplace_qu_cats[0])
    return data
```

```python
training_data = ohe_fireplace_qu(training_data)
training_data.filter(regex='Fireplace_Qu').head(10)
```

# Part 5: Improved Linear Models

In this section, we will create linear models that produce more accurate estimates of the housing prices in Ames than the model created in Homework 5, but at the expense of increased complexity.

## Question 3: Adding Covariates to our Model

It's finally time to fit our updated linear regression model using the ordinary least squares estimator! Our new model consists of the linear model from Homework 5, with the addition of the our newly created `in_rich_neighborhood` variable and our one-hot-encoded fireplace quality variables:

$$\begin{align}
\text{SalePrice} & = \theta_0 + \theta_1 \cdot \text{Gr_Liv_Area} + \theta_2 \cdot \text{Garage_Area} +
\theta_3 \cdot \text{TotalBathrooms} + \theta_4 \cdot \text{in_rich_neighborhood} + \\
& \quad \: \theta_5 \cdot \text{Fireplace_Qu=Excellent} + \theta_6 \cdot \text{Fireplace_Qu=Fair} + \theta_7 \cdot \text{Fireplace_Qu=Good} + \\
& \quad \: \theta_8 \cdot \text{Fireplace_Qu=No Fireplace} + \theta_9 \cdot \text{Fireplace_Qu=Poor}
\end{align}$$

### Question 3a

Although the fireplace quality variable that we explored in Question 2 has six categories, only five of these categories' indicator variables are included in our model. Is this a mistake, or is it done intentionally? Why?

*Write your answer here, replacing this text.*

### Question 3b

We still have a little bit of work to do prior to esimating our linear regression model's coefficients. Instead of having you go through the process of selecting the pertinent convariates and creating a [`sklearn.linear_model.LinearRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) object for our linear model again, we will provide the necessary code from Homework 5. However, we will now use cross validation to help validate our model instead of explicitly splitting the data into a training and testing set.

First, we will re-import the data.

```python
training_data = pd.read_csv("ames_train_cleaned.csv")
```

Next, we will implement a reusable pipeline that selects the required variables in our data and splits our covariates and response variable into a matrix and a vector, respectively.

```python
def select_columns(data, *columns):
    """Select only columns passed as arguments."""
    return data.loc[:, columns]

def process_data_gm(data):
    """Process the data for a guided model."""
    # One-hot-encode fireplace quality feature
    data = fix_fireplace_qu(data)
    data = ohe_fireplace_qu(data)

    # Use rich_neighborhoods computed earlier to add in_rich_neighborhoods feature
    data = add_in_rich_neighborhood(data, rich_neighborhoods)

    # Transform Data, Select Features
    data = select_columns(data,
                          'SalePrice',
                          'Gr_Liv_Area',
                          'Garage_Area',
                          'TotalBathrooms',
                          'in_rich_neighborhood',
                          'Fireplace_Qu=Excellent',
                          'Fireplace_Qu=Fair',
                          'Fireplace_Qu=Good',
                          'Fireplace_Qu=No Fireplace',
                          'Fireplace_Qu=Poor'
                         )

    # Return predictors and response variables separately
    X = data.drop(['SalePrice'], axis = 1)
    y = data.loc[:, 'SalePrice']

    return X, y
```

We then split our dataset into training and testing sets using our data cleaning pipeline.

```python
# Pre-process the training data
# Our functions make this very easy!
X_train, y_train = process_data_gm(training_data)
X_train.head()
```

Finally, we initialize a [`sklearn.linear_model.LinearRegression`](https://scikit-learn.org/stable/modules/generated/sklearn.linear_model.LinearRegression.html) object as our linear model. We set the `fit_intercept=True` to ensure that the linear model has a non-zero intercept.

```python
from sklearn import linear_model as lm

linear_model = lm.LinearRegression(fit_intercept=True)
```

After a little bit of work, it's finally time to fit our updated linear regression model. Use the cell below to estimate the model, and then use it to compute the fitted value of `SalePrice` over the training data.

*The provided tests check that you answered correctly, so that future analyses are not corrupted by a mistake.*

```python
# Fit the model below

# Compute the fitted and predicted values of SalePrice
y_fitted = ...
```

```python
ok.grade("q3b");
```

You can see that, as we consider more features in our model, its computational complexity grows. It isn’t just computational, however - increased complexity requires greater expertise and understanding of data science.

### Question 3c

Let's assess the performance of our new linear regression model using the Root Mean Squared Error function that we created in Homework 5.

$$RMSE = \sqrt{\dfrac{\sum_{\text{houses}}(\text{actual price for house} - \text{predicted price for house})^2}{\text{# of houses}}}$$

The function is provided below.

```python
def rmse(predicted, actual):
    """
    Calculates RMSE from actual and predicted values
    Input:
      predicted (1D array): vector of predicted/fitted values
      actual (1D array): vector of actual values
    Output:
      a float, the root-mean square error
    """
    return np.sqrt(np.mean((actual - predicted)**2))
```

Please compute the training error using the `rmse` function above.

*The provided tests for this question do not confirm that you have answered correctly; only that you have assigned each variable to a non-negative number.*

```python
training_error = ...
print("Training RMSE: {}".format(training_error))
```

```python
ok.grade("q3c");
```

A slighlty modified version of the `cross_validate_rmse` function from Lecture 18 is also provided below.

```python
from sklearn.model_selection import KFold
from sklearn.base import clone

def cross_validate_rmse(model, X, y):
    model = clone(model)
    five_fold = KFold(n_splits=5)
    rmse_values = []
    for tr_ind, va_ind in five_fold.split(X):
        model.fit(X.iloc[tr_ind,:], y.iloc[tr_ind])
        rmse_values.append(rmse(y.iloc[va_ind], model.predict(X.iloc[va_ind,:])))
    return np.mean(rmse_values)
```

Now use the `cross_validate_rmse` functions to calculate the cross validation error in the cell below.

*The provided tests for this question do not confirm that you have answered correctly; only that you have assigned each variable to a non-negative number.*

```python
cv_error = ...
print("Cross Validation RMSE: {}".format(cv_error))
```

```python
ok.grade("q3d");
```

# Part 6: Open-Response

The following part is purposefully left nearly open-ended.  The Ames data in your possession comes from a larger data set.  Your goal is to provide a linear regression model that accurately predicts the prices of the held-out homes, measured by root mean square error.

$$RMSE = \sqrt{\dfrac{\sum_{\text{houses in public test set}}(\text{actual price for house} - \text{predicted price for house})^2}{\text{# of houses}}}$$

Perfect prediction of house prices would have a score of 0, so you want your score to be as low as possible!

### Grading Scheme

Your grade for Question 4 will be based on your training RMSE and test RMSE. The thresholds are as follows:

Points | 3 | 2 | 1 | 0
--- | --- | --- | --- | ---
Training RMSE | Less than 36k | 36k - 38k | 38k - 40k | More than 40k

Points | 3 | 2 | 1 | 0
--- | --- | --- | --- | ---
Test RMSE | Less than 37k | 37k - 40k | 40k - 43k | More than 43k

### One Hot Encoding

If you choose to include more categorical features in your model, you'll need to one-hot-encode each one. Remember that if a categorical variable has a unique value that is present in the training set but not in the test set, one-hot-encoding this variable will result in different outputs for the training and test sets (different numbers of one-hot columns). Watch out for this! Feel free to look back at how we [one-hot-encoded `Fireplace_Qu`](#important_note).

To generate all possible categories for a categorical variable, we suggest reading through `codebook.txt` or finding the values programmatically across both the training and test datasets.

## Question 4: Your Own Linear Model

Just as in the guided model above, you should encapsulate as much of your workflow into functions as possible. Below, we have initialized `final_model` for you. Your job is to select better features and define your own feature engineering pipeline in `process_data_fm`. We recommend using cross validation to help inform your feature selection process.

To evaluate your model, we will process training data using your `process_data_fm`, fit `final_model` with this training data, and compute the training RMSE. Then, we will process the test data with your `process_data_fm`, use `final_model` to predict sale prices for the test data, and compute the test RMSE. See below for an example of the code we will run to grade your model:

```
training_data = pd.read_csv('ames_train_cleaned.csv')
test_data = pd.read_csv('ames_test_cleaned.csv')

X_train, y_train = process_data_fm(training_data)
X_test, y_test = process_data_fm(test_data)

final_model.fit(X_train, y_train)
y_predicted_train = final_model.predict(X_train)
y_predicted_test = final_model.predict(X_test)

training_rmse = rmse(y_predicted_train, y_train)
test_rmse = rmse(y_predicted_test, y_test)
```

**Note:** It is your duty to make sure that all of your feature engineering and selection happens in `process_data_fm`, and that the function performs as expected without errors. We will **NOT** accept regrade requests that require us to go back and run code that require typo/bug fixes.

**Hint:** Some features may have missing values in the test set but not in the training set. Make sure `process_data_fm` handles missing values appropriately for each feature!

```python
final_model = lm.LinearRegression(fit_intercept=True) # No need to change this!

def process_data_fm(data):
    ...
    # Return predictors and response variables separately
    X = data.drop(['SalePrice'], axis = 1)
    y = data.loc[:, 'SalePrice']
    return X, y
```

```python
ok.grade("q4");
```

## Question 5: EDA for Feature Selection

In the following question, explain a choice you made in designing your custom linear model in Question 4. First, make a plot to show something interesting about the data. Then explain your findings from the plot, and describe how these findings motivated a change to your model.

### Question 5a

In the cell below, create a visualization that shows something interesting about the dataset.

```python
# Code for visualization goes here
...
```

### Question 5b

Explain any conclusions you draw from the plot above, and describe how these conclusions affected the design of your model. After creating the plot, did you add/remove certain features from your model, or did you perform some other type of feature engineering? How significantly did these changes affect your rmse?

*Write your answer here, replacing this text.*

### Bias in Modeling

As the previous question highlighted, a series of decisions are built into your model: When the goal is to minimize rmse (and receive full homework points!), it makes sense to use features that carry a lot of weight. These choices, however, create a representation of reality - intentional or not - and dictate what a model considers valuable when generating any type of prediction.
Having worked through a structured data analysis/modeling process and built your own model, we’ll now take a step back and look at how your work fits into the real world. Because civic data initiatives have multiple stakeholders, it’s imperative to understand how different groups interact with particular aspects of the data. For a student, it’s important to minimize a model’s error for the sake of completing this assignment and learning basic data science principles. For the CCAO, it’s important to project a vision of fairness for housing assessments in order to maintain the trust of Cook County’s constituents. An assessment model has the potential to reach stakeholders beyond the assessor’s office, so let’s explore this further!

## HCE: Question 3
Above, we included `Floodplain` in our model to predict property values. This column could be useful to organizations outside of the CCAO - insurance companies, for example, calculate risk and insure houses based on their characteristics. If an insurance company wanted to assign an insurance rate to a house, why might they use `Floodplain` from the county assessor as part of their calculation?

`TODO:` *Write your answer here*

## HCE: Question 4
This dataset, and many like it around the country, are available free or for sale to any business. What kinds of businesses and industries would be interested in the feature(s) you identified? How could a business or industry use the feature(s) to make a decision that would help them?

`TODO:` *Write your answer here*

## HCE: Question 5
In the cell below, generate a visualization of the feature(s) you selected which could help the business or industry use this data to make a decision.

`TODO:` *Write your answer here*

## HCE: Question 6
While you have argued that this feature could help a given business make an informed decision, is there a potential for the feature(s) to mislead them? How so?

*Hint:* If you’re stuck, consider the limitations of bias in data collection.

`TODO:` *Write your answer here*

As your work through these questions demonstrates, your features and modeling process have relevance to a diversity of fields. You may have noticed, however, that the quality of this information is contingent on its explainability, i.e. what a particular variable would mean or represent to different industries. This need for context and clarity not only reveals the underlying biases of our work but also characterizes another common approach to fairness: transparency.

## HCE: Question 7
What would you consider a transparent and fair process in regard to home assessment, and how might you implement these ideas in your modeling process?

`TODO:` *Write your answer here*

# Part 7: Approaching Fairness through Transparency

In this homework, the validity of your model’s assessments is determined by how closely your test set’s results align with the CCAO’s residential sales dataset. The CCAO, however, does not have an autograder to check its work; its predictions, after all, draw on this dataset to re-establish the standard for fair and accurate property values throughout the triennial assessment period. Instead, the Office champions transparency as a guiding principle in regard to fairness.

### Transparency and the CCAO

After a lawsuit was filed against the CCAO for producing [“racially discriminatory assessments and taxes,"](https://harris.uchicago.edu/news-events/news/prof-chris-berry-testifies-institutional-racism-cook-county-property-taxes) the Office decided to tackle these inequities by committing to transparency initiatives. The hope was that, by exposing the inner workings of the CCAO’s property valuation process, their assessment results could be publicly verified as accurate and therefore trusted to be fair.

These transparency initiatives include publishing all of the CCAO’s work on [GitLab](https://gitlab.com/ccao-data-science---modeling). By allowing the public to access any updates to the system in real-time, the Office argues that they increase accessibility to a process that had previously been blackboxed - obscured and hidden - from the public. Empowered by transparency, the citizens of Cook County would ideally hold the Assessor’s Office accountable for their work, redistributing the balance of power between the CCAO and its constituents. And in this scenario, this form of transparency would thus contribute to the legitimacy and fairness of the Office’s property assessments.

Additionally, these measures were, in part, developed to push back against the inequities of the tax lawyer industry. Because hiring a tax lawyer to negotiate for lower valuations (and therefore taxes) is limited to the wealthy, property owners with a lower socioeconomic status paid a disproportionate amount of tax. However, because the CCAO’s assessment process is now public, tax lawyers can only contest the CCAO’s work through technical means - in other words, tax lawyers must abide by the rules set by the CCAO. In this way, the transparency initiatives aim to shift the balance of power in property assessment and taxes away from the tax lawyer industry.

## HCE: Question 8

How do the CCAO’s transparency initiatives aim to redistribute power between the tax lawyer industry, the CCAO, and the constituents of Cook County?

`TODO:` *Write your answer here*

With these balances of power in mind, the next step is to critically examine the effectiveness and reach of the CCAO’s transparency initiatives. We can assess them in three ways:

* Accessibility - Are there barriers or limits to participating in the transparency initiatives implemented by the CCAO? More specifically, who can and cannot interact with and understand the CCAO’s published code on GitLab?
* Explainability - What efforts has the CCAO made to effectively communicate information about their assessment process? To what extent do they elaborate on the documentation of their GitLab repository?
* Accountability - In what ways can the CCAO be held accountable through their transparency initiatives? Are there barriers to who can hold them accountable?

As you may have noticed, these terms are closely linked to one another. We’ll now examine these standards for transparency by diving deeper into the CCAO’s assessment process.

According to the CCAO’s [Progress Report on Implementation of 100 Day Objectives](https://gitlab.com/ccao-data-science---modeling/ccao_sf_cama_dev/-/blob/master/documentation/Progress%20Report%20on%20Implementation%20of%20100%20Day%20Objectives.pdf), their  assessment system operates beyond a single model or series of models: It allows “any number of models to be specified and runs all of them (...) Each model is subjected to a battery of tests as outlined by the International Association of Assessing Officers. The algorithm then recommends a model based on its performance on these tests.” This complex modeling, coupled with real estate-based data collection practices, enable the CCAO to leverage their data scientists’ expertise in their final assessments. It’s important to acknowledge this power dynamic because it directly interacts with the notion of transparency in the assessment system.

Let’s start with accessibility. To be frank, the CCAO’s algorithmic modeling system is almost completely inaccessible to the everyday person, despite its viewability on GitLab. Understanding code - much less algorithms and statistical models written in code - is a high barrier to entry, severely restricting accessibility on the basis of expertise. This is where the next metric for transparency, explainability, comes in.

## HCE: Question 9

Take a look at the Residential Automated Valuation Model files under the Models subgroup in the CCAO’s [GitLab](https://gitlab.com/ccao-data-science---modeling). Without directly looking at any code, does the documentation sufficiently explain how the residential valuation model works?

`TODO:` *Write your answer here*

With the barrier of expertise, explanations are perhaps the only way that the CCAO can bridge the technical gaps throughout the assessment process. However, as you might’ve noticed, these explanations are also lacking. That leaves the final piece of the puzzle - the ideal product of transparency, accountability.

Without any measures to expand the accessibility and explainability of their algorithmic modeling, the prospects of holding the CCAO accountable for their assessments are dim. The level of expertise needed to check the CCAO’s systems excludes participatory voices from the community. Even community members who have valuable insights into measures of inequity - itself another form of expertise - cannot engage with technical expertise in data and modeling.

WIth all that said, it’s imperative to acknowledge that the CCAO is a government institution. Although their work relies on technical expertise, anyone who wishes to critique - or even just interact with - the assessment pipeline should have the means to; all of Cook County’s constituents are affected by property assessments, and they should not be required to have a technical background in order to participate in a technical space.

## HCE: Question 10

In what ways does the CCAO’s implementation of transparency fail? What aspects of it are inaccessible and to whom? Consider the concepts of expertise and power.

`TODO:` *Write your answer here*

So what role does transparency really play in relation to fairness? Given its limitations in regard to accessibility, the CCAO’s transparency initiatives can come across as merely performative. That said, they certainly still demonstrate the Office’s commitment to fairness and equity, performing the social function of instilling faith in its work. And while there is validity in fostering trust with the local community as a governing institution, it’s nonetheless vital to understand what this faith is built on: The CCAO operates its assessment model by leveraging its expertise, and transparency is then used to reinforce the legitimacy of the CCAO’s work by exemplifying a gesture of good will. And through this process, the CCAO’s power to determine property assessments and taxes is thus maintained through this process.

## HCE: Question 11

How does the CCAO maintain its power in housing assessments and its open-data initiative?

`TODO:` *Write your answer here*

Fairness, in the end, is as difficult to define as it is to implement. What you make of this process - from the ingrained bias throughout the data lifecycle to the role of transparency in the CCAO’s work - is for your consideration. Regardless, it raises several challenging but pressing questions: How can we envision fair and equitable data science beyond transparency? How can we go one step further and incorporate justice in data science?

The Cook County Assessor’s Office is just one case study. Although it is unique in being the first Assessor’s Office to publicly publish its assessment model and data, it is not exempt from the same scrutiny and critique that other public and private institutions receive. Consider how expertise serves as the baseline and standard for fairness - and consider how data science becomes authoritative despite (or, perhaps, because of) its many exceptions, parameters, and metrics. Continue to reflect on these concepts in your data science work as you learn more and more.

## Before You Submit

Make sure that if you run Kernel > Restart & Run All, your notebook produces the expected outputs for each cell. Congratulations on finishing the assignment!

# Submit
Make sure you have run all cells in your notebook in order before running the cell below, so that all images/graphs appear in the output.
**Please save before submitting!**

```python
# Save your notebook first, then run this cell to submit.
import jassign.to_pdf
jassign.to_pdf.generate_pdf('hw6.ipynb', 'hw6.pdf')
ok.submit()
```


# Companion notebook of the paper _Fast sampling of $\beta$-ensembles_
by [Guillaume Gautier](http://guilgautier.github.io/), [Rémi Bardenet](https://rbardenet.github.io/), and [Michal Valko](http://researchers.lille.inria.fr/~valko/hp/)

See also the [arXiv preprint: 2003.02344](http://arxiv.org/abs/2003.02344)

Table of Contents

1  Companion notebook of the paper Fast sampling of $\beta$-ensembles

1.1  Summary

2  Imports

2.1  $V(x) = g_{2m} x^{2m}$

2.1.1  $V(x) = \frac{1}{2} x^2$ (Hermite ensemble)

2.1.2  $V(x) = \frac{1}{4} x^4$

2.1.3  $V(x) = \frac{1}{6} x^6$

2.2  $V(x) = g_2 x^2 + g_4 x^4$

2.2.1  $V(x)= \frac{1}{4} x^4 + \frac{1}{2} x^2$

2.2.2  $V(x)= \frac{1}{4} x^4 - x^2$ (onset of two-cut solution)

2.2.3  $V(x)= \frac{1}{4} x^4 - \frac{5}{4} x^2$ (Two-cut eigenvalue distribution)

2.3  $V(x) = \frac{1}{20} x^4 - \frac{4}{15}x^3 + \frac{1}{5}x^2 + \frac{8}{5}x$

## Summary
We focus on sampling $\beta$-ensembles with $N$ points associated to polynomial potentials $V$ with even degree which take the form

$$
\begin{equation}
\label{eq:potential_V}
V(x) = \frac{g_6}{6} x^6
    + \frac{g_4}{4} x^4
    + \frac{g_3}{3} x^3
    + \frac{g_2}{2} x^2
    + g_1 x
\end{equation}
$$

We derive a **fast but approximate procedure to generate samples from the targeted $\beta$-ensemble**. In other words, we wish to sample from $(x_{1},\dots, x_N)$ with joint distribution proportional to

$$
\begin{equation}
\label{eq:joint_x}
\left|\Delta(x_1,\dots,x_N)\right|^{\beta}
    ~ \exp^{-\sum_{i=1}^N V(x_i)}
    \prod_{i=1}^N d x_i
\end{equation}
$$

where $\Delta(x_1,\dots,x_N) = \prod_{1\leq i < j \leq N} (x_j-x_i)$.

To do this we view $x_1, \dots, x_N$ as the eigenvalues of a random *Jacobi* matrix, i.e., a real-symmetric, tridiagonal matrix with positive subdiagonal

$$
\begin{equation}
\label{eq:jacobi_matrix_J_N_a_b}
J_{ab}=
\begin{bmatrix}
        a_1 		& \sqrt{b_1}& 0              &    0           \\
        \sqrt{b_1}  & a_2 		& \ddots         &    0           \\
        0    		& \ddots   	& \ddots      	 & \sqrt{b_{N-1}} \\
        0 			& 0 		& \sqrt{b_{N-1}} & a_{N}
\end{bmatrix}
\end{equation}.
$$

---

To draw the correspondence with Jacobi matrices, we first augment the distribution of the points $x_1,\dots,x_N$ with auxilary weights $w_{1}, \dots, w_{N}$, distributed independently from the points as a Dirichlet $\operatorname{Dir}\left(\frac{\beta}{2}\right)$, so that

$$
\begin{equation}
\label{eq:joint_x_w}
    (x_{1:N}, w_{1:N-1})
    \sim
    \left|\Delta(x_1,\dots,x_N)\right|^{\beta}
        ~ \exp^{- \sum_{i=1}^N V(x_i)}
            \prod_{i=1}^N d x_i
    \prod_{i=1}^{N-1} w_i^{\frac{\beta}{2}-1} d w_i
\end{equation}
$$

This allows to consider the random measure $\mu = \sum_{n=1}^N w_n \delta_{x_n}$ supported on the targeted $\beta$-ensemble.
Taking for
The corresponding Jacobi matrix $J_{ab}$ characterizes the three-term recurrence relation between orthonormal polynomials w.r.t. $\mu$ and Favard's theorem give that the mapping $\mu \mapsto J_{ab}$ is a diffeomorphism.
We can thus convert the distribution \eqref{eq:joint_x_w} of the nodes and weights of $\mu$ to the distribution of the entries of $J_{ab}$, namely

$$
\begin{equation}
\label{eq:joint_a_b}
    (a_{1:N}, b_{1:N-1})
    \sim
    \prod_{i=1}^{N-1}
    b_{i}^{\frac{\beta}{2}(N-i)-1}
    \exp^{-\operatorname{Tr}[V(J_N(a, b)]}
        d a_{1:N}, b_{1:N-1}
\end{equation}
$$

see, e.g., [[KrRiVi13, Proposition 2]](http://de.arxiv.org/pdf/1306.4832.pdf).

---

Hence, computing the eigenvalues of the Jacobi matrix $J_{ab}$ \eqref{eq:jacobi_matrix_J_N_a_b} with entries sampled from \eqref{eq:joint_a_b} provides a way to sample the corresponding $\beta$-ensemble \eqref{eq:joint_x} with $\mathcal{O}(N^2)$ complexity.
The question remains as to sample efficiently from \eqref{eq:joint_a_b}.

For specific potentials, namely $V(x)=x^2, x - \log(x), - \log(1-x) + \log(1+x)$ that are respectively associated to the Hermite, Laguerre and Jacobi ensembles, the stars align perfectly to make the entries of $J_{ab}$ independent with easy-to-sample distributions: Gaussian, Gamma, Beta.

But for more general potentials $V$, the Jacobi coefficients are no longer indenpendent and sampling from the distribution \eqref{eq:joint_a_b} remains a challenge.
Yet, when considering polynomials potentials, the interaction between the different parameters $a_n, b_n$ has short range, driven by the degree of $V$.
We exploit this short range of interaction using a Gibbs sampler on Jacobi matrices to generate fast but approximate samples from the targeted $\beta$ ensemble.

**After rescaling the polynomial potentials \eqref{eq:potential_V} as $V\leftarrow\frac{\beta N}{2} V$**,

for the chain of Jacobi matrices, we compare the empirical distribution of
- the eigenvalues to the expected equilibrium distribution
- the largest eigenvalue to the Tracy-Widom distribution, as expected for some potentials $V$ when $\beta=2$.

**Our empirical study seems to confirm the fast $\log(N)$ mixing time sugested by [[KrRiVi13, p.6]](http://de.arxiv.org/pdf/1306.4832.pdf).**

**Note:**
If you are interested in sampling exactly from the classical (Hermite, Laguerre and Jacobi) $\beta$-ensembles, please refer to the corresponding section in the [tutorial notebook of the DPPy toolbox](https://github.com/guilgautier/DPPy/tree/master/notebooks).

# Imports

**Here are the detailed [INSTALLATION INSTRUCTIONS](https://github.com/guilgautier/DPPy#installation) of DPPy**

To install the last version available on PyPI you can use [](https://badge.fury.io/py/dppy)

```python
# !pip install dppy
```

💣 **Note: The version available on PyPI might not be the latest version of DPPy.
Please consider forking or cloning DPPy using**

```python
# !rm -r DPPy
# !git clone https://github.com/guilgautier/DPPy.git
# !pip install scipy --upgrade

# Then
# !pip install DPPy/.
# OR
# !pip install DPPy/.['zonotope','trees','docs'] to perform a full installation.
```

💣 If you have chosen to clone the repo and now wish to interact with the source code while running this notebook.
You can uncomment the following cell.

```python
%load_ext autoreload
%autoreload 2
import os
import sys
sys.path.insert(0, os.path.abspath('..'))
```

```python
import numpy as np
import matplotlib.pyplot as plt
%config InlineBackend.figure_format = 'retina'

from dppy.beta_ensemble_polynomial_potential import BetaEnsemblePolynomialPotential
```

💻**You can play with the various parameters, e.g.,** $N, \beta, V$ `nb_gibbs_passes` 💻

## $V(x) = g_{2m} x^{2m}$

We first consider even monomial potentials whose equilibrium distribution can be derived from [[Dei00, Proposition 6.156]](https://bookstore.ams.org/cln-3)

### $V(x) = \frac{1}{2} x^2$ (Hermite ensemble)

This this the potential associated to the Hermite ensemble.

In this case, the Jacobi parameters are all independent and sampling is exact [[DuEd02, II C]](https://arxiv.org/abs/math-ph/0206043).

In our setting, this corresponds to a single pass of the Gibbs sampler over each variable.

```python
beta, V = 2, np.poly1d([0.5, 0, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x2 = be.sample_mcmc(N=1000, nb_gibbs_passes=1,
                          sample_exact_cond=True)
```

```python
be.hist(sampl_x2)
```

### $V(x) = \frac{1}{4} x^4$

To depart from the classical quadratic potential we consider the quartic potential, which has been sampled by
[[LiMe13]](https://arxiv.org/abs/1306.1179)
[[OlNaTr15]](https://arxiv.org/abs/1404.0071)
[[ChFe19, Section 3.1]](https://arxiv.org/abs/1806.05985)

```python
beta, V = 2, np.poly1d([1/4, 0, 0, 0, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x4 = be.sample_mcmc(N=200, nb_gibbs_passes=10,
                          sample_exact_cond=True)
#                           sample_exact_cond=False,
#                           nb_mala_steps=100)
```

```python
be.hist(sampl_x4)
```

### $V(x) = \frac{1}{6} x^6$

This is the first time the sextic ensemble is (approximately) sampled to the best of our knowledge.

In this case, the conditionals associated to the $a_n$ parameters are not $\log$-concave and we do not support exact sampling but perform a few steps (100 by defaults) of MALA.
For this reason, we set `sample_exact_cond=False`.

```python
beta, V = 2, np.poly1d([1/6, 0, 0, 0, 0, 0, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x6 = be.sample_mcmc(N=200, nb_gibbs_passes=10,
                          sample_exact_cond=False,
                          nb_mala_steps=100)
```

```python
be.hist(sampl_x6)
```

## $V(x) = g_2 x^2 + g_4 x^4$

We consider quartic potentials where $g_2$ varies to reveal equilibrium distributions with support which are connected, about to be disconnect and fully disconnected.

We refer to
[[DuKu06, p.2-3]](https://arxiv.org/pdf/math/0605201.pdf),
[[Molinari, Example 3.3]](http://pcteserver.mi.infn.it/~molinari/RMT/RM2.pdf) and
[[LiMe13, Section 2]](https://arxiv.org/pdf/1306.1179.pdf)
for the exact shape of the corresponding equilibrium densities.

### $V(x)= \frac{1}{4} x^4 + \frac{1}{2} x^2$

This case reveals an equilibrium density with a connected support.

```python
beta, V = 2, np.poly1d([1/4, 0, 1/2, 0, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x4_x2 = be.sample_mcmc(N=1000, nb_gibbs_passes=10,
                             sample_exact_cond=True)
#                              sample_exact_cond=False,
#                              nb_mala_steps=100)
```

```python
be.hist(sampl_x4_x2)
```

### $V(x)= \frac{1}{4} x^4 - x^2$ (onset of two-cut solution)

This case reveal an equilibrium density with support which is about to be disconnected.

The conditionals associated to the $a_n$ parameters are not $\log$-concave and we do not support exact sampling but perform a few steps (100 by defaults) of MALA.
For this reason, we set `sample_exact_cond=False`.

```python
beta, V = 2, np.poly1d([1/4, 0, -1, 0, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x4_x2_onset_2cut = be.sample_mcmc(N=1000, nb_gibbs_passes=10,
                                        sample_exact_cond=False,
                                        nb_mala_steps=100)
```

```python
be.hist(sampl_x4_x2_onset_2cut)
```

### $V(x)= \frac{1}{4} x^4 - \frac{5}{4} x^2$ (Two-cut eigenvalue distribution)

This case reveals an equilibrium density with support having two connected components.

The conditionals associated to the $a_n$ parameters are not $\log$-concave and we do not support exact sampling but perform a few steps (100 by defaults) of MALA.
For this reason, we set `sample_exact_cond=False`.

```python
beta, V = 2, np.poly1d([1/4, 0, -1.25, 0, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x4_x2_2cut = be.sample_mcmc(N=200, nb_gibbs_passes=10,
                                  sample_exact_cond=False,
                                  nb_mala_steps=100)
```

```python
be.hist(sampl_x4_x2_2cut)
```

## $V(x) = \frac{1}{20} x^4 - \frac{4}{15}x^3 + \frac{1}{5}x^2 + \frac{8}{5}x$

This case reveals a singular behavior at the right edge of the support of the equilibrium density

The conditionals associated to the $a_n$ parameters are not $\log$-concave and we do not support exact sampling but perform a few steps (100 by defaults) of MALA.
For this reason, we set `sample_exact_cond=False`.

We refer to [[ClItsKr10, Example 1.2]](https://arxiv.org/abs/0901.2473)
[[OlNaTr14, Section 3.2]](https://arxiv.org/pdf/1404.0071.pdf) for the expression of the corresponding equilibrium density.

```python
beta, V = 2, np.poly1d([1/20, -4/15, 1/5, 8/5, 0])
be = BetaEnsemblePolynomialPotential(beta, V)
```

```python
sampl_x4_x3_x2_x = be.sample_mcmc(N=200, nb_gibbs_passes=10,
                                  sample_exact_cond=False,
                                  nb_mala_steps=100)
```

```python
be.hist(sampl_x4_x3_x2_x)
```


# Inference example for trained 2D U-Net model on BraTS.
Takes a trained model and performs inference on a few validation examples.

```python
import sys
import platform
print("Python version:\n", sys.version)
print ("Path to the python executable:\n", sys.executable)
```

    Python version:
     3.6.3 |Intel Corporation| (default, May  4 2018, 04:22:28)
    [GCC 4.8.2 20140120 (Red Hat 4.8.2-15)]
    Path to the python executable:
     /home/user01/miniconda3/envs/idp/bin/python

```python
%matplotlib inline
import os
import psutil
import numpy as np
import keras as K
import h5py
import time
import tensorflow as tf
import sys; sys.argv=['']; del sys

from argparser import args
from model import unet

if args.keras_api:
    import keras as K
else:
    from tensorflow import keras as K

import matplotlib.pyplot as plt

onnx=False
plot_result = True
#TODO - Enable nGraph Bridge - Switch to (decathlon) venv!
# import ngraph_bridge

if onnx:
    #TODO - Include ngraph onnx backend
    import onnx
    from ngraph_onnx.onnx_importer.importer import import_onnx_model
    import ngraph as ng

print ("We are using Tensorflow version", tf.__version__,\
       "with Intel(R) MKL", "enabled" if tf.pywrap_tensorflow.IsMklEnabled() else "disabled",)
```

    /home/user01/miniconda3/envs/idp/lib/python3.6/site-packages/h5py/__init__.py:36: FutureWarning: Conversion of the second argument of issubdtype from `float` to `np.floating` is deprecated. In future, it will be treated as `np.float64 == np.dtype(float).type`.
      from ._conv import register_converters as _register_converters
    Using TensorFlow backend.

    We are using Tensorflow version 1.12.0 with Intel(R) MKL enabled

```python
# Create output directory for images
png_directory = "inference_examples"
if not os.path.exists(png_directory):
    os.makedirs(png_directory)

data_filename = os.path.join(args.data_path, args.data_filename)
model_filename = os.path.join(args.output_path, args.inference_filename)
```

#### Define the DICE coefficient and loss function

The Sørensen–Dice coefficient is a statistic used for comparing the similarity of two samples. Given two sets, X and Y, it is defined as

\begin{equation}
dice = \frac{2|X\cap Y|}{|X|+|Y|}
\end{equation}

```python
def calc_dice(target, prediction, smooth=0.01):
    """
    Sorensen Dice coefficient
    """
    prediction = np.round(prediction)

    numerator = 2.0 * np.sum(target * prediction) + smooth
    denominator = np.sum(target) + np.sum(prediction) + smooth
    coef = numerator / denominator

    return coef

def calc_soft_dice(target, prediction, smooth=0.01):
    """
    Sorensen (Soft) Dice coefficient - Don't round preictions
    """
    numerator = 2.0 * np.sum(target * prediction) + smooth
    denominator = np.sum(target) + np.sum(prediction) + smooth
    coef = numerator / denominator

    return coef
```

## Inference Time!

Inferencing in this example can be done in 3 simple steps:
1. Load the data
1. Load the Keras model
1. Perform a `model.predict` on an input image (or set of images)

#### Step 1 : Load data

```python
df = h5py.File(data_filename, "r")
imgs_testing = df["imgs_testing"]
msks_testing = df["msks_testing"]
files_testing = df["testing_input_files"]
```

#### Step 2 : Load the model

```python
unet_model = unet()
model = unet_model.load_model(model_filename)
```

    Data format = channels_last

#### Step 3 (final step): Perform prediction with `model.predict`

We first define a convenience function that performs a prediction on one image. This function takes an image ID and a flag `plot_result` which we can use to toggle displaying the results graphically.

After we have performed a prediction, we calculate the dice score to analyze how good out prediction was compared to the ground truth.

```python
def plot_results(model, imgs_validation, msks_validation, img_no, png_directory):

    img = imgs_validation[[img_no], ]
    msk = msks_validation[[img_no], ]

    #TODO load onnx model in ngraph
    if onnx:
        onnx_protobuf = onnx.load('./output/unet_model_for_decathlon_100_iter.onnx')
        ng_models = import_onnx_model(onnx_protobuf)
        ng_model = ng_models[0]
        runtime = ng.runtime(backend_name='CPU')
        unet = runtime.computation(ng_model['output'], *ng_model['inputs'])

        start_time = time.time()
        pred_mask= unet(img)[0]
        print ("Time for prediction ngraph: ", '%.0f'%((time.time()-start_time)*1000),"ms")

    else:
        start_time = time.time()
        pred_mask = model.predict(img, verbose=0, steps=None)
        print ("Time for prediction TF: ", '%.0f'%((time.time()-start_time)*1000),"ms")

    if args.plot:
        plt.figure(figsize=(10, 10))
        plt.subplot(1, 3, 1)
        plt.imshow(img[0, :, :, 0], cmap="bone", origin="lower")
        plt.title("MRI")
        plt.axis("off")
        plt.subplot(1, 3, 2)
        plt.imshow(msk[0, :, :, 0], origin="lower")
        plt.title("Ground Truth")
        plt.axis("off")
        plt.subplot(1, 3, 3)
        plt.imshow(pred_mask[0, :, :, 0], origin="lower")
        plt.title("Prediction\n(Dice = {:.4f})".format(calc_dice(msk, pred_mask)))
        plt.axis("off")

        png_filename = os.path.join(png_directory, "pred_{}.png".format(img_no))
        plt.savefig(png_filename, bbox_inches="tight", pad_inches=0)
        print("Dice {:.4f}, Soft Dice {:.4f}, Saved png file to: {}".format(
            calc_dice(msk, pred_mask), calc_soft_dice(msk, pred_mask), png_filename))
```

#### Step 3 (continued) : Perform prediction on some images.
Use `plot_result=True/False` to toggle plotting the results. If `plot_result=True`, the prediction results will be saved in the output directory for images, which is defined by the `png_directory` variable.

```python
indicies_validation = [40, 63, 43, 55, 99, 101, 19, 46, ] #[40]

for idx in indicies_validation:
    plot_results(model, imgs_testing, msks_testing,
                 idx, args.output_pngs)
```

# Can we perform inference even faster? Hmm..

Let's find out. Move on the the next tutorial section.

`Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. SPDX-License-Identifier: EPL-2.0`

`Copyright (c) 2019 Intel Corporation`