# Regression

### Preliminaries

- Goal
  - Introduction to Bayesian (Linear) Regression
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp. 152-158
    - In this and forthcoming lectures, we will make use of some elementary matrix calculus. The most important formulas are summarized at the bottom of this notebook in an [OPTIONAL SLIDE on matrix calculus](#matrix-calculus).

### Regression - Illustration

Given a set of (noisy) data measurements, find the 'best' relation between an input variable $x \in \mathbb{R}^D$ and input-dependent outcomes $y \in \mathbb{R}$

### Regression vs Density Estimation

- Observe $N$ IID data **pairs** $D=\{(x_1,y_1),\dotsc,(x_N,y_N)\}$ with $x_n \in \mathbb{R}^M$ and $y_n \in \mathbb{R}$.

- Assume that we are interested in (a model for) the responses $y_n$ for **given inputs** $x_n$?, I.o.w. we are interested in building a model for the conditional distribution $p(y|x)$.

- Note that, since $p(x,y)=p(y|x)\, p(x)$, building a model $p(y|x)$ is similar to density estimation with the assumption that $x$ is drawn from a uniform distribution.

### Bayesian Linear Regression

- Next, we discuss (1) model specification, (2) Inference and (3) a prediction application for a Bayesian linear regression problem.

#####  1. Model Specification

- In a _regression_ model, we try to 'explain the data' by a purely deterministic term $f(x_n,w)$, plus a purely random term $\epsilon_n$ for 'unexplained noise',

    $$
    y_n  = f(x_n,w) + \epsilon_n
    $$

-  In a _linear regression_ model, i.e., linear w.r.t. the parameters $w$, we assume that
$$f(x_n,w)= \sum_{j=0}^{M-1} w_j \phi_j(x_n) = w^T \phi(x_n)$$
where $\phi_j(x)$ are called basis functions.
  - For notational simplicity, from now on we will assume $f(x_n,w) = w^T x_n$, with $x_n \in \mathbb{R}^M$.

-  In _ordinary linear regression_ , the noise process $\epsilon_n$ is zero-mean Gaussian with constant variance, i.e.
$$
y_n  = w^T x_n  + \mathcal{N}(0,\beta^{-1}) \,.
$$

- Hence, given a data set  $D=\{(x_1,y_1),\dotsc,(x_N,y_N)\}$, the likelihood for an ordinary linear regression model is

$$\begin{align*}
p(y\,|\,\mathbf{X},w,\beta) &= \mathcal{N}(y\,|\,\mathbf{X} w,\beta^{-1} \mathbf{I}) \\
  &= \prod_n \mathcal{N}(y_n\,|\,w^T x_n,\beta^{-1}) \tag{B-3.10}
\end{align*}$$

where $w = \left(\begin{matrix} w_1 \\ w_2 \\ \vdots \\ w_{M} \end{matrix} \right)$, the $(N\times M)$-dim matrix $\mathbf{X}  = \left(\begin{matrix}x_1^T \\ x_2^T \\ \vdots \\ x_N^T \end{matrix} \right) = \left(\begin{matrix}x_{11},x_{12},\dots,x_{1M}\\ x_{21},x_{22},\dots,x_{2M} \\ \vdots \\ x_{N1},x_{N2},\dots,x_{NM} \end{matrix} \right) $  and $y = \left(\begin{matrix} y_1 \\ y_2 \\ \vdots \\ y_N \end{matrix} \right)$.

- For full Bayesian learning we should also choose a prior $p(w)$:

$$\begin{equation}
p(w\,|\,\alpha) = \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \tag{B-3.52}
\end{equation}$$

- For simplicity, we will assume that $\alpha$ and $\beta$ are known.

##### 2. Inference

- We'll do Bayesian inference for the parameters $w$.

$$\begin{align*}
p(w|D) &\propto p(D|w)\cdot p(w) \\
   &= \mathcal{N}(y\,|\,\mathbf{X} w,\beta^{-1} \mathbf{I}) \cdot \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \\
   &\propto \exp \big( -\frac{\beta}{2} \big( {y - \mathbf{X}w } \big)^T \big( {y - \mathbf{X}w } \big)  - \frac{\alpha}{2}w^T w \big)\tag{B-3.55} \\
   &= \exp\big( -\frac{1}{2} w^T\big(\underbrace{\beta \mathbf{X}^T\mathbf{X} + \alpha \mathbf{I}}_{\Lambda_N}\big)w + \big(\underbrace{\beta \mathbf{X}^Ty}_{\eta_N}\big)^T w - \frac{\beta}{2}y^T y \big) \\
     &\propto \mathcal{N}_c\left(w\,|\,\eta_N,\Lambda_N \right)
\end{align*}$$

with natural parameters (see the [natural parameterization of Gaussian](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/notebooks/The-Gaussian-Distribution.ipynb#natural-parameterization)):

$$\begin{align*}
\eta_N &= \beta\mathbf{X}^Ty \\
\Lambda_N &= \beta \mathbf{X}^T\mathbf{X} + \alpha \mathbf{I}
\end{align*}$$

- Or equivalently (in the moment parameterization of the Gaussian):
$$\begin{align*}
p(w|D) &= \mathcal{N}\left(w\,|\,m_N,S_N \right) \tag{B-3.49} \\
m_N &= \beta S_N \mathbf{X}^T y  \tag{B-3.53}\\
S_N &= \left(\alpha \mathbf{I} + \beta \mathbf{X}^T \mathbf{X}\right)^{-1} \tag{B-3.54}
\end{align*}$$

- Note that B-3.53 and B-3.54 combine to
$$
m_N = \left(\frac{\alpha}{\beta}\mathbf{I} + \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T y\,.
$$

 - (Bishop Fig.3.7) Illustration of sequential Bayesian learning for a simple linear model of the form $y(x, w) =
w_0 + w_1 x$. (Bishop Fig.3.7, detailed description at Bishop, pg.154.)

##### 3. Application: predictive distribution

- Assume we are interested in the distribution $p(y_\bullet \,|\, x_\bullet, D)$ for a new input $x_\bullet$. This can be worked out as (exercise B-3.10)

$$\begin{align*}
p(y_\bullet \,|\, x_\bullet, D) &= \int p(y_\bullet \,|\, x_\bullet, w) p(w\,|\,D)\,\mathrm{d}w \\
&= \int \mathcal{N}(y_\bullet \,|\, w^T x_\bullet, \beta^{-1}) \mathcal{N}(w\,|\,m_N,S_N)\,\mathrm{d}w \\
&= \int \mathcal{N}(y_\bullet \,|\, z, \beta^{-1}) \mathcal{N}(z\,|\,x_\bullet^T m_N,x_\bullet^T S_N x_\bullet)\,\mathrm{d}z \quad \text{(sub. }z=x_\bullet^T w \text{)} \\
&= \int \underbrace{\mathcal{N}(z \,|\, y_\bullet, \beta^{-1}) \mathcal{N}(z\,|\,x_\bullet^T m_N,x_\bullet^T S_N x_\bullet)}_{\text{Use Gaussian product formula SRG-6}}\,\mathrm{d}z  \\
&= \int \mathcal{N}\left(y_\bullet\,|\, m_N^T x_\bullet, \sigma_N^2(x_\bullet) \right) \underbrace{\mathcal{N}\left(z\,|\, m_z, S_z\right)}_{\text{integrate this out}} \mathrm{d}z\\
&= \mathcal{N}\left(y_\bullet\,|\, m_N^T x_\bullet, \sigma_N^2(x_\bullet) \right)
\end{align*}$$
with
$$\begin{align*}
\sigma_N^2(x_\bullet) = \beta^{-1} + x^T_\bullet S_N x_\bullet \tag{B-3.59}
\end{align*}$$

- So, the uncertainty $\sigma_N^2(x_\bullet)$ about the output contains comprises both uncertainty about the process ($\beta^{-1}$) and about the model parameters ($x^T_\bullet S_N x_\bullet$).

### Example Predictive Distribution

- As an example, let's do Bayesian Linear Regression for a synthetic sinusoidal data set and a model with 9 Gaussian basis functions
$$\begin{align*}
y &=\sum_{m=1}^9 w_m \phi_m(x_n) + \mathcal{N}(0,\beta^{-1}) \\
\phi_m(x_n) &= \exp\left( \frac{x_n-\mu_m}{\sigma^2}\right)
\end{align*}$$

- The predictive distributions for $y$ are shown in the following plots (Bishop, Fig.3.8)

- And some plots of draws of posteriors for the functions $w^T \phi(x)$ (Bishop, Fig.3.9)

###  Maximum Likelihood Estimation for Linear Regression Model

- Recall the posterior mean for the weight vector
$$
m_N = \left(\frac{\alpha}{\beta}\mathbf{I} + \mathbf{X}^T \mathbf{X} \right)^{-1} \mathbf{X}^T y
$$
where $\alpha$ is the prior precision for the weights.

- The Maximum Likelihood solution for $w$ is obtained by letting $\alpha \rightarrow 0$, which leads to

$$\begin{equation*}
\hat w_{\text{ML}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T y
\end{equation*}$$

-  The matrix $\mathbf{X}^\dagger \equiv  (\mathbf{X}^T \mathbf{X})^{-1}\mathbf{X}^T$ is also known as the **Moore-Penrose pseudo-inverse** (which is sort-of-an-inverse for non-square matrices).

-  Note that if we have fewer training samples than input dimensions, i.e., if $N<M$, then $\mathbf{X}^T \mathbf{X}$  will not be invertible and maximum likelihood blows up. The Bayesian solution does not suffer from this problem.

###  Least-Squares Regression

-  (You may say that) we don't need to work with probabilistic models. E.g., there's also the deterministic **least-squares** solution: minimize sum of squared errors,

$$\begin{align*} \hat w_{\text{LS}} &= \arg\min_{w} \sum_n {\left( {y_n  - w ^T x_n } \right)} ^2
  = \arg\min_{w} \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X} w } \right)
\end{align*}$$

-  Setting the gradient
$ \frac{\partial \left( {y - \mathbf{X}w } \right)^T \left( {y - \mathbf{X}w } \right)}{\partial w} = -2 \mathbf{X}^T \left(y - \mathbf{X} w  \right)
$ to zero yields the so-called  **normal equations**
$\mathbf{X}^T\mathbf{X} \hat w_{\text{LS}} = \mathbf{X}^T y$  and consequently

$$
\hat w_{\text{LS}} = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T y
$$

which is the same answer as we got for the maximum likelihood weights $\hat w_{\text{ML}}$.

- $\Rightarrow$ Least-squares regression ($\hat w_{\text{LS}}$) corresponds to the (probabilistic) maximum likelihood solution ($\hat w_{\text{ML}}$) if the probabilistic model includes the following assumptions:
  1. The observations are independently and identically distributed (**IID**) (this determines how errors are combined), and
  1. The noise signal $\epsilon_n \sim \mathcal{N}(0,\,\beta^{-1})$ is **Gaussian** distributed (determines the error metric)

- If you use the Least-Squares method, you cannot see (nor modify) these assumptions. The probabilistic method forces you to state all assumptions explicitly!

###  Not Identically Distributed Data

-  Let's do an example regarding changing our assumptions. What if we assume that the variance of the measurement error varies with the sampling index,  $\epsilon_n \sim \mathcal{N}(0,\beta_n^{-1})$?

- The likelihood is now (using $\Lambda \triangleq \mathrm{diag}(\beta_n)$ )
$$
p(y\,|\,\mathbf{X},w,\Lambda) = \mathcal{N}(y\,|\,\mathbf{X} w,\Lambda^{-1} ) \,.
$$

- Combining this likelihood with the prior $p(w) = \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I})$ leads to a posterior

$$\begin{align*}
p(w|D) &\propto p(D|w)\cdot p(w) \\
   &= \mathcal{N}(y\,|\,\mathbf{X} w,\Lambda^{-1} \mathbf{I}) \cdot \mathcal{N}(w\,|\,0,\alpha^{-1}\mathbf{I}) \\
   &\propto \exp \left\{ \frac{1}{2} \left( {y - \mathbf{X}w } \right)^T \Lambda \left( {y - \mathbf{X}w } \right)  + \frac{\alpha}{2}w^T w \right\} \\
   &\propto \mathcal{N}\left(w\,|\,m_N,S_N \right)
\end{align*}$$
with
$$\begin{align*}
m_N &=  S_N \mathbf{X}^T \Lambda y  \\
S_N &= \left(\alpha \mathbf{I} +  \mathbf{X}^T \Lambda \mathbf{X}\right)^{-1}
\end{align*}$$

- And maximum likelihood solution
$$
\hat{w}_{\text{ML}} = \left. m_N\right|_{\alpha \rightarrow 0} = \left(\mathbf{X}^T \Lambda \mathbf{X}\right)^{-1} \mathbf{X}^T \Lambda y
$$

-  This maximum likelihood solution is also called the **Weighted Least Squares** (WLS) solution. (Note that we just stumbled upon it, the crucial aspect is appropriate model specification!)

-  Note also that the dimension of $\Lambda$ grows with the number of data points. In general, models for which the number of parameters grow as the number of observations increase are called **non-parametric models**.

### Code Example: Least Squares vs Weighted Least Squares

- We'll compare the Least Squares and Weighted Least Squares solutions for a simple linear regression model with input-dependent noise:

$$\begin{align*}
x &\sim \text{Unif}[0,1]\\
y|x &\sim \mathcal{N}(f(x), v(x))\\
f(x) &= 5x - 2\\
v(x) &= 10e^{2x^2}-9.5\\
\mathcal{D} &= \{(x_1,y_1),\ldots,(x_N,y_N)\}
\end{align*}$$

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate();
IJulia.clear_output();
```

```julia
using PyPlot, LinearAlgebra

# Model specification: y|x ~ 𝒩(f(x), v(x))
f(x) = 5*x .- 2
v(x) = 10*exp.(2*x.^2) .- 9.5 # input dependent noise variance
x_test = [0.0, 1.0]
plot(x_test, f(x_test), "k--") # plot f(x)

# Generate N samples (x,y), where x ~ Unif[0,1]
N = 50
x = rand(N)
y = f(x) + sqrt.(v(x)) .* randn(N)
plot(x, y, "kx"); xlabel("x"); ylabel("y") # Plot samples

# Add constant to input so we can estimate both the offset and the slope
_x = [x ones(N)]
_x_test = hcat(x_test, ones(2))

# LS regression
w_ls = pinv(_x) * y
plot(x_test, _x_test*w_ls, "b-") # plot LS solution

# Weighted LS regression
W = Diagonal(1 ./ v(x)) # weight matrix
w_wls = inv(_x'*W*_x) * _x' * W * y
plot(x_test, _x_test*w_wls, "r-") # plot WLS solution
ylim([-5,8]); legend(["f(x)", "D", "LS linear regr.", "WLS linear regr."],loc=2);
```

##  OPTIONAL SLIDES

### Some Useful Matrix Calculus

When doing derivatives with matrices, e.g. for maximum likelihood estimation, it will be helpful to be familiar with some matrix calculus. We shortly recapitulate used formulas here.

- We define the **gradient** of a scalar function $f(A)$ w.r.t. an $n \times k$ matrix $A$ as
$$
\nabla_A f \triangleq
    \begin{bmatrix}
\frac{\partial{f}}{\partial a_{11}} & \frac{\partial{f}}{\partial a_{12}} & \cdots & \frac{\partial{f}}{\partial a_{1k}}\\
\frac{\partial{f}}{\partial a_{21}} & \frac{\partial{f}}{\partial a_{22}} & \cdots & \frac{\partial{f}}{\partial a_{2k}}\\
\vdots & \vdots & \cdots & \vdots\\
\frac{\partial{f}}{\partial a_{n1}} & \frac{\partial{f}}{\partial a_{n2}} & \cdots & \frac{\partial{f}}{\partial a_{nk}}
    \end{bmatrix}
$$

- The following formulas are useful (see Bishop App.-C)
$$\begin{align*}
|A^{-1}|&=|A|^{-1} \tag{B-C.4} \\
\nabla_A \log |A| &= (A^{T})^{-1} = (A^{-1})^T \tag{B-C.28} \\
\mathrm{Tr}[ABC]&= \mathrm{Tr}[CAB] = \mathrm{Tr}[BCA] \tag{B-C.9} \\
\nabla_A \mathrm{Tr}[AB] &=\nabla_A \mathrm{Tr}[BA]= B^T \tag{B-C.25} \\
\nabla_A \mathrm{Tr}[ABA^T] &= A(B+B^T)  \tag{B-C.27}\\
 \nabla_x x^TAx &= (A+A^T)x \tag{from B-C.27}\\
\nabla_X a^TXb &= \nabla_X \mathrm{Tr}[ba^TX] = ab^T \notag
\end{align*}$$

```julia

```

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>


# Generative Adversarial Networks with Keras (Work In Progress)

Do you remember how do we generate random values starting from a uniformely distributed random variable $U$? We can do that using the inverse transform techinique, which states that $X=F^{-1}(U)$ is a random variable with [cumulative distribution function](https://en.wikipedia.org/wiki/Cumulative_distribution_function) (CDF) given by $F_X(X) = F$.

For example, assume we want to generate an exponentialy distributed random variable. Wikipedia gives us the following facts:
1. Probability disbutition function (PDF):
\begin{equation}
f_X(x) = \lambda e^{-\lambda x}
\end{equation}
2. CDF:
\begin{equation}
F_X(x) = \int_{-inf}^x f_X(z)dz = 1-e^{-\lambda x}
\end{equation}
3. Inverting that CDF we have:
\begin{equation}
F_X^{-1}(u) = -\frac{1}{\lambda}log(1-u)
\end{equation}

Let's check that with numpy

```python
%matplotlib inline

import numpy as np
import matplotlib.pyplot as plt
l = 1 # lambda
u = np.random.uniform(0, 1, (500,))
e = -np.log(1-u) / l

plt.figure(figsize=(10, 3))

plt.subplot(121)
plt.hist(u)
plt.title('uniform')

plt.subplot(122)
plt.hist(e)
_ = plt.title('exponential')
```

Cool! But what if we want to generate pictures like those in the MNIST dataset? Can we estimate something like and inverse CDF in that case? Well, it happens that this is a hard problem and it has an entire field dedicated to it. That field is called Generative Modelling by the machine learning community. If you remember Restricted Boltzman Machines (RBM), Autoencoders (AE), Sparse Coding, etc, that's what I'm talking about. Here we are interested in a generative model proposed by Goodfellow et. al [1] called Generative Adversarial Networks (GAN).

GANs propose to generate samples from a dataset using (guess what?) adaptive neural networks. Thus, given ac uniformely distributed vector $\mathbf{U}$, the generated samples are given by $\mathbf{X} = G(\mathbf{U})$. Which doesn't quite solve the problem before we define how to train the DNN $G(\cdot)$. Let us think about that next, following Goodfellow et. al [1].

Given a dataset $S$, and a test statistic $D(\cdot)$, we say that a sample $\mathbf{X}$ comes from the underlying PDF that generated the samples in $S$ if $D(\mathbf{X}) = 1$. We reject that hypothesis if $D(\mathbf{X}) = 0$. I would give you a dollar if you can guess how the guys from Bengio's lab proposed to parameterize $D(\cdot)$...

WRONG! They used DNNs.

As a note, their derivations are general enough, and we can use other adaptive models instead. That being said, if $G$ wants to be good enough it must pass the test defined by $D$. On the other hand, $D$ must play it hard and only pass the real samples coming from $S$ itself. This way one can define a min-max game where the cost function for $G$ is the binary crossentropy between $D(G(\mathbf{U}))$ and 1, i.e

\begin{equation}
\mathcal{L}_G = log(D(G(\mathbf{U}))),
\end{equation}

the cost function for $D$, on the other hand, is

\begin{equation}
\mathcal{L}_D = \begin{cases}
        \begin{array}{lcl}
        log(D(\mathbf{X})), \quad\quad if: \mathbf{X}\in S \\
        log(1 - D(\mathbf{X})), \quad if: \mathbf{X}=G(\mathbf{U}).
        \end{array}
\end{cases}
\end{equation}

Note that the final solution (minimum for the combined cost functions above) should give us realistic looking samples $G(\mathbf{U})$ and a totally clueless $D$ returning $D(X) = \frac{1}{2}$. Hopefully we understood that well enough to hack something up with Keras. Let us do that for the MNIST dataset.

```python
# These first lines are a copy paste from keras/examples/mnist_mlp.py
%matplotlib inline

from __future__ import absolute_import
from __future__ import print_function

import numpy as np
import matplotlib.pyplot as plt
import theano
np.random.seed(1337)  # for reproducibility

from copy import copy, deepcopy

from keras.datasets import mnist
from keras.models import Sequential, Graph
from keras.layers.core import Dense, Dropout, Activation, Reshape, Flatten
from keras.optimizers import SGD, Adam, RMSprop
from keras.utils import np_utils, generic_utils
from keras.objectives import binary_crossentropy
from keras.callbacks import Callback
from keras.layers.advanced_activations import LeakyReLU as lrelu
from keras.layers.convolutional import Convolution2D, MaxPooling2D

batch_size = 128
nb_classes = 10
nb_epoch = 20

# the data, shuffled and split between tran and test sets
(X_train, y_train), (X_test, y_test) = mnist.load_data()

X_train = X_train.reshape(60000, 784)
X_test = X_test.reshape(10000, 784)
X_train = X_train.astype("float32")
X_test = X_test.astype("float32")
X_train /= 255
X_test /= 255
print(X_train.shape[0], 'train samples')
print(X_test.shape[0], 'test samples')
```

First we define our model for $D$.

```python
dim = 3000  # dim of the random number vector
mnist_dim = 784

detector = Sequential()
detector.add(Dense(dim, input_dim=mnist_dim))
detector.add(lrelu())
detector.add(Dropout(.3))
detector.add(Dense(dim))
detector.add(Activation('tanh'))
detector.add(Dropout(.3))
detector.add(Dense(1)) # 1: Yes, it belongs to S, 0: fake!
detector.add(Activation('sigmoid'))

```

Next, we make a copy of $D$ that we should use to adapt the cost function for $G$. Note that this copy should not be adapted with gradients coming from $\mathcal{L}_G$, otherwise $D$ will be cheating and we won't learn anything useful. To do that with Keras, we simply set `trainable=False`.

Finally we can define $G$ using our hacked detector.

```python
# Fully Connected model

sampler = Sequential()
sampler.add(Dense(dim, input_dim=dim))
sampler.add(lrelu())
sampler.add(Dense(dim))
sampler.add(lrelu())
sampler.add(Dense(mnist_dim))
sampler.add(Activation('sigmoid'))

# This is G itself!!!
sample_fake = theano.function([sampler.get_input()], sampler.get_output())

# We add the detector G on top, but it won't be adapted with this cost function.
# But here is a dirty hack: Theano shared variables on the GPU are the same for
# `detector` and `detector_no_grad`, so, when we adapt `detector` the values of
# `detector_no_grad` will be updated as well. But this only happens following the
# correct gradients.
# Don't you love pointers? Aliasing can be our friend sometimes.
detector.trainable = False
sampler.add(detector)

opt_g = Adam(lr=.001) # I got better results when
                      # detector's learning rate is faster
sampler.compile(loss='binary_crossentropy', optimizer=opt_g)
```

```python
# debug
opt_d = Adam(lr=.002)
detector.trainable = True
detector.compile(loss='binary_crossentropy', optimizer=opt_d)
detector.predict(np.ones((3, mnist_dim))).shape
```

Here is how we can train our networks. First we sample a uniform random vector $\mathbf{U}$, using that we sample fake images with $G(\mathbf{U})$. We train $G$ to get passing values in the $D$ test, I mean 1. Later, we concatenate the fake images with some real MNIST images and train $D$ to return zeros for the formers and ones for the later.

```python
nb_epoch = 1000 # it takes some time to get something recognizable.
batch_size = 128
num_batches = X_train.shape[0] / 128
fig = plt.figure()
fixed_noise = np.random.uniform(-1, 1, (9, dim)).astype('float32') # Let us visualize how these sampes evolve with training

progbar = generic_utils.Progbar(1000)
try:
    for e in range(nb_epoch):
        loss0 = 0
        loss1 = 0

        for (first, last) in zip(range(0, X_train.shape[0]-batch_size, batch_size),
                                 range(batch_size, X_train.shape[0], batch_size)):
            noise_batch = np.random.uniform(-1, 1, (batch_size, dim)).astype('float32')
            fake_samples = sample_fake(noise_batch)
            true_n_fake = np.concatenate([X_train[first: last],
                                          fake_samples], axis=0)
            y_batch = np.concatenate([np.ones((batch_size, 1)),
                                      np.zeros((batch_size, 1))], axis=0).astype('float32')
            all_fake = np.ones((batch_size, 1)).astype('float32')
            # We take turns adapting G and D. We may give D an upper hand,
            #  letting it train for more turns, keeping G fixed.
            #  Do that increasing the upper hand `uh` variable.
            uh = 2
            if e % uh == 0:
                loss0 += sampler.train_on_batch(noise_batch, all_fake)[0]
            else:
                loss1 += detector.train_on_batch(true_n_fake, y_batch)[0]
            loss = loss0 + loss1

        progbar.add(1, values=[("train loss", loss),
                               ("G loss", loss0),
                               ("D loss", loss1)])

        if e % 10 == 0: # visualize results once in a while
            fixed_fake = sample_fake(fixed_noise)
            plt.clf()
            for i in range(9):
                plt.subplot(3, 3, i+1)
                plt.imshow(fixed_fake[i].reshape((28,28)), cmap='gray')
                plt.axis('off')
            fig.canvas.draw()
            plt.show()
            plt.savefig(str(e)+'.png')
except:
    pass
```

# Convolutional GAN

As an easy exercise for the reader (best way to look knowledgeable when your lecture notes aren't finished yet), I recommend you to modify this code to parameterize $G$ and $D$ as convolutional networks. Check [DCGAN](https://github.com/Newmu/dcgan_code) out for inspiration.


 INF-285 - Computación Científica / ILI-285 - Computación Científica I

 Floating Point Arithmetic

  [S]cientific [C]omputing [T]eam

 Version: 1.17

## Table of Contents
* [Introduction](#intro)
* [The nature of floating point numbers](#nature)
* [Visualization of floating point numbers](#visualization)
* [What is the first integer that is not representable in double precision?](#firstinteger)
* [Loss of significance](#loss)
* [Loss of significance in funcion evaluation](#func)
* [Another analysis (example from textbook)](#another)
* [Acknowledgements](#acknowledgements)

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

## Introduction

Hello! This notebook is an introduction to how our computers handle the representation of real numbers using double-presicion floating-point format. To understand the contents of this notebook you should have at least a basic notion of how binary numbers work.

The aforementioned format occupies 64 bits which are divided as follows:

* 1 bit for the sign
* 11 bits for the exponent
* 52 bits for the mantissa

This means that the very next representable number after $1$ is $1 + 2^{-52}$, and their difference, $2^{-52}$, is the $\epsilon_{mach}$.

Additionally, if you'd like to quickly go from a base-2 integer to a base-10 integer and viceversa, Python has some functions that can help you with that.

```python
int('0b11', 2)
```

    3

```python
bin(9)
```

    '0b1001'

```python
bin(2**53)
```

    '0b100000000000000000000000000000000000000000000000000000'

## The nature of floating point numbers

As we know until now, float representations of real numbers are just a finite and bounded represetation of them. But another interesting thing, is that these floating numbers are distributed across the real numbers.

To see that, it's really important to keep in mind the following property:

\begin{equation} \left|\frac{\text{fl}(x)-x}{x}\right| \leq \frac{1}{2} \epsilon_{\text{mach}} \end{equation}

where $\text{fl}(x)$ means the float representation of $x \in R$. What it says is that **the relative error in representing any non-zero real number x, is bounded by a quantity that depends on the system representation** ($\epsilon_{\text{mach}}$).

Maybe now you're thinking: what relationship does this have with the distribution of floating point numbers? So if we rewrite the previous property like this:

\begin{equation} |\text{fl}(x)-x| \leq \frac{1}{2} \epsilon_{\text{mach}} |x| \end{equation}

it's clearer: **The absolute error (distance) between a real number and its floating point representation is proportional to the real number's magnitude.**

Intuitively speaking, if the representation error of a number increases as its magnitude increases, then it's quite natural that **the distance between a floating point number and the next representable floating point number will increase as the magnitude of such number increases (and conversely)**. Could you prove that?  For now we will prove it experimentally.

We will use a library named **bitstring** to handle different number representations. You can install it with:
```
pip install bitstring
```

```python
import bitstring as bs
```

The next two functions are self-explanatory:

1. `next_float(f)` computes the next representable float number.
2. `epsilon(f)` computes the difference between f and the next representable float number

```python
def next_float(f):
    #packing double-precision foat
    b = bs.pack('>d', f)

    #extracting mantisa as unsigned int
    #and adding up 1
    m = b[12:].uint
    m += 1

    #putting the result in his place
    b[12:] = m

    return b.float

def epsilon(f):
    next_f = next_float(f)
    return next_f - f
```

So if we compute `epsilon(1)` we should get the epsilon machine number. Let's try it:

```python
epsilon(1)
```

    2.220446049250313e-16

In order to prove our hypotesis, we will create an array of values: [1e-32, 1e-31, ..., 1e31, 1e32] and compute their corresponding epsilon.

```python
#values between 10**-32 and 10**+32
values = np.array([10**i for i in range(-32,32)]).astype(float)

#corresponding epsilons
vepsilon = np.vectorize(epsilon)
eps = vepsilon(values)
```

We include now a comparison between a linear scale plot and a loglog scale plot. Which one is more useful here?

```python
fig = plt.figure(figsize=(10,5))

plt.subplot(121)
plt.plot(values, eps,'.',markersize=20)
plt.xlabel('Values')
plt.ylabel('Corresponding Epsilons')
plt.title('Epsilons v/s Values')
plt.grid(True)

plt.subplot(122)
plt.loglog(values, eps,'.')
plt.xlabel('Values')
plt.ylabel('Corresponding Epsilons')
plt.title('Epsilons v/s Values')
plt.grid(True)

fig.tight_layout()
plt.show()
```

As you can see, the hypotesis was right. In other words: Floating point numbers are not linearly distributed across the real numbers, and the distance between them is proportional to their magnitude. **Tiny numbers (~ 0) are closer between each other than bigger numbers are.**

## Visualization of floating point numbers

With the help of `bitstring` library we could write a function to visualize floating point numbers in his binary representation

```python
def to_binary(f):
    b = bs.pack('>d', f)
    b = b.bin
    #show sign + exponent + mantisa
    print(b[0]+' '+b[1:12]+ ' '+b[12:])
```

Let's see some intereseting examples

```python
to_binary(1.)
```

    0 01111111111 0000000000000000000000000000000000000000000000000000

```python
int('0b01111111111', 2)
```

    1023

```python
to_binary(1.+epsilon(1.))
```

    0 01111111111 0000000000000000000000000000000000000000000000000001

```python
to_binary(+0.)
```

    0 00000000000 0000000000000000000000000000000000000000000000000000

```python
to_binary(-0.)
```

    1 00000000000 0000000000000000000000000000000000000000000000000000

```python
to_binary(np.inf)
```

    0 11111111111 0000000000000000000000000000000000000000000000000000

```python
to_binary(-np.inf)
```

    1 11111111111 0000000000000000000000000000000000000000000000000000

```python
to_binary(np.nan)
```

    0 11111111111 1000000000000000000000000000000000000000000000000000

```python
to_binary(-np.nan)
```

    1 11111111111 1000000000000000000000000000000000000000000000000000

```python
to_binary(2.**-1074)
```

    0 00000000000 0000000000000000000000000000000000000000000000000001

```python
print(2.**-1074)
```

    5e-324

```python
to_binary(2.**-1075)
```

    0 00000000000 0000000000000000000000000000000000000000000000000000

```python
print(2.**-1075)
```

    0.0

```python
to_binary(9.4)
```

    0 10000000010 0010110011001100110011001100110011001100110011001101

## What is the first integer that is not representable in double precision?

Recall that $\epsilon_{\text{mach}}=2^{-52}$ in double precision.

```python
to_binary(1)
to_binary(1+2**-52)
```

    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 01111111111 0000000000000000000000000000000000000000000000000001

This means that if we want to store any number in the interval $[1,1+\epsilon_{\text{mach}}]$, only the numbers $1$ and $1+\epsilon_{\text{mach}}$ will be stored. For example, compare the exponent and the mantissa in the previous cell with the following outputs:

```python
for i in np.arange(11):
    to_binary(1+i*2**-55)
```

    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 01111111111 0000000000000000000000000000000000000000000000000001
    0 01111111111 0000000000000000000000000000000000000000000000000001
    0 01111111111 0000000000000000000000000000000000000000000000000001
    0 01111111111 0000000000000000000000000000000000000000000000000001
    0 01111111111 0000000000000000000000000000000000000000000000000001
    0 01111111111 0000000000000000000000000000000000000000000000000001

Now, we can scale this difference such that the scaling factor multiplied but $\epsilon_{\text{mach}}$ is one. The factor will be $2^{52}$. This means $2^{52}\,\epsilon_{\text{mach}}=1$. Repeating the same example as before but with the scaling factor we obtain:

```python
for i in np.arange(11):
    to_binary((1+i*2**-55)*2**52)
```

    0 10000110011 0000000000000000000000000000000000000000000000000000
    0 10000110011 0000000000000000000000000000000000000000000000000000
    0 10000110011 0000000000000000000000000000000000000000000000000000
    0 10000110011 0000000000000000000000000000000000000000000000000000
    0 10000110011 0000000000000000000000000000000000000000000000000000
    0 10000110011 0000000000000000000000000000000000000000000000000001
    0 10000110011 0000000000000000000000000000000000000000000000000001
    0 10000110011 0000000000000000000000000000000000000000000000000001
    0 10000110011 0000000000000000000000000000000000000000000000000001
    0 10000110011 0000000000000000000000000000000000000000000000000001
    0 10000110011 0000000000000000000000000000000000000000000000000001

Which means we can only store exactly the numbers:

```python
to_binary(2**52)
to_binary(2**52+1)
```

    0 10000110011 0000000000000000000000000000000000000000000000000000
    0 10000110011 0000000000000000000000000000000000000000000000000001

This means, the distance now from $2^{52}$ and the following number representable is $1$ !!!! So, what would happend if I can to store $2^{53}+1$?

```python
to_binary(2**53)
to_binary(2**53+1)
```

    0 10000110100 0000000000000000000000000000000000000000000000000000
    0 10000110100 0000000000000000000000000000000000000000000000000000

I can't stored the **Integer** $2^{53}+1$! Thus, the first integer not representable is $2^{53}+1$.

## Loss of significance

As we mentioned, there's a small leap between 1 and the next representable number, which means that if you want to represent a number between those two, you won't be able to do so; that number is nonexistent as it is for the computer, so it'll have to round it to a representable number before storing it in memory.

```python
a = 1.
b = 2.**(-52) #emach
result_1 = a + b     # arithmetic result is 1.0000000000000002220446049250313080847263336181640625
result_1b = result_1-1.0
print("{0:.1000}".format(result_1))
print(result_1b)
print(b)
```

    1.0000000000000002220446049250313080847263336181640625
    2.220446049250313e-16
    2.220446049250313e-16

```python
c = 2.**(-53)
result_2 = a + c     # arithmetic result is 1.00000000000000011102230246251565404236316680908203125
np.set_printoptions(precision=16)
print("{0:.1000}".format(result_2))
print(result_2-a)
```

    1.0
    0.0

```python
to_binary(result_2)
to_binary(result_2-a)
```

    0 01111111111 0000000000000000000000000000000000000000000000000000
    0 00000000000 0000000000000000000000000000000000000000000000000000

```python
d = 2.**(-53) + 2.**(-54)

result_3 = a + d     # arithmetic result is 1.000000000000000166533453693773481063544750213623046875
print("{0:.1000}".format(result_3))
to_binary(result_3)
to_binary(d)
```

    1.0000000000000002220446049250313080847263336181640625
    0 01111111111 0000000000000000000000000000000000000000000000000001
    0 01111001010 1000000000000000000000000000000000000000000000000000

As you can see, if you try to save a number between $1$ and $1 + \epsilon _{mach}$, it will have to be rounded (according to some criteria) to a representable number before being stored, thus creating a difference between the real number and the stored number. This situation is an example of loss of significance.

Does that mean that the "leap" between representable numbers is always going to be $\epsilon _{mach}$? Of course not! Some numbers will require smaller leaps, and some others will require bigger leaps.

In any interval of the form $[2^n,2^{n+1}]$ for $n\in \mathbb{Z}$ and representables, is constant. For example, all the numbers between $2^{-1}$ and $2^0$ (but excluding $2^0$) have a distance of $\epsilon _{mach}/2$ between them. All the numbers between $2^0$ and $2^1$ (excluding $2^1$) have a distance of $\epsilon _{mach}$ between them. Those between $2^1$ and $2^2$ (not including $2^2$) have a distance of $2\,\epsilon _{mach}$ between them, and so on and so forth.

```python
e = 2.**(-1)
f = b/2. # emach/2

result_4 = e + f     # 0.50000000000000011102230246251565404236316680908203125
print("{0:.1000}".format(result_4))

result_5 = e + b     # 0.5000000000000002220446049250313080847263336181640625
print("{0:.1000}".format(result_5))
```

    0.50000000000000011102230246251565404236316680908203125
    0.5000000000000002220446049250313080847263336181640625

```python
g = b/4.

result_5 = e + g     # 0.500000000000000055511151231257827021181583404541015625
print("{0:.1000}".format(result_5))
```

    0.5

We'll let the students find some representable numbers and some non-representable numbers. It's important to note that loss significance can occur in many more operations and functions other that the simple addition of two numbers.

```python
num_1 = a
num_2 = b
result = a + b
print("{0:.1000}".format(result))
```

    1.0000000000000002220446049250313080847263336181640625

## Loss of significance in function evaluation

Loss of Significance is present too in the representation of **functions**. A classical example (which you can see in the guide book), is the next function:

\begin{equation}f(x)= \frac{1 - \cos x}{\sin^{2}x} \end{equation}

Applying trigonometric identities, we can obtain the 'equivalent' function:

\begin{equation}f(x)= \frac{1}{1 + \cos x} \end{equation}

Both of these functions are apparently equals. Nevertheless, its graphics say to us another thing when $x$ is equal to zero.

```python
x = np.arange(-10,10,0.1)
y = (1-np.cos(x))/(np.sin(x)**2)
plt.figure()
plt.plot(x,y,'.')
plt.grid(True)
plt.show()
```

```python
x = np.arange(-10,10,0.1)
y = 1/(1+np.cos(x))
plt.figure()
plt.plot(x,y,'.')
plt.grid(True)
plt.show()
```

```python
x = np.arange(-1,1,0.01)
y = (1-np.cos(x))/np.sin(x)**2
plt.figure()
plt.plot(x,y,'.',markersize=10)
plt.grid(True)
plt.show()
```

```python
y = 1/(1+np.cos(x))
plt.figure()
plt.plot(x,y,'.',markersize=10)
plt.grid(True)
plt.show()
```

This happens because when $x$ is equal to zero, the first function has an indetermination, but previously, the computer makes a subtraction between numbers that are almost equals. This generates a loss of significance, turning the expression close to this point to be zero. However, modifying this expression to make the second function, eliminates this substraction, fixing the error in its calculation when $x=0$.

In conclusion, for us, two representations of a function can be equals, but for the computer they can be different!

## Another analysis (example from textbook)

```python
f1 = lambda x: (1.-np.cos(x))/(np.sin(x)**2)
f2 = lambda x: 1./(1+np.cos(x))
f3 = lambda x: (1.-np.cos(x))
f4 = lambda x: (np.sin(x)**2)
x = np.logspace(-19,0,20)[-1:0:-1]
o1 = f1(x)
o2 = f2(x)
o3 = f3(x)
o4 = f4(x)

print("x,            f1(x),        f2(x),        f3(x),                       f4(x)")
for i in np.arange(len(x)):
    print("%1.10f, %1.10f, %1.10f, %1.25f, %1.25f" % (x[i],o1[i],o2[i],o3[i],o4[i]))
```

    x,            f1(x),        f2(x),        f3(x),                       f4(x)
    1.0000000000, 0.6492232052, 0.6492232052, 0.4596976941318602349895173, 0.7080734182735711756961905
    0.1000000000, 0.5012520863, 0.5012520863, 0.0049958347219742904599116, 0.0099667110793791851425238
    0.0100000000, 0.5000125002, 0.5000125002, 0.0000499995833347366414046, 0.0000999966667111107946004
    0.0010000000, 0.5000001250, 0.5000001250, 0.0000004999999583255032576, 0.0000009999996666667110814
    0.0001000000, 0.4999999986, 0.5000000012, 0.0000000049999999696126451, 0.0000000099999999666666680
    0.0000100000, 0.5000000414, 0.5000000000, 0.0000000000500000041370185, 0.0000000000999999999966667
    0.0000010000, 0.5000444503, 0.5000000000, 0.0000000000005000444502912, 0.0000000000009999999999997
    0.0000001000, 0.4996003611, 0.5000000000, 0.0000000000000049960036108, 0.0000000000000100000000000
    0.0000000100, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000001000000000
    0.0000000010, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000010000000
    0.0000000001, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000100000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000001000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000010
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000000
    0.0000000000, 0.0000000000, 0.5000000000, 0.0000000000000000000000000, 0.0000000000000000000000000

## Libraries
### Please make sure you make all of them your BFF!!

* Numpy - IEEE 754 Floating Point Special Values: https://docs.scipy.org/doc/numpy-1.10.0/user/misc.html
* Matplotlib: http://matplotlib.org/examples/pylab_examples/simple_plot.html
* Nice Trick: https://www.dataquest.io/blog/jupyter-notebook-tips-tricks-shortcuts/

# Acknowledgements
* _Material originally created by professor Claudio Torres_ (`ctorres@inf.utfsm.cl`) _and assistants: Laura Bermeo, Alvaro Salinas, Axel Simonsen and Martín Villanueva. v.1.1. DI UTFSM. March 2016._
* _Update April 2020 - v1.14 - C.Torres_ : Fixing some issues.
* _Update April 2020 - v1.15 - C.Torres_ : Adding subplot.
* _Update April 2020 - v1.16 - C.Torres_ : Adding value of numerator and denominator in example of f1 = lambda x: (1.-np.cos(x))/(np.sin(x)** 2).
* _Update April 2020 - v1.17 - C.Torres_ : Adding section "What is the first integer that is not representable in double precision?"

```python

```


Table of Contents

1  Basic Python

2  Matplotlib

3  Pandas

# Extra exercises
Some extra exercises to practice with Python, Pandas, and Matplotlib.

## Basic Python
The following ToDos help you practice with basic Python syntax and operations.

    ToDo: Add five to the value in the variable <tt>x</tt> below and then raise the result to the third power. Store the result in a variable named <tt>y</tt>.

```python
""" Implement the ToDo here. """
x = -3.5

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(y == 3.375)
print("Well done!")
```

    ToDo: From the list below (<tt>my_list</tt>), extract every element at an odd index (at index 1, at index 3, at index 5, etc.) and store it in a new variable named <tt>my_list_index_odd</tt>. Note: you don't have to use a for-loop for this (but you can, if you want). Remember: Python is zero-indexed (i.e., the first element is at index 0).

```python
""" Implement the ToDo here. """
my_list = ['a', 'b', 'c', 'd', 'e', 'f', 'g', 'h']

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(my_list_index_odd == ['b', 'd', 'f', 'h'])
print("Well done!")
```

    ToDo: From the list below (<tt>my_list2</tt>), extract the strings with five or more letters and store the result (another list) in a variable named <tt>my_list_filtered</tt>.

```python
""" Implement the ToDo here. """
my_list2 = ['hello', 'dear', 'student', ',', 'good', 'luck', 'on', 'this', 'ToDo!']

# YOUR CODE HERE
raise NotImplementedError()
my_list_filtered
```

```python
""" Tests the above ToDo. """
assert(my_list_filtered == ['hello', 'student', 'ToDo!'])
print("Well done!")
```

    ToDo: From the list below (<tt>my_list3</tt>, extract all even numbers (e.g., 0, 2, 100, 52) and store the result (another list) in a variable named <tt>my_list_even</tt>. Note: this ToDo is testing your google skills!

```python
""" Implement the ToDo here. """
my_list3 = [1, 8, -4, 0, 5820, 6823591, 99, 87, 27, 2386, 1242, 1111, 582353]
# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(sorted(my_list_even) == [-4, 0, 8, 1242, 2386, 5820])
print("Well done!")
```

    ToDo: The list below (<tt>graded_per_day</tt>) contains integers that represent the number of essays I graded in the past week (Monday-Friday). Compute the percentage of essays I graded for each day and store the result (another list) in a new variable with the name <tt>percentage_per_day</tt>. Hint: to get a percetage, you divide a number by the sum of the collection and then multiply it by 100.

```python
""" Implement the ToDo here. """
graded_per_day = [20, 10, 8, 12]

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert([float(p) for p in percentage_per_day] == [40.0, 20.0, 16.0, 24.0])
print("Well done!")
```

    ToDo (difficult!): The covariance between two variables ($x$, $y$) of length $n$ is computed as follows:

\begin{align}
\mathrm{cov}_{xy} = \frac{\Sigma_{i}^{n} (x_{i} - \bar{x})(y_{i} - \bar{y})}{n-1}
\end{align}

Compute the covariance (a single float!) between the two lists (<tt>z1</tt> and <tt>z2</tt>) below and store it in a variable with the name <tt>cov_z1z2</tt>. You may use the order of steps below:

* Compute the mean of the two lists
* Create two new lists in which each value has been subtracted with the list's mean
* Write a for-loop (or list comprehension) that multiplies each value in one list with the corresponding value in the other list (resulting in another list)
* Sum the results from the previous step
* Divide the result by the length of the list minus 1

```python
""" Implement the ToDo here. """
import random
z1 = [random.uniform(0, 1) for _ in range(20)]
z2 = [random.uniform(0, 1) for _ in range(20)]

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
import numpy as np
assert(round(cov_z1z2, 3) == round(np.cov(z1, z2)[0, 1], 3))
print("Well done!")
```

## Matplotlib
Two Matplotlib exercises (both relatively difficult).

    ToDo: Below, we simulate the resting heart rate (<tt>rhr</tt>, in beats per minute) for 100 people, who are either male ('M') or female ('F') in this example (stored in the <tt>gend</tt> variable). Using Matplotlib, plot the histogram of the resting heart rate of males and females separately, but in the same plot! Make sure that the transparency of the bars of the histograms are set to 50% (google this!). Add a legend and sensible axis labels.

```python
""" Implement the ToDo here. """
np.random.seed(42)
rhr = np.random.normal(62, 4, size=500).astype(int).tolist()
gend = [np.random.choice(['M', 'F']) for _ in range(500)]

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
# Your plot should look like the one below!
from IPython.display import Image
Image('rhr_by_gender.png')
```

    ToDo: Create a bar graph with the average heart rate for three groups:

People with a RHR < 58 ("athletic")

People with a 58 ≤ RHR ≤ 65("average")

People with a RHR > 65 ("sedentary")

Do this separately for men ("M") and women ("F"), such that there are six bars (athletic/men, athletic/women, average/men, average/women, sedentary/men, sedentary/women).

```python
""" Implement the ToDo here. Requires quite a lot of code! """

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
# Your plot should look like the one below!
from IPython.display import Image
Image('rhr_stratified.png')
```

## Pandas
Some Pandas exercises.

```python
import random
import pandas as pd

n = 30
df = pd.DataFrame({
    'participant_id': ['sub-' + str(i).zfill(2) for i in range(1, n + 1)],
    'gender': [random.choice(['M', 'F']) for _ in range(n)],
    'condition': ['A', 'B', 'C'] * (n // 3),
    'prop_correct': np.random.uniform(0.4, 1, size=n),
    'mean_rt': np.random.normal(200, 20, n)
})
df.iloc[np.random.choice(range(n), size=2), -1] = np.nan
```

    ToDo: Remove all participants with missing reaction time data ("mean_rt" column) and store it in a new variable named <tt>df_clean</tt>.

```python
""" Implement the ToDo here. """

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(df_clean.shape[0] == df.shape[0] - 2)
assert(df_clean.shape[1] == df.shape[1])

print("Well done.")
```

Let's delete the original `df` for now:

```python
del df
```

    ToDo: Select from the <tt>df_clean</tt> dataframe the subset of male participants and save them in a new dataframe with the name <tt>df_m</tt>.

```python
""" Implement the ToDo here. """

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(all(df_m.loc[:, 'gender'] == 'M'))
print("Well done!")
```

    ToDo: Select from the <tt>df_clean</tt> dataframe the subset of participants who have more than 65% correct ("prop_correct") and a mean reaction time smaller than 100 ms. Store the result in a new variable named <tt>df_select</tt>.

```python
""" Implement the ToDo here. """

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(all(df_select.loc[:, 'mean_rt'] < 200))
assert(all(df_select.loc[:, 'prop_correct'] > 0.65))

print("Well done!")
```

    ToDo: Add a new column to the dataframe, "status", which contains either the string "above_average" or "below_average" depending on whether the participant has a "prop_correct" score higher than the average or lower than the average, respectively. This column should be added to the <tt>df_clean</tt> dataframe. Note: if you get a "SettingWithCopyWarning", you may ignore this.

```python
""" Implement the ToDo here. """

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. """
assert(all(df_clean.query("status == 'below_average'")['prop_correct'] < df_clean['prop_correct'].mean()))
assert(all(df_clean.query("status == 'above_average'")['prop_correct'] > df_clean['prop_correct'].mean()))
print("Well done!")
```

    ToDo: Using Matplotlib, create a scatterplot with the variables "prop_correct" (on the x-axis) and "mean_rt" (on the y-axis).

```python
""" Implement the ToDo here. """
# YOUR CODE HERE
raise NotImplementedError()
```

    ToDo: Compute the correlation between proportion correct and mean RT (using, e.g., he <tt>pearsonr</tt> function from the <tt>scipy.stats</tt> module) for each group ("A", "B", and "C") separately. Store these three floats (i.e., the correlations) in a list with the variable name <tt>corrs_conditions</tt>. Hint: take a good look at what the <tt>pearsonr</tt> function returns exactly!

Note that, in real life, doing such "subgroup" analyses is probably a bad idea ;-)

```python
""" Implement the ToDo here. """
from scipy.stats import pearsonr

# YOUR CODE HERE
raise NotImplementedError()
```

```python
""" Tests the above ToDo. Don't use this implementation ;-)"""
ans = df_clean.groupby('condition')[['mean_rt', 'prop_correct']].corr().iloc[1::2, 0].tolist()
np.testing.assert_array_almost_equal(ans, corrs_conditions, decimal=4)
print("Well done!")
```


## Regression Discontinuity Design

We don't stop to think about it much, but it is impressive how smooth nature is. You can't grow a tree without first getting a bud, you can't teleport from one place to another, a wound takes its time to heal. Even in the social realm, smoothness seems to be the norm. You can't grow a business in one day, consistency and hard work are required to build wealth and it takes years before you learn how linear regression works. Under normal circumstances, nature is very cohesive and doesn't jump around much.

```
When the intelligent and animal souls are held together in one embrace, they can be kept from separating.
```
\- Tao Te Ching, Lao Tzu.

Which means that **when we do see jumps and spikes, they are probably artificial** and often man made situations. These events are usually accompanied by counterfactuals to the normal way of things: if a weird thing happens, this gives us some insight into what would have happened if nature was to work in a different way. Exploring these artificial jumps is at the core of Regression Discontinuity Design.

The basic setup goes like this. Imagine that you have a treatment variable \\(T\\) and potential outcomes \\(Y_0\\) and \\(Y_1\\). The treatment T is a discontinuous function of an observed running variable \\(R\\) such that

$
D_i = \mathcal{1}\{R_i>c\}
$

In other words, this is saying that treatment is zero when \\(R\\) is below a threshold \\(c\\) and one otherwise. This means that we get to observe \\(Y_1\\) when \\(R>c\\) and \\(Y_0\\) when \\(R<c\\). To wrap our head around this, think about the potential outcomes as 2 functions that we can't observe entirely. Both \\(Y_0(R)\\) and \\(Y_1(R)\\) are there, we just can't see that. The threshold acts as a switch that allows us to see one or the other of those function, but never both, much like in the image below:

The idea of regression discontinuity is to compare the outcome just above and just below the threshold to identify the treatment effect at the threshold. This is called a **sharp RD** design, since the probability of getting the treatment jumps from 0 to 1 at the threshold, but we could also think about a **fuzzy RD** design, where the probability also jumps, but is a less dramatic manner.

## Is Alcohol Killing You?

A very relevant public policy question is what should be the minimal drinking age. Most countries, Brazil included, set it to be 18 year, but in the US (most states) it is currently 21. So, is it the case that the US is being overly prudent and that they should lower their minimal drinking age? Or is it the case that other countries should make their legal drinking age higher?

One way to look at this question is from a [mortality rate perspective (Carpenter and Dobkin, 2009)](https://www.aeaweb.org/articles?id=10.1257/app.1.1.164). From the public policy standpoint, one could argue that we should lower the mortality rate as much as possible. If alcohol consumption increases the mortality rate by a lot, we should avoid lowering the minimum drinking age. This would be consistent with the objective of lowering deaths caused by alcohol consumption.

To estimate the impacts of alcohol, we could use the fact that legal drinking age imposes a discontinuity on nature. In the US, those just under 21 years don't drink (or drink much less) while those just older than 21 do drink. This means that the probability of drinking jumps at 21 years and that is something we can explore with an RDD.

```python
import warnings
warnings.filterwarnings('ignore')

import pandas as pd
import numpy as np
from matplotlib import style
from matplotlib import pyplot as plt
import seaborn as sns
import statsmodels.formula.api as smf

%matplotlib inline

style.use("fivethirtyeight")
```

To do so we can grab some mortality data aggregated data by age. Each row is a has the average age of a group of people and the average mortality by all causes (`all`), by moving veiche accident (`mva`) and by suicide (`suicide`).

```python
drinking = pd.read_csv("./data/drinking.csv")
drinking.head()[["agecell", "all", "mva", "suicide"]]
```

      agecell
      all
      mva
      suicide

      0
      19.068493
      92.825400
      35.829327
      11.203714

      1
      19.150684
      95.100740
      35.639256
      12.193368

      2
      19.232876
      92.144295
      34.205650
      11.715812

      3
      19.315070
      88.427760
      32.278957
      11.275010

      4
      19.397260
      88.704940
      32.650967
      10.984314

Just to aid visibility, and for another important reason we will see later, we will centralize the running variable `agecell` at the threshold 21.

```python
drinking["agecell"] -= 21
```

If we plot the multiple outcome variables (`all`, `mva`, `suicide`) with the runing variable on the x axis, we get some visual cue that there is also a jump in mortality as we cross the legal drinking age.

```python
plt.figure(figsize=(8,8))
ax = plt.subplot(3,1,1)
drinking.plot.scatter(x="agecell", y="all", ax=ax)
plt.title("Death Cause by Age (Centered at 0)")

ax = plt.subplot(3,1,2, sharex=ax)
drinking.plot.scatter(x="agecell", y="mva", ax=ax)

ax = plt.subplot(3,1,3, sharex=ax)
drinking.plot.scatter(x="agecell", y="suicide", ax=ax);

```

There are some cues, but we need more than that. What exactly is the effect of drinking on mortality at the threshold? And what is the standard error on that estimate?

## RDD Estimation

The key assumption that RDD relies on is the smoothness of the potential outcome at the threshold. Formally, the limits of the potential outcomes as the running variable approaches the threshold from the right and from the left should be the same.

$$
\lim_{r \to c^-} E[Y_{ti}|R_i=r] = \lim_{r \to c^+} E[Y_{ti}|R_i=r]
$$

If this holds true, we can find the causal effect at the threshold

\begin{align}
\lim_{r \to c^+} E[Y_{ti}|R_i=r] - \lim_{r \to c^-} E[Y_{ti}|R_i=r]=&\lim_{r \to c^+} E[Y_{1i}|R_i=r] - \lim_{r \to c^-} E[Y_{0i}|R_i=r] \\
=& E[Y_{1i}|R_i=r] - E[Y_{0i}|R_i=r] \\
=& E[Y_{1i} - Y_{0i}|R_i=r]
\end{align}

This is, in its own way, a sort of Local Average Treatment Effect (LATE), since we can only know it at the threshold. In this setting, we can think of RDD as a local randomized trial. For those at the threshold, the treatment could have gone either way and, by chance, some people fell below the threshold, and some people fell above. In our example, at the same point in time, some people are just above 21 years and some people are just below 21. What determines this is if someone was born some days later or not, which is pretty random. For this reason, RDD provides a very compelling causal story. It is not the golden standard of RCT, but it is close.

Now, to estimate the treatment effect at the threshold, all we need to do is estimate both of the limits in the formula above and compare them. The simplest way to do that is running a linear regression

To make it work, we interact a dummy for being above the threshold with the running variable

$
y_i = \beta_0 + \beta_1 r_i + \beta_2 \mathcal{1}\{r_i>c\} + \beta_3 \mathcal{1}\{r_i>c\} r_i
$

What this is doing is fitting a linear regression above the threshold and another below it. The parameter \\(\beta_0\\) is the intercept of the regression below the threshold and \\(\beta_0+\beta_2\\) is the intercept for the regression above the threshold.

Here is where the trick of centering the running variable at the threshold comes in play. After this pre-processing step, the threshold becomes zero. This causes the intercept \\(\beta_0\\) to be the predicted value at the threshold, for the regression below it. In other words, \\(\beta_0=\lim_{r \to c^-} E[Y_{ti}|R_i=r]\\). By the same reasoning, \\(\beta_0+\beta_2\\) is the limit of the outcome from above. Wich means, that

$
\lim_{r \to c^+} E[Y_{ti}|R_i=r] - \lim_{r \to c^-} E[Y_{ti}|R_i=r]=\beta_2=E[ATE|R=c]
$

Here is what this looks like in code for the case where we want to estimate the effect of alcohol consumption on death by all causes at 21 years.

```python
rdd_df = drinking.assign(threshold=(drinking["agecell"] > 0).astype(int))

model = smf.wls("all~agecell*threshold", rdd_df).fit()

model.summary().tables[1]
```

          	             coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	            93.6184	     0.932	   100.399	  0.000	    91.739	    95.498

  agecell	               0.8270	     0.819	     1.010	  0.318	    -0.823	     2.477

  threshold	             7.6627	     1.319	     5.811	  0.000	     5.005	    10.320

  agecell:threshold	    -3.6034	     1.158	    -3.111	  0.003	    -5.937	    -1.269

This model is telling us that mortality increases by 7.6627 points with the consumption of alcohol. Another way of putting this is that alcohol increases the chance of death by all causes by 8% ((7.6627+93.6184)/93.6184). Notice that this also gives us standard errors for our causal effect estimate. In this case, the effect is statistically significant, since the p-value is below 0.01.

If we want to verify this model visually, we can show the predicted values on the data that we have. You can see that it is as though we have 2 regression models: one for those above the threshold and one for below it.

```python
ax = drinking.plot.scatter(x="agecell", y="all", color="C0")
drinking.assign(predictions=model.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
plt.title("Regression Discontinuity");
```

If we do the same for the other causes, this is what we get.

```python
plt.figure(figsize=(8,8))

for p, cause in enumerate(["all", "mva", "suicide"], 1):
    ax = plt.subplot(3,1,p)
    drinking.plot.scatter(x="agecell", y=cause, ax=ax)
    m = smf.wls(f"{cause}~agecell*threshold", rdd_df).fit()
    ate_pct = 100*((m.params["threshold"] + m.params["Intercept"])/m.params["Intercept"] - 1)
    drinking.assign(predictions=m.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
    plt.title(f"Impact of Alcohol on Death: {np.round(ate_pct, 2)}%")

plt.tight_layout()
```

RDD is telling us that alcohol increases the chance of deth by suicide and car accidents by 15%, wich is a pretty significant ammount. These results are compelling arguments to not lower the drinking age, if we want to minimize mortality rates.

### Kernel Weighting

Regression Discontinuity relies heavily on the extrapolations properties of linear regression. Since we are looking at the values at the beginning and end of 2 regression lines, we better get those limits right. What can happen is that regression might focus too much on fitting the other data points at the cost of a poor fit at the threshold. If this happens, we might get the wrong measure of the treatment effect.

One way to solve this is to give higher weights for the points that are closer to the threshold. There are many ways to do this, but a popular one is to reweight the samples through the **triangular kernel**

$
K(R, c, h) = \mathcal{1}\{|R-c| \leq h\} * \bigg(1-\frac{|R-c|}{h}\bigg)
$

The first part of this kernel is an indicator function to whether we are close to the threshold. How close? This is determined by a bandwidth parameter \\(h\\). The second part of this kernel is a weighting function. As we move away from the threshold, the weights get smaller and smaller. These weights are divided by the bandwidth. If the bandwidth is large, the weights get smaller at a slow rate. If the bandwidth is small, the weights quickly go to zero.

To make it easier to understand, here is what the weights look like for this kernel applied to our problem. I've set the bandwidth to be 1 here, meaning we will only consider data from people that are no older than 22 years and no younger than 20 years.

```python
def kernel(R, c, h):
    indicator = (np.abs(R-c) <= h).astype(float)
    return indicator * (1 - np.abs(R-c)/h)
```

```python
plt.plot(drinking["agecell"], kernel(drinking["agecell"], c=0, h=1))
plt.xlabel("agecell")
plt.ylabel("Weight")
plt.title("Kernel Weight by Age");
```

If we apply these weights to our original problem, the impact of alcohol gets bigger, at least for all causes. It jumps from 7.6627 to 9.7004. The result remains very significant. Also, notice that I'm using `wls` instead of `ols`

```python
model = smf.wls("all~agecell*threshold", rdd_df,
                weights=kernel(drinking["agecell"], c=0, h=1)).fit()

model.summary().tables[1]
```

          	             coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	            93.2002	     0.731	   127.429	  0.000	    91.726	    94.674

  agecell	               0.4109	     1.789	     0.230	  0.819	    -3.196	     4.017

  threshold	             9.7004	     1.034	     9.378	  0.000	     7.616	    11.785

  agecell:threshold	    -7.1759	     2.531	    -2.835	  0.007	   -12.276	    -2.075

```python
ax = drinking.plot.scatter(x="agecell", y="all", color="C0")
drinking.assign(predictions=model.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
plt.title("Regression Discontinuity (Local Regression)");
```

And here is what it looks like for the other causes of death. Notice how the regression on the right is more negatively sloped since it disconsiders the right most points.

```python
plt.figure(figsize=(8,8))
weights = kernel(drinking["agecell"], c=0, h=1)

for p, cause in enumerate(["all", "mva", "suicide"], 1):
    ax = plt.subplot(3,1,p)
    drinking.plot.scatter(x="agecell", y=cause, ax=ax)
    m = smf.wls(f"{cause}~agecell*threshold", rdd_df, weights=weights).fit()
    ate_pct = 100*((m.params["threshold"] + m.params["Intercept"])/m.params["Intercept"] - 1)
    drinking.assign(predictions=m.fittedvalues).plot(x="agecell", y="predictions", ax=ax, color="C1")
    plt.title(f"Impact of Alcohol on Death: {np.round(ate_pct, 2)}%")

plt.tight_layout()
```

With the exception of suicide, it looks like adding the kernel weight made the negative impact on alcohol bigger. Once again, if we want to minimize the death rate, we should NOT recommend lowering the legal drinking age, since there clear impact of alcohol on the death rate.

This simple case covers what happens when regression discontinuity design works perfectly. Next, we will see some diagnostics that we should run in order to check how much we can trust RDD and talk about a topic that is very dear to our heart: the effect of education on earnings.

## Sheepskin Effect and Fuzzy RDD

When it comes to the effect of education on earnings, there are two major views in economics. The first one is the widely known argument that education increases human capital, increasing productivity and thus, earnings. In this view, education actually changes you for the better. Another view is that education is simply a signaling mechanism. It just puts you through all these hard tests and academic tasks and, if you can make it, it signals to the market that you are a good employee. In this view, education doesn't make you more productive. It only tells the market how productive you have always been. What matters here is the diploma. If you have it, you will be paid more. We refer to this as the **sheepskin effect**, since diplomas were printed in sheepskin in the past.

To test this hypothesis, [Clark and Martorell](https://faculty.smu.edu/millimet/classes/eco7321/papers/clark%20martorell%202014.pdf) used regression discontinuity to measure the effect of graduating 12th grade on earnings. In order to do that, they had to think about some running variable where students that fall above it graduate and those who fall below it, don't. They found such data in the Texas education system.

In order to graduate in Texas, one has to pass an exam. Testing starts at 10th grade and students can do it multiple times, but eventually, they face a last chance exam at the end of 12th grade. The idea was to get data from students who took those last chance exams and compare those that had barely failed it to those that barely passed it. These students will have very similar human capital, but different signaling credentials. Namely, those that barely passed it, will receive a diploma.

```python
sheepskin = pd.read_csv("./data/sheepskin.csv")[["avgearnings", "minscore", "receivehsd", "n"]]
sheepskin.head()
```

      avgearnings
      minscore
      receivehsd
      n

      0
      11845.086
      -30.0
      0.416667
      12

      1
      9205.679
      -29.0
      0.387097
      31

      2
      8407.745
      -28.0
      0.318182
      44

      3
      11114.087
      -27.0
      0.377778
      45

      4
      10814.624
      -26.0
      0.306667
      75

One again, this data is grouped by the running variable. It contains not only the running variable (`minscore`, already centered at zero) and the outcome (`avgearnings`), but it also has the probability of receiving a diploma in that score cell and the size of the call (`n`). So, for example, out of the 12 students in the cell -30 below the score threshold, only 5 were able to get the diploma.

This means that there is some slippage in the treatment assignment. Some students that are below the passing threshold managed to get the diploma anyway. Here, the regression discontinuity is **fuzzy**, rather than sharp. Notice how the probability of getting the diploma doesn't jump from zero to one at the threshold. But it does jump from something like 50% to 90%.

```python
sheepskin.plot.scatter(x="minscore", y="receivehsd", figsize=(10,5))
plt.xlabel("Test Scores Relative to Cutoff")
plt.ylabel("Fraction Receiving Diplomas")
plt.title("Last-chance Exams");
```

We can think of fuzzy RD as a sort of non compliance. Passing the threshold should make everyone receive the diploma, but some students, the never takers, don't get it. Likewise, being below the threshold should prevent you from getting a diploma, but some students, the allways takers, manage to get it anyway.

Just like we have the potential outcome, we have the potential treatment status in this situation. \\(T_1\\) is the treatment everyone would have received had they been above the threshold. \\(T_0\\) is the treatment everyone would have received had they been below the threshold. As you've might have noticed, we can think of the **threshold as an Instrumental Variable**. Just as in IV, this non compliance makes the treatment effect we can naively estimate biased to zero.

The probability of treatment being less than one, even above the threshold, makes the outcome we observe less than the true potential outcome \\(Y_1\\). By the same token, the outcome we observe below the threshold is higher than the true potential outcome \\(Y_0\\). This makes it look like the treatment effect at the threshold is smaller than it actually is and we will have to use IV techniques to correct for that.

Just like we've assumed smoothness on the potential outcome, we now assume it for the potential treatment. Also, we need to assume monotonicity, just like in IV. In case you don't remember, it states that \\(T_{i1}>T_{i0} \ \forall i\\). This means that crossing the threshold from the left to the right only increases your chance of getting a diploma (or that there are no defiers). With these 2 assumptions, we have a Wald Estimator for LATE.

$$
\dfrac{\lim_{r \to c^+} E[Y_i|R_i=r] - \lim_{r \to c^-} E[Y_i|R_i=r]}{\lim_{r \to c^+} E[T_i|R_i=r] - \lim_{r \to c^-} E[T_i|R_i=r]} = E[Y_{1i} - Y_{0i} | T_{1i} > T_{0i}, R_i=c]
$$

Notice how this is a local estimate in two senses. First, it is local because it only gives the treatment effect at the threshold \\(c\\). This is the RD locality. Second, it is local because it only estimates the treatment effect for the compliers. This is the IV locality.

To estimate this, we will use 2 linear regression. The numerator can be estimated just like we've done before. To get the denominator, we simply replace the outcome with the treatment. But first, let's talk about a sanity check we need to run to make sure we can trust our RDD estimates.

### The McCrary Test

One thing that could break our RDD argument is if people manipulate where they stand at the threshold. In the sheepskin example this could happen if students just below the threshold found a way around the system to increase their test score by just a bit. Another example is when you need to be below a certain income level to get a government benefit. Some families might lower their income on purpose, just to be just eligible for the program.

In these sorts of situations, we tend to see a phenomenon called bunching on the density of the running variable. This means that we will have a lot of entities just above or just below the threshold. To check this, we can plot the density function of the running variable and see if there are any spikes around the threshold. For our case, the density is given by the `n` column in our data.

```python
plt.figure(figsize=(8,8))

ax = plt.subplot(2,1,1)
sheepskin.plot.bar(x="minscore", y="n", ax=ax)
plt.title("McCrary Test")
plt.ylabel("Smoothness at the Threshold")

ax = plt.subplot(2,1,2, sharex=ax)
sheepskin.replace({1877:1977, 1874:2277}).plot.bar(x="minscore", y="n", ax=ax)
plt.xlabel("Test Scores Relative to Cutoff")
plt.ylabel("Spike at the Threshold");
```

The first plot shows how our data density looks like. As we can see, there are no spikes around the threshold, meaning there is no bunching. Students are not manipulating where they fall on the threshold. Just for illustrative purposes, the second plot shows what bunching would look like if students could manipulate where they fall on the threshold. We would see a spike in the density for the cells just above the threshold, since many students would be there. Getting this out of the way, we can go back to estimate the sheepskin effect.

As I've said before, the numerator of the Wald estimator can be estimated just like we did in the Sharp RD. Here, we will use as weight the kernel with a bandwidth of 15. Since we also have the cell size, we will multiply the kernel by the sample size to get a final weight for the cell.

```python
sheepsking_rdd = sheepskin.assign(threshold=(sheepskin["minscore"]>0).astype(int))
model = smf.wls("avgearnings~minscore*threshold",
                sheepsking_rdd,
                weights=kernel(sheepsking_rdd["minscore"], c=0, h=15)*sheepsking_rdd["n"]).fit()

model.summary().tables[1]
```

           	             coef	     std err	      t	      P>|t|	  [0.025	    0.975]

  Intercept	           1.399e+04	    83.678	   167.181	  0.000	  1.38e+04	  1.42e+04

  minscore	             181.6636	    16.389	    11.084	  0.000	   148.588	   214.739

  threshold	            -97.7571	   145.723	    -0.671	  0.506	  -391.839	   196.325

  minscore:threshold	    18.1955	    30.311	     0.600	  0.552	   -42.975	    79.366

This is telling us that the effect of a diploma is -97.7571, but this is not statistically significant (P-value of 0.5). If we plot these results, we get a very continuous line at the threshold. More educated people indeed make more money, but there isn't a jump at the point where they receive the 12th grade diploma. This is an argument in favor of the view that says that education increases earnings by making people more productive, since there is no sheepskin effect.

```python
ax = sheepskin.plot.scatter(x="minscore", y="avgearnings", color="C0")
sheepskin.assign(predictions=model.fittedvalues).plot(x="minscore", y="predictions", ax=ax, color="C1", figsize=(8,5))
plt.xlabel("Test Scores Relative to Cutoff")
plt.ylabel("Average Earnings")
plt.title("Last-chance Exams");
```

However, as we know from the way non compliance bias works, this result is biased towards zero. To correct for this, we need to scale it by the first stage and get the Wald estimator. Unfortunately, there isn't a good Python implementation for this, so we will have to do it manually and use bootstrap to get the standard errors.

The code below rund the numerator of the Wald estimator just like we did before and it also constructs the denominator by replacing the target variable with the treatment variable `receivehsd`. The final step just divides the numerator by the denominator.

```python
def wald_rdd(data):
    weights=kernel(data["minscore"], c=0, h=15)*data["n"]
    denominator = smf.wls("receivehsd~minscore*threshold", data, weights=weights).fit()
    numerator = smf.wls("avgearnings~minscore*threshold", data, weights=weights).fit()
    return numerator.params["threshold"]/denominator.params["threshold"]
```

```python
from joblib import Parallel, delayed

np.random.seed(45)
bootstrap_sample = 1000
ates = Parallel(n_jobs=4)(delayed(wald_rdd)(sheepsking_rdd.sample(frac=1, replace=True))
                          for _ in range(bootstrap_sample))
ates = np.array(ates)
```

With the bootstrap samples, we can plot the distribution of ATEs and see where the 95% confidence interval is.

```python
sns.distplot(ates, kde=False)
plt.vlines(ates.mean()-1.96*ates.std(), 0, 100, linestyles="dotted")
plt.vlines(ates.mean()+1.96*ates.std(), 0, 100, linestyles="dotted", label="95% CI")
plt.title("ATE Bootstrap Distribution")
plt.xlim([-10000, 10000])
plt.legend();
```

As you can see, even when we scale the effect by the first stage, it is still not statistically different from zero. This means that education doesn't increase earnings by a simple sheepskin effect, but rather by increasing one's productivity.

## Key Ideas

We learned how to take advantage of artificial discontinuities to estimate the causal effect. The idea is that we will have some artificial threshold that makes the probability of treatment jump. One example that we saw was how age makes the probability of drinking jump at 21 how we can use this to estimate the impact of drinking. We use the fact that very close to the threshold, we have something close to a randomized trial. Entities very close to the threshold could have gone either way and what determines where they've landed is essentially random. With this, we can compare those just above and just below to get the treatment effect. We saw how we could do that with weighted linear regression using a kernel and how this even gave us, for free, standard errors for our ATE.

Then, we look at what would happen in the fuzzy RD design, where we have non compliance. We saw how we could approach the situation much like we did with IV.

## References

I like to think of this entire series as a tribute to Joshua Angrist, Alberto Abadie and Christopher Walters for their amazing Econometrics class. Most of the ideas here are taken from their classes at the American Economic Association. Watching them is what is keeping me sane during this tough year of 2020.
* [Cross-Section Econometrics](https://www.aeaweb.org/conference/cont-ed/2017-webcasts)
* [Mastering Mostly Harmless Econometrics](https://www.aeaweb.org/conference/cont-ed/2020-webcasts)

I'll also like to reference the amazing books from Angrist. They have shown me that Econometrics, or 'Metrics as they call it, is not only extremely useful but also profoundly fun.

* [Mostly Harmless Econometrics](https://www.mostlyharmlesseconometrics.com/)
* [Mastering 'Metrics](https://www.masteringmetrics.com/)

Other important reference is Miguel Hernan and Jamie Robins' book. It has been my trustworthy companion in the most thorny causal questions I had to answer.

* [Causal Inference Book](https://www.hsph.harvard.edu/miguel-hernan/causal-inference-book/)

```python

```


# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Cascaded Structures

The realization of recursive filters with a high order may be subject to numerical issues. For instance, when the coefficients span a wide amplitude range, their quantization may require a small quantization step or may impose a large relative error for small coefficients. The basic concept of cascaded structures is to decompose a high order filter into a cascade of lower order filters, typically first and second order recursive filters.

### Decomposition into Second-Order Sections

The rational transfer function $H(z)$ of a linear time-invariant (LTI) recursive system can be [expressed by its zeros and poles](introduction.ipynb#Transfer-Function) as

\begin{equation}
H(z) = \frac{b_M}{a_N} \cdot \frac{\prod_{\mu=1}^{P} (z - z_{0\mu})^{m_\mu}}{\prod_{\nu=1}^{Q} (z - z_{\infty\nu})^{n_\nu}}
\end{equation}

where $z_{0\mu}$ and $z_{\infty\nu}$ denote the $\mu$-th zero and $\nu$-th pole of degree $m_\mu$ and $n_\nu$ of $H(z)$, respectively. The total number of zeros and poles is denoted by $P$ and $Q$.

The poles and zeros of a real-valued filter $h[k] \in \mathbb{R}$ are either single real valued or conjugate complex pairs. This motivates to split the transfer function into

* first order filters constructed from a single pole and zero
* second order filters constructed from a pair of conjugated complex poles and zeros

Decomposing the transfer function into these two types by grouping the poles and zeros into single poles/zeros and conjugate complex pairs of poles/zeros results in

\begin{equation}
H(z) = K \cdot \prod_{\eta=1}^{S_1} \frac{(z - z_{0\eta})}{(z - z_{\infty\eta})}
\cdot \prod_{\eta=1}^{S_2} \frac{(z - z_{0\eta}) (z - z_{0\eta}^*)} {(z - z_{\infty\eta})(z - z_{\infty\eta}^*)}
\end{equation}

where $K$ denotes a constant and $S_1 + 2 S_2 = N$ with $N$ denoting the order of the system. The cascade of two systems results in a multiplication of their transfer functions. Above decomposition represents a cascade of first- and second-order recursive systems. The former can be treated as a special case of second-order recursive systems. The decomposition is therefore known as decomposition into second-order sections (SOSs) or [biquad filters](https://en.wikipedia.org/wiki/Digital_biquad_filter). Using a cascade of SOSs the transfer function of the recursive system can be rewritten as

\begin{equation}
H(z) = \prod_{\mu=1}^{S} \frac{b_{0, \mu} + b_{1, \mu} \, z^{-1} + b_{2, \mu} \, z^{-2}}{1 + a_{1, \mu} \, z^{-1} + a_{2, \mu} \, z^{-2}}
\end{equation}

where $S = \lceil \frac{N}{2} \rceil$ denotes the total number of SOSs. These results state that any real valued system of order $N > 2$ can be decomposed into SOSs. This has a number of benefits

* quantization effects can be reduced by sensible grouping of poles/zeros, e.g. such that the spanned amplitude range of the filter coefficients is limited
* A SOS may be extended by a gain factor to further reduce quantization effects by normalization of the coefficients
* efficient and numerically stable SOSs serve as generic building blocks for higher-order recursive filters

### Example

The following example illustrates the decomposition of a higher-order recursive Butterworth lowpass filter into a cascade of second-order sections.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.markers import MarkerStyle
from matplotlib.patches import Circle
import scipy.signal as sig

N = 9  # order of recursive filter

def zplane(z, p):

    ax = plt.gca()
    plt.hold(True)

    unit_circle = Circle((0,0), radius=1, fill=False,
                         color='black', ls='solid', alpha=0.9)
    ax.add_patch(unit_circle)
    ax.axvline(0, color='0.7')
    ax.axhline(0, color='0.7')
    plt.axis('equal')
    plt.xlim((-2, 2))
    plt.ylim((-2, 2))
    plt.grid()
    plt.xlabel(r'Re{$z$}')
    plt.ylabel(r'Im{$z$}')

    ax.plot(np.real(z), np.imag(z), 'bo', fillstyle='none', ms = 10)
    ax.plot(np.real(p), np.imag(p), 'rx', fillstyle='none', ms = 10)

    plt.hold(False)

# design filter
b, a = sig.butter(N, 0.2)
# decomposition into SOS
sos = sig.tf2sos(b, a, pairing='nearest')

# print filter coefficients
print('Coefficients of the recursive part')
print(['%1.2f'%ai for ai in a])
print('\n')
print('Coefficients of the recursive part of the SOSs')
print('Section \t a1 \t\t a2')
for n in range(sos.shape[0]):
    print('%d \t\t %1.5f \t %1.5f'%(n, sos[n, 4], sos[n, 5]))

# plot poles and zeros
plt.figure(figsize=(5,5))
zplane(np.roots(b), np.roots(a))
plt.title('Overall')

for n in range(sos.shape[0]):
    if not n%3:
        plt.figure(figsize=(10, 3.5))
    plt.subplot(131+n%3)
    zplane(np.roots(sos[n, 0:3]), np.roots(sos[n, 3:6]))
    plt.title('Section %d'%n)
    plt.tight_layout()

# compute and plot frequency response of sections
plt.figure(figsize=(10,5))
for n in range(sos.shape[0]):
    Om, H = sig.freqz(sos[n, 0:3], sos[n, 3:6])
    plt.plot(Om, 20*np.log10(np.abs(H)), label=r'Section $n=%d$'%n)
    plt.hold(True)
plt.hold(False)
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H_n(e^{j \Omega})|$ in dB')
plt.legend()
plt.grid();
```

**Exercise**

* What amplitude range is spanned by the filter coefficients?
* What amplitude range is spanned by the SOS coefficients?
* Change the pole/zero grouping strategy from `pairing='nearest'` to `pairing='keep_odd'`. What changes?
* Increase the order `N` of the filter. What changes?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.