# Sequence Density Estimation using Field Theory (SeqDEFT)

This notebook demonstrates how to do the following tasks in SeqDEFT:

(1) Trace the MAP curve, and find the optimal hyperparameter;

(2) Visualize the optimal density;

(3) Compute pairwise associations of the optimal density;

(4) Perform posterior sampling using Hamiltonian Monte Carlo.

This demonstration employs a small example dataset simulated from the prior. Applications to sequence spaces with several hundred thousands of genotypes can be done on a typical laptop computer within days. For larger sequence spaces up to low millions of genotypes, it is strongly recommended to deploy the calculations on a cluster.

### Example

We consider a biological sequence space formed by short (length = 5) DNA sequences. The dimension of the space is equal to 1,024. These sequences are ordered in the lexicographical order: `AAAAA`, `AAAAC`, `AAAAG`, `AAAAT`, ..., `TTTTT`. The number of occurrence of each sequence will be simulated from the prior.

We use the 2nd order association (i.e. log odds ratio) as the "smoothness" measure of the density. The corresponding MaxEnt solution is then the independent sites model.

```python
#
# Import packages that will be needed later
#

import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import itertools

from matplotlib import gridspec
from matplotlib.collections import LineCollection
```

```python
#
# Import functions of SeqDEFT
#

from functions import preliminary_preparation, simulate_data_from_prior, \
                      trace_MAP_curve, compute_log_Es, compute_log_Es_bounds, compute_log_Ls, \
                      make_visualization, get_nodes, get_edges, \
                      compute_log_ORs, \
                      posterior_sampling, compute_R_hat, combine_samples, plot_distribution
```

## Preliminary preparation

Specify the following parameters:

`alpha` = number of alleles

`l` = sequence length

`P` = order of association to compute

This will set global parameters, and create/load graph Laplacian and its kernel basis.

```python
alpha = 4
l = 5
P = 2

# ----------

preliminary_preparation(alpha, l, P)
```

    Setting global parameters ...
    Loading L ...
    Loading D kernel basis ...
    Constructing D spectrum ...

## Simulate data from the prior

Specify the following parameters:

`N` = number of data

`a_true` = assumed value of the hyperparameter

This will create a dictionary `data_dict` that contains the number `N` and the frequency `R` of the data, as well as the density `Q_true` generating the data. The product of `N` and `R` gives the number of occurrence of the sequences.

```python
N = 10000
a_true = 100

# ----------

data_dict = simulate_data_from_prior(N, a_true, random_seed=0)
data_dict
```

    {'N': 10000,
     'Q_true': array([3.02898538e-06, 1.00150515e-04, 4.02625966e-05, ...,
            4.64508913e-05, 1.51595234e-04, 1.14668340e-03]),
     'R': array([0.    , 0.0002, 0.    , ..., 0.0001, 0.0001, 0.0012])}

## (1) Trace the MAP curve, and find the optimal hyperparameter

### (1a) Trace the MAP curve

Specify the following parameters:

`resolution` = target geodesic distance between two densities

`num_a` = number of values of the hyperparameter to compute along the MAP curve

This will trace the MAP curve. If `num_a` is specified, `num_a` values of the hyperparameter will be computed, in addition to zero and infinity. Otherwise, the function will find as many values of the hyperparameter as it needs such that any pair of adjacent densities has a geodesic distance less than `resolution`. In either case, `resolution` is also used to find a minimum and a maximum finite value of the hyperparameter. The resulting MAP curve will consist of hyperparameter values between these two extremes. The result will be saved in the dataframe `df_map`.

**Note 1.** The computation will take about 5 seconds.

**Note 2.** In some cases, when the hyperparameter is very large or very small, the minimization algorithm may return an error message, such as `ABNORMAL_TERMINATION_IN_LNSRCH` or `STOP: TOTAL NO. of f AND g EVALUATIONS EXCEEDS LIMIT`. There are (at least) two ways to deal with this situation. (1) We can simply discard these extreme values of the hyperparameter, as long as this does not affect the determination of the optimal hyperparameter. (2) We can make the numerical convergence criterion less strict by increasing `gtol` in `options`, as long as the (premature) solution is good enough. The MAP estimate of the field, denoted $\phi^a$, must satisfy the following equation:
\begin{equation}
\sum_{i=1}^G e^{-\phi^a_i} = 1
\end{equation}
where $G$ is the dimension of the space. This equation holds for any value of the hyperparameter.

```python
resolution = 0.1
num_a = 20

# ----------

df_map = trace_MAP_curve(data_dict, resolution, num_a, options={'ftol':1e-100, 'gtol':1e-3})
```

    Computing a = inf ...
    Computing a_max = 23040.000000 ...
    ... D_geo(Q_max, Q_inf) = 0.693012
    Computing a_max = 230400.000000 ...
    ... D_geo(Q_max, Q_inf) = 0.106456
    Computing a_max = 2304000.000000 ...
    ... D_geo(Q_max, Q_inf) = 0.010510

    Computing a = 0 ...
    Computing a_min = 0.023040 ...
    ... D_geo(Q_min, Q_0) = 0.007539

    Partitioning the MAP curve into 20 points ...
    Computing a_18 = 873835.819945 ...
    Computing a_17 = 331418.854261 ...
    Computing a_16 = 125696.903758 ...
    Computing a_15 = 47672.941389 ...
    Computing a_14 = 18080.869717 ...
    Computing a_13 = 6857.513722 ...
    Computing a_12 = 2600.842502 ...
    Computing a_11 = 986.418985 ...
    Computing a_10 = 374.118161 ...
    Computing a_9 = 141.891428 ...
    Computing a_8 = 53.815023 ...
    Computing a_7 = 20.410371 ...
    Computing a_6 = 7.741021 ...
    Computing a_5 = 2.935930 ...
    Computing a_4 = 1.113507 ...
    Computing a_3 = 0.422319 ...
    Computing a_2 = 0.160172 ...
    Computing a_1 = 0.060748 ...

### (1b) Compute Bayesian evidence ratio and its bounds

This will compute Bayesian evidence ratio (which is suitable for small problems only) and its lower and upper bounds. The result will be saved in the dataframe `df_map`.

**Note.** The computation of Bayesian evidence ratio will take about 1 minute, and the computation of the bounds will take less than 1 second.

```python
df_map = compute_log_Es(data_dict, df_map)
df_map = compute_log_Es_bounds(data_dict, df_map)
```

```python
#
# Make plot
#

aa = df_map['a'].values
log_Es = df_map['log_E'].values
log_Es_lb, log_Es_ub = df_map['log_E_lb'].values, df_map['log_E_ub'].values

a_star = aa[log_Es.argmax()]
max_log_E = log_Es.max()

# ----------

plt.figure(figsize=(6,5))

with np.errstate(divide='ignore'):
    plt.fill_between(np.log10(aa), log_Es_lb, log_Es_ub, color='orange', alpha=0.5, zorder=1)
    plt.hlines(0, xmin=-4, xmax=9, color='grey', linestyles='--', zorder=2)
    plt.scatter(np.log10(aa), log_Es, color='blue', s=15, zorder=3)
    plt.scatter(9, log_Es[-1], color='blue', s=15, zorder=3)
    plt.scatter(np.log10(a_star), max_log_E, color='red', s=15, zorder=4)

plt.annotate(s=('a* = %.1f'%a_star), xy=(5, 14000), fontsize=14)

plt.xlim(-4, 9)
plt.ylim(-1000, 16000)
plt.xticks([-4, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 9], [r'$-\infty$','-2','-1','0','1','2','3','4','5','6','7',r'$\infty$'])
plt.xlabel('log10 (a)', fontsize=14)
plt.ylabel('log (E)', fontsize=14)
plt.show()
```

### (1c) Compute cross-validated likelihood

Specify the following parameters:

`cv_fold` = number of partitions of the data

This will compute data likelihood by doing `cv_fold`-fold cross validation. The result will be saved in the dataframe `df_map`.

**Note 1.** The computation will take about 12 seconds.

**Note 2.** See Note 2 in (1a) for details about the numerical solutions and related issues.

```python
cv_fold = 5

# ----------

df_map = compute_log_Ls(data_dict, df_map, cv_fold, random_seed=0, options={'ftol':1e-100, 'gtol':1e-3})
```

    Doing cross validation fold # 0 ...
    Doing cross validation fold # 1 ...
    Doing cross validation fold # 2 ...
    Doing cross validation fold # 3 ...
    Doing cross validation fold # 4 ...

```python
#
# Make plot
#

aa = df_map['a'].values
log_Ls = df_map['log_L'].values

a_star = aa[log_Ls.argmax()]
max_log_L = log_Ls.max()

# ----------

plt.figure(figsize=(6,5))

with np.errstate(divide='ignore'):
    plt.scatter(np.log10(aa), log_Ls, color='blue', s=15, zorder=1)
    plt.scatter(9, log_Ls[-1], color='blue', s=15, zorder=1)
    plt.scatter(np.log10(a_star), max_log_L, color='red', s=15, zorder=2)

plt.annotate(s=('a* = %.1f'%a_star), xy=(5, -10500), fontsize=14)

plt.xlim(-4, 9)
plt.ylim(-14000, -10000)
plt.xticks([-4, -2, -1, 0, 1, 2, 3, 4, 5, 6, 7, 9], [r'$-\infty$','-2','-1','0','1','2','3','4','5','6','7',r'$\infty$'])
plt.xlabel('log10 (a)', fontsize=14)
plt.ylabel('log (L)', fontsize=14)
plt.show()
```

### (1d) Determine the optimal hyperparameter, and compare the MAP estimate with the true distribution

We use the more general cross-validated likelihood to determine the optimal hyperparameter and the corresponding field and density. Then we compare the corresponding MAP estimate to the true distribution.

```python
aa = df_map['a'].values
phis = df_map['phi'].values
log_Ls = df_map['log_L'].values

a_star = aa[log_Ls.argmax()]
phi_star = phis[log_Ls.argmax()]
Q_star = np.exp(-phi_star) / np.sum(np.exp(-phi_star))

print('a_true = %.1f' % a_true)
print('a_star = %.1f' % a_star)
```

    a_true = 100.0
    a_star = 141.9

```python
#
# Make plot
#

Q_true = data_dict['Q_true']

plt.figure(figsize=(6,6))
plt.plot([-7,0], [-7,0], color='grey', linewidth=0.5, alpha=0.5, zorder=1)
plt.scatter(np.log10(Q_true), np.log10(Q_star), color='black', s=5, alpha=0.4, zorder=2)
plt.xlim(-7, 0)
plt.ylim(-7, 0)
plt.xlabel('Q_true', fontsize=14)
plt.ylabel('Q*', fontsize=14)
plt.show()
```

## (2) Visualize the optimal density

### (2a) Compute visualization coordinates

Specify the following parameters:

`markov_chain` = model that will be used to build transition matrix of the Markov chain

`K` = number of visualization coordinates

This will compute `K` visualization coordinates of the optimal density with the `markov_chain` model. The result will be saved in the dataframe `df_visual`. The first visualization coordinate is a constant vector, and hence, will not be used.

**Note.** The computation will take about 1 second.

```python
markov_chain = 'evolutionary'
K = 20

# ----------

df_visual, _ = make_visualization(Q_star, markov_chain, K)
```

    Constructing A and c ...
    Computing dominant eigenvalues and eigenvectors of A ...

### (2b) Plot spectrum of the transition matrix

Plot reciprocal eigenvalue spectrum of the transition matrix. The reciprocal of an eigenvalue is proportional to the fraction of variation explained in that direction.

```python
#
# Make plot
#

x = range(1, K)
y = 1 / abs(df_visual['eigenvalue'].values[1:])

plt.figure(figsize=(6,5))
plt.scatter(x, y, color='blue', s=15)
plt.xlim(0,)
plt.xticks([0, 2, 4, 6, 8, 10, 12, 14, 16, 18, 20])
plt.xlabel('k', fontsize=14)
plt.ylabel('1 / |Eigenvalue|', fontsize=14)
plt.show()
```

### (2c) Plot visualization of the optimal density

Specify the following parameters:

`kx` = the visualization coordinate to be plotted on x-axis

`ky` = the visualization coordinate to be plotted on y-axis

This will get the nodes and edges of the Hamming graph first, and then plot them in the figure. Each node is colored according to its density. The squared distance between two nodes in the figure is proportional to the commute time between these two nodes.

```python
#
# Make plot
#

kx = 1
ky = 2

# ----------

df_nodes = get_nodes(df_visual, kx, ky)
df_edges = get_edges(df_visual, kx, ky)

plt.figure(figsize=(9.5,8))

edges = df_edges['edge'].values
ln_coll = LineCollection(edges, color='grey', linewidths=0.5, alpha=0.5, zorder=1)
ax = plt.gca()
ax.add_collection(ln_coll)
plt.draw()

nodes_x, nodes_y, nodes_c = df_nodes['x'].values, df_nodes['y'].values, np.log10(Q_star)
vmin, vmax = np.floor(np.log10(Q_star).min()), np.ceil(np.log10(Q_star).max())
cm = plt.cm.get_cmap('coolwarm')
sc = plt.scatter(nodes_x, nodes_y, c=nodes_c, vmin=vmin, vmax=vmax, cmap=cm, s=10, zorder=2)
plt.colorbar(sc).set_label(label='log10 (Q*)', size=14)

plt.xlabel('Diffusion Axis %d'%kx, fontsize=14)
plt.ylabel('Diffusion Axis %d'%ky, fontsize=14)
plt.show()
```

## (3) Compute pairwise associations of the optimal density

### (3a) Specify mutations at each site

Here we simply generate mutations at each site alphabetically, and save the result in the dictionary `mutations_dict`.

```python
bases = ['A','C','G','T']

mutations_dict = {}
for i in range(l):
    mutations_dict[i] = list(itertools.combinations(bases, 2))
mutations_dict
```

    {0: [('A', 'C'), ('A', 'G'), ('A', 'T'), ('C', 'G'), ('C', 'T'), ('G', 'T')],
     1: [('A', 'C'), ('A', 'G'), ('A', 'T'), ('C', 'G'), ('C', 'T'), ('G', 'T')],
     2: [('A', 'C'), ('A', 'G'), ('A', 'T'), ('C', 'G'), ('C', 'T'), ('G', 'T')],
     3: [('A', 'C'), ('A', 'G'), ('A', 'T'), ('C', 'G'), ('C', 'T'), ('G', 'T')],
     4: [('A', 'C'), ('A', 'G'), ('A', 'T'), ('C', 'G'), ('C', 'T'), ('G', 'T')]}

### (3b) Compute log odds ratios and their rms values

For every pair of sites, compute log odds ratios for all possible mutations at the two sites.

**Note.** The computation will take about 5 seconds.

```python
num_mut = int(alpha*(alpha-1)/2)

log_ORs_dict, rms_log_ORs = {}, np.zeros([l*num_mut,l*num_mut])
for i in range(l):
    for j in range(l):
        if i == j:
            square_ij = np.zeros([num_mut,num_mut])
            for m in range(num_mut):
                for n in range(num_mut):
                    square_ij[m,n] = np.nan
            rms_log_ORs[i*num_mut:(i+1)*num_mut,j*num_mut:(j+1)*num_mut] = square_ij
        else:
            print('Computing log_ORs with given mutations at site %d & site %d ...'%(i,j))
            site_i, site_j = i, j
            site_i_muts, site_j_muts = mutations_dict[site_i], mutations_dict[site_j]
            dict_ij, k, square_ij = {}, 0, np.zeros([num_mut,num_mut])
            for m in range(num_mut):
                for n in range(num_mut):
                    site_i_mut, site_j_mut = site_i_muts[m], site_j_muts[n]
                    df_log_ORs = compute_log_ORs(phi_star, site_i, site_j, site_i_mut, site_j_mut,
                                                 coding_dict={'A':0,'C':1,'G':2,'T':3})
                    log_ORs = df_log_ORs['log_OR'].values
                    dict_ij[k] = log_ORs
                    square_ij[m,n] = np.sqrt(np.mean(log_ORs**2))
                    k += 1
            log_ORs_dict[(i,j)] = dict_ij
            rms_log_ORs[i*num_mut:(i+1)*num_mut,j*num_mut:(j+1)*num_mut] = square_ij
```

    Computing log_ORs with given mutations at site 0 & site 1 ...
    Computing log_ORs with given mutations at site 0 & site 2 ...
    Computing log_ORs with given mutations at site 0 & site 3 ...
    Computing log_ORs with given mutations at site 0 & site 4 ...
    Computing log_ORs with given mutations at site 1 & site 0 ...
    Computing log_ORs with given mutations at site 1 & site 2 ...
    Computing log_ORs with given mutations at site 1 & site 3 ...
    Computing log_ORs with given mutations at site 1 & site 4 ...
    Computing log_ORs with given mutations at site 2 & site 0 ...
    Computing log_ORs with given mutations at site 2 & site 1 ...
    Computing log_ORs with given mutations at site 2 & site 3 ...
    Computing log_ORs with given mutations at site 2 & site 4 ...
    Computing log_ORs with given mutations at site 3 & site 0 ...
    Computing log_ORs with given mutations at site 3 & site 1 ...
    Computing log_ORs with given mutations at site 3 & site 2 ...
    Computing log_ORs with given mutations at site 3 & site 4 ...
    Computing log_ORs with given mutations at site 4 & site 0 ...
    Computing log_ORs with given mutations at site 4 & site 1 ...
    Computing log_ORs with given mutations at site 4 & site 2 ...
    Computing log_ORs with given mutations at site 4 & site 3 ...

### (3c) Plot effective association magnitudes

This will plot effective association magnitudes between any two sites, defined as exponential rms log odds ratios. Need to specify the following parameters:

`vmin` = minimum of exponential rms log odds ratios

`vmax` = maximum of exponential rms log odds ratios

```python
#
# Make plot
#

vmin = 5
vmax = 30

# ----------

labels = ['1','2','3','4','5']
ticks = [0.5, 1.5, 2.5, 3.5, 4.5, 5.5]
ticklabels_dict = {}
for i in range(l):
    ticklabels_dict[i] = [mut[0]+' > '+mut[1] for mut in mutations_dict[i]]

fig = plt.figure(figsize=(8.5,8))
gs = fig.add_gridspec(nrows=1, ncols=2, width_ratios=[10,1], height_ratios=[1])
gs0 = gridspec.GridSpecFromSubplotSpec(l, l, subplot_spec=gs[0])
gs1 = gridspec.GridSpecFromSubplotSpec(1, 1, subplot_spec=gs[1])

for i in range(l):
    for j in range(l):
        ax0 = fig.add_subplot(gs0[i,j])
        sns.heatmap(np.exp(rms_log_ORs[i*num_mut:(i+1)*num_mut,j*num_mut:(j+1)*num_mut]),
                    vmin=vmin, vmax=vmax, square=True, annot=False, cmap='Blues', cbar=False)
        ax0.set_xticks([])
        ax0.set_yticks([])
        if i == (l-1):
            ax0.set_xticks(ticks=ticks)
            ax0.set_xticklabels(labels=ticklabels_dict[j], rotation='vertical')
            ax0.set_xlabel('%s'%labels[j], fontsize=14)
        if j == 0:
            ax0.set_yticks(ticks=ticks)
            ax0.set_yticklabels(labels=ticklabels_dict[i], rotation='horizontal')
            ax0.set_ylabel('%s'%labels[i], fontsize=14)

plt.subplots_adjust(wspace=0.01, hspace=0.01)

ax1 = fig.add_subplot(gs1[0,0])
sns.heatmap(np.exp(rms_log_ORs), vmin=vmin, vmax=vmax, square=False, annot=False, cmap='Blues', cbar=False)
cbar = ax1.figure.colorbar(ax1.collections[0], fraction=0.5, aspect=30)
cbar.set_ticks([5, 10, 15, 20, 25, 30])
cbar.set_ticklabels(['5', '10', '15', '20', '25', '30'])
cbar.ax.tick_params(labelsize=12)
cbar.set_label('Effective Association Magnitude', size=14)
plt.gca().set_visible(False)

plt.show()
```

### (3d) Plot histograms of log odds ratios

Specify the following parameters:

`site_i` = one site

`site_j` = another site

This will plot histograms of log odds ratios for all possible mutations at the two sites `site_i` and `site_j`.

```python
#
# Make plot
#

site_i = 1
site_j = 3

# ----------

labels = ['1','2','3','4','5']
ticklabels_dict = {}
for i in range(l):
    ticklabels_dict[i] = [r'%s$\rightarrow$%s'%(mut[0],mut[1]) for mut in mutations_dict[i]]

fig, axes = plt.subplots(num_mut, num_mut, figsize=(8,8))
k = 0
for m in range(num_mut):
    for n in range(num_mut):
        k += 1
        log_ORs = log_ORs_dict[(site_i-1,site_j-1)][k-1]
        plt.subplot(num_mut, num_mut, k)
        plt.hist(log_ORs, bins=15, density=True, color='orange')
        plt.ylim(0, 0.45)
        plt.xlim(-6, 6)
        plt.xticks([-6, -4, -2, 0, 2, 4, 6], [])
        plt.yticks([])
        plt.tick_params(axis='x', direction='in')
        if m == (num_mut-1):
            plt.xlabel('%s'%ticklabels_dict[site_j-1][n], fontsize=14)
        if n == 0:
            plt.ylabel('%s'%ticklabels_dict[site_i-1][m], fontsize=14)

plt.subplots_adjust(wspace=0, hspace=0)

fig.add_subplot(111, frameon=False)
plt.tick_params(labelcolor='none', top=False, bottom=False, left=False, right=False)
plt.xlabel('Site %s'%labels[site_j-1], fontsize=16)
plt.ylabel('Site %s'%labels[site_i-1], fontsize=16)

plt.show()
```

## (4) Perform posterior sampling using Hamiltonian Monte Carlo

### (4a) Run multiple HMC chains with different initial phi

Specify the following parameters:

`num_chains` = number of HMC chains

`num_samples_per_chain` = number of samples per HMC chain

`num_warmup` = number of warmup draws

`num_thinning` = number of thinning draws

`e` = (initial) stepsize

`L` = (initial) number of leapfrog steps

`Le` = (fixed) product of `L` and `e`

`window` = stepsize adaptation window

This will run multiple HMC chains with different initial phi. The first `num_warmup` draws will be discarded, and from then on, samples will be picked every `num_thinning` draws. The stepsize `e` will be automatically adapted according to the acceptance rate in each period made of `window` draws. (For HMC, the target acceptance rate is 0.65.) The number of leapfrog steps `L` will be adapted accordingly, such that the product `Le` remains the same. The result will be saved in the 3D numpy array `multi_phi_samples`.

**Note.** The computation will take about 14 seconds.

```python
num_chains = 10
num_samples_per_chain = 100
num_warmup = 1000
num_thinning = 10
e = 0.2
L = 5
Le = 1
window = 100

# ----------

method = 'hmc'
args = {'e': e,
        'L': L,
        'Le': Le,
        'L_max': np.inf,
        'm': 1,
        'f': 0,
        'window': window,
        'gamma_old': 1,
        'gamma_new': 1,
        'perturbation': 2,
        'num_warmup': num_warmup,
        'num_thinning': num_thinning}
seeds = range(num_chains)

multi_phi_samples = np.zeros([num_chains, alpha**l, num_samples_per_chain])
for k in range(num_chains):
    print('Running HMC chain # %d ...' % k)
    phi_initial, phi_samples, acceptance_rates = \
        posterior_sampling(phi_star, a_star, data_dict, num_samples_per_chain, method, args, random_seed=seeds[k])
    multi_phi_samples[k,:,:] = phi_samples
    print('acceptance_rates in the sampling phase per window =', acceptance_rates[10:])
    print()
```

    Running HMC chain # 0 ...
    acceptance_rates in the sampling phase per window = [0.73, 0.66, 0.64, 0.7, 0.49, 0.8, 0.61, 0.68, 0.67, 0.67]

    Running HMC chain # 1 ...
    acceptance_rates in the sampling phase per window = [0.7, 0.64, 0.65, 0.63, 0.66, 0.63, 0.55, 0.66, 0.57, 0.65]

    Running HMC chain # 2 ...
    acceptance_rates in the sampling phase per window = [0.59, 0.63, 0.71, 0.7, 0.6, 0.66, 0.67, 0.75, 0.56, 0.66]

    Running HMC chain # 3 ...
    acceptance_rates in the sampling phase per window = [0.63, 0.71, 0.64, 0.67, 0.74, 0.65, 0.61, 0.57, 0.56, 0.65]

    Running HMC chain # 4 ...
    acceptance_rates in the sampling phase per window = [0.71, 0.64, 0.7, 0.6, 0.69, 0.65, 0.64, 0.62, 0.62, 0.6]

    Running HMC chain # 5 ...
    acceptance_rates in the sampling phase per window = [0.56, 0.68, 0.66, 0.76, 0.71, 0.54, 0.57, 0.69, 0.76, 0.68]

    Running HMC chain # 6 ...
    acceptance_rates in the sampling phase per window = [0.62, 0.59, 0.69, 0.64, 0.63, 0.64, 0.72, 0.72, 0.75, 0.65]

    Running HMC chain # 7 ...
    acceptance_rates in the sampling phase per window = [0.53, 0.72, 0.6, 0.63, 0.63, 0.65, 0.61, 0.78, 0.67, 0.63]

    Running HMC chain # 8 ...
    acceptance_rates in the sampling phase per window = [0.74, 0.64, 0.68, 0.61, 0.69, 0.7, 0.64, 0.68, 0.69, 0.77]

    Running HMC chain # 9 ...
    acceptance_rates in the sampling phase per window = [0.59, 0.6, 0.67, 0.52, 0.79, 0.65, 0.65, 0.71, 0.71, 0.56]

### (4b) Compute R_hat for each component of phi

If the HMC chains are well mixed, the "potential scale reduction factors" `R_hats` of all the components of phi should be around 1. If this is not the case, redo (4a) with a larger value of `Le`.

```python
R_hats = compute_R_hat(multi_phi_samples)
print('R_hat: %.3f (# %d) ~ %.3f (# %d)' % (R_hats.min(), R_hats.argmin(), R_hats.max(), R_hats.argmax()))
```

    R_hat: 0.996 (# 995) ~ 1.030 (# 802)

### (4c) Combine all samples

If satisfied with the samples, turn the 3D numpy array `multi_phi_samples` into a 2D numpy array `phi_samples`.

```python
phi_samples = combine_samples(multi_phi_samples)
```

### (4d) Plot distribution of the i-th component of phi

Specify the following parameters:

`ii` = list of components of phi

This will plot marginal distribution of these `ii` components of phi, along with the MAP estimate.

```python
ii = [0, 1, 2]

# ----------

phi_samples_list = [phi_samples]
phi_map = phi_star
num_bins = 15
colors=['blue']

for i in ii:
    plot_distribution(i, phi_samples_list, phi_map, num_bins, colors)
```

```python

```


# "모분산 불편 추정량과 모 표준편차"
> "모분산의 불편 추정량과 모 표준편차의 관계를 예제를 통하여 계산해 본다."

- toc: false
- badges: true
- author: 단호진
- categories: [statistics]

수리통계학 연습 문제 4.2.10을 살피다 지금까지 별 의심 없이 써왔던 모분산의 불편 추청량 $S^2$을 다시 생각해보게 되었다. 모 표준편차의 분편 추정량을 $\sqrt{S^2}$으로 볼 수 있냐는 문제이다. 모분산의 불편 추정량은 다음과 같다.

$S^2 = \frac{\sum_{i=1}^{n} (X_i - \bar X_n)^2}{n - 1}$

$X_i \sim N(\mu, \sigma^2)$이고, $\bar X_n = \sum_{i=1}^{n} X_i$이다. 불편 추청량이라 함은 $E(S^2) = \sigma^2$ 식이 성립한다는 것이다. 그렇다면, $E(S) = \sigma$이라고 볼 수 있나? 그렇지 않다. 더 자세한 내용은 모 표준 편차의 불편 추청에 관한 위키피디아 문서를 참고하기 바란다[2]. 문제의 힌트를 사용해서 $E(S)$를 계산해 보겠다.

$E(S) = \frac{\sigma}{\sqrt{n-1}} E\left[\sqrt{\frac{(n-1)S^2}{\sigma^2}}\right]$

제곱근 안의 항은 $\frac{(n-1)S^2}{\sigma^2} \sim \chi^2(n-1)$이므로 감마분포의 확률밀도함수(pdf) $f(x)$를 이용하여 다음과 같이 쓸 수 있다.

$E(S) = \frac{\sigma}{\sqrt{n-1}} \int_0^{\infty} x^{1/2} f(x) dx$

단, 확률밀도함수 f(x)는 자유도 $r$과 정의 구간 $0<x<\infty$에 대하여 다음과 같다.

$f(x) = \frac{1}{\Gamma(r/2) 2^{r/2}} x^{r/2 - 1} e^{-x/2}$

연습문제에서 $n=9$이고, $r=n-1=8$이다. 자유도를 적분식에 넣고 감마 함수로 정리하면 어렵지 않게 $E(s)$ 값을 계산할 수도 있지만, 여기에서는 sympy 패키지를 이용하여 적분을 풀어보겠다.

참고 문헌

1. 호그, 매킨, 크레이그, 박태영 옮김, 수리통계학 개론, 7판, Pearson/경문사, 2018
1. 위키피디아, Unbiased estimation of standard deviation, 최종 편집 2020년 5월 7일, https://en.wikipedia.org/wiki/Unbiased_estimation_of_standard_deviation

```python
from sympy import *
init_printing()

x, sigma = symbols('x, sigma')
n = 9
r = Rational(n - 1, 1)

f = 1 / gamma(r / 2) / 2**(r / 2) * x**(r / 2 - 1) * exp(-x / 2)
f
```

```python
E_S = sigma / sqrt(r) * integrate(sqrt(x) * f, [x, 0, oo])
# print(E_S)
E_S
```

```python
# print(E_S.evalf())
E_S.evalf()
```

불편 분산의 제곱근 기댓값은 $E(S) = 0.969 \sigma$로 **불편 분산의 제곱근은 모 표준 편차의 불편량으로 사용할 수 없다**는 점을 확인하였다.

4.2.10 (b)에서 신뢰구간이 확률변수 $t(8) = \sqrt{9} (\bar X - \mu) / S$에 근거를 두므로 95% 신뢰 구간의 길이는 $2t_{\alpha/2, n-1} S / \sqrt{n}$이다. 마지막으로 $S$와 $\sigma$ 관계를 삽입하면 신뢰 구간의 길이를 얻을 수 있다.

```python
from scipy.stats import t
import numpy as np

rv = t(n - 1)

2 * rv.ppf(0.975) / np.sqrt(9) * 0.96931
```

최종적으로 계산된 신뢰 구간의 길이는 1.49 $\sigma$이다.


# Preliminaries

## Libraries
We use scikit-learn library to achieve goals in task 1 & task 2. In addition, we use networkx library to perform propensity score matching at task 1.

```python
import pandas as pd
import seaborn as sns
import networkx as nx
import itertools
import numpy as np

from sklearn.linear_model import LinearRegression, LogisticRegression
from sklearn.model_selection import cross_val_predict
from sklearn.model_selection import cross_val_score
from sklearn.datasets import fetch_20newsgroups
from sklearn.feature_extraction.text import TfidfVectorizer
from sklearn.model_selection import train_test_split
from sklearn.naive_bayes import MultinomialNB
from sklearn import metrics
from sklearn.metrics import confusion_matrix
from sklearn.metrics import f1_score
from sklearn.ensemble import RandomForestClassifier
from sklearn.model_selection import GridSearchCV
from sklearn.model_selection import cross_val_score

from IPython.display import display
import matplotlib.pyplot as plt
%matplotlib inline

import networkx as nx
```

## Theme for plots
Overall, we use seaborn to plot graphs and charts. We pick whitegrid for simplicity.

```python
sns.set(style="whitegrid")
```

## Predefined Functions
We define six functions reduce the number of lines of code especially in task 1 and provide reusability.

#### 1. Splitting Dataset into two group based on treat or not
* Simply divide the dataset based on value on treat column.

```python
def split_data(df):
    return df.loc[df["treat"]==1], df.loc[df["treat"]==0]
```

#### 2. Plotting Continuous Features Distribution
* Draw distribution of each continuous feature (age, educ, re74, and re75) in treated and control group
* This will result 4 plots with 2 charts each.

```python
def plot_continuous_features_distribution(df_treat_group, df_control_group):
    # Plot 2 x 2
    fig, ([ax1, ax2], [ax3, ax4]) = plt.subplots(nrows=2, ncols=2)
    fig.set_size_inches(16, 12)

    # Draw distribution of age
    ax1 = sns.distplot(df_treat_group['age'], ax=ax1)
    ax1 = sns.distplot(df_control_group['age'], ax=ax1)
    ax1.set_title("Distribution of Age in Treated (Blue) and Control(Green)")

    # Draw distribution of years of education
    ax2 = sns.distplot(df_treat_group['educ'], ax=ax2)
    ax2 = sns.distplot(df_control_group['educ'], ax=ax2)
    ax2.set_title("Distribution of Years of Education in Treated (Blue) and Control(Green)")

    # Draw distribution of earnings in 1974
    ax3 = sns.distplot(df_treat_group['re74'], ax=ax3)
    ax3 = sns.distplot(df_control_group['re74'], ax=ax3)
    ax3.set_title("Distribution of Earings 1975 in Treated (Blue) and Control(Green)")

    # Draw distribution of earnings in 1975
    ax4 = sns.distplot(df_treat_group['re75'], ax=ax4)
    ax4 = sns.distplot(df_control_group['re75'], ax=ax4)
    ax4.set_title("Distribution of Earings 1975 in Treated (Blue) and Control(Green)")
```

#### 3. Getting the percentages of each binary feature within dataset
* Account the percentage of each binary features (race [black, white, hispan], married, nodegree) within dataset.
* The corresponding result will be returned in panda dataframe object

```python
def get_binary_features_distribution(df_treat_group, df_control_group):
    # Get the number of subjects for each group
    n_treated = df_treat_group.shape[0]
    n_control = df_control_group.shape[0]

    # gather the number of black, white, hispan, married, and nodegree subjects for both treated and control group
    n_black_treated    = df_treat_group.loc[df_treat_group["black"]==1].shape[0]
    n_black_control    = df_control_group.loc[df_control_group["black"]==1].shape[0]
    n_white_treated    = df_treat_group.loc[(df_treat_group["hispan"]==0) & (df_treat_group["black"]==0)].shape[0]
    n_white_control    = df_control_group.loc[(df_control_group["hispan"]==0) & (df_control_group["black"]==0)].shape[0]
    n_hispan_treated   = df_treat_group.loc[df_treat_group["hispan"]==1].shape[0]
    n_hispan_control   = df_control_group.loc[df_control_group["hispan"]==1].shape[0]
    n_married_treated  = df_treat_group.loc[df_treat_group["married"]==1].shape[0]
    n_married_control  = df_control_group.loc[df_control_group["married"]==1].shape[0]
    n_nodegree_treated = df_treat_group.loc[df_treat_group["nodegree"]==1].shape[0]
    n_nodegree_control = df_control_group.loc[df_control_group["nodegree"]==1].shape[0]

    # print the number of each feature
    print("Number of Black Subject in Treated Group: ",n_black_treated)
    print("Number of Black Subject in Control Group: ",n_black_control)
    print("Number of White Subject in Treated Group After Matching: ",n_white_treated)
    print("Number of White Subject in Control Group After Matching: ",n_white_control)
    print("Number of Hispan Subject in Treated Group: ",n_hispan_treated)
    print("Number of Hispan Subject in Control Group: ",n_hispan_control)
    print("Number of Married Subject in Treated Group: ",n_married_treated)
    print("Number of Married Subject in Control Group: ",n_married_control)
    print("Number of Nodegree Subject in Treated Group: ",n_nodegree_treated)
    print("Number of Nodegree Subject in Control Group: ",n_nodegree_control)

    # percentage_of_feature = count(features) / n_data
    pr_black_treated    = n_black_treated    / n_treated
    pr_black_control    = n_black_control    / n_control
    pr_white_treated    = n_white_treated    / n_treated
    pr_white_control    = n_white_control    / n_control
    pr_hispan_treated   = n_hispan_treated   / n_treated
    pr_hispan_control   = n_hispan_control   / n_control
    pr_married_treated  = n_married_treated  / n_treated
    pr_married_control  = n_married_control  / n_control
    pr_nodegree_treated = n_nodegree_treated / n_treated
    pr_nodegree_control = n_nodegree_control / n_control

    data = {
        'features':['black','hispan','white', 'married', 'not married', 'non degree', 'with degree'],
        'pr_treated':[pr_black_treated, pr_hispan_treated, pr_white_treated, \
                      pr_married_treated, 1 - pr_married_treated, \
                      pr_nodegree_treated, 1 - pr_nodegree_treated],
        'pr_control':[pr_black_control, pr_hispan_control, pr_white_control, \
                      pr_married_control, 1 - pr_married_control, \
                      pr_nodegree_control, 1 - pr_nodegree_control]
       }

    # form dataframe of percentage of binary features (race, married, nodegree)
    df_categorical_features_pr = (pd.DataFrame(data, columns = ['features', 'pr_treated', 'pr_control']))
    display(df_categorical_features_pr[0:3])
    display(df_categorical_features_pr[3:5])
    display(df_categorical_features_pr[5:])

    return df_categorical_features_pr
```

#### 4. Plotting binary features distribution
* Draw distribution of each binary feature (race [black, white, hispan], married, and nodegree) in treated and control group
* This will result 1 figure with 3 plot distributions.

```python
# function for ploting each distribution
# use bar chart to depict percentage of binary values in dataset
def add_subplot(data, position, title_):
    ax = data.sort_values(["pr_treated"]) \
                          .plot.bar(ax = position, y=["pr_treated","pr_control"],x=["features"], figsize=(20,4), rot = 0)
    ax.set_xlabel('Percentage of Features');
    ax.set_ylabel('Percentage of Features');
    ax.set_title(title_);

# call this function to plot binary features
def plot_binary_features_distribution(df_categorical_features_pr):
    # create subplot
    fig, axes = plt.subplots(nrows=1, ncols=3)

    # first, let's plot the distribution of race
    add_subplot(df_categorical_features_pr[0:3],
                axes[0],
                'Percentage of Race in Treated Group and Control Group'
                )

    # second, let's plot the distribution of marriage
    add_subplot(df_categorical_features_pr[3:5],
                axes[1],
                'Percentage of Marriage in Treated Group and Control Group'
                )

    # third, let's plot the degree
    add_subplot(df_categorical_features_pr[5:7],
                axes[2],
                'Percentage of School Degree in Treated Group and Control Group'
                )
```

#### 5. Matching the pair control-treated subject with networkx
* Use networkx graph package to perform max_weight_matching based on "costs" between control and treated subject.
* Assign the matching result into dataframe and return the dataframe

```python
def graph_matching(df_with_costs):
    # Build empty graph
    G = nx.Graph()

    # Build graph with vertices sources, destinations, and distance obtained directly from panda dataframe
    G = nx.from_pandas_dataframe(df_with_costs, '0', 'b', ['costs'])

    # Perform max weight matching
    match_control_treated = nx.max_weight_matching(G, maxcardinality=True)

    # Build dataframe from tree result, containing matching pair treated-control subject
    df_prop_matched            = pd.DataFrame(columns=["control","treated"])
    df_prop_matched['treated'] = match_control_treated.keys()
    df_prop_matched['control'] = match_control_treated.values()

    # The dataframe contains redundant entries, e.g. PSID1 - NSW185, NSW185 - PSID1
    # So we sort the values and only pick the first half
    n_treated       = int((df_prop_matched.shape[0])/2)
    df_prop_matched = df_prop_matched.sort_values(by="treated")[:n_treated]

    return df_prop_matched
```

#### 6. Plotting distribution of earnings outcome (re78)
* Plot the earnings outcome with boxplot
* Use the boxplots to compare earnings between treated and control group

```python
def plot_re78_distribution(df, str_flag=""):
    # draw 1 x 2 plot
    fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True)
    fig.set_size_inches(8, 8)
    ax1 = sns.boxplot(y = 're78_control' , data=df, ax = ax1, palette="Set3", showfliers=False)
    ax1.set_title("Real earnings in 1978 for Control Group")

    ax2 = sns.boxplot(y = 're78_treated' , data=df, ax = ax2, palette="Set3", showfliers=False)
    ax2.set_title("Real earnings in 1978 for Treated Group")

    print("Average earning of Control group " + str_flag + ": ",df['re78_control'].mean())
    print("Median earning of Control group " + str_flag + ": ",df['re78_control'].median())
    print("Average earning of Treated group " + str_flag + ": ",df['re78_treated'].mean())
    print("Median earning of Treated group " + str_flag + ": ",df['re78_treated'].median())
```

# Main Homework

## 1. Propensity Score Matching

Read data from the provided dataset and form raw dataframe.

```python
# df_raw: bulk classic dataset from Robert LaLonde's study
dataset_file = 'lalonde.csv'
df_raw       = pd.read_csv(dataset_file)

# print general information of the datasets
print("Number of subjects: ",df_raw.shape[0])

# preview the data
display(df_raw.head(5))
```

    Number of subjects:  614

      id
      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78

      0
      NSW1
      1
      37
      11
      1
      0
      1
      1
      0.0
      0.0
      9930.0460

      1
      NSW2
      1
      22
      9
      0
      1
      0
      1
      0.0
      0.0
      3595.8940

      2
      NSW3
      1
      30
      12
      1
      0
      0
      0
      0.0
      0.0
      24909.4500

      3
      NSW4
      1
      27
      11
      1
      0
      0
      1
      0.0
      0.0
      7506.1460

      4
      NSW5
      1
      33
      8
      1
      0
      0
      1
      0.0
      0.0
      289.7899

### 1a. Naive Analysis

First, we split data into two groups, treatment and control, based on treatment on subject or not (treat 1 or 0)

```python
# split using predefined function split_data
def split_data(df):
    return df.loc[df["treat"]==1], df.loc[df["treat"]==0]

df_treated_raw, df_control_raw = split_data(df_raw)

# print number of treated and control subjects
print("Number of treated subjects: ",df_treated_raw.shape[0])
print("Number of control subjects: ",df_control_raw.shape[0])
```

    Number of treated subjects:  185
    Number of control subjects:  429

* We discover that zeros are frequently presence in re78 column. Zeros mean unemployed condition (no salaries) and thus we cannot really analyze the results of job training program. Because we strictly rely with real earnings in 1978 to determine the outcome of job training program, there will be some cases where we decide to ignore zeros.
* Hence we create dataframes with zeros replaces with NaN. By default, pandas operation will not include zeros entries and we can count statistics of the rest non-zeros values.

```python
# we don't want to account for zero, thus we show the dataframe with NaN
# note that this replaces binaries features as well, but we can ignore that since here at this point we focus on re78
df_treated_with_nan    = df_treated_raw.replace(0,np.NAN)
df_control_with_nan    = df_control_raw.replace(0,np.NAN)
```

Then, we compare 4 different statistics of re78:
1. Earnings in 78 of **treated group (NSW)**
2. Earnings in 78 of **control group (PSID)**
3. Earnings in 78 of **treated group (NSW)**, employed subject only (ignore subject with zeros earnings)
4. Earnings in 78 of **control group (PSID)**, employed subject only (ignore subject with zeros earnings)

```python
print("Re78 statistics of Treated vs Control Group")

# form stats dataframe
stats         = pd.DataFrame(df_treated_raw.describe()[['re78']])
stats.columns = ["treated_re78"]

# add statistics of re78
stats["control_re78"]                     = df_control_raw.describe()[['re78']]
stats["treated_re78_without_unemployed"]  = df_treated_with_nan.describe()[['re78']]
stats["control_re78_without_unemployed"]  = df_control_with_nan.describe()[['re78']]

display(stats)
```

    Re78 statistics of Treated vs Control Group

      treated_re78
      control_re78
      treated_re78_without_unemployed
      control_re78_without_unemployed

      count
      185.000000
      429.000000
      140.000000
      331.000000

      mean
      6349.143530
      6984.169742
      8389.939665
      9051.990391

      std
      7867.402218
      7294.161791
      8042.944693
      7087.340468

      min
      0.000000
      0.000000
      289.789900
      31.032260

      25%
      485.229800
      220.181300
      3167.353750
      2978.358000

      50%
      4232.309000
      4975.505000
      6504.144500
      7310.313000

      75%
      9642.999000
      11688.820000
      11017.730000
      14131.500000

      max
      60307.930000
      25564.670000
      60307.930000
      25564.670000

#### Distribution of re78 between two groups - Include outliers and zeros (unemployed subjects)
Now we want to see the distributions of outcome variable re78 between two groups.

First we see the distributions when we include zeros and outliers

```python
# Plot 2 x 1
fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True)
fig.set_size_inches(16, 8)
ax1 = sns.boxplot(y = 're78', data=df_treated_raw, ax = ax1, palette="Set3")
ax1 = sns.swarmplot(y = 're78', data=df_treated_raw, ax = ax1, color=".25")
ax1.set_title("Real earnings in 1978 for Treated Group")

ax2 = sns.boxplot(y = 're78', data=df_control_raw, ax = ax2, palette="Set3")
ax2 = sns.swarmplot(y = 're78', data=df_control_raw, ax = ax2, color=".25")
ax2.set_title("Real earnings in 1978 for Control Group")

# Print mean and median of re78 between two groups
print("Mean earnings in 1978 of Treated group: ", df_treated_raw['re78'].mean())
print("Median earnings in 1978 of Treated group: ", df_treated_raw['re78'].median())
print("Mean earnings in 1978 of Control group: ", df_control_raw['re78'].mean())
print("Median earnings in 1978 of Control group: ", df_control_raw['re78'].median())
```

#### Observing outliers in treated group & zeros earnings in both group
As we can see in above figure, there are several outliers in treated group with earnings at least 5 times larger than the average group earnings.

```python
# consider outliers are ones with re78 > 5*mean(re78)
df_treated_raw_outliers = df_treated_raw.loc[df_treated_raw['re78'] > 5 * stats.loc["mean"]["treated_re78"]]

# inspect zeros
n_zeros_treated_raw = df_treated_raw.loc[df_treated_raw["re78"]==0].shape[0]
n_zeros_control_raw = df_control_raw.loc[df_control_raw["re78"]==0].shape[0]

# Print brief information about outliers and zeros
print("Number of outliers in treated group: ", df_treated_raw_outliers.shape[0])
print("Number of zeros earnings in 1978 of treated group: ", n_zeros_treated_raw)
print("Number of zeros earnings in 1978 of control group: ", n_zeros_control_raw)

# Display the outliers
display(df_treated_raw_outliers.head(5))
```

    Number of outliers in treated group:  3
    Number of zeros earnings in 1978 of treated group:  45
    Number of zeros earnings in 1978 of control group:  98

      id
      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78

      106
      NSW107
      1
      27
      13
      1
      0
      0
      0
      0.00
      0.000
      34099.28

      131
      NSW132
      1
      28
      11
      1
      0
      0
      1
      0.00
      1284.079
      60307.93

      181
      NSW182
      1
      25
      14
      1
      0
      1
      0
      35040.07
      11536.570
      36646.95

It is astonishing to see **45 out of 185** and **98 out of 429** (almost quarter each) subjects in treated group  and control group are unemployed persons. Thus this bold our reason to plot only non-zeros values of re78.

#### Distribution of re78 between two groups - Exclude outliers and zeros (unemployed subjects)
Then let's see the distributions when we exclude zeros and outliers from both group

```python
# Plot 2 x 1
fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True)
fig.set_size_inches(12, 6)
ax1 = sns.boxplot(y = 're78' , data=df_treated_with_nan, ax = ax1, palette="Set3", showfliers=False)
ax1.set_title("Real earnings in 1978 for Treated Group")

ax2 = sns.boxplot(y = 're78' , data=df_control_with_nan, ax = ax2, palette="Set3", showfliers=False)
ax2.set_title("Real earnings in 1978 for Control Group")

print("Average earnings in 1978 of Treated group (zeros excluded): ",df_treated_with_nan['re78'].mean())
print("Median earnings in 1978 of Treated group (zeros excluded): ",df_treated_with_nan['re78'].median())
print("Average earnings in 1978 of Control group (zeros excluded): ",df_control_with_nan['re78'].mean())
print("Median earnings in 1978 of Control group (zeros excluded): ",df_control_with_nan['re78'].median())
```

### Naive observations

> At this point, naive researchers may conclude that:
1. There is **no impact of job training program** to subject's earning as outcome (re78). Subjects in treated group tends to have lower outcome as we can see from both their earnings mean and median. Recall that median is sensitive to outliers.
2. The training is effective only to specific indivinduals, several subjects whom we can see as outliers in treated group.

### 1b. A Closer Look Into the Data
Now we delve deeper into distibutions of features between two groups. We can separate the analysis based on:
1. Binary features (either one or zero/ yes or no): { married, nodegree }
2. Discrete features (either black, white, hispan) : { race }
3. Continuous features : { age, educ, re74, re75 }

In order to make things simpler, we categorize binary features as : { race, married, nodegree }

```python
# first lets peek for number of data in each group
n_treated_raw = df_treated_raw.shape[0]
n_control_raw = df_control_raw.shape[0]

print("Number of Treated Subjects: ", n_treated_raw)
print("Number of Control Subjects: ", n_control_raw)
```

    Number of Treated Subjects:  185
    Number of Control Subjects:  429

#### Binary Features (race, married, nodegree)
Please note that to see distribution of binary features, it is better if we see the distribution in form of percentages. As we can see that the number of subject is imbalance between two groups (n_control > n_treated)

```python
# use predefined function to get dataframe of percentage of each binary feature in dataset
df_binary_features_pr = get_binary_features_distribution(df_treated_raw, df_control_raw)
```

    Number of Black Subject in Treated Group:  156
    Number of Black Subject in Control Group:  87
    Number of White Subject in Treated Group After Matching:  18
    Number of White Subject in Control Group After Matching:  281
    Number of Hispan Subject in Treated Group:  11
    Number of Hispan Subject in Control Group:  61
    Number of Married Subject in Treated Group:  35
    Number of Married Subject in Control Group:  220
    Number of Nodegree Subject in Treated Group:  131
    Number of Nodegree Subject in Control Group:  256

      features
      pr_treated
      pr_control

      0
      black
      0.843243
      0.202797

      1
      hispan
      0.059459
      0.142191

      2
      white
      0.097297
      0.655012

      features
      pr_treated
      pr_control

      3
      married
      0.189189
      0.512821

      4
      not married
      0.810811
      0.487179

      features
      pr_treated
      pr_control

      5
      non degree
      0.708108
      0.596737

      6
      with degree
      0.291892
      0.403263

#### Plotting Distribution of Binary Features (race, married, nodegree)

```python
# use predefined function to plot binary features
plot_binary_features_distribution(df_binary_features_pr)
```

#### Observations of binary features
We see that there is a significant difference in distribution of **"Race"** and **"Married"** features.

#### Plotting Continous Features (age, educ, re74, re75) - Box Plots (without outliers)

```python
# Plot 2 x 2
fig, ([ax1, ax2], [ax3, ax4]) = plt.subplots(nrows=2, ncols=2)
fig.set_size_inches(12, 12)
ax1 = sns.boxplot(y="age", x="treat", ax=ax1, data=df_raw, palette="Set3", showfliers=False)
ax1.set_title("Distribution of Age (Green=Control, Yellow=Treated)")

ax2 = sns.boxplot(y="educ", x="treat", ax=ax2, data=df_raw, palette="Set3", showfliers=False)
ax2.set_title("Distibution of Years of Education (Green=Control, Yellow=Treated)")

ax3 = sns.boxplot(y="re74", x="treat", ax=ax3, data=df_raw, palette="Set3", showfliers=False)
ax3.set_title("Distribution of Earnings in 1974 (Green=Control, Yellow=Treated)")

ax4 = sns.boxplot(y="re75", x="treat", ax=ax4, data=df_raw, palette="Set3", showfliers=False)
ax4.set_title("Distribution of Earnings in 1975 (Green=Control, Yellow=Treated)")
```

We can sure from boxplot above that there are plenty of zeros present in re74 and re75.
As we see that the mean for treated group is close to zero.

#### Plotting Continous Features (age, educ, re74, re75) - Distribution Plot With Outliers
Disclaimer: Although there are zeros in re74 and re75, we cannot just exclude these values since we assume that job training program made unemployed subjects in 1974 and 1975 gained job in 1978.

```python
# use our predefined function to plot continuous features-values distribution
plot_continuous_features_distribution(df_treated_raw, df_control_raw)
```

#### Observations of continuous features
We see that the distribution of continious features between two groups are roughly same. The only obvious imbalance distribution is in feature **"age"**.

#### What does it mean to conclusion we had on previous question?
> In naive/superficial analysis, we compare and draw conclusion for earnings outcome as if we have the identical subjects for both group. For example, we tend to think as if we are comparing an earning outcome of a same person X before and after treatment.

> We have already known that even the numbers of subjects between two groups are different.
Thus we know that distribution of features are most likely different. We cannot infer the training program does not work without considering underlying factor (features) that might give significant correlation to the earnings outcome.

> Then at this point, we assume that "Race", "Married", "Age" have significant impact to earning outcome and we want to avoid pitfalls in observational studies as we had in 1a.

### 1c. Propensity Score Model
From [Rosenbaum](http://www.stewartschultz.com/statistics/books/Design%20of%20observational%20studies.pdf):

In the population before matching, we imagine that subject received treatment
with probability π, independently of other subjects, where π may vary from one
person to the next and is not known. More precisely, π

\begin{align}
\pi_l & = Pr(Z_l = 1 | r_tl, r_cl, x_l, u_l) \\
\end{align}

Here we define propensity score of each subject is equal to its probability of being treated. To calculate propensity score, we use logistic regression model.

```python
# Now we have eight features; 4 binaries and 4 continuous features
features = ['age','educ','black','hispan','married','nodegree','re74','re75']

# Logistic Regression model
lr = LogisticRegression()

# X = { features }, y = treat
X = df_raw[features]
y = df_raw.treat

# train the model with entire dataset
lr.fit(X, y)

# print the model of our logistic regression
for f in range(len(features)):
    print("{0} * {1} + ".format(lr.coef_[0][f], features[f]))
print(lr.intercept_)
```

    -0.006794203178905679 * age +
    -0.06602229029570279 * educ +
    2.584312591583524 * black +
    -0.0406653837003028 * hispan +
    -0.8414791176919386 * married +
    -0.36073074418825135 * nodegree +
    -7.541712717668056e-05 * re74 +
    5.759223918378022e-05 * re75 +
    [-0.63204746]

$$y = \beta_0 + \beta_1  \times educ + \beta_2  \times black + \beta_3  \times hispan + \beta_4  \times married + \beta_5  \times nodegree + \beta_6  \times re74 + \beta_7  \times re75$$
$$y = -0.6320 - 0.0068 \times age - 0.0660 \times educ + 2.5843  \times black - 0.0407 \times hispan - 0.8415 \times married - 0.3607 \times nodegree - 7.5417 \times re74 + 5.7592 \times re75$$

As we explained before, we use the logistic regression to get probability of being treated for each subject.

```python
# We only pick the probability of being treated by accessing [:,1]
pred = pd.DataFrame(lr.predict_proba(X)[:,1])
pred.columns = ["propensity_score"]
pred.head(5)
```

      propensity_score

      0
      0.443350

      1
      0.144660

      2
      0.722355

      3
      0.664151

      4
      0.698286

Let's split the data again into treated and control group but this time we add **propensity_score** to every subject.

```python
# Create new dataframe, add "propensity_score" column
df_prop_scr                     = df_raw
df_prop_scr['propensity_score'] = pred['propensity_score']

# split the data using predefined split_data function
df_treated_prop_scr, df_control_prop_scr = split_data(df_prop_scr)

display(df_control_prop_scr.head(5))
display(df_treated_prop_scr.head(5))
```

      id
      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78
      propensity_score

      185
      PSID1
      0
      30
      12
      0
      0
      1
      0
      20166.73
      18347.23
      25564.67
      0.050504

      186
      PSID2
      0
      26
      12
      0
      0
      1
      0
      25862.32
      17806.55
      25564.67
      0.033330

      187
      PSID3
      0
      25
      16
      0
      0
      1
      0
      25862.32
      15316.21
      25564.67
      0.022574

      188
      PSID4
      0
      42
      11
      0
      0
      1
      1
      21787.05
      14265.29
      15491.01
      0.024906

      189
      PSID5
      0
      25
      9
      1
      0
      1
      1
      14829.69
      13776.53
      0.00
      0.416054

      id
      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78
      propensity_score

      0
      NSW1
      1
      37
      11
      1
      0
      1
      1
      0.0
      0.0
      9930.0460
      0.443350

      1
      NSW2
      1
      22
      9
      0
      1
      0
      1
      0.0
      0.0
      3595.8940
      0.144660

      2
      NSW3
      1
      30
      12
      1
      0
      0
      0
      0.0
      0.0
      24909.4500
      0.722355

      3
      NSW4
      1
      27
      11
      1
      0
      0
      1
      0.0
      0.0
      7506.1460
      0.664151

      4
      NSW5
      1
      33
      8
      1
      0
      0
      1
      0.0
      0.0
      289.7899
      0.698286

#### Plotting the distribution of propensity scores

```python
# create 2 x 1 plot
fig, (ax1, ax2) = plt.subplots(ncols=2, sharey=True)
fig.set_size_inches(16, 4)
ax1 = sns.distplot(df_control_prop_scr['propensity_score'], ax = ax1)
ax1.set_title("Distribution of Propensity scores of Control Group")

ax2 = sns.distplot(df_treated_prop_scr['propensity_score'], ax = ax2)
ax2.set_title("Distribution of Propensity scores of Treated Group")
```

#### Observations - Propensity Score Matching with Logistic Regression

> We observe that we are dealing with biased data as some subjects in control group have higher score of getting the treatment and vice versa for some subjects in treated group

### 1d. Balancing dataset via matching Propensity Score
Steps for this part
* Measuring costs between all subjects in treated group with all subjects in control group
* Use networkx graph function to maximize similiarity between matched subjects.
* Data wrangling
* Analysis of re78 after matching
* Analysis of features-values distribution after matching

#### Costs
We define cost as:
$$cost_{ij} = | PropensityScore_i - PropensityScore_j |$$

Where; i = subject i from either control or treated group, j = subject in different group from i.

For each subject from treated group, we measure its costs to every datapoint in control group.

```python
# Init an empty dataframe
df_pr_costs = pd.DataFrame()

# Loop through each subject in control group.
for row in df_control_prop_scr[['id','propensity_score']].iterrows():
    # get subject id and subject propensity score
    subject_id     = row[1][0]
    subject_pr_scr = row[1][1]

    # measure absolute costs from all subject in treated group to this control subject
    costs               = abs(subject_pr_scr - df_treated_prop_scr.propensity_score)

    # form dataframe contains: "0" source, "b" destination, "costs" of "0"-"b"
    df_pr_cost          = pd.DataFrame(df_treated_prop_scr[['id']])
    df_pr_cost.columns  = [['b']]
    df_pr_cost['0']     = subject_id
    df_pr_cost["costs"] = pd.DataFrame(costs)

    # append distance per subject
    df_pr_costs           = pd.concat([df_pr_costs, df_pr_cost])

print("Dataframe with costs:")
display(df_pr_costs.head(5))
```

    Dataframe with costs:

      b
      0
      costs

      0
      NSW1
      PSID1
      0.392846

      1
      NSW2
      PSID1
      0.094156

      2
      NSW3
      PSID1
      0.671851

      3
      NSW4
      PSID1
      0.613647

      4
      NSW5
      PSID1
      0.647782

#### Matching with Networkx Graph
To ensure that each data point from the control group is matched with at most one data point from the treated group, we use networkx package.

```python
# Perform max weight matching using our predefined function
df_prop_matched = graph_matching(df_pr_costs)

print("Matched control-treated subject:")
df_prop_matched.head()
```

    Matched control-treated subject:

      control
      treated

      1
      PSID429
      NSW1

      19
      PSID420
      NSW10

      199
      PSID330
      NSW100

      201
      PSID329
      NSW101

      203
      PSID328
      NSW102

#### Data wrangling
Finally, we add corresponding values of re78 to each pair of control-treated subject.

```python
# set "id" as index of df_raw, form new dataframe
df_raw_indexed = df_raw.set_index("id")

# Get value of re78 each matched control-treated subject
re78_control = df_raw_indexed.loc[df_prop_matched["control"]]["re78"]
re78_treated = df_raw_indexed.loc[df_prop_matched["treated"]]["re78"]

# add the re78 into df_prop_matched
df_prop_matched["re78_control"] = list(re78_control)
df_prop_matched["re78_treated"] = list(re78_treated)

print("Matched control-treated subject with their corresponding re78 value")
df_prop_matched.head()
```

    Matched control-treated subject with their corresponding re78 value

      control
      treated
      re78_control
      re78_treated

      1
      PSID429
      NSW1
      1495.4590
      9930.046

      19
      PSID420
      NSW10
      964.9555
      12418.070

      199
      PSID330
      NSW100
      116.7404
      26817.600

      201
      PSID329
      NSW101
      1717.1180
      0.000

      203
      PSID328
      NSW102
      14339.8600
      9265.788

#### Analysis of re78 after propensity score matching
To get brief stats, we compare using descibe()

```python
df_prop_matched.describe()
```

      re78_control
      re78_treated

      count
      185.000000
      185.000000

      mean
      5525.962462
      6349.143530

      std
      6637.979050
      7867.402218

      min
      0.000000
      0.000000

      25%
      0.000000
      485.229800

      50%
      2813.591000
      4232.309000

      75%
      8844.194000
      9642.999000

      max
      25564.670000
      60307.930000

##### Plotting Distribution of re78 between two groups after propensity score matching
Note that we don't include outliers in plot below.

```python
# draw distribution of outcomes (re78) using our predefined functions
plot_re78_distribution(df_prop_matched)
```

##### Observations of re78 values
We can spot that after the matching, the treated group has higher mean, median, and 25% quartile. This means at this point, we can show that the earnings outcome of treated group is better than control group.
> We conclude at this point that the training is working and effective

#### Analysis of features-values distribution after propensity score matching
Now we repeat our result in task 1b but using matched data of control-treated subjects. This is necessary to observe features distribution between matched control and treated group.

```python
# Get only matched control-treated pair from raw dataframe
df_control = df_raw_indexed.loc[df_prop_matched["control"]]
df_treated = df_raw_indexed.loc[df_prop_matched["treated"]]

# note that n_treated should be equal to n_control
n_treated = df_treated.shape[0]
n_control = df_control.shape[0]

# Check how many pairs we have
print("Number of treated subject: ",n_treated)
print("Number of control subject: ",n_control)

# Display pairs-example
display(df_control.head(3))
display(df_treated.head(3))
```

    Number of treated subject:  185
    Number of control subject:  185

      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78
      propensity_score

      id

      PSID429
      0
      16
      9
      0
      0
      0
      1
      0.00
      0.0
      1495.4590
      0.155030

      PSID420
      0
      39
      2
      1
      0
      1
      1
      0.00
      0.0
      964.9555
      0.587354

      PSID330
      0
      22
      16
      1
      0
      0
      0
      2564.68
      0.0
      116.7404
      0.634835

      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78
      propensity_score

      id

      NSW1
      1
      37
      11
      1
      0
      1
      1
      0.0
      0.0
      9930.046
      0.443350

      NSW10
      1
      33
      12
      0
      0
      1
      0
      0.0
      0.0
      12418.070
      0.076562

      NSW100
      1
      31
      9
      0
      1
      0
      1
      0.0
      0.0
      26817.600
      0.137257

###### Binary Features (race, married, nodegree)

```python
# use predefined function to get dataframe of percentage of each binary feature in dataset
df_binary_features_pr = get_binary_features_distribution(df_treated, df_control)
```

    Number of Black Subject in Treated Group:  156
    Number of Black Subject in Control Group:  51
    Number of White Subject in Treated Group After Matching:  18
    Number of White Subject in Control Group After Matching:  112
    Number of Hispan Subject in Treated Group:  11
    Number of Hispan Subject in Control Group:  22
    Number of Married Subject in Treated Group:  35
    Number of Married Subject in Control Group:  76
    Number of Nodegree Subject in Treated Group:  131
    Number of Nodegree Subject in Control Group:  121

      features
      pr_treated
      pr_control

      0
      black
      0.843243
      0.275676

      1
      hispan
      0.059459
      0.118919

      2
      white
      0.097297
      0.605405

      features
      pr_treated
      pr_control

      3
      married
      0.189189
      0.410811

      4
      not married
      0.810811
      0.589189

      features
      pr_treated
      pr_control

      5
      non degree
      0.708108
      0.654054

      6
      with degree
      0.291892
      0.345946

###### Plotting Distribution of Binary Features (race, married, nodegree) after propensity score matching

```python
# use predefined function to plot binary features
plot_binary_features_distribution(df_binary_features_pr)
```

###### Observations of binary features
> After matching, the distibutions of **race** and **married** still have huge differences. At this point we have to **withdraw our previous conclusion** as we cannot infer the result of observational studies in which treatment assignment and response may be caused by featues correlate (black or married).

###### Plotting Distribution of Continous Features (age, educ, re74, re75) after propensity score matching:  Distribution Plot With Outliers

**Disclaimer about zeros in re74 and re75**: Although there are zeros in re74 and re75, we cannot just exclude these values since we **assume** that job training program made unemployed subjects in 1974 and 1975 gained job in 1978.

```python
# use our predefined function to plot continuous features-values distribution
plot_continuous_features_distribution(df_treated, df_control)
```

###### Observations of Continuous features
> We can see that there is no significance difference of distribution between continuous features, before and after matching. After matching we notice that the distribution of continuous features between two groups are roughly same. Consistenly, the only obvious imbalance distribution is in feature **"age"**, same as before matching.

#### 1d recap: We are not satisfied with the result! :(
> After propensity score matching, we still are not happy with the results since the features distribution are not well-balanced. Let's say that training program is well-suited for Husbands (married) to manage work-life balanced or the training is designed to white (non-black) people only. This will mislead our conclusion and it's better if we try to balance the distribution even further, as we are not sure the implication of underlying features!

### 1e. Balancing Groups Further: Binary Features Matching
From the results obtained by 1b and 1d, we can see from the plots that two binaries features are heavily imbalanced; race and married. We aware that the ages distribution is not suifficiently balanced as well. But out strategy is to  pay more attention to "**black**" and "**married**" feature.

Almost the same as 1d, steps for this part:
* Measuring costs between all subjects in treated group with all subjects in control group
* Use networkx graph function to maximize similiarity between matched subjects.
* Data wrangling
* Analysis of features-values distribution after matching

> We use suffix \_fm (final matching) to distinguish variable names in this part and onwards with previous ones.

#### Costs
Important notes:
* Now we cut the node connections if subject in control does not has the same property as subject in treatment. E.g. for a subject i in control, who is black, we measure costs only between it to other black subjects in treatment group.
* We filter "black" and "married" features only.

```python
# Init an empty dataframe: costs of propensity scores between matched features only
df_matched_f_pr_costs = pd.DataFrame()

# Loop through each subject in control group.
for row in df_control_prop_scr[['id','propensity_score','black','married']].iterrows():
    # get subject id and subject propensity score
    subject_id     = row[1][0]
    subject_pr_scr = row[1][1]

    # get black and married property of the control subject
    subject_black   = row[1][2]
    subject_married = row[1][3]

    # measure absolute costs from all subject in treated group to this control subject
    costs = abs(subject_pr_scr - df_treated_prop_scr.loc[(df_treated_prop_scr["black"] == subject_black ) \
                                                         & (df_treated_prop_scr["married"] == subject_married)]
                                                        .propensity_score)

    # form dataframe contains: "0" source, "b" destination, "costs" of "0"-"b"
    df_pr_cost          = pd.DataFrame(df_treated_prop_scr[['id']])
    df_pr_cost.columns  = [['b']]
    df_pr_cost['0']     = subject_id
    df_pr_cost["costs"] = pd.DataFrame(costs)

    # append distance per subject
    df_matched_f_pr_costs           = pd.concat([df_matched_f_pr_costs, df_pr_cost.dropna(axis=0)])

print("Dataframe with costs:")
display(df_matched_f_pr_costs.head(5))
```

    Dataframe with costs:

      b
      0
      costs

      9
      NSW10
      PSID1
      0.026058

      43
      NSW44
      PSID1
      0.023593

      123
      NSW124
      PSID1
      0.010333

      136
      NSW137
      PSID1
      0.031153

      166
      NSW167
      PSID1
      0.005501

#### Matching with Networkx Graph
To ensure that each data point from the control group is matched with at most one data point from the treated group, we use networkx package.

```python
# Perform max weight matching using our predefined function
df_prop_fm = graph_matching(df_matched_f_pr_costs)

print("Matched control-treated subject:")
df_prop_fm.head()
```

    Matched control-treated subject:

      control
      treated

      3
      PSID428
      NSW1

      7
      PSID426
      NSW10

      55
      PSID402
      NSW100

      59
      PSID400
      NSW11

      203
      PSID98
      NSW110

#### Data wrangling
Finally, we add corresponding values of re78 to each pair of control-treated subject.

```python
# Get value of re78 each matched control-treated subject
re78_control = df_raw_indexed.loc[df_prop_fm["control"]]["re78"]
re78_treated = df_raw_indexed.loc[df_prop_fm["treated"]]["re78"]

# add the re78 into df_prop_matched
df_prop_fm["re78_control"] = list(re78_control)
df_prop_fm["re78_treated"] = list(re78_treated)

print("Matched control-treated subject with their corresponding re78 value")
df_prop_fm.head()
```

    Matched control-treated subject with their corresponding re78 value

      control
      treated
      re78_control
      re78_treated

      3
      PSID428
      NSW1
      187.6713
      9930.046

      7
      PSID426
      NSW10
      19464.6100
      12418.070

      55
      PSID402
      NSW100
      0.0000
      26817.600

      59
      PSID400
      NSW11
      648.7220
      8173.908

      203
      PSID98
      NSW110
      0.0000
      0.000

**Note that we will use df_prop_fm for final part 1f**

#### Analysis of features-values distribution after propensity score matching + binary features matching
Now we confirm the distribution of features in our matched dataset control and treated group.

```python
# Get only matched control-treated pair from raw dataframe
df_control_fm = df_raw_indexed.loc[df_prop_fm["control"]]
df_treated_fm = df_raw_indexed.loc[df_prop_fm["treated"]]

# note that n_treated should be equal to n_control
n_treated_fm = df_treated_fm.shape[0]
n_control_fm = df_control_fm.shape[0]

# Check how many pairs we have
print("Number of treated subject: ",n_treated_fm)
print("Number of control subject: ",n_control_fm)

# Display pairs-example
display(df_control_fm.head(3))
display(df_treated_fm.head(3))
```

    Number of treated subject:  116
    Number of control subject:  116

      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78
      propensity_score

      id

      PSID428
      0
      32
      5
      1
      0
      1
      1
      0.0
      0.0
      187.6713
      0.550460

      PSID426
      0
      24
      1
      0
      1
      1
      1
      0.0
      0.0
      19464.6100
      0.108708

      PSID402
      0
      16
      8
      0
      0
      0
      1
      0.0
      0.0
      0.0000
      0.163876

      treat
      age
      educ
      black
      hispan
      married
      nodegree
      re74
      re75
      re78
      propensity_score

      id

      NSW1
      1
      37
      11
      1
      0
      1
      1
      0.0
      0.0
      9930.046
      0.443350

      NSW10
      1
      33
      12
      0
      0
      1
      0
      0.0
      0.0
      12418.070
      0.076562

      NSW100
      1
      31
      9
      0
      1
      0
      1
      0.0
      0.0
      26817.600
      0.137257

> Note that we have less number of matched treated-control subject in dataset. Now we have **116 out of 185** pairs subject in maximum.

###### Binary Features (race, married, nodegree) after Features + Propensity score matching
Recall that we use suffix \_fm (final matching) to distinguish variable names in this part with previous ones.

Now we form dataframe consisting percentages of each binary features

```python
# use predefined function to get dataframe of percentage of each binary feature in dataset
df_pr_binary_features_fm = get_binary_features_distribution(df_treated_fm, df_control_fm)
```

    Number of Black Subject in Treated Group:  87
    Number of Black Subject in Control Group:  87
    Number of White Subject in Treated Group After Matching:  18
    Number of White Subject in Control Group After Matching:  24
    Number of Hispan Subject in Treated Group:  11
    Number of Hispan Subject in Control Group:  5
    Number of Married Subject in Treated Group:  31
    Number of Married Subject in Control Group:  31
    Number of Nodegree Subject in Treated Group:  80
    Number of Nodegree Subject in Control Group:  74

      features
      pr_treated
      pr_control

      0
      black
      0.750000
      0.750000

      1
      hispan
      0.094828
      0.043103

      2
      white
      0.155172
      0.206897

      features
      pr_treated
      pr_control

      3
      married
      0.267241
      0.267241

      4
      not married
      0.732759
      0.732759

      features
      pr_treated
      pr_control

      5
      non degree
      0.689655
      0.637931

      6
      with degree
      0.310345
      0.362069

###### Plotting Distribution of Binary Features (race, married, nodegree) after features + propensity score matching

```python
# use predefined function to plot binary features
plot_binary_features_distribution(df_pr_binary_features_fm)
```

###### Observations of binary features :)
> Now we are happy about the balanced distribution of binary features between treated and control group. We can give toleration to the rest features; nodegree and hispan. Because if we want the distibution to be exactly the same, we will reduce the number of datapoints and yet we have only 116 out of 185 (62%). Therefore we are satisfied with our distribution of binary features.

###### Plotting Distribution of Continous Features (age, educ, re74, re75) after features + propensity score matching:  Distribution Plot With Outliers

**Disclaimer about zeros in re74 and re75**: Although there are zeros in re74 and re75, we cannot just exclude these values since we **assume** that job training program made unemployed subjects in 1974 and 1975 gained job in 1978.

```python
# use our predefined function to plot continuous features-values distribution
plot_continuous_features_distribution(df_treated_fm, df_control_fm)
```

###### Observations of Continuous features ?
We are a little bit disappointed with distribution of "age" which is quite imbalance after all features and propensity score matching that we have done. However, as we have explained before, we decide to focus more on binary features "black" and "married" as they had huge imbalance distribution between control and treated group.
* If we want to balance the "age" distribution we may end up with a few data point (maybe less than 50%) and thus we don't want to draw conclusions using less than 50% data of maximum pairs of subject.

> To sum up, now we are satisfied with out final matching pair of treated and control group since we have better balanced features-values distribution, regardless of the only "age" quite imbalance distribution. We jump to next step with our final dataframe result "df_prop_fm" in order to draw the conclusion of this observational studies.

### 1f. A less naive analysis
Now we compare the outcomes (re78) between treated and control group using number and plots.
We use very final version of matching treated-control subject (result from 1e), namely "df_prop_fm"

```python
# To get brief stats, we compare using descibe()
df_prop_fm.describe()
```

      re78_control
      re78_treated

      count
      116.000000
      116.000000

      mean
      4837.325071
      6220.832549

      std
      6321.256830
      6408.060991

      min
      0.000000
      0.000000

      25%
      0.000000
      733.987100

      50%
      1514.669500
      4277.007000

      75%
      7641.324000
      9970.742000

      max
      23764.800000
      26817.600000

###### Plotting the outcomes (re78) for control and treated group after final matching

```python
# draw distribution of outcomes (re78) using our predefined functions
plot_re78_distribution(df_prop_fm)
```

#### Conclusion of the observational studies on job training program
> We can observe directly from plot above that outcome earnings (re78) for subjects in treated group are larger than those in control group. Mean, median, 25% percentile, of treated group are generally larger than control group. The increase in average earnings is:

$$(6220.832-4837.325)/4837.325 = 28.60\%$$

> Consequently, we conclude that **the job training program is genuinely effective to increase indivindual's earnings**.

## 2. Applied ML

#### Problem description
We are going to build a classifier of news to directly assign them to 20 news categories. Note that the pipeline that you will build in this exercise could be of great help during your project if you plan to work with text!

1. Load the 20newsgroup dataset. It is, again, a classic dataset that can directly be loaded using sklearn ([link](http://scikit-learn.org/stable/datasets/twenty_newsgroups.html)).
[TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), short for term frequency–inverse document frequency, is of great help when if comes to compute textual features. Indeed, it gives more importance to terms that are more specific to the considered articles (TF) but reduces the importance of terms that are very frequent in the entire corpus (IDF). Compute TF-IDF features for every article using [TfidfVectorizer](http://scikit-learn.org/stable/modules/generated/sklearn.feature_extraction.text.TfidfVectorizer.html). Then, split your dataset into a training, a testing and a validation set (10% for validation and 10% for testing). Each observation should be paired with its corresponding label (the article category).

2. Train a random forest on your training set. Try to fine-tune the parameters of your predictor on your validation set using a simple grid search on the number of estimator "n_estimators" and the max depth of the trees "max_depth". Then, display a confusion matrix of your classification pipeline. Lastly, once you assessed your model, inspect the `feature_importances_` attribute of your random forest and discuss the obtained results.

### Data collection

#### Steps in data collection
1. load the whole dataset
2. vectorize the news category with [TF-IDF](https://en.wikipedia.org/wiki/Tf%E2%80%93idf)
3. split the data into 80% training set, 10% test set and 10% validation set

   we use function **train_test_split** to split 80% training set and 20% test set firstly, then use the function again to split test set into 10% test set and 10% validation set

#### Note
Because we are going to use random forest, we don't need to perform standardization on the features

```python
# load the whole dataset
newsgroups = fetch_20newsgroups(subset='all')

# vectorize the dataset with TF-IDF
vectorizer = TfidfVectorizer()
x = vectorizer.fit_transform(newsgroups.data)
y = newsgroups.target

# split the original dataset into 20% validation set, 10% test set and 80% training set
x_train, x_test, y_train, y_test = train_test_split(x, y, test_size=0.2, random_state=1)
# split the test set into 50% validation set and 50% test set (both are 10% of the original dataset)
x_test, x_val, y_test, y_val = train_test_split(x_test, y_test, test_size=0.5, random_state=1)

```

### Model selection

#### Random forest
As required, we select random forest and use grid search to find the best parameters for
- n_estimators : number of decision trees
- max_depth    : depth of each tree

We constrain our max_depth upto 30 and n_estimator upto 1500, and we also test on several setting in other arguments such as min_samples_leaf and max_features, but we find that the default setting has the best performance:

| argument         | range              |
|------------------|--------------------|
| n_estimator      | 100 - 1500      |
| max_depth        | 10, 20, 30 |
| max_features     | default: "sqrt"    |
| min_samples_leaf | default: 1         |

#### Evaluation metric
We use the function [f1 score](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html) to evaluate the performance of classifier because it considers both metrics **accuracy** and **recall**.

#### Note
In this part we don't use cross validation during grid search. Based on the requirement of 80% training set and 10% test set, we need to perform 9-fold cross-validation and it would take a huge amount of time when n_estimators and max_depth are large. Our concern is the execution time, so we use same validation set and training set to evaluate the metric. Overfitting can actually be avoided by increasing n_estimators, and therefore cross validation is not really necessary here.

#### Grid search

```python
# variables to store the best parameters and model
best_fscore_tr = fscore = best_fscore = best_n_estimators = best_max_depth = 0

n1 = 15
n2 = 3

n_estimators_arr = [i*100 + 100 for i in range(n1)]
max_depth_arr = [i*10 + 10 for i in range(n2)]
fscore_test_array = np.zeros([n1,n2])
fscore_train_array = np.zeros([n1,n2])

# grid search the best parameters
for i, n_estimators in enumerate(n_estimators_arr):
    for j, max_depth in enumerate(max_depth_arr):

        # build the model and train
        clf = RandomForestClassifier(n_estimators=n_estimators,
                                     max_depth=max_depth,
                                     n_jobs=4)
        clf.fit(x_train, y_train)

        # test the performance with validation set
        # 'macro' means calculating metrics for each label, and finding their unweighted mean
        fscore = fscore_test_array[i,j] = f1_score(y_val, clf.predict(x_val), average = 'macro')
        fscore_train_array[i,j] = f1_score(y_train, clf.predict(x_train), average = 'macro')

        # record the best one
        if fscore > best_fscore:
            best_fscore = fscore
            best_fscore_tr = fscore_train_array[i,j]
            best_n_estimators = n_estimators
            best_max_depth = max_depth
            best_model = clf

print('best n_estimators            :', n_estimators)
print('best max_depth               :', max_depth)
print('best training f1 score       :', best_fscore_tr)
print('best validation f1 score     :', best_fscore)
print('test f1 score                :', f1_score(y_test, best_model.predict(x_test), average = 'macro'))

```

    best n_estimators            : 1500
    best max_depth               : 30
    best training f1 score       : 0.963655927926
    best validation f1 score     : 0.82110746907
    test f1 score                : 0.829473569572

#### Learning curve
Let's see how f1 score in test set and training set move with n_estimator and max_depth

```python
f, (ax1, ax2) = plt.subplots(1,2, sharex=True, sharey=True)
f.set_figwidth(12)
ax1.plot(n_estimators_arr, fscore_test_array[:,0], label = 'test accuracy')
ax1.plot(n_estimators_arr, fscore_train_array[:,0], label = 'train accuracy')
ax1.set_xlabel('n_estimators')
ax1.set_ylabel('f1 score')
ax1.set_title('comparison of train accuracy and test accuracy')
ax1.legend()

ax2.plot(n_estimators_arr, fscore_test_array[:,0], label = 'max_depth = 10')
ax2.plot(n_estimators_arr, fscore_test_array[:,1], label = 'max_depth = 20')
ax2.plot(n_estimators_arr, fscore_test_array[:,2], label = 'max_depth = 30')
ax2.set_xlabel('n_estimators')
ax2.set_ylabel('f1 score')
ax2.set_title('comparison of different max_depth')
ax2.legend()

plt.show()
```

Comment: We could see usually the higher the n_estimator and max_depth are, the better the accuracy is. However, the accuracy is only improved a little bit when n_estimator is larger than 400.

#### Confusion matrix
By definition from [sklearn](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html)

Confusion matrix $C$ is such that $C_{ij}$ is equal to to the number of observations known to be in group $i$ but predicted to be in group $j$. Hence for each category i:

####  $accuracy = {{C_{ii}}\over{\sum_{j}C_{ij}}}, recall = {{C_{ii}}\over{\sum_{j}C_{ji}}}$

Now, let's plot confusion matrix to visualize the performance of our model.

```python
# create confusion matrix
sns.set(style="dark")
cnf_matrix = confusion_matrix(y_test, best_model.predict(x_test))

def plot_confusion_matrix(cm, classes,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """Function to plot confusion matrix
    Args:
        cm          : confusion matrix
        classes     : array storing the name of categories
        camp        : color bar of confusion matrix
    """

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=90)
    plt.yticks(tick_marks, classes)

    fmt = 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(i, j, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')

# plot the confusion matrix
plt.figure(figsize=(12,12))
plot_confusion_matrix(cnf_matrix,
                      classes=newsgroups.target_names,
                      title='Confusion matrix')
plt.show()
```

Comment: We can see if we use f1 score as our metric, usually both accracy and recall are low because f1 score is the harmonic mean of them. Here we can only see the classification of misc.forsale and comp.os.ms-windows-misc are worse in accuracy

### Model assessment

#### Cross-validation
To assess our model based on best n_estimator and max_depth got from grid search, we perform 10-fold cross validation on our best model. Note that this time we use all the data to train random forest because we are sure that the parameters we choose do not cause overfit (i.e. test accuracy and train accuracy are nearly the same in previous result). Hence, the more the data, the better the model is.

```python
def build_k_indices(y, k_fold):
    """Function to build k indices for k-fold.

    Args:
        y      (numpy array): Matrix output of size N x 1.
        k_fold (int)        : The value k, of k-fold cross validation.
        seed   (int)        : Integer value to seed the random generator.

    Returns:
        k_indices (numpy array) : Matrix of K x floor(N/K), as indices for unbiased test error of K-cross Validation.
    """
    seed = 1
    num_row    = y.shape[0]
    interval   = int(num_row / k_fold)
    np.random.seed(seed)
    indices    = np.random.permutation(num_row)
    k_indices  = np.array([indices[k * interval: (k + 1) * interval]
                 for k in range(k_fold)])
    return k_indices

def data_split(y, x, k_indices, k):
    """Function to split training data and test data using K-cross validation based on given k_indices.

    Args:
        y         (numpy array): Matrix output of size N x 1.
        tx        (numpy array): Matrix input of size N x D.
        k_indices (numpy array): Matrix of K x floor(N/K), as indices for unbiased test error of K-cross Validation.
        k         (int)        : For choosing index in the k_indices

    Returns:
        x_train (numpy array) : Matrix input (features) for training set with size of (ratio*N) x D.
        y_train (numpy array) : Matrix output for training set with size of (ratio*N) x 1.
        x_test  (numpy array) : Matrix input (features) for testing set with size of ((1-ratio)*N) x D.
        y_test  (numpy array) : Matrix output for testing set with size of ((1-ratio)*N) x 1.
    """

    train_set_indices               = np.ones(x.shape[0], dtype = bool)
    train_set_indices[k_indices[k]] = False

    x_train = x[train_set_indices]
    y_train = y[train_set_indices]

    x_test  = x[k_indices[k]]
    y_test  = y[k_indices[k]]

    return x_train, y_train, x_test, y_test

# build validation set
k_fold = 10
k_indices = build_k_indices(newsgroups.target, k_fold)

best_n_estimators = 1500
best_max_depth = 30

# best classifier from grid search
clf = RandomForestClassifier(n_estimators=best_n_estimators,
                             max_depth=best_max_depth,
                             n_jobs=4)

accuracy_arr = np.zeros(k_fold)
para_ = []
# perform 10-fold cross validation
for k in range(k_fold):
    x_train, y_train, x_test, y_test = data_split(y = y, x = x, k_indices=k_indices, k=k)
    clf.fit(x_train, y_train)
    accuracy_arr[k] = f1_score(y_test, clf.predict(x_test), average = 'macro')
    para_.append(clf.feature_importances_)
print('average f1 score = ', np.mean(accuracy_arr))
```

    average f1 score =  0.815357300641

#### Observe feature_importance
We can see there're lots of useless features (feature_importance = 0)

And we observe the value of non-zero features in log scale and plot it in histogram. Based on the histogram, we could see the majority of feature importance is really small, and we think there're two reasons. The first reason is that there exists lots of useless features, and the second reason is that there're too many features and makes the importance of each feature decrease.

```python
sns.set(style="whitegrid")
feature_importance = np.mean(para_, axis=0)

print('total number of features:', feature_importance.shape[0])
# number of 0's in the feature importance
print('number of features with importance:', np.sum(feature_importance == 0))

# let's see features with non-zero value in log scale
plt.hist(np.log(feature_importance[feature_importance != 0]), bins = 25)
plt.show()
```