<p align="center">

</p>

## Interactive Confidence Interval Demonstration

### Boostrap and Analytical Confidence Intervals

* we calculate the confidence interval in the mean with boostrap and compare to the analytical expression

* with this workflow we all provide an interactive plot demonstration with matplotlib and ipywidget packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Confidence Intevals

Confidence intervals are the uncertainty in a sample statistic or model parameter

* for uncertainty in the sample mean we have:

    * center on the sample proportion, $\hat{p}$

    * the standard error in the proportion for the dispersion (spread)

    * Student's t distributed for small samples and Gaussian distributed for large sample sizes

The analytical form for small samples ($n \lt 30$) is:

\begin{equation}
CI: \hat{p} \pm t_{\frac{\alpha}{2},n-1} \times \frac {\sqrt{p(1-p)}}{\sqrt{n}}
\end{equation}

where the sampling distribution of the proportion is student's t distributed with number of samples, $n$ - 1, degrees of freedom and $\alpha$ is the signficance level divded by 2 for the two tails.

When the number of samples is large ($n \ge 30$) then the analytical form converges to Gaussian distributed:

\begin{equation}
CI: \hat{p} \pm N_{\frac{\alpha}{2}} \times \frac {\sqrt{p(1-p)}}{\sqrt{n}}
\end{equation}

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out and compare the result to the analytical form of the confidence interval for the sample mean.

#### Objective

Provide an example and demonstration for:

1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets
2. provide an intuitive hands-on example of confidence intervals and compare to statistical boostrap

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

#### Load the Required Libraries

The following code loads the required libraries.

```python
%matplotlib inline
from ipywidgets import interactive                        # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
import matplotlib.pyplot as plt                           # plotting
from matplotlib.ticker import (MultipleLocator, AutoMinorLocator) # control of axes ticks
plt.rc('axes', axisbelow=True)                            # set axes and grids in the background for all plots
import numpy as np                                        # working with arrays
import pandas as pd                                       # working with DataFrames
import seaborn as sns                                     # for matrix scatter plots
from scipy.stats import triang                            # parametric distributions
from scipy.stats import binom
from scipy.stats import norm
from scipy.stats import uniform
from scipy.stats import triang
from scipy.stats import t
from scipy import stats                                   # statistical calculations
import random                                             # random drawing / bootstrap realizations of the data
from matplotlib.gridspec import GridSpec                  # nonstandard subplots
import math                                               # square root operator
```

#### Make a Synthetic Dataset

This is an interactive method to:

* select a parametric distribution

* select the distribution parameters

* select the number of samples and visualize the synthetic dataset distribution

```python
# parameters for the synthetic dataset
bins = np.linspace(0,1000,1000)

# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                                Simple Boostrap Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

a = widgets.IntSlider(min=0, max = 100, value = 2, step = 1, description = '$n_{red}$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
a.style.handle_color = 'red'

b = widgets.IntSlider(min=0, max = 100, value = 3, step = 1, description = '$n_{green}$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
b.style.handle_color = 'green'

c = widgets.IntSlider(min=1, max = 16, value = 3, step = 1, description = '$L$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
c.style.handle_color = 'gray'

ui = widgets.HBox([a,b,c],)                               # basic widget formatting
ui2 = widgets.VBox([l,ui],)

def f_make(a, b, c):                                      # function to take parameters, make sample and plot
    red_freq = make_data(a, b, c)

    labels = ['Red', 'Green']
    nrows = np.round(np.sqrt(c)+0.4,0); ncols = np.round(c / nrows + 0.4,0)
    plt.clf()

    for i in range(0, c):
        plt.subplot(ncols,nrows,i + 1)
        draw = [red_freq[i],a + b - red_freq[i]]
        plt.grid(zorder=0, color='black', axis = 'y', alpha = 0.2); plt.ylim(0,a + b);
        plt.ylabel('Frequency'); plt.xlabel('Balls Drawn')
        plt.yticks(np.arange(0,a + b + 1,max(1,round((a+b)/10))))
        barlist = plt.bar(labels,draw,edgecolor = "black",linewidth = 1,alpha = 0.8); plt.title('Realization #' + str(i+1),zorder = 1)
        barlist[0].set_color('r'); barlist[1].set_color('g')

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2 * nrows, wspace=0.2, hspace=0.2)
    plt.show()

def make_data(a, b, c):                                   # function to check parameters and make sample
    prop_red = np.zeros(c)
    for i in range(0, c):
        prop_red[i] = np.random.multinomial(a+b,[a/(a+b),b/(a+b)], size = 1)[0][0]
    return prop_red

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'a': a, 'b': b, 'c': c})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

### Simple Bootstrap Demonstration - Drawing Red and Green Balls from a Virtual Cowboy Hat

* drawing red and green balls from a hat with replacement to access uncertainty in the proportion

* interactive plot demonstration with ipywidget, matplotlib packages

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Let's simulate bootstrap, resampling with replacement from a hat with $n_{red}$ and $n_{green}$ balls

* **$n_{red}$**: number of red balls in the sample (placed in the hat)

* **$n_{green}$**: number of green balls in the sample (placed in the hat)

* **$L$**: number of bootstrap realizations

```python
display(ui2, interactive_plot)                            # display the interactive plot
```

    VBox(children=(Text(value='                                Simple Boostrap Demonstration, Michael Pyrcz, Assoc…

    Output()

#### Summarizing Bootstrap Uncertainty

* Run more bootstrap realizations and evaluate the uncertianty model

Now instead of looking at each bootstrap result, let's make many and summarize with:

* **box and whisker plot** of the red and green ball frequencies

* **histograms** of the red and green ball frequencies.

```python
# parameters for the synthetic dataset
bins = np.linspace(0,1000,1000)

# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l2 = widgets.Text(value='     Confidence Interval for Proportions, Analytical and Bootstrap Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))

a2 = widgets.IntSlider(min=0, max = 100, value = 20, step = 1, description = '$n_{red}$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
a2.style.handle_color = 'red'

b2 = widgets.IntSlider(min=0, max = 100, value = 30, step = 1, description = '$n_{green}$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
b2.style.handle_color = 'green'

c2 = widgets.IntSlider(min=5, max = 1000, value = 1000, step = 1, description = '$L$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
c2.style.handle_color = 'gray'

alpha = widgets.FloatSlider(min=0.01, max = 0.40, value = 0.05, step = 0.01, description = r'$\alpha$',orientation='horizontal',layout=Layout(width='400px', height='20px'),continuous_update=False)
alpha.style.handle_color = 'gray'

uib = widgets.HBox([a2,b2,c2,alpha],)                            # basic widget formatting
uib2 = widgets.VBox([l2,uib],)

def s_make(a, b, c, alpha):                                      # function to take parameters, make sample and plot

    dof = a + b - 1

    red_freq = make_data(a, b, c)
    pred = red_freq/(a+b)
    red_prop = (a / (a+b))
    red_SE = math.sqrt((red_prop * (1.0 - red_prop)) / (a+b))

    green_freq = (a + b) - red_freq
    pgreen = green_freq/(a+b)
    green_prop = (b / (a+b))
    green_SE = math.sqrt((green_prop * (1.0 - green_prop)) / (a+b))

    prop_red = red_freq / (a + b)
    prop_green = green_freq / (a + b)
    labels = ['Red Balls', 'Green Balls']
    bins = np.linspace(0,a + b, a + b)

    fig = plt.figure(constrained_layout=False)
    gs = GridSpec(3, 2, figure=fig)

    ax1 = fig.add_subplot(gs[:, 0])
    boxplot = ax1.boxplot([pred,pgreen],labels = labels, notch = True, sym = '+',patch_artist=True)
    colors = ['red','green']
    for patch, color in zip(boxplot['boxes'], colors):
        patch.set_facecolor(color)
    for patch, color in zip(boxplot['medians'], colors):
        patch.set_color('black')

    ax1.set_ylim([0,1])
    ax1.grid(zorder=0, color='black', axis = 'y', alpha = 0.2)
    ax1.set_ylabel('Proportion of Balls'); ax1.set_xlabel('Ball Color');ax1.set_title('Bootstrap Uncertainty - Proportion Distributions')
    ax1.grid(True, which='major',axis='y',linewidth = 1.0); ax1.grid(True, which='minor',axis='y',linewidth = 0.2) # add y grids
    ax1.tick_params(which='major',length=7); ax1.tick_params(which='minor', length=4)
    ax1.xaxis.set_minor_locator(AutoMinorLocator()); ax1.yaxis.set_minor_locator(AutoMinorLocator()) # turn on minor ticks

    cumul_prob = np.linspace(0.0,1.0,100)
    if a <= 30 or b <= 30:
        red_prop_values = t.ppf(cumul_prob, dof)
        red_lower = t.ppf(alpha/2, dof); red_upper = t.ppf(1-alpha/2, dof)
    else:
        red_prop_values = norm.ppf(cumul_prob)
        red_lower = norm.ppf(alpha/2); red_upper = norm.ppf(1-alpha/2)

    red_prop_values = red_prop_values * red_SE + red_prop
    red_lower = red_lower * red_SE + red_prop
    red_upper = red_upper * red_SE + red_prop

    cumul_prob = np.linspace(0.01,0.99,100)
    if a <= 30 or b <= 30:
        green_prop_values = t.ppf(cumul_prob, dof)
        green_lower = t.ppf(alpha/2, dof); green_upper = t.ppf(1-alpha/2, dof)
    else:
        green_prop_values = norm.ppf(cumul_prob)
        green_lower = norm.ppf(alpha/2); green_upper = norm.ppf(1-alpha/2)

    green_prop_values = green_prop_values * green_SE + green_prop
    green_lower = green_lower * green_SE + green_prop
    green_upper = green_upper * green_SE + green_prop

    ax2 = fig.add_subplot(gs[0, 1])
    ax2.hist(prop_red,cumulative = True, density = True, alpha=0.7,color="red",edgecolor="black",linewidth=2,bins = np.linspace(0,1,50), label = 'Bootstrap')
    ax2.plot([red_lower,red_lower],[0,1],color='black',linewidth=2,linestyle='--',label='Lower/Upper')
    ax2.plot([red_upper,red_upper],[0,1],color='black',linewidth=2,linestyle='--')
    ax2.plot([red_prop,red_prop],[0,1],color='black',linewidth=3,label='Exp.')
    ax2.set_title('Uncertainty in Proportion of Red Balls'); ax2.set_xlabel('Proportion of Red Balls'); ax2.set_ylabel('Cumulative Probability')
    ax2.set_xlim([0,1]); ax2.set_ylim([0,1])

    ax2.plot(red_prop_values, cumul_prob, color = 'black', linewidth = 2, label = 'Analytical')
    ax2.legend()

    ax3 = fig.add_subplot(gs[1, 1])
    ax3.hist(prop_green,cumulative = True, density = True, alpha=0.7,color="green",edgecolor="black",linewidth=2,bins = np.linspace(0,1,50), label = 'Bootstrap')
    ax3.plot([green_lower,green_lower],[0,1],color='black',linewidth=2,linestyle='--',label='Lower/Upper')
    ax3.plot([green_upper,green_upper],[0,1],color='black',linewidth=2,linestyle='--')
    ax3.plot([green_prop,green_prop],[0,1],color='black',linewidth=3,label='Exp.')
    ax3.set_title('Uncertainty in Proportion of Green Balls'); ax3.set_xlabel('Proportion of Green Balls'); ax3.set_ylabel('Cumulative Probability')
    ax3.set_xlim([0,1]); ax3.set_ylim([0,1])

    ax3.plot(green_prop_values, cumul_prob, color = 'black', linewidth = 2, label = 'Analytical')
    ax3.legend()

    ax4 = fig.add_subplot(gs[2, 1])
    ax4.hist(prop_green,cumulative = False, density = True, alpha=0.7,color="green",edgecolor="black",linewidth=2, bins = np.linspace(0,1,50), label = 'Bootstrap Prop. Green')
    ax4.hist(prop_red,cumulative = False, density = True, alpha=0.7,color="red",edgecolor="black",linewidth=2, bins = np.linspace(0,1,50), label = 'Bootstrap Prop. Red')

    ax4.set_title('Confidence Interval in Proportion of Red and Green Balls (Alpha = ' + str(alpha) + ')'); ax3.set_xlabel('Proportion of Green Balls')
    ax4.set_xlabel('Proportion of Red and Green Balls'); ax4.set_ylabel('Frequency')
    ax4.set_xlim([0,1])

    prop_values = np.linspace(0.0,1.0,100)
    if a <= 30 and b <= 30:
        green_density = t.pdf(prop_values,loc = green_prop, df = dof, scale = green_SE)
    else:
        green_density = norm.pdf(prop_values,loc = green_prop, scale = green_SE)
    ax4.plot(prop_values, green_density, color = 'black', linewidth = 5,zorder=99)
    ax4.plot(prop_values, green_density, color = 'green', linewidth = 3, label = 'Analytical Prop. Green',zorder=100)

    if a <= 30 and b <= 30:
        red_density = t.pdf(prop_values,loc = red_prop, df = dof, scale = red_SE)
    else:
        red_density = norm.pdf(prop_values,loc = red_prop, scale = red_SE)
    ax4.plot(prop_values, red_density, color = 'black', linewidth = 5,zorder=99)
    ax4.plot(prop_values, red_density, color = 'red', linewidth = 3, label = 'Analytical Prop. Red',zorder=100)

    ax4.fill_between(prop_values, 0, green_density, where = prop_values <= green_lower, facecolor='green', interpolate=True, alpha = 0.9,zorder=101)
    ax4.fill_between(prop_values, 0, green_density, where = prop_values >= green_upper, facecolor='green', interpolate=True, alpha = 0.9,zorder=101)
    ax4.fill_between(prop_values, 0, red_density, where = prop_values <= red_lower, facecolor='darkred', interpolate=True, alpha = 0.9,zorder=101)
    ax4.fill_between(prop_values, 0, red_density, where = prop_values >= red_upper, facecolor='darkred', interpolate=True, alpha = 0.9,zorder=101)
    ax4.legend()

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.5, top=3.0, wspace=0.2, hspace=0.3)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(s_make, {'a': a2, 'b': b2, 'c': c2, 'alpha': alpha})
interactive_plot.clear_output(wait = True)                # reduce flickering by delaying plot updating
```

### Simple Bootstrap and Analytical Confidence Interval Demonstration for Sample Proportions

* drawing red and green balls from a hat with replacement to access uncertainty in the proportion

* run many bootstrap realizations and summarize the results and compare to the analytical sampling distribution for the proportion

* interactive plot demonstration with ipywidget, matplotlib packages

#### Michael Pyrcz, Associate Professor, The University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Let's simulate bootstrap, resampling with replacement from a hat with $n_{red}$ and $n_{green}$ balls

* **$n_{red}$**: number of red balls in the sample (placed in the hat)

* **$n_{green}$**: number of green balls in the sample (placed in the hat)

* **$L$**: number of bootstrap realizations

* **$\alpha$**: alpha level for the confidence interval (significance level)

and then compare the uncertainty in the proportion of balls to the analytical expression.

### Confidence Interval Demonstration for Sample Proportions, Analytical and Bootstrap

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

```python
display(uib2, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='     Confidence Interval for Proportions, Analytical and Bootstrap Demonstration, M…

    Output()

#### Observations

Some observations:

* sampling distribution for proportions become discrete with too few samples, as only $n$ cases are possible

* enough bootstrap realizations are required for stable statistics

* the analytical sampling distribution for the uncertainty in the sample proportion matches the results from bootstrap

#### Comments

This was a simple demonstration of interactive plots in Jupyter Notebook Python with the ipywidgets and matplotlib packages.

I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


# Structural Estimation

1. This notebook shows how to **estimate** the consumption model in **ConsumptionSaving.pdf** using **Simulated Minimum Distance (SMD)**
2. It also shows how to calculate **standard errors** and **sensitivity measures**

## Simulated Minimum Distance

**Data:** We assume that we have data available for $N$ households over $T$ periods, collected in $\{w_i\}_i^N$.

**Goal:** We wish to estimate the true, unknown, parameter vector $\theta_0$. We assume our model is correctly specified in the sense that the observed data stems from the model.

**Overview:**

1. We focus on matching certain (well-chosen) **empirical moments** in the data to **simulated moments** from the model.

2. We calculate a $J\times1$ vector of moments in the data, $\Lambda_{data} = \frac{1}{N}\sum_{i=1}^N m(\theta_0|w_i)$. This could e.g. be average consumption over the life-cycle, the income variance or regressions coefficients from some statistical model.

3. To estimate $\theta$ we chose $\theta$ as to **minimize the (squared) distance** between the moments in the data and the same moments calculated from simulated data. Let $\Lambda_{sim}(\theta) = \frac{1}{N_{sim}}\sum_{s=1}^{N_{sim}} m(\theta|w_s)$ be the same moments calculated on simulated data for $N_{sim}=S\times N$ observations for $T_{sim}$ periods from the model for a given value of $\theta$. As we change $\theta$, the simulated outomes will change and the moments will too.

The **Simulated Minimum Distance (SMD)** estimator then is

$$
\hat{\theta} = \arg\min_{\theta} g(\theta)'Wg(\theta)
$$

where $W$ is a $J\times J$ positive semidefinite **weighting matrix** and

$$
g(\theta)=\Lambda_{data}-\Lambda_{sim}(\theta)
$$

is the distance between $J\times1$ vectors of moments calculated in the data and the simulated data, respectively. Concretely,

$$
\Lambda_{data} = \frac{1}{N}\sum_{i=1}^N m(\theta_0|w_i) \\
\Lambda_{sim}(\theta) = \frac{1}{N_{sim}}\sum_{s=1}^{N_{sim}} m(\theta|w_s)
$$

are $J\times1$ vectors of moments calculated in the data and the simulated data, respectively.

**Settings:** In our baseline setup, we will have $N=5,000$ observations for $T=40$ periods, and simulate $N_{sim}=100,000$ synthetic consumers for $T_{sim} = 40$ periods when estimating the model.

**Solution of consumption-saving model:** This estimator requires the solution (and simulation) of the model each trial guess of $\theta$ as we search for the one that minimizes the objective function. Therefore, structural estimation can in general be quite time-consuming. We will use the EGM to solve the consumption model quite fast and thus be able to estimate parameters within a couple of minutes. Estimation of more complex models might take significantly longer.

> **Note I:** When regressions coefficients are used as moments, they are sometimes referred to as **auxiliary parameters** (APs) and the estimator using these APs as an **Indirect Inference (II)** estimator ([Gouriéroux, Monfort and Renault, 1993](https://onlinelibrary.wiley.com/doi/abs/10.1002/jae.3950080507)).

> **Note II:** The estimator used is also called a **simulated method of momoments (SMM)** estimator. I.e. a simulated General Method of Moments (GMM) estimator.

# Setup

```python
%matplotlib inline
%load_ext autoreload
%autoreload 2

import time
import numpy as np
import scipy.optimize as optimize

import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
prop_cycle = plt.rcParams['axes.prop_cycle']
colors = prop_cycle.by_key()['color']

import figs
from ConsumptionSavingModel import ConsumptionSavingModelClass
from SimulatedMinimumDistance import SimulatedMinimumDistanceClass
```

# Estimation choices

```python
# a. model settings
N = 5_000
N_sim = 100_000

par = {'simlifecycle':True,
       'sim_mini':1.0 ,
       'simT':40,
       'simN':N_sim,
       'Nxi':4,
       'Npsi':4,
       'Na':100}

par_true = par.copy()
par_true['simN'] = N

# b. parameters to estimate
est_par = {
    'rho': {'guess':2.0,'lower':0.5,'upper':5.0,},
    'beta': {'guess':0.97,'lower':0.90,'upper':0.999},
}

est_par_names = [key for key in est_par.keys()]

# c. moment function used in estimation.
def mom_func(data,ids=None):
    """ returns the age profile of wealth """

    if ids is None:
        mean_A = np.mean(data.A[:,1:],axis=0)
    else:
        mean_A = np.mean(data.A[ids,1:],axis=0)

    return mean_A

# d. choose weighting matrix
weighting_matrix = 0

# 0: identity (equal weight),
# 1: inverse of variance on the diagonal (removes scale),
# 2: inverse of covaraince matrix between estimation moments (optimal weighting matrix)
```

# Data and estimator

Construct **data**.

```python
# a. setup model to simulate data
true = ConsumptionSavingModelClass(name='true',solmethod='egm',**par_true)
true.solve()
true.simulate(seed=2019) # this seed is different from the default

# b. data moments
datamoms = mom_func(true.sim)
moment_names = [i for i in range(true.par.age_min+1,true.par.age_min+true.par.simT)]
```

    model solved in 1.3 secs
    model simulated in 1.3 secs

**Bootstrap** variance of estimation moments used when later calculting standard errors below (and potentially for weighting matrix).

```python
num_boot = 200
num_moms = datamoms.size
smd = SimulatedMinimumDistanceClass(est_par,mom_func,datamoms=datamoms)
smd.Omega = smd.bootstrap_mom_var(true.sim,N,num_boot,num_moms)
```

**Setup estimator**.

```python
smd.plot({'data':moment_names},{'data':datamoms},xlabel='age',ylabel='wealth',hide_legend=True)
```

# Estimate the model

```python
model = ConsumptionSavingModelClass(name='estimated',solmethod='egm',**par)
```

Choose **weighting matrix**:

```python
if weighting_matrix == 0:
    W = np.eye(smd.datamoms.size) # identity
elif weighting_matrix == 1:
    W = np.diag(1.0/np.diag(smd.Omega)) # inverse of variance on the diagonal
else:
    W = np.linalg.inv(smd.Omega) # optimal weighting matrix
```

## Estimation results

```python
# a. estimate the model (can take several minutes)
%time est = smd.estimate(model,W)

# b. print estimation results
print(f'\n       True  Est. ')
for key in est_par.keys():
    print(f'{key:5s} {getattr(true.par,key):2.3f} {est[key]:2.3f}')
```

    objective function at starting values: 5.961003893464155
    Wall time: 47.7 s

           True  Est.
    rho   2.000 2.018
    beta  0.960 0.960

Show **model-fit**:

```python
plot_data_x = {'data':moment_names,'simulated':moment_names}
plot_data_y = {'data':datamoms,'simulated':mom_func(model.sim)}
smd.plot(plot_data_x,plot_data_y,xlabel='age',ylabel='wealth')
```

## Standard errors

The SMD estimator is **asymptotic Normal** and standard errors have the same form as standard GMM estimators scaled with the adjustment factor $(1+S^{-1})$ due to the fact that we use $S$ simulations of the model.

The **standard errors** are thus

$$
\begin{align}
\text{Var}(\hat{\theta})&=(1+S^{-1})\Gamma\Omega\Gamma'/N \\
\Gamma &= -(G'WG)^{-1}G'W \\
\Omega & = \text{Var}(m(\theta_0|w_i))
\end{align}
$$

where $G=\frac{\partial g(\theta)}{\partial \theta}$ is the $J\times K$ **Jacobian** with respect to $\theta$. $\Gamma$ is related to what is sometimes called the "influence function".

**Calculating $\Omega$**:

1. Can sometimes be done **analytically**
2. Can always be done using a **bootstrap** as done above

**Calculating the Jacobian, $G$:** This is done using numerical finite differences.

```python
# a. number of datasets simulated per individual in original data
S = model.par.simN/N

# b. find standard errors
Gamma, grad_theta = smd.calc_influence_function(est['theta'],model,W)
Var_theta = (1.0+1.0/S) * Gamma @ smd.Omega @ Gamma.T /N
se = np.sqrt(np.diag(Var_theta))

# b. print estimation results
print(f'       True  Est.  (se)')
for i,(key,val) in enumerate(est_par.items()):
    print(f'{key:5s} {getattr(true.par,key):2.3f} {est[key]:2.3f} ({se[i]:2.3f})')
```

           True  Est.  (se)
    rho   2.000 2.018 (0.043)
    beta  0.960 0.960 (0.001)

# Sensitivity Analysis

We now look into a **sensitivity analysis** of our estimation. Concretely, we implement the **informativeness measure** from [Honoré, Jørgensen and de Paula (2019)](https://arxiv.org/abs/1907.02101v2 "The Informativeness of Estimation Moments") and the **sensitivity to calibrated parameters** in [Jørgensen (2020)](https://www.dropbox.com/s/g8ip7h051dyhn3r/Sensitivity.pdf?dl=0). Further details can be found in these papers.

## The informativeness of estimation moments

The measures are motivated by those proposed in [Honoré, Jørgensen and de Paula (2019)](https://arxiv.org/abs/1907.02101v2 "The Informativeness of Estimation Moments"). All the measures proposed in that paper is calculated, but we will focus on their measure 4 that asks **"what is the change in the asymptotic variance from completely excluding the k'th moment?"**. If the *k*th is very informative about a parameter, the asymptotic varaince of that parameter should increase significantly, if we leave out the *k*th moment.

```python
info = smd.informativeness_moments(grad_theta,smd.Omega,W)
smd.plot_heat(info['M4e'],est_par_names,moment_names,annot=False)
```

**Conclusion:** We can see that especially the wealth level for younger households are very informative regarding both $\rho$ and $\beta$. This is likely due to the fact that for low level of resources (which is the case at younger ages), the value of both these parameters affect consumption and saving decisions a lot. Thus, the level of saving especially in young ages are very informative and help to identify the two parameters.

## Sensitivity to calibrated parameters

The mesure is motivated by the one proposed in [Jørgensen (2020)](www.tjeconomics.com "Sensitivity to Calibrated Parameters"). Note that the estimation moments are all functions of the $L$ calibrated parameters, which we will denote $\gamma$, $g(\theta|\gamma)$.

The **sensitivity measure** is defined as

$$
\begin{align}
S &= \Gamma D
\end{align}
$$

where $D=\frac{\partial g(\theta|\gamma)}{\partial \gamma}$ is the $J\times L$ **Jacobian** with respect to $\gamma$.

*We only need to calculate $D$* since we have already calculated $\Gamma$ when we calculated standard errors above. We use numerical finite differences to calcualte this object.

**Chosen calibrated paramters:** $R$, $G$, $\sigma_{\psi}$, $\sigma_{\xi}$.

```python
cali_par_names = ('R','G','sigma_psi','sigma_xi')
cali_par = np.array([getattr(model.par,name) for name in cali_par_names])
```

**Calculate the sensitivty measure:**

```python
grad_gamma = smd.num_grad(cali_par,model,cali_par_names)
sens_cali = Gamma @ grad_gamma
```

**Plot sensitivity measure**

```python
smd.plot_heat(sens_cali,est_par_names,cali_par_names)
```

**Check:** We can compare this to a brute-force approach in which we re-estimate the model for marginal changes in the calibrated parameters. This takes considerable time, however. The results are almost identical.

```python
sens_cali_brute = smd.sens_cali_brute_force(model,est['theta'],W,cali_par_names)
smd.plot_heat(sens_cali_brute,est_par_names,cali_par_names)
```

**Arbitrary changes in $\gamma$**: We can also investigate larger simultaneous changes in $\gamma$.

```python
# a. set new calibrated parameters
cali_par_new = {'G':1.05}

# b. update calibrated parameters in new version of the model
model_new = model.copy()
for key,val in cali_par_new.items():
    setattr(model_new.par,key,val)

# c. calculate new objective function
obj_vec = smd.diff_vec_func(est['theta'],model,est_par_names)
obj_vec_new = smd.diff_vec_func(est['theta'],model_new,est_par_names)

# d. approximate change in theta
Gamma_new,_ = smd.calc_influence_function(est['theta'],model_new,W)
theta_delta =  Gamma_new @ obj_vec_new - Gamma @ obj_vec

# e. extrapolate the gradient
theta_delta_extrap = np.zeros(theta_delta.size)
for j,key in enumerate(cali_par_new):
    theta_delta_extrap += sens_cali[:,j]*(cali_par_new[key]-getattr(model.par,key))

print(theta_delta_extrap)
```

    [ 0.17300506 -0.02961878]

**Check:** Again, we can compare this approximation to a brute-force re-estimation of the model for the changed $\gamma$.

```python
est_new = smd.estimate(model_new,W)
theta_delta_brute = est_new['theta'] - est['theta']
print(theta_delta_brute)
```

    objective function at starting values: 5.463540643940908
    [ 0.11275468 -0.05969197]


### Lab 3: Expectation Maximization and Variational Autoencoder

### Machine Learning 2 (2017/2018)

* The lab exercises should be made in groups of two or three people.
* The deadline is Friday, 01.06.
* Assignment should be submitted through BlackBoard! Make sure to include your and your teammates' names with the submission.
* Attach the .IPYNB (IPython Notebook) file containing your code and answers. Naming of the file should be "studentid1\_studentid2\_lab#", for example, the attached file should be "12345\_12346\_lab1.ipynb". Only use underscores ("\_") to connect ids, otherwise the files cannot be parsed.

Notes on implementation:

* You should write your code and answers in an IPython Notebook: http://ipython.org/notebook.html. If you have problems, please ask.
* Use __one cell__ for code and markdown answers only!
    * Put all code in the cell with the ```# YOUR CODE HERE``` comment and overwrite the ```raise NotImplementedError()``` line.
    * For theoretical questions, put your solution using LaTeX style formatting in the YOUR ANSWER HERE cell.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Large parts of you notebook will be graded automatically. Therefore it is important that your notebook can be run completely without errors and within a reasonable time limit. To test your notebook before submission, select Kernel -> Restart \& Run All.
$\newcommand{\bx}{\mathbf{x}} \newcommand{\bpi}{\mathbf{\pi}} \newcommand{\bmu}{\mathbf{\mu}} \newcommand{\bX}{\mathbf{X}} \newcommand{\bZ}{\mathbf{Z}} \newcommand{\bz}{\mathbf{z}}$

### Installing PyTorch

In this lab we will use PyTorch. PyTorch is an open source deep learning framework primarily developed by Facebook's artificial-intelligence research group. In order to install PyTorch in your conda environment go to https://pytorch.org and select your operating system, conda, Python 3.6, no cuda. Copy the text from the "Run this command:" box. Now open a terminal and activate your 'ml2labs' conda environment. Paste the text and run. After the installation is done you should restart Jupyter.

### MNIST data

In this Lab we will use several methods for unsupervised learning on the MNIST dataset of written digits. The dataset contains digital images of handwritten numbers $0$ through $9$. Each image has 28x28 pixels that each take 256 values in a range from white ($= 0$) to  black ($=1$). The labels belonging to the images are also included.
Fortunately, PyTorch comes with a MNIST data loader. The first time you run the box below it will download the MNIST data set. That can take a couple of minutes.
The main data types in PyTorch are tensors. For Part 1, we will convert those tensors to numpy arrays. In Part 2, we will use the torch module to directly work with PyTorch tensors.

```python
%pylab inline
import torch
from torchvision import datasets, transforms

train_dataset = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ]))

train_labels = train_dataset.train_labels.numpy()
train_data = train_dataset.train_data.numpy()
# For EM we will use flattened data
train_data = train_data.reshape(train_data.shape[0], -1)

```

    Populating the interactive namespace from numpy and matplotlib

## Part 1: Expectation Maximization
We will use the Expectation Maximization (EM) algorithm for the recognition of handwritten digits in the MNIST dataset. The images are modelled as a Bernoulli mixture model (see Bishop $\S9.3.3$):
$$
p(\bx|\bmu, \bpi) = \sum_{k=1}^K  \pi_k \prod_{i=1}^D \mu_{ki}^{x_i}(1-\mu_{ki})^{(1-x_i)}
$$
where $x_i$ is the value of pixel $i$ in an image, $\mu_{ki}$ represents the probability that pixel $i$ in class $k$ is black, and $\{\pi_1, \ldots, \pi_K\}$ are the mixing coefficients of classes in the data. We want to use this data set to classify new images of handwritten numbers.

### 1.1 Binary data (5 points)
As we like to apply our Bernoulli mixture model, write a function `binarize` to convert the (flattened) MNIST data to binary images, where each pixel $x_i \in \{0,1\}$, by thresholding at an appropriate level.

```python
def binarize(X):
    return 1. * (X > 128)

```

```python
# Test test test!
bin_train_data = binarize(train_data)
assert bin_train_data.dtype == np.float
assert bin_train_data.shape == train_data.shape

```

Sample a few images of digits $2$, $3$ and $4$; and show both the original and the binarized image together with their label.

```python
for digit in [2,3,4]:

    fig = plt.figure(figsize=(15,10))
    index = np.where(train_labels == digit)[0][:2]

    plt.subplot(141)
    plt.imshow(train_data[index[0]].reshape(28,28),cmap='gray')
    plt.title("Label:" + str(train_labels[index[0]]) + ", Non Binarized")
    plt.subplot(142)
    plt.imshow(bin_train_data[index[0]].reshape(28,28),cmap='gray')
    plt.title("Label:" + str(train_labels[index[0]])+ ", Binarized")
    plt.subplot(143)
    plt.imshow(train_data[index[1]].reshape(28,28),cmap='gray')
    plt.title("Label:" + str(train_labels[index[1]]) + ", Non Binarized")
    plt.subplot(144)
    plt.imshow(bin_train_data[index[1]].reshape(28,28),cmap='gray')
    plt.title("Label:" + str(train_labels[index[1]])+ ", Binarized")
    plt.show()
```

### 1.2 Implementation (40 points)
You are going to write a function ```EM(X, K, max_iter)``` that implements the EM algorithm on the Bernoulli mixture model.

The only parameters the function has are:
* ```X``` :: (NxD) array of input training images
* ```K``` :: size of the latent space
* ```max_iter``` :: maximum number of iterations, i.e. one E-step and one M-step

You are free to specify your return statement.

Make sure you use a sensible way of terminating the iteration process early to prevent unnecessarily running through all epochs. Vectorize computations using ```numpy``` as  much as possible.

You should implement the `E_step(X, mu, pi)` and `M_step(X, gamma)` separately in the functions defined below. These you can then use in your function `EM(X, K, max_iter)`.

```python
def E_step(X, mu, pi):
    # YOUR CODE HERE
    N, D = shape(X)
    K = shape(pi)[0]

    gamma = np.zeros((N,K))
    for n in range(N):
        gamma_n = np.zeros(K)

        for k in range(K):
            p1_array = mu[k,:]**X[n,:]
            p0_array = (1-mu[k,:])**(1-X[n,:])
            p_array = p1_array * p0_array
            gamma_n[k] = pi[k]*np.prod(p_array)

        normalizing_denom_n = np.sum(gamma_n)
        gamma_n /= normalizing_denom_n
#         print(np.sum(gamma_n)) # SHOULD BE 1

        gamma[n] = gamma_n

    return gamma
```

```python
# Let's test on 5 datapoints
n_test = 5
X_test = bin_train_data[:n_test]
D_test, K_test = X_test.shape[1], 10

np.random.seed(2018)
mu_test = np.random.uniform(low=.25, high=.75, size=(K_test,D_test))
pi_test = np.ones(K_test) / K_test

gamma_test = E_step(X_test, mu_test, pi_test)
assert gamma_test.shape == (n_test, K_test)

```

```python
def M_step(X, gamma):
    # YOUR CODE HERE
    N, D = shape(X)
    K = shape(gamma)[1]

    Nk = np.sum(gamma, axis=0)
    pi = Nk/N
    mu = np.zeros((K, D))

    for k in range(K):
        mu[k] = (np.transpose(gamma[:,k]) @ X) / Nk[k]

    return mu, pi
```

```python
# Oh, let's test again
mu_test, pi_test = M_step(X_test, gamma_test)

assert mu_test.shape == (K_test,D_test)
assert pi_test.shape == (K_test, )

```

```python
def EM(X, K, max_iter, mu=None, pi=None):
    N, D = shape(X)
    convergence_value = 1e-5

    gamma = np.zeros((N, K))
    if mu is None:
        mu = np.random.uniform(low=.25, high=.75, size=(K, D))
    if pi is None:
        pi = np.ones(K) / K

    for i in range(1, max_iter):
        mu_prev, pi_prev = mu, pi

        gamma = E_step(X, mu, pi)
        mu, pi = M_step(X, gamma)

        mu_update = np.linalg.norm(mu-mu_prev)
        pi_update = np.linalg.norm(pi-pi_prev)
        if i%10 == 0:
            print("Iteration ", i, "| delta_mu = %.7f" % mu_update, "| delta_pi = %.7f" % pi_update, "|")

        if mu_update < convergence_value and pi_update < convergence_value:
            print("Convergence reached at iteration ", i, ", early stopping!")
            return gamma, mu, pi

    return gamma, mu, pi

# EXPERIMENT with 10 classes, TO BE REMOVED
# gamma, mu, pi = EM(bin_train_data[:10000, :], 10, 100) #Subset

# fig = plt.figure(figsize=(15,10))

# for k in range(9):
#     plt.subplot(191+k)
#     plt.imshow(mu[k].reshape(28,28),cmap='gray') #Greys swaps colors
#     plt.title("Weights for Class:"+str(k))

# plt.show()
```

### 1.3 Three digits experiment (10 points)
In analogue with Bishop $\S9.3.3$, sample a training set consisting of only __binary__ images of written digits $2$, $3$, and $4$. Run your EM algorithm and show the reconstructed digits.

```python
# YOUR CODE HERE

mask = np.isin(train_labels, [2,3,4])
reduced_train_labels = train_labels[mask]
reduced_train_data = bin_train_data[mask]
# print(shape(reduced_train_data))

gamma3, mu3, pi3 = EM(reduced_train_data, 3, 100)

fig = plt.figure(figsize=(15,10))
for k in range(3):
    plt.subplot(131+k)
    plt.imshow(mu3[k].reshape(28,28),cmap='gray') #Greys swaps colors
    plt.title("Weights for Class:"+str(k))
plt.show()

from scipy.stats import itemfreq
frequencies = itemfreq(reduced_train_labels)[:, 1].flatten()

print('Mixture Coefficients (pi): {}'.format(pi3))
print('Class-frequencies in the dataset:')
print('; '.join(['{}: {}'.format(d, frequencies[i]/len(reduced_train_labels)) for i, d in enumerate([2, 3, 4])]))

```

Can you identify which element in the latent space corresponds to which digit? What are the identified mixing coefficients for digits $2$, $3$ and $4$, and how do these compare to the true ones?

It is evident how different elements in latent space corresponds to the digits from different classes.
In the pictures above we show how, in different digits, the regions with active weight values correspond to the regions representing the shape of that particular digit.

The mixing coeffincients ($\pi$) follow roughly an uniform distribution over the 3 classes and resemble the distribution of the classes in the dataset

### 1.4 Experiments (20 points)
Perform the follow-up experiments listed below using your implementation of the EM algorithm. For each of these, describe/comment on the obtained results and give an explanation. You may still use your dataset with only digits 2, 3 and 4 as otherwise computations can take very long.

#### 1.4.1 Size of the latent space (5 points)
Run EM with $K$ larger or smaller than the true number of classes. Describe your results.

```python
K = 2
_, mu2, _ = EM(reduced_train_data, K, 100)

fig = plt.figure(figsize=(15,10))
for k in range(K):
    plt.subplot(121+k)
    plt.imshow(mu2[k].reshape(28,28),cmap='gray') #Greys swaps colors
    plt.title("Weights for Class:"+str(k))
plt.show()

K = 8
_, mu8, _ = EM(reduced_train_data, K, 100)

fig2 = plt.figure(figsize=(15,10))
for k in range(K):
    plt.subplot(181+k)
    plt.imshow(mu8[k].reshape(28,28),cmap='gray') #Greys swaps colors
    plt.title("Weights for Class:"+str(k))
plt.show()
```

When the number of latent classes K is lower than the true classes, some of the classes (the most similar ones) appear to be clustered together. For example, using K=2 for classes [2,3,4], the spaces for digits 2 and 3 (more similar) are merged together and result in some blurry weight matrix.

On the other way, when more latent classes are present with respect to the number of true classes, the spaces will represent digits from the same classes with different shapes. Namely, subcluster in the larger cluster corresponding to a digit are created and represented with different weights

#### 1.4.2 Identify misclassifications (10 points)
How can you use the data labels to assign a label to each of the clusters/latent variables? Use this to identify images that are 'misclassified' and try to understand why they are. Report your findings.

```python
from collections import defaultdict

def test_classification(data, labels, classes, assigned_classes, gamma):

    predictions = np.zeros(len(data), dtype=int)

    for i in range(len(data)):
        predictions[i] = assigned_classes[np.argmax(gamma[i, :])]

    misclassified_indexes = predictions != labels
    number_of_errors = sum(misclassified_indexes)

    print('{} misclassified images out of {}'.format(number_of_errors, len(data)))

    sampled_indexes = np.where(misclassified_indexes)[0]
    np.random.shuffle(sampled_indexes)

    rows = 5
    columns = 5

    sampled_indexes = sampled_indexes[:rows*columns]

    fig, axes = plt.subplots(rows, columns, figsize=(15, rows*4))

    for i in range(len(sampled_indexes)):

        ax = axes[i//columns, i % columns]

        ax.imshow(data[sampled_indexes[i]].reshape(28,28),cmap='gray') #Greys swaps colors
        ax.set_title("{} misclassified as {}".format(labels[sampled_indexes[i]], predictions[sampled_indexes[i]]))

    plt.show()

classes = [2, 3, 4]

scores = [defaultdict(float) for i in range(len(classes))]

for x, l in zip(reduced_train_data, reduced_train_labels):
    for k in range(len(classes)):
        #log_prob = sum(np.log(mu[k][x == 1])) + sum(np.log(1 - mu[k][x != 1]))
        #scores[k][l] += log_prob
        scores[k][l] += -sum((mu3[k] - x)**2)

scores = [dict(scores[i]) for i in range(len(classes))]

assigned_classes = [max(scores[i], key=scores[i].get) for i in range(len(classes))]

#     print(scores)
#     print(assigned_classes)

test_classification(reduced_train_data, reduced_train_labels, classes, assigned_classes, gamma3)

```

From the plots we are showing we see how recurrent characteristic and shapes for different digits cause them to be classified wrongly.

For example, we notice how digits "2" which are shifted upwards are often misclassified as "4". Probably this happens because the number "4" has a characteristic horizontal line in the middle, while usually "2" as the orizontal line in the bottom part.

Similarly, "3" digits that appear sheared/rotated assuming an oblungated shape from top-right to bottom-left are misclassified as "2", whose shape specifically contain a main line on the diagonal spanning top-right to bottom-left, with a rounded part on the top which is in common with the digit "3", and thus cannot be used to discriminate between the two classes.

On the other way, we realized that few digit "4" are misclassified. Why is it more common, for example, to have "4" classified as "2" then "2" as "4"? One explanation could be that "4" digit contain characteristical vertical lines which are never found in "2" and "3", in partiuclar the straight line at the top left.

#### 1.4.3 Initialize with true values (5 points)
Initialize the three classes with the true values of the parameters and see what happens. Report your results.

```python

frequencies = itemfreq(reduced_train_labels)
pi = frequencies[:, 1].astype(float) / len(reduced_train_labels)

K = len(classes)
mu = np.zeros((K, 28*28))

classes_map = {}
for i, c in enumerate(classes):
    classes_map[c] = i

classes_count = np.zeros(K)

for x, l in zip(reduced_train_data, reduced_train_labels):
    mu[classes_map[l], :] += x
    classes_count[classes_map[l]] += 1

mu = mu / classes_count.reshape(-1, 1)

gamma_i, mu_i, pi_i = EM(reduced_train_data, K, 100, mu=mu, pi=pi)

test_classification(reduced_train_data, reduced_train_labels, classes, classes, gamma_i)

```

By initializing the three classes with the true parameters we see a small improvement in accuracy, which stands between 5 and 10 percent. This is probably because random initialization increases the chances to get stuck into a local optima in comparison with true parameter initialization.
Regarding the type of misclassification errors, we see the same patterns described in the previous experiment.

In addition, this way of initialize the classes significantly increases the speed in converging. Using a convergenge threshold for $\mu$ and $\pi$ updates of $1e-5$, the random initialization approach usually takes between 30 and 80 EM iterations, while using initialization with true parameters results in convergence after 10 to 30 iterations.

## Part 2: Variational Auto-Encoder

A Variational Auto-Encoder (VAE) is a probabilistic model $p(\bx, \bz)$ over observed variables $\bx$ and latent variables and/or parameters $\bz$. Here we distinguish the decoder part, $p(\bx | \bz) p(\bz)$ and an encoder part $p(\bz | \bx)$ that are both specified with a neural network. A lower bound on the log marginal likelihood $\log p(\bx)$ can be obtained by approximately inferring the latent variables z from the observed data x using an encoder distribution $q(\bz| \bx)$ that is also specified as a neural network. This lower bound is then optimized to fit the model to the data.

The model was introduced by Diederik Kingma (during his PhD at the UVA) and Max Welling in 2013, https://arxiv.org/abs/1312.6114.

Since it is such an important model there are plenty of well written tutorials that should help you with the assignment. E.g: https://jaan.io/what-is-variational-autoencoder-vae-tutorial/.

In the following, we will make heavily use of the torch module, https://pytorch.org/docs/stable/index.html. Most of the time replacing `np.` with `torch.` will do the trick, e.g. `np.sum` becomes `torch.sum` and `np.log` becomes `torch.log`. In addition, we will use `torch.FloatTensor()` as an equivalent to `np.array()`. In order to train our VAE efficiently we will make use of batching. The number of data points in a batch will become the first dimension of our data tensor, e.g. A batch of 128 MNIST images has the dimensions [128, 1, 28, 28]. To check check the dimensions of a tensor you can call `.size()`.

### 2.1 Loss function
The objective function (variational lower bound), that we will use to train the VAE, consists of two terms: a log Bernoulli loss (reconstruction loss) and a Kullback–Leibler divergence. We implement the two terms separately and combine them in the end.
As seen in Part 1: Expectation Maximization, we can use a multivariate Bernoulli distribution to model the likelihood $p(\bx | \bz)$ of black and white images. Formally, the variational lower bound is maximized but in PyTorch we are always minimizing therefore we need to calculate the negative log Bernoulli loss and Kullback–Leibler divergence.

### 2.1.1 Negative Log Bernoulli loss (5 points)
The negative log Bernoulli loss is defined as,

\begin{align}
loss = - (\sum_i^D \bx_i \log \hat{\bx_i} + (1 − \bx_i) \log(1 − \hat{\bx_i})).
\end{align}

Write a function `log_bernoulli_loss` that takes a D dimensional vector `x`, its reconstruction `x_hat` and returns the negative log Bernoulli loss. Make sure that your function works for batches of arbitrary size.

```python
def log_bernoulli_loss(x_hat, x):

#     loss = -x * torch.log(x_hat) - (1 - x)*torch.log(1 - x_hat)

#     loss = loss.sum() #loss.sum(2) ??

    loss = F.binary_cross_entropy(x_hat, x, size_average=False)

    return loss

```

```python
### Test test test
x_test = torch.FloatTensor([[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8], [0.9, 0.9, 0.9, 0.9]])
x_hat_test = torch.FloatTensor([[0.11, 0.22, 0.33, 0.44], [0.55, 0.66, 0.77, 0.88], [0.99, 0.99, 0.99, 0.99]])

assert log_bernoulli_loss(x_hat_test, x_test) > 0.0
assert log_bernoulli_loss(x_hat_test, x_test) < 10.0

```

### 2.1.2 Negative Kullback–Leibler divergence (10 Points)
The variational lower bound (the objective to be maximized) contains a KL term $D_{KL}(q(\bz)||p(\bz))$ that can often be calculated analytically. In the VAE we assume $q = N(\bz, \mu, \sigma^2I)$ and $p = N(\bz, 0, I)$. Solve analytically!

$\mathcal{KL}(q || p) = \int q (\log q - \log p ) dx \\
= \frac{1}{2} \int q (-\log |\sigma^2 I | + z^T z - (z-\mu)^T (\sigma^2 I )^{-1}(z-\mu) ) dx \\
=\frac{1}{2} (-\log |\sigma^2 I | + \mu^T \mu + Tr(\sigma^2 I ) - Tr(I^{-1}I) ) \\
=- \frac{1}{2}\sum_i^D \log \sigma_i^2 + \frac{1}{2} \mu^T \mu + \frac{1}{2} \sum_i^D \sigma_i^2 - \frac{D}{2} $

Since in the definition of $q$ provided $\sigma^2$ appears to be a scalar (then the covariance between every dimension is equal), we also provided the solution for the KL divergence in that case:

$\mathcal{KL}(q || p) = -\frac{1}{2} D \ln \sigma^2 + \frac{1}{2} \mu^T\mu -\frac{1}{2} D + \frac{1}{2} D \sigma^2$

Write a function `KL_loss` that takes two J dimensional vectors `mu` and `logvar` and returns the negative Kullback–Leibler divergence. Where `logvar` is $\log(\sigma^2)$. Make sure that your function works for batches of arbitrary size.

```python
def KL_loss(mu, logvar):
#     N, D = mu.size()
#     norm = mu.pow(2)
#     loss = N*(-0.5*D) -0.5*torch.sum(logvar) + 0.5*torch.sum(norm) + 0.5*torch.sum(torch.exp(logvar))

    loss = -0.5 * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())

    return loss

```

```python
### Test test test
mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

assert KL_loss(mu_test, logvar_test) > 0.0
assert KL_loss(mu_test, logvar_test) < 10.0

```

### 2.1.3 Putting the losses together (5 points)
Write a function `loss_function` that takes a D dimensional vector `x`, its reconstruction `x_hat`, two J dimensional vectors `mu` and `logvar` and returns the final loss. Make sure that your function works for batches of arbitrary size.

```python
def loss_function(x_hat, x, mu, logvar):
    return log_bernoulli_loss(x_hat, x) + KL_loss(mu, logvar)

```

```python
x_test = torch.FloatTensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
x_hat_test = torch.FloatTensor([[0.11, 0.22, 0.33], [0.44, 0.55, 0.66], [0.77, 0.88, 0.99]])
mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

assert loss_function(x_hat_test, x_test, mu_test, logvar_test) > 0.0
assert loss_function(x_hat_test, x_test, mu_test, logvar_test) < 10.0

```

### 2.2 The model
Below you see a data structure for the VAE. The modell itself consists of two main parts the encoder (images $\bx$ to latent variables $\bz$) and the decoder (latent variables $\bz$ to images $\bx$). The encoder is using 3 fully-connected layers, whereas the decoder is using fully-connected layers. Right now the data structure is quite empty, step by step will update its functionality. For test purposes we will initialize a VAE for you. After the data structure is completed you will do the hyperparameter search.

```python
from torch import nn
from torch.nn import functional as F

class VAE(nn.Module):
    def __init__(self, fc1_dims, fc21_dims, fc22_dims, fc3_dims, fc4_dims):
        super(VAE, self).__init__()

        self.fc1 = nn.Linear(*fc1_dims)
        self.fc21 = nn.Linear(*fc21_dims)
        self.fc22 = nn.Linear(*fc22_dims)
        self.fc3 = nn.Linear(*fc3_dims)
        self.fc4 = nn.Linear(*fc4_dims)

    def encode(self, x):
        # To be implemented
        raise Exception('Method not implemented')

    def reparameterize(self, mu, logvar):
        # To be implemented
        raise Exception('Method not implemented')

    def decode(self, z):
        # To be implemented
        raise Exception('Method not implemented')

    def forward(self, x):
        # To be implemented
        raise Exception('Method not implemented')

VAE_test = VAE(fc1_dims=(784, 4), fc21_dims=(4, 2), fc22_dims=(4, 2), fc3_dims=(2, 4), fc4_dims=(4, 784))

```

### 2.3 Encoding (10 points)
Write a function `encode` that gets a vector `x` with 784 elements (flattened MNIST image) and returns `mu` and `logvar`. Your function should use three fully-connected layers (`self.fc1()`, `self.fc21()`, `self.fc22()`). First, you should use `self.fc1()` to embed `x`. Second, you should use `self.fc21()` and `self.fc22()` on the embedding of `x` to compute `mu` and `logvar` respectively. PyTorch comes with a variety of activation functions, the most common calls are `F.relu()`, `F.sigmoid()`, `F.tanh()`. Make sure that your function works for batches of arbitrary size.

```python
def encode(self, x):

    e = self.fc1(x)
    e = F.relu(e)

    mu = self.fc21(e)
#     mu = F.tanh(mu)

    logvar = self.fc22(e)
#     logvar = F.relu(logvar)

    return mu, logvar

```

```python
### Test, test, test
VAE.encode = encode

x_test = torch.ones((5,784))
mu_test, logvar_test = VAE_test.encode(x_test)

assert np.allclose(mu_test.size(), [5, 2])
assert np.allclose(logvar_test.size(), [5, 2])

```

### 2.4 Reparameterization (10 points)
One of the major question that the VAE is answering, is 'how to take derivatives with respect to the parameters of a stochastic variable?', i.e. if we are given $\bz$ that is drawn from a distribution $q(\bz|\bx)$, and we want to take derivatives. This step is necessary to be able to use gradient-based optimization algorithms like SGD.
For some distributions, it is possible to reparameterize samples in a clever way, such that the stochasticity is independent of the parameters. We want our samples to deterministically depend on the parameters of the distribution. For example, in a normally-distributed variable with mean $\mu$ and standard deviation $\sigma$, we can sample from it like this:

\begin{align}
\bz = \mu + \sigma \odot \epsilon,
\end{align}

where $\odot$ is the element-wise multiplication and $\epsilon$ is sampled from $N(0, I)$.

Write a function `reparameterize` that takes two J dimensional vectors `mu` and `logvar`. It should return $\bz = \mu + \sigma \odot \epsilon$.

```python
def reparameterize(self, mu, logvar):

    z = mu + torch.exp(0.5*logvar)*torch.randn_like(mu)

    return z

```

```python
### Test, test, test
VAE.reparameterize = reparameterize
VAE_test.train()

mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

z_test = VAE_test.reparameterize(mu_test, logvar_test)

assert np.allclose(z_test.size(), [3, 2])
assert z_test[0][0] < 5.0
assert z_test[0][0] > -5.0

```

### 2.5 Decoding (10 points)
Write a function `decode` that gets a vector `z` with J elements and returns a vector `x_hat` with 784 elements (flattened MNIST image). Your function should use two fully-connected layers (`self.fc3()`, `self.fc4()`). PyTorch comes with a variety of activation functions, the most common calls are `F.relu()`, `F.sigmoid()`, `F.tanh()`. Make sure that your function works for batches of arbitrary size.

```python
def decode(self, z):

    y = self.fc3(z)
    y = F.relu(y)

    y = self.fc4(y)

    x_hat = F.sigmoid(y)

    return x_hat

```

```python
# test test test
VAE.decode = decode

z_test = torch.ones((5,2))
x_hat_test = VAE_test.decode(z_test)

assert np.allclose(x_hat_test.size(), [5, 784])
assert (x_hat_test <= 1).all()
assert (x_hat_test >= 0).all()

```

### 2.6 Forward pass (10)
To complete the data structure you have to define a forward pass through the VAE. A single forward pass consists of the encoding of an MNIST image $\bx$ into latent space $\bz$, the reparameterization of $\bz$ and the decoding of $\bz$ into an image $\bx$.

Write a function `forward` that gets a a vector `x` with 784 elements (flattened MNIST image) and returns a vector `x_hat` with 784 elements (flattened MNIST image), `mu` and `logvar`.

```python
def forward(self, x):
    x = x.view(-1, 784)

    mu, logvar = self.encode(x)

    z = self.reparameterize(mu, logvar)

    x_hat = self.decode(z)

    return x_hat, mu, logvar

```

```python
# test test test
VAE.forward = forward

x_test = torch.ones((5,784))
x_hat_test, mu_test, logvar_test = VAE_test.forward(x_test)

assert np.allclose(x_hat_test.size(), [5, 784])
assert np.allclose(mu_test.size(), [5, 2])
assert np.allclose(logvar_test.size(), [5, 2])

```

### 2.7 Training (15)
We will now train the VAE using an optimizer called Adam, https://arxiv.org/abs/1412.6980. The code to train a model in PyTorch is given below.

```python
from torch.autograd import Variable

def train(epoch, train_loader, model, optimizer):
    model.train()
    train_loss = 0
    for batch_idx, (data, _) in enumerate(train_loader):
        data = Variable(data)
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data)
        loss = loss_function(recon_batch, data.view(-1, 784), mu, logvar)
        loss.backward()
        train_loss += loss.data
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.data / len(data)))

    print('====> Epoch: {} Average loss: {:.4f}'.format(
          epoch, train_loss / len(train_loader.dataset)))

```

Let's train. You have to choose the hyperparameters. Make sure your loss is going down in a reasonable amount of epochs (around 10).

```python
# Hyperparameters
# fc1_dims = (?,?)
# fc21_dims =
# fc22_dims =
# fc3_dims =
# fc4_dims =
# lr =
# batch_size =
# epochs =

hidden_layer_size = 400
latent_space_size = 20

fc1_dims = (28*28, hidden_layer_size)
fc21_dims = (hidden_layer_size, latent_space_size)
fc22_dims = (hidden_layer_size, latent_space_size)
fc3_dims = (latent_space_size, hidden_layer_size)
fc4_dims = (hidden_layer_size, 28*28)
lr = 0.001
batch_size = 128
epochs = 10

```

```python
# This cell contains a hidden test, please don't delete it, thx
```

Run the box below to train the model using the hyperparameters you entered above.

```python
from torchvision import datasets, transforms
from torch import nn, optim

# Load data
train_data = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size, shuffle=True, **{})

# Init model
VAE_MNIST = VAE(fc1_dims=fc1_dims, fc21_dims=fc21_dims, fc22_dims=fc22_dims, fc3_dims=fc3_dims, fc4_dims=fc4_dims)

# Init optimizer
optimizer = optim.Adam(VAE_MNIST.parameters(), lr=lr)

# Train
for epoch in range(1, epochs + 1):
    train(epoch, train_loader, VAE_MNIST, optimizer)

```

    Train Epoch: 1 [0/60000 (0%)]	Loss: 548.218994
    Train Epoch: 1 [12800/60000 (21%)]	Loss: 179.038406
    Train Epoch: 1 [25600/60000 (43%)]	Loss: 154.299728
    Train Epoch: 1 [38400/60000 (64%)]	Loss: 136.146194
    Train Epoch: 1 [51200/60000 (85%)]	Loss: 137.696609
    ====> Epoch: 1 Average loss: 163.9566
    Train Epoch: 2 [0/60000 (0%)]	Loss: 128.446014
    Train Epoch: 2 [12800/60000 (21%)]	Loss: 118.466515
    Train Epoch: 2 [25600/60000 (43%)]	Loss: 121.666725
    Train Epoch: 2 [38400/60000 (64%)]	Loss: 118.164062
    Train Epoch: 2 [51200/60000 (85%)]	Loss: 118.403358
    ====> Epoch: 2 Average loss: 121.3418
    Train Epoch: 3 [0/60000 (0%)]	Loss: 115.683067
    Train Epoch: 3 [12800/60000 (21%)]	Loss: 115.601486
    Train Epoch: 3 [25600/60000 (43%)]	Loss: 116.874527
    Train Epoch: 3 [38400/60000 (64%)]	Loss: 116.663383
    Train Epoch: 3 [51200/60000 (85%)]	Loss: 113.716934
    ====> Epoch: 3 Average loss: 114.6640
    Train Epoch: 4 [0/60000 (0%)]	Loss: 109.423950
    Train Epoch: 4 [12800/60000 (21%)]	Loss: 114.831970
    Train Epoch: 4 [25600/60000 (43%)]	Loss: 111.892365
    Train Epoch: 4 [38400/60000 (64%)]	Loss: 116.666138
    Train Epoch: 4 [51200/60000 (85%)]	Loss: 112.022842
    ====> Epoch: 4 Average loss: 111.6554
    Train Epoch: 5 [0/60000 (0%)]	Loss: 115.271133
    Train Epoch: 5 [12800/60000 (21%)]	Loss: 105.700172
    Train Epoch: 5 [25600/60000 (43%)]	Loss: 108.677246
    Train Epoch: 5 [38400/60000 (64%)]	Loss: 109.566368
    Train Epoch: 5 [51200/60000 (85%)]	Loss: 107.932693
    ====> Epoch: 5 Average loss: 109.8876
    Train Epoch: 6 [0/60000 (0%)]	Loss: 108.103485
    Train Epoch: 6 [12800/60000 (21%)]	Loss: 107.634300
    Train Epoch: 6 [25600/60000 (43%)]	Loss: 108.621140
    Train Epoch: 6 [38400/60000 (64%)]	Loss: 107.716660
    Train Epoch: 6 [51200/60000 (85%)]	Loss: 109.582489
    ====> Epoch: 6 Average loss: 108.7154
    Train Epoch: 7 [0/60000 (0%)]	Loss: 110.990593
    Train Epoch: 7 [12800/60000 (21%)]	Loss: 108.023628
    Train Epoch: 7 [25600/60000 (43%)]	Loss: 103.764626
    Train Epoch: 7 [38400/60000 (64%)]	Loss: 105.314987
    Train Epoch: 7 [51200/60000 (85%)]	Loss: 108.633560
    ====> Epoch: 7 Average loss: 107.8690
    Train Epoch: 8 [0/60000 (0%)]	Loss: 107.899307
    Train Epoch: 8 [12800/60000 (21%)]	Loss: 110.971703
    Train Epoch: 8 [25600/60000 (43%)]	Loss: 109.482376
    Train Epoch: 8 [38400/60000 (64%)]	Loss: 110.997002
    Train Epoch: 8 [51200/60000 (85%)]	Loss: 102.966225
    ====> Epoch: 8 Average loss: 107.1769
    Train Epoch: 9 [0/60000 (0%)]	Loss: 106.253616
    Train Epoch: 9 [12800/60000 (21%)]	Loss: 107.286575
    Train Epoch: 9 [25600/60000 (43%)]	Loss: 102.333618
    Train Epoch: 9 [38400/60000 (64%)]	Loss: 102.734947
    Train Epoch: 9 [51200/60000 (85%)]	Loss: 104.835625
    ====> Epoch: 9 Average loss: 106.6546
    Train Epoch: 10 [0/60000 (0%)]	Loss: 103.383324
    Train Epoch: 10 [12800/60000 (21%)]	Loss: 108.156723
    Train Epoch: 10 [25600/60000 (43%)]	Loss: 106.962585
    Train Epoch: 10 [38400/60000 (64%)]	Loss: 104.326218
    Train Epoch: 10 [51200/60000 (85%)]	Loss: 103.240379
    ====> Epoch: 10 Average loss: 106.2311

Run the box below to check if the model you trained above is able to correctly reconstruct images.

```python
### Let's check if the reconstructions make sense
# Set model to test mode
VAE_MNIST.eval()

# Reconstructed
train_data_plot = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader_plot = torch.utils.data.DataLoader(train_data_plot,
                                           batch_size=1, shuffle=False, **{})

for batch_idx, (data, _) in enumerate(train_loader_plot):
    x_hat, mu, logvar = VAE_MNIST(data)
    plt.imshow(x_hat.view(1,28,28).squeeze().data.numpy(), cmap='gray')
    plt.title('%i' % train_data.train_labels[batch_idx])
    plt.show()
    if batch_idx == 3:
        break

```

### 2.8 Visualize latent space (20 points)
Now, implement the auto-encoder now with a 2-dimensional latent space, and train again over the MNIST data. Make a visualization of the learned manifold by using a linearly spaced coordinate grid as input for the latent space, as seen in  https://arxiv.org/abs/1312.6114 Figure 4.

```python
hidden_layer_size = 400
latent_space_size = 2

fc1_dims = (28*28, hidden_layer_size)
fc21_dims = (hidden_layer_size, latent_space_size)
fc22_dims = (hidden_layer_size, latent_space_size)
fc3_dims = (latent_space_size, hidden_layer_size)
fc4_dims = (hidden_layer_size, 28*28)
lr = 0.001
batch_size = 64
epochs = 10

# Init model
VAE_MNIST_2 = VAE(fc1_dims=fc1_dims, fc21_dims=fc21_dims, fc22_dims=fc22_dims, fc3_dims=fc3_dims, fc4_dims=fc4_dims)

# Init optimizer
optimizer_2 = optim.Adam(VAE_MNIST_2.parameters(), lr=lr)

# Train
for epoch in range(1, epochs + 1):
    train(epoch, train_loader, VAE_MNIST_2, optimizer_2)

def box_muller(u1, u2):
    R = np.sqrt(-2*np.log(u1))
    theta = 2*np.pi*u2

    z1 = R*np.cos(theta)
    z2 = R*np.sin(theta)
    return z1, z2

N = 15

fig, axes = plt.subplots(N, N, figsize=(N, N))

for i, u1 in enumerate(np.linspace(0.1, 1, N)):
    for j, u2 in enumerate(np.linspace(0.1, 1, N)):

        z1, z2 = box_muller(u1, u2)

        rec = VAE_MNIST_2.decode(torch.FloatTensor([z1, z2]).view(1, 2))

        ax = axes[i, j]
        ax.xaxis.set_visible(False)
        ax.yaxis.set_visible(False)
        ax.imshow(rec.view(28,28).squeeze().data.numpy(),cmap='gray') #Greys swaps colors

plt.show()

```

### 2.8 Amortized inference (10 points)
What is amortized inference? Where in the code of Part 2 is it used? What is the benefit of using it?

In Amortized Inference, instead of using a different set of parameters for each datapoint as in Mean-Field Inference, part of the parameters are shared between the datapoints.

In Part2, it is used in the decoder: while the mean and the variances are parameters of a particular datapoint, the parameters of the decoder network are the same for any datapoint.

The main benefit is a smaller set of parameters to learn, reducing the cost of the algorithm.


# Everything you need to know about: Linear Regression

> *Since the title is "Everything you need to know" keep in mind that main goal here is to explain concepts with a very simple approach, give some examples and references. This is not a deep study of the subject.*

## Very basic concepts

Linear Regression is a statistic method to find relatioships between dependent and independent variables. It can be used to do machine learning, since some simple problems can be solved easily with regression.

We find the relationship by finding a line that best fits the actual data we have, giving us power to predict new data.

We can minimize the error between the line and actual data to improve our prediction capability.

The way we minimize the error is by changing the line position using the line equation:

\begin{equation}
y = c + b*x
\end{equation}

Where:
+ y = estimated dependent variable score
+ c = constant
+ b = regression coefficient
+ x = score on the independent variable

## Examples

A good example of Linear Regression is to predict an animal weight based on his brain weight.

We can do it easily using scikit-learn:

```python
%matplotlib inline

## Importing dependencies
import pandas as pd
from sklearn import linear_model
import matplotlib.pyplot as plt

## Loading our data
df = pd.read_fwf('brain_body.txt')

## Setting brain as independent variable
x_values = df[['Brain']].values.tolist()
## Setting brain as dependent variable
y_values = df[['Body']].values.tolist()

## Training model on our data
body_weight_reg = linear_model.LinearRegression()
body_weight_reg.fit(x_values, y_values)

## Plot the results
plt.scatter(x_values, y_values)
plt.plot(x_values, body_weight_reg.predict(x_values))
plt.show()
```

## Resources

### Articles
+ [Wikipedia Article](https://en.wikipedia.org/wiki/Linear_regression)
+ [Statistics Solutions Article](http://www.statisticssolutions.com/what-is-linear-regression/)
+ [Lesson from PennState College of Science](https://onlinecourses.science.psu.edu/stat501/node/250)
+ [Scikit-learn Linear Regression Examples](http://scikit-learn.org/stable/auto_examples/linear_model/plot_ols.html)

```python

```