# Von Karman vortex street

$$
\renewcommand{\DdQq}[2]{{\mathrm D}_{#1}{\mathrm Q}_{#2}}
\renewcommand{\drondt}{\partial_t}
\renewcommand{\drondx}{\partial_x}
\renewcommand{\drondy}{\partial_y}
\renewcommand{\drondtt}{\partial_{tt}}
\renewcommand{\drondxx}{\partial_{xx}}
\renewcommand{\drondyy}{\partial_{yy}}
\renewcommand{\dx}{\Delta x}
\renewcommand{\dt}{\Delta t}
\renewcommand{\grandO}{{\mathcal O}}
\renewcommand{\density}[2]{\,f_{#1}^{#2}}
\renewcommand{\fk}[1]{\density{#1}{\vphantom{\star}}}
\renewcommand{\fks}[1]{\density{#1}{\star}}
\renewcommand{\moment}[2]{\,m_{#1}^{#2}}
\renewcommand{\mk}[1]{\moment{#1}{\vphantom{\star}}}
\renewcommand{\mke}[1]{\moment{#1}{e}}
\renewcommand{\mks}[1]{\moment{#1}{\star}}
$$

In this tutorial, we consider the classical $\DdQq{2}{9}$ to simulate the Von Karman vortex street modeling by the Navier-Stokes equations.

In fluid dynamics, a Von Karman vortex street is a repeating pattern of swirling vortices caused by the unsteady separation of flow of a fluid around blunt bodies. It is named after the engineer and fluid dynamicist Theodore von Karman. For the simulation, we propose to simulate the Navier-Stokes equation into a rectangular domain with a circular hole of diameter $d$.

The $\DdQq{2}{9}$ is defined by:

* a space step $\dx$ and a time step $\dt$ related to the scheme velocity $\lambda$ by the relation $\lambda=\dx/\dt$,
* nine velocities $\{(0,0), (\pm1,0), (0,\pm1), (\pm1, \pm1)\}$, identified in pylbm by the numbers $0$ to $8$,
* nine polynomials used to build the moments

$$
\{1, \lambda X, \lambda Y, 3E-4, (9E^2-21E+8)/2, 3XE-5X, 3YE-5Y,X^2-Y^2, XY\},
$$

where $E = X^2+Y^2$.

* three conserved moments $\rho$, $q_x$, and $q_y$,
* nine relaxation parameters (three are $0$ corresponding to conserved moments): $\{0,0,0,s_\mu,s_\mu,s_\eta,s_\eta,s_\eta,s_\eta\}$, where $s_\mu$ and $s_\eta$ are in $(0,2)$,
* equilibrium value of the non conserved moments

$$
\begin{aligned}\mke{3} &= -2\rho + 3(q_x^2+q_y^2)/(\rho_0\lambda^2), \\ \mke{4} &= \rho-3(q_x^2+q_y^2)/(\rho_0\lambda^2), \\ \mke{5} &= -q_x/\lambda, \\ \mke{6} &= -q_y/\lambda, \\ \mke{7} &= (q_x^2-q_y^2)/(\rho_0\lambda^2), \\ \mke{8} &= q_xq_y/(\rho_0\lambda^2),\end{aligned}
$$

where $\rho_0$ is a given scalar.

This scheme is consistant at second order with the following equations (taken $\rho_0=1$)

$$
\begin{gathered} \drondt\rho + \drondx q_x + \drondy q_y = 0,\\ \drondt q_x + \drondx (q_x^2+p) + \drondy (q_xq_y) = \mu \drondx (\drondx q_x + \drondy q_y ) + \eta (\drondxx+\drondyy)q_x, \\ \drondt q_y + \drondx (q_xq_y) + \drondy (q_y^2+p) = \mu \drondy (\drondx q_x + \drondy q_y ) + \eta (\drondxx+\drondyy)q_y,\end{gathered}
$$

with $p=\rho\lambda^2/3$.

We write a dictionary for a simulation of the Navier-Stokes equations on $(0,1)^2$.

In order to impose the boundary conditions, we use the bounce-back conditions to fix $q_x=q_y=\rho v_0$ at south, east, and north where the velocity $v_0$ could be $v_0=\lambda/20$. At west, we impose the simple output condition of Neumann by repeating the second to last cells into the last cells.

The solution is governed by the Reynolds number $Re = \rho_0v_0d / \eta$, where $d$ is the diameter of the circle. Fix the relaxation parameters to have $Re=500$. The relaxation parameters related to the bulk viscosity $\mu$ should be large enough to ensure the stability (for instance $\mu=10^{-3}$).

We compute the stationary solution of the problem obtained for large enough final time. We plot the vorticity of the solution with the function imshow of matplotlib.

```python

from six.moves import range
%matplotlib inline
```

```python
import numpy as np
import sympy as sp
import pylbm

X, Y, LA = sp.symbols('X, Y, LA')
rho, qx, qy = sp.symbols('rho, qx, qy')

def bc_in(f, m, x, y):
    m[qx] = rhoo * v0

def vorticity(sol):
    ux = sol.m[qx] / sol.m[rho]
    uy = sol.m[qy] / sol.m[rho]
    V = np.abs(uy[2:,1:-1] - uy[0:-2,1:-1] - ux[1:-1,2:] + ux[1:-1,0:-2])/(2*sol.domain.dx)
    return -V

# parameters
rayon = 0.05
Re = 500
dx = 1./64   # spatial step
la = 1.      # velocity of the scheme
Tf = 75      # final time of the simulation
v0 = la/20   # maximal velocity obtained in the middle of the channel
rhoo = 1.    # mean value of the density
mu = 1.e-3   # bulk viscosity
eta = rhoo*v0*2*rayon/Re  # shear viscosity
# initialization
xmin, xmax, ymin, ymax = 0., 3., 0., 1.
dummy = 3.0/(la*rhoo*dx)
s_mu = 1.0/(0.5+mu*dummy)
s_eta = 1.0/(0.5+eta*dummy)
s_q = s_eta
s_es = s_mu
s  = [0.,0.,0.,s_mu,s_es,s_q,s_q,s_eta,s_eta]
dummy = 1./(LA**2*rhoo)
qx2 = dummy*qx**2
qy2 = dummy*qy**2
q2  = qx2+qy2
qxy = dummy*qx*qy

print("Reynolds number: {0:10.3e}".format(Re))
print("Bulk viscosity : {0:10.3e}".format(mu))
print("Shear viscosity: {0:10.3e}".format(eta))
print("relaxation parameters: {0}".format(s))

dico = {
    'box':{'x':[xmin, xmax], 'y':[ymin, ymax], 'label':[0,2,0,0]},
    'elements':[pylbm.Circle([.3, 0.5*(ymin+ymax)+dx], rayon, label=1)],
    'space_step':dx,
    'scheme_velocity':la,
    'parameters':{LA:la},
    'schemes':[
        {
            'velocities':list(range(9)),
            'conserved_moments':[rho, qx, qy],
            'polynomials':[
                1, LA*X, LA*Y,
                3*(X**2+Y**2)-4,
                (9*(X**2+Y**2)**2-21*(X**2+Y**2)+8)/2,
                3*X*(X**2+Y**2)-5*X, 3*Y*(X**2+Y**2)-5*Y,
                X**2-Y**2, X*Y
            ],
            'relaxation_parameters':s,
            'equilibrium':[
                rho, qx, qy,
                -2*rho + 3*q2,
                rho-3*q2,
                -qx/LA, -qy/LA,
                qx2-qy2, qxy
            ],
            'init':{rho:rhoo, qx:0., qy:0.},
        },
    ],
    'boundary_conditions':{
        0:{'method':{0: pylbm.bc.BouzidiBounceBack}, 'value':bc_in},
        1:{'method':{0: pylbm.bc.BouzidiBounceBack}, 'value':None},
        2:{'method':{0: pylbm.bc.NeumannX}, 'value':None},
    },
    'generator': 'cython',
}

sol = pylbm.Simulation(dico)
while sol.t < Tf:
    sol.one_time_step()

viewer = pylbm.viewer.matplotlib_viewer
fig = viewer.Fig()
ax = fig[0]
im = ax.image(vorticity(sol).transpose(), clim = [-3., 0])
ax.ellipse([.3/dx, 0.5*(ymin+ymax)/dx], [rayon/dx,rayon/dx], 'r')
ax.title = 'Von Karman vortex street at t = {0:f}'.format(sol.t)
fig.show()
```


# 5 - Ensemble Methods - Bagging

by [Alejandro Correa Bahnsen](albahnsen.com/) and [Jesus Solano](https://github.com/jesugome)

version 1.6, June 2020

## Part of the class [Advanced Methods in Data Analysis](https://github.com/albahnsen/AdvancedMethodsDataAnalysisClass)

This notebook is licensed under a [Creative Commons Attribution-ShareAlike 3.0 Unported License](http://creativecommons.org/licenses/by-sa/3.0/deed.en_US). Special thanks goes to [Kevin Markham](https://github.com/justmarkham)

Why are we learning about ensembling?

- Very popular method for improving the predictive performance of machine learning models
- Provides a foundation for understanding more sophisticated models

## Lesson objectives

Students will be able to:

- Define ensembling and its requirements
- Identify the two basic methods of ensembling
- Decide whether manual ensembling is a useful approach for a given problem
- Explain bagging and how it can be applied to decision trees
- Explain how out-of-bag error and feature importances are calculated from bagged trees
- Explain the difference between bagged trees and Random Forests
- Build and tune a Random Forest model in scikit-learn
- Decide whether a decision tree or a Random Forest is a better model for a given problem

# Part 1: Introduction

Ensemble learning is a widely studied topic in the machine learning community. The main idea behind
the ensemble methodology is to combine several individual base classifiers in   order to have a
classifier that outperforms each of them.

Nowadays,   ensemble methods are  one
of the most popular and well studied machine learning techniques, and it can be
noted that since 2009 all the first-place and   second-place winners of the KDD-Cup https://www.sigkdd.org/kddcup/   used  ensemble methods. The core
principle in ensemble learning, is to induce random perturbations into  the learning procedure in
order to produce several different base classifiers from a single  training set, then combining the
base classifiers in order to make the final prediction.  In order to induce the random permutations
and therefore create the different base classifiers,   several methods have been proposed, in
particular:
* bagging
* pasting
* random forests
* random patches

Finally, after  the base   classifiers
are trained, they are typically   combined using either:
* majority voting
* weighted  voting
* stacking

There are three main reasons regarding why ensemble
methods perform better than single models: statistical, computational and representational . First, from a statistical point of view, when the learning set is too
small, an algorithm can find several good models within the search space, that arise to the same
performance on the training set $\mathcal{S}$. Nevertheless, without a validation set, there is
a risk of choosing the wrong model. The second reason is computational; in general, algorithms
rely on some local search optimization and may get stuck in a local optima. Then, an ensemble may
solve this by focusing different algorithms to different spaces across the training set. The last
reason is representational. In most cases, for a learning set of finite size, the  true function
$f$ cannot be represented by any of the candidate models. By combining several  models in an
ensemble, it may be possible to obtain a model with a larger coverage across the  space of
representable functions.

## Example

Let's pretend that instead of building a single model to solve a binary classification problem, you created **five independent models**, and each model was correct about 70% of the time. If you combined these models into an "ensemble" and used their majority vote as a prediction, how often would the ensemble be correct?

```python
import numpy as np

# set a seed for reproducibility
np.random.seed(1234)

# generate 1000 random numbers (between 0 and 1) for each model, representing 1000 observations
mod1 = np.random.rand(1000)
mod2 = np.random.rand(1000)
mod3 = np.random.rand(1000)
mod4 = np.random.rand(1000)
mod5 = np.random.rand(1000)

# each model independently predicts 1 (the "correct response") if random number was at least 0.3
preds1 = np.where(mod1 > 0.3, 1, 0)
preds2 = np.where(mod2 > 0.3, 1, 0)
preds3 = np.where(mod3 > 0.3, 1, 0)
preds4 = np.where(mod4 > 0.3, 1, 0)
preds5 = np.where(mod5 > 0.3, 1, 0)

# print the first 20 predictions from each model
print(preds1[:20])
print(preds2[:20])
print(preds3[:20])
print(preds4[:20])
print(preds5[:20])
```

    [0 1 1 1 1 0 0 1 1 1 1 1 1 1 1 1 1 0 1 1]
    [1 1 1 1 1 1 1 0 1 0 0 0 1 1 1 0 1 0 0 0]
    [1 1 1 1 0 1 1 0 0 1 1 1 1 1 1 1 1 0 1 1]
    [1 1 0 0 0 0 1 1 0 1 1 1 1 1 1 0 1 1 1 0]
    [0 0 1 0 0 0 1 0 1 0 0 0 1 1 1 1 1 1 1 1]

```python
# average the predictions and then round to 0 or 1
ensemble_preds = np.round((preds1 + preds2 + preds3 + preds4 + preds5)/5.0).astype(int)

# print the ensemble's first 20 predictions
print(ensemble_preds[:20])
```

    [1 1 1 1 0 0 1 0 1 1 1 1 1 1 1 1 1 0 1 1]

```python
# how accurate was each individual model?
print(preds1.mean())
print(preds2.mean())
print(preds3.mean())
print(preds4.mean())
print(preds5.mean())
```

    0.713
    0.665
    0.717
    0.712
    0.687

```python
# how accurate was the ensemble?
print(ensemble_preds.mean())
```

    0.841

**Note:** As you add more models to the voting process, the probability of error decreases, which is known as [Condorcet's Jury Theorem](http://en.wikipedia.org/wiki/Condorcet%27s_jury_theorem).

## What is ensembling?

**Ensemble learning (or "ensembling")** is the process of combining several predictive models in order to produce a combined model that is more accurate than any individual model.

- **Regression:** take the average of the predictions
- **Classification:** take a vote and use the most common prediction, or take the average of the predicted probabilities

For ensembling to work well, the models must have the following characteristics:

- **Accurate:** they outperform the null model
- **Independent:** their predictions are generated using different processes

**The big idea:** If you have a collection of individually imperfect (and independent) models, the "one-off" mistakes made by each model are probably not going to be made by the rest of the models, and thus the mistakes will be discarded when averaging the models.

There are two basic **methods for ensembling:**

- Manually ensemble your individual models
- Use a model that ensembles for you

### Theoretical performance of an ensemble
  If we assume that each one of the $T$ base classifiers has a probability $\rho$ of
  being correct, the probability of an ensemble making the correct decision, assuming independence,
  denoted by $P_c$, can be calculated using the binomial distribution

$$P_c = \sum_{j>T/2}^{T} {{T}\choose{j}} \rho^j(1-\rho)^{T-j}.$$

  Furthermore, as shown, if $T\ge3$ then:

$$
  \lim_{T \to  \infty} P_c= \begin{cases}
            1  &\mbox{if } \rho>0.5 \\
            0  &\mbox{if } \rho<0.5 \\
            0.5  &\mbox{if } \rho=0.5 ,
            \end{cases}
$$
	leading to the conclusion that
$$
  \rho \ge 0.5 \quad \text{and} \quad T\ge3 \quad \Rightarrow \quad P_c\ge \rho.
$$

# Part 2: Manual ensembling

What makes a good manual ensemble?

- Different types of **models**
- Different combinations of **features**
- Different **tuning parameters**

*Machine learning flowchart created by the [winner](https://github.com/ChenglongChen/Kaggle_CrowdFlower) of Kaggle's [CrowdFlower competition](https://www.kaggle.com/c/crowdflower-search-relevance)*

```python
# read in and prepare the vehicle training data
import pandas as pd
url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/vehicles_train.csv'
train = pd.read_csv(url)
train['vtype'] = train.vtype.map({'car':0, 'truck':1})
# read in and prepare the vehicle testing data
url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/vehicles_test.csv'
test = pd.read_csv(url)
test['vtype'] = test.vtype.map({'car':0, 'truck':1})
```

```python
train.head()
```

      price
      year
      miles
      doors
      vtype

      0
      22000
      2012
      13000
      2
      0

      1
      14000
      2010
      30000
      2
      0

      2
      13000
      2010
      73500
      4
      0

      3
      9500
      2009
      78000
      4
      0

      4
      9000
      2007
      47000
      4
      0

### Train different models

```python
from sklearn.linear_model import LinearRegression
from sklearn.tree import DecisionTreeRegressor
from sklearn.naive_bayes import GaussianNB
from sklearn.neighbors import KNeighborsRegressor

models = {'lr': LinearRegression(),
          'dt': DecisionTreeRegressor(),
          'nb': GaussianNB(),
          'kn': KNeighborsRegressor()}
```

```python
# Train all the models
X_train = train.iloc[:, 1:]
X_test = test.iloc[:, 1:]
y_train = train.price
y_test = test.price

for model in models.keys():
    models[model].fit(X_train, y_train)
```

```python
# predict test for each model
y_pred = pd.DataFrame(index=test.index, columns=models.keys())
for model in models.keys():
    y_pred[model] = models[model].predict(X_test)

```

```python
# Evaluate each model
from sklearn.metrics import mean_squared_error

for model in models.keys():
    print(model,np.sqrt(mean_squared_error(y_pred[model], y_test)))
```

    lr 2138.3579028745116
    dt 1414.213562373095
    nb 5477.2255750516615
    kn 1671.3268182295567

### Evaluate the error of the mean of the predictions

```python
np.sqrt(mean_squared_error(y_pred.mean(axis=1), y_test))
```

    1193.164765760328

## Comparing manual ensembling with a single model approach

**Advantages of manual ensembling:**

- Increases predictive accuracy
- Easy to get started

**Disadvantages of manual ensembling:**

- Decreases interpretability
- Takes longer to train
- Takes longer to predict
- More complex to automate and maintain
- Small gains in accuracy may not be worth the added complexity

# Part 3: Bagging

The primary weakness of **decision trees** is that they don't tend to have the best predictive accuracy. This is partially due to **high variance**, meaning that different splits in the training data can lead to very different trees.

**Bagging** is a general purpose procedure for reducing the variance of a machine learning method, but is particularly useful for decision trees. Bagging is short for **bootstrap aggregation**, meaning the aggregation of bootstrap samples.

What is a **bootstrap sample**? A random sample with replacement:

```python
# set a seed for reproducibility
np.random.seed(1)

# create an array of 1 through 20
nums = np.arange(1, 21)
print(nums)

# sample that array 20 times with replacement
print(np.random.choice(a=nums, size=20, replace=True))
```

    [ 1  2  3  4  5  6  7  8  9 10 11 12 13 14 15 16 17 18 19 20]
    [ 6 12 13  9 10 12  6 16  1 17  2 13  8 14  7 19  6 19 12 11]

**How does bagging work (for decision trees)?**

1. Grow B trees using B bootstrap samples from the training data.
2. Train each tree on its bootstrap sample and make predictions.
3. Combine the predictions:
    - Average the predictions for **regression trees**
    - Take a vote for **classification trees**

Notes:

- **Each bootstrap sample** should be the same size as the original training set.
- **B** should be a large enough value that the error seems to have "stabilized".
- The trees are **grown deep** so that they have low bias/high variance.

Bagging increases predictive accuracy by **reducing the variance**, similar to how cross-validation reduces the variance associated with train/test split (for estimating out-of-sample error) by splitting many times an averaging the results.

```python
# set a seed for reproducibility
np.random.seed(123)

n_samples = train.shape[0]
n_B = 10

# create ten bootstrap samples (will be used to select rows from the DataFrame)
samples = [np.random.choice(a=n_samples, size=n_samples, replace=True) for _ in range(1, n_B +1 )]
samples
```

    [array([13,  2, 12,  2,  6,  1,  3, 10, 11,  9,  6,  1,  0,  1]),
     array([ 9,  0,  0,  9,  3, 13,  4,  0,  0,  4,  1,  7,  3,  2]),
     array([ 4,  7,  2,  4,  8, 13,  0,  7,  9,  3, 12, 12,  4,  6]),
     array([ 1,  5,  6, 11,  2,  1, 12,  8,  3, 10,  5,  0, 11,  2]),
     array([10, 10,  6, 13,  2,  4, 11, 11, 13, 12,  4,  6, 13,  3]),
     array([10,  0,  6,  4,  7, 11,  6,  7,  1, 11, 10,  5,  7,  9]),
     array([ 2,  4,  8,  1, 12,  2,  1,  1,  3, 12,  5,  9,  0,  8]),
     array([11,  1,  6,  3,  3, 11,  5,  9,  7,  9,  2,  3, 11,  3]),
     array([ 3,  8,  6,  9,  7,  6,  3,  9,  6, 12,  6, 11,  6,  1]),
     array([13, 10,  3,  4,  3,  1, 13,  0,  5,  8, 13,  6, 11,  8])]

```python
# show the rows for the first decision tree
train.iloc[samples[0], :]
```

      price
      year
      miles
      doors
      vtype

      13
      1300
      1997
      138000
      4
      0

      2
      13000
      2010
      73500
      4
      0

      12
      1800
      1999
      163000
      2
      1

      2
      13000
      2010
      73500
      4
      0

      6
      3000
      2004
      177000
      4
      0

      1
      14000
      2010
      30000
      2
      0

      3
      9500
      2009
      78000
      4
      0

      10
      2500
      2003
      190000
      2
      1

      11
      5000
      2001
      62000
      4
      0

      9
      1900
      2003
      160000
      4
      0

      6
      3000
      2004
      177000
      4
      0

      1
      14000
      2010
      30000
      2
      0

      0
      22000
      2012
      13000
      2
      0

      1
      14000
      2010
      30000
      2
      0

 Build one tree for each sample

```python
from sklearn.tree import DecisionTreeRegressor

# grow each tree deep
treereg = DecisionTreeRegressor(max_depth=None, random_state=123)

# DataFrame for storing predicted price from each tree
y_pred = pd.DataFrame(index=test.index, columns=[list(range(n_B))])

# grow one tree for each bootstrap sample and make predictions on testing data
for i, sample in enumerate(samples):
    X_train = train.iloc[sample, 1:]
    y_train = train.iloc[sample, 0]
    treereg.fit(X_train, y_train)
    y_pred[i] = treereg.predict(X_test)
```

```python
y_pred
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9

      0
      1300.0
      1300.0
      3000.0
      4000.0
      1300.0
      4000.0
      4000.0
      4000.0
      3000.0
      4000.0

      1
      5000.0
      1300.0
      3000.0
      5000.0
      5000.0
      5000.0
      4000.0
      5000.0
      5000.0
      5000.0

      2
      14000.0
      13000.0
      13000.0
      13000.0
      13000.0
      14000.0
      13000.0
      13000.0
      9500.0
      9000.0

Results of each tree

```python
for i in range(n_B):
    print(i, np.sqrt(mean_squared_error(y_pred[i], y_test)))
```

    0 1621.7274740226856
    1 2942.7877939124323
    2 1825.7418583505537
    3 1000.0
    4 1276.7145334803704
    5 1414.213562373095
    6 1414.213562373095
    7 1000.0
    8 1554.5631755148024
    9 1914.854215512676

Results of the ensemble

```python
y_pred.mean(axis=1)
```

    0     2990.0
    1     4330.0
    2    12450.0
    dtype: float64

```python
np.sqrt(mean_squared_error(y_test, y_pred.mean(axis=1)))
```

    998.5823284370031

## Bagged decision trees in scikit-learn (with B=500)

```python
# define the training and testing sets
X_train = train.iloc[:, 1:]
y_train = train.iloc[:, 0]
X_test = test.iloc[:, 1:]
y_test = test.iloc[:, 0]
```

```python
# instruct BaggingRegressor to use DecisionTreeRegressor as the "base estimator"
from sklearn.ensemble import BaggingRegressor
bagreg = BaggingRegressor(DecisionTreeRegressor(), n_estimators=500,
                          bootstrap=True, oob_score=True, random_state=1)
```

    C:\Users\albah\Anaconda3\lib\site-packages\sklearn\ensemble\weight_boosting.py:29: DeprecationWarning: numpy.core.umath_tests is an internal NumPy module and should not be imported. It will be removed in a future NumPy release.
      from numpy.core.umath_tests import inner1d

```python
# fit and predict
bagreg.fit(X_train, y_train)
y_pred = bagreg.predict(X_test)
y_pred
```

    array([ 3344.2,  5395. , 12902. ])

```python
# calculate RMSE
np.sqrt(mean_squared_error(y_test, y_pred))
```

    657.8000304043775

## Estimating out-of-sample error

For bagged models, out-of-sample error can be estimated without using **train/test split** or **cross-validation**!

On average, each bagged tree uses about **two-thirds** of the observations. For each tree, the **remaining observations** are called "out-of-bag" observations.

```python
# show the first bootstrap sample
samples[0]
```

    array([13,  2, 12,  2,  6,  1,  3, 10, 11,  9,  6,  1,  0,  1])

```python
# show the "in-bag" observations for each sample
for sample in samples:
    print(set(sample))
```

    {0, 1, 2, 3, 6, 9, 10, 11, 12, 13}
    {0, 1, 2, 3, 4, 7, 9, 13}
    {0, 2, 3, 4, 6, 7, 8, 9, 12, 13}
    {0, 1, 2, 3, 5, 6, 8, 10, 11, 12}
    {2, 3, 4, 6, 10, 11, 12, 13}
    {0, 1, 4, 5, 6, 7, 9, 10, 11}
    {0, 1, 2, 3, 4, 5, 8, 9, 12}
    {1, 2, 3, 5, 6, 7, 9, 11}
    {1, 3, 6, 7, 8, 9, 11, 12}
    {0, 1, 3, 4, 5, 6, 8, 10, 11, 13}

```python
# show the "out-of-bag" observations for each sample
for sample in samples:
    print(sorted(set(range(n_samples)) - set(sample)))
```

    [4, 5, 7, 8]
    [5, 6, 8, 10, 11, 12]
    [1, 5, 10, 11]
    [4, 7, 9, 13]
    [0, 1, 5, 7, 8, 9]
    [2, 3, 8, 12, 13]
    [6, 7, 10, 11, 13]
    [0, 4, 8, 10, 12, 13]
    [0, 2, 4, 5, 10, 13]
    [2, 7, 9, 12]

How to calculate **"out-of-bag error":**

1. For every observation in the training data, predict its response value using **only** the trees in which that observation was out-of-bag. Average those predictions (for regression) or take a vote (for classification).
2. Compare all predictions to the actual response values in order to compute the out-of-bag error.

When B is sufficiently large, the **out-of-bag error** is an accurate estimate of **out-of-sample error**.

```python
# compute the out-of-bag R-squared score (not MSE, unfortunately!) for B=500
bagreg.oob_score_
```

    0.7986955133989982

## Estimating feature importance

Bagging increases **predictive accuracy**, but decreases **model interpretability** because it's no longer possible to visualize the tree to understand the importance of each feature.

However, we can still obtain an overall summary of **feature importance** from bagged models:

- **Bagged regression trees:** calculate the total amount that **MSE** is decreased due to splits over a given feature, averaged over all trees
- **Bagged classification trees:** calculate the total amount that **Gini index** is decreased due to splits over a given feature, averaged over all trees

# Part 4: Combination of classifiers - Majority Voting

  The most typical form of an ensemble is made by combining $T$ different base classifiers.
  Each  base classifier $M(\mathcal{S}_j)$ is trained by applying algorithm $M$ to a random subset
  $\mathcal{S}_j$ of the training set $\mathcal{S}$.
  For simplicity we define $M_j \equiv  M(\mathcal{S}_j)$ for $j=1,\dots,T$, and
  $\mathcal{M}=\{M_j\}_{j=1}^{T}$ a set of base classifiers.
  Then, these models are combined using majority voting to create the ensemble $H$ as follows
  $$
    f_{mv}(\mathcal{S},\mathcal{M}) = max_{c \in \{0,1\}} \sum_{j=1}^T
    \mathbf{1}_c(M_j(\mathcal{S})).
  $$

```python
# read in and prepare the churn data
# Download the dataset
import pandas as pd
import numpy as np

url = 'https://raw.githubusercontent.com/albahnsen/PracticalMachineLearningClass/master/datasets/churn.csv'
data = pd.read_csv(url)

# Create X and y

# Select only the numeric features
X = data.iloc[:, [1,2,6,7,8,9,10]].astype(np.float)
# Convert bools to floats
X = X.join((data.iloc[:, [4,5]] == 'no').astype(np.float))

y = (data.iloc[:, -1] == 'True.').astype(np.int)
```

```python
X.head()
```

      Account Length
      Area Code
      VMail Message
      Day Mins
      Day Calls
      Day Charge
      Eve Mins
      Int'l Plan
      VMail Plan

      0
      128.0
      415.0
      25.0
      265.1
      110.0
      45.07
      197.4
      1.0
      0.0

      1
      107.0
      415.0
      26.0
      161.6
      123.0
      27.47
      195.5
      1.0
      0.0

      2
      137.0
      415.0
      0.0
      243.4
      114.0
      41.38
      121.2
      1.0
      1.0

      3
      84.0
      408.0
      0.0
      299.4
      71.0
      50.90
      61.9
      0.0
      1.0

      4
      75.0
      415.0
      0.0
      166.7
      113.0
      28.34
      148.3
      0.0
      1.0

```python
y.value_counts().to_frame('count').assign(percentage = lambda x: x/x.sum())
```

      count
      percentage

      0
      2850
      0.855086

      1
      483
      0.144914

```python
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.33, random_state=42)
```

Create 100 decision trees

```python
n_estimators = 100
# set a seed for reproducibility
np.random.seed(123)

n_samples = X_train.shape[0]

# create bootstrap samples (will be used to select rows from the DataFrame)
samples = [np.random.choice(a=n_samples, size=n_samples, replace=True) for _ in range(n_estimators)]
```

```python
from sklearn.tree import DecisionTreeClassifier

np.random.seed(123)
seeds = np.random.randint(1, 10000, size=n_estimators)

trees = {}
for i in range(n_estimators):
    trees[i] = DecisionTreeClassifier(max_features="sqrt", max_depth=None, random_state=seeds[i])
    trees[i].fit(X_train.iloc[samples[i]], y_train.iloc[samples[i]])
```

```python
# Predict
y_pred_df = pd.DataFrame(index=X_test.index, columns=list(range(n_estimators)))
for i in range(n_estimators):
    y_pred_df.iloc[:, i] = trees[i].predict(X_test)

y_pred_df.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      90
      91
      92
      93
      94
      95
      96
      97
      98
      99

      438
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0

      2674
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1345
      0
      0
      0
      1
      0
      0
      0
      0
      0
      1
      ...
      0
      0
      0
      1
      1
      0
      0
      1
      1
      0

      1957
      0
      0
      0
      0
      0
      0
      0
      0
      0
      1
      ...
      1
      0
      1
      0
      0
      0
      0
      0
      1
      0

      2148
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      1
      0
      0
      1
      0

5 rows × 100 columns

Predict using majority voting

```python
y_pred_df.sum(axis=1)[:10]
```

    438      2
    2674     5
    1345    35
    1957    17
    2148     3
    3106     4
    1786    22
    321      6
    3082    10
    2240     5
    dtype: int64

```python
y_pred = (y_pred_df.sum(axis=1) >= (n_estimators / 2)).astype(np.int)

from sklearn import metrics
metrics.f1_score(y_pred, y_test)
```

    0.5245901639344264

```python
metrics.accuracy_score(y_pred, y_test)
```

    0.8945454545454545

### Using majority voting with sklearn

```python
from sklearn.ensemble import BaggingClassifier
clf = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=100, bootstrap=True,
                        random_state=42, n_jobs=-1, oob_score=True)
```

```python
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.536, 0.8945454545454545)

# Part 5: Combination of classifiers - Weighted Voting

The majority voting approach gives the same weight to each classfier regardless of the performance of each one. Why not take into account the oob performance of each classifier

First, in the traditional approach, a
similar comparison of the votes of the base classifiers is made, but giving a weight $\alpha_j$
to each classifier $M_j$ during the voting phase
$$
  f_{wv}(\mathcal{S},\mathcal{M}, \alpha)
  =\max_{c \in \{0,1\}} \sum_{j=1}^T \alpha_j \mathbf{1}_c(M_j(\mathcal{S})),
$$
where $\alpha=\{\alpha_j\}_{j=1}^T$.
The calculation of $\alpha_j$ is related to the performance of each classifier $M_j$.
It is usually defined as the normalized misclassification error   $\epsilon$ of the base
classifier $M_j$  in the out of bag set   $\mathcal{S}_j^{oob}=\mathcal{S}-\mathcal{S}_j$
\begin{equation}
  \alpha_j=\frac{1-\epsilon(M_j(\mathcal{S}_j^{oob}))}{\sum_{j_1=1}^T
  1-\epsilon(M_{j_1}(\mathcal{S}_{j_1}^{oob}))}.
\end{equation}

Select each oob sample

```python
samples_oob = []
# show the "out-of-bag" observations for each sample
for sample in samples:
    samples_oob.append(sorted(set(range(n_samples)) - set(sample)))
```

Estimate the oob error of each classifier

```python
errors = np.zeros(n_estimators)

for i in range(n_estimators):
    y_pred_ = trees[i].predict(X_train.iloc[samples_oob[i]])
    errors[i] = 1 - metrics.accuracy_score(y_train.iloc[samples_oob[i]], y_pred_)
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')

plt.scatter(range(n_estimators), errors)
plt.xlim([0, n_estimators])
plt.title('OOB error of each tree')
```

Estimate $\alpha$

```python
alpha = (1 - errors) / (1 - errors).sum()
```

```python
weighted_sum_1 = ((y_pred_df) * alpha).sum(axis=1)
```

```python
weighted_sum_1.head(20)
```

    438     0.019993
    2674    0.050009
    1345    0.350236
    1957    0.170230
    2148    0.030047
    3106    0.040100
    1786    0.219819
    321     0.059707
    3082    0.100178
    2240    0.050128
    1910    0.180194
    2124    0.190111
    2351    0.049877
    1736    0.950014
    879     0.039378
    785     0.219632
    2684    0.010104
    787     0.710568
    170     0.220390
    1720    0.020166
    dtype: float64

```python
y_pred = (weighted_sum_1 >= 0.5).astype(np.int)

metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.5267489711934156, 0.8954545454545455)

### Using Weighted voting with sklearn

```python
clf = BaggingClassifier(base_estimator=DecisionTreeClassifier(), n_estimators=100, bootstrap=True,
                        random_state=42, n_jobs=-1, oob_score=True)
clf.fit(X_train, y_train)
y_pred = clf.predict(X_test)
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.536, 0.8945454545454545)

```python
errors = np.zeros(clf.n_estimators)
y_pred_all_ = np.zeros((X_test.shape[0], clf.n_estimators))

for i in range(clf.n_estimators):
    oob_sample = ~clf.estimators_samples_[i]
    y_pred_ = clf.estimators_[i].predict(X_train.values[oob_sample])
    errors[i] = metrics.accuracy_score(y_pred_, y_train.values[oob_sample])
    y_pred_all_[:, i] = clf.estimators_[i].predict(X_test)

alpha = (1 - errors) / (1 - errors).sum()
y_pred = (np.sum(y_pred_all_ * alpha, axis=1) >= 0.5).astype(np.int)
```

```python
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.5533596837944664, 0.8972727272727272)

# Part 5: Combination of classifiers - Stacking

The staking method consists in combining the different base classifiers by learning a
second level algorithm on top of them. In this framework, once the base
classifiers are constructed using the training set  $\mathcal{S}$, a new set is constructed
where the output of the base classifiers  are now considered as the features while keeping the
class labels.

Even though there is no restriction on which algorithm can be used as a second level learner,
it is common to use a linear model, such as
$$
  f_s(\mathcal{S},\mathcal{M},\beta) =
  g \left( \sum_{j=1}^T \beta_j M_j(\mathcal{S}) \right),
$$
where $\beta=\{\beta_j\}_{j=1}^T$, and $g(\cdot)$ is the sign function
$g(z)=sign(z)$ in the case of a linear regression or the sigmoid function, defined
as $g(z)=1/(1+e^{-z})$, in the case of a logistic regression.

Lets first get a new training set consisting of the output of every classifier

```python
X_train_2 = pd.DataFrame(index=X_train.index, columns=list(range(n_estimators)))

for i in range(n_estimators):
    X_train_2[i] = trees[i].predict(X_train)
```

```python
X_train_2.head()
```

      0
      1
      2
      3
      4
      5
      6
      7
      8
      9
      ...
      90
      91
      92
      93
      94
      95
      96
      97
      98
      99

      2360
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1412
      0
      0
      1
      0
      0
      0
      0
      0
      0
      0
      ...
      1
      0
      0
      0
      0
      0
      0
      0
      0
      0

      1404
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      1
      0
      0
      0
      0
      0

      626
      1
      1
      0
      1
      1
      1
      1
      1
      1
      1
      ...
      1
      1
      1
      1
      1
      1
      1
      1
      1
      1

      347
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0
      ...
      0
      0
      0
      0
      0
      0
      0
      0
      0
      0

5 rows × 100 columns

```python
from sklearn.linear_model import LogisticRegressionCV
```

```python
lr = LogisticRegressionCV(cv = 5 )
lr.fit(X_train_2, y_train)
```

    LogisticRegressionCV(Cs=10, class_weight=None, cv=5, dual=False,
               fit_intercept=True, intercept_scaling=1.0, max_iter=100,
               multi_class='ovr', n_jobs=1, penalty='l2', random_state=None,
               refit=True, scoring=None, solver='lbfgs', tol=0.0001, verbose=0)

```python
lr.coef_
```

    array([[0.10093164, 0.10422116, 0.09431398, 0.09652653, 0.09708914,
            0.09902868, 0.11099774, 0.09661942, 0.09339793, 0.09113735,
            0.10012431, 0.09821555, 0.09383391, 0.09553896, 0.09147924,
            0.09649782, 0.08966216, 0.09196528, 0.09684185, 0.09020504,
            0.0983949 , 0.09514009, 0.10446051, 0.10029114, 0.09671778,
            0.09725594, 0.10912314, 0.10590886, 0.10274701, 0.10275977,
            0.10607442, 0.09803138, 0.1031967 , 0.09266065, 0.09702167,
            0.095245  , 0.08884686, 0.0996088 , 0.09053837, 0.09010279,
            0.09905727, 0.09880662, 0.10538906, 0.09584236, 0.09633239,
            0.09001192, 0.09181503, 0.08995192, 0.10130381, 0.10827454,
            0.10065035, 0.09770659, 0.08922769, 0.10078159, 0.10173676,
            0.10522662, 0.0974279 , 0.09597549, 0.08932533, 0.1003361 ,
            0.10345933, 0.1014522 , 0.09016942, 0.10348487, 0.09335792,
            0.09796407, 0.10166743, 0.09307337, 0.09538791, 0.10997033,
            0.09352554, 0.09860746, 0.10597265, 0.09583425, 0.0982285 ,
            0.09994926, 0.10224051, 0.10065239, 0.10209171, 0.11258262,
            0.09956141, 0.11516098, 0.09798579, 0.10092722, 0.10149644,
            0.10275359, 0.09181294, 0.09903724, 0.10016702, 0.10146037,
            0.09848365, 0.10322647, 0.09913428, 0.08925698, 0.0994986 ,
            0.10277998, 0.09249995, 0.09541316, 0.10532089, 0.09850201]])

```python
y_pred = lr.predict(y_pred_df)
```

```python
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.5365853658536585, 0.8963636363636364)

### Using sklearn

```python
y_pred_all_ = np.zeros((X_test.shape[0], clf.n_estimators))
X_train_3 = np.zeros((X_train.shape[0], clf.n_estimators))

for i in range(clf.n_estimators):

    X_train_3[:, i] = clf.estimators_[i].predict(X_train)
    y_pred_all_[:, i] = clf.estimators_[i].predict(X_test)

lr = LogisticRegressionCV(cv=5)
lr.fit(X_train_3, y_train)

y_pred = lr.predict(y_pred_all_)
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.5625000000000001, 0.8981818181818182)

vs using only one dt

```python
dt = DecisionTreeClassifier()
dt.fit(X_train, y_train)
y_pred = dt.predict(X_test)
metrics.f1_score(y_pred, y_test), metrics.accuracy_score(y_pred, y_test)
```

    (0.44510385756676557, 0.83)


## Contents

* amortized variational inference (VAEs)
* improving amortized variational inference using KL annleaing
* improving amortized variational inference using constraint optimization

Contact: mihaelacr@google.com

Check out the slides for this colab [here](https://github.com/eeml2019/PracticalSessions/tree/master/unsupervised).

```
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import math

import tensorflow as tf
import sonnet as snt

import numpy as np

# Plotting library.
from matplotlib import pyplot as plt
import seaborn as sns

# Tensorflow probability utilities
import tensorflow_probability as tfp

tfd = tfp.distributions

```

    WARNING: Logging before flag parsing goes to stderr.
    W0708 13:08:49.969654 140497286846336 deprecation_wrapper.py:119] From /usr/local/lib/python2.7/dist-packages/sonnet/python/custom_getters/restore_initializer.py:27: The name tf.GraphKeys is deprecated. Please use tf.compat.v1.GraphKeys instead.

    W0708 13:08:52.114116 140497286846336 lazy_loader.py:50]
    The TensorFlow contrib module will not be included in TensorFlow 2.0.
    For more information, please see:
      * https://github.com/tensorflow/community/blob/master/rfcs/20180907-contrib-sunset.md
      * https://github.com/tensorflow/addons
      * https://github.com/tensorflow/io (for I/O related ops)
    If you depend on functionality not listed there, please file an issue.

```
sns.set(rc={"lines.linewidth": 2.8}, font_scale=2)
sns.set_style("whitegrid")
```

```
# Don't forget to select GPU runtime environment in Runtime -> Change runtime type
device_name = tf.test.gpu_device_name()
if device_name != '/device:GPU:0':
  raise SystemError('GPU device not found')
print('Found GPU at: {}'.format(device_name))
```

    Found GPU at: /device:GPU:0

```
def gallery(array, ncols=10, rescale=False):
    """Data visualization code."""
    if rescale:
      array = (array + 1.) / 2
    nindex, height, width, intensity = array.shape
    nrows = nindex//ncols
    assert nindex == nrows*ncols
    # want result.shape = (height*nrows, width*ncols, intensity)
    result = (array.reshape(nrows, ncols, height, width, intensity)
              .swapaxes(1,2)
              .reshape(height*nrows, width*ncols, intensity))
    return result
```

```
def show_digits(axis, digits, title=''):
  axis.axis('off')
  ncols = int(np.sqrt(digits.shape[0]))
  axis.imshow(gallery(digits, ncols=ncols).squeeze(axis=2),
                 cmap='gray')
  axis.set_title(title, fontsize=15)
```

```
def show_latent_interpolations(generator, prior, session):
  a = np.linspace(0.0, 1.0, BATCH_SIZE)
  a = np.expand_dims(a, axis=1)

  first_latents = prior.sample()[0]
  second_latents = prior.sample()[0]

  # To ensure that the interpolation is still likely under the Gaussian prior,
  # we use Gaussian interpolation - rather than linear interpolation.
  interpolations =  np.sqrt(a) * first_latents + np.sqrt(1 - a) * second_latents

  ncols = int(np.sqrt(BATCH_SIZE))
  samples_from_interpolations = generator(interpolations)
  samples_from_interpolations_np = sess.run(samples_from_interpolations)
  plt.gray()
  axis = plt.gca()
  show_digits(
     axis, samples_from_interpolations_np, title='Latent space interpolations')
```

## Hyperparameters

```
BATCH_SIZE = 64
NUM_LATENTS = 10
TRAINING_STEPS = 10000
NUM_SVI_UPDATES = 100
```

```
DECODER_VARIABLE_SCOPE = "decoder"
```

```
tf.reset_default_graph()
```

## Get the data

We will use the MNIST dataset. Luckly, TensorFlow comes with a simple way to load it.

```
mnist = tf.contrib.learn.datasets.load_dataset("mnist")
```

    W0708 13:08:53.497503 140497286846336 deprecation.py:323] From <ipython-input-10-5f45bee47b71>:1: load_dataset (from tensorflow.contrib.learn.python.learn.datasets) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use tf.data.
    W0708 13:08:53.499013 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/__init__.py:80: load_mnist (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
    W0708 13:08:53.500588 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:300: read_data_sets (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.
    W0708 13:08:53.501789 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:260: maybe_download (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please write your own downloading logic.
    W0708 13:08:53.506086 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/base.py:252: wrapped_fn (from tensorflow.contrib.learn.python.learn.datasets.base) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use urllib or similar directly.
    W0708 13:08:53.927912 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:262: extract_images (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use tf.data to implement this functionality.

    Successfully downloaded train-images-idx3-ubyte.gz 9912422 bytes.
    Extracting MNIST-data/train-images-idx3-ubyte.gz

    W0708 13:08:54.442007 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:267: extract_labels (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use tf.data to implement this functionality.

    Successfully downloaded train-labels-idx1-ubyte.gz 28881 bytes.
    Extracting MNIST-data/train-labels-idx1-ubyte.gz
    Successfully downloaded t10k-images-idx3-ubyte.gz 1648877 bytes.
    Extracting MNIST-data/t10k-images-idx3-ubyte.gz

    W0708 13:08:54.833554 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/contrib/learn/python/learn/datasets/mnist.py:290: __init__ (from tensorflow.contrib.learn.python.learn.datasets.mnist) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use alternatives such as official/mnist/dataset.py from tensorflow/models.

    Successfully downloaded t10k-labels-idx1-ubyte.gz 4542 bytes.
    Extracting MNIST-data/t10k-labels-idx1-ubyte.gz

```
print(mnist.train.images.shape)
print(type(mnist.train.images))
```

    (55000, 784)
    <type 'numpy.ndarray'>

### Transform the data from numpy arrays to in graph tensors.

This allows us to use TensorFlow datasets, which ensure that a new batch from the data is being fed at each session.run. This means that we do not need to use feed_dicts to feed data to each session.

```
def make_tf_data_batch(np_data, shuffle=True):
  # Reshape the data to image size.
  images = np_data.reshape((-1, 28, 28, 1))

  # Create the TF dataset.
  dataset = tf.data.Dataset.from_tensor_slices(images)

  # Shuffle and repeat the dataset for training.
  # This is required because we want to do multiple passes through the entire
  # dataset when training.
  if shuffle:
    dataset = dataset.shuffle(100000).repeat()

  # Batch the data and return the data batch.
  one_shot_iterator = dataset.batch(BATCH_SIZE).make_one_shot_iterator()
  data_batch = one_shot_iterator.get_next()
  return data_batch
```

```
real_data = make_tf_data_batch(mnist.train.images)
print(real_data.shape)
```

    W0708 13:08:56.027748 140497286846336 deprecation.py:323] From <ipython-input-12-54f78434392b>:15: make_one_shot_iterator (from tensorflow.python.data.ops.dataset_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use `for ... in dataset:` to iterate over a dataset. If using `tf.estimator`, return the `Dataset` object directly from your input function. As a last resort, you can use `tf.compat.v1.data.make_one_shot_iterator(dataset)`.

    (?, 28, 28, 1)

# Amortized variational inference (VAEs)

Instead of learning one set of posterior variables per data point, we can use function approximation to learn the distributional variables. Specifically, the posterior parameters for $x_i$ will be the output of a *learned* function $f_\theta(x_i)$, where $\theta$ are parameters shared across all data points. Can you think of why this is useful?

Objective - maximize:
\begin{equation}
 \mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x) \log p_\theta(x|z)}  - \mathbb{E}_{p^*(x)} KL(q(z|x)||p(z))
\end{equation}

For more information, see:
  * https://arxiv.org/abs/1312.6114

## Define the decoder

```
def standard_decoder(z):
  with tf.variable_scope(DECODER_VARIABLE_SCOPE, reuse=tf.AUTO_REUSE):
    h = tf.layers.dense(z, 7 * 7 * 64, activation=tf.nn.relu)
    h = tf.reshape(h, shape=[BATCH_SIZE, 7, 7, 64])
    h = tf.layers.Conv2DTranspose(
          filters=32,
          kernel_size=5,
          strides=2,
          activation=tf.nn.relu,
          padding='same')(h)
    h = tf.layers.Conv2DTranspose(
          filters=1,
          kernel_size=5,
          strides=2,
          activation=None,  # Do not activate the last layer.
          padding='same')(h)
    return tf.distributions.Bernoulli(h)
```

```
def big_decoder(z):
  with tf.variable_scope(DECODER_VARIABLE_SCOPE, reuse=tf.AUTO_REUSE):
    h = tf.layers.dense(z, 4000, activation=tf.nn.relu)
    h = tf.layers.dense(z, 4000, activation=tf.nn.relu)
    h = tf.layers.dense(z, 4000, activation=tf.nn.relu)
    h = tf.layers.dense(h, 7 * 7 * 64, activation=tf.nn.relu)
    h = tf.reshape(h, shape=[BATCH_SIZE, 7, 7, 64])
    h = tf.layers.Conv2DTranspose(
          filters=32,
          kernel_size=5,
          strides=2,
          activation=tf.nn.relu,
          padding='same')(h)
    h = tf.layers.Conv2DTranspose(
          filters=32,
          kernel_size=5,
          strides=1,
          activation=tf.nn.relu,
          padding='same')(h)
    h = tf.layers.Conv2DTranspose(
          filters=32,
          kernel_size=5,
          strides=1,
          activation=tf.nn.relu,
          padding='same')(h)
    h = tf.layers.Conv2DTranspose(
          filters=1,
          kernel_size=5,
          strides=2,
          activation=None,  # Do not activate the last layer.
          padding='same')(h)
    return tf.distributions.Bernoulli(h)
```

```
ENCODER_VARIABLE_SCOPE = 'encoder'
```

```
def encoder(x):
  with tf.variable_scope(ENCODER_VARIABLE_SCOPE, reuse=tf.AUTO_REUSE):

    h = tf.layers.Conv2D(
          filters=8,
          kernel_size=5,
          strides=2,
          activation=tf.nn.relu,
          padding='same')(x)
    h = tf.layers.Conv2D(
          filters=16,
          kernel_size=5,
          strides=2,
          activation=tf.nn.relu,
          padding='same')(h)
    h = tf.layers.Conv2D(
          filters=32,
          kernel_size=5,
          strides=1,
          activation=tf.nn.relu,
          padding='same')(h)

    out_shape = 1
    for s in h.shape.as_list()[1:]:
      out_shape*= s

    h = tf.reshape(h, shape=[BATCH_SIZE, out_shape])
    mean = tf.layers.dense(h, NUM_LATENTS, activation=None)
    scale = tf.layers.dense(h, NUM_LATENTS, activation=None)
    return multi_normal(loc=mean, log_scale=scale)
```

```
def multi_normal(loc, log_scale):
  # We model the latent variables as independent
  return tfd.Independent(
      distribution=tfd.Normal(loc=loc, scale=tf.exp(log_scale)),
      reinterpreted_batch_ndims=1)
```

```
def make_prior():
  # Zero mean, unit variance prior.
  prior_mean = tf.zeros(shape=(BATCH_SIZE, NUM_LATENTS), dtype=tf.float32)
  prior_log_scale = tf.zeros(shape=(BATCH_SIZE, NUM_LATENTS), dtype=tf.float32)

  return multi_normal(prior_mean, prior_log_scale)
```

```
def bound_terms(data_batch, variational_posterior, decoder_fn):
  latent_sample = variational_posterior.sample()
  all_log_probs = decoder_fn(latent_sample).log_prob(data_batch)

  # Since we used broadcasting, log_prob of a Bernoulli will return a log_prob for
  # each dimension.
  all_log_probs.shape.assert_is_compatible_with([BATCH_SIZE, 28, 28, 1])

  # Reduce sum over the data dimensions
  likelihood_term = tf.reduce_sum(all_log_probs, axis=[1, 2, 3])

  # Reduce mean over the batch dimensions
  likelihood_term = tf.reduce_mean(likelihood_term)

  kl_term = variational_posterior.kl_divergence(prior)
  kl_term.shape.assert_is_compatible_with([BATCH_SIZE])

  # Reduce over the batch dimension.
  kl_term = tf.reduce_mean(kl_term)

  return likelihood_term, kl_term
```

```
real_data = make_tf_data_batch(mnist.train.images)
print(real_data.shape)
```

    (?, 28, 28, 1)

```
prior = make_prior()
```

```
# Note: we no longer have to use a variable to store the data.
# We will perform one encoder update per decoder update, so it is OK for the
# data batch to be refreshed at each run.
variational_posterior = encoder(real_data)
decoder = standard_decoder
```

    W0708 13:09:00.230885 140497286846336 deprecation.py:506] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling __init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Call initializer instance with the dtype argument instead of passing it to the constructor
    W0708 13:09:00.533052 140497286846336 deprecation.py:323] From <ipython-input-17-997b61b0c8ff>:28: dense (from tensorflow.python.layers.core) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use keras.layers.dense instead.

```
# Maximize the data likelihodd and minimize the KL divergence between the prior
# and posterior. We use the exact same loss as in the SVI case.
likelihood_term, kl_term = bound_terms(
    real_data, variational_posterior, decoder)
train_elbo = likelihood_term - kl_term
loss =  - train_elbo
```

    W0708 13:09:01.631365 140497286846336 deprecation.py:323] From <ipython-input-14-a043c76715d6>:17: __init__ (from tensorflow.python.ops.distributions.bernoulli) is deprecated and will be removed after 2019-01-01.
    Instructions for updating:
    The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
    W0708 13:09:01.633912 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/distributions/bernoulli.py:97: __init__ (from tensorflow.python.ops.distributions.distribution) is deprecated and will be removed after 2019-01-01.
    Instructions for updating:
    The TensorFlow Distributions library has moved to TensorFlow Probability (https://github.com/tensorflow/probability). You should update all references to use `tfp.distributions` instead of `tf.distributions`.
    W0708 13:09:01.646147 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/nn_impl.py:180: where (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where

```
samples = decoder(prior.sample()).mean()
samples.shape.assert_is_compatible_with([BATCH_SIZE, 28, 28, 1])

reconstructions = decoder(variational_posterior.sample()).mean()
```

```
# We now perform joint optimization on the encoder and decoder variables.
optimizer = tf.train.AdamOptimizer(0.001, beta1=0.9, beta2=0.9)
decoder_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES, scope=DECODER_VARIABLE_SCOPE)
encoder_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES, scope=ENCODER_VARIABLE_SCOPE)
autoencoder_vars = encoder_vars + decoder_vars
update_op = optimizer.minimize(loss, var_list=autoencoder_vars)
```

## Training

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())
```

    W0708 13:09:02.493182 140497286846336 deprecation.py:323] From /usr/local/lib/python2.7/dist-packages/tensorflow/python/util/tf_should_use.py:193: initialize_all_variables (from tensorflow.python.ops.variables) is deprecated and will be removed after 2017-03-02.
    Instructions for updating:
    Use `tf.global_variables_initializer` instead.

```
# %hide_pyerr  # - uncomment to interrupt training without a stacktrace
losses = []
kls = []
likelihood_terms = []

for i in xrange(TRAINING_STEPS):
  sess.run(update_op)

  if i % 100 == 0:
    iteration_loss, iteration_likelihood, iteration_kl  = sess.run(
          [loss, likelihood_term, kl_term])
    print('Iteration {}. Loss {}. KL {}'.format(
          i, iteration_loss, iteration_kl))
    losses.append(iteration_loss)
    kls.append(iteration_kl)
    likelihood_terms.append(iteration_likelihood)
```

    Iteration 0. Loss 543.149597168. KL 0.0311325900257
    Iteration 100. Loss 212.25730896. KL 5.6214094162
    Iteration 200. Loss 174.790023804. KL 7.67844295502
    Iteration 300. Loss 142.881057739. KL 13.4385919571
    Iteration 400. Loss 135.431091309. KL 16.2140541077
    Iteration 500. Loss 122.805511475. KL 16.5219306946
    Iteration 600. Loss 129.848358154. KL 17.4595108032
    Iteration 700. Loss 119.759521484. KL 16.9787559509
    Iteration 800. Loss 121.922439575. KL 18.7446975708
    Iteration 900. Loss 118.180030823. KL 18.2364273071
    Iteration 1000. Loss 114.578315735. KL 18.1579208374
    Iteration 1100. Loss 119.28993988. KL 18.9148769379
    Iteration 1200. Loss 112.884460449. KL 18.0944690704
    Iteration 1300. Loss 110.759017944. KL 18.4087028503
    Iteration 1400. Loss 111.300521851. KL 17.4910926819
    Iteration 1500. Loss 111.921302795. KL 18.8771114349
    Iteration 1600. Loss 110.051803589. KL 18.0699653625
    Iteration 1700. Loss 108.457572937. KL 17.9535446167
    Iteration 1800. Loss 108.490524292. KL 19.3655204773
    Iteration 1900. Loss 100.225669861. KL 18.7643070221
    Iteration 2000. Loss 111.785705566. KL 19.4207115173
    Iteration 2100. Loss 104.484207153. KL 18.5088977814
    Iteration 2200. Loss 115.497138977. KL 18.7223434448
    Iteration 2300. Loss 114.926834106. KL 18.7673416138
    Iteration 2400. Loss 106.025680542. KL 18.5278453827
    Iteration 2500. Loss 105.173126221. KL 18.6448173523
    Iteration 2600. Loss 109.301971436. KL 18.9525566101
    Iteration 2700. Loss 108.767593384. KL 19.2736968994
    Iteration 2800. Loss 111.36567688. KL 18.4386901855
    Iteration 2900. Loss 109.312240601. KL 19.2203826904
    Iteration 3000. Loss 104.537918091. KL 19.1003456116
    Iteration 3100. Loss 110.757202148. KL 19.0079498291
    Iteration 3200. Loss 112.999229431. KL 18.8419589996
    Iteration 3300. Loss 109.117324829. KL 18.6015930176
    Iteration 3400. Loss 111.633560181. KL 19.2635688782
    Iteration 3500. Loss 104.650123596. KL 19.2261276245
    Iteration 3600. Loss 107.909301758. KL 18.4298820496
    Iteration 3700. Loss 107.513908386. KL 19.2823562622
    Iteration 3800. Loss 99.3676986694. KL 17.9559078217
    Iteration 3900. Loss 106.782066345. KL 19.6768741608
    Iteration 4000. Loss 110.721343994. KL 18.8887825012
    Iteration 4100. Loss 107.074325562. KL 18.9987812042
    Iteration 4200. Loss 106.425239563. KL 19.6298599243
    Iteration 4300. Loss 106.66569519. KL 19.651052475
    Iteration 4400. Loss 111.59274292. KL 19.445810318
    Iteration 4500. Loss 102.762084961. KL 19.0068473816
    Iteration 4600. Loss 108.971916199. KL 19.381111145
    Iteration 4700. Loss 107.235610962. KL 19.8888092041
    Iteration 4800. Loss 108.421539307. KL 18.8425006866
    Iteration 4900. Loss 109.713317871. KL 20.2253990173
    Iteration 5000. Loss 102.892028809. KL 19.3477287292
    Iteration 5100. Loss 101.523200989. KL 19.5116329193
    Iteration 5200. Loss 105.531776428. KL 19.4702854156
    Iteration 5300. Loss 103.905708313. KL 20.0379638672
    Iteration 5400. Loss 105.870521545. KL 20.0232620239
    Iteration 5500. Loss 109.295715332. KL 19.609998703
    Iteration 5600. Loss 107.15145874. KL 19.2891845703
    Iteration 5700. Loss 108.174583435. KL 19.7180328369
    Iteration 5800. Loss 108.477203369. KL 19.9281005859
    Iteration 5900. Loss 107.612701416. KL 19.5747203827
    Iteration 6000. Loss 107.639404297. KL 19.7801189423
    Iteration 6100. Loss 105.317398071. KL 19.6310386658
    Iteration 6200. Loss 103.571769714. KL 18.7017593384
    Iteration 6300. Loss 103.673660278. KL 19.6920547485
    Iteration 6400. Loss 105.396484375. KL 19.6332759857
    Iteration 6500. Loss 99.8526000977. KL 19.1468315125
    Iteration 6600. Loss 104.205543518. KL 18.9545192719
    Iteration 6700. Loss 99.4845733643. KL 19.6858062744
    Iteration 6800. Loss 104.343544006. KL 19.8048858643
    Iteration 6900. Loss 99.3537597656. KL 18.9388313293
    Iteration 7000. Loss 101.224441528. KL 19.9195518494
    Iteration 7100. Loss 100.044769287. KL 19.2435569763
    Iteration 7200. Loss 104.803375244. KL 19.2749214172
    Iteration 7300. Loss 104.603050232. KL 19.175951004
    Iteration 7400. Loss 101.972198486. KL 18.6278858185
    Iteration 7500. Loss 97.816696167. KL 19.6141624451
    Iteration 7600. Loss 109.782745361. KL 20.5181312561
    Iteration 7700. Loss 105.940322876. KL 20.5412864685
    Iteration 7800. Loss 104.402236938. KL 19.1177558899
    Iteration 7900. Loss 104.112945557. KL 19.043548584
    Iteration 8000. Loss 103.723693848. KL 18.6456279755
    Iteration 8100. Loss 98.1162872314. KL 19.4400405884
    Iteration 8200. Loss 106.893859863. KL 19.2208862305
    Iteration 8300. Loss 100.480285645. KL 19.347366333
    Iteration 8400. Loss 101.485412598. KL 19.6425189972
    Iteration 8500. Loss 103.959197998. KL 19.8362579346
    Iteration 8600. Loss 93.5329742432. KL 18.2440299988
    Iteration 8700. Loss 106.767196655. KL 18.9457569122
    Iteration 8800. Loss 101.940284729. KL 19.9074783325
    Iteration 8900. Loss 102.059799194. KL 19.8515605927
    Iteration 9000. Loss 104.046653748. KL 19.5471572876
    Iteration 9100. Loss 103.739982605. KL 18.6642303467
    Iteration 9200. Loss 103.349441528. KL 19.708568573
    Iteration 9300. Loss 105.681678772. KL 19.9007949829
    Iteration 9400. Loss 105.354469299. KL 20.4583835602
    Iteration 9500. Loss 103.603713989. KL 19.7070903778
    Iteration 9600. Loss 101.325996399. KL 18.7148971558
    Iteration 9700. Loss 104.791824341. KL 19.368938446
    Iteration 9800. Loss 105.478645325. KL 19.5082492828
    Iteration 9900. Loss 103.824905396. KL 20.3832130432

### Visualize the loss in time

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(losses, label='Negative ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].legend()

axes[1].plot(kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(likelihood_terms, label='Likelihood Term')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

### Generate samples and latent interpolations

```
real_data_vals, final_samples_vals, data_reconstructions_vals = sess.run(
      [real_data, samples, reconstructions])
```

```
fig, axes = plt.subplots(1, 3, figsize=(3*4,4))

show_digits(axes[0], real_data_vals, 'Data')
show_digits(axes[1], data_reconstructions_vals, 'Reconstructions')
show_digits(axes[2], final_samples_vals, 'Samples')
```

```
show_latent_interpolations(lambda x: decoder(x).mean(), prior, sess)
```

## Questions about amortized variational inference

* What do you observe about sample quality and reconstruction quality?
* What do you observe about the ELBO and KL term?
* Which gradient estimation method is used to compute the gradients with respect to the encoder parameters?

## Answers
 * Gap between reconstruction and sample quality due to difficulty in latent space distribution matching via KL divergence.
 * KL term goes up during training. Loss (negative elbo) goes down.
 * Reparametrization.

# KL annealing

Objective - maximize:
\begin{equation}
 \mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x) \log p_\theta(x|z)}  - \alpha \mathbb{E}_{p^*(x)} KL(q(z|x)||p(z))
\end{equation}

Where $\alpha$ changes during training, to weigh in the KL term more.

```
tf.reset_default_graph()
```

```
real_data = make_tf_data_batch(mnist.train.images)
prior = make_prior()
decoder = standard_decoder
```

```
kl_coefficient = tf.Variable(
    tf.zeros(shape=(1), dtype=tf.float32),
    trainable=False,
    name='kl_coeff')

kl_step = 1.0 / TRAINING_STEPS
update_kl_coeff = tf.assign(kl_coefficient, kl_coefficient + kl_step)
```

```
variational_posterior = encoder(real_data)# Maximize the data likelihodd and minimize the KL divergence between the prior
# and posterior. We use the exact same loss as in the SVI case.
likelihood_term, kl_term = bound_terms(
    real_data, variational_posterior, decoder)
train_elbo = likelihood_term - kl_coefficient * kl_term
loss =  - train_elbo
```

```
samples = decoder(prior.sample()).mean()
samples.shape.assert_is_compatible_with([BATCH_SIZE, 28, 28, 1])

reconstructions = decoder(variational_posterior.sample()).mean()
```

```
# We now perform joint optimization on the encoder and decoder variables.
optimizer = tf.train.AdamOptimizer(0.001, beta1=0.9, beta2=0.9)
decoder_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES, scope=DECODER_VARIABLE_SCOPE)
encoder_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES, scope=ENCODER_VARIABLE_SCOPE)
autoencoder_vars = encoder_vars + decoder_vars
variables_update_op = optimizer.minimize(loss, var_list=autoencoder_vars)

# Ensure that a variable update is followed by an update in the KL coefficient.
with tf.control_dependencies([variables_update_op]):
  update_op = tf.identity(update_kl_coeff)
```

## Training

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())
```

```
# %hide_pyerr  # - uncomment to interrupt training without a stacktrace
losses = []
kls = []
likelihood_terms = []

for i in xrange(TRAINING_STEPS):
  sess.run(update_op)

  if i % 100 == 0:
    iteration_loss, iteration_likelihood, iteration_kl  = sess.run(
          [loss, likelihood_term, kl_term])
    print('Iteration {}. Loss {}. KL {}'.format(
          i, iteration_loss, iteration_kl))
    losses.append(iteration_loss)
    kls.append(iteration_kl)
    likelihood_terms.append(iteration_likelihood)
```

    Iteration 0. Loss [538.7282]. KL 0.0849007740617
    Iteration 100. Loss [207.6383]. KL 53.8163986206
    Iteration 200. Loss [163.20004]. KL 55.5619697571
    Iteration 300. Loss [126.80246]. KL 67.9112243652
    Iteration 400. Loss [106.59318]. KL 56.4268226624
    Iteration 500. Loss [105.78894]. KL 48.3723678589
    Iteration 600. Loss [95.13674]. KL 47.0271224976
    Iteration 700. Loss [98.70525]. KL 43.0538406372
    Iteration 800. Loss [96.5647]. KL 41.1986846924
    Iteration 900. Loss [90.7644]. KL 38.3425712585
    Iteration 1000. Loss [86.22786]. KL 37.3697891235
    Iteration 1100. Loss [80.22088]. KL 34.7906951904
    Iteration 1200. Loss [84.72531]. KL 33.7538909912
    Iteration 1300. Loss [96.80273]. KL 31.1420459747
    Iteration 1400. Loss [82.68167]. KL 33.0395469666
    Iteration 1500. Loss [93.63019]. KL 30.8407268524
    Iteration 1600. Loss [97.53325]. KL 31.0071678162
    Iteration 1700. Loss [97.375435]. KL 31.1036338806
    Iteration 1800. Loss [95.40447]. KL 30.0751857758
    Iteration 1900. Loss [83.09582]. KL 28.6838340759
    Iteration 2000. Loss [92.76718]. KL 27.8191661835
    Iteration 2100. Loss [87.869316]. KL 29.5407428741
    Iteration 2200. Loss [89.413956]. KL 28.4358310699
    Iteration 2300. Loss [93.984]. KL 27.9070892334
    Iteration 2400. Loss [92.83576]. KL 28.690574646
    Iteration 2500. Loss [91.39958]. KL 27.4778823853
    Iteration 2600. Loss [90.22681]. KL 26.7947235107
    Iteration 2700. Loss [91.12179]. KL 27.6318130493
    Iteration 2800. Loss [91.660446]. KL 26.9869651794
    Iteration 2900. Loss [88.58613]. KL 26.7378807068
    Iteration 3000. Loss [95.650536]. KL 25.8180961609
    Iteration 3100. Loss [91.4081]. KL 26.5886802673
    Iteration 3200. Loss [84.056015]. KL 25.650554657
    Iteration 3300. Loss [95.398254]. KL 25.8928642273
    Iteration 3400. Loss [91.88904]. KL 25.7199287415
    Iteration 3500. Loss [93.8163]. KL 25.7762794495
    Iteration 3600. Loss [89.50023]. KL 26.0213069916
    Iteration 3700. Loss [93.09817]. KL 25.1856880188
    Iteration 3800. Loss [95.68851]. KL 25.0011882782
    Iteration 3900. Loss [93.84276]. KL 26.0200195312
    Iteration 4000. Loss [95.64398]. KL 24.8965587616
    Iteration 4100. Loss [92.84069]. KL 25.6173610687
    Iteration 4200. Loss [96.570465]. KL 24.2456817627
    Iteration 4300. Loss [92.831696]. KL 25.0357170105
    Iteration 4400. Loss [93.47231]. KL 25.8835449219
    Iteration 4500. Loss [96.91273]. KL 25.1160774231
    Iteration 4600. Loss [89.63088]. KL 24.1783161163
    Iteration 4700. Loss [99.182144]. KL 24.9961032867
    Iteration 4800. Loss [90.9977]. KL 23.5637969971
    Iteration 4900. Loss [88.59955]. KL 24.5034503937
    Iteration 5000. Loss [93.72626]. KL 23.578037262
    Iteration 5100. Loss [89.66024]. KL 23.0985145569
    Iteration 5200. Loss [88.95053]. KL 23.2904891968
    Iteration 5300. Loss [98.10858]. KL 23.0870475769
    Iteration 5400. Loss [88.058304]. KL 23.5322914124
    Iteration 5500. Loss [96.32849]. KL 23.6962146759
    Iteration 5600. Loss [91.41071]. KL 24.3380966187
    Iteration 5700. Loss [93.821754]. KL 23.0876960754
    Iteration 5800. Loss [95.6335]. KL 23.6332015991
    Iteration 5900. Loss [89.201904]. KL 22.907661438
    Iteration 6000. Loss [90.392395]. KL 23.7171058655
    Iteration 6100. Loss [94.46007]. KL 23.1119766235
    Iteration 6200. Loss [97.29648]. KL 22.1558513641
    Iteration 6300. Loss [99.77679]. KL 23.06275177
    Iteration 6400. Loss [97.482895]. KL 22.6422805786
    Iteration 6500. Loss [97.07002]. KL 23.0295562744
    Iteration 6600. Loss [98.088425]. KL 22.2883167267
    Iteration 6700. Loss [95.350235]. KL 22.2409420013
    Iteration 6800. Loss [98.70405]. KL 22.4547996521
    Iteration 6900. Loss [101.46624]. KL 23.0755977631
    Iteration 7000. Loss [98.34372]. KL 22.8184280396
    Iteration 7100. Loss [96.793686]. KL 21.6929588318
    Iteration 7200. Loss [95.252594]. KL 21.1330528259
    Iteration 7300. Loss [98.296814]. KL 21.8783473969
    Iteration 7400. Loss [94.1511]. KL 21.441860199
    Iteration 7500. Loss [99.43497]. KL 20.6506614685
    Iteration 7600. Loss [95.55722]. KL 21.0348091125
    Iteration 7700. Loss [100.44975]. KL 21.2759113312
    Iteration 7800. Loss [100.59696]. KL 21.7520713806
    Iteration 7900. Loss [89.30542]. KL 20.3898582458
    Iteration 8000. Loss [101.869576]. KL 20.3803787231
    Iteration 8100. Loss [98.239204]. KL 21.0610351562
    Iteration 8200. Loss [102.14479]. KL 22.0289936066
    Iteration 8300. Loss [94.70732]. KL 20.2863121033
    Iteration 8400. Loss [101.938614]. KL 21.7867889404
    Iteration 8500. Loss [95.559265]. KL 20.5674705505
    Iteration 8600. Loss [100.93642]. KL 20.6077346802
    Iteration 8700. Loss [102.08324]. KL 20.9716529846
    Iteration 8800. Loss [98.366425]. KL 21.2855529785
    Iteration 8900. Loss [100.21767]. KL 20.3810443878
    Iteration 9000. Loss [108.61808]. KL 20.4927768707
    Iteration 9100. Loss [101.76357]. KL 20.1575584412
    Iteration 9200. Loss [106.28869]. KL 20.5190467834
    Iteration 9300. Loss [105.35385]. KL 20.2669067383
    Iteration 9400. Loss [98.641266]. KL 20.0853347778
    Iteration 9500. Loss [96.2671]. KL 20.2059440613
    Iteration 9600. Loss [101.07809]. KL 20.3420009613
    Iteration 9700. Loss [103.353165]. KL 20.0434856415
    Iteration 9800. Loss [101.18288]. KL 18.7720680237
    Iteration 9900. Loss [101.64004]. KL 18.4157562256

```
fig, axes = plt.subplots(1, 3, figsize=(3*8,5))

axes[0].plot(losses, label='Negative ELBO')
axes[0].set_title('Time', fontsize=15)
axes[0].legend()

axes[1].plot(kls, label='KL')
axes[1].set_title('Time', fontsize=15)
axes[1].legend()

axes[2].plot(likelihood_terms, label='Likelihood Term')
axes[2].set_title('Time', fontsize=15)
axes[2].legend()
```

```
real_data_vals, final_samples_vals, data_reconstructions_vals = sess.run(
      [real_data, samples, reconstructions])
```

```
fig, axes = plt.subplots(1, 3, figsize=(3*4,4))

show_digits(axes[0], real_data_vals, 'Data')
show_digits(axes[1], data_reconstructions_vals, 'Reconstructions')
show_digits(axes[2], final_samples_vals, 'Samples')
```

```
show_latent_interpolations(lambda x: decoder(x).mean(), prior, sess)
```

## Questions about KL annealing

* What do you observe about the KL behaviour throughout training as opposed to amortized variational inference without any KL annealing?

# Constrained optimization

Instead of using KL annealing, constrained optimization can be used to automatically tune the relative weight of the likelihood and kl terms. This removes the need to manually create an optimization schedule, which can be problem specific.

The objective now becomes:

\begin{equation}
  \text{minimize } \mathbb{E}_{p^*(x)} KL(q(z|x)||p(z)) \text{ such that }  \mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x) \log p_\theta(x|z)} > \alpha
\end{equation}

This can be solved using the use of Lagrange multipliers. The objective becomes:

\begin{equation}
  \text{minimize }  \mathbb{E}_{p^*(x)} KL(q(z|x)||p(z)) + \lambda  (\mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x)} (\alpha - \log p_\theta(x|z)))
\end{equation}

The difference compared to the KL annealing is that:

   * $\lambda$ is a learned parameter - it will be learned using stochastic gradient descent, like the network parameters.  The difference is that the lagrangian has to solve a maximization problem. You can see this intuitively: the graadient with respect to $\lambda$ in the objective above is $\mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x)} (\alpha - \log p_\theta(x|z))$. If $ \mathbb{E}_{p^*(x)} \mathbb{E}_{q(z|x)} (\alpha - \log p_\theta(x|z))> 0$, the constraint is not being satisfied, so the value of the lagrangian needs to increase. This will be done by doing gradient ascent, instead of gradient descent. Note that for $\lambda$ to be a valid lagranian in a minimization problem, it has to be positive.
   * The practicioner has to specify the hyperparameter $\alpha$, which determines the reoncstruction quality of the model.
   * the coefficient is in front of the likelihood term, not the KL term. This is mainly for convenience, as it is easier to specify the hyperparameter $\alpha$ for the likelihood (reconstruction loss).

For more assumptions made by this method, see the Karush–Kuhn–Tucker conditions.

For more information, see:
  * http://bayesiandeeplearning.org/2018/papers/33.pdf

  If you want to use this in practice, I suggest the implementation [here](https://github.com/deepmind/sonnet/blob/master/sonnet/python/modules/optimization_constraints.py).

```
tf.reset_default_graph()
```

```
real_data = make_tf_data_batch(mnist.train.images)
prior = make_prior()
decoder = standard_decoder
```

```
# Unlike in the KL annealing case, we learn the coefficient.
# We initialize the lagrangian such that after the softplus the coefficient is
# approximately 1.
lagrangian_var = tf.Variable(
    0.55 * tf.ones(shape=(1), dtype=tf.float32),
    trainable=True,
    name='lagrangian')

# Ensure that the lagrangian is positive and has stable dynamics.
lagrangian = tf.nn.softplus(lagrangian_var)
```

```
# How good do we want the reconstruction loss to be?
# We can look at previous runs to get an idea what a reasonable value would be.
reconstruction_target = - 90
```

```
variational_posterior = encoder(real_data)
likelihood_term, kl_term = bound_terms(
    real_data, variational_posterior, decoder)
# Note: now the elbo is a different quanitity to what we optimize.
train_elbo = likelihood_term - kl_term
loss =  kl_term + lagrangian * (reconstruction_target - likelihood_term )
```

```
samples = decoder(prior.sample()).mean()
samples.shape.assert_is_compatible_with([BATCH_SIZE, 28, 28, 1])

reconstructions = decoder(variational_posterior.sample()).mean()
```

```
tf.trainable_variables()
```

    [<tf.Variable 'lagrangian:0' shape=(1,) dtype=float32_ref>,
     <tf.Variable 'encoder/conv2d/kernel:0' shape=(5, 5, 1, 8) dtype=float32_ref>,
     <tf.Variable 'encoder/conv2d/bias:0' shape=(8,) dtype=float32_ref>,
     <tf.Variable 'encoder/conv2d_1/kernel:0' shape=(5, 5, 8, 16) dtype=float32_ref>,
     <tf.Variable 'encoder/conv2d_1/bias:0' shape=(16,) dtype=float32_ref>,
     <tf.Variable 'encoder/conv2d_2/kernel:0' shape=(5, 5, 16, 32) dtype=float32_ref>,
     <tf.Variable 'encoder/conv2d_2/bias:0' shape=(32,) dtype=float32_ref>,
     <tf.Variable 'encoder/dense/kernel:0' shape=(1568, 10) dtype=float32_ref>,
     <tf.Variable 'encoder/dense/bias:0' shape=(10,) dtype=float32_ref>,
     <tf.Variable 'encoder/dense_1/kernel:0' shape=(1568, 10) dtype=float32_ref>,
     <tf.Variable 'encoder/dense_1/bias:0' shape=(10,) dtype=float32_ref>,
     <tf.Variable 'decoder/dense/kernel:0' shape=(10, 3136) dtype=float32_ref>,
     <tf.Variable 'decoder/dense/bias:0' shape=(3136,) dtype=float32_ref>,
     <tf.Variable 'decoder/conv2d_transpose/kernel:0' shape=(5, 5, 32, 64) dtype=float32_ref>,
     <tf.Variable 'decoder/conv2d_transpose/bias:0' shape=(32,) dtype=float32_ref>,
     <tf.Variable 'decoder/conv2d_transpose_1/kernel:0' shape=(5, 5, 1, 32) dtype=float32_ref>,
     <tf.Variable 'decoder/conv2d_transpose_1/bias:0' shape=(1,) dtype=float32_ref>]

```
lagrangian_optimizer = tf.train.GradientDescentOptimizer(0.001)

# We now perform joint optimization on the encoder and decoder variables.
optimizer = tf.train.AdamOptimizer(0.001, beta1=0.9, beta2=0.9)
decoder_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES, scope=DECODER_VARIABLE_SCOPE)
encoder_vars = tf.get_collection(
    tf.GraphKeys.TRAINABLE_VARIABLES, scope=ENCODER_VARIABLE_SCOPE)
autoencoder_vars = encoder_vars + decoder_vars
autoencoder_variables_update_op = optimizer.minimize(
      loss, var_list=autoencoder_vars)

# Ensure that a variable update is followed by an update to the Lagrangian.
with tf.control_dependencies([autoencoder_variables_update_op]):
  # Ensure that the lagrangian solves a maximization problem instead of a
  # minimization problem by changing the sign of the loss function.
  update_op = lagrangian_optimizer.minimize(- loss, var_list=[lagrangian_var])
```

## Training

```
sess = tf.Session()

# Initialize all variables
sess.run(tf.initialize_all_variables())
```

```
# %hide_pyerr  # - uncomment to interrupt training without a stacktrace
losses = []
kls = []
likelihood_terms = []
lagrangian_values = []

for i in xrange(TRAINING_STEPS):
  sess.run(update_op)

  if i % 100 == 0:
    iteration_loss, iteration_likelihood, iteration_kl, lag_val  = sess.run(
          [loss, likelihood_term, kl_term, lagrangian])
    print('Iteration {}. Loss {}. KL {}. Lagrangian {}'.format(
          i, iteration_loss, iteration_kl, lag_val))
    losses.append(iteration_loss)
    kls.append(iteration_kl)
    likelihood_terms.append(iteration_likelihood)
    lagrangian_values.append(lag_val)
```

    Iteration 0. Loss [543.1117]. KL 0.0468898043036. Lagrangian [1.1982032]
    Iteration 100. Loss [2151.1316]. KL 28.4300880432. Lagrangian [20.542797]
    Iteration 200. Loss [2224.2332]. KL 39.7065124512. Lagrangian [29.597895]
    Iteration 300. Loss [861.8726]. KL 58.4612426758. Lagrangian [33.975483]
    Iteration 400. Loss [637.9816]. KL 58.8706054688. Lagrangian [35.98191]
    Iteration 500. Loss [99.82016]. KL 60.9041404724. Lagrangian [37.12914]
    Iteration 600. Loss [546.16095]. KL 52.4665489197. Lagrangian [37.918278]
    Iteration 700. Loss [234.68088]. KL 55.6101303101. Lagrangian [38.35677]
    Iteration 800. Loss [124.71093]. KL 57.8047180176. Lagrangian [38.650856]
    Iteration 900. Loss [-89.92343]. KL 50.5216674805. Lagrangian [38.724487]
    Iteration 1000. Loss [-274.25394]. KL 52.6793518066. Lagrangian [38.667652]
    Iteration 1100. Loss [19.260248]. KL 49.6854743958. Lagrangian [38.596394]
    Iteration 1200. Loss [-113.308655]. KL 49.5601844788. Lagrangian [38.40778]
    Iteration 1300. Loss [-54.74331]. KL 49.0904312134. Lagrangian [38.05258]
    Iteration 1400. Loss [-143.32037]. KL 47.7855491638. Lagrangian [37.706814]
    Iteration 1500. Loss [-264.0155]. KL 49.648765564. Lagrangian [37.364395]
    Iteration 1600. Loss [-89.39512]. KL 48.1656417847. Lagrangian [36.9197]
    Iteration 1700. Loss [-243.0066]. KL 47.3575935364. Lagrangian [36.381886]
    Iteration 1800. Loss [-87.81668]. KL 46.8487854004. Lagrangian [35.858994]
    Iteration 1900. Loss [-128.1641]. KL 44.9706954956. Lagrangian [35.202892]
    Iteration 2000. Loss [-217.43582]. KL 44.6783180237. Lagrangian [34.560177]
    Iteration 2100. Loss [-194.64182]. KL 46.5217933655. Lagrangian [33.888237]
    Iteration 2200. Loss [-200.41522]. KL 43.1292495728. Lagrangian [33.17446]
    Iteration 2300. Loss [-144.71468]. KL 43.0004692078. Lagrangian [32.48504]
    Iteration 2400. Loss [-83.79274]. KL 46.6880531311. Lagrangian [31.790047]
    Iteration 2500. Loss [-238.00589]. KL 42.5690345764. Lagrangian [30.968412]
    Iteration 2600. Loss [-295.2543]. KL 41.9660072327. Lagrangian [30.114298]
    Iteration 2700. Loss [-209.98837]. KL 43.3855819702. Lagrangian [29.269682]
    Iteration 2800. Loss [-240.09467]. KL 40.4892501831. Lagrangian [28.42518]
    Iteration 2900. Loss [-219.23083]. KL 41.1703491211. Lagrangian [27.577219]
    Iteration 3000. Loss [-46.26053]. KL 42.3831253052. Lagrangian [26.70978]
    Iteration 3100. Loss [-116.477844]. KL 41.1179656982. Lagrangian [25.806303]
    Iteration 3200. Loss [-151.38455]. KL 41.4860687256. Lagrangian [24.897219]
    Iteration 3300. Loss [-172.95479]. KL 39.49949646. Lagrangian [23.965107]
    Iteration 3400. Loss [-78.90642]. KL 39.1470146179. Lagrangian [23.03269]
    Iteration 3500. Loss [-121.48421]. KL 37.3902549744. Lagrangian [22.0124]
    Iteration 3600. Loss [-88.64246]. KL 41.2223892212. Lagrangian [21.034946]
    Iteration 3700. Loss [-103.45243]. KL 38.6824569702. Lagrangian [20.071976]
    Iteration 3800. Loss [-90.24272]. KL 37.5775642395. Lagrangian [19.058521]
    Iteration 3900. Loss [-185.22794]. KL 37.5042190552. Lagrangian [18.037323]
    Iteration 4000. Loss [-116.51072]. KL 37.9734115601. Lagrangian [17.057495]
    Iteration 4100. Loss [-155.50908]. KL 38.0620384216. Lagrangian [16.009325]
    Iteration 4200. Loss [-125.57202]. KL 36.5806732178. Lagrangian [14.967052]
    Iteration 4300. Loss [-182.9829]. KL 36.1292762756. Lagrangian [13.971183]
    Iteration 4400. Loss [-99.363785]. KL 36.9285392761. Lagrangian [12.90584]
    Iteration 4500. Loss [-106.959915]. KL 33.3790931702. Lagrangian [11.871457]
    Iteration 4600. Loss [-110.164215]. KL 35.6553535461. Lagrangian [10.811469]
    Iteration 4700. Loss [-99.40161]. KL 34.1581268311. Lagrangian [9.734447]
    Iteration 4800. Loss [-67.8123]. KL 32.9110183716. Lagrangian [8.6663]
    Iteration 4900. Loss [-60.20204]. KL 31.938741684. Lagrangian [7.580799]
    Iteration 5000. Loss [-96.74039]. KL 32.0885734558. Lagrangian [6.524331]
    Iteration 5100. Loss [-10.517544]. KL 30.4894180298. Lagrangian [5.4422064]
    Iteration 5200. Loss [-41.880127]. KL 28.7457180023. Lagrangian [4.395103]
    Iteration 5300. Loss [-17.397053]. KL 28.1027488708. Lagrangian [3.4133966]
    Iteration 5400. Loss [3.2369251]. KL 26.3471107483. Lagrangian [2.494329]
    Iteration 5500. Loss [8.020703]. KL 23.8433113098. Lagrangian [1.769626]
    Iteration 5600. Loss [11.126582]. KL 21.5599899292. Lagrangian [1.3192866]
    Iteration 5700. Loss [15.181192]. KL 20.9147186279. Lagrangian [1.0089493]
    Iteration 5800. Loss [14.443428]. KL 18.848777771. Lagrangian [0.81762713]
    Iteration 5900. Loss [22.189003]. KL 19.6187133789. Lagrangian [0.7028593]
    Iteration 6000. Loss [22.202225]. KL 18.4728336334. Lagrangian [0.6367048]
    Iteration 6100. Loss [12.531754]. KL 17.013469696. Lagrangian [0.59461105]
    Iteration 6200. Loss [14.333203]. KL 16.687828064. Lagrangian [0.5632765]
    Iteration 6300. Loss [17.570522]. KL 16.6683692932. Lagrangian [0.54682696]
    Iteration 6400. Loss [17.840134]. KL 16.5294799805. Lagrangian [0.54610014]
    Iteration 6500. Loss [18.79185]. KL 16.365688324. Lagrangian [0.54157174]
    Iteration 6600. Loss [18.892363]. KL 15.5037021637. Lagrangian [0.5348891]
    Iteration 6700. Loss [15.21629]. KL 16.0394458771. Lagrangian [0.5307114]
    Iteration 6800. Loss [15.7559805]. KL 16.5639038086. Lagrangian [0.5277575]
    Iteration 6900. Loss [14.931731]. KL 15.7383422852. Lagrangian [0.51278555]
    Iteration 7000. Loss [19.56498]. KL 16.4491100311. Lagrangian [0.5056475]
    Iteration 7100. Loss [19.576942]. KL 16.1137161255. Lagrangian [0.50865453]
    Iteration 7200. Loss [15.451592]. KL 15.7040147781. Lagrangian [0.5201391]
    Iteration 7300. Loss [14.791081]. KL 15.4733705521. Lagrangian [0.5202902]
    Iteration 7400. Loss [17.569696]. KL 16.0807380676. Lagrangian [0.5198521]
    Iteration 7500. Loss [16.089928]. KL 15.9870471954. Lagrangian [0.5270175]
    Iteration 7600. Loss [13.272612]. KL 15.0041370392. Lagrangian [0.5332477]
    Iteration 7700. Loss [14.104158]. KL 15.9515972137. Lagrangian [0.5200584]
    Iteration 7800. Loss [14.199192]. KL 16.3426208496. Lagrangian [0.5135497]
    Iteration 7900. Loss [14.111576]. KL 15.195432663. Lagrangian [0.5150878]
    Iteration 8000. Loss [14.772502]. KL 15.3920574188. Lagrangian [0.50922656]
    Iteration 8100. Loss [12.457115]. KL 15.3535823822. Lagrangian [0.50688916]
    Iteration 8200. Loss [14.97758]. KL 15.119140625. Lagrangian [0.51678526]
    Iteration 8300. Loss [15.871786]. KL 15.623336792. Lagrangian [0.5170641]
    Iteration 8400. Loss [19.64407]. KL 16.4168167114. Lagrangian [0.51305974]
    Iteration 8500. Loss [14.533511]. KL 15.6900234222. Lagrangian [0.51203316]
    Iteration 8600. Loss [12.111099]. KL 15.4349803925. Lagrangian [0.5070485]
    Iteration 8700. Loss [15.735896]. KL 15.2515888214. Lagrangian [0.5097336]
    Iteration 8800. Loss [16.15351]. KL 15.3506851196. Lagrangian [0.50386757]
    Iteration 8900. Loss [16.38898]. KL 15.8036651611. Lagrangian [0.50163394]
    Iteration 9000. Loss [13.862732]. KL 15.408575058. Lagrangian [0.49713355]
    Iteration 9100. Loss [19.131302]. KL 15.6372089386. Lagrangian [0.50165915]
    Iteration 9200. Loss [13.162114]. KL 15.7860622406. Lagrangian [0.49521977]
    Iteration 9300. Loss [13.502721]. KL 14.2125453949. Lagrangian [0.50200534]
    Iteration 9400. Loss [13.477545]. KL 15.0501804352. Lagrangian [0.4931091]
    Iteration 9500. Loss [14.764007]. KL 14.9095439911. Lagrangian [0.49174684]
    Iteration 9600. Loss [13.92013]. KL 15.4130897522. Lagrangian [0.49334994]
    Iteration 9700. Loss [13.595222]. KL 14.6033401489. Lagrangian [0.49307796]
    Iteration 9800. Loss [13.88134]. KL 14.5327234268. Lagrangian [0.4895476]
    Iteration 9900. Loss [14.949783]. KL 14.9303512573. Lagrangian [0.4920855]

```
fig, axes = plt.subplots(2, 2, figsize=(2*8, 2* 5))

axes[0, 0].plot(losses, label='Negative ELBO')
axes[0, 0].set_title('Time', fontsize=15)
axes[0, 0].legend()

axes[0, 1].plot(kls, label='KL')
axes[0, 1].set_title('Time', fontsize=15)
axes[0, 1].legend()

axes[1, 0].plot(likelihood_terms, label='Likelihood Term')
axes[1, 0].set_title('Time', fontsize=15)
axes[1, 0].legend()

axes[1, 1].plot(lagrangian_values, label='Lagrangian Values')
axes[1, 1].set_title('Time', fontsize=15)
axes[1, 1].legend()
```

### Generate samples and latent interpolations

```
real_data_vals, final_samples_vals, data_reconstructions_vals = sess.run(
      [real_data, samples, reconstructions])
```

```
fig, axes = plt.subplots(1, 3, figsize=(3*4,4))

show_digits(axes[0], real_data_vals, 'Data')
show_digits(axes[1], data_reconstructions_vals, 'Reconstructions')
show_digits(axes[2], final_samples_vals, 'Samples')
```

```
show_latent_interpolations(lambda x: decoder(x).mean(), prior, sess)
```

## Discussion about constrained optimization

* What do you observe about the behaviour of the likelihood and KL term throughout training? How is it different than in Amortized Variational inference with and without KL annealing?
* What do you notice about the behaviour of the lagrangian during training?
* How would you choose the hyperparameters of the constrained optimization problem?


# Stochastic Variational Inference for scalability with SVGP

One of the main criticisms of Gaussian processes is their scalability to large datasets. In this notebook, we illustrate how to use the state-of-the-art Stochastic Variational Gaussian Process (SVGP) (*Hensman, et. al. 2013*) to overcome this problem.

```python
%matplotlib inline
import itertools
import numpy as np
import time
import gpflow
import tensorflow as tf
import matplotlib.pyplot as plt
from gpflow.ci_utils import ci_niter

plt.style.use("ggplot")

# for reproducibility of this notebook:
rng = np.random.RandomState(123)
tf.random.set_seed(42)
```

## Generating data
For this notebook example, we generate 10,000 noisy observations from a test function:
\begin{equation}
f(x) = \sin(3\pi x) + 0.3\cos(9\pi x) + \frac{\sin(7 \pi x)}{2}
\end{equation}

```python
def func(x):
    return np.sin(x * 3 * 3.14) + 0.3 * np.cos(x * 9 * 3.14) + 0.5 * np.sin(x * 7 * 3.14)

N = 10000  # Number of training observations

X = rng.rand(N, 1) * 2 - 1  # X values
Y = func(X) + 0.2 * rng.randn(N, 1)  # Noisy Y values
data = (X, Y)
```

We plot the data along with the noiseless generating function:

```python
plt.plot(X, Y, "x", alpha=0.2)
Xt = np.linspace(-1.1, 1.1, 1000)[:, None]
Yt = func(Xt)
_ = plt.plot(Xt, Yt, c="k")
```

## Building the model
The main idea behind SVGP is to approximate the true GP posterior with a GP conditioned on a small set of "inducing" values. This smaller set can be thought of as summarizing the larger dataset. For this example, we will select a set of 50 inducing locations that are initialized from the training dataset:

```python
M = 50  # Number of inducing locations

kernel = gpflow.kernels.SquaredExponential()
Z = X[:M, :].copy()  # Initialize inducing locations to the first M inputs in the dataset

m = gpflow.models.SVGP(kernel, gpflow.likelihoods.Gaussian(), Z, num_data=N)
```

## Likelihood computation: batch vs. minibatch
First we showcase the model's performance using the whole dataset to compute the ELBO.

```python
elbo = tf.function(m.elbo)
```

```python
# TensorFlow re-traces & compiles a `tf.function`-wrapped method at *every* call if the arguments are numpy arrays instead of tf.Tensors. Hence:
tensor_data = tuple(map(tf.convert_to_tensor, data))
elbo(tensor_data)  # run it once to trace & compile
```

    <tf.Tensor: shape=(), dtype=float64, numpy=-17730.676725310346>

```python
%%timeit
elbo(tensor_data)
```

    9.74 ms ± 190 µs per loop (mean ± std. dev. of 7 runs, 100 loops each)

We can speed up this calculation by using minibatches of the data. For this example, we use minibatches of size 100.

```python
minibatch_size = 100

train_dataset = tf.data.Dataset.from_tensor_slices((X, Y)).repeat().shuffle(N)

train_iter = iter(train_dataset.batch(minibatch_size))

ground_truth = elbo(tensor_data).numpy()
```

```python
%%timeit
elbo(next(train_iter))
```

    1.3 ms ± 165 µs per loop (mean ± std. dev. of 7 runs, 1 loop each)

### Stochastical estimation of ELBO
The minibatch estimate should be an unbiased estimator of the `ground_truth`. Here we show a histogram of the value from different evaluations, together with its mean and the ground truth. The small difference between the mean of the minibatch estimations and the ground truth shows that the minibatch estimator is working as expected.

```python
evals = [elbo(minibatch).numpy() for minibatch in itertools.islice(train_iter, 100)]
```

```python
plt.hist(evals, label="Minibatch estimations")
plt.axvline(ground_truth, c="k", label="Ground truth")
plt.axvline(np.mean(evals), c="g", ls="--", label="Minibatch mean")
plt.legend()
plt.title("Histogram of ELBO evaluations using minibatches")
print("Discrepancy between ground truth and minibatch estimate:", ground_truth - np.mean(evals))
```

### Minibatches speed up computation
The reason for using minibatches is that it decreases the time needed to make an optimization step, because estimating the objective is computationally cheaper with fewer data points. Here we plot the change in time required with the size of the minibatch. We see that smaller minibatches result in a cheaper estimate of the objective.

```python
# Evaluate objective for different minibatch sizes
minibatch_proportions = np.logspace(-2, 0, 10)
times = []
objs = []
for mbp in minibatch_proportions:
    batchsize = int(N * mbp)
    train_iter = iter(train_dataset.batch(batchsize))
    start_time = time.time()
    objs.append([elbo(minibatch) for minibatch in itertools.islice(train_iter, 20)])
    times.append(time.time() - start_time)
```

```python
f, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 6))
ax1.plot(minibatch_proportions, times, "x-")
ax1.set_xlabel("Minibatch proportion")
ax1.set_ylabel("Time taken")

ax2.plot(minibatch_proportions, np.array(objs), "kx")
ax2.set_xlabel("Minibatch proportion")
ax2.set_ylabel("ELBO estimates")
```

### Running stochastic optimization

First we create a utility function that plots the model's predictions:

```python
def plot(title=""):
    plt.figure(figsize=(12, 4))
    plt.title(title)
    pX = np.linspace(-1, 1, 100)[:, None]  # Test locations
    pY, pYv = m.predict_y(pX)  # Predict Y values at test locations
    plt.plot(X, Y, "x", label="Training points", alpha=0.2)
    (line,) = plt.plot(pX, pY, lw=1.5, label="Mean of predictive posterior")
    col = line.get_color()
    plt.fill_between(
        pX[:, 0],
        (pY - 2 * pYv ** 0.5)[:, 0],
        (pY + 2 * pYv ** 0.5)[:, 0],
        color=col,
        alpha=0.6,
        lw=1.5,
    )
    Z = m.inducing_variable.Z.numpy()
    plt.plot(Z, np.zeros_like(Z), "k|", mew=2, label="Inducing locations")
    plt.legend(loc="lower right")

plot(title="Predictions before training")
```

Now we can train our model. For optimizing the ELBO, we use the Adam Optimizer *(Kingma and Ba 2015)* which is designed for stochastic objective functions. We create a `run_adam` utility function  to perform the optimization.

```python
minibatch_size = 100

# We turn off training for inducing point locations
gpflow.set_trainable(m.inducing_variable, False)

def run_adam(model, iterations):
    """
    Utility function running the Adam optimizer

    :param model: GPflow model
    :param interations: number of iterations
    """
    # Create an Adam Optimizer action
    logf = []
    train_iter = iter(train_dataset.batch(minibatch_size))
    training_loss = model.training_loss_closure(train_iter, compile=True)
    optimizer = tf.optimizers.Adam()

    @tf.function
    def optimization_step():
        optimizer.minimize(training_loss, model.trainable_variables)

    for step in range(iterations):
        optimization_step()
        if step % 10 == 0:
            elbo = -training_loss().numpy()
            logf.append(elbo)
    return logf
```

Now we run the optimization loop for 20,000 iterations.

```python
maxiter = ci_niter(20000)

logf = run_adam(m, maxiter)
plt.plot(np.arange(maxiter)[::10], logf)
plt.xlabel("iteration")
_ = plt.ylabel("ELBO")
```

Finally, we plot the model's predictions.

```python
plot("Predictions after training")
```

## Further reading

Several notebooks expand on this one:

- [Advanced Sparse GP regression](../advanced/advanced_many_points.ipynb), which goes into deeper detail on sparse Gaussian process methods.
- [Optimization](../advanced/optimisation.ipynb) discussing optimizing GP models.
- [Natural gradients](../advanced/natural_gradients.ipynb) for optimizing SVGP models efficiently.

## References:
Hensman, James, Nicolo Fusi, and Neil D. Lawrence. "Gaussian processes for big data." Uncertainty in Artificial Intelligence (2013).

Kingma, Diederik P., and Jimmy Ba. "Adam: A method for stochastic optimization." arXiv preprint arXiv:1412.6980 (2014).