```python
from google.colab import drive
drive.mount('/content/drive',force_remount=True)
```

    Mounted at /content/drive

```python
import os
os.chdir('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week3')
```

```python
!pip install tensorflow=='2.3.0'
```

    Collecting tensorflow==2.3.0
      Downloading tensorflow-2.3.0-cp37-cp37m-manylinux2010_x86_64.whl (320.4 MB)
         |████████████████████████████████| 320.4 MB 53 kB/s
    Collecting numpy<1.19.0,>=1.16.0
      Downloading numpy-1.18.5-cp37-cp37m-manylinux1_x86_64.whl (20.1 MB)
         |████████████████████████████████| 20.1 MB 93.2 MB/s
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.0.0)
    Requirement already satisfied: protobuf>=3.9.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (3.17.3)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (0.37.1)
    Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (0.2.0)
    Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.6.3)
    Collecting h5py<2.11.0,>=2.10.0
      Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
         |████████████████████████████████| 2.9 MB 61.3 MB/s
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.43.0)
    Collecting tensorflow-estimator<2.4.0,>=2.3.0
      Downloading tensorflow_estimator-2.3.0-py2.py3-none-any.whl (459 kB)
         |████████████████████████████████| 459 kB 50.1 MB/s
    Collecting gast==0.3.3
      Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (3.3.0)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.13.3)
    Collecting tensorboard<3,>=2.3.0
      Downloading tensorboard-2.8.0-py3-none-any.whl (5.8 MB)
         |████████████████████████████████| 5.8 MB 53.9 MB/s
    Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.15.0)
    Requirement already satisfied: keras-preprocessing<1.2,>=1.1.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.1.2)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.1.0)
    Requirement already satisfied: scipy==1.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.3.0) (1.4.1)
    Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (2.23.0)
    Requirement already satisfied: tensorboard-data-server<0.7.0,>=0.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (0.6.1)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (3.3.6)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (57.4.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (1.0.1)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (1.8.1)
    Requirement already satisfied: google-auth<3,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (1.35.0)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<3,>=2.3.0->tensorflow==2.3.0) (0.4.6)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (4.2.4)
    Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (4.8)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (0.2.8)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (1.3.1)
    Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (4.10.1)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (3.10.0.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (3.7.0)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<3,>=1.6.3->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (0.4.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (3.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (2.10)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (1.24.3)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<3,>=2.3.0->tensorflow==2.3.0) (3.2.0)
    Installing collected packages: numpy, tensorflow-estimator, tensorboard, h5py, gast, tensorflow
      Attempting uninstall: numpy
        Found existing installation: numpy 1.19.5
        Uninstalling numpy-1.19.5:
          Successfully uninstalled numpy-1.19.5
      Attempting uninstall: tensorflow-estimator
        Found existing installation: tensorflow-estimator 2.1.0
        Uninstalling tensorflow-estimator-2.1.0:
          Successfully uninstalled tensorflow-estimator-2.1.0
      Attempting uninstall: tensorboard
        Found existing installation: tensorboard 2.1.1
        Uninstalling tensorboard-2.1.1:
          Successfully uninstalled tensorboard-2.1.1
      Attempting uninstall: h5py
        Found existing installation: h5py 3.1.0
        Uninstalling h5py-3.1.0:
          Successfully uninstalled h5py-3.1.0
      Attempting uninstall: gast
        Found existing installation: gast 0.2.2
        Uninstalling gast-0.2.2:
          Successfully uninstalled gast-0.2.2
      Attempting uninstall: tensorflow
        Found existing installation: tensorflow 2.1.0
        Uninstalling tensorflow-2.1.0:
          Successfully uninstalled tensorflow-2.1.0
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    tables 3.7.0 requires numpy>=1.19.0, but you have numpy 1.18.5 which is incompatible.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed gast-0.3.3 h5py-2.10.0 numpy-1.18.5 tensorboard-2.8.0 tensorflow-2.3.0 tensorflow-estimator-2.3.0

```python
!pip install tensorflow_probability=='0.11.1'
```

    Collecting tensorflow_probability==0.11.1
      Downloading tensorflow_probability-0.11.1-py2.py3-none-any.whl (4.3 MB)
         |████████████████████████████████| 4.3 MB 10.6 MB/s
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.11.1) (1.18.5)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.11.1) (1.15.0)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.11.1) (4.4.2)
    Requirement already satisfied: cloudpickle>=1.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.11.1) (1.3.0)
    Requirement already satisfied: dm-tree in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.11.1) (0.1.6)
    Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.11.1) (0.3.3)
    Installing collected packages: tensorflow-probability
      Attempting uninstall: tensorflow-probability
        Found existing installation: tensorflow-probability 0.9.0
        Uninstalling tensorflow-probability-0.9.0:
          Successfully uninstalled tensorflow-probability-0.9.0
    Successfully installed tensorflow-probability-0.11.1

# Programming Assignment

## RealNVP for the LSUN bedroom dataset

### Instructions

In this notebook, you will develop the RealNVP normalising flow architecture from scratch, including the affine coupling layers, checkerboard and channel-wise masking, and combining into a multiscale architecture. You will train the normalising flow on a subset of the LSUN bedroom dataset.

Some code cells are provided for you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line:

`#### GRADED CELL ####`

Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly.

### How to submit

Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the **Submit Assignment** button at the top of this notebook.

### Let's get started!

We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here.

```python
#### PACKAGE IMPORTS ####

# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook

import tensorflow as tf
import tensorflow_probability as tfp
import numpy as np
import matplotlib.pyplot as plt
from IPython.display import Image

from tensorflow.keras import Model, Input
from tensorflow.keras.layers import Conv2D, BatchNormalization
from tensorflow.keras.optimizers import Adam

tfd = tfp.distributions
tfb = tfp.bijectors

# If you would like to make further imports from tensorflow, add them here

```

#### The LSUN Bedroom Dataset

In this assignment, you will use a subset of the [LSUN dataset](https://www.yf.io/p/lsun). This is a large-scale image dataset with 10 scene and 20 object categories. A subset of the LSUN bedroom dataset has been provided, and has already been downsampled and preprocessed into smaller, fixed-size images.

* F. Yu, A. Seff, Y. Zhang, S. Song, T. Funkhouser and J. Xia. "LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loop". [arXiv:1506.03365](https://arxiv.org/abs/1506.03365), 10 Jun 2015

Your goal is to develop the RealNVP normalising flow architecture using bijector subclassing, and use it to train a generative model of the LSUN bedroom data subset. For full details on the RealNVP model, refer to the original paper:

* L. Dinh, J. Sohl-Dickstein and S. Bengio. "Density estimation using Real NVP". [arXiv:1605.08803](https://arxiv.org/abs/1605.08803), 27 Feb 2017.

#### Import the data

The dataset required for this project can be downloaded from the following link:

https://drive.google.com/file/d/1scbDZrn5pkRjF_CeZp66uHVQC9o1gIsg/view?usp=sharing

You should upload this file to Drive for use in this Colab notebook. It is recommended to unzip it on Drive, which can be done using the `zipfile` package:

>```
import zipfile
with zipfile.ZipFile("/path/to/lsun_bedroom.zip","r") as zip_ref:
   zip_ref.extractall('lsun_bedroom_data')
```

```python
#import zipfile
#with zipfile.ZipFile("lsun_bedroom.zip","r") as zip_ref:
#  zip_ref.extractall('lsun_bedroom_data')
```

#### Load the dataset

The following functions will be useful for loading and preprocessing the dataset. The subset you will use for this assignment consists of 10,000 training images, 1000 validation images and 1000 test images.

The images have been downsampled to 32 x 32 x 3 in order to simplify the training process.

```python
# Functions for loading and preprocessing the images

def load_image(filepath):
    raw_img = tf.io.read_file(filepath)
    img_tensor_int = tf.image.decode_jpeg(raw_img, channels=3)
    img_tensor_flt = tf.image.convert_image_dtype(img_tensor_int, tf.float32)
    img_tensor_flt = tf.image.resize(img_tensor_flt, [32, 32])
    img_tensor_flt = tf.image.random_flip_left_right(img_tensor_flt)
    return img_tensor_flt, img_tensor_flt

def load_dataset(split):
    train_list_ds = tf.data.Dataset.list_files('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week3/lsun_bedroom_data/{}/*.jpg'.format(split), shuffle=False)
    train_ds = train_list_ds.map(load_image)
    return train_ds
```

```python
# Load the training, validation and testing datasets splits

train_ds = load_dataset('train')
val_ds = load_dataset('val')
test_ds = load_dataset('test')
```

```python
# Shuffle the datasets

shuffle_buffer_size = 1000
train_ds = train_ds.shuffle(shuffle_buffer_size)
val_ds = val_ds.shuffle(shuffle_buffer_size)
test_ds = test_ds.shuffle(shuffle_buffer_size)
```

```python
# Display a few examples

n_img = 4
f, axs = plt.subplots(n_img, n_img, figsize=(14, 14))

for k, image in enumerate(train_ds.take(n_img**2)):
    i = k // n_img
    j = k % n_img
    axs[i, j].imshow(image[0])
    axs[i, j].axis('off')
f.subplots_adjust(wspace=0.01, hspace=0.03)
```

```python
# Batch the Dataset objects

batch_size = 64
train_ds = train_ds.batch(batch_size)
val_ds = val_ds.batch(batch_size)
test_ds = test_ds.batch(batch_size)
```

### Affine coupling layer

We will begin the development of the RealNVP architecture with the core bijector that is called the _affine coupling layer_. This bijector can be described as follows: suppose that $x$ is a $D$-dimensional input, and let $d<D$. Then the output $y$ of the affine coupling layer is given by the following equations:

$$
\begin{align}
y_{1:d} &= x_{1:d} \tag{1}\\
    y_{d+1:D} &= x_{d+1:D}\odot \exp(s(x_{1:d})) + t(x_{1:d}), \tag{2}
\end{align}
$$

where $s$ and $t$ are functions from $\mathbb{R}^d\rightarrow\mathbb{R}^{D-d}$, and define the log-scale and shift operations on the vector $x_{d+1:D}$ respectively.

The log of the Jacobian determinant for this layer is given by $\sum_{j}s(x_{1:d})_j$.

The inverse operation can be easily computed as

$$
\begin{align}
x_{1:d} &= y_{1:d}\tag{3}\\
x_{d+1:D} &= \left(y_{d+1:D} - t(y_{1:d})\right)\odot \exp(-s(y_{1:d})),\tag{4}
\end{align}
$$

In practice, we will implement equations $(1)$ and $(2)$ using a binary mask $b$:

$$
\begin{align}
\text{Forward pass:}\qquad y &= b\odot x + (1-b)\odot\left(x\odot\exp(s(b\odot x)) + t(b\odot x)\right),\tag{5}\\
\text{Inverse pass:}\qquad x &= b\odot y + (1-b)\odot\left(y - t(b\odot x) \odot\exp( -s(b\odot x))\right).\tag{6}
\end{align}
$$

Our inputs $x$ will be a batch of 3-dimensional Tensors with `height`, `width` and `channels` dimensions. As in the original architecture, we will use both spatial 'checkerboard' masks and channel-wise masks:

```python
# Run this cell to download and view a figure to illustrate the checkerboard and binary masks

!wget -q -O binary_masks.png --no-check-certificate "https://docs.google.com/uc?export=download&id=1d_cBjyPGm8i0l5GsRSspPoAlxiPo3HGt"
Image("binary_masks.png", width=800)
```

Figure 1. Spatial checkerboard mask (left) and channel-wise mask (right). From the original paper.

#### Custom model for log-scale and shift

You should now create a custom model for the shift and log-scale parameters that are used in the affine coupling layer bijector. We will use a convolutional residual network, with two residual blocks and a final convolutional layer. Using the functional API, build the model according to the following specifications:

* The function takes the `input_shape` and `filters` as arguments
* The model should use the `input_shape` in the function argument to set the shape in the Input layer (call this layer `h0`).
* The first hidden layer should be a Conv2D layer with number of filters set by the `filters` argument, and a ReLU activation
* The second hidden layer should be a BatchNormalization layer
* The third hidden layer should be a Conv2D layer with the same number of filters as the input `h0` to the model, and a ReLU activation
* The fourth hidden layer should be a BatchNormalization layer
* The fifth hidden layer should be the sum of the fourth hidden layer output and the inputs `h0`. Call this layer `h1`
* The sixth hidden layer should be a Conv2D layer with filters set by the `filters` argument, and a ReLU activation
* The seventh hidden layer should be a BatchNormalization layer
* The eighth hidden layer should be a Conv2D layer with the same number of filters as `h1` (and `h0`), and a ReLU activation
* The ninth hidden layer should be a BatchNormalization layer
* The tenth hidden layer should be the sum of the ninth hidden layer output and `h1`
* The eleventh hidden layer should be a Conv2D layer with the number of filters equal to twice the number of channels of the model input, and a linear activation. Call this layer `h2`
* The twelfth hidden layer should split `h2` into two equal-sized Tensors along the final channel axis. These two Tensors are the shift and log-scale Tensors, and should each have the same shape as the model input
* The final layer should then apply the `tanh` nonlinearity to the log_scale Tensor. The outputs to the model should then be the list of Tensors `[shift, log_scale]`

All Conv2D layers should use a 3x3 kernel size, `"SAME"` padding and an $l2$ kernel regularizer with regularisation coefficient of `5e-5`.

_Hint: use_ `tf.split` _with arguments_ `num_or_size_splits=2, axis=-1` _to create the output Tensors_.

In total, the network should have 14 layers (including the `Input` layer).

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_conv_resnet(input_shape, filters):
    """
    This function should build a CNN ResNet model according to the above specification,
    using the functional API. The function takes input_shape as an argument, which should be
    used to specify the shape in the Input layer, as well as a filters argument, which
    should be used to specify the number of filters in (some of) the convolutional layers.
    Your function should return the model.
    """
    def ConvBlock(hi,filters_i,activation='relu'):
      h=tf.keras.layers.Conv2D(filters=filters_i,activation=activation,kernel_size=(3,3),padding='SAME',kernel_regularizer=tf.keras.regularizers.l2(5e-5))(hi)
      h=tf.keras.layers.BatchNormalization()(h)
      return h

    h0=tf.keras.layers.Input(shape=input_shape)
    h01=ConvBlock(h0,filters)
    h02=ConvBlock(h01,input_shape[-1])

    #h1=tf.keras.layers.concatenate([h0,h02],axis=-1)
    h1=tf.keras.Add()([h0,h02])
    h11=ConvBlock(h1,filters)
    h12=ConvBlock(h11,input_shape[-1])

    #h2=tf.keras.layers.concatenate([h1,h12],axis=-1)
    h2=tf.keras.Add()([h1,h12])
    h2=ConvBlock(h2,2*input_shape[-1],'linear')

    shift,log_scale=tf.keras.layers.Lambda(lambda value: tf.split(value,num_or_size_splits=2,axis=-1))(h2)

    log_scale=tf.keras.layers.Activation(activation='tanh')(log_scale)
    conv_resnet=Model(inputs=h0,outputs=[shift,log_scale])
    return conv_resnet

```

```python
# Test your function and print the model summary

conv_resnet = get_conv_resnet((32, 32, 3), 32)
conv_resnet.summary()
```

    Model: "functional_1"
    __________________________________________________________________________________________________
    Layer (type)                    Output Shape         Param #     Connected to
    ==================================================================================================
    input_1 (InputLayer)            [(None, 32, 32, 3)]  0
    __________________________________________________________________________________________________
    conv2d (Conv2D)                 (None, 32, 32, 32)   896         input_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization (BatchNorma (None, 32, 32, 32)   128         conv2d[0][0]
    __________________________________________________________________________________________________
    conv2d_1 (Conv2D)               (None, 32, 32, 3)    867         batch_normalization[0][0]
    __________________________________________________________________________________________________
    batch_normalization_1 (BatchNor (None, 32, 32, 3)    12          conv2d_1[0][0]
    __________________________________________________________________________________________________
    concatenate (Concatenate)       (None, 32, 32, 6)    0           input_1[0][0]
                                                                     batch_normalization_1[0][0]
    __________________________________________________________________________________________________
    conv2d_2 (Conv2D)               (None, 32, 32, 32)   1760        concatenate[0][0]
    __________________________________________________________________________________________________
    batch_normalization_2 (BatchNor (None, 32, 32, 32)   128         conv2d_2[0][0]
    __________________________________________________________________________________________________
    conv2d_3 (Conv2D)               (None, 32, 32, 3)    867         batch_normalization_2[0][0]
    __________________________________________________________________________________________________
    batch_normalization_3 (BatchNor (None, 32, 32, 3)    12          conv2d_3[0][0]
    __________________________________________________________________________________________________
    concatenate_1 (Concatenate)     (None, 32, 32, 9)    0           concatenate[0][0]
                                                                     batch_normalization_3[0][0]
    __________________________________________________________________________________________________
    conv2d_4 (Conv2D)               (None, 32, 32, 6)    492         concatenate_1[0][0]
    __________________________________________________________________________________________________
    batch_normalization_4 (BatchNor (None, 32, 32, 6)    24          conv2d_4[0][0]
    __________________________________________________________________________________________________
    lambda (Lambda)                 [(None, 32, 32, 3),  0           batch_normalization_4[0][0]
    __________________________________________________________________________________________________
    activation (Activation)         (None, 32, 32, 3)    0           lambda[0][1]
    ==================================================================================================
    Total params: 5,186
    Trainable params: 5,034
    Non-trainable params: 152
    __________________________________________________________________________________________________

You can also inspect your model architecture graphically by running the following cell. It should look something like the following:

```python
# Run this cell to download and view an example model plot

!wget -q -O model_plot.png --no-check-certificate "https://docs.google.com/uc?export=download&id=1I9MhFGquwyHsJlDgO8hItXnMqAvc5cQg"
Image("model_plot.png", width=1400)
```

```python
# Plot the model graph

tf.keras.utils.plot_model(conv_resnet, show_layer_names=False, rankdir='LR')
```

```python
# Check the output shapes are as expected

print(conv_resnet(tf.random.normal((1, 32, 32, 3)))[0].shape)
print(conv_resnet(tf.random.normal((1, 32, 32, 3)))[1].shape)
```

    (1, 32, 32, 3)
    (1, 32, 32, 3)

#### Binary masks

Now that you have a shift and log-scale model built, we will now implement the affine coupling layer. We will first need functions to create the binary masks $b$ as described above. The following function creates the spatial 'checkerboard' mask.

It takes a rank-2 `shape` as input, which correspond to the `height` and `width` dimensions, as well as an `orientation` argument (an integer equal to `0` or `1`) that determines which way round the zeros and ones are entered into the Tensor.

```python
# Function to create the checkerboard mask

def checkerboard_binary_mask(shape, orientation=0):
    height, width = shape[0], shape[1]
    height_range = tf.range(height)
    width_range = tf.range(width)
    height_odd_inx = tf.cast(tf.math.mod(height_range, 2), dtype=tf.bool)
    width_odd_inx = tf.cast(tf.math.mod(width_range, 2), dtype=tf.bool)
    odd_rows = tf.tile(tf.expand_dims(height_odd_inx, -1), [1, width])
    odd_cols = tf.tile(tf.expand_dims(width_odd_inx, 0), [height, 1])
    checkerboard_mask = tf.math.logical_xor(odd_rows, odd_cols)
    if orientation == 1:
        checkerboard_mask = tf.math.logical_not(checkerboard_mask)
    return tf.cast(tf.expand_dims(checkerboard_mask, -1), tf.float32)
```

This function creates a rank-3 Tensor to mask the `height`, `width` and `channels` dimensions of the input. We can take a look at this checkerboard mask for some example inputs below. In order to make the Tensors easier to inspect, we will squeeze out the single channel dimension (which is always 1 for this mask).

```python
# Run the checkerboard_binary_mask function to see an example
# NB: we squeeze the shape for easier viewing. The full shape is (4, 4, 1)

tf.squeeze(checkerboard_binary_mask((4, 4), orientation=0))
```

    <tf.Tensor: shape=(4, 4), dtype=float32, numpy=
    array([[0., 1., 0., 1.],
           [1., 0., 1., 0.],
           [0., 1., 0., 1.],
           [1., 0., 1., 0.]], dtype=float32)>

```python
# The `orientation` should be 0 or 1, and determines which way round the binary entries are

tf.squeeze(checkerboard_binary_mask((4, 4), orientation=1))
```

    <tf.Tensor: shape=(4, 4), dtype=float32, numpy=
    array([[1., 0., 1., 0.],
           [0., 1., 0., 1.],
           [1., 0., 1., 0.],
           [0., 1., 0., 1.]], dtype=float32)>

You should now complete the following function to create a channel-wise mask. This function takes a single integer `num_channels` as an input, as well as an `orientation` argument, similar to above. You can assume that the `num_channels` integer is even.

The function should return a rank-3 Tensor with singleton entries for `height` and `width`. In the channel axis, the first `num_channels // 2` entries should be zero (for `orientation=0`) and the final `num_channels // 2` entries should be one (for `orientation=0`). The zeros and ones should be reversed for `orientation=1`. The `dtype` of the returned Tensor should be `tf.float32`.

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def channel_binary_mask(num_channels, orientation=0):
    """
    This function takes an integer num_channels and orientation (0 or 1) as
    arguments. It should create a channel-wise binary mask with
    dtype=tf.float32, according to the above specification.
    The function should then return the binary mask.
    """
    if orientation==0:
      return tf.cast(tf.keras.layers.concatenate([tf.zeros((1,1,num_channels//2)),
                                            tf.ones((1,1,num_channels - num_channels//2))],axis=-1),tf.float32)
    elif orientation==1:
      return tf.cast(tf.keras.layers.concatenate([tf.ones((1,1,num_channels//2)),
                                            tf.zeros((1,1,num_channels-num_channels//2))],axis=-1),tf.float32)
    else:
      print('Bad input')

```

```python
# Run your function to see an example channel-wise binary mask

channel_binary_mask(6, orientation=0)
```

    <tf.Tensor: shape=(1, 1, 6), dtype=float32, numpy=array([[[0., 0., 0., 1., 1., 1.]]], dtype=float32)>

```python
#### GRADED CELL ####

# Complete the following functions.
# Make sure to not change the function names or arguments.

def forward(x, b, shift_and_log_scale_fn):
    """
    This function takes the input Tensor x, binary mask b and callable
    shift_and_log_scale_fn as arguments.
    This function should implement the forward transformation in equation (5)
    and return the output Tensor y, which will have the same shape as x
    """
    t,s=shift_and_log_scale_fn(b*x)
    return tf.cast(b*x+(1-b)*(x*tf.math.exp(s)+t),tf.float32)

def inverse(y, b, shift_and_log_scale_fn):
    """
    This function takes the input Tensor x, binary mask b and callable
    shift_and_log_scale_fn as arguments.
    This function should implement the forward transformation in equation (5)
    and return the output Tensor y, which will have the same shape as x
    """
    t,s=shift_and_log_scale_fn(b*y)
    return tf.cast(b*y+(1-b)*((y-t)*tf.math.exp(-s)),tf.float32)

```

The new bijector class also requires the `log_det_jacobian` methods to be implemented. Recall that the log of the Jacobian determinant of the forward transformation is given by $\sum_{j}s(x_{1:d})_j$, where $s$ is the log-scale function of the affine coupling layer.

You should now complete the following functions to define the `forward_log_det_jacobian` and `inverse_log_det_jacobian` methods of the affine coupling layer bijector.

* Both functions `forward_log_det_jacobian` and `inverse_log_det_jacobian` takes an input Tensor `x` (or `y`), a rank-3 binary mask `b`, and the `shift_and_log_scale_fn` callable
* These arguments are the same as the description for the `forward` and `inverse` functions
* The `forward_log_det_jacobian` function should implement the log of the Jacobian determinant for the transformation $(5)$
* The `inverse_log_det_jacobian` function should implement the log of the Jacobian determinant for the transformation $(6)$
* Both functions should reduce sum over the last three axes of the input Tensor (`height`, `width` and `channels`)

```python
#### GRADED CELL ####

# Complete the following functions.
# Make sure to not change the function names or arguments.

def forward_log_det_jacobian(x, b, shift_and_log_scale_fn):
    """
    This function takes the input Tensor x, binary mask b and callable
    shift_and_log_scale_fn as arguments.
    This function should compute and return the log of the Jacobian determinant
    of the forward transformation in equation (5)
    """
    t,s=shift_and_log_scale_fn(b*x)
    return tf.reduce_sum((1-b)*s,axis=[-1,-2,-3])

def inverse_log_det_jacobian(y, b, shift_and_log_scale_fn):
    """
    This function takes the input Tensor y, binary mask b and callable
    shift_and_log_scale_fn as arguments.
    This function should compute and return the log of the Jacobian determinant
    of the forward transformation in equation (6)
    """
    t,s=shift_and_log_scale_fn(b*y)
    return -tf.reduce_sum((1-b)*s,axis=[-1,-2,-3])

```

You are now ready to create the coupling layer bijector, using bijector subclassing. You should complete the class below to define the `AffineCouplingLayer`.

* You should complete the initialiser `__init__`, and the internal class method `_get_mask`
* The `_forward`, `_inverse`, `_forward_log_det_jacobian` and `_inverse_log_det_jacobian` methods are completed for you using the functions you have written above. Do not modify these methods
* The initialiser takes the `shift_and_log_scale_fn` callable, `mask_type` string (either `"checkerboard"` or `"channel"`, `orientation` (integer, either `0` or `1`) as required arguments, and allows for extra keyword arguments
  * The required arguments should be set as class attributes in the initialiser (note that the `shift_and_log_scale_fn` attribute is being used in the `_forward`, `_inverse`, `_forward_log_det_jacobian` and `_inverse_log_det_jacobian` methods)
  * The initialiser should call the base class initialiser, and pass in any extra keyword arguments
  * The class should have a required number of event dimensions equal to 3
* The internal method `_get_mask` takes a `shape` as an argument, which is the shape of an input Tensor
  * This method should use the `checkerboard_binary_mask` and `channel_binary_mask` functions above, as well as the `mask_type` and `orientation` arguments passed to the initialiser to compute and return the required binary mask
  * This method is used in each of the `_forward`, `_inverse`, `_forward_log_det_jacobian` and `_inverse_log_det_jacobian` methods

```python
#### GRADED CELL ####

# Complete the following class.
# Make sure to not change the class or method names or arguments.

class AffineCouplingLayer(tfb.Bijector):
    """
    Class to implement the affine coupling layer.
    Complete the __init__ and _get_mask methods according to the instructions above.
    """

    def __init__(self, shift_and_log_scale_fn, mask_type, orientation, **kwargs):
        """
        The class initialiser takes the shift_and_log_scale_fn callable, mask_type,
        orientation and possibly extra keywords arguments. It should call the
        base class initialiser, passing any extra keyword arguments along.
        It should also set the required arguments as class attributes.
        """
        super(AffineCouplingLayer,self).__init__(forward_min_event_ndims=3,**kwargs)
        self.shift_and_log_scale_fn=shift_and_log_scale_fn
        self.mask_type=mask_type
        self.orientation=orientation

    def _get_mask(self, shape):
        """
        This internal method should use the binary mask functions above to compute
        and return the binary mask, according to the arguments passed in to the
        initialiser.
        """
        if self.mask_type!='channel':
          return checkerboard_binary_mask(shape[1:],self.orientation)
        elif self.mask_type=='channel':
          return channel_binary_mask(shape[-1],self.orientation)

    def _forward(self, x):
        b = self._get_mask(x.shape)
        return forward(x, b, self.shift_and_log_scale_fn)

    def _inverse(self, y):
        b = self._get_mask(y.shape)
        return inverse(y, b, self.shift_and_log_scale_fn)

    def _forward_log_det_jacobian(self, x):
        b = self._get_mask(x.shape)
        return forward_log_det_jacobian(x, b, self.shift_and_log_scale_fn)

    def _inverse_log_det_jacobian(self, y):
        b = self._get_mask(y.shape)
        return inverse_log_det_jacobian(y, b, self.shift_and_log_scale_fn)
```

```python
# Test your function by creating an instance of the AffineCouplingLayer class

affine_coupling_layer = AffineCouplingLayer(conv_resnet, 'channel', orientation=1,
                                            name='affine_coupling_layer')
```

```python
# The following should return a Tensor of the same shape as the input

affine_coupling_layer.forward(tf.random.normal((16, 32, 32, 3))).shape
```

    TensorShape([16, 32, 32, 3])

```python
# The following should compute a log_det_jacobian for each event in the batch

affine_coupling_layer.forward_log_det_jacobian(tf.random.normal((16, 32, 32, 3)), event_ndims=3).shape
```

    TensorShape([16])

#### Combining the affine coupling layers

In the affine coupling layer, part of the input remains unchanged in the transformation $(5)$. In order to allow transformation of all of the input, several coupling layers are composed, with the orientation of the mask being reversed in subsequent layers.

```python
# Run this cell to download and view a sketch of the affine coupling layers

!wget -q -O alternating_masks.png --no-check-certificate "https://docs.google.com/uc?export=download&id=1r1vASfLOW3kevxRzFUXhCtHN8dzldHve"
Image("alternating_masks.png", width=800)
```

Figure 2. RealNVP alternates the orientation of masks from one affine coupling layer to the next. From the original paper.

Our model design will be similar to the original architecture; we will compose three affine coupling layers with checkerboard masking, followed by a batch normalization bijector (`tfb.BatchNormalization` is a built-in bijector), followed by a squeezing operation, followed by three more affine coupling layers with channel-wise masking and a final batch normalization bijector.

The squeezing operation divides the spatial dimensions into 2x2 squares, and reshapes a Tensor of shape `(H, W, C)` into a Tensor of shape `(H // 2, W // 2, 4 * C)` as shown in Figure 1.

The squeezing operation is also a bijective operation, and has been provided for you in the class below.

```python
# Bijector class for the squeezing operation

class Squeeze(tfb.Bijector):

    def __init__(self, name='Squeeze', **kwargs):
        super(Squeeze, self).__init__(forward_min_event_ndims=3, is_constant_jacobian=True,
                                      name=name, **kwargs)

    def _forward(self, x):
        input_shape = x.shape
        height, width, channels = input_shape[-3:]
        y = tfb.Reshape((height // 2, 2, width // 2, 2, channels), event_shape_in=(height, width, channels))(x)
        y = tfb.Transpose(perm=[0, 2, 1, 3, 4])(y)
        y = tfb.Reshape((height // 2, width // 2, 4 * channels),
                        event_shape_in=(height // 2, width // 2, 2, 2, channels))(y)
        return y

    def _inverse(self, y):
        input_shape = y.shape
        height, width, channels = input_shape[-3:]
        x = tfb.Reshape((height, width, 2, 2, channels // 4), event_shape_in=(height, width, channels))(y)
        x = tfb.Transpose(perm=[0, 2, 1, 3, 4])(x)
        x = tfb.Reshape((2 * height, 2 * width, channels // 4),
                        event_shape_in=(height, 2, width, 2, channels // 4))(x)
        return x

    def _forward_log_det_jacobian(self, x):
        return tf.constant(0., x.dtype)

    def _inverse_log_det_jacobian(self, y):
        return tf.constant(0., y.dtype)

    def _forward_event_shape_tensor(self, input_shape):
        height, width, channels = input_shape[-3], input_shape[-2], input_shape[-1]
        return height // 2, width // 2, 4 * channels

    def _inverse_event_shape_tensor(self, output_shape):
        height, width, channels = output_shape[-3], output_shape[-2], output_shape[-1]
        return height * 2, width * 2, channels // 4
```

You can see the effect of the squeezing operation on some example inputs in the cells below. In the forward transformation, each spatial dimension is halved, whilst the channel dimension is multiplied by 4. The opposite happens in the inverse transformation.

```python
# Test the Squeeze bijector

squeeze = Squeeze()
squeeze(tf.ones((10, 32, 32, 3))).shape
```

    TensorShape([10, 16, 16, 12])

```python
# Test the inverse operation

squeeze.inverse(tf.ones((10, 4, 4, 96))).shape
```

    TensorShape([10, 8, 8, 24])

We can now construct a block of coupling layers according to the architecture described above. You should complete the following function to chain together the bijectors that we have constructed, to form a bijector that performs the following operations in the forward transformation:

* Three `AffineCouplingLayer` bijectors with `"checkerboard"` masking with orientations `0, 1, 0` respectively
* A `BatchNormalization` bijector
* A `Squeeze` bijector
* Three more `AffineCouplingLayer` bijectors with `"channel"` masking with orientations `0, 1, 0` respectively
* Another `BatchNormalization` bijector

The function takes the following arguments:
* `shift_and_log_scale_fns`: a list or tuple of six conv_resnet models
  * The first three models in this list are used in the three coupling layers with checkerboard masking
  * The last three models in this list are used in the three coupling layers with channel masking
* `squeeze`: an instance of the `Squeeze` bijector

_NB: at this point, we would like to point out that we are following the exposition in the original paper, and think of the forward transformation as acting on the input image. Note that this is in contrast to the convention of using the forward transformation for sampling, and the inverse transformation for computing log probs._

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def realnvp_block(shift_and_log_scale_fns, squeeze):
    """
    This function takes a list or tuple of six conv_resnet models, and an
    instance of the Squeeze bijector.
    The function should construct the chain of bijectors described above,
    using the conv_resnet models in the coupling layers.
    The function should then return the chained bijector.
    """
    l1=AffineCouplingLayer(shift_and_log_scale_fns[0],'checkerboard',orientation=0)
    l2=AffineCouplingLayer(shift_and_log_scale_fns[1],'checkerboard',orientation=1)
    l3=AffineCouplingLayer(shift_and_log_scale_fns[2],'checkerboard',orientation=0)
    l4=tfb.BatchNormalization()
    l5=squeeze
    l6=AffineCouplingLayer(shift_and_log_scale_fns[3],'channel',orientation=0)
    l7=AffineCouplingLayer(shift_and_log_scale_fns[4],'channel',orientation=1)
    l8=AffineCouplingLayer(shift_and_log_scale_fns[5],'channel',orientation=0)
    l9=tfb.BatchNormalization()
    return tfb.Chain([l9,l8,l7,l6,l5,l4,l3,l2,l1])

```

```python
# Run your function to create an instance of the bijector

checkerboard_fns = []
for _ in range(3):
    checkerboard_fns.append(get_conv_resnet((32, 32, 3), 512))
channel_fns = []
for _ in range(3):
    channel_fns.append(get_conv_resnet((16, 16, 12), 512))

block = realnvp_block(checkerboard_fns + channel_fns, squeeze)
```

```python
# Test the bijector on a dummy input

block.forward(tf.random.normal((10, 32, 32, 3))).shape
```

    TensorShape([10, 16, 16, 12])

#### Multiscale architecture

The final component of the RealNVP is the multiscale architecture. The squeeze operation reduces the spatial dimensions but increases the channel dimensions. After one of the blocks of coupling-squeeze-coupling that you have implemented above, half of the dimensions are factored out as latent variables, while the other half is further processed through subsequent layers. This results in latent variables that represent different scales of features in the model.

```python
# Run this cell to download and view a sketch of the multiscale architecture

!wget -q -O multiscale.png --no-check-certificate "https://docs.google.com/uc?export=download&id=19Sc6PKbc8Bi2DoyupHZxHvB3m6tw-lki"
Image("multiscale.png", width=700)
```

Figure 3. RealNVP creates latent variables at different scales by factoring out half of the dimensions at each scale. From the original paper.

The final scale does not use the squeezing operation, and instead applies four affine coupling layers with alternating checkerboard masks.

The multiscale architecture for two latent variable scales is implemented for you in the following bijector.

```python
# Bijector to implement the multiscale architecture

class RealNVPMultiScale(tfb.Bijector):

    def __init__(self, **kwargs):
        super(RealNVPMultiScale, self).__init__(forward_min_event_ndims=3, **kwargs)

        # First level
        shape1 = (32, 32, 3)  # Input shape
        shape2 = (16, 16, 12)  # Shape after the squeeze operation
        shape3 = (16, 16, 6)  # Shape after factoring out the latent variable
        self.conv_resnet1 = get_conv_resnet(shape1, 64)
        self.conv_resnet2 = get_conv_resnet(shape1, 64)
        self.conv_resnet3 = get_conv_resnet(shape1, 64)
        self.conv_resnet4 = get_conv_resnet(shape2, 128)
        self.conv_resnet5 = get_conv_resnet(shape2, 128)
        self.conv_resnet6 = get_conv_resnet(shape2, 128)
        self.squeeze = Squeeze()
        self.block1 = realnvp_block([self.conv_resnet1, self.conv_resnet2,
                                    self.conv_resnet3, self.conv_resnet4,
                                    self.conv_resnet5, self.conv_resnet6], self.squeeze)

        # Second level
        self.conv_resnet7 = get_conv_resnet(shape3, 128)
        self.conv_resnet8 = get_conv_resnet(shape3, 128)
        self.conv_resnet9 = get_conv_resnet(shape3, 128)
        self.conv_resnet10 = get_conv_resnet(shape3, 128)
        self.coupling_layer1 = AffineCouplingLayer(self.conv_resnet7, 'checkerboard', 0)
        self.coupling_layer2 = AffineCouplingLayer(self.conv_resnet8, 'checkerboard', 1)
        self.coupling_layer3 = AffineCouplingLayer(self.conv_resnet9, 'checkerboard', 0)
        self.coupling_layer4 = AffineCouplingLayer(self.conv_resnet10, 'checkerboard', 1)
        self.block2 = tfb.Chain([self.coupling_layer4, self.coupling_layer3,
                                 self.coupling_layer2, self.coupling_layer1])

    def _forward(self, x):
        h1 = self.block1.forward(x)
        z1, h2 = tf.split(h1, 2, axis=-1)
        z2 = self.block2.forward(h2)
        return tf.concat([z1, z2], axis=-1)

    def _inverse(self, y):
        z1, z2 = tf.split(y, 2, axis=-1)
        h2 = self.block2.inverse(z2)
        h1 = tf.concat([z1, h2], axis=-1)
        return self.block1.inverse(h1)

    def _forward_log_det_jacobian(self, x):
        log_det1 = self.block1.forward_log_det_jacobian(x, event_ndims=3)
        h1 = self.block1.forward(x)
        _, h2 = tf.split(h1, 2, axis=-1)
        log_det2 = self.block2.forward_log_det_jacobian(h2, event_ndims=3)
        return log_det1 + log_det2

    def _inverse_log_det_jacobian(self, y):
        z1, z2 = tf.split(y, 2, axis=-1)
        h2 = self.block2.inverse(z2)
        log_det2 = self.block2.inverse_log_det_jacobian(z2, event_ndims=3)
        h1 = tf.concat([z1, h2], axis=-1)
        log_det1 = self.block1.inverse_log_det_jacobian(h1, event_ndims=3)
        return log_det1 + log_det2

    def _forward_event_shape_tensor(self, input_shape):
        height, width, channels = input_shape[-3], input_shape[-2], input_shape[-1]
        return height // 4, width // 4, 16 * channels

    def _inverse_event_shape_tensor(self, output_shape):
        height, width, channels = output_shape[-3], output_shape[-2], output_shape[-1]
        return 4 * height, 4 * width, channels // 16
```

```python
# Create an instance of the multiscale architecture

multiscale_bijector = RealNVPMultiScale()
```

#### Data preprocessing bijector

We will also preprocess the image data before sending it through the RealNVP model. To do this, for a Tensor $x$ of pixel values in $[0, 1]^D$, we transform $x$ according to the following:

$$
T(x) = \text{logit}\left(\alpha + (1 - 2\alpha)x\right),\tag{7}
$$

where $\alpha$ is a parameter, and the logit function is the inverse of the sigmoid function, and is given by

$$
\text{logit}(p) = \log (p) - \log (1 - p).
$$

You should now complete the following function to construct this bijector from in-built bijectors from the bijectors module.

* The function takes the parameter `alpha` as an input, which you can assume to take a small positive value ($\ll0.5$)
* The function should construct and return a bijector that computes $(7)$ in the forward pass

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_preprocess_bijector(alpha):
    """
    This function should create a chained bijector that computes the
    transformation T in equation (7) above.
    This can be computed using in-built bijectors from the bijectors module.
    Your function should then return the chained bijector.
    """
    l1=tfb.Scale(1-2*alpha)
    l2=tfb.Shift(alpha)
    l3=tfb.Invert(tfb.Sigmoid())
    return tfb.Chain([l3,l2,l1])

```

```python
# Create an instance of the preprocess bijector

preprocess = get_preprocess_bijector(0.05)
```

#### Train the RealNVP model

Finally, we will use our RealNVP model to train

We will use the following model class to help with the training process.

```python
# Helper class for training

class RealNVPModel(Model):

    def __init__(self, **kwargs):
        super(RealNVPModel, self).__init__(**kwargs)
        self.preprocess = get_preprocess_bijector(0.05)
        self.realnvp_multiscale = RealNVPMultiScale()
        self.bijector = tfb.Chain([self.realnvp_multiscale, self.preprocess])

    def build(self, input_shape):
        output_shape = self.bijector(tf.expand_dims(tf.zeros(input_shape[1:]), axis=0)).shape
        self.base = tfd.Independent(tfd.Normal(loc=tf.zeros(output_shape[1:]), scale=1.),
                                    reinterpreted_batch_ndims=3)
        self._bijector_variables = (
            list(self.bijector.variables))
        self.flow = tfd.TransformedDistribution(
            distribution=self.base,
            bijector=tfb.Invert(self.bijector),
        )
        super(RealNVPModel, self).build(input_shape)

    def call(self, inputs, training=None, **kwargs):
        return self.flow

    def sample(self, batch_size):
        sample = self.base.sample(batch_size)
        return self.bijector.inverse(sample)
```

```python
# Create an instance of the RealNVPModel class

realnvp_model = RealNVPModel()
realnvp_model.build((1, 32, 32, 3))
```

```python
# Compute the number of variables in the model

print("Total trainable variables:")
print(sum([np.prod(v.shape) for v in realnvp_model.trainable_variables]))
```

    Total trainable variables:
    411468

Note that the model's `call` method returns the `TransformedDistribution` object. Also, we have set up our datasets to return the input image twice as a 2-tuple. This is so we can train our model with negative log-likelihood as normal.

```python
# Define the negative log-likelihood loss function

def nll(y_true, y_pred):
    return -y_pred.log_prob(y_true)
```

It is recommended to use the GPU accelerator hardware on Colab to train this model, as it can take some time to train. Note that it is not required to train the model in order to pass this assignment. For optimal results, a larger model should be trained for longer.

```python
# Compile and train the model

optimizer = Adam()
realnvp_model.compile(loss=nll, optimizer=Adam())
realnvp_model.fit(train_ds, validation_data=val_ds, epochs=30)
```

    Epoch 1/30
    938/938 [==============================] - 157s 167ms/step - loss: -4198.9170 - val_loss: -6479.9341
    Epoch 2/30
    938/938 [==============================] - 155s 165ms/step - loss: -6878.4443 - val_loss: -7294.2310
    Epoch 3/30
    938/938 [==============================] - 181s 193ms/step - loss: -7429.2651 - val_loss: -7641.1729
    Epoch 4/30
    938/938 [==============================] - 158s 168ms/step - loss: -7658.2246 - val_loss: -7765.8506
    Epoch 5/30
    938/938 [==============================] - 154s 164ms/step - loss: -7988.6016 - val_loss: -8165.0649
    Epoch 6/30
    938/938 [==============================] - 154s 165ms/step - loss: -8197.5889 - val_loss: -8311.8115
    Epoch 7/30
    938/938 [==============================] - 154s 164ms/step - loss: -8205.1367 - val_loss: -8361.0029
    Epoch 8/30
    938/938 [==============================] - 154s 164ms/step - loss: -8431.5410 - val_loss: -8531.9053
    Epoch 9/30
    938/938 [==============================] - 154s 164ms/step - loss: -8528.5713 - val_loss: -8578.6035
    Epoch 10/30
    938/938 [==============================] - 155s 165ms/step - loss: -8611.7568 - val_loss: -8685.4248
    Epoch 11/30
    938/938 [==============================] - 155s 165ms/step - loss: -8699.3584 - val_loss: -8798.7529
    Epoch 12/30
    938/938 [==============================] - 153s 163ms/step - loss: -8789.7012 - val_loss: -8657.1240
    Epoch 13/30
    938/938 [==============================] - 154s 165ms/step - loss: -8840.0957 - val_loss: -8809.8057
    Epoch 14/30
    938/938 [==============================] - 153s 163ms/step - loss: -8889.6670 - val_loss: -8928.7236
    Epoch 15/30
    938/938 [==============================] - 153s 163ms/step - loss: -8931.1396 - val_loss: -8976.1582
    Epoch 16/30
    938/938 [==============================] - 154s 164ms/step - loss: -8956.2129 - val_loss: -9022.7129
    Epoch 17/30
    938/938 [==============================] - 155s 165ms/step - loss: -9004.2812 - val_loss: -8862.6758
    Epoch 18/30
    938/938 [==============================] - 153s 163ms/step - loss: -9029.0273 - val_loss: -8976.9365
    Epoch 19/30
    938/938 [==============================] - 154s 164ms/step - loss: -9052.1084 - val_loss: -9025.9404
    Epoch 20/30
    938/938 [==============================] - 154s 164ms/step - loss: -9065.9424 - val_loss: -9100.0156
    Epoch 21/30
    938/938 [==============================] - 154s 164ms/step - loss: -9086.8291 - val_loss: -9065.6318
    Epoch 22/30
    938/938 [==============================] - 153s 163ms/step - loss: -9112.7607 - val_loss: -9095.4609
    Epoch 23/30
    938/938 [==============================] - 154s 164ms/step - loss: -9128.1982 - val_loss: -9161.4229
    Epoch 24/30
    938/938 [==============================] - 153s 163ms/step - loss: -9147.2607 - val_loss: -9148.3809
    Epoch 25/30
    938/938 [==============================] - 152s 162ms/step - loss: -9157.2627 - val_loss: -9189.7930
    Epoch 26/30
    938/938 [==============================] - 153s 163ms/step - loss: -9175.0059 - val_loss: -9199.8809
    Epoch 27/30
    938/938 [==============================] - 152s 162ms/step - loss: -9173.3311 - val_loss: -9208.4639
    Epoch 28/30
    938/938 [==============================] - 152s 162ms/step - loss: -9192.8428 - val_loss: -9200.8281
    Epoch 29/30
    938/938 [==============================] - 152s 162ms/step - loss: -9204.8057 - val_loss: -9239.0176
    Epoch 30/30
    938/938 [==============================] - 153s 163ms/step - loss: -9215.8760 - val_loss: -9232.9316

    <tensorflow.python.keras.callbacks.History at 0x7fc2a1ae7990>

```python
# Evaluate the model

realnvp_model.evaluate(test_ds)
```

    157/157 [==============================] - 14s 90ms/step - loss: -9226.8008

    -9226.80078125

#### Generate some samples

```python
# Sample from the model

samples = realnvp_model.sample(8).numpy()
```

    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow_probability/python/bijectors/batch_normalization.py:210: Layer.apply (from tensorflow.python.keras.engine.base_layer) is deprecated and will be removed in a future version.
    Instructions for updating:
    Please use `layer.__call__` method instead.

```python
# Display the samples

n_img = 8
f, axs = plt.subplots(2, n_img // 2, figsize=(14, 7))

for k, image in enumerate(samples):
    i = k % 2
    j = k // 2
    axs[i, j].imshow(image[0])
    axs[i, j].axis('off')
f.subplots_adjust(wspace=0.01, hspace=0.03)
```

Congratulations on completing this programming assignment! In the next week of the course we will look at the variational autoencoder.

Week3/Week 3 Programming Assignment.ipynb


```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))
from formats import load_style
load_style(css_style = 'custom2.css')
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. a ipython magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format = 'retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.metrics import mean_squared_error
from sklearn.tree import DecisionTreeRegressor
from sklearn.model_selection import train_test_split, GridSearchCV
from sklearn.ensemble import RandomForestRegressor, GradientBoostingRegressor

%watermark -d -t -v -p numpy,pandas,matplotlib,sklearn
```

    2017-09-30 09:54:48

    CPython 3.5.2
    IPython 6.1.0

    numpy 1.13.1
    pandas 0.20.3
    matplotlib 2.0.0
    sklearn 0.18.1

# Gradient Boosting Machine (GBM)

Just like [Random Forest and Extra Trees](http://nbviewer.jupyter.org/github/ethen8181/machine-learning/blob/master/trees/random_forest.ipynb), Gradient Boosting Machine is also a type of Ensemble Tree method, the only difference is it is stemmed from the the boosting framework. The idea of boosting is to add a weak classifier to the ensemble at a time, and this newly added weak classifier is trained to improve upon the already trained ensemble. Meaning it will pay higher attention on examples which are misclassiﬁed or have higher errors and focus on mitigating those errors. Boosting is a general framework can be applied to any sort of weak learner, although Decision Tree models is by far the commonly used due to the fact that they have the flexibility to be weak learners by simply restricting their depth and they are quite fast to train.

Suppose we are given some dataset $(x_1, y_1), (x_2, y_2), ...,(x_n, y_n)$, and the task is to fit a model $F(x)$ to minimize square loss. After training the model, we discovered the model is good but not perfect.
There are some mistakes: $F(x_1) = 0.8$, while $y_1 = 0.9$, and $F(x_2) = 1.4$ while $y_2 = 1.3$ .... Now the question is, how can we improve this model without changing anything from $F(x)$?

How about we simply add an additional model (e.g. regression tree) $h$ to the already existing $F$, so the new prediction becomes $F(x) + h(x)$. In other words, we wish to improve upon the existing model so that $F(x_1) + h(x_1) = y_1, F(x_2) + h(x_2) = y_2 ...$ or equivalent we wish to find a new model $h$ such that $h(x_1) = y_1 - F(x_1), h(x_2) = y_2 - F(x_2) ...$. The idea is all well and good, but the bad news is probably no model $h$ (e.g. regression tree) will be able to do this perfectly. Fortunately, the good news is, some $h$ might be able to do this approximately.

The idea is, we fit the model $h$ to the data using $y_1 - F(x_1), y_2 - F(x_2)$ as the response variable. And the intuition for this is: the $y_i - F(x_i)$s are the residuals. These are the areas that the existing
model $F$ cannot do well, so now the role of $h$ is to compensate the shortcoming of existing model $F$. And if the model after adding the new model $h$, $F + h$ is still unsatisfactory, we will just add another new one.

To make sure we're actually learning the residuals, we'll employ the idea of gradient descent. Say our goal is to minimize $J$, an overall loss function additively calculated from all observations with regard to $F$, a classifier with some parameters. More formally, we're given the formula:

$$J(y, F) = \sum_i^n L\big(y_i, F(x_i)\big)$$

Where:

- $L$ is a cost/loss function comparing the response variable's value and the prediction of the model for each observation

Instead of trying to solve it directly, gradient descent is an iterative technique that allows us to approach the solution of an optimization problem. At each step of the algorithm, it will perform the following operations:

$$F_b(x_i) = F_{b-1}(x_i) - \eta \times \nabla L\big(y_i, F(x_i)\big)$$

Where:

- $F_b$ is the version of classifier at step/iteration $b$
- $\eta$ is the learning rate which controls the size of the learning process
- $\nabla$ is the gradient i.e. the first order partial derivative of the cost function with respect to the classifier
- The formula above actually refers to stochastic gradient descent only computing the function for a single observation, $x_i$

For example, say we're given, sum of squares errors, a well-known quality indicator for regression model as our loss function. So now our loss function $L\big(y_i, F(x_i)\big)$ is defined as: $\frac{1}{2} \big( y_i - F(x_i) \big)^2$ (the 1/2 is simply to make the notation cleaner later). Taking the gradient of this loss function we get:

$$\frac{ \partial L\big(y_i, F(x_i)\big) }{ \partial F(x_i) } = \frac{ \partial \frac{1}{2} \big( y_i - F(x_i) \big)^2 }{ \partial F(x_i) } = F(x_i) - y_i$$

Tying this back to our original problem, we wish to update our function $F$ at iteration $b$ with a new model $h$:

\begin{align}
F_b(x_i) &= F_{b-1}(x_i) + h(x_i) \nonumber \\
         &= F_{b-1}(x_i) + y_i - F_{b-1}(x_i) \nonumber \\
         &= F_{b-1}(x_i) - 1 \times \frac{ \partial L\big(y_i, F_{b-1}(x_i)\big) }{ \partial F_{b-1}(x_i) }
         \nonumber \\
\end{align}

As we can see, the formula above is 99% the same as as the gradient descent formula, $F_b(x_i) = F_{b-1}(x_i) - \eta \times \nabla L\big(y_i, F(x_i)\big)$. The only difference is that the learning rate $\eta$ is  1. Thus, we now have an iterative process constructing the additive model that minimizes our loss function (residuals).

In practice though, Gradient Boosting Machine is more prone to overfitting, since the week learner is tasked with optimally fitting the gradient. This means that boosting will select the optimal learner at each stage of the algorithm, although this strategy generates an optimal solution at the current stage, it has the drawbacks of not finding the optimal global model as well as overfitting the training data. A remedy for greediness is to constrain the learning process by setting the learning rate $\eta$ (also known as shrinkage). In the above algorithm, instead of directly adding the predicted value for a sample to next iteration's predicted value, so that only a fraction of the current predicted value is added to the previous iteration's predicted value. This parameter can take values between 0 and 1 and becomes another tuning parameter for the model. Small values of the learning parameter such as 0.1 tends to work better, but the value of the parameter is inversely proportional to the computation time required to find an optimal model, because more iterations is required.

To sum it all up, the process of training a GBM for regression is:

1. Initialize a predicted value for each observation (e.g. the original response or the average response or a value that minimizes the loss function). This will be our initial "residuals", $r$. It can be called the residuals because we're dealing with a regression task, but this quantity is more often referred to as the negative gradient, this terminology makes the $- \nabla \times L\big(y_i, F(x_i) \big)$ part generalizes to any loss function we might wish to employ. In short, GBM is fitting to the gradient of the loss function
2. For step = 1 to $B$ (number of iterations that we specify) do:
    - Fit a regression tree $F_b$ to the training data $(X, r)$, where we use the residuals as the response variable
    - Update model $F$ by adding a shrunken version of the newly fitted regression tree. Translating it to code, this means we append the new tree to the array of trees we've already stored:
    $F(X) = F(X) + \eta F_{b}(X)$
    - Update each observation's residual by adding the predicted value to it:
    $r_{b + 1} = r_b - \eta F_b(X)$
3. In the end, our final output boosted model becomes $F(x) = \sum_{b = 1}^B \eta F_b(x)$, where we sum the values that each individual tree gives (times the learning rate)

To hit the notion home, let's conside an example using made up numbers. Suppose we have 5 observations, with responses 10, 20, 30, 40, 50. The first tree is built and gives predictions of 12, 18, 27, 39, 54 (these predictions are made up numbers). If our learning rate $\eta$ = 0.1, all trees will have their predictions scaled down by $\eta$, so the first tree will instead "predict" 1.2, 1.8, 2.7, 3.9, 5.4. The response variable passed to the next tree will then have values 8.8, 18.2, 27.3, 36.1, 44.6 (the difference between the prediction that was scaled down by the prediction and the true response). The second round then uses these response values to build another tree - and again the predictions are scaled down by the learning rate $\eta$. So tree 2 predicts say, 7, 18, 25, 40, 40, which, once scaled, become 0.7, 1.8, 2.5, 4.0, 4.0. As before, the third tree will be passed the difference between these values and the previous tree's response variable (so 8.1, 16.4, 24.8, 32.1. 40.6). And we keep iterating this process until we finished training all the trees (a parameter that we specify), in the end, the sum of the predictions from all trees will give the final prediction.

## Implementation

Here, we will use the [Wine Quality Data Set](https://archive.ics.uci.edu/ml/datasets/Wine+Quality) to test our implementation. This [link](https://archive.ics.uci.edu/ml/machine-learning-databases/wine-quality/winequality-white.csv) should download the .csv file. The task is to predict the quality of the wine (a scale of 1 ~ 10) given some of its features.

```python
# read in the data and shuffle the row order for model stability
np.random.seed(4321)
wine_path = os.path.join('..', 'winequality-white.csv')
wine = pd.read_csv(wine_path, sep = ';')
wine = wine.sample(frac = 1)

# train/test split the features and response column
y = wine['quality'].values
X = wine.drop('quality', axis = 1).values
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state = 1234)

print('dimension of the dataset: ', wine.shape)
wine.head()
```

    dimension of the dataset:  (4898, 12)

      fixed acidity
      volatile acidity
      citric acid
      residual sugar
      chlorides
      free sulfur dioxide
      total sulfur dioxide
      density
      pH
      sulphates
      alcohol
      quality

      857
      8.2
      0.40
      0.48
      13.70
      0.042
      59.0
      169.0
      0.99860
      3.10
      0.52
      9.4
      5

      2325
      7.0
      0.35
      0.17
      1.10
      0.049
      7.0
      119.0
      0.99297
      3.13
      0.36
      9.7
      6

      279
      7.0
      0.30
      0.49
      4.70
      0.036
      17.0
      105.0
      0.99160
      3.26
      0.68
      12.4
      7

      1687
      7.3
      0.26
      0.33
      17.85
      0.049
      41.5
      195.0
      1.00000
      3.06
      0.44
      9.1
      7

      531
      6.4
      0.18
      0.48
      4.00
      0.186
      64.0
      150.0
      0.99450
      3.06
      0.40
      9.3
      5

```python
class GBMReg:
    """
    Regression gradient boosting machine using scikit learn's
    decision tree as the base tree

    Parameters
    ----------
    n_estimators: int
        number of trees to train

    learning_rate: float
        learning rate, some calls it shrinkage,
        shrinks the contribution of each tree
        to prevent overfitting

    max_depth: int
        controls how deep to grow the tree;
        this is more of a decision tree parameter,
        it is tune here to make later comparison fair

    all the other parameters for a decision tree like
    max_features or min_sample_split also applies to GBM,
    it is just not used here as that is more
    related to a single decision tree
    """

    def __init__(self, n_estimators, learning_rate, max_depth):
        self.max_depth = max_depth
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate

    def fit(self, X, y):
        self.estimators = []

        # simply use the response as the original residuals
        # and covert it to float type to prevent error warning
        # that it's converting from int to float
        residual = y.astype(np.float)
        for i in range(self.n_estimators):
            tree = DecisionTreeRegressor(max_depth = self.max_depth)
            tree.fit(X, residual)
            y_pred = tree.predict(X)
            self.estimators.append(tree)
            residual -= self.learning_rate * y_pred

        return self

    def predict(self, X):
        y_pred = np.zeros(X.shape[0])
        for tree in self.estimators:
            y_pred += self.learning_rate * tree.predict(X)

        return y_pred
```

```python
# compare the results between a single decision tree,
# gradient boosting, the lower the mean square
# error, the better
tree = DecisionTreeRegressor(max_depth = 6)
tree.fit(X_train, y_train)
tree_y_pred = tree.predict(X_test)
print('tree: ', mean_squared_error(y_test, tree_y_pred))

# library to confirm result
gbm_reg = GBMReg(n_estimators = 100, learning_rate = 0.1, max_depth = 6)
gbm_reg.fit(X_train, y_train)
gbm_reg_y_pred = gbm_reg.predict(X_test)
print('gbm: ', mean_squared_error(y_test, gbm_reg_y_pred))

# gradient boosting for 100 trees and learning rate of 0.1
gbm = GradientBoostingRegressor(n_estimators = 100, learning_rate = 0.1, max_depth = 6)
gbm.fit(X_train, y_train)
gbm_y_pred = gbm.predict(X_test)
print('gbm library: ', mean_squared_error(y_test, gbm_y_pred))
```

    tree:  0.547700721197
    gbm:  0.420772942612
    gbm library:  0.422106982302

Clearly, Gradient Boosting has some similarities to Random Forests and Extra Trees: the final prediction is based on an ensemble of models, and trees are used as the base learner, so all the tuning parameters for the tree model also controls the variability of Gradient Boosting. And for interpretability we can also access the feature importance attribute.

```python
def viz_importance(model, feature_names, n_features):
    """Visualize the relative importance of predictors"""
    # sort the importance in decreasing order
    importances = model.feature_importances_
    idx = np.argsort(importances)[-n_features:]
    names = feature_names[idx]
    scores = importances[idx]

    y_pos = np.arange(1, n_features + 1)
    plt.barh(y_pos, scores, color = 'lightskyblue', align = 'center')
    plt.yticks(y_pos, names)
    plt.xlabel('Importance')
    plt.title('Feature Importance Plot')
```

```python
# change default figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12

viz_importance(gbm, wine.columns[:-1], X.shape[1])
```

But the way the ensembles are constructed differs substantially between each model. In Random Forests and Extra Trees, all trees are created independently and each tree contributes equally to the final model. The trees in Gradient Boosting, however, are dependent on past trees and contribute unequally to the final model. Despite these differences, Random Forests, Extra Trees and Gradient Boosting all offer competitive predictive performance (Gradient Boosting often wins when carefully tuned). As for computation time, Gradient Boosting is often greater than for Random Forests, Extra Trees, since the two former models' procedure can be easily parallel processed given that their individual trees are created independently.

## Classification

Gradient Boosting Machine can also be extended to handle classification tasks, as we'll soon see, even in the classification context, the underlying algorithm is still a regression tree. To adapt the algorithm to a classification process, we start by defining a new loss function, cross entropy (also known as multinomial deviance), denoted as:

$$L\big(y_i, F(x_i)\big) = -\sum_k ^ K y_k(x_i) \log p_k(x_i)$$

The notation above says:

- We have a total of $K$ output class (categorical response variable) that ranges from $1, ..., K$
- $y_k(x_i)$ is a dummy indicator of the response variable that takes the value of 1 if the $i_{th}$ observation belongs to class $k$ and 0 otherwise
- $p_k(x_i)$ is the predicted probability of the $i_{th}$ observation belonging to class $k$

So the next question is how do we get $p_k(x_i)$?

### Softmax

Softmax function takes an $N$-dimensional vector of arbitrary real values and produces another $N$-dimensional vector with real values in the range (0, 1) that add up to 1. The function's formula can be written as:

$$p_i = \frac{e^{o_i}}{\sum_k^K e^{o_k}}$$

For example, in the following code chunk, we see that how the softmax function transforms a 3-element vector 1.0, 2.0, 3.0 into probabilities that sums up to 1, while still preserving the relative size of the original elements.

```python
def compute_softmax(x):
    """compute the softmax of vector"""
    exp_x = np.exp(x)
    softmax = exp_x / np.sum(exp_x)
    return softmax

# this can be interpreted as the probability
# of belonging to the three classes
compute_softmax([1, 2, 3])
```

    array([ 0.09003057,  0.24472847,  0.66524096])

Next, we wish to compute the derivative of this function with respect to the input $o_i$ so we can use it later when computing the derivative of the loss function. To be explicit we wish to find:

$$\frac{\partial p_i}{\partial o_j} = \frac{\partial \frac{e^{o_i}}{\sum_{k=1}^{N}e^{o_k}}}{\partial o_j}$$

For any arbitrary output $i$ and input $j$. To do so, We'll be using the quotient rule of derivatives. The rule tells us that for a function $f(x) = \frac{g(x)}{h(x)}$:

$$f'(x) = \frac{g'(x)h(x) - h'(x)g(x)}{[h(x)]^2}$$

In our case, we have:

\begin{align*}
g &= e^{o_i} \nonumber \\
h &= \sum_{k=1}^{K}e^{o_k} \nonumber
\end{align*}

It's important to notice that no matter which $o_j$ we compute the derivative of $h$ for the output will always be $e^{o_j}$. Howewer, this is not the case for $g$. It's derivative will be $e^{o_j}$ only if $i = j$, because only then will it have the term $e^{o_j}$. Otherwise, the derivative is simply 0 (because it's simply taking the derivative of a constant).

So going back to using our quotient rule, we start with the $i = j$ case. In the following derivation we'll use the $\Sigma$ (Sigma) sign to represent $\sum_{k=1}^{K}e^{o_k}$ for simplicity and to prevent cluttering up the notation.

\begin{align*}
\frac{\partial \frac{e^{o_i}}{\sum_{k = 1}^{N} e^{o_k}}}{\partial o_j}
&= \frac{e^{o_i}\Sigma-e^{o_j}e^{o_i}}{\Sigma^2} \nonumber \\
&= \frac{e^{o_i}}{\Sigma}\frac{\Sigma - e^{o_j}}{\Sigma} \nonumber \\
&= p_i(1 - p_j) \nonumber \\
&= p_i(1 - p_i) \nonumber
\end{align*}

The reason we can perform the operation in the last line is because we're considering the scenario where $i = j$. Similarly we can do the case where $i \neq j$.

\begin{align*}
\frac{\partial \frac{e^{o_i}}{\sum_{k = 1}^{N} e^{o_k}}}{\partial o_j}
&= \frac{0-e^{o_j}e^{o_i}}{\Sigma^2} \nonumber \\
&= -\frac{e^{o_j}}{\Sigma}\frac{e^{o_i}}{\Sigma} \nonumber \\
&= -p_j p_i \nonumber \\
&= -p_i p_j \nonumber
\end{align*}

Just to sum it up, we now have:

$$\frac{\partial p_i}{\partial o_j} = p_i(1 - p_i),\quad i = j$$

$$\frac{\partial p_i}{\partial o_j} = -p_i p_j,\quad i \neq j$$

Given this, we can tie this back to the original loss function $-\sum_k^K y_k \log p_k$ and compute its negative gradient.

\begin{align}
\frac{\partial L}{\partial o_i}
&= -\sum_k y_k\frac{\partial \log p_k}{\partial o_i} \nonumber \\
&= -\sum_k y_k\frac{1}{p_k}\frac{\partial p_k}{\partial o_i} \nonumber \\
&= -y_i(1-p_i) - \sum_{k \neq i}y_k\frac{1}{p_k}(-p_kp_i) \nonumber \\
&= -y_i(1 - p_i) + \sum_{k \neq i}y_k(p_i) \nonumber \\
&= -y_i + y_i p_i + \sum_{k \neq i}y_k(p_i) \nonumber \\
&= p_i\left(\sum_ky_k\right) - y_i \nonumber \\
&= p_i - y_i \nonumber
\end{align}

Remember $\sum_ky_k=1$ (as $y$ is a vector with only one non-zero element, which is $1$ when the indicating the observation belongs to the $k_{th}$ class.

After a long journey, we now see, for every class $k$, the gradient is the difference between the associated dummy variable and the predicted probability of belonging to that class. This is essentially the "residuals" from the classification gradient boosting. Given this, we can now implement the algorithm, the overall process of training a regression tree has still not changed, only now we must deal with the dummy variables, $y_k$ and fit a regression tree on the negative gradient for each dummy variable.

## Implementation

For the dataset, we'll still use the Wine Quality Data Set that was used for the regression task, except we now treat the quality of the wine (a scale of 1 ~ 10) as categorical instead of numeric.

```python
from sklearn.metrics import accuracy_score
from sklearn.preprocessing import LabelEncoder
from sklearn.tree import DecisionTreeClassifier
from sklearn.ensemble import GradientBoostingClassifier
```

```python
class GBMClass:
    """
    Classification gradient boosting machine using scikit learn's
    decision tree as the base tree

    Parameters
    ----------
    n_estimators: int
        number of trees to train

    learning_rate: float
        learning rate, some calls it shrinkage,
        shrinks the contribution of each tree
        to prevent overfitting

    max_depth: int
        controls how deep to grow the tree;
        this is more of a decision tree parameter,
        it is tune here to make later comparison fair

    all the other parameters for a decision tree like
    max_features or min_sample_split also applies to GBM,
    it is just not used here as that is more
    related to a single decision tree
    """

    def __init__(self, n_estimators, learning_rate, max_depth):
        self.max_depth = max_depth
        self.n_estimators = n_estimators
        self.learning_rate = learning_rate

    def fit(self, X, y):
        # encode labels with value between 0 and n_classes - 1,
        # so we can easily one-hot encode them
        self.le = LabelEncoder()
        labels = self.le.fit_transform(y)
        Y = self._to_categorical(labels)
        del labels

        # the predicted probability starts out with
        # a value that's uniform over all classes;
        # then we compute the residuals (negative gradient),
        # which is the difference between the predicted
        # probability and the class label
        y_proba = np.full(Y.shape, 1 / Y.shape[1])
        residuals = Y - y_proba

        # train a base decision tree on the residuals
        # for every single class, hence we end up with
        # n_estimators * n_classes base tree models
        self.estimators = []
        for i in range(self.n_estimators):
            for j in range(self.n_classes):
                tree = DecisionTreeRegressor(max_depth = self.max_depth)
                tree.fit(X, residuals[:, j])
                y_pred = tree.predict(X)
                self.estimators.append(tree)
                residuals[:, j] -= self.learning_rate * y_pred

        return self

    def _to_categorical(self, y):
        """one hot encode class vector y"""
        self.n_classes = np.amax(y) + 1
        Y = np.zeros((y.shape[0], self.n_classes))
        for i in range(y.shape[0]):
            Y[i, y[i]] = 1.0

        return Y

    def predict(self, X):
        # after predicting the class remember to
        # transform it back to the actual class label
        y_prob = self.predict_proba(X)
        y_pred = np.argmax(y_prob, axis = 1)
        y_pred = self.le.inverse_transform(y_pred)
        return y_pred

    def predict_proba(self, X):
        # add up raw score for every class and convert
        # it to probability using softmax
        y_raw = np.zeros((X.shape[0], self.n_classes))

        # obtain the tree for each class and add up the prediction
        for c in range(self.n_classes):
            class_tree = self.estimators[c::self.n_classes]
            for tree in class_tree:
                y_raw[:, c] += self.learning_rate * tree.predict(X)

        y_proba = self._compute_softmax(y_raw)
        return y_proba

    def _compute_softmax(self, z):
        """
        compute the softmax of matrix z in a numerically stable way,
        by substracting each row with the max of each row
        """
        shift_z = z - np.amax(z, axis = 1, keepdims = 1)
        exp_z = np.exp(shift_z)
        softmax = exp_z / np.sum(exp_z, axis = 1, keepdims = 1)
        return softmax
```

```python
# compare the results between a single decision tree,
# gradient boosting, the higher the accuracy, the better
tree = DecisionTreeClassifier(max_depth = 6)
tree.fit(X_train, y_train)
tree_y_pred = tree.predict(X_test)
print('tree: ', accuracy_score(y_test, tree_y_pred))

# gradient boosting for 150 trees and learning rate of 0.2
# unlike random forest, gradient boosting's base tree can be shallower
# meaning that there depth can be smaller
gbm_class = GBMClass(n_estimators = 150, learning_rate = 0.2, max_depth = 3)
gbm_class.fit(X_train, y_train)
gbm_class_y_pred = gbm_class.predict(X_test)
print('gbm: ', accuracy_score(y_test, gbm_class_y_pred))

# library to confirm results are comparable
gbm = GradientBoostingClassifier(n_estimators = 150, learning_rate = 0.2, max_depth = 3)
gbm.fit(X_train, y_train)
gbm_y_pred = gbm.predict(X_test)
print('gbm library: ', accuracy_score(y_test, gbm_y_pred))
```

    tree:  0.54387755102
    gbm:  0.601020408163
    gbm library:  0.616326530612

## Understanding Model Complexity

In the following section, we generate a Sinoide function + random gaussian noise, with 80 training samples (blue points) and 20 test samples (red points).

```python
def ground_truth(x):
    """Ground truth -- function to approximate"""
    return x * np.sin(x) + np.sin(2 * x)

def gen_data(low, high, n_samples):
    """generate training and testing data from the ground truth function"""
    np.random.seed(15)
    X = np.random.uniform(low, high, size = n_samples)

    # generate the response from the ground truth function and add
    # some random noise to it
    y = ground_truth(X) + np.random.normal(scale = 2, size = n_samples)
    X_train, X_test, y_train, y_test = train_test_split(
        X, y, test_size = 0.2, random_state = 3)

    return X_train, X_test, y_train, y_test

def plot_data(x_plot, X_train, X_test, y_train, y_test):
    """plot training and testing data"""
    s = 20
    alpha = 0.4
    plt.plot(x_plot, ground_truth(x_plot), alpha = alpha, label = 'ground truth')
    plt.scatter(X_train, y_train, s = s, alpha = alpha)
    plt.scatter(X_test, y_test, s = s, alpha = alpha, color = 'red')
    plt.xlim(( 0, 10 ))
    plt.ylabel('y')
    plt.xlabel('x')
    plt.legend(loc = 'upper left')
    plt.show()
```

```python
low = 0
high = 10
x_plot = np.linspace(low, high, 500)
X_train, X_test, y_train, y_test = gen_data(low = low, high = high, n_samples = 100)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

Recall that in a single regression tree, we can use the `max_depth` parameter to control how deep to grow the tree and the deeper the tree the more variance can be explained.

```python
# when using scikit-learn, the training data has to be
# a 2d-array even if it only has 1 features
tree1 = DecisionTreeRegressor(max_depth = 1)
tree1.fit(X_train[:, np.newaxis], y_train)
tree2 = DecisionTreeRegressor(max_depth = 3)
tree2.fit(X_train[:, np.newaxis], y_train)

plt.plot(x_plot, tree1.predict(x_plot[:, np.newaxis]),
         label = 'RT max_depth=1', color = 'g', alpha = 0.9, linewidth = 2)
plt.plot(x_plot, tree2.predict(x_plot[:, np.newaxis]),
         label = 'RT max_depth=3', color = 'g', alpha = 0.7, linewidth = 1)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

The plot above shows that the decision boundaries made by decision trees are always perpendicular to $x$ and $y$ axis (due to the fact that they consists of nested if-else statements). Let's see what happens when we use gradient boosting without tuning the parameters (by specifying a fix `max_depth`).

```python
gbm = GradientBoostingRegressor(n_estimators = 300, max_depth = 6, learning_rate = 0.1)
gbm.fit(X_train[:, np.newaxis], y_train)

plt.plot(x_plot, gbm.predict(x_plot[:, np.newaxis]),
         label = 'GBM max_depth=6', color = 'r', alpha = 0.9, linewidth = 2)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

Hopefully, it should be clear that compared with decision trees, gradient boosting machine is far more susceptible to overfitting the training data, hence it is common to tune parameters including `max_depth`, `max_features`, `min_samples_leaf`, `subsample` (explained below) to reduce the overfitting phenomenon from  occurring.

The parameter `subsample` (technically called *stochastic gradient boosting*) borrows some idea from bagging techniques. What it does is: while iterating through each individual tree building process, it randomly select a fraction of the training data. Then the residuals and models in the remaining steps of the current iteration are based only on that sample of data. It turns out that this simple modification improved the predictive accuracy of boosting while also reducing the required computational resources (of course, this is based on the fact that you have enough observations to subsample).

The following section tunes the commonly tuned parameter and find the best one and draws the decision boundary. The resulting plot should be self-explanatory.

```python
param_grid = {
    'max_depth': [4, 6],
    'min_samples_leaf': [3, 5, 8],
    'subsample': [0.9, 1]
    # 'max_features': [1.0, 0.3, 0.1] # not possible in this example (there's only 1)
}
gs_gbm = GridSearchCV(gbm, param_grid, scoring = 'neg_mean_squared_error', n_jobs = 4)
gs_gbm.fit(X_train[:, np.newaxis], y_train)
print('Best hyperparameters: %r' % gs_gbm.best_params_)

plt.plot(x_plot, gs_gbm.predict(x_plot[:, np.newaxis]),
         label = 'GBM tuned', color = 'r', alpha = 0.9, linewidth = 2)
plot_data(x_plot, X_train, X_test, y_train, y_test)
```

# Reference

- [Slide: Gradent boosting](http://eric.univ-lyon2.fr/~ricco/cours/slides/en/gradient_boosting.pdf)
- [Slide: A gentle introduction to gradient boosting](http://www.ccs.neu.edu/home/vip/teach/MLcourse/4_boosting/slides/gradient_boosting.pdf)
- [Blog: The Softmax function and its derivative](http://eli.thegreenplace.net/2016/the-softmax-function-and-its-derivative/)
- [Notebook: Regression Trees and Rule-Based Models](http://nbviewer.jupyter.org/github/leig/Applied-Predictive-Modeling-with-Python/blob/master/notebooks/Chapter%208.ipynb)
- [Notebook: Gradient Boosted Regression Trees in scikit-learn](http://nbviewer.jupyter.org/github/pprett/pydata-gbrt-tutorial/blob/master/gbrt-tutorial.ipynb)
- [StackExchange: Derivative of Softmax loss function](http://math.stackexchange.com/questions/945871/derivative-of-softmax-loss-function)
- [StackExchange: How are individual trees added together in boosted regression tree?](http://stats.stackexchange.com/questions/135378/how-are-individual-trees-added-together-in-boosted-regression-tree)
- [Stackoverflow: How to access weighting of indiviual decision trees in xgboost?](https://stackoverflow.com/questions/32950607/how-to-access-weighting-of-indiviual-decision-trees-in-xgboost/34331573#34331573)


# Pytex

Usage examples of pytex package.

```python
import pytex
```

Create a document object and specify the configuration.

```python
doc = pytex.Document(
    filename="example",
    institution="Institution Name",
    mayor_heading="Mayor Heading",
    minor_heading="Minor Heading",
    title="Probably a very long title",
    author="Author's name",
    supervisor="Supervisor's name"
)
```

Add a table of contents.

```python
doc.add_table_of_contents()
```

You can add multiple elements secuentially.

```python
doc.add_section("First Section's Title")

doc.add_paragraph("""
First test paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
""")

doc.add_paragraph("""
Second test paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
Excepteur sint occaecat cupidatat non proident,
sunt in culpa qui officia deserunt mollit anim id est laborum consectetur adipiscing elit,
sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
""")

doc.add_paragraph(r"""
This is the black-scholes equation:
""")

doc.add_equation(
    eq=r"\frac{\partial V}{\partial t} + \sigma^2 S^2 \frac{1}{2} \frac{\partial^2V}{\partial S^2} + rS\frac{\partial V}{\partial S} - rV = 0",
    label="bs")

doc.add_paragraph(r"""References work! See equation \ref{bs}.""")

doc.add_paragraph("""
These are some items:
""")

doc.add_items([
        "item1",
        "item2",
        "item3"
    ])

doc.add_paragraph("""
Now let's code some stuff.
""")

doc.add_code(
    code_string="""
    import pytex
    import math
    doc = pytex.Document(filename="file")
    doc.add_section("pytex is great")
    doc.add_paragraph("The number pi is: {}".format(math.pi))
    for i in range(10):
        doc.add_section("Section: " + str(i+2))
    """)

doc.add_paragraph("""
Great!
""")

```

Take a look to the tex file.

```python
print(doc.to_tex())
```

                \documentclass[11pt]{article}
                \usepackage[utf8]{inputenc}
                \usepackage[T1]{fontenc}
                \usepackage{amsmath}
                \usepackage{fancyhdr}
                \usepackage{listings}
                \usepackage{color}

                \definecolor{codegreen}{rgb}{0,0.6,0}
                \definecolor{codegray}{rgb}{0.5,0.5,0.5}
                \definecolor{codepurple}{rgb}{0.58,0,0.82}
                \definecolor{backcolour}{rgb}{0.95,0.95,0.92}

                \lstdefinestyle{mystyle}{
                    backgroundcolor=\color{backcolour},
                    commentstyle=\color{codegreen},
                    keywordstyle=\color{magenta},
                    numberstyle=\tiny\color{codegray},
                    stringstyle=\color{codepurple},
                    basicstyle=\footnotesize,
                    breakatwhitespace=false,
                    breaklines=true,
                    captionpos=b,
                    keepspaces=true,
                    numbers=left,
                    numbersep=5pt,
                    showspaces=false,
                    showstringspaces=false,
                    showtabs=false,
                    tabsize=2
                }

                \lstset{style=mystyle}

                \parindent 0em
                \parskip 2ex
                \pagestyle{fancy}
                \setlength{\textfloatsep}{5pt}
                \begin{document}

                \begin{titlepage}
                \newcommand{\HRule}{\rule{\linewidth}{0.5mm}}
                \center

                \textsc{\LARGE Institution Name}\\[1.5cm]
                \textsc{\Large Mayor Heading}\\[0.5cm]
                \textsc{\large Minor Heading}\\[0.5cm]

                \HRule\\[0.4cm]
                {\huge\bfseries Probably a very long title}\\[0.4cm]
                \HRule\\[1.5cm]

                \begin{minipage}{0.4\textwidth}
                    \begin{flushleft}
                    \large
                    \textit{Author}\\
                    Author's name
                    \end{flushleft}
                \end{minipage}
                ~
                \begin{minipage}{0.4\textwidth}
                    \begin{flushright}
                    \large
                    \textit{Supervisor}\\
                    Supervisor's name
                    \end{flushright}
                \end{minipage}

                \vfill\vfill\vfill
                {\large\today}

                \vfill
                \end{titlepage}

    \tableofcontents\newpage
    \section{First Section's Title}

    First test paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
    sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
    Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
    Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
    Excepteur sint occaecat cupidatat non proident,
    sunt in culpa qui officia deserunt mollit anim id est laborum consectetur adipiscing elit,
    sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
    Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

    Second test paragraph. Lorem ipsum dolor sit amet, consectetur adipiscing elit,
    sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
    Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.
    Duis aute irure dolor in reprehenderit in voluptate velit esse cillum dolore eu fugiat nulla pariatur.
    Excepteur sint occaecat cupidatat non proident,
    sunt in culpa qui officia deserunt mollit anim id est laborum consectetur adipiscing elit,
    sed do eiusmod tempor incididunt ut labore et dolore magna aliqua.
    Ut enim ad minim veniam, quis nostrud exercitation ullamco laboris nisi ut aliquip ex ea commodo consequat.

    This is the black-scholes equation:

                \begin{equation} \label{bs}
                \frac{\partial V}{\partial t} + \sigma^2 S^2 \frac{1}{2} \frac{\partial^2V}{\partial S^2} + rS\frac{\partial V}{\partial S} - rV = 0
                \end{equation}

    References work! See equation \ref{bs}.

    These are some items:

                \begin{itemize}
                \item item1\item item2\item item3
                \end{itemize}

    Now let's code some stuff.

                \begin{lstlisting}[language=Python]

        import pytex
        import math
        doc = pytex.Document(filename="file")
        doc.add_section("pytex is great")
        doc.add_paragraph("The number pi is: {}".format(math.pi))
        for i in range(10):
            doc.add_section("Section: " + str(i+2))

                \end{lstlisting}

    Great!

    \end{document}

Compile to pdf file.

```python
doc.compile()
```

```python

```