# Probabilistic Matrix Factorization for Making Personalized Recommendations

The model discussed in this analysis was developed by Ruslan Salakhutdinov and Andriy Mnih. All of the code and supporting text, when not referenced, is the original work of [Mack Sweeney](https://www.linkedin.com/in/macksweeney).

## Motivation

Say I download a handbook of a hundred jokes, and I'd like to know very quickly which ones will be my favorite. So maybe I read a few, I laugh, I read a few more, I stop laughing, and I indicate on a scale of -10 to 10 how funny I thought each joke was. Maybe I do this for 5 jokes out of the 100. Now I go to the back of the book, and there's a little program included for calculating my preferences for all the other jokes. I enter in my preference numbers and shazam! The program spits out a list of all 100 jokes, sorted in the order I'll like them. That certainly would be nice. Today we'll write a program that does exactly this.

We'll start out by getting some intuition for how our model will work. Then we'll formalize our intuition. Afterwards, we'll examine the dataset we are going to use. Once we have some notion of what our data looks like, we'll define some baseline methods for predicting preferences for jokes. Following that, we'll look at Probabilistic Matrix Factorization (PMF), which is a more sophisticated Bayesian method for predicting preferences. Having detailed the PMF model, we'll use PyMC3 for MAP estimation and MCMC inference. Finally, we'll compare the results obtained with PMF to those obtained from our baseline methods and discuss the outcome.

## Intuition

Normally if we want recommendations for something, we try to find people who are similar to us and ask their opinions. If Bob, Alice, and Monty are all similar to me, and they all like knock-knock jokes, I'll probably like knock-knock jokes. Now this isn't always true. It depends on what we consider to be "similar". In order to get the best bang for our buck, we really want to look for people who have the most similar sense of humor. Humor being a complex beast, we'd probably like to break it down into something more understandable. We might try to characterize each joke in terms of various factors. Perhaps jokes can be dry, sarcastic, crude, sexual, political, etc. Now imagine we go through our handbook of jokes and assign each joke a rating in each of the categories. How dry is it? How sarcastic is it? How much does it use sexual innuendos? Perhaps we use numbers between 0 and 1 for each category. Intuitively, we might call this the joke's humor profile.

Now let's suppose we go back to those 5 jokes we rated. At this point, we can get a richer picture of our own preferences by looking at the humor profiles of each of the jokes we liked and didn't like. Perhaps we take the averages across the 5 humor profiles and call this our ideal type of joke. In other words, we have computed some notion of our inherent _preferences_ for various types of jokes. Suppose Bob, Alice, and Monty all do the same. Now we can compare our preferences and determine how similar each of us really are. I might find that Bob is the most similar and the other two are still more similar than other people, but not as much as Bob. So I want recommendations from all three people, but when I make my final decision, I'm going to put more weight on Bob's recommendation than those I get from Alice and Monty.

While the above procedure sounds fairly effective as is, it also reveals an unexpected additional source of information. If we rated a particular joke highly, and we know its humor profile, we can compare with the profiles of other jokes. If we find one with very close numbers, it is probable we'll also enjoy this joke. Both this approach and the one above are commonly known as _neighborhood approaches_. Techniques that leverage both of these approaches simultaneously are often called _collaborative filtering_ [[1]](http://www2.research.att.com/~volinsky/papers/ieeecomputer.pdf). The first approach we talked about uses user-user similarity, while the second uses item-item similarity. Ideally, we'd like to use both sources of information. The idea is we have a lot of items available to us, and we'd like to work together with others to filter the list of items down to those we'll each like best. My list should have the items I'll like best at the top and those I'll like least at the bottom. Everyone else wants the same. If I get together with a bunch of other people, we all read 5 jokes, and we have some efficient computational process to determine similarity, we can very quickly order the jokes to our liking.

## Formalization

Let's take some time to make the intuitive notions we've been discussing more concrete. We have a set of $M$ jokes, or _items_ ($M = 100$ in our example above). We also have $N$ people, whom we'll call _users_ of our recommender system. For each item, we'd like to find a $D$ dimensional factor composition (humor profile above) to describe the item. Ideally, we'd like to do this without actually going through and manually labeling all of the jokes. Manual labeling would be both slow and error-prone, as different people will likely label jokes differently. So we model each joke as a $D$ dimensional vector, which is its latent factor composition. Furthermore, we expect each user to have some preferences, but without our manual labeling and averaging procedure, we have to rely on the latent factor compositions to learn $D$ dimensional latent preference vectors for each user. The only thing we get to observe is the $N \times M$ ratings matrix $R$ provided by the users. Entry $R_{ij}$ is the rating user $i$ gave to item $j$. Many of these entries may be missing, since most users will not have rated all 100 jokes. Our goal is to fill in the missing values with predicted ratings based on the latent variables $U$ and $V$. We denote the predicted ratings by $R_{ij}^*$. We also define an indicator matrix $I$, with entry $I_{ij} = 0$ if $R_{ij}$ is missing and $I_{ij} = 1$ otherwise.

So we have an $N \times D$ matrix of user preferences which we'll call $U$ and an $M \times D$ factor composition matrix we'll call $V$. We also have a $N \times M$ rating matrix we'll call $R$. We can think of each row $U_i$ as indications of how much each user prefers each of the $D$ latent factors. Each row $V_j$ can be thought of as how much each item can be described by each of the latent factors. In order to make a recommendation, we need a suitable prediction function which maps a user preference vector $U_i$ and an item latent factor vector $V_j$ to a predicted ranking. The choice of this prediction function is an important modeling decision, and a variety of prediction functions have been used. Perhaps the most common is the dot product of the two vectors, $U_i \cdot V_j$ [[1]](http://www2.research.att.com/~volinsky/papers/ieeecomputer.pdf).

To better understand CF techniques, let us explore a particular example. Imagine we are seeking to recommend jokes using a model which infers five latent factors, $V_j$, for $j = 1,2,3,4,5$. In reality, the latent factors are often unexplainable in a straightforward manner, and most models make no attempt to understand what information is being captured by each factor.  However, for the purposes of explanation, let us assume the five latent factors might end up capturing the humor profile we were discussing above. So our five latent factors are: dry, sarcastic, crude, sexual, and political. Then for a particular user $i$, imagine we infer a preference vector $U_i = <0.2, 0.1, 0.3, 0.1, 0.3>$. Also, for a particular item $j$, we infer these values for the latent factors: $V_j = <0.5, 0.5, 0.25, 0.8, 0.9>$. Using the dot product as the prediction function, we would calculate 0.575 as the ranking for that item, which is more or less a neutral preference given our -10 to 10 rating scale.

$$ 0.2 \times 0.5 + 0.1 \times 0.5 + 0.3 \times 0.25 + 0.1 \times 0.8 + 0.3 \times 0.9 = 0.575 $$

## Data

The [v1 Jester dataset](http://eigentaste.berkeley.edu/dataset/) provides something very much like the handbook of jokes we have been discussing. The original version of this dataset was constructed in conjunction with the development of the [Eigentaste recommender system](http://eigentaste.berkeley.edu/about.html) [[2]](http://goldberg.berkeley.edu/pubs/eigentaste.pdf). At this point in time, v1 contains over 4.1 million continuous ratings in the range [-10, 10] of 100 jokes from 73,421 users. These ratings were collected between Apr. 1999 and May 2003. In order to reduce the training time of the model for illustrative purposes, 1,000 users who have rated all 100 jokes will be selected randomly. We will implement a model that is suitable for collaborative filtering on this data and evaluate it in terms of root mean squared error (RMSE) to validate the results.

Let's begin by exploring our data. We want to get a general feel for what it looks like and a sense for what sort of patterns it might contain.

```python
% matplotlib inline
import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import os
import shutil
%precision 4
plt.style.use('bmh')
DATA_DIR = './data'

data = pd.read_csv(os.path.join(DATA_DIR, 'jester-dataset-v1-dense-first-1000.csv'))
data.head()
```

      1
      2
      3
      4
      5
      6
      7
      8
      9
      10
      ...
      91
      92
      93
      94
      95
      96
      97
      98
      99
      100

      0
      4.08
      -0.29
      6.36
      4.37
      -2.38
      -9.66
      -0.73
      -5.34
      8.88
      9.22
      ...
      2.82
      -4.95
      -0.29
      7.86
      -0.19
      -2.14
      3.06
      0.34
      -4.32
      1.07

      1
      -6.17
      -3.54
      0.44
      -8.50
      -7.09
      -4.32
      -8.69
      -0.87
      -6.65
      -1.80
      ...
      -3.54
      -6.89
      -0.68
      -2.96
      -2.18
      -3.35
      0.05
      -9.08
      -5.05
      -3.45

      2
      6.84
      3.16
      9.17
      -6.21
      -8.16
      -1.70
      9.27
      1.41
      -5.19
      -4.42
      ...
      7.23
      -1.12
      -0.10
      -5.68
      -3.16
      -3.35
      2.14
      -0.05
      1.31
      0.00

      3
      -3.79
      -3.54
      -9.42
      -6.89
      -8.74
      -0.29
      -5.29
      -8.93
      -7.86
      -1.60
      ...
      4.37
      -0.29
      4.17
      -0.29
      -0.29
      -0.29
      -0.29
      -0.29
      -3.40
      -4.95

      4
      1.31
      1.80
      2.57
      -2.38
      0.73
      0.73
      -0.97
      5.00
      -7.23
      -1.36
      ...
      1.46
      1.70
      0.29
      -3.30
      3.45
      5.44
      4.08
      2.48
      4.51
      4.66

5 rows × 100 columns

```python
# Extract the ratings from the DataFrame
all_ratings = np.ndarray.flatten(data.values)
ratings = pd.Series(all_ratings)

# Plot histogram and density.
fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(16, 7))
ratings.plot(kind='density', ax=ax1, grid=False)
ax1.set_ylim(0, 0.08)
ax1.set_xlim(-11, 11)

# Plot histogram
ratings.plot(kind='hist', ax=ax2, bins=20, grid=False)
ax2.set_xlim(-11, 11)
plt.show()
```

```python
ratings.describe()
```

    count    100000.000000
    mean          0.996219
    std           5.265215
    min          -9.950000
    25%          -2.860000
    50%           1.650000
    75%           5.290000
    max           9.420000
    dtype: float64

This must be a decent batch of jokes. From our exploration above, we know most ratings are in the range -1 to 10, and positive ratings are more likely than negative ratings. Let's look at the means for each joke to see if we have any particularly good (or bad) humor here.

```python
joke_means = data.mean(axis=0)
joke_means.plot(kind='bar', grid=False, figsize=(16, 6),
                title="Mean Ratings for All 100 Jokes")
```

While the majority of the jokes generally get positive feedback from users, there are definitely a few that stand out as poor humor. Let's take a look at the worst and best joke, just for fun.

```python
import json
# Worst and best joke?
worst_joke_id = joke_means.argmin()
best_joke_id = joke_means.argmax()

# Let's see for ourselves. Load the jokes.
with open(os.path.join(DATA_DIR, 'jokes.json')) as buff:
    joke_dict = json.load(buff)

print('The worst joke:\n---------------\n%s\n' % joke_dict[worst_joke_id])
print('The best joke:\n--------------\n%s' % joke_dict[best_joke_id])
```

    The worst joke:
    ---------------
    How many teddybears does it take to change a lightbulb?

    It takes only one teddybear, but it takes a whole lot of lightbulbs.

    The best joke:
    --------------
    *A radio conversation of a US naval
    ship with Canadian authorities ... *

    Americans: Please divert your course 15 degrees to the North to avoid a
    collision.

    Canadians: Recommend you divert YOUR course 15 degrees to the South to
    avoid a collision.

    Americans: This is the Captain of a US Navy ship.  I say again, divert
    YOUR course.

    Canadians: No.  I say again, you divert YOUR course.

    Americans: This is the aircraft carrier USS LINCOLN, the second largest ship in the United States' Atlantic Fleet. We are accompanied by three destroyers, three cruisers and numerous support vessels. I demand that you change your course 15 degrees north, that's ONE FIVE DEGREES NORTH, or counter-measures will be undertaken to ensure the safety of this ship.

    Canadians: *This is a lighthouse.  Your call*.

Make sense to me. We now know there are definite popularity differences between the jokes. Some of them are simply funnier than others, and some are downright lousy. Looking at the joke means allowed us to discover these general trends. Perhaps there are similar trends across users. It might be the case that some users are simply more easily humored than others. Let's take a look.

```python
user_means = data.mean(axis=1)
_, ax = plt.subplots(figsize=(16, 6))
user_means.plot(kind='bar', grid=False, ax=ax,
                title="Mean Ratings for All 1000 Users")
ax.set_xticklabels('')  # 1000 labels is nonsensical
```

We see even more significant trends here. Some users rate nearly everything highly, and some (though not as many) rate nearly everything negatively. These observations will come in handy when considering models to use for predicting user preferences on unseen jokes.

## Methods

Having explored the data, we're now ready to dig in and start addressing the problem. We want to predict how much each user is going to like all of the jokes he or she has not yet read.

### Baselines

Every good analysis needs some kind of baseline methods to compare against. It's difficult to claim we've produced good results if we have no reference point for what defines "good". We'll define three very simple baseline methods and find the RMSE using these methods. Our goal will be to obtain lower RMSE scores with whatever model we produce.

#### Uniform Random Baseline

Our first baseline is about as dead stupid as you can get. Every place we see a missing value in $R$, we'll simply fill it with a number drawn uniformly at random in the range [-10, 10]. We expect this method to do the worst by far.

$$R_{ij}^* \sim Uniform$$

#### Global Mean Baseline

This method is only slightly better than the last. Wherever we have a missing value, we'll fill it in with the mean of all observed ratings.

$$\text{global_mean} = \frac{1}{N \times M} \sum_{i=1}^N \sum_{j=1}^M I_{ij}(R_{ij})$$

$$R_{ij}^* = \text{global_mean}$$

#### Mean of Means Baseline

Now we're going to start getting a bit smarter. We imagine some users might be easily amused, and inclined to rate all jokes more highly. Other users might be the opposite. Additionally, some jokes might simply be more witty than others, so all users might rate some jokes more highly than others in general. We can clearly see this in our graph of the joke means above. We'll attempt to capture these general trends through per-user and per-joke rating means. We'll also incorporate the global mean to smooth things out a bit. So if we see a missing value in cell $R_{ij}$, we'll average the global mean with the mean of $U_i$ and the mean of $V_j$ and use that value to fill it in.

$$\text{user_means} = \frac{1}{M} \sum_{j=1}^M I_{ij}(R_{ij})$$

$$\text{joke_means} = \frac{1}{N} \sum_{i=1}^N I_{ij}(R_{ij})$$

$$R_{ij}^* = \frac{1}{3} \left(\text{user_means}_i + \text{ joke_means}_j + \text{ global_mean} \right)$$

```python
from collections import OrderedDict

# Create a base class with scaffolding for our 3 baselines.

def split_title(title):
    """Change "BaselineMethod" to "Baseline Method"."""
    words = []
    tmp = [title[0]]
    for c in title[1:]:
        if c.isupper():
            words.append(''.join(tmp))
            tmp = [c]
        else:
            tmp.append(c)
    words.append(''.join(tmp))
    return ' '.join(words)

class Baseline(object):
    """Calculate baseline predictions."""

    def __init__(self, train_data):
        """Simple heuristic-based transductive learning to fill in missing
        values in data matrix."""
        self.predict(train_data.copy())

    def predict(self, train_data):
        raise NotImplementedError(
            'baseline prediction not implemented for base class')

    def rmse(self, test_data):
        """Calculate root mean squared error for predictions on test data."""
        return rmse(test_data, self.predicted)

    def __str__(self):
        return split_title(self.__class__.__name__)

# Implement the 3 baselines.

class UniformRandomBaseline(Baseline):
    """Fill missing values with uniform random values."""

    def predict(self, train_data):
        nan_mask = np.isnan(train_data)
        masked_train = np.ma.masked_array(train_data, nan_mask)
        pmin, pmax = masked_train.min(), masked_train.max()
        N = nan_mask.sum()
        train_data[nan_mask] = np.random.uniform(pmin, pmax, N)
        self.predicted = train_data

class GlobalMeanBaseline(Baseline):
    """Fill in missing values using the global mean."""

    def predict(self, train_data):
        nan_mask = np.isnan(train_data)
        train_data[nan_mask] = train_data[~nan_mask].mean()
        self.predicted = train_data

class MeanOfMeansBaseline(Baseline):
    """Fill in missing values using mean of user/item/global means."""

    def predict(self, train_data):
        nan_mask = np.isnan(train_data)
        masked_train = np.ma.masked_array(train_data, nan_mask)
        global_mean = masked_train.mean()
        user_means = masked_train.mean(axis=1)
        item_means = masked_train.mean(axis=0)
        self.predicted = train_data.copy()
        n, m = train_data.shape
        for i in range(n):
            for j in range(m):
                if np.ma.isMA(item_means[j]):
                    self.predicted[i,j] = np.mean(
                        (global_mean, user_means[i]))
                else:
                    self.predicted[i,j] = np.mean(
                        (global_mean, user_means[i], item_means[j]))

baseline_methods = OrderedDict()
baseline_methods['ur'] = UniformRandomBaseline
baseline_methods['gm'] = GlobalMeanBaseline
baseline_methods['mom'] = MeanOfMeansBaseline
```

## Probabilistic Matrix Factorization

[Probabilistic Matrix Factorization (PMF)](http://papers.nips.cc/paper/3208-probabilistic-matrix-factorization.pdf) [3] is a probabilistic approach to the collaborative filtering problem that takes a Bayesian perspective. The ratings $R$ are modeled as draws from a Gaussian distribution.  The mean for $R_{ij}$ is $U_i V_j^T$. The precision $\alpha$ is a fixed parameter that reflects the uncertainty of the estimations; the normal distribution is commonly reparameterized in terms of precision, which is the inverse of the variance. Complexity is controlled by placing zero-mean spherical Gaussian priors on $U$ and $V$. In other words, each row of $U$ is drawn from a multivariate Gaussian with mean $\mu = 0$ and precision which is some multiple of the identity matrix $I$. Those multiples are $\alpha_U$ for $U$ and $\alpha_V$ for $V$. So our model is defined by:

$\newcommand\given[1][]{\:#1\vert\:}$

\begin{equation}
P(R \given U, V, \alpha^2) =
    \prod_{i=1}^N \prod_{j=1}^M
        \left[ \mathcal{N}(R_{ij} \given U_i V_j^T, \alpha^{-1}) \right]^{I_{ij}}
\end{equation}

\begin{equation}
P(U \given \alpha_U^2) =
    \prod_{i=1}^N \mathcal{N}(U_i \given 0, \alpha_U^{-1} \boldsymbol{I})
\end{equation}

\begin{equation}
P(V \given \alpha_U^2) =
    \prod_{j=1}^M \mathcal{N}(V_j \given 0, \alpha_V^{-1} \boldsymbol{I})
\end{equation}

Given small precision parameters, the priors on $U$ and $V$ ensure our latent variables do not grow too far from 0. This prevents overly strong user preferences and item factor compositions from being learned. This is commonly known as complexity control, where the complexity of the model here is measured by the magnitude of the latent variables. Controlling complexity like this helps prevent overfitting, which allows the model to generalize better for unseen data. We must also choose an appropriate $\alpha$ value for the normal distribution for $R$. So the challenge becomes choosing appropriate values for $\alpha_U$, $\alpha_V$, and $\alpha$. This challenge can be tackled with the soft weight-sharing methods discussed by [Nowland and Hinton, 1992](http://www.cs.toronto.edu/~fritz/absps/sunspots.pdf) [4]. However, for the purposes of this analysis, we will stick to using point estimates obtained from our data.

```python
import time
import logging
import pymc3 as pm
import theano
import scipy as sp

# Enable on-the-fly graph computations, but ignore
# absence of intermediate test values.
theano.config.compute_test_value = 'ignore'

# Set up logging.
logger = logging.getLogger()
logger.setLevel(logging.INFO)

class PMF(object):
    """Probabilistic Matrix Factorization model using pymc3."""

    def __init__(self, train, dim, alpha=2, std=0.01, bounds=(-10, 10)):
        """Build the Probabilistic Matrix Factorization model using pymc3.

        :param np.ndarray train: The training data to use for learning the model.
        :param int dim: Dimensionality of the model; number of latent factors.
        :param int alpha: Fixed precision for the likelihood function.
        :param float std: Amount of noise to use for model initialization.
        :param (tuple of int) bounds: (lower, upper) bound of ratings.
            These bounds will simply be used to cap the estimates produced for R.

        """
        self.dim = dim
        self.alpha = alpha
        self.std = np.sqrt(1.0 / alpha)
        self.bounds = bounds
        self.data = train.copy()
        n, m = self.data.shape

        # Perform mean value imputation
        nan_mask = np.isnan(self.data)
        self.data[nan_mask] = self.data[~nan_mask].mean()

        # Low precision reflects uncertainty; prevents overfitting.
        # Set to the mean variance across users and items.
        self.alpha_u = 1 / self.data.var(axis=1).mean()
        self.alpha_v = 1 / self.data.var(axis=0).mean()

        # Specify the model.
        logging.info('building the PMF model')
        with pm.Model() as pmf:
            U = pm.MvNormal(
                'U', mu=0, tau=self.alpha_u * np.eye(dim),
                shape=(n, dim), testval=np.random.randn(n, dim) * std)
            V = pm.MvNormal(
                'V', mu=0, tau=self.alpha_v * np.eye(dim),
                shape=(m, dim), testval=np.random.randn(m, dim) * std)
            R = pm.Normal(
                'R', mu=theano.tensor.dot(U, V.T), tau=self.alpha * np.ones((n, m)),
                observed=self.data)

        logging.info('done building the PMF model')
        self.model = pmf

    def __str__(self):
        return self.name

```

We'll also need functions for calculating the MAP and performing sampling on our PMF model. When the observation noise variance $\alpha$ and the prior variances $\alpha_U$ and $\alpha_V$ are all kept fixed, maximizing the log posterior is equivalent to minimizing the sum-of-squared-errors objective function with quadratic regularization terms.

$$ E = \frac{1}{2} \sum_{i=1}^N \sum_{j=1}^M I_{ij} (R_{ij} - U_i V_j^T)^2 + \frac{\lambda_U}{2} \sum_{i=1}^N \|U\|_{Fro}^2 + \frac{\lambda_V}{2} \sum_{j=1}^M \|V\|_{Fro}^2, $$

where $\lambda_U = \alpha_U / \alpha$, $\lambda_V = \alpha_V / \alpha$, and $\|\cdot\|_{Fro}^2$ denotes the Frobenius norm [3]. Minimizing this objective function gives a local minimum, which is essentially a maximum a posteriori (MAP) estimate. While it is possible to use a fast Stochastic Gradient Descent procedure to find this MAP, we'll be finding it using the utilities built into `pymc3`. In particular, we'll use `find_MAP` with Powell optimization (`scipy.optimize.fmin_powell`). Having found this MAP estimate, we can use it as our starting point for MCMC sampling.

Since it is a reasonably complex model, we expect the MAP estimation to take some time. So let's save it after we've found it. Note that we define a function for finding the MAP below, assuming it will receive a namespace with some variables in it. Then we attach that function to the PMF class, where it will have such a namespace after initialization. The PMF class is defined in pieces this way so I can say a few things between each piece to make it clearer.

```python
try:
    import ujson as json
except ImportError:
    import json

# First define functions to save our MAP estimate after it is found.
# We adapt these from `pymc3`'s `backends` module, where the original
# code is used to save the traces from MCMC samples.
def save_np_vars(vars, savedir):
    """Save a dictionary of numpy variables to `savedir`. We assume
    the directory does not exist; an OSError will be raised if it does.
    """
    logging.info('writing numpy vars to directory: %s' % savedir)
    if not os.path.isdir(savedir):
        os.mkdir(savedir)
    shapes = {}
    for varname in vars:
        data = vars[varname]
        var_file = os.path.join(savedir, varname + '.txt')
        np.savetxt(var_file, data.reshape(-1, data.size))
        shapes[varname] = data.shape

        ## Store shape information for reloading.
        shape_file = os.path.join(savedir, 'shapes.json')
        with open(shape_file, 'w') as sfh:
            json.dump(shapes, sfh)

def load_np_vars(savedir):
    """Load numpy variables saved with `save_np_vars`."""
    shape_file = os.path.join(savedir, 'shapes.json')
    with open(shape_file, 'r') as sfh:
        shapes = json.load(sfh)

    vars = {}
    for varname, shape in shapes.items():
        var_file = os.path.join(savedir, varname + '.txt')
        vars[varname] = np.loadtxt(var_file).reshape(shape)

    return vars

# Now define the MAP estimation infrastructure.
def _map_dir(self):
    basename = 'pmf-map-d%d' % self.dim
    return os.path.join(DATA_DIR, basename)

def _find_map(self):
    """Find mode of posterior using Powell optimization."""
    tstart = time.time()
    with self.model:
        logging.info('finding PMF MAP using Powell optimization...')
        self._map = pm.find_MAP(fmin=sp.optimize.fmin_powell, disp=True)

    elapsed = int(time.time() - tstart)
    logging.info('found PMF MAP in %d seconds' % elapsed)

    # This is going to take a good deal of time to find, so let's save it.
    save_np_vars(self._map, self.map_dir)

def _load_map(self):
    self._map = load_np_vars(self.map_dir)

def _map(self):
    try:
        return self._map
    except:
        if os.path.isdir(self.map_dir):
            self.load_map()
        else:
            self.find_map()
        return self._map

# Update our class with the new MAP infrastructure.
PMF.find_map = _find_map
PMF.load_map = _load_map
PMF.map_dir = property(_map_dir)
PMF.map = property(_map)
print("ready")
```

    ready

So now our PMF class has a `map` `property` which will either be found using Powell optimization or loaded from a previous optimization. Once we have the MAP, we can use it as a starting point for our MCMC sampler. We'll need a sampling function in order to draw MCMC samples to approximate the posterior distribution of the PMF model.

```python
# Draw MCMC samples.
def _trace_dir(self):
    basename = 'pmf-mcmc-d%d' % self.dim
    return os.path.join(DATA_DIR, basename)

def _draw_samples(self, nsamples=1000, njobs=2):
    # First make sure the trace_dir does not already exist.
    if os.path.isdir(self.trace_dir):
        shutil.rmtree(self.trace_dir)

    with self.model:
        logging.info('drawing %d samples using %d jobs' % (nsamples, njobs))
        backend = pm.backends.Text(self.trace_dir)
        logging.info('backing up trace to directory: %s' % self.trace_dir)
        self.trace = pm.sample(draws=nsamples, init='advi',
                               n_init=150000, njobs=njobs, trace=backend)

def _load_trace(self):
    with self.model:
        self.trace = pm.backends.text.load(self.trace_dir)

# Update our class with the sampling infrastructure.
PMF.trace_dir = property(_trace_dir)
PMF.draw_samples = _draw_samples
PMF.load_trace = _load_trace
print("ready")
```

    ready

We could define some kind of default trace property like we did for the MAP, but that would mean using possibly nonsensical values for `nsamples` and `njobs`. Better to leave it as a non-optional call to `draw_samples`. Finally, we'll need a function to make predictions using our inferred values for $U$ and $V$. For user $i$ and joke $j$, a prediction is generated by drawing from $\mathcal{N}(U_i V_j^T, \alpha)$. To generate predictions from the sampler, we generate an $R$ matrix for each $U$ and $V$ sampled, then we combine these by averaging over the $K$ samples.

\begin{equation}
P(R_{ij}^* \given R, \alpha, \alpha_U, \alpha_V) \approx
    \frac{1}{K} \sum_{k=1}^K \mathcal{N}(U_i V_j^T, \alpha)
\end{equation}

We'll want to inspect the individual $R$ matrices before averaging them for diagnostic purposes. So we'll write code for the averaging piece during evaluation. The function below simply draws an $R$ matrix given a $U$ and $V$ and the fixed $\alpha$ stored in the PMF object.

```python
def _predict(self, U, V):
    """Estimate R from the given values of U and V."""
    R = np.dot(U, V.T)
    n, m = R.shape
    sample_R = np.array([
        [np.random.normal(R[i,j], self.std) for j in range(m)]
        for i in range(n)
    ])

    # bound ratings
    low, high = self.bounds
    sample_R[sample_R < low] = low
    sample_R[sample_R > high] = high
    return sample_R

PMF.predict = _predict
```

One final thing to note: the dot products in this model are often constrained using a logistic function $g(x) = 1/(1 + exp(-x))$, that bounds the predictions to the range [0, 1]. To facilitate this bounding, the ratings are also mapped to the range [0, 1] using $t(x) = (x + min) / range$. The authors of PMF also introduced a constrained version which performs better on users with less ratings [3]. Both models are generally improvements upon the basic model presented here. However, in the interest of time and space, these will not be implemented here.

## Evaluation

### Metrics

In order to understand how effective our models are, we'll need to be able to evaluate them. We'll be evaluating in terms of root mean squared error (RMSE), which looks like this:

\begin{equation}
RMSE = \sqrt{ \frac{ \sum_{i=1}^N \sum_{j=1}^M I_{ij} (R_{ij} - R_{ij}^*)^2 }
                   { \sum_{i=1}^N \sum_{j=1}^M I_{ij} } }
\end{equation}

In this case, the RMSE can be thought of as the standard deviation of our predictions from the actual user preferences.

```python
# Define our evaluation function.
def rmse(test_data, predicted):
    """Calculate root mean squared error.
    Ignoring missing values in the test data.
    """
    I = ~np.isnan(test_data)   # indicator for missing values
    N = I.sum()                # number of non-missing values
    sqerror = abs(test_data - predicted) ** 2  # squared error array
    mse = sqerror[I].sum() / N                 # mean squared error
    return np.sqrt(mse)                        # RMSE
```

### Training Data vs. Test Data

The next thing we need to do is split our data into a training set and a test set. Matrix factorization techniques use [transductive learning](http://en.wikipedia.org/wiki/Transduction_%28machine_learning%29) rather than inductive learning. So we produce a test set by taking a random sample of the cells in the full $N \times M$ data matrix. The values selected as test samples are replaced with `nan` values in a copy of the original data matrix to produce the training set. Since we'll be producing random splits, let's also write out the train/test sets generated. This will allow us to replicate our results. We'd like to be able to idenfity which split is which, so we'll take a hash of the indices selected for testing and use that to save the data.

```python
import hashlib

# Define a function for splitting train/test data.
def split_train_test(data, percent_test=10):
    """Split the data into train/test sets.
    :param int percent_test: Percentage of data to use for testing. Default 10.
    """
    n, m = data.shape             # # users, # jokes
    N = n * m                     # # cells in matrix
    test_size = N / percent_test  # use 10% of data as test set
    train_size = N - test_size    # and remainder for training

    # Prepare train/test ndarrays.
    train = data.copy().values
    test = np.ones(data.shape) * np.nan

    # Draw random sample of training data to use for testing.
    tosample = np.where(~np.isnan(train))       # ignore nan values in data

    #idx_pairs = zip(tosample[0], tosample[1])   # tuples of row/col index pairs
    idx_pairs = list(zip(tosample[0], tosample[1]))   # tuples of row/col index pairs

    indices = np.arange(len(idx_pairs))         # indices of index pairs
    sample = np.random.choice(indices, replace=False, size=int(test_size))

    # Transfer random sample from train set to test set.
    for idx in sample:
        idx_pair = idx_pairs[idx]
        test[idx_pair] = train[idx_pair]  # transfer to test set
        train[idx_pair] = np.nan          # remove from train set

    # Verify everything worked properly
    assert(np.isnan(train).sum() == test_size)
    assert(np.isnan(test).sum() == train_size)

    # Finally, hash the indices and save the train/test sets.
    index_string = ''.join(map(str, np.sort(sample)))
    name = hashlib.sha1(index_string).hexdigest()
    savedir = os.path.join(DATA_DIR, name)
    save_np_vars({'train': train, 'test': test}, savedir)

    # Return train set, test set, and unique hash of indices.
    return train, test, name

def load_train_test(name):
    """Load the train/test sets."""
    savedir = os.path.join(DATA_DIR, name)
    vars = load_np_vars(savedir)
    return vars['train'], vars['test']

train, test, name = split_train_test(data)
print("done")
```

In order to facilitate reproducibility, I've produced a train/test split using the code above which we'll now use for all the evaluations below.

```python
train, test = load_train_test('2f7be834c19f630bc89d550e41485a5a54d28af7')
```

## Results

```python
# Let's see the results:
baselines = {}
for name in baseline_methods:
    Method = baseline_methods[name]
    method = Method(train)
    baselines[name] = method.rmse(test)
    print('%s RMSE:\t%.5f' % (method, baselines[name]))

```

As expected: the uniform random baseline is the worst by far, the global mean baseline is next best, and the mean of means method is our best baseline. Now let's see how PMF stacks up.

```python
# We use a fixed precision for the likelihood.
# This reflects uncertainty in the dot product.
# We choose 2 in the footsteps Salakhutdinov
# Mnihof.
ALPHA = 2

# The dimensionality D; the number of latent factors.
# We can adjust this higher to try to capture more subtle
# characteristics of each joke. However, the higher it is,
# the more expensive our inference procedures will be.
# Specifically, we have D(N + M) latent variables. For our
# Jester dataset, this means we have D(1100), so for 5
# dimensions, we are sampling 5500 latent variables.
DIM = 5

pmf = PMF(train, DIM, ALPHA, std=0.05)
```

### Predictions Using MAP

```python
# Find MAP for PMF.
pmf.find_map()
```

Excellent. The first thing we want to do is make sure the MAP estimate we obtained is reasonable. We can do this by computing RMSE on the predicted ratings obtained from the MAP values of $U$ and $V$. First we define a function for generating the predicted ratings $R$ from $U$ and $V$. We ensure the actual rating bounds are enforced by setting all values below -10 to -10 and all values above 10 to 10. Finally, we compute RMSE for both the training set and the test set. We expect the test RMSE to be higher. The difference between the two gives some idea of how much we have overfit. Some difference is always expected, but a very low RMSE on the training set with a high RMSE on the test set is a definite sign of overfitting.

```python
def eval_map(pmf_model, train, test):
    U = pmf_model.map['U']
    V = pmf_model.map['V']

    # Make predictions and calculate RMSE on train & test sets.
    predictions = pmf_model.predict(U, V)
    train_rmse = rmse(train, predictions)
    test_rmse = rmse(test, predictions)
    overfit = test_rmse - train_rmse

    # Print report.
    print('PMF MAP training RMSE: %.5f' % train_rmse)
    print('PMF MAP testing RMSE:  %.5f' % test_rmse)
    print('Train/test difference: %.5f' % overfit)

    return test_rmse

# Add eval function to PMF class.
PMF.eval_map = eval_map
```

```python
# Evaluate PMF MAP estimates.
pmf_map_rmse = pmf.eval_map(train, test)
pmf_improvement = baselines['mom'] - pmf_map_rmse
print('PMF MAP Improvement:   %.5f' % pmf_improvement)
```

So we see a pretty nice improvement here when compared to our best baseline, which was the mean of means method. We also have a fairly small difference in the RMSE values between the train and the test sets. This indicates that the point estimates for $\alpha_U$ and $\alpha_V$ that we calculated from our data are doing a good job of controlling model complexity. Now let's see if we can improve our estimates by approximating our posterior distribution with MCMC sampling. We'll draw 1000 samples and back them up using the `pymc3.backend.Text` backend.

### Predictions using MCMC

```python
# Draw MCMC samples.
pmf.draw_samples(300)

# uncomment to load previous trace rather than drawing new samples.
# pmf.load_trace()
```

### Diagnostics and Posterior Predictive Check

The next step is to check how many samples we should discard as burn-in. Normally, we'd do this using a traceplot to get some idea of where the sampled variables start to converge. In this case, we have high-dimensional samples, so we need to find a way to approximate them. One way was proposed by [Salakhutdinov and Mnih, p.886](https://www.cs.toronto.edu/~amnih/papers/bpmf.pdf). We can calculate the Frobenius norms of $U$ and $V$ at each step and monitor those for convergence. This essentially gives us some idea when the average magnitude of the latent variables is stabilizing. The equations for the Frobenius norms of $U$ and $V$ are shown below. We will use `numpy`'s `linalg` package to calculate these.

$$ \|U\|_{Fro}^2 = \sqrt{\sum_{i=1}^N \sum_{d=1}^D |U_{id}|^2}, \hspace{40pt} \|V\|_{Fro}^2 = \sqrt{\sum_{j=1}^M \sum_{d=1}^D |V_{jd}|^2} $$

```python
def _norms(pmf_model, monitor=('U', 'V'), ord='fro'):
    """Return norms of latent variables at each step in the
    sample trace. These can be used to monitor convergence
    of the sampler.
    """
    monitor = ('U', 'V')
    norms = {var: [] for var in monitor}
    for sample in pmf_model.trace:
        for var in monitor:
            norms[var].append(np.linalg.norm(sample[var], ord))
    return norms

def _traceplot(pmf_model):
    """Plot Frobenius norms of U and V as a function of sample #."""
    trace_norms = pmf_model.norms()
    u_series = pd.Series(trace_norms['U'])
    v_series = pd.Series(trace_norms['V'])
    fig, (ax1, ax2) = plt.subplots(1, 2, figsize=(15, 7))
    u_series.plot(kind='line', ax=ax1, grid=False,
                  title="$\|U\|_{Fro}^2$ at Each Sample")
    v_series.plot(kind='line', ax=ax2, grid=False,
                  title="$\|V\|_{Fro}^2$ at Each Sample")
    ax1.set_xlabel("Sample Number")
    ax2.set_xlabel("Sample Number")

PMF.norms = _norms
PMF.traceplot = _traceplot
```

```python
pmf.traceplot()
```

It appears we get convergence of $U$ and $V$ after about 200 samples. When testing for convergence, we also want to see convergence of the particular statistics we are looking for, since different characteristics of the posterior may converge at different rates. Let's also do a traceplot of the RSME. We'll compute RMSE for both the train and the test set, even though the convergence is indicated by RMSE on the training set alone. In addition, let's compute a running RMSE on the train/test sets to see how aggregate performance improves or decreases as we continue to sample.

```python
def _running_rmse(pmf_model, test_data, train_data, burn_in=0, plot=True):
    """Calculate RMSE for each step of the trace to monitor convergence.
    """
    burn_in = burn_in if len(pmf_model.trace) >= burn_in else 0
    results = {'per-step-train': [], 'running-train': [],
               'per-step-test': [], 'running-test': []}
    R = np.zeros(test_data.shape)
    for cnt, sample in enumerate(pmf_model.trace[burn_in:]):
        sample_R = pmf_model.predict(sample['U'], sample['V'])
        R += sample_R
        running_R = R / (cnt + 1)
        results['per-step-train'].append(rmse(train_data, sample_R))
        results['running-train'].append(rmse(train_data, running_R))
        results['per-step-test'].append(rmse(test_data, sample_R))
        results['running-test'].append(rmse(test_data, running_R))

    results = pd.DataFrame(results)

    if plot:
        results.plot(
            kind='line', grid=False, figsize=(15, 7),
            title='Per-step and Running RMSE From Posterior Predictive')

    # Return the final predictions, and the RMSE calculations
    return running_R, results

PMF.running_rmse = _running_rmse
```

```python
predicted, results = pmf.running_rmse(test, train, burn_in=200)
```

```python
# And our final RMSE?
final_test_rmse = results['running-test'].values[-1]
final_train_rmse = results['running-train'].values[-1]
print('Posterior predictive train RMSE: %.5f' % final_train_rmse)
print('Posterior predictive test RMSE:  %.5f' % final_test_rmse)
print('Train/test difference:           %.5f' % (final_test_rmse - final_train_rmse))
print('Improvement from MAP:            %.5f' % (pmf_map_rmse - final_test_rmse))
print('Improvement from Mean of Means:  %.5f' % (baselines['mom'] - final_test_rmse))
```

We have some interesting results here. As expected, our MCMC sampler provides lower error on the training set. However, it seems it does so at the cost of overfitting the data. This results in a decrease in test RMSE as compared to the MAP, even though it is still much better than our best baseline. So why might this be the case? Recall that we used point estimates for our precision paremeters $\alpha_U$ and $\alpha_V$ and we chose a fixed precision $\alpha$. It is quite likely that by doing this, we constrained our posterior in a way that biased it towards the training data. In reality, the variance in the user ratings and the joke ratings is unlikely to be equal to the means of sample variances we used. Also, the most reasonable observation precision $\alpha$ is likely different as well.

### Summary of Results

Let's summarize our results.

```python
size = 100  # RMSE doesn't really change after 100th sample anyway.
all_results = pd.DataFrame({
    'uniform random': np.repeat(baselines['ur'], size),
    'global means': np.repeat(baselines['gm'], size),
    'mean of means': np.repeat(baselines['mom'], size),
    'PMF MAP': np.repeat(pmf_map_rmse, size),
    'PMF MCMC': results['running-test'][:size],
})
fig, ax = plt.subplots(figsize=(10, 5))
all_results.plot(kind='line', grid=False, ax=ax,
                 title='RMSE for all methods')
ax.set_xlabel("Number of Samples")
ax.set_ylabel("RMSE")
```

## Summary

We set out to predict user preferences for unseen jokes. First we discussed the intuitive notion behind the user-user and item-item neighborhood approaches to collaborative filtering. Then we formalized our intuitions. With a firm understanding of our problem context, we moved on to exploring our subset of the Jester data. After discovering some general patterns, we defined three baseline methods: uniform random, global mean, and mean of means. With the goal of besting our baseline methods, we implemented the basic version of Probabilistic Matrix Factorization (PMF) using `pymc3`.

Our results demonstrate that the mean of means method is our best baseline on our prediction task. As expected, we are able to obtain a significant decrease in RMSE using the PMF MAP estimate obtained via Powell optimization. We illustrated one way to monitor convergence of an MCMC sampler with a high-dimensionality sampling space using the Frobenius norms of the sampled variables. The traceplots using this method seem to indicate that our sampler converged to the posterior. Results using this posterior showed that attempting to improve the MAP estimation using MCMC sampling actually overfit the training data and increased test RMSE. This was likely caused by the constraining of the posterior via fixed precision parameters $\alpha$, $\alpha_U$, and $\alpha_V$.

As a followup to this analysis, it would be interesting to also implement the logistic and constrained versions of PMF. We expect both models to outperform the basic PMF model. We could also implement the [fully Bayesian version of PMF](https://www.cs.toronto.edu/~amnih/papers/bpmf.pdf) (BPMF), which places hyperpriors on the model parameters to automatically learn ideal mean and precision parameters for $U$ and $V$. This would likely resolve the issue we faced in this analysis. We would expect BPMF to improve upon the MAP estimation produced here by learning more suitable hyperparameters and parameters. For a basic (but working!) implementation of BPMF in `pymc3`, see [this gist](https://gist.github.com/macks22/00a17b1d374dfc267a9a).

If you made it this far, then congratulations! You now have some idea of how to build a basic recommender system. These same ideas and methods can be used on many different recommendation tasks. Items can be movies, products, advertisements, courses, or even other people. Any time you can build yourself a user-item matrix with user preferences in the cells, you can use these types of collaborative filtering algorithms to predict the missing values. If you want to learn more about recommender systems, the first reference is a good place to start.

## References

1.  Y. Koren, R. Bell, and C. Volinsky, “Matrix Factorization Techniques for Recommender Systems,” Computer, vol. 42, no. 8, pp. 30–37, Aug. 2009.
2.  K. Goldberg, T. Roeder, D. Gupta, and C. Perkins, “Eigentaste: A constant time collaborative filtering algorithm,” Information Retrieval, vol. 4, no. 2, pp. 133–151, 2001.
3.  A. Mnih and R. Salakhutdinov, “Probabilistic matrix factorization,” in Advances in neural information processing systems, 2007, pp. 1257–1264.
4.  S. J. Nowlan and G. E. Hinton, “Simplifying Neural Networks by Soft Weight-sharing,” Neural Comput., vol. 4, no. 4, pp. 473–493, Jul. 1992.
5.  R. Salakhutdinov and A. Mnih, “Bayesian Probabilistic Matrix Factorization Using Markov Chain Monte Carlo,” in Proceedings of the 25th International Conference on Machine Learning, New York, NY, USA, 2008, pp. 880–887.

```python

```


# Deep Learning

**CS5483 Data Warehousing and Data Mining**
___

```python
%reset -f
%load_ext tensorboard
%matplotlib inline
import jupyter_manim
from manimlib.imports import *
import pprint as pp
import tensorflow_datasets as tfds
import tensorflow.compat.v2 as tf
import tensorflow_addons as tfa
import os, datetime, pytz
import tensorboard as tb
from matplotlib import pyplot as plt
import numpy as np
from IPython import display
import matplotlib.pyplot as plt
# produce vector inline graphics
from IPython.display import set_matplotlib_formats
set_matplotlib_formats('svg')
```

## Introduction

In this notebook, we will train the following classifier using deep learning:
1. Handwrite a digit from 0, ..., 9.
1. Click predict to see if the app can recognize the digit.

```python
display.IFrame(src="https://www.cs.cityu.edu.hk/~ccha23/mnist/", width=805, height=450)
```

**What is deep learning?**

[Deep learning](https://en.wikipedia.org/wiki/Deep_learning) is a technique of training a neural network with many layers of computational units called neurons. The following videos showcase some interesting applications of the technique.

```python
display.IFrame(src="https://slides.com/ccha23/dl_intro/embed", width=805, height=450)
```

**How to train a neural network?**

We can visualize the training process in the following application, which trains a neural network that predicts the color of a given point when given its coordinate $(x_1,x_2)$.
- Choose a data set from the `DATA` column.
- Click the `play` button to start training the network.
- `Epoch` is the number of times a neural network is trained iteratively using the data selected.
- In the `OUTPUT` column,
  - points in the blue region are predicted blue, and
  - points in the orange region are predicted orange.

```python
display.IFrame(src="https://www.cs.cityu.edu.hk/~ccha23/playground", width=900, height=800)
```

The above app is a slight modification of the open source project [Tensorflow Playground](https://playground.tensorflow.org) with the additional features that:
- You can save your configuration to the browser session by clicking the button `Save to browser session`. If you reopen the browser, it will load your previously saved configuration automatically.
- You can reset the configuration by clicking the `reset` button.
- The last button copies the permalink to your configuration to the clipboard. You can save multiple configurations and share them by keeping the permalinks.

For instance, the following uses the permalink to initialize the simplest neural network for the linearly separable data:

```python
display.IFrame(src="https://www.cs.cityu.edu.hk/~ccha23/playground/#activation=linear&batchSize=10&dataset=gauss&regDataset=reg-plane&learningRate=0.03&regularizationRate=0&noise=0&networkShape=&seed=0.15891&showTestData=false&discretize=false&percTrainData=50&x=true&y=true&xTimesY=false&xSquared=false&ySquared=false&cosX=false&sinX=false&cosY=false&sinY=false&collectStats=false&problem=classification&initZero=false&hideText=false", width=900, height=800)
```

**Exercise** Try to tune the parameters to classify the Spiral dataset until the testing loss reaches 0.02. Include a screenshot below and give the permalink to your configuration.

YOUR ANSWER HERE

You can also [fork and modify the code on GitHub](https://github.com/tensorflow/tensorflow/pulls) to add new features, e.g., to customize the datasets and store the trained neural network. However, since the visualization is limited 2D, it is difficult to extend the app for higher-dimensional dataset with multiple class values.

Nevertheless, it is possible to train a practical neural network without any coding, by using a service called the [Teachable Machine](https://teachablemachine.withgoogle.com/). E.g., you may follow the interactive slides below to learn to train a machine that recognizes musical notes.
(Click the `play` button at the bottom of the slides to start the presentation.)

```python
%%html

```

**Exercise** Use [Teachable Machine](https://teachablemachine.withgoogle.com/) to train your machine. Explain what your machine does and include a link to it.

YOUR ANSWER HERE

## Preliminaries

The neural network is trained using an iterative algorithm called the *stochastic gradient descent*, which requires some background on vector calculus and probability theory. The lecture series below explain the idea nicely with mathematical animations made using the python library [mathematical animation engine (`manim`)](https://github.com/3b1b/manim).

```python
display.IFrame(src="https://www.youtube.com/embed/videoseries?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi", width=805, height=450)
```

Instead of using the mean-squared error for the loss function, we will consider the [cross-entropy](https://en.wikipedia.org/wiki/Cross_entropy) loss, which is more suitable for training a neural network classifier. We will have a glimpse of the information theory involved.
$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}
$

**What to know about vector calculus?**

To help explain the theory, we will use some notations from vector linear algebra and probability theory.

- [Vectors](https://en.wikipedia.org/wiki/Vector_(mathematics_and_physics)) in lowercase boldface font:

$$\M{x}=\begin{bmatrix}x_1 \\ x_2 \\ \vdots \end{bmatrix}$$

- [Matrices](https://en.wikipedia.org/wiki/Matrix_(mathematics)) in uppercase boldface font:

$$\M{W}=\begin{bmatrix}w_{11} & w_{12} & \cdots \\
w_{21} & \ddots &  \\
\vdots &  &  \end{bmatrix}$$

- [Matrix multiplication](https://en.wikipedia.org/wiki/Matrix_multiplication):

$$\M{W}\M{x} = \begin{bmatrix}w_{11} & w_{12} & \cdots \\
w_{21} & \ddots &  \\
\vdots &  &  \end{bmatrix}
\begin{bmatrix}x_1 \\ x_2 \\ \vdots \end{bmatrix}
=
\begin{bmatrix}w_{11}x_1 + w_{12}x_2 + \cdots \\ w_{21}x_1+\cdots \\ \vdots \end{bmatrix}
$$

**What to know about Probability Theory?**

- [Random variables](https://en.wikipedia.org/wiki/Random_variable) in sanserif font:

  $$\underbrace{\R{y}}_{\text{random variable}}, \underbrace{\RM{x}=\begin{bmatrix}\R{x}_1 \\  \R{x}_2 \\  \vdots \end{bmatrix}}_{\text{random vector}}$$

- [Support sets](https://en.wikipedia.org/wiki/Support_(mathematics)) in calligraphic font:

$$\underbrace{\mc{Y}=\Set{0,1,\dots,k-1}}_{\text{finite set}}, \underbrace{\mc{X}=\mc{X}_1\times \mc{X}_2 \times \cdots}_{\text{product space}}$$

- [Joint distribution](https://en.wikipedia.org/wiki/Joint_probability_distribution#Mixed_case):

$$p_{\RM{x}\R{y}}(\M{x},y)= \underbrace{p_{\R{y}|\RM{x}}(y|\M{x})}_{\underbrace{\Pr}_{\text{probability measure}\kern-3em}\Set{\R{y}=y|\RM{x}=\M{x}}} \cdot \underbrace{p_{\RM{x}}(\M{x})}_{(\underbrace{\partial_{x_1}}_{\text{partial derivative w.r.t. $x_1$}\kern-5em} \partial_{x_2}\cdots)\Pr\Set{\RM{x} \leq \M{x}}\kern-4em}\kern1em \text{where}$$
  - $p_{\R{y}|\RM{x}}(y|\M{x})$ is the *probability mass function [(pmf)](https://en.wikipedia.org/wiki/Probability_mass_function)* of $\R{y}=y\in \mc{Y}$ [conditioned](https://en.wikipedia.org/wiki/Conditional_probability_distribution) on $\RM{x}=\M{x}\in \mc{X}$, and
  - $p_{\RM{x}}(\M{x})$ is the *(multivariate) probability density function [(pdf)](https://en.wikipedia.org/wiki/Probability_density_function#Densities_associated_with_multiple_variables)* of $\RM{x}=\M{x}\in \mc{X}$.

- For any function $g$ of $(\RM{x},y)$, the expectations are:

\begin{align}
\E[g(\RM{x},\R{y})|\RM{x}]&=\sum_{y\in \mc{Y}} g(\RM{x},y)\cdot p_{\R{y}|\RM{x}}(y|\RM{x})\tag{conditional expectation}
\\
\E[g(\RM{x},\R{y})] &=\int_{\mc{X}} \underbrace{\sum_{y\in \mc{Y}} g(\RM{x},y)\cdot \underbrace{p_{\RM{x},\R{y}}(\M{x},y)}_{p_{\R{y}|\RM{x}}(y|\M{x}) p_{\R{x}}(\M{x})}\kern-1.7em}_{\E[g(\RM{x},\R{y})|\RM{x}]}\kern1.4em\,\d \M{x} \tag{expectation}\\
&= \E[\E[g(\RM{x},\R{y})|\RM{x}]] \tag{iterated expectation}
\end{align}

The followings are some manims created to introduce the above notions.

```python
%%html
<b>Notation:</b><br>

```

Notation:

```python
%%html
<b>Distribution:</b><br>

```

Distribution:

```python
%%html
<b>Expectation:<br>

```

Expectation:

You may also create your own animations with `manim` in the jupyter notebook using `jupyter_manim` and `manimlib` as described [here](https://ccha23.github.io/CS1302ICP/Lecture5/Objects.html).

```python
import jupyter_manim
from manimlib.imports import *
```

**Exercise** Run the following cell and see the effect when changing

- Mobjects: `TextMobject('Hello, World!')` to `TexMobject(r'E=mc^2')` or `Circle()` or `Square()`.
- Animation objects: `Write` to `FadeIn` or `GrowFromCenter`.

You may take a look at the documentation [here](https://docs.manim.community/en/v0.2.0/index.html) and a more detailed [tutorial](https://talkingphysics.wordpress.com/2019/01/08/getting-started-animating-with-manim-and-python-3-7/) here.

```python
%%manim HelloWorld -l
class HelloWorld(Scene):
    def construct(self):
        self.play(Write(TextMobject('Hello, World!')))
```

## Classification Problem

A neural network learns from many examples collected together as a *dataset*. For instance, the [MNIST (Modified National Institute of Standards and Technology)](https://en.wikipedia.org/wiki/MNIST_database) dataset consists of labeled handwritten digits.

**What is an example in a dataset?**

A dataset is a sequence

\begin{align}
(\RM{x}_1,\R{y}_1),(\RM{x}_2,\R{y}_2), \dots\tag{dataset}
\end{align}

of *tuples/instances* $(\RM{x}_i,\R{y}_i)$, each of which consists of
- an *input feature vector* $\RM{x}_i$ such as an image of a handwritten digit and
- a *label* $\R{y}_i$ such as the digit type of the handwritten digit.

**What to learn from the dataset?**

One of the problem in Machine Learning is *classification*: The goal is to train a *classifier* that predicts a label $\R{y}$ for an input feature $\RM{x}$:
-  A hard-decision classifier is a function $f:\mc{X}\to \mc{Y}$ such that
$f(\RM{x})$ predicts $\R{y}$.
- A probabilistic classifier is a conditional distribution $q_{\R{y}|\RM{x}}$ that estimates $p_{\R{y}|\RM{x}}$.

For MNIST, the goal is to classify the digit type of a handwritten digit.  When given a handwritten digit,
- a hard-decision classifier returns a digit type, and
- a probabilistic classifier returns a distribution of the digit types.

**Why consider a probabilistic classifier?**

We often train a neural network as a probabilistic classifer because:
- A probabilistic classifer is more general and can give a hard decision as well

  $$f(\RM{x}):=\arg\max_{y\in \mc{Y}} q_{\R{y}|\RM{x}}(y|\RM{x})$$
  by returning the estimated most likely digit type.

- A neural network can model the distribution $p_{\R{y}|\RM{x}}(\cdot|\RM{x})$ better than $\R{y}$ because its output is continous.

**Why can we learn from examples?**

For the problem to be meaningful, $(\RM{x},\R{y})$ is assumed to be random with some unknown joint distribution $p_{\RM{x},\R{y}}$.

- If we always had $\R{y}=y$ instead, then a perfect classifier can just return $y$ without even looking at $\RM{x}$.
- If $p_{\RM{x},\R{y}}$ were known instead, then $p_{\R{y}|\RM{x}}$ would also be known and therefore needed not be estimated.

Examples are often collected randomly and independently from a population, i.e., as [i.i.d. samples](https://en.wikipedia.org/wiki/Independent_and_identically_distributed_random_variables) of $(\RM{x},\R{y})$:

$$(\RM{x}_1,\R{y}_1), (\RM{x}_2,\R{y}_2), \dots\sim_{\text{i.i.d.}} p_{\RM{x},\R{y}}.$$

- If all the examples were the same instead, the examples could not show the pattern of how $\R{y}$ depended on $\RM{x}$.
- The observed distribution of i.i.d. samples converge to the unknown distribution by the [law of large number](https://en.wikipedia.org/wiki/Law_of_large_numbers).

**How to determine if a classifier is good?**

Ultimately, we desire a classifier with the maximum accuracy in predicting $\R{y}$ but doing so is [computationally too difficult](https://en.wikipedia.org/wiki/Loss_functions_for_classification).

Instead, we regard a classification algorithm to be reasonably good if it can achieve the maximum possible accuracy as the number of training samples goes to $\infty$. This is more formally stated below:

**Definition** A probabilistic classifier for the input feature $\RM{x}$ and label $\R{y}$ with unknown joint distribution  is a conditional pmf

$$
\R{q}_{\R{y}|\RM{x}}(y|\RM{x})\qquad \text{for }\M{x}\in \mc{X}, y\in \mc{Y}
$$

defined as a function of the i.i.d. samples

$$\{(\RM{x}_i,\R{y}_i)\}_{i=1}^N$$

of $(\RM{x},\R{y})$ (but independent of $(\RM{x},\R{y})$).
The classifier is said to be a *consistent* estimate (of $p_{\R{y}|\M{x}}$) if

\begin{align}
\lim_{N\to \infty} \Pr\Set{\R{q}_{\R{y}|\RM{x}}(y|\RM{x})=p_{\R{y}|\RM{x}}(y|\RM{x})\text{ for all } y\in \mc{Y}}=1,\tag{consistency}
\end{align}

namely, $\R{q}_{\R{y}|\RM{x}}(y|\RM{x})$ converges almost surely (a.s.) to $p_{\R{y}|\RM{x}}$. $\square$

A consistent probabilistic classifier gives rise to an asymptotically optimal hard-decision classifier that achieves the maximum accuracy.

**Proposition** If for some $\epsilon\geq 0$ that

$$\Pr\Set{\R{q}_{\R{y}|\RM{x}}(y|\RM{x})=p_{\R{y}|\RM{x}}(y|\RM{x}) \text{ for all } y\in \mc{Y}}=1-\epsilon,$$

the (hard-decision) classifier

\begin{align}\R{f}(\RM{x}):=\arg\max_{y\in \mc{Y}} \R{q}_{\R{y}|\RM{x}}(y|\RM{x})\tag{hard decision}\end{align}

achieves an accuracy

\begin{align}
\sup_{f:\mc{X}\to \mc{Y}} \Pr(\R{y}= f(\RM{x})) &\geq \E\left[\max_{y\in \mc{Y}} p_{\R{y}|\M{x}}(y|\RM{x})\right] - \epsilon.\tag{accuracy lower bound}
\end{align}

where the expectation is the maximum possible accuracy. $\square$

*Proof:* For any classifier $f$,

$$ \begin{align*}
\Pr(\R{y}= f(\RM{x}))
&= \E[p_{\R{y}|\M{x}}(f(\RM{x})|\RM{x})] \\
&\leq \E\left[\max_{y\in \mc{Y}} p_{\R{y}|\M{x}}(y|\RM{x})\right]\end{align*}
$$

where the last inequality is achievable with equality with the hard-decision classifier and $\R{q}$ replaced by $p$. This implies the desired accuracy lower bound for the case $\epsilon=0$. The more general case with $\epsilon\geq 0$ can be derived similarly. $\blacksquare$

**How can we obtain a consistent classifier?**

We train a neural network to minimize certain *loss*. A common loss function for classification uses the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) measure in information theory.

The theoretical underpinning is the following identity that relates three information quantities:

\begin{align}
\overbrace{\E\left[\log \frac{1}{q_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right]}^{ \text{Cross entropy}\\ H(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}}):=} &\equiv \overbrace{\E\left[\log \frac{1}{p_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right]}^{\text{Conditional entropy}\\ H(\R{y}|\RM{x}):=} + \overbrace{\E\left[\log \frac{p_{\R{y}|\RM{x}}(\R{y}|\RM{x})}{q_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right].}^{\text{Divergence}\\ D(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}}):=}\tag{identity}
\end{align}

The identity can be proved quite easily using the linearity of expectation

$$ \E[\R{u}+\R{v}]=\E[\R{u}]+\E[\R{v}],$$

and a property of logarithm that

$$\log uv = \log u+ \log v.$$

```python
%%html
<b>Information Identity</b><br>

```

Information Identity

**Proposition** With $q_{\R{y}|\RM{x}}(y|\M{x})$ being a valid pmf of a random variable taking values from $\mc{Y}$ conditioned on a random variable taking values from $\mc{X}$,

\begin{align}
\min_{q_{\R{y}|\RM{x}}} H(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}})
&= H(\R{y}|\RM{x})
\end{align}

and the optimal solution equals $p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. for all $y\in \mc{Y}$. $\square$

Hence, a neural network that minimizes the cross entropy equals $p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. for all $y\in \mc{Y}$ and any possible input image $\RM{x}$.

*Proof:* It suffices to show that

\begin{align}
D(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}})\geq 0 \tag{positivity of divergence}
\end{align}

with equality iff $q_{\R{y}|\RM{x}}(y|\RM{x})=p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. This, in turn, can be proved using the [log-sum inequality](https://en.wikipedia.org/wiki/Log_sum_inequality):

\begin{align}
\sum_{i} a_i \log\left(\frac{a_i}{b_i}\right) \geq (\textstyle \sum_{i} a_i) \log\left(\frac{\sum_{i} a_i}{\sum_{i} b_i}\right)\tag{log-sum inequality}
\end{align}

for any sequences $\{a_i\}$, $\{b_i\}$, and $\{c_i\}$. $\blacksquare$

## Data Preparation

### Load the dataset

Like the iris dataset, the MNIST dataset can be obtained in many ways due to its popularity in image recognition. For instance, one may use `tensorflow.keras.datasets.mnist.load_data` to load the data as tuples/arrays and convert it to `DataFrame`. However, training a neural network often requires a lot of data and computational power. It may be inefficient or impossible to load all data into memory.

A better way is to use the package [`tensorflow_datasets`](https://blog.tensorflow.org/2019/02/introducing-tensorflow-datasets.html), which lazily load the dataset and prepare the data as [`Tensor`s](https://www.tensorflow.org/guide/tensor), which can be operated faster by GPU or TPU instead of CPU.

```python
import tensorflow_datasets as tfds  # give a shorter name tfds for convenience
import os

user_home = os.getenv("HOME")  # get user home directory
data_dir = os.path.join(user_home, "data")  # download folder for data

ds, ds_info = tfds.load(
    'mnist',
    data_dir=data_dir,  # download location
    as_supervised=True,  # separate input features and label
    with_info=True,  # return information of the dataset
)

# print information related to loading the dataset
import pprint as pp
print('-' * 79)
print(f'Data downloaded to {data_dir}')
print(f'Data to be loaded in memory:')
pp.pprint(ds)
```

The function `tfds.load` download the data to `data_dir` and prepare it for loading using variable `ds`. In particular, the dataset is split into
- a training set `ds["train"]` and
- a test set `ds["test"]`.

`tfds.load?` shows more information about the function. E.g., we can control the split ratio using the argument [`split`](https://www.tensorflow.org/datasets/splits).

**Why split the data?**

The test set is used to evaluate the performance of a neural network trained using the training set (separate from the test set).

The purpose of separating the test set from the training set is to avoid *overly-optimistic* performance estimate. Why?

Suppose the final exam questions (test set) are the same as the previous homework questions (training set).
- Students may get a high exam score simply by studying the model answers to the homework instead of understanding entire subject.
- The exam score is therefore an overly-optimistic estimate of the students' understanding of the subject.

**Exercise** Assign to `train_size` and `test_size` the numbers of instances in the training set and test set respectively.

*Hint: Both the training and test sets are loaded as [`Dataset` objects](https://www.tensorflow.org/api_docs/python/tf/data/Dataset). Since the loading is lazy, i.e., the data is not yet in memory, we cannot count the number of instances directly. Instead, we obtain such information from `ds_info`.*

```python
# YOUR CODE HERE
raise NotImplementedError()
train_size, test_size
```

```python
# tests
```

Note that the training set is often much larger than the test set especially for deep learning because
- training a neural network requires many examples but
- estimating its performance does not.

### Show an example

The following retrieves an example from the training set.

```python
for image, label in ds["train"].take(1):
    print(
        f'image dtype: {type(image)} shape: {image.shape} element dtype: {image.dtype}'
    )
    print(f'label dtype: {label.dtype}')
```

The for loop above takes one example from `ds["train"]` using the method [`take`](https://www.tensorflow.org/api_docs/python/tf/data/Dataset#take) and print its data types.
- The handwritten digit is represented by a 28x28x1 [`EagerTensor`](https://www.tensorflow.org/guide/eager), which is essentially a 2D array of bytes (8-bit unsigned integers `uint8`).
- The digit type is an integer.

The following function plots the image using the `imshow` function from `matplotlib.pyplot`. We set the parameter `cmap` to `gray_r` so the color is darker if the pixel value is larger. The slice operator `[:,:,0]` for the image reshaped the numpy array from 3 dimensions to 2 dimensions.

```python
def plot_mnist_image(example, ax=None, pixel_format=None):
    (image, label) = example
    if ax == None:
        ax = plt.gca()
    ax.title.set_text(label.numpy())  # show digit type as plot title
    ax.imshow(image[:, :, 0], cmap="gray_r")  # show image
    # Major ticks
    ax.set_xticks(np.arange(0, 28, 3))
    ax.set_yticks(np.arange(0, 28, 3))
    # Minor ticks
    ax.set_xticks(np.arange(-.5, 28, 1), minor=True)
    ax.set_yticks(np.arange(-.5, 28, 1), minor=True)
    if pixel_format is not None:
        for i in range(28):
            for j in range(28):
                ax.text(
                    j,
                    i,
                    pixel_format.format(image[i, j,
                                              0].numpy()),  # show pixel value
                    va='center',
                    ha='center',
                    color='white',
                    fontweight='bold',
                    fontsize='small')
        ax.grid(color='lightblue', linestyle='-', linewidth=1, which='minor')
        ax.set_xlabel('2nd dimension')
        ax.set_ylabel('1st dimension')
        ax.title.set_text('Image with label ' + ax.title.get_text())

if input('Execute? [Y/n]').lower != 'n':
    plt.figure(figsize=(11, 11), dpi=80)
    for example in ds["train"].take(1):
        plot_mnist_image(example, pixel_format='{}')
    plt.show()
```

**Exercise** Complete the following code to generate a matrix plot of the first 50 examples from the training sets.
The parameter `nrows` and `ncols` specify the number of rows and columns respectively. You code may look like
```Python
...
        for ax, example in zip(axes.flat, ds["train"].____(nrows * ncols)):
            plot_mnist_image(_______, ax)
            ax.axes.xaxis.set_visible(False)
            ax.axes.yaxis.set_visible(False)
...
```
and the output image should look like

```python
if input('Execute? [Y/n]').lower != 'n':
    def plot_mnist_image_matrix(ds, nrows=5, ncols=10):
        fig, axes = plt.subplots(nrows=nrows, ncols=ncols)

        # YOUR CODE HERE
        raise NotImplementedError()

        fig.tight_layout()  # adjust spacing between subplots automatically
        return fig, axes

    fig, axes = plot_mnist_image_matrix(ds, nrows=5)
    fig.set_figwidth(9)
    fig.set_figheight(6)
    fig.set_dpi(80)
    # plt.savefig('mnist_examples.svg')
    plt.show()
```

### Preprocess the data

We will use the [`tensorflow`](https://www.tensorflow.org/) library to process the data and train the neural network. (Another popular library is [PyTorch](https://pytorch.org/).)

```python
import tensorflow.compat.v2 as tf  # explicitly use tensorflow version 2
```

Each pixel is stored as an integer from $\Set{0,\dots,255}$ ($2^8$ possible values). However, for computations by the neural network, we need to convert it to a floating point number. We will also normalize each pixel value to be within the unit interval $[0,1]$:

\begin{align}
v \mapsto \frac{v - v_{\min}}{v_{\max} - v_{\min}} = \frac{v}{255}\tag{min-max normalization}
\end{align}

**Exercise** Using the function `map`, normalize each element of an image to the unit interval $[0,1]$ after converting them to `tf.float32` using [`tf.cast`](https://www.tensorflow.org/api_docs/python/tf/cast).

*Hint:* The normalization factor is NOT 256. You code may look like
```Python
...
        ds_n[part] = ds[part].map(
                    lambda image, label: (_____(image, _____) / ___, label),
                    num_parallel_calls=tf.data.experimental.AUTOTUNE)
...
```
and the output image should look like

```python
def normalize_mnist(ds):
    """
  Returns:
  MNIST Dataset with image pixel values normalized to float32 in [0,1].
  """
    ds_n = dict.fromkeys(ds.keys())  # initialize the normalized dataset
    for part in ds.keys():
        # normalize pixel values to [0,1]
        # YOUR CODE HERE
        raise NotImplementedError()
    return ds_n

ds_n = normalize_mnist(ds)
pp.pprint(ds_n)
plt.figure(figsize=(11, 11), dpi=80)
for example in ds_n["train"].take(1):
    plot_mnist_image(example,
                     pixel_format='{:.2f}')  # show pixel value to 2 d.p.s
plt.savefig('mnist_example_normalized.svg')
plt.show()
```

```python
# tests
```

To avoid overfitting, the training of a neural network uses *stochastic gradient descent* which
- divides the training into many steps where
- each step uses a *randomly* selected minibatch of samples
- to improve the neural network *bit-by-bit*.

The following code specifies the batch size and enables caching and prefetching to reduce the latency in loading examples repeatedly for training and testing.

```python
def batch_mnist(ds_n):
    ds_b = dict.fromkeys(ds_n.keys())  # initialize the batched dataset
    for part in ds_n.keys():
        ds_b[part] = (
            ds_n[part].batch(
                128)  # Use a minibatch of examples for each training step
            .shuffle(
                ds_info.splits[part].num_examples,
                reshuffle_each_iteration=True)  # shuffle data for each epoch
            .cache()  # cache current elements
            .prefetch(tf.data.experimental.AUTOTUNE)
        )  # preload subsequent elements
    return ds_b

ds_b = batch_mnist(ds_n)
pp.pprint(ds_b)
```

**Exercise** The output to the above cell should look like
```Python
{'test': <PrefetchDataset shapes: ((None, 28, 28, 1), (None,)), types: (tf.float32, tf.int64)>,
 'train': <PrefetchDataset shapes: ((None, 28, 28, 1), (None,)), types: (tf.float32, tf.int64)>}
```
with a new first dimension of unknown size `None`. Why?

YOUR ANSWER HERE

## Training and Testing

### Define the neural network architecture

As mentioned earlier, the neural network computes an estimate $q_{\R{y}|\RM{x}}(y|\M{x})$ of the unknown probability $p_{\R{y}|\RM{x}}(y|\M{x})$ for any image $\M{x}\in \mc{X}$ and label $y\in \mc{Y}$. The computation is organized into layers of computation units called the *neurons*.

For $\ell\in \{0,\dots,L\}$ and integer $L\geq 1$, let
- $\M{a}^{(\ell)}$ be the output of the $\ell$-th layer of the neural network, and
- $a^{(\ell)}_i$ be the $i$-th element of $\M{a}^{(\ell)}$. The element is computed from the output $\M{a}^{(\ell-1)}$ of its previous layer except for $\ell=0$.

The $0$-th layer is called the *input layer* while the $L$-th layer is called the *output layer*. All other layers are called the *hidden layers*.

A basic neural network architecture computes $q_{\R{y}|\RM{x}}(y|\M{x})$ as

\begin{align}
[q_{\R{y}|\RM{x}}(y|\M{x})]_{y\in \mc{Y}} &:= \M{a}^{(L)}
\end{align}

where

\begin{align}
\M{a}^{(\ell)}&:=\begin{cases}
\M{x} & \ell=0\\
\sigma^{(\ell)}(\overbrace{\M{W}^{(\ell)}\M{a}^{(\ell-1)}+\M{b}^{(\ell)}}^{\RM{z}^{(\ell)}:=})& \ell>0;
\end{cases}\tag{net}
\end{align}

- $\M{W}^{(\ell)}$ is a matrix of weights;
- $\M{b}^{(\ell)}$ is a vector called bias; and
- $\sigma^{(\ell)}$ is a reveal-valued function called the *activation function*.

To ensure $\M{a}^{(L)}$ is a valid probability vector, the soft-max activation function is often used for the last layer:

$$ \sigma^{(L)}\left(\left[\begin{smallmatrix}z^{(\ell)}_1 \\ \vdots \\ z^{(\ell)}_k\end{smallmatrix}\right]\right) := \frac{1}{\sum_{i=1}^k e^{z^{(\ell)}_i}} \left[\begin{smallmatrix}e^{z^{(\ell)}_1} \\ \vdots \\ e^{z^{(\ell)}_k}\end{smallmatrix}\right] $$

where $k:=\abs{\mc{Y}}=10$ is the number of distinct class labels.

This ensures $\M{a}^{(L)}$ (the output of soft-max) is stochastic, i.e.,

$$\sum_{i=1}^k a_i^{(L)}  = 1\kern1em \text{and} \kern1em a_i^{(L)}\geq 0\qquad \forall i\in \{1,\dots,k\}.$$

The activation functions $\sigma^{(\ell)}$ for other layers $1\leq \ell<L$ is often the vectorized version of
-  sigmoid:

  $$\sigma_{\text{sigmoid}}(z) = \frac{1}{1+e^{-z}}$$
-  rectified linear unit (ReLU):

  $$ \sigma_{\text{ReLU}}(z) = \max\{0,z\}. $$

```python
%%html
<b>Activation function:</b><br>

```

Activation function:

The following plots the ReLU activation function.

```python
def ReLU(z):
    return np.max([np.zeros(z.shape), z], axis=0)

z = np.linspace(-5, 5, 100)
plt.plot(z, ReLU(z))
plt.xlim(-5, 5)
plt.title(r'ReLU: $\max\{0,z\}$')
plt.xlabel(r'$z$')
plt.show()
```

**Exercise** Complete the vectorized function `sigmoid` using the vectorized exponentiation `np.exp`.

```python
def sigmoid(z):
    # YOUR CODE HERE
    raise NotImplementedError()

plt.plot(z, ReLU(z))
plt.xlim(-5, 5)
plt.title(r'Sigmoid function: $\frac{1}{1+e^{-z}}$')
plt.xlabel(r'$z$')
plt.show()
```

```python
# tests
```

The following uses the [`keras`](https://keras.io/) library to define the basic neural network achitecture. `keras` runs on top of `tensorflow` and offers a higher-level abstraction to simplify the construction and training of a neural network. ([`tflearn`](https://github.com/tflearn/tflearn) is another library that provides a higher-level API for `tensorflow`.)

```python
def create_simple_model():
    tf.keras.backend.clear_session() # clear keras cache.
                        # See https://github.com/keras-team/keras/issues/7294
    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
        tf.keras.layers.Dense(16, activation=tf.keras.activations.relu),
        tf.keras.layers.Dense(16, activation=tf.keras.activations.relu),
        tf.keras.layers.Dense(10, activation=tf.keras.activations.softmax)
    ], 'Simple_sequential')
    return model

model = create_simple_model()
model.summary()
```

The above defines [a linear stack](https://www.tensorflow.org/api_docs/python/tf/keras/layers) of [fully-connected layers](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Dense) after [flattening the input](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Flatten). The method `summary` is useful for [debugging in Keras](https://keras.io/examples/keras_recipes/debugging_tips/).

**Execise** Assign to `n_hidden_layers` the number of hidden layers for the above simple sequential model.

*Hint:* The layer `Flatten` do not counts as a hidden layer since it simply reshape the input without using any trainable parameters. The output layer also do not count as a hidden layer since its output is the output of the neural network, not intermediate (hidden) values that require further processing by the neurons.

```python
# YOUR CODE HERE
raise NotImplementedError()
n_hidden_layers
```

```python
# tests
```

### Train and validate

Recall that a neural network that minimizes the cross entropy gives $p_{\R{y}|\RM{x}}(y|\RM{x})$ a.s. for all $y\in \mc{Y}$ and any possible input image $\RM{x}$:

\begin{align}
\min_{q_{\R{y}|\RM{x}}} \overbrace{\E\left[\log \frac{1}{q_{\R{y}|\RM{x}}(\R{y}|\RM{x})}\right]}^{ \text{Cross entropy}\\ H(p_{\R{y}|\RM{x}}\|q_{\R{y}|\RM{x}}|p_{\RM{x}}):=}
&= H(\R{y}|\RM{x})
\end{align}

The cross entropy cannot be computed exactly without knowing the joint distribution $p_{\RM{x}\R{y}}$. Nevertheless, it can be estimated from a batch of i.i.d. samples $(\RM{x}_{\R{j}_i},\R{y}_{\R{j}_i})$ for $1\leq i\leq n$:

\begin{align}
\R{L}(\theta)&:=\frac1n \sum_{i=1}^n \log \frac{1}{q_{\R{y}|\RM{x}}(\R{y}_{\R{j}_i}|\RM{x}_{\R{j}_i})}\tag{empirical loss}
\end{align}

where

$$\theta := \operatorname{flat}(\M{W}^{(\ell)},\M{b}^{(\ell)}\mid 0\leq \ell \leq L)$$

is the vector of parameters of the neural network defined in (net).

A mini-batch [gradient descent algorithm](https://en.wikipedia.org/wiki/Stochastic_gradient_descent) is often used to reduce the loss. It iteratively updates/trains the neural network parameters:

$$\theta \leftarrow \theta -s\nabla \R{L}(\theta)$$

by computing the gradient $\nabla \R{L}(\theta)$ on a randomly selected minibatch of examples and choosing an appropriate learning rate $s$.

As explained in the [lecture series on deep learning](https://www.youtube.com/embed/videoseries?list=PLZHQObOWTQDNU6R1_67000Dx_ZCJB-3pi) in the introduction section:
- The gradient can be computed systematically using a technique called *backpropagation* due to the structure of the neural network in (net).
- The learning rate can affect the convergence rate of the loss to a local minima:
    - $\theta$ may overshoot its optimal value if $s$ is too large, and
    - the convergence can be very slow if $\theta$ is too small.

A more advanced method called [Adam (Adaptive Momentum Estimation)](https://en.wikipedia.org/wiki/Stochastic_gradient_descent#Adam) can adaptively choose $s$ to speed up the convergence.

```python
%%html
<div>
<b>What is gradient descent?</b><br>

</div>
<div>
<b>How to choose the step size?</b><br>

</div>
```

What is gradient descent?

How to choose the step size?

The [loss function](https://keras.io/api/losses/), gradient descent algorithm, and the performance metrics can be specified using the [`compile` method](https://keras.io/api/losses/).

```python
def compile_model(model):
    model.compile(loss=tf.keras.losses.SparseCategoricalCrossentropy(),
                  optimizer=tf.keras.optimizers.Adam(0.001),
                  metrics=[tf.keras.metrics.SparseCategoricalAccuracy()])
    return model

compile_model(model)
model.loss, model.optimizer
```

We can now train the neural network using the method [`fit`](https://www.tensorflow.org/api_docs/python/tf/keras/Model#fit) of the compiled model:

```python
if input('Train? [Y/n]').lower() != 'n':
    model.fit(ds_b["train"])
```

**Exercise** By default, the neural network is trained for 1 epoch. What happens to the training accuracy if you rerun the above cell to train the model for another epoch?

YOUR ANSWER HERE

We can set the parameter `epochs` to train the neural network for multiple epochs since it is quite unlikely to train a neural network well with just one epoch. To determine whether the neural network is well-trained (when to stop training), we should also use a separate validation set to evaluate the performance of the neural network. The validation set can be specified using the parameter `validation_set` as follows:

```python
if input('Train? [Y/n]').lower() != 'n':
    model.fit(ds_b["train"], epochs=6, validation_data=ds_b["test"])
```

**Exercise** Is the maximum validation accuracy `val_sparse_categorical_accuracy` (over different epoches) an unbiased estimate of the performance of deep learning for the given dataset? If not, how to avoid the bias?

YOUR ANSWER HERE

**Exercise** Is it a good idea to use cross-validation to evaluate the neural network? Why or why not?

YOUR ANSWER HERE

### Log the results

To call additional functions during training, we can add the functions to the `callbacks` parameter of the model `fit` method. For instance:

```python
import tqdm.keras

if input('Train? [Y/n]').lower() != 'n':
    model.fit(ds_b["train"],
              epochs=6,
              validation_data=ds_b["test"],
              verbose=0,
              callbacks=[tqdm.keras.TqdmCallback(verbose=2)])
```

The above code uses [`tqdm.keras.TqdmCallback()`](https://tqdm.github.io/docs/keras/) to return a callback function that displays a graphical progress bar:
- Setting `verbose=0` for the method `fit` disables the default text-based progress bar.
- Setting `verbose=2` for the class `TqdmCallback` show and keep the progress bars for training each batch. Try changing `verbose` to other values to see different effects.

An import use of callback functions is to save the models and results during training for further analysis. We define the following function `train_model` for this purpose:
- Take a look at the docstring to learn its basic usage, and then
- learn the implementations in the source code.

```python
import os, datetime, pytz

def train_model(model,
                fit_params={},
                log_root='.',
                save_log_params=None,
                save_model_params=None,
                debug_params=None):
    '''Train and test the model, and return the log directory path name.

    Parameters:
    ----------
    log_root (str): the root directory for creating log directory

    fit_params (dict): dictionary of parameters to pass to model.fit.
    save_log_params (dict): dictionary of parameters to pass to
        tf.keras.callbacks.TensorBoard to save the results for TensorBoard.
        The default value None means no logging of the results.
    save_model_params (dict): dictionary of parameters to pass to
        tf.keras.callbacks.ModelCheckpoint to save the model to checkpoint
        files.
        The default value None means no saving of the models.
    debug_params (dict): dictionary of parameters to pass to
        tf.debugging.experimental.enable_dump_debug_info for debugger
        v2 in tensorboard.
        The default value None means no logging of the debug information.

    Returns:
    -------
    str: log directory path that points to a subfolder of log_root named
        using the current time.
    '''
    # use a subfolder named by the current time to distinguish repeated runs
    log_dir = os.path.join(
        log_root,
        datetime.datetime.now(
            tz=pytz.timezone('Asia/Hong_Kong')).strftime("%Y%m%d-%H%M%S"))

    callbacks = fit_params.pop('callbacks', []).copy()

    if save_log_params is not None:
        # add callback to save the training log for further analysis by tensorboard
        callbacks.append(
            tf.keras.callbacks.TensorBoard(log_dir,
                                           **save_log_params))

    if save_model_params is not None:
        # save the model as checkpoint files after each training epoch
        callbacks.append(
            tf.keras.callbacks.ModelCheckpoint(os.path.join(log_dir, '{epoch}.ckpt'),
                                               **save_model_params))

    if debug_params is not None:
        # save information for debugger v2 in tensorboard
        tf.debugging.experimental.enable_dump_debug_info(
            log_dir, **debug_params)

    # training + testing (validation)
    model.fit(ds_b['train'],
              validation_data=ds_b['test'],
              callbacks=callbacks,
              **fit_params)

    return log_dir
```

For example:

```python
fit_params = {'epochs': 6, 'callbacks': [tqdm.keras.TqdmCallback()], 'verbose': 0}
log_root = 'private/demo/'
save_log_params = {'update_freq': 100, 'histogram_freq': 1}
save_model_params = {'save_weights_only': True, 'verbose': 1}
debug_params = {'tensor_debug_mode': "FULL_HEALTH", 'circular_buffer_size': -1}

if input('Train? [Y/n]').lower() != 'n':
    model = compile_model(create_simple_model())
    log_dir = train_model(model,
                          fit_params = fit_params,
                          log_root=log_root,
                          save_log_params=save_log_params,
                          save_model_params=save_model_params,
                          debug_params=debug_params)
```

By providing the `save_model_params` to the callback [`tf.keras.callbacks.ModelCheckpoint`](https://www.tensorflow.org/tutorials/keras/save_and_load#save_checkpoints_during_training), the model is saved at the end of each epoch to `log_dir`.

```python
!ls {log_dir}
```

Saving the model is useful because it often takes a long time to train a neural network. To reload the model from the latest checkpoint and continue to train it:

```python
if input('Continue to train? [Y/n]').lower() != 'n':
    # load the weights of the previously trained model
    restored_model = compile_model(create_simple_model())
    restored_model.load_weights(tf.train.latest_checkpoint(log_dir))
    # continue to train
    train_model(restored_model,
                log_root=log_root,
                save_log_params=save_log_params)
```

By providing [`tf.keras.callbacks.TensorBoard`](https://www.tensorflow.org/tensorboard/get_started#using_tensorboard_with_keras_modelfit) as a callback function to the `fit` method earlier, the training logs can be analyzed using TensorBoard.

```python
if input('Execute? [Y/n]').lower() != 'n':
    %load_ext tensorboard
    %tensorboard --logdir {log_dir}
```

The `SCALARS` tab shows the curves of training and validation losses/accuracies after different batches/epoches. The curves often have jitters as the gradient descent is stochastic (random). To see the typical performance, a smoothing factor $\theta\in [0,1]$ can be applied on the left panel. The smoothed curve $\bar{l}(t)$ of the original curve $l(t)$ is defined as

\begin{align}
\bar{l}(t) = \theta \bar{l}(t-1) + (1-\theta) l(t)
\end{align}

which is called the moving average. Try changing the smoothing factor on the left panel to see the effect.

**Exercise**  If the smoothing factor $\theta$ is too large, would it cause bias when using empirical loss or performance to estimate the actual loss or performance? If so, is estimate overly optimistic or pessimistic?

YOUR ANSWER HERE

We can also visualize the input images in TensorBoard:
- Run the following cell to write the images to the log directory.
- Click the `refresh` button on the top of the previous TensorBoard panel.
- Click the `IMAGE` tab to show the images.

```python
if input('Execute? [Y/n]').lower() != 'n':
    file_writer = tf.summary.create_file_writer(log_dir)

    with file_writer.as_default():
        # Don't forget to reshape.
        images = np.reshape([image for (image, label) in ds["train"].take(25)],
                            (-1, 28, 28, 1))
        tf.summary.image("25 training data examples",
                         images,
                         max_outputs=25,
                         step=0)
```

In addition to presenting the results, TensorBoard is useful for debugging deep learning. In particular, learn
- to check the model graph under the [`GRAPHS`](https://www.tensorflow.org/tensorboard/graphs) tab,
- to debug using the [`DEBUGGER v2` tab](https://www.tensorflow.org/tensorboard/debugger_v2), and
- to [publish your results](https://www.tensorflow.org/tensorboard/get_started#tensorboarddev_host_and_share_your_ml_experiment_results).

TensorBoard can also show simultaneously the logs of different runs stored in different subfolders of the log directory:

```python
if input('Execute? [Y/n]').lower() != 'n':
    %load_ext tensorboard
    %tensorboard --logdir {log_root}
```

You can select different runs on the left panel to compare their performance.

Note that loading the log to TensorBoard may consume a lot of memory. You can list the TensorBoard notebook instances and kill those you do not need anymore by running `!kill {pid}`.

```python
import tensorboard as tb
tb.notebook.list() # list all the running TensorBoard notebooks.

pids_to_kill = [] # modify the list of pid to kill
for pid in pids_to_kill:
    !kill {pid}
```

**Exercise** Train the following network with [dropout](https://en.wikipedia.org/wiki/Dilution_(neural_networks)#Dropout). Try to tune the network for the best accuracy. Use `log_root='logs/dropout/'` to so your log will also be submitted along with your notebook. Put your training code inside the body of the conditional `if input...` for autograding to work.

```python
def create_dropout_model():
    model = tf.keras.models.Sequential([
        tf.keras.layers.Flatten(input_shape=(28, 28, 1)),
        tf.keras.layers.Dense(128, activation=tf.keras.activations.relu),
        tf.keras.layers.Dropout(0.2),  # dropout
        tf.keras.layers.Dense(10, activation=tf.keras.activations.softmax)
    ], name="Dropout")
    return model

model = compile_model(create_dropout_model())
print(model.summary())
log_root = 'logs/dropout/'

if input('Train? [Y/n]').lower() != 'n':
    # YOUR CODE HERE
    raise NotImplementedError()
```

**Exercise** Explore the [convolutional neural network (CNN)](https://en.wikipedia.org/wiki/Convolutional_neural_network). Try to tune the network for the best accuracy.

```python
def create_cnn_model():
    model = tf.keras.models.Sequential([
        tf.keras.layers.Conv2D(32,
                               3,
                               activation='relu',
                               input_shape=(28, 28, 1)),
        tf.keras.layers.MaxPooling2D(),
        tf.keras.layers.Flatten(),
        tf.keras.layers.Dense(64, activation='relu'),
        tf.keras.layers.Dense(10, activation='softmax')
    ], name="CNN")
    return model

model = compile_model(create_cnn_model())
print(model.summary())
log_root = 'logs/cnn/'

if input('Train? [Y/n]').lower() != 'n':
    # YOUR CODE HERE
    raise NotImplementedError()
```

**Exercise** Launch TensorBoard to show the best performances of each of the two neural network architectures. Note that to clean up the log of the inferior results, you may need to kill the TensorBoard instance. It is easier to use the vscode interface or the terminal in the lab interface to remove folders.

```python
if input('Execute? [Y/n]').lower() != 'n':
    # YOUR CODE HERE
    raise NotImplementedError()
```

## Deployment

Once you are satisfied with the result, you can deploy the model as a web application.

```python
model.save('model.h5')
```

Convert the model to files in `mnist/model` that can be loaded by [`tensorflow.js`](https://www.tensorflow.org/js) on the page `mnist/index.html`.

```python
!tensorflowjs_converter --input_format keras 'model.h5' 'mnist/model'
```

To download the `mnist` folder, we first compress it as follows:

```python
!zip -r mnist.zip mnist/* index.html
```

Finally, you can download the zip file [here](./mnist.zip) and host the web application on a static web server.