# Introduction to Quantum Cryptography
In this notebook we are going to introduce the theory of **Quantum Cryptography** and some of its applications.

The **preface** consists in a quick recap about traditional cryptography, its progress, and the reasons that led to the conception of quantum cryptography and its contemporary achievements.

Following up, we recall some useful concepts of **quantum mechanics** without being too pedantic, more as a tool needed to argue the enforcement of this practice compared to classical crypto. In fact, this tutorial is intended to be divulgative and concise, not to substitute a scientific publication; to maintain the aim of it being more accessible, we renounce to some formality (studies will be linked where possible).

## Requirements
Throughout the notebook we will make use of:
- Python 3.6 (although python 3.5 works as well)
- QISKit (with a Qconfig file for remote backend)
- Standard python scientific libs stack: numpy etc.
- Knowledge of basic Quantum Mechanics is assumed
- Notions of traditional Cryptography are helpful although not strictly necessary

## Preface
Traditional **Cryptography** uses mathematical techniques to secure the communication of information. The procedures, that explain how to perform the operations, give shape to a certain **protocol** or **cipher**, a policy to be followed to enforce the achievement of different cryptographic tasks, such as: confidentiality, authenticity and data integrity.

The related field of **cryptanalysis** focuses on protocol breaking, elaborating its weaknesses, and  finding a way of getting access to secret information. The cat-and-mouse game played by cryptologists and cryptoanalysts has pushed the field and driven progress in it tremendously.

The reliability of a protocol is defined as the resilience to a certain attack performed in a given amount of time. Thus, it was thought that there was not such a thing as **intrinsic security** and that, given enough time or enough resource of computation any protocol would be broken. This is actually not true; perfectly concealing a message is indeed possible, although it is somewhat inconvenient to implement.

Now suppose we have the simple scenario: a sender, **Alice**, who wants to deliver a secret message to a recipient, **Bob**, unbeknown of **Eve**, an eavesdropper. Alice takes the **plaintext**, the unencrypted information, and produces the **ciphertext** with a given criterion, using a **cryptographic key**.

This key is a string of bits that is applied to the plaintext using a mathematical function, such as the bitwise XOR or a transposition/substitution map etc. The encrypted message is sent over a channel and delivered to Bob, who also possesses the same cryptographic key and performs the inverse operation, he applies the key to the ciphertext to recover the plaintext.

In the case in which they have pre-shared the **same** key, called **symmetric cryptography**, this is the only element that Eve, who has intercepted only the ciphertext, needs to retrieve the plaintext message. Simple ciphers, such as transposition and substitution maps, are vulnerable to frequency analysis, meaning that the plaintext characters could be inferred counting the occurrences of characters in the ciphertext, and using statistical methods plaintext words can be reconstructed. Once the relations have been established, the key is obtained.

To ensure perfect secrecy one would have to use a completely random generated key, long as much as the plaintext, and never use it again, a cipher known as **one-time pad** (OTP). Once this random key is generated, Alice can apply for example the bitwise XOR to the plaintext and, being the XOR symmetrical, Bob just needs to reapply the XOR with the same key to the ciphertext to obtain the plaintext.

I stress the word random because this is the whole strength of the protocol: if Eve attempts to bruteforce the ciphertext and tries all the different keys to a one-word message for example, she would eventually find the right word, but she would find as well all the other possible one-words and she would be unable to decide which one to pick!

However OTP has never gained widespread popularity for several practical reasons: there is not such a thing as a perfectly random key, careful storing of the key (and safe disposal to prevent any reuse) is a problem, and going through the generation of a pre-shared key for every message to send is a huge effort. The latter is the biggest problem of all and it makes this cipher extremely impractical for real use cases.

The invention of **asymmetric cryptography**, also known as **public-key cryptography**, has been the most important achievement of modern cryptography. In this system each user generates a pair of keys, a **public key**, spread freely to everyone, and a **private key**, kept in maximum secrecy. Alice can encrypt a message using Bob's public key, but only Bob is able to decrypt it using his private key.

However, to implement this feature, we have to use mathematical functions that are generally difficult to solve but, once a solution is known, it is not so difficult to verify it. Contemporary implementations of mathematical algorithms make use of integer factorization, discrete logarithm, and elliptic curve relationships. It is not the scope of this notebook to go into the details of this process, which is widely discussed in literature.

Let's just say it offers a good compromise between security and practicality, overcoming the problem of one-time key generation and, most importantly, **key exchange**. In fact in real use cases almost all major implementations of cryptography are realized asymmetrically (although sometimes in combination with symmetrical keys), such as TLS, S/MIME, PGP, OMEMO, OTR etc.

The advent of **quantum computers** has raised an unexpected problem for public key cryptography. It has been proven that a quantum computer is able to solve all mathematical algorithms currently in use in a much more efficient way. Thanks to the properties of the **Quantum Fourier Transform** (QFT) and its implementation in Shor's algorithm, the time needed to find a general solution for the problem is reduced from being exponential to  being only polynomial, thus opening a window for possible brute-force attacks.

At the present time quantum computers are not powerful enough to actually perform these operations on big numbers, but in the not so distant future they might as well be, so much so that cryptography experts have begun to discuss strategies to harden cryptographic protocols in case of widely available quantum computers; this field is called **post-quantum cryptography**.

Quantum computers also present another extraordinary advantage, that may lead to perfect secrecy. Storing information in the form of qubits is the first step in implementing **quantum cryptography**, the science of safely exchanging information using quantum physics. This is achieved through the establishment of a **quantum channel**, a special apparatus capable of delivering quantum information in the form of qubits. Confidentiality is guaranteed by laws of quantum mechanics, a remarkable way of calling physics in aid of mathematics.

We will see in the next notebook how **quantum key distribution** solves, once and for all (?), the problems associated with key sniffing. Although it may seem hard to do in practice, it has actually been accomplished multiple times, given the right machinery, up to the realization of quantum encrypted satellite communication. In the following section we recall relevant properties of quantum mechanics needed to put in place a quantum channel.

## Quantum mechanics
### Qubits
A **quantum channel** provides a medium and an apparatus to exchange quantum information between two parties using **qubits**, a set of two state quantum mechanical systems. There are many kinds of qubits such as electrons, photons, quantum dots, Josephson junctions etc. **Polarized photons** were the first type of qubits to be used in practice and they continue to be popular today, since they are relatively easy to obtain compared to other qubits, and they can be exchanged either through fiber optic cables or via laser beacons. According to quantum mechanics, electromagnetic waves can be thought as stream of photons. The term **polarization** classically refers to the direction of the electric field of the wave, but can be also seen as an intrinsic  property of single photons. Natural sources of light are incoherent and unpolarized because they consist of a random mixture of waves having different frequencies, phases, and polarization states, but polarized light can be produced by passing light through a **polarizing filter**, which allows waves of only one polarization to pass through.

Consider a **monochromatic electromagnetic wave** that propagates along the $\hat{z}$ axis. The electric field, $\vec{E}$, lies in the $\hat{x}\hat{y}$ plane, the angle $\theta$ that it makes with the $\hat{x}$ axis is the  angle of monochromatic radiation. In general the evolution of the electric field in time can be described according to the following equation (using complex notation):

\begin{equation}
    \vec{E}(\vec{r},t)=|\vec{E}|\ \Re\big{(}\lvert\psi\rangle e^{i(kz - \omega t)}\ \big{)}
\end{equation}

where $|\vec{E}|$ is the amplitude of the electric field, $\omega$ is the angular frequency of the wave in terms of the wave number $k$, and $ \lvert\psi\rangle = \big{(} cos\theta e^{i\phi_x},\ sin\theta e^{i\phi_y}\big{)}$ is **Jones vector** that describes the **polarization** of the photon. This vector is composed of two complex amplitudes, each with its own magnitude and phase, and represents the projection of $\vec{E}$ on the $\hat{x}$ and $\hat{y}$ axis, so that $|E_x| = |\vec{E}|cos\theta$ and $|E_y| = |\vec{E}|sin\theta$, and thus $|\vec{E}|^2 = |E_x|^2 + |E_y|^2$. These phases are usually absorbed in an overall phase $\phi=\phi_x - \phi_y$ because, as $t$ and $z$ change, $\phi_x$ and $\phi_y$ change in the same way so that $\phi$ is actually unchanged in the evolution. The complex notation can be somewhat scary to non accustomed readers, but it's easy to rewrite the general expression in a real form:

\begin{equation}
   \vec{E}(\vec{r},t)=E_x\ cos(kz-\omega t+\phi_x)\ \hat{x}+E_y\ cos(kz-\omega t+\phi_y)\ \hat{y}
\end{equation}

The wave is said to be **linearly polarized** if $\phi = 0$, there is no phase difference between $E_x$ and $E_y$ and $\phi_x=\phi_y$. If we set $z=0$ and fix the phases at $\phi_x=\phi_y=0$ we obtain a very simple representation of linear polarization in a generic direction, at an angle $\theta$ with respect to the $\hat{x}$ axis:

\begin{equation}
   \vec{E}(\vec{r},t)=E_x\ cos(\omega t)\ \hat{x}+E_y\ cos(\omega t)\ \hat{y} = |\vec{E}|\ cos\theta\ cos(\omega t)\ \hat{x}+|\vec{E}|\ sin\theta\ cos(\omega t)\ \hat{y}
\end{equation}

We can see explicitly that the electric field oscillates in time but maintains its direction constant.
If we have $sin\theta=0$, only the $\hat{x}$ component is present, and the wave is said to be linearly polarized in the $\hat{x}$ direction, typically called **horizontal polarization**; viceversa if $cos\theta=0$, only the $\hat{y}$ component is present, and the wave is said to be linearly polarized in the $\hat{y}$ direction, typically called **vertical polarization**.

This prompts us to introduce orthonormal unit vectors such that:

\begin{equation}
\lvert 0 \rangle =
\lvert x \rangle =
  \begin{pmatrix}
    1 \\
    0 \\
    \end{pmatrix}
    , \ \ \
    \lvert 1 \rangle =
    \lvert y \rangle =
  \begin{pmatrix}
    0 \\
    1 \\
    \end{pmatrix}
\end{equation}

so that the generic linearly polarized state can be written, in terms of the Jones vector, as:

\begin{equation}
\lvert \psi \rangle = e^{i \alpha}\big{(}cos\theta \lvert 0 \rangle + sin\theta \lvert 1 \rangle\big{)}
\end{equation}

If it looks familiar it is probably because the $\lvert 0 \rangle$ and $\lvert 1 \rangle$ vectors are the usual qubits used in the [Bloch sphere](https://www.qiskit.org/ibmqx-user-guides/beginners-guide/004-The_Weird_and_Wonderful_World_of_the_Qubit/001-The_Weird_and_Wonderful_World_of_the_Qubit.html). But these are not the only states that we can use with polarized photons. It's useful to introduce the quantum superpositions of these two states, that together make up another valid orthonormal basis called **Hadamard basis** that we will use, and the vectors of the photons that represent **diagonal polarization**, or linearly polarized at 45° from the $\hat{x}$ axis, and **anti-diagonal polarization**, or linearly polarized at −45° from the $\hat{x}$ axis:

\begin{equation}
\lvert \nearrow \rangle =
\frac{1}{\sqrt{2}} \big{(}\lvert 0 \rangle + \lvert 1 \rangle \big{)}=
  \frac{1}{\sqrt{2}}
  \begin{pmatrix}
    1 \\
    1 \\
    \end{pmatrix}
    , \ \ \ \
\lvert \searrow \rangle =
\frac{1}{\sqrt{2}} \big{(}\lvert 0 \rangle - \lvert 1 \rangle \big{)}=
  \frac{1}{\sqrt{2}}
  \begin{pmatrix}
    1 \\
    -1 \\
    \end{pmatrix}
\end{equation}

From the equation of the electric field it's straightforward to see that in linear polarization, the fields oscillate in a single direction. Also note that diagonal polarization states, are linear combinations of horizontal and vertical polarization states, and thus they are **not orthogonal** to these. For the sake of completeness, we have to mention that the electric field can also rotate at a constant rate with respect to $\hat{z}$ in which case we would have **circular polarization**. The rotation can have two possible directions: clockwise or anti-clockwise anti-clockwise: if the fields rotate clockwise, the result is called right circular polarization, viceversa if they rotate anti-clockwise, it is called left circular polarization:

\begin{equation}
\lvert \circlearrowright \rangle =
\frac{1}{\sqrt{2}} \big{(}\lvert 0 \rangle - i \lvert 1 \rangle \big{)}=
  \frac{1}{\sqrt{2}}
  \begin{pmatrix}
    1 \\
    -i \\
    \end{pmatrix}
    , \ \ \ \
\lvert \circlearrowleft \rangle =
\frac{1}{\sqrt{2}} \big{(}\lvert 0 \rangle + i \lvert 1 \rangle \big{)}=
  \frac{1}{\sqrt{2}}
  \begin{pmatrix}
    1 \\
    i \\
    \end{pmatrix}
\end{equation}

Note that all the other points on the Bloch sphere that are not represented by these states are still valid polarization states for the photon, known as **elliptical polarization**, we will not make use of these states.

Now that we know how to obtain and manipulate working qubits we want to know their characteristics that make them suitable for key sharing. In particular the two fundamental properties that help us in our task are the fact that a measurement of an unknown quantum state causes it to collapse in an unknown eigenstate (Heisenberg's principle) and that it's impossible to create an identical copy of an arbitrary quantum state (no-cloning theorem).

### No-cloning Theorem
In traditional electronics, digital signals are carried by conductive wires or traces in which current flows. If we wish to clone an arbitrary unknown signal on another track, the only thing we need to do is to bifurcate the wire and, given the right accuracy on the circuitry, we end up with the same signal duplicated on two tracks. In quantum computers things work differently: if the state is in an arbitrary state, we have no way of making exact copies of it. This is due do the fact that, when we work with quantum systems, there are only two permissible quantum operations with which we may manipulate the state: we can perform an observation, which irreversibly collapses the system into some eigenstate of an observable, corrupting the information contained in the original state, or we can utilize a unitary operator.

Suppose that we have a quantum state $\lvert \psi \rangle$ that we wish to duplicate using a unitary operator, $U$: $\lvert \psi \rangle \rightarrow \lvert \psi \rangle\lvert \psi \rangle$. Since unitary transformations connect spaces of the same dimension we have to introduce a service state, $\lvert s \rangle$, so that:

\begin{equation}
U\big{(}\lvert \psi \rangle \lvert s \rangle \big{)} = \lvert \psi \rangle\lvert \psi \rangle
\end{equation}

Select an arbitrary state, $\lvert \chi \rangle$. Unitary operators are useful because they conserve the scalar product between states. Recall the definition of a unitary operator: $U^{+}U = UU^{+}=I$, from which conservation of scalar product follows:

\begin{equation}
    \langle \psi \lvert \chi \rangle = \langle U\psi \lvert U\chi \rangle = \langle U^{+}U\psi \lvert \chi \rangle  = \langle \psi \lvert \chi \rangle
\end{equation}

Now, knowing that the cloning machine has to work for any arbitrary state, we can apply it to $\chi$ as well: $U\big{(}\lvert \chi \rangle \lvert s \rangle \big{)} = \lvert \chi \rangle\lvert \chi \rangle$. We take the scalar product between the two states and we apply $U$:

\begin{equation}
    \langle \psi \lvert \chi \rangle \langle s \lvert s \rangle = \langle \psi \lvert \chi \rangle \langle \psi \lvert \chi \rangle
\end{equation}

We can normalize $\lvert s \rangle $ so that we obtain in the end:

\begin{equation}
    \langle \psi \lvert \chi \rangle = {\langle \psi \lvert \chi \rangle}^{2}
\end{equation}

This equation has only two solutions: $ \langle \psi \lvert \chi \rangle = 1$ or $ \langle \psi \lvert \chi \rangle = 0$ which means that either the two states are orthogonal or they are the same state, thus proving that we cannot perfectly clone an arbitrary state.

The no-cloning theorem has profound implications in quantum computing. It prevents classical **error correction** on quantum states, since backup copies of a state cannot be made in the middle of a quantum computation and used for correcting subsequent errors. It is also a requirement for the no-communication theorem, which states that it's impossible to trasmit quantum information over a quantum channel only by measuring entangled states, as outlined in the famous EPR experiment.

The most relevant consequence of the no-cloning theorem for our case is the fact that it's impossible to distinguish non-orthogonal states with only one measurement. Suppose that we have a state $\lvert \psi \rangle$ that can be one of two states: either $\lvert 0 \rangle$ or $\lvert \nearrow \rangle = \frac{1}{\sqrt{2}} \big{(}\lvert 0 \rangle + \lvert 1 \rangle \big{)}$. We have at our service a measuring device that operates on the computational basis $\{\lvert 0 \rangle,\  \lvert 1 \rangle\}$. According to the principle of quantum mechanics, the result of the measure on the system will return one of the possible eigenvalues, either '$0$' or '$1$' and the state would collapse in the relative eigenstate. If we obtain '$1$' the state would now become $\lvert 1 \rangle$ and right away we know that we measured state $\lvert \nearrow \rangle$. But if we obtain '$0$' the state would collapse in $\lvert 0 \rangle$ and we have no way of saying in which one of the two states it was in before the measurement. If we had a cloning machine we could use it to make many copies of the state and, by performing a measurement on each state, we could use statistics to infer which state we began with.

This property of non-orthogonal states is the core idea behind quantum cryptography. It's worth to introduce the first thought application of qubits in the real world that later served as an inspiration for quantum key sharing. Quantum money was the first conceptually idealized currency that used quantum mechanics to guarantee its security.

## Quantum Money
In 1970 Stephen Wiesner, a graduate student at Columbia University, came up with an interesting idea that was actually rejected by several journals but was eventually published in 1983. He envisioned a new kind of bank note that, in addition to a unique serial number, would also contain a certain number of qubits. These qubits could be made using polarized photons, quantum dots, isolate nuclei or any other two state systems, the important thing is that they are fixed in a very specific state that only the bank knows. In particular each qubit is randomly set in an eigenstate of one of two basis that are not orthogonal to each other, such as $\{\lvert 0 \rangle,\  \lvert 1 \rangle\}$ and $\{\lvert \nearrow \rangle,\  \lvert \searrow \rangle\}$, and they are made in such a way that phase coherence is maintained in time.

The bank keeps a record composed of every bank note number associated with its quantum state so that, when questioned about the validity of the bank note, the bank performs a check to see if each qubit is still in its initial state or it has switched to the orthogonal state. Now consider a counterfeiter that tries to forge a new bank note from an old one. Only the bank knows the right sequence of basis used to measure the quantum state, so the counterfeiter can only guess which one of the four eigenstates each qubit is in but he needs to measure each qubit in one of the two bases to find its eigenstate. He has $1/2$ chance of picking the wrong basis, so he will guess rightly about half of the times, but even in the case he picks the wrong basis, in this event he has a probability $1/2$ of ending up in the right eigenstate, since the wrong basis can be seen as a superposition of the base elements of the right basis; thus, he will correctly guess each qubit with a probability of $3/4$. If we imagine to have $n$ qubits embedded on the bank note, the probability of getting all of them right and perfectly forge an identical bank note would be ${(3/4)}^{n}$; for n=20 this is  $0.00317$ or about $0.3\%$, which is low enough to discourage any potential forgery.

The act of observing the qubit, which is indeed a projective measure, causes the qubit to collapse in one of its eigenstate, so that not only the bank note is unsuccessfully cloned, but also the original bank note is now useless. This peculiar property of quantum mechanics, wave function collapse, in conjunction with non-orthogonal states, is what enforces security in quantum cryptography.


```python
# Imports
import torch
cuda = torch.cuda.is_available()
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
import sys
from tqdm.auto import tqdm
from sklearn.decomposition import PCA
from sklearn.manifold import TSNE
sys.path.append("../../semi-supervised")
```

```python
from utils import batch_normalization, dynamic_partition, dynamic_stitch
```

# HIAuxiliary Deep Generative Model

The Auxiliary Deep Generative Model [[Maaløe, 2016]](https://arxiv.org/abs/1602.05473) posits a model that with an auxiliary latent variable $a$ that infers the variables $z$ and $y$. This helps in terms of semi-supervised learning by delegating causality to their respective variables. This model was state-of-the-art in semi-supervised until 2017, and is still very powerful with an MNIST accuracy of *99.4%* using just 10 labelled examples per class.

## Training

The lower bound we derived in the notebook for the **deep generative model** is similar to the one for the ADGM. Here, we also need to integrate over a continuous auxiliary variable $a$.

For labelled data, the lower bound is given by.
\begin{align}
\log p(x,y) &= \log \int \int p(x, y, a, z) \ dz \ da\\
&\geq \mathbb{E}_{q(a,z|x,y)} \bigg [\log \frac{p(x,y,a,z)}{q(a,z|x,y)} \bigg ] = - \mathcal{L}(x,y)
\end{align}

Again when no label information is available we sum out all of the labels.

\begin{align}
\log p(x) &= \log \int \sum_{y} \int p(x, y, a, z) \ dz \ da\\
&\geq \mathbb{E}_{q(a,y,z|x)} \bigg [\log \frac{p(x,y,a,z)}{q(a,y,z |x)} \bigg ] = - \mathcal{U}(x)
\end{align}

Where we decompose the q-distribution into its constituent parts. $q(a, y, z|x) = q(z|a,y,x)q(y|a,x)q(a|x)$, which is also what can be seen in the figure.

The distribution over $a$ is similar to $z$ in the sense that it is also a diagonal Gaussian distribution. However by introducing the auxiliary variable we allow for $z$ to become arbitrarily complex - something we can also see when using normalizing flows.

```python
# from datautils import get_mnist

# # Only use 10 labelled examples per class
# # The rest of the data is unlabelled.
# labelled, unlabelled, validation = get_mnist(location="./", batch_size=64, labels_per_class=10)
# alpha = 0.1 * (len(unlabelled) + len(labelled)) / len(labelled)
```

```python
import read_functions as rf
```

```python
dataset = 'defaultCredit'
data_file = dataset+'/data.csv'
types_file = dataset+'/data_types.csv'
miss_file = dataset+'/Missing10-50_1.csv'
true_miss_file = None
batch_size = 128
```

```python
train_data, types_dict, miss_mask, true_miss_mask, n_samples = rf.read_data(data_file, types_file,
                                                                                miss_file,
                                                                                true_miss_file)
# Randomize the data in the mini-batches
random_perm = np.random.permutation(range(np.shape(train_data)[0]))
train_data = train_data[random_perm, :]
miss_mask = miss_mask[random_perm, :]
true_miss_mask = true_miss_mask[random_perm, :]

# Check batch size
if batch_size > n_samples:
    batch_size = n_samples

for t in types_dict:
    t['dim'] = int(t['dim'])

# Compute the real miss_mask
miss_mask = np.multiply(miss_mask, true_miss_mask)

train_data = torch.Tensor(train_data)
miss_mask = torch.Tensor(miss_mask)

labelled = train_data[miss_mask[:,-1] == 1, :-2], train_data[miss_mask[:,-1] == 1, -2:]
unlabelled = train_data[miss_mask[:,-1] == 0, :-2], train_data[miss_mask[:,-1] == 0, -2:]
miss_mask = miss_mask[:,:-1]
type_label = types_dict[-1]
types_dict = types_dict[:-1]
```

```python
alpha = 0.1 * (len(unlabelled) + len(labelled)) / len(labelled)
```

```python
x, y = labelled
u, v = unlabelled
```

```python
# Get an integer number of batches
n_batches = int(np.floor(max(x.size()[0], u.size()[0]) / batch_size))
n_batches
```

    117

```python
if x.size()[0] < u.size()[0]:
    xx = torch.zeros_like(u)
    yy = torch.zeros_like(v)
    xx[:x.size()[0], :] = x
    xx[x.size()[0]:, :] = x[:u.size()[0] - x.size()[0], :]
    x = xx
    yy[:y.size()[0], :] = y
    yy[y.size()[0]:, :] = y[:v.size()[0] - y.size()[0], :]
    y = yy
```

```python
u.size(), x.size(), y.size(), v.size(), len(types_dict)
```

    (torch.Size([15000, 94]),
     torch.Size([15000, 94]),
     torch.Size([15000, 2]),
     torch.Size([15000, 2]),
     23)

```python
from models import HIAuxiliaryDeepGenerativeModel

y_dim = y.size()[1]
z_dim = 5
a_dim = 5
gamma_dim = 5
h_dim = [5]
input_dim = x.size()[1]

model = HIAuxiliaryDeepGenerativeModel([input_dim, y_dim, z_dim, a_dim, gamma_dim, h_dim], types_dict)
model
```

```python
x.shape
```

    torch.Size([15000, 94])

```python
types_dict
```

    [{'type': 'pos', 'dim': 1, 'nclass': ''},
     {'type': 'cat', 'dim': 3, 'nclass': '3'},
     {'type': 'cat', 'dim': 7, 'nclass': '7'},
     {'type': 'cat', 'dim': 4, 'nclass': '4'},
     {'type': 'count', 'dim': 1, 'nclass': ''},
     {'type': 'ordinal', 'dim': 11, 'nclass': '11'},
     {'type': 'ordinal', 'dim': 11, 'nclass': '11'},
     {'type': 'ordinal', 'dim': 11, 'nclass': '11'},
     {'type': 'ordinal', 'dim': 11, 'nclass': '11'},
     {'type': 'ordinal', 'dim': 11, 'nclass': '11'},
     {'type': 'ordinal', 'dim': 11, 'nclass': '11'},
     {'type': 'real', 'dim': 1, 'nclass': ''},
     {'type': 'real', 'dim': 1, 'nclass': ''},
     {'type': 'real', 'dim': 1, 'nclass': ''},
     {'type': 'real', 'dim': 1, 'nclass': ''},
     {'type': 'real', 'dim': 1, 'nclass': ''},
     {'type': 'real', 'dim': 1, 'nclass': ''},
     {'type': 'pos', 'dim': 1, 'nclass': ''},
     {'type': 'pos', 'dim': 1, 'nclass': ''},
     {'type': 'pos', 'dim': 1, 'nclass': ''},
     {'type': 'pos', 'dim': 1, 'nclass': ''},
     {'type': 'pos', 'dim': 1, 'nclass': ''},
     {'type': 'pos', 'dim': 1, 'nclass': ''}]

```python
def binary_cross_entropy(r, x):
    return -torch.sum(x * torch.log(r + 1e-8) + (1 - x) * torch.log(1 - r + 1e-8), dim=-1)

optimizer = torch.optim.Adam(model.parameters(), lr=3e-4, betas=(0.9, 0.999))
```

```python
from itertools import cycle
from inference import HISVI, DeterministicWarmup

# We will need to use warm-up in order to achieve good performance.
# Over 200 calls to SVI we change the autoencoder from
# deterministic to stochastic.
beta = DeterministicWarmup(n=200)

if cuda: model = model.cuda()
elbo = HISVI(model, beta=beta)
```

```python
alpha *= 100
```

```python
losses = []
for epoch in tqdm(range(100)):
    model.train()
    total_loss, L_loss, U_loss, class_loss, accuracy = (0, 0, 0, 0, 0)
    for i in range(n_batches):
        # Create inputs for the feed_dict
        data_tensor, labels, batch_miss_mask = rf.next_batch(x, y, types_dict, miss_mask,
                                                         batch_size, index_batch=i)
        unl_data_tensor, hid_labels, unl_batch_miss_mask = rf.next_batch(u, v, types_dict, miss_mask,
                                                         batch_size, index_batch=i)

        # Delete not known data (input zeros)
        data_observed = data_tensor * batch_miss_mask
        unl_data_observed = unl_data_tensor * unl_batch_miss_mask

        # easier names
        x_batch = data_observed
        u_batch = unl_data_observed
        y_batch = labels
        v_batch = hid_labels

        ######
        if cuda:
            # They need to be on the same device and be synchronized.
            x_batch, y_batch = x_batch.cuda(device=0), y_batch.cuda(device=0)
            u_batch, v_batch = u_batch.cuda(device=0), v_batch.cuda(device=0)

        L = -elbo(x_batch, y_batch, batch_miss_mask)

        # Add auxiliary classification loss q(y|x,a)
        logits = model.classify(x_batch)

        U = -elbo(u_batch, y=None, miss_list=unl_batch_miss_mask)

        # Regular cross entropy
        classication_loss = torch.sum(y_batch * torch.log(logits + 1e-8), dim=1).mean()
        J_alpha = L - alpha * classication_loss + U
#         J_alpha = L

        J_alpha.backward()
        optimizer.step()
        optimizer.zero_grad()

        total_loss += J_alpha.item()
        U_loss += U.item()
        L_loss += L.item()
        class_loss += classication_loss.item()
        accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y_batch, 1)[1].data).float()).item() / n_batches
        ######
    samples_z = model.samples_z
    samples_qa = model.samples_qa
    samples_pa = model.samples_pa
    model.samples_z = []
    model.samples_qa = []
    model.samples_pa = []
    print('Epoch:', epoch+1)
    print('Loss:', total_loss)
    print('---- U:', U_loss)
    print('---- L:', L_loss)
    print('---- cl/tion loss:', class_loss)
    print('Accuracy:', accuracy)
    print('-------------------------------------------\n')
    losses.append((total_loss, U_loss, L_loss, class_loss, accuracy))

         # TODO
#     if epoch % 1 == 0:
#         model.eval()
#         m = len(unlabelled)
#         print("Epoch: {}".format(epoch))
#         print("[Train]\t\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))

#         total_loss, accuracy = (0, 0)
#         for x, y in validation:
#             x, y = Variable(x), Variable(y)

#             if cuda:
#                 x, y = x.cuda(device=0), y.cuda(device=0)

#             L = -elbo(x, y)
#             U = -elbo(x)

#             logits = model.classify(x)
#             classication_loss = -torch.sum(y * torch.log(logits + 1e-8), dim=1).mean()

#             J_alpha = L + alpha * classication_loss + U

#             total_loss += J_alpha.data[0]

#             _, pred_idx = torch.max(logits, 1)
#             _, lab_idx = torch.max(y, 1)
#             accuracy += torch.mean((torch.max(logits, 1)[1].data == torch.max(y, 1)[1].data).float())

#         m = len(validation)
#         print("[Validation]\t J_a: {:.2f}, accuracy: {:.2f}".format(total_loss / m, accuracy / m))
```

```python
alpha
```

```python
losses = np.array(losses)
for l, title in zip(losses.T, ['Total loss', "Unlabeled loss", 'Labeled loss', 'Classification loss', 'accuracy']):
    plt.plot(l)
    plt.title(title)
    plt.xlabel('epoch')
    plt.show()
```

# plot a
# try without classloss
# try breast
# try more dimensions
# try different α because classloss is very small

```python
ZZ = torch.cat([z for z, _ in samples_z], dim=0)
ZY = torch.cat([y for _, y in samples_z], dim=0)
```

```python
pca = PCA(n_components=2)
ZZ_pca = pca.fit_transform(ZZ.detach().numpy())
```

```python
plt.scatter(ZZ_pca[:,0], ZZ_pca[:,1], c=np.argmax(ZY, axis=1), alpha=0.5, s=4);
```

```python
tsne = TSNE(n_components=2, verbose=1, perplexity=20, n_iter=250)
ZZ_tsne = tsne.fit_transform(ZZ.detach().numpy())
```

```python
plt.scatter(ZZ_tsne[:,0], ZZ_tsne[:,1], c=np.argmax(ZY, axis=1), alpha=0.5, s=4);
```

```python
for zz, yy in samples_z:
    zz = zz.detach().numpy()
    plt.scatter(zz[:,0], zz[:,1], c=np.argmax(yy, axis=1), alpha=0.5, s=4)
plt.title('Latent Space for z')
plt.show()
```

```python
for qa, yy in samples_qa:
    qa = qa.detach().numpy()
    plt.scatter(qa[:,0], qa[:,1], c=np.argmax(yy, axis=1), alpha=0.5, s=4)
plt.title('Latent Space for a')
plt.show()
```

```python
for pa, yy in samples_pa:
    pa = pa.detach().numpy()
    plt.scatter(pa[:,0], pa[:,1], c=np.argmax(yy, axis=1), alpha=0.5, s=4)
plt.title('Latent Space for a')
plt.show()
```

The library is conventially packed with the `SVI` method that does all of the work of calculating the lower bound for both labelled and unlabelled data depending on whether the label is given. It also manages to perform the enumeration of all the labels.

Remember that the labels have to be in a *one-hot encoded* format in order to work with SVI.

## Conditional generation

When the model is done training you can generate samples conditionally given some normal distributed noise $z$ and a label $y$.

*The model below has only trained for 10 iterations, so the perfomance is not representative*.

```python
from utils import onehot
model.eval()

z = Variable(torch.randn(16, 32))

# Generate a batch of 5s
y = Variable(onehot(10)(5).repeat(16, 1))

x_mu = model.sample(z, y)
```

```python
f, axarr = plt.subplots(1, 16, figsize=(18, 12))

samples = x_mu.data.view(-1, 28, 28).numpy()

for i, ax in enumerate(axarr.flat):
    ax.imshow(samples[i])
    ax.axis("off")
```

examples/notebooks/HIAuxiliary Deep Generative Model.ipynb


$$ \LaTeX \text{ command declarations here.}
\newcommand{\R}{\mathbb{R}}
\renewcommand{\vec}[1]{\mathbf{#1}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\D}{\mathcal{D}}
\newcommand{\G}{\mathcal{G}}
\newcommand{\L}{\mathcal{L}}
\newcommand{\X}{\mathcal{X}}
\newcommand{\Parents}{\mathrm{Parents}}
\newcommand{\NonDesc}{\mathrm{NonDesc}}
\newcommand{\I}{\mathcal{I}}
\newcommand{\dsep}{\text{d-sep}}
\newcommand{\Cat}{\mathrm{Categorical}}
\newcommand{\Bin}{\mathrm{Binomial}}
$$

```python
from __future__ import division

# scientific
%matplotlib inline
from matplotlib import pyplot as plt;
import matplotlib as mpl;
import numpy as np;
import sklearn as skl;
import sklearn.datasets;
import sklearn.cluster;
import sklearn.mixture;

# ipython
import IPython;

# python
import os;
import random;

#####################################################

# image processing
import PIL;

# trim and scale images
def trim(im, percent=100):
    print("trim:", percent);
    bg = PIL.Image.new(im.mode, im.size, im.getpixel((0,0)))
    diff = PIL.ImageChops.difference(im, bg)
    diff = PIL.ImageChops.add(diff, diff, 2.0, -100)
    bbox = diff.getbbox()
    if bbox:
        x = im.crop(bbox)
        return x.resize(((x.size[0]*percent)//100, (x.size[1]*percent)//100), PIL.Image.ANTIALIAS);

#####################################################

# daft (rendering PGMs)
import daft;

# set to FALSE to load PGMs from static images
RENDER_PGMS = True;

# decorator for pgm rendering
def pgm_render(pgm_func):
    def render_func(path, percent=100, render=None, *args, **kwargs):
        print("render_func:", percent);
        # render
        render = render if (render is not None) else RENDER_PGMS;

        if render:
            print("rendering");
            # render
            pgm = pgm_func(*args, **kwargs);
            pgm.render();
            pgm.figure.savefig(path, dpi=300);

            # trim
            img = trim(PIL.Image.open(path), percent);
            img.save(path, 'PNG');
        else:
            print("not rendering");

        # error
        if not os.path.isfile(path):
            raise Exception("Error:  Graphical model image %s not found.  You may need to set RENDER_PGMS=True." % path);

        # display
        return IPython.display.Image(filename=path);#trim(PIL.Image.open(path), percent);

    return render_func;

######################################################
```

# EECS 445:  Machine Learning

## Lecture 19: Baum-Welch Algorithm, and Inference in PGMs

- Instructor:  **Jacob Abernethy**
- Date:  November 21, 2016

*Lecture Exposition:*  Benjamin Bray

## References

- **[MLAPP]** Murphy, Kevin. [*Machine Learning:  A Probabilistic Perspective*](https://mitpress.mit.edu/books/machine-learning-0).  2012.
- **[PRML]** Bishop, Christopher.  [*Pattern Recognition and Machine Learning*](http://research.microsoft.com/en-us/um/people/cmbishop/prml/).  2006.
- **[Koller & Friedman 2009]** Koller, Daphne and Nir Friedman.  [*Probabilistic Graphical Models*](https://mitpress.mit.edu/books/probabilistic-graphical-models)

# **Review**: Hidden Markov Models

> Uses material from **[MLAPP]**

### Hidden Markov Models

Noisy observations $X_k$ generated from *discrete* hidden Markov chain $Z_k$.
$$
P(\vec{X}, \vec{Z}) = P(Z_1) P(X_1 \mid Z_1) \prod_{k=2}^T P(Z_k \mid Z_{k-1}) P(X_k \mid Z_k)
$$

```python
@pgm_render
def pgm_hmm():
    pgm = daft.PGM([7, 7], origin=[0, 0])

    # Nodes
    pgm.add_node(daft.Node("Z1", r"$Z_1$", 1, 3.5))
    pgm.add_node(daft.Node("Z2", r"$Z_2$", 2, 3.5))
    pgm.add_node(daft.Node("Z3", r"$\dots$", 3, 3.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("Z4", r"$Z_T$", 4, 3.5))

    pgm.add_node(daft.Node("x1", r"$X_1$", 1, 2.5, observed=True))
    pgm.add_node(daft.Node("x2", r"$X_2$", 2, 2.5, observed=True))
    pgm.add_node(daft.Node("x3", r"$\dots$", 3, 2.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("x4", r"$X_T$", 4, 2.5, observed=True))

    # Add in the edges.
    pgm.add_edge("Z1", "Z2", head_length=0.08)
    pgm.add_edge("Z2", "Z3", head_length=0.08)
    pgm.add_edge("Z3", "Z4", head_length=0.08)

    pgm.add_edge("Z1", "x1", head_length=0.08)
    pgm.add_edge("Z2", "x2", head_length=0.08)
    pgm.add_edge("Z4", "x4", head_length=0.08)

    return pgm;
```

```python
%%capture
pgm_hmm("images/pgm/hmm.png")
```

### HMM:  Parameters

For a Hidden Markov Model with $N$ hidden states and $M$ observed states, there are three *row-stochastic* parameters $\theta=(A,B,\pi)$,
- Transition matrix  $A \in \R^{N \times N}$
    $$
    A_{ij} = p(Z_t = j | Z_{t-1} = i)
    $$
- Emission matrix $B \in \R^{N \times M}$
    $$
    B_{jk} = p(X_t = k | Z_t = j)
    $$
- Initial distribution $\pi \in \R^N$,
    $$
    \pi_j = p(Z_1 = j)
    $$

### HMM:  Filtering Problem

**Filtering** means to compute the current *belief state* $p(z_t | x_1, \dots, x_t,\theta)$.
    $$
    p(z_t | x_1,\dots,x_t) = \frac{p(x_1,\dots,x_t,z_t)}{p(x_1,\dots,x_t)}
    $$
- Given observations $x_{1:t}$ so far, infer $z_t$.
- *Example*:  Estimate robot position given previous sensor readings.

> Solved by the **forward algorithm**.

## How do we infer values of hidden variables?

- One of the most challenging part of HMMs is to try to "predict" what are the values of the hidden variables $z_t$, having observed all the $x_1, \ldots, x_T$.
- Computing $p(z_t \mid \X)$ is known on *smoothing*. More on this soon.
- But it turns out that this probability can be computed from two other quantities:
    - $p(x_1,\dots,x_t,z_t)$, which we are going to label $\alpha_t(z_t)$
    - $p(x_{t+1},\dots,x_{T} | z_t)$, which we are going to label $\beta_t(z_t)$

### HMM:  Forward Algorithm

The **forward algorithm** computes $\alpha_t(z_t) \equiv p(x_1,\dots,x_t,z_t)$.

> Looks kinda hard to compute this probability, there are a lot of variables! Maybe it can be computed recursively? **Yes!**

$$
    \begin{align}
    \alpha_t(z_t)
    &= \sum_{z_{t-1}} p(x_1, \dots, x_t, z_{t-1}, z_t) \\
    &= \sum_{z_{t-1}} p(x_1, \dots, x_{t-1}, z_{t-1}) p(z_t | z_{t-1}) p(x_t | z_t) \\
    &= p(x_t | z_t) \sum_{z_{t-1}} \alpha_{t-1}(z_{t-1}) p(z_t | z_{t-1})\\
    &= B_{z_t,x_t} \sum_{z_{t-1}} \alpha_{t-1}(z_{t-1}) A_{z_{t-1}, z_t}
    \end{align}
$$

> Recursion starts from the *front* of the chain.
> (Suggestion: you need to work out the above recursion on your own, it's a really important exercise)

### HMM:  Backward Algorithm

The **backward algorithm** computes $\beta_t(z_t) \equiv p(x_{t+1},\dots,x_{T} | z_t)$,
> Again, maybe we can try to compute this recursively?

$$
    \begin{align}
    \beta(z_t)
    &= \sum_{z_{t+1}} p(x_{t+1},\dots,x_{T},z_{t+1} | z_t) \\
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}, \dots, x_T |z_{t+1}, z_t)\\
        (\text{since } x_{t+1:T} \perp z_t \mid z_{t+1}) \quad
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}, \dots, x_T |z_{t+1})\\
        (\text{since } x_{t+2:T} \perp x_{t+1} \mid z_{t+1}) \quad
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}|z_{t+1})  p(x_{t+2}, \dots, x_T |z_{t+1})\\
    &= \sum_{z_{t+1}} p(z_{t+1}|z_t) p(x_{t+1}|z_{t+1}) \beta_{t+1}(z_{t+1}) \\
    &= \sum_{z_{t+1}} A_{z_t, z_{t+1}} B_{z_{t+1}, x_{t+1}} \beta_{t+1}(z_{t+1})
    \end{align}
$$

> Recursion starts from the *back* of the chain.
> (Suggestion: you need to work out the above recursion on your own, it's a really important exercise)

### HMM:  Smoothing Problem

Compute $p(z_t | \X)$ offline, given all observations.
- Retroactively infer $z_t$.  (Hindsight!)

We can break the chain into two parts, the *past* and *future*:
    $$
    \begin{align}
    p(z_t | \X)
    &= p(x_{1:t}, z_t, x_{t+1:T}) \frac{1}{p(\X)} \\
    &= p(x_{1:t}, z_t) p(x_{t+1:T} | x_{1:t}, z_t)\frac{1}{p(\X)} \\
    &= p(x_{1:t}, z_t) p(x_{t+1:T} | z_t)\frac{1}{p(\X)}
    \end{align}
    $$
**Exercise**: How did we get that last line?

### HMM:  Smoothing Problem

Overall, the **smoothing problem** is to compute
$$
\begin{align*}
    \gamma_t(j) \equiv p(z_t = j \mid \X)
    & = \frac{p(\X \mid z_t = j)p(z_t = j)}{p(\X)} \\
    & = \frac{p(\X_{1:t} \mid z_t = j) p(\X_{t+1:T} | z_t=j) p(z_t = j)}{p(\X)} \\
    & = \frac{\alpha_t(j) \beta_t(j)}{p(\X)} = \frac{\alpha_t(j) \beta_t(j)}{\sum_k \alpha_t(k)\beta_t(k)}
\end{align*}
$$
It is an easy exercise to check that, for any $t$, $p(\X) = \sum_k \alpha_t(k)\beta_t(k)$.

We solve this via the **forward-backward algorithm**, where
- $\alpha_t(z_t) \equiv p(x_1,\dots,x_t,z_t)$ is found with the **forward algorithm**
- $\beta_t(z_t) \equiv p(x_{t+1},\dots,x_{T} | z_t)$ is found with the **backward algorithm**

### HMM:  Decoding Problem

**Decoding** computes the most probable state sequence, given observations.
    $$
    \vec{z}^* = \arg\max_{z_1,\dots,z_T} p(z_1,\dots,z_T | x_1, \dots, x_T, \theta)
    $$

> The decoding problem is solved by the **Viterbi algorithm**, which uses dynamic programming.  See **[PRML]** or **[MLAPP]** for more details.

### Viterbi Algorithm

* Viterbi is another recursive procedure for computing the most likely sequence of hidden states.
* We define $V_t(z_t)$ to be the probability of the most likely sequence of states up to time $t$ *that ended in state* $z_t$, given observed data
* These $V(z_t)$ values satisfies the recursion:

\begin{align}
V_1(j) & = p(x_1 \mid z_1=j) = B_{x_1, j}\\
V_{t+1}(j) & = p(x_{t+1} \mid z_{t+1} = j) \max_{k} \bigg\{ V_{t}(k) p(z_{t+1} = j \mid z_t = k) \bigg\}\\
& = B_{x_{t+1},j} \max_{k} \bigg\{ V_{t}(k) A_{j,k} \bigg\}
\end{align}

* Can be computed very quickly!

### HMM:  Part-of-Speech Tagging

In English, some words can have multiple parts of speech.  For instance,

- Business is going **well** (*Adverb*).
- All is **well** with us (*Adjective*).
- **Well**, who would have thought he could do it? (*Interjection*)
- The **well** was drilled fifty meters deep. (*Noun*)
- Tears **well** up in my eyes. (*Verb*)

(Example taken from [here](http://english.stackexchange.com/questions/46277/what-word-can-fulfill-the-most-parts-of-speech).)

### HMM:  Part-of-Speech Tagging

We can use a Hidden Markov Model to **disambiguate** the part of speech using context clues!
- Hidden states $z_t$ are parts-of-speech
- Observed states $x_t$ are words

Certain sequences of POS tags are unlikely.  This allows us to infer the correct tags!

## EM for HMMs: The Baum-Welch Algorithm

### HMM:  Learning Problem

It is usually necessary to **learn** the model parameters $\theta = (A,B,\pi)$ from data.
- Given observations $\mathcal{X} = \{ x_1, \dots, x_T \}$
- Given model dimensions $N$ and $M$
- Find parameters, i.e. the matrices $A$ & $B$, and the vector $\pi$ that best fit the data.

> The learning problem is solved by the **Baum-Welch algorithm**, a special case of expectation maximization.

### Recall:  Expectation-Maximization

**E-Step:** Write down an expression for
    $$
    Q(\theta_t, \theta) = E_{q}[ \log p(\X,Z | \theta) ] \quad \quad \text{ where } q = q(\cdot | \theta_t)
    $$

**M-Step:** Maximize the auxiliary function,
    $$
    \theta_{t+1}
    = \arg\max_\theta Q(\theta_t, \theta)
    $$

> Recall $q_t(Z) = p(Z|\X,\theta_t)$

### HMM:  Complete-Data Log-Likelihood

The joint likelihood of the hidden and observed states is

$$
\begin{align}
\log p(x_{1:T}, z_{1:T} | \theta)
&= \log \left[ p(z_1|\pi)
    \prod_{t=2}^T p(z_t | z_{t-1}, A)
    \prod_{t=1}^T p(x_t | z_t, B) \right] \\
&=  \log p(z_1|\pi)
    + \sum_{t=2}^T \log p(z_t | z_{t-1}, A) \\
    & \qquad \qquad + \sum_{t=1}^T \log p(x_t | z_t, B)
\end{align}
$$

### HMM:  Complete-Data Log-Likelihood

Each term of the complete-data log-likelihood is:

$$
\begin{align}
\log p(z_1 | \pi)
    &= \sum_{j=1}^N \mathbb{I}(z_1=j) \log \pi_j \\
\log p(z_t | z_{t-1}, A)
    &= \sum_{i=1}^N \sum_{j=1}^N
        \mathbb{I}(z_{t-1}=i)\mathbb{I}(z_t=j) \log A_{ij} \\
\log p(x_t | z_t, B)
    &= \sum_{j=1}^N
        \mathbb{I}(z_t=j) \log B_{j,x_t}
\end{align}
$$

### HMM:  Expected Complete Likelihood

The expected complete likelihood $Q(\theta_t, \theta)$ is

$$
\begin{align}
Q(\theta_t,\theta)
&= E_q[ \log p(\X,Z|\theta) ] \\
&= E_q[ \log p(z_1|\pi) ]
    + E_q\left[ \sum_{t=2}^T \log p(z_t | z_{t-1}, A) \right]\\
    &\qquad + E_q\left[ \sum_{t=1}^T \log p(x_t | z_t, B) \right]
\end{align}
$$

### HMM:  Expected Complete Likelihood

Fixing $t > 1$, and taking expectations with respect to $q(Z) = p(Z|\X,\theta)$,

$$
\begin{align*}
E_q[\log p(z_1 | \pi)]
    &= \sum_{j=1}^N q(z_1=j) \log \pi_j \\
E_q[\log p(z_t | z_{t-1}, A)]
    &= \sum_{i=1}^N \sum_{j=1}^N
        q(z_{t-1}=i, z_t=j) \log A_{ij} \\
E_q[\log p(x_t | z_t, B)]
    &= \sum_{j=1}^N
        q(z_t=j) \log B_{j, x_t}
\end{align*}
$$

### HMM:  Baum-Welch

- The **E-Step** consists of computing the $q$ terms from the previous slide, which can all be computed using the **forward-backward algorithm**!

- The **M-Step** consists of normalizing the expected transition and emission counts
    - Similar to MLE for complete data
    - Requires some careful calculations. See details on next slide.

You must compute the key quantities for $q()$ using the forward-backward algorithm. In the E step you can assume the parameters $\theta = (A, B, \pi)$, so we'll drop dependence on $\theta$.
1. First note that $q(z_t=j) = p(z_t = j \mid \X, \theta) = \gamma_t(j)$ which we computed above.
1. We then compute $q(z_{t-1}=i, z_t=j) = p(z_{t-1}=i, z_t=j | \X, \theta)$.
$$
\begin{align*}
    p(z_{t-1}, z_t | \X, \theta)
    & = \frac{p(\X \mid z_t, z_{t-1})p(z_t, z_{t-1})}{p(\X)} \\
    & = \frac{p(\X_{1:t-1} \mid z_t, z_{t-1}) p(\X_{t:T} | z_t, z_{t-1}) p(z_t, z_{t-1})}{p(\X)} \\
    & = \frac{p(\X_{1:t-1} \mid z_{t-1}) p(\X_{t:T} | z_t) p(z_t, z_{t-1})}{p(\X)} \\
    & = \frac{p(\X_{1:t-1}, z_{t-1}) p(x_t \mid z_t) p(\X_{t+1:T} | z_t) p(z_t, z_{t-1})}{p(z_{t-1})p(\X)} \\
    & = \frac{\alpha_{t-1}(z_{t-1}) \beta_t(z_t) p(z_t \mid z_{t-1}) p(x_t \mid z_t)}{p(\X)} \\
    & = \frac{\alpha_{t-1}(z_{t-1}) \beta_t(z_t) A_{z_{t-1}, z_t} B_{z_t, x_t}}{\sum_k \alpha_t(k)\beta_t(k)}
\end{align*}
$$

## Break Time!

# Graphical Models: Wrap-Up

We've just barely scratched the surface of what **probabilistic graphical models** have to offer.
- Bayesian Networks
- Mixture Models
- Hidden Markov Models
- Latent Variable Models & Expectation-Maximization

This lecture will be a grab-bag of some of the things we missed.

> Accordingly, we won't test you on the details.

### Graphical Models:  Related Courses

These courses are being offered next semester:
- **IOE 591**, [Bayesian Data Analysis](https://d1b10bmlvqabco.cloudfront.net/attach/iizaobacnl55nb/hku8ttycli2/ilxshlclzteu/IOE591_flyer.pdf)
- **EECS 592**, Advanced Artificial Intelligence

The following courses have been offered in the past:
- **STATS 601**, Analysis of Multivariate and Categorical Data
- **EECS 598**, [Probabilistic Graphical Models in Vision and Beyond](http://web.eecs.umich.edu/~jjcorso/t/598W15/)
- **STATS 700**, [Probabilistic Graphical Models](http://dept.stat.lsa.umich.edu/~xuanlong/courses/stat700-f09/)

### Graphical Models:  Other Resources

- **[Koller & Friedman 2009]** Koller, Daphne and Nir Friedman.  [*Probabilistic Graphical Models*](https://mitpress.mit.edu/books/probabilistic-graphical-models)
- **[Wainwright & Jordan 2008]** Wainwright, Martin, and Michael Jordan.  [Graphical Models, Exponential Families, and Variational Inference](https://www.eecs.berkeley.edu/~wainwrig/Papers/WaiJor08_FTML.pdf).  2008
- **Coursera,** [Probabilistic Graphical Models](https://www.coursera.org/course/pgm)

## Outline

- Undirected Graphical Models
    - Markov Random Fields
- Summary of Exact Inference
- Approximate Inference
    - Gibbs Sampling
    - Variational Inference
- Applications
    - Latent Dirichlet Allocation

# Undirected Graphical Models

> Uses material from **[MLAPP]** Chapter 19 and **[Koller & Friedman 2009]** Chapter 4

---

> Also known as **Markov Random Field**

### Markov Random Fields:  Motivation

So far, we have talked only about **directed models**.
- Directed edges represent an underlying generative process.
- Formalizes independence properties between variables

In some cases, **undirected models** are more natural.
- In many applications, it's hard to assign a direction to the interactions between variables.
- Offer a different, sometimes simpler, perspective

### Markov Random Fields:  Motivation

Consider the problem of modeling pixels in an image.
- One random variable $X_{ij}$ per pixel $(i,j)$ in the image.
- Intensity values of neighboring pixels are *locally* correlated

How can we represent this situation as a Bayesian network?
- Which directions do the arrows go?
- Which independence properties are desirable?

### Markov Random Fields:  Motivation

We could arbitrarily add directed edges, but this leads to strange independence structures.
- Below, $X_8$ is independent of all other variables given the colored nodes.

(Figure 19.1a from **[MLAPP]**)

### Markov Random Fields:  Motivation

Since images have no natural "directionality", why bother with undirected edges?
- Now, $X_8$ is independent of all other variables given its immediate neighbors.

(Figure 19.1b from **[MLAPP]**)

### Markov Random Fields:  Parameterization

Directed models come with a natural topological ordering.
- Applying the chain rule yields a factorization.
- It is only necessary to store **conditional probability tables**

Undirected models have no natural ordering!
- Instead, we describe the model in terms of **factors** describing the local interactions between variables.

### Markov Random Fields:  Factors

Let $\mathbf{D}$ be a set of random variables.  A **factor** is a function $\phi : \mathrm{Val}(\mathbf{D}) \mapsto \R$.
- where $\mathrm{Val}(\mathbf{D})$ is the set of possible configurations of the variables.
- where $\mathbf{D}$ is called the *scope* of $\phi$

> A factor assigns a real number to every possible configuration of the variables $\mathbf{D}$

### Markov Random Fields:  Factors

Probability distributions are *normalized* factors.
- $P(X_1,X_2,X_3)$ is a factor on $\mathbf{D}=(X_1,X_2,X_3)$
- $P(X | Y)$ is a factor on $\mathbf{D}=(X,Y)$

Other factors need not be normalized.  For example, $\phi(A,B)$ assigns high *affinity* to configurations where $A$ and $B$ agree:

$A$	$B$	$\phi(A,B)$

0	0	100

0	1	-50

1	0	-50

1	1	100

### Markov Random Fields:  Parameterization

Given a set of factors $\Phi=(\phi_1(\mathbf{D}_1), \phi_2(\mathbf{D}_2), \dots, \phi_K(\mathbf{D}_K))$, we can define a **Gibbs Distribution**,
    $$
    \begin{align}
    P_\Phi(X_1,\dots,X_n)
    &= \frac{1}{Z} \phi_1(\mathbf{D}_1) \phi_2(\mathbf{D}_2) \cdots \phi_K(\mathbf{D}_K) \\
    &= \frac{1}{Z} \exp\left[ \sum_{k=1}^K \log \phi_k(\mathbf{D}_k) \right]
    \end{align}
    $$
where $Z$ is a normalization constant.  Sometimes log-potentials are referred to as **energies** associated with each configuration of variables.

> This model was stolen from physicists :)

### Markov Random Fields:  Texture Modeling

Suppose we have a **pairwise Markov Random Field** over binary pixels.
- *Binary:* Each pixel is either $-1$ (black) or $1$ (white)
- *Pairwise:* Only model interactions between directly neighboring pixels.

Define the **energy function**, summing over all edges $(i,j)$:
$$
U(\X) = -\beta \sum_{(i,j)} x_i x_j \qquad (\beta > 0)
$$

### Markov Random Fields:  Texture Modeling

Then, the probability of an image is
$$
P(\X)
= \frac{1}{Z} \exp[-U(\X)]
= \frac{1}{Z} \exp\left[-\beta \sum_{(i,j)} x_i x_j \right]
$$

The *affinity parameter* $\beta > 0$ determines how much we want neighboring pixels to agree with each other.
- Low energy = High probability = Neighbors Agree
- High energy = Low probability = Neighbors Disagree

### Markov Random Fields:  Texture Modeling

Using Gibbs Sampling (later!) we can generate image samples! (low $\beta$)

### Markov Random Fields:  Texture Modeling

Using Gibbs Sampling (later!) we can generate image samples! (high $\beta$)

### Markov Random Fields:  Texture Modeling

We can also model more complex interactions between pixels.
- Force vertical neighbors to agree, horizontal neighbors to disagree.
- Rather than taking direct neighbors, look at a *window* around each pixel.
- Learn different $\beta$ values for different regions of the image.

### Markov Random Fields:  Texture Synthesis

We can learn to sample large textures from a small starting seed:

(Example taken from Efros & Leung, [Texture Synthesis by Non-Parametric Sampling](http://ieeexplore.ieee.org.proxy.lib.umich.edu/stamp/stamp.jsp?tp=&arnumber=790383))

### Markov Random Fields:  Texture Synthesis

The same algorithm can even be applied to text!

(Example taken from Efros & Leung, [Texture Synthesis by Non-Parametric Sampling](http://ieeexplore.ieee.org.proxy.lib.umich.edu/stamp/stamp.jsp?tp=&arnumber=790383))

### Undirected Models:  Other Applications

Undirected models are incredibly useful for probabilistic modeling.
- Image Processing
    - Segmentation, Denoising
- Natural Language Processing
    - Named entity recognition
    - Parsing
    - Computational Humor
- Maximum entropy models (related to exponential family)
- Conditional random fields (discriminative model)
- Deep learning

# Inference in Graphical Models

> Uses material from **[MLAPP]** Chapters 20, 21

### Inference in Graphical Models

**Inference**:  Estimate hidden variables $Z$ from observed variables $X$.
$$
P(Z | X,\theta) = \frac{P(X,Z | \theta)}{P(X|\theta)}
$$

- Denominator $P(X|\theta)$ is sometimes called the probability of the **evidence**.
- Occasionally we care only about a subset of the hidden variables, and marginalize out the rest.

### Inference:  Examples

We have already described the inference procedure for simple models:

- Expectation Maximization
- Mixture Models
- Hidden Markov Models

> Given an arbitrary directed or undirected graph, can we systematically perform inference?

## Exact Inference

> Uses material from **[MLAPP]** Chapter 20

### Exact Inference

We can recover $P(Z | X,\theta)$ exactly when the hidden variables $Z$ are **discrete**.
- Belief Propagation
- Variable Elimination
- Junction Tree

Many exact inference algorithms can be seen as a generalization of the **fowards-backwards algorithm** for HMMs.
> Unfortunately, they have undesirable time and space complexity properties.

### Exact Inference:  HMMs

Recall that a **Hidden Markov Model** is a chain-structured directed model:

```python
@pgm_render
def pgm_hmm():
    pgm = daft.PGM([7, 7], origin=[0, 0])

    # Nodes
    pgm.add_node(daft.Node("Z1", r"$Z_1$", 1, 3.5))
    pgm.add_node(daft.Node("Z2", r"$Z_2$", 2, 3.5))
    pgm.add_node(daft.Node("Z3", r"$\dots$", 3, 3.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("Z4", r"$Z_T$", 4, 3.5))

    pgm.add_node(daft.Node("x1", r"$X_1$", 1, 2.5, observed=True))
    pgm.add_node(daft.Node("x2", r"$X_2$", 2, 2.5, observed=True))
    pgm.add_node(daft.Node("x3", r"$\dots$", 3, 2.5, plot_params={'ec':'none'}))
    pgm.add_node(daft.Node("x4", r"$X_T$", 4, 2.5, observed=True))

    # Add in the edges.
    pgm.add_edge("Z1", "Z2", head_length=0.08)
    pgm.add_edge("Z2", "Z3", head_length=0.08)
    pgm.add_edge("Z3", "Z4", head_length=0.08)

    pgm.add_edge("Z1", "x1", head_length=0.08)
    pgm.add_edge("Z2", "x2", head_length=0.08)
    pgm.add_edge("Z4", "x4", head_length=0.08)

    return pgm;
```

```python
%%capture
pgm_hmm("images/pgm/hmm.png")
```

### Exact Inference:  HMMs

We used the **fowards-backwards algorithm** to find $P(Z_t | \X,\theta)$.
- We split the graph at $Z_t$
- Performed the **foward algorithm** to propagate information from the start state *forwards* to $Z_t$
- Performed the **backward algorithm** to propagate information *backwards* to $Z_t$

### Belief Propagation:  From Chains to Trees

The **Belief Propagation** algorithm generalizes the forwards-backwards algorithm from chains to tree-structured graphs.
- *Idea:*  Every path from the root node to a leaf node is a chain!

> Let's do some hand-waving...

### Belief Propagation:  Collect-to-Root Phase

Much like before, we split the tree at node $t$ and propagate evidence upwards:

(Figure 20.1a from **[MLAPP]**)

### Belief Propagation:  Distribute-from-Root Phase

Next, we propagate the collected evidence back down the tree:

(Figure 20.1b from **[MLAPP]**)

### Junction Tree:  From Trees to Graphs

The **Junction Tree** algorithm generalizes Belief Propagation from trees to arbitary graphs.

> See **[MLAPP]** or **[Koller & Friedman 2009]** for details.

### Exact Inference:  Summary

**Omitting some slides from lecture** -- but still available on notes

Exact inference is possible in discrete graphical models.
- Unfortunately, algorithms to do so are ugly and complicated.
- Moreover, exact inference is NP-Hard, so these algorithms are very inefficient.
- Incapable of dealing with general, continuous distributions.

Fortunately, **approximate inference** works well in practice.

## Approximate Inference

> Uses material from **[MLAPP]** Chapter 21 and [MLSS 2009](http://videolectures.net/site/normal_dl/tag=50736/mlss09uk_murray_mcmc.pdf)

### Approximate Inference:  Intro

Exact inference is too difficult, so we turn to **approximate inference**.
- Theory is lacking--no guarantees about accuracy or runtime
- However, they seem to work well in practice.
- Capable of handling continuous variables!

The two most common methods are
- **Markov Chain Monte Carlo:** Estimate $P(Z|X,\theta)$ by sampling from it!
- **Variational Inference:** Cast inference as a deterministic optimization problem.

### Integration is Hard

In general, we want to find $p(X|Y)$ for some variables $X$ and $Y$.  Using Bayes' Rule,
    $$p(y|x)= \frac{p(x|y)p(y)}{p(x)}= \frac{p(x|y)p(y)}{\int_Y p(x|y)p(y)\, dy}$$
Usually, the hard part is evaluating the normalization integral,
    $$
    \int_Y p(x|y)p(y)\, dy
    $$

> **Key Point:** Many inference problems can be reduced to the problem of *evaluating an integral*.

### Monte Carlo Integration

Suppose we wish to evaluate $\int_X f(x) p(x) \, dx$, where

- $f(x)$ is an arbitrary function
- $p(x)$ is a probability distribution

We can express this as an expectation, with $X \sim p$,
    $$
    \int_{X \sim p} f(x) p(x) \, dx = E_p[f(X)]
    $$

### Monte Carlo Integration

We can approximate any expectation by averaging over samples $X_1,\dots,X_N \stackrel{iid}{\sim} p(X)$,
    $$
    \int_X f(x) p(x) \, dx
    = E_p[f(X)] \approx \bar{f}_N
    = \frac{1}{N} \sum_{n=1}^N f(x_n)
    $$

> **Key Point:** We can reduce any integration problem to a *sampling problem*.

### Example:  Dumb Approximation of $\pi$

We can express $\pi$ as an integral over the unit circle $C$, which has area $\pi r^2 = \pi$.

$$
\pi
= \int_{\R^2} \mathbb{I}_C(x) \, dx
= \int_{[-1,1]\times[-1,1]} \mathbb{I}_C(x) \,dx
$$

Let $p$ be the uniform distribution on the square of size $2$.
$$
p(x)=\begin{cases} \frac{1}{4} & x \in [-1,1] \times [-1,1] \\
0 & \text{ otherwise.}
\end{cases}
$$

Then,
$$
\pi
= 4\; \int_{[-1,1]\times[-1,1]} \mathbb{I}_C(x) p(x) \,dx \approx E_p[4\cdot \mathbb{I}_C(X) ]
$$

### Example:  Dumb Approximation of $\pi$

Therefore, we can estimate $\pi$ by drawing samples uniformly from the square $[-1,1] \times [-1,1]$ and computing the fraction of samples that fall within the unit circle $C$!
> This method is **not ideal** and requires tons of samples.

$$
\frac{\text{Area of Circle}}{\text{Area of Square}}
= \frac{\pi r^2}{4r^2} = \frac{\pi}{4}
$$

### Example:  Dumb Approximation of $\pi$

```python
def estimate_pi(n):
    # draw samples
    samples = np.random.random((2,n)) * 2 - 1;
    # separate samples inside/outside circle
    r = np.sum(samples**2, axis=0);
    inside = samples[:,r < 1];
    outside = samples[:,r >= 1];
    # estimate pi
    return inside.shape[1] / n * 4;

def estimate_pi_plot(n):
    # draw samples
    samples = np.random.random((2,n)) * 2 - 1;
    # separate samples inside/outside circle
    r = np.sum(samples**2, axis=0);
    inside = samples[:,r < 1];
    outside = samples[:,r >= 1];
    # estimate pi
    pi_estimate = inside.shape[1] / n * 4;
    print("pi estimated to be %.5f" % pi_estimate)

    # plot
    plt.figure(figsize=(6,6))
    plt.plot(inside[0,:], inside[1,:], "rx");
    plt.plot(outside[0,:], outside[1,:], "bx");
    plt.axis("equal");
    plt.title(r"$\pi \approx %0.2f$ from %d samples" % (pi_estimate,n));
```

```python
estimate_pi_plot(5000)
```

### Monte Carlo Integration:  Sampling

We can compute any integral $\int_X f(x)p(x) \, dx$ provided that we know how to sample from $p$.  Different methods have been devloped to sample from arbitrary distributions:
- Sampling from Inverse CDF
- Rejection Sampling
- Importance Sampling
- Particle Filtering

The most popular is **Markov Chain Monte Carlo**, which beats out the other methods in high-dimensional spaces.

### Markov Chain Monte Carlo

**Goal:** Draw samples from $P(X_1,\dots,X_N)$.
- Usually, $N$ is large, so $P$ is high-dimensional.

**Idea:** Construct a Markov Chain on the state space $\X$ whose *stationary distribution* is the target density.

> We will perform a (biased) random walk on the state space, such that the fraction of time we spend at each state $(x_1,\dots,x_N)$ is proportional to $p(x_1,\dots,x_N)$

### Markov Chain Monte Carlo

(Figure taken from [MLSS 2009 Slides](http://videolectures.net/site/normal_dl/tag=50736/mlss09uk_murray_mcmc.pdf))

### MCMC:  Gibbs Sampling

Sample each variable in turn, conditioned on all the others.
- Assumes the **full conditionals** $p(x_k | \mathbf{x}_{-k})$ are known

Initialize $\mathbf{x}^{0}=(x_1^0, x_2^0, \dots, x_N^0)$ randomly.

For $k \in {1,\dots,N}$ in a random order,

Set $\mathbf{x}^{t+1} = \mathbf{x}^t$

Resample $x_k^{t+1} \sim p(x_k \mid \mathbf{x}_{-k}^t)$

### MCMC:  Gibbs Sampling

(Figure 11.11 from **[PRML]**)

### MCMC:  Summary

1. Inference can often be reduced to an integration problem.
2. Integration problems can be expressed as an expection.
3. Expectations can be approximated by averaging over samples.
4. **Inference can be reduced to a sampling problem.**

### MCMC:  Summary

Advantages:
- Easy to implement
- Works well in high dimensions
- Guaranteed to converge to the true distribution

Disadvantages:
- The Markov Chain may not **mix** fast enough
- Need lots of samples
- Convergence rate unknown

# Application:  Latent Dirichlet Allocation

> This was originally a homework problem, but you lucked out :)  See [**[Blei, Ng, and Jordan 2003]**](http://ai.stanford.edu/~ang/papers/nips01-lda.pdf) for details.

### Latent Dirichlet Allocation

**Latent Dirichlet Allocation** models documents as mixtures of latent *topics*.
- Makes the **bag-of-words** assumption.
- A **topic** is a probability distribution over words.
- Is a **mixed-membership** mixture model.  Each document expresses more than one topic with different *proportions*.

### LDA:  Generative Process

Given a set $\beta=(\beta_1,\dots,\beta_K)$ of $K$ topics and a fixed vocabulary, generate documents in the following way:

- For document $d=1,2, \dots$
    - Choose topic proportions $\theta_d \sim \mathrm{Dirichlet}(\alpha)$
    - For word position $n=1,2,\dots$,
        - Sample topic $z_{dn} \sim  \Cat(\theta_d)$
        - Sample word $w_{dn} \sim \Cat(\beta_{z_{dn}})$

### LDA:  Graphical Model

(Taken from [MLSS 2012 Slides](http://yosinski.com/mlss12/media/slides/MLSS-2012-Blei-Probabilistic-Topic-Models.pdf) by David Blei)

### LDA:  Intuitive View

(Taken from [MLSS 2012 Slides](http://yosinski.com/mlss12/media/slides/MLSS-2012-Blei-Probabilistic-Topic-Models.pdf) by David Blei)

### LDA:  Example of Learned Topics

(Taken from [MLSS 2012 Slides](http://yosinski.com/mlss12/media/slides/MLSS-2012-Blei-Probabilistic-Topic-Models.pdf) by David Blei)

### LDA:  Inference

Latent Dirichlet Allocation is arguably the simplest model that requires approximate inference.  It is often used as the "hello world" example when new inference techniques are introduced.

- Mean field variational inference
- Expectation propagation
- Collapsed Gibbs sampling
- Distributed sampling
- Collapsed variational inference
- Online variational inference
- Factorization based inference

Latent Dirichlet Allocation has **many, many** applications to areas other than text modeling.


>[Quantile regression from OLS to TensorFlow](#scrollTo=PQdJbWCS9N3G)

>>[Setup](#scrollTo=VjvK4vP2-X_D)

>>>[Graph options](#scrollTo=sm3RTN850S70)

>>>[Data](#scrollTo=vdaqXkSCe7gG)

>>>>[Load](#scrollTo=vdaqXkSCe7gG)

>>>>[Normalize](#scrollTo=xuphFNDqtEiW)

>>>>[Make DataFrames](#scrollTo=HDT3Cs7ptIYG)

>>>[Initialize result](#scrollTo=utzAT-gdBTqj)

>>[Quantile loss example](#scrollTo=OZ7dkNLfyZ3r)

>>[OLS](#scrollTo=6dRLoTDhCrpO)

>>[QuantReg](#scrollTo=esTAKyTyG1TS)

>>[Random forests](#scrollTo=qCt2S4V9Uepc)

>>[Gradient boosted trees](#scrollTo=6ssXXxsCA7FS)

>>[Keras](#scrollTo=H55y_74W97jH)

>>>[Quantiles 0.1, 0.5, 0.9](#scrollTo=SeMIPNsO-qxI)

>>[TensorFlow](#scrollTo=PEy2M6X89tbD)

>>[Visualize quantiles](#scrollTo=NG7Oqjp-7htm)

>>[Compare quantile loss](#scrollTo=42LIBXl933YI)

>>[All features](#scrollTo=ig_Wywm05sjA)

>>>[OLS](#scrollTo=ImnTiUft-Yhq)

>>>[Quantreg](#scrollTo=sC29BePy_dtj)

>>>[Random forests](#scrollTo=Z3g3cXQVDvBJ)

>>>[Gradient boosting](#scrollTo=kfHp4qvBDeA3)

>>>[Keras](#scrollTo=tLm8TjoV_qap)

>>>[TensorFlow](#scrollTo=G_lApuMnAoR9)

>>>[Compare](#scrollTo=hn3sxw4yDXdz)

# Quantile regression from OLS to TensorFlow
[An Awesome Medium Post Notebook:](https://towardsdatascience.com/quantile-regression-from-linear-models-to-trees-to-deep-learning-af3738b527c3)

Use the Boston housing dataset.

Based on blog posts using:
* Keras: https://towardsdatascience.com/deep-quantile-regression-c85481548b5a
* Tensorflow: https://towardsdatascience.com/deep-quantile-regression-in-tensorflow-1dbc792fe597
* statsmodels quantile regression: https://www.statsmodels.org/dev/examples/notebooks/generated/quantile_regression.html

## Setup

```
import pandas as pd
import numpy as np

import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

import statsmodels.api as sm

from scipy.stats import norm

from sklearn import ensemble

import tensorflow as tf
from tensorflow import keras
```

    /usr/local/lib/python3.6/dist-packages/statsmodels/compat/pandas.py:56: FutureWarning: The pandas.core.datetools module is deprecated and will be removed in a future version. Please use the pandas.tseries module instead.
      from pandas.core import datetools

Keras and TensorFlow options

```
EPOCHS = 200
BATCH_SIZE = 32
UNITS = 512
```

### Graph options

```
!wget https://github.com/MaxGhenis/random/raw/master/Roboto-Regular.ttf -P /usr/local/lib/python3.6/dist-packages/matplotlib/mpl-data/fonts/ttf
mpl.font_manager._rebuild()
```

    --2018-11-29 19:21:44--  https://github.com/MaxGhenis/random/raw/master/Roboto-Regular.ttf
    Resolving github.com (github.com)... 192.30.255.113, 192.30.255.112
    Connecting to github.com (github.com)|192.30.255.113|:443... connected.
    HTTP request sent, awaiting response... 302 Found
    Location: https://raw.githubusercontent.com/MaxGhenis/random/master/Roboto-Regular.ttf [following]
    --2018-11-29 19:21:45--  https://raw.githubusercontent.com/MaxGhenis/random/master/Roboto-Regular.ttf
    Resolving raw.githubusercontent.com (raw.githubusercontent.com)... 151.101.0.133, 151.101.64.133, 151.101.128.133, ...
    Connecting to raw.githubusercontent.com (raw.githubusercontent.com)|151.101.0.133|:443... connected.
    HTTP request sent, awaiting response... 200 OK
    Length: 145348 (142K) [application/octet-stream]
    Saving to: ‘/usr/local/lib/python3.6/dist-packages/matplotlib/mpl-data/fonts/ttf/Roboto-Regular.ttf.3’

    Roboto-Regular.ttf. 100%[===================>] 141.94K  --.-KB/s    in 0.02s

    2018-11-29 19:21:45 (6.36 MB/s) - ‘/usr/local/lib/python3.6/dist-packages/matplotlib/mpl-data/fonts/ttf/Roboto-Regular.ttf.3’ saved [145348/145348]

```
sns.set_style('white')
DPI = 200
mpl.rc('savefig', dpi=DPI)
mpl.rcParams['figure.dpi'] = DPI
mpl.rcParams['figure.figsize'] = 6.4, 4.8  # Default.
mpl.rcParams['font.sans-serif'] = 'Roboto'
mpl.rcParams['font.family'] = 'sans-serif'

# Set title text color to dark gray (https://material.io/color) not black.
TITLE_COLOR = '#212121'
mpl.rcParams['text.color'] = TITLE_COLOR

# Axis titles and tick marks are medium gray.
AXIS_COLOR = '#757575'
mpl.rcParams['axes.labelcolor'] = AXIS_COLOR
mpl.rcParams['xtick.color'] = AXIS_COLOR
mpl.rcParams['ytick.color'] = AXIS_COLOR
```

### Data

#### Load

```
boston_housing = keras.datasets.boston_housing

(train_data, train_labels), (test_data, test_labels) = \
    boston_housing.load_data()
```

```
train_data.shape
```

    (404, 13)

```
test_data.shape
```

    (102, 13)

#### Make DataFrames

```
column_names = ['CRIM', 'ZN', 'INDUS', 'CHAS', 'NOX', 'RM', 'AGE', 'DIS', 'RAD',
                'TAX', 'PTRATIO', 'B', 'LSTAT']

train_df = pd.DataFrame(train_data, columns=column_names)
test_df = pd.DataFrame(test_data, columns=column_names)
```

#### Normalize

```
train_df.describe()
```

      CRIM
      ZN
      INDUS
      CHAS
      NOX
      RM
      AGE
      DIS
      RAD
      TAX
      PTRATIO
      B
      LSTAT

      count
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000
      404.000000

      mean
      3.745111
      11.480198
      11.104431
      0.061881
      0.557356
      6.267082
      69.010644
      3.740271
      9.440594
      405.898515
      18.475990
      354.783168
      12.740817

      std
      9.240734
      23.767711
      6.811308
      0.241238
      0.117293
      0.709788
      27.940665
      2.030215
      8.698360
      166.374543
      2.200382
      94.111148
      7.254545

      min
      0.006320
      0.000000
      0.460000
      0.000000
      0.385000
      3.561000
      2.900000
      1.129600
      1.000000
      188.000000
      12.600000
      0.320000
      1.730000

      25%
      0.081437
      0.000000
      5.130000
      0.000000
      0.453000
      5.874750
      45.475000
      2.077100
      4.000000
      279.000000
      17.225000
      374.672500
      6.890000

      50%
      0.268880
      0.000000
      9.690000
      0.000000
      0.538000
      6.198500
      78.500000
      3.142300
      5.000000
      330.000000
      19.100000
      391.250000
      11.395000

      75%
      3.674808
      12.500000
      18.100000
      0.000000
      0.631000
      6.609000
      94.100000
      5.118000
      24.000000
      666.000000
      20.200000
      396.157500
      17.092500

      max
      88.976200
      100.000000
      27.740000
      1.000000
      0.871000
      8.725000
      100.000000
      10.710300
      24.000000
      711.000000
      22.000000
      396.900000
      37.970000

```
mean = train_df.mean(axis=0)
std = train_df.std(axis=0)
train_df = (train_df - mean) / std
test_df = (test_df - mean) / std
```

```
x_train = train_df.AGE
x_test = test_df.AGE
```

Denormalize.

```
def denorm_AGE(x):
  return x * std.AGE + mean.AGE

x_train_denorm = denorm_AGE(x_train)
x_test_denorm = denorm_AGE(x_test)
```

Reformat data for `statsmodels`.

```
X_train = sm.add_constant(x_train)
X_test = sm.add_constant(x_test)
```

Reformat data for `tensorflow`.

```
x_train_expanded = np.expand_dims(x_train, 1)
x_test_expanded = np.expand_dims(x_test, 1)
train_labels_expanded = np.expand_dims(train_labels, 1)
```

### Initialize result

Dataset per method, quantile, and `x` value.

```
METHODS = ['OLS', 'QuantReg', 'Random forests', 'Gradient boosting', 'Keras',
           'TensorFlow']

QUANTILES = [0.1, 0.3, 0.5, 0.7, 0.9]
# QUANTILES.reverse()  # Test out to see if we're getting different results.

quantiles_legend = [str(int(q * 100)) + 'th percentile' for q in QUANTILES]
```

```
# sns.set_palette(sns.color_palette('Blues', len(QUANTILES)))
sns.set_palette(sns.color_palette('Blues'))
# Set dots to a light gray
dot_color = sns.color_palette('coolwarm', 3)[1]
```

```
preds = np.array([(method, q, x)
                  for method in METHODS
                  for q in QUANTILES
                  for x in x_test])
preds = pd.DataFrame(preds)
preds.columns = ['method', 'q', 'x']
preds = preds.apply(lambda x: pd.to_numeric(x, errors='ignore'))
```

```
preds['label'] = np.resize(test_labels, preds.shape[0])
```

## Quantile loss example

```
# pandas version rather than Keras.
def quantile_loss(q, y, f):
  # q: Quantile to be evaluated, e.g., 0.5 for median.
  # y: True value.
  # f: Fitted or predicted value.
  e = y - f
  return np.maximum(q * e, (q - 1) * e)
```

```
quantile_loss_example_e = np.linspace(-1, 1, 1000)
quantile_loss_example_loss_10 = quantile_loss(0.1, 0, quantile_loss_example_e)
quantile_loss_example_loss_50 = quantile_loss(0.5, 0, quantile_loss_example_e)
quantile_loss_example_loss_90 = quantile_loss(0.9, 0, quantile_loss_example_e)
```

```
with sns.color_palette('Blues', 3):
  plt.plot(quantile_loss_example_e, quantile_loss_example_loss_10)
  plt.plot(quantile_loss_example_e, quantile_loss_example_loss_50)
  plt.plot(quantile_loss_example_e, quantile_loss_example_loss_90)
  plt.legend([str(int(q * 100)) + 'th percentile' for q in [0.1, 0.5, 0.9]])
  sns.despine(left=True, bottom=True)
  plt.xlabel('Error')
  plt.ylabel('Quantile loss')
  plt.title('Quantile loss by error and quantile', loc='left');
```

```
quantile_loss_example_q = np.linspace(0.01, 0.99, 99)
quantile_loss_example_loss_neg1 = quantile_loss(quantile_loss_example_q, 0, -1)
quantile_loss_example_loss_pos1 = quantile_loss(quantile_loss_example_q, 0, 1)
```

```
with sns.color_palette('Blues', 2):
  plt.plot(quantile_loss_example_q, quantile_loss_example_loss_neg1)
  plt.plot(quantile_loss_example_q, quantile_loss_example_loss_pos1)
  plt.legend(['Error of -1', 'Error of +1'])
  sns.despine(left=True, bottom=True)
  plt.xlabel('Quantile')
  plt.ylabel('Quantile loss')
  plt.title('Quantile loss by quantile and error', loc='left');
```

### Scatter plot of AGE

```
mean.AGE
```

    69.01064356435643

```
std.AGE
```

    27.940664897596555

```
from matplotlib.ticker import FuncFormatter

ax = plt.scatter(x_train_denorm, train_labels, color=dot_color)
plt.title('Age vs. value, Boston housing dataset (training slice)', loc='left')
sns.despine(left=True, bottom=True)
ax.axes.xaxis.set_major_formatter(FuncFormatter(
    lambda x, _: '{:.0%}'.format(x / 100)))
ax.axes.yaxis.set_major_formatter(FuncFormatter(
    lambda y, _: '${:.0f}k'.format(y)))
plt.xlabel('Proportion of owner-occupied units built prior to 1940')
plt.ylabel('Median value of owner-occupied homes')
plt.show()
```

## OLS

https://stackoverflow.com/questions/17559408/confidence-and-prediction-intervals-with-statsmodels

```
ols = sm.OLS(train_labels, X_train).fit()
```

```
def ols_quantile(m, X, q):
  # m: OLS model.
  # X: X matrix.
  # q: Quantile.
  #
  # Set alpha based on q. Vectorized for different values of q.
  mean_pred = m.predict(X)
  se = np.sqrt(m.scale)
  return mean_pred + norm.ppf(q) * se
```

```
preds.loc[preds.method == 'OLS', 'pred'] = np.concatenate(
    [ols_quantile(ols, X_test, q) for q in QUANTILES])
```

## QuantReg

https://www.statsmodels.org/dev/examples/notebooks/generated/quantile_regression.html

```
quantreg = sm.QuantReg(train_labels, X_train)  # Don't fit yet, since we'll fit once per quantile.
```

```
preds.loc[preds.method == 'QuantReg', 'pred'] = np.concatenate(
    [quantreg.fit(q=q).predict(X_test) for q in QUANTILES])
```

## Random forests

```
N_ESTIMATORS = 1000
rf = ensemble.RandomForestRegressor(n_estimators=N_ESTIMATORS,
                                    min_samples_leaf=1, random_state=3,
                                    verbose=True,
                                    n_jobs=-1)  # Use maximum number of cores.
rf.fit(X_train, train_labels)
```

    [Parallel(n_jobs=-1)]: Done  46 tasks      | elapsed:    0.0s
    [Parallel(n_jobs=-1)]: Done 196 tasks      | elapsed:    0.2s
    [Parallel(n_jobs=-1)]: Done 446 tasks      | elapsed:    0.4s
    [Parallel(n_jobs=-1)]: Done 796 tasks      | elapsed:    0.8s
    [Parallel(n_jobs=-1)]: Done 1000 out of 1000 | elapsed:    1.0s finished

    RandomForestRegressor(bootstrap=True, criterion='mse', max_depth=None,
               max_features='auto', max_leaf_nodes=None,
               min_impurity_decrease=0.0, min_impurity_split=None,
               min_samples_leaf=1, min_samples_split=2,
               min_weight_fraction_leaf=0.0, n_estimators=1000, n_jobs=-1,
               oob_score=False, random_state=3, verbose=True, warm_start=False)

```
def rf_quantile(m, X, q):
    rf_preds = []
    for estimator in m.estimators_:
        rf_preds.append(estimator.predict(X))
    rf_preds = np.array(rf_preds).transpose()  # One row per record.
    return np.percentile(rf_preds, q * 100, axis=1)
```

```
preds.loc[preds.method == 'Random forests', 'pred'] = np.concatenate(
    [rf_quantile(rf, X_test, q) for q in QUANTILES])
```

## Gradient boosted trees

```
def gb_quantile(X_train, train_labels, X, q):
  gbf = ensemble.GradientBoostingRegressor(loss='quantile', alpha=q,
                                           n_estimators=N_ESTIMATORS,
                                           max_depth=3,
                                           learning_rate=0.1, min_samples_leaf=9,
                                           min_samples_split=9)
  gbf.fit(X_train, train_labels)
  return gbf.predict(X)
```

```
preds.loc[preds.method == 'Gradient boosting', 'pred'] = np.concatenate(
    [gb_quantile(X_train, train_labels, X_test, q) for q in QUANTILES])
```

## Keras

From https://github.com/sachinruk/KerasQuantileModel/blob/master/Keras%20Quantile%20Model.ipynb

One area that Deep Learning has not explored extensively is the uncertainty in estimates. However, as far as decision making goes, most people actually require quantiles as opposed to true uncertainty in an estimate. eg. For a given age the weight of an individual will vary. What would be interesting is the (for arguments sake) the 10th and 90th percentile. The uncertainty of the estimate of an individuals weight is less interesting.

### Quantiles 0.1, 0.5, 0.9

The loss for an individual data point is defined as:
$$
\begin{align}
\mathcal{L}(\xi_i|\alpha)=\begin{cases}
\alpha \xi_i &\text{if }\xi_i\ge 0, \\
(\alpha-1) \xi_i &\text{if }\xi_i< 0.
\end{cases}
\end{align}
$$
where $\alpha$ is the required quantile and $\xi_i = y_i - f(\mathbf{x}_i)$ and, $f(\mathbf{x}_i)$ is the predicted (quantile) model. The final overall loss is defines as:
$$\mathcal{L}(\mathbf{y},\mathbf{f}|\alpha)=\frac{1}{N} \sum_{i=1}^N \mathcal{L}(y_i-f(\mathbf{x}_i)|\alpha)$$

The following function defines the loss function for a quantile model.

**Note**: The following 4 lines is ALL that you change in comparison to a normal Deep Learning method, i.e. The loss function is all that changes.

```
def tilted_loss(q, y, f):
    e = (y - f)
    return keras.backend.mean(keras.backend.maximum(q * e, (q - 1) * e),
                              axis=-1)
```

```
optimizer = tf.train.AdamOptimizer(0.001)
early_stop = keras.callbacks.EarlyStopping(monitor='val_loss', patience=20)
```

```
def keras_pred(x_train, train_labels, x_test, q):
    print(q)
    # Set input_dim for the number of features.
    if len(x_train.shape) == 1:
        input_dim = 1
    else:
        input_dim = x_train.shape[1]
    model = keras.Sequential([
      keras.layers.Dense(UNITS, activation=tf.nn.relu,
                         input_dim=input_dim),
      keras.layers.Dense(UNITS, activation=tf.nn.relu),
      keras.layers.Dense(1)
    ])

    model.compile(loss=lambda y, f: tilted_loss(q, y, f), optimizer=optimizer)
    model.fit(x_train, train_labels, epochs=EPOCHS, batch_size=BATCH_SIZE,
              verbose=0, validation_split=0.2, callbacks=[early_stop])

    # Predict the quantile
    return model.predict(x_test)
```

```
preds.loc[preds.method == 'Keras', 'pred'] = np.concatenate(
    [keras_pred(x_train_expanded, train_labels, x_test_expanded, q)
     for q in QUANTILES])
```

    0.1
    0.3
    0.5
    0.7
    0.9

## TensorFlow

Adapted from https://github.com/strongio/quantile-regression-tensorflow/blob/master/Quantile%20Loss.ipynb

```
# Initialize session
sess = tf.Session()
```

```
# Create network
class q_model:
    def __init__(self,
                 sess,
                 quantiles,
                 in_shape=1,
                 out_shape=1,
                 batch_size=32):

        self.sess = sess

        self.quantiles = quantiles
        self.num_quantiles = len(quantiles)

        self.in_shape = in_shape
        self.out_shape = out_shape
        self.batch_size = batch_size

        self.outputs = []
        self.losses = []
        self.loss_history = []

        self.build_model()

    def build_model(self, scope='q_model', reuse=tf.AUTO_REUSE):
        with tf.variable_scope(scope, reuse=reuse) as scope:
            self.x = tf.placeholder(tf.float32, shape=(None, self.in_shape))
            self.y = tf.placeholder(tf.float32, shape=(None, self.out_shape))

            self.layer0 = tf.layers.dense(self.x,
                                          units=UNITS,
                                          activation=tf.nn.relu)
            self.layer1 = tf.layers.dense(self.layer0,
                                          units=UNITS,
                                          activation=tf.nn.relu)

            # Create outputs and losses for all quantiles
            for i, q in enumerate(self.quantiles):
                # Get output layers
                output = tf.layers.dense(self.layer1, self.out_shape,
                                         name="{}_q{}".format(i, int(q * 100)))
                self.outputs.append(output)

                # Create losses
                error = tf.subtract(self.y, output)
                loss = tf.reduce_mean(tf.maximum(q * error, (q - 1) * error),
                                      axis=-1)

                self.losses.append(loss)

            # Create combined loss
            self.combined_loss = tf.reduce_mean(tf.add_n(self.losses))
            self.train_step = tf.train.AdamOptimizer().minimize(
                self.combined_loss)

    def fit(self, x, y, epochs=EPOCHS):
        for epoch in range(epochs):
            epoch_losses = []
            for idx in range(0, x.shape[0], self.batch_size):
                batch_x = x[idx : min(idx + self.batch_size, x.shape[0]), :]
                batch_y = y[idx : min(idx + self.batch_size, y.shape[0]), :]

                feed_dict = {self.x: batch_x,
                             self.y: batch_y}

                _, c_loss = self.sess.run([self.train_step, self.combined_loss],
                                          feed_dict)
                epoch_losses.append(c_loss)

            epoch_loss =  np.mean(epoch_losses)
            self.loss_history.append(epoch_loss)
            if epoch % 100 == 0:
                print("Epoch {}: {}".format(epoch, epoch_loss))

    def predict(self, x):
        # Run model to get outputs
        feed_dict = {self.x: x}
        predictions = sess.run(self.outputs, feed_dict)

        return predictions
```

```
# Instantiate model
tf_model = q_model(sess, QUANTILES, in_shape=1, out_shape=1,
                   batch_size=BATCH_SIZE)

# Initialize all variables
init_op = tf.global_variables_initializer()
sess.run(init_op)
```

```
# Run training
tf_model.fit(x_train_expanded, train_labels_expanded, EPOCHS)
```

    Epoch 0: 54.22074890136719
    Epoch 100: 10.759406089782715

```
preds.loc[preds.method == 'TensorFlow', 'pred'] = \
    np.array([item for sublist in tf_model.predict(x_test_expanded)
              for item in sublist])
```

## Visualize quantiles

```
for i, method in enumerate(METHODS):
    ax = plt.scatter(x_test_denorm, test_labels, color=dot_color)
    plt.plot(preds[preds.method == method].pivot_table(
        index='AGE_denorm', columns='q', values='pred'))
    plt.legend(quantiles_legend)#, rev=True)
#     handles, labels = ax.get_legend_handles_labels()
#     ax.legend(handles[::-1], labels[::-1])
    plt.xlim((0, 100))
    ax.axes.xaxis.set_major_formatter(FuncFormatter(
        lambda x, _: '{:.0%}'.format(x / 100)))
    ax.axes.yaxis.set_major_formatter(FuncFormatter(
        lambda y, _: '${:.0f}k'.format(y)))
    plt.xlabel('Proportion of owner-occupied units built prior to 1940')
    plt.ylabel('Median value of owner-occupied homes')
    plt.title(method + ' quantiles', loc='left')
    sns.despine(left=True, bottom=True)
    plt.show()
```

## Compare quantile loss

```
preds['quantile_loss'] = quantile_loss(preds.q, preds.label, preds.pred)
```

```
def plot_loss_comparison(preds):
  overall_loss_comparison = preds[~preds.quantile_loss.isnull()].\
    pivot_table(index='method', values='quantile_loss').\
    sort_values('quantile_loss')
  # Show overall table.
  print(overall_loss_comparison)

  # Plot overall.
  with sns.color_palette('Blues', 1):
    ax = overall_loss_comparison.plot.barh()
    plt.title('Total quantile loss', loc='left')
    sns.despine(left=True, bottom=True)
    plt.xlabel('Quantile loss')
    plt.ylabel('')
    ax.legend_.remove()

  # Per quantile.
  per_quantile_loss_comparison = preds[~preds.quantile_loss.isnull()].\
    pivot_table(index='q', columns='method', values='quantile_loss')
  # Sort by overall quantile loss.
  per_quantile_loss_comparison = \
    per_quantile_loss_comparison[overall_loss_comparison.index]
  print(per_quantile_loss_comparison)

  # Plot per quantile.
  with sns.color_palette('Blues'):
    ax = per_quantile_loss_comparison.plot.barh()
    plt.title('Quantile loss per quantile', loc='left')
    sns.despine(left=True, bottom=True)
    handles, labels = ax.get_legend_handles_labels()
    plt.xlabel('Quantile loss')
    plt.ylabel('Quantile')
    # Reverse legend.
    ax.legend(reversed(handles), reversed(labels));
```

```
plot_loss_comparison(preds)
```

## All features

Same analysis but using all `x` features.

```
preds_full = preds[['method', 'q', 'label']].copy(deep=True)
```

```
x_train_full = train_df.copy(deep=True)
x_test_full = test_df.copy(deep=True)
```

Reformat data for `statsmodels`.

```
X_train_full = sm.add_constant(x_train_full)
X_test_full = sm.add_constant(x_test_full)
```

### OLS

```
ols_full = sm.OLS(train_labels, X_train_full).fit()
```

```
preds_full.loc[preds_full.method == 'OLS', 'pred'] = np.concatenate(
    [ols_quantile(ols_full, X_test_full, q) for q in QUANTILES])
```

### Quantreg

```
# Don't fit yet, since we'll fit once per quantile.
quantreg_full = sm.QuantReg(train_labels, X_train_full)
```

```
preds_full.loc[preds_full.method == 'QuantReg', 'pred'] = np.concatenate(
    [quantreg_full.fit(q=q).predict(X_test_full) for q in QUANTILES])
```

### Random forests

```
rf_full = ensemble.RandomForestRegressor(n_estimators=N_ESTIMATORS,
                                         min_samples_leaf=1, random_state=3,
                                         n_jobs=-1)
rf_full.fit(X_train_full, train_labels)
```

```
preds_full.loc[preds_full.method == 'Random forests', 'pred'] = np.concatenate(
    [rf_quantile(rf_full, X_test_full, q) for q in QUANTILES])
```

### Gradient boosting

```
preds_full.loc[preds_full.method == 'Gradient boosting', 'pred'] = \
    np.concatenate([gb_quantile(X_train_full, train_labels, X_test_full, q)
                    for q in QUANTILES])
```

### Keras

```
preds_full.loc[preds_full.method == 'Keras', 'pred'] = np.concatenate(
    [keras_pred(x_train_full, train_labels, x_test_full, q) for q in QUANTILES])
```

### TensorFlow

Reset graph to avoid conflicts with previous model.

```
tf.reset_default_graph()
sess = tf.Session()
```

```
# Instantiate model
tf_model_full = q_model(sess, QUANTILES, in_shape=train_data.shape[1],
                        batch_size=BATCH_SIZE)

# Initialize all variables
init_op = tf.global_variables_initializer()
sess.run(init_op)

# Run training
tf_model_full.fit(train_data, train_labels_expanded, EPOCHS)

preds_full.loc[preds_full.method == 'TensorFlow', 'pred'] = np.array(
    [item for sublist in tf_model_full.predict(test_data)
     for item in sublist]
)
```

### Compare

```
preds_full['quantile_loss'] = quantile_loss(preds_full.q, preds_full.label,
                                            preds_full.pred)
```

```
plot_loss_comparison(preds_full)
```

Save everything manually using `pickle` (`shelve` and `dill` fail for some objects ([issue](https://github.com/uqfoundation/dill/issues/155))).

```
import pickle

!mkdir quantile_regression_comparison

for i in ['preds']:
  pickle.dump(eval(i), open('quantile_regression_comparison/' + i + '.p', 'wb'))
```