### Lab 3: Expectation Maximization and Variational Autoencoder

### Machine Learning 2 (2019)

* The lab exercises can be done in groups of two people, or individually.
* The deadline is Tuesday, October 15th at 17:00.
* Assignment should be submitted through Canvas! Make sure to include your and your teammates' names with the submission.
* Attach the .IPYNB (IPython Notebook) file containing your code and answers. Naming of the file should be "studentid1\_studentid2\_lab#", for example, the attached file should be "12345\_12346\_lab1.ipynb". Only use underscores ("\_") to connect ids, otherwise the files cannot be parsed.

Notes on implementation:

* You should write your code and answers in an IPython Notebook: http://ipython.org/notebook.html. If you have problems, please ask.
* Use __one cell__ for code and markdown answers only!
    * Put all code in the cell with the ```# YOUR CODE HERE``` comment and overwrite the ```raise NotImplementedError()``` line.
    * For theoretical questions, put your solution using LaTeX style formatting in the YOUR ANSWER HERE cell.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Large parts of you notebook will be graded automatically. Therefore it is important that your notebook can be run completely without errors and within a reasonable time limit. To test your notebook before submission, select Kernel -> Restart \& Run All.
$\newcommand{\bx}{\mathbf{x}} \newcommand{\bpi}{\mathbf{\pi}} \newcommand{\bmu}{\mathbf{\mu}} \newcommand{\bX}{\mathbf{X}} \newcommand{\bZ}{\mathbf{Z}} \newcommand{\bz}{\mathbf{z}}$

### Installing PyTorch

In this lab we will use PyTorch. PyTorch is an open source deep learning framework primarily developed by Facebook's artificial-intelligence research group. In order to install PyTorch in your conda environment go to https://pytorch.org and select your operating system, conda, Python 3.6, no cuda. Copy the text from the "Run this command:" box. Now open a terminal and activate your 'ml2labs' conda environment. Paste the text and run. After the installation is done you should restart Jupyter.

### MNIST data

In this Lab we will use several methods for unsupervised learning on the MNIST dataset of written digits. The dataset contains digital images of handwritten numbers $0$ through $9$. Each image has 28x28 pixels that each take 256 values in a range from white ($= 0$) to  black ($=1$). The labels belonging to the images are also included.
Fortunately, PyTorch comes with a MNIST data loader. The first time you run the box below it will download the MNIST data set. That can take a couple of minutes.
The main data types in PyTorch are tensors. For Part 1, we will convert those tensors to numpy arrays. In Part 2, we will use the torch module to directly work with PyTorch tensors.

```python
%pylab inline
import torch
from torchvision import datasets, transforms

train_dataset = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.Compose([
                       transforms.ToTensor(),
                       transforms.Normalize((0.1307,), (0.3081,))
                   ]))

train_labels = train_dataset.train_labels.numpy()
train_data = train_dataset.train_data.numpy()
# For EM we will use flattened data
train_data = train_data.reshape(train_data.shape[0], -1)

```

    Populating the interactive namespace from numpy and matplotlib

    C:\Users\lintl\Anaconda3\envs\rl2019\lib\site-packages\torchvision\datasets\mnist.py:43: UserWarning: train_labels has been renamed targets
      warnings.warn("train_labels has been renamed targets")
    C:\Users\lintl\Anaconda3\envs\rl2019\lib\site-packages\torchvision\datasets\mnist.py:53: UserWarning: train_data has been renamed data
      warnings.warn("train_data has been renamed data")

## Part 1: Expectation Maximization
We will use the Expectation Maximization (EM) algorithm for the recognition of handwritten digits in the MNIST dataset. The images are modelled as a Bernoulli mixture model (see Bishop $\S9.3.3$):
$$
p(\bx|\bmu, \bpi) = \sum_{k=1}^K  \pi_k \prod_{i=1}^D \mu_{ki}^{x_i}(1-\mu_{ki})^{(1-x_i)}
$$
where $x_i$ is the value of pixel $i$ in an image, $\mu_{ki}$ represents the probability that pixel $i$ in class $k$ is black, and $\{\pi_1, \ldots, \pi_K\}$ are the mixing coefficients of classes in the data. We want to use this data set to classify new images of handwritten numbers.

### 1.1 Binary data (5 points)
As we like to apply our Bernoulli mixture model, write a function `binarize` to convert the (flattened) MNIST data to binary images, where each pixel $x_i \in \{0,1\}$, by thresholding at an appropriate level.

```python
def binarize(X):
    return (X > 128).astype(np.float)
```

```python
# Test test test!
bin_train_data = binarize(train_data)
assert bin_train_data.dtype == np.float
assert bin_train_data.shape == train_data.shape

```

Sample a few images of digits $2$, $3$ and $4$; and show both the original and the binarized image together with their label.

```python
numbers = [2,3,4]

def plot_image(number, images, labels, sample):

    index = np.where(labels == number)[0][sample]

    title = "Digit" + str(labels[index]) + ":"
    plt.imshow(images[index].reshape(28,28))
    plt.title(title + " Non Binarized")

def plot_bin_non_bin(numbers, train, train_labels, bin_train, sample):

    counter = 0
    for i in range(len(numbers)):
        fig = plt.figure(figsize=(15,10))

        plt.subplot(341)
        plot_image(numbers[i], train_data, train_labels, sample[0])
        plt.subplot(341 + 1)
        plot_image(numbers[i], bin_train, train_labels, sample[0])
        plt.subplot(341 + 2)
        plot_image(numbers[i], train_data, train_labels, sample[1])
        plt.subplot(341 + 3)
        plot_image(numbers[i], bin_train, train_labels, sample[1])
        plt.show()

plot_bin_non_bin(numbers, train_data, train_labels, bin_train_data, [3,4])
```

### 1.2 Implementation (40 points)
You are going to write a function ```EM(X, K, max_iter)``` that implements the EM algorithm on the Bernoulli mixture model.

The only parameters the function has are:
* ```X``` :: (NxD) array of input training images
* ```K``` :: size of the latent space
* ```max_iter``` :: maximum number of iterations, i.e. one E-step and one M-step

You are free to specify your return statement.

Make sure you use a sensible way of terminating the iteration process early to prevent unnecessarily running through all epochs. Vectorize computations using ```numpy``` as  much as possible.

You should implement the `E_step(X, mu, pi)` and `M_step(X, gamma)` separately in the functions defined below. These you can then use in your function `EM(X, K, max_iter)`.

```python
def E_step(X, mu, pi):
    # get dimensions of gamma
    X_expand = np.expand_dims(X, axis=1)

    unnorm_gamma = pi * np.prod((mu ** X_expand) * ((1 - mu) ** (1 - X_expand)), axis=2)

    denominator = np.expand_dims(np.sum(unnorm_gamma, axis=1), axis=1)

    gamma = unnorm_gamma / denominator
    return gamma
```

```python
# Let's test on 5 datapoints
n_test = 5
X_test = bin_train_data[:n_test]
D_test, K_test = X_test.shape[1], 10

np.random.seed(2018)
mu_test = np.random.uniform(low=.25, high=.75, size=(K_test,D_test))
pi_test = np.ones(K_test) / K_test

gamma_test = E_step(X_test, mu_test, pi_test)
assert gamma_test.shape == (n_test, K_test)

```

```python
def M_step(X, gamma):
    # we sum over the number of datapoints
    N_k = gamma.sum(axis=0)

    # pi is the fraction of N_k over the total sum ofer N_k
    pi = N_k / np.sum(N_k)

    # expanding with [None,:] and [:, None]
    tmp_mu = gamma[None,:].T * X[None, :]

    # unnormalized mu
    unnorm_mu = tmp_mu.sum(axis=1)

    # normalized mu
    mu = unnorm_mu / N_k[:, None]

    return mu, pi
```

```python
# Oh, let's test again
mu_test, pi_test = M_step(X_test, gamma_test)

assert mu_test.shape == (K_test,D_test)
assert pi_test.shape == (K_test, )

```

```python
def EM(X, K, max_iter, mu=None, pi=None):
    # dimensions
    N = X.shape[0]
    D = X.shape[1]

    # parameter initialization
    if mu is None:
        mu = np.random.uniform(0.25, 0.75, (K,D))
    if pi is None:
        pi = np.ones(K) / K

    conv_crit = 1e-4

    for i in range(max_iter):
        mu_old = mu
        pi_old = pi

        gamma = E_step(X, mu, pi)
        mu, pi = M_step(X, gamma)

        mu_update = np.linalg.norm(mu-mu_old)
        pi_update = np.linalg.norm(pi-pi_old)

        print("Iteration: {}, delta_mu: {}, delta_pi: {}".format(i, np.round(mu_update, 5),
                                                                     np.round(pi_update, 5)))

        if mu_update < conv_crit and pi_update < conv_crit:
            print("After iteration {}, convergence is reached".format(i))
            break

    return gamma, mu, pi

```

### 1.3 Three digits experiment (10 points)
In analogue with Bishop $\S9.3.3$, sample a training set consisting of only __binary__ images of written digits $2$, $3$, and $4$. Run your EM algorithm and show the reconstructed digits.

```python
def plot_mu(mu):

    K = mu.shape[0]
    fig, axes = plt.subplots(1, K, figsize=(15, 10))

    for k, ax in enumerate(axes):
        ax.imshow(mu[k].reshape((28,28)))
    plt.show()

numbers = [2,3,4]
max_iter = 100

example_train_labels = train_labels[np.isin(train_labels, numbers)]
example_trainbinary = bin_train_data[np.isin(train_labels, numbers), :]

```

```python
gamma, mu, pi = EM(example_trainbinary, 3, 100)

#show_mu(mu_3)
```

    Iteration: 0, delta_mu: 20.62302, delta_pi: 0.47597
    Iteration: 1, delta_mu: 2.69712, delta_pi: 0.16751
    Iteration: 2, delta_mu: 1.76431, delta_pi: 0.12087
    Iteration: 3, delta_mu: 1.19617, delta_pi: 0.11609
    Iteration: 4, delta_mu: 0.6832, delta_pi: 0.06307
    Iteration: 5, delta_mu: 0.23769, delta_pi: 0.01967
    Iteration: 6, delta_mu: 0.06954, delta_pi: 0.00507
    Iteration: 7, delta_mu: 0.02194, delta_pi: 0.00145
    Iteration: 8, delta_mu: 0.00868, delta_pi: 0.00052
    Iteration: 9, delta_mu: 0.00348, delta_pi: 0.00019
    Iteration: 10, delta_mu: 0.00165, delta_pi: 6e-05
    Iteration: 11, delta_mu: 0.00181, delta_pi: 8e-05
    Iteration: 12, delta_mu: 0.00072, delta_pi: 5e-05
    Iteration: 13, delta_mu: 0.00044, delta_pi: 3e-05
    Iteration: 14, delta_mu: 0.00095, delta_pi: 4e-05
    Iteration: 15, delta_mu: 0.0011, delta_pi: 5e-05
    Iteration: 16, delta_mu: 0.00022, delta_pi: 2e-05
    Iteration: 17, delta_mu: 9e-05, delta_pi: 1e-05
    After iteration 17, convergence is reached

```python
plot_mu(mu)
print("Mixing coefficients: {}".format(pi))
```

Can you identify which element in the latent space corresponds to which digit? What are the identified mixing coefficients for digits $2$, $3$ and $4$, and how do these compare to the true ones?

Considering the visualizations, it is obvious that the elements in the latent space illustrate the numbers of the classes, namely 2,3,4. So, in the above shown images, the activated regions illustrate probabilities represent if a pixel is active and therefore, the shape of the digits is detectable.

The mixing coefficients are uniformly distributed, which is in line with the assumption, that each number appears equally often.

### 1.4 Experiments (20 points)
Perform the follow-up experiments listed below using your implementation of the EM algorithm. For each of these, describe/comment on the obtained results and give an explanation. You may still use your dataset with only digits 2, 3 and 4 as otherwise computations can take very long.

#### 1.4.1 Size of the latent space (5 points)
Run EM with $K$ larger or smaller than the true number of classes. Describe your results.

```python
gamma_2, mu_2, pi_2 = EM(example_trainbinary, 2, 100)
plot_mu(mu_2)
print("Mixing coefficients for 2 classes: {}".format(pi_2))
```

```python
gamma_5, mu_5, pi_5 = EM(example_trainbinary, 5, 100)
plot_mu(mu_5)
print("Mixing coefficients for 5 classes: {}".format(pi))
```

The first case, a number smaller than the actual number of classes, the convergence of $\mu_k$s to the actual classes is not guaranteed anymore, as seen in the picture, the first image is not clearly a three, but rather a mixture between three and two. The second image actually looks like it converged to a proper 4.

For the larger case, we see, that convergence in itself takes much longer. Also, it is notable that the digits 4 and 2 are represented doubly. Therefore, compared to the smaller number of classes each $\mu_k$ seemingly actually converged to a true class, as none of the shown digits seems distorted or as a mixture of classes. It is also notable, that the digits of the same class vary in their look (the two 2s and 4s seem different), which might stemm from the fact that they reflect a different region of the possible shapes of the respective digits.

The mixture components seem to adapt, as they stay relatively uniform.

So, both a too small and a too large number of classes lead to acuracy loss. Once, the classes are mixed and once multiple representations appear.

#### 1.4.2 Identify misclassifications (10 points)
How can you use the data labels to assign a label to each of the clusters/latent variables? Use this to identify images that are 'misclassified' and try to understand why they are. Report your findings.

```python

```

YOUR ANSWER HERE

#### 1.4.3 Initialize with true values (5 points)
Initialize the three classes with the true values of the parameters and see what happens. Report your results.

```python

```

YOUR ANSWER HERE

## Part 2: Variational Auto-Encoder

A Variational Auto-Encoder (VAE) is a probabilistic model $p(\bx, \bz)$ over observed variables $\bx$ and latent variables and/or parameters $\bz$. Here we distinguish the decoder part, $p(\bx | \bz) p(\bz)$ and an encoder part $p(\bz | \bx)$ that are both specified with a neural network. A lower bound on the log marginal likelihood $\log p(\bx)$ can be obtained by approximately inferring the latent variables z from the observed data x using an encoder distribution $q(\bz| \bx)$ that is also specified as a neural network. This lower bound is then optimized to fit the model to the data.

The model was introduced by Diederik Kingma (during his PhD at the UVA) and Max Welling in 2013, https://arxiv.org/abs/1312.6114.

Since it is such an important model there are plenty of well written tutorials that should help you with the assignment. E.g: https://jaan.io/what-is-variational-autoencoder-vae-tutorial/.

In the following, we will make heavily use of the torch module, https://pytorch.org/docs/stable/index.html. Most of the time replacing `np.` with `torch.` will do the trick, e.g. `np.sum` becomes `torch.sum` and `np.log` becomes `torch.log`. In addition, we will use `torch.FloatTensor()` as an equivalent to `np.array()`. In order to train our VAE efficiently we will make use of batching. The number of data points in a batch will become the first dimension of our data tensor, e.g. A batch of 128 MNIST images has the dimensions [128, 1, 28, 28]. To check check the dimensions of a tensor you can call `.size()`.

### 2.1 Loss function
The objective function (variational lower bound), that we will use to train the VAE, consists of two terms: a log Bernoulli loss (reconstruction loss) and a Kullback–Leibler divergence. We implement the two terms separately and combine them in the end.
As seen in Part 1: Expectation Maximization, we can use a multivariate Bernoulli distribution to model the likelihood $p(\bx | \bz)$ of black and white images. Formally, the variational lower bound is maximized but in PyTorch we are always minimizing therefore we need to calculate the negative log Bernoulli loss and Kullback–Leibler divergence.

### 2.1.1 Negative Log Bernoulli loss (5 points)
The negative log Bernoulli loss is defined as,

\begin{align}
loss = - (\sum_i^D \bx_i \log \hat{\bx_i} + (1 − \bx_i) \log(1 − \hat{\bx_i})).
\end{align}

Write a function `log_bernoulli_loss` that takes a D dimensional vector `x`, its reconstruction `x_hat` and returns the negative log Bernoulli loss. Make sure that your function works for batches of arbitrary size.

```python
def log_bernoulli_loss(x_hat, x):

    loss = -torch.sum(x * torch.log(x_hat) + (1 - x) * torch.log(1 - x_hat))
    return loss

```

```python
### Test test test
x_test = torch.FloatTensor([[0.1, 0.2, 0.3, 0.4], [0.5, 0.6, 0.7, 0.8], [0.9, 0.9, 0.9, 0.9]])
x_hat_test = torch.FloatTensor([[0.11, 0.22, 0.33, 0.44], [0.55, 0.66, 0.77, 0.88], [0.99, 0.99, 0.99, 0.99]])

assert log_bernoulli_loss(x_hat_test, x_test) > 0.0
assert log_bernoulli_loss(x_hat_test, x_test) < 10.0

```

### 2.1.2 Negative Kullback–Leibler divergence (10 Points)
The variational lower bound (the objective to be maximized) contains a KL term $D_{KL}(q(\bz)||p(\bz))$ that can often be calculated analytically. In the VAE we assume $q = N(\bz, \mu, \sigma^2I)$ and $p = N(\bz, 0, I)$. Solve analytically!

YOUR ANSWER HERE

Write a function `KL_loss` that takes two J dimensional vectors `mu` and `logvar` and returns the negative Kullback–Leibler divergence. Where `logvar` is $\log(\sigma^2)$. Make sure that your function works for batches of arbitrary size.

```python
def KL_loss(mu, logvar):

    loss = -(1/2) * torch.sum(1 + logvar - mu.pow(2) - logvar.exp())
    return loss

```

```python
### Test test test
mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

assert KL_loss(mu_test, logvar_test) > 0.0
assert KL_loss(mu_test, logvar_test) < 10.0

```

### 2.1.3 Putting the losses together (5 points)
Write a function `loss_function` that takes a D dimensional vector `x`, its reconstruction `x_hat`, two J dimensional vectors `mu` and `logvar` and returns the final loss. Make sure that your function works for batches of arbitrary size.

```python
def loss_function(x_hat, x, mu, logvar):

    loss = log_bernoulli_loss(x_hat, x) + KL_loss(mu, logvar)
    return loss

```

```python
x_test = torch.FloatTensor([[0.1, 0.2, 0.3], [0.4, 0.5, 0.6], [0.7, 0.8, 0.9]])
x_hat_test = torch.FloatTensor([[0.11, 0.22, 0.33], [0.44, 0.55, 0.66], [0.77, 0.88, 0.99]])
mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

assert loss_function(x_hat_test, x_test, mu_test, logvar_test) > 0.0
assert loss_function(x_hat_test, x_test, mu_test, logvar_test) < 10.0

```

### 2.2 The model
Below you see a data structure for the VAE. The modell itself consists of two main parts the encoder (images $\bx$ to latent variables $\bz$) and the decoder (latent variables $\bz$ to images $\bx$). The encoder is using 3 fully-connected layers, whereas the decoder is using fully-connected layers. Right now the data structure is quite empty, step by step will update its functionality. For test purposes we will initialize a VAE for you. After the data structure is completed you will do the hyperparameter search.

```python
from torch import nn
from torch.nn import functional as F

class VAE(nn.Module):
    def __init__(self, fc1_dims, fc21_dims, fc22_dims, fc3_dims, fc4_dims):
        super(VAE, self).__init__()

        self.fc1 = nn.Linear(*fc1_dims)
        self.fc21 = nn.Linear(*fc21_dims)
        self.fc22 = nn.Linear(*fc22_dims)
        self.fc3 = nn.Linear(*fc3_dims)
        self.fc4 = nn.Linear(*fc4_dims)

    def encode(self, x):
        # To be implemented
        raise Exception('Method not implemented')

    def reparameterize(self, mu, logvar):
        # To be implemented
        raise Exception('Method not implemented')

    def decode(self, z):
        # To be implemented
        raise Exception('Method not implemented')

    def forward(self, x):
        # To be implemented
        raise Exception('Method not implemented')

VAE_test = VAE(fc1_dims=(784, 4), fc21_dims=(4, 2), fc22_dims=(4, 2), fc3_dims=(2, 4), fc4_dims=(4, 784))

```

### 2.3 Encoding (10 points)
Write a function `encode` that gets a vector `x` with 784 elements (flattened MNIST image) and returns `mu` and `logvar`. Your function should use three fully-connected layers (`self.fc1()`, `self.fc21()`, `self.fc22()`). First, you should use `self.fc1()` to embed `x`. Second, you should use `self.fc21()` and `self.fc22()` on the embedding of `x` to compute `mu` and `logvar` respectively. PyTorch comes with a variety of activation functions, the most common calls are `F.relu()`, `F.sigmoid()`, `F.tanh()`. Make sure that your function works for batches of arbitrary size.

```python
def encode(self, x):

    h1 = F.relu(self.fc1(x))
    mu = self.fc21(h1)
    logvar = self.fc22(h1)

    return mu, logvar

```

```python
### Test, test, test
VAE.encode = encode

x_test = torch.ones((5,784))
mu_test, logvar_test = VAE_test.encode(x_test)

assert np.allclose(mu_test.size(), [5, 2])
assert np.allclose(logvar_test.size(), [5, 2])

```

### 2.4 Reparameterization (10 points)
One of the major question that the VAE is answering, is 'how to take derivatives with respect to the parameters of a stochastic variable?', i.e. if we are given $\bz$ that is drawn from a distribution $q(\bz|\bx)$, and we want to take derivatives. This step is necessary to be able to use gradient-based optimization algorithms like SGD.
For some distributions, it is possible to reparameterize samples in a clever way, such that the stochasticity is independent of the parameters. We want our samples to deterministically depend on the parameters of the distribution. For example, in a normally-distributed variable with mean $\mu$ and standard deviation $\sigma$, we can sample from it like this:

\begin{align}
\bz = \mu + \sigma \odot \epsilon,
\end{align}

where $\odot$ is the element-wise multiplication and $\epsilon$ is sampled from $N(0, I)$.

Write a function `reparameterize` that takes two J dimensional vectors `mu` and `logvar`. It should return $\bz = \mu + \sigma \odot \epsilon$.

```python
def reparameterize(self, mu, logvar):

        return  mu + torch.exp(1/2 * logvar) * torch.randn_like(mu)

```

```python
### Test, test, test
VAE.reparameterize = reparameterize
VAE_test.train()

mu_test = torch.FloatTensor([[0.1, 0.2], [0.3, 0.4], [0.5, 0.6]])
logvar_test = torch.FloatTensor([[0.01, 0.02], [0.03, 0.04], [0.05, 0.06]])

z_test = VAE_test.reparameterize(mu_test, logvar_test)

assert np.allclose(z_test.size(), [3, 2])
assert z_test[0][0] < 5.0
assert z_test[0][0] > -5.0

```

### 2.5 Decoding (10 points)
Write a function `decode` that gets a vector `z` with J elements and returns a vector `x_hat` with 784 elements (flattened MNIST image). Your function should use two fully-connected layers (`self.fc3()`, `self.fc4()`). PyTorch comes with a variety of activation functions, the most common calls are `F.relu()`, `F.sigmoid()`, `F.tanh()`. Make sure that your function works for batches of arbitrary size.

```python
def decode(self, z):

    h1 = self.fc3(z)
    h1 = F.relu(h1)
    h1 = self.fc4(h1)
    x_hat = F.sigmoid(h1)
    return x_hat

```

```python
# test test test
VAE.decode = decode

z_test = torch.ones((5,2))
x_hat_test = VAE_test.decode(z_test)

assert np.allclose(x_hat_test.size(), [5, 784])
assert (x_hat_test <= 1).all()
assert (x_hat_test >= 0).all()

```

    C:\Users\lintl\Anaconda3\envs\rl2019\lib\site-packages\torch\nn\functional.py:1350: UserWarning: nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.
      warnings.warn("nn.functional.sigmoid is deprecated. Use torch.sigmoid instead.")

### 2.6 Forward pass (10)
To complete the data structure you have to define a forward pass through the VAE. A single forward pass consists of the encoding of an MNIST image $\bx$ into latent space $\bz$, the reparameterization of $\bz$ and the decoding of $\bz$ into an image $\bx$.

Write a function `forward` that gets a a vector `x` with 784 elements (flattened MNIST image) and returns a vector `x_hat` with 784 elements (flattened MNIST image), `mu` and `logvar`.

```python
def forward(self, x):
    x = x.view(-1, 784)

    mu, logvar = self.encode(x)

    z = self.reparameterize(mu, logvar)

    x_hat = self.decode(z)

    return x_hat, mu, logvar

```

```python
# test test test
VAE.forward = forward

x_test = torch.ones((5,784))
x_hat_test, mu_test, logvar_test = VAE_test.forward(x_test)

assert np.allclose(x_hat_test.size(), [5, 784])
assert np.allclose(mu_test.size(), [5, 2])
assert np.allclose(logvar_test.size(), [5, 2])

```

### 2.7 Training (15)
We will now train the VAE using an optimizer called Adam, https://arxiv.org/abs/1412.6980. The code to train a model in PyTorch is given below.

```python
from torch.autograd import Variable

def train(epoch, train_loader, model, optimizer):
    model.train()
    train_loss = 0
    for batch_idx, (data, _) in enumerate(train_loader):
        data = Variable(data)
        optimizer.zero_grad()
        recon_batch, mu, logvar = model(data)
        loss = loss_function(recon_batch, data.view(-1, 784), mu, logvar)
        loss.backward()
        train_loss += loss.data
        optimizer.step()
        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch, batch_idx * len(data), len(train_loader.dataset),
                100. * batch_idx / len(train_loader),
                loss.data / len(data)))

    print('====> Epoch: {} Average loss: {:.4f}'.format(
          epoch, train_loss / len(train_loader.dataset)))

```

Let's train. You have to choose the hyperparameters. Make sure your loss is going down in a reasonable amount of epochs (around 10).

```python
# Hyperparameters
fc1_dims = (784, 400)
fc21_dims = (400, 20)
fc22_dims = (400, 20)
fc3_dims = (20, 400)
fc4_dims = (400, 784)
lr = 1e-4
batch_size = 120
epochs = 10

```

```python
# This cell contains a hidden test, please don't delete it, thx
```

Run the box below to train the model using the hyperparameters you entered above.

```python
from torchvision import datasets, transforms
from torch import nn, optim

# Load data
train_data = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size, shuffle=True, **{})

# Init model
VAE_MNIST = VAE(fc1_dims=fc1_dims, fc21_dims=fc21_dims, fc22_dims=fc22_dims, fc3_dims=fc3_dims, fc4_dims=fc4_dims)

# Init optimizer
optimizer = optim.Adam(VAE_MNIST.parameters(), lr=lr)

# Train
for epoch in range(1, epochs + 1):
    train(epoch, train_loader, VAE_MNIST, optimizer)

```

    Train Epoch: 1 [0/60000 (0%)]	Loss: 550.567566
    Train Epoch: 1 [12000/60000 (20%)]	Loss: 268.343658
    Train Epoch: 1 [24000/60000 (40%)]	Loss: 235.497818
    Train Epoch: 1 [36000/60000 (60%)]	Loss: 212.567719
    Train Epoch: 1 [48000/60000 (80%)]	Loss: 207.858917
    ====> Epoch: 1 Average loss: 254.7273
    Train Epoch: 2 [0/60000 (0%)]	Loss: 192.445572
    Train Epoch: 2 [12000/60000 (20%)]	Loss: 184.283844
    Train Epoch: 2 [24000/60000 (40%)]	Loss: 174.772934
    Train Epoch: 2 [36000/60000 (60%)]	Loss: 176.407883
    Train Epoch: 2 [48000/60000 (80%)]	Loss: 168.189941
    ====> Epoch: 2 Average loss: 174.8130
    Train Epoch: 3 [0/60000 (0%)]	Loss: 161.005066
    Train Epoch: 3 [12000/60000 (20%)]	Loss: 158.476639
    Train Epoch: 3 [24000/60000 (40%)]	Loss: 156.709290
    Train Epoch: 3 [36000/60000 (60%)]	Loss: 158.456818
    Train Epoch: 3 [48000/60000 (80%)]	Loss: 151.390640
    ====> Epoch: 3 Average loss: 155.7453
    Train Epoch: 4 [0/60000 (0%)]	Loss: 151.145935
    Train Epoch: 4 [12000/60000 (20%)]	Loss: 137.667313
    Train Epoch: 4 [24000/60000 (40%)]	Loss: 143.940842
    Train Epoch: 4 [36000/60000 (60%)]	Loss: 144.788986
    Train Epoch: 4 [48000/60000 (80%)]	Loss: 138.139908
    ====> Epoch: 4 Average loss: 144.3600
    Train Epoch: 5 [0/60000 (0%)]	Loss: 140.378143
    Train Epoch: 5 [12000/60000 (20%)]	Loss: 138.112839
    Train Epoch: 5 [24000/60000 (40%)]	Loss: 139.341797
    Train Epoch: 5 [36000/60000 (60%)]	Loss: 140.853287
    Train Epoch: 5 [48000/60000 (80%)]	Loss: 130.976883
    ====> Epoch: 5 Average loss: 136.7890
    Train Epoch: 6 [0/60000 (0%)]	Loss: 131.719711
    Train Epoch: 6 [12000/60000 (20%)]	Loss: 132.215408
    Train Epoch: 6 [24000/60000 (40%)]	Loss: 131.544479
    Train Epoch: 6 [36000/60000 (60%)]	Loss: 137.173676
    Train Epoch: 6 [48000/60000 (80%)]	Loss: 134.207764
    ====> Epoch: 6 Average loss: 131.5690
    Train Epoch: 7 [0/60000 (0%)]	Loss: 130.113449
    Train Epoch: 7 [12000/60000 (20%)]	Loss: 134.579498
    Train Epoch: 7 [24000/60000 (40%)]	Loss: 127.660347
    Train Epoch: 7 [36000/60000 (60%)]	Loss: 130.052444
    Train Epoch: 7 [48000/60000 (80%)]	Loss: 126.369286
    ====> Epoch: 7 Average loss: 127.7944
    Train Epoch: 8 [0/60000 (0%)]	Loss: 127.497429
    Train Epoch: 8 [12000/60000 (20%)]	Loss: 124.193436
    Train Epoch: 8 [24000/60000 (40%)]	Loss: 124.477249
    Train Epoch: 8 [36000/60000 (60%)]	Loss: 126.167824
    Train Epoch: 8 [48000/60000 (80%)]	Loss: 124.408363
    ====> Epoch: 8 Average loss: 124.9271
    Train Epoch: 9 [0/60000 (0%)]	Loss: 121.864845
    Train Epoch: 9 [12000/60000 (20%)]	Loss: 120.650017
    Train Epoch: 9 [24000/60000 (40%)]	Loss: 118.843796
    Train Epoch: 9 [36000/60000 (60%)]	Loss: 121.871872
    Train Epoch: 9 [48000/60000 (80%)]	Loss: 121.571045
    ====> Epoch: 9 Average loss: 122.5673
    Train Epoch: 10 [0/60000 (0%)]	Loss: 127.931313
    Train Epoch: 10 [12000/60000 (20%)]	Loss: 123.621147
    Train Epoch: 10 [24000/60000 (40%)]	Loss: 119.833107
    Train Epoch: 10 [36000/60000 (60%)]	Loss: 114.540863
    Train Epoch: 10 [48000/60000 (80%)]	Loss: 118.989517
    ====> Epoch: 10 Average loss: 120.6853

Run the box below to check if the model you trained above is able to correctly reconstruct images.

```python
### Let's check if the reconstructions make sense
# Set model to test mode
VAE_MNIST.eval()

# Reconstructed
train_data_plot = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader_plot = torch.utils.data.DataLoader(train_data_plot,
                                           batch_size=1, shuffle=False, **{})

for batch_idx, (data, _) in enumerate(train_loader_plot):
    x_hat, mu, logvar = VAE_MNIST(data)
    plt.imshow(x_hat.view(1,28,28).squeeze().data.numpy(), cmap='gray')
    plt.title('%i' % train_data.train_labels[batch_idx])
    plt.show()
    if batch_idx == 3:
        break

```

### 2.8 Visualize latent space (20 points)
Now, implement the auto-encoder now with a 2-dimensional latent space, and train again over the MNIST data. Make a visualization of the learned manifold by using a linearly spaced coordinate grid as input for the latent space, as seen in  https://arxiv.org/abs/1312.6114 Figure 4.

```python
fc1_dims = (784, 256)
fc21_dims = (256, 2)
fc22_dims = (256, 2)
fc3_dims = (2, 256)
fc4_dims = (256, 784)
lr = 1e-4
batch_size = 128
epochs = 10

# Load data
train_data = datasets.MNIST('../data', train=True, download=True,
                   transform=transforms.ToTensor())

train_loader = torch.utils.data.DataLoader(train_data,
                                           batch_size=batch_size, shuffle=True, **{})

# Init model
VAE_MNIST = VAE(fc1_dims=fc1_dims,
                fc21_dims=fc21_dims,
                fc22_dims=fc22_dims,
                fc3_dims=fc3_dims,
                fc4_dims=fc4_dims)

# Init optimizer
optimizer = optim.Adam(VAE_MNIST.parameters(), lr=lr)

# Train
for epoch in range(1, epochs + 1):
    train(epoch, train_loader, VAE_MNIST, optimizer)
```

    Train Epoch: 1 [0/60000 (0%)]	Loss: 551.562500
    Train Epoch: 1 [12800/60000 (21%)]	Loss: 263.747467
    Train Epoch: 1 [25600/60000 (43%)]	Loss: 238.770248
    Train Epoch: 1 [38400/60000 (64%)]	Loss: 221.702301
    Train Epoch: 1 [51200/60000 (85%)]	Loss: 223.060852
    ====> Epoch: 1 Average loss: 264.5905
    Train Epoch: 2 [0/60000 (0%)]	Loss: 208.024170
    Train Epoch: 2 [12800/60000 (21%)]	Loss: 211.439774
    Train Epoch: 2 [25600/60000 (43%)]	Loss: 202.066025
    Train Epoch: 2 [38400/60000 (64%)]	Loss: 198.744766
    Train Epoch: 2 [51200/60000 (85%)]	Loss: 193.100998
    ====> Epoch: 2 Average loss: 199.6053
    Train Epoch: 3 [0/60000 (0%)]	Loss: 190.765076
    Train Epoch: 3 [12800/60000 (21%)]	Loss: 192.474442
    Train Epoch: 3 [25600/60000 (43%)]	Loss: 187.244492
    Train Epoch: 3 [38400/60000 (64%)]	Loss: 187.458191
    Train Epoch: 3 [51200/60000 (85%)]	Loss: 190.158081
    ====> Epoch: 3 Average loss: 190.6890
    Train Epoch: 4 [0/60000 (0%)]	Loss: 187.374985
    Train Epoch: 4 [12800/60000 (21%)]	Loss: 180.420563
    Train Epoch: 4 [25600/60000 (43%)]	Loss: 188.807617
    Train Epoch: 4 [38400/60000 (64%)]	Loss: 187.241302
    Train Epoch: 4 [51200/60000 (85%)]	Loss: 185.843384
    ====> Epoch: 4 Average loss: 185.7557
    Train Epoch: 5 [0/60000 (0%)]	Loss: 178.010010
    Train Epoch: 5 [12800/60000 (21%)]	Loss: 180.615494
    Train Epoch: 5 [25600/60000 (43%)]	Loss: 186.501984
    Train Epoch: 5 [38400/60000 (64%)]	Loss: 182.561722
    Train Epoch: 5 [51200/60000 (85%)]	Loss: 176.204254
    ====> Epoch: 5 Average loss: 182.3810
    Train Epoch: 6 [0/60000 (0%)]	Loss: 184.070938
    Train Epoch: 6 [12800/60000 (21%)]	Loss: 186.254715
    Train Epoch: 6 [25600/60000 (43%)]	Loss: 174.112152
    Train Epoch: 6 [38400/60000 (64%)]	Loss: 181.852188
    Train Epoch: 6 [51200/60000 (85%)]	Loss: 178.781525
    ====> Epoch: 6 Average loss: 179.9251
    Train Epoch: 7 [0/60000 (0%)]	Loss: 185.035431
    Train Epoch: 7 [12800/60000 (21%)]	Loss: 179.148438
    Train Epoch: 7 [25600/60000 (43%)]	Loss: 170.358719
    Train Epoch: 7 [38400/60000 (64%)]	Loss: 168.970840
    Train Epoch: 7 [51200/60000 (85%)]	Loss: 173.702774
    ====> Epoch: 7 Average loss: 177.9628
    Train Epoch: 8 [0/60000 (0%)]	Loss: 171.521820
    Train Epoch: 8 [12800/60000 (21%)]	Loss: 171.705261
    Train Epoch: 8 [25600/60000 (43%)]	Loss: 178.153992
    Train Epoch: 8 [38400/60000 (64%)]	Loss: 172.498566
    Train Epoch: 8 [51200/60000 (85%)]	Loss: 174.557022
    ====> Epoch: 8 Average loss: 176.2270
    Train Epoch: 9 [0/60000 (0%)]	Loss: 173.551559
    Train Epoch: 9 [12800/60000 (21%)]	Loss: 172.849670
    Train Epoch: 9 [25600/60000 (43%)]	Loss: 174.010452
    Train Epoch: 9 [38400/60000 (64%)]	Loss: 174.835938
    Train Epoch: 9 [51200/60000 (85%)]	Loss: 183.921448
    ====> Epoch: 9 Average loss: 174.4494
    Train Epoch: 10 [0/60000 (0%)]	Loss: 169.704361
    Train Epoch: 10 [12800/60000 (21%)]	Loss: 183.385513
    Train Epoch: 10 [25600/60000 (43%)]	Loss: 163.904984
    Train Epoch: 10 [38400/60000 (64%)]	Loss: 171.295914
    Train Epoch: 10 [51200/60000 (85%)]	Loss: 172.903488
    ====> Epoch: 10 Average loss: 172.6839

### 2.8 Amortized inference (10 points)
What is amortized inference? Where in the code of Part 2 is it used? What is the benefit of using it?


# Singular Matrices

Recently I've been working a lot on coding models and functions 'from scratch'. The point is to force myself to understand not only how each model/method/algorithm works, but also really understand the implementations and parts of the models.

One of the problems I've hit with a few different statistical models is singular matrices. In this post I'm going to talk about:

1. What is a singular matrix?
2. Why are singular matrices a problem?
3. How can I identify a singular matrix?
4. How can I work with/around a singular matrix?

## What is a singular matrix?

A singular (or degenerate) matrix is a matrix that can not be inverted. A lot of this post will discuss the bridge between theory and practice, so I'll further specify that a singular amtrix is a matrix that can't theoretically be inverted.

Of course, practically we can do all sorts of silly things. There are many functions that one can run on a computer that will try and 'succeed' at inverting a sigular matrix. I put succeed in quotes as any function that returns a result when trying to invert a singular matrix is returning nonsense. This is why it is important not to blindly trust functions for random packages you find on the internet! On the other hand, there may be matrices that are theoretically invertable but impossible to practically invert for a variety of reasons that I'll discuss in a bit.

First, let's review the definition of 'invertable'. A matrix is invertable if there exists a square ($nxn$) matrix B such that $AB = BA = I$ where $I$ is the identity matrix. Matrix inversion is the process of finding matrix $B$ for matrix $A$ that satisfies the equation above.

**Technical Asside:** I'd like to burrow a little deeper into what a singular matrix is, but it's a bit mathy. Feel free to skip this if you aren't a hard core math nerd. One can show that non-singular coefficient matrices lead to unique solutions for every vector of constants one could choose. Singular matrices, on the other hand, have non-trivial nullspaces (see proof NMTNS at bottom). For vector contraints b, the system $\mathcal{LS}(A,b)$ could be inconsistant (i.e. no solution). However, if $\mathcal{LS}(A,b)$ has at elast one solution $(w)$, then it will have infinitely many solutions (see proof PSPHS)! A system of equations with a singular coefficient matrix will never have a unique solution.

We'll also note that for singular matrices that there will often be a way to write one row of the matrix as a linear combination of the other rows (may also be true for the columns).

## Why are singular matrices a problem?

Why are singular matrices a problem? Well, as it turns out, we often need to invert matrices. For example, what if we want to evaluate the probability density function of a multivariate gaussian distribution?

$$
p(x;\mu,\Sigma)= \frac{1}{(2\pi)^{\frac{n}{2}} \left\lvert \Sigma \right\rvert ^\frac{1}{2}} \exp\biggl( \frac{-1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu) \biggr)
$$

We would need to find $\Sigma^{-1}$, the inverse of the caovariance matrix. Or what if we wanted to evaulate the posterior of a Gaussian Process Model?

$$
\overline{\mathcal{f}}_* = k^T_*(K+\sigma^2_nI)^-1y
$$

$$
\mathcal{V}[\mathcal{f}_*] = k(x+*,x_*)-k^T_*(K+\sigma^2_nI)^-1k_*
$$

I borrowed the notation above from Gaussian Processes for MAchine Learning Eq. 2.25-26. I could go on listing examples of important equations that require matrix inversions but I think you get the point.

The problem is, if we ever need to invert a singular matrix we are in big trouble!

## How can I identify a singular matrix?

In many classrooms we teach that the simplest way to find out if a matrix is singular is to find the determinant. If the determinant is zero, then the matrix is singular. This would work fine if theory and practice always went hand in hand, but in the real world things can go terribly wrong with using the determinant to find out if a matrix is singular.

Here is a good example (courtesy of anonymous user 85109 on stack exchange). Let's take the determinant of the matrix below.

```python
import numpy as np

arr = np.array([
    [16,     2,     3,    13],
    [5,    11,    10,     8],
    [9,     7,     6,    12],
    [4,    14,    15,     1]
    ])

np.linalg.det(arr)
```

    -1.449507180950607e-12

Well that's not zero! Awesome, so the matrix must be non-singular, right? Nope. We can see that there is a way to write a row on this matrix as a linear combination of the other raws (and the same for the columns). This implies that the matrix is singular!

Let's check the symbolic determinant to get a second opinion.

```python
import sympy as sym

M = sym.Matrix(arr)
M.det()
```

    0

Wow! The symbolic determinant is exactly what we expect for a singular matrix (zero). So why did numpy give us a different answer?

Well, calcualting the determinant of large matrices is very inefficient. A nice approximation that is commonly leveraged by apckages like numpy is to use the product of the diagonal elements of a specific matrix factorization of the array (LU factorization as of version 1.15). Let's look at this factorization below.

```python
import scipy.linalg as la

P, L, U = la.lu(arr)

print(L)
print(U)
print(P)
```

    [[1.         0.         0.         0.        ]
     [0.25       1.         0.         0.        ]
     [0.3125     0.76851852 1.         0.        ]
     [0.5625     0.43518519 1.         1.        ]]
    [[ 1.60000000e+01  2.00000000e+00  3.00000000e+00  1.30000000e+01]
     [ 0.00000000e+00  1.35000000e+01  1.42500000e+01 -2.25000000e+00]
     [ 0.00000000e+00  0.00000000e+00 -1.88888889e+00  5.66666667e+00]
     [ 0.00000000e+00  0.00000000e+00  0.00000000e+00  3.55271368e-15]]
    [[1. 0. 0. 0.]
     [0. 0. 1. 0.]
     [0. 0. 0. 1.]
     [0. 1. 0. 0.]]

The diagonal of the lower triable (L) are all ones and the diagnonal of the upper triangle (U) are all on-zero! This makes for nice easy math then writing statistical/scientific computing packages. We can take the product of the diagonal of the upper triagle to approximate the determinant of the original matrix.

```python
np.prod(np.diag(U))
```

    -1.4495071809506048e-12

We got the same answer as when we called the determinant function from numpy! Neat. Now this LU decomposition technique is super fast, but it relies on floating point arithmatic. The product of the daigonal of the upper triangle is not quite zero as we would expect. This is why using standard functions that calcualte determinants to identify singualr matrices is a bad idea.

Here are a few other weird examples where using the determinant misleads us! Now, the identity matrix is NOT singular,

```python
np.linalg.det(np.eye(100))
```

    1.0

But by multiplying our matrix by a very samll number, we suddenly see a determinant value that is WAY closer to zero than the determinant value for the singular matrix above!

```python
np.linalg.det(0.1*np.eye(100))
```

    1.00000000000033e-100

Now this matrix is NOT singular (for any constant $c$ with identiy matrix $I$, $c*I=D$, matrix D is non-singular jsut like $I$), but with a determinant of $1e^{-100}$ we might easily be fooled into thinking that it is....just wait, it gets worse. Look at the example below.

```python
np.linalg.det(.0001*np.eye(100))
```

    0.0

The determinant should jsut be the determinant of $I$ scaled by $.0001^{-100}$...but numpy can't represent that number! Instead the number underflows and becomes zero, thus tricking us into thinking that this matrix is singular. We could easily invert this matrix and get the correct inversion. We can try the same trick with a large constant to get overflow issues (at least this time numpy warns us!).

```python
np.linalg.det(10000*np.eye(100))
```

    /anaconda3/envs/squidward_env/lib/python3.6/site-packages/numpy/linalg/linalg.py:2022: RuntimeWarning: overflow encountered in det
      r = _umath_linalg.det(a, signature=signature)

    inf

What other tests might we try for identifying if a matrix is singular? One common tool is using the matrix rank. If the rank of an NxM matrix is less than the minimum of N and M, then we call the matrix singular.

The [rank](https://stattrek.com/matrix-algebra/matrix-rank.aspx) of a matrix is defined as either 1) the maximum number of linearly independent column vectors in the matrix or 2) the maximum number of linearly independent row vectors in the matrix. Both definitions are equivalent.

```python
A = .0001*np.eye(100)
rank = np.linalg.matrix_rank(A)
size_M = A.shape[0]
det = np.linalg.det(A)

print("rank {} = dimension {}".format(rank, size_M))
print("determinant {}".format(det))
```

    rank 100 = dimension 100
    determinant 0.0

The scaled identity matrix from above still fails to pass the determinant test (due to underflow issues), passes the rank test. We can try this for our original array as well!

```python
rank = np.linalg.matrix_rank(arr)
size_M = arr.shape[0]
det = np.linalg.det(arr)

print("rank {} != dimension {}".format(rank, size_M))
print("determinant {}".format(det))
```

    rank 3 != dimension 4
    determinant -1.449507180950607e-12

This array passes the determinant test (even though it is singular), but fails to pass the rank test.

Another test that we can try is the [condition](https://en.wikipedia.org/wiki/Condition_number) test. The condition of a matrix can be thought of as a measure of how easy the matrix is to invert. The best condition is one. The higher the condition number the harder a matrix is to invert and the more errors may propagate through to the inverted matrix. This is nice, because it not only gives us a clue as to whether a matrix is singular, but also whether the matrix is close enough to singular that we can expect errors when computing the inversion on a computer (due to floating point errors and what not).

The condition is technically the norm of a matrix times the norm of it's 'inverse' (or the matrix the computer gets when it tries to invert the matrix). If these two norms are very dissimilar (meaning the norm changed a lot when the matrix was inverted) then we say that the matrix is poorly (or ill) conditioned. The condition number will be high in this case.

Now the computer may still invert ill conditioned matrices. In fact, it takes the same amount of steps to invert a matrix using Gaussian elimination no matter the condition. However, ill conditioned matrices will have many errors in their inverted counter parts (even to the point of being completely useless). The condition becomes a kind of error multiplier.

When solving the linear system $Ax = b$, you might expect that a small error in $b$ would result in a small error in $x$. That’s true if $A$ is well-conditioned. But small changes in $b$ could result in large changes in x if $A$ is ill-conditioned. Any error (like measurement error from real world observations) will be multiplied by poor conditioning (not just floating point errors).

As a rule of thumb in double precision, a condition greater than 1e15 is really bad.

```python
np.linalg.cond(arr)
```

    8.147992566622624e+16

Our original matrix (the tricky singular one) has a HUGE condition and is probably even singular based only on looking at the condition. Obviously it will be bad to try to invert this matrix without taking the proper precautions.

One good check is to see if the reciprocal of the condition is larger than than float epsilon. If it is clsot to epsilon then you are bound to run into some issues.

```python
import sys

1.0 / np.linalg.cond(arr) >= sys.float_info.epsilon
```

    False

Finally there is [svd](https://en.wikipedia.org/wiki/Singular_value_decomposition) (singular value decomposition). This is what rank and condition are based on! When an of the singular values of a matrix are small compared to the largest singular value...beware!

```python
np.set_printoptions(suppress=True)
singular_values = np.linalg.svd(arr)[1]
max_sv = np.max(singular_values)
min_sv = np.min(singular_values)

min_sv / max_sv
```

    1.2272961613838391e-17

We notice that the ratio of the largest and smallest value is REALLY small...that's a bad sign. The svd can tell us if a amtrix is close to singularity. If multiple singualr values are really small it can tell us about the matrix rank.

All of the tools above are easy to use and pretty efficient. A careful scientific coder should always check if his/her matrices are invertable.

So what do we do if we find a singular matrix?!

## How can I work with/around a singular matrix?

Singular marticies are, as it turns out, a very small subset of the space of all possible square matricies. In fact, if you were to fill matricies with random uniform samples, you would almost NEVER get a singular matrix.

So the easiest trick to work with a singular amtrix is to add a very small value to the diagonal of the matrix to 'nudge' it out of the singular subset.

```python
arr_nudged = arr + np.eye(arr.shape[0])*1e-10
print("Original Matrix Condition: {}".format(np.linalg.cond(arr)))
print("Nudged Matrix Condition: {}".format(np.linalg.cond(arr_nudged)))
```

    Original Matrix Condition: 8.147992566622624e+16
    Nudged Matrix Condition: 339998354530.4601

The condition of our nudged matrix is still really big...but not NEARLY as bad as it's original condition! Adding a tiny value like 1e-10 to the diagonal of a covariance matrix (for example) might not change the matrix in any meaningful way from a scientific standpoint, but it can be many fewer errors when calculating the matrix's inverse.

Another good piece of advice to to looka t different methods of inverting matrices. Instead of using [Cramer's formula](https://en.wikipedia.org/wiki/Invertible_matrix#Analytic_solution) or standard funtions like `np.linal.inv`, try using SVD decomposition or LU decomposition. You can even find some very nice numerically stable methods leveraging cholesky decomposition (a favorite for Gaussian Process models).

**Author's Note:** The observant reader may note that earlier I said that singular matrices are very rare...so why worry about them? Well, they are rare in the sense that you are unlikely to stumble arocss one when randomly sampling many common distributions from the exponential family. However, there are good reasons why we may run acorss them commonly in the real world. Covariances matrices, for example, are often build around multiple samples from a test set. Many data points/samples may be identical or very close resulting in rows/columns in the matrix that are identical/close to identical. This is why we regularize the matrix we want to invert by adding a very small number to the 'ridge' or 'principal diagonal' of the matrix (just like in [ridge regression](https://link.springer.com/content/pdf/10.3758/BF03208332.pdf)) in the same way that we might add a noise term in the noisy case of Gaussian Process regression! In layman's terms: this is why we add a small number to the matrix diagonal. If you'd like to read more about this in the case of Gaussian Processes, you can check out equation 3.26 on page 45 of Gaussian Processes for Machine Learning.

## Closing Remarks

Well now you know how to find hot singular matrices in your area and even how to work around them! My advice is to always check your matrices before you try to invert them and have a plan for how to treat the matrix if it is poorly conditioned.

## Proofs
I credit these to 'A First Course in Linear Algrebra' by Robert Breezer from which I took these proofs. I thank Robert for releasing this great reference for free under the GNU open source liscence!

**Theorem NMTNS:** Nonsingular Matrices have Trivial Null Spaces.

Suppose that $A$ is a square matrix. Then $A$ is nonsingular if and only if the null space of $A$ is the set containing only the zero vector, i.e. $\mathcal{N}(A)=\{0\}$.

Proof: The null space of a square matrix, $A$, is equal to the set of solutions to the homogeneous system, $\mathcal{LS}(A,0)$. A matrix is nonsingular if and only if the set of solutions to the homogeneous system, $\mathcal{LS}(A,0)$, has only a trivial solution. These two observations may be chained together to construct the two proofs necessary for each half of this theorem.

**Theorem PSPHS:** Particular Solution Plus Homogeneous Solutions.
 Suppose that $w$ s one solution to the linear system of equations $\mathcal{LS}(A,b)$. Then $y$ is a solution to $\mathcal{LS}(A,b)$ if and only if $y=w+z$ for some vector $z \in \mathcal{N}(A)$.

Proof: [PSPHS Proof](http://linear.ups.edu/html/section-LC.html)

```python

```