```python
import mxnet as mx
import numpy as np
import os
import logging
import matplotlib.pyplot as plt
import matplotlib.cm as cm
```

# Building a Variational Autoencoder in MXNet

#### Xiaoyu Lu,  July 5th, 2017

This tutorial guides you through the process of building a variational encoder in MXNet. in this notebook we'll focus on an example unsing the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.

## 1. Loading the Data

We first load the MNIST dataset, which contains 60000 trainings and 10000 test examples. The following code import required modules and load the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization.

```python
mnist = mx.test_utils.get_mnist()
image = np.reshape(mnist['train_data'],(60000,28*28))
label = image
image_test = np.reshape(mnist['test_data'],(10000,28*28))
label_test = image_test
[N,features] = np.shape(image)          #number of examples and features
```

```python
f, (ax1, ax2, ax3, ax4) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))
ax1.imshow(np.reshape(image[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax2.imshow(np.reshape(image[1,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax3.imshow(np.reshape(image[2,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax4.imshow(np.reshape(image[3,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
plt.show()
```

We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images.

```python
model_prefix = None

batch_size = 100
nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},
                            batch_size = batch_size)
nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},
                            batch_size = batch_size)
```

## 2.  Building the Network Architecture

### 2.1 Gaussian MLP as encoder
Next we constuct the neural network, as in the paper, we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used:

\begin{align}
\log q_{\phi}(z|x) &= \log \mathcal{N}(z:\mu,\sigma^2I) \\
\textit{ where } \mu &= W_2h+b_2, \log \sigma^2 = W_3h+b_3\\
h &= \tanh(W_1x+b_1)
\end{align}

where $\{W_1,W_2,W_3,b_1,b_2,b_3\}$ are the weights and biases of the MLP.
Note below that `encoder_mu` and `encoder_logvar` are symbols, can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.

```python
## define data and loss labels as symbols
data = mx.sym.var('data')
loss_label = mx.sym.var('loss_label')

## define fully connected and activation layers for the encoder, where we used tanh activation function.
encoder_h  = mx.sym.FullyConnected(data=data, name="encoder_h",num_hidden=400)
act_h = mx.sym.Activation(data=encoder_h, act_type="tanh",name="activation_h")

## define mu and log variance which are the fully connected layers of the previous activation layer
mu  = mx.sym.FullyConnected(data=act_h, name="mu",num_hidden = 5)
logvar  = mx.sym.FullyConnected(data=act_h, name="logvar",num_hidden = 5)

## sample the latent variables z according to Normal(mu,var)
z = mu + np.multiply(mx.symbol.exp(0.5*logvar),mx.symbol.random_normal(loc=0, scale=1,shape=np.shape(logvar.get_internals()["logvar_output"])))
```

### 2.2 Bernoulli MLP as decoder

In this case let $p_\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:

\begin{align}
\log p(x|z) &= \sum_{i=1}^D x_i\log y_i + (1-x_i)\log (1-y_i) \\
\textit{ where }  y &= f_\sigma(W_5\tanh (W_4z+b_4)+b_5)
\end{align}

where $f_\sigma(\dot)$ is the elementwise sigmoid activation function, $\{W_4,W_5,b_4,b_5\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details.

```python
# define fully connected and tanh activation layers for the decoder
decoder_z = mx.sym.FullyConnected(data=z, name="decoder_z",num_hidden=400)
act_z = mx.sym.Activation(data=decoder_z, act_type="tanh",name="activation_z")

# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension
decoder_x = mx.sym.FullyConnected(data=act_z, name="decoder_x",num_hidden=features)
y = mx.sym.Activation(data=decoder_x, act_type="sigmoid",name='activation_x')
```

### 2.3 Joint Loss Function for the Encoder and the Decoder

The variational lower bound can be estimated as:

\begin{align}
\mathcal{L}(\theta,\phi;x_{(i)}) \approx \frac{1}{2}\left(1+\log ((\sigma_j^{(i)})^2)-(\mu_j^{(i)})^2-(\sigma_j^{(i)})^2\right) + \log p_\theta(x^{(i)}|z^{(i)})
\end{align}

where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\mathcal{L}$ for MXNet to minimize.

```python
# define the objective loss function that needs to be minimized
KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)
loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y)) + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL
output = mx.symbol.MakeLoss(sum(loss),name='loss')
```

## 3. Training the model
Now we can define the model and train it, we initilize the weights and the biases to be Gaussian(0,0.01), and use stochastic gradient descent for optimization. To warm start the training, one may initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`. To save intermediate results, we can optionally use `epoch_end_callback  = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output minus ELBO $-\mathcal{L}$ after each epoch, with the command `eval_metric = 'Loss'`. We can plot the training loss for mini batches by accessing the log and save it to a list, then parse it to the argument `batch_end_callback`.

```python
# set up the log
nd_iter.reset()
logging.getLogger().setLevel(logging.DEBUG)

#define function to trave back training loss
def log_to_list(period, lst):
    def _callback(param):
        """The checkpoint function."""
        if param.nbatch % period == 0:
            name, value = param.eval_metric.get()
            lst.append(value)
    return _callback

# define the model
model = mx.mod.Module(
    symbol = output ,
    data_names=['data'],
    label_names = ['loss_label'])
```

```python
# training the model, save training loss as a list.
training_loss=list()

# initilize the parameters for training using Normal.
init = mx.init.Normal(0.01)
model.fit(nd_iter,  # train data
              initializer=init,
              #if eval_data is supplied, test loss will also be reported
              #eval_data = nd_iter_test,
              optimizer='sgd',  # use SGD to train
              optimizer_params={'learning_rate':1e-3,'wd':1e-2},
              epoch_end_callback  = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),   #save parameters for each epoch if model_prefix is supplied
              batch_end_callback = log_to_list(N/batch_size,training_loss),
              num_epoch=100,
              eval_metric = 'Loss')
```

    INFO:root:Epoch[0] Train-loss=375.023381
    INFO:root:Epoch[0] Time cost=6.127
    INFO:root:Epoch[1] Train-loss=212.780315
    INFO:root:Epoch[1] Time cost=6.409
    INFO:root:Epoch[2] Train-loss=208.209400
    INFO:root:Epoch[2] Time cost=6.619
    INFO:root:Epoch[3] Train-loss=206.146854
    INFO:root:Epoch[3] Time cost=6.648
    INFO:root:Epoch[4] Train-loss=204.530598
    INFO:root:Epoch[4] Time cost=7.000
    INFO:root:Epoch[5] Train-loss=202.799992
    INFO:root:Epoch[5] Time cost=6.778
    INFO:root:Epoch[6] Train-loss=200.333474
    INFO:root:Epoch[6] Time cost=7.187
    INFO:root:Epoch[7] Train-loss=197.506393
    INFO:root:Epoch[7] Time cost=6.712
    INFO:root:Epoch[8] Train-loss=195.969775
    INFO:root:Epoch[8] Time cost=6.896
    INFO:root:Epoch[9] Train-loss=195.418288
    INFO:root:Epoch[9] Time cost=6.887
    INFO:root:Epoch[10] Train-loss=194.739763
    INFO:root:Epoch[10] Time cost=6.745
    INFO:root:Epoch[11] Train-loss=194.380536
    INFO:root:Epoch[11] Time cost=6.706
    INFO:root:Epoch[12] Train-loss=193.955462
    INFO:root:Epoch[12] Time cost=6.592
    INFO:root:Epoch[13] Train-loss=193.493671
    INFO:root:Epoch[13] Time cost=6.775
    INFO:root:Epoch[14] Train-loss=192.958739
    INFO:root:Epoch[14] Time cost=6.600
    INFO:root:Epoch[15] Train-loss=191.928542
    INFO:root:Epoch[15] Time cost=6.586
    INFO:root:Epoch[16] Train-loss=189.797939
    INFO:root:Epoch[16] Time cost=6.700
    INFO:root:Epoch[17] Train-loss=186.672446
    INFO:root:Epoch[17] Time cost=6.869
    INFO:root:Epoch[18] Train-loss=184.616599
    INFO:root:Epoch[18] Time cost=7.144
    INFO:root:Epoch[19] Train-loss=183.305978
    INFO:root:Epoch[19] Time cost=6.997
    INFO:root:Epoch[20] Train-loss=181.944634
    INFO:root:Epoch[20] Time cost=6.481
    INFO:root:Epoch[21] Train-loss=181.005329
    INFO:root:Epoch[21] Time cost=6.754
    INFO:root:Epoch[22] Train-loss=178.363118
    INFO:root:Epoch[22] Time cost=7.000
    INFO:root:Epoch[23] Train-loss=176.363421
    INFO:root:Epoch[23] Time cost=6.923
    INFO:root:Epoch[24] Train-loss=174.573954
    INFO:root:Epoch[24] Time cost=6.510
    INFO:root:Epoch[25] Train-loss=173.245940
    INFO:root:Epoch[25] Time cost=6.926
    INFO:root:Epoch[26] Train-loss=172.082522
    INFO:root:Epoch[26] Time cost=6.733
    INFO:root:Epoch[27] Train-loss=171.123084
    INFO:root:Epoch[27] Time cost=6.616
    INFO:root:Epoch[28] Train-loss=170.239300
    INFO:root:Epoch[28] Time cost=7.004
    INFO:root:Epoch[29] Train-loss=169.538416
    INFO:root:Epoch[29] Time cost=6.341
    INFO:root:Epoch[30] Train-loss=168.952901
    INFO:root:Epoch[30] Time cost=6.736
    INFO:root:Epoch[31] Train-loss=168.169076
    INFO:root:Epoch[31] Time cost=6.616
    INFO:root:Epoch[32] Train-loss=167.208973
    INFO:root:Epoch[32] Time cost=6.446
    INFO:root:Epoch[33] Train-loss=165.732213
    INFO:root:Epoch[33] Time cost=6.405
    INFO:root:Epoch[34] Train-loss=163.606801
    INFO:root:Epoch[34] Time cost=6.139
    INFO:root:Epoch[35] Train-loss=161.985880
    INFO:root:Epoch[35] Time cost=6.678
    INFO:root:Epoch[36] Train-loss=160.763072
    INFO:root:Epoch[36] Time cost=8.749
    INFO:root:Epoch[37] Train-loss=160.025193
    INFO:root:Epoch[37] Time cost=6.519
    INFO:root:Epoch[38] Train-loss=159.319723
    INFO:root:Epoch[38] Time cost=7.584
    INFO:root:Epoch[39] Train-loss=158.670701
    INFO:root:Epoch[39] Time cost=6.874
    INFO:root:Epoch[40] Train-loss=158.225733
    INFO:root:Epoch[40] Time cost=6.402
    INFO:root:Epoch[41] Train-loss=157.741337
    INFO:root:Epoch[41] Time cost=8.617
    INFO:root:Epoch[42] Train-loss=157.301411
    INFO:root:Epoch[42] Time cost=6.515
    INFO:root:Epoch[43] Train-loss=156.765170
    INFO:root:Epoch[43] Time cost=6.447
    INFO:root:Epoch[44] Train-loss=156.389668
    INFO:root:Epoch[44] Time cost=6.130
    INFO:root:Epoch[45] Train-loss=155.815434
    INFO:root:Epoch[45] Time cost=6.155
    INFO:root:Epoch[46] Train-loss=155.432254
    INFO:root:Epoch[46] Time cost=6.158
    INFO:root:Epoch[47] Train-loss=155.114027
    INFO:root:Epoch[47] Time cost=6.749
    INFO:root:Epoch[48] Train-loss=154.612441
    INFO:root:Epoch[48] Time cost=6.255
    INFO:root:Epoch[49] Train-loss=154.137659
    INFO:root:Epoch[49] Time cost=7.813
    INFO:root:Epoch[50] Train-loss=153.634072
    INFO:root:Epoch[50] Time cost=7.408
    INFO:root:Epoch[51] Train-loss=153.417397
    INFO:root:Epoch[51] Time cost=7.747
    INFO:root:Epoch[52] Train-loss=152.851887
    INFO:root:Epoch[52] Time cost=8.587
    INFO:root:Epoch[53] Train-loss=152.575068
    INFO:root:Epoch[53] Time cost=7.554
    INFO:root:Epoch[54] Train-loss=152.084419
    INFO:root:Epoch[54] Time cost=6.628
    INFO:root:Epoch[55] Train-loss=151.724836
    INFO:root:Epoch[55] Time cost=6.535
    INFO:root:Epoch[56] Train-loss=151.302525
    INFO:root:Epoch[56] Time cost=7.148
    INFO:root:Epoch[57] Train-loss=150.960916
    INFO:root:Epoch[57] Time cost=7.195
    INFO:root:Epoch[58] Train-loss=150.603895
    INFO:root:Epoch[58] Time cost=6.649
    INFO:root:Epoch[59] Train-loss=150.237795
    INFO:root:Epoch[59] Time cost=6.222
    INFO:root:Epoch[60] Train-loss=149.936080
    INFO:root:Epoch[60] Time cost=8.450
    INFO:root:Epoch[61] Train-loss=149.514617
    INFO:root:Epoch[61] Time cost=6.113
    INFO:root:Epoch[62] Train-loss=149.229345
    INFO:root:Epoch[62] Time cost=6.088
    INFO:root:Epoch[63] Train-loss=148.893769
    INFO:root:Epoch[63] Time cost=6.558
    INFO:root:Epoch[64] Train-loss=148.526837
    INFO:root:Epoch[64] Time cost=7.590
    INFO:root:Epoch[65] Train-loss=148.249951
    INFO:root:Epoch[65] Time cost=6.180
    INFO:root:Epoch[66] Train-loss=147.940414
    INFO:root:Epoch[66] Time cost=6.242
    INFO:root:Epoch[67] Train-loss=147.621304
    INFO:root:Epoch[67] Time cost=8.501
    INFO:root:Epoch[68] Train-loss=147.294314
    INFO:root:Epoch[68] Time cost=7.645
    INFO:root:Epoch[69] Train-loss=147.074479
    INFO:root:Epoch[69] Time cost=7.092
    INFO:root:Epoch[70] Train-loss=146.796387
    INFO:root:Epoch[70] Time cost=6.914
    INFO:root:Epoch[71] Train-loss=146.508842
    INFO:root:Epoch[71] Time cost=6.606
    INFO:root:Epoch[72] Train-loss=146.230444
    INFO:root:Epoch[72] Time cost=7.755
    INFO:root:Epoch[73] Train-loss=145.970296
    INFO:root:Epoch[73] Time cost=6.409
    INFO:root:Epoch[74] Train-loss=145.711610
    INFO:root:Epoch[74] Time cost=6.334
    INFO:root:Epoch[75] Train-loss=145.460053
    INFO:root:Epoch[75] Time cost=7.269
    INFO:root:Epoch[76] Train-loss=145.156451
    INFO:root:Epoch[76] Time cost=6.744
    INFO:root:Epoch[77] Train-loss=144.957674
    INFO:root:Epoch[77] Time cost=7.100
    INFO:root:Epoch[78] Train-loss=144.729749
    INFO:root:Epoch[78] Time cost=6.242
    INFO:root:Epoch[79] Train-loss=144.481728
    INFO:root:Epoch[79] Time cost=6.865
    INFO:root:Epoch[80] Train-loss=144.236061
    INFO:root:Epoch[80] Time cost=6.632
    INFO:root:Epoch[81] Train-loss=144.030473
    INFO:root:Epoch[81] Time cost=6.764
    INFO:root:Epoch[82] Train-loss=143.776374
    INFO:root:Epoch[82] Time cost=6.564
    INFO:root:Epoch[83] Train-loss=143.538847
    INFO:root:Epoch[83] Time cost=6.181
    INFO:root:Epoch[84] Train-loss=143.326444
    INFO:root:Epoch[84] Time cost=6.220
    INFO:root:Epoch[85] Train-loss=143.078987
    INFO:root:Epoch[85] Time cost=6.823
    INFO:root:Epoch[86] Train-loss=142.877117
    INFO:root:Epoch[86] Time cost=7.755
    INFO:root:Epoch[87] Train-loss=142.667316
    INFO:root:Epoch[87] Time cost=6.068
    INFO:root:Epoch[88] Train-loss=142.461755
    INFO:root:Epoch[88] Time cost=6.111
    INFO:root:Epoch[89] Train-loss=142.270438
    INFO:root:Epoch[89] Time cost=6.221
    INFO:root:Epoch[90] Train-loss=142.047086
    INFO:root:Epoch[90] Time cost=8.061
    INFO:root:Epoch[91] Train-loss=141.855774
    INFO:root:Epoch[91] Time cost=6.433
    INFO:root:Epoch[92] Train-loss=141.688955
    INFO:root:Epoch[92] Time cost=7.153
    INFO:root:Epoch[93] Train-loss=141.442910
    INFO:root:Epoch[93] Time cost=7.113
    INFO:root:Epoch[94] Train-loss=141.279274
    INFO:root:Epoch[94] Time cost=7.152
    INFO:root:Epoch[95] Train-loss=141.086522
    INFO:root:Epoch[95] Time cost=6.472
    INFO:root:Epoch[96] Train-loss=140.901925
    INFO:root:Epoch[96] Time cost=6.767
    INFO:root:Epoch[97] Train-loss=140.722496
    INFO:root:Epoch[97] Time cost=7.044
    INFO:root:Epoch[98] Train-loss=140.579295
    INFO:root:Epoch[98] Time cost=7.040
    INFO:root:Epoch[99] Train-loss=140.386067
    INFO:root:Epoch[99] Time cost=6.669

```python
ELBO = [-training_loss[i] for i in range(len(training_loss))]
plt.plot(ELBO)
plt.ylabel('ELBO');plt.xlabel('epoch');plt.title("training curve for mini batches")
plt.show()
```

As expected, the ELBO is monotonically increasing over epoch, and we reproduced the resutls given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set.

```python
arg_params = model.get_params()[0]

#if saved the parameters, can load them at e.g. 100th epoch
#sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)
#assert sym.tojson() == output.tojson()

e = y.bind(mx.cpu(), {'data': nd_iter_test.data[0][1],
                     'encoder_h_weight': arg_params['encoder_h_weight'],
                     'encoder_h_bias': arg_params['encoder_h_bias'],
                     'mu_weight': arg_params['mu_weight'],
                     'mu_bias': arg_params['mu_bias'],
                     'logvar_weight':arg_params['logvar_weight'],
                     'logvar_bias':arg_params['logvar_bias'],
                     'decoder_z_weight':arg_params['decoder_z_weight'],
                     'decoder_z_bias':arg_params['decoder_z_bias'],
                     'decoder_x_weight':arg_params['decoder_x_weight'],
                     'decoder_x_bias':arg_params['decoder_x_bias'],
                     'loss_label':label})

x_fit = e.forward()
x_construction = x_fit[0].asnumpy()
```

```python
# learning images on the test set
f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))
ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax1.set_title('True image')
ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax2.set_title('Learned image')
ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax3.set_title('Learned image')
ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax4.set_title('Learned image')
plt.show()
```

```python
#calculate the ELBO which is minus the loss for test set
metric = mx.metric.Loss()
model.score(nd_iter_test, metric)
```

    [('loss', 139.73684648437501)]

## 4. All together: MXNet-based class VAE

```python
from VAE import VAE
```

One can directly call the class `VAE` to do the training. The outputs are the learned model and training loss.
```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None, initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```

```python
# can initilize weights and biases with the learned parameters
#init = mx.initializer.Load(params)

# call the VAE , output model contains the learned model and training loss
out = VAE(n_latent=2,x_train=image,x_valid=None,num_epoch=200)
```

    INFO:root:Epoch[0] Train-loss=377.146422
    INFO:root:Epoch[0] Time cost=5.989
    INFO:root:Epoch[1] Train-loss=211.998043
    INFO:root:Epoch[1] Time cost=6.303
    INFO:root:Epoch[2] Train-loss=207.103096
    INFO:root:Epoch[2] Time cost=7.368
    INFO:root:Epoch[3] Train-loss=204.958183
    INFO:root:Epoch[3] Time cost=7.530
    INFO:root:Epoch[4] Train-loss=203.342700
    INFO:root:Epoch[4] Time cost=8.887
    INFO:root:Epoch[5] Train-loss=201.649251
    INFO:root:Epoch[5] Time cost=9.147
    INFO:root:Epoch[6] Train-loss=199.782661
    INFO:root:Epoch[6] Time cost=8.924
    INFO:root:Epoch[7] Train-loss=198.044015
    INFO:root:Epoch[7] Time cost=8.920
    INFO:root:Epoch[8] Train-loss=195.732077
    INFO:root:Epoch[8] Time cost=8.857
    INFO:root:Epoch[9] Train-loss=194.070547
    INFO:root:Epoch[9] Time cost=9.216
    INFO:root:Epoch[10] Train-loss=193.186871
    INFO:root:Epoch[10] Time cost=8.966
    INFO:root:Epoch[11] Train-loss=192.700208
    INFO:root:Epoch[11] Time cost=8.843
    INFO:root:Epoch[12] Train-loss=192.191504
    INFO:root:Epoch[12] Time cost=8.152
    INFO:root:Epoch[13] Train-loss=191.842837
    INFO:root:Epoch[13] Time cost=6.180
    INFO:root:Epoch[14] Train-loss=191.310450
    INFO:root:Epoch[14] Time cost=6.067
    INFO:root:Epoch[15] Train-loss=190.520681
    INFO:root:Epoch[15] Time cost=6.058
    INFO:root:Epoch[16] Train-loss=189.784146
    INFO:root:Epoch[16] Time cost=6.046
    INFO:root:Epoch[17] Train-loss=188.515020
    INFO:root:Epoch[17] Time cost=6.062
    INFO:root:Epoch[18] Train-loss=187.530712
    INFO:root:Epoch[18] Time cost=6.088
    INFO:root:Epoch[19] Train-loss=186.194826
    INFO:root:Epoch[19] Time cost=6.491
    INFO:root:Epoch[20] Train-loss=185.492288
    INFO:root:Epoch[20] Time cost=6.182
    INFO:root:Epoch[21] Train-loss=184.922654
    INFO:root:Epoch[21] Time cost=6.058
    INFO:root:Epoch[22] Train-loss=184.677911
    INFO:root:Epoch[22] Time cost=6.042
    INFO:root:Epoch[23] Train-loss=183.921396
    INFO:root:Epoch[23] Time cost=5.994
    INFO:root:Epoch[24] Train-loss=183.600690
    INFO:root:Epoch[24] Time cost=6.038
    INFO:root:Epoch[25] Train-loss=183.388476
    INFO:root:Epoch[25] Time cost=6.025
    INFO:root:Epoch[26] Train-loss=182.972208
    INFO:root:Epoch[26] Time cost=6.014
    INFO:root:Epoch[27] Train-loss=182.561678
    INFO:root:Epoch[27] Time cost=6.064
    INFO:root:Epoch[28] Train-loss=182.475261
    INFO:root:Epoch[28] Time cost=5.983
    INFO:root:Epoch[29] Train-loss=182.308808
    INFO:root:Epoch[29] Time cost=6.371
    INFO:root:Epoch[30] Train-loss=182.135900
    INFO:root:Epoch[30] Time cost=6.038
    INFO:root:Epoch[31] Train-loss=181.978367
    INFO:root:Epoch[31] Time cost=6.924
    INFO:root:Epoch[32] Train-loss=181.677153
    INFO:root:Epoch[32] Time cost=8.205
    INFO:root:Epoch[33] Train-loss=181.677775
    INFO:root:Epoch[33] Time cost=6.017
    INFO:root:Epoch[34] Train-loss=181.257998
    INFO:root:Epoch[34] Time cost=6.056
    INFO:root:Epoch[35] Train-loss=181.125288
    INFO:root:Epoch[35] Time cost=6.020
    INFO:root:Epoch[36] Train-loss=181.018858
    INFO:root:Epoch[36] Time cost=6.035
    INFO:root:Epoch[37] Train-loss=180.785110
    INFO:root:Epoch[37] Time cost=6.049
    INFO:root:Epoch[38] Train-loss=180.452598
    INFO:root:Epoch[38] Time cost=6.083
    INFO:root:Epoch[39] Train-loss=180.362733
    INFO:root:Epoch[39] Time cost=6.198
    INFO:root:Epoch[40] Train-loss=180.060788
    INFO:root:Epoch[40] Time cost=6.049
    INFO:root:Epoch[41] Train-loss=180.022728
    INFO:root:Epoch[41] Time cost=6.135
    INFO:root:Epoch[42] Train-loss=179.648499
    INFO:root:Epoch[42] Time cost=6.055
    INFO:root:Epoch[43] Train-loss=179.507952
    INFO:root:Epoch[43] Time cost=6.108
    INFO:root:Epoch[44] Train-loss=179.303132
    INFO:root:Epoch[44] Time cost=6.020
    INFO:root:Epoch[45] Train-loss=178.945211
    INFO:root:Epoch[45] Time cost=6.004
    INFO:root:Epoch[46] Train-loss=178.808598
    INFO:root:Epoch[46] Time cost=6.016
    INFO:root:Epoch[47] Train-loss=178.550906
    INFO:root:Epoch[47] Time cost=6.050
    INFO:root:Epoch[48] Train-loss=178.403674
    INFO:root:Epoch[48] Time cost=6.115
    INFO:root:Epoch[49] Train-loss=178.237544
    INFO:root:Epoch[49] Time cost=6.004
    INFO:root:Epoch[50] Train-loss=178.033747
    INFO:root:Epoch[50] Time cost=6.051
    INFO:root:Epoch[51] Train-loss=177.802884
    INFO:root:Epoch[51] Time cost=6.028
    INFO:root:Epoch[52] Train-loss=177.533980
    INFO:root:Epoch[52] Time cost=6.052
    INFO:root:Epoch[53] Train-loss=177.490143
    INFO:root:Epoch[53] Time cost=6.019
    INFO:root:Epoch[54] Train-loss=177.136637
    INFO:root:Epoch[54] Time cost=6.014
    INFO:root:Epoch[55] Train-loss=177.062524
    INFO:root:Epoch[55] Time cost=6.024
    INFO:root:Epoch[56] Train-loss=176.869033
    INFO:root:Epoch[56] Time cost=6.065
    INFO:root:Epoch[57] Train-loss=176.704606
    INFO:root:Epoch[57] Time cost=6.037
    INFO:root:Epoch[58] Train-loss=176.470091
    INFO:root:Epoch[58] Time cost=6.012
    INFO:root:Epoch[59] Train-loss=176.261440
    INFO:root:Epoch[59] Time cost=6.215
    INFO:root:Epoch[60] Train-loss=176.133904
    INFO:root:Epoch[60] Time cost=6.042
    INFO:root:Epoch[61] Train-loss=175.941920
    INFO:root:Epoch[61] Time cost=6.000
    INFO:root:Epoch[62] Train-loss=175.731296
    INFO:root:Epoch[62] Time cost=6.025
    INFO:root:Epoch[63] Train-loss=175.613303
    INFO:root:Epoch[63] Time cost=6.002
    INFO:root:Epoch[64] Train-loss=175.438844
    INFO:root:Epoch[64] Time cost=5.982
    INFO:root:Epoch[65] Train-loss=175.254716
    INFO:root:Epoch[65] Time cost=6.016
    INFO:root:Epoch[66] Train-loss=175.090210
    INFO:root:Epoch[66] Time cost=6.008
    INFO:root:Epoch[67] Train-loss=174.895443
    INFO:root:Epoch[67] Time cost=6.008
    INFO:root:Epoch[68] Train-loss=174.701321
    INFO:root:Epoch[68] Time cost=6.418
    INFO:root:Epoch[69] Train-loss=174.553292
    INFO:root:Epoch[69] Time cost=6.072
    INFO:root:Epoch[70] Train-loss=174.349379
    INFO:root:Epoch[70] Time cost=6.048
    INFO:root:Epoch[71] Train-loss=174.174641
    INFO:root:Epoch[71] Time cost=6.036
    INFO:root:Epoch[72] Train-loss=173.966333
    INFO:root:Epoch[72] Time cost=6.017
    INFO:root:Epoch[73] Train-loss=173.798454
    INFO:root:Epoch[73] Time cost=6.018
    INFO:root:Epoch[74] Train-loss=173.635657
    INFO:root:Epoch[74] Time cost=5.985
    INFO:root:Epoch[75] Train-loss=173.423795
    INFO:root:Epoch[75] Time cost=6.016
    INFO:root:Epoch[76] Train-loss=173.273981
    INFO:root:Epoch[76] Time cost=6.018
    INFO:root:Epoch[77] Train-loss=173.073401
    INFO:root:Epoch[77] Time cost=5.996
    INFO:root:Epoch[78] Train-loss=172.888044
    INFO:root:Epoch[78] Time cost=6.035
    INFO:root:Epoch[79] Train-loss=172.694943
    INFO:root:Epoch[79] Time cost=8.492
    INFO:root:Epoch[80] Train-loss=172.504260
    INFO:root:Epoch[80] Time cost=7.380
    INFO:root:Epoch[81] Train-loss=172.323245
    INFO:root:Epoch[81] Time cost=6.063
    INFO:root:Epoch[82] Train-loss=172.131274
    INFO:root:Epoch[82] Time cost=6.209
    INFO:root:Epoch[83] Train-loss=171.932986
    INFO:root:Epoch[83] Time cost=6.060
    INFO:root:Epoch[84] Train-loss=171.755262
    INFO:root:Epoch[84] Time cost=6.068
    INFO:root:Epoch[85] Train-loss=171.556803
    INFO:root:Epoch[85] Time cost=6.004
    INFO:root:Epoch[86] Train-loss=171.384773
    INFO:root:Epoch[86] Time cost=6.059
    INFO:root:Epoch[87] Train-loss=171.185034
    INFO:root:Epoch[87] Time cost=6.001
    INFO:root:Epoch[88] Train-loss=170.995980
    INFO:root:Epoch[88] Time cost=6.143
    INFO:root:Epoch[89] Train-loss=170.818701
    INFO:root:Epoch[89] Time cost=6.690
    INFO:root:Epoch[90] Train-loss=170.629929
    INFO:root:Epoch[90] Time cost=6.869
    INFO:root:Epoch[91] Train-loss=170.450824
    INFO:root:Epoch[91] Time cost=7.156
    INFO:root:Epoch[92] Train-loss=170.261806
    INFO:root:Epoch[92] Time cost=6.972
    INFO:root:Epoch[93] Train-loss=170.070318
    INFO:root:Epoch[93] Time cost=6.595
    INFO:root:Epoch[94] Train-loss=169.906993
    INFO:root:Epoch[94] Time cost=6.561
    INFO:root:Epoch[95] Train-loss=169.734455
    INFO:root:Epoch[95] Time cost=6.744
    INFO:root:Epoch[96] Train-loss=169.564318
    INFO:root:Epoch[96] Time cost=6.601
    INFO:root:Epoch[97] Train-loss=169.373926
    INFO:root:Epoch[97] Time cost=6.725
    INFO:root:Epoch[98] Train-loss=169.215408
    INFO:root:Epoch[98] Time cost=6.391
    INFO:root:Epoch[99] Train-loss=169.039854
    INFO:root:Epoch[99] Time cost=6.677
    INFO:root:Epoch[100] Train-loss=168.869222
    INFO:root:Epoch[100] Time cost=6.370
    INFO:root:Epoch[101] Train-loss=168.703175
    INFO:root:Epoch[101] Time cost=6.607
    INFO:root:Epoch[102] Train-loss=168.523054
    INFO:root:Epoch[102] Time cost=6.368
    INFO:root:Epoch[103] Train-loss=168.365964
    INFO:root:Epoch[103] Time cost=10.267
    INFO:root:Epoch[104] Train-loss=168.181174
    INFO:root:Epoch[104] Time cost=11.132
    INFO:root:Epoch[105] Train-loss=168.021498
    INFO:root:Epoch[105] Time cost=10.187
    INFO:root:Epoch[106] Train-loss=167.858251
    INFO:root:Epoch[106] Time cost=10.676
    INFO:root:Epoch[107] Train-loss=167.690670
    INFO:root:Epoch[107] Time cost=10.973
    INFO:root:Epoch[108] Train-loss=167.535069
    INFO:root:Epoch[108] Time cost=10.108
    INFO:root:Epoch[109] Train-loss=167.373971
    INFO:root:Epoch[109] Time cost=11.013
    INFO:root:Epoch[110] Train-loss=167.207507
    INFO:root:Epoch[110] Time cost=11.427
    INFO:root:Epoch[111] Train-loss=167.043077
    INFO:root:Epoch[111] Time cost=10.349
    INFO:root:Epoch[112] Train-loss=166.884060
    INFO:root:Epoch[112] Time cost=13.129
    INFO:root:Epoch[113] Train-loss=166.746976
    INFO:root:Epoch[113] Time cost=11.255
    INFO:root:Epoch[114] Train-loss=166.572499
    INFO:root:Epoch[114] Time cost=10.037
    INFO:root:Epoch[115] Train-loss=166.445170
    INFO:root:Epoch[115] Time cost=10.406
    INFO:root:Epoch[116] Train-loss=166.284912
    INFO:root:Epoch[116] Time cost=10.170
    INFO:root:Epoch[117] Train-loss=166.171475
    INFO:root:Epoch[117] Time cost=10.034
    INFO:root:Epoch[118] Train-loss=166.015457
    INFO:root:Epoch[118] Time cost=10.047
    INFO:root:Epoch[119] Train-loss=165.882208
    INFO:root:Epoch[119] Time cost=10.008
    INFO:root:Epoch[120] Train-loss=165.753836
    INFO:root:Epoch[120] Time cost=10.056
    INFO:root:Epoch[121] Train-loss=165.626045
    INFO:root:Epoch[121] Time cost=10.704
    INFO:root:Epoch[122] Train-loss=165.492859
    INFO:root:Epoch[122] Time cost=10.609
    INFO:root:Epoch[123] Train-loss=165.361132
    INFO:root:Epoch[123] Time cost=10.027
    INFO:root:Epoch[124] Train-loss=165.256487
    INFO:root:Epoch[124] Time cost=11.225
    INFO:root:Epoch[125] Train-loss=165.119995
    INFO:root:Epoch[125] Time cost=11.266
    INFO:root:Epoch[126] Train-loss=165.012773
    INFO:root:Epoch[126] Time cost=10.547
    INFO:root:Epoch[127] Train-loss=164.898748
    INFO:root:Epoch[127] Time cost=10.339
    INFO:root:Epoch[128] Train-loss=164.775702
    INFO:root:Epoch[128] Time cost=10.875
    INFO:root:Epoch[129] Train-loss=164.692449
    INFO:root:Epoch[129] Time cost=8.412
    INFO:root:Epoch[130] Train-loss=164.564323
    INFO:root:Epoch[130] Time cost=7.239
    INFO:root:Epoch[131] Train-loss=164.468273
    INFO:root:Epoch[131] Time cost=10.096
    INFO:root:Epoch[132] Train-loss=164.328320
    INFO:root:Epoch[132] Time cost=9.680
    INFO:root:Epoch[133] Train-loss=164.256156
    INFO:root:Epoch[133] Time cost=10.707
    INFO:root:Epoch[134] Train-loss=164.151625
    INFO:root:Epoch[134] Time cost=13.835
    INFO:root:Epoch[135] Train-loss=164.046402
    INFO:root:Epoch[135] Time cost=10.049
    INFO:root:Epoch[136] Train-loss=163.960676
    INFO:root:Epoch[136] Time cost=9.625
    INFO:root:Epoch[137] Train-loss=163.873193
    INFO:root:Epoch[137] Time cost=9.845
    INFO:root:Epoch[138] Train-loss=163.783837
    INFO:root:Epoch[138] Time cost=9.618
    INFO:root:Epoch[139] Train-loss=163.658903
    INFO:root:Epoch[139] Time cost=10.411
    INFO:root:Epoch[140] Train-loss=163.588920
    INFO:root:Epoch[140] Time cost=9.633
    INFO:root:Epoch[141] Train-loss=163.493254
    INFO:root:Epoch[141] Time cost=10.668
    INFO:root:Epoch[142] Train-loss=163.401188
    INFO:root:Epoch[142] Time cost=10.644
    INFO:root:Epoch[143] Train-loss=163.334470
    INFO:root:Epoch[143] Time cost=9.665
    INFO:root:Epoch[144] Train-loss=163.235133
    INFO:root:Epoch[144] Time cost=9.612
    INFO:root:Epoch[145] Train-loss=163.168029
    INFO:root:Epoch[145] Time cost=9.578
    INFO:root:Epoch[146] Train-loss=163.092392
    INFO:root:Epoch[146] Time cost=10.215
    INFO:root:Epoch[147] Train-loss=163.014362
    INFO:root:Epoch[147] Time cost=12.296
    INFO:root:Epoch[148] Train-loss=162.891574
    INFO:root:Epoch[148] Time cost=9.578
    INFO:root:Epoch[149] Train-loss=162.831664
    INFO:root:Epoch[149] Time cost=9.536
    INFO:root:Epoch[150] Train-loss=162.768784
    INFO:root:Epoch[150] Time cost=9.607
    INFO:root:Epoch[151] Train-loss=162.695416
    INFO:root:Epoch[151] Time cost=9.681
    INFO:root:Epoch[152] Train-loss=162.620814
    INFO:root:Epoch[152] Time cost=9.464
    INFO:root:Epoch[153] Train-loss=162.527031
    INFO:root:Epoch[153] Time cost=9.518
    INFO:root:Epoch[154] Train-loss=162.466575
    INFO:root:Epoch[154] Time cost=9.562
    INFO:root:Epoch[155] Train-loss=162.409388
    INFO:root:Epoch[155] Time cost=9.483
    INFO:root:Epoch[156] Train-loss=162.308957
    INFO:root:Epoch[156] Time cost=9.545
    INFO:root:Epoch[157] Train-loss=162.211725
    INFO:root:Epoch[157] Time cost=9.542
    INFO:root:Epoch[158] Train-loss=162.141098
    INFO:root:Epoch[158] Time cost=9.768
    INFO:root:Epoch[159] Train-loss=162.124311
    INFO:root:Epoch[159] Time cost=7.155
    INFO:root:Epoch[160] Train-loss=162.013039
    INFO:root:Epoch[160] Time cost=6.147
    INFO:root:Epoch[161] Train-loss=161.954485
    INFO:root:Epoch[161] Time cost=9.121
    INFO:root:Epoch[162] Train-loss=161.913859
    INFO:root:Epoch[162] Time cost=9.936
    INFO:root:Epoch[163] Train-loss=161.830799
    INFO:root:Epoch[163] Time cost=8.612
    INFO:root:Epoch[164] Train-loss=161.768672
    INFO:root:Epoch[164] Time cost=9.722
    INFO:root:Epoch[165] Train-loss=161.689120
    INFO:root:Epoch[165] Time cost=9.478
    INFO:root:Epoch[166] Train-loss=161.598279
    INFO:root:Epoch[166] Time cost=9.466
    INFO:root:Epoch[167] Train-loss=161.551172
    INFO:root:Epoch[167] Time cost=9.419
    INFO:root:Epoch[168] Train-loss=161.488880
    INFO:root:Epoch[168] Time cost=9.457
    INFO:root:Epoch[169] Train-loss=161.410458
    INFO:root:Epoch[169] Time cost=9.504
    INFO:root:Epoch[170] Train-loss=161.340681
    INFO:root:Epoch[170] Time cost=9.866
    INFO:root:Epoch[171] Train-loss=161.281700
    INFO:root:Epoch[171] Time cost=9.526
    INFO:root:Epoch[172] Train-loss=161.215523
    INFO:root:Epoch[172] Time cost=9.511
    INFO:root:Epoch[173] Train-loss=161.152452
    INFO:root:Epoch[173] Time cost=9.498
    INFO:root:Epoch[174] Train-loss=161.058544
    INFO:root:Epoch[174] Time cost=9.561
    INFO:root:Epoch[175] Train-loss=161.036475
    INFO:root:Epoch[175] Time cost=9.463
    INFO:root:Epoch[176] Train-loss=161.009996
    INFO:root:Epoch[176] Time cost=9.629
    INFO:root:Epoch[177] Train-loss=160.853546
    INFO:root:Epoch[177] Time cost=9.518
    INFO:root:Epoch[178] Train-loss=160.860520
    INFO:root:Epoch[178] Time cost=9.395
    INFO:root:Epoch[179] Train-loss=160.810621
    INFO:root:Epoch[179] Time cost=9.452
    INFO:root:Epoch[180] Train-loss=160.683071
    INFO:root:Epoch[180] Time cost=9.411
    INFO:root:Epoch[181] Train-loss=160.674101
    INFO:root:Epoch[181] Time cost=8.784
    INFO:root:Epoch[182] Train-loss=160.554823
    INFO:root:Epoch[182] Time cost=7.265
    INFO:root:Epoch[183] Train-loss=160.536528
    INFO:root:Epoch[183] Time cost=6.108
    INFO:root:Epoch[184] Train-loss=160.525913
    INFO:root:Epoch[184] Time cost=6.349
    INFO:root:Epoch[185] Train-loss=160.399412
    INFO:root:Epoch[185] Time cost=7.364
    INFO:root:Epoch[186] Train-loss=160.380027
    INFO:root:Epoch[186] Time cost=7.651
    INFO:root:Epoch[187] Train-loss=160.272921
    INFO:root:Epoch[187] Time cost=7.309
    INFO:root:Epoch[188] Train-loss=160.243907
    INFO:root:Epoch[188] Time cost=7.162
    INFO:root:Epoch[189] Train-loss=160.194351
    INFO:root:Epoch[189] Time cost=8.941
    INFO:root:Epoch[190] Train-loss=160.130400
    INFO:root:Epoch[190] Time cost=10.242
    INFO:root:Epoch[191] Train-loss=160.073841
    INFO:root:Epoch[191] Time cost=10.528
    INFO:root:Epoch[192] Train-loss=160.021623
    INFO:root:Epoch[192] Time cost=9.482
    INFO:root:Epoch[193] Train-loss=159.938673
    INFO:root:Epoch[193] Time cost=9.465
    INFO:root:Epoch[194] Train-loss=159.885823
    INFO:root:Epoch[194] Time cost=9.523
    INFO:root:Epoch[195] Train-loss=159.886516
    INFO:root:Epoch[195] Time cost=9.599
    INFO:root:Epoch[196] Train-loss=159.797400
    INFO:root:Epoch[196] Time cost=8.675
    INFO:root:Epoch[197] Train-loss=159.705562
    INFO:root:Epoch[197] Time cost=9.551
    INFO:root:Epoch[198] Train-loss=159.738354
    INFO:root:Epoch[198] Time cost=9.919
    INFO:root:Epoch[199] Train-loss=159.619932
    INFO:root:Epoch[199] Time cost=10.121

```python
# encode test images to obtain mu and logvar which are used for sampling
[mu,logvar] = VAE.encoder(out,image_test)
#sample in the latent space
z = VAE.sampler(mu,logvar)
# decode from the latent space to obtain reconstructed images
x_construction = VAE.decoder(out,z)

```

```python
f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))
ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax1.set_title('True image')
ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax2.set_title('Learned image')
ax3.imshow(np.reshape(x_construction[999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax3.set_title('Learned image')
ax4.imshow(np.reshape(x_construction[9999,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax4.set_title('Learned image')
plt.show()
```

```python
z1 = z[:,0]
z2 = z[:,1]

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z1,z2,'ko')
plt.title("latent space")

#np.where((z1>3) & (z2<2) & (z2>0))
#select the points from the latent space
a_vec = [2,5,7,789,25,9993]
for i in range(len(a_vec)):
    ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro')
    ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]),
                xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)

f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6,  sharex='col', sharey='row',figsize=(12,2.5))
for i in range(len(a_vec)):
    eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)
    eval('ax%d' %(i)).set_title('z%d'%i)

plt.show()
```

Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right.


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, C.D. Cooper, G.F. Forsyth.

# Riding the wave

## Numerical schemes for hyperbolic PDEs

Welcome back! This is the second notebook of *Riding the wave: Convection problems*, the third module of  ["Practical Numerical Methods with Python"](https://openedx.seas.gwu.edu/courses/course-v1:MAE+MAE6286+2017/about).

The first notebook of this module discussed conservation laws and developed the non-linear traffic equation. We learned about the effect of the wave speed on the stability of the numerical method, and on the CFL number. We also realized that the forward-time/backward-space difference scheme really has many limitations: it cannot deal with wave speeds that move in more than one direction. It is also first-order accurate in space and time, which often is just not good enough. This notebook will introduce some new numerical schemes for conservation laws, continuing with the traffic-flow problem as motivation.

## Red light!

Let's explore the behavior of different numerical schemes for a moving shock wave. In the context of the traffic-flow model of the previous notebook, imagine a very busy road and a red light at $x=4$. Cars accumulate quickly in the front, where we have the maximum allowed density of cars between $x=3$ and $x=4$, and there is an incoming traffic of 50% the maximum allowed density $(\rho = 0.5\rho_{\rm max})$.

Mathematically, this is:

$$
\begin{equation}
\rho(x,0) = \left\{
\begin{array}{cc}
0.5 \rho_{\rm max} & 0 \leq x < 3  \\
\rho_{\rm max} & 3 \leq x \leq 4 \\
\end{array}
\right.
\end{equation}
$$

Let's find out what the initial condition looks like.

```python
import numpy
from matplotlib import pyplot
%matplotlib inline
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
def rho_red_light(x, rho_max):
    """
    Computes the "red light" initial condition with shock.

    Parameters
    ----------
    x : numpy.ndaray
        Locations on the road as a 1D array of floats.
    rho_max : float
        The maximum traffic density allowed.

    Returns
    -------
    rho : numpy.ndarray
        The initial car density along the road
        as a 1D array of floats.
    """
    rho = rho_max * numpy.ones_like(x)
    mask = numpy.where(x < 3.0)
    rho[mask] = 0.5 * rho_max
    return rho
```

```python
# Set parameters.
nx = 81  # number of locations on the road
L = 4.0  # length of the road
dx = L / (nx - 1)  # distance between two consecutive locations
nt = 40  # number of time steps to compute
rho_max = 10.0  # maximum taffic density allowed
u_max = 1.0  # maximum speed traffic

# Get the road locations.
x = numpy.linspace(0.0, L, num=nx)

# Compute the initial traffic density.
rho0 = rho_red_light(x, rho_max)
```

```python
# Plot the initial traffic density.
fig = pyplot.figure(figsize=(6.0, 4.0))
pyplot.xlabel(r'$x$')
pyplot.ylabel(r'$\rho$')
pyplot.grid()
line = pyplot.plot(x, rho0,
                   color='C0', linestyle='-', linewidth=2)[0]
pyplot.xlim(0.0, L)
pyplot.ylim(4.0, 11.0)
pyplot.tight_layout()
```

The question we would like to answer is: **How will cars accumulate at the red light?**

We will solve this problem using different numerical schemes, to see how they perform. These schemes are:

   * Lax-Friedrichs
   * Lax-Wendroff
   * MacCormack

Before we do any coding, let's think about the equation a little bit. The wave speed $u_{\rm wave}$ is $-1$ for $\rho = \rho_{\rm max}$ and $\rho \leq \rho_{\rm max}/2$, making all velocities negative. We should see a solution moving left, maintaining the shock geometry.

#### Figure 1. The exact solution is a shock wave moving left.

Now to some coding! First, let's define some useful functions and prepare to make some nice animations later.

```python
def flux(rho, u_max, rho_max):
    """
    Computes the traffic flux F = V * rho.

    Parameters
    ----------
    rho : numpy.ndarray
       Traffic density along the road as a 1D array of floats.
    u_max : float
        Maximum speed allowed on the road.
    rho_max : float
        Maximum car density allowed on the road.

    Returns
    -------
    F : numpy.ndarray
        The traffic flux along the road as a 1D array of floats.
    """
    F = rho * u_max * (1.0 - rho / rho_max)
    return F
```

Before we investigate different schemes, let's create the function to update the Matplotlib figure during the animation.

```python
from matplotlib import animation
from IPython.display import HTML
```

```python
def update_plot(n, rho_hist):
    """
    Update the line y-data of the Matplotlib figure.

    Parameters
    ----------
    n : integer
        The time-step index.
    rho_hist : list of numpy.ndarray objects
        The history of the numerical solution.
    """
    fig.suptitle('Time step {:0>2}'.format(n))
    line.set_ydata(rho_hist[n])
```

## Lax-Friedrichs scheme

Recall the conservation law for vehicle traffic, resulting in the following equation for the traffic density:

$$
\begin{equation}
\frac{\partial \rho}{\partial t} + \frac{\partial F}{\partial x} = 0
\end{equation}
$$

$F$ is the *traffic flux*, which in the linear traffic-speed model is given by:

$$
\begin{equation}
F = \rho u_{\rm max} \left(1-\frac{\rho}{\rho_{\rm max}}\right)
\end{equation}
$$

In the time variable, the natural choice for discretization is always a forward-difference formula; time invariably moves forward!

$$
\begin{equation}
\frac{\partial \rho}{\partial t}\approx \frac{1}{\Delta t}( \rho_i^{n+1}-\rho_i^n )
\end{equation}
$$

As is usual, the discrete locations on the 1D spatial grid are denoted by indices $i$ and the discrete time instants are denoted by indices $n$.

In a convection problem, using first-order discretization in space leads to excessive numerical diffusion (as you probably observed in [Lesson 1 of Module 2](https://nbviewer.jupyter.org/github/numerical-mooc/numerical-mooc/blob/master/lessons/02_spacetime/02_01_1DConvection.ipynb)). The simplest approach to get second-order accuracy in space is to use a central difference:

$$
\begin{equation}
\frac{\partial F}{\partial x} \approx \frac{1}{2\Delta x}( F_{i+1}-F_{i-1})
\end{equation}
$$

But combining these two choices for time and space discretization in the convection equation has catastrophic results! The "forward-time, central scheme" (FTCS) is **unstable**. (Go on: try it; you know you want to!)

The Lax-Friedrichs scheme was proposed by Lax (1954) as a clever trick to stabilize the forward-time, central scheme. The idea was to replace the solution value at $\rho^n_i$ by the average of the values at the neighboring grid points. If we do that replacement, we get the following discretized equation:

$$
\begin{equation}
\frac{\rho_i^{n+1}-\frac{1}{2}(\rho^n_{i+1}+\rho^n_{i-1})}{\Delta t} = -\frac{F^n_{i+1}-F^n_{i-1}}{2 \Delta x}
\end{equation}
$$

Take a careful look: the difference formula no longer uses the value at $\rho^n_i$ to obtain $\rho^{n+1}_i$. The stencil of the Lax-Friedrichs scheme is slightly different than that for the forward-time, central scheme.

#### Figure 2. Stencil of the forward-time/central scheme.

#### Figure 3. Stencil of the Lax-Friedrichs scheme.

This numerical discretization is **stable**. Unfortunately, substituting  $\rho^n_i$ by the average of its neighbors introduces a first-order error. _Nice try, Lax!_

To implement the scheme in code, we need to isolate the value at the next time step, $\rho^{n+1}_i$, so we can write a time-stepping loop:

$$
\begin{equation}
\rho_i^{n+1} = \frac{1}{2}(\rho^n_{i+1}+\rho^n_{i-1}) - \frac{\Delta t}{2 \Delta x}(F^n_{i+1}-F^n_{i-1})
\end{equation}
$$

The function below implements Lax-Friedrichs for our traffic model.  All the schemes in this notebook are wrapped in their own functions to help with displaying animations of the results.  This is also good practice for developing modular, reusable code.

In order to display animations, we're going to hold the results of each time step in the variable `rho`, a 2D array.  The resulting array `rho_n` has `nt` rows and `nx` columns.

```python
def lax_friedrichs(rho0, nt, dt, dx, bc_values, *args):
    """
    Computes the traffic density on the road
    at a certain time given the initial traffic density.
    Integration using Lax-Friedrichs scheme.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial traffic density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_values : 2-tuple of floats
        The value of the density at the first and last locations.
    args : list or tuple
        Positional arguments to be passed to the flux function.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Advance in time using Lax-Friedrichs scheme.
        rho[1:-1] = (0.5 * (rho[:-2] + rho[2:]) -
                     dt / (2.0 * dx) * (F[2:] - F[:-2]))
        # Set the value at the first location.
        rho[0] = bc_values[0]
        # Set the value at the last location.
        rho[-1] = bc_values[1]
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

### Lax-Friedrichs with $\frac{\Delta t}{\Delta x}=1$

We are now all set to run! First, let's try with CFL=1

```python
# Set the time-step size based on CFL limit.
sigma = 1.0
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = lax_friedrichs(rho0, nt, dt, dx, (rho0[0], rho0[-1]),
                          u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

##### Think

* What do you see in the animation above? How does the numerical solution compare with the exact solution (a left-traveling shock wave)?
* What types of errors do you think we see?
* What do you think of the Lax-Friedrichs scheme, so far?

### Lax-Friedrichs with $\frac{\Delta t}{\Delta x} = 0.5$

Would the solution improve if we use smaller time steps? Let's check that!

```python
# Set the time-step size based on CFL limit.
sigma = 0.5
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = lax_friedrichs(rho0, nt, dt, dx, (rho0[0], rho0[-1]),
                          u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

##### Dig deeper

Notice the strange "staircase" behavior on the leading edge of the wave?  You may be interested to learn more about this: a feature typical of what is sometimes called "odd-even decoupling." Last year we published a collection of lessons in Computational Fluid Dynamics, called _CFD Python_, where we discuss [odd-even decoupling](https://nbviewer.jupyter.org/github/barbagroup/CFDPython/blob/14b56718ac1508671de66bab3fe432e93cb59fcb/lessons/19_Odd_Even_Decoupling.ipynb).

* How does this solution compare with the previous one, where the Courant number was $\frac{\Delta t}{\Delta x}=1$?

## Lax-Wendroff scheme

The Lax-Friedrichs method uses a clever trick to stabilize the central difference in space for convection, but loses an order of accuracy in doing so. First-order methods are just not good enough for convection problems, especially when you have sharp gradients (shocks).

The Lax-Wendroff (1960) method was the _first_ scheme ever to achieve second-order accuracy in both space and time. It is therefore a landmark in the history of computational fluid dynamics.

To develop the Lax-Wendroff scheme, we need to do a bit of work. Sit down, grab a notebook and grit your teeth. We want you to follow this derivation in your own hand. It's good for you! Start with the Taylor series expansion (in the time variable) about $\rho^{n+1}$:

$$
\begin{equation}
\rho^{n+1} = \rho^n + \frac{\partial\rho^n}{\partial t} \Delta t + \frac{(\Delta t)^2}{2}\frac{\partial^2\rho^n}{\partial t^2} + \ldots
\end{equation}
$$

For the conservation law with $F=F(\rho)$, and using our beloved chain rule, we can write:

$$
\begin{equation}
\frac{\partial \rho}{\partial t} = -\frac{\partial F}{\partial x} = -\frac{\partial F}{\partial \rho} \frac{\partial \rho}{\partial x} = -J \frac{\partial \rho}{\partial x}
\end{equation}
$$

where

$$
\begin{equation}
J = \frac{\partial F}{\partial \rho} = u _{\rm max} \left(1-2\frac{\rho}{\rho_{\rm max}} \right)
\end{equation}
$$

is the _Jacobian_ for the traffic model. Next, we can do a little trickery:

$$
\begin{equation}
\frac{\partial F}{\partial t} = \frac{\partial F}{\partial \rho} \frac{\partial \rho}{\partial t} = J \frac{\partial \rho}{\partial t} = -J \frac{\partial F}{\partial x}
\end{equation}
$$

In the last step above, we used the differential equation of the traffic model to replace the time derivative by a spatial derivative. These equivalences imply that

$$
\begin{equation}
\frac{\partial^2\rho}{\partial t^2} = \frac{\partial}{\partial x} \left( J \frac{\partial F}{\partial x} \right)
\end{equation}
$$

Let's use all this in the Taylor expansion:

$$
\begin{equation}
\rho^{n+1} = \rho^n - \frac{\partial F^n}{\partial x} \Delta t + \frac{(\Delta t)^2}{2} \frac{\partial}{\partial x} \left(J\frac{\partial F^n}{\partial x} \right)+ \ldots
\end{equation}
$$

We can now reorganize this and discretize the spatial derivatives with central differences to get the following discrete equation:

$$
\begin{equation}
\frac{\rho_i^{n+1} - \rho_i^n}{\Delta t} = -\frac{F^n_{i+1}-F^n_{i-1}}{2 \Delta x} + \frac{\Delta t}{2} \left(\frac{(J \frac{\partial F}{\partial x})^n_{i+\frac{1}{2}}-(J \frac{\partial F}{\partial x})^n_{i-\frac{1}{2}}}{\Delta x}\right)
\end{equation}
$$

Now, approximate the rightmost term (inside the parenthesis) in the above equation as follows:
\begin{equation} \frac{J^n_{i+\frac{1}{2}}\left(\frac{F^n_{i+1}-F^n_{i}}{\Delta x}\right)-J^n_{i-\frac{1}{2}}\left(\frac{F^n_i-F^n_{i-1}}{\Delta x}\right)}{\Delta x}\end{equation}

Then evaluate the Jacobian at the midpoints by using averages of the points on either side:

\begin{equation}\frac{\frac{1}{2 \Delta x}(J^n_{i+1}+J^n_i)(F^n_{i+1}-F^n_i)-\frac{1}{2 \Delta x}(J^n_i+J^n_{i-1})(F^n_i-F^n_{i-1})}{\Delta x}.\end{equation}

Our equation now reads:

\begin{align}
&\frac{\rho_i^{n+1} - \rho_i^n}{\Delta t} =
-\frac{F^n_{i+1}-F^n_{i-1}}{2 \Delta x} + \cdots \\ \nonumber
&+ \frac{\Delta t}{4 \Delta x^2} \left( (J^n_{i+1}+J^n_i)(F^n_{i+1}-F^n_i)-(J^n_i+J^n_{i-1})(F^n_i-F^n_{i-1})\right)
\end{align}

Solving for $\rho_i^{n+1}$:

\begin{align}
&\rho_i^{n+1} = \rho_i^n - \frac{\Delta t}{2 \Delta x} \left(F^n_{i+1}-F^n_{i-1}\right) + \cdots \\ \nonumber
&+ \frac{(\Delta t)^2}{4(\Delta x)^2} \left[ (J^n_{i+1}+J^n_i)(F^n_{i+1}-F^n_i)-(J^n_i+J^n_{i-1})(F^n_i-F^n_{i-1})\right]
\end{align}

with

\begin{equation}J^n_i = \frac{\partial F}{\partial \rho} = u_{\rm max} \left(1-2\frac{\rho^n_i}{\rho_{\rm max}} \right).\end{equation}

Lax-Wendroff is a little bit long.  Remember that you can use \ slashes to split up a statement across several lines.  This can help make code easier to parse (and also easier to debug!).

```python
def jacobian(rho, u_max, rho_max):
    """
    Computes the Jacobian for our traffic model.

    Parameters
    ----------
    rho : numpy.ndarray
       Traffic density along the road as a 1D array of floats.
    u_max : float
        Maximum speed allowed on the road.
    rho_max : float
        Maximum car density allowed on the road.

    Returns
    -------
    J : numpy.ndarray
        The Jacobian as a 1D array of floats.
    """
    J = u_max * (1.0 - 2.0 * rho / rho_max)
    return J
```

```python
def lax_wendroff(rho0, nt, dt, dx, bc_values, *args):
    """
    Computes the traffic density on the road
    at a certain time given the initial traffic density.
    Integration using Lax-Wendroff scheme.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial traffic density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_values : 2-tuple of floats
        The value of the density at the first and last locations.
    args : list or tuple
        Positional arguments to be passed to the
        flux and Jacobien functions.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Compute the Jacobian.
        J = jacobian(rho, *args)
        # Advance in time using Lax-Wendroff scheme.
        rho[1:-1] = (rho[1:-1] -
                     dt / (2.0 * dx) * (F[2:] - F[:-2]) +
                     dt**2 / (4.0 * dx**2) *
                     ((J[1:-1] + J[2:]) * (F[2:] - F[1:-1]) -
                      (J[:-2] + J[1:-1]) * (F[1:-1] - F[:-2])))
        # Set the value at the first location.
        rho[0] = bc_values[0]
        # Set the value at the last location.
        rho[-1] = bc_values[1]
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

Now that's we've defined a function for the Lax-Wendroff scheme, we can use the same procedure as above to animate and view our results.

### Lax-Wendroff with $\frac{\Delta t}{\Delta x}=1$

```python
# Set the time-step size based on CFL limit.
sigma = 1.0
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = lax_wendroff(rho0, nt, dt, dx, (rho0[0], rho0[-1]),
                        u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

Interesting! The Lax-Wendroff method captures the sharpness of the shock much better than the Lax-Friedrichs scheme, but there is a new problem: a strange wiggle appears right at the tail of the shock. This is typical of many second-order methods: they introduce _numerical oscillations_ where the solution is not smooth. Bummer.

### Lax-Wendroff with $\frac{\Delta t}{\Delta x} =0.5$

How do the oscillations at the shock front vary with changes to the CFL condition? You might think that the solution will improve if you make the time step smaller ... let's see.

```python
# Set the time-step size based on CFL limit.
sigma = 0.5
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = lax_wendroff(rho0, nt, dt, dx, (rho0[0], rho0[-1]),
                        u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

Eek! The numerical oscillations got worse. Double bummer!

Why do we observe oscillations with second-order methods? This is a question of fundamental importance!

## MacCormack Scheme

The numerical oscillations that you observed with the Lax-Wendroff method on the traffic model can become severe in some problems. But actually the main drawback of the Lax-Wendroff method is having to calculate the Jacobian in every time step. With more complicated equations (like the Euler equations), calculating the Jacobian is a large computational expense.

Robert W. MacCormack introduced the first version of his now-famous method at the 1969 AIAA Hypervelocity Impact Conference, held in Cincinnati, Ohio, but the paper did not at first catch the attention of the aeronautics community. The next year, however, he presented at the 2nd International Conference on Numerical Methods in Fluid Dynamics at Berkeley. His paper there (MacCormack, 1971) was a landslide. MacCormack got a promotion and continued to work on applications of his method to the compressible Navier-Stokes equations. In 1973, NASA gave him the prestigious H. Julian Allen award for his work.

The MacCormack scheme is a two-step method, in which the first step is called a _predictor_ and the second step is called a _corrector_. It achieves second-order accuracy in both space and time. One version is as follows:

$$
\begin{equation}
\rho^*_i = \rho^n_i - \frac{\Delta t}{\Delta x} (F^n_{i+1}-F^n_{i}) \ \ \ \ \ \ \text{(predictor)}
\end{equation}
$$

$$
\begin{equation}
\rho^{n+1}_i = \frac{1}{2} (\rho^n_i + \rho^*_i - \frac{\Delta t}{\Delta x} (F^*_i - F^{*}_{i-1})) \ \ \ \ \ \ \text{(corrector)}
\end{equation}
$$

If you look closely, it appears like the first step is a forward-time/forward-space scheme, and the second step is like a forward-time/backward-space scheme (these can also be reversed), averaged with the first result. What is so cool about this? You can compute problems with left-running waves and right-running waves, and the MacCormack scheme gives you a stable method (subject to the CFL condition). Nice! Let's try it.

```python
def maccormack(rho0, nt, dt, dx, bc_values, *args):
    """
    Computes the traffic density on the road
    at a certain time given the initial traffic density.
    Integration using MacCormack scheme.

    Parameters
    ----------
    rho0 : numpy.ndarray
        The initial traffic density along the road
        as a 1D array of floats.
    nt : integer
        The number of time steps to compute.
    dt : float
        The time-step size to integrate.
    dx : float
        The distance between two consecutive locations.
    bc_values : 2-tuple of floats
        The value of the density at the first and last locations.
    args : list or tuple
        Positional arguments to be passed to the flux function.

    Returns
    -------
    rho_hist : list of numpy.ndarray objects
        The history of the car density along the road.
    """
    rho_hist = [rho0.copy()]
    rho = rho0.copy()
    rho_star = rho.copy()
    for n in range(nt):
        # Compute the flux.
        F = flux(rho, *args)
        # Predictor step of the MacCormack scheme.
        rho_star[1:-1] = (rho[1:-1] -
                          dt / dx * (F[2:] - F[1:-1]))
        # Compute the flux.
        F = flux(rho_star, *args)
        # Corrector step of the MacCormack scheme.
        rho[1:-1] = 0.5 * (rho[1:-1] + rho_star[1:-1] -
                           dt / dx * (F[1:-1] - F[:-2]))
        # Set the value at the first location.
        rho[0] = bc_values[0]
        # Set the value at the last location.
        rho[-1] = bc_values[1]
        # Record the time-step solution.
        rho_hist.append(rho.copy())
    return rho_hist
```

### MacCormack with $\frac{\Delta t}{\Delta x} = 1$

```python
# Set the time-step size based on CFL limit.
sigma = 1.0
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = maccormack(rho0, nt, dt, dx, (rho0[0], rho0[-1]),
                      u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

### MacCormack with $\frac{\Delta t}{\Delta x}= 0.5$

Once again, we ask: how does the CFL number affect the errors? Which one gives better results? You just have to try it.

```python
# Set the time-step size based on CFL limit.
sigma = 0.5
dt = sigma * dx / u_max  # time-step size

# Compute the traffic density at all time steps.
rho_hist = maccormack(rho0, nt, dt, dx, (rho0[0], rho0[-1]),
                      u_max, rho_max)
```

```python
# Create an animation of the traffic density.
anim = animation.FuncAnimation(fig, update_plot,
                               frames=nt, fargs=(rho_hist,),
                               interval=100)
# Display the video.
HTML(anim.to_html5_video())
```

##### Dig Deeper

You can also obtain a MacCormack scheme by reversing the predictor and corrector steps. For shocks, the best resolution will occur when the difference in the predictor step is in the direction of propagation. Try it out! Was our choice here the ideal one? In which case is the shock better resolved?

##### Challenge task

In the *red light* problem, $\rho \geq \rho_{\rm max}/2$, making the wave speed negative at all points . You might be wondering why we introduced these new methods; couldn't we have just used a forward-time/forward-space scheme? But, what if $\rho_{\rm in} < \rho_{\rm max}/2$? Now, a whole region has negative wave speeds and forward-time/backward-space is unstable.

* How do Lax-Friedrichs, Lax-Wendroff and MacCormack behave in this case? Try it out!

* As you decrease $\rho_{\rm in}$, what happens to the velocity of the shock? Why do you think that happens?

## References

* Peter D. Lax (1954), "Weak solutions of nonlinear hyperbolic equations and their numerical computation," _Commun. Pure and Appl. Math._, Vol. 7, pp. 159–193.

* Peter D. Lax and Burton Wendroff (1960), "Systems of conservation laws,"  _Commun. Pure and Appl. Math._, Vol. 13, pp. 217–237.

* R. W. MacCormack (1969), "The effect of viscosity in hypervelocity impact cratering," AIAA paper 69-354. Reprinted on _Journal of Spacecraft and Rockets_, Vol. 40, pp. 757–763 (2003). Also on _Frontiers of Computational Fluid Dynamics_, edited by D. A. Caughey, M. M. Hafez (2002), chapter 2: [read on Google Books](http://books.google.com/books?id=QBsnMOz_8qcC&lpg=PA27&ots=uqCeuH1U6S&lr&pg=PA27#v=onepage&q&f=false).

* R. W. MacCormack (1971), "Numerical solution of the interaction of a shock wave with a laminar boundary layer," _Proceedings of the 2nd Int. Conf. on Numerical Methods in Fluid Dynamics_, Lecture Notes in Physics, Vol. 8, Springer, Berlin, pp. 151–163.

---

###### The cell below loads the style of the notebook.

```python
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<link href='https://fonts.googleapis.com/css?family=Source+Code+Pro' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
    margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
    font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
    font-family: "Source Code Pro";
    font-size: 90%;
}

.alert-box {
    padding:10px 10px 10px 36px;
    margin:5px;
}

.success {
    color:#666600;
    background:rgb(240,242,229);
}
</style>


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli and Rajath Kumar Mysore Pradeep Kumar

Note:  This material largely follows the text "Numerical Linear Algebra" by Trefethen and Bau (SIAM, 1997) and is meant as a guide and supplement to the material presented there.

```python
from __future__ import print_function

%precision 6
%matplotlib inline
import numpy
import matplotlib.pyplot as plt

```

# Understanding the Singular Value Decomposition

## Factorizations:  A review

**Matrix Factorization** is a fundamental concept in Linear Algebra and refers to the ability to write general matrices as the products of simpler matrices with special properties.

In particular, all of the great algorithms encountered in numerical linear Algebra can be succinctly described by their factorizations.  Examples include

### **The LU decomposition**
$$PA=LU$$

* **Algorithms** *Gaussian Elimination with partial pivoting*
* **Factorization** $P$ Permutation matrix (from pivoting), $L,U$: Lower and upper triangular matrices
* **Application** direct solution of $A\mathbf{x}=\mathbf{b}$ for $A\in\mathbb{R}^{n\times n}$

### **The QR decomposition**
$$A=QR$$

* **Algorithms** Gram-Schmidt Orthogonalization, Householder Triangularization
* **Factorization**  $Q$:  Orthonormal Basis for $C(A)$, $R$: upper triangular matrix
* **Applications** Least-Squares solutions, Projection problems,  Eigenvalues $QR/RQ$

### **EigenProblems**
$$ AX = X\Lambda, \quad (A\in\mathbb{R}^{n\times n})$$

* Diagonalizable matrices $A = X\Lambda X^{-1}$
* Hermitian/Symmetric Matrices $A = Q\Lambda Q^T$
* Schur Factorization $A = QTQ^T$

* **Algorithms**: Power Method, Inverse Power with shifts,  $QR/RQ$
* **Factorization**-- $X$, $Q$: matrix of Eigenvectors,  $\Lambda$ and diagonal matrix of eigenvalues
* **Application** Solution of Dynamical systems,  Iterative Maps, Markov Chains,  Vibrational analysis$\ldots$, Quantum Mechanics

*But perhaps the most beautiful factorization, which contains aspects of all of these problems$\ldots$

## Singular Value Decomposition (SVD)
$$
    A = U\Sigma V^{T}
$$
for *any* $A\in\mathbb{R}^{m\times n}$, where
* $U \in \mathbb R^{m \times m}$ and is the orthogonal matrix whose columns are the eigenvectors of $AA^{T}$
* $V \in \mathbb R^{n \times n}$ and is the orthogonal matrix whose columns are the eigenvectors of $A^{T}A$
* $\Sigma \in \mathbb R^{m \times n}$ and is a diagonal matrix with elements $\sigma_{1}, \sigma_{2}, \sigma_{3}, ... \sigma_{r}$ where $r = rank(A)$ corresponding to the square roots of the eigenvalues of $A^{T}A$. They are called the singular values of $A$ and are positive arranged in descending order. ($\sigma_{1} \geq \sigma_{2} \geq \sigma_{3} \geq ... \sigma_{r} > 0$).

Or a picture is worth a thousand words...

### Existence and Uniqueness

Every matrix $A \in \mathbb{C}^{m \times n}$ has a singular value decomposition. Furthermore, the singular values $\{\sigma_{j}\}$ are uniquely determined, and if $A$ is square and the $\sigma_{j}$ are distinct, the left and right singular vectors $\{u_{j}\}$ and $\{v_{j}\}$ are uniquely determined up to complex signs (i.e., complex scalar factors of absolute value 1).

### Eigenvalue Decomposition vs. SVD Decomposition

Let the matrix $X$ contain the eigenvectors of $A$ which are linearly independent, then we can write a decomposition of the matrix $A$ as
$$
    A = X \Lambda X^{-1}.
$$

How does this differ from the SVD?
 - The basis of the SVD representation differs from the eigenvalue decomposition
   - The basis vectors are not in general orthogonal for the eigenvalue decomposition where it is for the SVD
   - The SVD effectively contains two basis sets.
 - All matrices have an SVD decomposition whereas not all have eigenvalue decompositions.

#### Applications
* Matrix Pseudo-Inverse for ill-conditions problems
* Principal Component analysis
* Total Least Squares
* Image Compression
* Data Analysis for interpretation of high-dimensional data
* and more$\ldots$

### A conceptual computation of the SVD (not how it's really done)

Given $A = U\Sigma V^T$, we can form the matrix

$$
    A^TA = V\Sigma^T\Sigma V^T
$$

where $V\in\mathbb{R}^{n\times n}$ is unitary and
$$
    \Sigma^T\Sigma = \begin{bmatrix}
                \sigma_1^2 & & &    \\
                    & \sigma_2^2 & & \\
                   & & \ddots &   \\
                   & & &   \sigma_n^2  \\
                                   \end{bmatrix}
$$

However, because $A^TA$ is clearly **Symmetric** it can also has the eigen factorization

$$
    A^TA = Q\Lambda Q^T
$$

where $Q$ is unitary and $\Lambda$ is real diagonal.

Moreover, we can show that $A^T A$ is at least Positive semi-definite so all the eigenvalues are $\geq 0$.

In particular,  if $\mathrm{rank}(A)=r$ we know that $r$ eigenvalues are $>0$ and $n-r$ eigenvalues are equal to zero.

Given that

$$\begin{align}
    A^TA &= V\Sigma^T\Sigma V^T \\
        &= Q\Lambda Q^T \\
\end{align}
$$

If we simply order the eigenvalues from largest to smallest (and rearrange the columns of $Q$ to match), then we can make the association

$$V = Q\quad, \Sigma^T\Sigma = \Lambda$$ or

$$\sigma_i^2 = \lambda_i\quad\textrm{or}\quad \sigma_i = \sqrt{\lambda_i}$$

A similar argument can be made about $U$ and the eigenvectors of $AA^T$,  however, a better way to think about the relationship between $U$, $V$ and $\Sigma$ is to rearrange the SVD as

$$
    AV = U\Sigma
$$

or we can look at this column-by-column

$$
    A\mathbf{v}_i = \sigma_i\mathbf{u}_i
$$

(in the same way the Eigen decomposition $AX = X\Lambda$ is equivalent to $A\mathbf{x}_i = \lambda_i\mathbf{x}_i$)

or rearranging
$$
    \mathbf{u}_i = \frac{A\mathbf{v}_i}{\sigma_i},\quad i=1,2,\ldots,r
$$

lets us solve for the first $r$ columns of $U$ given $V$ and $\Sigma$.

As we'll show, we usually don't need to solve for the remaing $m-r$ columns of $U$ for reasons that relate to the "4 fundamental subspaces of $A$"

### The Strangian view of the 4 Subspaces: a quick refresher

### The Big idea:  The columns of $U$ and $V$ contain orthonormal bases for the 4 fundamental subspaces

Returning to $AV = U\Sigma$ it follows that

$$
\begin{align}
        A\mathbf{v}_i &= \sigma_i\mathbf{u_i},\quad\mathrm{for}\quad i=1,2\ldots r \\
        A\mathbf{v}_i &= \mathbf{0},\quad\mathrm{for}\quad i=r+1,\ldots n \\
\end{align}
$$

Therefore the last $n-r$ columns of $V$ must be in the Null space of $A$, and in fact must form an orthonormal basis for the Null space $N(A)$.

Since the first $r$ columns of $V$ are all orthogonal to the last $n-r$ columns,  they must form an orthonormal basis for the row space $C(A^T)$.

Likewise,  since the first $r$ columns of $U$ satisfy

$$
    \mathbf{u}_i = \frac{A\mathbf{v}_i}{\sigma_i},\quad i=1,2,\ldots,r
$$

they must form an orthonormal basis for $C(A)$

Leaving only the last $m-r$ columns of $U$ as an orthonormal basis for the left null space $N(A^T)$

Again a picture is worth a lot here

$$
A\begin{bmatrix} &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
 \mathbf{v}_1 & \cdots & \mathbf{v}_r & | &\mathbf{v}_{r+1} & \cdots & \mathbf{v}_n \\
 &  &  & |&  &  &  \\
  &  &  & |&  &  &  \\
   \end{bmatrix} = \begin{bmatrix} &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
 \mathbf{u}_1 & \cdots & \mathbf{u}_r & | &\mathbf{u}_{r+1} & \cdots & \mathbf{u}_m \\
 &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
  &  &  & |&  &  &  \\
    \end{bmatrix} \begin{bmatrix} \sigma_1 &  &  & |  &  &    \\
 & \ddots &  & |   &   \mathbf{0}  \\
 &  & \sigma_r &|    &     \\
 -&- &-  & | & - &  -\\
 & \mathbf{0} &  & |   & \mathbf{0}   \\
 &  &  & |  &    \\
  &  &  & |  &     \\
    \end{bmatrix}
$$

## The Economy (or Skinny) SVD

As it turns out, because of all the 0's  all we actually need to reconstruct $A$ is the the first $r$ columns of $U$, and $V$,  and the square sub-block of $\Sigma$ with just the singular values.  i.e.

$$
A\begin{bmatrix} &  &  & \\
 &  &  &    \\
 \mathbf{v}_1 & \cdots & \mathbf{v}_r   \\
 &  &  &   \\
  &  &  &    \\
   \end{bmatrix} = \begin{bmatrix} &  &  &   \\
 &  &  &    \\
  &  &  &    \\
 \mathbf{u}_1 & \cdots & \mathbf{u}_r     \\
 &  &  &    \\
 &  &  &    \\
  &  &  &    \\
    \end{bmatrix} \begin{bmatrix} \sigma_1 &  &        \\
 & \ddots &    \\
 &  & \sigma_r       \\
    \end{bmatrix}
$$
or

$$
    A = U_r\Sigma_r V_r^T
$$

And it is this object (The Economy (or Skinny) SVD), that helps us understand what the SVD does.

### Spectral Theorem

The Economy SVD can also be written as
$$
    A = U_r\Sigma_r V_r^T = \sum_{i=1}^r \sigma_i \mathbf{u}_i\mathbf{v}^T_i
$$

Which says we can expand $A$ as a series of rank-1 matrices $\mathbf{u}_i\mathbf{v}^T_i$ weighted by the singular values...

and it is this picture that leads to many of the important approximating properties of the SVD.

### Matrix Multiplication

Consider the Matrix Vector product $A\mathbf{x}$ which maps a vector $\mathbf{x}$ in the row space $C(A^T)$ to its image in column space $C(A)$.  In the context of the Economy SVD
$$
    A\mathbf{x} = U_r\Sigma_r V_r^T\mathbf{x}
$$
or (since $V_r^TV_r = I$)

$$
\begin{align}
    A\mathbf{x} &= U_r\Sigma_r V_r^T\mathbf{x} \\
    &= U_r\Sigma_r (V_r^T V_r) V_r^T\mathbf{x} \\
    & = (U_r\Sigma_r V_r^T) (V_r V_r^T)\mathbf{x} \\
    & = A \mathbf{x}^+
\end{align}
$$
where $$\mathbf{x}^+ = (V_r V_r^T)\mathbf{x}$$
is the projection of $\mathbf{x}$ onto the row space $C(A^T)$

**Question**: If we wanted it, how would we  find the projection of $\mathbf{x}$ onto the Null space of $A$?

### SVD and the Matrix Inverse

If a matrix is square and invertible, it implies that $m=n=r$ and $U$, $\Sigma$ and $V$ are all square invertible matrices,  therefore if

$$
    A = U\Sigma V^T
$$
then

$$
    A^{-1} = V\Sigma^{-1} U^T
$$
where
$$
    \Sigma^{-1} = \begin{bmatrix} 1/\sigma_1 &  &    &    \\
 & 1/\sigma_2 &   & \\
  &   &  \ddots &  \\
 &  &  & 1/\sigma_n       \\
    \end{bmatrix}
$$

and clearly
$$
    A^{-1}\mathbf{b} = \mathbf{x}
$$
maps any vector $\mathbf{b}\in C(A)$ back to a unique vector $\mathbf{x}\in C(A^T)$

### SVD and the Matrix Pseudo-Inverse

But suppose $A$ is not square.  Then it cannot have an inverse,  however, the SVD allows us to define a "Pseudo-Inverse"

Given, the skinny SVD

$$
    A = U_r\Sigma_r V_r^T
$$

we can define

$$
    A^{+} = V_r \Sigma_r^{-1} U_r^T
$$
because $\Sigma_r$ is square and invertible (but it's size is set by the rank $r$)

### Action of the  Pseudo-Inverse

In general, if $A$ is not square,  the problem $A\mathbf{x} = \mathbf{b}$ has either no solution or an infinite number of solutions.  However, it always has a minimal least square's solution given by
$$
    x^{+} = A^{+}\mathbf{b}
$$
which maps any vector $\mathbf{b}\in \mathbb{R}^m$ to a unique vector $\mathbf{x}^+\in C(A^T)$

As we did with $A\mathbf{x}$, we can consider the action of because $A^{+}$ on any vector $\mathbf{b}$ as a projection problem.
$$
\begin{align}
    A^{+}\mathbf{b} &= V_r\Sigma^{-1}_r U_r^T\mathbf{b} \\
    &= V_r\Sigma^{-1}_r (U_r^T U_r) U_r^T\mathbf{b} \\
    & = (V_r\Sigma^{-1}_r U_r^T) (U_r U_r^T)\mathbf{b} \\
    & = A^{+} \mathbf{p}
\end{align}
$$
where $$\mathbf{p}  = (U_r U_r^T)\mathbf{b}$$
is projection of $\mathbf{b}$ onto $C(A)$

### The minimal Least-Squares solution

It should be clear that
$$
    \mathbf{x}^+ = A^{+}\mathbf{b}
$$
lies entirely in the Row space of $A$ (Why?)

Therefore it has no component in the $N(A)$ and is the shortest solution to $A\hat{\mathbf{x}}=\mathbf{p}$ which is the least squares problem

### The Pseudo-Inverse and ill-conditioned problems

Technically, a square matrix is either invertible or singular,  however, in many real problems, a matrix can be close to singular or "ill-conditioned".

Again, the condition number of a matrix defined by an induced $p$ norm is
$$
    \kappa_p(A) = ||A||_p ||A^{-1}||_p
$$

In particular,  for $p=2$, it is easy to show (using the SVD) that for a square matrix
$$
    \kappa_2(A) =\frac{\sigma_1}{\sigma_n}
$$

and thus for a singular matrix with $r<n$, $\kappa_2 =\infty$.

###  ill-conditioned matrices

However, it is possible that while $\sigma_n>0$,  it is significantly smaller than $\sigma_1$, leading to a very large condition number.  In particular if the last $j$ singular values are very small, it suggest that the matrix has a "near Null Space",  with a basis defined by the last $j$ columns of $V$.  Thus, while strictly invertible,  if a direct solver is used to solve
$$
A\mathbf{x} =\mathbf{b}
$$
any component in the near null space will be amplified by $1/\sigma_j$ and completely pollute your answer.

However, a useful fix can be to approximate $A$ as a lower rank matrix

$$A_k = U_k\Sigma_kV_k^T$$

which just keeps the first $k$ singular values (i.e. pretends $r=k$).  If chosen wisely,  The approximate solution

$$
    \mathbf{x}^{+} = A^{+}_k\mathbf{b}
$$

can be considerably more accurate as it does not allow the near-null space to contaminate the solution.

# Examples

### Full SVD example

Consider the matrix
$$
    A = \begin{bmatrix}
        2 & 0 & 3 \\
        5 & 7 & 1 \\
        0 & 6 & 2
    \end{bmatrix}.
$$
Confirm the SVD representation using `numpy` functions as appropriate.

```python
A = numpy.array([
    [2.0, 0.0, 3.0],
    [5.0, 7.0, 1.0],
    [0.0, 6.0, 2.0]
])

U, sigma, V_T = numpy.linalg.svd(A, full_matrices=True)
```

```python
print('U:\n{}\n'.format(U))
print('Sigma:\n{}\n'.format(numpy.diag(sigma)))
print('V:\n{}\n'.format(V_T.T))

```

```python
Aprime = numpy.dot(U, numpy.dot(numpy.diag(sigma), V_T))
print('A - USV^T:\n{}'.format(A - Aprime))
```

#### Example: a rank-1 Matrix

$$A = \mathbf{x}\mathbf{y}^T$$

```python
x = numpy.array([ 1., 2., 1. ])
y = numpy.array([2., -1., 1. ])
A = numpy.outer(x,y)
print('A:\n{}\n'.format(A))
U, sigma, V_T = numpy.linalg.svd(A, full_matrices=False)
```

```python
print('U:\n{}\n'.format(U))
print('sigma:\n{}\n'.format(numpy.diag(sigma)))
print('V:\n{}\n'.format(V_T.T))
```

```python
Aprime = numpy.dot(U, numpy.dot(numpy.diag(sigma), V_T))
print('A - USV^T:\n{}'.format(A - Aprime))
```

```python
# rank-1 reconstruction
k = 1
A1 = numpy.dot(U[:,:k], numpy.dot(numpy.diag(sigma[:k]), V_T[:k,:]))
print('A_k =\n{}\n'.format(A1))
print('A - A_k:\n{}'.format(A - A1))
```

### An ill-conditioned example

Consider the matrix
$$
    A = \begin{bmatrix}
        1 & 0 & 1 \\
        1 & 1 & 2+\epsilon \\
        0 & 1 & 1\\
    \end{bmatrix}.
$$
where the last column is almost the sum of the first two.

```python
epsilon = 1.e-5
A = numpy.array([
    [1.0, 0.0, 1.0],
    [1.0, 1.0, 2+epsilon],
    [0.0, 1.0, 1.0]
])

U, S, V_T = numpy.linalg.svd(A, full_matrices=True)
```

```python
print('U:\n{}\n'.format(U))
print('S:\n{}\n'.format(numpy.diag(S)))
print('V:\n{}\n'.format(V_T.T))
print('sigma_1/sigma_n = {}, cond_2(A) = {}'.format(S[0]/S[2], numpy.linalg.cond(A, p=2)))
```

### Try to solve $A\mathbf{x}=\mathbf{b}$

```python
finfo = numpy.finfo(numpy.float64)
epsilon = 2*finfo.eps
A = numpy.array([
    [1.0, 0.0, 1.0],
    [1.0, 1.0, 2+epsilon],
    [0.0, 1.0, 1.0]
])
U, S, VT = numpy.linalg.svd(A)
x_true = numpy.array([1., 2., 3.])
b = A.dot(x_true)
x = numpy.linalg.solve(A,b)
print('x = {}: cond(A)={}, S ={}'.format(x, numpy.linalg.cond(A, p=2), S))
```

```python
# rank-2 reconstruction
k = 2
A2 = numpy.dot(U[:,:k], numpy.dot(numpy.diag(S[:k]), VT[:k,:]))
#Pseudo Inverse
Ap2 = numpy.dot(VT.T[:,:k], numpy.dot(numpy.diag(1./S[:k]), U.T[:k,:]))
xp = Ap2.dot(b)
print('x = {}: cond(A)={}, S ={}'.format(x, numpy.linalg.cond(A, p=2), S))
print('x^+ = {}'.format(xp))
```

## Review:  the Singular Value Decomposition (SVD)
$$
    A = U\Sigma V^{T}
$$
for *any* $A\in\mathbb{R}^{m\times n}$, where
* $U \in \mathbb R^{m \times m}$ and is the orthogonal matrix whose columns are the eigenvectors of $AA^{T}$
* $V \in \mathbb R^{n \times n}$ and is the orthogonal matrix whose columns are the eigenvectors of $A^{T}A$
* $\Sigma \in \mathbb R^{m \times n}$ and is a diagonal matrix with elements $\sigma_{1}, \sigma_{2}, \sigma_{3}, ... \sigma_{r}$ where $r = rank(A)$ corresponding to the square roots of the eigenvalues of $A^{T}A$. They are called the singular values of $A$ and are positive arranged in descending order. ($\sigma_{1} \geq \sigma_{2} \geq \sigma_{3} \geq ... \sigma_{r} > 0$).

Again a picture is worth a lot here

$$
A\begin{bmatrix} &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
 \mathbf{v}_1 & \cdots & \mathbf{v}_r & | &\mathbf{v}_{r+1} & \cdots & \mathbf{v}_n \\
 &  &  & |&  &  &  \\
  &  &  & |&  &  &  \\
   \end{bmatrix} = \begin{bmatrix} &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
 \mathbf{u}_1 & \cdots & \mathbf{u}_r & | &\mathbf{u}_{r+1} & \cdots & \mathbf{u}_m \\
 &  &  & |&  &  &  \\
 &  &  & |&  &  &  \\
  &  &  & |&  &  &  \\
    \end{bmatrix} \begin{bmatrix} \sigma_1 &  &  & |  &  &    \\
 & \ddots &  & |   &   \mathbf{0}  \\
 &  & \sigma_r &|    &     \\
 -&- &-  & | & - &  -\\
 & \mathbf{0} &  & |   & \mathbf{0}   \\
 &  &  & |  &    \\
  &  &  & |  &     \\
    \end{bmatrix}
$$

### $U$ and $V$ contain orthonormal bases for the 4 subspaces of $A$

* The first $r$ columns of $U$ form an orthonormal bases for $C(A)\in\mathbb{R}^m$
* The first $r$ columns of $V$ form an orthonormal bases for $C(A^T)\in\mathbb{R}^n$
* The last $n-r$ columns of $V$ form an orthonormal basis for $N(A)\in\mathbb{R}^n$
* The last $m-r$ columns of $U$ form an orthonormal basis for $N(A^T)\in\mathbb{R}^m$

## The Economy (or Skinny) SVD

As it turns out, because of all the 0's  all we actually need to reconstruct $A$ is the the first $r$ columns of $U$, and $V$,  and the square sub-block of $\Sigma$ with just the singular values.  i.e.

$$
A\begin{bmatrix} &  &  & \\
 &  &  &    \\
 \mathbf{v}_1 & \cdots & \mathbf{v}_r   \\
 &  &  &   \\
  &  &  &    \\
   \end{bmatrix} = \begin{bmatrix} &  &  &   \\
 &  &  &    \\
  &  &  &    \\
 \mathbf{u}_1 & \cdots & \mathbf{u}_r     \\
 &  &  &    \\
 &  &  &    \\
  &  &  &    \\
    \end{bmatrix} \begin{bmatrix} \sigma_1 &  &        \\
 & \ddots &    \\
 &  & \sigma_r       \\
    \end{bmatrix}
$$

* $V_rV_r^T$ is an orthogonal projector onto $C(A^T)$
* $(I - V_rV_r^T)$ projects onto $N(A)$
* $U_rU_r^T$ is an orthogonal projector onto $C(A)$
* $(I - U_rU_r^T)$ projects onto $N(A^T)$

## The Pseudo-Inverse

Given the SVD,  every matrix $A\in\mathbb{R}^{m\times n}$  has a 'pseudo-inverse'

$$
    A^{+} = V_r\Sigma_r^{-1}U_r^T
$$

with the properties
* if $A\in\mathbb{R}^{n\times n}$ and $r=n$, then $A^{+}=A^{-1}$
* $\mathbf{x}^+ = A^{+}\mathbf{b}$ is always the shortest least squares solution s.t. $(I - V_rV_r^T)\mathbf{x}^+=\mathbf{0}$
* $AA^+$ projects $\mathbf{b}$ onto ______?
* $A^+A$ projects $\mathbf{x}$ onto ______?

### Matrix Properties via the SVD

 - The $\text{rank}(A) = r$ where $r$ is the number of non-zero singular values.
 - The $\text{range}(A) = \mathrm{span}\langle u_1, ... , u_r\rangle$ and $\text{null}(a) = \mathrm{span}\langle v_{r+1}, ... , v_n\rangle$.
 - The $|| A ||_2 = \sigma_1$ and $||A||_F = \sqrt{\sigma_{1}^{2}+\sigma_{2}^{2}+...+\sigma_{r}^{2}}$.
 - The nonzero singular values of A are the square roots of the nonzero eigenvalues of $A^{T}A$ or $AA^{T}$.
 - If $A = A^{T}$, then the singular values of $A$ are the absolute values of the eigenvalues of $A$.
 - For $A \in \mathbb{C}^{m \times m}$ then $|det(A)| = \Pi_{i=1}^{m} \sigma_{i}$

### Low-Rank Approximations

 - $A$ is the sum of the $r$ rank-one matrices:
$$
    A = U \Sigma V^T = \sum_{j=1}^{r} \sigma_{j}u_{j}v_{j}^{T}
$$
 - For any $k$ with $0 \leq k \leq r$, define
$$
    A_{k} = \sum_{j=1}^{k} \sigma_{j}u_{j}v_{j}^{T}
$$

then

$$
     ||A - A_{k}||_{2} = ||\sum_{j=k+1}^{r} \sigma_{j}u_{j}v_{j}^{T}||_{2} = \sigma_{k+1}
$$

It can then be shown that for all matrices $B\in \mathbb{C}^{m \times n}$ with rank $k\leq r$, then

$$
    ||A - A_{k}||_{2} \leq || A-B||_{2}
$$

i.e. that $A_k$ is the best rank-k approximation to $A$ in the 2-norm.

A similar result can be shown for the Frobenious norm

- For any $k$ with $0 \leq k \leq r$, the matrix $A_{k}$ also satisfies

$$
    ||A - A_{k}||_{F} = \text{inf}_{B \in \mathbb{C}^{m \times n}} \text{rank}(B)\leq v ||A-B||_{F} = \sqrt{\sigma_{v+1}^{2} + ... + \sigma_{r}^{2}}
$$

## Example:   "Image compression"

How does this work in practice?

### the original Matrix (From Durer's Melancholia)

```python
from matplotlib import image
data = image.imread('images/melancholia-magic-square.png')
m,n = data.shape
```

```python
print('{}x{} pixel image'.format(m,n))
plt.figure(figsize=(8,6))
plt.imshow(data,cmap='gray')
plt.show()
```

```python
u, s, vt = numpy.linalg.svd(data)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.semilogy(s,'bo-')
axes.set_ylabel('$\sigma$', fontsize=16)
axes.grid()
axes.set_title('Spectrum of Singular Values')
plt.show()
```

### First 4 Modes

where mode $i$ is the rank-1 matrix $\sigma_i\mathbf{u}_i\mathbf{v}_i^T$

```python
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 4)
fig.set_figheight(fig.get_figheight() * 1)
for i in range(4):
    mode = s[i]*numpy.outer(u[:,i], vt[i,:])

    axes = fig.add_subplot(1, 4, i+1)
    mappable = axes.imshow(mode, cmap='gray')
    axes.set_title('Mode = {}, $\sigma$={:3.3f}'.format(i+1,s[i]),fontsize=18)

plt.show()
```

### Compressed Reconstructions

We can now view compressed reconstructions that sum the first $k$ modes
$$
    A_k = \sum_{i=1}^k \sigma_i\mathbf{u}_i\mathbf{v}_i^T = U_k\Sigma_kV_k^T
$$

the amount of memory required to store $A_k$ is $O(k(m +n + 1))$ whereas the storage of $A$ is $O(mn)$ so the relative compression is

$$
    \frac{k(m+n + 1)}{mn}
$$

The question is how many modes are required to adequately reconstruct the original image?

```python
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 3)
fig.set_figheight(fig.get_figheight() *2)
for i,k in enumerate([1, 10, 20]):
    Ak = u[:,:k].dot(numpy.diag(s[:k]).dot(vt[:k,:]))
    storage = 100.*k*(m+n + 1)/(m*n)
    axes = fig.add_subplot(1, 3, i+1)
    mappable = axes.imshow(Ak, vmin=0.0, vmax=1.0, cmap='gray')
    axes.set_title('$A_{{{}}}$, $storage={:2.2f}\%$'.format(k,storage),fontsize=16)

plt.show()
```

```python
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 3)
fig.set_figheight(fig.get_figheight() *2)
for i,k in enumerate([50, 100, 200]):
    Ak = u[:,:k].dot(numpy.diag(s[:k]).dot(vt[:k,:]))
    storage = 100.*k*(m+n + 1)/(m*n)
    axes = fig.add_subplot(1, 3, i+1)
    mappable = axes.imshow(Ak, vmin=0.0, vmax=1.0, cmap='gray')
    axes.set_title('$A_{{{}}}$, $storage={:2.2f}\%$'.format(k,storage),fontsize=16)

plt.show()
```

### Other Applications

* Total Least-squares
* PCA

# Total Least Squares

**GOAL:** Demonstrate the use of the SVD to calculate total least squares regression and compare it to the classical least squares problem that assumes only errors in y.

## Random data:

We start by constructing a random data set that approximates a straight line but has random errors in both x and y coordinates

```python
# npoints uniformly randomly distributed points in the interval [0,3]
npnts =100
x = numpy.random.uniform(0.,3.,npnts)
# set y = mx + b plus random noise of size err
slope = 2.
intercept = 1.
err = .5

y = slope*x + intercept
y += numpy.random.normal(loc=y,scale=err)

# add some random noise to x variable as well
x += numpy.random.normal(loc=x,scale=err)

# And plot out the data
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.scatter(x,y)
axes.set_xlabel('x',fontsize=16)
axes.set_ylabel('y', fontsize=16)
axes.set_title('Data', fontsize=18)
axes.grid()
plt.show()
```

### Classical Least Squares:

We first calculate the best fit straight line assuming all the error is in the y variable using the a QR decomposition of the Vandermonde matrix [ 1 x ]

```python
# Vandermonde matrix
A = numpy.array([ numpy.ones(x.shape), x]).T

# solve  Ac = y using the QR decomposition via scipy
c_ls,res,rank,s = numpy.linalg.lstsq(A,y, rcond=None)
print('Best fit Linear Least Squares:')
print('    slope={}'.format(c_ls[1]))
print('    intercept={}'.format(c_ls[0]))

```

```python
# dummy variables
t_ls = numpy.linspace(0,x.max())

# And plot out the data
fig  = plt.figure(figsize=(10,8))
axes = fig.add_subplot(1,1,1)
axes.scatter(x,y)
axes.set_xlabel('x')
axes.set_ylabel('y')
# plot the least squares solution
axes.plot(t_ls,c_ls[0]+t_ls*c_ls[1],'r-',label='Least Squares')
axes.legend()
axes.grid()
plt.show()
```

### Total Least Squares:

Suppose we wanted to find the line defined by its unit normal vector $\mathbf{u}$ that instead minimized the sum of **orthogonal** distance between the data and the line, i.e.

```python
# make up a line through the center of the data
X = numpy.array([ x , y]).T
X_mean = numpy.mean(X,0)
v = numpy.array([1,2])
v = v/numpy.linalg.norm(v, ord=2)
u = numpy.array([-v[1],v[0]])
#print(u.dot(v))
t = numpy.linspace(0,x.max())
t =  2*(t - numpy.mean(t_ls))

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.scatter(x,y)
l = X_mean + numpy.outer(t,v)
lu = X_mean + u
axes.plot(l[:,0],l[:,1],'g')
axes.plot([ X_mean[0], X_mean[0] + u[0]], [X_mean[1], X_mean[1] + u[1]], 'r')
axes.text( X_mean[0] + 1.3*u[0],  X_mean[1] + 1.3*u[1], '$\mathbf{u}$',fontsize=14)
axes.set_aspect('equal')
axes.set_xlabel('x',fontsize=16)
axes.set_ylabel('y', fontsize=16)
axes.set_title('Data', fontsize=18)

axes.grid()
plt.show()
```

This is equivalent to finding $\mathbf{u}$ that minimizes

$$
    ||M\mathbf{u}||_2
$$
where

$$
    M = X - \bar{X} =
\begin{bmatrix} (x_1 - \bar{x}) & (y_1 - \bar{y}) \\
                                        (x_2 - \bar{x}) & (y_2 - \bar{y}) \\
                                        (x_3 - \bar{x}) & (y_3 - \bar{y}) \\
                                        \vdots & \vdots \\
                                        (x_n - \bar{x}) & (y_n - \bar{y}) \\
                                       \end{bmatrix}
$$
is a matrix of the de-meaned data

but
$$
    ||M\mathbf{u}||^2_2 = \mathbf{u}^TM^TM\mathbf{u}
$$

or substituting in the SVD of $M=U\Sigma V^T$ the problem becomes find $\mathbf{u}$ that **minimizes**

$$
    (\mathbf{u}^TV) \Sigma^T\Sigma (V^T\mathbf{u})
$$

Let $\mathbf{y} = V^T\mathbf{u} \in \mathbb{R}^2$ where $||\mathbf{y}||_2 =1$

then we seek $\mathbf{y}$ that minimizes

$$
    \mathbf{y}^T\Sigma^T\Sigma \mathbf{y} = \sigma_1^2 y_1^2 + \sigma_2^2 y_2^2 \geq \sigma_2^2(y_1^2 + y_2^2) \geq \sigma_2^2
$$

because $$||\mathbf{y}||_2 =  y_1^2 + y_2^2 = 1$$

hopefully it's clear that the vector

$$
    \mathbf{y} = \begin{bmatrix} 0 \\ 1 \\\end{bmatrix}
$$
provides the minimum value

But  $\mathbf{u} = V\mathbf{y}$

So, the best fit line is the one that is orthogonal to $\mathbf{v}_2$ which is the singular vector corresponding to the smallest singular value.

The line we seek is orthogonal to that, so must lie in the direction of $\mathbf{v}_1$

### Total Least-Squares (a recipe)

* form a matrix $M$ whose rows are the de-meaned data
* find the SVD of $M = U\Sigma V^T$,
* set $\mathbf{u} = \mathbf{v}_2$
* and in 2-D,  the best fit line is parallel to $\mathbf{v}_1$ and goes through the mean of the data

```python
# Prepare the data matrix
X = numpy.array([ x , y]).T
print('Shape of data Matrix: {}'.format(X.shape))

```

```python
# and remove the mean
X_mean = numpy.mean(X,0)
print('Mean of data matrix={}'.format(X_mean))
M = X - X_mean
```

```python
# now calculate the SVD of the de-meaned data matrix
U,S,VT = numpy.linalg.svd(M,full_matrices=False)
V = VT.T
print('Singular values = {}'.format(S))
print('First Right singular vector V_1= {}'.format(V[:,0]))
```

Now plot and compare the two solutions

```python
# dummy variables
t_ls = numpy.linspace(0,x.max())
t_svd =  2*(t_ls - numpy.mean(t_ls))

# make figure
plt.figure(figsize=(10,8))
# plot data
plt.scatter(x,y)
# plot the least squares solution
plt.plot(t_ls,c_ls[0]+t_ls*c_ls[1],'r-',label='Least Squares')

# plot the total least Squares solution
# plot the mean
plt.plot(X_mean[0],X_mean[1],'go', markersize=12, label='X_mean')
# calculate a line through the mean with the first principal component as a basis
L_tls = X_mean + numpy.outer(t_svd,V[:,0])
plt.plot(L_tls[:,0],L_tls[:,1],'c-',label='Total Least Squares')
plt.xlabel('x')
plt.ylabel('y')
plt.title('Comparison Least Squares vs Total Least Squares')
plt.legend(loc='best')
plt.grid()
plt.show()
```

### Notes

This technique can be extended into higher dimensions.  For example, we could find a best fit plane through a 3-D cloud of data.

For this problem again, we would find the SVD of the demeaned data matrix $M$, and the best fit plane would be normal to the last singular vector $\mathbf{v}_3$ or  $\mathrm{span}\langle \mathbf{v}_1, \mathbf{v}_2\rangle$

In general, for a demeaned data matrix $M$,  columns of the matrix $V$ form an orthonormal basis for the row space of $M$ and describe the axes of the best fit Ellipsoid describing the data.