# Image compression for acoustic transmissions.

```python
__author__ = 'andrea munafo'
```

This notebook investigates some image compression methods using neural networks.

The point is to obtain a dimentionality reduction that can be supported over very bandwidth constrained media, such as acoustics.

This description of autoencoders is based on [Understanding Variational Autoencoders](
https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73), on [Autoencoding Variational Bayes](https://arxiv.org/pdf/1312.6114.pdf) and on [Tutorial on variational autoencoders](https://arxiv.org/abs/1606.05908).

and is reported here to provide a description for the implementation reported below.

All implementations are applied to the MNIST dataset.

## What is dimentionality reduction?

Dimensionality reduction is the process of reducing the number of features that describe some data.

This reduction can be done, either selecting specific features or by reducing to a new set of features created from the original ones (feature extraction).
Applications range from data visualisation, data storage, heavy computation and communications.

First, let’s call encoder the process that produce the “new features” representation from the “old features” representation (by selection or by extraction) and decoder the reverse process.

Dimensionality reduction can then be interpreted as data compression where the encoder compress the data (from the initial space to the encoded space, also called latent space) whereas the decoder decompress them.
Of course, depending on the initial data distribution, the latent space dimension and the encoder definition, this compression can be lossy, meaning that a part of the information is lost during the encoding process and cannot be recovered when decoding.

The problem can be formulated as finding the optimal encoder $e$ in the set of possible encoders $E$, and the optimal decoder $d$, in the set of possible dencoders $D$, such that:

$(e*,d*) = argmin f(x, d(e(x))$

where $e \in E$, $d \in D$, and $f(.)$ is the function that defines the reconstruction error measure between the input data $x$ and the encoded-decoded data $d(e(x))$.

One way to solve this problem is using Principal Components Analysis (PCA).

The idea of PCA is to build $n_e$ new features that are linear combinations of the $n_d$ original features.
This is done so that the projections of the original data on the subspace defined by these new features are as close as possible to the initial data (in term of euclidean distance).
This is shown in the figure below:

For example, the encoded version of point B is obtained as the projection of B onto the line that minimises the distance from all the points.

It can be shown that, the unitary eigenvectors corresponding to the $n_e$ greatest eigenvalues (in norm) of the covariance features matrix C are orthogonal (or can be chosen to be so) and define the best subspace of dimension $n_e$ to project data on with minimal error of approximation.
These $n_e$ eigenvectors can then be chosen as the new features.
In the PCA context, the problem of dimension reduction can be expressed as an eigenvalue/eigenvector problem.
Moreover, it can also be shown that the decoder matrix is the transposed of the encoder matrix.

Encoding and decoding matrices obtained with PCA define one of the solutions.
However, several basis can be chosen to describe the same optimal subspace and, so, several encoder/decoder pairs can give the optimal reconstruction error.

Now, let’s assume that both the encoder and the decoder are deep and non-linear. In such case, the more complex the architecture is, the more the autoencoder can proceed to a high dimensionality reduction while keeping reconstruction loss low. Intuitively, if our encoder and our decoder have enough degrees of freedom, we can reduce any initial dimensionality to 1. Indeed, an encoder with “infinite power” could theoretically takes our N initial data points and encodes them as 1, 2, 3, … up to N (or more generally, as N integer on the real axis) and the associated decoder could make the reverse transformation, with no loss during the process.

Here, we should however keep two things in mind. First, an important dimensionality reduction with no reconstruction loss often comes with a price: the lack of interpretable and exploitable structures in the latent space (lack of regularity). Second, most of the time the final purpose of dimensionality reduction is not to only reduce the number of dimensions of the data but to reduce this number of dimensions while keeping the major part of the data structure information in the reduced representations. For these two reasons, the dimension of the latent space and the “depth” of autoencoders (that define degree and quality of compression) have to be carefully controlled and adjusted depending on the final purpose of the dimensionality reduction.

The concept is shown in the picture below. We would like to reduce the dimentionality, while keeping some of the structure of the data.

# Autoencoders

```python
import os

import pylab as plt
import numpy as np

import torch
import torchvision
from torch import nn
from torch.autograd import Variable
from torch.utils.data import DataLoader
from torchvision import transforms
from torchvision.datasets import MNIST
from torchvision.utils import save_image
```

```python
print(torch.__version__)
```

    1.3.0

```python
import pathlib

pathlib.Path("../results/01-autoencoder/ae").mkdir(parents=True, exist_ok=True)
pathlib.Path("../results/01-autoencoder/convae").mkdir(parents=True, exist_ok=True)
pathlib.Path("../results/01-autoencoder/vae").mkdir(parents=True, exist_ok=True)
pathlib.Path("../saved-mdls/01-autoencoder").mkdir(parents=True, exist_ok=True)
```

```python
num_epochs = 100
batch_size = 64
learning_rate = 1e-3

device = 'cuda'
```

```python
def toImg(x, mu=0.5, std=1):
    """Converts x to an image shape. It works for batches of inputs."""
    x = mu * (x + std)
    x = x.clamp(0, 1)
    x = x.view(x.size(0), 1, 28, 28)
    return x
```

## Get the data

### Define some transforms to normalise the images

```python
ds_mean = 0.1307
ds_std = 0.3081

img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((ds_mean,), (ds_std,)) # The first tuple (0.5, 0.5, 0.5) is the mean for all three channels and the second (0.5, 0.5, 0.5) is the standard deviation for all three channels.
])
```

### Grab the dataset

```python
train_ds = MNIST('./data', train=True, transform=img_transform, download=True)
valid_ds = MNIST('./data', train=False, transform=img_transform, download=True)
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to ./data/MNIST/raw/train-images-idx3-ubyte.gz

    100.1%

    Extracting ./data/MNIST/raw/train-images-idx3-ubyte.gz to ./data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to ./data/MNIST/raw/train-labels-idx1-ubyte.gz

    113.5%

    Extracting ./data/MNIST/raw/train-labels-idx1-ubyte.gz to ./data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw/t10k-images-idx3-ubyte.gz

    100.4%

    Extracting ./data/MNIST/raw/t10k-images-idx3-ubyte.gz to ./data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz

    180.4%

    Extracting ./data/MNIST/raw/t10k-labels-idx1-ubyte.gz to ./data/MNIST/raw
    Processing...
    Done!

```python
# plt.imshow(train_ds.data[1])
```

```python
train_dl = DataLoader(train_ds, batch_size=batch_size, shuffle=True)
valid_dl = DataLoader(valid_ds, batch_size=batch_size, shuffle=True)
```

## Define the model

Defining an Autoencoder model is not difficult. The following class shows one potential option as a combination of Linear and ReLU layers.

```python
class Autoencoder(nn.Module):
    def __init__(self):
        super(Autoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Linear(28 * 28, 128),
            nn.ReLU(True),
            nn.Linear(128, 64),
            nn.ReLU(True),
            nn.Linear(64, 12),
            nn.ReLU(True),
            nn.Linear(12, 3))

        self.decoder = nn.Sequential(
            nn.Linear(3, 12),
            nn.ReLU(True),
            nn.Linear(12, 64),
            nn.ReLU(True),
            nn.Linear(64, 128),
            nn.ReLU(True),
            nn.Linear(128, 28 * 28),
            nn.Tanh())

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

    def encode(self, x):
        x = self.encoder(x)
        return x

    def decode(self, x):
        x = self.decoder(x)
        return x

```

## Train the model

```python
model = Autoencoder()
```

```python
loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(
    model.parameters(), lr=learning_rate, weight_decay=1e-5)
```

```python
if device == 'cuda':
    model.cuda()
```

```python
for epoch in range(num_epochs):
    for x, y in train_dl:
        x = x.view(x.size(0), -1).to(device)  # resize to be a vector bsx28*28
#        x = Variable(x).cuda()

        output = model(x)
        loss = loss_fn(output, x)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    print('epoch [{}/{}], loss:{:.4f}'.format(epoch + 1, num_epochs, loss.data))

    if epoch % 10 == 0:
        ipic = toImg(x.cpu().data)
        opic = toImg(output.cpu().data)

        save_image(opic, '../results/ae/image_{}_o.png'.format(epoch))
        save_image(ipic, '../results/ae/image_{}_i.png'.format(epoch))
```

    epoch [1/100], loss:0.5602
    epoch [2/100], loss:0.5473
    epoch [3/100], loss:0.5371
    epoch [4/100], loss:0.5408
    epoch [5/100], loss:0.4876
    epoch [6/100], loss:0.4854
    epoch [7/100], loss:0.4918
    epoch [8/100], loss:0.5050
    epoch [9/100], loss:0.5683
    epoch [10/100], loss:0.4808
    epoch [11/100], loss:0.4911
    epoch [12/100], loss:0.4622
    epoch [13/100], loss:0.5145
    epoch [14/100], loss:0.5030
    epoch [15/100], loss:0.4437
    epoch [16/100], loss:0.5159
    epoch [17/100], loss:0.5257
    epoch [18/100], loss:0.4939
    epoch [19/100], loss:0.4832
    epoch [20/100], loss:0.4450
    epoch [21/100], loss:0.4416
    epoch [22/100], loss:0.4433
    epoch [23/100], loss:0.4908
    epoch [24/100], loss:0.4583
    epoch [25/100], loss:0.4830
    epoch [26/100], loss:0.4803
    epoch [27/100], loss:0.4719
    epoch [28/100], loss:0.4317
    epoch [29/100], loss:0.4332
    epoch [30/100], loss:0.5100
    epoch [31/100], loss:0.4836
    epoch [32/100], loss:0.4686
    epoch [33/100], loss:0.4797
    epoch [34/100], loss:0.5120
    epoch [35/100], loss:0.4823
    epoch [36/100], loss:0.4763
    epoch [37/100], loss:0.4901
    epoch [38/100], loss:0.4556
    epoch [39/100], loss:0.4851
    epoch [40/100], loss:0.4474
    epoch [41/100], loss:0.4293
    epoch [42/100], loss:0.4440
    epoch [43/100], loss:0.4943
    epoch [44/100], loss:0.5166
    epoch [45/100], loss:0.5518
    epoch [46/100], loss:0.5341
    epoch [47/100], loss:0.5265
    epoch [48/100], loss:0.5320
    epoch [49/100], loss:0.4872
    epoch [50/100], loss:0.4489
    epoch [51/100], loss:0.5037
    epoch [52/100], loss:0.4680
    epoch [53/100], loss:0.4607
    epoch [54/100], loss:0.4225
    epoch [55/100], loss:0.4512
    epoch [56/100], loss:0.5028
    epoch [57/100], loss:0.4769
    epoch [58/100], loss:0.5159
    epoch [59/100], loss:0.4600
    epoch [60/100], loss:0.4927
    epoch [61/100], loss:0.4544
    epoch [62/100], loss:0.4826
    epoch [63/100], loss:0.5090
    epoch [64/100], loss:0.5271
    epoch [65/100], loss:0.5107
    epoch [66/100], loss:0.4235
    epoch [67/100], loss:0.4578
    epoch [68/100], loss:0.4514
    epoch [69/100], loss:0.4575
    epoch [70/100], loss:0.4789
    epoch [71/100], loss:0.4688
    epoch [72/100], loss:0.5022
    epoch [73/100], loss:0.4651
    epoch [74/100], loss:0.4402
    epoch [75/100], loss:0.4553
    epoch [76/100], loss:0.5108
    epoch [77/100], loss:0.4504
    epoch [78/100], loss:0.4580
    epoch [79/100], loss:0.5241
    epoch [80/100], loss:0.4497
    epoch [81/100], loss:0.4625
    epoch [82/100], loss:0.4352
    epoch [83/100], loss:0.5229
    epoch [84/100], loss:0.5016
    epoch [85/100], loss:0.4678
    epoch [86/100], loss:0.4940
    epoch [87/100], loss:0.4866
    epoch [88/100], loss:0.5085
    epoch [89/100], loss:0.4982
    epoch [90/100], loss:0.4447
    epoch [91/100], loss:0.4906
    epoch [92/100], loss:0.4760
    epoch [93/100], loss:0.4070
    epoch [94/100], loss:0.4130
    epoch [95/100], loss:0.4456
    epoch [96/100], loss:0.4077
    epoch [97/100], loss:0.4823
    epoch [98/100], loss:0.4681
    epoch [99/100], loss:0.4975
    epoch [100/100], loss:0.4695

We are now done training. We can save the model so that we can use it later on.

```python
# As suggested in https://pytorch.org/tutorials/beginner/saving_loading_models.html
torch.save(model, '../saved-mdls/01-autoencoder/autoencoder_{}e.pt'.format(epoch+1))
```

    /opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:256: UserWarning: Couldn't retrieve source code for container of type Autoencoder. It won't be checked for correctness upon loading.
      "type " + obj.__name__ + ". It won't be checked "

# Using and validating the simple autoencoder

```python
# Model class must be defined somewhere
model = torch.load('../saved-mdls/01-autoencoder/autoencoder.pt')
model.eval()
```

```python
for count, (x, _) in enumerate(valid_dl):
    x = x.view(x.size(0), -1)
    encoding = model.encode(x)
    decoding = model.decode(encoding)

    # save (some) results
    if count % 10 == 0:
        ipic = to_img(x.cpu().data)
        opic = to_img(decoding.cpu().data)
        save_image(ipic, '../results/01-autoencoder/ae/val-{}-enc.png'.format(count))
        save_image(opic, '../results/01-autoencoder/ae/val-{}-dec.png'.format(count))
```

Let's see how each image was encoded.
The next line prints the encodings for the last batch.

```python
print('encoded output: {}'.format(encoding.detach().cpu().numpy()))
```

If we want to analyse the results inline we can grab the encoding/decoding variables and play with them.

```python
idx = 0
```

```python
plt.imshow(decoding[idx].cpu().data.view(28, 28))
```

```python
f, axarr = plt.subplots(1,2)
axarr[0].imshow(x[idx].cpu().data.view(28,28))
axarr[1].imshow(decoding[idx].cpu().data.view(28, 28))
```

## Encoding / Decoding single images

To do this, I am going to use the validation set again, but one can do the same with any image.

```python
model.eval()
```

    Autoencoder(
      (encoder): Sequential(
        (0): Linear(in_features=784, out_features=128, bias=True)
        (1): ReLU(inplace)
        (2): Linear(in_features=128, out_features=64, bias=True)
        (3): ReLU(inplace)
        (4): Linear(in_features=64, out_features=12, bias=True)
        (5): ReLU(inplace)
        (6): Linear(in_features=12, out_features=3, bias=True)
      )
      (decoder): Sequential(
        (0): Linear(in_features=3, out_features=12, bias=True)
        (1): ReLU(inplace)
        (2): Linear(in_features=12, out_features=64, bias=True)
        (3): ReLU(inplace)
        (4): Linear(in_features=64, out_features=128, bias=True)
        (5): ReLU(inplace)
        (6): Linear(in_features=128, out_features=784, bias=True)
        (7): Tanh()
      )
    )

```python
idx = 1001 # select an index in the validation set
```

```python
# Define the transforms to normalise the image
pic_tt = transforms.ToTensor() # Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]
pic_n = transforms.Normalize((ds_mean,), (ds_std,))

if torch.__version__ != '1.3.0':
    pic_data = valid_ds.test_data[idx].numpy()
    pic_data = pic_data[:, :, None] # (H x W x C)
    print('pic_data.dtype: {}'.format(pic_data.dtype))
else:
    pic_data = valid_ds.data[idx].numpy()

pic = pic_tt(pic_data)
pic = pic_n(pic)

# print stats
print('pic mean: {}, std: {}'.format(pic.mean(), pic.std()))

# print pic shape
print('\npic shape (C x H x W): {}'.format(pic.shape))
```

    pic_data.dtype: uint8
    pic mean: 0.02145381271839142, std: 1.0174704790115356

    pic shape (C x H x W): torch.Size([1, 28, 28])

```python
plt.imshow(pic.view(28,28))
```

### Run the model to encode

```python
encoding = model.encode(pic.to(device).view(1, -1)) # note that we need to reshape the pic to be: bsx28*28

# print encoding result
print('encoded output: {}'.format(encoding.detach().cpu().numpy()))
```

    encoded output: [[-9.222232   9.739955  -7.7938967]]

### Run the decoder and show the result

```python
decoding = model.decode(encoding)
```

```python
plt.imshow(decoding.cpu().data.view(28, 28))
```

## Generating new images

```python
encoding = torch.tensor([
    [17., -10., +0.]])

decoding = model.decode(encoding)
plt.imshow(decoding.cpu().data.view(28, 28))
```

# Convolutional Autoencoders

Alternatively, one can define a Convolutional Autoencoder, where the network is composed of a sequence of convolutional layers.

```python
class ConvAutoencoder(nn.Module):
    def __init__(self):
        super(ConvAutoencoder, self).__init__()
        self.encoder = nn.Sequential(
            nn.Conv2d(1, 16, 3, stride=3, padding=1),  # b, 16, 10, 10
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=2),  # b, 16, 5, 5
            nn.Conv2d(16, 8, 3, stride=2, padding=1),  # b, 8, 3, 3
            nn.ReLU(True),
            nn.MaxPool2d(2, stride=1)  # b, 8, 2, 2
        )
        self.decoder = nn.Sequential(
            nn.ConvTranspose2d(8, 16, 3, stride=2),  # b, 16, 5, 5
            nn.ReLU(True),
            nn.ConvTranspose2d(16, 8, 5, stride=3, padding=1),  # b, 8, 15, 15
            nn.ReLU(True),
            nn.ConvTranspose2d(8, 1, 2, stride=2, padding=1),  # b, 1, 28, 28
            nn.Tanh()
        )

    def forward(self, x):
        x = self.encoder(x)
        x = self.decoder(x)
        return x

    def encode(self, x):
        x = self.encoder(x)
        return x

    def decode(self, x):
        x = self.decoder(x)
        return x
```

```python
model = ConvAutoencoder()

loss_fn = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate, weight_decay=1e-5)

if device == 'cuda': model.cuda()

for epoch in range(num_epochs):
    for x, y in train_dl:
        x = x.to(device)  # resize to be a vector bsx28*28

        output = model(x)
        loss = loss_fn(output, x)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

    print('epoch [{}/{}], loss:{:.4f}'.format(epoch + 1, num_epochs, loss.data))

    if epoch % 10 == 0:
        ipic = toImg(x.cpu().data)
        opic = toImg(output.cpu().data)

        save_image(opic, '../results/01-autoencoder/convae/image_{}_conv_o.png'.format(epoch))
        save_image(ipic, '../results/01-autoencoder/convae/image_{}_conv_i.png'.format(epoch))
```

    epoch [1/100], loss:0.5109
    epoch [2/100], loss:0.4628
    epoch [3/100], loss:0.5199
    epoch [4/100], loss:0.4988
    epoch [5/100], loss:0.4752
    epoch [6/100], loss:0.4783
    epoch [7/100], loss:0.4565
    epoch [8/100], loss:0.4774
    epoch [9/100], loss:0.4367
    epoch [10/100], loss:0.5051
    epoch [11/100], loss:0.4323
    epoch [12/100], loss:0.4938
    epoch [13/100], loss:0.4625
    epoch [14/100], loss:0.4442
    epoch [15/100], loss:0.4359
    epoch [16/100], loss:0.4216
    epoch [17/100], loss:0.4128
    epoch [18/100], loss:0.5216
    epoch [19/100], loss:0.4163
    epoch [20/100], loss:0.4229
    epoch [21/100], loss:0.4854
    epoch [22/100], loss:0.4435
    epoch [23/100], loss:0.4405
    epoch [24/100], loss:0.3971
    epoch [25/100], loss:0.4503
    epoch [26/100], loss:0.4532
    epoch [27/100], loss:0.4384
    epoch [28/100], loss:0.4189
    epoch [29/100], loss:0.4507
    epoch [30/100], loss:0.4159
    epoch [31/100], loss:0.4487
    epoch [32/100], loss:0.4177
    epoch [33/100], loss:0.4478
    epoch [34/100], loss:0.4493
    epoch [35/100], loss:0.4836
    epoch [36/100], loss:0.4739
    epoch [37/100], loss:0.4223
    epoch [38/100], loss:0.4179
    epoch [39/100], loss:0.4717
    epoch [40/100], loss:0.4035
    epoch [41/100], loss:0.4273
    epoch [42/100], loss:0.4396
    epoch [43/100], loss:0.4240
    epoch [44/100], loss:0.4513
    epoch [45/100], loss:0.4021
    epoch [46/100], loss:0.4341
    epoch [47/100], loss:0.3915
    epoch [48/100], loss:0.4056
    epoch [49/100], loss:0.4917
    epoch [50/100], loss:0.4463
    epoch [51/100], loss:0.4041
    epoch [52/100], loss:0.4991
    epoch [53/100], loss:0.4324
    epoch [54/100], loss:0.4214
    epoch [55/100], loss:0.4178
    epoch [56/100], loss:0.4286
    epoch [57/100], loss:0.4134
    epoch [58/100], loss:0.4234
    epoch [59/100], loss:0.4489
    epoch [60/100], loss:0.4033
    epoch [61/100], loss:0.4821
    epoch [62/100], loss:0.4093
    epoch [63/100], loss:0.4131
    epoch [64/100], loss:0.4737
    epoch [65/100], loss:0.4279
    epoch [66/100], loss:0.4410
    epoch [67/100], loss:0.4481
    epoch [68/100], loss:0.3976
    epoch [69/100], loss:0.4504
    epoch [70/100], loss:0.3888
    epoch [71/100], loss:0.3833
    epoch [72/100], loss:0.4279
    epoch [73/100], loss:0.3964
    epoch [74/100], loss:0.3978
    epoch [75/100], loss:0.4131
    epoch [76/100], loss:0.4012
    epoch [77/100], loss:0.4147
    epoch [78/100], loss:0.4499
    epoch [79/100], loss:0.3835
    epoch [80/100], loss:0.4146
    epoch [81/100], loss:0.4270
    epoch [82/100], loss:0.3776
    epoch [83/100], loss:0.4476
    epoch [84/100], loss:0.4090
    epoch [85/100], loss:0.4182
    epoch [86/100], loss:0.4433
    epoch [87/100], loss:0.4532
    epoch [88/100], loss:0.4058
    epoch [89/100], loss:0.4659
    epoch [90/100], loss:0.4265
    epoch [91/100], loss:0.4224
    epoch [92/100], loss:0.4296
    epoch [93/100], loss:0.4599
    epoch [94/100], loss:0.4055
    epoch [95/100], loss:0.4219
    epoch [96/100], loss:0.4640
    epoch [97/100], loss:0.4497
    epoch [98/100], loss:0.4238
    epoch [99/100], loss:0.4529
    epoch [100/100], loss:0.3765

```python
# As suggested in https://pytorch.org/tutorials/beginner/saving_loading_models.html
filename = '../saved-mdls/01-autoencoder/convautoencoder_{}e.pt'.format(epoch+1)
torch.save(model, filename)

print('model saved as: {}'.format(filename))
```

    model saved as: ./saved-mdls/01-autoencoder/convautoencoder_100e.pt

    /opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:256: UserWarning: Couldn't retrieve source code for container of type ConvAutoencoder. It won't be checked for correctness upon loading.
      "type " + obj.__name__ + ". It won't be checked "

```python
model.eval()
```

    ConvAutoencoder(
      (encoder): Sequential(
        (0): Conv2d(1, 16, kernel_size=(3, 3), stride=(3, 3), padding=(1, 1))
        (1): ReLU(inplace)
        (2): MaxPool2d(kernel_size=2, stride=2, padding=0, dilation=1, ceil_mode=False)
        (3): Conv2d(16, 8, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1))
        (4): ReLU(inplace)
        (5): MaxPool2d(kernel_size=2, stride=1, padding=0, dilation=1, ceil_mode=False)
      )
      (decoder): Sequential(
        (0): ConvTranspose2d(8, 16, kernel_size=(3, 3), stride=(2, 2))
        (1): ReLU(inplace)
        (2): ConvTranspose2d(16, 8, kernel_size=(5, 5), stride=(3, 3), padding=(1, 1))
        (3): ReLU(inplace)
        (4): ConvTranspose2d(8, 1, kernel_size=(2, 2), stride=(2, 2), padding=(1, 1))
        (5): Tanh()
      )
    )

```python
idx = 1001 # select an index in the validation set
```

```python
# Define the transforms to normalise the image
pic_tt = transforms.ToTensor() # Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]
pic_n = transforms.Normalize((ds_mean,), (ds_std,))

if torch.__version__ != '1.3.0':
    pic_data = valid_ds.test_data[idx].numpy()
    pic_data = pic_data[:, :, None] # (H x W x C)
    print('pic_data.dtype: {}'.format(pic_data.dtype))
else:
    pic_data = valid_ds.data[idx].numpy()

pic = pic_tt(pic_data)
pic = pic_n(pic)

# print stats
print('pic mean: {}, std: {}'.format(pic.mean(), pic.std()))

# print pic shape
print('\npic shape (C x H x W): {}'.format(pic.shape))
```

    pic_data.dtype: uint8
    pic mean: 0.02145381271839142, std: 1.0174704790115356

    pic shape (C x H x W): torch.Size([1, 28, 28])

```python
plt.imshow(pic.view(28,28))
```

### Run the model to encode

```python
print(pic.to(device)[None, :, :, :].shape)
```

    torch.Size([1, 1, 28, 28])

```python
encoding = model.encode(pic.to(device)[None, :, :, :]) # note that we need to reshape the pic to be: bsx28*28

# print encoding result
print('encoded output: {}'.format(encoding.detach().cpu().numpy()))
```

    encoded output: [[[[10.795247    6.077096  ]
       [ 5.69029     0.        ]]

      [[ 2.3519301   2.3519301 ]
       [ 2.3519301   2.3519301 ]]

      [[ 6.9652967   6.9652967 ]
       [ 0.5121887   0.5121887 ]]

      [[ 1.8640186   7.8633065 ]
       [ 5.015077    9.722703  ]]

      [[ 0.46509135  0.46509135]
       [ 8.607014    8.607014  ]]

      [[ 9.282421    5.930196  ]
       [ 4.88288     1.791863  ]]

      [[ 0.          7.888112  ]
       [ 1.9434392   6.8836126 ]]

      [[12.865752   12.865752  ]
       [12.865752   12.865752  ]]]]

### Run the decoder and show the result

```python
decoding = model.decode(encoding)
```

```python
plt.imshow(decoding.cpu().data.view(28, 28))
```

# Variational Autoencoders

We have discussed the problem of dimentionality reduction, and used encoder-decoder architectures to address it.

At this point, one question might arise: what is the link between autoencoders and content generation?

we could be tempted to think that, if the latent space is regular enough (well “organized” by the encoder during the training process), we could take a point randomly from that latent space and decode it to get a new content. The decoder would then act more or less like the generator of a Generative Adversarial Network.

As done before, we can generate new data by decoding points that are randomly sampled from the latent space. The quality and relevance of generated data depend on the regularity of the latent space.

The problem is that the regularity of the latent space for autoencoders depends on the distribution of the data in the initial space, the dimension of the latent space and the architecture of the encoder.
It is difficult to ensure that the encoder will organize the latent space in a smart way compatible with the generative process we just described.

This lack of structure among the encoded data into the latent space is normal since during training we never enforced such organization of the latern space.
We only trained to encode and decode so that we can minimize the loss.

To use the decoder of our autoencoder for generative purpose, we have to be sure that the latent space is regular enough. One possible solution to obtain such regularity is to introduce explicit regularisation during the training process.

A variational autoencoder is an autoencoder whose training is regularised to avoid overfitting and ensure that the latent space has good properties that enable generative process.

Just as a standard autoencoder, a variational autoencoder is an architecture composed of both an encoder and a decoder and that is trained to minimise the reconstruction error between the encoded-decoded data and the initial data. However, in order to introduce some regularisation of the latent space, we proceed to a slight modification of the encoding-decoding process: instead of encoding an input as a single point, we encode it as a distribution over the latent space.

To do this, the loss function that is minimised when training a VAE is composed of a “reconstruction term” (on the final layer), that tends to make the encoding-decoding scheme as performant as possible, and a “regularisation term” (on the latent layer), that tends to regularise the organisation of the latent space by making the distributions returned by the encoder close to a standard normal distribution.

One way to measure the difference between probability distribution is using the Kullback-Leibler divergence.

In practice, the encoded distributions are chosen to be normal so that the encoder can be trained to return the mean and the covariance matrix that describe these Gaussians.
The reason why an input is encoded as a distribution with some variance instead of a single point is that it makes possible to express very naturally the latent space regularisation: the distributions returned by the encoder are enforced to be close to a standard normal distribution.
Moreover, the Kullback-Leibler divergence between two Gaussian distributions has a closed form that can be directly expressed in terms of the means and the covariance matrices of the two distributions.

## Intuition

The regularity that is expected from the latent space in order to make generative process possible can be expressed through two main properties:
- continuity (two close points in the latent space should not give two completely different contents once decoded)
- completeness (for a chosen distribution, a point sampled from the latent space should give “meaningful” content once decoded)

A point of the latent space that would be halfway between the means of two encoded distributions coming from different training data should be decoded in something that is somewhere between the data that gave the first distribution and the data that gave the second distribution as it may be sampled by the autoencoder in both cases.

## Probabilistic framework

Variational autoencoders encode inputs as distributions (e.g. mean and std) and not as simple points (or vectors). The regularisation of the latent space is obtained constraining the distribution to be as close as possible to a Gaussian.

$x$ : data
$z$ : encoded rapresentation of $x$

We also assume that $z$ cannot be directly observed. Hewover, the latent representation $z$ can be sampled from the prior distribution $p(z)$

The data $x$ can be sampled from the conditional likelihood distribution: $p(x|z)$.

Within this framework, we can consider the probabilistic versions of encoders and decoders (instead of the deterministic ones that we described before).

- The “probabilistic decoder” is defined by $p(x|z)$, that describes the distribution of the decoded variable given the encoded one.

- The “probabilistic encoder” is defined by $p(z|x)$, that describes the distribution of the encoded variable given the decoded one.

Where of course, the link between the two is obtained through the Bayes' theorem: $p(z|x) = p(x|z)p(z)/p(x)$.

The previous equation also means that the encoder $p(z|x)$ depends on the prior $p(z)$, or on the encoded representation of $z$ in the latent space.

Let's assume:

$p(z) = \mathcal{N}(0, I)$
$p(x|z) = \mathcal{N}(f(z), cI)$, where $f \in F$ (family of functions), and $I$ is the identity matrix, $c$ constant > 0

If we consider $f$ to be well defined, fixed (and known), and if we know $p(x)$, can have all the elements to solve this problem and calculate $p(z|x)$. However calculating $p(x)$ is often intractable as $p(x)=\int p(x|u)p(u)du$

### Variational inference

Variational inference (VI) can be used to approximate complex distributions and hence we can use it to approximate $p(z|x)$.

The idea is to set a parametrised family of distribution (for example the family of Gaussians, whose parameters are the mean and the covariance) and to look for the best approximation of the target distribution among this family.

The best element in the family is one that minimise a given approximation error measurement (most of the time the Kullback-Leibler divergence between approximation and target).

This can be done using gradient descent over the parameters that describe the family [REF].

To do so, we assume $p(z|x)$, our target distribution as a Gaussian:

$p(z|x) = \mathcal{N}(g(x), h(x))$,

This means that we have chosen a Gaussian to approximate $p(z|x)$, whose mean and covariance are defined by two functions, $g$ and $h$, of the parameter $x$ (the data).

The problem than become that of finding the best approximation among this family by optimising the functions g and h (or better, their parameters) to minimise the Kullback-Leibler (KL) divergence between the approximation $\mathcal{N}(g(x), h(x))$ and the target $p(z|x)$, or:

\begin{equation*}
(g^*, h^*) = arg \min_{g,h} KL(\mathcal{N}(g(x), h(x)), p(z|x)).
\label{eq:optimal-encoder} \tag{1}
\end{equation*}

This would give us the optimal encoder $p^*(z|x) = \mathcal{N}(g^*(x), h^*(x))$.

Eq. (1), with a little bit of work (see [Understanding Variational Autoencoders](
https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73)) can be written as:

\begin{equation*}
(g^*, h^*) = \underset{g,h\in (GxH)}{\operatorname{argmax}} \{
\mathbb{E}_{z \sim \mathcal{N}(g(x), h(x))} [  log(p(x|z) ] - KL( \mathcal{N}(g(x), h(x)), p(z)) \}
\label{eq:optimal-encoder-2} \tag{2}
\end{equation*}

### Comment
All this reasoning was based on the assumption that we know $f(z)$. In fact, given that we know $f(z)$, we can calculate $p(x|z)$ (the decoder) and then we calculate the encoder $p(z|x)$ approximating it via variational inference.

However, in practise, $f(z)$ is not known and need to be chosen. In fact, the objective is to find a couple encoder-decoder with a regularised latent space, and the decoder $p(x|z)$ is a function of the function $f(z)$.

We have also realised that the encoder $p(z|x)$ depends on the prior $p(z)$ (Bayes' theorem).

As $p(z|x)$ can be approximated (by variational inference) from $p(z)$, and given that $p(x|z)$ and $p(z)$ have been set to be simple standard Gaussian,
we only have two parameters in our model: $c$ (that defines the variance of the posterior, $p(x|z)$) and the function $f$ (that defines the mean of the posterior).

In short:
- the encoder-decoder performance depends on the function $f(z)$
- the regularity of the latent space depends on the prior $p(z)$

This makes sense intuitively, as $f(z)$ defines the encoder, and $p(z)$ defines the latent space distribution.

## The encoder

Different functions $f(z)$ define different encoders.

We would like to find the best $f$ that maximise the encoding/decoding performance, or:

We want to choose the function $f$ that maximises the expected log-likelihood of $x$ (the data) given $z$ (its latent representation or encoding) when $z$ is sampled from $p^*(z|x)$. Where $p^*(z|x)$ is the optimal encoder, calculated as before.

For a given input $x$, we want to:
- maximise the probability to have $\hat x = x$
- when we sample $z$ from the distribution $p^*(z|x)$ (encoding)
- and then sample $\hat x$ from the distribution $p(x|z)$ (decoding).

We can write this as:

\begin{equation}
f^* = \underset{f\in F}{\operatorname{argmax}}
\mathbb{E}_{z \sim p^*(z|x)} [ \log p(x|z) ]
\label{eq:optimal-decoder} \tag{3}
\end{equation}

Note that, if $p(x|z) = \mathcal{N}(f(z), cI)$, then
$\log p(x|z) = - \frac{||x-f(z)||^2}{2c}$

## Encoder/Decoder

If we now put everything together (Eq.(1), (2), (3)):

\begin{equation*}
(f^*, g^*, h^*) = \underset{f,g,h\in (FxGxH)}{\operatorname{argmax}} \{
\mathbb{E}_{z \sim \mathcal{N}(g(x), h(x))} [  log(p(x|z) ] -  KL( \mathcal{N}(g(x), h(x)), p(z)) \}
\label{eq:optimal-encoder-2}
\end{equation*}

$\tag{4}$

with $\log p(x|z) = - \frac{||x-f(z)||^2}{2c}$

*Comments:*

The previous Eq (4), includes:
- the reconstruction error between $x$ and $f(z)$ (objective 1)
- the regularisation term given by the $KL$ divergence between $\mathcal{N}(g(x), h(x))$ and $p(z)$ (which is a standard Gaussian) (objective 2).

The constant $c$ acts as the balance between these two objectives:

The higher $c$ is, the more we assume a high variance around $f(z)$ for the probabilistic decoder in our model and, the more we favour the regularisation term over the reconstruction term.
The opposite stands if $c$ is small.

## Implementing a variational encoder with neural nets

We can’t easily optimise over the entire space of functions.
One way to solve this problem is to constrain the optimisation domain and decide to express $f$, $g$ and $h$ as neural networks.
In this case, $F$, $G$ and $H$ correspond respectively to the families of functions defined by the networks architectures and the optimisation is done over the parameters of these networks.

In practise, $g$ and $h$, which together define the encoder $p(z|x)$ are defined to share the initial part of their network architecture.

**Figure: Encoder part of the VAE (from [Understanding Variational Autoencoders](
https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73))**

Moreover, to simplify the problem, $\mathcal{N}(g(x), h(x))$ is assumed as a multidimensional Gaussian distribution with diagonal covariance matrix.

Under this assumption, the function $h(x)$ which defines the covariance matrix, is the vector of the diagonal elements.

To summarise, the output of the encoder part is a Gaussian with both mean $g$ and covariance $h$ that are functions of $x$.

The decoder network $f$ needs instead to model $p(x|z)$ that we defined as a Gaussian with unit variance.

As usual, the overall architecture is then obtained by concatenating the encoder and the decoder parts.

## Reparametrisation

The last element that remains is related to the fact that we need to sample from the encoded distribution $p(z|x)$ in a way that we can backpropagate the error through the network.

The reparametrisation allows us to do exactly that: makes the gradient descent possible despite the random sampling that occurs halfw aythrough the architecture.

The real trick is to understand that $z$ is a random variable sampled from a Gaussian distribution with mean $g(x)$ and variance $h(x)$.

This means that we can express $z$ as:

$z=h(x) \eta + g(x)$,

with $\eta \sim \mathcal{N}(0, I)$

In this way, we have sandboxed the sampling to only happen on the variable $\eta$, while we can do gradient descent over $h(x)$ and $g(x)$.

## Objective function

Finally, the _objective function_ of the variational autoencoder is defined by Eq. (4), where the expectation operator is replaced, most of the time by a Monte-Carlo approximation (sometimes obtained as a single draw).

The final architecture is sketched out in the next figure (from [Understanding Variational Autoencoders](
https://towardsdatascience.com/understanding-variational-autoencoders-vaes-f70510919f73)):

**Figure: architecture of a variational autoencoder (C = $\frac{1}{2c}$)**

# Implementing a variational autoencoder

## Grab the data

```python
num_epochs = 100
batch_size = 128
learning_rate = 1e-3

ds_mean = 0.1307
ds_std = 0.3081

img_transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize((ds_mean,), (ds_std,)) # The first tuple (0.5, 0.5, 0.5) is the mean for all three channels and the second (0.5, 0.5, 0.5) is the standard deviation for all three channels.
])

### Grab the dataset

train_ds = MNIST('./data', train=True, transform=img_transform, download=True)
valid_ds = MNIST('./data', train=False, transform=img_transform, download=True)

# plt.imshow(train_ds.data[1])

train_dl = DataLoader(train_ds, batch_size=batch_size, shuffle=True)
valid_dl = DataLoader(valid_ds, batch_size=batch_size, shuffle=True)
```

## Define the model

```python
class VAE(nn.Module):
    def __init__(self):
        super(VAE, self).__init__()

        self.base_encoder = nn.Sequential(
            nn.Linear(28 * 28, 400),
            nn.ReLU(True)
        )

        self.mean_encoder = nn.Sequential(
            self.base_encoder,
            nn.Linear(400, 20)
        )

        self.var_encoder = nn.Sequential(
            self.base_encoder,
            nn.Linear(400, 20)
        )

        self.decoder = nn.Sequential(
            nn.Linear(20, 400),
            nn.ReLU(True),
            nn.Linear(400, 784),
            nn.Sigmoid()
        )

    def forward(self, x):
        mu, logvar = self.mean_encoder(x), self.var_encoder(x)
        z = self.reparametrize(mu, logvar)
        return self.decoder(z), mu, logvar

    def encode(self, x):
        return self.mean_encoder(x), self.var_encoder(x)

    def decode(self, x):
        mu, logvar = x[0], x[1]
        z = self.reparametrize(mu, logvar)
        return self.decoder(z), mu, logvar

    def reparametrize(self, mu, logvar):
        """This function implements the reparametrisation trick:
        z = h(x)*eta+g(x), eta is Gaussian
        """
        std = logvar.mul(0.5).exp_()

        # sample the normal distribution
        if torch.cuda.is_available():
            eta = torch.cuda.FloatTensor(std.size()).normal_()
        else:
            eta = torch.FloatTensor(std.size()).normal_()

        return eta.mul(std).add_(mu)

```

```python
model = VAE()
optimizer = torch.optim.Adam(
    model.parameters(), lr=learning_rate, weight_decay=1e-5)

if torch.cuda.is_available():
    model.cuda()

model.train()
```

    VAE(
      (base_encoder): Sequential(
        (0): Linear(in_features=784, out_features=400, bias=True)
        (1): ReLU(inplace)
      )
      (mean_encoder): Sequential(
        (0): Sequential(
          (0): Linear(in_features=784, out_features=400, bias=True)
          (1): ReLU(inplace)
        )
        (1): Linear(in_features=400, out_features=20, bias=True)
      )
      (var_encoder): Sequential(
        (0): Sequential(
          (0): Linear(in_features=784, out_features=400, bias=True)
          (1): ReLU(inplace)
        )
        (1): Linear(in_features=400, out_features=20, bias=True)
      )
      (decoder): Sequential(
        (0): Linear(in_features=20, out_features=400, bias=True)
        (1): ReLU(inplace)
        (2): Linear(in_features=400, out_features=784, bias=True)
        (3): Sigmoid()
      )
    )

```python
mse = nn.MSELoss(size_average=False)
bce = nn.BCELoss(size_average=False)

def loss_f(x_hat, x, mu, logvar):
    """
    x_hat: generated img.
    x: origin img.
    mu: latent mean
    logvar: latent log variance (log(sigma^2))
    """

    # E[log P(x|z)]
    approx_err = bce(x_hat, x)  # (-||x-f(z)||^2/(2c))

    # The divergence between two Gaussians can be calculated in closed form.
    # loss = 0.5 * sum(1 + log(sigma^2) - mu^2 - sigma^2)
    _kl = mu.pow(2).add_(logvar.exp()).mul_(-1).add_(1).add_(logvar)
    KL = torch.sum(_kl).mul_(-0.5)

    return approx_err + KL
```

## Train

```python
model.train()
num_epochs = 100

for epoch in range(num_epochs):
    train_loss = 0
    for batch_idx, (x, y) in enumerate(train_dl):
        x = x.view(x.size(0), -1) # batchsize x 128

        if torch.cuda.is_available():
            x = x.cuda()

        y_hat, mu, logvar = model(x)

        loss = loss_f(y_hat, x, mu, logvar)

        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        train_loss += loss.data.item()

        if batch_idx % 100 == 0:
            print('Train Epoch: {} [{}/{} ({:.0f}%)]\tLoss: {:.6f}'.format(
                epoch,
                batch_idx * len(x),
                len(train_dl.dataset), 100. * batch_idx / len(train_dl),
                loss.data.item() / len(x)))

    print('>>  Epoch: {} Average loss: {:.4f}'.format(
        epoch, train_loss / len(train_dl.dataset)))

    if epoch % 10 == 0:
        save_image(toImg(x.cpu().data), '../results/01-autoencoder/vae/image_{}_i.png'.format(epoch))
        save_image(toImg(y_hat.cpu().data), '../results/01-autoencoder/vae/image_{}_o.png'.format(epoch))
```

    Train Epoch: 0 [0/60000 (0%)]	Loss: 548.798096
    Train Epoch: 0 [12800/60000 (21%)]	Loss: -10400.995117
    Train Epoch: 0 [25600/60000 (43%)]	Loss: -13520.972656
    Train Epoch: 0 [38400/60000 (64%)]	Loss: -15273.527344
    Train Epoch: 0 [51200/60000 (85%)]	Loss: -15952.123047
    >>  Epoch: 0 Average loss: -12749.0210
    Train Epoch: 1 [0/60000 (0%)]	Loss: -16029.354492
    Train Epoch: 1 [12800/60000 (21%)]	Loss: -16067.035156
    Train Epoch: 1 [25600/60000 (43%)]	Loss: -16273.087891
    Train Epoch: 1 [38400/60000 (64%)]	Loss: -16572.113281
    Train Epoch: 1 [51200/60000 (85%)]	Loss: -16374.041016
    >>  Epoch: 1 Average loss: -16285.1504
    Train Epoch: 2 [0/60000 (0%)]	Loss: -16515.455078
    Train Epoch: 2 [12800/60000 (21%)]	Loss: -16544.494141
    Train Epoch: 2 [25600/60000 (43%)]	Loss: -16339.613281
    Train Epoch: 2 [38400/60000 (64%)]	Loss: -16133.729492
    Train Epoch: 2 [51200/60000 (85%)]	Loss: -16039.532227
    >>  Epoch: 2 Average loss: -16280.5579
    Train Epoch: 3 [0/60000 (0%)]	Loss: -16026.655273
    Train Epoch: 3 [12800/60000 (21%)]	Loss: -16169.336914
    Train Epoch: 3 [25600/60000 (43%)]	Loss: -16350.911133
    Train Epoch: 3 [38400/60000 (64%)]	Loss: -16016.779297
    Train Epoch: 3 [51200/60000 (85%)]	Loss: -16301.458008
    >>  Epoch: 3 Average loss: -16044.4319
    Train Epoch: 4 [0/60000 (0%)]	Loss: -16284.871094
    Train Epoch: 4 [12800/60000 (21%)]	Loss: -16331.748047
    Train Epoch: 4 [25600/60000 (43%)]	Loss: -15999.494141
    Train Epoch: 4 [38400/60000 (64%)]	Loss: -15769.664062
    Train Epoch: 4 [51200/60000 (85%)]	Loss: -15759.160156
    >>  Epoch: 4 Average loss: -15910.8688
    Train Epoch: 5 [0/60000 (0%)]	Loss: -16160.987305
    Train Epoch: 5 [12800/60000 (21%)]	Loss: -16652.466797
    Train Epoch: 5 [25600/60000 (43%)]	Loss: -15019.367188
    Train Epoch: 5 [38400/60000 (64%)]	Loss: -15418.454102
    Train Epoch: 5 [51200/60000 (85%)]	Loss: -15785.478516
    >>  Epoch: 5 Average loss: -15739.4408
    Train Epoch: 6 [0/60000 (0%)]	Loss: -15510.125977
    Train Epoch: 6 [12800/60000 (21%)]	Loss: -15574.984375
    Train Epoch: 6 [25600/60000 (43%)]	Loss: -15398.647461
    Train Epoch: 6 [38400/60000 (64%)]	Loss: -15153.815430
    Train Epoch: 6 [51200/60000 (85%)]	Loss: -15735.817383
    >>  Epoch: 6 Average loss: -15629.2851
    Train Epoch: 7 [0/60000 (0%)]	Loss: -15564.513672
    Train Epoch: 7 [12800/60000 (21%)]	Loss: -15361.147461
    Train Epoch: 7 [25600/60000 (43%)]	Loss: -15257.525391
    Train Epoch: 7 [38400/60000 (64%)]	Loss: -15549.546875
    Train Epoch: 7 [51200/60000 (85%)]	Loss: -15512.875977
    >>  Epoch: 7 Average loss: -15618.3374
    Train Epoch: 8 [0/60000 (0%)]	Loss: -15687.753906
    Train Epoch: 8 [12800/60000 (21%)]	Loss: -15689.877930
    Train Epoch: 8 [25600/60000 (43%)]	Loss: -15374.916016
    Train Epoch: 8 [38400/60000 (64%)]	Loss: -15942.115234
    Train Epoch: 8 [51200/60000 (85%)]	Loss: -14915.119141
    >>  Epoch: 8 Average loss: -15655.1822
    Train Epoch: 9 [0/60000 (0%)]	Loss: -14978.934570
    Train Epoch: 9 [12800/60000 (21%)]	Loss: -15950.153320
    Train Epoch: 9 [25600/60000 (43%)]	Loss: -16483.396484
    Train Epoch: 9 [38400/60000 (64%)]	Loss: -15933.653320
    Train Epoch: 9 [51200/60000 (85%)]	Loss: -16408.232422
    >>  Epoch: 9 Average loss: -15699.3873
    Train Epoch: 10 [0/60000 (0%)]	Loss: -15647.618164
    Train Epoch: 10 [12800/60000 (21%)]	Loss: -16330.379883
    Train Epoch: 10 [25600/60000 (43%)]	Loss: -15949.466797
    Train Epoch: 10 [38400/60000 (64%)]	Loss: -16028.821289
    Train Epoch: 10 [51200/60000 (85%)]	Loss: -15954.647461
    >>  Epoch: 10 Average loss: -15707.1465
    Train Epoch: 11 [0/60000 (0%)]	Loss: -15906.910156
    Train Epoch: 11 [12800/60000 (21%)]	Loss: -16228.386719
    Train Epoch: 11 [25600/60000 (43%)]	Loss: -15131.728516
    Train Epoch: 11 [38400/60000 (64%)]	Loss: -15891.877930
    Train Epoch: 11 [51200/60000 (85%)]	Loss: -16029.520508
    >>  Epoch: 11 Average loss: -15761.8662
    Train Epoch: 12 [0/60000 (0%)]	Loss: -15814.916016
    Train Epoch: 12 [12800/60000 (21%)]	Loss: -15640.537109
    Train Epoch: 12 [25600/60000 (43%)]	Loss: -15681.725586
    Train Epoch: 12 [38400/60000 (64%)]	Loss: -15612.296875
    Train Epoch: 12 [51200/60000 (85%)]	Loss: -15969.593750
    >>  Epoch: 12 Average loss: -15755.1473
    Train Epoch: 13 [0/60000 (0%)]	Loss: -15734.547852
    Train Epoch: 13 [12800/60000 (21%)]	Loss: -15912.906250
    Train Epoch: 13 [25600/60000 (43%)]	Loss: -15386.783203
    Train Epoch: 13 [38400/60000 (64%)]	Loss: -15979.217773
    Train Epoch: 13 [51200/60000 (85%)]	Loss: -15417.867188
    >>  Epoch: 13 Average loss: -15720.0671
    Train Epoch: 14 [0/60000 (0%)]	Loss: -15656.513672
    Train Epoch: 14 [12800/60000 (21%)]	Loss: -16088.073242
    Train Epoch: 14 [25600/60000 (43%)]	Loss: -15439.639648
    Train Epoch: 14 [38400/60000 (64%)]	Loss: -15694.291992
    Train Epoch: 14 [51200/60000 (85%)]	Loss: -15735.992188
    >>  Epoch: 14 Average loss: -15731.8497
    Train Epoch: 15 [0/60000 (0%)]	Loss: -16031.658203
    Train Epoch: 15 [12800/60000 (21%)]	Loss: -15363.505859
    Train Epoch: 15 [25600/60000 (43%)]	Loss: -15335.965820
    Train Epoch: 15 [38400/60000 (64%)]	Loss: -15611.947266
    Train Epoch: 15 [51200/60000 (85%)]	Loss: -15649.885742
    >>  Epoch: 15 Average loss: -15741.5805
    Train Epoch: 16 [0/60000 (0%)]	Loss: -15287.258789
    Train Epoch: 16 [12800/60000 (21%)]	Loss: -15810.520508
    Train Epoch: 16 [25600/60000 (43%)]	Loss: -15941.052734
    Train Epoch: 16 [38400/60000 (64%)]	Loss: -15806.700195
    Train Epoch: 16 [51200/60000 (85%)]	Loss: -16056.616211
    >>  Epoch: 16 Average loss: -15741.6051
    Train Epoch: 17 [0/60000 (0%)]	Loss: -15775.357422
    Train Epoch: 17 [12800/60000 (21%)]	Loss: -15751.205078
    Train Epoch: 17 [25600/60000 (43%)]	Loss: -16333.038086
    Train Epoch: 17 [38400/60000 (64%)]	Loss: -16151.360352
    Train Epoch: 17 [51200/60000 (85%)]	Loss: -16332.794922
    >>  Epoch: 17 Average loss: -15736.9253
    Train Epoch: 18 [0/60000 (0%)]	Loss: -15776.823242
    Train Epoch: 18 [12800/60000 (21%)]	Loss: -15912.367188
    Train Epoch: 18 [25600/60000 (43%)]	Loss: -15632.143555
    Train Epoch: 18 [38400/60000 (64%)]	Loss: -15429.823242
    Train Epoch: 18 [51200/60000 (85%)]	Loss: -16084.385742
    >>  Epoch: 18 Average loss: -15756.3129
    Train Epoch: 19 [0/60000 (0%)]	Loss: -15728.345703
    Train Epoch: 19 [12800/60000 (21%)]	Loss: -15664.066406
    Train Epoch: 19 [25600/60000 (43%)]	Loss: -15685.458984
    Train Epoch: 19 [38400/60000 (64%)]	Loss: -15815.618164
    Train Epoch: 19 [51200/60000 (85%)]	Loss: -14626.311523
    >>  Epoch: 19 Average loss: -15748.6681
    Train Epoch: 20 [0/60000 (0%)]	Loss: -15371.667969
    Train Epoch: 20 [12800/60000 (21%)]	Loss: -15351.987305
    Train Epoch: 20 [25600/60000 (43%)]	Loss: -15516.359375
    Train Epoch: 20 [38400/60000 (64%)]	Loss: -15894.355469
    Train Epoch: 20 [51200/60000 (85%)]	Loss: -15774.041016
    >>  Epoch: 20 Average loss: -15764.9767
    Train Epoch: 21 [0/60000 (0%)]	Loss: -16372.184570
    Train Epoch: 21 [12800/60000 (21%)]	Loss: -15606.973633
    Train Epoch: 21 [25600/60000 (43%)]	Loss: -15101.914062
    Train Epoch: 21 [38400/60000 (64%)]	Loss: -15878.699219
    Train Epoch: 21 [51200/60000 (85%)]	Loss: -15291.368164
    >>  Epoch: 21 Average loss: -15746.8408
    Train Epoch: 22 [0/60000 (0%)]	Loss: -15855.291992
    Train Epoch: 22 [12800/60000 (21%)]	Loss: -16354.753906
    Train Epoch: 22 [25600/60000 (43%)]	Loss: -16244.956055
    Train Epoch: 22 [38400/60000 (64%)]	Loss: -16145.729492
    Train Epoch: 22 [51200/60000 (85%)]	Loss: -15999.291992
    >>  Epoch: 22 Average loss: -15754.9529
    Train Epoch: 23 [0/60000 (0%)]	Loss: -15501.169922
    Train Epoch: 23 [12800/60000 (21%)]	Loss: -15767.891602
    Train Epoch: 23 [25600/60000 (43%)]	Loss: -15450.406250
    Train Epoch: 23 [38400/60000 (64%)]	Loss: -16731.027344
    Train Epoch: 23 [51200/60000 (85%)]	Loss: -15620.701172
    >>  Epoch: 23 Average loss: -15784.9061
    Train Epoch: 24 [0/60000 (0%)]	Loss: -15639.106445
    Train Epoch: 24 [12800/60000 (21%)]	Loss: -16171.797852
    Train Epoch: 24 [25600/60000 (43%)]	Loss: -16093.580078
    Train Epoch: 24 [38400/60000 (64%)]	Loss: -15425.541992
    Train Epoch: 24 [51200/60000 (85%)]	Loss: -15944.828125
    >>  Epoch: 24 Average loss: -15775.7069
    Train Epoch: 25 [0/60000 (0%)]	Loss: -15554.327148
    Train Epoch: 25 [12800/60000 (21%)]	Loss: -15968.566406
    Train Epoch: 25 [25600/60000 (43%)]	Loss: -15659.537109
    Train Epoch: 25 [38400/60000 (64%)]	Loss: -15878.447266
    Train Epoch: 25 [51200/60000 (85%)]	Loss: -16219.924805
    >>  Epoch: 25 Average loss: -15797.3291
    Train Epoch: 26 [0/60000 (0%)]	Loss: -15969.476562
    Train Epoch: 26 [12800/60000 (21%)]	Loss: -15485.171875
    Train Epoch: 26 [25600/60000 (43%)]	Loss: -15726.582031
    Train Epoch: 26 [38400/60000 (64%)]	Loss: -15938.704102
    Train Epoch: 26 [51200/60000 (85%)]	Loss: -15694.555664
    >>  Epoch: 26 Average loss: -15808.7790
    Train Epoch: 27 [0/60000 (0%)]	Loss: -15663.542969
    Train Epoch: 27 [12800/60000 (21%)]	Loss: -15626.659180
    Train Epoch: 27 [25600/60000 (43%)]	Loss: -16402.958984
    Train Epoch: 27 [38400/60000 (64%)]	Loss: -15684.948242
    Train Epoch: 27 [51200/60000 (85%)]	Loss: -15679.839844
    >>  Epoch: 27 Average loss: -15828.3127
    Train Epoch: 28 [0/60000 (0%)]	Loss: -15980.693359
    Train Epoch: 28 [12800/60000 (21%)]	Loss: -15336.357422
    Train Epoch: 28 [25600/60000 (43%)]	Loss: -15683.900391
    Train Epoch: 28 [38400/60000 (64%)]	Loss: -15997.925781
    Train Epoch: 28 [51200/60000 (85%)]	Loss: -16086.597656
    >>  Epoch: 28 Average loss: -15855.4075
    Train Epoch: 29 [0/60000 (0%)]	Loss: -15888.863281
    Train Epoch: 29 [12800/60000 (21%)]	Loss: -15684.571289
    Train Epoch: 29 [25600/60000 (43%)]	Loss: -15961.804688
    Train Epoch: 29 [38400/60000 (64%)]	Loss: -15849.370117
    Train Epoch: 29 [51200/60000 (85%)]	Loss: -15698.311523
    >>  Epoch: 29 Average loss: -15878.0026
    Train Epoch: 30 [0/60000 (0%)]	Loss: -16085.376953
    Train Epoch: 30 [12800/60000 (21%)]	Loss: -16326.175781
    Train Epoch: 30 [25600/60000 (43%)]	Loss: -15949.460938
    Train Epoch: 30 [38400/60000 (64%)]	Loss: -16597.437500
    Train Epoch: 30 [51200/60000 (85%)]	Loss: -16148.072266
    >>  Epoch: 30 Average loss: -15926.7902
    Train Epoch: 31 [0/60000 (0%)]	Loss: -15720.159180
    Train Epoch: 31 [12800/60000 (21%)]	Loss: -15995.453125
    Train Epoch: 31 [25600/60000 (43%)]	Loss: -16108.129883
    Train Epoch: 31 [38400/60000 (64%)]	Loss: -16219.212891
    Train Epoch: 31 [51200/60000 (85%)]	Loss: -15874.607422
    >>  Epoch: 31 Average loss: -15964.4744
    Train Epoch: 32 [0/60000 (0%)]	Loss: -15681.047852
    Train Epoch: 32 [12800/60000 (21%)]	Loss: -15873.328125
    Train Epoch: 32 [25600/60000 (43%)]	Loss: -15883.899414
    Train Epoch: 32 [38400/60000 (64%)]	Loss: -16537.060547
    Train Epoch: 32 [51200/60000 (85%)]	Loss: -15421.623047
    >>  Epoch: 32 Average loss: -16004.1485
    Train Epoch: 33 [0/60000 (0%)]	Loss: -16307.760742
    Train Epoch: 33 [12800/60000 (21%)]	Loss: -16323.730469
    Train Epoch: 33 [25600/60000 (43%)]	Loss: -15635.327148
    Train Epoch: 33 [38400/60000 (64%)]	Loss: -15912.753906
    Train Epoch: 33 [51200/60000 (85%)]	Loss: -16238.466797
    >>  Epoch: 33 Average loss: -16063.7850
    Train Epoch: 34 [0/60000 (0%)]	Loss: -16172.517578
    Train Epoch: 34 [12800/60000 (21%)]	Loss: -15760.675781
    Train Epoch: 34 [25600/60000 (43%)]	Loss: -16394.808594
    Train Epoch: 34 [38400/60000 (64%)]	Loss: -16498.275391
    Train Epoch: 34 [51200/60000 (85%)]	Loss: -15712.812500
    >>  Epoch: 34 Average loss: -16138.8057
    Train Epoch: 35 [0/60000 (0%)]	Loss: -16025.337891
    Train Epoch: 35 [12800/60000 (21%)]	Loss: -16540.029297
    Train Epoch: 35 [25600/60000 (43%)]	Loss: -15970.066406
    Train Epoch: 35 [38400/60000 (64%)]	Loss: -15916.967773
    Train Epoch: 35 [51200/60000 (85%)]	Loss: -16363.433594
    >>  Epoch: 35 Average loss: -16232.1018
    Train Epoch: 36 [0/60000 (0%)]	Loss: -15838.261719
    Train Epoch: 36 [12800/60000 (21%)]	Loss: -15826.887695
    Train Epoch: 36 [25600/60000 (43%)]	Loss: -16310.962891
    Train Epoch: 36 [38400/60000 (64%)]	Loss: -16837.082031
    Train Epoch: 36 [51200/60000 (85%)]	Loss: -16399.076172
    >>  Epoch: 36 Average loss: -16367.7808
    Train Epoch: 37 [0/60000 (0%)]	Loss: -15912.750000
    Train Epoch: 37 [12800/60000 (21%)]	Loss: -16463.654297
    Train Epoch: 37 [25600/60000 (43%)]	Loss: -16507.283203
    Train Epoch: 37 [38400/60000 (64%)]	Loss: -16087.551758
    Train Epoch: 37 [51200/60000 (85%)]	Loss: -16615.046875
    >>  Epoch: 37 Average loss: -16458.7341
    Train Epoch: 38 [0/60000 (0%)]	Loss: -17020.490234
    Train Epoch: 38 [12800/60000 (21%)]	Loss: -16587.585938
    Train Epoch: 38 [25600/60000 (43%)]	Loss: -16319.344727
    Train Epoch: 38 [38400/60000 (64%)]	Loss: -16382.368164
    Train Epoch: 38 [51200/60000 (85%)]	Loss: -16504.347656
    >>  Epoch: 38 Average loss: -16632.4627
    Train Epoch: 39 [0/60000 (0%)]	Loss: -16381.448242
    Train Epoch: 39 [12800/60000 (21%)]	Loss: -16697.320312
    Train Epoch: 39 [25600/60000 (43%)]	Loss: -16804.128906
    Train Epoch: 39 [38400/60000 (64%)]	Loss: -16615.097656
    Train Epoch: 39 [51200/60000 (85%)]	Loss: -16981.410156
    >>  Epoch: 39 Average loss: -16810.5873
    Train Epoch: 40 [0/60000 (0%)]	Loss: -16861.273438
    Train Epoch: 40 [12800/60000 (21%)]	Loss: -17466.968750
    Train Epoch: 40 [25600/60000 (43%)]	Loss: -17402.136719
    Train Epoch: 40 [38400/60000 (64%)]	Loss: -16999.201172
    Train Epoch: 40 [51200/60000 (85%)]	Loss: -17495.255859
    >>  Epoch: 40 Average loss: -17060.1137
    Train Epoch: 41 [0/60000 (0%)]	Loss: -17004.109375
    Train Epoch: 41 [12800/60000 (21%)]	Loss: -17359.103516
    Train Epoch: 41 [25600/60000 (43%)]	Loss: -17511.921875
    Train Epoch: 41 [38400/60000 (64%)]	Loss: -17389.585938
    Train Epoch: 41 [51200/60000 (85%)]	Loss: -17074.890625
    >>  Epoch: 41 Average loss: -17282.8759
    Train Epoch: 42 [0/60000 (0%)]	Loss: -17634.613281
    Train Epoch: 42 [12800/60000 (21%)]	Loss: -17447.382812
    Train Epoch: 42 [25600/60000 (43%)]	Loss: -17465.964844
    Train Epoch: 42 [38400/60000 (64%)]	Loss: -17410.089844
    Train Epoch: 42 [51200/60000 (85%)]	Loss: -17276.812500
    >>  Epoch: 42 Average loss: -17441.8183
    Train Epoch: 43 [0/60000 (0%)]	Loss: -17767.224609
    Train Epoch: 43 [12800/60000 (21%)]	Loss: -17329.523438
    Train Epoch: 43 [25600/60000 (43%)]	Loss: -17272.613281
    Train Epoch: 43 [38400/60000 (64%)]	Loss: -17227.455078
    Train Epoch: 43 [51200/60000 (85%)]	Loss: -17635.570312
    >>  Epoch: 43 Average loss: -17500.9153
    Train Epoch: 44 [0/60000 (0%)]	Loss: -17490.695312
    Train Epoch: 44 [12800/60000 (21%)]	Loss: -17791.738281
    Train Epoch: 44 [25600/60000 (43%)]	Loss: -17543.478516
    Train Epoch: 44 [38400/60000 (64%)]	Loss: -17350.167969
    Train Epoch: 44 [51200/60000 (85%)]	Loss: -17349.462891
    >>  Epoch: 44 Average loss: -17593.8134
    Train Epoch: 45 [0/60000 (0%)]	Loss: -17580.316406
    Train Epoch: 45 [12800/60000 (21%)]	Loss: -17385.085938
    Train Epoch: 45 [25600/60000 (43%)]	Loss: -17382.095703
    Train Epoch: 45 [38400/60000 (64%)]	Loss: -17429.287109
    Train Epoch: 45 [51200/60000 (85%)]	Loss: -17536.187500
    >>  Epoch: 45 Average loss: -17564.7666
    Train Epoch: 46 [0/60000 (0%)]	Loss: -17598.343750
    Train Epoch: 46 [12800/60000 (21%)]	Loss: -17388.191406
    Train Epoch: 46 [25600/60000 (43%)]	Loss: -17293.800781
    Train Epoch: 46 [38400/60000 (64%)]	Loss: -17492.017578
    Train Epoch: 46 [51200/60000 (85%)]	Loss: -17604.408203
    >>  Epoch: 46 Average loss: -17465.0792
    Train Epoch: 47 [0/60000 (0%)]	Loss: -17610.906250
    Train Epoch: 47 [12800/60000 (21%)]	Loss: -17191.480469
    Train Epoch: 47 [25600/60000 (43%)]	Loss: -17691.476562
    Train Epoch: 47 [38400/60000 (64%)]	Loss: -17332.033203
    Train Epoch: 47 [51200/60000 (85%)]	Loss: -17568.968750
    >>  Epoch: 47 Average loss: -17531.3173
    Train Epoch: 48 [0/60000 (0%)]	Loss: -17307.283203
    Train Epoch: 48 [12800/60000 (21%)]	Loss: -17492.416016
    Train Epoch: 48 [25600/60000 (43%)]	Loss: -17337.798828
    Train Epoch: 48 [38400/60000 (64%)]	Loss: -17336.730469
    Train Epoch: 48 [51200/60000 (85%)]	Loss: -17419.738281
    >>  Epoch: 48 Average loss: -17327.9903
    Train Epoch: 49 [0/60000 (0%)]	Loss: -17431.869141
    Train Epoch: 49 [12800/60000 (21%)]	Loss: -17365.650391
    Train Epoch: 49 [25600/60000 (43%)]	Loss: -17182.134766
    Train Epoch: 49 [38400/60000 (64%)]	Loss: -16896.330078
    Train Epoch: 49 [51200/60000 (85%)]	Loss: -17013.939453
    >>  Epoch: 49 Average loss: -17136.9201
    Train Epoch: 50 [0/60000 (0%)]	Loss: -17015.525391
    Train Epoch: 50 [12800/60000 (21%)]	Loss: -16782.992188
    Train Epoch: 50 [25600/60000 (43%)]	Loss: -17026.062500
    Train Epoch: 50 [38400/60000 (64%)]	Loss: -17217.783203
    Train Epoch: 50 [51200/60000 (85%)]	Loss: -17103.363281
    >>  Epoch: 50 Average loss: -17111.1771
    Train Epoch: 51 [0/60000 (0%)]	Loss: -17389.720703
    Train Epoch: 51 [12800/60000 (21%)]	Loss: -17199.408203
    Train Epoch: 51 [25600/60000 (43%)]	Loss: -17102.441406
    Train Epoch: 51 [38400/60000 (64%)]	Loss: -17427.525391
    Train Epoch: 51 [51200/60000 (85%)]	Loss: -16863.777344
    >>  Epoch: 51 Average loss: -17124.5835
    Train Epoch: 52 [0/60000 (0%)]	Loss: -16963.330078
    Train Epoch: 52 [12800/60000 (21%)]	Loss: -16904.535156
    Train Epoch: 52 [25600/60000 (43%)]	Loss: -17380.410156
    Train Epoch: 52 [38400/60000 (64%)]	Loss: -16948.828125
    Train Epoch: 52 [51200/60000 (85%)]	Loss: -17177.294922
    >>  Epoch: 52 Average loss: -17001.3476
    Train Epoch: 53 [0/60000 (0%)]	Loss: -16871.671875
    Train Epoch: 53 [12800/60000 (21%)]	Loss: -16856.982422
    Train Epoch: 53 [25600/60000 (43%)]	Loss: -16822.964844
    Train Epoch: 53 [38400/60000 (64%)]	Loss: -16449.992188
    Train Epoch: 53 [51200/60000 (85%)]	Loss: -17396.875000
    >>  Epoch: 53 Average loss: -16953.9299
    Train Epoch: 54 [0/60000 (0%)]	Loss: -17157.412109
    Train Epoch: 54 [12800/60000 (21%)]	Loss: -16841.775391
    Train Epoch: 54 [25600/60000 (43%)]	Loss: -16886.615234
    Train Epoch: 54 [38400/60000 (64%)]	Loss: -16942.320312
    Train Epoch: 54 [51200/60000 (85%)]	Loss: -16818.152344
    >>  Epoch: 54 Average loss: -16996.4448
    Train Epoch: 55 [0/60000 (0%)]	Loss: -16912.093750
    Train Epoch: 55 [12800/60000 (21%)]	Loss: -16525.378906
    Train Epoch: 55 [25600/60000 (43%)]	Loss: -17093.421875
    Train Epoch: 55 [38400/60000 (64%)]	Loss: -17006.072266
    Train Epoch: 55 [51200/60000 (85%)]	Loss: -16766.722656
    >>  Epoch: 55 Average loss: -16809.1004
    Train Epoch: 56 [0/60000 (0%)]	Loss: -16664.152344
    Train Epoch: 56 [12800/60000 (21%)]	Loss: -16771.865234
    Train Epoch: 56 [25600/60000 (43%)]	Loss: -16739.525391
    Train Epoch: 56 [38400/60000 (64%)]	Loss: -16652.421875
    Train Epoch: 56 [51200/60000 (85%)]	Loss: -17241.503906
    >>  Epoch: 56 Average loss: -16875.5294
    Train Epoch: 57 [0/60000 (0%)]	Loss: -16731.355469
    Train Epoch: 57 [12800/60000 (21%)]	Loss: -16529.789062
    Train Epoch: 57 [25600/60000 (43%)]	Loss: -16789.970703
    Train Epoch: 57 [38400/60000 (64%)]	Loss: -16686.048828
    Train Epoch: 57 [51200/60000 (85%)]	Loss: -17631.421875
    >>  Epoch: 57 Average loss: -16833.4013
    Train Epoch: 58 [0/60000 (0%)]	Loss: -16920.191406
    Train Epoch: 58 [12800/60000 (21%)]	Loss: -17039.376953
    Train Epoch: 58 [25600/60000 (43%)]	Loss: -16618.492188
    Train Epoch: 58 [38400/60000 (64%)]	Loss: -16558.892578
    Train Epoch: 58 [51200/60000 (85%)]	Loss: -16741.796875
    >>  Epoch: 58 Average loss: -16736.3788
    Train Epoch: 59 [0/60000 (0%)]	Loss: -16661.085938
    Train Epoch: 59 [12800/60000 (21%)]	Loss: -16362.338867
    Train Epoch: 59 [25600/60000 (43%)]	Loss: -16555.357422
    Train Epoch: 59 [38400/60000 (64%)]	Loss: -16383.298828
    Train Epoch: 59 [51200/60000 (85%)]	Loss: -16890.437500
    >>  Epoch: 59 Average loss: -16539.4592
    Train Epoch: 60 [0/60000 (0%)]	Loss: -16634.007812
    Train Epoch: 60 [12800/60000 (21%)]	Loss: -16557.408203
    Train Epoch: 60 [25600/60000 (43%)]	Loss: -16968.511719
    Train Epoch: 60 [38400/60000 (64%)]	Loss: -16812.199219
    Train Epoch: 60 [51200/60000 (85%)]	Loss: -16647.976562
    >>  Epoch: 60 Average loss: -16661.2688
    Train Epoch: 61 [0/60000 (0%)]	Loss: -17091.906250
    Train Epoch: 61 [12800/60000 (21%)]	Loss: -16241.574219
    Train Epoch: 61 [25600/60000 (43%)]	Loss: -16217.345703
    Train Epoch: 61 [38400/60000 (64%)]	Loss: -16891.384766
    Train Epoch: 61 [51200/60000 (85%)]	Loss: -16775.207031
    >>  Epoch: 61 Average loss: -16609.7100
    Train Epoch: 62 [0/60000 (0%)]	Loss: -17134.527344
    Train Epoch: 62 [12800/60000 (21%)]	Loss: -16760.613281
    Train Epoch: 62 [25600/60000 (43%)]	Loss: -16567.193359
    Train Epoch: 62 [38400/60000 (64%)]	Loss: -16474.320312
    Train Epoch: 62 [51200/60000 (85%)]	Loss: -16455.552734
    >>  Epoch: 62 Average loss: -16746.5140
    Train Epoch: 63 [0/60000 (0%)]	Loss: -17373.203125
    Train Epoch: 63 [12800/60000 (21%)]	Loss: -16973.087891
    Train Epoch: 63 [25600/60000 (43%)]	Loss: -16784.939453
    Train Epoch: 63 [38400/60000 (64%)]	Loss: -17112.052734
    Train Epoch: 63 [51200/60000 (85%)]	Loss: -16774.017578
    >>  Epoch: 63 Average loss: -16980.7624
    Train Epoch: 64 [0/60000 (0%)]	Loss: -17215.111328
    Train Epoch: 64 [12800/60000 (21%)]	Loss: -16998.052734
    Train Epoch: 64 [25600/60000 (43%)]	Loss: -16724.621094
    Train Epoch: 64 [38400/60000 (64%)]	Loss: -16849.472656
    Train Epoch: 64 [51200/60000 (85%)]	Loss: -16990.341797
    >>  Epoch: 64 Average loss: -16977.7651
    Train Epoch: 65 [0/60000 (0%)]	Loss: -16878.310547
    Train Epoch: 65 [12800/60000 (21%)]	Loss: -16966.154297
    Train Epoch: 65 [25600/60000 (43%)]	Loss: -16739.839844
    Train Epoch: 65 [38400/60000 (64%)]	Loss: -16463.107422
    Train Epoch: 65 [51200/60000 (85%)]	Loss: -17488.527344
    >>  Epoch: 65 Average loss: -16794.3002
    Train Epoch: 66 [0/60000 (0%)]	Loss: -16805.675781
    Train Epoch: 66 [12800/60000 (21%)]	Loss: -16661.839844
    Train Epoch: 66 [25600/60000 (43%)]	Loss: -16651.310547
    Train Epoch: 66 [38400/60000 (64%)]	Loss: -17002.986328
    Train Epoch: 66 [51200/60000 (85%)]	Loss: -17010.789062
    >>  Epoch: 66 Average loss: -16781.7397
    Train Epoch: 67 [0/60000 (0%)]	Loss: -17037.335938
    Train Epoch: 67 [12800/60000 (21%)]	Loss: -16572.474609
    Train Epoch: 67 [25600/60000 (43%)]	Loss: -16418.179688
    Train Epoch: 67 [38400/60000 (64%)]	Loss: -17102.197266
    Train Epoch: 67 [51200/60000 (85%)]	Loss: -16854.257812
    >>  Epoch: 67 Average loss: -16788.2711
    Train Epoch: 68 [0/60000 (0%)]	Loss: -16557.568359
    Train Epoch: 68 [12800/60000 (21%)]	Loss: -16540.048828
    Train Epoch: 68 [25600/60000 (43%)]	Loss: -16928.925781
    Train Epoch: 68 [38400/60000 (64%)]	Loss: -16511.935547
    Train Epoch: 68 [51200/60000 (85%)]	Loss: -16502.919922
    >>  Epoch: 68 Average loss: -16653.5732
    Train Epoch: 69 [0/60000 (0%)]	Loss: -17173.617188
    Train Epoch: 69 [12800/60000 (21%)]	Loss: -16657.203125
    Train Epoch: 69 [25600/60000 (43%)]	Loss: -16737.843750
    Train Epoch: 69 [38400/60000 (64%)]	Loss: -16758.394531
    Train Epoch: 69 [51200/60000 (85%)]	Loss: -16613.031250
    >>  Epoch: 69 Average loss: -16693.8696
    Train Epoch: 70 [0/60000 (0%)]	Loss: -16670.583984
    Train Epoch: 70 [12800/60000 (21%)]	Loss: -16123.583984
    Train Epoch: 70 [25600/60000 (43%)]	Loss: -17051.111328
    Train Epoch: 70 [38400/60000 (64%)]	Loss: -16375.357422
    Train Epoch: 70 [51200/60000 (85%)]	Loss: -16055.228516
    >>  Epoch: 70 Average loss: -16574.8070
    Train Epoch: 71 [0/60000 (0%)]	Loss: -17300.732422
    Train Epoch: 71 [12800/60000 (21%)]	Loss: -16737.800781
    Train Epoch: 71 [25600/60000 (43%)]	Loss: -16490.570312
    Train Epoch: 71 [38400/60000 (64%)]	Loss: -16617.035156
    Train Epoch: 71 [51200/60000 (85%)]	Loss: -17057.218750
    >>  Epoch: 71 Average loss: -16812.1958
    Train Epoch: 72 [0/60000 (0%)]	Loss: -16844.027344
    Train Epoch: 72 [12800/60000 (21%)]	Loss: -17196.638672
    Train Epoch: 72 [25600/60000 (43%)]	Loss: -17155.652344
    Train Epoch: 72 [38400/60000 (64%)]	Loss: -16815.871094
    Train Epoch: 72 [51200/60000 (85%)]	Loss: -16868.396484
    >>  Epoch: 72 Average loss: -17024.8841
    Train Epoch: 73 [0/60000 (0%)]	Loss: -16993.830078
    Train Epoch: 73 [12800/60000 (21%)]	Loss: -16626.347656
    Train Epoch: 73 [25600/60000 (43%)]	Loss: -16572.027344
    Train Epoch: 73 [38400/60000 (64%)]	Loss: -16964.886719
    Train Epoch: 73 [51200/60000 (85%)]	Loss: -16832.048828
    >>  Epoch: 73 Average loss: -16820.0449
    Train Epoch: 74 [0/60000 (0%)]	Loss: -16726.564453
    Train Epoch: 74 [12800/60000 (21%)]	Loss: -17005.005859
    Train Epoch: 74 [25600/60000 (43%)]	Loss: -16827.363281
    Train Epoch: 74 [38400/60000 (64%)]	Loss: -16812.699219
    Train Epoch: 74 [51200/60000 (85%)]	Loss: -17007.255859
    >>  Epoch: 74 Average loss: -16855.9665
    Train Epoch: 75 [0/60000 (0%)]	Loss: -17013.611328
    Train Epoch: 75 [12800/60000 (21%)]	Loss: -16890.156250
    Train Epoch: 75 [25600/60000 (43%)]	Loss: -16731.054688
    Train Epoch: 75 [38400/60000 (64%)]	Loss: -16648.509766
    Train Epoch: 75 [51200/60000 (85%)]	Loss: -16863.269531
    >>  Epoch: 75 Average loss: -16755.0141
    Train Epoch: 76 [0/60000 (0%)]	Loss: -16395.185547
    Train Epoch: 76 [12800/60000 (21%)]	Loss: -16981.302734
    Train Epoch: 76 [25600/60000 (43%)]	Loss: -16936.773438
    Train Epoch: 76 [38400/60000 (64%)]	Loss: -16885.177734
    Train Epoch: 76 [51200/60000 (85%)]	Loss: -16895.365234
    >>  Epoch: 76 Average loss: -16824.4441
    Train Epoch: 77 [0/60000 (0%)]	Loss: -16717.771484
    Train Epoch: 77 [12800/60000 (21%)]	Loss: -16638.263672
    Train Epoch: 77 [25600/60000 (43%)]	Loss: -16863.125000
    Train Epoch: 77 [38400/60000 (64%)]	Loss: -16758.496094
    Train Epoch: 77 [51200/60000 (85%)]	Loss: -16392.998047
    >>  Epoch: 77 Average loss: -16704.3087
    Train Epoch: 78 [0/60000 (0%)]	Loss: -16794.507812
    Train Epoch: 78 [12800/60000 (21%)]	Loss: -16618.425781
    Train Epoch: 78 [25600/60000 (43%)]	Loss: -16530.740234
    Train Epoch: 78 [38400/60000 (64%)]	Loss: -16447.982422
    Train Epoch: 78 [51200/60000 (85%)]	Loss: -16567.976562
    >>  Epoch: 78 Average loss: -16514.9580
    Train Epoch: 79 [0/60000 (0%)]	Loss: -16274.107422
    Train Epoch: 79 [12800/60000 (21%)]	Loss: -15965.790039
    Train Epoch: 79 [25600/60000 (43%)]	Loss: -16976.404297
    Train Epoch: 79 [38400/60000 (64%)]	Loss: -16533.996094
    Train Epoch: 79 [51200/60000 (85%)]	Loss: -16301.074219
    >>  Epoch: 79 Average loss: -16574.6297
    Train Epoch: 80 [0/60000 (0%)]	Loss: -16861.246094
    Train Epoch: 80 [12800/60000 (21%)]	Loss: -16715.041016
    Train Epoch: 80 [25600/60000 (43%)]	Loss: -16621.761719
    Train Epoch: 80 [38400/60000 (64%)]	Loss: -16638.947266
    Train Epoch: 80 [51200/60000 (85%)]	Loss: -16915.890625
    >>  Epoch: 80 Average loss: -16756.9602
    Train Epoch: 81 [0/60000 (0%)]	Loss: -17538.429688
    Train Epoch: 81 [12800/60000 (21%)]	Loss: -17190.730469
    Train Epoch: 81 [25600/60000 (43%)]	Loss: -17047.847656
    Train Epoch: 81 [38400/60000 (64%)]	Loss: -16756.128906
    Train Epoch: 81 [51200/60000 (85%)]	Loss: -16829.742188
    >>  Epoch: 81 Average loss: -16969.9282
    Train Epoch: 82 [0/60000 (0%)]	Loss: -17068.365234
    Train Epoch: 82 [12800/60000 (21%)]	Loss: -16725.025391
    Train Epoch: 82 [25600/60000 (43%)]	Loss: -16801.292969
    Train Epoch: 82 [38400/60000 (64%)]	Loss: -16457.238281
    Train Epoch: 82 [51200/60000 (85%)]	Loss: -16701.607422
    >>  Epoch: 82 Average loss: -16702.8252
    Train Epoch: 83 [0/60000 (0%)]	Loss: -17212.253906
    Train Epoch: 83 [12800/60000 (21%)]	Loss: -16595.898438
    Train Epoch: 83 [25600/60000 (43%)]	Loss: -16469.062500
    Train Epoch: 83 [38400/60000 (64%)]	Loss: -16930.515625
    Train Epoch: 83 [51200/60000 (85%)]	Loss: -16346.410156
    >>  Epoch: 83 Average loss: -16681.1220
    Train Epoch: 84 [0/60000 (0%)]	Loss: -16859.251953
    Train Epoch: 84 [12800/60000 (21%)]	Loss: -16471.759766
    Train Epoch: 84 [25600/60000 (43%)]	Loss: -16511.751953
    Train Epoch: 84 [38400/60000 (64%)]	Loss: -16661.656250
    Train Epoch: 84 [51200/60000 (85%)]	Loss: -16598.855469
    >>  Epoch: 84 Average loss: -16600.5367
    Train Epoch: 85 [0/60000 (0%)]	Loss: -16690.406250
    Train Epoch: 85 [12800/60000 (21%)]	Loss: -16376.686523
    Train Epoch: 85 [25600/60000 (43%)]	Loss: -17155.906250
    Train Epoch: 85 [38400/60000 (64%)]	Loss: -16336.201172
    Train Epoch: 85 [51200/60000 (85%)]	Loss: -16828.732422
    >>  Epoch: 85 Average loss: -16589.3381
    Train Epoch: 86 [0/60000 (0%)]	Loss: -16457.542969
    Train Epoch: 86 [12800/60000 (21%)]	Loss: -16272.050781
    Train Epoch: 86 [25600/60000 (43%)]	Loss: -16127.432617
    Train Epoch: 86 [38400/60000 (64%)]	Loss: -16412.441406
    Train Epoch: 86 [51200/60000 (85%)]	Loss: -15926.803711
    >>  Epoch: 86 Average loss: -16343.3976
    Train Epoch: 87 [0/60000 (0%)]	Loss: -16248.798828
    Train Epoch: 87 [12800/60000 (21%)]	Loss: -16318.914062
    Train Epoch: 87 [25600/60000 (43%)]	Loss: -16081.777344
    Train Epoch: 87 [38400/60000 (64%)]	Loss: -17063.484375
    Train Epoch: 87 [51200/60000 (85%)]	Loss: -16452.720703
    >>  Epoch: 87 Average loss: -16501.6742
    Train Epoch: 88 [0/60000 (0%)]	Loss: -16617.968750
    Train Epoch: 88 [12800/60000 (21%)]	Loss: -16762.095703
    Train Epoch: 88 [25600/60000 (43%)]	Loss: -16962.550781
    Train Epoch: 88 [38400/60000 (64%)]	Loss: -17069.365234
    Train Epoch: 88 [51200/60000 (85%)]	Loss: -17031.849609
    >>  Epoch: 88 Average loss: -16713.8787
    Train Epoch: 89 [0/60000 (0%)]	Loss: -16634.357422
    Train Epoch: 89 [12800/60000 (21%)]	Loss: -16780.300781
    Train Epoch: 89 [25600/60000 (43%)]	Loss: -17194.730469
    Train Epoch: 89 [38400/60000 (64%)]	Loss: -16642.835938
    Train Epoch: 89 [51200/60000 (85%)]	Loss: -16811.687500
    >>  Epoch: 89 Average loss: -16672.3675
    Train Epoch: 90 [0/60000 (0%)]	Loss: -16412.855469
    Train Epoch: 90 [12800/60000 (21%)]	Loss: -16578.832031
    Train Epoch: 90 [25600/60000 (43%)]	Loss: -16710.269531
    Train Epoch: 90 [38400/60000 (64%)]	Loss: -16771.345703
    Train Epoch: 90 [51200/60000 (85%)]	Loss: -16795.388672
    >>  Epoch: 90 Average loss: -16634.6537
    Train Epoch: 91 [0/60000 (0%)]	Loss: -16775.830078
    Train Epoch: 91 [12800/60000 (21%)]	Loss: -16958.330078
    Train Epoch: 91 [25600/60000 (43%)]	Loss: -16458.888672
    Train Epoch: 91 [38400/60000 (64%)]	Loss: -16233.967773
    Train Epoch: 91 [51200/60000 (85%)]	Loss: -16599.865234
    >>  Epoch: 91 Average loss: -16741.0825
    Train Epoch: 92 [0/60000 (0%)]	Loss: -17074.335938
    Train Epoch: 92 [12800/60000 (21%)]	Loss: -16677.009766
    Train Epoch: 92 [25600/60000 (43%)]	Loss: -16861.306641
    Train Epoch: 92 [38400/60000 (64%)]	Loss: -16204.630859
    Train Epoch: 92 [51200/60000 (85%)]	Loss: -16753.148438
    >>  Epoch: 92 Average loss: -16639.2805
    Train Epoch: 93 [0/60000 (0%)]	Loss: -16560.656250
    Train Epoch: 93 [12800/60000 (21%)]	Loss: -17046.308594
    Train Epoch: 93 [25600/60000 (43%)]	Loss: -16815.138672
    Train Epoch: 93 [38400/60000 (64%)]	Loss: -16509.044922
    Train Epoch: 93 [51200/60000 (85%)]	Loss: -16683.046875
    >>  Epoch: 93 Average loss: -16743.7790
    Train Epoch: 94 [0/60000 (0%)]	Loss: -16674.994141
    Train Epoch: 94 [12800/60000 (21%)]	Loss: -16944.917969
    Train Epoch: 94 [25600/60000 (43%)]	Loss: -16855.167969
    Train Epoch: 94 [38400/60000 (64%)]	Loss: -16546.781250
    Train Epoch: 94 [51200/60000 (85%)]	Loss: -16807.929688
    >>  Epoch: 94 Average loss: -16767.7904
    Train Epoch: 95 [0/60000 (0%)]	Loss: -17421.816406
    Train Epoch: 95 [12800/60000 (21%)]	Loss: -16699.886719
    Train Epoch: 95 [25600/60000 (43%)]	Loss: -16530.775391
    Train Epoch: 95 [38400/60000 (64%)]	Loss: -16462.689453
    Train Epoch: 95 [51200/60000 (85%)]	Loss: -16422.068359
    >>  Epoch: 95 Average loss: -16710.3526
    Train Epoch: 96 [0/60000 (0%)]	Loss: -17130.708984
    Train Epoch: 96 [12800/60000 (21%)]	Loss: -16741.439453
    Train Epoch: 96 [25600/60000 (43%)]	Loss: -16863.515625
    Train Epoch: 96 [38400/60000 (64%)]	Loss: -16242.845703
    Train Epoch: 96 [51200/60000 (85%)]	Loss: -16575.855469
    >>  Epoch: 96 Average loss: -16707.6568
    Train Epoch: 97 [0/60000 (0%)]	Loss: -16601.457031
    Train Epoch: 97 [12800/60000 (21%)]	Loss: -16559.931641
    Train Epoch: 97 [25600/60000 (43%)]	Loss: -16509.224609
    Train Epoch: 97 [38400/60000 (64%)]	Loss: -16780.644531
    Train Epoch: 97 [51200/60000 (85%)]	Loss: -16513.462891
    >>  Epoch: 97 Average loss: -16538.1087
    Train Epoch: 98 [0/60000 (0%)]	Loss: -17080.683594
    Train Epoch: 98 [12800/60000 (21%)]	Loss: -16576.988281
    Train Epoch: 98 [25600/60000 (43%)]	Loss: -16717.597656
    Train Epoch: 98 [38400/60000 (64%)]	Loss: -16582.261719
    Train Epoch: 98 [51200/60000 (85%)]	Loss: -16598.505859
    >>  Epoch: 98 Average loss: -16717.3681
    Train Epoch: 99 [0/60000 (0%)]	Loss: -16601.992188
    Train Epoch: 99 [12800/60000 (21%)]	Loss: -16578.425781
    Train Epoch: 99 [25600/60000 (43%)]	Loss: -16143.469727
    Train Epoch: 99 [38400/60000 (64%)]	Loss: -16556.958984
    Train Epoch: 99 [51200/60000 (85%)]	Loss: -16283.593750
    >>  Epoch: 99 Average loss: -16547.9954

```python
# As suggested in https://pytorch.org/tutorials/beginner/saving_loading_models.html
filename = '../saved-mdls/01-autoencoder/vae_{}e.pt'.format(epoch+1)
torch.save(model, filename)

print('model saved as: {}'.format(filename))
```

    model saved as: ./saved-mdls/01-autoencoder/vae_100e.pt

    /opt/anaconda3/lib/python3.7/site-packages/torch/serialization.py:256: UserWarning: Couldn't retrieve source code for container of type VAE. It won't be checked for correctness upon loading.
      "type " + obj.__name__ + ". It won't be checked "

## Using the Variational autoencoder

```python
# Model class must be defined somewhere
model = torch.load('./saved-mdls/01-autoencoder/vae_100e.pt',
                  map_location=torch.device('cpu'))
model.eval()
```

    /Users/andreamunafo/opt/anaconda3/envs/compiling-ai/lib/python3.6/site-packages/torch/serialization.py:493: SourceChangeWarning: source code of class 'torch.nn.modules.activation.ReLU' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
      warnings.warn(msg, SourceChangeWarning)
    /Users/andreamunafo/opt/anaconda3/envs/compiling-ai/lib/python3.6/site-packages/torch/serialization.py:493: SourceChangeWarning: source code of class 'torch.nn.modules.activation.Sigmoid' has changed. you can retrieve the original source code by accessing the object's source attribute or set `torch.nn.Module.dump_patches = True` and use the patch tool to revert the changes.
      warnings.warn(msg, SourceChangeWarning)

    VAE(
      (base_encoder): Sequential(
        (0): Linear(in_features=784, out_features=400, bias=True)
        (1): ReLU(inplace=True)
      )
      (mean_encoder): Sequential(
        (0): Sequential(
          (0): Linear(in_features=784, out_features=400, bias=True)
          (1): ReLU(inplace=True)
        )
        (1): Linear(in_features=400, out_features=20, bias=True)
      )
      (var_encoder): Sequential(
        (0): Sequential(
          (0): Linear(in_features=784, out_features=400, bias=True)
          (1): ReLU(inplace=True)
        )
        (1): Linear(in_features=400, out_features=20, bias=True)
      )
      (decoder): Sequential(
        (0): Linear(in_features=20, out_features=400, bias=True)
        (1): ReLU(inplace=True)
        (2): Linear(in_features=400, out_features=784, bias=True)
        (3): Sigmoid()
      )
    )

Choose an image

```python
idx = 1010
```

Define the transforms to normalise the image (as done during training)

```python
# Define the transforms to normalise the image
pic_tt = transforms.ToTensor() # Converts a PIL Image or numpy.ndarray (H x W x C) in the range [0, 255] to a torch.FloatTensor of shape (C x H x W) in the range [0.0, 1.0]
pic_n = transforms.Normalize((ds_mean,), (ds_std,))

if torch.__version__ != '1.3.0':
    pic_data = valid_ds.test_data[idx].numpy()
    pic_data = pic_data[:, :, None] # (H x W x C)
    print('pic_data.dtype: {}'.format(pic_data.dtype))
else:
    pic_data = valid_ds.data[idx].numpy()

pic = pic_tt(pic_data)
pic = pic_n(pic)

# print stats
print('pic mean: {}, std: {}'.format(pic.mean(), pic.std()))

# print pic shape
print('\npic shape (C x H x W): {}'.format(pic.shape))
```

    pic mean: 0.0921572670340538, std: 1.0990101099014282

    pic shape (C x H x W): torch.Size([1, 28, 28])

```python
plt.imshow(pic.view(28,28))
```

```python
# check size
print(pic.to(device)[None, :, :, :].shape)
print(pic.to(device).view(1, -1).shape)
```

    torch.Size([1, 1, 28, 28])
    torch.Size([1, 784])

### Encode

```python
device = 'cpu'
mean_encoding, var_encoding = model.encode(pic.to(device).view(1, -1)) # note that we need to reshape the pic to be: bsx28*28

# print encoding result
print('encoded mean: {}'.format(mean_encoding.detach().cpu().numpy()))
print('encoded  var: {}'.format(var_encoding.detach().cpu().numpy()))
```

    encoded mean: [[-0.5592429   1.4444453   1.1667817  -0.18782139  1.0589974  -0.55701214
      -0.07533638  0.1629481   0.64944714  5.653008    0.3813714  -0.9495977
       1.085453   -1.5471504  -3.5718756   1.2370685  -0.7264838   0.53405297
       1.5231957  -1.6848253 ]]
    encoded  var: [[-5.571127  -6.57798   -6.2529154 -5.8869295 -5.8825827 -5.62205
      -5.6434155 -6.084442  -6.771547  -5.3372474 -5.7165003 -6.550778
      -5.552048  -6.1818843 -5.855712  -5.3861585 -6.335955  -6.6406946
      -6.5768113 -6.869463 ]]

### Decode

```python
decoding, _, _ = model.decode((mean_encoding, var_encoding))
```

```python
plt.imshow(decoding.cpu().data.view(28, 28))
```

## fin.

```python

```