# VAE for MNIST clustering and generation

The goal of this notebook is to explore some recent works dealing with variational auto-encoder (VAE).

We will use MNIST dataset and a basic VAE architecture.

```python
import os
import torch
import torch.nn as nn
import torch.nn.functional as F
import torchvision
from torchvision import transforms
from torchvision.utils import save_image

import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

from sklearn.metrics.cluster import normalized_mutual_info_score

def show(img):
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1,2,0)), interpolation='nearest')

def plot_reconstruction(model, n=24):
    x,_ = next(iter(data_loader))
    x = x[:n,:,:,:].to(device)
    try:
        out, _, _, log_p = model(x.view(-1, image_size))
    except:
        out, _, _ = model(x.view(-1, image_size))
    x_concat = torch.cat([x.view(-1, 1, 28, 28), out.view(-1, 1, 28, 28)], dim=3)
    out_grid = torchvision.utils.make_grid(x_concat).cpu().data
    show(out_grid)

def plot_generation(model, n=24):
    with torch.no_grad():
        z = torch.randn(n, z_dim).to(device)
        out = model.decode(z).view(-1, 1, 28, 28)

    out_grid = torchvision.utils.make_grid(out).cpu()
    show(out_grid)

def plot_conditional_generation(model, n=8, fix_number=None):
    with torch.no_grad():
        matrix = np.zeros((n,n_classes))
        matrix[:,0] = 1

        if fix_number is None:
            final = matrix[:]
            for i in range(1,n_classes):
                final = np.vstack((final,np.roll(matrix,i)))
            z = torch.randn(8*n_classes, z_dim).to(device)
            y_onehot = torch.tensor(final).type(torch.FloatTensor).to(device)
            out = model_G.decode(z,y_onehot).view(-1, 1, 28, 28)
        else:
            z = torch.randn(n, z_dim).to(device)
            y_onehot = torch.tensor(np.roll(matrix, fix_number)).type(torch.FloatTensor).to(device)
            out = model_G.decode(z,y_onehot).view(-1, 1, 28, 28)

    out_grid = torchvision.utils.make_grid(out).cpu()
    show(out_grid)
```

```python
# Device configuration
device = torch.device('cuda' if torch.cuda.is_available() else 'cpu')

# Create a directory if not exists
sample_dir = 'samples'
if not os.path.exists(sample_dir):
    os.makedirs(sample_dir)
```

```python
batch_size = 128

data_dir = 'data'
# MNIST dataset
dataset = torchvision.datasets.MNIST(root=data_dir,
                                     train=True,
                                     transform=transforms.ToTensor(),
                                     download=True)

# Data loader
data_loader = torch.utils.data.DataLoader(dataset=dataset,
                                          batch_size=batch_size,
                                          shuffle=True)

test_loader = torch.utils.data.DataLoader(
    torchvision.datasets.MNIST(data_dir, train=False, download=True, transform=transforms.ToTensor()),
    batch_size=10, shuffle=False)
```

    Downloading http://yann.lecun.com/exdb/mnist/train-images-idx3-ubyte.gz to data/MNIST/raw/train-images-idx3-ubyte.gz

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    Extracting data/MNIST/raw/train-images-idx3-ubyte.gz to data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/train-labels-idx1-ubyte.gz to data/MNIST/raw/train-labels-idx1-ubyte.gz

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    Extracting data/MNIST/raw/train-labels-idx1-ubyte.gz to data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-images-idx3-ubyte.gz to data/MNIST/raw/t10k-images-idx3-ubyte.gz

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    Extracting data/MNIST/raw/t10k-images-idx3-ubyte.gz to data/MNIST/raw
    Downloading http://yann.lecun.com/exdb/mnist/t10k-labels-idx1-ubyte.gz to data/MNIST/raw/t10k-labels-idx1-ubyte.gz

Failed to display Jupyter Widget of type HBox.

  If you're reading this message in the Jupyter Notebook or JupyterLab Notebook, it may mean
  that the widgets JavaScript is still loading. If this message persists, it
  likely means that the widgets JavaScript library is either not installed or
  not enabled. See the Jupyter
  Widgets Documentation for setup instructions.

  If you're reading this message in another frontend (for example, a static
  rendering on GitHub or NBViewer),
  it may mean that your frontend doesn't currently support widgets.

    Extracting data/MNIST/raw/t10k-labels-idx1-ubyte.gz to data/MNIST/raw
    Processing...
    Done!

    /Users/megidervishi/anaconda3/lib/python3.6/site-packages/torchvision/datasets/mnist.py:469: UserWarning: The given NumPy array is not writeable, and PyTorch does not support non-writeable tensors. This means you can write to the underlying (supposedly non-writeable) NumPy array using the tensor. You may want to copy the array to protect its data or make it writeable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at  ../torch/csrc/utils/tensor_numpy.cpp:141.)
      return torch.from_numpy(parsed.astype(m[2], copy=False)).view(*s)

# Variational Autoencoders

Consider a latent variable model with a data variable $x\in \mathcal{X}$ and a latent variable $z\in \mathcal{Z}$, $p(z,x) = p(z)p_\theta(x|z)$. Given the data $x_1,\dots, x_n$, we want to train the model by maximizing the marginal log-likelihood:
\begin{eqnarray*}
\mathcal{L} = \mathbf{E}_{p_d(x)}\left[\log p_\theta(x)\right]=\mathbf{E}_{p_d(x)}\left[\log \int_{\mathcal{Z}}p_{\theta}(x|z)p(z)dz\right],
  \end{eqnarray*}
  where $p_d$ denotes the empirical distribution of $X$: $p_d(x) =\frac{1}{n}\sum_{i=1}^n \delta_{x_i}(x)$.

 To avoid the (often) difficult computation of the integral above, the idea behind variational methods is to instead maximize a lower bound to the log-likelihood:
  \begin{eqnarray*}
\mathcal{L} \geq L(p_\theta(x|z),q(z|x)) =\mathbf{E}_{p_d(x)}\left[\mathbf{E}_{q(z|x)}\left[\log p_\theta(x|z)\right]-\mathrm{KL}\left( q(z|x)||p(z)\right)\right].
  \end{eqnarray*}
  Any choice of $q(z|x)$ gives a valid lower bound. Variational autoencoders replace the variational posterior $q(z|x)$ by an inference network $q_{\phi}(z|x)$ that is trained together with $p_{\theta}(x|z)$ to jointly maximize $L(p_\theta,q_\phi)$.

The variational posterior $q_{\phi}(z|x)$ is also called the **encoder** and the generative model $p_{\theta}(x|z)$, the **decoder** or generator.

The first term $\mathbf{E}_{q(z|x)}\left[\log p_\theta(x|z)\right]$ is the negative reconstruction error. Indeed under a gaussian assumption i.e. $p_{\theta}(x|z) = \mathcal{N}(\mu_{\theta}(z), I)$ the term $\log p_\theta(x|z)$ reduces to $\propto \|x-\mu_\theta(z)\|^2$, which is often used in practice. The term $\mathrm{KL}\left( q(z|x)||p(z)\right)$ can be seen as a regularization term, where the variational posterior $q_\phi(z|x)$ should be matched to the prior $p(z)= \mathcal{N}(0, I)$.

Variational Autoencoders were introduced by [Kingma and Welling (2013)](https://arxiv.org/abs/1312.6114), see also [(Doersch, 2016)](https://arxiv.org/abs/1606.05908) for a tutorial.

There are various examples of VAE in PyTorch available [here](https://github.com/pytorch/examples/tree/master/vae?raw=1) or [here](https://github.com/yunjey/pytorch-tutorial/blob/master/tutorials/03-advanced/variational_autoencoder/main.py#L38-L65). The code below is taken from this last source.

```python
# Hyper-parameters
image_size = 784
h_dim = 400
z_dim = 20
num_epochs = 15
learning_rate = 1e-3

# VAE model
class VAE(nn.Module):
    def __init__(self, image_size=784, h_dim=400, z_dim=20):
        super(VAE, self).__init__()
        self.fc1 = nn.Linear(image_size, h_dim)
        self.fc2 = nn.Linear(h_dim, z_dim)
        self.fc3 = nn.Linear(h_dim, z_dim)
        self.fc4 = nn.Linear(z_dim, h_dim)
        self.fc5 = nn.Linear(h_dim, image_size)

    def encode(self, x):
        h = F.relu(self.fc1(x))
        return self.fc2(h), self.fc3(h)

    def reparameterize(self, mu, log_var):
        std = torch.exp(log_var/2)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z):
        h = F.relu(self.fc4(z))
        return torch.sigmoid(self.fc5(h))

    def forward(self, x):
        mu, log_var = self.encode(x)
        z = self.reparameterize(mu, log_var)
        x_reconst = self.decode(z)
        return x_reconst, mu, log_var

model = VAE().to(device)
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)
```

Here for the loss, instead of MSE for the reconstruction loss, we take Binary Cross-Entropy. The code below is still from the PyTorch tutorial (with minor modifications to avoid warnings!).

```python
# Start training
for epoch in range(num_epochs):
    for i, (x, _) in enumerate(data_loader):
        # Forward pass
        x = x.to(device).view(-1, image_size)
        x_reconst, mu, log_var = model(x)

        # Compute reconstruction loss and kl divergence
        # For KL divergence between Gaussians, see Appendix B in VAE paper or (Doersch, 2016):
        # https://arxiv.org/abs/1606.05908
        reconst_loss = F.binary_cross_entropy(x_reconst, x, reduction='sum')
        kl_div = - 0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())

        # Backprop and optimize
        loss = reconst_loss + kl_div
        optimizer.zero_grad()
        loss.backward()
        optimizer.step()

        if (i+1) % 10 == 0:
            print ("Epoch[{}/{}], Step [{}/{}], Reconst Loss: {:.4f}, KL Div: {:.4f}"
                   .format(epoch+1, num_epochs, i+1, len(data_loader), reconst_loss.item()/batch_size, kl_div.item()/batch_size))
```

    Epoch[1/15], Step [10/469], Reconst Loss: 283.4444, KL Div: 30.6976
    Epoch[1/15], Step [20/469], Reconst Loss: 227.9601, KL Div: 8.2261
    Epoch[1/15], Step [30/469], Reconst Loss: 212.8159, KL Div: 9.0650
    Epoch[1/15], Step [40/469], Reconst Loss: 207.7409, KL Div: 5.5895
    Epoch[1/15], Step [50/469], Reconst Loss: 206.4438, KL Div: 5.5463
    Epoch[1/15], Step [60/469], Reconst Loss: 187.1308, KL Div: 8.9796
    Epoch[1/15], Step [70/469], Reconst Loss: 191.9458, KL Div: 7.0730
    Epoch[1/15], Step [80/469], Reconst Loss: 192.5746, KL Div: 7.5109
    Epoch[1/15], Step [90/469], Reconst Loss: 168.9218, KL Div: 10.4938
    Epoch[1/15], Step [100/469], Reconst Loss: 172.0278, KL Div: 11.4910
    Epoch[1/15], Step [110/469], Reconst Loss: 174.3038, KL Div: 11.2750
    Epoch[1/15], Step [120/469], Reconst Loss: 156.0820, KL Div: 12.8584
    Epoch[1/15], Step [130/469], Reconst Loss: 158.1327, KL Div: 14.0804
    Epoch[1/15], Step [140/469], Reconst Loss: 156.8378, KL Div: 13.2619
    Epoch[1/15], Step [150/469], Reconst Loss: 140.6765, KL Div: 14.8911
    Epoch[1/15], Step [160/469], Reconst Loss: 149.6428, KL Div: 12.8916
    Epoch[1/15], Step [170/469], Reconst Loss: 145.0909, KL Div: 14.7677
    Epoch[1/15], Step [180/469], Reconst Loss: 147.8302, KL Div: 14.4155
    Epoch[1/15], Step [190/469], Reconst Loss: 135.2172, KL Div: 15.6616
    Epoch[1/15], Step [200/469], Reconst Loss: 137.6193, KL Div: 14.4592
    Epoch[1/15], Step [210/469], Reconst Loss: 129.6897, KL Div: 15.9617
    Epoch[1/15], Step [220/469], Reconst Loss: 131.3563, KL Div: 16.1370
    Epoch[1/15], Step [230/469], Reconst Loss: 132.5214, KL Div: 17.0562
    Epoch[1/15], Step [240/469], Reconst Loss: 131.8430, KL Div: 16.3170
    Epoch[1/15], Step [250/469], Reconst Loss: 127.4480, KL Div: 17.4930
    Epoch[1/15], Step [260/469], Reconst Loss: 128.4877, KL Div: 16.9097
    Epoch[1/15], Step [270/469], Reconst Loss: 126.9615, KL Div: 17.6919
    Epoch[1/15], Step [280/469], Reconst Loss: 123.1542, KL Div: 17.4141
    Epoch[1/15], Step [290/469], Reconst Loss: 129.2113, KL Div: 17.8016
    Epoch[1/15], Step [300/469], Reconst Loss: 120.7165, KL Div: 17.3990
    Epoch[1/15], Step [310/469], Reconst Loss: 118.3080, KL Div: 18.5025
    Epoch[1/15], Step [320/469], Reconst Loss: 125.9189, KL Div: 18.9179
    Epoch[1/15], Step [330/469], Reconst Loss: 118.6567, KL Div: 18.6880
    Epoch[1/15], Step [340/469], Reconst Loss: 118.3391, KL Div: 19.1130
    Epoch[1/15], Step [350/469], Reconst Loss: 113.1558, KL Div: 19.5981
    Epoch[1/15], Step [360/469], Reconst Loss: 113.9478, KL Div: 19.3660
    Epoch[1/15], Step [370/469], Reconst Loss: 104.4730, KL Div: 19.3657
    Epoch[1/15], Step [380/469], Reconst Loss: 113.6559, KL Div: 19.8323
    Epoch[1/15], Step [390/469], Reconst Loss: 115.7506, KL Div: 20.0228
    Epoch[1/15], Step [400/469], Reconst Loss: 110.9799, KL Div: 20.2173
    Epoch[1/15], Step [410/469], Reconst Loss: 108.9616, KL Div: 20.0049
    Epoch[1/15], Step [420/469], Reconst Loss: 109.1234, KL Div: 20.4970
    Epoch[1/15], Step [430/469], Reconst Loss: 112.2262, KL Div: 21.0618
    Epoch[1/15], Step [440/469], Reconst Loss: 111.9787, KL Div: 21.4799
    Epoch[1/15], Step [450/469], Reconst Loss: 108.2216, KL Div: 20.4694
    Epoch[1/15], Step [460/469], Reconst Loss: 105.4037, KL Div: 21.0004
    Epoch[2/15], Step [10/469], Reconst Loss: 105.9310, KL Div: 21.0563
    Epoch[2/15], Step [20/469], Reconst Loss: 105.1231, KL Div: 20.3092
    Epoch[2/15], Step [30/469], Reconst Loss: 104.1421, KL Div: 20.3578
    Epoch[2/15], Step [40/469], Reconst Loss: 105.7685, KL Div: 22.0663
    Epoch[2/15], Step [50/469], Reconst Loss: 105.2155, KL Div: 20.7031
    Epoch[2/15], Step [60/469], Reconst Loss: 111.4215, KL Div: 21.6333
    Epoch[2/15], Step [70/469], Reconst Loss: 105.5619, KL Div: 22.4659
    Epoch[2/15], Step [80/469], Reconst Loss: 99.4786, KL Div: 20.5324
    Epoch[2/15], Step [90/469], Reconst Loss: 100.7465, KL Div: 21.6921
    Epoch[2/15], Step [100/469], Reconst Loss: 99.4157, KL Div: 21.3379
    Epoch[2/15], Step [110/469], Reconst Loss: 99.0465, KL Div: 22.0048
    Epoch[2/15], Step [120/469], Reconst Loss: 103.7931, KL Div: 22.0202
    Epoch[2/15], Step [130/469], Reconst Loss: 100.8131, KL Div: 21.4609
    Epoch[2/15], Step [140/469], Reconst Loss: 98.1795, KL Div: 22.2518
    Epoch[2/15], Step [150/469], Reconst Loss: 98.1612, KL Div: 22.6991
    Epoch[2/15], Step [160/469], Reconst Loss: 101.9489, KL Div: 23.0266
    Epoch[2/15], Step [170/469], Reconst Loss: 99.5322, KL Div: 22.7491
    Epoch[2/15], Step [180/469], Reconst Loss: 98.7404, KL Div: 21.8232
    Epoch[2/15], Step [190/469], Reconst Loss: 98.0366, KL Div: 22.3969
    Epoch[2/15], Step [200/469], Reconst Loss: 96.0463, KL Div: 21.9491
    Epoch[2/15], Step [210/469], Reconst Loss: 100.6761, KL Div: 23.0246
    Epoch[2/15], Step [220/469], Reconst Loss: 95.7314, KL Div: 23.0936
    Epoch[2/15], Step [230/469], Reconst Loss: 96.0871, KL Div: 24.0227
    Epoch[2/15], Step [240/469], Reconst Loss: 99.4487, KL Div: 22.1608
    Epoch[2/15], Step [250/469], Reconst Loss: 105.8416, KL Div: 22.4669
    Epoch[2/15], Step [260/469], Reconst Loss: 91.6712, KL Div: 23.9247
    Epoch[2/15], Step [270/469], Reconst Loss: 97.3091, KL Div: 22.9843
    Epoch[2/15], Step [280/469], Reconst Loss: 94.8199, KL Div: 23.6393
    Epoch[2/15], Step [290/469], Reconst Loss: 93.1443, KL Div: 21.9787
    Epoch[2/15], Step [300/469], Reconst Loss: 95.7690, KL Div: 22.1131
    Epoch[2/15], Step [310/469], Reconst Loss: 93.1821, KL Div: 22.6409
    Epoch[2/15], Step [320/469], Reconst Loss: 96.1784, KL Div: 23.8287
    Epoch[2/15], Step [330/469], Reconst Loss: 97.9764, KL Div: 22.9079
    Epoch[2/15], Step [340/469], Reconst Loss: 94.9379, KL Div: 24.1052
    Epoch[2/15], Step [350/469], Reconst Loss: 99.3805, KL Div: 22.9294
    Epoch[2/15], Step [360/469], Reconst Loss: 91.7861, KL Div: 23.1244
    Epoch[2/15], Step [370/469], Reconst Loss: 100.4690, KL Div: 23.0284
    Epoch[2/15], Step [380/469], Reconst Loss: 95.5059, KL Div: 23.7523
    Epoch[2/15], Step [390/469], Reconst Loss: 90.9013, KL Div: 23.6550
    Epoch[2/15], Step [400/469], Reconst Loss: 91.0478, KL Div: 23.0732
    Epoch[2/15], Step [410/469], Reconst Loss: 93.3272, KL Div: 23.5283
    Epoch[2/15], Step [420/469], Reconst Loss: 94.3256, KL Div: 22.9782
    Epoch[2/15], Step [430/469], Reconst Loss: 96.4375, KL Div: 23.6157
    Epoch[2/15], Step [440/469], Reconst Loss: 91.7329, KL Div: 23.1831
    Epoch[2/15], Step [450/469], Reconst Loss: 95.5454, KL Div: 23.4689
    Epoch[2/15], Step [460/469], Reconst Loss: 94.1071, KL Div: 23.5192
    Epoch[3/15], Step [10/469], Reconst Loss: 93.9816, KL Div: 23.5487
    Epoch[3/15], Step [20/469], Reconst Loss: 91.1747, KL Div: 23.6346
    Epoch[3/15], Step [30/469], Reconst Loss: 93.5215, KL Div: 23.2735
    Epoch[3/15], Step [40/469], Reconst Loss: 92.2423, KL Div: 23.9977
    Epoch[3/15], Step [50/469], Reconst Loss: 93.5404, KL Div: 23.5699
    Epoch[3/15], Step [60/469], Reconst Loss: 94.0113, KL Div: 25.0209
    Epoch[3/15], Step [70/469], Reconst Loss: 95.3747, KL Div: 23.4426
    Epoch[3/15], Step [80/469], Reconst Loss: 90.6839, KL Div: 23.5498
    Epoch[3/15], Step [90/469], Reconst Loss: 91.9212, KL Div: 24.5197
    Epoch[3/15], Step [100/469], Reconst Loss: 91.3125, KL Div: 24.9629
    Epoch[3/15], Step [110/469], Reconst Loss: 94.0002, KL Div: 23.7098
    Epoch[3/15], Step [120/469], Reconst Loss: 88.2450, KL Div: 24.0887
    Epoch[3/15], Step [130/469], Reconst Loss: 95.6015, KL Div: 23.6309
    Epoch[3/15], Step [140/469], Reconst Loss: 90.8833, KL Div: 24.0844
    Epoch[3/15], Step [150/469], Reconst Loss: 90.8171, KL Div: 23.4383
    Epoch[3/15], Step [160/469], Reconst Loss: 89.9201, KL Div: 24.0236
    Epoch[3/15], Step [170/469], Reconst Loss: 88.7880, KL Div: 23.5757
    Epoch[3/15], Step [180/469], Reconst Loss: 90.2315, KL Div: 24.2829
    Epoch[3/15], Step [190/469], Reconst Loss: 88.4064, KL Div: 23.9626
    Epoch[3/15], Step [200/469], Reconst Loss: 91.2867, KL Div: 23.5976
    Epoch[3/15], Step [210/469], Reconst Loss: 87.5000, KL Div: 24.3524
    Epoch[3/15], Step [220/469], Reconst Loss: 95.5598, KL Div: 24.3092
    Epoch[3/15], Step [230/469], Reconst Loss: 91.6443, KL Div: 24.3416
    Epoch[3/15], Step [240/469], Reconst Loss: 90.0155, KL Div: 24.0487
    Epoch[3/15], Step [250/469], Reconst Loss: 89.1162, KL Div: 24.4466
    Epoch[3/15], Step [260/469], Reconst Loss: 88.1867, KL Div: 24.6152
    Epoch[3/15], Step [270/469], Reconst Loss: 90.1912, KL Div: 23.2492
    Epoch[3/15], Step [280/469], Reconst Loss: 90.5498, KL Div: 24.5827
    Epoch[3/15], Step [290/469], Reconst Loss: 92.3270, KL Div: 24.4327
    Epoch[3/15], Step [300/469], Reconst Loss: 92.1999, KL Div: 25.0825
    Epoch[3/15], Step [310/469], Reconst Loss: 89.2087, KL Div: 24.4912
    Epoch[3/15], Step [320/469], Reconst Loss: 89.1233, KL Div: 24.5143
    Epoch[3/15], Step [330/469], Reconst Loss: 85.4465, KL Div: 24.3192
    Epoch[3/15], Step [340/469], Reconst Loss: 92.9438, KL Div: 24.4922
    Epoch[3/15], Step [350/469], Reconst Loss: 91.7630, KL Div: 24.3071
    Epoch[3/15], Step [360/469], Reconst Loss: 89.6768, KL Div: 24.7731
    Epoch[3/15], Step [370/469], Reconst Loss: 90.3116, KL Div: 24.9383
    Epoch[3/15], Step [380/469], Reconst Loss: 88.5050, KL Div: 24.6078
    Epoch[3/15], Step [390/469], Reconst Loss: 87.3001, KL Div: 24.2962
    Epoch[3/15], Step [400/469], Reconst Loss: 90.1058, KL Div: 24.6141
    Epoch[3/15], Step [410/469], Reconst Loss: 86.3154, KL Div: 23.5284
    Epoch[3/15], Step [420/469], Reconst Loss: 91.4078, KL Div: 24.1764
    Epoch[3/15], Step [430/469], Reconst Loss: 88.7384, KL Div: 23.4781
    Epoch[3/15], Step [440/469], Reconst Loss: 86.8329, KL Div: 24.4642
    Epoch[3/15], Step [450/469], Reconst Loss: 87.2882, KL Div: 23.9975
    Epoch[3/15], Step [460/469], Reconst Loss: 88.5201, KL Div: 23.8401
    Epoch[4/15], Step [10/469], Reconst Loss: 87.8535, KL Div: 24.7104
    Epoch[4/15], Step [20/469], Reconst Loss: 85.7178, KL Div: 24.7419
    Epoch[4/15], Step [30/469], Reconst Loss: 88.0464, KL Div: 24.3962
    Epoch[4/15], Step [40/469], Reconst Loss: 91.4711, KL Div: 24.9714
    Epoch[4/15], Step [50/469], Reconst Loss: 88.9789, KL Div: 24.3557
    Epoch[4/15], Step [60/469], Reconst Loss: 90.7421, KL Div: 24.8282
    Epoch[4/15], Step [70/469], Reconst Loss: 89.4918, KL Div: 24.2088
    Epoch[4/15], Step [80/469], Reconst Loss: 86.4651, KL Div: 25.5775
    Epoch[4/15], Step [90/469], Reconst Loss: 88.2781, KL Div: 24.7471
    Epoch[4/15], Step [100/469], Reconst Loss: 85.5209, KL Div: 23.9165
    Epoch[4/15], Step [110/469], Reconst Loss: 88.1973, KL Div: 24.7870
    Epoch[4/15], Step [120/469], Reconst Loss: 88.9421, KL Div: 24.6597
    Epoch[4/15], Step [130/469], Reconst Loss: 87.7594, KL Div: 24.8715
    Epoch[4/15], Step [140/469], Reconst Loss: 86.5109, KL Div: 25.1788
    Epoch[4/15], Step [150/469], Reconst Loss: 87.6462, KL Div: 24.5078
    Epoch[4/15], Step [160/469], Reconst Loss: 87.5189, KL Div: 24.8149
    Epoch[4/15], Step [170/469], Reconst Loss: 87.9809, KL Div: 24.0752
    Epoch[4/15], Step [180/469], Reconst Loss: 86.7625, KL Div: 24.0332
    Epoch[4/15], Step [190/469], Reconst Loss: 85.7749, KL Div: 24.4013
    Epoch[4/15], Step [200/469], Reconst Loss: 87.2622, KL Div: 24.5264
    Epoch[4/15], Step [210/469], Reconst Loss: 85.3630, KL Div: 25.1704
    Epoch[4/15], Step [220/469], Reconst Loss: 88.9856, KL Div: 25.1470
    Epoch[4/15], Step [230/469], Reconst Loss: 88.2323, KL Div: 25.6389
    Epoch[4/15], Step [240/469], Reconst Loss: 85.9350, KL Div: 23.8111
    Epoch[4/15], Step [250/469], Reconst Loss: 87.3459, KL Div: 25.8176
    Epoch[4/15], Step [260/469], Reconst Loss: 89.8858, KL Div: 24.6134
    Epoch[4/15], Step [270/469], Reconst Loss: 86.1639, KL Div: 24.4116
    Epoch[4/15], Step [280/469], Reconst Loss: 90.9889, KL Div: 24.8326
    Epoch[4/15], Step [290/469], Reconst Loss: 85.2899, KL Div: 24.2143
    Epoch[4/15], Step [300/469], Reconst Loss: 85.6406, KL Div: 25.2123
    Epoch[4/15], Step [310/469], Reconst Loss: 87.8218, KL Div: 25.1302
    Epoch[4/15], Step [320/469], Reconst Loss: 85.4952, KL Div: 25.3915
    Epoch[4/15], Step [330/469], Reconst Loss: 84.2406, KL Div: 24.2393
    Epoch[4/15], Step [340/469], Reconst Loss: 84.4898, KL Div: 24.2280
    Epoch[4/15], Step [350/469], Reconst Loss: 90.2897, KL Div: 25.1727
    Epoch[4/15], Step [360/469], Reconst Loss: 86.3158, KL Div: 24.9330
    Epoch[4/15], Step [370/469], Reconst Loss: 84.1187, KL Div: 25.0259
    Epoch[4/15], Step [380/469], Reconst Loss: 87.3169, KL Div: 25.0864
    Epoch[4/15], Step [390/469], Reconst Loss: 90.9448, KL Div: 25.2770
    Epoch[4/15], Step [400/469], Reconst Loss: 87.7719, KL Div: 24.8677
    Epoch[4/15], Step [410/469], Reconst Loss: 88.9668, KL Div: 24.9036
    Epoch[4/15], Step [420/469], Reconst Loss: 88.1392, KL Div: 24.8606
    Epoch[4/15], Step [430/469], Reconst Loss: 86.3506, KL Div: 24.9003
    Epoch[4/15], Step [440/469], Reconst Loss: 80.6923, KL Div: 23.8933
    Epoch[4/15], Step [450/469], Reconst Loss: 84.5354, KL Div: 24.6591
    Epoch[4/15], Step [460/469], Reconst Loss: 87.5532, KL Div: 25.1095
    Epoch[5/15], Step [10/469], Reconst Loss: 88.2910, KL Div: 25.2066
    Epoch[5/15], Step [20/469], Reconst Loss: 87.2305, KL Div: 25.2307
    Epoch[5/15], Step [30/469], Reconst Loss: 85.2947, KL Div: 25.1637
    Epoch[5/15], Step [40/469], Reconst Loss: 87.4641, KL Div: 24.8535
    Epoch[5/15], Step [50/469], Reconst Loss: 83.2807, KL Div: 24.8668
    Epoch[5/15], Step [60/469], Reconst Loss: 87.8426, KL Div: 25.6473
    Epoch[5/15], Step [70/469], Reconst Loss: 86.0558, KL Div: 24.8693
    Epoch[5/15], Step [80/469], Reconst Loss: 80.3174, KL Div: 23.9642
    Epoch[5/15], Step [90/469], Reconst Loss: 86.6175, KL Div: 25.8233
    Epoch[5/15], Step [100/469], Reconst Loss: 85.5950, KL Div: 23.9763
    Epoch[5/15], Step [110/469], Reconst Loss: 84.4369, KL Div: 24.9535
    Epoch[5/15], Step [120/469], Reconst Loss: 82.4522, KL Div: 25.0258
    Epoch[5/15], Step [130/469], Reconst Loss: 86.4585, KL Div: 25.1122
    Epoch[5/15], Step [140/469], Reconst Loss: 82.5199, KL Div: 24.4801
    Epoch[5/15], Step [150/469], Reconst Loss: 83.3701, KL Div: 25.2063
    Epoch[5/15], Step [160/469], Reconst Loss: 86.6209, KL Div: 24.8109
    Epoch[5/15], Step [170/469], Reconst Loss: 87.7937, KL Div: 25.2079
    Epoch[5/15], Step [180/469], Reconst Loss: 85.6239, KL Div: 25.7219
    Epoch[5/15], Step [190/469], Reconst Loss: 84.5028, KL Div: 24.3708
    Epoch[5/15], Step [200/469], Reconst Loss: 85.8104, KL Div: 25.2320
    Epoch[5/15], Step [210/469], Reconst Loss: 85.8518, KL Div: 25.2819
    Epoch[5/15], Step [220/469], Reconst Loss: 87.0708, KL Div: 24.6563
    Epoch[5/15], Step [230/469], Reconst Loss: 88.9952, KL Div: 25.0600
    Epoch[5/15], Step [240/469], Reconst Loss: 84.9305, KL Div: 25.0094
    Epoch[5/15], Step [250/469], Reconst Loss: 88.7408, KL Div: 25.1974
    Epoch[5/15], Step [260/469], Reconst Loss: 84.1129, KL Div: 24.6099
    Epoch[5/15], Step [270/469], Reconst Loss: 84.8932, KL Div: 25.8207
    Epoch[5/15], Step [280/469], Reconst Loss: 85.0033, KL Div: 24.1807
    Epoch[5/15], Step [290/469], Reconst Loss: 84.9218, KL Div: 25.0973
    Epoch[5/15], Step [300/469], Reconst Loss: 82.4030, KL Div: 25.0214
    Epoch[5/15], Step [310/469], Reconst Loss: 88.8630, KL Div: 25.3028
    Epoch[5/15], Step [320/469], Reconst Loss: 85.5015, KL Div: 26.4271
    Epoch[5/15], Step [330/469], Reconst Loss: 79.5160, KL Div: 24.5511
    Epoch[5/15], Step [340/469], Reconst Loss: 87.7504, KL Div: 25.9175
    Epoch[5/15], Step [350/469], Reconst Loss: 88.5888, KL Div: 24.2734
    Epoch[5/15], Step [360/469], Reconst Loss: 85.0149, KL Div: 24.5350
    Epoch[5/15], Step [370/469], Reconst Loss: 81.9515, KL Div: 24.6611
    Epoch[5/15], Step [380/469], Reconst Loss: 83.4394, KL Div: 24.4014
    Epoch[5/15], Step [390/469], Reconst Loss: 82.4500, KL Div: 24.8810
    Epoch[5/15], Step [400/469], Reconst Loss: 81.7111, KL Div: 25.0739
    Epoch[5/15], Step [410/469], Reconst Loss: 85.3557, KL Div: 25.5719
    Epoch[5/15], Step [420/469], Reconst Loss: 84.3453, KL Div: 25.4795
    Epoch[5/15], Step [430/469], Reconst Loss: 85.2970, KL Div: 24.1444
    Epoch[5/15], Step [440/469], Reconst Loss: 84.8419, KL Div: 25.2844
    Epoch[5/15], Step [450/469], Reconst Loss: 84.1942, KL Div: 25.3932
    Epoch[5/15], Step [460/469], Reconst Loss: 84.6566, KL Div: 24.1237
    Epoch[6/15], Step [10/469], Reconst Loss: 86.5202, KL Div: 24.7561
    Epoch[6/15], Step [20/469], Reconst Loss: 83.9635, KL Div: 25.6936
    Epoch[6/15], Step [30/469], Reconst Loss: 85.0224, KL Div: 25.4233
    Epoch[6/15], Step [40/469], Reconst Loss: 80.4348, KL Div: 25.1287
    Epoch[6/15], Step [50/469], Reconst Loss: 85.9459, KL Div: 24.0085
    Epoch[6/15], Step [60/469], Reconst Loss: 82.2489, KL Div: 25.0038
    Epoch[6/15], Step [70/469], Reconst Loss: 84.6643, KL Div: 24.9898
    Epoch[6/15], Step [80/469], Reconst Loss: 84.6424, KL Div: 25.5093
    Epoch[6/15], Step [90/469], Reconst Loss: 82.0240, KL Div: 25.5950
    Epoch[6/15], Step [100/469], Reconst Loss: 86.1515, KL Div: 24.5458
    Epoch[6/15], Step [110/469], Reconst Loss: 85.8601, KL Div: 25.8252
    Epoch[6/15], Step [120/469], Reconst Loss: 81.4193, KL Div: 24.8079
    Epoch[6/15], Step [130/469], Reconst Loss: 81.0166, KL Div: 24.7519
    Epoch[6/15], Step [140/469], Reconst Loss: 84.2782, KL Div: 25.2445
    Epoch[6/15], Step [150/469], Reconst Loss: 81.9854, KL Div: 24.5616
    Epoch[6/15], Step [160/469], Reconst Loss: 83.6724, KL Div: 25.3713
    Epoch[6/15], Step [170/469], Reconst Loss: 85.4816, KL Div: 25.2363
    Epoch[6/15], Step [180/469], Reconst Loss: 80.2557, KL Div: 24.4230
    Epoch[6/15], Step [190/469], Reconst Loss: 85.7166, KL Div: 24.2247
    Epoch[6/15], Step [200/469], Reconst Loss: 82.0852, KL Div: 25.3454
    Epoch[6/15], Step [210/469], Reconst Loss: 83.4794, KL Div: 24.0721
    Epoch[6/15], Step [220/469], Reconst Loss: 84.8882, KL Div: 24.9672
    Epoch[6/15], Step [230/469], Reconst Loss: 86.1325, KL Div: 24.9857
    Epoch[6/15], Step [240/469], Reconst Loss: 85.4666, KL Div: 25.0496
    Epoch[6/15], Step [250/469], Reconst Loss: 84.5038, KL Div: 24.8124
    Epoch[6/15], Step [260/469], Reconst Loss: 85.6003, KL Div: 24.9404
    Epoch[6/15], Step [270/469], Reconst Loss: 79.3931, KL Div: 25.1200
    Epoch[6/15], Step [280/469], Reconst Loss: 83.3944, KL Div: 25.6857
    Epoch[6/15], Step [290/469], Reconst Loss: 85.4072, KL Div: 25.2534
    Epoch[6/15], Step [300/469], Reconst Loss: 83.6718, KL Div: 25.6995
    Epoch[6/15], Step [310/469], Reconst Loss: 82.8890, KL Div: 24.4419
    Epoch[6/15], Step [320/469], Reconst Loss: 82.2516, KL Div: 25.1420
    Epoch[6/15], Step [330/469], Reconst Loss: 79.6404, KL Div: 24.1768
    Epoch[6/15], Step [340/469], Reconst Loss: 85.2030, KL Div: 24.6116
    Epoch[6/15], Step [350/469], Reconst Loss: 83.8459, KL Div: 24.9919
    Epoch[6/15], Step [360/469], Reconst Loss: 83.1405, KL Div: 24.9290
    Epoch[6/15], Step [370/469], Reconst Loss: 82.0676, KL Div: 24.6917
    Epoch[6/15], Step [380/469], Reconst Loss: 79.7863, KL Div: 25.0822
    Epoch[6/15], Step [390/469], Reconst Loss: 80.8166, KL Div: 24.9663
    Epoch[6/15], Step [400/469], Reconst Loss: 81.3594, KL Div: 25.9871
    Epoch[6/15], Step [410/469], Reconst Loss: 85.1776, KL Div: 24.9474
    Epoch[6/15], Step [420/469], Reconst Loss: 84.0786, KL Div: 25.7846
    Epoch[6/15], Step [430/469], Reconst Loss: 86.5337, KL Div: 24.1882
    Epoch[6/15], Step [440/469], Reconst Loss: 82.8182, KL Div: 25.9264
    Epoch[6/15], Step [450/469], Reconst Loss: 82.8128, KL Div: 24.7601
    Epoch[6/15], Step [460/469], Reconst Loss: 77.7635, KL Div: 25.1753
    Epoch[7/15], Step [10/469], Reconst Loss: 80.6468, KL Div: 24.9600
    Epoch[7/15], Step [20/469], Reconst Loss: 83.6550, KL Div: 25.5652
    Epoch[7/15], Step [30/469], Reconst Loss: 81.5574, KL Div: 24.5524
    Epoch[7/15], Step [40/469], Reconst Loss: 82.6108, KL Div: 25.7569
    Epoch[7/15], Step [50/469], Reconst Loss: 82.7740, KL Div: 25.4590
    Epoch[7/15], Step [60/469], Reconst Loss: 83.5893, KL Div: 24.5068
    Epoch[7/15], Step [70/469], Reconst Loss: 84.8288, KL Div: 25.1250
    Epoch[7/15], Step [80/469], Reconst Loss: 79.3136, KL Div: 24.9009
    Epoch[7/15], Step [90/469], Reconst Loss: 80.4136, KL Div: 25.4357
    Epoch[7/15], Step [100/469], Reconst Loss: 83.0263, KL Div: 25.0121
    Epoch[7/15], Step [110/469], Reconst Loss: 84.6097, KL Div: 24.7008
    Epoch[7/15], Step [120/469], Reconst Loss: 83.6644, KL Div: 24.9952
    Epoch[7/15], Step [130/469], Reconst Loss: 83.6823, KL Div: 24.1137
    Epoch[7/15], Step [140/469], Reconst Loss: 82.4178, KL Div: 25.1361
    Epoch[7/15], Step [150/469], Reconst Loss: 81.0761, KL Div: 25.1334
    Epoch[7/15], Step [160/469], Reconst Loss: 82.7316, KL Div: 25.0230
    Epoch[7/15], Step [170/469], Reconst Loss: 83.4943, KL Div: 25.2700
    Epoch[7/15], Step [180/469], Reconst Loss: 81.9597, KL Div: 25.4583
    Epoch[7/15], Step [190/469], Reconst Loss: 80.7743, KL Div: 25.0830
    Epoch[7/15], Step [200/469], Reconst Loss: 78.2664, KL Div: 24.6211
    Epoch[7/15], Step [210/469], Reconst Loss: 84.3199, KL Div: 25.3144
    Epoch[7/15], Step [220/469], Reconst Loss: 84.4103, KL Div: 25.0889
    Epoch[7/15], Step [230/469], Reconst Loss: 80.2304, KL Div: 25.8412
    Epoch[7/15], Step [240/469], Reconst Loss: 79.6539, KL Div: 24.4213
    Epoch[7/15], Step [250/469], Reconst Loss: 81.1095, KL Div: 25.0268
    Epoch[7/15], Step [260/469], Reconst Loss: 80.9673, KL Div: 25.0405
    Epoch[7/15], Step [270/469], Reconst Loss: 84.8583, KL Div: 25.5107
    Epoch[7/15], Step [280/469], Reconst Loss: 81.6953, KL Div: 24.6200
    Epoch[7/15], Step [290/469], Reconst Loss: 84.8295, KL Div: 25.5083
    Epoch[7/15], Step [300/469], Reconst Loss: 81.8489, KL Div: 24.8037
    Epoch[7/15], Step [310/469], Reconst Loss: 84.7073, KL Div: 25.8787
    Epoch[7/15], Step [320/469], Reconst Loss: 83.7603, KL Div: 25.2930
    Epoch[7/15], Step [330/469], Reconst Loss: 82.8853, KL Div: 24.8227
    Epoch[7/15], Step [340/469], Reconst Loss: 77.9564, KL Div: 24.7301
    Epoch[7/15], Step [350/469], Reconst Loss: 81.5303, KL Div: 25.6097
    Epoch[7/15], Step [360/469], Reconst Loss: 85.1276, KL Div: 25.6996
    Epoch[7/15], Step [370/469], Reconst Loss: 85.7714, KL Div: 25.3135
    Epoch[7/15], Step [380/469], Reconst Loss: 81.4562, KL Div: 25.1178
    Epoch[7/15], Step [390/469], Reconst Loss: 82.4593, KL Div: 25.2926
    Epoch[7/15], Step [400/469], Reconst Loss: 84.4015, KL Div: 25.3417
    Epoch[7/15], Step [410/469], Reconst Loss: 83.9317, KL Div: 25.5044
    Epoch[7/15], Step [420/469], Reconst Loss: 84.4789, KL Div: 25.1679
    Epoch[7/15], Step [430/469], Reconst Loss: 76.5526, KL Div: 25.5680
    Epoch[7/15], Step [440/469], Reconst Loss: 83.6662, KL Div: 25.5697
    Epoch[7/15], Step [450/469], Reconst Loss: 80.8138, KL Div: 24.9163
    Epoch[7/15], Step [460/469], Reconst Loss: 86.8420, KL Div: 25.6811
    Epoch[8/15], Step [10/469], Reconst Loss: 83.2994, KL Div: 25.1344
    Epoch[8/15], Step [20/469], Reconst Loss: 84.2518, KL Div: 24.9367
    Epoch[8/15], Step [30/469], Reconst Loss: 80.1642, KL Div: 24.8459
    Epoch[8/15], Step [40/469], Reconst Loss: 81.3635, KL Div: 25.5214
    Epoch[8/15], Step [50/469], Reconst Loss: 86.5173, KL Div: 25.4071
    Epoch[8/15], Step [60/469], Reconst Loss: 75.9721, KL Div: 25.3317
    Epoch[8/15], Step [70/469], Reconst Loss: 80.5894, KL Div: 25.0090
    Epoch[8/15], Step [80/469], Reconst Loss: 83.1256, KL Div: 25.6666
    Epoch[8/15], Step [90/469], Reconst Loss: 84.4044, KL Div: 25.9635
    Epoch[8/15], Step [100/469], Reconst Loss: 80.5516, KL Div: 24.9887
    Epoch[8/15], Step [110/469], Reconst Loss: 81.3484, KL Div: 24.6945
    Epoch[8/15], Step [120/469], Reconst Loss: 84.6154, KL Div: 25.5398
    Epoch[8/15], Step [130/469], Reconst Loss: 80.8338, KL Div: 25.2936
    Epoch[8/15], Step [140/469], Reconst Loss: 83.6054, KL Div: 25.4787
    Epoch[8/15], Step [150/469], Reconst Loss: 80.6582, KL Div: 24.7066
    Epoch[8/15], Step [160/469], Reconst Loss: 80.9495, KL Div: 25.1322
    Epoch[8/15], Step [170/469], Reconst Loss: 81.8409, KL Div: 24.9814
    Epoch[8/15], Step [180/469], Reconst Loss: 82.4147, KL Div: 25.5714
    Epoch[8/15], Step [190/469], Reconst Loss: 80.8678, KL Div: 25.0309
    Epoch[8/15], Step [200/469], Reconst Loss: 82.3757, KL Div: 25.3534
    Epoch[8/15], Step [210/469], Reconst Loss: 81.7085, KL Div: 25.2561
    Epoch[8/15], Step [220/469], Reconst Loss: 81.3324, KL Div: 25.4627
    Epoch[8/15], Step [230/469], Reconst Loss: 82.4192, KL Div: 26.1692
    Epoch[8/15], Step [240/469], Reconst Loss: 83.1952, KL Div: 26.2652
    Epoch[8/15], Step [250/469], Reconst Loss: 84.9444, KL Div: 25.2237
    Epoch[8/15], Step [260/469], Reconst Loss: 80.2504, KL Div: 25.9367
    Epoch[8/15], Step [270/469], Reconst Loss: 82.3273, KL Div: 24.8051
    Epoch[8/15], Step [280/469], Reconst Loss: 80.0720, KL Div: 25.4103
    Epoch[8/15], Step [290/469], Reconst Loss: 82.1304, KL Div: 26.0242
    Epoch[8/15], Step [300/469], Reconst Loss: 83.4693, KL Div: 25.2064
    Epoch[8/15], Step [310/469], Reconst Loss: 79.2998, KL Div: 25.3382
    Epoch[8/15], Step [320/469], Reconst Loss: 79.8693, KL Div: 24.2435
    Epoch[8/15], Step [330/469], Reconst Loss: 78.4780, KL Div: 25.6689
    Epoch[8/15], Step [340/469], Reconst Loss: 83.7385, KL Div: 25.0886
    Epoch[8/15], Step [350/469], Reconst Loss: 78.7697, KL Div: 24.7184
    Epoch[8/15], Step [360/469], Reconst Loss: 80.6483, KL Div: 24.4894
    Epoch[8/15], Step [370/469], Reconst Loss: 82.5044, KL Div: 25.5970
    Epoch[8/15], Step [380/469], Reconst Loss: 81.1190, KL Div: 25.6516
    Epoch[8/15], Step [390/469], Reconst Loss: 83.2109, KL Div: 25.1419
    Epoch[8/15], Step [400/469], Reconst Loss: 78.7704, KL Div: 25.2869
    Epoch[8/15], Step [410/469], Reconst Loss: 79.5177, KL Div: 25.2170
    Epoch[8/15], Step [420/469], Reconst Loss: 81.4032, KL Div: 24.2997
    Epoch[8/15], Step [430/469], Reconst Loss: 81.1955, KL Div: 24.7273
    Epoch[8/15], Step [440/469], Reconst Loss: 77.9980, KL Div: 24.9363
    Epoch[8/15], Step [450/469], Reconst Loss: 80.9998, KL Div: 24.9318
    Epoch[8/15], Step [460/469], Reconst Loss: 82.1973, KL Div: 24.5933
    Epoch[9/15], Step [10/469], Reconst Loss: 79.0835, KL Div: 24.9193
    Epoch[9/15], Step [20/469], Reconst Loss: 80.8568, KL Div: 25.3240
    Epoch[9/15], Step [30/469], Reconst Loss: 84.1270, KL Div: 25.5732
    Epoch[9/15], Step [40/469], Reconst Loss: 82.5442, KL Div: 25.0875
    Epoch[9/15], Step [50/469], Reconst Loss: 87.6259, KL Div: 25.5161
    Epoch[9/15], Step [60/469], Reconst Loss: 78.7696, KL Div: 24.7656
    Epoch[9/15], Step [70/469], Reconst Loss: 82.7669, KL Div: 26.9254
    Epoch[9/15], Step [80/469], Reconst Loss: 81.0623, KL Div: 24.7591
    Epoch[9/15], Step [90/469], Reconst Loss: 77.6811, KL Div: 24.1094
    Epoch[9/15], Step [100/469], Reconst Loss: 84.9075, KL Div: 25.9289
    Epoch[9/15], Step [110/469], Reconst Loss: 78.6138, KL Div: 25.1645
    Epoch[9/15], Step [120/469], Reconst Loss: 81.9030, KL Div: 25.2650
    Epoch[9/15], Step [130/469], Reconst Loss: 80.6560, KL Div: 25.1632
    Epoch[9/15], Step [140/469], Reconst Loss: 81.5721, KL Div: 25.0648
    Epoch[9/15], Step [150/469], Reconst Loss: 79.7174, KL Div: 25.3214
    Epoch[9/15], Step [160/469], Reconst Loss: 77.8527, KL Div: 24.6552
    Epoch[9/15], Step [170/469], Reconst Loss: 77.9473, KL Div: 25.3427
    Epoch[9/15], Step [180/469], Reconst Loss: 81.3381, KL Div: 25.0449
    Epoch[9/15], Step [190/469], Reconst Loss: 81.1151, KL Div: 24.9550
    Epoch[9/15], Step [200/469], Reconst Loss: 81.1918, KL Div: 25.0823
    Epoch[9/15], Step [210/469], Reconst Loss: 83.2743, KL Div: 24.6867
    Epoch[9/15], Step [220/469], Reconst Loss: 83.6120, KL Div: 25.9657
    Epoch[9/15], Step [230/469], Reconst Loss: 84.3184, KL Div: 24.8615
    Epoch[9/15], Step [240/469], Reconst Loss: 81.9734, KL Div: 26.2982
    Epoch[9/15], Step [250/469], Reconst Loss: 83.1803, KL Div: 25.0265
    Epoch[9/15], Step [260/469], Reconst Loss: 81.6270, KL Div: 25.6505
    Epoch[9/15], Step [270/469], Reconst Loss: 79.1631, KL Div: 25.3920
    Epoch[9/15], Step [280/469], Reconst Loss: 81.6192, KL Div: 25.0086
    Epoch[9/15], Step [290/469], Reconst Loss: 80.8764, KL Div: 25.6099
    Epoch[9/15], Step [300/469], Reconst Loss: 81.3799, KL Div: 25.2033
    Epoch[9/15], Step [310/469], Reconst Loss: 78.5640, KL Div: 25.3889
    Epoch[9/15], Step [320/469], Reconst Loss: 84.0987, KL Div: 25.7778
    Epoch[9/15], Step [330/469], Reconst Loss: 83.1055, KL Div: 25.2823
    Epoch[9/15], Step [340/469], Reconst Loss: 79.5353, KL Div: 25.0711
    Epoch[9/15], Step [350/469], Reconst Loss: 81.8427, KL Div: 25.9122
    Epoch[9/15], Step [360/469], Reconst Loss: 79.8661, KL Div: 25.8983
    Epoch[9/15], Step [370/469], Reconst Loss: 85.2796, KL Div: 26.6296
    Epoch[9/15], Step [380/469], Reconst Loss: 81.2579, KL Div: 25.2337
    Epoch[9/15], Step [390/469], Reconst Loss: 78.0883, KL Div: 24.3847
    Epoch[9/15], Step [400/469], Reconst Loss: 83.9071, KL Div: 25.9286
    Epoch[9/15], Step [410/469], Reconst Loss: 81.1363, KL Div: 25.0857
    Epoch[9/15], Step [420/469], Reconst Loss: 83.4873, KL Div: 25.9355
    Epoch[9/15], Step [430/469], Reconst Loss: 74.5237, KL Div: 24.3632
    Epoch[9/15], Step [440/469], Reconst Loss: 80.8283, KL Div: 25.5265
    Epoch[9/15], Step [450/469], Reconst Loss: 79.2071, KL Div: 24.7768
    Epoch[9/15], Step [460/469], Reconst Loss: 82.7189, KL Div: 24.8872
    Epoch[10/15], Step [10/469], Reconst Loss: 82.1753, KL Div: 25.4847
    Epoch[10/15], Step [20/469], Reconst Loss: 81.4575, KL Div: 24.6253
    Epoch[10/15], Step [30/469], Reconst Loss: 81.0452, KL Div: 26.3479
    Epoch[10/15], Step [40/469], Reconst Loss: 82.1087, KL Div: 25.3971
    Epoch[10/15], Step [50/469], Reconst Loss: 79.4922, KL Div: 25.2366
    Epoch[10/15], Step [60/469], Reconst Loss: 81.4662, KL Div: 25.8512
    Epoch[10/15], Step [70/469], Reconst Loss: 78.3633, KL Div: 24.0209
    Epoch[10/15], Step [80/469], Reconst Loss: 78.9551, KL Div: 25.2981
    Epoch[10/15], Step [90/469], Reconst Loss: 83.7023, KL Div: 25.9187
    Epoch[10/15], Step [100/469], Reconst Loss: 81.7896, KL Div: 25.4817
    Epoch[10/15], Step [110/469], Reconst Loss: 79.9857, KL Div: 24.9807
    Epoch[10/15], Step [120/469], Reconst Loss: 79.5856, KL Div: 24.9535
    Epoch[10/15], Step [130/469], Reconst Loss: 82.2993, KL Div: 25.7959
    Epoch[10/15], Step [140/469], Reconst Loss: 78.8660, KL Div: 24.8448
    Epoch[10/15], Step [150/469], Reconst Loss: 82.8781, KL Div: 26.4704
    Epoch[10/15], Step [160/469], Reconst Loss: 82.9250, KL Div: 25.2591
    Epoch[10/15], Step [170/469], Reconst Loss: 76.9899, KL Div: 25.4454
    Epoch[10/15], Step [180/469], Reconst Loss: 83.2137, KL Div: 24.6744
    Epoch[10/15], Step [190/469], Reconst Loss: 81.2325, KL Div: 25.1663
    Epoch[10/15], Step [200/469], Reconst Loss: 78.5267, KL Div: 25.4523
    Epoch[10/15], Step [210/469], Reconst Loss: 81.6571, KL Div: 26.0793
    Epoch[10/15], Step [220/469], Reconst Loss: 80.1339, KL Div: 24.4524
    Epoch[10/15], Step [230/469], Reconst Loss: 78.4396, KL Div: 25.6655
    Epoch[10/15], Step [240/469], Reconst Loss: 83.6883, KL Div: 25.4438
    Epoch[10/15], Step [250/469], Reconst Loss: 80.1721, KL Div: 25.9917
    Epoch[10/15], Step [260/469], Reconst Loss: 78.5464, KL Div: 25.0210
    Epoch[10/15], Step [270/469], Reconst Loss: 77.7731, KL Div: 24.5727
    Epoch[10/15], Step [280/469], Reconst Loss: 79.3820, KL Div: 26.0662
    Epoch[10/15], Step [290/469], Reconst Loss: 79.3194, KL Div: 24.4346
    Epoch[10/15], Step [300/469], Reconst Loss: 81.1566, KL Div: 24.4988
    Epoch[10/15], Step [310/469], Reconst Loss: 79.9159, KL Div: 26.1577
    Epoch[10/15], Step [320/469], Reconst Loss: 82.4326, KL Div: 24.7820
    Epoch[10/15], Step [330/469], Reconst Loss: 80.9608, KL Div: 25.8137
    Epoch[10/15], Step [340/469], Reconst Loss: 84.7856, KL Div: 25.5698
    Epoch[10/15], Step [350/469], Reconst Loss: 81.6917, KL Div: 25.0993
    Epoch[10/15], Step [360/469], Reconst Loss: 82.9423, KL Div: 26.2104
    Epoch[10/15], Step [370/469], Reconst Loss: 80.3977, KL Div: 24.8040
    Epoch[10/15], Step [380/469], Reconst Loss: 82.7262, KL Div: 26.1026
    Epoch[10/15], Step [390/469], Reconst Loss: 77.6057, KL Div: 24.5409
    Epoch[10/15], Step [400/469], Reconst Loss: 79.7156, KL Div: 25.4468
    Epoch[10/15], Step [410/469], Reconst Loss: 77.0931, KL Div: 24.9702
    Epoch[10/15], Step [420/469], Reconst Loss: 82.6472, KL Div: 25.5653
    Epoch[10/15], Step [430/469], Reconst Loss: 80.3614, KL Div: 26.0397
    Epoch[10/15], Step [440/469], Reconst Loss: 81.7831, KL Div: 24.2268
    Epoch[10/15], Step [450/469], Reconst Loss: 81.9273, KL Div: 26.4486
    Epoch[10/15], Step [460/469], Reconst Loss: 79.0945, KL Div: 25.1865
    Epoch[11/15], Step [10/469], Reconst Loss: 81.3269, KL Div: 25.3449
    Epoch[11/15], Step [20/469], Reconst Loss: 82.4200, KL Div: 25.5905
    Epoch[11/15], Step [30/469], Reconst Loss: 80.2343, KL Div: 25.5021
    Epoch[11/15], Step [40/469], Reconst Loss: 81.9987, KL Div: 24.9979
    Epoch[11/15], Step [50/469], Reconst Loss: 81.3767, KL Div: 26.4194
    Epoch[11/15], Step [60/469], Reconst Loss: 79.2351, KL Div: 24.8611
    Epoch[11/15], Step [70/469], Reconst Loss: 80.4274, KL Div: 25.1910
    Epoch[11/15], Step [80/469], Reconst Loss: 81.0855, KL Div: 25.4724
    Epoch[11/15], Step [90/469], Reconst Loss: 78.0985, KL Div: 25.4388
    Epoch[11/15], Step [100/469], Reconst Loss: 78.7710, KL Div: 25.5180
    Epoch[11/15], Step [110/469], Reconst Loss: 81.5187, KL Div: 25.3222
    Epoch[11/15], Step [120/469], Reconst Loss: 83.3076, KL Div: 25.5328
    Epoch[11/15], Step [130/469], Reconst Loss: 79.2349, KL Div: 25.3751
    Epoch[11/15], Step [140/469], Reconst Loss: 81.8621, KL Div: 24.9382
    Epoch[11/15], Step [150/469], Reconst Loss: 81.4174, KL Div: 25.1285
    Epoch[11/15], Step [160/469], Reconst Loss: 81.4647, KL Div: 26.0012
    Epoch[11/15], Step [170/469], Reconst Loss: 83.7344, KL Div: 25.1536
    Epoch[11/15], Step [180/469], Reconst Loss: 78.9804, KL Div: 25.7035
    Epoch[11/15], Step [190/469], Reconst Loss: 79.9718, KL Div: 25.6583
    Epoch[11/15], Step [200/469], Reconst Loss: 82.0243, KL Div: 24.3348
    Epoch[11/15], Step [210/469], Reconst Loss: 75.3854, KL Div: 25.3928
    Epoch[11/15], Step [220/469], Reconst Loss: 81.9210, KL Div: 24.5277
    Epoch[11/15], Step [230/469], Reconst Loss: 78.9593, KL Div: 26.2109
    Epoch[11/15], Step [240/469], Reconst Loss: 77.9231, KL Div: 25.6205
    Epoch[11/15], Step [250/469], Reconst Loss: 78.8128, KL Div: 25.4733
    Epoch[11/15], Step [260/469], Reconst Loss: 82.3756, KL Div: 24.7012
    Epoch[11/15], Step [270/469], Reconst Loss: 80.4883, KL Div: 25.7740
    Epoch[11/15], Step [280/469], Reconst Loss: 81.4550, KL Div: 25.3725
    Epoch[11/15], Step [290/469], Reconst Loss: 77.6456, KL Div: 25.0046
    Epoch[11/15], Step [300/469], Reconst Loss: 83.6936, KL Div: 25.0141
    Epoch[11/15], Step [310/469], Reconst Loss: 78.7763, KL Div: 25.4609
    Epoch[11/15], Step [320/469], Reconst Loss: 79.4224, KL Div: 25.6144
    Epoch[11/15], Step [330/469], Reconst Loss: 79.2705, KL Div: 24.6755
    Epoch[11/15], Step [340/469], Reconst Loss: 79.8311, KL Div: 25.1374
    Epoch[11/15], Step [350/469], Reconst Loss: 83.1249, KL Div: 25.3839
    Epoch[11/15], Step [360/469], Reconst Loss: 79.9598, KL Div: 25.2443
    Epoch[11/15], Step [370/469], Reconst Loss: 81.0097, KL Div: 25.9255
    Epoch[11/15], Step [380/469], Reconst Loss: 80.6209, KL Div: 25.8971
    Epoch[11/15], Step [390/469], Reconst Loss: 80.7376, KL Div: 25.0892
    Epoch[11/15], Step [400/469], Reconst Loss: 80.7466, KL Div: 25.7320
    Epoch[11/15], Step [410/469], Reconst Loss: 81.4588, KL Div: 25.9351
    Epoch[11/15], Step [420/469], Reconst Loss: 79.4719, KL Div: 24.5379
    Epoch[11/15], Step [430/469], Reconst Loss: 81.7722, KL Div: 26.4869
    Epoch[11/15], Step [440/469], Reconst Loss: 80.8953, KL Div: 25.1019
    Epoch[11/15], Step [450/469], Reconst Loss: 80.1982, KL Div: 25.6955
    Epoch[11/15], Step [460/469], Reconst Loss: 82.8763, KL Div: 24.5053
    Epoch[12/15], Step [10/469], Reconst Loss: 78.0871, KL Div: 24.9657
    Epoch[12/15], Step [20/469], Reconst Loss: 78.0917, KL Div: 24.8710
    Epoch[12/15], Step [30/469], Reconst Loss: 81.0662, KL Div: 25.6844
    Epoch[12/15], Step [40/469], Reconst Loss: 81.0481, KL Div: 25.2845
    Epoch[12/15], Step [50/469], Reconst Loss: 86.0144, KL Div: 25.7664
    Epoch[12/15], Step [60/469], Reconst Loss: 82.9201, KL Div: 24.9147
    Epoch[12/15], Step [70/469], Reconst Loss: 80.9565, KL Div: 25.2537
    Epoch[12/15], Step [80/469], Reconst Loss: 81.6467, KL Div: 25.6834
    Epoch[12/15], Step [90/469], Reconst Loss: 81.3523, KL Div: 25.4150
    Epoch[12/15], Step [100/469], Reconst Loss: 83.9734, KL Div: 25.9917
    Epoch[12/15], Step [110/469], Reconst Loss: 82.6414, KL Div: 25.6876
    Epoch[12/15], Step [120/469], Reconst Loss: 79.9054, KL Div: 25.6890
    Epoch[12/15], Step [130/469], Reconst Loss: 80.6320, KL Div: 25.7946
    Epoch[12/15], Step [140/469], Reconst Loss: 82.5407, KL Div: 25.0998
    Epoch[12/15], Step [150/469], Reconst Loss: 78.1238, KL Div: 25.5628
    Epoch[12/15], Step [160/469], Reconst Loss: 81.5852, KL Div: 25.4603
    Epoch[12/15], Step [170/469], Reconst Loss: 79.7386, KL Div: 25.4694
    Epoch[12/15], Step [180/469], Reconst Loss: 81.4807, KL Div: 25.8463
    Epoch[12/15], Step [190/469], Reconst Loss: 82.0069, KL Div: 25.4319
    Epoch[12/15], Step [200/469], Reconst Loss: 79.0649, KL Div: 25.7112
    Epoch[12/15], Step [210/469], Reconst Loss: 77.6934, KL Div: 25.0970
    Epoch[12/15], Step [220/469], Reconst Loss: 79.9647, KL Div: 26.0611
    Epoch[12/15], Step [230/469], Reconst Loss: 77.7318, KL Div: 24.7429
    Epoch[12/15], Step [240/469], Reconst Loss: 79.8627, KL Div: 25.2986
    Epoch[12/15], Step [250/469], Reconst Loss: 77.7265, KL Div: 25.9675
    Epoch[12/15], Step [260/469], Reconst Loss: 82.4231, KL Div: 25.2075
    Epoch[12/15], Step [270/469], Reconst Loss: 80.4948, KL Div: 25.1094
    Epoch[12/15], Step [280/469], Reconst Loss: 81.4790, KL Div: 25.7851
    Epoch[12/15], Step [290/469], Reconst Loss: 81.8665, KL Div: 25.4227
    Epoch[12/15], Step [300/469], Reconst Loss: 79.3176, KL Div: 25.4005
    Epoch[12/15], Step [310/469], Reconst Loss: 83.2655, KL Div: 25.2819
    Epoch[12/15], Step [320/469], Reconst Loss: 76.2343, KL Div: 25.2915
    Epoch[12/15], Step [330/469], Reconst Loss: 78.5690, KL Div: 24.2394
    Epoch[12/15], Step [340/469], Reconst Loss: 80.7908, KL Div: 25.4832
    Epoch[12/15], Step [350/469], Reconst Loss: 79.7970, KL Div: 25.4078
    Epoch[12/15], Step [360/469], Reconst Loss: 81.7827, KL Div: 25.2942
    Epoch[12/15], Step [370/469], Reconst Loss: 82.2201, KL Div: 25.6688
    Epoch[12/15], Step [380/469], Reconst Loss: 77.7119, KL Div: 24.9353
    Epoch[12/15], Step [390/469], Reconst Loss: 84.6291, KL Div: 25.8510
    Epoch[12/15], Step [400/469], Reconst Loss: 82.9305, KL Div: 25.8565
    Epoch[12/15], Step [410/469], Reconst Loss: 82.8232, KL Div: 25.4759
    Epoch[12/15], Step [420/469], Reconst Loss: 81.1320, KL Div: 25.5605
    Epoch[12/15], Step [430/469], Reconst Loss: 76.1791, KL Div: 25.2261
    Epoch[12/15], Step [440/469], Reconst Loss: 78.1530, KL Div: 24.7712
    Epoch[12/15], Step [450/469], Reconst Loss: 80.9111, KL Div: 25.2736
    Epoch[12/15], Step [460/469], Reconst Loss: 82.8898, KL Div: 25.9192
    Epoch[13/15], Step [10/469], Reconst Loss: 81.1904, KL Div: 25.4812
    Epoch[13/15], Step [20/469], Reconst Loss: 81.5334, KL Div: 25.2996
    Epoch[13/15], Step [30/469], Reconst Loss: 78.3835, KL Div: 25.6803
    Epoch[13/15], Step [40/469], Reconst Loss: 80.5652, KL Div: 25.7653
    Epoch[13/15], Step [50/469], Reconst Loss: 80.5329, KL Div: 25.5047
    Epoch[13/15], Step [60/469], Reconst Loss: 77.5572, KL Div: 25.7582
    Epoch[13/15], Step [70/469], Reconst Loss: 74.8325, KL Div: 24.0017
    Epoch[13/15], Step [80/469], Reconst Loss: 78.9147, KL Div: 25.9084
    Epoch[13/15], Step [90/469], Reconst Loss: 78.6312, KL Div: 25.4255
    Epoch[13/15], Step [100/469], Reconst Loss: 81.0994, KL Div: 25.4389
    Epoch[13/15], Step [110/469], Reconst Loss: 76.4075, KL Div: 25.7223
    Epoch[13/15], Step [120/469], Reconst Loss: 80.9502, KL Div: 25.1831
    Epoch[13/15], Step [130/469], Reconst Loss: 82.8090, KL Div: 25.5760
    Epoch[13/15], Step [140/469], Reconst Loss: 79.0186, KL Div: 24.7029
    Epoch[13/15], Step [150/469], Reconst Loss: 80.2117, KL Div: 26.3687
    Epoch[13/15], Step [160/469], Reconst Loss: 81.8945, KL Div: 25.7391
    Epoch[13/15], Step [170/469], Reconst Loss: 80.7304, KL Div: 25.5473
    Epoch[13/15], Step [180/469], Reconst Loss: 81.9524, KL Div: 25.7723
    Epoch[13/15], Step [190/469], Reconst Loss: 76.2498, KL Div: 24.7319
    Epoch[13/15], Step [200/469], Reconst Loss: 80.8863, KL Div: 25.3866
    Epoch[13/15], Step [210/469], Reconst Loss: 77.2035, KL Div: 24.9184
    Epoch[13/15], Step [220/469], Reconst Loss: 81.4882, KL Div: 26.2688
    Epoch[13/15], Step [230/469], Reconst Loss: 83.3025, KL Div: 25.7013
    Epoch[13/15], Step [240/469], Reconst Loss: 85.1575, KL Div: 26.3229
    Epoch[13/15], Step [250/469], Reconst Loss: 78.8237, KL Div: 25.4030
    Epoch[13/15], Step [260/469], Reconst Loss: 79.2389, KL Div: 24.5529
    Epoch[13/15], Step [270/469], Reconst Loss: 80.5692, KL Div: 25.6988
    Epoch[13/15], Step [280/469], Reconst Loss: 83.2591, KL Div: 25.4824
    Epoch[13/15], Step [290/469], Reconst Loss: 78.3958, KL Div: 25.6455
    Epoch[13/15], Step [300/469], Reconst Loss: 83.0644, KL Div: 26.1527
    Epoch[13/15], Step [310/469], Reconst Loss: 79.5108, KL Div: 25.2392
    Epoch[13/15], Step [320/469], Reconst Loss: 81.5407, KL Div: 25.5366
    Epoch[13/15], Step [330/469], Reconst Loss: 82.0558, KL Div: 25.2214
    Epoch[13/15], Step [340/469], Reconst Loss: 81.1264, KL Div: 25.1372
    Epoch[13/15], Step [350/469], Reconst Loss: 79.8261, KL Div: 25.6791
    Epoch[13/15], Step [360/469], Reconst Loss: 78.7042, KL Div: 24.9986
    Epoch[13/15], Step [370/469], Reconst Loss: 82.0932, KL Div: 26.3858
    Epoch[13/15], Step [380/469], Reconst Loss: 79.9886, KL Div: 26.3256
    Epoch[13/15], Step [390/469], Reconst Loss: 80.8247, KL Div: 25.0925
    Epoch[13/15], Step [400/469], Reconst Loss: 78.0183, KL Div: 24.7194
    Epoch[13/15], Step [410/469], Reconst Loss: 76.7849, KL Div: 25.6365
    Epoch[13/15], Step [420/469], Reconst Loss: 83.1875, KL Div: 25.2420
    Epoch[13/15], Step [430/469], Reconst Loss: 79.3331, KL Div: 25.9699
    Epoch[13/15], Step [440/469], Reconst Loss: 81.1375, KL Div: 25.5769
    Epoch[13/15], Step [450/469], Reconst Loss: 79.5066, KL Div: 25.0094
    Epoch[13/15], Step [460/469], Reconst Loss: 81.7754, KL Div: 25.8637
    Epoch[14/15], Step [10/469], Reconst Loss: 78.5350, KL Div: 25.3790
    Epoch[14/15], Step [20/469], Reconst Loss: 77.3757, KL Div: 25.6061
    Epoch[14/15], Step [30/469], Reconst Loss: 81.2092, KL Div: 25.6415
    Epoch[14/15], Step [40/469], Reconst Loss: 79.7324, KL Div: 24.5458
    Epoch[14/15], Step [50/469], Reconst Loss: 79.6952, KL Div: 25.4341
    Epoch[14/15], Step [60/469], Reconst Loss: 79.8843, KL Div: 25.3621
    Epoch[14/15], Step [70/469], Reconst Loss: 81.4862, KL Div: 25.3876
    Epoch[14/15], Step [80/469], Reconst Loss: 78.9900, KL Div: 25.5733
    Epoch[14/15], Step [90/469], Reconst Loss: 78.5505, KL Div: 25.7702
    Epoch[14/15], Step [100/469], Reconst Loss: 82.5692, KL Div: 26.0789
    Epoch[14/15], Step [110/469], Reconst Loss: 77.7710, KL Div: 25.6365
    Epoch[14/15], Step [120/469], Reconst Loss: 79.4527, KL Div: 26.0826
    Epoch[14/15], Step [130/469], Reconst Loss: 79.4439, KL Div: 25.4844
    Epoch[14/15], Step [140/469], Reconst Loss: 83.2154, KL Div: 26.0082
    Epoch[14/15], Step [150/469], Reconst Loss: 78.8878, KL Div: 25.3778
    Epoch[14/15], Step [160/469], Reconst Loss: 78.6515, KL Div: 25.2636
    Epoch[14/15], Step [170/469], Reconst Loss: 75.8635, KL Div: 24.7826
    Epoch[14/15], Step [180/469], Reconst Loss: 79.3048, KL Div: 25.2690
    Epoch[14/15], Step [190/469], Reconst Loss: 78.2646, KL Div: 25.3400
    Epoch[14/15], Step [200/469], Reconst Loss: 77.3817, KL Div: 25.7637
    Epoch[14/15], Step [210/469], Reconst Loss: 78.9489, KL Div: 25.5684
    Epoch[14/15], Step [220/469], Reconst Loss: 77.7267, KL Div: 25.7162
    Epoch[14/15], Step [230/469], Reconst Loss: 77.3374, KL Div: 25.2003
    Epoch[14/15], Step [240/469], Reconst Loss: 78.6105, KL Div: 25.5992
    Epoch[14/15], Step [250/469], Reconst Loss: 81.4851, KL Div: 25.4490
    Epoch[14/15], Step [260/469], Reconst Loss: 81.4021, KL Div: 25.6057
    Epoch[14/15], Step [270/469], Reconst Loss: 79.5215, KL Div: 24.7424
    Epoch[14/15], Step [280/469], Reconst Loss: 76.9483, KL Div: 25.5064
    Epoch[14/15], Step [290/469], Reconst Loss: 80.4043, KL Div: 25.3503
    Epoch[14/15], Step [300/469], Reconst Loss: 83.4428, KL Div: 25.6133
    Epoch[14/15], Step [310/469], Reconst Loss: 79.8034, KL Div: 25.1495
    Epoch[14/15], Step [320/469], Reconst Loss: 81.6979, KL Div: 25.0916
    Epoch[14/15], Step [330/469], Reconst Loss: 78.4999, KL Div: 25.1405
    Epoch[14/15], Step [340/469], Reconst Loss: 78.7892, KL Div: 25.5711
    Epoch[14/15], Step [350/469], Reconst Loss: 79.0046, KL Div: 25.4135
    Epoch[14/15], Step [360/469], Reconst Loss: 82.0038, KL Div: 25.7562
    Epoch[14/15], Step [370/469], Reconst Loss: 77.0693, KL Div: 25.1360
    Epoch[14/15], Step [380/469], Reconst Loss: 83.4615, KL Div: 25.4235
    Epoch[14/15], Step [390/469], Reconst Loss: 79.9559, KL Div: 25.4437
    Epoch[14/15], Step [400/469], Reconst Loss: 80.0277, KL Div: 25.5238
    Epoch[14/15], Step [410/469], Reconst Loss: 79.1139, KL Div: 25.0590
    Epoch[14/15], Step [420/469], Reconst Loss: 75.9453, KL Div: 24.4021
    Epoch[14/15], Step [430/469], Reconst Loss: 77.7330, KL Div: 25.3021
    Epoch[14/15], Step [440/469], Reconst Loss: 78.3513, KL Div: 25.8471
    Epoch[14/15], Step [450/469], Reconst Loss: 81.7533, KL Div: 24.9906
    Epoch[14/15], Step [460/469], Reconst Loss: 76.0457, KL Div: 25.1263
    Epoch[15/15], Step [10/469], Reconst Loss: 79.7691, KL Div: 25.3802
    Epoch[15/15], Step [20/469], Reconst Loss: 77.8079, KL Div: 25.5221
    Epoch[15/15], Step [30/469], Reconst Loss: 81.8969, KL Div: 25.4933
    Epoch[15/15], Step [40/469], Reconst Loss: 80.5397, KL Div: 25.3296
    Epoch[15/15], Step [50/469], Reconst Loss: 77.7470, KL Div: 25.6413
    Epoch[15/15], Step [60/469], Reconst Loss: 75.9614, KL Div: 25.2043
    Epoch[15/15], Step [70/469], Reconst Loss: 81.9444, KL Div: 26.1145
    Epoch[15/15], Step [80/469], Reconst Loss: 78.7787, KL Div: 24.6590
    Epoch[15/15], Step [90/469], Reconst Loss: 79.2274, KL Div: 26.2333
    Epoch[15/15], Step [100/469], Reconst Loss: 80.4397, KL Div: 25.3362
    Epoch[15/15], Step [110/469], Reconst Loss: 81.1702, KL Div: 25.9141
    Epoch[15/15], Step [120/469], Reconst Loss: 79.6282, KL Div: 25.8796
    Epoch[15/15], Step [130/469], Reconst Loss: 79.8895, KL Div: 26.1675
    Epoch[15/15], Step [140/469], Reconst Loss: 75.8992, KL Div: 25.1448
    Epoch[15/15], Step [150/469], Reconst Loss: 79.4789, KL Div: 25.8143
    Epoch[15/15], Step [160/469], Reconst Loss: 84.4682, KL Div: 26.2439
    Epoch[15/15], Step [170/469], Reconst Loss: 78.2709, KL Div: 26.0026
    Epoch[15/15], Step [180/469], Reconst Loss: 79.8718, KL Div: 25.8113
    Epoch[15/15], Step [190/469], Reconst Loss: 80.8025, KL Div: 25.2451
    Epoch[15/15], Step [200/469], Reconst Loss: 82.9244, KL Div: 25.0495
    Epoch[15/15], Step [210/469], Reconst Loss: 81.5660, KL Div: 25.6745
    Epoch[15/15], Step [220/469], Reconst Loss: 79.4027, KL Div: 25.4569
    Epoch[15/15], Step [230/469], Reconst Loss: 78.0706, KL Div: 25.9456
    Epoch[15/15], Step [240/469], Reconst Loss: 79.0586, KL Div: 25.3234
    Epoch[15/15], Step [250/469], Reconst Loss: 77.4862, KL Div: 25.5026
    Epoch[15/15], Step [260/469], Reconst Loss: 79.5378, KL Div: 25.2674
    Epoch[15/15], Step [270/469], Reconst Loss: 78.9801, KL Div: 25.3041
    Epoch[15/15], Step [280/469], Reconst Loss: 80.2667, KL Div: 25.8804
    Epoch[15/15], Step [290/469], Reconst Loss: 78.0092, KL Div: 24.6465
    Epoch[15/15], Step [300/469], Reconst Loss: 82.2234, KL Div: 26.3831
    Epoch[15/15], Step [310/469], Reconst Loss: 80.8803, KL Div: 25.7925
    Epoch[15/15], Step [320/469], Reconst Loss: 78.1355, KL Div: 25.4706
    Epoch[15/15], Step [330/469], Reconst Loss: 76.8196, KL Div: 25.4953
    Epoch[15/15], Step [340/469], Reconst Loss: 83.0936, KL Div: 26.6631
    Epoch[15/15], Step [350/469], Reconst Loss: 81.9319, KL Div: 25.1247
    Epoch[15/15], Step [360/469], Reconst Loss: 80.5722, KL Div: 25.5892
    Epoch[15/15], Step [370/469], Reconst Loss: 81.5478, KL Div: 25.5967
    Epoch[15/15], Step [380/469], Reconst Loss: 83.0195, KL Div: 25.6249
    Epoch[15/15], Step [390/469], Reconst Loss: 79.5094, KL Div: 25.3942
    Epoch[15/15], Step [400/469], Reconst Loss: 82.0050, KL Div: 25.3936
    Epoch[15/15], Step [410/469], Reconst Loss: 78.4521, KL Div: 24.4835
    Epoch[15/15], Step [420/469], Reconst Loss: 77.6122, KL Div: 25.0833
    Epoch[15/15], Step [430/469], Reconst Loss: 79.9671, KL Div: 24.6797
    Epoch[15/15], Step [440/469], Reconst Loss: 82.6777, KL Div: 25.5717
    Epoch[15/15], Step [450/469], Reconst Loss: 82.1045, KL Div: 25.0951
    Epoch[15/15], Step [460/469], Reconst Loss: 74.2894, KL Div: 25.3463

Let see how our network reconstructs our last batch. We display pairs of original digits and reconstructed version.

```python
plt.figure(figsize=(10,5))
plot_reconstruction(model)
```

Let's see now how our network generates new samples.

```python
plt.figure(figsize=(10,5))
plot_generation(model)
```

Not great, but we did not train our network for long... That being said, we have no control of the generated digits. In the rest of this notebook, we explore ways to generates zeroes, ones, twos and so on. As a by-product, we show how our VAE will allow us to do clustering.

The main idea is to build what we call a Gumbel VAE as described below.

# Gumbel VAE

Implement a VAE where you add a categorical variable $c\in \{0,\dots 9\}$ so that your latent variable model is $p(c,z,x) = p(c)p(z)p_{\theta}(x|,c,z)$ and your variational posterior is $q_{\phi}(c|x)q_{\phi}(z|x)$ as described in this NeurIPS paper: [(Dupont, 2018)](https://arxiv.org/abs/1804.00104). Make minimal modifications to previous architecture.

The idea is to incorporate a categorical variable in your latent space. You hope that this categorical variable will encode the class of the digit, so that your network can use it for a better reconstruction. Moreover, if things work as planned, you will then be able to generate digits conditionally to the class, i.e. you can choose the class thanks to the latent categorical variable $c$ and then generate digits from this class.

As noticed above, in order to sample random variables while still being able to use backpropagation, we need to use the reparameterization trick which is easy for Gaussian random variables. For categorical random variables, the reparameterization trick is explained in [(Jang et al., 2016)](https://arxiv.org/abs/1611.01144). This is implemented in PyTorch thanks to [F.gumbel_softmax](https://pytorch.org/docs/stable/nn.html?highlight=gumbel_softmax#torch.nn.functional.gumbel_softmax).

```python
n_classes = 10

class VAE_Gumbel(nn.Module):
    def __init__(self, image_size=784, h_dim=400, z_dim=20, n_classes = 10):
        super(VAE_Gumbel, self).__init__()
        self.fc1 = nn.Linear(image_size, h_dim)
        self.fc2 = nn.Linear(h_dim, z_dim)
        self.fc3 = nn.Linear(h_dim, z_dim)
        self.fc4 = nn.Linear(h_dim, n_classes)
        self.fc5 = nn.Linear(z_dim + n_classes , h_dim)
        self.fc6 = nn.Linear(h_dim, image_size)

    def encode(self, x):
        # use F.log_softmax
        h = F.relu(self.fc1(x))
        return self.fc2(h), self.fc3(h), F.log_softmax(self.fc4(h), dim=1)

    def reparameterize(self, mu, log_var):
        std = torch.exp(log_var/2)
        eps = torch.randn_like(std)
        return mu + eps * std

    def decode(self, z, y_onehot):
        # use torch.cat
        zy = torch.cat((z, y_onehot), dim =1)
        h = F.relu(self.fc5(zy))
        return torch.sigmoid(self.fc6(h))

    def forward(self, x):
        # use F.gumbel_softmax
        mu, log_var, alpha = self.encode(x)
        z = self.reparameterize(mu, log_var)
        y_onehot = F.gumbel_softmax(alpha, tau=0.5)
        x_reconst = self.decode(z, y_onehot)
        return x_reconst, mu, log_var, y_onehot

model_G = VAE_Gumbel().to(device)
optimizer = torch.optim.Adam(model_G.parameters(), lr=learning_rate)
```

You need to modify the loss to take into account the categorical random variable with an uniform prior on $\{0,\dots 9\}$, see Appendix A.2 in [(Dupont, 2018)](https://arxiv.org/abs/1804.00104)

## EXPLAIN:
From the paper:
$$D_{\mathrm{KL}}\left( q_{\phi}(z|c,
x)||p(z,c)\right) = D_{\mathrm{KL}}\left( q_{\phi}(z|x)||p(z)\right) + D_{\mathrm{KL}}\left( q_{\phi}(c|x)||p(c)\right)$$

$$D_{\mathrm{KL}}\left( q_{\phi}(z|x)||p(z)\right) = -\frac{1}{2} \sum _j(1 + \log(\sigma_j) - \mu_j^2 - \sigma_j^2 ) $$

$c \sim g(\alpha)$ where $g$ is Gumbel softmax distribution.
$p(c)$ follows a Uniform Gumbel Softmax distribution. Hence in our case $p(c = 0) = \frac{1}{{n_{classes}}}$. We also have that $q_{\phi}(c|x)$ is a categorial random variable. Hence:

\begin{align} D_{\mathrm{KL}}\left( q_{\phi}(c|x)||p(c)\right) &= \sum_{i=1}^{n_{classes}} q_{i} \log \frac{q_i}{p_i} \\ &= \sum_{i=1}^{n_{classes}} q_i \log \frac{q_i}{1/n_{classes}} \\ &= \sum_{i=1}^{n_{classes}} q_i \log q_i + \sum_{i=1}^{n_{classes}} q_i\log n_{classes} \\ &= -H(q_{\phi}(c|x)) + \log(n_{classes})
\end{align}

we know that $q_i$ is returned by the model `g_alpha` and $p_i = 1/{n_{classes}}$ so we can compute the above straightforward.

```python
def train_G(model, data_loader=data_loader,num_epochs=num_epochs, beta = 1., verbose=True):
    nmi_scores = []
    model.train(True)
    for epoch in range(num_epochs):
        all_labels = []
        all_labels_est = []
        for i, (x, labels) in enumerate(data_loader):

            # Forward pass
            x = x.to(device).view(-1, image_size)
            x_reconst, mu, log_var, g_alpha = model(x)

            #NMI
            all_labels += labels.tolist()
            all_labels_est += torch.argmax(g_alpha, dim=1).tolist()

            reconst_loss = F.binary_cross_entropy(x_reconst, x, reduction='sum')
            KL_div_z = - 0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
            pi = torch.tensor([1/n_classes], device=device)
            H_cat = - torch.sum(g_alpha * g_alpha.log(), dim=1) #entropy
            KL_div_c = torch.sum(-H_cat - torch.log(pi))#expectency
            H_cat = torch.sum(H_cat) #helps for the print

            KL_div = KL_div_z + KL_div_c

            # Backprop and optimize
            loss = reconst_loss + beta*KL_div
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if verbose:
                if (i+1) % 10 == 0:
                    print ("Epoch[{}/{}], Step [{}/{}], Reconst Loss: {:.4f}, KL Div: {:.4f}, kl_div_c: {:.4f}, Entropy: {:.4f}"
                           .format(epoch+1, num_epochs, i+1, len(data_loader), reconst_loss.item()/batch_size,
                                   KL_div.item()/batch_size, KL_div_c.item()/batch_size, H_cat.item()/batch_size))

        nmi = normalized_mutual_info_score(all_labels, all_labels_est)
        nmi_scores.append(nmi)
    return nmi_scores
```

```python
nmi = train_G(model_G,num_epochs=10,verbose=True)
```

    Epoch[1/10], Step [10/469], Reconst Loss: 295.8097, KL Div: 26.3921, kl_div_c: 1.4442, Entropy: 0.8584
    Epoch[1/10], Step [20/469], Reconst Loss: 237.9132, KL Div: 11.6071, kl_div_c: 1.5115, Entropy: 0.7911
    Epoch[1/10], Step [30/469], Reconst Loss: 214.0738, KL Div: 12.1082, kl_div_c: 1.5578, Entropy: 0.7448
    Epoch[1/10], Step [40/469], Reconst Loss: 210.5674, KL Div: 6.6489, kl_div_c: 1.5111, Entropy: 0.7915
    Epoch[1/10], Step [50/469], Reconst Loss: 207.7448, KL Div: 7.2551, kl_div_c: 1.4965, Entropy: 0.8061
    Epoch[1/10], Step [60/469], Reconst Loss: 196.2546, KL Div: 7.8084, kl_div_c: 1.5598, Entropy: 0.7428
    Epoch[1/10], Step [70/469], Reconst Loss: 198.5005, KL Div: 7.8504, kl_div_c: 1.4972, Entropy: 0.8054
    Epoch[1/10], Step [80/469], Reconst Loss: 193.8957, KL Div: 8.8493, kl_div_c: 1.5796, Entropy: 0.7230
    Epoch[1/10], Step [90/469], Reconst Loss: 179.6999, KL Div: 10.7978, kl_div_c: 1.5632, Entropy: 0.7394
    Epoch[1/10], Step [100/469], Reconst Loss: 182.9502, KL Div: 11.5317, kl_div_c: 1.5646, Entropy: 0.7380
    Epoch[1/10], Step [110/469], Reconst Loss: 165.8508, KL Div: 13.5462, kl_div_c: 1.5259, Entropy: 0.7767
    Epoch[1/10], Step [120/469], Reconst Loss: 163.7408, KL Div: 12.5379, kl_div_c: 1.6064, Entropy: 0.6962
    Epoch[1/10], Step [130/469], Reconst Loss: 155.5877, KL Div: 13.9326, kl_div_c: 1.5652, Entropy: 0.7374
    Epoch[1/10], Step [140/469], Reconst Loss: 143.0812, KL Div: 15.7212, kl_div_c: 1.5446, Entropy: 0.7580
    Epoch[1/10], Step [150/469], Reconst Loss: 153.8530, KL Div: 14.6770, kl_div_c: 1.5803, Entropy: 0.7223
    Epoch[1/10], Step [160/469], Reconst Loss: 148.7237, KL Div: 14.7831, kl_div_c: 1.5979, Entropy: 0.7047
    Epoch[1/10], Step [170/469], Reconst Loss: 141.6917, KL Div: 15.3613, kl_div_c: 1.5823, Entropy: 0.7202
    Epoch[1/10], Step [180/469], Reconst Loss: 140.0088, KL Div: 16.1007, kl_div_c: 1.5696, Entropy: 0.7330
    Epoch[1/10], Step [190/469], Reconst Loss: 143.6349, KL Div: 16.2724, kl_div_c: 1.5796, Entropy: 0.7229
    Epoch[1/10], Step [200/469], Reconst Loss: 144.0509, KL Div: 16.3169, kl_div_c: 1.5510, Entropy: 0.7516
    Epoch[1/10], Step [210/469], Reconst Loss: 135.4578, KL Div: 16.2302, kl_div_c: 1.6034, Entropy: 0.6992
    Epoch[1/10], Step [220/469], Reconst Loss: 138.2269, KL Div: 16.5874, kl_div_c: 1.5571, Entropy: 0.7455
    Epoch[1/10], Step [230/469], Reconst Loss: 135.6042, KL Div: 16.7409, kl_div_c: 1.5914, Entropy: 0.7112
    Epoch[1/10], Step [240/469], Reconst Loss: 130.7148, KL Div: 18.5083, kl_div_c: 1.6191, Entropy: 0.6835
    Epoch[1/10], Step [250/469], Reconst Loss: 131.4883, KL Div: 17.5735, kl_div_c: 1.6433, Entropy: 0.6593
    Epoch[1/10], Step [260/469], Reconst Loss: 126.9528, KL Div: 19.4707, kl_div_c: 1.5532, Entropy: 0.7494
    Epoch[1/10], Step [270/469], Reconst Loss: 131.5822, KL Div: 18.6858, kl_div_c: 1.6040, Entropy: 0.6986
    Epoch[1/10], Step [280/469], Reconst Loss: 130.8069, KL Div: 19.6888, kl_div_c: 1.6514, Entropy: 0.6512
    Epoch[1/10], Step [290/469], Reconst Loss: 126.6651, KL Div: 18.1131, kl_div_c: 1.5983, Entropy: 0.7043
    Epoch[1/10], Step [300/469], Reconst Loss: 123.2835, KL Div: 20.2194, kl_div_c: 1.6112, Entropy: 0.6914
    Epoch[1/10], Step [310/469], Reconst Loss: 121.4461, KL Div: 19.1399, kl_div_c: 1.6798, Entropy: 0.6228
    Epoch[1/10], Step [320/469], Reconst Loss: 123.6997, KL Div: 20.2656, kl_div_c: 1.6398, Entropy: 0.6628
    Epoch[1/10], Step [330/469], Reconst Loss: 116.5174, KL Div: 20.3636, kl_div_c: 1.5592, Entropy: 0.7434
    Epoch[1/10], Step [340/469], Reconst Loss: 123.8798, KL Div: 20.1290, kl_div_c: 1.6828, Entropy: 0.6197
    Epoch[1/10], Step [350/469], Reconst Loss: 118.1562, KL Div: 20.1963, kl_div_c: 1.6381, Entropy: 0.6645
    Epoch[1/10], Step [360/469], Reconst Loss: 116.3649, KL Div: 20.3666, kl_div_c: 1.5853, Entropy: 0.7173
    Epoch[1/10], Step [370/469], Reconst Loss: 123.5720, KL Div: 20.1809, kl_div_c: 1.5386, Entropy: 0.7639
    Epoch[1/10], Step [380/469], Reconst Loss: 116.9432, KL Div: 20.7896, kl_div_c: 1.6233, Entropy: 0.6793
    Epoch[1/10], Step [390/469], Reconst Loss: 116.2906, KL Div: 21.1667, kl_div_c: 1.6157, Entropy: 0.6869
    Epoch[1/10], Step [400/469], Reconst Loss: 107.0432, KL Div: 21.0290, kl_div_c: 1.5124, Entropy: 0.7901
    Epoch[1/10], Step [410/469], Reconst Loss: 117.6930, KL Div: 21.6393, kl_div_c: 1.5827, Entropy: 0.7199
    Epoch[1/10], Step [420/469], Reconst Loss: 110.0228, KL Div: 21.1658, kl_div_c: 1.6103, Entropy: 0.6923
    Epoch[1/10], Step [430/469], Reconst Loss: 106.8650, KL Div: 22.3406, kl_div_c: 1.6316, Entropy: 0.6710
    Epoch[1/10], Step [440/469], Reconst Loss: 111.8057, KL Div: 21.3999, kl_div_c: 1.6132, Entropy: 0.6894
    Epoch[1/10], Step [450/469], Reconst Loss: 113.6897, KL Div: 22.1589, kl_div_c: 1.6307, Entropy: 0.6719
    Epoch[1/10], Step [460/469], Reconst Loss: 109.5054, KL Div: 22.1046, kl_div_c: 1.5685, Entropy: 0.7341
    Epoch[2/10], Step [10/469], Reconst Loss: 105.8199, KL Div: 21.4837, kl_div_c: 1.6591, Entropy: 0.6434
    Epoch[2/10], Step [20/469], Reconst Loss: 109.6519, KL Div: 22.1956, kl_div_c: 1.6323, Entropy: 0.6702
    Epoch[2/10], Step [30/469], Reconst Loss: 108.4492, KL Div: 21.7629, kl_div_c: 1.6532, Entropy: 0.6494
    Epoch[2/10], Step [40/469], Reconst Loss: 108.2049, KL Div: 23.2601, kl_div_c: 1.5468, Entropy: 0.7558
    Epoch[2/10], Step [50/469], Reconst Loss: 104.3150, KL Div: 23.3348, kl_div_c: 1.5581, Entropy: 0.7445
    Epoch[2/10], Step [60/469], Reconst Loss: 107.7467, KL Div: 22.5503, kl_div_c: 1.5400, Entropy: 0.7626
    Epoch[2/10], Step [70/469], Reconst Loss: 106.2855, KL Div: 22.4544, kl_div_c: 1.6489, Entropy: 0.6537
    Epoch[2/10], Step [80/469], Reconst Loss: 102.3881, KL Div: 23.8529, kl_div_c: 1.5344, Entropy: 0.7682
    Epoch[2/10], Step [90/469], Reconst Loss: 102.3313, KL Div: 22.6483, kl_div_c: 1.5696, Entropy: 0.7329
    Epoch[2/10], Step [100/469], Reconst Loss: 105.0673, KL Div: 22.6514, kl_div_c: 1.6093, Entropy: 0.6933
    Epoch[2/10], Step [110/469], Reconst Loss: 104.0163, KL Div: 23.1691, kl_div_c: 1.6526, Entropy: 0.6500
    Epoch[2/10], Step [120/469], Reconst Loss: 105.9471, KL Div: 22.2200, kl_div_c: 1.5561, Entropy: 0.7465
    Epoch[2/10], Step [130/469], Reconst Loss: 103.3039, KL Div: 22.7783, kl_div_c: 1.5313, Entropy: 0.7713
    Epoch[2/10], Step [140/469], Reconst Loss: 98.0113, KL Div: 23.7123, kl_div_c: 1.6053, Entropy: 0.6973
    Epoch[2/10], Step [150/469], Reconst Loss: 101.5639, KL Div: 22.9122, kl_div_c: 1.5876, Entropy: 0.7150
    Epoch[2/10], Step [160/469], Reconst Loss: 100.2914, KL Div: 23.4200, kl_div_c: 1.5962, Entropy: 0.7063
    Epoch[2/10], Step [170/469], Reconst Loss: 105.3157, KL Div: 23.2348, kl_div_c: 1.5612, Entropy: 0.7414
    Epoch[2/10], Step [180/469], Reconst Loss: 102.3168, KL Div: 23.4811, kl_div_c: 1.5963, Entropy: 0.7063
    Epoch[2/10], Step [190/469], Reconst Loss: 101.0537, KL Div: 22.7332, kl_div_c: 1.5575, Entropy: 0.7451
    Epoch[2/10], Step [200/469], Reconst Loss: 103.0697, KL Div: 23.4821, kl_div_c: 1.6942, Entropy: 0.6084
    Epoch[2/10], Step [210/469], Reconst Loss: 99.0255, KL Div: 23.1827, kl_div_c: 1.5925, Entropy: 0.7101
    Epoch[2/10], Step [220/469], Reconst Loss: 103.1606, KL Div: 23.2392, kl_div_c: 1.6389, Entropy: 0.6637
    Epoch[2/10], Step [230/469], Reconst Loss: 96.9066, KL Div: 23.1336, kl_div_c: 1.5698, Entropy: 0.7328
    Epoch[2/10], Step [240/469], Reconst Loss: 101.1309, KL Div: 24.0240, kl_div_c: 1.5765, Entropy: 0.7260
    Epoch[2/10], Step [250/469], Reconst Loss: 98.1825, KL Div: 23.0537, kl_div_c: 1.5893, Entropy: 0.7133
    Epoch[2/10], Step [260/469], Reconst Loss: 100.8906, KL Div: 24.3669, kl_div_c: 1.6141, Entropy: 0.6885
    Epoch[2/10], Step [270/469], Reconst Loss: 103.6113, KL Div: 23.6534, kl_div_c: 1.6189, Entropy: 0.6837
    Epoch[2/10], Step [280/469], Reconst Loss: 100.1171, KL Div: 23.3242, kl_div_c: 1.4948, Entropy: 0.8078
    Epoch[2/10], Step [290/469], Reconst Loss: 96.2596, KL Div: 23.8986, kl_div_c: 1.6089, Entropy: 0.6937
    Epoch[2/10], Step [300/469], Reconst Loss: 98.5585, KL Div: 23.4561, kl_div_c: 1.5901, Entropy: 0.7125
    Epoch[2/10], Step [310/469], Reconst Loss: 91.2010, KL Div: 22.9394, kl_div_c: 1.6278, Entropy: 0.6748
    Epoch[2/10], Step [320/469], Reconst Loss: 93.8557, KL Div: 24.2604, kl_div_c: 1.5320, Entropy: 0.7706
    Epoch[2/10], Step [330/469], Reconst Loss: 95.3995, KL Div: 24.6029, kl_div_c: 1.5867, Entropy: 0.7159
    Epoch[2/10], Step [340/469], Reconst Loss: 99.2123, KL Div: 24.1032, kl_div_c: 1.6053, Entropy: 0.6973
    Epoch[2/10], Step [350/469], Reconst Loss: 95.5485, KL Div: 24.1051, kl_div_c: 1.5198, Entropy: 0.7828
    Epoch[2/10], Step [360/469], Reconst Loss: 91.5845, KL Div: 24.3046, kl_div_c: 1.6097, Entropy: 0.6929
    Epoch[2/10], Step [370/469], Reconst Loss: 94.5486, KL Div: 24.0813, kl_div_c: 1.6275, Entropy: 0.6750
    Epoch[2/10], Step [380/469], Reconst Loss: 98.0099, KL Div: 24.5214, kl_div_c: 1.6182, Entropy: 0.6843
    Epoch[2/10], Step [390/469], Reconst Loss: 96.9200, KL Div: 24.1616, kl_div_c: 1.6085, Entropy: 0.6940
    Epoch[2/10], Step [400/469], Reconst Loss: 96.7135, KL Div: 23.9197, kl_div_c: 1.5738, Entropy: 0.7288
    Epoch[2/10], Step [410/469], Reconst Loss: 98.1384, KL Div: 25.1735, kl_div_c: 1.6148, Entropy: 0.6878
    Epoch[2/10], Step [420/469], Reconst Loss: 95.2170, KL Div: 24.2239, kl_div_c: 1.5883, Entropy: 0.7142
    Epoch[2/10], Step [430/469], Reconst Loss: 94.0985, KL Div: 24.2298, kl_div_c: 1.6604, Entropy: 0.6422
    Epoch[2/10], Step [440/469], Reconst Loss: 92.5431, KL Div: 24.4869, kl_div_c: 1.6405, Entropy: 0.6621
    Epoch[2/10], Step [450/469], Reconst Loss: 94.8092, KL Div: 24.0105, kl_div_c: 1.5276, Entropy: 0.7750
    Epoch[2/10], Step [460/469], Reconst Loss: 91.8664, KL Div: 24.9804, kl_div_c: 1.5915, Entropy: 0.7111
    Epoch[3/10], Step [10/469], Reconst Loss: 90.8702, KL Div: 24.6528, kl_div_c: 1.6290, Entropy: 0.6736
    Epoch[3/10], Step [20/469], Reconst Loss: 94.9081, KL Div: 24.2712, kl_div_c: 1.6346, Entropy: 0.6680
    Epoch[3/10], Step [30/469], Reconst Loss: 92.3664, KL Div: 25.1807, kl_div_c: 1.5738, Entropy: 0.7288
    Epoch[3/10], Step [40/469], Reconst Loss: 91.0745, KL Div: 24.7335, kl_div_c: 1.5896, Entropy: 0.7130
    Epoch[3/10], Step [50/469], Reconst Loss: 91.7773, KL Div: 24.7270, kl_div_c: 1.5503, Entropy: 0.7523
    Epoch[3/10], Step [60/469], Reconst Loss: 93.8614, KL Div: 25.8279, kl_div_c: 1.6717, Entropy: 0.6309
    Epoch[3/10], Step [70/469], Reconst Loss: 95.0383, KL Div: 24.4232, kl_div_c: 1.5634, Entropy: 0.7392
    Epoch[3/10], Step [80/469], Reconst Loss: 91.6873, KL Div: 24.6065, kl_div_c: 1.5735, Entropy: 0.7291
    Epoch[3/10], Step [90/469], Reconst Loss: 93.0638, KL Div: 25.1497, kl_div_c: 1.5396, Entropy: 0.7630
    Epoch[3/10], Step [100/469], Reconst Loss: 96.7036, KL Div: 25.1796, kl_div_c: 1.6349, Entropy: 0.6677
    Epoch[3/10], Step [110/469], Reconst Loss: 93.3583, KL Div: 25.3399, kl_div_c: 1.6535, Entropy: 0.6491
    Epoch[3/10], Step [120/469], Reconst Loss: 92.9861, KL Div: 24.6848, kl_div_c: 1.6236, Entropy: 0.6790
    Epoch[3/10], Step [130/469], Reconst Loss: 89.7346, KL Div: 24.8840, kl_div_c: 1.5945, Entropy: 0.7081
    Epoch[3/10], Step [140/469], Reconst Loss: 95.9199, KL Div: 24.9760, kl_div_c: 1.6619, Entropy: 0.6407
    Epoch[3/10], Step [150/469], Reconst Loss: 91.1054, KL Div: 24.6462, kl_div_c: 1.5945, Entropy: 0.7081
    Epoch[3/10], Step [160/469], Reconst Loss: 90.2808, KL Div: 25.3967, kl_div_c: 1.5700, Entropy: 0.7326
    Epoch[3/10], Step [170/469], Reconst Loss: 91.4181, KL Div: 25.0372, kl_div_c: 1.6446, Entropy: 0.6580
    Epoch[3/10], Step [180/469], Reconst Loss: 93.5807, KL Div: 25.1535, kl_div_c: 1.5363, Entropy: 0.7662
    Epoch[3/10], Step [190/469], Reconst Loss: 93.5025, KL Div: 25.2183, kl_div_c: 1.5777, Entropy: 0.7249
    Epoch[3/10], Step [200/469], Reconst Loss: 92.2743, KL Div: 25.1205, kl_div_c: 1.6038, Entropy: 0.6988
    Epoch[3/10], Step [210/469], Reconst Loss: 92.1523, KL Div: 25.7696, kl_div_c: 1.5918, Entropy: 0.7108
    Epoch[3/10], Step [220/469], Reconst Loss: 88.0824, KL Div: 25.1307, kl_div_c: 1.5921, Entropy: 0.7105
    Epoch[3/10], Step [230/469], Reconst Loss: 91.9359, KL Div: 24.4425, kl_div_c: 1.6164, Entropy: 0.6862
    Epoch[3/10], Step [240/469], Reconst Loss: 94.5062, KL Div: 25.6117, kl_div_c: 1.6322, Entropy: 0.6703
    Epoch[3/10], Step [250/469], Reconst Loss: 89.3999, KL Div: 25.0052, kl_div_c: 1.5723, Entropy: 0.7303
    Epoch[3/10], Step [260/469], Reconst Loss: 87.3882, KL Div: 24.9532, kl_div_c: 1.6593, Entropy: 0.6433
    Epoch[3/10], Step [270/469], Reconst Loss: 90.1601, KL Div: 24.4949, kl_div_c: 1.5063, Entropy: 0.7962
    Epoch[3/10], Step [280/469], Reconst Loss: 89.3063, KL Div: 26.0614, kl_div_c: 1.6379, Entropy: 0.6647
    Epoch[3/10], Step [290/469], Reconst Loss: 90.9888, KL Div: 24.7031, kl_div_c: 1.5651, Entropy: 0.7375
    Epoch[3/10], Step [300/469], Reconst Loss: 91.4464, KL Div: 25.0830, kl_div_c: 1.6312, Entropy: 0.6714
    Epoch[3/10], Step [310/469], Reconst Loss: 91.2445, KL Div: 25.5849, kl_div_c: 1.6914, Entropy: 0.6112
    Epoch[3/10], Step [320/469], Reconst Loss: 91.5661, KL Div: 25.0788, kl_div_c: 1.5913, Entropy: 0.7113
    Epoch[3/10], Step [330/469], Reconst Loss: 90.1799, KL Div: 25.1233, kl_div_c: 1.6846, Entropy: 0.6180
    Epoch[3/10], Step [340/469], Reconst Loss: 89.5160, KL Div: 24.8637, kl_div_c: 1.6057, Entropy: 0.6969
    Epoch[3/10], Step [350/469], Reconst Loss: 94.1532, KL Div: 25.4592, kl_div_c: 1.6078, Entropy: 0.6948
    Epoch[3/10], Step [360/469], Reconst Loss: 92.0052, KL Div: 25.4761, kl_div_c: 1.6766, Entropy: 0.6260
    Epoch[3/10], Step [370/469], Reconst Loss: 91.6859, KL Div: 24.6147, kl_div_c: 1.5593, Entropy: 0.7433
    Epoch[3/10], Step [380/469], Reconst Loss: 93.1045, KL Div: 25.0688, kl_div_c: 1.6801, Entropy: 0.6225
    Epoch[3/10], Step [390/469], Reconst Loss: 91.2828, KL Div: 25.7013, kl_div_c: 1.6063, Entropy: 0.6963
    Epoch[3/10], Step [400/469], Reconst Loss: 91.1644, KL Div: 25.1444, kl_div_c: 1.5898, Entropy: 0.7128
    Epoch[3/10], Step [410/469], Reconst Loss: 90.2725, KL Div: 25.4445, kl_div_c: 1.6447, Entropy: 0.6579
    Epoch[3/10], Step [420/469], Reconst Loss: 89.1491, KL Div: 25.2842, kl_div_c: 1.6439, Entropy: 0.6587
    Epoch[3/10], Step [430/469], Reconst Loss: 85.6870, KL Div: 24.9598, kl_div_c: 1.6399, Entropy: 0.6627
    Epoch[3/10], Step [440/469], Reconst Loss: 89.7748, KL Div: 26.5693, kl_div_c: 1.5629, Entropy: 0.7397
    Epoch[3/10], Step [450/469], Reconst Loss: 88.4643, KL Div: 24.5542, kl_div_c: 1.5726, Entropy: 0.7300
    Epoch[3/10], Step [460/469], Reconst Loss: 85.5936, KL Div: 25.4856, kl_div_c: 1.5527, Entropy: 0.7499
    Epoch[4/10], Step [10/469], Reconst Loss: 85.4612, KL Div: 24.2264, kl_div_c: 1.5900, Entropy: 0.7125
    Epoch[4/10], Step [20/469], Reconst Loss: 91.4314, KL Div: 25.6450, kl_div_c: 1.6058, Entropy: 0.6968
    Epoch[4/10], Step [30/469], Reconst Loss: 91.6349, KL Div: 25.5752, kl_div_c: 1.6520, Entropy: 0.6505
    Epoch[4/10], Step [40/469], Reconst Loss: 88.1954, KL Div: 25.6376, kl_div_c: 1.6594, Entropy: 0.6432
    Epoch[4/10], Step [50/469], Reconst Loss: 87.5905, KL Div: 25.9057, kl_div_c: 1.5943, Entropy: 0.7083
    Epoch[4/10], Step [60/469], Reconst Loss: 89.7926, KL Div: 25.2104, kl_div_c: 1.6271, Entropy: 0.6755
    Epoch[4/10], Step [70/469], Reconst Loss: 84.0165, KL Div: 24.5996, kl_div_c: 1.6290, Entropy: 0.6736
    Epoch[4/10], Step [80/469], Reconst Loss: 90.4523, KL Div: 25.6365, kl_div_c: 1.6178, Entropy: 0.6848
    Epoch[4/10], Step [90/469], Reconst Loss: 90.9932, KL Div: 25.9107, kl_div_c: 1.5751, Entropy: 0.7275
    Epoch[4/10], Step [100/469], Reconst Loss: 88.6991, KL Div: 25.4853, kl_div_c: 1.4740, Entropy: 0.8286
    Epoch[4/10], Step [110/469], Reconst Loss: 89.4764, KL Div: 25.5911, kl_div_c: 1.5905, Entropy: 0.7121
    Epoch[4/10], Step [120/469], Reconst Loss: 87.5611, KL Div: 25.6410, kl_div_c: 1.6411, Entropy: 0.6615
    Epoch[4/10], Step [130/469], Reconst Loss: 88.1394, KL Div: 25.9691, kl_div_c: 1.5637, Entropy: 0.7388
    Epoch[4/10], Step [140/469], Reconst Loss: 86.4929, KL Div: 25.5667, kl_div_c: 1.6278, Entropy: 0.6747
    Epoch[4/10], Step [150/469], Reconst Loss: 87.9040, KL Div: 25.4843, kl_div_c: 1.6494, Entropy: 0.6532
    Epoch[4/10], Step [160/469], Reconst Loss: 87.6536, KL Div: 25.4408, kl_div_c: 1.6227, Entropy: 0.6799
    Epoch[4/10], Step [170/469], Reconst Loss: 88.1137, KL Div: 25.8815, kl_div_c: 1.5516, Entropy: 0.7510
    Epoch[4/10], Step [180/469], Reconst Loss: 85.2637, KL Div: 25.2117, kl_div_c: 1.6467, Entropy: 0.6559
    Epoch[4/10], Step [190/469], Reconst Loss: 87.6505, KL Div: 26.4855, kl_div_c: 1.5304, Entropy: 0.7721
    Epoch[4/10], Step [200/469], Reconst Loss: 90.2454, KL Div: 25.3498, kl_div_c: 1.5335, Entropy: 0.7691
    Epoch[4/10], Step [210/469], Reconst Loss: 87.1921, KL Div: 26.7188, kl_div_c: 1.5606, Entropy: 0.7420
    Epoch[4/10], Step [220/469], Reconst Loss: 87.6114, KL Div: 25.4604, kl_div_c: 1.6348, Entropy: 0.6677
    Epoch[4/10], Step [230/469], Reconst Loss: 88.9066, KL Div: 25.5588, kl_div_c: 1.5158, Entropy: 0.7867
    Epoch[4/10], Step [240/469], Reconst Loss: 90.0244, KL Div: 26.0645, kl_div_c: 1.6203, Entropy: 0.6823
    Epoch[4/10], Step [250/469], Reconst Loss: 87.9486, KL Div: 25.8408, kl_div_c: 1.5021, Entropy: 0.8005
    Epoch[4/10], Step [260/469], Reconst Loss: 85.3125, KL Div: 24.9394, kl_div_c: 1.5895, Entropy: 0.7131
    Epoch[4/10], Step [270/469], Reconst Loss: 83.9700, KL Div: 26.6745, kl_div_c: 1.5101, Entropy: 0.7925
    Epoch[4/10], Step [280/469], Reconst Loss: 84.8126, KL Div: 24.6888, kl_div_c: 1.6225, Entropy: 0.6800
    Epoch[4/10], Step [290/469], Reconst Loss: 88.3761, KL Div: 26.2917, kl_div_c: 1.5825, Entropy: 0.7200
    Epoch[4/10], Step [300/469], Reconst Loss: 90.2729, KL Div: 26.6937, kl_div_c: 1.5880, Entropy: 0.7145
    Epoch[4/10], Step [310/469], Reconst Loss: 83.3308, KL Div: 25.1734, kl_div_c: 1.5839, Entropy: 0.7187
    Epoch[4/10], Step [320/469], Reconst Loss: 85.9036, KL Div: 24.7941, kl_div_c: 1.5232, Entropy: 0.7794
    Epoch[4/10], Step [330/469], Reconst Loss: 84.9798, KL Div: 25.7846, kl_div_c: 1.5356, Entropy: 0.7670
    Epoch[4/10], Step [340/469], Reconst Loss: 85.9315, KL Div: 25.4394, kl_div_c: 1.6328, Entropy: 0.6698
    Epoch[4/10], Step [350/469], Reconst Loss: 92.3363, KL Div: 25.8579, kl_div_c: 1.5251, Entropy: 0.7775
    Epoch[4/10], Step [360/469], Reconst Loss: 89.0324, KL Div: 25.3299, kl_div_c: 1.5181, Entropy: 0.7845
    Epoch[4/10], Step [370/469], Reconst Loss: 84.8704, KL Div: 25.7173, kl_div_c: 1.5526, Entropy: 0.7500
    Epoch[4/10], Step [380/469], Reconst Loss: 87.1488, KL Div: 26.7133, kl_div_c: 1.5829, Entropy: 0.7197
    Epoch[4/10], Step [390/469], Reconst Loss: 83.5699, KL Div: 25.4502, kl_div_c: 1.5310, Entropy: 0.7716
    Epoch[4/10], Step [400/469], Reconst Loss: 85.5685, KL Div: 25.6921, kl_div_c: 1.5313, Entropy: 0.7712
    Epoch[4/10], Step [410/469], Reconst Loss: 89.9600, KL Div: 26.0595, kl_div_c: 1.6181, Entropy: 0.6845
    Epoch[4/10], Step [420/469], Reconst Loss: 87.6755, KL Div: 26.2476, kl_div_c: 1.6245, Entropy: 0.6781
    Epoch[4/10], Step [430/469], Reconst Loss: 85.9530, KL Div: 26.0025, kl_div_c: 1.6386, Entropy: 0.6640
    Epoch[4/10], Step [440/469], Reconst Loss: 87.6889, KL Div: 24.7977, kl_div_c: 1.6027, Entropy: 0.6999
    Epoch[4/10], Step [450/469], Reconst Loss: 85.2331, KL Div: 26.5771, kl_div_c: 1.5237, Entropy: 0.7789
    Epoch[4/10], Step [460/469], Reconst Loss: 86.2621, KL Div: 24.0178, kl_div_c: 1.5750, Entropy: 0.7276
    Epoch[5/10], Step [10/469], Reconst Loss: 83.8076, KL Div: 24.9905, kl_div_c: 1.6454, Entropy: 0.6572
    Epoch[5/10], Step [20/469], Reconst Loss: 88.4694, KL Div: 26.3278, kl_div_c: 1.5981, Entropy: 0.7045
    Epoch[5/10], Step [30/469], Reconst Loss: 89.7541, KL Div: 26.1990, kl_div_c: 1.6313, Entropy: 0.6713
    Epoch[5/10], Step [40/469], Reconst Loss: 85.6198, KL Div: 25.4353, kl_div_c: 1.5980, Entropy: 0.7046
    Epoch[5/10], Step [50/469], Reconst Loss: 82.8976, KL Div: 25.2545, kl_div_c: 1.4755, Entropy: 0.8270
    Epoch[5/10], Step [60/469], Reconst Loss: 89.5831, KL Div: 25.8525, kl_div_c: 1.5784, Entropy: 0.7242
    Epoch[5/10], Step [70/469], Reconst Loss: 89.6556, KL Div: 26.3569, kl_div_c: 1.5919, Entropy: 0.7107
    Epoch[5/10], Step [80/469], Reconst Loss: 79.9631, KL Div: 24.9534, kl_div_c: 1.6210, Entropy: 0.6816
    Epoch[5/10], Step [90/469], Reconst Loss: 85.6872, KL Div: 25.7741, kl_div_c: 1.6776, Entropy: 0.6250
    Epoch[5/10], Step [100/469], Reconst Loss: 87.7975, KL Div: 26.3043, kl_div_c: 1.6743, Entropy: 0.6283
    Epoch[5/10], Step [110/469], Reconst Loss: 85.1242, KL Div: 26.2697, kl_div_c: 1.5601, Entropy: 0.7425
    Epoch[5/10], Step [120/469], Reconst Loss: 86.3182, KL Div: 26.2607, kl_div_c: 1.5787, Entropy: 0.7239
    Epoch[5/10], Step [130/469], Reconst Loss: 84.9939, KL Div: 25.0902, kl_div_c: 1.6313, Entropy: 0.6712
    Epoch[5/10], Step [140/469], Reconst Loss: 84.6050, KL Div: 26.0249, kl_div_c: 1.6514, Entropy: 0.6512
    Epoch[5/10], Step [150/469], Reconst Loss: 90.8246, KL Div: 25.0272, kl_div_c: 1.5792, Entropy: 0.7234
    Epoch[5/10], Step [160/469], Reconst Loss: 83.3074, KL Div: 25.5134, kl_div_c: 1.5912, Entropy: 0.7114
    Epoch[5/10], Step [170/469], Reconst Loss: 84.1351, KL Div: 25.5811, kl_div_c: 1.6575, Entropy: 0.6451
    Epoch[5/10], Step [180/469], Reconst Loss: 83.2642, KL Div: 25.5095, kl_div_c: 1.5549, Entropy: 0.7477
    Epoch[5/10], Step [190/469], Reconst Loss: 82.8225, KL Div: 25.9724, kl_div_c: 1.6320, Entropy: 0.6706
    Epoch[5/10], Step [200/469], Reconst Loss: 87.1048, KL Div: 26.3800, kl_div_c: 1.5979, Entropy: 0.7047
    Epoch[5/10], Step [210/469], Reconst Loss: 84.3079, KL Div: 25.9184, kl_div_c: 1.6832, Entropy: 0.6194
    Epoch[5/10], Step [220/469], Reconst Loss: 86.3511, KL Div: 25.4415, kl_div_c: 1.5915, Entropy: 0.7111
    Epoch[5/10], Step [230/469], Reconst Loss: 88.6996, KL Div: 25.7616, kl_div_c: 1.5835, Entropy: 0.7191
    Epoch[5/10], Step [240/469], Reconst Loss: 81.8327, KL Div: 26.5649, kl_div_c: 1.6060, Entropy: 0.6965
    Epoch[5/10], Step [250/469], Reconst Loss: 84.1026, KL Div: 25.9128, kl_div_c: 1.6432, Entropy: 0.6594
    Epoch[5/10], Step [260/469], Reconst Loss: 86.6073, KL Div: 25.9840, kl_div_c: 1.6459, Entropy: 0.6567
    Epoch[5/10], Step [270/469], Reconst Loss: 84.0389, KL Div: 24.9473, kl_div_c: 1.5879, Entropy: 0.7147
    Epoch[5/10], Step [280/469], Reconst Loss: 84.3521, KL Div: 26.5918, kl_div_c: 1.6707, Entropy: 0.6319
    Epoch[5/10], Step [290/469], Reconst Loss: 83.9224, KL Div: 25.5627, kl_div_c: 1.5521, Entropy: 0.7504
    Epoch[5/10], Step [300/469], Reconst Loss: 88.0452, KL Div: 26.3488, kl_div_c: 1.7081, Entropy: 0.5945
    Epoch[5/10], Step [310/469], Reconst Loss: 83.2952, KL Div: 25.2576, kl_div_c: 1.6057, Entropy: 0.6969
    Epoch[5/10], Step [320/469], Reconst Loss: 88.1100, KL Div: 26.6168, kl_div_c: 1.6439, Entropy: 0.6587
    Epoch[5/10], Step [330/469], Reconst Loss: 83.5364, KL Div: 25.5744, kl_div_c: 1.6594, Entropy: 0.6432
    Epoch[5/10], Step [340/469], Reconst Loss: 85.5254, KL Div: 26.9645, kl_div_c: 1.7191, Entropy: 0.5835
    Epoch[5/10], Step [350/469], Reconst Loss: 80.8250, KL Div: 25.1961, kl_div_c: 1.6721, Entropy: 0.6305
    Epoch[5/10], Step [360/469], Reconst Loss: 85.1733, KL Div: 25.8953, kl_div_c: 1.6395, Entropy: 0.6631
    Epoch[5/10], Step [370/469], Reconst Loss: 86.0402, KL Div: 26.2907, kl_div_c: 1.6680, Entropy: 0.6345
    Epoch[5/10], Step [380/469], Reconst Loss: 87.1191, KL Div: 26.0947, kl_div_c: 1.6460, Entropy: 0.6565
    Epoch[5/10], Step [390/469], Reconst Loss: 87.0195, KL Div: 25.6757, kl_div_c: 1.6005, Entropy: 0.7021
    Epoch[5/10], Step [400/469], Reconst Loss: 82.5659, KL Div: 25.8820, kl_div_c: 1.5939, Entropy: 0.7087
    Epoch[5/10], Step [410/469], Reconst Loss: 86.8784, KL Div: 25.3114, kl_div_c: 1.6558, Entropy: 0.6468
    Epoch[5/10], Step [420/469], Reconst Loss: 82.1487, KL Div: 26.0710, kl_div_c: 1.4662, Entropy: 0.8364
    Epoch[5/10], Step [430/469], Reconst Loss: 88.5582, KL Div: 26.0300, kl_div_c: 1.6319, Entropy: 0.6707
    Epoch[5/10], Step [440/469], Reconst Loss: 85.6926, KL Div: 26.1543, kl_div_c: 1.6275, Entropy: 0.6751
    Epoch[5/10], Step [450/469], Reconst Loss: 87.3347, KL Div: 25.2817, kl_div_c: 1.6147, Entropy: 0.6879
    Epoch[5/10], Step [460/469], Reconst Loss: 83.4598, KL Div: 25.4829, kl_div_c: 1.6446, Entropy: 0.6580
    Epoch[6/10], Step [10/469], Reconst Loss: 84.2995, KL Div: 26.5021, kl_div_c: 1.6434, Entropy: 0.6591
    Epoch[6/10], Step [20/469], Reconst Loss: 85.4725, KL Div: 26.3706, kl_div_c: 1.6513, Entropy: 0.6512
    Epoch[6/10], Step [30/469], Reconst Loss: 86.7407, KL Div: 26.3403, kl_div_c: 1.6973, Entropy: 0.6053
    Epoch[6/10], Step [40/469], Reconst Loss: 85.2834, KL Div: 26.2768, kl_div_c: 1.6576, Entropy: 0.6450
    Epoch[6/10], Step [50/469], Reconst Loss: 84.1891, KL Div: 25.5753, kl_div_c: 1.6249, Entropy: 0.6777
    Epoch[6/10], Step [60/469], Reconst Loss: 84.8539, KL Div: 25.9951, kl_div_c: 1.6482, Entropy: 0.6544
    Epoch[6/10], Step [70/469], Reconst Loss: 82.6947, KL Div: 25.8271, kl_div_c: 1.5503, Entropy: 0.7523
    Epoch[6/10], Step [80/469], Reconst Loss: 79.8114, KL Div: 25.0429, kl_div_c: 1.6821, Entropy: 0.6204
    Epoch[6/10], Step [90/469], Reconst Loss: 81.1008, KL Div: 26.5265, kl_div_c: 1.6043, Entropy: 0.6983
    Epoch[6/10], Step [100/469], Reconst Loss: 79.9116, KL Div: 25.9542, kl_div_c: 1.6591, Entropy: 0.6434
    Epoch[6/10], Step [110/469], Reconst Loss: 83.4840, KL Div: 25.7516, kl_div_c: 1.6144, Entropy: 0.6881
    Epoch[6/10], Step [120/469], Reconst Loss: 89.1966, KL Div: 26.5799, kl_div_c: 1.6396, Entropy: 0.6630
    Epoch[6/10], Step [130/469], Reconst Loss: 87.0957, KL Div: 26.6882, kl_div_c: 1.6399, Entropy: 0.6626
    Epoch[6/10], Step [140/469], Reconst Loss: 84.6365, KL Div: 26.5954, kl_div_c: 1.6177, Entropy: 0.6849
    Epoch[6/10], Step [150/469], Reconst Loss: 86.4093, KL Div: 25.9136, kl_div_c: 1.7023, Entropy: 0.6003
    Epoch[6/10], Step [160/469], Reconst Loss: 84.3710, KL Div: 26.3856, kl_div_c: 1.6617, Entropy: 0.6409
    Epoch[6/10], Step [170/469], Reconst Loss: 80.7053, KL Div: 25.6782, kl_div_c: 1.6058, Entropy: 0.6968
    Epoch[6/10], Step [180/469], Reconst Loss: 84.8329, KL Div: 26.0547, kl_div_c: 1.6769, Entropy: 0.6257
    Epoch[6/10], Step [190/469], Reconst Loss: 85.6038, KL Div: 25.8590, kl_div_c: 1.6467, Entropy: 0.6559
    Epoch[6/10], Step [200/469], Reconst Loss: 82.1713, KL Div: 25.7958, kl_div_c: 1.6271, Entropy: 0.6755
    Epoch[6/10], Step [210/469], Reconst Loss: 85.0586, KL Div: 26.4007, kl_div_c: 1.5949, Entropy: 0.7077
    Epoch[6/10], Step [220/469], Reconst Loss: 80.6312, KL Div: 25.2151, kl_div_c: 1.5656, Entropy: 0.7370
    Epoch[6/10], Step [230/469], Reconst Loss: 80.9638, KL Div: 25.2720, kl_div_c: 1.6984, Entropy: 0.6041
    Epoch[6/10], Step [240/469], Reconst Loss: 84.1463, KL Div: 26.3452, kl_div_c: 1.6539, Entropy: 0.6486
    Epoch[6/10], Step [250/469], Reconst Loss: 83.5393, KL Div: 25.1596, kl_div_c: 1.7295, Entropy: 0.5731
    Epoch[6/10], Step [260/469], Reconst Loss: 85.8471, KL Div: 26.7077, kl_div_c: 1.5891, Entropy: 0.7135
    Epoch[6/10], Step [270/469], Reconst Loss: 85.4695, KL Div: 25.5895, kl_div_c: 1.6585, Entropy: 0.6441
    Epoch[6/10], Step [280/469], Reconst Loss: 80.6780, KL Div: 26.1678, kl_div_c: 1.6875, Entropy: 0.6151
    Epoch[6/10], Step [290/469], Reconst Loss: 83.9970, KL Div: 25.5167, kl_div_c: 1.7067, Entropy: 0.5959
    Epoch[6/10], Step [300/469], Reconst Loss: 84.4932, KL Div: 26.1193, kl_div_c: 1.6655, Entropy: 0.6371
    Epoch[6/10], Step [310/469], Reconst Loss: 84.9242, KL Div: 25.6061, kl_div_c: 1.7112, Entropy: 0.5914
    Epoch[6/10], Step [320/469], Reconst Loss: 82.1243, KL Div: 26.4435, kl_div_c: 1.7520, Entropy: 0.5506
    Epoch[6/10], Step [330/469], Reconst Loss: 81.8684, KL Div: 25.7239, kl_div_c: 1.5978, Entropy: 0.7048
    Epoch[6/10], Step [340/469], Reconst Loss: 86.9229, KL Div: 25.9360, kl_div_c: 1.6484, Entropy: 0.6542
    Epoch[6/10], Step [350/469], Reconst Loss: 81.1840, KL Div: 24.7460, kl_div_c: 1.6745, Entropy: 0.6281
    Epoch[6/10], Step [360/469], Reconst Loss: 79.5844, KL Div: 24.9408, kl_div_c: 1.6552, Entropy: 0.6474
    Epoch[6/10], Step [370/469], Reconst Loss: 85.0625, KL Div: 26.1047, kl_div_c: 1.6938, Entropy: 0.6088
    Epoch[6/10], Step [380/469], Reconst Loss: 82.0092, KL Div: 25.9942, kl_div_c: 1.6253, Entropy: 0.6772
    Epoch[6/10], Step [390/469], Reconst Loss: 83.4455, KL Div: 25.9845, kl_div_c: 1.6021, Entropy: 0.7005
    Epoch[6/10], Step [400/469], Reconst Loss: 85.9179, KL Div: 26.0679, kl_div_c: 1.6539, Entropy: 0.6487
    Epoch[6/10], Step [410/469], Reconst Loss: 84.3264, KL Div: 26.6504, kl_div_c: 1.7019, Entropy: 0.6007
    Epoch[6/10], Step [420/469], Reconst Loss: 80.2802, KL Div: 24.7591, kl_div_c: 1.6648, Entropy: 0.6378
    Epoch[6/10], Step [430/469], Reconst Loss: 86.0216, KL Div: 26.1775, kl_div_c: 1.6102, Entropy: 0.6924
    Epoch[6/10], Step [440/469], Reconst Loss: 85.2343, KL Div: 26.2527, kl_div_c: 1.6014, Entropy: 0.7012
    Epoch[6/10], Step [450/469], Reconst Loss: 87.1116, KL Div: 26.5017, kl_div_c: 1.6941, Entropy: 0.6085
    Epoch[6/10], Step [460/469], Reconst Loss: 81.7077, KL Div: 24.8645, kl_div_c: 1.6520, Entropy: 0.6506
    Epoch[7/10], Step [10/469], Reconst Loss: 84.6185, KL Div: 25.0932, kl_div_c: 1.6786, Entropy: 0.6239
    Epoch[7/10], Step [20/469], Reconst Loss: 85.0459, KL Div: 26.5278, kl_div_c: 1.6767, Entropy: 0.6259
    Epoch[7/10], Step [30/469], Reconst Loss: 85.1646, KL Div: 25.9958, kl_div_c: 1.7304, Entropy: 0.5722
    Epoch[7/10], Step [40/469], Reconst Loss: 85.6791, KL Div: 25.9037, kl_div_c: 1.7250, Entropy: 0.5775
    Epoch[7/10], Step [50/469], Reconst Loss: 82.3978, KL Div: 26.4632, kl_div_c: 1.6739, Entropy: 0.6287
    Epoch[7/10], Step [60/469], Reconst Loss: 83.6100, KL Div: 25.9473, kl_div_c: 1.6511, Entropy: 0.6515
    Epoch[7/10], Step [70/469], Reconst Loss: 80.3013, KL Div: 26.7978, kl_div_c: 1.6916, Entropy: 0.6110
    Epoch[7/10], Step [80/469], Reconst Loss: 82.5301, KL Div: 25.6932, kl_div_c: 1.6320, Entropy: 0.6706
    Epoch[7/10], Step [90/469], Reconst Loss: 81.4338, KL Div: 25.7387, kl_div_c: 1.6373, Entropy: 0.6653
    Epoch[7/10], Step [100/469], Reconst Loss: 80.8819, KL Div: 26.1689, kl_div_c: 1.6987, Entropy: 0.6039
    Epoch[7/10], Step [110/469], Reconst Loss: 82.6190, KL Div: 26.3763, kl_div_c: 1.6997, Entropy: 0.6029
    Epoch[7/10], Step [120/469], Reconst Loss: 82.2106, KL Div: 25.2348, kl_div_c: 1.7157, Entropy: 0.5869
    Epoch[7/10], Step [130/469], Reconst Loss: 81.4868, KL Div: 25.9647, kl_div_c: 1.7168, Entropy: 0.5858
    Epoch[7/10], Step [140/469], Reconst Loss: 80.5667, KL Div: 26.4811, kl_div_c: 1.7276, Entropy: 0.5750
    Epoch[7/10], Step [150/469], Reconst Loss: 82.8455, KL Div: 25.7099, kl_div_c: 1.6468, Entropy: 0.6558
    Epoch[7/10], Step [160/469], Reconst Loss: 83.8200, KL Div: 25.7042, kl_div_c: 1.7400, Entropy: 0.5626
    Epoch[7/10], Step [170/469], Reconst Loss: 84.1828, KL Div: 25.8847, kl_div_c: 1.6454, Entropy: 0.6572
    Epoch[7/10], Step [180/469], Reconst Loss: 86.0885, KL Div: 26.4862, kl_div_c: 1.7400, Entropy: 0.5625
    Epoch[7/10], Step [190/469], Reconst Loss: 80.9738, KL Div: 25.6855, kl_div_c: 1.6623, Entropy: 0.6403
    Epoch[7/10], Step [200/469], Reconst Loss: 85.8181, KL Div: 27.0355, kl_div_c: 1.7118, Entropy: 0.5908
    Epoch[7/10], Step [210/469], Reconst Loss: 75.3472, KL Div: 25.3264, kl_div_c: 1.7415, Entropy: 0.5611
    Epoch[7/10], Step [220/469], Reconst Loss: 82.5348, KL Div: 25.9616, kl_div_c: 1.6847, Entropy: 0.6179
    Epoch[7/10], Step [230/469], Reconst Loss: 81.7264, KL Div: 26.0912, kl_div_c: 1.6384, Entropy: 0.6642
    Epoch[7/10], Step [240/469], Reconst Loss: 81.2147, KL Div: 25.4997, kl_div_c: 1.7082, Entropy: 0.5943
    Epoch[7/10], Step [250/469], Reconst Loss: 83.8901, KL Div: 26.2477, kl_div_c: 1.7594, Entropy: 0.5432
    Epoch[7/10], Step [260/469], Reconst Loss: 83.0348, KL Div: 26.1412, kl_div_c: 1.7241, Entropy: 0.5785
    Epoch[7/10], Step [270/469], Reconst Loss: 79.5512, KL Div: 25.6287, kl_div_c: 1.7332, Entropy: 0.5694
    Epoch[7/10], Step [280/469], Reconst Loss: 80.8483, KL Div: 26.4707, kl_div_c: 1.7368, Entropy: 0.5658
    Epoch[7/10], Step [290/469], Reconst Loss: 86.9592, KL Div: 25.6225, kl_div_c: 1.7114, Entropy: 0.5912
    Epoch[7/10], Step [300/469], Reconst Loss: 82.5302, KL Div: 26.2335, kl_div_c: 1.6660, Entropy: 0.6366
    Epoch[7/10], Step [310/469], Reconst Loss: 80.9222, KL Div: 25.5699, kl_div_c: 1.6432, Entropy: 0.6594
    Epoch[7/10], Step [320/469], Reconst Loss: 81.6264, KL Div: 25.6922, kl_div_c: 1.7852, Entropy: 0.5174
    Epoch[7/10], Step [330/469], Reconst Loss: 83.3576, KL Div: 25.6584, kl_div_c: 1.6231, Entropy: 0.6795
    Epoch[7/10], Step [340/469], Reconst Loss: 84.1464, KL Div: 26.0233, kl_div_c: 1.7334, Entropy: 0.5692
    Epoch[7/10], Step [350/469], Reconst Loss: 82.3435, KL Div: 26.0191, kl_div_c: 1.7281, Entropy: 0.5745
    Epoch[7/10], Step [360/469], Reconst Loss: 80.0016, KL Div: 25.5599, kl_div_c: 1.7572, Entropy: 0.5453
    Epoch[7/10], Step [370/469], Reconst Loss: 80.9443, KL Div: 26.5330, kl_div_c: 1.7498, Entropy: 0.5527
    Epoch[7/10], Step [380/469], Reconst Loss: 82.1880, KL Div: 25.1236, kl_div_c: 1.7476, Entropy: 0.5549
    Epoch[7/10], Step [390/469], Reconst Loss: 85.8801, KL Div: 26.4168, kl_div_c: 1.6629, Entropy: 0.6397
    Epoch[7/10], Step [400/469], Reconst Loss: 83.6929, KL Div: 26.0389, kl_div_c: 1.6417, Entropy: 0.6609
    Epoch[7/10], Step [410/469], Reconst Loss: 81.3813, KL Div: 25.8464, kl_div_c: 1.6657, Entropy: 0.6369
    Epoch[7/10], Step [420/469], Reconst Loss: 83.2146, KL Div: 26.7198, kl_div_c: 1.7101, Entropy: 0.5925
    Epoch[7/10], Step [430/469], Reconst Loss: 80.2909, KL Div: 26.1419, kl_div_c: 1.7326, Entropy: 0.5700
    Epoch[7/10], Step [440/469], Reconst Loss: 81.4065, KL Div: 25.8167, kl_div_c: 1.6815, Entropy: 0.6211
    Epoch[7/10], Step [450/469], Reconst Loss: 86.1867, KL Div: 26.5322, kl_div_c: 1.7012, Entropy: 0.6014
    Epoch[7/10], Step [460/469], Reconst Loss: 81.0252, KL Div: 26.4906, kl_div_c: 1.7359, Entropy: 0.5667
    Epoch[8/10], Step [10/469], Reconst Loss: 79.3101, KL Div: 25.7539, kl_div_c: 1.6518, Entropy: 0.6508
    Epoch[8/10], Step [20/469], Reconst Loss: 86.0751, KL Div: 26.0924, kl_div_c: 1.7144, Entropy: 0.5882
    Epoch[8/10], Step [30/469], Reconst Loss: 81.2953, KL Div: 25.9641, kl_div_c: 1.6953, Entropy: 0.6073
    Epoch[8/10], Step [40/469], Reconst Loss: 84.7822, KL Div: 26.2986, kl_div_c: 1.6304, Entropy: 0.6722
    Epoch[8/10], Step [50/469], Reconst Loss: 81.8291, KL Div: 26.4819, kl_div_c: 1.6079, Entropy: 0.6946
    Epoch[8/10], Step [60/469], Reconst Loss: 86.2114, KL Div: 26.5420, kl_div_c: 1.6909, Entropy: 0.6117
    Epoch[8/10], Step [70/469], Reconst Loss: 83.8712, KL Div: 26.1399, kl_div_c: 1.7262, Entropy: 0.5764
    Epoch[8/10], Step [80/469], Reconst Loss: 83.1733, KL Div: 25.9746, kl_div_c: 1.8151, Entropy: 0.4875
    Epoch[8/10], Step [90/469], Reconst Loss: 80.7502, KL Div: 26.0485, kl_div_c: 1.6774, Entropy: 0.6252
    Epoch[8/10], Step [100/469], Reconst Loss: 80.9986, KL Div: 26.3387, kl_div_c: 1.7980, Entropy: 0.5046
    Epoch[8/10], Step [110/469], Reconst Loss: 80.1805, KL Div: 24.8531, kl_div_c: 1.6779, Entropy: 0.6247
    Epoch[8/10], Step [120/469], Reconst Loss: 85.3226, KL Div: 26.7899, kl_div_c: 1.7368, Entropy: 0.5658
    Epoch[8/10], Step [130/469], Reconst Loss: 79.3371, KL Div: 25.5619, kl_div_c: 1.6435, Entropy: 0.6591
    Epoch[8/10], Step [140/469], Reconst Loss: 79.6524, KL Div: 26.1021, kl_div_c: 1.7033, Entropy: 0.5993
    Epoch[8/10], Step [150/469], Reconst Loss: 86.5080, KL Div: 26.0309, kl_div_c: 1.7348, Entropy: 0.5677
    Epoch[8/10], Step [160/469], Reconst Loss: 83.3129, KL Div: 26.6608, kl_div_c: 1.7745, Entropy: 0.5281
    Epoch[8/10], Step [170/469], Reconst Loss: 76.5122, KL Div: 25.7232, kl_div_c: 1.5946, Entropy: 0.7079
    Epoch[8/10], Step [180/469], Reconst Loss: 82.0454, KL Div: 25.7378, kl_div_c: 1.6367, Entropy: 0.6659
    Epoch[8/10], Step [190/469], Reconst Loss: 76.4986, KL Div: 25.8716, kl_div_c: 1.7412, Entropy: 0.5613
    Epoch[8/10], Step [200/469], Reconst Loss: 86.4439, KL Div: 27.0891, kl_div_c: 1.8121, Entropy: 0.4905
    Epoch[8/10], Step [210/469], Reconst Loss: 79.2952, KL Div: 25.6944, kl_div_c: 1.7840, Entropy: 0.5186
    Epoch[8/10], Step [220/469], Reconst Loss: 81.5079, KL Div: 26.0541, kl_div_c: 1.7310, Entropy: 0.5716
    Epoch[8/10], Step [230/469], Reconst Loss: 82.5948, KL Div: 26.1945, kl_div_c: 1.7048, Entropy: 0.5978
    Epoch[8/10], Step [240/469], Reconst Loss: 79.4035, KL Div: 25.5750, kl_div_c: 1.7273, Entropy: 0.5753
    Epoch[8/10], Step [250/469], Reconst Loss: 79.4874, KL Div: 26.8456, kl_div_c: 1.6714, Entropy: 0.6311
    Epoch[8/10], Step [260/469], Reconst Loss: 81.6080, KL Div: 26.4295, kl_div_c: 1.7677, Entropy: 0.5348
    Epoch[8/10], Step [270/469], Reconst Loss: 83.3624, KL Div: 26.6378, kl_div_c: 1.7124, Entropy: 0.5901
    Epoch[8/10], Step [280/469], Reconst Loss: 84.6972, KL Div: 26.9349, kl_div_c: 1.7427, Entropy: 0.5599
    Epoch[8/10], Step [290/469], Reconst Loss: 81.8299, KL Div: 25.9874, kl_div_c: 1.6654, Entropy: 0.6372
    Epoch[8/10], Step [300/469], Reconst Loss: 81.2409, KL Div: 25.4666, kl_div_c: 1.6783, Entropy: 0.6243
    Epoch[8/10], Step [310/469], Reconst Loss: 80.5840, KL Div: 25.4804, kl_div_c: 1.7055, Entropy: 0.5971
    Epoch[8/10], Step [320/469], Reconst Loss: 87.5064, KL Div: 26.5571, kl_div_c: 1.7485, Entropy: 0.5541
    Epoch[8/10], Step [330/469], Reconst Loss: 82.2105, KL Div: 26.3070, kl_div_c: 1.7609, Entropy: 0.5417
    Epoch[8/10], Step [340/469], Reconst Loss: 82.8404, KL Div: 25.1093, kl_div_c: 1.6661, Entropy: 0.6365
    Epoch[8/10], Step [350/469], Reconst Loss: 82.4144, KL Div: 26.6468, kl_div_c: 1.7463, Entropy: 0.5563
    Epoch[8/10], Step [360/469], Reconst Loss: 82.1274, KL Div: 25.4358, kl_div_c: 1.6566, Entropy: 0.6460
    Epoch[8/10], Step [370/469], Reconst Loss: 83.9332, KL Div: 25.9957, kl_div_c: 1.6225, Entropy: 0.6801
    Epoch[8/10], Step [380/469], Reconst Loss: 81.1490, KL Div: 26.5095, kl_div_c: 1.6702, Entropy: 0.6324
    Epoch[8/10], Step [390/469], Reconst Loss: 78.9934, KL Div: 26.2309, kl_div_c: 1.7265, Entropy: 0.5761
    Epoch[8/10], Step [400/469], Reconst Loss: 83.8099, KL Div: 26.3778, kl_div_c: 1.7452, Entropy: 0.5573
    Epoch[8/10], Step [410/469], Reconst Loss: 82.3000, KL Div: 27.2222, kl_div_c: 1.6744, Entropy: 0.6282
    Epoch[8/10], Step [420/469], Reconst Loss: 83.0790, KL Div: 26.7113, kl_div_c: 1.7106, Entropy: 0.5919
    Epoch[8/10], Step [430/469], Reconst Loss: 82.7640, KL Div: 26.0481, kl_div_c: 1.7591, Entropy: 0.5435
    Epoch[8/10], Step [440/469], Reconst Loss: 80.7136, KL Div: 26.0856, kl_div_c: 1.6572, Entropy: 0.6454
    Epoch[8/10], Step [450/469], Reconst Loss: 81.3454, KL Div: 26.5779, kl_div_c: 1.6986, Entropy: 0.6040
    Epoch[8/10], Step [460/469], Reconst Loss: 83.2538, KL Div: 25.7245, kl_div_c: 1.8209, Entropy: 0.4817
    Epoch[9/10], Step [10/469], Reconst Loss: 84.3277, KL Div: 26.6217, kl_div_c: 1.7494, Entropy: 0.5531
    Epoch[9/10], Step [20/469], Reconst Loss: 79.8514, KL Div: 25.4132, kl_div_c: 1.7513, Entropy: 0.5513
    Epoch[9/10], Step [30/469], Reconst Loss: 79.2233, KL Div: 26.1368, kl_div_c: 1.7038, Entropy: 0.5988
    Epoch[9/10], Step [40/469], Reconst Loss: 80.3272, KL Div: 26.2572, kl_div_c: 1.7408, Entropy: 0.5618
    Epoch[9/10], Step [50/469], Reconst Loss: 76.8585, KL Div: 25.2572, kl_div_c: 1.7008, Entropy: 0.6018
    Epoch[9/10], Step [60/469], Reconst Loss: 83.1481, KL Div: 26.2593, kl_div_c: 1.7117, Entropy: 0.5909
    Epoch[9/10], Step [70/469], Reconst Loss: 79.9600, KL Div: 25.4298, kl_div_c: 1.6990, Entropy: 0.6036
    Epoch[9/10], Step [80/469], Reconst Loss: 83.3037, KL Div: 25.6380, kl_div_c: 1.7301, Entropy: 0.5725
    Epoch[9/10], Step [90/469], Reconst Loss: 77.7883, KL Div: 26.6014, kl_div_c: 1.7511, Entropy: 0.5515
    Epoch[9/10], Step [100/469], Reconst Loss: 82.6545, KL Div: 25.8706, kl_div_c: 1.7598, Entropy: 0.5428
    Epoch[9/10], Step [110/469], Reconst Loss: 80.2121, KL Div: 25.5131, kl_div_c: 1.7695, Entropy: 0.5330
    Epoch[9/10], Step [120/469], Reconst Loss: 82.1474, KL Div: 25.9261, kl_div_c: 1.6933, Entropy: 0.6093
    Epoch[9/10], Step [130/469], Reconst Loss: 81.4940, KL Div: 25.9509, kl_div_c: 1.7789, Entropy: 0.5237
    Epoch[9/10], Step [140/469], Reconst Loss: 80.6777, KL Div: 26.1977, kl_div_c: 1.7019, Entropy: 0.6007
    Epoch[9/10], Step [150/469], Reconst Loss: 87.1775, KL Div: 26.5559, kl_div_c: 1.6974, Entropy: 0.6052
    Epoch[9/10], Step [160/469], Reconst Loss: 81.4370, KL Div: 26.1403, kl_div_c: 1.6890, Entropy: 0.6136
    Epoch[9/10], Step [170/469], Reconst Loss: 81.8072, KL Div: 26.0084, kl_div_c: 1.6635, Entropy: 0.6390
    Epoch[9/10], Step [180/469], Reconst Loss: 83.9134, KL Div: 25.6723, kl_div_c: 1.7022, Entropy: 0.6003
    Epoch[9/10], Step [190/469], Reconst Loss: 82.7810, KL Div: 25.9767, kl_div_c: 1.7238, Entropy: 0.5788
    Epoch[9/10], Step [200/469], Reconst Loss: 82.2661, KL Div: 26.1080, kl_div_c: 1.7273, Entropy: 0.5753
    Epoch[9/10], Step [210/469], Reconst Loss: 80.0547, KL Div: 27.1209, kl_div_c: 1.6305, Entropy: 0.6721
    Epoch[9/10], Step [220/469], Reconst Loss: 85.0820, KL Div: 25.6112, kl_div_c: 1.7957, Entropy: 0.5069
    Epoch[9/10], Step [230/469], Reconst Loss: 78.7923, KL Div: 27.1751, kl_div_c: 1.8111, Entropy: 0.4915
    Epoch[9/10], Step [240/469], Reconst Loss: 83.6495, KL Div: 26.5158, kl_div_c: 1.8114, Entropy: 0.4912
    Epoch[9/10], Step [250/469], Reconst Loss: 79.2639, KL Div: 25.1871, kl_div_c: 1.7665, Entropy: 0.5361
    Epoch[9/10], Step [260/469], Reconst Loss: 79.7349, KL Div: 26.8379, kl_div_c: 1.7504, Entropy: 0.5522
    Epoch[9/10], Step [270/469], Reconst Loss: 82.1969, KL Div: 25.8323, kl_div_c: 1.7411, Entropy: 0.5615
    Epoch[9/10], Step [280/469], Reconst Loss: 79.0796, KL Div: 25.8455, kl_div_c: 1.6764, Entropy: 0.6261
    Epoch[9/10], Step [290/469], Reconst Loss: 80.0680, KL Div: 25.8942, kl_div_c: 1.7284, Entropy: 0.5741
    Epoch[9/10], Step [300/469], Reconst Loss: 82.0987, KL Div: 26.1857, kl_div_c: 1.6076, Entropy: 0.6950
    Epoch[9/10], Step [310/469], Reconst Loss: 81.1156, KL Div: 26.1990, kl_div_c: 1.6303, Entropy: 0.6723
    Epoch[9/10], Step [320/469], Reconst Loss: 82.0214, KL Div: 25.7137, kl_div_c: 1.7539, Entropy: 0.5487
    Epoch[9/10], Step [330/469], Reconst Loss: 78.4941, KL Div: 25.7333, kl_div_c: 1.7624, Entropy: 0.5402
    Epoch[9/10], Step [340/469], Reconst Loss: 77.3844, KL Div: 26.5223, kl_div_c: 1.7010, Entropy: 0.6016
    Epoch[9/10], Step [350/469], Reconst Loss: 79.6441, KL Div: 26.0941, kl_div_c: 1.6748, Entropy: 0.6277
    Epoch[9/10], Step [360/469], Reconst Loss: 81.6610, KL Div: 26.3258, kl_div_c: 1.8006, Entropy: 0.5020
    Epoch[9/10], Step [370/469], Reconst Loss: 83.0548, KL Div: 27.4775, kl_div_c: 1.8058, Entropy: 0.4968
    Epoch[9/10], Step [380/469], Reconst Loss: 84.5611, KL Div: 26.4529, kl_div_c: 1.8067, Entropy: 0.4959
    Epoch[9/10], Step [390/469], Reconst Loss: 81.7165, KL Div: 25.9372, kl_div_c: 1.7128, Entropy: 0.5898
    Epoch[9/10], Step [400/469], Reconst Loss: 81.2576, KL Div: 26.3591, kl_div_c: 1.7227, Entropy: 0.5799
    Epoch[9/10], Step [410/469], Reconst Loss: 81.3379, KL Div: 25.9060, kl_div_c: 1.7223, Entropy: 0.5803
    Epoch[9/10], Step [420/469], Reconst Loss: 78.8244, KL Div: 26.3701, kl_div_c: 1.7572, Entropy: 0.5454
    Epoch[9/10], Step [430/469], Reconst Loss: 79.6145, KL Div: 25.8130, kl_div_c: 1.7361, Entropy: 0.5665
    Epoch[9/10], Step [440/469], Reconst Loss: 82.9903, KL Div: 26.5819, kl_div_c: 1.7874, Entropy: 0.5152
    Epoch[9/10], Step [450/469], Reconst Loss: 80.8525, KL Div: 26.0057, kl_div_c: 1.7105, Entropy: 0.5921
    Epoch[9/10], Step [460/469], Reconst Loss: 83.9691, KL Div: 25.9790, kl_div_c: 1.7201, Entropy: 0.5825
    Epoch[10/10], Step [10/469], Reconst Loss: 84.6509, KL Div: 26.3910, kl_div_c: 1.7843, Entropy: 0.5183
    Epoch[10/10], Step [20/469], Reconst Loss: 81.3388, KL Div: 26.3566, kl_div_c: 1.7372, Entropy: 0.5654
    Epoch[10/10], Step [30/469], Reconst Loss: 78.6926, KL Div: 25.8625, kl_div_c: 1.7799, Entropy: 0.5227
    Epoch[10/10], Step [40/469], Reconst Loss: 79.1927, KL Div: 25.5510, kl_div_c: 1.7606, Entropy: 0.5419
    Epoch[10/10], Step [50/469], Reconst Loss: 81.7427, KL Div: 26.9990, kl_div_c: 1.7282, Entropy: 0.5744
    Epoch[10/10], Step [60/469], Reconst Loss: 80.9173, KL Div: 26.1169, kl_div_c: 1.8539, Entropy: 0.4487
    Epoch[10/10], Step [70/469], Reconst Loss: 84.6382, KL Div: 26.4832, kl_div_c: 1.6685, Entropy: 0.6341
    Epoch[10/10], Step [80/469], Reconst Loss: 84.1370, KL Div: 26.4602, kl_div_c: 1.7556, Entropy: 0.5470
    Epoch[10/10], Step [90/469], Reconst Loss: 80.7792, KL Div: 26.2180, kl_div_c: 1.7201, Entropy: 0.5825
    Epoch[10/10], Step [100/469], Reconst Loss: 80.3058, KL Div: 26.6194, kl_div_c: 1.6692, Entropy: 0.6334
    Epoch[10/10], Step [110/469], Reconst Loss: 79.9794, KL Div: 25.4179, kl_div_c: 1.7454, Entropy: 0.5572
    Epoch[10/10], Step [120/469], Reconst Loss: 82.7514, KL Div: 26.2939, kl_div_c: 1.7204, Entropy: 0.5822
    Epoch[10/10], Step [130/469], Reconst Loss: 81.3144, KL Div: 25.7045, kl_div_c: 1.6973, Entropy: 0.6053
    Epoch[10/10], Step [140/469], Reconst Loss: 81.6158, KL Div: 26.5005, kl_div_c: 1.7627, Entropy: 0.5399
    Epoch[10/10], Step [150/469], Reconst Loss: 82.0381, KL Div: 27.0073, kl_div_c: 1.8218, Entropy: 0.4807
    Epoch[10/10], Step [160/469], Reconst Loss: 83.5648, KL Div: 25.6440, kl_div_c: 1.7022, Entropy: 0.6003
    Epoch[10/10], Step [170/469], Reconst Loss: 82.8050, KL Div: 26.1016, kl_div_c: 1.6501, Entropy: 0.6524
    Epoch[10/10], Step [180/469], Reconst Loss: 83.2472, KL Div: 26.4807, kl_div_c: 1.7437, Entropy: 0.5589
    Epoch[10/10], Step [190/469], Reconst Loss: 80.3203, KL Div: 26.3707, kl_div_c: 1.7752, Entropy: 0.5273
    Epoch[10/10], Step [200/469], Reconst Loss: 81.7718, KL Div: 26.2828, kl_div_c: 1.7322, Entropy: 0.5704
    Epoch[10/10], Step [210/469], Reconst Loss: 80.2395, KL Div: 26.1439, kl_div_c: 1.7606, Entropy: 0.5420
    Epoch[10/10], Step [220/469], Reconst Loss: 81.4182, KL Div: 26.9130, kl_div_c: 1.7489, Entropy: 0.5537
    Epoch[10/10], Step [230/469], Reconst Loss: 84.1852, KL Div: 26.0682, kl_div_c: 1.7259, Entropy: 0.5766
    Epoch[10/10], Step [240/469], Reconst Loss: 81.8582, KL Div: 26.1033, kl_div_c: 1.8154, Entropy: 0.4871
    Epoch[10/10], Step [250/469], Reconst Loss: 82.4586, KL Div: 26.8427, kl_div_c: 1.7845, Entropy: 0.5181
    Epoch[10/10], Step [260/469], Reconst Loss: 77.6665, KL Div: 26.0776, kl_div_c: 1.8464, Entropy: 0.4562
    Epoch[10/10], Step [270/469], Reconst Loss: 83.3983, KL Div: 26.7908, kl_div_c: 1.7382, Entropy: 0.5644
    Epoch[10/10], Step [280/469], Reconst Loss: 82.6404, KL Div: 25.5614, kl_div_c: 1.6815, Entropy: 0.6211
    Epoch[10/10], Step [290/469], Reconst Loss: 81.4572, KL Div: 26.5617, kl_div_c: 1.7556, Entropy: 0.5470
    Epoch[10/10], Step [300/469], Reconst Loss: 81.0497, KL Div: 26.0682, kl_div_c: 1.7425, Entropy: 0.5601
    Epoch[10/10], Step [310/469], Reconst Loss: 79.8778, KL Div: 26.1744, kl_div_c: 1.7378, Entropy: 0.5648
    Epoch[10/10], Step [320/469], Reconst Loss: 81.5134, KL Div: 25.7940, kl_div_c: 1.6971, Entropy: 0.6055
    Epoch[10/10], Step [330/469], Reconst Loss: 82.6496, KL Div: 26.4097, kl_div_c: 1.8093, Entropy: 0.4932
    Epoch[10/10], Step [340/469], Reconst Loss: 79.8910, KL Div: 25.4569, kl_div_c: 1.7560, Entropy: 0.5466
    Epoch[10/10], Step [350/469], Reconst Loss: 78.6526, KL Div: 26.0519, kl_div_c: 1.8028, Entropy: 0.4998
    Epoch[10/10], Step [360/469], Reconst Loss: 80.6216, KL Div: 26.6758, kl_div_c: 1.7918, Entropy: 0.5107
    Epoch[10/10], Step [370/469], Reconst Loss: 79.5593, KL Div: 26.1562, kl_div_c: 1.6999, Entropy: 0.6027
    Epoch[10/10], Step [380/469], Reconst Loss: 79.0427, KL Div: 25.7465, kl_div_c: 1.7459, Entropy: 0.5567
    Epoch[10/10], Step [390/469], Reconst Loss: 81.7966, KL Div: 26.9920, kl_div_c: 1.7914, Entropy: 0.5112
    Epoch[10/10], Step [400/469], Reconst Loss: 78.9095, KL Div: 25.0713, kl_div_c: 1.7025, Entropy: 0.6001
    Epoch[10/10], Step [410/469], Reconst Loss: 76.8131, KL Div: 26.0223, kl_div_c: 1.7824, Entropy: 0.5202
    Epoch[10/10], Step [420/469], Reconst Loss: 79.3607, KL Div: 26.0679, kl_div_c: 1.7124, Entropy: 0.5902
    Epoch[10/10], Step [430/469], Reconst Loss: 86.3665, KL Div: 25.8332, kl_div_c: 1.7459, Entropy: 0.5567
    Epoch[10/10], Step [440/469], Reconst Loss: 78.9826, KL Div: 25.7291, kl_div_c: 1.7678, Entropy: 0.5347
    Epoch[10/10], Step [450/469], Reconst Loss: 78.5583, KL Div: 25.7672, kl_div_c: 1.7948, Entropy: 0.5078
    Epoch[10/10], Step [460/469], Reconst Loss: 83.6455, KL Div: 26.3997, kl_div_c: 1.7080, Entropy: 0.5946

```python
plt.plot(nmi)
plt.show()
```

```python
plt.figure(figsize=(10,5))
plot_reconstruction(model_G)
```

This was for reconstruction, but we care more about generation. For each category, we are generating 8 samples thanks to the `plot_conditional_generation()` function.

```python
plt.figure(figsize=(10,10))
plot_conditional_generation(model_G, n=8)
```

It does not look like our original idea is working...

What is happening is that our network is not using the categorical variable. We can track the [normalized mutual information](https://en.wikipedia.org/wiki/Mutual_information#Normalized_variants) (see [this method in scikit-learn](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.normalized_mutual_info_score.html)) between the true labels and the labels predicted by our network (just by taking the category with maximal probability).

Change your training loop to return the normalized mutual information (NMI) for each epoch. Plot the curve to check that the NMI is actually decreasing.

This problem is explained in [(Burgess et al., 2018)](https://arxiv.org/abs/1804.03599) and a solution is proposed in Section 5.

In order to force our network to use the categorical variable, we will change the loss according to [(Dupont, 2018)](https://arxiv.org/abs/1804.00104), Section 3 Equation (7).

Implement this change in the training loop and plot the new NMI curve. For $\beta = 20, C_z=100, C_c=100$, you should see that NMI increases.

```python
model_G = VAE_Gumbel().to(device)
optimizer = torch.optim.Adam(model_G.parameters(), lr=learning_rate)
```

```python
def train_G_modified_loss(model, data_loader=data_loader,num_epochs=num_epochs, beta = 20, C_z_fin=100, C_c_fin=100, verbose=True):
    #progress of C
    Cz = np.linspace(0, C_z_fin, num_epochs*len(data_loader))
    Cc = np.linspace(0, C_c_fin, num_epochs*len(data_loader))
    nmi_scores = []
    model.train(True)
    for epoch in range(num_epochs):
        all_labels = []
        all_labels_est = []
        for i, (x, labels) in enumerate(data_loader):

            # Forward pass
            x = x.to(device).view(-1, image_size)
            x_reconst, mu, log_var, g_alpha = model(x)

            #NMI
            all_labels += labels.tolist()
            all_labels_est += torch.argmax(g_alpha, dim=1).tolist()

            #update C
            cz_i = Cz[epoch * len(data_loader) + i]
            cc_i = Cc[epoch * len(data_loader) + i]

            #compute KL+reconst_loss
            reconst_loss = F.binary_cross_entropy(x_reconst, x, reduction='sum')
            KL_div_z = - 0.5 * torch.sum(1 + log_var - mu.pow(2) - log_var.exp())
            pi = torch.tensor([1/n_classes], device=device)
            H_cat = - torch.sum(g_alpha * g_alpha.log(), dim=1) #entropy
            KL_div_c = torch.sum(-H_cat - torch.log(pi))#expectency
            H_cat = torch.sum(H_cat) #print

            KL_div = torch.abs(KL_div_z - cz_i) + torch.abs(KL_div_c - cc_i)

            # Backprop and optimize
            loss = reconst_loss + beta * KL_div
            optimizer.zero_grad()
            loss.backward()
            optimizer.step()

            if verbose:
                if (i+1) % 10 == 0:
                    print ("Epoch[{}/{}], Step [{}/{}], Reconst Loss: {:.4f}, KL Div: {:.4f}, kl_div_c: {:.4f}, Entropy: {:.4f}"
                           .format(epoch+1, num_epochs, i+1, len(data_loader), reconst_loss.item()/batch_size,
                                   KL_div.item()/batch_size, KL_div_c.item()/batch_size, H_cat.item()/batch_size))

        nmi = normalized_mutual_info_score(all_labels, all_labels_est)
        nmi_scores.append(nmi)
    return nmi_scores
```

```python
# Hyper-parameters
num_epochs = 10
learning_rate = 1e-3
beta = 20
C_z_fin=200
C_c_fin=200

model_G = VAE_Gumbel(z_dim = z_dim).to(device)
optimizer = torch.optim.Adam(model_G.parameters(), lr=learning_rate)

nmi2 = train_G_modified_loss(model_G, data_loader, num_epochs=num_epochs, beta=beta, C_z_fin=C_z_fin, C_c_fin=C_c_fin)
```

    Epoch[1/10], Step [10/469], Reconst Loss: 365.7806, KL Div: 1.8152, kl_div_c: 1.3675, Entropy: 0.9351
    Epoch[1/10], Step [20/469], Reconst Loss: 266.7787, KL Div: 1.6835, kl_div_c: 1.3680, Entropy: 0.9346
    Epoch[1/10], Step [30/469], Reconst Loss: 237.5932, KL Div: 1.4773, kl_div_c: 1.3538, Entropy: 0.9488
    Epoch[1/10], Step [40/469], Reconst Loss: 226.8783, KL Div: 1.4861, kl_div_c: 1.4351, Entropy: 0.8675
    Epoch[1/10], Step [50/469], Reconst Loss: 225.4799, KL Div: 1.3478, kl_div_c: 1.3421, Entropy: 0.9605
    Epoch[1/10], Step [60/469], Reconst Loss: 203.9574, KL Div: 1.4929, kl_div_c: 1.4784, Entropy: 0.8242
    Epoch[1/10], Step [70/469], Reconst Loss: 211.2693, KL Div: 1.3851, kl_div_c: 1.3854, Entropy: 0.9171
    Epoch[1/10], Step [80/469], Reconst Loss: 215.0709, KL Div: 1.4747, kl_div_c: 1.4838, Entropy: 0.8187
    Epoch[1/10], Step [90/469], Reconst Loss: 205.7009, KL Div: 1.4042, kl_div_c: 1.4088, Entropy: 0.8938
    Epoch[1/10], Step [100/469], Reconst Loss: 207.1015, KL Div: 1.4121, kl_div_c: 1.4279, Entropy: 0.8747
    Epoch[1/10], Step [110/469], Reconst Loss: 207.5055, KL Div: 1.3689, kl_div_c: 1.3790, Entropy: 0.9236
    Epoch[1/10], Step [120/469], Reconst Loss: 211.6981, KL Div: 1.3863, kl_div_c: 1.4244, Entropy: 0.8782
    Epoch[1/10], Step [130/469], Reconst Loss: 204.6374, KL Div: 1.3652, kl_div_c: 1.3982, Entropy: 0.9044
    Epoch[1/10], Step [140/469], Reconst Loss: 205.4488, KL Div: 1.3616, kl_div_c: 1.3724, Entropy: 0.9302
    Epoch[1/10], Step [150/469], Reconst Loss: 205.4825, KL Div: 1.3804, kl_div_c: 1.4179, Entropy: 0.8847
    Epoch[1/10], Step [160/469], Reconst Loss: 202.2792, KL Div: 1.3622, kl_div_c: 1.4071, Entropy: 0.8955
    Epoch[1/10], Step [170/469], Reconst Loss: 197.3674, KL Div: 1.4109, kl_div_c: 1.4468, Entropy: 0.8558
    Epoch[1/10], Step [180/469], Reconst Loss: 210.9661, KL Div: 1.3143, kl_div_c: 1.3727, Entropy: 0.9299
    Epoch[1/10], Step [190/469], Reconst Loss: 204.2466, KL Div: 1.3891, kl_div_c: 1.4257, Entropy: 0.8769
    Epoch[1/10], Step [200/469], Reconst Loss: 213.4306, KL Div: 1.3253, kl_div_c: 1.3639, Entropy: 0.9387
    Epoch[1/10], Step [210/469], Reconst Loss: 204.0423, KL Div: 1.4319, kl_div_c: 1.4553, Entropy: 0.8473
    Epoch[1/10], Step [220/469], Reconst Loss: 198.6331, KL Div: 1.3471, kl_div_c: 1.4083, Entropy: 0.8943
    Epoch[1/10], Step [230/469], Reconst Loss: 209.3045, KL Div: 1.3804, kl_div_c: 1.4161, Entropy: 0.8865
    Epoch[1/10], Step [240/469], Reconst Loss: 207.2680, KL Div: 1.3414, kl_div_c: 1.4145, Entropy: 0.8880
    Epoch[1/10], Step [250/469], Reconst Loss: 204.7468, KL Div: 1.3251, kl_div_c: 1.3490, Entropy: 0.9535
    Epoch[1/10], Step [260/469], Reconst Loss: 205.9723, KL Div: 1.4007, kl_div_c: 1.4416, Entropy: 0.8609
    Epoch[1/10], Step [270/469], Reconst Loss: 206.7068, KL Div: 1.4118, kl_div_c: 1.4708, Entropy: 0.8318
    Epoch[1/10], Step [280/469], Reconst Loss: 199.2047, KL Div: 1.4126, kl_div_c: 1.4755, Entropy: 0.8271
    Epoch[1/10], Step [290/469], Reconst Loss: 197.8575, KL Div: 1.3791, kl_div_c: 1.4387, Entropy: 0.8639
    Epoch[1/10], Step [300/469], Reconst Loss: 210.9990, KL Div: 1.3454, kl_div_c: 1.3871, Entropy: 0.9155
    Epoch[1/10], Step [310/469], Reconst Loss: 205.9446, KL Div: 1.3614, kl_div_c: 1.4199, Entropy: 0.8827
    Epoch[1/10], Step [320/469], Reconst Loss: 199.9848, KL Div: 1.4440, kl_div_c: 1.4323, Entropy: 0.8703
    Epoch[1/10], Step [330/469], Reconst Loss: 194.2483, KL Div: 1.3401, kl_div_c: 1.4421, Entropy: 0.8605
    Epoch[1/10], Step [340/469], Reconst Loss: 199.1043, KL Div: 1.4565, kl_div_c: 1.4923, Entropy: 0.8102
    Epoch[1/10], Step [350/469], Reconst Loss: 203.4531, KL Div: 1.3216, kl_div_c: 1.4288, Entropy: 0.8737
    Epoch[1/10], Step [360/469], Reconst Loss: 207.0937, KL Div: 1.4065, kl_div_c: 1.4467, Entropy: 0.8559
    Epoch[1/10], Step [370/469], Reconst Loss: 205.9380, KL Div: 1.3324, kl_div_c: 1.4102, Entropy: 0.8924
    Epoch[1/10], Step [380/469], Reconst Loss: 198.7399, KL Div: 1.4078, kl_div_c: 1.4372, Entropy: 0.8654
    Epoch[1/10], Step [390/469], Reconst Loss: 210.4839, KL Div: 1.3120, kl_div_c: 1.4100, Entropy: 0.8926
    Epoch[1/10], Step [400/469], Reconst Loss: 203.6008, KL Div: 1.2817, kl_div_c: 1.3829, Entropy: 0.9197
    Epoch[1/10], Step [410/469], Reconst Loss: 202.9372, KL Div: 1.3429, kl_div_c: 1.3536, Entropy: 0.9490
    Epoch[1/10], Step [420/469], Reconst Loss: 203.9925, KL Div: 1.3576, kl_div_c: 1.4187, Entropy: 0.8839
    Epoch[1/10], Step [430/469], Reconst Loss: 204.2941, KL Div: 1.2776, kl_div_c: 1.3823, Entropy: 0.9202
    Epoch[1/10], Step [440/469], Reconst Loss: 203.8614, KL Div: 1.3978, kl_div_c: 1.4896, Entropy: 0.8130
    Epoch[1/10], Step [450/469], Reconst Loss: 197.2280, KL Div: 1.2999, kl_div_c: 1.4296, Entropy: 0.8730
    Epoch[1/10], Step [460/469], Reconst Loss: 198.5396, KL Div: 1.3768, kl_div_c: 1.4606, Entropy: 0.8419
    Epoch[2/10], Step [10/469], Reconst Loss: 203.4960, KL Div: 1.3726, kl_div_c: 1.4880, Entropy: 0.8146
    Epoch[2/10], Step [20/469], Reconst Loss: 205.8861, KL Div: 1.2943, kl_div_c: 1.3908, Entropy: 0.9118
    Epoch[2/10], Step [30/469], Reconst Loss: 203.5964, KL Div: 1.2375, kl_div_c: 1.3747, Entropy: 0.9279
    Epoch[2/10], Step [40/469], Reconst Loss: 197.0630, KL Div: 1.3064, kl_div_c: 1.4446, Entropy: 0.8580
    Epoch[2/10], Step [50/469], Reconst Loss: 195.6449, KL Div: 1.2794, kl_div_c: 1.3895, Entropy: 0.9131
    Epoch[2/10], Step [60/469], Reconst Loss: 197.1300, KL Div: 1.3458, kl_div_c: 1.4687, Entropy: 0.8339
    Epoch[2/10], Step [70/469], Reconst Loss: 202.8020, KL Div: 1.3889, kl_div_c: 1.4958, Entropy: 0.8068
    Epoch[2/10], Step [80/469], Reconst Loss: 196.0045, KL Div: 1.2028, kl_div_c: 1.3745, Entropy: 0.9281
    Epoch[2/10], Step [90/469], Reconst Loss: 202.1400, KL Div: 1.3214, kl_div_c: 1.4308, Entropy: 0.8718
    Epoch[2/10], Step [100/469], Reconst Loss: 205.9220, KL Div: 1.3310, kl_div_c: 1.4430, Entropy: 0.8596
    Epoch[2/10], Step [110/469], Reconst Loss: 203.3112, KL Div: 1.2995, kl_div_c: 1.4406, Entropy: 0.8620
    Epoch[2/10], Step [120/469], Reconst Loss: 199.1581, KL Div: 1.3111, kl_div_c: 1.4349, Entropy: 0.8677
    Epoch[2/10], Step [130/469], Reconst Loss: 190.7327, KL Div: 1.3035, kl_div_c: 1.4548, Entropy: 0.8477
    Epoch[2/10], Step [140/469], Reconst Loss: 199.3186, KL Div: 1.2376, kl_div_c: 1.4034, Entropy: 0.8992
    Epoch[2/10], Step [150/469], Reconst Loss: 198.6394, KL Div: 1.3741, kl_div_c: 1.5237, Entropy: 0.7789
    Epoch[2/10], Step [160/469], Reconst Loss: 191.7486, KL Div: 1.2727, kl_div_c: 1.4514, Entropy: 0.8511
    Epoch[2/10], Step [170/469], Reconst Loss: 195.0895, KL Div: 1.2628, kl_div_c: 1.4559, Entropy: 0.8466
    Epoch[2/10], Step [180/469], Reconst Loss: 203.0213, KL Div: 1.4144, kl_div_c: 1.5458, Entropy: 0.7568
    Epoch[2/10], Step [190/469], Reconst Loss: 201.5511, KL Div: 1.2927, kl_div_c: 1.4929, Entropy: 0.8097
    Epoch[2/10], Step [200/469], Reconst Loss: 195.9098, KL Div: 1.3157, kl_div_c: 1.5232, Entropy: 0.7794
    Epoch[2/10], Step [210/469], Reconst Loss: 198.9702, KL Div: 1.2553, kl_div_c: 1.4661, Entropy: 0.8364
    Epoch[2/10], Step [220/469], Reconst Loss: 193.4492, KL Div: 1.2848, kl_div_c: 1.5067, Entropy: 0.7959
    Epoch[2/10], Step [230/469], Reconst Loss: 193.8467, KL Div: 1.3161, kl_div_c: 1.5326, Entropy: 0.7700
    Epoch[2/10], Step [240/469], Reconst Loss: 199.3786, KL Div: 1.3996, kl_div_c: 1.6178, Entropy: 0.6847
    Epoch[2/10], Step [250/469], Reconst Loss: 193.5779, KL Div: 1.3755, kl_div_c: 1.5916, Entropy: 0.7110
    Epoch[2/10], Step [260/469], Reconst Loss: 191.3688, KL Div: 1.2858, kl_div_c: 1.5082, Entropy: 0.7944
    Epoch[2/10], Step [270/469], Reconst Loss: 191.6511, KL Div: 1.3839, kl_div_c: 1.5597, Entropy: 0.7429
    Epoch[2/10], Step [280/469], Reconst Loss: 195.3796, KL Div: 1.3359, kl_div_c: 1.5654, Entropy: 0.7372
    Epoch[2/10], Step [290/469], Reconst Loss: 191.7183, KL Div: 1.3595, kl_div_c: 1.5874, Entropy: 0.7152
    Epoch[2/10], Step [300/469], Reconst Loss: 197.1807, KL Div: 1.5104, kl_div_c: 1.6933, Entropy: 0.6093
    Epoch[2/10], Step [310/469], Reconst Loss: 192.1837, KL Div: 1.3562, kl_div_c: 1.6064, Entropy: 0.6961
    Epoch[2/10], Step [320/469], Reconst Loss: 182.5093, KL Div: 1.3722, kl_div_c: 1.6178, Entropy: 0.6848
    Epoch[2/10], Step [330/469], Reconst Loss: 190.5924, KL Div: 1.2740, kl_div_c: 1.5135, Entropy: 0.7891
    Epoch[2/10], Step [340/469], Reconst Loss: 192.3715, KL Div: 1.4847, kl_div_c: 1.7206, Entropy: 0.5820
    Epoch[2/10], Step [350/469], Reconst Loss: 186.1114, KL Div: 1.3323, kl_div_c: 1.5152, Entropy: 0.7874
    Epoch[2/10], Step [360/469], Reconst Loss: 195.6768, KL Div: 1.3573, kl_div_c: 1.6205, Entropy: 0.6821
    Epoch[2/10], Step [370/469], Reconst Loss: 187.8044, KL Div: 1.3878, kl_div_c: 1.6275, Entropy: 0.6751
    Epoch[2/10], Step [380/469], Reconst Loss: 181.3711, KL Div: 1.3298, kl_div_c: 1.6011, Entropy: 0.7015
    Epoch[2/10], Step [390/469], Reconst Loss: 189.4561, KL Div: 1.3894, kl_div_c: 1.6748, Entropy: 0.6278
    Epoch[2/10], Step [400/469], Reconst Loss: 184.9549, KL Div: 1.3772, kl_div_c: 1.6033, Entropy: 0.6993
    Epoch[2/10], Step [410/469], Reconst Loss: 191.0608, KL Div: 1.3133, kl_div_c: 1.6001, Entropy: 0.7025
    Epoch[2/10], Step [420/469], Reconst Loss: 184.5422, KL Div: 1.3743, kl_div_c: 1.6689, Entropy: 0.6337
    Epoch[2/10], Step [430/469], Reconst Loss: 185.9262, KL Div: 1.3748, kl_div_c: 1.6217, Entropy: 0.6809
    Epoch[2/10], Step [440/469], Reconst Loss: 185.7413, KL Div: 1.3643, kl_div_c: 1.6059, Entropy: 0.6967
    Epoch[2/10], Step [450/469], Reconst Loss: 191.6344, KL Div: 1.3430, kl_div_c: 1.5787, Entropy: 0.7239
    Epoch[2/10], Step [460/469], Reconst Loss: 180.3888, KL Div: 1.3012, kl_div_c: 1.6021, Entropy: 0.7005
    Epoch[3/10], Step [10/469], Reconst Loss: 176.5178, KL Div: 1.3249, kl_div_c: 1.6397, Entropy: 0.6629
    Epoch[3/10], Step [20/469], Reconst Loss: 184.3883, KL Div: 1.3888, kl_div_c: 1.6553, Entropy: 0.6472
    Epoch[3/10], Step [30/469], Reconst Loss: 186.7247, KL Div: 1.4115, kl_div_c: 1.6735, Entropy: 0.6291
    Epoch[3/10], Step [40/469], Reconst Loss: 180.6637, KL Div: 1.3465, kl_div_c: 1.6057, Entropy: 0.6969
    Epoch[3/10], Step [50/469], Reconst Loss: 182.3732, KL Div: 1.3955, kl_div_c: 1.6660, Entropy: 0.6365
    Epoch[3/10], Step [60/469], Reconst Loss: 182.2766, KL Div: 1.3764, kl_div_c: 1.6845, Entropy: 0.6181
    Epoch[3/10], Step [70/469], Reconst Loss: 186.5266, KL Div: 1.3211, kl_div_c: 1.6442, Entropy: 0.6584
    Epoch[3/10], Step [80/469], Reconst Loss: 189.3331, KL Div: 1.2896, kl_div_c: 1.6142, Entropy: 0.6884
    Epoch[3/10], Step [90/469], Reconst Loss: 188.1027, KL Div: 1.4569, kl_div_c: 1.6824, Entropy: 0.6202
    Epoch[3/10], Step [100/469], Reconst Loss: 184.9888, KL Div: 1.2726, kl_div_c: 1.6093, Entropy: 0.6933
    Epoch[3/10], Step [110/469], Reconst Loss: 178.9727, KL Div: 1.4405, kl_div_c: 1.6733, Entropy: 0.6293
    Epoch[3/10], Step [120/469], Reconst Loss: 182.3946, KL Div: 1.3179, kl_div_c: 1.6604, Entropy: 0.6422
    Epoch[3/10], Step [130/469], Reconst Loss: 184.2013, KL Div: 1.3686, kl_div_c: 1.7079, Entropy: 0.5947
    Epoch[3/10], Step [140/469], Reconst Loss: 181.5211, KL Div: 1.2872, kl_div_c: 1.6457, Entropy: 0.6569
    Epoch[3/10], Step [150/469], Reconst Loss: 185.5801, KL Div: 1.3179, kl_div_c: 1.6216, Entropy: 0.6810
    Epoch[3/10], Step [160/469], Reconst Loss: 174.6873, KL Div: 1.3062, kl_div_c: 1.6493, Entropy: 0.6532
    Epoch[3/10], Step [170/469], Reconst Loss: 186.5756, KL Div: 1.2841, kl_div_c: 1.6503, Entropy: 0.6523
    Epoch[3/10], Step [180/469], Reconst Loss: 176.1456, KL Div: 1.4509, kl_div_c: 1.7230, Entropy: 0.5796
    Epoch[3/10], Step [190/469], Reconst Loss: 188.5756, KL Div: 1.3006, kl_div_c: 1.6312, Entropy: 0.6714
    Epoch[3/10], Step [200/469], Reconst Loss: 187.4419, KL Div: 1.3729, kl_div_c: 1.6495, Entropy: 0.6531
    Epoch[3/10], Step [210/469], Reconst Loss: 182.9089, KL Div: 1.3768, kl_div_c: 1.6903, Entropy: 0.6123
    Epoch[3/10], Step [220/469], Reconst Loss: 185.0822, KL Div: 1.3038, kl_div_c: 1.6842, Entropy: 0.6184
    Epoch[3/10], Step [230/469], Reconst Loss: 180.7024, KL Div: 1.2860, kl_div_c: 1.6626, Entropy: 0.6400
    Epoch[3/10], Step [240/469], Reconst Loss: 180.8629, KL Div: 1.4418, kl_div_c: 1.8123, Entropy: 0.4903
    Epoch[3/10], Step [250/469], Reconst Loss: 182.4550, KL Div: 1.3920, kl_div_c: 1.7000, Entropy: 0.6026
    Epoch[3/10], Step [260/469], Reconst Loss: 178.4521, KL Div: 1.3751, kl_div_c: 1.7438, Entropy: 0.5588
    Epoch[3/10], Step [270/469], Reconst Loss: 184.4082, KL Div: 1.2930, kl_div_c: 1.6849, Entropy: 0.6177
    Epoch[3/10], Step [280/469], Reconst Loss: 180.8120, KL Div: 1.3859, kl_div_c: 1.7107, Entropy: 0.5919
    Epoch[3/10], Step [290/469], Reconst Loss: 175.5583, KL Div: 1.3793, kl_div_c: 1.7512, Entropy: 0.5513
    Epoch[3/10], Step [300/469], Reconst Loss: 174.4447, KL Div: 1.3391, kl_div_c: 1.7409, Entropy: 0.5617
    Epoch[3/10], Step [310/469], Reconst Loss: 188.4864, KL Div: 1.3577, kl_div_c: 1.6518, Entropy: 0.6508
    Epoch[3/10], Step [320/469], Reconst Loss: 177.0939, KL Div: 1.2601, kl_div_c: 1.6147, Entropy: 0.6878
    Epoch[3/10], Step [330/469], Reconst Loss: 170.2322, KL Div: 1.4159, kl_div_c: 1.8242, Entropy: 0.4783
    Epoch[3/10], Step [340/469], Reconst Loss: 177.2252, KL Div: 1.3073, kl_div_c: 1.7203, Entropy: 0.5823
    Epoch[3/10], Step [350/469], Reconst Loss: 173.9044, KL Div: 1.3684, kl_div_c: 1.7500, Entropy: 0.5526
    Epoch[3/10], Step [360/469], Reconst Loss: 176.2756, KL Div: 1.3898, kl_div_c: 1.7932, Entropy: 0.5094
    Epoch[3/10], Step [370/469], Reconst Loss: 176.2525, KL Div: 1.3344, kl_div_c: 1.6860, Entropy: 0.6166
    Epoch[3/10], Step [380/469], Reconst Loss: 176.4805, KL Div: 1.4326, kl_div_c: 1.7896, Entropy: 0.5129
    Epoch[3/10], Step [390/469], Reconst Loss: 174.3938, KL Div: 1.3668, kl_div_c: 1.7015, Entropy: 0.6011
    Epoch[3/10], Step [400/469], Reconst Loss: 172.2226, KL Div: 1.3249, kl_div_c: 1.7213, Entropy: 0.5813
    Epoch[3/10], Step [410/469], Reconst Loss: 175.5396, KL Div: 1.3623, kl_div_c: 1.8045, Entropy: 0.4981
    Epoch[3/10], Step [420/469], Reconst Loss: 178.9061, KL Div: 1.4288, kl_div_c: 1.8282, Entropy: 0.4743
    Epoch[3/10], Step [430/469], Reconst Loss: 178.8146, KL Div: 1.3644, kl_div_c: 1.7731, Entropy: 0.5295
    Epoch[3/10], Step [440/469], Reconst Loss: 172.5052, KL Div: 1.3738, kl_div_c: 1.7819, Entropy: 0.5207
    Epoch[3/10], Step [450/469], Reconst Loss: 177.2525, KL Div: 1.2648, kl_div_c: 1.7124, Entropy: 0.5902
    Epoch[3/10], Step [460/469], Reconst Loss: 176.2922, KL Div: 1.3582, kl_div_c: 1.8070, Entropy: 0.4956
    Epoch[4/10], Step [10/469], Reconst Loss: 164.0246, KL Div: 1.3699, kl_div_c: 1.7718, Entropy: 0.5308
    Epoch[4/10], Step [20/469], Reconst Loss: 174.9942, KL Div: 1.2828, kl_div_c: 1.7468, Entropy: 0.5558
    Epoch[4/10], Step [30/469], Reconst Loss: 169.9272, KL Div: 1.3552, kl_div_c: 1.8060, Entropy: 0.4966
    Epoch[4/10], Step [40/469], Reconst Loss: 176.4989, KL Div: 1.3721, kl_div_c: 1.7663, Entropy: 0.5363
    Epoch[4/10], Step [50/469], Reconst Loss: 164.6479, KL Div: 1.3430, kl_div_c: 1.8243, Entropy: 0.4783
    Epoch[4/10], Step [60/469], Reconst Loss: 174.2584, KL Div: 1.3315, kl_div_c: 1.8084, Entropy: 0.4942
    Epoch[4/10], Step [70/469], Reconst Loss: 176.1416, KL Div: 1.4404, kl_div_c: 1.7999, Entropy: 0.5027
    Epoch[4/10], Step [80/469], Reconst Loss: 173.9711, KL Div: 1.2957, kl_div_c: 1.7620, Entropy: 0.5405
    Epoch[4/10], Step [90/469], Reconst Loss: 170.2739, KL Div: 1.3884, kl_div_c: 1.8321, Entropy: 0.4705
    Epoch[4/10], Step [100/469], Reconst Loss: 171.4061, KL Div: 1.3428, kl_div_c: 1.8257, Entropy: 0.4769
    Epoch[4/10], Step [110/469], Reconst Loss: 173.8055, KL Div: 1.3908, kl_div_c: 1.8176, Entropy: 0.4849
    Epoch[4/10], Step [120/469], Reconst Loss: 170.5880, KL Div: 1.3989, kl_div_c: 1.8574, Entropy: 0.4452
    Epoch[4/10], Step [130/469], Reconst Loss: 169.6180, KL Div: 1.3408, kl_div_c: 1.7666, Entropy: 0.5360
    Epoch[4/10], Step [140/469], Reconst Loss: 177.2692, KL Div: 1.3827, kl_div_c: 1.8545, Entropy: 0.4481
    Epoch[4/10], Step [150/469], Reconst Loss: 176.3629, KL Div: 1.4497, kl_div_c: 1.8469, Entropy: 0.4556
    Epoch[4/10], Step [160/469], Reconst Loss: 170.6961, KL Div: 1.3844, kl_div_c: 1.8375, Entropy: 0.4651
    Epoch[4/10], Step [170/469], Reconst Loss: 165.3121, KL Div: 1.3273, kl_div_c: 1.8351, Entropy: 0.4675
    Epoch[4/10], Step [180/469], Reconst Loss: 172.4039, KL Div: 1.4178, kl_div_c: 1.8798, Entropy: 0.4227
    Epoch[4/10], Step [190/469], Reconst Loss: 168.7992, KL Div: 1.3935, kl_div_c: 1.8520, Entropy: 0.4506
    Epoch[4/10], Step [200/469], Reconst Loss: 163.5796, KL Div: 1.3746, kl_div_c: 1.8655, Entropy: 0.4371
    Epoch[4/10], Step [210/469], Reconst Loss: 171.2927, KL Div: 1.4533, kl_div_c: 1.8890, Entropy: 0.4136
    Epoch[4/10], Step [220/469], Reconst Loss: 162.2920, KL Div: 1.3349, kl_div_c: 1.8663, Entropy: 0.4363
    Epoch[4/10], Step [230/469], Reconst Loss: 167.9840, KL Div: 1.3684, kl_div_c: 1.8771, Entropy: 0.4254
    Epoch[4/10], Step [240/469], Reconst Loss: 165.2819, KL Div: 1.3736, kl_div_c: 1.8219, Entropy: 0.4806
    Epoch[4/10], Step [250/469], Reconst Loss: 168.2566, KL Div: 1.4094, kl_div_c: 1.8894, Entropy: 0.4132
    Epoch[4/10], Step [260/469], Reconst Loss: 171.5186, KL Div: 1.2814, kl_div_c: 1.8179, Entropy: 0.4846
    Epoch[4/10], Step [270/469], Reconst Loss: 169.3375, KL Div: 1.4157, kl_div_c: 1.8671, Entropy: 0.4355
    Epoch[4/10], Step [280/469], Reconst Loss: 167.2933, KL Div: 1.2956, kl_div_c: 1.8301, Entropy: 0.4725
    Epoch[4/10], Step [290/469], Reconst Loss: 166.8329, KL Div: 1.3164, kl_div_c: 1.8263, Entropy: 0.4763
    Epoch[4/10], Step [300/469], Reconst Loss: 163.8744, KL Div: 1.3072, kl_div_c: 1.8720, Entropy: 0.4306
    Epoch[4/10], Step [310/469], Reconst Loss: 165.8472, KL Div: 1.4656, kl_div_c: 1.9285, Entropy: 0.3741
    Epoch[4/10], Step [320/469], Reconst Loss: 170.3104, KL Div: 1.2857, kl_div_c: 1.8319, Entropy: 0.4707
    Epoch[4/10], Step [330/469], Reconst Loss: 166.7438, KL Div: 1.3997, kl_div_c: 1.8975, Entropy: 0.4050
    Epoch[4/10], Step [340/469], Reconst Loss: 160.8812, KL Div: 1.3206, kl_div_c: 1.8571, Entropy: 0.4455
    Epoch[4/10], Step [350/469], Reconst Loss: 162.1740, KL Div: 1.3050, kl_div_c: 1.8678, Entropy: 0.4348
    Epoch[4/10], Step [360/469], Reconst Loss: 164.0703, KL Div: 1.3653, kl_div_c: 1.9023, Entropy: 0.4003
    Epoch[4/10], Step [370/469], Reconst Loss: 169.9361, KL Div: 1.3369, kl_div_c: 1.8949, Entropy: 0.4077
    Epoch[4/10], Step [380/469], Reconst Loss: 175.0498, KL Div: 1.3410, kl_div_c: 1.8694, Entropy: 0.4332
    Epoch[4/10], Step [390/469], Reconst Loss: 171.1697, KL Div: 1.3530, kl_div_c: 1.8641, Entropy: 0.4385
    Epoch[4/10], Step [400/469], Reconst Loss: 159.5576, KL Div: 1.2953, kl_div_c: 1.8808, Entropy: 0.4218
    Epoch[4/10], Step [410/469], Reconst Loss: 160.7331, KL Div: 1.3970, kl_div_c: 1.9063, Entropy: 0.3963
    Epoch[4/10], Step [420/469], Reconst Loss: 165.4639, KL Div: 1.3031, kl_div_c: 1.9043, Entropy: 0.3983
    Epoch[4/10], Step [430/469], Reconst Loss: 163.1567, KL Div: 1.2829, kl_div_c: 1.8611, Entropy: 0.4415
    Epoch[4/10], Step [440/469], Reconst Loss: 168.7412, KL Div: 1.3323, kl_div_c: 1.9370, Entropy: 0.3656
    Epoch[4/10], Step [450/469], Reconst Loss: 167.3279, KL Div: 1.3449, kl_div_c: 1.9091, Entropy: 0.3935
    Epoch[4/10], Step [460/469], Reconst Loss: 171.4743, KL Div: 1.3303, kl_div_c: 1.9302, Entropy: 0.3723
    Epoch[5/10], Step [10/469], Reconst Loss: 165.5883, KL Div: 1.3263, kl_div_c: 1.9346, Entropy: 0.3680
    Epoch[5/10], Step [20/469], Reconst Loss: 169.7424, KL Div: 1.3308, kl_div_c: 1.8913, Entropy: 0.4112
    Epoch[5/10], Step [30/469], Reconst Loss: 164.1293, KL Div: 1.2940, kl_div_c: 1.8672, Entropy: 0.4353
    Epoch[5/10], Step [40/469], Reconst Loss: 164.5310, KL Div: 1.3109, kl_div_c: 1.9001, Entropy: 0.4025
    Epoch[5/10], Step [50/469], Reconst Loss: 167.2325, KL Div: 1.3452, kl_div_c: 1.9544, Entropy: 0.3481
    Epoch[5/10], Step [60/469], Reconst Loss: 160.6992, KL Div: 1.3418, kl_div_c: 1.9340, Entropy: 0.3686
    Epoch[5/10], Step [70/469], Reconst Loss: 164.3478, KL Div: 1.3894, kl_div_c: 1.9317, Entropy: 0.3709
    Epoch[5/10], Step [80/469], Reconst Loss: 170.3328, KL Div: 1.3035, kl_div_c: 1.9051, Entropy: 0.3975
    Epoch[5/10], Step [90/469], Reconst Loss: 156.9251, KL Div: 1.4179, kl_div_c: 1.9477, Entropy: 0.3548
    Epoch[5/10], Step [100/469], Reconst Loss: 162.8214, KL Div: 1.2429, kl_div_c: 1.9006, Entropy: 0.4020
    Epoch[5/10], Step [110/469], Reconst Loss: 166.8151, KL Div: 1.4161, kl_div_c: 1.9285, Entropy: 0.3741
    Epoch[5/10], Step [120/469], Reconst Loss: 161.2068, KL Div: 1.3066, kl_div_c: 1.9314, Entropy: 0.3712
    Epoch[5/10], Step [130/469], Reconst Loss: 163.7292, KL Div: 1.2818, kl_div_c: 1.9192, Entropy: 0.3834
    Epoch[5/10], Step [140/469], Reconst Loss: 172.1300, KL Div: 1.3181, kl_div_c: 1.9368, Entropy: 0.3658
    Epoch[5/10], Step [150/469], Reconst Loss: 164.9560, KL Div: 1.2747, kl_div_c: 1.9414, Entropy: 0.3612
    Epoch[5/10], Step [160/469], Reconst Loss: 157.5615, KL Div: 1.2904, kl_div_c: 1.9421, Entropy: 0.3605
    Epoch[5/10], Step [170/469], Reconst Loss: 171.9198, KL Div: 1.2184, kl_div_c: 1.8804, Entropy: 0.4222
    Epoch[5/10], Step [180/469], Reconst Loss: 157.4635, KL Div: 1.3357, kl_div_c: 1.9662, Entropy: 0.3364
    Epoch[5/10], Step [190/469], Reconst Loss: 162.0916, KL Div: 1.2607, kl_div_c: 1.9307, Entropy: 0.3719
    Epoch[5/10], Step [200/469], Reconst Loss: 162.1835, KL Div: 1.2461, kl_div_c: 1.9325, Entropy: 0.3701
    Epoch[5/10], Step [210/469], Reconst Loss: 160.9282, KL Div: 1.2698, kl_div_c: 1.9402, Entropy: 0.3624
    Epoch[5/10], Step [220/469], Reconst Loss: 162.5893, KL Div: 1.2761, kl_div_c: 1.9398, Entropy: 0.3627
    Epoch[5/10], Step [230/469], Reconst Loss: 161.5721, KL Div: 1.3231, kl_div_c: 1.9638, Entropy: 0.3388
    Epoch[5/10], Step [240/469], Reconst Loss: 163.5063, KL Div: 1.2845, kl_div_c: 1.9291, Entropy: 0.3735
    Epoch[5/10], Step [250/469], Reconst Loss: 160.6834, KL Div: 1.4275, kl_div_c: 2.0216, Entropy: 0.2810
    Epoch[5/10], Step [260/469], Reconst Loss: 158.7650, KL Div: 1.3304, kl_div_c: 1.9740, Entropy: 0.3286
    Epoch[5/10], Step [270/469], Reconst Loss: 157.9880, KL Div: 1.3661, kl_div_c: 1.9896, Entropy: 0.3130
    Epoch[5/10], Step [280/469], Reconst Loss: 165.5455, KL Div: 1.2816, kl_div_c: 1.9948, Entropy: 0.3077
    Epoch[5/10], Step [290/469], Reconst Loss: 159.4724, KL Div: 1.3295, kl_div_c: 2.0009, Entropy: 0.3016
    Epoch[5/10], Step [300/469], Reconst Loss: 162.2845, KL Div: 1.3010, kl_div_c: 1.9690, Entropy: 0.3336
    Epoch[5/10], Step [310/469], Reconst Loss: 168.9976, KL Div: 1.2706, kl_div_c: 1.8936, Entropy: 0.4090
    Epoch[5/10], Step [320/469], Reconst Loss: 159.0304, KL Div: 1.2639, kl_div_c: 1.9908, Entropy: 0.3118
    Epoch[5/10], Step [330/469], Reconst Loss: 166.6517, KL Div: 1.3154, kl_div_c: 2.0278, Entropy: 0.2748
    Epoch[5/10], Step [340/469], Reconst Loss: 159.9053, KL Div: 1.2912, kl_div_c: 1.9512, Entropy: 0.3514
    Epoch[5/10], Step [350/469], Reconst Loss: 155.9325, KL Div: 1.3355, kl_div_c: 2.0142, Entropy: 0.2884
    Epoch[5/10], Step [360/469], Reconst Loss: 160.4123, KL Div: 1.2447, kl_div_c: 1.9365, Entropy: 0.3661
    Epoch[5/10], Step [370/469], Reconst Loss: 154.9869, KL Div: 1.2357, kl_div_c: 1.9629, Entropy: 0.3396
    Epoch[5/10], Step [380/469], Reconst Loss: 162.1890, KL Div: 1.2611, kl_div_c: 1.9502, Entropy: 0.3524
    Epoch[5/10], Step [390/469], Reconst Loss: 168.0313, KL Div: 1.2454, kl_div_c: 1.9922, Entropy: 0.3104
    Epoch[5/10], Step [400/469], Reconst Loss: 159.5363, KL Div: 1.3342, kl_div_c: 1.9916, Entropy: 0.3110
    Epoch[5/10], Step [410/469], Reconst Loss: 161.0063, KL Div: 1.3273, kl_div_c: 1.9987, Entropy: 0.3039
    Epoch[5/10], Step [420/469], Reconst Loss: 162.2911, KL Div: 1.2522, kl_div_c: 1.9955, Entropy: 0.3071
    Epoch[5/10], Step [430/469], Reconst Loss: 160.6063, KL Div: 1.3149, kl_div_c: 1.9348, Entropy: 0.3678
    Epoch[5/10], Step [440/469], Reconst Loss: 160.8354, KL Div: 1.2459, kl_div_c: 1.9213, Entropy: 0.3813
    Epoch[5/10], Step [450/469], Reconst Loss: 165.3809, KL Div: 1.2987, kl_div_c: 2.0056, Entropy: 0.2969
    Epoch[5/10], Step [460/469], Reconst Loss: 158.5304, KL Div: 1.2590, kl_div_c: 2.0149, Entropy: 0.2877
    Epoch[6/10], Step [10/469], Reconst Loss: 164.1347, KL Div: 1.1823, kl_div_c: 1.9602, Entropy: 0.3424
    Epoch[6/10], Step [20/469], Reconst Loss: 161.7077, KL Div: 1.2496, kl_div_c: 2.0160, Entropy: 0.2866
    Epoch[6/10], Step [30/469], Reconst Loss: 155.8423, KL Div: 1.1818, kl_div_c: 1.9631, Entropy: 0.3395
    Epoch[6/10], Step [40/469], Reconst Loss: 155.0816, KL Div: 1.2339, kl_div_c: 1.9760, Entropy: 0.3266
    Epoch[6/10], Step [50/469], Reconst Loss: 159.9976, KL Div: 1.1702, kl_div_c: 1.9469, Entropy: 0.3557
    Epoch[6/10], Step [60/469], Reconst Loss: 163.9716, KL Div: 1.1216, kl_div_c: 1.9148, Entropy: 0.3877
    Epoch[6/10], Step [70/469], Reconst Loss: 156.6215, KL Div: 1.1464, kl_div_c: 1.9321, Entropy: 0.3705
    Epoch[6/10], Step [80/469], Reconst Loss: 152.9305, KL Div: 1.2804, kl_div_c: 2.0278, Entropy: 0.2748
    Epoch[6/10], Step [90/469], Reconst Loss: 155.6976, KL Div: 1.1121, kl_div_c: 1.9215, Entropy: 0.3811
    Epoch[6/10], Step [100/469], Reconst Loss: 159.7683, KL Div: 1.2414, kl_div_c: 2.0050, Entropy: 0.2976
    Epoch[6/10], Step [110/469], Reconst Loss: 155.8253, KL Div: 1.1954, kl_div_c: 1.9583, Entropy: 0.3443
    Epoch[6/10], Step [120/469], Reconst Loss: 155.9601, KL Div: 1.2084, kl_div_c: 1.9937, Entropy: 0.3089
    Epoch[6/10], Step [130/469], Reconst Loss: 156.1791, KL Div: 1.2240, kl_div_c: 1.9772, Entropy: 0.3254
    Epoch[6/10], Step [140/469], Reconst Loss: 160.1133, KL Div: 1.1481, kl_div_c: 1.9726, Entropy: 0.3299
    Epoch[6/10], Step [150/469], Reconst Loss: 163.2961, KL Div: 1.2583, kl_div_c: 1.9641, Entropy: 0.3385
    Epoch[6/10], Step [160/469], Reconst Loss: 164.3067, KL Div: 1.2050, kl_div_c: 1.9866, Entropy: 0.3160
    Epoch[6/10], Step [170/469], Reconst Loss: 156.6214, KL Div: 1.3127, kl_div_c: 1.9817, Entropy: 0.3208
    Epoch[6/10], Step [180/469], Reconst Loss: 159.9145, KL Div: 1.2464, kl_div_c: 1.9864, Entropy: 0.3162
    Epoch[6/10], Step [190/469], Reconst Loss: 165.2392, KL Div: 1.1541, kl_div_c: 1.9892, Entropy: 0.3134
    Epoch[6/10], Step [200/469], Reconst Loss: 160.2490, KL Div: 1.1666, kl_div_c: 1.9501, Entropy: 0.3525
    Epoch[6/10], Step [210/469], Reconst Loss: 163.0526, KL Div: 1.2581, kl_div_c: 2.0410, Entropy: 0.2616
    Epoch[6/10], Step [220/469], Reconst Loss: 158.4046, KL Div: 1.2698, kl_div_c: 2.0215, Entropy: 0.2811
    Epoch[6/10], Step [230/469], Reconst Loss: 149.7410, KL Div: 1.1526, kl_div_c: 1.9735, Entropy: 0.3291
    Epoch[6/10], Step [240/469], Reconst Loss: 147.3053, KL Div: 1.1812, kl_div_c: 2.0136, Entropy: 0.2890
    Epoch[6/10], Step [250/469], Reconst Loss: 157.9728, KL Div: 1.1474, kl_div_c: 2.0054, Entropy: 0.2972
    Epoch[6/10], Step [260/469], Reconst Loss: 149.5576, KL Div: 1.1631, kl_div_c: 2.0209, Entropy: 0.2817
    Epoch[6/10], Step [270/469], Reconst Loss: 160.8791, KL Div: 1.2520, kl_div_c: 2.0054, Entropy: 0.2972
    Epoch[6/10], Step [280/469], Reconst Loss: 159.5240, KL Div: 1.1372, kl_div_c: 1.9713, Entropy: 0.3313
    Epoch[6/10], Step [290/469], Reconst Loss: 164.5749, KL Div: 1.2136, kl_div_c: 2.0258, Entropy: 0.2768
    Epoch[6/10], Step [300/469], Reconst Loss: 154.3990, KL Div: 1.2151, kl_div_c: 2.0593, Entropy: 0.2433
    Epoch[6/10], Step [310/469], Reconst Loss: 158.5007, KL Div: 1.1336, kl_div_c: 2.0039, Entropy: 0.2987
    Epoch[6/10], Step [320/469], Reconst Loss: 151.0188, KL Div: 1.1807, kl_div_c: 2.0279, Entropy: 0.2747
    Epoch[6/10], Step [330/469], Reconst Loss: 160.5061, KL Div: 1.1377, kl_div_c: 2.0024, Entropy: 0.3002
    Epoch[6/10], Step [340/469], Reconst Loss: 155.6936, KL Div: 1.1932, kl_div_c: 1.9676, Entropy: 0.3350
    Epoch[6/10], Step [350/469], Reconst Loss: 155.8612, KL Div: 1.0936, kl_div_c: 1.9664, Entropy: 0.3362
    Epoch[6/10], Step [360/469], Reconst Loss: 152.8806, KL Div: 1.2153, kl_div_c: 2.0551, Entropy: 0.2475
    Epoch[6/10], Step [370/469], Reconst Loss: 149.4819, KL Div: 1.1231, kl_div_c: 2.0024, Entropy: 0.3002
    Epoch[6/10], Step [380/469], Reconst Loss: 156.3932, KL Div: 1.0986, kl_div_c: 1.9803, Entropy: 0.3223
    Epoch[6/10], Step [390/469], Reconst Loss: 158.6521, KL Div: 1.0863, kl_div_c: 1.9780, Entropy: 0.3246
    Epoch[6/10], Step [400/469], Reconst Loss: 165.1944, KL Div: 1.0654, kl_div_c: 1.9617, Entropy: 0.3409
    Epoch[6/10], Step [410/469], Reconst Loss: 157.3154, KL Div: 1.1047, kl_div_c: 1.9976, Entropy: 0.3050
    Epoch[6/10], Step [420/469], Reconst Loss: 154.8550, KL Div: 1.2934, kl_div_c: 2.0121, Entropy: 0.2904
    Epoch[6/10], Step [430/469], Reconst Loss: 156.4101, KL Div: 1.1350, kl_div_c: 2.0089, Entropy: 0.2937
    Epoch[6/10], Step [440/469], Reconst Loss: 152.1521, KL Div: 1.1147, kl_div_c: 1.9705, Entropy: 0.3321
    Epoch[6/10], Step [450/469], Reconst Loss: 163.2875, KL Div: 1.1554, kl_div_c: 2.0263, Entropy: 0.2763
    Epoch[6/10], Step [460/469], Reconst Loss: 158.5182, KL Div: 1.1199, kl_div_c: 1.9884, Entropy: 0.3142
    Epoch[7/10], Step [10/469], Reconst Loss: 162.3872, KL Div: 1.0304, kl_div_c: 1.9481, Entropy: 0.3545
    Epoch[7/10], Step [20/469], Reconst Loss: 155.5876, KL Div: 1.0865, kl_div_c: 1.9956, Entropy: 0.3070
    Epoch[7/10], Step [30/469], Reconst Loss: 154.8699, KL Div: 1.1638, kl_div_c: 2.0429, Entropy: 0.2596
    Epoch[7/10], Step [40/469], Reconst Loss: 158.2438, KL Div: 1.1311, kl_div_c: 1.9759, Entropy: 0.3267
    Epoch[7/10], Step [50/469], Reconst Loss: 161.0991, KL Div: 1.0627, kl_div_c: 2.0079, Entropy: 0.2947
    Epoch[7/10], Step [60/469], Reconst Loss: 150.3547, KL Div: 1.1494, kl_div_c: 2.0364, Entropy: 0.2662
    Epoch[7/10], Step [70/469], Reconst Loss: 158.2367, KL Div: 1.1330, kl_div_c: 1.9548, Entropy: 0.3478
    Epoch[7/10], Step [80/469], Reconst Loss: 156.5142, KL Div: 1.0902, kl_div_c: 2.0369, Entropy: 0.2657
    Epoch[7/10], Step [90/469], Reconst Loss: 157.9923, KL Div: 1.1279, kl_div_c: 2.0373, Entropy: 0.2653
    Epoch[7/10], Step [100/469], Reconst Loss: 157.4344, KL Div: 1.1772, kl_div_c: 2.0526, Entropy: 0.2500
    Epoch[7/10], Step [110/469], Reconst Loss: 155.2825, KL Div: 1.0723, kl_div_c: 2.0196, Entropy: 0.2830
    Epoch[7/10], Step [120/469], Reconst Loss: 161.7881, KL Div: 1.0694, kl_div_c: 1.9958, Entropy: 0.3068
    Epoch[7/10], Step [130/469], Reconst Loss: 154.4339, KL Div: 1.1118, kl_div_c: 2.0613, Entropy: 0.2413
    Epoch[7/10], Step [140/469], Reconst Loss: 147.8534, KL Div: 1.0912, kl_div_c: 2.0317, Entropy: 0.2709
    Epoch[7/10], Step [150/469], Reconst Loss: 156.4002, KL Div: 1.1248, kl_div_c: 2.0407, Entropy: 0.2618
    Epoch[7/10], Step [160/469], Reconst Loss: 156.8348, KL Div: 1.2340, kl_div_c: 2.0302, Entropy: 0.2724
    Epoch[7/10], Step [170/469], Reconst Loss: 148.3575, KL Div: 1.0381, kl_div_c: 2.0253, Entropy: 0.2772
    Epoch[7/10], Step [180/469], Reconst Loss: 158.4935, KL Div: 1.0938, kl_div_c: 2.0818, Entropy: 0.2208
    Epoch[7/10], Step [190/469], Reconst Loss: 158.7010, KL Div: 1.0464, kl_div_c: 2.0233, Entropy: 0.2793
    Epoch[7/10], Step [200/469], Reconst Loss: 150.1181, KL Div: 1.0403, kl_div_c: 2.0203, Entropy: 0.2823
    Epoch[7/10], Step [210/469], Reconst Loss: 154.7293, KL Div: 1.0571, kl_div_c: 2.0110, Entropy: 0.2916
    Epoch[7/10], Step [220/469], Reconst Loss: 153.9424, KL Div: 1.2033, kl_div_c: 2.0397, Entropy: 0.2629
    Epoch[7/10], Step [230/469], Reconst Loss: 155.8625, KL Div: 1.0075, kl_div_c: 2.0064, Entropy: 0.2962
    Epoch[7/10], Step [240/469], Reconst Loss: 155.0371, KL Div: 1.0230, kl_div_c: 2.0128, Entropy: 0.2898
    Epoch[7/10], Step [250/469], Reconst Loss: 162.2056, KL Div: 0.9931, kl_div_c: 1.9924, Entropy: 0.3102
    Epoch[7/10], Step [260/469], Reconst Loss: 152.2703, KL Div: 1.1326, kl_div_c: 2.0436, Entropy: 0.2590
    Epoch[7/10], Step [270/469], Reconst Loss: 152.0639, KL Div: 1.1217, kl_div_c: 2.0048, Entropy: 0.2978
    Epoch[7/10], Step [280/469], Reconst Loss: 149.6535, KL Div: 1.1271, kl_div_c: 2.0796, Entropy: 0.2230
    Epoch[7/10], Step [290/469], Reconst Loss: 158.7172, KL Div: 0.9521, kl_div_c: 1.9727, Entropy: 0.3299
    Epoch[7/10], Step [300/469], Reconst Loss: 154.3011, KL Div: 1.1259, kl_div_c: 2.0499, Entropy: 0.2527
    Epoch[7/10], Step [310/469], Reconst Loss: 160.5274, KL Div: 0.9756, kl_div_c: 1.9617, Entropy: 0.3409
    Epoch[7/10], Step [320/469], Reconst Loss: 153.7051, KL Div: 1.0764, kl_div_c: 2.0205, Entropy: 0.2821
    Epoch[7/10], Step [330/469], Reconst Loss: 160.6029, KL Div: 0.9560, kl_div_c: 1.9977, Entropy: 0.3049
    Epoch[7/10], Step [340/469], Reconst Loss: 148.0925, KL Div: 1.0687, kl_div_c: 2.0971, Entropy: 0.2055
    Epoch[7/10], Step [350/469], Reconst Loss: 158.2940, KL Div: 0.9617, kl_div_c: 1.9590, Entropy: 0.3436
    Epoch[7/10], Step [360/469], Reconst Loss: 153.9412, KL Div: 0.9974, kl_div_c: 1.9886, Entropy: 0.3140
    Epoch[7/10], Step [370/469], Reconst Loss: 146.7866, KL Div: 1.0369, kl_div_c: 2.0462, Entropy: 0.2564
    Epoch[7/10], Step [380/469], Reconst Loss: 156.2618, KL Div: 0.9932, kl_div_c: 1.9907, Entropy: 0.3119
    Epoch[7/10], Step [390/469], Reconst Loss: 151.4121, KL Div: 0.9288, kl_div_c: 1.9533, Entropy: 0.3493
    Epoch[7/10], Step [400/469], Reconst Loss: 151.1146, KL Div: 1.0224, kl_div_c: 2.0725, Entropy: 0.2301
    Epoch[7/10], Step [410/469], Reconst Loss: 153.6425, KL Div: 1.0530, kl_div_c: 2.0650, Entropy: 0.2376
    Epoch[7/10], Step [420/469], Reconst Loss: 150.8103, KL Div: 0.9121, kl_div_c: 1.9763, Entropy: 0.3263
    Epoch[7/10], Step [430/469], Reconst Loss: 160.9752, KL Div: 1.0201, kl_div_c: 2.0576, Entropy: 0.2450
    Epoch[7/10], Step [440/469], Reconst Loss: 157.2688, KL Div: 1.0656, kl_div_c: 2.0199, Entropy: 0.2827
    Epoch[7/10], Step [450/469], Reconst Loss: 160.6410, KL Div: 1.0029, kl_div_c: 2.0466, Entropy: 0.2559
    Epoch[7/10], Step [460/469], Reconst Loss: 150.1829, KL Div: 0.9707, kl_div_c: 2.0483, Entropy: 0.2543
    Epoch[8/10], Step [10/469], Reconst Loss: 154.5176, KL Div: 0.9197, kl_div_c: 1.9940, Entropy: 0.3086
    Epoch[8/10], Step [20/469], Reconst Loss: 148.4035, KL Div: 0.9792, kl_div_c: 2.0638, Entropy: 0.2388
    Epoch[8/10], Step [30/469], Reconst Loss: 159.1358, KL Div: 1.0018, kl_div_c: 2.0354, Entropy: 0.2672
    Epoch[8/10], Step [40/469], Reconst Loss: 154.3939, KL Div: 1.0937, kl_div_c: 2.0729, Entropy: 0.2297
    Epoch[8/10], Step [50/469], Reconst Loss: 154.6187, KL Div: 0.9052, kl_div_c: 2.0039, Entropy: 0.2987
    Epoch[8/10], Step [60/469], Reconst Loss: 147.9331, KL Div: 0.9231, kl_div_c: 2.0177, Entropy: 0.2849
    Epoch[8/10], Step [70/469], Reconst Loss: 152.5321, KL Div: 0.9403, kl_div_c: 2.0126, Entropy: 0.2900
    Epoch[8/10], Step [80/469], Reconst Loss: 153.7991, KL Div: 0.9319, kl_div_c: 1.9928, Entropy: 0.3097
    Epoch[8/10], Step [90/469], Reconst Loss: 153.7357, KL Div: 0.9516, kl_div_c: 2.0032, Entropy: 0.2994
    Epoch[8/10], Step [100/469], Reconst Loss: 148.4437, KL Div: 0.9628, kl_div_c: 2.0302, Entropy: 0.2723
    Epoch[8/10], Step [110/469], Reconst Loss: 145.4585, KL Div: 1.0655, kl_div_c: 2.0888, Entropy: 0.2138
    Epoch[8/10], Step [120/469], Reconst Loss: 153.2159, KL Div: 0.9878, kl_div_c: 2.0757, Entropy: 0.2269
    Epoch[8/10], Step [130/469], Reconst Loss: 152.5813, KL Div: 0.9595, kl_div_c: 1.9683, Entropy: 0.3343
    Epoch[8/10], Step [140/469], Reconst Loss: 152.1160, KL Div: 0.9613, kl_div_c: 2.0469, Entropy: 0.2557
    Epoch[8/10], Step [150/469], Reconst Loss: 149.2025, KL Div: 0.9289, kl_div_c: 2.0689, Entropy: 0.2337
    Epoch[8/10], Step [160/469], Reconst Loss: 146.2250, KL Div: 0.9609, kl_div_c: 2.0332, Entropy: 0.2693
    Epoch[8/10], Step [170/469], Reconst Loss: 152.7787, KL Div: 0.9649, kl_div_c: 2.0397, Entropy: 0.2629
    Epoch[8/10], Step [180/469], Reconst Loss: 156.6590, KL Div: 0.9724, kl_div_c: 2.0260, Entropy: 0.2766
    Epoch[8/10], Step [190/469], Reconst Loss: 154.4292, KL Div: 1.0239, kl_div_c: 2.0653, Entropy: 0.2372
    Epoch[8/10], Step [200/469], Reconst Loss: 150.3895, KL Div: 0.9448, kl_div_c: 2.0248, Entropy: 0.2778
    Epoch[8/10], Step [210/469], Reconst Loss: 155.4995, KL Div: 0.9985, kl_div_c: 2.0499, Entropy: 0.2526
    Epoch[8/10], Step [220/469], Reconst Loss: 146.5903, KL Div: 0.9398, kl_div_c: 2.0615, Entropy: 0.2411
    Epoch[8/10], Step [230/469], Reconst Loss: 153.6233, KL Div: 0.9550, kl_div_c: 2.0980, Entropy: 0.2046
    Epoch[8/10], Step [240/469], Reconst Loss: 153.3412, KL Div: 0.9474, kl_div_c: 2.0446, Entropy: 0.2580
    Epoch[8/10], Step [250/469], Reconst Loss: 148.0147, KL Div: 0.8528, kl_div_c: 2.0239, Entropy: 0.2787
    Epoch[8/10], Step [260/469], Reconst Loss: 152.8505, KL Div: 0.8888, kl_div_c: 2.0409, Entropy: 0.2617
    Epoch[8/10], Step [270/469], Reconst Loss: 155.0471, KL Div: 0.8856, kl_div_c: 2.0017, Entropy: 0.3008
    Epoch[8/10], Step [280/469], Reconst Loss: 153.7457, KL Div: 0.9824, kl_div_c: 2.0351, Entropy: 0.2675
    Epoch[8/10], Step [290/469], Reconst Loss: 162.8667, KL Div: 0.8179, kl_div_c: 1.9839, Entropy: 0.3186
    Epoch[8/10], Step [300/469], Reconst Loss: 157.9347, KL Div: 0.8454, kl_div_c: 1.9695, Entropy: 0.3331
    Epoch[8/10], Step [310/469], Reconst Loss: 149.7734, KL Div: 0.9150, kl_div_c: 2.0344, Entropy: 0.2682
    Epoch[8/10], Step [320/469], Reconst Loss: 153.9634, KL Div: 0.8053, kl_div_c: 1.9905, Entropy: 0.3121
    Epoch[8/10], Step [330/469], Reconst Loss: 153.0221, KL Div: 1.0388, kl_div_c: 2.0078, Entropy: 0.2948
    Epoch[8/10], Step [340/469], Reconst Loss: 150.8868, KL Div: 0.8154, kl_div_c: 1.9919, Entropy: 0.3107
    Epoch[8/10], Step [350/469], Reconst Loss: 153.6908, KL Div: 0.8682, kl_div_c: 2.0120, Entropy: 0.2906
    Epoch[8/10], Step [360/469], Reconst Loss: 148.8206, KL Div: 0.8612, kl_div_c: 2.0272, Entropy: 0.2754
    Epoch[8/10], Step [370/469], Reconst Loss: 156.3354, KL Div: 0.8789, kl_div_c: 2.0448, Entropy: 0.2578
    Epoch[8/10], Step [380/469], Reconst Loss: 151.7118, KL Div: 0.8877, kl_div_c: 2.0017, Entropy: 0.3009
    Epoch[8/10], Step [390/469], Reconst Loss: 146.5988, KL Div: 0.8157, kl_div_c: 2.0156, Entropy: 0.2870
    Epoch[8/10], Step [400/469], Reconst Loss: 159.4468, KL Div: 0.8918, kl_div_c: 2.0261, Entropy: 0.2765
    Epoch[8/10], Step [410/469], Reconst Loss: 145.5415, KL Div: 0.9502, kl_div_c: 2.0727, Entropy: 0.2299
    Epoch[8/10], Step [420/469], Reconst Loss: 150.6207, KL Div: 0.7938, kl_div_c: 1.9761, Entropy: 0.3265
    Epoch[8/10], Step [430/469], Reconst Loss: 153.7910, KL Div: 0.7687, kl_div_c: 2.0021, Entropy: 0.3005
    Epoch[8/10], Step [440/469], Reconst Loss: 152.4401, KL Div: 0.8952, kl_div_c: 2.0267, Entropy: 0.2759
    Epoch[8/10], Step [450/469], Reconst Loss: 149.7760, KL Div: 0.8876, kl_div_c: 2.0209, Entropy: 0.2817
    Epoch[8/10], Step [460/469], Reconst Loss: 156.8558, KL Div: 0.8240, kl_div_c: 2.0070, Entropy: 0.2955
    Epoch[9/10], Step [10/469], Reconst Loss: 155.5471, KL Div: 0.8583, kl_div_c: 2.0496, Entropy: 0.2529
    Epoch[9/10], Step [20/469], Reconst Loss: 151.2643, KL Div: 0.8956, kl_div_c: 2.0517, Entropy: 0.2508
    Epoch[9/10], Step [30/469], Reconst Loss: 147.3738, KL Div: 0.9220, kl_div_c: 2.0864, Entropy: 0.2162
    Epoch[9/10], Step [40/469], Reconst Loss: 149.8771, KL Div: 0.7811, kl_div_c: 1.9903, Entropy: 0.3123
    Epoch[9/10], Step [50/469], Reconst Loss: 159.3881, KL Div: 0.8119, kl_div_c: 2.0703, Entropy: 0.2322
    Epoch[9/10], Step [60/469], Reconst Loss: 151.5984, KL Div: 0.8050, kl_div_c: 2.0559, Entropy: 0.2466
    Epoch[9/10], Step [70/469], Reconst Loss: 158.1736, KL Div: 0.9297, kl_div_c: 2.0553, Entropy: 0.2473
    Epoch[9/10], Step [80/469], Reconst Loss: 156.8215, KL Div: 0.8323, kl_div_c: 2.0288, Entropy: 0.2737
    Epoch[9/10], Step [90/469], Reconst Loss: 148.7245, KL Div: 0.7864, kl_div_c: 2.0028, Entropy: 0.2998
    Epoch[9/10], Step [100/469], Reconst Loss: 148.5547, KL Div: 0.7777, kl_div_c: 1.9952, Entropy: 0.3073
    Epoch[9/10], Step [110/469], Reconst Loss: 153.1445, KL Div: 0.7663, kl_div_c: 1.9888, Entropy: 0.3138
    Epoch[9/10], Step [120/469], Reconst Loss: 153.3111, KL Div: 0.7940, kl_div_c: 2.0359, Entropy: 0.2667
    Epoch[9/10], Step [130/469], Reconst Loss: 148.7674, KL Div: 0.8402, kl_div_c: 2.0359, Entropy: 0.2667
    Epoch[9/10], Step [140/469], Reconst Loss: 148.1756, KL Div: 0.8043, kl_div_c: 2.0676, Entropy: 0.2349
    Epoch[9/10], Step [150/469], Reconst Loss: 149.0778, KL Div: 0.8525, kl_div_c: 2.0242, Entropy: 0.2784
    Epoch[9/10], Step [160/469], Reconst Loss: 143.1754, KL Div: 0.8015, kl_div_c: 2.0700, Entropy: 0.2326
    Epoch[9/10], Step [170/469], Reconst Loss: 150.9504, KL Div: 0.8415, kl_div_c: 2.0495, Entropy: 0.2531
    Epoch[9/10], Step [180/469], Reconst Loss: 148.3069, KL Div: 0.8434, kl_div_c: 2.1244, Entropy: 0.1782
    Epoch[9/10], Step [190/469], Reconst Loss: 151.9523, KL Div: 0.7947, kl_div_c: 1.9830, Entropy: 0.3195
    Epoch[9/10], Step [200/469], Reconst Loss: 152.1234, KL Div: 0.8597, kl_div_c: 2.0447, Entropy: 0.2579
    Epoch[9/10], Step [210/469], Reconst Loss: 151.2022, KL Div: 0.8747, kl_div_c: 2.0559, Entropy: 0.2467
    Epoch[9/10], Step [220/469], Reconst Loss: 149.2528, KL Div: 0.7774, kl_div_c: 2.0663, Entropy: 0.2363
    Epoch[9/10], Step [230/469], Reconst Loss: 152.0250, KL Div: 0.6610, kl_div_c: 1.9355, Entropy: 0.3671
    Epoch[9/10], Step [240/469], Reconst Loss: 156.0632, KL Div: 0.8161, kl_div_c: 2.0460, Entropy: 0.2566
    Epoch[9/10], Step [250/469], Reconst Loss: 145.0891, KL Div: 0.6476, kl_div_c: 1.9791, Entropy: 0.3235
    Epoch[9/10], Step [260/469], Reconst Loss: 145.1267, KL Div: 0.7285, kl_div_c: 2.0288, Entropy: 0.2738
    Epoch[9/10], Step [270/469], Reconst Loss: 148.8006, KL Div: 0.8623, kl_div_c: 2.0642, Entropy: 0.2384
    Epoch[9/10], Step [280/469], Reconst Loss: 150.9214, KL Div: 0.8626, kl_div_c: 2.0859, Entropy: 0.2167
    Epoch[9/10], Step [290/469], Reconst Loss: 152.2598, KL Div: 0.7846, kl_div_c: 2.0488, Entropy: 0.2538
    Epoch[9/10], Step [300/469], Reconst Loss: 144.6252, KL Div: 0.7658, kl_div_c: 2.0327, Entropy: 0.2699
    Epoch[9/10], Step [310/469], Reconst Loss: 148.3706, KL Div: 0.6606, kl_div_c: 2.0092, Entropy: 0.2933
    Epoch[9/10], Step [320/469], Reconst Loss: 152.2620, KL Div: 0.7785, kl_div_c: 2.0566, Entropy: 0.2460
    Epoch[9/10], Step [330/469], Reconst Loss: 153.7658, KL Div: 0.7170, kl_div_c: 2.0660, Entropy: 0.2366
    Epoch[9/10], Step [340/469], Reconst Loss: 146.1825, KL Div: 0.7898, kl_div_c: 2.0596, Entropy: 0.2430
    Epoch[9/10], Step [350/469], Reconst Loss: 144.4139, KL Div: 0.7817, kl_div_c: 2.0716, Entropy: 0.2310
    Epoch[9/10], Step [360/469], Reconst Loss: 149.3730, KL Div: 0.6521, kl_div_c: 1.9956, Entropy: 0.3070
    Epoch[9/10], Step [370/469], Reconst Loss: 151.2605, KL Div: 0.6720, kl_div_c: 2.0043, Entropy: 0.2983
    Epoch[9/10], Step [380/469], Reconst Loss: 147.8917, KL Div: 0.6988, kl_div_c: 2.0387, Entropy: 0.2639
    Epoch[9/10], Step [390/469], Reconst Loss: 152.1039, KL Div: 0.7257, kl_div_c: 2.0621, Entropy: 0.2405
    Epoch[9/10], Step [400/469], Reconst Loss: 149.0283, KL Div: 0.7943, kl_div_c: 2.0752, Entropy: 0.2274
    Epoch[9/10], Step [410/469], Reconst Loss: 149.8138, KL Div: 0.7016, kl_div_c: 2.0348, Entropy: 0.2678
    Epoch[9/10], Step [420/469], Reconst Loss: 151.0238, KL Div: 0.6869, kl_div_c: 2.0284, Entropy: 0.2741
    Epoch[9/10], Step [430/469], Reconst Loss: 148.2190, KL Div: 0.6763, kl_div_c: 2.0183, Entropy: 0.2843
    Epoch[9/10], Step [440/469], Reconst Loss: 147.7690, KL Div: 0.7468, kl_div_c: 2.0117, Entropy: 0.2909
    Epoch[9/10], Step [450/469], Reconst Loss: 142.3318, KL Div: 0.9178, kl_div_c: 2.1150, Entropy: 0.1876
    Epoch[9/10], Step [460/469], Reconst Loss: 153.3278, KL Div: 0.7200, kl_div_c: 2.0640, Entropy: 0.2386
    Epoch[10/10], Step [10/469], Reconst Loss: 148.6433, KL Div: 0.6906, kl_div_c: 1.9992, Entropy: 0.3034
    Epoch[10/10], Step [20/469], Reconst Loss: 150.2955, KL Div: 0.8558, kl_div_c: 2.0174, Entropy: 0.2852
    Epoch[10/10], Step [30/469], Reconst Loss: 148.0037, KL Div: 0.6789, kl_div_c: 2.0055, Entropy: 0.2971
    Epoch[10/10], Step [40/469], Reconst Loss: 153.3064, KL Div: 0.6795, kl_div_c: 1.9820, Entropy: 0.3206
    Epoch[10/10], Step [50/469], Reconst Loss: 154.7547, KL Div: 0.6182, kl_div_c: 2.0123, Entropy: 0.2903
    Epoch[10/10], Step [60/469], Reconst Loss: 141.4574, KL Div: 0.6672, kl_div_c: 2.0480, Entropy: 0.2546
    Epoch[10/10], Step [70/469], Reconst Loss: 141.3128, KL Div: 0.8191, kl_div_c: 2.0621, Entropy: 0.2405
    Epoch[10/10], Step [80/469], Reconst Loss: 152.3186, KL Div: 0.6865, kl_div_c: 2.0148, Entropy: 0.2878
    Epoch[10/10], Step [90/469], Reconst Loss: 147.3266, KL Div: 0.5626, kl_div_c: 1.9856, Entropy: 0.3170
    Epoch[10/10], Step [100/469], Reconst Loss: 152.7325, KL Div: 0.7392, kl_div_c: 2.0758, Entropy: 0.2267
    Epoch[10/10], Step [110/469], Reconst Loss: 145.8946, KL Div: 0.6162, kl_div_c: 2.0165, Entropy: 0.2861
    Epoch[10/10], Step [120/469], Reconst Loss: 144.9716, KL Div: 0.6593, kl_div_c: 2.0083, Entropy: 0.2942
    Epoch[10/10], Step [130/469], Reconst Loss: 148.3238, KL Div: 0.6319, kl_div_c: 1.9744, Entropy: 0.3282
    Epoch[10/10], Step [140/469], Reconst Loss: 148.6694, KL Div: 0.6775, kl_div_c: 2.0882, Entropy: 0.2144
    Epoch[10/10], Step [150/469], Reconst Loss: 146.7234, KL Div: 0.5987, kl_div_c: 2.0166, Entropy: 0.2860
    Epoch[10/10], Step [160/469], Reconst Loss: 150.4580, KL Div: 0.6468, kl_div_c: 2.0579, Entropy: 0.2447
    Epoch[10/10], Step [170/469], Reconst Loss: 148.0875, KL Div: 0.6811, kl_div_c: 2.0060, Entropy: 0.2966
    Epoch[10/10], Step [180/469], Reconst Loss: 149.3493, KL Div: 0.7509, kl_div_c: 2.0146, Entropy: 0.2880
    Epoch[10/10], Step [190/469], Reconst Loss: 149.0126, KL Div: 0.6520, kl_div_c: 2.0470, Entropy: 0.2556
    Epoch[10/10], Step [200/469], Reconst Loss: 142.6222, KL Div: 0.8897, kl_div_c: 2.0650, Entropy: 0.2376
    Epoch[10/10], Step [210/469], Reconst Loss: 147.0392, KL Div: 0.6229, kl_div_c: 2.0677, Entropy: 0.2349
    Epoch[10/10], Step [220/469], Reconst Loss: 139.0383, KL Div: 0.7631, kl_div_c: 2.0212, Entropy: 0.2814
    Epoch[10/10], Step [230/469], Reconst Loss: 150.5562, KL Div: 0.6363, kl_div_c: 2.0891, Entropy: 0.2135
    Epoch[10/10], Step [240/469], Reconst Loss: 142.2820, KL Div: 0.6154, kl_div_c: 2.0159, Entropy: 0.2866
    Epoch[10/10], Step [250/469], Reconst Loss: 141.7477, KL Div: 0.5704, kl_div_c: 2.0475, Entropy: 0.2551
    Epoch[10/10], Step [260/469], Reconst Loss: 151.0742, KL Div: 0.5347, kl_div_c: 2.0058, Entropy: 0.2968
    Epoch[10/10], Step [270/469], Reconst Loss: 148.3958, KL Div: 0.5575, kl_div_c: 2.0156, Entropy: 0.2870
    Epoch[10/10], Step [280/469], Reconst Loss: 150.7678, KL Div: 0.5681, kl_div_c: 1.9942, Entropy: 0.3084
    Epoch[10/10], Step [290/469], Reconst Loss: 144.3584, KL Div: 0.5231, kl_div_c: 2.0140, Entropy: 0.2885
    Epoch[10/10], Step [300/469], Reconst Loss: 144.3884, KL Div: 0.5897, kl_div_c: 2.0431, Entropy: 0.2595
    Epoch[10/10], Step [310/469], Reconst Loss: 148.7691, KL Div: 0.5389, kl_div_c: 2.0175, Entropy: 0.2851
    Epoch[10/10], Step [320/469], Reconst Loss: 149.1942, KL Div: 0.6768, kl_div_c: 2.0161, Entropy: 0.2865
    Epoch[10/10], Step [330/469], Reconst Loss: 144.8170, KL Div: 0.5879, kl_div_c: 2.0444, Entropy: 0.2582
    Epoch[10/10], Step [340/469], Reconst Loss: 151.5385, KL Div: 0.5885, kl_div_c: 2.0642, Entropy: 0.2384
    Epoch[10/10], Step [350/469], Reconst Loss: 147.8734, KL Div: 0.6160, kl_div_c: 2.0693, Entropy: 0.2333
    Epoch[10/10], Step [360/469], Reconst Loss: 140.5097, KL Div: 0.6783, kl_div_c: 2.0857, Entropy: 0.2169
    Epoch[10/10], Step [370/469], Reconst Loss: 143.6862, KL Div: 0.5675, kl_div_c: 2.0655, Entropy: 0.2371
    Epoch[10/10], Step [380/469], Reconst Loss: 152.1318, KL Div: 0.6758, kl_div_c: 2.0414, Entropy: 0.2612
    Epoch[10/10], Step [390/469], Reconst Loss: 148.0763, KL Div: 0.4691, kl_div_c: 1.9909, Entropy: 0.3117
    Epoch[10/10], Step [400/469], Reconst Loss: 142.9638, KL Div: 0.6742, kl_div_c: 2.0418, Entropy: 0.2608
    Epoch[10/10], Step [410/469], Reconst Loss: 149.9526, KL Div: 0.5520, kl_div_c: 2.0228, Entropy: 0.2798
    Epoch[10/10], Step [420/469], Reconst Loss: 145.4588, KL Div: 0.6024, kl_div_c: 2.0701, Entropy: 0.2324
    Epoch[10/10], Step [430/469], Reconst Loss: 143.0976, KL Div: 0.5430, kl_div_c: 2.0911, Entropy: 0.2114
    Epoch[10/10], Step [440/469], Reconst Loss: 139.2160, KL Div: 0.5783, kl_div_c: 2.0692, Entropy: 0.2334
    Epoch[10/10], Step [450/469], Reconst Loss: 146.1561, KL Div: 0.5545, kl_div_c: 2.0472, Entropy: 0.2554
    Epoch[10/10], Step [460/469], Reconst Loss: 145.7618, KL Div: 0.5073, kl_div_c: 2.0518, Entropy: 0.2508

```python
plt.plot(nmi2)
plt.show()
```

```python
plt.figure(figsize=(10,5))
plot_reconstruction(model_G)
```

```python
plt.figure(figsize=(10,10))
plot_conditional_generation(model_G, fix_number=None)
```

```python
plt.figure(figsize=(10,5))
plot_conditional_generation(model_G, fix_number=1)
```

```python

```