```python
from google.colab import drive
drive.mount('/content/drive',force_remount=True)
```

    Mounted at /content/drive

```python
import os
os.chdir('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week4')
```

```python
!pip install tensorflow=='2.2.0'
```

    Collecting tensorflow==2.2.0
      Downloading tensorflow-2.2.0-cp37-cp37m-manylinux2010_x86_64.whl (516.2 MB)
         |████████████████████████████████| 516.2 MB 3.5 kB/s
    Requirement already satisfied: numpy<2.0,>=1.16.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.19.5)
    Requirement already satisfied: wrapt>=1.11.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.13.3)
    Requirement already satisfied: google-pasta>=0.1.8 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (0.2.0)
    Requirement already satisfied: opt-einsum>=2.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (3.3.0)
    Requirement already satisfied: six>=1.12.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.15.0)
    Requirement already satisfied: absl-py>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.0.0)
    Requirement already satisfied: wheel>=0.26 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (0.37.1)
    Requirement already satisfied: scipy==1.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.4.1)
    Requirement already satisfied: protobuf>=3.8.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (3.17.3)
    Requirement already satisfied: astunparse==1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.6.3)
    Requirement already satisfied: keras-preprocessing>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.1.2)
    Requirement already satisfied: termcolor>=1.1.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.1.0)
    Collecting tensorboard<2.3.0,>=2.2.0
      Downloading tensorboard-2.2.2-py3-none-any.whl (3.0 MB)
         |████████████████████████████████| 3.0 MB 62.5 MB/s
    Collecting gast==0.3.3
      Downloading gast-0.3.3-py2.py3-none-any.whl (9.7 kB)
    Collecting tensorflow-estimator<2.3.0,>=2.2.0
      Downloading tensorflow_estimator-2.2.0-py2.py3-none-any.whl (454 kB)
         |████████████████████████████████| 454 kB 85.7 MB/s
    Collecting h5py<2.11.0,>=2.10.0
      Downloading h5py-2.10.0-cp37-cp37m-manylinux1_x86_64.whl (2.9 MB)
         |████████████████████████████████| 2.9 MB 76.9 MB/s
    Requirement already satisfied: grpcio>=1.8.6 in /usr/local/lib/python3.7/dist-packages (from tensorflow==2.2.0) (1.43.0)
    Requirement already satisfied: markdown>=2.6.8 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.3.6)
    Requirement already satisfied: tensorboard-plugin-wit>=1.6.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.8.1)
    Requirement already satisfied: google-auth<2,>=1.6.3 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.35.0)
    Requirement already satisfied: werkzeug>=0.11.15 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.0.1)
    Requirement already satisfied: setuptools>=41.0.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (57.4.0)
    Requirement already satisfied: google-auth-oauthlib<0.5,>=0.4.1 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.6)
    Requirement already satisfied: requests<3,>=2.21.0 in /usr/local/lib/python3.7/dist-packages (from tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.23.0)
    Requirement already satisfied: pyasn1-modules>=0.2.1 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.2.8)
    Requirement already satisfied: cachetools<5.0,>=2.0.0 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.2.4)
    Requirement already satisfied: rsa<5,>=3.1.4 in /usr/local/lib/python3.7/dist-packages (from google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.8)
    Requirement already satisfied: requests-oauthlib>=0.7.0 in /usr/local/lib/python3.7/dist-packages (from google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.3.1)
    Requirement already satisfied: importlib-metadata>=4.4 in /usr/local/lib/python3.7/dist-packages (from markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (4.10.1)
    Requirement already satisfied: typing-extensions>=3.6.4 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.10.0.2)
    Requirement already satisfied: zipp>=0.5 in /usr/local/lib/python3.7/dist-packages (from importlib-metadata>=4.4->markdown>=2.6.8->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.7.0)
    Requirement already satisfied: pyasn1<0.5.0,>=0.4.6 in /usr/local/lib/python3.7/dist-packages (from pyasn1-modules>=0.2.1->google-auth<2,>=1.6.3->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (0.4.8)
    Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (1.24.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2021.10.8)
    Requirement already satisfied: idna<3,>=2.5 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (2.10)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests<3,>=2.21.0->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.0.4)
    Requirement already satisfied: oauthlib>=3.0.0 in /usr/local/lib/python3.7/dist-packages (from requests-oauthlib>=0.7.0->google-auth-oauthlib<0.5,>=0.4.1->tensorboard<2.3.0,>=2.2.0->tensorflow==2.2.0) (3.2.0)
    Installing collected packages: tensorflow-estimator, tensorboard, h5py, gast, tensorflow
      Attempting uninstall: tensorflow-estimator
        Found existing installation: tensorflow-estimator 2.7.0
        Uninstalling tensorflow-estimator-2.7.0:
          Successfully uninstalled tensorflow-estimator-2.7.0
      Attempting uninstall: tensorboard
        Found existing installation: tensorboard 2.7.0
        Uninstalling tensorboard-2.7.0:
          Successfully uninstalled tensorboard-2.7.0
      Attempting uninstall: h5py
        Found existing installation: h5py 3.1.0
        Uninstalling h5py-3.1.0:
          Successfully uninstalled h5py-3.1.0
      Attempting uninstall: gast
        Found existing installation: gast 0.4.0
        Uninstalling gast-0.4.0:
          Successfully uninstalled gast-0.4.0
      Attempting uninstall: tensorflow
        Found existing installation: tensorflow 2.7.0
        Uninstalling tensorflow-2.7.0:
          Successfully uninstalled tensorflow-2.7.0
    Successfully installed gast-0.3.3 h5py-2.10.0 tensorboard-2.2.2 tensorflow-2.2.0 tensorflow-estimator-2.2.0

```python
!pip install tensorflow_probability=='0.10.0'
```

    Collecting tensorflow_probability==0.10.0
      Downloading tensorflow_probability-0.10.0-py2.py3-none-any.whl (3.5 MB)
         |████████████████████████████████| 3.5 MB 14.2 MB/s
    Requirement already satisfied: numpy>=1.13.3 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.10.0) (1.19.5)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.10.0) (4.4.2)
    Requirement already satisfied: gast>=0.3.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.10.0) (0.3.3)
    Requirement already satisfied: six>=1.10.0 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.10.0) (1.15.0)
    Requirement already satisfied: cloudpickle>=1.2.2 in /usr/local/lib/python3.7/dist-packages (from tensorflow_probability==0.10.0) (1.3.0)
    Installing collected packages: tensorflow-probability
      Attempting uninstall: tensorflow-probability
        Found existing installation: tensorflow-probability 0.15.0
        Uninstalling tensorflow-probability-0.15.0:
          Successfully uninstalled tensorflow-probability-0.15.0
    Successfully installed tensorflow-probability-0.10.0

# Programming Assignment

## VAE for the CelebA dataset

### Instructions
In this programming assignment, you will implement the variational autoencoder algorithm for an image dataset of celebrity faces. You will use the trained encoder and decoder networks to reconstruct and generate images. You will also see how the latent space encodes high-level information about the images.

Some code cells are provided for you in the notebook. You should avoid editing provided code, and make sure to execute the cells in order to avoid unexpected errors. Some cells begin with the line:

`#### GRADED CELL ####`

Don't move or edit this first line - this is what the automatic grader looks for to recognise graded cells. These cells require you to write your own code to complete them, and are automatically graded when you submit the notebook. Don't edit the function name or signature provided in these cells, otherwise the automatic grader might not function properly.

### How to submit

Complete all the tasks you are asked for in the worksheet. When you have finished and are happy with your code, press the Submit Assignment button at the top of this notebook.

### Let's get started!

We'll start running some imports, and loading the dataset. Do not edit the existing imports in the following cell. If you would like to make further Tensorflow imports, you should add them here.

```python
#### PACKAGE IMPORTS ####

# Run this cell first to import all required packages. Do not make any imports elsewhere in the notebook

import tensorflow as tf
import tensorflow_probability as tfp
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import os

from tensorflow.keras import Sequential, Model
from tensorflow.keras.layers import (Dense, Flatten, Reshape, Concatenate, Conv2D,
                                     UpSampling2D, BatchNormalization)
tfd = tfp.distributions
tfb = tfp.bijectors
tfpl = tfp.layers

# If you would like to make further imports from tensorflow, add them here

```

#### The Large-scale CelebFaces Attributes (CelebA) Dataset

For this assignment you will use a subset of the CelebFaces Attributes (CelebA) dataset. The full dataset contains over 200K images CelebA contains thousands of colour images of the faces of celebrities, together with tagged attributes such as 'Smiling', 'Wearing glasses', or 'Wearing lipstick'. It also contains information about bounding boxes and facial part localisation. CelebA is a popular dataset that is commonly used for face attribute recognition, face detection, landmark (or facial part) localization, and face editing & synthesis.

* Z. Liu, P. Luo, X. Wang, and X. Tang. "Deep Learning Face Attributes in the Wild", Proceedings of International Conference on Computer Vision (ICCV), 2015.

You can read about the dataset in more detail [here](http://mmlab.ie.cuhk.edu.hk/projects/CelebA.html).

Your goal is to implement the variational autoencoder algorithm for a subset of the CelebA dataset. For practical reasons we will keep the dataset and the network size relatively small.

#### Import the data

The dataset required for this project can be downloaded from the following link:

https://drive.google.com/file/d/1qzbcYRjxO6yU_usb4CXyGObUEjIVYWAp/view?usp=sharing

You should unzip this file and store in Drive for use in this Colab notebook.

```python
#import zipfile
#with zipfile.ZipFile("/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week4/CelebA subset.zip","r") as zip_ref:
#  zip_ref.extractall('CeleAsubset')
```

```python
# Run this cell to connect to your Drive folder

#from google.colab import drive
#drive.mount('/content/gdrive')
```

#### Load the dataset

The following functions will be useful for loading and preprocessing the dataset. The subset you will use for this assignment consists of 10,000 training images, 1000 validation images and 1000 test images. These examples have been chosen to respect the original training / validation / test split of the dataset.

```python
# Functions for loading and preprocessing the images

def load_image(filepath):
    raw_img = tf.io.read_file(filepath)
    img_tensor_int = tf.image.decode_jpeg(raw_img, channels=3)
    img_tensor_flt = tf.image.convert_image_dtype(img_tensor_int, tf.float32)
    return img_tensor_flt, img_tensor_flt

def load_dataset(split):
    train_list_ds = tf.data.Dataset.list_files('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week4/CeleAsubset/{}/*.jpg'.format(split), shuffle=False)
    train_ds = train_list_ds.map(load_image)
    return train_ds
```

```python
# Load the training, validation and testing datasets splits

train_ds = load_dataset('train')
val_ds = load_dataset('val')
test_ds = load_dataset('test')
```

```python
# Display a few examples

n_examples_shown = 6
f, axs = plt.subplots(1, n_examples_shown, figsize=(16, 3))

for j, image in enumerate(train_ds.take(n_examples_shown)):
    axs[j].imshow(image[0])
    axs[j].axis('off')
```

```python
# Batch the Dataset objects

batch_size = 32
train_ds = train_ds.batch(batch_size)
val_ds = val_ds.batch(batch_size)
test_ds = test_ds.batch(batch_size)
```

#### Mixture of Gaussians distribution

We will define a prior distribution that is a mixture of Gaussians. This is a more flexible distribution that is comprised of $K$ separate Gaussians, that are combined together with some weighting assigned to each.

Recall that the probability density function for a multivariate Gaussian distribution with mean $\mu\in\mathbb{R}^D$ and covariance matrix $\Sigma\in\mathbb{R}^{D\times D}$ is given by

$$
\mathcal{N}(\mathbf{z}; \mathbf{\mu}, \Sigma) = \frac{1}{(2\pi)^{D/2}|\Sigma|^{1/2}}
\exp\left(-\frac{1}{2}(\mathbf{z}-\mathbf{\mu})^T\Sigma^{-1}(\mathbf{z}-\mathbf{\mu})\right).
$$

A mixture of Gaussians with $K$ components defines $K$ Gaussians defined by means $\mathbf{\mu}_k$ and covariance matrices $\Sigma_k$, for $k=1,\ldots,K$. It also requires mixing coefficients $\pi_k$, $k=1,\ldots,K$ with $\sum_{k} \pi_k = 1$. These coefficients define a categorical distribution over the $K$ Gaussian components. To sample an event, we first sample from the categorical distribution, and then again from the corresponding Gaussian component.

The probability density function of the mixture of Gaussians is simply the weighted sum of probability density functions for each Gaussian component:

$$
p(\mathbf{z}) = \sum_{k=1}^K \pi_k \mathcal{N}(\mathbf{z}; \mathbf{\mu}_k, \Sigma_k)
$$

#### Define the prior distribution

You should now complete the following function to define the mixture of Gaussians distribution for the prior, for a given number of components and latent space dimension. Each Gaussian component will have a diagonal covariance matrix. This distribution will have fixed mixing coefficients, but trainable means and standard deviations.

* The function takes `num_modes` (number of components) and `latent_dim` as arguments
* Use the `tfd.MixtureSameFamily` for the prior distribution. Take a look at [the documentation](https://www.tensorflow.org/probability/api_docs/python/tfp/distributions/MixtureSameFamily) for this distribution
  * The constructor takes a `mixture_distribution` and `components_distribution` as required arguments
  * The `mixture_distribution` should be fixed to a uniform `tfd.Categorical` distribution, so that $pi_k = 1/K$ in the above equation. This argument will therefore not contain any trainable variables
  * The `components_distribution` should be a `tfd.MultivariateNormalDiag` distribution batch shape equal to `[num_modes]` and event shape equal to `[latent_dim]`.
    * The `tfd.MultivariateNormalDiag` distribution should have trainable `loc` parameter (initialised with a random normal distribution)  and trainable `scale_diag` parameter (initialised to ones)
    * The `scale_diag` variable should be enforced to be positive using `tfp.util.TransformedVariable` and the `tfb.Softplus` bijection

The function should return the instance of the `tfd.MixtureSameFamily` distribution.

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_prior(num_modes, latent_dim):
    """
    This function should create an instance of a MixtureSameFamily distribution
    according to the above specification.
    The function takes the num_modes and latent_dim as arguments, which should
    be used to define the distribution.
    Your function should then return the distribution instance.
    """
    mixture_distribution=tfd.Categorical(probs=(1./num_modes)*tf.ones([num_modes,]))
    components_distribution=tfd.MultivariateNormalDiag(loc=tf.Variable(tf.random.normal([num_modes,latent_dim]),trainable=True,dtype=tf.float32),
                                                       scale_diag=tfp.util.TransformedVariable(initial_value=tf.ones([num_modes,latent_dim]),
                                                                                bijector=tfb.Softplus()))
    prior=tfp.distributions.MixtureSameFamily(
    mixture_distribution, components_distribution, reparameterize=False,
    validate_args=False, allow_nan_stats=True, name='MixtureSameFamily'
    )
    return prior

```

```python
# Run your function to get the prior distribution with 2 components and latent_dim = 50

prior = get_prior(num_modes=2, latent_dim=50)
```

#### Define the encoder network

We will now define the encoder network as part of the VAE. First, we will define the `KLDivergenceRegularizer` to use in the encoder network to add the KL divergence part of the loss. This should be defined according to the following specification:

* The function takes the `prior_distribution` as an argument
* The function should use the `tfpl.KLDivergenceRegularizer` object to add the KL loss term
* The `tfpl.KLDivergenceRegularizer` should use a weight factor of 1.0 for the KL loss (standard ELBO objective)
* The KL loss cannot be computed exactly, so the `tfpl.KLDivergenceRegularizer` should compute a Monte Carlo approximation by drawing 3 samples from the posterior, and then averaging over the sample and batch axes

Your function should then return the `tfpl.KLDivergenceRegularizer` object.

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_kl_regularizer(prior_distribution):
    """
    This function should create an instance of the KLDivergenceRegularizer
    according to the above specification.
    The function takes the prior_distribution, which should be used to define
    the distribution.
    Your function should then return the KLDivergenceRegularizer instance.
    """
    divergence_regularizer = tfpl.KLDivergenceRegularizer(prior_distribution,
                                                          use_exact_kl=False,
                                                          test_points_fn=lambda q: q.sample(3),
                                                          test_points_reduce_axis=(0,1),
                                                          weight=1.0)
    return divergence_regularizer

```

```python
# Run your function to get the KLDivergenceRegularizer

kl_regularizer = get_kl_regularizer(prior)
```

You should now complete the following function to define the encoder network, according to the following specification:

* The function takes the `latent_dim` and `kl_regularizer` as arguments
* Use the `Sequential` class to define the model
  * The first layer should be a Conv2D layer with 32 filters, 4x4 kernel size, ReLU activation, stride of 2x2, and 'SAME' padding. It should also set the `input_shape` to `(64, 64, 3)`
  * The second layer should be a BatchNormalization layer
  * The third layer should be a Conv2D layer with 64 filters, 4x4 kernel size, ReLU activation, stride of 2x2, and 'SAME' padding
  * The fourth layer should be a BatchNormalization layer
  * The fifth layer should be a Conv2D layer with 128 filters, 4x4 kernel size, ReLU activation, stride of 2x2, and 'SAME' padding
  * The sixth layer should be a BatchNormalization layer
  * The seventh layer should be a Conv2D layer with 256 filters, 4x4 kernel size, ReLU activation, stride of 2x2, and 'SAME' padding
  * The eighth layer should be a BatchNormalization layer
  * The ninth layer should be a Flatten layer
  * The tenth layer should be a Dense layer with no activation function, and the right number of units to parameterise a `MultivariateNormalTriL` layer with event size equal to `latent_dim`
  * The final layer should be a `MultivariateNormalTriL` layer with event size equal to `latent_dim`, and it should use the `kl_regularizer` passed in as the argument as the activity regularizer

In total, your model should have 11 layers.

The function should then return the encoder model.

```python
#### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_encoder(latent_dim, kl_regularizer):
    """
    This function should build a CNN encoder model according to the above specification.
    The function takes latent_dim and kl_regularizer as arguments, which should be
    used to define the model.
    Your function should return the encoder model.
    """
    encoder=Sequential([
                    Conv2D(32,(4,4),strides=(2,2),activation='relu',padding='SAME',input_shape=(64,64,3)),
                    BatchNormalization(),
                    Conv2D(64,(4,4),strides=(2,2),activation='relu',padding='SAME'),
                    BatchNormalization(),
                    Conv2D(128,(4,4),strides=(2,2),activation='relu',padding='SAME'),
                    BatchNormalization(),
                    Conv2D(256,(4,4),strides=(2,2),activation='relu',padding='SAME'),
                    BatchNormalization(),
                    Flatten(),
                    Dense(tfpl.MultivariateNormalTriL.params_size(latent_dim)),
                    tfpl.MultivariateNormalTriL(latent_dim,activity_regularizer=kl_regularizer)])
    return encoder

```

```python
# Run your function to get the encoder

encoder = get_encoder(latent_dim=50, kl_regularizer=kl_regularizer)
```

    WARNING:tensorflow:From /usr/local/lib/python3.7/dist-packages/tensorflow/python/ops/linalg/linear_operator_lower_triangular.py:158: calling LinearOperator.__init__ (from tensorflow.python.ops.linalg.linear_operator) with graph_parents is deprecated and will be removed in a future version.
    Instructions for updating:
    Do not pass `graph_parents`.  They will  no longer be used.

```python
# Print the encoder summary

encoder.summary()
```

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    conv2d (Conv2D)              (None, 32, 32, 32)        1568
    _________________________________________________________________
    batch_normalization (BatchNo (None, 32, 32, 32)        128
    _________________________________________________________________
    conv2d_1 (Conv2D)            (None, 16, 16, 64)        32832
    _________________________________________________________________
    batch_normalization_1 (Batch (None, 16, 16, 64)        256
    _________________________________________________________________
    conv2d_2 (Conv2D)            (None, 8, 8, 128)         131200
    _________________________________________________________________
    batch_normalization_2 (Batch (None, 8, 8, 128)         512
    _________________________________________________________________
    conv2d_3 (Conv2D)            (None, 4, 4, 256)         524544
    _________________________________________________________________
    batch_normalization_3 (Batch (None, 4, 4, 256)         1024
    _________________________________________________________________
    flatten (Flatten)            (None, 4096)              0
    _________________________________________________________________
    dense (Dense)                (None, 1325)              5428525
    _________________________________________________________________
    multivariate_normal_tri_l (M ((None, 50), (None, 50))  200
    =================================================================
    Total params: 6,120,789
    Trainable params: 6,119,829
    Non-trainable params: 960
    _________________________________________________________________

#### Define the decoder network

You should now define the decoder network for the VAE, using the `Sequential` API. This should be a neural network that returns an IndependentBernoulli distribution of `event_shape=(64, 64, 3)`.

* The function takes the `latent_dim` as an argument
* Use the `Sequential` class to define the model
  * The first layer should be a Dense layer with 4096 units and ReLU activation. It should also set the `input_shape` to `(latent_dim,)`
  * The second layer should be a Reshape layer, that reshapes its input to `(4, 4, 256)`
  * The third layer should be an UpSampling2D layer with upsampling factor of `(2, 2)`
  * The fourth layer should be a Conv2D layer with 128 filters, 3x3 kernel size, ReLU activation, and 'SAME' padding
  * The fifth layer should be an UpSampling2D layer with upsampling factor of `(2, 2)`
  * The sixth layer should be a Conv2D layer with 64 filters, 3x3 kernel size, ReLU activation, and 'SAME' padding
  * The seventh layer should be an UpSampling2D layer with upsampling factor of `(2, 2)`
  * The eighth layer should be a Conv2D layer with 32 filters, 3x3 kernel size, ReLU activation, and 'SAME' padding
  * The ninth layer should be an UpSampling2D layer with upsampling factor of `(2, 2)`
  * The tenth layer should be a Conv2D layer with 128 filters, 3x3 kernel size, ReLU activation, and 'SAME' padding
  * The eleventh layer should be a Conv2D layer with 3 filters, 3x3 kernel size, no activation function, and 'SAME' padding
  * The twelfth layer should be a Flatten layer
  * The final layer should be a `IndependentBernoulli` layer with event size equal to `(64, 64, 3)`

In total, your model should have 13 layers.

The function should then return the decoder model.

```python
from tensorflow_probability.python.layers.distribution_layer import IndependentBernoulli
 #### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def get_decoder(latent_dim):
    """
    This function should build a CNN decoder model according to the above specification.
    The function takes latent_dim as an argument, which should be used to define the model.
    Your function should return the decoder model.
    """
    decoder=Sequential([
                    Dense(4096,activation='relu',input_shape=(latent_dim,)),
                    Reshape(target_shape=(4,4,256)),
                    UpSampling2D(size=(2,2)),
                    Conv2D(128,(3,3),activation='relu',padding='SAME'),
                    UpSampling2D(size=(2,2)),
                    Conv2D(64,(3,3),activation='relu',padding='SAME'),
                    UpSampling2D(size=(2,2)),
                    Conv2D(32,(3,3),activation='relu',padding='SAME'),
                    UpSampling2D(size=(2,2)),
                    Conv2D(128,(3,3),activation='relu',padding='SAME'),
                    Conv2D(3,(3,3),padding='SAME'),
                    Flatten(),
                    tfpl.IndependentBernoulli(event_shape=(64,64,3))])
    return decoder

```

```python
# Run your function to get the decoder

decoder = get_decoder(latent_dim=50)
```

```python
# Print the decoder summary

decoder.summary()
```

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_1 (Dense)              (None, 4096)              208896
    _________________________________________________________________
    reshape (Reshape)            (None, 4, 4, 256)         0
    _________________________________________________________________
    up_sampling2d (UpSampling2D) (None, 8, 8, 256)         0
    _________________________________________________________________
    conv2d_4 (Conv2D)            (None, 8, 8, 128)         295040
    _________________________________________________________________
    up_sampling2d_1 (UpSampling2 (None, 16, 16, 128)       0
    _________________________________________________________________
    conv2d_5 (Conv2D)            (None, 16, 16, 64)        73792
    _________________________________________________________________
    up_sampling2d_2 (UpSampling2 (None, 32, 32, 64)        0
    _________________________________________________________________
    conv2d_6 (Conv2D)            (None, 32, 32, 32)        18464
    _________________________________________________________________
    up_sampling2d_3 (UpSampling2 (None, 64, 64, 32)        0
    _________________________________________________________________
    conv2d_7 (Conv2D)            (None, 64, 64, 128)       36992
    _________________________________________________________________
    conv2d_8 (Conv2D)            (None, 64, 64, 3)         3459
    _________________________________________________________________
    flatten_1 (Flatten)          (None, 12288)             0
    _________________________________________________________________
    independent_bernoulli (Indep ((None, 64, 64, 3), (None 0
    =================================================================
    Total params: 636,643
    Trainable params: 636,643
    Non-trainable params: 0
    _________________________________________________________________

#### Link the encoder and decoder together

The following cell connects `encoder` and `decoder` to form the end-to-end architecture.

```python
# Connect the encoder and decoder

vae = Model(inputs=encoder.inputs, outputs=decoder(encoder.outputs))
```

#### Define the average reconstruction loss

You should now define the reconstruction loss that forms the remaining part of the negative ELBO objective. This function should take a batch of images of shape `(batch_size, 64, 64, 3)` in the first argument, and the output of the decoder after passing the batch of images through `vae` in the second argument.

The loss should be defined so that it returns
\begin{equation}
    -\frac{1}{n}\sum_{i=1}^n \log p(x_i|z_i)
\end{equation}
where $n$ is the batch size and $z_i$ is sampled from $q(z|x_i)$, the encoding distribution a.k.a. the approximate posterior. The value of this expression is always a scalar.

Expression (1) is, as you know, is an estimate of the (negative of the) batch's average expected reconstruction loss:

\begin{equation}
    -\frac{1}{n}\sum_{i=1}^n \mathrm{E}_{Z\sim q(z|x_i)}\big[\log p(x_i|Z)\big]
\end{equation}

_Hints_ :
- _You may find the `log_prob` method of the decoding distribution helpful._
- _The code you add does not need to be more than one line._

```python
 #### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def reconstruction_loss(batch_of_images, decoding_dist):
    """
    This function should compute and return the average expected reconstruction loss,
    as defined above.
    The function takes batch_of_images (Tensor containing a batch of input images to
    the encoder) and decoding_dist (output distribution of decoder after passing the
    image batch through the encoder and decoder) as arguments.
    The function should return the scalar average expected reconstruction loss.
    """
    return -tf.reduce_mean(decoding_dist.log_prob(batch_of_images))

```

#### Compile and fit the model

It's now time to compile and train the model. This may take some time, so as an alternative, you can also load a pre-trained model below if you wish.

The pre-trained model can be downloaded from the following link:

https://drive.google.com/file/d/15JMM4Pw4lnYvgR_5WOjW8ttkhglTGOGq/view?usp=sharing

You should unzip this file and store in Drive for use in this Colab notebook.

```python
# Compile the model

optimizer = tf.keras.optimizers.Adam(learning_rate=0.0005)
vae.compile(optimizer=optimizer, loss=reconstruction_loss)
```

```python
# EITHER... compile and fit the model

vae.fit(train_ds, validation_data=val_ds, epochs=1000)
```

    Epoch 1/1000
    313/313 [==============================] - 23s 74ms/step - loss: 7140.3750 - val_loss: 7029.6094
    Epoch 2/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6717.7231 - val_loss: 6613.9370
    Epoch 3/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6590.0298 - val_loss: 6562.4009
    Epoch 4/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6516.4360 - val_loss: 6494.5234
    Epoch 5/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6465.1475 - val_loss: 6452.4707
    Epoch 6/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6427.2642 - val_loss: 6425.9302
    Epoch 7/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6397.0498 - val_loss: 6392.1162
    Epoch 8/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6368.1182 - val_loss: 6368.2822
    Epoch 9/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6346.4595 - val_loss: 6357.8218
    Epoch 10/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6327.4053 - val_loss: 6337.4731
    Epoch 11/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6308.6318 - val_loss: 6321.9810
    Epoch 12/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6292.5845 - val_loss: 6318.3760
    Epoch 13/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6278.9614 - val_loss: 6299.2368
    Epoch 14/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6266.1343 - val_loss: 6296.1660
    Epoch 15/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6255.6631 - val_loss: 6280.7544
    Epoch 16/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6245.7036 - val_loss: 6280.8354
    Epoch 17/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6238.1211 - val_loss: 6285.6343
    Epoch 18/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6231.7163 - val_loss: 6275.7300
    Epoch 19/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6228.1724 - val_loss: 6275.6265
    Epoch 20/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6228.5156 - val_loss: 6297.8242
    Epoch 21/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6235.7964 - val_loss: 6281.5566
    Epoch 22/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6235.5366 - val_loss: 6317.9849
    Epoch 23/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6226.6670 - val_loss: 6311.0986
    Epoch 24/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6217.8613 - val_loss: 6300.5801
    Epoch 25/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6211.6577 - val_loss: 6322.3906
    Epoch 26/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6209.3193 - val_loss: 6309.4238
    Epoch 27/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6206.9287 - val_loss: 6325.4648
    Epoch 28/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6209.3843 - val_loss: 6300.6548
    Epoch 29/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6213.9863 - val_loss: 6283.5161
    Epoch 30/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6215.8965 - val_loss: 6304.6445
    Epoch 31/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6214.6631 - val_loss: 6320.8872
    Epoch 32/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6209.1494 - val_loss: 6334.1738
    Epoch 33/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6203.8086 - val_loss: 6321.6738
    Epoch 34/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6196.0234 - val_loss: 6292.6641
    Epoch 35/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6189.5342 - val_loss: 6291.5796
    Epoch 36/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6183.7935 - val_loss: 6281.3398
    Epoch 37/1000
    313/313 [==============================] - 14s 44ms/step - loss: 6179.5591 - val_loss: 6272.2119
    Epoch 38/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6176.8130 - val_loss: 6267.2437
    Epoch 39/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6176.3193 - val_loss: 6266.9087
    Epoch 40/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6177.6309 - val_loss: 6272.8877
    Epoch 41/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6181.6309 - val_loss: 6295.2065
    Epoch 42/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6183.2510 - val_loss: 6318.3550
    Epoch 43/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6185.4077 - val_loss: 6330.8203
    Epoch 44/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6186.2783 - val_loss: 6322.1548
    Epoch 45/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6179.8442 - val_loss: 6313.3364
    Epoch 46/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6174.6846 - val_loss: 6309.3560
    Epoch 47/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6172.1235 - val_loss: 6320.8379
    Epoch 48/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6172.0786 - val_loss: 6327.1216
    Epoch 49/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6175.9722 - val_loss: 6351.2905
    Epoch 50/1000
    313/313 [==============================] - 14s 44ms/step - loss: 6184.7207 - val_loss: 6351.1255
    Epoch 51/1000
    313/313 [==============================] - 14s 45ms/step - loss: 6188.4492 - val_loss: 6326.7153
    Epoch 52/1000
    313/313 [==============================] - 14s 44ms/step - loss: 6181.8267 - val_loss: 6303.0356
    Epoch 53/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6172.1777 - val_loss: 6290.7500
    Epoch 54/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6163.7764 - val_loss: 6290.8345
    Epoch 55/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6158.3560 - val_loss: 6287.3027
    Epoch 56/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6153.9995 - val_loss: 6279.5537
    Epoch 57/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6150.7461 - val_loss: 6274.3340
    Epoch 58/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6149.0977 - val_loss: 6278.8081
    Epoch 59/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6148.7275 - val_loss: 6288.4390
    Epoch 60/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6151.4736 - val_loss: 6300.9014
    Epoch 61/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6155.4170 - val_loss: 6331.1392
    Epoch 62/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6158.7246 - val_loss: 6299.7129
    Epoch 63/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6156.1382 - val_loss: 6279.1250
    Epoch 64/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6150.3462 - val_loss: 6266.8984
    Epoch 65/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6146.7417 - val_loss: 6264.8916
    Epoch 66/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6146.3750 - val_loss: 6266.9521
    Epoch 67/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6147.6421 - val_loss: 6264.5151
    Epoch 68/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6150.6821 - val_loss: 6270.8301
    Epoch 69/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6152.2710 - val_loss: 6269.8765
    Epoch 70/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6150.0942 - val_loss: 6269.8516
    Epoch 71/1000
    313/313 [==============================] - 16s 53ms/step - loss: 6150.1060 - val_loss: 6269.8794
    Epoch 72/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6148.9746 - val_loss: 6263.8096
    Epoch 73/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6148.8643 - val_loss: 6265.8423
    Epoch 74/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6147.1958 - val_loss: 6269.2285
    Epoch 75/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6147.0928 - val_loss: 6270.7910
    Epoch 76/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6149.8550 - val_loss: 6273.5718
    Epoch 77/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6152.3237 - val_loss: 6266.6294
    Epoch 78/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6160.7632 - val_loss: 6284.1421
    Epoch 79/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6163.1079 - val_loss: 6315.7388
    Epoch 80/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6161.0195 - val_loss: 6309.1323
    Epoch 81/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6156.5850 - val_loss: 6294.1973
    Epoch 82/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6153.6094 - val_loss: 6294.2925
    Epoch 83/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6151.6919 - val_loss: 6278.0176
    Epoch 84/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6150.2002 - val_loss: 6289.6396
    Epoch 85/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6147.9590 - val_loss: 6299.0947
    Epoch 86/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6145.6855 - val_loss: 6293.3506
    Epoch 87/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6141.9165 - val_loss: 6282.4233
    Epoch 88/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6138.6030 - val_loss: 6289.2754
    Epoch 89/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6134.6636 - val_loss: 6286.0894
    Epoch 90/1000
    313/313 [==============================] - 14s 45ms/step - loss: 6131.2637 - val_loss: 6282.8174
    Epoch 91/1000
    313/313 [==============================] - 14s 45ms/step - loss: 6128.5205 - val_loss: 6283.8696
    Epoch 92/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6127.3721 - val_loss: 6284.3804
    Epoch 93/1000
    313/313 [==============================] - 14s 43ms/step - loss: 6125.6738 - val_loss: 6273.6494
    Epoch 94/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6124.3828 - val_loss: 6277.0518
    Epoch 95/1000
    313/313 [==============================] - 13s 42ms/step - loss: 6124.3809 - val_loss: 6283.6787
    Epoch 96/1000
    313/313 [==============================] - 13s 42ms/step - loss: 6125.7705 - val_loss: 6294.7388
    Epoch 97/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6128.0410 - val_loss: 6299.2021
    Epoch 98/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6131.1665 - val_loss: 6309.3286
    Epoch 99/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6134.6631 - val_loss: 6337.6514
    Epoch 100/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6135.8198 - val_loss: 6362.9028
    Epoch 101/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6133.8472 - val_loss: 6356.4429
    Epoch 102/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6130.3452 - val_loss: 6350.0244
    Epoch 103/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6126.9609 - val_loss: 6337.3896
    Epoch 104/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6123.6265 - val_loss: 6332.4775
    Epoch 105/1000
    313/313 [==============================] - 14s 44ms/step - loss: 6121.3608 - val_loss: 6326.9121
    Epoch 106/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6120.0186 - val_loss: 6327.9805
    Epoch 107/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6120.0908 - val_loss: 6328.5557
    Epoch 108/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6121.7910 - val_loss: 6331.7114
    Epoch 109/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6124.1899 - val_loss: 6303.8950
    Epoch 110/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6122.0674 - val_loss: 6295.1211
    Epoch 111/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6118.8838 - val_loss: 6294.1519
    Epoch 112/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6117.2158 - val_loss: 6303.8198
    Epoch 113/1000
    313/313 [==============================] - 17s 53ms/step - loss: 6117.1006 - val_loss: 6298.1206
    Epoch 114/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6118.1157 - val_loss: 6305.1318
    Epoch 115/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6119.5391 - val_loss: 6324.6792
    Epoch 116/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6122.9077 - val_loss: 6331.9414
    Epoch 117/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6127.1816 - val_loss: 6317.9229
    Epoch 118/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6132.6338 - val_loss: 6318.0762
    Epoch 119/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6137.3921 - val_loss: 6328.0186
    Epoch 120/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6139.7998 - val_loss: 6332.6919
    Epoch 121/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6134.3726 - val_loss: 6346.2271
    Epoch 122/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6128.2988 - val_loss: 6347.9385
    Epoch 123/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6122.3638 - val_loss: 6346.8125
    Epoch 124/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6116.8286 - val_loss: 6346.8804
    Epoch 125/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6112.5659 - val_loss: 6337.7192
    Epoch 126/1000
    313/313 [==============================] - 14s 45ms/step - loss: 6109.2075 - val_loss: 6333.7329
    Epoch 127/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6107.4912 - val_loss: 6339.3950
    Epoch 128/1000
    313/313 [==============================] - 15s 46ms/step - loss: 6105.3076 - val_loss: 6338.2837
    Epoch 129/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6104.0493 - val_loss: 6351.4238
    Epoch 130/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6102.7754 - val_loss: 6352.1201
    Epoch 131/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6102.2769 - val_loss: 6340.5039
    Epoch 132/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6101.8018 - val_loss: 6330.2881
    Epoch 133/1000
    313/313 [==============================] - 17s 53ms/step - loss: 6102.6758 - val_loss: 6330.8784
    Epoch 134/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6103.3672 - val_loss: 6348.7744
    Epoch 135/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6103.2417 - val_loss: 6325.5396
    Epoch 136/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6102.2402 - val_loss: 6326.9902
    Epoch 137/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6101.6294 - val_loss: 6376.2935
    Epoch 138/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6103.4458 - val_loss: 6381.8120
    Epoch 139/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6106.4170 - val_loss: 6353.1934
    Epoch 140/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6111.9102 - val_loss: 6316.3467
    Epoch 141/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6116.5654 - val_loss: 6305.2573
    Epoch 142/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6119.1509 - val_loss: 6309.3574
    Epoch 143/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6121.6934 - val_loss: 6305.6284
    Epoch 144/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6124.8638 - val_loss: 6303.7803
    Epoch 145/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6128.2363 - val_loss: 6297.8135
    Epoch 146/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6128.3394 - val_loss: 6299.2114
    Epoch 147/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6125.8394 - val_loss: 6294.3530
    Epoch 148/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6124.3979 - val_loss: 6300.8999
    Epoch 149/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6121.8682 - val_loss: 6301.8730
    Epoch 150/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6118.3301 - val_loss: 6302.2290
    Epoch 151/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6115.4922 - val_loss: 6297.4209
    Epoch 152/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6113.1738 - val_loss: 6295.7061
    Epoch 153/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6109.9590 - val_loss: 6295.1035
    Epoch 154/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6107.4258 - val_loss: 6296.6064
    Epoch 155/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6105.0698 - val_loss: 6294.8345
    Epoch 156/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6103.1328 - val_loss: 6294.6299
    Epoch 157/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6101.4131 - val_loss: 6292.7632
    Epoch 158/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6100.1323 - val_loss: 6291.3120
    Epoch 159/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6100.1362 - val_loss: 6293.4507
    Epoch 160/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6100.9458 - val_loss: 6290.7964
    Epoch 161/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6103.9810 - val_loss: 6293.7788
    Epoch 162/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6107.8838 - val_loss: 6301.3770
    Epoch 163/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6111.0566 - val_loss: 6310.9604
    Epoch 164/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6112.9922 - val_loss: 6297.4258
    Epoch 165/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6110.0430 - val_loss: 6301.3105
    Epoch 166/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6107.2119 - val_loss: 6299.4868
    Epoch 167/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6105.0259 - val_loss: 6304.7417
    Epoch 168/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6104.8262 - val_loss: 6309.3691
    Epoch 169/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6105.3755 - val_loss: 6333.4644
    Epoch 170/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6107.6982 - val_loss: 6343.9146
    Epoch 171/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6109.7886 - val_loss: 6338.2080
    Epoch 172/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6111.3955 - val_loss: 6324.7197
    Epoch 173/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6111.4272 - val_loss: 6325.1216
    Epoch 174/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6110.8154 - val_loss: 6308.8408
    Epoch 175/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6109.8584 - val_loss: 6297.2544
    Epoch 176/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6108.4487 - val_loss: 6296.0337
    Epoch 177/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6107.0742 - val_loss: 6292.4141
    Epoch 178/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6105.6182 - val_loss: 6292.6890
    Epoch 179/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6104.3662 - val_loss: 6289.9604
    Epoch 180/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6101.7720 - val_loss: 6290.2900
    Epoch 181/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6099.5654 - val_loss: 6289.8638
    Epoch 182/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6097.8433 - val_loss: 6287.5854
    Epoch 183/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6097.0435 - val_loss: 6291.4561
    Epoch 184/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6096.9839 - val_loss: 6289.8369
    Epoch 185/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6098.0254 - val_loss: 6289.6304
    Epoch 186/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6099.1855 - val_loss: 6289.4834
    Epoch 187/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6099.2930 - val_loss: 6286.3687
    Epoch 188/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6098.1382 - val_loss: 6291.2515
    Epoch 189/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6098.1670 - val_loss: 6292.0039
    Epoch 190/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6098.7593 - val_loss: 6293.9185
    Epoch 191/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6098.3696 - val_loss: 6297.3398
    Epoch 192/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6097.4580 - val_loss: 6298.1006
    Epoch 193/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6096.9214 - val_loss: 6300.0605
    Epoch 194/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6095.4434 - val_loss: 6296.1934
    Epoch 195/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6094.6152 - val_loss: 6301.3208
    Epoch 196/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6094.8354 - val_loss: 6304.2153
    Epoch 197/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6095.9180 - val_loss: 6297.9932
    Epoch 198/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6096.8516 - val_loss: 6297.3350
    Epoch 199/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6098.3574 - val_loss: 6295.2510
    Epoch 200/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6100.2656 - val_loss: 6296.2559
    Epoch 201/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6103.4868 - val_loss: 6300.9365
    Epoch 202/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6106.9111 - val_loss: 6304.4727
    Epoch 203/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6112.8887 - val_loss: 6328.5376
    Epoch 204/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6120.2725 - val_loss: 6339.1558
    Epoch 205/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6122.3882 - val_loss: 6322.9536
    Epoch 206/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6119.5669 - val_loss: 6309.6577
    Epoch 207/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6115.9014 - val_loss: 6306.4434
    Epoch 208/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6112.4912 - val_loss: 6308.3813
    Epoch 209/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6110.1118 - val_loss: 6314.6108
    Epoch 210/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6107.4751 - val_loss: 6313.9814
    Epoch 211/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6104.4380 - val_loss: 6311.8452
    Epoch 212/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6102.4106 - val_loss: 6310.7012
    Epoch 213/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6100.3779 - val_loss: 6309.4697
    Epoch 214/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6099.0049 - val_loss: 6304.8296
    Epoch 215/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6097.8364 - val_loss: 6312.6133
    Epoch 216/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6097.4106 - val_loss: 6326.9321
    Epoch 217/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6097.4741 - val_loss: 6341.0371
    Epoch 218/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6099.1577 - val_loss: 6351.4648
    Epoch 219/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6103.2939 - val_loss: 6327.0762
    Epoch 220/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6105.2246 - val_loss: 6323.5571
    Epoch 221/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6103.4541 - val_loss: 6348.4585
    Epoch 222/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6098.4253 - val_loss: 6338.0547
    Epoch 223/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6093.9614 - val_loss: 6325.2153
    Epoch 224/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6089.8228 - val_loss: 6319.9414
    Epoch 225/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6087.8530 - val_loss: 6322.7959
    Epoch 226/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6086.3242 - val_loss: 6331.5723
    Epoch 227/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6085.5708 - val_loss: 6326.0762
    Epoch 228/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6084.6680 - val_loss: 6324.3110
    Epoch 229/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6083.9937 - val_loss: 6313.2661
    Epoch 230/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6083.4526 - val_loss: 6305.2490
    Epoch 231/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6082.6309 - val_loss: 6303.6328
    Epoch 232/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6082.4941 - val_loss: 6303.0356
    Epoch 233/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6082.0195 - val_loss: 6307.1870
    Epoch 234/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6082.3384 - val_loss: 6307.1284
    Epoch 235/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6083.2588 - val_loss: 6306.9883
    Epoch 236/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6085.6172 - val_loss: 6312.2520
    Epoch 237/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6086.9912 - val_loss: 6308.8179
    Epoch 238/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6087.5889 - val_loss: 6369.4541
    Epoch 239/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6086.1416 - val_loss: 6323.0337
    Epoch 240/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6084.6880 - val_loss: 6299.8921
    Epoch 241/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6085.2534 - val_loss: 6302.7988
    Epoch 242/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6086.8257 - val_loss: 6313.2798
    Epoch 243/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6088.5332 - val_loss: 6324.4341
    Epoch 244/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6090.6143 - val_loss: 6334.6440
    Epoch 245/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6092.0898 - val_loss: 6347.8887
    Epoch 246/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6093.3481 - val_loss: 6354.3418
    Epoch 247/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6094.0952 - val_loss: 6353.2358
    Epoch 248/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6094.9082 - val_loss: 6351.5449
    Epoch 249/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6095.6089 - val_loss: 6342.7690
    Epoch 250/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6097.3242 - val_loss: 6337.7637
    Epoch 251/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6097.7876 - val_loss: 6348.3281
    Epoch 252/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6098.5933 - val_loss: 6385.6406
    Epoch 253/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6097.7090 - val_loss: 6384.7656
    Epoch 254/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6095.2520 - val_loss: 6370.0068
    Epoch 255/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6092.8438 - val_loss: 6361.6284
    Epoch 256/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6090.9136 - val_loss: 6364.9663
    Epoch 257/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6089.7734 - val_loss: 6362.8931
    Epoch 258/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6089.0557 - val_loss: 6353.2769
    Epoch 259/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6088.3926 - val_loss: 6346.7261
    Epoch 260/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.9590 - val_loss: 6336.1724
    Epoch 261/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.5269 - val_loss: 6327.9795
    Epoch 262/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6086.0469 - val_loss: 6324.3721
    Epoch 263/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6085.3750 - val_loss: 6325.2466
    Epoch 264/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6085.4297 - val_loss: 6325.3472
    Epoch 265/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6085.6230 - val_loss: 6331.4224
    Epoch 266/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6086.4424 - val_loss: 6334.2202
    Epoch 267/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.9336 - val_loss: 6342.9419
    Epoch 268/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6090.5933 - val_loss: 6354.9883
    Epoch 269/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6095.0498 - val_loss: 6362.0054
    Epoch 270/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6103.1372 - val_loss: 6372.9121
    Epoch 271/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6109.6909 - val_loss: 6352.0659
    Epoch 272/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6107.7773 - val_loss: 6349.0708
    Epoch 273/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6101.4146 - val_loss: 6357.5830
    Epoch 274/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6096.1392 - val_loss: 6382.7788
    Epoch 275/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6092.7705 - val_loss: 6384.2559
    Epoch 276/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6089.8774 - val_loss: 6393.9053
    Epoch 277/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.6484 - val_loss: 6406.7168
    Epoch 278/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6086.2852 - val_loss: 6402.5703
    Epoch 279/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6084.6138 - val_loss: 6395.9233
    Epoch 280/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6083.0400 - val_loss: 6391.6548
    Epoch 281/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6081.0278 - val_loss: 6389.3799
    Epoch 282/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6079.4355 - val_loss: 6394.6909
    Epoch 283/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6077.9365 - val_loss: 6385.1040
    Epoch 284/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6076.5469 - val_loss: 6385.6484
    Epoch 285/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6075.3828 - val_loss: 6378.2407
    Epoch 286/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6074.3613 - val_loss: 6378.0337
    Epoch 287/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6073.7900 - val_loss: 6368.2979
    Epoch 288/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6073.0859 - val_loss: 6370.9829
    Epoch 289/1000
    313/313 [==============================] - 15s 46ms/step - loss: 6072.7349 - val_loss: 6364.8643
    Epoch 290/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6073.8706 - val_loss: 6358.0142
    Epoch 291/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6077.9023 - val_loss: 6350.0737
    Epoch 292/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6084.1118 - val_loss: 6354.0449
    Epoch 293/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6080.3574 - val_loss: 6352.8389
    Epoch 294/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6075.0615 - val_loss: 6355.2588
    Epoch 295/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6072.1680 - val_loss: 6344.7339
    Epoch 296/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.0400 - val_loss: 6344.9033
    Epoch 297/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.9912 - val_loss: 6341.6465
    Epoch 298/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.4233 - val_loss: 6342.9072
    Epoch 299/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.0454 - val_loss: 6340.4922
    Epoch 300/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6068.9917 - val_loss: 6338.9663
    Epoch 301/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6068.7568 - val_loss: 6345.9966
    Epoch 302/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6068.8579 - val_loss: 6356.5649
    Epoch 303/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6068.8442 - val_loss: 6350.7856
    Epoch 304/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.7651 - val_loss: 6351.0967
    Epoch 305/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6072.2720 - val_loss: 6345.2310
    Epoch 306/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6074.9692 - val_loss: 6322.2891
    Epoch 307/1000
    313/313 [==============================] - 16s 53ms/step - loss: 6076.7319 - val_loss: 6320.7500
    Epoch 308/1000
    313/313 [==============================] - 17s 53ms/step - loss: 6076.6650 - val_loss: 6332.6880
    Epoch 309/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6076.0249 - val_loss: 6345.1284
    Epoch 310/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6077.9258 - val_loss: 6348.8071
    Epoch 311/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6079.4590 - val_loss: 6340.8071
    Epoch 312/1000
    313/313 [==============================] - 17s 55ms/step - loss: 6079.8730 - val_loss: 6331.7642
    Epoch 313/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6080.5269 - val_loss: 6325.9502
    Epoch 314/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6081.2910 - val_loss: 6323.6138
    Epoch 315/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6082.3369 - val_loss: 6330.3862
    Epoch 316/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6083.9590 - val_loss: 6329.4858
    Epoch 317/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6086.1060 - val_loss: 6336.9600
    Epoch 318/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6088.3906 - val_loss: 6344.5571
    Epoch 319/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6090.8774 - val_loss: 6347.9302
    Epoch 320/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6092.3267 - val_loss: 6334.9531
    Epoch 321/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6093.0649 - val_loss: 6340.9053
    Epoch 322/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6092.1426 - val_loss: 6334.4331
    Epoch 323/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6089.9360 - val_loss: 6327.0537
    Epoch 324/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6086.8330 - val_loss: 6322.0527
    Epoch 325/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6083.6387 - val_loss: 6321.4053
    Epoch 326/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6081.7178 - val_loss: 6325.2964
    Epoch 327/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6080.3042 - val_loss: 6326.9917
    Epoch 328/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6079.1875 - val_loss: 6326.3257
    Epoch 329/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6078.3516 - val_loss: 6337.4883
    Epoch 330/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6077.7285 - val_loss: 6332.5781
    Epoch 331/1000
    313/313 [==============================] - 17s 54ms/step - loss: 6077.1465 - val_loss: 6337.2192
    Epoch 332/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6076.3013 - val_loss: 6343.5825
    Epoch 333/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6075.4575 - val_loss: 6338.0688
    Epoch 334/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6075.3198 - val_loss: 6335.9878
    Epoch 335/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6074.7358 - val_loss: 6332.3252
    Epoch 336/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6075.6694 - val_loss: 6323.2212
    Epoch 337/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6077.2173 - val_loss: 6315.8525
    Epoch 338/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6081.5601 - val_loss: 6329.9702
    Epoch 339/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6088.4595 - val_loss: 6333.5132
    Epoch 340/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6093.4233 - val_loss: 6330.6279
    Epoch 341/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6092.6919 - val_loss: 6321.0151
    Epoch 342/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6089.8169 - val_loss: 6316.9863
    Epoch 343/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6084.7910 - val_loss: 6315.7485
    Epoch 344/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6080.9102 - val_loss: 6313.2578
    Epoch 345/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6078.9863 - val_loss: 6315.1396
    Epoch 346/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6077.8013 - val_loss: 6314.7334
    Epoch 347/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6076.6895 - val_loss: 6317.0430
    Epoch 348/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6076.0972 - val_loss: 6318.2935
    Epoch 349/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6074.9600 - val_loss: 6316.3848
    Epoch 350/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6073.7041 - val_loss: 6318.5146
    Epoch 351/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6072.4321 - val_loss: 6320.1196
    Epoch 352/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.1626 - val_loss: 6322.3896
    Epoch 353/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6070.3164 - val_loss: 6322.7866
    Epoch 354/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6069.4287 - val_loss: 6323.4458
    Epoch 355/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.1343 - val_loss: 6325.6587
    Epoch 356/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6069.9502 - val_loss: 6319.7339
    Epoch 357/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6071.2075 - val_loss: 6325.8979
    Epoch 358/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6073.5190 - val_loss: 6322.5278
    Epoch 359/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6072.7197 - val_loss: 6314.2778
    Epoch 360/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6070.6636 - val_loss: 6321.7476
    Epoch 361/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.4443 - val_loss: 6333.7998
    Epoch 362/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.4565 - val_loss: 6336.5908
    Epoch 363/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.6826 - val_loss: 6320.9487
    Epoch 364/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6073.3896 - val_loss: 6317.7012
    Epoch 365/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6076.7534 - val_loss: 6323.6475
    Epoch 366/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6078.4453 - val_loss: 6332.7402
    Epoch 367/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6080.0439 - val_loss: 6334.5205
    Epoch 368/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6083.1738 - val_loss: 6372.8184
    Epoch 369/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6087.4307 - val_loss: 6376.2554
    Epoch 370/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6089.8521 - val_loss: 6333.3765
    Epoch 371/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6088.3140 - val_loss: 6322.2427
    Epoch 372/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6084.9028 - val_loss: 6330.3721
    Epoch 373/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6082.4248 - val_loss: 6335.7979
    Epoch 374/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6081.0952 - val_loss: 6338.0098
    Epoch 375/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6080.2378 - val_loss: 6340.5557
    Epoch 376/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6079.5234 - val_loss: 6347.1313
    Epoch 377/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6078.6274 - val_loss: 6341.5391
    Epoch 378/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6077.9097 - val_loss: 6346.6602
    Epoch 379/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6077.2734 - val_loss: 6349.0713
    Epoch 380/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6076.3843 - val_loss: 6351.5894
    Epoch 381/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6075.8838 - val_loss: 6345.3687
    Epoch 382/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6076.4502 - val_loss: 6343.1299
    Epoch 383/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6077.9702 - val_loss: 6352.0542
    Epoch 384/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6082.0630 - val_loss: 6367.6436
    Epoch 385/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6089.0229 - val_loss: 6317.0259
    Epoch 386/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6096.5698 - val_loss: 6310.0269
    Epoch 387/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6092.9077 - val_loss: 6311.4326
    Epoch 388/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6084.7593 - val_loss: 6313.6655
    Epoch 389/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6079.6133 - val_loss: 6313.9424
    Epoch 390/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6076.3853 - val_loss: 6316.3086
    Epoch 391/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6074.7090 - val_loss: 6319.7754
    Epoch 392/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6073.3384 - val_loss: 6315.9932
    Epoch 393/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6072.3794 - val_loss: 6314.3511
    Epoch 394/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.9873 - val_loss: 6318.0615
    Epoch 395/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6070.1802 - val_loss: 6321.0947
    Epoch 396/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.2666 - val_loss: 6320.5527
    Epoch 397/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6068.3823 - val_loss: 6322.0474
    Epoch 398/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6068.0562 - val_loss: 6321.2095
    Epoch 399/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6067.6182 - val_loss: 6321.5488
    Epoch 400/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6067.7510 - val_loss: 6324.4976
    Epoch 401/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6069.1704 - val_loss: 6314.5317
    Epoch 402/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.7056 - val_loss: 6322.2852
    Epoch 403/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6072.6807 - val_loss: 6321.1919
    Epoch 404/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.2432 - val_loss: 6321.8574
    Epoch 405/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.8716 - val_loss: 6319.1812
    Epoch 406/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.1748 - val_loss: 6317.4126
    Epoch 407/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.4297 - val_loss: 6324.1064
    Epoch 408/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6070.3130 - val_loss: 6343.0469
    Epoch 409/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6071.4360 - val_loss: 6332.6909
    Epoch 410/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6073.0371 - val_loss: 6325.4048
    Epoch 411/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6074.5317 - val_loss: 6324.8408
    Epoch 412/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6076.1045 - val_loss: 6324.0825
    Epoch 413/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6077.8384 - val_loss: 6332.4375
    Epoch 414/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6079.1816 - val_loss: 6346.4888
    Epoch 415/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6081.3838 - val_loss: 6342.7100
    Epoch 416/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6084.5508 - val_loss: 6351.2510
    Epoch 417/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6086.7554 - val_loss: 6377.0425
    Epoch 418/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.6978 - val_loss: 6383.2456
    Epoch 419/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.4678 - val_loss: 6356.1836
    Epoch 420/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6087.4600 - val_loss: 6357.2998
    Epoch 421/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6087.9756 - val_loss: 6362.1060
    Epoch 422/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6088.5117 - val_loss: 6384.5049
    Epoch 423/1000
    313/313 [==============================] - 17s 55ms/step - loss: 6088.7334 - val_loss: 6390.8940
    Epoch 424/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6088.6611 - val_loss: 6374.7153
    Epoch 425/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6088.2998 - val_loss: 6339.9526
    Epoch 426/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6088.0659 - val_loss: 6328.9224
    Epoch 427/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6086.2446 - val_loss: 6339.2388
    Epoch 428/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6083.6318 - val_loss: 6331.8745
    Epoch 429/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6081.1084 - val_loss: 6336.5098
    Epoch 430/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6078.7563 - val_loss: 6344.2939
    Epoch 431/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6076.4609 - val_loss: 6344.0259
    Epoch 432/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6075.0132 - val_loss: 6342.4404
    Epoch 433/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6073.1567 - val_loss: 6343.0020
    Epoch 434/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6071.4155 - val_loss: 6339.8838
    Epoch 435/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.8042 - val_loss: 6340.8423
    Epoch 436/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6068.4707 - val_loss: 6334.6206
    Epoch 437/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6067.1982 - val_loss: 6333.6304
    Epoch 438/1000
    313/313 [==============================] - 13s 43ms/step - loss: 6065.7324 - val_loss: 6329.0752
    Epoch 439/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6064.2158 - val_loss: 6330.7183
    Epoch 440/1000
    313/313 [==============================] - 15s 46ms/step - loss: 6063.1353 - val_loss: 6328.0396
    Epoch 441/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6062.3462 - val_loss: 6326.4263
    Epoch 442/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6061.8950 - val_loss: 6324.1123
    Epoch 443/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6062.0254 - val_loss: 6325.0547
    Epoch 444/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6063.9141 - val_loss: 6332.2842
    Epoch 445/1000
    313/313 [==============================] - 15s 46ms/step - loss: 6067.0142 - val_loss: 6330.8447
    Epoch 446/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6067.5039 - val_loss: 6346.4048
    Epoch 447/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6065.1567 - val_loss: 6334.5322
    Epoch 448/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6063.5117 - val_loss: 6325.9033
    Epoch 449/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6062.8691 - val_loss: 6335.9067
    Epoch 450/1000
    313/313 [==============================] - 15s 46ms/step - loss: 6062.8906 - val_loss: 6326.4409
    Epoch 451/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6062.7231 - val_loss: 6325.5078
    Epoch 452/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6063.1831 - val_loss: 6325.4316
    Epoch 453/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6064.0112 - val_loss: 6326.8140
    Epoch 454/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6064.6250 - val_loss: 6325.1689
    Epoch 455/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6065.8066 - val_loss: 6323.5771
    Epoch 456/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6065.9458 - val_loss: 6322.4502
    Epoch 457/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6065.7939 - val_loss: 6336.4072
    Epoch 458/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6065.6968 - val_loss: 6332.0908
    Epoch 459/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6064.8735 - val_loss: 6333.9736
    Epoch 460/1000
    313/313 [==============================] - 14s 46ms/step - loss: 6064.4658 - val_loss: 6333.2354
    Epoch 461/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6064.6895 - val_loss: 6336.2310
    Epoch 462/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6065.2588 - val_loss: 6340.0298
    Epoch 463/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6066.5396 - val_loss: 6353.9312
    Epoch 464/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6067.4688 - val_loss: 6369.2227
    Epoch 465/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6068.9414 - val_loss: 6378.3945
    Epoch 466/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.9668 - val_loss: 6416.3335
    Epoch 467/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6074.5469 - val_loss: 6430.8286
    Epoch 468/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6075.7148 - val_loss: 6401.3892
    Epoch 469/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6072.5210 - val_loss: 6403.8877
    Epoch 470/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6067.4390 - val_loss: 6408.2930
    Epoch 471/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6063.4619 - val_loss: 6397.0396
    Epoch 472/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6061.2285 - val_loss: 6393.9380
    Epoch 473/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6059.5576 - val_loss: 6389.8325
    Epoch 474/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6058.2529 - val_loss: 6390.6704
    Epoch 475/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6057.5859 - val_loss: 6389.1519
    Epoch 476/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6057.1201 - val_loss: 6381.9888
    Epoch 477/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6056.6987 - val_loss: 6381.9233
    Epoch 478/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6056.4048 - val_loss: 6386.8652
    Epoch 479/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.9946 - val_loss: 6384.1274
    Epoch 480/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6055.3203 - val_loss: 6393.3579
    Epoch 481/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.1802 - val_loss: 6382.3682
    Epoch 482/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.1104 - val_loss: 6381.0049
    Epoch 483/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.9414 - val_loss: 6378.2549
    Epoch 484/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6059.2529 - val_loss: 6369.9697
    Epoch 485/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.8926 - val_loss: 6408.4351
    Epoch 486/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6058.4639 - val_loss: 6385.9429
    Epoch 487/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6057.2217 - val_loss: 6368.4878
    Epoch 488/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6057.3647 - val_loss: 6374.9097
    Epoch 489/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6057.8687 - val_loss: 6375.9888
    Epoch 490/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6059.3330 - val_loss: 6380.4238
    Epoch 491/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6061.3486 - val_loss: 6401.0356
    Epoch 492/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6064.2944 - val_loss: 6404.3428
    Epoch 493/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6066.5430 - val_loss: 6414.8374
    Epoch 494/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6067.7085 - val_loss: 6396.7261
    Epoch 495/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6068.0586 - val_loss: 6397.2153
    Epoch 496/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6068.3306 - val_loss: 6411.6382
    Epoch 497/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.8926 - val_loss: 6423.4751
    Epoch 498/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6071.9595 - val_loss: 6431.6660
    Epoch 499/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6072.6982 - val_loss: 6435.7529
    Epoch 500/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6072.7222 - val_loss: 6446.6499
    Epoch 501/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6072.1450 - val_loss: 6447.0195
    Epoch 502/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.3970 - val_loss: 6439.4736
    Epoch 503/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.9673 - val_loss: 6436.6118
    Epoch 504/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.5137 - val_loss: 6434.4585
    Epoch 505/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.2778 - val_loss: 6449.8467
    Epoch 506/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.1211 - val_loss: 6427.5850
    Epoch 507/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.3486 - val_loss: 6417.9185
    Epoch 508/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6070.9487 - val_loss: 6415.7056
    Epoch 509/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.4233 - val_loss: 6418.4165
    Epoch 510/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6070.5776 - val_loss: 6405.9443
    Epoch 511/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6069.7378 - val_loss: 6405.5010
    Epoch 512/1000
    313/313 [==============================] - 16s 53ms/step - loss: 6068.3379 - val_loss: 6395.9873
    Epoch 513/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6066.8096 - val_loss: 6401.3765
    Epoch 514/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6064.6338 - val_loss: 6401.3320
    Epoch 515/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6063.6782 - val_loss: 6398.4629
    Epoch 516/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6063.1299 - val_loss: 6397.7661
    Epoch 517/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6062.5283 - val_loss: 6391.7900
    Epoch 518/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6061.5918 - val_loss: 6394.8369
    Epoch 519/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6060.9443 - val_loss: 6387.8960
    Epoch 520/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6061.0215 - val_loss: 6382.3730
    Epoch 521/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6060.8696 - val_loss: 6384.1929
    Epoch 522/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.7832 - val_loss: 6381.1982
    Epoch 523/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6060.9346 - val_loss: 6384.0005
    Epoch 524/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6061.9409 - val_loss: 6385.9453
    Epoch 525/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6063.9810 - val_loss: 6388.3071
    Epoch 526/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6067.3970 - val_loss: 6381.7007
    Epoch 527/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6072.3374 - val_loss: 6363.0903
    Epoch 528/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6075.4341 - val_loss: 6351.6025
    Epoch 529/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6073.1196 - val_loss: 6355.8286
    Epoch 530/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.1641 - val_loss: 6356.7822
    Epoch 531/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6065.7095 - val_loss: 6350.0190
    Epoch 532/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6062.8447 - val_loss: 6346.4951
    Epoch 533/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6060.4268 - val_loss: 6346.2192
    Epoch 534/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6059.3828 - val_loss: 6347.7988
    Epoch 535/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6058.2754 - val_loss: 6345.6255
    Epoch 536/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6057.2012 - val_loss: 6342.3799
    Epoch 537/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6056.5234 - val_loss: 6340.9824
    Epoch 538/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6056.0522 - val_loss: 6340.3696
    Epoch 539/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.5142 - val_loss: 6338.8198
    Epoch 540/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6054.9253 - val_loss: 6338.1421
    Epoch 541/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.4194 - val_loss: 6337.4185
    Epoch 542/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.3613 - val_loss: 6339.4517
    Epoch 543/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6054.4404 - val_loss: 6335.9678
    Epoch 544/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6055.0845 - val_loss: 6335.4053
    Epoch 545/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6057.2466 - val_loss: 6333.9082
    Epoch 546/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6058.5542 - val_loss: 6332.2324
    Epoch 547/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6058.4590 - val_loss: 6341.6230
    Epoch 548/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6059.2207 - val_loss: 6336.5977
    Epoch 549/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6063.0894 - val_loss: 6332.4111
    Epoch 550/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6066.8008 - val_loss: 6334.6045
    Epoch 551/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6065.7026 - val_loss: 6332.0864
    Epoch 552/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6064.1948 - val_loss: 6332.0654
    Epoch 553/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6065.7798 - val_loss: 6335.7944
    Epoch 554/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.9297 - val_loss: 6342.3579
    Epoch 555/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6074.6460 - val_loss: 6357.7480
    Epoch 556/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6081.7114 - val_loss: 6394.1099
    Epoch 557/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6084.4126 - val_loss: 6389.5586
    Epoch 558/1000
    313/313 [==============================] - 15s 47ms/step - loss: 6080.8794 - val_loss: 6364.8301
    Epoch 559/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6077.7549 - val_loss: 6358.5044
    Epoch 560/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6075.4263 - val_loss: 6357.5669
    Epoch 561/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6074.6802 - val_loss: 6357.7690
    Epoch 562/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6073.7227 - val_loss: 6358.3574
    Epoch 563/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6073.9473 - val_loss: 6364.7129
    Epoch 564/1000
    313/313 [==============================] - 16s 52ms/step - loss: 6073.3008 - val_loss: 6360.5693
    Epoch 565/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6072.5464 - val_loss: 6361.8232
    Epoch 566/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6071.7930 - val_loss: 6364.0098
    Epoch 567/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.5059 - val_loss: 6366.6577
    Epoch 568/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6070.3809 - val_loss: 6358.4297
    Epoch 569/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6069.1318 - val_loss: 6363.7949
    Epoch 570/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6068.2295 - val_loss: 6357.0752
    Epoch 571/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6068.4985 - val_loss: 6357.0967
    Epoch 572/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6069.4478 - val_loss: 6349.6152
    Epoch 573/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6074.7793 - val_loss: 6364.8735
    Epoch 574/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6083.9482 - val_loss: 6333.8018
    Epoch 575/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6086.2310 - val_loss: 6342.3188
    Epoch 576/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6078.5063 - val_loss: 6346.2202
    Epoch 577/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6070.8618 - val_loss: 6347.0488
    Epoch 578/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6066.5054 - val_loss: 6346.2534
    Epoch 579/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6063.3921 - val_loss: 6343.7969
    Epoch 580/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6061.5918 - val_loss: 6344.6689
    Epoch 581/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.4604 - val_loss: 6347.9185
    Epoch 582/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6058.9502 - val_loss: 6346.4390
    Epoch 583/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6057.7100 - val_loss: 6348.1143
    Epoch 584/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6056.7241 - val_loss: 6351.1982
    Epoch 585/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.8013 - val_loss: 6351.8940
    Epoch 586/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.4194 - val_loss: 6354.2344
    Epoch 587/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.9966 - val_loss: 6359.3213
    Epoch 588/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.4810 - val_loss: 6346.2266
    Epoch 589/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6053.5942 - val_loss: 6346.4409
    Epoch 590/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6056.4736 - val_loss: 6357.8813
    Epoch 591/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6059.5547 - val_loss: 6334.5552
    Epoch 592/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6057.7549 - val_loss: 6338.7520
    Epoch 593/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.4502 - val_loss: 6349.6650
    Epoch 594/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.6934 - val_loss: 6346.4927
    Epoch 595/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.0972 - val_loss: 6340.0801
    Epoch 596/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.4111 - val_loss: 6342.8770
    Epoch 597/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.1118 - val_loss: 6344.4033
    Epoch 598/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6054.2485 - val_loss: 6334.7515
    Epoch 599/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6056.6689 - val_loss: 6344.5635
    Epoch 600/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6058.1934 - val_loss: 6362.6792
    Epoch 601/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6059.2798 - val_loss: 6350.1035
    Epoch 602/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6057.5054 - val_loss: 6338.0601
    Epoch 603/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.6577 - val_loss: 6334.3960
    Epoch 604/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6052.9810 - val_loss: 6335.0781
    Epoch 605/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.2969 - val_loss: 6337.3999
    Epoch 606/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6052.1470 - val_loss: 6339.9224
    Epoch 607/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6052.0889 - val_loss: 6332.6196
    Epoch 608/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6053.6870 - val_loss: 6330.5815
    Epoch 609/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6055.9419 - val_loss: 6332.6348
    Epoch 610/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.9619 - val_loss: 6326.4819
    Epoch 611/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6065.8101 - val_loss: 6348.4155
    Epoch 612/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6063.0752 - val_loss: 6356.5020
    Epoch 613/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6056.4575 - val_loss: 6352.9443
    Epoch 614/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6052.8354 - val_loss: 6353.9429
    Epoch 615/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6051.2100 - val_loss: 6354.7598
    Epoch 616/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6050.2842 - val_loss: 6360.0889
    Epoch 617/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.8423 - val_loss: 6364.5386
    Epoch 618/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6050.0142 - val_loss: 6366.6304
    Epoch 619/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.7480 - val_loss: 6367.3330
    Epoch 620/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.7183 - val_loss: 6364.9595
    Epoch 621/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.6313 - val_loss: 6365.4404
    Epoch 622/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6049.6812 - val_loss: 6371.2671
    Epoch 623/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.4575 - val_loss: 6365.8462
    Epoch 624/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.9062 - val_loss: 6352.6299
    Epoch 625/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6050.3955 - val_loss: 6359.6841
    Epoch 626/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.4761 - val_loss: 6380.2876
    Epoch 627/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.9736 - val_loss: 6336.6377
    Epoch 628/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.5327 - val_loss: 6359.6680
    Epoch 629/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6056.3037 - val_loss: 6360.1084
    Epoch 630/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6056.5137 - val_loss: 6361.6646
    Epoch 631/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6055.7490 - val_loss: 6379.4600
    Epoch 632/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6055.3613 - val_loss: 6378.8887
    Epoch 633/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6055.8574 - val_loss: 6369.7998
    Epoch 634/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6056.8052 - val_loss: 6383.9922
    Epoch 635/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6058.6294 - val_loss: 6417.1938
    Epoch 636/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.3154 - val_loss: 6435.5146
    Epoch 637/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6061.1792 - val_loss: 6449.0591
    Epoch 638/1000
    313/313 [==============================] - 15s 50ms/step - loss: 6061.0942 - val_loss: 6427.7046
    Epoch 639/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.4834 - val_loss: 6400.1377
    Epoch 640/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6060.0894 - val_loss: 6386.7217
    Epoch 641/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6059.4771 - val_loss: 6381.3809
    Epoch 642/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6060.2129 - val_loss: 6371.6401
    Epoch 643/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6060.1514 - val_loss: 6370.6685
    Epoch 644/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6058.8950 - val_loss: 6357.1582
    Epoch 645/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6057.2373 - val_loss: 6349.4360
    Epoch 646/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.8232 - val_loss: 6352.0029
    Epoch 647/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.6802 - val_loss: 6348.2080
    Epoch 648/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.3066 - val_loss: 6349.8369
    Epoch 649/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6053.4395 - val_loss: 6364.2051
    Epoch 650/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6054.0449 - val_loss: 6384.1880
    Epoch 651/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6055.3384 - val_loss: 6382.5908
    Epoch 652/1000
    313/313 [==============================] - 16s 51ms/step - loss: 6057.7729 - val_loss: 6369.3281
    Epoch 653/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6060.6543 - val_loss: 6356.5454
    Epoch 654/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6063.0264 - val_loss: 6343.4814
    Epoch 655/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6062.0391 - val_loss: 6340.6846
    Epoch 656/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6058.4707 - val_loss: 6338.6001
    Epoch 657/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6055.5020 - val_loss: 6343.6069
    Epoch 658/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.5825 - val_loss: 6345.9058
    Epoch 659/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6050.5415 - val_loss: 6345.1421
    Epoch 660/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6049.6606 - val_loss: 6342.4771
    Epoch 661/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6049.0044 - val_loss: 6344.1289
    Epoch 662/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6048.6050 - val_loss: 6344.2471
    Epoch 663/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6048.2129 - val_loss: 6343.6138
    Epoch 664/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6047.9805 - val_loss: 6340.1934
    Epoch 665/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6048.0605 - val_loss: 6338.5771
    Epoch 666/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6048.0933 - val_loss: 6348.1367
    Epoch 667/1000
    313/313 [==============================] - 15s 48ms/step - loss: 6048.5137 - val_loss: 6372.9238
    Epoch 668/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6050.5723 - val_loss: 6387.2954
    Epoch 669/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.4258 - val_loss: 6335.6006
    Epoch 670/1000
    313/313 [==============================] - 15s 49ms/step - loss: 6052.4536 - val_loss: 6334.3223
    Epoch 671/1000
    313/313 [==============================] - 16s 50ms/step - loss: 6052.0928 - val_loss: 6335.8418
    Epoch 672/1000
    251/313 [=======================>......] - ETA: 2s - loss: 6056.5859

```python
# OR... load the pre-trained model

#ckpt = tf.train.Checkpoint(model=vae)
#ckpt.restore(tf.train.latest_checkpoint('/path/to/model'))
```

```python
# Evaluate the model on the test set

test_loss = vae.evaluate(test_ds)
print("Test loss: {}".format(test_loss))
```

#### Compute reconstructions of test images

We will now take a look at some image reconstructions from the encoder-decoder architecture.

You should complete the following function, that uses `encoder` and `decoder` to reconstruct images from the test dataset. This function takes the encoder, decoder and a Tensor batch of test images as arguments. The function should be completed according to the following specification:

* Get the mean of the encoding distributions from passing the batch of images into the encoder
* Pass these latent vectors through the decoder to get the output distribution

Your function should then return the mean of the output distribution, which will be a Tensor of shape `(batch_size, 64, 64, 3)`.

```python
 #### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def reconstruct(encoder, decoder, batch_of_images):
    """
    This function should compute reconstructions of the batch_of_images according
    to the above instructions.
    The function takes the encoder, decoder and batch_of_images as inputs, which
    should be used to compute the reconstructions.
    The function should then return the reconstructions Tensor.
    """
    return  decoder(encoder(batch_of_images).mean()).mean()

```

```python
# Run your function to compute reconstructions of random samples from the test dataset

n_reconstructions = 7
files = [f for f in os.listdir('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week4/CeleAsubset/test/') if f.endswith('.jpg')]
num_test_files = len(files)
test_ds_for_reconstructions = load_dataset('test')
for all_test_images, _ in test_ds_for_reconstructions.batch(num_test_files).take(1):
    all_test_images_np = all_test_images.numpy()
example_images = all_test_images_np[np.random.choice(num_test_files, n_reconstructions, replace=False)]

reconstructions = reconstruct(encoder, decoder, example_images).numpy()
```

```python
# Plot the reconstructions

f, axs = plt.subplots(2, n_reconstructions, figsize=(16, 6))
axs[0, n_reconstructions // 2].set_title("Original test images")
axs[1, n_reconstructions // 2].set_title("Reconstructed images")
for j in range(n_reconstructions):
    axs[0, j].imshow(example_images[j])
    axs[1, j].imshow(reconstructions[j])
    axs[0, j].axis('off')
    axs[1, j].axis('off')

plt.tight_layout();
```

#### Sample new images from the generative model

Now we will sample from the generative model; that is, first sample latent vectors from the prior, and then decode those latent vectors with the decoder.

You should complete the following function to generate new images. This function takes the prior distribution and decoder network as arguments, as well as the number of samples to generate. This function should be completed according to the following:

* Sample a batch of `n_samples` images from the prior distribution, to obtain a latent vector Tensor of shape `(n_samples, 50)`
* Pass this batch of latent vectors through the decoder, to obtain an Independent Bernoulli distribution with batch shape equal to `[n_samples]` and event shape equal to `[64, 64, 3]`.

The function should then return the mean of the Bernoulli distribution, which will be a Tensor of shape `(n_samples, 64, 64, 3)`.

```python
 #### GRADED CELL ####

# Complete the following function.
# Make sure to not change the function name or arguments.

def generate_images(prior, decoder, n_samples):
    """
    This function should compute generate new samples of images from the generative model,
    according to the above instructions.
    The function takes the prior distribution, decoder and number of samples as inputs, which
    should be used to generate the images.
    The function should then return the batch of generated images.
    """
    return decoder(prior.sample(n_samples)).mean()
```

```python
# Run your function to generate new images

n_samples = 10
sampled_images = generate_images(prior, decoder, n_samples)

f, axs = plt.subplots(1, n_samples, figsize=(16, 6))

for j in range(n_samples):
    axs[j].imshow(sampled_images[j])
    axs[j].axis('off')

plt.tight_layout();
```

### Modify generations with attribute vector

This final section is ungraded, but you will hopefully find it to be an interesting extension. We will see how the latent space encodes high-level information about the images, even though it has not been trained with any information apart from the images themselves.

As mentioned in the introduction, each image in the CelebA dataset is labelled according to the attributes of the person pictured.

Run the cells below to load these labels, along with a subset of the training data.

```python
# Function to load labels and images as a numpy array

def load_labels_and_image_arrays(split):
    dataset = load_dataset(split)
    files = [f for f in os.listdir('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week4/CeleAsubset/{}/'.format(split)) if f.endswith('.jpg')]

    for all_images, _ in dataset.batch(len(files)).take(1):
        all_images_np = all_images.numpy()

    labels = pd.read_csv('/content/drive/MyDrive/Colab_Notebooks/coursera/coursera_Probabilistic_Deep_Learning_TF2/Week4/CeleAsubset/list_attr_celeba_subset.csv')
    labels = labels[labels['image_id'].isin(files)]
    return labels, all_images_np
```

```python
# Load labels in a pandas DataFrame, training_subset is a numpy array

train_labels, training_subset = load_labels_and_image_arrays('train')
```

```python
# List the attributes contained in the DataFrame

train_labels.columns[2:]
```

Each image is labelled with a binary indicator (1 is True, -1 is False) according to whether it possesses the attribute.

```python
# View a sample from the labels data

train_labels.sample(5)
```

```python
# Select an attribute

attribute = 'Smiling'
```

```python
# Separate the images into those that have the attribute, and those that don't

attribute_mask = (train_labels[attribute] == 1)
images_with_attribute = training_subset[attribute_mask]

not_attribute_mask = (train_labels[attribute] == -1)
images_without_attribute = training_subset[not_attribute_mask]
```

#### Get the 'attribute vector'

We will now encode each of the images that have the chosen attribute into the latent space by passing them through the encoder. We then average the latent codes obtained for all of these images to obtain a single latent code.

We then do the same for the images that do not have the chosen attribute. This gives an average latent code for images with the attribute, and an average latent code for images without the attribute. Intuitively speaking, the difference between these two vectors then gives us the 'direction' in latent space that corresponds to the presence of the attribute.

```python
# Encode the images with and without the chosen attribute

encoded_images_with_attribute = encoder(images_with_attribute)
encoded_images_without_attribute = encoder(images_without_attribute)
```

```python
# Average the latent vectors for each batch of encodings

mean_encoded_images_with_attribute = tf.reduce_mean(encoded_images_with_attribute.mean(),
                                                    axis=0, keepdims=True)
mean_encoded_images_without_attribute = tf.reduce_mean(encoded_images_without_attribute.mean(),
                                                    axis=0, keepdims=True)
```

```python
# Get the attribute vector

attribute_vector = mean_encoded_images_with_attribute -  mean_encoded_images_without_attribute
```

We can view this attribute vector by decoding it:

```python
# Display the decoded attribute vector

decoded_a = decoder(attribute_vector).mean()
plt.imshow(decoded_a.numpy().squeeze())
plt.axis('off');
```

#### Modify reconstructions using the attribute vector

We can now use the attribute vector to add the attribute to an image reconstruction, where that attribute wasn't present before. To do this, we can just add the attribute vector to the latent vector encoding of the image, and then decode the result. We can also adjust the strength of the attribute vector by scaling with a multiplicative parameter.

```python
# Add the attribute vector to a sample of images that don't have the attribute

n_examples = 7
sampled_inx = np.random.choice(images_without_attribute.shape[0], n_examples, replace=False)
sample_images_without_attribute = images_without_attribute[sampled_inx]
sample_images_encodings = encoder(sample_images_without_attribute)
sample_images_reconstructions = decoder(sample_images_encodings).mean()

k = 2.5  # Weighting of attribute vector
modified_sample_images_encodings = sample_images_encodings + (k * attribute_vector)
modified_reconstructions = decoder(modified_sample_images_encodings).mean()
```

```python
# Display the original images, their reconstructions, and modified reconstructions

f, axs = plt.subplots(3, n_examples, figsize=(16, 6))
axs[0, n_examples // 2].set_title("Original images")
axs[1, n_examples // 2].set_title("Reconstructed images")
axs[2, n_examples // 2].set_title("Images with added attribute")
for j in range(n_examples):
    axs[0, j].imshow(sample_images_without_attribute[j])
    axs[1, j].imshow(sample_images_reconstructions[j])
    axs[2, j].imshow(modified_reconstructions[j])
    for ax in axs[:, j]: ax.axis('off')

plt.tight_layout();
```

You could also try removing the attribute from images that possess the attribute, or experiment with a different attribute.

---

Congratulations for completing this programming assignment! You're now ready to move on to the capstone project for this course.

```python

```

Week4/Week 4 Programming Assignment.ipynb