# Practical 4a: Recurrent Neural Networks (RNNs)

© Deep Learning Indaba. Apache License 2.0.

## Introduction

Feedforward models (eg deep MLPs and ConvNets) map fixed-size input-data (vectors of a fixed dimensionality) to their output labels. They're very powerful and have been successfully used for many tasks. However, a lot of data is not in the form of fixed-size vectors, but exists in the form of **sequences**. Language is one good example, where sentences are sequences of words. In some way, almost any data types can be considered as a sequence (for instance an image consists of a sequence of pixels, speech a sequence of phonemes, and so forth).

Recurrent neural networks (**RNNs**) were designed to be able to handle sequential data, and in this practical we will take a closer look at RNNs and then build a model that can generate English sentences in the style of Shakespeare!

## Learning Objectives
* Understand how RNNs model sequential data.
* Understand how the vanilla RNN is a generalization of feedforward models to incorporate sequential dependencies.
* Understand the issues involved when training RNNs.
* Know how to implement an RNN for time-series estimation (**regression**) and an RNN language model (character-level **classification**) in Tensorflow using Keras.

## Imports

```python
#@title Imports (RUN ME!) { display-mode: "form" }

#!pip install tensorflow-gpu==2.0.0-beta0 > /dev/null 2>&1
#!pip -q install pydot_ng > /dev/null 2>&1
#!pip -q install grapfrffhviz > /dev/null 2>&1
#!apt install graphviz > /dev/null 2>&1

import numpy as np
import tensorflow as tf
import math
import random
import ssl
import sys
import urllib

from IPython import display
import matplotlib.pyplot as plt
%matplotlib inline
```

##From Feedforward to Recurrent Models

### Intuition
RNNs generalize feedforward networks (FFNs) to be able to work with sequential data. FFNs take an input (e.g. an image) and immediately produce an output (e.g. a digit class), something like this:

```python
def ffn_forward(x, W_xh, W_ho, b_hid, b_out):

    # Compute activations on the hidden layer.
    hidden_layer = act_fn(np.dot(W_xh, x) + b_hid)

    # Compute the (linear) output layer activations.
    output = np.dot(W_ho, hidden_layer) + b_out

    return output
```

**NOTE**: You don't have to run this cell, it's just shown to illustrate the point.

RNNs, on the other hand, consider the data sequentially, and can remember what they have seen earlier in the sequence to help interpret or contextualize elements from later in the sequence when making predictions, something like this:

```python
def rnn_forward(data_sequence, initial_state):

    state = initial_state  # Reused at every time-step
    all_states = [state]  # Used to save all states
    all_ys = [] # Used to save all predictions

    for x in data_sequence:

      # recurrent_fn() takes the current input and the previous state and produces a new state
      # it is a place holder which could represent various kinds of recurrent connections
      # including vanilla RNN, GRU or LSTM (if you don't know what these mean yet don't worry)
      new_state, y_pred = recurrent_fn(x, state)

      all_states.append(new_state)
      all_ys.append(y_pred)

      # Update state for the next time-step
      state = new_state

    return all_states, all_ys
```

To understand the distinction between FFNs and RNNs, imagine we want to label words as the part-of-speech categories that they belong to: E.g. for the input sentence "I want a duck" and "He had to duck", we want our model to predict that duck is a `Noun` in the first sentence and a `Verb` in the second. To do this successfully, the model needs to be aware of the surrounding context. However, if we feed a FFN model only one word at a time, how could it know the difference? If we want to feed it all the words at once, how do we deal with the fact that sentences are of different lengths?

RNNs solve this issue by processing the sentence word-by-word, and maintaining an internal **state** summarizing what it has seen so far. This applies not only to words, but also to phonemes in speech, or even, as we will see, elements of a time-series.

### Unrolling the network

Imagine we are trying to classify a sequence $X = (x_1, x_2, \ldots, x_N)$ into labels $y$ (for now, let's keep it abstract). After running the `rnn_forward()` function of our RNN defined above on $X$, we would have a list of internal states and outputs of the model at each sequence position. This process is called **unrolling or unfolding in time**, because you can think of it as unrolling the *computations* defined by the RNN loop over the inputs at each position of the sequence.  RNNs are often used to model **time series data** (which we will do in this practical), and therefore these positions are referred to as **time-steps**, and hence we call this process "unrolling over time". See the visualization below from this great [blog post](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) (it uses slightly different notation, but the idea should be clear):

> **We can therefore think of an RNN as a composition of identical feedforward neural networks (with replicated/tied weights), one for each moment or step in time. **

These feedforward functions that make up the RNN(i.e. our `recurrent_fn` above) are typically referred to as **cells**, and the only restriction on its API is that the cell function needs to be a differentiable function that can map an input and a state vector to an output and a new state vector. What we have shown above is called the **vanilla RNN**, but there are many more possibilities. One of the most popular variants is called the **Long Short-Term Memory (LSTM)** cell, which we'll use later to create our Shakespeare language model.

### Putting this together

In the feedforward models we've seen before, the input $x$ is mapped to an intermediate hidden layer $h$ as follows:

\begin{equation}
  h = \sigma(\underbrace{W_{xh}x}_\text{current input (per-example)} + b)
\end{equation}

where $\sigma$ is some non-linear activation function like ReLU or tanh.  We can then make a prediction $\hat{y} = \sigma(W_{hy}h + b)$ based on $h$, or we can add another layer, etc.  **NOTE**: We use the weight subscript $W_{xz}$ to indicate a mapping from layer $x$ to layer $z$.

RNNs generalize this idea to a sequence of inputs $X = {x_1, x_2, ...}$ by maintaining a sequence of state vectors $h_t$, one for every time-step $t$, as follows:

\begin{equation}
  h_t = \sigma(\underbrace{W_{hh}h_{t-1}}_\text{previous state} + \underbrace{W_{xh}x_t}_\text{current input (per time-step)} + b)
\end{equation}

Feedforward models map one input to one output. On the other hand, RNNs can give **many-to-many** (many inputs, many labels), **many-to-one**, and **one-to-many** mappings. Examples of these tasks include machine translation, sentiment analysis, and image captioning, respectively. For example, we could use each $h_t$ to predict the part of speech for that time-step ($y_t= \sigma(W_{hy}h_t+b)$) or we could use the last state $h_T$ to predict the topic of the whole document. Which task the RNN performs is based on the training data (of course)! We can visualize these as follows (from this [excellent blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) by Andrej Karpathy):

**QUESTIONS**
* How are FFNs and RNNs **similar**?
* How are they **different**?
* Why do we call RNNs "recurrent"?
* Can you think of a one-to-many task?

##Modeling General Time-Series

We will train an RNN to model a time-series as a first step. A **time-series** is a series of data-points ordered over discrete time-steps. Examples include the hourly temperature of Stellenbosch over a month or a year, the market price of some asset (like a company's stock) over time, and so forth. We will generate a **sinusoidal time-series** (with or without noise) as a toy example, and then train a tiny RNN model with only 5 parameters on this data.

###Create some artificial data

```python
#@title Create sinusoidal data {run: "auto"}
steps_per_cycle = 20 #@param { type: "slider", min:1, max:100, step:1 }
number_of_cycles = 176 #@param { type: "slider", min:1, max:1000, step:1 }
noise_factor = 0.1 #@param { type: "slider", min:0, max:1, step:0.1 }
plot_num_cycles = 23 #@param { type: "slider", min:1, max:50, step:1 }

seq_len = steps_per_cycle * number_of_cycles
t = np.arange(seq_len)
sin_t_noisy = np.sin(2 * np.pi / steps_per_cycle * t + noise_factor * np.random.uniform(-1.0, +1.0, seq_len))
sin_t_clean = np.sin(2 * np.pi / steps_per_cycle * t)

upto = plot_num_cycles * steps_per_cycle
fig = plt.figure(figsize=(15,3))
plt.plot(t[:upto], sin_t_noisy[:upto])
plt.title("Showing first {} cycles.".format(plot_num_cycles))
plt.show()

#both = np.column_stack((t, sin_t_noisy))
#print("both.shape = {}".format(both.shape))

#print("both[:steps_per_cycle, :steps_per_cycle]")
#print(both[:steps_per_cycle,:steps_per_cycle])
```

**TASK**: Adjust the parameters above to generate data with different properties.

Now we pack the data into train and test batches. Note that while RNNs can in theory learn the dependencies across all inputs received so far (using an algorithm called **backpropagation through time**, or BPTT; see the Aside box below), in practice they are trained using an algorithm called **truncated BPTT** where we truncate the inputs to only the last $T$ symbols (this is the `truncated_seq_len` variable below).

**QUESTIONs**:
* What issues can you think may arise by truncating the training data in this way?
* Despite these issues, why do you think it might be necessary to do this?

```python
#@title Pack truncated sequence data {run: "auto"}

def pack_truncated_data(data, num_prev = 100):
    X, Y = [], []
    for i in range(len(data) - num_prev):
      X.append(data[i : i + num_prev])
      Y.append(data[i + num_prev])
    # NOTE: Keras expects input data in the shape (batch_size, truncated_seq_len, input_dim)
    # We have only one real-valued number per time-step, so we therefore expand
    # the last dimension from (batch_size, truncated_seq_len) to
    # (batch_size, truncated_seq_len, 1).
    X, Y = np.array(X)[:,:,np.newaxis], np.array(Y)[:,np.newaxis]
    return X, Y

# We only consider this many previous data points
truncated_seq_len = 2 #@param { type: "slider", min:1, max:10, step:1 }
test_split = 0.25  # Fraction of total data to keep out as test data

# We use only the sin(t) values, and discard the time values
data = sin_t_noisy
data_len = data.shape[0]
num_train = int(data_len * (1 - test_split))

train_data = data[:num_train]
test_data = data[num_train:]

X_train, y_train = pack_truncated_data(train_data, num_prev=truncated_seq_len)
X_test, y_test = pack_truncated_data(test_data, num_prev=truncated_seq_len)

print("Generated training/test data with shapes\nX_train: {}, y_train: {}\nX_test: {}, y_test: {}. ".format(
    X_train.shape, y_train.shape, X_test.shape, y_test.shape))

```

    Generated training/test data with shapes
    X_train: (2638, 2, 1), y_train: (2638, 1)
    X_test: (878, 2, 1), y_test: (878, 1).

**NOTE**: We reshape the training data into (batch_size, truncated_seq_len, 1) and (batch_size, 1) arrays.

### Intermediate optional extra reading: (Truncated) Backpropagation-through-Time and Vanishing and Exploding Gradients

RNNs model sequential data, and are designed to capture how ***outputs*** at the current time step are influenced by the ***inputs*** that came before them. This is referred to as **long-range dependencies**. At a high level, this allows the model to remember what it has seen so far in order to better contextualize what it is seeing at the moment (think about how knowing the context of the sentence or conversation can sometimes help one to better figure out the intended meaning of a misheard word or ambiguous statement). It is what makes these models so powerful, but it is also what makes them so hard to train!

The most well-known algorithm for training RNNs is called **back-propagation through time (BPTT**; there are other algorithms). BPTT conceptually amounts to unrolling the computations of the RNN over time, computing the errors, and backpropagating the gradients through the unrolled graph structure.  Ideally we want to unroll the graph up to the maximum sequence length, however in practice, since sequence lengths vary and memory is limited, we only end up unrolling sequences up to some length $T$. This is called **truncated BPTT**, and is the most used variant of BPTT.

At a high level, there are two main issues when using (truncated) BPTT to train RNNs:

* Having shared ("tied") recurrent weights ($W_{hh}$) mean that **the gradient on these weights at some time step $t$ depends on all time steps up to time-step $T$**, the length of the full (truncated) sequence. This also leads to the **vanishing/exploding gradients** problem, which we'll explain below.

* **Memory usage grows linearly with the total number of steps $T$ that we unroll for**, because we need to save/cache the activations at each time-step (look at the Python code above to convince yourself of this).  This matters computationally, since memory is a limited resource. It also matters statistically, because it puts a limit on the types of dependency that the model can successfully learn, by preventing it from correcting errors that stem from an input more than $T$ steps ago.

**NOTE**: Think about that last statement and make sure you understand those 2 points.

BPTT is very similar to the standard back-propagation algorithm. Key to understanding the BPTT algorithm is to realize that gradients on the non-recurrent weights (weights of a per time-step classifier that tries to predict the part-of-speech tag for each word for example) and recurrent weights (that transform $h_{t-1}$ into $h_t$) are computed differently:

* The gradients of **non-recurrent weights** ($W_{hy}$) depend only on the error at that time-step, $E_t$.
* The gradients of **recurrent weights** ($W_{hh}$) depend on all previous time-steps up to maximum length $T$.

The first point is fairly intuitive: predictions at time-step $t$ is related to the loss of that particular prediction.

The second point will be explained in more detail in the lectures (see also [this great blog post](http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/)), but briefly, this can be summarized in these equations:

1. The **current** state is a function of the **previous** state and the current input: $h_t = \sigma(W_{hh}h_{t-1} + W_{xh}x_t)$
2. The gradient of the loss $E_t$ at time $t$ on $W_{hh}$ is a function of the current hidden state and model predictions $\hat{y}_t$ at time t:
$\frac{\partial E_t}{\partial W_{hh}} = \frac{\partial E_t}{\partial \hat{y}_t}\frac{\partial\hat{y}_t}{\partial h_t}\frac{\partial h_t}{\partial W_{hh}}$
3. Substituting (1) into (2) results in a **sum over all previous time-steps**:
$\frac{\partial E_t}{\partial W_{hh}} = \sum\limits_{k=0}^{t} \underbrace{\frac{\partial E_t}{\partial \hat{y}_t}\frac{\partial\hat{y}_t}{\partial h_t}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial W_{hh}}}_\text{product of gradient terms}$

The problem is that $\frac{\partial h_t}{\partial h_k} = \Pi_j \frac{\partial h_j}{\partial h_{j-1}}$ for j from $k + 1$ to $t$.  Because of this **repeated multiplicative interaction**, as the sequence length $t$ gets longer, the gradients themselves can get diminishingly small (**vanish**) or grow too large and result in numeric overflow (**explode**). This has been shown to be related to the norms of the recurrent weight matrices being less than or equal to 1. Intuitively, it works very similar to how multiplying a small number $v<1.0$ with itself repeatedly can quickly go to zero, or conversely, a large number $v>1.0$ could quickly go to infinity; only this is for matrices.

###Build a tiny RNN in Keras

Building an RNN in Keras is quite simple. We simply chain the layers together as follows:

```python
def define_model(truncated_seq_len):

    input_dimension = 1
    hidden_dimension = 1
    output_dimension = 1

    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.SimpleRNN(
        # We need to specify the input_shape *without* leading batch_size (it is inferred)
        input_shape=(truncated_seq_len, input_dimension),
        units=hidden_dimension,
        return_sequences=False,
        name='hidden_layer'))
    model.add(tf.keras.layers.Dense(
        output_dimension,
        name='output_layer'))

    model.compile(loss="mean_squared_error",
                  optimizer=tf.compat.v1.train.AdamOptimizer(learning_rate=1e-3))

    return model

```

**NOTE**: We're building an RNN for **regression**. We therefore use a linear layer (which outputs real-valued numbers) at the output with the "*mean_squared_error*" loss function.

```python
model = define_model(truncated_seq_len = X_train.shape[1])
model.summary()
```

    WARNING: Logging before flag parsing goes to stderr.
    W0708 18:29:35.965700 139985615468416 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/init_ops.py:1251: calling VarianceScaling.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Call initializer instance with the dtype argument instead of passing it to the constructor

    Model: "sequential"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    hidden_layer (SimpleRNN)     (None, 1)                 3
    _________________________________________________________________
    output_layer (Dense)         (None, 1)                 2
    =================================================================
    Total params: 5
    Trainable params: 5
    Non-trainable params: 0
    _________________________________________________________________

**NOTE**: You need to re-run the above cell every time after training to reset the model weights!

###Train the tiny RNN
Now let's train the model. This may take a few minutes (it takes much longer if you increase `truncated_seq_len`). Set `verbose=1` **before** you run the cell to see the intermediate output as the model is training. Set it to 0 if you don't want any output.

```python
''' SOLUTION TO ONE OF TASKS [DELETE]
patience = 5
train_history = model.fit(X_train, y_train, batch_size=600, epochs=1000,
                          verbose=1, validation_split=0.05,
                          callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=patience, verbose=1)])
'''
```

    " SOLUTION TO ONE OF TASKS [DELETE]\npatience = 5\ntrain_history = model.fit(X_train, y_train, batch_size=600, epochs=1000,\n                          verbose=1, validation_split=0.05,\n                          callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=patience, verbose=1)])\n"

```python
train_history = model.fit(X_train, y_train, batch_size=600, epochs=1000,
                          verbose=1, validation_split=0.05)
```

    Train on 2506 samples, validate on 132 samples
    Epoch 1/1000
    2506/2506 [==============================] - 4s 1ms/sample - loss: 0.9979 - val_loss: 0.9847
    Epoch 2/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.9882 - val_loss: 0.9755
    Epoch 3/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.9788 - val_loss: 0.9662
    Epoch 4/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.9694 - val_loss: 0.9569
    Epoch 5/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.9601 - val_loss: 0.9476
    Epoch 6/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.9509 - val_loss: 0.9385
    Epoch 7/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.9419 - val_loss: 0.9296
    Epoch 8/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.9330 - val_loss: 0.9210
    Epoch 9/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.9243 - val_loss: 0.9126
    Epoch 10/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.9157 - val_loss: 0.9043
    Epoch 11/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.9073 - val_loss: 0.8961
    Epoch 12/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8990 - val_loss: 0.8881
    Epoch 13/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8909 - val_loss: 0.8801
    Epoch 14/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.8829 - val_loss: 0.8722
    Epoch 15/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.8751 - val_loss: 0.8644
    Epoch 16/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.8674 - val_loss: 0.8567
    Epoch 17/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.8598 - val_loss: 0.8492
    Epoch 18/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.8523 - val_loss: 0.8418
    Epoch 19/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.8449 - val_loss: 0.8345
    Epoch 20/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8377 - val_loss: 0.8274
    Epoch 21/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8307 - val_loss: 0.8205
    Epoch 22/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8237 - val_loss: 0.8137
    Epoch 23/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8170 - val_loss: 0.8071
    Epoch 24/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.8104 - val_loss: 0.8006
    Epoch 25/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.8039 - val_loss: 0.7941
    Epoch 26/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.7974 - val_loss: 0.7878
    Epoch 27/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.7911 - val_loss: 0.7816
    Epoch 28/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7849 - val_loss: 0.7755
    Epoch 29/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7787 - val_loss: 0.7695
    Epoch 30/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7728 - val_loss: 0.7637
    Epoch 31/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.7669 - val_loss: 0.7581
    Epoch 32/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7612 - val_loss: 0.7524
    Epoch 33/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.7555 - val_loss: 0.7470
    Epoch 34/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.7499 - val_loss: 0.7416
    Epoch 35/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.7444 - val_loss: 0.7361
    Epoch 36/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7390 - val_loss: 0.7307
    Epoch 37/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7337 - val_loss: 0.7255
    Epoch 38/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.7285 - val_loss: 0.7203
    Epoch 39/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7234 - val_loss: 0.7151
    Epoch 40/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.7185 - val_loss: 0.7101
    Epoch 41/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.7136 - val_loss: 0.7052
    Epoch 42/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.7088 - val_loss: 0.7004
    Epoch 43/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.7042 - val_loss: 0.6958
    Epoch 44/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.6996 - val_loss: 0.6915
    Epoch 45/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6951 - val_loss: 0.6871
    Epoch 46/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6907 - val_loss: 0.6828
    Epoch 47/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.6864 - val_loss: 0.6785
    Epoch 48/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6821 - val_loss: 0.6743
    Epoch 49/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6780 - val_loss: 0.6702
    Epoch 50/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.6739 - val_loss: 0.6661
    Epoch 51/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6699 - val_loss: 0.6622
    Epoch 52/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6660 - val_loss: 0.6583
    Epoch 53/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6622 - val_loss: 0.6546
    Epoch 54/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6585 - val_loss: 0.6508
    Epoch 55/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6548 - val_loss: 0.6472
    Epoch 56/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.6512 - val_loss: 0.6437
    Epoch 57/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6477 - val_loss: 0.6402
    Epoch 58/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6442 - val_loss: 0.6369
    Epoch 59/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6408 - val_loss: 0.6336
    Epoch 60/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6374 - val_loss: 0.6303
    Epoch 61/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6342 - val_loss: 0.6272
    Epoch 62/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6309 - val_loss: 0.6240
    Epoch 63/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6278 - val_loss: 0.6208
    Epoch 64/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6247 - val_loss: 0.6176
    Epoch 65/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6217 - val_loss: 0.6145
    Epoch 66/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6187 - val_loss: 0.6115
    Epoch 67/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.6159 - val_loss: 0.6087
    Epoch 68/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.6130 - val_loss: 0.6060
    Epoch 69/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6102 - val_loss: 0.6033
    Epoch 70/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6075 - val_loss: 0.6007
    Epoch 71/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6049 - val_loss: 0.5981
    Epoch 72/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.6023 - val_loss: 0.5957
    Epoch 73/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5998 - val_loss: 0.5933
    Epoch 74/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5973 - val_loss: 0.5909
    Epoch 75/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5949 - val_loss: 0.5886
    Epoch 76/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5925 - val_loss: 0.5864
    Epoch 77/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5901 - val_loss: 0.5841
    Epoch 78/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5878 - val_loss: 0.5819
    Epoch 79/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.5856 - val_loss: 0.5796
    Epoch 80/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5834 - val_loss: 0.5775
    Epoch 81/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5812 - val_loss: 0.5753
    Epoch 82/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5790 - val_loss: 0.5733
    Epoch 83/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5770 - val_loss: 0.5715
    Epoch 84/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5749 - val_loss: 0.5696
    Epoch 85/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5730 - val_loss: 0.5675
    Epoch 86/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.5710 - val_loss: 0.5656
    Epoch 87/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5691 - val_loss: 0.5636
    Epoch 88/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5672 - val_loss: 0.5616
    Epoch 89/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5653 - val_loss: 0.5597
    Epoch 90/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5635 - val_loss: 0.5579
    Epoch 91/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5618 - val_loss: 0.5562
    Epoch 92/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5600 - val_loss: 0.5545
    Epoch 93/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5583 - val_loss: 0.5528
    Epoch 94/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5567 - val_loss: 0.5510
    Epoch 95/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5550 - val_loss: 0.5494
    Epoch 96/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5534 - val_loss: 0.5479
    Epoch 97/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5518 - val_loss: 0.5463
    Epoch 98/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.5503 - val_loss: 0.5449
    Epoch 99/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5488 - val_loss: 0.5435
    Epoch 100/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5473 - val_loss: 0.5421
    Epoch 101/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5459 - val_loss: 0.5408
    Epoch 102/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5445 - val_loss: 0.5393
    Epoch 103/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5431 - val_loss: 0.5379
    Epoch 104/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5417 - val_loss: 0.5365
    Epoch 105/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5404 - val_loss: 0.5353
    Epoch 106/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5391 - val_loss: 0.5340
    Epoch 107/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5378 - val_loss: 0.5327
    Epoch 108/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5365 - val_loss: 0.5316
    Epoch 109/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5353 - val_loss: 0.5304
    Epoch 110/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5340 - val_loss: 0.5292
    Epoch 111/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5328 - val_loss: 0.5280
    Epoch 112/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5317 - val_loss: 0.5268
    Epoch 113/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5305 - val_loss: 0.5256
    Epoch 114/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5294 - val_loss: 0.5245
    Epoch 115/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5283 - val_loss: 0.5233
    Epoch 116/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.5272 - val_loss: 0.5223
    Epoch 117/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5261 - val_loss: 0.5212
    Epoch 118/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5250 - val_loss: 0.5201
    Epoch 119/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5240 - val_loss: 0.5191
    Epoch 120/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5230 - val_loss: 0.5182
    Epoch 121/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5219 - val_loss: 0.5172
    Epoch 122/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5209 - val_loss: 0.5163
    Epoch 123/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5200 - val_loss: 0.5155
    Epoch 124/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5190 - val_loss: 0.5146
    Epoch 125/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.5181 - val_loss: 0.5137
    Epoch 126/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5172 - val_loss: 0.5127
    Epoch 127/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5163 - val_loss: 0.5118
    Epoch 128/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5154 - val_loss: 0.5109
    Epoch 129/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5145 - val_loss: 0.5101
    Epoch 130/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5136 - val_loss: 0.5092
    Epoch 131/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5128 - val_loss: 0.5083
    Epoch 132/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5119 - val_loss: 0.5074
    Epoch 133/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5111 - val_loss: 0.5065
    Epoch 134/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5102 - val_loss: 0.5056
    Epoch 135/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.5094 - val_loss: 0.5048
    Epoch 136/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5086 - val_loss: 0.5041
    Epoch 137/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5079 - val_loss: 0.5035
    Epoch 138/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5071 - val_loss: 0.5028
    Epoch 139/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5063 - val_loss: 0.5020
    Epoch 140/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5056 - val_loss: 0.5012
    Epoch 141/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5048 - val_loss: 0.5005
    Epoch 142/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5041 - val_loss: 0.4998
    Epoch 143/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.5034 - val_loss: 0.4990
    Epoch 144/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5027 - val_loss: 0.4983
    Epoch 145/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5020 - val_loss: 0.4977
    Epoch 146/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.5013 - val_loss: 0.4970
    Epoch 147/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.5006 - val_loss: 0.4964
    Epoch 148/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4999 - val_loss: 0.4957
    Epoch 149/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4992 - val_loss: 0.4951
    Epoch 150/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4985 - val_loss: 0.4943
    Epoch 151/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4978 - val_loss: 0.4936
    Epoch 152/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4971 - val_loss: 0.4928
    Epoch 153/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4965 - val_loss: 0.4920
    Epoch 154/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4958 - val_loss: 0.4913
    Epoch 155/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4951 - val_loss: 0.4906
    Epoch 156/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4945 - val_loss: 0.4899
    Epoch 157/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4938 - val_loss: 0.4892
    Epoch 158/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4932 - val_loss: 0.4885
    Epoch 159/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4925 - val_loss: 0.4879
    Epoch 160/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4919 - val_loss: 0.4873
    Epoch 161/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.4913 - val_loss: 0.4866
    Epoch 162/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4906 - val_loss: 0.4861
    Epoch 163/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4900 - val_loss: 0.4855
    Epoch 164/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.4894 - val_loss: 0.4850
    Epoch 165/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4887 - val_loss: 0.4844
    Epoch 166/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4881 - val_loss: 0.4838
    Epoch 167/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4875 - val_loss: 0.4832
    Epoch 168/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4869 - val_loss: 0.4827
    Epoch 169/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4863 - val_loss: 0.4821
    Epoch 170/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4857 - val_loss: 0.4815
    Epoch 171/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4851 - val_loss: 0.4809
    Epoch 172/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4844 - val_loss: 0.4803
    Epoch 173/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4838 - val_loss: 0.4797
    Epoch 174/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4832 - val_loss: 0.4792
    Epoch 175/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4826 - val_loss: 0.4787
    Epoch 176/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4820 - val_loss: 0.4782
    Epoch 177/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4814 - val_loss: 0.4776
    Epoch 178/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4808 - val_loss: 0.4771
    Epoch 179/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4802 - val_loss: 0.4764
    Epoch 180/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4796 - val_loss: 0.4758
    Epoch 181/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4790 - val_loss: 0.4751
    Epoch 182/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4783 - val_loss: 0.4744
    Epoch 183/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4777 - val_loss: 0.4737
    Epoch 184/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4771 - val_loss: 0.4730
    Epoch 185/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4764 - val_loss: 0.4723
    Epoch 186/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4758 - val_loss: 0.4716
    Epoch 187/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4752 - val_loss: 0.4709
    Epoch 188/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4745 - val_loss: 0.4701
    Epoch 189/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4739 - val_loss: 0.4692
    Epoch 190/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.4733 - val_loss: 0.4684
    Epoch 191/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4726 - val_loss: 0.4677
    Epoch 192/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4719 - val_loss: 0.4670
    Epoch 193/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4713 - val_loss: 0.4663
    Epoch 194/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4706 - val_loss: 0.4657
    Epoch 195/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4699 - val_loss: 0.4652
    Epoch 196/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4692 - val_loss: 0.4645
    Epoch 197/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4686 - val_loss: 0.4638
    Epoch 198/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4679 - val_loss: 0.4632
    Epoch 199/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4672 - val_loss: 0.4626
    Epoch 200/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4665 - val_loss: 0.4620
    Epoch 201/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4658 - val_loss: 0.4613
    Epoch 202/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4651 - val_loss: 0.4607
    Epoch 203/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4644 - val_loss: 0.4602
    Epoch 204/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.4637 - val_loss: 0.4596
    Epoch 205/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4630 - val_loss: 0.4588
    Epoch 206/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4623 - val_loss: 0.4580
    Epoch 207/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4615 - val_loss: 0.4572
    Epoch 208/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4608 - val_loss: 0.4564
    Epoch 209/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4601 - val_loss: 0.4557
    Epoch 210/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.4594 - val_loss: 0.4549
    Epoch 211/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4586 - val_loss: 0.4542
    Epoch 212/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.4579 - val_loss: 0.4534
    Epoch 213/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4571 - val_loss: 0.4526
    Epoch 214/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4564 - val_loss: 0.4518
    Epoch 215/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4556 - val_loss: 0.4509
    Epoch 216/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4549 - val_loss: 0.4501
    Epoch 217/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4541 - val_loss: 0.4492
    Epoch 218/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4533 - val_loss: 0.4484
    Epoch 219/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4525 - val_loss: 0.4475
    Epoch 220/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4517 - val_loss: 0.4467
    Epoch 221/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4509 - val_loss: 0.4458
    Epoch 222/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4501 - val_loss: 0.4449
    Epoch 223/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4493 - val_loss: 0.4442
    Epoch 224/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4485 - val_loss: 0.4434
    Epoch 225/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4476 - val_loss: 0.4426
    Epoch 226/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4468 - val_loss: 0.4418
    Epoch 227/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4459 - val_loss: 0.4410
    Epoch 228/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4451 - val_loss: 0.4403
    Epoch 229/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4442 - val_loss: 0.4396
    Epoch 230/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4433 - val_loss: 0.4388
    Epoch 231/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4425 - val_loss: 0.4379
    Epoch 232/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.4416 - val_loss: 0.4370
    Epoch 233/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4408 - val_loss: 0.4360
    Epoch 234/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4399 - val_loss: 0.4350
    Epoch 235/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4390 - val_loss: 0.4341
    Epoch 236/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4381 - val_loss: 0.4332
    Epoch 237/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4372 - val_loss: 0.4323
    Epoch 238/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4363 - val_loss: 0.4315
    Epoch 239/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4354 - val_loss: 0.4307
    Epoch 240/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4345 - val_loss: 0.4299
    Epoch 241/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4336 - val_loss: 0.4290
    Epoch 242/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4327 - val_loss: 0.4280
    Epoch 243/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.4318 - val_loss: 0.4270
    Epoch 244/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4308 - val_loss: 0.4261
    Epoch 245/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4299 - val_loss: 0.4252
    Epoch 246/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4289 - val_loss: 0.4243
    Epoch 247/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4280 - val_loss: 0.4234
    Epoch 248/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4270 - val_loss: 0.4224
    Epoch 249/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4260 - val_loss: 0.4215
    Epoch 250/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4251 - val_loss: 0.4205
    Epoch 251/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4241 - val_loss: 0.4195
    Epoch 252/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4231 - val_loss: 0.4185
    Epoch 253/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.4222 - val_loss: 0.4175
    Epoch 254/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4212 - val_loss: 0.4166
    Epoch 255/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4202 - val_loss: 0.4156
    Epoch 256/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4192 - val_loss: 0.4147
    Epoch 257/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4182 - val_loss: 0.4137
    Epoch 258/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4172 - val_loss: 0.4127
    Epoch 259/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4162 - val_loss: 0.4118
    Epoch 260/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4152 - val_loss: 0.4109
    Epoch 261/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4142 - val_loss: 0.4098
    Epoch 262/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4132 - val_loss: 0.4087
    Epoch 263/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4122 - val_loss: 0.4077
    Epoch 264/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4111 - val_loss: 0.4066
    Epoch 265/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4101 - val_loss: 0.4056
    Epoch 266/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4091 - val_loss: 0.4045
    Epoch 267/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4080 - val_loss: 0.4035
    Epoch 268/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4069 - val_loss: 0.4024
    Epoch 269/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4059 - val_loss: 0.4013
    Epoch 270/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4049 - val_loss: 0.4002
    Epoch 271/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4038 - val_loss: 0.3991
    Epoch 272/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4028 - val_loss: 0.3980
    Epoch 273/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.4018 - val_loss: 0.3969
    Epoch 274/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.4008 - val_loss: 0.3959
    Epoch 275/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3997 - val_loss: 0.3949
    Epoch 276/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3987 - val_loss: 0.3940
    Epoch 277/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3977 - val_loss: 0.3930
    Epoch 278/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3967 - val_loss: 0.3921
    Epoch 279/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3957 - val_loss: 0.3912
    Epoch 280/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3947 - val_loss: 0.3901
    Epoch 281/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3937 - val_loss: 0.3891
    Epoch 282/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3927 - val_loss: 0.3880
    Epoch 283/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.3916 - val_loss: 0.3868
    Epoch 284/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3906 - val_loss: 0.3857
    Epoch 285/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3896 - val_loss: 0.3847
    Epoch 286/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3886 - val_loss: 0.3836
    Epoch 287/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3875 - val_loss: 0.3826
    Epoch 288/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3865 - val_loss: 0.3816
    Epoch 289/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3855 - val_loss: 0.3806
    Epoch 290/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3845 - val_loss: 0.3796
    Epoch 291/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3835 - val_loss: 0.3785
    Epoch 292/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3825 - val_loss: 0.3775
    Epoch 293/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3815 - val_loss: 0.3766
    Epoch 294/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3805 - val_loss: 0.3756
    Epoch 295/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3795 - val_loss: 0.3747
    Epoch 296/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3785 - val_loss: 0.3737
    Epoch 297/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3775 - val_loss: 0.3728
    Epoch 298/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3765 - val_loss: 0.3718
    Epoch 299/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3755 - val_loss: 0.3708
    Epoch 300/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3745 - val_loss: 0.3700
    Epoch 301/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3735 - val_loss: 0.3690
    Epoch 302/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3726 - val_loss: 0.3680
    Epoch 303/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3716 - val_loss: 0.3671
    Epoch 304/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3706 - val_loss: 0.3662
    Epoch 305/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3697 - val_loss: 0.3653
    Epoch 306/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3687 - val_loss: 0.3643
    Epoch 307/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3678 - val_loss: 0.3634
    Epoch 308/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3669 - val_loss: 0.3624
    Epoch 309/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3659 - val_loss: 0.3614
    Epoch 310/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3649 - val_loss: 0.3605
    Epoch 311/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3640 - val_loss: 0.3595
    Epoch 312/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3630 - val_loss: 0.3586
    Epoch 313/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3621 - val_loss: 0.3576
    Epoch 314/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3612 - val_loss: 0.3566
    Epoch 315/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3603 - val_loss: 0.3557
    Epoch 316/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3594 - val_loss: 0.3548
    Epoch 317/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3585 - val_loss: 0.3540
    Epoch 318/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3576 - val_loss: 0.3533
    Epoch 319/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3567 - val_loss: 0.3524
    Epoch 320/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3559 - val_loss: 0.3514
    Epoch 321/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3549 - val_loss: 0.3504
    Epoch 322/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3541 - val_loss: 0.3494
    Epoch 323/1000
    2506/2506 [==============================] - 0s 15us/sample - loss: 0.3532 - val_loss: 0.3485
    Epoch 324/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3524 - val_loss: 0.3476
    Epoch 325/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3515 - val_loss: 0.3466
    Epoch 326/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3507 - val_loss: 0.3459
    Epoch 327/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3498 - val_loss: 0.3451
    Epoch 328/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3490 - val_loss: 0.3444
    Epoch 329/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3482 - val_loss: 0.3438
    Epoch 330/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3473 - val_loss: 0.3430
    Epoch 331/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3465 - val_loss: 0.3422
    Epoch 332/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3457 - val_loss: 0.3413
    Epoch 333/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3449 - val_loss: 0.3406
    Epoch 334/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3441 - val_loss: 0.3398
    Epoch 335/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3433 - val_loss: 0.3391
    Epoch 336/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3426 - val_loss: 0.3383
    Epoch 337/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3418 - val_loss: 0.3376
    Epoch 338/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3410 - val_loss: 0.3369
    Epoch 339/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.3403 - val_loss: 0.3362
    Epoch 340/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3395 - val_loss: 0.3355
    Epoch 341/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3388 - val_loss: 0.3349
    Epoch 342/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3381 - val_loss: 0.3341
    Epoch 343/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3373 - val_loss: 0.3333
    Epoch 344/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3367 - val_loss: 0.3324
    Epoch 345/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3359 - val_loss: 0.3318
    Epoch 346/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3352 - val_loss: 0.3312
    Epoch 347/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3345 - val_loss: 0.3306
    Epoch 348/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3338 - val_loss: 0.3299
    Epoch 349/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3331 - val_loss: 0.3292
    Epoch 350/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3324 - val_loss: 0.3284
    Epoch 351/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3316 - val_loss: 0.3276
    Epoch 352/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3310 - val_loss: 0.3268
    Epoch 353/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3303 - val_loss: 0.3261
    Epoch 354/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3296 - val_loss: 0.3254
    Epoch 355/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3290 - val_loss: 0.3247
    Epoch 356/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3283 - val_loss: 0.3241
    Epoch 357/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3277 - val_loss: 0.3236
    Epoch 358/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3270 - val_loss: 0.3230
    Epoch 359/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3264 - val_loss: 0.3224
    Epoch 360/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.3258 - val_loss: 0.3218
    Epoch 361/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3252 - val_loss: 0.3211
    Epoch 362/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.3246 - val_loss: 0.3204
    Epoch 363/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3240 - val_loss: 0.3198
    Epoch 364/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.3234 - val_loss: 0.3193
    Epoch 365/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.3228 - val_loss: 0.3187
    Epoch 366/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.3222 - val_loss: 0.3183
    Epoch 367/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3217 - val_loss: 0.3177
    Epoch 368/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3211 - val_loss: 0.3170
    Epoch 369/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3205 - val_loss: 0.3164
    Epoch 370/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3200 - val_loss: 0.3159
    Epoch 371/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3194 - val_loss: 0.3156
    Epoch 372/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3189 - val_loss: 0.3151
    Epoch 373/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3184 - val_loss: 0.3145
    Epoch 374/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3178 - val_loss: 0.3139
    Epoch 375/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3173 - val_loss: 0.3134
    Epoch 376/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3168 - val_loss: 0.3130
    Epoch 377/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3163 - val_loss: 0.3127
    Epoch 378/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3158 - val_loss: 0.3123
    Epoch 379/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3153 - val_loss: 0.3120
    Epoch 380/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3148 - val_loss: 0.3115
    Epoch 381/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3143 - val_loss: 0.3111
    Epoch 382/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3139 - val_loss: 0.3106
    Epoch 383/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3134 - val_loss: 0.3102
    Epoch 384/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3129 - val_loss: 0.3096
    Epoch 385/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3124 - val_loss: 0.3092
    Epoch 386/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3120 - val_loss: 0.3087
    Epoch 387/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3115 - val_loss: 0.3081
    Epoch 388/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3111 - val_loss: 0.3076
    Epoch 389/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3106 - val_loss: 0.3070
    Epoch 390/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3102 - val_loss: 0.3066
    Epoch 391/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3097 - val_loss: 0.3061
    Epoch 392/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.3093 - val_loss: 0.3056
    Epoch 393/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3089 - val_loss: 0.3051
    Epoch 394/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3085 - val_loss: 0.3047
    Epoch 395/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3080 - val_loss: 0.3043
    Epoch 396/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3076 - val_loss: 0.3038
    Epoch 397/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3072 - val_loss: 0.3033
    Epoch 398/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3068 - val_loss: 0.3029
    Epoch 399/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.3064 - val_loss: 0.3027
    Epoch 400/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3060 - val_loss: 0.3027
    Epoch 401/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3056 - val_loss: 0.3025
    Epoch 402/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3052 - val_loss: 0.3021
    Epoch 403/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3048 - val_loss: 0.3016
    Epoch 404/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3044 - val_loss: 0.3011
    Epoch 405/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3040 - val_loss: 0.3006
    Epoch 406/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3036 - val_loss: 0.3003
    Epoch 407/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3032 - val_loss: 0.2998
    Epoch 408/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.3029 - val_loss: 0.2994
    Epoch 409/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3025 - val_loss: 0.2990
    Epoch 410/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3021 - val_loss: 0.2986
    Epoch 411/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.3018 - val_loss: 0.2983
    Epoch 412/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.3014 - val_loss: 0.2980
    Epoch 413/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3011 - val_loss: 0.2978
    Epoch 414/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.3007 - val_loss: 0.2977
    Epoch 415/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3004 - val_loss: 0.2973
    Epoch 416/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.3000 - val_loss: 0.2969
    Epoch 417/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2997 - val_loss: 0.2964
    Epoch 418/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2993 - val_loss: 0.2959
    Epoch 419/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2990 - val_loss: 0.2954
    Epoch 420/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2987 - val_loss: 0.2950
    Epoch 421/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2984 - val_loss: 0.2946
    Epoch 422/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2980 - val_loss: 0.2942
    Epoch 423/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2977 - val_loss: 0.2939
    Epoch 424/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2974 - val_loss: 0.2936
    Epoch 425/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2971 - val_loss: 0.2934
    Epoch 426/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2968 - val_loss: 0.2931
    Epoch 427/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2964 - val_loss: 0.2929
    Epoch 428/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2961 - val_loss: 0.2926
    Epoch 429/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2958 - val_loss: 0.2924
    Epoch 430/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2955 - val_loss: 0.2921
    Epoch 431/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2952 - val_loss: 0.2919
    Epoch 432/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2949 - val_loss: 0.2919
    Epoch 433/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2946 - val_loss: 0.2917
    Epoch 434/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2943 - val_loss: 0.2915
    Epoch 435/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2941 - val_loss: 0.2913
    Epoch 436/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2938 - val_loss: 0.2909
    Epoch 437/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2935 - val_loss: 0.2906
    Epoch 438/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2932 - val_loss: 0.2903
    Epoch 439/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2929 - val_loss: 0.2900
    Epoch 440/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2926 - val_loss: 0.2898
    Epoch 441/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2923 - val_loss: 0.2896
    Epoch 442/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2921 - val_loss: 0.2893
    Epoch 443/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2918 - val_loss: 0.2889
    Epoch 444/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2915 - val_loss: 0.2886
    Epoch 445/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2912 - val_loss: 0.2882
    Epoch 446/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2909 - val_loss: 0.2879
    Epoch 447/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2907 - val_loss: 0.2876
    Epoch 448/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2904 - val_loss: 0.2873
    Epoch 449/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2901 - val_loss: 0.2870
    Epoch 450/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2899 - val_loss: 0.2869
    Epoch 451/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2897 - val_loss: 0.2868
    Epoch 452/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2894 - val_loss: 0.2866
    Epoch 453/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2891 - val_loss: 0.2865
    Epoch 454/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2889 - val_loss: 0.2862
    Epoch 455/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2886 - val_loss: 0.2858
    Epoch 456/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2883 - val_loss: 0.2855
    Epoch 457/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2881 - val_loss: 0.2851
    Epoch 458/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2878 - val_loss: 0.2849
    Epoch 459/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2876 - val_loss: 0.2845
    Epoch 460/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2873 - val_loss: 0.2842
    Epoch 461/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2871 - val_loss: 0.2839
    Epoch 462/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2868 - val_loss: 0.2837
    Epoch 463/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2866 - val_loss: 0.2833
    Epoch 464/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2864 - val_loss: 0.2830
    Epoch 465/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2861 - val_loss: 0.2828
    Epoch 466/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2859 - val_loss: 0.2828
    Epoch 467/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2857 - val_loss: 0.2828
    Epoch 468/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2854 - val_loss: 0.2826
    Epoch 469/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2852 - val_loss: 0.2823
    Epoch 470/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2849 - val_loss: 0.2821
    Epoch 471/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2847 - val_loss: 0.2818
    Epoch 472/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2845 - val_loss: 0.2815
    Epoch 473/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2843 - val_loss: 0.2813
    Epoch 474/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2840 - val_loss: 0.2811
    Epoch 475/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2838 - val_loss: 0.2809
    Epoch 476/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2836 - val_loss: 0.2807
    Epoch 477/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2834 - val_loss: 0.2804
    Epoch 478/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2831 - val_loss: 0.2801
    Epoch 479/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2830 - val_loss: 0.2797
    Epoch 480/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2827 - val_loss: 0.2795
    Epoch 481/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.2825 - val_loss: 0.2794
    Epoch 482/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2823 - val_loss: 0.2794
    Epoch 483/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2821 - val_loss: 0.2793
    Epoch 484/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2818 - val_loss: 0.2793
    Epoch 485/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2816 - val_loss: 0.2790
    Epoch 486/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2814 - val_loss: 0.2788
    Epoch 487/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2812 - val_loss: 0.2784
    Epoch 488/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2810 - val_loss: 0.2781
    Epoch 489/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2808 - val_loss: 0.2779
    Epoch 490/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2806 - val_loss: 0.2777
    Epoch 491/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2804 - val_loss: 0.2775
    Epoch 492/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2802 - val_loss: 0.2773
    Epoch 493/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2800 - val_loss: 0.2769
    Epoch 494/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2798 - val_loss: 0.2767
    Epoch 495/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2796 - val_loss: 0.2765
    Epoch 496/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2794 - val_loss: 0.2765
    Epoch 497/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2792 - val_loss: 0.2764
    Epoch 498/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2790 - val_loss: 0.2763
    Epoch 499/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2788 - val_loss: 0.2762
    Epoch 500/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2786 - val_loss: 0.2758
    Epoch 501/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2784 - val_loss: 0.2755
    Epoch 502/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2782 - val_loss: 0.2754
    Epoch 503/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2780 - val_loss: 0.2753
    Epoch 504/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2778 - val_loss: 0.2751
    Epoch 505/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2776 - val_loss: 0.2747
    Epoch 506/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2774 - val_loss: 0.2746
    Epoch 507/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2772 - val_loss: 0.2746
    Epoch 508/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2771 - val_loss: 0.2747
    Epoch 509/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2769 - val_loss: 0.2747
    Epoch 510/1000
    2506/2506 [==============================] - 0s 15us/sample - loss: 0.2768 - val_loss: 0.2747
    Epoch 511/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2766 - val_loss: 0.2746
    Epoch 512/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2764 - val_loss: 0.2742
    Epoch 513/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2762 - val_loss: 0.2737
    Epoch 514/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2760 - val_loss: 0.2733
    Epoch 515/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2757 - val_loss: 0.2729
    Epoch 516/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2756 - val_loss: 0.2723
    Epoch 517/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2754 - val_loss: 0.2720
    Epoch 518/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2753 - val_loss: 0.2719
    Epoch 519/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2751 - val_loss: 0.2719
    Epoch 520/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2749 - val_loss: 0.2719
    Epoch 521/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2747 - val_loss: 0.2720
    Epoch 522/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2745 - val_loss: 0.2720
    Epoch 523/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2744 - val_loss: 0.2718
    Epoch 524/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2742 - val_loss: 0.2715
    Epoch 525/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2740 - val_loss: 0.2713
    Epoch 526/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2738 - val_loss: 0.2711
    Epoch 527/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2737 - val_loss: 0.2710
    Epoch 528/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2735 - val_loss: 0.2709
    Epoch 529/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2733 - val_loss: 0.2707
    Epoch 530/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2731 - val_loss: 0.2705
    Epoch 531/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2730 - val_loss: 0.2702
    Epoch 532/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2729 - val_loss: 0.2697
    Epoch 533/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2727 - val_loss: 0.2695
    Epoch 534/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2725 - val_loss: 0.2694
    Epoch 535/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2723 - val_loss: 0.2693
    Epoch 536/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2722 - val_loss: 0.2691
    Epoch 537/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2720 - val_loss: 0.2690
    Epoch 538/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2719 - val_loss: 0.2690
    Epoch 539/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2717 - val_loss: 0.2688
    Epoch 540/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2715 - val_loss: 0.2686
    Epoch 541/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.2714 - val_loss: 0.2684
    Epoch 542/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2712 - val_loss: 0.2682
    Epoch 543/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2710 - val_loss: 0.2681
    Epoch 544/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2709 - val_loss: 0.2680
    Epoch 545/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2707 - val_loss: 0.2679
    Epoch 546/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2706 - val_loss: 0.2677
    Epoch 547/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2704 - val_loss: 0.2674
    Epoch 548/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2703 - val_loss: 0.2671
    Epoch 549/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2701 - val_loss: 0.2669
    Epoch 550/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2700 - val_loss: 0.2668
    Epoch 551/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2698 - val_loss: 0.2667
    Epoch 552/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2697 - val_loss: 0.2665
    Epoch 553/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2695 - val_loss: 0.2663
    Epoch 554/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2694 - val_loss: 0.2661
    Epoch 555/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2693 - val_loss: 0.2659
    Epoch 556/1000
    2506/2506 [==============================] - 0s 15us/sample - loss: 0.2691 - val_loss: 0.2656
    Epoch 557/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2690 - val_loss: 0.2655
    Epoch 558/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2689 - val_loss: 0.2652
    Epoch 559/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2687 - val_loss: 0.2651
    Epoch 560/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2686 - val_loss: 0.2651
    Epoch 561/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2684 - val_loss: 0.2650
    Epoch 562/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2682 - val_loss: 0.2649
    Epoch 563/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2681 - val_loss: 0.2649
    Epoch 564/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2679 - val_loss: 0.2649
    Epoch 565/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2677 - val_loss: 0.2648
    Epoch 566/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2676 - val_loss: 0.2647
    Epoch 567/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2675 - val_loss: 0.2644
    Epoch 568/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2673 - val_loss: 0.2643
    Epoch 569/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2672 - val_loss: 0.2642
    Epoch 570/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2670 - val_loss: 0.2642
    Epoch 571/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2669 - val_loss: 0.2641
    Epoch 572/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2667 - val_loss: 0.2639
    Epoch 573/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2666 - val_loss: 0.2637
    Epoch 574/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2665 - val_loss: 0.2635
    Epoch 575/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2663 - val_loss: 0.2635
    Epoch 576/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2662 - val_loss: 0.2636
    Epoch 577/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2661 - val_loss: 0.2636
    Epoch 578/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2659 - val_loss: 0.2634
    Epoch 579/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2658 - val_loss: 0.2632
    Epoch 580/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2656 - val_loss: 0.2631
    Epoch 581/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2655 - val_loss: 0.2629
    Epoch 582/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2654 - val_loss: 0.2628
    Epoch 583/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2652 - val_loss: 0.2627
    Epoch 584/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2651 - val_loss: 0.2626
    Epoch 585/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2650 - val_loss: 0.2626
    Epoch 586/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2648 - val_loss: 0.2625
    Epoch 587/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2647 - val_loss: 0.2626
    Epoch 588/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2646 - val_loss: 0.2626
    Epoch 589/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2645 - val_loss: 0.2624
    Epoch 590/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2643 - val_loss: 0.2620
    Epoch 591/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2642 - val_loss: 0.2616
    Epoch 592/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2641 - val_loss: 0.2613
    Epoch 593/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2639 - val_loss: 0.2611
    Epoch 594/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2638 - val_loss: 0.2609
    Epoch 595/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2637 - val_loss: 0.2606
    Epoch 596/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2636 - val_loss: 0.2606
    Epoch 597/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2634 - val_loss: 0.2605
    Epoch 598/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2634 - val_loss: 0.2606
    Epoch 599/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2632 - val_loss: 0.2606
    Epoch 600/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2630 - val_loss: 0.2604
    Epoch 601/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2629 - val_loss: 0.2601
    Epoch 602/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2628 - val_loss: 0.2599
    Epoch 603/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2627 - val_loss: 0.2596
    Epoch 604/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2626 - val_loss: 0.2595
    Epoch 605/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2625 - val_loss: 0.2595
    Epoch 606/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2623 - val_loss: 0.2596
    Epoch 607/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2622 - val_loss: 0.2597
    Epoch 608/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2620 - val_loss: 0.2598
    Epoch 609/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2620 - val_loss: 0.2598
    Epoch 610/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2618 - val_loss: 0.2596
    Epoch 611/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2617 - val_loss: 0.2595
    Epoch 612/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2616 - val_loss: 0.2593
    Epoch 613/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2615 - val_loss: 0.2593
    Epoch 614/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2614 - val_loss: 0.2591
    Epoch 615/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2613 - val_loss: 0.2588
    Epoch 616/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2611 - val_loss: 0.2586
    Epoch 617/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2610 - val_loss: 0.2584
    Epoch 618/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2609 - val_loss: 0.2584
    Epoch 619/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2608 - val_loss: 0.2583
    Epoch 620/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2607 - val_loss: 0.2582
    Epoch 621/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2605 - val_loss: 0.2580
    Epoch 622/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2604 - val_loss: 0.2580
    Epoch 623/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2603 - val_loss: 0.2579
    Epoch 624/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2602 - val_loss: 0.2576
    Epoch 625/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2601 - val_loss: 0.2575
    Epoch 626/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2600 - val_loss: 0.2574
    Epoch 627/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2599 - val_loss: 0.2572
    Epoch 628/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2598 - val_loss: 0.2571
    Epoch 629/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2597 - val_loss: 0.2571
    Epoch 630/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2595 - val_loss: 0.2569
    Epoch 631/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2594 - val_loss: 0.2567
    Epoch 632/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2593 - val_loss: 0.2565
    Epoch 633/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2592 - val_loss: 0.2564
    Epoch 634/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2591 - val_loss: 0.2562
    Epoch 635/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2590 - val_loss: 0.2561
    Epoch 636/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2589 - val_loss: 0.2561
    Epoch 637/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2587 - val_loss: 0.2561
    Epoch 638/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2586 - val_loss: 0.2561
    Epoch 639/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2585 - val_loss: 0.2561
    Epoch 640/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2584 - val_loss: 0.2561
    Epoch 641/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2583 - val_loss: 0.2560
    Epoch 642/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2582 - val_loss: 0.2557
    Epoch 643/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2581 - val_loss: 0.2554
    Epoch 644/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2580 - val_loss: 0.2553
    Epoch 645/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2579 - val_loss: 0.2554
    Epoch 646/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2578 - val_loss: 0.2554
    Epoch 647/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2577 - val_loss: 0.2552
    Epoch 648/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2576 - val_loss: 0.2552
    Epoch 649/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2575 - val_loss: 0.2550
    Epoch 650/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2573 - val_loss: 0.2550
    Epoch 651/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2572 - val_loss: 0.2549
    Epoch 652/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2572 - val_loss: 0.2547
    Epoch 653/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2570 - val_loss: 0.2547
    Epoch 654/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2569 - val_loss: 0.2547
    Epoch 655/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2568 - val_loss: 0.2546
    Epoch 656/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2567 - val_loss: 0.2545
    Epoch 657/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2566 - val_loss: 0.2543
    Epoch 658/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2565 - val_loss: 0.2542
    Epoch 659/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2564 - val_loss: 0.2539
    Epoch 660/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2563 - val_loss: 0.2537
    Epoch 661/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2562 - val_loss: 0.2536
    Epoch 662/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2561 - val_loss: 0.2536
    Epoch 663/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2560 - val_loss: 0.2536
    Epoch 664/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2559 - val_loss: 0.2536
    Epoch 665/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2558 - val_loss: 0.2536
    Epoch 666/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2557 - val_loss: 0.2533
    Epoch 667/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2556 - val_loss: 0.2532
    Epoch 668/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2555 - val_loss: 0.2532
    Epoch 669/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2554 - val_loss: 0.2532
    Epoch 670/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2553 - val_loss: 0.2532
    Epoch 671/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2552 - val_loss: 0.2529
    Epoch 672/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2551 - val_loss: 0.2527
    Epoch 673/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2550 - val_loss: 0.2524
    Epoch 674/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2549 - val_loss: 0.2521
    Epoch 675/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2548 - val_loss: 0.2520
    Epoch 676/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2547 - val_loss: 0.2520
    Epoch 677/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2546 - val_loss: 0.2520
    Epoch 678/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2545 - val_loss: 0.2520
    Epoch 679/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2544 - val_loss: 0.2520
    Epoch 680/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2543 - val_loss: 0.2519
    Epoch 681/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2542 - val_loss: 0.2518
    Epoch 682/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2541 - val_loss: 0.2515
    Epoch 683/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2540 - val_loss: 0.2512
    Epoch 684/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2539 - val_loss: 0.2511
    Epoch 685/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2539 - val_loss: 0.2510
    Epoch 686/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2537 - val_loss: 0.2511
    Epoch 687/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2536 - val_loss: 0.2510
    Epoch 688/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2535 - val_loss: 0.2510
    Epoch 689/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2534 - val_loss: 0.2513
    Epoch 690/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2534 - val_loss: 0.2514
    Epoch 691/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2533 - val_loss: 0.2513
    Epoch 692/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2532 - val_loss: 0.2511
    Epoch 693/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2531 - val_loss: 0.2509
    Epoch 694/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2530 - val_loss: 0.2507
    Epoch 695/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2529 - val_loss: 0.2506
    Epoch 696/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2528 - val_loss: 0.2505
    Epoch 697/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2527 - val_loss: 0.2505
    Epoch 698/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2526 - val_loss: 0.2502
    Epoch 699/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2526 - val_loss: 0.2499
    Epoch 700/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2524 - val_loss: 0.2500
    Epoch 701/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2523 - val_loss: 0.2500
    Epoch 702/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2523 - val_loss: 0.2500
    Epoch 703/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2522 - val_loss: 0.2500
    Epoch 704/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2521 - val_loss: 0.2500
    Epoch 705/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2520 - val_loss: 0.2501
    Epoch 706/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2520 - val_loss: 0.2501
    Epoch 707/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2518 - val_loss: 0.2498
    Epoch 708/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2517 - val_loss: 0.2494
    Epoch 709/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2516 - val_loss: 0.2490
    Epoch 710/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2516 - val_loss: 0.2489
    Epoch 711/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2515 - val_loss: 0.2489
    Epoch 712/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2514 - val_loss: 0.2489
    Epoch 713/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2513 - val_loss: 0.2489
    Epoch 714/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.2512 - val_loss: 0.2487
    Epoch 715/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2511 - val_loss: 0.2486
    Epoch 716/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2510 - val_loss: 0.2484
    Epoch 717/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2509 - val_loss: 0.2483
    Epoch 718/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2508 - val_loss: 0.2483
    Epoch 719/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2508 - val_loss: 0.2482
    Epoch 720/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2507 - val_loss: 0.2479
    Epoch 721/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2506 - val_loss: 0.2478
    Epoch 722/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2505 - val_loss: 0.2478
    Epoch 723/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2504 - val_loss: 0.2477
    Epoch 724/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2503 - val_loss: 0.2475
    Epoch 725/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2503 - val_loss: 0.2474
    Epoch 726/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2502 - val_loss: 0.2473
    Epoch 727/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2501 - val_loss: 0.2472
    Epoch 728/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2500 - val_loss: 0.2474
    Epoch 729/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2499 - val_loss: 0.2474
    Epoch 730/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2498 - val_loss: 0.2474
    Epoch 731/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2498 - val_loss: 0.2476
    Epoch 732/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2497 - val_loss: 0.2477
    Epoch 733/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2496 - val_loss: 0.2475
    Epoch 734/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2495 - val_loss: 0.2475
    Epoch 735/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2495 - val_loss: 0.2475
    Epoch 736/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2494 - val_loss: 0.2473
    Epoch 737/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2493 - val_loss: 0.2472
    Epoch 738/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2492 - val_loss: 0.2472
    Epoch 739/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2491 - val_loss: 0.2470
    Epoch 740/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2490 - val_loss: 0.2468
    Epoch 741/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2489 - val_loss: 0.2465
    Epoch 742/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2488 - val_loss: 0.2462
    Epoch 743/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2488 - val_loss: 0.2461
    Epoch 744/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2487 - val_loss: 0.2460
    Epoch 745/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2486 - val_loss: 0.2461
    Epoch 746/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2485 - val_loss: 0.2464
    Epoch 747/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2484 - val_loss: 0.2465
    Epoch 748/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2484 - val_loss: 0.2464
    Epoch 749/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2483 - val_loss: 0.2462
    Epoch 750/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2482 - val_loss: 0.2462
    Epoch 751/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2481 - val_loss: 0.2462
    Epoch 752/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2481 - val_loss: 0.2461
    Epoch 753/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2480 - val_loss: 0.2460
    Epoch 754/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2479 - val_loss: 0.2458
    Epoch 755/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2478 - val_loss: 0.2458
    Epoch 756/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2477 - val_loss: 0.2458
    Epoch 757/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2477 - val_loss: 0.2456
    Epoch 758/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2476 - val_loss: 0.2455
    Epoch 759/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2475 - val_loss: 0.2453
    Epoch 760/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2475 - val_loss: 0.2452
    Epoch 761/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2473 - val_loss: 0.2452
    Epoch 762/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2472 - val_loss: 0.2453
    Epoch 763/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2472 - val_loss: 0.2453
    Epoch 764/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.2471 - val_loss: 0.2453
    Epoch 765/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2471 - val_loss: 0.2451
    Epoch 766/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2470 - val_loss: 0.2449
    Epoch 767/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2469 - val_loss: 0.2446
    Epoch 768/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2468 - val_loss: 0.2444
    Epoch 769/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2467 - val_loss: 0.2442
    Epoch 770/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2467 - val_loss: 0.2441
    Epoch 771/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2466 - val_loss: 0.2442
    Epoch 772/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2465 - val_loss: 0.2442
    Epoch 773/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2464 - val_loss: 0.2440
    Epoch 774/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2463 - val_loss: 0.2439
    Epoch 775/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2463 - val_loss: 0.2437
    Epoch 776/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2462 - val_loss: 0.2435
    Epoch 777/1000
    2506/2506 [==============================] - 0s 16us/sample - loss: 0.2462 - val_loss: 0.2433
    Epoch 778/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2461 - val_loss: 0.2434
    Epoch 779/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2460 - val_loss: 0.2432
    Epoch 780/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2460 - val_loss: 0.2433
    Epoch 781/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2458 - val_loss: 0.2432
    Epoch 782/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2458 - val_loss: 0.2430
    Epoch 783/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2457 - val_loss: 0.2429
    Epoch 784/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2457 - val_loss: 0.2428
    Epoch 785/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2456 - val_loss: 0.2428
    Epoch 786/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2455 - val_loss: 0.2429
    Epoch 787/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2454 - val_loss: 0.2429
    Epoch 788/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2453 - val_loss: 0.2430
    Epoch 789/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2452 - val_loss: 0.2429
    Epoch 790/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2452 - val_loss: 0.2429
    Epoch 791/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2451 - val_loss: 0.2426
    Epoch 792/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2451 - val_loss: 0.2425
    Epoch 793/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2450 - val_loss: 0.2425
    Epoch 794/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2449 - val_loss: 0.2426
    Epoch 795/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2448 - val_loss: 0.2427
    Epoch 796/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2447 - val_loss: 0.2428
    Epoch 797/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2447 - val_loss: 0.2428
    Epoch 798/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2446 - val_loss: 0.2426
    Epoch 799/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2445 - val_loss: 0.2423
    Epoch 800/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2444 - val_loss: 0.2421
    Epoch 801/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2444 - val_loss: 0.2421
    Epoch 802/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2443 - val_loss: 0.2422
    Epoch 803/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2442 - val_loss: 0.2421
    Epoch 804/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2442 - val_loss: 0.2419
    Epoch 805/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2442 - val_loss: 0.2415
    Epoch 806/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2441 - val_loss: 0.2415
    Epoch 807/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2440 - val_loss: 0.2415
    Epoch 808/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2439 - val_loss: 0.2416
    Epoch 809/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2439 - val_loss: 0.2418
    Epoch 810/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2438 - val_loss: 0.2416
    Epoch 811/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2437 - val_loss: 0.2415
    Epoch 812/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2436 - val_loss: 0.2413
    Epoch 813/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2435 - val_loss: 0.2411
    Epoch 814/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2435 - val_loss: 0.2409
    Epoch 815/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2434 - val_loss: 0.2409
    Epoch 816/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2433 - val_loss: 0.2410
    Epoch 817/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2433 - val_loss: 0.2411
    Epoch 818/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2432 - val_loss: 0.2412
    Epoch 819/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2431 - val_loss: 0.2410
    Epoch 820/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2431 - val_loss: 0.2409
    Epoch 821/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2430 - val_loss: 0.2408
    Epoch 822/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2429 - val_loss: 0.2407
    Epoch 823/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2428 - val_loss: 0.2408
    Epoch 824/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2428 - val_loss: 0.2409
    Epoch 825/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2428 - val_loss: 0.2406
    Epoch 826/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2427 - val_loss: 0.2407
    Epoch 827/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2426 - val_loss: 0.2406
    Epoch 828/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2425 - val_loss: 0.2401
    Epoch 829/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2425 - val_loss: 0.2398
    Epoch 830/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2424 - val_loss: 0.2397
    Epoch 831/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2424 - val_loss: 0.2396
    Epoch 832/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2423 - val_loss: 0.2397
    Epoch 833/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2422 - val_loss: 0.2397
    Epoch 834/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.2421 - val_loss: 0.2399
    Epoch 835/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2421 - val_loss: 0.2398
    Epoch 836/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2420 - val_loss: 0.2399
    Epoch 837/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2419 - val_loss: 0.2398
    Epoch 838/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2419 - val_loss: 0.2397
    Epoch 839/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2418 - val_loss: 0.2396
    Epoch 840/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2417 - val_loss: 0.2398
    Epoch 841/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2417 - val_loss: 0.2397
    Epoch 842/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2416 - val_loss: 0.2395
    Epoch 843/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2415 - val_loss: 0.2394
    Epoch 844/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2415 - val_loss: 0.2391
    Epoch 845/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2414 - val_loss: 0.2390
    Epoch 846/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2413 - val_loss: 0.2390
    Epoch 847/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2413 - val_loss: 0.2390
    Epoch 848/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2412 - val_loss: 0.2391
    Epoch 849/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2411 - val_loss: 0.2391
    Epoch 850/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2411 - val_loss: 0.2389
    Epoch 851/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2410 - val_loss: 0.2391
    Epoch 852/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2410 - val_loss: 0.2392
    Epoch 853/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2409 - val_loss: 0.2390
    Epoch 854/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2408 - val_loss: 0.2388
    Epoch 855/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2407 - val_loss: 0.2385
    Epoch 856/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2407 - val_loss: 0.2384
    Epoch 857/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2406 - val_loss: 0.2384
    Epoch 858/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2406 - val_loss: 0.2385
    Epoch 859/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2405 - val_loss: 0.2385
    Epoch 860/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2404 - val_loss: 0.2385
    Epoch 861/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2404 - val_loss: 0.2383
    Epoch 862/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2403 - val_loss: 0.2381
    Epoch 863/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2402 - val_loss: 0.2381
    Epoch 864/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2402 - val_loss: 0.2381
    Epoch 865/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2401 - val_loss: 0.2380
    Epoch 866/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2401 - val_loss: 0.2379
    Epoch 867/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2400 - val_loss: 0.2377
    Epoch 868/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2399 - val_loss: 0.2374
    Epoch 869/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2399 - val_loss: 0.2373
    Epoch 870/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2399 - val_loss: 0.2371
    Epoch 871/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2398 - val_loss: 0.2371
    Epoch 872/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2398 - val_loss: 0.2371
    Epoch 873/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2397 - val_loss: 0.2373
    Epoch 874/1000
    2506/2506 [==============================] - 0s 13us/sample - loss: 0.2396 - val_loss: 0.2374
    Epoch 875/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2396 - val_loss: 0.2376
    Epoch 876/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2395 - val_loss: 0.2376
    Epoch 877/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2395 - val_loss: 0.2377
    Epoch 878/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2394 - val_loss: 0.2376
    Epoch 879/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2393 - val_loss: 0.2375
    Epoch 880/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2392 - val_loss: 0.2374
    Epoch 881/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2392 - val_loss: 0.2372
    Epoch 882/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2391 - val_loss: 0.2371
    Epoch 883/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2390 - val_loss: 0.2372
    Epoch 884/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2390 - val_loss: 0.2376
    Epoch 885/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2390 - val_loss: 0.2375
    Epoch 886/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2389 - val_loss: 0.2373
    Epoch 887/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2388 - val_loss: 0.2369
    Epoch 888/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2388 - val_loss: 0.2365
    Epoch 889/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2387 - val_loss: 0.2365
    Epoch 890/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2386 - val_loss: 0.2365
    Epoch 891/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2386 - val_loss: 0.2364
    Epoch 892/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2385 - val_loss: 0.2363
    Epoch 893/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2385 - val_loss: 0.2360
    Epoch 894/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2384 - val_loss: 0.2359
    Epoch 895/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2384 - val_loss: 0.2358
    Epoch 896/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2383 - val_loss: 0.2358
    Epoch 897/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2382 - val_loss: 0.2360
    Epoch 898/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2381 - val_loss: 0.2361
    Epoch 899/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2381 - val_loss: 0.2363
    Epoch 900/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2380 - val_loss: 0.2362
    Epoch 901/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2380 - val_loss: 0.2360
    Epoch 902/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2379 - val_loss: 0.2359
    Epoch 903/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2379 - val_loss: 0.2358
    Epoch 904/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2378 - val_loss: 0.2358
    Epoch 905/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2377 - val_loss: 0.2358
    Epoch 906/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2377 - val_loss: 0.2359
    Epoch 907/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2377 - val_loss: 0.2359
    Epoch 908/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2376 - val_loss: 0.2357
    Epoch 909/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2375 - val_loss: 0.2357
    Epoch 910/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2375 - val_loss: 0.2355
    Epoch 911/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2374 - val_loss: 0.2355
    Epoch 912/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2374 - val_loss: 0.2356
    Epoch 913/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2373 - val_loss: 0.2356
    Epoch 914/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2373 - val_loss: 0.2355
    Epoch 915/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2372 - val_loss: 0.2353
    Epoch 916/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2371 - val_loss: 0.2351
    Epoch 917/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2371 - val_loss: 0.2349
    Epoch 918/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2370 - val_loss: 0.2349
    Epoch 919/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2369 - val_loss: 0.2349
    Epoch 920/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2369 - val_loss: 0.2346
    Epoch 921/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2369 - val_loss: 0.2345
    Epoch 922/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2368 - val_loss: 0.2347
    Epoch 923/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2367 - val_loss: 0.2348
    Epoch 924/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2367 - val_loss: 0.2346
    Epoch 925/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2366 - val_loss: 0.2347
    Epoch 926/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2366 - val_loss: 0.2345
    Epoch 927/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2365 - val_loss: 0.2342
    Epoch 928/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2365 - val_loss: 0.2340
    Epoch 929/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2364 - val_loss: 0.2339
    Epoch 930/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2364 - val_loss: 0.2339
    Epoch 931/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2363 - val_loss: 0.2339
    Epoch 932/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2362 - val_loss: 0.2339
    Epoch 933/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2362 - val_loss: 0.2339
    Epoch 934/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2361 - val_loss: 0.2339
    Epoch 935/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2360 - val_loss: 0.2341
    Epoch 936/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2360 - val_loss: 0.2343
    Epoch 937/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2360 - val_loss: 0.2343
    Epoch 938/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2359 - val_loss: 0.2342
    Epoch 939/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2359 - val_loss: 0.2340
    Epoch 940/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2358 - val_loss: 0.2340
    Epoch 941/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2357 - val_loss: 0.2340
    Epoch 942/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2357 - val_loss: 0.2337
    Epoch 943/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2356 - val_loss: 0.2335
    Epoch 944/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2356 - val_loss: 0.2331
    Epoch 945/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2355 - val_loss: 0.2330
    Epoch 946/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2356 - val_loss: 0.2330
    Epoch 947/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2355 - val_loss: 0.2331
    Epoch 948/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2354 - val_loss: 0.2332
    Epoch 949/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2353 - val_loss: 0.2333
    Epoch 950/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2353 - val_loss: 0.2334
    Epoch 951/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2352 - val_loss: 0.2333
    Epoch 952/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2352 - val_loss: 0.2332
    Epoch 953/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2351 - val_loss: 0.2329
    Epoch 954/1000
    2506/2506 [==============================] - 0s 12us/sample - loss: 0.2350 - val_loss: 0.2329
    Epoch 955/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2350 - val_loss: 0.2330
    Epoch 956/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2349 - val_loss: 0.2331
    Epoch 957/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2349 - val_loss: 0.2333
    Epoch 958/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2349 - val_loss: 0.2335
    Epoch 959/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2348 - val_loss: 0.2335
    Epoch 960/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2348 - val_loss: 0.2334
    Epoch 961/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2347 - val_loss: 0.2331
    Epoch 962/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2346 - val_loss: 0.2327
    Epoch 963/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2345 - val_loss: 0.2323
    Epoch 964/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2345 - val_loss: 0.2321
    Epoch 965/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2345 - val_loss: 0.2319
    Epoch 966/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2345 - val_loss: 0.2320
    Epoch 967/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2344 - val_loss: 0.2321
    Epoch 968/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2343 - val_loss: 0.2319
    Epoch 969/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2343 - val_loss: 0.2319
    Epoch 970/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2343 - val_loss: 0.2324
    Epoch 971/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2342 - val_loss: 0.2327
    Epoch 972/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2341 - val_loss: 0.2326
    Epoch 973/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2341 - val_loss: 0.2326
    Epoch 974/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2340 - val_loss: 0.2322
    Epoch 975/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2340 - val_loss: 0.2319
    Epoch 976/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2339 - val_loss: 0.2319
    Epoch 977/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2338 - val_loss: 0.2320
    Epoch 978/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2338 - val_loss: 0.2318
    Epoch 979/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2337 - val_loss: 0.2316
    Epoch 980/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2337 - val_loss: 0.2316
    Epoch 981/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2336 - val_loss: 0.2316
    Epoch 982/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2336 - val_loss: 0.2317
    Epoch 983/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2335 - val_loss: 0.2317
    Epoch 984/1000
    2506/2506 [==============================] - 0s 11us/sample - loss: 0.2335 - val_loss: 0.2317
    Epoch 985/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2334 - val_loss: 0.2317
    Epoch 986/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2334 - val_loss: 0.2317
    Epoch 987/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2333 - val_loss: 0.2316
    Epoch 988/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2333 - val_loss: 0.2313
    Epoch 989/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2332 - val_loss: 0.2310
    Epoch 990/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2332 - val_loss: 0.2307
    Epoch 991/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2333 - val_loss: 0.2307
    Epoch 992/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2332 - val_loss: 0.2310
    Epoch 993/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2330 - val_loss: 0.2313
    Epoch 994/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2330 - val_loss: 0.2315
    Epoch 995/1000
    2506/2506 [==============================] - 0s 14us/sample - loss: 0.2330 - val_loss: 0.2317
    Epoch 996/1000
    2506/2506 [==============================] - 0s 9us/sample - loss: 0.2330 - val_loss: 0.2317
    Epoch 997/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2329 - val_loss: 0.2316
    Epoch 998/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2329 - val_loss: 0.2312
    Epoch 999/1000
    2506/2506 [==============================] - 0s 8us/sample - loss: 0.2328 - val_loss: 0.2310
    Epoch 1000/1000
    2506/2506 [==============================] - 0s 10us/sample - loss: 0.2327 - val_loss: 0.2307

Let's visualize the training and validation losses.

```python
plt.figure(figsize=(15,5))

for label in ["loss","val_loss"]:
    plt.plot(train_history.history[label], label=label)

plt.ylabel("loss")
plt.xlabel("epoch")
plt.title("The final validation loss: {}".format(train_history.history["val_loss"][-1]))
plt.legend()
plt.show()
```

Finally, let's look at the parameters for the trained model.

```python
for layer in model.layers:
  print("{}, {}".format(layer.name, layer.get_weights()))
```

    hidden_layer, [array([[0.2797624]], dtype=float32), array([[-3.6514106]], dtype=float32), array([0.00288544], dtype=float32)]
    output_layer, [array([[-1.102846]], dtype=float32), array([-0.00604694], dtype=float32)]

**QUESTION**:
* Relate the above weights to the terms in the equation for the vanilla RNN we saw earlier, namely:
  * input-to-hidden $W_{xh}$,
  * hidden-to-hidden $W_{hh}$,
  * hidden-to-output weights $W_{hy}$
  * recurrent and out biases.

###Make predictions using the trained model

```python
y_pred = model.predict(X_test[:100])
plt.figure(figsize=(19,3))

plt.plot(y_test[:100], label="true")
plt.plot(y_pred, label="predicted")
plt.legend()
plt.show()
```

**YOUR TASKS**:
* [**ALL**] Change the learning rate up to 5 orders of magnitude larger and smaller and retrain. What happens when it is too large? What happens when it is too small?
* [**ALL**] Change the `SimpleRNN` to `GRU`.
  * You can read more about LSTMs in [this blog post](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Chris Olah.
  * What is the effect on the number of parameters? Can you explain why? Now do the same for `LSTM`.
* [**INTERMEDIATE**] Note that the loss does not decrease much after around epoch 400. Add "Early Stopping with patience" to the `model.fit()` function to stop it from training beyond this point. **Hint**: Look at tf.keras.callbacks.
  * *Early stopping* is a technique where we stop training the model once it's performance on validation data stops improving, and is a very common approach to prevent overfitting. Early stopping *with patience* means as soon as the model starts doing worse on validation we wait for at least `patience` more evaluations before stopping training, and if it improves within that time, we reset the counter. The patience parameter is a way to ensure we don't stop by accident, as the validation loss can fluctuate randomly from epoch to epoch.

##Generating Shakespeare

Now let's build an RNN language model to generate Shakespearian English! A language model is trained to assign high probabilities to sequences of words or sentences that are well formed, and low probabilities to sequences which are not realistic. When the model is trained, one can use it to *generate* data that is similar to the training data.

Our data is now sequences of discrete symbols (characters). But neural networks operate in continuous spaces, and so we need to take the discrete language data, and **embed** it in a continuous space. To do this, we'll simply break up the data into sequences of characters, and represent each character using a learned vector. This is a standard trick for processing text using neural networks.

### Download and Preprocess the Data

We first download the data and examine what it looks like:

```python
context = ssl._create_unverified_context()
shakespeare_url = 'https://cs.stanford.edu/people/karpathy/char-rnn/shakespeare_input.txt'

data = urllib.request.urlopen(shakespeare_url, context=context)
all_text = data.read().lower().decode('utf8')

print("Downloaded Shakespeare data with {} characters.".format(len(all_text)))
print("FIRST 1000 CHARACTERS: ")
print(all_text[:1000])
```

    Downloaded Shakespeare data with 4573338 characters.
    FIRST 1000 CHARACTERS:
    first citizen:
    before we proceed any further, hear me speak.

    all:
    speak, speak.

    first citizen:
    you are all resolved rather to die than to famish?

    all:
    resolved. resolved.

    first citizen:
    first, you know caius marcius is chief enemy to the people.

    all:
    we know't, we know't.

    first citizen:
    let us kill him, and we'll have corn at our own price.
    is't a verdict?

    all:
    no more talking on't; let it be done: away, away!

    second citizen:
    one word, good citizens.

    first citizen:
    we are accounted poor citizens, the patricians good.
    what authority surfeits on would relieve us: if they
    would yield us but the superfluity, while it were
    wholesome, we might guess they relieved us humanely;
    but they think we are too dear: the leanness that
    afflicts us, the object of our misery, is as an
    inventory to particularise their abundance; our
    sufferance is a gain to them let us revenge this with
    our pikes, ere we become rakes: for the gods know i
    speak this in hunger for bread, not in thirst for revenge.

```python
training_text = all_text[:1000000] # Keep only the first 1 million characters
```

We now preprocess the text data as follows:

1. Extract the vocabulary of all `vocab_size` unique characters appearing in the data.

2. Assign each character a unique integer id in `0 <= id < vocab_size`. This is so we can map the characters to unique embedding vectors. This is a common way to map discrete inputs to continuous vectors that neural networks can work with. See e.g. this [blog post](https://www.tensorflow.org/tutorials/representation/word2vec), or [this one](http://ruder.io/word-embeddings-1/) for more information.

3. Split the data into sequences ("windows") of `max_len` characters (the input to the model) followed by the next character as target. E.g. using `max_len=5` the sentence "I saw a cat" (11 characters) will get split into "I saw" and /space/, "/space/saw/space/" and "a", "saw a" and /space/, etc. To add some variation, we skip `step` characters between each sequence (i.e. we use a "sliding window of `max_len` with stride `step`").

```python
max_len = 30  # We only consider this many previous data points (characters)
step = 3 # We start a new training sequence every `step` characters
sentences = [] # This holds our extracted sequences
next_chars = [] # This holds the targets (the follow-up characters)

chars = sorted(list(set(training_text)))  # List of unique characters in the corpus
vocab_size = len(chars)
print('Number of unique characters: ', vocab_size)
print(chars)

# Construct dictionaries mapping unique characters to their index in `chars` and reverse
char2index = dict((c, chars.index(c)) for c in chars)
index2char = dict((chars.index(c), c) for c in chars)
```

    Number of unique characters:  39
    ['\n', ' ', '!', '$', '&', "'", ',', '-', '.', '3', ':', ';', '?', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Now we encode the training data by mapping each character to its unique integer id.

```python
for i in range(0, len(training_text) - max_len, step):
    sentences.append([char2index[s] for s in training_text[i: i + max_len]])
    next_chars.append([char2index[training_text[i + max_len]]])

print('Number of extracted sequences:', len(sentences))
```

    Number of extracted sequences: 333324

This yields the following numpy arrays:

```python
X, Y = np.array(sentences, dtype=np.int64), np.array(next_chars, dtype=np.int64)
X.shape, Y.shape
```

    ((333324, 30), (333324, 1))

Let's take a look at the first example.

```python
print("X[0].shape = {}, Y[0].shape = {}".format(X[0].shape, Y[0].shape))
print("X[0]: ", X[0])
print("Y[0]: ", Y[0])
```

    X[0].shape = (30,), Y[0].shape = (1,)
    X[0]:  [18 21 30 31 32  1 15 21 32 21 38 17 26 10  0 14 17 18 27 30 17  1 35 17
      1 28 30 27 15 17]
    Y[0]:  [17]

###Build an RNN language model

A **language model** estimates a probability distribution over sequences $\mathbb{x}_{1:N} = (x_1, x_2, ..., x_N)$ by breaking up the full joint probability into a sequence of conditional probabilities using the **[chain-rule of probability](https://en.wikipedia.org/wiki/Chain_rule_(probability))**:

\begin{align}
 p(\mathbb{x}_{1:N}) &= p(x_1) \cdot p(x_2 | x_1) \cdot p(x_3 | x_2, x_1) \ldots \\
 &= \Pi_1^N p(x_i | \mathbb{x}_{1:i-1})
\end{align}

In other words, to model the probability of the phrase "*i saw a cat*" at the character level, the model learns to estimate the probabilities for p(i), p(/space/| i), p(c | i, /space/), and so forth, and multiplies them together.

There are many different ways in which to estimate these individual probabilities. But one particularly effective way is to use an RNN! To do this, we'll therefore be modeling the  $p(x_i | \mathbb{x}_{1:i-1})$ terms using an RNN conditioned on $\mathbb{x}_{1:i-1}$.

* We model these probabilities at the character-level, so we'll use an `Embedding` layer as the first layer of our model to map the discrete character id's to real-valued embedding vectors.
* Next, the RNN-core will map these sequences of character embeddings to a probability distribution over all characters $p(x_i | \mathbb{x}_{1:i-1}) \in \mathbb{R}^\textrm{vocab_size}$ at every step of the sequence. To do this, the RNN will map the embeddings to a sequence of *hidden states*. We will then use a `Dense` layer to map from the RNN hidden state to an output distribution over the total number of characters using a [`softmax`](https://en.wikipedia.org/wiki/Softmax_function) activation.

We can do this with a few lines of code:

```python
embedding_dim = 32   # Map each character to a unique vector of this dimension
vocab_size = len(chars)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Embedding(
  vocab_size, embedding_dim,
  input_length=max_len,
  embeddings_initializer=tf.keras.initializers.TruncatedNormal))
model.add(tf.keras.layers.LSTM(
  128,
  input_shape=(max_len, embedding_dim),  # NB: Ensure this matches the embedding_dim!
  dropout=0.1,  # input-to-hidden drop-probability
  recurrent_dropout=0.2))  # hidden-to-hidden drop-probability
model.add(tf.keras.layers.Dense(vocab_size, activation='softmax'))

model.summary()
```

    W0708 18:30:15.728784 139985615468416 deprecation.py:506] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/keras/initializers.py:94: calling TruncatedNormal.__init__ (from tensorflow.python.ops.init_ops) with dtype is deprecated and will be removed in a future version.
    Instructions for updating:
    Call initializer instance with the dtype argument instead of passing it to the constructor

    Model: "sequential_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    embedding (Embedding)        (None, 30, 32)            1248
    _________________________________________________________________
    lstm (LSTM)                  (None, 128)               82432
    _________________________________________________________________
    dense (Dense)                (None, 39)                5031
    =================================================================
    Total params: 88,711
    Trainable params: 88,711
    Non-trainable params: 0
    _________________________________________________________________

###Select the optimizer and loss

Once we have a model that can map sequence of characters to a probability distribution over the next character in the sequence, we can train it using **maximum likelihood** on the training set to find the model parameters which maximizes the probability of the training data. Again, this is very simple to do by choosing an optimizer and selecting the `sparse_categorical_crossentropy` loss function:

```python
optimizer = tf.keras.optimizers.Adam()
loss='sparse_categorical_crossentropy'

model.compile(loss=loss, optimizer=optimizer)
```

###Helper functions

```python
def sample_with_temp(preds, temperature=1.0):
  preds = np.asarray(preds).astype('float64')
  preds = np.log(preds) / temperature
  exp_preds = np.exp(preds)
  preds = exp_preds / np.sum(exp_preds)
  probas = np.random.multinomial(1, preds, 1)
  return np.argmax(probas)
```

```python
def shift_and_append(test_arr, next_item):
  '''Returns a copy of test_arr with items shifted one position to the left and
     next_item appended.
  '''
  tmp = np.empty_like(test_arr)
  tmp[:,:-1] = test_arr[:,1:]
  tmp[:,-1] = next_item
  return tmp

## TEST the above function:
test_arr = np.array([[1,2,3,4]])

print("test_arr = {}".format(test_arr))
test_arr = shift_and_append(test_arr, 5)
print("roll_arr(test_arr, 5) = {}".format(test_arr))
```

    test_arr = [[1 2 3 4]]
    roll_arr(test_arr, 5) = [[2 3 4 5]]

```python
def sample_from_model(model,
                      num_generate=400,
                      prev_text=None,  # the text used to condition the model
                      temperatures=[0.2, 0.5, 1.0, 1.2]):

    if not prev_text:
      # Select a text seed at random
      start_index = random.randint(0, len(training_text) - max_len - 1)
      while ((start_index < (len(training_text) - max_len - 1)) and (
        training_text[start_index - 1] is not ' ')):
        start_index += 1  # Advance to beginning of new word
      prev_text = training_text[start_index: start_index + max_len]

    if len(prev_text) != max_len:
      print("`prev_text` must be of length `max_len`.")
      return

    print('GENERATING TEXT WITH SEED: \n"' + prev_text + '"')
    prev_text_arr = np.array(
      [[char2index[c] for c in prev_text]], dtype=np.int64)

    for temp in temperatures:
        print('==TEMPERATURE:', temp)
        sys.stdout.write(prev_text)

        # Start with the same sampled text for all temperatures
        generated_text = prev_text
        generated_text_arr = prev_text_arr

        # Now generate this many characters
        for i in range(num_generate):

            # Get the output softmax given the conditioning text
            #prev_text = generated_text_enc[np.newaxis,:]
            preds = model.predict(generated_text_arr, verbose=0)[0]

            next_index = sample_with_temp(preds, temp)
            next_char = index2char[next_index]
            generated_text += next_char
            generated_text = generated_text[1:]

            # Left-shift and add into encoded array
            generated_text_arr = shift_and_append(generated_text_arr, next_index)

            sys.stdout.write(next_char)
            sys.stdout.flush()
        print()
```

###Train the model

Let's train the model! The code below will train the model on a subset of the available data, and then generate from the model every `sample_every` number of batches.

To generate from the model, we use `model.predict()` on a sequence of `max_len` conditioning characters to produce an output distribution over `vocab_size` characters. We then sample one character from this distribution and shift everything up by one and append the new characters. By repeating this, we can generate text from the (partially-trained) model.

**NOTE**:
* It takes a while to train a model that starts generating anything resembling the Shapespeare text! In general it should start getting the rough structure in place around the 100K training example mark (examples, not batches). But to generate any meaningful words will need several hundred thousand examples.
* We sample with *temperature*. This is a way to sharpen or flatten the probabilities produced by the model. By lowering the temperature, we emphasize the modes of the predicted distribution, and by increasing the temperature, we flatten the modes (tends towards uniform). Higher temperatures therefore encourage the model to be more 'creative', instead of always choosing the most likely next character.

```python
from google.colab import files
uploaded = files.upload()
```

<input type="file" id="files-85bb6fdc-f708-4e58-9b59-84454b6b983c" name="files[]" multiple disabled />
<output id="result-85bb6fdc-f708-4e58-9b59-84454b6b983c">
 Upload widget is only available when the cell has been executed in the
 current browser session. Please rerun this cell to enable.
 </output>

```python
batch_size = 128
total_num_batches = X.shape[0] // batch_size
sample_every = 256  # Train on this many batches, then generate something

print("Training on {} batches in total.".format(total_num_batches))

for cur_batch in range(0, total_num_batches, sample_every):
    print('TRAINING ON BATCH {} to {} (example {} to {})'.format(
      cur_batch, cur_batch + sample_every,
      cur_batch * batch_size, (cur_batch + sample_every) * batch_size)
    )

    X_batch = X[batch_size * cur_batch : batch_size * (cur_batch + sample_every), :]
    Y_batch = Y[batch_size * cur_batch : batch_size * (cur_batch + sample_every), :]

    '''
    # Show the first 5 examples to make sure we're not training on garbage
    print("X_batch.shape = {}".format(X_batch.shape))
    print("Y_batch.shape = {}".format(Y_batch.shape))
    print("FIRST 5 EXAMPLES:")
    for num in range(5):
      in_seq = [index2char[int(indx)] for indx in np.nditer(X_batch[num, :])]
      next_char = index2char[Y_batch[num, 0]]
      print(str(num) + '. ' + ''.join(in_seq) + '-->' + next_char)
    '''

    model.fit(X_batch, Y_batch,
              batch_size=batch_size,
              epochs=1,
              verbose=1)

    print("GENERATING SOME RANDOM TEXT FROM THE MODEL")
    sample_from_model(model)
```

    W0708 18:30:57.057747 139985615468416 deprecation.py:323] From /usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/math_grad.py:1250: add_dispatch_support.<locals>.wrapper (from tensorflow.python.ops.array_ops) is deprecated and will be removed in a future version.
    Instructions for updating:
    Use tf.where in 2.0, which has the same broadcast rule as np.where

    Training on 2604 batches in total.
    TRAINING ON BATCH 0 to 256 (example 0 to 32768)
    32768/32768 [==============================] - 15s 464us/sample - loss: 2.9973
    GENERATING SOME RANDOM TEXT FROM THE MODEL
    GENERATING TEXT WITH SEED:
    "have poison'd good camillo's h"
    ==TEMPERATURE: 0.2
    have poison'd good camillo's he he he hous ore thon ar he hone ho hor houn hous houn the the he he con hour hothe he athe hon woo hor hor ho he the houn he he hothe he he hathe he ire in the the hous the ho hor athe hothe he he as hon hor he he the he he the hitar hon the the ho he hor hore hon are he hore an an ho ar he hor he hor are he he he an he ano hothe he ar hor he he ho he he the ire he hore as hor he he he the tor he
    ==TEMPERATURE: 0.5
    have poison'd good camillo's hol ho ire e toi bothan hiore weol ani ais aaf arare sod the holtoy toi he en then hicor paur chin he theit ee homis
    thon thi hon horen he hin antu uthori he he hei thoy wole the hen
     poin he thouds mord hoche hont fe or h thhouro the ittoran aore
    wonce hout moi hi or
    hos ye ol ton the ios cotli hohe an hee lio the toran atle are hed theu he yt hon oo hous sor an poyu hin the ul s wone heu ye iruun
    ==TEMPERATURE: 1.0
    have poison'd good camillo's hetuiuis adtr hyifmet ro chof whoui? ok men
    pami!amtmmm; louuslm piol edgivasou
    bine

    iot
    s aun h've feu fo sydiuilsn taf h urnumsn
    metllthe li.ent
    anuu:hirn ciachhedd
    visif weihuften e hedsgere davo mone mood:
    aise, helinulune'rir?r na s ahoiwit dh hoire arthscou
    hvahg  hivf he whonothe pothe? hheurs d womk wio iohesud heutui
    d tyeu
    yutwowh thetd, ungamar oed ffo whcar :it hoorlwiil
    og
    fnn bcde so
    ==TEMPERATURE: 1.2
    have poison'd good camillo's hetdteo hwha nis
    er
    ytlge.el,d, focldevsrtirnzops
    chotml, ictyethen we uh
    coutalotolswa
    putatcanlzeimlh omisctacwthedkares fe toeksdelsvl,i-mi fove vo gouu
    sn kinr 'iwe; velhm
    wamog'hihtve atcvoammo wgnolof hfchin .up? i iwwo 'o
    lcihiutd lounisd
    p,ord  tacsroorrhe ihis
    decmap.ni n hitheruuh!
    cthe putjpstpomoom
    shi-ab e tnrtvohima, weoyy irm horj
    limen
    mak miut hootlure-se.ur:k mof rogh
    hi tl.s,s 'r
    TRAINING ON BATCH 256 to 512 (example 32768 to 65536)
    32768/32768 [==============================] - 15s 447us/sample - loss: 2.5145
    GENERATING SOME RANDOM TEXT FROM THE MODEL
    GENERATING TEXT WITH SEED:
    "gods be good unto us!

    meneniu"
    ==TEMPERATURE: 0.2
    gods be good unto us!

    menenius ho he the the wore the the the the here wous hore wore the the the the the the het thare he the the the the wous the the the the the he the wore wore he the hone the hethe wous hind the the the the the the the tore th

**NOTE**: Even after training has stopped you can still generate from the (partially trained) model as follows:

```python
my_text = "       the meaning of life is:"  # Needs to be max_len characters
print(len(my_text))
sample_from_model(model, prev_text=my_text)
```

###IMPORTANT NOTES
* Even if you stop training the model weights are persistant. If you resume training it will start where you left off.
* To reset the weights, you need to recompile the model.
* Sampling is **stochastic** (random), so you'll get new outputs every time you rerun the sampling code.

### YOUR TASKS:
* [**ALL**] Read the generations from your model in a funny voice to your neighbour.
* [**ALL**] Increase `max_len` and regenerate the data and retrain the model.
 * What's the effect on training speed as you double `max_len`. Can you explain why?
 * Do you notice any effect on the quality of the model? Can you explain why?
* [**ALL**] Set the max_len to be roughly the length of half a word; one word; two words... What kind of samples do these models generate? Explain how they go wrong.
* [**ALL**] Change `embedding_dim` and the hidden size of the LSTM and observe the effect on training speed and quality.
* [**INTERMEDIATE**] Change the dropout rates & retrain the model.
  * What types of dropout do we get for recurrent models?
  * What's the effect on the text quality?
* [**ADVANCED**] Implement the "teacher forcing" training methodology, where the net must predict the entire output sequence shifted forward by one character, instead of just the next character. Compare the output of a model trained with teacher forcing versus the per-character model, given a similar training time. Is it fair to say that teacher forcing is a more efficient training methodology?

##Further reading

* https://distill.pub/2016/augmented-rnns/
* https://distill.pub/2017/ctc/
* https://algotravelling.com/en/machine-learning-fun-part-5/