# Practical 3: Recurrent Neural Networks (RNNs)

## Introduction

Feedforward models (eg deep MLPs and ConvNets) map fixed-size input-data (vectors of a fixed dimensionality) to their output labels. They're very powerful and have been successfully used for many tasks. However, a lot of data is not in the form of fixed-size vectors, but exists in the form of **sequences**. Language is one good example, where sentences are sequences of words. In some way, almost any data types can be considered as a sequence (for instance an image consists of a sequence of pixels, speech a sequence of phonemes, and so forth).

Recurrent neural networks (**RNNs**) were designed to be able to handle sequential data, and in this practical we will take a closer look at RNNs and then build a model that can generate English sentences in the style of Shakespeare!

## Learning Objectives
* Understand how RNNs model sequential data.
* Understand how the vanilla RNN is a generalization of feedforward models to incorporate sequential dependencies.
* Understand the issues involved when training RNNs.
* Know how to implement an RNN for time-series estimation (**regression**) and an RNN language model (character-level **classification**) in Tensorflow using Keras.

##Imports

```
#@title Imports (RUN ME!) { display-mode: "form" }

#!pip -q install pydot_ng
#!pip -q install graphviz
#!apt install graphviz > /dev/null

from __future__ import absolute_import, division, print_function

import numpy as np
import tensorflow as tf
import math
import random
import ssl
import sys
import urllib2
from IPython import display
import matplotlib.pyplot as plt
%matplotlib inline

print('Running TensorFlow version %s' % (tf.__version__))
try:
  tf.enable_eager_execution()
  print('Eager mode activated.')
except ValueError:
  print('Already running in Eager mode')
```

    Running TensorFlow version 1.10.1
    Eager mode activated.

##From Feedforward to Recurrent Models

### Intuition
RNNs generalize feedforward networks (FFNs) to be able to work with sequential data. FFNs take an input (e.g. an image) and immediately produce an output (e.g. a digit class), something like this:

```
def ffn_forward(x, W_xh, W_ho, b_hid, b_out):

    # Compute activations on the hidden layer.
    hidden_layer = act_fn(np.dot(W_xh, x) + b_hid)

    # Compute the (linear) output layer activations.
    output = np.dot(W_ho, hidden_layer) + b_out

    return output
```

**NOTE**: You don't have to run this cell, it's just shown to illustrate the point.

RNNs, on the other hand, consider the data sequentially, and can remember what they have seen earlier in the sequence to help interpret or contextualize elements from later in the sequence when making predictions, something like this:

```
def rnn_forward(data_sequence, initial_state):

    state = initial_state  # Reused at every time-step
    all_states, all_ys = [state], []  # Used to save all states and predictions

    for x, y in data_sequence:

      # recurrent_fn() takes the current input and the previous state and produces a new state
      new_state, y_pred = recurrent_fn(x, state)

      all_states.append(new_state)
      all_ys.append(y_pred)

      # Update state for the next time-step
      state = new_state

    return all_states, all_ys
```

To understand the distinction between FFNs and RNNs, imagine we want to label words as the part-of-speech categories that they belong to: E.g. for the input sentence "I want a duck" and "He had to duck", we want our model to predict that duck is a `Noun` in the first sentence and a `Verb` in the second. To do this successfully, the model needs to be aware of the surrounding context. However, if we feed a FFN model only one word at a time, how could it know the difference? If we want to feed it all the words at once, how do we deal with the fact that sentences are of different lengths?

RNNs solve this issue by processing the sentence word-by-word, and maintaining an internal **state** summarizing what it has seen so far. This applies not only to words, but also to phonemes in speech, or even, as we will see, elements of a time-series.

### Unrolling the network

Imagine we are trying to classify a sequence $X = (x_1, x_2, \ldots, x_N)$ into labels $y$ (for now, let's keep it abstract). After running the `rnn_forward()` function of our RNN defined above on $X$, we would have a list of internal states and outputs of the model at each sequence position. This process is called **unrolling or unfolding in time**, because you can think of it as unrolling the *computations* defined by the RNN loop over the inputs at each position of the sequence.  RNNs are often used to model **time series data** (which we will do in this practical), and therefore these positions are referred to as **time-steps**, and hence we call this process "unrolling over time". See the visualization below from this great [blog post](http://www.wildml.com/2015/09/recurrent-neural-networks-tutorial-part-1-introduction-to-rnns/) (it uses slightly different notation, but the idea should be clear):

> **We can therefore think of an RNN as a composition of identical feedforward neural networks (with replicated/tied weights), one for each moment or step in time. **

These feedforward functions that make up the RNN(i.e. our `recurrent_fn` above) are typically referred to as **cells**, and the only restriction on its API is that the cell function needs to be a differentiable function that can map an input and a state vector to an output and a new state vector. What we have shown above is called the **vanilla RNN**, but there are many more possibilities. One of the most popular variants is called the **Long Short-Term Memory (LSTM)** cell, which we'll use later to create our Shakespeare language model.

### Putting this together

In the feedforward models we've seen before, the input $x$ is mapped to an intermediate hidden layer $h$ as follows:

\begin{equation}
  h = \sigma(\underbrace{W_{xh}x}_\text{current input (per-example)} + b)
\end{equation}

where $\sigma$ is some non-linear activation function like ReLU or tanh.  We can then make a prediction $\hat{y} = \sigma(W_{hy}h + b)$ based on $h$, or we can add another layer, etc.  **NOTE**: We use the weight subscript $W_{xz}$ to indicate a mapping from layer $x$ to layer $z$.

RNNs generalize this idea to a sequence of inputs $X = {x_1, x_2, ...}$ by maintaining a sequence of state vectors $h_t$, one for every time-step $t$, as follows:

\begin{equation}
  h_t = \sigma(\underbrace{W_{hh}h_{t-1}}_\text{previous state} + \underbrace{W_{xh}x_t}_\text{current input (per time-step)} + b)
\end{equation}

Feedforward models map one input to one output. On the other hand, RNNs can give **many-to-many** (many inputs, many labels), **many-to-one**, and **one-to-many** mappings. For example, we could use each $h_t$ to predict the part of speech for that time-step ($y_t= \sigma(W_{hy}h_t+b)$) or we could use the last state $h_T$ to predict the topic of the whole document. Which task the RNN performs is based on the training data (of course)! We can visualize these as follows (fromt this [excellent blog post](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) by Andrej Karpathy):

**QUESTIONS**
* How are FFNs and RNNs **similar**?
* How are they **different**?
* Why do we call RNNs "recurrent"?
* Can you think of a one-to-many task?

##Modeling General Time-Series

We will train an RNN to model a time-series as a first step. A **time-series** is a series of data-points ordered over discrete time-steps. Examples include the hourly temperature of Stellenbosch over a month or a year, the market price of some asset (like a company's stock) over time, and so forth. We will generate a **sinusoidal time-series** (with or without noise) as a toy example, and then train a tiny RNN model with only 5 parameters on this data.

###Create some artificial data

```
#@title Create sinusoidal data {run: "auto"}
steps_per_cycle = 20 #@param { type: "slider", min:1, max:100, step:1 }
number_of_cycles = 176 #@param { type: "slider", min:1, max:1000, step:1 }
noise_factor = 0.1 #@param { type: "slider", min:0, max:1, step:0.1 }
plot_num_cycles = 23 #@param { type: "slider", min:1, max:50, step:1 }

seq_len = steps_per_cycle * number_of_cycles
t = np.arange(seq_len)
sin_t_noisy = np.sin(2 * np.pi / steps_per_cycle * t + noise_factor * np.random.uniform(-1.0, +1.0, seq_len))
sin_t_clean = np.sin(2 * np.pi / steps_per_cycle * t)

upto = plot_num_cycles * steps_per_cycle
fig = plt.figure(figsize=(15,3))
plt.plot(t[:upto], sin_t_noisy[:upto])
plt.title("Showing first {} cycles.".format(plot_num_cycles))
plt.show()

#both = np.column_stack((t, sin_t_noisy))
#print("both.shape = {}".format(both.shape))

#print("both[:steps_per_cycle, :steps_per_cycle]")
#print(both[:steps_per_cycle,:steps_per_cycle])
```

**TASK**: Adjust the parameters above to generate data with different properties.

Now we pack the data into train and test batches. Note that while RNNs can in theory learn the dependencies across all inputs received so far (using an algorithm called **backpropagation through time**, or BPTT; see the Aside box below), in practice they are trained using an algorithm called **truncated BPTT** where we truncate the inputs to only the last $T$ symbols (this is the `truncated_seq_len` variable below).

**QUESTIONs**:
* What issues can you think may arise by truncating the training data in this way?
* Despite these issues, why do you think it might be necessary to do this?

```
#@title Pack truncated sequence data {run: "auto"}

def pack_truncated_data(data, num_prev = 100):
    X, Y = [], []
    for i in range(len(data) - num_prev):
      X.append(data[i : i + num_prev])
      Y.append(data[i + num_prev])
    # NOTE: Keras expects input data in the shape (batch_size, truncated_seq_len, input_dim)
    # We have only one real-valued number per time-step, so we therefore expand
    # the last dimension from (batch_size, truncated_seq_len) to
    # (batch_size, truncated_seq_len, 1).
    X, Y = np.array(X)[:,:,np.newaxis], np.array(Y)[:,np.newaxis]
    return X, Y

# We only consider this many previous data points
truncated_seq_len = 2 #@param { type: "slider", min:1, max:10, step:1 }
test_split = 0.25  # Fraction of total data to keep out as test data

# We use only the sin(t) values, and discard the time values
data = sin_t_noisy
data_len = data.shape[0]
num_train = int(data_len * (1 - test_split))

train_data = data[:num_train]
test_data = data[num_train:]

X_train, y_train = pack_truncated_data(train_data, num_prev=truncated_seq_len)
X_test, y_test = pack_truncated_data(test_data, num_prev=truncated_seq_len)

print("Generated training/test data with shapes\nX_train: {}, y_train: {}\nX_test: {}, y_test: {}. ".format(
    X_train.shape, y_train.shape, X_test.shape, y_test.shape))

```

    Generated training/test data with shapes
    X_train: (2638, 2, 1), y_train: (2638, 1)
    X_test: (878, 2, 1), y_test: (878, 1).

**NOTE**: We reshape the training data into (batch_size, truncated_seq_len, 1) and (batch_size, 1) arrays.

### Intermediate Aside: (Truncated) Backpropagation-through-Time and Vanishing and Exploding Gradients

RNNs model sequential data, and are designed to capture how ***outputs*** at the current time step are influenced by the ***inputs*** that came before them. This is referred to as **long-range dependencies**. At a high level, this allows the model to remember what it has seen so far in order to better contextualize what it is seeing at the moment (think about how knowing the context of the sentence or conversation can sometimes help one to better figure out the intended meaning of a misheard word or ambiguous statement). It is what makes these models so powerful, but it is also what makes them so hard to train!

The most well-known algorithm for training RNNs is called **back-propagation through time (BPTT**; there are other algorithms). BPTT conceptually amounts to unrolling the computations of the RNN over time, computing the errors, and backpropagating the gradients through the unrolled graph structure.  Ideally we want to unroll the graph up to the maximum sequence length, however in practice, since sequence lengths vary and memory is limited, we only end up unrolling sequences up to some length $T$. This is called **truncated BPTT**, and is the most used variant of BPTT.

At a high level, there are two main issues when using (truncated) BPTT to train RNNs:

* Having shared ("tied") recurrent weights ($W_{hh}$) mean that **the gradient on these weights at some time step $t$ depends on all time steps up to time-step $T$**, the length of the full (truncated) sequence. This also leads to the **vanishing/exploding gradients** problem, which we'll explain below.

* **Memory usage grows linearly with the total number of steps $T$ that we unroll for**, because we need to save/cache the activations at each time-step (look at the Python code above to convince yourself of this).  This matters computationally, since memory is a limited resource. It also matters statistically, because it puts a limit on the types of dependency that the model can successfully learn, by preventing it from correcting errors that stem from an input more than $T$ steps ago.

**NOTE**: Think about that last statement and make sure you understand those 2 points.

BPTT is very similar to the standard back-propagation algorithm. Key to understanding the BPTT algorithm is to realize that gradients on the non-recurrent weights (weights of a per time-step classifier that tries to predict the part-of-speech tag for each word for example) and recurrent weights (that transform $h_{t-1}$ into $h_t$) are computed differently:

* The gradients of **non-recurrent weights** ($W_{hy}$) depend only on the error at that time-step, $E_t$.
* The gradients of **recurrent weights** ($W_{hh}$) depend on all previous time-steps up to maximum length $T$.

The first point is fairly intuitive: predictions at time-step $t$ is related to the loss of that particular prediction.

The second point will be explained in more detail in the lectures (see also [this great blog post](http://www.wildml.com/2015/10/recurrent-neural-networks-tutorial-part-3-backpropagation-through-time-and-vanishing-gradients/)), but briefly, this can be summarized in these equations:

1. The **current** state is a function of the **previous** state and the current input: $h_t = \sigma(W_{hh}h_{t-1} + W_{xh}x_t)$
2. The gradient of the loss $E_t$ at time $t$ on $W_{hh}$ is a function of the current hidden state and model predictions $\hat{y}_t$ at time t:
$\frac{\partial E_t}{\partial W_{hh}} = \frac{\partial E_t}{\partial \hat{y}_t}\frac{\partial\hat{y}_t}{\partial h_t}\frac{\partial h_t}{\partial W_{hh}}$
3. Substituting (1) into (2) results in a **sum over all previous time-steps**:
$\frac{\partial E_t}{\partial W_{hh}} = \sum\limits_{k=0}^{t} \underbrace{\frac{\partial E_t}{\partial \hat{y}_t}\frac{\partial\hat{y}_t}{\partial h_t}\frac{\partial h_t}{\partial h_k}\frac{\partial h_k}{\partial W_{hh}}}_\text{product of gradient terms}$

The problem is that $\frac{\partial h_t}{\partial h_k} = \Pi_j \frac{\partial h_j}{\partial h_{j-1}}$ for j from $k + 1$ to $t$.  Because of this **repeated multiplicative interaction**, as the sequence length $t$ gets longer, the gradients themselves can get diminishingly small (**vanish**) or grow too large and result in numeric overflow (**explode**). This has been shown to be related to the norms of the recurrent weight matrices being less than or equal to 1. Intuitively, it works very similar to how multiplying a small number $v<1.0$ with itself repeatedly can quickly go to zero, or conversely, a large number $v>1.0$ could quickly go to infinity; only this is for matrices.

###Build a tiny RNN in Keras

Building an RNN in Keras is quite simple. We simply chain the layers together as follows:

```
def define_model(truncated_seq_len):

    input_dimension = 1
    hidden_dimension = 1
    output_dimension = 1

    model = tf.keras.models.Sequential()
    model.add(tf.keras.layers.SimpleRNN(
        # We need to specify the input_shape *without* leading batch_size (it is inferred)
        input_shape=(truncated_seq_len, input_dimension),
        units=hidden_dimension,
        return_sequences=False,
        name='hidden_layer'))
    model.add(tf.keras.layers.Dense(
        output_dimension,
        name='output_layer'))

    model.compile(loss="mean_squared_error",
                  optimizer=tf.train.AdamOptimizer(learning_rate=1e-3))

    return model

```

**NOTE**: We're building an RNN for **regression**. We therefore use a linear layer (which outputs real-valued numbers) at the output with the "*mean_squared_error*" loss function.

```
model = define_model(truncated_seq_len = X_train.shape[1])
model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    hidden_layer (SimpleRNN)     (None, 1)                 3
    _________________________________________________________________
    output_layer (Dense)         (None, 1)                 2
    =================================================================
    Total params: 5
    Trainable params: 5
    Non-trainable params: 0
    _________________________________________________________________

**NOTE**: You need to re-run the above cell every time after training to reset the model weights!

###Train the tiny RNN
Now let's train the model. This may take a few minutes (it takes much longer if you increase `truncated_seq_len`). Set `verbose=1` **before** you run the cell to see the intermediate output as the model is training. Set it to 0 if you don't want any output.

```
''' SOLUTION TO ONE OF TASKS [DELETE]
patience = 5
train_history = model.fit(X_train, y_train, batch_size=600, epochs=1000,
                          verbose=1, validation_split=0.05,
                          callbacks=[tf.keras.callbacks.EarlyStopping(monitor='val_loss', patience=patience, verbose=1)])
'''
```

```
train_history = model.fit(X_train, y_train, batch_size=600, epochs=1000,
                          verbose=1, validation_split=0.05)
```

    Train on 2506 samples, validate on 132 samples
    Epoch 1/1000
    2506/2506 [==============================] - 0s 26us/step - loss: 0.4968 - val_loss: 0.4914
    Epoch 2/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4965 - val_loss: 0.4911
    Epoch 3/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4961 - val_loss: 0.4908
    Epoch 4/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4958 - val_loss: 0.4904
    Epoch 5/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4953 - val_loss: 0.4898
    Epoch 6/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.4949 - val_loss: 0.4893
    Epoch 7/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4944 - val_loss: 0.4887
    Epoch 8/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4939 - val_loss: 0.4883
    Epoch 9/1000
    2506/2506 [==============================] - 0s 13us/step - loss: 0.4933 - val_loss: 0.4877
    Epoch 10/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4927 - val_loss: 0.4872
    Epoch 11/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4920 - val_loss: 0.4865
    Epoch 12/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4913 - val_loss: 0.4856
    Epoch 13/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4905 - val_loss: 0.4847
    Epoch 14/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4896 - val_loss: 0.4838
    Epoch 15/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4887 - val_loss: 0.4828
    Epoch 16/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4876 - val_loss: 0.4819
    Epoch 17/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4865 - val_loss: 0.4808
    Epoch 18/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4853 - val_loss: 0.4797
    Epoch 19/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.4840 - val_loss: 0.4784
    Epoch 20/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.4826 - val_loss: 0.4771
    Epoch 21/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.4811 - val_loss: 0.4756
    Epoch 22/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.4794 - val_loss: 0.4741
    Epoch 23/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4777 - val_loss: 0.4725
    Epoch 24/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.4758 - val_loss: 0.4707
    Epoch 25/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4738 - val_loss: 0.4689
    Epoch 26/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4717 - val_loss: 0.4669
    Epoch 27/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4695 - val_loss: 0.4646
    Epoch 28/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4671 - val_loss: 0.4622
    Epoch 29/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4646 - val_loss: 0.4596
    Epoch 30/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4619 - val_loss: 0.4568
    Epoch 31/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.4590 - val_loss: 0.4540
    Epoch 32/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4560 - val_loss: 0.4510
    Epoch 33/1000
    2506/2506 [==============================] - 0s 13us/step - loss: 0.4529 - val_loss: 0.4479
    Epoch 34/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4496 - val_loss: 0.4446
    Epoch 35/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4461 - val_loss: 0.4412
    Epoch 36/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4425 - val_loss: 0.4376
    Epoch 37/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4388 - val_loss: 0.4337
    Epoch 38/1000
    2506/2506 [==============================] - 0s 13us/step - loss: 0.4349 - val_loss: 0.4297
    Epoch 39/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4308 - val_loss: 0.4256
    Epoch 40/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.4266 - val_loss: 0.4214
    Epoch 41/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4223 - val_loss: 0.4170
    Epoch 42/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4178 - val_loss: 0.4127
    Epoch 43/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.4133 - val_loss: 0.4082
    Epoch 44/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.4086 - val_loss: 0.4036
    Epoch 45/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.4038 - val_loss: 0.3988
    Epoch 46/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.3988 - val_loss: 0.3939
    Epoch 47/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.3938 - val_loss: 0.3889
    Epoch 48/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.3888 - val_loss: 0.3839
    Epoch 49/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.3836 - val_loss: 0.3789
    Epoch 50/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3785 - val_loss: 0.3738
    Epoch 51/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3732 - val_loss: 0.3686
    Epoch 52/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.3679 - val_loss: 0.3634
    Epoch 53/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.3625 - val_loss: 0.3581
    Epoch 54/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.3571 - val_loss: 0.3526
    Epoch 55/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3516 - val_loss: 0.3472
    Epoch 56/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3461 - val_loss: 0.3417
    Epoch 57/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.3406 - val_loss: 0.3362
    Epoch 58/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3350 - val_loss: 0.3306
    Epoch 59/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.3295 - val_loss: 0.3249
    Epoch 60/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3238 - val_loss: 0.3192
    Epoch 61/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.3182 - val_loss: 0.3136
    Epoch 62/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.3126 - val_loss: 0.3079
    Epoch 63/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.3069 - val_loss: 0.3022
    Epoch 64/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.3013 - val_loss: 0.2964
    Epoch 65/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.2956 - val_loss: 0.2908
    Epoch 66/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2900 - val_loss: 0.2851
    Epoch 67/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2844 - val_loss: 0.2794
    Epoch 68/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2787 - val_loss: 0.2737
    Epoch 69/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.2729 - val_loss: 0.2679
    Epoch 70/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.2673 - val_loss: 0.2621
    Epoch 71/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2615 - val_loss: 0.2563
    Epoch 72/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2557 - val_loss: 0.2505
    Epoch 73/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2500 - val_loss: 0.2446
    Epoch 74/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2442 - val_loss: 0.2388
    Epoch 75/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.2385 - val_loss: 0.2329
    Epoch 76/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2328 - val_loss: 0.2271
    Epoch 77/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.2270 - val_loss: 0.2213
    Epoch 78/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2212 - val_loss: 0.2155
    Epoch 79/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.2155 - val_loss: 0.2098
    Epoch 80/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2099 - val_loss: 0.2041
    Epoch 81/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.2042 - val_loss: 0.1984
    Epoch 82/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.1986 - val_loss: 0.1929
    Epoch 83/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1931 - val_loss: 0.1873
    Epoch 84/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.1876 - val_loss: 0.1819
    Epoch 85/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1822 - val_loss: 0.1764
    Epoch 86/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1768 - val_loss: 0.1710
    Epoch 87/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.1714 - val_loss: 0.1656
    Epoch 88/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.1661 - val_loss: 0.1605
    Epoch 89/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1610 - val_loss: 0.1554
    Epoch 90/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1559 - val_loss: 0.1505
    Epoch 91/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1509 - val_loss: 0.1456
    Epoch 92/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.1461 - val_loss: 0.1409
    Epoch 93/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.1414 - val_loss: 0.1363
    Epoch 94/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1368 - val_loss: 0.1317
    Epoch 95/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1323 - val_loss: 0.1273
    Epoch 96/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1279 - val_loss: 0.1231
    Epoch 97/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1237 - val_loss: 0.1190
    Epoch 98/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.1196 - val_loss: 0.1150
    Epoch 99/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.1156 - val_loss: 0.1111
    Epoch 100/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.1117 - val_loss: 0.1074
    Epoch 101/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.1080 - val_loss: 0.1039
    Epoch 102/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.1045 - val_loss: 0.1006
    Epoch 103/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.1011 - val_loss: 0.0973
    Epoch 104/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0978 - val_loss: 0.0941
    Epoch 105/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0946 - val_loss: 0.0911
    Epoch 106/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0916 - val_loss: 0.0882
    Epoch 107/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0887 - val_loss: 0.0854
    Epoch 108/1000
    2506/2506 [==============================] - 0s 14us/step - loss: 0.0859 - val_loss: 0.0828
    Epoch 109/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0833 - val_loss: 0.0803
    Epoch 110/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0808 - val_loss: 0.0779
    Epoch 111/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0783 - val_loss: 0.0757
    Epoch 112/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0761 - val_loss: 0.0736
    Epoch 113/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0739 - val_loss: 0.0716
    Epoch 114/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0719 - val_loss: 0.0697
    Epoch 115/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0699 - val_loss: 0.0679
    Epoch 116/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0681 - val_loss: 0.0663
    Epoch 117/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0663 - val_loss: 0.0646
    Epoch 118/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0647 - val_loss: 0.0631
    Epoch 119/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0631 - val_loss: 0.0616
    Epoch 120/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0617 - val_loss: 0.0603
    Epoch 121/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0603 - val_loss: 0.0591
    Epoch 122/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0590 - val_loss: 0.0579
    Epoch 123/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0578 - val_loss: 0.0567
    Epoch 124/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0566 - val_loss: 0.0557
    Epoch 125/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0556 - val_loss: 0.0547
    Epoch 126/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0545 - val_loss: 0.0538
    Epoch 127/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0536 - val_loss: 0.0530
    Epoch 128/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0527 - val_loss: 0.0522
    Epoch 129/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0519 - val_loss: 0.0515
    Epoch 130/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0511 - val_loss: 0.0507
    Epoch 131/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0504 - val_loss: 0.0500
    Epoch 132/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0497 - val_loss: 0.0494
    Epoch 133/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0490 - val_loss: 0.0488
    Epoch 134/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0484 - val_loss: 0.0482
    Epoch 135/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0478 - val_loss: 0.0477
    Epoch 136/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0472 - val_loss: 0.0472
    Epoch 137/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0467 - val_loss: 0.0467
    Epoch 138/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0462 - val_loss: 0.0463
    Epoch 139/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0457 - val_loss: 0.0459
    Epoch 140/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0453 - val_loss: 0.0455
    Epoch 141/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0448 - val_loss: 0.0451
    Epoch 142/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0444 - val_loss: 0.0448
    Epoch 143/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0440 - val_loss: 0.0444
    Epoch 144/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0437 - val_loss: 0.0441
    Epoch 145/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0433 - val_loss: 0.0438
    Epoch 146/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0430 - val_loss: 0.0435
    Epoch 147/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0427 - val_loss: 0.0431
    Epoch 148/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0424 - val_loss: 0.0428
    Epoch 149/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0421 - val_loss: 0.0426
    Epoch 150/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0418 - val_loss: 0.0423
    Epoch 151/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0415 - val_loss: 0.0421
    Epoch 152/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0412 - val_loss: 0.0419
    Epoch 153/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0410 - val_loss: 0.0416
    Epoch 154/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0407 - val_loss: 0.0414
    Epoch 155/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0405 - val_loss: 0.0411
    Epoch 156/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0402 - val_loss: 0.0409
    Epoch 157/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0400 - val_loss: 0.0407
    Epoch 158/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0398 - val_loss: 0.0405
    Epoch 159/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0395 - val_loss: 0.0403
    Epoch 160/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0393 - val_loss: 0.0401
    Epoch 161/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0391 - val_loss: 0.0399
    Epoch 162/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0389 - val_loss: 0.0397
    Epoch 163/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0387 - val_loss: 0.0395
    Epoch 164/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0385 - val_loss: 0.0393
    Epoch 165/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0382 - val_loss: 0.0391
    Epoch 166/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0380 - val_loss: 0.0389
    Epoch 167/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0378 - val_loss: 0.0387
    Epoch 168/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0376 - val_loss: 0.0385
    Epoch 169/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0374 - val_loss: 0.0383
    Epoch 170/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0372 - val_loss: 0.0381
    Epoch 171/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0370 - val_loss: 0.0379
    Epoch 172/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0368 - val_loss: 0.0377
    Epoch 173/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0366 - val_loss: 0.0375
    Epoch 174/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0364 - val_loss: 0.0373
    Epoch 175/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0362 - val_loss: 0.0372
    Epoch 176/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0360 - val_loss: 0.0370
    Epoch 177/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0358 - val_loss: 0.0368
    Epoch 178/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0356 - val_loss: 0.0366
    Epoch 179/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0354 - val_loss: 0.0364
    Epoch 180/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0352 - val_loss: 0.0362
    Epoch 181/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0351 - val_loss: 0.0360
    Epoch 182/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0349 - val_loss: 0.0359
    Epoch 183/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0347 - val_loss: 0.0357
    Epoch 184/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0345 - val_loss: 0.0355
    Epoch 185/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0343 - val_loss: 0.0353
    Epoch 186/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0341 - val_loss: 0.0351
    Epoch 187/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0339 - val_loss: 0.0349
    Epoch 188/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0337 - val_loss: 0.0347
    Epoch 189/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0335 - val_loss: 0.0345
    Epoch 190/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0333 - val_loss: 0.0343
    Epoch 191/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0331 - val_loss: 0.0342
    Epoch 192/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0329 - val_loss: 0.0340
    Epoch 193/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0327 - val_loss: 0.0338
    Epoch 194/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0325 - val_loss: 0.0336
    Epoch 195/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0323 - val_loss: 0.0334
    Epoch 196/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0321 - val_loss: 0.0332
    Epoch 197/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0320 - val_loss: 0.0330
    Epoch 198/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0318 - val_loss: 0.0328
    Epoch 199/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0316 - val_loss: 0.0326
    Epoch 200/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0314 - val_loss: 0.0324
    Epoch 201/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0312 - val_loss: 0.0322
    Epoch 202/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0310 - val_loss: 0.0321
    Epoch 203/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0308 - val_loss: 0.0319
    Epoch 204/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0306 - val_loss: 0.0317
    Epoch 205/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0304 - val_loss: 0.0315
    Epoch 206/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0302 - val_loss: 0.0314
    Epoch 207/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0301 - val_loss: 0.0312
    Epoch 208/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0299 - val_loss: 0.0310
    Epoch 209/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0297 - val_loss: 0.0308
    Epoch 210/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0295 - val_loss: 0.0306
    Epoch 211/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0293 - val_loss: 0.0304
    Epoch 212/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0291 - val_loss: 0.0302
    Epoch 213/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0289 - val_loss: 0.0300
    Epoch 214/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0287 - val_loss: 0.0298
    Epoch 215/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0286 - val_loss: 0.0296
    Epoch 216/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0284 - val_loss: 0.0295
    Epoch 217/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0282 - val_loss: 0.0293
    Epoch 218/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0280 - val_loss: 0.0291
    Epoch 219/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0278 - val_loss: 0.0289
    Epoch 220/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0276 - val_loss: 0.0288
    Epoch 221/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0275 - val_loss: 0.0286
    Epoch 222/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0273 - val_loss: 0.0285
    Epoch 223/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0271 - val_loss: 0.0283
    Epoch 224/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0269 - val_loss: 0.0281
    Epoch 225/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0268 - val_loss: 0.0279
    Epoch 226/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0266 - val_loss: 0.0277
    Epoch 227/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0264 - val_loss: 0.0275
    Epoch 228/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0262 - val_loss: 0.0274
    Epoch 229/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0260 - val_loss: 0.0272
    Epoch 230/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0259 - val_loss: 0.0270
    Epoch 231/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0257 - val_loss: 0.0268
    Epoch 232/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0255 - val_loss: 0.0267
    Epoch 233/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0253 - val_loss: 0.0265
    Epoch 234/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0252 - val_loss: 0.0263
    Epoch 235/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0250 - val_loss: 0.0262
    Epoch 236/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0248 - val_loss: 0.0260
    Epoch 237/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0247 - val_loss: 0.0258
    Epoch 238/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0245 - val_loss: 0.0257
    Epoch 239/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0243 - val_loss: 0.0255
    Epoch 240/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0242 - val_loss: 0.0253
    Epoch 241/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0240 - val_loss: 0.0252
    Epoch 242/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0238 - val_loss: 0.0250
    Epoch 243/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0237 - val_loss: 0.0249
    Epoch 244/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0235 - val_loss: 0.0247
    Epoch 245/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0233 - val_loss: 0.0246
    Epoch 246/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0232 - val_loss: 0.0244
    Epoch 247/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0230 - val_loss: 0.0242
    Epoch 248/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0229 - val_loss: 0.0241
    Epoch 249/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0227 - val_loss: 0.0239
    Epoch 250/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0226 - val_loss: 0.0238
    Epoch 251/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0224 - val_loss: 0.0236
    Epoch 252/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0222 - val_loss: 0.0235
    Epoch 253/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0221 - val_loss: 0.0233
    Epoch 254/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0219 - val_loss: 0.0232
    Epoch 255/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0218 - val_loss: 0.0230
    Epoch 256/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0216 - val_loss: 0.0229
    Epoch 257/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0215 - val_loss: 0.0227
    Epoch 258/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0213 - val_loss: 0.0226
    Epoch 259/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0212 - val_loss: 0.0224
    Epoch 260/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0210 - val_loss: 0.0223
    Epoch 261/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0209 - val_loss: 0.0222
    Epoch 262/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0208 - val_loss: 0.0220
    Epoch 263/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0206 - val_loss: 0.0219
    Epoch 264/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0205 - val_loss: 0.0217
    Epoch 265/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0203 - val_loss: 0.0216
    Epoch 266/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0202 - val_loss: 0.0215
    Epoch 267/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0201 - val_loss: 0.0213
    Epoch 268/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0199 - val_loss: 0.0212
    Epoch 269/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0198 - val_loss: 0.0211
    Epoch 270/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0197 - val_loss: 0.0210
    Epoch 271/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0195 - val_loss: 0.0208
    Epoch 272/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0194 - val_loss: 0.0207
    Epoch 273/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0193 - val_loss: 0.0206
    Epoch 274/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0191 - val_loss: 0.0205
    Epoch 275/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0190 - val_loss: 0.0203
    Epoch 276/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0189 - val_loss: 0.0202
    Epoch 277/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0188 - val_loss: 0.0201
    Epoch 278/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0186 - val_loss: 0.0200
    Epoch 279/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0185 - val_loss: 0.0198
    Epoch 280/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0184 - val_loss: 0.0197
    Epoch 281/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0183 - val_loss: 0.0196
    Epoch 282/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0182 - val_loss: 0.0195
    Epoch 283/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0180 - val_loss: 0.0194
    Epoch 284/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0179 - val_loss: 0.0193
    Epoch 285/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0178 - val_loss: 0.0192
    Epoch 286/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0177 - val_loss: 0.0191
    Epoch 287/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0176 - val_loss: 0.0190
    Epoch 288/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0175 - val_loss: 0.0188
    Epoch 289/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0174 - val_loss: 0.0187
    Epoch 290/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0173 - val_loss: 0.0186
    Epoch 291/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0172 - val_loss: 0.0185
    Epoch 292/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0171 - val_loss: 0.0185
    Epoch 293/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0170 - val_loss: 0.0184
    Epoch 294/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0169 - val_loss: 0.0183
    Epoch 295/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0168 - val_loss: 0.0182
    Epoch 296/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0167 - val_loss: 0.0181
    Epoch 297/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0166 - val_loss: 0.0180
    Epoch 298/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0165 - val_loss: 0.0179
    Epoch 299/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0164 - val_loss: 0.0178
    Epoch 300/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0163 - val_loss: 0.0177
    Epoch 301/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0162 - val_loss: 0.0176
    Epoch 302/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0161 - val_loss: 0.0175
    Epoch 303/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0160 - val_loss: 0.0174
    Epoch 304/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0159 - val_loss: 0.0173
    Epoch 305/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0158 - val_loss: 0.0173
    Epoch 306/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0157 - val_loss: 0.0172
    Epoch 307/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0156 - val_loss: 0.0171
    Epoch 308/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0156 - val_loss: 0.0170
    Epoch 309/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0155 - val_loss: 0.0169
    Epoch 310/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0154 - val_loss: 0.0169
    Epoch 311/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0153 - val_loss: 0.0168
    Epoch 312/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0152 - val_loss: 0.0167
    Epoch 313/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0152 - val_loss: 0.0166
    Epoch 314/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0151 - val_loss: 0.0166
    Epoch 315/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0150 - val_loss: 0.0165
    Epoch 316/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0149 - val_loss: 0.0164
    Epoch 317/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0149 - val_loss: 0.0164
    Epoch 318/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0148 - val_loss: 0.0163
    Epoch 319/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0147 - val_loss: 0.0162
    Epoch 320/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0146 - val_loss: 0.0161
    Epoch 321/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0146 - val_loss: 0.0161
    Epoch 322/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0145 - val_loss: 0.0160
    Epoch 323/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0144 - val_loss: 0.0159
    Epoch 324/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0144 - val_loss: 0.0159
    Epoch 325/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0143 - val_loss: 0.0158
    Epoch 326/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0143 - val_loss: 0.0158
    Epoch 327/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0142 - val_loss: 0.0157
    Epoch 328/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0141 - val_loss: 0.0157
    Epoch 329/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0141 - val_loss: 0.0156
    Epoch 330/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0140 - val_loss: 0.0156
    Epoch 331/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0140 - val_loss: 0.0155
    Epoch 332/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0139 - val_loss: 0.0154
    Epoch 333/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0138 - val_loss: 0.0154
    Epoch 334/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0138 - val_loss: 0.0153
    Epoch 335/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0137 - val_loss: 0.0153
    Epoch 336/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0137 - val_loss: 0.0152
    Epoch 337/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0136 - val_loss: 0.0152
    Epoch 338/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0136 - val_loss: 0.0151
    Epoch 339/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0135 - val_loss: 0.0151
    Epoch 340/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0135 - val_loss: 0.0150
    Epoch 341/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0134 - val_loss: 0.0150
    Epoch 342/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0134 - val_loss: 0.0149
    Epoch 343/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0133 - val_loss: 0.0149
    Epoch 344/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0133 - val_loss: 0.0149
    Epoch 345/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0132 - val_loss: 0.0148
    Epoch 346/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0132 - val_loss: 0.0148
    Epoch 347/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0132 - val_loss: 0.0147
    Epoch 348/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0131 - val_loss: 0.0147
    Epoch 349/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0131 - val_loss: 0.0147
    Epoch 350/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0130 - val_loss: 0.0146
    Epoch 351/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0130 - val_loss: 0.0146
    Epoch 352/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0130 - val_loss: 0.0146
    Epoch 353/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0129 - val_loss: 0.0145
    Epoch 354/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0129 - val_loss: 0.0145
    Epoch 355/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0129 - val_loss: 0.0144
    Epoch 356/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0128 - val_loss: 0.0144
    Epoch 357/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0128 - val_loss: 0.0144
    Epoch 358/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0127 - val_loss: 0.0143
    Epoch 359/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0127 - val_loss: 0.0143
    Epoch 360/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0127 - val_loss: 0.0143
    Epoch 361/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0126 - val_loss: 0.0143
    Epoch 362/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0126 - val_loss: 0.0142
    Epoch 363/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0126 - val_loss: 0.0142
    Epoch 364/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0126 - val_loss: 0.0142
    Epoch 365/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0125 - val_loss: 0.0142
    Epoch 366/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0125 - val_loss: 0.0141
    Epoch 367/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0125 - val_loss: 0.0141
    Epoch 368/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0125 - val_loss: 0.0141
    Epoch 369/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0124 - val_loss: 0.0141
    Epoch 370/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0124 - val_loss: 0.0140
    Epoch 371/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0124 - val_loss: 0.0140
    Epoch 372/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0124 - val_loss: 0.0140
    Epoch 373/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0123 - val_loss: 0.0140
    Epoch 374/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0123 - val_loss: 0.0140
    Epoch 375/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0123 - val_loss: 0.0139
    Epoch 376/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0123 - val_loss: 0.0139
    Epoch 377/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0122 - val_loss: 0.0139
    Epoch 378/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0122 - val_loss: 0.0139
    Epoch 379/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0122 - val_loss: 0.0138
    Epoch 380/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0122 - val_loss: 0.0138
    Epoch 381/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0122 - val_loss: 0.0138
    Epoch 382/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0121 - val_loss: 0.0138
    Epoch 383/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0121 - val_loss: 0.0138
    Epoch 384/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0121 - val_loss: 0.0138
    Epoch 385/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0121 - val_loss: 0.0137
    Epoch 386/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0121 - val_loss: 0.0137
    Epoch 387/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0121 - val_loss: 0.0137
    Epoch 388/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0120 - val_loss: 0.0137
    Epoch 389/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0120 - val_loss: 0.0137
    Epoch 390/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0120 - val_loss: 0.0137
    Epoch 391/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0120 - val_loss: 0.0137
    Epoch 392/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0120 - val_loss: 0.0137
    Epoch 393/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0120 - val_loss: 0.0136
    Epoch 394/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0120 - val_loss: 0.0136
    Epoch 395/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 396/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 397/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 398/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 399/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 400/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 401/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0119 - val_loss: 0.0136
    Epoch 402/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0119 - val_loss: 0.0135
    Epoch 403/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 404/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 405/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 406/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 407/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 408/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 409/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 410/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 411/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 412/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0118 - val_loss: 0.0135
    Epoch 413/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0118 - val_loss: 0.0134
    Epoch 414/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 415/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 416/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 417/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 418/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 419/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 420/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 421/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 422/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 423/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 424/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 425/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 426/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 427/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0117 - val_loss: 0.0134
    Epoch 428/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0116 - val_loss: 0.0134
    Epoch 429/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0134
    Epoch 430/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0134
    Epoch 431/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0134
    Epoch 432/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0134
    Epoch 433/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 434/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 435/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 436/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 437/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 438/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 439/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 440/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 441/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 442/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 443/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 444/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 445/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 446/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 447/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 448/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 449/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 450/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 451/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 452/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 453/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0116 - val_loss: 0.0133
    Epoch 454/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 455/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 456/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 457/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 458/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 459/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 460/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 461/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 462/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 463/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 464/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 465/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 466/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 467/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 468/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0133
    Epoch 469/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 470/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 471/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 472/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 473/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 474/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 475/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 476/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 477/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 478/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 479/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 480/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 481/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 482/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 483/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 484/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 485/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 486/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 487/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 488/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 489/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 490/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 491/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 492/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 493/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 494/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 495/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 496/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 497/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 498/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 499/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 500/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 501/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 502/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 503/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 504/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 505/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 506/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0115 - val_loss: 0.0132
    Epoch 507/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 508/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 509/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 510/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 511/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 512/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 513/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 514/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 515/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 516/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 517/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 518/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 519/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 520/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 521/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 522/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 523/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 524/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 525/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 526/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 527/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 528/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 529/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 530/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 531/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 532/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 533/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 534/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 535/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 536/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 537/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 538/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 539/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 540/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 541/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 542/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 543/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0114 - val_loss: 0.0132
    Epoch 544/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 545/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 546/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 547/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 548/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 549/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 550/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 551/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 552/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 553/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 554/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 555/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 556/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 557/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 558/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 559/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 560/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 561/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 562/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 563/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 564/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 565/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 566/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 567/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 568/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 569/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 570/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 571/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 572/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 573/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 574/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 575/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 576/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 577/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 578/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0114 - val_loss: 0.0131
    Epoch 579/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 580/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 581/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 582/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 583/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 584/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 585/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 586/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 587/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 588/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 589/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 590/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 591/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 592/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 593/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 594/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 595/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 596/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 597/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 598/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 599/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 600/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 601/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 602/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 603/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 604/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 605/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 606/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 607/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 608/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 609/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 610/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 611/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 612/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 613/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 614/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0131
    Epoch 615/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 616/1000
    2506/2506 [==============================] - 0s 21us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 617/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 618/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 619/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 620/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 621/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 622/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 623/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 624/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 625/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 626/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 627/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 628/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 629/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 630/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 631/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 632/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 633/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 634/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 635/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 636/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 637/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 638/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 639/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 640/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 641/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 642/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 643/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 644/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 645/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 646/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 647/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0113 - val_loss: 0.0130
    Epoch 648/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 649/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 650/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 651/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 652/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 653/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 654/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 655/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 656/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 657/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 658/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 659/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 660/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 661/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 662/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 663/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 664/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 665/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 666/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 667/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 668/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 669/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 670/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 671/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 672/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 673/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 674/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 675/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0130
    Epoch 676/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 677/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 678/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 679/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 680/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 681/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 682/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 683/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 684/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 685/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 686/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 687/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 688/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 689/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 690/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 691/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 692/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 693/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 694/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 695/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 696/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 697/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 698/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 699/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 700/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 701/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 702/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 703/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 704/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 705/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0112 - val_loss: 0.0129
    Epoch 706/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 707/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 708/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 709/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 710/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 711/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 712/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 713/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 714/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 715/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 716/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 717/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 718/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 719/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 720/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 721/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 722/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 723/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 724/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 725/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 726/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 727/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 728/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 729/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 730/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 731/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 732/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 733/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0129
    Epoch 734/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 735/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 736/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 737/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 738/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 739/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 740/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 741/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 742/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 743/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 744/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 745/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 746/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 747/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 748/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 749/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 750/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 751/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 752/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 753/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 754/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 755/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 756/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 757/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 758/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 759/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0111 - val_loss: 0.0128
    Epoch 760/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 761/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 762/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 763/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 764/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 765/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 766/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 767/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 768/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 769/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 770/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 771/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 772/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 773/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 774/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 775/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 776/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 777/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 778/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 779/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 780/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 781/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 782/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 783/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 784/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 785/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 786/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 787/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0110 - val_loss: 0.0128
    Epoch 788/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 789/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 790/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 791/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 792/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 793/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 794/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 795/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 796/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 797/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 798/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 799/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 800/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 801/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 802/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 803/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 804/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 805/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 806/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 807/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 808/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 809/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 810/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 811/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0110 - val_loss: 0.0127
    Epoch 812/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 813/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 814/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 815/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 816/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 817/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 818/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 819/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 820/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 821/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 822/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 823/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 824/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 825/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 826/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 827/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 828/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 829/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 830/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 831/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 832/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 833/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 834/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 835/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0127
    Epoch 836/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 837/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 838/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 839/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 840/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 841/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 842/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 843/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 844/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 845/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 846/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 847/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 848/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 849/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 850/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 851/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 852/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 853/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 854/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 855/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 856/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 857/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 858/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 859/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 860/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 861/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 862/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0109 - val_loss: 0.0126
    Epoch 863/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 864/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 865/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 866/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 867/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 868/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 869/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 870/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 871/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 872/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 873/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 874/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 875/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 876/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 877/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 878/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 879/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 880/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 881/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0126
    Epoch 882/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 883/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 884/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 885/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 886/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 887/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 888/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 889/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 890/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 891/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 892/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 893/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 894/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 895/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 896/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 897/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 898/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 899/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 900/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 901/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 902/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 903/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 904/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 905/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 906/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 907/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 908/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 909/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 910/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 911/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 912/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 913/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 914/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0108 - val_loss: 0.0125
    Epoch 915/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 916/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 917/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 918/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 919/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 920/1000
    2506/2506 [==============================] - 0s 20us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 921/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 922/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 923/1000
    2506/2506 [==============================] - 0s 23us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 924/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 925/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 926/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 927/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 928/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 929/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 930/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 931/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 932/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0125
    Epoch 933/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 934/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 935/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 936/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 937/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 938/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 939/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 940/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 941/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 942/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 943/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 944/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 945/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 946/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 947/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 948/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 949/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 950/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 951/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 952/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 953/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 954/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 955/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 956/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 957/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 958/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 959/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 960/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0107 - val_loss: 0.0124
    Epoch 961/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 962/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 963/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 964/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 965/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 966/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 967/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 968/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 969/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 970/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 971/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 972/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 973/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 974/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 975/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 976/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 977/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 978/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 979/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 980/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 981/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 982/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0106 - val_loss: 0.0124
    Epoch 983/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 984/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 985/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 986/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 987/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 988/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 989/1000
    2506/2506 [==============================] - 0s 17us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 990/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 991/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 992/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 993/1000
    2506/2506 [==============================] - 0s 18us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 994/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 995/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 996/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 997/1000
    2506/2506 [==============================] - 0s 15us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 998/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 999/1000
    2506/2506 [==============================] - 0s 19us/step - loss: 0.0106 - val_loss: 0.0123
    Epoch 1000/1000
    2506/2506 [==============================] - 0s 16us/step - loss: 0.0106 - val_loss: 0.0123

Let's visualize the training and validation losses.

```
plt.figure(figsize=(15,5))

for label in ["loss","val_loss"]:
    plt.plot(train_history.history[label], label=label)

plt.ylabel("loss")
plt.xlabel("epoch")
plt.title("The final validation loss: {}".format(train_history.history["val_loss"][-1]))
plt.legend()
plt.show()
```

Finally, let's look at the parameters for the trained model.

```
for layer in model.layers:
  print("{}, {}".format(layer.name, layer.get_weights()))
```

    hidden_layer, [array([[0.7633411]], dtype=float32), array([[-0.56165993]], dtype=float32), array([0.04831408], dtype=float32)]
    output_layer, [array([[2.4384913]], dtype=float32), array([-0.06346191], dtype=float32)]

**QUESTION**:
* Relate the above weights to the terms in the equation for the vanilla RNN we saw earlier, namely:
  * input-to-hidden $W_{xh}$,
  * hidden-to-hidden $W_{hh}$,
  * hidden-to-output weights $W_{hy}$
  * recurrent and out biases.

###Make predictions using the trained model

```
y_pred = model.predict(X_test[:100])
plt.figure(figsize=(19,3))

plt.plot(y_test[:100], label="true")
plt.plot(y_pred, label="predicted")
plt.legend()
plt.show()
```

**YOUR TASKS**:
* [**ALL**] Change the learning rate up to 5 orders of magnitude larger and smaller and retrain. What happens when it is too large? What happens when it is too small?
* [**ALL**] Change the `SimpleRNN` to `GRU`.
  * You can read more about LSTMs in [this blog post](https://colah.github.io/posts/2015-08-Understanding-LSTMs/) by Chris Olah.
  * What is the effect on the number of parameters? Can you explain why? Now do the same for `LSTM`.
* [**INTERMEDIATE**] Note that the loss does not decrease much after around epoch 400. Add "Early Stopping with patience" to the `model.fit()` function to stop it from training beyond this point. **Hint**: Look at tf.keras.callbacks.
  * *Early stopping* is a technique where we stop training the model once it's performance on validation data stops improving, and is a very common approach to prevent overfitting. Early stopping *with patience* means as soon as the model starts doing worse on validation we wait for at least `patience` more evaluations before stopping training, and if it improves within that time, we reset the counter. The patience parameter is a way to ensure we don't stop by accident, as the validation loss can fluctuate randomly from epoch to epoch.

##Generating Shakespeare

Now let's build an RNN language model to generate Shakespearian English! A language model is trained to assign high probabilities to sequences of words or sentences that are well formed, and low probabilities to sequences which are not realistic. When the model is trained, one can use it to *generate* data that is similar to the training data.

Our data is now sequences of discrete symbols (characters). But neural networks operate in continuous spaces, and so we need to take the discrete language data, and **embed** it in a continuous space. To do this, we'll simply break up the data into sequences of characters, and represent each character using a learned vector. This is a standard trick for processing text using neural networks.

### Download and Preprocess the Data

We first download the data and examine what it looks like:

```
context = ssl._create_unverified_context()
shakespeare_url = 'https://cs.stanford.edu/people/karpathy/char-rnn/shakespeare_input.txt'

data = urllib2.urlopen(shakespeare_url, context=context)
all_text = data.read().lower()

print("Downloaded Shakespeare data with {} characters.".format(len(all_text)))
print("FIRST 1000 CHARACTERS: ")
print(all_text[:1000])
```

    Downloaded Shakespeare data with 4573338 characters.
    FIRST 1000 CHARACTERS:
    first citizen:
    before we proceed any further, hear me speak.

    all:
    speak, speak.

    first citizen:
    you are all resolved rather to die than to famish?

    all:
    resolved. resolved.

    first citizen:
    first, you know caius marcius is chief enemy to the people.

    all:
    we know't, we know't.

    first citizen:
    let us kill him, and we'll have corn at our own price.
    is't a verdict?

    all:
    no more talking on't; let it be done: away, away!

    second citizen:
    one word, good citizens.

    first citizen:
    we are accounted poor citizens, the patricians good.
    what authority surfeits on would relieve us: if they
    would yield us but the superfluity, while it were
    wholesome, we might guess they relieved us humanely;
    but they think we are too dear: the leanness that
    afflicts us, the object of our misery, is as an
    inventory to particularise their abundance; our
    sufferance is a gain to them let us revenge this with
    our pikes, ere we become rakes: for the gods know i
    speak this in hunger for bread, not in thirst for revenge.

```
training_text = all_text[:1000000] # Keep only the first 1 million characters
```

We now preprocess the text data as follows:

1. Extract the vocabulary of all `vocab_size` unique characters appearing in the data.

2. Assign each character a unique integer id in `0 <= id < vocab_size`. This is so we can map the characters to unique embedding vectors. This is a common way to map discrete inputs to continuous vectors that neural networks can work with. See e.g. this [blog post](https://www.tensorflow.org/tutorials/representation/word2vec), or [this one](http://ruder.io/word-embeddings-1/) for more information.

3. Split the data into sequences ("windows") of `max_len` characters (the input to the model) followed by the next character as target. E.g. using `max_len=5` the sentence "I saw a cat" (11 characters) will get split into "I saw" and /space/, "/space/saw/space/" and "a", "saw a" and /space/, etc. To add some variation, we skip `step` characters between each sequence (i.e. we use a "sliding window of `max_len` with stride `step`").

```
max_len = 30  # We only consider this many previous data points (characters)
step = 3 # We start a new training sequence every `step` characters
sentences = [] # This holds our extracted sequences
next_chars = [] # This holds the targets (the follow-up characters)

chars = sorted(list(set(training_text)))  # List of unique characters in the corpus
vocab_size = len(chars)
print('Number of unique characters: ', vocab_size)
print(chars)

# Construct dictionaries mapping unique characters to their index in `chars` and reverse
char2index = dict((c, chars.index(c)) for c in chars)
index2char = dict((chars.index(c), c) for c in chars)
```

    Number of unique characters:  39
    ['\n', ' ', '!', '$', '&', "'", ',', '-', '.', '3', ':', ';', '?', 'a', 'b', 'c', 'd', 'e', 'f', 'g', 'h', 'i', 'j', 'k', 'l', 'm', 'n', 'o', 'p', 'q', 'r', 's', 't', 'u', 'v', 'w', 'x', 'y', 'z']

Now we encode the training data by mapping each character to its unique integer id.

```
for i in range(0, len(training_text) - max_len, step):
    sentences.append([char2index[s] for s in training_text[i: i + max_len]])
    next_chars.append([char2index[s] for s in training_text[i + max_len]])

print('Number of extracted sequences:', len(sentences))
```

    Number of extracted sequences: 333324

This yields the following numpy arrays:

```
X, Y = np.array(sentences, dtype=np.int64), np.array(next_chars, dtype=np.int64)
X.shape, Y.shape
```

    ((333324, 30), (333324, 1))

Let's take a look at the first example.

```
print("X[0].shape = {}, Y[0].shape = {}".format(X[0].shape, Y[0].shape))
print("X[0]: ", X[0])
print("Y[0]: ", Y[0])
```

    X[0].shape = (30,), Y[0].shape = (1,)
    X[0]:  [18 21 30 31 32  1 15 21 32 21 38 17 26 10  0 14 17 18 27 30 17  1 35 17
      1 28 30 27 15 17]
    Y[0]:  [17]

###Build an RNN language model

A **language model** estimates a probability distribution over sequences $\mathbb{x}_{1:N} = (x_1, x_2, ..., x_N)$ by breaking up the full joint probability into a sequence of conditional probabilities using the **[chain-rule of probability](https://en.wikipedia.org/wiki/Chain_rule_(probability))**:

\begin{align}
 p(\mathbb{x}_{1:N}) &= p(x_1) \cdot p(x_2 | x_1) \cdot p(x_3 | x_2, x_1) \ldots \\
 &= \Pi_1^N p(x_i | \mathbb{x}_{1:i-1})
\end{align}

In other words, to model the probability of the phrase "*i saw a cat*" at the character level, the model learns to estimate the probabilities for p(i), p(/space/| i), p(c | i, /space/), and so forth, and multiplies them together.

There are many different ways in which to estimate these individual probabilities. But one particularly effective way is to use an RNN! To do this, we'll therefore be modeling the  $p(x_i | \mathbb{x}_{1:i-1})$ terms using an RNN conditioned on $\mathbb{x}_{1:i-1}$.

* We model these probabilities at the character-level, so we'll use an `Embedding` layer as the first layer of our model to map the discrete character id's to real-valued embedding vectors.
* Next, the RNN-core will map these sequences of character embeddings to a probability distribution over all characters $p(x_i | \mathbb{x}_{1:i-1}) \in \mathbb{R}^\textrm{vocab_size}$ at every step of the sequence. To do this, the RNN will map the embeddings to a sequence of *hidden states*. We will then use a `Dense` layer to map from the RNN hidden state to an output distribution over the total number of characters using a [`softmax`](https://en.wikipedia.org/wiki/Softmax_function) activation.

We can do this with a few lines of code:

```
embedding_dim = 32   # Map each character to a unique vector of this dimension
vocab_size = len(chars)

model = tf.keras.models.Sequential()
model.add(tf.keras.layers.Embedding(
  vocab_size, embedding_dim,
  input_length=max_len,
  embeddings_initializer=tf.keras.initializers.TruncatedNormal))
model.add(tf.keras.layers.LSTM(
  128,
  input_shape=(max_len, embedding_dim),  # NB: Ensure this matches the embedding_dim!
  dropout=0.1,  # input-to-hidden drop-probability
  recurrent_dropout=0.2))  # hidden-to-hidden drop-probability
model.add(tf.keras.layers.Dense(vocab_size, activation='softmax'))

model.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    embedding_2 (Embedding)      (None, 30, 32)            1248
    _________________________________________________________________
    lstm_2 (LSTM)                (None, 128)               82432
    _________________________________________________________________
    dense_2 (Dense)              (None, 39)                5031
    =================================================================
    Total params: 88,711
    Trainable params: 88,711
    Non-trainable params: 0
    _________________________________________________________________

###Select the optimizer and loss

Once we have a model that can map sequence of characters to a probability distribution over the next character in the sequence, we can train it using **maximum likelihood** on the training set to find the model parameters which maximizes the probability of the training data. Again, this is very simple to do by choosing an optimizer and selecting the `sparse_categorical_crossentropy` loss function:

```
optimizer = tf.train.AdamOptimizer(learning_rate=0.001)
loss='sparse_categorical_crossentropy'

model.compile(loss=loss, optimizer=optimizer)
```

###Helper functions

```
def sample_with_temp(preds, temperature=1.0):
  preds = np.asarray(preds).astype('float64')
  preds = np.log(preds) / temperature
  exp_preds = np.exp(preds)
  preds = exp_preds / np.sum(exp_preds)
  probas = np.random.multinomial(1, preds, 1)
  return np.argmax(probas)
```

```
def shift_and_append(test_arr, next_item):
  '''Returns a copy of test_arr with items shifted one position to the left and
     next_item appended.
  '''
  tmp = np.empty_like(test_arr)
  tmp[:,:-1] = test_arr[:,1:]
  tmp[:,-1] = next_item
  return tmp

## TEST the above function:
test_arr = np.array([[1,2,3,4]])

print("test_arr = {}".format(test_arr))
test_arr = shift_and_append(test_arr, 5)
print("roll_arr(test_arr, 5) = {}".format(test_arr))
```

    test_arr = [[1 2 3 4]]
    roll_arr(test_arr, 5) = [[2 3 4 5]]

```
def sample_from_model(model,
                      num_generate=400,
                      prev_text=None,  # the text used to condition the model
                      temperatures=[0.2, 0.5, 1.0, 1.2]):

    if not prev_text:
      # Select a text seed at random
      start_index = random.randint(0, len(training_text) - max_len - 1)
      while ((start_index < (len(training_text) - max_len - 1)) and (
        training_text[start_index - 1] is not ' ')):
        start_index += 1  # Advance to beginning of new word
      prev_text = training_text[start_index: start_index + max_len]

    if len(prev_text) != max_len:
      print("`prev_text` must be of length `max_len`.")
      return

    print('GENERATING TEXT WITH SEED: \n"' + prev_text + '"')
    prev_text_arr = np.array(
      [[char2index[c] for c in prev_text]], dtype=np.int64)

    for temp in temperatures:
        print('==TEMPERATURE:', temp)
        sys.stdout.write(prev_text)

        # Start with the same sampled text for all temperatures
        generated_text = prev_text
        generated_text_arr = prev_text_arr

        # Now generate this many characters
        for i in range(num_generate):

            # Get the output softmax given the conditioning text
            #prev_text = generated_text_enc[np.newaxis,:]
            preds = model.predict(generated_text_arr, verbose=0)[0]

            next_index = sample_with_temp(preds, temp)
            next_char = index2char[next_index]
            generated_text += next_char
            generated_text = generated_text[1:]

            # Left-shift and add into encoded array
            generated_text_arr = shift_and_append(generated_text_arr, next_index)

            sys.stdout.write(next_char)
            sys.stdout.flush()
        print()
```

###Train the model

Let's train the model! The code below will train the model on a subset of the available data, and then generate from the model every `sample_every` number of batches.

To generate from the model, we use `model.predict()` on a sequence of `max_len` conditioning characters to produce an output distribution over `vocab_size` characters. We then sample one character from this distribution and shift everything up by one and append the new characters. By repeating this, we can generate text from the (partially-trained) model.

**NOTE**:
* It takes a while to train a model that starts generating anything resembling the Shapespeare text! In general it should start getting the rough structure in place around the 100K training example mark (examples, not batches). But to generate any meaningful words will need several hundred thousand examples.
* We sample with *temperature*. This is a way to sharpen or flatten the probabilities produced by the model. By lowering the temperature, we emphasize the modes of the predicted distribution, and by increasing the temperature, we flatten the modes (tends towards uniform). Higher temperatures therefore encourage the model to be more 'creative', instead of always choosing the most likely next character.

```
batch_size = 128
total_num_batches = X.shape[0] // batch_size
sample_every = 256  # Train on this many batches, then generate something

print("Training on {} batches in total.".format(total_num_batches))

for cur_batch in range(0, total_num_batches, sample_every):
    print('TRAINING ON BATCH {} to {} (example {} to {})'.format(
      cur_batch, cur_batch + sample_every,
      cur_batch * batch_size, (cur_batch + sample_every) * batch_size)
    )

    X_batch = X[batch_size * cur_batch : batch_size * (cur_batch + sample_every), :]
    Y_batch = Y[batch_size * cur_batch : batch_size * (cur_batch + sample_every), :]

    '''
    # Show the first 5 examples to make sure we're not training on garbage
    print("X_batch.shape = {}".format(X_batch.shape))
    print("Y_batch.shape = {}".format(Y_batch.shape))
    print("FIRST 5 EXAMPLES:")
    for num in range(5):
      in_seq = [index2char[int(indx)] for indx in np.nditer(X_batch[num, :])]
      next_char = index2char[Y_batch[num, 0]]
      print(str(num) + '. ' + ''.join(in_seq) + '-->' + next_char)
    '''

    model.fit(X_batch, Y_batch,
              batch_size=batch_size,
              epochs=1,
              verbose=1)

    print("GENERATING SOME RANDOM TEXT FROM THE MODEL")
    sample_from_model(model)
```

**NOTE**: Even after training has stopped you can still generate from the (partially trained) model as follows:

```
my_text = "       the meaning of life is:"  # Needs to be max_len characters
print(len(my_text))
sample_from_model(model, prev_text=my_text)
```

    30

###IMPORTANT NOTES
* Even if you stop training the model weights are persistant. If you resume training it will start where you left off.
* To reset the weights, you need to recompile the model.
* Sampling is **stochastic** (random), so you'll get new outputs every time you rerun the sampling code.

### YOUR TASKS:
* [**ALL**] Read the generations from your model in a funny voice to your neighbour.
* [**ALL**] Increase `max_len` and regenerate the data and retrain the model.
 * What's the effect on training speed as you double `max_len`. Can you explain why?
 * Do you notice any effect on the quality of the model? Can you explain why?
* [**ALL**] Set the max_len to be roughly the length of half a word; one word; two words... What kind of samples do these models generate? Explain how they go wrong.
* [**ALL**] Change `embedding_dim` and the hidden size of the LSTM and observe the effect on training speed and quality.
* [**INTERMEDIATE**] Change the dropout rates & retrain the model.
  * What types of dropout do we get for recurrent models?
  * What's the effect on the text quality?
* [**ADVANCED**] Implement the "teacher forcing" training methodology, where the net must predict the entire output sequence shifted forward by one character, instead of just the next character. Compare the output of a model trained with teacher forcing versus the per-character model, given a similar training time. Is it fair to say that teacher forcing is a more efficient training methodology?

##Further reading

* https://distill.pub/2016/augmented-rnns/
* https://distill.pub/2017/ctc/
* https://algotravelling.com/en/machine-learning-fun-part-5/