# Deep Learning

- 01 What is "Deep Learning?"
- 02 Why Deep Learning?
- 03 The Perceptron (Neural units)
- 04 Shallow Neural Network
- 05 Activation functions
- 06 Loss functions
- 07 Cross entropy

## What is "Deep Learning?"

[Deep learning](https://en.wikipedia.org/wiki/Deep_learning) is the stacking of artificial neural networks (ANNs) to create stacked neural networks, [deep belief networks](https://en.wikipedia.org/wiki/Deep_belief_network), [recurrent neural networks](https://en.wikipedia.org/wiki/Recurrent_neural_network) and deep generative models. A deep neural network (DNN) is an ANN with multiple hidden layers between the input and output layers.

An ANN is based on a collection of connected units called artificial neurons, (analogous to axons in a biological brain). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving (postsynaptic) neuron can process the signal(s) and then signal downstream neurons connected to it.

_ Deep learning is basically the deep stacking of artificial neurons to learn complex models of data. _

## Why Deep Learning?

- It works

Deep learning and neural networks are increasingly important concepts as demonstrated through their performance on difficult problems in computer vision medical diagnosis, natural language processing and many other domains.

- Learns feature selection

Deep learning algorithms are unique in that they try to learn latent features from data, as opposed to traditional machine learning where features selection is typically handcrafted.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import warnings
import random
from datetime import datetime
random.seed(datetime.now())
warnings.filterwarnings('ignore')

# Make plots larger
plt.rcParams['figure.figsize'] = (10, 6)
```

```python
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
```

## MNIST data

The [MNIST database](http://yann.lecun.com/exdb/mnist/) of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.

It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.

```python
# load the MNIST dataset
(X_train, y_train), (X_test, y_test) = mnist.load_data()
```

```python
# plot first 4 images as gray scale
plt.subplot(221)
plt.imshow(X_train[0], cmap=plt.get_cmap('gray'))
plt.subplot(222)
plt.imshow(X_train[1], cmap=plt.get_cmap('gray'))
plt.subplot(223)
plt.imshow(X_train[2], cmap=plt.get_cmap('gray'))
plt.subplot(224)
plt.imshow(X_train[3], cmap=plt.get_cmap('gray'))
# show the plot
plt.show()
```

The training dataset is structured as a 3-dimensional array of each image, and that images width and image height (28×28 pixels per image).

```python
## 60K 28×28 sized training images
print (X_train.shape)
```

    (60000, 28, 28)

```python
# plot 4 more images as gray scale
plt.subplot(221)
plt.imshow(X_train[55], cmap=plt.get_cmap('gray'))
plt.subplot(222)
plt.imshow(X_train[555], cmap=plt.get_cmap('gray'))
plt.subplot(223)
plt.imshow(X_train[5555], cmap=plt.get_cmap('gray'))
plt.subplot(224)
plt.imshow(X_train[55555], cmap=plt.get_cmap('gray'))
# show the plot
plt.show()
```

## The Perceptron (Neural units)

The [perceptron]()https://en.wikipedia.org/wiki/Perceptron is an algorithm for supervised learning of binary classifiers (functions that can decide
whether an input, represented by a vector of numbers, belongs to some
specific class or not). It is a type of linear classifier, i.e. a
classification algorithm that makes its predictions based on a linear
predictor function combining a set of weights with the feature
vector.

The perceptron algorithm dates back to the late 1950s, and is the basis of [artificial neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network).

Definition
----------

In the modern sense, the perceptron is an algorithm for learning a
binary classifier: a function that maps its input (a real-valued
[vector]) to an output value $f(x)$ (a single [binary] value):

$$f(x) = \begin{cases}1 & \text{if }\ w \cdot x + b > 0\\0 & \text{otherwise}\end{cases}$$

where is a vector of real-valued weights, $w \cdot x$ is the [dot
product] $\sum_{i=1}^m w_i x_i$, where m is the number of inputs to the
perceptron and is the *bias*. The bias shifts the decision boundary away
from the origin and does not depend on any input value.

The value of $f(x)$ (0 or 1) is used to classify as either a positive or
a negative instance, in the case of a binary classification problem. If
is negative, then the weighted combination of inputs must produce a
positive value greater than $|b|$ in order to push the classifier neuron
over the 0 threshold. Spatially, the bias alters the position (though
not the orientation) of the decision boundary. The perceptron learning
algorithm does not terminate if the learning set is not linearly
separable. If the vectors are not linearly separable learning will
never reach a point where all vectors are classified properly. The most
famous example of the perceptron's inability to solve problems with
linearly nonseparable vectors is the Boolean exclusive-or problem. The
solution spaces of decision boundaries for all binary functions and
learning behaviors are studied in the reference.

In the context of neural networks, a perceptron is an _artificial
neuron_ using the [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function) as the activation function.
The perceptron algorithm is also termed the **single-layer perceptron**,
to distinguish it from a multilayer perceptron, which is a misnomer
for a more complicated neural network. As a linear classifier, the
single-layer perceptron is the simplest [feedforward neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network).

Learning algorithm
------------------

Below is an example of a learning algorithm for a (single-layer)
perceptron. For [multilayer perceptrons], where a hidden layer exists,
more sophisticated algorithms such as [backpropagation] must be used.
Alternatively, methods such as the [delta rule] can be used if the
function is non-linear and differentiable, although the one below will
work as well.

When multiple perceptrons are combined in an artificial neural network,
each output neuron operates independently of all the others; thus,
learning each output can be considered in isolation.

### Definitions

We first define some variables:

-   $y = f(\mathbf{z})$ denotes the *output* from the perceptron for an
    input vector $\mathbf{z}$.
-   $D = \{(\mathbf{x}_1,d_1),\dots,(\mathbf{x}_s,d_s)\}$ is the
    *training set* of $s$ samples, where:
    -   $\mathbf{x}_j$ is the $n$-dimensional input vector.
    -   $d_j$ is the desired output value of the perceptron for that
        input.

We show the values of the features as follows:

-   $x_{j,i}$ is the value of the $i$th feature of the $j$th training
    *input vector*.
-   $x_{j,0} = 1$.

To represent the weights:

-   $w_i$ is the $i$th value in the *weight vector*, to be multiplied by
    the value of the $i$th input feature.
-   Because $x_{j,0} = 1$, the $w_0$ is effectively a bias that we use
    instead of the bias constant $b$.

To show the time-dependence of $\mathbf{w}$, we use:

-   $w_i(t)$ is the weight $i$ at time $t$.

Unlike other linear classification algorithms such as [logistic
regression], there is no need for a *learning rate* in the perceptron
algorithm. This is because multiplying the update by any constant simply
rescales the weights but never changes the sign of the prediction.

### Steps

1. Initialize the weights and the threshold. Weights may
be initialized to 0 or to a small random value. In the example below, we
use 0.
2. For each example in our training set , perform the following
steps over the input $\mathbf{x}_j$ and desired output $d_j$

Calculate the actual output:

a. $$\begin{align}
y_j(t) &= f[\mathbf{w}(t)\cdot\mathbf{x}_j] \\
&= f[w_0(t)x_{j,0} + w_1(t)x_{j,1} + w_2(t)x_{j,2} + \dotsb + w_n(t)x_{j,n}]
\end{align}$$

b. Update the weights:

$$w_i(t+1) = w_i(t) + (d_j - y_j(t)) x_{j,i}$$, for all features
$0 \leq i \leq n$.

**offline learning**

For offline learning, the step 2 may be repeated until the iteration error  $\frac{1}{s} \sum_{j=1}^s |d_j - y_j(t)|$ is less than a user-specified error threshold $\gamma$, or a predetermined number of iterations have been completed. The algorithm updates the weights after steps 2a and 2b. These weights are immediately applied to a pair in the training set, and subsequently updated, rather than waiting until all pairs in the training set have undergone these steps.

Multiclass perceptron
---------------------

Like most other techniques for training linear classifiers, the
perceptron generalizes naturally to multiclass classification. Here,
the input $x$ and the output $y$ are drawn from arbitrary sets. A
feature representation function $f(x,y)$ maps each possible input/output
pair to a finite-dimensional real-valued feature vector. As before, the
feature vector is multiplied by a weight vector $w$, but now the
resulting score is used to choose among many possible outputs:

$$\hat y = \operatorname{argmax}_y f(x,y) \cdot w.$$ ≈ Learning again
iterates over the examples, predicting an output for each, leaving the
weights unchanged when the predicted output matches the target, and
changing them when it does not. The update becomes:

$$w_{t+1} = w_t + f(x, y) - f(x,\hat y).$$

This multiclass feedback formulation reduces to the original perceptron
when $x$ is a real-valued vector, $y$ is chosen from $\{0,1\}$, and
$f(x,y) = y x$.

For certain problems, input/output representations and features can be
chosen so that $\mathrm{argmax}_y f(x,y) \cdot w$ can be found
efficiently even though $y$ is chosen from a very large or even infinite
set.

In recent years, perceptron training has become popular in the field of
[natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing) for such tasks as [part-of-speech tagging](https://en.wikipedia.org/wiki/Part-of-speech_tagging)
and [syntactic parsing](https://en.wikipedia.org/wiki/Parsing).

## Artificial neural network

[Artificial neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network) (**ANNs**) or **[connectionist] systems**
are computing systems inspired by the biological neural networks that
constitute animal brains. Such systems learn (progressively improve
performance) to do tasks by considering examples, generally without
task-specific programming. For example, in image recognition, they might
learn to identify images that contain cats by analyzing example images
that have been manually labeled as “cat” or “no cat” and using the
analytic results to identify cats in other images. They have found most
use in applications difficult to express in a traditional computer
algorithm using rule-based programming.

An ANN is based on a collection of connected units called artificial
neurons, (analogous to axons in a biological brain). Each
connection synapse) between neurons can transmit a signal to another
neuron. The receiving (postsynaptic) neuron can process the signal(s)
and then signal downstream neurons connected to it. Neurons may have
state, generally represented by [real numbers], typically between 0 and 1.

Neurons and synapses may also have a weight that varies as learning
proceeds, which can increase or decrease the strength of the signal that
it sends downstream. Further, they may have a threshold such that only
if the aggregate signal is below (or above) that level is the downstream
signal sent.

Typically, neurons are organized in layers. Different layers may perform
different kinds of transformations on their inputs. Signals travel from
the first (input), to the last (output) layer, possibly after traversing
the layers multiple times.

The original goal of the neural network approach was to solve problems
in the same way that a human brain would. Over time, attention focused
on matching specific mental abilities, leading to deviations from
biology such as backpropagation, or passing information in the reverse
direction and adjusting the network to reflect that information.

Neural networks have been used on a variety of tasks, including
computer vision, speech recognition, machine translation, social
network filtering, playing board and video games, medical diagnosis and
in many other domains.

## How many input neurons?

What is the input layer for the MNIST data?  Each image is $28x28 = 784$.

```python
## 10K 28×28 sized test images
X_test.shape
```

    (10000, 28, 28)

```python
28*28
```

    784

For a multi-layer perceptron model we must reduce the images down into a a single input layer as a vector of pixels. In this case the 28×28 sized images will be 784 pixel input values (28x28=784).

```python
X_train = X_train.reshape(60000, 784).astype('float32')
X_test = X_test.reshape(10000, 784).astype('float32')
```

The pixel values are gray scale between 0 and 255. Scaling of input values when using neural network models is a good idea. Neural network models propagate values and the rate of propagation can be effected by their scale.

```python
# Scale the data between 0 and 1

X_train /= 255.0
X_test /= 255.0
```

## Shallow Neural Network

A shallow neural network has few layers (just one dense layer in this case). Dense means every neuron connected to every other.

```python
X_test[0]
```

    array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.32941177,  0.72549021,  0.62352943,
            0.59215689,  0.23529412,  0.14117648,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.87058824,  0.99607843,  0.99607843,  0.99607843,  0.99607843,
            0.94509804,  0.7764706 ,  0.7764706 ,  0.7764706 ,  0.7764706 ,
            0.7764706 ,  0.7764706 ,  0.7764706 ,  0.7764706 ,  0.66666669,
            0.20392157,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.26274511,  0.44705883,
            0.28235295,  0.44705883,  0.63921571,  0.89019608,  0.99607843,
            0.88235295,  0.99607843,  0.99607843,  0.99607843,  0.98039216,
            0.89803922,  0.99607843,  0.99607843,  0.54901963,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.06666667,  0.25882354,  0.05490196,  0.26274511,
            0.26274511,  0.26274511,  0.23137255,  0.08235294,  0.9254902 ,
            0.99607843,  0.41568628,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.32549021,  0.99215686,  0.81960785,  0.07058824,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.08627451,  0.9137255 ,
            1.        ,  0.32549021,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.50588238,  0.99607843,  0.93333334,  0.17254902,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.23137255,  0.97647059,
            0.99607843,  0.24313726,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.52156866,  0.99607843,  0.73333335,  0.01960784,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.03529412,  0.80392158,
            0.97254902,  0.22745098,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.49411765,  0.99607843,  0.71372551,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.29411766,  0.98431373,
            0.94117647,  0.22352941,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.07450981,  0.86666667,  0.99607843,  0.65098041,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.01176471,  0.79607844,  0.99607843,
            0.85882354,  0.13725491,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.14901961,  0.99607843,  0.99607843,  0.3019608 ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.12156863,  0.87843138,  0.99607843,
            0.4509804 ,  0.00392157,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.52156866,  0.99607843,  0.99607843,  0.20392157,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.23921569,  0.94901961,  0.99607843,
            0.99607843,  0.20392157,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.47450981,  0.99607843,  0.99607843,  0.85882354,  0.15686275,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.47450981,  0.99607843,
            0.81176472,  0.07058824,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ], dtype=float32)

Finally, the output variable is an integer from 0 to 9. This is a multi-class classification problem. As such, it is good practice to use a one hot encoding of the class values, transforming the vector of class integers into a binary matrix.

We can easily do this using the built-in np_utils.to_categorical() helper function in Keras.

```python
n_classes = 10
y_train = keras.utils.to_categorical(y_train, n_classes)
y_test = keras.utils.to_categorical(y_test, n_classes)
```

```python
y_test[0:5]
```

    array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.]])

```python
y_train[0:5]
```

    array([[ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]])

```python
def shallow_net_A(n=55,i=784,o=10):
    # create simple one dense layer net
    # default 55 neurons, input 784, output 10
    net = Sequential()
    net.add(Dense(n, activation='sigmoid', input_shape=(i,)))
    net.add(Dense(10, activation='softmax'))
    # Compile net
    net.compile(loss='mean_squared_error', optimizer=SGD(lr=0.01), metrics=['accuracy'])
    return net
```

```python
nn=shallow_net_A()
```

```python
nn.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_7 (Dense)              (None, 55)                43175
    _________________________________________________________________
    dense_8 (Dense)              (None, 10)                560
    =================================================================
    Total params: 43,735
    Trainable params: 43,735
    Non-trainable params: 0
    _________________________________________________________________

```python
nn.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0911 - acc: 0.0969 - val_loss: 0.0908 - val_acc: 0.0973
    Epoch 2/99
    60000/60000 [==============================] - 0s - loss: 0.0906 - acc: 0.1062 - val_loss: 0.0904 - val_acc: 0.1102
    Epoch 3/99
    60000/60000 [==============================] - 0s - loss: 0.0902 - acc: 0.1206 - val_loss: 0.0901 - val_acc: 0.1321
    Epoch 4/99
    60000/60000 [==============================] - 0s - loss: 0.0899 - acc: 0.1409 - val_loss: 0.0898 - val_acc: 0.1559
    Epoch 5/99
    60000/60000 [==============================] - 0s - loss: 0.0896 - acc: 0.1677 - val_loss: 0.0895 - val_acc: 0.1873
    Epoch 6/99
    60000/60000 [==============================] - 0s - loss: 0.0893 - acc: 0.2019 - val_loss: 0.0892 - val_acc: 0.2171
    Epoch 7/99
    60000/60000 [==============================] - 0s - loss: 0.0891 - acc: 0.2343 - val_loss: 0.0889 - val_acc: 0.2516
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.0888 - acc: 0.2677 - val_loss: 0.0887 - val_acc: 0.2765
    Epoch 9/99
    60000/60000 [==============================] - 0s - loss: 0.0885 - acc: 0.2934 - val_loss: 0.0884 - val_acc: 0.3039
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.0883 - acc: 0.3146 - val_loss: 0.0881 - val_acc: 0.3254
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.0880 - acc: 0.3370 - val_loss: 0.0879 - val_acc: 0.3487
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.0877 - acc: 0.3617 - val_loss: 0.0876 - val_acc: 0.3744
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.0875 - acc: 0.3848 - val_loss: 0.0873 - val_acc: 0.3985
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.0872 - acc: 0.4067 - val_loss: 0.0871 - val_acc: 0.4189
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0869 - acc: 0.4255 - val_loss: 0.0868 - val_acc: 0.4343
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.0866 - acc: 0.4410 - val_loss: 0.0865 - val_acc: 0.4477
    Epoch 17/99
    60000/60000 [==============================] - 0s - loss: 0.0863 - acc: 0.4544 - val_loss: 0.0862 - val_acc: 0.4596
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0861 - acc: 0.4626 - val_loss: 0.0859 - val_acc: 0.4659
    Epoch 19/99
    60000/60000 [==============================] - 0s - loss: 0.0858 - acc: 0.4702 - val_loss: 0.0856 - val_acc: 0.4720
    Epoch 20/99
    60000/60000 [==============================] - 0s - loss: 0.0854 - acc: 0.4761 - val_loss: 0.0853 - val_acc: 0.4771
    Epoch 21/99
    60000/60000 [==============================] - 0s - loss: 0.0851 - acc: 0.4800 - val_loss: 0.0849 - val_acc: 0.4830
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.0848 - acc: 0.4839 - val_loss: 0.0846 - val_acc: 0.4869
    Epoch 23/99
    60000/60000 [==============================] - 0s - loss: 0.0845 - acc: 0.4857 - val_loss: 0.0843 - val_acc: 0.4886
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.0841 - acc: 0.4887 - val_loss: 0.0839 - val_acc: 0.4917
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.0838 - acc: 0.4900 - val_loss: 0.0836 - val_acc: 0.4944
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.0834 - acc: 0.4920 - val_loss: 0.0832 - val_acc: 0.4959
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.0830 - acc: 0.4930 - val_loss: 0.0828 - val_acc: 0.4969
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.0827 - acc: 0.4944 - val_loss: 0.0824 - val_acc: 0.4979
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.0823 - acc: 0.4952 - val_loss: 0.0820 - val_acc: 0.5001
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.0819 - acc: 0.4960 - val_loss: 0.0816 - val_acc: 0.5011
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.0815 - acc: 0.4972 - val_loss: 0.0812 - val_acc: 0.5014
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.0810 - acc: 0.4980 - val_loss: 0.0808 - val_acc: 0.5013
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.0806 - acc: 0.4987 - val_loss: 0.0803 - val_acc: 0.5032
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.0802 - acc: 0.5002 - val_loss: 0.0799 - val_acc: 0.5044
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.0797 - acc: 0.5014 - val_loss: 0.0794 - val_acc: 0.5056
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.0793 - acc: 0.5029 - val_loss: 0.0790 - val_acc: 0.5083
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.0788 - acc: 0.5038 - val_loss: 0.0785 - val_acc: 0.5093
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.0783 - acc: 0.5057 - val_loss: 0.0780 - val_acc: 0.5112
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.0778 - acc: 0.5075 - val_loss: 0.0775 - val_acc: 0.5126
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.0773 - acc: 0.5084 - val_loss: 0.0770 - val_acc: 0.5150
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.0768 - acc: 0.5103 - val_loss: 0.0765 - val_acc: 0.5167
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.0763 - acc: 0.5117 - val_loss: 0.0760 - val_acc: 0.5187
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.0758 - acc: 0.5141 - val_loss: 0.0755 - val_acc: 0.5212
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0753 - acc: 0.5163 - val_loss: 0.0749 - val_acc: 0.5229
    Epoch 45/99
    60000/60000 [==============================] - 0s - loss: 0.0748 - acc: 0.5181 - val_loss: 0.0744 - val_acc: 0.5254
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.0743 - acc: 0.5206 - val_loss: 0.0739 - val_acc: 0.5288
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.0737 - acc: 0.5232 - val_loss: 0.0734 - val_acc: 0.5306
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.0732 - acc: 0.5261 - val_loss: 0.0728 - val_acc: 0.5337
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.0727 - acc: 0.5285 - val_loss: 0.0723 - val_acc: 0.5359
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0722 - acc: 0.5305 - val_loss: 0.0718 - val_acc: 0.5385
    Epoch 51/99
    60000/60000 [==============================] - 0s - loss: 0.0716 - acc: 0.5333 - val_loss: 0.0712 - val_acc: 0.5421
    Epoch 52/99
    60000/60000 [==============================] - 0s - loss: 0.0711 - acc: 0.5358 - val_loss: 0.0707 - val_acc: 0.5447
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0706 - acc: 0.5399 - val_loss: 0.0702 - val_acc: 0.5479
    Epoch 54/99
    60000/60000 [==============================] - 0s - loss: 0.0701 - acc: 0.5432 - val_loss: 0.0697 - val_acc: 0.5520
    Epoch 55/99
    60000/60000 [==============================] - 0s - loss: 0.0695 - acc: 0.5463 - val_loss: 0.0691 - val_acc: 0.5543
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.0690 - acc: 0.5491 - val_loss: 0.0686 - val_acc: 0.5578
    Epoch 57/99
    60000/60000 [==============================] - 0s - loss: 0.0685 - acc: 0.5530 - val_loss: 0.0681 - val_acc: 0.5604
    Epoch 58/99
    60000/60000 [==============================] - 0s - loss: 0.0680 - acc: 0.5562 - val_loss: 0.0676 - val_acc: 0.5646
    Epoch 59/99
    60000/60000 [==============================] - 0s - loss: 0.0675 - acc: 0.5597 - val_loss: 0.0671 - val_acc: 0.5672
    Epoch 60/99
    60000/60000 [==============================] - 0s - loss: 0.0670 - acc: 0.5635 - val_loss: 0.0666 - val_acc: 0.5709
    Epoch 61/99
    60000/60000 [==============================] - 0s - loss: 0.0665 - acc: 0.5668 - val_loss: 0.0661 - val_acc: 0.5751
    Epoch 62/99
    60000/60000 [==============================] - 0s - loss: 0.0660 - acc: 0.5702 - val_loss: 0.0656 - val_acc: 0.5777
    Epoch 63/99
    60000/60000 [==============================] - 0s - loss: 0.0656 - acc: 0.5741 - val_loss: 0.0651 - val_acc: 0.5816
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0651 - acc: 0.5774 - val_loss: 0.0647 - val_acc: 0.5839
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0646 - acc: 0.5810 - val_loss: 0.0642 - val_acc: 0.5876
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0642 - acc: 0.5842 - val_loss: 0.0637 - val_acc: 0.5897
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0637 - acc: 0.5877 - val_loss: 0.0633 - val_acc: 0.5926
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0632 - acc: 0.5911 - val_loss: 0.0628 - val_acc: 0.5953
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0628 - acc: 0.5940 - val_loss: 0.0624 - val_acc: 0.5974
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0624 - acc: 0.5972 - val_loss: 0.0619 - val_acc: 0.5995
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0619 - acc: 0.6008 - val_loss: 0.0615 - val_acc: 0.6014
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0615 - acc: 0.6036 - val_loss: 0.0611 - val_acc: 0.6045
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0611 - acc: 0.6067 - val_loss: 0.0606 - val_acc: 0.6072
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0607 - acc: 0.6093 - val_loss: 0.0602 - val_acc: 0.6093
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0602 - acc: 0.6121 - val_loss: 0.0598 - val_acc: 0.6118
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0598 - acc: 0.6147 - val_loss: 0.0594 - val_acc: 0.6139
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0594 - acc: 0.6171 - val_loss: 0.0590 - val_acc: 0.6173
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0590 - acc: 0.6194 - val_loss: 0.0586 - val_acc: 0.6197
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0586 - acc: 0.6219 - val_loss: 0.0582 - val_acc: 0.6222
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0583 - acc: 0.6243 - val_loss: 0.0578 - val_acc: 0.6242
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0579 - acc: 0.6263 - val_loss: 0.0574 - val_acc: 0.6262
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0575 - acc: 0.6289 - val_loss: 0.0570 - val_acc: 0.6286
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0571 - acc: 0.6312 - val_loss: 0.0567 - val_acc: 0.6303
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0567 - acc: 0.6332 - val_loss: 0.0563 - val_acc: 0.6327
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0564 - acc: 0.6354 - val_loss: 0.0559 - val_acc: 0.6353
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0560 - acc: 0.6373 - val_loss: 0.0556 - val_acc: 0.6375
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0557 - acc: 0.6396 - val_loss: 0.0552 - val_acc: 0.6390
    Epoch 88/99
    60000/60000 [==============================] - 0s - loss: 0.0553 - acc: 0.6417 - val_loss: 0.0548 - val_acc: 0.6412
    Epoch 89/99
    60000/60000 [==============================] - 0s - loss: 0.0550 - acc: 0.6438 - val_loss: 0.0545 - val_acc: 0.6426
    Epoch 90/99
    60000/60000 [==============================] - 0s - loss: 0.0546 - acc: 0.6461 - val_loss: 0.0541 - val_acc: 0.6452
    Epoch 91/99
    60000/60000 [==============================] - 0s - loss: 0.0543 - acc: 0.6487 - val_loss: 0.0538 - val_acc: 0.6470
    Epoch 92/99
    60000/60000 [==============================] - 0s - loss: 0.0539 - acc: 0.6509 - val_loss: 0.0535 - val_acc: 0.6491
    Epoch 93/99
    60000/60000 [==============================] - 0s - loss: 0.0536 - acc: 0.6530 - val_loss: 0.0531 - val_acc: 0.6515
    Epoch 94/99
    60000/60000 [==============================] - 0s - loss: 0.0533 - acc: 0.6551 - val_loss: 0.0528 - val_acc: 0.6538
    Epoch 95/99
    60000/60000 [==============================] - 0s - loss: 0.0529 - acc: 0.6571 - val_loss: 0.0525 - val_acc: 0.6563
    Epoch 96/99
    60000/60000 [==============================] - 0s - loss: 0.0526 - acc: 0.6589 - val_loss: 0.0521 - val_acc: 0.6591
    Epoch 97/99
    60000/60000 [==============================] - 0s - loss: 0.0523 - acc: 0.6614 - val_loss: 0.0518 - val_acc: 0.6612
    Epoch 98/99
    60000/60000 [==============================] - 0s - loss: 0.0520 - acc: 0.6635 - val_loss: 0.0515 - val_acc: 0.6638
    Epoch 99/99
    60000/60000 [==============================] - 0s - loss: 0.0516 - acc: 0.6660 - val_loss: 0.0512 - val_acc: 0.6667

    <keras.callbacks.History at 0x12da5efd0>

```python
# 68% accuracy after 99 epochs
nn.evaluate(X_test, y_test)
```

     8160/10000 [=======================>......] - ETA: 0s

    [0.051163210290670398, 0.66669999999999996]

```python
nn.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0513 - acc: 0.6683 - val_loss: 0.0508 - val_acc: 0.6690
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0510 - acc: 0.6703 - val_loss: 0.0505 - val_acc: 0.6722
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0507 - acc: 0.6728 - val_loss: 0.0502 - val_acc: 0.6749
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0504 - acc: 0.6752 - val_loss: 0.0499 - val_acc: 0.6779
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0501 - acc: 0.6776 - val_loss: 0.0496 - val_acc: 0.6803
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0498 - acc: 0.6804 - val_loss: 0.0493 - val_acc: 0.6824
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0495 - acc: 0.6833 - val_loss: 0.0490 - val_acc: 0.6853
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.0492 - acc: 0.6859 - val_loss: 0.0487 - val_acc: 0.6879
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0489 - acc: 0.6879 - val_loss: 0.0484 - val_acc: 0.6905
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.0486 - acc: 0.6903 - val_loss: 0.0481 - val_acc: 0.6932
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.0484 - acc: 0.6923 - val_loss: 0.0479 - val_acc: 0.6959
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.0481 - acc: 0.6947 - val_loss: 0.0476 - val_acc: 0.6991
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.0478 - acc: 0.6970 - val_loss: 0.0473 - val_acc: 0.7015
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.0475 - acc: 0.6993 - val_loss: 0.0470 - val_acc: 0.7037
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0472 - acc: 0.7015 - val_loss: 0.0467 - val_acc: 0.7060
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.0470 - acc: 0.7035 - val_loss: 0.0464 - val_acc: 0.7084
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0467 - acc: 0.7059 - val_loss: 0.0462 - val_acc: 0.7106
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0464 - acc: 0.7081 - val_loss: 0.0459 - val_acc: 0.7122
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0461 - acc: 0.7102 - val_loss: 0.0456 - val_acc: 0.7145
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0459 - acc: 0.7119 - val_loss: 0.0454 - val_acc: 0.7162
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0456 - acc: 0.7143 - val_loss: 0.0451 - val_acc: 0.7185
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.0453 - acc: 0.7165 - val_loss: 0.0448 - val_acc: 0.7207
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0451 - acc: 0.7182 - val_loss: 0.0446 - val_acc: 0.7229
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.0448 - acc: 0.7206 - val_loss: 0.0443 - val_acc: 0.7250
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0446 - acc: 0.7231 - val_loss: 0.0440 - val_acc: 0.7275
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0443 - acc: 0.7252 - val_loss: 0.0438 - val_acc: 0.7294
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0441 - acc: 0.7276 - val_loss: 0.0435 - val_acc: 0.7312
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0438 - acc: 0.7301 - val_loss: 0.0433 - val_acc: 0.7332
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0436 - acc: 0.7323 - val_loss: 0.0430 - val_acc: 0.7356
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0433 - acc: 0.7346 - val_loss: 0.0428 - val_acc: 0.7383
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0431 - acc: 0.7369 - val_loss: 0.0425 - val_acc: 0.7403
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0428 - acc: 0.7388 - val_loss: 0.0423 - val_acc: 0.7432
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0426 - acc: 0.7420 - val_loss: 0.0420 - val_acc: 0.7458
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.0423 - acc: 0.7443 - val_loss: 0.0418 - val_acc: 0.7493
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0421 - acc: 0.7469 - val_loss: 0.0416 - val_acc: 0.7524
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0419 - acc: 0.7497 - val_loss: 0.0413 - val_acc: 0.7550
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.0416 - acc: 0.7524 - val_loss: 0.0411 - val_acc: 0.7574
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.0414 - acc: 0.7556 - val_loss: 0.0408 - val_acc: 0.7602
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0412 - acc: 0.7584 - val_loss: 0.0406 - val_acc: 0.7633
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0409 - acc: 0.7611 - val_loss: 0.0404 - val_acc: 0.7665
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0407 - acc: 0.7641 - val_loss: 0.0401 - val_acc: 0.7701
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0405 - acc: 0.7664 - val_loss: 0.0399 - val_acc: 0.7733
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0403 - acc: 0.7689 - val_loss: 0.0397 - val_acc: 0.7759
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0400 - acc: 0.7722 - val_loss: 0.0395 - val_acc: 0.7796
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0398 - acc: 0.7749 - val_loss: 0.0392 - val_acc: 0.7818
    Epoch 46/99
    60000/60000 [==============================] - 1s - loss: 0.0396 - acc: 0.7776 - val_loss: 0.0390 - val_acc: 0.7849
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0394 - acc: 0.7809 - val_loss: 0.0388 - val_acc: 0.7878
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0392 - acc: 0.7837 - val_loss: 0.0386 - val_acc: 0.7909
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.0390 - acc: 0.7868 - val_loss: 0.0384 - val_acc: 0.7941
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0387 - acc: 0.7898 - val_loss: 0.0381 - val_acc: 0.7972
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0385 - acc: 0.7928 - val_loss: 0.0379 - val_acc: 0.7996
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0383 - acc: 0.7952 - val_loss: 0.0377 - val_acc: 0.8025
    Epoch 53/99
    60000/60000 [==============================] - 0s - loss: 0.0381 - acc: 0.7975 - val_loss: 0.0375 - val_acc: 0.8053
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0379 - acc: 0.8002 - val_loss: 0.0373 - val_acc: 0.8075
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0377 - acc: 0.8030 - val_loss: 0.0371 - val_acc: 0.8095
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.0375 - acc: 0.8056 - val_loss: 0.0369 - val_acc: 0.8119
    Epoch 57/99
    60000/60000 [==============================] - 0s - loss: 0.0373 - acc: 0.8077 - val_loss: 0.0367 - val_acc: 0.8140
    Epoch 58/99
    60000/60000 [==============================] - 0s - loss: 0.0371 - acc: 0.8097 - val_loss: 0.0365 - val_acc: 0.8160
    Epoch 59/99
    60000/60000 [==============================] - 0s - loss: 0.0369 - acc: 0.8119 - val_loss: 0.0363 - val_acc: 0.8183
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0367 - acc: 0.8135 - val_loss: 0.0361 - val_acc: 0.8199
    Epoch 61/99
    60000/60000 [==============================] - 0s - loss: 0.0365 - acc: 0.8153 - val_loss: 0.0359 - val_acc: 0.8216
    Epoch 62/99
    60000/60000 [==============================] - 0s - loss: 0.0363 - acc: 0.8174 - val_loss: 0.0357 - val_acc: 0.8239
    Epoch 63/99
    60000/60000 [==============================] - 0s - loss: 0.0362 - acc: 0.8195 - val_loss: 0.0355 - val_acc: 0.8255
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0360 - acc: 0.8212 - val_loss: 0.0353 - val_acc: 0.8270
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0358 - acc: 0.8226 - val_loss: 0.0352 - val_acc: 0.8281
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0356 - acc: 0.8239 - val_loss: 0.0350 - val_acc: 0.8305
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0354 - acc: 0.8255 - val_loss: 0.0348 - val_acc: 0.8327
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0352 - acc: 0.8266 - val_loss: 0.0346 - val_acc: 0.8345
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0351 - acc: 0.8283 - val_loss: 0.0344 - val_acc: 0.8359
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0349 - acc: 0.8299 - val_loss: 0.0342 - val_acc: 0.8372
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0347 - acc: 0.8310 - val_loss: 0.0341 - val_acc: 0.8388
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0346 - acc: 0.8324 - val_loss: 0.0339 - val_acc: 0.8398
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0344 - acc: 0.8334 - val_loss: 0.0337 - val_acc: 0.8411
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0342 - acc: 0.8347 - val_loss: 0.0335 - val_acc: 0.8422
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0340 - acc: 0.8357 - val_loss: 0.0334 - val_acc: 0.8435
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0339 - acc: 0.8368 - val_loss: 0.0332 - val_acc: 0.8454
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0337 - acc: 0.8379 - val_loss: 0.0330 - val_acc: 0.8465
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0336 - acc: 0.8387 - val_loss: 0.0329 - val_acc: 0.8476
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0334 - acc: 0.8393 - val_loss: 0.0327 - val_acc: 0.8494
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0332 - acc: 0.8401 - val_loss: 0.0325 - val_acc: 0.8500
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0331 - acc: 0.8411 - val_loss: 0.0324 - val_acc: 0.8506
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0329 - acc: 0.8422 - val_loss: 0.0322 - val_acc: 0.8509
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0328 - acc: 0.8429 - val_loss: 0.0321 - val_acc: 0.8524
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0326 - acc: 0.8438 - val_loss: 0.0319 - val_acc: 0.8531
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0325 - acc: 0.8446 - val_loss: 0.0318 - val_acc: 0.8536
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0323 - acc: 0.8454 - val_loss: 0.0316 - val_acc: 0.8542
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0322 - acc: 0.8460 - val_loss: 0.0315 - val_acc: 0.8545
    Epoch 88/99
    60000/60000 [==============================] - 0s - loss: 0.0320 - acc: 0.8467 - val_loss: 0.0313 - val_acc: 0.8555
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.0319 - acc: 0.8474 - val_loss: 0.0312 - val_acc: 0.8563
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0317 - acc: 0.8479 - val_loss: 0.0310 - val_acc: 0.8570
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.0316 - acc: 0.8485 - val_loss: 0.0309 - val_acc: 0.8571
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0315 - acc: 0.8492 - val_loss: 0.0307 - val_acc: 0.8575
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0313 - acc: 0.8495 - val_loss: 0.0306 - val_acc: 0.8582
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0312 - acc: 0.8501 - val_loss: 0.0305 - val_acc: 0.8587
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0310 - acc: 0.8504 - val_loss: 0.0303 - val_acc: 0.8593
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0309 - acc: 0.8510 - val_loss: 0.0302 - val_acc: 0.8597
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0308 - acc: 0.8517 - val_loss: 0.0300 - val_acc: 0.8603
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0307 - acc: 0.8521 - val_loss: 0.0299 - val_acc: 0.8608
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0305 - acc: 0.8526 - val_loss: 0.0298 - val_acc: 0.8609

    <keras.callbacks.History at 0x12d552a58>

```python
# 86% accuracy after another 99 epochs
nn.evaluate(X_test, y_test)
```

     9408/10000 [===========================>..] - ETA: 0s

    [0.029782633084058763, 0.8609]

```python
nn.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0304 - acc: 0.8531 - val_loss: 0.0297 - val_acc: 0.8613
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0303 - acc: 0.8536 - val_loss: 0.0295 - val_acc: 0.8616
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0301 - acc: 0.8541 - val_loss: 0.0294 - val_acc: 0.8620
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0300 - acc: 0.8545 - val_loss: 0.0293 - val_acc: 0.8626
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0299 - acc: 0.8549 - val_loss: 0.0291 - val_acc: 0.8632
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0298 - acc: 0.8551 - val_loss: 0.0290 - val_acc: 0.8638
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0297 - acc: 0.8557 - val_loss: 0.0289 - val_acc: 0.8643
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.0295 - acc: 0.8561 - val_loss: 0.0288 - val_acc: 0.8647
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0294 - acc: 0.8565 - val_loss: 0.0287 - val_acc: 0.8651
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.0293 - acc: 0.8568 - val_loss: 0.0285 - val_acc: 0.8654
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.0292 - acc: 0.8572 - val_loss: 0.0284 - val_acc: 0.8658
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.0291 - acc: 0.8575 - val_loss: 0.0283 - val_acc: 0.8664
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.0290 - acc: 0.8579 - val_loss: 0.0282 - val_acc: 0.8670
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.0289 - acc: 0.8581 - val_loss: 0.0281 - val_acc: 0.8679
    Epoch 15/99
    60000/60000 [==============================] - 0s - loss: 0.0287 - acc: 0.8586 - val_loss: 0.0280 - val_acc: 0.8687
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.0286 - acc: 0.8590 - val_loss: 0.0279 - val_acc: 0.8688
    Epoch 17/99
    60000/60000 [==============================] - 0s - loss: 0.0285 - acc: 0.8594 - val_loss: 0.0277 - val_acc: 0.8689
    Epoch 18/99
    60000/60000 [==============================] - 0s - loss: 0.0284 - acc: 0.8595 - val_loss: 0.0276 - val_acc: 0.8692
    Epoch 19/99
    60000/60000 [==============================] - 0s - loss: 0.0283 - acc: 0.8599 - val_loss: 0.0275 - val_acc: 0.8691
    Epoch 20/99
    60000/60000 [==============================] - 0s - loss: 0.0282 - acc: 0.8602 - val_loss: 0.0274 - val_acc: 0.8693
    Epoch 21/99
    60000/60000 [==============================] - 0s - loss: 0.0281 - acc: 0.8605 - val_loss: 0.0273 - val_acc: 0.8696
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.0280 - acc: 0.8609 - val_loss: 0.0272 - val_acc: 0.8700
    Epoch 23/99
    60000/60000 [==============================] - 0s - loss: 0.0279 - acc: 0.8613 - val_loss: 0.0271 - val_acc: 0.8702
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.0278 - acc: 0.8617 - val_loss: 0.0270 - val_acc: 0.8707
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.0277 - acc: 0.8621 - val_loss: 0.0269 - val_acc: 0.8713
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.0276 - acc: 0.8623 - val_loss: 0.0268 - val_acc: 0.8719
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.0275 - acc: 0.8626 - val_loss: 0.0267 - val_acc: 0.8719
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.0274 - acc: 0.8626 - val_loss: 0.0266 - val_acc: 0.8723
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.0273 - acc: 0.8628 - val_loss: 0.0265 - val_acc: 0.8724
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.0272 - acc: 0.8634 - val_loss: 0.0264 - val_acc: 0.8723
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.0271 - acc: 0.8635 - val_loss: 0.0263 - val_acc: 0.8724
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.0270 - acc: 0.8637 - val_loss: 0.0262 - val_acc: 0.8729
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.0270 - acc: 0.8640 - val_loss: 0.0261 - val_acc: 0.8730
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.0269 - acc: 0.8645 - val_loss: 0.0260 - val_acc: 0.8732
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.0268 - acc: 0.8646 - val_loss: 0.0260 - val_acc: 0.8736
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.0267 - acc: 0.8649 - val_loss: 0.0259 - val_acc: 0.8739
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.0266 - acc: 0.8651 - val_loss: 0.0258 - val_acc: 0.8743
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.0265 - acc: 0.8654 - val_loss: 0.0257 - val_acc: 0.8744
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.0264 - acc: 0.8656 - val_loss: 0.0256 - val_acc: 0.8749
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.0263 - acc: 0.8659 - val_loss: 0.0255 - val_acc: 0.8749
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.0263 - acc: 0.8660 - val_loss: 0.0254 - val_acc: 0.8753
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.0262 - acc: 0.8665 - val_loss: 0.0253 - val_acc: 0.8756
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.0261 - acc: 0.8664 - val_loss: 0.0253 - val_acc: 0.8761
    Epoch 44/99
    60000/60000 [==============================] - 0s - loss: 0.0260 - acc: 0.8668 - val_loss: 0.0252 - val_acc: 0.8763
    Epoch 45/99
    60000/60000 [==============================] - 0s - loss: 0.0259 - acc: 0.8671 - val_loss: 0.0251 - val_acc: 0.8768
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.0259 - acc: 0.8673 - val_loss: 0.0250 - val_acc: 0.8770
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.0258 - acc: 0.8677 - val_loss: 0.0249 - val_acc: 0.8773
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.0257 - acc: 0.8678 - val_loss: 0.0249 - val_acc: 0.8774
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.0256 - acc: 0.8680 - val_loss: 0.0248 - val_acc: 0.8774
    Epoch 50/99
    60000/60000 [==============================] - 0s - loss: 0.0255 - acc: 0.8682 - val_loss: 0.0247 - val_acc: 0.8777
    Epoch 51/99
    60000/60000 [==============================] - 0s - loss: 0.0255 - acc: 0.8685 - val_loss: 0.0246 - val_acc: 0.8780
    Epoch 52/99
    60000/60000 [==============================] - 0s - loss: 0.0254 - acc: 0.8688 - val_loss: 0.0246 - val_acc: 0.8784
    Epoch 53/99
    60000/60000 [==============================] - 0s - loss: 0.0253 - acc: 0.8691 - val_loss: 0.0245 - val_acc: 0.8787
    Epoch 54/99
    60000/60000 [==============================] - 0s - loss: 0.0252 - acc: 0.8694 - val_loss: 0.0244 - val_acc: 0.8789
    Epoch 55/99
    60000/60000 [==============================] - 0s - loss: 0.0252 - acc: 0.8697 - val_loss: 0.0243 - val_acc: 0.8789
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.0251 - acc: 0.8698 - val_loss: 0.0243 - val_acc: 0.8789
    Epoch 57/99
    60000/60000 [==============================] - 0s - loss: 0.0250 - acc: 0.8702 - val_loss: 0.0242 - val_acc: 0.8796
    Epoch 58/99
    60000/60000 [==============================] - 0s - loss: 0.0250 - acc: 0.8704 - val_loss: 0.0241 - val_acc: 0.8796
    Epoch 59/99
    60000/60000 [==============================] - 0s - loss: 0.0249 - acc: 0.8706 - val_loss: 0.0240 - val_acc: 0.8796
    Epoch 60/99
    60000/60000 [==============================] - 0s - loss: 0.0248 - acc: 0.8708 - val_loss: 0.0240 - val_acc: 0.8798
    Epoch 61/99
    60000/60000 [==============================] - 0s - loss: 0.0248 - acc: 0.8709 - val_loss: 0.0239 - val_acc: 0.8799
    Epoch 62/99
    60000/60000 [==============================] - 0s - loss: 0.0247 - acc: 0.8712 - val_loss: 0.0238 - val_acc: 0.8801
    Epoch 63/99
    60000/60000 [==============================] - 0s - loss: 0.0246 - acc: 0.8714 - val_loss: 0.0238 - val_acc: 0.8803
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0246 - acc: 0.8715 - val_loss: 0.0237 - val_acc: 0.8807
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0245 - acc: 0.8718 - val_loss: 0.0236 - val_acc: 0.8807
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0244 - acc: 0.8720 - val_loss: 0.0236 - val_acc: 0.8810
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0244 - acc: 0.8724 - val_loss: 0.0235 - val_acc: 0.8812
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0243 - acc: 0.8725 - val_loss: 0.0234 - val_acc: 0.8815
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0242 - acc: 0.8726 - val_loss: 0.0234 - val_acc: 0.8817
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0242 - acc: 0.8728 - val_loss: 0.0233 - val_acc: 0.8818
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0241 - acc: 0.8730 - val_loss: 0.0232 - val_acc: 0.8818
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0240 - acc: 0.8734 - val_loss: 0.0232 - val_acc: 0.8820
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0240 - acc: 0.8734 - val_loss: 0.0231 - val_acc: 0.8821
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0239 - acc: 0.8737 - val_loss: 0.0231 - val_acc: 0.8822
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0239 - acc: 0.8739 - val_loss: 0.0230 - val_acc: 0.8823
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0238 - acc: 0.8740 - val_loss: 0.0229 - val_acc: 0.8827
    Epoch 77/99
    60000/60000 [==============================] - 1s - loss: 0.0237 - acc: 0.8742 - val_loss: 0.0229 - val_acc: 0.8830
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0237 - acc: 0.8744 - val_loss: 0.0228 - val_acc: 0.8836
    Epoch 79/99
    60000/60000 [==============================] - 1s - loss: 0.0236 - acc: 0.8744 - val_loss: 0.0228 - val_acc: 0.8838
    Epoch 80/99
    60000/60000 [==============================] - 1s - loss: 0.0236 - acc: 0.8746 - val_loss: 0.0227 - val_acc: 0.8838
    Epoch 81/99
    60000/60000 [==============================] - 1s - loss: 0.0235 - acc: 0.8748 - val_loss: 0.0226 - val_acc: 0.8841
    Epoch 82/99
    60000/60000 [==============================] - 1s - loss: 0.0235 - acc: 0.8751 - val_loss: 0.0226 - val_acc: 0.8844
    Epoch 83/99
    60000/60000 [==============================] - 1s - loss: 0.0234 - acc: 0.8751 - val_loss: 0.0225 - val_acc: 0.8846
    Epoch 84/99
    60000/60000 [==============================] - 1s - loss: 0.0233 - acc: 0.8753 - val_loss: 0.0225 - val_acc: 0.8849
    Epoch 85/99
    60000/60000 [==============================] - 1s - loss: 0.0233 - acc: 0.8754 - val_loss: 0.0224 - val_acc: 0.8852
    Epoch 86/99
    60000/60000 [==============================] - 1s - loss: 0.0232 - acc: 0.8756 - val_loss: 0.0224 - val_acc: 0.8852
    Epoch 87/99
    60000/60000 [==============================] - 1s - loss: 0.0232 - acc: 0.8758 - val_loss: 0.0223 - val_acc: 0.8852
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.0231 - acc: 0.8760 - val_loss: 0.0223 - val_acc: 0.8852
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.0231 - acc: 0.8761 - val_loss: 0.0222 - val_acc: 0.8853
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0230 - acc: 0.8762 - val_loss: 0.0222 - val_acc: 0.8854
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.0230 - acc: 0.8763 - val_loss: 0.0221 - val_acc: 0.8855
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0229 - acc: 0.8765 - val_loss: 0.0221 - val_acc: 0.8854
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0229 - acc: 0.8765 - val_loss: 0.0220 - val_acc: 0.8857
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0228 - acc: 0.8769 - val_loss: 0.0219 - val_acc: 0.8860
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0228 - acc: 0.8770 - val_loss: 0.0219 - val_acc: 0.8861
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0227 - acc: 0.8771 - val_loss: 0.0218 - val_acc: 0.8864
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0227 - acc: 0.8773 - val_loss: 0.0218 - val_acc: 0.8866
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0226 - acc: 0.8775 - val_loss: 0.0217 - val_acc: 0.8870
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0226 - acc: 0.8776 - val_loss: 0.0217 - val_acc: 0.8870

    <keras.callbacks.History at 0x12da6cef0>

```python
# 88% accuracy after another 99 epochs
nn.evaluate(X_test, y_test)
```

     7968/10000 [======================>.......] - ETA: 0s

    [0.021699914459884166, 0.88700000000000001]

## Activation functions

In computational networks, the [activation function](https://en.wikipedia.org/wiki/Activation_function) of a node
defines the output of that node given an input or set of inputs. A
standard computer chip circuit can be seen as a digital network of
activation functions that can be “ON” (1) or “OFF” (0), depending on
input. This is similar to the behavior of the linear perceptron in
neural networks. However, only *nonlinear* activation functions allow
such networks to compute nontrivial problems using only a small number
of nodes. In artificial neural networks this function is also called
the **transfer function**.

Functions
---------

In biologically inspired neural networks, the activation function is
usually an abstraction representing the rate of action potential
firing in the cell. In its simplest form, this function is binary—that
is, either the neuron is firing or not. The function looks like
$\phi(v_i)=U(v_i)$, where $U$ is the Heaviside step function. In this
case many neurons must be used in computation beyond linear separation
of categories.

A line of positive slope may be used to reflect the increase in firing
rate that occurs as input current increases. Such a function would be of
the form $\phi(v_i)=\mu v_i$, where $\mu$ is the slope. This activation
function is linear, and therefore has the same problems as the binary
function. In addition, networks constructed using this model have
unstable convergence because neuron inputs along favored paths tend to
increase without bound, as this function is not normalizable.

All problems mentioned above can be handled by using a normalizable
sigmoid activation function. One realistic model stays at zero until
input current is received, at which point the firing frequency increases
quickly at first, but gradually approaches an asymptote at 100% firing
rate. Mathematically, this looks like $\phi(v_i)=U(v_i)\tanh(v_i)$,
where the hyperbolic tangent function can be replaced by any sigmoid
function. This behavior is realistically reflected in the neuron, as
neurons cannot physically fire faster than a certain rate. This model
runs into problems, however, in computational networks as it is not
differentiable, a requirement to calculate backpropagation.

The final model, then, that is used in multilayer perceptrons is a
sigmoidal activation function in the form of a hyperbolic tangent. Two
forms of this function are commonly used: $\phi(v_i)=\tanh(v_i)$ whose
range is normalized from -1 to 1, and $\phi(v_i) = (1+\exp(-v_i))^{-1}$
is vertically translated to normalize from 0 to 1. The latter model is
often considered more biologically realistic, but it runs into
theoretical and experimental difficulties with certain types.

## Comparison of activation functions

Some desirable properties in an activation function include:

-   Nonlinear – When the activation function is non-linear, then a
    two-layer neural network can be proven to be a universal function
    approximator. The identity activation function does not satisfy
    this property. When multiple layers use the identity activation
    function, the entire network is equivalent to a single-layer model.
-   Continuously differentiable – This property is necessary for
    enabling gradient-based optimization methods. The binary step
    activation function is not differentiable at 0, and it
    differentiates to 0 for all other values, so gradient-based methods
    can make no progress with it.
-   Range – When the range of the activation function is finite,
    gradient-based training methods tend to be more stable, because
    pattern presentations significantly affect only limited weights.
    When the range is infinite, training is generally more efficient
    because pattern presentations significantly affect most of the
    weights. In the latter case, smaller learning rates are typically
    necessary.
-   Monotonic – When the activation function is monotonic, the error
    surface associated with a single-layer model is guaranteed to be
    convex.
-   Smooth Functions with a Monotonic derivative – These have been shown
    to generalize better in some cases. The argument for these
    properties suggests that such activation functions are more
    consistent with Occam's razor.
-   Approximates identity near the origin – When activation functions
    have this property, the neural network will learn efficiently when
    its weights are initialized with small random values. When the
    activation function does not approximate identity near the origin,
    special care must be used when initializing the weights.

## Rectified linear unit (ReLU) transfer function

Rectified linear unit (ReLU)

Activation identity

$f(x)=x$
$f'(x)=1$
$(-\infty,\infty)$
$C^\infty$

Logistic (a.k.a. Soft step)

$f(x)=\frac{1}{1+e^{-x}}$
$f'(x)=f(x)(1-f(x))$
$(0,1)$
$C^\infty$

TanH

$f(x)=\tanh(x)=\frac{2}{1+e^{-2x}}-1$
$f'(x)=1-f(x)^2$
$(-1,1)$
$C^\infty$

Rectified linear unit (ReLU)

$f(x) = \begin{cases}
    0 & \text{for } x < 0\\
    x & \text{for } x \ge 0\end{cases}$

$f'(x) = \begin{cases}
    0 & \text{for } x < 0\\
    1 & \text{for } x \ge 0\end{cases}$

$[0,\infty)$
$C^0$

The Rectified linear unit (ReLU) seem to work well empirically.

```python
def shallow_net_B(n=55,i=784,o=10):
    # create simple one dense layer net
    # default 55 neurons, input 784, output 10
    # Using relu
    net = Sequential()
    net.add(Dense(n, activation='relu', input_shape=(i,)))
    net.add(Dense(10, activation='softmax'))
    # Compile net
    net.compile(loss='mean_squared_error', optimizer=SGD(lr=0.01), metrics=['accuracy'])
    return net
```

```python
nn2=shallow_net_B()
nn2.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_9 (Dense)              (None, 55)                43175
    _________________________________________________________________
    dense_10 (Dense)             (None, 10)                560
    =================================================================
    Total params: 43,735
    Trainable params: 43,735
    Non-trainable params: 0
    _________________________________________________________________

```python
nn2.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0908 - acc: 0.1280 - val_loss: 0.0899 - val_acc: 0.1532
    Epoch 2/99
    60000/60000 [==============================] - 0s - loss: 0.0892 - acc: 0.1747 - val_loss: 0.0883 - val_acc: 0.2041
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0876 - acc: 0.2280 - val_loss: 0.0867 - val_acc: 0.2599
    Epoch 4/99
    60000/60000 [==============================] - 0s - loss: 0.0860 - acc: 0.2821 - val_loss: 0.0850 - val_acc: 0.3088
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0841 - acc: 0.3339 - val_loss: 0.0830 - val_acc: 0.3658
    Epoch 6/99
    60000/60000 [==============================] - 0s - loss: 0.0820 - acc: 0.3978 - val_loss: 0.0805 - val_acc: 0.4334
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0793 - acc: 0.4672 - val_loss: 0.0776 - val_acc: 0.4979
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.0762 - acc: 0.5256 - val_loss: 0.0743 - val_acc: 0.5482
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0728 - acc: 0.5646 - val_loss: 0.0708 - val_acc: 0.5871
    Epoch 10/99
    60000/60000 [==============================] - 1s - loss: 0.0693 - acc: 0.5964 - val_loss: 0.0672 - val_acc: 0.6210
    Epoch 11/99
    60000/60000 [==============================] - 1s - loss: 0.0656 - acc: 0.6282 - val_loss: 0.0634 - val_acc: 0.6523
    Epoch 12/99
    60000/60000 [==============================] - 1s - loss: 0.0620 - acc: 0.6607 - val_loss: 0.0597 - val_acc: 0.6871
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.0584 - acc: 0.6892 - val_loss: 0.0561 - val_acc: 0.7182
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.0549 - acc: 0.7147 - val_loss: 0.0526 - val_acc: 0.7407
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0517 - acc: 0.7391 - val_loss: 0.0494 - val_acc: 0.7622
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.0487 - acc: 0.7598 - val_loss: 0.0464 - val_acc: 0.7812
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0460 - acc: 0.7766 - val_loss: 0.0438 - val_acc: 0.7948
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0435 - acc: 0.7902 - val_loss: 0.0414 - val_acc: 0.8067
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0413 - acc: 0.8006 - val_loss: 0.0393 - val_acc: 0.8162
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0394 - acc: 0.8090 - val_loss: 0.0374 - val_acc: 0.8240
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0376 - acc: 0.8155 - val_loss: 0.0357 - val_acc: 0.8305
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0361 - acc: 0.8219 - val_loss: 0.0343 - val_acc: 0.8356
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0347 - acc: 0.8263 - val_loss: 0.0329 - val_acc: 0.8399
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0335 - acc: 0.8308 - val_loss: 0.0318 - val_acc: 0.8420
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.0324 - acc: 0.8345 - val_loss: 0.0307 - val_acc: 0.8457
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.0314 - acc: 0.8386 - val_loss: 0.0298 - val_acc: 0.8488
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.0305 - acc: 0.8417 - val_loss: 0.0289 - val_acc: 0.8509
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.0297 - acc: 0.8452 - val_loss: 0.0281 - val_acc: 0.8534
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.0289 - acc: 0.8481 - val_loss: 0.0274 - val_acc: 0.8551
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.0282 - acc: 0.8506 - val_loss: 0.0268 - val_acc: 0.8573
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.0276 - acc: 0.8531 - val_loss: 0.0262 - val_acc: 0.8598
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.0270 - acc: 0.8555 - val_loss: 0.0256 - val_acc: 0.8617
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.0265 - acc: 0.8573 - val_loss: 0.0251 - val_acc: 0.8636
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.0260 - acc: 0.8590 - val_loss: 0.0246 - val_acc: 0.8655
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.0255 - acc: 0.8608 - val_loss: 0.0242 - val_acc: 0.8672
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.0251 - acc: 0.8626 - val_loss: 0.0238 - val_acc: 0.8685
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.0247 - acc: 0.8639 - val_loss: 0.0234 - val_acc: 0.8699
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.0243 - acc: 0.8652 - val_loss: 0.0230 - val_acc: 0.8714
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.0239 - acc: 0.8663 - val_loss: 0.0227 - val_acc: 0.8726
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.0236 - acc: 0.8676 - val_loss: 0.0224 - val_acc: 0.8740
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.0233 - acc: 0.8690 - val_loss: 0.0221 - val_acc: 0.8754
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.0230 - acc: 0.8703 - val_loss: 0.0218 - val_acc: 0.8772
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.0227 - acc: 0.8716 - val_loss: 0.0215 - val_acc: 0.8778
    Epoch 44/99
    60000/60000 [==============================] - 0s - loss: 0.0224 - acc: 0.8727 - val_loss: 0.0213 - val_acc: 0.8793
    Epoch 45/99
    60000/60000 [==============================] - 0s - loss: 0.0222 - acc: 0.8733 - val_loss: 0.0210 - val_acc: 0.8806
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.0219 - acc: 0.8743 - val_loss: 0.0208 - val_acc: 0.8811
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.0217 - acc: 0.8752 - val_loss: 0.0206 - val_acc: 0.8823
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.0215 - acc: 0.8759 - val_loss: 0.0203 - val_acc: 0.8829
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.0212 - acc: 0.8768 - val_loss: 0.0201 - val_acc: 0.8836
    Epoch 50/99
    60000/60000 [==============================] - 0s - loss: 0.0210 - acc: 0.8775 - val_loss: 0.0200 - val_acc: 0.8840
    Epoch 51/99
    60000/60000 [==============================] - 0s - loss: 0.0208 - acc: 0.8782 - val_loss: 0.0198 - val_acc: 0.8846
    Epoch 52/99
    60000/60000 [==============================] - 0s - loss: 0.0207 - acc: 0.8794 - val_loss: 0.0196 - val_acc: 0.8855
    Epoch 53/99
    60000/60000 [==============================] - 0s - loss: 0.0205 - acc: 0.8801 - val_loss: 0.0194 - val_acc: 0.8863
    Epoch 54/99
    60000/60000 [==============================] - 0s - loss: 0.0203 - acc: 0.8809 - val_loss: 0.0193 - val_acc: 0.8870
    Epoch 55/99
    60000/60000 [==============================] - 0s - loss: 0.0201 - acc: 0.8816 - val_loss: 0.0191 - val_acc: 0.8877
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.0200 - acc: 0.8822 - val_loss: 0.0190 - val_acc: 0.8885
    Epoch 57/99
    60000/60000 [==============================] - 0s - loss: 0.0198 - acc: 0.8828 - val_loss: 0.0188 - val_acc: 0.8896
    Epoch 58/99
    60000/60000 [==============================] - 0s - loss: 0.0197 - acc: 0.8832 - val_loss: 0.0187 - val_acc: 0.8897
    Epoch 59/99
    60000/60000 [==============================] - 0s - loss: 0.0195 - acc: 0.8839 - val_loss: 0.0185 - val_acc: 0.8907
    Epoch 60/99
    60000/60000 [==============================] - 0s - loss: 0.0194 - acc: 0.8847 - val_loss: 0.0184 - val_acc: 0.8913
    Epoch 61/99
    60000/60000 [==============================] - 0s - loss: 0.0193 - acc: 0.8852 - val_loss: 0.0183 - val_acc: 0.8923
    Epoch 62/99
    60000/60000 [==============================] - 0s - loss: 0.0191 - acc: 0.8859 - val_loss: 0.0181 - val_acc: 0.8925
    Epoch 63/99
    60000/60000 [==============================] - 0s - loss: 0.0190 - acc: 0.8864 - val_loss: 0.0180 - val_acc: 0.8931
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0189 - acc: 0.8869 - val_loss: 0.0179 - val_acc: 0.8936
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0188 - acc: 0.8876 - val_loss: 0.0178 - val_acc: 0.8936
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0187 - acc: 0.8881 - val_loss: 0.0177 - val_acc: 0.8940
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0185 - acc: 0.8887 - val_loss: 0.0176 - val_acc: 0.8948
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0184 - acc: 0.8892 - val_loss: 0.0175 - val_acc: 0.8956
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0183 - acc: 0.8895 - val_loss: 0.0174 - val_acc: 0.8963
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0182 - acc: 0.8901 - val_loss: 0.0173 - val_acc: 0.8963
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0181 - acc: 0.8904 - val_loss: 0.0172 - val_acc: 0.8971
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0180 - acc: 0.8908 - val_loss: 0.0171 - val_acc: 0.8976
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0179 - acc: 0.8913 - val_loss: 0.0170 - val_acc: 0.8985
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0179 - acc: 0.8918 - val_loss: 0.0169 - val_acc: 0.8986
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0178 - acc: 0.8918 - val_loss: 0.0169 - val_acc: 0.8985
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0177 - acc: 0.8924 - val_loss: 0.0168 - val_acc: 0.8989
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0176 - acc: 0.8928 - val_loss: 0.0167 - val_acc: 0.8996
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0175 - acc: 0.8931 - val_loss: 0.0166 - val_acc: 0.9002
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0174 - acc: 0.8934 - val_loss: 0.0165 - val_acc: 0.9001
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0173 - acc: 0.8940 - val_loss: 0.0165 - val_acc: 0.9011
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0173 - acc: 0.8943 - val_loss: 0.0164 - val_acc: 0.9013
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0172 - acc: 0.8947 - val_loss: 0.0163 - val_acc: 0.9018
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0171 - acc: 0.8949 - val_loss: 0.0163 - val_acc: 0.9018
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0171 - acc: 0.8956 - val_loss: 0.0162 - val_acc: 0.9028
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0170 - acc: 0.8957 - val_loss: 0.0161 - val_acc: 0.9027
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0169 - acc: 0.8961 - val_loss: 0.0161 - val_acc: 0.9032
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0168 - acc: 0.8965 - val_loss: 0.0160 - val_acc: 0.9036
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.0168 - acc: 0.8968 - val_loss: 0.0159 - val_acc: 0.9041
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.0167 - acc: 0.8971 - val_loss: 0.0159 - val_acc: 0.9047
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0167 - acc: 0.8975 - val_loss: 0.0158 - val_acc: 0.9048
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.0166 - acc: 0.8978 - val_loss: 0.0158 - val_acc: 0.9048
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0165 - acc: 0.8981 - val_loss: 0.0157 - val_acc: 0.9059
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0165 - acc: 0.8985 - val_loss: 0.0156 - val_acc: 0.9058
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0164 - acc: 0.8987 - val_loss: 0.0156 - val_acc: 0.9060
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0164 - acc: 0.8991 - val_loss: 0.0155 - val_acc: 0.9061
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0163 - acc: 0.8995 - val_loss: 0.0155 - val_acc: 0.9065
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0162 - acc: 0.8999 - val_loss: 0.0154 - val_acc: 0.9066
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0162 - acc: 0.9001 - val_loss: 0.0154 - val_acc: 0.9070
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0161 - acc: 0.9005 - val_loss: 0.0153 - val_acc: 0.9072

    <keras.callbacks.History at 0x12de988d0>

```python
# 90% accuracy after first 99 epochs with Relu
nn2.evaluate(X_test, y_test)
```

     8256/10000 [=======================>......] - ETA: 0s

    [0.015325590824894608, 0.90720000000000001]

```python
nn2.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0161 - acc: 0.9008 - val_loss: 0.0153 - val_acc: 0.9072
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0160 - acc: 0.9011 - val_loss: 0.0152 - val_acc: 0.9075
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0160 - acc: 0.9013 - val_loss: 0.0152 - val_acc: 0.9079
    Epoch 4/99
    60000/60000 [==============================] - 0s - loss: 0.0159 - acc: 0.9016 - val_loss: 0.0151 - val_acc: 0.9084
    Epoch 5/99
    60000/60000 [==============================] - 0s - loss: 0.0159 - acc: 0.9019 - val_loss: 0.0151 - val_acc: 0.9087
    Epoch 6/99
    60000/60000 [==============================] - 0s - loss: 0.0158 - acc: 0.9021 - val_loss: 0.0150 - val_acc: 0.9090
    Epoch 7/99
    60000/60000 [==============================] - 0s - loss: 0.0158 - acc: 0.9022 - val_loss: 0.0150 - val_acc: 0.9093
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.0157 - acc: 0.9024 - val_loss: 0.0150 - val_acc: 0.9093
    Epoch 9/99
    60000/60000 [==============================] - 0s - loss: 0.0157 - acc: 0.9027 - val_loss: 0.0149 - val_acc: 0.9095
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.0156 - acc: 0.9028 - val_loss: 0.0149 - val_acc: 0.9095
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.0156 - acc: 0.9031 - val_loss: 0.0148 - val_acc: 0.9098
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.0156 - acc: 0.9033 - val_loss: 0.0148 - val_acc: 0.9101
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.0155 - acc: 0.9034 - val_loss: 0.0148 - val_acc: 0.9102
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.0155 - acc: 0.9039 - val_loss: 0.0147 - val_acc: 0.9108
    Epoch 15/99
    60000/60000 [==============================] - 0s - loss: 0.0154 - acc: 0.9039 - val_loss: 0.0147 - val_acc: 0.9107
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.0154 - acc: 0.9042 - val_loss: 0.0146 - val_acc: 0.9108
    Epoch 17/99
    60000/60000 [==============================] - 0s - loss: 0.0153 - acc: 0.9046 - val_loss: 0.0146 - val_acc: 0.9112
    Epoch 18/99
    60000/60000 [==============================] - 0s - loss: 0.0153 - acc: 0.9047 - val_loss: 0.0146 - val_acc: 0.9114
    Epoch 19/99
    60000/60000 [==============================] - 0s - loss: 0.0153 - acc: 0.9048 - val_loss: 0.0145 - val_acc: 0.9117
    Epoch 20/99
    60000/60000 [==============================] - 0s - loss: 0.0152 - acc: 0.9049 - val_loss: 0.0145 - val_acc: 0.9117
    Epoch 21/99
    60000/60000 [==============================] - 0s - loss: 0.0152 - acc: 0.9052 - val_loss: 0.0144 - val_acc: 0.9118
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.0151 - acc: 0.9054 - val_loss: 0.0144 - val_acc: 0.9118
    Epoch 23/99
    60000/60000 [==============================] - 0s - loss: 0.0151 - acc: 0.9056 - val_loss: 0.0144 - val_acc: 0.9119
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.0151 - acc: 0.9058 - val_loss: 0.0143 - val_acc: 0.9120
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.0150 - acc: 0.9059 - val_loss: 0.0143 - val_acc: 0.9123
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.0150 - acc: 0.9062 - val_loss: 0.0143 - val_acc: 0.9124
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.0150 - acc: 0.9064 - val_loss: 0.0142 - val_acc: 0.9125
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.0149 - acc: 0.9067 - val_loss: 0.0142 - val_acc: 0.9123
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.0149 - acc: 0.9069 - val_loss: 0.0142 - val_acc: 0.9125
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.0149 - acc: 0.9072 - val_loss: 0.0141 - val_acc: 0.9125
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.0148 - acc: 0.9074 - val_loss: 0.0141 - val_acc: 0.9125
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.0148 - acc: 0.9077 - val_loss: 0.0141 - val_acc: 0.9128
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9077 - val_loss: 0.0141 - val_acc: 0.9126
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9079 - val_loss: 0.0140 - val_acc: 0.9126
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9080 - val_loss: 0.0140 - val_acc: 0.9127
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9084 - val_loss: 0.0140 - val_acc: 0.9126
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.0146 - acc: 0.9086 - val_loss: 0.0139 - val_acc: 0.9127
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.0146 - acc: 0.9088 - val_loss: 0.0139 - val_acc: 0.9131
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.0146 - acc: 0.9089 - val_loss: 0.0139 - val_acc: 0.9134
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.0145 - acc: 0.9093 - val_loss: 0.0138 - val_acc: 0.9135
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.0145 - acc: 0.9095 - val_loss: 0.0138 - val_acc: 0.9136
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.0145 - acc: 0.9098 - val_loss: 0.0138 - val_acc: 0.9138
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.0144 - acc: 0.9098 - val_loss: 0.0138 - val_acc: 0.9138
    Epoch 44/99
    60000/60000 [==============================] - 0s - loss: 0.0144 - acc: 0.9099 - val_loss: 0.0137 - val_acc: 0.9141
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0144 - acc: 0.9100 - val_loss: 0.0137 - val_acc: 0.9140
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9104 - val_loss: 0.0137 - val_acc: 0.9141
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9104 - val_loss: 0.0137 - val_acc: 0.9143
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9107 - val_loss: 0.0136 - val_acc: 0.9145
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9108 - val_loss: 0.0136 - val_acc: 0.9145
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0142 - acc: 0.9111 - val_loss: 0.0136 - val_acc: 0.9146
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0142 - acc: 0.9114 - val_loss: 0.0136 - val_acc: 0.9149
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0142 - acc: 0.9115 - val_loss: 0.0135 - val_acc: 0.9147
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0141 - acc: 0.9116 - val_loss: 0.0135 - val_acc: 0.9150
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0141 - acc: 0.9117 - val_loss: 0.0135 - val_acc: 0.9151
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0141 - acc: 0.9118 - val_loss: 0.0135 - val_acc: 0.9150
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.0141 - acc: 0.9121 - val_loss: 0.0134 - val_acc: 0.9153
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0140 - acc: 0.9122 - val_loss: 0.0134 - val_acc: 0.9154
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0140 - acc: 0.9123 - val_loss: 0.0134 - val_acc: 0.9157
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0140 - acc: 0.9125 - val_loss: 0.0134 - val_acc: 0.9160
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0140 - acc: 0.9127 - val_loss: 0.0133 - val_acc: 0.9159
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0139 - acc: 0.9127 - val_loss: 0.0133 - val_acc: 0.9161
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0139 - acc: 0.9127 - val_loss: 0.0133 - val_acc: 0.9160
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0139 - acc: 0.9129 - val_loss: 0.0133 - val_acc: 0.9163
    Epoch 64/99
    60000/60000 [==============================] - 1s - loss: 0.0139 - acc: 0.9131 - val_loss: 0.0132 - val_acc: 0.9163
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9133 - val_loss: 0.0132 - val_acc: 0.9165
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9134 - val_loss: 0.0132 - val_acc: 0.9165
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9134 - val_loss: 0.0132 - val_acc: 0.9167
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9137 - val_loss: 0.0132 - val_acc: 0.9167
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9136 - val_loss: 0.0131 - val_acc: 0.9170
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9138 - val_loss: 0.0131 - val_acc: 0.9172
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9140 - val_loss: 0.0131 - val_acc: 0.9173
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9141 - val_loss: 0.0131 - val_acc: 0.9173
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9143 - val_loss: 0.0131 - val_acc: 0.9176
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9143 - val_loss: 0.0130 - val_acc: 0.9179
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9146 - val_loss: 0.0130 - val_acc: 0.9179
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9147 - val_loss: 0.0130 - val_acc: 0.9182
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9147 - val_loss: 0.0130 - val_acc: 0.9184
    Epoch 78/99
    60000/60000 [==============================] - 1s - loss: 0.0135 - acc: 0.9148 - val_loss: 0.0130 - val_acc: 0.9186
    Epoch 79/99
    60000/60000 [==============================] - 1s - loss: 0.0135 - acc: 0.9150 - val_loss: 0.0129 - val_acc: 0.9186
    Epoch 80/99
    60000/60000 [==============================] - 1s - loss: 0.0135 - acc: 0.9152 - val_loss: 0.0129 - val_acc: 0.9189
    Epoch 81/99
    60000/60000 [==============================] - 1s - loss: 0.0135 - acc: 0.9153 - val_loss: 0.0129 - val_acc: 0.9191
    Epoch 82/99
    60000/60000 [==============================] - 1s - loss: 0.0135 - acc: 0.9155 - val_loss: 0.0129 - val_acc: 0.9191
    Epoch 83/99
    60000/60000 [==============================] - 1s - loss: 0.0134 - acc: 0.9156 - val_loss: 0.0129 - val_acc: 0.9192
    Epoch 84/99
    60000/60000 [==============================] - 1s - loss: 0.0134 - acc: 0.9158 - val_loss: 0.0128 - val_acc: 0.9195
    Epoch 85/99
    60000/60000 [==============================] - 1s - loss: 0.0134 - acc: 0.9160 - val_loss: 0.0128 - val_acc: 0.9195
    Epoch 86/99
    60000/60000 [==============================] - 1s - loss: 0.0134 - acc: 0.9160 - val_loss: 0.0128 - val_acc: 0.9199
    Epoch 87/99
    60000/60000 [==============================] - 1s - loss: 0.0134 - acc: 0.9162 - val_loss: 0.0128 - val_acc: 0.9203
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.0133 - acc: 0.9163 - val_loss: 0.0128 - val_acc: 0.9200
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.0133 - acc: 0.9164 - val_loss: 0.0127 - val_acc: 0.9203
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0133 - acc: 0.9166 - val_loss: 0.0127 - val_acc: 0.9205
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.0133 - acc: 0.9167 - val_loss: 0.0127 - val_acc: 0.9204
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0133 - acc: 0.9169 - val_loss: 0.0127 - val_acc: 0.9208
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0132 - acc: 0.9171 - val_loss: 0.0127 - val_acc: 0.9210
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0132 - acc: 0.9170 - val_loss: 0.0127 - val_acc: 0.9209
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0132 - acc: 0.9170 - val_loss: 0.0126 - val_acc: 0.9213
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0132 - acc: 0.9173 - val_loss: 0.0126 - val_acc: 0.9215
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0132 - acc: 0.9174 - val_loss: 0.0126 - val_acc: 0.9218
    Epoch 98/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9174 - val_loss: 0.0126 - val_acc: 0.9217
    Epoch 99/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9177 - val_loss: 0.0126 - val_acc: 0.9217

    <keras.callbacks.History at 0x12dbfa128>

```python
# 92% accuracy after another 99 epochs with Relu
# Seems to be a plateau
nn2.evaluate(X_test, y_test)
```

     9216/10000 [==========================>...] - ETA: 0s

    [0.012574804448196664, 0.92169999999999996]

## Loss or cost functions

Loss function
-------------

Sometimes referred to as the **cost function** or **error function**
(not to be confused with the Gauss error function), the loss function
is a function that maps values of one or more variables onto a real
number intuitively representing some “cost” associated with those
values. For backpropagation, the loss function calculates the difference
between the network output and its expected output, after a case
propagates through the network.

### Assumptions

Two assumptions must be made about the form of the error function.

The first is that it can be written as an average
$E=\frac{1}{n}\sum_xE_x$ over error functions $E_x$, for individual
training examples, $x$. The reason for this assumption is that the
backpropagation algorithm calculates the gradient of the error function
for a single training example, which needs to be generalized to the
overall error function. The second assumption is that it can be written
as a function of the outputs from the neural network.

### Example loss function

Let $y,y'$ be vectors in $\mathbb{R}^n$.

Select an error function $E(y,y')$ measuring the difference between two
outputs.

The standard choice is $E(y,y') = \tfrac{1}{2} \lVert y-y'\rVert^2$,

the square of the Euclidean distance between the vectors $y$ and $y'$.

The factor of $\tfrac{1}{2}$ conveniently cancels the exponent when the
error function is subsequently differentiated.

The error function over $n$ training examples can be written as an
average$$E=\frac{1}{2n}\sum_x\lVert (y(x)-y'(x)) \rVert^2$$And the
partial derivative with respect to the
outputs$$\frac{\partial E}{\partial y'} = y'-y$$

## Cross entropy

In information theory, the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) between two probability
distributions $p$ and $q$ over the same underlying set of events
measures the average number of bits needed to identify an event drawn
from the set, if a coding scheme is used that is optimized for an
“unnatural” probability distribution $q$, rather than the “true”
distribution $p$.

The cross entropy for the distributions $p$ and $q$ over a given set is
defined as follows:

$$H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q),\!$$

where $H(p)$ is the entropy of $p$, and $D_{\mathrm{KL}}(p \| q)$ is
the [Kullback–Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) of $q$ from $p$ (also known as the
*relative entropy* of *p* with respect to *q* — note the reversal of
emphasis).

For discrete $p$ and $q$ this means

$$H(p, q) = -\sum_x p(x)\, \log q(x). \!$$

The situation for continuous distributions is analogous. We have to
assume that $p$ and $q$ are [absolutely continuous] with respect to some
reference [measure] $r$ (usually $r$ is a Lebesgue measure on a
Borel [σ-algebra]). Let $P$ and $Q$ be probability density functions
of $p$ and $q$ with respect to $r$. Then

$$-\int_X P(x)\, \log Q(x)\, dr(x) = \operatorname{E}_p[-\log Q]. \!$$

NB: The notation $H(p,q)$ is also used for a different concept, the
joint entropy of $p$ and $q$.

```python
def shallow_net_C(n=55,i=784,o=10):
    # create simple one dense layer net
    # default 55 neurons, input 784, output 10
    # Using relu and
    net = Sequential()
    net.add(Dense(n, activation='relu', input_shape=(i,)))
    net.add(Dense(10, activation='softmax'))
    # Compile net
    net.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01), metrics=['accuracy'])
    return net
```

```python
nn3=shallow_net_C()
nn3.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_11 (Dense)             (None, 55)                43175
    _________________________________________________________________
    dense_12 (Dense)             (None, 10)                560
    =================================================================
    Total params: 43,735
    Trainable params: 43,735
    Non-trainable params: 0
    _________________________________________________________________

```python
nn3.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 1.2930 - acc: 0.6790 - val_loss: 0.7058 - val_acc: 0.8356
    Epoch 2/99
    60000/60000 [==============================] - 0s - loss: 0.5927 - acc: 0.8564 - val_loss: 0.4847 - val_acc: 0.8768
    Epoch 3/99
    60000/60000 [==============================] - 0s - loss: 0.4621 - acc: 0.8795 - val_loss: 0.4096 - val_acc: 0.8915
    Epoch 4/99
    60000/60000 [==============================] - 0s - loss: 0.4067 - acc: 0.8904 - val_loss: 0.3723 - val_acc: 0.8990
    Epoch 5/99
    60000/60000 [==============================] - 0s - loss: 0.3747 - acc: 0.8968 - val_loss: 0.3471 - val_acc: 0.9027
    Epoch 6/99
    60000/60000 [==============================] - 0s - loss: 0.3528 - acc: 0.9021 - val_loss: 0.3297 - val_acc: 0.9063
    Epoch 7/99
    60000/60000 [==============================] - 0s - loss: 0.3365 - acc: 0.9057 - val_loss: 0.3159 - val_acc: 0.9101
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.3235 - acc: 0.9094 - val_loss: 0.3053 - val_acc: 0.9133
    Epoch 9/99
    60000/60000 [==============================] - 0s - loss: 0.3126 - acc: 0.9124 - val_loss: 0.2961 - val_acc: 0.9169
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.3032 - acc: 0.9152 - val_loss: 0.2884 - val_acc: 0.9185
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.2951 - acc: 0.9178 - val_loss: 0.2820 - val_acc: 0.9203
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.2878 - acc: 0.9193 - val_loss: 0.2748 - val_acc: 0.9231
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.2811 - acc: 0.9214 - val_loss: 0.2695 - val_acc: 0.9253
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.2749 - acc: 0.9230 - val_loss: 0.2649 - val_acc: 0.9260
    Epoch 15/99
    60000/60000 [==============================] - 0s - loss: 0.2694 - acc: 0.9250 - val_loss: 0.2591 - val_acc: 0.9280
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.2642 - acc: 0.9261 - val_loss: 0.2546 - val_acc: 0.9290
    Epoch 17/99
    60000/60000 [==============================] - 0s - loss: 0.2592 - acc: 0.9275 - val_loss: 0.2504 - val_acc: 0.9305
    Epoch 18/99
    60000/60000 [==============================] - 0s - loss: 0.2544 - acc: 0.9286 - val_loss: 0.2466 - val_acc: 0.9320
    Epoch 19/99
    60000/60000 [==============================] - 0s - loss: 0.2501 - acc: 0.9296 - val_loss: 0.2429 - val_acc: 0.9329
    Epoch 20/99
    60000/60000 [==============================] - 0s - loss: 0.2460 - acc: 0.9310 - val_loss: 0.2398 - val_acc: 0.9331
    Epoch 21/99
    60000/60000 [==============================] - 0s - loss: 0.2421 - acc: 0.9318 - val_loss: 0.2358 - val_acc: 0.9353
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.2383 - acc: 0.9331 - val_loss: 0.2326 - val_acc: 0.9347
    Epoch 23/99
    60000/60000 [==============================] - 0s - loss: 0.2347 - acc: 0.9339 - val_loss: 0.2299 - val_acc: 0.9363
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.2313 - acc: 0.9352 - val_loss: 0.2266 - val_acc: 0.9366
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.2279 - acc: 0.9361 - val_loss: 0.2237 - val_acc: 0.9375
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.2247 - acc: 0.9367 - val_loss: 0.2211 - val_acc: 0.9379
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.2216 - acc: 0.9379 - val_loss: 0.2192 - val_acc: 0.9380
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.2186 - acc: 0.9384 - val_loss: 0.2159 - val_acc: 0.9384
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.2157 - acc: 0.9392 - val_loss: 0.2135 - val_acc: 0.9396
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.2130 - acc: 0.9398 - val_loss: 0.2116 - val_acc: 0.9400
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.2103 - acc: 0.9410 - val_loss: 0.2090 - val_acc: 0.9408
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.2076 - acc: 0.9416 - val_loss: 0.2067 - val_acc: 0.9407
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.2051 - acc: 0.9422 - val_loss: 0.2048 - val_acc: 0.9416
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.2025 - acc: 0.9429 - val_loss: 0.2035 - val_acc: 0.9419
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.2002 - acc: 0.9436 - val_loss: 0.2007 - val_acc: 0.9422
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.1977 - acc: 0.9445 - val_loss: 0.1997 - val_acc: 0.9442
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.1954 - acc: 0.9450 - val_loss: 0.1982 - val_acc: 0.9434
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.1931 - acc: 0.9456 - val_loss: 0.1950 - val_acc: 0.9440
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.1909 - acc: 0.9464 - val_loss: 0.1933 - val_acc: 0.9444
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.1888 - acc: 0.9470 - val_loss: 0.1916 - val_acc: 0.9456
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.1866 - acc: 0.9476 - val_loss: 0.1898 - val_acc: 0.9451
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.1846 - acc: 0.9479 - val_loss: 0.1874 - val_acc: 0.9457
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.1826 - acc: 0.9489 - val_loss: 0.1860 - val_acc: 0.9466
    Epoch 44/99
    60000/60000 [==============================] - 0s - loss: 0.1807 - acc: 0.9493 - val_loss: 0.1844 - val_acc: 0.9466
    Epoch 45/99
    60000/60000 [==============================] - 0s - loss: 0.1787 - acc: 0.9499 - val_loss: 0.1828 - val_acc: 0.9471
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.1769 - acc: 0.9502 - val_loss: 0.1811 - val_acc: 0.9482
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.1749 - acc: 0.9509 - val_loss: 0.1798 - val_acc: 0.9482
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.1732 - acc: 0.9512 - val_loss: 0.1782 - val_acc: 0.9485
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.1713 - acc: 0.9517 - val_loss: 0.1772 - val_acc: 0.9495
    Epoch 50/99
    60000/60000 [==============================] - 0s - loss: 0.1696 - acc: 0.9526 - val_loss: 0.1753 - val_acc: 0.9495
    Epoch 51/99
    60000/60000 [==============================] - 0s - loss: 0.1679 - acc: 0.9532 - val_loss: 0.1738 - val_acc: 0.9495
    Epoch 52/99
    60000/60000 [==============================] - 0s - loss: 0.1661 - acc: 0.9534 - val_loss: 0.1721 - val_acc: 0.9510
    Epoch 53/99
    60000/60000 [==============================] - 0s - loss: 0.1644 - acc: 0.9541 - val_loss: 0.1708 - val_acc: 0.9509
    Epoch 54/99
    60000/60000 [==============================] - 0s - loss: 0.1628 - acc: 0.9547 - val_loss: 0.1694 - val_acc: 0.9508
    Epoch 55/99
    60000/60000 [==============================] - 0s - loss: 0.1612 - acc: 0.9551 - val_loss: 0.1678 - val_acc: 0.9506
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.1597 - acc: 0.9557 - val_loss: 0.1674 - val_acc: 0.9518
    Epoch 57/99
    60000/60000 [==============================] - 0s - loss: 0.1581 - acc: 0.9557 - val_loss: 0.1658 - val_acc: 0.9513
    Epoch 58/99
    60000/60000 [==============================] - 0s - loss: 0.1566 - acc: 0.9562 - val_loss: 0.1642 - val_acc: 0.9528
    Epoch 59/99
    60000/60000 [==============================] - 0s - loss: 0.1552 - acc: 0.9565 - val_loss: 0.1635 - val_acc: 0.9526
    Epoch 60/99
    60000/60000 [==============================] - 0s - loss: 0.1536 - acc: 0.9567 - val_loss: 0.1619 - val_acc: 0.9533
    Epoch 61/99
    60000/60000 [==============================] - 0s - loss: 0.1523 - acc: 0.9576 - val_loss: 0.1611 - val_acc: 0.9539
    Epoch 62/99
    60000/60000 [==============================] - 0s - loss: 0.1509 - acc: 0.9578 - val_loss: 0.1591 - val_acc: 0.9538
    Epoch 63/99
    60000/60000 [==============================] - 0s - loss: 0.1495 - acc: 0.9582 - val_loss: 0.1580 - val_acc: 0.9549
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.1482 - acc: 0.9587 - val_loss: 0.1574 - val_acc: 0.9534
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.1468 - acc: 0.9592 - val_loss: 0.1561 - val_acc: 0.9545
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.1456 - acc: 0.9594 - val_loss: 0.1546 - val_acc: 0.9555
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.1443 - acc: 0.9598 - val_loss: 0.1534 - val_acc: 0.9556
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.1430 - acc: 0.9602 - val_loss: 0.1530 - val_acc: 0.9562
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.1418 - acc: 0.9605 - val_loss: 0.1512 - val_acc: 0.9565
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.1405 - acc: 0.9609 - val_loss: 0.1508 - val_acc: 0.9563
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.1393 - acc: 0.9614 - val_loss: 0.1501 - val_acc: 0.9561
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.1383 - acc: 0.9615 - val_loss: 0.1488 - val_acc: 0.9574
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.1370 - acc: 0.9620 - val_loss: 0.1483 - val_acc: 0.9580
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.1359 - acc: 0.9624 - val_loss: 0.1468 - val_acc: 0.9582
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.1348 - acc: 0.9625 - val_loss: 0.1459 - val_acc: 0.9581
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.1337 - acc: 0.9627 - val_loss: 0.1446 - val_acc: 0.9596
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.1326 - acc: 0.9631 - val_loss: 0.1443 - val_acc: 0.9597
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.1315 - acc: 0.9631 - val_loss: 0.1434 - val_acc: 0.9595
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.1304 - acc: 0.9636 - val_loss: 0.1429 - val_acc: 0.9590
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.1294 - acc: 0.9643 - val_loss: 0.1413 - val_acc: 0.9596
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.1284 - acc: 0.9644 - val_loss: 0.1410 - val_acc: 0.9595
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.1274 - acc: 0.9647 - val_loss: 0.1397 - val_acc: 0.9606
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.1264 - acc: 0.9654 - val_loss: 0.1395 - val_acc: 0.9600
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.1254 - acc: 0.9650 - val_loss: 0.1384 - val_acc: 0.9600
    Epoch 85/99
    60000/60000 [==============================] - 1s - loss: 0.1244 - acc: 0.9657 - val_loss: 0.1369 - val_acc: 0.9611
    Epoch 86/99
    60000/60000 [==============================] - 1s - loss: 0.1234 - acc: 0.9659 - val_loss: 0.1370 - val_acc: 0.9601
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.1225 - acc: 0.9663 - val_loss: 0.1366 - val_acc: 0.9608
    Epoch 88/99
    60000/60000 [==============================] - 0s - loss: 0.1216 - acc: 0.9666 - val_loss: 0.1349 - val_acc: 0.9615
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.1208 - acc: 0.9667 - val_loss: 0.1345 - val_acc: 0.9615
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.1197 - acc: 0.9672 - val_loss: 0.1341 - val_acc: 0.9620
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.1189 - acc: 0.9676 - val_loss: 0.1330 - val_acc: 0.9620
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.1180 - acc: 0.9678 - val_loss: 0.1327 - val_acc: 0.9623
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.1172 - acc: 0.9675 - val_loss: 0.1316 - val_acc: 0.9628
    Epoch 94/99
    60000/60000 [==============================] - 0s - loss: 0.1163 - acc: 0.9681 - val_loss: 0.1307 - val_acc: 0.9625
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.1154 - acc: 0.9682 - val_loss: 0.1300 - val_acc: 0.9625
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.1146 - acc: 0.9682 - val_loss: 0.1303 - val_acc: 0.9624
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.1139 - acc: 0.9685 - val_loss: 0.1287 - val_acc: 0.9631
    Epoch 98/99
    60000/60000 [==============================] - 0s - loss: 0.1131 - acc: 0.9688 - val_loss: 0.1282 - val_acc: 0.9632
    Epoch 99/99
    60000/60000 [==============================] - 0s - loss: 0.1123 - acc: 0.9690 - val_loss: 0.1277 - val_acc: 0.9633

    <keras.callbacks.History at 0x12d765908>

```python
# 96% accuracy after first 99 epochs with Relu and Cross-entropy
nn3.evaluate(X_test, y_test)
```

     9152/10000 [==========================>...] - ETA: 0s

    [0.12768967846110463, 0.96330000000000005]

```python
nn3.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 0s - loss: 0.1115 - acc: 0.9694 - val_loss: 0.1281 - val_acc: 0.9638
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.1107 - acc: 0.9695 - val_loss: 0.1259 - val_acc: 0.9642
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.1100 - acc: 0.9696 - val_loss: 0.1265 - val_acc: 0.9636
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.1092 - acc: 0.9702 - val_loss: 0.1244 - val_acc: 0.9643
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.1085 - acc: 0.9701 - val_loss: 0.1245 - val_acc: 0.9645
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.1077 - acc: 0.9702 - val_loss: 0.1242 - val_acc: 0.9642
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.1070 - acc: 0.9706 - val_loss: 0.1237 - val_acc: 0.9643
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.1063 - acc: 0.9708 - val_loss: 0.1228 - val_acc: 0.9649
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.1056 - acc: 0.9712 - val_loss: 0.1229 - val_acc: 0.9645
    Epoch 10/99
    60000/60000 [==============================] - 1s - loss: 0.1049 - acc: 0.9709 - val_loss: 0.1224 - val_acc: 0.9646
    Epoch 11/99
    60000/60000 [==============================] - 1s - loss: 0.1042 - acc: 0.9714 - val_loss: 0.1216 - val_acc: 0.9651
    Epoch 12/99
    60000/60000 [==============================] - 1s - loss: 0.1035 - acc: 0.9716 - val_loss: 0.1218 - val_acc: 0.9646
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.1029 - acc: 0.9717 - val_loss: 0.1207 - val_acc: 0.9654
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.1022 - acc: 0.9719 - val_loss: 0.1201 - val_acc: 0.9649
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.1016 - acc: 0.9723 - val_loss: 0.1204 - val_acc: 0.9653
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.1010 - acc: 0.9726 - val_loss: 0.1190 - val_acc: 0.9657
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.1004 - acc: 0.9726 - val_loss: 0.1186 - val_acc: 0.9659
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0997 - acc: 0.9726 - val_loss: 0.1178 - val_acc: 0.9660
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0991 - acc: 0.9727 - val_loss: 0.1175 - val_acc: 0.9658
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0985 - acc: 0.9731 - val_loss: 0.1177 - val_acc: 0.9659
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0979 - acc: 0.9733 - val_loss: 0.1168 - val_acc: 0.9663
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0973 - acc: 0.9734 - val_loss: 0.1165 - val_acc: 0.9659
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0968 - acc: 0.9736 - val_loss: 0.1158 - val_acc: 0.9669
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0961 - acc: 0.9737 - val_loss: 0.1158 - val_acc: 0.9663
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0955 - acc: 0.9738 - val_loss: 0.1152 - val_acc: 0.9663
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0949 - acc: 0.9741 - val_loss: 0.1149 - val_acc: 0.9666
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0944 - acc: 0.9739 - val_loss: 0.1139 - val_acc: 0.9667
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0939 - acc: 0.9742 - val_loss: 0.1140 - val_acc: 0.9669
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0933 - acc: 0.9741 - val_loss: 0.1136 - val_acc: 0.9672
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0927 - acc: 0.9747 - val_loss: 0.1138 - val_acc: 0.9674
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0922 - acc: 0.9746 - val_loss: 0.1132 - val_acc: 0.9676
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0917 - acc: 0.9747 - val_loss: 0.1123 - val_acc: 0.9680
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0912 - acc: 0.9746 - val_loss: 0.1124 - val_acc: 0.9674
    Epoch 34/99
    60000/60000 [==============================] - 1s - loss: 0.0906 - acc: 0.9749 - val_loss: 0.1121 - val_acc: 0.9676
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0902 - acc: 0.9750 - val_loss: 0.1113 - val_acc: 0.9678
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0895 - acc: 0.9752 - val_loss: 0.1115 - val_acc: 0.9673
    Epoch 37/99
    60000/60000 [==============================] - 1s - loss: 0.0891 - acc: 0.9755 - val_loss: 0.1108 - val_acc: 0.9686
    Epoch 38/99
    60000/60000 [==============================] - 1s - loss: 0.0887 - acc: 0.9755 - val_loss: 0.1107 - val_acc: 0.9682
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0881 - acc: 0.9758 - val_loss: 0.1101 - val_acc: 0.9680
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0877 - acc: 0.9755 - val_loss: 0.1098 - val_acc: 0.9688
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0872 - acc: 0.9758 - val_loss: 0.1100 - val_acc: 0.9683
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0866 - acc: 0.9760 - val_loss: 0.1097 - val_acc: 0.9688
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0863 - acc: 0.9759 - val_loss: 0.1084 - val_acc: 0.9689
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0858 - acc: 0.9762 - val_loss: 0.1086 - val_acc: 0.9686
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0854 - acc: 0.9765 - val_loss: 0.1085 - val_acc: 0.9689
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.0848 - acc: 0.9765 - val_loss: 0.1078 - val_acc: 0.9694
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0844 - acc: 0.9763 - val_loss: 0.1077 - val_acc: 0.9689
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0839 - acc: 0.9767 - val_loss: 0.1075 - val_acc: 0.9690
    Epoch 49/99
    60000/60000 [==============================] - 1s - loss: 0.0835 - acc: 0.9770 - val_loss: 0.1063 - val_acc: 0.9695
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0831 - acc: 0.9768 - val_loss: 0.1068 - val_acc: 0.9698
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0826 - acc: 0.9771 - val_loss: 0.1063 - val_acc: 0.9697
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0822 - acc: 0.9773 - val_loss: 0.1059 - val_acc: 0.9695
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0817 - acc: 0.9773 - val_loss: 0.1062 - val_acc: 0.9694
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0814 - acc: 0.9775 - val_loss: 0.1053 - val_acc: 0.9699
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0810 - acc: 0.9775 - val_loss: 0.1053 - val_acc: 0.9699
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.0805 - acc: 0.9774 - val_loss: 0.1047 - val_acc: 0.9704
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0802 - acc: 0.9778 - val_loss: 0.1042 - val_acc: 0.9700
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0798 - acc: 0.9779 - val_loss: 0.1044 - val_acc: 0.9699
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0793 - acc: 0.9780 - val_loss: 0.1046 - val_acc: 0.9709
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0790 - acc: 0.9782 - val_loss: 0.1043 - val_acc: 0.9706
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0785 - acc: 0.9782 - val_loss: 0.1041 - val_acc: 0.9706
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0782 - acc: 0.9784 - val_loss: 0.1037 - val_acc: 0.9705
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0778 - acc: 0.9782 - val_loss: 0.1031 - val_acc: 0.9709
    Epoch 64/99
    60000/60000 [==============================] - 1s - loss: 0.0773 - acc: 0.9786 - val_loss: 0.1031 - val_acc: 0.9709
    Epoch 65/99
    60000/60000 [==============================] - 1s - loss: 0.0770 - acc: 0.9789 - val_loss: 0.1028 - val_acc: 0.9703
    Epoch 66/99
    60000/60000 [==============================] - 1s - loss: 0.0767 - acc: 0.9787 - val_loss: 0.1024 - val_acc: 0.9708
    Epoch 67/99
    60000/60000 [==============================] - 1s - loss: 0.0763 - acc: 0.9788 - val_loss: 0.1022 - val_acc: 0.9709
    Epoch 68/99
    60000/60000 [==============================] - 1s - loss: 0.0760 - acc: 0.9789 - val_loss: 0.1018 - val_acc: 0.9714
    Epoch 69/99
    60000/60000 [==============================] - 1s - loss: 0.0756 - acc: 0.9790 - val_loss: 0.1016 - val_acc: 0.9716
    Epoch 70/99
    60000/60000 [==============================] - 1s - loss: 0.0752 - acc: 0.9788 - val_loss: 0.1018 - val_acc: 0.9709
    Epoch 71/99
    60000/60000 [==============================] - 1s - loss: 0.0749 - acc: 0.9791 - val_loss: 0.1020 - val_acc: 0.9708
    Epoch 72/99
    60000/60000 [==============================] - 1s - loss: 0.0746 - acc: 0.9791 - val_loss: 0.1019 - val_acc: 0.9710
    Epoch 73/99
    60000/60000 [==============================] - 1s - loss: 0.0742 - acc: 0.9794 - val_loss: 0.1011 - val_acc: 0.9717
    Epoch 74/99
    60000/60000 [==============================] - 1s - loss: 0.0738 - acc: 0.9795 - val_loss: 0.1007 - val_acc: 0.9716
    Epoch 75/99
    60000/60000 [==============================] - 1s - loss: 0.0735 - acc: 0.9797 - val_loss: 0.1003 - val_acc: 0.9713
    Epoch 76/99
    60000/60000 [==============================] - 1s - loss: 0.0732 - acc: 0.9796 - val_loss: 0.1010 - val_acc: 0.9714
    Epoch 77/99
    60000/60000 [==============================] - 1s - loss: 0.0728 - acc: 0.9798 - val_loss: 0.1002 - val_acc: 0.9719
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0725 - acc: 0.9799 - val_loss: 0.1008 - val_acc: 0.9711
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0722 - acc: 0.9801 - val_loss: 0.1003 - val_acc: 0.9713
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0718 - acc: 0.9803 - val_loss: 0.1000 - val_acc: 0.9717
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0716 - acc: 0.9804 - val_loss: 0.0994 - val_acc: 0.9716
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0711 - acc: 0.9803 - val_loss: 0.1002 - val_acc: 0.9716
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0709 - acc: 0.9804 - val_loss: 0.0992 - val_acc: 0.9719
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0705 - acc: 0.9802 - val_loss: 0.0999 - val_acc: 0.9712
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0702 - acc: 0.9804 - val_loss: 0.0992 - val_acc: 0.9723
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0699 - acc: 0.9809 - val_loss: 0.0992 - val_acc: 0.9720
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0696 - acc: 0.9808 - val_loss: 0.0993 - val_acc: 0.9718
    Epoch 88/99
    60000/60000 [==============================] - 0s - loss: 0.0693 - acc: 0.9806 - val_loss: 0.0988 - val_acc: 0.9720
    Epoch 89/99
    60000/60000 [==============================] - 0s - loss: 0.0690 - acc: 0.9811 - val_loss: 0.0985 - val_acc: 0.9714
    Epoch 90/99
    60000/60000 [==============================] - 0s - loss: 0.0687 - acc: 0.9810 - val_loss: 0.0978 - val_acc: 0.9721
    Epoch 91/99
    60000/60000 [==============================] - 0s - loss: 0.0684 - acc: 0.9812 - val_loss: 0.0983 - val_acc: 0.9721
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0681 - acc: 0.9811 - val_loss: 0.0984 - val_acc: 0.9723
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0679 - acc: 0.9811 - val_loss: 0.0977 - val_acc: 0.9721
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0676 - acc: 0.9814 - val_loss: 0.0979 - val_acc: 0.9722
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0673 - acc: 0.9815 - val_loss: 0.0972 - val_acc: 0.9726
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0670 - acc: 0.9815 - val_loss: 0.0970 - val_acc: 0.9727
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0667 - acc: 0.9817 - val_loss: 0.0970 - val_acc: 0.9725
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0663 - acc: 0.9817 - val_loss: 0.0970 - val_acc: 0.9724
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0662 - acc: 0.9817 - val_loss: 0.0967 - val_acc: 0.9728

    <keras.callbacks.History at 0x12d232ba8>

```python
# 97% accuracy after another 99 epochs with Relu and Cross-entropy
nn3.evaluate(X_test, y_test)
```

    10000/10000 [==============================] - 0s

    [0.096727926008403298, 0.9728]

## Summary

With a fairly simple shallow net we've done fairly well classifying (97% accuracy after another 99 epochs with Relu and Cross-entropy) on the [MNIST](http://yann.lecun.com/exdb/mnist/)  handwritten digit classification problem.

Last update September 22, 2017

Prof. Nik Brown's Notes/C05. NeuralNetworks/1. NBB_Deep_Learning_Shallow_Neural_Networks.ipynb