# Deep Learning

- 01 What is "Deep Learning?"
- 02 Why Deep Learning?
- 03 The Perceptron (Neural units)
- 04 Shallow Neural Network
- 05 Activation functions
- 06 Loss functions
- 07 Cross entropy

## What is "Deep Learning?"

[Deep learning](https://en.wikipedia.org/wiki/Deep_learning) is the stacking of artificial neural networks (ANNs) to create stacked neural networks, [deep belief networks](https://en.wikipedia.org/wiki/Deep_belief_network), [recurrent neural networks](https://en.wikipedia.org/wiki/Recurrent_neural_network) and deep generative models. A deep neural network (DNN) is an ANN with multiple hidden layers between the input and output layers.

An ANN is based on a collection of connected units called artificial neurons, (analogous to axons in a biological brain). Each connection (synapse) between neurons can transmit a signal to another neuron. The receiving (postsynaptic) neuron can process the signal(s) and then signal downstream neurons connected to it.

_ Deep learning is basically the deep stacking of artificial neurons to learn complex models of data. _

## Why Deep Learning?

- It works

Deep learning and neural networks are increasingly important concepts as demonstrated through their performance on difficult problems in computer vision medical diagnosis, natural language processing and many other domains.

- Learns feature selection

Deep learning algorithms are unique in that they try to learn latent features from data, as opposed to traditional machine learning where features selection is typically handcrafted.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
from scipy import stats
import seaborn as sns
import warnings
import random
from datetime import datetime
random.seed(datetime.now())
warnings.filterwarnings('ignore')

# Make plots larger
plt.rcParams['figure.figsize'] = (10, 6)
```

```python
import keras
from keras.datasets import mnist
from keras.models import Sequential
from keras.layers import Dense
from keras.optimizers import SGD
```

    Using TensorFlow backend.

## MNIST data

The [MNIST database](http://yann.lecun.com/exdb/mnist/) of handwritten digits, available from this page, has a training set of 60,000 examples, and a test set of 10,000 examples. It is a subset of a larger set available from NIST. The digits have been size-normalized and centered in a fixed-size image.

It is a good database for people who want to try learning techniques and pattern recognition methods on real-world data while spending minimal efforts on preprocessing and formatting.

```python
# load the MNIST dataset
(X_train, y_train), (X_test, y_test) = mnist.load_data()
```

```python
# plot first 4 images as gray scale
plt.subplot(221)
plt.imshow(X_train[0], cmap=plt.get_cmap('gray'))
plt.subplot(222)
plt.imshow(X_train[1], cmap=plt.get_cmap('gray'))
plt.subplot(223)
plt.imshow(X_train[2], cmap=plt.get_cmap('gray'))
plt.subplot(224)
plt.imshow(X_train[3], cmap=plt.get_cmap('gray'))
# show the plot
plt.show()
```

The training dataset is structured as a 3-dimensional array of each image, and that images width and image height (28×28 pixels per image).

```python
## 60K 28×28 sized training images
print (X_train.shape)
```

    (60000, 28, 28)

```python
# plot 4 more images as gray scale
plt.subplot(221)
plt.imshow(X_train[55], cmap=plt.get_cmap('gray'))
plt.subplot(222)
plt.imshow(X_train[555], cmap=plt.get_cmap('gray'))
plt.subplot(223)
plt.imshow(X_train[5555], cmap=plt.get_cmap('gray'))
plt.subplot(224)
plt.imshow(X_train[55555], cmap=plt.get_cmap('gray'))
# show the plot
plt.show()
```

[But what *is* a Neural Network?](https://youtu.be/aircAruvnKk)

## The Perceptron (Neural units)

The [perceptron]()https://en.wikipedia.org/wiki/Perceptron is an algorithm for supervised learning of binary classifiers (functions that can decide
whether an input, represented by a vector of numbers, belongs to some
specific class or not). It is a type of linear classifier, i.e. a
classification algorithm that makes its predictions based on a linear
predictor function combining a set of weights with the feature
vector.

The perceptron algorithm dates back to the late 1950s, and is the basis of [artificial neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network).

Definition
----------

In the modern sense, the perceptron is an algorithm for learning a
binary classifier: a function that maps its input (a real-valued
[vector]) to an output value $f(x)$ (a single [binary] value):

$$f(x) = \begin{cases}1 & \text{if }\ w \cdot x + b > 0\\0 & \text{otherwise}\end{cases}$$

where is a vector of real-valued weights, $w \cdot x$ is the [dot
product] $\sum_{i=1}^m w_i x_i$, where m is the number of inputs to the
perceptron and is the *bias*. The bias shifts the decision boundary away
from the origin and does not depend on any input value.

The value of $f(x)$ (0 or 1) is used to classify as either a positive or
a negative instance, in the case of a binary classification problem. If
is negative, then the weighted combination of inputs must produce a
positive value greater than $|b|$ in order to push the classifier neuron
over the 0 threshold. Spatially, the bias alters the position (though
not the orientation) of the decision boundary. The perceptron learning
algorithm does not terminate if the learning set is not linearly
separable. If the vectors are not linearly separable learning will
never reach a point where all vectors are classified properly. The most
famous example of the perceptron's inability to solve problems with
linearly nonseparable vectors is the Boolean exclusive-or problem. The
solution spaces of decision boundaries for all binary functions and
learning behaviors are studied in the reference.

In the context of neural networks, a perceptron is an _artificial
neuron_ using the [Heaviside step function](https://en.wikipedia.org/wiki/Heaviside_step_function) as the activation function.
The perceptron algorithm is also termed the **single-layer perceptron**,
to distinguish it from a multilayer perceptron, which is a misnomer
for a more complicated neural network. As a linear classifier, the
single-layer perceptron is the simplest [feedforward neural network](https://en.wikipedia.org/wiki/Feedforward_neural_network).

Learning algorithm
------------------

Below is an example of a learning algorithm for a (single-layer)
perceptron. For [multilayer perceptrons], where a hidden layer exists,
more sophisticated algorithms such as [backpropagation] must be used.
Alternatively, methods such as the [delta rule] can be used if the
function is non-linear and differentiable, although the one below will
work as well.

When multiple perceptrons are combined in an artificial neural network,
each output neuron operates independently of all the others; thus,
learning each output can be considered in isolation.

### Definitions

We first define some variables:

-   $y = f(\mathbf{z})$ denotes the *output* from the perceptron for an
    input vector $\mathbf{z}$.
-   $D = \{(\mathbf{x}_1,d_1),\dots,(\mathbf{x}_s,d_s)\}$ is the
    *training set* of $s$ samples, where:
    -   $\mathbf{x}_j$ is the $n$-dimensional input vector.
    -   $d_j$ is the desired output value of the perceptron for that
        input.

We show the values of the features as follows:

-   $x_{j,i}$ is the value of the $i$th feature of the $j$th training
    *input vector*.
-   $x_{j,0} = 1$.

To represent the weights:

-   $w_i$ is the $i$th value in the *weight vector*, to be multiplied by
    the value of the $i$th input feature.
-   Because $x_{j,0} = 1$, the $w_0$ is effectively a bias that we use
    instead of the bias constant $b$.

To show the time-dependence of $\mathbf{w}$, we use:

-   $w_i(t)$ is the weight $i$ at time $t$.

Unlike other linear classification algorithms such as [logistic
regression], there is no need for a *learning rate* in the perceptron
algorithm. This is because multiplying the update by any constant simply
rescales the weights but never changes the sign of the prediction.

### Steps

1. Initialize the weights and the threshold. Weights may
be initialized to 0 or to a small random value. In the example below, we
use 0.
2. For each example in our training set , perform the following
steps over the input $\mathbf{x}_j$ and desired output $d_j$

Calculate the actual output:

a. $$\begin{align}
y_j(t) &= f[\mathbf{w}(t)\cdot\mathbf{x}_j] \\
&= f[w_0(t)x_{j,0} + w_1(t)x_{j,1} + w_2(t)x_{j,2} + \dotsb + w_n(t)x_{j,n}]
\end{align}$$

b. Update the weights:

$$w_i(t+1) = w_i(t) + (d_j - y_j(t)) x_{j,i}$$, for all features
$0 \leq i \leq n$.

**offline learning**

For offline learning, the step 2 may be repeated until the iteration error  $\frac{1}{s} \sum_{j=1}^s |d_j - y_j(t)|$ is less than a user-specified error threshold $\gamma$, or a predetermined number of iterations have been completed. The algorithm updates the weights after steps 2a and 2b. These weights are immediately applied to a pair in the training set, and subsequently updated, rather than waiting until all pairs in the training set have undergone these steps.

Multiclass perceptron
---------------------

Like most other techniques for training linear classifiers, the
perceptron generalizes naturally to multiclass classification. Here,
the input $x$ and the output $y$ are drawn from arbitrary sets. A
feature representation function $f(x,y)$ maps each possible input/output
pair to a finite-dimensional real-valued feature vector. As before, the
feature vector is multiplied by a weight vector $w$, but now the
resulting score is used to choose among many possible outputs:

$$\hat y = \operatorname{argmax}_y f(x,y) \cdot w.$$ ≈ Learning again
iterates over the examples, predicting an output for each, leaving the
weights unchanged when the predicted output matches the target, and
changing them when it does not. The update becomes:

$$w_{t+1} = w_t + f(x, y) - f(x,\hat y).$$

This multiclass feedback formulation reduces to the original perceptron
when $x$ is a real-valued vector, $y$ is chosen from $\{0,1\}$, and
$f(x,y) = y x$.

For certain problems, input/output representations and features can be
chosen so that $\mathrm{argmax}_y f(x,y) \cdot w$ can be found
efficiently even though $y$ is chosen from a very large or even infinite
set.

In recent years, perceptron training has become popular in the field of
[natural language processing](https://en.wikipedia.org/wiki/Natural_language_processing) for such tasks as [part-of-speech tagging](https://en.wikipedia.org/wiki/Part-of-speech_tagging)
and [syntactic parsing](https://en.wikipedia.org/wiki/Parsing).

## Artificial neural network

[Artificial neural networks](https://en.wikipedia.org/wiki/Artificial_neural_network) (**ANNs**) or **[connectionist] systems**
are computing systems inspired by the biological neural networks that
constitute animal brains. Such systems learn (progressively improve
performance) to do tasks by considering examples, generally without
task-specific programming. For example, in image recognition, they might
learn to identify images that contain cats by analyzing example images
that have been manually labeled as “cat” or “no cat” and using the
analytic results to identify cats in other images. They have found most
use in applications difficult to express in a traditional computer
algorithm using rule-based programming.

An ANN is based on a collection of connected units called artificial
neurons, (analogous to axons in a biological brain). Each
connection synapse) between neurons can transmit a signal to another
neuron. The receiving (postsynaptic) neuron can process the signal(s)
and then signal downstream neurons connected to it. Neurons may have
state, generally represented by [real numbers], typically between 0 and 1.

Neurons and synapses may also have a weight that varies as learning
proceeds, which can increase or decrease the strength of the signal that
it sends downstream. Further, they may have a threshold such that only
if the aggregate signal is below (or above) that level is the downstream
signal sent.

Typically, neurons are organized in layers. Different layers may perform
different kinds of transformations on their inputs. Signals travel from
the first (input), to the last (output) layer, possibly after traversing
the layers multiple times.

The original goal of the neural network approach was to solve problems
in the same way that a human brain would. Over time, attention focused
on matching specific mental abilities, leading to deviations from
biology such as backpropagation, or passing information in the reverse
direction and adjusting the network to reflect that information.

Neural networks have been used on a variety of tasks, including
computer vision, speech recognition, machine translation, social
network filtering, playing board and video games, medical diagnosis and
in many other domains.

## Backpropagation Algorithm

**Backpropagation** is a method used in artificial neural networks to
calculate a gradient that is needed in the calculation of the
weights to be used in the network. It is commonly used to train
deep neural networks, a term referring to neural networks with
more than one hidden layer.

Backpropagation is a special case of an older and more general technique
called automatic differentiation. In the context of learning,
backpropagation is commonly used by the gradient descent optimization
algorithm to adjust the weight of neurons by calculating the gradient
of the loss function. This technique is also sometimes called
**backward propagation of errors**, because the error is calculated at
the output and distributed back through the network layers.

[What is backpropagation really doing?](https://youtu.be/Ilg3gGewQ5U)

[Gradient descent, how neural networks learn?](https://youtu.be/IHZwWFHWa-w)

[Backpropagation calculus](https://youtu.be/tIeHLnjs5U8)

**Loss function**

Sometimes referred to as the **cost function** or **error function**
(not to be confused with the Gauss error function), the loss function
is a function that maps values of one or more variables onto a real
number intuitively representing some \"cost\" associated with those
values. For backpropagation, the loss function calculates the difference
between the network output and its expected output, after a case
propagates through the network.

### Assumptions

Two assumptions must be made about the form of the error function.^1
The first is that it can be written as an average
$E=\frac{1}{n}\sum_xE_x$ over error functions $E_x$, for $n$ individual
training examples, $x$. The reason for this assumption is that the
backpropagation algorithm calculates the gradient of the error function
for a single training example, which needs to be generalized to the
overall error function. The second assumption is that it can be written
as a function of the outputs from the neural network.

### Example loss function

Let $y,y'$ be vectors in $\mathbb{R}^n$.

Select an error function $E(y,y')$ measuring the difference between two
outputs. The standard choice is the square of the Euclidean distance
between the vectors $y$ and $y'$:

$E(y,y') = \tfrac{1}{2} \lVert y-y'\rVert^2$

Note that the factor of $\tfrac{1}{2}$ conveniently cancels the exponent
when the error function is subsequently differentiated.

The error function over $n$ training examples can simply be written as
an average of losses over individual
examples$$E=\frac{1}{2n}\sum_x\lVert (y(x)-y'(x)) \rVert^2$$

and therefore, the partial derivative with respect to the
outputs$$\frac{\partial E}{\partial y'} = y'-y$$

**Algorithm**

Let $N$ be a neural network with $e$ connections, $m$ inputs, and $n$
outputs.

Below, $x_1,x_2,\dots$ will denote vectors in $\mathbb{R}^m$,
$y_1,y_2,\dots$ vectors in $\mathbb{R}^n$, and $w_0, w_1, w_2, \ldots$
vectors in $\mathbb{R}^e$. These are called *inputs*, *outputs* and
*weights* respectively.

The neural network corresponds to a function $y = f_N(w, x)$ which,
given a weight $w$, maps an input $x$ to an output $y$.

The optimization takes as input a sequence of *training examples*
$(x_1,y_1), \dots, (x_p, y_p)$ and produces a sequence of weights
$w_0, w_1, \dots, w_p$ starting from some initial weight $w_0$, usually
chosen at random.

These weights are computed in turn: first compute $w_i$ using only
$(x_i, y_i, w_{i-1})$ for $i = 1, \dots, p$. The output of the algorithm
is then $w_p$, giving us a new function $x \mapsto f_N(w_p, x)$. The
computation is the same in each step, hence only the case $i = 1$ is
described.

Calculating $w_1$ from $(x_1, y_1, w_0)$ is done by considering a
variable weight $w$ and applying gradient descent to the function
$w\mapsto E(f_N(w, x_1), y_1)$ to find a local minimum, starting at
$w = w_0$.

This makes $w_1$ the minimizing weight found by gradient descent.

**Algorithm in code**

To implement the algorithm above, explicit formulas are required for the
gradient of the function $w \mapsto E(f_N(w, x), y)$ where the function
is $E(y,y')= |y-y'|^2$.

The learning algorithm can be divided into two phases: propagation and
weight update.

### Phase 1: propagation

Each propagation involves the following steps:

1.  Propagation forward through the network to generate the output
    value(s)
2.  Calculation of the cost (error term)
3.  Propagation of the output activations back through the network using
    the training pattern target in order to generate the deltas (the
    difference between the targeted and actual output values) of all
    output and hidden neurons.

### Phase 2: weight update

For each weight, the following steps must be followed:

1.  The weight\'s output delta and input activation are multiplied to
    find the gradient of the weight.
2.  A ratio (percentage) of the weight\'s gradient is subtracted from
    the weight.

This ratio (percentage) influences the speed and quality of learning; it
is called the *learning rate*. The greater the ratio, the faster the
neuron trains, but the lower the ratio, the more accurate the training
is. The sign of the gradient of a weight indicates whether the error
varies directly with, or inversely to, the weight. Therefore, the weight
must be updated in the opposite direction, \"descending\" the gradient.

Learning is repeated (on new batches) until the network performs
adequately.

### Pseudocode

The following is pseudocode for a stochastic gradient descent
algorithm for training a three-layer network (only one hidden layer):

```python
  initialize network weights (often small random values)\
  **do**\
     **forEach** training example named ex\
        prediction = _neural-net-output_(network, ex)  *// forward pass*\
        actual = _teacher-output_(ex)\
        compute error (prediction - actual) at the output units\
          *// backward pass*\
           *// backward pass continued*\
        update network weights *// input layer not modified by error estimate*\
  **until** all examples classified correctly or another stopping criterion satisfied\
  **return** the network
```

The lines labeled \"backward pass\" can be implemented using the
backpropagation algorithm, which calculates the gradient of the error of
the network regarding the network\'s modifiable weights.

## How many input neurons?

What is the input layer for the MNIST data?  Each image is $28x28 = 784$.

```python
## 10K 28×28 sized test images
X_test.shape
```

    (10000, 28, 28)

```python
28*28
```

    784

For a multi-layer perceptron model we must reduce the images down into a a single input layer as a vector of pixels. In this case the 28×28 sized images will be 784 pixel input values (28x28=784).

```python
X_train = X_train.reshape(60000, 784).astype('float32')
X_test = X_test.reshape(10000, 784).astype('float32')
```

The pixel values are gray scale between 0 and 255. Scaling of input values when using neural network models is a good idea. Neural network models propagate values and the rate of propagation can be effected by their scale.

```python
# Scale the data between 0 and 1

X_train /= 255.0
X_test /= 255.0
```

## Shallow Neural Network

A shallow neural network has few layers (just one dense layer in this case). Dense means every neuron connected to every other.

```python
X_test[0]
```

    array([ 0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.32941177,  0.72549021,  0.62352943,
            0.59215689,  0.23529412,  0.14117648,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.87058824,  0.99607843,  0.99607843,  0.99607843,  0.99607843,
            0.94509804,  0.7764706 ,  0.7764706 ,  0.7764706 ,  0.7764706 ,
            0.7764706 ,  0.7764706 ,  0.7764706 ,  0.7764706 ,  0.66666669,
            0.20392157,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.26274511,  0.44705883,
            0.28235295,  0.44705883,  0.63921571,  0.89019608,  0.99607843,
            0.88235295,  0.99607843,  0.99607843,  0.99607843,  0.98039216,
            0.89803922,  0.99607843,  0.99607843,  0.54901963,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.06666667,  0.25882354,  0.05490196,  0.26274511,
            0.26274511,  0.26274511,  0.23137255,  0.08235294,  0.9254902 ,
            0.99607843,  0.41568628,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.32549021,  0.99215686,  0.81960785,  0.07058824,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.08627451,  0.9137255 ,
            1.        ,  0.32549021,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.50588238,  0.99607843,  0.93333334,  0.17254902,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.23137255,  0.97647059,
            0.99607843,  0.24313726,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.52156866,  0.99607843,  0.73333335,  0.01960784,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.03529412,  0.80392158,
            0.97254902,  0.22745098,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.49411765,  0.99607843,  0.71372551,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.29411766,  0.98431373,
            0.94117647,  0.22352941,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.07450981,  0.86666667,  0.99607843,  0.65098041,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.01176471,  0.79607844,  0.99607843,
            0.85882354,  0.13725491,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.14901961,  0.99607843,  0.99607843,  0.3019608 ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.12156863,  0.87843138,  0.99607843,
            0.4509804 ,  0.00392157,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.52156866,  0.99607843,  0.99607843,  0.20392157,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.23921569,  0.94901961,  0.99607843,
            0.99607843,  0.20392157,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.47450981,  0.99607843,  0.99607843,  0.85882354,  0.15686275,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.47450981,  0.99607843,
            0.81176472,  0.07058824,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ,  0.        ,
            0.        ,  0.        ,  0.        ,  0.        ], dtype=float32)

Finally, the output variable is an integer from 0 to 9. This is a multi-class classification problem. As such, it is good practice to use a one hot encoding of the class values, transforming the vector of class integers into a binary matrix.

We can easily do this using the built-in np_utils.to_categorical() helper function in Keras.

```python
n_classes = 10
y_train = keras.utils.to_categorical(y_train, n_classes)
y_test = keras.utils.to_categorical(y_test, n_classes)
```

```python
y_test[0:5]
```

    array([[ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.],
           [ 0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.]])

```python
y_train[0:5]
```

    array([[ 0.,  0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.],
           [ 1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  1.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  1.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.],
           [ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  1.]])

```python
def shallow_net_A(n=55,i=784,o=10):
    # create simple one dense layer net
    # default 55 neurons, input 784, output 10
    net = Sequential()
    net.add(Dense(n, activation='sigmoid', input_shape=(i,)))
    net.add(Dense(10, activation='softmax'))
    # Compile net
    net.compile(loss='mean_squared_error', optimizer=SGD(lr=0.01), metrics=['accuracy'])
    return net
```

```python
nn=shallow_net_A()
```

```python
nn.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_1 (Dense)              (None, 55)                43175
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                560
    =================================================================
    Total params: 43,735
    Trainable params: 43,735
    Non-trainable params: 0
    _________________________________________________________________

```python
nn.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0943 - acc: 0.0993 - val_loss: 0.0932 - val_acc: 0.1032
    Epoch 2/99
    60000/60000 [==============================] - 0s - loss: 0.0927 - acc: 0.0984 - val_loss: 0.0922 - val_acc: 0.0985
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0920 - acc: 0.0984 - val_loss: 0.0916 - val_acc: 0.1011
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0914 - acc: 0.1155 - val_loss: 0.0911 - val_acc: 0.1264
    Epoch 5/99
    60000/60000 [==============================] - 0s - loss: 0.0909 - acc: 0.1447 - val_loss: 0.0907 - val_acc: 0.1545
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0905 - acc: 0.1690 - val_loss: 0.0903 - val_acc: 0.1787
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0901 - acc: 0.1906 - val_loss: 0.0899 - val_acc: 0.2001
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.0898 - acc: 0.2121 - val_loss: 0.0896 - val_acc: 0.2239
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0894 - acc: 0.2330 - val_loss: 0.0892 - val_acc: 0.2448
    Epoch 10/99
    60000/60000 [==============================] - 1s - loss: 0.0890 - acc: 0.2534 - val_loss: 0.0888 - val_acc: 0.2611
    Epoch 11/99
    60000/60000 [==============================] - 1s - loss: 0.0887 - acc: 0.2707 - val_loss: 0.0885 - val_acc: 0.2768
    Epoch 12/99
    60000/60000 [==============================] - 1s - loss: 0.0884 - acc: 0.2850 - val_loss: 0.0881 - val_acc: 0.2915
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.0880 - acc: 0.2965 - val_loss: 0.0878 - val_acc: 0.3022
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.0877 - acc: 0.3064 - val_loss: 0.0875 - val_acc: 0.3119
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0873 - acc: 0.3137 - val_loss: 0.0871 - val_acc: 0.3204
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.0870 - acc: 0.3232 - val_loss: 0.0868 - val_acc: 0.3300
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0867 - acc: 0.3309 - val_loss: 0.0864 - val_acc: 0.3370
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0863 - acc: 0.3379 - val_loss: 0.0861 - val_acc: 0.3431
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0860 - acc: 0.3438 - val_loss: 0.0857 - val_acc: 0.3479
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0856 - acc: 0.3500 - val_loss: 0.0854 - val_acc: 0.3552
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0853 - acc: 0.3551 - val_loss: 0.0850 - val_acc: 0.3597
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0849 - acc: 0.3593 - val_loss: 0.0847 - val_acc: 0.3642
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0846 - acc: 0.3639 - val_loss: 0.0843 - val_acc: 0.3699
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0842 - acc: 0.3689 - val_loss: 0.0839 - val_acc: 0.3746
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0839 - acc: 0.3733 - val_loss: 0.0836 - val_acc: 0.3790
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0835 - acc: 0.3801 - val_loss: 0.0832 - val_acc: 0.3843
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0831 - acc: 0.3853 - val_loss: 0.0828 - val_acc: 0.3897
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0827 - acc: 0.3910 - val_loss: 0.0824 - val_acc: 0.3968
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0823 - acc: 0.3987 - val_loss: 0.0820 - val_acc: 0.4045
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0819 - acc: 0.4060 - val_loss: 0.0816 - val_acc: 0.4113
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0815 - acc: 0.4127 - val_loss: 0.0811 - val_acc: 0.4189
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0811 - acc: 0.4193 - val_loss: 0.0807 - val_acc: 0.4249
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0807 - acc: 0.4261 - val_loss: 0.0803 - val_acc: 0.4330
    Epoch 34/99
    60000/60000 [==============================] - 1s - loss: 0.0803 - acc: 0.4326 - val_loss: 0.0799 - val_acc: 0.4392
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0798 - acc: 0.4388 - val_loss: 0.0794 - val_acc: 0.4450
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0794 - acc: 0.4454 - val_loss: 0.0790 - val_acc: 0.4509
    Epoch 37/99
    60000/60000 [==============================] - 1s - loss: 0.0789 - acc: 0.4510 - val_loss: 0.0785 - val_acc: 0.4580
    Epoch 38/99
    60000/60000 [==============================] - 1s - loss: 0.0785 - acc: 0.4570 - val_loss: 0.0781 - val_acc: 0.4633
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0780 - acc: 0.4619 - val_loss: 0.0776 - val_acc: 0.4680
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0776 - acc: 0.4671 - val_loss: 0.0771 - val_acc: 0.4735
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0771 - acc: 0.4727 - val_loss: 0.0767 - val_acc: 0.4795
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0767 - acc: 0.4785 - val_loss: 0.0762 - val_acc: 0.4847
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0762 - acc: 0.4833 - val_loss: 0.0757 - val_acc: 0.4888
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0758 - acc: 0.4880 - val_loss: 0.0752 - val_acc: 0.4934
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0753 - acc: 0.4932 - val_loss: 0.0748 - val_acc: 0.4980
    Epoch 46/99
    60000/60000 [==============================] - 1s - loss: 0.0748 - acc: 0.4972 - val_loss: 0.0743 - val_acc: 0.5021
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0744 - acc: 0.5029 - val_loss: 0.0738 - val_acc: 0.5059
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0739 - acc: 0.5077 - val_loss: 0.0733 - val_acc: 0.5106
    Epoch 49/99
    60000/60000 [==============================] - 1s - loss: 0.0734 - acc: 0.5124 - val_loss: 0.0729 - val_acc: 0.5144
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0730 - acc: 0.5175 - val_loss: 0.0724 - val_acc: 0.5197
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0725 - acc: 0.5213 - val_loss: 0.0719 - val_acc: 0.5245
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0720 - acc: 0.5255 - val_loss: 0.0714 - val_acc: 0.5291
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0716 - acc: 0.5306 - val_loss: 0.0710 - val_acc: 0.5340
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0711 - acc: 0.5351 - val_loss: 0.0705 - val_acc: 0.5387
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0706 - acc: 0.5397 - val_loss: 0.0700 - val_acc: 0.5434
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.0702 - acc: 0.5443 - val_loss: 0.0696 - val_acc: 0.5477
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0697 - acc: 0.5483 - val_loss: 0.0691 - val_acc: 0.5536
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0692 - acc: 0.5538 - val_loss: 0.0686 - val_acc: 0.5575
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0688 - acc: 0.5587 - val_loss: 0.0681 - val_acc: 0.5627
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0683 - acc: 0.5640 - val_loss: 0.0677 - val_acc: 0.5675
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0679 - acc: 0.5689 - val_loss: 0.0672 - val_acc: 0.5714
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0674 - acc: 0.5742 - val_loss: 0.0668 - val_acc: 0.5772
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0670 - acc: 0.5795 - val_loss: 0.0663 - val_acc: 0.5818
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0665 - acc: 0.5849 - val_loss: 0.0658 - val_acc: 0.5864
    Epoch 65/99
    60000/60000 [==============================] - 1s - loss: 0.0661 - acc: 0.5898 - val_loss: 0.0654 - val_acc: 0.5922
    Epoch 66/99
    60000/60000 [==============================] - 1s - loss: 0.0656 - acc: 0.5940 - val_loss: 0.0649 - val_acc: 0.5957
    Epoch 67/99
    60000/60000 [==============================] - 1s - loss: 0.0652 - acc: 0.5991 - val_loss: 0.0645 - val_acc: 0.6008
    Epoch 68/99
    60000/60000 [==============================] - 1s - loss: 0.0647 - acc: 0.6039 - val_loss: 0.0640 - val_acc: 0.6062
    Epoch 69/99
    60000/60000 [==============================] - 1s - loss: 0.0643 - acc: 0.6085 - val_loss: 0.0636 - val_acc: 0.6107
    Epoch 70/99
    60000/60000 [==============================] - 1s - loss: 0.0638 - acc: 0.6137 - val_loss: 0.0631 - val_acc: 0.6152
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0634 - acc: 0.6177 - val_loss: 0.0627 - val_acc: 0.6200
    Epoch 72/99
    60000/60000 [==============================] - 1s - loss: 0.0629 - acc: 0.6223 - val_loss: 0.0622 - val_acc: 0.6244
    Epoch 73/99
    60000/60000 [==============================] - 1s - loss: 0.0625 - acc: 0.6260 - val_loss: 0.0618 - val_acc: 0.6276
    Epoch 74/99
    60000/60000 [==============================] - 1s - loss: 0.0621 - acc: 0.6306 - val_loss: 0.0613 - val_acc: 0.6324
    Epoch 75/99
    60000/60000 [==============================] - 1s - loss: 0.0616 - acc: 0.6351 - val_loss: 0.0609 - val_acc: 0.6367
    Epoch 76/99
    60000/60000 [==============================] - 1s - loss: 0.0612 - acc: 0.6390 - val_loss: 0.0605 - val_acc: 0.6422
    Epoch 77/99
    60000/60000 [==============================] - 1s - loss: 0.0608 - acc: 0.6435 - val_loss: 0.0600 - val_acc: 0.6458
    Epoch 78/99
    60000/60000 [==============================] - 1s - loss: 0.0603 - acc: 0.6468 - val_loss: 0.0596 - val_acc: 0.6505
    Epoch 79/99
    60000/60000 [==============================] - 1s - loss: 0.0599 - acc: 0.6506 - val_loss: 0.0592 - val_acc: 0.6534
    Epoch 80/99
    60000/60000 [==============================] - 1s - loss: 0.0595 - acc: 0.6537 - val_loss: 0.0587 - val_acc: 0.6576
    Epoch 81/99
    60000/60000 [==============================] - 1s - loss: 0.0591 - acc: 0.6569 - val_loss: 0.0583 - val_acc: 0.6624
    Epoch 82/99
    60000/60000 [==============================] - 1s - loss: 0.0587 - acc: 0.6606 - val_loss: 0.0579 - val_acc: 0.6652
    Epoch 83/99
    60000/60000 [==============================] - 1s - loss: 0.0582 - acc: 0.6639 - val_loss: 0.0575 - val_acc: 0.6691
    Epoch 84/99
    60000/60000 [==============================] - 1s - loss: 0.0578 - acc: 0.6669 - val_loss: 0.0571 - val_acc: 0.6724
    Epoch 85/99
    60000/60000 [==============================] - 1s - loss: 0.0574 - acc: 0.6707 - val_loss: 0.0566 - val_acc: 0.6768
    Epoch 86/99
    60000/60000 [==============================] - 1s - loss: 0.0570 - acc: 0.6740 - val_loss: 0.0562 - val_acc: 0.6804
    Epoch 87/99
    60000/60000 [==============================] - 1s - loss: 0.0566 - acc: 0.6764 - val_loss: 0.0558 - val_acc: 0.6840
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.0562 - acc: 0.6801 - val_loss: 0.0554 - val_acc: 0.6877
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.0558 - acc: 0.6832 - val_loss: 0.0550 - val_acc: 0.6912
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0554 - acc: 0.6853 - val_loss: 0.0546 - val_acc: 0.6941
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.0550 - acc: 0.6883 - val_loss: 0.0542 - val_acc: 0.6977
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0546 - acc: 0.6904 - val_loss: 0.0538 - val_acc: 0.7018
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0542 - acc: 0.6930 - val_loss: 0.0534 - val_acc: 0.7040
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0539 - acc: 0.6957 - val_loss: 0.0531 - val_acc: 0.7073
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0535 - acc: 0.6986 - val_loss: 0.0527 - val_acc: 0.7105
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0531 - acc: 0.7017 - val_loss: 0.0523 - val_acc: 0.7124
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0527 - acc: 0.7038 - val_loss: 0.0519 - val_acc: 0.7155
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0524 - acc: 0.7056 - val_loss: 0.0516 - val_acc: 0.7179
    Epoch 99/99
    60000/60000 [==============================] - 0s - loss: 0.0520 - acc: 0.7085 - val_loss: 0.0512 - val_acc: 0.7199

    <keras.callbacks.History at 0x1226252b0>

```python
# 68% accuracy after 99 epochs
nn.evaluate(X_test, y_test)
```

     8608/10000 [========================>.....] - ETA: 0s

    [0.051187293976545332, 0.71989999999999998]

```python
nn.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 0s - loss: 0.0516 - acc: 0.7109 - val_loss: 0.0508 - val_acc: 0.7233
    Epoch 2/99
    60000/60000 [==============================] - 0s - loss: 0.0513 - acc: 0.7132 - val_loss: 0.0505 - val_acc: 0.7261
    Epoch 3/99
    60000/60000 [==============================] - 0s - loss: 0.0509 - acc: 0.7160 - val_loss: 0.0501 - val_acc: 0.7283
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0506 - acc: 0.7184 - val_loss: 0.0498 - val_acc: 0.7298
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0502 - acc: 0.7204 - val_loss: 0.0494 - val_acc: 0.7319
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0499 - acc: 0.7231 - val_loss: 0.0491 - val_acc: 0.7344
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0495 - acc: 0.7254 - val_loss: 0.0487 - val_acc: 0.7378
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.0492 - acc: 0.7279 - val_loss: 0.0484 - val_acc: 0.7405
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0489 - acc: 0.7300 - val_loss: 0.0480 - val_acc: 0.7425
    Epoch 10/99
    60000/60000 [==============================] - 1s - loss: 0.0485 - acc: 0.7323 - val_loss: 0.0477 - val_acc: 0.7451
    Epoch 11/99
    60000/60000 [==============================] - 1s - loss: 0.0482 - acc: 0.7345 - val_loss: 0.0474 - val_acc: 0.7463
    Epoch 12/99
    60000/60000 [==============================] - 1s - loss: 0.0479 - acc: 0.7366 - val_loss: 0.0471 - val_acc: 0.7486
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.0476 - acc: 0.7384 - val_loss: 0.0467 - val_acc: 0.7508
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.0472 - acc: 0.7409 - val_loss: 0.0464 - val_acc: 0.7532
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0469 - acc: 0.7432 - val_loss: 0.0461 - val_acc: 0.7562
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.0466 - acc: 0.7453 - val_loss: 0.0458 - val_acc: 0.7585
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0463 - acc: 0.7474 - val_loss: 0.0455 - val_acc: 0.7606
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0460 - acc: 0.7494 - val_loss: 0.0452 - val_acc: 0.7631
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0457 - acc: 0.7514 - val_loss: 0.0449 - val_acc: 0.7658
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0454 - acc: 0.7537 - val_loss: 0.0446 - val_acc: 0.7679
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0451 - acc: 0.7563 - val_loss: 0.0443 - val_acc: 0.7707
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0448 - acc: 0.7580 - val_loss: 0.0440 - val_acc: 0.7725
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0446 - acc: 0.7601 - val_loss: 0.0437 - val_acc: 0.7754
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0443 - acc: 0.7623 - val_loss: 0.0434 - val_acc: 0.7774
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0440 - acc: 0.7643 - val_loss: 0.0431 - val_acc: 0.7803
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0437 - acc: 0.7671 - val_loss: 0.0429 - val_acc: 0.7818
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0434 - acc: 0.7688 - val_loss: 0.0426 - val_acc: 0.7835
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0432 - acc: 0.7716 - val_loss: 0.0423 - val_acc: 0.7852
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0429 - acc: 0.7739 - val_loss: 0.0420 - val_acc: 0.7872
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0426 - acc: 0.7758 - val_loss: 0.0418 - val_acc: 0.7900
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0424 - acc: 0.7776 - val_loss: 0.0415 - val_acc: 0.7920
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0421 - acc: 0.7802 - val_loss: 0.0412 - val_acc: 0.7942
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0419 - acc: 0.7823 - val_loss: 0.0410 - val_acc: 0.7957
    Epoch 34/99
    60000/60000 [==============================] - 1s - loss: 0.0416 - acc: 0.7845 - val_loss: 0.0407 - val_acc: 0.7980
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0413 - acc: 0.7866 - val_loss: 0.0405 - val_acc: 0.7995
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0411 - acc: 0.7880 - val_loss: 0.0402 - val_acc: 0.8014
    Epoch 37/99
    60000/60000 [==============================] - 1s - loss: 0.0409 - acc: 0.7900 - val_loss: 0.0400 - val_acc: 0.8042
    Epoch 38/99
    60000/60000 [==============================] - 1s - loss: 0.0406 - acc: 0.7918 - val_loss: 0.0397 - val_acc: 0.8059
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0404 - acc: 0.7934 - val_loss: 0.0395 - val_acc: 0.8083
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0401 - acc: 0.7952 - val_loss: 0.0392 - val_acc: 0.8097
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0399 - acc: 0.7970 - val_loss: 0.0390 - val_acc: 0.8113
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0397 - acc: 0.7983 - val_loss: 0.0388 - val_acc: 0.8129
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0394 - acc: 0.8000 - val_loss: 0.0385 - val_acc: 0.8147
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0392 - acc: 0.8018 - val_loss: 0.0383 - val_acc: 0.8162
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0390 - acc: 0.8034 - val_loss: 0.0381 - val_acc: 0.8176
    Epoch 46/99
    60000/60000 [==============================] - 1s - loss: 0.0387 - acc: 0.8049 - val_loss: 0.0379 - val_acc: 0.8192
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0385 - acc: 0.8064 - val_loss: 0.0376 - val_acc: 0.8209
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0383 - acc: 0.8080 - val_loss: 0.0374 - val_acc: 0.8226
    Epoch 49/99
    60000/60000 [==============================] - 1s - loss: 0.0381 - acc: 0.8096 - val_loss: 0.0372 - val_acc: 0.8241
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0379 - acc: 0.8112 - val_loss: 0.0370 - val_acc: 0.8249
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0377 - acc: 0.8126 - val_loss: 0.0368 - val_acc: 0.8266
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0375 - acc: 0.8137 - val_loss: 0.0365 - val_acc: 0.8274
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0372 - acc: 0.8151 - val_loss: 0.0363 - val_acc: 0.8282
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0370 - acc: 0.8162 - val_loss: 0.0361 - val_acc: 0.8293
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0368 - acc: 0.8176 - val_loss: 0.0359 - val_acc: 0.8300
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.0366 - acc: 0.8185 - val_loss: 0.0357 - val_acc: 0.8309
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0364 - acc: 0.8199 - val_loss: 0.0355 - val_acc: 0.8321
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0362 - acc: 0.8209 - val_loss: 0.0353 - val_acc: 0.8331
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0361 - acc: 0.8224 - val_loss: 0.0351 - val_acc: 0.8340
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0359 - acc: 0.8234 - val_loss: 0.0349 - val_acc: 0.8351
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0357 - acc: 0.8246 - val_loss: 0.0347 - val_acc: 0.8360
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0355 - acc: 0.8258 - val_loss: 0.0346 - val_acc: 0.8374
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0353 - acc: 0.8269 - val_loss: 0.0344 - val_acc: 0.8379
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0351 - acc: 0.8283 - val_loss: 0.0342 - val_acc: 0.8391
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0349 - acc: 0.8294 - val_loss: 0.0340 - val_acc: 0.8396
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0348 - acc: 0.8302 - val_loss: 0.0338 - val_acc: 0.8402
    Epoch 67/99
    60000/60000 [==============================] - 1s - loss: 0.0346 - acc: 0.8309 - val_loss: 0.0336 - val_acc: 0.8414
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0344 - acc: 0.8320 - val_loss: 0.0335 - val_acc: 0.8428
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0342 - acc: 0.8328 - val_loss: 0.0333 - val_acc: 0.8435
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0341 - acc: 0.8337 - val_loss: 0.0331 - val_acc: 0.8442
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0339 - acc: 0.8346 - val_loss: 0.0329 - val_acc: 0.8450
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0337 - acc: 0.8357 - val_loss: 0.0328 - val_acc: 0.8457
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0335 - acc: 0.8362 - val_loss: 0.0326 - val_acc: 0.8477
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0334 - acc: 0.8370 - val_loss: 0.0324 - val_acc: 0.8486
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0332 - acc: 0.8378 - val_loss: 0.0323 - val_acc: 0.8490
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0331 - acc: 0.8385 - val_loss: 0.0321 - val_acc: 0.8498
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0329 - acc: 0.8388 - val_loss: 0.0320 - val_acc: 0.8503
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0327 - acc: 0.8398 - val_loss: 0.0318 - val_acc: 0.8505
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0326 - acc: 0.8404 - val_loss: 0.0316 - val_acc: 0.8506
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0324 - acc: 0.8413 - val_loss: 0.0315 - val_acc: 0.8509
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0323 - acc: 0.8416 - val_loss: 0.0313 - val_acc: 0.8516
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0321 - acc: 0.8425 - val_loss: 0.0312 - val_acc: 0.8522
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0320 - acc: 0.8433 - val_loss: 0.0310 - val_acc: 0.8524
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0318 - acc: 0.8439 - val_loss: 0.0309 - val_acc: 0.8528
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0317 - acc: 0.8444 - val_loss: 0.0307 - val_acc: 0.8534
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0315 - acc: 0.8453 - val_loss: 0.0306 - val_acc: 0.8540
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0314 - acc: 0.8457 - val_loss: 0.0305 - val_acc: 0.8547
    Epoch 88/99
    60000/60000 [==============================] - 0s - loss: 0.0313 - acc: 0.8465 - val_loss: 0.0303 - val_acc: 0.8556
    Epoch 89/99
    60000/60000 [==============================] - 0s - loss: 0.0311 - acc: 0.8471 - val_loss: 0.0302 - val_acc: 0.8562
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0310 - acc: 0.8475 - val_loss: 0.0300 - val_acc: 0.8568
    Epoch 91/99
    60000/60000 [==============================] - 0s - loss: 0.0309 - acc: 0.8481 - val_loss: 0.0299 - val_acc: 0.8572
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0307 - acc: 0.8484 - val_loss: 0.0298 - val_acc: 0.8579
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0306 - acc: 0.8487 - val_loss: 0.0296 - val_acc: 0.8581
    Epoch 94/99
    60000/60000 [==============================] - 0s - loss: 0.0305 - acc: 0.8492 - val_loss: 0.0295 - val_acc: 0.8585
    Epoch 95/99
    60000/60000 [==============================] - 0s - loss: 0.0303 - acc: 0.8498 - val_loss: 0.0294 - val_acc: 0.8594
    Epoch 96/99
    60000/60000 [==============================] - 0s - loss: 0.0302 - acc: 0.8503 - val_loss: 0.0292 - val_acc: 0.8599
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0301 - acc: 0.8511 - val_loss: 0.0291 - val_acc: 0.8607
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0299 - acc: 0.8512 - val_loss: 0.0290 - val_acc: 0.8612
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0298 - acc: 0.8519 - val_loss: 0.0289 - val_acc: 0.8616

    <keras.callbacks.History at 0x12dda54a8>

```python
# 86% accuracy after another 99 epochs
nn.evaluate(X_test, y_test)
```

     9024/10000 [==========================>...] - ETA: 0s

    [0.028857796752452852, 0.86160000000000003]

```python
nn.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0297 - acc: 0.8519 - val_loss: 0.0287 - val_acc: 0.8619
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0296 - acc: 0.8528 - val_loss: 0.0286 - val_acc: 0.8625
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0295 - acc: 0.8531 - val_loss: 0.0285 - val_acc: 0.8628
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0293 - acc: 0.8538 - val_loss: 0.0284 - val_acc: 0.8636
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0292 - acc: 0.8543 - val_loss: 0.0283 - val_acc: 0.8640
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0291 - acc: 0.8548 - val_loss: 0.0281 - val_acc: 0.8646
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0290 - acc: 0.8550 - val_loss: 0.0280 - val_acc: 0.8651
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.0289 - acc: 0.8557 - val_loss: 0.0279 - val_acc: 0.8655
    Epoch 9/99
    60000/60000 [==============================] - 0s - loss: 0.0288 - acc: 0.8562 - val_loss: 0.0278 - val_acc: 0.8656
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.0287 - acc: 0.8565 - val_loss: 0.0277 - val_acc: 0.8660
    Epoch 11/99
    60000/60000 [==============================] - 1s - loss: 0.0285 - acc: 0.8568 - val_loss: 0.0276 - val_acc: 0.8663
    Epoch 12/99
    60000/60000 [==============================] - 1s - loss: 0.0284 - acc: 0.8572 - val_loss: 0.0275 - val_acc: 0.8665
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.0283 - acc: 0.8577 - val_loss: 0.0274 - val_acc: 0.8672
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.0282 - acc: 0.8580 - val_loss: 0.0273 - val_acc: 0.8673
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0281 - acc: 0.8583 - val_loss: 0.0271 - val_acc: 0.8676
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.0280 - acc: 0.8588 - val_loss: 0.0270 - val_acc: 0.8679
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0279 - acc: 0.8593 - val_loss: 0.0269 - val_acc: 0.8681
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0278 - acc: 0.8596 - val_loss: 0.0268 - val_acc: 0.8687
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0277 - acc: 0.8599 - val_loss: 0.0267 - val_acc: 0.8691
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0276 - acc: 0.8602 - val_loss: 0.0266 - val_acc: 0.8695
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0275 - acc: 0.8606 - val_loss: 0.0265 - val_acc: 0.8697
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0274 - acc: 0.8610 - val_loss: 0.0264 - val_acc: 0.8701
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0273 - acc: 0.8612 - val_loss: 0.0263 - val_acc: 0.8702
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0272 - acc: 0.8615 - val_loss: 0.0263 - val_acc: 0.8709
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0271 - acc: 0.8618 - val_loss: 0.0262 - val_acc: 0.8709
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0270 - acc: 0.8621 - val_loss: 0.0261 - val_acc: 0.8713
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0270 - acc: 0.8623 - val_loss: 0.0260 - val_acc: 0.8716
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0269 - acc: 0.8625 - val_loss: 0.0259 - val_acc: 0.8721
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0268 - acc: 0.8628 - val_loss: 0.0258 - val_acc: 0.8723
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0267 - acc: 0.8630 - val_loss: 0.0257 - val_acc: 0.8727
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0266 - acc: 0.8633 - val_loss: 0.0256 - val_acc: 0.8726
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0265 - acc: 0.8636 - val_loss: 0.0255 - val_acc: 0.8727
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0264 - acc: 0.8638 - val_loss: 0.0254 - val_acc: 0.8728
    Epoch 34/99
    60000/60000 [==============================] - 1s - loss: 0.0263 - acc: 0.8640 - val_loss: 0.0254 - val_acc: 0.8733
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0262 - acc: 0.8643 - val_loss: 0.0253 - val_acc: 0.8734
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0262 - acc: 0.8645 - val_loss: 0.0252 - val_acc: 0.8736
    Epoch 37/99
    60000/60000 [==============================] - 1s - loss: 0.0261 - acc: 0.8649 - val_loss: 0.0251 - val_acc: 0.8739
    Epoch 38/99
    60000/60000 [==============================] - 1s - loss: 0.0260 - acc: 0.8653 - val_loss: 0.0250 - val_acc: 0.8745
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0259 - acc: 0.8655 - val_loss: 0.0249 - val_acc: 0.8749
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0258 - acc: 0.8660 - val_loss: 0.0249 - val_acc: 0.8750
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0258 - acc: 0.8661 - val_loss: 0.0248 - val_acc: 0.8751
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0257 - acc: 0.8664 - val_loss: 0.0247 - val_acc: 0.8754
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0256 - acc: 0.8667 - val_loss: 0.0246 - val_acc: 0.8758
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0255 - acc: 0.8671 - val_loss: 0.0246 - val_acc: 0.8760
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0255 - acc: 0.8674 - val_loss: 0.0245 - val_acc: 0.8759
    Epoch 46/99
    60000/60000 [==============================] - 1s - loss: 0.0254 - acc: 0.8676 - val_loss: 0.0244 - val_acc: 0.8764
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0253 - acc: 0.8678 - val_loss: 0.0243 - val_acc: 0.8765
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0252 - acc: 0.8680 - val_loss: 0.0243 - val_acc: 0.8766
    Epoch 49/99
    60000/60000 [==============================] - 1s - loss: 0.0252 - acc: 0.8684 - val_loss: 0.0242 - val_acc: 0.8770
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0251 - acc: 0.8687 - val_loss: 0.0241 - val_acc: 0.8773
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0250 - acc: 0.8689 - val_loss: 0.0240 - val_acc: 0.8772
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0249 - acc: 0.8693 - val_loss: 0.0240 - val_acc: 0.8776
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0249 - acc: 0.8696 - val_loss: 0.0239 - val_acc: 0.8778
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0248 - acc: 0.8696 - val_loss: 0.0238 - val_acc: 0.8780
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0247 - acc: 0.8701 - val_loss: 0.0238 - val_acc: 0.8782
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.0247 - acc: 0.8702 - val_loss: 0.0237 - val_acc: 0.8783
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0246 - acc: 0.8705 - val_loss: 0.0236 - val_acc: 0.8784
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0245 - acc: 0.8707 - val_loss: 0.0236 - val_acc: 0.8786
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0245 - acc: 0.8710 - val_loss: 0.0235 - val_acc: 0.8788
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0244 - acc: 0.8713 - val_loss: 0.0234 - val_acc: 0.8787
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0243 - acc: 0.8716 - val_loss: 0.0234 - val_acc: 0.8788
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0243 - acc: 0.8716 - val_loss: 0.0233 - val_acc: 0.8790
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0242 - acc: 0.8720 - val_loss: 0.0232 - val_acc: 0.8794
    Epoch 64/99
    60000/60000 [==============================] - 1s - loss: 0.0241 - acc: 0.8722 - val_loss: 0.0232 - val_acc: 0.8800
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0241 - acc: 0.8724 - val_loss: 0.0231 - val_acc: 0.8802
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0240 - acc: 0.8726 - val_loss: 0.0230 - val_acc: 0.8803
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0240 - acc: 0.8728 - val_loss: 0.0230 - val_acc: 0.8803
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0239 - acc: 0.8730 - val_loss: 0.0229 - val_acc: 0.8802
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0238 - acc: 0.8733 - val_loss: 0.0229 - val_acc: 0.8803
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0238 - acc: 0.8735 - val_loss: 0.0228 - val_acc: 0.8807
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0237 - acc: 0.8738 - val_loss: 0.0227 - val_acc: 0.8808
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0237 - acc: 0.8739 - val_loss: 0.0227 - val_acc: 0.8809
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0236 - acc: 0.8742 - val_loss: 0.0226 - val_acc: 0.8810
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0235 - acc: 0.8743 - val_loss: 0.0226 - val_acc: 0.8815
    Epoch 75/99
    60000/60000 [==============================] - 1s - loss: 0.0235 - acc: 0.8745 - val_loss: 0.0225 - val_acc: 0.8815
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0234 - acc: 0.8747 - val_loss: 0.0225 - val_acc: 0.8818
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0234 - acc: 0.8751 - val_loss: 0.0224 - val_acc: 0.8819
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0233 - acc: 0.8752 - val_loss: 0.0223 - val_acc: 0.8822
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0233 - acc: 0.8754 - val_loss: 0.0223 - val_acc: 0.8826
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0232 - acc: 0.8755 - val_loss: 0.0222 - val_acc: 0.8828
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0231 - acc: 0.8758 - val_loss: 0.0222 - val_acc: 0.8830
    Epoch 82/99
    60000/60000 [==============================] - 1s - loss: 0.0231 - acc: 0.8759 - val_loss: 0.0221 - val_acc: 0.8834
    Epoch 83/99
    60000/60000 [==============================] - 1s - loss: 0.0230 - acc: 0.8762 - val_loss: 0.0221 - val_acc: 0.8833
    Epoch 84/99
    60000/60000 [==============================] - 1s - loss: 0.0230 - acc: 0.8764 - val_loss: 0.0220 - val_acc: 0.8837
    Epoch 85/99
    60000/60000 [==============================] - 1s - loss: 0.0229 - acc: 0.8765 - val_loss: 0.0220 - val_acc: 0.8837
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0229 - acc: 0.8769 - val_loss: 0.0219 - val_acc: 0.8837
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0228 - acc: 0.8771 - val_loss: 0.0219 - val_acc: 0.8838
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.0228 - acc: 0.8772 - val_loss: 0.0218 - val_acc: 0.8844
    Epoch 89/99
    60000/60000 [==============================] - 0s - loss: 0.0227 - acc: 0.8773 - val_loss: 0.0218 - val_acc: 0.8844
    Epoch 90/99
    60000/60000 [==============================] - 0s - loss: 0.0227 - acc: 0.8775 - val_loss: 0.0217 - val_acc: 0.8848
    Epoch 91/99
    60000/60000 [==============================] - 0s - loss: 0.0226 - acc: 0.8776 - val_loss: 0.0217 - val_acc: 0.8854
    Epoch 92/99
    60000/60000 [==============================] - 0s - loss: 0.0226 - acc: 0.8778 - val_loss: 0.0216 - val_acc: 0.8855
    Epoch 93/99
    60000/60000 [==============================] - 0s - loss: 0.0225 - acc: 0.8780 - val_loss: 0.0216 - val_acc: 0.8858
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0225 - acc: 0.8780 - val_loss: 0.0215 - val_acc: 0.8860
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0224 - acc: 0.8780 - val_loss: 0.0215 - val_acc: 0.8862
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0224 - acc: 0.8783 - val_loss: 0.0214 - val_acc: 0.8865
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.0223 - acc: 0.8784 - val_loss: 0.0214 - val_acc: 0.8869
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0223 - acc: 0.8785 - val_loss: 0.0213 - val_acc: 0.8869
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0223 - acc: 0.8786 - val_loss: 0.0213 - val_acc: 0.8868

    <keras.callbacks.History at 0x12ddb3ef0>

```python
# 88% accuracy after another 99 epochs
nn.evaluate(X_test, y_test)
```

     9696/10000 [============================>.] - ETA: 0s

    [0.021296201345324516, 0.88680000000000003]

## Activation functions

In computational networks, the [activation function](https://en.wikipedia.org/wiki/Activation_function) of a node
defines the output of that node given an input or set of inputs. A
standard computer chip circuit can be seen as a digital network of
activation functions that can be “ON” (1) or “OFF” (0), depending on
input. This is similar to the behavior of the linear perceptron in
neural networks. However, only *nonlinear* activation functions allow
such networks to compute nontrivial problems using only a small number
of nodes. In artificial neural networks this function is also called
the **transfer function**.

Functions
---------

In biologically inspired neural networks, the activation function is
usually an abstraction representing the rate of action potential
firing in the cell. In its simplest form, this function is binary—that
is, either the neuron is firing or not. The function looks like
$\phi(v_i)=U(v_i)$, where $U$ is the Heaviside step function. In this
case many neurons must be used in computation beyond linear separation
of categories.

A line of positive slope may be used to reflect the increase in firing
rate that occurs as input current increases. Such a function would be of
the form $\phi(v_i)=\mu v_i$, where $\mu$ is the slope. This activation
function is linear, and therefore has the same problems as the binary
function. In addition, networks constructed using this model have
unstable convergence because neuron inputs along favored paths tend to
increase without bound, as this function is not normalizable.

All problems mentioned above can be handled by using a normalizable
sigmoid activation function. One realistic model stays at zero until
input current is received, at which point the firing frequency increases
quickly at first, but gradually approaches an asymptote at 100% firing
rate. Mathematically, this looks like $\phi(v_i)=U(v_i)\tanh(v_i)$,
where the hyperbolic tangent function can be replaced by any sigmoid
function. This behavior is realistically reflected in the neuron, as
neurons cannot physically fire faster than a certain rate. This model
runs into problems, however, in computational networks as it is not
differentiable, a requirement to calculate backpropagation.

The final model, then, that is used in multilayer perceptrons is a
sigmoidal activation function in the form of a hyperbolic tangent. Two
forms of this function are commonly used: $\phi(v_i)=\tanh(v_i)$ whose
range is normalized from -1 to 1, and $\phi(v_i) = (1+\exp(-v_i))^{-1}$
is vertically translated to normalize from 0 to 1. The latter model is
often considered more biologically realistic, but it runs into
theoretical and experimental difficulties with certain types.

## Comparison of activation functions

Some desirable properties in an activation function include:

-   Nonlinear – When the activation function is non-linear, then a
    two-layer neural network can be proven to be a universal function
    approximator. The identity activation function does not satisfy
    this property. When multiple layers use the identity activation
    function, the entire network is equivalent to a single-layer model.
-   Continuously differentiable – This property is necessary for
    enabling gradient-based optimization methods. The binary step
    activation function is not differentiable at 0, and it
    differentiates to 0 for all other values, so gradient-based methods
    can make no progress with it.
-   Range – When the range of the activation function is finite,
    gradient-based training methods tend to be more stable, because
    pattern presentations significantly affect only limited weights.
    When the range is infinite, training is generally more efficient
    because pattern presentations significantly affect most of the
    weights. In the latter case, smaller learning rates are typically
    necessary.
-   Monotonic – When the activation function is monotonic, the error
    surface associated with a single-layer model is guaranteed to be
    convex.
-   Smooth Functions with a Monotonic derivative – These have been shown
    to generalize better in some cases. The argument for these
    properties suggests that such activation functions are more
    consistent with Occam's razor.
-   Approximates identity near the origin – When activation functions
    have this property, the neural network will learn efficiently when
    its weights are initialized with small random values. When the
    activation function does not approximate identity near the origin,
    special care must be used when initializing the weights.

## Rectified linear unit (ReLU) transfer function

Rectified linear unit (ReLU)

Activation identity

$f(x)=x$
$f'(x)=1$
$(-\infty,\infty)$
$C^\infty$

Logistic (a.k.a. Soft step)

$f(x)=\frac{1}{1+e^{-x}}$
$f'(x)=f(x)(1-f(x))$
$(0,1)$
$C^\infty$

TanH

$f(x)=\tanh(x)=\frac{2}{1+e^{-2x}}-1$
$f'(x)=1-f(x)^2$
$(-1,1)$
$C^\infty$

Rectified linear unit (ReLU)

$f(x) = \begin{cases}
    0 & \text{for } x < 0\\
    x & \text{for } x \ge 0\end{cases}$

$f'(x) = \begin{cases}
    0 & \text{for } x < 0\\
    1 & \text{for } x \ge 0\end{cases}$

$[0,\infty)$
$C^0$

The Rectified linear unit (ReLU) seem to work well empirically.

```python
def shallow_net_B(n=55,i=784,o=10):
    # create simple one dense layer net
    # default 55 neurons, input 784, output 10
    # Using relu
    net = Sequential()
    net.add(Dense(n, activation='relu', input_shape=(i,)))
    net.add(Dense(10, activation='softmax'))
    # Compile net
    net.compile(loss='mean_squared_error', optimizer=SGD(lr=0.01), metrics=['accuracy'])
    return net
```

```python
nn2=shallow_net_B()
nn2.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_3 (Dense)              (None, 55)                43175
    _________________________________________________________________
    dense_4 (Dense)              (None, 10)                560
    =================================================================
    Total params: 43,735
    Trainable params: 43,735
    Non-trainable params: 0
    _________________________________________________________________

```python
nn2.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0912 - acc: 0.0756 - val_loss: 0.0902 - val_acc: 0.1054
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0892 - acc: 0.1552 - val_loss: 0.0883 - val_acc: 0.2029
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0873 - acc: 0.2435 - val_loss: 0.0863 - val_acc: 0.2915
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0852 - acc: 0.3212 - val_loss: 0.0841 - val_acc: 0.3640
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0829 - acc: 0.3821 - val_loss: 0.0816 - val_acc: 0.4106
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0803 - acc: 0.4162 - val_loss: 0.0788 - val_acc: 0.4353
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0775 - acc: 0.4381 - val_loss: 0.0758 - val_acc: 0.4511
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.0744 - acc: 0.4594 - val_loss: 0.0726 - val_acc: 0.4724
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0712 - acc: 0.4845 - val_loss: 0.0693 - val_acc: 0.5041
    Epoch 10/99
    60000/60000 [==============================] - 1s - loss: 0.0680 - acc: 0.5242 - val_loss: 0.0661 - val_acc: 0.5474
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.0649 - acc: 0.5734 - val_loss: 0.0630 - val_acc: 0.6030
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.0618 - acc: 0.6247 - val_loss: 0.0599 - val_acc: 0.6471
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.0587 - acc: 0.6640 - val_loss: 0.0568 - val_acc: 0.6821
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.0557 - acc: 0.6906 - val_loss: 0.0538 - val_acc: 0.7076
    Epoch 15/99
    60000/60000 [==============================] - 0s - loss: 0.0528 - acc: 0.7085 - val_loss: 0.0509 - val_acc: 0.7248
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.0501 - acc: 0.7229 - val_loss: 0.0483 - val_acc: 0.7400
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0477 - acc: 0.7377 - val_loss: 0.0458 - val_acc: 0.7556
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0454 - acc: 0.7532 - val_loss: 0.0436 - val_acc: 0.7726
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0433 - acc: 0.7701 - val_loss: 0.0414 - val_acc: 0.7882
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0413 - acc: 0.7853 - val_loss: 0.0395 - val_acc: 0.8010
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0395 - acc: 0.7971 - val_loss: 0.0377 - val_acc: 0.8104
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0378 - acc: 0.8064 - val_loss: 0.0361 - val_acc: 0.8188
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0363 - acc: 0.8137 - val_loss: 0.0346 - val_acc: 0.8254
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0350 - acc: 0.8196 - val_loss: 0.0333 - val_acc: 0.8324
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0337 - acc: 0.8244 - val_loss: 0.0321 - val_acc: 0.8362
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0326 - acc: 0.8294 - val_loss: 0.0311 - val_acc: 0.8407
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0316 - acc: 0.8339 - val_loss: 0.0301 - val_acc: 0.8450
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0307 - acc: 0.8371 - val_loss: 0.0292 - val_acc: 0.8479
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0299 - acc: 0.8400 - val_loss: 0.0284 - val_acc: 0.8509
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0291 - acc: 0.8433 - val_loss: 0.0276 - val_acc: 0.8539
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0284 - acc: 0.8466 - val_loss: 0.0269 - val_acc: 0.8558
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0277 - acc: 0.8489 - val_loss: 0.0263 - val_acc: 0.8587
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0271 - acc: 0.8512 - val_loss: 0.0257 - val_acc: 0.8613
    Epoch 34/99
    60000/60000 [==============================] - 1s - loss: 0.0266 - acc: 0.8538 - val_loss: 0.0252 - val_acc: 0.8628
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0261 - acc: 0.8562 - val_loss: 0.0247 - val_acc: 0.8640
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0256 - acc: 0.8582 - val_loss: 0.0242 - val_acc: 0.8662
    Epoch 37/99
    60000/60000 [==============================] - 1s - loss: 0.0252 - acc: 0.8601 - val_loss: 0.0238 - val_acc: 0.8674
    Epoch 38/99
    60000/60000 [==============================] - 1s - loss: 0.0247 - acc: 0.8618 - val_loss: 0.0234 - val_acc: 0.8689
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0244 - acc: 0.8637 - val_loss: 0.0230 - val_acc: 0.8704
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0240 - acc: 0.8654 - val_loss: 0.0227 - val_acc: 0.8712
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0236 - acc: 0.8668 - val_loss: 0.0223 - val_acc: 0.8730
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0233 - acc: 0.8681 - val_loss: 0.0220 - val_acc: 0.8744
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0230 - acc: 0.8693 - val_loss: 0.0217 - val_acc: 0.8749
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0227 - acc: 0.8702 - val_loss: 0.0215 - val_acc: 0.8762
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0224 - acc: 0.8714 - val_loss: 0.0212 - val_acc: 0.8770
    Epoch 46/99
    60000/60000 [==============================] - 1s - loss: 0.0222 - acc: 0.8726 - val_loss: 0.0209 - val_acc: 0.8788
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0219 - acc: 0.8739 - val_loss: 0.0207 - val_acc: 0.8795
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0217 - acc: 0.8751 - val_loss: 0.0205 - val_acc: 0.8811
    Epoch 49/99
    60000/60000 [==============================] - 1s - loss: 0.0215 - acc: 0.8760 - val_loss: 0.0203 - val_acc: 0.8820
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0213 - acc: 0.8771 - val_loss: 0.0200 - val_acc: 0.8829
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0211 - acc: 0.8778 - val_loss: 0.0198 - val_acc: 0.8844
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0209 - acc: 0.8789 - val_loss: 0.0197 - val_acc: 0.8853
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0207 - acc: 0.8797 - val_loss: 0.0195 - val_acc: 0.8862
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0205 - acc: 0.8806 - val_loss: 0.0193 - val_acc: 0.8870
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0203 - acc: 0.8816 - val_loss: 0.0191 - val_acc: 0.8884
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.0201 - acc: 0.8824 - val_loss: 0.0190 - val_acc: 0.8889
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0200 - acc: 0.8832 - val_loss: 0.0188 - val_acc: 0.8892
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0198 - acc: 0.8837 - val_loss: 0.0187 - val_acc: 0.8896
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0197 - acc: 0.8847 - val_loss: 0.0185 - val_acc: 0.8906
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0195 - acc: 0.8855 - val_loss: 0.0184 - val_acc: 0.8915
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0194 - acc: 0.8859 - val_loss: 0.0183 - val_acc: 0.8923
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0193 - acc: 0.8867 - val_loss: 0.0181 - val_acc: 0.8932
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0191 - acc: 0.8873 - val_loss: 0.0180 - val_acc: 0.8936
    Epoch 64/99
    60000/60000 [==============================] - 1s - loss: 0.0190 - acc: 0.8878 - val_loss: 0.0179 - val_acc: 0.8945
    Epoch 65/99
    60000/60000 [==============================] - 1s - loss: 0.0189 - acc: 0.8885 - val_loss: 0.0178 - val_acc: 0.8950
    Epoch 66/99
    60000/60000 [==============================] - 1s - loss: 0.0188 - acc: 0.8888 - val_loss: 0.0177 - val_acc: 0.8956
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0186 - acc: 0.8894 - val_loss: 0.0176 - val_acc: 0.8961
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0185 - acc: 0.8898 - val_loss: 0.0174 - val_acc: 0.8963
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0184 - acc: 0.8905 - val_loss: 0.0173 - val_acc: 0.8971
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0183 - acc: 0.8908 - val_loss: 0.0172 - val_acc: 0.8978
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0182 - acc: 0.8913 - val_loss: 0.0171 - val_acc: 0.8982
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0181 - acc: 0.8917 - val_loss: 0.0171 - val_acc: 0.8988
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0180 - acc: 0.8923 - val_loss: 0.0170 - val_acc: 0.8990
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0179 - acc: 0.8925 - val_loss: 0.0169 - val_acc: 0.8996
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0178 - acc: 0.8929 - val_loss: 0.0168 - val_acc: 0.9003
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0177 - acc: 0.8935 - val_loss: 0.0167 - val_acc: 0.9006
    Epoch 77/99
    60000/60000 [==============================] - 1s - loss: 0.0176 - acc: 0.8937 - val_loss: 0.0166 - val_acc: 0.9010
    Epoch 78/99
    60000/60000 [==============================] - 1s - loss: 0.0176 - acc: 0.8942 - val_loss: 0.0165 - val_acc: 0.9015
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0175 - acc: 0.8948 - val_loss: 0.0165 - val_acc: 0.9020
    Epoch 80/99
    60000/60000 [==============================] - 1s - loss: 0.0174 - acc: 0.8950 - val_loss: 0.0164 - val_acc: 0.9023
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0173 - acc: 0.8955 - val_loss: 0.0163 - val_acc: 0.9024
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0172 - acc: 0.8959 - val_loss: 0.0162 - val_acc: 0.9028
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0172 - acc: 0.8962 - val_loss: 0.0162 - val_acc: 0.9028
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0171 - acc: 0.8967 - val_loss: 0.0161 - val_acc: 0.9027
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0170 - acc: 0.8970 - val_loss: 0.0160 - val_acc: 0.9030
    Epoch 86/99
    60000/60000 [==============================] - 1s - loss: 0.0169 - acc: 0.8975 - val_loss: 0.0160 - val_acc: 0.9031
    Epoch 87/99
    60000/60000 [==============================] - 1s - loss: 0.0169 - acc: 0.8981 - val_loss: 0.0159 - val_acc: 0.9036
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.0168 - acc: 0.8980 - val_loss: 0.0158 - val_acc: 0.9040
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.0167 - acc: 0.8985 - val_loss: 0.0158 - val_acc: 0.9040
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.0167 - acc: 0.8987 - val_loss: 0.0157 - val_acc: 0.9039
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.0166 - acc: 0.8990 - val_loss: 0.0156 - val_acc: 0.9042
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.0165 - acc: 0.8993 - val_loss: 0.0156 - val_acc: 0.9045
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.0165 - acc: 0.8997 - val_loss: 0.0155 - val_acc: 0.9051
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.0164 - acc: 0.9000 - val_loss: 0.0155 - val_acc: 0.9048
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.0164 - acc: 0.9004 - val_loss: 0.0154 - val_acc: 0.9052
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.0163 - acc: 0.9008 - val_loss: 0.0154 - val_acc: 0.9054
    Epoch 97/99
    60000/60000 [==============================] - 0s - loss: 0.0162 - acc: 0.9011 - val_loss: 0.0153 - val_acc: 0.9053
    Epoch 98/99
    60000/60000 [==============================] - 0s - loss: 0.0162 - acc: 0.9012 - val_loss: 0.0152 - val_acc: 0.9056
    Epoch 99/99
    60000/60000 [==============================] - 0s - loss: 0.0161 - acc: 0.9016 - val_loss: 0.0152 - val_acc: 0.9056

    <keras.callbacks.History at 0x12e124b00>

```python
# 90% accuracy after first 99 epochs with Relu
nn2.evaluate(X_test, y_test)
```

     9984/10000 [============================>.] - ETA: 0s

    [0.015197866915352642, 0.90559999999999996]

```python
nn2.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 0s - loss: 0.0161 - acc: 0.9020 - val_loss: 0.0151 - val_acc: 0.9056
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0160 - acc: 0.9022 - val_loss: 0.0151 - val_acc: 0.9058
    Epoch 3/99
    60000/60000 [==============================] - 0s - loss: 0.0160 - acc: 0.9024 - val_loss: 0.0150 - val_acc: 0.9064
    Epoch 4/99
    60000/60000 [==============================] - 0s - loss: 0.0159 - acc: 0.9026 - val_loss: 0.0150 - val_acc: 0.9062
    Epoch 5/99
    60000/60000 [==============================] - 0s - loss: 0.0159 - acc: 0.9029 - val_loss: 0.0150 - val_acc: 0.9065
    Epoch 6/99
    60000/60000 [==============================] - 0s - loss: 0.0158 - acc: 0.9032 - val_loss: 0.0149 - val_acc: 0.9070
    Epoch 7/99
    60000/60000 [==============================] - 0s - loss: 0.0158 - acc: 0.9034 - val_loss: 0.0149 - val_acc: 0.9075
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.0157 - acc: 0.9037 - val_loss: 0.0148 - val_acc: 0.9077
    Epoch 9/99
    60000/60000 [==============================] - 0s - loss: 0.0157 - acc: 0.9040 - val_loss: 0.0148 - val_acc: 0.9079
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.0156 - acc: 0.9041 - val_loss: 0.0147 - val_acc: 0.9083
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.0156 - acc: 0.9043 - val_loss: 0.0147 - val_acc: 0.9089
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.0155 - acc: 0.9048 - val_loss: 0.0146 - val_acc: 0.9089
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.0155 - acc: 0.9048 - val_loss: 0.0146 - val_acc: 0.9093
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.0154 - acc: 0.9052 - val_loss: 0.0146 - val_acc: 0.9097
    Epoch 15/99
    60000/60000 [==============================] - 0s - loss: 0.0154 - acc: 0.9055 - val_loss: 0.0145 - val_acc: 0.9099
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.0153 - acc: 0.9056 - val_loss: 0.0145 - val_acc: 0.9106
    Epoch 17/99
    60000/60000 [==============================] - 0s - loss: 0.0153 - acc: 0.9059 - val_loss: 0.0144 - val_acc: 0.9106
    Epoch 18/99
    60000/60000 [==============================] - 0s - loss: 0.0152 - acc: 0.9062 - val_loss: 0.0144 - val_acc: 0.9112
    Epoch 19/99
    60000/60000 [==============================] - 0s - loss: 0.0152 - acc: 0.9062 - val_loss: 0.0144 - val_acc: 0.9111
    Epoch 20/99
    60000/60000 [==============================] - 0s - loss: 0.0152 - acc: 0.9064 - val_loss: 0.0143 - val_acc: 0.9111
    Epoch 21/99
    60000/60000 [==============================] - 0s - loss: 0.0151 - acc: 0.9067 - val_loss: 0.0143 - val_acc: 0.9118
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.0151 - acc: 0.9070 - val_loss: 0.0142 - val_acc: 0.9121
    Epoch 23/99
    60000/60000 [==============================] - 0s - loss: 0.0150 - acc: 0.9071 - val_loss: 0.0142 - val_acc: 0.9124
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.0150 - acc: 0.9071 - val_loss: 0.0142 - val_acc: 0.9125
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.0150 - acc: 0.9075 - val_loss: 0.0141 - val_acc: 0.9126
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.0149 - acc: 0.9078 - val_loss: 0.0141 - val_acc: 0.9127
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.0149 - acc: 0.9080 - val_loss: 0.0141 - val_acc: 0.9126
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.0148 - acc: 0.9081 - val_loss: 0.0140 - val_acc: 0.9131
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.0148 - acc: 0.9083 - val_loss: 0.0140 - val_acc: 0.9128
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.0148 - acc: 0.9085 - val_loss: 0.0140 - val_acc: 0.9130
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9088 - val_loss: 0.0139 - val_acc: 0.9132
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9089 - val_loss: 0.0139 - val_acc: 0.9136
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.0147 - acc: 0.9091 - val_loss: 0.0139 - val_acc: 0.9139
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.0146 - acc: 0.9094 - val_loss: 0.0138 - val_acc: 0.9143
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.0146 - acc: 0.9098 - val_loss: 0.0138 - val_acc: 0.9141
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.0146 - acc: 0.9097 - val_loss: 0.0138 - val_acc: 0.9145
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.0145 - acc: 0.9101 - val_loss: 0.0137 - val_acc: 0.9142
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.0145 - acc: 0.9102 - val_loss: 0.0137 - val_acc: 0.9147
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.0145 - acc: 0.9102 - val_loss: 0.0137 - val_acc: 0.9146
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.0144 - acc: 0.9104 - val_loss: 0.0137 - val_acc: 0.9147
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.0144 - acc: 0.9106 - val_loss: 0.0136 - val_acc: 0.9148
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.0144 - acc: 0.9110 - val_loss: 0.0136 - val_acc: 0.9149
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9111 - val_loss: 0.0136 - val_acc: 0.9149
    Epoch 44/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9113 - val_loss: 0.0135 - val_acc: 0.9148
    Epoch 45/99
    60000/60000 [==============================] - 0s - loss: 0.0143 - acc: 0.9115 - val_loss: 0.0135 - val_acc: 0.9151
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.0142 - acc: 0.9115 - val_loss: 0.0135 - val_acc: 0.9155
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.0142 - acc: 0.9117 - val_loss: 0.0135 - val_acc: 0.9156
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.0142 - acc: 0.9120 - val_loss: 0.0134 - val_acc: 0.9162
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.0141 - acc: 0.9122 - val_loss: 0.0134 - val_acc: 0.9160
    Epoch 50/99
    60000/60000 [==============================] - 0s - loss: 0.0141 - acc: 0.9123 - val_loss: 0.0134 - val_acc: 0.9163
    Epoch 51/99
    60000/60000 [==============================] - 0s - loss: 0.0141 - acc: 0.9122 - val_loss: 0.0134 - val_acc: 0.9165
    Epoch 52/99
    60000/60000 [==============================] - 0s - loss: 0.0141 - acc: 0.9126 - val_loss: 0.0133 - val_acc: 0.9166
    Epoch 53/99
    60000/60000 [==============================] - 0s - loss: 0.0140 - acc: 0.9126 - val_loss: 0.0133 - val_acc: 0.9166
    Epoch 54/99
    60000/60000 [==============================] - 0s - loss: 0.0140 - acc: 0.9126 - val_loss: 0.0133 - val_acc: 0.9171
    Epoch 55/99
    60000/60000 [==============================] - 0s - loss: 0.0140 - acc: 0.9130 - val_loss: 0.0133 - val_acc: 0.9170
    Epoch 56/99
    60000/60000 [==============================] - 0s - loss: 0.0139 - acc: 0.9132 - val_loss: 0.0132 - val_acc: 0.9177
    Epoch 57/99
    60000/60000 [==============================] - 0s - loss: 0.0139 - acc: 0.9134 - val_loss: 0.0132 - val_acc: 0.9176
    Epoch 58/99
    60000/60000 [==============================] - 0s - loss: 0.0139 - acc: 0.9133 - val_loss: 0.0132 - val_acc: 0.9179
    Epoch 59/99
    60000/60000 [==============================] - 0s - loss: 0.0139 - acc: 0.9137 - val_loss: 0.0132 - val_acc: 0.9177
    Epoch 60/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9138 - val_loss: 0.0131 - val_acc: 0.9178
    Epoch 61/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9138 - val_loss: 0.0131 - val_acc: 0.9181
    Epoch 62/99
    60000/60000 [==============================] - 0s - loss: 0.0138 - acc: 0.9140 - val_loss: 0.0131 - val_acc: 0.9183
    Epoch 63/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9142 - val_loss: 0.0131 - val_acc: 0.9186
    Epoch 64/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9142 - val_loss: 0.0130 - val_acc: 0.9185
    Epoch 65/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9142 - val_loss: 0.0130 - val_acc: 0.9189
    Epoch 66/99
    60000/60000 [==============================] - 0s - loss: 0.0137 - acc: 0.9146 - val_loss: 0.0130 - val_acc: 0.9190
    Epoch 67/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9147 - val_loss: 0.0130 - val_acc: 0.9193
    Epoch 68/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9150 - val_loss: 0.0129 - val_acc: 0.9192
    Epoch 69/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9151 - val_loss: 0.0129 - val_acc: 0.9191
    Epoch 70/99
    60000/60000 [==============================] - 0s - loss: 0.0136 - acc: 0.9151 - val_loss: 0.0129 - val_acc: 0.9192
    Epoch 71/99
    60000/60000 [==============================] - 0s - loss: 0.0135 - acc: 0.9155 - val_loss: 0.0129 - val_acc: 0.9192
    Epoch 72/99
    60000/60000 [==============================] - 0s - loss: 0.0135 - acc: 0.9156 - val_loss: 0.0129 - val_acc: 0.9191
    Epoch 73/99
    60000/60000 [==============================] - 0s - loss: 0.0135 - acc: 0.9156 - val_loss: 0.0128 - val_acc: 0.9193
    Epoch 74/99
    60000/60000 [==============================] - 0s - loss: 0.0135 - acc: 0.9159 - val_loss: 0.0128 - val_acc: 0.9192
    Epoch 75/99
    60000/60000 [==============================] - 0s - loss: 0.0134 - acc: 0.9159 - val_loss: 0.0128 - val_acc: 0.9191
    Epoch 76/99
    60000/60000 [==============================] - 0s - loss: 0.0134 - acc: 0.9162 - val_loss: 0.0128 - val_acc: 0.9196
    Epoch 77/99
    60000/60000 [==============================] - 0s - loss: 0.0134 - acc: 0.9161 - val_loss: 0.0128 - val_acc: 0.9199
    Epoch 78/99
    60000/60000 [==============================] - 0s - loss: 0.0134 - acc: 0.9163 - val_loss: 0.0127 - val_acc: 0.9199
    Epoch 79/99
    60000/60000 [==============================] - 0s - loss: 0.0133 - acc: 0.9165 - val_loss: 0.0127 - val_acc: 0.9201
    Epoch 80/99
    60000/60000 [==============================] - 0s - loss: 0.0133 - acc: 0.9166 - val_loss: 0.0127 - val_acc: 0.9202
    Epoch 81/99
    60000/60000 [==============================] - 0s - loss: 0.0133 - acc: 0.9167 - val_loss: 0.0127 - val_acc: 0.9201
    Epoch 82/99
    60000/60000 [==============================] - 0s - loss: 0.0133 - acc: 0.9169 - val_loss: 0.0127 - val_acc: 0.9205
    Epoch 83/99
    60000/60000 [==============================] - 0s - loss: 0.0133 - acc: 0.9169 - val_loss: 0.0126 - val_acc: 0.9205
    Epoch 84/99
    60000/60000 [==============================] - 0s - loss: 0.0132 - acc: 0.9171 - val_loss: 0.0126 - val_acc: 0.9205
    Epoch 85/99
    60000/60000 [==============================] - 0s - loss: 0.0132 - acc: 0.9172 - val_loss: 0.0126 - val_acc: 0.9206
    Epoch 86/99
    60000/60000 [==============================] - 0s - loss: 0.0132 - acc: 0.9174 - val_loss: 0.0126 - val_acc: 0.9208
    Epoch 87/99
    60000/60000 [==============================] - 0s - loss: 0.0132 - acc: 0.9176 - val_loss: 0.0126 - val_acc: 0.9209
    Epoch 88/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9177 - val_loss: 0.0125 - val_acc: 0.9208
    Epoch 89/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9179 - val_loss: 0.0125 - val_acc: 0.9210
    Epoch 90/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9181 - val_loss: 0.0125 - val_acc: 0.9211
    Epoch 91/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9182 - val_loss: 0.0125 - val_acc: 0.9211
    Epoch 92/99
    60000/60000 [==============================] - 0s - loss: 0.0131 - acc: 0.9184 - val_loss: 0.0125 - val_acc: 0.9211
    Epoch 93/99
    60000/60000 [==============================] - 0s - loss: 0.0130 - acc: 0.9182 - val_loss: 0.0124 - val_acc: 0.9214
    Epoch 94/99
    60000/60000 [==============================] - 0s - loss: 0.0130 - acc: 0.9185 - val_loss: 0.0124 - val_acc: 0.9218
    Epoch 95/99
    60000/60000 [==============================] - 0s - loss: 0.0130 - acc: 0.9186 - val_loss: 0.0124 - val_acc: 0.9217
    Epoch 96/99
    60000/60000 [==============================] - 0s - loss: 0.0130 - acc: 0.9187 - val_loss: 0.0124 - val_acc: 0.9219
    Epoch 97/99
    60000/60000 [==============================] - 0s - loss: 0.0130 - acc: 0.9189 - val_loss: 0.0124 - val_acc: 0.9221
    Epoch 98/99
    60000/60000 [==============================] - 0s - loss: 0.0129 - acc: 0.9191 - val_loss: 0.0124 - val_acc: 0.9218
    Epoch 99/99
    60000/60000 [==============================] - 0s - loss: 0.0129 - acc: 0.9192 - val_loss: 0.0123 - val_acc: 0.9220

    <keras.callbacks.History at 0x122654668>

```python
# 92% accuracy after another 99 epochs with Relu
# Seems to be a plateau
nn2.evaluate(X_test, y_test)
```

     7552/10000 [=====================>........] - ETA: 0s

    [0.012338483134144916, 0.92200000000000004]

## Loss or cost functions

Loss function
-------------

Sometimes referred to as the **cost function** or **error function**
(not to be confused with the Gauss error function), the loss function
is a function that maps values of one or more variables onto a real
number intuitively representing some “cost” associated with those
values. For backpropagation, the loss function calculates the difference
between the network output and its expected output, after a case
propagates through the network.

### Assumptions

Two assumptions must be made about the form of the error function.

The first is that it can be written as an average
$E=\frac{1}{n}\sum_xE_x$ over error functions $E_x$, for individual
training examples, $x$. The reason for this assumption is that the
backpropagation algorithm calculates the gradient of the error function
for a single training example, which needs to be generalized to the
overall error function. The second assumption is that it can be written
as a function of the outputs from the neural network.

### Example loss function

Let $y,y'$ be vectors in $\mathbb{R}^n$.

Select an error function $E(y,y')$ measuring the difference between two
outputs.

The standard choice is $E(y,y') = \tfrac{1}{2} \lVert y-y'\rVert^2$,

the square of the Euclidean distance between the vectors $y$ and $y'$.

The factor of $\tfrac{1}{2}$ conveniently cancels the exponent when the
error function is subsequently differentiated.

The error function over $n$ training examples can be written as an
average$$E=\frac{1}{2n}\sum_x\lVert (y(x)-y'(x)) \rVert^2$$And the
partial derivative with respect to the
outputs$$\frac{\partial E}{\partial y'} = y'-y$$

## Cross entropy

In information theory, the [cross entropy](https://en.wikipedia.org/wiki/Cross_entropy) between two probability
distributions $p$ and $q$ over the same underlying set of events
measures the average number of bits needed to identify an event drawn
from the set, if a coding scheme is used that is optimized for an
“unnatural” probability distribution $q$, rather than the “true”
distribution $p$.

The cross entropy for the distributions $p$ and $q$ over a given set is
defined as follows:

$$H(p, q) = \operatorname{E}_p[-\log q] = H(p) + D_{\mathrm{KL}}(p \| q),\!$$

where $H(p)$ is the entropy of $p$, and $D_{\mathrm{KL}}(p \| q)$ is
the [Kullback–Leibler divergence](https://en.wikipedia.org/wiki/Kullback%E2%80%93Leibler_divergence) of $q$ from $p$ (also known as the
*relative entropy* of *p* with respect to *q* — note the reversal of
emphasis).

For discrete $p$ and $q$ this means

$$H(p, q) = -\sum_x p(x)\, \log q(x). \!$$

The situation for continuous distributions is analogous. We have to
assume that $p$ and $q$ are [absolutely continuous] with respect to some
reference [measure] $r$ (usually $r$ is a Lebesgue measure on a
Borel [σ-algebra]). Let $P$ and $Q$ be probability density functions
of $p$ and $q$ with respect to $r$. Then

$$-\int_X P(x)\, \log Q(x)\, dr(x) = \operatorname{E}_p[-\log Q]. \!$$

NB: The notation $H(p,q)$ is also used for a different concept, the
joint entropy of $p$ and $q$.

```python
def shallow_net_C(n=55,i=784,o=10):
    # create simple one dense layer net
    # default 55 neurons, input 784, output 10
    # Using relu and
    net = Sequential()
    net.add(Dense(n, activation='relu', input_shape=(i,)))
    net.add(Dense(10, activation='softmax'))
    # Compile net
    net.compile(loss='categorical_crossentropy', optimizer=SGD(lr=0.01), metrics=['accuracy'])
    return net
```

```python
nn3=shallow_net_C()
nn3.summary()
```

    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    dense_5 (Dense)              (None, 55)                43175
    _________________________________________________________________
    dense_6 (Dense)              (None, 10)                560
    =================================================================
    Total params: 43,735
    Trainable params: 43,735
    Non-trainable params: 0
    _________________________________________________________________

```python
nn3.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 1.2643 - acc: 0.6854 - val_loss: 0.6699 - val_acc: 0.8520
    Epoch 2/99
    60000/60000 [==============================] - 0s - loss: 0.5697 - acc: 0.8599 - val_loss: 0.4638 - val_acc: 0.8839
    Epoch 3/99
    60000/60000 [==============================] - 0s - loss: 0.4490 - acc: 0.8814 - val_loss: 0.3969 - val_acc: 0.8955
    Epoch 4/99
    60000/60000 [==============================] - 0s - loss: 0.3977 - acc: 0.8917 - val_loss: 0.3615 - val_acc: 0.9024
    Epoch 5/99
    60000/60000 [==============================] - 0s - loss: 0.3674 - acc: 0.8981 - val_loss: 0.3388 - val_acc: 0.9072
    Epoch 6/99
    60000/60000 [==============================] - 0s - loss: 0.3467 - acc: 0.9027 - val_loss: 0.3231 - val_acc: 0.9101
    Epoch 7/99
    60000/60000 [==============================] - 0s - loss: 0.3310 - acc: 0.9069 - val_loss: 0.3107 - val_acc: 0.9134
    Epoch 8/99
    60000/60000 [==============================] - 0s - loss: 0.3183 - acc: 0.9105 - val_loss: 0.3005 - val_acc: 0.9167
    Epoch 9/99
    60000/60000 [==============================] - 0s - loss: 0.3076 - acc: 0.9130 - val_loss: 0.2924 - val_acc: 0.9186
    Epoch 10/99
    60000/60000 [==============================] - 0s - loss: 0.2982 - acc: 0.9162 - val_loss: 0.2848 - val_acc: 0.9202
    Epoch 11/99
    60000/60000 [==============================] - 0s - loss: 0.2901 - acc: 0.9188 - val_loss: 0.2792 - val_acc: 0.9206
    Epoch 12/99
    60000/60000 [==============================] - 0s - loss: 0.2827 - acc: 0.9203 - val_loss: 0.2719 - val_acc: 0.9229
    Epoch 13/99
    60000/60000 [==============================] - 0s - loss: 0.2759 - acc: 0.9220 - val_loss: 0.2664 - val_acc: 0.9238
    Epoch 14/99
    60000/60000 [==============================] - 0s - loss: 0.2698 - acc: 0.9241 - val_loss: 0.2612 - val_acc: 0.9253
    Epoch 15/99
    60000/60000 [==============================] - 0s - loss: 0.2640 - acc: 0.9260 - val_loss: 0.2567 - val_acc: 0.9264
    Epoch 16/99
    60000/60000 [==============================] - 0s - loss: 0.2586 - acc: 0.9267 - val_loss: 0.2519 - val_acc: 0.9285
    Epoch 17/99
    60000/60000 [==============================] - 0s - loss: 0.2536 - acc: 0.9284 - val_loss: 0.2478 - val_acc: 0.9280
    Epoch 18/99
    60000/60000 [==============================] - 0s - loss: 0.2487 - acc: 0.9300 - val_loss: 0.2427 - val_acc: 0.9311
    Epoch 19/99
    60000/60000 [==============================] - 0s - loss: 0.2441 - acc: 0.9315 - val_loss: 0.2392 - val_acc: 0.9334
    Epoch 20/99
    60000/60000 [==============================] - 0s - loss: 0.2395 - acc: 0.9329 - val_loss: 0.2352 - val_acc: 0.9348
    Epoch 21/99
    60000/60000 [==============================] - 0s - loss: 0.2355 - acc: 0.9338 - val_loss: 0.2314 - val_acc: 0.9351
    Epoch 22/99
    60000/60000 [==============================] - 0s - loss: 0.2313 - acc: 0.9348 - val_loss: 0.2278 - val_acc: 0.9350
    Epoch 23/99
    60000/60000 [==============================] - 0s - loss: 0.2271 - acc: 0.9366 - val_loss: 0.2257 - val_acc: 0.9364
    Epoch 24/99
    60000/60000 [==============================] - 0s - loss: 0.2233 - acc: 0.9375 - val_loss: 0.2213 - val_acc: 0.9365
    Epoch 25/99
    60000/60000 [==============================] - 0s - loss: 0.2196 - acc: 0.9386 - val_loss: 0.2176 - val_acc: 0.9382
    Epoch 26/99
    60000/60000 [==============================] - 0s - loss: 0.2161 - acc: 0.9393 - val_loss: 0.2141 - val_acc: 0.9388
    Epoch 27/99
    60000/60000 [==============================] - 0s - loss: 0.2124 - acc: 0.9403 - val_loss: 0.2108 - val_acc: 0.9396
    Epoch 28/99
    60000/60000 [==============================] - 0s - loss: 0.2092 - acc: 0.9412 - val_loss: 0.2085 - val_acc: 0.9403
    Epoch 29/99
    60000/60000 [==============================] - 0s - loss: 0.2059 - acc: 0.9423 - val_loss: 0.2053 - val_acc: 0.9405
    Epoch 30/99
    60000/60000 [==============================] - 0s - loss: 0.2029 - acc: 0.9432 - val_loss: 0.2025 - val_acc: 0.9413
    Epoch 31/99
    60000/60000 [==============================] - 0s - loss: 0.1997 - acc: 0.9438 - val_loss: 0.1996 - val_acc: 0.9425
    Epoch 32/99
    60000/60000 [==============================] - 0s - loss: 0.1967 - acc: 0.9449 - val_loss: 0.1968 - val_acc: 0.9435
    Epoch 33/99
    60000/60000 [==============================] - 0s - loss: 0.1938 - acc: 0.9456 - val_loss: 0.1949 - val_acc: 0.9446
    Epoch 34/99
    60000/60000 [==============================] - 0s - loss: 0.1910 - acc: 0.9468 - val_loss: 0.1924 - val_acc: 0.9453
    Epoch 35/99
    60000/60000 [==============================] - 0s - loss: 0.1883 - acc: 0.9474 - val_loss: 0.1897 - val_acc: 0.9456
    Epoch 36/99
    60000/60000 [==============================] - 0s - loss: 0.1856 - acc: 0.9481 - val_loss: 0.1874 - val_acc: 0.9465
    Epoch 37/99
    60000/60000 [==============================] - 0s - loss: 0.1831 - acc: 0.9490 - val_loss: 0.1845 - val_acc: 0.9474
    Epoch 38/99
    60000/60000 [==============================] - 0s - loss: 0.1806 - acc: 0.9500 - val_loss: 0.1830 - val_acc: 0.9477
    Epoch 39/99
    60000/60000 [==============================] - 0s - loss: 0.1781 - acc: 0.9500 - val_loss: 0.1809 - val_acc: 0.9487
    Epoch 40/99
    60000/60000 [==============================] - 0s - loss: 0.1757 - acc: 0.9508 - val_loss: 0.1781 - val_acc: 0.9488
    Epoch 41/99
    60000/60000 [==============================] - 0s - loss: 0.1734 - acc: 0.9513 - val_loss: 0.1762 - val_acc: 0.9494
    Epoch 42/99
    60000/60000 [==============================] - 0s - loss: 0.1711 - acc: 0.9518 - val_loss: 0.1748 - val_acc: 0.9498
    Epoch 43/99
    60000/60000 [==============================] - 0s - loss: 0.1689 - acc: 0.9525 - val_loss: 0.1730 - val_acc: 0.9501
    Epoch 44/99
    60000/60000 [==============================] - 0s - loss: 0.1667 - acc: 0.9538 - val_loss: 0.1715 - val_acc: 0.9512
    Epoch 45/99
    60000/60000 [==============================] - 0s - loss: 0.1646 - acc: 0.9537 - val_loss: 0.1686 - val_acc: 0.9509
    Epoch 46/99
    60000/60000 [==============================] - 0s - loss: 0.1625 - acc: 0.9539 - val_loss: 0.1669 - val_acc: 0.9518
    Epoch 47/99
    60000/60000 [==============================] - 0s - loss: 0.1606 - acc: 0.9549 - val_loss: 0.1658 - val_acc: 0.9521
    Epoch 48/99
    60000/60000 [==============================] - 0s - loss: 0.1586 - acc: 0.9550 - val_loss: 0.1630 - val_acc: 0.9532
    Epoch 49/99
    60000/60000 [==============================] - 0s - loss: 0.1567 - acc: 0.9557 - val_loss: 0.1618 - val_acc: 0.9533
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.1548 - acc: 0.9564 - val_loss: 0.1606 - val_acc: 0.9536
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.1530 - acc: 0.9566 - val_loss: 0.1584 - val_acc: 0.9548
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.1512 - acc: 0.9570 - val_loss: 0.1575 - val_acc: 0.9550
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.1494 - acc: 0.9576 - val_loss: 0.1559 - val_acc: 0.9555
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.1477 - acc: 0.9582 - val_loss: 0.1538 - val_acc: 0.9563
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.1460 - acc: 0.9586 - val_loss: 0.1536 - val_acc: 0.9556
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.1444 - acc: 0.9589 - val_loss: 0.1513 - val_acc: 0.9570
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.1428 - acc: 0.9592 - val_loss: 0.1496 - val_acc: 0.9573
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.1412 - acc: 0.9598 - val_loss: 0.1487 - val_acc: 0.9579
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.1397 - acc: 0.9605 - val_loss: 0.1471 - val_acc: 0.9577
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.1381 - acc: 0.9609 - val_loss: 0.1462 - val_acc: 0.9595
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.1367 - acc: 0.9613 - val_loss: 0.1446 - val_acc: 0.9587
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.1352 - acc: 0.9618 - val_loss: 0.1438 - val_acc: 0.9601
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.1337 - acc: 0.9621 - val_loss: 0.1430 - val_acc: 0.9606
    Epoch 64/99
    60000/60000 [==============================] - 1s - loss: 0.1325 - acc: 0.9627 - val_loss: 0.1418 - val_acc: 0.9609
    Epoch 65/99
    60000/60000 [==============================] - 1s - loss: 0.1312 - acc: 0.9628 - val_loss: 0.1400 - val_acc: 0.9601
    Epoch 66/99
    60000/60000 [==============================] - 1s - loss: 0.1298 - acc: 0.9633 - val_loss: 0.1395 - val_acc: 0.9614
    Epoch 67/99
    60000/60000 [==============================] - 1s - loss: 0.1285 - acc: 0.9637 - val_loss: 0.1380 - val_acc: 0.9610
    Epoch 68/99
    60000/60000 [==============================] - 1s - loss: 0.1273 - acc: 0.9644 - val_loss: 0.1369 - val_acc: 0.9624
    Epoch 69/99
    60000/60000 [==============================] - 1s - loss: 0.1261 - acc: 0.9642 - val_loss: 0.1360 - val_acc: 0.9621
    Epoch 70/99
    60000/60000 [==============================] - 1s - loss: 0.1249 - acc: 0.9647 - val_loss: 0.1346 - val_acc: 0.9624
    Epoch 71/99
    60000/60000 [==============================] - 1s - loss: 0.1238 - acc: 0.9650 - val_loss: 0.1340 - val_acc: 0.9634
    Epoch 72/99
    60000/60000 [==============================] - 1s - loss: 0.1227 - acc: 0.9655 - val_loss: 0.1328 - val_acc: 0.9633
    Epoch 73/99
    60000/60000 [==============================] - 1s - loss: 0.1215 - acc: 0.9657 - val_loss: 0.1319 - val_acc: 0.9631
    Epoch 74/99
    60000/60000 [==============================] - 1s - loss: 0.1204 - acc: 0.9657 - val_loss: 0.1313 - val_acc: 0.9643
    Epoch 75/99
    60000/60000 [==============================] - 1s - loss: 0.1194 - acc: 0.9664 - val_loss: 0.1303 - val_acc: 0.9646
    Epoch 76/99
    60000/60000 [==============================] - 1s - loss: 0.1183 - acc: 0.9668 - val_loss: 0.1297 - val_acc: 0.9643
    Epoch 77/99
    60000/60000 [==============================] - 1s - loss: 0.1173 - acc: 0.9673 - val_loss: 0.1286 - val_acc: 0.9648
    Epoch 78/99
    60000/60000 [==============================] - 1s - loss: 0.1162 - acc: 0.9673 - val_loss: 0.1283 - val_acc: 0.9645
    Epoch 79/99
    60000/60000 [==============================] - 1s - loss: 0.1154 - acc: 0.9674 - val_loss: 0.1270 - val_acc: 0.9649
    Epoch 80/99
    60000/60000 [==============================] - 1s - loss: 0.1143 - acc: 0.9679 - val_loss: 0.1263 - val_acc: 0.9653
    Epoch 81/99
    60000/60000 [==============================] - 1s - loss: 0.1134 - acc: 0.9681 - val_loss: 0.1255 - val_acc: 0.9654
    Epoch 82/99
    60000/60000 [==============================] - 1s - loss: 0.1124 - acc: 0.9686 - val_loss: 0.1247 - val_acc: 0.9651
    Epoch 83/99
    60000/60000 [==============================] - 1s - loss: 0.1114 - acc: 0.9686 - val_loss: 0.1249 - val_acc: 0.9653
    Epoch 84/99
    60000/60000 [==============================] - 1s - loss: 0.1107 - acc: 0.9689 - val_loss: 0.1236 - val_acc: 0.9658
    Epoch 85/99
    60000/60000 [==============================] - 1s - loss: 0.1097 - acc: 0.9695 - val_loss: 0.1231 - val_acc: 0.9658
    Epoch 86/99
    60000/60000 [==============================] - 1s - loss: 0.1089 - acc: 0.9694 - val_loss: 0.1219 - val_acc: 0.9669
    Epoch 87/99
    60000/60000 [==============================] - 1s - loss: 0.1080 - acc: 0.9702 - val_loss: 0.1220 - val_acc: 0.9661
    Epoch 88/99
    60000/60000 [==============================] - 1s - loss: 0.1072 - acc: 0.9702 - val_loss: 0.1208 - val_acc: 0.9672
    Epoch 89/99
    60000/60000 [==============================] - 1s - loss: 0.1063 - acc: 0.9704 - val_loss: 0.1203 - val_acc: 0.9671
    Epoch 90/99
    60000/60000 [==============================] - 1s - loss: 0.1056 - acc: 0.9704 - val_loss: 0.1193 - val_acc: 0.9678
    Epoch 91/99
    60000/60000 [==============================] - 1s - loss: 0.1047 - acc: 0.9708 - val_loss: 0.1192 - val_acc: 0.9673
    Epoch 92/99
    60000/60000 [==============================] - 1s - loss: 0.1039 - acc: 0.9712 - val_loss: 0.1182 - val_acc: 0.9669
    Epoch 93/99
    60000/60000 [==============================] - 1s - loss: 0.1032 - acc: 0.9711 - val_loss: 0.1179 - val_acc: 0.9673
    Epoch 94/99
    60000/60000 [==============================] - 1s - loss: 0.1024 - acc: 0.9715 - val_loss: 0.1172 - val_acc: 0.9674
    Epoch 95/99
    60000/60000 [==============================] - 1s - loss: 0.1017 - acc: 0.9714 - val_loss: 0.1169 - val_acc: 0.9678
    Epoch 96/99
    60000/60000 [==============================] - 1s - loss: 0.1010 - acc: 0.9719 - val_loss: 0.1160 - val_acc: 0.9676
    Epoch 97/99
    60000/60000 [==============================] - 1s - loss: 0.1003 - acc: 0.9720 - val_loss: 0.1155 - val_acc: 0.9676
    Epoch 98/99
    60000/60000 [==============================] - 1s - loss: 0.0995 - acc: 0.9724 - val_loss: 0.1160 - val_acc: 0.9675
    Epoch 99/99
    60000/60000 [==============================] - 1s - loss: 0.0989 - acc: 0.9725 - val_loss: 0.1150 - val_acc: 0.9679

    <keras.callbacks.History at 0x12e5bdeb8>

```python
# 96% accuracy after first 99 epochs with Relu and Cross-entropy
nn3.evaluate(X_test, y_test)
```

     8800/10000 [=========================>....] - ETA: 0s

    [0.11497668759040534, 0.96789999999999998]

```python
nn3.fit(X_train, y_train, batch_size=128, epochs=99, verbose=1, validation_data=(X_test, y_test))
```

    Train on 60000 samples, validate on 10000 samples
    Epoch 1/99
    60000/60000 [==============================] - 1s - loss: 0.0981 - acc: 0.9723 - val_loss: 0.1141 - val_acc: 0.9674
    Epoch 2/99
    60000/60000 [==============================] - 1s - loss: 0.0976 - acc: 0.9727 - val_loss: 0.1138 - val_acc: 0.9679
    Epoch 3/99
    60000/60000 [==============================] - 1s - loss: 0.0968 - acc: 0.9731 - val_loss: 0.1129 - val_acc: 0.9679
    Epoch 4/99
    60000/60000 [==============================] - 1s - loss: 0.0962 - acc: 0.9731 - val_loss: 0.1129 - val_acc: 0.9676
    Epoch 5/99
    60000/60000 [==============================] - 1s - loss: 0.0955 - acc: 0.9735 - val_loss: 0.1120 - val_acc: 0.9677
    Epoch 6/99
    60000/60000 [==============================] - 1s - loss: 0.0949 - acc: 0.9735 - val_loss: 0.1119 - val_acc: 0.9679
    Epoch 7/99
    60000/60000 [==============================] - 1s - loss: 0.0943 - acc: 0.9738 - val_loss: 0.1108 - val_acc: 0.9681
    Epoch 8/99
    60000/60000 [==============================] - 1s - loss: 0.0936 - acc: 0.9741 - val_loss: 0.1108 - val_acc: 0.9673
    Epoch 9/99
    60000/60000 [==============================] - 1s - loss: 0.0931 - acc: 0.9744 - val_loss: 0.1102 - val_acc: 0.9679
    Epoch 10/99
    60000/60000 [==============================] - 1s - loss: 0.0924 - acc: 0.9742 - val_loss: 0.1102 - val_acc: 0.9678
    Epoch 11/99
    60000/60000 [==============================] - 1s - loss: 0.0919 - acc: 0.9744 - val_loss: 0.1097 - val_acc: 0.9680
    Epoch 12/99
    60000/60000 [==============================] - 1s - loss: 0.0913 - acc: 0.9745 - val_loss: 0.1090 - val_acc: 0.9679
    Epoch 13/99
    60000/60000 [==============================] - 1s - loss: 0.0907 - acc: 0.9748 - val_loss: 0.1089 - val_acc: 0.9683
    Epoch 14/99
    60000/60000 [==============================] - 1s - loss: 0.0902 - acc: 0.9750 - val_loss: 0.1085 - val_acc: 0.9685
    Epoch 15/99
    60000/60000 [==============================] - 1s - loss: 0.0895 - acc: 0.9754 - val_loss: 0.1082 - val_acc: 0.9684
    Epoch 16/99
    60000/60000 [==============================] - 1s - loss: 0.0890 - acc: 0.9755 - val_loss: 0.1072 - val_acc: 0.9688
    Epoch 17/99
    60000/60000 [==============================] - 1s - loss: 0.0885 - acc: 0.9757 - val_loss: 0.1070 - val_acc: 0.9690
    Epoch 18/99
    60000/60000 [==============================] - 1s - loss: 0.0879 - acc: 0.9756 - val_loss: 0.1065 - val_acc: 0.9692
    Epoch 19/99
    60000/60000 [==============================] - 1s - loss: 0.0874 - acc: 0.9758 - val_loss: 0.1064 - val_acc: 0.9693
    Epoch 20/99
    60000/60000 [==============================] - 1s - loss: 0.0869 - acc: 0.9758 - val_loss: 0.1058 - val_acc: 0.9689
    Epoch 21/99
    60000/60000 [==============================] - 1s - loss: 0.0863 - acc: 0.9761 - val_loss: 0.1054 - val_acc: 0.9692
    Epoch 22/99
    60000/60000 [==============================] - 1s - loss: 0.0858 - acc: 0.9764 - val_loss: 0.1052 - val_acc: 0.9690
    Epoch 23/99
    60000/60000 [==============================] - 1s - loss: 0.0854 - acc: 0.9763 - val_loss: 0.1045 - val_acc: 0.9693
    Epoch 24/99
    60000/60000 [==============================] - 1s - loss: 0.0848 - acc: 0.9765 - val_loss: 0.1044 - val_acc: 0.9695
    Epoch 25/99
    60000/60000 [==============================] - 1s - loss: 0.0844 - acc: 0.9767 - val_loss: 0.1042 - val_acc: 0.9691
    Epoch 26/99
    60000/60000 [==============================] - 1s - loss: 0.0839 - acc: 0.9767 - val_loss: 0.1045 - val_acc: 0.9687
    Epoch 27/99
    60000/60000 [==============================] - 1s - loss: 0.0834 - acc: 0.9767 - val_loss: 0.1035 - val_acc: 0.9696
    Epoch 28/99
    60000/60000 [==============================] - 1s - loss: 0.0829 - acc: 0.9772 - val_loss: 0.1035 - val_acc: 0.9700
    Epoch 29/99
    60000/60000 [==============================] - 1s - loss: 0.0824 - acc: 0.9771 - val_loss: 0.1026 - val_acc: 0.9698
    Epoch 30/99
    60000/60000 [==============================] - 1s - loss: 0.0819 - acc: 0.9773 - val_loss: 0.1025 - val_acc: 0.9698
    Epoch 31/99
    60000/60000 [==============================] - 1s - loss: 0.0815 - acc: 0.9774 - val_loss: 0.1021 - val_acc: 0.9697
    Epoch 32/99
    60000/60000 [==============================] - 1s - loss: 0.0810 - acc: 0.9777 - val_loss: 0.1017 - val_acc: 0.9699
    Epoch 33/99
    60000/60000 [==============================] - 1s - loss: 0.0806 - acc: 0.9776 - val_loss: 0.1016 - val_acc: 0.9703
    Epoch 34/99
    60000/60000 [==============================] - 1s - loss: 0.0801 - acc: 0.9781 - val_loss: 0.1011 - val_acc: 0.9703
    Epoch 35/99
    60000/60000 [==============================] - 1s - loss: 0.0797 - acc: 0.9779 - val_loss: 0.1011 - val_acc: 0.9704
    Epoch 36/99
    60000/60000 [==============================] - 1s - loss: 0.0793 - acc: 0.9783 - val_loss: 0.1006 - val_acc: 0.9701
    Epoch 37/99
    60000/60000 [==============================] - 1s - loss: 0.0789 - acc: 0.9782 - val_loss: 0.1001 - val_acc: 0.9706
    Epoch 38/99
    60000/60000 [==============================] - 1s - loss: 0.0785 - acc: 0.9783 - val_loss: 0.1003 - val_acc: 0.9707
    Epoch 39/99
    60000/60000 [==============================] - 1s - loss: 0.0779 - acc: 0.9785 - val_loss: 0.1005 - val_acc: 0.9699
    Epoch 40/99
    60000/60000 [==============================] - 1s - loss: 0.0776 - acc: 0.9784 - val_loss: 0.1003 - val_acc: 0.9702
    Epoch 41/99
    60000/60000 [==============================] - 1s - loss: 0.0772 - acc: 0.9785 - val_loss: 0.0990 - val_acc: 0.9709
    Epoch 42/99
    60000/60000 [==============================] - 1s - loss: 0.0768 - acc: 0.9790 - val_loss: 0.0989 - val_acc: 0.9707
    Epoch 43/99
    60000/60000 [==============================] - 1s - loss: 0.0764 - acc: 0.9789 - val_loss: 0.0986 - val_acc: 0.9706
    Epoch 44/99
    60000/60000 [==============================] - 1s - loss: 0.0759 - acc: 0.9790 - val_loss: 0.0985 - val_acc: 0.9710
    Epoch 45/99
    60000/60000 [==============================] - 1s - loss: 0.0755 - acc: 0.9793 - val_loss: 0.0982 - val_acc: 0.9709
    Epoch 46/99
    60000/60000 [==============================] - 1s - loss: 0.0752 - acc: 0.9793 - val_loss: 0.0978 - val_acc: 0.9707
    Epoch 47/99
    60000/60000 [==============================] - 1s - loss: 0.0748 - acc: 0.9794 - val_loss: 0.0979 - val_acc: 0.9709
    Epoch 48/99
    60000/60000 [==============================] - 1s - loss: 0.0744 - acc: 0.9794 - val_loss: 0.0976 - val_acc: 0.9711
    Epoch 49/99
    60000/60000 [==============================] - 1s - loss: 0.0740 - acc: 0.9797 - val_loss: 0.0977 - val_acc: 0.9704
    Epoch 50/99
    60000/60000 [==============================] - 1s - loss: 0.0736 - acc: 0.9798 - val_loss: 0.0966 - val_acc: 0.9719
    Epoch 51/99
    60000/60000 [==============================] - 1s - loss: 0.0733 - acc: 0.9798 - val_loss: 0.0966 - val_acc: 0.9710
    Epoch 52/99
    60000/60000 [==============================] - 1s - loss: 0.0729 - acc: 0.9800 - val_loss: 0.0966 - val_acc: 0.9709
    Epoch 53/99
    60000/60000 [==============================] - 1s - loss: 0.0726 - acc: 0.9801 - val_loss: 0.0964 - val_acc: 0.9712
    Epoch 54/99
    60000/60000 [==============================] - 1s - loss: 0.0722 - acc: 0.9803 - val_loss: 0.0962 - val_acc: 0.9712
    Epoch 55/99
    60000/60000 [==============================] - 1s - loss: 0.0718 - acc: 0.9803 - val_loss: 0.0956 - val_acc: 0.9717
    Epoch 56/99
    60000/60000 [==============================] - 1s - loss: 0.0715 - acc: 0.9806 - val_loss: 0.0958 - val_acc: 0.9714
    Epoch 57/99
    60000/60000 [==============================] - 1s - loss: 0.0711 - acc: 0.9804 - val_loss: 0.0954 - val_acc: 0.9719
    Epoch 58/99
    60000/60000 [==============================] - 1s - loss: 0.0707 - acc: 0.9807 - val_loss: 0.0955 - val_acc: 0.9719
    Epoch 59/99
    60000/60000 [==============================] - 1s - loss: 0.0704 - acc: 0.9808 - val_loss: 0.0948 - val_acc: 0.9718
    Epoch 60/99
    60000/60000 [==============================] - 1s - loss: 0.0700 - acc: 0.9807 - val_loss: 0.0946 - val_acc: 0.9715
    Epoch 61/99
    60000/60000 [==============================] - 1s - loss: 0.0698 - acc: 0.9811 - val_loss: 0.0943 - val_acc: 0.9715
    Epoch 62/99
    60000/60000 [==============================] - 1s - loss: 0.0694 - acc: 0.9811 - val_loss: 0.0941 - val_acc: 0.9721
    Epoch 63/99
    60000/60000 [==============================] - 1s - loss: 0.0690 - acc: 0.9814 - val_loss: 0.0942 - val_acc: 0.9721
    Epoch 64/99
    60000/60000 [==============================] - 1s - loss: 0.0687 - acc: 0.9815 - val_loss: 0.0936 - val_acc: 0.9725
    Epoch 65/99
    60000/60000 [==============================] - 1s - loss: 0.0684 - acc: 0.9814 - val_loss: 0.0942 - val_acc: 0.9724
    Epoch 66/99
    60000/60000 [==============================] - 1s - loss: 0.0681 - acc: 0.9817 - val_loss: 0.0935 - val_acc: 0.9719
    Epoch 67/99
    60000/60000 [==============================] - 1s - loss: 0.0678 - acc: 0.9818 - val_loss: 0.0932 - val_acc: 0.9724
    Epoch 68/99
    60000/60000 [==============================] - 1s - loss: 0.0674 - acc: 0.9817 - val_loss: 0.0931 - val_acc: 0.9725
    Epoch 69/99
    60000/60000 [==============================] - 1s - loss: 0.0671 - acc: 0.9817 - val_loss: 0.0933 - val_acc: 0.9718
    Epoch 70/99
    60000/60000 [==============================] - 1s - loss: 0.0668 - acc: 0.9822 - val_loss: 0.0931 - val_acc: 0.9727
    Epoch 71/99
    60000/60000 [==============================] - 1s - loss: 0.0665 - acc: 0.9821 - val_loss: 0.0924 - val_acc: 0.9724
    Epoch 72/99
    60000/60000 [==============================] - 1s - loss: 0.0662 - acc: 0.9823 - val_loss: 0.0924 - val_acc: 0.9726
    Epoch 73/99
    60000/60000 [==============================] - 1s - loss: 0.0659 - acc: 0.9821 - val_loss: 0.0922 - val_acc: 0.9726
    Epoch 74/99
    60000/60000 [==============================] - 1s - loss: 0.0656 - acc: 0.9823 - val_loss: 0.0919 - val_acc: 0.9725
    Epoch 75/99
    60000/60000 [==============================] - 1s - loss: 0.0654 - acc: 0.9824 - val_loss: 0.0920 - val_acc: 0.9727
    Epoch 76/99
    60000/60000 [==============================] - 1s - loss: 0.0650 - acc: 0.9825 - val_loss: 0.0917 - val_acc: 0.9727
    Epoch 77/99
    60000/60000 [==============================] - 1s - loss: 0.0647 - acc: 0.9825 - val_loss: 0.0915 - val_acc: 0.9727
    Epoch 78/99
    60000/60000 [==============================] - 1s - loss: 0.0644 - acc: 0.9828 - val_loss: 0.0917 - val_acc: 0.9725
    Epoch 79/99
    60000/60000 [==============================] - 1s - loss: 0.0641 - acc: 0.9827 - val_loss: 0.0911 - val_acc: 0.9728
    Epoch 80/99
     5888/60000 [=>............................] - ETA: 0s - loss: 0.0604 - acc: 0.9845

```python
# 97% accuracy after another 99 epochs with Relu and Cross-entropy
nn3.evaluate(X_test, y_test)
```

## Summary

With a fairly simple shallow net we've done fairly well classifying (97% accuracy after another 99 epochs with Relu and Cross-entropy) on the [MNIST](http://yann.lecun.com/exdb/mnist/)  handwritten digit classification problem.

Last update September 22, 2017