# Inference and Machine Learning : Berlemont Kevin

## Statistical inference and MSE

We consider a system with a half-length decay :
$$ P_{\lambda} (x) =  \frac{e^{-x / \lambda}}{Z(\lambda)} \quad \text{if $ 1<x<20$}$$
$$ P_{\lambda} (x) = 0$$

For normalization, the constant $Z(\lambda)$ will be :
$$ Z(\lambda) = (\lambda e^{-1/\lambda} - e^{-20/\lambda} \lambda)^{-1} $$

If we want to observe a set of events $\{ x_1,\cdots, x_n \}$, because all particles are independent we have the relation :
$$ \{ x\} = \otimes_i \{ x_i \}$$
This leads easily to :
$$ P_{\lambda} (x) = \Pi_i P_{\lambda} (x_i)$$

```python
from IPython.display import HTML

HTML('''
The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click <a href="javascript:code_toggle()">here</a>.''')
```

The raw code for this IPython notebook is by default hidden for easier reading.
To toggle on/off the raw code, click here.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats as ss
% matplotlib inline

def trunc_exp_decay(low, high, scale, size): # inverse transfor msampling
    rnd_cdf = np.random.uniform(ss.expon.cdf(x=low, scale=scale),
                                ss.expon.cdf(x=high, scale=scale),
                                size=size)
    return ss.expon.ppf(q=rnd_cdf, scale=scale)

Lambda =5
Size = 600
plt.hist(trunc_exp_decay(1, 20, Lambda, Size))
plt.xlim(0, 20)
plt.title('Histogram of the probability distribution function')
```

```python
def observations(Lambda,n) :
    """Return n observations of the process"""
    sample = trunc_exp_decay(1, 20, Lambda, n)
    return sample
```

We now want to estimate the value of $\lambda$. First of all we use the maximum likehood estimator and we will optimize the value with the module scipy.

```python
import scipy
import scipy.stats as sciStat
import scipy.optimize as sciOpt

sample =observations(5,100)
def myMleEstimate(myFunc,par,data):

 def lnL_av(x,par):
  N = len(x)

  #Increase your efficiency using Scientific Python (scipy)
  lnL = 0.
  for i in range(N):
   lnL += scipy.log(myFunc(par,x[i]))
  return lnL/N

 objFunc = lambda s: -lnL_av(data,s)
 par_mle = sciOpt.minimize(objFunc,par,method='nelder-mead')
 return par_mle
```

```python
Decay = lambda s,x: (np.exp(-x/s))/(s*(np.exp(-1/s) - np.exp(-20/s)))
objFunc = lambda x: myMleEstimate(Decay,1,x)
sample =observations(5,1000)
objFunc(sample)
```

      status: 0
        nfev: 40
     success: True
         fun: 2.4604431572741339
           x: array([ 4.71884766])
     message: 'Optimization terminated successfully.'
         nit: 20

```python
sample_size=np.arange(10,500,20)
Lambda_list=[1,2,5]
N=len(sample_size)
estimator = np.zeros((len(Lambda_list),N))
for i in range(N):
    estimator[0][i] = objFunc(observations(5,sample_size[i])).x

for i in range(N):
    estimator[1][i] = objFunc(observations(2,sample_size[i])).x

for i in range(N):
    estimator[2][i] = objFunc(observations(1,sample_size[i])).x

fig=plt.figure()
ax = fig.add_subplot(111)
ax.plot(sample_size,estimator[0],'o',label='$\lambda=1$')
ax.plot(sample_size,estimator[1],'o',label='$\lambda=2$')
ax.plot(sample_size,estimator[2],'o',label='$\lambda=5$')
box = ax.get_position()
ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])

# Put a legend to the right of the current axis
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title('Values of the estimator for some $\lambda$ for several sample size')
plt.show()
```

As we can observe, we have indeed convergence of our estimator toward the true value of $\lambda$. We can note that the performance increases with the size of the sample, and we could show that this estimator is unbiased.

We want now to observe the behavior of the MSE with respect to the true value of $\lambda$.

```python
from sklearn.metrics import mean_squared_error

#sample_size=[10,50,100,200,300,500,1000]
sample_size=np.arange(1,50,1)
N=len(sample_size)
Lambda_list=[1,5,10]
y_true = [Lambda]
estimator = np.zeros((len(Lambda_list),N))
MSE_list=np.zeros((len(Lambda_list),N))

for j,Lambda in enumerate(Lambda_list):
 for i in range(N):
    estimator[j,i] = objFunc(observations(Lambda,sample_size[i])).x
    MSE_list[j,i]=mean_squared_error([Lambda], [estimator[j,i]])

 plt.plot(sample_size,MSE_list[j],'o',label='$\lambda=$'+str(Lambda))

plt.ylim((-5,150))
plt.legend()
plt.title('MSE with respect to the sample size for several values of $\lambda$')
```

As we could predict we observe a decreasing of the MSE with respect to the size sample. Indeed we have a consistant estimator so the MSE should decrease to have convergence of our estimator.

We can note that for high $\lambda$ it harder to find the right estiamtor. This comes from the fact that we are dealing with a truncated distribution, and not the full one. When we reach the truncation size, the estimator has difficulty to detect the true value.

```python
#sample_size=[10000]
from sklearn.metrics import mean_squared_error
sample_size=[10000]
N=1
Lambda_list = np.arange(1,20,1)
mean=10
estimator = np.zeros((len(Lambda_list),mean))
MSE_list=np.zeros(len(Lambda_list))

for j,Lambda in enumerate(Lambda_list):

    for k in range(mean):
     estimator[j,k] = objFunc(observations(Lambda,sample_size[0])).x
    MSE_list[j]=mean_squared_error([Lambda]*mean, estimator[j])

plt.plot(Lambda_list,MSE_list,'o')
plt.title('MSE with respect to $\lambda$')
```

```python
from IPython.display import Image
Image("mse_estime.png")
```

As we can observe for small $\lambda$ it follows the MSE, which is good because it shows that our estimator is consistant. However as we increase $\lambda$ the error increase even if we are optimal with respect ot the bound.

This can be explain by the fact that we are trying to estimate the parameter of a truncated distribution. For such distribution it is important to notice that as we reach the truncated value, the ML estimator can be unefficient sometimes because we don't have enough values to distinguish the several $\lambda$ possible.

But far from the boundary, our estimator is really good.

We want now to study the MAP estimator. The idea behing this kind of estimator is to estimate $\lambda$ as the mode of the posterior distribution. If the prior is uniform, then the MAP estiamtor coincide with the ML estimator, as we can easily deduce with the Bayes formula.

The next curve is analog to the one for ML estimator. It seems logic because our prior is the Fisher information. Moreover this estimator is asymptotically unbiased, because the argmax is a function of Fisher information and Likehood. The Likehood is asymptotically unbiased and the Fisher information, for such big sample does not impact the biais because we are taking the expectation.

```python
Image("est_bayes.png")
```

```python
from sympy import *

x=Symbol('x')
Lambda = Symbol('Lambda')

Z = Lambda*(exp(-1/Lambda)-exp(-20/Lambda))
DZ = diff(ln(Z),Lambda,2)
DZ

Z2=ln(exp(-x/Lambda))
DZ2 = diff(Z2,Lambda,2)
DZ2
Rep=DZ2.subs(x,(20*Lambda*exp(-20/Lambda)-Lambda*exp(-1/Lambda))/(Z)+(-20*Lambda*exp(-20/Lambda)+Lambda*exp(-1/Lambda))/(Z/Lambda))

simplify(-Rep+DZ)

```

    (Lambda**2*exp(38/Lambda) - 40*Lambda**2*exp(19/Lambda) + 39*Lambda**2 - 4*Lambda*exp(38/Lambda) + 84*Lambda*exp(19/Lambda) - 80*Lambda - 361*exp(19/Lambda))/(Lambda**4*(exp(38/Lambda) - 2*exp(19/Lambda) + 1))

```python
A=diff(ln(Lambda*exp(x*Lambda)),Lambda,2)
B=diff(ln(exp(-x/Lambda)/(Lambda*(exp(-1/Lambda)-exp(-20/Lambda)))),Lambda,2)
#simplify(A)
Simp=simplify(-B)
Simp
Rep=Simp.subs(x,1/Lambda)
simplify(Rep)
```

## The MNIST Dataset

We will now use the MNIST Dataset to train a machine learning algortihm with neural net.

For now we will use the sigmoid fonction as the transfer function.

We want to find the best estimator to our problem. For this, we use the mean square error as measure of performance of our model and we introduce regularization. Thus the total error function will be :
$$ (Xw -y)^\top (Xw - y) + \frac12 \Gamma w^\top w$$
The optimal estimator will be given for :
$$ \nabla_w (Xw -y)^\top (Xw - y) +  \Gamma w^\top w = 0$$
We expand and find :
$$  X^\top X w -  X^\top y + \Gamma w = 0$$
And finally :
$$ \hat w = (X^\top X + \Gamma I)^{-1} X^\top Y$$

As say at the end of part 2, the method with the matrix has the inconvenient of using a load of RAM when trying to invert it. In order to be able to obtain nice results we will  use a neural network architecture we will perform a stochastic gradient descent on an neural network with the following architecture. The architecture explain why $M$, or $n$ in this case, can be described as a number of hidden neurons. To have different behavior we will use several values of $M$ in the neural networks.
At the end, the learning is the same, but we didn't had to take the inverse of some big matrices, we will compute this method during this homework.

```python
from IPython.display import Image
Image("tikz12.png")
```

```python
"""
mnist_loader
~~~~~~~~~~~~

A library to load the MNIST image data.  For details of the data
structures that are returned, see the doc strings for ``load_data``
and ``load_data_wrapper``.  In practice, ``load_data_wrapper`` is the
function usually called by our neural network code.
"""

#### Libraries
# Standard library
import cPickle
import gzip

# Third-party libraries
import numpy as np

def load_data():
    """Return the MNIST data as a tuple containing the training data,
    the validation data, and the test data.

    The ``training_data`` is returned as a tuple with two entries.
    The first entry contains the actual training images.  This is a
    numpy ndarray with 50,000 entries.  Each entry is, in turn, a
    numpy ndarray with 784 values, representing the 28 * 28 = 784
    pixels in a single MNIST image.

    The second entry in the ``training_data`` tuple is a numpy ndarray
    containing 50,000 entries.  Those entries are just the digit
    values (0...9) for the corresponding images contained in the first
    entry of the tuple.

    The ``validation_data`` and ``test_data`` are similar, except
    each contains only 10,000 images.

    This is a nice data format, but for use in neural networks it's
    helpful to modify the format of the ``training_data`` a little.
    That's done in the wrapper function ``load_data_wrapper()``, see
    below.
    """
    f = gzip.open('./data/mnist.pkl.gz', 'rb')
    training_data, validation_data, test_data = cPickle.load(f)
    f.close()
    return (training_data, validation_data, test_data)

def load_data_wrapper():
    """Return a tuple containing ``(training_data, validation_data,
    test_data)``. Based on ``load_data``, but the format is more
    convenient for use in our implementation of neural networks.

    In particular, ``training_data`` is a list containing 50,000
    2-tuples ``(x, y)``.  ``x`` is a 784-dimensional numpy.ndarray
    containing the input image.  ``y`` is a 10-dimensional
    numpy.ndarray representing the unit vector corresponding to the
    correct digit for ``x``.

    ``validation_data`` and ``test_data`` are lists containing 10,000
    2-tuples ``(x, y)``.  In each case, ``x`` is a 784-dimensional
    numpy.ndarry containing the input image, and ``y`` is the
    corresponding classification, i.e., the digit values (integers)
    corresponding to ``x``.

    Obviously, this means we're using slightly different formats for
    the training data and the validation / test data.  These formats
    turn out to be the most convenient for use in our neural network
    code."""
    tr_d, va_d, te_d = load_data()
    training_inputs = [np.reshape(x, (784, 1)) for x in tr_d[0]]
    training_results = [vectorized_result(y) for y in tr_d[1]]
    training_data = zip(training_inputs, training_results)
    validation_inputs = [np.reshape(x, (784, 1)) for x in va_d[0]]
    validation_data = zip(validation_inputs, va_d[1])
    test_inputs = [np.reshape(x, (784, 1)) for x in te_d[0]]
    test_data = zip(test_inputs, te_d[1])
    return (training_data, validation_data, test_data)

def vectorized_result(j):
    """Return a 10-dimensional unit vector with a 1.0 in the jth
    position and zeroes elsewhere.  This is used to convert a digit
    (0...9) into a corresponding desired output from the neural
    network."""
    e = np.zeros((10, 1))
    e[j] = 1.0
    return e
```

```python

import sys

# My library

# Third-party libraries
import matplotlib
import matplotlib.pyplot as plt
import numpy as np

def plot_10_by_10_images(images):
    """ Plot 100 MNIST images in a 10 by 10 table. Note that we crop
    the images so that they appear reasonably close together.  The
    image is post-processed to give the appearance of being continued."""
    fig = plt.figure()
    images = [image[3:25, 3:25] for image in images]
    #image = np.concatenate(images, axis=1)
    for x in range(10):
        for y in range(10):
            ax = fig.add_subplot(10, 10, 10*y+x)
            ax.matshow(images[10*y+x], cmap = matplotlib.cm.binary)
            plt.xticks(np.array([]))
            plt.yticks(np.array([]))
    plt.show()

def get_images(training_set):
    """ Return a list containing the images from the MNIST data
    set. Each image is represented as a 2-d numpy array."""
    flattened_images = training_set[0]
    return [np.reshape(f, (-1, 28)) for f in flattened_images]

training_set, validation_set, test_set = load_data()
images = get_images(training_set)

```

The MNIST dataset is a set of number as on the next picture. Our goal is to learn to classify the number : from 0 to 9, with a neural network. At the beginning we will use the quadratic cost function and we will add some regularization to have networks which are performing better.

```python
plot_10_by_10_images(images)
```

We will plot the accuracy of the classification for several number of hidden neurons, with some vlaues of the learning parameters. We will then compare the performance for the training data and the evaluation data.

```python
import json
import random
import sys

# Third-party libraries
import numpy as np

#### Define the quadratic and cross-entropy cost functions

class QuadraticCost(object):

    @staticmethod
    def fn(a, y):
        """Return the cost associated with an output ``a`` and desired output
        ``y``.

        """
        return 0.5*np.linalg.norm(a-y)**2

    @staticmethod
    def delta(z, a, y):
        """Return the error delta from the output layer."""
        return (a-y) * sigmoid_prime(z)

#### Main Network class
class Network(object):

    def __init__(self, sizes, cost=QuadraticCost):
        """The list ``sizes`` contains the number of neurons in the respective
        layers of the network.  For example, if the list was [2, 3, 1]
        then it would be a three-layer network, with the first layer
        containing 2 neurons, the second layer 3 neurons, and the
        third layer 1 neuron.  The biases and weights for the network
        are initialized randomly

        """
        self.num_layers = len(sizes)
        self.sizes = sizes
        self.large_weight_initializer()
        self.cost=cost

    def large_weight_initializer(self):
        """Initialize the weights using a Gaussian distribution with mean 0
        and standard deviation 1.  Initialize the biases using a
        Gaussian distribution with mean 0 and standard deviation 1.

        """
        self.biases = [np.random.randn(y, 1) for y in self.sizes[1:]]
        self.weights = [np.random.randn(y, x)
                        for x, y in zip(self.sizes[:-1], self.sizes[1:])]

    def feedforward(self, a):
        """Return the output of the network if ``a`` is input."""
        for b, w in zip(self.biases, self.weights):
            a = sigmoid(np.dot(w, a)+b)
        return a

    def SGD(self, training_data, epochs, mini_batch_size, eta,
            lmbda = 0.0,
            evaluation_data=None,
            monitor_evaluation_cost=False,
            monitor_evaluation_accuracy=False,
            monitor_training_cost=False,
            monitor_training_accuracy=False):
        """Train the neural network using mini-batch stochastic gradient
        descent.  The ``training_data`` is a list of tuples ``(x, y)``
        representing the training inputs and the desired outputs.  The
        other non-optional parameters are self-explanatory, as is the
        regularization parameter ``lmbda``.  The method also accepts
        ``evaluation_data``, usually either the validation or test
        data.  We can monitor the cost and accuracy on either the
        evaluation data or the training data, by setting the
        appropriate flags.  The method returns a tuple containing four
        lists: the (per-epoch) costs on the evaluation data, the
        accuracies on the evaluation data, the costs on the training
        data, and the accuracies on the training data.  All values are
        evaluated at the end of each training epoch.  So, for example,
        if we train for 30 epochs, then the first element of the tuple
        will be a 30-element list containing the cost on the
        evaluation data at the end of each epoch. Note that the lists
        are empty if the corresponding flag is not set.

        """
        if evaluation_data: n_data = len(evaluation_data)
        n = len(training_data)
        evaluation_cost, evaluation_accuracy = [], []
        training_cost, training_accuracy = [], []
        for j in xrange(epochs):
            random.shuffle(training_data)
            mini_batches = [
                training_data[k:k+mini_batch_size]
                for k in xrange(0, n, mini_batch_size)]
            for mini_batch in mini_batches:
                self.update_mini_batch(
                    mini_batch, eta, lmbda, len(training_data))
            print "Epoch %s training complete" % j
            if monitor_training_cost:
                cost = self.total_cost(training_data, lmbda)
                training_cost.append(cost)
                print "Cost on training data: {}".format(cost)
            if monitor_training_accuracy:
                accuracy = self.accuracy(training_data, convert=True)
                training_accuracy.append(accuracy)
                print "Accuracy on training data: {} / {}".format(
                    accuracy, n)
            if monitor_evaluation_cost:
                cost = self.total_cost(evaluation_data, lmbda, convert=True)
                evaluation_cost.append(cost)
                print "Cost on evaluation data: {}".format(cost)
            if monitor_evaluation_accuracy:
                accuracy = self.accuracy(evaluation_data)
                evaluation_accuracy.append(accuracy)
                print "Accuracy on evaluation data: {} / {}".format(
                    self.accuracy(evaluation_data), n_data)
            print
        return evaluation_cost, evaluation_accuracy, \
            training_cost, training_accuracy

    def update_mini_batch(self, mini_batch, eta, lmbda, n):
        """Update the network's weights and biases by applying gradient
        descent using backpropagation to a single mini batch.  The
        ``mini_batch`` is a list of tuples ``(x, y)``, ``eta`` is the
        learning rate, ``lmbda`` is the regularization parameter, and
        ``n`` is the total size of the training data set.

        """
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        for x, y in mini_batch:
            delta_nabla_b, delta_nabla_w = self.backprop(x, y)
            nabla_b = [nb+dnb for nb, dnb in zip(nabla_b, delta_nabla_b)]
            nabla_w = [nw+dnw for nw, dnw in zip(nabla_w, delta_nabla_w)]
        self.weights = [(1-eta*(lmbda/n))*w-(eta/len(mini_batch))*nw
                        for w, nw in zip(self.weights, nabla_w)]
        self.biases = [b-(eta/len(mini_batch))*nb
                       for b, nb in zip(self.biases, nabla_b)]

    def backprop(self, x, y):
        """Return a tuple ``(nabla_b, nabla_w)`` representing the
        gradient for the cost function C_x.  ``nabla_b`` and
        ``nabla_w`` are layer-by-layer lists of numpy arrays, similar
        to ``self.biases`` and ``self.weights``."""
        nabla_b = [np.zeros(b.shape) for b in self.biases]
        nabla_w = [np.zeros(w.shape) for w in self.weights]
        # feedforward
        activation = x
        activations = [x] # list to store all the activations, layer by layer
        zs = [] # list to store all the z vectors, layer by layer
        for b, w in zip(self.biases, self.weights):
            z = np.dot(w, activation)+b
            zs.append(z)
            activation = sigmoid(z)
            activations.append(activation)
        # backward pass
        delta = (self.cost).delta(zs[-1], activations[-1], y)
        nabla_b[-1] = delta
        nabla_w[-1] = np.dot(delta, activations[-2].transpose())

        for l in xrange(2, self.num_layers):
            z = zs[-l]
            sp = sigmoid_prime(z)
            delta = np.dot(self.weights[-l+1].transpose(), delta) * sp
            nabla_b[-l] = delta
            nabla_w[-l] = np.dot(delta, activations[-l-1].transpose())
        return (nabla_b, nabla_w)

    def accuracy(self, data, convert=False):
        """Return the number of inputs in ``data`` for which the neural
        network outputs the correct result. The neural network's
        output is assumed to be the index of whichever neuron in the
        final layer has the highest activation.

        The flag ``convert`` should be set to False if the data set is
        validation or test data (the usual case), and to True if the
        data set is the training data. The need for this flag arises
        due to differences in the way the results ``y`` are
        represented in the different data sets.  In particular, it
        flags whether we need to convert between the different
        representations.

        """
        if convert:
            results = [(np.argmax(self.feedforward(x)), np.argmax(y))
                       for (x, y) in data]
        else:
            results = [(np.argmax(self.feedforward(x)), y)
                        for (x, y) in data]
        return sum(int(x == y) for (x, y) in results)

    def total_cost(self, data, lmbda, convert=False):
        """Return the total cost for the data set ``data``.  The flag
        ``convert`` should be set to False if the data set is the
        training data (the usual case), and to True if the data set is
        the validation or test data.
        """
        cost = 0.0
        for x, y in data:
            a = self.feedforward(x)
            if convert: y = vectorized_result(y)
            cost += self.cost.fn(a, y)/len(data)
        cost += 0.5*(lmbda/len(data))*sum(
            np.linalg.norm(w)**2 for w in self.weights)
        return cost

    def save(self, filename):
        """Save the neural network to the file ``filename``."""
        data = {"sizes": self.sizes,
                "weights": [w.tolist() for w in self.weights],
                "biases": [b.tolist() for b in self.biases],
                "cost": str(self.cost.__name__)}
        f = open(filename, "w")
        json.dump(data, f)
        f.close()

#### Loading a Network
def load(filename):
    """Load a neural network from the file ``filename``.
    """
    f = open(filename, "r")
    data = json.load(f)
    f.close()
    cost = getattr(sys.modules[__name__], data["cost"])
    net = Network(data["sizes"], cost=cost)
    net.weights = [np.array(w) for w in data["weights"]]
    net.biases = [np.array(b) for b in data["biases"]]
    return net

#### Miscellaneous functions
def vectorized_result(j):
    """Return a 10-dimensional unit vector with a 1.0 in the j'th position
    and zeroes elsewhere.  This is used to convert a digit (0...9)
    into a corresponding desired output from the neural network.

    """
    e = np.zeros((10, 1))
    e[j] = 1.0
    return e

def sigmoid(z):
    """The sigmoid function."""
    return 1.0/(1.0+np.exp(-z))

def sigmoid_prime(z):
    """Derivative of the sigmoid function."""
    return sigmoid(z)*(1-sigmoid(z))
```

```python
training_data, validation_data, test_data = load_data_wrapper()
```

```python

def plot_test_accuracy(test_accuracy, num_epochs, test_accuracy_xmin):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.plot(np.arange(test_accuracy_xmin, num_epochs),
            [accuracy/100.0
             for accuracy in test_accuracy[test_accuracy_xmin:num_epochs]],
            color='#2A6EA6')
    ax.set_xlim([test_accuracy_xmin, num_epochs])
    ax.grid(True)
    ax.set_xlabel('Epoch')
    ax.set_title('Accuracy (%) on the test data')
    plt.show()

def plot_training_accuracy(training_accuracy, num_epochs,
                           training_accuracy_xmin, training_set_size):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    ax.plot(np.arange(training_accuracy_xmin, num_epochs),
            [accuracy*100.0/training_set_size
             for accuracy in training_accuracy[training_accuracy_xmin:num_epochs]],
            color='#2A6EA6')
    ax.set_xlim([training_accuracy_xmin, num_epochs])
    ax.grid(True)
    ax.set_xlabel('Epoch')
    ax.set_title('Accuracy (%) on the training data')
    plt.show()

```

```python
M_list=[10,30,50,100,200]
net_list=[]
data_list=[]

for i,M in enumerate(M_list):
    f = open('results_totaux'+str(M), "w")
    net = Network([784, M, 10])
    net.large_weight_initializer()
    test_cost, test_accuracy, training_cost, training_accuracy = net.SGD(training_data[:50000], 40, 10, 0.5,
                  evaluation_data=test_data, lmbda = 5.0,
                  monitor_evaluation_cost=True,
                  monitor_evaluation_accuracy=True,
                  monitor_training_cost=True,
                  monitor_training_accuracy=True)
    data_list.append((test_cost, test_accuracy, training_cost, training_accuracy))
    net_list.append(net)
    net.save('nnM_'+str(M))
    json.dump([M,test_cost, test_accuracy, training_cost, training_accuracy], f)
f.close()

```

    Epoch 0 training complete
    Cost on training data: 0.458176078172
    Accuracy on training data: 36812 / 50000
    Cost on evaluation data: 1.44022357202
    Accuracy on evaluation data: 7430 / 10000

    Epoch 1 training complete
    Cost on training data: 0.281221002679
    Accuracy on training data: 43180 / 50000
    Cost on evaluation data: 0.91991232953
    Accuracy on evaluation data: 8743 / 10000

    Epoch 2 training complete
    Cost on training data: 0.204841836014
    Accuracy on training data: 44604 / 50000
    Cost on evaluation data: 0.634605375559
    Accuracy on evaluation data: 8940 / 10000

    Epoch 3 training complete
    Cost on training data: 0.163318295699
    Accuracy on training data: 45156 / 50000
    Cost on evaluation data: 0.466455928416
    Accuracy on evaluation data: 9059 / 10000

    Epoch 4 training complete
    Cost on training data: 0.140869242011
    Accuracy on training data: 45360 / 50000
    Cost on evaluation data: 0.369625647154
    Accuracy on evaluation data: 9066 / 10000

    Epoch 5 training complete
    Cost on training data: 0.126317876126
    Accuracy on training data: 45633 / 50000
    Cost on evaluation data: 0.311990699019
    Accuracy on evaluation data: 9122 / 10000

    Epoch 6 training complete
    Cost on training data: 0.117324127131
    Accuracy on training data: 45876 / 50000
    Cost on evaluation data: 0.278120691458
    Accuracy on evaluation data: 9148 / 10000

    Epoch 7 training complete
    Cost on training data: 0.11206942944
    Accuracy on training data: 45936 / 50000
    Cost on evaluation data: 0.257439992898
    Accuracy on evaluation data: 9189 / 10000

    Epoch 8 training complete
    Cost on training data: 0.113039695857
    Accuracy on training data: 45744 / 50000
    Cost on evaluation data: 0.249628311141
    Accuracy on evaluation data: 9114 / 10000

    Epoch 9 training complete
    Cost on training data: 0.108640666766
    Accuracy on training data: 45943 / 50000
    Cost on evaluation data: 0.240284087241
    Accuracy on evaluation data: 9149 / 10000

    Epoch 10 training complete
    Cost on training data: 0.10535788065
    Accuracy on training data: 46055 / 50000
    Cost on evaluation data: 0.234105426217
    Accuracy on evaluation data: 9193 / 10000

    Epoch 11 training complete
    Cost on training data: 0.106326665297
    Accuracy on training data: 45889 / 50000
    Cost on evaluation data: 0.232924869443
    Accuracy on evaluation data: 9133 / 10000

    Epoch 12 training complete
    Cost on training data: 0.103644422529
    Accuracy on training data: 46055 / 50000
    Cost on evaluation data: 0.227524156675
    Accuracy on evaluation data: 9216 / 10000

    Epoch 13 training complete
    Cost on training data: 0.103176490513
    Accuracy on training data: 46004 / 50000
    Cost on evaluation data: 0.226044718286
    Accuracy on evaluation data: 9185 / 10000

    Epoch 14 training complete
    Cost on training data: 0.0993016460276
    Accuracy on training data: 46373 / 50000
    Cost on evaluation data: 0.221781664591
    Accuracy on evaluation data: 9248 / 10000

    Epoch 15 training complete
    Cost on training data: 0.0994180143408
    Accuracy on training data: 46274 / 50000
    Cost on evaluation data: 0.221236250889
    Accuracy on evaluation data: 9234 / 10000

    Epoch 16 training complete
    Cost on training data: 0.0984523798645
    Accuracy on training data: 46350 / 50000
    Cost on evaluation data: 0.220711163405
    Accuracy on evaluation data: 9243 / 10000

    Epoch 17 training complete
    Cost on training data: 0.099120895361
    Accuracy on training data: 46385 / 50000
    Cost on evaluation data: 0.221334252211
    Accuracy on evaluation data: 9231 / 10000

    Epoch 18 training complete
    Cost on training data: 0.0985488332723
    Accuracy on training data: 46416 / 50000
    Cost on evaluation data: 0.219862613732
    Accuracy on evaluation data: 9277 / 10000

    Epoch 19 training complete
    Cost on training data: 0.0971619135548
    Accuracy on training data: 46498 / 50000
    Cost on evaluation data: 0.219946938941
    Accuracy on evaluation data: 9273 / 10000

    Epoch 20 training complete
    Cost on training data: 0.101459514544
    Accuracy on training data: 46164 / 50000
    Cost on evaluation data: 0.225581803115
    Accuracy on evaluation data: 9206 / 10000

    Epoch 21 training complete
    Cost on training data: 0.0985895657601
    Accuracy on training data: 46431 / 50000
    Cost on evaluation data: 0.222668014647
    Accuracy on evaluation data: 9254 / 10000

    Epoch 22 training complete
    Cost on training data: 0.0966114639604
    Accuracy on training data: 46556 / 50000
    Cost on evaluation data: 0.220066075447
    Accuracy on evaluation data: 9268 / 10000

    Epoch 23 training complete
    Cost on training data: 0.097960015684
    Accuracy on training data: 46516 / 50000
    Cost on evaluation data: 0.221492426188
    Accuracy on evaluation data: 9265 / 10000

    Epoch 24 training complete
    Cost on training data: 0.0970031819138
    Accuracy on training data: 46544 / 50000
    Cost on evaluation data: 0.220764317831
    Accuracy on evaluation data: 9294 / 10000

    Epoch 25 training complete
    Cost on training data: 0.0984711049332
    Accuracy on training data: 46397 / 50000
    Cost on evaluation data: 0.222090234315
    Accuracy on evaluation data: 9268 / 10000

    Epoch 26 training complete
    Cost on training data: 0.097115023078
    Accuracy on training data: 46454 / 50000
    Cost on evaluation data: 0.220743047651
    Accuracy on evaluation data: 9267 / 10000

    Epoch 27 training complete
    Cost on training data: 0.0969448932933
    Accuracy on training data: 46487 / 50000
    Cost on evaluation data: 0.220900214097
    Accuracy on evaluation data: 9275 / 10000

    Epoch 28 training complete
    Cost on training data: 0.0984500530146
    Accuracy on training data: 46401 / 50000
    Cost on evaluation data: 0.22242485868
    Accuracy on evaluation data: 9267 / 10000

    Epoch 29 training complete
    Cost on training data: 0.0961689650613
    Accuracy on training data: 46572 / 50000
    Cost on evaluation data: 0.220621504065
    Accuracy on evaluation data: 9283 / 10000

    Epoch 30 training complete
    Cost on training data: 0.0957654938188
    Accuracy on training data: 46563 / 50000
    Cost on evaluation data: 0.219198218476
    Accuracy on evaluation data: 9309 / 10000

    Epoch 31 training complete
    Cost on training data: 0.095581043104
    Accuracy on training data: 46598 / 50000
    Cost on evaluation data: 0.21926999935
    Accuracy on evaluation data: 9299 / 10000

    Epoch 32 training complete
    Cost on training data: 0.0964234662673
    Accuracy on training data: 46609 / 50000
    Cost on evaluation data: 0.220722961849
    Accuracy on evaluation data: 9291 / 10000

    Epoch 33 training complete
    Cost on training data: 0.0963648257377
    Accuracy on training data: 46562 / 50000
    Cost on evaluation data: 0.220408780274
    Accuracy on evaluation data: 9304 / 10000

    Epoch 34 training complete
    Cost on training data: 0.0973188677453
    Accuracy on training data: 46562 / 50000
    Cost on evaluation data: 0.221372816174
    Accuracy on evaluation data: 9277 / 10000

    Epoch 35 training complete
    Cost on training data: 0.0971380123591
    Accuracy on training data: 46593 / 50000
    Cost on evaluation data: 0.220826604777
    Accuracy on evaluation data: 9300 / 10000

    Epoch 36 training complete
    Cost on training data: 0.0986762771196
    Accuracy on training data: 46406 / 50000
    Cost on evaluation data: 0.222855498689
    Accuracy on evaluation data: 9242 / 10000

    Epoch 37 training complete
    Cost on training data: 0.0973346131584
    Accuracy on training data: 46456 / 50000
    Cost on evaluation data: 0.220982695377
    Accuracy on evaluation data: 9263 / 10000

    Epoch 38 training complete
    Cost on training data: 0.0950144979779
    Accuracy on training data: 46626 / 50000
    Cost on evaluation data: 0.219333546448
    Accuracy on evaluation data: 9290 / 10000

    Epoch 39 training complete
    Cost on training data: 0.0966498271659
    Accuracy on training data: 46527 / 50000
    Cost on evaluation data: 0.220543735769
    Accuracy on evaluation data: 9289 / 10000

    Epoch 0 training complete
    Cost on training data: 0.861972407897
    Accuracy on training data: 41561 / 50000
    Cost on evaluation data: 3.74341092706
    Accuracy on evaluation data: 8422 / 10000

    Epoch 1 training complete
    Cost on training data: 0.532568164198
    Accuracy on training data: 44874 / 50000
    Cost on evaluation data: 2.30560909986
    Accuracy on evaluation data: 9034 / 10000

    Epoch 2 training complete
    Cost on training data: 0.347967897447
    Accuracy on training data: 45978 / 50000
    Cost on evaluation data: 1.45271513159
    Accuracy on evaluation data: 9195 / 10000

    Epoch 3 training complete
    Cost on training data: 0.241111285404
    Accuracy on training data: 46417 / 50000
    Cost on evaluation data: 0.94451101609
    Accuracy on evaluation data: 9276 / 10000

    Epoch 4 training complete
    Cost on training data: 0.175132117103
    Accuracy on training data: 46810 / 50000
    Cost on evaluation data: 0.639558383129
    Accuracy on evaluation data: 9351 / 10000

    Epoch 5 training complete
    Cost on training data: 0.136952208313
    Accuracy on training data: 47030 / 50000
    Cost on evaluation data: 0.459762452294
    Accuracy on evaluation data: 9395 / 10000

    Epoch 6 training complete
    Cost on training data: 0.114002041574
    Accuracy on training data: 47159 / 50000
    Cost on evaluation data: 0.352571951238
    Accuracy on evaluation data: 9428 / 10000

    Epoch 7 training complete
    Cost on training data: 0.100347902067
    Accuracy on training data: 47307 / 50000
    Cost on evaluation data: 0.289615736281
    Accuracy on evaluation data: 9439 / 10000

    Epoch 8 training complete
    Cost on training data: 0.0922315066948
    Accuracy on training data: 47370 / 50000
    Cost on evaluation data: 0.252520602816
    Accuracy on evaluation data: 9460 / 10000

    Epoch 9 training complete
    Cost on training data: 0.0863678740554
    Accuracy on training data: 47467 / 50000
    Cost on evaluation data: 0.231190881878
    Accuracy on evaluation data: 9463 / 10000

    Epoch 10 training complete
    Cost on training data: 0.0831204583087
    Accuracy on training data: 47564 / 50000
    Cost on evaluation data: 0.21745725461
    Accuracy on evaluation data: 9500 / 10000

    Epoch 11 training complete
    Cost on training data: 0.0801729061994
    Accuracy on training data: 47626 / 50000
    Cost on evaluation data: 0.208382499936
    Accuracy on evaluation data: 9529 / 10000

    Epoch 12 training complete
    Cost on training data: 0.0811791181724
    Accuracy on training data: 47561 / 50000
    Cost on evaluation data: 0.206550459741
    Accuracy on evaluation data: 9488 / 10000

    Epoch 13 training complete
    Cost on training data: 0.0791646781201
    Accuracy on training data: 47690 / 50000
    Cost on evaluation data: 0.203173837859
    Accuracy on evaluation data: 9513 / 10000

    Epoch 14 training complete
    Cost on training data: 0.0776566746861
    Accuracy on training data: 47720 / 50000
    Cost on evaluation data: 0.200740071551
    Accuracy on evaluation data: 9527 / 10000

    Epoch 15 training complete
    Cost on training data: 0.0772951164629
    Accuracy on training data: 47712 / 50000
    Cost on evaluation data: 0.200193221567
    Accuracy on evaluation data: 9520 / 10000

    Epoch 16 training complete
    Cost on training data: 0.0772995107332
    Accuracy on training data: 47766 / 50000
    Cost on evaluation data: 0.199810866321
    Accuracy on evaluation data: 9528 / 10000

    Epoch 17 training complete
    Cost on training data: 0.0775021217657
    Accuracy on training data: 47726 / 50000
    Cost on evaluation data: 0.200070967512
    Accuracy on evaluation data: 9534 / 10000

    Epoch 18 training complete
    Cost on training data: 0.0771945401466
    Accuracy on training data: 47757 / 50000
    Cost on evaluation data: 0.199993684278
    Accuracy on evaluation data: 9550 / 10000

    Epoch 19 training complete
    Cost on training data: 0.0759150843321
    Accuracy on training data: 47828 / 50000
    Cost on evaluation data: 0.198601386966
    Accuracy on evaluation data: 9569 / 10000

    Epoch 20 training complete
    Cost on training data: 0.0764031169294
    Accuracy on training data: 47861 / 50000
    Cost on evaluation data: 0.199598023093
    Accuracy on evaluation data: 9563 / 10000

    Epoch 21 training complete
    Cost on training data: 0.075715100415
    Accuracy on training data: 47910 / 50000
    Cost on evaluation data: 0.199202631425
    Accuracy on evaluation data: 9566 / 10000

    Epoch 22 training complete
    Cost on training data: 0.075320141174
    Accuracy on training data: 47905 / 50000
    Cost on evaluation data: 0.198714047496
    Accuracy on evaluation data: 9570 / 10000

    Epoch 23 training complete
    Cost on training data: 0.0753163217794
    Accuracy on training data: 47932 / 50000
    Cost on evaluation data: 0.198910555187
    Accuracy on evaluation data: 9585 / 10000

    Epoch 24 training complete
    Cost on training data: 0.0755254211847
    Accuracy on training data: 47929 / 50000
    Cost on evaluation data: 0.199658312355
    Accuracy on evaluation data: 9568 / 10000

    Epoch 25 training complete
    Cost on training data: 0.0750626026167
    Accuracy on training data: 47920 / 50000
    Cost on evaluation data: 0.199156690688
    Accuracy on evaluation data: 9563 / 10000

    Epoch 26 training complete
    Cost on training data: 0.0765099822539
    Accuracy on training data: 47877 / 50000
    Cost on evaluation data: 0.201439129371
    Accuracy on evaluation data: 9534 / 10000

    Epoch 27 training complete
    Cost on training data: 0.0763954026594
    Accuracy on training data: 47910 / 50000
    Cost on evaluation data: 0.200725914178
    Accuracy on evaluation data: 9562 / 10000

    Epoch 28 training complete
    Cost on training data: 0.0752290225153
    Accuracy on training data: 47917 / 50000
    Cost on evaluation data: 0.199159421223
    Accuracy on evaluation data: 9572 / 10000

    Epoch 29 training complete
    Cost on training data: 0.0752334968892
    Accuracy on training data: 47967 / 50000
    Cost on evaluation data: 0.199907549312
    Accuracy on evaluation data: 9566 / 10000

    Epoch 30 training complete
    Cost on training data: 0.0751209064036
    Accuracy on training data: 47953 / 50000
    Cost on evaluation data: 0.199873006303
    Accuracy on evaluation data: 9580 / 10000

    Epoch 31 training complete
    Cost on training data: 0.0749644480154
    Accuracy on training data: 47925 / 50000
    Cost on evaluation data: 0.198845619785
    Accuracy on evaluation data: 9583 / 10000

    Epoch 32 training complete
    Cost on training data: 0.0754221961682
    Accuracy on training data: 47955 / 50000
    Cost on evaluation data: 0.199869296726
    Accuracy on evaluation data: 9566 / 10000

    Epoch 33 training complete
    Cost on training data: 0.0750306021252
    Accuracy on training data: 47964 / 50000
    Cost on evaluation data: 0.199907190653
    Accuracy on evaluation data: 9592 / 10000

    Epoch 34 training complete
    Cost on training data: 0.0752249631154
    Accuracy on training data: 47970 / 50000
    Cost on evaluation data: 0.19989491641
    Accuracy on evaluation data: 9558 / 10000

    Epoch 35 training complete
    Cost on training data: 0.0753670418496
    Accuracy on training data: 47926 / 50000
    Cost on evaluation data: 0.200089312469
    Accuracy on evaluation data: 9552 / 10000

    Epoch 36 training complete
    Cost on training data: 0.0742837506319
    Accuracy on training data: 47965 / 50000
    Cost on evaluation data: 0.199312727186
    Accuracy on evaluation data: 9582 / 10000

    Epoch 37 training complete
    Cost on training data: 0.0762079994807
    Accuracy on training data: 47913 / 50000
    Cost on evaluation data: 0.200660374682
    Accuracy on evaluation data: 9566 / 10000

    Epoch 38 training complete
    Cost on training data: 0.0754397993785
    Accuracy on training data: 47957 / 50000
    Cost on evaluation data: 0.200308081015
    Accuracy on evaluation data: 9565 / 10000

    Epoch 39 training complete
    Cost on training data: 0.0741362406536
    Accuracy on training data: 48011 / 50000
    Cost on evaluation data: 0.198877400116
    Accuracy on evaluation data: 9583 / 10000

    Epoch 0 training complete
    Cost on training data: 1.42975414854
    Accuracy on training data: 32417 / 50000
    Cost on evaluation data: 6.24144134801
    Accuracy on evaluation data: 6647 / 10000

    Epoch 1 training complete
    Cost on training data: 0.910308561491
    Accuracy on training data: 36094 / 50000
    Cost on evaluation data: 3.84473197607
    Accuracy on evaluation data: 7307 / 10000

    Epoch 2 training complete
    Cost on training data: 0.56849787671
    Accuracy on training data: 41534 / 50000
    Cost on evaluation data: 2.37330326207
    Accuracy on evaluation data: 8371 / 10000

    Epoch 3 training complete
    Cost on training data: 0.345551017571
    Accuracy on training data: 46634 / 50000
    Cost on evaluation data: 1.47402678624
    Accuracy on evaluation data: 9344 / 10000

    Epoch 4 training complete
    Cost on training data: 0.237854178947
    Accuracy on training data: 47031 / 50000
    Cost on evaluation data: 0.959674395369
    Accuracy on evaluation data: 9395 / 10000

    Epoch 5 training complete
    Cost on training data: 0.172009707577
    Accuracy on training data: 47322 / 50000
    Cost on evaluation data: 0.650655939764
    Accuracy on evaluation data: 9461 / 10000

    Epoch 6 training complete
    Cost on training data: 0.132925623166
    Accuracy on training data: 47493 / 50000
    Cost on evaluation data: 0.466808263836
    Accuracy on evaluation data: 9509 / 10000

    Epoch 7 training complete
    Cost on training data: 0.110514565095
    Accuracy on training data: 47619 / 50000
    Cost on evaluation data: 0.359148303473
    Accuracy on evaluation data: 9508 / 10000

    Epoch 8 training complete
    Cost on training data: 0.0958548429406
    Accuracy on training data: 47687 / 50000
    Cost on evaluation data: 0.294018448783
    Accuracy on evaluation data: 9533 / 10000

    Epoch 9 training complete
    Cost on training data: 0.0880799484647
    Accuracy on training data: 47787 / 50000
    Cost on evaluation data: 0.256567128188
    Accuracy on evaluation data: 9554 / 10000

    Epoch 10 training complete
    Cost on training data: 0.0826109115797
    Accuracy on training data: 47843 / 50000
    Cost on evaluation data: 0.233368397029
    Accuracy on evaluation data: 9554 / 10000

    Epoch 11 training complete
    Cost on training data: 0.079311146049
    Accuracy on training data: 47903 / 50000
    Cost on evaluation data: 0.220708421984
    Accuracy on evaluation data: 9559 / 10000

    Epoch 12 training complete
    Cost on training data: 0.0771986025677
    Accuracy on training data: 47960 / 50000
    Cost on evaluation data: 0.211757232305
    Accuracy on evaluation data: 9572 / 10000

    Epoch 13 training complete
    Cost on training data: 0.0753410207516
    Accuracy on training data: 47970 / 50000
    Cost on evaluation data: 0.207046731324
    Accuracy on evaluation data: 9577 / 10000

    Epoch 14 training complete
    Cost on training data: 0.0745022850736
    Accuracy on training data: 48028 / 50000
    Cost on evaluation data: 0.20420411345
    Accuracy on evaluation data: 9570 / 10000

    Epoch 15 training complete
    Cost on training data: 0.075408520212
    Accuracy on training data: 47953 / 50000
    Cost on evaluation data: 0.204352501768
    Accuracy on evaluation data: 9566 / 10000

    Epoch 16 training complete
    Cost on training data: 0.075624367312
    Accuracy on training data: 48035 / 50000
    Cost on evaluation data: 0.204304837207
    Accuracy on evaluation data: 9581 / 10000

    Epoch 17 training complete
    Cost on training data: 0.072595432028
    Accuracy on training data: 48116 / 50000
    Cost on evaluation data: 0.200904940659
    Accuracy on evaluation data: 9611 / 10000

    Epoch 18 training complete
    Cost on training data: 0.0730388720584
    Accuracy on training data: 48153 / 50000
    Cost on evaluation data: 0.201183508378
    Accuracy on evaluation data: 9618 / 10000

    Epoch 19 training complete
    Cost on training data: 0.0720846596894
    Accuracy on training data: 48206 / 50000
    Cost on evaluation data: 0.200106576148
    Accuracy on evaluation data: 9618 / 10000

    Epoch 20 training complete
    Cost on training data: 0.0725805231683
    Accuracy on training data: 48189 / 50000
    Cost on evaluation data: 0.200327609924
    Accuracy on evaluation data: 9623 / 10000

    Epoch 21 training complete
    Cost on training data: 0.0719251543502
    Accuracy on training data: 48207 / 50000
    Cost on evaluation data: 0.200731208768
    Accuracy on evaluation data: 9616 / 10000

    Epoch 22 training complete
    Cost on training data: 0.0712784529354
    Accuracy on training data: 48186 / 50000
    Cost on evaluation data: 0.200004553185
    Accuracy on evaluation data: 9612 / 10000

    Epoch 23 training complete
    Cost on training data: 0.0735629009285
    Accuracy on training data: 48173 / 50000
    Cost on evaluation data: 0.201596061373
    Accuracy on evaluation data: 9612 / 10000

    Epoch 24 training complete
    Cost on training data: 0.0712134892791
    Accuracy on training data: 48196 / 50000
    Cost on evaluation data: 0.199991505183
    Accuracy on evaluation data: 9629 / 10000

    Epoch 25 training complete
    Cost on training data: 0.0706783613569
    Accuracy on training data: 48258 / 50000
    Cost on evaluation data: 0.199617815737
    Accuracy on evaluation data: 9633 / 10000

    Epoch 26 training complete
    Cost on training data: 0.0732257162245
    Accuracy on training data: 48183 / 50000
    Cost on evaluation data: 0.202294906206
    Accuracy on evaluation data: 9608 / 10000

    Epoch 27 training complete
    Cost on training data: 0.0717474778501
    Accuracy on training data: 48210 / 50000
    Cost on evaluation data: 0.200933006453
    Accuracy on evaluation data: 9616 / 10000

    Epoch 28 training complete
    Cost on training data: 0.0723750189725
    Accuracy on training data: 48178 / 50000
    Cost on evaluation data: 0.201218037601
    Accuracy on evaluation data: 9604 / 10000

    Epoch 29 training complete
    Cost on training data: 0.0708755072604
    Accuracy on training data: 48252 / 50000
    Cost on evaluation data: 0.200079627289
    Accuracy on evaluation data: 9614 / 10000

    Epoch 30 training complete
    Cost on training data: 0.0705442536435
    Accuracy on training data: 48279 / 50000
    Cost on evaluation data: 0.199519019952
    Accuracy on evaluation data: 9633 / 10000

    Epoch 31 training complete
    Cost on training data: 0.0701940372347
    Accuracy on training data: 48281 / 50000
    Cost on evaluation data: 0.199399160677
    Accuracy on evaluation data: 9630 / 10000

    Epoch 32 training complete
    Cost on training data: 0.0701449924563
    Accuracy on training data: 48308 / 50000
    Cost on evaluation data: 0.199441479284
    Accuracy on evaluation data: 9644 / 10000

    Epoch 33 training complete
    Cost on training data: 0.0720386431053
    Accuracy on training data: 48211 / 50000
    Cost on evaluation data: 0.20223108616
    Accuracy on evaluation data: 9603 / 10000

    Epoch 34 training complete
    Cost on training data: 0.0709264962312
    Accuracy on training data: 48252 / 50000
    Cost on evaluation data: 0.200349037123
    Accuracy on evaluation data: 9617 / 10000

    Epoch 35 training complete
    Cost on training data: 0.0712024018311
    Accuracy on training data: 48294 / 50000
    Cost on evaluation data: 0.200689912517
    Accuracy on evaluation data: 9622 / 10000

    Epoch 36 training complete
    Cost on training data: 0.0701742957932
    Accuracy on training data: 48312 / 50000
    Cost on evaluation data: 0.19970797337
    Accuracy on evaluation data: 9628 / 10000

    Epoch 37 training complete
    Cost on training data: 0.0701860649665
    Accuracy on training data: 48315 / 50000
    Cost on evaluation data: 0.200010118823
    Accuracy on evaluation data: 9630 / 10000

    Epoch 38 training complete
    Cost on training data: 0.0706488623752
    Accuracy on training data: 48361 / 50000
    Cost on evaluation data: 0.200078220573
    Accuracy on evaluation data: 9644 / 10000

    Epoch 39 training complete
    Cost on training data: 0.070427577259
    Accuracy on training data: 48324 / 50000
    Cost on evaluation data: 0.199919749971
    Accuracy on evaluation data: 9640 / 10000

    Epoch 0 training complete
    Cost on training data: 2.63082244986
    Accuracy on training data: 32464 / 50000
    Cost on evaluation data: 12.2513770989
    Accuracy on evaluation data: 6454 / 10000

    Epoch 1 training complete
    Cost on training data: 1.58514642792
    Accuracy on training data: 41236 / 50000
    Cost on evaluation data: 7.42992072926
    Accuracy on evaluation data: 8236 / 10000

    Epoch 2 training complete
    Cost on training data: 0.969289074322
    Accuracy on training data: 45935 / 50000
    Cost on evaluation data: 4.53501921371
    Accuracy on evaluation data: 9150 / 10000

    Epoch 3 training complete
    Cost on training data: 0.606697493418
    Accuracy on training data: 46870 / 50000
    Cost on evaluation data: 2.80041310355
    Accuracy on evaluation data: 9361 / 10000

    Epoch 4 training complete
    Cost on training data: 0.395439380806
    Accuracy on training data: 47268 / 50000
    Cost on evaluation data: 1.7632826142
    Accuracy on evaluation data: 9437 / 10000

    Epoch 5 training complete
    Cost on training data: 0.266628750447
    Accuracy on training data: 47609 / 50000
    Cost on evaluation data: 1.13851632599
    Accuracy on evaluation data: 9500 / 10000

    Epoch 6 training complete
    Cost on training data: 0.188730778588
    Accuracy on training data: 47777 / 50000
    Cost on evaluation data: 0.763274659086
    Accuracy on evaluation data: 9523 / 10000

    Epoch 7 training complete
    Cost on training data: 0.142198868502
    Accuracy on training data: 47968 / 50000
    Cost on evaluation data: 0.53907604704
    Accuracy on evaluation data: 9562 / 10000

    Epoch 8 training complete
    Cost on training data: 0.115330619524
    Accuracy on training data: 47998 / 50000
    Cost on evaluation data: 0.406323012551
    Accuracy on evaluation data: 9562 / 10000

    Epoch 9 training complete
    Cost on training data: 0.0971880357973
    Accuracy on training data: 48121 / 50000
    Cost on evaluation data: 0.324442897345
    Accuracy on evaluation data: 9593 / 10000

    Epoch 10 training complete
    Cost on training data: 0.0885962988981
    Accuracy on training data: 48099 / 50000
    Cost on evaluation data: 0.278440845143
    Accuracy on evaluation data: 9590 / 10000

    Epoch 11 training complete
    Cost on training data: 0.0809856295121
    Accuracy on training data: 48236 / 50000
    Cost on evaluation data: 0.248255848371
    Accuracy on evaluation data: 9609 / 10000

    Epoch 12 training complete
    Cost on training data: 0.0775452027918
    Accuracy on training data: 48218 / 50000
    Cost on evaluation data: 0.231598473601
    Accuracy on evaluation data: 9600 / 10000

    Epoch 13 training complete
    Cost on training data: 0.077244848003
    Accuracy on training data: 48159 / 50000
    Cost on evaluation data: 0.22315586015
    Accuracy on evaluation data: 9587 / 10000

    Epoch 14 training complete
    Cost on training data: 0.0738529871425
    Accuracy on training data: 48254 / 50000
    Cost on evaluation data: 0.214774175084
    Accuracy on evaluation data: 9621 / 10000

    Epoch 15 training complete
    Cost on training data: 0.0726153160714
    Accuracy on training data: 48342 / 50000
    Cost on evaluation data: 0.211553016072
    Accuracy on evaluation data: 9635 / 10000

    Epoch 16 training complete
    Cost on training data: 0.0719727225417
    Accuracy on training data: 48294 / 50000
    Cost on evaluation data: 0.209514502505
    Accuracy on evaluation data: 9611 / 10000

    Epoch 17 training complete
    Cost on training data: 0.073803451792
    Accuracy on training data: 48273 / 50000
    Cost on evaluation data: 0.210487752919
    Accuracy on evaluation data: 9603 / 10000

    Epoch 18 training complete
    Cost on training data: 0.0710905917012
    Accuracy on training data: 48329 / 50000
    Cost on evaluation data: 0.206396692066
    Accuracy on evaluation data: 9624 / 10000

    Epoch 19 training complete
    Cost on training data: 0.0696140295526
    Accuracy on training data: 48429 / 50000
    Cost on evaluation data: 0.204686285756
    Accuracy on evaluation data: 9643 / 10000

    Epoch 20 training complete
    Cost on training data: 0.070000267812
    Accuracy on training data: 48366 / 50000
    Cost on evaluation data: 0.205466192185
    Accuracy on evaluation data: 9616 / 10000

    Epoch 21 training complete
    Cost on training data: 0.0690897146766
    Accuracy on training data: 48453 / 50000
    Cost on evaluation data: 0.204200032141
    Accuracy on evaluation data: 9640 / 10000

    Epoch 22 training complete
    Cost on training data: 0.0693821300692
    Accuracy on training data: 48435 / 50000
    Cost on evaluation data: 0.204425696398
    Accuracy on evaluation data: 9643 / 10000

    Epoch 23 training complete
    Cost on training data: 0.0693532724089
    Accuracy on training data: 48392 / 50000
    Cost on evaluation data: 0.204748074206
    Accuracy on evaluation data: 9639 / 10000

    Epoch 24 training complete
    Cost on training data: 0.0703227041671
    Accuracy on training data: 48413 / 50000
    Cost on evaluation data: 0.20587772223
    Accuracy on evaluation data: 9641 / 10000

    Epoch 25 training complete
    Cost on training data: 0.0687386332413
    Accuracy on training data: 48478 / 50000
    Cost on evaluation data: 0.204163597377
    Accuracy on evaluation data: 9664 / 10000

    Epoch 26 training complete
    Cost on training data: 0.0684178219363
    Accuracy on training data: 48545 / 50000
    Cost on evaluation data: 0.203875362307
    Accuracy on evaluation data: 9659 / 10000

    Epoch 27 training complete
    Cost on training data: 0.0694916746048
    Accuracy on training data: 48476 / 50000
    Cost on evaluation data: 0.204922591077
    Accuracy on evaluation data: 9647 / 10000

    Epoch 28 training complete
    Cost on training data: 0.0695106454
    Accuracy on training data: 48479 / 50000
    Cost on evaluation data: 0.205078463502
    Accuracy on evaluation data: 9646 / 10000

    Epoch 29 training complete
    Cost on training data: 0.0696386635033
    Accuracy on training data: 48447 / 50000
    Cost on evaluation data: 0.205554437146
    Accuracy on evaluation data: 9639 / 10000

    Epoch 30 training complete
    Cost on training data: 0.0709283825484
    Accuracy on training data: 48372 / 50000
    Cost on evaluation data: 0.206791436247
    Accuracy on evaluation data: 9634 / 10000

    Epoch 31 training complete
    Cost on training data: 0.0683367659826
    Accuracy on training data: 48509 / 50000
    Cost on evaluation data: 0.204377327604
    Accuracy on evaluation data: 9655 / 10000

    Epoch 32 training complete
    Cost on training data: 0.0691175101122
    Accuracy on training data: 48484 / 50000
    Cost on evaluation data: 0.205084275755
    Accuracy on evaluation data: 9662 / 10000

    Epoch 33 training complete
    Cost on training data: 0.0696022705428
    Accuracy on training data: 48512 / 50000
    Cost on evaluation data: 0.205342921393
    Accuracy on evaluation data: 9642 / 10000

    Epoch 34 training complete
    Cost on training data: 0.0695198298452
    Accuracy on training data: 48464 / 50000
    Cost on evaluation data: 0.205704241859
    Accuracy on evaluation data: 9644 / 10000

    Epoch 35 training complete
    Cost on training data: 0.0695102937703
    Accuracy on training data: 48460 / 50000
    Cost on evaluation data: 0.205579249348
    Accuracy on evaluation data: 9640 / 10000

    Epoch 36 training complete
    Cost on training data: 0.0678409800366
    Accuracy on training data: 48564 / 50000
    Cost on evaluation data: 0.203939197127
    Accuracy on evaluation data: 9666 / 10000

    Epoch 37 training complete
    Cost on training data: 0.07027444492
    Accuracy on training data: 48403 / 50000
    Cost on evaluation data: 0.20611673117
    Accuracy on evaluation data: 9630 / 10000

    Epoch 38 training complete
    Cost on training data: 0.0683722959656
    Accuracy on training data: 48525 / 50000
    Cost on evaluation data: 0.20429278246
    Accuracy on evaluation data: 9667 / 10000

    Epoch 39 training complete
    Cost on training data: 0.0672118882159
    Accuracy on training data: 48589 / 50000
    Cost on evaluation data: 0.203621631122
    Accuracy on evaluation data: 9684 / 10000

    Epoch 0 training complete
    Cost on training data: 5.1745183444
    Accuracy on training data: 19876 / 50000
    Cost on evaluation data: 24.4962297911
    Accuracy on evaluation data: 4013 / 10000

    Epoch 1 training complete
    Cost on training data: 3.2241509914
    Accuracy on training data: 23347 / 50000
    Cost on evaluation data: 14.9456529872
    Accuracy on evaluation data: 4676 / 10000

    Epoch 2 training complete
    Cost on training data: 2.02046348238
    Accuracy on training data: 28178 / 50000
    Cost on evaluation data: 9.14068778368
    Accuracy on evaluation data: 5604 / 10000

    Epoch 3 training complete
    Cost on training data: 1.19130022056
    Accuracy on training data: 42306 / 50000
    Cost on evaluation data: 5.52689870669
    Accuracy on evaluation data: 8476 / 10000

    Epoch 4 training complete
    Cost on training data: 0.729916111587
    Accuracy on training data: 46917 / 50000
    Cost on evaluation data: 3.39362372267
    Accuracy on evaluation data: 9387 / 10000

    Epoch 5 training complete
    Cost on training data: 0.465595575924
    Accuracy on training data: 47512 / 50000
    Cost on evaluation data: 2.12536149515
    Accuracy on evaluation data: 9502 / 10000

    Epoch 6 training complete
    Cost on training data: 0.311912589875
    Accuracy on training data: 47653 / 50000
    Cost on evaluation data: 1.36524429501
    Accuracy on evaluation data: 9518 / 10000

    Epoch 7 training complete
    Cost on training data: 0.215396081156
    Accuracy on training data: 47934 / 50000
    Cost on evaluation data: 0.905468450821
    Accuracy on evaluation data: 9576 / 10000

    Epoch 8 training complete
    Cost on training data: 0.161355518182
    Accuracy on training data: 47801 / 50000
    Cost on evaluation data: 0.632160392799
    Accuracy on evaluation data: 9558 / 10000

    Epoch 9 training complete
    Cost on training data: 0.125276845264
    Accuracy on training data: 48135 / 50000
    Cost on evaluation data: 0.464389298513
    Accuracy on evaluation data: 9605 / 10000

    Epoch 10 training complete
    Cost on training data: 0.10533316191
    Accuracy on training data: 48110 / 50000
    Cost on evaluation data: 0.365618088807
    Accuracy on evaluation data: 9599 / 10000

    Epoch 11 training complete
    Cost on training data: 0.0917651526037
    Accuracy on training data: 48215 / 50000
    Cost on evaluation data: 0.303947391525
    Accuracy on evaluation data: 9617 / 10000

    Epoch 12 training complete
    Cost on training data: 0.0852197971391
    Accuracy on training data: 48280 / 50000
    Cost on evaluation data: 0.26840490972
    Accuracy on evaluation data: 9640 / 10000

    Epoch 13 training complete
    Cost on training data: 0.077714863899
    Accuracy on training data: 48343 / 50000
    Cost on evaluation data: 0.243547312752
    Accuracy on evaluation data: 9657 / 10000

    Epoch 14 training complete
    Cost on training data: 0.077986384013
    Accuracy on training data: 48222 / 50000
    Cost on evaluation data: 0.233206511761
    Accuracy on evaluation data: 9623 / 10000

    Epoch 15 training complete
    Cost on training data: 0.0765077739791
    Accuracy on training data: 48238 / 50000
    Cost on evaluation data: 0.225538011033
    Accuracy on evaluation data: 9623 / 10000

    Epoch 16 training complete
    Cost on training data: 0.0720232096791
    Accuracy on training data: 48355 / 50000
    Cost on evaluation data: 0.217142007655
    Accuracy on evaluation data: 9655 / 10000

    Epoch 17 training complete
    Cost on training data: 0.0724238659631
    Accuracy on training data: 48364 / 50000
    Cost on evaluation data: 0.215584048724
    Accuracy on evaluation data: 9640 / 10000

    Epoch 18 training complete
    Cost on training data: 0.0727829469583
    Accuracy on training data: 48294 / 50000
    Cost on evaluation data: 0.214672021961
    Accuracy on evaluation data: 9626 / 10000

    Epoch 19 training complete
    Cost on training data: 0.0721098829158
    Accuracy on training data: 48333 / 50000
    Cost on evaluation data: 0.212864200707
    Accuracy on evaluation data: 9632 / 10000

    Epoch 20 training complete
    Cost on training data: 0.0718628830548
    Accuracy on training data: 48373 / 50000
    Cost on evaluation data: 0.21176831328
    Accuracy on evaluation data: 9649 / 10000

    Epoch 21 training complete
    Cost on training data: 0.0697850690867
    Accuracy on training data: 48468 / 50000
    Cost on evaluation data: 0.209565195629
    Accuracy on evaluation data: 9658 / 10000

    Epoch 22 training complete
    Cost on training data: 0.0693635253792
    Accuracy on training data: 48462 / 50000
    Cost on evaluation data: 0.208820237969
    Accuracy on evaluation data: 9653 / 10000

    Epoch 23 training complete
    Cost on training data: 0.0710427147179
    Accuracy on training data: 48375 / 50000
    Cost on evaluation data: 0.209834853814
    Accuracy on evaluation data: 9642 / 10000

    Epoch 24 training complete
    Cost on training data: 0.0699777310838
    Accuracy on training data: 48485 / 50000
    Cost on evaluation data: 0.209673306335
    Accuracy on evaluation data: 9679 / 10000

    Epoch 25 training complete
    Cost on training data: 0.0710727337154
    Accuracy on training data: 48402 / 50000
    Cost on evaluation data: 0.210611216577
    Accuracy on evaluation data: 9644 / 10000

    Epoch 26 training complete
    Cost on training data: 0.0710299440324
    Accuracy on training data: 48471 / 50000
    Cost on evaluation data: 0.210735582866
    Accuracy on evaluation data: 9650 / 10000

    Epoch 27 training complete
    Cost on training data: 0.0689616618556
    Accuracy on training data: 48550 / 50000
    Cost on evaluation data: 0.2083193359
    Accuracy on evaluation data: 9675 / 10000

    Epoch 28 training complete
    Cost on training data: 0.0689251115695
    Accuracy on training data: 48483 / 50000
    Cost on evaluation data: 0.208043187883
    Accuracy on evaluation data: 9659 / 10000

    Epoch 29 training complete
    Cost on training data: 0.0687200613751
    Accuracy on training data: 48558 / 50000
    Cost on evaluation data: 0.208025431277
    Accuracy on evaluation data: 9662 / 10000

    Epoch 30 training complete
    Cost on training data: 0.068028539343
    Accuracy on training data: 48539 / 50000
    Cost on evaluation data: 0.20735543297
    Accuracy on evaluation data: 9668 / 10000

    Epoch 31 training complete
    Cost on training data: 0.0709934526561
    Accuracy on training data: 48414 / 50000
    Cost on evaluation data: 0.210130297509
    Accuracy on evaluation data: 9645 / 10000

    Epoch 32 training complete
    Cost on training data: 0.0688941788057
    Accuracy on training data: 48522 / 50000
    Cost on evaluation data: 0.20823681365
    Accuracy on evaluation data: 9661 / 10000

    Epoch 33 training complete
    Cost on training data: 0.0719136066014
    Accuracy on training data: 48408 / 50000
    Cost on evaluation data: 0.210221671956
    Accuracy on evaluation data: 9649 / 10000

    Epoch 34 training complete
    Cost on training data: 0.071661726771
    Accuracy on training data: 48401 / 50000
    Cost on evaluation data: 0.210944931909
    Accuracy on evaluation data: 9648 / 10000

    Epoch 35 training complete
    Cost on training data: 0.0684698107141
    Accuracy on training data: 48589 / 50000
    Cost on evaluation data: 0.207418832033
    Accuracy on evaluation data: 9680 / 10000

    Epoch 36 training complete
    Cost on training data: 0.0676313829221
    Accuracy on training data: 48591 / 50000
    Cost on evaluation data: 0.207123482779
    Accuracy on evaluation data: 9683 / 10000

    Epoch 37 training complete
    Cost on training data: 0.0700040520312
    Accuracy on training data: 48486 / 50000
    Cost on evaluation data: 0.209088529232
    Accuracy on evaluation data: 9647 / 10000

    Epoch 38 training complete
    Cost on training data: 0.0697589467638
    Accuracy on training data: 48575 / 50000
    Cost on evaluation data: 0.209088396759
    Accuracy on evaluation data: 9675 / 10000

    Epoch 39 training complete
    Cost on training data: 0.0686697889581
    Accuracy on training data: 48550 / 50000
    Cost on evaluation data: 0.207777292798
    Accuracy on evaluation data: 9668 / 10000

```python

def plot_overlay_train(data, num_epochs, xmin,
                 training_set_size,M_list):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    for i,M in enumerate(M_list):
     test_accuracy = data[i][1]
     training_accuracy = data[i][3]
     ax.plot(np.arange(xmin, num_epochs),
            [accuracy/100.0 for accuracy in test_accuracy],

            label="Accuracy on the test data, M="+str(M))

     ax.grid(True)
     ax.set_xlim([xmin, num_epochs])
     ax.set_xlabel('Epoch')
     ax.set_ylim([90, 100])
    # Shrink current axis by 20%
    box = ax.get_position()
    ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])

# Put a legend to the right of the current axis
    ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    #plt.legend(loc="lower right")
    plt.show()

def plot_overlay_test(data, num_epochs, xmin,
                 training_set_size,M_list):
    fig = plt.figure()
    ax = fig.add_subplot(111)
    for i,M in enumerate(M_list):
     test_accuracy = data[i][1]
     training_accuracy = data[i][3]
     ax.plot(np.arange(xmin, num_epochs),
            [accuracy*100.0/training_set_size
             for accuracy in training_accuracy],

            label="Accuracy on the training data, M="+str(M))
     ax.grid(True)
     ax.set_xlim([xmin, num_epochs])
     ax.set_xlabel('Epoch')
     ax.set_ylim([90, 100])
    # Shrink current axis by 20%
    box = ax.get_position()
    ax.set_position([box.x0, box.y0, box.width * 0.8, box.height])

# Put a legend to the right of the current axis
    ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
    #plt.legend(loc="lower right")
    plt.show()

```

```python
import json
M_list=[10,30,50,100,200]
net_list=[]
data_list=[]

for i,M in enumerate(M_list):
    with open('results_totaux'+str(M)) as json_data:
        d = json.loads(json_data.read())
        data_list.append(d[1:])

#%print data_list
plot_overlay_test(data_list, 40, 0,50000,M_list)

plot_overlay_train(data_list, 40, 0,50000,M_list)

```

#### What can we deduce from this results ?

As we can see, increasing the number of hidden neurons $M$ lead to a larger accuracy of the system. On the first plot we notice that increasing $M$ first leads to an increase of learning on the training datas, but after $M=100$ the accuracy seems to be similar between the networks.

On the test data, globally we observe the same behavior but the results are similar as soon as we reach $M=50$. We can obtain better accuracy by increasing the size of the hidden layer, but at some point that's the way to go to 100 % accuracy.

### Improvment on the results with Deep Learning

We are going to improve with the network we have presented. To improve more the results we should use the convolutional network architecture, in order to recognize features but this is behond tis homework.

One way to improve the categorisation of digits without changing the architecture of the neural network is to work with another cost function. We will try to see what happend with the cost Entropy function. this one is defined by :
$$ C = - \frac{1}{n} \sum_x [y \ln y + (1-y) \ln (1-y)]$$
The cross entropy comes from the field of information theory, and is sometimes called the binary entropy. Just to introduce briefly this quantity, we can say that this quantity measure how much we are "surprised" when learning the true value of $y$. This quantity is equal to the Kullback_Leibner divergence up to some constant, which is appropriate to the course while highlighting the link between statistical physics and machine learning.

Now we just have to add a class for this cost function and we can use the same program to see the difference with the quadratic cost function.

```python
class CrossEntropyCost(object):

    @staticmethod
    def fn(a, y):
        """Return the cost associated with an output ``a`` and desired output
        ``y``.  Note that np.nan_to_num is used to ensure numerical
        stability.  In particular, if both ``a`` and ``y`` have a 1.0
        in the same slot, then the expression (1-y)*np.log(1-a)
        returns nan.  The np.nan_to_num ensures that that is converted
        to the correct value (0.0).

        """
        return np.sum(np.nan_to_num(-y*np.log(a)-(1-y)*np.log(1-a)))

    @staticmethod
    def delta(z, a, y):
        """Return the error delta from the output layer.  Note that the
        parameter ``z`` is not used by the method.  It is included in
        the method's parameters in order to make the interface
        consistent with the delta method for other cost classes.

        """
        return (a-y)

```

```python

net = Network([784, 100, 10], cost=CrossEntropyCost)
net.large_weight_initializer()
test_cost, test_accuracy, training_cost, training_accuracy=net.SGD(training_data, 60, 10, 0.1, lmbda=5.0,evaluation_data=validation_data,monitor_evaluation_accuracy=True)

plot_test_accuracy(test_accuracy,60, 0)
```

As we can see we have better results. Indeed we are able to reach an accuracy of $98$ %. We highlight the fact that for machine learning problem, the choice of the cost function is important.

If we work on the other parameter : architecture, transfer function, weight initialization .... we could still improve the classification. We are going briefly to introduce the results we can obtained with convolutional nets. For this we will use a python implementation : theano.

For optimization we will still use backpropagation and stochastic gradient descent as in the previous part. The algorithm is pretty huge and not really interessant because of the use of a package, thus it remains in another file.

```python
import network_homework_5
from network_homework_5 import Network
from network_homework_5 import ConvPoolLayer, FullyConnectedLayer, SoftmaxLayer

training_data, validation_data, test_data = network_homework_5.load_data_shared()
mini_batch_size = 10
net = Network([
        ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
                      filter_shape=(20, 1, 5, 5),
                      poolsize=(2, 2)),
        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 12),
                      filter_shape=(40, 20, 5, 5),
                      poolsize=(2, 2)),
        FullyConnectedLayer(n_in=40*4*4, n_out=100),
        SoftmaxLayer(n_in=100, n_out=10)], mini_batch_size)
validation =net.SGD(training_data, 40, mini_batch_size, 0.1, validation_data, test_data)
```

    Training mini-batch number 0
    Training mini-batch number 1000
    Training mini-batch number 2000
    Training mini-batch number 3000
    Training mini-batch number 4000
    Epoch 0: validation accuracy 93.00%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 93.25%
    Training mini-batch number 5000
    Training mini-batch number 6000
    Training mini-batch number 7000
    Training mini-batch number 8000
    Training mini-batch number 9000
    Epoch 1: validation accuracy 96.94%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 96.59%
    Training mini-batch number 10000
    Training mini-batch number 11000
    Training mini-batch number 12000
    Training mini-batch number 13000
    Training mini-batch number 14000
    Epoch 2: validation accuracy 97.55%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 97.31%
    Training mini-batch number 15000
    Training mini-batch number 16000
    Training mini-batch number 17000
    Training mini-batch number 18000
    Training mini-batch number 19000
    Epoch 3: validation accuracy 97.89%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 97.76%
    Training mini-batch number 20000
    Training mini-batch number 21000
    Training mini-batch number 22000
    Training mini-batch number 23000
    Training mini-batch number 24000
    Epoch 4: validation accuracy 98.07%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.00%
    Training mini-batch number 25000
    Training mini-batch number 26000
    Training mini-batch number 27000
    Training mini-batch number 28000
    Training mini-batch number 29000
    Epoch 5: validation accuracy 98.30%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.26%
    Training mini-batch number 30000
    Training mini-batch number 31000
    Training mini-batch number 32000
    Training mini-batch number 33000
    Training mini-batch number 34000
    Epoch 6: validation accuracy 98.41%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.48%
    Training mini-batch number 35000
    Training mini-batch number 36000
    Training mini-batch number 37000
    Training mini-batch number 38000
    Training mini-batch number 39000
    Epoch 7: validation accuracy 98.55%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.57%
    Training mini-batch number 40000
    Training mini-batch number 41000
    Training mini-batch number 42000
    Training mini-batch number 43000
    Training mini-batch number 44000
    Epoch 8: validation accuracy 98.60%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.64%
    Training mini-batch number 45000
    Training mini-batch number 46000
    Training mini-batch number 47000
    Training mini-batch number 48000
    Training mini-batch number 49000
    Epoch 9: validation accuracy 98.66%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.68%
    Training mini-batch number 50000
    Training mini-batch number 51000
    Training mini-batch number 52000
    Training mini-batch number 53000
    Training mini-batch number 54000
    Epoch 10: validation accuracy 98.72%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.70%
    Training mini-batch number 55000
    Training mini-batch number 56000
    Training mini-batch number 57000
    Training mini-batch number 58000
    Training mini-batch number 59000
    Epoch 11: validation accuracy 98.75%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.72%
    Training mini-batch number 60000
    Training mini-batch number 61000
    Training mini-batch number 62000
    Training mini-batch number 63000
    Training mini-batch number 64000
    Epoch 12: validation accuracy 98.79%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.74%
    Training mini-batch number 65000
    Training mini-batch number 66000
    Training mini-batch number 67000
    Training mini-batch number 68000
    Training mini-batch number 69000
    Epoch 13: validation accuracy 98.83%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.83%
    Training mini-batch number 70000
    Training mini-batch number 71000
    Training mini-batch number 72000
    Training mini-batch number 73000
    Training mini-batch number 74000
    Epoch 14: validation accuracy 98.86%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.85%
    Training mini-batch number 75000
    Training mini-batch number 76000
    Training mini-batch number 77000
    Training mini-batch number 78000
    Training mini-batch number 79000
    Epoch 15: validation accuracy 98.86%
    Training mini-batch number 80000
    Training mini-batch number 81000
    Training mini-batch number 82000
    Training mini-batch number 83000
    Training mini-batch number 84000
    Epoch 16: validation accuracy 98.84%
    Training mini-batch number 85000
    Training mini-batch number 86000
    Training mini-batch number 87000
    Training mini-batch number 88000
    Training mini-batch number 89000
    Epoch 17: validation accuracy 98.87%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.87%
    Training mini-batch number 90000
    Training mini-batch number 91000
    Training mini-batch number 92000
    Training mini-batch number 93000
    Training mini-batch number 94000
    Epoch 18: validation accuracy 98.88%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.91%
    Training mini-batch number 95000
    Training mini-batch number 96000
    Training mini-batch number 97000
    Training mini-batch number 98000
    Training mini-batch number 99000
    Epoch 19: validation accuracy 98.88%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.90%
    Training mini-batch number 100000
    Training mini-batch number 101000
    Training mini-batch number 102000
    Training mini-batch number 103000
    Training mini-batch number 104000
    Epoch 20: validation accuracy 98.89%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.93%
    Training mini-batch number 105000
    Training mini-batch number 106000
    Training mini-batch number 107000
    Training mini-batch number 108000
    Training mini-batch number 109000
    Epoch 21: validation accuracy 98.88%
    Training mini-batch number 110000
    Training mini-batch number 111000
    Training mini-batch number 112000
    Training mini-batch number 113000
    Training mini-batch number 114000
    Epoch 22: validation accuracy 98.89%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.92%
    Training mini-batch number 115000
    Training mini-batch number 116000
    Training mini-batch number 117000
    Training mini-batch number 118000
    Training mini-batch number 119000
    Epoch 23: validation accuracy 98.89%
    Training mini-batch number 120000
    Training mini-batch number 121000
    Training mini-batch number 122000
    Training mini-batch number 123000
    Training mini-batch number 124000
    Epoch 24: validation accuracy 98.90%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.94%
    Training mini-batch number 125000
    Training mini-batch number 126000
    Training mini-batch number 127000
    Training mini-batch number 128000
    Training mini-batch number 129000
    Epoch 25: validation accuracy 98.94%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 98.92%
    Training mini-batch number 130000
    Training mini-batch number 131000
    Training mini-batch number 132000
    Training mini-batch number 133000
    Training mini-batch number 134000
    Epoch 26: validation accuracy 98.93%
    Training mini-batch number 135000
    Training mini-batch number 136000
    Training mini-batch number 137000
    Training mini-batch number 138000
    Training mini-batch number 139000
    Epoch 27: validation accuracy 98.92%
    Training mini-batch number 140000
    Training mini-batch number 141000
    Training mini-batch number 142000
    Training mini-batch number 143000
    Training mini-batch number 144000
    Epoch 28: validation accuracy 98.92%
    Training mini-batch number 145000
    Training mini-batch number 146000
    Training mini-batch number 147000
    Training mini-batch number 148000
    Training mini-batch number 149000
    Epoch 29: validation accuracy 98.93%
    Training mini-batch number 150000
    Training mini-batch number 151000
    Training mini-batch number 152000
    Training mini-batch number 153000
    Training mini-batch number 154000
    Epoch 30: validation accuracy 98.92%
    Training mini-batch number 155000
    Training mini-batch number 156000
    Training mini-batch number 157000
    Training mini-batch number 158000
    Training mini-batch number 159000
    Epoch 31: validation accuracy 98.92%
    Training mini-batch number 160000
    Training mini-batch number 161000
    Training mini-batch number 162000
    Training mini-batch number 163000
    Training mini-batch number 164000
    Epoch 32: validation accuracy 98.92%
    Training mini-batch number 165000
    Training mini-batch number 166000
    Training mini-batch number 167000
    Training mini-batch number 168000
    Training mini-batch number 169000
    Epoch 33: validation accuracy 98.93%
    Training mini-batch number 170000
    Training mini-batch number 171000
    Training mini-batch number 172000
    Training mini-batch number 173000
    Training mini-batch number 174000
    Epoch 34: validation accuracy 98.92%
    Training mini-batch number 175000
    Training mini-batch number 176000
    Training mini-batch number 177000
    Training mini-batch number 178000
    Training mini-batch number 179000
    Epoch 35: validation accuracy 98.92%
    Training mini-batch number 180000
    Training mini-batch number 181000
    Training mini-batch number 182000
    Training mini-batch number 183000
    Training mini-batch number 184000
    Epoch 36: validation accuracy 98.93%
    Training mini-batch number 185000
    Training mini-batch number 186000
    Training mini-batch number 187000
    Training mini-batch number 188000
    Training mini-batch number 189000
    Epoch 37: validation accuracy 98.93%
    Training mini-batch number 190000
    Training mini-batch number 191000
    Training mini-batch number 192000
    Training mini-batch number 193000
    Training mini-batch number 194000
    Epoch 38: validation accuracy 98.92%
    Training mini-batch number 195000
    Training mini-batch number 196000
    Training mini-batch number 197000
    Training mini-batch number 198000
    Training mini-batch number 199000
    Epoch 39: validation accuracy 98.92%
    Finished training network.
    Best validation accuracy of 98.94% obtained at iteration 129999
    Corresponding test accuracy of 98.92%

```python
print validation
plot_test_accuracy(validation*100,40, 0)
```

As we can see we have improved our results with fully connected Networks. The architecture we were using is the one developped in 1998, known as LeNet-5. This accuracy is already amazing, but we can still improve it with a last trick.

We could modify the transfer function of the neurons, or modify the architecture. But we are going to highlight the importance of the training data. Indeed we are going to expand our dataset b displacing each training image up, down, left and right by only one pixel. We are going to see that it improves our accuracy by using the fact that the convolutional net will be recognize feature easily because it recognizes pattern.

Here is the result of this transformation :

```python
import network_homework_5
from network_homework_5 import Network
from network_homework_5 import ConvPoolLayer, FullyConnectedLayer, SoftmaxLayer

mini_batch_size=10
training_data, validation_data, test_data = network_homework_5.load_data_shared()
expanded_training_data, _, _ = network_homework_5.load_data_shared(
        "../data/mnist_expanded.pkl.gz")
```

```python
from network_homework_5 import ReLU
reseau = Network([
        ConvPoolLayer(image_shape=(mini_batch_size, 1, 28, 28),
                      filter_shape=(20, 1, 5, 5),
                      poolsize=(2, 2),
                      activation_fn=ReLU),
        ConvPoolLayer(image_shape=(mini_batch_size, 20, 12, 12),
                      filter_shape=(40, 20, 5, 5),
                      poolsize=(2, 2),
                      activation_fn=ReLU),
        FullyConnectedLayer(n_in=40*4*4, n_out=100, activation_fn=ReLU),
        SoftmaxLayer(n_in=100, n_out=10)], mini_batch_size)
reseau.SGD(expanded_training_data, 60, mini_batch_size, 0.03,
            validation_data, test_data, lmbda=0.1)
```

    WARNING (theano.gof.compilelock): Overriding existing lock by dead process '15464' (I am process '4860')

    Training mini-batch number 0
    Training mini-batch number 1000
    Training mini-batch number 2000
    Training mini-batch number 3000
    Training mini-batch number 4000
    Training mini-batch number 5000
    Training mini-batch number 6000
    Training mini-batch number 7000
    Training mini-batch number 8000
    Training mini-batch number 9000
    Training mini-batch number 10000
    Training mini-batch number 11000
    Training mini-batch number 12000
    Training mini-batch number 13000
    Training mini-batch number 14000
    Training mini-batch number 15000
    Training mini-batch number 16000
    Training mini-batch number 17000
    Training mini-batch number 18000
    Training mini-batch number 19000
    Training mini-batch number 20000
    Training mini-batch number 21000
    Training mini-batch number 22000
    Training mini-batch number 23000
    Training mini-batch number 24000
    Epoch 0: validation accuracy 99.06%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.04%
    Training mini-batch number 25000
    Training mini-batch number 26000
    Training mini-batch number 27000
    Training mini-batch number 28000
    Training mini-batch number 29000
    Training mini-batch number 30000
    Training mini-batch number 31000
    Training mini-batch number 32000
    Training mini-batch number 33000
    Training mini-batch number 34000
    Training mini-batch number 35000
    Training mini-batch number 36000
    Training mini-batch number 37000
    Training mini-batch number 38000
    Training mini-batch number 39000
    Training mini-batch number 40000
    Training mini-batch number 41000
    Training mini-batch number 42000
    Training mini-batch number 43000
    Training mini-batch number 44000
    Training mini-batch number 45000
    Training mini-batch number 46000
    Training mini-batch number 47000
    Training mini-batch number 48000
    Training mini-batch number 49000
    Epoch 1: validation accuracy 98.98%
    Training mini-batch number 50000
    Training mini-batch number 51000
    Training mini-batch number 52000
    Training mini-batch number 53000
    Training mini-batch number 54000
    Training mini-batch number 55000
    Training mini-batch number 56000
    Training mini-batch number 57000
    Training mini-batch number 58000
    Training mini-batch number 59000
    Training mini-batch number 60000
    Training mini-batch number 61000
    Training mini-batch number 62000
    Training mini-batch number 63000
    Training mini-batch number 64000
    Training mini-batch number 65000
    Training mini-batch number 66000
    Training mini-batch number 67000
    Training mini-batch number 68000
    Training mini-batch number 69000
    Training mini-batch number 70000
    Training mini-batch number 71000
    Training mini-batch number 72000
    Training mini-batch number 73000
    Training mini-batch number 74000
    Epoch 2: validation accuracy 99.23%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.21%
    Training mini-batch number 75000
    Training mini-batch number 76000
    Training mini-batch number 77000
    Training mini-batch number 78000
    Training mini-batch number 79000
    Training mini-batch number 80000
    Training mini-batch number 81000
    Training mini-batch number 82000
    Training mini-batch number 83000
    Training mini-batch number 84000
    Training mini-batch number 85000
    Training mini-batch number 86000
    Training mini-batch number 87000
    Training mini-batch number 88000
    Training mini-batch number 89000
    Training mini-batch number 90000
    Training mini-batch number 91000
    Training mini-batch number 92000
    Training mini-batch number 93000
    Training mini-batch number 94000
    Training mini-batch number 95000
    Training mini-batch number 96000
    Training mini-batch number 97000
    Training mini-batch number 98000
    Training mini-batch number 99000
    Epoch 3: validation accuracy 99.12%
    Training mini-batch number 100000
    Training mini-batch number 101000
    Training mini-batch number 102000
    Training mini-batch number 103000
    Training mini-batch number 104000
    Training mini-batch number 105000
    Training mini-batch number 106000
    Training mini-batch number 107000
    Training mini-batch number 108000
    Training mini-batch number 109000
    Training mini-batch number 110000
    Training mini-batch number 111000
    Training mini-batch number 112000
    Training mini-batch number 113000
    Training mini-batch number 114000
    Training mini-batch number 115000
    Training mini-batch number 116000
    Training mini-batch number 117000
    Training mini-batch number 118000
    Training mini-batch number 119000
    Training mini-batch number 120000
    Training mini-batch number 121000
    Training mini-batch number 122000
    Training mini-batch number 123000
    Training mini-batch number 124000
    Epoch 4: validation accuracy 99.24%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.19%
    Training mini-batch number 125000
    Training mini-batch number 126000
    Training mini-batch number 127000
    Training mini-batch number 128000
    Training mini-batch number 129000
    Training mini-batch number 130000
    Training mini-batch number 131000
    Training mini-batch number 132000
    Training mini-batch number 133000
    Training mini-batch number 134000
    Training mini-batch number 135000
    Training mini-batch number 136000
    Training mini-batch number 137000
    Training mini-batch number 138000
    Training mini-batch number 139000
    Training mini-batch number 140000
    Training mini-batch number 141000
    Training mini-batch number 142000
    Training mini-batch number 143000
    Training mini-batch number 144000
    Training mini-batch number 145000
    Training mini-batch number 146000
    Training mini-batch number 147000
    Training mini-batch number 148000
    Training mini-batch number 149000
    Epoch 5: validation accuracy 99.20%
    Training mini-batch number 150000
    Training mini-batch number 151000
    Training mini-batch number 152000
    Training mini-batch number 153000
    Training mini-batch number 154000
    Training mini-batch number 155000
    Training mini-batch number 156000
    Training mini-batch number 157000
    Training mini-batch number 158000
    Training mini-batch number 159000
    Training mini-batch number 160000
    Training mini-batch number 161000
    Training mini-batch number 162000
    Training mini-batch number 163000
    Training mini-batch number 164000
    Training mini-batch number 165000
    Training mini-batch number 166000
    Training mini-batch number 167000
    Training mini-batch number 168000
    Training mini-batch number 169000
    Training mini-batch number 170000
    Training mini-batch number 171000
    Training mini-batch number 172000
    Training mini-batch number 173000
    Training mini-batch number 174000
    Epoch 6: validation accuracy 99.26%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.24%
    Training mini-batch number 175000
    Training mini-batch number 176000
    Training mini-batch number 177000
    Training mini-batch number 178000
    Training mini-batch number 179000
    Training mini-batch number 180000
    Training mini-batch number 181000
    Training mini-batch number 182000
    Training mini-batch number 183000
    Training mini-batch number 184000
    Training mini-batch number 185000
    Training mini-batch number 186000
    Training mini-batch number 187000
    Training mini-batch number 188000
    Training mini-batch number 189000
    Training mini-batch number 190000
    Training mini-batch number 191000
    Training mini-batch number 192000
    Training mini-batch number 193000
    Training mini-batch number 194000
    Training mini-batch number 195000
    Training mini-batch number 196000
    Training mini-batch number 197000
    Training mini-batch number 198000
    Training mini-batch number 199000
    Epoch 7: validation accuracy 99.25%
    Training mini-batch number 200000
    Training mini-batch number 201000
    Training mini-batch number 202000
    Training mini-batch number 203000
    Training mini-batch number 204000
    Training mini-batch number 205000
    Training mini-batch number 206000
    Training mini-batch number 207000
    Training mini-batch number 208000
    Training mini-batch number 209000
    Training mini-batch number 210000
    Training mini-batch number 211000
    Training mini-batch number 212000
    Training mini-batch number 213000
    Training mini-batch number 214000
    Training mini-batch number 215000
    Training mini-batch number 216000
    Training mini-batch number 217000
    Training mini-batch number 218000
    Training mini-batch number 219000
    Training mini-batch number 220000
    Training mini-batch number 221000
    Training mini-batch number 222000
    Training mini-batch number 223000
    Training mini-batch number 224000
    Epoch 8: validation accuracy 99.23%
    Training mini-batch number 225000
    Training mini-batch number 226000
    Training mini-batch number 227000
    Training mini-batch number 228000
    Training mini-batch number 229000
    Training mini-batch number 230000
    Training mini-batch number 231000
    Training mini-batch number 232000
    Training mini-batch number 233000
    Training mini-batch number 234000
    Training mini-batch number 235000
    Training mini-batch number 236000
    Training mini-batch number 237000
    Training mini-batch number 238000
    Training mini-batch number 239000
    Training mini-batch number 240000
    Training mini-batch number 241000
    Training mini-batch number 242000
    Training mini-batch number 243000
    Training mini-batch number 244000
    Training mini-batch number 245000
    Training mini-batch number 246000
    Training mini-batch number 247000
    Training mini-batch number 248000
    Training mini-batch number 249000
    Epoch 9: validation accuracy 99.24%
    Training mini-batch number 250000
    Training mini-batch number 251000
    Training mini-batch number 252000
    Training mini-batch number 253000
    Training mini-batch number 254000
    Training mini-batch number 255000
    Training mini-batch number 256000
    Training mini-batch number 257000
    Training mini-batch number 258000
    Training mini-batch number 259000
    Training mini-batch number 260000
    Training mini-batch number 261000
    Training mini-batch number 262000
    Training mini-batch number 263000
    Training mini-batch number 264000
    Training mini-batch number 265000
    Training mini-batch number 266000
    Training mini-batch number 267000
    Training mini-batch number 268000
    Training mini-batch number 269000
    Training mini-batch number 270000
    Training mini-batch number 271000
    Training mini-batch number 272000
    Training mini-batch number 273000
    Training mini-batch number 274000
    Epoch 10: validation accuracy 99.19%
    Training mini-batch number 275000
    Training mini-batch number 276000
    Training mini-batch number 277000
    Training mini-batch number 278000
    Training mini-batch number 279000
    Training mini-batch number 280000
    Training mini-batch number 281000
    Training mini-batch number 282000
    Training mini-batch number 283000
    Training mini-batch number 284000
    Training mini-batch number 285000
    Training mini-batch number 286000
    Training mini-batch number 287000
    Training mini-batch number 288000
    Training mini-batch number 289000
    Training mini-batch number 290000
    Training mini-batch number 291000
    Training mini-batch number 292000
    Training mini-batch number 293000
    Training mini-batch number 294000
    Training mini-batch number 295000
    Training mini-batch number 296000
    Training mini-batch number 297000
    Training mini-batch number 298000
    Training mini-batch number 299000
    Epoch 11: validation accuracy 99.23%
    Training mini-batch number 300000
    Training mini-batch number 301000
    Training mini-batch number 302000
    Training mini-batch number 303000
    Training mini-batch number 304000
    Training mini-batch number 305000
    Training mini-batch number 306000
    Training mini-batch number 307000
    Training mini-batch number 308000
    Training mini-batch number 309000
    Training mini-batch number 310000
    Training mini-batch number 311000
    Training mini-batch number 312000
    Training mini-batch number 313000
    Training mini-batch number 314000
    Training mini-batch number 315000
    Training mini-batch number 316000
    Training mini-batch number 317000
    Training mini-batch number 318000
    Training mini-batch number 319000
    Training mini-batch number 320000
    Training mini-batch number 321000
    Training mini-batch number 322000
    Training mini-batch number 323000
    Training mini-batch number 324000
    Epoch 12: validation accuracy 99.26%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.33%
    Training mini-batch number 325000
    Training mini-batch number 326000
    Training mini-batch number 327000
    Training mini-batch number 328000
    Training mini-batch number 329000
    Training mini-batch number 330000
    Training mini-batch number 331000
    Training mini-batch number 332000
    Training mini-batch number 333000
    Training mini-batch number 334000
    Training mini-batch number 335000
    Training mini-batch number 336000
    Training mini-batch number 337000
    Training mini-batch number 338000
    Training mini-batch number 339000
    Training mini-batch number 340000
    Training mini-batch number 341000
    Training mini-batch number 342000
    Training mini-batch number 343000
    Training mini-batch number 344000
    Training mini-batch number 345000
    Training mini-batch number 346000
    Training mini-batch number 347000
    Training mini-batch number 348000
    Training mini-batch number 349000
    Epoch 13: validation accuracy 99.17%
    Training mini-batch number 350000
    Training mini-batch number 351000
    Training mini-batch number 352000
    Training mini-batch number 353000
    Training mini-batch number 354000
    Training mini-batch number 355000
    Training mini-batch number 356000
    Training mini-batch number 357000
    Training mini-batch number 358000
    Training mini-batch number 359000
    Training mini-batch number 360000
    Training mini-batch number 361000
    Training mini-batch number 362000
    Training mini-batch number 363000
    Training mini-batch number 364000
    Training mini-batch number 365000
    Training mini-batch number 366000
    Training mini-batch number 367000
    Training mini-batch number 368000
    Training mini-batch number 369000
    Training mini-batch number 370000
    Training mini-batch number 371000
    Training mini-batch number 372000
    Training mini-batch number 373000
    Training mini-batch number 374000
    Epoch 14: validation accuracy 99.26%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.34%
    Training mini-batch number 375000
    Training mini-batch number 376000
    Training mini-batch number 377000
    Training mini-batch number 378000
    Training mini-batch number 379000
    Training mini-batch number 380000
    Training mini-batch number 381000
    Training mini-batch number 382000
    Training mini-batch number 383000
    Training mini-batch number 384000
    Training mini-batch number 385000
    Training mini-batch number 386000
    Training mini-batch number 387000
    Training mini-batch number 388000
    Training mini-batch number 389000
    Training mini-batch number 390000
    Training mini-batch number 391000
    Training mini-batch number 392000
    Training mini-batch number 393000
    Training mini-batch number 394000
    Training mini-batch number 395000
    Training mini-batch number 396000
    Training mini-batch number 397000
    Training mini-batch number 398000
    Training mini-batch number 399000
    Epoch 15: validation accuracy 99.29%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.28%
    Training mini-batch number 400000
    Training mini-batch number 401000
    Training mini-batch number 402000
    Training mini-batch number 403000
    Training mini-batch number 404000
    Training mini-batch number 405000
    Training mini-batch number 406000
    Training mini-batch number 407000
    Training mini-batch number 408000
    Training mini-batch number 409000
    Training mini-batch number 410000
    Training mini-batch number 411000
    Training mini-batch number 412000
    Training mini-batch number 413000
    Training mini-batch number 414000
    Training mini-batch number 415000
    Training mini-batch number 416000
    Training mini-batch number 417000
    Training mini-batch number 418000
    Training mini-batch number 419000
    Training mini-batch number 420000
    Training mini-batch number 421000
    Training mini-batch number 422000
    Training mini-batch number 423000
    Training mini-batch number 424000
    Epoch 16: validation accuracy 99.38%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.33%
    Training mini-batch number 425000
    Training mini-batch number 426000
    Training mini-batch number 427000
    Training mini-batch number 428000
    Training mini-batch number 429000
    Training mini-batch number 430000
    Training mini-batch number 431000
    Training mini-batch number 432000
    Training mini-batch number 433000
    Training mini-batch number 434000
    Training mini-batch number 435000
    Training mini-batch number 436000
    Training mini-batch number 437000
    Training mini-batch number 438000
    Training mini-batch number 439000
    Training mini-batch number 440000
    Training mini-batch number 441000
    Training mini-batch number 442000
    Training mini-batch number 443000
    Training mini-batch number 444000
    Training mini-batch number 445000
    Training mini-batch number 446000
    Training mini-batch number 447000
    Training mini-batch number 448000
    Training mini-batch number 449000
    Epoch 17: validation accuracy 99.25%
    Training mini-batch number 450000
    Training mini-batch number 451000
    Training mini-batch number 452000
    Training mini-batch number 453000
    Training mini-batch number 454000
    Training mini-batch number 455000
    Training mini-batch number 456000
    Training mini-batch number 457000
    Training mini-batch number 458000
    Training mini-batch number 459000
    Training mini-batch number 460000
    Training mini-batch number 461000
    Training mini-batch number 462000
    Training mini-batch number 463000
    Training mini-batch number 464000
    Training mini-batch number 465000
    Training mini-batch number 466000
    Training mini-batch number 467000
    Training mini-batch number 468000
    Training mini-batch number 469000
    Training mini-batch number 470000
    Training mini-batch number 471000
    Training mini-batch number 472000
    Training mini-batch number 473000
    Training mini-batch number 474000
    Epoch 18: validation accuracy 99.29%
    Training mini-batch number 475000
    Training mini-batch number 476000
    Training mini-batch number 477000
    Training mini-batch number 478000
    Training mini-batch number 479000
    Training mini-batch number 480000
    Training mini-batch number 481000
    Training mini-batch number 482000
    Training mini-batch number 483000
    Training mini-batch number 484000
    Training mini-batch number 485000
    Training mini-batch number 486000
    Training mini-batch number 487000
    Training mini-batch number 488000
    Training mini-batch number 489000
    Training mini-batch number 490000
    Training mini-batch number 491000
    Training mini-batch number 492000
    Training mini-batch number 493000
    Training mini-batch number 494000
    Training mini-batch number 495000
    Training mini-batch number 496000
    Training mini-batch number 497000
    Training mini-batch number 498000
    Training mini-batch number 499000
    Epoch 19: validation accuracy 99.22%
    Training mini-batch number 500000
    Training mini-batch number 501000
    Training mini-batch number 502000
    Training mini-batch number 503000
    Training mini-batch number 504000
    Training mini-batch number 505000
    Training mini-batch number 506000
    Training mini-batch number 507000
    Training mini-batch number 508000
    Training mini-batch number 509000
    Training mini-batch number 510000
    Training mini-batch number 511000
    Training mini-batch number 512000
    Training mini-batch number 513000
    Training mini-batch number 514000
    Training mini-batch number 515000
    Training mini-batch number 516000
    Training mini-batch number 517000
    Training mini-batch number 518000
    Training mini-batch number 519000
    Training mini-batch number 520000
    Training mini-batch number 521000
    Training mini-batch number 522000
    Training mini-batch number 523000
    Training mini-batch number 524000
    Epoch 20: validation accuracy 99.21%
    Training mini-batch number 525000
    Training mini-batch number 526000
    Training mini-batch number 527000
    Training mini-batch number 528000
    Training mini-batch number 529000
    Training mini-batch number 530000
    Training mini-batch number 531000
    Training mini-batch number 532000
    Training mini-batch number 533000
    Training mini-batch number 534000
    Training mini-batch number 535000
    Training mini-batch number 536000
    Training mini-batch number 537000
    Training mini-batch number 538000
    Training mini-batch number 539000
    Training mini-batch number 540000
    Training mini-batch number 541000
    Training mini-batch number 542000
    Training mini-batch number 543000
    Training mini-batch number 544000
    Training mini-batch number 545000
    Training mini-batch number 546000
    Training mini-batch number 547000
    Training mini-batch number 548000
    Training mini-batch number 549000
    Epoch 21: validation accuracy 99.27%
    Training mini-batch number 550000
    Training mini-batch number 551000
    Training mini-batch number 552000
    Training mini-batch number 553000
    Training mini-batch number 554000
    Training mini-batch number 555000
    Training mini-batch number 556000
    Training mini-batch number 557000
    Training mini-batch number 558000
    Training mini-batch number 559000
    Training mini-batch number 560000
    Training mini-batch number 561000
    Training mini-batch number 562000
    Training mini-batch number 563000
    Training mini-batch number 564000
    Training mini-batch number 565000
    Training mini-batch number 566000
    Training mini-batch number 567000
    Training mini-batch number 568000
    Training mini-batch number 569000
    Training mini-batch number 570000
    Training mini-batch number 571000
    Training mini-batch number 572000
    Training mini-batch number 573000
    Training mini-batch number 574000
    Epoch 22: validation accuracy 99.29%
    Training mini-batch number 575000
    Training mini-batch number 576000
    Training mini-batch number 577000
    Training mini-batch number 578000
    Training mini-batch number 579000
    Training mini-batch number 580000
    Training mini-batch number 581000
    Training mini-batch number 582000
    Training mini-batch number 583000
    Training mini-batch number 584000
    Training mini-batch number 585000
    Training mini-batch number 586000
    Training mini-batch number 587000
    Training mini-batch number 588000
    Training mini-batch number 589000
    Training mini-batch number 590000
    Training mini-batch number 591000
    Training mini-batch number 592000
    Training mini-batch number 593000
    Training mini-batch number 594000
    Training mini-batch number 595000
    Training mini-batch number 596000
    Training mini-batch number 597000
    Training mini-batch number 598000
    Training mini-batch number 599000
    Epoch 23: validation accuracy 99.21%
    Training mini-batch number 600000
    Training mini-batch number 601000
    Training mini-batch number 602000
    Training mini-batch number 603000
    Training mini-batch number 604000
    Training mini-batch number 605000
    Training mini-batch number 606000
    Training mini-batch number 607000
    Training mini-batch number 608000
    Training mini-batch number 609000
    Training mini-batch number 610000
    Training mini-batch number 611000
    Training mini-batch number 612000
    Training mini-batch number 613000
    Training mini-batch number 614000
    Training mini-batch number 615000
    Training mini-batch number 616000
    Training mini-batch number 617000
    Training mini-batch number 618000
    Training mini-batch number 619000
    Training mini-batch number 620000
    Training mini-batch number 621000
    Training mini-batch number 622000
    Training mini-batch number 623000
    Training mini-batch number 624000
    Epoch 24: validation accuracy 99.33%
    Training mini-batch number 625000
    Training mini-batch number 626000
    Training mini-batch number 627000
    Training mini-batch number 628000
    Training mini-batch number 629000
    Training mini-batch number 630000
    Training mini-batch number 631000
    Training mini-batch number 632000
    Training mini-batch number 633000
    Training mini-batch number 634000
    Training mini-batch number 635000
    Training mini-batch number 636000
    Training mini-batch number 637000
    Training mini-batch number 638000
    Training mini-batch number 639000
    Training mini-batch number 640000
    Training mini-batch number 641000
    Training mini-batch number 642000
    Training mini-batch number 643000
    Training mini-batch number 644000
    Training mini-batch number 645000
    Training mini-batch number 646000
    Training mini-batch number 647000
    Training mini-batch number 648000
    Training mini-batch number 649000
    Epoch 25: validation accuracy 99.18%
    Training mini-batch number 650000
    Training mini-batch number 651000
    Training mini-batch number 652000
    Training mini-batch number 653000
    Training mini-batch number 654000
    Training mini-batch number 655000
    Training mini-batch number 656000
    Training mini-batch number 657000
    Training mini-batch number 658000
    Training mini-batch number 659000
    Training mini-batch number 660000
    Training mini-batch number 661000
    Training mini-batch number 662000
    Training mini-batch number 663000
    Training mini-batch number 664000
    Training mini-batch number 665000
    Training mini-batch number 666000
    Training mini-batch number 667000
    Training mini-batch number 668000
    Training mini-batch number 669000
    Training mini-batch number 670000
    Training mini-batch number 671000
    Training mini-batch number 672000
    Training mini-batch number 673000
    Training mini-batch number 674000
    Epoch 26: validation accuracy 99.28%
    Training mini-batch number 675000
    Training mini-batch number 676000
    Training mini-batch number 677000
    Training mini-batch number 678000
    Training mini-batch number 679000
    Training mini-batch number 680000
    Training mini-batch number 681000
    Training mini-batch number 682000
    Training mini-batch number 683000
    Training mini-batch number 684000
    Training mini-batch number 685000
    Training mini-batch number 686000
    Training mini-batch number 687000
    Training mini-batch number 688000
    Training mini-batch number 689000
    Training mini-batch number 690000
    Training mini-batch number 691000
    Training mini-batch number 692000
    Training mini-batch number 693000
    Training mini-batch number 694000
    Training mini-batch number 695000
    Training mini-batch number 696000
    Training mini-batch number 697000
    Training mini-batch number 698000
    Training mini-batch number 699000
    Epoch 27: validation accuracy 99.36%
    Training mini-batch number 700000
    Training mini-batch number 701000
    Training mini-batch number 702000
    Training mini-batch number 703000
    Training mini-batch number 704000
    Training mini-batch number 705000
    Training mini-batch number 706000
    Training mini-batch number 707000
    Training mini-batch number 708000
    Training mini-batch number 709000
    Training mini-batch number 710000
    Training mini-batch number 711000
    Training mini-batch number 712000
    Training mini-batch number 713000
    Training mini-batch number 714000
    Training mini-batch number 715000
    Training mini-batch number 716000
    Training mini-batch number 717000
    Training mini-batch number 718000
    Training mini-batch number 719000
    Training mini-batch number 720000
    Training mini-batch number 721000
    Training mini-batch number 722000
    Training mini-batch number 723000
    Training mini-batch number 724000
    Epoch 28: validation accuracy 99.23%
    Training mini-batch number 725000
    Training mini-batch number 726000
    Training mini-batch number 727000
    Training mini-batch number 728000
    Training mini-batch number 729000
    Training mini-batch number 730000
    Training mini-batch number 731000
    Training mini-batch number 732000
    Training mini-batch number 733000
    Training mini-batch number 734000
    Training mini-batch number 735000
    Training mini-batch number 736000
    Training mini-batch number 737000
    Training mini-batch number 738000
    Training mini-batch number 739000
    Training mini-batch number 740000
    Training mini-batch number 741000
    Training mini-batch number 742000
    Training mini-batch number 743000
    Training mini-batch number 744000
    Training mini-batch number 745000
    Training mini-batch number 746000
    Training mini-batch number 747000
    Training mini-batch number 748000
    Training mini-batch number 749000
    Epoch 29: validation accuracy 99.22%
    Training mini-batch number 750000
    Training mini-batch number 751000
    Training mini-batch number 752000
    Training mini-batch number 753000
    Training mini-batch number 754000
    Training mini-batch number 755000
    Training mini-batch number 756000
    Training mini-batch number 757000
    Training mini-batch number 758000
    Training mini-batch number 759000
    Training mini-batch number 760000
    Training mini-batch number 761000
    Training mini-batch number 762000
    Training mini-batch number 763000
    Training mini-batch number 764000
    Training mini-batch number 765000
    Training mini-batch number 766000
    Training mini-batch number 767000
    Training mini-batch number 768000
    Training mini-batch number 769000
    Training mini-batch number 770000
    Training mini-batch number 771000
    Training mini-batch number 772000
    Training mini-batch number 773000
    Training mini-batch number 774000
    Epoch 30: validation accuracy 99.26%
    Training mini-batch number 775000
    Training mini-batch number 776000
    Training mini-batch number 777000
    Training mini-batch number 778000
    Training mini-batch number 779000
    Training mini-batch number 780000
    Training mini-batch number 781000
    Training mini-batch number 782000
    Training mini-batch number 783000
    Training mini-batch number 784000
    Training mini-batch number 785000
    Training mini-batch number 786000
    Training mini-batch number 787000
    Training mini-batch number 788000
    Training mini-batch number 789000
    Training mini-batch number 790000
    Training mini-batch number 791000
    Training mini-batch number 792000
    Training mini-batch number 793000
    Training mini-batch number 794000
    Training mini-batch number 795000
    Training mini-batch number 796000
    Training mini-batch number 797000
    Training mini-batch number 798000
    Training mini-batch number 799000
    Epoch 31: validation accuracy 99.31%
    Training mini-batch number 800000
    Training mini-batch number 801000
    Training mini-batch number 802000
    Training mini-batch number 803000
    Training mini-batch number 804000
    Training mini-batch number 805000
    Training mini-batch number 806000
    Training mini-batch number 807000
    Training mini-batch number 808000
    Training mini-batch number 809000
    Training mini-batch number 810000
    Training mini-batch number 811000
    Training mini-batch number 812000
    Training mini-batch number 813000
    Training mini-batch number 814000
    Training mini-batch number 815000
    Training mini-batch number 816000
    Training mini-batch number 817000
    Training mini-batch number 818000
    Training mini-batch number 819000
    Training mini-batch number 820000
    Training mini-batch number 821000
    Training mini-batch number 822000
    Training mini-batch number 823000
    Training mini-batch number 824000
    Epoch 32: validation accuracy 99.38%
    Training mini-batch number 825000
    Training mini-batch number 826000
    Training mini-batch number 827000
    Training mini-batch number 828000
    Training mini-batch number 829000
    Training mini-batch number 830000
    Training mini-batch number 831000
    Training mini-batch number 832000
    Training mini-batch number 833000
    Training mini-batch number 834000
    Training mini-batch number 835000
    Training mini-batch number 836000
    Training mini-batch number 837000
    Training mini-batch number 838000
    Training mini-batch number 839000
    Training mini-batch number 840000
    Training mini-batch number 841000
    Training mini-batch number 842000
    Training mini-batch number 843000
    Training mini-batch number 844000
    Training mini-batch number 845000
    Training mini-batch number 846000
    Training mini-batch number 847000
    Training mini-batch number 848000
    Training mini-batch number 849000
    Epoch 33: validation accuracy 99.40%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.34%
    Training mini-batch number 850000
    Training mini-batch number 851000
    Training mini-batch number 852000
    Training mini-batch number 853000
    Training mini-batch number 854000
    Training mini-batch number 855000
    Training mini-batch number 856000
    Training mini-batch number 857000
    Training mini-batch number 858000
    Training mini-batch number 859000
    Training mini-batch number 860000
    Training mini-batch number 861000
    Training mini-batch number 862000
    Training mini-batch number 863000
    Training mini-batch number 864000
    Training mini-batch number 865000
    Training mini-batch number 866000
    Training mini-batch number 867000
    Training mini-batch number 868000
    Training mini-batch number 869000
    Training mini-batch number 870000
    Training mini-batch number 871000
    Training mini-batch number 872000
    Training mini-batch number 873000
    Training mini-batch number 874000
    Epoch 34: validation accuracy 99.41%
    This is the best validation accuracy to date.
    The corresponding test accuracy is 99.36%
    Training mini-batch number 875000
    Training mini-batch number 876000
    Training mini-batch number 877000
    Training mini-batch number 878000
    Training mini-batch number 879000
    Training mini-batch number 880000
    Training mini-batch number 881000
    Training mini-batch number 882000
    Training mini-batch number 883000
    Training mini-batch number 884000
    Training mini-batch number 885000
    Training mini-batch number 886000
    Training mini-batch number 887000
    Training mini-batch number 888000
    Training mini-batch number 889000
    Training mini-batch number 890000
    Training mini-batch number 891000
    Training mini-batch number 892000
    Training mini-batch number 893000
    Training mini-batch number 894000
    Training mini-batch number 895000
    Training mini-batch number 896000
    Training mini-batch number 897000
    Training mini-batch number 898000
    Training mini-batch number 899000
    Epoch 35: validation accuracy 99.39%
    Training mini-batch number 900000
    Training mini-batch number 901000
    Training mini-batch number 902000
    Training mini-batch number 903000
    Training mini-batch number 904000
    Training mini-batch number 905000
    Training mini-batch number 906000
    Training mini-batch number 907000
    Training mini-batch number 908000
    Training mini-batch number 909000
    Training mini-batch number 910000
    Training mini-batch number 911000
    Training mini-batch number 912000
    Training mini-batch number 913000
    Training mini-batch number 914000
    Training mini-batch number 915000
    Training mini-batch number 916000
    Training mini-batch number 917000
    Training mini-batch number 918000
    Training mini-batch number 919000
    Training mini-batch number 920000
    Training mini-batch number 921000
    Training mini-batch number 922000
    Training mini-batch number 923000
    Training mini-batch number 924000
    Epoch 36: validation accuracy 99.38%
    Training mini-batch number 925000
    Training mini-batch number 926000
    Training mini-batch number 927000
    Training mini-batch number 928000
    Training mini-batch number 929000
    Training mini-batch number 930000
    Training mini-batch number 931000
    Training mini-batch number 932000
    Training mini-batch number 933000
    Training mini-batch number 934000
    Training mini-batch number 935000
    Training mini-batch number 936000
    Training mini-batch number 937000
    Training mini-batch number 938000
    Training mini-batch number 939000
    Training mini-batch number 940000
    Training mini-batch number 941000
    Training mini-batch number 942000
    Training mini-batch number 943000
    Training mini-batch number 944000
    Training mini-batch number 945000
    Training mini-batch number 946000
    Training mini-batch number 947000
    Training mini-batch number 948000
    Training mini-batch number 949000
    Epoch 37: validation accuracy 99.38%
    Training mini-batch number 950000
    Training mini-batch number 951000
    Training mini-batch number 952000
    Training mini-batch number 953000
    Training mini-batch number 954000
    Training mini-batch number 955000
    Training mini-batch number 956000
    Training mini-batch number 957000
    Training mini-batch number 958000
    Training mini-batch number 959000
    Training mini-batch number 960000
    Training mini-batch number 961000
    Training mini-batch number 962000
    Training mini-batch number 963000
    Training mini-batch number 964000
    Training mini-batch number 965000
    Training mini-batch number 966000
    Training mini-batch number 967000
    Training mini-batch number 968000
    Training mini-batch number 969000
    Training mini-batch number 970000
    Training mini-batch number 971000
    Training mini-batch number 972000
    Training mini-batch number 973000
    Training mini-batch number 974000
    Epoch 38: validation accuracy 99.38%
    Training mini-batch number 975000
    Training mini-batch number 976000
    Training mini-batch number 977000
    Training mini-batch number 978000
    Training mini-batch number 979000
    Training mini-batch number 980000
    Training mini-batch number 981000
    Training mini-batch number 982000
    Training mini-batch number 983000
    Training mini-batch number 984000
    Training mini-batch number 985000
    Training mini-batch number 986000
    Training mini-batch number 987000
    Training mini-batch number 988000
    Training mini-batch number 989000
    Training mini-batch number 990000
    Training mini-batch number 991000
    Training mini-batch number 992000
    Training mini-batch number 993000
    Training mini-batch number 994000
    Training mini-batch number 995000
    Training mini-batch number 996000
    Training mini-batch number 997000
    Training mini-batch number 998000
    Training mini-batch number 999000
    Epoch 39: validation accuracy 99.38%
    Training mini-batch number 1000000
    Training mini-batch number 1001000
    Training mini-batch number 1002000
    Training mini-batch number 1003000
    Training mini-batch number 1004000
    Training mini-batch number 1005000
    Training mini-batch number 1006000
    Training mini-batch number 1007000
    Training mini-batch number 1008000
    Training mini-batch number 1009000
    Training mini-batch number 1010000
    Training mini-batch number 1011000
    Training mini-batch number 1012000
    Training mini-batch number 1013000
    Training mini-batch number 1014000
    Training mini-batch number 1015000
    Training mini-batch number 1016000
    Training mini-batch number 1017000
    Training mini-batch number 1018000
    Training mini-batch number 1019000
    Training mini-batch number 1020000
    Training mini-batch number 1021000
    Training mini-batch number 1022000
    Training mini-batch number 1023000
    Training mini-batch number 1024000
    Epoch 40: validation accuracy 99.38%
    Training mini-batch number 1025000
    Training mini-batch number 1026000
    Training mini-batch number 1027000
    Training mini-batch number 1028000
    Training mini-batch number 1029000
    Training mini-batch number 1030000
    Training mini-batch number 1031000
    Training mini-batch number 1032000
    Training mini-batch number 1033000
    Training mini-batch number 1034000
    Training mini-batch number 1035000
    Training mini-batch number 1036000
    Training mini-batch number 1037000
    Training mini-batch number 1038000
    Training mini-batch number 1039000
    Training mini-batch number 1040000
    Training mini-batch number 1041000
    Training mini-batch number 1042000
    Training mini-batch number 1043000
    Training mini-batch number 1044000
    Training mini-batch number 1045000
    Training mini-batch number 1046000
    Training mini-batch number 1047000
    Training mini-batch number 1048000
    Training mini-batch number 1049000
    Epoch 41: validation accuracy 99.38%
    Training mini-batch number 1050000
    Training mini-batch number 1051000
    Training mini-batch number 1052000
    Training mini-batch number 1053000
    Training mini-batch number 1054000
    Training mini-batch number 1055000
    Training mini-batch number 1056000
    Training mini-batch number 1057000
    Training mini-batch number 1058000
    Training mini-batch number 1059000
    Training mini-batch number 1060000
    Training mini-batch number 1061000
    Training mini-batch number 1062000
    Training mini-batch number 1063000
    Training mini-batch number 1064000
    Training mini-batch number 1065000
    Training mini-batch number 1066000
    Training mini-batch number 1067000
    Training mini-batch number 1068000
    Training mini-batch number 1069000
    Training mini-batch number 1070000
    Training mini-batch number 1071000
    Training mini-batch number 1072000
    Training mini-batch number 1073000
    Training mini-batch number 1074000
    Epoch 42: validation accuracy 99.38%
    Training mini-batch number 1075000
    Training mini-batch number 1076000
    Training mini-batch number 1077000
    Training mini-batch number 1078000
    Training mini-batch number 1079000
    Training mini-batch number 1080000
    Training mini-batch number 1081000
    Training mini-batch number 1082000
    Training mini-batch number 1083000
    Training mini-batch number 1084000
    Training mini-batch number 1085000
    Training mini-batch number 1086000
    Training mini-batch number 1087000
    Training mini-batch number 1088000
    Training mini-batch number 1089000
    Training mini-batch number 1090000
    Training mini-batch number 1091000
    Training mini-batch number 1092000
    Training mini-batch number 1093000
    Training mini-batch number 1094000
    Training mini-batch number 1095000
    Training mini-batch number 1096000
    Training mini-batch number 1097000
    Training mini-batch number 1098000
    Training mini-batch number 1099000
    Epoch 43: validation accuracy 99.38%
    Training mini-batch number 1100000
    Training mini-batch number 1101000
    Training mini-batch number 1102000
    Training mini-batch number 1103000
    Training mini-batch number 1104000
    Training mini-batch number 1105000
    Training mini-batch number 1106000
    Training mini-batch number 1107000
    Training mini-batch number 1108000
    Training mini-batch number 1109000
    Training mini-batch number 1110000
    Training mini-batch number 1111000
    Training mini-batch number 1112000
    Training mini-batch number 1113000
    Training mini-batch number 1114000
    Training mini-batch number 1115000
    Training mini-batch number 1116000
    Training mini-batch number 1117000
    Training mini-batch number 1118000
    Training mini-batch number 1119000
    Training mini-batch number 1120000
    Training mini-batch number 1121000
    Training mini-batch number 1122000
    Training mini-batch number 1123000
    Training mini-batch number 1124000
    Epoch 44: validation accuracy 99.39%
    Training mini-batch number 1125000
    Training mini-batch number 1126000
    Training mini-batch number 1127000
    Training mini-batch number 1128000
    Training mini-batch number 1129000
    Training mini-batch number 1130000
    Training mini-batch number 1131000
    Training mini-batch number 1132000
    Training mini-batch number 1133000
    Training mini-batch number 1134000
    Training mini-batch number 1135000
    Training mini-batch number 1136000
    Training mini-batch number 1137000
    Training mini-batch number 1138000
    Training mini-batch number 1139000
    Training mini-batch number 1140000
    Training mini-batch number 1141000
    Training mini-batch number 1142000
    Training mini-batch number 1143000
    Training mini-batch number 1144000
    Training mini-batch number 1145000
    Training mini-batch number 1146000
    Training mini-batch number 1147000
    Training mini-batch number 1148000
    Training mini-batch number 1149000
    Epoch 45: validation accuracy 99.39%
    Training mini-batch number 1150000
    Training mini-batch number 1151000
    Training mini-batch number 1152000
    Training mini-batch number 1153000
    Training mini-batch number 1154000
    Training mini-batch number 1155000
    Training mini-batch number 1156000
    Training mini-batch number 1157000
    Training mini-batch number 1158000
    Training mini-batch number 1159000
    Training mini-batch number 1160000
    Training mini-batch number 1161000
    Training mini-batch number 1162000
    Training mini-batch number 1163000
    Training mini-batch number 1164000
    Training mini-batch number 1165000
    Training mini-batch number 1166000
    Training mini-batch number 1167000
    Training mini-batch number 1168000
    Training mini-batch number 1169000
    Training mini-batch number 1170000
    Training mini-batch number 1171000
    Training mini-batch number 1172000
    Training mini-batch number 1173000
    Training mini-batch number 1174000
    Epoch 46: validation accuracy 99.39%
    Training mini-batch number 1175000
    Training mini-batch number 1176000
    Training mini-batch number 1177000
    Training mini-batch number 1178000
    Training mini-batch number 1179000
    Training mini-batch number 1180000
    Training mini-batch number 1181000
    Training mini-batch number 1182000
    Training mini-batch number 1183000
    Training mini-batch number 1184000
    Training mini-batch number 1185000
    Training mini-batch number 1186000
    Training mini-batch number 1187000
    Training mini-batch number 1188000
    Training mini-batch number 1189000
    Training mini-batch number 1190000
    Training mini-batch number 1191000
    Training mini-batch number 1192000
    Training mini-batch number 1193000
    Training mini-batch number 1194000
    Training mini-batch number 1195000
    Training mini-batch number 1196000
    Training mini-batch number 1197000
    Training mini-batch number 1198000
    Training mini-batch number 1199000
    Epoch 47: validation accuracy 99.39%
    Training mini-batch number 1200000
    Training mini-batch number 1201000
    Training mini-batch number 1202000
    Training mini-batch number 1203000
    Training mini-batch number 1204000
    Training mini-batch number 1205000
    Training mini-batch number 1206000
    Training mini-batch number 1207000
    Training mini-batch number 1208000
    Training mini-batch number 1209000
    Training mini-batch number 1210000
    Training mini-batch number 1211000
    Training mini-batch number 1212000
    Training mini-batch number 1213000
    Training mini-batch number 1214000
    Training mini-batch number 1215000
    Training mini-batch number 1216000
    Training mini-batch number 1217000
    Training mini-batch number 1218000
    Training mini-batch number 1219000
    Training mini-batch number 1220000
    Training mini-batch number 1221000
    Training mini-batch number 1222000
    Training mini-batch number 1223000
    Training mini-batch number 1224000
    Epoch 48: validation accuracy 99.38%
    Training mini-batch number 1225000
    Training mini-batch number 1226000
    Training mini-batch number 1227000
    Training mini-batch number 1228000
    Training mini-batch number 1229000
    Training mini-batch number 1230000
    Training mini-batch number 1231000
    Training mini-batch number 1232000
    Training mini-batch number 1233000
    Training mini-batch number 1234000
    Training mini-batch number 1235000
    Training mini-batch number 1236000
    Training mini-batch number 1237000
    Training mini-batch number 1238000
    Training mini-batch number 1239000
    Training mini-batch number 1240000
    Training mini-batch number 1241000
    Training mini-batch number 1242000
    Training mini-batch number 1243000
    Training mini-batch number 1244000
    Training mini-batch number 1245000
    Training mini-batch number 1246000
    Training mini-batch number 1247000
    Training mini-batch number 1248000
    Training mini-batch number 1249000
    Epoch 49: validation accuracy 99.37%
    Training mini-batch number 1250000
    Training mini-batch number 1251000
    Training mini-batch number 1252000
    Training mini-batch number 1253000
    Training mini-batch number 1254000
    Training mini-batch number 1255000
    Training mini-batch number 1256000
    Training mini-batch number 1257000
    Training mini-batch number 1258000
    Training mini-batch number 1259000
    Training mini-batch number 1260000
    Training mini-batch number 1261000
    Training mini-batch number 1262000
    Training mini-batch number 1263000
    Training mini-batch number 1264000
    Training mini-batch number 1265000
    Training mini-batch number 1266000
    Training mini-batch number 1267000
    Training mini-batch number 1268000
    Training mini-batch number 1269000
    Training mini-batch number 1270000
    Training mini-batch number 1271000
    Training mini-batch number 1272000
    Training mini-batch number 1273000
    Training mini-batch number 1274000
    Epoch 50: validation accuracy 99.37%
    Training mini-batch number 1275000
    Training mini-batch number 1276000
    Training mini-batch number 1277000
    Training mini-batch number 1278000
    Training mini-batch number 1279000
    Training mini-batch number 1280000
    Training mini-batch number 1281000
    Training mini-batch number 1282000
    Training mini-batch number 1283000
    Training mini-batch number 1284000
    Training mini-batch number 1285000
    Training mini-batch number 1286000
    Training mini-batch number 1287000
    Training mini-batch number 1288000
    Training mini-batch number 1289000
    Training mini-batch number 1290000
    Training mini-batch number 1291000
    Training mini-batch number 1292000
    Training mini-batch number 1293000
    Training mini-batch number 1294000
    Training mini-batch number 1295000
    Training mini-batch number 1296000
    Training mini-batch number 1297000
    Training mini-batch number 1298000
    Training mini-batch number 1299000
    Epoch 51: validation accuracy 99.38%
    Training mini-batch number 1300000
    Training mini-batch number 1301000
    Training mini-batch number 1302000
    Training mini-batch number 1303000
    Training mini-batch number 1304000
    Training mini-batch number 1305000
    Training mini-batch number 1306000
    Training mini-batch number 1307000
    Training mini-batch number 1308000
    Training mini-batch number 1309000
    Training mini-batch number 1310000
    Training mini-batch number 1311000
    Training mini-batch number 1312000
    Training mini-batch number 1313000
    Training mini-batch number 1314000
    Training mini-batch number 1315000
    Training mini-batch number 1316000
    Training mini-batch number 1317000
    Training mini-batch number 1318000
    Training mini-batch number 1319000
    Training mini-batch number 1320000
    Training mini-batch number 1321000
    Training mini-batch number 1322000
    Training mini-batch number 1323000
    Training mini-batch number 1324000
    Epoch 52: validation accuracy 99.38%
    Training mini-batch number 1325000
    Training mini-batch number 1326000
    Training mini-batch number 1327000
    Training mini-batch number 1328000
    Training mini-batch number 1329000
    Training mini-batch number 1330000
    Training mini-batch number 1331000
    Training mini-batch number 1332000
    Training mini-batch number 1333000
    Training mini-batch number 1334000
    Training mini-batch number 1335000
    Training mini-batch number 1336000
    Training mini-batch number 1337000
    Training mini-batch number 1338000
    Training mini-batch number 1339000
    Training mini-batch number 1340000
    Training mini-batch number 1341000
    Training mini-batch number 1342000
    Training mini-batch number 1343000
    Training mini-batch number 1344000
    Training mini-batch number 1345000
    Training mini-batch number 1346000
    Training mini-batch number 1347000
    Training mini-batch number 1348000
    Training mini-batch number 1349000
    Epoch 53: validation accuracy 99.39%
    Training mini-batch number 1350000
    Training mini-batch number 1351000
    Training mini-batch number 1352000
    Training mini-batch number 1353000
    Training mini-batch number 1354000
    Training mini-batch number 1355000
    Training mini-batch number 1356000
    Training mini-batch number 1357000
    Training mini-batch number 1358000
    Training mini-batch number 1359000
    Training mini-batch number 1360000
    Training mini-batch number 1361000
    Training mini-batch number 1362000
    Training mini-batch number 1363000
    Training mini-batch number 1364000
    Training mini-batch number 1365000
    Training mini-batch number 1366000
    Training mini-batch number 1367000
    Training mini-batch number 1368000
    Training mini-batch number 1369000
    Training mini-batch number 1370000
    Training mini-batch number 1371000
    Training mini-batch number 1372000
    Training mini-batch number 1373000
    Training mini-batch number 1374000
    Epoch 54: validation accuracy 99.39%
    Training mini-batch number 1375000
    Training mini-batch number 1376000
    Training mini-batch number 1377000
    Training mini-batch number 1378000
    Training mini-batch number 1379000
    Training mini-batch number 1380000
    Training mini-batch number 1381000
    Training mini-batch number 1382000
    Training mini-batch number 1383000
    Training mini-batch number 1384000
    Training mini-batch number 1385000
    Training mini-batch number 1386000
    Training mini-batch number 1387000
    Training mini-batch number 1388000
    Training mini-batch number 1389000
    Training mini-batch number 1390000
    Training mini-batch number 1391000
    Training mini-batch number 1392000
    Training mini-batch number 1393000
    Training mini-batch number 1394000
    Training mini-batch number 1395000
    Training mini-batch number 1396000
    Training mini-batch number 1397000
    Training mini-batch number 1398000
    Training mini-batch number 1399000
    Epoch 55: validation accuracy 99.39%
    Training mini-batch number 1400000
    Training mini-batch number 1401000
    Training mini-batch number 1402000
    Training mini-batch number 1403000
    Training mini-batch number 1404000
    Training mini-batch number 1405000
    Training mini-batch number 1406000
    Training mini-batch number 1407000
    Training mini-batch number 1408000
    Training mini-batch number 1409000
    Training mini-batch number 1410000
    Training mini-batch number 1411000
    Training mini-batch number 1412000
    Training mini-batch number 1413000
    Training mini-batch number 1414000
    Training mini-batch number 1415000
    Training mini-batch number 1416000
    Training mini-batch number 1417000
    Training mini-batch number 1418000
    Training mini-batch number 1419000
    Training mini-batch number 1420000
    Training mini-batch number 1421000
    Training mini-batch number 1422000
    Training mini-batch number 1423000
    Training mini-batch number 1424000
    Epoch 56: validation accuracy 99.39%
    Training mini-batch number 1425000
    Training mini-batch number 1426000
    Training mini-batch number 1427000
    Training mini-batch number 1428000
    Training mini-batch number 1429000
    Training mini-batch number 1430000
    Training mini-batch number 1431000
    Training mini-batch number 1432000
    Training mini-batch number 1433000
    Training mini-batch number 1434000
    Training mini-batch number 1435000
    Training mini-batch number 1436000
    Training mini-batch number 1437000
    Training mini-batch number 1438000
    Training mini-batch number 1439000
    Training mini-batch number 1440000
    Training mini-batch number 1441000
    Training mini-batch number 1442000
    Training mini-batch number 1443000
    Training mini-batch number 1444000
    Training mini-batch number 1445000
    Training mini-batch number 1446000
    Training mini-batch number 1447000
    Training mini-batch number 1448000
    Training mini-batch number 1449000
    Epoch 57: validation accuracy 99.40%
    Training mini-batch number 1450000
    Training mini-batch number 1451000
    Training mini-batch number 1452000
    Training mini-batch number 1453000
    Training mini-batch number 1454000
    Training mini-batch number 1455000
    Training mini-batch number 1456000
    Training mini-batch number 1457000
    Training mini-batch number 1458000
    Training mini-batch number 1459000
    Training mini-batch number 1460000
    Training mini-batch number 1461000
    Training mini-batch number 1462000
    Training mini-batch number 1463000
    Training mini-batch number 1464000
    Training mini-batch number 1465000
    Training mini-batch number 1466000
    Training mini-batch number 1467000
    Training mini-batch number 1468000
    Training mini-batch number 1469000
    Training mini-batch number 1470000
    Training mini-batch number 1471000
    Training mini-batch number 1472000
    Training mini-batch number 1473000
    Training mini-batch number 1474000
    Epoch 58: validation accuracy 99.39%
    Training mini-batch number 1475000
    Training mini-batch number 1476000
    Training mini-batch number 1477000
    Training mini-batch number 1478000
    Training mini-batch number 1479000
    Training mini-batch number 1480000
    Training mini-batch number 1481000
    Training mini-batch number 1482000
    Training mini-batch number 1483000
    Training mini-batch number 1484000
    Training mini-batch number 1485000
    Training mini-batch number 1486000
    Training mini-batch number 1487000
    Training mini-batch number 1488000
    Training mini-batch number 1489000
    Training mini-batch number 1490000
    Training mini-batch number 1491000
    Training mini-batch number 1492000
    Training mini-batch number 1493000
    Training mini-batch number 1494000
    Training mini-batch number 1495000
    Training mini-batch number 1496000
    Training mini-batch number 1497000
    Training mini-batch number 1498000
    Training mini-batch number 1499000
    Epoch 59: validation accuracy 99.39%
    Finished training network.
    Best validation accuracy of 99.41% obtained at iteration 874999
    Corresponding test accuracy of 99.36%

    [0.99059999999999859,
     0.98979999999999835,
     0.99229999999999874,
     0.99119999999999853,
     0.99239999999999873,
     0.99199999999999866,
     0.99259999999999882,
     0.99249999999999883,
     0.99229999999999885,
     0.99239999999999884,
     0.99189999999999889,
     0.99229999999999885,
     0.99259999999999882,
     0.9916999999999988,
     0.99259999999999893,
     0.99289999999999901,
     0.99379999999999902,
     0.99249999999999883,
     0.99289999999999901,
     0.99219999999999864,
     0.99209999999999876,
     0.99269999999999903,
     0.99289999999999889,
     0.99209999999999865,
     0.99329999999999896,
     0.99179999999999868,
     0.99279999999999891,
     0.99359999999999904,
     0.99229999999999885,
     0.99219999999999864,
     0.99259999999999893,
     0.99309999999999887,
     0.99379999999999891,
     0.993999999999999,
     0.99409999999999898,
     0.99389999999999901,
     0.99379999999999902,
     0.99379999999999902,
     0.99379999999999902,
     0.99379999999999902,
     0.99379999999999902,
     0.99379999999999902,
     0.99379999999999902,
     0.99379999999999902,
     0.99389999999999912,
     0.99389999999999912,
     0.99389999999999912,
     0.99389999999999912,
     0.99379999999999902,
     0.99369999999999903,
     0.99369999999999903,
     0.99379999999999902,
     0.99379999999999902,
     0.99389999999999912,
     0.99389999999999912,
     0.99389999999999912,
     0.99389999999999912,
     0.99399999999999911,
     0.99389999999999912,
     0.99389999999999912]

Hence we obtained 99,3 percent of accuracy. A small change in the expansion of the training data thus leads to a substantial improvement of the accuracy of out network.

In this part we have highlight the several way to improve accuracy on neural network : configuration of the network, datas ...

#### To finish this homework we can insist on the fact that the datas are essential to a good accuracy of learning with the following fact, and we can note that our result is not that far away of this one :

In 2003 Simard, Steinkraus and Platt improved their MNIST performance to 99.6 percent using a neural network otherwise very similar to ours, using two convolutional-pooling layers, followed by a hidden fully-connected layer with 100 neurons. There were a few differences of detail in their architecture - they didn't have the advantage of using rectified linear units, for instance - but the key to their improved performance was expanding the training data. They did this by rotating, translating, and skewing the MNIST training images. They also developed a process of "elastic distortion", a way of emulating the random oscillations hand muscles undergo when a person is writing. By combining all these processes they substantially increased the effective size of their training data, and that's how they achieved 99.6 percent accuracy.