# A short & practical introduction to Tensor Flow!

Part 5

Example of a single-layer one-directional long short-term memory network (LSTM) trained with
[connectionist temporal classification](http://www.cs.toronto.edu/~graves/icml_2006.pdf) to predict character sequences from nFeatures x nFrames
arrays of Mel-Frequency Cepstral Coefficients.  This is test code to run on the
8-item data set in the "sample_data" directory, for those without access to TIMIT.

Author: [Jon Rein](https://github.com/jonrein/tensorflow_CTC_example)

Adapted by: Pablo M. Olmos (olmos@tsc.uc3m.es)

Date: March 2017

```python
from __future__ import absolute_import
from __future__ import division
from __future__ import print_function

import tensorflow as tf
from tensorflow.python.ops import ctc_ops as ctc
import numpy as np
from utils import load_batched_data
```

```python
# Lets check what version of tensorflow we have installed. The provided scripts should run with tf 1.0 and above

print(tf.__version__)
```

    1.3.0

```python
batchSize = 4 # Batch of sequences

## Change according to the folder where you saved the dataset provided
INPUT_PATH = '../../DataSets/MCC_sample_data_phoneme_recog/mfcc' #directory of MFCC nFeatures x nFrames 2-D array .npy files
TARGET_PATH = '../../DataSets/MCC_sample_data_phoneme_recog/char_y/' #directory of nCharacters 1-D array .npy files

####Load data
print('Loading data')
batchedData, maxTimeSteps, totalN = load_batched_data(INPUT_PATH, TARGET_PATH, batchSize)
```

    Loading data

```python
####Learning Parameters
learningRate = 0.001
momentum = 0.9
nEpochs = 200

####Network Parameters
nFeatures = 26 #12 MFCC coefficients + energy, and derivatives
nHidden = 128
nClasses = 28#27 characters, plus the "blank" for CTC
```

## Creating the computation graph

We will create a LSTM layer with 128 memory cells. On top of this, we use a fully connected soft-max layer.

We use [CTC classification](http://www.cs.toronto.edu/~graves/icml_2006.pdf) to compute the loss function that we can optimize by gradient descend. This function is already provided in the TF [contributions library](        https://www.tensorflow.org/versions/r0.10/api_docs/python/nn/conectionist_temporal_classification__ctc_)

### LSTMs

Recall the fundamental model

Also, the un-regularized cost function is

\begin{align}
J(\boldsymbol{\theta})=\frac{1}{N}\sum_{n=1}^N\sum_{t=1}^{T_n}d(\boldsymbol{y}_t^{(n)},\text{softmax}(\boldsymbol{W}_h \boldsymbol{h}_t^{(n)}+\mathbf{b}))
\end{align}
where $d(\cdot,\cdot)$ is the cross-entropy loss function.

```python
####Define graph
print('Defining graph')
graph = tf.Graph()
with graph.as_default():

    #### We start by encoding the forward LSTM with nHidden cells

    #i(t) parameters
    # Input gate: input, previous output, and bias.
    ix = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.1, 0.1))   ##W^ix
    im = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.1, 0.1)) ## W^ih
    ib = tf.Variable(tf.zeros([1, nHidden])) ##b_i

    #f(t) parameters
    # Forget gate: input, previous output, and bias.
    fx = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.1, 0.1)) ##W^fx
    fm = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.1, 0.1)) ##W^fh
    fb = tf.Variable(tf.zeros([1, nHidden])) ##b_f

    #g(t) parameters
    # Memory cell: input, state and bias.
    cx = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.1, 0.1)) ##W^gx
    cm = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.1, 0.1)) ##W^gh
    cb = tf.Variable(tf.zeros([1, nHidden]))  ##b_g

    #o(t) parameters
    # Output gate: input, previous output, and bias.
    ox = tf.Variable(tf.truncated_normal([nFeatures, nHidden], -0.1, 0.1))  ##W^ox
    om = tf.Variable(tf.truncated_normal([nHidden, nHidden], -0.1, 0.1))  ##W^oh
    ob = tf.Variable(tf.zeros([1, nHidden])) ##b_o

    # Variable saving state across unrollings.
    saved_output = tf.Variable(tf.zeros([batchSize, nHidden]), trainable=False) #h(t)
    saved_state = tf.Variable(tf.zeros([batchSize, nHidden]), trainable=False) #s(t)
    ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####

    # Definition of the cell computation.
    def lstm_cell(i, o, state):
        """Create a LSTM cell. See e.g.: http://arxiv.org/pdf/1402.1128v1.pdf
        Note that in this formulation, we omit the various connections between the
        previous state and the gates."""

        input_gate = tf.sigmoid(tf.matmul(i, ix) + tf.matmul(o, im) + ib)
        forget_gate = tf.sigmoid(tf.matmul(i, fx) + tf.matmul(o, fm) + fb)
        update = tf.matmul(i, cx) + tf.matmul(o, cm) + cb
        state = forget_gate * state + input_gate * tf.tanh(update)    #tf.tanh(update) is g(t)
        output_gate = tf.sigmoid(tf.matmul(i, ox) + tf.matmul(o, om) + ob)
        return output_gate * tf.tanh(state), state      #h(t) is output_gate * tf.tanh(state)
    ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####

    # Classifier weights and biases (over h(t) and b_h(t) to labels)
    w = tf.Variable(tf.truncated_normal([nHidden, nClasses], -0.1, 0.1))
    b = tf.Variable(tf.zeros([nClasses]))

    ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####  ####

    # Now we define the placeholders for the input data

    train_data = list()
    for _ in range(maxTimeSteps):
        train_data.append(tf.placeholder(tf.float32, shape=[batchSize,nFeatures]))

    targetIxs = tf.placeholder(tf.int64)
    targetVals = tf.placeholder(tf.int32)
    targetShape = tf.placeholder(tf.int64)
    targetY = tf.SparseTensor(targetIxs, targetVals, targetShape)
    seqLengths = tf.placeholder(tf.int32, shape=(batchSize))

    # Given the input, we indicate how to compute the hidden states

    # Unrolled forward LSTM loop.
    outputs = list()
    output = saved_output
    state = saved_state
    for i in train_data:
        output, state = lstm_cell(i, output, state)
        outputs.append(output)

    # State saving across unrollings.

    # With this, we make sure that the loss is not computed until all states are first evaluate
    with tf.control_dependencies([saved_output.assign(output),saved_state.assign(state)]):
        #Classifier.
        logits = tf.reshape(tf.matmul(tf.concat(axis=0,values=outputs),w)+b,[-1,batchSize,nClasses])
        loss = tf.reduce_mean(ctc.ctc_loss(inputs=logits, labels=targetY,  sequence_length=seqLengths))

    ####Optimizing
    optimizer = tf.train.MomentumOptimizer(learningRate, momentum).minimize(loss)

    ####Evaluating
    # We use greedy decoding to estimate label sequences for training data
    logitsMaxTest = tf.slice(tf.argmax(logits, 2), [0, 0], [seqLengths[0], 1])
    predictions = tf.to_int32(ctc.ctc_beam_search_decoder(logits, seqLengths)[0][0])
    errorRate = tf.reduce_sum(tf.edit_distance(predictions, targetY, normalize=False)) / tf.to_float(tf.size(targetY.values))
```

    Defining graph

```python
####Run session
with tf.Session(graph=graph) as session:
    print('Initializing')
    tf.global_variables_initializer().run()

    for epoch in range(nEpochs):
        print('Epoch', epoch+1, '...')
        batchErrors = np.zeros(len(batchedData))
        batchRandIxs = np.random.permutation(len(batchedData)) #randomize batch order
        for batch, batchOrigI in enumerate(batchRandIxs):
            batchInputs, batchTargetSparse, batchSeqLengths = batchedData[batchOrigI]
            batchTargetIxs, batchTargetVals, batchTargetShape = batchTargetSparse
            feedDict = {targetIxs: batchTargetIxs, targetVals: batchTargetVals,
                        targetShape: batchTargetShape, seqLengths: batchSeqLengths}
            for i in range(maxTimeSteps):
                feedDict[train_data[i]] = batchInputs[i,:,:]

            _, l, er, lmt, logits_out = session.run([optimizer, loss, errorRate, logitsMaxTest,logits], feed_dict=feedDict)
            print(np.unique(lmt)) #print unique argmax values of first sample in batch; should be blank for a while, then spit out target values
            if (batch % 1) == 0:
                print('Minibatch', batch, '/', batchOrigI, 'loss:', l)
                print('Minibatch', batch, '/', batchOrigI, 'error rate:', er)
            batchErrors[batch] = er*len(batchSeqLengths)
        epochErrorRate = batchErrors.sum() / totalN
        print('Epoch', epoch+1, 'error rate:', epochErrorRate)

```

    Initializing
    Epoch 1 ...
    [ 0  1  2  3  4  5  6  7 10 11 12 13 14 15 16 17 18 20 21 23 24 25 26 27]
    Minibatch 0 / 0 loss: 908.24
    Minibatch 0 / 0 error rate: 1.72816
    [ 0  1  2  3  4  5  6  7  9 10 11 12 13 14 15 16 17 18 21 22 23 24 25 26 27]
    Minibatch 1 / 1 loss: 868.786
    Minibatch 1 / 1 error rate: 1.65306
    Epoch 1 error rate: 1.69060832262
    Epoch 2 ...
    [ 0  2  3  5 10 12 17 18 21 24 26 27]
    Minibatch 0 / 1 loss: 807.372
    Minibatch 0 / 1 error rate: 1.36395
    [27]
    Minibatch 1 / 0 loss: 701.823
    Minibatch 1 / 0 error rate: 0.919094
    Epoch 2 error rate: 1.14151972532
    Epoch 3 ...
    [27]
    Minibatch 0 / 1 loss: 529.089
    Minibatch 0 / 1 error rate: 0.659864
    [27]
    Minibatch 1 / 0 loss: 379.771
    Minibatch 1 / 0 error rate: 0.776699
    Epoch 3 error rate: 0.71828147769
    Epoch 4 ...
    [27]
    Minibatch 0 / 0 loss: 285.586
    Minibatch 0 / 0 error rate: 0.928803
    [27]
    Minibatch 1 / 1 loss: 235.717
    Minibatch 1 / 1 error rate: 0.979592
    Epoch 4 error rate: 0.954197227955
    Epoch 5 ...
    [27]
    Minibatch 0 / 0 loss: 249.514
    Minibatch 0 / 0 error rate: 0.987055
    [27]
    Minibatch 1 / 1 loss: 254.973
    Minibatch 1 / 1 error rate: 0.986395
    Epoch 5 error rate: 0.986724793911
    Epoch 6 ...
    [27]
    Minibatch 0 / 0 loss: 295.442
    Minibatch 0 / 0 error rate: 0.987055
    [27]
    Minibatch 1 / 1 loss: 289.621
    Minibatch 1 / 1 error rate: 0.989796
    Epoch 6 error rate: 0.988425463438
    Epoch 7 ...
    [27]
    Minibatch 0 / 1 loss: 287.891
    Minibatch 0 / 1 error rate: 0.986395
    [ 0 27]
    Minibatch 1 / 0 loss: 293.471
    Minibatch 1 / 0 error rate: 0.97411
    Epoch 7 error rate: 0.980252295732
    Epoch 8 ...
    [ 0 17 26 27]
    Minibatch 0 / 0 loss: 269.742
    Minibatch 0 / 0 error rate: 0.92233
    [ 0 26 27]
    Minibatch 1 / 1 loss: 226.844
    Minibatch 1 / 1 error rate: 0.870748
    Epoch 8 error rate: 0.896539181471
    Epoch 9 ...
    [ 0 26 27]
    Minibatch 0 / 0 loss: 236.611
    Minibatch 0 / 0 error rate: 0.783172
    [ 0 26 27]
    Minibatch 1 / 1 loss: 211.439
    Minibatch 1 / 1 error rate: 0.782313
    Epoch 9 error rate: 0.782742232084
    Epoch 10 ...
    [ 0 26 27]
    Minibatch 0 / 1 loss: 210.825
    Minibatch 0 / 1 error rate: 0.717687
    [ 0 12 26 27]
    Minibatch 1 / 0 loss: 226.738
    Minibatch 1 / 0 error rate: 0.699029
    Epoch 10 error rate: 0.708358108997
    Epoch 11 ...
    [ 0 12 26 27]
    Minibatch 0 / 0 loss: 227.267
    Minibatch 0 / 0 error rate: 0.699029
    [ 0 12 26 27]
    Minibatch 1 / 1 loss: 216.027
    Minibatch 1 / 1 error rate: 0.687075
    Epoch 11 error rate: 0.693051993847
    Epoch 12 ...
    [ 0 12 26 27]
    Minibatch 0 / 1 loss: 216.15
    Minibatch 0 / 1 error rate: 0.680272
    [ 0 12 26 27]
    Minibatch 1 / 0 loss: 226.51
    Minibatch 1 / 0 error rate: 0.68932
    Epoch 12 error rate: 0.684796243906
    Epoch 13 ...
    [ 0 12 26 27]
    Minibatch 0 / 0 loss: 224.573
    Minibatch 0 / 0 error rate: 0.692557
    [ 0 12 26 27]
    Minibatch 1 / 1 loss: 208.249
    Minibatch 1 / 1 error rate: 0.673469
    Epoch 13 error rate: 0.683012992144
    Epoch 14 ...
    [ 0  4 12 26 27]
    Minibatch 0 / 0 loss: 219.478
    Minibatch 0 / 0 error rate: 0.686084
    [ 0  4 12 15 26 27]
    Minibatch 1 / 1 loss: 201.333
    Minibatch 1 / 1 error rate: 0.663265
    Epoch 14 error rate: 0.674674719572
    Epoch 15 ...
    [ 0  4 12 26 27]
    Minibatch 0 / 0 loss: 214.9
    Minibatch 0 / 0 error rate: 0.699029
    [ 0  4 12 15 26 27]
    Minibatch 1 / 1 loss: 195.367
    Minibatch 1 / 1 error rate: 0.670068
    Epoch 15 error rate: 0.684548586607
    Epoch 16 ...
    [ 0  4 12 15 26 27]
    Minibatch 0 / 1 loss: 192.873
    Minibatch 0 / 1 error rate: 0.673469
    [ 0  4 12 15 22 26 27]
    Minibatch 1 / 0 loss: 210.695
    Minibatch 1 / 0 error rate: 0.695793
    Epoch 16 error rate: 0.684631109238
    Epoch 17 ...
    [ 0 12 15 22 26 27]
    Minibatch 0 / 1 loss: 188.494
    Minibatch 0 / 1 error rate: 0.666667
    [ 0  4 12 14 22 26 27]
    Minibatch 1 / 0 loss: 209.153
    Minibatch 1 / 0 error rate: 0.692557
    Epoch 17 error rate: 0.67961165309
    Epoch 18 ...
    [ 0 12 15 22 26 27]
    Minibatch 0 / 1 loss: 185.349
    Minibatch 0 / 1 error rate: 0.663265
    [ 0  4 12 14 15 22 26 27]
    Minibatch 1 / 0 loss: 207.611
    Minibatch 1 / 0 error rate: 0.686084
    Epoch 18 error rate: 0.674674719572
    Epoch 19 ...
    [ 0  1 12 15 22 26 27]
    Minibatch 0 / 1 loss: 182.62
    Minibatch 0 / 1 error rate: 0.659864
    [ 0  4 12 14 15 19 22 26 27]
    Minibatch 1 / 0 loss: 205.552
    Minibatch 1 / 0 error rate: 0.666667
    Epoch 19 error rate: 0.663265317678
    Epoch 20 ...
    [ 0  1  4 12 14 15 19 22 26 27]
    Minibatch 0 / 0 loss: 204.12
    Minibatch 0 / 0 error rate: 0.679612
    [ 0  1 12 15 22 26 27]
    Minibatch 1 / 1 loss: 178.352
    Minibatch 1 / 1 error rate: 0.64966
    Epoch 20 error rate: 0.664635747671
    Epoch 21 ...
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 0 / 0 loss: 200.672
    Minibatch 0 / 0 error rate: 0.650485
    [ 0  1  4 12 15 22 26 27]
    Minibatch 1 / 1 loss: 175.233
    Minibatch 1 / 1 error rate: 0.639456
    Epoch 21 error rate: 0.644970625639
    Epoch 22 ...
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 0 / 0 loss: 197.065
    Minibatch 0 / 0 error rate: 0.660194
    [ 0  1  4 12 15 19 22 26 27]
    Minibatch 1 / 1 loss: 172.153
    Minibatch 1 / 1 error rate: 0.62585
    Epoch 22 error rate: 0.643022239208
    Epoch 23 ...
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 0 / 0 loss: 193.703
    Minibatch 0 / 0 error rate: 0.656958
    [ 0  1  4  5  8 12 15 19 22 26 27]
    Minibatch 1 / 1 loss: 169.473
    Minibatch 1 / 1 error rate: 0.619048
    Epoch 23 error rate: 0.63800278306
    Epoch 24 ...
    [ 0  1  4  5  8 12 15 19 22 26 27]
    Minibatch 0 / 1 loss: 168.14
    Minibatch 0 / 1 error rate: 0.608844
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 1 / 0 loss: 189.673
    Minibatch 1 / 0 error rate: 0.647249
    Epoch 24 error rate: 0.628046363592
    Epoch 25 ...
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 0 / 0 loss: 188.361
    Minibatch 0 / 0 error rate: 0.63754
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 1 / 1 loss: 164.492
    Minibatch 1 / 1 error rate: 0.598639
    Epoch 25 error rate: 0.618089944124
    Epoch 26 ...
    [ 0  1  4  8 12 14 15 19 22 26 27]
    Minibatch 0 / 1 loss: 163.242
    Minibatch 0 / 1 error rate: 0.591837
    [ 0  1  4 12 14 15 19 21 22 26 27]
    Minibatch 1 / 0 loss: 184.54
    Minibatch 1 / 0 error rate: 0.631068
    Epoch 26 error rate: 0.61145234108
    Epoch 27 ...
    [ 0  1  4 12 14 15 19 22 26 27]
    Minibatch 0 / 1 loss: 160.287
    Minibatch 0 / 1 error rate: 0.57483
    [ 0  1  4 12 14 15 19 21 22 26 27]
    Minibatch 1 / 0 loss: 182.239
    Minibatch 1 / 0 error rate: 0.618123
    Epoch 27 error rate: 0.596476465464
    Epoch 28 ...
    [ 0  1  4  5 12 14 15 19 22 25 26 27]
    Minibatch 0 / 1 loss: 157.25
    Minibatch 0 / 1 error rate: 0.57483
    [ 0  1  4 12 14 15 19 21 22 26 27]
    Minibatch 1 / 0 loss: 179.874
    Minibatch 1 / 0 error rate: 0.614887
    Epoch 28 error rate: 0.59485834837
    Epoch 29 ...
    [ 0  1  4  5 12 14 15 19 22 25 26 27]
    Minibatch 0 / 1 loss: 154.278
    Minibatch 0 / 1 error rate: 0.55102
    [ 0  1  4 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 177.471
    Minibatch 1 / 0 error rate: 0.61165
    Epoch 29 error rate: 0.581335425377
    Epoch 30 ...
    [ 0  1  4 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 176.09
    Minibatch 0 / 0 error rate: 0.601942
    [ 0  1  4  5 12 14 15 19 22 25 26 27]
    Minibatch 1 / 1 loss: 150.539
    Minibatch 1 / 1 error rate: 0.547619
    Epoch 30 error rate: 0.574780404568
    Epoch 31 ...
    [ 0  1  4 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 173.141
    Minibatch 0 / 0 error rate: 0.592233
    [ 0  1  4  5 12 14 15 19 22 25 26 27]
    Minibatch 1 / 1 loss: 148.47
    Minibatch 1 / 1 error rate: 0.534014
    Epoch 31 error rate: 0.563123315573
    Epoch 32 ...
    [ 0  1  4  8 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 170.327
    Minibatch 0 / 0 error rate: 0.582524
    [ 0  1  4  5 12 14 15 19 22 25 26 27]
    Minibatch 1 / 1 loss: 146.39
    Minibatch 1 / 1 error rate: 0.534014
    Epoch 32 error rate: 0.558268964291
    Epoch 33 ...
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 167.599
    Minibatch 0 / 0 error rate: 0.559871
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 1 / 1 loss: 144.265
    Minibatch 1 / 1 error rate: 0.534014
    Epoch 33 error rate: 0.546942085028
    Epoch 34 ...
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 0 / 1 loss: 142.985
    Minibatch 0 / 1 error rate: 0.52381
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 164.078
    Minibatch 1 / 0 error rate: 0.563107
    Epoch 34 error rate: 0.543458163738
    Epoch 35 ...
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 0 / 1 loss: 140.362
    Minibatch 0 / 1 error rate: 0.510204
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 161.975
    Minibatch 1 / 0 error rate: 0.563107
    Epoch 35 error rate: 0.536655426025
    Epoch 36 ...
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 0 / 1 loss: 137.907
    Minibatch 0 / 1 error rate: 0.517007
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 159.724
    Minibatch 1 / 0 error rate: 0.556634
    Epoch 36 error rate: 0.536820560694
    Epoch 37 ...
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 0 / 1 loss: 135.645
    Minibatch 0 / 1 error rate: 0.496599
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 157.342
    Minibatch 1 / 0 error rate: 0.559871
    Epoch 37 error rate: 0.52823458612
    Epoch 38 ...
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 0 / 1 loss: 133.549
    Minibatch 0 / 1 error rate: 0.482993
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 154.899
    Minibatch 1 / 0 error rate: 0.556634
    Epoch 38 error rate: 0.519813746214
    Epoch 39 ...
    [ 0  1  4  5 12 14 19 22 25 26 27]
    Minibatch 0 / 1 loss: 131.578
    Minibatch 0 / 1 error rate: 0.479592
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 152.459
    Minibatch 1 / 0 error rate: 0.563107
    Epoch 39 error rate: 0.521349310875
    Epoch 40 ...
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 151.079
    Minibatch 0 / 0 error rate: 0.556634
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 129.183
    Minibatch 1 / 1 error rate: 0.462585
    Epoch 40 error rate: 0.509609669447
    Epoch 41 ...
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 128.295
    Minibatch 0 / 1 error rate: 0.459184
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 147.242
    Minibatch 1 / 0 error rate: 0.556634
    Epoch 41 error rate: 0.507908985019
    Epoch 42 ...
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 126.195
    Minibatch 0 / 1 error rate: 0.462585
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 145.114
    Minibatch 1 / 0 error rate: 0.533981
    Epoch 42 error rate: 0.498282819986
    Epoch 43 ...
    [ 0  1  4  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 143.94
    Minibatch 0 / 0 error rate: 0.527508
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 123.57
    Minibatch 1 / 1 error rate: 0.462585
    Epoch 43 error rate: 0.495046555996
    Epoch 44 ...
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 141.283
    Minibatch 0 / 0 error rate: 0.517799
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 122.069
    Minibatch 1 / 1 error rate: 0.445578
    Epoch 44 error rate: 0.481688797474
    Epoch 45 ...
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 121.14
    Minibatch 0 / 1 error rate: 0.442177
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 138.064
    Minibatch 1 / 0 error rate: 0.501618
    Epoch 45 error rate: 0.471897512674
    Epoch 46 ...
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 118.954
    Minibatch 0 / 1 error rate: 0.435374
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 136.229
    Minibatch 1 / 0 error rate: 0.501618
    Epoch 46 error rate: 0.468496143818
    Epoch 47 ...
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 116.918
    Minibatch 0 / 1 error rate: 0.431973
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 134.307
    Minibatch 1 / 0 error rate: 0.498382
    Epoch 47 error rate: 0.465177342296
    Epoch 48 ...
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 133.176
    Minibatch 0 / 0 error rate: 0.488673
    [ 0  1  4  5 12 14 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 114.495
    Minibatch 1 / 1 error rate: 0.431973
    Epoch 48 error rate: 0.460322976112
    Epoch 49 ...
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 130.587
    Minibatch 0 / 0 error rate: 0.469256
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 113.142
    Minibatch 1 / 1 error rate: 0.428571
    Epoch 49 error rate: 0.448913544416
    Epoch 50 ...
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 128.199
    Minibatch 0 / 0 error rate: 0.462783
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 111.65
    Minibatch 1 / 1 error rate: 0.421769
    Epoch 50 error rate: 0.442275926471
    Epoch 51 ...
    [ 0  1  4  5  8 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 125.96
    Minibatch 0 / 0 error rate: 0.446602
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 110.002
    Minibatch 1 / 1 error rate: 0.418367
    Epoch 51 error rate: 0.432484641671
    Epoch 52 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 108.991
    Minibatch 0 / 1 error rate: 0.414966
    [ 0  1  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 123.246
    Minibatch 1 / 0 error rate: 0.427184
    Epoch 52 error rate: 0.421075224876
    Epoch 53 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 106.751
    Minibatch 0 / 1 error rate: 0.408163
    [ 0  1  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 121.681
    Minibatch 1 / 0 error rate: 0.430421
    Epoch 53 error rate: 0.419292002916
    Epoch 54 ...
    [ 0  1  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 120.7
    Minibatch 0 / 0 error rate: 0.420712
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 104.076
    Minibatch 1 / 1 error rate: 0.401361
    Epoch 54 error rate: 0.411036252975
    Epoch 55 ...
    [ 0  1  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 118.294
    Minibatch 0 / 0 error rate: 0.423948
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 102.603
    Minibatch 1 / 1 error rate: 0.391156
    Epoch 55 error rate: 0.407552346587
    Epoch 56 ...
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 115.963
    Minibatch 0 / 0 error rate: 0.407767
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 101.095
    Minibatch 1 / 1 error rate: 0.384354
    Epoch 56 error rate: 0.396060362458
    Epoch 57 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 100.204
    Minibatch 0 / 1 error rate: 0.380952
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 112.991
    Minibatch 1 / 0 error rate: 0.398058
    Epoch 57 error rate: 0.389505326748
    Epoch 58 ...
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 112.038
    Minibatch 0 / 0 error rate: 0.391586
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 97.4281
    Minibatch 1 / 1 error rate: 0.37415
    Epoch 58 error rate: 0.382867708802
    Epoch 59 ...
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 109.809
    Minibatch 0 / 0 error rate: 0.391586
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 95.8622
    Minibatch 1 / 1 error rate: 0.370748
    Epoch 59 error rate: 0.381167039275
    Epoch 60 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 94.9914
    Minibatch 0 / 1 error rate: 0.360544
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 107.024
    Minibatch 1 / 0 error rate: 0.381877
    Epoch 60 error rate: 0.371210619807
    Epoch 61 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 92.8948
    Minibatch 0 / 1 error rate: 0.343537
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 105.358
    Minibatch 1 / 0 error rate: 0.378641
    Epoch 61 error rate: 0.361089095473
    Epoch 62 ...
    [ 0  1  2  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 104.367
    Minibatch 0 / 0 error rate: 0.375405
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 90.4112
    Minibatch 1 / 1 error rate: 0.336735
    Epoch 62 error rate: 0.356069609523
    Epoch 63 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 89.6665
    Minibatch 0 / 1 error rate: 0.333333
    [ 0  1  2  3  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 101.33
    Minibatch 1 / 0 error rate: 0.36246
    Epoch 63 error rate: 0.347896441817
    Epoch 64 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 87.7248
    Minibatch 0 / 1 error rate: 0.329932
    [ 0  1  2  3  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 99.5808
    Minibatch 1 / 0 error rate: 0.349515
    Epoch 64 error rate: 0.339723274112
    Epoch 65 ...
    [ 0  1  2  3  4  5  8  9 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 98.589
    Minibatch 0 / 0 error rate: 0.33657
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 85.3978
    Minibatch 1 / 1 error rate: 0.312925
    Epoch 65 error rate: 0.324747368693
    Epoch 66 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 84.7029
    Minibatch 0 / 1 error rate: 0.309524
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 95.6024
    Minibatch 1 / 0 error rate: 0.326861
    Epoch 66 error rate: 0.318192332983
    Epoch 67 ...
    [ 0  1  4  5 12 14 15 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 82.8116
    Minibatch 0 / 1 error rate: 0.292517
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 93.8895
    Minibatch 1 / 0 error rate: 0.323625
    Epoch 67 error rate: 0.308070793748
    Epoch 68 ...
    [ 0  1  4  5 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 81.0889
    Minibatch 0 / 1 error rate: 0.289116
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 92.1255
    Minibatch 1 / 0 error rate: 0.313916
    Epoch 68 error rate: 0.301515743136
    Epoch 69 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 91.1243
    Minibatch 0 / 0 error rate: 0.291262
    [ 0  1  4  5 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 78.9687
    Minibatch 1 / 1 error rate: 0.282313
    Epoch 69 error rate: 0.286787539721
    Epoch 70 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 88.8712
    Minibatch 0 / 0 error rate: 0.281553
    [ 0  1  4  5 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 77.7516
    Minibatch 1 / 1 error rate: 0.282313
    Epoch 70 error rate: 0.281933158636
    Epoch 71 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 86.7908
    Minibatch 0 / 0 error rate: 0.275081
    [ 0  1  4  5 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 76.4276
    Minibatch 1 / 1 error rate: 0.27551
    Epoch 71 error rate: 0.275295555592
    Epoch 72 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 84.8351
    Minibatch 0 / 0 error rate: 0.268608
    [ 0  1  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 75.009
    Minibatch 1 / 1 error rate: 0.268707
    Epoch 72 error rate: 0.268657952547
    Epoch 73 ...
    [ 0  1  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 74.3089
    Minibatch 0 / 1 error rate: 0.268707
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 82.4506
    Minibatch 1 / 0 error rate: 0.255663
    Epoch 73 error rate: 0.262185454369
    Epoch 74 ...
    [ 0  1  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 72.3404
    Minibatch 0 / 1 error rate: 0.251701
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 81.0593
    Minibatch 1 / 0 error rate: 0.255663
    Epoch 74 error rate: 0.253682047129
    Epoch 75 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 80.2114
    Minibatch 0 / 0 error rate: 0.255663
    [ 0  1  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 70.1542
    Minibatch 1 / 1 error rate: 0.231293
    Epoch 75 error rate: 0.243477970362
    Epoch 76 ...
    [ 0  1  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 69.6143
    Minibatch 0 / 1 error rate: 0.231293
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 77.6016
    Minibatch 1 / 0 error rate: 0.242718
    Epoch 76 error rate: 0.237005479634
    Epoch 77 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 76.7655
    Minibatch 0 / 0 error rate: 0.239482
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 67.4454
    Minibatch 1 / 1 error rate: 0.221088
    Epoch 77 error rate: 0.230285316706
    Epoch 78 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 74.8553
    Minibatch 0 / 0 error rate: 0.229773
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 66.273
    Minibatch 1 / 1 error rate: 0.214286
    Epoch 78 error rate: 0.222029589117
    Epoch 79 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 73.0838
    Minibatch 0 / 0 error rate: 0.220065
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 65.0191
    Minibatch 1 / 1 error rate: 0.204082
    Epoch 79 error rate: 0.21207318455
    Epoch 80 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 64.4405
    Minibatch 0 / 1 error rate: 0.204082
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 70.923
    Minibatch 1 / 0 error rate: 0.20712
    Epoch 80 error rate: 0.205600693822
    Epoch 81 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 70.2092
    Minibatch 0 / 0 error rate: 0.20712
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 62.1906
    Minibatch 1 / 1 error rate: 0.190476
    Epoch 81 error rate: 0.19879797101
    Epoch 82 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 68.4836
    Minibatch 0 / 0 error rate: 0.203883
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 61.0409
    Minibatch 1 / 1 error rate: 0.187075
    Epoch 82 error rate: 0.195479162037
    Epoch 83 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 66.8293
    Minibatch 0 / 0 error rate: 0.203883
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 59.9067
    Minibatch 1 / 1 error rate: 0.187075
    Epoch 83 error rate: 0.195479162037
    Epoch 84 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 59.3943
    Minibatch 0 / 1 error rate: 0.180272
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 64.787
    Minibatch 1 / 0 error rate: 0.190939
    Epoch 84 error rate: 0.185605309904
    Epoch 85 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 57.7066
    Minibatch 0 / 1 error rate: 0.173469
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 63.6146
    Minibatch 1 / 0 error rate: 0.190939
    Epoch 85 error rate: 0.182203955948
    Epoch 86 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 62.9136
    Minibatch 0 / 0 error rate: 0.184466
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 55.9066
    Minibatch 1 / 1 error rate: 0.166667
    Epoch 86 error rate: 0.175566345453
    Epoch 87 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 61.2751
    Minibatch 0 / 0 error rate: 0.177994
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 54.9166
    Minibatch 1 / 1 error rate: 0.156463
    Epoch 87 error rate: 0.16722805053
    Epoch 88 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 54.4954
    Minibatch 0 / 1 error rate: 0.156463
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 59.2487
    Minibatch 1 / 0 error rate: 0.171521
    Epoch 88 error rate: 0.163991808891
    Epoch 89 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 52.997
    Minibatch 0 / 1 error rate: 0.156463
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 58.0848
    Minibatch 1 / 0 error rate: 0.171521
    Epoch 89 error rate: 0.163991808891
    Epoch 90 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 57.4129
    Minibatch 0 / 0 error rate: 0.171521
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 51.3824
    Minibatch 1 / 1 error rate: 0.153061
    Epoch 90 error rate: 0.162291131914
    Epoch 91 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 55.8964
    Minibatch 0 / 0 error rate: 0.171521
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 50.4699
    Minibatch 1 / 1 error rate: 0.14966
    Epoch 91 error rate: 0.160590447485
    Epoch 92 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 54.492
    Minibatch 0 / 0 error rate: 0.165049
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 1 / 1 loss: 49.5126
    Minibatch 1 / 1 error rate: 0.14966
    Epoch 92 error rate: 0.157354198396
    Epoch 93 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 49.1084
    Minibatch 0 / 1 error rate: 0.146259
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 52.7747
    Minibatch 1 / 0 error rate: 0.158576
    Epoch 93 error rate: 0.15241727978
    Epoch 94 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 47.6847
    Minibatch 0 / 1 error rate: 0.139456
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 51.7958
    Minibatch 1 / 0 error rate: 0.15534
    Epoch 94 error rate: 0.147397793829
    Epoch 95 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 21 22 25 26 27]
    Minibatch 0 / 1 loss: 46.5226
    Minibatch 0 / 1 error rate: 0.139456
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 50.7579
    Minibatch 1 / 0 error rate: 0.148867
    Epoch 95 error rate: 0.14416154474
    Epoch 96 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 45.4419
    Minibatch 0 / 1 error rate: 0.132653
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 49.6843
    Minibatch 1 / 0 error rate: 0.135922
    Epoch 96 error rate: 0.134287692606
    Epoch 97 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 44.4349
    Minibatch 0 / 1 error rate: 0.129252
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 48.6112
    Minibatch 1 / 0 error rate: 0.135922
    Epoch 97 error rate: 0.132587015629
    Epoch 98 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 48.0176
    Minibatch 0 / 0 error rate: 0.135922
    [ 0  1  2  3  4  5  8 12 14 15 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 43.1858
    Minibatch 1 / 1 error rate: 0.129252
    Epoch 98 error rate: 0.132587015629
    Epoch 99 ...
    [ 0  1  2  3  4  5  8 12 14 15 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 42.9167
    Minibatch 0 / 1 error rate: 0.129252
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 46.3079
    Minibatch 1 / 0 error rate: 0.122977
    Epoch 99 error rate: 0.126114524901
    Epoch 100 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 45.7775
    Minibatch 0 / 0 error rate: 0.119741
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 41.4465
    Minibatch 1 / 1 error rate: 0.122449
    Epoch 100 error rate: 0.12109503895
    Epoch 101 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 41.1696
    Minibatch 0 / 1 error rate: 0.119048
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 44.225
    Minibatch 1 / 0 error rate: 0.110032
    Epoch 101 error rate: 0.114539992064
    Epoch 102 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 43.7239
    Minibatch 0 / 0 error rate: 0.106796
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 39.7339
    Minibatch 1 / 1 error rate: 0.119048
    Epoch 102 error rate: 0.11292186752
    Epoch 103 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 42.6086
    Minibatch 0 / 0 error rate: 0.10356
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 39.003
    Minibatch 1 / 1 error rate: 0.115646
    Epoch 103 error rate: 0.109603065997
    Epoch 104 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 38.706
    Minibatch 0 / 1 error rate: 0.115646
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 41.2605
    Minibatch 1 / 0 error rate: 0.0938511
    Epoch 104 error rate: 0.104748696089
    Epoch 105 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 37.5728
    Minibatch 0 / 1 error rate: 0.108844
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 40.5013
    Minibatch 1 / 0 error rate: 0.0938511
    Epoch 105 error rate: 0.101347334683
    Epoch 106 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 36.6489
    Minibatch 0 / 1 error rate: 0.102041
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 39.7202
    Minibatch 1 / 0 error rate: 0.0873786
    Epoch 106 error rate: 0.0947097316384
    Epoch 107 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 35.7862
    Minibatch 0 / 1 error rate: 0.0986395
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 38.9056
    Minibatch 1 / 0 error rate: 0.0873786
    Epoch 107 error rate: 0.0930090509355
    Epoch 108 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 34.979
    Minibatch 0 / 1 error rate: 0.0952381
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 38.0806
    Minibatch 1 / 0 error rate: 0.0809061
    Epoch 108 error rate: 0.0880721211433
    Epoch 109 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 37.6293
    Minibatch 0 / 0 error rate: 0.0776699
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 33.9991
    Minibatch 1 / 1 error rate: 0.0952381
    Epoch 109 error rate: 0.086454000324
    Epoch 110 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 36.6212
    Minibatch 0 / 0 error rate: 0.0711974
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 33.4162
    Minibatch 1 / 1 error rate: 0.0952381
    Epoch 110 error rate: 0.0832177549601
    Epoch 111 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 35.7088
    Minibatch 0 / 0 error rate: 0.0711974
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 32.7846
    Minibatch 1 / 1 error rate: 0.0952381
    Epoch 111 error rate: 0.0832177549601
    Epoch 112 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 34.8706
    Minibatch 0 / 0 error rate: 0.0711974
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 32.1321
    Minibatch 1 / 1 error rate: 0.0918367
    Epoch 112 error rate: 0.0815170742571
    Epoch 113 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 31.8611
    Minibatch 0 / 1 error rate: 0.0918367
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 33.8494
    Minibatch 1 / 0 error rate: 0.0679612
    Epoch 113 error rate: 0.0798989497125
    Epoch 114 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 30.9229
    Minibatch 0 / 1 error rate: 0.085034
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 33.248
    Minibatch 1 / 0 error rate: 0.0679612
    Epoch 114 error rate: 0.0764975883067
    Epoch 115 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 32.8992
    Minibatch 0 / 0 error rate: 0.0647249
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 29.9397
    Minibatch 1 / 1 error rate: 0.0748299
    Epoch 115 error rate: 0.0697774253786
    Epoch 116 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 29.7115
    Minibatch 0 / 1 error rate: 0.0680272
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 31.8424
    Minibatch 1 / 0 error rate: 0.0647249
    Epoch 116 error rate: 0.066376067698
    Epoch 117 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 28.8933
    Minibatch 0 / 1 error rate: 0.0680272
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 31.2487
    Minibatch 1 / 0 error rate: 0.0614887
    Epoch 117 error rate: 0.0647579431534
    Epoch 118 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 30.92
    Minibatch 0 / 0 error rate: 0.0614887
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 28.0056
    Minibatch 1 / 1 error rate: 0.0612245
    Epoch 118 error rate: 0.0613565817475
    Epoch 119 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 27.7873
    Minibatch 0 / 1 error rate: 0.0612245
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 29.9137
    Minibatch 1 / 0 error rate: 0.0582524
    Epoch 119 error rate: 0.0597384590656
    Epoch 120 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 29.5975
    Minibatch 0 / 0 error rate: 0.0614887
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 26.8777
    Minibatch 1 / 1 error rate: 0.0578231
    Epoch 120 error rate: 0.0596559010446
    Epoch 121 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 26.6562
    Minibatch 0 / 1 error rate: 0.0578231
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 28.6398
    Minibatch 1 / 0 error rate: 0.0582524
    Epoch 121 error rate: 0.0580377783626
    Epoch 122 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 25.9766
    Minibatch 0 / 1 error rate: 0.0578231
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 28.1089
    Minibatch 1 / 0 error rate: 0.0582524
    Epoch 122 error rate: 0.0580377783626
    Epoch 123 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 27.814
    Minibatch 0 / 0 error rate: 0.0582524
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 25.209
    Minibatch 1 / 1 error rate: 0.0578231
    Epoch 123 error rate: 0.0580377783626
    Epoch 124 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 27.1287
    Minibatch 0 / 0 error rate: 0.0582524
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 24.7723
    Minibatch 1 / 1 error rate: 0.0578231
    Epoch 124 error rate: 0.0580377783626
    Epoch 125 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 26.4991
    Minibatch 0 / 0 error rate: 0.0582524
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 24.3227
    Minibatch 1 / 1 error rate: 0.0544218
    Epoch 125 error rate: 0.0563370976597
    Epoch 126 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 24.0823
    Minibatch 0 / 1 error rate: 0.0510204
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 25.731
    Minibatch 1 / 0 error rate: 0.0550162
    Epoch 126 error rate: 0.0530182961375
    Epoch 127 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 23.4722
    Minibatch 0 / 1 error rate: 0.0510204
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 25.2911
    Minibatch 1 / 0 error rate: 0.0550162
    Epoch 127 error rate: 0.0530182961375
    Epoch 128 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 22.9322
    Minibatch 0 / 1 error rate: 0.0510204
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 24.8313
    Minibatch 1 / 0 error rate: 0.0517799
    Epoch 128 error rate: 0.0514001734555
    Epoch 129 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 22.4309
    Minibatch 0 / 1 error rate: 0.047619
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 24.3595
    Minibatch 1 / 0 error rate: 0.0517799
    Epoch 129 error rate: 0.0496994927526
    Epoch 130 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 24.101
    Minibatch 0 / 0 error rate: 0.0485437
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 21.8291
    Minibatch 1 / 1 error rate: 0.047619
    Epoch 130 error rate: 0.0480813682079
    Epoch 131 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 21.6317
    Minibatch 0 / 1 error rate: 0.047619
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 23.3096
    Minibatch 1 / 0 error rate: 0.0485437
    Epoch 131 error rate: 0.0480813682079
    Epoch 132 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 23.073
    Minibatch 0 / 0 error rate: 0.0485437
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 21.025
    Minibatch 1 / 1 error rate: 0.0442177
    Epoch 132 error rate: 0.046380687505
    Epoch 133 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 20.8189
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 22.3458
    Minibatch 1 / 0 error rate: 0.0420712
    Epoch 133 error rate: 0.0397430807352
    Epoch 134 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 20.3761
    Minibatch 0 / 1 error rate: 0.0408163
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 21.9372
    Minibatch 1 / 0 error rate: 0.0420712
    Epoch 134 error rate: 0.0414437614381
    Epoch 135 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 21.7127
    Minibatch 0 / 0 error rate: 0.0420712
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 19.828
    Minibatch 1 / 1 error rate: 0.0408163
    Epoch 135 error rate: 0.0414437614381
    Epoch 136 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 21.1885
    Minibatch 0 / 0 error rate: 0.038835
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 19.5169
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 136 error rate: 0.0381249580532
    Epoch 137 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 19.3032
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 20.5628
    Minibatch 1 / 0 error rate: 0.038835
    Epoch 137 error rate: 0.0381249580532
    Epoch 138 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 20.3732
    Minibatch 0 / 0 error rate: 0.038835
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 18.7692
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 138 error rate: 0.0381249580532
    Epoch 139 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 18.557
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 19.7713
    Minibatch 1 / 0 error rate: 0.0355987
    Epoch 139 error rate: 0.0365068353713
    Epoch 140 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 18.187
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 19.429
    Minibatch 1 / 0 error rate: 0.0355987
    Epoch 140 error rate: 0.0365068353713
    Epoch 141 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 17.8114
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 19.0788
    Minibatch 1 / 0 error rate: 0.0355987
    Epoch 141 error rate: 0.0365068353713
    Epoch 142 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 18.8914
    Minibatch 0 / 0 error rate: 0.0355987
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 17.3601
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 142 error rate: 0.0365068353713
    Epoch 143 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 18.4381
    Minibatch 0 / 0 error rate: 0.0323625
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 17.1011
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 143 error rate: 0.0348887126893
    Epoch 144 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 16.9021
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 17.9039
    Minibatch 1 / 0 error rate: 0.0323625
    Epoch 144 error rate: 0.0348887126893
    Epoch 145 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 16.5858
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 17.6049
    Minibatch 1 / 0 error rate: 0.02589
    Epoch 145 error rate: 0.031652466394
    Epoch 146 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 17.4415
    Minibatch 0 / 0 error rate: 0.0226537
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 16.1555
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 146 error rate: 0.0300343427807
    Epoch 147 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 17.0359
    Minibatch 0 / 0 error rate: 0.0226537
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 15.9126
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 147 error rate: 0.0300343427807
    Epoch 148 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 16.6715
    Minibatch 0 / 0 error rate: 0.0226537
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 15.6635
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 148 error rate: 0.0300343427807
    Epoch 149 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 16.3303
    Minibatch 0 / 0 error rate: 0.0226537
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 15.4034
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 149 error rate: 0.0300343427807
    Epoch 150 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 16.0075
    Minibatch 0 / 0 error rate: 0.0194175
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 15.1392
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 150 error rate: 0.0284162200987
    Epoch 151 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 14.9414
    Minibatch 0 / 1 error rate: 0.0340136
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 15.6135
    Minibatch 1 / 0 error rate: 0.0161812
    Epoch 151 error rate: 0.0250974185765
    Epoch 152 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 14.6636
    Minibatch 0 / 1 error rate: 0.037415
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 15.3823
    Minibatch 1 / 0 error rate: 0.0161812
    Epoch 152 error rate: 0.0267980974168
    Epoch 153 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 15.2572
    Minibatch 0 / 0 error rate: 0.0161812
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 14.2719
    Minibatch 1 / 1 error rate: 0.037415
    Epoch 153 error rate: 0.0267980974168
    Epoch 154 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 14.0946
    Minibatch 0 / 1 error rate: 0.0340136
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 14.8271
    Minibatch 1 / 0 error rate: 0.0161812
    Epoch 154 error rate: 0.0250974185765
    Epoch 155 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 14.7083
    Minibatch 0 / 0 error rate: 0.0161812
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 13.7824
    Minibatch 1 / 1 error rate: 0.0340136
    Epoch 155 error rate: 0.0250974185765
    Epoch 156 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 14.3919
    Minibatch 0 / 0 error rate: 0.0161812
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 13.5738
    Minibatch 1 / 1 error rate: 0.0340136
    Epoch 156 error rate: 0.0250974185765
    Epoch 157 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 13.3958
    Minibatch 0 / 1 error rate: 0.0340136
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 14.0264
    Minibatch 1 / 0 error rate: 0.0161812
    Epoch 157 error rate: 0.0250974185765
    Epoch 158 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 13.9202
    Minibatch 0 / 0 error rate: 0.0161812
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 13.1001
    Minibatch 1 / 1 error rate: 0.0340136
    Epoch 158 error rate: 0.0250974185765
    Epoch 159 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 12.9246
    Minibatch 0 / 1 error rate: 0.0306122
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 13.5495
    Minibatch 1 / 0 error rate: 0.0161812
    Epoch 159 error rate: 0.0233967378736
    Epoch 160 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 13.447
    Minibatch 0 / 0 error rate: 0.012945
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 12.6522
    Minibatch 1 / 1 error rate: 0.0306122
    Epoch 160 error rate: 0.0217786147259
    Epoch 161 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 13.1735
    Minibatch 0 / 0 error rate: 0.012945
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 12.4632
    Minibatch 1 / 1 error rate: 0.0272109
    Epoch 161 error rate: 0.020077934023
    Epoch 162 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 12.9288
    Minibatch 0 / 0 error rate: 0.012945
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 12.2709
    Minibatch 1 / 1 error rate: 0.0272109
    Epoch 162 error rate: 0.020077934023
    Epoch 163 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 12.696
    Minibatch 0 / 0 error rate: 0.012945
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 12.0759
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 163 error rate: 0.0183772542514
    Epoch 164 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 12.4764
    Minibatch 0 / 0 error rate: 0.012945
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 11.879
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 164 error rate: 0.0183772542514
    Epoch 165 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 12.2673
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 11.6836
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 165 error rate: 0.0167591311038
    Epoch 166 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 11.5177
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 12.0069
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 166 error rate: 0.0167591311038
    Epoch 167 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 11.9327
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 11.2677
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 167 error rate: 0.0167591311038
    Epoch 168 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 11.7114
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 11.097
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 168 error rate: 0.0167591311038
    Epoch 169 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 10.94
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 11.4544
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 169 error rate: 0.0167591311038
    Epoch 170 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 11.385
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 10.7209
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 170 error rate: 0.0167591311038
    Epoch 171 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 10.57
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 11.1149
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 171 error rate: 0.0167591311038
    Epoch 172 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 11.0445
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 10.3773
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 172 error rate: 0.0167591311038
    Epoch 173 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 10.2297
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 10.7812
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 173 error rate: 0.0167591311038
    Epoch 174 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 10.1118
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 10.6379
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 174 error rate: 0.0167591311038
    Epoch 175 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 10.5685
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 9.88518
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 175 error rate: 0.0167591311038
    Epoch 176 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 10.3711
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 9.75605
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 176 error rate: 0.0167591311038
    Epoch 177 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 9.62226
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 10.146
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 177 error rate: 0.0167591311038
    Epoch 178 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 9.51055
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 10.0184
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 178 error rate: 0.0167591311038
    Epoch 179 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 9.3521
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 9.88622
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 179 error rate: 0.0167591311038
    Epoch 180 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 9.2008
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 9.75327
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 180 error rate: 0.0167591311038
    Epoch 181 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 9.69217
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 9.01305
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 181 error rate: 0.0167591311038
    Epoch 182 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 8.89586
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 9.46162
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 182 error rate: 0.0167591311038
    Epoch 183 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 9.40556
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 8.76349
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 183 error rate: 0.0167591311038
    Epoch 184 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 9.23882
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 8.65625
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 184 error rate: 0.0167591311038
    Epoch 185 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 8.53383
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 9.05034
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 185 error rate: 0.0167591311038
    Epoch 186 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 9.0024
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 8.4056
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 186 error rate: 0.0167591311038
    Epoch 187 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 8.2851
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 8.81004
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 187 error rate: 0.0167591311038
    Epoch 188 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 8.76425
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 8.16582
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 188 error rate: 0.0167591311038
    Epoch 189 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 8.04718
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 8.58029
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 189 error rate: 0.0167591311038
    Epoch 190 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 7.97785
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 8.48184
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 190 error rate: 0.0167591311038
    Epoch 191 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 8.4368
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 7.81743
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 191 error rate: 0.0167591311038
    Epoch 192 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 8.29807
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 7.72376
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 192 error rate: 0.0167591311038
    Epoch 193 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 8.17924
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 7.62856
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 193 error rate: 0.0167591311038
    Epoch 194 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 7.51901
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 8.03248
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 194 error rate: 0.0167591311038
    Epoch 195 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 7.74435
    Minibatch 0 / 1 error rate: 0.0238095
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 7.94914
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 195 error rate: 0.0167591311038
    Epoch 196 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 7.9139
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 7.51295
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 196 error rate: 0.0167591311038
    Epoch 197 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 7.78702
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 7.36959
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 197 error rate: 0.0167591311038
    Epoch 198 ...
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 0 / 0 loss: 7.68333
    Minibatch 0 / 0 error rate: 0.00970874
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 1 / 1 loss: 7.23937
    Minibatch 1 / 1 error rate: 0.0238095
    Epoch 198 error rate: 0.0167591311038
    Epoch 199 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 7.05715
    Minibatch 0 / 1 error rate: 0.0204082
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 7.55549
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 199 error rate: 0.0150584504008
    Epoch 200 ...
    [ 0  1  2  3  4  5  8 12 14 15 16 18 19 22 25 26 27]
    Minibatch 0 / 1 loss: 7.0425
    Minibatch 0 / 1 error rate: 0.0204082
    [ 0  1  2  3  4  5  8  9 10 12 14 15 17 19 21 22 25 26 27]
    Minibatch 1 / 0 loss: 7.47882
    Minibatch 1 / 0 error rate: 0.00970874
    Epoch 200 error rate: 0.0150584504008

```python
logits_out.shape
```

    (423, 4, 28)

```python
import matplotlib.pyplot as plt
```

```python
# Lets visualize the prediction for the first training sequence
np.argmax(logits_out[:,0,:],axis=1)
```

    array([27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 10, 26, 26,  9,
           12, 19, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,  0, 26, 26, 22, 25,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,  5, 12, 12, 12, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27,  0,  8, 26, 26, 17, 26, 26, 27, 27, 27, 14, 12, 27,
           27, 27, 27, 27, 27, 27, 27,  2,  0,  0, 12, 21, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,  4,  0,
            0, 15,  4, 27, 27, 27, 27, 27, 14, 27, 27, 27, 27, 27, 27, 27, 27,
           12, 19, 15, 12,  0, 15, 27, 27, 27, 27, 27, 27, 27, 27, 27,  5,  5,
           22, 22, 15, 19, 25,  4, 27, 27, 27, 27, 27, 27, 27, 27, 27,  0,  0,
           15, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 14,
           12, 12, 26,  1,  0, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,  4, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 14, 14, 26, 26,  0, 12, 27, 27, 27, 27, 27,
           27, 27, 27,  5,  1,  1, 26, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 26,  0,  0,  0, 12, 21, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27,  4, 27, 27, 27, 27,  0,  0,  0,  3,
            4, 27, 22, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 14, 12,  0,  0, 12, 21,  4,  4, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27,  0, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,  8,
           26, 17, 17, 26, 27, 27, 27, 27, 14, 14, 14, 14, 14, 14, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27,
           27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27, 27])

```python
%matplotlib notebook
plt.plot(logits_out[:,0,27],'k-',label='blank')
plt.plot(logits_out[:,0,10],'b-',label='Index 10')
plt.plot(logits_out[:,0,9],'r-',label='Index 9')
plt.plot(logits_out[:,0,12],'g-',label='Index 12')
plt.legend(loc=4)
plt.xlabel('Time Steps')
plt.ylabel('Output Logits')
```

    <IPython.core.display.Javascript object>

    <matplotlib.text.Text at 0x141ed2438>

More about the spykes that tend to apear at the RNN output when using CTC

[Supervised Sequence Labelling](https://www.cs.toronto.edu/~graves/preprint.pdf)

```python

```


# Universidad de Costa Rica

## Facultad de Ciencias

## Escuela de Química

## Sección de Fisicoquímica

## Qu-0560 Tópicos de Química Física: Herramientas Computacionales para Química

### Profesor Mauricio Gutiérrez
### Asistente no oficial: Yefry López

Objetivos

* Usar el módulo scipython y numpy para resolución de problemas básicos
* Emplear el módulo Symbolic Python para resolver diversos problemas matemáticas

### Tópico 1. Resolviendo sistemas de ecuaciones con fsolve de Scipython

Los ejemplos son aportados por AP Monitor Python y Scipython

Para resolver ecuaciones debemos importar el fsolve de scipython y así mismo numpy

```python
import numpy as np
from scipy.optimize import fsolve
import matplotlib.pyplot as plt

```

Ejemplo 1. Encontrar las soluciones o raices de la ecuación  $f(x) = 2x^2 + 3x - 10$ usando fsolve

* Para resolver el problema, debe definir una función en Pytnon que nos de f(x).

```python
def f(x):
    return 2.0*x**2 + 3.0*x -10.0
```

¿ Que tal si creamos un espacio lineal de x, para evaluar en f?

```python
x = np.linspace(-5,3)
print(x)
```

```python
Veamos mas o menos donde se encuentran las soluciones de f, graficando!
```

```python
plt.plot(x,f(x))
```

Sabemos que las soluciones de f se encuentran en x = 0

```python
plt.plot(x,f(x))

#Para agregar los ceros de la función usas el siguiente código
plt.plot(x,np.zeros(len(x)))
```

Con lo anterios gráficamente, mas o menos ya sabemos donde se encuentran las soluciones!

Con fsolve simplemente tenes agregar la función a optimizar en nuestro caso f y un valor inicial donde buscar!

Por ejemplo, a continuación daremos fsolve, con el valor inicial cero

```python
solucion_aprox = fsolve(f,0)
```

```python
print(solucion_aprox)
```

Ahora bien de la gráfica podemos decir que la solución se acerca como a 1.5 talvez....

```python
x1 = fsolve(f,1.5)
```

```python
print(x1)
```

Para la segunda solución, podemos decir que -3 es un buen valor cercano ( ver gráfica, intercepción entre curvas)

```python
x2 = fsolve(f,-3)
print(x2)
```

Ejercicio 1. Encontrar las soluciones o raices de la ecuación  $f(x) = 2x^3 + 3x^2 - 10*x + 2 $ usando fsolve

Tiene tres soluciones

### Tópico 2. Resolviendo sistemas de ecuaciones, derivadas e integrales con Simbolic Python.

Simbolic Python es el relativo a Python a Maple, Matlab o Mathematica, sólo que usando la sintaxis de Python.

En general vamos a importar lo siguiente al inicio

```python
import sympy as sp
sym.init_printing()
```

Con sym.inir_print() te permite obtener resultados similares a Maple en cuanto a manejo de datos.

Hagamos una prueba

Queremos resolver el siguiente sistema de ecuaciones.

$$2*x^{2}+y+z = 1$$
$$ x + 2*y + z = c_{1}$$
$$ -2*x + y = -z$$

#### Paso 1Primero debemos definir los símbolis de interés ( son las variable o parámetros de tu problema)
Por ejemplo sí el simbolo es x

Código

x = sp.simbols('x')

```python
x = sp.symbols('x')

```

```python
x
```

Viste? Ya tenemos x como simbolo, como si se tratase de Maple

Podemos definir los simbolos de forma mas rápida de la siguiente manera

```python
x,y,z = sp.symbols('x,y,z')
c1 = sp.Symbol('c1')
```

```python
x
```

```python
y
```

```python
z
```

```python
c1
```

#### Paso 2. ¿Como ingresar ecuaciones?

Con el código sym.Eq( ecuación, valor al cual está igualado)

Por ejemplo nombremos f,g y h a las tres ecuaciones el ejercicio.

Entonces

```python
f = sp.Eq(2*x**2+y+z,1)
g = sp.Eq(x+2*y+z,c1)
h = sp.Eq(-2*x+y,-z)
```

Preguntemos por f g y h

```python
f
```

```python
g
```

```python
h
```

#### Paso 3. Resolviendo con sym.solve([ecuaciones],(para quienes resolver))

Por ejemplo, nosotros queremos resolver f, g y h para las variables x y y z.

El código es

sp.solve([f,g,h],(x,y,z))

```python
sp.solve([f,g,h],(x,y,z))
```

Listo :3 resolviste el sistema de ecuacion

#### Ejercicio 1.

Convierta a Simbolic Python la expresión , guarde la expresión en h

$$x^2+2x-5.$$

```python
h = x**2 + 2*x - 5
h
```

#### Ejercicio 2.1
Evaluar en x = 1.5 con el método .subs(variable, lo que quiero sustituir)

.subs() viene de ( Sustitución)

```python
h.subs(x,1.5)
```

#### Ejercicio 2.2 Haga un cambio de variable de x a z con subs()

```python
h.subs(x,z)
```

#### Ejercicio 2.3. Haga un cambio de variable de x a $y^{2}$ con .subs()

#### Ejercicio 3.1

Simplifique la siguiente expresión
$$\frac{x^2 - x - 6}{x^2-3x}.$$

Con el código __sp.simplify(nombre de expresión)__

```python
f = (x**2-x-6)/(x**-3*x)
```

```python
sp.simplify(f)
```

#### Ejercicio 3.2

Expandir simbólicamente la expresión

 $$(x+1)^3(x-2)^2.$$

 Con el código __sp.expand(nombre de expresión)__

```python

```

#### Ejercicio 3.3

Factorice la siguien expresión

$$3x^4 - 36x^3+99x^2-6x-144.$$

Con el código __sp.factor(nombre de la expresión)__

### Derivadas en integrales

sp puede resolver multitud de derivadas e integrales.

Por deficiòn tiene guardados diveras expresiones por ejemplo

* sp.sin(x) seno de x
* sp.exp(x)
* sp.cos(x)

Y así sucesivamente! Es como cuando aprendiste a usar math.exp() o np.exp(), sólo que ahora lo hacemos con sp de Symbolic Python

Hagamos una prueba, escribamos una expresión para $$cos(x)*e^{5x}$$ y la guardamos en y

```python
y = sp.cos(x) * sp.exp(2*x)
y
```

Para calcular la derivadas usas el código
__sp.diff(expresión, variable)__

```python
sp.diff(y,x)
```

#### Ejercicio 4.1

Calcule la derivada $$\frac{d}{dx}\sin^2(x)e^{2x}.$$
Luego evalue para $x=3.3.$

```python
# Primero guarde la expresión a derivar en h

# Segundo aplique la derivadad y guardela en z

# Haga sp.subs(z,3.3)
```

#### Ejercicio 4.2

Represente de forma símbólica la siguiente integral y resuelva.

$$\int_0^5x^2\sin(x^2)dx.$$

Para resolver integrales el código es el siguiente

__sp.integrate(expresión a integrar , (variable de integración , límite inicial de integración, límite final de integración))__

```python
f = x**2*sp.sin(x**2)
```

```python
g = sp.integrate(f,(x,0,5))
g
```

Se ve espantasos :/

Pero esta es la solución analítica de dicha integral, esa simbolos que parecen L invertida se llaman función Gamma de Euler.

Sí, deseamos una aproximación numérica, podemos usar __g.evalf()__

```python
g.evalf()
```

Mucho mejor!

#### Ejercicio 4.3

Represente de forma símbólica la siguiente integral y resuelva

$$\int_0^2xe^{x^{2}}dx.$$

#### Ejercicio 5.1

Encuentre las 3 a soluciones de la expresión $$x^3+15x^2=3x-10.$$

La expresión debe igualarse a cero

* Primero guarde la expresión  use la función sp.Eq(acá va la expresión, a lo que es igual) y guarde en  y
* Por ejemplo h = sp.Eq(x**3 + 15*x**2,3*x-10)
* Segundo realice sp.solve(y,x) y guardela en z
* Pregunte por z ( como son tres raices, z es una lista. Podes preguntar po los elentos de la lista, por ejemplo z[0] la primera raíz
* Sí el resultado es horrible, puede usar z[0].evalf(),z[1].evalf()

```python

```

```python
z[0].evalf()
```

Yup, las soluciones se encuentran dentro de los números imaginarioa

#### Ejercicio 5.2
Resuelva el siguiente sistema de ecuaciones

\begin{align}
x+y+z&=0 \\
2x-y-z&=10 \\
y+2z&=5
\end{align}

HINT.

Defina tres ecuaciones, eq1, eq2, eq3

Luego use sp.solve(eq1,eq2,eq3, [x,y,z])

#### Ejercicio 6.1 Resolviendo ecuaciones diferenciales

Resuelva la siguiente ecuación diferenciales usando la función ```dsolve``` :
$$\frac{df(x)}{dx} = x\cos(x).$$

Para este caso, primero es necesario, decirle a Symbolic Python hay que resolver una función

```python
f = sp.Function('f')
```

La expresión es la siguiente

$$\frac{df(x)}{dx} - x\cos(x) = 0.$$

Guardemos la expresión en h.

Para hacer la derivada de f respecto a x , el código es sp.Derivative(f(x),x)

```python
h = sp.Derivative(f(x),x) - x*sp.cos(x)
```

Usamos sp.disolve(ecuación diferencial , para quién queremos resolver)

En nuestro caso, queremos resolver para f(x)

```python
y = sp.dsolve(h,f(x))
```

```python
y
```

Y esa es la solución1

sp tiene muchas mas opciones!

Como Algebra de matrices y otros tópicos de algebra lineal.

Podes sacarle mucho mas el provecho!

[Volver al índice](https://notebooks.azure.com/jeff0994/libraries/QU-0560-1-1/html/01Introducci%C3%B3n-Indice.ipynb)

```python

```


# Auto Encoder

## 符号定义

|符号|含义|
|:--:|:--:|
|$\pmb{x}$|输入数据|
|$\pmb{y}$|输出数据|
|$\pmb{h}$|低维表示向量|
|$f_{w, b}$|映射函数|
|$g_{w, b}$|映射函数|
|$\mathcal{L}$|损失函数|
|$m$|特征空间维度|

## 概念

Auto Encoder通常是一个多层前馈网络。为了实现可靠的监督学习，在参数学习阶段需要编码器和解码器两部分。其中编码器将输入数据映射到低维特征空间，解码器又将低维特征映射回原空间。通过对输入输出进行监督实现参数学习。这个学习过程的损失可由下式表示

$$
\begin{equation}
    \begin{split}
    \mathcal{L}
    &= \sum_{i=1}^N||\pmb{x_i}-\pmb{y_i}||^2 \\
    &= \sum_{i=1}^N||\pmb{x_i}-g_{w, b}(f_{w, b}(\pmb{x_i}))||^2
    \end{split}
\end{equation}
$$

其中，$g_{w, b}$为解码器拟合得到的函数，$f_{w, b}$为编码器拟合得到的函数。

上式中的$f_{w, b}$即为Auto Encoder，它能实现将输入数据嵌入映射到低维特征空间。

$$
\begin{equation}
    \pmb{h_i} = f_{w, b}(\pmb{x_i}), \pmb{h_i} \in \mathcal{R}^m
\end{equation}
$$

上式并没指出输入数据所在的空间，根据输入数据类型不同完全可以定义不同的编码器形式。

## 算法步骤

1. 根据输入数据设计相应的编码器结构和解码器结构
2. 利用输入输出监督训练编码器和解码器
3. 得到编码器用于降维

```python
from collections import OrderedDict
import numpy as np
import random

import torch
from torch import nn
from torchvision import transforms, datasets
from torch.utils.data import DataLoader

import matplotlib.pyplot as plt

```

```python
# ------------------------ model ------------------------

class MyAE(nn.Module):

    def __init__(self):

        super(MyAE, self).__init__()

        self.encoder = nn.Sequential(OrderedDict({
            "linear1": nn.Linear(28*28, 128),
            "BN1": nn.BatchNorm1d(128),
            "ReLU1": nn.ReLU(inplace=True),
            "linear2": nn.Linear(128, 64),
            "BN2": nn.BatchNorm1d(64),
            "ReLU2": nn.ReLU(inplace=True),
            "linear3": nn.Linear(64, 32),
            "BN3": nn.BatchNorm1d(32),
            "ReLU3": nn.ReLU(inplace=True),
            "linear4": nn.Linear(32, 8),
            "BN4": nn.BatchNorm1d(8),
            "ReLU4": nn.ReLU(inplace=True),
            "linear5": nn.Linear(8, 2),
        }))

        self.decoder = nn.Sequential(OrderedDict({
            "linear1": nn.Linear(2, 8),
            "BN1": nn.BatchNorm1d(8),
            "ReLU1": nn.ReLU(inplace=True),
            "linear2": nn.Linear(8, 32),
            "BN2": nn.BatchNorm1d(32),
            "ReLU2": nn.ReLU(inplace=True),
            "linear3": nn.Linear(32, 64),
            "BN3": nn.BatchNorm1d(64),
            "ReLU3": nn.ReLU(inplace=True),
            "linear4": nn.Linear(64, 128),
            "BN4": nn.BatchNorm1d(128),
            "ReLU4": nn.ReLU(inplace=True),
            "linear5": nn.Linear(128, 28*28),
            "sigmoid": nn.Sigmoid()
        }))

    def forward(self, x):
        feature_vector = self.encoder(x)
        output_ = self.decoder(feature_vector)
        return output_

    def encoder_func(self, x):
        return self.encoder(x)

    def decoder_func(self, x):
        return self.decoder(x)
```

```python
# --------------------- global ------------------
np.random.seed(1024)
torch.manual_seed(1024)
random.seed(1024)

if torch.cuda.is_available():
    torch.cuda.manual_seed(1024)
    device = torch.device("cuda")
else:
    device = torch.device("cpu")
print("device: ", device)

batch_size = 128
basic_lr = 5e-3
epochs = 1

# ------------------------ data ------------------------
transform_ = transforms.Compose([transforms.ToTensor()])
train_data = datasets.MNIST('./mnist', train=True, transform=transform_, download=True)
test_data = datasets.MNIST('./mnist', train=False, transform=transform_)

train_data_loader = DataLoader(dataset=train_data, batch_size=batch_size, shuffle=True)
test_data_loader = DataLoader(dataset=test_data, batch_size=batch_size, shuffle=False)

# ------------------------ model -------------------------
model = MyAE().to(device)

# ----------------------- optimizer and loss -----------------------
loss_func = nn.MSELoss()
optimizer = torch.optim.Adam(model.parameters(), lr=basic_lr)

# ------------------------ train ----------------------
for epoch in range(1, epochs+1):
    for i, (images, labels) in enumerate(train_data_loader):
        images = images.reshape(-1, 28*28).to(device)

        output_ = model(images)

        train_loss = loss_func(images, output_)
        optimizer.zero_grad()
        train_loss.backward()
        optimizer.step()

        if i % 200 == 0:
            print("[{}/{}]\t[{}/{}]\ttrain loss:{:.4f}".format(epoch, epochs, i, len(train_data_loader), train_loss.item()))
    print("[{}/{}]\t[{}/{}]\ttrain loss:{:.4f}".format(epoch, epochs, len(train_data_loader), len(train_data_loader), train_loss.item()))
```

    device:  cuda
    [1/1]	[0/469]	train loss:0.2405
    [1/1]	[200/469]	train loss:0.0429
    [1/1]	[400/469]	train loss:0.0442
    [1/1]	[469/469]	train loss:0.0470

```python
# ------------------------------ draw --------------------------

draw_train_data_list = np.zeros((len(train_data), 2))
draw_train_label_list = np.zeros((len(train_data)))

draw_test_data_list = np.zeros((len(train_data), 2))
draw_test_label_list = np.zeros((len(train_data)))

model.eval()
data_count = 0
with torch.no_grad():
    for i, (images, labels) in enumerate(train_data_loader):
        images = images.reshape(-1, 28*28).to(device)

        feature_vector = model.encoder_func(images).detach().cpu().numpy()

        draw_train_data_list[data_count:data_count+images.shape[0], :] = feature_vector.copy()
        draw_train_label_list[data_count:data_count+images.shape[0]] = labels.reshape(-1)

        data_count += images.shape[0]

data_count = 0
with torch.no_grad():
    for i, (images, labels) in enumerate(test_data_loader):
        images = images.reshape(-1, 28*28).to(device)

        feature_vector = model.encoder_func(images).detach().cpu().numpy()

        draw_test_data_list[data_count:data_count+images.shape[0], :] = feature_vector.copy()
        draw_test_label_list[data_count:data_count+images.shape[0]] = labels.reshape(-1)

        data_count += images.shape[0]

# draw func
fig = plt.figure(figsize=(20, 10))

ax = fig.add_subplot(121)
plt.title("Projection of training data using AutoEncoder", fontsize=15)
ax.scatter(draw_train_data_list[:, 0], draw_train_data_list[:, 1], c=draw_train_label_list, cmap=plt.cm.Spectral)

ax = fig.add_subplot(122)
plt.title("Projection of testing data using AutoEncoder", fontsize=15)
ax.scatter(draw_test_data_list[:, 0], draw_test_data_list[:, 1], c=draw_test_label_list, cmap=plt.cm.Spectral)

plt.show()

```


# A simple analysis of inflation and bnp in a closed economy

Imports and set magics:

```python
import numpy as np
from scipy import optimize
import sympy as sm
import ipywidgets as wg
import matplotlib.pyplot as plt
from IPython.display import display

# autoreload modules when code is run
%load_ext autoreload
%autoreload 2

# local modules
import modelproject as mp

```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

# Model description

We consider  the following model of inflation and GDP.

<!---
it looks like a mess because I don't know how to indent, " " is equal to one space.
-->

|||
|:---|---|
|      AD.   |      $y_t = \bar{y} - \alpha (\pi_t - \pi^*)$   |
|      AS.   |      $\pi_t = \pi_{t-1} + \gamma (y_t - \bar{y})$   |

Using $\hat{y}_t = y_t - \bar{y}$ and $\hat{\pi}_t = \pi_t - \pi^*$ we get:

|||
|:---|---|
|      AD.   |      $\hat{y}_t = - \alpha \hat{\pi}_t$   |
|      AS.   |      $\pi_t = \pi_{t-1} + \gamma (y_t - \bar{y})$ $\rightarrow$ $\pi_t = \hat{\pi}_{t-1} + \pi^* + \gamma \hat{y_t}$ $\rightarrow$  |
|      AS. |      $\hat{\pi}_t = \hat{\pi}_{t-1} + \gamma \hat{y_t}$ |

$\pi_{t}$ is inflation at period t, $y_t$ is GDP at period t, and $\gamma$ and $\alpha$ are parameters larger than 0.

$\pi^*$ is the central banks inflation goal

$\bar{y}$ and $\bar{\pi}$ are the natural levels of output and inflation, meaning the levels if every resource in the economy is used effeciently.

$\hat{y}_t$ and $\hat{\pi}_t$ are the differences between the actual value and the efficient value of GDP and the inflation goal at period t.

# Steady state

```python
#using sympy to find steady state

# Defining the symbols
yt, ybar, pit, pistar, pitm1, yhat_t, yhat_t1, alpha, pihat_t, pihat_t1, gamma = sm.symbols('y_t ybar pi_t pi^* pi_t-1 yhat_t yhat_t-1 alpha pihat_t pihat_t-1 gamma')
```

```python
# Defining our equations
AD_yt = sm.Eq(yhat_t, - alpha*pihat_t)
AS_pit = sm.Eq(pihat_t, pihat_t1+gamma*yhat_t)
display("AD = ", AD_yt, "AS = ", AS_pit)

# the steady state for pi is found by inserting AD into AS
ss1 = sm.Eq(pihat_t, pihat_t1-gamma*alpha*pihat_t)
#solving for pihat_t
sspi1 = sm.solve(ss1, pihat_t) # solving for pihat_t
sspi = sm.Eq(pihat_t,(pihat_t1/(alpha*gamma + 1))) # converting result to equation
display('steady state for pi is: ', sspi) # displaying result

# the steady state for y is found by inserting AS into AD
ss2 = sm.Eq(yhat_t, - alpha*(pihat_t1+gamma*yhat_t))
ssy1 = sm.solve(ss2, yhat_t) # solving for yhat_t
#by inserting AD in time t-1 we get yhat_t as a function of y_t-1
ss3 = sm.Eq(yhat_t,(alpha/(alpha*gamma + 1)*(yhat_t1/alpha)))
ssy2 = sm.solve(ss3, yhat_t) # solve again for yhat_t
ssy3 = sm.Eq(yhat_t,(yhat_t1/(alpha*gamma + 1))) # converting result to equation
display('steady state for y is: ', ssy3) # displaying result
```

    'AD = '

$\displaystyle \hat{y}_{t} = - \alpha \hat{\pi}_{t}$

    'AS = '

$\displaystyle \hat{\pi}_{t} = \gamma \hat{y}_{t} + \hat{\pi}_{t-1}$

    'steady state for pi is: '

$\displaystyle \hat{\pi}_{t} = \frac{\hat{\pi}_{t-1}}{\alpha \gamma + 1}$

    'steady state for y is: '

$\displaystyle \hat{y}_{t} = \frac{\hat{y}_{t-1}}{\alpha \gamma + 1}$

By solving the model we find that steady state for both $\hat{y}_t$ and $\hat{\pi}_t$ is equal to the the previouse periods value times $\frac{1}{\alpha \gamma + 1}$

# Graphing the equations

```python
# Defining AD and AS as graphs
def AD_graph(a):
    yt = np.linspace(0,100)
    ybar = 10
    pistar = 0.02
    pit = (a*pistar - yt + ybar)/a

    # Making the graph
    plt.plot(yt, pit ,'r-')
    plt.ylabel('pi')
    plt.xlabel('yt')
    plt.ylim([0,0.1])
    plt.xlim([9.8,10.2])

def AS_graph(g):
    yt = np.linspace(0,100)
    ybar = 10
    pitm1 = 0.01
    pit = g*(yt-ybar)+pitm1
    # Making the graph

    plt.plot(yt, pit, 'b-')
    plt.ylabel('pi')
    plt.xlabel('yt')
    plt.ylim([0,0.1])
    plt.xlim([9.8,10.2])

```

```python
# Makeing a slider for alpha and gamma
a_slider = wg.FloatSlider(value=0.5, min=0, max=1, step = 0.01)
g_slider = wg.FloatSlider(value=0.5, min=0, max=1, step = 0.01)

AD = wg.interact(AD_graph, a = a_slider)
AS = wg.interact(AS_graph, g = g_slider)
```

    interactive(children=(FloatSlider(value=0.5, description='a', max=1.0, step=0.01), Output()), _dom_classes=('w…

    interactive(children=(FloatSlider(value=0.5, description='g', max=1.0, step=0.01), Output()), _dom_classes=('w…

```python
# Graphic solution for alpha = gamma = 1/3
AD_graph(0.33)
AS_graph(0.33)
```

## Unforseen changes to inflation

Because of unforseen consequenses the inflation in period t-1 is now 1 pct., where as the goal for the inflation is 2 pct.
By using a solver we can see that the optimal level for the inflation in our model is 2 pct. meaning the inflation will settle at 2 pct. in the steady state.

```python
#Defining values used in this exampla.
pi_t1 = 0.01 # the inflation is 1% instead of the 2% inflation goal.
gamma = 0.33
ybar = 10
pistar = 0.02
alpha = 0.33

def AS(y_t):
    return pi_t1 + gamma*(y_t - ybar)
def obj(y_t):
    return alpha * (AS(y_t) - pistar)
```

```python
#using solver from modelproject to find the optimal pi
optimum = mp.solve_for_ss(pi_t1, gamma, ybar, pistar, alpha)
```

## Conclusion

```python
yoptimum = optimum.x[0]
print(f'the optimal y is equal to {yoptimum:.2f} which is also equal to the most effecient output')
pioptimum = AS(optimum.x[0])
print(f'the optimal pi is equal to {pioptimum:.2f} which is also equal to the inflation goal')
```

    the optimal y is equal to 10.03 which is also equal to the most effecient output
    the optimal pi is equal to 0.02 which is also equal to the inflation goal