# Intro to Pattern Recognition
# Problem Set 4: Neural Networks
### M S Mohamed Fazil
### UB Id : mm549

## Introduction
We will be using neural networks to build a model that classifies hand written digits using thr MNIST database. The MNIST database (Modified National Institute of Standards and Technology Database) is a large collection of handwritten digits which happens to be a state of the art data set to train and test machine learning models using image processing.It consist of 60000 Training samples and 10000 Testing Samples. Each sample image consist of normalized $28X28$ pixels grayscale image stored as 784 dimensional vector for each sample.This is a way of dimension reduction done by the MNIST to use it for model developments.

### Libraries Used
For loading the given MNIST datasets we will be using the mnist library to fetch the data.The MNIST() function will be used to fetch the datasets from the directory './data'. For the purpose of building neural networks and trainning we will be using Keras library which is backed by the Tensorflow. We will use OpenCv for the image processing applications.

```python
from keras.models import Sequential
from keras.layers import Dense, Activation, Conv2D, MaxPooling2D, Dropout, Flatten
from keras.callbacks import LambdaCallback,ModelCheckpoint
from keras.utils import to_categorical
from keras import optimizers,regularizers
import numpy as np
import cv2
import os
from mnist.loader import MNIST
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
import random as rd
os.environ["CUDA_VISIBLE_DEVICES"]="1"
%matplotlib inline

m = MNIST('./data')
```

    Using TensorFlow backend.

## 1. Neural Network to Maximize the log Likehood

### Minimization of Negative Log Likehood with Softmax Output Layer
Softmax is an activation function of a node which is commonly placed as the output layer of a Neural Network. It is mostly used in multiclass classification problems. The Function is defined as
\begin{equation}
S(V_i) = \frac{e^{V_i}}{\sum_je^{V_i}}\
\end{equation}
The softmax function squashes the input vector values between 0 to 1 and because of the normalization of exponential , the sum of the values in the vector tends to be 1. The output values of a Softmax function can be interpretted as a probabilities of the different labels present in the multiclass.In our case the Softmax outputs a vector of 10 values corresponding to the class labels.
Negative Log-Likehood is a loss function (aka criterion function) which is used with the Softmax activation function to minimize the criterion function. It is given as
\begin{equation}
J_0 = -log\,p(D;w) = -log\,p(\{(x_n,t_n):n=1,2,4,...\};w)
\end{equation}
\begin{equation}
J_0 =-log\prod_n\prod_{m=0}^9p(t_n=m|x_n;w)
\end{equation}
To minimize the criterion function we must differentiate the softmax function with respect to the Negative Log Likehood function.
### Neural Network to Maximize the Posterior Likehood by Minimizing Criterion Function with L2 Regularization
Let us assume that a Gaussion Prior of the weight distribution is given with the $\alpha$.
\begin{equation}
p(w;\alpha)=N(N,\alpha l)
\end{equation}
Regularization is used to avoid overfitting of the traning datasets into the model.Therefore depending on the gradient of the weights between the epochs a part of the weights is subtracted by the regularization component determined with the regularization parameter $\alpha$. The Gaussian Prior function can be used in par with the L2 regularization during the minimization of the Criterion function which will maximize the Posterior Likehood. The combined loss function can be given as
\begin{equation}
J(w) = J_0(w)-log\,p(w;\alpha^{-1})
\end{equation}

## Preparing the Datasets
We prepare a dataset which will consist of 1000 training samples with labels where 100 images for each digit and another 1000 testing samples with label where 100 images for each digit. The target consist of a class with ten classification labels.

```python
classes = [0,1,2,3,4,5,6,7,8,9]
xtrain,ytrain = m.load_training()
xtest,ytest = m.load_testing()

# Divide the given datasets to 1000 training dataset and 1000 testing dataset
xtrain = xtrain[0:1000]
ytrain = ytrain[0:1000]
xtest = xtest[0:1000]
ytest = ytest[0:1000]
xtrain = np.asarray(xtrain).astype(np.float32)
ytrain = np.asarray(ytrain).astype(np.float32)
xtest = np.asarray(xtest).astype(np.float32)
ytest = np.asarray(ytest).astype(np.float32)

n_classes = len(classes)
#0-1 Hot encoding
label_train = np.zeros((ytrain.shape[0], n_classes))
a = np.arange(ytrain.shape[0], dtype=np.int64)
b = np.array(ytrain, dtype=np.int64).reshape((ytrain.shape[0],))
label_train[a, b] = 1
label_test = np.zeros((ytest.shape[0], n_classes))
c = np.arange(ytest.shape[0], dtype=np.int64)
d = np.array(ytest, dtype=np.int64).reshape((ytest.shape[0],))
label_test[c, d] = 1
```

### Function to Plot the Results of a Training Model
This fuction plots the Traing and Testing's loss,criterion function, accuracy and the learning speed of each hidden layer with respect to each epoch during training phase. The learning Speed is the measure of the speed of change of weights after every epoch.

```python
def plot_model_result(history,weights,final_w):
    plt.plot(history.history['loss'], label='Training data')
    plt.plot(history.history['val_loss'], label='Testing data')
    plt.legend(['train', 'test'], loc='upper left')
    plt.title('Loss of Training and Testing Data (Criterion Function)')
    plt.ylabel('MAE value')
    plt.xlabel('No. epoch')
    plt.show()
    plt.plot(history.history['accuracy'], label='Training data')
    plt.plot(history.history['val_accuracy'], label='Testing data')
    plt.legend(['train', 'test'], loc='upper left')
    plt.title('Accuracy of Training and Testing Data')
    plt.ylabel('Accuracy value')
    plt.xlabel('No. epoch')
    plt.show()
    cnt = 0
    '''
    Plot the Learning Speed of the Model
    '''
    np.seterr(divide='ignore', invalid='ignore')
    for i in range(0,len(weights)):
        w = weights[i]
        W = final_w[i]
        learning_speed = [0]
        for i in range(1,len(w)):
            wold = w[i-1]
            wnew = w[i]
            diff = wold-wnew
            d = wnew
            m = np.median(d[d>0])
            d[d==0] = m
            diff = abs(diff)/d
            if diff.size:
                avg = abs(np.mean(diff))
            else:
                avg = 0
            learning_speed.append(avg)
        plt.plot(learning_speed, label='Learning Speed')
        cnt = cnt+1
        plt.title('Learning Speed of Hidden Layer {}'.format(cnt))
        plt.ylabel('Learning Speed')
        plt.xlabel('No. epoch')
        plt.show()

    print('Training Accuracy -',history.history['accuracy'][29])
    print('Training Loss -',history.history['loss'][29])
    print('Testing Accuracy -',history.history['val_accuracy'][29])
    print('Testing Loss -',history.history['val_loss'][29])
```

## 2(a). Neural Network with 1 Hidden Layer without regularization

```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1]))
model.add(Dense(10, activation='softmax'))
weights = []
save_weights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[0].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights])
final_w = [model.layers[0].get_weights()[0]]
weights = [weights]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 505us/step - loss: 0.3249 - accuracy: 0.8999 - val_loss: 0.3080 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 323us/step - loss: 0.2965 - accuracy: 0.9000 - val_loss: 0.2910 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.2822 - accuracy: 0.9000 - val_loss: 0.2849 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.2640 - accuracy: 0.9015 - val_loss: 0.2661 - val_accuracy: 0.9001
    Epoch 5/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.2475 - accuracy: 0.9041 - val_loss: 0.2513 - val_accuracy: 0.9071
    Epoch 6/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.2381 - accuracy: 0.9070 - val_loss: 0.2443 - val_accuracy: 0.9095
    Epoch 7/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.2319 - accuracy: 0.9082 - val_loss: 0.2372 - val_accuracy: 0.9092
    Epoch 8/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.2224 - accuracy: 0.9087 - val_loss: 0.2269 - val_accuracy: 0.9104
    Epoch 9/30
    1000/1000 [==============================] - 0s 313us/step - loss: 0.2101 - accuracy: 0.9119 - val_loss: 0.2213 - val_accuracy: 0.9106
    Epoch 10/30
    1000/1000 [==============================] - 0s 312us/step - loss: 0.2048 - accuracy: 0.9126 - val_loss: 0.2174 - val_accuracy: 0.9133
    Epoch 11/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.2020 - accuracy: 0.9161 - val_loss: 0.2135 - val_accuracy: 0.9161
    Epoch 12/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.1939 - accuracy: 0.9190 - val_loss: 0.2101 - val_accuracy: 0.9159
    Epoch 13/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.1959 - accuracy: 0.9158 - val_loss: 0.2097 - val_accuracy: 0.9150
    Epoch 14/30
    1000/1000 [==============================] - 0s 311us/step - loss: 0.1902 - accuracy: 0.9178 - val_loss: 0.1968 - val_accuracy: 0.9169
    Epoch 15/30
    1000/1000 [==============================] - 0s 310us/step - loss: 0.1833 - accuracy: 0.9249 - val_loss: 0.2056 - val_accuracy: 0.9175
    Epoch 16/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.1747 - accuracy: 0.9285 - val_loss: 0.2009 - val_accuracy: 0.9232
    Epoch 17/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.1899 - accuracy: 0.9228 - val_loss: 0.2166 - val_accuracy: 0.9153
    Epoch 18/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.1925 - accuracy: 0.9206 - val_loss: 0.2165 - val_accuracy: 0.9113
    Epoch 19/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.1996 - accuracy: 0.9146 - val_loss: 0.2226 - val_accuracy: 0.9101
    Epoch 20/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.1864 - accuracy: 0.9238 - val_loss: 0.2153 - val_accuracy: 0.9166
    Epoch 21/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.1847 - accuracy: 0.9255 - val_loss: 0.1938 - val_accuracy: 0.9202
    Epoch 22/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.1612 - accuracy: 0.9370 - val_loss: 0.1815 - val_accuracy: 0.9279
    Epoch 23/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.1527 - accuracy: 0.9420 - val_loss: 0.1809 - val_accuracy: 0.9278
    Epoch 24/30
    1000/1000 [==============================] - 0s 310us/step - loss: 0.1602 - accuracy: 0.9381 - val_loss: 0.1783 - val_accuracy: 0.9304
    Epoch 25/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.1503 - accuracy: 0.9426 - val_loss: 0.1863 - val_accuracy: 0.9250
    Epoch 26/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.1487 - accuracy: 0.9450 - val_loss: 0.1781 - val_accuracy: 0.9313
    Epoch 27/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.1516 - accuracy: 0.9402 - val_loss: 0.1790 - val_accuracy: 0.9301
    Epoch 28/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.1494 - accuracy: 0.9429 - val_loss: 0.1693 - val_accuracy: 0.9344
    Epoch 29/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.1460 - accuracy: 0.9455 - val_loss: 0.1616 - val_accuracy: 0.9386
    Epoch 30/30
    1000/1000 [==============================] - 0s 309us/step - loss: 0.1388 - accuracy: 0.9495 - val_loss: 0.1621 - val_accuracy: 0.9367

```python
plot_model_result(history,weights,final_w)
```

## 2(b). Neural Network with 2 Hidden Layer and 3 Hidden Layer

## 2 Hidden Layer without Regularization

```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1]))
model.add(Dense(30, activation='sigmoid'))
model.add(Dense(10, activation='softmax'))
weights1 = []
weights2 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0]]
weights = [weights1,weights2]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 523us/step - loss: 0.3258 - accuracy: 0.9000 - val_loss: 0.3214 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 327us/step - loss: 0.3157 - accuracy: 0.9000 - val_loss: 0.3153 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.3090 - accuracy: 0.9000 - val_loss: 0.3101 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.3043 - accuracy: 0.9000 - val_loss: 0.3056 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.2982 - accuracy: 0.9000 - val_loss: 0.3004 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.2922 - accuracy: 0.9000 - val_loss: 0.2953 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.2861 - accuracy: 0.9000 - val_loss: 0.2892 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.2786 - accuracy: 0.9000 - val_loss: 0.2817 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.2722 - accuracy: 0.9000 - val_loss: 0.2769 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.2658 - accuracy: 0.9000 - val_loss: 0.2700 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.2587 - accuracy: 0.9000 - val_loss: 0.2649 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.2528 - accuracy: 0.9000 - val_loss: 0.2602 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.2452 - accuracy: 0.9000 - val_loss: 0.2529 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.2384 - accuracy: 0.9000 - val_loss: 0.2474 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.2318 - accuracy: 0.9025 - val_loss: 0.2447 - val_accuracy: 0.9006
    Epoch 16/30
    1000/1000 [==============================] - 0s 337us/step - loss: 0.2252 - accuracy: 0.9059 - val_loss: 0.2338 - val_accuracy: 0.9059
    Epoch 17/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.2179 - accuracy: 0.9087 - val_loss: 0.2302 - val_accuracy: 0.9081
    Epoch 18/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.2111 - accuracy: 0.9138 - val_loss: 0.2264 - val_accuracy: 0.9102
    Epoch 19/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.2074 - accuracy: 0.9155 - val_loss: 0.2185 - val_accuracy: 0.9145
    Epoch 20/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.1993 - accuracy: 0.9179 - val_loss: 0.2126 - val_accuracy: 0.9175
    Epoch 21/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.1926 - accuracy: 0.9230 - val_loss: 0.2121 - val_accuracy: 0.9186
    Epoch 22/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.1946 - accuracy: 0.9208 - val_loss: 0.2116 - val_accuracy: 0.9168
    Epoch 23/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.1900 - accuracy: 0.9225 - val_loss: 0.2027 - val_accuracy: 0.9211
    Epoch 24/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.1810 - accuracy: 0.9269 - val_loss: 0.1967 - val_accuracy: 0.9244
    Epoch 25/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.1746 - accuracy: 0.9295 - val_loss: 0.1957 - val_accuracy: 0.9243
    Epoch 26/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.1716 - accuracy: 0.9282 - val_loss: 0.1906 - val_accuracy: 0.9255
    Epoch 27/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.1710 - accuracy: 0.9305 - val_loss: 0.1938 - val_accuracy: 0.9264
    Epoch 28/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.1661 - accuracy: 0.9332 - val_loss: 0.1813 - val_accuracy: 0.9291
    Epoch 29/30
    1000/1000 [==============================] - 0s 323us/step - loss: 0.1622 - accuracy: 0.9361 - val_loss: 0.1795 - val_accuracy: 0.9295
    Epoch 30/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.1555 - accuracy: 0.9374 - val_loss: 0.1821 - val_accuracy: 0.9289

```python
plot_model_result(history,weights,final_w)
```

## 3 Hidden Layer Without Regularization

```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1]))
model.add(Dense(30, activation='sigmoid'))
model.add(Dense(30, activation='sigmoid'))
model.add(Dense(10, activation='softmax'))
weights1 = []
weights2 = []
weights3 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
save_weights3 = LambdaCallback(on_epoch_end=lambda batch, logs: weights3.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2,save_weights3])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0],model.layers[2].get_weights()[0]]
weights = [weights1,weights2,weights3]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 561us/step - loss: 0.3428 - accuracy: 0.9000 - val_loss: 0.3306 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 339us/step - loss: 0.3274 - accuracy: 0.9000 - val_loss: 0.3247 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3242 - accuracy: 0.9000 - val_loss: 0.3234 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 341us/step - loss: 0.3231 - accuracy: 0.9000 - val_loss: 0.3231 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 346us/step - loss: 0.3226 - accuracy: 0.9000 - val_loss: 0.3227 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 340us/step - loss: 0.3220 - accuracy: 0.9000 - val_loss: 0.3221 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3215 - accuracy: 0.9000 - val_loss: 0.3217 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3209 - accuracy: 0.9000 - val_loss: 0.3212 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3203 - accuracy: 0.9000 - val_loss: 0.3208 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3198 - accuracy: 0.9000 - val_loss: 0.3204 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3193 - accuracy: 0.9000 - val_loss: 0.3199 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3187 - accuracy: 0.9000 - val_loss: 0.3195 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3182 - accuracy: 0.9000 - val_loss: 0.3189 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3176 - accuracy: 0.9000 - val_loss: 0.3184 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 337us/step - loss: 0.3169 - accuracy: 0.9000 - val_loss: 0.3178 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3162 - accuracy: 0.9000 - val_loss: 0.3170 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3153 - accuracy: 0.9000 - val_loss: 0.3165 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3145 - accuracy: 0.9000 - val_loss: 0.3156 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3136 - accuracy: 0.9000 - val_loss: 0.3148 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3125 - accuracy: 0.9000 - val_loss: 0.3138 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3115 - accuracy: 0.9000 - val_loss: 0.3129 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3104 - accuracy: 0.9000 - val_loss: 0.3120 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3093 - accuracy: 0.9000 - val_loss: 0.3106 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3078 - accuracy: 0.9000 - val_loss: 0.3095 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3064 - accuracy: 0.9000 - val_loss: 0.3082 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3048 - accuracy: 0.9000 - val_loss: 0.3068 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3033 - accuracy: 0.9000 - val_loss: 0.3049 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.3015 - accuracy: 0.9000 - val_loss: 0.3031 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.2994 - accuracy: 0.9000 - val_loss: 0.3012 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.2973 - accuracy: 0.9000 - val_loss: 0.2993 - val_accuracy: 0.9000

```python
plot_model_result(history,weights,final_w)
```

## 1 Hidden layer with Regularization - l2(5)

```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1],
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(10, activation='softmax',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
weights = []
save_weights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[0].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights])
final_w = [model.layers[0].get_weights()[0]]
weights = [weights]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 567us/step - loss: 3.9812 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 323us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 318us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 322us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 314us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 316us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 319us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 312us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 315us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 320us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 321us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 317us/step - loss: 0.3257 - accuracy: 0.9000 - val_loss: 0.3253 - val_accuracy: 0.9000

```python
plot_model_result(history,weights,final_w)
```

## 2 Hidden Layer with Regularization - l2(5)

```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1],
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(30, activation='sigmoid',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(10, activation='softmax',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
weights1 = []
weights2 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0]]
weights = [weights1,weights2]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 630us/step - loss: 5.4465 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 326us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 332us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 324us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 328us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 334us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 325us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 329us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 331us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 335us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 336us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 330us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 333us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000

```python
plot_model_result(history,weights,final_w)
```

## 3 Hidden Layer with Regularization - l2(5)

```python
model = Sequential()
model.add(Dense(30, activation='sigmoid', input_dim=xtrain.shape[1],
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(30, activation='sigmoid',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(30, activation='sigmoid',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
model.add(Dense(10, activation='softmax',
                kernel_regularizer=regularizers.l2(5),
                bias_regularizer=regularizers.l2(5)))
weights1 = []
weights2 = []
weights3 = []
save_weights1 = LambdaCallback(on_epoch_end=lambda batch, logs: weights1.append(model.layers[0].get_weights()[0]))
save_weights2 = LambdaCallback(on_epoch_end=lambda batch, logs: weights2.append(model.layers[1].get_weights()[0]))
save_weights3 = LambdaCallback(on_epoch_end=lambda batch, logs: weights3.append(model.layers[1].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(xtrain, label_train,validation_data=(xtest, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights1,save_weights2,save_weights3])
final_w = [model.layers[0].get_weights()[0],model.layers[1].get_weights()[0],model.layers[2].get_weights()[0]]
weights = [weights1,weights2,weights3]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 695us/step - loss: 6.9392 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 2/30
    1000/1000 [==============================] - 0s 355us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 3/30
    1000/1000 [==============================] - 0s 355us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 4/30
    1000/1000 [==============================] - 0s 356us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 5/30
    1000/1000 [==============================] - 0s 357us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 6/30
    1000/1000 [==============================] - 0s 352us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 7/30
    1000/1000 [==============================] - 0s 354us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 8/30
    1000/1000 [==============================] - 0s 352us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 9/30
    1000/1000 [==============================] - 0s 348us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 10/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 11/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 12/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 13/30
    1000/1000 [==============================] - 0s 352us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 14/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 15/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 16/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 17/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 18/30
    1000/1000 [==============================] - 0s 354us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 19/30
    1000/1000 [==============================] - 0s 350us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 20/30
    1000/1000 [==============================] - 0s 349us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 21/30
    1000/1000 [==============================] - 0s 347us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 22/30
    1000/1000 [==============================] - 0s 349us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 23/30
    1000/1000 [==============================] - 0s 348us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 24/30
    1000/1000 [==============================] - 0s 353us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 25/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 26/30
    1000/1000 [==============================] - 0s 357us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 27/30
    1000/1000 [==============================] - 0s 353us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 28/30
    1000/1000 [==============================] - 0s 356us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 29/30
    1000/1000 [==============================] - 0s 351us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000
    Epoch 30/30
    1000/1000 [==============================] - 0s 347us/step - loss: 0.3256 - accuracy: 0.9000 - val_loss: 0.3255 - val_accuracy: 0.9000

```python
plot_model_result(history,weights,final_w)
```

## 2(c) . Construct and Train Convolutional Neural network
For Convolutional Neural Networks we first change shape of th Datasets from set of array of 1 Dimensional array of $28x28$ image to a set of 3-Dimensinal Image with Value.

```python
x_train_imgs = xtrain.reshape(xtrain.shape[0],28,28,1)
x_test_imgs = xtrain.reshape(xtest.shape[0],28,28,1)
```

### Preprocessing the Image :
We use th ImageDataGenerator library from keras to generate a augumented form of out original training dataset.

```python
from keras.preprocessing.image import ImageDataGenerator
aug_data = ImageDataGenerator(rotation_range=3,
                              width_shift_range=0.25, # 0.25 * 28 = 8 pixels approx
                              height_shift_range=0.25, # 0.25 * 28 = 8 pixels approx
                              horizontal_flip=True,
                              fill_mode="nearest")
aug_data.fit(x_train_imgs)
n = 0
for x_batch, y_batch in aug_data.flow(x_train_imgs, label_train, batch_size=1000):
    x_batch = np.reshape(x_batch, [-1, 28*28])
    model.fit(x_batch, y_batch)
    n += 1
    if n >= len(x_train_imgs) / 1000:
        break
x_batch = x_batch.reshape(x_batch.shape[0],28,28,1)
```

    Epoch 1/1
    1000/1000 [==============================] - 0s 78us/step - loss: 0.3252 - accuracy: 0.9000

Now we build the Convolutional Neutral Network and feed the Augumented Training Dataset to our model.

```python
model = Sequential()
model.add(Conv2D(30, activation='sigmoid',kernel_size=(5, 5), input_shape=(28,28,1)))
model.add(MaxPooling2D(pool_size=(2, 2), strides=(2, 2)))
model.add(Flatten())
model.add(Dropout(0.25))
model.add(Dense(10, activation='softmax'))
weights = []
save_weights = LambdaCallback(on_epoch_end=lambda batch, logs: weights.append(model.layers[0].get_weights()[0]))
sgd = optimizers.SGD(learning_rate=0.1)
model.compile(optimizer=sgd,
              loss='binary_crossentropy',
              metrics=['accuracy'])
history = model.fit(x_batch, y_batch,validation_data=(x_test_imgs, label_test),
                    epochs=30, batch_size=10, shuffle = False,callbacks = [save_weights])
final_w = [model.layers[0].get_weights()[0]]
weights = [weights]
```

    Train on 1000 samples, validate on 1000 samples
    Epoch 1/30
    1000/1000 [==============================] - 1s 1ms/step - loss: 0.7134 - accuracy: 0.8686 - val_loss: 0.3678 - val_accuracy: 0.8987
    Epoch 2/30
    1000/1000 [==============================] - 1s 883us/step - loss: 0.3261 - accuracy: 0.8984 - val_loss: 0.3791 - val_accuracy: 0.8962
    Epoch 3/30
    1000/1000 [==============================] - 1s 874us/step - loss: 0.3104 - accuracy: 0.8997 - val_loss: 0.3911 - val_accuracy: 0.8927
    Epoch 4/30
    1000/1000 [==============================] - 1s 881us/step - loss: 0.2959 - accuracy: 0.9031 - val_loss: 0.3962 - val_accuracy: 0.8924
    Epoch 5/30
    1000/1000 [==============================] - 1s 880us/step - loss: 0.2871 - accuracy: 0.9046 - val_loss: 0.4069 - val_accuracy: 0.8873
    Epoch 6/30
    1000/1000 [==============================] - 1s 878us/step - loss: 0.2766 - accuracy: 0.9055 - val_loss: 0.4134 - val_accuracy: 0.8900
    Epoch 7/30
    1000/1000 [==============================] - 1s 871us/step - loss: 0.2653 - accuracy: 0.9073 - val_loss: 0.4232 - val_accuracy: 0.8732
    Epoch 8/30
    1000/1000 [==============================] - 1s 880us/step - loss: 0.2541 - accuracy: 0.9090 - val_loss: 0.4219 - val_accuracy: 0.8821
    Epoch 9/30
    1000/1000 [==============================] - 1s 880us/step - loss: 0.2451 - accuracy: 0.9130 - val_loss: 0.4370 - val_accuracy: 0.8836
    Epoch 10/30
    1000/1000 [==============================] - 1s 884us/step - loss: 0.2386 - accuracy: 0.9130 - val_loss: 0.4234 - val_accuracy: 0.8808
    Epoch 11/30
    1000/1000 [==============================] - 1s 891us/step - loss: 0.2294 - accuracy: 0.9174 - val_loss: 0.4404 - val_accuracy: 0.8792
    Epoch 12/30
    1000/1000 [==============================] - 1s 887us/step - loss: 0.2202 - accuracy: 0.9187 - val_loss: 0.4411 - val_accuracy: 0.8747
    Epoch 13/30
    1000/1000 [==============================] - 1s 879us/step - loss: 0.2136 - accuracy: 0.9212 - val_loss: 0.4486 - val_accuracy: 0.8742
    Epoch 14/30
    1000/1000 [==============================] - 1s 884us/step - loss: 0.2072 - accuracy: 0.9237 - val_loss: 0.4560 - val_accuracy: 0.8691
    Epoch 15/30
    1000/1000 [==============================] - 1s 888us/step - loss: 0.2011 - accuracy: 0.9252 - val_loss: 0.4632 - val_accuracy: 0.8716
    Epoch 16/30
    1000/1000 [==============================] - 1s 882us/step - loss: 0.1938 - accuracy: 0.9269 - val_loss: 0.4816 - val_accuracy: 0.8676
    Epoch 17/30
    1000/1000 [==============================] - 1s 879us/step - loss: 0.1862 - accuracy: 0.9307 - val_loss: 0.4618 - val_accuracy: 0.8724
    Epoch 18/30
    1000/1000 [==============================] - 1s 869us/step - loss: 0.1757 - accuracy: 0.9356 - val_loss: 0.4740 - val_accuracy: 0.8679
    Epoch 19/30
    1000/1000 [==============================] - 1s 874us/step - loss: 0.1700 - accuracy: 0.9373 - val_loss: 0.5001 - val_accuracy: 0.8602
    Epoch 20/30
    1000/1000 [==============================] - 1s 867us/step - loss: 0.1668 - accuracy: 0.9384 - val_loss: 0.4896 - val_accuracy: 0.8624
    Epoch 21/30
    1000/1000 [==============================] - 1s 898us/step - loss: 0.1617 - accuracy: 0.9416 - val_loss: 0.4837 - val_accuracy: 0.8652
    Epoch 22/30
    1000/1000 [==============================] - 1s 877us/step - loss: 0.1561 - accuracy: 0.9414 - val_loss: 0.5013 - val_accuracy: 0.8626
    Epoch 23/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1498 - accuracy: 0.9442 - val_loss: 0.5084 - val_accuracy: 0.8612
    Epoch 24/30
    1000/1000 [==============================] - 1s 866us/step - loss: 0.1417 - accuracy: 0.9490 - val_loss: 0.5055 - val_accuracy: 0.8630
    Epoch 25/30
    1000/1000 [==============================] - 1s 870us/step - loss: 0.1356 - accuracy: 0.9500 - val_loss: 0.5158 - val_accuracy: 0.8606
    Epoch 26/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1332 - accuracy: 0.9516 - val_loss: 0.5064 - val_accuracy: 0.8595
    Epoch 27/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1319 - accuracy: 0.9520 - val_loss: 0.5240 - val_accuracy: 0.8585
    Epoch 28/30
    1000/1000 [==============================] - 1s 873us/step - loss: 0.1241 - accuracy: 0.9568 - val_loss: 0.5211 - val_accuracy: 0.8576
    Epoch 29/30
    1000/1000 [==============================] - 1s 871us/step - loss: 0.1264 - accuracy: 0.9538 - val_loss: 0.5257 - val_accuracy: 0.8551
    Epoch 30/30
    1000/1000 [==============================] - 1s 872us/step - loss: 0.1169 - accuracy: 0.9588 - val_loss: 0.5339 - val_accuracy: 0.8546

```python
plot_model_result(history,weights,final_w)
```

```python

```


```python
import sympy as sym
x, L, C, D, c_0, c_1, = sym.symbols('x L C D c_0 c_1')

class TwoPtBoundaryValueProblem(object):
    """
    Solve -(a*u')' = f(x) with boundary conditions
    specified in subclasses (method get_bc).
    a and f must be sympy expressions of x.
    """
    def __init__(self, f, a=1, L=L, C=C, D=D):
        """Default values for L, C, D are symbols."""
        self.f = f
        self.a = a
        self.L = L
        self.C = C
        self.D = D

        # Integrate twice
        u_x = - sym.integrate(f, (x, 0, x)) + c_0
        u = sym.integrate(u_x/a, (x, 0, x)) + c_1
        # Set up 2 equations from the 2 boundary conditions and solve
        # with respect to the integration constants c_0, c_1
        eq = self.get_bc(u)
        eq = [sym.simplify(eq_) for eq_ in eq]
        print(('BC eq:', eq))
        self.u = self.apply_bc(eq, u)

    def apply_bc(self, eq, u):
        # Solve BC eqs respect to the integration constants
        r = sym.solve(eq, [c_0, c_1])
        # Substitute the integration constants in the solution
        u = u.subs(c_0, r[c_0]).subs(c_1, r[c_1])
        u = sym.simplify(sym.expand(u))
        return u

    def get_solution(self, latex=False):
        return sym.latex(self.u, mode='plain') if latex else self.u

    def get_residuals(self):
        """Return the residuals in the equation and BCs."""
        R_eq = sym.diff(sym.diff(self.u, x)*self.a, x) + self.f
        R_0, R_L = self.get_bc(self.u)
        residuals = [sym.simplify(R) for R in (R_eq, R_0, R_L)]
        return residuals

    def get_bc(self, u):
        raise NotImplementedError(
            'class %s has not implemented get_bc' %
            self.__class__.__name__)

class Model1(TwoPtBoundaryValueProblem):
    """u(0)=0, u(L)=D."""
    def get_bc(self, u):
        return [u.subs(x, 0)-0,               # x=0 condition
                u.subs(x, self.L) - self.D]   # x=L condition

class Model2(TwoPtBoundaryValueProblem):
    """u'(0)=C, u(L)=D."""
    def get_bc(self, u):
        return [sym.diff(u,x).subs(x, 0) - self.C, # x=0 cond.
                u.subs(x, self.L) - self.D]        # x=L cond.

class Model3(TwoPtBoundaryValueProblem):
    """u(0)=C, u(L)=D."""
    def get_bc(self, u):
        return [u.subs(x, 0) - self.C,
                u.subs(x, self.L) - self.D]

class Model4(TwoPtBoundaryValueProblem):
    """u(0)=0, -u'(L)=C*(u-D)."""
    def get_bc(self, u):
        return [u.subs(x, 0) - 0,
                -sym.diff(u, x).subs(x, self.L) -
                self.C*(u.subs(x, self.L) - self.D)]

def test_TwoPtBoundaryValueProblem():
    f = 2
    model = Model1(f)
    print(('Model 1, u:', model.get_solution()))
    for R in model.get_residuals():
        assert R == 0

    f = x
    model = Model2(f)
    print(('Model 2, u:', model.get_solution()))
    for R in model.get_residuals():
        assert R == 0

    f = 0
    a = 1 + x**2
    model = Model3(f, a=a)
    print(('Model 3, u:', model.get_solution()))
    for R in model.get_residuals():
        assert R == 0

def demo_Model4():
    f = 0
    model = Model4(f, a=sym.sqrt(1+x))
    print(('Model 4, u:', model.get_solution()))

if __name__ == '__main__':
    test_TwoPtBoundaryValueProblem()
    demo_Model4()

```

    ('BC eq:', [c_1, -D - L**2 + L*c_0 + c_1])
    ('Model 1, u:', x*(D + L*(L - x))/L)
    ('BC eq:', [-C + c_0, -D - L**3/6 + L*c_0 + c_1])
    ('Model 2, u:', -C*L + C*x + D + L**3/6 - x**3/6)
    ('BC eq:', [-C + c_1, -D + c_0*atan(L) + c_1])
    ('Model 3, u:', (C*atan(L) - C*atan(x) + D*atan(x))/atan(L))
    ('BC eq:', [c_1, C*D - 2*C*c_0*sqrt(L + 1) + 2*C*c_0 - C*c_1 - c_0/sqrt(L + 1)])
    ('Model 4, u:', 2*C*D*sqrt(L + 1)*(sqrt(x + 1) - 1)/(2*C*L - 2*C*sqrt(L + 1) + 2*C + 1))

```python

```

Data Science and Machine Learning/Machine-Learning-In-Python-THOROUGH/EXAMPLES/FINITE_ELEMENTS/INTRO/EXERCICES/27_U_XX_F_SYMPY_CLASS.ipynb


```python
# %load science.py
%matplotlib notebook
from sympy import init_session, symbols
import numpy as np
import matplotlib.pyplot as plt
import scipy as sp
import pandas as pd
import theano as T
#from sympy.geometry import *
init_session()
a,b,c = symbols('a b c')
r, theta, phi = symbols('r theta phi', positive=True)
print('Loaded a b c theta and phi')
print('Load Theano as T')
```

    IPython console for SymPy 1.0 (Python 3.5.2-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.0/
    Loaded a b c theta and phi
    Load Theano as T

# Henon Map
This notebook explores, or is it plays, with the construction of the 2D Henon map. The equations are:
$$ x_{n + 1} = y_{n} + 1 - ax^2_n$$
$$ y_{n + 1} = b x_n$$

These are broken into three mapping steps. My first idea is to look at how each step transforms a line $-1 \leq x \leq 1$.The line is viewed as a 2d object with countable width and infinitely small thickness.

The second idea is to plot successive iterations of that line in their final form on one plot to see how each iteration maps the original and subsequent objects.

See also [Wen](http://chaosbook.org/projects/Wen14.pdf)

### Bayesian NUTS and Henon

It may be unrelated, but he Bayesian NUTS sampler appears to move around a similar shape, in a similar mapping process.

```python
#Step 1 of the iterate. Bend the line.
def Step1(xn, yn, a):
    return xn, yn + 1 -a*xn**2
#Step2: Squeeze up the x-axis
def Step2(xn, yn, b):
    return b*xn, yn
#Step3: Reverse the axes (90 degree rotation)
def Step3(xn,yn):
    return yn, xn
def Henon_map(xn,yn,a,b): #Made this explicit with unneeded intermediate variables.
    x = xn
    y = yn
    x1,y1 = Step1(x,y,a)
    x2,y2 = Step2(x1,y1,b)
    x3,y3 = Step3(x2,y2)
    return x3,y3
```

```python
#Set some initial values; I use the one's from Henon.
a = 1.4
b = 0.3
num_values = int(2e5)
x_min = -1
x_max = 1
#Create two arrays. It would be better code to create an array and manipulate it
#but in this instance maybe clearer in the code using two arrays.
xn = np.linspace(x_min,x_max, num_values)
yn = np.empty_like(xn)#Could declare zeroes but left obvious that one sets the value
yn.fill(0)#Filling with =1 blows up the model quickly
xn[:10],yn[:10]
```

    (array([-1.     , -0.99999, -0.99998, -0.99997, -0.99996, -0.99995,
            -0.99994, -0.99993, -0.99992, -0.99991]),
     array([ 0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.,  0.]))

```python
xn2, yn2 = Henon_map(xn,yn,a,b)
xn3, yn3 = Henon_map(xn2,yn2,a,b)
xn4, yn4 = Henon_map(xn3,yn3,a,b)
xn5, yn5 = Henon_map(xn4,yn4,a,b)
xn6, yn6 = Henon_map(xn5,yn5,a,b)
xn7, yn7 = Henon_map(xn6,yn6,a,b)
xn8, yn8 = Henon_map(xn7,yn7,a,b)
xn9, yn9 = Henon_map(xn8,yn8,a,b)
```

```python
fixed1 = (-(1-b)+np.sqrt((1-b)**2 +4*a))/(2*a)
fixed2 = (-(1-b)-np.sqrt((1-b)**2 +4*a))/(2*a)
```

```python
from mpl_toolkits.axes_grid1.inset_locator import zoomed_inset_axes
from mpl_toolkits.axes_grid1.inset_locator import mark_inset

#Plot the results of the iterations.
fig, ax0 = plt.subplots(1,1,figsize=(8,6))
ax0.set_facecolor(plt.cm.gray(.85))
ax0.set_title('Henon Map',size=16)
ax0.set_xlabel('$x_{n + i}$',size=14)
ax0.set_ylabel('$y_{n + i}$',size=14)
# ax0.plot(fixed1,fixed1,'ro')
# ax0.plot(fixed2,fixed2,'ro')
ax0.plot(xn,yn, 'k-',lw=1,label='Start')
ax0.plot(xn2,yn2, 'b.',markersize=.1,label='1: Blue',alpha=.3)
ax0.plot(xn3,yn3, 'g.',markersize=.1,label='2: Green',alpha=.3)
ax0.plot(xn4,yn4, 'r.',markersize=.1,label='3: Red',alpha=.3)
ax0.plot(xn5,yn5, 'c.',markersize=.1,label='4: Cyan',alpha=.3)
ax0.plot(xn6,yn6, 'm.',markersize=.1,label='5: Magenta',alpha=.3)
# ax0.plot(xn7,yn7, 'y.',markersize=.1,label='6: Yellow',alpha=.3)
# ax0.plot(xn8,yn8, 'k.',markersize=.1,label='7: Black',alpha=.3)
# ax0.plot(xn9,yn9, 'b.',markersize=.1,label='8: Blue',alpha=.3)
ax0.grid()
ax0.legend()

#Create an inset box.
axins = zoomed_inset_axes(ax0, 10, loc=6) # zoom-factor: 2.5, location: lower right
#axins.set_facecolor(plt.cm.gray(.9))
axins.plot(xn2, yn2,'b,')
axins.plot(xn3, yn3,'g,')
axins.plot(xn4, yn4,'r,')
axins.plot(xn5, yn5,'c,')
axins.plot(xn6, yn6,'m,')
# axins.plot(xn7, yn7,'y,')
# axins.plot(xn8, yn8,'k,')
# axins.plot(xn9, yn9,'b,')
x1, x2, y1, y2 = -1.25, -1.15, 0.36, 0.39 # specify the limits
axins.set_xlim(x1, x2) # apply the x-limits
axins.set_ylim(y1, y2) # apply the y-limits
plt.yticks(visible=False)
plt.xticks(visible=False)
mark_inset(ax0, axins, loc1=2, loc2=4, fc="none", ec="0.5")
```

    <IPython.core.display.Javascript object>

    (<mpl_toolkits.axes_grid1.inset_locator.BboxPatch at 0x1078527f0>,
     <mpl_toolkits.axes_grid1.inset_locator.BboxConnector at 0x1078524a8>,
     <mpl_toolkits.axes_grid1.inset_locator.BboxConnector at 0x1288286d8>)

If one sets the line to be between -1 and 1, the mapping rapidly becomes a long exponential type tail. You can zoom in on a thin x-slice of this to see the Henon 'tick'.

```python

```


# Introduction

Now that we have the symbolic equations of motion we need to transform them into Python functions that can be evaluated for use in numerical integration. [Numerical integration](http://en.wikipedia.org/wiki/Numerical_methods_for_ordinary_differential_equations) is required to solve the ordinary differential initial value problem and allow us to see how the states change through time.

# Setup

Load the solutions from the previous notebooks:

```python
from solution.equations_of_motion import *
```

To setup the numerical values and integrate the equations of motion we will need some functions from NumPy for numerical arrays:

```python
from numpy import deg2rad, rad2deg, array, zeros, linspace
```

We will need a ODE numerical integration routine from SciPy:

```python
from scipy.integrate import odeint
```

We can use PyDy's ODE function generator to transform the symbolic equations into numerical functions:

```python
from pydy.codegen.ode_function_generators import generate_ode_function
```

Once again, let's display the symbolics nicely.

```python
from sympy.physics.vector import init_vprinting, vlatex
```

```python
init_vprinting(use_latex='mathjax', pretty_print=False)
```

Once we get a solution for how the state changes over time, it is nice to visualize it. The simplest way to do this is to plot the trajectories versus time. We can use the matplotlib library to do this. First enable inline plotting:

```python
%matplotlib inline
```

Import a few functions for plotting:

```python
from matplotlib.pyplot import plot, legend, xlabel, ylabel, rcParams
```

And set the default figure size to be larger:

```python
rcParams['figure.figsize'] = (14.0, 6.0)
```

# Variables

The first step is to gather all of the variables in the equations of motion into lists. We will need the constants, coordinates, speeds, and the specified inputs.

## Constants

There are twelve constants in the equations. Put them into a list.

```python
constants = [lower_leg_length,
             lower_leg_com_length,
             lower_leg_mass,
             lower_leg_inertia,
             upper_leg_length,
             upper_leg_com_length,
             upper_leg_mass,
             upper_leg_inertia,
             torso_com_length,
             torso_mass,
             torso_inertia,
             g]
constants

```

## Time Varying

The coordinates and speeds make up the states and there are three time varying specified inputs to the system, the joint torques.

```python
coordinates = [theta1, theta2, theta3]
coordinates
```

```python
speeds = [omega1, omega2, omega3]
speeds
```

## Exercise

Make a list called `specified` that contains the three torque magnitude variables: $T_a$, $T_k$, and $T_h$.

```python

```

```python
%load exercise_solutions/n07_simulation_torque-magnitude.py
```

# Generate the Numerical ODE Function

Ordinary differential equation integrators, like `scipy.integrate.odeint`, require a function that numerically evaluates the right hand side of the coupled first order ordinary differential equations. We have the symbolic form of the mass matrix and the forcing vector available. The `generate_ode_function` function generates a function from the symbolic expressions that fits the form needed for `odeint`.

`odeint` is an ODE integrator based on the `lsoda` routine from ODEPACK that works well for both non-stiff and stiff ODEs. Notice that it requres the right hand side function, the initial conditions of the state, a time vector. We will also pass in extra arguments, `args`, of the right hand side function.

```python
help(odeint)
```

To create the function, simply pass in $\mathbf{M}$, $\mathbf{f}$, and the lists of variables in the system.

```python
right_hand_side = generate_ode_function(forcing_vector, coordinates,
                                        speeds, constants,
                                        mass_matrix=mass_matrix,
                                        specifieds=specified)
```

We see that the result is a function.

```python
type(right_hand_side)
```

And the doc string gives information on the type of the arguments needed to evaluate it:

```python
help(right_hand_side)
```

# Set the Initial Conditions, Parameter Values, and Time Array

We will set the intial values of the speeds to be zero and the coordinates to be offset from vertical at 2 degrees. First make an array of zeros:

```python
x0 = zeros(6)
x0
```

And then set the first three values, $\theta_{1,2,3}$, to 2 degrees:

```python
x0[:3] = deg2rad(2.0)
x0
```

The right hand side function requires numerical values of all the constants to be passed in and values for the specified joint torques. Here we will use typical values from body segment parameter measurements which were generated from the [Yeadon](http://yeadon.readthedocs.org/en/latest/) Python package (`male1.txt`). Make sure the units are all consistent!

```python
numerical_constants = array([0.611,  # lower_leg_length [m]
                             0.387,  # lower_leg_com_length [m]
                             6.769,  # lower_leg_mass [kg]
                             0.101,  # lower_leg_inertia [kg*m^2]
                             0.424,  # upper_leg_length [m]
                             0.193,  # upper_leg_com_length
                             17.01,  # upper_leg_mass [kg]
                             0.282,  # upper_leg_inertia [kg*m^2]
                             0.305,  # torso_com_length [m]
                             32.44,  # torso_mass [kg]
                             1.485,  # torso_inertia [kg*m^2]
                             9.81],  # acceleration due to gravity [m/s^2]
                            )
```

## Exercise

For this first simulation we will set the three joint torques equal to zero for the duration of the simulation. Created a Python variale `numerical_specified` which is a NumPy array of length three and each entry is equal to zero.

```python

```

We can use the `linspace` function to generate a time vector over 10 secs such that `odeint` returns results at 60 Hz.

```python

```

```python
%load exercise_solutions/n07_simulation_sim-setup.py
```

# Integrate the Equations of Motion

The right hand side function can now be evaluated numerically given a current value of the states, a value for time, and the numerical values for all the constants and specified values in the equations of motion:

```python
right_hand_side(x0, 0.0, numerical_specified, numerical_constants)
```

Now we can solve the initial value problem and simulate the motion. As shown above, `odeint` requires the function to integrate `right_hand_side`, the initial conditions `x0`, the time vector `t`, and the extra arguments `args`:

```python
y = odeint(right_hand_side, x0, t, args=(numerical_specified, numerical_constants))
```

The `y` variable now contains a 2D array that gives the trajectories of the states as a function of time.

```python
y.shape
```

# Plot the results

We can plot the first 3 columns of `y` versus `t` so see how the three angles change throughout time.

```python
plot(t, rad2deg(y[:, :3]))
xlabel('Time [s]')
ylabel('Angle [deg]')
legend(["${}$".format(vlatex(c)) for c in coordinates])
```

What does this graph tell us? How does the system behave?

## Exercise

Now as an exercise, plot the generalized speeds (i.e. the last three states).

```python

```

```python
%load exercise_solutions/n07_simulation_plot-speeds.py
```


```python
round(22/7, 3)
```

```python
import random
```

```python
def zero_or_one():
    r = random.choice([0, 1])
    return r

trial = [zero_or_one() for i in range(20)]

print(trial, sum(trial))
```

```python
N_coins = 50
N_heads = 25

N_simul = 20000

count = 0
for i in range(N_simul):
    trial = [zero_or_one() for i in range(N_coins)]
    if sum(trial) >= N_heads:
        count += 1

print( 1.0 * count / N_simul )
```

```python
trial = np.random.randint(2, size=10)
print(trial)
print(trial.sum())
```

```python
N = 100000
n_coins = 50
n_heads = 25

count = 0
for i in range(N):
    trial = np.random.randint(2, size=n_coins)
    if trial.sum() >= 25:
        count += 1

1.0 * count / N
```

# Sympy

```python
from sympy import symbols, solve

x = symbols('x')

solve(x**2 - 2, x)
```

```python
from sympy import diff, sin, exp
diff(sin(x) * exp(x), x)
```

```python
from sympy import init_printing
init_printing(use_unicode=True)

diff(sin(x) * exp(x), x)
```

```python
from sympy import integrate, cos
integrate(exp(x)*sin(x) + exp(x)*cos(x), x)
```

# Prettier graphs

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# seaborn package is not include in anaconda
# neet to run `conda install seaborn` or `pip install seaborn`

import seaborn
seaborn.set_style('whitegrid')
```

```python
def f(x):
    return x*x

x = np.linspace(-5, 5, 100)
plt.plot(x, f(x))

s = x[::5]
plt.plot(s, f(s), 'ro')

#plt.savefig('/tmp/toto.png', dpi=300)
```


Autograd: automatic differentiation
===================================
Central to all neural networks in PyTorch is the ``autograd`` package.
Let’s first briefly visit this, and we will then go to training our
first neural network.

The ``autograd`` package provides automatic differentiation for all operations
on Tensors. It is a define-by-run framework, which means that your backprop is
defined by how your code is run, and that every single iteration can be
different.

Let us see this in more simple terms with some examples.

Tensor
--------
``torch.Tensor`` is the central class of the package. If you set its attribute
``.requires_grad`` as ``True``, it starts to track all operations on it. When
you finish your computation you can call ``.backward()`` and have all the
gradients computed automatically. The gradient for this tensor will be
accumulated into ``.grad`` attribute.

To stop a tensor from tracking history, you can call ``.detach()`` to detach
it from the computation history, and to prevent future computation from being
tracked.

To prevent tracking history (and using memory), you can also wrap the code block
in ``with torch.no_grad():``. This can be particularly helpful when evaluating a
model because the model may have trainable parameters with `requires_grad=True`,
but for which we don't need the gradients.

There’s one more class which is very important for autograd
implementation - a ``Function``.

``Tensor`` and ``Function`` are interconnected and build up an acyclic
graph, that encodes a complete history of computation. Each tensor has
a ``.grad_fn`` attribute that references a ``Function`` that has created
the ``Tensor`` (except for Tensors created by the user - their
``grad_fn is None``).

If you want to compute the derivatives, you can call ``.backward()`` on
a ``Tensor``. If ``Tensor`` is a scalar (i.e. it holds a one element
data), you don’t need to specify any arguments to ``backward()``,
however if it has more elements, you need to specify a ``gradient``
argument that is a tensor of matching shape.

```
import torch
```

Create a tensor and set requires_grad=True to track computation with it

```
x = torch.ones(2, 2, requires_grad=True)
```

Do an operation of tensor:

```
y = x + 2
print(y)
```

``y`` was created as a result of an operation, so it has a ``grad_fn``.

```
print(y.grad_fn)
```

Do more operations on y

```
z = y * y * 3
out = z.mean()

print(z, out)
```

``.requires_grad_( ... )`` changes an existing Tensor's ``requires_grad``
flag in-place. The input flag defaults to ``True`` if not given.

```
a = torch.randn(2, 2)
a = ((a * 3) / (a - 1))
print(a.requires_grad)
a.requires_grad_(True)
print(a.requires_grad)
b = (a * a).sum()
print(b.grad_fn)
```

Gradients
---------
Let's backprop now
Because ``out`` contains a single scalar, ``out.backward()`` is
equivalent to ``out.backward(torch.tensor(1))``.

```
out.backward()
```

print gradients d(out)/dx

```
print(x.grad)
```

You should have got a matrix of ``4.5``.

Let’s call the ``out``
*Tensor* $o$.
We have that $o = \frac{1}{4}\sum_i z_i$,
$z_i = 3(x_i+2)^2$ and $z_i\bigr\rvert_{x_i=1} = 27$.
Therefore,

\begin{equation}
\frac{\partial o}{\partial x_i} = \frac{3}{2}(x_i+2)
\end{equation}

hence

\begin{equation}
\frac{\partial o}{\partial x_i}\bigr\rvert_{x_i=1} = \frac{9}{2} = 4.5
\end{equation}

You can do many crazy things with autograd!

```
x = torch.randn(3, requires_grad=True)

y = x * 2
while y.data.norm() < 1000:
    y = y * 2

print(y)
```

```
gradients = torch.tensor([0.1, 1.0, 0.0001], dtype=torch.float)
y.backward(gradients)

print(x.grad)
```

You can also stop autograd from tracking history on Tensors
with ``.requires_grad``=True by wrapping the code block in
``with torch.no_grad():``

```
print(x.requires_grad)
print((x ** 2).requires_grad)

with torch.no_grad():
	print((x ** 2).requires_grad)
```

**Read Later:**

Documentation of ``autograd`` and ``Function`` is at
http://pytorch.org/docs/autograd


```python
import numpy as np
import cv2
from matplotlib import pyplot as plt
import math
import sys

num_frames = 166
video_imgs = []
readdir = 'D:/44754/Documents/robotics/Perception/RoboticsPerceptionWeek3Code/data/apriltagims/image{:03d}.jpg'
for i in range(num_frames):
    tmp = cv2.imread(readdir.format(i+1))
    video_imgs.append(tmp)
video_imgs = np.array(video_imgs)
print(video_imgs.shape)
```

    (166, 480, 640, 3)

### Pose Estimation
Given the initial corner pixel coordinates, the corner coordinates over time $p_{im}$ are extracted via KLT method. With intrinsic camera parameter matrix $K$ and $p_w$ the cube real world coordinate, homography $H$ can be solved and hence give the rotation $R$ and translation $t$. The homography is of the form,

$$
\begin{align}
Hp_w &\sim p_c = Kp_{im} \\
H &= \lambda
\begin{pmatrix}
| & | & | \\ r_1 & r_2 & t \\ | & | & | \end{pmatrix}
\end{align}$$

Assuming all the world points lie on $Z=0$,

\begin{align}
\begin{pmatrix} x_c \\ y_c \\ z_c \end{pmatrix} &= \left[\begin{array}{cc} R & t \end{array} \right]\begin{pmatrix} x_w \\ y_w \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} |&|&|&|\\r_1&r_2&r_3&t\\|&|&|&|\end{pmatrix}\begin{pmatrix} x_w \\ y_w \\ 0 \\ 1 \end{pmatrix} \\
\Longrightarrow z_c \begin{pmatrix} x_c/z_c \\ y_c/z_c \\ 1 \end{pmatrix} &= \begin{pmatrix} |&|&|\\r_1&r_2&t\\|&|&|\end{pmatrix}\begin{pmatrix} x_w \\ y_w \\ 1 \end{pmatrix} \\
\Longrightarrow \begin{pmatrix}x_{im}\\y_{im}\\1\end{pmatrix}  &= \underbrace{\frac{1}{z_c}\begin{pmatrix} |&|&|\\r_1&r_2&t\\|&|&|\end{pmatrix}}_{H}\begin{pmatrix} x_w \\ y_w \\ 1  \end{pmatrix} = \begin{pmatrix} |&|&|\\h_1&h_2&h_3\\|&|&|\end{pmatrix}\begin{pmatrix} x_w \\ y_w \\ 1  \end{pmatrix}
\end{align}

To extract $R$ and $t$ from $H$, assume that there was no noise, then $h_1=\lambda r_1$, $h_2=\lambda r_2$ and $h_3=\lambda t$, for some arbitrary scale factor $\lambda$. Since $r_1$ has norm 1, then $r_1 = \frac{h_1}{\|h_1\|}$, $r_2 = \frac{h_2}{\|h_2\|}$, $r_3 = r_1\times r_2$ and $t = \frac{h_3}{\|h_1\|}$. However, because of noise, $h_1$ and $h_2$ may not be orthogonal and thus not be a rotation matrix. To fix this, SVD is usedm giving estimate of $R$ and $t$,

\begin{align}
R' &=
\begin{pmatrix}
|&|&|\\h_1&h_2&(h_1\times h_2)\\|&|&|
\end{pmatrix} = USV^\top \\
R &=
U\begin{pmatrix}
1&0&0\\0&1&0\\0&0&\det(UV^\top)
\end{pmatrix}V^\top  \\
\end{align}

$R$ and $t$ can now be extracted from $H$,
\begin{align}
R &=
U\begin{pmatrix}
1&0&0\\0&1&0\\0&0&\det(UV^\top)
\end{pmatrix}V^\top  \\
t &= \frac{h_3}{\|h_1\|}
\end{align}

The above determines the basic algorithm, however, since a homography is only determined up to a scale, there is a sign ambiguity, as if $H$ is negated, the approximate rotation is obtained,

$$R' =
\begin{pmatrix}
|&|&|\\-h_1&-h_2&(-h_1\times -h_2)\\|&|&|
\end{pmatrix} = \begin{pmatrix}
|&|&|\\-h_1&-h_2&(h_1\times h_2)\\|&|&|
\end{pmatrix}$$

This gives a flip of the $x$ and $y$ axes and thus two valid rotation matrices are available, corresponding to the translations $t$ and $-t$  respectively. To disambiguate this, the code should enforce that the $z$ component of $t$ is positive (i.e. $t_3 = H_{33}>0$ and multiply the $H$ matrix by the appropriate sign to do this.

### Projecting Points
Once $R$ and $T$ are computed, this can be used to place virtual objects into the world. Given a set of points $X$, project them onto the image,

\begin{align}
\begin{pmatrix}x_c\\y_c\\z_c\end{pmatrix} &= X_c = K(RX+t) \\
X_{im} &= \frac{X_c}{z_c}
\end{align}

```python
initial_points = 100 * np.array([
   [1.981631469726562,3.165294189453125],
   [3.786268920898438,3.424402770996094],
   [4.036800842285157,1.686005859375000],
   [2.333528289794922,1.491907043457031] ]);
K = 100*np.array([[7.661088867187500,0,3.139585628047498],
                 [0,7.699354248046875,2.503607131410900],
                 [0,0,0.010000000000000]])
pr = np.array([[1,0,0],[0,1,0]])
tag_width = 0.13
tag_height = 0.13
cube_depth = 0.13
print(initial_points.shape)
print(K.shape)
```

    (4, 2)
    (3, 3)

```python
def vectorize(r,rp):
    x1=r[0]; x2=r[1]; x3=1
    x1p=rp[0]; x2p=rp[1]
    ax=[-x1,-x2,-1,0,0,0,x1*x1p,x2*x1p,x1p]
    ay=[0,0,0,-x1,-x2,-x3,x1*x2p,x2*x2p,x2p]
    return ax,ay
def est_homography(video_pts,logo_pts):
    A = []
    for i in range(len(logo_pts)):
        r = video_pts[i,:]; rp = logo_pts[i,:]
        ax,ay = vectorize(r,rp)
        A.append(ax)
        A.append(ay)
    A = np.array(A)
    u, s, vh = np.linalg.svd(A, full_matrices=True)
    H = vh[-1,:].reshape((3,3))
    return H
```

```python
class KLTtrack():
    def __init__(self,imglist,trackpts):
        self.vid = imglist
        self.pts = trackpts
        # Parameters for lucas kanade optical flow
        self.lk_params = dict(winSize  = (31,31),
                  maxLevel = 3,
                  criteria = (cv2.TERM_CRITERIA_EPS|cv2.TERM_CRITERIA_COUNT, 5, 0.03))

    def pointTracker(self):
        # Create some random colors
        color = np.random.randint(0,255,(self.pts.shape[0],3))
        # Take first frame and find corners in it
        old_frame = self.vid[0,:,:,:]
        old_gray = cv2.cvtColor(old_frame, cv2.COLOR_BGR2GRAY)
        p0 = np.floor(self.pts).reshape(-1,1,2).astype(np.float32)

        # Create a mask image for drawing purposes
        mask = np.zeros_like(old_frame)
        corners = []
        corners.append(self.pts)

        for i in range(1,num_frames):
            frame = video_imgs[i,:,:,:]
            frame_gray = cv2.cvtColor(frame, cv2.COLOR_BGR2GRAY)
            # calculate optical flow
            p1, st, err = cv2.calcOpticalFlowPyrLK(old_gray, frame_gray, p0, None, **self.lk_params)
            # Select good points
            good_new = p1[st==1]
            good_old = p0[st==1]
            # draw the tracks
            for i,(new,old) in enumerate(zip(good_new, good_old)):
                a,b = new.ravel()
                c,d = old.ravel()
                mask = cv2.line(mask, (a,b),(c,d), color[i].tolist(), 2)
                frame = cv2.circle(frame,(a,b),5,color[i].tolist(),-1)
            img = cv2.add(frame,mask)

            k = cv2.waitKey(5) & 0xff
            if k == 27:
                break
            # Now update the previous frame and previous points
            old_gray = frame_gray.copy()
            corners.append(good_new)
            p0 = good_new.reshape(-1,1,2)

        return np.array(corners)
```

```python
## generate klt images
demo = KLTtrack(video_imgs,initial_points)
corners = demo.pointTracker()
```

```python
%matplotlib inline
%load_ext autoreload
%autoreload 2
%matplotlib notebook

fig = plt.figure()
ax = plt.gca()
plt.ion()
fig.show()
fig.canvas.draw()

for i in range(num_frames):
    ax.clear()
    ax.imshow(video_imgs[i,:,:,:])
    ax.plot(corners[i,:,0],corners[i,:,1],'ro')
    fig.canvas.draw()
```

    <IPython.core.display.Javascript object>

```python
generated_imgs = []
corner_pts = .5*tag_width*np.array([[1,1],[-1,1],
                                    [-1,-1],[1,-1]])
render_pts1 = np.hstack((corner_pts,np.zeros((4,1))))
render_pts2 = np.hstack((corner_pts,cube_depth*np.ones((4,1))))
render_pts = np.vstack((render_pts1,render_pts2))

```

```python
def ar_cube(H,render_pts,K):
    H = H*np.sign(H[2,2])
    h1 = H[:,0]; h2 = H[:,1]
    h3 = H[:,2]
    Rp = np.vstack((h1,h2,np.cross(h1,h2))).T
    u, s, vh = np.linalg.svd(Rp, full_matrices=True)
    S = np.diag(np.array([1,1,np.linalg.det(np.matmul(u,vh.T))]))
    R = np.matmul(np.matmul(u,S),vh.T)
    t = h3/np.linalg.norm(h1)

    proj_pts = []
    for i in range(len(render_pts)):
        xc = np.matmul(K,np.matmul(R,render_pts[i,:])+t)
        tmppj = xc[0:2].T/xc[-1]
        proj_pts.append(tmppj)

    return np.array(proj_pts),t,R
```

```python
def draw_ar_cube(points,im):
    inds=np.array([0,1,0,3,0,4,1,2,1,5,
                  2,3,2,6,3,7,4,5,4,7,
                  5,6,6,7])
    X = points[inds,:]
    for i in np.arange(0,len(X),2):
        Xj = X[np.array([i,i+1]),:];
        im = cv2.line(tmp,tuple(Xj[0,:].astype(int)),
                   tuple(Xj[1,:].astype(int)),(0,200,200),5)
    return im
```

```python
fig = plt.figure()
ax = plt.gca()
plt.ion()
fig.show()
fig.canvas.draw()

generated_imgs = video_imgs.copy()
pos = []; rot = []
for i in range(len(generated_imgs)):
# for i in range(1):
    tmp = generated_imgs[i,:,:,:]
    tmpcorner = corners[i,:,:]
    for j in tmpcorner:
        x,y = np.floor(j.ravel()).astype(int)
        tmp = cv2.circle(tmp,(x,y),10,(0,255,0),5)

    tmp1 = np.vstack((tmpcorner.T,np.ones((1,4))))
    tmp2 = np.array(np.linalg.lstsq(K,tmp1)[0])
    p = np.matmul(pr,tmp2).T
    H = est_homography(corner_pts,p)

    proj_pts,tmppos,tmprot = ar_cube(H,render_pts,K)
    pos.append(tmppos)
    rot.append(tmprot)

    frame = .05*np.array([[0,0,0],[1,0,0],[0,1,0],[0,0,1]])
    frame = np.matmul(np.matmul(frame,rot[i].T)+pos[i].T,K.T)
    frame[:,0] = frame[:,0]/frame[:,2]
    frame[:,1] = frame[:,1]/frame[:,2]
    frame = frame[:,0:2]

    tmp = cv2.line(tmp,tuple(frame[0,:].astype(int)),
                   tuple(frame[1,:].astype(int)),(255,0,0),5)
    tmp = cv2.line(tmp,tuple(frame[0,:].astype(int)),
                   tuple(frame[2,:].astype(int)),(0,255,0),5)
    tmp = cv2.line(tmp,tuple(frame[0,:].astype(int)),
                   tuple(frame[3,:].astype(int)),(0,100,255),5)

    tmp = draw_ar_cube(proj_pts,tmp)

    ax.clear()
    ax.imshow(tmp)
    fig.canvas.draw()
```

    <IPython.core.display.Javascript object>

    C:\Users\44754\Anaconda3\lib\site-packages\ipykernel_launcher.py:18: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
    To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.

```python

```

Computer Vision/3D object projection/3D Homography.ipynb

Perception/3D object projection/3D Homography.ipynb