 # Modelling part 3 (Week 8)

Code at: https://github.com/markbriers/data-science-jupyter

## Recap - the story so far

Our goal is to follow a data science process (CRISP-DM) to deliver a successful business outcome.

Our learning objectives are as follows:

* Describe the six stages of a data processing pipeline (using CRISP-DM)

* Demonstrate an understanding of the python programming language through the production of elementary data analysis programme

* Analyse at least three different data sources by applying at least one python data processing library to extract and explore pertinent features

* Be able to design a set of data requirements for a specified business problem

* Describe and apply (using the python programming language) the main approaches to supervised learning for a given classification problem

* Understand the use cases of Big Data technology (in particular Spark)

* Produce a report including appropriate data visualisations covering the analysis of a business problem using a data science based approach

## Learning outcome

* By the end of the lecture, you will be able to use the tensorflow library to estimate parameters of a binary classification model.

* By the end of the lecture, you will be able to use the tensorflow library to estimate parameters of a deep neural network.

* By the end of the lecture, you will have a high level understanding of Generative Adversarial Networks (GANs), and how they can be used to generate "fake news".

# Bernoulli distribution and coin tossing

Let's compute the following probabilities:
* Tossing T with a fair coin
* Tossing HH with a fair coin
* Tossing HT with a fair coin
* Tossing HTH with a fair coin
* Tossing H with a biased coin ($p(head) = \lambda = 0.1$)
* Tossing HT with a biased coin ($p(head) = \lambda = 0.1$)

Now we'll (informally) introduce the concept of a random variable, denoted $C$, where $C=0$ corresponds to coin landing on heads and $C=1$ corresponds to coin landing on tails. The probability of $C=1$ is equal to $\lambda$. If $\lambda=0.5$ then the coin is unbiased (and we're maximally uncertain about the outcome of a coin toss.

We can say that $C$ follows a Bernoulli distribution, with *probability mass function* equal to the following expression:
\begin{equation}
p(C|\lambda) = \lambda^{C}(1-\lambda)^{1-C}
\end{equation}

## Logistic regression - binary classification (supervised learning)

Consider the scenario where we wish to partition objects into two categories (e.g. spam or not spam, fraud or not fraud, positive sentiment or negative sentiment, face present or face not present, ...) given a set of attributes. The simplest (and widest used) model to achieve this is known as a Logistic Regression model. The Logistic Regression model "squashes" the output of a linear regression model, so that the resultant squashed output is between $0$ and $1$. This "squashed" output can be interpreted as a class probabaility (e.g. probability of spam or not spam). So what is the squashing function?

### Sigmoid function

$sigm(\eta)$ is known as the sigmoid function (also known as the _logistic_ or _logit_ function):

\begin{equation}
sigm(\eta) = \frac{1}{1+e^{-\eta}} = \frac{e^{\eta}}{e^{\eta}+1}
\end{equation}

This function has the property that is "squashes" any input (from $-\infty < \eta < \infty$) to a value between 0 and 1.

We can use the Bernouilli expression introduced above to model input data and (probabilistically) ascertain whether the data are from one class or another:

\begin{equation}
\prod_{i=1}^{N} Ber(C_{i}|sigm(\theta x_{i})) = \prod_{i=1}^{N} \left[\frac{1}{1+e^{-\theta x_{i}}} \right]^{C_{i}} \left[1-\frac{1}{1+e^{-\theta x_{i}}} \right]^{1-C_{i}}
\end{equation}

So how do we estimate model parameters $\theta$? What is the loss function?

### Cross-entropy loss

Let's introduce entropy: https://en.wikipedia.org/wiki/Entropy_(information_theory). Entropy can be considered to be a measure of uncertainty. For example, in the case of coin tossing, the value oof $\lambda$ that maximises entropy is $\lambda = 0.5$ - we are maximally uncertain about the outcome of the "experiment". We therefore want to chose a value of $\theta$ (in the above model) that minimises *cross entropy loss*:
\begin{equation}
c(\theta) = -\frac{1}{N}\sum_{i=1}^{N} \left[C_{i}log\hat{C}_{i}+(1-C_{i})log(1-\hat{C}_{i})\right]
\end{equation}

where:
\begin{equation}
\hat{C}_{i} = \frac{1}{1+e^{-\theta x_{i}}}
\end{equation}

In order to compute the optimal value of $\theta$ that minimises the cross entropy loss, we use the chain rule (differentiation of composed functions) and gradient descent.

### Tensorflow implementation

```
import pandas as pd
import numpy as np
import seaborn as sns
import tensorflow as tf
import matplotlib.pyplot as plt
%matplotlib inline
```

The dataset basically consists of 7 columns:

* price (the buying price of the car)
* maint (the maintenance cost)
* doors (number of doors)
* persons (the seating capacity)
* lug_capacity (the luggage capacity)
* safety (how safe is the car)
* output (the condition of the car)

Given the first 6 columns, the task is to predict the value for the 7th column i.e. the output column.

```
cols = ['price', 'maint', 'doors', 'persons', 'lug_capacity', 'safety','output']
cars = pd.read_csv('https://raw.githubusercontent.com/vortexeye/Car-Evaluation/master/car_evaluation.csv', names=cols, header=None)
cars.head()
```

      price
      maint
      doors
      persons
      lug_capacity
      safety
      output

      0
      vhigh
      vhigh
      2
      2
      small
      low
      unacc

      1
      vhigh
      vhigh
      2
      2
      small
      med
      unacc

      2
      vhigh
      vhigh
      2
      2
      small
      high
      unacc

      3
      vhigh
      vhigh
      2
      2
      med
      low
      unacc

      4
      vhigh
      vhigh
      2
      2
      med
      med
      unacc

### Brief exploratory analysis

Let's perform a brief exploratory analysis of the data:

```
cars.output.value_counts().plot(kind='pie', autopct='%0.05f%%', colors=['lightblue', 'lightgreen', 'orange', 'pink'], explode=(0.05, 0.05, 0.05,0.05))
```

Let's convert the categorical values into numerical values (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.get_dummies.html?highlight=get_dummies#pandas.get_dummies):

```
price = pd.get_dummies(cars.price, prefix='price') #The get_dummies() function is used to convert categorical variable into dummy/indicator variables
maint = pd.get_dummies(cars.maint, prefix='maint')

doors = pd.get_dummies(cars.doors, prefix='doors')
persons = pd.get_dummies(cars.persons, prefix='persons')

lug_capacity = pd.get_dummies(cars.lug_capacity, prefix='lug_capacity')
safety = pd.get_dummies(cars.safety, prefix='safety')

labels = pd.get_dummies(cars.output, prefix='condition')
```

Now concatonate the two data frames (https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.concat.html):

```
X = pd.concat([price, maint, doors, persons, lug_capacity, safety] , axis=1)
X.head()
```

      price_high
      price_low
      price_med
      price_vhigh
      maint_high
      maint_low
      maint_med
      maint_vhigh
      doors_2
      doors_3
      doors_4
      doors_5more
      persons_2
      persons_4
      persons_more
      lug_capacity_big
      lug_capacity_med
      lug_capacity_small
      safety_high
      safety_low
      safety_med

      0
      0
      0
      0
      1
      0
      0
      0
      1
      1
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0
      1
      0

      1
      0
      0
      0
      1
      0
      0
      0
      1
      1
      0
      0
      0
      1
      0
      0
      0
      0
      1
      0
      0
      1

      2
      0
      0
      0
      1
      0
      0
      0
      1
      1
      0
      0
      0
      1
      0
      0
      0
      0
      1
      1
      0
      0

      3
      0
      0
      0
      1
      0
      0
      0
      1
      1
      0
      0
      0
      1
      0
      0
      0
      1
      0
      0
      1
      0

      4
      0
      0
      0
      1
      0
      0
      0
      1
      1
      0
      0
      0
      1
      0
      0
      0
      1
      0
      0
      0
      1

The labels dataframe corresponds to the output category (i.e. the condition of the vehicle):

```
labels.head()
```

      condition_acc
      condition_good
      condition_unacc
      condition_vgood

      0
      0
      0
      1
      0

      1
      0
      0
      1
      0

      2
      0
      0
      1
      0

      3
      0
      0
      1
      0

      4
      0
      0
      1
      0

```
y = labels.values
```

We'll randomly split the data into training and test data:

```
from sklearn.model_selection import train_test_split
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
```

And create the model as layers in tensorflow:

```
from tensorflow.keras.layers import Input, Dense, Activation,Dropout
from tensorflow.keras.models import Model
```

```
input_layer = Input(shape=(X.shape[1],))
output = Dense(y.shape[1], activation='softmax')(input_layer)

model = Model(inputs=input_layer, outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='sgd', metrics=['acc'])
```

The softmax activation function acts as a generalisation of the sigmoid function in logistic regression, to allow for multi-class classification problems (i.e. it extends binary classification) - see https://www.kdnuggets.com/2016/07/softmax-regression-related-logistic-regression.html

```
print(model.summary())
```

    Model: "model_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_2 (InputLayer)         [(None, 21)]              0
    _________________________________________________________________
    dense_1 (Dense)              (None, 4)                 88
    =================================================================
    Total params: 88
    Trainable params: 88
    Non-trainable params: 0
    _________________________________________________________________
    None

Now we can fit the model (i.e. estimate the parameters):

```
history = model.fit(X_train, y_train, batch_size=8, epochs=50, verbose=1, validation_split=0.2)
```

    Epoch 1/50
    139/139 [==============================] - 1s 3ms/step - loss: 1.3379 - acc: 0.3924 - val_loss: 0.9411 - val_acc: 0.6570
    Epoch 2/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.8254 - acc: 0.7237 - val_loss: 0.8297 - val_acc: 0.6787
    Epoch 3/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.7189 - acc: 0.7477 - val_loss: 0.7780 - val_acc: 0.6679
    Epoch 4/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.6592 - acc: 0.7414 - val_loss: 0.7412 - val_acc: 0.6751
    Epoch 5/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.6408 - acc: 0.7369 - val_loss: 0.7131 - val_acc: 0.6823
    Epoch 6/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.5840 - acc: 0.7513 - val_loss: 0.6889 - val_acc: 0.6968
    Epoch 7/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.5348 - acc: 0.7739 - val_loss: 0.6688 - val_acc: 0.7148
    Epoch 8/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.5326 - acc: 0.7701 - val_loss: 0.6522 - val_acc: 0.7148
    Epoch 9/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.5369 - acc: 0.7852 - val_loss: 0.6374 - val_acc: 0.7473
    Epoch 10/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.5074 - acc: 0.8055 - val_loss: 0.6239 - val_acc: 0.7653
    Epoch 11/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4957 - acc: 0.8323 - val_loss: 0.6116 - val_acc: 0.7762
    Epoch 12/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.5143 - acc: 0.8206 - val_loss: 0.6013 - val_acc: 0.7690
    Epoch 13/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4824 - acc: 0.8334 - val_loss: 0.5906 - val_acc: 0.7834
    Epoch 14/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4975 - acc: 0.8379 - val_loss: 0.5816 - val_acc: 0.7798
    Epoch 15/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4663 - acc: 0.8378 - val_loss: 0.5730 - val_acc: 0.7870
    Epoch 16/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4630 - acc: 0.8441 - val_loss: 0.5646 - val_acc: 0.7834
    Epoch 17/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4571 - acc: 0.8408 - val_loss: 0.5565 - val_acc: 0.7978
    Epoch 18/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4635 - acc: 0.8463 - val_loss: 0.5499 - val_acc: 0.7906
    Epoch 19/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4651 - acc: 0.8402 - val_loss: 0.5426 - val_acc: 0.7978
    Epoch 20/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4458 - acc: 0.8494 - val_loss: 0.5359 - val_acc: 0.8123
    Epoch 21/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4172 - acc: 0.8541 - val_loss: 0.5299 - val_acc: 0.8087
    Epoch 22/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4007 - acc: 0.8731 - val_loss: 0.5239 - val_acc: 0.8087
    Epoch 23/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4103 - acc: 0.8645 - val_loss: 0.5179 - val_acc: 0.8051
    Epoch 24/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4081 - acc: 0.8641 - val_loss: 0.5129 - val_acc: 0.8051
    Epoch 25/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4377 - acc: 0.8400 - val_loss: 0.5079 - val_acc: 0.8195
    Epoch 26/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.4026 - acc: 0.8714 - val_loss: 0.5031 - val_acc: 0.8231
    Epoch 27/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3976 - acc: 0.8573 - val_loss: 0.4984 - val_acc: 0.8231
    Epoch 28/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3931 - acc: 0.8590 - val_loss: 0.4942 - val_acc: 0.8231
    Epoch 29/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3793 - acc: 0.8741 - val_loss: 0.4900 - val_acc: 0.8231
    Epoch 30/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4017 - acc: 0.8641 - val_loss: 0.4862 - val_acc: 0.8231
    Epoch 31/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.4072 - acc: 0.8444 - val_loss: 0.4821 - val_acc: 0.8231
    Epoch 32/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3736 - acc: 0.8687 - val_loss: 0.4787 - val_acc: 0.8231
    Epoch 33/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3768 - acc: 0.8661 - val_loss: 0.4751 - val_acc: 0.8231
    Epoch 34/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3859 - acc: 0.8510 - val_loss: 0.4717 - val_acc: 0.8231
    Epoch 35/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3722 - acc: 0.8576 - val_loss: 0.4682 - val_acc: 0.8231
    Epoch 36/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3625 - acc: 0.8643 - val_loss: 0.4650 - val_acc: 0.8267
    Epoch 37/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3729 - acc: 0.8656 - val_loss: 0.4620 - val_acc: 0.8231
    Epoch 38/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3599 - acc: 0.8600 - val_loss: 0.4591 - val_acc: 0.8231
    Epoch 39/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3642 - acc: 0.8593 - val_loss: 0.4562 - val_acc: 0.8267
    Epoch 40/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3638 - acc: 0.8632 - val_loss: 0.4535 - val_acc: 0.8267
    Epoch 41/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3685 - acc: 0.8575 - val_loss: 0.4508 - val_acc: 0.8267
    Epoch 42/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3706 - acc: 0.8713 - val_loss: 0.4482 - val_acc: 0.8267
    Epoch 43/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3434 - acc: 0.8689 - val_loss: 0.4458 - val_acc: 0.8267
    Epoch 44/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3681 - acc: 0.8437 - val_loss: 0.4433 - val_acc: 0.8267
    Epoch 45/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3482 - acc: 0.8527 - val_loss: 0.4410 - val_acc: 0.8267
    Epoch 46/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3564 - acc: 0.8537 - val_loss: 0.4387 - val_acc: 0.8267
    Epoch 47/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3361 - acc: 0.8738 - val_loss: 0.4364 - val_acc: 0.8303
    Epoch 48/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3389 - acc: 0.8678 - val_loss: 0.4342 - val_acc: 0.8303
    Epoch 49/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.3428 - acc: 0.8565 - val_loss: 0.4325 - val_acc: 0.8303
    Epoch 50/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3503 - acc: 0.8568 - val_loss: 0.4302 - val_acc: 0.8303

And evaluate how well the model has performed on the held-out test set:

```
score = model.evaluate(X_test, y_test, verbose=1)

print("Test Score:", score[0])
print("Test Accuracy:", score[1])
```

    11/11 [==============================] - 0s 1ms/step - loss: 0.3828 - acc: 0.8497
    Test Score: 0.38277667760849
    Test Accuracy: 0.849711000919342

## Deep Neural Network

Finally, we can make our model "Deep" (i.e. we can create a deep neural network) by adding additional layers:

```
input_layer = Input(shape=(X.shape[1],))
dense_layer_1 = Dense(15, activation='relu')(input_layer)
dense_layer_2 = Dense(10, activation='relu')(dense_layer_1)
dense_layer_3 = Dense(20, activation='relu')(dense_layer_2)
output = Dense(y.shape[1], activation='softmax')(dense_layer_3)

model = Model(inputs=input_layer, outputs=output)
model.compile(loss='categorical_crossentropy', optimizer='adam', metrics=['acc'])
```

Here we use a relu (rectified linear unit) activation function: https://www.tinymind.com/learn/terms/relu

```
print(model.summary())
```

    Model: "model_1"
    _________________________________________________________________
    Layer (type)                 Output Shape              Param #
    =================================================================
    input_2 (InputLayer)         [(None, 21)]              0
    _________________________________________________________________
    dense_1 (Dense)              (None, 15)                330
    _________________________________________________________________
    dense_2 (Dense)              (None, 10)                160
    _________________________________________________________________
    dense_3 (Dense)              (None, 20)                220
    _________________________________________________________________
    dense_4 (Dense)              (None, 4)                 84
    =================================================================
    Total params: 794
    Trainable params: 794
    Non-trainable params: 0
    _________________________________________________________________
    None

```
history = model.fit(X_train, y_train, batch_size=8, epochs=50, verbose=1, validation_split=0.2)
```

    Epoch 1/50
    139/139 [==============================] - 1s 3ms/step - loss: 1.0615 - acc: 0.7142 - val_loss: 0.8876 - val_acc: 0.6498
    Epoch 2/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.7216 - acc: 0.7251 - val_loss: 0.6928 - val_acc: 0.6498
    Epoch 3/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.5149 - acc: 0.7491 - val_loss: 0.5129 - val_acc: 0.7617
    Epoch 4/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.3965 - acc: 0.8296 - val_loss: 0.4182 - val_acc: 0.8159
    Epoch 5/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.2844 - acc: 0.8903 - val_loss: 0.3337 - val_acc: 0.8664
    Epoch 6/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.2358 - acc: 0.9124 - val_loss: 0.2880 - val_acc: 0.8773
    Epoch 7/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.2078 - acc: 0.9032 - val_loss: 0.2575 - val_acc: 0.8845
    Epoch 8/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.1677 - acc: 0.9430 - val_loss: 0.2176 - val_acc: 0.9097
    Epoch 9/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.1438 - acc: 0.9501 - val_loss: 0.2065 - val_acc: 0.9097
    Epoch 10/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.1192 - acc: 0.9670 - val_loss: 0.1767 - val_acc: 0.9495
    Epoch 11/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.1062 - acc: 0.9722 - val_loss: 0.1581 - val_acc: 0.9531
    Epoch 12/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0907 - acc: 0.9697 - val_loss: 0.1612 - val_acc: 0.9495
    Epoch 13/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0837 - acc: 0.9776 - val_loss: 0.1315 - val_acc: 0.9639
    Epoch 14/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0870 - acc: 0.9768 - val_loss: 0.1208 - val_acc: 0.9675
    Epoch 15/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0720 - acc: 0.9806 - val_loss: 0.1125 - val_acc: 0.9603
    Epoch 16/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0547 - acc: 0.9909 - val_loss: 0.1201 - val_acc: 0.9567
    Epoch 17/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0529 - acc: 0.9900 - val_loss: 0.0982 - val_acc: 0.9603
    Epoch 18/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0469 - acc: 0.9927 - val_loss: 0.0932 - val_acc: 0.9675
    Epoch 19/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0407 - acc: 0.9915 - val_loss: 0.0974 - val_acc: 0.9567
    Epoch 20/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0319 - acc: 0.9923 - val_loss: 0.0955 - val_acc: 0.9639
    Epoch 21/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0389 - acc: 0.9889 - val_loss: 0.0799 - val_acc: 0.9603
    Epoch 22/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0282 - acc: 0.9989 - val_loss: 0.0840 - val_acc: 0.9639
    Epoch 23/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0277 - acc: 0.9950 - val_loss: 0.0925 - val_acc: 0.9639
    Epoch 24/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0245 - acc: 0.9975 - val_loss: 0.0644 - val_acc: 0.9747
    Epoch 25/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0176 - acc: 0.9971 - val_loss: 0.0821 - val_acc: 0.9639
    Epoch 26/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0181 - acc: 0.9981 - val_loss: 0.0708 - val_acc: 0.9675
    Epoch 27/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0152 - acc: 0.9973 - val_loss: 0.0581 - val_acc: 0.9747
    Epoch 28/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0200 - acc: 0.9960 - val_loss: 0.0550 - val_acc: 0.9747
    Epoch 29/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0132 - acc: 0.9999 - val_loss: 0.0690 - val_acc: 0.9747
    Epoch 30/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0095 - acc: 0.9993 - val_loss: 0.0543 - val_acc: 0.9747
    Epoch 31/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0107 - acc: 1.0000 - val_loss: 0.0572 - val_acc: 0.9711
    Epoch 32/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0078 - acc: 0.9997 - val_loss: 0.0605 - val_acc: 0.9747
    Epoch 33/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0114 - acc: 0.9983 - val_loss: 0.0460 - val_acc: 0.9856
    Epoch 34/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0077 - acc: 1.0000 - val_loss: 0.0451 - val_acc: 0.9747
    Epoch 35/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0069 - acc: 1.0000 - val_loss: 0.0508 - val_acc: 0.9711
    Epoch 36/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0042 - acc: 1.0000 - val_loss: 0.0565 - val_acc: 0.9711
    Epoch 37/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0050 - acc: 1.0000 - val_loss: 0.0678 - val_acc: 0.9747
    Epoch 38/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0044 - acc: 1.0000 - val_loss: 0.0437 - val_acc: 0.9819
    Epoch 39/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0038 - acc: 1.0000 - val_loss: 0.0479 - val_acc: 0.9783
    Epoch 40/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0035 - acc: 1.0000 - val_loss: 0.0455 - val_acc: 0.9819
    Epoch 41/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0030 - acc: 1.0000 - val_loss: 0.0508 - val_acc: 0.9819
    Epoch 42/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0035 - acc: 1.0000 - val_loss: 0.0431 - val_acc: 0.9783
    Epoch 43/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0025 - acc: 1.0000 - val_loss: 0.0480 - val_acc: 0.9783
    Epoch 44/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0023 - acc: 1.0000 - val_loss: 0.0452 - val_acc: 0.9819
    Epoch 45/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0022 - acc: 1.0000 - val_loss: 0.0390 - val_acc: 0.9819
    Epoch 46/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0017 - acc: 1.0000 - val_loss: 0.0372 - val_acc: 0.9819
    Epoch 47/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0019 - acc: 1.0000 - val_loss: 0.0534 - val_acc: 0.9711
    Epoch 48/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0019 - acc: 1.0000 - val_loss: 0.0431 - val_acc: 0.9783
    Epoch 49/50
    139/139 [==============================] - 0s 2ms/step - loss: 0.0016 - acc: 1.0000 - val_loss: 0.0284 - val_acc: 0.9856
    Epoch 50/50
    139/139 [==============================] - 0s 1ms/step - loss: 0.0015 - acc: 1.0000 - val_loss: 0.0329 - val_acc: 0.9856

Notice the difference (improvement) in classification performance when we added more layers into the model:

```
score = model.evaluate(X_test, y_test, verbose=1)

print("Test Score:", score[0])
print("Test Accuracy:", score[1])
```

    11/11 [==============================] - 0s 1ms/step - loss: 0.0248 - acc: 0.9942
    Test Score: 0.02476133219897747
    Test Accuracy: 0.9942196607589722

The art and science of deep neural networks involves designing appropriately large networks (with billions of parameters), defining the "activation" functions, and using specific "bells and whistles" to optimise performance. A further analysis of deep neural networks is beyond the scope of this module. The interested reader is referred to https://www.coursera.org/learn/neural-networks-deep-learning for an in-depth exposition.

## Generative Adversarial Networks

The user of Generative Adversarial Networks is becoming widespread, underpinning lots of misinformation on the internet. The ability to use a pair of deep neural networks in order to produce realistic, but deceptive, information, is likely to be societally disruptive in the coming years. A recent example of a "Deepfake" is in the video below (is this real or fake?!):

```
from IPython.display import YouTubeVideo
YouTubeVideo('iyiOVUbsPcM')
```

Here's another example, this time in text generation: https://deepai.org/machine-learning-model/text-generator

A GAN uses two deep neural networks in order to build up a statistical representation of a data set, which can be used to generate synthetic data. A detailed methodological review of GANs is beyond the scope of this module. However, those interested in the detail should read: https://machinelearningmastery.com/what-are-generative-adversarial-networks-gans/ for example.

***
Mandatory exercises:

- [ ] Carefully read the material above, and ask questions over email.
- [ ] Replicate all of the code in this notebook.


```python
%load_ext autoreload
%autoreload 2

import os
os.chdir("..")
```

```python
import particlezoo as zoo
import liesym as ls
```

```python
p = zoo.Field("psi", "1/2", {"U(1)_x": "q"})
p.conjugate_name
```

$\displaystyle \overline{\psi}$

```python
s = zoo.Symmetry("U(1)_x", ls.U1(), True, "g", tag="x")
s
```

$\displaystyle U(1)_{x}$

```python
l = zoo.Lagrangian([p], [s], "test")
```

```python
l._get_sym("U(1)_x")
```

$\displaystyle U(1)_{x}$

```python
from particlezoo import Field, Symmetry
from tikzfeynwrap import TikzFeynWrap
from typing import NamedTuple
from sympy import latex
```

```python
def id_gen():
    import string
    i = -1
    def _get_id():
        nonlocal i
        i += 1
        return "a" + str(i)
    return _get_id

class Node2(NamedTuple):
    name: str
    connectedTo: list[tuple]
    key: str

class Diagram(NamedTuple):
    heads: list[Node2]
    horizontal: tuple

allowed = [
            "charged scalar",
            "scalar",
            "majorana",
            "anti majorana",
            "gluon",
            "photon",
            "ghost",
            "charged boson",
            "fermion"
        ]

def _create_vertex(key, name=None):
    if name:
        return f"{key} [particle=\({name}\)]"
    return key

def _create_line(key, line_type=None):
    if line_type:
        if line_type in allowed:
            return f"-- [{line_type}]"
        else:
            raise ValueError(f"Bad tikz-feynman option: {line_type}")
    return f"--"

def fermion_kinetic_energy(field: Field, sym: Symmetry):
    id_ = id_gen()

    fname = latex(field.name)
    fname_conj = latex(field.conjugate_name)
    gauge_boson = latex(sym.gauge_name)

    gbout = Node2(gauge_boson, None, id_())
    conjout = Node2(fname_conj, None, id_())
    interaction = Node2(None,[
        (gbout, 'fermion', id_()),
        (conjout, 'photon', id_())
    ], id_())
    pin = Node2(fname, [(interaction, 'fermion', id_())], id_())
    # return Diagram([pin], (interaction, gbout))
    return pin

ex = fermion_kinetic_energy(p, s)

print(ex)

total = []

# def parse_path(n: Node2, log: list):
import copy

def parse_heads(head: Node2, l: list):
    vertex = _create_vertex(head.key, head.name)
    l.append(vertex)

    if head.connectedTo is None:
        total.append(copy.deepcopy(l))
        l.clear()
        return

    for node, name, key in head.connectedTo:
        path = _create_line(key, name)
        l.append(path)
        parse_heads(node, copy.deepcopy(l))

l = []
parse_heads(ex, l)
total

```

    Node2(name='\\psi', connectedTo=[(Node2(name=None, connectedTo=[(Node2(name='\\mathtt{\\text{A\\_x}}', connectedTo=None, key='a0'), 'fermion', 'a2'), (Node2(name='\\overline{\\psi}', connectedTo=None, key='a1'), 'photon', 'a3')], key='a4'), 'fermion', 'a5')], key='a6')

    [['a6 [particle=\\(\\psi\\)]',
      '-- [fermion]',
      'a4',
      '-- [fermion]',
      'a0 [particle=\\(\\mathtt{\\text{A\\_x}}\\)]'],
     ['a6 [particle=\\(\\psi\\)]',
      '-- [fermion]',
      'a4',
      '-- [fermion]',
      '-- [photon]',
      'a1 [particle=\\(\\overline{\\psi}\\)]']]

```python
# wrapper = TikzFeynWrap().startup()
```

```python
def dec(x):
    return f"""\\feynmandiagram [horizontal=a6 to a0] {{
        {x}
        }};"""

t = dec(" ".join(total[0]))
print(t)
x = wrapper(t, return_path=True)
```

    \feynmandiagram [horizontal=a6 to a0] {
            a6 [particle=\(\psi\)] -- [fermion] a4 -- [fermion] a0 [particle=\(\mathtt{\text{A\_x}}\)]
            };

```python
a,r = wrapper(r"""
\feynmandiagram [horizontal=a to b] {
        i1 [particle=\(e^{-}\)] -- [fermion] a -- [fermion] i2 [particle=\(e^{+}\)],
        a -- [photon, edge label=\(\gamma\), momentum'=\(k\)] b,
        f1 [particle=\(\mu^{+}\)] -- [fermion] b -- [fermion] f2 [particle=\(\mu^{-}\)],

        };
""", return_path=True, return_output=True)
```

```python
from IPython.display import SVG, display

display(SVG(a))
```

```python
print(str(r))
```

    ExecResult(exit_code=0, output=b'This is LuaHBTeX, Version 1.13.2 (TeX Live 2021) \n restricted system commands enabled.\n(./output.tex\nLaTeX2e <2021-06-01> patch level 1\n L3 programming layer <2021-08-27>\n(/usr/local/texlive/2021/texmf-dist/tex/latex/standalone/standalone.cls\nDocument Class: standalone 2018/03/26 v1.3a Class to compile TeX sub-files stan\ndalone\n(/usr/local/texlive/2021/texmf-dist/tex/latex/tools/shellesc.sty)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/iftex/ifluatex.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/iftex/iftex.sty))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/xkeyval/xkeyval.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/xkeyval/xkeyval.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/xkeyval/xkvutils.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/xkeyval/keyval.tex))))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/standalone/standalone.cfg)\n(/usr/local/texlive/2021/texmf-dist/tex/latex/base/article.cls\nDocument Class: article 2021/02/12 v1.4n Standard LaTeX document class\n(/usr/local/texlive/2021/texmf-dist/tex/latex/base/size10.clo))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/luatex85/luatex85.sty)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pdftexcmds/pdftexcmds.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/infwarerr/infwarerr.sty)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/ltxcmds/ltxcmds.sty))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/frontendlayer/tikz.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/basiclayer/pgf.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/utilities/pgfrcs.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfutil-common.te\nx\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfutil-common-li\nsts.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfutil-latex.def\n) (/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfrcs.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/pgf.revision.tex)))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/basiclayer/pgfcore.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/graphics/graphicx.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/graphics/graphics.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/graphics/trig.sty)\n(/usr/local/texlive/2021/texmf-dist/tex/latex/graphics-cfg/graphics.cfg)\n(/usr/local/texlive/2021/texmf-dist/tex/latex/graphics-def/luatex.def)))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/systemlayer/pgfsys.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/systemlayer/pgfsys.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfkeysfiltered.c\node.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/systemlayer/pgf.cfg)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/systemlayer/pgfsys-luatex.d\nef\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/systemlayer/pgfsys-common-p\ndf.def)))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/systemlayer/pgfsyssoftpath.\ncode.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/systemlayer/pgfsysprotocol.\ncode.tex)) (/usr/local/texlive/2021/texmf-dist/tex/latex/xcolor/xcolor.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/graphics-cfg/color.cfg))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcore.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathcalc.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathutil.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathparser.code.tex\n)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.code.\ntex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.basic\n.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.trigo\nnometric.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.rando\nm.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.compa\nrison.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.base.\ncode.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.round\n.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.misc.\ncode.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfunctions.integ\nerarithmetics.code.tex)))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmathfloat.code.tex)\n) (/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfint.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepoints.co\nde.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathconst\nruct.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathusage\n.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorescopes.co\nde.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoregraphicst\nate.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoretransform\nations.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorequick.cod\ne.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreobjects.c\node.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepathproce\nssing.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorearrows.co\nde.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreshade.cod\ne.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreimage.cod\ne.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoreexternal.\ncode.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorelayers.co\nde.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcoretranspare\nncy.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorepatterns.\ncode.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/basiclayer/pgfcorerdf.code.\ntex)))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/modules/pgfmoduleshapes.cod\ne.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/modules/pgfmoduleplot.code.\ntex)\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/compatibility/pgfcomp-version\n-0-65.sty)\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/compatibility/pgfcomp-version\n-1-18.sty))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/utilities/pgffor.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/utilities/pgfkeys.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgfkeys.code.tex)\n) (/usr/local/texlive/2021/texmf-dist/tex/latex/pgf/math/pgfmath.sty\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/utilities/pgffor.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/math/pgfmath.code.tex)))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/tikz.cod\ne.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/pgflibraryplothan\ndlers.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/modules/pgfmodulematrix.cod\ne.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarytopaths.code.tex))))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/tikz-feynman/tikz-feynman.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/pgfopts/pgfopts.sty)\n(/usr/local/texlive/2021/texmf-dist/tex/latex/tikz-feynman/tikzlibraryfeynman.c\node.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/pgflibraryarrows.\nmeta.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarycalc.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarydecorations.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/modules/pgfmoduledecoration\ns.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarydecorations.markings.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/decorations/pgfli\nbrarydecorations.markings.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarydecorations.pathmorphing.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/decorations/pgfli\nbrarydecorations.pathmorphing.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarydecorations.pathreplacing.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/decorations/pgfli\nbrarydecorations.pathreplacing.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/graphs/tikzlibrarygraphs.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarypatterns.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/pgflibrarypattern\ns.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibrarypositioning.code.tex)\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/frontendlayer/tikz/librarie\ns/tikzlibraryshapes.geometric.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/libraries/shapes/pgflibrary\nshapes.geometric.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/graphdrawing/tex/tikzlibrar\nygraphdrawing.code.tex\n(/usr/local/texlive/2021/texmf-dist/tex/generic/pgf/graphdrawing/tex/pgflibrary\ngraphdrawing.code.tex))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/tikz-feynman/tikzfeynman.keys.cod\ne.tex)))\n(/usr/local/texlive/2021/texmf-dist/tex/latex/l3backend/l3backend-luatex.def)\n(./output.aux) (/usr/local/texlive/2021/texmf-dist/tex/latex/base/ts1cmr.fd)\n(/usr/local/texlive/2021/texmf-dist/tex/context/base/mkii/supp-pdf.mkii\n[Loading MPS to PDF converter (version 2006.09.02).]\n) (/usr/local/texlive/2021/texmf-dist/tex/latex/epstopdf-pkg/epstopdf-base.sty\n(/usr/local/texlive/2021/texmf-dist/tex/latex/latexconfig/epstopdf-sys.cfg))\n\nPackage tikz-feynman Warning: Consider loading TikZ-Feynman with \\usepackage[co\nmpat=1.1.0]{tikz-feynman} so that you can be warned if TikZ-Feynman changes. on\n input line 3.\n\n[1] [2{/usr/local/texlive/2021/texmf-var/fonts/map/pdftex/updmap/pdftex.map}]\n(./output.aux))\n(see the transcript file for additional information)\n 413 words of node memory still in use:\n   3 hlist, 1 vlist, 1 rule, 2 glue, 3 kern, 1 glyph, 5 attribute, 48 glue_spec\n, 5 attribute_list, 2 write nodes\n   avail lists: 1:9,2:70,3:25,4:20,5:27,6:2,7:416,8:14,9:42\n</usr/local/texlive/2021/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi10.pfb><\n/usr/local/texlive/2021/texmf-dist/fonts/type1/public/amsfonts/cm/cmmi7.pfb></u\nsr/local/texlive/2021/texmf-dist/fonts/type1/public/amsfonts/cm/cmr10.pfb>\nOutput written on output.pdf (2 pages, 25543 bytes).\nTranscript written on output.log.\n')


# Neuromatch Academy: Week 1, Day 5, Tutorial 1
# Regularization

__Content creators:__ Lyle Ungar, Ravi Teja Konkimalla, Mohitrajhu Lingan Kumaraian, Kevin Machado Gamboa, Kelson Shilling-Scrivo

__Content reviewers:__ Piyush Chauhan, Kelson Shilling-Scrivo

__Content editors:__ Roberto Guidotti

__Production editors:__ Saeed Salehi, Spiros Chavlis

---
# Tutorial Objectives

1.   **Big ANNs are efficient universal approximators due to adaptive basis functions**
2.   **ANN’s memorize some but generalize well**
3.   Regularization as shrinkage of overparameterized models: L1, L2, early stopping
4.   Regularization by Dropout
5. Regularization by Data Augmentation
6. Perils of Hyper-Parameter Tuning
7.   Rethinking generalization

```
#@title Video : Introduction to Regularization
import time
from IPython.display import YouTubeVideo

try: t1;
except NameError: t1=time.time()

video = YouTubeVideo(id="bc1nsP4htVg", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=bc1nsP4htVg

A key idea of neural nets, is that they use models that are "too complex" - complex enough to fit all the noise in the data. One then needs to "regularize" them to make the models fit complex enough, but not too complex. The more complex the model, the better it fits the training data, but if it is too complex, it generalizes less well; it memorizes the training data but is less accurate on future test data.

```
#@markdown Tutorial slides

#@markdown **Do not read them now.**
# you should link the slides for all tutorial videos here (we will store pdfs on osf)

from IPython.display import HTML
HTML('')
```

---
# Setup
Note that some of the code for today can take up to an hour to run. We have therefore "hidden" that code and shown the resulting outputs.

**Ensure you're running a GPU notebook:**
From "Runtime" in the drop-down menu above, click "Change runtime type". Ensure that "Hardware Accelerator" says "GPU".

**Ensure you can save:** From "File", click "Save a copy in Drive"

```
#@title Imports
# Imports
from __future__ import print_function
import torch
import pathlib
import random
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
from torchvision import datasets, transforms
from torchvision.datasets import ImageFolder
from torch.utils.data import DataLoader, TensorDataset
import torch.nn.utils.prune as prune
from torch.optim.lr_scheduler import StepLR
import time
import numpy as np
import matplotlib.pyplot as plt
import matplotlib.animation as animation
import copy
from tqdm.auto import tqdm
from IPython.display import HTML, display
```

```
# @title Figure Settings
import ipywidgets as widgets

%matplotlib inline
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```
# @title Loading Animal Faces data
%%capture
!rm -r AnimalFaces32x32/
!git clone https://github.com/arashash/AnimalFaces32x32
!rm -r afhq/
!unzip ./AnimalFaces32x32/afhq_32x32.zip
```

```
# @title Loading Animal Faces Randomized data
%%capture
!rm -r Animal_faces_random/
!git clone https://github.com/Ravi3191/Animal_faces_random.git
!rm -r afhq_random_32x32/
!unzip ./Animal_faces_random/afhq_random_32x32.zip
!rm -r afhq_10_32x32/
!unzip ./Animal_faces_random/afhq_10_32x32.zip
```

```
#@title Seeding for Reproducibility
seed = 90108
random.seed(seed)
np.random.seed(seed)
torch.manual_seed(seed)
torch.cuda.manual_seed(seed)
torch.cuda.manual_seed_all(seed)
torch.backends.cudnn.deterministic = True
torch.backends.cudnn.benchmark = False
torch.backends.cudnn.enabled = False
# torch.torch.use_deterministic_algorithms(True)
def seed_worker(worker_id):
  worker_seed = seed % (worker_id+1)
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```
# @title Plotting functions
def imshow(img):
    img = img / 2 + 0.5     # unnormalize
    npimg = img.numpy()
    plt.imshow(np.transpose(npimg, (1, 2, 0)))
    plt.axis(False)
    plt.show()

def plot_weights(norm, labels, ws, title='Weight Size Measurement'):

  plt.figure(figsize=[8,6])
  plt.title(title)
  plt.ylabel('Frobenius Norm Value')
  plt.xlabel('Model Layers')
  plt.bar(labels, ws)
  plt.axhline(y=norm, linewidth=1, color='r', ls='--', label='Total Model F-Norm')
  plt.legend()
  plt.show()
```

Now, lets define a Animal Net model, train, test and main functions which we will use frequently today.

```
##Network Class - Animal Faces
class Animal_Net(nn.Module):
  def __init__(self):
    torch.manual_seed(104)
    super(Animal_Net, self).__init__()
    self.fc1 = nn.Linear(3*32*32, 128)
    self.fc2 = nn.Linear(128, 32)
    self.fc3 = nn.Linear(32, 3)

  def forward(self, x):
    x = x.view(x.shape[0],-1)
    x = F.relu(self.fc1(x))
    x = F.relu(self.fc2(x))
    x = self.fc3(x)
    output = F.log_softmax(x, dim=1)
    return output
```

The train function takes in the current model along with the train_loader and loss function and updates the parameters for a single pass of the entire dataset. The test function takes in the current model after every epoch and calculates the accuracy on the test dataset.

```
def train(args, model, device, train_loader, optimizer, epoch,
          reg_function1=None, reg_function2=None, criterion=F.nll_loss):
  """
  Trains the current inpur model using the data
  from Train_loader and Updates parameters for a single pass
  """
  model.train()
  for batch_idx, (data, target) in enumerate(train_loader):
    data, target = data.to(device), target.to(device)
    optimizer.zero_grad()
    output = model(data)
    if reg_function1 is None:
      loss = criterion(output, target)
    elif reg_function2 is None:
      loss = criterion(output, target)+args['lambda']*reg_function1(model)
    else:
      loss = criterion(output, target)+args['lambda1']*reg_function1(model)+args['lambda2']*reg_function2(model)
    loss.backward()
    optimizer.step()

def test(model, device, test_loader, loader = 'Test', criterion=F.nll_loss):
  """
  Tests the current Model
  """
  model.eval()
  test_loss = 0
  correct = 0
  with torch.no_grad():
    for data, target in test_loader:
      data, target = data.to(device), target.to(device)
      output = model(data)
      test_loss += criterion(output, target, reduction='sum').item()  # sum up batch loss
      pred = output.argmax(dim=1, keepdim=True)  # get the index of the max log-probability
      correct += pred.eq(target.view_as(pred)).sum().item()

  test_loss /= len(test_loader.dataset)
  return 100. * correct / len(test_loader.dataset)

def main(args, model, train_loader, val_loader, test_data, reg_function1=None,
         reg_function2=None, criterion=F.nll_loss):
  """
  Trains the model with train_loader and tests the learned model using val_loader
  """

  use_cuda = not args['no_cuda'] and torch.cuda.is_available()
  device = torch.device('cuda' if use_cuda else 'cpu')

  model = model.to(device)
  optimizer = optim.SGD(model.parameters(), lr=args['lr'], momentum=args['momentum'])

  val_acc_list, train_acc_list,param_norm_list = [], [], []
  for epoch in tqdm(range(args['epochs'])):
    train(args, model, device, train_loader, optimizer, epoch,reg_function1=reg_function1,reg_function2=reg_function2)
    train_acc = test(model,device,train_loader, 'Train')
    val_acc = test(model,device,val_loader, 'Val')
    param_norm = calculate_frobenius_norm(model)
    train_acc_list.append(train_acc)
    val_acc_list.append(val_acc)
    param_norm_list.append(param_norm)

  return val_acc_list, train_acc_list, param_norm_list, model, 0
```

---
#Section 1: Regularization is Shrinkage

```
#@title Video : Regularization as Shrinkage
try: t2;
except NameError: t2=time.time()

video = YouTubeVideo(id="B4CsCKViB3k", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=B4CsCKViB3k

One way to think about Regularization is to think in terms of the magnitude of the overall weights of the model. A model with big weights can fit more data perfectly. Wheras a model with smaller weights tend to underperform on the train set but can suprisingly do very well on the test set. Too small of a weights can also be as issue an it can the underfit the model.

This week we use the sum of Frobenius Norm of all the tensors in the model as a measure of the "size of the model".

##Exercise 1: Frobenius Norm
Before we start, let's define the Frobenius norm, sometimes also called the Euclidean norm of an m×n matrix $A$  as the square root of the sum of the absolute squares of its elements.
\begin{equation}
||A||_F= \sqrt{\sum_{i=1}^m\sum_{j=1}^n|a_{ij}|^2}
\end{equation}

This is just a measure of how big the matrix is, analagous to how big a vector is.

 **Hint:** Use functions model.parameters() or model.named_parameters()

```
def calculate_frobenius_norm(model):

  ####################################################################
  # Fill in all missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Define `calculate_frobenius_norm` function")
  ####################################################################

  norm = 0.0

  # Sum the square of all parameters
  for param in model.parameters():
      norm += ...

  # Take a square root of the sum of squares of all the parameters
  norm = ...
  return norm

# # uncomment to run
# net = nn.Linear(10, 1)
# print(f'Frobenius Norm of Single Linear Layer: {calculate_frobenius_norm(net)}')
```

```
# to_remove solution
def calculate_frobenius_norm(model):

  norm = 0.0

  # Sum the square of all parameters
  for param in model.parameters():
      norm += torch.sum(param**2)

  # Take a square root of the sum of squares of all the parameters
  norm = norm**0.5
  return norm

# # uncomment to run
net = nn.Linear(10, 1)
print(f'Frobenius Norm of Single Linear Layer: {calculate_frobenius_norm(net)}')
```

    Frobenius Norm of Single Linear Layer: 0.44787365198135376

Apart from calculating the weight size for an entire model, we could also determine the weight size in every layer. For this, we can modify our `calculate_frobenius_norm` function as shown below.

**Have a look how it works!!**

```
#@markdown Frobenius Norm per Layer
def calculate_frobenius_norm(model):

  # initialization of variables
  norm, ws, labels = 0.0, [], []

  # Sum all the parameters
  for name, parameters in model.named_parameters():
    p = torch.sum(parameters**2)
    norm += p

    ws.append((p**0.5).cpu().detach().numpy())
    labels.append(name)

  # Take a square root of the sum of squares of all the parameters
  norm = (norm**0.5).cpu().detach().numpy()

  return norm, ws, labels

#uncomment to run
net = nn.Linear(10,1)
norm, ws, labels = calculate_frobenius_norm(net)
print(f'Frobenius Norm of Single Linear Layer: {norm}')
# Plots the weights
plot_weights(norm, labels, ws)
```

Using the last function `calculate_frobenius_norm`, we can also obtain the Frobenius Norm per layer for a whole NN model and use the `plot_weigts` function to visualize them.

```
# Creates a new model
model = Animal_Net()

# Calculates the forbenius norm per layer
norm, ws, labels = calculate_frobenius_norm(model)
print(f'Frobenius Norm of Models weights: {norm}')

# Plots the weights
plot_weights(norm, labels, ws)
```

---
#Section 2: Overfitting

```
#@title Video : Overfitting
try: t3;
except NameError: t3=time.time()

from IPython.display import YouTubeVideo
video = YouTubeVideo(id="RlaGyRKP2nY", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=RlaGyRKP2nY

##Visualizing Overfitting

Let's create some synthetic dataset that we will use to illustrate overfitting in neural networks.

```
#creating train data
X = torch.rand((10, 1))
X.sort(dim = 0)
Y = 2*X + 2*torch.empty((X.shape[0], 1)).normal_(mean=0, std=1) #adding small error in the data

X = X.unsqueeze_(1)
Y = Y.unsqueeze_(1)

#visualizing trian data
plt.figure(figsize=(8, 6))
plt.scatter(X.numpy(),Y.numpy())
plt.xlabel('input (x)')
plt.ylabel('output(y)')
plt.title('toy dataset')
plt.show()

#creating test dataset
X_test = torch.linspace(0, 1, 40)
X_test = X_test.reshape((40, 1, 1))
```

Let's create an overparametrized NN that can fit on the dataset we just created and train it.

First, let's build the model architecture:

```
##Network Class - 2D
class Net(nn.Module):
  def __init__(self):
    super(Net, self).__init__()

    self.fc1 = nn.Linear(1, 300)
    self.fc2 = nn.Linear(300, 500)
    self.fc3 = nn.Linear(500, 1)

  def forward(self, x):
    x = F.leaky_relu(self.fc1(x))
    x = F.leaky_relu(self.fc2(x))
    output = self.fc3(x)
    return output
```

Next, let's define the different parameters for training our model:

```
#train the network on toy dataset
model = Net()

criterion = nn.MSELoss()
optimizer = optim.Adam(model.parameters(), lr = 1e-4)
max_epochs = 10000
iters = 0
# Calculates frobenius before training
normi, wsi, label = calculate_frobenius_norm(model)
```

At this point, we can now train our model.

```
# Initializing variables to store weights
per_layer = []

running_predictions = np.empty((40, (int)(max_epochs/500 + 1)))

train_loss = []
test_loss = []
model_norm = []

for epoch in tqdm(range(max_epochs)):

  norm, pl, layer_names = calculate_frobenius_norm(model)

  #training
  model_norm.append(norm)
  per_layer.append(pl)
  model.train()
  optimizer.zero_grad()
  predictions = model(X)
  loss = criterion(predictions, Y)
  loss.backward()
  optimizer.step()

  train_loss.append(loss.data)
  model.eval()
  Y_test = model(X_test)
  loss = criterion(Y_test, 2*X_test)
  test_loss.append(loss.data)

  if(epoch % 500 == 0 or epoch == max_epochs - 1):
    running_predictions[:, iters] = Y_test[:, 0, 0].detach().numpy()
    iters += 1
```

    HBox(children=(FloatProgress(value=0.0, max=10000.0), HTML(value='')))

Now that we have finished training, let's see how the model has evolved over the training process.

```
#@title Animation (Run Me!)

# create a figure and axes
fig = plt.figure(figsize=(14, 5))
ax1 = plt.subplot(1,2,1)
ax2 = plt.subplot(1,2,2)
# organizing subplots
plot1, = ax1.plot([],[])
plot2 = ax2.bar([], [])

def frame(i):
  ax1.clear()
  title1 = ax1.set_title('')
  ax1.set_xlabel("Input(x)")
  ax1.set_ylabel("Output(y)")

  ax2.clear()
  ax2.set_xlabel('Layer names')
  ax2.set_ylabel('Frobenius norm')
  title2 = ax2.set_title('Weight Measurement: Forbenius Norm')

  ax1.scatter(X[:,0,:].numpy(),Y[:,0,:].numpy())
  plot1 = ax1.plot(X_test[:,0,:].detach().numpy(),running_predictions[:,i])
  title1.set_text(f'Epochs: {i * 500}')
  plot2 = ax2.bar(label, per_layer[i*500])
  plt.axhline(y=model_norm[i*500], linewidth=1,
              color='r', ls='--', label=f'Norm: {model_norm[i*500]:.2f}')
  plt.legend()

  return plot1, plot2

anim = animation.FuncAnimation(fig, frame, frames=range(20),
                               blit=False, repeat=False, repeat_delay=10000)
html_anim = HTML(anim.to_html5_video());
plt.close()
display(html_anim)
```

```
#@title Plot the train and test losses
plt.figure(figsize=(8, 6))
plt.plot(train_loss,label='train_loss')
plt.plot(test_loss,label='test_loss')
plt.ylabel('loss')
plt.xlabel('epochs')
plt.title('loss vs epoch')
plt.legend()
plt.show()
```

Now let's visualize the Frobenious norm of the model as we trained. You should see that the value of weights increases over the epochs.

```
#@title Frobenious norm of the model
plt.figure(figsize=(8, 6))
plt.plot(model_norm)
plt.ylabel('norm of the model')
plt.xlabel('epochs')
plt.title('Size of the model vs Epochs') ## Change title to Frobenious norm of the model
plt.show()
```

Finally, you can compare the Frobenius norm for every layer in the model, before and after training.

```
#@title Frobenius norm per layer before and after training
normf, wsf, label = calculate_frobenius_norm(model)

plot_weights(float(normi), label, wsi, title='Weight Size Before Training')
plot_weights(float(normf), label, wsf, title='Weight Size After Training')
```

What trend do you see w.r.t to train and test losses ( Where do you see the minimum of these losses?) What does it tell about the model we trained? Discuss among yourselves.

## Overfitting on Test Dataset

In principle, we should not touch our test set until after we have chosen all our hyperparameters. Were we to use the test data in the model selection process, there is a risk that we might overfit the test data. Then we would be in serious trouble. If we overfit our training data, there is always the evaluation on test data to keep us honest. But if we overfit the test data, how would we ever know?

Note that there is another kind of overfitting: you do "honest" fitting on one set of images or posts, or medical records, but it may not generalize to other sets of images, posts or medical records.

##Validation Dataset
A common practice to address this problem is to split our data three ways, using a validation dataset (or validation set) to tune the hyperparameters. Ideally, we would only touch the test data once, to assess the very best model or to compare a small number of models to each other, real-world test data is seldom discarded after just one use.

---
# Section 3: Memorization

Given sufficiently large networks and enough training, Neural Networks can acheive almost 100% train accuracy.

In this section we train three MLP's one each on:

1.   Animal Faces Dataset
2.   A Completely Noisy Dataset (Random Shuffling of all labels)
3.   Partially Noisy Dataset (Random Shuffling of 15% labels)

Now, think for a couple of minutes as to what the train and test accuracies of each of these models might be, given you train for sufficient time and use a powerful network.

First, create the required dataloaders for all three datasets. Notice how we split the data. We train on a fraction of the dataset as it will be faster to train and will overfit more clearly.

```
##Dataloaders for the Dataset
batch_size = 128
classes = ('cat', 'dog', 'wild')

train_transform = transforms.Compose([
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
     ])
data_path = pathlib.Path('.')/'afhq' # using pathlib to be compatible with all OS's
img_dataset = ImageFolder(data_path/'train', transform=train_transform)

####################################################

##Dataloaders for the  Original Dataset

img_train_data, img_val_data,_ = torch.utils.data.random_split(img_dataset, [100,100,14430])

#Creating train_loader and Val_loader
train_loader = torch.utils.data.DataLoader(img_train_data,
                                           batch_size=batch_size,
                                           worker_init_fn=seed_worker)
val_loader = torch.utils.data.DataLoader(img_val_data,
                                         batch_size=1000,
                                         worker_init_fn=seed_worker)

#creating test dataset
test_transform = transforms.Compose([
     transforms.ToTensor(),
     transforms.Normalize((0.5, 0.5, 0.5), (0.5, 0.5, 0.5))
     ])
img_test_dataset = ImageFolder(data_path/'val', transform=test_transform)

####################################################

##Dataloaders for the  Random Dataset

#splitting randomized data into training and validation data
data_path = pathlib.Path('.')/'afhq_random_32x32/afhq_random' # using pathlib to be compatible with all OS's
img_dataset = ImageFolder(data_path/'train', transform=train_transform)
random_img_train_data, random_img_val_data,_ = torch.utils.data.random_split(img_dataset, [100, 100, 14430])

#Randomized train and validation dataloader
rand_train_loader = torch.utils.data.DataLoader(random_img_train_data,
                                                batch_size=batch_size,
                                                num_workers=0,
                                                worker_init_fn=seed_worker)
rand_val_loader = torch.utils.data.DataLoader(random_img_val_data,
                                              batch_size=1000,
                                              num_workers=0,
                                              worker_init_fn=seed_worker)

####################################################

##Dataloaders for the Partially Random Dataset

#Splitting data between training and validation dataset for partially randomized data
data_path = pathlib.Path('.')/'afhq_10_32x32/afhq_10' # using pathlib to be compatible with all OS's
img_dataset = ImageFolder(data_path/'train', transform=train_transform)
partially_random_train_data, partially_random_val_data,_ = torch.utils.data.random_split(img_dataset, [100, 100, 14430])

#Training and Validation loader for partially randomized data
partial_rand_train_loader = torch.utils.data.DataLoader(partially_random_train_data,
                                                        batch_size=batch_size,
                                                        num_workers=0,
                                                        worker_init_fn=seed_worker)
partial_rand_val_loader = torch.utils.data.DataLoader(partially_random_val_data,
                                                      batch_size=1000,
                                                      num_workers=0,
                                                      worker_init_fn=seed_worker)
```

Now let's define a model which has many parameters compared to the training dataset size, and train it on these datasets.

```
# # Network Class - Animal Faces
class Big_Animal_Net(nn.Module):
  def __init__(self):
    torch.manual_seed(104)
    super(Big_Animal_Net, self).__init__()
    self.fc1 = nn.Linear(3*32*32, 124)
    self.fc2 = nn.Linear(124, 64)
    self.fc3 = nn.Linear(64, 3)

  def forward(self, x):
    x = x.view(x.shape[0], -1)
    x = F.leaky_relu(self.fc1(x))
    x = F.leaky_relu(self.fc2(x))
    x = self.fc3(x)
    output = F.log_softmax(x, dim=1)
    return output
```

Before training our `Big_AnimalNet()`, calculate the Frobenius norm again.

```
model = Big_Animal_Net()
normi, wsi, label = calculate_frobenius_norm(model)
```

Now, train our `Big_AnimalNet()` model

```
# # Here we have 100 true train data.

args = {'test_batch_size': 1000,
        'epochs': 200,
        'lr': 5e-3,
        'momentum': 0.9,
        'no_cuda': False,
        }

acc_dict = {}

start_time = time.time()
val_acc_pure, train_acc_pure, _, model ,_ = main(args,
                                                 model,
                                                 train_loader,
                                                 val_loader,
                                                 img_test_dataset)
end_time = time.time()

print("Time to memorize the dataset:",end_time - start_time)

# # Train and Test accuracy plot
plt.figure(figsize=(8, 6))
plt.plot(val_acc_pure, label='Val Accuracy Pure', c='red', ls='dashed')
plt.plot(train_acc_pure, label='Train Accuracy Pure', c='red', ls='solid')
plt.axhline(y=max(val_acc_pure), c='green', ls='dashed')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

```
#@title Frobenius norm for AnimalNet before and after training
normf, wsf, label = calculate_frobenius_norm(model)

plot_weights(float(normi), label, wsi, title='Weight Size Before Training')
plot_weights(float(normf), label, wsf, title='Weight Size After Training')
```

##Exercise 2: Data Visualizer
Before we train the model on a data with random labels, let's visualize and verify for ourselves that the data is random. Here, we have classes = ("cat", "dog", "wild").

**Hint:** Use .permute() method. plt.imshow() expects imput to be in numpy format and in the format (Px,Py,3), where Px and Py are the number of pixels along axis x and y respectively.

```
def Visualize_data(dataloader):
  ####################################################################
  # Fill in all missing code below (...),
  # then remove or comment the line below to test your function
  # The dataloader here gives out mini batches of 100 data points.
  raise NotImplementedError("Complete the Visualize_random_data function")
  ####################################################################

  for idx, (data,label) in enumerate(dataloader):
    plt.figure(idx)
    # Choose the datapoint you would like to visualize
    index = ...

    # choose that datapoint using index and permute the dimensions
    # and bring the pixel values between [0,1]
    data = ...

    # Convert the torch tensor into numpy
    data = ...

    plt.imshow(data)
    plt.axis(False)
    image_class = classes[...]
    print(f'The image belongs to : {image_class}')

  plt.show()

# # uncomment to run the function
# Visualize_data(rand_train_loader)
```

```
# to_remove solution
def Visualize_data(dataloader):

  for idx, (data,label) in enumerate(dataloader):
    plt.figure(idx)
    # Choose the datapoint you would like to visualize
    index = 22

    # choose that datapoint using index and permute the dimensions
    # and bring the pixel values between [0,1]
    data = data[index].permute(1, 2, 0) * \
           torch.tensor([0.5, 0.5, 0.5]) + \
           torch.tensor([0.5, 0.5, 0.5])

    # Convert the torch tensor into numpy
    data = data.numpy()

    plt.imshow(data)
    plt.axis(False)
    image_class = classes[label[index].item()]
    print(f'The image belongs to : {image_class}')

  plt.show()

# # uncomment to run the function
Visualize_data(rand_train_loader)
```

Now let's train the network on the shuffled data and see if it memorizes.

```
##Here we have 100 completely shuffled train data.
args = {'epochs': 200,
        'lr': 5e-3,
        'momentum': 0.9,
        'no_cuda': False
        }

acc_dict = {}
model = Big_Animal_Net()

val_acc_random, train_acc_random, _,model,_ = main(args,
                                                   model,
                                                   rand_train_loader,
                                                   val_loader,
                                                   img_test_dataset)

##Train and Test accuracy plot
plt.figure(figsize=(8, 6))
plt.plot(val_acc_random, label='Val Accuracy random', c='red', ls='dashed')
plt.plot(train_acc_random, label='Train Accuracy random', c='red', ls='solid')
plt.axhline(y=max(val_acc_random), c='green', ls='dashed')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

Finally lets train on a parially shuffled dataset where 15% of the labels are noisy.

```
##Here we have 100 partially shuffled train data.
args = {'test_batch_size': 1000,
        'epochs': 200,
        'lr': 5e-3,
        'momentum': 0.9,
        'no_cuda': False,
        }

acc_dict = {}
model = Big_Animal_Net()

val_acc_shuffle, train_acc_shuffle, _,_,_ = main(args,
                                                 model,
                                                 partial_rand_train_loader,
                                                 val_loader,
                                                 img_test_dataset)

#train and test acc plot
plt.figure(figsize=(8, 6))
plt.plot(val_acc_shuffle, label='Val Accuracy shuffle', c='red', ls='dashed')
plt.plot(train_acc_shuffle, label='Train Accuracy shuffle', c='red', ls='solid')
plt.axhline(y=max(val_acc_shuffle), c='green', ls='dashed')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

```
#@title Plotting them all together (Run Me!)
plt.figure(figsize=(8, 6))
plt.plot(val_acc_pure,label='Val - Pure',c='red',ls = 'dashed')
plt.plot(train_acc_pure,label='Train - Pure',c='red',ls = 'solid')
plt.plot(val_acc_random,label='Val - Random',c='blue',ls = 'dashed')
plt.plot(train_acc_random,label='Train - Random',c='blue',ls = 'solid')
plt.title('Memorization')
plt.ylabel('Accuracy (%)')
plt.xlabel('Epoch')
plt.legend()
plt.show()
```

Given that the NN fit/memorize the training data perfectly, Do you think it generalizes well? What makes you think it does or doesn't?

Isn't it supprising to see that the NN was able to acheive 100% training accuracy on randomly shuffled labels. This is one of the reasons why training accuracy is not a good indicator of model performance.

Also it is interesting to note that sometimes the model trained on slightly shuffled data does slightly better than the one trained on pure data.  Shuffling some of the data is a form of regularization--one of many ways of adding noise to the training data.

---
# Section 4: Early Stopping

```
#@title Video : Early Stopping
try: t4;
except NameError: t4=time.time()

from IPython.display import YouTubeVideo
video = YouTubeVideo(id="GA6J-50GCWs", width=854, height=480, fs=1)
print("Video available at https://youtube.com/watch?v=" + video.id)

video
```

    Video available at https://youtube.com/watch?v=GA6J-50GCWs

Now that we have established that the validation accuracy reaches the peak well before the model overfits we want to somehow stop the training early. You should have also observed from the above plots that the train/test loss on real data is not very smooth and hence you might guess that the choice of epoch can play a very large role on the val/test accuracy of your model.

Early stopping stops training when the validation accuracies stop increasing.

## Exercise 3: Early Stopping
Reimplement the main function to include early stopping as described above. Then run the code below to validate your implementation.

```
def early_stopping_main(args, model, train_loader, val_loader, test_data):

  ####################################################################
  # Fill in all missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Complete the early_stopping_main function")
  ####################################################################

  use_cuda = not args['no_cuda'] and torch.cuda.is_available()
  device = torch.device('cuda' if use_cuda else 'cpu')

  model = model.to(device)
  optimizer = optim.SGD(model.parameters(),
                        lr=args['lr'],
                        momentum=args['momentum'])

  best_acc = 0.0
  best_epoch = 0

  # Number of successive epochs that you want to wait before stopping training process
  patience = 20

  # Keps track of number of epochs during which the val_acc was less than best_acc
  wait = 0

  val_acc_list, train_acc_list = [], []
  for epoch in tqdm(range(args['epochs'])):
    # train the model
    ...
    # calculate training accuracy
    train_acc = ...
    # calculate validation accuracy
    val_acc = ...
    if (val_acc > best_acc):
      best_acc = val_acc
      best_epoch = epoch
      best_model = copy.deepcopy(model)
      wait = 0
    else:
      wait += 1
    if (wait > patience):
      print(f'early stopped on epoch:{epoch}')
      break
    train_acc_list.append(train_acc)
    val_acc_list.append(val_acc)

  return val_acc_list, train_acc_list, best_model, best_epoch
```

```
# to_remove solution
def early_stopping_main(args, model, train_loader, val_loader, test_data):

  use_cuda = not args['no_cuda'] and torch.cuda.is_available()
  device = torch.device('cuda' if use_cuda else 'cpu')

  model = model.to(device)
  optimizer = optim.SGD(model.parameters(),
                        lr=args['lr'],
                        momentum=args['momentum'])

  best_acc = 0.0
  best_epoch = 0

  # Number of successive epochs that you want to wait before stopping training process
  patience = 20

  # Keps track of number of epochs during which the val_acc was less than best_acc
  wait = 0

  val_acc_list, train_acc_list = [], []
  for epoch in tqdm(range(args['epochs'])):
    # train the model
    train(args, model, device, train_loader, optimizer, epoch)
    # calculate training accuracy
    train_acc = test(model, device, train_loader, 'Train')
    # calculate validation accuracy
    val_acc = test(model, device, val_loader, 'Val')
    if (val_acc > best_acc):
      best_acc = val_acc
      best_epoch = epoch
      best_model = copy.deepcopy(model)
      wait = 0
    else:
      wait += 1
    if (wait > patience):
      print(f'early stopped on epoch:{epoch}')
      break
    train_acc_list.append(train_acc)
    val_acc_list.append(val_acc)

  return val_acc_list, train_acc_list, best_model, best_epoch
```

```
args = {'epochs': 200,
        'lr': 5e-4,
        'momentum': 0.99,
        'no_cuda': False,
        }

model = Animal_Net()
val_acc_earlystop, train_acc_earlystop, _, best_epoch = early_stopping_main(args,
                                                                            model,
                                                                            train_loader,
                                                                            val_loader,
                                                                            img_test_dataset)

print("Maximum Validation Accuracy is reached at epoch:%2d"%(best_epoch))
```

    HBox(children=(FloatProgress(value=0.0, max=200.0), HTML(value='')))

    early stopped on epoch:71
    Maximum Validation Accuracy is reached at epoch:50

Do you think early stopping can be harmful for the training of your network? Discuss among your pod why or why not?


# Usikkerhetsanalyse av Biot-Savarts lov

## FY1003 Elektrisitet og magnetisme

### Laboratorieøvelse 2 Statisk magnetfelt

#### Norges Teknisk-Naturvitenskapelige Universitet

Av Håkon W. Ånes

Dato: 2018-02-18

___

**Mål:** Plotte målt og beregnet $B(x)$ i hver posisjon $x$ med beregnet usikkerhet for de fem geometriene kort spole, helmholtzspole ($a = (2R, R, R/2)$) og solenoide.

**Fremgangsmåte:**
1. Ser på usikkerhetsbidraget til $\Delta B$ fra hver av variablene som inngår i Biot-Savarts lov ved å benytte Gauss' feilforplantningslov.
2. Partiellderiverer hvert uttrykk ved bruk av Python-biblioteket `sympy`.
3. Setter dette som usikkerhet for den beregnede kurven.
4. Plotter måledata og beregnet kurve med $\pm$ to ganger beregnet usikkerhet (analogt med to standardavvik).

Noe av `sympy`s funksjoner benyttet i denne notebooken finnes her [http://docs.sympy.org/latest/tutorial/calculus.html#](http://docs.sympy.org/latest/tutorial/calculus.html#).

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import sympy as sp

# Initialiserer pen visning av uttrykkene
sp.init_printing()

# Plotteparametre for å få store, tydelige plott som utnytter tilgjengelig skjermareal
fontsize = 20
newparams = {'axes.titlesize': fontsize, 'axes.labelsize': fontsize,
             'lines.linewidth': 2, 'lines.markersize': 7,
             'figure.figsize': (16, 5), 'ytick.labelsize': fontsize,
             'xtick.labelsize': fontsize, 'legend.fontsize': fontsize,
            'legend.handlelength': 1.5}
plt.rcParams.update(newparams)
```

```python
# Definerer variablene som inngår i uttrykkene for Biot-Savarts lov for alle geometriene
mu0, I, R, N, l, x, z = sp.symbols('mu0 I R N l x z')
```

# Kort spole

Lager uttrykk for Biot-Savarts lov for kort spole og skriver til skjerm.

```python
kort_spole = (N*I*mu0)/(2*R)*(1 + (x/R)**2)**(-3/2)
kort_spole
```

Da partiellderiverer vi med hensyn på f.eks. $x$, $I$ og $R$ og legger uttrykkene i en liste. For å sjekke om det faktisk er så lett som det ser ut skriver vi $\partial B/\partial I$ til skjerm.

```python
dBbd = [sp.diff(kort_spole, x), sp.diff(kort_spole, I), sp.diff(kort_spole, R)]
dBbd[1]
```

Suksess!

For å finne usikkerheter $\Delta B(x)$ for den beregnede kurven lager vi et array med avstander $x_\mathrm{i}$ tilsvarende måleposisjonene, og korrigerer posisjonene iht. spolens midtplan. Vi lager også en beregnet kurve fra Biot-Savarts lov. I tillegg skriver vi eksperimentelle måleposisjoner og magnetfeltstyrker til arrayer.

```python
# Eksperimentelle måleposisjoner
xe = np.array([0.01, 0.02, 0.03, 0.04, 0.05, 0.06, 0.07, 0.08, 0.09, 0.10, 0.11, 0.12, 0.13, 0.14,
               0.15, 0.16, 0.17, 0.18, 0.19, 0.20, 0.21, 0.22, 0.23, 0.24, 0.25, 0.26, 0.27, 0.28,
               0.285, 0.29, 0.295, 0.3, 0.305, 0.31, 0.315, 0.32, 0.33, 0.34, 0.35, 0.36, 0.37,
               0.38, 0.39, 0.4, 0.41, 0.42, 0.43, 0.44, 0.45, 0.46, 0.47, 0.48, 0.49, 0.5])

# For beregnet kurve
xb = np.linspace(0.01, 0.5, 200)

# Måleposisjonene er absolutte posisjoner avlest fra linjal. Spolens midtplan er i x = 0.30 m, og
# vi må korrigere for dette.
x0 = 0.30
xe -= x0
xb -= x0

# Eksperimentelle magnetfeltstyrker
Be = np.array([0.22, 0.26, 0.3, 0.36, 0.42, 0.49, 0.58, 0.67, 0.79, 0.92, 1.09, 1.27, 1.51, 1.79,
               2.14, 2.57, 3.1, 3.75, 4.58, 5.62, 6.97, 8.59, 10.74, 13.3, 16.36, 19.85, 23.41,
               26.57, 27.76, 28.71, 29.2, 29.26, 28.91, 28.13, 26.97, 25.52, 22.19, 18.53, 15.21,
               12.31, 9.96, 8.04, 6.47, 5.26, 4.31, 3.51, 2.9, 2.41, 2.02, 1.7, 1.43, 1.21, 1.03,
               0.87])

# Beregnede magnetfeltstyrker i Gauss (ganger med 10**4)
Bb = [kort_spole.subs([(N, 330), (I, 1), (mu0, 4*np.pi*1e-7), (R, 0.07), (x, xbi)])*1e4
      for xbi in xb]

# Gjør om hvert element i Bb til numpy float-verdier
Bb = np.array(Bb).astype(np.float64)
```

Spesifiserer så tenkte usikkerheter for variablene (ikke bruk disse, tenk ut dine egne!), beregner usikkerheter med Gauss' feilforplantningslov

$$
\Delta B_\mathrm{b} = \sqrt{\left(\frac{\partial B_\mathrm{b}}{\partial x_\mathrm{b}} \cdot \Delta x_\mathrm{b}\right)^2
+ \left(\frac{\partial B_\mathrm{b}}{\partial I} \cdot \Delta I\right)^2
+ \left(\frac{\partial B_\mathrm{b}}{\partial R} \cdot \Delta R\right)^2}.
$$

og plotter resultatene.

```python
# Usikkerheter for beregnet kurve (tenk ut disse selv, valg av disse må begrunnes i rapporten!)
deltaxb = 0.01  # [m]
deltaIb = 0.01  # [A]
deltaRb = 0.007  # [m]

# Legger dem i en liste i samme rekkefølge (! VIKTIG!) som dBed
deltas = [deltaxb, deltaIb, deltaRb]

# Oppdiktet usikkerhet for gaussmeteret for eksperimentelle måledata (finnes ved å følge
# prosedyren i 3.3.4 i labheftet)
deltaBe = 1  # [Gauss]

# Definerer et array av samme form som Be
deltaBb = np.zeros(xb.shape)

# Beregner usikkerhetene for hver av måleposisjonene. Én iterasjon, én deltaB(x_i).
for i in range(len(xb)):

    # Benytter list comprehension her, se f.eks.
    # https://www.pythonforbeginners.com/basics/list-comprehensions-in-python
    # Eksempel syntaks: variabel = [en operasjon for i in range(3)]
    # Hva vi gjør (fra innerst til ytterst i linjen):
    #   1. Med dBkd[j].subs() bytter vi ut variablene N, I, mu0, R og x med tallverdier. x endrer seg
    #      for hver i-iterasjon i den ytre løkken.
    #   2. Kvadrerer
    #   3. Summerer de tre leddene ()^2
    deltaBb[i] = np.sum([(dBbd[j].subs([(N, 330), (I, 1), (mu0, 4*np.pi*1e-7), (R, 0.07),
                                        (x, xb[i])])*deltas[j])**2 for j in range(len(deltas))])
    # Så tar vi roten og gjør om fra Tesla til Gauss
    deltaBb[i] = np.sqrt(deltaBb[i])*1e4

# Plotter
plt.fill_between(xb, Bb - 1*deltaBb, Bb + 1*deltaBb,
                 label='Beregnet kurve med $\pm \Delta B_\mathrm{b}$', alpha=0.5)
plt.errorbar(xe, Be, yerr=deltaBe, fmt='r.', label='Måledata')
plt.ylim([0, 1.1*np.max(Bb + deltaBb)])
plt.xlabel('$x$ [m]')
plt.ylabel('$B$ [Gauss]')
plt.legend(loc='upper left');
```

Her har vi plottet måledata og beregnet kurve med en usikkerhet, altså benytter vi et konfidensintervall på 68% ([https://en.wikipedia.org/wiki/Confidence_interval](https://en.wikipedia.org/wiki/Confidence_interval)), dvs. at vi med 68% sannsynlighet kan si at magnetfeltstyrkens "sanne" verdi for hver posisjon ligger innenfor det tegnede intervallet ([https://en.wikipedia.org/wiki/68-95-99.7_rule](https://en.wikipedia.org/wiki/68%E2%80%9395%E2%80%9399.7_rule)).

Vi kan også undersøke de relative bidragene fra hver av usikkerhetene $\Delta x$, $\Delta I$ og $\Delta R$ for hver av posisjonene $x_\mathrm{i}$ til den beregnede kurven, som

$$
\frac{\left(\frac{\partial B}{\partial x}\Delta x\right)^2}{\Delta B^2},
$$

osv.

```python
# Beregner bidragene
bidrag = np.zeros((len(deltas), len(xb)))
for i in range(len(deltas)):
    bidrag[i] = [dBbd[i].subs([(N, 330), (I, 1), (mu0, 4*np.pi*1e-7), (R, 0.07),
                               (x, xb[j])])*1e4*deltas[i] for j in range(len(xb))]

# Plotter bidrag til usikkerheten
plt.plot(xb, np.abs(bidrag[0])**2/deltaBb**2, '-',
         label=r'$(\frac{\partial B}{\partial x}\Delta x)^2/(\Delta B)^2$')
plt.plot(xb, np.abs(bidrag[1])**2/deltaBb**2, '--',
         label=r'$(\frac{\partial B}{\partial I}\Delta I)^2/(\Delta B)^2$')
plt.plot(xb, np.abs(bidrag[2])**2/deltaBb**2, '-.',
         label=r'$(\frac{\partial B}{\partial R}\Delta R)^2/(\Delta B)^2$')
plt.ylabel('Relative bidrag til $\Delta B$ [-]')
plt.xlabel('$x$ [m]')
plt.ylim([0, 1.1])
plt.legend(loc='upper left');
```

# Helmholtzspoler og solenoide

... Dette gjør dere selv. Hold tungen rett i munnen!