#  Introduction to Neural Networks and Pytorch

    Notebook version: 0.1 (Nov 14, 2020)

    Authors: Jerónimo Arenas García (jarenas@ing.uc3m.es)

    Changes: v.0.1. (Nov 14, 2020) - First version

    Pending changes: - Use epochs instead of iters in first part of notebook
                     - Add an example with dropout
                     - Add theory about CNNs
                     - Define functions for the training of neural nets and display of the results
                       in order to simplify code cells

```python
import numpy as np
import matplotlib.pyplot as plt

%matplotlib inline

size=18
params = {'legend.fontsize': 'Large',
          'axes.labelsize': size,
          'axes.titlesize': size,
          'xtick.labelsize': size*0.75,
          'ytick.labelsize': size*0.75}
plt.rcParams.update(params)
```

##  1. Introduction and purpose of this Notebook

###  1.1. About Neural Networks

* Neural Networks (NN) have become the state of the art for many machine learning problems
    * Natural Language Processing
    * Computer Vision
    * Image Recognition

* They are in widespread use for many applications, e.g.,
    * Language Translantion (Google Neural Machine Translation System)
    * Automatic Speech recognition (Hey Siri! DNN overview)
    * Autonomous Navigation (Facebook Robot Autonomous 3D Navigation)
    * Automatic Plate recognition

Feed Forward Neural Networks have been around since 1960 but only recently (last 10-12 years) have they met their expectations, and improve other machine learning algorithms

* Computation resources are now available at large scale
* Cloud Computing (AWS, Azure)
* From MultiLayer Perceptrons to Deep Learning
* Big Data sets
* This has also made possible an intense research effort resulting in
    * Topologies better suited to particular problems (CNNs, RNNs)
    * New training strategies providing better generalization

In parallel, Deep Learning Platforms have emerged that make design, implementation, training, and production of DNNs feasible for everyone

###  1.2. Scope

* To provide just an overview of most important NNs and DNNs concepts
* Connecting with already studied methods as starting point
* Introduction to PyTorch
* Providing links to external sources for further study

###  1.3. Outline

1. Introduction and purpose of this Notebook
2. Introduction to Neural Networks
3. Implementing Deep Networks with PyTorch

###  1.4. Other resources

* We point here to external resources and tutorials that are excellent material for further study of the topic
* Most of them include examples and exercises using numpy and PyTorch
* This notebook uses examples and other material from some of these sources

|Tutorial|Description|
|-----|---------------------|
|  |Very general tutorial including videos and an overview of top deep learning platforms|
|  |Very complete book with a lot of theory and examples for MxNET, PyTorch, and TensorFlow|
|  |Official tutorials from the PyTorch project. Contains a 60 min overview, and a very practical *learning PyTorch with examples* tutorial|
|  |Kaggle tutorials covering an introduction to Neural Networks using Numpy, and a second one offering a PyTorch tutorial|

In addition to this, PyTorch MOOCs can be followed for free in main sites: edX, Coursera, Udacity

##  2. Introduction to Neural Networks

In this section, we will implement neural networks from scratch using Numpy arrays

* No need to learn any new Python libraries
* But we need to deal with complexity of multilayer networks
* Low-level implementation will be useful to grasp the most important concepts concerning DNNs
    * Back-propagation
    * Activation functions
    * Loss functions
    * Optimization methods
    * Generalization
    * Special layers and configurations

###  2.0. Data preparation

We start by loading some data sets that will be used to carry out the exercises

### Sign language digits data set

* Dataset is taken from  Kaggle and used in the above referred tutorial
* 2062 digits in sign language. $64 \times 64$ images
* Problem with 10 classes. One hot encoding for the label matrix
* Input data are images, we create also a flattened version

```python
digitsX = np.load('./data/Sign-language-digits-dataset/X.npy')
digitsY = np.load('./data/Sign-language-digits-dataset/Y.npy')
K = digitsX.shape[0]
img_size = digitsX.shape[1]
digitsX_flatten = digitsX.reshape(K,img_size*img_size)

print('Size of Input Data Matrix:', digitsX.shape)
print('Size of Flattned Input Data Matrix:', digitsX_flatten.shape)
print('Size of label Data Matrix:', digitsY.shape)
selected = [260, 1400]
plt.subplot(1, 2, 1), plt.imshow(digitsX[selected[0]].reshape(img_size, img_size)), plt.axis('off')
plt.subplot(1, 2, 2), plt.imshow(digitsX[selected[1]].reshape(img_size, img_size)), plt.axis('off')
plt.show()
print('Labels corresponding to figures:', digitsY[selected,])
```

### Dogs vs Cats data set

* Dataset is taken from  Kaggle
* 25000 pictures of dogs and cats
* Binary problem
* Input data are images, we create also a flattened version
* Original images are RGB, and arbitrary size
* Preprocessed images are $64 \times 64$ and gray scale

```python
# Preprocessing of original Dogs and Cats Pictures
# Adapted from https://medium.com/@mrgarg.rajat/kaggle-dogs-vs-cats-challenge-complete-step-by-step-guide-part-1-a347194e55b1
# RGB channels are collapsed in GRAYSCALE
# Images are resampled to 64x64

"""
import os, cv2  # cv2 -- OpenCV

train_dir = './data/DogsCats/train/'
rows = 64
cols = 64
train_images = sorted([train_dir+i for i in os.listdir(train_dir)])

def read_image(file_path):
    image = cv2.imread(file_path, cv2.IMREAD_GRAYSCALE)
    return cv2.resize(image, (rows, cols),interpolation=cv2.INTER_CUBIC)

def prep_data(images):
    m = len(images)
    X = np.ndarray((m, rows, cols), dtype=np.uint8)
    y = np.zeros((m,))
    print("X.shape is {}".format(X.shape))

    for i,image_file in enumerate(images) :
        image = read_image(image_file)
        X[i,] = np.squeeze(image.reshape((rows, cols)))
        if 'dog' in image_file.split('/')[-1].lower():
            y[i] = 1
        elif 'cat' in image_file.split('/')[-1].lower():
            y[i] = 0

        if i%5000 == 0 :
            print("Proceed {} of {}".format(i, m))

    return X,y

X_train, y_train = prep_data(train_images)
np.save('./data/DogsCats/X.npy', X_train)
np.save('./data/DogsCats/Y.npy', y_train)
"""
```

```python
DogsCatsX = np.load('./data/DogsCats/X.npy')
DogsCatsY = np.load('./data/DogsCats/Y.npy')
K = DogsCatsX.shape[0]
img_size = DogsCatsX.shape[1]
DogsCatsX_flatten = DogsCatsX.reshape(K,img_size*img_size)

print('Size of Input Data Matrix:', DogsCatsX.shape)
print('Size of Flattned Input Data Matrix:', DogsCatsX_flatten.shape)
print('Size of label Data Matrix:', DogsCatsY.shape)
selected = [260, 16000]
plt.subplot(1, 2, 1), plt.imshow(DogsCatsX[selected[0]].reshape(img_size, img_size)), plt.axis('off')
plt.subplot(1, 2, 2), plt.imshow(DogsCatsX[selected[1]].reshape(img_size, img_size)), plt.axis('off')
plt.show()
print('Labels corresponding to figures:', DogsCatsY[selected,])
```

###  2.1. Logistic Regression as a Simple Neural Network

* We can consider logistic regression as an extremely simple (1 layer) neural network

* In this context, $\text{NLL}({\bf w})$ is normally referred to as cross-entropy loss

* We need to find parameters $\bf w$ and $b$ to minimize the loss $\rightarrow$ GD / SGD
* Gradient computation can be simplified using the **chain rule**

\begin{align}
\frac{\partial \text{NLL}}{\partial {\bf w}} & = \frac{\partial \text{NLL}}{\partial {\hat y}} \cdot \frac{\partial \hat y}{\partial o} \cdot \frac{\partial o}{\partial {\bf w}} \\
& = \sum_{k=0}^{K-1} \left[\frac{1 - y_k}{1 - \hat y_k} - \frac{y_k}{\hat y_k}\right]\hat y_k (1-\hat y_k) {\bf x}_k \\
& = \sum_{k=0}^{K-1} \left[(1 - y_k) \hat y_k - y_k (1 - \hat y_k) \right] {\bf x}_k \\
\frac{\partial \text{NLL}}{\partial b} & = \sum_{k=0}^{K-1} \left[(1 - y_k) \hat y_k - y_k (1 - \hat y_k) \right]
\end{align}

* Gradient Descent Optimization

$${\bf w}_{n+1} = {\bf w}_n + \rho_n \sum_{k=0}^{K-1} \left[y_k (1 - \hat y_k) - (1 - y_k) \hat y_k \right] {\bf x}_k = {\bf w}_n + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k){\bf x}_k$$
$$b_{n+1} = b_n + \rho_n \sum_{k=0}^{K-1} \left[y_k (1 - \hat y_k) - (1 - y_k) \hat y_k \right] = b_n + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)$$

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

#dataset = 'DogsCats'
dataset = 'digits'

if dataset=='DogsCats':
    X = DogsCatsX_flatten
    y = DogsCatsY

else:
    #Zero and Ones are one hot encoded in columns 1 and 4
    X0 = digitsX_flatten[np.argmax(digitsY, axis=1)==1,]
    X1 = digitsX_flatten[np.argmax(digitsY, axis=1)==4,]
    X = np.vstack((X0, X1))
    y = np.zeros(X.shape[0])
    y[X0.shape[0]:] = 1

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(X)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, shuffle=True)
```

```python
# Define some useful functions
def logistic(t):
    return 1.0 / (1 + np.exp(-t))

def forward(w,b,x):
    #Calcula la salida de la red
    return logistic(x.dot(w)+b)

def backward(y,y_hat,x):
    #Calcula los gradientes
    #w_grad = x.T.dot((1-y)*y_hat - y*(1-y_hat))/len(y)
    #b_grad = np.sum((1-y)*y_hat - y*(1-y_hat))/len(y)
    w_grad = x.T.dot(y_hat-y)/len(y)
    b_grad = np.sum(y_hat-y)/len(y)
    return w_grad, b_grad

def accuracy(y, y_hat):
    return np.mean(y == (y_hat>=0.5))

def loss(y, y_hat):
    return -np.sum(y*np.log(y_hat)+(1-y)*np.log(1-y_hat))/len(y)
```

```python
#Neural Network Training

epochs = 50
rho = .05 #Use this setting for Sign Digits Dataset

#Parameter initialization
w = .1 * np.random.randn(X.shape[1])
b = .1 * np.random.randn(1)

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in np.arange(epochs):
    y_hat_train = forward(w, b, X_train)
    y_hat_val = forward(w, b, X_val)
    w_grad, b_grad = backward(y_train, y_hat_train, X_train)
    w = w - rho * w_grad
    b = b - rho * b_grad

    loss_train[epoch] = loss(y_train, y_hat_train)
    loss_val[epoch] = loss(y_val, y_hat_val)
    acc_train[epoch] = accuracy(y_train, y_hat_train)
    acc_val[epoch] = accuracy(y_val, y_hat_val)
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Exercise

* Study the behavior of the algorithm changing the number of epochs and the learning rate

* Repeat the analysis for the other dataset, trying to obtain as large an accuracy value as possible

* What do you believe are the reasons for the very different performance for both datasets?

Linear logistic regression allowed us to review a few concepts that are key for Neural Networks:

* Network topology (In this case, a linear network with one layer)
* Activation functions
* Parametric approach ($\bf w$/$b$)
* Parameter initialization
* Obtaining the network prediction using *forward* computation
* Loss function
* Parameter gradient calculus using *backward* computation
* Optimization method for parameters update (here, GD)

###  2.2. (Multiclass) SoftMax Regression

* One hot encoding output, e.g., $[0, 1, 0, 0]$, $[0, 0, 0, 1]$
* Used to encode categorial variables without predefined order
* Similar to logistic regression, network tries to predict class probability
$$\hat y_{k,j} = \hat P(y_k=j|{\bf x}_k)$$
* Network output should satisfy "probability constraints"
$$\hat y_{k,j} \in [0,1]\qquad \text{and} \qquad \sum_j \hat y_{k,j} = 1$$

* Softmax regression network topology:

### Notation

In this section, it is important to pay attention to subindexes:

|Notation/ Variable Name|Definition|
|-----------------------|---------------------------------|
|$y_k \in [0,\dots,M-1]$|The label of pattern $k$|
|${\bf y}_k$|One hot encoding of the label of pattern $k$|
|$y_{k,m}$|$m$-th component of vector ${\bf y}_k$|
|$y_{m}$|$m$-th component of generic vector ${\bf y}$ (i.e., for an undefined pattern)|
|$\hat {\bf y}_k$|Network output for pattern $k$|
|$\hat y_{k,m}$|$m$-th network output for pattern $k$|
|$\hat y_{m}$|$m$-th network output for an undefined pattern)|
|$k$|Index used for pattern enumeration|
|$m$|Index used for network output enumeration|
|$j$|Secondary index for selected network output|

### The softmax function

* It is to multiclass problems as the logistic function for binary classification
* Invented in 1959 by the social scientist R. Duncan Luce
* Transforms a set of $M$ real numbers to satisfy "probability" constraints

$${\bf \hat y} = \text{softmax}({\bf o}) \qquad \text{where} \qquad \hat y_j = \frac{\exp(o_j)}{\sum_m \exp(o_m)} $$

* Continuous and **differentiable** function

$$\frac{\partial \hat y_j}{\partial o_j} = \hat y_j (1 - \hat y_j) \qquad \text{and} \qquad \frac{\partial \hat y_j}{\partial o_m} = - \hat y_j \hat y_m$$

* The classifier is still linear, since

$$\arg\max \hat {\bf y} = \arg\max \hat {\bf o} = \arg\max {\bf W} {\bf x} + {\bf b}$$

### Cross-entropy loss for multiclass problems

* Similarly to logistic regression, minimization of the log-likelihood can be stated to obtain ${\bf W}$ and ${\bf b}$

$$\text{Binary}: \text{NLL}({\bf w}, b) = - \sum_{k=0}^{K-1} \log \hat P(y_k|{\bf x}_k)$$
$$\text{Multiclass}: \text{NLL}({\bf W}, {\bf b}) = - \sum_{k=0}^{K-1} \log \hat P(y_k|{\bf x}_k)$$

* Using one hot encoding for the label vector of each sample, e.g., $y_k = 2 \rightarrow {\bf y}_k = [0, 0, 1, 0]$

$$\text{NLL}({\bf W}, {\bf b}) = - \sum_{k=0}^{K-1} \sum_{m=0}^{M-1} y_{k,m} \log \hat P(m|{\bf x}_k)= - \sum_{k=0}^{K-1} \sum_{m=0}^{M-1} y_{k,m} \log \hat y_{k,m} = \sum_{k=0}^{K-1} l({\bf y}_k, \hat {\bf y}_k)$$

* Note that for each pattern, only one element in the inner sum (the one indexed with $m$) is non-zero

* In the context of Neural Networks, this cost is referred to as the cross-entropy loss

$$l({\bf y}, \hat {\bf y}) = - \sum_{m=0}^{M-1} y_{m} \log \hat y_{m}$$

### Network optimization

* Gradient Descent Optimization

$${\bf W}_{n+1} = {\bf W}_n - \rho_n \sum_{k=0}^{K-1} \frac{\partial l({\bf y}_k,{\hat {\bf y}_k})}{\partial {\bf W}}$$
$${\bf b}_{n+1} = {\bf b}_n - \rho_n \sum_{k=0}^{K-1} \frac{\partial l({\bf y}_k,{\hat {\bf y}_k})}{\partial {\bf b}}$$

* We compute derivatives using the chain rule (we ignore dimension mismatchs, and rearrange at the end)

\begin{align}
\frac{\partial l({\bf y},{\hat {\bf y}})}{\partial {\bf W}} &= \frac{\partial l({\bf y},{\hat {\bf y}})}{\partial \hat {\bf y}} \cdot \frac{\partial \hat {\bf y}}{\partial {\bf o}} \cdot \frac{\partial {\bf o}}{\partial {\bf W}} \\ & = \left[\begin{array}{c} 0 \\ 0 \\ \vdots \\ - 1/\hat y_j \\ \vdots \end{array}\right] \left[ \begin{array}{ccccc} \hat y_1 (1 - \hat y_1) & -\hat y_1 \hat y_2 & \dots & -\hat y_1 \hat y_j & \dots \\ -\hat y_2 \hat y_1 & \hat y_2 (1 - \hat y_2) & \dots & -\hat y_2 \hat y_j & \dots \\ \vdots & \vdots & \ddots & \vdots & \vdots \\ - \hat y_j \hat y_1 & -\hat y_j \hat y_2 & \dots & \hat y_j (1-\hat y_j) & \dots \\ \vdots & \vdots & \ddots & \vdots & \vdots \end{array}\right] {\bf x}^\top \\
& = \left[\begin{array}{c}\hat y_1 \\ \hat y_2 \\ \vdots \\ \hat y_j - 1 \\ \vdots \end{array} \right] {\bf x}^\top \\
& = (\hat {\bf y} - {\bf y}){\bf x}^\top \\
\\
\frac{\partial l({\bf y},{\hat {\bf y}})}{\partial {\bf b}} & = (\hat {\bf y} - {\bf y}){\bf 1}^\top
\end{align}

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

dataset = 'digits'

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(digitsX_flatten)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, digitsY, test_size=0.2, random_state=42, shuffle=True)
```

```python
# Define some useful functions
def softmax(t):
    """Compute softmax values for each sets of scores in t."""
    e_t = np.exp(t)
    return e_t / e_t.sum(axis=1)[:,np.newaxis]

def forward(w,b,x):
    #Calcula la salida de la red
    return softmax(x.dot(w.T)+b.T)

def backward(y,y_hat,x):
    #Calcula los gradientes
    W_grad = (y_hat-y).T.dot(x)/len(y)
    b_grad = ((y_hat-y).sum(axis=0)[:,np.newaxis])/len(y)
    return W_grad, b_grad

def accuracy(y, y_hat):
    return np.mean(np.argmax(y, axis=1) == np.argmax(y_hat, axis=1))

def loss(y, y_hat):
    return -np.sum(y * np.log(y_hat))/len(y)
```

```python
#Neural Network Training

epochs = 300
rho = .1

#Parameter initialization
W = .1 * np.random.randn(y_train.shape[1], X_train.shape[1])
b = .1 * np.random.randn(y_train.shape[1],1)

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in np.arange(epochs):
    y_hat_train = forward(W, b, X_train)
    y_hat_val = forward(W, b, X_val)
    W_grad, b_grad = backward(y_train, y_hat_train, X_train)
    W = W - rho * W_grad
    b = b - rho * b_grad

    loss_train[epoch] = loss(y_train, y_hat_train)
    loss_val[epoch] = loss(y_val, y_hat_val)
    acc_train[epoch] = accuracy(y_train, y_hat_train)
    acc_val[epoch] = accuracy(y_val, y_hat_val)
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Exercise

* Study the behavior of the algorithm changing the number of iterations and the learning rate

* Obtain the confusion matrix, and study which classes are more difficult to classify

* Think about the differences between using this 10-class network, vs training 10 binary classifiers, one for each class

As in linear logistic regression note that we covered the following aspects of neural network design, implementation, and training:

* Network topology (In this case, a linear network with one layer and $M$ ouptuts)
* Activation functions (softmax activation)
* Parameter initialization ($\bf W$/$b$)
* Obtaining the network prediction using *forward* computation
* Loss function
* Parameter gradient calculus using *backward* computation
* Optimization method for parameters update (here, GD)

###  2.3. Multi Layer Networks (Deep Networks)

Previous networks are constrained in the sense that they can only implement linear classifiers. In this section we analyze how we can extend them to implement non-linear classification:
* Fixed non-linear transformations of inputs: ${\bf z} = {\bf{f}}({\bf x})$

* Parametrize the transformation using additional non-linear layers

* When counting layers, we normally ignore the input layer, since there is no computation involved
* Intermediate layers are normally referred to as "hidden" layers
* Non-linear activations result in an overall non-linear classifier
* We can still use Gradient Descent Optimization as long as the network loss derivatives with respect to all parameters exist and are continuous
* This is already deep learning. We can have two layers or more, each with different numbers of neurons. But as long as derivatives with respect to parameters can be calculated, the network can be optimized
* Finding an appropriate number of layers for a particular problem, as well as the number of neurons per layer, requires exploration
* The more data we have for training the network, the more parameters we can afford, making feasible the use of more complex topologies

### Example: 2-layer network for binary classification

* Network topology
    * Hidden layer with $n_h$ neurons
    * Hyperbolic tangent activation function for the hidden layer
    $${\bf h} = \text{tanh}({\bf o}^{(1)})= \text{tanh}\left({\bf W}^{(1)} {\bf x} + {\bf b}^{(1)}\right)$$
    * Output layer is linear with logistic activation (as in logistic regression)
    $$\hat y = \text{logistic}(o) = \text{logistic}\left({{\bf w}^{(2)}}^\top {\bf h} + b^{(2)}\right)$$

* Cross-entropy loss

$$l(y,\hat y) = -\left[ y \log(\hat y) + (1 - y ) \log(1 - \hat y) \right], \qquad \text{with } y\in [0,1]$$

* Update of output layer weights as in logistic regression (use ${\bf h}$ instead of ${\bf x}$)

$${\bf w}_{n+1}^{(2)} = {\bf w}_n^{(2)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k){\bf h}_k$$
$$b_{n+1}^{(2)} = b_n^{(2)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)$$

* For updating the input layer parameters we need to use the chain rule (we ignore dimensions and rearrange at the end)

\begin{align}\frac{\partial l(y, \hat y)}{\partial {\bf W}^{(1)}} & = \frac{\partial l(y, \hat y)}{\partial o} \cdot \frac{\partial o}{\partial {\bf h}} \cdot \frac{\partial {\bf h}}{\partial {\bf o}^{(1)}} \cdot \frac{\partial {\bf o}^{(1)}}{\partial {\bf W}^{(1)}} \\
& = (\hat y - y) [{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h})^2] {\bf x}^{\top}
\end{align}

\begin{align}\frac{\partial l(y, \hat y)}{\partial {\bf b}^{(1)}} & = \frac{\partial l(y, \hat y)}{\partial o} \cdot \frac{\partial o}{\partial {\bf h}} \cdot \frac{\partial {\bf h}}{\partial {\bf o}^{(1)}} \cdot \frac{\partial {\bf o}^{(1)}}{\partial {\bf b}^{(1)}} \\
& = (\hat y - y) [{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h})^2]
\end{align}

* GD update rules become
$${\bf W}_{n+1}^{(1)} = {\bf W}_n^{(1)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)[{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h}_k)^2] {\bf x}_k^{\top}$$
$${\bf b}_{n+1}^{(1)} = {\bf b}_n^{(1)} + \rho_n \sum_{k=0}^{K-1} (y_k - \hat y_k)[{\bf w}^{(2)} .\ast ({\bf 1}-{\bf h}_k)^2]$$

* The process can be implemented as long as the derivatives of the network overall loss with respect to parameters can be computed

* Forward computation graphs represent how the network output can be computed

* We can then reverse the graph to compute derivatives with respect to parameters

* Deep Learning libraries implement automatic gradient camputation
   * We just define network topology
   * Computation of gradients is carried out automatically

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

#dataset = 'DogsCats'
dataset = 'digits'

if dataset=='DogsCats':
    X = DogsCatsX_flatten
    y = DogsCatsY

else:
    #Zero and Ones are one hot encoded in columns 1 and 4
    X0 = digitsX_flatten[np.argmax(digitsY, axis=1)==1,]
    X1 = digitsX_flatten[np.argmax(digitsY, axis=1)==4,]
    X = np.vstack((X0, X1))
    y = np.zeros(X.shape[0])
    y[X0.shape[0]:] = 1

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(X)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, y, test_size=0.2, random_state=42, shuffle=True)
```

```python
# Define some useful functions
def logistic(t):
    return 1.0 / (1 + np.exp(-t))

def forward(W1,b1,w2,b2,x):
    #Calcula la salida de la red
    h = x.dot(W1.T)+b1
    y_hat = logistic(h.dot(w2)+b2)
    #Provide also hidden units value for backward gradient step
    return h, y_hat

def backward(y,y_hat,h,x,w2):
    #Calcula los gradientes
    w2_grad = h.T.dot(y_hat-y)/len(y)
    b2_grad = np.sum(y_hat-y)/len(y)
    W1_grad = ((w2[np.newaxis,]*((1-h)**2)*(y_hat - y)[:,np.newaxis]).T.dot(x))/len(y)
    b1_grad = ((w2[np.newaxis,]*((1-h)**2)*(y_hat - y)[:,np.newaxis]).sum(axis=0))/len(y)
    return w2_grad, b2_grad, W1_grad, b1_grad

def accuracy(y, y_hat):
    return np.mean(y == (y_hat>=0.5))

def loss(y, y_hat):
    return -np.sum(y*np.log(y_hat)+(1-y)*np.log(1-y_hat))/len(y)
```

```python
#Neural Network Training
epochs = 1000
rho = .05

#Parameter initialization
n_h = 5
W1 = .01 * np.random.randn(n_h, X_train.shape[1])
b1 = .01 * np.random.randn(n_h)
w2 = .01 * np.random.randn(n_h)
b2 = .01 * np.random.randn(1)

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in np.arange(epochs):
    h, y_hat_train = forward(W1, b1, w2, b2, X_train)
    dum, y_hat_val = forward(W1, b1, w2, b2, X_val)
    w2_grad, b2_grad, W1_grad, b1_grad = backward(y_train, y_hat_train, h, X_train, w2)
    W1 = W1 - rho/10 * W1_grad
    b1 = b1 - rho/10 * b1_grad
    w2 = w2 - rho * w2_grad
    b2 = b2 - rho * b2_grad

    loss_train[epoch] = loss(y_train, y_hat_train)
    loss_val[epoch] = loss(y_val, y_hat_val)
    acc_train[epoch] = accuracy(y_train, y_hat_train)
    acc_val[epoch] = accuracy(y_val, y_hat_val)

    if not ((epoch+1)%(epochs/5)):
        print('Número de iteraciones:', epoch+1)
```

### Results in Dogs vs Cats dataset ($epochs = 1000$ and $\rho = 0.05$)

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Results in Binary Sign Digits Dataset ($epochs = 10000$ and $\rho = 0.001$)

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

### Exercises

* Train the network using other settings for:
    * The number of iterations
    * The learning step
    * The number of neurons in the hidden layer

* You may find divergence issues for some settings
    * Related to the use of the hyperbolic tangent function in the hidden layer (numerical issues)
    * This is also why learning step was selected smaller for the hidden layer
    * Optimized libraries rely on certain modifications to obtain more robust implementations

* Try to solve both problems using scikit-learn implementation
    * You can also explore other activation functions
    * You can also explore other solvers to speed up convergence
    * You can also adjust the size of minibatches
    * Take a look at the *early_stopping* parameter

###  2.4. Multi Layer Networks for Regression

* Deep Learning networks can be used to solve regression problems with the following common adjustments

    * Linear activation for the output unit

    * Square loss:
    $$l(y, \hat y) = (y - \hat y)^2, \qquad \text{where} \qquad y, \hat y \in \Re$$

###  2.5. Activation Functions

You can refer to the Dive into Deep Learning book for a more detailed discussion on common actiation functions for the hidden units.

We extract some information about the very important **ReLU** function

> *The most popular choice, due to both simplicity of implementation and its good performance on a variety of predictive tasks, is the rectified linear unit (ReLU). ReLU provides a very simple nonlinear transformation. Given an element $x$, the function is defined as the maximum of that element and 0.*

> *When the input is negative, the derivative of the ReLU function is 0, and when the input is positive, the derivative of the ReLU function is 1. When the input takes value precisely equal to 0, we say that the derivative is 0 when the input is 0.*

> *The reason for using ReLU is that its derivatives are particularly well behaved: either they vanish or they just let the argument through. This makes optimization better behaved and it mitigated the well-documented problem of vanishing gradients that plagued previous versions of neural networks.*

```python
x_array = np.linspace(-6,6,100)
y_array = np.clip(x_array, 0, a_max=None)
plt.plot(x_array, y_array)
plt.title('ReLU activation function')
plt.show()
```

##  3. Implementing Deep Networks with PyTorch

* Pytorch is a Python library that provides different levels of abstraction for implementing deep neural networks

* The main features of PyTorch are:

    * Definition of numpy-like n-dimensional *tensors*. They can be stored in / moved to GPU for parallel execution of operations
    * Automatic calculation of gradients, making *backward gradient calculation* transparent to the user
    * Definition of common loss functions, NN layers of different types, optimization methods, data loaders, etc, simplifying NN implementation and training
    * Provides different levels of abstraction, thus a good balance between flexibility and simplicity

* This notebook provides just a basic review of the main concepts necessary to train NNs with PyTorch taking materials from:
    * Learning PyTorch with Examples, by Justin Johnson
    * What is *torch.nn* really?, by Jeremy Howard
    * Pytorch Tutorial for Deep Learning Lovers, by Kaggle user kanncaa1

###  3.0. Installation and PyTorch introduction

* PyTorch can be installed with or without GPU support
    * If you have an Anaconda installation, you can install from the command line, using the instructions of the project website

* PyTorch is also preinstalled in Google Collab with free GPU access
    * Follow RunTime -> Change runtime type, and select GPU for HW acceleration

* Please, refer to Pytorch getting started tutorial for a quick introduction regarding tensor definition, GPU vs CPU storage of tensors, operations, and bridge to Numpy

###  3.1. Torch tensors (very) general overview

* We can create tensors with different construction methods provided by the library, either to create new tensors from scratch or from a Numpy array

```python
import torch

x = torch.rand((100,200))
digitsX_flatten_tensor = torch.from_numpy(digitsX_flatten)

print(x.type())
print(digitsX_flatten_tensor.size())
```

    torch.FloatTensor
    torch.Size([2062, 4096])

* Tensors can be converted back to numpy arrays

* Note that in this case, a tensor and its corresponding numpy array **will share memory**

* Operations and slicing use a syntax similar to numpy

```python
print('Size of tensor x:', x.size())
print('Tranpose of vector has size', x.t().size()) #Transpose and compute size
print('Extracting upper left matrix of size 3 x 3:', x[:3,:3])
print(x.mm(x.t()).size())  #mm for matrix multiplications
xpx = x.add(x)
xpx2 = torch.add(x,x)
print((xpx!=xpx2).sum())   #Since all are equal, count of different terms is zero
```

    Size of tensor x: torch.Size([100, 200])
    Tranpose of vector has size torch.Size([200, 100])
    Extracting upper left matrix of size 3 x 3: tensor([[0.6252, 0.0318, 0.4039],
            [0.0704, 0.7540, 0.4963],
            [0.1201, 0.1661, 0.0053]])
    torch.Size([100, 100])
    tensor(0)

* Adding underscore performs operations "*in place*", e.g., ```x.add_(y)```

* If a GPU is available, tensors can be moved to and from the GPU device

* Operations on tensors stored in a GPU will be carried out using GPU resources and will typically be highly parallelized

```python
if torch.cuda.is_available():
    device = torch.device('cuda')
    x = x.to(device)
    y = x.add(x)
    y = y.to('cpu')
else:
    print('No GPU card is available')
```

    No GPU card is available

###  3.2. Automatic gradient calculation

* PyTorch tensors have a property ```requires_grad```. When true, PyTorch automatic gradient calculation will be activated for that variable

* In order to compute these derivatives numerically, PyTorch keeps track of all operations carried out on these variables, organizing them in a forward computation graph.

* When executing the ```backward()``` method, derivatives will be calculated

* However, this should only be activated when necessary, to save computation

```python
x.requires_grad = True
y = (3 * torch.log(x)).sum()
y.backward()
print(x.grad[:2,:2])
print(3/x[:2,:2])

x.requires_grad = False
x.grad.zero_()
print('Automatic gradient calculation is deactivated, and gradients set to zero')
```

    tensor([[ 4.7981, 94.4270],
            [42.5862,  3.9788]])
    tensor([[ 4.7981, 94.4270],
            [42.5862,  3.9788]], grad_fn=<MulBackward0>)
    Automatic gradient calculation is deactivated, and gradients set to zero

Exercise

* Initialize a tensor ```x``` with the upper right $5 \times 10$ submatrix of flattened digits
* Compute output vector ```y``` applying a function of your choice to ```x```
* Compute scalar value ```z``` as the sum of all elements in ```y``` squared
* Check that ```x.grad``` calculation is correct using the ```backward``` method
* Try to run your cell multiple times to see if the calculation is still correct. If not, implement the necessary mnodifications so that you can run the cell multiple times, but the gradient does not change from run to run

**Note:** The backward method can only be run on scalar variables

###  3.2. Feed Forward Network using PyTorch

* In this section we will change our code for a neural network to use tensors instead of numpy arrays. We will work with the sign digits datasets.

* We will introduce all concepts using a single layer perceptron (softmax regression), and then implement networks with additional hidden layers

###  3.2.1. Using Automatic differentiation

* We start by loading the data, and converting to tensors.

* As a first step, we refactor our code to use tensor operations

* We do not need to pay too much attention to particular details regarding tensor operations, since these will not be necessary when moving to higher PyTorch abstraction levels

* We do not need to implement gradient calculation. PyTorch will take care of that

```python
from sklearn.preprocessing import MinMaxScaler
from sklearn.model_selection import train_test_split

dataset = 'digits'

#Joint normalization of all data. For images [-.5, .5] scaling is frequent
min_max_scaler = MinMaxScaler(feature_range=(-.5, .5))
X = min_max_scaler.fit_transform(digitsX_flatten)

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(X, digitsY, test_size=0.2, random_state=42, shuffle=True)

#Convert to Torch tensors
X_train_torch = torch.from_numpy(X_train)
X_val_torch = torch.from_numpy(X_val)
y_train_torch = torch.from_numpy(y_train)
y_val_torch = torch.from_numpy(y_val)
```

```python
# Define some useful functions
def softmax(t):
    """Compute softmax values for each sets of scores in t"""
    return t.exp() / t.exp().sum(-1).unsqueeze(-1)

def model(w,b,x):
    #Calcula la salida de la red
    return softmax(x.mm(w) + b)

def accuracy(y, y_hat):
    return (y.argmax(axis=-1) == y_hat.argmax(axis=-1)).float().mean()

def nll(y, y_hat):
    return -(y * y_hat.log()).mean()
```

* Syntaxis is a bit different because input variables are tensors, not arrays

* This time we did not need to implement the backward function

```python
#Parameter initialization
W = .1 * torch.randn(X_train_torch.size()[1], y_train_torch.size()[1])
W.requires_grad_()
b = torch.zeros(y_train_torch.size()[1], requires_grad=True)

epochs = 500
rho = .5

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)
```

```python
# Network training

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = model(W,b,X_train_torch)
    loss = nll(y_train_torch, pred)

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = model(W, b, X_val_torch)
        loss_val[epoch] = nll(y_val_torch, pred_val).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        W -= rho * W.grad
        b -= rho * b.grad
        #Reset gradients
        W.grad.zero_()
        b.grad.zero_()
```

It is important to deactivate gradient updates after the network has been evaluated on training data, and gradients of the loss function have been computed

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

###  3.2.2. Using torch *nn* module

* PyTorch *nn* module provides many attributes and methods that make the implementation and training of Neural Networks simpler

* ```nn.Module``` and ```nn.Parameter``` allow to implement a more concise training loop

* ```nn.Module``` is a PyTorch class that will be used to encapsulate and design a specific neural network, thus, it is central to the implementation of deep neural nets using PyTorch

* ```nn.Parameter``` allow the definition of trainable network parameters. In this way, we will simplify the implementation of the training loop.

* All parameters defined with ```nn.Parameter``` will have ```requires_grad = True```

```python
from torch import nn

class my_multiclass_net(nn.Module):
    def __init__(self, nin, nout):
        """This method initializes the network parameters
        Parameters nin and nout stand for the number of input parameters (features in X)
        and output parameters (number of classes)"""
        super().__init__()
        self.W = nn.Parameter(.1 * torch.randn(nin, nout))
        self.b = nn.Parameter(torch.zeros(nout))

    def forward(self, x):
        return softmax(x.mm(self.W) + self.b)

    def softmax(t):
        """Compute softmax values for each sets of scores in t"""
        return t.exp() / t.exp().sum(-1).unsqueeze(-1)
```

```python
my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])

epochs = 500
rho = .5

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = nll(y_train_torch, pred)

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = nll(y_val_torch, pred_val).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        for p in my_net.parameters():
            p -= p.grad * rho
        #Reset gradients
        my_net.zero_grad()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

* ```nn.Module``` comes with several kinds of pre-defined layers, thus making it even simpler to implement neural networks

* We can also import the Cross Entropy Loss from ```nn.Module```. When doing so:
    - We do not have to compute the softmax, since the ```nn.CrossEntropyLoss``` already does so
    - ```nn.CrossEntropyLoss``` receives two input arguments, the first is the output of the network, and the second is the true label as a 1-D tensor (i.e., an array of integers, one-hot encoding should not be used)

```python
from torch import nn

class my_multiclass_net(nn.Module):
    def __init__(self, nin, nout):
        """Note that now, we do not even need to initialize network parameters ourselves"""
        super().__init__()
        self.lin = nn.Linear(nin, nout)

    def forward(self, x):
        return self.lin(x)

loss_func = nn.CrossEntropyLoss()
```

```python
my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])

epochs = 500
rho = .1

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = loss_func(pred, y_train_torch.argmax(axis=-1))

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

        #Weight update
        for p in my_net.parameters():
            p -= p.grad * rho
        #Reset gradients
        my_net.zero_grad()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

Note faster convergence is observed in this case. It is actually due to a more convenient initialization of the hidden layer

###  3.2.3. Network Optimization

* We cover in this subsection two different aspects about network training using PyTorch:

    + Using ```torch.optim``` allows an easier and more interpretable encoding of neural network training, and opens the door to more sophisticated training algorithms

    + Using minibatches can speed up network convergence

* ```torch.optim``` provides two convenient methods for neural network training:
    - ```opt.step()``` updates all network parameters using current gradients
    - ```opt.zero_grad()``` resets all network parameters

```python
from torch import optim

my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])
opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 500

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    #Compute network output and cross-entropy loss
    pred = my_net(X_train_torch)
    loss = loss_func(pred, y_train_torch.argmax(axis=-1))

    #Compute gradients
    loss.backward()

    #Deactivate gradient automatic updates
    with torch.no_grad():
        #Computing network performance after iteration
        loss_train[epoch] = loss.item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()

    opt.step()
    opt.zero_grad()
```

Note network optimization is carried out outside ```torch.no_grad()``` but network evaluation (other than forward output calculation for the training patterns) still need to deactivate gradient updates

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

###  Exercise

Implement network training with other optimization methods. You can refer to the official documentation and select a couple of methods. You can also try to implement adaptive learning rates using ```torch.optim.lr_scheduler```

* Each epoch of the previous implementation of network training was actually implementing Gradient Descent

* In SGD only a *minibatch* of training patterns are used at every iteration

* In each epoch we iterate over all training patterns sequentially selecting non-overlapping *minibatches*

* Overall, convergence is usually faster than when using Gradient Descent

* Torch provides methods that simplify the implementation of this strategy

```python
from torch.utils.data import TensorDataset, DataLoader

train_ds = TensorDataset(X_train_torch, y_train_torch)
train_dl = DataLoader(train_ds, batch_size=64)
```

```python
from torch import optim

my_net = my_multiclass_net(X_train_torch.size()[1], y_train_torch.size()[1])
opt = optim.SGD(my_net.parameters(), lr=0.1)

epochs = 200

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Current epoch:', epoch+1)

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

###  3.2.4. Multi Layer networks using ```nn.Sequential```

* PyTorch simplifies considerably the implementation of neural network training, since we do not need to implement derivatives ourselves

* We can also make a simpler implementation of multilayer networks using ```nn.Sequential``` function

* It returns directly a network with the requested topology, including parameters **and forward evaluation method**

```python
my_net = nn.Sequential(
    nn.Linear(X_train_torch.size()[1], 200),
    nn.ReLU(),
    nn.Linear(200,50),
    nn.ReLU(),
    nn.Linear(50,20),
    nn.ReLU(),
    nn.Linear(20,y_train_torch.size()[1])
)

opt = optim.SGD(my_net.parameters(), lr=0.1)
```

```python
epochs = 200

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Número de épocas:', epoch+1)

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()
```

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

```python
print('Validation accuracy with this net:', acc_val[-1])
```

    Validation accuracy with this net: 0.8692494034767151

###  3.3. Generalization

* For complex network topologies (i.e., many parameters), network training can incur in over-fitting issues

* Some common strategies to avoid this are:

    - Early stopping
    - Dropout regularization

Image Source

* Data augmentation can also be used to avoid overfitting, as well as to achieve improved accuracy by providing the network some a priori expert knowledge
    - E.g., if image rotations and scalings do not affect the correct class, we could enlarge the dataset by creating artificial images with these transformations

###  3.5. Convolutional Networks for Image Processing

* PyTorch implements other layers that are better suited for different applications

* In image processing, we normally recur to Convolutional Neural Networks, since they are able to capture the true spatial information of the image

Image Source

```python
dataset = 'digits'

#Generate train and validation data, shuffle
X_train, X_val, y_train, y_val = train_test_split(digitsX[:,np.newaxis,:,:], digitsY, test_size=0.2, random_state=42, shuffle=True)

#Convert to Torch tensors
X_train_torch = torch.from_numpy(X_train)
X_val_torch = torch.from_numpy(X_val)
y_train_torch = torch.from_numpy(y_train)
y_val_torch = torch.from_numpy(y_val)

train_ds = TensorDataset(X_train_torch, y_train_torch)
train_dl = DataLoader(train_ds, batch_size=64)
```

```python
class Lambda(nn.Module):
    def __init__(self, func):
        super().__init__()
        self.func = func

    def forward(self, x):
        return self.func(x)

my_net = nn.Sequential(
    nn.Conv2d(1, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 16, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.Conv2d(16, 10, kernel_size=3, stride=2, padding=1),
    nn.ReLU(),
    nn.AvgPool2d(4),
    Lambda(lambda x: x.view(x.size(0), -1)),
)

opt = optim.SGD(my_net.parameters(), lr=0.1)
```

```python
epochs = 2500

loss_train = np.zeros(epochs)
loss_val = np.zeros(epochs)
acc_train = np.zeros(epochs)
acc_val = np.zeros(epochs)

for epoch in range(epochs):

    if not ((epoch+1)%(epochs/5)):
        print('Número de épocas:', epoch+1)

    for xb, yb in train_dl:

        #Compute network output and cross-entropy loss for current minibatch
        pred = my_net(xb)
        loss = loss_func(pred, yb.argmax(axis=-1))

        #Compute gradients and optimize parameters
        loss.backward()
        opt.step()
        opt.zero_grad()

    #At the end of each epoch, evaluate overall network performance
    with torch.no_grad():
        #Computing network performance after iteration
        pred = my_net(X_train_torch)
        loss_train[epoch] = loss_func(pred, y_train_torch.argmax(axis=-1)).item()
        acc_train[epoch] = accuracy(y_train_torch, pred).item()
        pred_val = my_net(X_val_torch)
        loss_val[epoch] = loss_func(pred_val, y_val_torch.argmax(axis=-1)).item()
        acc_val[epoch] = accuracy(y_val_torch, pred_val).item()
```

    Número de épocas: 500
    Número de épocas: 1000
    Número de épocas: 1500
    Número de épocas: 2000
    Número de épocas: 2500

```python
plt.figure(figsize=(14,5))
plt.subplot(1, 2, 1), plt.plot(loss_train, 'b'), plt.plot(loss_val, 'r'), plt.legend(['train', 'val']), plt.title('Cross-entropy loss')
plt.subplot(1, 2, 2), plt.plot(acc_train, 'b'), plt.plot(acc_val, 'r'), plt.legend(['train', 'val']), plt.title('Accuracy')
plt.show()
```

```python

```


# Heteroskedasticity
## Consequences of Heteroskedasticity for OLS

$\DeclareMathOperator*{\argmin}{argmin}
\DeclareMathOperator*{\argmax}{argmax}
\DeclareMathOperator*{\plim}{plim}
\newcommand{\using}[1]{\stackrel{\mathrm{#1}}{=}}
\newcommand{\ffrac}{\displaystyle \frac}
\newcommand{\asim}{\overset{\text{a}}{\sim}}
\newcommand{\space}{\text{ }}
\newcommand{\bspace}{\;\;\;\;}
\newcommand{\QQQ}{\boxed{?\:}}
\newcommand{\void}{\left.\right.}
\newcommand{\Tran}[1]{{#1}^{\mathrm{T}}}
\newcommand{\d}[1]{\displaystyle{#1}}
\newcommand{\CB}[1]{\left\{ #1 \right\}}
\newcommand{\SB}[1]{\left[ #1 \right]}
\newcommand{\P}[1]{\left( #1 \right)}
\newcommand{\abs}[1]{\left| #1 \right|}
\newcommand{\norm}[1]{\left\| #1 \right\|}
\newcommand{\dd}{\mathrm{d}}
\newcommand{\Exp}{\mathrm{E}}
\newcommand{\RR}{\mathbb{R}}
\newcommand{\EE}{\mathbb{E}}
\newcommand{\II}{\mathbb{I}}
\newcommand{\NN}{\mathbb{N}}
\newcommand{\ZZ}{\mathbb{Z}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\AcA}{\mathcal{A}}
\newcommand{\FcF}{\mathcal{F}}
\newcommand{\AsA}{\mathscr{A}}
\newcommand{\FsF}{\mathscr{F}}
\newcommand{\Var}[2][\,\!]{\mathrm{Var}_{#1}\left[#2\right]}
\newcommand{\Avar}[2][\,\!]{\mathrm{Avar}_{#1}\left[#2\right]}
\newcommand{\Cov}[2][\,\!]{\mathrm{Cov}_{#1}\left(#2\right)}
\newcommand{\Corr}[2][\,\!]{\mathrm{Corr}_{#1}\left(#2\right)}
\newcommand{\I}[1]{\mathrm{I}\left( #1 \right)}
\newcommand{\N}[1]{\mathcal{N} \left( #1 \right)}
\newcommand{\ow}{\text{otherwise}}
\newcommand{\FSD}{\text{FSD}}Review$

>Homoskedasticity assumption $\text{MLR}.5$, that $\Var{u\mid x_1,x_2,\dots, x_k} = \sigma^2$, plays no role in showing whether OLS was unbiased or consistent. Only something like omitting an important variable would have this effect.
>
>Also, $R^2$ and $\bar R^2$ are unaffected by the presence of heteroskedasticity. They are the estimators of population $R^2 = 1 - \sigma_u^2/\sigma_y^2$ where the two variances are *un*conditional while heteroskedasticity is under conditioning on $\mathbf{x}$.

Under $\text{MLR}.1$ to $\text{MLR}.4$, $\text{SSR}/n$ consistently estimates $\sigma_u^2$ and $\text{SST}/n$ consistently estimates $\sigma_y^2$. Therefore, $R^2$ and $\bar R^2$ are both consistant estimators of the population $R^2$, whether or not the homoskedasticity assumption holds.

But to do the inference by $t$ statistic and $F$ statistic, we still need that assumption. Besides, if $\Var{u\mid\mathbf{x}}$ is no longer constant, OLS is no longer BLUE.

## Heteroskedasticity-Robust Inference after OLS Estimation

Consider the simple linear regression, the estimator of slope parameter is

$$\hat\beta_1 = \beta_1 + \ffrac{\d{\sum_{i=1}^{n} \P{x_i - \bar x}} u_i}{\d{\sum_{i=1}^{n} \P{x_i - \bar x}^2}}$$

$$\Var{u_i\mid x_i} = \sigma_i^2\;{\Longrightarrow}\;\Var{\hat\beta_1} = \ffrac{\sum \P{x_i - \bar x}^2\sigma_i^2}{\text{SST}_x^2}$$

Now we give a valid estimator for general MLR

$$\widehat{\Var{\hat \beta_j}} = \ffrac{\d{\sum_{i=1}^{n}\hat r_{ij}^2 \hat u_i^2}}{\text{SSR}_j^2}$$

where $\hat r_{ij}$ denotes the $i$th residual from regressing $x_j$ on all other independent variables, and $\text{SSR}_j$ is the sum of squared residuals from this regression. More than that, we have the ***heteroskedasticity-robust standard error*** for $\hat\beta_j$:

$$\sqrt{\widehat{\Var{\hat \beta_j}}}= \ffrac{\d{\sqrt{\sum_{i=1}^{n}\hat r_{ij}^2 \hat u_i^2}}}{\text{SSR}_j^2}$$

Here the $\text{SSR}_j^2$ can be replaced by $\text{SST}_j^2\P{1-R_j^2}$, where $\text{SST}_j^2$ is the total sum of squares of $x_j$, and $R_j^2$ is the usual $R^2$ from regressing $x_j$ on all other explanatory variables.

Then the ***heteroskedasticity-robust $t$ statistic***.

$$t = \ffrac{\text{estimate} - \text{hypothesized value}}{\textbf{heteroskedasticity-robust} \text{ standard error}}$$

Using these formulas, the usual $t$ test is valid asymptotically; but the usual $F$-statistic does not work under heteroskedasticity.

### Computing Heteroskedasticity-Robust LM Tests

Skipped, however. OK, the usual LM statistic:

1. Estimate the restricted model to obtain the residual $\tilde u$
2. regress $\tilde u$ on all of the independent variables
3. $\text{LM} = n \cdot R_{\tilde u}^2$, where $R_{\tilde u}^2$ is the $R^2$ from this regression

Then the HR LM statistic in the general case

1. The same: estimate the restricted model to obtain the residuals $\tilde u$
2. Regress each of the independent variables which are excluded under the null hypothesis, on all of the included variables; say there're $q$ excluded variables (restricted model for them: $x_j = \beta_0 + \beta_1 x_1 + \cdots + \beta_{k-q} x_{k-q} + u$, $j = k-q+1,\dots,k$)
3. Obtain $q$ sets of residuals $\P{\tilde r_1,\tilde r_2,\dots, \tilde r_q}$, then find the element-wise production of $\tilde r_j$ and $\tilde u$
4. Regress $y\equiv 1$ on $\beta = 0, \tilde r_1 \tilde u, \tilde r_2 \tilde u,\dots,\tilde r_q \tilde u$
5. The **Heteroskedasticity-Robust LM Statistic** is now $n-\text{SSR}_1$ (I bet this is a minus sign, it could be some other signs though...), where $\text{SSR}_1$ is just the usual sum of squared residuals from the regression in the final step.

By the way, under $H_0$, $\text{LM}$ is distributed approximately as $\chi_q^2$

## Testing for Heteroskedasticity

Model: $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$; assumption: $\text{MLR}.1$ through $\text{MLR}.4$, so that the OLS estimators are still unbiased and consistent.

To test the heteroskedasticity, we have $H_0: \Var{u\mid x_1,x_2,\dots,x_k} = \sigma^2$. Since $u$ has a zero conditional expectation, this is equivalent to

$$H_0: \Exp\SB{u^2\mid x_1,\dots,x_k} = \Exp\SB{u^2} = \sigma^2$$

So we are actually testing weather $u^2$ is related (in expected value) to one or more of the explanatory variables. Then we assume the linear function

$$u^2 = \delta_0 + \delta_1 x_1 + \delta_2 x_2 + \cdots + \delta_k x_k + v$$

where $v$ is an error term with mean zero given the $x_j$. Then we rewrite the null hypothesis of homoskedasticity as $H_0: \delta_1 = \delta_2 = \cdots = \delta_k = 0$.

Then we can use $F$ statistic or $\text{LM}$ to test this. And to do so, we first need to estimate the left side, the residual and since it's unable to obtain, we will use its estimation $\hat u_i$ so actually the equation to be process is actually

$$\hat u^2 = \delta_0 + \delta_1 x_1 + \delta_2 x_2 + \cdots + \delta_k x_k + \text{error}$$

Then apply the $F$ test or $\text{LM}$ test. And to distinguish two different $R$-square, we denote the $F$ statistic in this regression

$$F = \ffrac{\ffrac{R_{\hat u^2}^2}{k}}{\ffrac{1-R_{\hat u^2}^2}{n-k-1}}$$

And the $\text{LM}$ statistic is $\text{LM} = n\cdot R_{\hat u^2}^2$. Under $H_0$, it's distributed asymptotically as $\chi_k^2$. The $\text{LM}$ version of the test is typically called the ***Breusch-Pagan test for heteroskedasticity (BP test)***.

$Remark$

>Larger $R_{\hat u^2}^2$ could be the evidence against the null hypothesis.

**Steps for BP test**:

1. Estimate the model $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$ by OLS, as usual. And for each observation, find the residual $\hat u^2$
2. Regression the equation $\hat u^2 = \delta_0 + \delta_1 x_1 + \delta_2 x_2 + \cdots + \delta_k x_k + \text{error}$, with the $R$-squared $R_{\hat u^2}^2$.
3.  Form either the $F$ statistic or the $\text{LM}$ statistic and compute the $p$-value. Use $F_{k,n-k-1}$ distribution for $F$ statistic and $\chi_k^2$ distribution for the otherone. If the $p$-value is below the chosen significance level, meaning that it's sufficiently small, we reject the null hypothesis and admit the heteroskedasticity.

$Remark$

> If we suspect that heteroskedasticity depends only upon certain independent variables, we can simple modify the BP test that we regress $\hat u^2$ only on the chosen variables and then carry out the appropriate $F$ or $\text{LM}$ test.

### The White Test for Heteroskedasticity

The ***White test for heteroskedasticity*** is the $\text{LM}$ statistic for testing that all of the $\delta_j$ where $j=1,2,\dots$ (no $0$ here), defined by

$$\begin{align}
\hat u^2 &= \P{y - \hat\beta_0 - \hat\beta_1 x_1 - \cdots - \hat\beta_k x_k}^2 \\
&\equiv \delta_0 + \sum_{i=1}^k \delta_i x_i + \sum_{i=1}^k \delta_{k+i} x_i^2  + \sum_{j<k} \delta_{i\P{j,k}} x_jx_k + \text{error}
\end{align}$$

$F$ statistics can also do, however one drawback is that this method is of too many parameters so that with less $df$. So here's a better one.

$$\hat u^2 = \delta_0 + \delta_1 \hat y + \delta_2 \hat y^2 + \text{error}$$

$H_0: \delta_1 = \delta_2 = 0$; $\text{LM} = n \cdot R_{\hat u^2}^2\sim \chi_2^2$

**Case**:

1. Estimate the model $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$ by OLS, as usual. And for each observation, find the residual $\hat u$ and fitted value $\hat y$; then use the two to compute $\hat u^2$ and $\hat y^2$
2. Regress $\hat u^2 = \delta_0 + \delta_1 \hat y + \delta_2 \hat y^2 + \text{error}$. Keep the $R$-squared from this regression, $R_{\hat u^2}^2$
3. Form either the $F$ or $\text{LM}$ statistic and compute the $p$-value. Use $F_{2,n-3}$ or $\chi_2^2$.

## Weighted Least Squares Estimation

If we have correctly specified the form of the variance, then weighted least squares (WLS) is more efficient than OLS, and WLS leads to new $t$ and $F$ statistics that have $t$ and $F$ distributions.

### The Heteroskedasticity is Known up to a Multiplicative Constant

Model: $y = \beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$, and let $\mathbf{x}$ denote all the explanatory variables. We assume that

$$\Var{u\mid\mathbf{x}} = \sigma^2 h\P{\mathbf x}$$

where $h\P{\mathbf x}$ is some function of the explanatory variables that determines the heteroskedasticity. It's *known* and *positive*. Only the population parameter $\sigma^2$ is required to be estimated. Then we take the original equation

$$y_i = \beta_0 + \beta_1 x_{i1} + \beta_2 x_{i2} + \cdots + \beta_k x_{ik} + u_i$$

We'll transform it into an equation that has homoskedastic errors under Gauss-Markov Assumptions.

$$\Exp\SB{u_i\mid\mathbf x_i} = 0\Rightarrow\Exp\SB{\ffrac{u_i}{h_i}\mid\mathbf x_i} = 0$$

$$\Var{u_i\mid \mathbf x_i} = \Exp\SB{u_i^2\mid \mathbf x_i} = \sigma^2 h_i$$

$$\Exp\SB{\P{\ffrac{u_i}{\sqrt{h_i}}}^2\mid\mathbf x_i} = \ffrac{\Exp\SB{u_i^2\mid\mathbf x_i}}{h_i} = \ffrac{\sigma^2 h_i}{h_i} = \sigma^2$$

Then inspired by this, we divide $\sqrt{h_i}$ on both sides of the equation.

$$\ffrac{y_i}{\sqrt{h_i}} = \ffrac{\beta_0}{\sqrt{h_i}} + \beta_1\P{\ffrac{x_{i1}}{\sqrt{h_i}}} + \beta_2\P{\ffrac{x_{i2}}{\sqrt{h_i}}} + \cdots + \beta_k\P{\ffrac{x_{ik}}{\sqrt{h_i}}}+ \ffrac{u_i}{\sqrt{h_i}}$$

and we'll rewrite this as

$$y_i^* = \beta_0x_{i0}^* + \beta_1x_{i1}^* + \cdots + \beta_kx_{ik}^* + u_{i}^*,\bspace x_{i0}^* = \ffrac{1}{\sqrt{h_i}}$$

then we can use the Gauss-Markov assumptions. More than that, $\text{MLR}.6$ can be kept holding true after that transform.

The estimators $\beta_j^*$ here are examples of ***generalized least squares (GLS) estimators***. And WLS is one special case that is used to account for heteroskedasticity in the errors. Mathematically, the WLS estimators are the values of the $b_j$ that make

$$\sum_{i=1}^{n}\ffrac{1}{h_i} \P{y_i - b_0 - b_1 x_{i1} - \cdots - b_kx_{ik}}^2$$

as small as possible.

***
One special case that when the observations are reported as averages of certain level, the variables should be weighted by the size of the unit. A simple example:

$$\overline{\text{contrib}}_i = \beta_0 + \beta_1 \overline{\text{earns}}_i + \beta_2 \overline{\text{age}}_i + \beta_3 \text{mrate}_i + \overline u_i$$

It can be considered an example for a average from a firm $i$'s total $m_i$ workers, where $\overline u_i = m_i^{-1} \sum_{e=1}^{m_i} u_{i,e}$.

If, from the respect of individuals, the data are homoskedastic, $\Var{u_{i,e}} = \sigma^2$, then from the respect of firms, we have $\Var{\bar u_i} = \sigma^2/m_i$.

### The Heteroskedasticity Function Must Be Estimated: Feasible GLS

For the unknown heteroscedasticity function, we assume a general one

$$\Var{u\mid \mathbf x} = \sigma^2 \exp\CB{\delta_0 + \delta_1 x_1 + \cdots + \delta_k x_k} = \sigma^2 h\P{\mathbf x}$$

To estimate $\delta_j$, we write

$$u^2 = \sigma^2 \exp\CB{\delta_0 + \delta_1 x_1 + \cdots + \delta_k x_k}v$$

Further, if we assume that $\Exp\SB{v \mid \mathbf x} = 1$, we have

$$\log\P{u^2} = \alpha_0 + \delta_1 x_1 + \cdots + \delta_k x_k + e$$

where $e$ has a zero mean expectation and is independent from $\mathbf x$. This is the one that satisfies all Gauss-Markov assumptions so that we can obtain $\hat \delta_j$. Denote the fitted value in this regression as $\hat g_i$ then we have the weight:

$$\ffrac{1}{h_i} = \ffrac{1}{\hat h_i} = e^{-\hat g_i}$$

**Step**:

1. Regress $y$ on $x_1,x_2,\dots,x_k$ and obtain the residual $\hat u$
2. Obtain $\log\P{\hat u^2}$
3. Regress $\log\P{u^2} = \alpha_0 + \delta_1 x_1 + \cdots + \delta_k x_k + e$, then let $\hat g_i$ denote the fitted value
4. $\hat h = \exp\CB{\hat g} = \exp\CB{\hat \alpha_0 + \hat\delta_1 x_1 + \cdots + \hat\delta_k x_k}$
5. Use weight $1/\hat h$ and then apply WLS on $y =\beta_0 + \beta_1 x_1 + \cdots + \beta_k x_k + u$
***
Feasible GLS is consistent and asymptotically more efficient than OLS.
***

### What if the Assumed Heteroskedasticity Function is Wrong?

- WLS is still consistent under $\text{MLR}.1$ through $\text{MLR}.4$
- robust standard errors should be computed
- WLS is consistent under $\text{MLR}.4$ but not necessarily under $\text{MLR}.4'$

### Prediction and Prediction Intervals with Heteroskedasticity

## The Linear Probability Model Revisited

$$\Var{y\mid \mathbf x} = p\P{\mathbf x} \P{1-p\P{\mathbf x}}\Rightarrow \hat h_i = \hat y_i \P{1-\hat y_i}$$

***


# MAE 3120 Methods of Engineering Experiments

>__Philippe M Bardet__

>__Mechanical and Aerospace Engineering__

>__The George Washington University__

# Module 01: Introduction to measurement system

This class will be focused mainly on experiments, but many of the concepts seen here are also applied in many other fields where analytical thinking is needed.  In fact, with the advancement of computers one talks more and more of numerical experiments.  The stock market could also be considered as a huge real time experiment...

This is an introductory lecture and we need to establish a common (rigorous) language for the rest of the class (and your career).  We are going to introduce a lot of definitions that we will use for the rest of the semester.  Many definitions and notions introduced here should only be reviews at this point (I hope).

Along with the common language, we will also adopt a notation convention that is consistent throughout the class.  Depending on the textbook you choose to follow, the convention might be slightly different than the one adopted here.

## DIKW pyramid

In engineering, knowledge could be defined as a model that describes and (ideally) predicts the behavior of a complex system.  The context of data acquisition, analysis, and model development can be put in the context of the wisdom pyramid (or DIKW pyramyd).  This concept has been developed over the years by the information theory field.

https://en.wikipedia.org/wiki/DIKW_pyramid

Starting from the bottom of the pyramid:

__data__: Signal reading from sensor/transducer

__information__: "organized or structured data, which has been processed in such a way that the information now has relevance for a specific purpose or context, and is therefore meaningful, valuable, useful and relevant."  information + data: "know what".

__knowledge__: "organization and processing to convey understanding, experience, and accumulated learning".  It can be seen as having an engineering model that describes a phenomena or system. "Why is".

__wisdom__: "Why do".  Applicability of model to predict new behaviors.

Use example of taking data at 3 points, to extract mean values, from mean values, trend line, and then knowing when to use the trendline.

Here is another representation of the pyramid:

## Goals of experiments

Experiments serve two main purposes:

>__1- Engineering/scientific experimentation__:
>The goal is to seek new information.  For example when developing a new product one needs to know: how hot does it get? When will it fail?  Another example would be to determine a model that describes the behavior of a system.

>__2- Operational system__:
>The goal is to monitor and control processes. In other words to create a reliable operational sysem. This is generally applied to existing equipment (or equipment under design), rather than used to design a new equipment (first application of measurement).  For example, this could be the A/C control system of a room: one needs to measure temperature and regulate the heating/cooling based on a set point.

It is convenient to think of the measurement process with a block diagram.

## A brief history of measurement (length)

In Ancient Egypt, (~3,000 BC) people used a measure called a CUBIT.  This word comes from a Latin word “cubitum” which meant elbow and it was the length of a person’s outstretched forearm -  from the elbow to the tip of the middle finger.  It was based on the Pharaoh’s body: the ROYAL CUBIT.  A stick was marked with this distance and copies were distributed to merchants throughout the land.  When the Pharaoh died, a new Pharaoh took the throne, a new Royal Cubit came into being, and a new stick and it’s copies had to be made and sent across the land.  Plus, there were OTHER problems:  The lengths that people wanted to measure were sometimes shorter or longer than a cubit.  So other lengths like the PALM, DIGIT, and FOOT were used.  7 PALMS was the same as 1 CUBIT.  The ROYAL FOOT was equal to about 18 fingers or ⅔ of a Royal Cubit.

Alongside the Ancient Egyptians, people throughout the Mediterranean used parts of their bodies to create units of measurement.  Mediterranean sailors used Fathoms to measure depth.  The Hebrew people of long ago used a measurement called a SPAN.  Hand spans are still a unit used to measure horses!  The people of Ancient Greece adapted the Egyptian-Hebrew measurements and added more measurements based upon multiples of fingers.  Ancient Romans created a measurement meant based on the width of the thumb or “uncia” in Latin.  That’s where the word for INCH comes from.  Roman armies measured the distance from one step to another using PACES.  A PACE was equal to 2 Egyptian cubits and is still used to describe speed in foot races.  The word MILE comes from the Latin word “milliare,” a distance of 1,000 paces covered by the Roman army at a forced march.

As the Roman empire spread, the Roman measuring units became the accepted system of measurement throughout Europe.  A Roman FOOT was equal to 12 UNCIA.  These uncia came to England and over time became INCHES.  Like the Pharaohs of Ancient Egypt, the King of England at the time also wanted to STANDARDIZE (or make the same) the units of measurement across the land based on his own body.  A royal decree went out: a YARD was to be the distance from the nose to the tip of the middle finger of the outstretched arm...or about 3 FEET.  From this time, all units were derived from the King’s foot and yard.  The English Mile was derived from that.  This English or Customary system spread back down through Europe and across the ocean to North America with the English who arrived here.  As trade and communication increased again, once again a more uniform or regulated system was needed.  It took several hundred years for the Customary system to become more and more dependable and standardized.  In 1855, a new distance for a yard was formalized.  This was still about the size of the original Roman yard.  Since then all of the other units of measurement for length have been derived from multiplications and divisions of the Yard.  1 Inch was 1/36 of a yard.  1 Foot was ⅓ yard.

While the English or Customary system came to be widely used in Europe, and is still used here in the United States, today there is another widely used system of measurement that is not based on the measurements of the human body.  It’s beginnings can be found in the 1600s in Europe, when people began to talk about finding better STANDARDS for measurement.  In the 1790s, a group of French scientists decided to create a new standard of measurement that would be unchangeable.  The name for the unit of measure chosen was the METER; it was based upon a measurement of the Earth.  The system they developed is called the METRIC SYSTEM and it is still in use today, throughout France, Europe and almost every nation in the world.  All scientists use the Metric system because it is the most precise.  The length of the meter was taken to be one ten-millionth of the distance from the North Pole to the equator along a line of longitude near Paris, France.  The word meters comes from the Greek word “metron” which means to measure.  The metric system is based on multiples of 10.

## Dimensions and Units
For measured data to be useful, one needs to have a common language: i.e. a unique definition of dimensions (length, time, etc.) with associated dimensional units (meter, second, etc.).

There are two types of dimensions:

>__1- Primary or base.__ 7 in total.

\begin{array}{l l l}
\hline
\mathrm{primary\, dimension} & \mathrm{symbol} & \mathrm{unit} \\
\hline
\text{mass} & m & \mathrm{kg}\\
\mathrm{length} & L & \mathrm{m}\\
\mathrm{time} & t & \mathrm{s}\\
\mathrm{thermodynamic\,temperature} & T& \mathrm{K}\\
\mathrm{electrical \,current} & I & \mathrm{A}\\
\mathrm{amount\,of\,light} & C& \mathrm{Cd,\,Candela}\\
\mathrm{amount\,of\,matter} & mol & \mathrm{mole}\\
\hline
\end{array}

>__2- Secondary or derived.__  They are made of a combination of primary/base dimensions

\begin{equation}
\mathrm{force} = \frac{ m \times L}{t^2}
\end{equation}

All other dimensions and units can be derived as combinations of the primaries.  Here is a table with examples.

\begin{array}{l l l l}
\hline
\mathrm{secondary\, dimension}  & \mathrm{Symbol} & \mathrm{unit} & \text{unit name}\\
\hline
\mathrm{force} &  F & \mathrm{N= kg \cdot m/s^2} & \text{Newton}\\
\mathrm{pressure} & P \,(p) & \mathrm{Pa = N/m^2} & \text{Pascal}\\
\mathrm{energy} & E & \mathrm{J = N \cdot m = kg \cdot m^2 / s^2} & \text{Joule} \\
\mathrm{power} & \dot{W} \, (P) &  \mathrm{W = N \cdot m / s = kg \cdot m^2 / s^3} & \text{Watt}\\
\hline
\end{array}

To avoid confusions we will use the SI (International Standard) system of units.

Please also note the notation in how we report the units and symbols.  We will use the same notation throughout the class and you will also throughtout your career.  In scientific notation:
> - mathematical symbols are reported as italic (e.g. temperature $T$, pressure $P$, velocity $U$),
> - while units are reported in roman fonts and with a space in front of the value it characterizes (e.g. $P$ = 100 Pa, $U$ = 5 m/s, $T$ = 400 K).

Anecdote: the Mars Climate Orbiter crashed in 1990's due to a problem of unit conversion.  Source of the failure (from official report): ''failure using metric units''.

While we will use the SI system in the class it is useful to know how to convert dimensions from one unit system to another (i.e. imperial to SI).  Here are some useful quantities to keep handy.

### Unit conversion

\begin{array}{l l l}
\hline
\mathrm{length} & & \\
\hline
1 \,\mathrm{in} & = & 25.4 \times 10^{-3}\,\mathrm{m}\\
1 \,\mathrm{ft} & = & 0.3048 \,\mathrm{m} \\
 & = & 12 \,\mathrm{in}\\
1\, \AA & = & 10^{-10}\,\mathrm{m} \\
1\,\mathrm{mile \,(statute)} & = & 1,609 \,\mathrm{m}\\
1 \,\mathrm{mile \,(nautical)} & = & 1,852 \,\mathrm{m} \\
%
\hline
\mathrm{volume} & & \\
\hline
1 \,\mathrm{l\, (liter)} & = & 10^{-3}\,\mathrm{m}^3 \\
1\,\mathrm{ in}^3 & = & 16.387 \,\mathrm{cm}^3\\
1 \,\mathrm{gal\, (U.S.\, liq.)} & = & 3.785\,\mathrm{l} \\
1 \,\mathrm{gal\, (U.S.\, dry)} & = & 1.164\,\mathrm{ U.S.-liq.\, gal}\\
1 \,\mathrm{gal \,(British)} & = & 1.201\,\mathrm{ U.S.-liq.\, gal}\\
%
\hline
\mathrm{mass} & &\\
\hline
1 \,\mathrm{lb \,(mass)} & = & 0.454\,\mathrm{ kg}\\
%
\hline
\mathrm{force}& &\\
\hline
1 \,\mathrm{N }& = & 1\,\mathrm{ kg\cdot m/s}^2\\
1\,\mathrm{ dynes} & = & 10^{-5}\,\mathrm{ N}\\
1 \,\mathrm{lb \,(force)} & = & 4.448 \,\mathrm{N}\\
%
\hline
\mathrm{energy} & & \\
\hline
1 \,\mathrm{J }& = & 1\, \mathrm{kg \cdot m}^2/\mathrm{s}^2\\
1 \,\mathrm{BTU} & = & 1,055.1\,\mathrm{ J}\\
1 \,\mathrm{cal} & \equiv & 4.184\,\mathrm{ J}\\
1 \,\mathrm{kg-TNT} & \equiv & 4.184\,\mathrm{ MJ}\\
%
\hline
\mathrm{power} & & \\
\hline
1 \,\mathrm{W} & \equiv & 1 \,\mathrm{J/s}\\
1 \,\mathrm{HP\, (imperial)} & \equiv & 745.7 \,\mathrm{W}\\
1 \,\mathrm{HP\, (metric)} & \equiv & 735.5 \,\mathrm{W}\\
\hline
\end{array}

### Dimensionless numbers

\begin{array}{l l l}
\mathrm{Reynolds\, number} & Re & U L / \nu \\
\mathrm{Mach\, number} & M & U/a \\
\mathrm{Prandtl\, number} & Pr & \mu c_p / k = \nu / \kappa \\
\mathrm{Strouhal\, number} & St & L/U \tau  \\
\mathrm{Knudsen\, number} & Kn & \Lambda / L \\
\mathrm{Peclet\, number} & Pe & U L / \kappa = Pr \cdot Re \\
\mathrm{Schmidt\, number} & Sc & \nu / D \\
\mathrm{Lewis\, number} & Le & D / \kappa \\
\end{array}

### Useful constants

Avogadro's number:
\begin{align*}
N_A & = 6.022\, 1367 \times 10^{23} \mathrm{\, molecules/(mol)} %\nolabel
\end{align*}

Boltzman constant:
\begin{align*}
k_B & = 1.380\, 69 \times 10^{-23} \mathrm{\, J/K} \\
k_B T & = 2.585 \times 10^{-2} \mathrm{\,eV} \sim \frac{1}{40} \mathrm{\, eV, at \,} T = 300 \mathrm{K}
\end{align*}

Universal gas constant:
\begin{align*}
R_u & = 8.314\, 510 \mathrm{\, J/(mol} \cdot \mathrm{K)}
\end{align*}

Earth radius (at equator):
\begin{align*}
r_{earth} & = 6\,378.1370 \mathrm{\, km}
\end{align*}

## Dimensional analysis

Now that we know the primary dimensionns, we can make use of it to reduce the number of experimental runs one needs to perform.  This is the foundation of dimensional analysis, which you have seen in MAE 3126 (Fluid Mechanics).  To reduce the number of experimental runs will see other techniques, such as Taguchi arrays in a few weeks when we treat design of experiments.  The benefit of dimensional analysis is best seen through the graph below:

Please review your notes of Fluid Mechanics on dimensional analysis and the  method of repeating variables (also called Buckingham $\Pi$ theorem)

## Errors and Uncertainties

An __error__ is defined as:

\begin{align*}
\epsilon = x_m - x_{true}
\end{align*}

where $x_m$ is the measured value and $x_{true}$ the true value.  The problem is that we do not always (rarely in fact) know the true value.  This will lead to the concept of uncertainty.

Errors can be categorized into two types:

>__1- Systematic or bias error__: Those are errors that are consistent or repeatable.  For example, I use a ruler with the first 3 mm missing, all the measurements will be short by 3 mm.

>__2- Random or precision error__: errors that are inconsistent or unrepeatable.  This will be seen as scatter in the measured data.  For examl]ple, this could be caused by electo-magnetic noise in a voltmeter (with implication on grounding and shielding of the instrument).

Systematic vs random errors can be best seen visually:

In light of the two types of errors defined above, one would like to define mathematical formulas to quantify them.

__systematic/bias error__

\begin{align*}
\epsilon_b = x_m - x_{true}
\end{align*}

_Question_: What are sources of bias errors?
How can we reduce bias errors?

__relative mean bias error__: non-dimensional (normalized) form of the mean bias error.

\begin{align*}
\frac{<x_m> - x_{true}}{x_{true}}
\end{align*}

__random/precision error__

\begin{align*}
\epsilon_p = x_m - <x_m>
\end{align*}

_example_: We have five temperature measurements:  Can you find the maximum precision error?

The __overal precision error or standard error__ is found by computing the standard deviation, $S$, divided by the square root of the number of samples, $n$.

\begin{align*}
\epsilon_{op} = \frac{S}{\sqrt{n}}
\end{align*}

```python
import numpy

T=[372.80, 373.00, 372.90, 373.30, 373.10]

Tm = numpy.mean(T)
print(Tm)
ep_max = 373.30-Tm
print(ep_max)
```

    373.02
    0.28000000000002956

_Question_: Are five measurement enough to quantify the precision error?  We also need a statistics that represents the __mean__ precision error.  We will see this soon.

Now that we have described the two types of errors and hinted at how to estimate them, let's look at instruments specifically.

### Calibration

Calibration aims to determine and imrove its accuracy.  Calibration can be accomplished in 3 manners: 1- comparison with a primary standard (such as developed by NIST, like the mass or meter defined earlier), 2- a secondary standard, such as another instrument of known and higher accuracy, 3- a known input source.

Here are examples of primary standards for temperature that are ''easy'' to implement in any labs:
\begin{array}{l l}
\hline
\mathrm{definition} & \text{temperature (K)}\\
\hline
\mathrm{triple\, point\, of\, hydrogen} & 13.8033 \\
\mathrm{triple\, point\, of\, oxygen} & 54.3584\\
\mathrm{triple\, point\, of\, water} & 273.16 \\
\mathrm{Ice\, point} & 273.15 \\
\mathrm{normal\, boiling\, point\, of\, water} & 373.15 \\
\hline
\end{array}

Calibration can be done in a static and/or dynamic manner.  It is a very important step to verify the accuracy of an instrument or sensor.  In most laboratories, calibration has to be performed regularly.  Some commercial entities are specialised in doing so.

Before each important test campaign or after a company recertify instrument, the results have be to be documented for traceability.  Here is an example for load balance.

### Uncertainty

The concept of __uncertainty__ needs to be taken into account when we conduct experiments.  Uncertainty can be defined as (S.J. Kline) ''What we think the error would be if we could and did measure it by calibration''.  Taking data is a very small part of doing an experiment and we are going to spend a lot of time doing uncertainty analysis.

An error, $\epsilon$ has a particular sign and magnitude (see the equations above). If it is known, then if can be removed from the measurments (through calibration for example).  Any remaining error that does not have a sign and mangitude cannot be removed.  We will define an uncertainty $\pm u$ as the range that contains the remaining (unknown) errors.

Because we are doing measurement in an uncertain world, we need to be able to express our __confidence level__ in our results.  This wil require set of sophisticated statistical tools.

__Uncertainty analysis__ is an extremely important tool and step in experiments and we are going to spend a significant amount of time on it during the class.  This analysis is performed typically in the experimental planning phase (to help in determining the appropriate components to use in our instrumentation chain see in the first figure).  An extensive analysis is also performed after the campaign to characterize the actual uncertainties in the measurement.

### Instrument rating

When selecting an instrument for a measurement, one has many options.  Ideally, one would like to select a system that will meet our requirements for the measurement (such as expected range, but also for uncertainty) while not breaking the bank... Luckily manufacturers report a lot of data with their sensor/transducer that can help us in making an educated guess of the expected performance without having to characterize it ourselves.  Here is an example from Omega Scientific for a pressure transducer:

Let's define a few of the terms used.

__Accuracy__ is difference between true and measured value.

\begin{align*}
u_a = x_m - x_{true}
\end{align*}

A small difference between true and measured value leads to a high accuracy and vice-versa.  It can be expressed as a percentage of reading, of full-scale, or an absolute value.  Accuracy can be assessed and minimized by calibrating the system.

__Precision__ of an instrument is the reading minus the average of readings.  It characterizes random error of instrument output or the reproducibility of an instrument.

\begin{align*}
u_p = x_m - <x_m>
\end{align*}

_Questions_:

>Can we improve precision by calibrating the system?

> Is there a limit up to which we can improve the accuracy of a system?

If we recall our previous definitions for errors, you will remark that they are the same than the two terms introduced above and imply that we compare the instrument readings to the true, known, value.  However, in most cases, we do not know the true value and instead we are only confident that we are within a certain range ($\pm$) of the true value.  Therefore to be consistent with the definitions introduced so far, we should use the term uncertainty and not error, when describing experimental results (except for a few cases).

Accuracy and precision are the two main categories of uncertainties in our measurements; however, they are each comprised of elemental components.  A non-exhaustive list includes:

__resolution__: smallest change or increment in the measurand that the instrumente can detect. Note for digital instrument, resolution is associated with the number of digits on display, ie a 5 digit Digital Multi-Meter (DMM) has better resolution than a 4 digit DMM.  The values reported by the DMM will be at $\pm$ the last digit.

__sensitivity__: it is defined

Other sources of errors are zero, linearity, sensitivity, hysteresis, etc.

## Experiment planning

```python

```