# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.
    	There will be an option for Colab users and another for Jupyter (local) users.

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    ('X_val: ', (1000, 3, 32, 32))
    ('X_train: ', (49000, 3, 32, 32))
    ('X_test: ', (1000, 3, 32, 32))
    ('y_val: ', (1000,))
    ('y_train: ', (49000,))
    ('y_test: ', (1000,))

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
    ('  means: ', array([ -2.3814598 , -13.18038246,   1.91780462]))
    ('  stds:  ', array([ 27.18502186,  34.21455511,  37.68611762]))
    ()
    After batch normalization (gamma=1, beta=0)
    ('  means: ', array([  4.21884749e-17,   5.10702591e-17,  -3.51801921e-17]))
    ('  stds:  ', array([ 0.99999999,  1.        ,  1.        ]))
    ()
    ('After batch normalization (gamma=', array([ 1.,  2.,  3.]), ', beta=', array([ 11.,  12.,  13.]), ')')
    ('  means: ', array([ 11.,  12.,  13.]))
    ('  stds:  ', array([ 0.99999999,  1.99999999,  2.99999999]))
    ()

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
    ('  means: ', array([-0.03927354, -0.04349152, -0.10452688]))
    ('  stds:  ', array([ 1.01531428,  1.01238373,  0.97819988]))
    ()

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
#print(dx)
#print(dx_num)
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    ('dx error: ', 1.7029261167605239e-09)
    ('dgamma error: ', 7.4204142162470875e-13)
    ('dbeta error: ', 2.8795057655839487e-12)

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    ('dx difference: ', 1.0776809016429724e-12)
    ('dgamma difference: ', 0.0)
    ('dbeta difference: ', 0.0)
    speedup: 2.74x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    ('Running check with reg = ', 0)
    ('Initial loss: ', 2.2611955101340957)
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 3.92e-10
    b1 relative error: 2.22e-08
    b2 relative error: 7.63e-09
    b3 relative error: 1.01e-10
    beta1 relative error: 7.85e-09
    beta2 relative error: 1.07e-09
    gamma1 relative error: 6.44e-09
    gamma2 relative error: 1.56e-09
    ()
    ('Running check with reg = ', 3.14)
    ('Initial loss: ', 6.9965332201083026)
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 2.23e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 4.23e-09
    gamma1 relative error: 5.94e-09
    gamma2 relative error: 3.72e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.315000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.386000; val_acc: 0.279000
    (Iteration 41 / 200) loss: 2.041103
    (Epoch 3 / 10) train acc: 0.494000; val_acc: 0.307000
    (Iteration 61 / 200) loss: 1.753902
    (Epoch 4 / 10) train acc: 0.532000; val_acc: 0.309000
    (Iteration 81 / 200) loss: 1.246584
    (Epoch 5 / 10) train acc: 0.575000; val_acc: 0.314000
    (Iteration 101 / 200) loss: 1.323436
    (Epoch 6 / 10) train acc: 0.648000; val_acc: 0.325000
    (Iteration 121 / 200) loss: 1.135111
    (Epoch 7 / 10) train acc: 0.691000; val_acc: 0.324000
    (Iteration 141 / 200) loss: 1.180117
    (Epoch 8 / 10) train acc: 0.750000; val_acc: 0.312000
    (Iteration 161 / 200) loss: 0.693384
    (Epoch 9 / 10) train acc: 0.811000; val_acc: 0.336000
    (Iteration 181 / 200) loss: 0.934761
    (Epoch 10 / 10) train acc: 0.772000; val_acc: 0.305000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696059
    (Epoch 6 / 10) train acc: 0.535000; val_acc: 0.345000
    (Iteration 121 / 200) loss: 1.557987
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.304000
    (Iteration 141 / 200) loss: 1.432189
    (Epoch 8 / 10) train acc: 0.628000; val_acc: 0.339000
    (Iteration 161 / 200) loss: 1.034116
    (Epoch 9 / 10) train acc: 0.654000; val_acc: 0.342000
    (Iteration 181 / 200) loss: 0.905795
    (Epoch 10 / 10) train acc: 0.712000; val_acc: 0.328000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
Models with batch normalization seem more robust to weight initialization scale than models without batch normalization

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    ('No normalization: batch size = ', 5)
    ('Normalization: batch size = ', 5)
    ('Normalization: batch size = ', 10)
    ('Normalization: batch size = ', 50)

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
Batch normalization provides better validation accuracy when batch size is large

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
Batch normalization - Method 3
Layer normalization - Method 2

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
    ('  means: ', array([-59.06673243, -47.60782686, -43.31137368, -26.40991744]))
    ('  stds:  ', array([ 10.07429373,  28.39478981,  35.28360729,   4.01831507]))
    ()
    After layer normalization (gamma=1, beta=0)
    ('  means: ', array([ -9.62193288e-16,   2.22044605e-16,   7.40148683e-17,
            -3.70074342e-17]))
    ('  stds:  ', array([ 0.99999995,  0.99999999,  1.        ,  0.99999969]))
    ()
    ('After layer normalization (gamma=', array([ 3.,  3.,  3.]), ', beta=', array([ 5.,  5.,  5.]), ')')
    ('  means: ', array([ 5.,  5.,  5.,  5.]))
    ('  stds:  ', array([ 2.99999985,  2.99999998,  2.99999999,  2.99999907]))
    ()

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    ('dx error: ', 1.4336161688733359e-09)
    ('dgamma error: ', 4.5194895460327993e-12)
    ('dbeta error: ', 2.276445013433725e-12)

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
With a small number of features, layer normalization is likely to not work well since the estimates of mean and variance are likely to have larger standard errors.

```python

```


Principal receipe found here:
https://github.com/parrt/msds621

## Creditcard-Fraud: imbalanced data

```python
import pandas as pd
import numpy as np
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestRegressor, RandomForestClassifier
from sklearn.metrics import r2_score, mean_absolute_error, mean_squared_error, f1_score, accuracy_score,\
                            roc_auc_score, average_precision_score, precision_recall_curve, auc,\
                            roc_curve, precision_score, recall_score

from sklearn.model_selection import KFold
from sklearn.model_selection import cross_val_score
from sklearn.tree import DecisionTreeClassifier
from sklearn.metrics import confusion_matrix
from copy import copy
from numpy import random

import matplotlib.pyplot as plt
%config InlineBackend.figure_format = 'png'
```

```python
# https://www.kaggle.com/mlg-ulb/creditcardfraud
df = pd.read_csv("../data/creditcard.csv")
df.shape
```

    (284807, 31)

### the data-set is too large - we subsample the majority class

```python
no_fraud = df.index[df['Class']==0]
fraud = df.index[df['Class']==1].tolist()
sampled = random.choice(no_fraud, size=100000, replace=False).tolist()
df = df.loc[fraud + sampled, ].reset_index()
df.to_csv("../data/creditcard_subsampled.csv", header=True, index=False)
```

```python
print(f"num anomalies {np.sum(df['Class']==1)}/{len(df)} = {100*np.sum(df['Class']==1)/len(df):.2f}%")
```

    num anomalies 492/100492 = 0.49%

## what will our accuracy be ad hoc?

```python
pd.crosstab(df.Class.astype(str) + "_true", pd.Series(np.zeros_like(df.Class)).astype(str) + "_Actual")
```

      col_0
      0_Actual

      Class

      0_true
      100000

      1_true
      492

 - $\text{accuracy} = \frac{\text{true positives + true negatives}}{\text{true negatives + false negatives + true positives + false positives}} = \frac{0 + 284315}{492 + 284315} = 0.998$

## get a model

```python
import lightgbm
model = lightgbm.LGBMClassifier(boosting_type='gbdt', num_leaves=31, max_depth=- 1, learning_rate=0.1,
                                n_estimators=500, subsample_for_bin=20000, objective='binary',
                                subsample=1.0, subsample_freq=0, colsample_bytree=1.0,
                                n_jobs=- 1, silent=True, importance_type='split',
                                is_unbalance = False, scale_pos_weight = 1.0)

```

## train / test split

```python
X, y = df.drop('Class', axis=1), df['Class']
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20)
clf = copy(model)
clf.fit(X_train,y_train)

```

    LGBMClassifier(is_unbalance=False, n_estimators=500, objective='binary',
                   scale_pos_weight=1.0, subsample_for_bin=20000)

```python
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
FPR, TPR, _ = roc_curve(y_test, y_pred_proba)
df_conf = pd.DataFrame(confusion, columns=['non-Fraud','Fraud'], index=['non-Fraud','Fraud'])
df_conf
```

      non-Fraud
      Fraud

      non-Fraud
      20003
      6

      Fraud
      15
      75

```python
sensitivity, recall, hit rate, or true positive rate (TPR)
```

 \begin{equation}
 \text{Precision}=\frac{\text{TP}}{\text{TP + FP}}
   \end{equation}

  \begin{equation}
 \text{Recall}=\text{Sensitivity}=\text{TPR}=\frac{\text{TP}}{\text{P}} =\frac{\text{TP}}{\text{TP + FN}}
    \end{equation}

   \begin{equation}
 \text{F}_1\text{-Score} = 2\frac{\text{Precision} \cdot \text{Recall}}{\text{Precision} + \text{Recall}}
     \end{equation}

   \begin{equation}
  \text{FPR}=\frac{\text{FP}}{\text{N}} =\frac{\text{FP}}{\text{FP + TN}}
    \end{equation}

```python
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.926, Recall 0.833
    F1 0.88, Accuracy 0.9990
    ROC 0.99, AUC PR 0.89

```python
plt.plot(FPR, TPR, 'b.-', markersize=1, alpha=0.5)
plt.xlabel("FPR")
plt.ylabel("TPR / Recall / Sensitivity")
plt.title(f"credit card $\\bf ROC$ curve")
plt.savefig("/tmp/ROC-curve.png", dpi=200)
plt.show()
```

```python
plt.hist(y_pred_proba)
```

```python
plt.plot(recall, precision, 'b.-', markersize=1, alpha=0.5)
plt.xlabel("recall")
plt.ylabel("precision")
plt.title(f"credit card $\\bf PR$ curve")
plt.savefig("/tmp/PR-curve.png", dpi=200)
plt.show()
```

```python
# keep the workspace tidy
y_pred, confusion, y_pred_proba, precision, recall = None, None, None, None, None
del y_pred, confusion, y_pred_proba, precision, recall
```

## proper test set?

```python
# must split out test set first but get 20% from each class at original ratio of fraud/good
df_good = df[df['Class']==0]
df_fraud = df[df['Class']==1]

df_train_good, df_test_good   = train_test_split(df_good, test_size=0.20)
df_train_fraud, df_test_fraud = train_test_split(df_fraud, test_size=0.20)

df_train = pd.concat([df_train_good, df_train_fraud], axis=0)
df_test  = pd.concat([df_test_good, df_test_fraud], axis=0)

print(f"TRAIN num fraud {np.sum(df_train['Class']==1)}/{len(df_train)} = {100*np.sum(df_train['Class']==1)/len(df_train):.2f}%")
print(f"TEST num fraud {np.sum(df_test['Class']==1)}/{len(df_test)} = {100*np.sum(df_test['Class']==1)/len(df_test):.2f}%")
```

    TRAIN num fraud 393/80393 = 0.49%
    TEST num fraud 99/20099 = 0.49%

```python
# keep the workspace tidy
df_good, df_fraud, df_train_good, df_test_good, df_train_fraud, df_test_fraud, df_train, df_test =None,None,None,None, None, None, None,None
del df_good, df_fraud, df_train_good, df_test_good, df_train_fraud, df_test_fraud, df_train, df_test
```

we can also do this with a sklearn data-set splitter:

```python
from sklearn.model_selection import StratifiedShuffleSplit
splitter = StratifiedShuffleSplit(n_splits=2, test_size=0.2, train_size=0.8)
idx = next(splitter.split(df, df.Class))
train_idx = idx[0]
test_idx = idx[1]
```

```python
len(test_idx)
```

    20099

```python
df_train = df.loc[train_idx]
df_test = df.loc[test_idx]
print(f"TRAIN num fraud {np.sum(df_train['Class']==1)}/{len(df_train)} = {100*np.sum(df_train['Class']==1)/len(df_train):.2f}%")
print(f"TEST num fraud {np.sum(df_test['Class']==1)}/{len(df_test)} = {100*np.sum(df_test['Class']==1)/len(df_test):.2f}%")
```

    TRAIN num fraud 394/80393 = 0.49%
    TEST num fraud 98/20099 = 0.49%

```python
X_train, y_train = df_train.drop('Class', axis=1), df_train['Class']
X_test, y_test = df_test.drop('Class', axis=1), df_test['Class']
clf = copy(model)
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19964
      37

      T
      43
      55

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.598, Recall 0.561
    F1 0.58, Accuracy 0.9960
    ROC 0.78, AUC PR 0.55

```python
# keep workspace tidy
df, idx, df_train, df_test, clf, precision, recall, y_pred, y_pred_proba = None, None, None, None, None, None, None, None,None
del df, idx, df_train, df_test, clf, precision, recall, y_pred, y_pred_proba
```

# Strategies for the imbalanced case:
 - oversample the minority class
 - undersample the majority class
 - do both of the former two suggestions
 - oversample only the cases that get missclassified
 - set `is_unbalance` and/or `scale_pos_weight` parameters in lightgbm

## oversample fraud

### Synthetic Minority Oversampling Technique = SMOTE
The [original paper is here](https://arxiv.org/abs/1106.1813)

The idea is to connect very close points in the feature space and generate new samples lying on the connecting line. This is done for the minority class only.

```python
import imblearn
from imblearn.over_sampling import SMOTE, ADASYN
X_train_ov, y_train_ov = SMOTE(sampling_strategy=0.1).fit_resample(X_train, y_train)
```

```python
clf = copy(model)
clf.fit(X_train_ov,y_train_ov)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19997
      4

      T
      13
      85

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.955, Recall 0.867
    F1 0.91, Accuracy 0.9992
    ROC 0.97, AUC PR 0.89

```python
X_train_ov, y_train_ov, y_pred, y_pred_proba, precision, recall, clf = None, None, None, None, None, None, None
del X_train_ov, y_train_ov, y_pred, y_pred_proba, precision, recall, clf
```

### Adaptive Synthetic Sampling = ADASYN
The [paper can be found here](https://www.ele.uri.edu/faculty/he/PDFfiles/adasyn.pdf).

More synthetic observations are generated in areas where the density of the minority class is very low. Those cases are hard to learn and upsampling them with SMOTE should make the easier to learn.

```python
# needs too much RAM:
X_train_ov, y_train_ov = ADASYN(sampling_strategy=0.05).fit_resample(X_train, y_train)
```

```python
clf = copy(model)
clf.fit(X_train_ov,y_train_ov)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19998
      3

      T
      14
      84

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.966, Recall 0.857
    F1 0.91, Accuracy 0.9992
    ROC 0.98, AUC PR 0.88

```python
X_train_ov, y_train_ov, y_pred, y_pred_proba, precision, recall, clf = None, None, None, None, None, None, None
del X_train_ov, y_train_ov, y_pred, y_pred_proba, precision, recall, clf
```

### Borderline-SMOTE
The [paper can be found here](https://sci2s.ugr.es/keel/keel-dataset/pdfs/2005-Han-LNCS.pdf)

The principal idea is to upsample only those observations that are not classified correctly. The missclassified instances are found by k-nearest-neighbors.

```python
# needs to much RAM for me:
from imblearn.over_sampling import BorderlineSMOTE
X_train_ov, y_train_ov = BorderlineSMOTE(sampling_strategy=0.1).fit_resample(X_train, y_train)
```

```python
clf = copy(model)
clf.fit(X_train_ov,y_train_ov)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19999
      2

      T
      13
      85

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.977, Recall 0.867
    F1 0.92, Accuracy 0.9993
    ROC 0.98, AUC PR 0.88

```python
X_train_ov, y_train_ov, y_pred, y_pred_proba, precision, recall, clf = None, None, None, None, None, None, None
del X_train_ov, y_train_ov, y_pred, y_pred_proba, precision, recall, clf
```

### Support-Vector-Machine-SMOTE = SVMSMOTE
The class-boundaries for the Borderline are learned as the support-vectors of a Support-Vector-Machine.

```python
from imblearn.over_sampling import SVMSMOTE
X_train_ov, y_train_ov = SVMSMOTE(sampling_strategy=0.1).fit_resample(X_train, y_train)
```

```python
clf = copy(model)
clf.fit(X_train_ov,y_train_ov)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19999
      2

      T
      14
      84

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.977, Recall 0.857
    F1 0.91, Accuracy 0.9992
    ROC 0.98, AUC PR 0.89

```python
# keep the workspace tidy
X_train_ov, y_train_ovm, y_pred, clf = None, None, None, None
del X_train_ov, y_train_ovm, y_pred, clf
```

## weigh minority class

```python
clf = copy(model)
clf.set_params(is_unbalance = True)
#, scale_pos_weight
```

    LGBMClassifier(is_unbalance=True, n_estimators=500, objective='binary',
                   scale_pos_weight=1.0, subsample_for_bin=20000)

```python
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19127
      874

      T
      12
      86

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.090, Recall 0.878
    F1 0.16, Accuracy 0.9559
    ROC 0.92, AUC PR 0.48

```python
clf.set_params(is_unbalance=False, scale_pos_weight=200.0)
clf.fit(X_train,y_train)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19438
      563

      T
      16
      82

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.127, Recall 0.837
    F1 0.22, Accuracy 0.9712
    ROC 0.90, AUC PR 0.48

```python
# house-keeping
y_pred_proba, precision, recall, clf, y_pred = None, None, None, None, None
del y_pred_proba, precision, recall, clf, y_pred
```

## Undersample Fraud

```python
from imblearn.under_sampling import RandomUnderSampler
X_train_u, y_train_u = RandomUnderSampler(sampling_strategy=0.5).fit_resample(X_train, y_train)
print(X_train.shape, X_train_u.shape)
```

    (80393, 31) (1182, 31)

```python
clf = copy(model)
clf.fit(X_train_u,y_train_u)
y_pred = clf.predict(X_test)
confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19571
      430

      T
      10
      88

```python
y_pred_proba = clf.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.170, Recall 0.898
    F1 0.29, Accuracy 0.9781
    ROC 0.99, AUC PR 0.85

```python
y_pred_proba, y_pred, precision, recall, clf, X_train_u, y_train_u = None, None, None, None, None, None, None
y_pred_proba, y_pred, precision, recall, clf, X_train_u, y_train_u
```

    (None, None, None, None, None, None, None)

## do both: oversample minority class and undersample majority class

### SMOTE and Edited Nearest Neighbors = SMOTE-ENN
  - SMOTE is used for upsampling
  - ENN removes an observation and its (k=3) nearest neighbors when the majority class of the nearest neighbors is different from the observation's class

### SMOTETOMEK
  - this method finds reciprocal nearest-neighbors that are in different classes; These pairs are called Tomek-Links
  - the observation in such a pair, belonging to the majority class is deleted

```python
# not enough RAM
# from imblearn.combine import SMOTEENN
# X_train_uo, y_train_uo = SMOTEENN().fit_resample(X_train, y_train)
# X_train_uo, y_train_uo = SMOTETomek(random_state=0).fit_resample(X_train, y_train)
# print(X_train.shape, X_train_uo.shape)
```

## get best combination: sampling and hyper-parameters

```python
from imblearn.pipeline import make_pipeline, Pipeline
from sklearn.model_selection import GridSearchCV
from imblearn.over_sampling import BorderlineSMOTE
clf = copy(model)
clf.set_params(num_leaves=64)
borderline = BorderlineSMOTE(random_state=88)
grid = {'class__n_estimators': [200, 500],
        'class__neg_bagging_fraction': [0.1, 0.3],
        'class__max_depth' : [4, 6, 8,],
        'class__learning_rate' :[0.05, 0.1],
        'sampling__sampling_strategy' :[0.1, 0.3]}
pipeline = Pipeline([('sampling', borderline), ('class', clf)])
grid_cv = GridSearchCV(pipeline, grid, scoring = 'f1', cv = 5, refit=True)

grid_cv.fit(X_train, y_train)
display(grid_cv.best_score_, grid_cv.best_params_)
```

```python
pipeline.set_params(**grid_cv.best_params_)
pipeline.fit(X_train, y_train)
y_pred = pipeline.predict(X_test)#[:, 1]

confusion = confusion_matrix(y_test, y_pred)
df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

      F
      T

      F
      19996
      5

      T
      20
      78

```python
y_pred_proba = pipeline.predict_proba(X_test)[:,1] # 2nd column is p(fraud)
AUC = roc_auc_score(y_test, y_pred_proba)
precision, recall, _ = precision_recall_curve(y_test, y_pred_proba, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred):.3f}, Recall {recall_score(y_test, y_pred):.3f}")
print(f"F1 {f1_score(y_test, y_pred):.2f}, Accuracy {accuracy_score(y_test, y_pred):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.940, Recall 0.796
    F1 0.86, Accuracy 0.9988
    ROC 0.98, AUC PR 0.85

## One-Class SVM
Train on majority class only and classifiy test-set in in-class examples and out-class examples:

```python
from sklearn.svm import OneClassSVM

train_normal = X_train[y_train == 0]
train_outliers = X_train[y_train == 1]
```

```python
outlier_prop = len(train_outliers) / len(train_normal)
svm = OneClassSVM(kernel='rbf', nu=outlier_prop, gamma=0.000001)
svm.fit(train_normal)
```

    OneClassSVM(gamma=1e-06, nu=0.004925061563269541)

```python
y_pred = svm.predict(X_test)
y_pred_corrected = np.zeros_like(y_pred)
y_pred_corrected[y_pred == -1] = 1
```

```python
y_test

```

    58311    0
    56095    0
    41235    0
    18471    0
    37850    0
            ..
    77565    0
    50273    0
    69171    0
    61624    0
    82742    0
    Name: Class, Length: 20099, dtype: int64

```python
confusion = confusion_matrix(y_test, y_pred_corrected)

df_conf = pd.DataFrame(confusion, columns=['F','T'], index=['F','T'])
df_conf
```

```python
AUC = roc_auc_score(y_test, y_pred_corrected)
precision, recall, _ = precision_recall_curve(y_test, y_pred_corrected, pos_label=1)
PR = auc(recall, precision)
print(f"Precision {precision_score(y_test, y_pred_corrected):.3f}, Recall {recall_score(y_test, y_pred_corrected):.3f}")
print(f"F1 {f1_score(y_test, y_pred_corrected):.2f}, Accuracy {accuracy_score(y_test, y_pred_corrected):.4f}")
print(f"ROC {AUC:.2f}, AUC PR {PR:.2f}")
```

    Precision 0.031, Recall 0.255
    F1 0.06, Accuracy 0.9580
    ROC 0.61, AUC PR 0.15

Pros:
  - It works really well with a clear margin of separation
  - It is effective in high dimensional spaces.
  - It is effective in cases where the number of dimensions is greater than the number of samples.
  - It uses a subset of training points in the decision function (called support vectors), so it is also memory efficient.

Cons:
  - It doesn’t perform well when we have large data set because the required training time is higher
  - It also doesn’t perform very well, when the data set has more noise i.e. target classes are overlapping
  - SVM doesn’t directly provide probability estimates, these are calculated using an expensive five-fold cross-validation. It is included in the related SVC method of Python scikit-learn library.


```python
import pandas as pd
import numpy as np
from sklearn import datasets

data = pd.read_csv('lecture4data.csv')
#X = pd.Dataframe(data,columns = ['Country Name','Economic Stability','Gap between Rich and Poor','Instability of neighbours','Military Power','Technological developments of arms and warfare','Immigration','Stability of Government','Diversity','Public Unrest','Geographical Factors'])
data.head(5)
```

      name
      city
      state
      adm_rate
      undergrads
      cost
      compl_4
      median_hh_inc
      median_earnings

      0
      Alabama A & M University
      Normal
      AL
      0.6538
      4505
      20809
      0.1104
      49720.22
      29900

      1
      University of Alabama at Birmingham
      Birmingham
      AL
      0.6043
      11269
      22232
      0.3173
      55735.22
      40200

      2
      University of Alabama in Huntsville
      Huntsville
      AL
      0.8120
      5829
      20999
      0.1848
      58688.62
      45600

      3
      Alabama State University
      Montgomery
      AL
      0.4639
      4740
      18100
      0.1216
      46065.20
      26700

      4
      The University of Alabama
      Tuscaloosa
      AL
      0.5359
      31005
      27205
      0.4135
      57928.41
      42700

```python
#features = data[['Country Name','Economic Stability','Gap between Rich and Poor','Instability of neighbours','Military Power','Technological developments of arms and warfare','Immigration','Stability of Government','Diversity','Public Unrest','Geographical Factors']]
#target = data['War Index']
features = data[['adm_rate','undergrads','cost','compl_4','median_hh_inc']]
target = data['median_earnings']
```

```python
from sklearn.linear_model import LinearRegression
from sklearn.pipeline import make_pipeline
from sklearn.preprocessing import PolynomialFeatures
import matplotlib.pyplot as plt
test = pd.DataFrame({'cost': pd.Series(range(10000, 67000, 100))})
poly_model = make_pipeline(PolynomialFeatures(5), LinearRegression())
poly_model = poly_model.fit(data[['cost']], data[['median_earnings']])
predictions = poly_model.predict(test)
plt.scatter(data['cost'], data['median_earnings'])
plt.plot(test, predictions, 'k', linewidth=4)
plt.title('Gap between Rich and Poor vs War Index')
plt.xlabel('Gap between Rich and Poor')
plt.ylabel('War Index')
plt.show()
```

```python
from sympy import S, symbols
arr1 = test.flatten()
p = np.polyfit(arr1,predictions,5)
x = symbols("x")
poly = sum(S("{:6.2f}".format(v))*x**i for i, v in enumerate(p[::-1]))
eq_latex = sympy.printing.latex(poly)

plt.plot(test, predictions, 'k', label="${}$".format(eq_latex),linewidth=4)
plt.legend(fontsize="small")
plt.title('Gap between Rich and Poor vs War Index')
plt.xlabel('Gap between Rich and Poor')
plt.ylabel('War Index')
plt.show()

```

```python
from sympy import S, symbols
import sympy as smp

from matplotlib import pyplot as plt
import numpy as np

x=np.linspace(0,1,100)
y=np.sin(2 * np.pi * x)

p = np.polyfit(x, y, 5)
f = np.poly1d(p)

# calculate new x's and y's
x_new = np.linspace(x[0], x[-1], 50)
y_new = f(x_new)

x = symbols("x")
poly = sum(S("{:6.2f}".format(v))*x**i for i, v in enumerate(p[::-1]))
eq_latex = smp.printing.latex(poly)

plt.plot(x_new, y_new, label="${}$".format(eq_latex))
plt.legend(fontsize="small")
```

```python
import matplotlib as plt
Z, hmx, hmy = np.histogram2d(X['Country Name'],X['War Index'])

plt.title('Heatmap')
plt.xlabel('Country Name')
plt.ylabel('War Index')
plt.pcolormesh(hmx,hmy,Z.T)

plt.show()
```

```python

```


# Yearly number of carriers' flights provided for each origin-dest pair (route)

In this notebook we compute the yearly number of flights provided by the carriers for each route.

In the other notebooks other aspects of the dataset were analysed but no information about the carriers were inspected.

\begin{equation}
CountOfFlights(carrier, year) = \sum_{f\space in \space flights(year)} x_f\, , \,\,\, x_f = \begin{cases} 1, & \text{if} \space \space f[Carrier] == carrier \\ 0, & \text{otherwise} \end{cases}
\end{equation}

#### Execution mode

If development mode is turned on, all the calculations will be performed on a reduced instance of the data set.

This allows to *save computation time* while developing.

```python
# development mode
# False when performing real analytics
DEV = False

### threads to be used to run spark worker nodes locally
spark_local_threads = 4

# default transformed dataset path
transformed_dataset_path = '../dataset/05_cache_yearly_carries_flights_count_for_each_route.parquet'
missings_dataframe_path = '../dataset/05_cache_yearly_carries_flights_count_for_each_route_missings.pkl'
origins_dests_path = '../dataset/05_cache_origins_dests.pkl'
```

#### Find and import Apache Spark on current machine

```python
import findspark
findspark.init()
```

```python
# imports
import pickle
from pathlib import Path

from pyspark.sql import SparkSession
import pyspark.sql.functions as F
from pyspark.sql.types import *

import numpy as np
import pandas as pd

from missing_values_utils import *
```

#### Create a new session

```python
# Build a Spark SQL Session for DataFrames
master = 'local[{}]'.format(spark_local_threads)
appName = 'Yearly carriers filghts count for each route'
spark = SparkSession  \
    .builder          \
    .appName(appName) \
    .master(master)   \
    .getOrCreate()
```

### Load data

If there exists a cached version of the already transformed dataset, just load it and skip all the unnecessary computation.

```python
do_computation = True

if not DEV and Path(transformed_dataset_path).is_dir():
    df = spark.read.load(transformed_dataset_path)
    do_computation = False
```

Try to load the optimized parquet format data set. If parquet data set is not found, load full compressed data sets, reduce and save them.

```python
from preprocessing_utils import *

if do_computation:
    if DEV:
        # DEV preprocessing
        perform_DEV_dataset_preprocessing(spark)
    else:
        # Production preprocessing
        perform_dataset_preprocessing(spark)
```

    Starting preprocessing of ../dataset/*.csv.bz2
    Preprocessing NOT performed.
    Preprocessed dataset already exists: ../dataset/preprocessed_dataset.parquet

```python
# Load the parquet dataset
if do_computation:
    if DEV:
        # Load DEV dataset
        df = load_DEV_preprocessed_dataset(spark)
    else:
        # Load production dataset
        df = load_preprocessed_dataset(spark)
```

    Peprocessed dataset loaded.
    ../dataset/preprocessed_dataset.parquet

### Attributes selection

```python
if do_computation:
    # Keep only the dimensions we need
    df = df.select('Year', 'Cancelled' ,'Origin', 'Dest', 'UniqueCarrier')
    # Explore the schema
    df.printSchema()
```

    root
     |-- Year: integer (nullable = true)
     |-- Cancelled: integer (nullable = true)
     |-- Origin: string (nullable = true)
     |-- Dest: string (nullable = true)
     |-- UniqueCarrier: string (nullable = true)

```python
# get years from dataset
if do_computation:
    df_years = df.select('Year').distinct().toPandas()['Year']
else:
    df_years = range(1994,2009)

df_years = sorted(df_years)
print(df_years)
```

    [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008]

### Missing values exploration and exclusion

```python
df.count()
```

    91469371

```python
df, df_missing = get_df_missing_and_drop_all_null(df, do_computation, missings_dataframe_path)
```

    Years examined:
     [1994, 1995, 1996, 1997, 1998, 1999, 2000, 2001, 2002, 2003, 2004, 2005, 2006, 2007, 2008]

    Missing values inspection:
    0  null values in:  Year
    0  null values in:  Cancelled
    0  null values in:  Origin
    0  null values in:  Dest
    0  null values in:  UniqueCarrier

            Count  MissingsCount  MissingsPercentage
    1994  5180048              0                 0.0
    1995  5327435              0                 0.0
    1996  5351983              0                 0.0
    1997  5411843              0                 0.0
    1998  5384721              0                 0.0
    1999  5527884              0                 0.0
    2000  5683047              0                 0.0
    2001  5967780              0                 0.0
    2002  5271359              0                 0.0
    2003  6488540              0                 0.0
    2004  7129270              0                 0.0
    2005  7140596              0                 0.0
    2006  7141922              0                 0.0
    2007  7453215              0                 0.0
    2008  7009728              0                 0.0

```python
df.count()
```

    91469371

### Compute yearly carriers filghts count per route

```python
# Drop cancelled flights
if do_computation:
    df = df.drop(df['Cancelled'] == 1)
    df = df.drop('Cancelled')
```

```python
if do_computation:
    df = df.withColumn("Count", F.lit(1))

    df = df.groupBy(['Year', 'Origin', 'Dest', 'UniqueCarrier'])  \
           .sum('Count')                                          \
           .withColumnRenamed('sum(Count)', 'Count')

    df.show(10)
```

    +----+------+----+-------------+-----+
    |Year|Origin|Dest|UniqueCarrier|Count|
    +----+------+----+-------------+-----+
    |2007|   BNA| TPA|           WN| 1756|
    |2007|   LAX| OAK|           WN| 7496|
    |2007|   PHX| DTW|           WN|  667|
    |2007|   LAS| OKC|           YV|  123|
    |2007|   CMH| DCA|           OH|   91|
    |2007|   ORD| CVG|           OH| 1878|
    |2007|   BOS| BWI|           OH| 2562|
    |2007|   CVG| RIC|           OH|  628|
    |2007|   SMF| SLC|           OO|  990|
    |2007|   ACV| SFO|           OO| 2887|
    +----+------+----+-------------+-----+
    only showing top 10 rows

```python
if do_computation:
    df.write.mode('overwrite').save(transformed_dataset_path, format='parquet')
```

```python
df = spark.read.load(transformed_dataset_path)
```

#### Data format optimazion

```python
#We create a dictionary where for each origin there are associated all the possible destinations
def get_origins_dests_dict(df):
    print('Started. Please wait...')
    from datetime import datetime
    begin = datetime.now()

    origin_dests = {}

    origins = df.select('Origin')     \
                .orderBy('Origin')    \
                .distinct()           \
                .rdd                  \
                .map(lambda x : x[0]) \
                .collect()

    for origin in origins:
        dests = df.filter(df.Origin == origin) \
                  .select('Dest')              \
                  .orderBy('Dest')             \
                  .distinct()                  \
                  .rdd                         \
                  .map(lambda x : x[0])        \
                  .collect()
        origin_dests[origin] = dests

    end = datetime.now()
    print('Complete. Elapsed time: {}'.format(end - begin))

    return origin_dests
```

```python
if Path(origins_dests_path).is_file() and not do_computation:
    # load from cache
    with open(r'' + origins_dests_path, 'rb') as pf:
        origins_dests = pickle.load(pf)
else:
    # compute
    origins_dests = get_origins_dests_dict(df)

    # cache it
    with open(r'' + origins_dests_path, 'wb') as pf:
        pickle.dump(origins_dests, pf)
```

    Started. Please wait...
    Complete. Elapsed time: 0:05:45.479995

```python
# Carriers
unique_carriers = df.select('UniqueCarrier').distinct().toPandas()['UniqueCarrier']
unique_carriers = sorted(unique_carriers)
unique_carriers
```

    ['9E',
     'AA',
     'AQ',
     'AS',
     'B6',
     'CO',
     'DH',
     'DL',
     'EV',
     'F9',
     'FL',
     'HA',
     'HP',
     'MQ',
     'NW',
     'OH',
     'OO',
     'TW',
     'TZ',
     'UA',
     'US',
     'WN',
     'XE',
     'YV']

```python
# Airports
airports_origin = df.select('Origin').distinct().orderBy('Origin').rdd.map(lambda x : x[0]).collect()
airports_dest = df.select('Dest').distinct().orderBy('Dest').rdd.map(lambda x : x[0]).collect()
```

##### Data visualization functions

Scroll down to see the charts

```python
# Hide warnings if there are any
import warnings
warnings.filterwarnings('ignore')

%matplotlib inline

from ipywidgets import interact_manual
import ipywidgets as widgets
import matplotlib.pyplot as plt
from matplotlib.font_manager import FontProperties
```

```python
def get_count_df(airport_origin, airport_dest, years, df):
    df_selection = df.filter(df.Year.isin(*years))        \
                     .filter(df.Origin == airport_origin) \
                     .filter(df.Dest == airport_dest)     \
                     .select('UniqueCarrier', 'Count' )   \
                     .toPandas()

    # create dictionary: key-> carrier, value-> 0
    data = dict.fromkeys(unique_carriers, 0)
    for idx, r in df_selection.iterrows():
        data[r['UniqueCarrier']] += r['Count']

    return pd.DataFrame.from_dict(data, orient='index')

def plot_count_carriers(airport_origin, airport_dest, years, df, ax):
    df = get_count_df(airport_origin, airport_dest, years, df)
    title = 'Unique Carrier count for Origin-Dest and Year'
    df.plot.bar(title=title, rot=90, ax=ax, legend=False)

def on_change(change):
    # exclude the origin ariport and all the aiports with count 0 for all the carriers
    global airports2_w
    global df
    airports_dx = origins_dests[change['new']]
    airports2_w.options = airports_dx
    airports2_w.value = 'N/A' if not airports_dx else airports_dx[0]
```

```python
def ui_callback(airport_origin, airport_dest, years, df):
    plt.figure(figsize=(12,15))
    plt.clf()
    ax = plt.subplot(211)
    plot_count_carriers(airport_origin, airport_dest,range(years[0], years[1] + 1), df, ax)

    plt.subplots_adjust(top=0.92, bottom=0.08, left=0.10, right=0.95, hspace=0.25, wspace=0.35)
    plt.show()

# Years selection range
years = [(str(y), y) for y in df_years]
years_w = widgets.SelectionRangeSlider(options=years,
                                       index=(0, 2),
                                       description='Years',
                                       continuous_update=False)
```

```python
# Airports selection menus
airports1_w = widgets.Dropdown(options     = airports_origin,
                               value       = airports_origin[0],
                               description = 'Airport_Origin')
# set the listener function to the observer
airports1_w.observe(on_change, 'value')

airports2_w = widgets.Dropdown(options     = origins_dests['ABE'],
                               value       = origins_dests['ABE'][0],
                               description = 'Airport_Dest')
```

```python
out = widgets.interactive_output(ui_callback, {'airport_origin': airports1_w,'airport_dest': airports2_w, 'years': years_w, 'df': widgets.fixed(df)})
ui = widgets.HBox([airports1_w, airports2_w, years_w])
```

## Data Visualization

Analytics showhing yearly number of flights provided by the carriers for each route are reported below.

### Missing values and valid data

```python
plot_missing_values_stacked_bar(df_missing)
```

### Yearly number of flights provided by the carriers for each route

For the user convenience, the Aiport_Dest dropdown is automatically filled with only the airports that are valid destinations with respect to the selected Airport_Origin. Change the Airport_Origin value to update the Aiport_Dest dropdown.

> **WARNING**
>
>If the resulting chart is empty, it means that for the selected years that are no flights. In this case, by selecting all the available years, at least one bar will be visualized.

#### Data discoveries

We decided to keep the absolute value of flights managed by each carrier and not to show the percentage w.r.t. the totality of flights on the route because we noticed a very high variance of the possible results. Some routes are much more covered than others.

#### Suggested tests
- ABE-AZO: 1 flight only (2006)
- ABE-ATL interesting before/after 2000
- DEN-ABE
- DEN-COS
- BOS-JFK

List of airports in the United States: https://en.wikipedia.org/wiki/List_of_airports_in_the_United_States
<br/>List of airline codes: https://en.wikipedia.org/wiki/List_of_airline_codes

```python
display(ui, out)
```

    HBox(children=(Dropdown(description='Airport_Origin', options=('ABE', 'ABI', 'ABQ', 'ABY', 'ACK', 'ACT', 'ACV'…

    Output()


```python
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```python
#export
from exp.nb_06 import *
```

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=5899)

```python
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```python
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```python
nfs = [8,16,32,64,64]
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(2, learn)
```

    train: [1.286835234375, tensor(0.5818, device='cuda:0')]
    valid: [0.21955908203125, tensor(0.9303, device='cuda:0')]
    train: [0.1570641796875, tensor(0.9517, device='cuda:0')]
    valid: [0.09984942626953125, tensor(0.9713, device='cuda:0')]
    Wall time: 5.28 s

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6018)

```python
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.ones(1,nf,1,1)) #register_buffer will move this variable to GPU and saved in the model
        self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        m = x.mean((0,2,3), keepdim=True) #(0,2,3) 0:batches axis, 2 and 3 are x and y axis
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)  #lerp_  :linear interpolation, the momentum used in this function is the oppsite of
                                       #the normal sense(as shown in __init__, it's 0.1 not 0.9)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
        if self.training:
            with torch.no_grad(): m,v = self.update_stats(x)
        else:
            m,v = self.means,self.vars
        x = (x-m) / (v+self.eps).sqrt()
        return x*self.mults + self.adds
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs): #bn:batchnorm
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn), #batch norm is also bias, so if bn is true, no
                                                                                #need for bias
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None: m.bias.data.zero_()
    for l in m.children(): init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

```python
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```python
learn,run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.27727037109375, tensor(0.9114, device='cuda:0')]
    valid: [0.11842855224609375, tensor(0.9666, device='cuda:0')]
    train: [0.085093349609375, tensor(0.9733, device='cuda:0')]
    valid: [0.1284952880859375, tensor(0.9591, device='cuda:0')]
    train: [0.059765107421875, tensor(0.9815, device='cuda:0')]
    valid: [0.0684951171875, tensor(0.9796, device='cuda:0')]
    Wall time: 5.14 s

### Builtin batchnorm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6679)

```python
#export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.22859607421875, tensor(0.9278, device='cuda:0')]
    valid: [0.09754116821289062, tensor(0.9717, device='cuda:0')]
    train: [0.0672201611328125, tensor(0.9790, device='cuda:0')]
    valid: [0.080840380859375, tensor(0.9762, device='cuda:0')]
    train: [0.0445792431640625, tensor(0.9858, device='cuda:0')]
    valid: [0.05909752197265625, tensor(0.9828, device='cuda:0')]
    Wall time: 4.44 s

### With scheduler

Now let's add the usual warm-up/annealing.

```python
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched)])
```

```python
run.fit(8, learn)
```

    train: [0.2556219140625, tensor(0.9226, device='cuda:0')]
    valid: [0.134628955078125, tensor(0.9574, device='cuda:0')]
    train: [0.091288759765625, tensor(0.9715, device='cuda:0')]
    valid: [0.07984010009765626, tensor(0.9732, device='cuda:0')]
    train: [0.055083505859375, tensor(0.9825, device='cuda:0')]
    valid: [0.06229658813476562, tensor(0.9809, device='cuda:0')]
    train: [0.0338397412109375, tensor(0.9886, device='cuda:0')]
    valid: [0.0544700439453125, tensor(0.9836, device='cuda:0')]
    train: [0.019713880615234373, tensor(0.9939, device='cuda:0')]
    valid: [0.047903341674804686, tensor(0.9863, device='cuda:0')]
    train: [0.011677803955078125, tensor(0.9967, device='cuda:0')]
    valid: [0.04176881408691406, tensor(0.9889, device='cuda:0')]
    train: [0.006602901611328125, tensor(0.9987, device='cuda:0')]
    valid: [0.042446771240234375, tensor(0.9881, device='cuda:0')]
    train: [0.004530086669921875, tensor(0.9994, device='cuda:0')]
    valid: [0.04200081787109375, tensor(0.9883, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6717)

```python
class LayerNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    def forward(self, x):
        m = x.mean((1,2,3), keepdim=True)
        v = x.var ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```python
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.1264, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    Wall time: 7.98 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7114)

```python
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
        m = x.mean((2,3), keepdim=True)
        v = x.var ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

```python
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    Wall time: 6.09 s

*Question*: why can't this classify anything?

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

### Group norm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7213)

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7304)

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [2.3196465625, tensor(0.1821, device='cuda:0')]
    valid: [4759502.4384, tensor(0.2565, device='cuda:0')]
    Wall time: 2min 18s

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7516)

```python
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel()/nc)
        mom1 = 1 - (1-self.mom)/math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias*(1-self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step<100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c    = c    / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```python
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn,run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [0.4374879296875, tensor(0.8819, device='cuda:0')]
    valid: [0.1823290771484375, tensor(0.9584, device='cuda:0')]
    Wall time: 6min 37s

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=8068)

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 32), c)
```

```python
learn,run = get_learn_run(nfs, data, 0.9, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

```python
%time run.fit(1, learn)
```

    train: [0.16663080078125, tensor(0.9471, device='cuda:0')]
    valid: [0.14945870361328126, tensor(0.9646, device='cuda:0')]
    Wall time: 26.3 s

## Export

```python
nb_auto_export()
```

    <IPython.core.display.Javascript object>

```python

```