# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [ 1.33226763e-17 -3.94129174e-17  3.29597460e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029261167605239e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  1.1612359861380333e-12
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 1.00x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.248698913380842
    W1 relative error: 1.31e-04
    W2 relative error: 1.07e-05
    W3 relative error: 2.52e-10
    b1 relative error: 1.11e-08
    b2 relative error: 1.11e-08
    b3 relative error: 1.30e-10
    beta1 relative error: 2.34e-09
    beta2 relative error: 1.33e-09
    gamma1 relative error: 4.12e-08
    gamma2 relative error: 3.75e-09

    Running check with reg =  3.14
    Initial loss:  16.585926023164685
    W1 relative error: 1.00e+00
    W2 relative error: 1.00e+00
    W3 relative error: 1.00e+00
    b1 relative error: 2.41e-09
    b2 relative error: 5.55e-09
    b3 relative error: 5.55e-10
    beta1 relative error: 1.39e-09
    beta2 relative error: 4.06e-08
    gamma1 relative error: 3.65e-08
    gamma2 relative error: 1.59e-06

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.329570
    (Epoch 0 / 10) train acc: 0.105000; val_acc: 0.104000
    (Epoch 1 / 10) train acc: 0.408000; val_acc: 0.321000
    (Iteration 21 / 200) loss: 1.760356
    (Epoch 2 / 10) train acc: 0.510000; val_acc: 0.295000
    (Iteration 41 / 200) loss: 1.411037
    (Epoch 3 / 10) train acc: 0.545000; val_acc: 0.305000
    (Iteration 61 / 200) loss: 1.151660
    (Epoch 4 / 10) train acc: 0.595000; val_acc: 0.272000
    (Iteration 81 / 200) loss: 1.141470
    (Epoch 5 / 10) train acc: 0.751000; val_acc: 0.302000
    (Iteration 101 / 200) loss: 0.715621
    (Epoch 6 / 10) train acc: 0.840000; val_acc: 0.314000
    (Iteration 121 / 200) loss: 0.791894
    (Epoch 7 / 10) train acc: 0.832000; val_acc: 0.265000
    (Iteration 141 / 200) loss: 0.676013
    (Epoch 8 / 10) train acc: 0.869000; val_acc: 0.304000
    (Iteration 161 / 200) loss: 0.343583
    (Epoch 9 / 10) train acc: 0.940000; val_acc: 0.284000
    (Iteration 181 / 200) loss: 0.178243
    (Epoch 10 / 10) train acc: 0.887000; val_acc: 0.282000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.301973
    (Epoch 0 / 10) train acc: 0.093000; val_acc: 0.112000
    (Epoch 1 / 10) train acc: 0.264000; val_acc: 0.246000
    (Iteration 21 / 200) loss: 1.983225
    (Epoch 2 / 10) train acc: 0.397000; val_acc: 0.298000
    (Iteration 41 / 200) loss: 1.735059
    (Epoch 3 / 10) train acc: 0.469000; val_acc: 0.318000
    (Iteration 61 / 200) loss: 1.420312
    (Epoch 4 / 10) train acc: 0.536000; val_acc: 0.315000
    (Iteration 81 / 200) loss: 1.166277
    (Epoch 5 / 10) train acc: 0.617000; val_acc: 0.313000
    (Iteration 101 / 200) loss: 0.973184
    (Epoch 6 / 10) train acc: 0.674000; val_acc: 0.330000
    (Iteration 121 / 200) loss: 0.769337
    (Epoch 7 / 10) train acc: 0.732000; val_acc: 0.312000
    (Iteration 141 / 200) loss: 0.546289
    (Epoch 8 / 10) train acc: 0.785000; val_acc: 0.343000
    (Iteration 161 / 200) loss: 0.580997
    (Epoch 9 / 10) train acc: 0.785000; val_acc: 0.295000
    (Iteration 181 / 200) loss: 0.899831
    (Epoch 10 / 10) train acc: 0.899000; val_acc: 0.334000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers_ws[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
Small initial weights creates vanishing gradients problem in baseline model, but still works in BN model.

Baseline model very sensitive to correct weight initialization, BN model more robust.

First plot: BN model starts to overfit later than baseline model, shows it is more robust to overfitting.

Third plot: shows explosing gradient problem where loss shoots to the sky with baseline model, as usual, BN model more robust to this issue.

In general with batch normalization we can avoid the problem of vanishing and exploding gradients because it normalizes every affine layer (xW+b), avoiding very large/small values. Moreover, its regularization properties allow to decrease overfitting.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
According to the results, we can see that the batch size affects directly the performance of batch normalization (the smaller the batch size the worse). Even the baseline model outperforms the batchnorm model when using a very small batch size. This problem occurs because when we calculate the statistics of a batch, i.e., mean and variance, we try to find an approximation of the statistics of the entire dataset. Therefore with a small batch size, these statistics can be very noisy. On the other hand, with a large batch size we can obtain a better approximation.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
1. 2. layer because working on within an image only, 3. 4. batch because working across all images.

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_backward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -1.48029737e-16  0.00000000e+00  3.33066907e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.4336160411201157e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
2


# GPS-Logger elemzo Peter Bence X89O8X

## Feladatok

1. Megadva egy v_max sebességhatárt, megállapítja, hány másodpercig volt a mozgás sebesség-nagysága ennél nagyobb.
2. Kiszámolja azt is, hogy mennyivel nőtt volna meg a menetidő, ha azonos útvonalon mennénk, de ahol átléptük v_max-ot, ott csak v_max-szal mentünk volna.

## Felhasznalt txt file-ok

* vezetes-1.txt
* vezetes-2.txt

```python
from sympy import *
import numpy as np
import pandas as pd
%matplotlib inline
import matplotlib.pyplot as plt
```

```python
# numerikus derivalo fuggveny derivalas bal-, jobboldali es centralis kozelitessel
def deriv(xx_tab, ff_tab): # xx_tab ido tablazat, ff_tab f(t) tablazat
    N=xx_tab.shape[0]
    dff_dxx=np.zeros(N, np.float64)  # a deriváltak tömbje
    for i in range(N):
        if i==0:        # bal széle
            deriv=(ff_tab[1]-ff_tab[0])/(xx_tab[1]-xx_tab[0])
        elif i==N-1:    # jobb széle
            deriv=(ff_tab[N-1]-ff_tab[N-2])/(xx_tab[N-1]-xx_tab[N-2])
        else:          # közepe
            deriv=(ff_tab[i+1]-ff_tab[i-1])/(xx_tab[i+1]-xx_tab[i-1])
        dff_dxx[i]=deriv
    return(dff_dxx)
```

```python
# numerikus integralo fuggveny, F0 kezdeti ertek
def integral(xx_tab, ff_tab, F0=0.0): # xx_tab ido tablazat, ff_tab f(t) tablazat, ks pedig a kozeliteshez hasznalt sema
    N=xx_tab.shape[0]
    Ff_tab=np.zeros(N, dtype=np.float64)
    Ff_tab=F0
    for i in range(1,N):
        intff[i]=intff[i-1]+(xx_tab[i]-xx_tab[i-1])*(ff_tab[i]+ff_tab[i-1])/2.0
    return(intff)
```

```python
# Adatok beolvasasa es konvertalasa hasznalhato formatumba vezetes-1.txt bol
filename = 'vezetes-1.txt'
data = pd.read_csv(filename, header=0)
v_tab = data['speed(m/s)']
data['tdat'] = pd.to_datetime(data['date time'], format="%Y-%m-%d %H:%M:%S")
tdat_arr=data.tdat.to_numpy()
t_arr=(tdat_arr-tdat_arr[0])/np.timedelta64(1, 's') # ido datum adatok secundum-ma alakitasa
R_F=6378000.0 # a Föld sugara m-ben
long_arr=np.radians(data.longitude).to_numpy()  # hosszúsági adatok radiánba
lat_arr=np.radians(data.latitude).to_numpy()    # szélességi adatok radiánba
x_arr=(long_arr-long_arr[0])*np.cos(lat_arr[0])*R_F
y_arr=(lat_arr-lat_arr[0])*R_F

# Adatok beolvasasa es konvertalasa hasznalhato formatumba vezetes-1.txt bol
filename2 = 'vezetes-2.txt'
data2 = pd.read_csv(filename2, header=0)
v_tab2 = data2['speed(m/s)']
data2['tdat'] = pd.to_datetime(data2['date time'], format="%Y-%m-%d %H:%M:%S")
tdat_arr2=data2.tdat.to_numpy()
t_arr2=(tdat_arr2-tdat_arr2[0])/np.timedelta64(1, 's') # ido datum adatok secundum-ma alakitasa
```

```python
# Néhány általános adat nyomtatása
print("Mérési pontok száma   : %d"%(len(t_arr)))

delta_t=t_arr[-1]-t_arr[0]
print("Időtartam             : %.2f s"%(delta_t))

delta_x=x_arr[-1]-x_arr[0]
delta_y=y_arr[-1]-y_arr[0]
print("Elmozdulás-komponensek: %.2f m, %.2f m"%(delta_x, delta_y))

delta_r_abs=(delta_x**2+delta_y**2)**0.5   # Pitagorasz-tétel
print("Elmozdulás nagysága   : %.2f m"%(delta_r_abs))
print("Átlagsebesség-komponensek: %.2f m/s, %.2f m/s"%(delta_x/delta_t, delta_y/delta_t))
print("Átlagsebesség nagysága   : %.2f m/s"%(delta_r_abs/delta_t))

```

    Mérési pontok száma   : 1494
    Időtartam             : 1493.00 s
    Elmozdulás-komponensek: -2957.50 m, 4795.22 m
    Elmozdulás nagysága   : 5633.92 m
    Átlagsebesség-komponensek: -1.98 m/s, 3.21 m/s
    Átlagsebesség nagysága   : 3.77 m/s

```python
# sebesség-komponensek
vx_arr=deriv(t_arr, x_arr)
vy_arr=deriv(t_arr, y_arr)
# sebesség-nagyság:
vabs_arr=(vx_arr**2+vy_arr**2)**0.5
```

```python
fig=plt.figure(figsize=(20,15))  # rajzméret
ax1=fig.add_subplot(211) # két rész-grafikon
ax2=fig.add_subplot(212)
ax1.set_ylabel("x(t)")
ax1.plot(t_arr, x_arr, color="red") # az elsőbe az x(t)
ax2.set_ylabel("y(t)")
ax2.plot(t_arr, y_arr, color="green") # a másodikba az y(t)
ax1.grid()
ax2.grid()
```

```python
fig=plt.figure(figsize=(15,15))  # rajzméret
ax1=fig.add_subplot(111)
ax1.plot(x_arr, y_arr, color="red") # x függvényében y
ax1.grid()
```

```python
# út számítás
ut_arr=np.zeros(t_arr.shape, np.float64)
for i in range(1,len(ut_arr)):
    delta_x=x_arr[i]-x_arr[i-1]
    delta_y=y_arr[i]-y_arr[i-1]
    delta_ut=(delta_x**2+delta_y**2)**0.5
    ut_arr[i]=ut_arr[i-1]+ delta_ut
```

```python
# teljes utat jellemző adatok:
print("Megtett út              : %.2f m"%(ut_arr[-1]))
print("Átlagos sebesség-nagyság: %0.2f m/s"%(ut_arr[-1]/t_arr[-1]))
```

    Megtett út              : 9650.21 m
    Átlagos sebesség-nagyság: 6.46 m/s

```python
# néhány általános adat: (ismétlés)
print("Mérési pontok száma   : %d"%(len(t_arr)))
delta_t=t_arr[-1]-t_arr[0]
print("Időtartam             : %.2f s"%(delta_t))
delta_x=x_arr[-1]-x_arr[0]
delta_y=y_arr[-1]-y_arr[0]
print("Elmozdulás-komponensek: %.2f m, %.2f m"%(delta_x, delta_y))
delta_r_abs=(delta_x**2+delta_y**2)**0.5   # Pitagorasz-tétel
print("Elmozdulás nagysága   : %.2f m"%(delta_r_abs))
print("Átlagsebesség-komponensek: %.2f m/s, %.2f m/s"%(delta_x/delta_t, delta_y/delta_t))
print("Átlagsebesség nagysága   : %.2f m/s"%(delta_r_abs/delta_t))

```

    Mérési pontok száma   : 1494
    Időtartam             : 1493.00 s
    Elmozdulás-komponensek: -2957.50 m, 4795.22 m
    Elmozdulás nagysága   : 5633.92 m
    Átlagsebesség-komponensek: -1.98 m/s, 3.21 m/s
    Átlagsebesség nagysága   : 3.77 m/s

```python
# Adatok simitasa
from scipy.signal import savgol_filter   # telepíteni kell a SciPy modult!

fokszam=5  # ennyied fokú csúszó polinom illesztés
ablak=21   # ennyi adatponttal (páratlan!)

xs_arr=savgol_filter(x_arr, ablak, fokszam)
ys_arr=savgol_filter(y_arr, ablak, fokszam)
```

```python
# sebesség-komponensek
vxs_arr=deriv(t_arr, xs_arr)
vys_arr=deriv(t_arr, ys_arr)
# sebesség-nagyság:
vsabs_arr=(vxs_arr**2+vys_arr**2)**0.5
```

```python
# gyorsulás-komponensek
axs_arr=deriv(t_arr, vxs_arr)
ays_arr=deriv(t_arr, vys_arr)
a_abs_arr=(axs_arr**2+ays_arr**2)**0.5
```

```python
# sebesseg adatok simitasa
vxs_arr=savgol_filter(x_arr, ablak, fokszam, deriv=1)
vys_arr=savgol_filter(y_arr, ablak, fokszam, deriv=1)
# gyorsulas adatok simitasa
axs_arr=savgol_filter(x_arr, ablak, fokszam, deriv=2)
ays_arr=savgol_filter(y_arr, ablak, fokszam, deriv=2)
# sebesseg vektorok nagysaganak kiszamitasa x es y komponensekbol
vsabs_arr=(vxs_arr**2+vys_arr**2)**0.5
# gyorsulas vektorok nagysaganak kiszamitasa x es y komponensekbol
as_abs_arr=(axs_arr**2+ays_arr**2)**0.5
```

### 1. Feladat
#### Megadva egy v_max sebességhatárt, megállapítja, hány másodpercig volt a mozgás sebesség-nagysága ennél nagyobb.

```python
def calc_time_over_vmax(v_max, v_arr, t_arr):
    v_max_ms = v_max / 3.6
    duration = 0.0
    for i in range(len(t_arr)):
        if v_arr[i] > v_max_ms:
            duration += t_arr[i]-t_arr[i-1]
    return duration
```

```python
v_max = 50.0
time_over_vmax = calc_time_over_vmax(v_max, vsabs_arr, t_arr)
print("A sebesseg-nagysaga %.2f masodpercig volt nagyobb mint a v_max."%time_over_vmax)
```

    A sebesseg-nagysaga 69.00 masodpercig volt nagyobb mint a v_max.

### 2. Feladat
#### Kiszámolja azt is, hogy mennyivel nőtt volna meg a menetidő, ha azonos útvonalon mennénk, de ahol átléptük v_max-ot, ott csak v_max-szal mentünk volna.

```python
v_max_ms = v_max / 3.6 # km/h atvaltasa m/s-ba
v_legal_arr = np.where(vsabs_arr < v_max_ms, vsabs_arr, v_max_ms) # v_max nal nagyobb sebessegek kicserelese v_max-ra
```

```python
# v = s / t
# t = r / v
v_atlag = np.sum(v_legal_arr) / len(v_legal_arr)
s = ut_arr[-1] - ut_arr[0]
t_legal = s / v_atlag
t_tobblet = t_legal - (t_arr[-1] - t_arr[0])
print("%.2f s-el none meg  az ido ha v_max ot nem lepjuk at."%t_tobblet)
```

    24.33 s-el none meg  az ido ha v_max ot nem lepjuk at.

```python

```


# Práctica Background Subtraction

En esta segunda práctica del bloque de análisis de secuencias haremos trabajo con 3 técnicas de background subtraction.

En primer lugar debéis implementar vuestro propio método de eliminación de fondo basado en la idea que explicamos del filtro exponencial. Podéis hacer las adaptaciones que encontréis necesario a la idea general del método para mejorar su rendimiento.

En segundo lugar debéis de usar las implementaciones del MOG y del MOG2 que se encuentran en la librería
 [OpenCV](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.html).

En tercer lugar lo que quiero es que hagáis una comparativa entre su método y los dos basados en Mixture de gausiano utilizando el dataset de la competición [changedetection.net](http://changedetection.net/) del año 2012.

En concreto debe usar una secuencia de cada categoría exceptuando la de *Thermal*.

Las métricas que a usar para cada secuencia son:

* **TP** : True Positive
* **FP** : False Positive
* **FN** : False Negative
* **TN** : True Negative
* **Re (Recall)** : TP / (TP + FN)
* **F-Measure** : (2 * Precision * Recall) / (Precision + Recall)
* **Precision** : TP / (TP + FP)

Finalmente se pide que hagáis un análisis de los resultados obtenidos por cada secuencia: problemas, carencias, puntos fuertes de los algoritmos ...

**Entrega:**

* Explicación del algoritmo implementado y código fuente.
* Resultados de las secuencias en formato imagen o video.
* Resultados de las secuencias en formato tabla resumen de las métricas.
* Análisis de los resultados

Secuencias a usar:
- baseline: todo
- cameraJitter: badminton, traffic
- dynamicBackground: canoe, fall, fountain02
- intermittentObjectMotion: sofa,
- shadow: busStation, copyMachine, cubicle, peopleInShade

### Métricas

```python
def recall(tp, fn):
    return (tp/(tp+fn))

def precision(tp, fp):
    return (tp/(tp+fp))

def f_measure(precision, recall):
    return ((2*precision*recall)/(precision+recall))

def all_metrics (resultPath, groundtruthPath, start_frame, end_frame):
    # Cargamos el resultado del filtro indicado
    result = loadImages(resultPath)
    groundTruth = loadImages(groundtruthPath)

    # ground truth empieza en start_frame, los frames están numerados de 1 hasta n, pero el array de frames va de 0 a n-1
    total_frames = end_frame-start_frame

    # Inicializamos un array para cada métrica donde se el resultado para cada frame
    tp =  np.zeros(total_frames)
    fp =  np.zeros(total_frames)
    fn =  np.zeros(total_frames)
    tn =  np.zeros(total_frames)

    # Calculamos las métricas True Positive, False Positive, False Negative, True Negative para cada frame
    for i in range(total_frames):
        tp[i], fp[i], fn[i], tn[i] = comparator(result[i+start_frame], groundTruth[i+start_frame])

    # Calculamos las métricas Recall, Precision y F-measure para cada frame
    recall_frame = recall(tp=tp, fn=fn)
    precision_frame = precision(tp=tp, fp=fp)
    fMeasure_frame = f_measure(precision=precision_frame, recall=recall_frame)

    # Calculamos las métricas anteriores para toda la secuencia en conjunto
    tp_secuencia = tp.sum()
    fp_secuencia = fp.sum()
    fn_secuencia = fn.sum()
    tn_secuencia = tn.sum()

    recall_secuencia = recall(tp=tp_secuencia, fn=fn_secuencia)
    precision_secuencia = precision(tp=tp_secuencia, fp=fp_secuencia)
    fMeasure_secuencia = f_measure(precision=precision_secuencia, recall=recall_secuencia)

    cm_frames = np.array([tp,fp,fn,tn])
    cm_secuencia = np.array ([tp_secuencia, fp_secuencia, fn_secuencia, tn_secuencia])

    r_p_fm_frames = np.array([recall_frame, precision_frame, fMeasure_frame])
    r_p_fm_secuencia = np.array([recall_secuencia, precision_secuencia, fMeasure_secuencia])

    return cm_frames, cm_secuencia, r_p_fm_frames, r_p_fm_secuencia
```

### Filtro exponencial para _background subtraction_
Parte de la idea de eliminación de fondo más básica: restar el fotograman **_fondo_** (_background_) del fotograma actual para obtener el **_primer plano_** (_foreground_).
El primer problema o tarea para realizar este algoritmo es conseguir ese fotograma **_fondo_**. Para ello se necesita una imagen que represente el fondo sin ningún otro objeto, pero esto no siempre es posible, por lo tanto hay que buscar otras soluciones. Una de ellas es estimar el fondo como la media o la mediana a partir de los primeros _n_ fotogramas.
El segundo problema que surge es que el fondo en los casos prácticos varía con el tiempo y por lo tanto es posible que ese fotograma _fondo_ que hemos estimado no nos sirve más tarde. Por lo tanto la mejora que se propone es estimar el fondo recursivamente respecto el fotograma anterior siguiendo la siguiente fórmula:
$$
\begin{align}
B_t=(1-{\alpha})B_{t-1}+{\alpha}I_t
\end{align}
$$
Donde $B_t$ es la estimación del fondo para el fotograma actual, $B_{t-1}$ es el fondo para el fotograma anterior, $I_t$ es el fotograma actual y $\alpha$ es el factor de aprendizaje que indica cuánto del fotograma actual se considerará fondo para el siguiente fotograma.

```python
from utils import *
import numpy as np
from tqdm.notebook import tqdm
import matplotlib.pyplot as plt
```

Para mantener la limpieza del _notebook_ se han definidos las funciones para realizar el análisis dentro del archivo _utils.py_.
Las funciones que se encuentran son las siguientes:
- **comparator**: compara un _frame_ resultante de un modelo de _background subtraction_ con el resultado esperado (_groundtruth_)
- **loadImages**: devuelve una lista con las imágenes dentro de la carpeta especificada
- **exponentialFilter**: implementación del algoritmo basado en filtro exponencial
- **MOG**: implementación del algoritmo MOG disponible [aquí](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.html)
- **MOG2**: implementación del algoritmo MOG2 disponible [aquí](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.html)
- **im2vid**: genera video a partir de las imágenes ubicadas en una carpeta
- **showVideo**: muestra el video en una ventana flotante para visualizar mejor el resultado del modelo

Notar que, a pesar de tratarse de algoritmos que trabajan con video, el _dataset_ que se proporciona para realizar las pruebas son imágenes que representan los fotogramas de una secuencia. Por lo tanto, las funciones que se han realizado, se han pensado para trabajar con imágenes en vez de video.

Pasos a seguir para cada secuencia:
1. Cargar las imágenes en una variable (path de las imágenes)
2. Aplicar el filtro deseado EF, MOG y MOG2 (path donde se guardarán las imágenes aplicadas por el filtro)
3. Obtener las métricas con comparator (hacer bucle a lo largo de las imágenes resultado -> path del groundtruth)
4. Mostrar los resultados y videos
5. Comparar los resultados (hacer gráfica a lo largo del

```python
# dynamicBackground canoe dataset output paths
expFilter_canoe_path = 'DATA/dynamicBackground/results/canoe/expFilter/'
MOG_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG/'
MOG2_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG2'

# intermittentObject sofa dataset output paths
expFilter_sofa_path = 'DATA/intermittentObjectMotion/results/sofa/expFilter/'
MOG_sofa_path = 'DATA/intermittentObjectMotion/results/sofa/MOG'
MOG2_sofa_path = 'DATA/intermittentObjectMotion/results/sofa/MOG2'

# shadow copyMachine dataset output paths
expFilter_copyMachine_path = 'DATA/shadow/results/copyMachine/expFilter/'
MOG_copyMachine_path = 'DATA/shadow/results/copyMachine/MOG'
MOG2_copyMachine_path = 'DATA/shadow/results/copyMachine/MOG2'
```

# 1. Secuencia _highway_ del dataset _baseline_

Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.

```python
# baseline highway dataset groundtruth y input paths
gt_highway_path = 'DATA/baseline/baseline/highway/groundtruth/*.png'
in_highway_path = 'DATA/baseline/baseline/highway/input/*.jpg'

groundTruth = loadImages(gt_highway_path)
frames = loadImages(in_highway_path)
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2667.40it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:01<00:00, 1208.82it/s]

## 1.1. Algoritmo filtro exponencial

Aplicamos el algoritmo basado en el filtro exponencial que hemos implementado.

```python
# baseline highway dataset output path
expFilter_highway_path = 'DATA/baseline/results/highway/expFilter/'

exponentialFilter(frames, 0.3, expFilter_highway_path, 20, bgFilter ='MEDIAN')
```

    Algorithm application: 100%|█████████████████████████████████████████████████████| 1680/1680 [00:01<00:00, 1032.18it/s]

Convertimos la secuencia de imágenes en video para poder visualizar mejor el resultado.

```python
#im2vid(expFilter_highway_path+'*.png','testvid.mp4')
```

Mostramos el video en una ventana flotante.

```python
#showVideo('testvid.mp4')
```

### 1.1.1. Cálculo de métricas
Calculamos las métricas comparando con el _ground truth_ propocionado. Para cada secuencia se proporciona una ROI (Region Of Interest) sobre la cual se realiza el groundtruth, para simplificar el cálculo se han seleccionado aquellas secuencias en las que la ROI sea toda la imagen. Por otro lado, también se especifica el _frame_ de inicio y final sobre los que se ha de realizar la verificación del algoritmo, por lo tanto solo se calcularán las métricas dentro de ese rango.

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_EF = all_metrics (expFilter_highway_path+'*.png', gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1700/1700 [00:06<00:00, 262.06it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2594.43it/s]

```python
fig, (ax1, ax2, ax3) = plt.subplots(3, sharex=True, sharey=True)
ax1.plot(r_p_fm_frames_EF[0])
ax1.set_title("recall")
ax2.plot(r_p_fm_frames_EF[1])
ax2.set_title("precision")
ax3.plot(r_p_fm_frames_EF[2])
ax3.set_title("f-measure")
plt.ylim([0,1])
fig.tight_layout()
plt.show()
```

```python
precision_expFilter
```

    array([0.05653631, 0.04029126, 0.01318373, ..., 0.62876909, 0.61493709,
           0.6044432 ])

```python
plt.plot(tp_expFilter)

plt.show()
```

## 1.2. Algoritmo MOG

Aplicamos el algoritmo MOG (Mixture Of Gaussians)

```python
# baseline highway dataset output path
MOG_highway_path = 'DATA/baseline/results/highway/MOG/'

MOG(frames,MOG_highway_path)
```

    Algorithm application2: 100%|█████████████████████████████████████████████████████| 1700/1700 [00:03<00:00, 498.40it/s]

```python
im2vid(MOG_highway_path+'*.png','testvid.mp4')
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1700/1700 [00:06<00:00, 260.80it/s]
    Converting to video: 100%|███████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 3580.98it/s]

```python
showVideo('testvid.mp4')
```

### 1.2.1. Cálculo de métricas

```python
cm_frames_MOG,\
cm_secuencia_MOG,\
r_p_fm_frames_MOG,\
r_p_fm_secuencia_MOG = all_metrics (MOG_highway_path+'*.png', gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2133.43it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2555.43it/s]

```python
fig, (ax1, ax2, ax3) = plt.subplots(3, sharex=True, sharey=True)
ax1.plot(r_p_fm_frames_MOG[0])
ax2.plot(r_p_fm_frames_MOG[1])
ax3.plot(r_p_fm_frames_MOG[2])
plt.ylim([0,1])
plt.show()
```

```python
plt.plot(r_p_fm_frames_MOG[0])

plt.show()
```

## 1.3. Algoritmo MOG2

```python
# baseline highway dataset output path
MOG2_highway_path = 'DATA/baseline/results/highway/MOG2/'

MOG2(frames,MOG2_highway_path)
```

    Algorithm application: 100%|██████████████████████████████████████████████████████| 1700/1700 [00:04<00:00, 407.06it/s]

### 1.3.1. Cálculo de métricas

```python
cm_frames_MOG2,\
cm_secuencia_MOG2,\
r_p_fm_frames_MOG2,\
r_p_fm_secuencia_MOG2 = all_metrics (MOG2_highway_path+'*.png', gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1700/1700 [00:06<00:00, 255.43it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2536.53it/s]

## Comparación resultados

```python
cm_frames_gt,\
cm_secuencia_gt,\
r_p_fm_frames_gt,\
r_p_fm_secuencia_gt = all_metrics (gt_highway_path, gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2622.26it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2626.55it/s]

```python
fig, ax = plt.subplots(3,3, sharex=True, sharey=True)
ax[0,0].plot(r_p_fm_frames_EF[0])
ax[0,0].set_title("recall-EF")
ax[1,0].plot(r_p_fm_frames_EF[1])
ax[1,0].set_title("precision-EF")
ax[2,0].plot(r_p_fm_frames_EF[2])
ax[2,0].set_title("f-measure-EF")

ax[0,1].plot(r_p_fm_frames_MOG[0])
ax[0,1].set_title("recall-MOG")
ax[1,1].plot(r_p_fm_frames_MOG[1])
ax[1,1].set_title("precision-MOG")
ax[2,1].plot(r_p_fm_frames_MOG[2])
ax[2,1].set_title("f-measure-MOG")

ax[0,2].plot(r_p_fm_frames_MOG2[0])
ax[0,2].set_title("recall-MOG2")
ax[1,2].plot(r_p_fm_frames_MOG2[1])
ax[1,2].set_title("precision-MOG2")
ax[2,2].plot(r_p_fm_frames_MOG2[2])
ax[2,2].set_title("f-measure-MOG2")

plt.ylim([0,1])
fig.tight_layout()
plt.show()
```

```python
import pandas as pd
```

```python
r_p_fm_secuencia_EF
```

    array([0.60293747, 0.5       , 0.54666514])

```python
r_p_fm_secuencia_MOG
```

    array([0.50234751, 0.5       , 0.501171  ])

```python
r_p_fm_secuencia_MOG2
```

    array([0.92913693, 0.5       , 0.65013849])

```python
cm_secuencia_gt
```

    array([ 5448641.,        0.,        0., 86228220.])

```python
cm_secuencia_MOG-cm_secuencia_gt
```

    array([-3168986.,    68877.,  2258349.,   -14606.])

```python
np.concatenate((r_p_fm_secuencia_EF,cm_secuencia_EF))
```

    array([6.02937474e-01, 5.00000000e-01, 5.46665145e-01, 2.95299300e+06,
           3.57410400e+06, 1.94468400e+06, 6.80320250e+07])

```python
cm_secuencia_EF.sum()
```

    94464000.0

```python
cm_secuencia_MOG.sum()
```

    94464000.0

```python
cm_secuencia_MOG2.sum()
```

    92811260.0

```python
cm_secuencia_gt.sum()
```

    91676861.0

```python
mog_result = loadImages(MOG2_highway_path+'*.png')
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 1703.61it/s]

```python
mog_result[469].shape
```

    (240, 320, 3)

```python
ef_result = loadImages(expFilter_highway_path+'*.png')
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:01<00:00, 1313.18it/s]

```python
ef_result[469] = cv2.cvtColor(ef_result[469], cv2.COLOR_BGR2GRAY)
```

```python
ef_result[469]
```

    array([[  0,   0,   1, ..., 254,   3,   0],
           [  2,   0,   0, ..., 255,   0,   0],
           [254, 255,   0, ...,   0,   2,   1],
           ...,
           [  2,   0,   2, ...,   0,   0,   0],
           [  1,   0,   0, ...,   0,   1,   1],
           [  0,   1,   0, ...,   1,   1,   0]], dtype=uint8)

```python
df_highway = pd.DataFrame(np.array([np.concatenate((r_p_fm_secuencia_EF,cm_secuencia_EF)),
                                    np.concatenate((r_p_fm_secuencia_MOG,cm_secuencia_MOG)),
                                    np.concatenate((r_p_fm_secuencia_MOG2,cm_secuencia_MOG2)),
                                    np.concatenate((r_p_fm_secuencia_gt,cm_secuencia_gt))]),
                         columns = ['Recall', 'Precision','F-measure','TP','FP','FN','TN'])
df_highway['Algorithm'] = ['Exponential Filter','MOG','MOG2','Ground truth']
df_highway.set_index('Algorithm')
```

      Recall
      Precision
      F-measure
      TP
      FP
      FN
      TN

      Algorithm

      Exponential Filter
      0.602937
      0.5
      0.546665
      2952993.0
      3574104.0
      1944684.0
      68032025.0

      MOG
      0.502348
      0.5
      0.501171
      2279655.0
      68877.0
      2258349.0
      86213614.0

      MOG2
      0.929137
      0.5
      0.650138
      4012554.0
      1135816.0
      306028.0
      78852773.0

      Ground truth
      1.000000
      1.0
      1.000000
      5448641.0
      0.0
      0.0
      86228220.0

```python
r_p_fm_secuencia_EF
```

    array([0.60293747, 0.5       , 0.54666514])

```python
df_highway
```

      Recall
      Precision
      F-measure
      Algorithm

      0
      0.602937
      0.5
      0.546665
      Exponential Filter

      1
      0.502348
      0.5
      0.501171
      MOG

      2
      0.929137
      0.5
      0.650138
      MOG2

```python
df2 = pd.DataFrame(np.array([[1, 2, 3], [4, 5, 6], [7, 8, 9],[10,11,12]]),
                   columns=['a', 'b', 'c'])
df2
```

      a
      b
      c

      0
      1
      2
      3

      1
      4
      5
      6

      2
      7
      8
      9

      3
      10
      11
      12

# 2. Secuencia _badminton_ del dataset _cameraJitter_

Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.

```python
#camaraJitter  badminton dataset input
gt_badminton_path = 'DATA/cameraJitter/cameraJitter/badminton/groundtruth/*.png'
in_badminton_path = 'DATA/cameraJitter/cameraJitter/badminton/input/*.jpg'

groundTruth = loadImages(gt_badminton_path)
frames = loadImages(in_badminton_path)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1150/1150 [00:03<00:00, 296.71it/s]
    Loading images: 100%|█████████████████████████████████████████████████████████████| 1150/1150 [00:08<00:00, 129.45it/s]

## 2.1. Algoritmo filtro exponencial

```python
# camaraJitter badminton dataset output path
expFilter_badminton_path = 'DATA/cameraJitter/results/badminton/expFilter/'

```

### 2.1.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_EF = all_metrics (expFilter_badminton_path+'*.jpg', gt_badminton_path, start_frame=800, end_frame=1150)
```

## 2.2. Algoritmo MOG

```python
# camaraJitter badminton dataset output paths
MOG_badminton_path = 'DATA/cameraJitter/results/badminton/MOG/'
```

### 2.2.1. Cálculo de métricas

```python
cm_frames_MOG,\
cm_secuencia_MOG,\
r_p_fm_frames_MOG,\
r_p_fm_secuencia_MOG = all_metrics (MOG_badminton_path+'*.jpg', gt_badminton_path, start_frame=800, end_frame=1150)
```

## 2.3. Algoritmo MOG2

```python
# camaraJitter badminton dataset output paths
MOG2_badminton_path = 'DATA/cameraJitter/results/badminton/MOG2'
```

### 2.3.1. Cálculo de métricas

```python
cm_frames_MOG2,\
cm_secuencia_MOG2,\
r_p_fm_frames_MOG2,\
r_p_fm_secuencia_MOG2 = all_metrics (MOG2_badminton_path+'*.jpg', gt_badminton_path, start_frame=800, end_frame=1150)
```

## Comparación resultados

# 3. Secuencia _canoe_ del dataset _dynamicBackground_

```python
Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.
```

```python
# dynamicBackground  canoe dataset input
gt_canoe_path = 'DATA/dynamicBackground/dynamicBackground/canoe/groundtruth/*.png'
in_canoe_path = 'DATA/dynamicBackground/dynamicBackground/canoe/input/*.jpg'

groundTruth = loadImages(gt_canoe_path)
frames = loadImages(in_canoe_path)
```

```python
# dynamicBackground canoe dataset output path
expFilter_canoe_path = 'DATA/dynamicBackground/results/canoe/expFilter/'
MOG_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG/'
MOG2_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG2'
```

## 3.1. Algoritmo filtro exponencial

```python
# dynamicBackground canoe dataset output paths
expFilter_canoe_path = 'DATA/dynamicBackground/results/canoe/expFilter/'
```

### 3.1.1 Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_canoe_path+'*.jpg', gt_canoe_path, start_frame=469, end_frame=1699)
```

## 3.2. Algoritmo MOG

```python
# dynamicBackground canoe dataset output path
MOG_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG/'
```

### 3.2.1 Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (MOG_canoe_path+'*.jpg', gt_canoe_path, start_frame=469, end_frame=1699)
```

## 3.3. Algoritmo MOG2

```python
# dynamicBackground canoe dataset output path
MOG2_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG2'
```

### 3.3.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (MOG2_canoe_path+'*.jpg', gt_canoe_path, start_frame=469, end_frame=1699)
```

## 3.4. Comparación resultados

# 4. Secuencia _sofa_ del dataset _intermittentObjectMotion_

```python
Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.
```

```python
# intermittentObjectMotion sofa dataset input
gt_sofa_path = 'DATA/intermittentObjectMotion/intermittentObjectMotion/sofa/groundtruth/*.png'
in_sofa_path = 'DATA/intermittentObjectMotion/intermittentObjectMotion/sofa/input/*.jpg'

groundTruth = loadImages(gt_sofa_path)
frames = loadImages(in_sofa_path)
```

## 4.1. Algoritmo filtro exponencial

### 4.1.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 4.2. Algoritmo MOG

### 4.2.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 4.3. Algoritmo MOG2

### 4.3.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 4.4. Comparación resultados

# 5. Secuencia _copyMachine_ del dataset _shadow_

```python
Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.
```

```python
# shadow copyMachine dataset input
gt_copyMachine_path = 'DATA/shadow/shadow/copyMachine/groundtruth/*.png'
in_copyMachine_path = 'DATA/shadow/shadow/copyMachine/input/*.jpg'

groundTruth = loadImages(gt_copyMachine_path)
frames = loadImages(in_copyMachine_path)
```

## 5.1. Algoritmo filtro exponencial

### 5.1.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 5.2. Algoritmo MOG

### 5.2.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 5.3. Algoritmo MOG2

### 5.3.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 5.4. Comparación resultados


# Reinforcement Learning

In this lab, you will have the opportunity to explore Reinforcement Learning (RL) algorithms in practice, training agents to navigate through a 2D grid world to maximize their rewards. In a sequence of tasks, you will

1. discover the effect of randomness on RL training,
2. compare different RL algorithms trained on the same task,
3. implement the Q-Learning algorithm
4. explore whether the task is harder to learn if the agent started in a different random position in each episode, and
5. explore different tasks by modifying the environment's reward structure

## Setup

We start by importing dependencies and setting up some functions that will be useful in completing these tasks.

### Imports

```python
import numpy as np
from matplotlib import pyplot as plt
from grid_world import GridWorld  # Ensure the file grid_world.py is located in the same folder as this notebook.
```

### Environment

The class below defines the grid world environment that we've already seen in the lecture. It consists of a grid with 4 rows and 5 columns, specifying 20 environment states (one state for each grid cell). In each episode, the agent starts in state (0, 3). The cell (4, 0) is declared a terminal state, ending an episode whenever the agent transitions into that state. All state transitions yield a reward of 0 by default. The reward for transitions into two specific states are set differently, yielding a reward of 1 when an agent's action results in a transition into state (4, 0), and yielding a reward of -1 when an agent's action results in a transition into state (1, 2).

```python
class ExampleGrid(GridWorld):

    def __init__(self):
        super().__init__(rows = 4, cols = 5, agent_start = (0, 3))
        self.update_terminal(4, 0)  # Make state (4, 0) a terminal state.
        self.update_reward(4, 0, 1)  # Make transitions into state (4, 0) yield a reward of 1.
        self.update_reward(1, 2, -1)  # Make transitions into state (1, 2) yield a reward of -1.

env = ExampleGrid()  # Instantiate the example grid.
env.render()  # Display the environment state(green dot), terminal states (gray shaded cells), and rewards (numbers inside cells).
```

### Tabular Softmax Agent

This agent class implements an explicit policy where the probability distribution over actions given a state $\pi(a|s)$ is defined as the softmax over action-specific 'logits' $z$:
\begin{equation}
\pi(a=a_i|s) = \sigma(z_{s,a}) = \frac{e^{z_{s, a_i}}}{\sum_j e^{z_{s, a_j}}}
\end{equation}

Using a tabular state representation, the policy is parameterized with a separate parameter for each state-action pair $z_{s, a}$ with state $s: (x, y) \in \mathcal{S}$ and action $a \in \mathcal{A}$.

```python
class SoftmaxAgent():
    """
    A tabular agent parameterizing the logits of a softmax policy.

    Args:
        z (np.array): Parameters organized by [y, x, a], where (y, x) encode
        the state (as a cell in a grid world) and 'a' each possible action.
    """

    def __init__(self, z):
        self.z = z

    def act(self, state, epsilon):
        """
        Sample from the policy pi(a, s), where pi is defined as the
        softmax function with parameters self.z[y, x].
        """

        x, y = state
        p = np.exp(self.z[y, x]) / np.sum(np.exp(self.z[y, x]))
        a = np.random.choice(len(p), p = p)
        return a

    def rollout(self, env, epsilon = 0):
        """
        Complete a full episode interacting with the environment
        by following the current policy.
        """

        states, actions, rewards = [], [], []
        done = False

        s = env.reset()
        states.append(s)

        while not done:
            a = self.act(s, epsilon)
            s, r, done, _ = env.step(a)
            states.append(s)
            actions.append(a)
            rewards.append(r)

        return states, actions, rewards

    def render_pi(self, env, ax = None):
        """
        Display the learned policy.
        """

        if ax is None:
            fig, ax = env._create_fig()

        env.render_grid(ax)
        pi = np.exp(self.z) / np.expand_dims(np.sum(np.exp(self.z), axis = 2), axis = 2)

        for y in range(env.rows):
            for x in range(env.cols):
                # render distribution over actions
                p =  pi[y, x]
                p_max = np.max(p)
                for a in range(len(env.action_space)):
                    length = p[a] / (1e-6 + p_max) * 0.3
                    env.render_action(ax, action = env.action_space[a], agent = (x, y), length = length, hw = 0.1, hl = 0.15, ox = 0, oy = 0)
```

## Tasks

### 1. How does randomness affect training of REINFORCE ?

The code snippet below trains a Tabular Softmax policy with REINFORCE for 1000 rollouts (episodes) and then plots the return of each rollout. It also illustrates the trained policy and shows an example rollout from the last training step.

Write a function that repeats training 5 times and then plots mean, min and max reward after each rollout.

#### Solution Steps (implemented in the code below)
1. wrap code inside a function that returns training returns
2. call function 5 times
3. *convert all results into (5xN) numpy array (for convenience)
4. plot mean, min and max along axis=0

```python
def train_reinforce() -> list:
    '''
    Train a Tabular Softmax policy with REINFORCE.

    Returns
    -------
    returns : training returns
    '''

    env = ExampleGrid()  # Instantiate environment.
    logits = np.ones((env.rows, env.cols, len(env.action_space)))  # Instantiate policy.
    agent = SoftmaxAgent(z = logits)
    eta = 0.01
    gamma = 0.95
    returns = []

    for i in range(1000):
        states, actions, rewards = agent.rollout(env)
        G = np.sum(rewards * gamma**np.arange(len(rewards)))  # Here, the decay gamma is omitted, implicitly assuming it to be 1.
        returns.append(G)
        z = agent.z
        dz = np.zeros_like(z)

        for t in range(len(states) - 1):
            x, y = states[t]
            a = actions[t]
            G = np.sum(rewards[t:] * gamma**np.arange(len(rewards[t:])))
            dz[y,x] -= G
            dz[y,x, a] += G

        z = agent.z + eta * dz  # Apply gradient with learning rate eta.
        agent.z = np.maximum(-6, np.minimum(6, z))  # Saturate logits to avoid distribution collapse.

    return returns

reinforce_returns = []

for _ in range(5):
    reinforce_returns.append(train_reinforce())

reinforce_returns = np.array(reinforce_returns)
fig, ax = plt.subplots(figsize = (10, 8))
ax.plot(np.mean(reinforce_returns, axis = 0), 'r', label = 'mean')
ax.plot(np.min(reinforce_returns, axis = 0), 'r--', alpha = 0.25)
ax.plot(np.max(reinforce_returns, axis = 0), 'r--', alpha = 0.25)
plt.xlabel('Episode')
plt.ylabel('Returns')
plt.legend()
plt.show()
```

## 2. How does Actor-Critic compare to REINFORCE in training performance?

The code snippet below trains a Tabular Softmax policy with Actor-Critic for 1000 rollouts (episodes) and then plots the return of each rollout. It also illustrates the trained policy and shows an example rollout from the last training step.

Train Actor-Critic 5 times and compare mean, min and max return in each rollout with the mean min and max return observed during REINFORCE training.

#### Solution Steps (implemented in the code below)
1. apply all steps from task 1. above to the code below
    1. wrap code inside a function that returns training returns
    2. call function 5 times
    3. *convert all results into (5xN) numpy array (for convenience)
    4. plot mean, min and max along axis=0
2. plot results from task 1. in the same figure

```python
def train_ac():
    '''
    Train a Tabular Softmax policy with Actor-Critic.

    Returns
    -------
    returns : training returns
    '''

    env = ExampleGrid()  # Instantiate environment.
    logits = np.ones((env.rows, env.cols, len(env.action_space)))  # Instantiate policy.
    agent = SoftmaxAgent(z = logits)
    eta = 0.2
    gamma = 1
    Q = np.zeros_like(logits)  # Initialize the critic.
    returns = []

    for i in range(1000):
        states, actions, rewards = agent.rollout(env)
        G = np.sum(rewards)
        returns.append(G)
        G -= np.mean(returns)
        p = np.exp(agent.z) / np.expand_dims(np.sum(np.exp(agent.z), axis = 2), axis = 2)
        dz = np.zeros_like(agent.z)

        for t in range(len(states) - 1):
            x, y = states[t]
            a = actions[t]

            # Estimate updated Q.
            x_next, y_next = states[t + 1]
            v = np.sum(Q * p, axis = 2)[y_next, x_next]
            Q[y, x, a] = rewards[t] + gamma * v

            # Estimate gradient,
            # Here, we scale gradients with the relevant Q-value in each step instead of using the episode return at the end.
            dz[y, x] -= Q[y, x, a]
            dz[y, x, a] += Q[y, x, a]

        z = 0.99*agent.z + eta * dz  # Update parameters.
        agent.z = np.maximum(-6, np.minimum(6, z))  # Saturate parameters to prevent distribution collapse.

    return returns

ac_returns = []

for _ in range(5):
    ac_returns.append(train_ac())

ac_returns = np.array(ac_returns)
fig, ax= plt.subplots(figsize=(10, 8))
ax.plot(np.mean(reinforce_returns, axis = 0), 'r', label = 'REINFORCE')
ax.plot(np.min(reinforce_returns, axis = 0), 'r--', alpha = 0.25)
ax.plot(np.max(reinforce_returns, axis = 0), 'r--', alpha = 0.25)
ax.plot(np.mean(ac_returns, axis = 0), 'b', label = 'Actor-Critic')
ax.plot(np.min(ac_returns, axis = 0), 'b--', alpha = 0.25)
ax.plot(np.max(ac_returns, axis = 0), 'b--', alpha = 0.25)
plt.legend()
plt.show()
```

## 3. Implement the Q-Learning algorithm

### 3.1 Action Selection

Implement the ```act``` method of the `QAgent` below to select the action with maximum Q-value.

Amend your implementation to an $\epsilon$-greedy policy that returns a random action with probability $\epsilon$ and returns the action with maximum Q-value with probability $(1-\epsilon)$.

```python
class QAgent(SoftmaxAgent):
    """
    A tabular agent parameterizing the Q-values corresponding to the policy.

    Args:
        q (np.array): Parameters organized by [y, x, a], where (y, x) encode
                    the state (as a cell in a grid world) and 'a' each possible action.

    """

    def __init__(self, q):
        self.q = q

    def act(self, state, epsilon = 0.1):
        """ Select the action with maximum q-value for the given state.

        Steps:
            1. return the action with maximum Q-value. Assume all relevant Q-values
               are stored in self.q[y, x]. The function np.argmax returns the index
               of the maximum value (consult the numpy documentation online for
               examples).

            2. implement an epsilon-greedy policy with random exploration probability
               passed in through the parameter "epsilon".

        """

        if np.random.uniform() < epsilon:
            a = np.random.randint(4)  # Number of actions hard-coded here for simplicity; could also read it from "self.q.shape[-1]".
        else:
            x, y = state
            q_s = self.q[y, x]
            a = 0  # Set "a" to the correct return value.

        return a

    def render_pi(self, env, ax = None):
        if ax is None:
            fig, ax = env._create_fig()

        env.render_grid(ax)

        for y in range(env.rows):
            for x in range(env.cols):
                env.render_action(ax, action = np.argmax(self.q[y, x]), agent = (x, y), length = 0.3, hw = 0.1, hl = 0.15, ox = 0, oy = 0)
```

### 3.2 Q-Learning Update

Implement the Q-Learning update $Q(s_t, a_t) \leftarrow r + \gamma \, \text{argmax}_j \, Q(s_{t+1}, a_j)$

```python
def train_qlearning():
    env = ExampleGrid()
    Q = np.zeros((env.rows, env.cols, len(env.action_space)))
    agent = QAgent(q = Q)
    epsilon = 1
    gamma = 0.95
    returns = []

    for i in range(200):
        # Do training rollout with epsilon-greedy policy.
        states, actions, rewards = agent.rollout(env, epsilon)
        G = np.sum(rewards * gamma**np.arange(len(rewards)))
        returns.append(G)

        # Update Q-Table.
        for t in range(len(states) - 1):
            x, y = states[t]
            a = actions[t]
            x_next, y_next = states[t + 1]
            Q[y, x, a] = rewards[t] + gamma * np.max(Q[y_next, x_next])

        epsilon = max(0.05, 0.9 * epsilon)  # Decay epsilon.

    return returns
```

### 3.3 Evaluation

Evaluate your algorithm, training 5 times and plot mean, min and max return after each rollout

```python
q_returns = []

for _ in range(5):
    q_returns.append(train_qlearning())

q_returns = np.array(q_returns)
n_steps = 100  # Number of training steps to plot.
fig, ax= plt.subplots(figsize=(10, 8))
ax.plot(np.mean(reinforce_returns, axis = 0)[:n_steps], 'r', label = 'REINFORCE')
ax.plot(np.min(reinforce_returns, axis = 0)[:n_steps], 'r--', alpha = 0.25)
ax.plot(np.max(reinforce_returns, axis = 0)[:n_steps], 'r--', alpha = 0.25)
ax.plot(np.mean(ac_returns, axis = 0)[:n_steps], 'b', label = 'Actor-Critic')
ax.plot(np.min(ac_returns, axis = 0)[:n_steps], 'b--', alpha = 0.25)
ax.plot(np.max(ac_returns, axis = 0)[:n_steps], 'b--', alpha = 0.25)
ax.plot(np.mean(q_returns, axis = 0)[:n_steps], 'g', label = 'Q-Learning')
ax.plot(np.min(q_returns, axis = 0)[:n_steps], 'g--', alpha = 0.25)
ax.plot(np.max(q_returns, axis = 0)[:n_steps], 'g--', alpha = 0.25)
plt.legend()
plt.show()
```

## 4. Does the task get harder to learn if the agent starts in a different random position?

Adapt the environmnet such that the the agent starts at a different (random) position in each rollout and explore the training progress of one RL algorithm. You may need to adapt the number of rollouts, the learning rate, or the exploration rate.

```python
class ExampleGrid(GridWorld):

    def __init__(self):
        agent_x = np.random.randint(5)
        agent_y = np.random.randint(4)
        super().__init__(rows = 4, cols = 5, agent_start = (agent_x, agent_y))
        self.update_terminal(4, 0)  # Make state (4, 0) a terminal state.
        self.update_reward(4, 0, 1)  # Make transitions into state (4, 0) yield a reward of 1.
        self.update_reward(1, 2, -1)  # Make transitions into state (1, 2) yield a reward of -1.

env = ExampleGrid()  # Instantiate the example grid.
env.render()  # Display the environment state(green dot), terminal states (gray shaded cells), and rewards (numbers inside cells).
```

```python
random_starts = []

for _ in range(5):
    random_starts.append(train_reinforce())

random_starts = np.array(random_starts)
n_steps = 1000  # Number of training steps to plot.
fig, ax= plt.subplots(figsize=(10, 8))
ax.plot(np.mean(reinforce_returns, axis = 0)[:n_steps], 'k', label = 'REINFORCE')
ax.plot(np.min(reinforce_returns, axis = 0)[:n_steps], 'k--', alpha = 0.25)
ax.plot(np.max(reinforce_returns, axis = 0)[:n_steps], 'k--', alpha = 0.25)
ax.plot(np.mean(random_starts, axis = 0)[:n_steps], 'g', label = 'Random Starts')
ax.plot(np.min(random_starts, axis = 0)[:n_steps], 'g--', alpha = 0.25)
ax.plot(np.max(random_starts, axis = 0)[:n_steps], 'g--', alpha = 0.25)
plt.legend()
plt.show()
```

## 5. Explore different tasks by modifying the environment's reward structure

Change the reward structure of the environment to represent a different custom task of your choice and explore the training progress of one RL algorithm, and interpret the learned policy. You may need to adapt the number of rollouts, the learning rate, or the exploration rate (epsilon).

#### Solution

This is an open-ended exercise, where you can use `self.update_reward` to change the reward for other cells in the grid and re-run their evaluation code.

```python
class ExampleGrid(GridWorld):

    def __init__(self):
        super().__init__(rows = 4, cols = 5, agent_start = (0, 3))
        self.update_terminal(4, 0)  # Make state (4, 0) a terminal state.
        self.update_reward(4, 0, 1)  # Make transitions into state (4, 0) yield a reward of 1.
        self.update_reward(1, 2, -1)  # Make transitions into state (1, 2) yield a reward of -1.

env = ExampleGrid()  # Instantiate the example grid.
env.render()  # Display the environment state(green dot), terminal states (gray shaded cells), and rewards (numbers inside cells).
```

Labs/Week5/MLAIDS - Reinforcement Learning.ipynb


# CHEM 1000 - Spring 2022
Prof. Geoffrey Hutchison, University of Pittsburgh

## 9 Probability

Chapter 9 in [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

(These lectures notes on probability and statistics will include substantial material not found in the text.)

By the end of this session, you should be able to:
- Understand the binomial and multinomial processes
- Compute cumulative chances (e.g., lottery)
- Understand calculating moments from probability distributions
  - Mean, variance, skew, and kurtosis

### Randomness, Probability and Chemistry

A common technique in simulating chemistry and physical behavior is called [Monte Carlo](https://en.wikipedia.org/wiki/Monte_Carlo_method) - essentially using random processes to solve complicated problems. For example, one can randomly sample many possible shapes of a polymer or sample from multiple possible arrangements in a nanoparticle, etc.

I will assume you have some background in probability and statistics and focus only on some key areas.

### Coin Flips and Binomial Distribution

We're generally familiar with *descrete* random numbers, like flipping a coin heads or tails.

If we flip a coin once, it's 1/2 chance of heads or tails.

Over multiple events - each one is independent - the probability of a particular number of heads (n) in a total of N flips is:

$$
p(n, N) =\frac{1}{2^{N}} \frac{N !}{n !(N-n) !}
$$

For 6 flips, this looks like:

Image from [*Mathematical Methods for Chemists*](http://sites.bu.edu/straub/mathematical-methods-for-molecular-science/)

We can calculate the counts either using the formula above, or via [Pascal's Triangle](https://en.wikipedia.org/wiki/Pascal%27s_triangle)

We can relate coin-flips with electron spin or nuclear spin (for NMR), etc. For example, given 3 unpaired electrons, how many arrangements are there?

- Up, Up, Up
- Up, Up, Down
.. (etc)

### Multinomial Probability

Obviously we don't just flip coins (or electrons) so we also need to consider a multinomial distribution (i.e., rolling a 6-sided dice, or a system that can be in multiple equivalent states):

For example, if we roll a six-sided die, 5 times how many ways can we get 2 ones and 3 sixes (1 1 6 6 6):

$$
W(2,0,0,0,0,3 ; 5)=\frac{5 !}{2 ! 0 ! 0 ! 0 ! 0 ! 3 !}=10
$$

(Important to remember that 0! = 1)

### Cumulative Probability

One common real-world probability question is about cumulative chances.

My son came to me, asking about an iPad game where he can win a prize every time he opens a gift. Each gift has a chance of winning the prize. So he asks me if he buys 30 gifts, what's the chance he'll win the super-awesome dragon?

That's a cumulative probability - he doesn't care *which* gift gives him the dragon, only that one of the thirty gifts works.

Here's the catch - the game only gives dragons with 1% chance - you're more likely to get other prizes.

While there's a formula, it's really, really easy to compute this with a for() loop.
- what's the total cumulative chance?
- what's the chance we didn't get the prize on the last round

e.g.
- first time through, there's a 1% chance of a win
- second time, there was a 99% chance we didn't win, times the 1% chance I win on this round = 0.99% chance on this gift
- third time, there's a 99.01% we didn't win on the 2nd round, times the 1% chance on the 3rd round
- etc.

```python
# help danny
total = 0.0 # start out with no chance to win the prize
missed = 1.0 # i.e., he doesn't have the prize yet
chance = 0.01 # chance of winning each time he opens the gift

for egg in range(1,31): # remember the loop will go from start to end - 1 = 30
    total = total + chance * missed
    missed = missed * (1.0 - chance)
    print(egg, round(total, 4))
```

Notice that even though there are 30 gifts, his cumulative probability is **not** 30 * 0.01, but lower...

Not surprisingly, he decides a 26% chance of getting a dragon isn't very good and he picks a different game.

No dragons, but this new game has a 4% chance of winning each time he plays.

How long does he have to go for a good chance of winning a prize?

```python
# is this a better game?
total = 0.0 # start out with no chance to win the prize
missed = 1.0 # i.e., he doesn't have the prize yet
chance = 0.04 # chance of winning each time he opens the gift

for egg in range(1,21): # remember the loop will go from start to end - 1 = 20
    total = total + chance * missed
    missed = missed * (1.0 - chance)
    print(egg, round(total, 4))
```

So it's 17 rounds before we break 50% so it's clearly better than the first game. Still, the cumulative chance is not N * 0.04...

(Danny decided neither game was worth the money, incidentally.)

### Moments from Distributions

Sometimes people will discuss "[*moments*](https://en.wikipedia.org/wiki/Moment_(mathematics))" of probability distributions or statistical distributions.

These are related to the shape of the distribution.
- the "zeroth" moment is the total (e.g., for a probability it should be 1 = 100%)
- first moment is the [mean](https://en.wikipedia.org/wiki/Expected_value) $\mu$ (i.e., the center or "expected value")
- second moment is the [variance](https://en.wikipedia.org/wiki/Variance) $\sigma^2$ (i.e., the width)
    - you're probably more familiar with the standard deviation $\sigma$
- third moment is the [skewness](https://en.wikipedia.org/wiki/Skewness) (i.e., the asymmetry of the distribution)
- fourth moment, the [kurtosis](https://en.wikipedia.org/wiki/Kurtosis) (i.e., how thin or thick the "tail" of the distribution)

In general, the mean, variance (or standard deviation which is the square root of the variance) and the skewness are the most useful measures of a distribution.

#### Skewness

Not all distributions are "normal" or symmetric. For example, the number of people waiting for a bus is never negative. Even if I tell you the average is 5 people in the morning, sometimes it's zero (when the bus just arrived) and it's sometimes much higher (right before the bus comes and someone runs to catch it).

Image from Wikipedia:

#### Kurtosis

Kurtosis is a measure of whether the data are heavy-tailed or light-tailed relative to a normal distribution. That is, distributions with high kurtosis tend to have many outliers.

This is probably easier to plot:

```python
# let's plot this
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

%config InlineBackend.figure_format = 'retina'
plt.style.use('../lectures/chem1000.mplstyle')
```

```python
# generate a "normal" distribution

mu, sigma = 0, 0.1 # mean and standard deviation
points = np.random.normal(mu, sigma, 1000) # 1,000 points from a normal distribution
```

```python
import scipy.stats

print('mean', np.mean(points))
print('variance', np.var(points)) # variance

print('skewness', scipy.stats.skew(points))
print('kurtosis', scipy.stats.kurtosis(points))
```

```python
count, bins, ignored = plt.hist(points, 30, density=True)

# add a red line with the perfect curve from a Gaussian distribution
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
               np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
         linewidth=2, color='r')

plt.show()
```

```python
# here's a different distribution (Cauchy - the same as a Lorentzian peak in spectra)
from scipy.stats import cauchy

mu, sigma = 0, 0.1 # mean and standard deviation
points = cauchy.rvs(mu, sigma, 1000)

print('mean', np.mean(points))
print('variance', np.var(points)) # variance

print('skew', scipy.stats.skew(points))
print('kurtosis', scipy.stats.kurtosis(points))
```

```python
count, bins, ignored = plt.hist(points, 1000, density=True)

# add a red line with the perfect curve from a Gaussian distribution
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
               np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
         linewidth=2, color='r')

plt.xlim(-0.5, 0.5)
plt.show()
```

Notice that there's not as much in the middle and more on the outside? That's **kurtosis**.

Can we generate some skew?

Yes, there are many kinds of distributions, including intentionally skewed distributions

```python
from scipy.stats import skewnorm

mu, sigma = 0, 0.1 # mean and standard deviation
asym = 4 # skew parameter
points = skewnorm.rvs(asym, mu, sigma, 1000)

print('mean', np.mean(points))
print('variance', np.var(points)) # variance

print('skew', scipy.stats.skew(points))
print('kurtosis', scipy.stats.kurtosis(points))
```

```python
count, bins, ignored = plt.hist(points, 30, density=True)

# add a red line with the perfect curve from a Gaussian distribution
plt.plot(bins, 1/(sigma * np.sqrt(2 * np.pi)) *
               np.exp( - (bins - mu)**2 / (2 * sigma**2) ),
         linewidth=2, color='r')

plt.show()
```

### Common Probability Distributions

The following are all continuous distributions:

- Uniform random (`scipy.stats.uniform`)
- Gaussian / normal (`scipy.stats.norm`)
- Cauchy (Lorentzian) (`scipy.stats.cauchy`)
- Exponential (`scipy.stats.expon`)
  - example: exponential decay of radioactive elements

The [Poisson distribution](https://en.wikipedia.org/wiki/Poisson_distribution) and [binomial distribution](https://en.wikipedia.org/wiki/Poisson_distribution) are discrete probability distributions (e.g., 5 people at the bus stop, or 6 heads when flipping coins).

### Calculating Mean, Variance, etc. for a Probability Distribution

When we have discrete data, it's easy to calculate a mean. We add up the values and divide by the count. However, it's not so hard when we have a continuous probability distribution either.

Consider an equivalent way to calculate the mean - it's the value times the probability (e.g., a weighted mean):

$$
\bar{x}=\sum_{k=1}^{n} x_{k} p_{k}
$$

In other words, we add up the values times the probability of occurring.

With a continuous probability distribution, we "just" change the sum to an integral.

$$
\sum_{k} p_{k} \rightarrow \int p(x) d x
$$

So now if we want to calculate the mean of a probability distribution, we need:

$$
\int x p(x) dx
$$

For example:

$$
\bar{x}=\int_{a}^{b} x p(x) d x
$$

Similarly, if we want the average of $x^2$ we can use:
$$
\overline{x^{2}}=\int_{a}^{b} x^{2} p(x) d x
$$

To calculate the variance, we need:
$$
\sigma_{x}^{2}=\overline{(x-\bar{x})^{2}}=\overline{x^{2}}-(\bar{x})^{2}
$$

That might look confusing, but just means we want the difference:
- the average of $x^2$
- the average of $x$ and then square that number

Below I've taken the integrals for the particle in a box, e.g.:

$$
p(x) = \psi^*\psi = \frac{2}{L} \sin^2 (\frac{n \pi x}{L})
$$

```python
from sympy import init_session
init_session()
```

```python
L = symbols('L')
f = 2*x * sin(n*pi*x/L)**2/L
simplify(integrate(f, (x, 0, L)))
```

```python
x_sq = 2*x**2*sin(n*pi*x/L)**2 / L
simplify(integrate(x_sq, (x, 0, L)))
```

```python
# variance = average(x**2) - average(x)**2
var = integrate(x_sq, (x, 0, L)) - integrate(f, (x, 0, L))**2
simplify(var)
```

-------
This notebook is from Prof. Geoffrey Hutchison, University of Pittsburgh
https://github.com/ghutchis/chem1000