# Signals and Sampling

In this notebook, we introduce signals and discuss the concept of sampling to convert a continuous into a discrete time axis. Using sinusoidal signals as example, we then cover phenomena such as aliasing and interference. This notebook is based on material from the
[FMP Notebook on Sampling](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C2/C2S2_DigitalSignalSampling.html) and the [FMP Notebook on Interference and Beating](https://www.audiolabs-erlangen.de/resources/MIR/FMP/C2/C2S3_InterferenceBeating.html).

Signals

Sampling

Aliasing

Interference

Exercise 1: Beating

Exercise 2: Aliasing with Sinsuoids

Exercise 3: Visual Aliasing

## Signals

A **sound** is generated by a vibrating object such as the vocal cords of a singer, the string and soundboard of a violin, or the diaphragm of a kettledrum. In signal processing, such a sound is typically represented by a function or **signal** $f\colon\mathbb{R}\to\mathbb{R}$, which encodes the sound's air pressure changes over time. The signal is called **periodic**, if high and low air pressure repeat in an alternating and regular fashion. In this case, the **period** of the signal is defined as the time required to complete a cycle. The **frequency**, measured in **Hertz** (Hz), is the reciprocal of the period. The prototype of such a periodic signal is a **sinusoid**, which is specified by its **frequency**, its **amplitude** (the peak deviation of the sinusoid from its mean), and its **phase** (determining where in its cycle the sinusoid is at time zero). In the following code cell, we provide a function for generating a sinusoid.

```python
import numpy as np
from matplotlib import pyplot as plt
%matplotlib inline

def generate_sinusoid(dur=1, amp=1, freq=1, phase=0, Fs=100):
    """Generation of sinusoid

    Notebook: PCP_signal.ipynb

    Args:
        dur: Duration (in seconds)
        amp: Amplitude of sinusoid
        freq: Frequency of sinusoid
        phase: Phase of sinusoid
        Fs: Sampling rate

    Returns:
        x: Signal
        t: Time axis (in seconds)
    """
    num_samples = int(Fs * dur)
    t = np.arange(num_samples) / Fs
    x = amp * np.sin(2 * np.pi * (freq * t - phase))
    return x, t

amp = 1.2
freq = 2
phase = 0.2
x, t = generate_sinusoid(dur=5, amp=amp, freq=freq, phase=phase, Fs=1000)

plt.figure(figsize=(8, 2.5))
plt.plot(t, x, color='blue', linewidth=2.0, linestyle='-')
plt.xlim([0, t[-1]])
plt.ylim([-amp*1.2, amp*1.2])
plt.xlabel('Time (seconds)')
plt.ylabel('Amplitude')
plt.title('Sinusoid with freq = %.1f, amp = %.1f, and phase = %.1f' % (freq, amp, phase))
plt.grid()
plt.tight_layout()
```

Interpreted as audio signal, one may listen to the sound of a sinusoid as long as its frequency lies in the human hearing range (roughly between $20$ and $20,000~\mathrm{Hz}$). To integrate an audio signal into notebook, we can use the module [`IPython.display`](https://ipython.readthedocs.io/en/stable/api/generated/IPython.display.html), which is an application programming interface (API) for displaying various tools in IPython. As for audio, one can use the class [`IPython.display.Audio`](https://ipython.org/ipython-doc/3/api/generated/IPython.display.html),  which creates an in-browser audio player. In the following code cell, this functionality is demonstrated using a sinusoid of $440~\mathrm{Hz}$.

```python
import IPython.display as ipd

Fs = 4000
dur = 1
x, t = generate_sinusoid(dur=dur, amp=1, freq=440, Fs=Fs)

ipd.display(ipd.Audio(x, rate=Fs))
```

## Sampling

In digital signal processing, one often reduces a **continuous-time** (CT) signal $f\colon\mathbb{R}\to\mathbb{R}$ into a **discrete-time** (DT) signal $x\colon\mathbb{Z}\to\mathbb{R}$ by a procedure known as **equidistant sampling**. Fixing a positive real number $T>0$, the DT-signal $x$ is obtained by setting

\begin{equation}
   x(n):= f(n \cdot T)
\end{equation}

for $n\in\mathbb{Z}$. The value $x(n)$ is called the **sample** taken at time $t=n\cdot T$ of the original analog signal $f$. In short, this procedure is also called **$T$-sampling**. The number $T$ is referred to as the **sampling period** and the inverse $F_\mathrm{s}:=1/T$ as the **sampling rate**. The sampling rate specifies the number of samples per second and is measured in Hertz (Hz).

In the following code cell, we start with a CT-signal $f$ that is defined via linear interpolation of a DT-signal sampled at a high sampling rate (generated by the `generate_example_function`). In the plot, this CT-signal is shown as black curve. Applying equidistant sampling (`sampling_equidistant`), we obtain the DT-signal visualized as the red stem plot.

```python
def generate_example_signal(dur=1, Fs=100):
    """Generate example signal

    Notebook: PCP_signal.ipynb

    Args:
        dur: Duration (in seconds) of signal to be generated
        Fs: Sampling rate

    Returns:
        x: Signal
        t: Time axis (in seconds)
    """
    N = int(Fs * dur)
    t = np.arange(N) / Fs
    x = 1 * np.sin(2 * np.pi * (1.9 * t - 0.3))
    x += 0.5 * np.sin(2 * np.pi * (6.1 * t - 0.1))
    x += 0.1 * np.sin(2 * np.pi * (20 * t - 0.2))
    return x, t

def sampling_equidistant(x_1, t_1, Fs_2, dur=None):
    """Equidistant sampling of interpolated signal

    Notebook: PCP_signal.ipynb

    Args:
        x_1: Signal to be interpolated and sampled
        t_1: Time axis (in seconds) of x_1
        Fs_2: Sampling rate used for equidistant sampling
        dur: Duration (in seconds) of sampled signal

    Returns:
        x_2: Sampled signal
        t_2: time axis (in seconds) of sampled signal
    """
    if dur is None:
        dur = len(t_1) * t_1[1]
    N = int(Fs_2 * dur)
    t_2 = np.arange(N) / Fs_2
    x_2 = np.interp(t_2, t_1, x_1)
    return x_2, t_2

Fs_1 = 100
x_1, t_1 = generate_example_signal(Fs=Fs_1, dur=2)

Fs_2 = 20
x_2, t_2 = sampling_equidistant(x_1, t_1, Fs_2)

plt.figure(figsize=(8, 2.2))
plt.plot(t_1, x_1, 'k')
plt.title('Original CT-signal')
plt.xlabel('Time (seconds)')
plt.ylim([-1.8, 1.8])
plt.xlim([t_1[0], t_1[-1]])
plt.tight_layout()

plt.figure(figsize=(8, 2.2))
plt.stem(t_2, x_2, linefmt='r', markerfmt='ro', basefmt='None', use_line_collection=True)
plt.plot(t_1, x_1, 'k', linewidth=1, linestyle='dotted')
plt.title(r'DT-Signal obtained by sampling with $F_\mathrm{s} = %.0f$' % Fs_2)
plt.xlabel('Time (seconds)')
plt.ylim([-1.8, 1.8])
plt.xlim([t_1[0], t_1[-1]])
plt.tight_layout()

plt.figure(figsize=(8, 2.2))
plt.stem(x_2, linefmt='r', markerfmt='ro', basefmt='None', use_line_collection=True)
plt.title(r'DT-Signal')
plt.xlabel('Time (samples)')
plt.ylim([-1.8, 1.8])
plt.xlim([0, len(t_2)])
plt.tight_layout()
```

## Aliasing

In general, sampling is a **lossy** operation in the sense that information is lost in this process and that the original CT-signal cannot be recovered from its sampled version. Only if the CT-signal has additional properties in terms of its frequency spectrum (it needs to be **bandlimited**), a perfect reconstruction is possible. This is the assertion of the famous **sampling theorem**. The sampling theorem also shows how the original CT-signal can be reconstructed by superimposing suitably shifted $\mathrm{sinc}$-functions weighted by the samples of the DT-signal.

Recap: Sampling Theorem

The sampling theorem, which is often associated with the names Harry Nyquist and Claude Shannon, states that a continuous-time (CT) signal that is bandlimited can be reconstructed perfectly under certain conditions. More precisely, a CT-signal $f\in L^2(\mathbb{R})$ is called $\Omega$-bandlimited if the Fourier transform $\hat{f}$ vanishes for $|\omega|>\Omega$ (i.e., $\hat{f}(\omega) = 0$ for $|\omega|>\Omega$). Let $f\in L^2(\mathbb{R})$ be an $\Omega$-bandlimited function and let $x$ be the $T$-sampled version of $f$ with $T:=1/(2\Omega)$ (i.e., $x(n)=f(nT)$, $n\in\mathbb{Z}$). Then $f$ can be reconstructed from $x$ by

$$
 f(t)=\sum_{n\in\mathbb{Z}}x(n)\mathrm{sinc}\left(\frac{t-nT}{T}\right)
 =\sum_{n\in\mathbb{Z}}f\left(\frac{n}{2\Omega}\right) \mathrm{sinc}\left(2\Omega t-n\right),
$$

where the $\mathrm{sinc}$-function is defined as

\begin{equation}
    \mathrm{sinc}(t):=\left\{\begin{array}{ll}
    \frac{\sin \pi t}{\pi t},&\mbox{ if $t\not= 0$,}\\
    1,&\mbox{ if $t= 0$.}
\end{array}\right.
\end{equation}

In other words, the CT-signal $f$ can be perfectly reconstructed from the DT-signal obtained by equidistant sampling if the bandlimit is no greater than half the sampling rate. This reconstruction based on the $\mathrm{sinc}$-function has been used in the function `reconstruction_sinc`.

Without additional properties, sampling may cause an effect known as **aliasing** where certain frequency components of the signal become indistinguishable. This effect is illustrated by the following figure. Using a high sampling rate, the original CT-signal can be reconstructed with high accuracy. However, when decreasing the sampling rate, the higher-frequency components are not captured well and only a coarse approximation of the original signal remains. This phenomenon is demonstrated by the following example.

```python
def reconstruction_sinc(x, t, t_sinc):
    """Reconstruction from sampled signal using sinc-functions

    Notebook: PCP_signal.ipynb

    Args:
        x: Sampled signal
        t: Equidistant discrete time axis (in seconds) of x
        t_sinc: Equidistant discrete time axis (in seconds) of signal to be reconstructed

    Returns:
        x_sinc: Reconstructed signal having time axis t_sinc
    """
    Fs = 1 / t[1]
    x_sinc = np.zeros(len(t_sinc))
    for n in range(0, len(t)):
        x_sinc += x[n] * np.sinc(Fs * t_sinc - n)
    return x_sinc

def plot_signal_reconstructed(t_1, x_1, t_2, x_2, t_sinc, x_sinc, figsize=(8, 2.2)):
    """Plotting three signals
    Notebook: PCP_signal.ipynb
    """
    plt.figure(figsize=figsize)
    plt.plot(t_1, x_1, 'k', linewidth=1, linestyle='dotted', label='Orignal signal')
    plt.stem(t_2, x_2, linefmt='r:', markerfmt='r.', basefmt='None', label='Samples', use_line_collection=True)
    plt.plot(t_1, x_sinc, 'b', label='Reconstructed signal')
    plt.title(r'Sampling rate $F_\mathrm{s} = %.0f$' % (1/t_2[1]))
    plt.xlabel('Time (seconds)')
    plt.ylim([-1.8, 1.8])
    plt.xlim([t_1[0], t_1[-1]])
    plt.legend(loc='upper right', framealpha=1)
    plt.tight_layout()
    plt.show()

Fs_2 = 40
x_2, t_2 = sampling_equidistant(x_1, t_1, Fs_2)
t_sinc = t_1
x_sinc = reconstruction_sinc(x_2, t_2, t_sinc)
plot_signal_reconstructed(t_1, x_1, t_2, x_2, t_sinc, x_sinc)

Fs_2 = 20
x_2, t_2 = sampling_equidistant(x_1, t_1, Fs_2)
t_sinc = t_1
x_sinc = reconstruction_sinc(x_2, t_2, t_sinc)
plot_signal_reconstructed(t_1, x_1, t_2, x_2, t_sinc, x_sinc)

Fs_2 = 10
x_2, t_2 = sampling_equidistant(x_1, t_1, Fs_2)
t_sinc = t_1
x_sinc = reconstruction_sinc(x_2, t_2, t_sinc)
plot_signal_reconstructed(t_1, x_1, t_2, x_2, t_sinc, x_sinc)
```

## Interference

In signal processing, **interference** occurs when a wave is superimposed with another wave of similar frequency. When a crest of one wave meets a crest of the other wave at some point, then the individual magnitudes add up for a certain period of time, which is known as **constructive interference**. Vice versa, when a crest of one wave meets a trough of the other wave, then the magnitudes cancel out for a certain period of time, which is known as **destructive interference**. We illustrate these phenomena in the following code cell.

```python
def plot_interference(t, x1, x2, figsize=(8, 2), xlim=None, ylim=None, title=''):
    """Plotting two signals and its superposition
    Notebook: PCP_signal.ipynb
    """
    plt.figure(figsize=(8, 2))
    plt.plot(t, x1, color='gray', linewidth=1.0, linestyle='-', label='x1')
    plt.plot(t, x2, color='cyan', linewidth=1.0, linestyle='-', label='x2')
    plt.plot(t, x1+x2, color='red', linewidth=2.0, linestyle='-', label='x1+x2')
    if xlim is None:
        plt.xlim([0, t[-1]])
    else:
        plt.xlim(xlim)
    if ylim is not None:
        plt.ylim(ylim)
    plt.xlabel('Time (seconds)')
    plt.ylabel('Amplitude')
    plt.title(title)
    plt.legend(loc='upper right')
    plt.tight_layout()
    plt.show()

dur = 5
x1, t = generate_sinusoid(dur=dur, Fs=100, amp=1, freq=1.05, phase=0.0)
x2, t = generate_sinusoid(dur=dur, Fs=100, amp=1, freq=0.95, phase=0.8)
plot_interference(t, x1, x2, xlim=[0, dur], ylim=[-2.2,2.2], title='Constructive Interference')

dur = 5
x1, t = generate_sinusoid(dur=dur, Fs=100, amp=1, freq=1.05, phase=0.0)
x2, t = generate_sinusoid(dur=dur, Fs=100, amp=1, freq=1.00, phase=0.4)
plot_interference(t, x1, x2, xlim=[0, dur], ylim=[-2.2,2.2], title='Destructive Interference')
```

## Exercises and Results

```python
import LibPCP.signal
show_result = True
```

Exercise 1: Beating

Let $f_1(t)=\sin(2\pi \omega_1 t)$ and $f_2(t)=\sin(2\pi \omega_2 t)$ be two sinusoids with distinct but nearby frequencies $\omega_1\approx\omega_2$. The superposition $f_1+f_2$ of these two sinusoids results in a function that looks like a single sine wave with a slowly varying amplitude, a phenomenon also known as beating. Mathematically, this phenomenon results from a trigonometric identity yielding

$$
\sin(2\pi \omega_1 t)+\sin(2\pi \omega_2 t)=
2\cos\left(2\pi\frac{\omega_1-\omega_2}{2}t\right)\sin\left(2\pi\frac{\omega_1+\omega_2}{2}t\right).
$$

If the difference $\omega_1-\omega_2$ is small, the cosine term has a low frequency compared to the sine term. As a result the signal $f_1+f_2$ can be seen as a sine wave of frequency $(\omega_1+\omega_2)/2$ with a slowly varying amplitude envelope of frequency $|\omega_1-\omega_2|$. Note that this rate is twice the frequency $(\omega_1-\omega_2)/2$ of the cosine term.

Use the functions generate_sinusoid and plot_interference to  illustrate the beating effect for various parameters $\omega_1$ and $\omega_2$.

In particular, consider $\omega_1=200$ and $\omega_2=203$ to generate a superimposed signal x at a sampling rate of Fs=4000. Listen to x using ipd.display(ipd.Audio(x, rate=Fs)).

```python
#<solution>
# Your Solution
#</solution>
```

```python
LibPCP.signal.exercise_beating(show_result=show_result)
```

Exercise 2: Aliasing with Sinsuoids

First, using x, t = generate_sinusoid(dur=2, Fs=128, freq=10), generate a sinusoid of frequency $\omega=10~\mathrm{Hz}$ at Fs=128 to mimic a CT-signal. Then, use the function sampling_equidistant to successively decrease the sampling rate using [64, 32, 20, 16, 12, 8, 4]. Visualize the result via plot_signal_reconstructed and discuss your observations.

```python
#<solution>
# Your Solution
#</solution>
```

```python
LibPCP.signal.exercise_aliasing_sinus(show_result=show_result)
```

Exercise 3: Visual Aliasing

Aliasing effects occur also in other areas such as the visual domain. One famous example is the wagon-wheel effect, where a wheel appears to rotate at a slower speed or even backwards when the speed is actually increasing. Identify some YouTube videos that illustrate this effect. Furthermore, integrate the video into your notebooks. To this end, do the following:

 Import the API for display tools in IPython: import IPython.display as ipd.

 Use the following commands: ipd.display, ipd.YouTubeVideo

```python
#<solution>
# Your Solution
#</solution>
```

```python
LibPCP.signal.exercise_aliasing_visual(show_result=show_result)
```


# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [2.66453526e-17 7.43849426e-17 3.98986399e-17]
      stds:   [0.99999963 0.99999971 0.99999973]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999963 1.99999942 2.9999992 ]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029241291468676e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  0.0
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 0.93x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 3.92e-10
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 4.78e-11
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 7.57e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 5.55e-09
    b2 relative error: 5.55e-09
    b3 relative error: 2.23e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 5.69e-09
    gamma1 relative error: 8.80e-09
    gamma2 relative error: 4.14e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.101000; val_acc: 0.112000
    (Epoch 1 / 10) train acc: 0.244000; val_acc: 0.212000
    (Iteration 21 / 200) loss: 1.916710
    (Epoch 2 / 10) train acc: 0.347000; val_acc: 0.280000
    (Iteration 41 / 200) loss: 2.155512
    (Epoch 3 / 10) train acc: 0.406000; val_acc: 0.304000
    (Iteration 61 / 200) loss: 1.910217
    (Epoch 4 / 10) train acc: 0.410000; val_acc: 0.296000
    (Iteration 81 / 200) loss: 1.539788
    (Epoch 5 / 10) train acc: 0.476000; val_acc: 0.299000
    (Iteration 101 / 200) loss: 1.435907
    (Epoch 6 / 10) train acc: 0.519000; val_acc: 0.320000
    (Iteration 121 / 200) loss: 1.379069
    (Epoch 7 / 10) train acc: 0.581000; val_acc: 0.312000
    (Iteration 141 / 200) loss: 1.598973
    (Epoch 8 / 10) train acc: 0.641000; val_acc: 0.329000
    (Iteration 161 / 200) loss: 0.842891
    (Epoch 9 / 10) train acc: 0.665000; val_acc: 0.319000
    (Iteration 181 / 200) loss: 1.106657
    (Epoch 10 / 10) train acc: 0.709000; val_acc: 0.320000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.155000; val_acc: 0.140000
    (Epoch 1 / 10) train acc: 0.182000; val_acc: 0.160000
    (Iteration 21 / 200) loss: 2.143169
    (Epoch 2 / 10) train acc: 0.213000; val_acc: 0.179000
    (Iteration 41 / 200) loss: 1.919161
    (Epoch 3 / 10) train acc: 0.225000; val_acc: 0.241000
    (Iteration 61 / 200) loss: 1.841960
    (Epoch 4 / 10) train acc: 0.263000; val_acc: 0.238000
    (Iteration 81 / 200) loss: 1.804382
    (Epoch 5 / 10) train acc: 0.293000; val_acc: 0.247000
    (Iteration 101 / 200) loss: 1.929585
    (Epoch 6 / 10) train acc: 0.345000; val_acc: 0.251000
    (Iteration 121 / 200) loss: 1.706305
    (Epoch 7 / 10) train acc: 0.342000; val_acc: 0.246000
    (Iteration 141 / 200) loss: 1.762569
    (Epoch 8 / 10) train acc: 0.356000; val_acc: 0.242000
    (Iteration 161 / 200) loss: 1.648601
    (Epoch 9 / 10) train acc: 0.384000; val_acc: 0.273000
    (Iteration 181 / 200) loss: 1.536583
    (Epoch 10 / 10) train acc: 0.405000; val_acc: 0.267000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers_ws[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
[FILL THIS IN]

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
[FILL THIS IN]

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_backward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [-0.58416774  0.03223092 -0.29106935  0.84300618]
      stds:   [0.42903404 1.0673565  1.17954475 0.38429471]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [3.24749679 5.09669275 4.12679194 7.52901853]
      stds:   [1.28710213 3.2020695  3.53863425 1.15288413]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029241291468676e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
[FILL THIS IN]


# Learning disentangled representations in a 'sphere-world' with continuous actions

## 1. Preamble

```python
import os

import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim
import numpy as np
import matplotlib.pyplot as plt
import random

import time
from IPython import display

try:
    import seaborn as sns
    sns.set()
    print("Set seaborn plotting defaults.")
except:
    print("seaborn not found.")
```

```python
def mk_dir(export_dir, quite=False):
    if not os.path.exists(export_dir):
            try:
                os.makedirs(export_dir)
                print('created dir: ', export_dir)
            except OSError as exc: # Guard against race condition
                 if exc.errno != exc.errno.EEXIST:
                    raise
            except Exception:
                pass
    else:
        print('dir already exists: ', export_dir)

mk_dir("_data/")
```

## 2. Defining objects

### SphereWorld environment

We use a cubic 3D world in which a ball evolves on a sphere. Observations are a 3-dimensional tensor with values 0 except at the voxel occupied by the ball which has value 1. When the ball is not at the center of the voxel (which is most of the time) its density is distributed around neighbouring voxels. Actions are continuous rotations around the three axes.

```python
class SphereWorld():

    class action_space():
        def __init__(self,n_actions):
            self.n = n_actions

        def sample(self, k=1):
            return torch.randint(0,self.n,(k,))

    class observation_space():
        def __init__(self,n_features):
            self.shape = [n_features]

    def __init__(self,dim=5,radius=3.5):

        self.size = dim

        self.radius = radius
        self.action_space = self.action_space(6)
        self.observation_space = self.observation_space((2*dim)**3)

        self.reset()

    def reset(self):

        self.theta = random.uniform(0, np.pi)
        self.phi = random.uniform(0, np.pi*2)

        self.ball_coordinates = [self.radius*np.sin(self.theta)*np.cos(self.phi), \
                                 self.radius*np.sin(self.theta)*np.sin(self.phi), \
                                 self.radius*np.cos(self.theta)]   #Start from an outside position

        self.state = self.get_state()
        return self.get_observation()

    def get_state(self, ball_coordinates=None):
        if ball_coordinates is None:
            ball_coordinates=self.ball_coordinates

        state = torch.zeros((2*self.size, 2*self.size, 2*self.size))

        ball_low_x = ball_coordinates[0] + self.size - .5
        ball_low_y = ball_coordinates[1] + self.size - .5
        ball_low_z = ball_coordinates[2] + self.size - .5

        fill_x = (int(ball_low_x + 1) - ball_low_x)
        fill_y = (int(ball_low_y + 1) - ball_low_y)
        fill_z = (int(ball_low_z + 1) - ball_low_z)

        state[int(ball_low_x), int(ball_low_y), int(ball_low_z)] = fill_x * fill_y * fill_z
        state[(int(ball_low_x) + 1), int(ball_low_y), int(ball_low_z)] = (1 - fill_x) * fill_y * fill_z
        state[int(ball_low_x), (int(ball_low_y) + 1), int(ball_low_z)] = fill_x * (1 - fill_y) * fill_z
        state[(int(ball_low_x) + 1), (int(ball_low_y) +1), int(ball_low_z)] = (1 - fill_x) * (1 - fill_y) * fill_z
        state[int(ball_low_x), int(ball_low_y), int(ball_low_z) + 1] = fill_x * fill_y * (1 - fill_z)
        state[(int(ball_low_x) + 1), int(ball_low_y), int(ball_low_z) + 1] = (1 - fill_x) * fill_y * (1 - fill_z)
        state[int(ball_low_x), (int(ball_low_y) + 1), int(ball_low_z) + 1] = fill_x * (1 - fill_y) * (1 - fill_z)
        state[(int(ball_low_x) + 1), (int(ball_low_y) +1), int(ball_low_z) + 1] = (1 - fill_x) * (1 - fill_y) * (1 - fill_z)

        self.state = state
        return state

    def get_observation(self):
        return self.state.flatten()

    def step(self, action, angle):

        ball_coordinates = self.ball_coordinates

        if action == 0:
            new_ball_coordinates = [ball_coordinates[0]*np.cos(angle) + ball_coordinates[1]*np.sin(angle), \
                                    - ball_coordinates[0]*np.sin(angle) + ball_coordinates[1]*np.cos(angle), \
                                    ball_coordinates[2]]
        elif action == 1:
            new_ball_coordinates = [ball_coordinates[0]*np.cos(angle) + ball_coordinates[2]*np.sin(angle), \
                                    ball_coordinates[1], \
                                    - ball_coordinates[0]*np.sin(angle) + ball_coordinates[2]*np.cos(angle)]
        elif action == 2:
            new_ball_coordinates = [ball_coordinates[0], \
                                    ball_coordinates[1]*np.cos(angle) + ball_coordinates[2]*np.sin(angle), \
                                    - ball_coordinates[1]*np.sin(angle) + ball_coordinates[2]*np.cos(angle)]
        else:
            raise Exception("INVALID ACTION")

        self.ball_coordinates = new_ball_coordinates
        self.state = self.get_state()
        return self.get_observation()
```

**Show consecutive states from this environment**

```python
def plot_state(state, ax):
    ax[0].pcolormesh(env.state.sum(axis=0), edgecolors='gray', linewidth=2, )
    ax[0].set_aspect('equal')
    ax[0].set_xticks([])
    ax[0].set_yticks([])

    ax[1].pcolormesh(env.state.sum(axis=1), edgecolors='gray', linewidth=2, )
    ax[1].set_aspect('equal')
    ax[1].set_xticks([])
    ax[1].set_yticks([])

    ax[2].pcolormesh(env.state.sum(axis=2), edgecolors='gray', linewidth=2, )
    ax[2].set_aspect('equal')
    ax[2].set_xticks([])
    ax[2].set_yticks([])

    return ax

n_steps = 8

env = SphereWorld()

fig, axs = plt.subplots(3, n_steps+1,figsize=(15, 5))

plot_state(env.state,axs[:,0])

for i in range(n_steps):
    action = 2
    env.step(action, np.pi/4)
    plot_state(env.state,axs[:,i+1])
```

### Latent space

**Encoder/Decoder**

Now we want to learn to represent this environment in some latent space (which we, for now, simply assume to be 3-dimensional).  We will require both an encoder and decoder, which will be simple MLPs

```python
class Coder(nn.Module):

    def __init__(self, n_in, n_out, n_hid=[16], output_activation=nn.Sigmoid):
        super().__init__()

        if type(n_hid) != list:
            n_hid = [n_hid]
        n_layers = [n_in] + n_hid + [n_out]

        self.layers = []
        for i_layer, (n1, n2) in enumerate(zip(n_layers, n_layers[1:])):
            mods = [nn.Linear(n1, n2, bias=True)]
            act_fn = nn.ReLU if i_layer < len(n_layers) - 2 else output_activation
            if act_fn is not None:
                mods.append(act_fn())
            layer = nn.Sequential(*mods)
            self.layers.append(layer)

        self.layers = nn.ModuleList(self.layers)

    def forward(self, x):
        for layer in self.layers:
            x = layer(x)
        return x

class nnNorm(nn.Module):

    def __init__(self, dim=-1):
        super().__init__()
        self.dim=dim

    def forward(self, x):
        return F.normalize(x, dim=self.dim)

class Decoder(Coder):

    def __init__(self, n_in, n_out, n_hid=[32]):
        super().__init__(n_in, n_out, n_hid, output_activation=nn.Sigmoid)

class Encoder(Coder):

    def __init__(self, n_in, n_out, n_hid=[32]):
        super().__init__(n_in, n_out, n_hid, output_activation=nnNorm)

class Angles(Coder):

    def __init__(self, n_in, n_out, n_hid=[32]):
        super().__init__(n_in, n_out, n_hid, output_activation=None)

```

```python
encoder = Encoder(n_in=5*5*5, n_out=6, n_hid=[64])
print(encoder)

decoder = Decoder(n_in=6, n_out=5*5*5, n_hid=[64])
print(decoder)

def count_parameters(model):
    return sum(p.numel() for p in model.parameters() if p.requires_grad)

count_parameters(encoder)
```

**Check tensor dimensions**

```python
x = torch.rand(125)
z = encoder(x)
x_recon = decoder(z)
print("Observable rep (shape = {}) : \n {}".format(x.shape, x))
print("Latent rep (shape = {}, norm = {}) : \n {}".format(z.shape, torch.norm(z), z))
print("Reconstructed Observable rep (shape = {}) : \n {}".format(x_recon.shape, x_recon))
```

**Representation**

The crux of the matter is learning to 'represent' actions in the observation space with actions in latent space.  Here, we will do this by assuming every action is a generalized rotation in latent space, which we denote with a series of 2-dimensional rotations.

A 2-d rotation is given by:

\begin{pmatrix}
\cos(\theta) & \sin(\theta) \\
-\sin(\theta) & \cos(\theta)
\end{pmatrix}

and we denote a rotation in dimensions $i$ and $j$ of a higher dimensional space as $R_{i,j}(\theta)$.  For $i=1$, $j=4$, in a 4-dimensional space:

\begin{equation}
R_{1,4}(\theta) =
\begin{pmatrix}
\cos(\theta) & 0 & 0 & \sin(\theta) \\
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
-\sin(\theta) & 0 & 0 & \cos(\theta)
\end{pmatrix}
\end{equation}

An arbitrary rotation, denoted $g$ as I am subtly moving towards this being a group action, can then be written as:

\begin{equation}
    g(\theta_{1,2},\theta_{1,3},\dots,\theta_{n-1,n}) = \prod_{i=1}^{n-1} \prod_{j=1+1}^{n} R_{i,j}(\theta_{i,j})
\end{equation}

which has $n(n-1)/2$ free parameters (i.e. $\theta_{i,j}$'s).

To learn continuous Lie groups instead of discrete rotations, each of these $\theta_{i,j}$ is now parameterized by a neural network.

```python
class Representation():

    def __init__(self, dim=3):
        self.dim = dim
        self.params = dim*(dim-1)//2
        self.thetas = torch.autograd.Variable(2*torch.rand(self.params)-1, requires_grad=True)

        self.__matrix = None

    def set_thetas(self, thetas):
        self.thetas = thetas
        self.clear_matrix()

    def clear_matrix(self):
        self.__matrix = None

    def get_matrix(self):
        if self.__matrix is None:
            k = 0
            mats = []
            for i in range(self.dim-1):
                for j in range(self.dim-1-i):
                    theta_ij = self.thetas[k]
                    k+=1
                    c, s = torch.cos(theta_ij), torch.sin(theta_ij)

                    rotation_i = torch.eye(self.dim, self.dim)
                    rotation_i[i, i] = c
                    rotation_i[i, i+j+1] = s
                    rotation_i[j+i+1, i] = -s
                    rotation_i[j+i+1, j+i+1] = c

                    mats.append(rotation_i)

            def chain_mult(l):
                if len(l)>=3:
                    return l[0]@l[1]@chain_mult(l[2:])
                elif len(l)==2:
                    return l[0]@l[1]
                else:
                    return l[0]

            self.__matrix = chain_mult(mats)

        return self.__matrix
```

**LatentWorld**

Now, for symmetry's sake, we'll also have a `LatentWorld` which acts as the environment in the latent space.

```python
class LatentWorld():

    class action_space():
        def __init__(self,n_actions):
            self.n = n_actions

        def sample(self, k=1):
            return torch.randint(0,self.n,(k,))

    class observation_space():
        def __init__(self,n_features):
            self.shape = [n_features]

    def __init__(self,
                 dim=3,
                 n_actions=3,
                 n_hid=[64]):

        self.dim = dim
        self.angle_computer = Angles(n_in = 2, n_out = dim*(dim-1)//2, n_hid=n_hid)
        self.action_space = self.action_space(n_actions)
        self.observation_space = self.observation_space(dim)

        self.representation = Representation()

    def reset(self, state):
        self.state = state
        return self.get_observation()

    def clear_representations(self):
        self.representation.clear_matrix()

    def get_representation_params(self):
        params = []
        for rep in self.action_reps:
            params.append(rep.thetas)
        return params

    def save_representations(self, path):
        if os.path.splitext(path)[-1] != '.pth':
            path += '.pth'
        return torch.save(self.angle_computer.state_dict(), path)

    def load_reprentations(self, path):
        self.representation.load_state_dict(torch.load(path))

    def get_observation(self):
        return self.state

    def step(self,action_value):
        self.representation.set_thetas(self.angle_computer(action_value))
        self.state = torch.mv(self.representation.get_matrix(), self.state)
        obs = self.get_observation()
        return obs
```

## 5. Disentanglement

***Some jargon***

We focus on learning a *disentangled* representation of the actions.

Before considering how best to do this, we want to define a metric of 'disentanglement'.  We consider the evolution of an observable (latent) vector, $x \in X$ ($z \in Z$), under the element $g \in G$ of the group of symmetries generating transformations of the object.  Then we are looking for a representation, $\rho:G \rightarrow GL(V)$, such that the transformation is linear in the latent space, i.e.
\begin{equation}
    z^{\prime} = \rho(g) \cdot z.
\end{equation}
Note, in our case, the representations are the rotation matrices we learn.

For this representation to be disentangled, it means that if there exists a subgroup decomposition of $G$
\begin{equation}
    G = G_1 \times G_2 \times \dots \times G_n,
\end{equation}
then we equivalently decompose the representation, $(\rho, G)$, into subrepresentations:
\begin{equation}
    V = V_1 \oplus V_2 \oplus \dots \oplus V_n
\end{equation}
such that the restricted subrepresentations $(\rho_{\vert G_i}, V_i)_i$ are non-trivial, and the restricted subrepresentations $(\rho_{\vert G_i}, V_j)_{j \neq i}$ are trivial.

In our context, a SphereWorld can be represented as a subgroup of $\mathrm{SO}(3)$, therefore we hope to find the disentangled representation of the three continuous rotations.

***Some practicalities***

Our intuition is that the disentangled representation acts as the identity on as many dimensions as possible.  We could attempt to enforce this with some regularization during training.  Normal weight decay won't cut it, as that tries to reduce all weights, where as what we really want to do is have all *but one* of our thetas (which corresponds to the rotation/coupling of two dimensions) to be zero.

**1. Entanglement regularisation**

So for $m$ parameters, ${\theta_1, \dots, \theta_m}$, we want to regularise with
\begin{equation}
    \sum_{i \neq j} \vert\theta_i\vert^2, \mathrm{where\ } \theta_j {=} \mathrm{max_k}({\vert\theta_k\vert}).
\end{equation}
We will also use this term as our metric of 'entanglement'.

```python
def calc_entanglement(params):
    params = params.abs().pow(2)
    return params.sum() - params.max()

params = torch.FloatTensor([1,1,0.5,0,0])
calc_entanglement(params)
```

### Training

```python
obs_env = SphereWorld(dim = 5, radius = 4.)
lat_env = LatentWorld(dim = 4,
                      n_actions = obs_env.action_space.n
                     )
decoder = Decoder(n_in = sum(lat_env.observation_space.shape),
                  n_out = sum(obs_env.observation_space.shape),
                  n_hid = [64])

encoder = Encoder(n_in = sum(obs_env.observation_space.shape),
                  n_out = sum(lat_env.observation_space.shape),
                  n_hid = [64])

optimizer_dec = optim.Adam(decoder.parameters(),
                           lr=1e-2,
#                            betas=(0.9, 0.99),
                           weight_decay=0)

optimizer_enc = optim.Adam(encoder.parameters(),
                           lr=1e-2,
#                            betas=(0.9, 0.99),
                           weight_decay=0)

optimizer_rep = optim.Adam(lat_env.angle_computer.parameters(),
                           lr=1e-2,
#                            betas=(0.9, 0.99),
                           weight_decay=0)

losses = []

```

```python
n_sgd_steps = 10000
ep_steps = 5
batch_eps = 16

#Starting by learning small angles before moving on the larger angles helps
angle_max = np.pi * 2 / 5

i = 1

t_start = time.time()

temp = 0

while i < n_sgd_steps:
    if i == 4000:
        angle_max = np.pi * 4 / 5
    if i == 7000:
        angle_max = np.pi

    loss = torch.zeros(1)

    for _ in range(batch_eps):
        t_ep = -1
        while t_ep < ep_steps:
            angle = random.uniform(-angle_max, angle_max)
            if t_ep == -1:
                obs_x = obs_env.reset()
                obs_z = lat_env.reset(encoder(obs_x))
            else:
                action = random.randrange(3)
                obs_x = obs_env.step(action, angle)
                obs_z = lat_env.step(torch.tensor([action, angle]))

            t_ep += 1

            obs_x_recon = decoder(obs_z)
            (obs_x_recon==obs_x_recon.max()).float()

            loss += F.binary_cross_entropy(obs_x_recon, obs_x)
            loss += calc_entanglement(lat_env.angle_computer(torch.tensor([action, angle])))*1e-3

    loss /= (batch_eps * ep_steps)
    loss_raw = loss

    losses.append(loss_raw.item())

    optimizer_dec.zero_grad()
    optimizer_rep.zero_grad()
    optimizer_enc.zero_grad()
    loss.backward()
    optimizer_dec.step()
    optimizer_rep.step()
    optimizer_enc.step()

    # Rember to clear the cached action representations after we update the parameters!
    lat_env.clear_representations()

    i+=1

    if i%10==0:
        print("iter {} : loss={:.5f} : last 10 iters in {:.3f}s".format(
            i, loss.item(), time.time() - t_start
            ), end="\r" if i%100 else "\n")
        t_start = time.time()
```

```python
exp_name = "ent_reg_1e-2_orthNorm_reg_1e-2_20steps_16batch"
# exp_name = "testing"

save_folder = os.path.join("_data/decoder_only", exp_name)
train_folder = os.path.join(save_folder,"train")
test_folder = os.path.join(save_folder,"test")

mk_dir(save_folder)
mk_dir(train_folder)
mk_dir(test_folder)
```

**Plot training curve**

```python
with plt.style.context('seaborn-paper', after_reset=True):

    fig, (ax1) = plt.subplots(1, 1, figsize=(8, 8), gridspec_kw={"wspace":0.3})

    window = 25
    avg_mask = np.ones(window) / window

    ax1.plot(np.convolve(range(len(losses)), avg_mask, 'valid'),
             np.convolve(np.log(losses), avg_mask, 'valid'),
             linewidth=0.75,
             alpha=0.8)
#     ax1.set_yscale("log")

    ax1.set_xlabel("Num. parameter updates")
    ax1.set_ylabel("Log reconstruction loss")

    fig_fname = os.path.join(train_folder, "training_curves")
    plt.savefig(fig_fname + ".pdf", bbox_inches='tight')
    plt.savefig(fig_fname + ".png", bbox_inches='tight')
```

```python
torch.save(decoder.state_dict(), os.path.join(save_folder,"decoder.pth"))
torch.save(optimizer_dec.state_dict(), os.path.join(train_folder,"optimizer_dec.pth"))
torch.save(optimizer_rep.state_dict(), os.path.join(train_folder,"optimizer_rep.pth"))
```

### Testing

**1) Representations learned for specific values of the rotations**

```python
width=0.75

plt_lim = 0.22
titles = ["-0.2", "-0.1", "0", "0.1", "0.2"]

with plt.style.context('seaborn-paper', after_reset=True):

    fig, axs = plt.subplots(3, 5, figsize=(15, 5), gridspec_kw={"wspace":0.4})

    for i in range(15):
        thetas = lat_env.angle_computer(torch.tensor([i//5, (-0.2+(i%5)/10.)*2*np.pi])).detach()
        print(thetas)
        x = np.arange(len(thetas))
        axs[i//5,i%5].bar(x - width/2, thetas/(2*np.pi), width, label='Rep {}'.format(i))

        axs[i//5,i%5].set_yticks([-0.2, 0., 0.2])
        axs[i//5,i%5].set_xticks(x-0.25)
        axs[i//5,i%5].set_xticklabels(["12","13","23"])
        axs[i//5,i%5].set_xlabel("$ij$")

        axs[i//5,i%5].set_ylim(-plt_lim,plt_lim)

        axs[i//5,i%5].set_title(titles[i%5])

    axs[0,0].set_ylabel(r"$x : \theta / 2\pi$" )
    axs[1,0].set_ylabel(r"$y : \theta / 2\pi$" )
    axs[2,0].set_ylabel(r"$y : \theta / 2\pi$" )

    fig_fname = os.path.join(test_folder, "thetas")

    plt.savefig(fig_fname + ".pdf", bbox_inches='tight')
    plt.savefig(fig_fname + ".png", bbox_inches='tight')
```

**2) Learned representations as a function of rotation angle**

```python
fig, axs = plt.subplots(1, 3, figsize=(17, 5))

x = np.linspace(-np.pi, np.pi, 500)

titles = ["rotation around $x$", "rotation around $y$", "rotation around $z$"]

for i in range(3):
    y0 = []
    y1 = []
    y2 = []
    for k in range(500):
        y = lat_env.angle_computer(torch.tensor([i, x[k]])).detach().numpy()
        y0 += [y[0]]
        y1 += [y[1]]
        y2 += [y[2]]
    a = axs[i].plot(x,y0,c="r",linewidth=2)[0]
    b = axs[i].plot(x,y1,c="b",linewidth=2)[0]
    c = axs[i].plot(x,y2,c="g",linewidth=2)[0]
    axs[i].set_xlabel("rotation angle", fontsize=24)
    axs[i].set_title(titles[i], fontsize=24)
    axs[i].tick_params(labelsize=18)
axs[0].set_ylabel(r"$\theta_{ij}$", fontsize=24)
fig.subplots_adjust(right=0.83)
fig.legend((a,b,c),(r'ij=12',r'ij=13',r'ij=23'), loc='center right',fontsize=24)
fig.show()
plt.savefig("lie_group.png", bbox_inches='tight')
```

**3) See predictions made by network**

```python
def plot_state(obs, ax, one_hot=True):
    if one_hot:
        obs=(obs == obs.max())
    ax.pcolormesh(obs.reshape(2*obs_env.size, 2*obs_env.size, 2*obs_env.size).sum(axis=2), edgecolors='gray', linewidth=2)
    ax.set_aspect('equal')
    ax.set_xticks([])
    ax.set_yticks([])

    return ax

n_steps = 10

fig, (ax1,ax2) = plt.subplots(2, 1)

ax1.set_title("Ground truth")
ax2.set_title("Reconstruction")

for i in range(n_steps+1):

    if i==0:
        action = "N\A"
        obs_x = obs_env.reset()
        obs_z = lat_env.reset(encoder(obs_x))
    else:
        action = random.randrange(3)
        angle = random.uniform(-angle_max, angle_max)
        obs_x = obs_env.step(action, angle)
        obs_z = lat_env.step(torch.tensor([action, angle]))

    obs_x_recon = decoder(obs_z)

    fig.suptitle('step {} : last action = {} in the x-y plane'.format(i, action), fontsize=16)

    plot_state(obs_x.detach().numpy(),ax1)
    plot_state(obs_x_recon.detach().numpy(),ax2, one_hot=False)

    fig_fname = os.path.join(test_folder, "step_{}".format(i))

    plt.savefig(fig_fname + ".pdf", bbox_inches='tight')
    plt.savefig(fig_fname + ".png", bbox_inches='tight')

    display.clear_output(wait=True)
    display.display(plt.gcf())
    time.sleep(1.)

display.clear_output(wait=False)
```

```python

```


# Práctica Background Subtraction

En esta segunda práctica del bloque de análisis de secuencias haremos trabajo con 3 técnicas de background subtraction.

En primer lugar debéis implementar vuestro propio método de eliminación de fondo basado en la idea que explicamos del filtro exponencial. Podéis hacer las adaptaciones que encontréis necesario a la idea general del método para mejorar su rendimiento.

En segundo lugar debéis de usar las implementaciones del MOG y del MOG2 que se encuentran en la librería
 [OpenCV](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.html).

En tercer lugar lo que quiero es que hagáis una comparativa entre su método y los dos basados en Mixture de gausiano utilizando el dataset de la competición [changedetection.net](http://changedetection.net/) del año 2012.

En concreto debe usar una secuencia de cada categoría exceptuando la de *Thermal*.

Las métricas que a usar para cada secuencia son:

* **TP** : True Positive
* **FP** : False Positive
* **FN** : False Negative
* **TN** : True Negative
* **Re (Recall)** : TP / (TP + FN)
* **F-Measure** : (2 * Precision * Recall) / (Precision + Recall)
* **Precision** : TP / (TP + FP)

Finalmente se pide que hagáis un análisis de los resultados obtenidos por cada secuencia: problemas, carencias, puntos fuertes de los algoritmos ...

**Entrega:**

* Explicación del algoritmo implementado y código fuente.
* Resultados de las secuencias en formato imagen o video.
* Resultados de las secuencias en formato tabla resumen de las métricas.
* Análisis de los resultados

Secuencias a usar:
- baseline: todo
- cameraJitter: badminton, traffic
- dynamicBackground: canoe, fall, fountain02
- intermittentObjectMotion: sofa,
- shadow: busStation, copyMachine, cubicle, peopleInShade

### Métricas

```python
def recall(tp, fn):
    return (tp/(tp+fn))

def precision(tp, fp):
    return (tp/(tp+fp))

def f_measure(precision, recall):
    return ((2*precision*recall)/(precision+recall))

def all_metrics (resultPath, groundtruthPath, start_frame, end_frame):
    # Cargamos el resultado del filtro indicado
    result = loadImages(resultPath)
    groundTruth = loadImages(groundtruthPath)

    # ground truth empieza en start_frame, los frames están numerados de 1 hasta n, pero el array de frames va de 0 a n-1
    total_frames = end_frame-start_frame

    # Inicializamos un array para cada métrica donde se el resultado para cada frame
    tp =  np.zeros(total_frames)
    fp =  np.zeros(total_frames)
    fn =  np.zeros(total_frames)
    tn =  np.zeros(total_frames)

    # Calculamos las métricas True Positive, False Positive, False Negative, True Negative para cada frame
    for i in range(total_frames):
        tp[i], fp[i], fn[i], tn[i] = comparator(result[i+start_frame], groundTruth[i+start_frame])

    # Calculamos las métricas Recall, Precision y F-measure para cada frame
    recall_frame = recall(tp=tp, fn=fn)
    precision_frame = precision(tp=tp, fp=fp)
    fMeasure_frame = f_measure(precision=precision_frame, recall=recall_frame)

    # Calculamos las métricas anteriores para toda la secuencia en conjunto
    tp_secuencia = tp.sum()
    fp_secuencia = fp.sum()
    fn_secuencia = fn.sum()
    tn_secuencia = tn.sum()

    recall_secuencia = recall(tp=tp_secuencia, fn=fn_secuencia)
    precision_secuencia = precision(tp=tp_secuencia, fp=fp_secuencia)
    fMeasure_secuencia = f_measure(precision=precision_secuencia, recall=recall_secuencia)

    cm_frames = np.array([tp,fp,fn,tn])
    cm_secuencia = np.array ([tp_secuencia, fp_secuencia, fn_secuencia, tn_secuencia])

    r_p_fm_frames = np.array([recall_frame, precision_frame, fMeasure_frame])
    r_p_fm_secuencia = np.array([recall_secuencia, precision_secuencia, fMeasure_secuencia])

    return cm_frames, cm_secuencia, r_p_fm_frames, r_p_fm_secuencia
```

### Filtro exponencial para _background subtraction_
Parte de la idea de eliminación de fondo más básica: restar el fotograman **_fondo_** (_background_) del fotograma actual para obtener el **_primer plano_** (_foreground_).
El primer problema o tarea para realizar este algoritmo es conseguir ese fotograma **_fondo_**. Para ello se necesita una imagen que represente el fondo sin ningún otro objeto, pero esto no siempre es posible, por lo tanto hay que buscar otras soluciones. Una de ellas es estimar el fondo como la media o la mediana a partir de los primeros _n_ fotogramas.
El segundo problema que surge es que el fondo en los casos prácticos varía con el tiempo y por lo tanto es posible que ese fotograma _fondo_ que hemos estimado no nos sirve más tarde. Por lo tanto la mejora que se propone es estimar el fondo recursivamente respecto el fotograma anterior siguiendo la siguiente fórmula:
$$
\begin{align}
B_t=(1-{\alpha})B_{t-1}+{\alpha}I_t
\end{align}
$$
Donde $B_t$ es la estimación del fondo para el fotograma actual, $B_{t-1}$ es el fondo para el fotograma anterior, $I_t$ es el fotograma actual y $\alpha$ es el factor de aprendizaje que indica cuánto del fotograma actual se considerará fondo para el siguiente fotograma.

```python
from utils import *
import numpy as np
from tqdm.notebook import tqdm
import matplotlib.pyplot as plt
import pandas as pd
```

Para mantener la limpieza del _notebook_ se han definidos las funciones para realizar el análisis dentro del archivo _utils.py_.
Las funciones que se encuentran son las siguientes:
- **comparator**: compara un _frame_ resultante de un modelo de _background subtraction_ con el resultado esperado (_groundtruth_)
- **loadImages**: devuelve una lista con las imágenes dentro de la carpeta especificada
- **exponentialFilter**: implementación del algoritmo basado en filtro exponencial
- **MOG**: implementación del algoritmo MOG disponible [aquí](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.html)
- **MOG2**: implementación del algoritmo MOG2 disponible [aquí](https://opencv-python-tutroals.readthedocs.io/en/latest/py_tutorials/py_video/py_bg_subtraction/py_bg_subtraction.html)
- **im2vid**: genera video a partir de las imágenes ubicadas en una carpeta
- **showVideo**: muestra el video en una ventana flotante para visualizar mejor el resultado del modelo

Notar que, a pesar de tratarse de algoritmos que trabajan con video, el _dataset_ que se proporciona para realizar las pruebas son imágenes que representan los fotogramas de una secuencia. Por lo tanto, las funciones que se han realizado, se han pensado para trabajar con imágenes en vez de video.

Pasos a seguir para cada secuencia:
1. Cargar las imágenes en una variable (path de las imágenes)
2. Aplicar el filtro deseado EF, MOG y MOG2 (path donde se guardarán las imágenes aplicadas por el filtro)
3. Obtener las métricas con comparator (hacer bucle a lo largo de las imágenes resultado -> path del groundtruth)
4. Mostrar los resultados y videos
5. Comparar los resultados (hacer gráfica a lo largo del

```python
# dynamicBackground canoe dataset output paths
expFilter_canoe_path = 'DATA/dynamicBackground/results/canoe/expFilter/'
MOG_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG/'
MOG2_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG2'

# intermittentObject sofa dataset output paths
expFilter_sofa_path = 'DATA/intermittentObjectMotion/results/sofa/expFilter/'
MOG_sofa_path = 'DATA/intermittentObjectMotion/results/sofa/MOG'
MOG2_sofa_path = 'DATA/intermittentObjectMotion/results/sofa/MOG2'

# shadow copyMachine dataset output paths
expFilter_copyMachine_path = 'DATA/shadow/results/copyMachine/expFilter/'
MOG_copyMachine_path = 'DATA/shadow/results/copyMachine/MOG'
MOG2_copyMachine_path = 'DATA/shadow/results/copyMachine/MOG2'
```

# 1. Secuencia _highway_ del dataset _baseline_

Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.

```python
# baseline highway dataset groundtruth y input paths
gt_highway_path = 'DATA/baseline/baseline/highway/groundtruth/*.png'
in_highway_path = 'DATA/baseline/baseline/highway/input/*.jpg'

groundTruth = loadImages(gt_highway_path)
frames = loadImages(in_highway_path)
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2667.40it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:01<00:00, 1208.82it/s]

## 1.1. Algoritmo filtro exponencial

Aplicamos el algoritmo basado en el filtro exponencial que hemos implementado.

```python
# baseline highway dataset output path
expFilter_highway_path = 'DATA/baseline/results/highway/expFilter/'

exponentialFilter(frames, 0.3, expFilter_highway_path, 20, bgFilter ='MEDIAN')
```

    Algorithm application: 100%|█████████████████████████████████████████████████████| 1680/1680 [00:01<00:00, 1032.18it/s]

Convertimos la secuencia de imágenes en video para poder visualizar mejor el resultado.

```python
#im2vid(expFilter_highway_path+'*.png','testvid.mp4')
```

Mostramos el video en una ventana flotante.

```python
#showVideo('testvid.mp4')
```

### 1.1.1. Cálculo de métricas
Calculamos las métricas comparando con el _ground truth_ propocionado. Para cada secuencia se proporciona una ROI (Region Of Interest) sobre la cual se realiza el groundtruth, para simplificar el cálculo se han seleccionado aquellas secuencias en las que la ROI sea toda la imagen. Por otro lado, también se especifica el _frame_ de inicio y final sobre los que se ha de realizar la verificación del algoritmo, por lo tanto solo se calcularán las métricas dentro de ese rango.

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_EF = all_metrics (expFilter_highway_path+'*.png', gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1700/1700 [00:06<00:00, 262.06it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2594.43it/s]

## 1.2. Algoritmo MOG

Aplicamos el algoritmo MOG (Mixture Of Gaussians)

```python
# baseline highway dataset output path
MOG_highway_path = 'DATA/baseline/results/highway/MOG/'

MOG(frames,MOG_highway_path)
```

    Algorithm application2: 100%|█████████████████████████████████████████████████████| 1700/1700 [00:03<00:00, 496.46it/s]

```python
#im2vid(MOG_highway_path+'*.png','testvid.mp4')
```

```python
#showVideo('testvid.mp4')
```

### 1.2.1. Cálculo de métricas

```python
cm_frames_MOG,\
cm_secuencia_MOG,\
r_p_fm_frames_MOG,\
r_p_fm_secuencia_MOG = all_metrics (MOG_highway_path+'*.png', gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1700/1700 [00:06<00:00, 275.06it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2675.90it/s]

## 1.3. Algoritmo MOG2

```python
# baseline highway dataset output path
MOG2_highway_path = 'DATA/baseline/results/highway/MOG2/'

MOG2(frames,MOG2_highway_path)
```

    Algorithm application: 100%|██████████████████████████████████████████████████████| 1700/1700 [00:03<00:00, 448.95it/s]

### 1.3.1. Cálculo de métricas

```python
cm_frames_MOG2,\
cm_secuencia_MOG2,\
r_p_fm_frames_MOG2,\
r_p_fm_secuencia_MOG2 = all_metrics (MOG2_highway_path+'*.png', gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1700/1700 [00:06<00:00, 276.62it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2637.81it/s]

## Comparación resultados

Obtenemos también las métricas para el _ground truth_ para poder comparar los valores TP, FP, FN y TN

```python
cm_frames_gt,\
cm_secuencia_gt,\
r_p_fm_frames_gt,\
r_p_fm_secuencia_gt = all_metrics (gt_highway_path, gt_highway_path, start_frame=469, end_frame=1699)
```

    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2642.47it/s]
    Loading images: 100%|████████████████████████████████████████████████████████████| 1700/1700 [00:00<00:00, 2634.41it/s]

```python
r_p_fm_frames_EF[0].shape
```

    (1230,)

```python
fig, ax = plt.subplots(3, sharex=True, sharey=True, figsize=(12,8))
ax[0].plot(r_p_fm_frames_EF[0], label="Exponential Filter")
ax[0].plot(r_p_fm_frames_MOG[0],label="MOG")
ax[0].plot(r_p_fm_frames_MOG2[0],label="MOG2")
ax[0].set_title("Recall")

ax[1].plot(r_p_fm_frames_EF[1],label="Exponential Filter")
ax[1].plot(r_p_fm_frames_MOG[1],label="MOG")
ax[1].plot(r_p_fm_frames_MOG2[1],label="MOG2")
ax[1].set_title("Precision")

ax[2].plot(r_p_fm_frames_EF[2],label="Exponential Filter")
ax[2].plot(r_p_fm_frames_MOG[2],label="MOG")
ax[2].plot(r_p_fm_frames_MOG2[2],label="MOG2")
ax[2].set_title("f-measure")

plt.ylim([0,1])
legend = ax[0].legend(loc='lower right')
legend = ax[1].legend(loc='lower right')
legend = ax[2].legend(loc='lower right')
fig.tight_layout()
plt.show()
```

Guardamos en un _dataframe_ los resultados obtenidos para esta secuencia

```python
df_highway = pd.DataFrame(np.array([np.concatenate((r_p_fm_secuencia_EF,cm_secuencia_EF)),
                                    np.concatenate((r_p_fm_secuencia_MOG,cm_secuencia_MOG)),
                                    np.concatenate((r_p_fm_secuencia_MOG2,cm_secuencia_MOG2)),
                                    np.concatenate((r_p_fm_secuencia_gt,cm_secuencia_gt))]),
                         columns = ['Recall', 'Precision','F-measure','TP','FP','FN','TN'])
df_highway['Algorithm'] = ['Exponential Filter','MOG','MOG2','Ground truth']
df_highway.set_index('Algorithm')
```

      Recall
      Precision
      F-measure
      TP
      FP
      FN
      TN

      Algorithm

      Exponential Filter
      0.590599
      0.434918
      0.500942
      4165508.0
      5412167.0
      2887516.0
      81998809.0

      MOG
      0.500938
      0.967187
      0.660027
      3156445.0
      107085.0
      3144618.0
      88055852.0

      MOG2
      0.933806
      0.774233
      0.846565
      6843616.0
      1995606.0
      485117.0
      85139661.0

      Ground truth
      1.000000
      1.000000
      1.000000
      5448641.0
      0.0
      0.0
      86228220.0

# 2. Secuencia _badminton_ del dataset _cameraJitter_

Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.

```python
#camaraJitter  badminton dataset input
gt_badminton_path = 'DATA/cameraJitter/cameraJitter/badminton/groundtruth/*.png'
in_badminton_path = 'DATA/cameraJitter/cameraJitter/badminton/input/*.jpg'

groundTruth = loadImages(gt_badminton_path)
frames = loadImages(in_badminton_path)
```

    Loading images: 100%|█████████████████████████████████████████████████████████████| 1150/1150 [00:03<00:00, 296.71it/s]
    Loading images: 100%|█████████████████████████████████████████████████████████████| 1150/1150 [00:08<00:00, 129.45it/s]

## 2.1. Algoritmo filtro exponencial

```python
# camaraJitter badminton dataset output path
expFilter_badminton_path = 'DATA/cameraJitter/results/badminton/expFilter/'

exponentialFilter(frames, 0.3, expFilter_badminton_path, 20, bgFilter ='MEDIAN')
```

Convertimos la secuencia de imágenes en video para poder visualizar mejor el resultado y mostramos el video en una ventana flotante.

```python
#im2vid(expFilter_badminton_path+'*.png','testvid.mp4')
#showVideo('testvid.mp4')
```

### 2.1.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_EF = all_metrics (expFilter_badminton_path+'*.png', gt_badminton_path, start_frame=800, end_frame=1150)
```

## 2.2. Algoritmo MOG

```python
# camaraJitter badminton dataset output paths
MOG_badminton_path = 'DATA/cameraJitter/results/badminton/MOG/'

MOG(frames, MOG_badminton_path)
```

### 2.2.1. Cálculo de métricas

```python
cm_frames_MOG,\
cm_secuencia_MOG,\
r_p_fm_frames_MOG,\
r_p_fm_secuencia_MOG = all_metrics (MOG_badminton_path+'*.png', gt_badminton_path, start_frame=800, end_frame=1150)
```

## 2.3. Algoritmo MOG2

```python
# camaraJitter badminton dataset output paths
MOG2_badminton_path = 'DATA/cameraJitter/results/badminton/MOG2'

MOG2(frames, MOG2_badminton_path)
```

### 2.3.1. Cálculo de métricas

```python
cm_frames_MOG2,\
cm_secuencia_MOG2,\
r_p_fm_frames_MOG2,\
r_p_fm_secuencia_MOG2 = all_metrics (MOG2_badminton_path+'*.png', gt_badminton_path, start_frame=800, end_frame=1150)
```

## Comparación resultados

```python
cm_frames_gt,\
cm_secuencia_gt,\
r_p_fm_frames_gt,\
r_p_fm_secuencia_gt = all_metrics (gt_badminton_path, gt_badminton_path, start_frame=800, end_frame=1150)
```

# 3. Secuencia _canoe_ del dataset _dynamicBackground_

```python
Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.
```

```python
# dynamicBackground  canoe dataset input
gt_canoe_path = 'DATA/dynamicBackground/dynamicBackground/canoe/groundtruth/*.png'
in_canoe_path = 'DATA/dynamicBackground/dynamicBackground/canoe/input/*.jpg'

groundTruth = loadImages(gt_canoe_path)
frames = loadImages(in_canoe_path)
```

```python
# dynamicBackground canoe dataset output path
expFilter_canoe_path = 'DATA/dynamicBackground/results/canoe/expFilter/'
MOG_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG/'
MOG2_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG2'
```

## 3.1. Algoritmo filtro exponencial

```python
# dynamicBackground canoe dataset output paths
expFilter_canoe_path = 'DATA/dynamicBackground/results/canoe/expFilter/'
```

### 3.1.1 Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_canoe_path+'*.jpg', gt_canoe_path, start_frame=469, end_frame=1699)
```

## 3.2. Algoritmo MOG

```python
# dynamicBackground canoe dataset output path
MOG_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG/'
```

### 3.2.1 Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (MOG_canoe_path+'*.jpg', gt_canoe_path, start_frame=469, end_frame=1699)
```

## 3.3. Algoritmo MOG2

```python
# dynamicBackground canoe dataset output path
MOG2_canoe_path = 'DATA/dynamicBackground/results/canoe/MOG2'
```

### 3.3.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (MOG2_canoe_path+'*.jpg', gt_canoe_path, start_frame=469, end_frame=1699)
```

## 3.4. Comparación resultados

# 4. Secuencia _sofa_ del dataset _intermittentObjectMotion_

```python
Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.
```

```python
# intermittentObjectMotion sofa dataset input
gt_sofa_path = 'DATA/intermittentObjectMotion/intermittentObjectMotion/sofa/groundtruth/*.png'
in_sofa_path = 'DATA/intermittentObjectMotion/intermittentObjectMotion/sofa/input/*.jpg'

groundTruth = loadImages(gt_sofa_path)
frames = loadImages(in_sofa_path)
```

## 4.1. Algoritmo filtro exponencial

### 4.1.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 4.2. Algoritmo MOG

### 4.2.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 4.3. Algoritmo MOG2

### 4.3.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 4.4. Comparación resultados

# 5. Secuencia _copyMachine_ del dataset _shadow_

```python
Cargamos las imágenes de entrada y el ground truth con el que se calcularán las métricas.
```

```python
# shadow copyMachine dataset input
gt_copyMachine_path = 'DATA/shadow/shadow/copyMachine/groundtruth/*.png'
in_copyMachine_path = 'DATA/shadow/shadow/copyMachine/input/*.jpg'

groundTruth = loadImages(gt_copyMachine_path)
frames = loadImages(in_copyMachine_path)
```

## 5.1. Algoritmo filtro exponencial

### 5.1.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 5.2. Algoritmo MOG

### 5.2.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 5.3. Algoritmo MOG2

### 5.3.1. Cálculo de métricas

```python
cm_frames_EF,\
cm_secuencia_EF,\
r_p_fm_frames_EF,\
r_p_fm_secuencia_RF = all_metrics (expFilter_highway_path+'*.jpg', gt_highway_path, start_frame=469, end_frame=1699)
```

## 5.4. Comparación resultados