```
# this mounts your Google Drive to the Colab VM.
from google.colab import drive
drive.mount('/content/drive', force_remount=True)

# enter the foldername in your Drive where you have saved the unzipped
# assignment folder, e.g. 'cs231n/assignments/assignment3/'
FOLDERNAME = 'cs231n/assignments/assignment2/'
assert FOLDERNAME is not None, "[!] Enter the foldername."

# now that we've mounted your Drive, this ensures that
# the Python interpreter of the Colab VM can load
# python files from within it.
import sys
sys.path.append('/content/drive/My Drive/{}'.format(FOLDERNAME))

# this downloads the CIFAR-10 dataset to your Drive
# if it doesn't already exist.
%cd drive/My\ Drive/$FOLDERNAME/cs231n/datasets/
!bash get_datasets.sh
%cd /content
```

    Go to this URL in a browser: https://accounts.google.com/o/oauth2/auth?client_id=947318989803-6bn6qk8qdgf4n4g3pfee6491hc0brc4i.apps.googleusercontent.com&redirect_uri=urn%3aietf%3awg%3aoauth%3a2.0%3aoob&response_type=code&scope=email%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdocs.test%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive%20https%3a%2f%2fwww.googleapis.com%2fauth%2fdrive.photos.readonly%20https%3a%2f%2fwww.googleapis.com%2fauth%2fpeopleapi.readonly

    Enter your authorization code:
    ··········
    Mounted at /content/drive
    /content/drive/My Drive/cs231n/assignments/assignment2/cs231n/datasets
    /content

# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.
    	There will be an option for Colab users and another for Jupyter (local) users.

```
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [5.32907052e-17 7.04991621e-17 1.85962357e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029261167605239e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  6.284600172572596e-13
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 2.10x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.3004790897684924
    W1 relative error: 1.48e-07
    W2 relative error: 2.21e-05
    W3 relative error: 3.53e-07
    b1 relative error: 5.38e-09
    b2 relative error: 2.09e-09
    b3 relative error: 5.80e-11

    Running check with reg =  3.14
    Initial loss:  7.052114776533016
    W1 relative error: 3.90e-09
    W2 relative error: 6.87e-08
    W3 relative error: 2.13e-08
    b1 relative error: 1.48e-08
    b2 relative error: 1.72e-09
    b3 relative error: 1.57e-10

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.302838
    (Epoch 0 / 10) train acc: 0.131000; val_acc: 0.119000
    (Epoch 1 / 10) train acc: 0.227000; val_acc: 0.188000
    (Iteration 21 / 200) loss: 2.066770
    (Epoch 2 / 10) train acc: 0.274000; val_acc: 0.231000
    (Iteration 41 / 200) loss: 2.026292
    (Epoch 3 / 10) train acc: 0.301000; val_acc: 0.255000
    (Iteration 61 / 200) loss: 2.081528
    (Epoch 4 / 10) train acc: 0.368000; val_acc: 0.278000
    (Iteration 81 / 200) loss: 1.743705
    (Epoch 5 / 10) train acc: 0.405000; val_acc: 0.285000
    (Iteration 101 / 200) loss: 1.511911
    (Epoch 6 / 10) train acc: 0.479000; val_acc: 0.308000
    (Iteration 121 / 200) loss: 1.679106
    (Epoch 7 / 10) train acc: 0.501000; val_acc: 0.266000
    (Iteration 141 / 200) loss: 1.635350
    (Epoch 8 / 10) train acc: 0.508000; val_acc: 0.286000
    (Iteration 161 / 200) loss: 1.190941
    (Epoch 9 / 10) train acc: 0.594000; val_acc: 0.311000
    (Iteration 181 / 200) loss: 1.270953
    (Epoch 10 / 10) train acc: 0.650000; val_acc: 0.332000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696059
    (Epoch 6 / 10) train acc: 0.535000; val_acc: 0.345000
    (Iteration 121 / 200) loss: 1.557987
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.304000
    (Iteration 141 / 200) loss: 1.432189
    (Epoch 8 / 10) train acc: 0.628000; val_acc: 0.339000
    (Iteration 161 / 200) loss: 1.033931
    (Epoch 9 / 10) train acc: 0.661000; val_acc: 0.340000
    (Iteration 181 / 200) loss: 0.901034
    (Epoch 10 / 10) train acc: 0.726000; val_acc: 0.318000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
The second plot shows the problem of vanishing gradients (small initial weights). The baseline model is very sensitive to this problem (the accuracy is very low), therefore finding the correct weight scale is difficult. For this example, the baseline obtains the best result with a weight scale equal to 1e-1. On the other hand, we can see that the batchnorm model is less sensitive to weight initialization because its accuracy is around 30% for all the different weight scales.

The behaviour of the first plot is very similar to that of the second plot. The main difference is that the first plot shows that we are overfitting our model, besides that we can see that with the batchnorm model we obtained better results than the baseline model and that occurs because batch normalization has regularization properties.

The third plot depicts the problem of exploding gradients and it is very evident in the baseline model for weight scale values greater than 1e-1. However, the batchnorm model does not suffer from this problem.

In general with batch normalization we can avoid the problem of vanishing and exploding gradients because it normalizes every affine layer (xW+b), avoiding very large/small values. Moreover, its regularization properties allow to decrease overfitting.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
According to the results, we can see that the batch size affects directly the performance of batch normalization (the smaller the batch size the worse). Even the baseline model outperforms the batchnorm model when using a very small batch size. This problem occurs because when we calculate the statistics of a batch, i.e., mean and variance, we try to find an approximation of the statistics of the entire dataset. Therefore with a small batch size, these statistics can be very noisy. On the other hand, with a large batch size we can obtain a better approximation.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
Number 2 is analogous to layer normalization when we consider: mean = 0, beta parameter = 0 (at this point we have gamma*x/std where std=sqrt(sum(x^2))) and gamma=x/std. Thus the result of layer normalization will be x^2/sum(x^2).

Number 3 is analogous to batch normalization when we consider: batch size = size of the dataset, gamma parameter = standard deviation and beta parameter = 0. Thus the result of batch normalization will be std*(x-mean)/std + 0 = x-mean.

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.4336158494902849e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
1. [INCORRECT] In the previous example, the network had five layers and it can be considered as a deep network. Thus, using layer normalization in deep networks works correctly.

2. [CORRECT] Having a small dimension of features affects the performance of layer normalization. The problem is very similar to that of batch normalization with small batch size because in layer normalization we calculate the statistics according to the number of hidden units, which represent the features that the network is learning. Thus, the smaller the hidden size the noisier the statistics used in layer normalization.

3. [CORRECT] Having a high regularization term affects the performance of layer normalization. In general, when the regularization term is very high, the model learns very simple functions (underfitting).


```python
%matplotlib inline
```

```python
# Write your imports here
import sympy
import numpy as np
import matplotlib.pyplot as plt
import math
```

# High-School Maths Exercise
## Getting to Know Jupyter Notebook. Python Libraries and Best Practices. Basic Workflow

### Problem 1. Markdown
Jupyter Notebook is a very light, beautiful and convenient way to organize your research and display your results. Let's play with it for a while.

First, you can double-click each cell and edit its content. If you want to run a cell (that is, execute the code inside it), use Cell > Run Cells in the top menu or press Ctrl + Enter.

Second, each cell has a type. There are two main types: Markdown (which is for any kind of free text, explanations, formulas, results... you get the idea), and code (which is, well... for code :D).

Let me give you a...
#### Quick Introduction to Markdown
##### Text and Paragraphs
There are several things that you can do. As you already saw, you can write paragraph text just by typing it. In order to create a new paragraph, just leave a blank line. See how this works below:
```
This is some text.
This text is on a new line, but it will continue the same paragraph (so you can make your paragraphs more easily readable by just continuing on a new line, or just go on and on like this one line is ever continuing).

This text is displayed in a new paragraph.

And this is yet another paragraph.
```
**Result:**

This is some text.
This text is on a new line, but it will continue the same paragraph (so you can make your paragraphs more easily readable by just continuing on a new line, or just go on and on like this one line is ever continuing).

This text is displayed in a new paragraph.

And this is yet another paragraph.

##### Headings
There are six levels of headings. Level one is the highest (largest and most important), and level 6 is the smallest. You can create headings of several types by prefixing the header line with one to six "#" symbols (this is called a pound sign if you are ancient, or a sharp sign if you're a musician... or a hashtag if you're too young :D). Have a look:
```
# Heading 1
## Heading 2
### Heading 3
#### Heading 4
##### Heading 5
###### Heading 6
```

**Result:**

# Heading 1
## Heading 2
### Heading 3
#### Heading 4
##### Heading 5
###### Heading 6

It is recommended that you have **only one** H1 heading - this should be the header of your notebook (or scientific paper). Below that, you can add your name or just jump to the explanations directly.

##### Emphasis
You can create emphasized (stonger) text by using a **bold** or _italic_ font. You can do this in several ways (using asterisks (\*) or underscores (\_)). In order to "escape" a symbol, prefix it with a backslash (\). You can also strike thorugh your text in order to signify a correction.
```
**bold** __bold__
*italic* _italic_

This is \*\*not \*\* bold.

I ~~didn't make~~ a mistake.
```

**Result:**

**bold** __bold__
*italic* _italic_

This is \*\*not\*\* bold.

I ~~didn't make~~ a mistake.

##### Lists
You can add two types of lists: ordered and unordered. Lists can also be nested inside one another. To do this, press Tab once (it will be converted to 4 spaces).

To create an ordered list, just type the numbers. Don't worry if your numbers are wrong - Jupyter Notebook will create them properly for you. Well, it's better to have them properly numbered anyway...
```
1. This is
2. A list
10. With many
9. Items
    1. Some of which
    2. Can
        3. Be nested
42. You can also
    * Mix
    * list
    * types
```

**Result:**
1. This is
2. A list
10. With many
9. Items
    1. Some of which
    2. Can
        3. Be nested
42. You can also
    * Mix
    * list
    * types

To create an unordered list, type an asterisk, plus or minus at the beginning:
```
* This is
* An
    + Unordered
    - list
```

**Result:**
* This is
* An
    + Unordered
        - list

##### Links
There are many ways to create links but we mostly use one of them: we present links with some explanatory text. See how it works:
```
This is [a link](http://google.com) to Google.
```

**Result:**

This is [a link](http://google.com) to Google.

##### Images
They are very similar to links. Just prefix the image with an exclamation mark. The alt(ernative) text will be displayed if the image is not available. Have a look (hover over the image to see the title text):
```
 Do you know that "taco cat" is a palindrome? Thanks to The Oatmeal :)
```

**Result:**

 Do you know that "taco cat" is a palindrome? Thanks to The Oatmeal :)

If you want to resize images or do some more advanced stuff, just use HTML.

Did I mention these cells support HTML, CSS and JavaScript? Now I did.

##### Tables
These are a pain because they need to be formatted (somewhat) properly. Here's a good [table generator](http://www.tablesgenerator.com/markdown_tables). Just select File > Paste table data... and provide a tab-separated list of values. It will generate a good-looking ASCII-art table for you.
```
| Cell1 | Cell2 | Cell3 |
|-------|-------|-------|
| 1.1   | 1.2   | 1.3   |
| 2.1   | 2.2   | 2.3   |
| 3.1   | 3.2   | 3.3   |
```

**Result:**

| Cell1 | Cell2 | Cell3 |
|-------|-------|-------|
| 1.1   | 1.2   | 1.3   |
| 2.1   | 2.2   | 2.3   |
| 3.1   | 3.2   | 3.3   |

##### Code
Just use triple backtick symbols. If you provide a language, it will be syntax-highlighted. You can also use inline code with single backticks.

```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Result:**
```python
def square(x):
    return x ** 2
```
This is `inline` code. No syntax highlighting here.

**Now it's your turn to have some Markdown fun.** In the next cell, try out some of the commands. You can just throw in some things, or do something more structured (like a small notebook).

___some markdown here___

### Problem 2. Formulas and LaTeX
Writing math formulas has always been hard. But scientists don't like difficulties and prefer standards. So, thanks to Donald Knuth (a very popular computer scientist, who also invented a lot of algorithms), we have a nice typesetting system, called LaTeX (pronounced _lah_-tek). We'll be using it mostly for math formulas, but it has a lot of other things to offer.

There are two main ways to write formulas. You could enclose them in single `$` signs like this: `$ ax + b $`, which will create an **inline formula**: $ ax + b $. You can also enclose them in double `$` signs `$$ ax + b $$` to produce $$ ax + b $$.

Most commands start with a backslash and accept parameters either in square brackets `[]` or in curly braces `{}`. For example, to make a fraction, you typically would write `$$ \frac{a}{b} $$`: $$ \frac{a}{b} $$.

[Here's a resource](http://www.stat.pitt.edu/stoffer/freetex/latex%20basics.pdf) where you can look up the basics of the math syntax. You can also search StackOverflow - there are all sorts of solutions there.

You're on your own now. Research and recreate all formulas shown in the next cell. Try to make your cell look exactly the same as mine. It's an image, so don't try to cheat by copy/pasting :D.

Note that you **do not** need to understand the formulas, what's written there or what it means. We'll have fun with these later in the course.

$$ y = ax + b $$

$$ ax^2 + bx + c = 0 $$

$$ x_{1,2}= \frac{-b \pm\sqrt{b^2 - 4ac}}{2a} $$

\begin{equation}
f(x)|_{x=a} = f(a) + f\prime(a)(x-a) + \frac{f^n(a)}{2!}(x-a)^2 + ... + \frac{f^n(a)}{n!}(x-a)^n + ...
\end{equation}

\begin{equation}
(x + y)^n = {n\choose 0}x^ny^0 + {n\choose 1}x^{n-1}y^1 + ... + {n\choose n}x^0y^n = \sum_{k=0}^{n} {n\choose k}x^{n-k}y^k
\end{equation}

\begin{equation}
\int_{-\infty}^{+\infty} e^{-x^2}dx = \sqrt{\pi}
\end{equation}

\begin{equation}
\left( \begin{array}{ccc}
2 & 1 & 3 \\
2 & 6 & 8 \\
6 & 8 & 18 \end{array} \right)
\end{equation}

\begin{equation}
A = \begin{pmatrix}
    a_{11} & a_{12} & \dots & a_{1n} \\
    a_{21} & a_{22} & \dots & a_{2n} \\
    \vdots & \vdots & \ddots & \vdots \\
    a_{m1} & a_{m1} & \dots & a_{mn}
    \end{pmatrix}
\end{equation}

Write your formulas here.

### Problem 3. Solving with Python
Let's first do some symbolic computation. We need to import `sympy` first.

**Should your imports be in a single cell at the top or should they appear as they are used?** There's not a single valid best practice. Most people seem to prefer imports at the top of the file though. **Note: If you write new code in a cell, you have to re-execute it!**

Let's use `sympy` to give us a quick symbolic solution to our equation. First import `sympy` (you can use the second cell in this notebook):
```python
import sympy
```

Next, create symbols for all variables and parameters. You may prefer to do this in one pass or separately:
```python
x = sympy.symbols('x')
a, b, c = sympy.symbols('a b c')
```

Now solve:
```python
sympy.solve(a * x**2 + b * x + c)
```

Hmmmm... we didn't expect that :(. We got an expression for $a$ because the library tried to solve for the first symbol it saw. This is an equation and we have to solve for $x$. We can provide it as a second paramter:
```python
sympy.solve(a * x**2 + b * x + c, x)
```

Finally, if we use `sympy.init_printing()`, we'll get a LaTeX-formatted result instead of a typed one. This is very useful because it produces better-looking formulas.

```python
x = sympy.symbols('x')
a, b, c = sympy.symbols('a b c')

sympy.solve(a * x**2 + b * x + c)

sympy.init_printing()

a = 5
```

How about a function that takes $a, b, c$ (assume they are real numbers, you don't need to do additional checks on them) and returns the **real** roots of the quadratic equation?

Remember that in order to calculate the roots, we first need to see whether the expression under the square root sign is non-negative.

If $b^2 - 4ac > 0$, the equation has two real roots: $x_1, x_2$

If $b^2 - 4ac = 0$, the equation has one real root: $x_1 = x_2$

If $b^2 - 4ac < 0$, the equation has zero real roots

Write a function which returns the roots. In the first case, return a list of 2 numbers: `[2, 3]`. In the second case, return a list of only one number: `[2]`. In the third case, return an empty list: `[]`.

```python

def solve_quadratic_equation(a, b, c):
    """
    Returns the real solutions of the quadratic equation ax^2 + bx + c = 0
    """
    if a == 0:
        if b == 0:
            return math.nan
        elif c == 0:
            return b
        else:
            return -c / b
    else:
        d = b**2-4*a*c
        answer = []
        if d > 0:
            answer.append((-b - math.sqrt(d)) / (2*a))
            answer.append((-b + math.sqrt(d)) / (2*a))
        elif d == 0:
            answer.append(-b / (2*a))
        return answer
```

```python
# Testing: Execute this cell. The outputs should match the expected outputs. Feel free to write more tests
print(solve_quadratic_equation(1, -1, -2)) # [-1.0, 2.0]
print(solve_quadratic_equation(1, -8, 16)) # [4.0]
print(solve_quadratic_equation(1, 1, 1)) # []
```

    [-1.0, 2.0]
    [4.0]
    []

**Bonus:** Last time we saw how to solve a linear equation. Remember that linear equations are just like quadratic equations with $a = 0$. In this case, however, division by 0 will throw an error. Extend your function above to support solving linear equations (in the same way we did it last time).

### Problem 4. Equation of a Line
Let's go back to our linear equations and systems. There are many ways to define what "linear" means, but they all boil down to the same thing.

The equation $ax + b = 0$ is called *linear* because the function $f(x) = ax+b$ is a linear function. We know that there are several ways to know what one particular function means. One of them is to just write the expression for it, as we did above. Another way is to **plot** it. This is one of the most exciting parts of maths and science - when we have to fiddle around with beautiful plots (although not so beautiful in this case).

The function produces a straight line and we can see it.

How do we plot functions in general? Ww know that functions take many (possibly infinitely many) inputs. We can't draw all of them. We could, however, evaluate the function at some points and connect them with tiny straight lines. If the points are too many, we won't notice - the plot will look smooth.

Now, let's take a function, e.g. $y = 2x + 3$ and plot it. For this, we're going to use `numpy` arrays. This is a special type of array which has two characteristics:
* All elements in it must be of the same type
* All operations are **broadcast**: if `x = [1, 2, 3, 10]` and we write `2 * x`, we'll get `[2, 4, 6, 20]`. That is, all operations are performed at all indices. This is very powerful, easy to use and saves us A LOT of looping.

There's one more thing: it's blazingly fast because all computations are done in C, instead of Python.

First let's import `numpy`. Since the name is a bit long, a common convention is to give it an **alias**:
```python
import numpy as np
```

Import that at the top cell and don't forget to re-run it.

Next, let's create a range of values, e.g. $[-3, 5]$. There are two ways to do this. `np.arange(start, stop, step)` will give us evenly spaced numbers with a given step, while `np.linspace(start, stop, num)` will give us `num` samples. You see, one uses a fixed step, the other uses a number of points to return. When plotting functions, we usually use the latter. Let's generate, say, 1000 points (we know a straight line only needs two but we're generalizing the concept of plotting here :)).
```python
x = np.linspace(-3, 5, 1000)
```
Now, let's generate our function variable
```python
y = 2 * x + 3
```

We can print the values if we like but we're more interested in plotting them. To do this, first let's import a plotting library. `matplotlib` is the most commnly used one and we usually give it an alias as well.
```python
import matplotlib.pyplot as plt
```

Now, let's plot the values. To do this, we just call the `plot()` function. Notice that the top-most part of this notebook contains a "magic string": `%matplotlib inline`. This hints Jupyter to display all plots inside the notebook. However, it's a good practice to call `show()` after our plot is ready.
```python
plt.plot(x, y)
plt.show()
```

```python
x = np.linspace(-3, 5, 1000)
y = 2 * x + 3
plt.plot(x, y)
plt.show()
```

It doesn't look too bad bit we can do much better. See how the axes don't look like they should? Let's move them to zeto. This can be done using the "spines" of the plot (i.e. the borders).

All `matplotlib` figures can have many plots (subfigures) inside them. That's why when performing an operation, we have to specify a target figure. There is a default one and we can get it by using `plt.gca()`. We usually call it `ax` for "axis".
Let's save it in a variable (in order to prevent multiple calculations and to make code prettier). Let's now move the bottom and left spines to the origin $(0, 0)$ and hide the top and right one.
```python
ax = plt.gca()
ax.spines["bottom"].set_position("zero")
ax.spines["left"].set_position("zero")
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
```

**Note:** All plot manipulations HAVE TO be done before calling `show()`. It's up to you whether they should be before or after the function you're plotting.

This should look better now. We can, of course, do much better (e.g. remove the double 0 at the origin and replace it with a single one), but this is left as an exercise for the reader :).

```python
x = np.linspace(-3, 5, 1000)
y = 2 * x + 3
plt.plot(x, y)
ax = plt.gca()
ax.spines["bottom"].set_position("zero")
ax.spines["left"].set_position("zero")
ax.spines["top"].set_visible(False)
ax.spines["right"].set_visible(False)
plt.show()
```

### * Problem 5. Linearizing Functions
Why is the line equation so useful? The main reason is because it's so easy to work with. Scientists actually try their best to linearize functions, that is, to make linear functions from non-linear ones. There are several ways of doing this. One of them involves derivatives and we'll talk about it later in the course.

A commonly used method for linearizing functions is through algebraic transformations. Try to linearize
$$ y = ae^{bx} $$

Hint: The inverse operation of $e^{x}$ is $\ln(x)$. Start by taking $\ln$ of both sides and see what you can do. Your goal is to transform the function into another, linear function. You can look up more hints on the Internet :).

$$ ln(y) = ln(a e^{bx}) $$

$$ ln(y) = ln(a) + ln(e^{bx}) $$

$$ ln(y) = ln(a) + bx $$

### * Problem 6. Generalizing the Plotting Function
Let's now use the power of Python to generalize the code we created to plot. In Python, you can pass functions as parameters to other functions. We'll utilize this to pass the math function that we're going to plot.

Note: We can also pass *lambda expressions* (anonymous functions) like this:
```python
lambda x: x + 2```
This is a shorter way to write
```python
def some_anonymous_function(x):
    return x + 2
```

We'll also need a range of x values. We may also provide other optional parameters which will help set up our plot. These may include titles, legends, colors, fonts, etc. Let's stick to the basics now.

Write a Python function which takes another function, x range and number of points, and plots the function graph by evaluating it at every point.

**BIG hint:** If you want to use not only `numpy` functions for `f` but any one function, a very useful (and easy) thing to do, is to vectorize the function `f` (e.g. to allow it to be used with `numpy` broadcasting):
```python
f_vectorized = np.vectorize(f)
y = f_vectorized(x)
```

```python
def plot_math_function(f, min_x, max_x, num_points):
    xpts = np.linspace(min_x, max_x, num_points)
    plt.plot(xpts, [f(x) for x in xpts])
    ax = plt.gca()
    ax.spines["bottom"].set_position("zero")
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    plt.show()
```

```python
plot_math_function(lambda x: 2 * x + 3, -3, 5, 1000)
plot_math_function(lambda x: -x + 8, -1, 10, 1000)
plot_math_function(lambda x: x**2 - x - 2, -3, 4, 1000)
plot_math_function(lambda x: np.sin(x), -np.pi, np.pi, 1000)
plot_math_function(lambda x: np.sin(x) / x, -4 * np.pi, 4 * np.pi, 1000)
```

### * Problem 7. Solving Equations Graphically
Now that we have a general plotting function, we can use it for more interesting things. Sometimes we don't need to know what the exact solution is, just to see where it lies. We can do this by plotting the two functions around the "=" sign ans seeing where they intersect. Take, for example, the equation $2x + 3 = 0$. The two functions are $f(x) = 2x + 3$ and $g(x) = 0$. Since they should be equal, the point of their intersection is the solution of the given equation. We don't need to bother marking the point of intersection right now, just showing the functions.

To do this, we'll need to improve our plotting function yet once. This time we'll need to take multiple functions and plot them all on the same graph. Note that we still need to provide the $[x_{min}; x_{max}]$ range and it's going to be the same for all functions.

```python
vectorized_fs = [np.vectorize(f) for f in functions]
ys = [vectorized_f(x) for vectorized_f in vectorized_fs]
```

```python
def plot_math_functions(functions, min_x, max_x, num_points):
    xpts = np.linspace(min_x, max_x, num_points)
    vectorized_fs = [np.vectorize(f) for f in functions]
    ys = [vectorized_f(xpts) for vectorized_f in vectorized_fs]
    for f in ys:
        plt.plot(xpts, f)
    ax = plt.gca()
    ax.spines["bottom"].set_position("zero")
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    plt.show()
```

```python
plot_math_functions([lambda x: 2 * x + 3, lambda x: 0], -3, 5, 1000)
plot_math_functions([lambda x: 3 * x**2 - 2 * x + 5, lambda x: 3 * x + 7], -2, 3, 1000)
```

This is also a way to plot the solutions of systems of equation, like the one we solved last time. Let's actually try it.

```python
plot_math_functions([lambda x: (-4 * x + 7) / 3, lambda x: (-3 * x + 8) / 5, lambda x: (-x - 1) / -2], -1, 4, 1000)
```

### Problem 8. Trigonometric Functions
We already saw the graph of the function $y = \sin(x)$. But, how do we define the trigonometric functions once again? Let's quickly review that.

The two basic trigonometric functions are defined as the ratio of two sides:
$$ \sin(x) = \frac{\text{opposite}}{\text{hypotenuse}} $$
$$ \cos(x) = \frac{\text{adjacent}}{\text{hypotenuse}} $$

And also:
$$ \tan(x) = \frac{\text{opposite}}{\text{adjacent}} = \frac{\sin(x)}{\cos(x)} $$
$$ \cot(x) = \frac{\text{adjacent}}{\text{opposite}} = \frac{\cos(x)}{\sin(x)} $$

This is fine, but using this, "right-triangle" definition, we're able to calculate the trigonometric functions of angles up to $90^\circ$. But we can do better. Let's now imagine a circle centered at the origin of the coordinate system, with radius $r = 1$. This is called a "unit circle".

We can now see exactly the same picture. The $x$-coordinate of the point in the circle corresponds to $\cos(\alpha)$ and the $y$-coordinate - to $\sin(\alpha)$. What did we get? We're now able to define the trigonometric functions for all degrees up to $360^\circ$. After that, the same values repeat: these functions are **periodic**:
$$ \sin(k.360^\circ + \alpha) = \sin(\alpha), k = 0, 1, 2, \dots $$
$$ \cos(k.360^\circ + \alpha) = \cos(\alpha), k = 0, 1, 2, \dots $$

We can, of course, use this picture to derive other identities, such as:
$$ \sin(90^\circ + \alpha) = \cos(\alpha) $$

A very important property of the sine and cosine is that they accept values in the range $(-\infty; \infty)$ and produce values in the range $[-1; 1]$. The two other functions take values in the range $(-\infty; \infty)$ **except when their denominators are zero** and produce values in the same range.

#### Radians
A degree is a geometric object, $1/360$th of a full circle. This is quite inconvenient when we work with angles. There is another, natural and intrinsic measure of angles. It's called the **radian** and can be written as $\text{rad}$ or without any designation, so $\sin(2)$ means "sine of two radians".

It's defined as *the central angle of an arc with length equal to the circle's radius* and $1\text{rad} \approx 57.296^\circ$.

We know that the circle circumference is $C = 2\pi r$, therefore we can fit exactly $2\pi$ arcs with length $r$ in $C$. The angle corresponding to this is $360^\circ$ or $2\pi\ \text{rad}$. Also, $\pi rad = 180^\circ$.

(Some people prefer using $\tau = 2\pi$ to avoid confusion with always multiplying by 2 or 0.5 but we'll use the standard notation here.)

**NOTE:** All trigonometric functions in `math` and `numpy` accept radians as arguments. In order to convert between radians and degrees, you can use the relations $\text{[deg]} = 180/\pi.\text{[rad]}, \text{[rad]} =  \pi/180.\text{[deg]}$. This can be done using `np.deg2rad()` and `np.rad2deg()` respectively.

#### Inverse trigonometric functions
All trigonometric functions have their inverses. If you plug in, say $\pi/4$ in the $\sin(x)$ function, you get $\sqrt{2}/2$. The inverse functions (also called, arc-functions) take arguments in the interval $[-1; 1]$ and return the angle that they correspond to. Take arcsine for example:
$$ \arcsin(y) = x: sin(y) = x $$
$$ \arcsin\left(\frac{\sqrt{2}}{2}\right) = \frac{\pi}{4} $$

Please note that this is NOT entirely correct. From the relations we found:
$$\sin(x) = sin(2k\pi + x), k = 0, 1, 2, \dots $$

it follows that $\arcsin(x)$ has infinitely many values, separated by $2k\pi$ radians each:
$$ \arcsin\left(\frac{\sqrt{2}}{2}\right) = \frac{\pi}{4} + 2k\pi, k = 0, 1, 2, \dots $$

In most cases, however, we're interested in the first value (when $k = 0$). It's called the **principal value**.

Note 1: There are inverse functions for all four basic trigonometric functions: $\arcsin$, $\arccos$, $\arctan$, $\text{arccot}$. These are sometimes written as $\sin^{-1}(x)$, $cos^{-1}(x)$, etc. These definitions are completely equivalent.

Just notice the difference between $\sin^{-1}(x) := \arcsin(x)$ and $\sin(x^{-1}) = \sin(1/x)$.

#### Exercise
Use the plotting function you wrote above to plot the inverse trigonometric functions. Use `numpy` (look up how to use inverse trigonometric functions).

```python
def plot_math_functions(min_x, max_x, num_points):
    xpts = np.linspace(min_x, max_x)
    plt.plot(xpts, np.arcsin(xpts))
    plt.plot(xpts, np.arccos(xpts))
    plt.plot(xpts, np.arctan(xpts))
#     plt.plot(xpts, np.arccos(xpts) / np.arcsin(xpts))
    ax = plt.gca()
    ax.spines["bottom"].set_position("zero")
    ax.spines["left"].set_position("zero")
    ax.spines["top"].set_visible(False)
    ax.spines["right"].set_visible(False)
    plt.show()
plot_math_functions(1, -1, 20)
```

### ** Problem 9. Perlin Noise
This algorithm has many applications in computer graphics and can serve to demonstrate several things... and help us learn about math, algorithms and Python :).
#### Noise
Noise is just random values. We can generate noise by just calling a random generator. Note that these are actually called *pseudorandom generators*. We'll talk about this later in this course.
We can generate noise in however many dimensions we want. For example, if we want to generate a single dimension, we just pick N random values and call it a day. If we want to generate a 2D noise space, we can take an approach which is similar to what we already did with `np.meshgrid()`.

$$ \text{noise}(x, y) = N, N \in [n_{min}, n_{max}] $$

This function takes two coordinates and returns a single number N between $n_{min}$ and $n_{max}$. (This is what we call a "scalar field").

Random variables are always connected to **distributions**. We'll talk about these a great deal but now let's just say that these define what our noise will look like. In the most basic case, we can have "uniform noise" - that is, each point in our little noise space $[n_{min}, n_{max}]$ will have an equal chance (probability) of being selected.

#### Perlin noise
There are many more distributions but right now we'll want to have a look at a particular one. **Perlin noise** is a kind of noise which looks smooth. It looks cool, especially if it's colored. The output may be tweaked to look like clouds, fire, etc. 3D Perlin noise is most widely used to generate random terrain.

#### Algorithm
... Now you're on your own :). Research how the algorithm is implemented (note that this will require that you understand some other basic concepts like vectors and gradients).

#### Your task
1. Research about the problem. See what articles, papers, Python notebooks, demos, etc. other people have created
2. Create a new notebook and document your findings. Include any assumptions, models, formulas, etc. that you're using
3. Implement the algorithm. Try not to copy others' work, rather try to do it on your own using the model you've created
4. Test and improve the algorithm
5. (Optional) Create a cool demo :), e.g. using Perlin noise to simulate clouds. You can even do an animation (hint: you'll need gradients not only in space but also in time)
6. Communicate the results (e.g. in the Softuni forum)

Hint: [This](http://flafla2.github.io/2014/08/09/perlinnoise.html) is a very good resource. It can show you both how to organize your notebook (which is important) and how to implement the algorithm.


```python
from IPython.display import Image
Image('../../Python_probability_statistics_machine_learning_2E.png',width=200)
```

We considered Maximum Likelihood Estimation (MLE) and Maximum A-Posteriori
(MAP)
estimation and in each case we started out with a probability density
function
of some kind and we further assumed that the samples were identically
distributed and independent (iid). The idea behind robust statistics
[[maronna2006robust]](#maronna2006robust) is to construct estimators that can
survive the
weakening of either or both of these assumptions. More concretely,
suppose you
have a model that works great except for a few outliers. The
temptation is to
just ignore the outliers and proceed. Robust estimation methods
provide a
disciplined way to handle outliers without cherry-picking data that
works for
your favored model.

### The Notion of Location

The first notion we
need is *location*, which is  a generalization of the idea
of *central value*.
Typically, we just use an estimate of the mean for this,
but we will see later
why this could be a bad idea.  The general idea of
location satisfies the
following requirements Let $X$ be a random variable with
distribution $F$, and
let $\theta(X)$ be some descriptive measure of $F$. Then
$\theta(X)$ is said to
be a measure of *location* if for any constants *a* and
*b*, we have the
following:

$$
\begin{equation}
\theta(X+b)  = \theta(X) +b
\label{_auto1} \tag{1}
\end{equation}
$$

$$
\begin{equation} \
\theta(-X)   = -\theta(X)
\label{_auto2} \tag{2}
\end{equation}
$$

$$
\begin{equation} \
X \ge 0 \Rightarrow \theta(X)  \ge 0
\label{_auto3} \tag{3}
\end{equation}
$$

$$
\begin{equation} \
\theta(a X) = a\theta(X)
\label{_auto4} \tag{4}
\end{equation}
$$

 The first condition is called *location equivariance* (or *shift-invariance* in
signal processing lingo). The fourth condition is called *scale equivariance*,
which means that the units that $X$ is measured in should not effect the value
of the  location estimator.  These requirements capture the intuition of
*centrality* of a distribution, or where most of the
probability mass is
located.

For example, the sample mean estimator is $ \hat{\mu}=\frac{1}{n}\sum
X_i $. The first
requirement is obviously satisfied as $
\hat{\mu}=\frac{1}{n}\sum (X_i+b) = b +
\frac{1}{n}\sum X_i =b+\hat{\mu}$. Let
us consider the second requirement:$
\hat{\mu}=\frac{1}{n}\sum -X_i =
-\hat{\mu}$. Finally, the last requirement is
satisfied with $
\hat{\mu}=\frac{1}{n}\sum a X_i =a \hat{\mu}$.

### Robust Estimation and Contamination

Now that we have the generalized location of centrality embodied
in the
*location* parameter, what can we do with it?  Previously, we assumed
that our samples
were all identically distributed. The key idea is that the
samples might be
actually coming from a *single* distribution that is
contaminated by another nearby
distribution, as in the following:

$$
F(X) = \epsilon G(X) + (1-\epsilon)H(X)
$$

 where $ \epsilon $ randomly toggles between zero and one. This means
that our
data samples $\lbrace X_i \rbrace$ actually derived from two separate
distributions, $ G(X) $ and $ H(X) $. We just don't know how they are mixed
together. What we really want  is an estimator  that captures the location of $
G(X) $ in the face of random intermittent contamination by $ H(X)$.  For
example, it may be that this contamination is responsible for the outliers in a
model that otherwise works well with the dominant $F$ distribution. It can get
even worse than that because we don't know that there is only one contaminating
$H(X)$ distribution out there. There may be a whole family of distributions
that
are contaminating $G(X)$. This means that whatever estimators we construct
have
to be derived from a more generalized family of distributions instead of
from a
single distribution,  as the maximum-likelihood method assumes.  This is
what
makes robust estimation so difficult --- it has to deal with *spaces* of
function distributions instead of parameters from a particular probability
distribution.

### Generalized Maximum Likelihood Estimators

M-estimators are
generalized maximum likelihood estimators. Recall that for
maximum likelihood,
we want to maximize the likelihood function as in the
following:

$$
L_{\mu}(x_i) = \prod f_0(x_i-\mu)
$$

 and then to find the estimator $\hat{\mu}$ so that

$$
\hat{\mu} = \arg \max_{\mu} L_{\mu}(x_i)
$$

  So far, everything is the same as our usual maximum-likelihood
derivation
except for the fact that we don't assume a specific $f_0$ as the
distribution of
the $\lbrace X_i\rbrace$. Making the definition of

$$
\rho = -\log f_0
$$

 we obtain the more convenient form of the likelihood product and the
optimal
$\hat{\mu}$ as

$$
\hat{\mu} = \arg \min_{\mu} \sum \rho(x_i-\mu)
$$

 If $\rho$ is differentiable, then differentiating  this with respect
to $\mu$
gives

$$
\begin{equation}
\sum \psi(x_i-\hat{\mu}) = 0
\label{eq:muhat} \tag{5}
\end{equation}
$$

 with $\psi = \rho^\prime$, the first derivative of $\rho$ , and for technical
reasons we will assume that
$\psi$ is increasing. So far, it looks like we just
pushed some definitions
around, but the key idea is we want to consider general
$\rho$ functions that
may not be maximum likelihood estimators for *any*
distribution. Thus, our
focus is now on uncovering the nature of $\hat{\mu}$.

### Distribution of M-estimates

For a given distribution $F$, we define
$\mu_0=\mu(F)$ as the solution to the
following

$$
\mathbb{E}_F(\psi(x-\mu_0))= 0
$$

 It is technical to show, but it turns out that $\hat{\mu} \sim
\mathcal{N}(\mu_0,\frac{v}{n})$ with

$$
v =
\frac{\mathbb{E}_F(\psi(x-\mu_0)^2)}{(\mathbb{E}_F(\psi^\prime(x-\mu_0)))^2}
$$

 Thus, we can say that $\hat{\mu}$ is asymptotically normal with asymptotic
value $\mu_0$ and asymptotic variance $v$. This leads to the efficiency ratio
which is defined as  the following:

$$
\texttt{Eff}(\hat{\mu})= \frac{v_0}{v}
$$

 where $v_0$ is the asymptotic variance of the MLE and measures how
near
$\hat{\mu}$ is to the optimum. In other words, this provides a sense of
how much
outlier contamination costs in terms of samples. For example, if for
two
estimates with asymptotic variances $v_1$ and $v_2$, we have $v_1=3v_2$,
then
first estimate requires three times as many observations to obtain the
same
variance as the second. Furthermore, for the sample mean (i.e.,
$\hat{\mu}=\frac{1}{n} \sum X_i$) with $F=\mathcal{N}$, we have $\rho=x^2/2$
and
$\psi=x$ and also $\psi'=1$. Thus, we have $v=\mathbb{V}(x)$.
Alternatively,
using the sample median as the estimator for the location, we
have $v=1/(4
f(\mu_0)^2)$.  Thus, if we have $F=\mathcal{N}(0,1)$, for the
sample median, we
obtain $v={2\pi}/{4} \approx 1.571$. This means that the
sample median takes
approximately 1.6 times as many samples to obtain the same
variance for the
location as the sample mean. The sample median is
far more immune to the
effects of outliers than the sample mean, so this
gives a sense of how much
this robustness costs in samples.

** M-Estimates as Weighted Means.** One way
to think about M-estimates is a
weighted means. Operationally, this
means that
we want weight functions that can circumscribe the
influence of the individual
data points, but, when taken as a whole,
still provide good estimated
parameters. Most of the time, we have $\psi(0)=0$ and $\psi'(0)$ exists so
that
$\psi$ is approximately linear at the origin. Using the following
definition:

$$
W(x)  =  \begin{cases}
                \psi(x)/x & \text{if} \: x \neq 0 \\\
\psi'(x)  & \text{if} \: x =0
            \end{cases}
$$

 We can write our Equation [5](#eq:muhat) as follows:

$$
\begin{equation}
\sum W(x_i-\hat{\mu})(x_i-\hat{\mu}) = 0
\label{eq:Wmuhat}
\tag{6}
\end{equation}
$$

 Solving this for $\hat{\mu} $ yields the following,

$$
\hat{\mu} = \frac{\sum w_{i} x_i}{\sum w_{i}}
$$

 where $w_{i}=W(x_i-\hat{\mu})$. This is not practically useful
because the
$w_i$ contains $\hat{\mu}$, which is what we are trying to solve
for. The
question that remains is how to pick the $\psi$ functions. This is
still an open
question, but the Huber functions are a well-studied choice.

### Huber
Functions

The family of Huber functions is defined by the following:

$$
\rho_k(x ) = \begin{cases}
                x^2         & \mbox{if }  |x|\leq
k \\\
                2 k |x|-k^2 & \mbox{if }  |x| >  k
\end{cases}
$$

 with corresponding derivatives $2\psi_k(x)$ with

$$
\psi_k(x ) = \begin{cases}
                x              & \mbox{if } \: |x|
\leq k \\\
                \text{sgn}(x)k & \mbox{if } \: |x| >  k
\end{cases}
$$

 where the limiting cases $k \rightarrow \infty$ and $k \rightarrow 0$
correspond to the mean and median, respectively. To see this, take
$\psi_{\infty} = x$ and therefore $W(x) = 1$ and thus the defining Equation
[6](#eq:Wmuhat) results in

$$
\sum_{i=1}^{n} (x_i-\hat{\mu}) = 0
$$

 and then solving this leads to $\hat{\mu} = \frac{1}{n}\sum x_i$.
Note that
choosing $k=0$ leads to  the sample median, but that is not so
straightforward
to solve for. Nonetheless, Huber functions provide a way
to move between two
extremes of estimators for location (namely,
the mean vs. the median) with a
tunable parameter $k$.
The $W$ function corresponding to Huber's $\psi$ is the
following:

$$
W_k(x) = \min\Big{\lbrace} 1, \frac{k}{|x|} \Big{\rbrace}
$$

 [Figure](#fig:Robust_Statistics_0001) shows the Huber weight
function for $k=2$
with some sample points. The idea is that the computed
location, $\hat{\mu}$ is
computed from Equation [6](#eq:Wmuhat) to lie somewhere
in the middle of the
weight function so that those terms (i.e., *insiders*)
have their values fully
reflected in the location estimate. The black circles
are the *outliers* that
have their values attenuated by the weight function so
that only a fraction of
their presence is represented in the location estimate.

This shows the Huber weight function,
$W_2(x)$ and some cartoon data points that are insiders or outsiders as far as
the robust location estimate is concerned.

###
Breakdown Point

So far, our discussion of robustness has been very abstract.  A
more concrete
concept of robustness comes from the breakdown point.  In the
simplest terms,
the breakdown point describes what happens when a single data
point in an
estimator is changed in the most damaging way possible. For example,
suppose we
have the sample mean, $\hat{\mu}=\sum x_i/n$, and we take one of the
$x_i$
points to be infinite. What happens to this estimator? It also goes
infinite.
This means that the breakdown point of the estimator is 0%. On the
other hand,
the median has a breakdown point of 50%, meaning that half of the
data for
computing the median could go infinite without affecting the median
value. The median
is a *rank* statistic that cares more about the relative
ranking of the data
than the values of the data, which explains its robustness.
The simpliest but still formal way to express the breakdown point is to
take $n$
data points, $\mathcal{D} = \lbrace (x_i,y_i) \rbrace$. Suppose $T$
is a
regression estimator that yields a vector of regression coefficients,
$\boldsymbol{\theta}$,

$$
T(\mathcal{D}) = \boldsymbol{\theta}
$$

 Likewise, consider all possible corrupted samples of the data
$\mathcal{D}^\prime$. The maximum *bias* caused by this contamination is
the
following:

$$
\texttt{bias}_{m} = \sup_{\mathcal{D}^\prime} \Vert
T(\mathcal{D^\prime})-T(\mathcal{D}) \Vert
$$

 where the $\sup$ sweeps over all possible sets of $m$ contaminated samples.
Using this, the breakdown point is defined as the following:

$$
\epsilon_m = \min \Big\lbrace \frac{m}{n} \colon \texttt{bias}_{m}
\rightarrow \infty \Big\rbrace
$$

 For example, in our least-squares regression, even one point at
infinity causes
an infinite $T$. Thus, for least-squares regression,
$\epsilon_m=1/n$. In the
limit $n \rightarrow \infty$, we have $\epsilon_m
\rightarrow 0$.

###
Estimating Scale

In robust statistics, the concept of *scale* refers to a
measure of the
dispersion of the data. Usually, we use the
estimated standard
deviation for this, but this has a terrible breakdown point.
Even more
troubling, in order to get a good estimate of location, we have to
either
somehow know the scale ahead of time, or jointly estimate it. None of
these
methods have easy-to-compute closed form solutions and must be computed
numerically.

The most popular method for estimating scale is the *median
absolute deviation*

$$
\texttt{MAD} = \texttt{Med} (\vert \mathbf{x} -
\texttt{Med}(\mathbf{x})\vert)
$$

 In words, take the median of the data $\mathbf{x}$ and
then subtract that
median from the data itself, and then take the median of the
absolute value of
the result. Another good dispersion estimate is the *interquartile range*,

$$
\texttt{IQR} = x_{(n-m+1)} - x_{(n)}
$$

 where $m= [n/4]$. The $x_{(n)}$ notation means the $n^{th}$ data
element after
the data have been sorted. Thus, in this notation,
$\texttt{max}(\mathbf{x})=x_{(n)}$. In the case where $x \sim
\mathcal{N}(\mu,\sigma^2)$, then $\texttt{MAD}$ and $\texttt{IQR}$ are constant
multiples of $\sigma$ such that the normalized $\texttt{MAD}$ is the following,

$$
\texttt{MADN}(x) = \frac{\texttt{MAD} }{0.675}
$$

  The number comes from the inverse CDF of the normal distribution
corresponding
to the $0.75$ level. Given the complexity of the
calculations, *jointly*
estimating both location and scale is a purely
numerical matter. Fortunately,
the Statsmodels module has many of these
ready to use. Let's create some
contaminated data in the following code,

```python
import statsmodels.api as sm
import numpy as np

from scipy import stats
data=np.hstack([stats.norm(10,1).rvs(10),
                stats.norm(0,1).rvs(100)])
```

These data correspond to our model of contamination that we started
this
section with. As shown in the  histogram in
[Figure](#fig:Robust_Statistics_0002), there are two normal distributions, one
centered neatly at zero, representing the majority of the samples, and another
coming less regularly from the normal distribution on the right. Notice that
the
group of infrequent samples on the right separates the mean and median
estimates
(vertical dotted and dashed lines).  In the absence of the
contaminating
distribution on the right, the standard deviation for this data
should be close
to one. However, the usual non-robust estimate for standard
deviation (`np.std`)
comes out to approximately three.  Using the
$\texttt{MADN}$ estimator
(`sm.robust.scale.mad(data)`) we obtain approximately
1.25. Thus, the robust
estimate of dispersion is less moved by the presence of
the  contaminating
distribution.

Histogram of sample data. Notice that the group of infrequent samples on the
right separates the mean and median estimates indicated by the vertical
lines.

The generalized maximum likelihood M-estimation extends to
joint
scale and location estimation using Huber functions. For example,

```python
huber = sm.robust.scale.Huber()
loc,scl=huber(data)
```

which implements Huber's *proposal two* method of joint estimation of
location
and scale. This kind of estimation is the key ingredient to robust
regression
methods, many of which are implemented in Statsmodels in
`statsmodels.formula.api.rlm`. The corresponding documentation has more
information.


# Implementing Neural Networks with Numpy for Absolute Beginners - Part 1: Introduction

##### In this tutorial, you will get a brief understanding of what Neural Networks are and how they have been developed. In the end, you will gain a brief intuition as to how the network learns.

The field of Artificial Intelligence has gained a lot of popularity and momentum during the past 10 years, largely due to a huge increase in the computational capacity of computers with the use of GPUs and the availability of gigantic amounts of data. Deep Learning has become the buzzword everywhere!!
>>>>>

Although Artificial Intelligence (AI) resonates with the notion of the machines to think and behave impersonating humans, it is rather restricted to very nascent and small task-specific functions while the term Artificial General Intelligence (AGI) obliges to the terms of impersonating a human. Above these is the concept of Artificial Super Intelligence (ASI) which gives me the shrills as it represents intelligence of machines far exceeding human levels!!

The main concept for Artificial Intelligence currently holds that you have to train it before it learns to perform the task much like humans, except that here… you have to train it even for the simplest of the tasks like seeing and identifying objects!(This is surely a complex problem for our computers).

There are 3 situations that you can encounter in this domain:
1. When you have a lot of data...

> - Either your data is tagged, labelled, maintained or it is not.
 If the data is available and is fully labelled or tagged, you can train the model based on the given set of input-output pairs and ask the model to predict the output for a new set of data. This type of learning is called **Supervised Learning** (Since, you are giving the input and also mentioning that this is the correct output for the data).

Supervised Learning can be further divided into the two tasks as below:

> a. Classifcation - where you predict that the data belongs to a specific class. Eg.: Classfying a cat or a dog.

> b. Regression - where a real number value is predicted. Eg: Predicting the price of a house given it's dimensions.

>>In the below example, you can see that images are trained against their labels. You test the model by inputting an image and predicting it's class... like a cat.
>>>>>

> - When your data is unlabelled, the only option would be to let your model figure out by itself the patterns in the data. This is called **Unsupervised Learning**.

In the example shown below, you only provide the datapoints and the number of clusters(classes) that has to be formed and let the algorithm find out the best set of clusters.
>>>>>>>

> 2\. When you don't have data but instead have the environment itself to learn!

>Here, a learning agent is put in a predefined environment and made to learn by the actions it takes. It is either rewarded or punished based on its actions. This is the most interesting kind of learning and is also where a lot of exploration and research is happenning.It is called **Reinforcement Learning**.

As it can clearly be seen from the below image that the agent which is modelled as a person, learns to climb the wall through trial and error.
>>>>>>>

This tutorial focuses on Neural Networks which is a part of Supervised Learning.

## A little bit into the history of how Neural Networks evolved

The evolution of AI dates to back to 1950 when Alan Turing, the computer genius, came out with the Turing Test to distinguish between a Human and a Robot. He describes that when a machine performs so well, that we humans are not able to distinguish between the response given by a human and a machine, it has passed the Turing Test. Apparently this feat was achieved only in 2012, when a company named Vicarious cracked the captchas. Check out this video below on how Vicarious broke the captchas.

```python
#@title Vicarious Video
%%HTML

```

It must be noted that most of the Algorithms that were developed during that period(1950-2000) and now existing, are highly inspired by the working of our brain, the neurons and their structure with how they learn and transfer data. The most popular works include the Perceptron and the Neocognitron $-$(not covered in this article, but in a future article) based on which the Neural Networks have been developed.

Now, before you dive into what a perceptron is,  let's make sure you know a bit of all these... Although not necessarily required!

## Prerequisites

What you’ll need to know for the course:
1.   A little bit of Python &
2.   The eagerness to learn Neural Networks.

If you are unsure of which environment to use for implementing this, I recommend [Google Colab](https://colab.research.google.com/). The environment comes with many important packages already installed. Installing new packages and also importing and exporting the data is quite simple. Most of all, it also comes with GPU support. So go ahead and get coding with the platform!

Lastly, this article is directed for those who want to learn about Neural Networks or just Linear Regression. However, there would be an inclination towards Neural Networks!

## A biological Neuron

>>>>>

The figure above shows a biological neuron. It has *dendrites* that recieve information from neurons. The recieved information is passed on to the *cell body or the nucleus* of the neuron. The *nucleus* is where the information is processed. The processed information is passed on to the next layer of neurons through the *axons*.

Our brain consists of about 100 billion such neurons which communicate through electrochemical signals. Each neuron is connected to 100s and 1000s of other neurons which constantly transmit and recieve signals. When the sum of the signals recieved by a neuron exceeds a set threshold value, the cell is activated (although, it has been speculated that neurons use very complex activations to process the input data) and the signal is further transmitted to other neurons. You'll see that the artificial neuron or the perceptron adopts the same ideology to perform computation and transmit data in the next section.

You know that different regions of our brain are activated (/receptive) for different actions like seeing, hearing, creative thinking and so on. This is because the neurons belonging to a specific region in the brain are trained to process a certain kind of information better and hence get activated when only certain kinds of information is being sent.The figure below gives us a better understanding of the different receptive regions of the brain.

>>>>

It has also been shown through the concept of Neuroplasticity that the different regions of the brain can be rewired to perform totally different tasks. Such as the neurons responsible for touch sensing can be rewired to become sensitive to smell. Check out this great TEDx video below to know more about neuroplasticity.

Similarly, an artificial neuron/perceptron can be trained to recognize some of the most comlplex pattern. Hence, they can be called Universal Function Approximators.

In the next section, we'll explore the working of a perceptron and also gain a mathematical intuition.

```python
#@title Neuroplasticity
%%HTML

''
```

''

## Perceptron/Artificial Neuron

>>>>>>

From the figure, you can observe that the perceptron is a reflection of the biological neuron. The inputs combined with the weights($w_i$) are analogous to dendrties. These values are summed and passed through an activation function (like the thresholding function as shown in fig.). This is analogous to the nucleus. Finally, the activated value is transmitted to the next neuron/perceptron which is analogous to the axons.

The latent weights($w_i$) multiplied with each input($x_i$) depicts the significance of the respective input/feature. Larger the value of a weight, more important is the feature. Hence, the weights are what is learned in a perceptron so as to arrive at the required result. An additional bias($b$, here $w_0$) is also learned.

Hence, when there are multiple inputs (say n), the equation can be generalized as follows:
\begin{equation}
z=w_0+w_1.x_1+w_2.x_2+w_3.x_3+......+w_n.x_n \\
\therefore z=\sum_{i=0}^{n}w_i.x_i \qquad \text{where } x_0 = 1
\end{equation}

Finally, the output of summation (assume as $z$) is fed to the *thresholding activation function*, where the function outputs $ -1 \space \text{if } z < 0 \space \& \space 1 \space \text{if } z \geq 0$.

### An Example

Let us consider our perceptron to perform as *logic gates* to gain more intuition.

Let's choose an $AND \space gate$. The Truth Table for the $AND \space gate$ is shown below:

>>>>>>>>>

The perceptron for the $AND \space gate$ can be formed as shown in the figure. It is clear that the perceptron has two inputs (here $x1=A$ and $x2=B$)

>>>>>>>>>

\begin{equation}
\text{Threshold Function,} \qquad y = f(z) = \begin{cases}
1,& \text{if }z \geq 0.5\\
0,& \text{if } z< 0.5\\
\end{cases}
\end{equation}

We can see that for inputs $x1$, $x2$ & $x_0=1$, setting their weights as
\begin{equation}
w_0=-0.5, \\
w_1=0.6, \space \&\\
w_2=0.6
\end{equation}
respectively and keeping the *Threshold function* as the activation function we can arrive at the $AND \space Gate$.

Now, let's get our hands dirty and codify this and test it out!

```python
def and_perceptron(x1, x2):

    w0 = -0.5
    w1 = 0.6
    w2 = 0.6

    z = w0 + w1 * x1 + w2 * x2

    thresh = lambda x: 1 if x>= 0.5 else 0

    r = thresh(z)
    print(r)
```

```python
and_perceptron(1, 1)
```

    1

Similarly for $NOR \space Gate$ the Truth Table is,

>>>>>>>>>

The perceptron for $NOR \space Gate$ will be as below:

>>>>>>>>>

You can set the weights as
\begin{equation}
w_0 = 0.5 \\
w_1 = -0.6 \\
w_2 = -0.6
\end{equation}
so that you obtain a $NOR \space Gate$.

You can go ahead and implement this in code.

```python
def nor_perceptron(x1, x2):

    w0 = 0.5
    w1 = -0.6
    w2 = -0.6

    z = w0 + w1 * x1 + w2 * x2

    thresh = lambda x: 1 if x>= 0.5 else 0

    r = thresh(z)
    print(r)
```

```python
nor_perceptron(1, 1)
```

    0

Here, is the Truth Table for $NAND \space Gate$. Go ahead and guess the weights that fits the function and also implement in code.

>>>>>>>>>

## What you are actually calculating...

If you analyse what you were trying to do in the above examples, you will realize that you were actually trying to adjust the values of the weights to obtain the required output.

Lets consider the NOR Gate example and break it down to very miniscule steps to gain more understanding.

What you would usually do first is to simply set some values to the weights and observe the result, say

\begin{equation}
w_0 = 0.4 \\
w_1 = 0.7 \\
w_2 = -0.2
\end{equation}

Then the output will be as shown in below table:
>>>>>

So how can you fix the values of weights so that you get the right output?

By intuition, you can easily observe that $w_0$ must be increased and $w_1$ and $w_2$ must be reduced or rather made negative so that you obtain the actual output. But if you breakdown this intuition, you will observe that you are actually finding the difference between the actual output and the predicted output and finally reflecting that on the weights...

This is a very important concept that you will be digging deeper and will  be the core to formulate the ideas behind *gradient descent* and also *backward propagation*.

## Conclusion

In this tutorial you were introduced to the field of AI and went through an overview of perceptron. In the next tutorial, you'll learn to train a perceptron and do some predictions!!

NN with Numpy 1/Neural_Networks_for_Absolute_beginners_Part_1_Introduction.ipynb