# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
    print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [5.32907052e-17 7.04991621e-17 4.22578639e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
    X = np.random.randn(N, D1)
    a = np.maximum(0, X.dot(W1)).dot(W2)
    batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029235612572515e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\\vdots\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
\mu&=\frac{1}{N}\sum_{k=1}^N x_k\\
v&=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2\\
\sigma&=\sqrt{v+\epsilon}\\
y_i&=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  7.666703093525566e-13
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 1.97x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
    print('Running check with reg = ', reg)
    model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                              reg=reg, weight_scale=5e-2, dtype=np.float64,
                              normalization='batchnorm')

    loss, grads = model.loss(X, y)
    print('Initial loss: ', loss)

    for name in sorted(grads):
        f = lambda _: model.loss(X, y)[0]
        grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
        print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
    if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 3.92e-10
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 4.78e-11
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 7.57e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.29e-06
    W3 relative error: 1.11e-08
    b1 relative error: 5.55e-09
    b2 relative error: 5.55e-09
    b3 relative error: 2.23e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 3.48e-09
    gamma1 relative error: 5.94e-09
    gamma2 relative error: 4.14e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration     1 /   200) loss: 2.340974
    (Epoch   0 /  10) train acc: 0.107000; val_acc: 0.115000
    (Epoch   1 /  10) train acc: 0.314000; val_acc: 0.266000
    (Iteration    21 /   200) loss: 2.039365
    (Epoch   2 /  10) train acc: 0.385000; val_acc: 0.279000
    (Iteration    41 /   200) loss: 2.041103
    (Epoch   3 /  10) train acc: 0.495000; val_acc: 0.308000
    (Iteration    61 /   200) loss: 1.753902
    (Epoch   4 /  10) train acc: 0.532000; val_acc: 0.309000
    (Iteration    81 /   200) loss: 1.246584
    (Epoch   5 /  10) train acc: 0.574000; val_acc: 0.314000
    (Iteration   101 /   200) loss: 1.320589
    (Epoch   6 /  10) train acc: 0.633000; val_acc: 0.339000
    (Iteration   121 /   200) loss: 1.159471
    (Epoch   7 /  10) train acc: 0.686000; val_acc: 0.324000
    (Iteration   141 /   200) loss: 1.156478
    (Epoch   8 /  10) train acc: 0.770000; val_acc: 0.334000
    (Iteration   161 /   200) loss: 0.630540
    (Epoch   9 /  10) train acc: 0.803000; val_acc: 0.342000
    (Iteration   181 /   200) loss: 0.860585
    (Epoch  10 /  10) train acc: 0.779000; val_acc: 0.328000

    Solver without batch norm:
    (Iteration     1 /   200) loss: 2.302332
    (Epoch   0 /  10) train acc: 0.129000; val_acc: 0.131000
    (Epoch   1 /  10) train acc: 0.283000; val_acc: 0.250000
    (Iteration    21 /   200) loss: 2.041970
    (Epoch   2 /  10) train acc: 0.316000; val_acc: 0.277000
    (Iteration    41 /   200) loss: 1.900473
    (Epoch   3 /  10) train acc: 0.373000; val_acc: 0.282000
    (Iteration    61 /   200) loss: 1.713156
    (Epoch   4 /  10) train acc: 0.390000; val_acc: 0.310000
    (Iteration    81 /   200) loss: 1.662208
    (Epoch   5 /  10) train acc: 0.434000; val_acc: 0.300000
    (Iteration   101 /   200) loss: 1.696063
    (Epoch   6 /  10) train acc: 0.536000; val_acc: 0.346000
    (Iteration   121 /   200) loss: 1.550786
    (Epoch   7 /  10) train acc: 0.530000; val_acc: 0.310000
    (Iteration   141 /   200) loss: 1.436340
    (Epoch   8 /  10) train acc: 0.624000; val_acc: 0.348000
    (Iteration   161 /   200) loss: 0.998066
    (Epoch   9 /  10) train acc: 0.660000; val_acc: 0.332000
    (Iteration   181 /   200) loss: 0.943247
    (Epoch  10 /  10) train acc: 0.740000; val_acc: 0.345000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                    num_epochs=10, batch_size=50,
                    update_rule='adam',
                    optim_config={
                        'learning_rate': 1e-3,
                    },
                    verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                    num_epochs=10, batch_size=50,
                    update_rule='adam',
                    optim_config={
                        'learning_rate': 1e-3,
                    },
                    verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
    best_train_accs.append(max(solvers_ws[ws].train_acc_history))
    bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

    best_val_accs.append(max(solvers_ws[ws].val_acc_history))
    bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

    final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
    bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
The scale of weight initialization has a less affect on models with batch normalization, because they normalize the input activation.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
Too little batch size lead to low accuracy, because low batch size will make a inaccurate estimate.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
1 and 2 is analogous to layer normalization; 3 is analogous to batch normalization.

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_backward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [-4.81096644e-16  0.00000000e+00  7.40148683e-17 -5.55111512e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.433615657860454e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
Layer normalization does not work well when having a very small dimension of features, because layer normalization all units on one layer.


```python
import numpy as np
import sys
import matplotlib.pyplot as plt
import pandas as pd

from skimage import io
from scipy.interpolate import BSpline
from scipy.optimize import curve_fit
from scipy.signal import find_peaks
import sympy
from sympy import *
```

```python
import musicalrobot
from musicalrobot import edge_detection as ed
```

```python
frames = ed.input_file('../musicalrobot/data/PPA_Melting_6_14_19.tiff')
```

```python
plt.imshow(frames[0])
```

### Crop the input file if required to remove the noise and increase the accuracy of edge detection

```python
crop_frame = []
for frame in frames:
    crop_frame.append(frame[40:100])
plt.imshow(crop_frame[0])
```

```python
flip_frames = ed.flip_frame(crop_frame)
```

```python
labeled_samples = ed.edge_detection(flip_frames)
```

```python
plt.imshow(labeled_samples)
```

```python
regprops = ed.regprop(labeled_samples, flip_frames, 5, 1, 5)
```

```python
# Plotting the original image with the samples
# and centroid and plate location
plt.imshow(flip_frames[0])
plt.scatter(regprops[0]['Plate_coord'],regprops[0]['Row'],c='orange',s=6)
plt.scatter(regprops[0]['Column'],regprops[0]['Row'],s=6,c='red')
plt.title('Sample centroid and plate locations at which the temperature profile is monitored')
```

```python
s_temp,p_temp = ed.sample_temp(regprops,flip_frames)
```

```python
frame_len = np.linspace(1,len(p_temp[0]),len(p_temp[0]))
```

```python
temp = s_temp[0]
```

```python
bspl = BSpline(frame_len,temp,k=3)
```

```python
gradient_array = np.column_stack((frame_len,bspl(frame_len)))
```

```python
gradient = np.gradient(gradient_array,axis=0)
derivative = gradient[:,1]/gradient[:,0]
```

```python
plt.plot(frame_len,s_temp[0],label='Original temperature profile',linewidth=4,color='blue')
plt.plot(frame_len,bspl(frame_len),label='Spline fitting',color='yellow')
plt.legend()
```

```python
plt.scatter(frame_len,derivative)
plt.xlabel('Frames')
plt.ylabel('Derivative of sample temperature with respect to frames')
```

```python
plt.scatter(frame_len,s_temp[0],s=1.5)
plt.xlabel('Frames')
plt.ylabel('Temperature of the sample($^\circ$C)')
```

```python
plt.scatter(frame_len,p_temp[0],s=1.5)
plt.xlabel('Frames')
plt.ylabel('Temperature of the plate($^\circ$C)')
```

```python
# # Plotting the temperature profile of a sample against the temperature profile
# # of the plate at a location next to the sample.
# plt.scatter(p_temp[4],s_temp[4],s=2)
# plt.ylabel('Temperature of the sample($^\circ$C)')
# plt.xlabel('Temperature of the well plate($^\circ$C)')
# plt.title('Temperature of the sample against the temperature of the plate')
```

```python
# def inflection_point(s_temp,p_temp):
#     s_infl = []
#     p_infl = []
#     s_peaks = []
#     p_peaks = []
#     inf_peak = [] ; inf_temp = []
#     for temp in s_temp:
#         frames = np.linspace(1,len(temp),len(temp))
#         bspl = BSpline(frames,temp,k=3)
#         gradient_array = np.column_stack((frames,bspl(frames)))
#         gradient = np.gradient(gradient_array,axis=0)
#         derivative = gradient[:,1]/gradient[:,0]
#         peaks, properties = find_peaks(derivative,height=0.1)
#         max_height1 = np.max(properties['peak_heights'])
#         # To find the second highest peak
#         a = list(properties['peak_heights'])
#         a.remove(max_height1)
#         max_height2 = np.max(a)
#         inf_index1 = list(properties['peak_heights']).index(max_height1)
#         inf_index2 = list(properties['peak_heights']).index(max_height2)
#         s_peaks.append([peaks[inf_index1],peaks[inf_index2]])
#         s_infl.append([temp[peaks[inf_index1]],temp[peaks[inf_index2]]])
#     for temp in p_temp:
#         frames = np.linspace(1,len(temp),len(temp))
#         bspl = BSpline(frames,temp,k=3)
#         gradient_array = np.column_stack((frames,bspl(frames)))
#         gradient = np.gradient(gradient_array,axis=0)
#         derivative = gradient[:,1]/gradient[:,0]
#         peaks, properties = find_peaks(derivative,height=0.1)
#         max_height1 = np.max(properties['peak_heights'])
#         # To find the second highest peak
#         a = list(properties['peak_heights'])
#         a.remove(max_height1)
#         max_height2 = np.max(a)
#         inf_index1 = list(properties['peak_heights']).index(max_height1)
#         inf_index2 = list(properties['peak_heights']).index(max_height2)
#         p_peaks.append([peaks[inf_index1],peaks[inf_index2]])
#         p_infl.append([temp[peaks[inf_index1]],temp[peaks[inf_index2]]])
#     for i,peaks in enumerate(s_peaks):
#         for peak in peaks:
#             if peak - p_peaks[i][0] >= 3:
#                 inf_peak.append(peak)
#             else:
#                 pass
#     for i,temp in enumerate(s_temp):
#         inf_temp.append(temp[inf_peak[i]])
#     return inf_temp, s_peaks, p_peaks
```


```python
from IPython.core.display import HTML, Image
css_file = 'style.css'
HTML(open(css_file, 'r').read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Philosopher:400,700,400italic,700italic' rel='stylesheet' type='text/css'>

```python
from sympy import init_printing, symbols, Matrix
from warnings import filterwarnings
from IPython.display import Image
```

```python
init_printing(use_latex = 'mathjax')
filterwarnings('ignore')
```

# Graphs and networks
# Incidence matrices
# Kirchhoff's laws

* This lecture is about the application of matrices

## Graphs and networks

* In this instance we refer to nodes and there connections called edges
* Consider the graph below:

```python
Image(filename = 'Graph1.png')
```

* We will call the nodes *n* (columns), in this case *n* = 4
* The edges (connections) will be called *m* (rows), with *m* = 5 in this case
* This will give us a *m*×*n* = 5×4 matrix
* We will have to give a direction to every edge

## The incidence matrix

* This corresponds to the graph above

```python
A = Matrix([[-1, 1, 0, 0], [0, -1, 1, 0], [-1, 0, 1, 0], [-1, 0, 0, 1], [0, 0, -1, 1]])
A
# For each row (edge) look only at that edge (line)
# In the case of row (edge, line) 1, the arrow point away from node 1, hence the first -1 in the matrix
# The arrow point towards node 2, hence the 1
# It does not point to nodes 3 and 4, hence the 0's
```

$\displaystyle \left[\begin{matrix}-1 & 1 & 0 & 0\\0 & -1 & 1 & 0\\-1 & 0 & 1 & 0\\-1 & 0 & 0 & 1\\0 & 0 & -1 & 1\end{matrix}\right]$

* Edges 1, 2, and 3 form a loop
* Notice for the first loop (edges 1, 2, and 3) the corresponding third row is a linear combination of rows 1 and 2
* Intuitively, you can see that you can reach node 3 from node 1 by a combination of edges (rows) 1 and 2

```python
A.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & -1\\0 & 1 & 0 & -1\\0 & 0 & 1 & -1\\0 & 0 & 0 & 0\\0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1, \  2\right)\right)$

* We note that we have three pivot columns, hence a rank, *r* = 3
* We have one column without a pivot and will thus have one in the nullspace (*n* - *r* = 4 - 3 = 1)

```python
A.nullspace()
```

$\displaystyle \left[ \left[\begin{matrix}1\\1\\1\\1\end{matrix}\right]\right]$

*  The basis for this subspace is one dimensional and includes all scalar multiplications of this vector
* The meaning in our example is that nothing will happen when the solutions fall on this line in 4-dimensional space, i.e. no current will flow

* If you think of the solution **x** and every component of **x** being a potential at a node, the matrix multiplication A**x** gives you the potential differences along the edges
* The nullspace would then be the solution where all the potential differences are 0

```python
x1, x2, x3, x4 = symbols('x1, x2, x3, x4')
```

```python
x_vect = Matrix([x1, x2, x3, x4])
x_vect
```

$\displaystyle \left[\begin{matrix}x_{1}\\x_{2}\\x_{3}\\x_{4}\end{matrix}\right]$

```python
A * x_vect
```

$\displaystyle \left[\begin{matrix}- x_{1} + x_{2}\\- x_{2} + x_{3}\\- x_{1} + x_{3}\\- x_{1} + x_{4}\\- x_{3} + x_{4}\end{matrix}\right]$

* For the nullspace, each row now equals 0 (the potential difference between two nodes)

* Let's look at the row space and the nullspace of the row picture
* We now to get the rowspace by transposing the row that contain pivots

```python
A_row = Matrix([[1, 0, 0, -1], [0, 1, 0, -1], [0, 0, 1, -1]]).transpose()
A_row
```

$\displaystyle \left[\begin{matrix}1 & 0 & 0\\0 & 1 & 0\\0 & 0 & 1\\-1 & -1 & -1\end{matrix}\right]$

```python
A
```

$\displaystyle \left[\begin{matrix}-1 & 1 & 0 & 0\\0 & -1 & 1 & 0\\-1 & 0 & 1 & 0\\-1 & 0 & 0 & 1\\0 & 0 & -1 & 1\end{matrix}\right]$

```python
A.transpose()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & -1 & -1 & 0\\1 & -1 & 0 & 0 & 0\\0 & 1 & 1 & 0 & -1\\0 & 0 & 0 & 1 & 1\end{matrix}\right]$

```python
A.transpose().rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 1 & 0 & -1\\0 & 1 & 1 & 0 & -1\\0 & 0 & 0 & 1 & 1\\0 & 0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1, \  3\right)\right)$

* Note how the pivot columns are columns 1, 2, and 4
* These represent edges 1, 2, 4
* Note (form the graph above) that thye are independent as they are not a part of a loop
* A graph without a loop (with 1 less edge than nodes) is called a *tree*
* It has a nullspace of

```python
A.transpose().nullspace()
```

$\displaystyle \left[ \left[\begin{matrix}-1\\-1\\1\\0\\0\end{matrix}\right], \  \left[\begin{matrix}1\\1\\0\\-1\\1\end{matrix}\right]\right]$

* The dimension of the nullspace of AT is *m* - *r* = number of edges minus (number of nodes - 1)
* ∴ number of nodes - number of edges + number of loops = 1
* This is Euler's formula and works for all graphs
* It tells you how many independent loops there are

* There is a connection between potentials and currents
* With 5 edges we will have 5 currents, which we can represent as a vector **y**
$$ \overline { y } =\begin{bmatrix} { y }_{ 1 } & { y }_{ 2 } & { y }_{ 3 } & { y }_{ 4 } & { y }_{ 5 } \end{bmatrix} $$
* This relationship is Ohm's law

## Kirchhoff's law

* By the way, Kirchhoff's current law is: AT**y** = **0**
* We can look at it in the following way

```python
A.transpose()
```

$\displaystyle \left[\begin{matrix}-1 & 0 & -1 & -1 & 0\\1 & -1 & 0 & 0 & 0\\0 & 1 & 1 & 0 & -1\\0 & 0 & 0 & 1 & 1\end{matrix}\right]$

```python
y1, y2, y3, y4, y5 = symbols('y1, y2, y3, y4, y5')
```

```python
y_vect = Matrix([y1, y2, y3, y4, y5])
y_vect
```

$\displaystyle \left[\begin{matrix}y_{1}\\y_{2}\\y_{3}\\y_{4}\\y_{5}\end{matrix}\right]$

```python
A.transpose() * y_vect
```

$\displaystyle \left[\begin{matrix}- y_{1} - y_{3} - y_{4}\\y_{1} - y_{2}\\y_{2} + y_{3} - y_{5}\\y_{4} + y_{5}\end{matrix}\right]$

* For row 1 (setting it equal to 0 and looking at graph above tells us that current flows out from node 1 on all these 3 edges
* For row 2 (doing the same as above) we note that for node 2 current flow towards it on edge *y*1 and away from it along edge *y*2
* For row 3 we note that current flows from node three along edges 2 (edge *y*2) and 3 (edge *y*3) and away from it along edge 5 (edge *y*5)
* For row 4 we note that current flows towards it along edges 4 (edge *y*4) and 5 (edge *y*5)

* Look back at the nullspace of AT
* The two basis vectors show the flow in current that will allow for NO current to accumulate at a node
* In this example, current flowed along the loop of edges 1, 2, and 3 (with nothing along 4 and 5
* The other solution would be current flowing all along the periphery, with nothing along 3
* These are the basis vectors of the nullspace
* Another valid basis would include flow along the upper loop
* Notice that the basis is two dimensional as (between the 3 flows explained above) one is a linear combination of the other two

## Putting it all together

* All of the above can be stated as follows
$$ \overline {e} = {A} \overline {x} $$
$$ \overline {y} = {C} \overline {e} $$
$$ A^{ T }\overline { y } =\overline { f } $$
* Where
    * **e** is the potential differences
    * **f** is an external current in Kirchhoff's law
* This gives us the fundamental equation for applications as stated here
$$ {A}^{T}{C}{A} \overline{x}=\overline{f} $$
* These equations are for equilibrium (no Newton's law, no time)

* Remember that ATA is always symmetric

## Example problem

### Example problem 1

```python
Image(filename = 'Graph2.png')
```

* Calculate the incidence matrix A
* Calculate the nullspaces of A and AT
* Calculate the trace of ATA

#### Solution

```python
A = Matrix([[-1, 1, 0, 0, 0], [0, -1, 1, 0, 0], [-1, 0, 1, 0, 0], [0, -1, 0, 1, 0], [0, 0, 0, -1, 1], [0, 0, 1, 0, -1]])
A
```

$\displaystyle \left[\begin{matrix}-1 & 1 & 0 & 0 & 0\\0 & -1 & 1 & 0 & 0\\-1 & 0 & 1 & 0 & 0\\0 & -1 & 0 & 1 & 0\\0 & 0 & 0 & -1 & 1\\0 & 0 & 1 & 0 & -1\end{matrix}\right]$

```python
A.rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 0 & 0 & -1\\0 & 1 & 0 & 0 & -1\\0 & 0 & 1 & 0 & -1\\0 & 0 & 0 & 1 & -1\\0 & 0 & 0 & 0 & 0\\0 & 0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1, \  2, \  3\right)\right)$

* We note that we have 4 independent columns
* The dimension of the nullspace will be *n* - *r* = 5 - 4 = 1
* We will let *x*5 = *s*, then from the row-reduced echelon form abobe we have
$$ { x }_{ 1 }-{ x }_{ 5 }=0\\ { x }_{ 2 }-{ x }_{ 5 }=0\\ { x }_{ 3 }-{ x }_{ 5 }=0\\ { x }_{ 4 }-{ x }_{ 5 }=0\\ \begin{bmatrix} { x }_{ 1 } \\ { x }_{ 2 } \\ { x }_{ 3 } \\ { x }_{ 4 } \\ { x }_{ 5 } \end{bmatrix}=s\begin{bmatrix} 1 \\ 1 \\ 1 \\ 1 \\ 1 \end{bmatrix} $$

```python
A.nullspace()
```

$\displaystyle \left[ \left[\begin{matrix}1\\1\\1\\1\\1\end{matrix}\right]\right]$

* It represents a potential difference between all nodes t be zero: A**x** = **0**
* This means that the potential at all nodes must be a constant

```python
A.transpose().nullspace()
```

$\displaystyle \left[ \left[\begin{matrix}-1\\-1\\1\\0\\0\\0\end{matrix}\right], \  \left[\begin{matrix}0\\-1\\0\\1\\1\\1\end{matrix}\right]\right]$

* It is of dimension 2, as there are two independent loops
* As per Euler's formula
    * nodes - edges + loops = 1
    * 5 - 6 + 2 = 1
* This tells us about current that needs to flow so as not to accumulate current at a node
* It therefor indicates the independent loops
* It works out beautifully
    * Look at the two loops and assign flow as per the two vector columns for each edge and you will see perfect flow along either of the two independent loops with no current accumulating at any node

* We could calculate it from the row-reduced echelon for of AT

```python
A.transpose().rref()
```

$\displaystyle \left( \left[\begin{matrix}1 & 0 & 1 & 0 & 0 & 0\\0 & 1 & 1 & 0 & 0 & 1\\0 & 0 & 0 & 1 & 0 & -1\\0 & 0 & 0 & 0 & 1 & -1\\0 & 0 & 0 & 0 & 0 & 0\end{matrix}\right], \  \left( 0, \  1, \  3, \  4\right)\right)$

* This gives us 4 independent columns, with dependent *y*3 and *y*6
$$ y_{ 6 }=s\\ { y }_{ 3 }=t\\ { y }_{ 1 }+{ y }_{ 3 }={ y }_{ 1 }+t=0\\ \therefore \quad { y }_{ 1 }=-t\\ { y }_{ 2 }+{ y }_{ 3 }+{ y }_{ 6 }=\quad 0\\ \therefore \quad { y }_{ 2 }=-s-t\\ { y }_{ 4 }-{ y }_{ 6 }={ y }_{ 4 }-s=0\\ \therefore \quad { y }_{ 4 }=s\\ { y }_{ 5 }-{ y }_{ 6 }={ y }_{ 5 }-s=0\\ \therefore \quad { y }_{ 5 }=s\\ \begin{bmatrix} { y }_{ 1 } \\ { y }_{ 2 } \\ { y }_{ 3 } \\ { y }_{ 4 } \\ { y }_{ 5 } \\ { y }_{ 6 } \end{bmatrix}=\begin{bmatrix} -t \\ -s-t \\ t \\ s \\ s \\ s \end{bmatrix}=\begin{bmatrix} 0 \\ -s \\ 0 \\ s \\ s \\ s \end{bmatrix}+\begin{bmatrix} -t \\ -t \\ t \\ 0 \\ 0 \\ 0 \end{bmatrix}=s\begin{bmatrix} 0 \\ -1 \\ 0 \\ 1 \\ 1 \\ 1 \end{bmatrix}+t\begin{bmatrix} -1 \\ -1 \\ 1 \\ 0 \\ 0 \\ 0 \end{bmatrix} $$

```python
A.transpose() * A
```

$\displaystyle \left[\begin{matrix}2 & -1 & -1 & 0 & 0\\-1 & 3 & -1 & -1 & 0\\-1 & -1 & 3 & 0 & -1\\0 & -1 & 0 & 2 & -1\\0 & 0 & -1 & -1 & 2\end{matrix}\right]$

```python
(A.transpose() * A).trace()
```

$\displaystyle 12$

* The degree of the node is the number of edges it has
* Look at the columns of the incidence matrix A
* Every non-trivial (non-zero) entry represents an edge
* Note that there are 2 in column 1
    * This gives us a degree of 2, which will also be the first entry on the diagonal of ATA
* Column 2 has 3 entries representing 3 edges from node 2 and an entry of 3 on the diagonal of ATA
* ... and so on
* The trace is therefor just the sum of the degree of all the nodes

```python

```

Mathematics/Linear Algebra/0.0 MIT-18.06 - Jupyter/Lecture_12_Graphs_Incidence_matrices_Kirchhoff_laws.ipynb


```python
from IPython.display import Image
Image('../../../python_for_probability_statistics_and_machine_learning.jpg')
```

[Python for Probability, Statistics, and Machine Learning](https://www.springer.com/fr/book/9783319307152)

```python
from __future__ import division
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

Linear regression gets to the heart of statistics: Given a set of data points,
what is the relationship of the data in-hand to data yet seen?
How should information from one data set propagate to other data? Linear
regression offers the following model to address this question:

$$
\mathbb{E}(Y|X=x) \approx a x + b
$$

 That is, given specific values for $X$, assume that the conditional
expectation is a linear function of those specific values.  However, because
the observed values are not the expectations themselves, the model accommodates
this with an additive noise term. In other words, the observed variable (a.k.a.
response, target, dependent variable) is modeled as,

$$
\mathbb{E}(Y| X=x_i) + \epsilon_i \approx a x + b+ \epsilon_i=y
$$

 where $\mathbb{E}(\epsilon_i)=0$ and the $\epsilon_i$ are iid and
where the distribution function of $\epsilon_i$ depends on the problem, even
though it is often assumed Gaussian. The $X=x$ values are known as independent
variables, covariates, or regressors.

Let's see if we can use all of the methods we have developed so far to
understand this form of regression. The first task is to determine how to
estimate the unknown linear parameters, $a$ and $b$. To make this concrete,
let's assume that $\epsilon \sim \mathcal{N}(0,\sigma^2)$.  Bear in mind that
$\mathbb{E}(Y|X=x)$ is a deterministic function of $x$.  In other words, the
variable $x$ changes with each draw, but after the data have been collected
these are no longer random quantities. Thus, for fixed $x$, $y$ is a random
variable generated by $\epsilon$.  Perhaps we should denote $\epsilon$ as
$\epsilon_x$ to emphasize this, but because $\epsilon$ is an independent,
identically-distributed (iid) random variable at each fixed $x$, this would be
excessive.  Because of Gaussian additive noise, the
distribution of $y$ is completely characterized by its mean and variance.

$$
\mathbb{E}(y) = a x  + b
$$

$$
\
\mathbb{V}(y) = \sigma^2
$$

 Using the maximum likelihood procedure, we write
out the log-likelihood  function as

$$
\mathcal{L}(a,b)  = \sum_{i=1}^n \log \mathcal{N}(a x_i +b , \sigma^2) \propto \frac{1}{2 \sigma^2}\sum_{i=1}^n (y_i-a x_i-b)^2
$$

 Note that we suppressed  the terms that are irrelevent to
the maximum-finding. Taking the derivative of this with respect to $a$ gives
the following equation:

$$
\frac{\partial \mathcal{L}(a,b)}{\partial a}= 2 \sum_{i=1}^n x_i (b+ a x_i -y_i) =0
$$

 Likewise, we do the same for the $b$ parameter

$$
\frac{\partial \mathcal{L}(a,b)}{\partial b}=2\sum_{i=1}^n (b+a x_i-y_i) =0
$$

 The following code simulates some data and uses Numpy tools to
compute the parameters as shown,

```python
import numpy as np
a = 6;b = 1 # parameters to estimate
x = np.linspace(0,1,100)
y = a*x + np.random.randn(len(x))+b
p,var_=np.polyfit(x,y,1,cov=True) # fit data to line
y_ = np.polyval(p,x) # estimated by linear regression
```

```python
%matplotlib inline

from matplotlib.pylab import subplots
fig,axs=subplots(1,2)
fig.set_size_inches((9,3.5))
_=ax =axs[0]
_=ax.plot(x,y,'o',alpha=.3,color='gray',ms=10)
_=ax.plot(x,y_,color='k',lw=3)
_=ax.set_xlabel("$x$",fontsize=22)
_=ax.set_ylabel("$y$",fontsize=22)
_=ax.set_title("Linear regression; a=%3.3g b=%3.3g"%(p[0],p[1]))
_=ax = axs[1]
_=ax.hist(y_-y,color='gray')
_=ax.set_xlabel(r"$\Delta y$",fontsize=22)
fig.tight_layout()
#fig.savefig('fig-statistics/Regression_001.png')
```

The panel on the left shows the data and regression line. The panel on the right shows a histogram of the regression errors.

 The graph on the left of [Figure](#fig:Regression_001)
shows the regression line plotted against the data. The estimated
parameters are noted in the title.  The histogram on the right of
[Figure](#fig:Regression_001) shows the residual errors in the model.
It is always a good idea to inspect the residuals of any regression
for normality. These are the differences between the fitted line for
each $x_i$ value and the corresponding $y_i$ value in the data.
Note that the $x$ term does not have to be uniformly monotone.

To decouple the deterministic variation from the random variation, we can fix
the index and write separate problems of the form

$$
y_i = a x_i +b + \epsilon_i
$$

 where $\epsilon_i \sim \mathcal{N}(0,\sigma^2)$. What could we do with
just this one component of the problem? In other words, suppose we had
$m$-samples of this component as in $\lbrace y_{i,k}\rbrace_{k=1}^m$. Following
the usual procedure, we could obtain estimates of the mean of $y_i$ as

$$
\hat{y_i} = \frac{1}{m}\sum_{k=1}^m y_{i,k}
$$

 However, this tells us nothing about the individual parameters $a$
and $b$ because they are not separable in the terms that are computed, namely,
we may have

$$
\mathbb{E}(y_i) = a x_i +b
$$

 but we still only have one equation and the two unknowns, $a$ and
$b$. How about if we consider and fix another component $j$ as in

$$
y_j = a x_j +b + \epsilon_i
$$

 Then, we have

$$
\mathbb{E}(y_j) = a x_j +b
$$

 so at least now we have two equations and two unknowns and we know how to
estimate the left hand sides of these equations from the data using the
estimators $\hat{y_i}$ and  $\hat{y_j}$. Let's see how this works in the code
sample below.

```python
x0, xn =x[0],x[80]
# generate synthetic data
y_0 = a*x0 + np.random.randn(20)+b
y_1 = a*xn + np.random.randn(20)+b
# mean along sample dimension
yhat = np.array([y_0,y_1]).mean(axis=1)
a_,b_=np.linalg.solve(np.array([[x0,1],
                                [xn,1]]),yhat)
```

The fitted and true lines are plotted with the data values. The squares at either end of the solid line show the mean value for each of the data groups shown.

**Programming Tip.**

The prior code uses the `solve` function in the Numpy `linalg` module, which
contains the core linear algebra codes in Numpy that incorporate the
battle-tested LAPACK library.

 We can write out the solution for the estimated parameters for this
case where $x_0 =0$

$$
\hat{a} = \frac{\hat{y}_i - \hat{y}_0}{x_i}
$$

$$
\
\hat{b} = \hat{y_0}
$$

$$
\
$$

 The expectations and variances of these estimators are the following,

$$
\mathbb{E}(\hat{a}) = \frac{a x_i }{x_i}=a
$$

$$
\
\mathbb{E}(\hat{b}) =b
$$

$$
\
\mathbb{V}(\hat{a}) = \frac{2 \sigma^2}{x_i^2}
$$

$$
\
\mathbb{V}(\hat{b}) = \sigma^2
$$

 The expectations show that the estimators are unbiased.  The
estimator  $\hat{a}$ has a variance that decreases as larger points $x_i$ are
selected. That is, it is better to have samples further out along the
horizontal axis for fitting the line.  This variance quantifies the *leverage*
of those distant points.

**Regression From Projection Methods.** Let's see if we can apply our knowledge
of projection methods to the general case.  In vector notation, we can write
the following:

$$
\mathbf{y} = a \mathbf{x} + b\mathbf{1} + \boldsymbol{\epsilon}
$$

 where $\mathbf{1}$ is the vector of all ones.
Let's use the inner-product notation,

$$
\langle \mathbf{x},\mathbf{y} \rangle = \mathbb{E}(\mathbf{x}^T \mathbf{y})
$$

 Then, by taking the inner-product with some $\mathbf{x}_1 \in
\mathbf{1}^\perp$ we obtain [^perp],

[^perp]: The space of all vectors, $\mathbf{a}$ such that $\langle
\mathbf{a},\mathbf{1} \rangle = 0$ is denoted $\mathbf{1}^\perp$.

$$
\langle \mathbf{y},\mathbf{x}_1  \rangle = a \langle \mathbf{x},\mathbf{x}_1 \rangle
$$

 Recall that $\mathbb{E}(\boldsymbol{\epsilon})=\mathbf{0}$.  We
can finally solve for $a$ as

$$
\begin{equation}
\hat{a} = \frac{\langle\mathbf{y},\mathbf{x}_1 \rangle}{\langle \mathbf{x},\mathbf{x}_1 \rangle}
\end{equation}
\label{eq:ahat} \tag{1}
$$

 That was pretty neat but now we have the mysterious $\mathbf{x}_1$
vector.  Where does this come from?  If we project $\mathbf{x}$ onto the
$\mathbf{1}^\perp$, then we get the MMSE  approximation to $\mathbf{x}$ in the
$\mathbf{1}^\perp$ space. Thus, we take

$$
\mathbf{x}_1 = P_{\mathbf{1}^\perp} (\mathbf{x})
$$

 Remember that $P_{\mathbf{1}^\perp} $ is a projection matrix so the length of
$\mathbf{x}_1$ is at most $\mathbf{x}$. This means that the denominator in
the $\hat{a}$ equation above is really just the length of the $\mathbf{x}$
vector in the coordinate system of $P_{\mathbf{1}^\perp} $. Because the
projection is orthogonal (namely, of minimum length), the Pythagorean theorem
gives this length as the following:

$$
\langle \mathbf{x},\mathbf{x}_1 \rangle ^2=\langle \mathbf{x},\mathbf{x} \rangle- \langle\mathbf{1},\mathbf{x} \rangle^2
$$

 The first term on the right is the length of the $\mathbf{x}$ vector
and last term is the length of $\mathbf{x}$ in the coordinate system orthogonal
to $P_{\mathbf{1}^\perp} $, namely that of $\mathbf{1}$.  We
can use this geometric interpretation to understand what is going on in
typical linear regression in much more detail. The fact that the denominator is
the orthogonal projection of $\mathbf{x}$ tells us that the choice of
$\mathbf{x}_1$ has the strongest effect (i.e., largest value) on reducing
the variance of $\hat{a}$. That is, the more $\mathbf{x}$ is aligned with
$\mathbf{1}$, the worse the variance of $\hat{a}$. This makes intuitive sense
because the closer $\mathbf{x}$ is to $\mathbf{1}$, the more constant it is,
and we have already seen from our one-dimensional example that distance between
the $x$ terms pays off in reduced variance. We already know that $\hat{a}$
is an unbiased estimator and because we chose $\mathbf{x}_1$ deliberately as a
projection, we know that it is also of minimum variance. Such estimators are
known as Minimum-Variance Unbiased Estimators (MVUE).

In the same spirit, let's examine the numerator of $\hat{a}$ in Equation ref{eq:ahat}. We can write
$\mathbf{x}_{1}$ as the following

$$
\mathbf{x}_{1} = \mathbf{x} -  P_{\mathbf{1}} \mathbf{x}
$$

 where $P_{\mathbf{1}}$ is projection matrix  of $\mathbf{x}$ onto the
$\mathbf{1}$ vector. Using this, the numerator of $\hat{a}$ becomes

$$
\langle \mathbf{y}, \mathbf{x}_1\rangle  =\langle \mathbf{y}, \mathbf{x}\rangle -\langle \mathbf{y}, P_{\mathbf{1}} \mathbf{x}\rangle
$$

 Note that,

$$
P_{\mathbf{1}}  = \mathbf{1} \mathbf{1}^T \frac{1}{n}
$$

 so that writing this out explicitly gives

$$
\langle \mathbf{y}, P_{\mathbf{1}} \mathbf{x}\rangle = \left(\mathbf{y}^T \mathbf{1}\right) \left(\mathbf{1}^T \mathbf{x}\right)/n =\left(\sum y_i\right)\left(\sum x_{i}\right)/n
$$

 and similarly, we have the following for the denominator:

$$
\langle \mathbf{x}, P_{\mathbf{1}} \mathbf{x}\rangle = \left(\mathbf{x}^T \mathbf{1}\right) \left(\mathbf{1}^T \mathbf{x}\right)/n =\left(\sum x_i\right)\left(\sum x_{i}\right)/n
$$

 So, plugging all of this together gives the following,

$$
\hat{a} = \frac{\mathbf{x}^T\mathbf{y}-(\sum x_i)(\sum y_i)/n}{\mathbf{x}^T \mathbf{x}  -(\sum x_i)^2/n}
$$

 with corresponding variance,

$$
\mathbb{V}(\hat{a}) = \sigma^2 \frac{\|\mathbf{x}_1\|^2}{\langle\mathbf{x},\mathbf{x}_1\rangle^2}
$$

$$
\
                    = \frac{\sigma^2}{\Vert \mathbf{x}\Vert^2-n(\overline{x}^2)}
$$

 Using the same approach with $\hat{b}$ gives,

$$
\begin{equation}
\hat{b}  = \frac{\langle \mathbf{y},\mathbf{x}^{\perp} \rangle}{\langle \mathbf{1},\mathbf{x}^{\perp}\rangle}
\label{_auto1} \tag{2}
\end{equation}
$$

$$
\begin{equation} \
         = \frac{\langle \mathbf{y},\mathbf{1}-P_{\mathbf{x}}(\mathbf{1})\rangle}{\langle \mathbf{1},\mathbf{1}-P_{\mathbf{x}}(\mathbf{1})\rangle}
\label{_auto2} \tag{3}
\end{equation}
$$

$$
\begin{equation} \
         = \frac{\mathbf{x}^T \mathbf{x}(\sum y_i)/n -\mathbf{x}^T\mathbf{y}(\sum x_i)/n}{\mathbf{x}^T \mathbf{x} -(\sum x_i)^2/n}
\label{_auto3} \tag{4}
\end{equation}
$$

 where

$$
P_{\mathbf{x}} = \frac{\mathbf{\mathbf{x} \mathbf{x}^T}}{\| \mathbf{x} \|^2}
$$

 with variance

$$
\mathbb{V}(\hat{b})=\sigma^2 \frac{\langle \boldsymbol{\mathbf{1}-P_{\mathbf{x}}(\mathbf{1})},\boldsymbol{\mathbf{1}-P_{\mathbf{x}}(\mathbf{1})}\rangle}{\langle \mathbf{1},\boldsymbol{\mathbf{1}-P_{\mathbf{x}}(\mathbf{1})}\rangle^2}
$$

$$
\
                   =\frac{\sigma^2}{n-\frac{(n\overline{x})^2}{\Vert\mathbf{x}\Vert^2}}
$$

**Qualifying the Estimates.** Our formulas for the variance above include the
unknown $\sigma^2$, which we must estimate from the data itself using our
plug-in estimates.  We can form the residual sum of squares as

$$
\texttt{RSS} = \sum_i (\hat{a} x_i + \hat{b} - y_i)^2
$$

 Thus, the estimate of $\sigma^2$ can be expressed as

$$
\hat{\sigma}^2 = \frac{\texttt{RSS}}{n-2}
$$

 where $n$ is the number of samples. This is also known as the
*residual mean square*. The $n-2$  represents the *degrees of freedom* (`df`).
Because we estimated two parameters from the same data we have $n-2$ instead of
$n$. Thus, in general, $\texttt{df} = n - p$, where $p$ is the number of
estimated parameters.  Under the assumption that the noise is Gaussian,  the
$\texttt{RSS}/\sigma^2$ is chi-squared distributed with $n-2$ degrees of
freedom. Another important term is the *sum of squares about the mean*, (a.k.a
*corrected* sum of squares),

$$
\texttt{SYY} = \sum (y_i - \bar{y})^2
$$

 The $\texttt{SYY}$ captures the idea of not using the $x_i$ data and
just using the mean of the $y_i$ data to estimate $y$. These two terms lead
to the $R^2$ term,

$$
R^2=1-\frac{\texttt{RSS}}{ \texttt{SYY} }
$$

 Note that for perfect regression, $R^2=1$.  That is, if the
regression gets each $y_i$ data point exactly right, then
$\texttt{RSS}=0$ this term equals one.  Thus, this term is used to
measure of goodness-of-fit. The `stats` module in `scipy` computes
many of these terms automatically,

```python
from scipy import stats
slope,intercept,r_value,p_value,stderr = stats.linregress(x,y)
```

 where the square of the `r_value` variable is the $R^2$ above. The
computed p-value is the two-sided hypothesis test with a null hypothesis that
the slope of the line is zero. In other words, this tests whether or not the
linear regression  makes sense for the data for that hypothesis.  The
Statsmodels module provides a powerful extension to Scipy's stats module by
making it easy to do regression and keep track of these parameters. Let's
reformulate our problem using the Statsmodels framework by creating
a Pandas dataframe for the data,

```python
import statsmodels.formula.api as smf
from pandas import DataFrame
import numpy as np
d = DataFrame({'x':np.linspace(0,1,10)}) # create data
d['y'] = a*d.x+ b + np.random.randn(*d.x.shape)
```

 Now that we have  the input data in the above
Pandas dataframe, we can perform the regression as in the following,

```python
results = smf.ols('y ~ x', data=d).fit()
```

 The $\sim$ symbol is notation for $y = a x + b + \epsilon$, where the
constant $b$ is implicit in this usage of Statsmodels. The names in the string
are taken from the columns in the dataframe. This makes it very easy to build
models with complicated interactions between the named columns in the
dataframe. We can examine a report of the model fit by looking at the summary,

```python
print results.summary2()
```

                     Results: Ordinary least squares
    =================================================================
    Model:              OLS              Adj. R-squared:     0.862
    Dependent Variable: y                AIC:                27.3310
    Date:               2016-06-06 17:18 BIC:                27.9361
    No. Observations:   10               Log-Likelihood:     -11.665
    Df Model:           1                F-statistic:        57.12
    Df Residuals:       8                Prob (F-statistic): 6.56e-05
    R-squared:          0.877            Scale:              0.75455
    -------------------------------------------------------------------
                  Coef.    Std.Err.     t      P>|t|    [0.025   0.975]
    -------------------------------------------------------------------
    Intercept     1.3192     0.5106   2.5838   0.0324   0.1419   2.4965
    x             6.5051     0.8607   7.5578   0.0001   4.5203   8.4899
    -----------------------------------------------------------------
    Omnibus:               1.923        Durbin-Watson:          2.608
    Prob(Omnibus):         0.382        Jarque-Bera (JB):       0.439
    Skew:                  -0.505       Prob(JB):               0.803
    Kurtosis:              3.185        Condition No.:          4
    =================================================================

    d:\Miniconda64\Lib\site-packages\scipy\stats\stats.py:1277: UserWarning: kurtosistest only valid for n>=20 ... continuing anyway, n=10
      "anyway, n=%i" % int(n))

                 Results: Ordinary least squares
=================================================================
Model:              OLS              Adj. R-squared:     0.808
Dependent Variable: y                AIC:                28.1821
Date:               0000-00-00 00:00 BIC:                00.0000
No. Observations:   10               Log-Likelihood:     -12.091
Df Model:           1                F-statistic:        38.86
Df Residuals:       8                Prob (F-statistic): 0.000250
R-squared:          0.829            Scale:              0.82158
-------------------------------------------------------------------
              Coef.    Std.Err.     t      P>|t|    [0.025   0.975]
-------------------------------------------------------------------
Intercept     1.5352     0.5327   2.8817   0.0205   0.3067   2.7637
x             5.5990     0.8981   6.2340   0.0003   3.5279   7.6701
 There is a lot more here than we have discussed so far, but the
Statsmodels documentation is the best place to go for complete information
about this report. The F-statistic attempts to capture the contrast between
including the slope parameter or leaving it off. That is, consider two
hypotheses:

$$
H_0 \colon  \mathbb{E}(Y|X=x) = b
$$

$$
\
H_1 \colon  \mathbb{E}(Y|X=x) = b + a x
$$

 In order to quantify how much better adding the slope term is for
the regression, we compute  the following:

$$
F = \frac{\texttt{SYY} - \texttt{RSS}}{ \hat{\sigma}^2 }
$$

 The numerator computes the difference in the residual squared errors
between including the slope in the regression or just using the mean of the
$y_i$ values. Once again, if we assume (or can claim asymptotically) that the
$\epsilon$ noise term is Gaussian, $\epsilon \sim \mathcal{N}(0,\sigma^2)$,
then the $H_0$ hypothesis will follow an F-distribution [^fdist] with degrees of
freedom from the numerator and denominator. In this case, $F \sim F(1,n-2)$.
The value of this statistic is reported by Statsmodels above. The corresponding
reported probability shows the chance of $F$ exceeding its computed value  if
$H_0$ were true. So, the take-home message from all this is that including the
slope leads to a much smaller reduction in squared error than could be expected
from a favorable draw of $n$ points of this data, under the Gaussian additive
noise assumption. This is evidence that including the slope is meaningful for
this data.

[^fdist]: The $F(m,n)$ F-distribution has two integer degree-of-freedom parameters, $m$
and $n$.

The Statsmodels report also shows the adjusted $R^2$ term.
This is a correction to the $R^2$ calculation that accounts
for the number of parameters $p$ that the regression is
fitting and the sample size $n$,

$$
\texttt{Adjusted } R^2 = 1- \frac{\texttt{RSS}/(n-p)}{\texttt{SYY}/(n-1)}
$$

 This is always lower than $R^2$ except when $p=1$ (i.e., estimating
only $b$). This becomes a better way to compare regressions when one is
attempting to fit many parameters with comparatively small $n$.

**Linear Prediction.**  Using linear regression for prediction introduces
some other issues. Recall the following expectation,

$$
\mathbb{E}(Y|X=x) \approx \hat{a} x + \hat{b}
$$

 where we have determined $\hat{a}$  and $\hat{b}$  from the data.
Given a new point of interest, $x_p$, we would certainly compute

$$
\hat{y}_p = \hat{a} x_p + \hat{b}
$$

 as the predicted value for $\hat{ y_p }$. This is the same as saying
that our best prediction for $y$ based on $x_p$ is the above conditional
expectation. The variance for this is the following,

$$
\mathbb{V}(y_p) = x_p^2 \mathbb{V}(\hat{a}) +\mathbb{V}(\hat{b})+2 x_p \texttt{cov}(\hat{a}\hat{b})
$$

 Note that we have the covariance above because $\hat{a}$ and
$\hat{b}$ are derived from the same data.  We can work this out below using
our previous notation from ref{eq:ahat},

$$
\texttt{cov}(\hat{a}\hat{b})=\frac{\mathbf{x}_1^T \mathbb{V}\lbrace\mathbf{y}\mathbf{y}^T\rbrace\mathbf{x}^{\perp}}{(\mathbf{x}_1^T \mathbf{x})(\mathbf{1}^T \mathbf{x}^{\perp})} = \frac{\mathbf{x}_1^T \sigma^2\mathbf{I}\mathbf{x}^{\perp}}{(\mathbf{x}_1^T \mathbf{x})(\mathbf{1}^T \mathbf{x}^{\perp})}
$$

$$
\
        =\sigma^2\frac{\mathbf{x}_1^T\mathbf{x}^{\perp}}{(\mathbf{x}_1^T \mathbf{x})(\mathbf{1}^T \mathbf{x}^{\perp})} = \sigma^2\frac{\left(\mathbf{x}-P_1\mathbf{x}\right)^T\mathbf{x}^{\perp}}{(\mathbf{x}_1^T \mathbf{x})(\mathbf{1}^T \mathbf{x}^{\perp})}
$$

$$
\
        =\sigma^2\frac{-\mathbf{x}^T P_1^T\mathbf{x}^{\perp}}{(\mathbf{x}_1^T \mathbf{x})(\mathbf{1}^T \mathbf{x}^{\perp})} = \sigma^2\frac{-\mathbf{x}^T\frac{1}{n}\mathbf{1} \mathbf{1}^T\mathbf{x}^{\perp}}{(\mathbf{x}_1^T \mathbf{x})(\mathbf{1}^T \mathbf{x}^{\perp})}
$$

$$
\
        =\sigma^2\frac{-\mathbf{x}^T\frac{1}{n}\mathbf{1}}{(\mathbf{x}_1^T \mathbf{x})} = \frac{-\sigma^2\overline{x}}{\sum_{i=1}^n(x_i^2-\overline{x}^2)}
$$

$$
\
$$

 After plugging all this in, we obtain the following,

$$
\mathbb{V}(y_p)=\sigma^2 \frac{x_p^2-2 x_p\overline{x}+\Vert \mathbf{x}\Vert^2/n}{\Vert\mathbf{x}\Vert^2-n\overline{x}^2}
$$

 where, in practice, we use the plug-in estimate for
the  $\sigma^2$.

There is an important consequence for the confidence interval for
$y_p$. We cannot simply use the square root of $\mathbb{V}(y_p)$
to form the confidence interval because the model includes the
extra $\epsilon$ noise term. In particular, the parameters were
computed using a set of statistics from the data, but now must
include different realizations for the noise term for the
prediction part.  This means we have to compute

$$
\eta^2=\mathbb{V}(y_p)+\sigma^2
$$

 Then, the 95\% confidence interval $y_p \in
(y_p-2\hat{\eta},y_p+2\hat{\eta})$ is the following,

$$
\mathbb{P}(y_p-2\hat{\eta}< y_p < y_p+2\hat{\eta})\approx\mathbb{P}(-2<\mathcal{N}(0,1)<2) \approx 0.95
$$

 where $\hat{\eta}$ comes from substituting the
plug-in estimate for $\sigma$.

## Extensions to Multiple Covariates

With all the machinery we have, it is a short notational hop to consider
multiple regressors as in the following,

$$
\mathbf{Y} = \mathbf{X} \boldsymbol{\beta} +\boldsymbol{\epsilon}
$$

 with the usual $\mathbb{E}(\boldsymbol{\epsilon})=\mathbf{0}$ and
$\mathbb{V}(\boldsymbol{\epsilon})=\sigma^2\mathbf{I}$. Thus, $\mathbf{X}$ is a
$n \times p$ full rank matrix of regressors and $\mathbf{Y}$ is the $n$-vector
of observations. Note that the constant term has been incorporated into
$\mathbf{X}$ as a column of ones.  The corresponding estimated solution for
$\boldsymbol{\beta}$  is the following,

$$
\hat{ \boldsymbol{\beta} } = (\mathbf{X}^T \mathbf{X})^{-1} \mathbf{X}^T \mathbf{Y}
$$

 with corresponding variance,

$$
\mathbb{V}(\hat{\boldsymbol{\beta}})=\sigma^2(\mathbf{X}^T \mathbf{X})^{-1}
$$

 and with the assumption of Gaussian errors, we have

$$
\hat{\boldsymbol{\beta}}\sim \mathcal{N}(\boldsymbol{\beta}, \sigma^2(\mathbf{X}^T \mathbf{X})^{-1})
$$

 The unbiased estimate of $\sigma^2$ is the following,

$$
\hat{\sigma}^2 = \frac{1}{n-p}\sum \hat{\epsilon}_i^2
$$

 where $\hat{ \boldsymbol{\epsilon}}=\mathbf{X}\hat{\boldsymbol{\beta}} -\mathbf{Y}$
is the vector of residuals. Tukey christened the following matrix as the *hat*
matrix (a.k.a. influence matrix),

$$
\mathbf{V}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T
$$

 because it maps $\mathbf{Y}$ into $\hat{ \mathbf{Y} }$,

$$
\hat{ \mathbf{Y} } = \mathbf{V} \mathbf{Y}
$$

 As an exercise you can check that $\mathbf{V}$ is a projection
matrix. Note that that matrix is solely a function of $\mathbf{X}$. The
diagonal elements of $\mathbf{V}$ are called the *leverage values* and are
contained in the closed interval $[1/n,1]$. These terms measure of distance
between the values of $x_i$ and the mean values over the $n$ observations.
Thus, the leverage terms depend only on $\mathbf{X}$. This is the
generalization of our initial discussion of leverage where we had multiple
samples at only two $x_i$ points. Using the hat matrix, we can compute the
variance of each residual, $e_i = \hat{y}-y_i$ as

$$
\mathbb{V}(e_i) = \sigma^2 (1-v_{i})
$$

 where $v_i=V_{i,i}$. Given the above-mentioned bounds on $v_{i}$,
these are always less than $\sigma^2$.

Degeneracy in the columns of $\mathbf{X}$ can become a problem. This is when
two or more of the columns become co-linear. We have already seen this with our
single regressor example wherein $\mathbf{x}$ close to $\mathbf{1}$ was bad
news. To compensate for this effect we can load the diagonal elements and solve
for the unknown parameters as in the following,

$$
\hat{ \boldsymbol{\beta} } = (\mathbf{X}^T \mathbf{X}+\alpha \mathbf{I})^{-1} \mathbf{X}^T \mathbf{Y}
$$

 where $\alpha>0$ is a tunable hyper-parameter. This method is known
as *ridge regression* and was proposed in 1970 by Hoerl and Kenndard. It can be
shown that this is the equivalent to minimizing the following objective,

$$
\Vert \mathbf{Y}- \mathbf{X} \boldsymbol{\beta}\Vert^2  + \alpha \Vert \boldsymbol{\beta}\Vert^2
$$

 In other words, the length of the estimated $\boldsymbol{\beta}$ is
penalized with larger $\alpha$. This has the effect of stabilizing the
subsequent inverse calculation and also providing a means to trade bias and
variance, which we will discuss at length in the section ref{ch:ml:sec:regularization}

**Interpreting Residuals.**  Our model assumes an additive Gaussian noise term.
We can check the voracity of this assumption by examining the residuals after
fitting.  The residuals are the difference between the fitted values and the
original data

$$
\hat{\epsilon}_i = \hat{a} x_i + \hat{b} - y_i
$$

 While the p-value and the F-ratio provide some indication of whether
or not computing the slope of the regression makes sense, we can get directly
at the key assumption of additive Gaussian noise.

For sufficiently small dimensions, the `scipy.stats.probplot` we discussed in
the last chapter provides quick visual evidence one way or another by plotting
the standardized residuals,

$$
r_i = \frac{e_i}{\hat{\sigma}\sqrt{1-v_i}}
$$

 The other part of the iid assumption implies homoscedasticity (all
$r_i$ have equal variances). Under the additive Gaussian noise assumption, the
$e_i$ should also be distributed according to $\mathcal{N}(0,\sigma^2(1-v_i))$.
The normalized residuals $r_i$ should then be distributed according to
$\mathcal{N}(0,1)$. Thus, the presence of any $r_i \notin [-1.96,1.96]$ should
not be common at the 5% significance level and is thereby breeds suspicion
regarding the homoscedasticity assumption.

The Levene test in `scipy.stats.leven` tests the null hypothesis that all the
variances are equal. This basically checks whether or not the standardized
residuals vary across $x_i$ more than expected. Under the homoscedasticity
assumption, the variance should be independent of $x_i$. If not, then this is a
clue that there is a missing variable in the analysis or that the variables
themselves should be transformed (e.g., using the $\log$ function) into another
format that can reduce this effect. Also, we can use weighted least-squares
instead of ordinary least-squares.

**Variable Scaling.** It is tempting to conclude in a multiple
regression that small coefficients in any of the $\boldsymbol{\beta}$ terms
implies that those terms are not important.  However, simple unit conversions
can cause this effect. For example, if one of the regressors is in
units of kilometers and the others are in meters, then just the scale
factor can give the impression of outsized or under-sized effects. The
common way to account for this is to scale the regressors so that

$$
x^\prime = \frac{x-\bar{x}}{\sigma_x}
$$

 This has the side effect of converting the slope parameters
into correlation coefficients, which is bounded by $\pm 1$.

**Influential Data.** We have already discussed the idea
of leverage. The concept of *influence* combines leverage with
outliers. To understand influence, consider [Figure](#fig:Regression_005).

The point on the right has outsized influence in this data because it is the only one used to determine the slope of the fitted line.

The point on the right in [Figure](#fig:Regression_005) is the only one that
is contributing to the calculation of the slope for the fitted line. Thus, it
is very influential in this sense. Cook's distance is a good way to get at this
concept numerically.  To compute this, we have to compute the $j^{th}$
component of the estimated target variable with the $i^{th}$ point deleted. We
call this $\hat{y}_{j(i)}$.  Then, we compute the following,

$$
D_i =\frac{\sum_j (\hat{y}_j- \hat{y}_{j(i)})^2}{p/n \sum_j (\hat{y}_j-y_j)^2}
$$

 where, as before, $p$ is the number of estimated terms (e.g., $p=2$
in the bivariate case). This calculation emphasizes the effect of the outlier
by predicting the target variable with and without each point. In the case of
[Figure](#fig:Regression_005), losing any of the points on the left cannot
change the estimated target variable much, but losing the single point on the
right surely does. The point on the right does not seem to be an outlier (it
*is* on the fitted line), but this is because it is influential enough to
rotate the line to align with it. Cook's distance helps capture this effect by
leaving each sample out and re-fitting the remainder as shown in the last
equation. [Figure](#fig:Regression_006) shows the calculated Cook's
distance for the data in [Figure](#fig:Regression_005), showing that
the data point on the right (sample index `5`) has outsized influence on the fitted line. As a rule
of thumb, Cook's distance values greater than one are suspect.

The calculated Cook's distance for the data in [Figure](#fig:Regression_005).

As another illustration of influence, consider [Figure](#fig:Regression_007)
which shows some data that nicely line up, but with one outlier (filled black
circle) in the upper panel.  The lower panel shows so-computed Cook's distance
for this data. As shown Cook's distance emphasizes the presence of the outlier.
Because the calculation involves leaving a single sample out and
re-calculating the rest, it can be a time-consuming operation suitable to
relatively small data sets. There is always the temptation to downplay the
importance of outliers because they conflict with a favored model, but
outliers must be carefully examined to understand why the model is unable
to capture them. It could be something as simple as faulty data
collection, or it could be an indication of deeper issues that
have been overlooked. The following code shows how Cook's distance
was compute for [Figure](#fig:Regression_006) and
[Figure](#fig:Regression_007).

```python
fit = lambda i,x,y: np.polyval(np.polyfit(x,y,1),i)
omit = lambda i,x: ([k for j,k in enumerate(x) if j !=i])
def cook_d(k):
   num = sum((fit(j,omit(k,x),omit(k,y))- fit(j,x,y))**2 for j in x)
   den = sum((y-np.polyval(np.polyfit(x,y,1),x))**2/len(x)*2)
   return num/den
```

**Programming Tip.**

The function `omit` sweeps through the data and excludes
the $i^{th}$ data element. The embedded `enumerate` function
associates every element in the iterable with its corresponding
index.

The upper panel shows data that fit on a line and an outlier point (filled black circle). The lower panel shows the calculated Cook's distance for the data in upper panel and shows that the tenth point (i.e., the outlier) has disproportionate influence.