# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
a = np.array([[1,-1,5],[4,5,8],[7,8,6]])
b = np.array([1,2,3]).reshape(1,3)
print(b.shape)
print(b)
print(a)
b = a.mean(axis=1)
c = a.var(axis=1)
print(b)
print(a-b)
print(c+1)
print((a-b)/(c+1))
print(-6.666667/7.2222)
print(-6.66667/3.8888)
```

    (1, 3)
    [[1 2 3]]
    [[ 1 -1  5]
     [ 4  5  8]
     [ 7  8  6]]
    [1.66666667 5.66666667 7.        ]
    [[-0.66666667 -6.66666667 -2.        ]
     [ 2.33333333 -0.66666667  1.        ]
     [ 5.33333333  2.33333333 -1.        ]]
    [7.22222222 3.88888889 1.66666667]
    [[-0.09230769 -1.71428571 -1.2       ]
     [ 0.32307692 -0.17142857  0.6       ]
     [ 0.73846154  0.6        -0.6       ]]
    -0.9230798094763369
    -1.7143257560172804

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [2.22044605e-17 8.16013923e-17 4.46864767e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
a = np.array([[1,2],[1,2]])
b = np.array([-3,-2])
print(a.T)
print(b.shape)
print(np.transpose(b).shape)
print(a * b)
```

    [[1 1]
     [2 2]]
    (2,)
    (2,)
    [[-3 -4]
     [-3 -4]]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029275364845641e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  0.0
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 0.89x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
%load_ext autoreload
%autoreload 2

np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.

for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization=  'batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload
    Running check with reg =  0
    Initial loss:  2.2861785657847262
    W0 relative error: 5.66e-05
    W1 relative error: 6.02e-06
    W2 relative error: 3.01e-10
    b0 relative error: 4.16e-09
    b1 relative error: 1.67e-08
    b2 relative error: 1.69e-10
    beta0 relative error: 3.73e-09
    beta1 relative error: 5.92e-09
    gamma0 relative error: 9.41e-09
    gamma1 relative error: 3.79e-08

    Running check with reg =  3.14
    Initial loss:  7.003236268442882
    W0 relative error: 2.42e-06
    W1 relative error: 1.58e-06
    W2 relative error: 6.57e-09
    b0 relative error: 2.22e-08
    b1 relative error: 4.16e-08
    b2 relative error: 1.77e-10
    beta0 relative error: 6.63e-09
    beta1 relative error: 5.47e-08
    gamma0 relative error: 6.42e-09
    gamma1 relative error: 9.92e-08

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

```python

np.random.seed(231)

# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.324357
    (Epoch 0 / 10) train acc: 0.112000; val_acc: 0.127000
    (Epoch 1 / 10) train acc: 0.280000; val_acc: 0.226000
    (Iteration 21 / 200) loss: 2.121563
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.235000
    (Iteration 41 / 200) loss: 2.136386
    (Epoch 3 / 10) train acc: 0.384000; val_acc: 0.264000
    (Iteration 61 / 200) loss: 1.921752
    (Epoch 4 / 10) train acc: 0.410000; val_acc: 0.245000
    (Iteration 81 / 200) loss: 1.529470
    (Epoch 5 / 10) train acc: 0.469000; val_acc: 0.282000
    (Iteration 101 / 200) loss: 1.659143
    (Epoch 6 / 10) train acc: 0.515000; val_acc: 0.302000
    (Iteration 121 / 200) loss: 1.595251
    (Epoch 7 / 10) train acc: 0.519000; val_acc: 0.302000
    (Iteration 141 / 200) loss: 1.564859
    (Epoch 8 / 10) train acc: 0.534000; val_acc: 0.279000
    (Iteration 161 / 200) loss: 1.437147
    (Epoch 9 / 10) train acc: 0.573000; val_acc: 0.281000
    (Iteration 181 / 200) loss: 1.297278
    (Epoch 10 / 10) train acc: 0.565000; val_acc: 0.271000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302566
    (Epoch 0 / 10) train acc: 0.128000; val_acc: 0.135000
    (Epoch 1 / 10) train acc: 0.254000; val_acc: 0.228000
    (Iteration 21 / 200) loss: 2.194719
    (Epoch 2 / 10) train acc: 0.249000; val_acc: 0.217000
    (Iteration 41 / 200) loss: 2.050111
    (Epoch 3 / 10) train acc: 0.317000; val_acc: 0.251000
    (Iteration 61 / 200) loss: 1.833334
    (Epoch 4 / 10) train acc: 0.333000; val_acc: 0.263000
    (Iteration 81 / 200) loss: 1.953899
    (Epoch 5 / 10) train acc: 0.363000; val_acc: 0.250000
    (Iteration 101 / 200) loss: 2.130708
    (Epoch 6 / 10) train acc: 0.363000; val_acc: 0.247000
    (Iteration 121 / 200) loss: 1.810888
    (Epoch 7 / 10) train acc: 0.386000; val_acc: 0.253000
    (Iteration 141 / 200) loss: 1.788717
    (Epoch 8 / 10) train acc: 0.406000; val_acc: 0.246000
    (Iteration 161 / 200) loss: 1.659366
    (Epoch 9 / 10) train acc: 0.409000; val_acc: 0.251000
    (Iteration 181 / 200) loss: 1.535577
    (Epoch 10 / 10) train acc: 0.407000; val_acc: 0.256000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers_ws[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
[FILL THIS IN]

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
[FILL THIS IN]

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_backward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
print(a.shape)
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
    (4, 3)
      means:  [-4.81096644e-16  0.00000000e+00  7.40148683e-17 -5.55111512e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)
bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
print(dgamma)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    [2.29048278 1.39248907 2.93350569 0.98234546 2.08326113]
    dx error:  1.7029275364845641e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

```python

# Gradient check layernorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.433615657860454e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
[FILL THIS IN]


"For our purposes, it is convenient to place the boundary of the learning agent not at the limit of its physical body, but at the limit of its control."  - 3.2

## 3.1

##### Exercise 3.2

Is the reinforcement learning framework adequate to usefully represent all goal-directed learning tasks? Can you think of any clear exceptions?

The goal-directed task doesn't work for tasks where failure is not an option (i.e. the problem doesn't reset). One example would be learning how to walk near a cliff, where failure means death with no ability to reset the episode.

Another exception would be  a task in which the rewards cannot be easily reduced to a single number. For example, in teaching a car how to drive we would want to minimize amount of travel time while maximizing safetiness. With the current framework, we would have to reduce all these desirable outcomes into a single number returned by the environment.

##### Exercise 3.4

Suppose you treated pole-balancing as an episodic task but also used discounting, with all rewards zero except for  upon failure. What then would the return be at each time? How does this return differ from that in the discounted, continuing formulation of this task?

For an episodic task with the above rewards, the discounted future reward would be exactly -1 * gamma^k, where k is the time step of failure, since that is when the episode ends. In the continuing formulation task, the reward is -1 * gamma^k1 + -1 * gamma^k2, etc. for each failure.

## 3.6 - MDPs

##### Exercise 3.7
Assuming a finite MDP with a finite number of reward values, write an equation for the transition probabilities and the expected rewards in terms of the joint conditional distribution in (3.5) $Pr(s_{t+1} = s', r_{t+1} = r| s_t, a_t)$

$P^a_{s,s'} = Pr(s_{t+1} = s' | s_t = s, a_t = a) = \sum_r Pr(s_{t+1} = s', r_{t+1} = r| s_t, a_t) $

$R^a_{s, s'} = E[r_{t+1} | s_t = s, a_t = a, s_{t+1} = s'] = \sum_r r \cdot Pr(s_{t+1} = s', r_{t+1} = r| s_t, a_t)$

## 3.7 - Value Functions

##### Exercise 3.8
What is the Bellman equation for action values?

$
\begin{equation}
\begin{split}
Q^{\pi}(s, a) &= E_{\pi} [R_t | s_t = s, a_t = a]\\
=& E_{\pi} [\sum_{k=0}^{\infty} \gamma^k r_{t + k + 1} | s_t = s, a_t = a ]\\
=& E_{\pi} [r_{t + 1} + \gamma \sum_{k=0}^{\infty} \gamma^k r_{t + k + 2} | s_t = s, a_t = a]\\
=& \sum_{s'} P^a_{ss'} R^a_{ss'} + \sum_{s'} P^a_{ss'} \gamma \sum_{k=0}^{\infty} \gamma^k r_{t + k + 2}\\
=& \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma V^{\pi}(s')]\\
=& \sum_{s'} P^a_{ss'} [R^a_{ss'} + \gamma \sum_{a'} \pi(s', a') Q^{\pi}(s', a')]\\
\end{split}
\end{equation}
$

##### Exercise 3.9
The Bellman equation (3.10) must hold for each state for the value function  shown in Figure  3.5b. As an example, show numerically that this equation holds for the center state, valued at 0.7 ,

$V^{\pi}(s_{mid}) = \frac{1}{4} [(0 + .9 * .7) + (0 + 0.9 * 2.3) + (0 + .9 * .4) + (0 - .9 * .4)] = .675 \approx .7$

##### Exercise 3.10
In the gridworld example, rewards are positive for goals, negative for running into the edge of the world, and zero the rest of the time. Are the signs of these rewards important, or only the intervals between them? Prove, using (3.2), that adding a constant $C$ to all the rewards adds a constant, $K$, to the values of all states, and thus does not affect the relative values of any states under any policies. What is $K$ in terms of $C$ and $\gamma$?

We have from previously that $V^{\pi}(s) = E_{\pi} [\sum_{k=0}^{\infty} \gamma^k r_{t+k+1} | s_t = s]$.

Let's add a constant $C$ to all rewards so that $r'_{t+1} = r_{t+1} + C$.

Then, $$V'^{\pi}(s) = E_{\pi} [\sum_{k=0}^{\infty} \gamma^k (r_{t+k+1} + C) | s_t = s]$$
$$ = V^{\pi}(s) + E_{\pi}[ \sum_{k=0}^{\infty} \gamma^k C | s_t = s ]$$
$$ = V^{\pi}(s) + K $$

We have,

$
\begin{equation}
\begin{split}
K =&  E_{\pi}[\sum_{k=0}^{\infty} \gamma^k C | s_t = s]\\
=& \sum_a \pi(s, a) \sum_s' P^a_{ss'} (C \frac{1}{1 - \gamma})\\
=& \frac{C}{1 - \gamma}
\end{split}
\end{equation}
$

##### Exercise 3.11

Now consider adding a constant C to all the rewards in an episodic task, such as maze running. Would this have any effect, or would it leave the task unchanged as in the continuing task above? Why or why not? Give an example.

For episodic tasks we have,

$$K = \sum_a \pi(s, a) \sum_s' P^a_{ss'} \sum_k^{N}(C \gamma^k)$$

where $N$ is the number of time steps until the episode is over. It seems like the reward is maximized as N goes to infiinity, so adding a constant reward to all actions in an episodic task makes the agent never want to end interacting with the environment to avoid finishing the episode.

##### Exercise 3.12

The value of a state depends on the the values of the actions possible in that state and on how likely each action is to be taken under the current policy. We can think of this in terms of a small backup diagram rooted at the state and considering each possible action:

$
\begin{equation}
\begin{split}
V^{\pi} (s) =& E_{\pi} [Q^{\pi} (s, a) | s_t = s]\\
=& \sum_a \pi(s, a) Q^{\pi} (s, a)
\end{split}
\end{equation}
$

##### Exercise 3.13

The value of an action, , can be divided into two parts, the expected next reward, which does not depend on the policy , and the expected sum of the remaining rewards, which depends on the next state and the policy. Again we can think of this in terms of a small backup diagram, this one rooted at an action (state-action pair) and branching to the possible next states:

$
\begin{equation}
\begin{split}
Q^{\pi} (s, a) =& E[ r_{t+1} | s_t  = s, a_t = a] + \gamma E_{\pi}[ V^{\pi}(s') | s_t = s, a_t = a] \\
Q^{\pi} (s, a) =& \sum_{s'} P^a_{ss'} [R_{ss'} + \gamma V^{\pi}(s')]
\end{split}
\end{equation}
$

## 3.8 - Optimal Value Functions

##### 3.16
Give the Bellman equation for  for the recycling robot.
s = search, w = wait, re = recharge
h = high, l = low

$Q^*(s, a) = \sum_{s'} P^a_{s, s'} [R^a_{s,s'} + \gamma max_{a'} Q^*(s', a')]$

$Q^*(h, s) = \alpha (R^s + \gamma max_{a'} Q^*(h, a')) + (1 - \alpha) (R^s + \gamma max_{a'} Q^*(l, a'))$

$Q^*(l, s) = \beta (R^s + \gamma max_{a'} Q^*(l, a')) + (1 - \beta) (-3 + \gamma max_{a'} Q^*(h, a'))$

$Q^*(l, w) = (R^w + \gamma max_{a'} Q^*(l, a'))$

$Q^*(h, w) = (R^w + \gamma max_{a'} Q^*(h, a'))$

$Q^*(l, r) = (0 + \gamma max_{a'} Q^*(h, a'))$

$Q^*(h, r) = 0$

##### 3.17
Figure  3.8 gives the optimal value of the best state of the gridworld as 24.4, to one decimal place. Use your knowledge of the optimal policy and (3.2) to express this value symbolically, and then to compute it to three decimal places.

$V^*(s) = max_{a \in A(s)} \sum_{s'} P^a_{s,s'}[R^a_{s, s'} + \gamma V^*(s')]$

So we have,

$V^*(s) = max_a E[r_{t+1}  + \gamma \sum_{k=0}^{\infty} \gamma^k r_{t + k+2} | s_t=s, s_t = a]$

$ = max_a [10 + \gamma \cdot 0 + \gamma^2 \cdot 0 + \gamma^3 \cdot 0 + \gamma^4 + \gamma^5 \cdot 10 + ...]$

$ = 10(\sum_{k=0}^\infty \gamma^{5k})$

$ = 10 \frac{1}{1 - \gamma^5} \approx 24.419$

# Cart-pole balancing

### Neuron-like adaptive elements

https://webdocs.cs.ualberta.ca/~sutton/papers/barto-sutton-anderson-83.pdf

https://webdocs.cs.ualberta.ca/~sutton/book/code/pole.c

```python
# https://raw.githubusercontent.com/stober/cartpole/master/src/__init__.py

import numpy as np
import math
import random

class CartPole(object):

    def __init__(self):

        # Constants
        self.gravity = 9.8
        self.masscart = 1.0
        self.masspole = 0.1
        self.total_mass = (self.masspole + self.masscart)
        self.length = 0.5 # actually half the pole's length
        self.polemass_length = (self.masspole * self.length)
        self.force_mag = 10.0
        self.tau = 0.02 # seconds between state updates
        self.fourthirds = 1.3333333333333

    def cart_pole(self, action, x = 0.0, xdot = 0.0, theta = 0.0, thetadot = 0.0):
        # action must be binary

        force = self.force_mag if action > 0 else -self.force_mag

        costheta = math.cos(theta)
        sintheta = math.sin(theta)

        tmp = (force + self.polemass_length * (thetadot ** 2) * sintheta) / self.total_mass
        thetaacc = (self.gravity * sintheta - costheta * tmp) / (self.length * (self.fourthirds - self.masspole * costheta ** 2 / self.total_mass))
        xacc = tmp - self.polemass_length * thetaacc * costheta / self.total_mass

        # Update the four state variables, using Euler's method
        x += self.tau * xdot
        xdot += self.tau * xacc
        theta += self.tau * thetadot
        thetadot += self.tau * thetaacc

        return [x, xdot, theta, thetadot]

    def get_box(self, x = 0.0, xdot = 0.0, theta = 0.0, thetadot = 0.0):
        #          get_box:  Given the current state, returns a number from 1 to 162
        #   designating the region of the state space encompassing the current state.
        #   Returns a value of -1 if a failure state is encountered.
        one_degree = 0.0174532 # 2pi/360
        six_degrees = 0.1047192
        twelve_degrees = 0.2094384
        fifty_degrees = 0.87266

        if (x < -2.4 or x > 2.4) or (theta < -twelve_degrees or theta > twelve_degrees):
            return -1

        if x < -0.8:
            box = 0
        elif x < 0.8:
            box = 1
        else:
            box = 2

        if xdot < -0.5:
            pass
        elif xdot < 0.5:
            box += 3
        else:
            box += 6

        if theta < -six_degrees:
            pass
        elif theta < -one_degree:
            box += 9
        elif theta < 0:
            box += 18
        elif theta < one_degree:
            box += 27
        elif theta < six_degrees:
            box += 36
        else:
            box += 45

        if thetadot < -fifty_degrees:
            pass
        elif thetadot < fifty_degrees:
            box += 54
        else:
            box += 108

        return box

    def prob_push_right(self, s):
        return (1.0 / (1.0 + np.exp(-max(-50.0, min(s, 50.0)))))

```

```python
cp = CartPole()

N_BOXES = 162
MAX_FAILURES = 100
MAX_STEPS = 100000
LAMBDAw = 0.9         #/* Decay rate for w eligibility trace. */
LAMBDAv = 0.8         # /* Decay rate for v eligibility trace. */
GAMMA = 0.95          # /* Discount factor for critic. */
ALPHA = 1000          # /* Learning rate for action weights, w. */
BETA = 0.5            # /* Learning rate for critic weights, v. */

failures = steps = 0
w = []; v = []; xbar = []; e = []

# Initialize action and heuristic critic weights and traces.
for i in range(N_BOXES):
    w.append(0.0); v.append(0.0); xbar.append(0.0); e.append(0.0)

# Starting state is (0 0 0 0)
x = x_dot = theta = theta_dot = 0.0

# Find box in state space containing start state
box = cp.get_box(x, x_dot, theta, theta_dot)

# Iterate through the action-learn loop. ---*/
while steps < MAX_STEPS and failures < MAX_FAILURES:
    # Choose action randomly, biased by current weight
    y = (random.uniform(0, 1) < cp.prob_push_right(w[box]))

    # Update traces.
    e[box] += (1.0 - LAMBDAw) * (y - 0.5)
    xbar[box] += (1.0 - LAMBDAv)

    # Remember prediction of failure for current stat
    oldp = v[box]

    # /*--- Apply action to the simulated cart-pole ---*/
    x, x_dot, theta, theta_dot = cp.cart_pole(y, x, x_dot, theta, theta_dot)

    # /*--- Get box of state space containing the resulting state. ---*/
    box = cp.get_box(x, x_dot, theta, theta_dot)

    if box < 0:
        # /*--- Failure occurred. ---*/
        failed = 1
        failures += 1
        print("Trial %d was %d steps.\n" %(failures, steps))
        steps = 0

        # /*--- Reset state to (0 0 0 0).  Find the box. ---*/
        x = x_dot = theta = theta_dot = 0.0
        box = cp.get_box(x, x_dot, theta, theta_dot)

        # /*--- Reinforcement upon failure is -1. Prediction of failure is 0. ---*/
        r = -1.0
        p = 0.0
    else:
        # /*--- Not a failure. ---*/
        failed = 0

        # /*--- Reinforcement is 0. Prediction of failure given by v weight. ---*/
        r = 0
        p = v[box]

    steps += 1

    # /*--- Heuristic reinforcement is:   current reinforcement
    #      + gamma * new failure prediction - previous failure prediction ---*/
    rhat = r + GAMMA * p - oldp

    for i in range(N_BOXES):
        # /*--- Update all weights. ---*/

        w[i] += ALPHA * rhat * e[i]
        v[i] += BETA * rhat * xbar[i]

        if v[i] < -1.0:
            v[i] = v[i]

        if failed == 1:
            #/*--- If failure, zero all traces. ---*/
            e[i] = 0.0
            xbar[i] = 0.0
        else:
            # /*--- Otherwise, update (decay) the traces. ---*/
            e[i] *= LAMBDAw
            xbar[i] *= LAMBDAv

if (failures == MAX_FAILURES):
    print("Pole not balanced. Stopping after %d failures." %(failures))
else:
    print("Pole balanced successfully for at least %d steps\n" %(steps))

```

    Trial 1 was 17 steps.

    Trial 2 was 14 steps.

    Trial 3 was 10 steps.

    Trial 4 was 10 steps.

    Trial 5 was 17 steps.

    Trial 6 was 71 steps.

    Trial 7 was 23 steps.

    Trial 8 was 26 steps.

    Trial 9 was 88 steps.

    Trial 10 was 39 steps.

    Trial 11 was 15 steps.

    Trial 12 was 43 steps.

    Trial 13 was 121 steps.

    Trial 14 was 30 steps.

    Trial 15 was 28 steps.

    Trial 16 was 171 steps.

    Trial 17 was 193 steps.

    Trial 18 was 210 steps.

    Trial 19 was 222 steps.

    Trial 20 was 185 steps.

    Trial 21 was 207 steps.

    Trial 22 was 99 steps.

    Trial 23 was 85 steps.

    Trial 24 was 185 steps.

    Trial 25 was 213 steps.

    Trial 26 was 230 steps.

    Trial 27 was 283 steps.

    Trial 28 was 281 steps.

    Trial 29 was 234 steps.

    Trial 30 was 292 steps.

    Trial 31 was 732 steps.

    Trial 32 was 455 steps.

    Trial 33 was 321 steps.

    Trial 34 was 264 steps.

    Trial 35 was 380 steps.

    Trial 36 was 294 steps.

    Trial 37 was 318 steps.

    Trial 38 was 241 steps.

    Trial 39 was 1554 steps.

    Trial 40 was 1792 steps.

    Trial 41 was 2523 steps.

    Trial 42 was 1555 steps.

    Trial 43 was 56268 steps.

    Trial 44 was 12440 steps.

    Trial 45 was 84286 steps.

    Pole balanced successfully for at least 100000 steps

```python
# Us AI Gym

import gym
import random
env = gym.make('CartPole-v0')

cp = CartPole()
N_BOXES = 162
MAX_FAILURES = 100
MAX_STEPS = 100000
LAMBDAw = 0.9         #/* Decay rate for w eligibility trace. */
LAMBDAv = 0.8         # /* Decay rate for v eligibility trace. */
GAMMA = 0.95          # /* Discount factor for critic. */
ALPHA = 1000          # /* Learning rate for action weights, w. */
BETA = 0.5            # /* Learning rate for critic weights, v. */

failures = steps = 0
w = []; v = []; xbar = []; e = []

# Initialize action and heuristic critic weights and traces.
for i in range(N_BOXES):
    w.append(0.0); v.append(0.0); xbar.append(0.0); e.append(0.0)

# Starting state is (0 0 0 0)
observation = env.reset()
x, x_dot, theta, theta_dot = observation

# Find box in state space containing start state
box = cp.get_box(x, x_dot, theta, theta_dot)

# Iterate through the action-learn loop. ---*/
while steps < MAX_STEPS and failures < MAX_FAILURES:
    env.render()

    # Choose action randomly, biased by current weight
    y = (random.uniform(0, 1) < cp.prob_push_right(w[box]))

    # Update traces.
    e[box] += (1.0 - LAMBDAw) * (y - 0.5)
    xbar[box] += (1.0 - LAMBDAv)

    # Remember prediction of failure for current stat
    oldp = v[box]

    # /*--- Apply action to the simulated cart-pole ---*/
    observation, reward, done, info = env.step(y)
    x, x_dot, theta, theta_dot = observation

    # /*--- Get box of state space containing the resulting state. ---*/
    box = cp.get_box(x, x_dot, theta, theta_dot)

    if done:
        # /*--- Failure occurred. ---*/
        failed = 1
        failures += 1
        print("Trial %d was %d steps.\n" %(failures, steps))
        steps = 0

        # /*--- Reset state to (0 0 0 0).  Find the box. ---*/
        observation = env.reset()
        x, x_dot, theta, theta_dot = observation
        box = cp.get_box(x, x_dot, theta, theta_dot)

        # /*--- Reinforcement upon failure is -1. Prediction of failure is 0. ---*/
        r = -1
        p = 0.0
    else:
        # /*--- Not a failure. ---*/
        failed = 0

        # /*--- Reinforcement is 0. Prediction of failure given by v weight. ---*/
        r = 0
        p = v[box]

    steps += 1

    # /*--- Heuristic reinforcement is:   current reinforcement
    #      + gamma * new failure prediction - previous failure prediction ---*/
    rhat = r + GAMMA * p - oldp

    for i in range(N_BOXES):
        # /*--- Update all weights. ---*/

        w[i] += ALPHA * rhat * e[i]
        v[i] += BETA * rhat * xbar[i]

        if v[i] < -1.0:
            v[i] = v[i]

        if failed == 1:
            #/*--- If failure, zero all traces. ---*/
            e[i] = 0.0
            xbar[i] = 0.0
        else:
            # /*--- Otherwise, update (decay) the traces. ---*/
            e[i] *= LAMBDAw
            xbar[i] *= LAMBDAv

if (failures == MAX_FAILURES):
    print("Pole not balanced. Stopping after %d failures." %(failures))
else:
    print("Pole balanced successfully for at least %d steps\n" %(steps))

```

    [2016-07-19 01:55:07,269] Making new env: CartPole-v0

    Trial 1 was 20 steps.

    Trial 2 was 12 steps.

    Trial 3 was 9 steps.

    Trial 4 was 155 steps.

    Trial 5 was 16 steps.

    Trial 6 was 103 steps.

    Trial 7 was 34 steps.

    Trial 8 was 38 steps.

    Trial 9 was 12 steps.

    Trial 10 was 27 steps.

    Trial 11 was 10 steps.

    Trial 12 was 58 steps.

    Trial 13 was 188 steps.

    Trial 14 was 144 steps.

    Trial 15 was 186 steps.

    Trial 16 was 101 steps.

    Trial 17 was 122 steps.

    Trial 18 was 40 steps.

    Trial 19 was 26 steps.

    Trial 20 was 48 steps.

    Trial 21 was 16 steps.

    Trial 22 was 11 steps.

    Trial 23 was 714 steps.

    Trial 24 was 412 steps.

    Trial 25 was 620 steps.

    Trial 26 was 470 steps.

    Trial 27 was 580 steps.

    Trial 28 was 498 steps.

    Trial 29 was 523 steps.

    Trial 30 was 736 steps.

    Trial 31 was 597 steps.

    Trial 32 was 825 steps.

    Trial 33 was 759 steps.

    Trial 34 was 342 steps.

    Trial 35 was 482 steps.

    Trial 36 was 598 steps.

    Trial 37 was 601 steps.

    Trial 38 was 614 steps.

    Trial 39 was 868 steps.

    Trial 40 was 1189 steps.

    Trial 41 was 1358 steps.

    Trial 42 was 427 steps.

    Trial 43 was 785 steps.

    Trial 44 was 387 steps.

    Trial 45 was 287 steps.

    Trial 46 was 3246 steps.

    Trial 47 was 10808 steps.

    Trial 48 was 15820 steps.

    Trial 49 was 22176 steps.

    Trial 50 was 4548 steps.

    Trial 51 was 11423 steps.

    Trial 52 was 4957 steps.

    Trial 53 was 37452 steps.

    Trial 54 was 6652 steps.

    Pole balanced successfully for at least 100000 steps

```python

```

### Simulated Annealing

https://gym.openai.com/evaluations/eval_7PjfKVdYRMuKyahjgTbrA#reproducibility

```python
import gym
import random
env = gym.make('CartPole-v0')

# env.monitor.start('/tmp/cartpole-experiment-3', force=True)

bestSteps = 0
best = [0, 0, 0, 0]
alpha = 1

for i_episode in xrange(80):

    test = [best[i] + (random.random() - 0.5) * alpha for i in range(4)]

    score = 0
    for ep in range(10):  # <-- key thing was to figure out that you need to do 10 tests per point
        observation = env.reset()
        for t in xrange(200): # <-- because you can't go over 200 you need to gain score hight else where
            env.render()
            if sum(observation[i] * test[i] for i in range(4)) > 0:
                action = 1
            else:
                action = 0
            observation, reward, done, info = env.step(action)
            if done:
                break

        score += t

    if bestSteps < score:
        bestSteps = score
        best = test
        alpha *= .9

    print "test:", "[%+1.2f %+1.2f %+1.2f %+1.2f]" % tuple(test), score,
    print "best:", "[%+1.2f %+1.2f %+1.2f %+1.2f]" % tuple(best), bestSteps, alpha

print "best", best, bestSteps

# env.monitor.close()
```

    [2016-07-19 01:43:02,117] Making new env: CartPole-v0

    test: [-0.47 +0.06 +0.29 -0.32] 85 best: [-0.47 +0.06 +0.29 -0.32] 85 0.9
    test: [-0.81 -0.11 +0.70 -0.55] 88 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.49 -0.13 +0.60 -0.53] 85 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.98 -0.30 +1.09 -0.47] 88 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.50 +0.21 +0.46 -0.53] 86 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.59 -0.15 +0.57 -0.91] 84 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-1.01 -0.45 +0.43 -0.57] 83 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.68 +0.05 +0.43 -0.18] 82 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-1.05 +0.07 +0.80 -0.15] 88 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.97 -0.07 +1.00 -0.36] 83 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.82 +0.15 +0.85 -0.47] 88 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.91 +0.29 +0.65 -0.24] 83 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.96 -0.18 +0.34 -0.44] 86 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-0.66 -0.18 +0.40 -0.34] 86 best: [-0.81 -0.11 +0.70 -0.55] 88 0.81
    test: [-1.11 +0.17 +0.99 -0.32] 89 best: [-1.11 +0.17 +0.99 -0.32] 89 0.729
    test: [-1.16 +0.43 +0.99 -0.02] 87 best: [-1.11 +0.17 +0.99 -0.32] 89 0.729
    test: [-0.80 -0.11 +0.79 -0.03] 543 best: [-0.80 -0.11 +0.79 -0.03] 543 0.6561
    test: [-0.91 -0.26 +0.93 -0.21] 317 best: [-0.80 -0.11 +0.79 -0.03] 543 0.6561
    test: [-0.99 -0.32 +0.67 +0.28] 618 best: [-0.99 -0.32 +0.67 +0.28] 618 0.59049
    test: [-0.72 -0.45 +0.39 +0.03] 560 best: [-0.99 -0.32 +0.67 +0.28] 618 0.59049
    test: [-1.24 -0.19 +0.53 +0.31] 856 best: [-1.24 -0.19 +0.53 +0.31] 856 0.531441
    test: [-1.36 -0.41 +0.77 +0.15] 727 best: [-1.24 -0.19 +0.53 +0.31] 856 0.531441
    test: [-1.04 -0.13 +0.51 +0.25] 610 best: [-1.24 -0.19 +0.53 +0.31] 856 0.531441
    test: [-1.47 -0.18 +0.78 +0.40] 728 best: [-1.24 -0.19 +0.53 +0.31] 856 0.531441
    test: [-1.24 +0.00 +0.27 +0.41] 962 best: [-1.24 +0.00 +0.27 +0.41] 962 0.4782969
    test: [-1.32 -0.12 +0.30 +0.19] 618 best: [-1.24 +0.00 +0.27 +0.41] 962 0.4782969
    test: [-1.27 -0.06 +0.21 +0.27] 607 best: [-1.24 +0.00 +0.27 +0.41] 962 0.4782969
    test: [-1.25 -0.20 +0.39 +0.62] 740 best: [-1.24 +0.00 +0.27 +0.41] 962 0.4782969
    test: [-1.07 +0.02 +0.10 +0.43] 721 best: [-1.24 +0.00 +0.27 +0.41] 962 0.4782969
    test: [-1.11 -0.19 +0.31 +0.59] 794 best: [-1.24 +0.00 +0.27 +0.41] 962 0.4782969
    test: [-1.28 -0.12 +0.27 +0.61] 1068 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.33 +0.02 +0.46 +0.72] 939 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.38 -0.20 +0.10 +0.76] 1018 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.15 -0.24 +0.22 +0.67] 827 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.39 -0.12 +0.35 +0.70] 907 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.07 -0.20 +0.19 +0.80] 1048 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.24 -0.14 +0.43 +0.53] 941 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.10 -0.28 +0.38 +0.64] 899 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.47 -0.06 +0.15 +0.66] 920 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.43 +0.03 +0.30 +0.40] 640 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.11 +0.10 +0.14 +0.53] 970 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.23 +0.03 +0.12 +0.60] 1033 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.13 -0.17 +0.41 +0.71] 822 best: [-1.28 -0.12 +0.27 +0.61] 1068 0.43046721
    test: [-1.39 +0.04 +0.31 +0.78] 1099 best: [-1.39 +0.04 +0.31 +0.78] 1099 0.387420489
    test: [-1.23 +0.14 +0.37 +0.88] 1555 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.15 +0.28 +0.40 +1.06] 1539 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.07 +0.12 +0.50 +0.83] 1214 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.30 +0.21 +0.50 +0.80] 1064 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.30 +0.12 +0.24 +0.76] 1089 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.39 -0.02 +0.35 +0.96] 848 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.38 +0.10 +0.51 +0.96] 1334 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.21 +0.14 +0.45 +0.72] 1173 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.24 +0.27 +0.54 +0.76] 1172 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.21 +0.27 +0.54 +0.95] 1433 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.11 +0.16 +0.39 +0.93] 1553 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.20 +0.17 +0.33 +1.02] 1346 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.07 +0.26 +0.51 +0.85] 1344 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.12 +0.20 +0.21 +0.97] 963 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.40 +0.10 +0.39 +1.00] 934 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.27 +0.15 +0.21 +0.81] 1026 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.35 +0.19 +0.42 +0.96] 1062 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.20 +0.31 +0.30 +0.73] 1162 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.24 +0.23 +0.24 +0.99] 1357 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.24 +0.29 +0.27 +0.90] 1245 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.28 +0.07 +0.28 +0.99] 1056 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.22 -0.02 +0.50 +0.89] 1012 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.26 +0.16 +0.40 +0.97] 996 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.07 +0.26 +0.38 +0.76] 1166 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.11 +0.17 +0.43 +0.93] 1201 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.37 +0.08 +0.20 +0.78] 1068 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.07 +0.02 +0.24 +0.90] 1518 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.24 +0.28 +0.40 +0.92] 1266 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.28 +0.23 +0.28 +1.00] 1214 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.16 +0.29 +0.23 +0.77] 1232 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.23 +0.26 +0.21 +0.78] 1328 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.25 +0.20 +0.23 +0.98] 917 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.24 +0.06 +0.30 +0.91] 1234 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.10 +0.07 +0.29 +0.83] 1089 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.20 +0.06 +0.29 +1.03] 1375 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    test: [-1.19 +0.27 +0.30 +0.88] 1378 best: [-1.23 +0.14 +0.37 +0.88] 1555 0.3486784401
    best [-1.2322893214490043, 0.14150894065716918, 0.36790056804535143, 0.881835892179857] 1555


# Programming Exercise 5:
# Regularized Linear Regression and Bias vs Variance

## Introduction

In this exercise, you will implement regularized linear regression and use it to study models with different bias-variance properties. Before starting on the programming exercise, we strongly recommend watching the video lectures and completing the review questions for the associated topics.

All the information you need for solving this assignment is in this notebook, and all the code you will be implementing will take place within this notebook. The assignment can be promptly submitted to the coursera grader directly from this notebook (code and instructions are included below).

Before we begin with the exercises, we need to import all libraries required for this programming exercise. Throughout the course, we will be using [`numpy`](http://www.numpy.org/) for all arrays and matrix operations, [`matplotlib`](https://matplotlib.org/) for plotting, and [`scipy`](https://docs.scipy.org/doc/scipy/reference/) for scientific and numerical computation functions and tools. You can find instructions on how to install required libraries in the README file in the [github repository](https://github.com/dibgerge/ml-coursera-python-assignments).

```python
# used for manipulating directory paths
import os

# Scientific and vector computation for python
import numpy as np

# Plotting library
from matplotlib import pyplot

# Optimization module in scipy
from scipy import optimize

# will be used to load MATLAB mat datafile format
from scipy.io import loadmat

# library written for this exercise providing additional functions for assignment submission, and others
import utils

# define the submission/grader object for this exercise
grader = utils.Grader()

# tells matplotlib to embed plots within the notebook
%matplotlib inline
```

## Submission and Grading

After completing each part of the assignment, be sure to submit your solutions to the grader. The following is a breakdown of how each part of this exercise is scored.

| Section | Part                                             | Submitted Function                | Points |
| :-      |:-                                                |:-                                 | :-:    |
| 1       | [Regularized Linear Regression Cost Function](#section1)      | [`linearRegCostFunction`](#linearRegCostFunction) |  25    |
| 2       | [Regularized Linear Regression Gradient](#section2)           | [`linearRegCostFunction`](#linearRegCostFunction) |25      |
| 3       | [Learning Curve](#section3)                                   | [`learningCurve`](#func2)         | 20     |
| 4       | [Polynomial Feature Mapping](#section4)                       | [`polyFeatures`](#polyFeatures)          | 10     |
| 5       | [Cross Validation Curve](#section5)                           | [`validationCurve`](#validationCurve)       | 20     |
|         | Total Points                                     |                                   |100     |

You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.

At the end of each section in this notebook, we have a cell which contains code for submitting the solutions thus far to the grader. Execute the cell to see your score up to the current section. For all your work to be submitted properly, you must execute those cells at least once.

## 1 Regularized Linear Regression

In the first half of the exercise, you will implement regularized linear regression to predict the amount of water flowing out of a dam using the change of water level in a reservoir. In the next half, you will go through some diagnostics of debugging learning algorithms and examine the effects of bias v.s.
variance.

### 1.1 Visualizing the dataset

We will begin by visualizing the dataset containing historical records on the change in the water level, $x$, and the amount of water flowing out of the dam, $y$. This dataset is divided into three parts:

- A **training** set that your model will learn on: `X`, `y`
- A **cross validation** set for determining the regularization parameter: `Xval`, `yval`
- A **test** set for evaluating performance. These are “unseen” examples which your model did not see during training: `Xtest`, `ytest`

Run the next cell to plot the training data. In the following parts, you will implement linear regression and use that to fit a straight line to the data and plot learning curves. Following that, you will implement polynomial regression to find a better fit to the data.

```python
# Load from ex5data1.mat, where all variables will be store in a dictionary
data = loadmat(os.path.join('Data', 'ex5data1.mat'))

# Extract train, test, validation data from dictionary
# and also convert y's form 2-D matrix (MATLAB format) to a numpy vector
X, y = data['X'], data['y'][:, 0]
Xtest, ytest = data['Xtest'], data['ytest'][:, 0]
Xval, yval = data['Xval'], data['yval'][:, 0]

# m = Number of examples
m = y.size

# Plot training data
pyplot.plot(X, y, 'ro', ms=10, mec='k', mew=1)
pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)');
```

### 1.2 Regularized linear regression cost function

Recall that regularized linear regression has the following cost function:

$$ J(\theta) = \frac{1}{2m} \left( \sum_{i=1}^m \left( h_\theta\left( x^{(i)} \right) - y^{(i)} \right)^2 \right) + \frac{\lambda}{2m} \left( \sum_{j=1}^n \theta_j^2 \right)$$

where $\lambda$ is a regularization parameter which controls the degree of regularization (thus, help preventing overfitting). The regularization term puts a penalty on the overall cost J. As the magnitudes of the model parameters $\theta_j$ increase, the penalty increases as well. Note that you should not regularize
the $\theta_0$ term.

You should now complete the code in the function `linearRegCostFunction` in the next cell. Your task is to calculate the regularized linear regression cost function. If possible, try to vectorize your code and avoid writing loops.

```python
def linearRegCostFunction(X, y, theta, lambda_=0.0):
    """
    Compute cost and gradient for regularized linear regression
    with multiple variables. Computes the cost of using theta as
    the parameter for linear regression to fit the data points in X and y.

    Parameters
    ----------
    X : array_like
        The dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each datapoint. A vector of
        shape (m, ).

    theta : array_like
        The parameters for linear regression. A vector of shape (n+1,).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    J : float
        The computed cost function.

    grad : array_like
        The value of the cost function gradient w.r.t theta.
        A vector of shape (n+1, ).

    Instructions
    ------------
    Compute the cost and gradient of regularized linear regression for
    a particular choice of theta.
    You should set J to the cost and grad to the gradient.
    """

    m = X.shape[0]
    h = X.dot(theta)

    J = (1/(2*m))*np.sum(np.square(h-y)) + (lambda_/(2*m))*np.sum(np.square(theta[1:]))

    mask = np.eye(len(theta))
    mask[0, 0] = 0
    grad = 1.0 / m * X.T.dot(X.dot(theta) - y) + 1.0 * lambda_ / m * (mask.dot(theta))

    return J, grad
```

When you are finished, the next cell will run your cost function using `theta` initialized at `[1, 1]`. You should expect to see an output of 303.993.

```python
theta = np.array([1, 1])
J, _ = linearRegCostFunction(np.concatenate([np.ones((m, 1)), X], axis=1), y, theta, 1)

print('Cost at theta = [1, 1]:\t   %f ' % J)
print('This value should be about 303.993192)\n' % J)
```

    Cost at theta = [1, 1]:	   303.993192
    This value should be about 303.993192)

After completing a part of the exercise, you can submit your solutions for grading by first adding the function you modified to the submission object, and then sending your function to Coursera for grading.

The submission script will prompt you for your login e-mail and submission token. You can obtain a submission token from the web page for the assignment. You are allowed to submit your solutions multiple times, and we will take only the highest score into consideration.

*Execute the following cell to grade your solution to the first part of this exercise.*

```python
grader[1] = linearRegCostFunction
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (jdgs.gt@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |   0 /  25 |
                                 Learning Curve |   0 /  20 |
                     Polynomial Feature Mapping |   0 /  10 |
                               Validation Curve |   0 /  20 |
                                      --------------------------------
                                                |  25 / 100 |

### 1.3 Regularized linear regression gradient

Correspondingly, the partial derivative of the cost function for regularized linear regression is defined as:

$$
\begin{align}
& \frac{\partial J(\theta)}{\partial \theta_0} = \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left(x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} & \qquad \text{for } j = 0 \\
& \frac{\partial J(\theta)}{\partial \theta_j} = \left( \frac{1}{m} \sum_{i=1}^m \left( h_\theta \left( x^{(i)} \right) - y^{(i)} \right) x_j^{(i)} \right) + \frac{\lambda}{m} \theta_j & \qquad \text{for } j \ge 1
\end{align}
$$

In the function [`linearRegCostFunction`](#linearRegCostFunction) above, add code to calculate the gradient, returning it in the variable `grad`. Do not forget to re-execute the cell containing this function to update the function's definition.

When you are finished, use the next cell to  run your gradient function using theta initialized at `[1, 1]`. You should expect to see a gradient of `[-15.30, 598.250]`.

```python
theta = np.array([1, 1])
J, grad = linearRegCostFunction(np.concatenate([np.ones((m, 1)), X], axis=1), y, theta, 1)

print('Gradient at theta = [1, 1]:  [{:.6f}, {:.6f}] '.format(*grad))
print(' (this value should be about [-15.303016, 598.250744])\n')
```

    Gradient at theta = [1, 1]:  [-15.303016, 598.250744]
     (this value should be about [-15.303016, 598.250744])

*You should now submit your solutions.*

```python
grader[2] = linearRegCostFunction
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (jdgs.gt@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |  25 /  25 | Nice work!
                                 Learning Curve |   0 /  20 |
                     Polynomial Feature Mapping |   0 /  10 |
                               Validation Curve |   0 /  20 |
                                      --------------------------------
                                                |  50 / 100 |

### Fitting linear regression

Once your cost function and gradient are working correctly, the next cell will run the code in `trainLinearReg` (found in the module `utils.py`) to compute the optimal values of $\theta$. This training function uses `scipy`'s optimization module to minimize the cost function.

In this part, we set regularization parameter $\lambda$ to zero. Because our current implementation of linear regression is trying to fit a 2-dimensional $\theta$, regularization will not be incredibly helpful for a $\theta$ of such low dimension. In the later parts of the exercise, you will be using polynomial regression with regularization.

Finally, the code in the next cell should also plot the best fit line, which should look like the figure below.

The best fit line tells us that the model is not a good fit to the data because the data has a non-linear pattern. While visualizing the best fit as shown is one possible way to debug your learning algorithm, it is not always easy to visualize the data and model. In the next section, you will implement a function to generate learning curves that can help you debug your learning algorithm even if it is not easy to visualize the
data.

```python
# add a columns of ones for the y-intercept
X_aug = np.concatenate([np.ones((m, 1)), X], axis=1)
theta = utils.trainLinearReg(linearRegCostFunction, X_aug, y, lambda_=0)

#  Plot fit over the data
pyplot.plot(X, y, 'ro', ms=10, mec='k', mew=1.5)
pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.plot(X, np.dot(X_aug, theta), '--', lw=2);
```

## 2 Bias-variance

An important concept in machine learning is the bias-variance tradeoff. Models with high bias are not complex enough for the data and tend to underfit, while models with high variance overfit to the training data.

In this part of the exercise, you will plot training and test errors on a learning curve to diagnose bias-variance problems.

### 2.1 Learning Curves

You will now implement code to generate the learning curves that will be useful in debugging learning algorithms. Recall that a learning curve plots training and cross validation error as a function of training set size. Your job is to fill in the function `learningCurve` in the next cell, so that it returns a vector of errors for the training set and cross validation set.

To plot the learning curve, we need a training and cross validation set error for different training set sizes. To obtain different training set sizes, you should use different subsets of the original training set `X`. Specifically, for a training set size of $i$, you should use the first $i$ examples (i.e., `X[:i, :]`
and `y[:i]`).

You can use the `trainLinearReg` function (by calling `utils.trainLinearReg(...)`) to find the $\theta$ parameters. Note that the `lambda_` is passed as a parameter to the `learningCurve` function.
After learning the $\theta$ parameters, you should compute the error on the training and cross validation sets. Recall that the training error for a dataset is defined as

$$ J_{\text{train}} = \frac{1}{2m} \left[ \sum_{i=1}^m \left(h_\theta \left( x^{(i)} \right) - y^{(i)} \right)^2 \right] $$

In particular, note that the training error does not include the regularization term. One way to compute the training error is to use your existing cost function and set $\lambda$ to 0 only when using it to compute the training error and cross validation error. When you are computing the training set error, make sure you compute it on the training subset (i.e., `X[:n,:]` and `y[:n]`) instead of the entire training set. However, for the cross validation error, you should compute it over the entire cross validation set. You should store
the computed errors in the vectors error train and error val.

```python
def learningCurve(X, y, Xval, yval, lambda_=0):
    """
    Generates the train and cross validation set errors needed to plot a learning curve
    returns the train and cross validation set errors for a learning curve.

    In this function, you will compute the train and test errors for
    dataset sizes from 1 up to m. In practice, when working with larger
    datasets, you might want to do this in larger intervals.

    Parameters
    ----------
    X : array_like
        The training dataset. Matrix with shape (m x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    y : array_like
        The functions values at each training datapoint. A vector of
        shape (m, ).

    Xval : array_like
        The validation dataset. Matrix with shape (m_val x n + 1) where m is the
        total number of examples, and n is the number of features
        before adding the bias term.

    yval : array_like
        The functions values at each validation datapoint. A vector of
        shape (m_val, ).

    lambda_ : float, optional
        The regularization parameter.

    Returns
    -------
    error_train : array_like
        A vector of shape m. error_train[i] contains the training error for
        i examples.
    error_val : array_like
        A vecotr of shape m. error_val[i] contains the validation error for
        i training examples.

    Instructions
    ------------
    Fill in this function to return training errors in error_train and the
    cross validation errors in error_val. i.e., error_train[i] and
    error_val[i] should give you the errors obtained after training on i examples.

    Notes
    -----
    - You should evaluate the training error on the first i training
      examples (i.e., X[:i, :] and y[:i]).

      For the cross-validation error, you should instead evaluate on
      the _entire_ cross validation set (Xval and yval).

    - If you are using your cost function (linearRegCostFunction) to compute
      the training and cross validation error, you should call the function with
      the lambda argument set to 0. Do note that you will still need to use
      lambda when running the training to obtain the theta parameters.

    Hint
    ----
    You can loop over the examples with the following:

           for i in range(1, m+1):
               # Compute train/cross validation errors using training examples
               # X[:i, :] and y[:i], storing the result in
               # error_train[i-1] and error_val[i-1]
               ....
    """
    # Number of training examples
    m = y.size

    # You need to return these values correctly
    error_train = np.zeros(m)
    error_val = np.zeros(m)

    for i in np.arange(m):
       # Compute train/cross validation errors using training examples
       # X[:i, :] and y[:i], storing the result in
       # error_train[i-1] and error_val[i-1]

        # compute theta for dataset size `i`
        theta = utils.trainLinearReg(linearRegCostFunction, X[:i+1], y[:i+1], lambda_)

        # compute training error for each dataset size
        # and compute validation error for the full dataset
        error_train[i], _ = linearRegCostFunction(X[:i+1], y[:i+1], theta, lambda_=0)
        error_val[i], _ = linearRegCostFunction(Xval, yval, theta,  lambda_=0)

    return error_train, error_val
```

When you are finished implementing the function `learningCurve`, executing the next cell prints the learning curves and produce a plot similar to the figure below.

In the learning curve figure, you can observe that both the train error and cross validation error are high when the number of training examples is increased. This reflects a high bias problem in the model - the linear regression model is too simple and is unable to fit our dataset well. In the next section, you will implement polynomial regression to fit a better model for this dataset.

```python
X_aug = np.concatenate([np.ones((m, 1)), X], axis=1)
Xval_aug = np.concatenate([np.ones((yval.size, 1)), Xval], axis=1)
error_train, error_val = learningCurve(X_aug, y, Xval_aug, yval, lambda_=0)

pyplot.plot(np.arange(1, m+1), error_train, np.arange(1, m+1), error_val, lw=2)
pyplot.title('Learning curve for linear regression')
pyplot.legend(['Train', 'Cross Validation'])
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 150])

print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```

*You should now submit your solutions.*

```python
grader[3] = learningCurve
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (jdgs.gt@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |  25 /  25 | Nice work!
                                 Learning Curve |  20 /  20 | Nice work!
                     Polynomial Feature Mapping |   0 /  10 |
                               Validation Curve |   0 /  20 |
                                      --------------------------------
                                                |  70 / 100 |

## 3 Polynomial regression

The problem with our linear model was that it was too simple for the data
and resulted in underfitting (high bias). In this part of the exercise, you will address this problem by adding more features. For polynomial regression, our hypothesis has the form:

$$
\begin{align}
h_\theta(x)  &= \theta_0 + \theta_1 \times (\text{waterLevel}) + \theta_2 \times (\text{waterLevel})^2 + \cdots + \theta_p \times (\text{waterLevel})^p \\
& = \theta_0 + \theta_1 x_1 + \theta_2 x_2 + \cdots + \theta_p x_p
\end{align}
$$

Notice that by defining $x_1 = (\text{waterLevel})$, $x_2 = (\text{waterLevel})^2$ , $\cdots$, $x_p =
(\text{waterLevel})^p$, we obtain a linear regression model where the features are the various powers of the original value (waterLevel).

Now, you will add more features using the higher powers of the existing feature $x$ in the dataset. Your task in this part is to complete the code in the function `polyFeatures` in the next cell. The function should map the original training set $X$ of size $m \times 1$ into its higher powers. Specifically, when a training set $X$ of size $m \times 1$ is passed into the function, the function should return a $m \times p$ matrix `X_poly`, where column 1 holds the original values of X, column 2 holds the values of $X^2$, column 3 holds the values of $X^3$, and so on. Note that you don’t have to account for the zero-eth power in this function.

```python
from pprint import pprint
def polyFeatures(X, p):
    """
    Maps X (1D vector) into the p-th power.

    Parameters
    ----------
    X : array_like
        A data vector of size m, where m is the number of examples.

    p : int
        The polynomial power to map the features.

    Returns
    -------
    X_poly : array_like
        A matrix of shape (m x p) where p is the polynomial
        power and m is the number of examples. That is:

        X_poly[i, :] = [X[i], X[i]**2, X[i]**3 ...  X[i]**p]

    Instructions
    ------------
    Given a vector X, return a matrix X_poly where the p-th column of
    X contains the values of X to the p-th power.
    """

    # You need to return the following variables correctly.
    X_poly = np.zeros((X.shape[0], p))

    # set the first column to the original values of X
    X_poly[:, 0] = X[:, 0]

    # set each column (starting from the second one) to its i-th powered value
    # the second column is X to the power of 2
    # the third is X to the power of 3 and so on
    for i in range(1, p):
        X_poly[:, i] = np.power(X, i+1)[:, 0]

    return X_poly
```

Now you have a function that will map features to a higher dimension. The next cell will apply it to the training set, the test set, and the cross validation set.

```python
p = 8

# Map X onto Polynomial Features and Normalize
X_poly = polyFeatures(X, p)
X_poly, mu, sigma = utils.featureNormalize(X_poly)
X_poly = np.concatenate([np.ones((m, 1)), X_poly], axis=1)

# Map X_poly_test and normalize (using mu and sigma)
X_poly_test = polyFeatures(Xtest, p)
X_poly_test -= mu
X_poly_test /= sigma
X_poly_test = np.concatenate([np.ones((ytest.size, 1)), X_poly_test], axis=1)

# Map X_poly_val and normalize (using mu and sigma)
X_poly_val = polyFeatures(Xval, p)
X_poly_val -= mu
X_poly_val /= sigma
X_poly_val = np.concatenate([np.ones((yval.size, 1)), X_poly_val], axis=1)

print('Normalized Training Example 1:')
X_poly[0, :]
```

    Normalized Training Example 1:

    array([ 1.        , -0.36214078, -0.75508669,  0.18222588, -0.70618991,
            0.30661792, -0.59087767,  0.3445158 , -0.50848117])

*You should now submit your solutions.*

```python
grader[4] = polyFeatures
grader.grade()
```

    Submitting Solutions | Programming Exercise regularized-linear-regression-and-bias-variance

    Use token from last successful submission (jdgs.gt@gmail.com)? (Y/n): y
                                      Part Name |     Score | Feedback
                                      --------- |     ----- | --------
    Regularized Linear Regression Cost Function |  25 /  25 | Nice work!
         Regularized Linear Regression Gradient |  25 /  25 | Nice work!
                                 Learning Curve |  20 /  20 | Nice work!
                     Polynomial Feature Mapping |  10 /  10 | Nice work!
                               Validation Curve |   0 /  20 |
                                      --------------------------------
                                                |  80 / 100 |

## 3.1 Learning Polynomial Regression

After you have completed the function `polyFeatures`, we will proceed to train polynomial regression using your linear regression cost function.

Keep in mind that even though we have polynomial terms in our feature vector, we are still solving a linear regression optimization problem. The polynomial terms have simply turned into features that we can use for linear regression. We are using the same cost function and gradient that you wrote for the earlier part of this exercise.

For this part of the exercise, you will be using a polynomial of degree 8. It turns out that if we run the training directly on the projected data, will not work well as the features would be badly scaled (e.g., an example with $x = 40$ will now have a feature $x_8 = 40^8 = 6.5 \times 10^{12}$). Therefore, you will
need to use feature normalization.

Before learning the parameters $\theta$ for the polynomial regression, we first call `featureNormalize` and normalize the features of the training set, storing the mu, sigma parameters separately. We have already implemented this function for you (in `utils.py` module) and it is the same function from the first exercise.

After learning the parameters $\theta$, you should see two plots generated for polynomial regression with $\lambda = 0$, which should be similar to the ones here:

You should see that the polynomial fit is able to follow the datapoints very well, thus, obtaining a low training error. The figure on the right shows that the training error essentially stays zero for all numbers of training samples. However, the polynomial fit is very complex and even drops off at the extremes. This is an indicator that the polynomial regression model is overfitting the training data and will not generalize well.

To better understand the problems with the unregularized ($\lambda = 0$) model, you can see that the learning curve  shows the same effect where the training error is low, but the cross validation error is high. There is a gap between the training and cross validation errors, indicating a high variance problem.

```python
lambda_ = 100
theta = utils.trainLinearReg(linearRegCostFunction, X_poly, y,
                             lambda_=lambda_, maxiter=55)

# Plot training data and fit
pyplot.plot(X, y, 'ro', ms=10, mew=1.5, mec='k')

utils.plotFit(polyFeatures, np.min(X), np.max(X), mu, sigma, theta, p)

pyplot.xlabel('Change in water level (x)')
pyplot.ylabel('Water flowing out of the dam (y)')
pyplot.title('Polynomial Regression Fit (lambda = %f)' % lambda_)
pyplot.ylim([-20, 50])

pyplot.figure()
error_train, error_val = learningCurve(X_poly, y, X_poly_val, yval, lambda_)
pyplot.plot(np.arange(1, 1+m), error_train, np.arange(1, 1+m), error_val)

pyplot.title('Polynomial Regression Learning Curve (lambda = %f)' % lambda_)
pyplot.xlabel('Number of training examples')
pyplot.ylabel('Error')
pyplot.axis([0, 13, 0, 100])
pyplot.legend(['Train', 'Cross Validation'])

print('Polynomial Regression (lambda = %f)\n' % lambda_)
print('# Training Examples\tTrain Error\tCross Validation Error')
for i in range(m):
    print('  \t%d\t\t%f\t%f' % (i+1, error_train[i], error_val[i]))
```