# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.
    	There will be an option for Colab users and another for Jupyter (local) users.

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [7.10542736e-17 7.82707232e-17 9.71445147e-18]
      stds:[0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029261167605239e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  6.117609279344672e-13
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 2.21x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 4.05e-10
    b1 relative error: 3.11e-07
    b2 relative error: 1.94e-08
    b3 relative error: 1.01e-10
    beta1 relative error: 2.99e-10
    beta2 relative error: 5.87e-10
    gamma1 relative error: 4.22e-09
    gamma2 relative error: 1.18e-10

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 1.38e-08
    b2 relative error: 7.99e-07
    b3 relative error: 1.73e-10
    beta1 relative error: 3.25e-09
    beta2 relative error: 4.69e-10
    gamma1 relative error: 8.22e-10
    gamma2 relative error: 9.12e-11

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340975
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.279000; val_acc: 0.249000
    (Iteration 21 / 200) loss: 1.963015
    (Epoch 2 / 10) train acc: 0.413000; val_acc: 0.300000
    (Iteration 41 / 200) loss: 2.044604
    (Epoch 3 / 10) train acc: 0.455000; val_acc: 0.280000
    (Iteration 61 / 200) loss: 1.836792
    (Epoch 4 / 10) train acc: 0.524000; val_acc: 0.319000
    (Iteration 81 / 200) loss: 1.243883
    (Epoch 5 / 10) train acc: 0.588000; val_acc: 0.318000
    (Iteration 101 / 200) loss: 1.309097
    (Epoch 6 / 10) train acc: 0.651000; val_acc: 0.341000
    (Iteration 121 / 200) loss: 1.081845
    (Epoch 7 / 10) train acc: 0.695000; val_acc: 0.312000
    (Iteration 141 / 200) loss: 1.110367
    (Epoch 8 / 10) train acc: 0.745000; val_acc: 0.335000
    (Iteration 161 / 200) loss: 0.689478
    (Epoch 9 / 10) train acc: 0.795000; val_acc: 0.329000
    (Iteration 181 / 200) loss: 0.687122
    (Epoch 10 / 10) train acc: 0.836000; val_acc: 0.323000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.245000; val_acc: 0.212000
    (Iteration 21 / 200) loss: 2.062544
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.270000
    (Iteration 41 / 200) loss: 1.820483
    (Epoch 3 / 10) train acc: 0.340000; val_acc: 0.260000
    (Iteration 61 / 200) loss: 1.755108
    (Epoch 4 / 10) train acc: 0.382000; val_acc: 0.291000
    (Iteration 81 / 200) loss: 1.539316
    (Epoch 5 / 10) train acc: 0.405000; val_acc: 0.289000
    (Iteration 101 / 200) loss: 1.751918
    (Epoch 6 / 10) train acc: 0.461000; val_acc: 0.315000
    (Iteration 121 / 200) loss: 1.406195
    (Epoch 7 / 10) train acc: 0.481000; val_acc: 0.307000
    (Iteration 141 / 200) loss: 1.374834
    (Epoch 8 / 10) train acc: 0.551000; val_acc: 0.321000
    (Iteration 161 / 200) loss: 1.294639
    (Epoch 9 / 10) train acc: 0.594000; val_acc: 0.317000
    (Iteration 181 / 200) loss: 0.979029
    (Epoch 10 / 10) train acc: 0.611000; val_acc: 0.306000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    acc: 0.107000
    (Epoch 2 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 3 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 4 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 5 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 6 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 7 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 8 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 9 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 10 / 10) train acc: 0.112000; val_acc: 0.119000
    Running weight scale 5 / 20
    (Epoch 0 / 10) train acc: 0.120000; val_acc: 0.123000
    (Epoch 1 / 10) train acc: 0.175000; val_acc: 0.163000
    (Epoch 2 / 10) train acc: 0.230000; val_acc: 0.210000
    (Epoch 3 / 10) train acc: 0.252000; val_acc: 0.210000
    (Epoch 4 / 10) train acc: 0.251000; val_acc: 0.199000
    (Epoch 5 / 10) train acc: 0.275000; val_acc: 0.215000
    (Epoch 6 / 10) train acc: 0.275000; val_acc: 0.227000
    (Epoch 7 / 10) train acc: 0.273000; val_acc: 0.236000
    (Epoch 8 / 10) train acc: 0.310000; val_acc: 0.247000
    (Epoch 9 / 10) train acc: 0.311000; val_acc: 0.240000
    (Epoch 10 / 10) train acc: 0.320000; val_acc: 0.253000
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 1 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 2 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 3 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 4 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 5 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 6 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 7 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 8 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 9 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 10 / 10) train acc: 0.112000; val_acc: 0.119000
    Running weight scale 6 / 20
    (Epoch 0 / 10) train acc: 0.132000; val_acc: 0.135000
    (Epoch 1 / 10) train acc: 0.169000; val_acc: 0.183000
    (Epoch 2 / 10) train acc: 0.212000; val_acc: 0.218000
    (Epoch 3 / 10) train acc: 0.241000; val_acc: 0.200000
    (Epoch 4 / 10) train acc: 0.258000; val_acc: 0.230000
    (Epoch 5 / 10) train acc: 0.300000; val_acc: 0.234000
    (Epoch 6 / 10) train acc: 0.348000; val_acc: 0.259000
    (Epoch 7 / 10) train acc: 0.341000; val_acc: 0.252000
    (Epoch 8 / 10) train acc: 0.357000; val_acc: 0.245000
    (Epoch 9 / 10) train acc: 0.401000; val_acc: 0.274000
    (Epoch 10 / 10) train acc: 0.392000; val_acc: 0.258000
    (Epoch 0 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 1 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 2 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 3 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 4 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 5 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 6 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 7 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 8 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 9 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 10 / 10) train acc: 0.112000; val_acc: 0.119000
    Running weight scale 7 / 20
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 1 / 10) train acc: 0.164000; val_acc: 0.151000
    (Epoch 2 / 10) train acc: 0.201000; val_acc: 0.168000
    (Epoch 3 / 10) train acc: 0.219000; val_acc: 0.189000
    (Epoch 4 / 10) train acc: 0.301000; val_acc: 0.267000
    (Epoch 5 / 10) train acc: 0.310000; val_acc: 0.248000
    (Epoch 6 / 10) train acc: 0.333000; val_acc: 0.248000
    (Epoch 7 / 10) train acc: 0.361000; val_acc: 0.266000
    (Epoch 8 / 10) train acc: 0.365000; val_acc: 0.273000
    (Epoch 9 / 10) train acc: 0.383000; val_acc: 0.246000
    (Epoch 10 / 10) train acc: 0.422000; val_acc: 0.270000
    (Epoch 0 / 10) train acc: 0.099000; val_acc: 0.079000
    (Epoch 1 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 2 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 3 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 4 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 5 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 6 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 7 / 10) train acc: 0.112000; val_acc: 0.130000
    (Epoch 8 / 10) train acc: 0.195000; val_acc: 0.214000
    (Epoch 9 / 10) train acc: 0.182000; val_acc: 0.177000
    (Epoch 10 / 10) train acc: 0.193000; val_acc: 0.191000
    Running weight scale 8 / 20
    (Epoch 0 / 10) train acc: 0.085000; val_acc: 0.086000
    (Epoch 1 / 10) train acc: 0.164000; val_acc: 0.168000
    (Epoch 2 / 10) train acc: 0.250000; val_acc: 0.231000
    (Epoch 3 / 10) train acc: 0.310000; val_acc: 0.262000
    (Epoch 4 / 10) train acc: 0.306000; val_acc: 0.258000
    (Epoch 5 / 10) train acc: 0.335000; val_acc: 0.262000
    (Epoch 6 / 10) train acc: 0.358000; val_acc: 0.286000
    (Epoch 7 / 10) train acc: 0.365000; val_acc: 0.262000
    (Epoch 8 / 10) train acc: 0.385000; val_acc: 0.262000
    (Epoch 9 / 10) train acc: 0.421000; val_acc: 0.247000
    (Epoch 10 / 10) train acc: 0.461000; val_acc: 0.271000
    (Epoch 0 / 10) train acc: 0.099000; val_acc: 0.079000
    (Epoch 1 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 2 / 10) train acc: 0.099000; val_acc: 0.079000
    (Epoch 3 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 4 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 5 / 10) train acc: 0.133000; val_acc: 0.142000
    (Epoch 6 / 10) train acc: 0.172000; val_acc: 0.162000
    (Epoch 7 / 10) train acc: 0.183000; val_acc: 0.187000
    (Epoch 8 / 10) train acc: 0.179000; val_acc: 0.165000
    (Epoch 9 / 10) train acc: 0.191000; val_acc: 0.188000
    (Epoch 10 / 10) train acc: 0.179000; val_acc: 0.170000
    Running weight scale 9 / 20
    (Epoch 0 / 10) train acc: 0.131000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.186000; val_acc: 0.159000
    (Epoch 2 / 10) train acc: 0.212000; val_acc: 0.168000
    (Epoch 3 / 10) train acc: 0.284000; val_acc: 0.246000
    (Epoch 4 / 10) train acc: 0.304000; val_acc: 0.250000
    (Epoch 5 / 10) train acc: 0.333000; val_acc: 0.258000
    (Epoch 6 / 10) train acc: 0.333000; val_acc: 0.244000
    (Epoch 7 / 10) train acc: 0.347000; val_acc: 0.236000
    (Epoch 8 / 10) train acc: 0.377000; val_acc: 0.261000
    (Epoch 9 / 10) train acc: 0.412000; val_acc: 0.262000
    (Epoch 10 / 10) train acc: 0.409000; val_acc: 0.269000
    (Epoch 0 / 10) train acc: 0.099000; val_acc: 0.113000
    (Epoch 1 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 2 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 3 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 4 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 5 / 10) train acc: 0.105000; val_acc: 0.110000
    (Epoch 6 / 10) train acc: 0.164000; val_acc: 0.141000
    (Epoch 7 / 10) train acc: 0.170000; val_acc: 0.156000
    (Epoch 8 / 10) train acc: 0.178000; val_acc: 0.155000
    (Epoch 9 / 10) train acc: 0.191000; val_acc: 0.177000
    (Epoch 10 / 10) train acc: 0.206000; val_acc: 0.156000
    Running weight scale 10 / 20
    (Epoch 0 / 10) train acc: 0.098000; val_acc: 0.098000
    (Epoch 1 / 10) train acc: 0.204000; val_acc: 0.198000
    (Epoch 2 / 10) train acc: 0.247000; val_acc: 0.229000
    (Epoch 3 / 10) train acc: 0.308000; val_acc: 0.257000
    (Epoch 4 / 10) train acc: 0.352000; val_acc: 0.278000
    (Epoch 5 / 10) train acc: 0.371000; val_acc: 0.276000
    (Epoch 6 / 10) train acc: 0.384000; val_acc: 0.285000
    (Epoch 7 / 10) train acc: 0.390000; val_acc: 0.275000
    (Epoch 8 / 10) train acc: 0.436000; val_acc: 0.276000
    (Epoch 9 / 10) train acc: 0.455000; val_acc: 0.276000
    (Epoch 10 / 10) train acc: 0.493000; val_acc: 0.299000
    (Epoch 0 / 10) train acc: 0.102000; val_acc: 0.107000
    (Epoch 1 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 2 / 10) train acc: 0.112000; val_acc: 0.119000
    (Epoch 3 / 10) train acc: 0.179000; val_acc: 0.162000
    (Epoch 4 / 10) train acc: 0.186000; val_acc: 0.164000
    (Epoch 5 / 10) train acc: 0.161000; val_acc: 0.150000
    (Epoch 6 / 10) train acc: 0.197000; val_acc: 0.162000
    (Epoch 7 / 10) train acc: 0.206000; val_acc: 0.173000
    (Epoch 8 / 10) train acc: 0.205000; val_acc: 0.171000
    (Epoch 9 / 10) train acc: 0.206000; val_acc: 0.176000
    (Epoch 10 / 10) train acc: 0.227000; val_acc: 0.183000
    Running weight scale 11 / 20
    (Epoch 0 / 10) train acc: 0.096000; val_acc: 0.092000
    (Epoch 1 / 10) train acc: 0.215000; val_acc: 0.214000
    (Epoch 2 / 10) train acc: 0.283000; val_acc: 0.257000
    (Epoch 3 / 10) train acc: 0.334000; val_acc: 0.269000
    (Epoch 4 / 10) train acc: 0.373000; val_acc: 0.292000
    (Epoch 5 / 10) train acc: 0.398000; val_acc: 0.299000
    (Epoch 6 / 10) train acc: 0.430000; val_acc: 0.321000
    (Epoch 7 / 10) train acc: 0.484000; val_acc: 0.339000
    (Epoch 8 / 10) train acc: 0.510000; val_acc: 0.296000
    (Epoch 9 / 10) train acc: 0.536000; val_acc: 0.310000
    (Epoch 10 / 10) train acc: 0.578000; val_acc: 0.325000
    (Epoch 0 / 10) train acc: 0.103000; val_acc: 0.105000
    (Epoch 1 / 10) train acc: 0.119000; val_acc: 0.119000
    (Epoch 2 / 10) train acc: 0.165000; val_acc: 0.161000
    (Epoch 3 / 10) train acc: 0.188000; val_acc: 0.190000
    (Epoch 4 / 10) train acc: 0.205000; val_acc: 0.167000
    (Epoch 5 / 10) train acc: 0.188000; val_acc: 0.157000
    (Epoch 6 / 10) train acc: 0.194000; val_acc: 0.156000
    (Epoch 7 / 10) train acc: 0.203000; val_acc: 0.191000
    (Epoch 8 / 10) train acc: 0.202000; val_acc: 0.183000
    (Epoch 9 / 10) train acc: 0.210000; val_acc: 0.195000
    (Epoch 10 / 10) train acc: 0.205000; val_acc: 0.163000
    Running weight scale 12 / 20
    (Epoch 0 / 10) train acc: 0.097000; val_acc: 0.123000
    (Epoch 1 / 10) train acc: 0.223000; val_acc: 0.223000
    (Epoch 2 / 10) train acc: 0.332000; val_acc: 0.269000
    (Epoch 3 / 10) train acc: 0.340000; val_acc: 0.280000
    (Epoch 4 / 10) train acc: 0.380000; val_acc: 0.286000
    (Epoch 5 / 10) train acc: 0.423000; val_acc: 0.301000
    (Epoch 6 / 10) train acc: 0.462000; val_acc: 0.302000
    (Epoch 7 / 10) train acc: 0.510000; val_acc: 0.295000
    (Epoch 8 / 10) train acc: 0.546000; val_acc: 0.290000
    (Epoch 9 / 10) train acc: 0.583000; val_acc: 0.330000
    (Epoch 10 / 10) train acc: 0.604000; val_acc: 0.318000
    (Epoch 0 / 10) train acc: 0.103000; val_acc: 0.105000
    (Epoch 1 / 10) train acc: 0.137000; val_acc: 0.140000
    (Epoch 2 / 10) train acc: 0.197000; val_acc: 0.191000
    (Epoch 3 / 10) train acc: 0.224000; val_acc: 0.172000
    (Epoch 4 / 10) train acc: 0.221000; val_acc: 0.206000
    (Epoch 5 / 10) train acc: 0.237000; val_acc: 0.194000
    (Epoch 6 / 10) train acc: 0.283000; val_acc: 0.237000
    (Epoch 7 / 10) train acc: 0.291000; val_acc: 0.234000
    (Epoch 8 / 10) train acc: 0.314000; val_acc: 0.234000
    (Epoch 9 / 10) train acc: 0.323000; val_acc: 0.264000
    (Epoch 10 / 10) train acc: 0.337000; val_acc: 0.249000
    Running weight scale 13 / 20
    (Epoch 0 / 10) train acc: 0.088000; val_acc: 0.097000
    (Epoch 1 / 10) train acc: 0.253000; val_acc: 0.228000
    (Epoch 2 / 10) train acc: 0.310000; val_acc: 0.256000
    (Epoch 3 / 10) train acc: 0.352000; val_acc: 0.282000
    (Epoch 4 / 10) train acc: 0.422000; val_acc: 0.287000
    (Epoch 5 / 10) train acc: 0.444000; val_acc: 0.303000
    (Epoch 6 / 10) train acc: 0.518000; val_acc: 0.306000
    (Epoch 7 / 10) train acc: 0.526000; val_acc: 0.308000
    (Epoch 8 / 10) train acc: 0.575000; val_acc: 0.326000
    (Epoch 9 / 10) train acc: 0.642000; val_acc: 0.312000
    (Epoch 10 / 10) train acc: 0.688000; val_acc: 0.303000
    (Epoch 0 / 10) train acc: 0.109000; val_acc: 0.117000
    (Epoch 1 / 10) train acc: 0.174000; val_acc: 0.155000
    (Epoch 2 / 10) train acc: 0.196000; val_acc: 0.196000
    (Epoch 3 / 10) train acc: 0.194000; val_acc: 0.171000
    (Epoch 4 / 10) train acc: 0.235000; val_acc: 0.198000
    (Epoch 5 / 10) train acc: 0.257000; val_acc: 0.231000
    (Epoch 6 / 10) train acc: 0.279000; val_acc: 0.256000
    (Epoch 7 / 10) train acc: 0.317000; val_acc: 0.262000
    (Epoch 8 / 10) train acc: 0.307000; val_acc: 0.240000
    (Epoch 9 / 10) train acc: 0.331000; val_acc: 0.253000
    (Epoch 10 / 10) train acc: 0.347000; val_acc: 0.261000
    Running weight scale 14 / 20
    (Epoch 0 / 10) train acc: 0.091000; val_acc: 0.113000
    (Epoch 1 / 10) train acc: 0.251000; val_acc: 0.214000
    (Epoch 2 / 10) train acc: 0.339000; val_acc: 0.261000
    (Epoch 3 / 10) train acc: 0.400000; val_acc: 0.274000
    (Epoch 4 / 10) train acc: 0.452000; val_acc: 0.306000
    (Epoch 5 / 10) train acc: 0.488000; val_acc: 0.280000
    (Epoch 6 / 10) train acc: 0.532000; val_acc: 0.307000
    (Epoch 7 / 10) train acc: 0.589000; val_acc: 0.299000
    (Epoch 8 / 10) train acc: 0.614000; val_acc: 0.325000
    (Epoch 9 / 10) train acc: 0.669000; val_acc: 0.311000
    (Epoch 10 / 10) train acc: 0.704000; val_acc: 0.333000
    (Epoch 0 / 10) train acc: 0.140000; val_acc: 0.124000
    (Epoch 1 / 10) train acc: 0.207000; val_acc: 0.179000
    (Epoch 2 / 10) train acc: 0.302000; val_acc: 0.254000
    (Epoch 3 / 10) train acc: 0.313000; val_acc: 0.264000
    (Epoch 4 / 10) train acc: 0.353000; val_acc: 0.277000
    (Epoch 5 / 10) train acc: 0.395000; val_acc: 0.276000
    (Epoch 6 / 10) train acc: 0.398000; val_acc: 0.280000
    (Epoch 7 / 10) train acc: 0.459000; val_acc: 0.282000
    (Epoch 8 / 10) train acc: 0.444000; val_acc: 0.307000
    (Epoch 9 / 10) train acc: 0.516000; val_acc: 0.304000
    (Epoch 10 / 10) train acc: 0.531000; val_acc: 0.304000
    Running weight scale 15 / 20
    (Epoch 0 / 10) train acc: 0.120000; val_acc: 0.123000
    (Epoch 1 / 10) train acc: 0.205000; val_acc: 0.163000
    (Epoch 2 / 10) train acc: 0.295000; val_acc: 0.192000
    (Epoch 3 / 10) train acc: 0.349000; val_acc: 0.232000
    (Epoch 4 / 10) train acc: 0.414000; val_acc: 0.249000
    (Epoch 5 / 10) train acc: 0.469000; val_acc: 0.266000
    (Epoch 6 / 10) train acc: 0.486000; val_acc: 0.271000
    (Epoch 7 / 10) train acc: 0.554000; val_acc: 0.275000
    (Epoch 8 / 10) train acc: 0.615000; val_acc: 0.283000
    (Epoch 9 / 10) train acc: 0.623000; val_acc: 0.283000
    (Epoch 10 / 10) train acc: 0.674000; val_acc: 0.285000
    (Epoch 0 / 10) train acc: 0.105000; val_acc: 0.095000
    (Epoch 1 / 10) train acc: 0.281000; val_acc: 0.197000
    (Epoch 2 / 10) train acc: 0.330000; val_acc: 0.269000
    (Epoch 3 / 10) train acc: 0.403000; val_acc: 0.283000
    (Epoch 4 / 10) train acc: 0.460000; val_acc: 0.297000
    (Epoch 5 / 10) train acc: 0.492000; val_acc: 0.284000
    (Epoch 6 / 10) train acc: 0.548000; val_acc: 0.300000
    (Epoch 7 / 10) train acc: 0.567000; val_acc: 0.308000
    (Epoch 8 / 10) train acc: 0.648000; val_acc: 0.289000
    (Epoch 9 / 10) train acc: 0.667000; val_acc: 0.304000
    (Epoch 10 / 10) train acc: 0.689000; val_acc: 0.302000
    Running weight scale 16 / 20
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.102000
    (Epoch 1 / 10) train acc: 0.189000; val_acc: 0.152000
    (Epoch 2 / 10) train acc: 0.323000; val_acc: 0.195000
    (Epoch 3 / 10) train acc: 0.404000; val_acc: 0.209000
    (Epoch 4 / 10) train acc: 0.455000; val_acc: 0.245000
    (Epoch 5 / 10) train acc: 0.501000; val_acc: 0.241000
    (Epoch 6 / 10) train acc: 0.524000; val_acc: 0.253000
    (Epoch 7 / 10) train acc: 0.581000; val_acc: 0.246000
    (Epoch 8 / 10) train acc: 0.611000; val_acc: 0.251000
    (Epoch 9 / 10) train acc: 0.652000; val_acc: 0.261000
    (Epoch 10 / 10) train acc: 0.670000; val_acc: 0.277000
    (Epoch 0 / 10) train acc: 0.100000; val_acc: 0.091000
    (Epoch 1 / 10) train acc: 0.129000; val_acc: 0.116000
    (Epoch 2 / 10) train acc: 0.200000; val_acc: 0.127000
    (Epoch 3 / 10) train acc: 0.242000; val_acc: 0.131000
    (Epoch 4 / 10) train acc: 0.288000; val_acc: 0.149000
    (Epoch 5 / 10) train acc: 0.352000; val_acc: 0.148000
    (Epoch 6 / 10) train acc: 0.410000; val_acc: 0.152000
    (Epoch 7 / 10) train acc: 0.420000; val_acc: 0.148000
    (Epoch 8 / 10) train acc: 0.443000; val_acc: 0.138000
    (Epoch 9 / 10) train acc: 0.481000; val_acc: 0.163000
    (Epoch 10 / 10) train acc: 0.511000; val_acc: 0.146000
    Running weight scale 17 / 20
    (Epoch 0 / 10) train acc: 0.091000; val_acc: 0.095000
    (Epoch 1 / 10) train acc: 0.164000; val_acc: 0.134000
    (Epoch 2 / 10) train acc: 0.285000; val_acc: 0.161000
    (Epoch 3 / 10) train acc: 0.354000; val_acc: 0.179000
    (Epoch 4 / 10) train acc: 0.383000; val_acc: 0.190000
    (Epoch 5 / 10) train acc: 0.430000; val_acc: 0.212000
    (Epoch 6 / 10) train acc: 0.464000; val_acc: 0.216000
    (Epoch 7 / 10) train acc: 0.498000; val_acc: 0.234000
    (Epoch 8 / 10) train acc: 0.517000; val_acc: 0.230000
    (Epoch 9 / 10) train acc: 0.538000; val_acc: 0.225000
    (Epoch 10 / 10) train acc: 0.559000; val_acc: 0.232000
    (Epoch 0 / 10) train acc: 0.096000; val_acc: 0.088000
    (Epoch 1 / 10) train acc: 0.157000; val_acc: 0.107000
    (Epoch 2 / 10) train acc: 0.219000; val_acc: 0.139000
    (Epoch 3 / 10) train acc: 0.283000; val_acc: 0.144000
    (Epoch 4 / 10) train acc: 0.336000; val_acc: 0.138000
    (Epoch 5 / 10) train acc: 0.388000; val_acc: 0.133000
    (Epoch 6 / 10) train acc: 0.396000; val_acc: 0.149000
    (Epoch 7 / 10) train acc: 0.454000; val_acc: 0.137000
    (Epoch 8 / 10) train acc: 0.523000; val_acc: 0.142000
    (Epoch 9 / 10) train acc: 0.547000; val_acc: 0.148000
    (Epoch 10 / 10) train acc: 0.586000; val_acc: 0.147000
    Running weight scale 18 / 20
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.098000
    (Epoch 1 / 10) train acc: 0.149000; val_acc: 0.101000
    (Epoch 2 / 10) train acc: 0.240000; val_acc: 0.146000
    (Epoch 3 / 10) train acc: 0.323000; val_acc: 0.167000
    (Epoch 4 / 10) train acc: 0.350000; val_acc: 0.203000
    (Epoch 5 / 10) train acc: 0.413000; val_acc: 0.201000
    (Epoch 6 / 10) train acc: 0.428000; val_acc: 0.200000
    (Epoch 7 / 10) train acc: 0.463000; val_acc: 0.193000
    (Epoch 8 / 10) train acc: 0.484000; val_acc: 0.213000
    (Epoch 9 / 10) train acc: 0.524000; val_acc: 0.206000
    (Epoch 10 / 10) train acc: 0.549000; val_acc: 0.213000
    (Epoch 0 / 10) train acc: 0.112000; val_acc: 0.106000
    (Epoch 1 / 10) train acc: 0.125000; val_acc: 0.094000
    (Epoch 2 / 10) train acc: 0.201000; val_acc: 0.118000
    (Epoch 3 / 10) train acc: 0.244000; val_acc: 0.121000
    (Epoch 4 / 10) train acc: 0.286000; val_acc: 0.126000
    (Epoch 5 / 10) train acc: 0.331000; val_acc: 0.128000
    (Epoch 6 / 10) train acc: 0.379000; val_acc: 0.108000
    (Epoch 7 / 10) train acc: 0.397000; val_acc: 0.119000
    (Epoch 8 / 10) train acc: 0.464000; val_acc: 0.111000
    (Epoch 9 / 10) train acc: 0.480000; val_acc: 0.124000
    (Epoch 10 / 10) train acc: 0.504000; val_acc: 0.118000
    Running weight scale 19 / 20
    (Epoch 0 / 10) train acc: 0.102000; val_acc: 0.088000
    (Epoch 1 / 10) train acc: 0.130000; val_acc: 0.108000
    (Epoch 2 / 10) train acc: 0.193000; val_acc: 0.126000
    (Epoch 3 / 10) train acc: 0.237000; val_acc: 0.134000
    (Epoch 4 / 10) train acc: 0.265000; val_acc: 0.150000
    (Epoch 5 / 10) train acc: 0.292000; val_acc: 0.162000
    (Epoch 6 / 10) train acc: 0.338000; val_acc: 0.152000
    (Epoch 7 / 10) train acc: 0.366000; val_acc: 0.149000
    (Epoch 8 / 10) train acc: 0.380000; val_acc: 0.167000
    (Epoch 9 / 10) train acc: 0.395000; val_acc: 0.165000
    (Epoch 10 / 10) train acc: 0.413000; val_acc: 0.167000
    (Epoch 0 / 10) train acc: 0.088000; val_acc: 0.094000
    (Epoch 1 / 10) train acc: 0.175000; val_acc: 0.120000
    (Epoch 2 / 10) train acc: 0.218000; val_acc: 0.128000
    (Epoch 3 / 10) train acc: 0.251000; val_acc: 0.133000
    (Epoch 4 / 10) train acc: 0.308000; val_acc: 0.133000
    (Epoch 5 / 10) train acc: 0.328000; val_acc: 0.128000
    (Epoch 6 / 10) train acc: 0.382000; val_acc: 0.129000
    (Epoch 7 / 10) train acc: 0.401000; val_acc: 0.128000
    (Epoch 8 / 10) train acc: 0.456000; val_acc: 0.136000
    (Epoch 9 / 10) train acc: 0.455000; val_acc: 0.133000
    (Epoch 10 / 10) train acc: 0.496000; val_acc: 0.134000
    Running weight scale 20 / 20
    (Epoch 0 / 10) train acc: 0.097000; val_acc: 0.093000
    (Epoch 1 / 10) train acc: 0.110000; val_acc: 0.104000
    (Epoch 2 / 10) train acc: 0.136000; val_acc: 0.121000
    (Epoch 3 / 10) train acc: 0.170000; val_acc: 0.114000
    (Epoch 4 / 10) train acc: 0.173000; val_acc: 0.118000
    (Epoch 5 / 10) train acc: 0.205000; val_acc: 0.126000
    (Epoch 6 / 10) train acc: 0.232000; val_acc: 0.131000
    (Epoch 7 / 10) train acc: 0.238000; val_acc: 0.137000
    (Epoch 8 / 10) train acc: 0.259000; val_acc: 0.146000
    (Epoch 9 / 10) train acc: 0.276000; val_acc: 0.143000
    (Epoch 10 / 10) train acc: 0.294000; val_acc: 0.154000
    (Epoch 0 / 10) train acc: 0.106000; val_acc: 0.088000
    (Epoch 1 / 10) train acc: 0.140000; val_acc: 0.139000
    (Epoch 2 / 10) train acc: 0.166000; val_acc: 0.138000
    (Epoch 3 / 10) train acc: 0.208000; val_acc: 0.138000
    (Epoch 4 / 10) train acc: 0.244000; val_acc: 0.136000
    (Epoch 5 / 10) train acc: 0.269000; val_acc: 0.149000
    (Epoch 6 / 10) train acc: 0.299000; val_acc: 0.143000
    (Epoch 7 / 10) train acc: 0.318000; val_acc: 0.139000
    (Epoch 8 / 10) train acc: 0.348000; val_acc: 0.148000
    (Epoch 9 / 10) train acc: 0.351000; val_acc: 0.142000
    (Epoch 10 / 10) train acc: 0.375000; val_acc: 0.147000

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:

* The second plot shows the problem of vanishing gradients (small initial weights). The baseline model is very sensitive to this problem (the accuracy is very low), therefore finding the correct weight scale is difficult. For this example, the baseline obtains the best result with a weight scale equal to 1e-1. On the other hand, we can see that the batchnorm model is less sensitive to weight initialization because its accuracy is around 30% for a greater range of weight scales.
* The behavior of the first plot is very similar to that of the second plot. The main difference is that the first plot shows that we are overfitting our model, besides that we can see that with the batchnorm model we obtained better results than the baseline model and that occurs because batch normalization has regularization properties.
* The third plot depicts the problem of exploding gradients and it is very evident in the baseline model for weight scale values greater than 1e-1. However, the batchnorm model does not suffer from this problem.

In general with batch normalization we can avoid the problem of vanishing and exploding gradients because it normalizes every affine layer (xW+b), avoiding very large/small values. Moreover, its regularization properties allow to decrease overfitting.

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    (Epoch 0 / 10) train acc: 0.106000; val_acc: 0.108000
    (Epoch 1 / 10) train acc: 0.245000; val_acc: 0.206000
    (Epoch 2 / 10) train acc: 0.272000; val_acc: 0.233000
    (Epoch 3 / 10) train acc: 0.349000; val_acc: 0.278000
    (Epoch 4 / 10) train acc: 0.357000; val_acc: 0.278000
    (Epoch 5 / 10) train acc: 0.437000; val_acc: 0.284000
    (Epoch 6 / 10) train acc: 0.468000; val_acc: 0.287000
    (Epoch 7 / 10) train acc: 0.517000; val_acc: 0.311000
    (Epoch 8 / 10) train acc: 0.566000; val_acc: 0.337000
    (Epoch 9 / 10) train acc: 0.593000; val_acc: 0.340000
    (Epoch 10 / 10) train acc: 0.617000; val_acc: 0.295000
    Normalization: batch size =  5
    (Epoch 0 / 10) train acc: 0.095000; val_acc: 0.114000
    (Epoch 1 / 10) train acc: 0.270000; val_acc: 0.237000
    (Epoch 2 / 10) train acc: 0.288000; val_acc: 0.275000
    (Epoch 3 / 10) train acc: 0.325000; val_acc: 0.289000
    (Epoch 4 / 10) train acc: 0.371000; val_acc: 0.309000
    (Epoch 5 / 10) train acc: 0.414000; val_acc: 0.311000
    (Epoch 6 / 10) train acc: 0.426000; val_acc: 0.309000
    (Epoch 7 / 10) train acc: 0.413000; val_acc: 0.322000
    (Epoch 8 / 10) train acc: 0.457000; val_acc: 0.323000
    (Epoch 9 / 10) train acc: 0.439000; val_acc: 0.322000
    (Epoch 10 / 10) train acc: 0.481000; val_acc: 0.344000
    Normalization: batch size =  10
    (Epoch 0 / 10) train acc: 0.103000; val_acc: 0.093000
    (Epoch 1 / 10) train acc: 0.324000; val_acc: 0.271000
    (Epoch 2 / 10) train acc: 0.385000; val_acc: 0.284000
    (Epoch 3 / 10) train acc: 0.425000; val_acc: 0.316000
    (Epoch 4 / 10) train acc: 0.474000; val_acc: 0.330000
    (Epoch 5 / 10) train acc: 0.516000; val_acc: 0.324000
    (Epoch 6 / 10) train acc: 0.585000; val_acc: 0.326000
    (Epoch 7 / 10) train acc: 0.585000; val_acc: 0.325000
    (Epoch 8 / 10) train acc: 0.613000; val_acc: 0.327000
    (Epoch 9 / 10) train acc: 0.657000; val_acc: 0.337000
    (Epoch 10 / 10) train acc: 0.710000; val_acc: 0.338000
    Normalization: batch size =  50
    (Epoch 0 / 10) train acc: 0.111000; val_acc: 0.127000
    (Epoch 1 / 10) train acc: 0.290000; val_acc: 0.225000
    (Epoch 2 / 10) train acc: 0.416000; val_acc: 0.306000
    (Epoch 3 / 10) train acc: 0.515000; val_acc: 0.325000
    (Epoch 4 / 10) train acc: 0.579000; val_acc: 0.317000
    (Epoch 5 / 10) train acc: 0.663000; val_acc: 0.333000
    (Epoch 6 / 10) train acc: 0.708000; val_acc: 0.337000
    (Epoch 7 / 10) train acc: 0.784000; val_acc: 0.342000
    (Epoch 8 / 10) train acc: 0.847000; val_acc: 0.338000
    (Epoch 9 / 10) train acc: 0.869000; val_acc: 0.347000
    (Epoch 10 / 10) train acc: 0.889000; val_acc: 0.339000

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:

According to the results, we can see that the batch size affects directly the performance of batch normalization (the smaller the batch size the worse the performance). Even the baseline model outperforms the batchnorm model when using a very small batch size. This problem occurs because when we calculate the statistics of a batch, i.e., mean and variance, we try to find an approximation of the statistics of the entire dataset. Therefore with a small batch size, these statistics can be very noisy. On the other hand, with a large batch size we can obtain a better approximation.

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:

Number 2 is analogous to layer normalization when we consider: mean = 0, beta parameter = 0 (at this point we have gamma*x/std where std=sqrt(sum(x^2))) and gamma=x/std. Thus the result of layer normalization will be x^2/sum(x^2).

Number 3 is analogous to batch normalization when we consider: batch size = size of the dataset, gamma parameter = standard deviation and beta parameter = 0. Thus the result of batch normalization will be std*(x-mean)/std + 0 = x-mean.

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.4336160411201157e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:

1. \[Will work well\] Like batch normalization, layer normalization helps mitigate the curse of vanishing gradients in very deep networks.

2. \[Will not work well\] Just as batch normalization fails at small batch sizes, layer normalization fails at small feature sizes. This is intuitive - layer normalization performs normalization across the feature dimension; the strength of layer normalization grows and diminishes with the size of this feature dimension.

3. \[Will not work well\] High regularization causes underfitting generally.


```python
%matplotlib inline
```

05: Aperiodic Component Fitting
===============================

Choosing and using different approaches for fitting the aperiodic component.

```python
# Import the FOOOF object
from fooof import FOOOF

# Import utility to download and load example data
from fooof.utils.download import load_fooof_data
```

Aperiodic Fitting Approaches
----------------------------

There are currently two approaches for fitting the aperiodic component:

- Fitting with just an offset and a exponent, equivalent to a linear fit in log-log space

  - `aperiodic_mode` = 'fixed'
- Including a 'knee' parameter, reflecting a fit with a bend, in log-log space

  - `aperiodic_mode` = 'knee'

Fitting in the 'fixed' mode assumes a single 1/f like characteristic to the aperiodic
component, meaning it looks linear across all frequencies in log-log space.

Though this assumption is true across *some* frequency ranges in neural data, it does
does not hold up across broad frequency ranges. If fitting is done in the 'fixed' mode,
but the assumption of a single 1/f is violated, then fitting will go wrong.

Broad frequency ranges (typically ranges greater than ~40 Hz range) don't meet this
criterion, as they typically exhibit a 'bend' in the aperiodic component, whereby there is
not a single 1/f property across all frequencies, but rather a 'bend' in the aperiodic
component. For these cases, fitting should be done using an extra parameter to capture
this, in 'knee' mode.

Fitting with an Aperiodic 'Knee'
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Let's explore fitting power spectrum models across a broader frequency range,
using some local field potential data.

```python
# Load example data files needed for this example
freqs = load_fooof_data('freqs_lfp.npy', folder='data')
spectrum = load_fooof_data('spectrum_lfp.npy', folder='data')
```

```python
# Initialize a FOOOF object, setting the aperiodic mode to use a 'knee' fit
fm = FOOOF(peak_width_limits=[2, 8], aperiodic_mode='knee')
```

```python
# Fit a power spectrum model
#  Note that this time we're specifying an optional parameter to plot in log-log
fm.report(freqs, spectrum, [2, 70], plt_log=True)
```

A note on interpreting the 'knee' parameter
-------------------------------------------

The aperiodic fit has the form:

\begin{align}AP = 10^b * \ \frac{1}{(k + F^\chi)}\end{align}

The knee parameter reported above corresponds to `k` in the equation.

This parameter is dependent on the frequency at which the aperiodic fit
transitions from horizontal to negatively sloped.

To interpret this parameter as a frequency, take the $\chi$-th root of `k`, i.e.:

\begin{align}knee \ frequency = k^{1/\ \chi}\end{align}

Interpreting the fit results when using knee fits is more complex, as the exponent result is
no longer a simple measure of the aperiodic component as a whole, but instead reflects the
aperiodic component past the 'knee' inflecting point. Because of this, when doing knee fits,
knee & exponent values should be considered together.

Example: Aperiodic Fitting Gone Wrong
-------------------------------------

In the example above, we jumped directly to fitting with a knee.

Here we will explore what it looks like if we don't use the appropriate mode for fitting
the aperiodic component - fitting in 'fixed' mode when we should use 'knee'

```python
# Create and fit a power spectrum model in fixed mode to the same data as above
fm = FOOOF(peak_width_limits=[2, 8], aperiodic_mode='fixed')
fm.report(freqs, spectrum, [2, 70], plt_log=True)
```

In this case, we see that the 'fixed' aperiodic component (equivalent to a line
in log-log space) is not able to capture the data, which has a curve.

To compensate, the model adds extra peaks, but these are also not a good
characterization of the data.

In this example, the data, over this frequency range, needs to be fit in
'knee' mode to be able to appropriately characterize the data.

Choosing an Aperiodic Fitting Procedure
---------------------------------------

It is important to choose the appropriate aperiodic fitting approach for your data.

If there is a clear knee in the power spectrum, fitting in 'fixed' mode will not
work well. However fitting with a knee may perform sub-optimally in ambiguous cases
(where the data may or may not have a knee), or if no knee is present.

Given this, we recommend:

- Check your data, across the frequency range of interest,
  for what the aperiodic component looks like.

  - If it looks roughly linear (in log-log space), fit without a knee.

    - This is likely across smaller frequency ranges, such as 3-30.
    - Do not perform no-knee fits across a range in which this does not hold.
  - If there is a clear knee, then use knee fits.

    - This is likely across larger fitting ranges such as 1-150 Hz.
- Be wary of ambiguous ranges, where there may or may not be a knee.

  - Trying to fit without a knee, when there is not a single consistent aperiodic component,
    can lead to very bad fits. But it is also a known issue that trying to fit with a knee
    can lead to suboptimal fits when no knee is present.

    - We therefore currently recommend picking frequency ranges in which the expected
      aperiodic component process is relatively clear.

Conclusion
----------

We have now explored the :class:`~fooof.FOOOF` object, and different fitting
approaches for the aperiodic component. Next up, we will be introducing how
to scale the fitting to apply across multiple power spectra.


# Using generalized tracers and power spectra
This example showcases how to use the generalized tracers and 2D power spectra implemented in CCL

```python
import numpy as np
import pylab as plt
import pyccl as ccl
%matplotlib inline
```

## Preliminaries
Let's just begin by setting up a cosmology and a couple of redshift distributions and bias functions

```python
# Cosmology
cosmo = ccl.Cosmology(Omega_c=0.27, Omega_b=0.045, h=0.67, A_s=2.1e-9, n_s=0.96)

# Redshift-dependent functions
z = np.linspace(0,1.2,1024)
# Redshift distributions
nz1 = np.exp(-((z-0.5)/0.05)**2/2)
nz2 = np.exp(-((z-0.65)/0.05)**2/2)
# Bias
bz = 0.95/ccl.growth_factor(cosmo,1./(1+z))
# Magnification bias
sz = np.ones_like(z)
# Intrinsic alignment amplitude
az = -0.004 * np.ones_like(z)
```

## Standard tracers
There are a set of standard tracers with built-in constructors.
First, there is number counts tracers (a.k.a. galaxy clustering). These include the standard contribution from the matter overdensity multiplied by a linear, scale-independent bias, redshift-space distortions and magnification. You can turn off any of them.

```python
# This tracer will only include the density contribution
gc_d = ccl.NumberCountsTracer(cosmo, has_rsd=False, dndz=(z,nz1), bias=(z,bz), mag_bias=None)
# If you pass bias=None, then the tracer will not have a density term. E.g. this one is an RSD-only tracer.
gc_r = ccl.NumberCountsTracer(cosmo, has_rsd=True, dndz=(z,nz1), bias=None, mag_bias=None)
# Now a magnification-only tracer. To turn on magnification just pass a non-None mag_bias.
gc_m = ccl.NumberCountsTracer(cosmo, has_rsd=False, dndz=(z,nz1), bias=None, mag_bias=(z,sz))
# You can of course create a tracer that includes all of these terms
gc_a = ccl.NumberCountsTracer(cosmo, has_rsd=True, dndz=(z,nz1), bias=(z,bz), mag_bias=(z,sz))
```

There are also standard weak lensing tracers with two terms: the cosmic shear term due to lensing and intrinsic alignments within the L-NLA model with a scale-independent IA amplitude.

```python
# This tracer will only include the lensing shear contribution (by setting the ia_bias to None)
wl_s = ccl.WeakLensingTracer(cosmo, dndz=(z,nz2), has_shear=True, ia_bias=None)
# This tracer will only include IAs
wl_i = ccl.WeakLensingTracer(cosmo, dndz=(z,nz2), has_shear=False, ia_bias=(z,az))
# And you can have the full monty
wl_a = ccl.WeakLensingTracer(cosmo, dndz=(z,nz2), has_shear=True, ia_bias=(z,az))
```

Finally, we also have CMB lensing tracers, defined by a single source redshift.

```python
# We put the source at z=1100, but you can put it at any other redshift if you want to!
# It won't be CMB lensing though.
cmbl = ccl.CMBLensingTracer(cosmo, z_source=1100.)
```

## Custom tracers

In general, tracers are things you can cross-correlate and get a power spectrum from. In the most general case, the angular power spectrum between two tracers is given by

\begin{equation}
   C^{\alpha\beta}_\ell=\frac{2}{\pi}\int d\chi_1\,d\chi_2\,dk\,k^2 P_{\alpha\beta}(k,\chi_1,\chi_2)\,\Delta^\alpha_\ell(k,\chi_1)\,\Delta^\beta_\ell(k,\chi_2).
\end{equation},

where $P_{\alpha\beta}$ is a generalized power spectrum (see below), and $\Delta^\alpha_\ell(k,\chi)$ is a sum over different contributions associated to tracer $\alpha$, where every contribution takes the form:
\begin{equation}
  \Delta^\alpha_\ell(k,\chi)=f^\alpha_\ell\,W_\alpha(\chi)\,T_\alpha(k,\chi)\,j^{(n_\alpha)}_\ell(k\chi).
\end{equation}

Here, $f^\alpha_\ell$ is an **$\ell$-dependent prefactor**, usually associated with angular derivatives, $W_\alpha(\chi)$ is the **radial kernel**, dependent only on redshift/distance, $T_\alpha(k,\chi)$ is the **transfer function**, dependent on both $k$ and $z/\chi$, and $j^{(n)}_\ell(x)$ is a generalized versions of the **spherical Bessel functions**, associated with radial derivatives or inverse Laplacians.

Generalized tracers can be created as instances of the `Tracer` class. This is done in two steps:
  1. First, generate an empty tracer:
  2. Then, add a contribution to this tracer. A contribution is defined by the 4 parameters above:
   - The **$\ell$-dependent prefactor** is defined by the argument `der_angles`. This can be 0, 1 or 2. 0 means no prefactor, 1 means a prefactor $\ell(\ell+1)$ and 2 means a prefactor $\sqrt{(\ell+2)!/(\ell-2)!}$.
   - The form of the **spherical Bessel function** is determined by the argument `der_bessel`, which can be -1, 0, 1 or 2. For 0, 1 or 2, this determines the derivative of the Bessel function that this contribution uses. For `der_bessel=-1`, $j^{(-1)}_\ell(x)=j_\ell(x)/x$.
   - The **radial kernel** is determined by two arrays corresponding to $\chi$ and $W(\chi)$, wrapped into a tuple and passed as `kernel`. If `kernel=None`, CCL assumes that the radial kernel is 1 everywhere for this tracer.
   - The **transfer function** can be passed in two different forms. If your transfer function is factorizable, then use two parameters: `transfer_k` and `transfer_a`, corresponding to the $k$-dependent and redshift-dependent factors respectively. `transfer_k` should be a tuple of two arrays $(\ln(k),K(k))$, and `transfer_a` should be $(a,A(a))$, where $a$ is the scale factor (which should be monotonically increasing!). The total transfer function is then given by $T(a,k)=K(k) A(a)$. If your transfer function is not factorizable, then use the argument `transfer_ka`. This should be a tuple of three arrays $(a,\ln(k),T(a,k))$, where $a$ is an array of scale factors in ascending order, $\ln(k)$ is an array containing the logarithms of wavenumbers, and $T(a,k)$ is a 2D array containing the values of the transfer function at the corresponding values of $a$ and $k$. If $a$ and $\ln(k)$ have sizes $n_a$ and $n_k$ respectively, then $T(a,k)$ should have shape $(n_a,n_k)$. If `transfer_ka`, `transfer_a` or `transfer_k` are `None`, CCL assumes that the corresponding quantities are 1 everywhere.

Then, keep adding contributions until you're done.

Let's try this out by building a number-counts tracer that contains a linear bias, RSDs and magnification. We could do this easily with the built-in constructors above, but it's always satisfying when you build something with your own hands.

```python
# First, initialize empty tracer
gc_custom = ccl.Tracer()
```

```python
# Now we need to generate a radial kernel and a transfer function.
# To generate the radial kernel we can use a convenience function that exists in CCL:
kernel = ccl.get_density_kernel(cosmo, (z,nz1))
# Let's plot it out of curiosity:
plt.plot(kernel[0],kernel[1])
plt.xlabel('$\\chi\\,[{\\rm Mpc}]$',fontsize=14)
plt.ylabel('$W(\\chi)$',fontsize=14)
plt.show()
```

```python
# We also need a transfer function.
# Because this tracer has a simple linear bias, the transfer function
# is only z-dependent and not k-dependent. So it's factorizable.
# The scale factor values need to be increasing, so we'll reverse
# The order of all arrays:
sf = (1./(1+z))[::-1]
transfer_a = (sf,bz[::-1])
```

```python
# Now we're ready to add the density contribution to the tracer:
gc_custom.add_tracer(cosmo, kernel=kernel, transfer_a=transfer_a)
```

```python
# Now let's add RSDs.
# In this case the transfer function is just given by the growth rate.
# Also, RSDs use the second derivative of the Bessel function
transfer_a_rsd = (sf, -ccl.growth_rate(cosmo,sf))
gc_custom.add_tracer(cosmo, kernel=kernel, transfer_a=transfer_a_rsd, der_bessel=2)
```

```python
# And now magnification.
# Again, there is a built-in function to compute the radial kernel for
# lensing observables.
# For magnification, der_bessel=-1 and der_angles=1.
# The transfer function is just 1 everywhere
chis, w = ccl.get_lensing_kernel(cosmo, (z,nz1), mag_bias=(z,sz))
kernel_m = (chis, -2*w)
gc_custom.add_tracer(cosmo, kernel=kernel_m, der_bessel=-1, der_angles=1)
```

```python
# Now let's check that both tracers, the one we just
# created and the original one created using the built-in constructors
# give exactly the same result. To do that, let's compute their
# angular power spectra
ells = np.geomspace(2,1000,20)
cl_gc_a = ccl.angular_cl(cosmo, gc_a, gc_a, ells)
cl_gc_custom = ccl.angular_cl(cosmo, gc_custom, gc_custom, ells)

# Let's plot the result
plt.plot(ells, 1E4*cl_gc_a, 'r-', label='built-in tracer')
plt.plot(ells, 1E4*cl_gc_custom, 'k--', label='custom tracer')
plt.xscale('log')
plt.xlabel('$\\ell$', fontsize=14)
plt.ylabel('$10^3\\times C_\\ell$', fontsize=14)
plt.legend(loc='upper right', fontsize=12, frameon=False)
plt.show()
```

OK, but so far we haven't done anything that we couldn't have done with the built-in tracers. Let's have some fun!

How about adding a contribution to the custom tracer corresponding to the effect of non-Gaussianity. Non-Gaussianity generates a scale-dependent bias on biased tracers that becomes relevant on large scales. Using Eq. 9 of arXiv:0710.4560, the bias receives an additive term of the form:
\begin{equation}
 \Delta b = (b-1)f_{\rm NL}\frac{3\,\delta_c\,\Omega_M\,H_0^2}{D(a)\,k^2}.
\end{equation}
So we can include this effect as an additional contribution to the custom tracer that looks like a standard galaxy clustering term with this extra bias. Since the corresponding transfer function is still factorizable, this should be pretty easy:

```python
# Let's create an array of k values and the corresponding
# k-dependent part of the transfer function
ks = np.logspace(-5,2,512)
transfer_k_fnl = (np.log(ks),1./ks**2)

# Now the z-dependent part of the transfer function
fNL = 10
delta_c = 1.686
H0 = (cosmo.cosmo.params.h/ccl.physical_constants.CLIGHT_HMPC)
Omega_M = cosmo.cosmo.params.Omega_c+cosmo.cosmo.params.Omega_b
transfer_a_fnl = (sf, (bz[::-1]-1)*3*fNL*delta_c*Omega_M*H0**2/ccl.growth_factor(cosmo,sf))

# Now let's add that contribution
gc_custom.add_tracer(cosmo, kernel=kernel, transfer_a=transfer_a_fnl,
                     transfer_k=transfer_k_fnl)

# And let's check the effect in the power spectrum:
cl_gc_custom_fnl = ccl.angular_cl(cosmo, gc_custom, gc_custom, ells)

# Let's plot the result
plt.plot(ells, 1E4*cl_gc_custom, 'r-', label='$f_{\\rm NL}=0$')
plt.plot(ells, 1E4*cl_gc_custom_fnl, 'k--', label='$f_{\\rm NL}$=%d'%(int(fNL)))
plt.xscale('log')
plt.xlabel('$\\ell$', fontsize=14)
plt.ylabel('$10^3\\times C_\\ell$', fontsize=14)
plt.legend(loc='upper right', fontsize=12, frameon=False)
plt.show()
```

## Generalized power spectra

By default, when you call `ccl.angular_cl`, CCL will use the non-linear matter power spectrum in the equation for $C_\ell$ above to compute the angular power spectrum. You can however, pass a more general `Pk2D` object. These objects encapsulate generic power spectra that depend simultaneously on $k$ and $a$. They are versatile and can be created in different ways, but for our purposes here, we will generate them in the same way that we generated the tracer transfer functions (i.e. through arrays of scale factors and wavenumbers). For more information, check out the documentation for this class.

To illustrate the use of `Pk2D` objects, let's read off the matter power spectrum from CCL, include a high-$k$ Gaussian cutoff, and generate a `Pk2D` object from that.

```python
# OK, let's first read off the matter power spectrum:
lpk_array = np.log(np.array([ccl.nonlin_matter_power(cosmo,ks,a) for a in sf]))

# Now let's impose a cutoff at a scale k_cut corresponding to roughly ~10 Mpc
k_cut = 0.4
lpk_array -= (ks/k_cut)**2

# Create a Pk2D object
pk_cut = ccl.Pk2D(a_arr=sf, lk_arr=np.log(ks), pk_arr=lpk_array, is_logp=True)

```

OK, now let's check out the impact of the cutoff on the angular power spectrum.

```python
# Compute power spectra with and without cutoff
cl_gc_ncut = ccl.angular_cl(cosmo, gc_d, gc_d, ells)
cl_gc_ycut = ccl.angular_cl(cosmo, gc_d, gc_d, ells, p_of_k_a=pk_cut)

# Plot stuff
plt.plot(ells, 1E4*cl_gc_ncut, 'r-', label='No cut-off')
plt.plot(ells, 1E4*cl_gc_ycut, 'k--', label='Cut-off at $k_* = %.1lf\,{\\rm Mpc}^{-1}$'%(k_cut))
plt.xscale('log')
plt.yscale('log')
plt.xlabel('$\\ell$', fontsize=14)
plt.ylabel('$10^3\\times C_\\ell$', fontsize=14)
plt.legend(loc='lower left', fontsize=12, frameon=False)
plt.show()
```

```python

```


```python
import holoviews as hv
hv.extension('bokeh')
hv.opts.defaults(hv.opts.Curve(width=500),
                 hv.opts.Histogram(width=500),
                 hv.opts.HLine(alpha=0.5, color='k', line_dash='dashed'))
```

```python
import numpy as np
import scipy.stats
```

# Markov Chain Monte Carlo (MCMC)

## Usos principales de las cadenas de Markov

Las cadenas de Markov tienen dos usos principales. En primer lugar las cadenas de Markov se ocupan como **modelo o aproximación de fenómenos que evolucionan en el tiempo**. Esto es lo que vimos la lección anterior.

En estos casos corresponde hacer la pregunta de si acaso el fenómeno que estoy estudiando cumple con la propiedad de Markov. Por ejemplo ¿Es la evolución del clima un proceso de Markov?

En segundo lugar las cadenas de Markov son un **componente fundamental de una clase de algoritmos conocidos como Markov Chain Monte Carlo** (MCMC). El objetivo de MCMC es crear sintéticamente una cadena de Markov que converge a una distribución en la cual estamos interesados y que no podemos simular de forma analítica y/o explícita. MCMC es considerado una revolución en computación científica y tiene aplicaciones en prácticamente todos las disciplinas.

En esta lección estudiaremos el algoritmos de Metropolis, una de las formulaciones originales de MCMC y también uno de los [diez algoritmos más importantes del Siglo XX](https://www.andrew.cmu.edu/course/15-355/misc/Top%20Ten%20Algorithms.html).

## Monte Carlo y muestreo por importancia (IS)

MCMC es una poderosa herramienta para muestrear y calcular valores esperados a partir de distribuciones complejas. En este sentido es una extensión de la idea básica de Monte Carlo que vimos en las primeras lecciones.

Recordemos, con Monte Carlo podemos estimar el valor esperado de una función $f(x)$ en base a muestras usando

$$
\mathbb{E}_{x\sim p(x)}[f(X)] \approx \frac{1}{N} \sum_{i=1}^N f(x_i) \quad x_i \sim p(x)
$$

:::{warning}

El estimador anterior sólo se puede usar si es posible muestrear directamente de $p(x)$

:::

Si no puedo muestrear de $p(x)$ pero si puedo evaluarla, se puede recurrir a la técnica de muestreo por importancia (IS) que se define a continuación.

Sea una distribución de propuestas o distribución de importancia $q(x)$ de la cual es posible evaluar y además muestrear. En ese caso se puede reescribir la ecuación anterior como sigue

$$
\begin{align}
\mathbb{E}_{x\sim p(x)}[f(X)] &= \int p(x) f(x) \,dx = \int q(x)  \frac{p(x)}{q(x)} f(x) \,dx \nonumber \\
&= \mathbb{E}_{x\sim q(x)}\left[ \frac{p(x)}{q(x)} f(X)\right] \nonumber \\
&\approx \frac{1}{N} \sum_{i=1}^N w_i f(x_i) \quad x_i \sim q(x) \nonumber
\end{align}
$$

donde $w_i = \frac{p(x)}{q(x)}$ se llama la ponderación de importancia.

Una distribución de importancia adecuada no sólo nos permite resolver el problema sino que tiende a presentar una varianza más baja que el estimador original de Monte Carlo. Notar que no es necesario escoger una distribución de importancia que sea igual a la distribución original, pero se debe tener en cuanta que que $q(x)$ debe ser tal que $p(x)=0$ cuando $q(x)=0$

### Ejercicio formativo: Muestreo por importancia

Sea una linea de teléfono de soporte tecnológico que recibe en promedio 2 llamadas por minuto

> ¿Cuál es la probabilidad de que ellos tengan que esperar por lo menos 10 minutos para recibir 9 llamadas?

Para modelar el tiempo de espera utilizaremos una distribución gamma

```python
b = 2. # Eventos promedio por minuto
a = 9 # Cantidad de eventos
p = scipy.stats.gamma(a, scale=1/b)
```

Luego implementamos la función que queremos estimar

```python
f = lambda x: x > 10
```

Para IS utilizaremos la siguiente función de propuesta

```python
q = scipy.stats.norm(scale=10)
```

A continuación simulamos para Monte Carlo e IS usando número de muestras cada vez más grandes e inspeccionamos como evoluciona nuestro estimador del valor esperado de $f(x)$.

:::{note}

En este caso el problema tiene solución teórica conocida. Veamos como se compara MC e IS al resultado teórico.

:::

```python
# Simulación
mc_result, is_result = {}, {}
true_result = 1 - p.cdf(10)
for N in np.logspace(1, 4, num=100):
    # Monte carlo clasico
    x = p.rvs(size=int(N))
    mc_result[N] = np.mean(f(x))
    # Muestreo por importancia
    x = q.rvs(size=int(N))
    w = p.pdf(x)/q.pdf(x)
    is_result[N] = np.mean(w*f(x))
```

```python
mc_plot = hv.Curve((list(mc_result.keys()), list(mc_result.values())),
                   'Cantidad de muestras',
                   'Probabilidad de recibir 10\n llamadas en 9 minutos',
                   label='Monte Carlo').opts(logx=True)
is_plot = hv.Curve((list(is_result.keys()), list(is_result.values())), label='Importance Sampling')
true_plot = hv.HLine(true_result)
hv.Overlay([mc_plot, is_plot, true_plot])
```

:::{note}

El estimador basado en IS converge más rápido y con menos varianza que el estimador basado en Monte Carlo convencional

:::

### Problemas de IS

Las técnicas de muestreo por importancia y muestreo por rechazo permiten calcular valores esperados de distribuciones que puedo evaluar pero no muestrear. También vimos que favorece en la disminución de la varianza del estimador.

Pero existen casos más complicados aun, por ejemplo

**No puedo muestrear ni evaluar la distribución de interés**

Digamos que estamos interesados en la distribución de una variable $\theta$ condicionada a un conjunto de observaciones $D$, esto corresponde al posterior $p(\theta|D)$. Sólo en contadas ocasiones este posterior corresponderá a una distribución teórica como las que hemos visto en este curso.

Más en general tendremos

$$
p(\theta|x) = \frac{p(x|\theta) p(\theta)}{p(x)}
$$

donde $p(x|\theta)$ es la verosimilitud, $p(\theta)$ es el prior y

$$
p(x) = \int_\theta p(x, \theta) \,d\theta
$$

es la evidencia o verosimilitud marginal que no depende de $\theta$. Si la dimensionalidad de $\theta$ es grande la integral será muy difícil o derechamente imposible de calcular analiticamente.

:::{note}

En este caso sólo podemos evaluar la verosimilitud y el prior, es decir que podemos evaluar una cantidad proporcional al posterior

$$
p(\theta|x) \propto p(x|\theta) p(\theta)
$$

:::

**Espacios demasiado grandes**

Otro problema de los espacios de alta dimensionalidad es que recorrer ese espacio completo de forma independiente puede ser muy lento o de plano infactible

:::{tip}

A diferencia de IS, MCMC puede usarse aun cuando existen las dos dificultades anteriormente mencionadas

:::

## Intuición de MCMC

En MCMC en lugar de muestrear de manera i.i.d., utilizamos una cadena de Markov que corresponde a la secuencia de pasos que damos en el espacio de alta dimensionalidad.

En la siguiente figura la distribución de interés se muestra de color rojo. En la subfigura de la izquierda usamos una distribución de importancia simple (contorno verde). Muchos de los puntos tendrán un ponderador de importancia cercano a cero.

Los métodos de MCMC se basan en "diseñar" una cadena de Markov tal que converja a la distribución complicada que nos interesa, como muestra la subfigura de la derecha.

Luego sólo tenemos que dejar que la cadena corra "por un tiempo largo" para que la convergencia se cumpla y finalmente usar los valores de los estados de la cadena como una representación de la distribución a la cual no tenemos acceso.

Glosario MCMC:

- La secuencia de valores de los estados de la cadena se llama **trace** (traza)
- El tiempo que demora en converger la cadena se llama **mixing time** (tiempo de mezcla)
- Es común eliminar de la trasa los primeros estados de la cadena pues no están cerca de la distribución de interés.  Esto se denomina **burn-in** (quema).

**¿Qué es diseñar una cadena de Markov?**

Extendiendo al caso de un estado continuo en lugar de discreto, la distribución estacionaria $\pi$ debe cumplir

$$
\int \pi(\theta_t) q(\theta_{t+1}|\theta_t) \,d\theta_t = \pi (\theta_{t+1})
$$

Luego diseñar la cadena de Markov consiste en encontrar las probabilidades de transición $q(\theta_{t+1}|\theta_t)$ dado que conozco $\pi$.

:::{note}

Esto es "al reves" de lo que hicimos en la lección pasada, que era encontrar $\pi$ dado que conozco la matriz de transición

:::

A continuación veremos veremos que no necesitamos conocer "completamente" $\pi$ para lograr esto, basta conocerlo hasta una constante.

##  Algoritmo de Metropolis

El algoritmo de Metropolis fue el primer algoritmo de tipo MCMC. Fue propuesto por Nicholas Metropolis, colega de Ulam y Von Neumann, [en el año 1953 para entender la transición de fase que experimetan los materiales](https://www.semanticscholar.org/paper/Equation-of-state-calculations-by-fast-computing-Metropolis-Rosenbluth/f6a13f116e270dde9d67848495f801cdb8efa25d). El paper original sentó las bases de lo que hoy conocemos como el algoritmo de Metropolis y el algoritmo de Simulated Annealing (SA)

El algoritmo de Metropolis utiliza un random walk para definir las probabilidades de transición de la cadena. Sea

$$
\theta_{t+1} = \theta_{t} + \epsilon
$$

donde $\epsilon$ tiene una distribución centrada en cero y simétrica, típicamente una gaussiana

$$
\epsilon \sim \mathcal{N}(0, I\sigma_\epsilon^2),
$$

donde $\sigma_\epsilon$ es hiper parámetro del algoritmo. Por definición tenemos entonces que

$$
\theta^* \sim q(\theta_{t+1}|\theta_{t}) = \mathcal{N}(\theta_{t}, I \sigma_\epsilon^2)
$$

donde $q$ se denomina **distribución de propuestas** y su objetivo es **proponer** un valor para $\theta_{t+1}$.

Luego el nuevo valor de $\theta$ se acepta con una tasa definida como

$$
\alpha(\theta^*|\theta_{t}) = \min(1, r)
$$

donde

$$
r = \frac{ p(\theta^*)q(\theta_{t}|\theta^*) }{ p(\theta_t)q(\theta^*|\theta_{t})} = \frac{p(\theta^*)}{p(\theta_t)}
$$

donde la última equivalencia se tiene por la simetría

Entonces

- Si $\theta^*$ es mucho mejor que $\theta_t$ entonces se acepta
- Si $\theta^*$ es mucho peor que $\theta_t$ entonces se rechaza
- En caso de duda se deja al azar

Respecto de $\sigma_\epsilon$

- Si su valor es grande tendremos muchos rechazos
- Si su valor es pequeño la difusión será lenta y podrían requerirse muchas iteraciones

Juntando lo anterior se tiene que el algoritmo completo es

1. Escoger una distribución de propuestas simétrica
1. Escoger un valor inicial $\theta_0$
1. Para $n=1,2,\ldots, N$
    1. Muestrear $\theta^* \sim q(\theta_{t+1}|\theta_{t})$
    1. Muestrear $u \sim U[0, 1]$
    1. Luego si

    $$
    u < \alpha(\theta^*|\theta_{t})
    $$

    entonces

    $$
    \theta_{t+1} = \theta^*
    $$

    de lo contrario

    $$
    \theta_{t+1} = \theta_{t}
    $$

A continuación se muestra una implementación en Python de este algoritmo.

- Se utiliza una normal como distribución de propuestas
- Se precalculan los valores de la distribución de propuesta de la variable U para mayor eficiencia
- El argumento `p` es una función de Python con una variable (theta)
- El argumento `mix_time` define el largo de la traza resultante
- El argumento `sigma_eps` es la desviación estándar de la distribución de propuestas

```python
def metropolis(p, mix_time, sigma_eps=1.):
    trace_theta = np.zeros(shape=(mix_time, )) # Retorna la traza de theta
    trace_theta[0] = np.random.randn() # Theta inicial
    qs = scipy.stats.norm(loc=0, scale=sigma_eps).rvs(size=mix_time)
    us = scipy.stats.uniform.rvs(size=mix_time)
    for n in range(1, mix_time):
        theta_star = trace_theta[n-1] + qs[n] # Theta propuesto
        r = p(theta_star)/p(trace_theta[n-1])
        alpha = np.amin([1., r])
        if us[n] < alpha:
            trace_theta[n] = theta_star
        else:
            trace_theta[n] = trace_theta[n-1]
    return trace_theta
```

### Algoritmo de Metropolis-Hastings

El algoritmo de Metropolis-Hastings es una generalización del algoritmo de Metropolis para el caso donde la distribución de propuestas ya no es simétrica por lo que $r$ es ahora

$$
r = \frac{ p(\theta^*)q(\theta_{t}|\theta^*) }{ p(\theta_t)q(\theta^*|\theta_{t})}
$$

El algoritmo procede de forma idéntica al caso anterior

## MCMC para estimar posteriors

Si estamos interesados en un posterior $p(\theta_t|\mathcal{D})$ cuya evidencia $p(\mathcal{D})$ es difícil o imposible de calcular, podemos usar el algoritmo de Metropolis sin problema pues

$$
r = \frac{p(\theta^*|\mathcal{D})}{p(\theta_t|\mathcal{D})} = \frac{p(\mathcal{D}|\theta^*)p(\theta^*)}{p(\mathcal{D}|\theta_t)p(\theta_t)} \frac{p(\mathcal{D})}{p(\mathcal{D})} = \frac{p(\mathcal{D}|\theta^*)p(\theta^*)}{p(\mathcal{D}|\theta_t)p(\theta_t)}
$$

Es decir que basta con conocer la verosimilitud y el prior. Pongamos en práctica esto con un ejemplo

Sea un conjunto de muestras con $N=5$

$$
\mathcal{D} = \{ 9.37, 10.18, 9.16, 11.60, 10.33 \}
$$

que corresponden a realizaciones i.i.d

$$
X_1, X_2, \ldots, X_5|\theta \sim \mathcal{N}(\theta, \sigma^2=1)
$$

donde

$$
\theta \sim \mathcal{N}(\mu=5, \tau^2=10)
$$

y nos interesa encontrar el posterior $p(\theta|\mathcal{D})$. Intentemos simular este posterior con el algoritmo de Metropolis.

Primero implementamos la verosimilitud y el prior

```python
prior = lambda theta : scipy.stats.norm(loc=5, scale=np.sqrt(10)).pdf(theta)
likelihood = lambda theta, data : np.prod([scipy.stats.norm(loc=theta, scale=1).pdf(x) for x in data])
p = lambda theta, data: likelihood(theta, data)*prior(theta)
```

Luego obtenemos una traza entregandole `p` a la función `metropolis` que implementamos anteriormente. Se queman (descartan) las primeras 100 muestras de la traza

```python
from functools import partial

D = np.array([9.37, 10.18, 9.16, 11.60, 10.33])
burn_in = 100
theta_trace = metropolis(partial(p, data=D), mix_time=5000)[burn_in:]
edges, bins = np.histogram(theta_trace, bins=20, density=True)
```

La siguiente figura muestra la evolución de la traza (izquierda) y la densidad estimada a partir de la traza (derecha).

```python
tn2 = (len(D)/1. + 1./10)**(-1)
wn = tn2/10.
theta_plot = np.linspace(np.amin(theta_trace), np.amax(theta_trace), num=100)
theorical_dist = scipy.stats.norm(loc=np.mean(D)*(1-wn)+5*wn,
                                  scale=np.sqrt(tn2)).pdf(theta_plot)

p_trace = hv.Curve((theta_trace), 'Step', 'Theta')
p_true = hv.HLine(np.mean(D)*(1-wn) + 5*wn)
p_dist = hv.Histogram((edges, bins), kdims='Theta', label='MCMC').opts(xlim=(8, 12), alpha=0.75, width=250)
p_dist_true = hv.Curve((theta_plot, theorical_dist),
                       label='Teórica').opts(color='k', line_dash='dashed')
((p_trace * p_true) << (p_dist * p_dist_true)).opts(hv.opts.Overlay(legend_position='top'))
```

:::{note}

En este caso (muy particular) el posterior se puede calcular analíticamente como

$$
p(\theta|\mathcal{D}) = \mathcal{N}\left ( \bar x (1- w_N) + \mu w_N , \tau_N^2 \right)
$$

donde $w_N = \tau_N^2/\tau^2$ y $\tau_N^2 = (N/\sigma^2 + 1/\tau^2)^{-1}$

:::

El posterior analítico se muestra con linea negra punteada en la figura anterior. La solución de MCMC estima con bastante fidelidad la distribución teórica.

:::{warning}

Pero, ¿Qué pasa en el caso general donde hay un posterior análitco para comparar? ¿Cómo puedo comprobar si la cadena convergió a su estado estacionario y que el posterior obtenido es válido?

:::

En la próxima lección veremos algunas técnicas para diagnosticar la convergencia de la cadena y también veremos en mayor detalle la influencia de los hiperparámetros del algoritmo de metrópolis

```python

```