# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.
    	There will be an option for Colab users and another for Jupyter (local) users.

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [5.32907052e-17 7.04991621e-17 1.85962357e-17]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03917523 -0.04338266 -0.10426524]
      stds:   [1.01277281 1.0098496  0.97575131]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029261167605239e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

    dx difference:  1.937389231917781e-12
    dgamma difference:  0.0
    dbeta difference:  0.0
    speedup: 0.01x

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 4.05e-10
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 1.01e-10
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 6.96e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 2.78e-09
    b2 relative error: 2.22e-08
    b3 relative error: 2.10e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 4.23e-09
    gamma1 relative error: 6.27e-09
    gamma2 relative error: 5.28e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340975
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.318000; val_acc: 0.278000
    (Iteration 21 / 200) loss: 1.968982
    (Epoch 2 / 10) train acc: 0.402000; val_acc: 0.305000
    (Iteration 41 / 200) loss: 2.164388
    (Epoch 3 / 10) train acc: 0.478000; val_acc: 0.310000
    (Iteration 61 / 200) loss: 1.625863
    (Epoch 4 / 10) train acc: 0.496000; val_acc: 0.311000
    (Iteration 81 / 200) loss: 1.261797
    (Epoch 5 / 10) train acc: 0.582000; val_acc: 0.303000
    (Iteration 101 / 200) loss: 1.267744
    (Epoch 6 / 10) train acc: 0.620000; val_acc: 0.327000
    (Iteration 121 / 200) loss: 1.082156
    (Epoch 7 / 10) train acc: 0.712000; val_acc: 0.355000
    (Iteration 141 / 200) loss: 1.109741
    (Epoch 8 / 10) train acc: 0.752000; val_acc: 0.329000
    (Iteration 161 / 200) loss: 0.786908
    (Epoch 9 / 10) train acc: 0.785000; val_acc: 0.316000
    (Iteration 181 / 200) loss: 0.781644
    (Epoch 10 / 10) train acc: 0.855000; val_acc: 0.319000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.245000; val_acc: 0.212000
    (Iteration 21 / 200) loss: 2.062543
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.270000
    (Iteration 41 / 200) loss: 1.820479
    (Epoch 3 / 10) train acc: 0.340000; val_acc: 0.258000
    (Iteration 61 / 200) loss: 1.755715
    (Epoch 4 / 10) train acc: 0.379000; val_acc: 0.287000
    (Iteration 81 / 200) loss: 1.544602
    (Epoch 5 / 10) train acc: 0.400000; val_acc: 0.289000
    (Iteration 101 / 200) loss: 1.751304
    (Epoch 6 / 10) train acc: 0.466000; val_acc: 0.311000
    (Iteration 121 / 200) loss: 1.382958
    (Epoch 7 / 10) train acc: 0.509000; val_acc: 0.322000
    (Iteration 141 / 200) loss: 1.348556
    (Epoch 8 / 10) train acc: 0.539000; val_acc: 0.316000
    (Iteration 161 / 200) loss: 1.294357
    (Epoch 9 / 10) train acc: 0.579000; val_acc: 0.320000
    (Iteration 181 / 200) loss: 0.971533
    (Epoch 10 / 10) train acc: 0.605000; val_acc: 0.311000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
[FILL THIS IN]

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
[FILL THIS IN]

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.433615657860454e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
[FILL THIS IN]


```python
%run header.ipynb
```

```python
from sympy import pi
from sympy.physics.units import meter, foot
```

```python
# dictionary that holds all values.
# if already defined (such as by another notebook) then don't override
if "values" not in vars():
    values={"d": 12.1 * meter}

Formula.set_global_values(values)
```

```python
d = Formula("d")
circ_formula = Formula(value_name="C", formula="pi*d", sigfigs=4)
```

The circumference of a circle with diameter {{ d.get_inline() }} is:

{{ circ_formula.get_display(oneline=True) }}

```python

```


# Dispersal in metacommunities

```python
import numpy as np
import matplotlib.pyplot as plt
from tqdm.notebook import tqdm
import ipywidgets as widgets
%matplotlib widget
```

## Vellend's dispersal

Vellend's basic description is incredibly simple. Say we have some number of patches $M$ making up our metacommunity, each with the same number of individuals. The only change we have to make is that with probability $m$ we choose a reproducing individual from the entire metacommunity rather than only the local community.

Conceptually, and in the code, this mean that now we also have to check each patch in addition to each individual in each year. Now we have a total number of timesteps $=Y \times N \times M$.

One other change is that when we choose an individual to die we also have to chose which patch the death occurs in. We uniformly select an individual in the entire metacommunity at each time step.

We can define the probability of a member of species 1 migrating into the patch where death occured ($P(S = 1)$) as the frequency of species 1 in the entire meta community ($f_1^m$).

\begin{align}
f_1^M = \frac{\sum_{i=0}^{M} N f_1^i}{NM} \\
P_m(S=1) = f_1^M
\end{align}

```python
def moran_dispersal(n_indiv, n_years, m_patches, init_f1, dispersal):
    """ Implements the Moran model including constant selection and simulates it in time.

        Parameters
        ----------
        n_indiv: int
            number of individuals in the community
        n_years: int
            number of potential 'turnovers' of the community
        m_patches: int
            number of separate patches linked by dispersal
        init_f1: float
            initial frequency of species 1
        dispersal: float
            probability that a new birth is drawn from metacommunity

        Return
        ------
        moran: np.array (n_years, 2)
            contains the species frequencies for each year of the simulation """

    # For this we're going to track only species 1 and instead of the columns representing
    # different species they will be different patches
    moran = np.zeros((n_years, m_patches))
    moran[0] = init_f1

    # Now our community vector is instead a metacommunity matrix
    meta = np.random.choice([0, 1], size=(m_patches, n_indiv), replace=True, p=[init_f1, 1 - init_f1])

    # now we can do the loop as in vellend. I'm going to write it a bit differently
    # but the idea is basically the same. (yes its slower)
    for year in tqdm(range(1, n_years)):

        # we now need to check each patch as well as each year
        for patch in range(m_patches):

            # for each year in each patch we do replacement for every individual
            for indiv in range(n_indiv):

                # select patch where death occurs
                death_patch = np.random.choice(range(m_patches))

                # now every time we replace an individual we need to check whether or not
                # dispersal occurs. Draw from entire metacommunity
                if np.random.uniform() < dispersal:
                    pr1 = np.sum(meta == 0) / (n_indiv * m_patches)
                # or draw from local community
                else:
                    pr1 = np.sum(meta[death_patch] == 0) / n_indiv

                # replace one individual with another
                death = np.random.randint(n_indiv)
                birth = np.random.choice([0, 1], p=[pr1, 1 - pr1])
                meta[death_patch, death] = birth

            # when we're done looping over all of the individuals we can update
            # frequencies for the year (the timescale we care about)
            f1 = np.sum(meta[patch] == 0) / n_indiv
            moran[year, patch] = f1

    return moran
```

```python
def draw_simulation(individuals=250, years=50, patches=10, initial_freq=0.5, dispersal=0.0):
    # the plot bit, this just makes a blank figure
    fig, ax = plt.subplots(ncols=2, figsize=(10,4), gridspec_kw = {'width_ratios':[3, 1]})

    # trajectory labels
    ax[0].set_xlabel('Years')
    ax[0].set_ylabel('Species 1 frequency by patch')
    ax[0].set_ylim([0, 1])
    # distribution labels and bins
    hist_bins = np.arange(0, 1.1, 0.1)
    ax[1].set_xlabel('Count')
    ax[1].set_xlim([0, patches])
    ax[1].set_ylim([0, 1])

    # we're going to plot a distribution too need an array for it
    dist = np.zeros(patches)

    simulation = moran_dispersal(individuals, years, patches, initial_freq, dispersal)
    ax[0].plot(range(years), simulation)
    # add final freq to our dist array
    dist = simulation[-1, :]

    # plot the distribution too
    ax[1].hist(dist, bins=hist_bins,  orientation='horizontal')
    ax[1].axhline(np.mean(dist), linestyle='--')

    plt.tight_layout()
    plt.show()

# set up the interface
widgets.interact_manual(draw_simulation, individuals=(10, 1000, 10), years=(5, 200, 5),
                  patches=(1, 20, 1), initial_freq=(0.0, 1.0, 0.01), dispersal=(0, 1.0, 0.01))
```

    interactive(children=(IntSlider(value=250, description='individuals', max=1000, min=10, step=10), IntSlider(va…

    <function __main__.draw_simulation(individuals=250, years=50, patches=10, initial_freq=0.5, dispersal=0.0)>

```python

```


# Computational Astrophysics
## Fundamental Concepts. Examples

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this Notebook

In this notebook we use `python` to illustrate some basic concepts of astrophysics such as distances and brightness.

---

## The `astropy` Project

The `astropy` project is a core package for astronomy and astrophysics using `Python`. It includes a lot of functions that we will use along this course.

`astropy` is included in Anaconda. To update to the latest version, use

`conda update astropy`

This package includes module with physical constants and a module to manipulate units. The complete list of constants included can be found at

https://docs.astropy.org/en/stable/constants/index.html

and the complete list of units included can be found at

https://docs.astropy.org/en/stable/units/index.html

```python
import numpy as np
import astropy.units as u
from astropy import constants as const
```

Newton's gravitational constant is

```python
const.G
```

$6.67408 \times 10^{-11} \; \mathrm{\frac{m^{3}}{kg\,s^{2}}}$

Speed of light in vacuum is

```python
const.c
```

$2.9979246 \times 10^{8} \; \mathrm{\frac{m}{s}}$

Mass of the Sun

```python
const.M_sun
```

$1.9884754 \times 10^{30} \; \mathrm{kg}$

Reduced Planck's constant, $\hbar$, and Boltzmann's constant, $k_B$, are

```python
const.hbar
```

$1.0545718 \times 10^{-34} \; \mathrm{J\,s}$

```python
const.k_B
```

$1.3806485 \times 10^{-23} \; \mathrm{\frac{J}{K}}$

Information about a particular constant can be obtained as

```python
print(const.c)
```

      Name   = Speed of light in vacuum
      Value  = 299792458.0
      Uncertainty  = 0.0
      Unit  = m / s
      Reference = CODATA 2014

The value of a constant can be given in other units,

```python
const.c.to('pc/yr')
```

$0.30660139 \; \mathrm{\frac{pc}{yr}}$

```python
const.c.to('lyr/yr')
```

$1 \; \mathrm{\frac{lyr}{yr}}$

It is possible to define a mass equal of $1$ solar mass

```python
M = 1*u.Msun
M
```

$1 \; \mathrm{M_{\odot}}$

and we can calculate the corresponding Schwarzschild radius using the imported constants,

```python
G = const.G
c = const.c
rS = 2*G*M/c**2
rS
```

$1.4851831 \times 10^{-27} \; \mathrm{\frac{m\,M_{\odot}}{kg}}$

Note that the solar mass units can be transformed into kg to obtain a radius in units of meters. This is acomplished by the method `.decompose()`,

```python
rS = rS.decompose()
rS
```

$2953.2501 \; \mathrm{m}$

and it is transformed into other units with the method `.to()`

```python
rS.to(u.km)
```

$2.9532501 \; \mathrm{km}$

---
## Distances in Astrophysics

**`Carroll and Ostlie (2017). Section 3.1`**

The first concept in astrophysics that we will present is that of distance. Here, we will present some units of measurement of distances used in astronomy and astrophysics through the package `astropy.units`. A complete list of units in the package can be found at
https://docs.astropy.org/en/stable/units/index.html

First, we introduce the **light-year** (lyr) as the distance travelled by light in vacuum in one Julian year (i.e. 365.25 days):

```python
ly = 1*u.lyr
ly
```

$1 \; \mathrm{lyr}$

This quantity can be written in other units such as meters or kilometers by usint the `.to()` method,

```python
ly.to(u.m)
```

$9.4607305 \times 10^{15} \; \mathrm{m}$

Due to our location in the Solar System , it is usual that all distances are given with respect to the Sun and that the unit of measurement at this scale is chosen as the mean distance Sun-Earth. This is called an **Astronomical Unit** (au),

```python
au = 1*u.au
au
```

$1 \; \mathrm{AU}$

$1$ au is equivalent to

```python
au.to(u.m)
```

$1.4959787 \times 10^{11} \; \mathrm{m}$

```python
au.to(u.lyr)
```

$1.5812507 \times 10^{-5} \; \mathrm{lyr}$

### Parallax and the definition of parsec ###

One of the initial forms of measurment of distances to stars in the Milky Way si known as the **parallax method**, in which it is used the annual shift of the angular position of an object in the sky. Using basic trigonometry, the distance to a satr is given by

\begin{equation}
d = \frac{b}{\tan \alpha}
\end{equation}

where $b$ is the baseline of the triangle and $\alpha$ is the parallax angle. For measurements from Earth, the baseline is chosen as $1$ au.

Now, we define a function to calculate the distance for a given parallax angle,

```python
def d(alpha=1*u.arcsec, baseline = 1*u.au):
    '''
    ------------------------------------------
    d(alpha, baseline)
    ------------------------------------------
    Returns the distance to a star using the
    parallax method.

    Arguments:
    baseline in au
    parallax angle alpha given in arsec.
    ------------------------------------------
    '''
    return baseline/np.tan(alpha)
```

#### Example. $\alpha$-Centauri ####
The closest star system to the Sun is called $\alpha$-Centauri. It is a triple system, consisting of three stars: $\alpha$-Centauri A (Rigil Kentaurus), $\alpha$-Centauri B (Toliman) and $\alpha$-Centauri C (Proxima Centauri). The parallax angle for this system is 754.81 milli-arsec. Therefore, it is located at a distance of

```python
alphaCentauri_distance = d(0.75481*u.arcsec)
alphaCentauri_distance
```

$273267.19 \; \mathrm{AU}$

This is clearly a large value but it can be converted to light-years,

```python
alphaCentauri_distance.to(u.lyr)
```

$4.3210394 \; \mathrm{lyr}$

---
The parallax method permits to define a distance unit called **parsec** (pc), corresponding to the distance at which $1$ au subtends a parallax angle of $1$ arcsec. This unit is equivalent to

```python
pc = d(1*u.arcsec)
pc
```

$206264.81 \; \mathrm{AU}$

```python
pc.to(u.pc)
```

$1 \; \mathrm{pc}$

```python
pc.to(u.lyr)
```

$3.2615638 \; \mathrm{lyr}$

Today, parallax angles can be measured up to the order of milli-arcsec. This corresponds to distance of the order of kpc,

```python
d(1*u.marcsec).to(u.pc)
```

$1000 \; \mathrm{pc}$

For far objects, for example at distances of the order of Mpc, the parallax method can not be used. Instead, methods such as relative brightness are used to estimate distances.

#### Example. $\alpha$-Centauri ####
The distance to $\alpha$-Centauri in pc is

```python
alphaCentauri_distance.to(u.pc)
```

$1.3248367 \; \mathrm{pc}$

#### Example. 61 Cygni ####
After 4 years of observation, in 1838 Bessel reported a parallax angle of $alpha=0.316$ arcsec for the star 61-Cygni. The distance to this star is

```python
cygni_distance = d(0.316*u.arcsec)
cygni_distance
```

$652736.73 \; \mathrm{AU}$

```python
cygni_distance.to(u.pc)
```

$3.164557 \; \mathrm{pc}$

---
## The Continuous Spectrum of Light
**`Carroll and Ostlie (2017). Section 3.2`**

### Apparent Magnitude ###

The magnitude scale defined by the Hipparchus used numbers from $1$ to $6$ where $1$ corresponds to the brightest objects. Hence, the magnitude scale difference in the visual range is 5. This is known as the apparent magnitude ($m$).
Today, we know that the magnitude scale difference $\Delta m = 5$ corresponds to a brightness (luminous energy) ratio of about 100. This means that magnitudes are logaritmic measures of brightness (similar to the case of audio loudness in which decibels define a base 10 logarithmic scale).

Concerning magnitudes, we can obtain the logarithm base $b$ of the scale by solving the equation

\begin{equation}
b^{\Delta m} = b^5 = 100
\end{equation}

The `sympy` functions `Symbol` and `solve` gives the solution,

```python
from sympy import Symbol, solve
b = Symbol('b')
solve(100.-b**5, b)
```

    [2.51188643150958,
     -2.03215881103109 - 1.4764497998749*I,
     -2.03215881103109 + 1.4764497998749*I,
     0.776215595276303 - 2.38894595888057*I,
     0.776215595276303 + 2.38894595888057*I]

The real value corresponds to the logarithm base $b=2.51188643150958$.

### Flux and Luminosity

The modern measurement of "brightness" is given in terms of the **radiant flux** $F$, defined as the total amount of energy (in all wavelengths) crossing a unit area oriented perpendicular to the direction of the light ray per unit time. It is measured in Watts (Joules per second).

$F$ depends on the **intrinsec luminosity** of the star, $L$, and the distance from the observer. The inverse square law for light states that

\begin{equation}
F = \frac{L}{4\pi r^2}
\end{equation}

The relation between flux and apparent magnitude is given by considering that a difference of $\Delta m = m_1 - m_2 = 5$ correspond to a ratio of $100$ in the fluxes, i.e.

\begin{equation}
\frac{F_2}{F_1} = 100^{\frac{\Delta m}{5}}.
\end{equation}

or inverting

\begin{equation}
\Delta m = m_1 - m_2 = - 2.5 \log_{10} \left( \frac{F_1}{F_2} \right)
\end{equation}

Some examples of apparent magnitudes of stars are:

| *Star*  |  $m$  |
|:-------:|:-----:|
| Sun     | -27   |
|Sirius   | -1.46 |
|Arcturus | -0.04 |
|Aldebaran| 0.85  |
|Spica    | 1.04  |
|Procyon  | 0.34  |

### Absolute Magnitude ###

The absolute magnitude ($M$) describes the intrinsic luminosity emitted by an object and is defined to be equal to the apparent magnitude that the object would have if it were placed at a certain distance from Earth. Since there are no major stars within 1 parsec from the Sun, but there are many important stellar objects within 10 parsecs, it has been historically chosen a reference distance of 10 pc for stars. There exist a more complex definition of absolute magnitude for planets and otrher Solar System bodies.

Then, the absolute magnitude is obtained from the apparent magnitude through

\begin{equation}
M = m - 5(\log_{10} D -1)
\end{equation}

where $D$ stands for the object's distance measured in parsecs. Solving for $D$, we can write

\begin{equation}
D = 10^\frac{m - M + 5}{5}
\end{equation}

#### Example. Absolute Magnitude of the Sun ####

The Sun is located at a distance of $1$ au from Earth and its apparent magnitude is $m=-26.83$. Lets calculate its absolute magnitude

```python
def M(m,D):
    '''
    --------------------------------------------
    M(m,D)
    --------------------------------------------
    Returns the absolute magnitude of a star.

    Arguments:
    m: apparent magnitude
    D: Distance to the Sun in parsecs
    --------------------------------------------
    '''
    return m - 5*(np.log10(D) - 1)

```

```python
D_sun = 1*u.au.to(u.pc)

M(-26.83, D_sun)
```

    4.742125665865284

```python

```


# `GiRaFFE_NRPy_staggered`: Solving the Induction Equation

## Author: Patrick Nelson

**Notebook Status:** Validated

**Validation Notes:** This code is a port from the old `GiRaFFE`

### NRPy+ Source Code for this module:
* [GiRaFFE_NRPy/GiRaFFE_NRPy_staggered_Afield_flux.py](../../edit/in_progress/GiRaFFE_NRPy/GiRaFFE_NRPy_staggered_Afield_flux.py)

## Introduction

This notebook presents an alternate algorithm for computing the flux term in the GRFFE induction equation in a staggered prescription. It is a direct port of the old `GiRaFFE` implementation, which implements the algorithm described in [Del Zanna, et al. (2003)](https://arxiv.org/abs/astro-ph/0210618)

# Table of Contents
$$\label{toc}$$

This notebook is organized as follows

1. [Step 1](#prelim): Preliminaries
1. [Step 2](#code): Write the C code
    1. [Step 2.a](#declaration): Write the function declaration and map inputs to $A_z$ formula
    1. [Step 2.a](#declaration): Write the function declaration
1. [Step 3](#code_validation): Code Validation
1. [Step 4](#latex_pdf_output): Output this notebook to $\LaTeX$-formatted PDF file

# Step 1: Preliminaries \[Back to [top](#toc)\]
$$\label{prelim}$$

This first block of code just sets up a subdirectory within `GiRaFFE_standalone_Ccodes/` to which we will write the C code.

```python
# Step 0: Add NRPy's directory to the path
# https://stackoverflow.com/questions/16780014/import-file-from-parent-directory
import os,sys
nrpy_dir_path = os.path.join("..")
if nrpy_dir_path not in sys.path:
    sys.path.append(nrpy_dir_path)

import cmdline_helper as cmd     # NRPy+: Multi-platform Python command-line interface
Ccodesdir = "GiRaFFE_standalone_Ccodes/RHSs"
cmd.mkdir(os.path.join(Ccodesdir))
```

# Step 2: Write the C code \[Back to [top](#toc)\]
$$\label{code}$$

Our goal here is to compute the part of A_i_rhs that excludes the gauge terms:
$$
\partial_t A_z = \psi^{6} (v^z B^x - v^x B^z)
$$
As can be seen above, we can do this by coding just one component ($z$) and permuting the inputs.

## Step 2.a: Write the function declaration and map inputs to $A_z$ formula \[Back to [top](#toc)\]
$$\label{declaration}$$

Our inputs include the component of $A_i$ (as a number between 1 and 3), the usual NRPy+ `paramstruct`, the reconstructed Valencia three-velocity on the left and right faces (as contained in the `gf_and_gz_struct`s), the square root of the metric determinant $\sqrt{\gamma} = \psi^6$, and the characteristic speeds. The output is the right-hand side (RHS) of the $A_i$ equation.

We then use the `#include` to set parameters; if the declaration of a certain variable is otherwise not seen in this file, it was performed here.

Next, we will define the *actual* directions that correspond to `v1` and `v2` from the input `A_dirn`.
* If `A_dirn=1`, then `v1_offset=1` (v1=VY) and `v2_offset=2` (v2=VZ)
* If `A_dirn=2`, then `v1_offset=2` (v1=VZ) and `v2_offset=0` (v2=VX)
* If `A_dirn=3`, then `v1_offset=0` (v1=VX) and `v2_offset=1` (v2=VY)

```python
%%writefile $Ccodesdir/A_i_rhs_no_gauge_terms.h
/* Compute the part of A_i_rhs that excludes the gauge terms. I.e., we set
 *   A_i_rhs = \partial_t A_i = \psi^{6} (v^z B^x - v^x B^z)   here.
 */
void A_i_rhs_no_gauge_terms(const int A_dirn,const paramstruct *params,gf_and_gz_struct *out_prims_r,gf_and_gz_struct *out_prims_l,
                                   REAL *psi6_pointer,REAL *cmax_1,REAL *cmin_1,REAL *cmax_2,REAL *cmin_2, REAL *A3_rhs) {
  #include "../set_Cparameters.h"

  // If A_dirn=1, then v1_offset=1 (v1=VY) and v2_offset=2 (v2=VZ)
  // If A_dirn=2, then v1_offset=2 (v1=VZ) and v2_offset=0 (v2=VX)
  // If A_dirn=3, then v1_offset=0 (v1=VX) and v2_offset=1 (v2=VY)
  const int v1_offset  = ((A_dirn-1)+1)%3,        v2_offset = ((A_dirn-1)+2)%3;

```

    Overwriting GiRaFFE_standalone_Ccodes/RHSs/A_i_rhs_no_gauge_terms.h

With the directions defined, we now have a mapping for the formulae for each of the $A_i$ RHSs onto the $A_z$ RHS formula. We use that here to set the pointers to the reconstructed velocities and magnetic fields that we have already computed, allowing this function to be completely general while using a single naming scheme for convenience.

```python
%%writefile -a $Ccodesdir/A_i_rhs_no_gauge_terms.h

  const REAL *v1rr=out_prims_r[VXR+v1_offset].gf, *v2rr=out_prims_r[VXR+v2_offset].gf;
  const REAL *v1rl=out_prims_l[VXR+v1_offset].gf, *v2rl=out_prims_l[VXR+v2_offset].gf;
  const REAL *v1lr=out_prims_r[VXL+v1_offset].gf, *v2lr=out_prims_r[VXL+v2_offset].gf;
  const REAL *v1ll=out_prims_l[VXL+v1_offset].gf, *v2ll=out_prims_l[VXL+v2_offset].gf;

  const REAL *B1r=out_prims_r[BX_STAGGER+v1_offset].gf, *B1l=out_prims_l[BX_STAGGER+v1_offset].gf;
  const REAL *B2r=out_prims_r[BX_STAGGER+v2_offset].gf, *B2l=out_prims_l[BX_STAGGER+v2_offset].gf;

```

    Appending to GiRaFFE_standalone_Ccodes/RHSs/A_i_rhs_no_gauge_terms.h

There is one more wrinkle to consider in mapping $A_i$ to $A_z$, which is where the reconstructions of the velocities and magnetic fields are stored. So, we up two arrays of offsets to coordinate this.
* $\bar{v}^i$ dependencies:
    * In the case of `Ax_rhs`, we need `v{y,z}{r,l}` at $(i,j+1/2,k+1/2)$.
        *    However, `v{y,z}{r,l}{r,l}` are defined at $(i,j-1/2,k-1/2)$, so
        *    `v{y,z}{r,l}` at $(i,j+1/2,k+1/2)$ is stored at `v{y,z}{r,l}{r,l}(i,j+1,k+1)`.
    * In the case of `Ay_rhs`, we need `v{x,z}{r,l}` at $(i+1/2,j,k+1/2)$.
        *    However, `v{x,z}{r,l}{r,l}` are defined at $(i-1/2,j,k-1/2)$, so
        *    `v{x,z}{r,l}` at $(i+1/2,j,k+1/2)$ is stored at `v{x,z}{r,l}{r,l}(i+1,j,k+1)`.
    * In the case of `Az_rhs`, we need `v{x,y}{r,l}` at $(i+1/2,j+1/2,k)$.
        *    However, `v{x,y}{r,l}{r,l}` are defined at $(i-1/2,j-1/2,k)$, so
        *    `v{x,y}{r,l}` at $(i+1/2,j+1/2,k)$ is stored at `v{x,y}{r,l}{r,l}(i+1,j+1,k)`.
* $B^i$ dependencies:
    * In the case of `Ax_rhs`, we need `B{y,z}_stagger{r,l}` at $(i,j+1/2,k+1/2)$.
        *    However, `By_stagger{r,l}` is defined at $(i,j+1/2,k-1/2)$, and
        *             `Bz_stagger{r,l}` is defined at $(i,j-1/2,k+1/2)$, so
        *             `By_stagger{r,l}` at $(i,j+1/2,k+1/2)$ is stored at `By_stagger{r,l}(i,j,k+1)`, and
        *             `Bz_stagger{r,l}` at $(i,j+1/2,k+1/2)$ is stored at `Bz_stagger{r,l}(i,j+1,k)`.
    * In the case of `Ay_rhs`, we need `B{z,x}_stagger{r,l}` at $(i+1/2,j,k+1/2)$.
        *    However, `Bz_stagger{r,l}` is defined at $(i-1/2,j,k+1/2)$, and
        *             `Bx_stagger{r,l}` is defined at $(i+1/2,j,k-1/2)$, so
        *             `Bz_stagger{r,l}` at $(i+1/2,j,k+1/2)$ is stored at `Bz_stagger{r,l}(i+1,j,k)`, and
        *             `Bx_stagger{r,l}` at $(i+1/2,j,k+1/2)$ is stored at `Bx_stagger{r,l}(i,j,k+1)`.
    * In the case of `Az_rhs`, we need `B{x,y}{r,l}_stagger` at $(i+1/2,j+1/2,k)$.
        *    However, `Bx_stagger{r,l}` is defined at $(i+1/2,j-1/2,k)$, and
        *             `By_stagger{r,l}` is defined at $(i-1/2,j+1/2,k)$, so
        *             `Bx_stagger{r,l}` at $(i+1/2,j+1/2,k)$ is stored at `Bx_stagger{r,l}(i,j+1,k)`, and
        *             `By_stagger{r,l}` at $(i+1/2,j+1/2,k)$ is stored at `By_stagger{r,l}(i+1,j,k)`.

```python
%%writefile -a $Ccodesdir/A_i_rhs_no_gauge_terms.h

  /**** V DEPENDENCIES ****/
  /* In the case of Ax_rhs, we need v{y,z}{r,l} at (i,j+1/2,k+1/2).
   *    However, v{y,z}{r,l}{r,l} are defined at (i,j-1/2,k-1/2), so
   *    v{y,z}{r,l} at (i,j+1/2,k+1/2) is stored at v{y,z}{r,l}{r,l}(i,j+1,k+1).
   * In the case of Ay_rhs, we need v{x,z}{r,l} at (i+1/2,j,k+1/2).
   *    However, v{x,z}{r,l}{r,l} are defined at (i-1/2,j,k-1/2), so
   *    v{x,z}{r,l} at (i+1/2,j,k+1/2) is stored at v{x,z}{r,l}{r,l}(i+1,j,k+1).
   * In the case of Az_rhs, we need v{x,y}{r,l} at (i+1/2,j+1/2,k).
   *    However, v{x,y}{r,l}{r,l} are defined at (i-1/2,j-1/2,k), so
   *    v{x,y}{r,l} at (i+1/2,j+1/2,k) is stored at v{x,y}{r,l}{r,l}(i+1,j+1,k). */
  static const int vs_ijk_offset[4][3] = { {0,0,0} , {0,1,1} , {1,0,1} , {1,1,0} }; // Note that vs_ijk_offset[0] is UNUSED; we choose a 1-offset for convenience.

  /**** B DEPENDENCIES ****/
  /* In the case of Ax_rhs, we need B{y,z}{r,l} at (i,j+1/2,k+1/2).
   *    However, By_stagger{r,l} is defined at (i,j+1/2,k-1/2), and
   *             Bz_stagger{r,l} is defined at (i,j-1/2,k+1/2), so
   *             By_stagger{r,l} at (i,j+1/2,k+1/2) is stored at By_stagger{r,l}(i,j,k+1), and
   *             Bz_stagger{r,l} at (i,j+1/2,k+1/2) is stored at Bz_stagger{r,l}(i,j+1,k).
   * In the case of Ay_rhs, we need B{z,x}_stagger{r,l} at (i+1/2,j,k+1/2).
   *    However, Bz_stagger{r,l} is defined at (i-1/2,j,k+1/2), and
   *             Bx_stagger{r,l} is defined at (i+1/2,j,k-1/2), so
   *             Bz_stagger{r,l} at (i+1/2,j,k+1/2) is stored at Bz_stagger{r,l}(i+1,j,k), and
   *             Bx_stagger{r,l} at (i+1/2,j,k+1/2) is stored at Bx_stagger{r,l}(i,j,k+1).
   * In the case of Az_rhs, we need B{x,y}_stagger{r,l} at (i+1/2,j+1/2,k).
   *    However, Bx_stagger{r,l} is defined at (i+1/2,j-1/2,k), and
   *             By_stagger{r,l} is defined at (i-1/2,j+1/2,k), so
   *             Bx_stagger{r,l} at (i+1/2,j+1/2,k) is stored at Bx_stagger{r,l}(i,j+1,k), and
   *             By_stagger{r,l} at (i+1/2,j+1/2,k) is stored at By_stagger{r,l}(i+1,j,k).
   */
  static const int B1_ijk_offset[4][3] = { {0,0,0} , {0,0,1} , {1,0,0} , {0,1,0} }; // Note that B1_ijk_offset[0] is UNUSED; we choose a 1-offset for convenience.
  static const int B2_ijk_offset[4][3] = { {0,0,0} , {0,1,0} , {0,0,1} , {1,0,0} }; // Note that B2_ijk_offset[0] is UNUSED; we choose a 1-offset for convenience.

```

    Appending to GiRaFFE_standalone_Ccodes/RHSs/A_i_rhs_no_gauge_terms.h

## Step 2.b: Loop over the grid interior \[Back to [top](#toc)\]
$$\label{loop_interior}$$

Now, we start a loop over the entire grid interior. We define the memory indices from which we will read the input data, using the offset arrays we just defined. Then, we will read in the values that we need from memory and calculate the flux terms
$$
\psi^6 \left( v^x B^y - v^y B^x \right)
$$
for all combinations of reconstructions: RR, RL, LR, and LL.

The values of $\psi^6$ were interpolated to the correct locations outside of this function.

We also read in the values of the characteristic speeds at a given point and calculate the difference of the state vectors, $\psi^6 (B^x_{\rm R} - B^x_{\rm L})$ and $\psi^6 (B^y_{\rm R} - B^y_{\rm L})$.

```python
%%writefile -a $Ccodesdir/A_i_rhs_no_gauge_terms.h

#pragma omp parallel for
  for(int k=NGHOSTS;k<Nxx_plus_2NGHOSTS2-NGHOSTS;k++) for(int j=NGHOSTS;j<Nxx_plus_2NGHOSTS1-NGHOSTS;j++) for(int i=NGHOSTS;i<Nxx_plus_2NGHOSTS0-NGHOSTS;i++) {
        const int index=IDX3S(i,j,k);
        // The following lines set the indices appropriately. See justification in exorbitant comments above.
        const int index_v =IDX3S(i+vs_ijk_offset[A_dirn][0],j+vs_ijk_offset[A_dirn][1],k+vs_ijk_offset[A_dirn][2]);
        const int index_B1=IDX3S(i+B1_ijk_offset[A_dirn][0],j+B1_ijk_offset[A_dirn][1],k+B1_ijk_offset[A_dirn][2]);
        const int index_B2=IDX3S(i+B2_ijk_offset[A_dirn][0],j+B2_ijk_offset[A_dirn][1],k+B2_ijk_offset[A_dirn][2]);

        // Stores 1/sqrt(gamma)==exp(6 phi) at (i+1/2,j+1/2,k) for Az, (i+1/2,j,k+1/2) for Ay, and (i,j+1/2,k+1/2) for Az.
        const REAL psi6_interped=psi6_pointer[index];

        const REAL B1lL = B1l[index_B1];
        const REAL B1rL = B1r[index_B1];
        const REAL B2lL = B2l[index_B2];
        const REAL B2rL = B2r[index_B2];

        const REAL A3_rhs_rr = psi6_interped*(v1rr[index_v]*B2rL - v2rr[index_v]*B1rL);
        const REAL A3_rhs_rl = psi6_interped*(v1rl[index_v]*B2rL - v2rl[index_v]*B1lL);
        const REAL A3_rhs_lr = psi6_interped*(v1lr[index_v]*B2lL - v2lr[index_v]*B1rL);
        const REAL A3_rhs_ll = psi6_interped*(v1ll[index_v]*B2lL - v2ll[index_v]*B1lL);

        // All variables for the A_i_rhs computation are now at the appropriate staggered point,
        //   so it's time to compute the HLL flux!

        // Note that with PPM, cmin and cmax are defined between ijk=3 and ijk<cctk_lsh[]-2 for all directions.
        const REAL cmax_1L = cmax_1[index_B2];
        const REAL cmin_1L = cmin_1[index_B2];
        const REAL cmax_2L = cmax_2[index_B1];
        const REAL cmin_2L = cmin_2[index_B1];

        const REAL B1tilder_minus_B1tildel = psi6_interped*( B1rL - B1lL );
        const REAL B2tilder_minus_B2tildel = psi6_interped*( B2rL - B2lL );

```

    Appending to GiRaFFE_standalone_Ccodes/RHSs/A_i_rhs_no_gauge_terms.h

With all the input variables read from memory, we are finally able implement the 2D HLL flux [(Del Zanna et al., 2003)](https://arxiv.org/abs/astro-ph/0210618).
This is given by
\begin{align}
-E_z &= \frac{
c_{1{\rm max}} c_{2{\rm max}} A_{z{\rm LL}} +
c_{1{\rm max}} c_{2{\rm min}} A_{z{\rm LR}} +
c_{1{\rm min}} c_{2{\rm max}} A_{z{\rm RL}} +
c_{1{\rm min}} c_{2{\rm min}} A_{z{\rm RR}}
}{(c_{1{\rm max}}+c_{1{\rm min}})(c_{2{\rm max}}+c_{2{\rm min}})} \\
&
- \frac{c_{1{\rm max}} c_{1{\rm min}} (B^y_{\rm R}-B^y_{\rm L})}{c_{1{\rm max}}+c_{1{\rm min}}}
+ \frac{c_{2{\rm max}} c_{2{\rm min}} (B^x_{\rm R}-B^x_{\rm L})}{c_{2{\rm max}}+c_{2{\rm min}}}
\end{align}
(Recall that $\partial_t A_z = -E_z + [{\rm Gauge\ Terms}]$)

```python
%%writefile -a $Ccodesdir/A_i_rhs_no_gauge_terms.h

        /*---------------------------
         * Implement 2D HLL flux
         * [see Del Zanna, Bucciantini & Londrillo A&A 400, 397 (2003), Eq. (44)]
         *
         * Note that cmax/cmin (\alpha^{\pm}  as defined in that paper) is at a slightly DIFFERENT
         * point than that described in the Del Zanna et al paper (e.g., (i+1/2,j,k) instead of
         * (i+1/2,j+1/2,k) for F3).  Yuk Tung Liu discussed this point with M. Shibata,
         * who found that the effect is negligible.
         ---------------------------*/
        A3_rhs[index] = (cmax_1L*cmax_2L*A3_rhs_ll + cmax_1L*cmin_2L*A3_rhs_lr +
                         cmin_1L*cmax_2L*A3_rhs_rl + cmin_1L*cmin_2L*A3_rhs_rr)
          /( (cmax_1L+cmin_1L)*(cmax_2L+cmin_2L) )
          - cmax_1L*cmin_1L*(B2tilder_minus_B2tildel)/(cmax_1L+cmin_1L)
          + cmax_2L*cmin_2L*(B1tilder_minus_B1tildel)/(cmax_2L+cmin_2L);
      }
}

```

    Appending to GiRaFFE_standalone_Ccodes/RHSs/A_i_rhs_no_gauge_terms.h

# Step 3: Code Validation \[Back to [top](#toc)\]
$$\label{code_validation}$$

To validate the code in this tutorial we check for agreement between the files

1. that were written in this tutorial and
1. those that are generated by the python module

```python
# Define the directory that we wish to validate against:
valdir = "GiRaFFE_NRPy/GiRaFFE_Ccode_library/RHSs/"

import GiRaFFE_NRPy.GiRaFFE_NRPy_staggered_Afield_flux as Af
Af.GiRaFFE_NRPy_Afield_flux(valdir)

import difflib
import sys

print("Printing difference between original C code and this code...")
# Open the files to compare
files = ["A_i_rhs_no_gauge_terms.h"]

for file in files:
    print("Checking file " + file)
    with open(os.path.join(valdir,file)) as file1, open(os.path.join(Ccodesdir,file)) as file2:
        # Read the lines of each file
        file1_lines = file1.readlines()
        file2_lines = file2.readlines()
        num_diffs = 0
        for line in difflib.unified_diff(file1_lines, file2_lines, fromfile=os.path.join(valdir+file), tofile=os.path.join(Ccodesdir+file)):
            sys.stdout.writelines(line)
            num_diffs = num_diffs + 1
        if num_diffs == 0:
            print("No difference. TEST PASSED!")
        else:
            print("ERROR: Disagreement found with .py file. See differences above.")
            sys.exit(1)
```

    Printing difference between original C code and this code...
    Checking file A_i_rhs_no_gauge_terms.h
    No difference. TEST PASSED!

# Step 4: Output this notebook to $\LaTeX$-formatted PDF file \[Back to [top](#toc)\]
$$\label{latex_pdf_output}$$

The following code cell converts this Jupyter notebook into a proper, clickable $\LaTeX$-formatted PDF file. After the cell is successfully run, the generated PDF may be found in the root NRPy+ tutorial directory, with filename
[Tutorial-GiRaFFE_NRPy_staggered-Afield_flux.pdf](Tutorial-GiRaFFE_NRPy_staggered-Afield_flux.pdf) (Note that clicking on this link may not work; you may need to open the PDF file through another means.)

```python
import cmdline_helper as cmd    # NRPy+: Multi-platform Python command-line interface
cmd.output_Jupyter_notebook_to_LaTeXed_PDF("Tutorial-GiRaFFE_NRPy_staggered-Afield_flux",location_of_template_file=os.path.join(".."))
```

    Created Tutorial-GiRaFFE_NRPy_staggered-Afield_flux.tex, and compiled LaTeX
        file to PDF file Tutorial-GiRaFFE_NRPy_staggered-Afield_flux.pdf