# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train. One idea along these lines is batch normalization, proposed by [1] in 2015.

To understand the goal of batch normalization, it is important to first recognize that machine learning methods tend to perform better with input data consisting of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features. This will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance, since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, they propose to insert into the network layers that normalize batches. At training time, such a layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```
# Setup cell.
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams["figure.figsize"] = (10.0, 8.0)  # Set default size of plots.
plt.rcParams["image.interpolation"] = "nearest"
plt.rcParams["image.cmap"] = "gray"

%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """Returns relative error."""
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print(f"  means: {x.mean(axis=axis)}")
    print(f"  stds:  {x.std(axis=axis)}\n")
```

    =========== You can safely ignore the message below if you are NOT working on ConvolutionalNetworks.ipynb ===========
    	You will need to compile a Cython extension for a portion of this assignment.
    	The instructions to do this will be given in a section of the notebook below.

```
# Load the (preprocessed) CIFAR-10 data.
data = get_CIFAR10_data()
for k, v in list(data.items()):
    print(f"{k}: {v.shape}")
```

    X_train: (49000, 3, 32, 32)
    y_train: (49000,)
    X_val: (1000, 3, 32, 32)
    y_val: (1000,)
    X_test: (1000, 3, 32, 32)
    y_test: (1000,)

# Batch Normalization: Forward Pass
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))

# Means should be close to zero and stds close to one.
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])

# Now means should be close to beta and stds close to gamma.
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means: [ -2.3814598  -13.18038246   1.91780462]
      stds:  [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means: [4.66293670e-17 5.27355937e-17 9.57567359e-18]
      stds:  [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means: [11. 12. 13.]
      stds:  [0.999995 1.99999  2.999985]

```
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    {'mode': 'test', 'running_mean': array([-0.32415038, 18.55718135, 14.18894184]), 'running_var': array([1205.42219358, 1249.80138292, 1311.43080532])}
    After batch normalization (test-time):
      means: [-0.03927354 -0.04349152 -0.10452688]
      stds:  [1.01531427 1.01238373 0.97819987]

# Batch Normalization: Backward Pass
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)

# You should expect to see relative errors between 1e-13 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029235612572515e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

# Batch Normalization: Alternative Backward Pass
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hard part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

# Fully Connected Networks with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

**Hint:** You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`.

```
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 5.65e-06
    W3 relative error: 4.14e-10
    b1 relative error: 2.22e-08
    b2 relative error: 5.55e-09
    b3 relative error: 1.02e-10
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.17e-09
    gamma1 relative error: 7.47e-09
    gamma2 relative error: 3.35e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 5.55e-09
    b2 relative error: 2.22e-08
    b3 relative error: 2.10e-10
    beta1 relative error: 6.32e-09
    beta2 relative error: 3.48e-09
    gamma1 relative error: 6.27e-09
    gamma2 relative error: 4.14e-09

# Batch Normalization for Deep Networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.314000; val_acc: 0.266000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.385000; val_acc: 0.279000
    (Iteration 41 / 200) loss: 2.041103
    (Epoch 3 / 10) train acc: 0.494000; val_acc: 0.308000
    (Iteration 61 / 200) loss: 1.753903
    (Epoch 4 / 10) train acc: 0.531000; val_acc: 0.307000
    (Iteration 81 / 200) loss: 1.246584
    (Epoch 5 / 10) train acc: 0.574000; val_acc: 0.313000
    (Iteration 101 / 200) loss: 1.320589
    (Epoch 6 / 10) train acc: 0.634000; val_acc: 0.338000
    (Iteration 121 / 200) loss: 1.159472
    (Epoch 7 / 10) train acc: 0.686000; val_acc: 0.325000
    (Iteration 141 / 200) loss: 1.156478
    (Epoch 8 / 10) train acc: 0.769000; val_acc: 0.334000
    (Iteration 161 / 200) loss: 0.630541
    (Epoch 9 / 10) train acc: 0.801000; val_acc: 0.344000
    (Iteration 181 / 200) loss: 0.862817
    (Epoch 10 / 10) train acc: 0.788000; val_acc: 0.335000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662209
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696062
    (Epoch 6 / 10) train acc: 0.536000; val_acc: 0.346000
    (Iteration 121 / 200) loss: 1.550785
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.310000
    (Iteration 141 / 200) loss: 1.436308
    (Epoch 8 / 10) train acc: 0.622000; val_acc: 0.342000
    (Iteration 161 / 200) loss: 1.000868
    (Epoch 9 / 10) train acc: 0.654000; val_acc: 0.328000
    (Iteration 181 / 200) loss: 0.925457
    (Epoch 10 / 10) train acc: 0.726000; val_acc: 0.335000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch Normalization and Initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train eight-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```
np.random.seed(231)

# Try training a very deep net with batchnorm.
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
    print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
    bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

    bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    bn_solver.train()
    bn_solvers_ws[weight_scale] = bn_solver

    solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
    solver.train()
    solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20

    /home/dishank/Desktop/misclenus/cs231n/assignment2/cs231n/layers.py:151: RuntimeWarning: divide by zero encountered in log
      loss = -np.sum(np.log(probs[np.arange(N), y])) / N

    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```
# Plot results of weight scale experiment.
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs. weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the weight initialization scale affect models with/without batch normalization differently, and why?

## Answer:
[FILL THIS IN]

# Batch Normalization and Batch Size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)

    # Try training a very deep net with batchnorm.
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
[FILL THIS IN]

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_forward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization.

# Simulate the forward pass for a two-layer network.
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)

# Means should be close to zero and stds close to one.
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])

# Now means should be close to beta and stds close to gamma.
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

```
# Gradient check batchnorm backward pass.
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

# You should expect to see relative errors between 1e-12 and 1e-8.
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

# Layer Normalization and Batch Size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
[FILL THIS IN]


## Subsurface Data Analytics

### Bayesian Linear Regression for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Bayesian Linear Regression for Subsurface Modeling in Python

Here's a simple workflow, demonstration of Bayesian linear regression for subsurface modeling workflows. This should help you get started with building subsurface models with data analytics and machine learning. Here's some basic details about Bayesian linear regression.

#### Bayesian Updating

The prediction for Bayesian linear regression is distributed:

\begin{equation}
y \sim N(\beta^{T}X, \sigma^{2} I)
\end{equation}

We estimate the model parameter distributions through Bayesian updating for infering the model parameters from a prior and likelihood from training data.

\begin{equation}
p(\beta | y, X) = \frac{p(y,X| \beta) p(\beta)}{p(y,X)}
\end{equation}

In general for continuous features, we must use a sampling method, such as Markov chain Monte Carlo to sample the posterior.

#### Workflow Goals

Learn the basics of isotonic regression in Python for analysis, modeling and prediction of porosity from density. This includes:

* Basic Python workflows and data preparation

* Training / fitting a ridge regression model

* Checking the model and learning about the impact of hyperparameters

#### Objective

Content to support the PGE 383: Subsurface Machine Learning class.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

There are examples below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Import Required Packages

Let's import the GeostatsPy package.

```python
import os                                                   # to set current working directory
import numpy as np                                          # arrays and matrix math
import scipy.stats as st                                    # statistical methods
import pandas as pd                                         # DataFrames
import matplotlib.pyplot as plt                             # for plotting
from sklearn.isotonic import IsotonicRegression
from sklearn.model_selection import train_test_split        # train and test split
from sklearn.metrics import explained_variance_score
import seaborn as sns
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

We will need to install a couple of packages that are not available in the standard Anaconda 3.7 build.

* pymc3 - streamlines Markov chain Monte Carlo Sampling

* arviz - exploratory analysis of Bayesian models

This can be accomplished with 'pip install [package_name]' in a terminal window.

```python
import warnings
warnings.filterwarnings('ignore')
import pymc3 as pm
```

    WARNING (theano.configdefaults): g++ not available, if using conda: `conda install m2w64-toolchain`
    WARNING (theano.configdefaults): g++ not detected ! Theano will be unable to execute optimized C-implementations (for both CPU and GPU) and will default to Python implementations. Performance will be severely degraded. To remove this warning, set Theano flags cxx to an empty string.
    WARNING (theano.tensor.blas): Using NumPy C-API based implementation for BLAS functions.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see below) data file in this working directory.

```python
os.chdir("d:\PGE383")                                       # set the working directory
```

#### Loading Data

Let's load the provided dataset. 'GrainSize_por.csv' is available at https://github.com/GeostatsGuy/GeoDataSets. It is a comma delimited file with 20 grain size ($mm$) and porosity (as a percentage) measures from the subsurface. We load the data file with the pandas 'read_csv' function into a data frame we called 'df'.  We preview each with the head function from Pandas DataFrames.

```python
df = pd.read_csv("GrainSize_Por.csv")                    # read a .csv file in as a DataFrame
df.head()
```

      Grainsize
      Porosity

      0
      2.689284
      5.400000

      1
      6.683638
      10.647582

      2
      9.877106
      7.926271

      3
      12.239344
      11.278953

      4
      27.623312
      10.508753

#### Visualize the Data

Looks like a reasonable dataset for a linear regression model.

```python
X_data = df.iloc[:,0]
y_data = df.iloc[:,1]
plt.subplot(111)
plt.scatter(X_data, y_data, color='black', s = 20, alpha = 0.5, label='sample data')
plt.title('Porosity from Grainsize with Training Data'); plt.xlabel('Grain Size (mm)'); plt.ylabel('Porosity (%)')
plt.xlim(0,100)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
plt.show()
```

#### Least Squares Linear Regression

Let's calculate the ordinary least squares (OLS) linear regression model for comparison.

* for a non-informative prior our Bayesian linear regression model parameters will be centered on the OLS solution.

```python
OLS_slope, OLS_intercept, r_value, p_value, std_err = st.linregress(X_data,y_data)

print('The model parameters are, slope (b1) = ' + str(round(OLS_slope,2)) + ', and the intercept (b0) = ' + str(round(OLS_intercept,2)))
```

    The model parameters are, slope (b1) = 0.16, and the intercept (b0) = 8.12

```python
plt.subplot(111)
plt.plot(X_data, y_data, 'o', label='sample data', color = 'red', alpha = 0.2, markeredgecolor = 'black')
plt.plot(X_data, OLS_intercept + OLS_slope*X_data, label='model', color = 'black')
plt.title('Porosity vs Grain Size')
plt.xlabel('Grain Size (mm)')
plt.ylabel('Porosity (%)')
plt.legend()
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

The following code was based on the work of Will Koehrsen [blog post](https://towardsdatascience.com/introduction-to-bayesian-linear-regression-e66e60791ea7) and example workflow in [Jupyter Notebook](https://github.com/WillKoehrsen/Data-Analysis/blob/master/bayesian_lr/Bayesian%20Linear%20Regression%20Demonstration.ipynb).

https://github.com/WillKoehrsen/Data-Analysis/blob/master/bayesian_lr/Bayesian%20Linear%20Regression%20Demonstration.ipynb

#### Bayesian Linear Regression

In the following code we take advantage of the pymc3 package's API to build a very readible and compact model.

The parts include:

* specify the prior models for $\beta_0$, $\beta_1$ and $\sigma^2$.

* specify the model estimates

* define the distribution of the predictions and the observated predictor features

* specify the specific form for the MCMC steps

* conduct MCMC sampling

Note the default MCMC method in pymc3 is the Hamiltonian Monte Carlo (HMC).

```python
with pm.Model() as linear_all_data:
    # prior distributions, Naive - set to Gaussian with very large variance
    intercept = pm.Normal('Intercept', mu = 0, sd = 10)# model B0 - Intercept Prior
    slope = pm.Normal('slope', mu = 0, sd = 10)        # model B1 - Slope Prior
    sigma = pm.HalfNormal('sigma', sd = 10)            # homoscedastic error in standard deviation

    # specify the model, linear regression
    mean = intercept + slope * X_data                  # estimate of model

    # define the distribution of the predictions from the model and provide the sampled response features
    Y_obs = pm.Normal('Y_obs', mu = mean, sd = sigma, observed = y_data.values[:]) # Observed values

    # specify the sampler
    step = pm.NUTS() # use the No-U-Turn Sampler (NUTS) for Markov Chains Monte Carlo MCMC

    # Posterior distribution
    linear_trace_all_data = pm.sample(1000, step)
```

    Multiprocess sampling (4 chains in 4 jobs)
    NUTS: [sigma, slope, Intercept]
    Sampling 4 chains: 100%|██████████| 6000/6000 [00:47<00:00, 126.61draws/s]
    The acceptance probability does not match the target. It is 0.9011898422570589, but should be close to 0.8. Try to increase the number of tuning steps.
    The acceptance probability does not match the target. It is 0.8891872075972533, but should be close to 0.8. Try to increase the number of tuning steps.

#### Summary Statistics for the Model Parameter Posterior Samples

It is useful to assess the mean, dispersion and credible intervals for the model parameters.

```python
def trace_mean(x):
    return pd.Series(np.average(x, 0), name='mean')

def trace_sd(x):
    return pd.Series(np.std(x, 0), name='sd')

def trace_quantiles(x):
    return pd.DataFrame(pm.quantiles(x, [2.5, 97.5]))

# Get the expectation / average for the model parameters from sampling the posterior distribution
bayes_exp_intercept = pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean]).iloc[0,0]
bayes_exp_slope = pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean]).iloc[1,0]

pm.stats.summary(linear_trace_all_data, stat_funcs = [trace_mean, trace_sd, trace_quantiles])
```

      mean
      sd
      2.5
      97.5

      Intercept
      8.048910
      0.921594
      6.258840
      9.825338

      slope
      0.159634
      0.018504
      0.123918
      0.196745

      sigma
      2.141588
      0.393305
      1.521258
      3.087641

#### Diagnostic Plots

For each of the model parameters, let's look at the results for the 4 Markov chains.

* distributions / PDFs based on kernel density estimates (KDE) from the 1000 MCMC states

* trace of the samples over each state to observe for burn-in, stuck in local minimums / sufficient variation

```python
pm.traceplot(linear_trace_all_data)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Posterior Distributions

We can take our 1000 states from our 4 Markov chains as samples of the posterior distributions of our Bayesian linear regresion model parameters.

* once again the distributions are represented as PDFs from kernel density estimation

```python
pm.plot_posterior(linear_trace_all_data, credible_interval = 0.95)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.0, wspace=0.2, hspace=0.2)
plt.show()
```

#### Visualize the Credible Intervals for the Model Parameters

We can visualize the credible intervals for all of the model parameters

* one for each Markov chain

```python
pm.forestplot(linear_trace_all_data, credible_interval = 0.95)
```

#### Sampling the Bayesian Regression Model

We can sample from the resulting uncertainty distributions for the model parameters.

* we'll sample 1000 times and visualize the reuslting models

```python
plt.subplot(111)
plt.scatter(X_data, y_data, color='black', s = 20, alpha = 0.5, label='sample data', zorder = 3)
pm.plot_posterior_predictive_glm(linear_trace_all_data, samples = 1000, eval=np.linspace(2, 100, 100), linewidth = 1, color = 'red', alpha = 0.02, label = 'Bayesian Posterior Models',lm = lambda x, sample: sample['Intercept'] + sample['slope'] * x, zorder = 1)
plt.plot(X_data, bayes_exp_intercept + bayes_exp_slope*X_data, label='OLS Model', color = 'black', zorder= 2)
plt.title('Porosity from Grainsize Bayesian Model'); plt.xlabel('Grain Size (mm)'); plt.ylabel('Porosity (%)')
plt.xlim(0,100)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
plt.show()
```

#### Posterior Prediction

Let's pick a specific grain size and apply the above 1000 samples of the model to sample the Bayes posterior prediction.

* note we would still need to add in the noise term, $\sigma$, to the model samples

```python
grain_size = 40
nruns = linear_trace_all_data['slope'].shape[0]

plt.subplot(121)
plt.scatter(X_data, y_data, color='black', s = 20, alpha = 0.5, label='sample data', zorder = 3)
pm.plot_posterior_predictive_glm(linear_trace_all_data, samples = 1000, eval=np.linspace(2, 100, 100), linewidth = 1, color = 'red', alpha = 0.02, label = 'Bayesian Posterior Models',lm = lambda x, sample: sample['Intercept'] + sample['slope'] * x, zorder = 1)

plt.plot(X_data, OLS_intercept + OLS_slope*X_data, label='model', color = 'black', zorder= 2)
plt.vlines(x = grain_size,ymin = 0, ymax = 30, label = 'OLS Prediction',colors = 'black', linestyles='--')
plt.title('Porosity from Grainsize Bayesian Model'); plt.xlabel('Grain Size (mm)'); plt.ylabel('Porosity (%)')
plt.xlim(0,100)#; plt.ylim(0,1500000)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()

plt.subplot(122)
model_uncert = linear_trace_all_data['Intercept'] + linear_trace_all_data['slope'] * grain_size
sns.kdeplot(model_uncert, label = 'Bayes Model Uncertainty', color = 'blue')
bayes_prediction = linear_trace_all_data['Intercept'] + linear_trace_all_data['slope'] * grain_size + np.random.normal(loc=0,scale=linear_trace_all_data['sigma'],size=nruns)
sns.kdeplot(bayes_prediction, label = 'Bayes Posterior Prediction', color = 'black')
plt.vlines(x = OLS_intercept + OLS_slope * grain_size,ymin = 0, ymax = 2.5, label = 'OLS Prediction',colors = 'red', linestyles='--')
plt.xlim(0,30),plt.ylim(0,1.0)
plt.xlabel('Porosity (%)'), plt.ylabel('Density');
plt.title('Posterior Prediction for Grain Size = ' + str(grain_size));
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.2, hspace=0.2)
plt.legend()
plt.show()
```

#### Comments

This was a basic demonstration of Bayesian linear regression. Motivation to work with Bayesian methods:

* work with the model parameter uncertainty distributions
* ingrate prior information.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


##### Exercise 5.1

Consider the diagrams on the right in Figure  5.2. Why does the value function jump up for the last two rows in the rear? Why does it drop off for the whole last row on the left? Why are the frontmost values higher in the upper diagrams than in the lower?

Jumps up for last two rows in the rear since sticking for 20 and 21 in blackjack usually result in a win. It drops off for the last row on the left since if the dealer has an Ace, it's bad news for the player. Finally, the frontmost values are higher in the upper diagrams than in the lower since having a usable Ace makes it less likely that an Ace will make the player bust.

##### Exercise 5.2

The backup diagram for the Monte Carlo estimation of $Q^\pi$ is similar to the backup diagram for $V^\pi$, but the root is a state-action pair, not just a state. The diagram ends in a terminal state.

##### Some notes
"Without a model (as we had in DP chapter 4)... state values alone are not sufficient. One must explicitly estimate the value of each action in order for the values to be useful in suggesting a policy. "

By model, the author means, having the transition probabilities and transition rewards available at hand.

" For policy evaluation to work for action values, we must assure continual exploration. One way to do this is by specifying that the first step of each episode starts at a state-action pair, and that every such pair has a nonzero probability of being selected as the start. This guarantees that all state-action pairs will be visited an infinite number of times in the limit of an infinite number of episodes. We call this the assumption of exploring starts."

Has the following been proved on Monte Carlo ES?
"Convergence to this optimal fixed point seems inevitable as the changes to the action-value function decrease over time, but has not yet been formally proved. In our opinion, this is one of the most fundamental open questions in reinforcement learning."

##### Questions

**Question**: In 5.6, in the Figure 5.7, (c), the update to w, why is there a 1 in the numerator? It's because $\pi(s,a)$ is a deterministic policy!

Why are we taking $\tau$ to be the latest time at which the actions are not greedy? Is it because our estimate for $Q^\pi$ only improves for nongreedy actions as stated in the section?

**Question**: In 5.4, the conditions for the policy improvement theorem require optimal substructure right? So even though Monte Carlo is more robust to violations of the Markov Property since it doesn't bootstrap $V$ or $Q$, we are still assuming that greedy updates in GPI (Generalized Policy Improvement) will allow us to arrive at the optimal action-value functions due to the Markov Property?

##### Exercise 5.3

What is the Monte Carlo estimate analogous to (5.3) for action values, given returns generated using $\pi'$?

I'm not sure, but I think it would be something like:

$$Q(s,a) = \frac{\sum_i^{n_s} \frac{p_i(s,a)}{p_i'(s,a)} R_i(s,a) }{ \sum_i^{n_s} \frac{p_i(s,a)}{p_i'(s,a)} }$$

Where $R_i(s,a)$ is the reward following state $s$ and action $a$, and $p_i(s_t, a_t) = P^{a_t}_{s_t, s_{t+1}}  \prod_{k=t+1}^{T_i(s_t, a_t) - 1} \pi(s_k, a_k)P^{a_k}_{s_k, s_{k+1}} $.

##### Exercise 5.4

See the code in `chapter5_racetrack` for the solution!

##### Exercise 5.5

Modify the algorithm for first-visit MC policy evaluation (Figure  5.1) to use the incremental implementation for stationary averages described in Section 2.5.

We just need to update the algorithm so that $Returns(s)$ is a 1x1 array for each $s \in S$. Before part(b) of the algorithm, we need to initialize $k = 0$, and in part (b) in the loop, we do:

- $Returns(s) \leftarrow Returns(s) + \frac{1}{k + 1}[R - Returns(s)]$
- $k \leftarrow k + 1$

##### Exercise 5.6

Derive the weighted-average update rule (5.5) from (5.4). Follow the pattern of the derivation of the unweighted rule (2.4) from (2.1).

We have,

$
\begin{equation}
\begin{split}
V_{n+1} =& \frac{\sum_{k=1}^{n+1} w_k R_k }{\sum{k+1}{n+1} w_k}\\
=& \frac{w_{n+1} R_{n+1} + \sum_{k=1}^n w_k R_k }{W_{n+1}}
\end{split}
\end{equation}
$

where $W_{n+1} = W_n + w_{n+1}$ and $W_0 = 0$. Then we have:

$
\begin{equation}
\begin{split}
V_{n+1} =& \frac{1}{W_{n+1}} [w_{n+1}R_{n+1} + V_n W_n]\\
=& \frac{1}{W_{n+1}} [w_{n+1}R_{n+1} + V_n (W_{n+1} - w_{n+1})]\\
=& V_n +  \frac{w_{n+1}}{W_{n+1}} [R_{n+1} - V_n ]
\end{split}
\end{equation}
$

##### Exercise 5.7

Modify the algorithm for the off-policy Monte Carlo control algorithm (Figure  5.7) to use the method described above for incrementally computing weighted averages.

Before the repeat loop, we need to initialize $W = 0$. Then in the repeat loop we do:

- get $w$ and $t$ as usual
- delete lines for $N(s,a)$ and $D(s,a)$
- $W \leftarrow w + W$
- $Q(s,a) \leftarrow Q(s,a) + \frac{w}{W} [R_t - Q(s,a)] $

```python

```


>###  Ejemplo 3  **Tarea**
>El tiempo en el cual un movimiento browniano se mantiene sobre su punto máximo en el intervalo [0,1] tiene una distribución
>$$F(x)=\frac{2}{\pi}\sin^{-1}(\sqrt x),\quad 0\leq x\leq 1$$
Genere muestres aleatorias que distribuyan según la función dada usando el método de la transformada inversa y grafique en una gráfica el historias 100 muestras generadas y comparela con el función F(x) dada, esto con el fín de validar que el procedimiento fue realizado de manera correcta

```python
import numpy as np
import scipy.stats as ss
from scipy.stats import rayleigh
import statsmodels as st
import matplotlib.pyplot as plt
from sympy import Derivative, diff, simplify
import matplotlib.pyplot as plt
```

```python
u=(2/np.pi)*np.arcos(root(x))

```

```python
N=10**6
def d_fun(N):
    U= np.random(N)
    return np.sin(np.pi*U/2)**2
f= lambda x: 1/(np.pi*np.sqrt(x)*np.sqrt(1-x))
#dominio 0<=x<=1
x=np.arange(0.1,.99,.01)
plt.plot(x,f(x))

```

```python
d = d_fun(N)
plt.hist(d, bins= 100, density=True)
plt.show()
```

### Ejemplo 4
Distribución de Rayleigh
$$F(x)=1-e^{-2x(x-b)},\quad x\geq b $$

```python
#derivar F(x) para obtener f(x)
import numpy as np
import scipy.stats as ss
from scipy.stats import rayleigh
import statsmodels as st
import matplotlib.pyplot as plt
from sympy import Derivative, diff, simplify
import matplotlib.pyplot as plt
#lambda x: np.diff(1-np.exp*(-2*x*(x-b)))
```

```python

```

```python
def D_ray(x,b):
    return np.diff(1-np.exp*(-2*x*(x-b)))
r = rayleigh.rvs(size=1000)
plt.hist(r, density=True, histtype='stepfilled', alpha=0.2)
plt.legend(loc='best', frameon=False)
plt.show()

```

```python
values = plt.hist(np.random.rayleigh(3, 100000), bins=200, density=True)
```

### Ejemplo binomial:
La distribución binomial modela el número de éxitos de n ensayos independientes donde hay una probabilidad p de éxito en cada ensayo.

Generar una variable aletoria binomial con parámetros $n=10$ y $p=0.7$. Recordar que
$$X\sim binomial(n,p) \longrightarrow p_i=P(X=i)=\frac{n!}{i!(n-i)!}p^i(1-p)^{n-i},\quad i=0,1,\cdots,n$$
> ## Tarea: Demostrar la validez de la siguiente ecuación
>$$p_{i+1}=\frac{n-i}{i+1}\frac{p}{1-p} p_i \longrightarrow \text{Hablar de las ventajas que sea recursiva}$$

```python
#generarU, verificar si cumple X=0
#Comparar la aumulada hasta n
```

```python
n=
p=np.zeros(N)
N=
P[0]=(1-p)**n

----
for i in range (N):
    p.append(pi+1#toda la expresion)((n-i)/(i+1))*(p/(1-p)*p[-1])

#crear la binomial
# Función que calcula la probabilidad acumulada
def P_acum_Binomial(n,p):
    c  = p/(1-p)
    pr = [(1-p)**n]
    add = pr.append
    for i in range(n+1):
        add((c*(n-i)/(i+1))*pr[-1])
    return np.cumsum(pr)
----
#quitar for
#se crea un vector y se llena
n=
p=np.zeros(N)
N=
P[0]=(1-p)**n
def P_acum_Binomial(n,p): #sera global para usarla en cualqioer momento, para recursividad crear otra func dentro de la global
             p=np.zeros(N)
             P[0]=(1-p)**n
             def llenar_vector(i): #contador de 0 a 1
             global p
             p[i+1]=((c*(n-i)/(i+1))*p[i])
             #compresion de listas, llama al vector de ceros
             [llenar_vector(i) for i in range(N-1)]
             return np.cumsum(p)
```

```python

```


########################################################
### This file is used to generate Table 7-8, Fig 4-5 ###
########################################################

- [Forward Problem](#Forward-Problem)
    - [Verify Assumption 1](#Verify-Assumption-1)
        - [Table 7](#Table-7)
        - [Table 8](#Table-8)
    - [Verify Lemma 1](#Verify-Lemma-1)
        - [Left plot in Figure 4](#Left-plot-in-Figure-4)
    - [Verify Theorem 3.1](#Verify-Theorem-3.1)
        - [Right plot in Figure 4](#Right-plot-in-Figure-4)
- [Inverse Problem](#Inverse-Problem)
    - [Verify Assumption 2](#Verify-Assumption-2)
    - [Verify Theorem 4.2](#Verify-Theorem-4.2)
        - [Figure 5](#Figure-5)

**Note(Important!!!!!)**
Since this file would require data from Data directory, you will need to generate the data files, there are two ways to generate the data.

- Way 1: run 'GenerateData.ipynb' or 'GenerateData.py' to get QoI values at quadrature points in order to get coefficients in PCE; and all the ratio evaluations
- Way 2: run 'GenerateData_ParallelVersion.ipynb' and set multiple processors

```python
import os
import scipy.io as sio #for the i/o
import numpy as np
import numpy.polynomial.hermite_e as H
from math import factorial
from scipy.stats import norm
from scipy.stats import gaussian_kde as kde
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import pyplot as plt

import ipywidgets as widgets
from ipywidgets import interact, interactive, fixed, interact_manual
from matplotlib import cm

%matplotlib inline
```

```python
####### Plot Formatting ######
plt.rc('lines', linewidth = 1.5)
plt.rc('xtick', labelsize = 14)
plt.rc('ytick', labelsize = 14)
plt.rc('legend',fontsize=14)
# plt.rcParams["font.family"] = "serif"
plt.rcParams['axes.labelsize'] = 20
plt.rcParams['axes.titlesize'] = 12
plt.rcParams['lines.markersize'] = 6
plt.rcParams['figure.figsize'] = (8.0, 6.0)
```

Problem:

\begin{align*}
- \nabla\cdot(A\nabla u) &= (e^{\lambda_1}\lambda_1^2\pi^2 + e^{\lambda_2}\lambda_2^2\pi^2)u \\
u &= 0 \, \text{ on } \Gamma_0 \,\text{(   Left edge)}\\
(A\nabla u)\cdot n &= -e^{\lambda_2}\lambda_2\pi \sin\lambda_1\pi x\sin \lambda_2\pi y \, \text{ on } \Gamma_1  \, \text{(   Top edge)}\\
(A\nabla u)\cdot n &= e^{\lambda_2}\lambda_2\pi \sin\lambda_1\pi x\sin \lambda_2\pi y \, \text{ on } \Gamma_2  \,\text{(   Bottom edge)}\\
(A\nabla u)\cdot n &= e^{\lambda_1}\lambda_1\pi \cos\lambda_1\pi x\cos \lambda_2\pi y \, \text{ on } \Gamma_3  \,\text{(   Right edge)}\\
\end{align*}

where
$$ A = \begin{bmatrix} e^{\lambda_1} & 0 \\ 0 & e^{\lambda_2} \end{bmatrix} $$
and $\Omega = [0,1]\times [0,1]$.

**Exact solution:**
$$ u = \sin \lambda_1\pi x \cos \lambda_2 \pi y$$

QoI is:

\begin{align*}
Q(\lambda_1,\lambda_2) &=  \frac{1}{(b-a)(d-c)}\int_{c}^{d} \int_{a}^{b} u(x,y,\lambda_1,\lambda_2)\, dx\, dy \\
&= \frac{1}{(b-a)(d-c)} \int_{a}^{b} \sin \lambda_1\pi x \, dx \int_{c}^{d} \cos \lambda_2 \pi y \, dy\\
&= \frac{1}{(b-a)(d-c)} \left( \frac{\cos \lambda_1\pi a - \cos\lambda_1\pi b}{\lambda_1\pi}\right) \left( \frac{\sin \lambda_2\pi d - \sin \lambda_2\pi c}{\lambda_2\pi}\right) \\
&= \frac{(\cos \lambda_1\pi a - \cos\lambda_1\pi b)(\sin \lambda_2\pi d - \sin\lambda_2\pi c) }{(b-a)(d-c)\lambda_1\lambda_2\pi^2}
\end{align*}

### When standard normal

$$ \Phi_{ij}(x,y) = \Phi_i(x) \Phi_j(y)$$

$$ (\Phi_{ij}, \Phi_{kl}) = \int_{-\infty}^{\infty} \int_{-\infty}^{\infty} \Phi_{ij}(x,y)\Phi_{kl}(x,y) e^{-\frac{x^2+y^2}{2}}\, dx\, dy = (\sqrt{2\pi})^2 i! j! \delta_{ik}\delta_{jl}$$

$$ Q(\lambda_1, \lambda_2) = \sum_{i,j=0}^{\infty} q_{ij} \Phi_{ij}(\lambda_1,\lambda_2)$$

$$ q_{ij} = \frac{1}{(\Phi_{ij},\Phi_{ij})} \biggr(Q(\lambda_1, \lambda_2), \  \Phi_{ij}(\lambda_1,\lambda_2)\biggr) = \frac{1}{(\sqrt{2\pi})^2 i! j! } \biggr(Q(\lambda_1, \lambda_2), \  \Phi_{ij}(\lambda_1,\lambda_2)\biggr)$$

**Gauss-Hermite Quadrature**

\begin{align*}
\biggr(Q(\lambda_1, \lambda_2), \  \Phi_{ij}(\lambda_1,\lambda_2)\biggr) &= \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} Q(\lambda_1,\lambda_2) \Phi_i(\lambda_1) e^{-\frac{\lambda_1^2}{2}} \, d\lambda_1 \Phi_j(\lambda_2) e^{-\frac{\lambda_2^2}{2}} \, d\lambda_2\\
&= \int_{-\infty}^{\infty} \biggr( \sum_{k=1}^n w_k Q(\lambda_1^{(k)}, \lambda_2)\Phi_i(\lambda_1^{(k)}) \biggr) \Phi_j(\lambda_2) e^{-\frac{\lambda_2^2}{2}} \, d\lambda_2\\
&= \sum_{l=1}^n  w_l \biggr( \sum_{k=1}^n w_k Q(\lambda_1^{(k)}, \lambda_2^{(l)})\Phi_i(\lambda_1^{(k)}) \biggr) \Phi_j(\lambda_2^{(l)})
\end{align*}

### When generalized normal

**Quadrature Rule**

If
$$ \int_{-\infty}^{\infty} f(x) e^{-\frac{x^2}{2}}\, dx \approx \sum_{i=1}^n w_if(x_i)$$
we will have
$$ \int_{-\infty}^{\infty} f(x) e^{-\frac{(x-\mu)^2}{2\sigma^2}}\, dx \approx \sigma \sum_{i=1}^n w_if(\mu + \sigma x_i)$$

If $\lambda_1$, $\lambda_2$ don't follow standard normal $N(0,1)$, assume $\lambda_1\sim N(\mu_1, \sigma_1^2)$, $\lambda_2\sim N(\mu_2, \sigma_2^2)$, we will have

$$ Q(\lambda_1, \lambda_2) = \sum_{i,j=0}^{\infty} q_{ij} \Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right)$$

Still use above $\Phi_{ij}$, but in order to make it orthogonal, we should have basis like

$$ \Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right) $$

then

\begin{align*}
&\,\,\, \left(  \Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right),  \Phi_{kl}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right)  \right)\\
&= \left(  \Phi_{i}\left(\frac{\lambda_1 - \mu_1}{\sigma_1}\right)\Phi_{j}\left(\frac{\lambda_2 - \mu_2}{\sigma_2}\right),  \Phi_{k}\left(\frac{\lambda_1 - \mu_1}{\sigma_1}\right) \Phi_l\left(\frac{\lambda_2 - \mu_2}{\sigma_2}\right)  \right)\\
&= \int_{R} \int_{R} \Phi_{i}\left(\frac{\lambda_1 - \mu_1}{\sigma_1}\right)\Phi_{j}\left(\frac{\lambda_2 - \mu_2}{\sigma_2}\right)  \Phi_{k}\left(\frac{\lambda_1 - \mu_1}{\sigma_1}\right) \Phi_l\left(\frac{\lambda_2 - \mu_2}{\sigma_2}\right)e^{-\frac{(\lambda_1-\mu_1)^2}{2\sigma_1^2}} e^{-\frac{(\lambda_2-\mu_2)^2}{2\sigma_2^2}} d\lambda_1 d\lambda_2 \\
&= \sigma_1\sigma_2 \int_{R} \int_{R} \Phi_{i}(y_1)\Phi_j(y_2)\Phi_{k}(y_1)\Phi_l(y_2) e^{-\frac{y_1^2}{2}}e^{-\frac{y_2^2}{2}}\, dy_1\, dy_2 \\
&= \sigma_1\sigma_2 2\pi i! j! \delta_{ik}\delta_{jl}
\end{align*}

Since
$$ Q(\lambda_1, \lambda_2) = \sum_{i,j=0}^{\infty} q_{ij} \Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right)$$
then

\begin{align*}
q_{ij} &= \frac{1}{\left(\Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right),\Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right)\right)} \biggr(Q(\lambda_1, \lambda_2), \  \Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right)\biggr) \\
&= \frac{1}{\sigma_1 \sigma_2 2\pi i! j! } \biggr(Q(\lambda_1, \lambda_2), \  \Phi_{ij}\left(\frac{\lambda_1 - \mu_1}{\sigma_1},\frac{\lambda_2 - \mu_2}{\sigma_2}\right)\biggr)\\
&= \frac{1}{\sigma_1 \sigma_2 2\pi i! j! } \int_{-\infty}^{\infty}\int_{-\infty}^{\infty} Q(\lambda_1,\lambda_2) \Phi_i\left(\frac{\lambda_1-\mu_1}{\sigma_1}\right) e^{-\frac{(\lambda_1-\mu_1)^2}{2\sigma_1^2}} \, d\lambda_1 \Phi_j\left(\frac{\lambda_2-\mu_2}{\sigma_2}\right) e^{-\frac{(\lambda_2-\mu_2)^2}{2\sigma_2^2}} \, d\lambda_2\\
&= \frac{1}{\sigma_1 \sigma_2 2\pi i! j! } \int_{-\infty}^{\infty} \sigma_1 \biggr( \sum_{k=1}^n w_k Q(\mu_1+\sigma_1\lambda_1^{(k)}, \lambda_2)\Phi_i(\lambda_1^{(k)}) \biggr) \Phi_j\left(\frac{\lambda_2 - \mu_2}{\sigma_2}\right) e^{-\frac{(\lambda_2-\mu_2)^2}{2\sigma_2^2}} \, d\lambda_2\\
&= \frac{1}{\sigma_1 \sigma_2 2\pi i! j! } \sigma_2 \sum_{l=1}^n  w_l \sigma_1 \biggr( \sum_{k=1}^n w_k Q(\mu_1+\sigma_1\lambda_1^{(k)}, \mu_2+\sigma_2\lambda_2^{(l)})\Phi_i(\lambda_1^{(k)}) \biggr) \Phi_j(\lambda_2^{(l)})\\
&= \frac{1}{2\pi i! j! } \sum_{l=1}^n  w_l \biggr( \sum_{k=1}^n w_k Q(\mu_1+\sigma_1\lambda_1^{(k)}, \mu_2+\sigma_2\lambda_2^{(l)})\Phi_i(\lambda_1^{(k)}) \biggr) \Phi_j(\lambda_2^{(l)})
\end{align*}

Assume
$$ \lambda_1 \sim N(\mu_1, \sigma_1^2) = N(0, 0.1^2) \ \ \ \lambda_2 \sim N(\mu_2, \sigma_2^2) = N(0, 0.1^2) $$

```python
proc_size = 25   # this number is determined by the number of data files
```

```python
mu1 = 0
mu2 = 0
sigma1 = 0.1
sigma2 = 0.1
```

## Get $Q_n(\lambda_1,\lambda_2)$ (need to compute coef $q_{ij}$)

```python
def Hermite_2d(i,j,x,y):
    '''
    Phi_{i,j}(x,y) = Phi_i(x) * Phi_j(y)  (left: 2d; right: 1d)
    '''
    c = np.zeros((20,20))
    c[i,j] = 1
    return H.hermeval2d(x, y, c)

Q_FEM_quad = np.zeros(int(400))   #already include information of mu1, mu2, sigma1, sigma2
for i in range(proc_size):
    filename = os.path.join(os.getcwd(), "Data", "Q_FEM_quad_") + str(i) + ".mat"
    partial_data = sio.loadmat(filename)
    Q_FEM_quad += partial_data['Q_FEM'].reshape(int(400))

def Phi(n):
    #define H_n
    coeffs = [0]*(n+1)
    coeffs[n] = 1
    return coeffs

def q(i,j):
    '''
    copmute coefficient q_{ij}
    Set up Gauss-Hermite quadrature, weighting function is exp^{-x^2}
    '''
    x, w=H.hermegauss(20)
    Q=sum([w[ldx]*sum([w[kdx] * Q_FEM_quad[ldx*20+kdx] * H.hermeval(x[kdx],Phi(i)) for kdx in range(20)])*H.hermeval(x[ldx],Phi(j)) for ldx in range(20)])
    q= Q/(2*np.pi*factorial(i)*factorial(j))

    return q

qij = np.zeros((10,10))
for i in range(10):
    for j in range(10):
        qij[i,j] = q(i,j)

def Q(n,x,y):
    result = 0
    for i in range(n+1):
        for j in range(n+1):
            if i+j <=n:
                result += qij[i,j]*Hermite_2d(i,j,(x-mu1)/sigma1,(y-mu2)/sigma2)
    return result

def Qexact(x,y,a=0.4,b=0.6,c=0.4,d=0.6):
    sol = (np.cos(x*np.pi*a)-np.cos(x*np.pi*b))*(np.sin(y*np.pi*d)-np.sin(y*np.pi*c))/((b-a)*(d-c)*x*y*np.pi**2)
    return sol
```

```python
#Visualize the error between PCE and exact
fig = plt.figure(figsize=(8,7))
ax = fig.gca(projection='3d')

# Make data.
X = np.arange(-0.5, 0.5, 0.1)
Y = np.arange(-0.5, 0.5, 0.1)
X, Y = np.meshgrid(X, Y)
# Z = Q(1, X, Y)
# Z = Q(5,X,Y) - Qexact(X,Y)
Z = Qexact(X,Y)

# Plot the surface.
surf = ax.plot_surface(X, Y, Z, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)

# Customize the z axis.
ax.set_zlim(-0.5, 0.5)        #Qexact zlim(-0.5,0.5); Qi-Q_exact zlim(-1.2,1.2)
ax.set_xlabel("$\lambda_1$", fontsize = 16)
ax.set_ylabel("$\lambda_2$", fontsize = 16)
ax.tick_params(direction='out', length=6, width=2, pad=1 ,colors='black',  \
               grid_color='r', grid_alpha=0.5, labelsize=14)    #pad:dist b/t label and axis
cbaxes = fig.add_axes([0.88, 0.25, 0.03, 0.5])     #1st:horizontal, 2&4th:vertical, 3rd:width
fig.colorbar(surf, shrink=5, aspect=14, cax=cbaxes)
plt.show()
# fig.savefig("images/2Q_exact_5std")   #2Q_exact_5std, 2Q1_5std
```

## Forward Problem
Assume
$$ \lambda_1 \sim N(\mu_1, \sigma_1^2) = N(0, 0.1^2) \ \ \ \lambda_2 \sim N(\mu_2, \sigma_2^2) = N(0, 0.1^2) $$

### Verify Assumption 1

```python
##### Generate data in Table 4 and 5 #####
def assumption1(n, J):
    np.random.seed(123456)
    lam1sample = np.random.normal(mu1, sigma1, J)
    lam2sample = np.random.normal(mu2, sigma2, J)
    pfprior_sample_n = Q(n, lam1sample, lam2sample)
    pfprior_dens_n = kde(pfprior_sample_n)
    x = np.linspace(-1, 1, 1000)
    return np.round(np.max(np.abs(np.gradient(pfprior_dens_n(x), x))), 2), np.round(np.max(pfprior_dens_n(x)), 2)

size_J = [int(1E3), int(1E4), int(1E5)]
degree_n = [1, 2, 3, 4, 5]
Bound_matrix, Lip_Bound_matrix = np.zeros((3, 5)), np.zeros((3, 5))
for i in range(3):
    for j in range(5):
        n, J = degree_n[j], size_J[i]
        Lip_Bound_matrix[i, j] = assumption1(n, J)[0]
        Bound_matrix[i, j] = assumption1(n, J)[1]
```

#### Table 7

```python
###########################################
################ Table 7 ##################
###########################################
print('Table 7')
print('Bound under certain n and J values')
print(Bound_matrix)
```

    Table 7
    Bound under certain n and J values
    [[2.63 2.63 2.61 2.61 2.61]
     [2.64 2.64 2.61 2.61 2.61]
     [2.6  2.6  2.57 2.57 2.57]]

#### Table 8

```python
###########################################
################ Table 8 ##################
###########################################
print('Table 8')
print('Lipschitz bound under certain n and J values')
print(Lip_Bound_matrix)
```

    Table 8
    Lipschitz bound under certain n and J values
    [[12.11 12.11 11.95 11.95 11.95]
     [11.47 11.47 11.41 11.41 11.41]
     [11.07 11.07 10.86 10.86 10.86]]

```python
#### Use plot to show the difference between the exact pushforward and approximate pushforward #####
fig = plt.figure()
def plot_pushforward(n,J):
    #pfprior_dens = kde(Q_FEM)

    np.random.seed(123456)
    lam1sample = np.random.normal(mu1,sigma1,J)
    lam2sample = np.random.normal(mu2,sigma2,J)

    pfprior_sample = Qexact(lam1sample,lam2sample)
    pfprior_dens = kde(pfprior_sample)

    pfprior_sample_n = Q(n,lam1sample,lam2sample)
    pfprior_dens_n = kde(pfprior_sample_n)

    fig.clear()
    qplot = np.linspace(-1,1, num=1000)

    plt.plot(qplot,pfprior_dens(qplot),color='r', linestyle='-.', linewidth=4,label="$\pi_{\mathcal{D}}^{Q}$")
    plt.plot(qplot,pfprior_dens_n(qplot),label="$\pi_{\mathcal{D}}^{Q_n}$")

    plt.title('Lipschitz const. = %4.2f and Bound = %2.2f' %(np.max(np.abs(np.gradient(pfprior_dens_n(qplot), qplot))),
                                                                             np.max(pfprior_dens_n(qplot))))
    plt.legend()

interact(plot_pushforward,
         n = widgets.IntSlider(value=int(1),min=int(1),max=int(5),step=1),
         J = widgets.IntSlider(value=int(1E3),min=int(1E3),max=int(1E5),step=int(1E3)))
```

    interactive(children=(IntSlider(value=1, description='n', max=5, min=1), IntSlider(value=1000, description='J'…

    <function __main__.plot_pushforward(n, J)>

### Verify Lemma 1
**Print out Monte Carlo Approximation of $	\|\pi_{\mathcal{D}}^Q(q)-\pi_{\mathcal{D}}^{Q_n}(q)\|_{L^r(\mathcal{D_c})} $ where $r>0$ and $D_c=[-1,1]$**

```python
##### Generate data for the left plot of Fig 3 #####
# Define push-forward densities
N_kde = int(1E4)
N_mc = int(1E4)

np.random.seed(123456)
lam1sample = np.random.normal(mu1,sigma1,N_kde)
lam2sample = np.random.normal(mu2,sigma2,N_kde)
pfprior_dens = kde(Qexact(lam1sample,lam2sample))

def pfprior_dens_n(n,x):
    pfprior_sample_n =  Q(n,lam1sample,lam2sample)
    pdf = kde(pfprior_sample_n)
    return pdf(x)
```

```python
# **Print out Monte Carlo Approximation of $	\|\pi_{\mathcal{D}}^Q(q)-\pi_{\mathcal{D}}^{Q_n}(q)\|_{L^r(\mathcal{D_c})} $ where $r>0$ and $D_c=[-1,1]$**
np.random.seed(123456)
qsample = np.random.uniform(-1,1,N_mc)

def error_r_onD(r,n):
    diff = (np.mean((np.abs(pfprior_dens_n(n,qsample) - pfprior_dens(qsample)))**r))**(1/r)
    return diff

error_r_D = np.zeros((5,5))
for i in range(5):
    for j in range(5):
        error_r_D[i,j] = error_r_onD(i+1,j+1)
```

```python
np.set_printoptions(linewidth=110)
print('L^r error on data space for Forward Problem',end='\n\n')
print(error_r_D)
```

    L^r error on data space for Forward Problem

    [[5.78985998e-03 5.78986860e-03 3.66465835e-05 3.66458716e-05 4.68447031e-06]
     [1.00098540e-02 1.00098559e-02 6.48003418e-05 6.48018329e-05 6.83125067e-06]
     [1.29985177e-02 1.29985144e-02 8.36155928e-05 8.36174583e-05 7.98012310e-06]
     [1.51923261e-02 1.51923210e-02 9.70172096e-05 9.70187251e-05 8.72619888e-06]
     [1.68583067e-02 1.68583012e-02 1.07115873e-04 1.07116859e-04 9.26298356e-06]]

```python
#### To make it cleaner, create Directory "images" to store all the figures ####
imagepath = os.path.join(os.getcwd(),"images")
os.makedirs(imagepath,exist_ok=True)
```

#### Left plot in Figure 4

```python
###########################################
######### The left plot of Fig 4 ##########
###########################################
fig = plt.figure()
plt.xlim([0,6])
marker = ['-D', '-o', '-v', '-s', '-.']
for i in range(5):
    plt.semilogy([1,2,3,4,5],error_r_D[i,:],marker[i],label='r = ' + np.str(i+1))
plt.xlabel('Order of PCE (n)')
plt.ylabel('$L^r$'+' Error in Push-Forward on '+'$\mathcal{D}$')
plt.legend()
# fig.savefig("images/2forward_error_D.png")
fig.savefig("images/Fig4(Left).png")
```

### Verify Theorem 3.1
**Print out Monte Carlo Approximation of $	\|\pi_{\mathcal{D}}^Q(Q(\lambda))-\pi_{\mathcal{D}}^{Q_n}(Q_n(\lambda))\|_{L^2(\Lambda)} $**

```python
##### Generate data for the right plot of Fig 4 #####
np.random.seed(123456)
lam1_seed = np.random.normal(mu1,sigma1,int(1E4))
lam2_seed = np.random.normal(mu2,sigma2,int(1E4))   #int(1E4) since Q_FEM size

error_2_Lam = np.zeros(5)
for i in range(5):
    pfprior_sample = Qexact(lam1_seed,lam2_seed)
    error_2_Lam[i] = (np.mean((np.abs(pfprior_dens_n(i+1,Q(i+1,lam1_seed,lam2_seed))\
                                      - pfprior_dens(pfprior_sample)))**2))**(1/2)
```

```python
np.set_printoptions(linewidth=110)
print('L^2 error on parameter space for Forward Problem',end='\n\n')
print(error_2_Lam)
```

    L^2 error on parameter space for Forward Problem

    [2.78836391e-02 2.78836376e-02 2.33315440e-04 2.33317057e-04 1.82851667e-05]

#### Right plot in Figure 4

```python
############################################
######### The right plot of Fig 4 ##########
############################################
fig = plt.figure()
plt.xlim([0,6])
plt.semilogy([1,2,3,4,5], error_2_Lam, '-s' )#, label='$L^2(\Lambda)$ error')
plt.xlabel('Order of PCE (n)')
plt.ylabel('$L^2$'+' Error in Push-Forward on '+'$\Lambda$');
# fig.savefig("images/2forward_error_lam.png")
fig.savefig("images/Fig4(Right).png")
```

## Inverse Problem

Compute $\pi_{\Lambda}^u$ and $\pi_{\Lambda}^{u,n}$

Observed pdf is $\pi_{\mathcal{D}} \sim N(0.3,0.1^2)$

Guess is $\lambda_1\sim N(0,0.1)$, $\lambda_2\sim N(0,0.1)$

Verify Result of Theorem 4.2:

$Q_n(\lambda)\to Q(\lambda)$ in $L^p(\Lambda)$, $\pi_{\Lambda}^{init}\in L^p(\mathcal{D})$. If Assumptions 1, 2 hold, $\{\pi_{\mathcal{D}}^{Q_n}\}$ are uniformly integrable in $L^p(\mathcal{D})$, then
\begin{equation}
        \pi_{\Lambda}^{u,n}(\lambda) \to \pi_{\Lambda}^{u}(\lambda) \text{ in } L^p(\Lambda)
\end{equation}

```python
def rejection_sampling(r):
    N = r.size # size of proposal sample set
    check = np.random.uniform(low=0,high=1,size=N) # create random uniform weights to check r against
    M = np.max(r)
    new_r = r/M     # normalize weights
    idx = np.where(new_r>=check)[0] # rejection criterion
    return idx

def pdf_obs(x):
    return norm.pdf(x, loc=0.1, scale=0.1)
```

```python
#### Use plot to show the difference between the pushforward of the init and the observed #####
plt.figure()
xx = np.linspace(-1,1,100)
plt.plot(xx,pdf_obs(xx),label="$\pi_{\mathcal{D}}^{obs}$")
plt.plot(xx,pfprior_dens(xx), label="$\pi_{\mathcal{D}}^{Q(init)}$")
plt.xlabel("$\mathcal{D}$")
plt.legend();
# fig.savefig("images/2obs_pushforward.png")
```

### Verify Assumption 2

```python
def Meanr(n):
    if n==0:
        pfprior_sample = Qexact(lam1_seed,lam2_seed)
        r = pdf_obs(pfprior_sample)/pfprior_dens(pfprior_sample)
    else:
        pfprior_sample_n = Q(n,lam1_seed,lam2_seed)
        r = pdf_obs(pfprior_sample_n)/pfprior_dens_n(n,pfprior_sample_n)
    return np.mean(r)

Expect_r = np.zeros(6)
for i in range(6):
    Expect_r[i] = Meanr(i)
```

```python
print('Expected ratio for verifying Assumption 2')
print(Expect_r[1:])
```

    Expected ratio for verifying Assumption 2
    [1.00179843 1.00179844 1.00197    1.00197    1.00197088]

### Verify Theorem 4.2

Print out Monte Carlo Approximation of $\|\pi_{\Lambda}^{u,n}(\lambda)-\pi_{\Lambda}^u(\lambda)\|_{L^2(\Lambda)} $

\begin{align*}
\|\pi_{\Lambda}^{u,n}(\lambda)-\pi_{\Lambda}^u(\lambda)\|^2_{L^2(\Lambda)} &= \int (\pi^i(\lambda))^2 (r_n(\lambda) - r(\lambda))^2\, d\mu_{\Lambda}\\
&= \mathbb{E}_i (\pi^i(\lambda)(r_n(\lambda) - r(\lambda))^2)\\
&\approx \frac{1}{N} \sum_{j=1}^N \pi^i(\lambda^{(j)})(r_n(\lambda^{(j)}) -r(\lambda^{(j)}) )^2
\end{align*}

```python
##### Load data for Fig 5 #####
# Print out Monte Carlo Approximation of $\|\pi_{\Lambda}^{u,n}(\lambda)-\pi_{\Lambda}^u(\lambda)\|_{L^2(\Lambda)} $
init_eval = np.zeros(int(1E4))
for i in range(int(1E4)):
    init_eval[i] = norm.pdf(lam1_seed[i], loc=0.1, scale=0.1)*norm.pdf(lam2_seed[i], loc=0.1, scale=0.1)

r = np.zeros(int(1E4))
for i in range(proc_size):
    filename = os.path.join(os.getcwd(), "Data", "r_") + str(i) + ".mat"
    partial_data = sio.loadmat(filename)
    r += partial_data['r'].reshape(int(1E4))

rn = np.zeros((6,int(1E4)))
for i in range(6):
    for j in range(proc_size):
        filename = os.path.join(os.getcwd(), "Data", "r")  + str(i+1) + '_' + str(j) + ".mat"
        partial_data = sio.loadmat(filename)
        rn[i,:] += partial_data['r'].reshape(int(1E4))
```

```python
error_Update = np.zeros(5)

for i in range(5):
    error_Update[i] = (np.mean(init_eval*(rn[i,:] - r)**2))**(1/2)
```

```python
np.set_printoptions(linewidth=110)
print('L^2 Error for Inverse Problem',end='\n\n')
print(error_Update)
```

    L^2 Error for Inverse Problem

    [4.25879362e-01 4.25878780e-01 5.37649557e-03 5.37799511e-03 3.87794081e-04]

#### Figure 5

```python
###########################################
################ Figure 5 #################
###########################################
fig = plt.figure()
plt.xlim([0,6])
plt.semilogy([1,2,3,4,5], error_Update, '-s')#, label='$L^2(\Lambda)$ error')
plt.xlabel('Order of PCE (n)')
plt.ylabel('$L^2$'+' Error in Update')
# fig.savefig("images/2inverse_error.png")
fig.savefig("images/Fig5.png")
```