# Batch Normalization
One way to make deep networks easier to train is to use more sophisticated optimization procedures such as SGD+momentum, RMSProp, or Adam. Another strategy is to change the architecture of the network to make it easier to train.
One idea along these lines is batch normalization which was proposed by [1] in 2015.

The idea is relatively straightforward. Machine learning methods tend to work better when their input data consists of uncorrelated features with zero mean and unit variance. When training a neural network, we can preprocess the data before feeding it to the network to explicitly decorrelate its features; this will ensure that the first layer of the network sees data that follows a nice distribution. However, even if we preprocess the input data, the activations at deeper layers of the network will likely no longer be decorrelated and will no longer have zero mean or unit variance since they are output from earlier layers in the network. Even worse, during the training process the distribution of features at each layer of the network will shift as the weights of each layer are updated.

The authors of [1] hypothesize that the shifting distribution of features inside deep neural networks may make training deep networks more difficult. To overcome this problem, [1] proposes to insert batch normalization layers into the network. At training time, a batch normalization layer uses a minibatch of data to estimate the mean and standard deviation of each feature. These estimated means and standard deviations are then used to center and normalize the features of the minibatch. A running average of these means and standard deviations is kept during training, and at test time these running averages are used to center and normalize features.

It is possible that this normalization strategy could reduce the representational power of the network, since it may sometimes be optimal for certain layers to have features that are not zero-mean or unit variance. To this end, the batch normalization layer includes learnable shift and scale parameters for each feature dimension.

[1] [Sergey Ioffe and Christian Szegedy, "Batch Normalization: Accelerating Deep Network Training by Reducing
Internal Covariate Shift", ICML 2015.](https://arxiv.org/abs/1502.03167)

```python
# As usual, a bit of setup
import time
import numpy as np
import matplotlib.pyplot as plt
from cs231n.classifiers.fc_net import *
from cs231n.data_utils import get_CIFAR10_data
from cs231n.gradient_check import eval_numerical_gradient, eval_numerical_gradient_array
from cs231n.solver import Solver

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0) # set default size of plots
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'

# for auto-reloading external modules
# see http://stackoverflow.com/questions/1907993/autoreload-of-modules-in-ipython
%load_ext autoreload
%autoreload 2

def rel_error(x, y):
    """ returns relative error """
    return np.max(np.abs(x - y) / (np.maximum(1e-8, np.abs(x) + np.abs(y))))

def print_mean_std(x,axis=0):
    print('  means: ', x.mean(axis=axis))
    print('  stds:  ', x.std(axis=axis))
    print()
```

```python
# Load the (preprocessed) CIFAR10 data.
data = get_CIFAR10_data()
for k, v in data.items():
  print('%s: ' % k, v.shape)
```

    X_train:  (49000, 3, 32, 32)
    y_train:  (49000,)
    X_val:  (1000, 3, 32, 32)
    y_val:  (1000,)
    X_test:  (1000, 3, 32, 32)
    y_test:  (1000,)

## Batch normalization: forward
In the file `cs231n/layers.py`, implement the batch normalization forward pass in the function `batchnorm_forward`. Once you have done so, run the following to test your implementation.

Referencing the paper linked to above in [1] may be helpful!

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after batch normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before batch normalization:')
print_mean_std(a,axis=0)

gamma = np.ones((D3,))
beta = np.zeros((D3,))
# Means should be close to zero and stds close to one
print('After batch normalization (gamma=1, beta=0)')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)

gamma = np.asarray([1.0, 2.0, 3.0])
beta = np.asarray([11.0, 12.0, 13.0])
# Now means should be close to beta and stds close to gamma
print('After batch normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = batchnorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=0)
```

    Before batch normalization:
      means:  [ -2.3814598  -13.18038246   1.91780462]
      stds:   [27.18502186 34.21455511 37.68611762]

    After batch normalization (gamma=1, beta=0)
      means:  [7.10542736e-17 7.82707232e-17 9.71445147e-18]
      stds:   [0.99999999 1.         1.        ]

    After batch normalization (gamma= [1. 2. 3.] , beta= [11. 12. 13.] )
      means:  [11. 12. 13.]
      stds:   [0.99999999 1.99999999 2.99999999]

```python
# Check the test-time forward pass by running the training-time
# forward pass many times to warm up the running averages, and then
# checking the means and variances of activations after a test-time
# forward pass.

np.random.seed(231)
N, D1, D2, D3 = 200, 50, 60, 3
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)

bn_param = {'mode': 'train'}
gamma = np.ones(D3)
beta = np.zeros(D3)

for t in range(50):
  X = np.random.randn(N, D1)
  a = np.maximum(0, X.dot(W1)).dot(W2)
  batchnorm_forward(a, gamma, beta, bn_param)

bn_param['mode'] = 'test'
X = np.random.randn(N, D1)
a = np.maximum(0, X.dot(W1)).dot(W2)
a_norm, _ = batchnorm_forward(a, gamma, beta, bn_param)

# Means should be close to zero and stds close to one, but will be
# noisier than training-time forward passes.
print('After batch normalization (test-time):')
print_mean_std(a_norm,axis=0)
```

    After batch normalization (test-time):
      means:  [-0.03927354 -0.04349152 -0.10452688]
      stds:   [1.01531428 1.01238373 0.97819988]

## Batch normalization: backward
Now implement the backward pass for batch normalization in the function `batchnorm_backward`.

To derive the backward pass you should write out the computation graph for batch normalization and backprop through each of the intermediate nodes. Some intermediates may have multiple outgoing branches; make sure to sum gradients across these branches in the backward pass.

Once you have finished, run the following to numerically check your backward pass.

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
fx = lambda x: batchnorm_forward(x, gamma, beta, bn_param)[0]
fg = lambda a: batchnorm_forward(x, a, beta, bn_param)[0]
fb = lambda b: batchnorm_forward(x, gamma, b, bn_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = batchnorm_forward(x, gamma, beta, bn_param)
dx, dgamma, dbeta = batchnorm_backward(dout, cache)
#You should expect to see relative errors between 1e-13 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.7029258328157158e-09
    dgamma error:  7.420414216247087e-13
    dbeta error:  2.8795057655839487e-12

## Batch normalization: alternative backward
In class we talked about two different implementations for the sigmoid backward pass. One strategy is to write out a computation graph composed of simple operations and backprop through all intermediate values. Another strategy is to work out the derivatives on paper. For example, you can derive a very simple formula for the sigmoid function's backward pass by simplifying gradients on paper.

Surprisingly, it turns out that you can do a similar simplification for the batch normalization backward pass too!

In the forward pass, given a set of inputs $X=\begin{bmatrix}x_1\\x_2\\...\\x_N\end{bmatrix}$,

we first calculate the mean $\mu$ and variance $v$.
With $\mu$ and $v$ calculated, we can calculate the standard deviation $\sigma$  and normalized data $Y$.
The equations and graph illustration below describe the computation ($y_i$ is the i-th element of the vector $Y$).

\begin{align}
& \mu=\frac{1}{N}\sum_{k=1}^N x_k  &  v=\frac{1}{N}\sum_{k=1}^N (x_k-\mu)^2 \\
& \sigma=\sqrt{v+\epsilon}         &  y_i=\frac{x_i-\mu}{\sigma}
\end{align}

The meat of our problem during backpropagation is to compute $\frac{\partial L}{\partial X}$, given the upstream gradient we receive, $\frac{\partial L}{\partial Y}.$ To do this, recall the chain rule in calculus gives us $\frac{\partial L}{\partial X} = \frac{\partial L}{\partial Y} \cdot \frac{\partial Y}{\partial X}$.

The unknown/hart part is $\frac{\partial Y}{\partial X}$. We can find this by first deriving step-by-step our local gradients at
$\frac{\partial v}{\partial X}$, $\frac{\partial \mu}{\partial X}$,
$\frac{\partial \sigma}{\partial v}$,
$\frac{\partial Y}{\partial \sigma}$, and $\frac{\partial Y}{\partial \mu}$,
and then use the chain rule to compose these gradients (which appear in the form of vectors!) appropriately to compute $\frac{\partial Y}{\partial X}$.

If it's challenging to directly reason about the gradients over $X$ and $Y$ which require matrix multiplication, try reasoning about the gradients in terms of individual elements $x_i$ and $y_i$ first: in that case, you will need to come up with the derivations for $\frac{\partial L}{\partial x_i}$, by relying on the Chain Rule to first calculate the intermediate $\frac{\partial \mu}{\partial x_i}, \frac{\partial v}{\partial x_i}, \frac{\partial \sigma}{\partial x_i},$ then assemble these pieces to calculate $\frac{\partial y_i}{\partial x_i}$.

You should make sure each of the intermediary gradient derivations are all as simplified as possible, for ease of implementation.

After doing so, implement the simplified batch normalization backward pass in the function `batchnorm_backward_alt` and compare the two implementations by running the following. Your two implementations should compute nearly identical results, but the alternative implementation should be a bit faster.

```python
np.random.seed(231)
N, D = 100, 500
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

bn_param = {'mode': 'train'}
out, cache = batchnorm_forward(x, gamma, beta, bn_param)

t1 = time.time()
dx1, dgamma1, dbeta1 = batchnorm_backward(dout, cache)
t2 = time.time()
dx2, dgamma2, dbeta2 = batchnorm_backward_alt(dout, cache)
t3 = time.time()

print('dx difference: ', rel_error(dx1, dx2))
print('dgamma difference: ', rel_error(dgamma1, dgamma2))
print('dbeta difference: ', rel_error(dbeta1, dbeta2))
print('speedup: %.2fx' % ((t2 - t1) / (t3 - t2)))
```

## Fully Connected Nets with Batch Normalization
Now that you have a working implementation for batch normalization, go back to your `FullyConnectedNet` in the file `cs231n/classifiers/fc_net.py`. Modify your implementation to add batch normalization.

Concretely, when the `normalization` flag is set to `"batchnorm"` in the constructor, you should insert a batch normalization layer before each ReLU nonlinearity. The outputs from the last layer of the network should not be normalized. Once you are done, run the following to gradient-check your implementation.

HINT: You might find it useful to define an additional helper layer similar to those in the file `cs231n/layer_utils.py`. If you decide to do so, do it in the file `cs231n/classifiers/fc_net.py`.

```python
np.random.seed(231)
N, D, H1, H2, C = 2, 15, 20, 30, 10
X = np.random.randn(N, D)
y = np.random.randint(C, size=(N,))

# You should expect losses between 1e-4~1e-10 for W,
# losses between 1e-08~1e-10 for b,
# and losses between 1e-08~1e-09 for beta and gammas.
for reg in [0, 3.14]:
  print('Running check with reg = ', reg)
  model = FullyConnectedNet([H1, H2], input_dim=D, num_classes=C,
                            reg=reg, weight_scale=5e-2, dtype=np.float64,
                            normalization='batchnorm')

  loss, grads = model.loss(X, y)
  print('Initial loss: ', loss)

  for name in sorted(grads):
    f = lambda _: model.loss(X, y)[0]
    grad_num = eval_numerical_gradient(f, model.params[name], verbose=False, h=1e-5)
    print('%s relative error: %.2e' % (name, rel_error(grad_num, grads[name])))
  if reg == 0: print()
```

    Running check with reg =  0
    Initial loss:  2.2611955101340957
    W1 relative error: 1.10e-04
    W2 relative error: 2.85e-06
    W3 relative error: 4.05e-10
    b1 relative error: 4.44e-08
    b2 relative error: 2.22e-08
    b3 relative error: 1.01e-10
    beta1 relative error: 7.33e-09
    beta2 relative error: 1.89e-09
    gamma1 relative error: 6.96e-09
    gamma2 relative error: 1.96e-09

    Running check with reg =  3.14
    Initial loss:  6.996533220108303
    W1 relative error: 1.98e-06
    W2 relative error: 2.28e-06
    W3 relative error: 1.11e-08
    b1 relative error: 5.55e-09
    b2 relative error: 2.22e-08
    b3 relative error: 2.10e-10
    beta1 relative error: 6.65e-09
    beta2 relative error: 4.23e-09
    gamma1 relative error: 6.27e-09
    gamma2 relative error: 5.28e-09

# Batchnorm for deep networks
Run the following to train a six-layer network on a subset of 1000 training examples both with and without batch normalization.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [100, 100, 100, 100, 100]

num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

weight_scale = 2e-2
bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

print('Solver with batch norm:')
bn_solver = Solver(bn_model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True,print_every=20)
bn_solver.train()

print('\nSolver without batch norm:')
solver = Solver(model, small_data,
                num_epochs=10, batch_size=50,
                update_rule='adam',
                optim_config={
                  'learning_rate': 1e-3,
                },
                verbose=True, print_every=20)
solver.train()
```

    Solver with batch norm:
    (Iteration 1 / 200) loss: 2.340974
    (Epoch 0 / 10) train acc: 0.107000; val_acc: 0.115000
    (Epoch 1 / 10) train acc: 0.313000; val_acc: 0.267000
    (Iteration 21 / 200) loss: 2.039365
    (Epoch 2 / 10) train acc: 0.385000; val_acc: 0.279000
    (Iteration 41 / 200) loss: 2.041103
    (Epoch 3 / 10) train acc: 0.493000; val_acc: 0.309000
    (Iteration 61 / 200) loss: 1.753903
    (Epoch 4 / 10) train acc: 0.535000; val_acc: 0.307000
    (Iteration 81 / 200) loss: 1.246584
    (Epoch 5 / 10) train acc: 0.573000; val_acc: 0.314000
    (Iteration 101 / 200) loss: 1.323436
    (Epoch 6 / 10) train acc: 0.647000; val_acc: 0.325000
    (Iteration 121 / 200) loss: 1.135112
    (Epoch 7 / 10) train acc: 0.691000; val_acc: 0.322000
    (Iteration 141 / 200) loss: 1.180117
    (Epoch 8 / 10) train acc: 0.749000; val_acc: 0.311000
    (Iteration 161 / 200) loss: 0.693385
    (Epoch 9 / 10) train acc: 0.806000; val_acc: 0.337000
    (Iteration 181 / 200) loss: 0.935037
    (Epoch 10 / 10) train acc: 0.791000; val_acc: 0.301000

    Solver without batch norm:
    (Iteration 1 / 200) loss: 2.302332
    (Epoch 0 / 10) train acc: 0.129000; val_acc: 0.131000
    (Epoch 1 / 10) train acc: 0.283000; val_acc: 0.250000
    (Iteration 21 / 200) loss: 2.041970
    (Epoch 2 / 10) train acc: 0.316000; val_acc: 0.277000
    (Iteration 41 / 200) loss: 1.900473
    (Epoch 3 / 10) train acc: 0.373000; val_acc: 0.282000
    (Iteration 61 / 200) loss: 1.713156
    (Epoch 4 / 10) train acc: 0.390000; val_acc: 0.310000
    (Iteration 81 / 200) loss: 1.662208
    (Epoch 5 / 10) train acc: 0.434000; val_acc: 0.300000
    (Iteration 101 / 200) loss: 1.696062
    (Epoch 6 / 10) train acc: 0.536000; val_acc: 0.346000
    (Iteration 121 / 200) loss: 1.550785
    (Epoch 7 / 10) train acc: 0.530000; val_acc: 0.310000
    (Iteration 141 / 200) loss: 1.436307
    (Epoch 8 / 10) train acc: 0.622000; val_acc: 0.342000
    (Iteration 161 / 200) loss: 1.000868
    (Epoch 9 / 10) train acc: 0.654000; val_acc: 0.328000
    (Iteration 181 / 200) loss: 0.925455
    (Epoch 10 / 10) train acc: 0.726000; val_acc: 0.335000

Run the following to visualize the results from two networks trained above. You should find that using batch normalization helps the network to converge much faster.

```python
def plot_training_history(title, label, baseline, bn_solvers, plot_fn, bl_marker='.', bn_marker='.', labels=None):
    """utility function for plotting training history"""
    plt.title(title)
    plt.xlabel(label)
    bn_plots = [plot_fn(bn_solver) for bn_solver in bn_solvers]
    bl_plot = plot_fn(baseline)
    num_bn = len(bn_plots)
    for i in range(num_bn):
        label='with_norm'
        if labels is not None:
            label += str(labels[i])
        plt.plot(bn_plots[i], bn_marker, label=label)
    label='baseline'
    if labels is not None:
        label += str(labels[0])
    plt.plot(bl_plot, bl_marker, label=label)
    plt.legend(loc='lower center', ncol=num_bn+1)

plt.subplot(3, 1, 1)
plot_training_history('Training loss','Iteration', solver, [bn_solver], \
                      lambda x: x.loss_history, bl_marker='o', bn_marker='o')
plt.subplot(3, 1, 2)
plot_training_history('Training accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.train_acc_history, bl_marker='-o', bn_marker='-o')
plt.subplot(3, 1, 3)
plot_training_history('Validation accuracy','Epoch', solver, [bn_solver], \
                      lambda x: x.val_acc_history, bl_marker='-o', bn_marker='-o')

plt.gcf().set_size_inches(15, 15)
plt.show()
```

# Batch normalization and initialization
We will now run a small experiment to study the interaction of batch normalization and weight initialization.

The first cell will train 8-layer networks both with and without batch normalization using different scales for weight initialization. The second layer will plot training accuracy, validation set accuracy, and training loss as a function of the weight initialization scale.

```python
np.random.seed(231)
# Try training a very deep net with batchnorm
hidden_dims = [50, 50, 50, 50, 50, 50, 50]
num_train = 1000
small_data = {
  'X_train': data['X_train'][:num_train],
  'y_train': data['y_train'][:num_train],
  'X_val': data['X_val'],
  'y_val': data['y_val'],
}

bn_solvers_ws = {}
solvers_ws = {}
weight_scales = np.logspace(-4, 0, num=20)
for i, weight_scale in enumerate(weight_scales):
  print('Running weight scale %d / %d' % (i + 1, len(weight_scales)))
  bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization='batchnorm')
  model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)

  bn_solver = Solver(bn_model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  bn_solver.train()
  bn_solvers_ws[weight_scale] = bn_solver

  solver = Solver(model, small_data,
                  num_epochs=10, batch_size=50,
                  update_rule='adam',
                  optim_config={
                    'learning_rate': 1e-3,
                  },
                  verbose=False, print_every=200)
  solver.train()
  solvers_ws[weight_scale] = solver
```

    Running weight scale 1 / 20
    Running weight scale 2 / 20
    Running weight scale 3 / 20
    Running weight scale 4 / 20
    Running weight scale 5 / 20
    Running weight scale 6 / 20
    Running weight scale 7 / 20
    Running weight scale 8 / 20
    Running weight scale 9 / 20
    Running weight scale 10 / 20
    Running weight scale 11 / 20
    Running weight scale 12 / 20
    Running weight scale 13 / 20
    Running weight scale 14 / 20
    Running weight scale 15 / 20
    Running weight scale 16 / 20
    Running weight scale 17 / 20
    Running weight scale 18 / 20
    Running weight scale 19 / 20
    Running weight scale 20 / 20

```python
# Plot results of weight scale experiment
best_train_accs, bn_best_train_accs = [], []
best_val_accs, bn_best_val_accs = [], []
final_train_loss, bn_final_train_loss = [], []

for ws in weight_scales:
  best_train_accs.append(max(solvers_ws[ws].train_acc_history))
  bn_best_train_accs.append(max(bn_solvers_ws[ws].train_acc_history))

  best_val_accs.append(max(solvers_ws[ws].val_acc_history))
  bn_best_val_accs.append(max(bn_solvers_ws[ws].val_acc_history))

  final_train_loss.append(np.mean(solvers_ws[ws].loss_history[-100:]))
  bn_final_train_loss.append(np.mean(bn_solvers_ws[ws].loss_history[-100:]))

plt.subplot(3, 1, 1)
plt.title('Best val accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best val accuracy')
plt.semilogx(weight_scales, best_val_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_val_accs, '-o', label='batchnorm')
plt.legend(ncol=2, loc='lower right')

plt.subplot(3, 1, 2)
plt.title('Best train accuracy vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Best training accuracy')
plt.semilogx(weight_scales, best_train_accs, '-o', label='baseline')
plt.semilogx(weight_scales, bn_best_train_accs, '-o', label='batchnorm')
plt.legend()

plt.subplot(3, 1, 3)
plt.title('Final training loss vs weight initialization scale')
plt.xlabel('Weight initialization scale')
plt.ylabel('Final training loss')
plt.semilogx(weight_scales, final_train_loss, '-o', label='baseline')
plt.semilogx(weight_scales, bn_final_train_loss, '-o', label='batchnorm')
plt.legend()
plt.gca().set_ylim(1.0, 3.5)

plt.gcf().set_size_inches(15, 15)
plt.show()
```

## Inline Question 1:
Describe the results of this experiment. How does the scale of weight initialization affect models with/without batch normalization differently, and why?

## Answer:
weight比较小时，batchnorm效果更明显

# Batch normalization and batch size
We will now run a small experiment to study the interaction of batch normalization and batch size.

The first cell will train 6-layer networks both with and without batch normalization using different batch sizes. The second layer will plot training accuracy and validation set accuracy over time.

```python
def run_batchsize_experiments(normalization_mode):
    np.random.seed(231)
    # Try training a very deep net with batchnorm
    hidden_dims = [100, 100, 100, 100, 100]
    num_train = 1000
    small_data = {
      'X_train': data['X_train'][:num_train],
      'y_train': data['y_train'][:num_train],
      'X_val': data['X_val'],
      'y_val': data['y_val'],
    }
    n_epochs=10
    weight_scale = 2e-2
    batch_sizes = [5,10,50]
    lr = 10**(-3.5)
    solver_bsize = batch_sizes[0]

    print('No normalization: batch size = ',solver_bsize)
    model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=None)
    solver = Solver(model, small_data,
                    num_epochs=n_epochs, batch_size=solver_bsize,
                    update_rule='adam',
                    optim_config={
                      'learning_rate': lr,
                    },
                    verbose=False)
    solver.train()

    bn_solvers = []
    for i in range(len(batch_sizes)):
        b_size=batch_sizes[i]
        print('Normalization: batch size = ',b_size)
        bn_model = FullyConnectedNet(hidden_dims, weight_scale=weight_scale, normalization=normalization_mode)
        bn_solver = Solver(bn_model, small_data,
                        num_epochs=n_epochs, batch_size=b_size,
                        update_rule='adam',
                        optim_config={
                          'learning_rate': lr,
                        },
                        verbose=False)
        bn_solver.train()
        bn_solvers.append(bn_solver)

    return bn_solvers, solver, batch_sizes

batch_sizes = [5,10,50]
bn_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('batchnorm')
```

    No normalization: batch size =  5
    Normalization: batch size =  5
    Normalization: batch size =  10
    Normalization: batch size =  50

```python
plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Batch Normalization)','Epoch', solver_bsize, bn_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 2:
Describe the results of this experiment. What does this imply about the relationship between batch normalization and batch size? Why is this relationship observed?

## Answer:
batch_size越大，效果基本越好

# Layer Normalization
Batch normalization has proved to be effective in making networks easier to train, but the dependency on batch size makes it less useful in complex networks which have a cap on the input batch size due to hardware limitations.

Several alternatives to batch normalization have been proposed to mitigate this problem; one such technique is Layer Normalization [2]. Instead of normalizing over the batch, we normalize over the features. In other words, when using Layer Normalization, each feature vector corresponding to a single datapoint is normalized based on the sum of all terms within that feature vector.

[2] [Ba, Jimmy Lei, Jamie Ryan Kiros, and Geoffrey E. Hinton. "Layer Normalization." stat 1050 (2016): 21.](https://arxiv.org/pdf/1607.06450.pdf)

## Inline Question 3:
Which of these data preprocessing steps is analogous to batch normalization, and which is analogous to layer normalization?

1. Scaling each image in the dataset, so that the RGB channels for each row of pixels within an image sums up to 1.
2. Scaling each image in the dataset, so that the RGB channels for all pixels within an image sums up to 1.
3. Subtracting the mean image of the dataset from each image in the dataset.
4. Setting all RGB values to either 0 or 1 depending on a given threshold.

## Answer:
[FILL THIS IN]

# Layer Normalization: Implementation

Now you'll implement layer normalization. This step should be relatively straightforward, as conceptually the implementation is almost identical to that of batch normalization. One significant difference though is that for layer normalization, we do not keep track of the moving moments, and the testing phase is identical to the training phase, where the mean and variance are directly calculated per datapoint.

Here's what you need to do:

* In `cs231n/layers.py`, implement the forward pass for layer normalization in the function `layernorm_backward`.

Run the cell below to check your results.
* In `cs231n/layers.py`, implement the backward pass for layer normalization in the function `layernorm_backward`.

Run the second cell below to check your results.
* Modify `cs231n/classifiers/fc_net.py` to add layer normalization to the `FullyConnectedNet`. When the `normalization` flag is set to `"layernorm"` in the constructor, you should insert a layer normalization layer before each ReLU nonlinearity.

Run the third cell below to run the batch size experiment on layer normalization.

```python
# Check the training-time forward pass by checking means and variances
# of features both before and after layer normalization

# Simulate the forward pass for a two-layer network
np.random.seed(231)
N, D1, D2, D3 =4, 50, 60, 3
X = np.random.randn(N, D1)
W1 = np.random.randn(D1, D2)
W2 = np.random.randn(D2, D3)
a = np.maximum(0, X.dot(W1)).dot(W2)

print('Before layer normalization:')
print_mean_std(a,axis=1)

gamma = np.ones(D3)
beta = np.zeros(D3)
# Means should be close to zero and stds close to one
print('After layer normalization (gamma=1, beta=0)')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)

gamma = np.asarray([3.0,3.0,3.0])
beta = np.asarray([5.0,5.0,5.0])
# Now means should be close to beta and stds close to gamma
print('After layer normalization (gamma=', gamma, ', beta=', beta, ')')
a_norm, _ = layernorm_forward(a, gamma, beta, {'mode': 'train'})
print_mean_std(a_norm,axis=1)
```

    Before layer normalization:
      means:  [-59.06673243 -47.60782686 -43.31137368 -26.40991744]
      stds:   [10.07429373 28.39478981 35.28360729  4.01831507]

    After layer normalization (gamma=1, beta=0)
      means:  [ 4.81096644e-16 -7.40148683e-17  2.22044605e-16 -5.92118946e-16]
      stds:   [0.99999995 0.99999999 1.         0.99999969]

    After layer normalization (gamma= [3. 3. 3.] , beta= [5. 5. 5.] )
      means:  [5. 5. 5. 5.]
      stds:   [2.99999985 2.99999998 2.99999999 2.99999907]

```python
# Gradient check batchnorm backward pass
np.random.seed(231)
N, D = 4, 5
x = 5 * np.random.randn(N, D) + 12
gamma = np.random.randn(D)
beta = np.random.randn(D)
dout = np.random.randn(N, D)

ln_param = {}
fx = lambda x: layernorm_forward(x, gamma, beta, ln_param)[0]
fg = lambda a: layernorm_forward(x, a, beta, ln_param)[0]
fb = lambda b: layernorm_forward(x, gamma, b, ln_param)[0]

dx_num = eval_numerical_gradient_array(fx, x, dout)
da_num = eval_numerical_gradient_array(fg, gamma.copy(), dout)
db_num = eval_numerical_gradient_array(fb, beta.copy(), dout)

_, cache = layernorm_forward(x, gamma, beta, ln_param)
dx, dgamma, dbeta = layernorm_backward(dout, cache)

#You should expect to see relative errors between 1e-12 and 1e-8
print('dx error: ', rel_error(dx_num, dx))
print('dgamma error: ', rel_error(da_num, dgamma))
print('dbeta error: ', rel_error(db_num, dbeta))
```

    dx error:  1.433615146847572e-09
    dgamma error:  4.519489546032799e-12
    dbeta error:  2.276445013433725e-12

# Layer Normalization and batch size

We will now run the previous batch size experiment with layer normalization instead of batch normalization. Compared to the previous experiment, you should see a markedly smaller influence of batch size on the training history!

```python
ln_solvers_bsize, solver_bsize, batch_sizes = run_batchsize_experiments('layernorm')

plt.subplot(2, 1, 1)
plot_training_history('Training accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.train_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)
plt.subplot(2, 1, 2)
plot_training_history('Validation accuracy (Layer Normalization)','Epoch', solver_bsize, ln_solvers_bsize, \
                      lambda x: x.val_acc_history, bl_marker='-^', bn_marker='-o', labels=batch_sizes)

plt.gcf().set_size_inches(15, 10)
plt.show()
```

## Inline Question 4:
When is layer normalization likely to not work well, and why?

1. Using it in a very deep network
2. Having a very small dimension of features
3. Having a high regularization term

## Answer:
layer_norm在2的情况下的效果会不好

```python

```


```python
import mxnet as mx
import numpy as np
import os
import logging
import matplotlib.pyplot as plt
import matplotlib.cm as cm
```

# Building a Variational Autoencoder in MXNet

#### Xiaoyu Lu,  July 5th, 2017

This tutorial guides you through the process of building a variational encoder in MXNet. In this notebook we'll focus on an example using the MNIST handwritten digit recognition dataset. Refer to [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/) for more details on the model description.

## Prerequisites

To complete this tutorial, we need following python packages:

- numpy, matplotlib

## 1. Loading the Data

We first load the MNIST dataset, which contains 60000 training and 10000 test examples. The following code imports required modules and loads the data. These images are stored in a 4-D matrix with shape (`batch_size, num_channels, width, height`). For the MNIST dataset, there is only one color channel, and both width and height are 28, so we reshape each image as a 28x28 array. See below for a visualization:

```python
mnist = mx.test_utils.get_mnist()
image = np.reshape(mnist['train_data'],(60000,28*28))
label = image
image_test = np.reshape(mnist['test_data'],(10000,28*28))
label_test = image_test
[N,features] = np.shape(image)          #number of examples and features
print(N,features)
```

    60000 784

```python
nsamples = 5
idx = np.random.choice(len(mnist['train_data']), nsamples)
_, axarr = plt.subplots(1, nsamples, sharex='col', sharey='row',figsize=(12,3))

for i,j in enumerate(idx):
    axarr[i].imshow(np.reshape(image[j,:],(28,28)), interpolation='nearest', cmap=cm.Greys)

plt.show()
```

We can optionally save the parameters in the directory variable 'model_prefix'. We first create data iterators for MXNet, with each batch of data containing 100 images.

```python
model_prefix = None

batch_size = 100
latent_dim = 5
nd_iter = mx.io.NDArrayIter(data={'data':image},label={'loss_label':label},
                            batch_size = batch_size)
nd_iter_test = mx.io.NDArrayIter(data={'data':image_test},label={'loss_label':label_test},
                            batch_size = batch_size)
```

## 2.  Building the Network Architecture

### 2.1 Gaussian MLP as encoder
Next we constuct the neural network, as in the [paper](https://arxiv.org/abs/1312.6114/), we use *Multilayer Perceptron (MLP)* for both the encoder and decoder. For encoder, a Gaussian MLP is used as follows:

\begin{align}
\log q_{\phi}(z|x) &= \log \mathcal{N}(z:\mu,\sigma^2I) \\
\textit{ where } \mu &= W_2h+b_2, \log \sigma^2 = W_3h+b_3\\
h &= \tanh(W_1x+b_1)
\end{align}

where $\{W_1,W_2,W_3,b_1,b_2,b_3\}$ are the weights and biases of the MLP.
Note below that `encoder_mu`(`mu`) and `encoder_logvar`(`logvar`) are symbols. So, we can use `get_internals()` to get the values of them, after which we can sample the latent variable $z$.

```python
## define data and loss labels as symbols
data = mx.sym.var('data')
loss_label = mx.sym.var('loss_label')

## define fully connected and activation layers for the encoder, where we used tanh activation function.
encoder_h  = mx.sym.FullyConnected(data=data, name="encoder_h",num_hidden=400)
act_h = mx.sym.Activation(data=encoder_h, act_type="tanh",name="activation_h")

## define mu and log variance which are the fully connected layers of the previous activation layer
mu  = mx.sym.FullyConnected(data=act_h, name="mu",num_hidden = latent_dim)
logvar  = mx.sym.FullyConnected(data=act_h, name="logvar",num_hidden = latent_dim)

## sample the latent variables z according to Normal(mu,var)
z = mu + mx.symbol.broadcast_mul(mx.symbol.exp(0.5 * logvar),
                                 mx.symbol.random_normal(loc=0, scale=1, shape=(batch_size, latent_dim)),
                                 name="z")
```

### 2.2 Bernoulli MLP as decoder

In this case let $p_\theta(x|z)$ be a multivariate Bernoulli whose probabilities are computed from $z$ with a feed forward neural network with a single hidden layer:

\begin{align}
\log p(x|z) &= \sum_{i=1}^D x_i\log y_i + (1-x_i)\log (1-y_i) \\
\textit{ where }  y &= f_\sigma(W_5\tanh (W_4z+b_4)+b_5)
\end{align}

where $f_\sigma(\dot)$ is the elementwise sigmoid activation function, $\{W_4,W_5,b_4,b_5\}$ are the weights and biases of the decoder MLP. A Bernouilli likelihood is suitable for this type of data but you can easily extend it to other likelihood types by parsing into the argument `likelihood` in the `VAE` class, see section 4 for details.

```python
# define fully connected and tanh activation layers for the decoder
decoder_z = mx.sym.FullyConnected(data=z, name="decoder_z",num_hidden=400)
act_z = mx.sym.Activation(data=decoder_z, act_type="tanh",name="activation_z")

# define the output layer with sigmoid activation function, where the dimension is equal to the input dimension
decoder_x = mx.sym.FullyConnected(data=act_z, name="decoder_x",num_hidden=features)
y = mx.sym.Activation(data=decoder_x, act_type="sigmoid",name='activation_x')
```

### 2.3 Joint Loss Function for the Encoder and the Decoder

The variational lower bound also called evidence lower bound (ELBO) can be estimated as:

\begin{align}
\mathcal{L}(\theta,\phi;x_{(i)}) \approx \frac{1}{2}\left(1+\log ((\sigma_j^{(i)})^2)-(\mu_j^{(i)})^2-(\sigma_j^{(i)})^2\right) + \log p_\theta(x^{(i)}|z^{(i)})
\end{align}

where the first term is the KL divergence of the approximate posterior from the prior, and the second term is an expected negative reconstruction error. We would like to maximize this lower bound, so we can define the loss to be $-\mathcal{L}$(minus ELBO) for MXNet to minimize.

```python
# define the objective loss function that needs to be minimized
KL = 0.5*mx.symbol.sum(1+logvar-pow( mu,2)-mx.symbol.exp(logvar),axis=1)
loss = -mx.symbol.sum(mx.symbol.broadcast_mul(loss_label,mx.symbol.log(y))
                      + mx.symbol.broadcast_mul(1-loss_label,mx.symbol.log(1-y)),axis=1)-KL
output = mx.symbol.MakeLoss(sum(loss),name='loss')
```

## 3. Training the model

Now, we can define the model and train it. First we will initilize the weights and the biases to be Gaussian(0,0.01), and then use stochastic gradient descent for optimization. To warm start the training, one may also initilize with pre-trainined parameters `arg_params` using `init=mx.initializer.Load(arg_params)`.

To save intermediate results, we can optionally use `epoch_end_callback = mx.callback.do_checkpoint(model_prefix, 1)` which saves the parameters to the path given by model_prefix, and with period every $1$ epoch. To assess the performance, we output $-\mathcal{L}$(minus ELBO) after each epoch, with the command `eval_metric = 'Loss'` which is defined above. We will also plot the training loss for mini batches by accessing the log and saving it to a list, and then parsing it to the argument `batch_end_callback`.

```python
# set up the log
nd_iter.reset()
logging.getLogger().setLevel(logging.DEBUG)

# define function to trave back training loss
def log_to_list(period, lst):
    def _callback(param):
        """The checkpoint function."""
        if param.nbatch % period == 0:
            name, value = param.eval_metric.get()
            lst.append(value)
    return _callback

# define the model
model = mx.mod.Module(
    symbol = output ,
    data_names=['data'],
    label_names = ['loss_label'])
```

```python
# training the model, save training loss as a list.
training_loss=list()

# initilize the parameters for training using Normal.
init = mx.init.Normal(0.01)
model.fit(nd_iter,  # train data
          initializer=init,
          # if eval_data is supplied, test loss will also be reported
          # eval_data = nd_iter_test,
          optimizer='sgd',  # use SGD to train
          optimizer_params={'learning_rate':1e-3,'wd':1e-2},
          # save parameters for each epoch if model_prefix is supplied
          epoch_end_callback = None if model_prefix==None else mx.callback.do_checkpoint(model_prefix, 1),
          batch_end_callback = log_to_list(N/batch_size,training_loss),
          num_epoch=100,
          eval_metric = 'Loss')
```

    INFO:root:Epoch[0] Train-loss=373.547317
    INFO:root:Epoch[0] Time cost=5.020
    INFO:root:Epoch[1] Train-loss=212.232684
    INFO:root:Epoch[1] Time cost=4.651
    INFO:root:Epoch[2] Train-loss=207.448528
    INFO:root:Epoch[2] Time cost=4.665
    INFO:root:Epoch[3] Train-loss=205.369479
    INFO:root:Epoch[3] Time cost=4.758
    INFO:root:Epoch[4] Train-loss=203.651983
    INFO:root:Epoch[4] Time cost=4.672
    INFO:root:Epoch[5] Train-loss=202.061007
    INFO:root:Epoch[5] Time cost=5.087
    INFO:root:Epoch[6] Train-loss=199.348143
    INFO:root:Epoch[6] Time cost=5.056
    INFO:root:Epoch[7] Train-loss=196.266242
    INFO:root:Epoch[7] Time cost=4.813
    INFO:root:Epoch[8] Train-loss=194.694945
    INFO:root:Epoch[8] Time cost=4.776
    INFO:root:Epoch[9] Train-loss=193.699284
    INFO:root:Epoch[9] Time cost=4.756
    INFO:root:Epoch[10] Train-loss=193.036517
    INFO:root:Epoch[10] Time cost=4.757
    INFO:root:Epoch[11] Train-loss=192.555736
    INFO:root:Epoch[11] Time cost=4.678
    INFO:root:Epoch[12] Train-loss=192.020813
    INFO:root:Epoch[12] Time cost=4.630
    INFO:root:Epoch[13] Train-loss=191.648876
    INFO:root:Epoch[13] Time cost=5.158
    INFO:root:Epoch[14] Train-loss=191.057798
    INFO:root:Epoch[14] Time cost=4.781
    INFO:root:Epoch[15] Train-loss=190.315835
    INFO:root:Epoch[15] Time cost=5.117
    INFO:root:Epoch[16] Train-loss=189.311271
    INFO:root:Epoch[16] Time cost=4.707
    INFO:root:Epoch[17] Train-loss=187.285967
    INFO:root:Epoch[17] Time cost=4.745
    INFO:root:Epoch[18] Train-loss=185.271324
    INFO:root:Epoch[18] Time cost=4.692
    INFO:root:Epoch[19] Train-loss=183.510888
    INFO:root:Epoch[19] Time cost=4.762
    INFO:root:Epoch[20] Train-loss=181.756008
    INFO:root:Epoch[20] Time cost=4.838
    INFO:root:Epoch[21] Train-loss=180.546818
    INFO:root:Epoch[21] Time cost=4.764
    INFO:root:Epoch[22] Train-loss=179.479776
    INFO:root:Epoch[22] Time cost=4.791
    INFO:root:Epoch[23] Train-loss=178.352077
    INFO:root:Epoch[23] Time cost=4.981
    INFO:root:Epoch[24] Train-loss=177.385084
    INFO:root:Epoch[24] Time cost=5.292
    INFO:root:Epoch[25] Train-loss=175.920123
    INFO:root:Epoch[25] Time cost=5.097
    INFO:root:Epoch[26] Train-loss=174.377171
    INFO:root:Epoch[26] Time cost=4.907
    INFO:root:Epoch[27] Train-loss=172.590589
    INFO:root:Epoch[27] Time cost=4.484
    INFO:root:Epoch[28] Train-loss=170.933683
    INFO:root:Epoch[28] Time cost=4.348
    INFO:root:Epoch[29] Train-loss=169.866807
    INFO:root:Epoch[29] Time cost=4.647
    INFO:root:Epoch[30] Train-loss=169.182084
    INFO:root:Epoch[30] Time cost=5.034
    INFO:root:Epoch[31] Train-loss=168.121719
    INFO:root:Epoch[31] Time cost=5.615
    INFO:root:Epoch[32] Train-loss=167.389992
    INFO:root:Epoch[32] Time cost=4.733
    INFO:root:Epoch[33] Train-loss=166.189067
    INFO:root:Epoch[33] Time cost=5.041
    INFO:root:Epoch[34] Train-loss=163.783392
    INFO:root:Epoch[34] Time cost=5.168
    INFO:root:Epoch[35] Train-loss=162.167959
    INFO:root:Epoch[35] Time cost=5.019
    INFO:root:Epoch[36] Train-loss=161.192039
    INFO:root:Epoch[36] Time cost=5.064
    INFO:root:Epoch[37] Train-loss=160.307114
    INFO:root:Epoch[37] Time cost=5.180
    INFO:root:Epoch[38] Train-loss=159.591957
    INFO:root:Epoch[38] Time cost=5.440
    INFO:root:Epoch[39] Train-loss=159.109593
    INFO:root:Epoch[39] Time cost=5.119
    INFO:root:Epoch[40] Train-loss=158.463844
    INFO:root:Epoch[40] Time cost=5.299
    INFO:root:Epoch[41] Train-loss=158.037287
    INFO:root:Epoch[41] Time cost=4.856
    INFO:root:Epoch[42] Train-loss=157.598576
    INFO:root:Epoch[42] Time cost=5.227
    INFO:root:Epoch[43] Train-loss=157.097344
    INFO:root:Epoch[43] Time cost=5.237
    INFO:root:Epoch[44] Train-loss=156.594472
    INFO:root:Epoch[44] Time cost=4.783
    INFO:root:Epoch[45] Train-loss=156.177069
    INFO:root:Epoch[45] Time cost=4.834
    INFO:root:Epoch[46] Train-loss=155.825302
    INFO:root:Epoch[46] Time cost=4.902
    INFO:root:Epoch[47] Train-loss=155.318117
    INFO:root:Epoch[47] Time cost=4.966
    INFO:root:Epoch[48] Train-loss=154.890766
    INFO:root:Epoch[48] Time cost=5.012
    INFO:root:Epoch[49] Train-loss=154.504158
    INFO:root:Epoch[49] Time cost=4.844
    INFO:root:Epoch[50] Train-loss=154.035214
    INFO:root:Epoch[50] Time cost=4.736
    INFO:root:Epoch[51] Train-loss=153.692903
    INFO:root:Epoch[51] Time cost=5.057
    INFO:root:Epoch[52] Train-loss=153.257554
    INFO:root:Epoch[52] Time cost=5.044
    INFO:root:Epoch[53] Train-loss=152.849715
    INFO:root:Epoch[53] Time cost=4.783
    INFO:root:Epoch[54] Train-loss=152.483047
    INFO:root:Epoch[54] Time cost=4.842
    INFO:root:Epoch[55] Train-loss=152.091617
    INFO:root:Epoch[55] Time cost=5.044
    INFO:root:Epoch[56] Train-loss=151.715490
    INFO:root:Epoch[56] Time cost=5.029
    INFO:root:Epoch[57] Train-loss=151.362293
    INFO:root:Epoch[57] Time cost=4.873
    INFO:root:Epoch[58] Train-loss=151.003241
    INFO:root:Epoch[58] Time cost=4.729
    INFO:root:Epoch[59] Train-loss=150.619678
    INFO:root:Epoch[59] Time cost=5.068
    INFO:root:Epoch[60] Train-loss=150.296043
    INFO:root:Epoch[60] Time cost=4.458
    INFO:root:Epoch[61] Train-loss=149.964152
    INFO:root:Epoch[61] Time cost=4.828
    INFO:root:Epoch[62] Train-loss=149.694102
    INFO:root:Epoch[62] Time cost=5.012
    INFO:root:Epoch[63] Train-loss=149.290113
    INFO:root:Epoch[63] Time cost=5.193
    INFO:root:Epoch[64] Train-loss=148.934186
    INFO:root:Epoch[64] Time cost=4.999
    INFO:root:Epoch[65] Train-loss=148.657502
    INFO:root:Epoch[65] Time cost=4.810
    INFO:root:Epoch[66] Train-loss=148.331948
    INFO:root:Epoch[66] Time cost=5.201
    INFO:root:Epoch[67] Train-loss=148.018539
    INFO:root:Epoch[67] Time cost=4.833
    INFO:root:Epoch[68] Train-loss=147.746825
    INFO:root:Epoch[68] Time cost=5.187
    INFO:root:Epoch[69] Train-loss=147.406399
    INFO:root:Epoch[69] Time cost=5.355
    INFO:root:Epoch[70] Train-loss=147.181831
    INFO:root:Epoch[70] Time cost=4.989
    INFO:root:Epoch[71] Train-loss=146.860770
    INFO:root:Epoch[71] Time cost=4.934
    INFO:root:Epoch[72] Train-loss=146.604369
    INFO:root:Epoch[72] Time cost=5.283
    INFO:root:Epoch[73] Train-loss=146.351628
    INFO:root:Epoch[73] Time cost=5.062
    INFO:root:Epoch[74] Train-loss=146.102506
    INFO:root:Epoch[74] Time cost=4.540
    INFO:root:Epoch[75] Train-loss=145.828805
    INFO:root:Epoch[75] Time cost=4.875
    INFO:root:Epoch[76] Train-loss=145.571626
    INFO:root:Epoch[76] Time cost=4.856
    INFO:root:Epoch[77] Train-loss=145.365383
    INFO:root:Epoch[77] Time cost=5.003
    INFO:root:Epoch[78] Train-loss=145.101047
    INFO:root:Epoch[78] Time cost=4.718
    INFO:root:Epoch[79] Train-loss=144.810765
    INFO:root:Epoch[79] Time cost=5.127
    INFO:root:Epoch[80] Train-loss=144.619876
    INFO:root:Epoch[80] Time cost=4.737
    INFO:root:Epoch[81] Train-loss=144.399066
    INFO:root:Epoch[81] Time cost=4.742
    INFO:root:Epoch[82] Train-loss=144.220090
    INFO:root:Epoch[82] Time cost=4.810
    INFO:root:Epoch[83] Train-loss=143.904279
    INFO:root:Epoch[83] Time cost=5.176
    INFO:root:Epoch[84] Train-loss=143.734935
    INFO:root:Epoch[84] Time cost=4.921
    INFO:root:Epoch[85] Train-loss=143.499403
    INFO:root:Epoch[85] Time cost=4.692
    INFO:root:Epoch[86] Train-loss=143.304287
    INFO:root:Epoch[86] Time cost=4.778
    INFO:root:Epoch[87] Train-loss=143.096145
    INFO:root:Epoch[87] Time cost=4.962
    INFO:root:Epoch[88] Train-loss=142.877920
    INFO:root:Epoch[88] Time cost=4.815
    INFO:root:Epoch[89] Train-loss=142.677429
    INFO:root:Epoch[89] Time cost=5.127
    INFO:root:Epoch[90] Train-loss=142.499622
    INFO:root:Epoch[90] Time cost=5.463
    INFO:root:Epoch[91] Train-loss=142.300291
    INFO:root:Epoch[91] Time cost=4.639
    INFO:root:Epoch[92] Train-loss=142.111362
    INFO:root:Epoch[92] Time cost=5.064
    INFO:root:Epoch[93] Train-loss=141.912848
    INFO:root:Epoch[93] Time cost=4.894
    INFO:root:Epoch[94] Train-loss=141.723130
    INFO:root:Epoch[94] Time cost=4.635
    INFO:root:Epoch[95] Train-loss=141.516580
    INFO:root:Epoch[95] Time cost=5.063
    INFO:root:Epoch[96] Train-loss=141.362380
    INFO:root:Epoch[96] Time cost=4.785
    INFO:root:Epoch[97] Train-loss=141.178878
    INFO:root:Epoch[97] Time cost=4.699
    INFO:root:Epoch[98] Train-loss=141.004168
    INFO:root:Epoch[98] Time cost=4.959
    INFO:root:Epoch[99] Train-loss=140.865592
    INFO:root:Epoch[99] Time cost=5.155

```python
ELBO = [-training_loss[i] for i in range(len(training_loss))]
plt.plot(ELBO)
plt.ylabel('ELBO');plt.xlabel('epoch');plt.title("training curve for mini batches")
plt.show()
```

As expected, the ELBO is monotonically increasing over epoch, and we reproduced the results given in the paper [Auto-Encoding Variational Bayes](https://arxiv.org/abs/1312.6114/). Now we can extract/load the parameters and then feed the network forward to calculate $y$ which is the reconstructed image, and we can also calculate the ELBO for the test set.

```python
arg_params = model.get_params()[0]
nd_iter_test.reset()
test_batch = nd_iter_test.next()

# if saved the parameters, can load them using `load_checkpoint` method at e.g. 100th epoch
# sym, arg_params, aux_params = mx.model.load_checkpoint(model_prefix, 100)
# assert sym.tojson() == output.tojson()

e = y.bind(mx.cpu(), {'data': test_batch.data[0],
                     'encoder_h_weight': arg_params['encoder_h_weight'],
                     'encoder_h_bias': arg_params['encoder_h_bias'],
                     'mu_weight': arg_params['mu_weight'],
                     'mu_bias': arg_params['mu_bias'],
                     'logvar_weight':arg_params['logvar_weight'],
                     'logvar_bias':arg_params['logvar_bias'],
                     'decoder_z_weight':arg_params['decoder_z_weight'],
                     'decoder_z_bias':arg_params['decoder_z_bias'],
                     'decoder_x_weight':arg_params['decoder_x_weight'],
                     'decoder_x_bias':arg_params['decoder_x_bias'],
                     'loss_label':label})

x_fit = e.forward()
x_construction = x_fit[0].asnumpy()
```

```python
# learning images on the test set
f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))
ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax1.set_title('True image')
ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax2.set_title('Learned image')
ax3.imshow(np.reshape(image_test[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax3.set_title('True image')
ax4.imshow(np.reshape(x_construction[99,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax4.set_title('Learned image')
plt.show()
```

```python
# calculate the ELBO which is minus the loss for test set
metric = mx.metric.Loss()
model.score(nd_iter_test, metric)
```

    [('loss', 140.17346005859375)]

## 4. All together: MXNet-based class VAE

```python
from VAE import VAE
```

One can directly call the class `VAE` to do the training:

```VAE(n_latent=5,num_hidden_ecoder=400,num_hidden_decoder=400,x_train=None,x_valid=None,
batch_size=100,learning_rate=0.001,weight_decay=0.01,num_epoch=100,optimizer='sgd',model_prefix=None,
initializer = mx.init.Normal(0.01),likelihood=Bernoulli)```

The outputs are the learned model and training loss.

```python
# can initilize weights and biases with the learned parameters as follows:
# init = mx.initializer.Load(params)

# call the VAE, output model contains the learned model and training loss
out = VAE(n_latent=2, x_train=image, x_valid=None, num_epoch=200)
```

    INFO:root:Epoch[0] Train-loss=383.478870
    INFO:root:Epoch[0] Time cost=5.075
    INFO:root:Epoch[1] Train-loss=211.923867
    INFO:root:Epoch[1] Time cost=4.741
    INFO:root:Epoch[2] Train-loss=206.789445
    INFO:root:Epoch[2] Time cost=4.601
    INFO:root:Epoch[3] Train-loss=204.428186
    INFO:root:Epoch[3] Time cost=4.865
    INFO:root:Epoch[4] Train-loss=202.417322
    INFO:root:Epoch[4] Time cost=4.606
    INFO:root:Epoch[5] Train-loss=200.635136
    INFO:root:Epoch[5] Time cost=4.711
    INFO:root:Epoch[6] Train-loss=199.009614
    INFO:root:Epoch[6] Time cost=5.159
    INFO:root:Epoch[7] Train-loss=197.565788
    INFO:root:Epoch[7] Time cost=4.588
    INFO:root:Epoch[8] Train-loss=196.524507
    INFO:root:Epoch[8] Time cost=4.905
    INFO:root:Epoch[9] Train-loss=195.725745
    INFO:root:Epoch[9] Time cost=4.426
    INFO:root:Epoch[10] Train-loss=194.902025
    INFO:root:Epoch[10] Time cost=4.685
    INFO:root:Epoch[11] Train-loss=194.026873
    INFO:root:Epoch[11] Time cost=4.622
    INFO:root:Epoch[12] Train-loss=193.350646
    INFO:root:Epoch[12] Time cost=4.712
    INFO:root:Epoch[13] Train-loss=192.737502
    INFO:root:Epoch[13] Time cost=4.618
    INFO:root:Epoch[14] Train-loss=192.338165
    INFO:root:Epoch[14] Time cost=4.763
    INFO:root:Epoch[15] Train-loss=191.888625
    INFO:root:Epoch[15] Time cost=5.168
    INFO:root:Epoch[16] Train-loss=191.170650
    INFO:root:Epoch[16] Time cost=4.809
    INFO:root:Epoch[17] Train-loss=190.307264
    INFO:root:Epoch[17] Time cost=4.622
    INFO:root:Epoch[18] Train-loss=188.988063
    INFO:root:Epoch[18] Time cost=4.543
    INFO:root:Epoch[19] Train-loss=187.616311
    INFO:root:Epoch[19] Time cost=5.154
    INFO:root:Epoch[20] Train-loss=186.352783
    INFO:root:Epoch[20] Time cost=4.661
    INFO:root:Epoch[21] Train-loss=185.428020
    INFO:root:Epoch[21] Time cost=5.193
    INFO:root:Epoch[22] Train-loss=184.543097
    INFO:root:Epoch[22] Time cost=4.519
    INFO:root:Epoch[23] Train-loss=184.029907
    INFO:root:Epoch[23] Time cost=4.732
    INFO:root:Epoch[24] Train-loss=183.643270
    INFO:root:Epoch[24] Time cost=5.011
    INFO:root:Epoch[25] Train-loss=183.246912
    INFO:root:Epoch[25] Time cost=4.706
    INFO:root:Epoch[26] Train-loss=183.065233
    INFO:root:Epoch[26] Time cost=4.673
    INFO:root:Epoch[27] Train-loss=182.680542
    INFO:root:Epoch[27] Time cost=4.628
    INFO:root:Epoch[28] Train-loss=182.428677
    INFO:root:Epoch[28] Time cost=4.772
    INFO:root:Epoch[29] Train-loss=182.219946
    INFO:root:Epoch[29] Time cost=4.571
    INFO:root:Epoch[30] Train-loss=182.070927
    INFO:root:Epoch[30] Time cost=4.603
    INFO:root:Epoch[31] Train-loss=181.837968
    INFO:root:Epoch[31] Time cost=4.559
    INFO:root:Epoch[32] Train-loss=181.624303
    INFO:root:Epoch[32] Time cost=5.069
    INFO:root:Epoch[33] Train-loss=181.534547
    INFO:root:Epoch[33] Time cost=4.654
    INFO:root:Epoch[34] Train-loss=181.239556
    INFO:root:Epoch[34] Time cost=4.776
    INFO:root:Epoch[35] Train-loss=181.098188
    INFO:root:Epoch[35] Time cost=4.571
    INFO:root:Epoch[36] Train-loss=180.820560
    INFO:root:Epoch[36] Time cost=4.815
    INFO:root:Epoch[37] Train-loss=180.828095
    INFO:root:Epoch[37] Time cost=4.455
    INFO:root:Epoch[38] Train-loss=180.495569
    INFO:root:Epoch[38] Time cost=5.096
    INFO:root:Epoch[39] Train-loss=180.389106
    INFO:root:Epoch[39] Time cost=4.797
    INFO:root:Epoch[40] Train-loss=180.200965
    INFO:root:Epoch[40] Time cost=5.054
    INFO:root:Epoch[41] Train-loss=179.851014
    INFO:root:Epoch[41] Time cost=4.642
    INFO:root:Epoch[42] Train-loss=179.719933
    INFO:root:Epoch[42] Time cost=4.603
    INFO:root:Epoch[43] Train-loss=179.431740
    INFO:root:Epoch[43] Time cost=4.341
    INFO:root:Epoch[44] Train-loss=179.235384
    INFO:root:Epoch[44] Time cost=4.638
    INFO:root:Epoch[45] Train-loss=179.108771
    INFO:root:Epoch[45] Time cost=4.754
    INFO:root:Epoch[46] Train-loss=178.714163
    INFO:root:Epoch[46] Time cost=4.457
    INFO:root:Epoch[47] Train-loss=178.508338
    INFO:root:Epoch[47] Time cost=4.960
    INFO:root:Epoch[48] Train-loss=178.288002
    INFO:root:Epoch[48] Time cost=4.562
    INFO:root:Epoch[49] Train-loss=178.083288
    INFO:root:Epoch[49] Time cost=4.619
    INFO:root:Epoch[50] Train-loss=177.791330
    INFO:root:Epoch[50] Time cost=4.580
    INFO:root:Epoch[51] Train-loss=177.570741
    INFO:root:Epoch[51] Time cost=4.704
    INFO:root:Epoch[52] Train-loss=177.287114
    INFO:root:Epoch[52] Time cost=5.172
    INFO:root:Epoch[53] Train-loss=177.122645
    INFO:root:Epoch[53] Time cost=4.678
    INFO:root:Epoch[54] Train-loss=176.816022
    INFO:root:Epoch[54] Time cost=4.819
    INFO:root:Epoch[55] Train-loss=176.670484
    INFO:root:Epoch[55] Time cost=4.568
    INFO:root:Epoch[56] Train-loss=176.459671
    INFO:root:Epoch[56] Time cost=4.450
    INFO:root:Epoch[57] Train-loss=176.174175
    INFO:root:Epoch[57] Time cost=4.579
    INFO:root:Epoch[58] Train-loss=175.935856
    INFO:root:Epoch[58] Time cost=4.552
    INFO:root:Epoch[59] Train-loss=175.739928
    INFO:root:Epoch[59] Time cost=4.385
    INFO:root:Epoch[60] Train-loss=175.579695
    INFO:root:Epoch[60] Time cost=4.496
    INFO:root:Epoch[61] Train-loss=175.403871
    INFO:root:Epoch[61] Time cost=5.088
    INFO:root:Epoch[62] Train-loss=175.157114
    INFO:root:Epoch[62] Time cost=4.628
    INFO:root:Epoch[63] Train-loss=174.953950
    INFO:root:Epoch[63] Time cost=4.826
    INFO:root:Epoch[64] Train-loss=174.743393
    INFO:root:Epoch[64] Time cost=4.832
    INFO:root:Epoch[65] Train-loss=174.554056
    INFO:root:Epoch[65] Time cost=4.375
    INFO:root:Epoch[66] Train-loss=174.366719
    INFO:root:Epoch[66] Time cost=4.583
    INFO:root:Epoch[67] Train-loss=174.160622
    INFO:root:Epoch[67] Time cost=4.586
    INFO:root:Epoch[68] Train-loss=173.981699
    INFO:root:Epoch[68] Time cost=5.149
    INFO:root:Epoch[69] Train-loss=173.751617
    INFO:root:Epoch[69] Time cost=4.495
    INFO:root:Epoch[70] Train-loss=173.548732
    INFO:root:Epoch[70] Time cost=4.588
    INFO:root:Epoch[71] Train-loss=173.380950
    INFO:root:Epoch[71] Time cost=5.042
    INFO:root:Epoch[72] Train-loss=173.158519
    INFO:root:Epoch[72] Time cost=4.817
    INFO:root:Epoch[73] Train-loss=172.970726
    INFO:root:Epoch[73] Time cost=4.791
    INFO:root:Epoch[74] Train-loss=172.782357
    INFO:root:Epoch[74] Time cost=4.377
    INFO:root:Epoch[75] Train-loss=172.581992
    INFO:root:Epoch[75] Time cost=4.518
    INFO:root:Epoch[76] Train-loss=172.385020
    INFO:root:Epoch[76] Time cost=4.863
    INFO:root:Epoch[77] Train-loss=172.198309
    INFO:root:Epoch[77] Time cost=5.104
    INFO:root:Epoch[78] Train-loss=172.022333
    INFO:root:Epoch[78] Time cost=4.571
    INFO:root:Epoch[79] Train-loss=171.816585
    INFO:root:Epoch[79] Time cost=4.557
    INFO:root:Epoch[80] Train-loss=171.643714
    INFO:root:Epoch[80] Time cost=4.567
    INFO:root:Epoch[81] Train-loss=171.460581
    INFO:root:Epoch[81] Time cost=4.735
    INFO:root:Epoch[82] Train-loss=171.284854
    INFO:root:Epoch[82] Time cost=5.012
    INFO:root:Epoch[83] Train-loss=171.113129
    INFO:root:Epoch[83] Time cost=4.877
    INFO:root:Epoch[84] Train-loss=170.947790
    INFO:root:Epoch[84] Time cost=4.487
    INFO:root:Epoch[85] Train-loss=170.766223
    INFO:root:Epoch[85] Time cost=4.723
    INFO:root:Epoch[86] Train-loss=170.602559
    INFO:root:Epoch[86] Time cost=4.803
    INFO:root:Epoch[87] Train-loss=170.448713
    INFO:root:Epoch[87] Time cost=4.636
    INFO:root:Epoch[88] Train-loss=170.273053
    INFO:root:Epoch[88] Time cost=4.562
    INFO:root:Epoch[89] Train-loss=170.099485
    INFO:root:Epoch[89] Time cost=4.567
    INFO:root:Epoch[90] Train-loss=169.934289
    INFO:root:Epoch[90] Time cost=4.905
    INFO:root:Epoch[91] Train-loss=169.768920
    INFO:root:Epoch[91] Time cost=4.636
    INFO:root:Epoch[92] Train-loss=169.620803
    INFO:root:Epoch[92] Time cost=4.429
    INFO:root:Epoch[93] Train-loss=169.448189
    INFO:root:Epoch[93] Time cost=4.985
    INFO:root:Epoch[94] Train-loss=169.295794
    INFO:root:Epoch[94] Time cost=4.649
    INFO:root:Epoch[95] Train-loss=169.143627
    INFO:root:Epoch[95] Time cost=4.602
    INFO:root:Epoch[96] Train-loss=168.989410
    INFO:root:Epoch[96] Time cost=4.904
    INFO:root:Epoch[97] Train-loss=168.841089
    INFO:root:Epoch[97] Time cost=4.602
    INFO:root:Epoch[98] Train-loss=168.694906
    INFO:root:Epoch[98] Time cost=4.589
    INFO:root:Epoch[99] Train-loss=168.527604
    INFO:root:Epoch[99] Time cost=4.560
    INFO:root:Epoch[100] Train-loss=168.385596
    INFO:root:Epoch[100] Time cost=4.835
    INFO:root:Epoch[101] Train-loss=168.246526
    INFO:root:Epoch[101] Time cost=4.558
    INFO:root:Epoch[102] Train-loss=168.093663
    INFO:root:Epoch[102] Time cost=4.609
    INFO:root:Epoch[103] Train-loss=167.938807
    INFO:root:Epoch[103] Time cost=4.599
    INFO:root:Epoch[104] Train-loss=167.814916
    INFO:root:Epoch[104] Time cost=4.394
    INFO:root:Epoch[105] Train-loss=167.676473
    INFO:root:Epoch[105] Time cost=4.724
    INFO:root:Epoch[106] Train-loss=167.560241
    INFO:root:Epoch[106] Time cost=4.316
    INFO:root:Epoch[107] Train-loss=167.424132
    INFO:root:Epoch[107] Time cost=4.646
    INFO:root:Epoch[108] Train-loss=167.284482
    INFO:root:Epoch[108] Time cost=4.472
    INFO:root:Epoch[109] Train-loss=167.184511
    INFO:root:Epoch[109] Time cost=4.768
    INFO:root:Epoch[110] Train-loss=167.037793
    INFO:root:Epoch[110] Time cost=4.717
    INFO:root:Epoch[111] Train-loss=166.916652
    INFO:root:Epoch[111] Time cost=4.803
    INFO:root:Epoch[112] Train-loss=166.796803
    INFO:root:Epoch[112] Time cost=4.617
    INFO:root:Epoch[113] Train-loss=166.655028
    INFO:root:Epoch[113] Time cost=4.420
    INFO:root:Epoch[114] Train-loss=166.561129
    INFO:root:Epoch[114] Time cost=4.333
    INFO:root:Epoch[115] Train-loss=166.434593
    INFO:root:Epoch[115] Time cost=4.526
    INFO:root:Epoch[116] Train-loss=166.322805
    INFO:root:Epoch[116] Time cost=4.310
    INFO:root:Epoch[117] Train-loss=166.195452
    INFO:root:Epoch[117] Time cost=4.458
    INFO:root:Epoch[118] Train-loss=166.073792
    INFO:root:Epoch[118] Time cost=4.333
    INFO:root:Epoch[119] Train-loss=165.967437
    INFO:root:Epoch[119] Time cost=4.459
    INFO:root:Epoch[120] Train-loss=165.876094
    INFO:root:Epoch[120] Time cost=5.070
    INFO:root:Epoch[121] Train-loss=165.748064
    INFO:root:Epoch[121] Time cost=4.782
    INFO:root:Epoch[122] Train-loss=165.656283
    INFO:root:Epoch[122] Time cost=4.640
    INFO:root:Epoch[123] Train-loss=165.540462
    INFO:root:Epoch[123] Time cost=4.522
    INFO:root:Epoch[124] Train-loss=165.448734
    INFO:root:Epoch[124] Time cost=4.858
    INFO:root:Epoch[125] Train-loss=165.347751
    INFO:root:Epoch[125] Time cost=4.842
    INFO:root:Epoch[126] Train-loss=165.230048
    INFO:root:Epoch[126] Time cost=4.495
    INFO:root:Epoch[127] Train-loss=165.147932
    INFO:root:Epoch[127] Time cost=4.766
    INFO:root:Epoch[128] Train-loss=165.036021
    INFO:root:Epoch[128] Time cost=4.526
    INFO:root:Epoch[129] Train-loss=164.977613
    INFO:root:Epoch[129] Time cost=5.091
    INFO:root:Epoch[130] Train-loss=164.881467
    INFO:root:Epoch[130] Time cost=5.223
    INFO:root:Epoch[131] Train-loss=164.785627
    INFO:root:Epoch[131] Time cost=4.165
    INFO:root:Epoch[132] Train-loss=164.707629
    INFO:root:Epoch[132] Time cost=4.527
    INFO:root:Epoch[133] Train-loss=164.598039
    INFO:root:Epoch[133] Time cost=4.167
    INFO:root:Epoch[134] Train-loss=164.502932
    INFO:root:Epoch[134] Time cost=4.354
    INFO:root:Epoch[135] Train-loss=164.422286
    INFO:root:Epoch[135] Time cost=4.387
    INFO:root:Epoch[136] Train-loss=164.344749
    INFO:root:Epoch[136] Time cost=4.662
    INFO:root:Epoch[137] Train-loss=164.264898
    INFO:root:Epoch[137] Time cost=4.671
    INFO:root:Epoch[138] Train-loss=164.178707
    INFO:root:Epoch[138] Time cost=4.776
    INFO:root:Epoch[139] Train-loss=164.109071
    INFO:root:Epoch[139] Time cost=4.787
    INFO:root:Epoch[140] Train-loss=163.993291
    INFO:root:Epoch[140] Time cost=4.726
    INFO:root:Epoch[141] Train-loss=163.956234
    INFO:root:Epoch[141] Time cost=4.337
    INFO:root:Epoch[142] Train-loss=163.845638
    INFO:root:Epoch[142] Time cost=4.787
    INFO:root:Epoch[143] Train-loss=163.790882
    INFO:root:Epoch[143] Time cost=5.563
    INFO:root:Epoch[144] Train-loss=163.723495
    INFO:root:Epoch[144] Time cost=4.529
    INFO:root:Epoch[145] Train-loss=163.634262
    INFO:root:Epoch[145] Time cost=5.028
    INFO:root:Epoch[146] Train-loss=163.552854
    INFO:root:Epoch[146] Time cost=4.933
    INFO:root:Epoch[147] Train-loss=163.501429
    INFO:root:Epoch[147] Time cost=4.912
    INFO:root:Epoch[148] Train-loss=163.444245
    INFO:root:Epoch[148] Time cost=5.034
    INFO:root:Epoch[149] Train-loss=163.348476
    INFO:root:Epoch[149] Time cost=4.600
    INFO:root:Epoch[150] Train-loss=163.256955
    INFO:root:Epoch[150] Time cost=4.704
    INFO:root:Epoch[151] Train-loss=163.216139
    INFO:root:Epoch[151] Time cost=4.670
    INFO:root:Epoch[152] Train-loss=163.144691
    INFO:root:Epoch[152] Time cost=4.678
    INFO:root:Epoch[153] Train-loss=163.050236
    INFO:root:Epoch[153] Time cost=4.595
    INFO:root:Epoch[154] Train-loss=162.991225
    INFO:root:Epoch[154] Time cost=5.307
    INFO:root:Epoch[155] Train-loss=162.907200
    INFO:root:Epoch[155] Time cost=4.684
    INFO:root:Epoch[156] Train-loss=162.838075
    INFO:root:Epoch[156] Time cost=4.686
    INFO:root:Epoch[157] Train-loss=162.759286
    INFO:root:Epoch[157] Time cost=4.750
    INFO:root:Epoch[158] Train-loss=162.725998
    INFO:root:Epoch[158] Time cost=4.637
    INFO:root:Epoch[159] Train-loss=162.635852
    INFO:root:Epoch[159] Time cost=4.498
    INFO:root:Epoch[160] Train-loss=162.563777
    INFO:root:Epoch[160] Time cost=5.048
    INFO:root:Epoch[161] Train-loss=162.527387
    INFO:root:Epoch[161] Time cost=5.040
    INFO:root:Epoch[162] Train-loss=162.395881
    INFO:root:Epoch[162] Time cost=4.764
    INFO:root:Epoch[163] Train-loss=162.353654
    INFO:root:Epoch[163] Time cost=4.561
    INFO:root:Epoch[164] Train-loss=162.285584
    INFO:root:Epoch[164] Time cost=5.051
    INFO:root:Epoch[165] Train-loss=162.204332
    INFO:root:Epoch[165] Time cost=4.455
    INFO:root:Epoch[166] Train-loss=162.147100
    INFO:root:Epoch[166] Time cost=5.021
    INFO:root:Epoch[167] Train-loss=162.051296
    INFO:root:Epoch[167] Time cost=4.551
    INFO:root:Epoch[168] Train-loss=161.978708
    INFO:root:Epoch[168] Time cost=4.744
    INFO:root:Epoch[169] Train-loss=161.927990
    INFO:root:Epoch[169] Time cost=4.821
    INFO:root:Epoch[170] Train-loss=161.883088
    INFO:root:Epoch[170] Time cost=4.365
    INFO:root:Epoch[171] Train-loss=161.785367
    INFO:root:Epoch[171] Time cost=4.448
    INFO:root:Epoch[172] Train-loss=161.716386
    INFO:root:Epoch[172] Time cost=4.622
    INFO:root:Epoch[173] Train-loss=161.656391
    INFO:root:Epoch[173] Time cost=4.500
    INFO:root:Epoch[174] Train-loss=161.598127
    INFO:root:Epoch[174] Time cost=4.677
    INFO:root:Epoch[175] Train-loss=161.518613
    INFO:root:Epoch[175] Time cost=4.958
    INFO:root:Epoch[176] Train-loss=161.418783
    INFO:root:Epoch[176] Time cost=4.607
    INFO:root:Epoch[177] Train-loss=161.407767
    INFO:root:Epoch[177] Time cost=4.427
    INFO:root:Epoch[178] Train-loss=161.319552
    INFO:root:Epoch[178] Time cost=4.930
    INFO:root:Epoch[179] Train-loss=161.234087
    INFO:root:Epoch[179] Time cost=4.240
    INFO:root:Epoch[180] Train-loss=161.187404
    INFO:root:Epoch[180] Time cost=4.484
    INFO:root:Epoch[181] Train-loss=161.123118
    INFO:root:Epoch[181] Time cost=4.937
    INFO:root:Epoch[182] Train-loss=160.999420
    INFO:root:Epoch[182] Time cost=4.489
    INFO:root:Epoch[183] Train-loss=160.955369
    INFO:root:Epoch[183] Time cost=4.894
    INFO:root:Epoch[184] Train-loss=160.908542
    INFO:root:Epoch[184] Time cost=4.269
    INFO:root:Epoch[185] Train-loss=160.846908
    INFO:root:Epoch[185] Time cost=4.998
    INFO:root:Epoch[186] Train-loss=160.765964
    INFO:root:Epoch[186] Time cost=4.467
    INFO:root:Epoch[187] Train-loss=160.687773
    INFO:root:Epoch[187] Time cost=4.609
    INFO:root:Epoch[188] Train-loss=160.652674
    INFO:root:Epoch[188] Time cost=5.327
    INFO:root:Epoch[189] Train-loss=160.551175
    INFO:root:Epoch[189] Time cost=4.267
    INFO:root:Epoch[190] Train-loss=160.477424
    INFO:root:Epoch[190] Time cost=4.798
    INFO:root:Epoch[191] Train-loss=160.501221
    INFO:root:Epoch[191] Time cost=4.695
    INFO:root:Epoch[192] Train-loss=160.370335
    INFO:root:Epoch[192] Time cost=4.640
    INFO:root:Epoch[193] Train-loss=160.279749
    INFO:root:Epoch[193] Time cost=4.653
    INFO:root:Epoch[194] Train-loss=160.242415
    INFO:root:Epoch[194] Time cost=5.044
    INFO:root:Epoch[195] Train-loss=160.197063
    INFO:root:Epoch[195] Time cost=4.684
    INFO:root:Epoch[196] Train-loss=160.132983
    INFO:root:Epoch[196] Time cost=4.460
    INFO:root:Epoch[197] Train-loss=160.083149
    INFO:root:Epoch[197] Time cost=4.713
    INFO:root:Epoch[198] Train-loss=160.025012
    INFO:root:Epoch[198] Time cost=4.779
    INFO:root:Epoch[199] Train-loss=159.945513
    INFO:root:Epoch[199] Time cost=4.659

```python
# encode test images to obtain mu and logvar which are used for sampling
[mu,logvar] = VAE.encoder(out,image_test)
# sample in the latent space
z = VAE.sampler(mu,logvar)
# decode from the latent space to obtain reconstructed images
x_construction = VAE.decoder(out,z)

```

```python
f, ((ax1, ax2, ax3, ax4)) = plt.subplots(1,4,  sharex='col', sharey='row',figsize=(12,3))
ax1.imshow(np.reshape(image_test[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax1.set_title('True image')
ax2.imshow(np.reshape(x_construction[0,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax2.set_title('Learned image')
ax3.imshow(np.reshape(image_test[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax3.set_title('True image')
ax4.imshow(np.reshape(x_construction[146,:],(28,28)), interpolation='nearest', cmap=cm.Greys)
ax4.set_title('Learned image')
plt.show()
```

```python
z1 = z[:,0]
z2 = z[:,1]

fig = plt.figure()
ax = fig.add_subplot(111)
ax.plot(z1,z2,'ko')
plt.title("latent space")

#np.where((z1>3) & (z2<2) & (z2>0))
#select the points from the latent space
a_vec = [2,5,7,789,25,9993]
for i in range(len(a_vec)):
    ax.plot(z1[a_vec[i]],z2[a_vec[i]],'ro')
    ax.annotate('z%d' %i, xy=(z1[a_vec[i]],z2[a_vec[i]]),
                xytext=(z1[a_vec[i]],z2[a_vec[i]]),color = 'r',fontsize=15)

f, ((ax0, ax1, ax2, ax3, ax4,ax5)) = plt.subplots(1,6,  sharex='col', sharey='row',figsize=(12,2.5))
for i in range(len(a_vec)):
    eval('ax%d' %(i)).imshow(np.reshape(x_construction[a_vec[i],:],(28,28)), interpolation='nearest', cmap=cm.Greys)
    eval('ax%d' %(i)).set_title('z%d'%i)

plt.show()
```

Above is a plot of points in the 2D latent space and their corresponding decoded images, it can be seen that points that are close in the latent space get mapped to the same digit from the decoder, and we can see how it evolves from left to right.


# Python for Engineers Assignment 2018
Refer to file Assignment.ipynb

Submission deadline 07/16 (July 16).

提出締め切り07/16 (7月16日）。

Please submit your answer by email.

どうぞ、メールによってあなたの答えを提出してください。

philamore.hemma.5s@kyoto-u.ac.jp

You may use the following file formats:

あなたは以下のファイル形式を使ってもよい:
- .py
- .ipynb
- .txt

You may use multiple files.

あなたは複数のファイルを使ってもよい。

### Rescheduled Class
You should all have received an email concerning the 2 rescheduled class.

Rescheduled classes will be held on:
- 06/26 (June 26, Tuesday ), period 1, room CALL 23
- 06/29 (June 29, Friday ), period 5, room CALL 23

You may attend either class.

These classes are for you to ask questions and get help with your assignment.

There will be *no new taught material* in these classes.

__THERE IS A CLASS ON JUNE 25, MONDAY, PERIOD 5 AS USUAL__

## Please download the new class notes.
### Step 1 : Navigate to the directory where your files are stored.
Open a terminal.

Using `cd`, navigate to *inside* the ILAS_Python_for_engineers folder on your computer.
### Step 3 : Update the course notes by downloading the changes
In the terminal type:

>`git add -A
git commit -m "commit"
git fetch upstream
git merge -X theirs upstream/master`

# Solving Differential and Integration Equations with Programming and Symbolic Mathematics

 Systems of Coupled Equations

    Linear Equations

       Example: An Electrical Circuit

    Linear Equations

 Root Finding

    Iterative Solutions

         Bisection

             Example: Bisection Algorithm

   Importing Specific Algorithms

    Finding the nth Root of a Periodic Function

 Coupled Non-Linear Equations (continued)

 Summary

 Test-Yourself Exercises

### Lesson Goal

Use analytical and numerical techniuqes to:
- solve differentiation and integration problems
- find solutions to ordinary differential equations
- represnt and solve equations symbolically

### Fundamental programming concepts
Solving problems programmatically to save time and to check your solutions.

Many of the practical situations that engineers have to analyse involve quantities that change with respect to time or another parameter.

Examples:
 - Temperature of a substance/system
 - Signal voltage
 - Torque on a turbine blade

The calculus techniques that we use to solve these problems are
- differentiation
- integration

You will probably be familiar with all of the techniques we will study today.

The main challenge is knowing how to represent processes that you are familiar with solving manually, using the Python syntax.

There are multiple ways to solve differentiation and integration problems programmatically.

Today we will study several of these methods using:
- Numpy
- Scipy
- Sympy : A Python library for __symbolic__ mathematics.

```python
import numpy as np
import scipy
import sympy

import matplotlib.pyplot as plt
%matplotlib inline
```

In the past two seminars we considered functions that can be represented in a 2D coordinate system by considering the relation ship between:
- input (independent variable)
- output (dependent variable)

For example, a function, $f(t)$

The definition of the function $f'(t)$ at a point $t$:

$$ f' = \lim_{x \to 0} \frac{f(t + \Delta t)-f(t)}{\Delta t} = \lim_{x \to 0} \frac{\Delta f}{\Delta t}$$

where $\Delta f$ is the rate of change of $f(t)$ corresponding to a change $\Delta t$ in $t$

This from this definition, we can *approximate* the rate of change of function $f(t)$ with respect to parameter $t$, from the difference in $f$ between two different values of $t$.

*Image source: James, Modern Engineering Mathematics*

Forward difference:

$$ f' \approx \frac{f(t + h)-f(t)}{h} $$

Backward difference:

$$ f' \approx \frac{f(t)-f(t - h)}{h} $$

Central difference:

$$ f' \approx \frac{f(t + h)-f(t - h)}{2h} $$

## Numerical Derivatives by Difference Methods
If your data points are equally space, these equations are very easy to solve programmatically for a data set.

```python
def forward_diff(x, y):
    #all_but_last = (y[1:] - y[:-1]) / (x[1:] - x[:-1])
    all_but_last = np.diff(y) / np.diff(x)
    # solve last element by backward difference
    last = (y[-1] - y[-2]) / (x[-1] - x[-2])
    return(np.append(all_but_last, last))

def backward_diff(x, y):
    #all_but_first = (y[1:] - y[:-1]) / (x[1:] - x[:-1])
    all_but_first = np.diff(y) / np.diff(x)
    # solve fisrst element by forward difference
    first = (y[1] - y[0]) / (x[1] - x[0])
    return(np.insert(all_but_first, 0, first))

def central_diff(x, y):
    centre = (y[2:] - y[:-2]) / (x[2:] - x[:-2])
    # solve first and last element
    first = (y[1] - y[0]) / (x[1] - x[0])
    last = (y[-1] - y[-2]) / (x[-1] - x[-2])

    return(np.insert(np.append(centre, last), 0, first))
```

This gives a very simple method for solving numerical derivatives.

Let's test these functions.

Analytical solution:

$$f(t)=sin(t), \qquad f'(t)=cos(t)$$

To show the accuracy of the solution, we first consider a very small t interval.

```python
t = np.linspace(0.78,0.79,100)
f = np.sin(t)

dfdt_analytical = np.cos(t)

dfdt_fwd = forward_diff(t, f)
dfdt_bck = backward_diff(t, f)
dfdt_ctr = central_diff(t, f)

plt.plot(t, dfdt_analytical, label='forward difference approximation')
plt.plot(t, dfdt_analytical, label='backward difference approximation')
plt.plot(t, dfdt_analytical, label='central difference approximation')
plt.plot(t, dfdt_analytical, 'k--', label='analytical derivative')
plt.legend();
```

The solution for a greater t interval:

```python
t = np.linspace(0, 2 * np.pi, 100)
f = np.sin(t)

dfdt_analytical = np.cos(t)

dfdt_fwd = forward_diff(t, f)
dfdt_bck = backward_diff(t, f)
dfdt_ctr = central_diff(t, f)

plt.plot(t, dfdt_analytical, label='forward difference approximation')
plt.plot(t, dfdt_analytical, label='backward difference approximation')
plt.plot(t, dfdt_analytical, label='central difference approximation')
plt.plot(t, dfdt_analytical, 'k--', label='analytical derivative')
plt.plot(t, f, label='function')
plt.legend(loc='best');
```

In computing terms, calculation that use loops are slow.

However, for practical purposes, this is a fast way to approximate the numerical derivative for *smooth* function data.

## Numerical Derivatives by Difference Methods

Some of the functions that we considered in the previous classes we generated by fitting a polynomial curve to empirical (and sometimes noisy) data.

To find the derivative of a polynomial function, we can use the numpy function, `polyder`

Example:
    $$p(x) = x^3 + x^2 + x^1 + 1$$

Find $p'(x)$

```python
p = np.poly1d([1,1,1,1])
pd = np.polyder(p)
print(pd)
```

       2
    3 x + 2 x + 1

Find $p'(2)$

```python
pd(2.)
```

    17.0

We can check the solution using the forward/backward/central difference approximation.

Forward difference approximation:

$$ f' \approx \frac{f(x + \Delta x)-f(x)}{\Delta x} $$

let $\Delta x = 0.001$

```python
pd2 = (p(2. + 0.001) - p(2.)) / 0.001
print(pd2)
```

    17.007000999997857

`polyder` can be be used to solve higher order derivatives by giving a second argument:

```python
# second derivative of p
print(np.polyder(p, 2))

# third derivative of p
print(np.polyder(p, 3))

# fourth derivative of p
print(np.polyder(p, 4))
```

    6 x + 2

    6

    0

The fourth derivative of a 3rd-order polynomial is zero.

## Piecewise Functions
Occaisionally we need to define piecewise functions e.g.:

$$f(x)=
\begin{cases}
      0   & ,x < 0 \\
      x   & ,0\leq x \leq 1 \\
      2-x & ,1 < x \leq 2 \\
      x+1 & ,x > 2
\end{cases}
$$

A simple way to solve this is to use conditional statements.

However, the function is not vectorised (we cannot apply it directly to a data structure).

```python
def f1(x):
    if x < 0:
        return 0
    elif 0 <= x <= 1:
        return x
    elif 1 < x <= 2:
        return 2.0 - x
    else:
         return x + 1.0

print(f1(-1))

# Application to data structure causes an error
#print(f1([0, 1, 2, 3, 4]))
```

    0

You can achieve vectorized behavior (apply the function to each element) by looping through each value in the data structure.

This is not ideal e.g. we cannot directly plot or interrogate the function.

Ideally we want a function where both of the following statements are allowable:

        print(f1([0, 1, 2, 3, 4]))
        print(f1(0))

In seminar 04_AdvancedFunctions_Libraries.ipynb we learnt to use the numpy function `vectorise` to allow a function to accept both:
- single variables
- data structures

as input arguments.

```python
f2 = np.vectorize(f1)
print(f2([0, 1, 2, 3, 4]))
print(f2(0))
```

    [0 1 0 4 5]
    0

Dealing with piecewise functions is an example of where vectorising a function is very useful.

The main advantages of vectorization are:
- simplicity of code
- speed

Loops in python are usually very slow compared to vectorized functions.

## Integrating Equations
A common need in engineering calculations is to __integrate__ an equation over some range to determine the total change.

For example, find the integral of a function f(x) from a to b i.e.

$$S = \int_{a}^{b} f(x) dx$$

This area or interal can be estimated from the sum of trapezoid strips used to represent the curve.

As a specific example, lets find the integrate:
$$f(x) = x^2$$

with respect to $x$,

within the limits $x=0$ to $x=1$.

\begin{align*}
S &= \int_{0}^{1} x^2 dx\\
  &= \frac{1}{3}x^3 \biggr\rvert_0^{1}\\
  &= \frac{1}{3}(1)^3 - \frac{1}{3}(0)^3\\
  &= \frac{1}{3}
\end{align*}

The scipy function `quad` is very convenient for solving integration problems:

Input arguments :
- function to integrate
- lower limit
- upper limit

```python
from scipy.integrate import quad
```

```python
# first create a function to describe the integrand
def integrand(x):
    return x**2

ans, err = quad(integrand, 0, 1)

print(ans)
print(f'The estimated error is {err}')
```

    0.33333333333333337
    The estimated error is 3.700743415417189e-15

#### Example :  The quantity of water that flows into a tank.

The change in volumetric flow rate, $v'$, into a tank (with respect to time):

$$ v' =\frac{dv}{dt}=\alpha t $$

where $\alpha = 1\textrm{L/s}$

__How much liquid flows into the tank over 10 seconds?__

Initial flow rate:

$v_0=5\textrm{L/s,} \qquad t=0$

__Solution__

We can see that at

\begin{align*}
v_1 &= v_0 + \frac{dv}{dt}\\
    &= v_0 + \alpha (1) \\
\end{align*}

i.e.

$$v_n = v_0 + \alpha n$$

To find the total __volume__ of liquid the enters the tank in $n$ seconds:

$$V = \int_{0}^{n} v_0 + \alpha t \quad dt$$

```python
from scipy.integrate import quad

t0=0.0
tfinal = 10.0
v0=5       # L/s
alpha = 1  # L/s

# create a function to describe the integrand
def integrand(t):
    return v0 + alpha * t

V, err = quad(integrand, t0, tfinal)

print(f'{V} L flowed into the tank in 10 seconds')
print(f'The estimated error is {err}')
```

    100.0 L flowed into the tank in 10 seconds
    The estimated error is 1.1102230246251565e-12

We can check the solution:

\begin{align*}
V &= \int_{0}^{10} v_0 + \alpha t \quad dt\\
  &= v_0(t) + \frac{1}{2}\alpha t^2 \biggr\rvert_0^{10}\\
  &=100
\end{align*}

### Nested Integrals
Sometimes engineering problems require us to integrate an expression multiple times to find the desired solution.

In that case, we can instead use the function:
- `scipy.integrate.dblquad` : double integral
- `scipy.integrate.tplquad` : triple integral
- `scipy.integrate.nquad`   : any number of intergals

```python
from scipy.integrate import dblquad
from scipy.integrate import tplquad
from scipy.integrate import nquad
```

__Example:__ Area of a rectangle.

A very simple application of a nested integral is to find the area, $A$, of a rectangle using the upper and lower x and y limits that define the position of the retangles vertices.

$$ A = \int_{y=c}^{d}\int_{x=a}^{b} 1\quad dxdy \quad = \quad \int_{y=c}^{d}\int_{x=a}^{b} \quad dxdy$$

```python

# Some arbitrary values
a, b, c, d = 1, 10, 0, 0.5

f = lambda x, y: 0.5

# Arguments:
# function, outer integral limits --> inner integral limits
area = dblquad(f, c, d,     # y limits
               lambda x: a, # x limits
               lambda x: b)
print(area)
```

    (2.25, 4.9960036108132044e-14)

```python
# Arguments:
# function, inner integral limits --> outer integral limits
area = nquad(f, [[a, b],   # x limits
                 [c, d]])  # y limits
print(area)
```

    (4.5, 9.992007221626409e-14)

##### Triple Integral
$$ A = \int_{z=e}^{f} \int_{y=c}^{d}\int_{x=a}^{b} kxyz\quad dxdy$$

```python
# Some arbitrary values
#a, b, c, d, e, f = 0, 1, 2, 3, 1, 2

f = lambda z, y, x, k: k*x*y*z

# Arguments:
# function, outer integral limits --> inner integral limits
a = tplquad(f, 1, 2,                          # z limits
            lambda x: 2,    lambda x: 3,      # y limits
            lambda x, y: 0, lambda x, y: 1,   # x limits
            args=(3,))
print(a)

```

    (5.625, 9.97393438277222e-14)

We can represent the same problem using `nquad`.

We can nest any number of (`n`) integrals.

Notice how the syntax of this function is different from `dblquad` and `tplquad`.

```python
# Arguments:
# function, inner integral limits --> outer integral limits

a = nquad(f, [[0, 1],    # x limits
              [2, 3],    # y limits
              [1, 2]],   # z limits
              args=(3,)) # additional arguments of function f

print(a)
```

    (5.625, 9.97393438277222e-14)

## Symbolic Mathematics
Sometimes we want to express the solution in the form of a symbolic expression (rather than an exact solution).

The `Sympy` package is designed for symbolic mathematics.

```python
from sympy import solve, symbols, Symbol, Function, Eq, pprint
```

### Symbolic Polynomials
To write a symbolic equation, we first create variables to store the symbols we want to use:

```python
a, b, c, x = symbols('a, b, c, x')
```

We then write an equation, using the variable names.

$$f(x) = ax^2 + bx + c$$

```python
f = a*x**2 + b*x + c
```

Let's *symbolically* find the roots of the function $f(x)$.

`sympy.solve()` works in the same way as `numpy.linalg.solve()` (07_RootFinding_CoupledEquations.ipynb) but for symbolic mathematics.

```python
solution = solve(f, x)

print(solution)
```

    [(-b + sqrt(-4*a*c + b**2))/(2*a), -(b + sqrt(-4*a*c + b**2))/(2*a)]

This equation takes the familar form of the __quadratic formula__ $$\frac{-b \pm \sqrt{b^2-4ac}}{2a}$$

However, it is difficult to tell from the way it is written.

`sympy.pprint` (pretty print) makes the solution more readable:

```python
from sympy import pprint

pprint(solution)
print(solution)
```

    ⎡        _____________   ⎛       _____________⎞ ⎤
    ⎢       ╱           2    ⎜      ╱           2 ⎟ ⎥
    ⎢-b + ╲╱  -4⋅a⋅c + b    -⎝b + ╲╱  -4⋅a⋅c + b  ⎠ ⎥
    ⎢─────────────────────, ────────────────────────⎥
    ⎣         2⋅a                     2⋅a           ⎦
    [(-b + sqrt(-4*a*c + b**2))/(2*a), -(b + sqrt(-4*a*c + b**2))/(2*a)]

We can use `subs` to substitute in numerical values for the symbolic variables in the solution.

In this case, there are two solutions (returned as a list) so we address each index of the list by index:

```python
A, B, C = -1, 2, 3

x0 = solution[0].subs([(a, A),
                       (b, B),
                       (c, C)])

x1 = solution[1].subs([(a, A),
                       (b, B),
                       (c, C)])

display(x0, x1)

```

    -1

    3

### Symbolic Differentiation
`sympy.diff`

```python
from sympy import diff

```

Using the same functionas the previous example:
    $$f(x) = ax^2 + bx + c$$

```python
f = a*x**2 + b*x + c

```

$f'(x)$
(first derivative w.r.t $x$)

```python
print(diff(f, x))

```

    2*a*x + b

$f''(x)$
(second derivative w.r.t $x$)

```python
print(diff(f, x, 2))

```

    2*a

$f'(a)$
(first derivative w.r.t $a$)

```python
print(diff(f, a))

```

    x**2

We can substitute numerical values into the solution:

Find $f'(a), \quad x=2$

```python
diff(f, a).subs([(x, 2)])

```

    4

### Symbolic Integration
`sympy.integrate`

```python
from sympy import integrate

```

Using the same functionas the previous example:
    $$f(x) = ax^2 + bx + c$$

```python
f = a*x**2 + b*x + c

```

Find the general solution to
$ \int f(x)  dx$

```python
print(integrate(f, x))

```

    a*x**3/3 + b*x**2/2 + c*x

Find the definite integral
$\int_{x=0}^{1} f(x)dx$

```python
print(integrate(f, (x, 0, 1))) # definite integral from x=0..1

```

    a/3 + b/2 + c

Find the integral shown where constants $a,b,c$ gievn as numerical values

$\int_{x=0}^{1} f(x)dx$

$a=b=c=2 $

### Differentiating a Symbolic Function
We can also symbolically represent:
- a function.
- a *single* symbol using `Symbol()`.

*Notice how this differs from `symbols` for creating multiple symbols.*

```python
from sympy import Function, Symbol
f = Function('f')
x = Symbol('x')

```

The syntax for differentiation of a *symbolic function* is also different from the syntax for a function comprised of *symbolic variables*.

```python
fd = f(x).diff(x)

pprint(fd)

```

    d
    ──(f(x))
    dx

## Ordinary Differential Equations
__Ordinary differential equation__

A differential equation containing one or more functions of one independent variable and its derivatives.

Sometimes you want to solve a problem of the form:

*Given an equation containing the derivatives of the function $f(x)$, find $f(x)$.*

### Symbolic Analytical Solution to ODE
The function `sympy.dsolve` can be used to find the solution to any ordinary differential equation.

```python
from sympy import dsolve, Eq, sin, cos

```

Syntax to represent $f'(x)$

        f(x).diff(x)

Syntax to represent $f''(x)$

        f(x).diff(x, x)

In the next examples we will:
- solve ODE symbolically to find the general solution, $f(x)$
- substitute an initial condition to find constants of the general solution
- find the value of the function for a specific input variable.

#### Solving a First Order ODE

Let's start with an example we used earlier:

$$f'(x) = cos(x), \quad f(x) = sin(x)$$

We create a variable to represent:
- the function $f(x)$
- the input variable $x$

```python
f = Function('f')
x = Symbol('x')

```

We create a Sympy `Eq` equation.

The first argument (left hand side of equation) is equal to the second argument (right hand side of equation).

```python
ode = Eq(f(x).diff(x), cos(x))

```

We find the general solution, with unknown constant, `C1`:

```python
# General solution
gen_sol = dsolve(ode, f(x))

pprint(gen_sol)

```

    f(x) = C₁ + sin(x)

The package `display` can also be used to achieve more readable print formatting for mathematical equations:

```python
from IPython.display import display

```

```python
display(gen_sol)

```

    Eq(f(x), C1 + sin(x))

Use an initial value, $x=0, \quad f(x)=0$ to find `C1`.

`subs` takes a list og the substitutions we want to make as arguments.

```python
# Method 1
cnd = gen_sol.subs([(x,0),       # x = 0
                    (f(0), 0)])  # f(0) = 0

# which can also be written:
# Method 2
# cnd = Eq(gen_sol.rhs.subs(x, 0), 0) # x = 0, f(0) = 0

display(cnd)

```

    Eq(0, C1)

Substitute `C1` into the general solution, noting if `C1` appears on left or right of equation

```python
# Method 1, C1 rhs
ode_sol = gen_sol.subs([(cnd.rhs, cnd.lhs)])

# Method 2, C1 lhs
# ode_sol = gen_sol.subs([(cnd.lhs, cnd.rhs)])

display (ode_sol)

```

    Eq(f(x), sin(x))

##### What does this code do?

>`ode_sol = A_sol.subs([(cnd.rhs, cnd.lhs)])`

`cnd` is a Sympy __`Eq`__ (equation) object.

The first argument (left hand side of equation) is equal to the second argument (right hand side of equation).

```python
print(cnd)

```

    Eq(0, C1)

This means we can conveniently access the left and right hand using `lhs` and `rhs`.

The expression on the right hand side of `const` is replaced by the expression on the left hand side of `const` wherever it appears in the general solution.

We can use `subs` again to find the value of $f(x)$ at any input value $x$.

```python
spec_sol = ode_sol.subs([(x, np.pi/2)])
display(spec_sol)

```

    Eq(f(1.5707963267949), 1.00000000000000)

Lastly, we convert the symbolic representation to a numerical value using the method `evalf`.

```python
display(spec_sol.evalf())

```

    Eq(f(1.5707963267949), 1.00000000000000)

#### Solving a Second Order ODE
Taking another expression containing higher order derivatives,

we can write the equation:

$$A=f''(x) =12x^2$$

```python
f = Function('f')
x = Symbol('x')

A = Eq(f(x).diff(x, x), 12*x**2)

```

The highest order differential is a second order differential.

The general solution will have two constants to solve.

```python
# General solution
gen_sol = dsolve(A, f(x))
display(gen_sol)

```

    Eq(f(x), C1 + C2*x + x**4)

Find the constants by substitution.

$f(0)=0$

$f(1)=3$

$f'(1)=6$

```python
cnd0 = Eq(gen_sol.rhs.subs(x, 0), 0)           # f(0) = 0

cnd1 = Eq(gen_sol.rhs.subs(x, 1), 3)           # f(1) = 3

# cnd1 = Eq(gen_sol.rhs.diff(x).subs(x, 1), 6)  # f'(1) = 6

display(cnd0, cnd1)

```

    Eq(C1, 0)

    Eq(C1 + sin(1), 3)

Create symbolic representations of the variables.

This time we use `sympy.solve` to solve the coupled equations `cnd0` and `cnd1` for constants `C1, C2`:

```python
C1, C2 = symbols("C1, C2")  # generic constants

C1C2_sol = solve([cnd0, cnd1], (C1, C2))
display(C1C2_sol)

```

    {C1: 0, C2: 2}

Substitute the two constants into the general solution.

```python
ode2_sol = gen_sol.subs(C1C2_sol)
display(ode2_sol)

```

    Eq(f(x), sin(x))

## Numerial Solutions to  Ordinary Differential Equations

The function `scipy.integrate.odeint` estmates the solution to first order differential equations.

In contrast to `dsolve`, this gives the numerical solution for a range of input values, rather than a symbolic expression.

We will cover this function in greater depth in the next class.

```python
from scipy.integrate import odeint

```

The function `odeint` takes a user-defined function as an argument.

This input function should define the derivative you want to solve in the form:

$f' = \frac{df}{dt} = g(t)$

For example:

$f' = cos(t), \quad f(t=0)=0$

```python
def dfdt(x, t):
    return np.cos(t)
```

Two additional arguments are needed:

- the initial value of x
- the value(s) of t at which to evaluate x, starting with the initial value
- (if `dx_dt` takes any other argumemts they are entered as a tuple as the third argument)

```python
ts = np.linspace(0,5,100) # the value(s) of t at which to evaluate x
f0 = 0                    # the initial value

# odeint returns x at each value of t
fs = odeint(dfdt, # function
            f0,   # initial value
            ts)   # time span

```

The function `odeint` returns an $n\times 1$ array (2D column).

To plot the output we have to "flatten" it to a 1D array.

Recall, $f' = cos(t), \quad f = sin(t)$

```python
fs = np.array(fs).flatten()

```

Plot the function

```python
plt.xlabel("t")
plt.ylabel("f")
plt.plot(ts, np.sin(ts), 'c', label='analytical solution');
plt.plot(ts, fs, 'r--', label='numerical solution');
plt.legend(loc='best')

```

The numerical and analytical solutions agree.

Lastly we can add the sybolically derived function:

```python
sym_sol = []

for t in ts:
    sym_sol.append(ode_sol.subs([(x, t)]).evalf().rhs)

```

Plot the function

```python
plt.xlabel("t")
plt.ylabel("f")
plt.plot(ts, np.sin(ts), 'c', label='analytical solution');
plt.plot(ts, fs, 'r--', label='numerical solution');
plt.plot(ts, fs, 'b:', label='symbolic solution');
plt.legend(loc='best')

```

All solutions agree:

# Test-Yourself Exercises

Compete the Test-Youself exercises below.

Save your answers as .py files and email them to:

philamore.hemma.5s@kyoto-u.ac.jp

### Test Yourself-Exercise : Root Finding

Find the roots of the function.

i.e. find all $x$ where $f(x)=0$

$$f(x)= ax^3 - bx^2 - ax + b$$

Use the values $a=3, b=1$ to find the exact vaue of the roots.

### Test Yourself-Exercise : Car Breaking

During braking a car’s velocity is given by:

$$v = v_{0} e^{−t/\tau}$$

$v_{0}=15$ m/s

$\tau = 2$ s

Find the distance travelled during the 10s after the breaks are applied.

```python

```