# Fastai Course DL from the Foundations Batch Norm
> Implementing Batch Norm, Layer Norm, Instance Norm, Group Norm and running batch norm (Lesson 3 Part 4)

- toc: true
- badges: true
- comments: true
- categories: [jupyter]
- image: images/norms.png

```python
#collapse
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```python
#collapse
from exp.nb_06 import *
```

## ConvNet

Let's get the data and training interface from where we left in the last notebook.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=5899)

```python
#collapse
x_train,y_train,x_valid,y_valid = get_data()

x_train,x_valid = normalize_to(x_train,x_valid)
train_ds,valid_ds = Dataset(x_train, y_train),Dataset(x_valid, y_valid)

nh,bs = 50,512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```python
#collapse
mnist_view = view_tfm(1,28,28)
cbfs = [Recorder,
        partial(AvgStatsCallback,accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```python
#collapse
nfs = [8,16,32,64,64]
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
#collapse
%time run.fit(2, learn)
```

    train: [1.7780409375, tensor(0.3814, device='cuda:0')]
    valid: [0.457603125, tensor(0.8491, device='cuda:0')]
    train: [0.2377014453125, tensor(0.9279, device='cuda:0')]
    valid: [0.1322774658203125, tensor(0.9603, device='cuda:0')]
    CPU times: user 3.81 s, sys: 782 ms, total: 4.59 s
    Wall time: 5.29 s

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch. We should be able to improve performance a lot. While training we keep a running exponentially weighted mean and variance average in the update_stats function. During inference we only use running mean and variance that we keep track off. We use `register_buffer`to create vars and means, this still creates `self.vars` and `self.means`, but if the model is moved to the GPU so will all buffers. Also it will be saved along with everything else in the model.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6018)

```python
#collapse_show
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('vars',  torch.ones(1,nf,1,1))
        self.register_buffer('means', torch.zeros(1,nf,1,1))

    def update_stats(self, x):
        #we average over all batches (0) and over x,y(2,3) coordinates (each filter)
        #keepdim=True means we can still broadcast nicely as these dimensions will be left empty
        m = x.mean((0,2,3), keepdim=True)
        v = x.var ((0,2,3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_ (v, self.mom)
        return m,v

    def forward(self, x):
        if self.training:
            with torch.no_grad(): m,v = self.update_stats(x)
        else: m,v = self.means,self.vars
        x = (x-m) / (v+self.eps).sqrt()
        return x*self.mults + self.adds
```

### Exponential moving average
We use exp. moving average, that way we only need to keep track of one element.The next value is computed with linear interpolation. PyTorch mom=0.1 is actually 0.9 in math terms. (1-mom)

Now we define our batch norm conv_layer, when we use batch norm we can remove the bias layer as batch norm adds are a bias.

```python
#collapse_show
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
#collapse_show
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None: m.bias.data.zero_()
    for l in m.children(): init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

Above the modules are initalized recursively.
We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```python
#collapse
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

Train with Hooks :

```python
#collapse_show
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6));

    fig,(ax0,ax1) = plt.subplots(1,2, figsize=(10,4))
    for h in hooks[:-1]:
        ms,ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

```python
#collapse
%time run.fit(3, learn)
```

    train: [0.26908765625, tensor(0.9149, device='cuda:0')]
    valid: [0.151610400390625, tensor(0.9536, device='cuda:0')]
    train: [0.086416064453125, tensor(0.9732, device='cuda:0')]
    valid: [0.1458560791015625, tensor(0.9568, device='cuda:0')]
    train: [0.0614888330078125, tensor(0.9807, device='cuda:0')]
    valid: [0.08743798828125, tensor(0.9744, device='cuda:0')]
    CPU times: user 3.12 s, sys: 221 ms, total: 3.34 s
    Wall time: 3.28 s

### Builtin batchnorm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6679)

```python
#collapse_show
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```python
#collapse
%time run.fit(3, learn)
```

    train: [0.21728392578125, tensor(0.9316, device='cuda:0')]
    valid: [0.08256806640625, tensor(0.9763, device='cuda:0')]
    train: [0.0592223681640625, tensor(0.9822, device='cuda:0')]
    valid: [0.06992761840820312, tensor(0.9806, device='cuda:0')]
    train: [0.038743759765625, tensor(0.9881, device='cuda:0')]
    valid: [0.05885752563476562, tensor(0.9837, device='cuda:0')]
    CPU times: user 3.12 s, sys: 92.9 ms, total: 3.22 s
    Wall time: 3.17 s

### With scheduler

Now let's add the usual warm-up/annealing.

```python
#collapse
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```python
#collapse_show
learn,run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched)])
```

```python
#collapse
run.fit(8, learn)
```

    train: [0.24476150390625, tensor(0.9275, device='cuda:0')]
    valid: [0.107988427734375, tensor(0.9694, device='cuda:0')]
    train: [0.081850380859375, tensor(0.9743, device='cuda:0')]
    valid: [0.11897130126953125, tensor(0.9625, device='cuda:0')]
    train: [0.057883427734375, tensor(0.9814, device='cuda:0')]
    valid: [0.0801966064453125, tensor(0.9750, device='cuda:0')]
    train: [0.0324224951171875, tensor(0.9900, device='cuda:0')]
    valid: [0.05837650146484375, tensor(0.9829, device='cuda:0')]
    train: [0.01931252197265625, tensor(0.9942, device='cuda:0')]
    valid: [0.048959814453125, tensor(0.9843, device='cuda:0')]
    train: [0.011390489501953125, tensor(0.9969, device='cuda:0')]
    valid: [0.04460760192871094, tensor(0.9857, device='cuda:0')]
    train: [0.00623075439453125, tensor(0.9988, device='cuda:0')]
    valid: [0.04336626892089844, tensor(0.9878, device='cuda:0')]
    train: [0.004166948547363282, tensor(0.9995, device='cuda:0')]
    valid: [0.042969384765625, tensor(0.9882, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*". This is the case for large Nets that only allow for small batch sizes. Also RNNs are a problem, as our for loop can not vary the batch size easily.

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6717)

```python
#collapse_show
class LayerNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add  = nn.Parameter(tensor(0.))

    def forward(self, x):
        m = x.mean((1,2,3), keepdim=True)
        v = x.var ((1,2,3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

Keep in mind that compared to BN we use `m = x.mean((1,2,3), keepdim=True)` instead of `m=x.mean((0,2,3), keepdim=True)` and we do not use the exp. moving average. The reason is that every image has it's own mean as we do not use batches anymore.

```python
#collapse
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

```python
#collapse_show
learn,run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```python
#collapse_show
%time run.fit(3, learn)
```

    train: [nan, tensor(0.1330, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 3.81 s, sys: 56.3 ms, total: 3.87 s
    Wall time: 3.82 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?
No we can not, layer norm will lead to the same normalization for both pictures. As we can see LN is not as good as BN, but it can be used for RNNS.

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7114)

```python
#collapse_show
class InstanceNorm(nn.Module):
    __constants__ = ['eps']
    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds  = nn.Parameter(torch.zeros(nf,1,1))

    def forward(self, x):
        m = x.mean((2,3), keepdim=True)
        v = x.var ((2,3), keepdim=True)
        res = (x-m) / ((v+self.eps).sqrt())
        return res*self.mults + self.adds
```

Keep in mind that compared to LN we use `m = x.mean((2,3), keepdim=True)` instead of `m=x.mean((1,2,3), keepdim=True)`.

```python
#collapse
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn: layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```python
#collapse
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 3.74 s, sys: 69.5 ms, total: 3.81 s
    Wall time: 3.78 s

*Question*: why can't this classify anything?
We are now using the mean and variance for every image and every channel, throwing away the things that allow classification.
It was not designed for Classification, but rather for style transfer where the differences in contrast and overall amount are not important according to the authors.

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) have you covered:

### Group norm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7213)

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7304)

```python
#collapse
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```python
#collapse_show
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
#collapse
%time run.fit(1, learn)
```

    train: [2.3492684375, tensor(0.1668, device='cuda:0')]
    valid: [27557812.6336, tensor(0.1274, device='cuda:0')]
    CPU times: user 1min 32s, sys: 766 ms, total: 1min 33s
    Wall time: 1min 34s

### Running Batch Norm

To solve this problem we introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std. Eps is used to avoid divergence, it is used as a hyperparameter. Running Batch Norm is a good solution (best so far according to Jeremy) for the small batch size problem.

1) It does not divide by the batch standard deviation, but it uses the moving average stats at training time as well, just like during inference. Accuracy increases a lot ! As we should not compute the running average of the variances, especially as there might be different batch sizes as well. We use the formula :
$E[X^{2}]-E[X]^{2}$
So we use the squares and the sums with a buffer.

2) And then we take the exp. moving average of these and interpolate. We also take the exponential moving average of the batch sizes, it tells us what we need to divide by : (total number of elements by the mini batch divided by number of channels)

3) Debiasing
Make sure that at every point, no observation is weighted too much. (early elements have more relevance as they appear more often)

4) For the first elements :
We might be unlucky, so that our first mini batch is very close to zero. So we clamp the first few elements (for example 20) variance to be 0.01.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7516)

```python
#collapse_show
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom,self.eps = mom,eps
        self.mults = nn.Parameter(torch.ones (nf,1,1))
        self.adds = nn.Parameter(torch.zeros(nf,1,1))
        self.register_buffer('sums', torch.zeros(1,nf,1,1))
        self.register_buffer('sqrs', torch.zeros(1,nf,1,1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs,nc,*_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0,2,3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel()/nc)
        mom1 = 1 - (1-self.mom)/math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias*(1-self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training: self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step<100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c    = c    / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20): vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```python
#collapse_show
def get_cnn_layers(data, nfs, layer,conv_dim,n_res_block, **kwargs):
    nfs = [1] + nfs
    res = resblocks_create(conv_dim,1)
    print(res)
    layers= [layer(nfs[i], nfs[i+1], 5 if i==0 else 3, **kwargs)
            for i in range(len(nfs)-1)] + list(res) + [nn.AdaptiveAvgPool2d(1), Lambda(flatten),nn.Dropout(0.4),
        nn.Linear(nfs[-1], data.c)]
    print(layers)
    return layers

def conv_layer(ni, nf, ks=3, stride=2, **kwargs):
    return nn.Sequential(
        nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride), GeneralRelu(**kwargs))

class GeneralRelu(nn.Module):
    def __init__(self, leak=None, sub=None, maxv=None):
        super().__init__()
        self.leak,self.sub,self.maxv = leak,sub,maxv

    def forward(self, x):
        x = F.leaky_relu(x,self.leak) if self.leak is not None else F.relu(x)
        if self.sub is not None: x.sub_(self.sub)
        if self.maxv is not None: x.clamp_max_(self.maxv)
        return x

def init_cnn_(m,f):
    if isinstance(m,nn.Conv2d):
            f(m.weight,a = 0.1)
            if getattr(m,'bias',None) is not None : m.bias.data.zero_()
    for l in m.children() : init_cnn_(l,f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m,f)

def get_cnn_model(data, nfs, layer,conv_dim,n_res_block, **kwargs):
    return nn.Sequential(*get_cnn_layers(data, nfs, layer,conv_dim,n_res_block, **kwargs))
```

```python
#collapse_show
class Runner():
    def __init__(self, cbs=None, cb_funcs=None):
        cbs = listify(cbs)
        for cbf in listify(cb_funcs):
            cb = cbf()
            setattr(self, cb.name, cb)
            cbs.append(cb)
        self.stop,self.cbs = False,[TrainEvalCallback()]+cbs

    @property
    def opt(self):       return self.learn.opt
    @property
    def model(self):     return self.learn.model
    @property
    def loss_func(self): return self.learn.loss_func
    @property
    def data(self):      return self.learn.data

    def one_batch(self, xb, yb):
        try:
            self.xb,self.yb = xb,yb
            self('begin_batch')
            self.pred = self.model(self.xb)
            self('after_pred')
            self.loss = self.loss_func(self.pred, self.yb)
            self('after_loss')
            if not self.in_train: return
            self.loss.backward()
            self('after_backward')
            self.opt.step()
            self('after_step')
            self.opt.zero_grad()
        except CancelBatchException: self('after_cancel_batch')
        finally: self('after_batch')

    def all_batches(self, dl):
        self.iters = len(dl)
        try:
            for xb,yb in dl:
                print('Batch')
                self.one_batch(xb, yb)
        except CancelEpochException: self('after_cancel_epoch')

    def fit(self, epochs, learn):
        self.epochs,self.learn,self.loss = epochs,learn,tensor(0.)

        try:
            for cb in self.cbs: cb.set_runner(self)
            self('begin_fit')
            for epoch in range(epochs):
                self.epoch = epoch
                if not self('begin_epoch'): self.all_batches(self.data.train_dl)

                with torch.no_grad():
                    if not self('begin_validate'): self.all_batches(self.data.valid_dl)
                self('after_epoch')

        except CancelTrainException: self('after_cancel_train')
        finally:
            self('after_fit')
            self.learn = None

    def __call__(self, cb_name):
        res = False
        for cb in sorted(self.cbs, key=lambda x: x._order): res = cb(cb_name) or res
        return res
```

If you recall from the lecture, a simple CNN is used. The goal was to improve one epoch training validation error as a practice toy problem. I did beat the 0.975 slightly from the video with Dropout in the fully connected part, but further added a Residual block after the normal Conv Part, allowing u.

```python
#collapse_show
def get_runner(model, data, lr=0.6, cbs=None, opt_func=None, loss_func = F.cross_entropy):
    if opt_func is None: opt_func = optim.SGD
    opt = opt_func(model.parameters(), lr=lr)
    learn = Learner(model, opt, loss_func, data)
    return learn, Runner(cb_funcs=listify(cbs))
```

```python
#collapse_show
def conv(in_channels, out_channels, kernel_size, stride=2, padding=1, batch_norm=True):
    """Creates a convolutional layer, with optional batch normalization.
    """
    layers = []
    conv_layer = nn.Conv2d(in_channels=in_channels, out_channels=out_channels,
                           kernel_size=kernel_size, stride=stride, padding=padding, bias=False)

    layers.append(conv_layer)

    if batch_norm:
        layers.append(nn.BatchNorm2d(out_channels))
    return nn.Sequential(*layers)
```

```python
#collapse_show
class Resblock(nn.Module):
    def __init__(self,conv_dim):
        super(Resblock,self).__init__()
        self.conv1 = conv(conv_dim, conv_dim, kernel_size=3, stride=2, batch_norm=True)
        self.conv2 = conv(conv_dim, conv_dim, kernel_size=3, stride=2, batch_norm=True)
    def forward(self,x):
        out1 = F.leaky_relu(self.conv1(x))
        out =  x + F.leaky_relu(self.conv2(out1))
        return out
```

```python
#collapse_show
def resblocks_create(conv_dim,n_res_blocks):
    res_layers = []
    for l in range(0,n_res_blocks):
        res_layers.append(Resblock(conv_dim))
    return nn.Sequential(*res_layers)
```

```python
#collapse_show
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn: layers.append(RunningBatchNorm(nf))
    print(**kwargs)
    return nn.Sequential(*layers)
```

```python
#collapse
def get_learn_run(nfs, data,conv_dim,n_res_block, lr, layer, cbs=None, opt_func=None, uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer,conv_dim,n_res_block, **kwargs)
    init_cnn(model, uniform=False)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

```python
#collapse
nfs = [8,16,32,64]
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 64,1,0.4, conv_rbn, cbs=cbfs)
```

    Sequential(
      (0): Resblock(
        (conv1): Sequential(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Sequential(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )

    [Sequential(
      (0): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Sequential(
      (0): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Sequential(
      (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Sequential(
      (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Resblock(
      (conv1): Sequential(
        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (conv2): Sequential(
        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    ), AdaptiveAvgPool2d(output_size=1), Lambda(), Dropout(p=0.4, inplace=False), Linear(in_features=64, out_features=10, bias=True)]

```python
learn.model
```

    Sequential(
      (0): Sequential(
        (0): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (1): Sequential(
        (0): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (2): Sequential(
        (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (3): Sequential(
        (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): GeneralRelu()
        (2): RunningBatchNorm()
      )
      (4): Resblock(
        (conv1): Sequential(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Sequential(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
      (5): AdaptiveAvgPool2d(output_size=1)
      (6): Lambda()
      (7): Dropout(p=0.4, inplace=False)
      (8): Linear(in_features=64, out_features=10, bias=True)
    )

```python
#collapse
%time run.fit(1, learn)
```

    train: [34.587845, tensor(0.3369, device='cuda:0')]
    valid: [96743292.928, tensor(0.4051, device='cuda:0')]
    CPU times: user 3min 41s, sys: 1.03 s, total: 3min 42s
    Wall time: 3min 43s

This solves the small batch size issue!

### What can we do in a single epoch?

Now let's see with a decent batch size what result we can get.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=8068)

```python
#collapse
data = DataBunch(*get_dls(train_ds, valid_ds, 16), c)
```

```python
#collapse
learn,run = get_learn_run(nfs, data, 64,1, 0.9, conv_rbn, cbs=cbfs
                          +[partial(ParamScheduler,'lr', sched_lin(1., 0.2))])
```

    Sequential(
      (0): Resblock(
        (conv1): Sequential(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
        (conv2): Sequential(
          (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
          (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
        )
      )
    )

    [Sequential(
      (0): Conv2d(1, 8, kernel_size=(5, 5), stride=(2, 2), padding=(2, 2), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Sequential(
      (0): Conv2d(8, 16, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Sequential(
      (0): Conv2d(16, 32, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Sequential(
      (0): Conv2d(32, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
      (1): GeneralRelu()
      (2): RunningBatchNorm()
    ), Resblock(
      (conv1): Sequential(
        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
      (conv2): Sequential(
        (0): Conv2d(64, 64, kernel_size=(3, 3), stride=(2, 2), padding=(1, 1), bias=False)
        (1): BatchNorm2d(64, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
      )
    ), AdaptiveAvgPool2d(output_size=1), Lambda(), Dropout(p=0.4, inplace=False), Linear(in_features=64, out_features=10, bias=True)]

```python
#collapse
%time run.fit(1, learn)
```

    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    Batch
    train: [0.12176001953125, tensor(0.9641, device='cuda:0')]
    valid: [0.053298565673828124, tensor(0.9842, device='cuda:0')]
    CPU times: user 29.5 s, sys: 327 ms, total: 29.8 s
    Wall time: 29.5 s

```python

```


```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
import numpy as np
import pandas as pd
import sympy as sym

import solowpy
```

```python
# define model parameters
ces_params = {'A0': 1.0, 'L0': 1.0, 'g': 0.02, 'n': 0.03, 's': 0.15,
              'delta': 0.05, 'alpha': 0.33, 'sigma': 0.95}

# create an instance of the solow.Model class
ces_model = solowpy.CESModel(params=ces_params)
```

# 3. Graphical analysis using Matplotlib and IPython widgets

As the primary use of this module is for teaching purposes, there are a number of pedagogically useful plotting methods. I will demonstrate the basic usage of only a few of them below. To see a full listing of the available plotting methods use tab-completion on the cell below.

```python
# use tab completion to see complete list
ces_model.plot_
```

### Static example:
Creating a static plot of the classic Solow diagram is done as follows.

```python
fig, ax = plt.subplots(1, 1, figsize=(8,6))
ces_model.plot_solow_diagram(ax)
fig.show()
```

## Interactive example:
All of the various plotting methods can be made interactive using IPython widgets. To construct an IPython widget we need the following additional `import` statements.

```python
from IPython.html.widgets import fixed, interact, FloatSliderWidget
```

Creating an interactive plot of the classic Solow diagram is done as follows.

```python
# wrap the static plotting code in a function
def interactive_solow_diagram(model, **params):
    """Interactive widget for the factor shares."""
    fig, ax = plt.subplots(1, 1, figsize=(8, 6))
    model.plot_solow_diagram(ax, Nk=1000, **params)

# define some widgets for the various parameters
eps = 1e-2
technology_progress_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
population_growth_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
savings_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
output_elasticity_widget = FloatSliderWidget(min=eps, max=1.0, step=eps, value=0.5)
depreciation_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
elasticity_substitution_widget = FloatSliderWidget(min=eps, max=10.0, step=0.01, value=1.0+eps)

# create the widget!
interact(interactive_solow_diagram,
         model=fixed(ces_model),
         g=technology_progress_widget,
         n=population_growth_widget,
         s=savings_widget,
         alpha=output_elasticity_widget,
         delta=depreciation_widget,
         sigma=elasticity_substitution_widget,
         )
```

## 3.1 Intensive production function
Creating an interactive plot of the intensive production function is done as follows.

```python
model.plot_intensive_output?
```

```python
def interactive_intensive_output(model, **params):
    """Interactive widget for the intensive production function."""
    fig, ax = plt.subplots(1, 1, figsize=(8, 6))
    model.plot_intensive_output(ax, Nk=1000, **params)

# define some widgets for the various parameters
eps = 1e-2
output_elasticity_widget = FloatSliderWidget(min=eps, max=1-eps, step=0.1, value=0.5)
elasticity_substitution_widget = FloatSliderWidget(min=eps, max=10.0, step=0.5, value=1.0+eps)

# create the interactive plot
interact(interactive_intensive_output,
         model=fixed(ces_model),
         alpha=output_elasticity_widget,
         sigma=elasticity_substitution_widget
         )
```

##3.2  Factor shares
Creating an interactive plot of factor shares for capital and labor is done as follows.

```python
def interactive_factor_shares(model, **params):
    """Interactive widget for the factor shares."""
    fig, ax = plt.subplots(1, 1, figsize=(8, 6))
    model.plot_factor_shares(ax, Nk=1000, **params)

# define some widgets for the various parameters
eps = 1e-2
technology_progress_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
population_growth_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
savings_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
output_elasticity_widget = FloatSliderWidget(min=eps, max=1.0-eps, step=eps, value=0.5)
depreciation_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
elasticity_substitution_widget = FloatSliderWidget(min=eps, max=10.0, step=0.5, value=1.0+eps)

# create the widget!
interact(interactive_factor_shares,
         model=fixed(ces_model),
         g=technology_progress_widget,
         n=population_growth_widget,
         s=savings_widget,
         alpha=output_elasticity_widget,
         delta=depreciation_widget,
         sigma=elasticity_substitution_widget,
         )
```

## 3.4 Phase Diagram
Creating an interactive plot of the phase diagram for the Solow model is done as follows.

```python
def interactive_phase_diagram(model, **params):
    """Interactive widget for the phase diagram."""
    fig, ax = plt.subplots(1, 1, figsize=(8, 6))
    model.plot_phase_diagram(ax, Nk=1000, **params)

# define some widgets for the various parameters
eps = 1e-2
technology_progress_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
population_growth_widget = FloatSliderWidget(min=-0.05, max=0.05, step=eps, value=0.02)
savings_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
output_elasticity_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
depreciation_widget = FloatSliderWidget(min=eps, max=1-eps, step=eps, value=0.5)
elasticity_substitution_widget = FloatSliderWidget(min=eps, max=10.0, step=0.5, value=1.0+eps)

# create the widget!
phase_diagram_widget = interact(interactive_phase_diagram,
                                model=fixed(ces_model),
                                g=technology_progress_widget,
                                n=population_growth_widget,
                                s=savings_widget,
                                alpha=output_elasticity_widget,
                                delta=depreciation_widget,
                                sigma=elasticity_substitution_widget
                                )
```

```python

```


# Ellipsometer Basics

**Scott Prahl**

*April 2020*

This notebook reviews the basic equations used in ellipsometry.  It also demonstrates how the Fresnel reflection equations are related to the ellipsometry parameter $\rho = \tan\psi \cdot e^{j\Delta}$.

References

* Archer, *Manual on Ellipsometry* 1968.
* Azzam, *Ellipsometry and Polarized Light*, 1977.
* Fujiwara, *Spectroscopic Ellipsometry*, 2007.
* Tompkins, *A User's Guide to Ellipsometry*, 1993
* Tompkins, *Handbook of Ellipsometry*, 2005.
* Woollam, *A short course in ellipsometry*, 2001.

```python
import sympy
import numpy as np
import matplotlib.pyplot as plt

import pypolar.fresnel as fresnel
import pypolar.ellipsometry as ellipse

sympy.init_printing(use_unicode=True)
```

## Ellipsometry

### Layout
A basic ellipsometer configuration is shown below where the lights hits the sample at an angle $\theta$ from a normal to the surface.  The  incident electric field $E_{ip}$ is parallel to the plane-of-incidence (which contains the incoming vector and the vector normal to the surface).  The electric field $E_{is}$ is perpendicular to the plane of incidence

Usually, the incident light $\mathbf{E}_i$ is linearly polarized but does not need to be.  The reflected light $\mathbf{E}_r$ is, in general, elliptically polarized.

$$
\mathbf{E}_i =
\begin{bmatrix}
E_{ip}e^{j\delta_{ip}}\\
E_{is}e^{j\delta_{is}}\\
\end{bmatrix}
\qquad\mbox{and}\qquad
\mathbf{E}_r =
\begin{bmatrix}
E_{rp}e^{j\delta_{rp}}\\
E_{rs}e^{j\delta_{rs}}\\
\end{bmatrix}
$$

### $\Delta$ and $\tan\psi$

The effect of reflection is characterized by the angle $\Delta$, defined as the change in phase, and the angle $\psi$, the arctangent of the factor by which the amplitude ratio changes.

$$
\Delta = (\delta_{rp} - \delta_{rs}) - (\delta_{ip}-\delta_{is})
$$

and

$$
\tan\psi = \frac{E_{is}}{E_{ip}} \cdot \frac{E_{rp}}{E_{rs}}
$$

In the special (but common case) of a smooth surface, there will be no mixing of parallel and perpendicular light, i.e.,

$$
\begin{bmatrix}
E_{rp}e^{j\delta_{rp}}\\
E_{rs}e^{j\delta_{rs}}
\end{bmatrix}
=
\begin{bmatrix}
r_p & 0 \\
0 & r_s
\end{bmatrix}
\begin{bmatrix}
E_{ip}e^{j\delta_{ip}}\\
E_{is}e^{j\delta_{is}}
\end{bmatrix}
=
\begin{bmatrix}
r_p E_{ip}e^{j\delta_{ip}}\\
s_s E_{is}e^{j\delta_{is}}
\end{bmatrix}
$$

the $\tan\psi$ equation simplifies to

$$
\tan\psi = \frac{r_p}{r_s}
$$

These overall change in field is often written as a single complex number $\rho$

$$
\rho = \tan\psi e^{j\Delta}
$$

Ellipsometry is the science of measuring and interpreting $\Delta$ and $\psi$ for a surface. A simple use of ellipsometry is to determine the complex index of refraction $m$ for a thick uniform flat substrate.  More elaborate ellisometric techniques allow one to determine one or more coatings on the substrate.

### Refractive index

When light reflects off a surface, the amplitude and phase of the electric changes.  These changes depend on the wavelength $\lambda$ of the light, the angle of incidence $\theta$, the complex refractive index of the material $m= n(1-j \kappa)$, and the polarization state of the incident beam:

* The plane of incidence contains the incident electric field propagation vector and the normal to the surface.

* The angle of incidence is the angle beween these two directions

* The real part of the refractive index $n$ determines the speed of light in the material

* The imaginary part of the refractive index $\kappa$ determines the light absorption of the material

* Linearly polarized light parallel to the plane of incidence is **p-polarized**.

* Linearly polarized light perpendicular to the plane of incident is **s-polarized**.

* The phase shift and amplitude change is different for p and s- polarized light.

* For dielectrics like glass, the amount of reflected light is determined by a single number, the index of refraction $n$.

* Semi-conductors and metals have a complex index of refraction  $m = n(1 - j \kappa)$.

### Fresnel Reflection

The Fresnel formulas for light incident from a vacuum onto a flat surface at an angle $\theta$ from the normal with refractive index $m$ varies with the orientation of the electric field.  The plane of incidence is defined as containing both the incident direction and the normal to the surface.  If the incident field is parallel to the plane of incidence then
$$
r_p=r_\parallel ={m^2\cos\theta-\sqrt{m^2-\sin^2\theta}\over
                        m^2\cos\theta+\sqrt{m^2-\sin^2\theta}}
$$
If the incident field is perpendicular to the plane of incidence then
$$
r_s=r_\perp ={\cos\theta-\sqrt{m^2-\sin^2\theta}\over
                    \cos\theta+\sqrt{m^2-\sin^2\theta}}
$$

### Fundamental Equation of Ellipsometry

Ellipsometers are used to determine the parameters $\psi$ and $\Delta$ which can be used to calculate $\rho$
$$
\rho = {r_p(\theta)\over r_s(\theta)} = \tan\psi \cdot e^{j \Delta}
$$

The graph below shows how $\psi$ and $\Delta$ vary with the incidence angle

### Determining the complex index of refraction

A convenient formula for an isotropic, uniform sample is given in
[Measurement of the Thickness and Refractive Index of Very Thin Films and the Optical Properties of Surfaces by Ellipsometry](https://archive.org/details/jresv67An4p363) so that when $\rho$ and $\theta$ are known, the complex index of refraction can be calculated.
$$
m = \tan\theta \sqrt{1-{4\rho\sin^2\theta\over (1+\rho)^2}}
$$

```python
def plot_rho(m):
    angles = np.linspace(0.001,90,91)
    rho = ellipse.rho_from_m(m, angles, deg=True)

    plt.plot(angles, np.abs(rho), color='red')
    plt.plot(angles, np.angle(rho),color='blue')

    plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
    plt.ylabel(r'$\tan\psi$ or $\Delta$')
    plt.title('Refractive Index %.3f%.3fj' % (m.real,m.imag))

    plt.xlim(0,90)
    plt.text(30, 3, r'$\Delta$', color='blue')
    plt.text(30, 0.8, 'tan($\psi$)', color='red')

m = 1.5
plot_rho(m)
plt.show()

m      = 1.5-1.0j
plot_rho(m)
plt.show()

m      = 3-3j
plot_rho(m)
plt.show()
```

Replicate figures 2.28 and 2.29 from Woollam's Short Course.

```python
def plot_rho2(m1,m2):
    angles = np.linspace(65,85,181)
    rho1 = ellipse.rho_from_m(m1,angles,deg=True)
    rho2 = ellipse.rho_from_m(m2,angles,deg=True)

    psi1 = np.degrees(np.arctan(np.abs(rho1)))
    psi2 = np.degrees(np.arctan(np.abs(rho2)))

    Delta1 = np.degrees(np.angle(rho1))
    Delta2 = np.degrees(np.angle(rho2))

    plt.subplots(2,1,figsize=(8,8))
    plt.subplot(2,1,1)

    plt.plot(angles, psi1, color='red')
    plt.plot(angles, psi2,color='blue')
    plt.ylabel(r'$\psi$ (degrees)')
    plt.title('%.3f%+.3fj and %.3f%+.3fj' % (m1.real,m1.imag,m2.real,m2.imag))
    plt.grid(True)

    plt.subplot(2,1,2)
    plt.plot(angles, Delta1, color='red')
    plt.plot(angles, Delta2,color='blue')
    plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
    plt.ylabel(r'$\Delta$ (degrees)')
    plt.grid(True)

m1 = 4.516 - 0.249j  # amorphous silicon
m2 = 3.875 - 0.023j  # crystalline silicon
plot_rho2(m1,m2)
plt.show()
```

## Extracting Refractive Index from $\rho$

First we will calculate $\rho$ for a known complex index of refraction.

```python
m = 1.5-1.3j
theta_i = np.linspace(1,89,15)
rho = ellipse.rho_from_m(m, theta_i, deg=True)

plt.plot(theta_i, np.abs(rho), 'o', color='red')
plt.plot(theta_i, np.angle(rho), 'o', color='blue')

plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
plt.ylabel(r'$\tan\psi$ or $\Delta$')
plt.title('Refractive Index %.3f%.3fj' % (m.real,m.imag))

plt.xlim(0,90)
plt.text(30, 3, r'$\Delta$', color='blue')
plt.text(30, 1.0, 'tan($\psi$)', color='red')

plt.show()
```

Then, we will see if we can recover $m$ using $\rho$ and $\theta_i$.
We will test with incidence angles from 1° to 89°.  We avoid 0° and 90° because these angles are either impossible or do not contain sufficient information to invert.

```python
m = 1.5-1.3j
theta_i = np.linspace(1,89,15)
rho = ellipse.rho_from_m(m, theta_i, deg=True)

m2 = ellipse.m_from_rho(rho, theta_i, deg=True)

plt.plot(theta_i, m2.real, 'o', color = 'blue')
plt.plot(theta_i, m2.imag, 'o', color='red')

plt.text(theta_i[0], m2[0].real-0.05, r'n', color='blue', va="top")
plt.text(theta_i[0], m2[0].imag+0.1, r'$n \kappa$', color='red')

plt.axhline(m.real,color='blue')
plt.axhline(m.imag,color='red')
plt.xlabel(r"Incidence angle $\theta_i$ (degrees)")
plt.ylabel('Real or Imaginary part of Refractive Index')
plt.title('Refractive Index = %.4f %.4fj'%(m.real,m.imag))

plt.show()
```

Quantitatively show that things work at an incidence angle of 70°

```python
theta_i = 70
m = 1.5 - 2.0j
rho = ellipse.rho_from_m(m, theta_i, deg=True)
print('Incidence angle  = %.1f°'%(theta_i))
print('rho              = %.3f%+.3fj'%(rho.real,rho.imag))
print('Refractive index = %.3f%+.3fj'%(m.real,m.imag))
print()
m2 = ellipse.m_from_rho(rho, theta_i, deg=True)
print('Extracted index  = %.3f%+.3fj using rho'%(m2.real,m2.imag))
```

```python

```


# Quantum Chemistry

#### Chris-Kriton Skylaris

## Outline

* Schrödinger equation
* Observable properties
* Hamiltonian operators for molecules
* Molecular orbitals and many-electron wavefunctions
* Energy of the many-electron wavefunction
* Molecular integrals
* Hartree-Fock

# Schrödinger's equation and Dirac's bra-ket notation

Key breakthrough: Schrödinger's equation

\begin{equation}
  \hat{H} \Psi = E \Psi.
\end{equation}

Examples of systems that can be solved exactly:

* Particle in a box: $E_n = \frac{n^2 h^2}{8 m l^2}$
* Harmonic oscillator: $E_n = \left( n + \frac{1}{2} \right) h \nu$
* Hydrogen atom: $E_n = -\frac{Z^2}{n^2} \left( \frac{e^2}{2 a} \right)$.

All are quantized.

The Schrödinger equation splits into two parts: the kinetic energy piece $\hat{T}$ and the potential energy piece $\hat{V}$, giving

\begin{equation}
  \left( \hat{T} + \hat{V} \right) \Psi = E \Psi.
\end{equation}

All are operators, taking one function (the wavefunction here) to another function.

The wavefunction is a function of the $3N$ coordinates, where there are $N$ particles. This will be written as

\begin{equation}
  \Psi(x_1, y_1, z_1, x_2, y_2, z_2, \dots, x_N, y_N, z_N) = \Psi({\bf r}_1, {\bf r}_2, \dots, {\bf r}_N).
\end{equation}

For a hydrogen molecule (two hydrogen nuclei connected by a bond), we consider the nuclei as *one particle each*, with a single coordinate. For the purposes of counting for now, we note that we have two H nuclei, plus two electrons, for four particles, hence 12 coordinates.

For a benzene molecule (six carbon atoms connected by various bonds, each connected to a hydrogen atom) we have six carbon nuclei and six hydrogen nuclei, and each carbon atom has six electrons (for 36 electrons), and each hydrogen atom has six electrons. This gives 54 particles for 162 coordinates.

Observables or expectation values correspond to experimental measurements. According to quantum theory we compute the outcome of any measurement by "averaging" the appropriate operator as

\begin{equation}
  < x > = \frac{\int_{-\infty}^{\infty} \Psi^*(x) \hat{x} \Psi(x) \, \text{d}x}{\int_{-\infty}^{\infty} \Psi^*(x) \Psi(x) \, \text{d}x}.
\end{equation}

This particular example gives the expectation value, or average value, of the position along the $x$-axis, where the operator for position $\hat{x}$ is the multiplicative operator $\hat{x} \Psi(x) = x \Psi(x)$.

## Bra-ket notation

Write

\begin{equation}
  \int \Psi^*({\bf x}) \, \hat{C} \, \Phi({\bf x}) \, \text{d}{\bf x} \longrightarrow \langle \Psi | \hat{C} | \Phi \rangle.
\end{equation}

The "bra" is

\begin{equation}
  \langle \Psi | \equiv \int \text{d}{\bf x} \, \Psi^*({\bf x}).
\end{equation}

The "ket" is

\begin{equation}
  | \Phi \rangle \equiv \Phi({\bf x}).
\end{equation}

The operator doesn't appear explicitly; it's the $\hat{C}$ in "bra"$\hat{C}$"ket".

As an example, the Schrödinger equation is written $\hat{H} | \Psi \rangle = E | \Psi \rangle$.

The key result of the examples is that

\begin{equation}
  E = \frac{\langle \Psi | \hat{H} | \Psi \rangle}{\langle \Psi | \Psi \rangle}.
\end{equation}

# Hamiltonian operators for molecules

Most operators are obvious: take the classical expression directly (eg potential $V(x)$ has operator $V(x)$). However, momentum is different:

\begin{equation}
  m v_x \longrightarrow - i \hbar \frac{\text{d}}{\text{d}x}
\end{equation}

Now, the Hamiltonian is the sum of the kinetic and potential energy operators. Kinetic energy is

\begin{align}
  T &= \frac{1}{2} m v_x^2 \\ &= \frac{(m v_x)^2}{2 m} \\ &= \frac{1}{2m} p_x^2.
\end{align}

Hence the previous result gives

\begin{align}
  \hat{T} &= \frac{1}{2m} \hat{p}_x^2 \\ &= \frac{1}{2m} \hat{p}_x \hat{p}_x \\ &= \frac{1}{2m} \left( -i \hbar \frac{\text{d}}{\text{d}x} \right) \left( -i \hbar \frac{\text{d}}{\text{d}x} \right) \\ & = -\frac{\hbar^2}{2m} \frac{\text{d}^2}{\text{d}x^2}.
\end{align}

Hence the Hamiltonian operator is

\begin{equation}
  \hat{H} = -\frac{\hbar^2}{2m} \frac{\text{d}^2}{\text{d}x^2} + V(x).
\end{equation}

Simplest case is a hydrogen atom. The location of the nucleus is given by ${\bf R}$. The location of the electron is given by ${\bf r}$. The vector connecting the nucleus to the electron is therefore ${\bf r} - {\bf R}$.

We remember that the force between two charged particles separated by distance $r$ is given by Coulomb's Law $F = q_1 q_2 / r^2$ and has potential $v({\bf r}) = 1 / (4 \pi \epsilon_0) q_1 q_2 / |{\bf r}|$.

The Hamiltonian operator for this single-atom system is

\begin{equation}
  \hat{H} = -\frac{\hbar^2}{2M} \nabla^2_{\bf R} -\frac{\hbar^2}{2m} \nabla^2_{\bf r}  - \frac{1}{4 \pi \epsilon_0} \frac{e^2}{|{\bf r} - {\bf R}|},
\end{equation}

where $m$ is the mass of the electron and $M$ the mass of the nucleus.

### Atomic units

Choose units so that $\hbar^2 / m_e = 1 = e^2 / (4 \pi \epsilon_0)$. This simplifies the Hamiltonian to

\begin{equation}
  \hat{H} = -\frac{1}{2M} \nabla^2_{\bf R} -\frac{1}{2} \nabla^2_{\bf r}  - \frac{1}{|{\bf r} - {\bf R}|},
\end{equation}

where $M$ is now written in terms of electron masses, and energies are now in *Hartrees*.

### Hamiltonian for Helium

\begin{equation}
  \hat{H} = -\frac{1}{2M} \nabla^2_{\bf R} -\frac{1}{2} \nabla^2_{{\bf r}_1} -\frac{1}{2} \nabla^2_{{\bf r}_2}  - \frac{2}{|{\bf r}_1 - {\bf R}|} - \frac{2}{|{\bf r}_2 - {\bf R}|} + \frac{1}{|{\bf r}_1 - {\bf r}_2|}.
\end{equation}

The final term is the repulsion between the two electrons. The factors of two follow as the nucleus now contains two protons, so has charge 2.

### Hamiltonian for water

\begin{equation}
  \hat{H} = -\frac{1}{2M_O} \nabla^2_{{\bf R}_O}-\frac{1}{2M_{H_1}} \nabla^2_{{\bf R}_{H_1}}-\frac{1}{2M_{H_2}} \nabla^2_{{\bf R}_{H_2}} - \sum_{i=1}^{10} \frac{1}{2} \nabla^2_{{\bf r}_i} - \sum_{i=1}^{10} \frac{8}{|{\bf r}_i - {\bf R}_O|} - \sum_{i=1}^{10} \frac{1}{|{\bf r}_i - {\bf R}_{H_1}|} - \sum_{i=1}^{10} \frac{1}{|{\bf r}_i - {\bf R}_{H_2}|} + \sum_{i=1}^{10} \sum_{j=i+1}^{10} \frac{1}{|{\bf r}_i - {\bf r}_j|} + \sum_{I=1}^{3} \sum_{J=I+1}^{3} \frac{Z_I Z_J}{|{\bf R}_I - {\bf R}_J|} .
\end{equation}

Notation is such that ${\bf R}_1 = {\bf R}_O$, ${\bf R}_2 = {\bf R}_{H_1}$, and so on. Note how the double sums only count each term once.

## Separating electronic from nuclear coordinates

Nuclei are much heavier than electrons ($m_p \approx 1800 m_e$), so it's a good approximation to assume that the electrons move much faster than nuclei: the electrons *instantaneously* equilibriate to the locations of the nuclei.

This is the core of the *Born-Oppenheimer* approximation: Solve Schrödinger's equation only in the electronic coordinates, with the nuclear coordinates held fixed. We then split the full Hamiltonian into the nuclear piece and the electronic piece $\hat{H}_{\text{elec}}$, and only solve for

\begin{align}
  \hat{H}_{\text{elec}} \Phi_{\text{elec}} &= E_{\text{elec}} \Phi_{\text{elec}} \\
  \Phi_{\text{elec}} &= \Phi_{\text{elec}} \left( \left\{ {\bf r}_i \right\}; \left\{ {\bf R}_I \right\} \right) \\
  E_{\text{elec}} &= E_{\text{elec}} \left( \left\{ {\bf R}_I \right\} \right)
\end{align}

That is, the solutions that we get depend *parametrically* on the coordinates of the nuclei.

For the water molecule, the Born-Oppenheimer approximation gives

\begin{equation}
  \hat{H}_{\text{elec}} = - \sum_{i=1}^{10} \frac{1}{2} \nabla^2_{{\bf r}_i} - \sum_{i=1}^{10} \frac{8}{|{\bf r}_i - {\bf R}_O|} - \sum_{i=1}^{10} \frac{1}{|{\bf r}_i - {\bf R}_{H_1}|} - \sum_{i=1}^{10} \frac{1}{|{\bf r}_i - {\bf R}_{H_2}|} + \sum_{i=1}^{10} \sum_{j=i+1}^{10} \frac{1}{|{\bf r}_i - {\bf r}_j|} + \sum_{I=1}^{3} \sum_{J=I+1}^{3} \frac{Z_I Z_J}{|{\bf R}_I - {\bf R}_J|}.
\end{equation}

That is, terms given by varying the location of the nuclei vanish. In addition, the final nuclear interaction term becomes constant.

# Slater determinant wavefunctions

Now the approximations get more serious.

A wavefunction for a single electron is called a *molecular orbital* (MO). MOs with spatial *and* spin coordinates are called *spin orbitals* and are products of a spatial orbital and a spin function. There are only two kinds of spin function: "up" ($\alpha$) and "down" ($\beta$). We write

\begin{align}
  \chi^{\uparrow}({\bf x}) &= \psi({\bf r}) \alpha(\omega), \\
  \chi^{\downarrow}({\bf x}) &= \psi({\bf r}) \beta(\omega)
\end{align}

In order to ensure that our (approximate) full wavefunction has the right symmetries, but is made out of single electron MOs, we use the *Slater determinant*, eg

\begin{equation}
  \Psi({\bf x}_1, {\bf x}_2) = \begin{vmatrix} \chi_1({\bf x}_1) & \chi_2({\bf x}_1) \\ \chi_1({\bf x}_2) & \chi_2({\bf x}_2) \end{vmatrix}
\end{equation}

This can be extended to an arbitrary number of MOs by increasing the number of terms in the determinant. However, it should include a $N!^{-1/2}$ normalization so that $\langle \Psi_{\text{SD}} | \Psi_{\text{SD}} \rangle = 1$.

You will often see the representation

\begin{equation}
  \Psi_{\text{SD}}({\bf x}_1, {\bf x}_2, \dots {\bf x}_N) = | \chi_1({\bf x}_1) \chi_2({\bf x}_2) \dots \chi_N({\bf x}_N) \rangle
\end{equation}

The spin orbitals are always orthonormal $\langle \chi_i | \chi_j \rangle = \delta_{ij}$.

## Pauli principle

A determinant is zero if two or more of its rows or columns are identical: therefore a spin orbital can be included **only once** in a Slater determinant (otherwise the wavefunction is zero everywhere, which isn't physically acceptable). This is the Pauli exclusion principle successfully encoded.

# Electronic energy of a Slater determinant

The electronic energy of a single SD is

\begin{align}
  E_{\text{SD}} &= \frac{ \langle \Psi_{\text{SD}} | \hat{H}_{\text{elec}} | \Psi_{\text{SD}} \rangle }{ \langle \Psi_{\text{SD}}  | \Psi_{\text{SD}} \rangle } \\
  & = \langle \Psi_{\text{SD}} | \hat{H}_{\text{elec}} | \Psi_{\text{SD}} \rangle
\end{align}

with

\begin{equation}
  \hat{H}_{\text{elec}} = - \sum_{i=1}^{N_{\text{elec}}} \frac{1}{2} \nabla_i^2 - \sum_{I=1}^{N_{\text{nuclei}}} \sum_{i=1}^{N_{\text{elec}}} \frac{Z_I}{|{\bf r}_i - {\bf R}_I|} + \sum_{i=1}^{N_{\text{elec}}} \sum_{j=i+1}^{N_{\text{elec}}} \frac{1}{|{\bf r}_i - {\bf r}_j|}.
\end{equation}

After manipulations we find

\begin{equation}
  E_{\text{kin}} = \sum_i \langle \chi_i | - \nabla_i^2 | \chi_i \rangle
\end{equation}

and the nuclear energies give

\begin{equation}
  E_{e-n} = \sum_i \langle \chi_i | \sum_I \frac{-Z_I}{|{\bf r}_i - {\bf R}_I|} | \chi_i \rangle
\end{equation}

The nuclear attraction potential for each electron is

\begin{equation}
  \hat{v}_{\text{ext}} =
\end{equation}

The *difficult* term is the electron-electron repulsion energy, or the two-electron integrals

\begin{equation}
  E_{e-e} = \frac{1}{2} \sum_i \sum_j \left[ \int\int \dots - \int\int \dots \right]
\end{equation}

where $r_{12} = |{\bf r}_1 - {\bf r}_2|$.

Remember that here $1, 2$ are dummy indices for the integration, not the first and second electrons.

The computational cost of computing these integrals goes as the *fourth* power of the number of electrons.

The part of the energy that doesn't include the two integral terms is written $\sum_i \langle \chi_i | \hat{h} | \chi_i \rangle$, where $\hat{h} = - \frac{1}{2} \nabla^2 + \sum_I \frac{-Z_I}{|{\bf r}_i - {\bf R}_I|}$ is the *core Hamiltonian*, which neglects the electron-electron coupling.

The spin terms can be integrated out, so that the sums over each term can become $2 \sum_{i=1}^{N_{\text{elec}/2}}$ (integer division). The energy of the Slater determinant then breaks down into the core Hamiltonian $h_i$, the Coulomb integral $J_{ij}$ which is symmetric in the coordinates ${\bf r}_{1,2}$, and the Exchange integral $K_{ij}$, which is not (and doesn't have a physical interpretation).

In summary,

\begin{equation}
  E_{\text{SD}} = 2 \sum_{i=1}^{N_{\text{elec}}/2} h_i + \sum_{i=1}^{N_{\text{elec}}/2} \sum_{j=1}^{N_{\text{elec}}/2} ( 2 J_{ij} - K_{ij}).
\end{equation}

# Hartree-Fock method

Assume we have two Slater determinants for a molecule, $\Psi^{A,B}_{\text{SD}}$, with energies $E^{A,B}_{\text{SD}} = \langle \Psi^{A,B}_{\text{SD}} | \hat{H} | \Psi^{A,B}_{\text{SD}} \rangle$. Then if $E^A_{\text{SD}} > E^B_{\text{SD}}$ then, according to teh variation principle, we take $\Psi^B$ to be a better approximation to the wavefunction.

By performing functional variation we can show that the minimal value occurs when

\begin{equation}
  \hat{f} \psi_i({\bf r}) = \epsilon_i \psi_i({\bf r})
\end{equation}

This is an eigenfunction equation involving the *Fock* operator $\hat{f}$ which is made of three pieces,

\begin{equation}
  \hat{f} = \hat{h} + \sum_i ( 2 \hat{J}_i - \hat{K}_i )
\end{equation}

where $h, J, K$ correspond to the core Hamiltonian, Coulomb and Exchange operators.

The eigenfunctions of the Fock operator are the optimal MOs, and the eigenvalues their associated energies.

An example: the Hartree-Fock eigenvalue equation for a molecule with 6 electrons is

\begin{equation}
  \left[ \hat{h} + \sum_{i=1}^3 \left( 2 \hat{J}_i - \hat{K}_i \right) \right] \Psi_j({\bf r}) = \epsilon_j \Psi_j({\bf r})
\end{equation}

The index $j$ can be anything. The Fock operator is made of the first three orbitals $\Psi_{1,2,3}$.

So we solve iteratively using the *Self-Consistent Field* procedure. First guess the MOs $\psi_i$. Construct the Fock operator and solve the eigenfunction problem to find a new set of MOs. If the new MOs are close enough to the old MOs the procedure has converged; otherwise, construct the Fock operator and repeat.

FEEG6016 Simulation and Modelling/Quantum-Chemistry-Lectures.ipynb