```python
%load_ext autoreload
%autoreload 2

%matplotlib inline
```

```python
#export
from exp.nb_06 import *
```

## ConvNet

We have seen what we can do with initialization, but to train really deep networks we need other tricks. One can use SeLU or mixup(?), but these are super-fiddly. J.H. would rather use the Layer-wise Sequential Unit Variance (**LSUV**) that we will encounter in one of the next notebooks. With just initialization, this is as far as we can go, but to go further, we need *normalization*, and in particular Batch Normalization.

The Batch Normalization paper shows the algorithm to be used, and it contains two parameters to be learned: $\gamma$ and $\beta$. In a neural network there are two main quantities: *parameters*, that are things we *learn*, and *activations*, that are things we *calculate*.

Let's get the data and training interface from where we left in the last notebook.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=5899)

```python
x_train, y_train, x_valid, y_valid = get_data()

x_train, x_valid = normalize_to(x_train, x_valid)
train_ds, valid_ds = Dataset(x_train, y_train), Dataset(x_valid, y_valid)

nh, bs = 50, 512
c = y_train.max().item()+1
loss_func = F.cross_entropy

data = DataBunch(*get_dls(train_ds, valid_ds, bs), c)
```

```python
mnist_view = view_tfm(1, 28, 28)
cbfs = [Recorder,
        partial(AvgStatsCallback, accuracy),
        CudaCallback,
        partial(BatchTransformXCallback, mnist_view)]
```

```python
nfs = [8, 16, 32, 64, 64]
```

0.4 was the highest learning rate we could get in our previous run.

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(2, learn)
```

    train: [1.60180046875, tensor(0.4719, device='cuda:0')]
    valid: [0.52043818359375, tensor(0.8313, device='cuda:0')]
    train: [0.2481643359375, tensor(0.9217, device='cuda:0')]
    valid: [0.151099365234375, tensor(0.9540, device='cuda:0')]
    CPU times: user 2.67 s, sys: 429 ms, total: 3.1 s
    Wall time: 2.99 s

## Batchnorm

### Custom

Let's start by building our own `BatchNorm` layer from scratch.

We compute the mean and the variance with the method `update_stats()`. Just to confuse things, J.H. calls $\gamma$ and $\beta$ `self.mults` (the thing that multiplies) and `self.adds` (the thing that adds). Why, oh why...

`self.mults` and `self.adds` are initialized in `__init__()` as `nn.Parameters` of, respectively, ones and zeros (neutral for multiplication and addition). `self.adds` is conceptually identical to *bias*.

There are a couple of things we have to keep track of: the first one is what happens at inference time. We want to normalize at training time, but if we normalize at inference time, we may be removing the interesting stuff (**Unclear - clarify**).

While we are training, we keep an exponentially moving average of the means and the variances, stored in `self.means.lerp_` and `self.vars.lerp_`. These are running averages of the last few batches. Then, at inference time, we don't use the current mean and variance, but rather these running averages.

In `__init__` we don't assign the mean and variance to `self` but rather we do:

```python
self.register_buffer('vars', torch.ones(1, nf, 1, 1))
self.register_buffer('means', torch.zeros(1, nf, 1, 1))
```

`register_buffer()` has two advantages: if we move the model to the GPU, anything registered with `register_buffer()` will also be moved to the GPU. The other nice thing is that the mean and the variance are part of the model. When we save the model, `register_buffer()` also forces the registered quantities to be saved.

When we compute the mean and the variance we averages over axes `(0, 2, 3)`, *i.e.*, over all the batches, and over all the x and y coordinates. We are therefore left only for a mean for each channel.

Why do we use an exponential moving average and not a standard moving average? In a standard moving average we would take a subset of the points, average, move one point to the right, average and so on. The problem is that in real models we can have hundreds of millions of parameters, and this becomes very expensive. Exponential Moving Averages use a nifty trick, of adding one value at a time, but applying the formula

$$\mu_{t} = \beta \mu_{t-1} + (1 - \beta) x_{t}$$

With this trick, the influence of the previous points decays exponentially. The formula above is a *linear interpolation* between $\mu_{t-1}$ and $x_t$, and in PyTorch, linear interpolation is obtained with the method `lerp()`. Typically, the *momentum* in an Exponential Moving Average would be $\beta \approx 0.9$, but `lerp()` uses the opposite, and we have to pass, as a momentum, $1 - \beta$. This is also how `nn.BatchNorm` works **check the code**.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6018)

```python
class BatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        # NB: pytorch bn mom is opposite of what you'd expect
        self.mom = mom
        self.eps = eps
        self.mults = nn.Parameter(torch.ones(nf, 1, 1))
        self.adds = nn.Parameter(torch.zeros(nf, 1, 1))
        self.register_buffer('vars',  torch.ones(1, nf, 1, 1))
        self.register_buffer('means', torch.zeros(1, nf, 1, 1))

    def update_stats(self, x):
        m = x.mean((0, 2, 3), keepdim=True)
        v = x.var((0, 2, 3), keepdim=True)
        self.means.lerp_(m, self.mom)
        self.vars.lerp_(v, self.mom)
        return m, v

    def forward(self, x):
        if self.training:
            with torch.no_grad():
                m, v = self.update_stats(x)
        else:
            m, v = self.means, self.vars
        x = (x-m) / (v + self.eps).sqrt()
        return x * self.mults + self.adds
```

We create a new `conv_layer()` function that can optionally add a `BatchNorm` layer. If we add a `BatchNorm` layer, we remove the bias because the `self.adds` is already a bias.

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    # No bias needed if using bn
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn:
        layers.append(BatchNorm(nf))
    return nn.Sequential(*layers)
```

`init_cnn_` is a *recursive* initialization that will go through all the parameters.

```python
#export
def init_cnn_(m, f):
    if isinstance(m, nn.Conv2d):
        f(m.weight, a=0.1)
        if getattr(m, 'bias', None) is not None:
            m.bias.data.zero_()
    for l in m.children():
        init_cnn_(l, f)

def init_cnn(m, uniform=False):
    f = init.kaiming_uniform_ if uniform else init.kaiming_normal_
    init_cnn_(m, f)

def get_learn_run(nfs, data, lr, layer, cbs=None, opt_func=None,
                  uniform=False, **kwargs):
    model = get_cnn_model(data, nfs, layer, **kwargs)
    init_cnn(model, uniform=uniform)
    return get_runner(model, data, lr=lr, cbs=cbs, opt_func=opt_func)
```

We can then use it in training and see how it helps keep the activations means to 0 and the std to 1.

```python
learn, run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs)
```

```python
with Hooks(learn.model, append_stats) as hooks:
    run.fit(1, learn)
    fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(10, 4))
    for h in hooks[:-1]:
        ms, ss = h.stats
        ax0.plot(ms[:10])
        ax1.plot(ss[:10])
        h.remove()
    plt.legend(range(6))

    fig, (ax0, ax1) = plt.subplots(1, 2, figsize=(10, 4))
    for h in hooks[:-1]:
        ms, ss = h.stats
        ax0.plot(ms)
        ax1.plot(ss)
```

Our training has gotten rid of all the exponential peaks and sudden crashes. There is something happening at the end of training, and we don't know what it is. Now we can push the learning rate of 1.0 and we have 97%.

```python
learn, run = get_learn_run(nfs, data, 1.0, conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.28963826171875, tensor(0.9103, device='cuda:0')]
    valid: [0.161962158203125, tensor(0.9515, device='cuda:0')]
    train: [0.093538662109375, tensor(0.9708, device='cuda:0')]
    valid: [0.15296573486328124, tensor(0.9544, device='cuda:0')]
    train: [0.0665977490234375, tensor(0.9794, device='cuda:0')]
    valid: [0.125709228515625, tensor(0.9580, device='cuda:0')]
    CPU times: user 2.38 s, sys: 178 ms, total: 2.55 s
    Wall time: 2.56 s

### Builtin batchnorm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6679)

Now that we have built our own `BatchNorm`, we can use PyTorch's `nn.BatchNorm` and see that we obtain pretty much the same results.

```python
# export
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn:
        layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn, run = get_learn_run(nfs, data, 1., conv_layer, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [0.26512009765625, tensor(0.9196, device='cuda:0')]
    valid: [0.10897271728515626, tensor(0.9678, device='cuda:0')]
    train: [0.0747459765625, tensor(0.9766, device='cuda:0')]
    valid: [0.087364794921875, tensor(0.9732, device='cuda:0')]
    train: [0.0520605029296875, tensor(0.9838, device='cuda:0')]
    valid: [0.0929110595703125, tensor(0.9706, device='cuda:0')]
    CPU times: user 2.16 s, sys: 218 ms, total: 2.38 s
    Wall time: 2.39 s

### With scheduler

Now let's add the usual warm-up/annealing, and push the learning rate up to 2.0, and we get almost at 99%.

```python
sched = combine_scheds([0.3, 0.7], [sched_lin(0.6, 2.), sched_lin(2., 0.1)])
```

```python
learn, run = get_learn_run(nfs, data, 0.9, conv_layer, cbs=cbfs
                           + [partial(ParamScheduler, 'lr', sched)])
```

```python
run.fit(8, learn)
```

    train: [0.32977328125, tensor(0.9017, device='cuda:0')]
    valid: [0.1280703369140625, tensor(0.9607, device='cuda:0')]
    train: [0.09645861328125, tensor(0.9697, device='cuda:0')]
    valid: [0.08916947021484375, tensor(0.9716, device='cuda:0')]
    train: [0.06404841796875, tensor(0.9801, device='cuda:0')]
    valid: [0.07270582275390625, tensor(0.9768, device='cuda:0')]
    train: [0.038807666015625, tensor(0.9879, device='cuda:0')]
    valid: [0.06498637084960937, tensor(0.9822, device='cuda:0')]
    train: [0.0254615283203125, tensor(0.9919, device='cuda:0')]
    valid: [0.05287564086914062, tensor(0.9853, device='cuda:0')]
    train: [0.01662402099609375, tensor(0.9952, device='cuda:0')]
    valid: [0.042873516845703126, tensor(0.9882, device='cuda:0')]
    train: [0.010115330810546875, tensor(0.9975, device='cuda:0')]
    valid: [0.04411027526855469, tensor(0.9872, device='cuda:0')]
    train: [0.006914703979492187, tensor(0.9987, device='cuda:0')]
    valid: [0.04352261962890625, tensor(0.9877, device='cuda:0')]

## More norms

### Layer norm

From [the paper](https://arxiv.org/abs/1607.06450): "*batch normalization cannot be applied to online learning tasks or to extremely large distributed models where the minibatches have to be small*".

If we have a batch size of 1 the variance is infinite, so batch-norm does not makes sense. If we have a segmentation task, with batch size of 2, we also have a problem, because at some point two values will be very similar, and we will be dividing by a variance or almost zero, so we will have unstable or impossible training.

It's also hard for RNNs (**I haven't understood the explanation of why it is a problem**). The paper above suggests the solution to these problems.

General equation for a norm layer with learnable affine:

$$y = \frac{x - \mathrm{E}[x]}{ \sqrt{\mathrm{Var}[x] + \epsilon}} * \gamma + \beta$$

The difference with BatchNorm is
1. we don't keep a moving average
2. we don't average over the batches dimension but over the hidden dimension, so it's independent of the batch size

In other words we get rid of all the Exponential Moving Averages, and we average over (1, 2, 3) rather than (0, 2, 3). Every image has its own mean and its own standard deviations, we don't average over the batches any more, but we do average over the channels.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=6717)

```python
class LayerNorm(nn.Module):
    __constants__ = ['eps']

    def __init__(self, eps=1e-5):
        super().__init__()
        self.eps = eps
        self.mult = nn.Parameter(tensor(1.))
        self.add = nn.Parameter(tensor(0.))

    def forward(self, x):
        m = x.mean((1, 2, 3), keepdim=True)
        v = x.var((1, 2, 3), keepdim=True)
        x = (x-m) / ((v+self.eps).sqrt())
        return x*self.mult + self.add
```

```python
def conv_ln(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn:
        layers.append(LayerNorm())
    return nn.Sequential(*layers)
```

The problem is that if we train, even at a lower learning rate, it doesn't work. We don't have a different normalization for each channel, and therefore we treat things as if they were the same, but they are not. In other words, `LayerNorm` can help for images, but it's not nearly as good as `BatchNorm`. For RNNs, however, this is what you have to use.

```python
learn, run = get_learn_run(nfs, data, 0.8, conv_ln, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.1118, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 2.74 s, sys: 241 ms, total: 2.98 s
    Wall time: 3 s

*Thought experiment*: can this distinguish foggy days from sunny days (assuming you're using it before the first conv)?

Foggy days will have 1) less activations on average as they are less bright, 2) less contrast, i.e. less average. `LayerNorm` will normalize the means to be the same and the variances to be the same, so now they sunny day picture and the hazy day picture will have the same amount of activation and of contrast, so the answer to the question is no. If you put it anywhere in the middle layers, the answer is the same.

### Instance norm

From [the paper](https://arxiv.org/abs/1607.08022):

The key difference between **contrast** and batch normalization is that the latter applies the normalization to a  whole batch of images instead for single ones:

\begin{equation}
\label{eq:bnorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{i}}{\sqrt{\sigma_i^2 + \epsilon}},
    \quad
    \mu_i = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_i^2 = \frac{1}{HWT}\sum_{t=1}^T\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_i)^2.
\end{equation}

In order to combine the effects of instance-specific normalization and batch normalization, we propose to replace the latter by the *instance normalization* (also known as *contrast normalization*) layer:

\begin{equation}
\label{eq:inorm}
    y_{tijk} =  \frac{x_{tijk} - \mu_{ti}}{\sqrt{\sigma_{ti}^2 + \epsilon}},
    \quad
    \mu_{ti} = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H x_{tilm},
    \quad
    \sigma_{ti}^2 = \frac{1}{HW}\sum_{l=1}^W \sum_{m=1}^H (x_{tilm} - mu_{ti})^2.
\end{equation}

So `InstanceNorm` averages over the indices (2, 3)

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7114)

```python
class InstanceNorm(nn.Module):
    __constants__ = ['eps']

    def __init__(self, nf, eps=1e-0):
        super().__init__()
        self.eps = eps
        self.mults = nn.Parameter(torch.ones(nf, 1, 1))
        self.adds = nn.Parameter(torch.zeros(nf, 1, 1))

    def forward(self, x):
        m = x.mean((2, 3), keepdim=True)
        v = x.var((2, 3), keepdim=True)
        res = (x-m) / ((v + self.eps).sqrt())
        return res*self.mults + self.adds
```

```python
def conv_in(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=True),
              GeneralRelu(**kwargs)]
    if bn:
        layers.append(InstanceNorm(nf))
    return nn.Sequential(*layers)
```

`InstanceNorm` even at a learning rate of 0.1 doesn't learn anything at all. Why? Because we are averaging over every channell and every image and throwing away anything that can distinguish them.

`InstanceNorm` was not designed for classification, but rather for style transfer where the authors guessed that the difference in contrast and overall amount were not important and could be removed. So there are many types of normalization, but you need to understand what each one of them does.

```python
learn, run = get_learn_run(nfs, data, 0.1, conv_in, cbs=cbfs)
```

```python
%time run.fit(3, learn)
```

    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    train: [nan, tensor(0.0986, device='cuda:0')]
    valid: [nan, tensor(0.0991, device='cuda:0')]
    CPU times: user 2.67 s, sys: 279 ms, total: 2.94 s
    Wall time: 2.95 s

*Question*: why can't this classify anything?

Lost in all those norms? The authors from the [group norm paper](https://arxiv.org/pdf/1803.08494.pdf) show the differences between the various types of normalization. Note that in `GroupNorm` the normalization arbitrarily operates on a group of channels.

### Group norm

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7213)

*From the PyTorch docs:*

`GroupNorm(num_groups, num_channels, eps=1e-5, affine=True)`

The input channels are separated into `num_groups` groups, each containing
``num_channels / num_groups`` channels. The mean and standard-deviation are calculated
separately over the each group. $\gamma$ and $\beta$ are learnable
per-channel affine transform parameter vectorss of size `num_channels` if
`affine` is ``True``.

This layer uses statistics computed from input data in both training and
evaluation modes.

Args:
-    num_groups (int): number of groups to separate the channels into
-    num_channels (int): number of channels expected in input
-    eps: a value added to the denominator for numerical stability. Default: 1e-5
-    affine: a boolean value that when set to ``True``, this module
        has learnable per-channel affine parameters initialized to ones (for weights)
        and zeros (for biases). Default: ``True``.

Shape:
- Input: `(N, num_channels, *)`
- Output: `(N, num_channels, *)` (same shape as input)

Examples::

    >>> input = torch.randn(20, 6, 10, 10)
    >>> # Separate 6 channels into 3 groups
    >>> m = nn.GroupNorm(3, 6)
    >>> # Separate 6 channels into 6 groups (equivalent with InstanceNorm)
    >>> m = nn.GroupNorm(6, 6)
    >>> # Put all 6 channels into a single group (equivalent with LayerNorm)
    >>> m = nn.GroupNorm(1, 6)
    >>> # Activating the module
    >>> output = m(input)

## Fix small batch sizes

There are a number of different approaches to normalization, but we have seen none of them is as good as `BatchNorm`. We don't know how to fix the problem for RNNs, but we may know how to fix the problem for small batches.

### What's the problem?

When we compute the statistics (mean and std) for a BatchNorm Layer on a small batch, it is possible that we get a standard deviation very close to 0. because there aren't many samples (the variance of one thing is 0. since it's equal to its mean).

We create a new `DataBunch` with a batch size of 2. If we run it we see it takes a long time, which makes sense. There is a lot of GPU overhead, so much that it may be faster on the CPU. The performance is also awful, because of the small batch size.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7304)

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 2), c)
```

```python
def conv_layer(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn:
        layers.append(nn.BatchNorm2d(nf, eps=1e-5, momentum=0.1))
    return nn.Sequential(*layers)
```

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_layer, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [2.33933921875, tensor(0.1750, device='cuda:0')]
    valid: [93125.184, tensor(0.1929, device='cuda:0')]
    CPU times: user 49.9 s, sys: 532 ms, total: 50.5 s
    Wall time: 50.8 s

### Running Batch Norm

In our `BatchNorm` above we included `self.eps` to avoid divisions by zero. But what if, instead of a fixed value, we *learn* it? If instead of 1e-5 we use `self.eps = 0.01`, even in the worst case, we would be multiplying by 10 (mind the square root) so it wouldn't easily blow up.

An even better solution is to introduce a Running BatchNorm that uses smoother running mean and variance for the mean and std. The idea is not to subtract the batch mean and divide by the batch standard deviation, but rather use the moving average statistics at training time as well, and not just at training time. According to J.H., even if you happen to have a very small variance, which will happen with small sample sizes, this will have a much smaller impact.

There are a number of details:

1. In `BatchNorm` we take the running average of the variance. J.H. says that you cannot take the average of variances. He uses the $E[X^2] - E^2[X]$ formula for this. **TODO: LOOK AT THE CODE**.
2. The batch size may vary so they are also taking a moving average of the batch sizes.
3. We need to do *debiasing*: we want to make sure that no observation is weighted too highly. The first point gets too much weight. You initialize both `sums` and `sqrs` to zero, then you do the `lerp()`. We have an exponentially debiased amount and we divide by the debiased amount.
4. If your first minibatch is really close to zero, that will ruin everything, so there is an additional guard rail that says "clamp the first 20 values to 0.01.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=7516)

```python
class RunningBatchNorm(nn.Module):
    def __init__(self, nf, mom=0.1, eps=1e-5):
        super().__init__()
        self.mom = mom
        self.eps = eps
        self.mults = nn.Parameter(torch.ones(nf, 1, 1))
        self.adds = nn.Parameter(torch.zeros(nf, 1, 1))
        self.register_buffer('sums', torch.zeros(1, nf, 1, 1))
        self.register_buffer('sqrs', torch.zeros(1, nf, 1, 1))
        self.register_buffer('batch', tensor(0.))
        self.register_buffer('count', tensor(0.))
        self.register_buffer('step', tensor(0.))
        self.register_buffer('dbias', tensor(0.))

    def update_stats(self, x):
        bs, nc, *_ = x.shape
        self.sums.detach_()
        self.sqrs.detach_()
        dims = (0, 2, 3)
        s = x.sum(dims, keepdim=True)
        ss = (x*x).sum(dims, keepdim=True)
        c = self.count.new_tensor(x.numel() / nc)
        mom1 = 1 - (1 - self.mom) / math.sqrt(bs-1)
        self.mom1 = self.dbias.new_tensor(mom1)
        self.sums.lerp_(s, self.mom1)
        self.sqrs.lerp_(ss, self.mom1)
        self.count.lerp_(c, self.mom1)
        self.dbias = self.dbias * (1 - self.mom1) + self.mom1
        self.batch += bs
        self.step += 1

    def forward(self, x):
        if self.training:
            self.update_stats(x)
        sums = self.sums
        sqrs = self.sqrs
        c = self.count
        if self.step < 100:
            sums = sums / self.dbias
            sqrs = sqrs / self.dbias
            c = c / self.dbias
        means = sums/c
        vars = (sqrs/c).sub_(means*means)
        if bool(self.batch < 20):
            vars.clamp_min_(0.01)
        x = (x-means).div_((vars.add_(self.eps)).sqrt())
        return x.mul_(self.mults).add_(self.adds)
```

```python
def conv_rbn(ni, nf, ks=3, stride=2, bn=True, **kwargs):
    layers = [nn.Conv2d(ni, nf, ks, padding=ks//2, stride=stride, bias=not bn),
              GeneralRelu(**kwargs)]
    if bn:
        layers.append(RunningBatchNorm(nf))
    return nn.Sequential(*layers)
```

```python
learn, run = get_learn_run(nfs, data, 0.4, conv_rbn, cbs=cbfs)
```

```python
%time run.fit(1, learn)
```

    train: [1.4375453125, tensor(0.5554, device='cuda:0')]
    valid: [10234328104776.5, tensor(0.7252, device='cuda:0')]
    CPU times: user 2min 1s, sys: 1.21 s, total: 2min 3s
    Wall time: 2min 4s

This solves the small batch size issue!

### What can we do in a single epoch?

What is the best accuracy you can get *in a single epoch*?
Now let's see with a decent batch size what result we can get.

[Jump_to lesson 10 video](https://course.fast.ai/videos/?lesson=10&t=8068)

```python
data = DataBunch(*get_dls(train_ds, valid_ds, 32), c)
```

```python
learn, run = get_learn_run(nfs, data, 0.9, conv_rbn, cbs=cbfs
                           + [partial(ParamScheduler, 'lr', sched_lin(1., 0.2))])
```

```python
%time run.fit(1, learn)
```

    train: [0.158750654296875, tensor(0.9526, device='cuda:0')]
    valid: [0.080205859375, tensor(0.9765, device='cuda:0')]
    CPU times: user 7.97 s, sys: 92.2 ms, total: 8.06 s
    Wall time: 8.15 s

## Simplified RunningBatchNorm

Stas Bekman noticed that we don't actually need to debias in the calculation above. In the code in this snippet

```python
sums = sums / self.dbias
sqrs = sqrs / self.dbias
c = c / self.dbias
```

there is no point in dividing by `self.dbias`. This means that we can remove all the debiasing and end up with something simpler.
Tom Viehmann also found a way to simplify these calculations.

**Important**: the simplified version is not on GitHub, I don't know why. Someone asked this same question [on the forum](https://forums.fast.ai/t/simplified-runningbatchnorm-in-lesson-11/56340).

## Export

```python
nb_auto_export()
```

    <IPython.core.display.Javascript object>

```python

```


# OpenGGCM_GM

The kamodofied OpenGGCM_GM reader requires `netCDF4` (and numpy, scipy, and pandas, but those are required by kamodo)

    pip install netCDF4

```python
#import block
%load_ext autoreload
%autoreload 2
from plotly.offline import init_notebook_mode, iplot, plot
init_notebook_mode(connected = True)
import numpy as np
from kamodo.readers.openggcm_gm_4Dcdf_xarray import MODEL, model_varnames
from kamodo.readers.reader_kplots import plot2D
from time import perf_counter
t0=perf_counter()
reader = MODEL()
print(perf_counter()-t0, 's')
```

    0.0008022000000003082 s

```python
#print possible variable names for given model
print('Standardized variables:\n---------------------------')
for key in model_varnames.keys():
    print(model_varnames[key][0]+':', model_varnames[key][1:])
```

    Standardized variables:
    ---------------------------
    B_x: ['x component of magnetic field', 0, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'nT']
    B_y: ['y component of magnetic field', 1, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'nT']
    B_z: ['z component of magnetic field', 2, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'nT']
    B1_x: ['x component of magnetic field (on grid cell faces)', 3, 'GSE', 'car', ['time', 'x', 'x', 'x'], 'nT']
    B1_y: ['y component of magnetic field (on grid cell faces)', 4, 'GSE', 'car', ['time', 'y', 'y', 'y'], 'nT']
    B1_z: ['z component of magnetic field (on grid cell faces)', 5, 'GSE', 'car', ['time', 'z', 'z', 'z'], 'nT']
    E_x: ['x component of electric field (on grid cell edges)', 6, 'GSE', 'car', ['time', 'x', 'x', 'x'], 'mV/m']
    E_y: ['y component of electric field (on grid cell edges)', 7, 'GSE', 'car', ['time', 'y', 'y', 'y'], 'mV/m']
    E_z: ['z component of electric field (on grid cell edges)', 8, 'GSE', 'car', ['time', 'z', 'z', 'z'], 'mV/m']
    V_x: ['x component of plasma velocity', 9, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'km/s']
    V_y: ['y component of plasma velocity', 10, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'km/s']
    V_z: ['z component of plasma velocity', 11, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'km/s']
    N_plasma: ['plasma number denstity (hydrogen equivalent)', 12, 'GSE', 'car', ['time', 'x', 'y', 'z'], '1/cm**3']
    eta: ['resistivity', 13, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'm**2/s']
    P_plasma: ['plasma pressure', 14, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'pPa']
    J_x: ['x component of current density', 15, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'muA/m**2']
    J_y: ['y component of current density', 16, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'muA/m**2']
    J_z: ['z component of current density', 17, 'GSE', 'car', ['time', 'x', 'y', 'z'], 'muA/m**2']

```python
#set filename, run_name, desired variable list, and whether gridded interpolators are desired
run_name =  "Yihua_Zheng_090721_1"
var_list, gridded_int = ['N_plasma','B_x','B1_x','E_x','V_x','eta'], True  #
var_3D = ''  #for testing of expected plot error in last cell, should be a 3D variable name (if possible)
```

```python
#set variables to be plotted
if gridded_int:  #set list of variables to check plots, list should include at least one 3D and at least one 4D variable
    plot_list = list(np.ravel([[var, var+'_ijk'] for var in var_list]))
else:
    plot_list = var_list
print(plot_list)
```

    ['N_plasma', 'N_plasma_ijk', 'B_x', 'B_x_ijk', 'B1_x', 'B1_x_ijk', 'E_x', 'E_x_ijk', 'V_x', 'V_x_ijk', 'eta', 'eta_ijk']

```python
file_prefix = 'D:/OpenGGCM_GM/Data/Yihua_1/Yihua_Zheng_090721_1.3df_2015-10-16_12'
tic = perf_counter()
kamodo_object = reader(file_prefix, runname=run_name)  #load all variables
#kamodo_object = reader(file_prefix, variables_requested=['N_plasma'], runname=run_name,
#                       gridded_int=True)  #load specified variables with gridded interpolators
print(perf_counter()-tic, 's for ', len(kamodo_object.variables.keys()), ' variables')
```

    3.4202832 s for  24  variables

```python
#show functionalized variables in kamodo object
kamodo_object
```

\begin{equation}\operatorname{B_{x}}{\left(\vec{x} \right)}[nT] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{B_{x ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[nT] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{B_{y}}{\left(\vec{x} \right)}[nT] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{B_{y ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[nT] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{B_{z}}{\left(\vec{x} \right)}[nT] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{B_{z ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[nT] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{V_{x}}{\left(\vec{x} \right)}[km / s] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{V_{x ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[km / s] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{V_{y}}{\left(\vec{x} \right)}[km / s] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{V_{y ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[km / s] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{V_{z}}{\left(\vec{x} \right)}[km / s] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{V_{z ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[km / s] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{J_{x}}{\left(\vec{x} \right)}[\frac{muA}{m^{2}}] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{J_{x ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[\frac{muA}{m^{2}}] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{J_{y}}{\left(\vec{x} \right)}[\frac{muA}{m^{2}}] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{J_{y ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[\frac{muA}{m^{2}}] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{J_{z}}{\left(\vec{x} \right)}[\frac{muA}{m^{2}}] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{J_{z ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[\frac{muA}{m^{2}}] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{N_{plasma}}{\left(\vec{x} \right)}[\frac{1}{cm^{3}}] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{N_{plasma ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[\frac{1}{cm^{3}}] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\eta{\left(\vec{x} \right)}[m^{2} / s] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\eta_{ijk}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[m^{2} / s] = \lambda{\left(time,x,y,z \right)}\end{equation} \begin{equation}\operatorname{P_{plasma}}{\left(\vec{x} \right)}[pPa] = \lambda{\left(\vec{x} \right)}\end{equation} \begin{equation}\operatorname{P_{plasma ijk}}(time[hr],x[R_{E}],y[R_{E}],z[R_{E}])[pPa] = \lambda{\left(time,x,y,z \right)}\end{equation}

```python
kamodo_object._time.min(), kamodo_object._time.max()
```

    (12.000021, 13.00001)

```python
kamodo_object._x.min(), kamodo_object._x.max()
```

    (-350.0119, 33.01)

```python
kamodo_object._y.min(), kamodo_object._y.max()
```

    (-96.01, 96.01)

```python
kamodo_object._z.min(), kamodo_object._z.max()
```

    (-96.01, 96.01)

```python
#show variable coordinate dependencies (especially for non-gridded variables)
for key in kamodo_object.variables.keys(): print(key, kamodo_object.variables[key]['xvec'])
```

    B_x {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    B_x_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    B_y {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    B_y_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    B_z {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    B_z_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    V_x {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    V_x_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    V_y {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    V_y_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    V_z {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    V_z_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    J_x {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    J_x_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    J_y {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    J_y_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    J_z {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    J_z_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    N_plasma {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    N_plasma_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    eta {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    eta_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    P_plasma {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}
    P_plasma_ijk {'time': 'hr', 'x': 'R_E', 'y': 'R_E', 'z': 'R_E'}

```python
kamodo_object.N_plasma.data
```

<svg style="position: absolute; width: 0; height: 0; overflow: hidden">
<defs>
<symbol id="icon-database" viewBox="0 0 32 32">
<path d="M16 0c-8.837 0-16 2.239-16 5v4c0 2.761 7.163 5 16 5s16-2.239 16-5v-4c0-2.761-7.163-5-16-5z"></path>
<path d="M16 17c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
<path d="M16 26c-8.837 0-16-2.239-16-5v6c0 2.761 7.163 5 16 5s16-2.239 16-5v-6c0 2.761-7.163 5-16 5z"></path>
</symbol>
<symbol id="icon-file-text2" viewBox="0 0 32 32">
<path d="M28.681 7.159c-0.694-0.947-1.662-2.053-2.724-3.116s-2.169-2.030-3.116-2.724c-1.612-1.182-2.393-1.319-2.841-1.319h-15.5c-1.378 0-2.5 1.121-2.5 2.5v27c0 1.378 1.122 2.5 2.5 2.5h23c1.378 0 2.5-1.122 2.5-2.5v-19.5c0-0.448-0.137-1.23-1.319-2.841zM24.543 5.457c0.959 0.959 1.712 1.825 2.268 2.543h-4.811v-4.811c0.718 0.556 1.584 1.309 2.543 2.268zM28 29.5c0 0.271-0.229 0.5-0.5 0.5h-23c-0.271 0-0.5-0.229-0.5-0.5v-27c0-0.271 0.229-0.5 0.5-0.5 0 0 15.499-0 15.5 0v7c0 0.552 0.448 1 1 1h7v19.5z"></path>
<path d="M23 26h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 22h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
<path d="M23 18h-14c-0.552 0-1-0.448-1-1s0.448-1 1-1h14c0.552 0 1 0.448 1 1s-0.448 1-1 1z"></path>
</symbol>
</defs>
</svg>

<xarray.DataArray 'rr' (time: 61, x: 656, y: 190, z: 190)>
dask.array<open_dataset-f10c69aac6d38ea0bbd6ef18e787ac85rr, shape=(61, 656, 190, 190), dtype=float32, chunksize=(61, 100, 100, 100), chunktype=numpy.ndarray>
Coordinates:
  * time     (time) float32 12.0 12.02 12.03 12.05 ... 12.95 12.97 12.98 13.0
  * x        (x) float32 -350.0 -348.4 -346.8 -345.2 ... 32.72 32.82 32.91 33.01
  * y        (y) float32 -96.01 -93.07 -90.2 -87.39 ... 87.39 90.2 93.07 96.01
  * z        (z) float32 -96.01 -93.07 -90.2 -87.39 ... 87.39 90.2 93.07 96.01
Attributes:
    units:    1/cm**3

xarray.DataArray

'rr'

time: 61

x: 656

y: 190

z: 190

<input id='section-249d9f5a-6403-4acf-88c4-b1fd190683fc' class='xr-array-in' type='checkbox' checked><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>
dask.array<chunksize=(61, 100, 100, 100), meta=np.ndarray>

                         Array
                         Chunk

                         Bytes
                         5.38 GiB
                         232.70 MiB

                         Shape
                         (61, 656, 190, 190)
                         (61, 100, 100, 100)

                         Count
                         29 Tasks
                         28 Chunks

                     Type
                     float32
                     numpy.ndarray

        <svg width="380" height="164" style="stroke:rgb(0,0,0);stroke-width:1" >

  <line x1="0" y1="0" x2="38" y2="0" style="stroke-width:2" />
  <line x1="0" y1="25" x2="38" y2="25" style="stroke-width:2" />

  <line x1="0" y1="0" x2="0" y2="25" style="stroke-width:2" />
  <line x1="38" y1="0" x2="38" y2="25" style="stroke-width:2" />

  <polygon points="0.0,0.0 38.19014606538945,0.0 38.19014606538945,25.412616514582485 0.0,25.412616514582485" style="fill:#ECB172A0;stroke-width:0"/>

  <text x="19.095073" y="45.412617" font-size="1.0rem" font-weight="100" text-anchor="middle" >61</text>
  <text x="58.190146" y="12.706308" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(0,58.190146,12.706308)">1</text>

  <line x1="108" y1="0" x2="178" y2="70" style="stroke-width:2" />
  <line x1="108" y1="22" x2="178" y2="93" />
  <line x1="108" y1="43" x2="178" y2="114" style="stroke-width:2" />

  <line x1="108" y1="0" x2="108" y2="43" style="stroke-width:2" />
  <line x1="118" y1="10" x2="118" y2="54" />
  <line x1="129" y1="21" x2="129" y2="64" />
  <line x1="140" y1="32" x2="140" y2="75" />
  <line x1="151" y1="43" x2="151" y2="86" />
  <line x1="161" y1="53" x2="161" y2="97" />
  <line x1="172" y1="64" x2="172" y2="107" />
  <line x1="178" y1="70" x2="178" y2="114" style="stroke-width:2" />

  <polygon points="108.0,0.0 178.58823529411765,70.58823529411765 178.58823529411765,114.01401521559819 108.0,43.42577992148055" style="fill:#ECB172A0;stroke-width:0"/>

  <line x1="108" y1="0" x2="151" y2="0" style="stroke-width:2" />
  <line x1="118" y1="10" x2="162" y2="10" />
  <line x1="129" y1="21" x2="172" y2="21" />
  <line x1="140" y1="32" x2="183" y2="32" />
  <line x1="151" y1="43" x2="194" y2="43" />
  <line x1="161" y1="53" x2="205" y2="53" />
  <line x1="172" y1="64" x2="215" y2="64" />
  <line x1="178" y1="70" x2="222" y2="70" style="stroke-width:2" />

  <line x1="108" y1="0" x2="178" y2="70" style="stroke-width:2" />
  <line x1="130" y1="0" x2="201" y2="70" />
  <line x1="151" y1="0" x2="222" y2="70" style="stroke-width:2" />

  <polygon points="108.0,0.0 151.42577992148054,0.0 222.0140152155982,70.58823529411765 178.58823529411765,70.58823529411765" style="fill:#ECB172A0;stroke-width:0"/>

  <line x1="178" y1="70" x2="222" y2="70" style="stroke-width:2" />
  <line x1="178" y1="93" x2="222" y2="93" />
  <line x1="178" y1="114" x2="222" y2="114" style="stroke-width:2" />

  <line x1="178" y1="70" x2="178" y2="114" style="stroke-width:2" />
  <line x1="201" y1="70" x2="201" y2="114" />
  <line x1="222" y1="70" x2="222" y2="114" style="stroke-width:2" />

  <polygon points="178.58823529411765,70.58823529411765 222.0140152155982,70.58823529411765 222.0140152155982,114.01401521559819 178.58823529411765,114.01401521559819" style="fill:#ECB172A0;stroke-width:0"/>

  <text x="200.301125" y="134.014015" font-size="1.0rem" font-weight="100" text-anchor="middle" >190</text>
  <text x="242.014015" y="92.301125" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(-90,242.014015,92.301125)">190</text>
  <text x="133.294118" y="98.719898" font-size="1.0rem" font-weight="100" text-anchor="middle" transform="rotate(45,133.294118,98.719898)">656</text>
</svg>

<input id='section-12d27bba-8085-4893-8b54-37b97a4a3e83' class='xr-section-summary-in' type='checkbox'  checked>Coordinates: (4)

time

(time)

float32

12.0 12.02 12.03 ... 12.98 13.0
<input id='attrs-d5af0b1d-b104-4928-9ad0-b9b7b601ff8a' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-46820e6a-fcc3-457b-9ee4-76adc70b329e' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([12.000021, 12.016686, 12.033352, 12.050017, 12.066677, 12.083367,
       12.100009, 12.116673, 12.133335, 12.150019, 12.166687, 12.183351,
       12.200014, 12.216678, 12.233337, 12.250031, 12.266703, 12.283367,
       12.300031, 12.316694, 12.333359, 12.350022, 12.366686, 12.383349,
       12.400013, 12.416677, 12.43334 , 12.450032, 12.466687, 12.483342,
       12.500036, 12.516691, 12.533346, 12.550001, 12.566671, 12.583369,
       12.600029, 12.616689, 12.633348, 12.650007, 12.666667, 12.683368,
       12.700031, 12.716679, 12.733337, 12.750032, 12.766691, 12.783349,
       12.800005, 12.816669, 12.833358, 12.850017, 12.866675, 12.883342,
       12.900029, 12.916687, 12.933346, 12.950012, 12.966695, 12.983348,
       13.00001 ], dtype=float32)

x

(x)

float32

-350.0 -348.4 ... 32.91 33.01
<input id='attrs-9c523b01-d6d8-400c-a210-d14bcb379dfc' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-723d60ee-4c9e-4087-a981-b08e45a010ad' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([-350.0119  , -348.40955 , -346.80716 , ...,   32.815464,   32.912773,
         33.01    ], dtype=float32)

y

(y)

float32

-96.01 -93.07 -90.2 ... 93.07 96.01
<input id='attrs-dc6d0765-c490-49a5-aebf-cb4d93c9fa96' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-a1801715-6505-4c30-a4e3-49e60034ba93' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([-96.01    , -93.0725  , -90.197945, -87.38569 , -84.635056, -81.94539 ,
       -79.31604 , -76.74634 , -74.235634, -71.78326 , -69.388565, -67.05089 ,
       -64.76957 , -62.543953, -60.37338 , -58.257195, -56.194733, -54.185345,
       -52.228363, -50.323135, -48.469   , -46.665302, -44.911377, -43.206566,
       -41.550205, -39.941635, -38.38019 , -36.865208, -35.396023, -33.97196 ,
       -32.59235 , -31.256525, -29.963804, -28.713509, -27.504953, -26.337452,
       -25.210306, -24.122822, -23.074287, -22.063993, -21.091215, -20.155222,
       -19.25527 , -18.39061 , -17.560472, -16.76408 , -16.000635, -15.269327,
       -14.569327, -13.89979 , -13.259845, -12.648605, -12.065161, -11.50858 ,
       -10.977909, -10.472171,  -9.990368,  -9.531482,  -9.094475,  -8.678293,
        -8.281869,  -7.904123,  -7.54397 ,  -7.20032 ,  -6.872088,  -6.558195,
        -6.257576,  -5.969182,  -5.691991,  -5.42501 ,  -5.167277,  -4.917874,
        -4.675923,  -4.440594,  -4.211107,  -3.986734,  -3.766799,  -3.550684,
        -3.337823,  -3.127704,  -2.919869,  -2.713911,  -2.509474,  -2.306248,
        -2.103966,  -1.902405,  -1.70138 ,  -1.500738,  -1.300361,  -1.100157,
        -0.900057,  -0.700016,  -0.500003,  -0.3     ,  -0.1     ,   0.1     ,
         0.3     ,   0.500003,   0.700016,   0.900057,   1.100157,   1.300361,
         1.500738,   1.70138 ,   1.902405,   2.103966,   2.306248,   2.509474,
         2.713911,   2.919869,   3.127704,   3.337823,   3.550684,   3.766799,
         3.986734,   4.211107,   4.440594,   4.675923,   4.917874,   5.167277,
         5.42501 ,   5.691991,   5.969182,   6.257576,   6.558195,   6.872088,
         7.20032 ,   7.54397 ,   7.904123,   8.281869,   8.678293,   9.094475,
         9.531482,   9.990368,  10.472171,  10.977909,  11.50858 ,  12.065161,
        12.648605,  13.259845,  13.89979 ,  14.569327,  15.269327,  16.000635,
        16.76408 ,  17.560472,  18.39061 ,  19.25527 ,  20.155222,  21.091215,
        22.063993,  23.074287,  24.122822,  25.210306,  26.337452,  27.504953,
        28.713509,  29.963804,  31.256525,  32.59235 ,  33.97196 ,  35.396023,
        36.865208,  38.38019 ,  39.941635,  41.550205,  43.206566,  44.911377,
        46.665302,  48.469   ,  50.323135,  52.228363,  54.185345,  56.194733,
        58.257195,  60.37338 ,  62.543953,  64.76957 ,  67.05089 ,  69.388565,
        71.78326 ,  74.235634,  76.74634 ,  79.31604 ,  81.94539 ,  84.635056,
        87.38569 ,  90.197945,  93.0725  ,  96.01    ], dtype=float32)

z

(z)

float32

-96.01 -93.07 -90.2 ... 93.07 96.01
<input id='attrs-60970d50-5961-489a-8773-f489b117993c' class='xr-var-attrs-in' type='checkbox' disabled><svg class='icon xr-icon-file-text2'><use xlink:href='#icon-file-text2'></use></svg><input id='data-30da3fce-7705-42b8-9563-1953e2117c83' class='xr-var-data-in' type='checkbox'><svg class='icon xr-icon-database'><use xlink:href='#icon-database'></use></svg>

array([-96.01    , -93.0725  , -90.197945, -87.38569 , -84.635056, -81.94539 ,
       -79.31604 , -76.74634 , -74.235634, -71.78326 , -69.388565, -67.05089 ,
       -64.76957 , -62.543953, -60.37338 , -58.257195, -56.194733, -54.185345,
       -52.228363, -50.323135, -48.469   , -46.665302, -44.911377, -43.206566,
       -41.550205, -39.941635, -38.38019 , -36.865208, -35.396023, -33.97196 ,
       -32.59235 , -31.256525, -29.963804, -28.713509, -27.504953, -26.337452,
       -25.210306, -24.122822, -23.074287, -22.063993, -21.091215, -20.155222,
       -19.25527 , -18.39061 , -17.560472, -16.76408 , -16.000635, -15.269327,
       -14.569327, -13.89979 , -13.259845, -12.648605, -12.065161, -11.50858 ,
       -10.977909, -10.472171,  -9.990368,  -9.531482,  -9.094475,  -8.678293,
        -8.281869,  -7.904123,  -7.54397 ,  -7.20032 ,  -6.872088,  -6.558195,
        -6.257576,  -5.969182,  -5.691991,  -5.42501 ,  -5.167277,  -4.917874,
        -4.675923,  -4.440594,  -4.211107,  -3.986734,  -3.766799,  -3.550684,
        -3.337823,  -3.127704,  -2.919869,  -2.713911,  -2.509474,  -2.306248,
        -2.103966,  -1.902405,  -1.70138 ,  -1.500738,  -1.300361,  -1.100157,
        -0.900057,  -0.700016,  -0.500003,  -0.3     ,  -0.1     ,   0.1     ,
         0.3     ,   0.500003,   0.700016,   0.900057,   1.100157,   1.300361,
         1.500738,   1.70138 ,   1.902405,   2.103966,   2.306248,   2.509474,
         2.713911,   2.919869,   3.127704,   3.337823,   3.550684,   3.766799,
         3.986734,   4.211107,   4.440594,   4.675923,   4.917874,   5.167277,
         5.42501 ,   5.691991,   5.969182,   6.257576,   6.558195,   6.872088,
         7.20032 ,   7.54397 ,   7.904123,   8.281869,   8.678293,   9.094475,
         9.531482,   9.990368,  10.472171,  10.977909,  11.50858 ,  12.065161,
        12.648605,  13.259845,  13.89979 ,  14.569327,  15.269327,  16.000635,
        16.76408 ,  17.560472,  18.39061 ,  19.25527 ,  20.155222,  21.091215,
        22.063993,  23.074287,  24.122822,  25.210306,  26.337452,  27.504953,
        28.713509,  29.963804,  31.256525,  32.59235 ,  33.97196 ,  35.396023,
        36.865208,  38.38019 ,  39.941635,  41.550205,  43.206566,  44.911377,
        46.665302,  48.469   ,  50.323135,  52.228363,  54.185345,  56.194733,
        58.257195,  60.37338 ,  62.543953,  64.76957 ,  67.05089 ,  69.388565,
        71.78326 ,  74.235634,  76.74634 ,  79.31604 ,  81.94539 ,  84.635056,
        87.38569 ,  90.197945,  93.0725  ,  96.01    ], dtype=float32)

<input id='section-0a410b2f-fe26-411b-8fbb-85db26555695' class='xr-section-summary-in' type='checkbox'  checked>Attributes: (1)

units :

1/cm**3

```python
kamodo_object.N_plasma.data.interp(time=12.8,x=5.,y=0.,z=0.).values
```

    array(2.60609573)

```python
kamodo_object.N_plasma([[12.8,5.,0.,0.]])
```

    Took 0.019 s for 1 positions.

    array([2.60609573])

```python
kamodo_object.N_plasma_ijk(time=12.8,x=5.,y=0.,z=0.)
```

    array(2.60609573)

```python
#show documentation for 2D plotting function
help(plot2D)
```

    Help on function plot2D in module kamodo.readers.reader_kplots:

    plot2D(kamodo_object, varname, plottype, t, lon, lat, h=-1)
        Use Kamodo's native plotting to generate 2D plot.
        t, lon, lat, and h also double as t, x, y, and z for cartesian inputs.
        Possible plot types are LonLat, LatH, LonH, TimeLat, TimeLon, and TimeH for
            spherical coordinates; and TimeX, TimeY, TimeX, XY, XZ, and YZ for
            cartesian coordinates.
        If a LonLat plot is requested, then the function expects a single value
            (integer, float, float32, or float64) for t and h (if h is given).
            In this case, lon and lat should be 1D arrays or flat lists.
        If the variable depends on height, then a value or array should be given for h.

```python
#generate each possible type of 2D plot for each variable in plot_list
xval, xarr = 0., np.linspace(-15.,15.,20)
yval, yarr = 0., np.linspace(-15.,15.,20)
zval, zarr = 2., np.linspace(-20.,20.,20)
timeval, timearr = 12.5, np.linspace(12.,13.,20)
for plot_var in plot_list:
    if plot_var not in kamodo_object.variables:
        print(plot_var)
        continue
    print(kamodo_object.variables[plot_var]['xvec'].keys())
    print('---------------------------------------------------------------')
    print(f'--------------  Plots for {plot_var}  -----------------------')
    print('---------------------------------------------------------------')
    if len(list(kamodo_object.variables[plot_var]['xvec'].keys()))==4:  #if height depedent
        #make 2D plots for variable
        iplot(plot2D(kamodo_object, plot_var,'XY',timeval,xarr,yarr,zval))
        iplot(plot2D(kamodo_object, plot_var,'TimeY',timearr,xval,yarr,zval))
        iplot(plot2D(kamodo_object, plot_var,'TimeX',timearr,xarr,yval,zval))
        iplot(plot2D(kamodo_object, plot_var,'XZ',timeval,xarr,yval,zarr))
        iplot(plot2D(kamodo_object, plot_var,'YZ',timeval,xval,yarr,zarr))
        iplot(plot2D(kamodo_object, plot_var,'TimeZ',timearr,xval,yval,zarr))
```

```python
#Show proper errors when user tries to plot a time+2D variable against height
if var_3D in kamodo_object.variables.keys():
    try:
        iplot(plot2D(kamodo_object, var_3D,'LonH',timeval,lonarr,latval,height_arr))
    except AttributeError as e:
        print(e)
    try:
        iplot(plot2D(kamodo_object, var_3D,'LatH',timeval,lonval,latarr,height_arr))
    except AttributeError as e:
        print(e)
    try:
        iplot(plot2D(kamodo_object, var_3D,'TimeH',timearr,lonval,latval,height_arr))
    except AttributeError as e:
        print(e)
else:
    print('No 3D variable given.')
```

    No 3D variable given.

```python
kamodo_object.datetimes
```

    ['2015-10-16 12:00:00.075531', '2015-10-16 13:00:00.034332']

```python
print('Inner boundary radius:', kamodo_object.near_Earth_boundary_radius,
                  kamodo_object.near_Earth_boundary_radius_unit)  #should be 2.5 R_E
```

    Inner boundary radius: 2.5 R_E

```python
kamodo_object.filetimes
```

```python

```


# Tutorial 1: Linear regression with MSE
**Week 1, Day 3: Model Fitting**

**By Neuromatch Academy**

**Content creators**: Pierre-Étienne Fiquet, Anqi Wu, Alex Hyafil with help from Byron Galbraith

**Content reviewers**: Lina Teichmann, Saeed Salehi, Patrick Mineault,  Ella Batty, Michael Waskom

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

___
# Tutorial Objectives

*Estimated timing of tutorial: 30 minutes*

This is Tutorial 1 of a series on fitting models to data. We start with simple linear regression, using least squares optimization (Tutorial 1) and Maximum Likelihood Estimation (Tutorial 2). We will use bootstrapping to build confidence intervals around the inferred linear model parameters (Tutorial 3). We'll finish our exploration of regression models by generalizing to multiple linear regression and polynomial regression (Tutorial 4). We end by learning how to choose between these various models. We discuss the bias-variance trade-off (Tutorial 5) and Cross Validation for model selection (Tutorial 6).

In this tutorial, we will learn how to fit simple linear models to data.
- Learn how to calculate the mean-squared error (MSE)
- Explore how model parameters (slope) influence the MSE
- Learn how to find the optimal model parameter using least-squares optimization

---

**acknowledgements:**
- we thank Eero Simoncelli, much of today's tutorials are inspired by exercises asigned in his mathtools class.

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/2mkq4/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

```python
import numpy as np
import matplotlib.pyplot as plt
```

```python
#@title Figure Settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/course-content/master/nma.mplstyle")
```

```python
#@title Plotting Functions

def plot_observed_vs_predicted(x, y, y_hat, theta_hat):
  """ Plot observed vs predicted data

  Args:
    x (ndarray): observed x values
    y (ndarray): observed y values
    y_hat (ndarray): predicted y values
    theta_hat (ndarray):
  """
  fig, ax = plt.subplots()
  ax.scatter(x, y, label='Observed')  # our data scatter plot
  ax.plot(x, y_hat, color='r', label='Fit')  # our estimated model
  # plot residuals
  ymin = np.minimum(y, y_hat)
  ymax = np.maximum(y, y_hat)
  ax.vlines(x, ymin, ymax, 'g', alpha=0.5, label='Residuals')
  ax.set(
      title=fr"$\hat{{\theta}}$ = {theta_hat:0.2f}, MSE = {mse(x, y, theta_hat):.2f}",
      xlabel='x',
      ylabel='y'
  )
  ax.legend()
```

---
# Section 1: Mean Squared Error (MSE)

```python
# @title Video 1: Linear Regression & Mean Squared Error
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
      def __init__(self, id, page=1, width=400, height=300, **kwargs):
          self.id=id
          src = 'https://player.bilibili.com/player.html?bvid={0}&page={1}'.format(id, page)
          super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id="BV1tA411e7NW", width=854, height=480, fs=1)
  print('Video available at https://www.bilibili.com/video/{0}'.format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id="HumajfjJ37E", width=854, height=480, fs=1, rel=0)
  print('Video available at https://youtube.com/watch?v=' + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

This video covers a 1D linear regression and mean squared error.

 Click here for text recap of video

**Linear least squares regression** is an old but gold  optimization procedure that we are going to use for data fitting. Least squares (LS) optimization problems are those in which the objective function is a quadratic function of the
parameter(s) being optimized.

Suppose you have a set of measurements: for each data point or measurement, you have $y_{i}$ (the "dependent" variable) obtained for a different input value, $x_{i}$ (the "independent" or "explanatory" variable).  Suppose we believe the measurements are proportional to the input values, but are corrupted by some (random) measurement errors, $\epsilon_{i}$, that is:

\begin{align}
y_{i}= \theta x_{i}+\epsilon_{i}
\end{align}

for some unknown slope parameter $\theta.$ The least squares regression problem uses **mean squared error (MSE)** as its objective function, it aims to find the value of the parameter $\theta$ by minimizing the average of squared errors:

\begin{align}
\min _{\theta} \frac{1}{N}\sum_{i=1}^{N}\left(y_{i}-\theta x_{i}\right)^{2}
\end{align}

We will now explore how MSE is used in fitting a linear regression model to data. For illustrative purposes, we will create a simple synthetic dataset where we know the true underlying model. This will allow us to see how our estimation efforts compare in uncovering the real model (though in practice we rarely have this luxury).

First we will generate some noisy samples $x$ from [0, 10) along the line $y = 1.2x$ as our dataset we wish to fit a model to.

```python
# @title

# @markdown Execute this cell to generate some simulated data

# setting a fixed seed to our random number generator ensures we will always
# get the same psuedorandom number sequence
np.random.seed(121)

# Let's set some parameters
theta = 1.2
n_samples = 30

# Draw x and then calculate y
x = 10 * np.random.rand(n_samples)  # sample from a uniform distribution over [0,10)
noise = np.random.randn(n_samples)  # sample from a standard normal distribution
y = theta * x + noise

# Plot the results
fig, ax = plt.subplots()
ax.scatter(x, y)  # produces a scatter plot
ax.set(xlabel='x', ylabel='y');
```

Now that we have our suitably noisy dataset, we can start trying to estimate the underlying model that produced it. We use MSE to evaluate how successful a particular slope estimate $\hat{\theta}$ is for explaining the data, with the closer to 0 the MSE is, the better our estimate fits the data.

## Coding Exercise 1: Compute MSE

In this exercise you will implement a method to compute the mean squared error for a set of inputs $\mathbf{x}$, measurements $\mathbf{y}$, and slope estimate $\hat{\theta}$. Here, $\mathbf{x}$ and $\mathbf{y}$ are vectors of data points. We will then compute and print the mean squared error for 3 different choices of theta.

As a reminder, the equation for computing the estimated y for a single data point is:

$$\hat{y}_{i}= \theta x_{i}$$

and for mean squared error is:

\begin{align}
\min _{\theta} \frac{1}{N}\sum_{i=1}^{N}\left(y_{i}-\hat{y}_i\right)^{2}
\end{align}

```python
def mse(x, y, theta_hat):
  """Compute the mean squared error

  Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.
    theta_hat (float): An estimate of the slope parameter

  Returns:
    float: The mean squared error of the data with the estimated parameter.
  """
  ####################################################
  ## TODO for students: compute the mean squared error
  # Fill out function and remove
  ####################################################

  # Compute the estimated y
  y_hat = theta_hat * x

  # Compute mean squared error

  mse = np.mean((y - y_hat)**2)

  return mse

theta_hats = [0.75, 1.0, 1.5]
for theta_hat in theta_hats:
  print(f"theta_hat of {theta_hat} has an MSE of {mse(x, y, theta_hat):.2f}")
```

    theta_hat of 0.75 has an MSE of 9.08
    theta_hat of 1.0 has an MSE of 3.01
    theta_hat of 1.5 has an MSE of 4.52

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial1_Solution_12a57de0.py)

The result should be:

theta_hat of 0.75 has an MSE of 9.08\
theta_hat of 1.0 has an MSE of 3.0\
theta_hat of 1.5 has an MSE of 4.52

We see that $\hat{\theta} = 1.0$ is our best estimate from the three we tried. Looking just at the raw numbers, however, isn't always satisfying, so let's visualize what our estimated model looks like over the data.

```python
#@title

#@markdown Execute this cell to visualize estimated models

fig, axes = plt.subplots(ncols=3, figsize=(18, 4))
for theta_hat, ax in zip(theta_hats, axes):

  # True data
  ax.scatter(x, y, label='Observed')  # our data scatter plot

  # Compute and plot predictions
  y_hat = theta_hat * x
  ax.plot(x, y_hat, color='r', label='Fit')  # our estimated model

  ax.set(
      title= fr'$\hat{{\theta}}$= {theta_hat}, MSE = {mse(x, y, theta_hat):.2f}',
      xlabel='x',
      ylabel='y'
  );

axes[0].legend()
```

## Interactive Demo 1: MSE Explorer

Using an interactive widget, we can easily see how changing our slope estimate changes our model fit. We display the **residuals**, the differences between observed and predicted data, as line segments between the data point (observed response) and the corresponding predicted response on the model fit line.

- What value of $\hat{\theta}$ results in the lowest MSE?
- Is this a good way of estimating $\theta$?

```python
#@title

#@markdown Make sure you execute this cell to enable the widget!

@widgets.interact(theta_hat=widgets.FloatSlider(1.0, min=0.0, max=2.0))
def plot_data_estimate(theta_hat):
  y_hat = theta_hat * x
  plot_observed_vs_predicted(x, y, y_hat, theta_hat)
```

    interactive(children=(FloatSlider(value=1.0, description='theta_hat', max=2.0), Output()), _dom_classes=('widg…

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial1_Solution_d5c4912a.py)

While visually exploring several estimates can be instructive, it's not the most efficient for finding the best estimate to fit our data. Another technique we can use is choose a reasonable range of parameter values and compute the MSE at several values in that interval. This allows us to plot the error against the parameter value (this is also called an **error landscape**, especially when we deal with more than one parameter). We can select the final $\hat{\theta}$  ($\hat{\theta}_\textrm{MSE}$) as the one which results in the lowest error.

```python
# @title

# @markdown Execute this cell to loop over theta_hats, compute MSE, and plot results

# Loop over different thetas, compute MSE for each
theta_hat_grid = np.linspace(-2.0, 4.0)
errors = np.zeros(len(theta_hat_grid))
for i, theta_hat in enumerate(theta_hat_grid):
  errors[i] = mse(x, y, theta_hat)

# Find theta that results in lowest error
best_error = np.min(errors)
theta_hat = theta_hat_grid[np.argmin(errors)]

# Plot results
fig, ax = plt.subplots()
ax.plot(theta_hat_grid, errors, '-o', label='MSE', c='C1')
ax.axvline(theta, color='g', ls='--', label=r"$\theta_{True}$")
ax.axvline(theta_hat, color='r', ls='-', label=r"$\hat{{\theta}}_{MSE}$")
ax.set(
  title=fr"Best fit: $\hat{{\theta}}$ = {theta_hat:.2f}, MSE = {best_error:.2f}",
  xlabel=r"$\hat{{\theta}}$",
  ylabel='MSE')
ax.legend();
```

We can see that our best fit is $\hat{\theta}=1.18$ with an MSE of 1.45. This is quite close to the original true value $\theta=1.2$!

---
# Section 2: Least-squares optimization

*Estimated timing to here from start of tutorial: 20 min*

While the approach detailed above (computing MSE at various values of $\hat\theta$) quickly got us to a good estimate, it still relied on evaluating the MSE value across a grid of hand-specified values. If we didn't pick a good range to begin with, or with enough granularity, we might miss the best possible estimator. Let's go one step further, and instead of finding the minimum MSE from a set of candidate estimates, let's solve for it analytically.

We can do this by minimizing the cost function. Mean squared error is a convex objective function, therefore we can compute its minimum using calculus. Please see video or Bonus Section 1 for this derivation! After computing the minimum, we find that:

\begin{align}
\hat\theta = \frac{\mathbf{x}^\top \mathbf{y}}{\mathbf{x}^\top \mathbf{x}}
\end{align}
where $\mathbf{x}$ and $\mathbf{y}$ are vectors of data points.

This is known as solving the normal equations. For different ways of obtaining the solution, see the notes on [Least Squares Optimization](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf) by Eero Simoncelli.

## Coding Exercise 2: Solve for the Optimal Estimator

In this exercise, you will write a function that finds the optimal $\hat{\theta}$ value using the least squares optimization approach (the equation above) to solve MSE minimization. It shoud take arguments $x$ and $y$ and return the solution $\hat{\theta}$.

We will then use your function to compute $\hat{\theta}$ and plot the resulting prediction on top of the data.

```python
def solve_normal_eqn(x, y):
  """Solve the normal equations to produce the value of theta_hat that minimizes
    MSE.

    Args:
    x (ndarray): An array of shape (samples,) that contains the input values.
    y (ndarray): An array of shape (samples,) that contains the corresponding
      measurement values to the inputs.

  Returns:
    float: the value for theta_hat arrived from minimizing MSE
  """

  ################################################################################
  ## TODO for students: solve for the best parameter using least squares
  # Fill out function and remove
  #raise NotImplementedError("Student exercise: solve for theta_hat using least squares")
  ################################################################################

  # Compute theta_hat analytically
  theta_hat = (np.transpose(x) @ y)/(np.transpose(x) @ x)

  return theta_hat
print(len(x))
print(len(x.T))

theta_hat = solve_normal_eqn(x, y)
y_hat = theta_hat * x
plot_observed_vs_predicted(x, y, y_hat, theta_hat)
```

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content/tree/master//tutorials/W1D3_ModelFitting/solutions/W1D3_Tutorial1_Solution_7a89ba24.py)

*Example output:*

We see that the analytic solution produces an even better result than our grid search from before, producing $\hat{\theta} = 1.21$ with MSE = 1.43!

---
# Summary

*Estimated timing of tutorial: 30 minutes*

Linear least squares regression is an optimization procedure that can be used for data fitting:

  - Task: predict a value for $y_i$ given $x_i$
  - Performance measure: $\textrm{MSE}$
  - Procedure: minimize $\textrm{MSE}$ by solving the normal equations

**Key point**: We fit the model by defining an *objective function* and minimizing it.

**Note**: In this case, there is an *analytical* solution to the minimization problem and in practice, this solution can be computed using *linear algebra*. This is *extremely* powerful and forms the basis for much of numerical computation throughout the sciences.

# Notation
\begin{align}
x_{i} &\quad \text{input, independent variable}\\
y_{i} &\quad \text{measurement, dependent variable}\\
\mathbf{x} &\quad \text{vector of input values}\\
\mathbf{y} &\quad \text{vector of measurements}\\
\hat{y}_{i} &\quad \text{estimate of dependent variable}\\
\epsilon_{i} &\quad \text{measurement error}\\
\theta &\quad \text{slope parameter}\\
\hat{\theta} &\quad \text{estimated slope parameter}\\
\hat{\theta}_\text{MSE} &\quad \text{slope parameter estimated via the mean squared error}\\
\textrm{MSE} &\quad \text{mean squared error}\\
\end{align}

---
# Bonus

---
## Bonus Section 1: Least Squares Optimization Derivation

We will outline here the derivation of the least squares solution.

We first set the derivative of the error expression with respect to $\theta$ equal to zero,

\begin{align}
\frac{d}{d\theta}\frac{1}{N}\sum_{i=1}^N(y_i - \theta x_i)^2 = 0 \\
\frac{1}{N}\sum_{i=1}^N-2x_i(y_i - \theta x_i) = 0
\end{align}

where we used the chain rule. Now solving for $\theta$, we obtain an optimal value of:

\begin{align}
\hat\theta = \frac{\sum_{i=1}^N x_i y_i}{\sum_{i=1}^N x_i^2}
\end{align}

Which we can write in vector notation as:

\begin{align}
\hat\theta = \frac{\mathbf{x}^\top \mathbf{y}}{\mathbf{x}^\top \mathbf{x}}
\end{align}

This is known as solving the *normal equations*. For different ways of obtaining the solution, see the notes on [Least Squares Optimization](https://www.cns.nyu.edu/~eero/NOTES/leastSquares.pdf) by Eero Simoncelli.


\title{Continues HDL Sinewave Generator via Chebyshev Polynomial Approximation in Python's myHDL}
\author{Steven K Armour}
\maketitle

# Table of Contents

1  Python Libraries Utilized

2  Acknowledgments

3  Derivation of the Chebyshev Polynomial Approximation for Cos

3.1  Chebyshev Polynomials

3.1.1  Plot of Chebyshev Polynomials

3.2  Chebyshev Polynomial $\cos$ Relationship

3.3  Sinusoid Generator Algorithm Development

3.3.1  Future additions

4  Test Bench for Sinusoid Generator

5  Post Processing of Test Bench

6  RTL Synthisis

7  Conclusion

8  Refrances

# Python Libraries Utilized

```python
import numpy as np
import pandas as pd
from sympy import *
init_printing()
from IPython.display import display, Math, Latex

from myhdl import *
from myhdlpeek import Peeker

import matplotlib.pyplot as plt
%matplotlib inline
```

# Acknowledgments
The orgianl Chebyshev Sinwave Genrator written in myHDL was done by ["HARDSOFTLUCID"](https://hardsoftlucid.wordpress.com/various-stuff/myhdl/)
(myHDL.old version [here](https://github.com/jandecaluwe/site-myhdl-retired/blob/master/_ori/pages/projects/continuous_wave_sinusgenerator.txt))

Author of myHDL [Jan Decaluwe](http://www.myhdl.org/users/jandecaluwe.html) and the author of the myHDL Peeker [XESS Corp.](https://github.com/xesscorp/myhdlpeek)

# Derivation of the Chebyshev Polynomial Approximation for Cos

## Chebyshev Polynomials

We Start with recalling that the double(n) angle trig identity of $\cos$ for $n=2$ is
$$\cos(2\theta)= \cos(\theta)^2 -\sin(\theta)^2 = 2\cos(\theta)^2 -1$$
and for $n=3$ is
$$\cos(3\theta)= cos(\theta)^3 -3\sin(\theta)^2 \cos(\theta)=4\cos(\theta)^3 -3\cos(\theta)$$

Now exploiting Chebyshev polynomials that come from the power series solution($y(x)=\sum_{n=0}^{\infty} a_n x^n$) of  Chebyshev differential equation:
$$(1-x^2)y" -xy'+p^2y=0$$

The Power series solution takes on the form of a Recurrence relation for the $a_n$ term in the Power series as
$$a_{n+2}=\dfrac{(n-p)(n+p)}{(n+1)(n+2)}a_n$$
for $x\in [-1, 1]$
that leads to the Chebyshev polynomial defined as
$$T_0(x)=1$$
$$T_1(x)=x$$
$$T_{n+1}(x)=2xT_n(x)-T_{n-1}(x)$$

### Plot of Chebyshev Polynomials

```python
x=np.linspace(-1.0, 1.0)
fig=plt.figure()
ax=plt.subplot(111)
for i in range(1,8+1):
    coeff=[0]*i
    coeff[-1]=i
    y=np.polynomial.Chebyshev(coeff)(x)
    ax.plot(x, y, label=f'$T_{i-1}(x)$')
bbox_to_anchor=ax.get_position()
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
ax.grid()
plt.title(r"Chebyshev Polynomials $T_0(x)-T_1(x), x\in[-1, 1]$" )
None
```

## Chebyshev Polynomial $\cos$ Relationship

If now $T_n(x)=T_n(cos(\theta))=cos(n\theta)$ we have

$$T_0(\cos(0\cdot\theta))=1$$
$$T_1(\cos(1\cdot\theta))=\cos(\theta)$$
$$T_{n+1}(\cos(\theta))=2 \cos(\theta)T_n(\cos(\theta))-T_{n-1}(\cos(\theta))$$
$$\cos((n+1)\theta)=2\cos(\theta)\cos(n\theta)-\cos((n-1)\theta)$$

solving for $\cos(\theta)$ we get

```python
n, theta=symbols('n, theta')
LHS=cos(theta)
RHS=(cos((n+1)*theta)+cos((n-1)*theta))/(2*cos(n*theta))
Eq(LHS, RHS)
```

notice that the RHS can be simplified to

```python
simplify(RHS)
```

```python
#numericalize symbolic
RHSN=lambdify((n, theta), RHS, dummify=False)

fig=plt.figure()
ax=plt.subplot(111)

thetaN=np.linspace(0, 2*np.pi)
for N in range(1, 8+1):
    y=RHSN(N, thetaN)
    ax.plot(thetaN, y, label=f'$C_{N-1} aprox$')
ax.plot(thetaN, np.cos(thetaN), label=r'$cos(\theta)$')
ax.grid()

ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title(r"Plot of $\cos(\theta), \theta \in[0, 2\pi]$ & $N\in[0, 7]$ CP Approx.")
None
```

```python
thetaN=np.linspace(0, 2*np.pi)
for N in range(1, 8+1):
    y=np.cos(thetaN)-RHSN(N, thetaN)
    plt.plot(thetaN, y, label=f'$C_{N-1} error$')
plt.grid()
plt.ticklabel_format(style='sci', axis='y', scilimits=(0,0))

plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title(r"Plot of error of $\cos(\theta), \theta \in[0, 2\pi]$ & $N\in[0, 7]$ CP Aprox")

None
```

##  Sinusoid Generator Algorithm Development

now letting $\cos(\theta)=\cos(2*\pi f_{\cos}/f_{clk})=T_1(x)$ we can translate the recursion relationship for the Chebyshev polynomials into relationships between regestes calls as follows. Let $$T_{n+1}(x)=2xT_n(x)-T_{n-1}(x)$$ become
$$R_2=K \cdot R_1 -R_0$$
that is we we replace the $2x$ by a constant factor $K$ and utilize the subscripts as designations of our registers. Furthermore we know that after one call of our recursion relationship $R_2$ ($T_{n+1}(x))$) will now become our old value $R_0$ ($T_{n-1}(x)$) thus we have
$$R_0=R_2$$
$$R_2=K \cdot R_1 -R_0$$

further it can be shown that $R_2$ is just the next state value of the $R_1$ so that the above becomes

$$R_0=R_1$$
$$R_1'=K \cdot R_1 -R_0$$

where $'$ is used to indicate next state

And because the multiplication of two 30 bit numbers will generate a 60 bit number the result needs to be down shifted since the full 30 bits of the register are not being utilized to prevent overflow.

$$R_0=R_1$$
$$R_1'=((K \cdot R_1)>>(\text{size of}R_1 -1 )) -R_0$$

```python
def SinGenerator(SinFreq_parm, ClkFreq_parm, SinValue_out,
                clk, rst, ena):
    #contorl byte size and works with -1 to translate cos to sin
    INTERNALWIDTH=len(SinValue_out)-2

    #qunitited version of the 2x for cos(\theta)
    KONSTANT_FACTOR=int(np.cos(2*np.pi * SinFreq_parm /ClkFreq_parm)* 2**(INTERNALWIDTH))

    #prep the Needed regesters at sysnthis
    Reg_T0=Signal(intbv((2**(INTERNALWIDTH))-1,
                        min=SinValue_out.min, max=SinValue_out.max))
    Reg_T1=Signal(intbv(KONSTANT_FACTOR,
                        min=SinValue_out.min, max=SinValue_out.max))

    #define the polynomal logic
    @always(clk.posedge,rst.negedge)
    def logicCP():
        #clear and prep the regesters
        if rst== 0 :
            Reg_T0.next=(2**(INTERNALWIDTH))-1
            Reg_T1.next=KONSTANT_FACTOR

        #run a single recursion iterration of the polynomal
        else:
            if ena==1:
                # recursive Chebyshev  formulation for sinus waveform calculation
                Reg_T0.next=Reg_T1
                #>> shift is a overflow wrapper
                Reg_T1.next=((KONSTANT_FACTOR * Reg_T1)>>(INTERNALWIDTH-1)) - Reg_T0

    #pole the R1 for the value of the sin function
    @always_comb
    def comb_logic():
        SinValue_out.next=Reg_T1

    return instances()
```

### Future additions
need to add a Amp, Freq, phase modulation to this so as to convert the cos output to any sinusoidal output

# Test Bench for Sinusoid Generator

```python
SinFreq=0.75e6   # make a 1.45 mhz Sinus
clkFreq=10e6   # 10 mhz
clkPeriod=1.0/clkFreq

OUTPUT_BITWIDTH=30

Peeker.clear()

SinValue_out=Signal(intbv(0, min=-2**OUTPUT_BITWIDTH, max=2**OUTPUT_BITWIDTH))
Peeker(SinValue_out, 'SinVal')
SinValueTracker=[]

clk=Signal(bool(0)); Peeker(clk, 'clk')
ena=Signal(bool(0)); Peeker(ena, 'ena')
rst=Signal(bool(0)); Peeker(rst, 'rst')

DUT=SinGenerator(SinFreq_parm=SinFreq, ClkFreq_parm=clkFreq, SinValue_out=SinValue_out,
                clk=clk, rst=rst, ena=ena)

def SinGenerator_TB(TestClkCyc=200):

    #clock genrator
    @always(delay(int(clkPeriod*0.5*1e9)))  ## delay in nano seconds
    def clkGen():
        clk.next = not clk

    # accterla test procdure
    @instance
    def stimulus():
        while 1:
            rst.next=0
            ena.next=0
            #wait one clock cycle
            yield clk.posedge

            #test reset
            rst.next=1

            #wait one clock cycle
            yield clk.posedge

            #run the sin wave genrator
            ena.next=1
            #run the test for 200 clock cycles
            for i in range(TestClkCyc):
                #wait for next clock cycle
                yield clk.posedge
                SinValueTracker.append(int(SinValue_out))
            raise StopSimulation

    return instances()
```

!? Peeker is failing for some reason to capture all these values so having to improvice

```python
N=200
sim = Simulation(DUT, SinGenerator_TB(TestClkCyc=N), *Peeker.instances()).run()
#Peeker.to_wavedrom(start_time=0, stop_time=20, tock=True)
```

# Post Processing of Test Bench

```python
SinGenOutDF=pd.DataFrame(columns=['GenValue'], data=SinValueTracker)
SinGenOutDF['Time[s]']=np.arange(0.0,clkPeriod*(len(SinGenOutDF)-0.5),clkPeriod)
SinGenOutDF['GenValueNorm']=SinGenOutDF['GenValue']/SinGenOutDF['GenValue'].max()
SinGenOutDF['f[Hz]']=np.arange(-clkFreq/2.0,clkFreq/2.0,clkFreq/(len(SinValueTracker)))

FFT=np.fft.fftshift(np.fft.fft(SinGenOutDF['GenValueNorm']))
SinGenOutDF['FFTMag']=np.abs(FFT)
SinGenOutDF['FFTPhase']=np.angle(FFT)
SinGenOutDF.head(5)
```

      GenValue
      Time[s]
      GenValueNorm
      f[Hz]
      FFTMag
      FFTPhase

      0
      478355485
      0.000000e+00
      0.891007
      -5000000.0
      1.676381e-08
      -5.921190e-15

      1
      315564804
      1.000000e-07
      0.587785
      -4950000.0
      1.169079e-08
      8.481712e-02

      2
      83985113
      2.000000e-07
      0.156434
      -4900000.0
      1.374530e-08
      -2.588747e-02

      3
      -165902237
      3.000000e-07
      -0.309017
      -4850000.0
      1.256693e-08
      8.504632e-03

      4
      -379625064
      4.000000e-07
      -0.707107
      -4800000.0
      1.237123e-08
      3.107669e-01

```python
CosDF=pd.DataFrame(columns=['Time[s]'], data=np.arange(0.0,clkPeriod*(len(SinGenOutDF)-0.5),clkPeriod))
CosDF['Cos']=np.cos(2*np.pi*SinFreq*CosDF['Time[s]'])
CosDF['CosS']=CosDF['Cos']*SinGenOutDF['GenValue'].max()
CosDF['f[Hz]']=np.arange(-clkFreq/2.0,clkFreq/2.0,clkFreq/(len(SinValueTracker)))
FFT=np.fft.fftshift(np.fft.fft(CosDF['Cos']))
CosDF['FFTMag']=np.abs(FFT)
CosDF['FFTPhase']=np.angle(FFT)
CosDF.head(5)
```

      Time[s]
      Cos
      CosS
      f[Hz]
      FFTMag
      FFTPhase

      0
      0.000000e+00
      1.000000
      5.368709e+08
      -5000000.0
      4.768857e-15
      -3.141593

      1
      1.000000e-07
      0.891007
      4.783555e+08
      -4950000.0
      3.754675e-14
      -1.502735

      2
      2.000000e-07
      0.587785
      3.155648e+08
      -4900000.0
      3.920159e-14
      0.642098

      3
      3.000000e-07
      0.156434
      8.398512e+07
      -4850000.0
      4.962420e-14
      -0.653919

      4
      4.000000e-07
      -0.309017
      -1.659022e+08
      -4800000.0
      9.565805e-14
      1.207279

```python
fig, [ax0, ax1]=plt.subplots(nrows=2, ncols=1, sharex=False)
plt.suptitle(f'Plots of Sin Generator output in time for {N} Cycles')

SinGenOutDF.plot(use_index=True ,y='GenValue', ax=ax0)
CosDF.plot(use_index=True, y='CosS', ax=ax0)
ax0.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
ax0.legend(loc='best')

SinGenOutDF.plot(x='Time[s]', y='GenValueNorm', ax=ax1)
CosDF.plot(x='Time[s]', y='CosS', ax=ax1)
ax1.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
ax1.legend(loc='best')

None
```

```python
fig, [ax0, ax1]=plt.subplots(nrows=2, ncols=1, sharex=True)
plt.suptitle(f'Plots of Sin Generator output in freq for {N} Cycles')

SinGenOutDF.plot(x='f[Hz]' ,y='FFTMag', logy=True, ax=ax0, label='GenFFTMag')
CosDF.plot(x='f[Hz]' ,y='FFTMag', logy=True, ax=ax0, label='SinFFTMag')

ax0.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
ax0.set_ylabel('Amp [dB]')
ax0.legend(loc='best')

SinGenOutDF.plot(x='f[Hz]', y='FFTPhase', ax=ax1, label='GenFFTPhase')
CosDF.plot(x='f[Hz]', y='FFTPhase', ax=ax1, label='CosFFTPhase')
ax1.ticklabel_format(style='sci', axis='x', scilimits=(0,0))
ax1.set_xlabel('f[Hz]'); ax1.set_ylabel('Phase [rad]')

ax1.legend(loc='best')

None
```

# RTL Synthisis

```python
SinFreq=0.75e6   # make a 1.45 mhz Sinus
clkFreq=10e6   # 10 mhz
clkPeriod=1.0/clkFreq

OUTPUT_BITWIDTH=30

Peeker.clear()

SinValue_out=Signal(intbv(0, min=-2**OUTPUT_BITWIDTH, max=2**OUTPUT_BITWIDTH))
Peeker(SinValue_out, 'SinVal')
SinValueTracker=[]

clk=Signal(bool(0))
ena=Signal(bool(0))
rst=Signal(bool(0))

toVerilog(SinGenerator, SinFreq, clkFreq, SinValue_out, clk, rst, ena)
toVHDL(SinGenerator, SinFreq, clkFreq, SinValue_out, clk, rst, ena)
None
```

Running the lines
```
toVerilog(SinGenerator, SinFreq, clkFreq, SinValue_out, clk, rst, ena)
toVHDL(SinGenerator, SinFreq, clkFreq, SinValue_out, clk, rst, ena)
```
called  **myHDL**'s conversion process that converted the function **SinGenerator(SinFreq_parm, ClkFreq_parm, SinValue_out, clk, rst, ena)** and the signals _SinValue_out, clk, rst, ena_ to be converted and written to _SinGenerator.v_ and _SinGenerator.vhd_ respectively in the folder where this _Jupyter Notebook_ is located.

```python
#helper  functions to read in the .v and .vhd generated files into python
def VerilogTextReader(loc, printresult=True):
    with open(f'{loc}.v', 'r') as vText:
        VerilogText=vText.read()
    if printresult:
        print(f'***Verilog modual from {loc}.v***\n\n', VerilogText)
    return VerilogText

def VHDLTextReader(loc, printresult=True):
    with open(f'{loc}.vhd', 'r') as vText:
        VerilogText=vText.read()
    if printresult:
        print(f'***VHDL modual from {loc}.vhd***\n\n', VerilogText)
    return VerilogText
```

```python
_=VerilogTextReader('SinGenerator', True)
```

    ***Verilog modual from SinGenerator.v***

     // File: SinGenerator.v
    // Generated by MyHDL 0.9.0
    // Date: Wed Oct 18 23:47:53 2017

    `timescale 1ns/10ps

    module SinGenerator (
        SinValue_out,
        clk,
        rst,
        ena
    );

    output signed [30:0] SinValue_out;
    wire signed [30:0] SinValue_out;
    input clk;
    input rst;
    input ena;

    reg signed [30:0] Reg_T1;
    reg signed [30:0] Reg_T0;

    assign SinValue_out = Reg_T1;

    always @(posedge clk, negedge rst) begin: SINGENERATOR_LOGICCP
        if ((rst == 0)) begin
            Reg_T0 <= ((2 ** 29) - 1);
            Reg_T1 <= 478355485;
        end
        else begin
            if ((ena == 1)) begin
                Reg_T0 <= Reg_T1;
                Reg_T1 <= ($signed((478355485 * Reg_T1) >>> (29 - 1)) - Reg_T0);
            end
        end
    end

    endmodule

```python
_=VHDLTextReader('SinGenerator', True)
```

    ***VHDL modual from SinGenerator.vhd***

     -- File: SinGenerator.vhd
    -- Generated by MyHDL 0.9.0
    -- Date: Wed Oct 18 23:47:53 2017

    library IEEE;
    use IEEE.std_logic_1164.all;
    use IEEE.numeric_std.all;
    use std.textio.all;

    use work.pck_myhdl_090.all;

    entity SinGenerator is
        port (
            SinValue_out: out signed (30 downto 0);
            clk: in std_logic;
            rst: in std_logic;
            ena: in std_logic
        );
    end entity SinGenerator;

    architecture MyHDL of SinGenerator is

    constant INTERNALWIDTH: integer := 29;
    constant KONSTANT_FACTOR: integer := 478355485;

    signal Reg_T1: signed (30 downto 0);
    signal Reg_T0: signed (30 downto 0);

    begin

    SinValue_out <= Reg_T1;

    SINGENERATOR_LOGICCP: process (clk, rst) is
    begin
        if (rst = '0') then
            Reg_T0 <= to_signed((2 ** INTERNALWIDTH) - 1, 31);
            Reg_T1 <= to_signed(KONSTANT_FACTOR, 31);
        elsif rising_edge(clk) then
            if (ena = '1') then
                Reg_T0 <= Reg_T1;
                Reg_T1 <= resize(shift_right((KONSTANT_FACTOR * Reg_T1), (INTERNALWIDTH - 1)) - Reg_T0, 31);
            end if;
        end if;
    end process SINGENERATOR_LOGICCP;

    end architecture MyHDL;

The RTL Schematic from the verilog myHDL synthesis via vavado 2016.1 of the Sine Generator is shown below

The RTL Synthesis in **Xilinx**'s _Vivado 2016.1_ shows 65 cells, 34 I/O ports 161 Nets, 2 Register Sets, and 3 RTL Operations (multiply, right shift, subtraction). Where the last two statistics are exactly as predicted from the myHDL (Python) function **SinGenerator**

# Conclusion

We can see that by using Python’s myHDL library one can synthesize a working sin generator that can be converted to both Verilog and VHDL making myHDL HDL language agnostic. Furthermore, by conducting the test in python we plot the data and perform the subsequent analysis in the same environment as our HDL function thus allowing for rapid prototyping. And further, with the utilization of the Peeker extension library for myHDL, we can generate a timing diagram to compare from the FPGA synthesis tools to confirm our results. And finally, by utilizing the Jupyter notebook and git,  documentation from theoretical development through algorithm design and HDL synthesis can be kept in one easy to read the living digital document that can be shared with ease. Thus banishing obscene separation on code, documentation, and testing that has plagued HDL DSP developers in the past

# Refrances

https://en.wikibooks.org/wiki/Trigonometry/For_Enthusiasts/Chebyshev_Polynomials

https://www.geophysik.uni-muenchen.de/~igel/Lectures/NMG/05_orthogonalfunctions.pdf