In [Ch1_SymbolicTaylorSeries](Ch1_SymbolicTaylorSeries.ipynb) we saw how to use the Taylor remainder theorem to estimate error in an expansion, and this is useful if you want to calculate a function like sine or cosine using only additions/subtractions and multiplications/division.

Let's try that with the exponential function.  This is a classic example in numerical analysis

```
# Load the package
import sympy as sym
from sympy import init_printing
init_printing()  # This will make output look very nice!

import math  # has things like math.pi (which is the numerical value of pi)
```

```
# First, all symbolic variables must be declared. This is in contrast to Mathematica
x = sym.Symbol('x')

# Recall the Taylor series for exp(x)... very simple!
sym.series( sym.exp(-x), x )
```

Let's try to find $e^{-30}$ with about 14 digits of precision.  Recally that Taylor's remainder theorem says that the error in the Taylor series (with $n-1$ terms) is the $n$th term but evaluate the derivative at some $\xi \in [x_0,x]$.  We'll expand around $0$, so $x_0=0$ (we do this because we *know* that $e^0=1$; we could choose a different $x_0$ to expand about if we knew what $e^{x_0}$ is).

For this remainder term, the $n$th derivative of $e^{-x}$ is $(-1)^ne^{-x}$, and since our target $x=30$ is positive, we know $\xi \ge 0$ and hence $e^{-\xi} \le 1$, so we can bound this derivative term by $1$ (in absolute value).  We also need to include the coefficient $1/n!$ and the polynomial term $(x-x_0)^n$.

```
X   = -30

# Taylor's remainder theorem says that the remainder term at the (n-1) step is the nth Taylor series term but evaluated at some \xi
Rem = lambda n : abs( X**n / math.factorial(n) )
for n in range(15):
  print( Rem(n) )
  # When n is large enough, eventually this sequence will start to decrease
```

    1.0
    30.0
    450.0
    4500.0
    33750.0
    202500.0
    1012500.0
    4339285.714285715
    16272321.42857143
    54241071.428571425
    162723214.2857143
    443790584.41558444
    1109476461.038961
    2560330294.7052946
    5486422060.082774

So if we want 14 digits, let's calculate a bound on the number of Taylor series we need:

```
tol = 1.0e-14
n = 0
while Rem(n) > tol and n < 1e8 : # this last check is to prevent infinite loops, which are a possibility due to bugs
  n += 1
N = n-1
print( "We need at least", N, "terms, since Rem(n+1) is %.2e" % Rem(N+1) )
```

    We need at least 106 terms, since Rem(n+1) is 9.19e-15

Now, let's sum the Taylor series. We should have an error of 1e-14, right?

```
e = 0.
for n in range(N+1):
  e += X**n / math.factorial(n)
  if ( not (n % 10 ) ) or ( n > N-10 ):  # save some output...
    print("Iteration %3d"% n, " estimate is %29.15f"% e, "and error is %9.3e" % abs(e - math.exp(X)) )
```

    Iteration   0  estimate is             1.000000000000000 and error is 1.000e+00
    Iteration  10  estimate is     121254849.571428596973419 and error is 1.213e+08
    Iteration  20  estimate is   85291712210.032180786132812 and error is 8.529e+10
    Iteration  30  estimate is  384842612535.907775878906250 and error is 3.848e+11
    Iteration  40  estimate is   63336540117.266479492187500 and error is 6.334e+10
    Iteration  50  estimate is     878229229.277898311614990 and error is 8.782e+08
    Iteration  60  estimate is       1685584.300283375196159 and error is 1.686e+06
    Iteration  70  estimate is           622.524572911353516 and error is 6.225e+02
    Iteration  80  estimate is             0.055867710574729 and error is 5.587e-02
    Iteration  90  estimate is            -0.000084070677839 and error is 8.407e-05
    Iteration  97  estimate is            -0.000085530630269 and error is 8.553e-05
    Iteration  98  estimate is            -0.000085530022814 and error is 8.553e-05
    Iteration  99  estimate is            -0.000085530206891 and error is 8.553e-05
    Iteration 100  estimate is            -0.000085530151668 and error is 8.553e-05
    Iteration 101  estimate is            -0.000085530168071 and error is 8.553e-05
    Iteration 102  estimate is            -0.000085530163247 and error is 8.553e-05
    Iteration 103  estimate is            -0.000085530164652 and error is 8.553e-05
    Iteration 104  estimate is            -0.000085530164246 and error is 8.553e-05
    Iteration 105  estimate is            -0.000085530164362 and error is 8.553e-05
    Iteration 106  estimate is            -0.000085530164330 and error is 8.553e-05

... but our error is in fact much worse. What went wrong?  Our estimate of the error was due to our mathematical approximation of the exponential, but we didn't include numerical error due to the summation.  We are adding and subtracting a lot of numbers of similar sizes, so we loose precision.

Attempted fix #1: use better summation strategies.  Unfortunately, we'll see that this doesn't quite fix it for us.
- Fix 1a: use Python's builtin `fsum`
- Fix 1b: use [Kahan summation](https://en.wikipedia.org/wiki/Kahan_summation_algorithm), which is a trick that effectively let's us work with twice as many digits, or use another good summing algorithm.

---

```
# Fix 1a.
# See https://www.math.ubc.ca/~pwalls/math-python/python/sequences/
series = [ X**n / math.factorial(n) for n in range(N+1) ]  # "lazy", or "iterable". This is a list comprehension
e = math.fsum( series )
abs( e - math.exp(X) )
```

```
# Kahan Summation, though doesn't work (Python might be optimizing?)
e = 0.
accum = 0.
for n in range(N+1):
  a = X**n / math.factorial(n) - accum
  b = e + a
  accum = (b-e) -a
  e = b
  if n > N-4:
    print("Iteration %2d"% n, " estimate is %21.15f"% e, "and error is %9.3e" % abs(e - math.exp(X)) )
```

    Iteration 103  estimate is    -0.000085530164652 and error is 8.553e-05
    Iteration 104  estimate is    -0.000085530164246 and error is 8.553e-05
    Iteration 105  estimate is    -0.000085530164362 and error is 8.553e-05
    Iteration 106  estimate is    -0.000085530164330 and error is 8.553e-05

Attempted fix #2: use some math.
- Fix 2a: split into positive and negative terms, add those separately, then combine at the end. Good idea, but again, doesn't quite fix it.
- Fix 2b: rewrite $e^{-x} = \frac{1}{e^x} = \frac{1}{1+x+x^2/2+\ldots}$.  Finally, we have a good solution!

```
# Fix 2a
seriesEven = [ X**n / math.factorial(n) for n in range(0,N+1,2) ]
seriesOdd = [ X**n / math.factorial(n) for n in range(1,N+1,2) ]
e = math.fsum( seriesEven ) + math.fsum( seriesOdd )
abs( e - math.exp(X) )
```

```
# Fix 2b
series = [ abs(X)**n / math.factorial(n) for n in range(N+1) ]
e = 1./math.fsum( series )
abs( e - math.exp(X) )
```

```

```


---
# Convolutional Neural Networks

In this assignment, you will first implement the forward and backward pass of a convolution layer. Then you can freely design your own CNN model for image classification on CIFAR-10.

As in previous assignments, you will see code blocks that look like this:
```python
###############################################################################
# TODO: Create a variable x with value 536.                                   #
###############################################################################
# Replace "pass" statement with your code
pass
# END OF YOUR CODE
```

You should replace the `pass` statement with your own code and leave the blocks intact, like this:
```python
###############################################################################
# TODO: Create a variable x with value 536.                                   #
###############################################################################
# Replace "pass" statement with your code
x = 536
# END OF YOUR CODE
```

Also, please remember:
- Do not write or modify any code outside of code blocks unless otherwise stated.
- Do not delete any cells from the notebook. You may add new cells to perform scratch work, but delete them before submitting.
- Run all cells before submitting. You will only get credit for code that has been run.
- Submit your notebook as `netid.ipynb`, where `netid` is your actual netid.
- Your submission will be graded with PyTorch 1.7.0 and Python 3.6, which are the default versions in Google Colab.

```python
from __future__ import print_function
from __future__ import division

import math
import torch
import torch.nn as nn
import torch.nn.functional as F
import torch.optim as optim

from torch.utils.data import random_split, DataLoader
from torchvision import transforms
from torchvision.datasets import CIFAR10

import matplotlib.pyplot as plt

%matplotlib inline
plt.rcParams['figure.figsize'] = (10.0, 8.0)
plt.rcParams['image.interpolation'] = 'nearest'
plt.rcParams['image.cmap'] = 'gray'
```

```python
assert torch.cuda.is_available(), 'GPU unavailable'
```

## Convolution - Forward Pass

Let us first implement the forward pass of 2D convolution and get a better understanding of the operations it performs. We will be replicating the functionality of PyTorch's built-in 2D convolution: [`nn.Conv2d()`](https://pytorch.org/docs/1.7.0/generated/torch.nn.Conv2d.html#torch.nn.Conv2d) and [`F.conv2d()`](https://pytorch.org/docs/1.7.0/nn.functional.html#torch.nn.functional.conv2d). Please refer to the documentation for more details of the input arguments and return values. It is acceptable to use nested loops here, but you may also develop a vectorized implementation.

```python
def conv2d(input, weight, bias=None, stride=1, padding=0):
  """
  Apply 2D convolution over a batch of input tensors.

  Input arguments and return value are the same as those of F.conv2d(),
  except that we don't care about `dilation` and `groups` here.
  Also, `padding_mode` is `zeros`.
  """
  output = None
  #############################################################################
  # TODO: Implement the same functionality as F.conv2d(). You are NOT allowed #
  # to use F.conv2d() or nn.Conv2d().                                         #
  #############################################################################
  # Replace "pass" statement with your code
  input_shape = input.shape
  weight_shape = weight.shape
  n = input_shape[0]
  c = input_shape[1]
  h = input_shape[2]
  w = input_shape[3]
  nk = weight_shape[0]
  hk = weight_shape[2]
  wk = weight_shape[3]
  if type(stride) is tuple:
    stride1 = stride[0]
    stride2 = stride[1]
  else:
    stride1 = stride2 = stride
  if type(padding) is tuple:
    padding1 = padding[0]
    padding2 = padding[1]
  else:
    padding1 = padding2 = padding
  oh = int((h - hk + 2*padding1)/stride1) + 1
  ow = int((w - wk + 2*padding2)/stride2) + 1
  hnew = h + 2*padding1
  wnew = w + 2*padding2

  m = nn.ZeroPad2d((padding2, padding2, padding1, padding1))
  input_new = m(input)

  # rearrange index
  idx1 = torch.arange(hk).repeat_interleave(wk).reshape(1, -1) + torch.arange(0, hnew-hk+1, step = stride1).repeat_interleave(oh).reshape(-1, 1)
  idx2 = torch.arange(wk).repeat(hk).reshape(1, -1) + torch.arange(0, wnew-wk+1, step = stride2).repeat(ow).reshape(-1, 1)

  # reshape input and weight
  input_new = input_new.reshape(n, 1, c, hnew, wnew)[:, :, :, idx1, idx2]
  weight = weight.reshape(1, nk, c, 1, hk*wk)
  if bias == None:
    output = (input_new*weight).sum(dim = 4).sum(dim = 2).reshape(n, nk, oh, ow)
  else:
    output = (input_new*weight).sum(dim = 4).sum(dim = 2).reshape(n, nk, oh, ow) + bias.reshape(1, nk, 1, 1)
  # END OF YOUR CODE
  return output
```

We will use the following helper functions to check your results:

```python
def max_diff(actual, expected):
  return (actual - expected).abs().max().item()
```

```python
def check_conv2d(batch_size, in_channels, in_height, in_width,
                 out_channels, kernel_size, stride=1, padding=0, use_bias=True):

  if isinstance(kernel_size, int):
    kernel_size = (kernel_size, kernel_size)

  torch.manual_seed(0)
  input = torch.empty(batch_size, in_channels, in_height, in_width).normal_()
  weight = torch.empty(out_channels, in_channels, kernel_size[0], kernel_size[1]).normal_()
  if use_bias:
    bias = torch.empty(out_channels).normal_()
  else:
    bias = None

  actual = conv2d(input, weight, bias, stride, padding)
  expected = F.conv2d(input, weight, bias, stride, padding)
  assert actual.shape == expected.shape, 'incorrect shape'

  diff = max_diff(actual, expected)
  print('max_diff = %e' % diff)
  assert diff < 1e-3, 'incorrect result'
```

```python
# check `kernel_size`
check_conv2d(batch_size=8, in_channels=3, in_height=8, in_width=8,
             out_channels=8, kernel_size=3, stride=1, padding=0, use_bias=True)
```

    max_diff = 3.814697e-06

```python
# check `stride`
check_conv2d(batch_size=4, in_channels=8, in_height=16, in_width=16,
             out_channels=3, kernel_size=5, stride=3, padding=0, use_bias=True)
```

    max_diff = 1.525879e-05

```python
# check `padding`
check_conv2d(batch_size=1, in_channels=1, in_height=32, in_width=32,
             out_channels=4, kernel_size=8, stride=5, padding=3, use_bias=True)
```

    max_diff = 5.722046e-06

```python
# check tuple arguments
check_conv2d(batch_size=16, in_channels=4, in_height=4, in_width=8,
             out_channels=1, kernel_size=(1, 3), stride=(1, 2), padding=(0, 1), use_bias=False)
```

    max_diff = 9.536743e-07

## Convolution - Backward Pass

In order to train a CNN, we need to backpropagate the gradient through the convolution. More precisely, suppose we have performed a convolution:
$$
\text{output} = \text{conv2d}(\text{input}, \text{weight}, \text{bias}),
$$
and the loss is defined as some function of the output:
$$
\mathcal{L} = f(\text{output}).
$$
Note that $\text{output}, \text{input}, \text{weight}, \text{bias}$ are tensors, and $\mathcal{L}$ is a scalar.

We need the gradients $\frac{\partial \mathcal{L}}{\partial \text{weight}}$ and $\frac{\partial \mathcal{L}}{\partial \text{bias}}$, which can be computed by the chain rule:
$$
\begin{align}
\frac{\partial \mathcal{L}}{\partial \text{weight}_i} &= \sum_{k} \frac{\partial \mathcal{L}}{\partial \text{output}_k} \frac{\partial \text{output}_k}{\partial \text{weight}_i},\\
\frac{\partial \mathcal{L}}{\partial \text{bias}_j} &= \sum_{k} \frac{\partial \mathcal{L}}{\partial \text{output}_k} \frac{\partial \text{output}_k}{\partial \text{bias}_j}.
\end{align}
$$
If $\text{input}$ is computed by another neural network with parameters $\theta$, then to obtain $\frac{\partial \mathcal{L}}{\partial \theta}$, we also need $\frac{\partial \mathcal{L}}{\partial \text{input}}$.

Therefore, in the backward pass that we will implement below, given $\frac{\partial \mathcal{L}}{\partial \text{output}}$, we need to compute $\frac{\partial \mathcal{L}}{\partial \text{input}}$, $\frac{\partial \mathcal{L}}{\partial \text{weight}}$, and $\frac{\partial \mathcal{L}}{\partial \text{bias}}$.

You are NOT allowed to use PyTorch's automatic differentiation, but you may use other built-in functions (in particular, [`F.conv2d()`](https://pytorch.org/docs/1.7.0/nn.functional.html#torch.nn.functional.conv2d) and [`F.conv_transpose2d()`](https://pytorch.org/docs/1.7.0/nn.functional.html#torch.nn.functional.conv_transpose2d)) if they can simplify your implementation. For simplicity, you only need to consider the case where `stride = 1`. You will get bonus points if your implementation also works for `stride > 1`.

```python
def conv2d_grad(output_grad, input, weight, bias=None, stride=1, padding=0):
  """
  Backward pass of 2D convolution.

  Inputs:
  - output_grad: A PyTorch tensor giving the gradient of loss wrt `output`
    computed by the forward pass conv2d(). It has the same shape as `output`.
  - Other arguments are the same as those of conv2d().

  Returns:
  - input_grad: A PyTorch tensor giving the gradient of loss wrt `input`.
    It has the same shape as `input`.
  - weight_grad: A PyTorch tensor giving the gradient of loss wrt `weight`.
    It has the same shape as `weight`.
  - bias_grad: A PyTorch tensor giving the gradient of loss wrt `bias`.
    It has the same shape as `bias` if `bias` is not `None`.
    Otherwise, it is `None`.
  """
  input_grad = None
  weight_grad = None
  bias_grad = None
  #############################################################################
  # TODO: Implement the backward pass of conv2d(). You are NOT allowed to use #
  # PyTorch's automatic differentiation.                                      #
  #############################################################################
  # Replace "pass" statement with your code
  input_shape = input.shape
  weight_shape = weight.shape
  n = input_shape[0]
  c = input_shape[1]
  h = input_shape[2]
  w = input_shape[3]
  nk = weight_shape[0]
  hk = weight_shape[2]
  wk = weight_shape[3]
  if type(padding) is tuple:
    padding1 = padding[0]
    padding2 = padding[1]
  else:
    padding1 = padding2 = padding
  if type(stride) is tuple:
    stride1 = stride[0]
    stride2 = stride[1]
  else:
    stride1 = stride2 = stride
  oh = int((h - hk + 2*padding1)/stride1) + 1
  ow = int((w - wk + 2*padding2)/stride2) + 1
  m = nn.ZeroPad2d((padding2, padding2, padding1, padding1))
  input_new = m(input)

  # bias gradient
  bias_grad = output_grad.sum(dim = (0, 2, 3))

  # weight gradient
  a = torch.arange(0, (oh-1)*stride1+1, stride1).repeat_interleave(ow).reshape(1, -1) + torch.arange(0, hk, 1).reshape(-1, 1)
  b = torch.arange(0, (ow-1)*stride2+1, stride2).repeat(oh).reshape(1, -1) + torch.arange(0, wk, 1).reshape(-1, 1)
  idx1 = a.repeat_interleave(b.shape[0], dim = 0)
  idx2 = b.repeat((a.shape[0], 1))
  yx = input_new[:, :, idx1, idx2].repeat(1, nk, 1, 1)*output_grad.reshape(n, nk, 1, oh*ow).repeat(1, c, 1, 1)
  weight_grad = yx.sum(dim = (0, 3)).reshape(nk, c, hk, wk)

  # input gradient
  dy = output_grad.reshape(n, nk, 1, oh*ow).repeat(1, 1, c, 1).reshape(n, nk, c, 1, oh*ow)
  dx = (dy*weight.reshape(1, nk, c, hk*wk, 1)).sum(dim = 1)
  x = torch.zeros_like(input_new)
  n, c, g, k = dx.shape
  for i in range(n):
    for j in range(c):
      for p in range(g):
        for q in range(k):
          x[i, j, idx1[p, q], idx2[p, q]] += dx[i, j, p, q]
  input_grad = x[:, :, padding1:(padding1+h), padding2:(padding2+w)]
  # END OF YOUR CODE
  return input_grad, weight_grad, bias_grad
```

To check correctness, let us define a simple loss function:
$$
\mathcal{L} = f(\text{output}) = \sum_k \text{output}_k,
$$
so that for each element in the output,
$$
\frac{\partial \mathcal{L}}{\partial \text{output}_k} = 1.
$$
Hence, we can use a tensor of ones for `output_grad`.

We will compare the returned gradients with those computed by PyTorch's automatic differentiation. To use automatic differentiation:
- Create tensors with `requires_grad=True`. PyTorch will automatically build up a computational graph of operations involving those tensors.
- Compute the loss (forward pass).
- Call `loss.backward()`. PyTorch will use backpropagation to compute the gradient of the loss wrt the tensors in the computational graph. The gradient is stored in the `.grad` attribute of each tensor.

We will use the following helper functions to check your results:

```python
def max_rel_error(actual, expected):

  rel_error_top = (actual - expected).abs()
  rel_error_bot = actual.abs() + expected.abs() + 1e-7
  return (rel_error_top / rel_error_bot).max().item()
```

```python
def check_conv2d_grad(batch_size, in_channels, in_height, in_width,
                      out_channels, kernel_size, stride=1, padding=0, use_bias=True,
                      variable='input'):

  if isinstance(kernel_size, int):
    kernel_size = (kernel_size, kernel_size)

  torch.manual_seed(0)
  input = torch.empty(batch_size, in_channels, in_height, in_width).normal_()
  weight = torch.empty(out_channels, in_channels, kernel_size[0], kernel_size[1]).normal_()
  if use_bias:
    bias = torch.empty(out_channels).normal_()
  else:
    bias = None

  output = F.conv2d(input, weight, bias, stride, padding)
  output_grad = torch.ones_like(output)

  input_grad, weight_grad, bias_grad = conv2d_grad(
    output_grad, input, weight, bias, stride, padding)

  # Use automatic differentiation to compute gradients
  # 1. Set `requires_grad` to `True`
  input.requires_grad_()
  weight.requires_grad_()
  if use_bias:
    bias.requires_grad_()

  # 2. Compute loss
  output = F.conv2d(input, weight, bias, stride, padding)
  loss = output.sum()

  # 3. Backpropagation
  loss.backward()
  # The gradients are stored in `input.grad`, `weight.grad`, and `bias.grad`

  if variable == 'input':
    assert input_grad.shape == input.grad.shape, 'incorrect shape'
    rel_error = max_rel_error(input_grad, input.grad)
    print('relative error = %e' % rel_error)
    assert rel_error < 1e-3, 'incorrect result'
  elif variable == 'weight':
    assert weight_grad.shape == weight.grad.shape, 'incorrect shape'
    rel_error = max_rel_error(weight_grad, weight.grad)
    print('relative error = %e' % rel_error)
    assert rel_error < 1e-3, 'incorrect result'
  else:
    if not use_bias:
      assert bias_grad is None
    else:
      assert bias_grad.shape == bias.grad.shape, 'incorrect shape'
      rel_error = max_rel_error(bias_grad, bias.grad)
      print('relative error = %e' % rel_error)
      assert rel_error < 1e-3, 'incorrect result'
```

```python
# check `kernel_size`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=8, in_width=8,
                  out_channels=1, kernel_size=3, stride=1, padding=0, use_bias=True,
                  variable='input')
```

    relative error = 0.000000e+00

```python
# check `kernel_size`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=8, in_width=8,
                  out_channels=1, kernel_size=3, stride=1, padding=0, use_bias=True,
                  variable='weight')
```

    relative error = 8.393523e-08

```python
# check `kernel_size`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=8, in_width=8,
                  out_channels=1, kernel_size=3, stride=1, padding=0, use_bias=True,
                  variable='bias')
```

    relative error = 0.000000e+00

```python
# check `padding`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=32, in_width=32,
                  out_channels=1, kernel_size=8, stride=1, padding=3, use_bias=True,
                  variable='input')
```

    relative error = 0.000000e+00

```python
# check `padding`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=32, in_width=32,
                  out_channels=1, kernel_size=8, stride=1, padding=3, use_bias=True,
                  variable='weight')
```

    relative error = 6.931216e-07

```python
# check `padding`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=32, in_width=32,
                  out_channels=1, kernel_size=8, stride=1, padding=3, use_bias=True,
                  variable='bias')
```

    relative error = 0.000000e+00

```python
# check `in_channels`
check_conv2d_grad(batch_size=1, in_channels=8, in_height=8, in_width=8,
                  out_channels=1, kernel_size=1, stride=1, padding=0, use_bias=True,
                  variable='input')
```

    relative error = 0.000000e+00

```python
# check `in_channels`
check_conv2d_grad(batch_size=1, in_channels=8, in_height=8, in_width=8,
                  out_channels=1, kernel_size=1, stride=1, padding=0, use_bias=True,
                  variable='weight')
```

    relative error = 1.528114e-07

```python
# check `in_channels`
check_conv2d_grad(batch_size=1, in_channels=8, in_height=8, in_width=8,
                  out_channels=1, kernel_size=1, stride=1, padding=0, use_bias=True,
                  variable='bias')
```

    relative error = 0.000000e+00

```python
# check `out_channels`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=32, in_width=32,
                  out_channels=2, kernel_size=9, stride=1, padding=2, use_bias=True,
                  variable='input')
```

    relative error = 3.020955e-06

```python
# check `out_channels`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=32, in_width=32,
                  out_channels=2, kernel_size=9, stride=1, padding=2, use_bias=True,
                  variable='weight')
```

    relative error = 1.284888e-06

```python
# check `out_channels`
check_conv2d_grad(batch_size=1, in_channels=1, in_height=32, in_width=32,
                  out_channels=2, kernel_size=9, stride=1, padding=2, use_bias=True,
                  variable='bias')
```

    relative error = 0.000000e+00

```python
# check `batch_size`
check_conv2d_grad(batch_size=8, in_channels=1, in_height=16, in_width=16,
                  out_channels=1, kernel_size=5, stride=1, padding=3, use_bias=True,
                  variable='input')
```

    relative error = 0.000000e+00

```python
# check `batch_size`
check_conv2d_grad(batch_size=8, in_channels=1, in_height=16, in_width=16,
                  out_channels=1, kernel_size=5, stride=1, padding=3, use_bias=True,
                  variable='weight')
```

    relative error = 1.159864e-06

```python
# check `batch_size`
check_conv2d_grad(batch_size=8, in_channels=1, in_height=16, in_width=16,
                  out_channels=1, kernel_size=5, stride=1, padding=3, use_bias=True,
                  variable='bias')
```

    relative error = 0.000000e+00

```python
# check tuple arguments
check_conv2d_grad(batch_size=16, in_channels=4, in_height=4, in_width=8,
                  out_channels=8, kernel_size=(1, 3), stride=1, padding=1, use_bias=True,
                  variable='input')
```

    relative error = 0.000000e+00

```python
# check tuple arguments
check_conv2d_grad(batch_size=16, in_channels=4, in_height=4, in_width=8,
                  out_channels=8, kernel_size=(1, 3), stride=1, padding=1, use_bias=True,
                  variable='weight')
```

    relative error = 1.197636e-05

```python
# check tuple arguments
check_conv2d_grad(batch_size=16, in_channels=4, in_height=4, in_width=8,
                  out_channels=8, kernel_size=(1, 3), stride=1, padding=1, use_bias=True,
                  variable='bias')
```

    relative error = 0.000000e+00

BONUS: `stride > 1`:

```python
# check `stride`
check_conv2d_grad(batch_size=16, in_channels=4, in_height=4, in_width=8,
                  out_channels=8, kernel_size=(1, 3), stride=(1, 2), padding=1, use_bias=True,
                  variable='input')
```

    relative error = 3.424135e-07

```python
# check `stride`
check_conv2d_grad(batch_size=16, in_channels=4, in_height=4, in_width=8,
                  out_channels=8, kernel_size=(1, 3), stride=(1, 2), padding=1, use_bias=True,
                  variable='weight')
```

    relative error = 6.547677e-07

## CIFAR-10 Open-Ended Challenge

In this section, you can experiment with whatever CNN architecture you'd like on CIFAR-10. Your model should be able to **achieve at least 75% validation accuracy** within 10-20 epochs.

This [tutorial](https://pytorch.org/tutorials/beginner/blitz/cifar10_tutorial.html#sphx-glr-beginner-blitz-cifar10-tutorial-py) will walk you through the process of training a CNN classifier using PyTorch. The model used in this tutorial, however, is too simple for our purpose. To conveniently create a deeper model, you can use [`nn.Sequential()`](https://pytorch.org/docs/1.7.0/generated/torch.nn.Sequential.html#torch.nn.Sequential) to stack multiple layers together. You can use built-in layers in the [`torch.nn`](https://pytorch.org/docs/1.7.0/nn.html) package, or you can develop custom layers. All custom layers should subclass [`nn.Module`](https://pytorch.org/docs/1.7.0/generated/torch.nn.Module.html#torch.nn.Module).

### Things you might try
- **Network architecture**: [VGG](https://arxiv.org/abs/1409.1556) can be a good starting point. Keep in mind that this model is designed for ImageNet, which is much larger and have a different image resolution than CIFAR-10. You need to make some adaptations. For example, you may tweak the number of layers and number of channels. [ResNet](https://arxiv.org/abs/1512.03385) and [DenseNet](https://arxiv.org/abs/1608.06993) are also good architectures to try.
- **Batch normalization**: You can conveniently add batch normalization layers using [`nn.BatchNorm2d()`](https://pytorch.org/docs/1.7.0/generated/torch.nn.BatchNorm2d.html#torch.nn.BatchNorm2d). Does your model train faster with batch normalization?
- **Optimizer**: The [`torch.optim`](https://pytorch.org/docs/1.7.0/optim.html) package implements many optimization algorithms that you can try, such as [SGD](https://pytorch.org/docs/1.7.0/optim.html#torch.optim.SGD), [Adam](https://pytorch.org/docs/1.7.0/optim.html#torch.optim.Adam), and [RMSprop](https://pytorch.org/docs/1.7.0/optim.html#torch.optim.RMSprop).
- **Regularization**: You may use weight decay or [Dropout](https://pytorch.org/docs/1.7.0/generated/torch.nn.Dropout.html#torch.nn.Dropout) to reduce overfitting.
- **Activation**: In addition to [ReLU](https://pytorch.org/docs/1.7.0/generated/torch.nn.ReLU.html#torch.nn.ReLU), some recent papers also use [ELU](https://pytorch.org/docs/1.7.0/generated/torch.nn.ELU.html#torch.nn.ELU) and [GELU](https://pytorch.org/docs/1.7.0/generated/torch.nn.GELU.html#torch.nn.GELU). Can they improve your model's performance?
- **Pooling vs Strided Convolution**: Similar to pooling, strided convolution can reduce the spatial size of feature maps. However, strided convolution contains learnable parameters, which make the model more flexible. Does strided convolution lead to improved performance?
- **Flattening vs Global Average Pooling**: [GoogLeNet](https://static.googleusercontent.com/media/research.google.com/en//pubs/archive/43022.pdf), [ResNet](https://arxiv.org/abs/1512.03385), and [DenseNet](https://arxiv.org/abs/1608.06993) all use global average pooling followed by a single linear layer to produce the class probabilities from the last feature map. How does this compare to the traditional approach of flattening the last feature map and then having several fully connected layers?

### Tips for training
For each network architecture that you try, you should tune the learning rate and other hyperparameters using the validation set. Remember the coarse-to-fine strategy.

### Have fun and happy training!

### Data Preprocessing

We will use the precomputed RGB mean and standard deviation to normalize the images. We then use PyTorch's built-in [`DataLoader`](https://pytorch.org/docs/1.7.0/data.html#torch.utils.data.DataLoader) for automatic mini-batch sampling.

```python
def get_data_loaders():
  """
  Normalize the data and split into (train 40K)/(val 10K)/(test 10K).

  Returns:
  - PyTorch DataLoaders that can automatically sample mini-batches.
  """
  loader_kwargs = {
    'batch_size': 100,
    'shuffle': True,
    'num_workers': 4,
    'pin_memory': True,
    'drop_last': True,
  }

  transform = transforms.Compose([
    transforms.ToTensor(),
    transforms.Normalize([0.4914, 0.4822, 0.4465], [0.2470, 0.2435, 0.2616]),
  ])

  CIFAR10_train = CIFAR10(root='.', train=True, transform=transform, download=True)
  CIFAR10_test = CIFAR10(root='.', train=False, transform=transform, download=True)

  train_set, val_set = random_split(CIFAR10_train, [40000, 10000],
                                    generator=torch.Generator().manual_seed(0))

  train_loader = DataLoader(train_set, **loader_kwargs)
  val_loader = DataLoader(val_set, **loader_kwargs)
  test_loader = DataLoader(CIFAR10_test, **loader_kwargs)

  return train_loader, val_loader, test_loader
```

### Model

Implement your CNN model below. If you implemented multiple models, you only need to show the best one. You may add helper functions and custom modules as necessary.

```python
class Net(nn.Module):

  def __init__(self):
    super().__init__()
    #########################################################################
    # TODO:                                                                 #
    # Specify your model architecture.                                      #
    #########################################################################
    # Replace "pass" statement with your code
    self.conv = nn.Sequential(
        nn.Conv2d(in_channels = 3, out_channels = 32, kernel_size = 3, padding = 1),
        nn.BatchNorm2d(32),
        nn.ReLU(),
        nn.Conv2d(in_channels = 32, out_channels = 64, kernel_size = 3, padding = 1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size = 2, stride = 2),
        nn.Conv2d(in_channels = 64, out_channels = 128, kernel_size = 3, padding = 1),
        nn.BatchNorm2d(128),
        nn.ReLU(),
        nn.Conv2d(in_channels = 128, out_channels = 128, kernel_size = 3, padding = 1),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size = 2, stride = 2),
        nn.Dropout(p = 0.05),
        nn.Conv2d(in_channels = 128, out_channels = 256, kernel_size = 3, padding = 1),
        nn.BatchNorm2d(256),
        nn.ReLU(),
        nn.MaxPool2d(kernel_size = 2, stride = 2),
    )

    self.linear = nn.Sequential(
        nn.Dropout(p = 0.1),
        nn.Linear(256*4*4, 1024),
        nn.ReLU(),
        nn.Linear(1024, 512),
        nn.ReLU(),
        nn.Linear(512, 128),
        nn.ReLU(),
        nn.Dropout(p = 0.1),
        nn.Linear(128, 10)
    )
    # END OF YOUR CODE

  def forward(self, images, labels):
    """
    Compute the loss for a mini-batch of training samples.

    Inputs:
    - images: A PyTorch tensor of shape (B, 3, 32, 32) containing
      a mini-batch of B training images.
    - labels: A PyTorch tensor of shape (B,) containing the corresponding
      training labels; labels[i] = k means that images[i] has label k,
      where 0 <= k < 10.

    Returns:
    - loss: A PyTorch scalar giving the training loss of this mini-batch.
    """
    loss = None
    #########################################################################
    # TODO:                                                                 #
    # Implement the forward pass.                                           #
    #########################################################################
    # Replace "pass" statement with your code
    x = self.conv(images)
    x = x.view(x.size(0), -1)
    x = self.linear(x)

    criterion = nn.CrossEntropyLoss()
    loss = criterion(x, labels)
    # END OF YOUR CODE
    return loss

  def predict(self, images):
    """
    Predict the labels for a mini-batch of images.

    Inputs:
    - images: A PyTorch tensor of shape (B, 3, 32, 32) containing
      a mini-batch of B images.

    Returns:
    - labels_pred: A PyTorch tensor of shape (B,) containing the predicted
      labels; labels_pred[i] = k means that k is the predicted label for
      images[i], where 0 <= k < 10.
    """
    labels_pred = None
    #########################################################################
    # TODO:                                                                 #
    # Implement the prediction phase.                                       #
    #########################################################################
    # Replace "pass" statement with your code
    x = self.conv(images)
    x = x.view(x.size(0), -1)
    x = self.linear(x)
    _, labels_pred = torch.max(x, 1)

    # END OF YOUR CODE
    return labels_pred
```

## Evaluation

We will use the following function to evaluate the classification accuracy:

```python
def eval_acc(model, data_loader):

  with torch.no_grad():
    model.eval()

    total = 0
    correct = 0

    for (images, labels) in data_loader:
      images = images.cuda()
      labels = labels.cuda()

      labels_pred = model.predict(images)

      total += labels.shape[0]
      correct += (labels_pred == labels).sum().item()

    acc = 100 * correct / total

  return acc
```

## Training

A basic training script is provided below. You can make modifications as appropriate. For example, you can save the model that achieves the best validation accuracy (instead of always using the latest model). Also, you may use learning rate schedules and gradient clipping. Only keep the best hyperparameters that you found.

```python
train_loader, val_loader, test_loader = get_data_loaders()

torch.manual_seed(0)
model = Net()
model = model.cuda()

num_epochs = 10
optimizer = optim.Adam(model.parameters(), lr = 0.001, weight_decay = 0.002)  # TODO: choose an optimizer

loss_history = []
for epoch in range(num_epochs):
  val_acc = eval_acc(model, val_loader)

  model.train()

  for batch, (images, labels) in enumerate(train_loader):
    images = images.cuda()
    labels = labels.cuda()

    optimizer.zero_grad()
    loss = model(images, labels)
    loss.backward()
    optimizer.step()

    with torch.no_grad():
      loss_history.append(loss.item())
      if batch == 0:
        print('Train Epoch: {:3} \t Loss: {:F} \t Val Acc: {:F}'.format(
          epoch, loss.item(), val_acc))

with torch.no_grad():
  plt.plot(loss_history, 'o')
  plt.xlabel('Iteration number')
  plt.ylabel('Loss value')
  plt.show()
```

## Results

```python
train_acc = eval_acc(model, train_loader)
print('Training Accuracy:', train_acc)
```

    Training Accuracy: 84.84

```python
val_acc = eval_acc(model, val_loader)
print('Validation Accuracy:', val_acc)
assert val_acc >= 75
```

    Validation Accuracy: 78.13

```python
test_acc = eval_acc(model, test_loader)
print('Test Accuracy:', test_acc)
```

    Test Accuracy: 77.74

```python

```


```python
from statsmodels.tsa.seasonal import seasonal_decompose
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import pandas as pd
from scipy.stats import norm, uniform, expon, t, probplot
import scipy.stats as st
from scipy.integrate import quad
from sympy.solvers import solve
from sympy import Symbol
import numpy as np
from pandas import Series, DataFrame
```

```python
AirP = pd.read_csv("AirPassengers.csv")
AirP.head()
AirP["TravelDate"] = pd.to_datetime(AirP["TravelDate"])
AirP.set_index("TravelDate", inplace = True)
AirP.head()
AirP.plot()
plt.xlabel("Reisedatum")
plt.ylabel("Anzahl Passagiere (in 1000)")
plt.show()
```

```python
AirP.head()
```

      Passengers

      TravelDate

      1949-01-01
      112

      1949-02-01
      118

      1949-03-01
      132

      1949-04-01
      129

      1949-05-01
      121

```python
AusBeer = pd.read_csv("AustralianBeer-2.csv", sep = ";", header = 0)
AusBeer1 = AusBeer.copy()
AusBeer1.head()
```

      Quarter
      megalitres

      0
      1956Q1
      284.4

      1
      1956Q2
      212.8

      2
      1956Q3
      226.9

      3
      1956Q4
      308.4

      4
      1957Q1
      262.0

```python
AusBeer1["Quarter"] = pd.to_datetime(AusBeer["Quarter"])
AusBeer1.set_index("Quarter", inplace = True)
AusBeer1.head()
```

      megalitres

      Quarter

      1956-01-01
      284.4

      1956-04-01
      212.8

      1956-07-01
      226.9

      1956-10-01
      308.4

      1957-01-01
      262.0

```python
AusBeer1.describe()
```

      megalitres

      count
      154.000000

      mean
      408.267532

      std
      97.598588

      min
      212.800000

      25%
      325.425000

      50%
      427.450000

      75%
      466.950000

      max
      600.000000

```python
AusBeer1.loc["1980-9" : "1990-3"].plot()
```

```python
AusBeer = pd.read_csv("AustralianBeer-2.csv", sep = ";", header = 0)
AusEl = pd.read_csv("AustralianElectricity-2.csv", sep = ";")
Aussie = AusBeer.copy()
# Hier wird der Datensatz um eine Spalte kilowatt erweitert
Aussie["kilowatt"] = AusEl["kilowatt"]
Aussie["Quarter"] = pd.to_datetime(Aussie["Quarter"])
Aussie.set_index("Quarter", inplace = True)
Aussie.loc["1970-9": "1994-1"].plot(subplots = True)
plt.show()
```

```python
def boxcox(x,lambd):
    return np.log(x) if (lambd==0) else (x**lambd-1)/lambd

AirP["l_2"] = boxcox(AirP["Passengers"],2)
AirP["l_0"] = boxcox(AirP["Passengers"],0)
AirP["l_-05"] = boxcox(AirP["Passengers"],-.5)
plt.subplot(221)
AirP["Passengers"].plot()
plt.title("Original")
plt.xlabel("")
plt.subplot(222)
AirP["l_-05"].plot()
plt.title("lambda = -0.5")
plt.xlabel("")
plt.subplot(223)
AirP["l_2"].plot()
plt.title("lambda = 2")
plt.xlabel("")
plt.subplot(224)
AirP["l_0"].plot()
plt.title("lambda = 0")
plt.xlabel("")
plt.show()
```

```python
AirQ = pd.read_csv("AirQualityUCI.csv", sep = ";", decimal = ",")
AirQ1 = AirQ.copy()
# pandas kennt das Zeitformat in der Tabelle nicht:
# Punkt muss durch - ersetzt werden
AirQ1["Time"] = AirQ1["Time"].str.replace(".", ":")
AirQ1["Date"] = pd.to_datetime(AirQ1["Date"] + " " + AirQ1["Time"])
AirQ1.set_index("Date", inplace = True)
# Einige Werte der Temperatur sind -200. Diese Zeilen werden weggelassen
AirQ1 = AirQ1[AirQ1["T"] > -20]
AirQ1["T"].plot()
plt.show()
```

```python
AirQ4 = AirQ1.loc["2004-3-10" : "2004-3-30" , "T"]
AirQ4.plot()
```

```python
AirQ1['Hour'] = AirQ1.index.hour
AirQ4 = AirQ1.loc["2004-3-10" : "2004-3-30"]
sns.boxplot(data=AirQ1, x='Hour', y="T")
```

```python
sns.set(rc={'figure.figsize':(11, 4)})
AirQ1['Hour'] = AirQ1.index.hour
AirQ4 = AirQ1.loc["2004-3-10" : "2004-3-30"]
sns.boxplot(data=AirQ1, x='Hour', y="T")
```

```python
AirQ1 = AirQ.copy()
# Einige Werte der Temperatur sind -200. Diese Zeilen werden weggelassen
AirQ1 = AirQ1[AirQ1["T"] > -20]
AirQ1["Time"] = AirQ1["Time"].str.replace(".", ":")
AirQ1["Date"] = pd.to_datetime(AirQ1["Date"] + " " + AirQ1["Time"])
AirQ1.set_index("Date", inplace = True)
# Resample to weekly frequency, aggregating with mean
AirQ1_weekly_mean = AirQ1['T'].resample('Y').mean()
AirQ1_weekly_mean.head(3)
print(AirQ1_weekly_mean.head(3))
```

    Date
    2004-12-31    20.861479
    2005-12-31    10.017496
    Freq: A-DEC, Name: T, dtype: float64

```python
from pandas.plotting import lag_plot
AirQ4 = AirQ.copy()
AirQ4 = AirQ4[AirQ4["T"] > -20]
AirQ4["Time"] = AirQ4["Time"].str.replace(".", ":")
AirQ4["Date"] = pd.to_datetime(AirQ4["Date"] + " " + AirQ4["Time"])
AirQ4.set_index("Date", inplace = True)
plt.subplot(121)
lag_plot(AirQ4)
plt.subplot(122)
lag_plot(AirQ4, 10)
plt.show()
```

```python
seasonal_decompose(AirP["Passengers"], model = "additive", freq = 12).plot()
plt.show()
```

```python
seasonal_decompose(np.log(AirP["Passengers"]), model = "add").resid.plot()
plt.show()
```

```python
pip install stldecompose
```

    Collecting stldecompose
      Downloading https://files.pythonhosted.org/packages/7f/0f/cc3cf33ae4ec1ffc83a2951dc37b5cdb49f9166c0e1de1a61e11b6d954fe/stldecompose-0.0.5-py2.py3-none-any.whl
    Requirement already satisfied: pandas in c:\users\flori\anaconda3\lib\site-packages (from stldecompose) (0.25.3)
    Requirement already satisfied: scipy in c:\users\flori\anaconda3\lib\site-packages (from stldecompose) (1.3.2)
    Requirement already satisfied: matplotlib in c:\users\flori\anaconda3\lib\site-packages (from stldecompose) (3.1.1)
    Requirement already satisfied: statsmodels in c:\users\flori\anaconda3\lib\site-packages (from stldecompose) (0.10.1)
    Requirement already satisfied: numpy in c:\users\flori\anaconda3\lib\site-packages (from stldecompose) (1.17.4)
    Requirement already satisfied: python-dateutil>=2.6.1 in c:\users\flori\anaconda3\lib\site-packages (from pandas->stldecompose) (2.8.1)
    Requirement already satisfied: pytz>=2017.2 in c:\users\flori\anaconda3\lib\site-packages (from pandas->stldecompose) (2019.3)
    Requirement already satisfied: cycler>=0.10 in c:\users\flori\anaconda3\lib\site-packages (from matplotlib->stldecompose) (0.10.0)
    Requirement already satisfied: kiwisolver>=1.0.1 in c:\users\flori\anaconda3\lib\site-packages (from matplotlib->stldecompose) (1.1.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\flori\anaconda3\lib\site-packages (from matplotlib->stldecompose) (2.4.5)
    Requirement already satisfied: patsy>=0.4.0 in c:\users\flori\anaconda3\lib\site-packages (from statsmodels->stldecompose) (0.5.1)
    Requirement already satisfied: six>=1.5 in c:\users\flori\anaconda3\lib\site-packages (from python-dateutil>=2.6.1->pandas->stldecompose) (1.13.0)
    Requirement already satisfied: setuptools in c:\users\flori\anaconda3\lib\site-packages (from kiwisolver>=1.0.1->matplotlib->stldecompose) (42.0.2.post20191203)
    Installing collected packages: stldecompose
    Successfully installed stldecompose-0.0.5
    Note: you may need to restart the kernel to use updated packages.

```python
from stldecompose import decompose
decompose(np.log(AirP["Passengers"]), period = 12).plot();
```

```python
seasonal_decompose(AirP["Passengers"], model = "additive", freq = 12).plot()
```

```python
plt.subplot(211)
seasonal_decompose(AirP["Passengers"], model = "additive", freq = 12).resid.plot()
plt.subplot(212)
decompose((AirP["Passengers"]), period = 12).resid.plot();
plt.show()
```

```python
AirP["Passengers"]
```

    TravelDate
    1949-01-01    112
    1949-02-01    118
    1949-03-01    132
    1949-04-01    129
    1949-05-01    121
                 ...
    1960-08-01    606
    1960-09-01    508
    1960-10-01    461
    1960-11-01    390
    1960-12-01    432
    Name: Passengers, Length: 144, dtype: int64

```python

```


# Music Machine Learning - Expectation-Maximization

### Author: Philippe Esling (esling@ircam.fr)

In this course we will cover
1. Unsupervised learning and [clustering](#clustering).
2. Motivating the need for [latent variable](#latent) models.
2. Approaching the problem naïvely with the [k-Means](#kmeans) algorithm.

## Probabilistic clustering

- Usually clustering is done in _hard_ version
    * Each point belongs to one cluster
    * For instance the basic k-Means algorithm
$$
cluster_{id} = f(x)
$$
- We might want to do probabilistic (soft) clustering
    * Each point belongs more or less to all clusters
    * Defines a probability of belonging
$$
p(cluster_{id} \vert x)
$$

Using probabilistic models has several interests
- Hyperparameter tuning
- Generative modeling

### Maximum likelihood

If we want to optimize our probabilistic model, we have to solve a typical maximum likelihood problem (considering that points $x_{i}$ are independent)
$$
\underset{\theta}{\text{max. }} p(x \vert \theta) = \prod_{i=1}^{N} p(x_{i} \vert \theta)
$$
So we could try to replace our full model inside this problem
$$
\underset{\theta}{\text{max. }} \prod_{i=1}^{N} \sum_{c} \pi_{c} \mathcal{N}\left(x_{i}\vert \mu_{c}, \Sigma_{c}\right)
$$

However, we have to include the following constraints
$$
\sum_{c} \pi_{c} = 1; \pi_{c} \geq 0, \forall c \\
\Sigma_{c} \succcurlyeq 0
$$

We could rely on optimization
- Hard to follow constraints
- Slow and unstable to train
Reasoning is way easier if we introduce a latent variable

### Introducing a latent variable

We will introduce a latent variable $t$, which _explains_ a given datapoint. In our case, this variable tells us _from which group_ the point was generated.

In our case, we can simply say that our latent variable corresponds to the mixing probabilities
$$
\begin{align}
p(z=c\vert\theta) &= \pi_{c} \\
p(x\vert t=c,\theta) &= \mathcal{N}\left(x\vert \mu_{c}, \Sigma_{c}\right)
\end{align}
$$

Therefore, our original model can be retrieved as a marginalization of the cluster latent variable such that
$$
p(x \vert \theta) = \sum_{c} p(x \vert t=c,\theta) p(t=c\vert\theta)
$$

#### Estimating parameters

Now in order to optimize the parameters, we should need to optimize both the $\theta$ and $t$ variables. If we know the cluster belonging (source) variable $t$, then the optimization is easy, since we know that
$$
p(x\vert z=c, \theta) = \mathcal{N}\left(x\vert \mu_{c}, \Sigma_{c}\right)
$$
And computing the parameters of this Gaussian is pretty easy, as we simply need to update the values depending on the corresponding points (weighted mean)
$$
\mu_{c} = \frac{\sum_{i} p(z=c\vert x_{i}, \theta) x_{i}}{\sum_{i} p(z=c\vert x_{i}, \theta)}
$$

#### Estimating sources

Now in order to optimize the source variable $t$, then the optimization is easy if we know the parameters as we have
$$
p(t = c \vert x, \theta) = \frac{p(x \vert t = c, \theta) p(t = c \vert \theta)}{Z}
$$
Note that here the normalization parameter

## Expectation-Maximization

As we just have seen, a natural alternating solution emerges in the form of the Expectation-Maximization algorithm
1. Start with a random set of Gaussians parameters $\theta$
2. Until convergence repeat
    - For each point compute $p(t=c \vert x_{i}, \theta)$
    - Update Gaussian parameters $\theta$

### Jensen inequality

We say that $f(x)$ is concave if, for any $a,b,\alpha$ and $0\leq\alpha\leq 1$ we have
$$
f(\alpha a + (1 - \alpha)b) \geq \alpha f(a) + (1 - a)f(b)
$$

We can generalize this definition to multiple points
$$
f(\alpha_{1}a_{1} + \alpha_{2}a_{2} + \alpha_{3}a_{3}) \geq \alpha_{1}f(a_{1})+\alpha_{2}f(a_{2}) + \alpha_{3}f(a_{3})
$$
with the constraint $\alpha_{1}+\alpha_{2}+\alpha_{3} = 1$. We can see that this formula can generalize if we consider $\alpha$ as probabilities. We obtain Jensen's inequality
$$
f\left(\mathbb{E}_{p(t)}[t]\right) \geq \mathbb{E}_{p(t)}\left[f(t)\right]
$$

### Kullback-Leibler divergence

The Kullback-Leibler divergence allows to compare two probability distributions
$$
\mathcal{KL}(q\Vert p) = \int q(x) \log \frac{q(x)}{p(x)} dx
$$

The KL divergence has several properties
1. $\mathcal{KL}(q\Vert p) \neq \mathcal{KL}(p\Vert q)$
2. $\mathcal{KL}(q\Vert q) = 0$
3. $\mathcal{KL}(q\Vert p) \geq 0$

### General form of EM

We are looking to maximize our model given our parameters, hence if we consider data points as independent we obtain
$$
\begin{align}
\underset{\theta}{\text{max. }} \log p(x \vert \theta) &= \log \prod_{i=1}^{N} p(x_{i} \vert \theta) \\
&= \sum_{i=1}^{N} \log p(x_{i} \vert \theta)
\end{align}
$$

Given that we want to maximize for our mixture model, we can write the final optimization problem as maximizing
$$
\log p(x \vert \theta) = \sum_{i=1}^{N} \log \sum_{c} p(x_{i}, t_{i}=c \vert \theta)
$$

#### Lower bounding the problem

Our goal here is to find a lower bound to this complicated expression so that
$$
\log p(x \vert \theta) \geq \mathcal{L}(\theta)
$$
Hence, if we can maximize the lower bound, it amounts to maximizing the original function. However, the problem is that the quality of our optimization is bound by the quality of our lower bound. Instead, we would like to introduce a lower bound that we could optimize. For instance, if we introduce a density
$$
\log p(x \vert \theta) = \sum_{i=1}^{N} \log \sum_{c} \frac{q(t_{i}=c)}{q(t_{i}=c)} p(x_{i}, t_{i}=c \vert \theta)
$$
Here, we can use Jensen's inequality such that
$$
\log\left(\sum_{c} \alpha_{c} v_{c}\right) \geq \sum_{c} \alpha_c \log(v_{c})
$$
Hence we obtain
$$
\log p(x \vert \theta) = \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)}
$$

To summarize, we now have a function $q$ that we can maximize and that should lower-bound so that
$$
\log p(x \vert \theta) \geq \mathcal{L}(\theta,q), \forall q
$$
Hence we can obtain the best optimization at a step as
$$
q^{k+1} = \underset{q}{\text{argmax. }} \mathcal{L}(\theta^{k}, q)
$$

### Summarizing the steps so far
E-step
$$
q^{k+1} = \underset{q}{\text{argmax. }} \mathcal{L}(\theta^{k}, q)
$$
M-step
$$
\theta^{k+1} = \underset{\theta}{\text{argmax. }} \mathcal{L}(\theta, q^{k+1})
$$

### Deriving the E-Step
Note that we want to minimize the gap between our lower bound and the true distribution, which can be written as
$$
\mathcal{D} = \log p(x \vert \theta) - \mathcal{L}(\theta\vert q)
$$
We can develop this expression
$$
\begin{align}
\mathcal{D} &= \sum_{i=1}^{N} \log p(x_{i} \vert \theta) - \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)} \\
&= \sum_{i=1}^{N} \left(\log p(x_{i} \vert \theta) - \sum_{c} q(t_{i}=c) \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)} \right) \\
&= \sum_{i=1}^{N} \left(\log p(x_{i} \vert \theta)\sum_{c} q(t_{i}=c) - \sum_{c} q(t_{i}=c) \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)} \right) \\
&= \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \left(\log p(x_{i} \vert \theta) - \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)} \right) \\
&= \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log \frac{p(x_{i} \vert \theta)q(t_{i}=c)}{p(x_{i}, t_{i}=c \vert \theta)} \\
&= \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log \frac{q(t_{i}=c)}{p(t_{i} = c \vert x_{i}, \theta)} \\
&= \sum_{i=1}^{N} \mathcal{KL}(q(t_{i}) \Vert p(t_{i} \Vert x_{i}, \theta)
\end{align}
$$
Hence, we need to minimize the KL divergence, which amounts to setting the approximate function as the posterior distribution
$$
q(t_{i}) = p(t_{i} \vert x_{i}, \theta)
$$

### Deriving the M-step

Based on the approximation that we want to maximize
$$
\mathcal{L}(\theta, q) = \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log\frac{p(x_{i}, t_{i}=c \vert \theta)}{q(t_{i}=c)}
$$
We can expand this expression as
$$
\mathcal{L}(\theta, q) = \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log p(x_{i}, t_{i}=c \vert \theta) - \sum_{i=1}^{N} \sum_{c} q(t_{i}=c) \log q(t_{i}=c)
$$
As we can see the second term is not dependent on $\theta$, so the maximization is
$$
\mathcal{L}(\theta, q) = \mathbb{E}_{q} \log p(x,t\vert \theta) + const
$$
Note that the fonction is usually concave, which makes it easy to optimize

#### Convergence guaranties

Note that by properties of the lower bound we have
$$
\log p (x \vert \theta^{k+1}) \geq \mathcal{L}(\theta^{k+1}, q^{k+1}) \geq \mathcal{L}(\theta^{k},q^{k+1})
$$
Therefore we have that
$$
\log p (x \vert \theta^{k+1}) \geq \log p (x \vert \theta^{k})
$$
Hence the EM algorithm is guaranteed to converge

### K-Means from GMM perspective

We can show that we can re-obtain the K-Means algorithm from our GMM formulation
- Fix covariances to be identical $\Sigma_{c}=I$
- Fix weights to be uniform $\pi_{c} = \frac{1}{C}$
Set the probability as
$$
p(x_{i}\vert t_{i}=c, \theta) = \frac{1}{Z} \exp (-0.5 \Vert x_{i} - \mu_{c} \Vert^{2})
$$
We can define $q\in Q$, where $Q$ is the set of delta-functions

# Expectation-maximization algorithm

In this assignment, we will derive and implement formulas for Gaussian Mixture Model — one of the most commonly used methods for performing soft clustering of the data.

### Setup
Loading auxiliary files and importing the necessary libraries.

```python
try:
    import google.colab
    IN_COLAB = True
except:
    IN_COLAB = False
if IN_COLAB:
    print("Downloading Colab files")
    ! shred -u setup_google_colab.py
    ! wget https://raw.githubusercontent.com/hse-aml/bayesian-methods-for-ml/master/setup_google_colab.py -O setup_google_colab.py
    import setup_google_colab
    setup_google_colab.load_data_week2()
```

```python
import numpy as np
from numpy.linalg import slogdet, det, solve
import matplotlib.pyplot as plt
import time
from sklearn.datasets import load_digits
from w2_grader import EMGrader
%matplotlib inline
```

### Grading
We will create a grader instance below and use it to collect your answers. Note that these outputs will be stored locally inside grader and will be uploaded to the platform only after running submitting function in the last part of this assignment. If you want to make a partial submission, you can run that cell anytime you want.

```python
grader = EMGrader()
```

## Implementing EM for GMM

For debugging, we will use samples from a Gaussian mixture model with unknown mean, variance, and priors. We also added initial values of parameters for grading purposes.

```python
samples = np.load('samples.npz')
X = samples['data']
pi0 = samples['pi0']
mu0 = samples['mu0']
sigma0 = samples['sigma0']
plt.scatter(X[:, 0], X[:, 1], c='grey', s=30)
plt.axis('equal')
plt.show()
```

### Reminder

Remember, that EM algorithm is a coordinate descent optimization of variational lower bound $\mathcal{L}(\theta, q) = \int q(T) \log\frac{p(X, T|\theta)}{q(T)}dT\to \max$.

E-step:

$\mathcal{L}(\theta, q) \to \max\limits_{q} \Leftrightarrow \mathcal{KL} [q(T) \,\|\, p(T|X, \theta)] \to \min \limits_{q\in Q} \Rightarrow q(T) = p(T|X, \theta)$

M-step:

$\mathcal{L}(\theta, q) \to \max\limits_{\theta} \Leftrightarrow \mathbb{E}_{q(T)}\log p(X,T | \theta) \to \max\limits_{\theta}$

For GMM, $\theta$ is a set of parameters that consists of mean vectors $\mu_c$, covariance matrices $\Sigma_c$ and priors $\pi_c$ for each component.

Latent variables $T$ are indices of components to which each data point is assigned, i.e. $t_i$  is the cluster index for object $x_i$.

The joint distribution can be written as follows: $\log p(T, X \mid \theta) =  \sum\limits_{i=1}^N \log p(t_i, x_i \mid \theta) = \sum\limits_{i=1}^N \sum\limits_{c=1}^C q(t_i = c) \log \left (\pi_c \, f_{\!\mathcal{N}}(x_i \mid \mu_c, \Sigma_c)\right)$,
where $f_{\!\mathcal{N}}(x \mid \mu_c, \Sigma_c) = \frac{1}{\sqrt{(2\pi)^n|\boldsymbol\Sigma_c|}}
\exp\left(-\frac{1}{2}({x}-{\mu_c})^T{\boldsymbol\Sigma_c}^{-1}({x}-{\mu_c})
\right)$ is the probability density function (pdf) of the normal distribution $\mathcal{N}(x_i \mid \mu_c, \Sigma_c)$.

### E-step
In this step we need to estimate the posterior distribution over the latent variables with fixed values of parameters: $q_i(t_i) = p(t_i \mid x_i, \theta)$. We assume that $t_i$ equals to the cluster index of the true component of the $x_i$ object. To do so we need to compute $\gamma_{ic} = p(t_i = c \mid x_i, \theta)$. Note that $\sum\limits_{c=1}^C\gamma_{ic}=1$.

Important trick 1: It is important to avoid numerical errors. At some point you will have to compute the formula of the following form: $\frac{e^{y_i}}{\sum_j e^{y_j}}$, which is called _softmax_. When you compute exponents of large numbers, some numbers may become infinity. You can avoid this by dividing numerator and denominator by $e^{\max(y)}$: $\frac{e^{y_i-\max(y)}}{\sum_j e^{y_j - \max(y)}}$. After this transformation maximum value in the denominator will be equal to one. All other terms will contribute smaller values. So, to compute desired formula you first subtract maximum value from each component in vector $\mathbf{y}$ and then compute everything else as before.

Important trick 2: You will probably need to compute formula of the form $A^{-1}x$ at some point. You would normally inverse $A$ and then multiply it by $x$. A bit faster and more numerically accurate way to do this is to directly solve equation $Ay = x$ by using a special function. Its solution is $y=A^{-1}x$, but the equation $Ay = x$ can be solved by methods which do not explicitely invert the matrix. You can use ```np.linalg.solve``` for this.

Other usefull functions:  ```slogdet``` and ```det```

Task 1: Implement E-step for GMM using template below.

```python
def E_step(X, pi, mu, sigma):
    """
    Performs E-step on GMM model
    Each input is numpy array:
    X: (N x d), data points
    pi: (C), mixture component weights
    mu: (C x d), mixture component means
    sigma: (C x d x d), mixture component covariance matrices

    Returns:
    gamma: (N x C), probabilities of clusters for objects
    """
    N = X.shape[0] # number of objects
    C = pi.shape[0] # number of clusters
    d = mu.shape[1] # dimension of each object
    gamma = np.zeros((N, C)) # distribution q(T)

    ### YOUR CODE HERE

    return gamma
```

```python
gamma = E_step(X, pi0, mu0, sigma0)
grader.submit_e_step(gamma)
```

### M-step

In M-step we need to maximize $\mathbb{E}_{q(T)}\log p(X,T | \theta)$ with respect to $\theta$. In our model this means that we need to find optimal values of $\pi$, $\mu$, $\Sigma$. To do so, you need to compute the derivatives and
set them to zero. You should start by deriving formulas for $\mu$ as it is the easiest part. Then move on to $\Sigma$. Here it is crucial to optimize function w.r.t. to $\Lambda = \Sigma^{-1}$ and then inverse obtained result. Finaly, to compute $\pi$, you will need Lagrange Multipliers technique to satisfy constraint $\sum\limits_{i=1}^{n}\pi_i = 1$.

Important note: You will need to compute derivatives of scalars with respect to matrices. To refresh this technique from previous courses, see  wiki article about it . Main formulas of matrix derivatives can be found in Chapter 2 of The Matrix Cookbook. For example, there you may find that $\frac{\partial}{\partial A}\log |A| = A^{-T}$.

Task 2: Implement M-step for GMM using template below.

```python
def M_step(X, gamma):
    """
    Performs M-step on GMM model
    Each input is numpy array:
    X: (N x d), data points
    gamma: (N x C), distribution q(T)

    Returns:
    pi: (C)
    mu: (C x d)
    sigma: (C x d x d)
    """
    N = X.shape[0] # number of objects
    C = gamma.shape[1] # number of clusters
    d = X.shape[1] # dimension of each object

    ### YOUR CODE HERE

    return pi, mu, sigma
```

```python
gamma = E_step(X, pi0, mu0, sigma0)
pi, mu, sigma = M_step(X, gamma)
grader.submit_m_step(pi, mu, sigma)
```

### Loss function

Finally, we need some function to track convergence. We will use variational lower bound $\mathcal{L}$ for this purpose. We will stop our EM iterations when $\mathcal{L}$ will saturate. Usually, you will need only about 10-20 iterations to converge. It is also useful to check that this function never decreases during training. If it does, you have a bug in your code.

Task 3: Implement a function that will compute $\mathcal{L}$ using template below.

$$\mathcal{L} = \sum_{i=1}^{N} \sum_{c=1}^{C} q(t_i =c) (\log \pi_c + \log f_{\!\mathcal{N}}(x_i \mid \mu_c, \Sigma_c)) - \sum_{i=1}^{N} \sum_{c=1}^{K} q(t_i =c) \log q(t_i =c)$$

```python
def compute_vlb(X, pi, mu, sigma, gamma):
    """
    Each input is numpy array:
    X: (N x d), data points
    gamma: (N x C), distribution q(T)
    pi: (C)
    mu: (C x d)
    sigma: (C x d x d)

    Returns value of variational lower bound
    """
    N = X.shape[0] # number of objects
    C = gamma.shape[1] # number of clusters
    d = X.shape[1] # dimension of each object

    ### YOUR CODE HERE

    return loss
```

```python
pi, mu, sigma = pi0, mu0, sigma0
gamma = E_step(X, pi, mu, sigma)
pi, mu, sigma = M_step(X, gamma)
loss = compute_vlb(X, pi, mu, sigma, gamma)
grader.submit_VLB(loss)
```

### Bringing it all together

Now that we have E step, M step and VLB, we can implement the training loop. We will initialize values of $\pi$, $\mu$ and $\Sigma$ to some random numbers, train until $\mathcal{L}$ stops changing, and return the resulting points. We also know that the EM algorithm converges to local optima. To find a better local optima, we will restart the algorithm multiple times from different (random) starting positions. Each training trial should stop either when maximum number of iterations is reached or when relative improvement is smaller than given tolerance ($|\frac{\mathcal{L}_i-\mathcal{L}_{i-1}}{\mathcal{L}_{i-1}}| \le \text{rtol}$).

Remember, that initial (random) values of $\pi$ that you generate must be non-negative and sum up to 1. Also, $\Sigma$ matrices must be symmetric and positive semi-definite. If you don't know how to generate those matrices, you can use $\Sigma=I$ as initialization.

You will also sometimes get numerical errors because of component collapsing. The easiest way to deal with this problems is to restart the procedure.

Task 4: Implement training procedure

```python
def train_EM(X, C, rtol=1e-3, max_iter=100, restarts=10):
    '''
    Starts with random initialization *restarts* times
    Runs optimization until saturation with *rtol* reached
    or *max_iter* iterations were made.

    X: (N, d), data points
    C: int, number of clusters
    '''
    N = X.shape[0] # number of objects
    d = X.shape[1] # dimension of each object
    best_loss = None
    best_pi = None
    best_mu = None
    best_sigma = None

    for _ in range(restarts):
        try:
            ### YOUR CODE HERE

        except np.linalg.LinAlgError:
            print("Singular matrix: components collapsed")
            pass

    return best_loss, best_pi, best_mu, best_sigma
```

```python
best_loss, best_pi, best_mu, best_sigma = train_EM(X, 3)
grader.submit_EM(best_loss)
```

If you implemented all the steps correctly, your algorithm should converge in about 20 iterations. Let's plot the clusters to see it. We will assign a cluster label as the most probable cluster index. This can be found using a matrix $\gamma$ computed on last E-step.

```python
gamma = E_step(X, best_pi, best_mu, best_sigma)
labels = gamma.argmax(axis=1)
colors = np.array([(31, 119, 180), (255, 127, 14), (44, 160, 44)]) / 255.
plt.scatter(X[:, 0], X[:, 1], c=colors[labels], s=30)
plt.axis('equal')
plt.show()
```


```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

```python
import pyflow
from pyflow.euler.supersonic_inlet import SupersonicInletProblem
```

# Supersonic Inlet Problem

## Description

Using the quasi-1D Euler equations, calculate the flow within the supersonic inlet for a supersonic ramjet represented schematic below (not drawn to scale). Assume the height of the entrance and exit areas are the same and equal to 0.2 m, and the distance between

Consider a simplification of the above supersonic inlet. At the left boundary of the boxed-domain, the inflow Mach number is defined as M=2.5 with a free-stream pressure and temperature are given as $p= 1 \times 10^6$ N/m$^2$ and $T= 300$ K, respectively. Thus, the flow is supersonic at the inflow. At the right boundary of the boxed-domain, the exit pressure is specified as $P_{\textrm{exit}}$, and it is unknown \textit{a priori} whether or not the outflow is supersonic or subsonic.

The inlet is symmetric and defined by a two straight lines inclined at the angle $\theta$ given by $\tan \theta = 0.25$ with the intersection rounded using a radius of curvature, $R_c = 0.5$.

Defining the lenght of the domain as $L = 1.0$ m and assuming the spacing between the 81 mesh points is uniform, we can find an equation for the variation of the height, $A(x_i)$, as function the $x_i$ location of each mesh point using the getometrical values $\theta$ and $R_c$.

## Quasi-One-dimensional Euler Equations

The quasi-1-D, unsteady Euler equations are given by the hyperbolic set of partial differential equations described by

$$
\frac{\partial \, \mathbf{Q}}{\partial t} + \frac{1}{A} \frac{\partial \, \big( \mathbf{F} A \big)}{\partial x} = \mathbf{S}
$$
where $\mathbf{Q}$ is defined as vector of conservative variables

$$
\mathbf{Q} = \begin{bmatrix}
    \rho & \\
    \rho u & \\
    e
  \end{bmatrix}
$$

and the flux vector $\mathbf{F}$ is defined as

$$
\mathbf{F} = \begin{bmatrix}
    \rho u & \\
    \rho u^2 + p & \\
    (e + p) u
  \end{bmatrix}
$$
and the source term vector is defined as

$$
\mathbf{S} = \begin{bmatrix}
    0 & \\
    \frac{p}{A}\frac{ \partial A }{\partial x} & \\
    0
  \end{bmatrix}
$$

and $A = A(x)$ is the area of the cross-sectional area as a function of $x$ normalized by the in-plane distance, i.e., the dimensions of A is in meters, not square-meters. For a calorically, perfect gas, we can use the thermodynamic relationship

$$
p = (\gamma - 1) \rho \epsilon
$$
and

$$
\epsilon = c_v T = \frac{e}{\rho} - \frac{1}{2} u^2
$$
to relate the conservative variables to the primitive variables of velocity, pressure, and internal energy. Note that this equation devolves the purely one-dimensional form of the Euler equations when $A(x) = 1$.

## Case 1 : No Shock Wave

First consider the case where the flow remains supersonic through the entire inlet. For the initial conditions

$$
\begin{align}
&M_0 = 2.5 \\
&p_0 = 1 \times 10^6 \, \textrm{N} / \textrm{m}^2 \\
&T_0 = 300 \, \textrm{K},
\end{align}
$$

the isentropic relations can be used to find the variation of the Mach number through the inlet. If there is no shock in the downstream flow, then the isentropic relationships should provide the exact solution and the numerical solution should not vary in time (the initial conditions are the solution). This will provide a verification that the boundary conditions were implemented correctly.

For the case of supersonic flow through the inlet, the variation of density, velocity, and pressure as a function of the area. From the isentropic relations, the pressure, density, and velocity can be determined analytically. This requires using Netwon's method to solve the following equation for Mach number given the known variation in the area.

$$
f(M) = \left(\frac{A}{A_0}\right)^2 - \left(\frac{M_0}{M}\right)^2 \left( \frac{ 1 + \frac{\gamma - 1}{2} M^2 }{ 1 + \frac{\gamma - 1}{2} M_0^2 } \right)^{\frac{\gamma + 1}{\gamma - 1}}
$$

Once the Mach number is computed, isentropic relationships can be used to find the density, pressure, and velocity through the inlets as

$$
\begin{align}
&\frac{\rho}{\rho_0} = \left( \frac{ 1 + \frac{\gamma -1 }{2} M_0^2 } { 1 + \frac{\gamma -1 }{2} M^2 } \right)^{\frac{1}{\gamma - 1}} \\
&\frac{p}{p_0} = \left( \frac{ 1 + \frac{\gamma -1 }{2} M_0^2 } { 1 + \frac{\gamma -1 }{2} M^2 } \right)^{\frac{\gamma}{\gamma - 1}} \\
&u  = \frac{\rho_0 A_0}{\rho A} u_0
\end{align}
$$

Using a CFL of 0.8, we can take a large number of time steps to verify that the solution does not change. Even if you are not verifying your own implementation of a code, test cases like this are excellent ways to verify that you using the CFD software correctly and that the numerical method is doing what you expect it to do.

```python
# Final time of 0.02 seconds -- for this problem the initial conditions are
# exact solution, so time should not matter.
#
t_final = 0.02

# Mesh size
#
Nx = 200

# CFL number
#
CFL = 0.5
```

```python
# MacCormack method requires numerical dissipation for stabilization
#
MC_inlet = SupersonicInletProblem(method='maccormack', dissipation=0.2, Nx=Nx, CFL=CFL, normalize_solution=True)
MC = MC_inlet.solve(t_final)
```

    2022-05-07 05:09:54,119 | pyflow.euler.supersonic_inlet | INFO | Finite-volume method selected: MacCormack predictor-correct

    Simulation Complete
    Final time = 0.02
    Total number of iterations = 9724
    2022-05-07 05:09:55,970 | pyflow.euler.supersonic_inlet | INFO | Returned solution variable speed is the Mach number

```python
# Steger-Warming
#
SW_inlet = SupersonicInletProblem(method='steger-warming', Nx=Nx, CFL=CFL, normalize_solution=True)
SW = SW_inlet.solve(t_final)
```

    2022-05-07 05:10:01,486 | pyflow.euler.supersonic_inlet | INFO | Finite-volume method selected: Steger-Warming

    Simulation Complete
    Final time = 0.02
    Total number of iterations = 9723
    2022-05-07 05:10:02,905 | pyflow.euler.supersonic_inlet | INFO | Returned solution variable speed is the Mach number

```python
fig, ax = plt.subplots()
ax.plot(MC.x, MC.speed, label='MacCormack')
ax.plot(SW.x, SW.speed, label='Steger-Warming')

ax.set_xlabel(r'$x$')
ax.set_ylabel('Mach Number')
ax.legend()
plt.title('Supersonic Inlet Problem')
```

    <IPython.core.display.Javascript object>

    Text(0.5, 1.0, 'Supersonic Inlet Problem')

## Case 2 : Stationary Shock Wave

By increasing the back pressure at the outflow boundary, a stationary shock will form in the diverging portion of the inlet. What is the solution for the following conditions?

$$
\begin{align}
&M_0 = 2.5 \\
&p_0 = 1 \times 10^6 \, \textrm{N} / \textrm{m}^2 \\
&T_0 = 300 \, \textrm{K} \\
&p_{exit} = 15 * p_0
\end{align}
$$

```python
# MacCormack method requires numerical dissipation for stabilization
#
MC_inlet_15 = SupersonicInletProblem(method='maccormack', dissipation=0.2, Nx=Nx, CFL=CFL, pExit_p0=15, normalize_solution=True)
MC_15 = MC_inlet_15.solve(t_final)
```

    2022-05-07 05:10:18,185 | pyflow.euler.supersonic_inlet | INFO | Finite-volume method selected: MacCormack predictor-correct

    Simulation Complete
    Final time = 0.02
    Total number of iterations = 9724
    2022-05-07 05:10:20,018 | pyflow.euler.supersonic_inlet | INFO | Returned solution variable speed is the Mach number

```python
# Steger-Warming
#
SW_inlet_15 = SupersonicInletProblem(method='maccormack', dissipation=0.2, Nx=Nx, CFL=CFL, pExit_p0=15, normalize_solution=True)
SW_15 = SW_inlet_15.solve(t_final)
```

    2022-05-07 05:10:23,157 | pyflow.euler.supersonic_inlet | INFO | Finite-volume method selected: MacCormack predictor-correct

    Simulation Complete
    Final time = 0.02
    Total number of iterations = 9724
    2022-05-07 05:10:25,004 | pyflow.euler.supersonic_inlet | INFO | Returned solution variable speed is the Mach number

```python
fig, ax = plt.subplots()
ax.plot(MC_15.x, MC_15.speed, label='MacCormack')
ax.plot(SW_15.x, SW_15.speed, label='Steger-Warming')

ax.set_xlabel(r'$x$')
ax.set_ylabel('Mach Number')
ax.legend()
plt.title('Supersonic Inlet Problem')
```

    <IPython.core.display.Javascript object>

    Text(0.5, 1.0, 'Supersonic Inlet Problem')

```python

```


```python
from neural_odes import *
%load_ext autoreload
%autoreload 2
```

## Solve the Discrete Lotka-Volterra Eqtn for general non-symmetric $A$
\begin{equation}
    \begin{aligned}
    p_i(t+1) &= p_i(t)\big[1+r_i\big(1-\frac{\sum_{j=1}^dA_{ij}p_j(t)}{k_i}\big)\big], i = 1, \dots d\\
    &= p_i(t)\big[1+r_i\big(1-\frac{\mathbf{b}_i^T\big(\sum_{j=1}^d\mathbf{c}_jp_j(t)\big)}{k_i}\big)\big], i = 1, \dots d \\
    &= p_i(t)\big[1+r_i\big(1-\frac{\mathbf{b}_i^TC\mathbf{p}}{k_i}\big)\big], i = 1, \dots d \\
    \end{aligned}
\end{equation}.

We approximate $A_{ij} = \mathbf{b}_i^T\mathbf{c}_j$ using the low rank matrix approximation $A= B^TC$, where $B = [\mathbf{b}_1, \cdots, \mathbf{b}_d] \in \mathbb{R}^{k \times d}$ and $C = [\mathbf{c}_1, \cdots, \mathbf{c}_d] \in \mathbb{R}^{k \times d}$.  Each $\mathbf{b}_i, \mathbf{c}_i \in \mathbb{R}^k$, where $k \ll d$ are the embeddings of time series $i$.

In matrix-vector form, we have $A\mathbf{p}$, which has computational complexity $\mathcal{O}(d^2)$ for $A \in \mathbb{R}^{d \times d}, \mathbf{p} \in \mathbb{R}^d$.  Using the low-rank form, we can write $A\mathbf{p} = B^T(C \mathbf{p})$.  We do not want to explicitly form the matrix $B^TC$, since this would have higher complexity of $\mathcal{O}(kd^2)$.  We instead break the computation into two matrix-vector products as indicated by the parathesis, each of complexity $\mathcal{O}(kd) \ll \mathcal{O}(d^2).$

We start with $d = 100$ for the number of time series and will learn the synthetic data from the equation for random initialized $A, \mathbf{r}, \mathbf{k}$

### Step 0. Set parameters

```python
num_time_series = 100
num_time_steps = 10
low_rank_param = 5
```

To enable the symmetric form, where $C = B$, set `is_sym = True`.

To explicitly form the low rank matrix matrix product $A = B^TC$, set `is_full_matrix = True`.

```python
is_sym = False
is_full_matrix = False
```

### Step 1. Generate data for $A \in \mathbb{R}^{d \times d}, \mathbf{r}, \mathbf{k} \in \mathbb{R}^d$ and the initial condition $\mathbf{p}(0) \in \mathbb{R}^d$.

```python
# Use gpu and higher precision for numerical computations
ctx = mx.gpu()
dtype = 'float64'
```

```python
_, _, p0, A = generate_data(num_time_series)
r = nd.ones(num_time_series, ctx=ctx, dtype=dtype)
k = 100*r
r.shape, k.shape, p0.shape, A.shape
```

    ((100,), (100,), (100,), (100, 100))

### Step 3. Solve the LV eqn for $p_i(t+1), 0 \le t < N - 1$.  We store $P$ as a matrix in $\mathbb{R}^{d \times N}$, whose first column is the initial condition $\mathbf{p}(0) \in \mathbb{R}^d$. Then $P = [\mathbf{p}(0), \dots, \mathbf{p}(N-1)].$  Use two embeddings to learn $B, C \in \mathbb{R}^{k \times d}$

```python
neural_lv = NeuralLV(num_time_series, num_time_steps, low_rank_param,
                     is_full_matrix, p0, r, k, A, is_sym)
```

```python
# Set number of iterations to run
num_epochs = 5000
```

```python
p_approx, p, A_approx, model= neural_lv.run(num_epochs)
```

    HBox(children=(IntProgress(value=0, max=5000), HTML(value='')))

```python
# Can re run until convergence extra 1000 at a time (Ran twice for total of 7000 iterations)
p_approx, p, A_approx, model= neural_lv.run(model=model)
```

    HBox(children=(IntProgress(value=0, max=1000), HTML(value='')))

### Step 5. Error Evaluation and Plotting

```python
print(f'l2 norm of the error = {nd.norm(p_approx-p)}')
```

    l2 norm of the error =
    [0.08958898]
    <NDArray 1 @gpu(0)>

```python
print(f'max norm of the error = {nd.max(nd.abs(p_approx-p))}')
```

    max norm of the error =
    [0.01711842]
    <NDArray 1 @gpu(0)>

```python
print(f'l2 matrix norm of the error of A and its low rank approx = {nd.norm(A_approx-A)}')
```

    l2 matrix norm of the error of A and its low rank approx =
    [39.32629733]
    <NDArray 1 @gpu(0)>

```python
# First ten time series
lv_plot_ts(p, p_approx, fig_size_width=20)
```


# Thinking probabilistically - Continuous variables
<b>

1. Probability density functions
2. Introduction to the Normal distribution
    1. The Normal PDF
    2. The Normal CDF
3. The Normal distribution: Properties and warnings
4. Are the Belmont Stakes results Normally distributed?
5. What are the chances of a horse matching or beating Secretariat's record?
6. The Exponential distribution
7. If you have a story, you can simulate it!
8. Distribution of no-hitters and cycles

</b>

## Probability density functions
- Continuous variables
    - Quantities that can take any value, not just discrete values
- Probability Density function (PDF)
    - Continuous analog to the PMF
    - Mathematical description of the relative likelihood of observing a value of a continuous variable

## Introduction to the Normal distribution

* Normal distribution: Describes a continuous variable whose PDF has a single symmetric peak.

$$ \begin{align} \text{mean of a Normal distribution} & \neq \text{mean computed from data} \\ \text{st. dev of a Normal distribution} & \neq \text{st. dev computed from data} \end{align} $$

### The Normal PDF:

In this exercise, you will explore the Normal PDF and also learn a way to plot a PDF of a known distribution using hacker statistics. Specifically, you will plot a Normal PDF for various values of the variance.

```
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
import pandas as pd
```

```
sns.set()
```

```
# with stds of interest: samples_std1, samples_std3, stamples_std10
samples_std1 = np.random.normal(20, 1, 100000)
samples_std3 = np.random.normal(20, 3, 100000)
samples_std10 = np.random.normal(20, 10, 100000)
```

```
# make histogram
plt.figure(figsize=(30,15))
_ = plt.hist(samples_std1,histtype='step',density=True,bins=100)
_ = plt.hist(samples_std3,histtype='step',density=True,bins=100)
_ = plt.hist(samples_std10,histtype='step',density=True,bins=100)

# Make a legends and set limits
_ = plt.legend(('std = 1', 'std = 3', 'std = 10'))
plt.ylim(-0.01, 0.42)
```

### The Normal CDF

Now that you have a feel for how the Normal PDF looks, let's consider its CDF. Using the samples you generated in the last exercise (in your namespace as samples_std1, samples_std3, and samples_std10), generate and plot the CDFs.

```
def ecdf(data):
    """Compute ECDF for a one-dimensional array of measurements."""
    # Number of data points: n
    n = len(data)

    # x-data for Ecdf
    x = np.sort(data)

    # y-data for the ecdf-y
    y =np.arange(1,n+1)/n

    return x,y
```

```
x_std1, y_std1 = ecdf(samples_std1)
x_std3, y_std3 = ecdf(samples_std3)
x_std10, y_std10 = ecdf(samples_std10)
```

```
# Plot CDF
_ = plt.plot(x_std1,y_std1,marker='.',linestyle=None)
_ = plt.plot(x_std3,y_std3,marker='*',linestyle=None)
_ = plt.plot(x_std10,y_std10,marker='o',linestyle=None)
# Make a legend
_ = plt.legend(('std = 1', 'std = 3', 'std = 10'), loc='lower right')
```

### The Normal distribution: Properties and warnings
**Are the Belmont Stakes results Normally distributed?**

Since 1926, the Belmont Stakes is a 1.5 mile-long race of 3-year old thoroughbred horses. [Secretariat](https://en.wikipedia.org/wiki/Secretariat_(horse) ) ran the fastest Belmont Stakes in history in 1973. While that was the fastest year, 1970 was the slowest because of unusually wet and sloppy conditions. With these two outliers removed from the data set, compute the mean and standard deviation of the Belmont winners' times. Sample out of a Normal distribution with this mean and standard deviation using the ```np.random.normal()``` function and plot a CDF. Overlay the ECDF from the winning Belmont times. Are these close to Normally distributed?

Note: Justin scraped the data concerning the Belmont Stakes from the [Belmont Wikipedia page](https://en.wikipedia.org/wiki/Belmont_Stakes).

```
url = "https://raw.githubusercontent.com/reddyprasade/Data-Analysis-with-Python/main/Statistics/Data/belmont.csv"
```

```
df = pd.read_csv(url)
df
```

```
df.head()
```

```
df.tail()
```

```
df.info()
```

```
df.describe()
```

```
belmont_no_outliers = np.array([148.51, 146.65, 148.52, 150.7 , 150.42, 150.88, 151.57, 147.54,
       149.65, 148.74, 147.86, 148.75, 147.5 , 148.26, 149.71, 146.56,
       151.19, 147.88, 149.16, 148.82, 148.96, 152.02, 146.82, 149.97,
       146.13, 148.1 , 147.2 , 146.  , 146.4 , 148.2 , 149.8 , 147.  ,
       147.2 , 147.8 , 148.2 , 149.  , 149.8 , 148.6 , 146.8 , 149.6 ,
       149.  , 148.2 , 149.2 , 148.  , 150.4 , 148.8 , 147.2 , 148.8 ,
       149.6 , 148.4 , 148.4 , 150.2 , 148.8 , 149.2 , 149.2 , 148.4 ,
       150.2 , 146.6 , 149.8 , 149.  , 150.8 , 148.6 , 150.2 , 149.  ,
       148.6 , 150.2 , 148.2 , 149.4 , 150.8 , 150.2 , 152.2 , 148.2 ,
       149.2 , 151.  , 149.6 , 149.6 , 149.4 , 148.6 , 150.  , 150.6 ,
       149.2 , 152.6 , 152.8 , 149.6 , 151.6 , 152.8 , 153.2 , 152.4 ,
       152.2 ])
```

```
# Compute mean and standard deviation: mu, sigma
mu = np.mean(belmont_no_outliers)
sigma = np.std(belmont_no_outliers)
```

```
# Sample out of a normal distribution with this mu and sigma: samples
samples = np.random.normal(mu, sigma, size=10000)
```

```
# Get the CDF of the samples and of the data
x_theor, y_theor = ecdf(belmont_no_outliers) # orginal data

x, y = ecdf(samples) # normal distribution with this mu and sigma: samples
```

```
# Plot the CDFs
plt.figure(figsize=(30,15))
_ = plt.plot(x_theor, y_theor) # Orginal data
_ = plt.plot(x, y, marker='.', linestyle='none') # Samples
_ = plt.xlabel('Belmont winning time (sec.)')
_ = plt.ylabel('CDF')
#plt.legend()
```

What are the chances of a horse matching or beating Secretariat's record?

Assume that the Belmont winners' times are Normally distributed (with the 1970 and 1973 years removed), what is the probability that the winner of a given Belmont Stakes will run it as fast or faster than Secretariat?

```
# Take a million samples out of the Normal distribution: samples
samples = np.random.normal(mu,sigma,size=1000000)

# Compute the fraction that are faster than 144 seconds: prob
prob = np.sum(samples/144)/len(samples)
# Print the result
print('Probability of besting Secretariat:', prob)
```

## The Exponential distribution
- The **waiting time** between **arrivals** of a Poisson process is Exponentially distributed

**If you have a story, you can simulate it!**

- Sometimes, the story describing our probability distribution does not have a named distribution to go along with it. In these cases, fear not! You can always simulate it. We'll do that in this and the next exercise.

- In earlier exercises, we looked at the rare event of no-hitters in Major League Baseball. Hitting the cycle is another rare baseball event. When a batter hits the cycle, he gets all four kinds of hits, a single, double, triple, and home run, in a single game. Like no-hitters, this can be modeled as a Poisson process, so the time between hits of the cycle are also Exponentially distributed.

- How long must we wait to see both a no-hitter and then a batter hit the cycle? The idea is that we have to wait some time for the no-hitter, and then after the no-hitter, we have to wait for hitting the cycle. Stated another way, what is the total waiting time for the arrival of two different Poisson processes? The total waiting time is the time waited for the no-hitter, plus the time waited for the hitting the cycle.

Now, you will write a function to sample out of the distribution described by this story.

```
def successive_poisson(tau1,tau2,size=1):
    """
    We ahave Compute time for arrival Successive Poisson.
    """
    # Draw samples out of first exponential distribution: t1
    t1 = np.random.exponential(tau1,size)

    # Draw samples out of Second exponential distribution: t2

    t2 = np.random.exponential(tau2,size)

    return t1,t2
```

### Distribution of no-hitters and cycles
Now, you'll use your sampling function to compute the waiting time to observe a no-hitter and hitting of the cycle. The mean waiting time for a no-hitter is 764 games, and the mean waiting time for hitting the cycle is 715 games.

```
# Draw samples of waiting times: waiting_times
waiting_times = successive_poisson(764, 715, size=100000)

# Make the histogram
plt.figure(figsize=(30,15))
_ = plt.hist(waiting_times,bins=100,histtype='step',density=True)

# Label axes
_ = plt.xlabel('waiting times')
_ = plt.ylabel('PDF')
```

Statistics/STAST-EN-2021-106 Thinking probabilistically - Continuous variables.ipynb


| |Pierre Proulx, ing, professeur|
|:---|:---|
|Département de génie chimique et de génie biotechnologique |** GCH200-Phénomènes d'échanges I **|

### Section 2.5 Écoulement de deux fluides visqueux adjacents sur un plan.
> Pour cet exemple on ira directement à l'équation diférentielle puisqu'on a déjà fait le même type de bilans à la section 2.2 en détails.

>

```python
#
# Pierre Proulx
#
# Préparation de l'affichage et des outils de calcul symbolique
#
import sympy as sp
from IPython.display import *
sp.init_printing(use_latex=True)
```

```python
# définition des variables symboliques et des lois de Newton, voir figure ci-haut
x,p_0,p_L,mu1,mu2,L,B=sp.symbols('x,p_0,p_L,mu_I,mu_II,L,B')
tau_xz1=sp.symbols('tau_xz^I')
tau_xz2=sp.symbols('tau_xz^II')
C11,C12,C21,C22,C1,C2=sp.symbols('C_11,C_12,C_21,C_22,C1,C2')
vz1=sp.Function('v_z^I')(x)
vz2=sp.Function('v_z^II')(x)
#
# Lois de Newton
#
newton1=-mu1*sp.Derivative(vz1,x)
newton2=-mu2*sp.Derivative(vz2,x)
```

#### Le bilan est identique à celui fait dans la section 2, sans la gravité, alors le résultat est simple, pour chacun des deux fluides on a le résultat du bilan montré sur la figure ci-haut:

```python
# Bilan des forces
eq1=sp.Eq(tau_xz1(x)-(p_0-p_L)*x/L+C11)
eq2=sp.Eq(tau_xz2(x)-(p_0-p_L)*x/L+C12)
eq1=eq1.subs(tau_xz1(x),newton1)
eq2=eq2.subs(tau_xz2(x),newton2)
display(eq1)
display(eq2)
```

```python
vz1=sp.dsolve(eq1,vz1)
vz2=sp.dsolve(eq2,vz2)
#
# Chaque dsolve génère une nouvelle constante par intégration, On les renomme ici pour respecter notre
# convention. Vous pouvez vérifier en faisant display des vz avant ces remplacements. On veut avoir
# la même notation que celle de Transport Phenomena autant que possible.
#
vz1=vz1.subs(C1,C21)
vz2=vz2.subs(C1,C22)
vz1=vz1
vz2=vz2
display(vz1)
display(vz2)
```

```python
# On peut solutionner directement les 4 constantes avec les 4 conditions aux limites
#
# Ici étant donné qu'on travaille avec sympy qui fera la solution automatiquement, on ne se préocuppe
# pas de décomposer le problème en plusieurs morceaux, on va directement à 4 équations et 4 inconnues
# en fonction des 4 conditions aux limites
#
vz1=vz1.rhs                 # prendre la partie de droite nous simplifiera les manips après
vz2=vz2.rhs
condition_1=sp.Eq(vz1.subs(x,0),vz2.subs(x,0))                                 # vitesses égales à l'interface
condition_2=sp.Eq(mu1*sp.diff(vz1,x).subs(x,0),mu2*sp.diff(vz2,x).subs(x,0))   # cisaillement égaux
condition_3=sp.Eq(vz1.subs(x,B),0)                                             # non glissement
condition_4=sp.Eq(vz2.subs(x,-B),0)                                            # non glissement
display(condition_1)
display(condition_2)
display(condition_3)
display(condition_4)
```

```python
constantes=sp.solve([condition_1,condition_2,condition_3,condition_4],sp.symbols('C_11,C_12,C_21,C_22'))
display(constantes)
```

```python
vz1=vz1.subs(constantes)
vz2=vz2.subs(constantes)
display(vz1)
display(vz2)
```

### Maintenant on tracera les profils en variant les viscosités.

```python
dico1={'mu_I':0.001,'mu_II':0.002,'B':0.01, 'L':10,'p_0':100,'p_L':0}
dico2={'mu_I':0.001,'mu_II':0.005,'B':0.01, 'L':10,'p_0':100,'p_L':0}
dico3={'mu_I':0.001,'mu_II':0.010,'B':0.01, 'L':10,'p_0':100,'p_L':0}
vz1p1=vz1.subs(dico1)
vz2p1=vz2.subs(dico1)
vz1p2=vz1.subs(dico2)
vz2p2=vz2.subs(dico2)
vz1p3=vz1.subs(dico3)
vz2p3=vz2.subs(dico3)
B=B.subs(dico1)                           # B est le même dans les 3 dicos...
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize']=10,8
p=sp.plot((vz1p1,(x,0,B)),(vz2p1,(x,-B,0)),
        (vz1p2,(x,0,B)),(vz2p2,(x,-B,0)),
        (vz1p3,(x,0,B)),(vz2p3,(x,-B,0)),ylabel='V',legend=True,show=False)
p[0].label='mu_II=0.002'
p[0].line_color='red'
p[1].label=''
p[1].line_color='red'
p[2].label='mu_II=0.005'
p[2].line_color='black'
p[3].label=''
p[3].line_color='black'
p[4].label='mu_II=0.01'
p[4].line_color='blue'
p[5].label=''
p[5].line_color='blue'
p.show();
```