# Week 7 - GAN Part 3 and Evaluation Metrics Notebook

In this notebook, we will solve questions on GANs and evaluation metrics of generative models in general.

 - This notebook is prepared using PyTorch. However, you can use any Python package you want to implement the necessary functions in questions.
 - If the question asks you to implement a specific function, please do not use its readily available version from a package and implement it yourself.

## Question 1

Please answer the questions below:

1. Please explain the concept of batch normalization. What problem does it solve and how?
2. What is spectral normalization and why do we use it?
3. What is the meaning of class-conditional GAN? How do we make GANs class conditional?
4. What are the main differences between StyleGAN-v1 and StyleGAN-v2?
5. Why is it so hard to quantitatively evaluate generative models?
6. What assumptions are we making on the data/model distribution when using Inception Score and Frechet Inception distance?

You can write your answer for each question in the markdown cell below:

**Please write your answer for each question here**

## Question 2

**Part 1**: Implement regular batch normalization for convolutional layers. Definition of the function and the input to test is given.

For reference, you can use the batch normalization paper given [here](https://arxiv.org/abs/1502.03167). Also, you can refer to the lecture slides.

Please note that we will implement a simple version of batch normalization, and in this simple version we will not be using *running mean*, *running variance*, and *momentum* parameters.

```python
import torch
torch.manual_seed(0)

batch_size = 16
num_channels = 12

input_tensor = torch.normal(3, 10, size=(batch_size, num_channels, 24, 24))
```

```python
def batch_normalization_conv(X, gamma, beta, eps=1e-8):
  """Performs batch normalization operation for convolutional layer output
    Args:
        X: input tensor (N x C x H x W)
        gamma: scale variable
        beta: offset variable
        eps: epsilon value
    Returns:
        the resulting tensor of the batch norm operation
  """
  #######################
  # Write code here
  #######################
  pass
```

```python
print("Expected Output:")
print(batch_normalization_conv(input_tensor, gamma=1, beta=0)[0,0,0,:])
```

    Expected Output:
    tensor([-1.1380, -1.1645, -0.2613, -0.4449,  0.8398,  0.6829, -0.3268, -2.1290,
             0.3125, -1.2757,  0.3403,  0.2984,  0.1098,  1.2294,  1.1083, -0.2580,
            -1.3651, -1.7090,  0.5573,  0.7845,  0.5895, -1.5679, -0.3522,  1.8458])

```python
input_tensor[0,0,0,:]
```

    tensor([ -8.2584,  -8.5236,   0.4942,  -1.3388,  11.4871,   9.9201,  -0.1601,
            -18.1522,   6.2227,  -9.6333,   6.4998,   6.0813,   4.1984,  15.3766,
             14.1678,   0.5272, -10.5265, -13.9593,   8.6665,  10.9351,   8.9884,
            -12.5510,  -0.4136,  21.5301])

Expected output:

tensor([-1.1380, -1.1645, -0.2613, -0.4449,  0.8398,  0.6829, -0.3268, -2.1290,
         0.3125, -1.2757,  0.3403,  0.2984,  0.1098,  1.2294,  1.1083, -0.2580,
        -1.3651, -1.7090,  0.5573,  0.7845,  0.5895, -1.5679, -0.3522,  1.8458])

**Part 2**: Implement class-conditional batch normalization for convolutional layers. You can copy-paste and modify your code from part 1 and use the same input above with the given **Y** vector below. You can refer to the lecture slides for the pseudocode.

This part is a bit tricky since we cannot directly use the class labels as inputs to a feed-forward neural network.

We therefore use the embeddings of the classes instead. We define 10-dimensional embeddings to represent our $y \in {0, 1}$ classes as float vectors.

We then randomly generate 0 and 1 values with the amount of **batch_size** and get their embeddings.

In our function, we will imitate a feed-forward neural network to implement class-conditional batch normalization, so we also define the weights and biases of this very simple perceptron as *gamma_w*, *gamma_b*, *beta_w*, and *beta_b*.

```python
import torch.nn as nn

# Assuming binary classification (binary labels)
num_classes = 2
embedding_dim = 10

# 10-dimensional embeddings for two classes: 2 x 10
class_embeddings = nn.Embedding(num_classes, embedding_dim)

# 16 random labels of 0 and 1
input_labels = torch.randint(0, 2, size=(batch_size,))

# Get class embeddings
input_label_embeddings = class_embeddings(input_labels)

gamma_w = torch.randn(embedding_dim, num_channels)
gamma_b = torch.zeros(1, num_channels)
beta_w = torch.randn(embedding_dim, num_channels)
beta_b = torch.zeros(1, num_channels)
```

```python
def cond_batch_normalization_conv(X, Y, gamma_w, gamma_b, beta_w, beta_b, eps=1e-8):
  """Performs conditional batch normalization operation for convolutional layer output
    Args:
        X: input tensor (N x C x H x W)
        Y: input labels (N x emb_dim)
        gamma_w: scale weights (emb_dim x C)
        gamma_b: scale bias (1 x C)
        beta_w: offset weights (emb_dim x C)
        beta_b: offset bias (1 x C)
        eps: epsilon value
    Returns:
        the resulting tensor of the batch norm operation
  """
  #######################
  # Write code here
  #######################
  pass
```

```python
print("Expected Output:")
print(cond_batch_normalization_conv(input_tensor, input_label_embeddings, gamma_w, gamma_b, beta_w, beta_b)[0, 0, 0, :].data)
```

    Expected Output:
    tensor([-4.8654, -4.9883, -0.8110, -1.6601,  4.2812,  3.5554, -1.1141, -9.4485,
             1.8426, -5.5023,  1.9710,  1.7771,  0.9049,  6.0829,  5.5230, -0.7957,
            -5.9161, -7.5062,  2.9747,  4.0255,  3.1238, -6.8539, -1.2315,  8.9334])

Expected output:

tensor([-4.8654, -4.9883, -0.8110, -1.6601,  4.2812,  3.5554, -1.1141, -9.4485,
         1.8426, -5.5023,  1.9710,  1.7771,  0.9049,  6.0829,  5.5230, -0.7957,
        -5.9161, -7.5062,  2.9747,  4.0255,  3.1238, -6.8539, -1.2315,  8.9334])

## Question 3

Implement the adaptive instance normalization (AdaIN) from StyleGAN. You can refer to the lecture slides or the StyleGAN paper [here](https://arxiv.org/abs/1812.04948).

Adaptive instance normalization is used in StyleGAN to incorporate the *style* information to the network through combining learned affine transformations and feature maps produced by convolutions.

AdaIN operation is defined mathemtically with the following equation:

\begin{equation}
\text{AdaIN}(\mathbf{x}_i, \mathbf{y}) = \mathbf{y}_{s, i}\frac{\mathbf{x}_i - \mu(\mathbf{x}_i)}{\sigma(\mathbf{x}_i)} + \mathbf{y}_{b,i}
\end{equation}

which takes the feature map $\mathbf{x}_i$ and the style vector $\mathbf{y}$ as parameters. Essentially, the operation normalizes the feature maps, scales it with half of the style vector and shifts it with the other half. Representations $\mathbf{y}_s$ and $\mathbf{y}_b$ correspond to *shift* and *bias* and they are simply two halves of the style vector $\mathbf{y} = (\mathbf{y}_s, \mathbf{y}_b)$

```python
input_feature_map = torch.randn(batch_size, num_channels, 24, 24)
style_vector = torch.randn(batch_size, 2 * num_channels)
```

```python
def adaptive_instance_normalization(X, y, eps=1e-8):
  """Performs adaptive instance normalization on the given feature map X with the
  style input y
    Args:
      X: Feature map (N x C x W x H)
      y: Style vector (N x 2C)
    Returns:
      The resulting tensor from the operation
  """
  pass
```

```python
print(adaptive_instance_normalization(input_feature_map, style_vector)[0,0,0,:])
```

    tensor([9.5492e-01, 3.1419e-01, 3.7104e+00, 1.3578e+00, 1.9867e+00, 2.0013e+00,
            3.8199e+00, 3.2714e+00, 3.1662e-04, 2.9241e+00, 1.2787e+00, 3.0599e+00,
            3.0669e+00, 4.9342e-01, 5.8750e-01, 2.1606e+00, 2.4507e+00, 2.4924e+00,
            2.1146e+00, 2.1017e+00, 2.4752e+00, 2.3877e+00, 3.2252e+00, 4.0635e+00])

Expected output:

tensor([9.5492e-01, 3.1419e-01, 3.7104e+00, 1.3578e+00, 1.9867e+00, 2.0013e+00,
        3.8199e+00, 3.2714e+00, 3.1662e-04, 2.9241e+00, 1.2787e+00, 3.0599e+00,
        3.0669e+00, 4.9342e-01, 5.8750e-01, 2.1606e+00, 2.4507e+00, 2.4924e+00,
        2.1146e+00, 2.1017e+00, 2.4752e+00, 2.3877e+00, 3.2252e+00, 4.0635e+00])

## Question 4

Implement a function that calculates the Frechet Inception Distance score from given real examples and fake examples.

You can refer to its original paper [here](https://arxiv.org/abs/1706.08500).

\begin{equation}
\text{FID} = ||\mu_1 – \mu_2||^2 + \text{Tr}(C_1 + C_2 – 2\sqrt{C_1C_2})
\end{equation}

where $\mu_1$ and $\mu_2$ are the feature-wise means of the real and generated samples, respectively. In addition, $C_1$ and $C_2$ are the covariance matrices of the real and generated samples, sometimes also referred as sigma ($\Sigma$).

```python
import torch
import torch.nn as nn
from torchvision.models import inception_v3
from torchvision.datasets import MNIST
from torchvision.transforms import Resize

%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
mnist_dataset = MNIST(root=".", download=True)

# Get 200 MNIST examples
mnist_examples = mnist_dataset.data[:32].type(torch.FloatTensor)
mnist_examples /= 255

resizer = Resize(299)
mnist_examples = resizer(mnist_examples)

plt.imshow(mnist_examples[2].numpy(), interpolation='nearest', cmap='gray')

# Reshape the images to 3D to give them as inputs to the Inception network
mnist_examples = mnist_examples.unsqueeze(dim=3).repeat((1, 1, 1, 3))
```

```python
# Create 200 noisy images as the same sizes with MNIST
noisy_examples = torch.randn(32, 299, 299, 1).clip(0, 1).repeat((1, 1, 1, 3))

plt.imshow(noisy_examples[2].numpy(), interpolation='nearest', cmap='gray')
```

```python
# Download the pretrained inception v3 model
inception_model = inception_v3(pretrained=True)
# Replace the classification layer with an identity layer to get the activations
inception_model.fc = nn.Identity()
# Evaluation mode
inception_model.eval()

with torch.no_grad():
  mnist_features = inception_model(mnist_examples.permute(0, 3, 1, 2))
  noisy_features = inception_model(noisy_examples.permute(0, 3, 1, 2))
```

    /usr/local/lib/python3.7/dist-packages/torch/nn/functional.py:718: UserWarning: Named tensors and all their associated APIs are an experimental feature and subject to change. Please do not use them for anything important until they are released as stable. (Triggered internally at  /pytorch/c10/core/TensorImpl.h:1156.)
      return torch.max_pool2d(input, kernel_size, stride, padding, dilation, ceil_mode)

```python
def calculate_fid(real_features, fake_features):
  """Calculates the Frechet Inception Distance of the given real and fake features
  to measure the similarity of two data distributions

  Args:
    real_features: Features taken from the real images (N x D)
    fake_features: Features taken from the fake images (N x D)
  Returns:
    A scalar number as the distance between two data distributions
  """
  pass
```

```python
calculate_fid(mnist_features.numpy(), noisy_features.numpy())
```

Expected output: 396.4476952524345

```python
calculate_fid(mnist_features.numpy()[:16], mnist_features.numpy()[16:])
```

Expected output: 95.29507726881653

inzva x METU ImageLab Joint Program/Week 7 - GANs 3 and Evaluation Metrics/Week_7_GAN_Part_3_and_Evaluation_Metrics.ipynb


```python

```

# Unconstrained Scalar Optimization

One of the problems studied in introductory calculus courses is the minimization or maximization of a function of a single variable. That is, given a function $f(x)$, find values $x^*$ such that $f(x^*) \leq f(x)$, or $f(x^*) \geq f(x)$, for all $x$ in an interval containing $x^*$. Such points are called local optima. If the derivative exists at all points in a given interval, then the local optima are found by solving for values $x^*$ that satisfy

\begin{align}
f'(x^*) = 0
\end{align}

Let's see how we can put this to work in a process engineering context.

## Example: Reactor for a Series Reaction

A desired product $B$ is produced as intermediate in a series reaction

\begin{align}
A \overset{k_A}{\longrightarrow} B \overset{k_B}{\longrightarrow} C
\end{align}

where $A$ is a raw material and $C$ is a undesired by-product. The reaction operates at temperature where the rate constants are $k_A = 0.5\ \mbox{min}^{-1}$ and $k_A = 0.1\ \mbox{min}^{-1}$. The raw material is available as solution with a concenration $$C_{A,f} = 2.0\ \mbox{moles/liter}$.

A 100 liter tank is avialable to run the reaction.

1. If the goal is obtain the maximum possible concentration of $B$, and the tank is operated as a continuous stirred tank reactor, what should be the flowrate?

2. What is the production rate of $B$ at maximum concentration?

3. [Bonus] Would it be better to operate the tank as a batch reactor?

## Continuous Stirred Tank Reactor

The reaction dynamics for an isothermal continuous stirred tank reactor with a volume $V = 40$ liters and feed concentration $C_{A,f}$ are modeled as

\begin{align}
V\frac{dC_A}{dt} & = q(C_{A,f} - C_A) - V k_A C_A \\
V\frac{dC_B}{dt} & = - q C_B + V k_A C_A - V k_B C_B
\end{align}

At steady-state the material balances become

\begin{align}
0 & = q(C_{A,f} - \bar{C}_A) - V k_A \bar{C}_A \\
0 & = - q \bar{C}_B + V k_A \bar{C}_A - V k_B \bar{C}_B
\end{align}

which can be solved for $C_A$

\begin{align}
\bar{C}_A & = \frac{qC_{A,f}}{q + Vk_A} \\
\end{align}

and then for $C_B$

\begin{align}
\bar{C}_B & = \frac{q V k_A C_{A,f}}{(q + V k_A)(q + Vk_B)}
\end{align}

The numerator is first-order in flowrate $q$, and the denominator is quadratic.  This is consistent with an intermediate value of $q$ corresponding to a maximum concentration $\bar{C}_B$.

The next cell plots $\bar{C}_B$ as a function of flowrate $q$.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

def cstr(q):
    return q*V*kA*CAf/(q + V*kB)/(q + V*kA)

q = np.linspace(0,30,200)
plt.plot(q, cstr(q))
plt.xlabel('flowrate q / liters per minute')
plt.ylabel('concentration C_B / moles per liter')
plt.title('Outlet concentration for a CSTR')
plt.grid()
```

We see that, for the parameters given, there is an optimal flowrate somewhere between 5 and 10 liters per minute.

## Analytical Solution using Calculus

As it happens, this problem has an interesting analytical solution that can be found by hand, and which can be used to check the accuracy of numerical solutions. Setting the first derivative of $\bar{C}_B$ to zero,

\begin{align}
\left.\frac{d\bar{C}_B}{dq}\right|_{q^*} = \frac{V k_A C_{A,f}}{(q^* + V k_A)(q^* + Vk_B)} - \frac{q^* V k_A C_{A,f}}{(q^* + V k_A)^2(q^* + Vk_B)} - \frac{q^* V k_A C_{A,f}}{(q^* + V k_A)(q^* + Vk_B)^2} = 0
\end{align}

Clearing out the non-negative common factors yields

\begin{align}
1 - \frac{q^*}{(q^* + V k_A)} - \frac{q^*}{(q^* + Vk_B)} = 0
\end{align}

and multiplying by the non-negative denominators produces

\begin{align}
{q^*}^2 + q^*V(k_A + k_B) + V^2k_Ak_B - q^*(q^* + Vk_B) - q^*(q^* + Vk_A) = 0
\end{align}

Expanding these expressions followed by arithmetic cancelations gives the final result

\begin{align}
q^*  =  V\sqrt{k_Ak_B}
\end{align}

which shows the optimal dilution rate, $\frac{q^*}{V}$, is equal the geomtric mean of the rate constants.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

qmax = V*np.sqrt(kA*kB)
CBmax = cstr(qmax)
print('Flowrate at maximum CB = ', qmax, 'liters per minute.')
print('Maximum CB =', CBmax, 'moles per liter.')
print('Productivity = ', qmax*CBmax, 'moles per minute.')
```

    Flowrate at maximum CB =  8.94427190999916 liters per minute.
    Maximum CB = 0.9549150281252629 moles per liter.
    Productivity =  8.541019662496845 moles per minute.

## Numerical Solution using `scipy.minimize_scalar`

In many applications, a simple analysis leading to an analytical solution is either difficult or intractable. In these cases, numerical solution solution is the only choice for calculating a solution to the optimization problem.

The following cell demonstrates the use of the `scipy` optimization library. The  value of $q$ that maximizes $\bar{C}_B$ can be found using the function `scipy.minimize_scalar`. Note the change in sign needed to convert the  `cstr` function from a maximization to a minimization problem for use with `minimize_scalar`.

```python
from scipy.optimize import minimize_scalar

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

def cstr(q):
    return -q*V*kA*CAf/(q + V*kB)/(q + V*kA)

results = minimize_scalar(cstr, bracket=[0,50])
results
```

         fun: -0.9549150281252629
        nfev: 18
         nit: 14
     success: True
           x: 8.944272016904383

The following cell shows how to access the components of the solution obtained from `scipy.minimize_scalar`.

```python
if results.success:
    qmax = results.x
    CBmax = -results.fun
    print('Flowrate at maximum CB = ', qmax, 'liters per minute.')
    print('Maximum CB =', CBmax, 'moles per liter.')
    print('Productivity = ', qmax*CBmax, 'moles per minute.')
else:
    print('No solution found.')
```

    Flowrate at maximum CB =  8.944272016904383 liters per minute.
    Maximum CB = 0.9549150281252629 moles per liter.
    Productivity =  8.54101976458225 moles per minute.

## [Bonus Discussion] Batch Reactor

A material balance for an isothermal stirred batch reactor with a volume $V = 40$ liters and an initial concentration $C_{A,f}$ is given by

\begin{align}
V\frac{dC_A}{dt} & = - V k_A C_A \\
V\frac{dC_B}{dt} & = - V k_A C_A - V k_B C_B
\end{align}

Eliminating the common factor $V$

\begin{align}
\frac{dC_A}{dt} & = - k_A C_A \\
\frac{dC_B}{dt} & = - k_A C_A - V k_B C_B
\end{align}

With an initial concentration $C_{A,f}$. A numerical solution to these equations is shown in the following cell.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
from scipy.integrate import odeint

V = 40     # liters
kA = 0.5   # 1/min
kB = 0.1   # l/min
CAf = 2.0  # moles/liter

def batch(X, t):
    CA, CB = X
    dCA_dt = -kA*CA
    dCB_dt = kA*CA - kB*CB
    return [dCA_dt, dCB_dt]

t = np.linspace(0,30,200)
soln = odeint(batch, [CAf,0], t)
plt.plot(t, soln)
plt.xlabel('time / minutes')
plt.ylabel('concentration / moles per liter')
plt.title('Batch Reactor')
plt.legend(['$C_A$','$C_B$'])
```

To find the maximum value, we first write a function to compute $C_B$ for any value of time $t$.

```python
def CB(t):
    soln = odeint(batch, [CAf, 0], [0, t])
    return soln[-1][1]
```

We gain use `minimize_scalar` to find the value of $t$ that minimizes the negative value of $C_B(t)$.|

```python
from scipy.optimize import minimize_scalar
minimize_scalar(lambda t: -CB(t), bracket=[0,50])
```

         fun: -1.3374806339222158
        nfev: 23
         nit: 19
     success: True
           x: 4.023594924340666

```python
tmax = minimize_scalar(lambda t: -CB(t), bracket=[0,50]).x

print('Concentration C_B has maximum', CB(tmax), 'moles/liter at time', tmax, 'minutes.')
print('Maximum productivity of the batch reactor is', V*CB(tmax)/tmax, 'moles per minute.')
```

    Concentration C_B has maximum 1.3374806339222158 moles/liter at time 4.023594924340666 minutes.
    Maximum productivity of the batch reactor is 13.296374601042967 moles per minute.

```python

```

```python

```

```python

```


```python
from logicqubit.logic import *
from cmath import *
import numpy as np
import sympy as sp
from scipy.optimize import *
import matplotlib.pyplot as plt
```

```python
def qft(qr):
    for i in range(len(qr)):
        for j in range(i):
            qr[i].CU1(qr[j], pi/float(2**(i-j)))
        qr[i].H()

def iqft(qr): # transformada quântica de Fourier inversa
    for i in range(len(qr)):
        for j in range(i):
            qr[i].CU1(qr[j], -pi/float(2**(i-j)))
        qr[i].H()

def swap(s1, s2):
    s2.CX(s1)
    s1.CX(s2)
    s2.CX(s1)

def ansatz(q, theta):
    return q.RY(theta)
```

```python
def expectation(theta):
    # H = Sz
    # <psi|H|psi> = <psi| [[1,0],[0,-1]] |psi>
    # |0><0|=[[1,0],[0,0]] e |1><1|=[[0,0],[0,1]]
    # <psi|H|psi> = <psi|0><0|psi> - <psi|1><1|psi> = <0> - <1>

    logicQuBit  = LogicQuBit(1)
    q = Qubit()

    ansatz(q, theta)

    res = logicQuBit.Measure([q])
    return res[0]-res[1]
```

```python
expectation(0)
```

    1.0

```python
params = np.linspace(0.0, 2 * np.pi, 25)
data = [expectation(theta) for theta in params]

plt.xlabel('angulo')
plt.ylabel('valor esperado')
plt.plot(params, data)
plt.show()
```

```python
theta = 0.0
minimum = minimize(expectation, theta, method='Nelder-Mead', options={'initial_simplex': np.array([[0.0], [0.05]]), 'xatol': 1.0e-2})
print(minimum)
```

     final_simplex: (array([[3.14375],
           [3.1375 ]]), array([-0.99999767, -0.99999163]))
               fun: -0.9999976729291358
           message: 'Optimization terminated successfully.'
              nfev: 28
               nit: 14
            status: 0
           success: True
                 x: array([3.14375])

```python

```


# Lorentz System

For reference, c.f. [wikipedia](https://en.wikipedia.org/wiki/Lorenz_system), [mathworld](http://mathworld.wolfram.com/LorenzAttractor.html), and [a solution by _Mathematica_](http://reference.wolfram.com/language/example/VisualizeTheLorenzAttractor.html).

For some $T > 0$ given, for $\forall t \in [0, T]$,
\begin{align}
  \frac{\mathrm{d} x}{\mathrm{d} t} &= \sigma ( y - x ), \\
  \frac{\mathrm{d} y}{\mathrm{d} t} &= x ( \rho - z ) - y, \\
  \frac{\mathrm{d} z}{\mathrm{d} t} &= x y - \beta z,
\end{align}
with boundary conditions $x(0) = x_0$, $y(0) = y_0$, and $z(0) = z_0$, for some $(x_0, y_0, z_0)$ given.

```python
%matplotlib notebook

import numpy as np
import tensorflow as tf
from matplotlib import pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from datetime import datetime
from collections import namedtuple
import random as rn
import os

# Global parameters
_SEED = 42
_EPSILON = 1e-8

# For reproducibility
os.environ['PYTHONHASHSEED'] = '0'
np.random.seed(_SEED)
rn.seed(_SEED)
tf.set_random_seed(_SEED)
```

    /Users/shuiruge/anaconda/lib/python3.6/importlib/_bootstrap.py:205: RuntimeWarning: compiletime version 3.5 of module 'tensorflow.python.framework.fast_tensor_util' does not match runtime version 3.6
      return f(*args, **kwds)

## Solving by Neural Network

```python
# -- Helpers

def ln_norm(x, n, name=None):
  """L`n`-norm of tensor `x`."""
  with tf.name_scope(name, 'L{}_Norm'.format(n), [x]):
    x = tf.convert_to_tensor(x)
    norm = tf.reduce_mean(x**n) ** (1/n)
    return norm
```

```python
def make_neural_network(x, hidden_layers, output_shape, name=None):
  """Implement the generic neural network with the dense linear output
  layer.
  Args:
    x: Tensor-like, as the input of the neural network. It's shape is
        of `[batch_size] + x_shape`.
    hidden_layers: List of objects of the classes in `tf.layers`.
    output_shape: List of integers.
  Returns:
    The output tensor of the neural network.
  """
  with tf.name_scope(name, 'NeuralNetwork', [x]):
    x = tf.convert_to_tensor(x)

    # Hidden layers
    hidden = x  # initialize.
    for layer in hidden_layers:
      hidden = layer(hidden)

    # Output layer
    flatten_hidden = tf.layers.flatten(hidden)
    output_size = sum(flatten(output_shape))
    output = tf.layers.dense(flatten_hidden, output_size)
    output = tf.reshape(output, [-1]+output_shape)

  return output

def flatten(nested_list):
    """Helper. (Recursively) flatten an arbitrarily nested list."""
    if nested_list == []:
        return nested_list
    if isinstance(nested_list[0], list):
        return flatten(nested_list[0]) + flatten(nested_list[1:])
    return nested_list[:1] + flatten(nested_list[1:])
```

Boundary is "re-parameterized" by $L^n \left( \zeta(0) - \zeta_0 \right)$ for $\zeta \in (x, y, z)$.

```python
def make_boundary_loss(output_boundary, boundary_target, lambda_, n=2, name=None):
  """Make the boundary term in loss, vanishing of which makes the fitting
  of boundary.

  $ \mathcal{L}_{\textrm{boundary}} := \lambda * L^n (\zeta(0) - \zeta_0)$
  for $\zeta \in (x, y, z)$.

  Args:
    XXX
    n: Positive integer, employ L-`n` norm.
    lambda: Positive float.
  Returns:
    Scalar.
  """
  with tf.name_scope(name, 'BoundaryLoss',
        [output_boundary, boundary_target]):
    output_boundary = tf.convert_to_tensor(output_boundary)
    boundary_target = tf.convert_to_tensor(boundary_target)
    print('test! output_boundary.shape', output_boundary.shape)
    print('test! boundary_target.shape', boundary_target.shape)
    # `delta` shall be vanishing
    delta = output_boundary - boundary_target
    boundary_loss = lambda_ * ln_norm(delta, n)
    return boundary_loss
```

```python
ODEParameter = namedtuple('ODEParameter', 'sigma, rho, beta')

def get_ode(ode_param):
  def ode(inputs, outputs, name=None):
    with tf.name_scope(name, 'ODE', [inputs, outputs]):
      inputs = tf.convert_to_tensor(inputs)
      outputs = tf.convert_to_tensor(outputs)
      x, y, z = tf.unstack(outputs, axis=1)
      f_value = [
          ode_param.sigma * (y - x),
          x * (ode_param.rho  - z) - y,
          x * y - ode_param.beta * z,
      ]
      f_value = tf.stack(f_value, axis=1)
      return f_value
  return ode

def make_ode_loss(inputs, outputs, grads, ode, n=2, name=None):
  """Make the ODE term in loss, vanishing if the ODE is satisfied.
  Args:
    inputs:
    outputs:
    grads:
    ode:
    ode_param: An instance of `ODEParameter`.
    n:
  Returns:
    Scalar.
  """
  with tf.name_scope(name, 'ODELoss', [inputs, outputs, grads]):
    inputs = tf.convert_to_tensor(inputs)
    outputs = tf.convert_to_tensor(outputs)
    grads = tf.convert_to_tensor(grads)
    deltas = grads - ode(inputs, outputs)
    ode_loss = ln_norm(deltas, n)
    return ode_loss
```

```python
ResultOps = namedtuple('ResultOps', 'inputs, outputs, output_bulk, '
    'output_boundary, grads, ode_loss, boundary_loss, loss')

def make_loss(bulk, boundary, boundary_target, make_solution,
              ode_param, lambda_, n=2, name=None):
  """Implements the loss.

  Args:
    bulk: Tensor-like, with the shape `[batch_size, param_space_dim]`.
    boundary: Tensor-like, with the shape `[n_boundaries, param_space_dim]`.
    boundary_target: Tensor-like, with the shape `[n_boundaries, phase_space_dim]`.
    make_solution: Callable that maps input of the shape `[None, param_space_dim]`
        to the neural network output of the shape `[None, phase_space_dim]`.
    ode_param: An instance of `ODEParameter`.
  Returns:
    Scalar.
  """
  with tf.name_scope(name, 'Loss', [bulk, boundary, boundary_target]):
    bulk = tf.convert_to_tensor(bulk)
    boundary = tf.convert_to_tensor(boundary)
    boundary_target = tf.convert_to_tensor(boundary_target)

    inputs = tf.concat([bulk, boundary], axis=0)
    outputs = make_solution(inputs)
    grads = tf.gradients(outputs, [inputs])[0]

    batch_size = bulk.get_shape().as_list()[0]
    n_boundaries = boundary.get_shape().as_list()[0]
    output_bulk, output_boundary = tf.split(
        outputs, [batch_size, n_boundaries], axis=0)

    ode = get_ode(ode_param)
    ode_loss = make_ode_loss(inputs, outputs, grads, ode, n=n)
    boundary_loss = make_boundary_loss(
        output_boundary, boundary_target, lambda_, n=n)
    loss = ode_loss + boundary_loss + 100 / (ln_norm(outputs, 2) + 1e-8)

    return ResultOps(inputs, outputs, output_bulk, output_boundary,
                     grads, ode_loss, boundary_loss, loss)
```

```python
def test(T, ode_param, boundary_target, n=2, lambda_=1.0, n_samples=10**3,
         hidden_layers=[lambda x: tf.layers.dense(x, 10)]*3,
         optimizer=None, n_iters=10**4, skip_step=10):

    # -- Build graph

    tf.reset_default_graph()

    param_space_dim = 1
    phase_space_dim = 3
    bulk = tf.placeholder(shape=[n_samples, param_space_dim],
                          dtype='float32', name='bulk')
    boundary = np.zeros([1, param_space_dim], dtype='float32')
    boundary_target = np.array(boundary_target, dtype='float32')

    def make_solution(x):
        return make_neural_network(x, hidden_layers, [phase_space_dim])

    ops = make_loss(bulk, boundary, boundary_target, make_solution,
                    ode_param, lambda_, n=n)

    # -- Optimization

    optimizer = optimizer if optimizer else tf.train.AdamOptimizer(1e-3)
    train_op = optimizer.minimize(ops.loss)

    with tf.Session() as sess:

        # Initialize all `tf.Variable`s, explicit or implicit
        sess.run(tf.global_variables_initializer())

        # -- Optimizing

        def get_bulk_value():
            """For feed-dict."""
            bulk_value = np.random.uniform(0, T,
                size=[n_samples, param_space_dim])
            bulk_value = bulk_value.astype('float32')
            bulk_value = np.sort(bulk_value, axis=0)
            return bulk_value

        # For visualization
        fig = plt.figure()
        ax = fig.add_subplot(111)
        ax = fig.gca(projection='3d')  # if 3-D.
        plt.ion()
        fig.show()
        fig.canvas.draw()

        # Iterations
        for i in range(n_iters):
            step = i + 1
            iter_ops = [train_op, ops.loss, ops.ode_loss, ops.boundary_loss,
                        ops.output_bulk]
            bulk_value = get_bulk_value()
            result = sess.run(iter_ops, feed_dict={bulk: bulk_value})
            _, loss, ode_loss, boundary_loss, output_bulk = result

            # Visualization
            if step % skip_step == 0:
                ax.clear()
                ax.plot(output_bulk[:,0], output_bulk[:,1], output_bulk[:,2])
                ax.set_title('{0} | {1:.3f} {2:.3f} {3:.3f}'\
                             .format(step, loss, ode_loss, boundary_loss))
                fig.canvas.draw()
```

```python
%%time
hidden_layers = []
for i in range(3):
    hidden_layers += [
        lambda x: tf.layers.dense(x, 300, activation=tf.nn.leaky_relu),
        lambda x: tf.layers.dropout(x),
    ]
test(T=40, ode_param=ODEParameter(10, 28, 8/3), boundary_target=[[1, 1, 1]],
     n_iters=10**5, hidden_layers=hidden_layers,
     optimizer=tf.train.AdamOptimizer(1e-3), lambda_=100)
```

    test! output_boundary.shape (1, 3)
    test! boundary_target.shape (1, 3)

    <IPython.core.display.Javascript object>

    CPU times: user 10h 39min, sys: 44min 22s, total: 11h 23min 23s
    Wall time: 10h 47min 13s

* It seems that optimizer hints a local minimum which is too deep to escape. This local minimum is an attractor of the Lorentz system, i.e. at $(x, y, z) = 0$.

## Solving by Traditional Method


# Atomic Orbitals
### Christina Lee
### Physics Prerequisites: Quantum Mechanics

Electrons around a nucleus.  Do they look like little well behaved planets orbiting a sun?

NOPE!

We get spread out blobs in special little patterns called orbitals.  Here, we will look at their shapes and properties a bit.  Today we will look at graphs in 1D and 2D.

The Hamiltonian for our problem is:

\begin{equation}
{\cal H}\Psi(x) =\left[ -\frac{\hbar}{2 m} \nabla^2 - \frac{Z e^2}{4 \pi \epsilon_0 r}\right]\Psi(x) = E \Psi(x)
\end{equation}
with
\begin{equation}
\nabla^2= \frac{1}{r^2}\frac{\partial}{\partial r} \left(
r^2 \frac{\partial}{\partial r}
\right)+
\frac{1}{r^2 \sin \theta} \frac{\partial}{\partial \theta} \left(
\sin \theta \frac{\partial}{\partial \theta}
\right)+
\frac{1}{r^2 \sin^2 \theta} \frac{\partial^2}{\partial \phi^2}
\end{equation}

To solve this problem, we begin by guessing a solution with seperated Radial and Angular variables,
\begin{equation}
\Psi(x) = R(r) \Theta ( \theta,\phi)
\end{equation}

\begin{equation}
\frac{E r^2 R(r)}{2r R^{\prime}(r) + r^2 R^{\prime \prime}(r)}=
\frac{\left( \frac{1}{\sin \theta} \frac{\partial}{\partial \theta} \left(
\sin \theta \frac{\partial \Theta(\theta,\phi)}{\partial \theta}
\right)+
\frac{1}{\sin^2 \theta} \frac{\partial^2 \Theta(\theta,\phi)}{\partial \phi^2}\right)      }{\Theta( \theta, \phi)}
=C
\end{equation}

Instead of going into the precise mechanisms of solving this differential equation from scratch seperate equations here, I'm only going to present the solution in terms of special functions that we can send to the [GNU Scientific Library](https://www.gnu.org/software/gsl/), GSL.

The overall eigenstate is given by
\begin{equation}
\Psi(\vec{x})= N R^{n,l}(\rho (r) )  Y^m_l (\theta,\phi)
\end{equation}

with the angular solution:
\begin{equation}
        Y^m_l(θ,ϕ) = (-1)^m e^{i m \phi} P^m_l (\cos(θ))
\end{equation}
where $P^m_l (x)$ is the [associated Legendre Polynomial](http://mathworld.wolfram.com/AssociatedLegendrePolynomial.html).

The radial solution is:
\begin{equation}
    R^{n,l} (\rho) = \rho^l e^{-\rho/2} L^{2 l+1}_{n-l-1} (\rho)
\end{equation}
where $L^{2 l+1}_{n-l-1}(\rho)$ is the [generalized Laguerre polynomial](http://mathworld.wolfram.com/LaguerrePolynomial.html).
Instead of $r$, we deal with $\rho$, a scaled function of $r$:
\begin{equation}
    \rho=\frac{2r}{n a_0}
\end{equation}
Everything is normalized by:
\begin{equation}
    N=\sqrt{\left(\frac{2}{n}\right)^3 \frac{(n-l-1)}{2n(n+l)!}}
\end{equation}

```julia
using GSL;    #GSL holds the special functions
using Plots;
pyplot()
```

    Plots.PyPlotBackend()

#### Cell to Evaluate
What's below is a bunch of definitions that makes our calculations easier later on.  Here I utilize the GNU scientific library, GSL imported above, to calculate the special functions.

#### Programming Tip!
Even though its not necessary, specifying the type of inputs to a function through `m::Int` helps prevent improper inputs and allows the compiler to perform additional optimizations.  Julia also implements Abstract Types, so we don't have to specify the exact type of Int.  Real allows and numerical, non-complex type.

Type greek characters in Jupyter notebooks via LaTeX syntax.  ex: \alpha+tab

The function `Orbital` throws `DomainError()` when `l` or `m` do not obey their bounds.  Julia supports a wide variety of easy to use error messages.

```julia
a0=1; #for convenience, or 5.2917721092(17)×10−11 m

# The unitless radial coordinate
ρ(r,n)=2r/(n*a0);

#The θ dependence
function Pmlh(m::Int,l::Int,θ::Real)
    return (-1.0)^m *sf_legendre_Plm(l,m,cos(θ));
end

#The θ and ϕ dependence
function Yml(m::Int,l::Int,θ::Real,ϕ::Real)
    return  (-1.0)^m*sf_legendre_Plm(l,m,cos(θ))*exp(im*m*ϕ)
end

#The Radial dependence
function R(n::Int,l::Int,ρ::Real)
    if isapprox(ρ,0)
        ρ=.01
    end
     return sf_laguerre_n(n-l-1,2*l+1,ρ)*exp(-ρ/2)*ρ^l
end

#A normalization: This is dependent on the choice of polynomial representation
function norm(n::Int,l::Int)
    return sqrt((2/n)^3 * factorial(n-l-1)/(2n*factorial(n+l)))
end
"Orbital Components"
```

    "Orbital Components"

```julia
#Generates an Orbital Funtion of (r,θ,ϕ) for a specificied n,l,m.
function Orbital(n::Int,l::Int,m::Int)
    if l>n    # we make sure l and m are within proper bounds
        throw(DomainError())
    end
    if abs(m)>l
        throw(DomainError())
    end
    psi(ρ,θ,ϕ)=norm(n, l)*R(n,l,ρ)*Yml(m,l,θ,ϕ);
    return psi
end
"Orbital Function"
```

    "Orbital Function"

```julia
#We will calculate in spherical coordinates, but plot in cartesian, so we need this array conversion
function SphtoCart(r::Array,θ::Array,ϕ::Array)
    x=r.*sin.(θ).*cos.(ϕ);
    y=r.*sin.(θ).*sin.(ϕ);
    z=r.*cos.(θ);
    return x,y,z;
end

function CarttoSph(x::Array,y::Array,z::Array)
    r=sqrt.(x.^2 .+y.^2 .+z.^2);
    θ=acos.(z./r);
    ϕ=atan.(y./x);
    return r,θ,ϕ;
end

"Coordinate Functions"
```

    "Coordinate Functions"

#### Parameters

For the Orbital Parameters:
\begin{equation}
0<n \qquad 0 \leq l < n \qquad -l \leq m \leq l
\qquad n,l,m \in {\mathbb Z}
\end{equation}

```julia
# Choose the Orbitals to look at
n=[3,3,2,2,2];
l=[1,2,1,1,0];
m=[0,0,1,0,0];
nΨ=length(n); # number of parameter combinations to look at

Nplot=100; # number of points to plot per dimension

Ψa=Function[] # Array of our Orbital Functions
for ii in 1:nΨ
    push!(Ψa,Orbital(n[ii],l[ii],m[ii]) )
end
"Defined parameters"
```

    "Defined parameters"

## Plotting

### Theta Slice
$l$ angular nodes exist for a given wavefunction, so as we proceed through a $\pi$ rotation, we should hit zero $l$ times in total.

$m$, the magnetic quantum number, rotates the location that the angular nodes occur. I chose one $m=1$ wavefunction, and this wavefunction intercepts zero at at a different location than its $m=0$ counterpart.

Go ahead and change the `r_loc` and `ϕ_loc` parameters to see how the slice changes as we change where we take it.

```julia
r_loc=1. # r location of slice
ϕ_loc=0. # ϕ location of slice

θ=range(0,stop=2π,length=Nplot)

plot()
for ii in 1:nΨ
    plot!(θ,abs.(Ψa[ii].(1.,θ,0.)),
        label="n=$(n[ii]) l=$(l[ii]) m=$(m[ii])")
end

plot!(xlabel="θ",ylabel="|Ψ |",
    title="Θ Slice of Wavefunction at r=$(r_loc), ϕ=$(ϕ_loc)")
```

### R Slice

The wavefunction has $n-1$ nodes in total, with $l$ of those going to angular nodes.  The remaining $n-l-1$ nodes are radial nodes, which we can see when taking a radial slice. The wavefunction hitting zero at the origin is a byproduct of angular nodes.

```julia
θ_loc=π/4 # θ location of slice
θ_loc_name="π/4" #format nice with pi
ϕ_loc=0   #  ϕ location of slice

r=range(0,stop=10,length=Nplot)

plot()
for ii in 1:nΨ
    plot!(r,abs.(Ψa[ii].(r,π/4,0.)) ,
        label="n=$(n[ii]) l=$(l[ii]) m=$(m[ii])")
end

plot!(xlabel="r",ylims=(0,.2),ylabel="|Ψ |",
    title="r Slice of Wavefunction at θ="*θ_loc_name*",ϕ=$(ϕ_loc)")
```

Next, we will look at two dimensional slices. I chose the $y=0$ plane, as the $z=0$ tends to correspond to an angular mode in several of my chosen wavefunctnions.  Much more clearly than the 1D slices, these show the presence of the nodes.

```julia
# the two dimensions we iterate over
x=range(-8,stop=8,length=Nplot)
z=range(-8,stop=8,length=Nplot)

# creating a 2D
xx=repeat(x,outer=[1,Nplot])
zz=repeat(transpose(z),outer=[Nplot,1])
yy=zeros(Float64,Nplot,Nplot)

r2, θ2, ϕ2=CarttoSph(xx,yy,zz);
```

```julia
ii=5
heatmap(x,z,abs.(Ψa[ii].(r2,θ2,ϕ2)))
plot!(xlabel="x",ylabel="z",
    title="Abs of Wavefunction n=$(n[ii]) l=$(l[ii]) m=$(m[ii])")
```

### Saved Examples

## Values on a sphere

Finally I pull out points at a fixed radius on a sphere.  By just using colored scatter points, we can still effectively get a picture of the angular behaviour of the wavefunction.

```julia
θ3=repeat(θ,outer=[1,100])
ϕ3=repeat(transpose(θ),outer=[100,1])
r3=ones(Float64,100,100)

x3,y3,z3=SphtoCart(r3,θ3,ϕ3);
```

```julia
choice=2

y3pr=reshape(y3,10000)
y3p=y3pr[y3pr.<0]
x3p=reshape(x3,10000)[y3pr.<0]
z3p=reshape(z3,10000)[y3pr.<0]
col=reshape(abs.(Ψa[choice].(r3,θ3,ϕ3)),10000)[y3pr.<0]

scatter(x3p,y3p,z3p,marker_z=col,
    label="",markerstrokewidth=0)
plot!(title="Angular Behaviour of n=$(n[choice]) l=$(l[choice]) m=$(m[choice])")
```

```julia

```

```julia

```

Happy Programming :)

```julia

```


# Finite-Volume Methods for Shocktube Problems

```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
sns.set()
```

```python
import pyflow
from pyflow.euler import PrimitiveState
from pyflow.euler.shocktube import ShocktubeProblem
```

# One-dimensional Shock Tube Problems

In this notebook, several finite-volume numerical alogirthms are presented for solving the 1-D Euler equations on the shock tube problem.  The shock tube problem consists of a flow at rest with a diaphragm separating two uniform regions at different densities and pressures. Upon removing the diaphragm at $t = t_0$, a shock wave travels into the low pressure side leaving a rarefaction in its wake. Between the rarefaction fan and the shock wave is a contact discontinuity forms, which is a remnant of the differences in the densities initially across the diaphragm. Both the pressure and velocity are constant across a contact discontinuity, which moves at the local velocity. The gas dynamics are often illustrated using a $x$-$-t$ diagram, which traces the location of the rarefaction, contact, and shock waves.

The shock tube problem is often used to test numerical algorithms for compressible flows. In this notebook, the Sod shocktube test case is solved using MacCormack's second-order, predictor-corrector method, which is compared to more sophisticated methods that are usually required for flows exhibiting shock waves and/or high pressure and density gradients.

## Euler Equations

The Euler equations in one-dimension are given as

$$
\begin{align}
& \frac{\partial \rho}{\partial t} + \frac{\partial \rho u}{\partial x} = 0 \\
& \frac{\partial \rho u}{\partial t} + \frac{\partial (\rho u^2 + p)}{\partial x} = 0 \\
& \frac{\partial \rho e}{\partial t} + \frac{\partial}{\partial x}\Big[ \left(e + p\right)u \Big] = 0
\end{align}
$$

which can be written in conservation law form

$$
\frac{\partial \mathbf{U} }{\partial t} + \frac{\partial \mathbf{F}}{\partial x} = 0
$$

where $\mathbf{U}$ is the vector of conserved variables and $\mathbf{F}(\mathbf{U})$ is the vector of fluxes. In general, these definition can be extended to any $m$-number of conservation laws and dimensions; however, for the one-dimensional Euler equations specifically we have

$$
\mathbf{U} = \begin{bmatrix}
\rho \\
\rho u \\
e
\end{bmatrix}
$$

and

$$
\mathbf{F} = \begin{bmatrix}
\rho u\\
\rho u^2 + p \\
 u (e + p)
\end{bmatrix}
$$

with

$$
p = (\gamma - 1)\left(e - \rho \frac{u^2}{2}\right)
$$

## MacCormack Method

MacCormck's method is a two-step, predictor-corrector method. In the predictor step, a forward difference approximation is used for the spatial derivative of the flux, which is then followed by a backward difference in the corrector step. The averaging of the spatial differences at the predictor ($t = n\Delta t$) and the corrector ($t = (n+1)\Delta t$) makes the method centered in both time and space. Thus, MacCormack's method is both second-order in space and time.

In one-dimension and assuming uniform spacing of the computational mesh, the method can be simplified to the following two-step, difference equations.

$$
\begin{align}
\textrm{predictor: } \qquad U^{\overline{n+1}}_i & = U^n_i - \frac{\Delta t}{\Delta x} \Big(F^n_{i+\frac{1}{2}} - F^n_{i-\frac{1}{2}} \Big) & \\
\textrm{corrector: } \qquad U^{n+1}_i & = \frac{1}{2} \left[ U^n_i + U^{\overline{n+1}}_i - \frac{\Delta t}{\Delta x} \Big(F^{\overline{n+1}}_{j+1} - F^{\overline{n+1}}_j \Big) \right]
\end{align}
$$

The indexing notations for the cell-centered variables, $U_i$, and the corresponding face-centered fluxes at the left $F^n_{i-\frac{1}{2}}$ and right $F^n_{i+\frac{1}{2}}$ of the $i^{th}$-element are shown in the figure below.

Note any finite-volume method requires an algorithm for computing the face-centered fluxes. This requires reconstructing the cell-centered quantities at each of the faces within the computational mesh. For MacCormack's method, this reconstruction process is linear.

In previous discussions when deriving the finite-volume method, we defined fluxes using the $j^{th}$ face of the volume element. In one-dimension, we can simplify the notation and use the half-mesh spacings as shown in the above figure. Since the volume-averaged conserved variables, $U_i$, are not defined at the surfaces of the volume element or cell, they must be reconstructed. For the MacCormack method, at the predictor step the values at the face are forward-biased as

$$
F^n_{i+\frac{1}{2}} = F^n \,\Big(U^n_{i+1}\Big)
$$

and

$$
F^n_{i-\frac{1}{2}} = F^n \,\Big(U^n_{i}\Big)
$$

and at the corrector step, the values at the face are backward-biased as

$$
F^{\overline{n+1}}_{i+\frac{1}{2}} = F^n \,\Big(U^{\overline{n+1}}_{i}\Big)
$$

and

$$
F^{\overline{n+1}}_{i-\frac{1}{2}} = F^n \,\Big(U^{\overline{n+1}}_{i-1}\Big)
$$

Since the Euler fluxes are expressed as functions of the primitive variables, they must be recomputed from the conservative vector. In particular, the pressure is required. The Euler flux vector is given as

$$
\mathbf{f} = \begin{bmatrix}
    \rho u & \\
    \rho u^2 + p & \\
    (e + p) u
  \end{bmatrix}
$$

From the conservative variables, the primitive variables can be computed, which requires a thermodynamic model to relate the total energy to the thermodynamic state. For a calorically-perfect gas, this relationship is straight-forward and the mapping between the conservative and primitive variables is linear.
$$
\mathbf{U}^n_i = \begin{bmatrix}
    \rho^n_i & \\
    (\rho u)^n_i & \\
    e^n_i
  \end{bmatrix} \to
\mathbf{V}^n_i = \begin{bmatrix}
    \rho^n_i & \\
    p^n_i & \\
    T^n_i
  \end{bmatrix}
$$

For a calorically perfect gas, the velocity is first computed using the momentum and total density as,
$$
u^n_i = \frac{(\rho u)^n_i}{\rho^n_i}
$$
Next, the internal energy or temperature is computed

$$
\epsilon^n_i = c^n_{v,i} T^n_i = \frac{e^n_i}{\rho^n_i} - \frac{1}{2} (u^n_i )^2
$$

And last we can compute the pressure,
$$
p^n_i = (\gamma - 1) \rho^n_i \epsilon^n_i
$$

## Shocktube Test Cases

Shocktube test cases are a standard way of testing the performance of compressible CFD solvers. For an extensive review of test problems for the Euler equations see the accompanying paper by Liska and Wendroff, 2003.

## Sod Test Case

For use with `pyflow` a shocktube problem is defined by specifying the primitive state vector at the left and right of the diaphragm. To do this, use the `PrimitiveState` object. For example, the following code defines the left and right state vectors.

    left = euler.PrimitiveState(density=1.0, speed=0.0, pressure=1.0)
    right = euler.PrimitiveState(density=0.125, speed=0.0, pressure=0.1)

These can then be used to define a shocktube problem:

    Sod = ShocktubeProblem(left, right, x0=0.5, Nx=50)

where `x0` is the location of the diaphragm and `Nx` is the number of cells used to discretize the domain. Additionally, the specific heat ratio $\gamma$ can also be specified. For example, to change the value of the specific heat ratio from the default of 1.4 to $\gamma = 5/3$,

    Sod = ShocktubeProblem(left, right, x0=0.5, Nx=50, gamma=5/3)

```python
# Define the left and right states for the initial conditions of Sod's shocktube test case
#
left = PrimitiveState(density=1.0, speed=0.0, pressure=1.0)
right = PrimitiveState(density=0.125, speed=0.0, pressure=0.1)

# Number of cells
Nx = 100

# CFL number
CFL = 0.8

# Diaphragm location
x0 = 0.5

# Final simulation time
#
time = 0.25
```

```python
SodExact = ShocktubeProblem(left, right, method='exact', x0=x0, Nx=Nx, CFL=CFL)
exact = SodExact.solve(time)
```

    2022-05-07 05:14:25,321 | pyflow.euler.shocktube | INFO | Compute exact solution to shocktube problem only.

```python
# Compute the solution using MacCormack's predictor-corrector method
#
SodMC = ShocktubeProblem(left, right, method='maccormack', x0=x0, Nx=Nx, CFL=CFL)

MC = SodMC.solve(time)
```

    2022-05-07 05:14:25,883 | pyflow.euler.shocktube | INFO | Finite-volume method selected: MacCormack predictor-correct

    Simulation Complete
    Final time = 0.25
    Total number of iterations = 77

```python
fig, ax = plt.subplots()
ax.plot(exact.x, exact.density, label="exact")
ax.plot(MC.x, MC.density, label="MacCormack")

ax.set_xlabel(r'$x$')
ax.set_ylabel('Density')
ax.set_xlim((0,1))
ax.legend()
plt.title('Sod Shocktube Problem')
```

    <IPython.core.display.Javascript object>

    Text(0.5, 1.0, 'Sod Shocktube Problem')

There are significant oscillations in the solution using MacCormack's methods, particularly at the discontinuities. In some instances for a given mesh spacing and time step, MacCormack's method can fail. This non-monotonic behavior is unavoidable for any second-order or higher numerical method that uses linear reconstruction functions for the numerical fluxes. Recall that similar observations were made about the finite-difference methods. See Godunov's order barrier theorem.

An early solution to this problem was the inclusion of the numerical dissipation; however, this approach is not ideal because the numerical dissipation is applied to all parts of the flow, when it only needs to be applied to regions of high gradients. Observe how the inclusion of numerical dissipation helps stabilize MacCormack's method.

```python
# Compute the solution using MacCormack's predictor-corrector method
#
SodMCdiss = ShocktubeProblem(left, right, method='maccormack', x0=x0, Nx=Nx, CFL=CFL, dissipation=0.1)

MCdiss = SodMCdiss.solve(time)
```

    2022-05-07 05:14:27,865 | pyflow.euler.shocktube | INFO | Finite-volume method selected: MacCormack predictor-correct

    Simulation Complete
    Final time = 0.25
    Total number of iterations = 68

```python
fig, ax = plt.subplots()
ax.plot(exact.x, exact.density, label="exact")
ax.plot(MC.x, MC.density, label="MacCormack")
ax.plot(MCdiss.x, MCdiss.density, label="MacCormack with dissipation")

ax.set_xlabel(r'$x$')
ax.set_ylabel('Density')
ax.set_xlim((0,1))
ax.legend()
plt.title('Sod Shocktube Problem')
```

    <IPython.core.display.Javascript object>

    Text(0.5, 1.0, 'Sod Shocktube Problem')

Computational scientists discovered that *non-linear* reconstructions were necessary for developing high-order methods for compressible flows. These non-linear reconstructions form the basis for the flux-differencing and flux-vector splitting algorithms that are illustrated below.

```python
# Compute the solution using MacCormack's predictor-corrector method
#
SodSW = ShocktubeProblem(left, right, method='steger-warming', x0=x0, Nx=Nx, CFL=CFL)

SW = SodSW.solve(time)
```

    2022-05-07 05:14:29,888 | pyflow.euler.shocktube | INFO | Finite-volume method selected: Steger-Warming

    Simulation Complete
    Final time = 0.25
    Total number of iterations = 68

```python
# Compute the solution using MacCormack's predictor-corrector method
#
SodHLLC = ShocktubeProblem(left, right, method='hllc', x0=x0, Nx=Nx, CFL=CFL)

HLLC = SodHLLC.solve(time)
```

    2022-05-07 05:14:30,677 | pyflow.euler.shocktube | INFO | Finite-volume method selected: Flux-difference splitting, HLLC

    Simulation Complete
    Final time = 0.25
    Total number of iterations = 67

```python
fig, ax = plt.subplots()
ax.plot(exact.x, exact.density, label="exact")
ax.plot(HLLC.x, HLLC.density, label="Flux-difference splitting method, HLLC")
ax.plot(SW.x, SW.density, label="Flux-vector splitting, Steger-Warming")

ax.set_xlabel(r'$x$')
ax.set_ylabel('Density')
ax.set_xlim((0,1))
ax.legend()
plt.title('Sod Shocktube Problem')
```

    <IPython.core.display.Javascript object>

    Text(0.5, 1.0, 'Sod Shocktube Problem')

# References

1. Liska, R. and B. Wendroff (2003), *Comparison of several difference schemes on 1D and 2D test problems for the Euler equations*, SIAM J. Sci. Comput., **25** (3), 995-1017


```python
# setup SymPy
from sympy import *
x, y, z, t = symbols('x y z t')
init_printing()

# setup plotting
%matplotlib notebook
import matplotlib.pyplot as mpl
from util.plot_helpers import plot_vec, plot_vecs, autoscale_arrows
```

```python
simplify(sin(2*x)*cos(2*x))
```

```python
a = Matrix([1,1,1])
a.
```

```python
S(1)/3
```

```python
type(1)
```

    int

```python
type(S(1))
```

    sympy.core.numbers.One

```python
float(S(1)/3)
```

```python
N(S(1)/3)
```

```python
pi.n(100)
type(pi.n(100))
```

    sympy.core.numbers.Float

```python
e
```

```python
E
```

```python

```


# Frequentist Inference Case Study - Part B

## Learning objectives

Welcome to Part B of the Frequentist inference case study! The purpose of this case study is to help you apply the concepts associated with Frequentist inference in Python. In particular, you'll practice writing Python code to apply the following statistical concepts:
* the _z_-statistic
* the _t_-statistic
* the difference and relationship between the two
* the Central Limit Theorem, including its assumptions and consequences
* how to estimate the population mean and standard deviation from a sample
* the concept of a sampling distribution of a test statistic, particularly for the mean
* how to combine these concepts to calculate a confidence interval

In the previous notebook, we used only data from a known normal distribution. **You'll now tackle real data, rather than simulated data, and answer some relevant real-world business problems using the data.**

## Hospital medical charges

Imagine that a hospital has hired you as their data scientist. An administrator is working on the hospital's business operations plan and needs you to help them answer some business questions.

In this assignment notebook, you're going to use frequentist statistical inference on a data sample to answer the questions:
* has the hospital's revenue stream fallen below a key threshold?
* are patients with insurance really charged different amounts than those without?

Answering that last question with a frequentist approach makes some assumptions, and requires some knowledge, about the two groups.

We are going to use some data on medical charges obtained from [Kaggle](https://www.kaggle.com/easonlai/sample-insurance-claim-prediction-dataset).

For the purposes of this exercise, assume the observations are the result of random sampling from our single hospital. Recall that in the previous assignment, we introduced the Central Limit Theorem (CLT), and its consequence that the distributions of sample statistics approach a normal distribution as $n$ increases. The amazing thing about this is that it applies to the sampling distributions of statistics that have been calculated from even highly non-normal distributions of data! Recall, also, that hypothesis testing is very much based on making inferences about such sample statistics. You're going to rely heavily on the CLT to apply frequentist (parametric) tests to answer the questions in this notebook.

```python
import pandas as pd
import numpy as np
import matplotlib.pyplot as plt
from scipy.stats import t
from numpy.random import seed
medical = pd.read_csv('insurance2.csv')
```

```python
medical.shape
```

    (1338, 8)

```python
medical.head()
```

      age
      sex
      bmi
      children
      smoker
      region
      charges
      insuranceclaim

      0
      19
      0
      27.900
      0
      1
      3
      16884.92400
      1

      1
      18
      1
      33.770
      1
      0
      2
      1725.55230
      1

      2
      28
      1
      33.000
      3
      0
      2
      4449.46200
      0

      3
      33
      1
      22.705
      0
      0
      1
      21984.47061
      0

      4
      32
      1
      28.880
      0
      0
      1
      3866.85520
      1

__Q1:__ Plot the histogram of charges and calculate the mean and standard deviation. Comment on the appropriateness of these statistics for the data.

__A:__

```python
estimate = medical.charges.mean()
sd = medical.charges.std()  # Pandas standard deviation defaults to the unbiased estimator, i.e., ddof=1
print(f"The mean of charges is ${estimate:.2f}, and the standard deviation is ${sd:.2f}")
```

    The mean of charges is $13270.42, and the standard deviation is $12110.01

```python
medical.charges.hist(figsize=(10, 8), bins='auto')
plt.axvline(estimate, color='r', label='mean')
plt.title('Histogram of medical charges')
plt.ylabel('Count of charges')
plt.xlabel('Amount charged')
plt.axvline(estimate + 2*sd, color='r', linestyle='--', label='+/- 2 sd')
plt.axvline(estimate - 2*sd, color='r', linestyle='--')
plt.legend();
```

This histogram shows clearly that the mean and standard deviation are misleading at best as summaries of the medical charges because the distribution of medical charges is highly right-skewed.

__Q2:__ The administrator is concerned that the actual average charge has fallen below 12,000, threatening the hospital's operational model. On the assumption that these data represent a random sample of charges, how would you justify that these data allow you to answer that question? And what would be the most appropriate frequentist test, of the ones discussed so far, to apply?

__A:__ This is a large sample (n=1338), and by the central limit theorem, the distribution of sample means is normally distributed, allowing us to quantify our uncertainty due to random (sampling) error by treating the sample mean as belonging to a normal distribution with sd = (standard deviation of our sample) / sqrt(n). Although this technically calls for a t test (since we do not know the population variance), the z-statistic and the t-statistic will be about the same, because we have 1337 degrees of freedom in our estimate. In other words, the t-distribution with dof=1337 is approximately normal.

__Q3:__ Given the nature of the administrator's concern, what is the appropriate confidence interval in this case? A ***one-sided*** or ***two-sided*** interval? (Refresh your understanding of this concept on p. 399 of the *AoS*). Calculate the critical value and the relevant 95% confidence interval for the mean, and comment on whether the administrator should be concerned.

__A:__ In this case, the administrator is concerned about the average charge falling below \\$12,000, so our null hypothesis is that the average charge is at or above \\$12,000, necessitating a one-sided t-test.

```python
h_null = 12000
n = medical.charges.count()
se = medical.charges.std() / np.sqrt(n)
t_score = (estimate - h_null) / se
p = t.cdf(-t_score, n-1)
conf_95 = np.round(estimate + t.ppf([0.025, 0.975], n-1) * se, 2)
```

```python
print(f"""One-tailed t-test:
The estimate of the mean is {estimate:.2f}, with a standard error of {se:.2f}.
The test statistic is {t_score:.2f}.
The one-tailed p-value for this test statistic is {p:.2e}.
The 95% confidence interval for the estimate is {conf_95}.
""")
```

    One-tailed t-test:
    The estimate of the mean is 13270.42, with a standard error of 331.07.
    The test statistic is 3.84.
    The one-tailed p-value for this test statistic is 6.51e-05.
    The 95% confidence interval for the estimate is [12620.95 13919.89].

With 95% confidence, we estimate that the average charge is \\$13,270, with a 95% confidence interval from \\$12,558 –\\$13982. Assuming that this sample is representative, we are highly confident that the average charge is not falling below \\$12,000.

The administrator then wants to know whether people with insurance really are charged a different amount to those without.

__Q4:__ State the null and alternative hypothesis here. Use the _t_-test for the difference between means, where the pooled standard deviation of the two groups is given by:
\begin{equation}
s_p = \sqrt{\frac{(n_0 - 1)s^2_0 + (n_1 - 1)s^2_1}{n_0 + n_1 - 2}}
\end{equation}

and the *t*-test statistic is then given by:

\begin{equation}
t = \frac{\bar{x}_0 - \bar{x}_1}{s_p \sqrt{1/n_0 + 1/n_1}}.
\end{equation}

(If you need some reminding of the general definition of ***t-statistic***, check out the definition on p. 404 of *AoS*).

What assumption about the variances of the two groups are we making here?

__A:__  The null hypothesis is that mean charge for people with insurance is the same as the mean charage as people without insurance; the alternative hypothesis is that these two population means differ. In using a t-test, we are assuming homogeneity of variance between the two groups.

__Q5:__ Perform this hypothesis test both manually, using the above formulae, and then using the appropriate function from [scipy.stats](https://docs.scipy.org/doc/scipy/reference/stats.html#statistical-tests) (hint, you're looking for a function to perform a _t_-test on two independent samples). For the manual approach, calculate the value of the test statistic and then its probability (the p-value). Verify you get the same results from both.

__A:__

```python
ins = medical[medical.insuranceclaim == 1].charges
noins = medical[medical.insuranceclaim == 0].charges
y_bar_ins = ins.mean()
y_bar_noins = noins.mean()
theta_hat = y_bar_ins - y_bar_noins
n_ins = ins.count()
n_noins = noins.count()
sd_theta = np.sqrt(( (n_ins - 1) * ins.var() + (n_noins - 1) * noins.var() ) / (n_ins + n_noins - 2))
se_theta = (sd_theta) * np.sqrt(1/n_ins + 1/n_noins)
t_score = np.abs(theta_hat) / se_theta
p_value = 2 * t.cdf(-t_score, n_ins + n_noins - 2)
```

```python
print(f"My manually calculated test statistic is {t_score} and p-value is {p_value}")
```

    My manually calculated test statistic is 11.89329903087671 and p-value is 4.461230231620972e-31

```python
from scipy.stats import ttest_ind

ttest_ind(ins, noins)
```

    Ttest_indResult(statistic=11.893299030876712, pvalue=4.461230231620717e-31)

Congratulations! Hopefully you got the exact same numerical results. This shows that you correctly calculated the numbers by hand. Secondly, you used the correct function and saw that it's much easier to use. All you need to do is pass your data to it.

__Q6:__ Conceptual question: look through the documentation for statistical test functions in scipy.stats. You'll see the above _t_-test for a sample, but can you see an equivalent one for performing a *z*-test from a sample? Comment on your answer.

__A:__  Scipy doesn't provide an equivalent function for a z-test from a sample; the results from an equivalent z-test will approach the results from the t-test as the sample sizes approach infinity. The t distribution accomodates the sampling uncertainty with fatter tails than the normal distribution for low degrees of freedom.

## Learning outcomes

Having completed this project notebook, you now have good hands-on experience:
* using the central limit theorem to help you apply frequentist techniques to answer questions that pertain to very non-normally distributed data from the real world
* performing inference using such data to answer business questions
* forming a hypothesis and framing the null and alternative hypotheses
* testing this using a _t_-test


# 11 ODE Applications (Baseball)

Simulate a semi-realistic baseball trajectory, involving air resistance and spin.

1. Baseball physics (theory),
2. Outline of baseball simulation code (see [baseball_solution](baseball_solution.ipynb) for the actual code)

```python
import numpy as np
import ode
```

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.matplotlib.style.use('ggplot')
```

## Theory: Simple Baseball physics

- quadratic air resistance (with velocity-dependent drag coefficient)
- Magnus force due to spin

### Quadratic air resistance
Occurs at high Reynolds numbers, i.e., turbulent flow. Only approximate:

$$
\mathbf{F}_2 = -b_2 v^2 \mathbf{\hat v} = -b_2 v \mathbf{v}
$$

Relate $b_2$ to the medium that is displaced:

$$
b_2 = \frac{1}{2} C_D \rho A
$$

where $C_D$ is called the _drag coefficient_.

### Magnus effect

**Magnus effect**: airflow is changed around a spinning object. The Magnus force is

$$
\mathbf{F}_M = \alpha \boldsymbol{\omega} \times \mathbf{v}
$$

where $\boldsymbol{\omega}$ is the ball's angular velocity in rad/s (e.g., 200/s for a baseball).

For a sphere the proportionality constant $\alpha$ can be written

$$
\mathbf{F}_M = \frac{1}{2} C_L \rho A \frac{v}{\omega} \boldsymbol{\omega} \times \mathbf{v}
$$

where $C_L$ is the lift coefficient, $\rho$ the air density, $A$ the ball's cross section. (Advantage of defining $C_L$ this way: when spin and velocity are perpendicular, the Magnus force is simply $F_M = \frac{1}{2} C_L \rho A v^2$.)

$C_L$ is mainly a function of the *spin parameter*

$$
S = \frac{r\omega}{v}
$$

with the radius $r$ of the ball. In general we write

$$
\mathbf{F}_M = \frac{1}{2} C_L  \frac{\rho A r}{S} \boldsymbol{\omega} \times \mathbf{v}
$$

For a baseball, experimental data show approximately a power law dependence of $C_L$ on $S$

$$
C_L = 0.62 \times S^{0.7}
$$

All together:

\begin{align}
\mathbf{F}_M &= \alpha\ \boldsymbol{\omega} \times \mathbf{v}\\
v &= \sqrt{\mathbf{v}\cdot\mathbf{v}}\\
S &= \frac{r\omega}{v}\\
C_L &= 0.62 \times S^{0.7}\\
\alpha &= \frac{1}{2} C_L  \frac{\rho A r}{S}
\end{align}

### Equations of motion

\begin{align}
\frac{d\mathbf{r}}{dt} &= \mathbf{v}\\
\frac{d\mathbf{v}}{dt} &= -g \hat{\mathbf{e}}_y -\frac{b_2}{m} v \mathbf{v} + \frac{\alpha}{m}\ \boldsymbol{\omega} \times \mathbf{v}
\end{align}

(quadratic drag $-\frac{b_2}{m} v \mathbf{v}$ included and dividing through by $m$ to get accelerations in the second equation.)

## Baseball simulation

Implement the full baseball equations of motions:
- gravity $a_\text{gravity}$
- quadratic drag $a_\text{drag}$
- Magnus effect $a_\text{Magnus}$

For the cross product you can look at [numpy.cross()](https://docs.scipy.org/doc/numpy/reference/generated/numpy.cross.html).

We will live-code the baseball simulation in class (i.e., build it from scratch), but if you want to work on this problem on your own and need some starter code, see below.

#### Parameters

* mass of baseball $m = 0.14883$ kg
* diameter of baseball $d = 0.07468$ m
* distance of homeplate from pitcher $R_\text{home} = 18.4$ m
* acceleration due to gravity $g = 9.81$ m/s$^2$
* air density $\rho = 1.225$ kg/m$^3$
* drag coefficient in air $C_D=0.40$

#### Baseball  code
Full solution will be posted as [`baseball_solution.ipynb`](baseball_solution.ipynb).

#### Simulate throws

Simulate baseball throw for initial velocity $\mathbf{v} = (30\,\text{m/s}, 0)$.

Plot x vs y and x vs z (to see curving).

Try out different spins; a good value is $\boldsymbol{\omega} = 200\,\text{rad/s} \times (0, 1, 1)$.

Simulate the baseball throw with
- almost no spin: $\omega = 0.001 \times (0, 0, 1)$  (our code does not handle $\omega = 0$ gracefully...)
- full upward spin: $\omega = 200 \times (0, 0, 1)$
- sideways spin: $\omega = 200 \times (0, 1, 1)$

#### Plotting

Plot the three scenarios in 2D planes: x-y (side view) and x-z (top view).

#### 3D plot
Use simple `matplotlib` 3D plot. (BONUS: Make it work with vpython)

If we use the [`ipympl` backend for matplotlib](http://ipython.readthedocs.io/en/stable/interactive/plotting.html) then we will be able to interactively rotate our [matplotlib 3D graphics](http://matplotlib.org/mpl_toolkits/mplot3d/tutorial.html). (Note: If this does not seem to work, disable adblockers and allow javascript on the page.)

```python
%matplotlib ipympl
```

```python
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.plot(r[:,1], r[:,3], r[:,2], 'o', label="no spin")
# ...

# hand of the catcher, 0.2m above homeplate
ax.plot([18.4, 18.4], [0, 0], [0, 0.2], color="black", lw=6)

ax.set_xlabel("$x$ (m)")
ax.set_ylabel("$z$ (m)")
ax.set_zlabel("$y$ (m)")
ax.legend(loc="upper left", numpoints=1)
ax.figure.tight_layout()
```

## Reynolds number

$$
\text{Re} = \frac{\rho v L}{\mu}\\
\text{Re} > 2300\quad\text{flow turbulent}
$$

* density $\rho$: air  1.275 kg/m^3 (kilograms per cubic meter)
* fluid viscosity $\mu$: air 1.845×10^-5 Pa s (pascal seconds) (at 25 °C)

(from Wolfram Alpha)

```python
rho_air = 1.275    # kg/m^3
mu_air = 1.845e-5  # Pa s
```

```python
L = 0.05  # m
v = 200   # m/s
```

```python
def ReynoldsNumber(v, L, rho=rho_air, mu=mu_air):
    return rho*v*L/mu
```

```python
ReynoldsNumber(v, L)
```

    691056.9105691056

This means that we really should have been using quadratic air resistance for the projectile that we simulated with linear air resistance... but we did this because (1) it was easy and (2) we will come back to the problem later when we look at root finding, and we can actually solve the trajectory analytically.

```python

```


# Normalizing Flows
## A short stroll from discrete to continuous time

## Aalto University, EEA
Adrien Corenflos, 2.11.2020

# Outline

1. Generative models

2. Problem formulation

3. Discrete time normalizing flows

4. Continuous time normalizing flows

# Generative models

## Definition

Statistical models are usually divided in two classes: discriminative and generative [1, Ch.2].

- discriminative:

    Discriminative models are a class of statistical models encode the conditional distribution of the observation given the input
    $$
    \begin{equation}
        \mathbb{P}(Y \mid X)
    \end{equation}
    $$

- generative:

    Generative models on the other end encode the joint distribution of the input and the observation
    $$
    \begin{equation}
        \mathbb{P}(X, Y)
    \end{equation}
    $$

## Some examples:

| Discriminative | Generative |
| --- | --- |
| Logistic regression | Naive Bayes classifier |
| Support vector machines | Gaussian Processes |
| Decision trees | Bayesian Networks |
| Neural Networks | ? |

# Problem formulation

## The data generation problem

We will consider the two following problems:

1. Given a series of observations $Y_n$'s, how do we calibrate a probabilistic model that is able to generate "similar" data?
    For example, how can we generate handwritten numbers:

```python
from sklearn.datasets import fetch_openml
mnist_X, mnist_y = fetch_openml('mnist_784', version=1, return_X_y=True)
```

```python
import matplotlib.pyplot as plt
_ = plt.imshow(1 - mnist_X[0].reshape(28, 28), cmap="gray")
```

2. Given a complicated likelihood function how do we sample from it? For example:

```python
import numpy as np

def square(x, y):
    abs_x = np.abs(x)
    abs_y = np.abs(y)
    return 0.25 * np.exp(-4*(np.maximum(abs_x, abs_y) - 1) ** 2) / np.sqrt(np.pi)

def plot_square(ax=None):
    linspace = np.linspace(-2., 2., 200)
    mesh_x, mesh_y = np.meshgrid(linspace, linspace)
    if ax is None:
        fig, ax = plt.subplots(figsize=(5, 5))
        ax.set_aspect("equal")
    _ = ax.pcolormesh(mesh_x, mesh_y, 1. - square(mesh_x, mesh_y), cmap="gray", shading='auto')

plot_square()
```

## Non exhaustive solutions

1. Data registration:
    a. Variational auto-encoders
    b. Generative adversarial networks
    c. Normalizing flows

2. Distribution sampling:
    a. MCMC
    b. Normalizing flows

1.a. VAE
    Learn a latent representation of the distribution

Image from Wikimedia. To go further see [2].

1.b. GAN
    Learn to fool a expert networks, or more recently learn to minimize Wasserstein distance between true and generated.
    $$\underset{\theta}{\min} \mathcal{W}\bigg(G_\theta, [Y_n]_{n=1}^N\bigg)$$

To go further see [3].

2.a. MCMC
    Acceptance-rejection method for exact sampling.

```python
def mh(f, n=1000, seed=0):
    rng = np.random.RandomState(seed)
    arr = np.empty((n, 2))
    x, y = rng.uniform(-1, 1, 2)
    p = f(x, y)
    for i in range(n):
        eps_x, eps_y = rng.uniform(-1, 1, 2)
        xi, yi = x + eps_x, y + eps_y
        pi = f(xi, yi)
        if rng.rand() <= pi / p:
            x, y = xi, yi
            p = pi
        arr[i] = x, y
    return arr
```

```python
fig, ax = plt.subplots(figsize=(5, 5))
ax.set_xlim(-2, 2), ax.set_ylim(-2, 2)
_ = ax.scatter(*mh(square, 5000).T, alpha=0.33)
```

# Discrete time normalizing flows

## Principle
- Flow: stands for composition
- Normalizing: stands for the fact that each transformation is normalizing: keeps total probability mass constant

$$
\begin{align}
    X_0 &\sim U([0, 1]^d) \quad \text{or} \quad \mathcal{N}(0, I_d) \\
    X_i &= f_i(X_{i-1}, \theta_i)
\end{align}
$$

Normalizing means that $f_i$ needs to be bijective.

## Change of variable

Let $Z = f(X)$,
$$p_Z(z) = p_X(f^{-1}(z)) \cdot \mathrm{d}V(f^{-1}(z))$$
where $dV$ is local the change of volume incurred by the change of variable.

Lebesgue theory tells us that
$$\mathrm{d}V = \left| \mathrm{det}J_f^{-1}\right| = \left| \mathrm{det}J_f\right|^{-1}$$

```python
import scipy.stats as st

def f(x): return np.exp(x)
def dV(z): return 1/z

linspace = np.linspace(1e-3, 3., 10000)
_ = plt.plot(linspace, st.lognorm.pdf(s=1., x=linspace), label="lognorm")
_ = plt.plot(linspace, st.norm.pdf(x=np.log(linspace)) * dV(linspace), label="normalized exp-norm")
_ = plt.legend()
```

## Flows  and chain rule

So now the composed model reads:
$$
\begin{align}
    Z = X_n = f_n(f_{n-1}(\dots, \theta_{n-1}), \theta_n))
\end{align}
$$

so that $p_Z(z) = p_X(x) \prod_{i=1}^n \left|\mathrm{det}J_{f_i}^{-1}\right|$, or said otherwise:
$$
\ln p_Z(z) = \ln p_X(x) - \sum_{i=1}^n \ln \left|\mathrm{det}J_{f_i}\right|
$$
for which we want to compute the gradient with respect to $\theta = [\theta_n, \dots, \theta_1]$

So how do we compute $\frac{\partial}{\partial \theta_i} \ln \left|\mathrm{det}J_{f_i}\right|$ ?

$$ \mathrm{d}(\ln\left| \mathrm{det}(M)\right|) = M^{-\top}\mathrm{d}M$$
So that we need to invert all the $J_{f_i}$: ($O(N^3)$)!!!

This restricts the practical form of the $f_i$'s quite a bit to functions for which the log-determinant of the Jacobian has an easy to compute form: triangular Jacobian, or...

## Planar flows [4]

$$
\begin{align}
    f(\mathbf{x}, \mathbf{a}, b, \mathbf{w}) = \mathbf{x} + \sigma(\mathbf{w}^\top \cdot \mathbf{x} + b) \mathbf{a}
\end{align}
$$

$$
\begin{align}
    J_f = I + \sigma'(\mathbf{w}^\top \cdot \mathbf{x} + b) \mathbf{a} \cdot \mathbf{w}^\top
\end{align}
$$

for which, using the [Matrix determinant lemma](https://en.wikipedia.org/wiki/Matrix_determinant_lemma), we get

$$
\begin{align}
    \mathrm{det}(J_f)= 1 + \sigma'(\mathbf{w}^\top \cdot \mathbf{x} + b) \cdot \mathbf{a}^\top \cdot \mathbf{w}
\end{align}
$$

```python
def sigmoid(y): return 1 / (1 + np.exp(-y))
def planar_flow_sigmoid(x, a, b, w): return x + sigmoid(x @ w + b).reshape(-1, 1) * a.reshape(1, -1)

uniform = np.random.uniform(-1, 1, (1000, 2))
a, b, w = np.array([0.5, -0.75]), 0., np.array([-1, 1])
transformed_uniform = planar_flow_sigmoid(uniform, a, b, w)

fig, ax = plt.subplots(figsize=(5, 5),)
_ = ax.scatter(*uniform.T, alpha=0.5)
_ = ax.scatter(*transformed_uniform.T, alpha=0.5)
_ = ax.quiver(*uniform[:20].T, *(transformed_uniform[:20].T - uniform[:20].T),
              units='xy', angles="xy", scale=1, alpha=0.9)
```

## MLE vs KL divergence
Does it mean we can compute the log-likelihood of a sample?

Well no (see [5] for a model where you can and it's efficient). To do this you need to compute $f^{-1}(y)$.
So instead we need to go forward: KL divergence!

$$
\begin{align}
    \textrm{KL}(q \mid\mid p)
        &= \int q \ln \frac q p\\
        &\approx \frac 1 N \sum_{i=1}^N \left(\ln f_0(u_i) - \sum_{i=1}^K \ln \left|\mathrm{det}J_{f_i}\right| - \ln p(z_i) \right)
\end{align}
$$

```python
import tensorflow as tf
import tensorflow_probability as tfp

@tf.function
def tf_planar_flow(x, a, b, w, sigma):
    scale = sigma(tf.linalg.matvec(x, w) + b)
    return x + tf.reshape(scale, (-1, 1)) * tf.reshape(a, (1, -1))

@tf.function
def tf_planar_log_det_jac(x, a, b, w, sigma):
    z = tf.linalg.matvec(x, w) + b
    with tf.GradientTape() as tape:
        tape.watch(z)
        val = sigma(z)
    scalar_prod = tf.tensordot(a, w, 1)
    sigma_prime = tape.gradient(val, z)
    return tf.math.log(tf.math.abs(1. + scalar_prod * sigma_prime))

@tf.function
def kl_planar(a_list, b_list, w_list, sigma, p, N, d, key):
    # without the initial distribution

    def fun(u, v):
        (f_k, log_det_jac), (a, b, w) = u, v
        return (tf_planar_flow(f_k, a, b, w, sigma),
                tf_planar_log_det_jac(f_k, a, b, w, sigma))

    us = tf.random.stateless_uniform((N, d), key, 0, 1)
    xks, log_det_jacs = tf.scan(fun, [a_list, b_list, w_list], (us, tf.zeros((us.shape[0],), dtype=us.dtype)))
    pus = p(xks[-1])
    logp = tf.math.log(pus)
    kl_val = -logp - tf.reduce_sum(log_det_jacs, axis=0)
    return tf.math.reduce_mean(kl_val), xks[-1]
```

```python
@tf.function
def tf_square(z):
    abs_x = tf.math.abs(z[:, 0])
    abs_y = tf.math.abs(z[:, 1])
    return 0.25 * tf.exp(-4.*(tf.math.maximum(abs_x, abs_y) - 1.) ** 2) / tf.math.sqrt(np.pi)
```

```python
tf_sigmoid = tf.function(tf.math.sigmoid)
```

```python
N = 5000
K = 5
n_iter = 1000
learning_rate = 0.1
decay = 0.95
keys = tfp.random.split_seed(42, n=n_iter + 3)

a_vars = tf.random.stateless_normal((K, 2), keys[0])
w_vars = tf.random.stateless_normal((K, 2), keys[1])
b_vars = tf.random.stateless_normal((K,), keys[2])
```

```python
fig, axes = plt.subplots(ncols=2, nrows=2, figsize=(11, 11), sharex=True, sharey=True)
plot_square(axes[0, 0])

lr = learning_rate
for i in range(n_iter):
    with tf.GradientTape() as tape:
        tape.watch([a_vars, b_vars, w_vars])
        loss, sample = kl_planar(a_vars,
                                 b_vars,
                                 w_vars,
                                 tf_sigmoid,
                                 tf_square,
                                 N,
                                 2,
                                 keys[i+3])
    if i == 0:
        axes[0, 1].scatter(*sample.numpy().T, alpha=0.5)
    grad_a, grad_b, grad_w = tape.gradient(loss, [a_vars, b_vars, w_vars])
    a_vars = a_vars - learning_rate * grad_a
    b_vars = b_vars - learning_rate * grad_b
    w_vars = w_vars - learning_rate * grad_w
    lr = lr * decay

axes[1, 1].scatter(*sample.numpy().T, alpha=0.5)
plot_square(axes[1, 0])

```

# Continuous time normalizing flows [6]

## Principle

Let's have a further look at the recursion induced by the planar flow:

$$
\begin{align}
    x_k
        &= x_{k-1} + f_{\theta_k}(x_{k-1})\\
    (x_k - x_{k-1})
        &= g(k, x_{k-1}, \theta_k) \Delta
\end{align}
$$

So that if $\Delta \to 0$, we can recognize an Euler discretization scheme for
$$
\begin{align}
    \mathrm{d}x_t &= f(x_t, x, t, \theta) \mathrm{d}t \\
    x_0 &\sim p_0
\end{align}
$$

## Why is it interesting?

- Invertibility guarantees
- Easy to compute change of variable scaling
- Gradient calculation happens to be easy

### Invertibility guarantees

Cauchy theorem: as long as $f$ is continuous, the trajectories are uniquely defined by their starting point (or any point for that matter).
Moreover, if we let $y_s = x_{t-s}$, then

$$
\begin{align}
    y_s' = f(y_s)
\end{align}
$$

### Change of variables scaling

We have

$$
\begin{align}
    \frac{\partial}{\partial t} \ln p_{t}(x_t) = - \mathrm{Tr}\left(J_x[f](x_t)\right)
\end{align}
$$

so that to compute the log-likelihood, you only need to augment the state

```python
def ode_fun(t, z, fun, fun_params):
    x, logp = z[..., :-1], z[..., -1]
    with tf.GradientTape() as tape:
        tape.watch(x)
        drift = fun(t, x, *fun_params)
    jac_drift = tape.batch_jacobian(drift, x)
    return tf.concat([drift,
                      tf.expand_dims(tf.linalg.trace(jac_drift), 1)], axis=-1)
```

For instance consider a linear system
$\mathrm{d}x_t = A x_t$

```python
@tf.function
def linear_fun(t, x, a):
    return tf.matmul(x, a, transpose_b=True)

key = tfp.random.split_seed(666, n=1)
N = 1000
D = 2
A = 0.5 * tf.eye(D)
t0, t1 = 0., 1.
x0_dist = tfp.distributions.MultivariateNormalDiag(loc=tf.zeros(D))
x0 = x0_dist.sample(N)
x0 = tf.concat([x0,
                tf.expand_dims(x0_dist.log_prob(x0), 1)], axis=1) # the log-likelihood

@tf.function
def linear_ode_fun(t, x, a):
    return ode_fun(t, x, linear_fun, [a])

res = tfp.math.ode.DormandPrince().solve(linear_ode_fun,
                                         t0, x0, solution_times=[t1],
                                         constants={'a': A},
                                         batch_ndims=0).states.numpy()
```

```python
from scipy.linalg import expm
from scipy.interpolate import griddata

m_x, m_y = np.meshgrid(np.linspace(-2, 2, 100),
                       np.linspace(-2, 2, 100))
ode_pdf = griddata(res[-1, :, :-1], np.exp(res[-1, :, -1]), (m_x, m_y), method='linear')

grid = np.stack([m_x, m_y], -1)

fig, axes = plt.subplots(ncols=2, figsize=(11, 5), sharex=True, sharey=True)
_ = axes[0].pcolormesh(m_x, m_y, st.multivariate_normal.pdf(grid, cov=expm(2 * A.numpy())))
_ = axes[1].pcolormesh(m_x, m_y, ode_pdf.reshape(100, 100))
```

### Informal proof (original so pardon me if you see a mistake)
For a formal proof, please see [6]. Consider a discretization of the ODE

$$
\begin{align}
    x_{t+h} = x_t + h f(x_{t})
\end{align}
$$
Now using the discrete time change of variable,
$$
\ln p(x_{t+h}) = \ln p(x_{t}) - \ln |\mathrm{det}(I + hJ_f)|
$$
Now in $\mathrm{det}(I + hJ_f)$ we recognize the characteristic polynomial of $J_f$:
$$
\begin{align}
\mathrm{det}(I + hJ_f)
    &= h^n \left (h^{-n} + h^{-n+1}Tr(J_f) + \dots \right)\\
    &= \left (1 + hTr(J_f) + \dots \right)\\
\end{align}
$$
and
$$
\ln(1 + hTr(J_f) + \dots) = h Tr(J_f) + o(h)
$$

### Gradient calculation

Now the question becomes: how do we compute the dependency of $−\mathrm{Tr}(𝐽_𝑥[𝑓](𝑥𝑡))$  with respect to its parameter $\theta$?

This can be done using backward dynamics by considering ([6]):
$$
\begin{align}
    \frac{\mathrm{d}a(t)}{\mathrm{d}t}& = a(t)^\top J_x[f] \\
    \frac{\mathrm{d}\frac{\mathrm{d}\ell(t)}{\mathrm{d}\theta}}{\mathrm{d}t}& = a(t)^\top J_\theta[f]
\end{align}
$$

Note that it involves vector-Jacobian products only!
(I'm not going to code it, it's plainly painful to do this in a notebook, but the [tensorflow_probability code](https://github.com/tensorflow/probability/blob/bf84eb67f342e332b7c09b8f435ba534cd91725e/tensorflow_probability/python/math/ode/base.py#L315) is pretty clean!)

```python
with tf.GradientTape() as tape:
    tape.watch(A)
    res = tfp.math.ode.DormandPrince().solve(linear_ode_fun,
                                             t0, x0, solution_times=[t1],
                                             constants={'a': A},
                                             batch_ndims=0)
    loss = tf.reduce_mean(res.states[-1, ..., -1])
tape.gradient(loss, A)
```

    WARNING:tensorflow:From /home/adrien/PycharmProjects/journal-club/venv/lib/python3.8/site-packages/tensorflow_probability/python/math/ode/base.py:459: calling while_loop_v2 (from tensorflow.python.ops.control_flow_ops) with back_prop=False is deprecated and will be removed in a future version.
    Instructions for updating:
    back_prop=False is deprecated. Consider using tf.stop_gradient instead.
    Instead of:
    results = tf.while_loop(c, b, vars, back_prop=False)
    Use:
    results = tf.nest.map_structure(tf.stop_gradient, tf.while_loop(c, b, vars))

    <tf.Tensor: shape=(2, 2), dtype=float32, numpy=
    array([[1.000001, 0.      ],
           [0.      , 1.000001]], dtype=float32)>

### In practice:
In this small example we revisit our square example:

```python
@tf.function
def tf_continuous_planar_flow(_t, x, a, b, w):
    scale = tf_sigmoid(tf.linalg.matvec(x, w) + b)
    return tf.reshape(scale, (-1, 1)) * tf.reshape(a, (1, -1))

@tf.function
def planar_flow_ode_fun(t, x, a, b, w):
    return ode_fun(t, x, tf_continuous_planar_flow, [a, b, w])
```

```python
N = 500
n_iter = 10
learning_rate = 0.1
decay = 0.95

keys = tfp.random.split_seed(31416, n=4)

a_ct = tf.random.stateless_normal((2,), keys[0])
w_ct = tf.random.stateless_normal((2,), keys[1])
b_ct = tf.random.stateless_normal((), keys[2])

x0 = tf.concat([tf.random.stateless_uniform((N, 2), keys[3], 0, 1),
                tf.zeros((N, 1))], axis=1)
dp = tfp.math.ode.DormandPrince()

```

```python
fig, axes = plt.subplots(ncols=2, nrows=2, figsize=(11, 11), sharex=True, sharey=True)
plot_square(axes[0, 0])

lr = learning_rate
for i in range(n_iter):
    with tf.GradientTape() as tape:
        tape.watch([a_ct, b_ct, w_ct])
        res = dp.solve(planar_flow_ode_fun,
                       0., x0, solution_times=[1.],
                       constants={'a': a_ct, 'b': b_ct, 'w': w_ct},
                       batch_ndims=0).states
        loss = tf.reduce_mean(res[-1, ..., -1] - tf.math.log(tf_square(res[-1, ..., :-1])))
        if i == 0:
            axes[0, 1].scatter(*res[-1, :, :2].numpy().T, alpha=0.5)

    grad_a, grad_b, grad_w = tape.gradient(loss, [a_ct, b_ct, w_ct])
    a_vars = a_vars - learning_rate * grad_a
    b_vars = b_vars - learning_rate * grad_b
    w_vars = w_vars - learning_rate * grad_w
    lr = lr * decay

axes[1, 1].scatter(*sample.numpy().T, alpha=0.5)
plot_square(axes[1, 0])
```

I didn't get a chance to show a proper likelihood fitting, but it should't be too hard using the code. In particular, one thing I would have liked to try out in this notebook would have been MCMC CNFs, but I ran out of time.

# To go further
- Augmented neural ODES [7]: some topologies cannot be representend by a bijective Neural-drift ODE. The authors show how to augment the state to improve sampling capabilities.
- Continuous Normalizing Flows via Optimal Transport [8]: efficient path sampling. Many vector fields can lead to the same final distribution, the authors find the most efficient one in the sense of OT.
- Scalable Gradients for Stochastic Differential Equations [9]: what happens when you add a stochastic component? Note: there are a few subtle mistakes in the paper but their code is working.
- FFJORD Free-form Continuous Dynamics for Scalable Reversible Generative Models [10]: An efficient way to compute the trace of the Jacobian.

# References

[1]: Jebara, Tony. Machine Learning: Discriminative and Generative, Springer US, 2004.
[2]: Doersch, Carl. Tutorial on variational autoencoders, arXiv:1606.05908.
[3]: Genevay, Aude and al. Learning Generative Models with Sinkhorn Divergences, AISTATS, 2018.
[4]: Rezende, Danilo and Mohamed, Shakir. Variational Inference with Normalizing Flows, ICML, 2015.
[5]: Papamakarios, George and al. Masked Autoregressive Flow for Density Estimation, NEURIPS, 2017.
[6]: Chen, Ricky T. & al. Neural ordinary differential equations. NEURIPS, 2018.
[7]: Dupont, Emilien & al. Augmented neural ODEs. NEURIPS, 2019.
[8]: Onken, Derek and al. OT-Flow: Fast and Accurate Continuous Normalizing Flows via Optimal Transport. ArXiv:2006.00104, 2020.
[9]: Li, Xuechen and al. Scalable Gradients for Stochastic Differential Equations. AISTATS, 2020.
[10]: Grathwohl, Will.FFJORD Free-form Continuous Dynamics for Scalable Reversible Generative Models, ICLR, 2019.

```python

```


# Introduction to Machine Learning - CSE 474/574

## A practical introduction to IPython Notebook
## Some notes about installation:

### Don't do this:

```
sudo apt-get install ipython-notebook
```

### Instead, do this:

```
pip install ipython tornado pyzmq
```

or install Anaconda from [http://store.continuum.io](http://store.continuum.io)

You can start IPython notebook by running

```
ipython notebook --pylab inline
```

### Lets get started with some simple Bayesian analysis
Adapted from [Probabilistic Programming and Bayesian Methods for Hackers](http://nbviewer.ipython.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Prologue/Prologue.ipynb)

#### Task - Estimate the probability of getting a heads for a coin (should be 0.5 for an unbiased coin)

- Frequentist Approach: Draw several samples and estimate the parameters that best explain the observation (maximum likelihood)
- Bayesian Approach: Start with an initial estimate of the parameter (Prior) and combine it with the likelihood to get an estimate that combines prior and observations (Posterior)

```python
%matplotlib inline
from IPython.core.pylabtools import figsize
import numpy as np
from matplotlib import pyplot as plt
figsize(11, 9)

import scipy.stats as stats

dist = stats.beta # this is the prior distribution
n_trials = [0, 1, 2, 3, 4, 5, 8, 15, 50, 500]
# here we draw samples from a bernoulli distribution which models a coin tossing
# note that in next line we are using 0.5 as the parameter for the Bernoulli distribution
# which means that we are using a fair coin. However during estimation we do not assume knowledge
# of the true parameter
data = stats.bernoulli.rvs(0.5, size=n_trials[-1])
x = np.linspace(0, 1, 100)

# For the already prepared, I'm using Binomial's conj. prior.
# (more about this in a few weeks when we get to Probability based methods)
for k, N in enumerate(n_trials):

    sx = plt.subplot(len(n_trials) / 2, 2, k + 1)

#    plt.setp(sx.get_yticklabels(), visible=False)
    heads = data[:N].sum() # choose only the first N samples in each iteration
    y = dist.pdf(x, 1 + heads, 1 + N - heads) # find the posterior distribution
    # note that in the above line we are computing the posterior distribution for the parameter
    # assuming that the prior distribution is uniform over [0,1]
    plt.plot(x, y, label="observe %d tosses,\n %d heads" % (N, heads))
    plt.fill_between(x, 0, y, color="#348ABD", alpha=0.4)
    plt.vlines(0.5, 0, 4, color="k", linestyles="--", lw=1)

    leg = plt.legend()
    leg.get_frame().set_alpha(0.4)
    plt.autoscale(tight=True)

plt.suptitle("Bayesian updating of posterior probabilities",
             y=1.02,
             fontsize=14)

plt.tight_layout()
```

### Some basic linear algebra

Our goal is to understand and, if possible, solve the system of $n$ linear equations

$$
\begin{align}
a_{00}\,x_0 + a_{01}\,x_1 + \ldots + a_{0,n-1}\,x_{n-1}   &= b_0 \\\
a_{10}\,x_0  + a_{11}\,x_1 + \ldots + a_{1,n-1}\,x_{n-1}   &= b_1 \\\
\vdots &  \\\
a_{n-1,0}\,x_0 + a_{n-1,1}\,x_1 + \ldots + a_{n-1,n-1}\,x_{n-1} &= b_{n-1}\ .
\end{align}
$$

In the system, the $a_{ij}$s and $b_i$s are known, while the $x_i$s are the unkown variables we wish to solve for. In other words, solving the system means finding the values for the $x_i$s using the $a_{ij}$s and $b_i$s.

Using matrix notation, we can write the system as

$$
\begin{pmatrix}
a_{00} & a_{01} & \ldots & a_{0,n-1} \\\
a_{10} & a_{11} & \ldots & a_{1,n-1} \\\
\vdots &  &   \ddots & \vdots \\\
a_{n-1,0} & a_{n-1,1} & \ldots & a_{n-1,n-1}
\end{pmatrix} \,
\begin{pmatrix} x_0 \\\ x_1 \\\ \vdots \\\ x_{n-1}\end{pmatrix}
=
\begin{pmatrix} b_0 \\\ b_1 \\\ \vdots \\\ b_{n-1}\end{pmatrix}\
$$

or $Ax = b$. In this form, a solution to the system is the vector $x$ that satisfies the equation.

```python
# a simple example
import numpy as np
import numpy.linalg as la

A = np.matrix([[  2, 3,  1],
              [0.5, 2, 0],
              [ -1, 5, -7]])
b = np.matrix([[10],[-3],[2]])
x = la.inv(A)*b
print(x)
```

    [[ 17.07692308]
     [ -5.76923077]
     [ -6.84615385]]

```python
# what happens with the following matrix
A = np.array([[  2, 3,  1],
              [0.5, 2, -1],
              [ -1, 5, -7]])
b = np.matrix([[10],[-3],[2]])
x = la.inv(A)*b
print(x)
```

Here the third column, A⋅2 is equal to 2×A⋅0−1×A⋅1, so the columns of this matrix are not linearly independent. This relationship means that x2=2x0−x1, so x2 is not an independent variable, and we really only have two variables in three equations. There will be an infinite number of combinations of x0 and x1 that solve the system.

When the columns of A are not linearly independent, and A has no inverse, it’s called singular or degenerate.


```python
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 5

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Feb. 9 at **2pm.**

## Exercise: AR(1) Process Simulation

For each of the AR(1) processes defined below:

1. Compute $y$ for $201$ periods (i.e., $t=0,\ldots, 200$).
2. Construct a well-labeled plot of $y$ for each simulation.
2. Print the mean and standard deviation of the stimulated process.

You may use the function from the Notebook for Lecture 10, write your own, or take a more brute force approach.

1. $y_{t} = 0.33 y_{t-1} + \epsilon_t, \; \; \; \; \epsilon \sim \mathcal{N}(0,1), \; \; \; \; y_0 = 0$

```python

```

```python
# print mean of y

# Print standard deviation of y

```

2. $y_{t} = y_{t-1} + \epsilon_t, \; \; \; \; \epsilon \sim \mathcal{N}(0,1), \; \; \; \; y_0 = 0$

```python

```

```python
# print mean of y

# Print standard deviation of y

```

3. $y_{t} = -0.95 y_{t-1} + \epsilon_t, \; \; \; \; \epsilon \sim \mathcal{N}(0,1), \; \; \; \; y_0 = 0$

```python

```

```python
# print mean of y

# Print standard deviation of y

```

4. $y_{t} = 0.75 y_{t-1} + \epsilon_t , \; \; \; \; \epsilon \sim \mathcal{N}(0,4), \; \; \; \; y_0 = 20$

```python

```

```python
# print mean of y

# Print standard deviation of y

```

5. $y_{t} = 0.9 y_{t-1} + \epsilon_t , \; \; \; \; \epsilon \sim \mathcal{N}(0,3), \; \; \; \; y_0 = 0$

```python

```

```python
# print mean of y

# Print standard deviation of y

```

## Exercise: Stochastic Growth

We've seen in class that total factory productivity (TFP) in the US fluctuates over the business cycle around a long-term growth trend. That is, total factor productivity is characterized by *stochastic growth*. In this problem, you will simulate a model of TFP with stochastic growth.

### Background

Suppose that the trend component of TFP $A^{trend}$ grows smoothly at the constant rate $g$:

\begin{align}
A^{trend}_t & = (1+g)A_{t-1}^{trend}, \tag{1}
\end{align}

where $A_0$ is given. The log-deviations of actual TFP $A$ from trend TFP are determined according to an AR(1) process:

\begin{align}
\log\left(A_t/A^{trend}_t\right) & = \rho \log \left(A_{t-1} / A^{trend}_{t-1}\right) + \epsilon_t, \tag{2}
\end{align}

where $A_0$ is given and $\epsilon_t \sim \mathcal{N}(0,\sigma^2)$. Solve quation (2) for $A_t$ to get an expression for $A_t$ in terms of $A_{t-1}$, $A^{trend}_{t}$, $A^{trend}_{t-1}$, and $\epsilon_t$:

\begin{align}
A_t & = A^{trend}_t \displaystyle e^{\rho \log \left(A_{t-1} / A^{trend}_{t-1}\right) + \epsilon_t} \tag{3}
\end{align}

### Simulation

To simulate this model:

1. Specify values for $A_0$ and $A^{trend}_0$.
2. For $t = 0, 1, \ldots... T$, compute $A^{trend}_{t+1}$ using equation (1) and then compute $A_{t+1}$ using equation (3)

Use the following prameter values for this simulation

| $$A^{trend}_0$$ | $$A_0$$ | $$g$$  | $$\rho$$ | $$\sigma$$ | $$T$$ |
|-----------------|---------|--------|----------|------------|-------|
| 1               | 1       | 0.02   | 0.7      | 0.025      | 101   |

In the follwing cell, simulate $A^{trend}$ and $A$ using the parameter values provided.

```python
# Simulate the stochastic growth model of TFP

```

Contruct a well-labeled plot of the simulated values of $A$ and $A^{trend}$

```python
# Construct a plot of simulated TFP with its trend with:
#    1. Actual line: blue with lw=1, alpha=0.7, label = 'actual'
#    2. Trend line:  red with lw=3, alpha=0.25, label = 'trend'

```

Contruct a well-labeled plot of the simulated values of $\log \left(A/A^{trend}\right)$

```python
# Construct a plot of simulated log deviation of TFP from trend

```

Compute the mean and standard deviation of $\log \left(A/A^{trend}\right)$

```python
# print mean of log(A/A trend)

# Print standard deviation of log(A/A trend)

```

## Exercise: Questions for Prescott (1986) Reading

The following questions are about Edward Prescott's 1986 article "Theory  Ahead of Business Cycle Measurement" from the Fall 1986 issue of the Minneapolis Fed's *Quarterly Review* (link to article: [https://www.minneapolisfed.org/research/qr/qr1042.pdf](https://www.minneapolisfed.org/research/qr/qr1042.pdf)). The article is a summary of the research agenda of the author, his main collaborator, Finn Kydland, and others. That agenda entailed incorporating stochastic growth of TFP into the neoclassical growth model (essentially a Solow model but with the saving rate determined as a consequence a of utility maximization problem) as a way of modeling business cycle fluctuations. This line of research would later be called *real business cycle* theory.

The below questions are specifically about:
* Pages 9-11 (including the table on page 12)
* The final paragraph on page 21

**Question:** On page 10, Prescott writes that the "models constructed within this theoretical framework are necessarily highly abstract. Consequently they are necessarily false." What does Kydland mean by this? In what sense are abstract models necessarily false?

**Answer**

**Question:** On page 10, Prescott lists two reasons for why he doesn't like the term *business cycle*. What is the first reason and what does it mean?

**Answer**

**Question:** Table 1 on page 12 shows that household consumption measures in the US fluctuate less than investment measures. What is the intuition for why consumption varies less than investment?

**Answer**

**Question:** In the last paragraph of page 21, to what does Prescott attribute economic fluctuations? Can you think of any reasons why this statement was, and still is, controversial?

**Answer**