# Tutorial 1: Biological vs. Artificial neurons

**Week 1, Day 3: Multi Layer Perceptrons**

**By Neuromatch Academy**

__Content creators:__ Arash Ash, Surya Ganguli

__Content reviewers:__ Saeed Salehi, Felix Bartsch, Yu-Fang Yang, Antoine De Comite, Melvin Selim Atay

__Content editors:__ Gagana B, Spiros Chavlis, Kelson Shilling-Scrivo

__Production editors:__ Anoop Kulkarni, Spiros Chavlis, Kelson Shilling-Scrivo

**Our 2021 Sponsors, including Presenting Sponsor Facebook Reality Labs**

---
# Tutorial objectives

In this tutorial we will explore the Multi-leayer Perceptrons (MLPs). MLPs are arguably one of the most tractable models that we can use to study deep learning fundamentals. Here we will learn why they are:

* similar to biological networks
* good at function approximation
* implemented the way they are in PyTorch

---
# Setup

This is a GPU free notebook!

```python
# @title Install dependencies
!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W1D3_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
```

```python
# Imports
import random

import torch
import numpy as np
import matplotlib.pyplot as plt

import torch.nn as nn
import torch.optim as optim
from torch.utils.data import DataLoader, TensorDataset
from tqdm.auto import tqdm
from IPython.display import display
```

```python
# @title Video 0: Introduction
# Insert the ID of the corresponding youtube video
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV11X4y1c7eK", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"kemIQ_8AHdA", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 0: Introduction')

display(out)
```

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/4ye56/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

def imshow(img):
  img = img / 2 + 0.5     # unnormalize
  npimg = img.numpy()
  plt.imshow(np.transpose(npimg, (1, 2, 0)))
  plt.axis(False)
  plt.show()

def plot_function_approximation(x, relu_acts, y_hat):
  fig, axes = plt.subplots(2, 1)

  # Plot ReLU Activations
  axes[0].plot(x, relu_acts.T);
  axes[0].set(xlabel='x',
              ylabel='Activation',
              title='ReLU Activations - Basis Functions')
  labels = [f"ReLU {i + 1}" for i in range(relu_acts.shape[0])]
  axes[0].legend(labels, ncol = 2)

  # Plot function approximation
  axes[1].plot(x, torch.sin(x), label='truth')
  axes[1].plot(x, y_hat, label='estimated')
  axes[1].legend()
  axes[1].set(xlabel='x',
              ylabel='y(x)',
              title='Function Approximation')

  plt.tight_layout()
  plt.show()
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

---
# Section 1: Neuron Physiology and Motivation to Deep Learning

```python
# @title Video 1: Biological to Artificial Neurons
# Insert the ID of the corresponding youtube video
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1mf4y157vf", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"ELAbflymSLo", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 1: Biological to Artificial Neurons')

display(out)
```

## Section 1.1: Leaky Integrate-and-fire (LIF)
The basic idea of LIF neuron was proposed in 1907 by Louis Édouard Lapicque, long before we understood the electrophysiology of a neuron (see a translation of [Lapicque's paper](https://pubmed.ncbi.nlm.nih.gov/17968583/) ). More details of the model can be found in the book [**Theoretical neuroscience**](http://www.gatsby.ucl.ac.uk/~dayan/book/) by Peter Dayan and Laurence F. Abbott.

The model dynamics is defined with the following formula,

\begin{equation}
\frac{d V}{d t}=\left\{\begin{array}{cc}
\frac{1}{C}\left(-\frac{V}{R}+I \right) & t>t_{r e s t} \\
0 & \text { otherwise }
\end{array}\right.
\end{equation}

Note that $V$, $C$, and $R$ are the membrane voltage, capacitance, and resitance of the neuron respectively and $-\frac{V}{R}$ is the leakage current. When $I$ is sufficiently strong such that $V$ reaches a certain threshold value $V_{\rm th}$, it momentarily spikes and then $V$ is reset to $V_{\rm reset}< V_{\rm th}$, and voltage stays at $V_{\rm reset}$ for $\tau_{\rm ref}$ ms, mimicking the refractoriness of the neuron during an action potential (note that $V_{\rm reset}$ and $\tau_{\rm ref}$ is assumed to be zero in the lecture):

\begin{eqnarray}
V(t)=V_{\rm reset} \text{  for } t\in(t_{\text{sp}}, t_{\text{sp}} + \tau_{\text{ref}}]
\end{eqnarray}

where $t_{\rm sp}$ is the spike time when $V(t)$ just exceeded $V_{\rm th}$.

Thus, the LIF model captures the facts that a neuron:
- performs spatial and temporal integration of synaptic inputs
- generates a spike when the voltage reaches a certain threshold
- goes refractory during the action potential
- has a leaky membrane

For in-depth content on computational models of neurons, follow the [NMA](https://www.neuromatchacademy.org/) tutorial 1 of *Biological Neuron Models*. Specifically, for NMA-CN 2021 follow this [Tutorial](https://github.com/NeuromatchAcademy/course-content/blob/master/tutorials/W2D3_BiologicalNeuronModels/W2D3_Tutorial1.ipynb).

## Section 1.2: Simulating an LIF Neuron

In the cell below is given a function for LIF neuron model with it's arguments described.

Note that we will use Euler's method to make a numerical approximation to a derivative. Hence we will use the following implementation of the model dynamics,

\begin{equation}
V_n=\left\{\begin{array}{cc}
V_{n-1} + \frac{1}{C}\left(-\frac{V}{R}+I \right) \Delta t & t>t_{r e s t} \\
0 & \text { otherwise }
\end{array}\right.
\end{equation}

```python
def run_LIF(I, T=50, dt=0.1, t_ref=10,
            Rm=1, Cm=10, Vth=1, V_spike=0.5):
  """
  Simulate the LIF dynamics with external input current

  Args:
    I          : input current (mA)
    T          : total time to simulate (msec)
    dt         : simulation time step (msec)
    t_ref      : refractory period (msec)
    Rm         : resistance (kOhm)
    Cm         : capacitance (uF)
    Vth        : spike threshold (V)
    V_spike    : spike delta (V)

  Returns:
    time       : time points
    Vm         : membrane potentials
  """

  # Set up array of time steps
  time = torch.arange(0, T+dt, dt)

  # Set up array for tracking Vm
  Vm = torch.zeros(len(time))

  # Iterate over each time step
  t_rest = 0
  for i, t in enumerate(time):

    # If t is after refractory period
    if t > t_rest:
      Vm[i] = Vm[i-1] + 1/Cm*(-Vm[i-1]/Rm + I)  * dt

    # If Vm is over the threshold
    if Vm[i] >= Vth:

      # Increase volatage by change due to spike
      Vm[i] += V_spike

      # Set up new refactory period
      t_rest = t + t_ref

  return time, Vm

sim_time, Vm = run_LIF(1.5)
# Plot the membrane voltage across time
plt.plot(sim_time, Vm)
plt.title('LIF Neuron Output')
plt.ylabel('Membrane Potential (V)')
plt.xlabel('Time (msec)')
plt.show()
```

### Interactive Demo 1.2: Neuron's transfer function explorer for different $R_m$ and $t_{ref}$
We know that real neurons communicate by modulating the spike count meaning that more input current causes a neuron to spike more often. Therefore, to find an input-output relationship, it makes sense to characterize their spike count as a function of input current. This is called the neuron's input-output transfer function. Let's plot the neuron's transfer function and see how it changes with respect to the **membrane resistance** and **refractory time**?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!
my_layout = widgets.Layout()

@widgets.interact(Rm=widgets.FloatSlider(1., min=1, max=100.,
                                         step=0.1, layout=my_layout),
                  t_ref=widgets.FloatSlider(1., min=1, max=100.,
                                            step=0.1, layout=my_layout)
                  )

def plot_IF_curve(Rm, t_ref):
  T = 1000 # total time to simulate (msec)
  dt = 1 # simulation time step (msec)
  Vth = 1 # spike threshold (V)
  Is_max = 2
  Is = torch.linspace(0, Is_max, 10)
  spike_counts = []
  for I in Is:
    _, Vm = run_LIF(I, T=T, dt=dt, Vth=Vth, Rm=Rm, t_ref=t_ref)
    spike_counts += [torch.sum(Vm > Vth)]

  plt.plot(Is, spike_counts)
  plt.title('LIF Neuron: Transfer Function')
  plt.ylabel('Spike count')
  plt.xlabel('I (mA)')
  plt.xlim(0, Is_max)
  plt.ylim(0, 80)
  plt.show()
```

### Think! 1.2: Real and Artificial neuron similarities

What happens at infinite membrane resistance (Rm) and small refactory time (t_ref)? Why?

Take 10 mins to discuss the similarity between a real neuron and an artificial one with your pod.

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type answer here and Push submit',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q1' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

```python
# to_remove explanation

"""
At infinite membrane resistance, the Neuron does not leak any current out,
and hence it starts firing with the slightest input current,
This shifts the transfer function towards 0, similar to ReLU activation (centered at 0).
Also, when there is minimal refractory time, the neuron can keep firing
at a high input current which avoids the saturation.
""";
```

---
# Section 2: The Need for MLPs

```python
# @title Video 2: Universal Approximation Theorem
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1SP4y147Uv", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"tg8HHKo1aH4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 2: Universal Approximation Theorem')

display(out)
```

## Coding Exercise 2: Function approximation with ReLU
We learned that one hidden layer MLPs are enough to approximate any smooth function! Now let's manually fit a sine function using ReLU activation.

We will approximate the sine function using a linear combination (a weighted sum) of ReLUs with slope 1. We need to determine the bias terms (which determines where the ReLU inflection point from 0 to linear occurs) and how to weight each ReLU. The idea is to set the weights iteratively so that the slope changes in the new sample's direction.

First, we generate our "training data" from a sine function using `torch.sine` function.

```python
>>> import torch
>>> torch.manual_seed(2021)
<torch._C.Generator object at 0x7f8734c83830>
>>> a = torch.randn(5)
>>> print(a)
tensor([ 2.2871,  0.6413, -0.8615, -0.3649, -0.6931])
>>> torch.sin(a)
tensor([ 0.7542,  0.5983, -0.7588, -0.3569, -0.6389])
```

These are the points we will use to learn how to approximate the function. We have 10 training data points so we will have 9 ReLUs (we don't need a ReLU for the last data point as we don't have anything to the right of it to model).

We first need to figure out the bias term for each ReLU and compute the activation of each ReLU where:

\begin{equation}
y(x) = \text{max}(0,x+b)
\end{equation}

We then need to figure out the correct weights on each ReLU so the linear combination approximates the desired function.

```python
def approximate_function(x_train, y_train):

  ####################################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Complete approximate_function!")
  ####################################################################

  # Number of relus
  n_relus = x_train.shape[0] - 1

  # x axis points (more than x train)
  x = torch.linspace(torch.min(x_train), torch.max(x_train), 1000)

  ## COMPUTE RELU ACTIVATIONS

  # First determine what bias terms should be for each of `n_relus` ReLUs
  b = ...

  # Compute ReLU activations for each point along the x axis (x)
  relu_acts = torch.zeros((n_relus, x.shape[0]))

  for i_relu in range(n_relus):
    relu_acts[i_relu, :] = torch.relu(x + b[i_relu])

  ## COMBINE RELU ACTIVATIONS

  # Set up weights for weighted sum of ReLUs
  combination_weights = torch.zeros((n_relus, ))

  # Figure out weights on each ReLU
  prev_slope = 0
  for i in range(n_relus):
    delta_x = x_train[i+1] - x_train[i]
    slope = (y_train[i+1] - y_train[i]) / delta_x
    combination_weights[i] = ...
    prev_slope = slope

  # Get output of weighted sum of ReLU activations for every point along x axis
  y_hat = ...

  return y_hat, relu_acts, x

# add event to airtable
atform.add_event('Coding Exercise 2: Function approximation with ReLU')

# Make training data from sine function
N_train = 10
x_train = torch.linspace(0, 2*np.pi, N_train).view(-1, 1)
y_train = torch.sin(x_train)

## Uncomment the lines below to test your function approximation
# y_hat, relu_acts, x = approximate_function(x_train, y_train)
# plot_function_approximation(x, relu_acts, y_hat)
```

```python
# to_remove solution
def approximate_function(x_train, y_train):

  # Number of relus
  n_relus = x_train.shape[0] - 1

  # x axis points (more than x train)
  x = torch.linspace(torch.min(x_train), torch.max(x_train), 1000)

  ## COMPUTE RELU ACTIVATIONS

  # First determine what bias terms should be for each of `n_relus` ReLUs
  b = -x_train[:-1]

  # Compute ReLU activations for each point along the x axis (x)
  relu_acts = torch.zeros((n_relus, x.shape[0]))

  for i_relu in range(n_relus):
    relu_acts[i_relu, :] = torch.relu(x + b[i_relu])

  ## COMBINE RELU ACTIVATIONS

  # Set up weights for weighted sum of ReLUs
  combination_weights = torch.zeros((n_relus, ))

  # Figure out weights on each ReLU
  prev_slope = 0
  for i in range(n_relus):
    delta_x = x_train[i+1] - x_train[i]
    slope = (y_train[i+1] - y_train[i]) / delta_x
    combination_weights[i] = slope - prev_slope
    prev_slope = slope

  # Get output of weighted sum of ReLU activations for every point along x axis
  y_hat = combination_weights @ relu_acts

  return y_hat, relu_acts, x

# add event to airtable
atform.add_event('Coding Exercise 2: Function approximation with ReLU')

# Make training data from sine function
N_train = 10
x_train = torch.linspace(0, 2*np.pi, N_train).view(-1, 1)
y_train = torch.sin(x_train)

## uncomment the lines below to test your function approximation
y_hat, relu_acts, x = approximate_function(x_train, y_train)
with plt.xkcd():
  plot_function_approximation(x, relu_acts, y_hat)
```

As you see in the top panel, we obtain 10 shifted ReLUs with the same slope. These are the basis functions that MLP uses to span the functional space, i.e., MLP finds a linear combination of these ReLUs.

---
# Section 3: MLPs in Pytorch

```python
# @title Video 3: Building MLPs in PyTorch
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1zh411z7LY", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"XtwLnaYJ7uc", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 3: Building MLPs in PyTorch')

display(out)
```

In the previous segment, we implemented a function to approximate any smooth function using MLPs. We saw that using Lipschitz continuity; we can prove that our approximation is mathematically correct. MLPs are fascinating, but before we get into the details on designing them, let's familiarize ourselves with some basic terminology of MLPs- layer, neuron, depth, width, weight, bias, and activation function. Armed with these ideas, we can now design an MLP given its input, hidden layers, and output size.

## Coding Exercise 3: Implement a general-purpose MLP in Pytorch
The objective is to design an MLP with these properties:
* works with any input (1D, 2D, etc.)
* construct any number of given hidden layers using `nn.Sequential()` and `add_module()` function
* use the same given activation function (i.e., [Leaky ReLU](https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html)) in all hidden layers.

**Leaky ReLU** is described by the following mathematical formula:

\begin{equation}
\text{LeakyReLU}(x) = \text{max}(0,x) + \text{negative_slope} \cdot \text{min}(0, x) =
\left\{
  \begin{array}{ll}
  x & ,\; \text{if} \; x \ge 0 \\
  \text{negative_slope} \cdot x & ,\;  \text{otherwise}
  \end{array}
\right.
\end{equation}

```python
class Net(nn.Module):
  def __init__(self, actv, input_feature_num, hidden_unit_nums, output_feature_num):
    super(Net, self).__init__()
    self.input_feature_num = input_feature_num # save the input size for reshapinng later
    self.mlp = nn.Sequential() # Initialize layers of MLP

    in_num = input_feature_num # initialize the temporary input feature to each layer
    for i in range(len(hidden_unit_nums)): # Loop over layers and create each one

      ####################################################################
      # Fill in missing code below (...),
      # then remove or comment the line below to test your function
      raise NotImplementedError("Create MLP Layer")
      ####################################################################

      out_num = hidden_unit_nums[i] # assign the current layer hidden unit from list
      layer = ... # use nn.Linear to define the layer
      in_num = out_num # assign next layer input using current layer output
      self.mlp.add_module('Linear_%d'%i, layer) # append layer to the model with a name

      actv_layer = eval('nn.%s'%actv) # Assign activation function (eval allows us to instantiate object from string)
      self.mlp.add_module('Activation_%d'%i, actv_layer) # append activation to the model with a name

    out_layer = nn.Linear(in_num, output_feature_num) # Create final layer
    self.mlp.add_module('Output_Linear', out_layer) # append the final layer

  def forward(self, x):
    # reshape inputs to (batch_size, input_feature_num)
    # just in case the input vector is not 2D, like an image!
    x = x.view(-1, self.input_feature_num)

    ####################################################################
    # Fill in missing code below (...),
    # then remove or comment the line below to test your function
    raise NotImplementedError("Run MLP model")
    ####################################################################

    logits = ... # forward pass of MLP
    return logits

input = torch.zeros((100, 2))
## Uncomment below to create network and test it on input
# net = Net(actv='LeakyReLU(0.1)', input_feature_num=2, hidden_unit_nums=[100, 10, 5], output_feature_num=1).to(DEVICE)
# y = net(input.to(DEVICE))
# print(f'The output shape is {y.shape} for an input of shape {input.shape}')
```

```python
# to_remove solution
class Net(nn.Module):
  def __init__(self, actv, input_feature_num, hidden_unit_nums, output_feature_num):
    super(Net, self).__init__()
    self.input_feature_num = input_feature_num # save the input size for reshapinng later
    self.mlp = nn.Sequential() # Initialize layers of MLP

    in_num = input_feature_num # initialize the temporary input feature to each layer
    for i in range(len(hidden_unit_nums)): # Loop over layers and create each one

      out_num = hidden_unit_nums[i] # assign the current layer hidden unit from list
      layer = nn.Linear(in_num, out_num) # use nn.Linear to define the layer
      in_num = out_num # assign next layer input using current layer output
      self.mlp.add_module('Linear_%d'%i, layer) # append layer to the model with a name

      actv_layer = eval('nn.%s'%actv) # Assign activation function (eval allows us to instantiate object from string)
      self.mlp.add_module('Activation_%d'%i, actv_layer) # append activation to the model with a name

    out_layer = nn.Linear(in_num, output_feature_num) # Create final layer
    self.mlp.add_module('Output_Linear', out_layer) # append the final layer

  def forward(self, x):
    # reshape inputs to (batch_size, input_feature_num)
    # just in case the input vector is not 2D, like an image!
    x = x.view(-1, self.input_feature_num)

    logits = self.mlp(x) # forward pass of MLP
    return logits

input = torch.zeros((100, 2))
## Uncomment below to create network and test it on input
net = Net(actv='LeakyReLU(0.1)', input_feature_num=2, hidden_unit_nums=[100, 10, 5], output_feature_num=1).to(DEVICE)
y = net(input.to(DEVICE))
print(f'The output shape is {y.shape} for an input of shape {input.shape}')
```

```
The output shape is torch.Size([100, 1]) for an input of shape torch.Size([100, 2])
```

## Section 3.1: Classification with MLPs

```python
# @title Video 4: Cross Entropy
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Ag41177mB", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"N8pVCbTlves", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 4: Cross Entropy')

display(out)
```

The main loss function we could use out of the box for multi-class classification for `N` samples and `C` number of classes is:
* CrossEntropyLoss:
This criterion expects a batch of predictions `x` with shape `(N, C)` and class index in the range $[0, C-1]$ as the target (label) for each `N` samples, hence a batch of `labels` with shape `(N, )`. There are other optional parameters like class weights and class ignores. Feel free to check the PyTorch documentation [here](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) for more detail. Additionally, [here](https://sparrow.dev/cross-entropy-loss-in-pytorch/) you can learn where is appropriate to use the CrossEntropyLoss.

To get CrossEntropyLoss of a sample $i$, we could first calculate $-\log(\text{softmax(x}))$ and then take the element corresponding to $\text { labels }_i$ as the loss. However, due to numerical stability, we implement this more stable equivalent form,

\begin{equation}
\operatorname{loss}(x_i, \text { labels }_i)=-\log \left(\frac{\exp (x[\text { labels }_i])}{\sum_{j} \exp (x[j])}\right)=-x_i[\text { labels }_i]+\log \left(\sum_{j=1}^C \exp (x_i[j])\right)
\end{equation}

### Coding Exercise 3.1: Implement Batch Cross Entropy Loss

To recap, since we will be doing batch learning, we'd like a loss function that given:
* a batch of predictions `x` with shape `(N, C)`
* a batch of `labels` with shape `(N, )` that ranges from `0` to `C-1`

returns the average loss $L$ calculated according to:

\begin{align}
loss(x_i, \text { labels }_i) &= -x_i[\text { labels }_i]+\log \left(\sum_{j=1}^C \exp (x_i[j])\right) \\
L &= \frac{1}{N} \sum_{i=1}^{N}{loss(x_i, \text { labels }_i)}
\end{align}

Steps:

1.   Use indexing operation to get predictions of class corresponding to the labels (i.e., $x_i[\text { labels }_i]$)
2.   Compute $loss(x_i, \text { labels }_i)$ vector (`losses`) using `torch.log()` and `torch.exp()` without Loops!
3. Return the average of the loss vector

```python
def cross_entropy_loss(x, labels):
  # x is the model predictions we'd like to evaluate using lables
  x_of_labels = torch.zeros(len(labels))
  ####################################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Cross Entropy Loss")
  ####################################################################
  # 1. prediction for each class corresponding to the label
  for i, label in enumerate(labels):
    x_of_labels[i] = x[i, label]
  # 2. loss vector for the batch
  losses = ...
  # 3. Return the average of the loss vector
  avg_loss = ...

  return avg_loss

# add event to airtable
atform.add_event('Coding Exercise 3.1: Implement Batch Cross Entropy Loss')

labels = torch.tensor([0, 1])
x = torch.tensor([[10.0, 1.0, -1.0, -20.0],  # correctly classified
                  [10.0, 10.0, 2.0, -10.0]])  # Not correctly classified
CE = nn.CrossEntropyLoss()
pytorch_loss = CE(x, labels).item()
## Uncomment below to test your function
# our_loss = cross_entropy_loss(x, labels).item()
# print(f'Our CE loss: {our_loss:0.8f}, Pytorch CE loss: {pytorch_loss:0.8f}')
# print(f'Difference: {np.abs(our_loss - pytorch_loss):0.8f}')
```

```python
# to_remove solution
def cross_entropy_loss(x, labels):
  # x is the model predictions we'd like to evaluate using lables
  x_of_labels = torch.zeros(len(labels))
  # 1. prediction for each class corresponding to the label
  for i, label in enumerate(labels):
    x_of_labels[i] = x[i, label]
  # 2. loss vector for the batch
  losses = -x_of_labels + torch.log(torch.sum(torch.exp(x), axis=1))
  # 3. Return the average of the loss vector
  avg_loss = losses.mean()

  return avg_loss

# add event to airtable
atform.add_event('Coding Exercise 3.1: Implement Batch Cross Entropy Loss')

labels = torch.tensor([0, 1])
x = torch.tensor([[10.0, 1.0, -1.0, -20.0],  # correctly classified
                  [10.0, 10.0, 2.0, -10.0]])  # Not correctly classified
CE = nn.CrossEntropyLoss()
pytorch_loss = CE(x, labels).item()
## Uncomment below to test your function
our_loss = cross_entropy_loss(x, labels).item()
print(f'Our CE loss: {our_loss:0.8f}, Pytorch CE loss: {pytorch_loss:0.8f}')
print(f'Difference: {np.abs(our_loss - pytorch_loss):0.8f}')
```

```
Our CE loss: 0.34672737, Pytorch CE loss: 0.34672749
Difference: 0.00000012
```

## Section 3.2: Spiral classification dataset
Before we could start optimizing these loss functions, we need a dataset!

Let's turn this fancy-looking equation into a classification dataset

\begin{equation}
\begin{array}{c}
X_{k}(t)=t\left(\begin{array}{c}
\sin \left[\frac{2 \pi}{K}\left(2 t+k-1\right)\right]+\mathcal{N}\left(0, \sigma\right) \\
\cos \left[\frac{2 \pi}{K}\left(2 t+k-1\right)\right]+\mathcal{N}\left(0, \sigma\right)
\end{array}\right)
\end{array}, \quad 0 \leq t \leq 1, \quad k=1, \ldots, K
\end{equation}

```python
def create_spiral_dataset(K, sigma, N):

  # Initialize t, X, y
  t = torch.linspace(0, 1, N)
  X = torch.zeros(K*N, 2)
  y = torch.zeros(K*N)

  # Create data
  for k in range(K):
    X[k*N:(k+1)*N, 0] = t*(torch.sin(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))
    X[k*N:(k+1)*N, 1] = t*(torch.cos(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))
    y[k*N:(k+1)*N] = k

  return X, y

# Set parameters
K = 4
sigma = 0.16
N = 1000

set_seed(seed=SEED)
X, y = create_spiral_dataset(K, sigma, N)
plt.scatter(X[:, 0], X[:, 1], c = y)
plt.show()
```

## Section 3.3: Training and Evaluation

```python
# @title Video 5: Training and Evaluating an MLP
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1QV411p7mF", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"DfXZhRfBEqQ", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 5: Training and Evaluating an MLP')

display(out)
```

### Coding Exercise 3.3: Implement it for a classfication task
Now that we have the Spiral dataset and a loss function, it's your turn to implement a simple train/test split for training and validation.

Steps to follow:
  * Dataset shuffle
  * Train/Test split (20% for test)
  * Dataloader definition
  * Training and Evaluation

```python
def shuffle_and_split_data(X, y, seed):
  # set seed for reproducibility
  torch.manual_seed(seed)
  # Number of samples
  N = X.shape[0]
  ####################################################################
  # Fill in missing code below (...),
  # then remove or comment the line below to test your function
  raise NotImplementedError("Shuffle & split data")
  ####################################################################
  # Shuffle data
  shuffled_indices = ...   # get indices to shuffle data, could use torch.randperm
  X = X[shuffled_indices]
  y = y[shuffled_indices]

  # Split data into train/test
  test_size = ...    # assign test datset size using 20% of samples
  X_test = X[:test_size]
  y_test = y[:test_size]
  X_train = X[test_size:]
  y_train = y[test_size:]

  return X_test, y_test, X_train, y_train

# add event to airtable
atform.add_event('Coding Exercise 3.3: Implement for a classfication task')

## Uncomment below to test your function
# X_test, y_test, X_train, y_train = shuffle_and_split_data(X, y, seed=SEED)
# plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test)
# plt.title('Test data')
# plt.show()
```

```python
# to_remove solution
def shuffle_and_split_data(X, y, seed):
  # set seed for reproducibility
  torch.manual_seed(seed)
  # Number of samples
  N = X.shape[0]
  # Shuffle data
  shuffled_indices = torch.randperm(N)   # get indices to shuffle data, could use torch.randperm
  X = X[shuffled_indices]
  y = y[shuffled_indices]

  # Split data into train/test
  test_size = int(0.2 * N)    # assign test datset size using 20% of samples
  X_test = X[:test_size]
  y_test = y[:test_size]
  X_train = X[test_size:]
  y_train = y[test_size:]

  return X_test, y_test, X_train, y_train

# add event to airtable
atform.add_event('Coding Exercise 3.3: Implement for a classfication task')

## Uncomment below to test your function
X_test, y_test, X_train, y_train = shuffle_and_split_data(X, y, seed=SEED)
with plt.xkcd():
  plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test)
  plt.title('Test data')
  plt.show()
```

And we need to make a Pytorch data loader out of it. Data loading in PyTorch can be separated in 2 parts:
* Data must be wrapped on a Dataset parent class where the methods __getitem__ and __len__ must be overrided. Note that at this point the data is not loaded on memory. PyTorch will only load what is needed to the memory. Here `TensorDataset` does this for us directly.
* Use a Dataloader that will actually read the data in batches and put into memory. Also, the option of `num_workers > 0` allows multithreading, which prepares multiple batches in the queue to speed things up.

```python
g_seed = torch.Generator()
g_seed.manual_seed(SEED)

batch_size = 128
test_data = TensorDataset(X_test, y_test)
test_loader = DataLoader(test_data, batch_size=batch_size,
                         shuffle=False, num_workers=2,
                         worker_init_fn=seed_worker,
                         generator=g_seed)

train_data = TensorDataset(X_train, y_train)
train_loader = DataLoader(train_data, batch_size=batch_size, drop_last=True,
                        shuffle=True, num_workers=2,
                         worker_init_fn=seed_worker,
                         generator=g_seed)
```

Let's write a general-purpose training and evaluation code and keep it in our pocket for next tutorial as well. So make sure you review it to see what it does.

Note that `model.train()` tells your model that you are training the model. So layers like dropout, batchnorm etc. which behave different on the train and test procedures know what is going on and hence can behave accordingly. And to turn off training mode we set `model.eval()`

```python
def train_test_classification(net, criterion, optimizer, train_loader,
                              test_loader, num_epochs=1, verbose=True,
                              training_plot=False, device='cpu'):

  net.train()
  training_losses = []
  for epoch in tqdm(range(num_epochs)):  # loop over the dataset multiple times
    running_loss = 0.0
    for i, data in enumerate(train_loader, 0):
      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = data
      inputs = inputs.to(device).float()
      labels = labels.to(device).long()

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = net(inputs)

      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()

      # print statistics
      if verbose:
        training_losses += [loss.item()]

  net.eval()
  def test(data_loader):
    correct = 0
    total = 0
    for data in data_loader:
      inputs, labels = data
      inputs = inputs.to(device).float()
      labels = labels.to(device).long()

      outputs = net(inputs)
      _, predicted = torch.max(outputs, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum().item()

    acc = 100 * correct / total
    return total, acc

  train_total, train_acc = test(train_loader)
  test_total, test_acc = test(test_loader)

  if verbose:
    print(f"Accuracy on the {train_total} training samples: {train_acc:0.2f}")
    print(f"Accuracy on the {test_total} testing samples: {test_acc:0.2f}")

  if training_plot:
    plt.plot(training_losses)
    plt.xlabel('Batch')
    plt.ylabel('Training loss')
    plt.show()

  return train_acc, test_acc
```

### Think! 3.3.1: What's the point of .eval() and .train()?

Is it necessary to use `net.train()` and `net.eval()` for our MLP model? why?

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type answer here and Push submit',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q2', text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

```python
# to_remove explanation

"""
Nope, since we did not use any layers that have different behaviours during test and train.
But these layers are very common so it's a good practice to always do it!
""";
```

Now let's put everything together and train your first deep-ish model!

```python
set_seed(SEED)
net = Net('ReLU()', X_train.shape[1], [128], K).to(DEVICE)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=1e-3)
num_epochs = 100

_, _ = train_test_classification(net, criterion, optimizer, train_loader,
                                 test_loader, num_epochs=num_epochs,
                                 training_plot=True, device=DEVICE)
```

And finally, let's visualize the learned decision-map. We know you're probably running out of time, so we won't make you write code now! But make sure you have reviewed it since we'll start with another visualization technique next time.

```python
def sample_grid(M=500, x_max=2.0):
  ii, jj = torch.meshgrid(torch.linspace(-x_max, x_max, M),
                          torch.linspace(-x_max, x_max, M))
  X_all = torch.cat([ii.unsqueeze(-1),
                     jj.unsqueeze(-1)],
                     dim=-1).view(-1, 2)
  return X_all

def plot_decision_map(X_all, y_pred, X_test, y_test,
                      M=500, x_max=2.0, eps=1e-3):
  decision_map = torch.argmax(y_pred, dim=1)

  for i in range(len(X_test)):
    indices = (X_all[:, 0] - X_test[i, 0])**2 + (X_all[:, 1] - X_test[i, 1])**2 < eps
    decision_map[indices] = (K + y_test[i]).long()

  decision_map = decision_map.view(M, M)
  plt.imshow(decision_map, extent=[-x_max, x_max, -x_max, x_max], cmap='jet')
  plt.show()
```

```python
X_all = sample_grid()
y_pred = net(X_all)
plot_decision_map(X_all, y_pred, X_test, y_test)
```

### Think! 3.3.2: Does it generalize well?
Do you think this model is performing well outside its training distribution? Why?

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type answer here and Push submit',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q3' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

```python
# to_remove explanation

"""
Not so much, since the true model is a spiral,
we'd expect that the model continues to have circular decision boundaries
as we go beyond the train distribution
""";
```

What would be your suggestions to increase models ability to generalize? Think about it and discuss with your pod.

---
# Summary

In this tutorial we have explored the Multi-leayer Perceptrons (MLPs). More specifically, we have discuss the similarities of artificial and biological neural networks, we have learned the Universal Approximation Theorem, and we have implemented MLPs in PyTorch.

```python

# @title Airtable Submission Link
from IPython import display as IPydisplay
IPydisplay.HTML(
   f"""
 <div>
   <a href= "{atform.url()}" target="_blank">
   </a>
   </div>""" )
```


# 第7章 確率ベクトルの変換

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from numpy.random import *
```

```python
def SampleMean(Π):
    N = len(Π)
    if N == 0:
        return False #計算不可能
    else:
        return sum(Π)/len(Π)

def SampleVariance(Π):
    m=SampleMean(Π)  #標本平均
    return SampleMean( (Π-m*np.ones_like(Π))**2 )

def Cov(X,Y):
    mx = SampleMean(X)
    my = SampleMean(Y)
    return SampleMean(X*Y)-mx*my
```

```python
def _gauss(x,m,v):
    return 1.0/np.sqrt(2*np.pi*v) * np.exp(-(x-m)**2/(2*v))

gauss = np.vectorize(_gauss,excluded=['m','v'])
```

## 7.1 確率変数の加法とスカラ倍

## 7.1.1 加法 $Z = X + Y$ の分布

### 例題7.1

```python
import sympy as sym
sym.init_printing(use_latex='mathjax') #整形した数式の表示

def Example_7_1(np_zz):
    x, y, z = sym.symbols('z y z')
    m, v = sym.symbols('m v')
    a, b = sym.symbols('a b')
    # pX(x) ガウス，pXzy = pX(z-y)
    pX = 1/sym.sqrt(2*sym.pi*v) * sym.exp(-(x-m)**2/(2*v))
    gauss_erf = sym.simplify( sym.integrate(pX,(x, 0, sym.Symbol('A'))) )
    pXzy = pX.subs([(x,z-y)])
    #print(pXzy)
    pZ = sym.simplify(sym.integrate(pXzy/(b-a), (y, a, b)))
    list_pZ = [ pZ.subs([(z,_z),(m,0),(v,1),(a,-2),(b,5)]) for _z in np_zz ]
    return pZ, np.array(list_pZ), gauss_erf

np_zz = np.linspace(-6,10,100)
sym_pZ, np_pZ, gauss_erf = Example_7_1(np_zz)
sym_pZ
```

$$\frac{1}{2 \left(a - b\right)} \left(\operatorname{erf}{\left (\frac{\sqrt{2}}{2 \sqrt{v}} \left(a + m - z\right) \right )} - \operatorname{erf}{\left (\frac{\sqrt{2}}{2 \sqrt{v}} \left(b + m - z\right) \right )}\right)$$

解答例の計算に使うガウス分布の累積積分 ※erf$(-x)$=$-$erf$(x)$に注意！

```python
gauss_erf
```

$$\frac{1}{2} \operatorname{erf}{\left (\frac{\sqrt{2} m}{2 \sqrt{v}} \right )} + \frac{1}{2} \operatorname{erf}{\left (\frac{\sqrt{2} \left(A - m\right)}{2 \sqrt{v}} \right )}$$

### 図7.1

```python
X = randn(100000) #平均0分散1の正規乱数
Y = uniform(-2, 5, 100000) #区間[-2,5]の一様乱数
Z = X + Y

plt.figure(figsize=(5,2))
plt.hist(Z, range=[-6,10], bins=16, density=True, color='w', edgecolor='k')
plt.plot(np_zz,np_pZ, 'k-') #例題7.1で導出した式
plt.xlim([-6,10])
plt.xticks( np.arange(-6, 11, 2) )
plt.xlabel('$z$', fontsize=12)
plt.ylabel('$p_Z(z)$', fontsize=12)
plt.text(5.7, 0.12,'$m=0$, $v=1$')
plt.text(5.7, 0.09,'$a=-2$, $b=5$')
plt.tight_layout()
plt.savefig('figs/Ch07-gauss_uniform.eps', bb_inches='tight')
```

## 7.2 ガウス確率変数の加法とスカラ倍

## 7.2.1 加法 $Z = X + Y$ の分布

```python
X = randn(100000) + 5*np.ones(100000)
Y = X + randn(100000)
```

```python
mx = SampleMean(X) # Xの期待値
my = SampleMean(Y) # Yの期待値
vx = SampleVariance(X) # Xの分散
vy = SampleVariance(Y) # Yの分散
Cxy = Cov(X,Y) # X,Yの共分散
mx, my, vx, vy, Cxy
```

$$\left ( 5.003215749042992, \quad 5.005103605232999, \quad 0.9971453405948929, \quad 1.9989074994142773, \quad 0.9978212271095934\right )$$

```python
mz = mx + my
vz = vx + 2*Cxy + vy
```

```python
Z = X + Y
```

```python
plt.hist(Z, range=[0,20], bins=21, normed=True)
xx = np.linspace(0,20,100)
plt.plot(xx, gauss(xx,mz,vz),'-')
plt.xlabel('$z$')
plt.ylabel('$p(z)$')
plt.grid(linestyle='--')
```

## 7.2.2 スカラ倍 $Z = a X$ の分布

```python
X = randn(100000) + 5*np.ones_like(X)
```

```python
mx = SampleMean(X) # Xの期待値
vx = SampleVariance(X) # Xの分散
mx, vx
```

$$\left ( 4.998777615535563, \quad 0.9971116584657035\right )$$

```python
a = 0.5
mz = a*mx
vz = (a**2)*vx
```

```python
Z = a*X
```

```python
plt.hist(Z, range=[0,5], bins=21, normed=True)
xx = np.linspace(0,5,100)
plt.plot(xx, gauss(xx,mz,vz),'-')
plt.xlabel('$z$')
plt.ylabel('$p(z)$')
plt.grid(linestyle='--')
```

## 7.2.3 線形結合 $Z = a_1 X_1 + a_2 X_2$ の分布

```python
X1 = randn(100000) + 5*np.ones_like(X)
X2 = X1 + randn(100000)
```

```python
m1 = SampleMean(X1) # X1の期待値
m2 = SampleMean(X2) # X2の期待値
v1 = SampleVariance(X1) # X1の分散
v2 = SampleVariance(X2) # X2の分散
C12 = Cov(X1,X2) # X,Yの共分散
m1, m2, v1, v2, C12
```

$$\left ( 4.993018944113484, \quad 4.994622009447813, \quad 1.0091661730120924, \quad 1.9940143270333988, \quad 1.0032547238840444\right )$$

```python
a1 = 0.5
a2 = 2
mz = a1*mx + a2*my
vz = (a1**2)*vx + 2*a1*a2*C12 + (a2**2)*vy
```

```python
Z = a1*X1 + a2*X2
```

```python
plt.hist(Z, range=[0,25], bins=21, normed=True)
xx = np.linspace(0,25,100)
plt.plot(xx, gauss(xx,mz,vz),'-')
plt.xlabel('$z$')
plt.ylabel('$p(z)$')
plt.grid(linestyle='--')
```

## 7.3 共分散行列と多変数ガウス分布

## 7.3.1 共分散行列

```python
def CovMatrix(XX,YY):
    dim = len(XX[0])
    cov = np.zeros([dim,dim])
    for i in range(dim):
        cov[i,i] = Cov(XX[:,i], YY[:,i])
        for j in range(i):
            cov[i,j] = Cov(XX[:,i], YY[:,j])
            cov[j,i] = cov[i,j]
    return cov
```

### 数値例：3次元ガウス分布のランダムデータ

```python
means = [1,2,3] #期待値ベクトル
covmat = [[1,0.5,0.2],[0.5,2,1],[0.2,1,3]] #分散共分散行列
data_for_test = np.random.multivariate_normal(means, covmat, 100000)
fig = plt.figure()
ax = Axes3D(fig)
ax.plot(data_for_test[:,0],data_for_test[:,1],data_for_test[:,2],".",markersize=0.5)
```

#### データの分散共分散行列

標本数が有限なので，ぴったりではないですが，covmat の成分と一致します．

```python
CovMatrix(data_for_test,data_for_test)
```

    array([[1.00379077, 0.50090574, 0.20194248],
           [0.50090574, 1.99579224, 0.99797172],
           [0.20194248, 0.99797172, 2.98543577]])

## 7.3.2 多変数ガウス分布

$n$次元の式を使って，2次元のガウス分布を記述してみる．

```python
def _Gauss2dv(_x1, _x2, _mm, _covmat):
    xx = np.array([_x1,_x2])
    mm = np.array(_mm)
    covmat = np.array(_covmat)
    n = len(mm)
    det = np.linalg.det(covmat)
    inv = np.linalg.pinv(covmat)
    xxm = xx - mm
    a = 1/( (np.sqrt(2*np.pi)**n)*np.sqrt(det) )
#    b = -0.5*np.dot(xxm,np.dot(inv,xxm))
    b = -0.5*xxm.dot(inv.dot(xxm))
    return a*np.exp(b)

Gauss2dv = np.vectorize(_Gauss2dv, excluded=[2,3])
```

```python
xx = np.linspace(-2,4,50)
yy = np.linspace(-5,5,50)
```

```python
Means = np.array([1,0])
CovMat = np.array([[1,0.8],[0.8,2]])

XX,YY = np.meshgrid(xx,yy)
ZZ = Gauss2dv(XX, YY, Means, CovMat)
```

```python
from matplotlib import cm

def density_plot(Xgrid, Ygrid, Zgrid, zlabel='zlabel', cmap=cm.coolwarm):
    fig = plt.figure(figsize=(10,4))
    ax2d = fig.add_subplot(121)
    ax3d = fig.add_subplot(122, projection='3d')
    cont = ax2d.contourf(Xgrid, Ygrid, Zgrid, cmap=cmap)
    ax2d.set_xlabel('$x$')
    ax2d.set_ylabel('$y$')
    surf = ax3d.plot_surface(Xgrid, Ygrid, Zgrid, cmap=cmap, linewidth=0)
    ax3d.set_xlabel('$x$')
    ax3d.set_ylabel('$y$')
    ax3d.set_zlabel(zlabel)
    fig.colorbar(cont, ax=ax2d, label=zlabel)
    plt.tight_layout(pad=1)
```

```python
density_plot(XX, YY, ZZ)
```

## 7.4 ガウス確率ベクトルの線形変換

## 7.4.2  2変数ガウス分布の例（標本バージョン）

上と同じ 2変数ガウス分布で生成した標本点

```python
X_data = np.random.multivariate_normal(Means, CovMat, 500000)
plt.plot(X_data[:,0],X_data[:,1],".",markersize=0.5)
```

標本の共分散を計算すると，所与の共分散 0.8 (CovMatの非対角要素)に近い値となる．

```python
Cov(X_data[:,0],X_data[:,1])
```

$$0.8013065611086408$$

```python
CovMat[0,1]
```

$$0.8$$

対角化の変換行列 $S$

```python
l,S = np.linalg.eig(CovMat)
l,S
```

    (array([0.55660189, 2.44339811]), array([[-0.87464248, -0.48476853],
            [ 0.48476853, -0.87464248]]))

この $S$ で実際に $S^T$(CovMat)$S$ は対角化される．

```python
(S.T).dot(CovMat.dot(S))
```

    array([[ 5.56601887e-01, -2.22044605e-16],
           [-1.66533454e-16,  2.44339811e+00]])

$S$ で変換した標本点 $Y = S^T X$

```python
Y_data = ( (S.T).dot(X_data.T) ).T
plt.plot(Y_data[:,0],Y_data[:,1],".",markersize=0.5)
```

分布がまっすぐになった．この標本の共分散を計算すると，無相関を意味する 0 に近い値となる．

```python
Cov(Y_data[:,0], Y_data[:,1])
```

$$0.0021889781040889678$$

## 7.4.2  2変数ガウス分布の例（理論計算バージョン）

```python
def density_plot2(Xgrid, Ygrid, Zgrid, zmax=0.31, zstep=0.1, cmap="binary", labels=['$x_1$','$x_2$','$p(x_1,x_2)$']):
    fig = plt.figure(figsize=(4,3))
    ax3d = fig.add_subplot(111, projection='3d')
    ax3d.view_init(elev=17, azim=-115)
    surf = ax3d.plot_surface(Xgrid, Ygrid, Zgrid, cmap=cmap, linewidth=0)
    zoff = -0.7*zmax
    ax3d.set_zticks(np.arange(0,zmax,zstep))
    ax3d.set_zlim([zoff,zmax])
    ax3d.contourf(Xgrid, Ygrid, Zgrid, cmap=cmap, offset=zoff)
    ax3d.set_xlabel(labels[0],fontsize=12)
    ax3d.set_ylabel(labels[1],fontsize=12)
    ax3d.zaxis.set_rotate_label(False)
    ax3d.set_zlabel(labels[2],fontsize=12,rotation=90)
    plt.tight_layout(pad=1)
```

### 元の2変数ガウス分布

```python
xx = np.linspace(-4,4,100)
yy = np.linspace(-4,4,100)
XX,YY = np.meshgrid(xx,yy)

Means = np.array([0,0])
CovMat_org = np.array([[1,0.8],[0.8,1]])

ZZ_org = Gauss2dv(XX, YY, Means, CovMat_org)
density_plot2(XX, YY, ZZ_org)
```

#### 式(7.31)

```python
eig_values, Smat=np.linalg.eig(CovMat_org)
print('固有値 =\n', eig_values)
print('行列 S =\n', Smat)
```

    固有値 =
     [1.8 0.2]
    行列 S =
     [[ 0.70710678 -0.70710678]
     [ 0.70710678  0.70710678]]

テキストの手計算とは違う並べ順で，固有値が返ってきました．その結果，行列Sの列が入れ替わってます．
固有値のソートが降順になる仕様なんですかね．

試しに，この並べ順のまま進めてみましょう．

#### 式(7.33)

```python
CovMat_new = Smat.T.dot(CovMat_org).dot(Smat)
print(CovMat_new)
```

    [[ 1.80000000e+00 -1.11022302e-16]
     [-1.38777878e-16  2.00000000e-01]]

```python
ZZ_new = Gauss2dv(XX, YY, Means, CovMat_new)
density_plot2(XX, YY, ZZ_new, labels=['$y_1$','$y_2$','$p\'(y_1,y_2)$'])
```

当然ながら，テキストの手計算による共分散行列とは，90度違う結果になります．別に間違ってはいません．対角化後の$y_1$と$y_2$の立場を入れ替えれば同じ結果です．

### テキストに合せた修正

間違いではありませんが，一応，手計算に合せた結果も出しときましょう．列を入れ替えるには・・・

```python
Smat2 = Smat[:,[1,0]]
print(Smat2)
```

    [[-0.70710678  0.70710678]
     [ 0.70710678  0.70710678]]

はい，これでテキストの手計算と同じ並べ順になりました．

#### 式(7.33)

これで，対角化後の共分散行列も揃いまして，

```python
CovMat_new2 = Smat2.T.dot(CovMat_org).dot(Smat2)
print(CovMat_new2)
```

    [[ 2.00000000e-01 -1.38777878e-16]
     [-1.11022302e-16  1.80000000e+00]]

#### 図7.1 (b)

以下がテキストにあるグラフになります．

```python
ZZ_new2 = Gauss2dv(XX, YY, Means, CovMat_new2)
density_plot2(XX, YY, ZZ_new2, labels=['$y_1$','$y_2$','$p\'(y_1,y_2)$'])
plt.savefig('figs/Ch07_gauss_2d_rot.eps', bbox_inches='tight')
```

### 答えが2種類あっていいの？

+ いーんです．
+ 固有ベクトルを並べて行列Sを作るときに，その並べ順については，計算する人の裁量で決めます．
+ ただし，その結果，変換後の独立変数$y_1$, $y_2$の立場が入れ替わります．


```python
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt
```

```javascript
%%javascript
MathJax.Hub.Config({
    TeX: { equationNumbers: { autoNumber: "AMS" } }
});

MathJax.Hub.Queue(
  ["resetEquationNumbers", MathJax.InputJax.TeX],
  ["PreProcess", MathJax.Hub],
  ["Reprocess", MathJax.Hub]
);
```

    <IPython.core.display.Javascript object>

# Agenda

- Introduction
- Strong convergence vs Weak convergence
- Euler-Maruyama and Milstein Method
- Runge-Kutta Methods
- Verlet Algorithm
- Exact Algorithm by Beskos et al.

# Introduction

- We want to simulate the solutions for an SDE (either paths, or distribution properties at specific timepoints).
- We want to understand the convergence properties of the solution.

# Convergence

### Strong convergence

The strong order of convergence is defined to be the largest exponent $\gamma$ such that if we numerically solve and SDE using $M = 1 / \Delta t$ steps of size $\Delta t$, then there exists a constant $K$ such that
$$
\mathrm{E}[\vert \mathbf{x}(t_M) - \mathbf{\hat{x}}(t_M)\vert] \leq K \Delta t^{\gamma}
$$

### Weak convergence

The weak order of convergence is the largest exponent $\alpha$ such that
$$
\left|\mathrm{E}\left[g\left(\mathbf{x}\left(t_{M}\right)\right)\right]-\mathrm{E}\left[g\left(\hat{\mathbf{x}}\left(t_{M}\right)\right)\right]\right| \leq K \Delta t^{\alpha},
$$
for any polynomial $g$. Note that for for the weak convergence we only need to approximate the distribution.

Under the _weak convergence_ criterion, the two solutions would have very similar error.

# Euler-Maruyama and Milstein methods

## ODE derivations

For an ODE of this form

$$
\frac{\mathrm{d} \mathbf{x}(t)}{\mathrm{d} t}=\mathbf{f}(\mathbf{x}(t), t), \quad \mathbf{x}\left(t_{0}\right)=\mathbf{x}_{0},
$$
the integral form reads
$$
\begin{equation}
\mathbf{x}(t)=\mathbf{x}\left(t_{0}\right)+\int_{t_{0}}^{t} \mathbf{f}(\mathbf{x}(\tau), \tau) \mathrm{d} \tau \label{eq:ode-integral}
\end{equation}
$$

If the function $\mathbf{f}$ is differentiable we can write it as the solution to the following ODE:

$$
\frac{\mathrm{df}(\mathbf{x}(t), t)}{\mathrm{d} t}=\frac{\partial}{\partial t} \mathbf{f}(\mathbf{x}(t), t)+\sum_{i} f_{i}(\mathbf{x}(t), t) \frac{\partial}{\partial x_{i}} \mathbf{f}(\mathbf{x}(t), t)
$$

with $\mathbf{f}(\mathbf{x}(t_0), t_0)$ as the initial condition. The integral form of this ODE is:

$$
\begin{array}{l}
\mathbf{f}(\mathbf{x}(t), t)=\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \\
\quad+\int_{t_{0}}^{t}\left[\frac{\partial}{\partial t} \mathbf{f}(\mathbf{x}(\tau), \tau)+\sum_{i} f_{i}(\mathbf{x}(\tau), \tau) \frac{\partial}{\partial x_{i}} \mathbf{f}(\mathbf{x}(\tau), \tau)\right] \mathrm{d} \tau,
\end{array}
$$

which can be neatly written as
$$
\mathbf{f}(\mathbf{x}(t), t)=\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)+\int_{t_{0}}^{t} \mathscr{L} \mathbf{f}(\mathbf{x}(\tau), \tau) \mathrm{d} \tau,
$$

where

$$
\mathscr{L}(\bullet)=\frac{\partial}{\partial t}(\bullet)+\sum_{i} f_{i} \frac{\partial}{\partial x_{i}}(\bullet)
$$

Substituting into $\eqref{eq:ode-integral}$, we obtain
$$
\begin{aligned}
\mathbf{x}(t) &=\mathbf{x}\left(t_{0}\right)+\int_{t_{0}}^{t}\left[\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)+\int_{t_{0}}^{\tau} \mathscr{L} \mathbf{f}\left(\mathbf{x}\left(\tau^{\prime}\right), \tau^{\prime}\right) \mathrm{d} \tau^{\prime}\right] \mathrm{d} \tau \\
&=\mathbf{x}\left(t_{0}\right)+\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)+\int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathscr{L} \mathbf{f}\left(\mathbf{x}\left(\tau^{\prime}\right), \tau^{\prime}\right) \mathrm{d} \tau^{\prime} \mathrm{d} \tau
\end{aligned}
$$

Repearing the process $n$ times we obtain:
$$
\begin{array}{l}
\mathbf{x}(t) & =\mathbf{x}\left(t_{0}\right)+\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)+\frac{1}{2 !} \mathscr{L} \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)^{2} \\
& \quad+\frac{1}{3 !} \mathscr{L}^{2} \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)^{3}+\cdots
\end{array}
$$

$$
\mathbf{r}_{n}(t)=\int_{t_{0}}^{t} \cdots \int_{t_{0}}^{\tau} \mathscr{L}^{n} \mathbf{f}(\mathbf{x}(\tau), \tau) \mathrm{d} \tau^{n+1}
$$

## SDE Derivations

For an SDE of the integral form
$$
\begin{equation}
\mathbf{x}(t)=\mathbf{x}\left(t_{0}\right)+\int_{t_{0}}^{t} \mathbf{f}(\mathbf{x}(\tau), \tau) \mathrm{d} \tau+\int_{t_{0}}^{t} \mathbf{L}(\mathbf{x}(\tau), \tau) \mathrm{d} \beta(\tau),
\label{eq:sde-integral}
\end{equation}
$$

we can analogously compute Ito differentials of $\mathbf{f}(\mathbf{x}(\tau), \tau)$ and $\mathbf{L}(\mathbf{x}(\tau), \tau)$, obtain their integral form, and substitute back into $\eqref{eq:sde-integral}$

$$
\begin{array}{l}
\mathbf{x}(t)=\mathbf{x}\left(t_{0}\right)+\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)+\mathbf{L}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(\beta(t)-\beta\left(t_{0}\right)\right) \\
\quad+\int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathscr{L}_{t} \mathbf{f}(\mathbf{x}(\tau), \tau) \mathrm{d} \tau \mathrm{d} \tau+\sum_{j} \int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathscr{L}_{\beta, j} \mathbf{f}(\mathbf{x}(\tau), \tau) \mathrm{d} \beta_{j}(\tau) \mathrm{d} \tau \\
\quad+\int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathscr{L}_{t} \mathbf{L}(\mathbf{x}(\tau), \tau) \mathrm{d} \tau \mathrm{d} \beta(\tau) \\
\quad+\sum_{j} \int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathscr{L}_{\beta, j} \mathbf{L}(\mathbf{x}(\tau), \tau) \mathrm{d} \beta_{j}(\tau) \mathrm{d} \mathbf{\beta}(\tau)
\end{array}
$$

This is the basis for most of the methods we mention.

$$
\begin{aligned}
\mathscr{L}_{t}(\bullet) &=\frac{\partial(\bullet)}{\partial t}+\sum_{i} \frac{\partial(\bullet)}{\partial x_{i}} f_{i}+\frac{1}{2} \sum_{i, j} \frac{\partial^{2}(\bullet)}{\partial x_{i} \partial x_{j}}\left[\mathbf{L} \mathbf{Q} \mathbf{L}^{\top}\right]_{i j} \\
\mathscr{L}_{\beta, j}(\bullet) &=\sum_{i} \frac{\partial(\bullet)}{\partial x_{i}} L_{i j}, \quad \text { for } j=1,2, \ldots, S
\end{aligned}
$$

## Euler-Maruyama

From the previous derivation, simply take
$$
\mathbf{x}(t)=\mathbf{x}\left(t_{0}\right)+\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)+\mathbf{L}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(\beta(t)-\beta\left(t_{0}\right)\right),
$$

resulting in the following update rule (starting at $\hat{\mathbf{x}}(t_0) \sim p(\mathbf{x}(t_0))$:
1. Draw the Brownian motion increments as
$$\Delta \beta_k \sim \mathrm{N}(\mathbf{0}, \mathbf{Q} \Delta t).$$
2. Compute:
$$ \hat{\mathbf{x}}\left(t_{k+1}\right)=\hat{\mathbf{x}}\left(t_{k}\right)+\mathbf{f}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k}\right) \Delta t+\mathbf{L}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k}\right) \Delta \boldsymbol{\beta}_{k}.
$$

Strong order: 0.5, weak order 1

## Milstein Method

But we ignored a term of order $\mathrm{d}t^{1/2}$. Namely $\mathrm{d} \beta_{j}(\tau) \mathrm{d} \mathbf{\beta}(\tau)$ if of order $\mathrm{d}t^{1/2}$ when integrated as above.

Further expansion of the term
$$
\sum_{j} \int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathscr{L}_{\beta, j} \mathbf{L}(\mathbf{x}(\tau), \tau) \mathrm{d} \beta_{j}(\tau) \mathrm{d} \mathbf{\beta}(\tau)
$$

yields
$$
\begin{aligned}
\mathbf{x}(t)=& \mathbf{x}\left(t_{0}\right)+\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(t-t_{0}\right)+\mathbf{L}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\left(\boldsymbol{\beta}(t)-\boldsymbol{\beta}\left(t_{0}\right)\right) \\
&+\sum_{j} \mathscr{L}_{\beta, j} \mathbf{L}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathrm{d} \beta_{j}(\tau) \mathrm{d} \boldsymbol{\beta}(\tau)+\text { remainder. }
\end{aligned}
$$

This approximation gives us the *Milstein* discretisation scheme

1.Jointly draw a Brownian motion increment and the iterated Ito integral of it
$$
\begin{aligned}
\Delta \boldsymbol{\beta}_{k} &=\boldsymbol{\beta}\left(t_{k+1}\right)-\boldsymbol{\beta}\left(t_{k}\right) \\
\Delta \boldsymbol{\chi}_{v, k} &=\int_{t_{k}}^{t_{k+1}} \int_{t_{k}}^{\tau} \mathrm{d} \beta_{j}(\tau) \mathrm{d} \boldsymbol{\beta}(\tau)
\end{aligned}
$$
2. Compute
$$
\begin{aligned}
\hat{\mathbf{x}}\left(t_{k+1}\right)=& \hat{\mathbf{x}}\left(t_{k}\right)+\mathbf{f}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k}\right) \Delta t+\mathbf{L}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k}\right) \Delta \boldsymbol{\beta}_{k} \\
&+\sum_{j}\left[\sum_{i} \frac{\partial \mathbf{L}}{\partial x_{i}}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k}\right) \mathbf{L}_{i, j}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k}\right)\right] \Delta \chi_{v, k}
\end{aligned}
$$

Strong order: 1, weak order: 1

### A few notes
1. Drawing the iterated stochastic integral jointly with the Brownian motion is hard.
2. We require the partial derivative of the $\mathbf{L}$ with respect to each component of $\mathbf{x}$
3. If the noise is additive , i.e. $\mathbf{L}(\mathbf{x}, t)=\mathbf{L}(t)$, the Milstein method reduces to the Euler-Maruyama method.
4. A special case is when $\mathbf{L}$ is constaint, i.e., $\mathbf{L}(\mathbf{x}, t)=L$, resulting in a discretisation scheme with strong order of convergence of *1.5*
5. In the scalar case, the iterated integral can be evaluated in closed-form.

### Scalar case
In this case we can compute the iterated integral:
$$
\int_{t_{0}}^{t} \int_{t_{0}}^{\tau} \mathrm{d} \beta(\tau) \mathrm{d} \beta(\tau)=\frac{1}{2}\left[\left(\beta(t)-\beta\left(t_{0}\right)\right)^{2}-q\left(t-t_{0}\right)\right]
$$

This follows from the fact that is derived earlier in the book (p.46)
$$
\int_{t_0}^{t} \beta(t) \mathrm{d} \beta(t) = -\frac{1}{2}q(t-t_0) + \frac{1}{2}\big( \beta^2(t) - \beta^2(t_0)\big)
$$

In fact, Ito showed that
$$
\begin{aligned}
n ! \int_{t_{0}}^{t} \int_{t_{0}}^{\tau_{n}} \cdots \int_{t_{0}}^{\tau_{2}} \mathrm{~d} \beta\left(\tau_{1}\right) & \mathrm{d} \beta\left(\tau_{2}\right) \ldots \mathrm{d} \beta\left(\tau_{n}\right) \\
&=q^{n / 2}\left(t-t_{0}\right)^{n / 2} \mathrm{H}_{n}\left(\frac{\beta(t)-\beta\left(t_{0}\right)}{\sqrt{q\left(t-t_{0}\right)}}\right),
\end{aligned}
$$

## Demonstration

Consider the scalar version of the *Black-Scholes* equation:
$$
\mathrm{d}x = \mu x \mathrm{d}t + \sigma x \mathrm{d}\beta,
$$
where $\mu$ is the drift constant, $\mathrm{d}\beta$ is the Brownian motion increment.

From Example 4.7 in the book, we have that
$$
x_t = \exp\bigg( \big( \mu - \frac{1}{2} \sigma^2 \big) t + \sigma \beta(t) \bigg) x_0,
$$
which we can use to compare the errors by different discretisation schemes.

```python
mu, sigma, x0 = 1, 1, 1

# Simulation parameters
T, N = 1, 2**7
dt = 1.0 / N
t = np.arange(dt, 1 + dt, dt)  # Start at dt because Y = X0 at t = 0

loc_1 = 0.4
loc_2 = 0.9

# Generate Brownian paths
num_paths = 10_000
dB = np.sqrt(dt) * np.random.normal(0, 1, (num_paths, N))
B  = np.cumsum(dB, axis=1)

x = x0 * np.exp((sigma * B) + ((mu - 0.5 * sigma**2) * t)[np.newaxis, :])

def plot_paths():
    max_x = 50
    fig, axes = plt.subplots(1, 2, figsize=(12, 6))

    # Left figure
    for i in range(100):
        axes[0].plot(t, x[i], linewidth=0.1, alpha=0.3, color='black')
    axes[0].plot(t, np.mean(x, axis=0), linewidth=2.5, color='blue', alpha=1.0)
    axes[0].axvline(x=loc_1, linestyle='--', color='green', alpha=1.0)
    axes[0].axvline(x=loc_2, linestyle='--', color='orange', alpha=1.0)

    # Right figure
    axes[1].hist(x[:, int(loc_1 * N)], range=(np.min(x), max_x), bins=1000, density=1.0, color='green', alpha=0.4, label=f"x({loc_1})")
    axes[1].hist(x[:, int(loc_2 * N)], range=(np.min(x), max_x), bins=1000, density=1.0, color='orange', alpha=0.4, label=f"x({loc_2})")
    axes[1].legend();
    plt.show()
```

```python
plot_paths()
```

```python
def plot_discretisation(chosen_brownian_path_idx=0, R=5):

    # EM Approximation - small dt
    x_em_small = np.zeros(N)
    x_em_small[0] = x0
    for j in range(1, N):
        x_em_small[j] = x_em_small[j - 1] + mu*x_em_small[j - 1]*dt + sigma*x_em_small[j - 1]*dB[chosen_brownian_path_idx, j-1]

    # EM Approximation - big dt
    coarse_grid = np.arange(0,1+dt,R*dt)
    x_em_big = np.zeros(coarse_grid.shape[0])
    x_em_big[0] = x0
    for j in range(coarse_grid.shape[0] - 1):
        x_em_big[j + 1] = x_em_big[j] + mu*x_em_big[j-1]*R*dt + sigma*x_em_big[j]*np.sum(dB[chosen_brownian_path_idx, int(R*j):int(R*(j+1))])

    x_mil_small = np.zeros(N)
    x_mil_small[0] = x0
    for j in range(1, N):
        x_mil_small[j] = x_mil_small[j - 1] + mu*x_mil_small[j - 1]*dt +  sigma * x_mil_small[j - 1] * dB[chosen_brownian_path_idx, j-1] + 0.5 * sigma**2 * x_mil_small[j - 1] * ((dB[chosen_brownian_path_idx, j-1])**2 - dt)

    fig, axes = plt.subplots(1, 2, figsize=(14, 7))
    axes[0].plot(t, x[chosen_brownian_path_idx, :], label="Exact ($x_t$)", color='blue')
    axes[0].plot(t, x_em_small, label="EM ($\hat{x}_t$): Fine Grid", color='orange', ls='--')
    axes[0].plot(coarse_grid, x_em_big, label="EM ($\hat{x}_t$): Coarse Grid", color='green', ls='--')
    axes[0].set_title('E-M Approximation vs. Exact Simulation');
    axes[0].set_xlabel('t');
    axes[0].legend();

    axes[1].plot(t, x[chosen_brownian_path_idx, :], label="Exact ($x_t$)", color='blue', alpha=0.5)
    axes[1].plot(t, x_em_small, label="EM ($\hat{x}_t$): Fine Grid", color='orange', ls='--')
    axes[1].plot(t, x_mil_small, label="Milstein ($\hat{x}_t$): Fine Grid", color='green', ls='--')
    axes[1].set_title('E-M vs. Milstein')
    axes[1].set_xlabel('t')
    axes[1].legend(loc = 2);

    plt.show()
```

```python
plot_discretisation(chosen_brownian_path_idx=6, R=10)
```

- Professor Des Higham has an [educational paper](https://doi.org/10.1109/MCSE.2004.62) on how to use Black Scholes SDE for option pricing.
- He also wrote a very accessible [paper](https://doi.org/10.1137/S0036144500378302) that discusses the simulation and the convergence of these two methods.

## Weak approximation of Ito-Taylor series

- Sometimes we are interested in the distributions of the paths, rather than paths themselves.
- Given that we only need to preserve the distribution, there is more freedom in forming approximation, mainly including stochastic integrals of higher order.
- As a result arbitrary order of convergence can be obtained in a tractable way (see chapter 14 of the Kloeden and Platen textbook)

# Runge-Kutta methods

1. Motivation
2. Ordinary Runge-Kutta
3. Stochastic Runge-Kutta Methods

## Motivation

- Milstein method requires partial derivatives of $\mathbf{L}(\mathbf{x}(t), t)$ with respect to each of $x_i$'s.
- Other methods also requite partial derivatives of $\mathbf{L}(\mathbf{x}(t), t)$ with respect to each of $x_i$'s as well as $t$.

Runge-Kutta:
- subdivide the integration interval into intermediate steps
- evaluating the function $\mathbf{f}(\cdot, \cdot)$ at a number of points and weighting these evaluations

## Ordinary Runge-Kutta

Consider the following ODE:

$$
\frac{\mathrm{d} \mathbf{x}(t)}{\mathrm{d} t}=\mathbf{f}(\mathbf{x}(t), t), \quad \mathbf{x}\left(t_{0}\right)=\mathbf{x}_{0}
$$

and its Taylor expansion up to $(\Delta t)^2$
$$
\begin{align}
\mathbf{x}\left(t_{0}+\Delta t\right) &\approx \mathbf{x}\left(t_{0}\right)+\mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \Delta t \nonumber\\
 & \quad +\frac{1}{2}\left\{\frac{\partial}{\partial t} \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) +\sum_{i} f_{i}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \frac{\partial}{\partial x_{i}} \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \right\}(\Delta t)^2 .
\label{eq:rk-taylor-expand}
\end{align}
$$

The above has got derivatives, but we are seeking to replace the term involving the derivatives with expressions involving evaluations of $\mathbf{f}(\cdot, \cdot)$ at various points.

We are looking for something of the form:

$$
\begin{aligned}
\mathbf{x}\left(t_{0}+\Delta t\right) & \approx \mathbf{x}\left(t_{0}\right)+a \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \Delta t \\
 & \quad +b \mathbf{f}\left(\mathbf{x}\left(t_{0}\right)+c \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \Delta t, t_{0}+d \Delta t\right) \Delta t,
\end{aligned}
$$

where if we linearise the last term around $\mathbf{f}(\mathbf{x}(t_0), t_0)$ with the increments chosen as follows:

$$
\begin{array}{l}
\mathbf{f}\left(\mathbf{x}\left(t_{0}\right)+c \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \Delta t, t_{0}+d \Delta t\right) \approx \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \\
\quad+c\left(\sum_{i} f_{i}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \frac{\partial}{\partial x_{i}} \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)\right) \Delta t+d \frac{\partial \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)}{\partial t} \Delta t
\end{array}
$$

which gives

$$
\begin{array}{l}
\mathbf{x}\left(t_{0}+\Delta t\right) \approx \mathbf{x}\left(t_{0}\right)+(a+b) \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \Delta t \\
\quad+b\left\{c \sum_{i} f_{i}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right) \frac{\partial}{\partial x_{i}} \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)+d \frac{\partial \mathbf{f}\left(\mathbf{x}\left(t_{0}\right), t_{0}\right)}{\partial t}\right\}(\Delta t)^{2}
\end{array}
$$

and can be pattern matched with $\eqref{eq:rk-taylor-expand}$ to give
$$
a = \frac{1}{2}, \quad b=\frac{1}{2}, \quad c = 1, \quad d = 1
$$

Coming up with these is an art form!

## The general explicit Runge-Kutta scheme

$$
\hat{\mathbf{x}}\left(t_{k+1}\right)=\hat{\mathbf{x}}\left(t_{k}\right)+\sum_{i=1}^{s} \alpha_{i} \mathbf{f}\left(\tilde{\mathbf{x}}_{i}, \tilde{t}_{i}\right) \Delta t,
$$
where $\tilde{t}_{i}=t_{k}+c_{i} \Delta t$ and $\tilde{\mathbf{x}}_{i}=\hat{\mathbf{x}}\left(t_{k}\right)+\sum_{j=1}^{s} A_{i, j} \mathbf{f}\left(\tilde{\mathbf{x}}_{j}, \tilde{t}_{j}\right) \Delta t$.

The corresponding Butcher's tableau is:
$$
\begin{array}{c|cccc}
c_{1} & A_{1,1} & & & \\
c_{2} & A_{2,1} & A_{2,2} & & \\
\vdots & \vdots & & \ddots & \\
c_{s} & A_{s, 1} & A_{s, 2} & \ldots & A_{s, s} \\
\hline & \alpha_{1} & \alpha_{2} & \ldots & \alpha_{s}
\end{array}
$$

### Fourth-order Runge-Kutta

$$
\begin{aligned}
\Delta \mathbf{x}_{k}^{1} &=\mathbf{f}\left(\hat{\mathbf{x}}\left(t_{k}\right), t_{k} + \color{blue}{0}\Delta t\right) \Delta t \\
\Delta \mathbf{x}_{k}^{2} &=\mathbf{f}\left(\hat{\mathbf{x}}\left(t_{k}\right)+\Delta  \color{green}{\frac{1}{2}}\mathbf{x}_{k}^{1}, t_{k}+ \color{blue}{\frac{1}{2}}\Delta t \right) \Delta t \\
\Delta \mathbf{x}_{k}^{3} &=\mathbf{f}\left(\hat{\mathbf{x}}\left(t_{k}\right)+\Delta  \color{green}{\frac{1}{2}} \mathbf{x}_{k}^{2}, t_{k}+ \color{blue}{\frac{1}{2}}\Delta t \right) \Delta t \\
\Delta \mathbf{x}_{k}^{4} &=\mathbf{f}\left(\hat{\mathbf{x}}\left(t_{k}\right)+ \color{green}{1}\Delta \mathbf{x}_{k}^{3}, t_{k}+ \color{blue}{1}\Delta t\right) \Delta t
\end{aligned}
$$

$$
\hat{\mathbf{x}}\left(t_{k+1}\right)=\hat{\mathbf{x}}\left(t_{k}\right)+\color{red}{\frac{1}{6}}\Delta \mathbf{x}_{k}^{1}+ \color{red}{\frac{1}{3}} \Delta \mathbf{x}_{k}^{2}+ \color{red}{\frac{1}{3}} \Delta \mathbf{x}_{k}^{3}+ \color{red}{\frac{1}{6}} \Delta \mathbf{x}_{k}^{4}
$$

The corresponding Butcher's tableau is
$$
\begin{array}{c|cccc}
\color{blue}{0} & & & & \\
\color{blue}{\frac{1}{2}} & \color{green}{\frac{1}{2}} & & & \\
\color{blue}{\frac{1}{2}} & 0 & \color{green}{\frac{1}{2}} & & \\
\color{blue}{1} & 0 & 0 & \color{green}{1} & \\
\hline & \color{red}{\frac{1}{6}} & \color{red}{\frac{1}{3}} & \color{red}{\frac{1}{3}} & \color{red}{\frac{1}{6}}
\end{array}
$$

## Strong stochastic Runge-Kutta

- Using the SDE expansion as above, but avoiding derivatives

$$
\begin{array}{c|c|c|c}
\mathbf{c}^{(0)} & \mathbf{A}^{(0)} & \mathbf{B}^{(0)} & \\
\hline \mathbf{c}^{(1)} & \mathbf{A}^{(1)} & \mathbf{B}^{(1)} & \\
\hline & \alpha^{\top} & {\left[\gamma^{(1)}\right]^{\top}} & {\left[\gamma^{(2)}\right]^{\top}}
\end{array}
$$
On each step $k$, approximate the solution trajectory as follows:
$\hat{\mathbf{x}}\left(t_{k+1}\right)=\hat{\mathbf{x}}\left(t_{k}\right)+\sum_{i} \alpha_{i} \mathbf{f}\left(\tilde{\mathbf{x}}_{i}^{(0)}, t_{k}+c_{i}^{(0)} \Delta t\right) \Delta t$
$$
+\sum_{i, n}\left(\gamma_{i}^{(1)} \Delta \beta_{k}^{(n)}+\gamma_{i}^{(2)} \sqrt{\Delta t}\right) \mathbf{L}_{n}\left(\tilde{\mathbf{x}}_{i}^{(n)}, t_{k}+c_{i}^{(1)} \Delta t\right)
$$
with the supporting values
$$
\begin{aligned}
\tilde{\mathbf{x}}_{i}^{(0)}=\hat{\mathbf{x}}\left(t_{k}\right) &+\sum_{j} A_{i, j}^{(0)} \mathbf{f}\left(\tilde{\mathbf{x}}_{j}^{(0)}, t_{k}+c_{j}^{(0)} \Delta t\right) \Delta t \\
&+\sum_{j, l} B_{i, j}^{(0)} \mathbf{L}_{l}\left(\tilde{\mathbf{x}}_{j}^{(l)}, t_{k}+c_{j}^{(1)} \Delta t\right) \Delta \beta_{k}^{(l)} \\
\tilde{\mathbf{x}}_{i}^{(n)}=\hat{\mathbf{x}}\left(t_{k}\right) &+\sum_{j} A_{i, j}^{(1)} \mathbf{f}\left(\tilde{\mathbf{x}}_{j}^{(0)}, t_{k}+c_{j}^{(0)} \Delta t\right) \Delta t \\
&+\sum_{j, l} B_{i, j}^{(1)} \mathbf{L}_{l}\left(\tilde{\mathbf{x}}_{j}^{(l)}, t_{k}+c_{j}^{(1)} \Delta t\right) \frac{\Delta \chi_{k}^{(l, n)}}{\sqrt{\Delta t}}
\end{aligned}
$$
for $i=1,2, \ldots, s$ and $n=1,2, \ldots, S$.
The increments in Algorithm 8.12 are given by the Itô integrals:
$$
\begin{aligned}
\Delta \beta_{k}^{(i)} &=\int_{t_{k}}^{t_{k+1}} \mathrm{~d} \beta_{i}(\tau) \text { and } \\
\Delta \chi_{k}^{(i, j)} &=\int_{t_{k}}^{t_{k+1}} \int_{t_{k}}^{\tau_{2}} \mathrm{~d} \beta_{i}\left(\tau_{1}\right) \mathrm{d} \beta_{j}\left(\tau_{2}\right),
\end{aligned}
$$
for $i, j=1,2, \ldots, S .$ The increments $\Delta \beta_{k}^{(i)}$ are normally distributed random variables such that jointly $\Delta \boldsymbol{\beta}_{k} \sim \mathrm{N}(\boldsymbol{0}, \mathbf{Q} \Delta t)$. The iterated stochastic Itô integrals $\Delta \chi_{k}^{(i, j)}$ are trickier. For these methods, when $i=j$ the multiple Itô integrals can be rewritten as
$$
\Delta \chi_{k}^{(i, i)}=\frac{1}{2}\left(\left[\Delta \beta_{k}^{(i)}\right]^{2}-Q_{i, i} \Delta t\right)
$$

## Weak Runge-Kutta

- the idea is the same -- if we are interested only in the distribution, the stochastic double integrals terms can be made tractable, thus allowing inclusion of higher-order terms.

# Verlet Algorithm

- Solving 2nd order stochastic SDEs
- Considering central difference

Example:
$$
\ddot{x}=f(x)-\eta b^{2}(x) \dot{x}+b(x) w(t),
$$
which can be rewritten into
$$
\begin{array}{l}
\mathrm{d} x(t)=\mathrm{d} v(t) \mathrm{d} t \\
\mathrm{~d} v(t)=-\eta b^{2}(x(t)) v(t) \mathrm{d} t+f(x(t)) \mathrm{d} t+b(x(t)) \mathrm{d} \beta(t) .
\end{array}
$$

### Leapfrog Verlet Algorithm

1. $\Delta \beta_{k} \sim \mathrm{N}(0, q \Delta t)$
2. $\tilde{x}=\hat{x}\left(t_{k}\right)+\frac{1}{2} v\left(t_{k}\right) \Delta t$
3.
$$
\begin{array}{l}
\hat{v}\left(t_{k+1}\right)=\hat{v}\left(t_{k}\right)-\eta b^{2}(\tilde{x}) \hat{v}\left(t_{k}\right) \Delta t+f(\tilde{x}) \Delta t+b(\tilde{x}) \Delta \beta_{k} \\
\hat{x}\left(t_{k+1}\right)=\tilde{x}+\frac{1}{2} \hat{v}\left(t_{k+1}\right) \Delta t
\end{array}
$$

As many evaluations as E-M but often performs better; time-reversible;

# Exact Algorithm

- Based on rejection sampling, where the sampled distribution is the Brownian motion
- From Girsanov theorem, we can compute the likelihood ratio between the process $\mathbf{x}$ and the Brownian motion, using Radon-Nikodym derivative
$$
\frac{\mathrm{d} P_{\mathbf{x}}}{\mathrm{d} P_{\boldsymbol{\beta}}}=\exp \left(\psi(\boldsymbol{\beta}(T))-\psi(\boldsymbol{\beta}(0))-\frac{1}{2} \int_{0}^{T}\left[\|\mathbf{f}(\boldsymbol{\beta})\|^{2}+(\nabla \cdot \mathbf{f}(\boldsymbol{\beta}))\right] \mathrm{d} t\right),
$$
where $\mathbf{f}(\mathbf{x}) = \nabla \psi (\mathbf{x})$.

By considering biased Brownian motion whose final point has density as

$$
h(\tilde{\boldsymbol{\beta}}(T)) \propto \exp \left(\psi(\tilde{\boldsymbol{\beta}}(T))-\frac{1}{2 T}\|\tilde{\boldsymbol{\beta}}(T)\|^{2}\right)
$$

we have

$$
\frac{\mathrm{d} P_{\mathbf{x}}}{\mathrm{d} P_{\tilde{\beta}}} \propto \exp \left(-\int_{0}^{T} \phi(\boldsymbol{\beta}) \mathrm{d} t\right) \leq 1,
$$

where
$$
\phi(\boldsymbol{\beta})=\frac{1}{2}\left[\|\mathbf{f}(\boldsymbol{\beta})\|^{2}+(\nabla \cdot \mathbf{f}(\boldsymbol{\beta}))\right]-c
$$
and we also assume that $\phi(\boldsymbol{\beta}) < M$

If all of the $x_i$'s lie above corresponding $\phi(\tilde{\beta}(t_i))$, then fill in with Brownian bridges between the points and accept.

### Black-scholes

- the more volatile the returns, the more valuable the option is

# References

- Applied Stochastic Differential Equations by Särkkä and Solin, CUP  2019
- [A blogpost on simulating SDEs in Python by Hautahi Kingi](https://hautahi.com/sde_simulation)


```python
%matplotlib inline
```

```python
import matplotlib.pyplot as plt
import numpy as np
import seaborn as sn

import pycollocation
```

Textbook example: Basic New Keynesian Model

 Households

Suppose that representative household wishes to maximize...

$$\int_{t=0}^{\infty} e^{-\rho t}u(C(t), N(t))L(t)dt$$

...where the flow utility function $u(C(t), N(t))$ is assumed to be additively separable in its two arguments consumption, $C(t)$, and labor supply, $N(t)$ (both measured per member of the household). Note that $L(t)$, the size of the representative household, is assumed to grow at a constant and exogenous rate $n$.

The representative household faces the following intertemporal budget constraint...

$$\dot{B}(t) = i(t)B(t) + W(t)N(t)L(t) - P(t)C(t)L(t)$$

...where $i(t)$ is the nominal interest rate, $B(t)$ is the quantity of bonds held by the representative household, $W(t)$ is the nominal wage paid for labor, and $P(t)$ is the price of consumption goods.

 Solution to the household problem

Form the Hamiltonian...

$$ H(t, B, C, N, \lambda) \equiv e^{-\rho t}u(C(t), N(t))L(t) + \lambda(t)\bigg[i(t)B(t) + W(t)N(t)L(t) - P(t)C(t)L(t)\bigg] $$

...differentiate with respect to control variables $C$ and $N$ and the state variable $B$...

\begin{align}
    \frac{\partial H}{\partial C} \equiv& e^{-\rho t}\frac{\partial u}{\partial C}L(t) - P(t)L(t)\lambda(t) \\
    \frac{\partial H}{\partial N} \equiv& e^{-\rho t}\frac{\partial u}{\partial N}L(t) - W(t)L(t)\lambda(t)\\
    \frac{\partial H}{\partial B} \equiv& i(t)\lambda(t)
\end{align}

...the state and costate equations are...

\begin{align}
    \dot{B}(t) = \frac{\partial H}{\partial \lambda} =& i(t)B(t) + W(t)N(t)L(t) - P(t)C(t)L(t) \\
    \dot{\lambda} = -\frac{\partial H}{\partial B} =& -i(t)\lambda(t)\\
\end{align}

After a bit of algebra (TODO: Add algebra!), we find that the behavior of the representative household is described by the consumption Euler equation...

$$ \frac{\dot{C}}{C} = \frac{1}{R(C)}\bigg[(i - \pi) - \rho\bigg] $$

...where...

$$ R(C) = -\frac{C\frac{\partial^2 u}{\partial C^2}}{\frac{\partial u}{\partial C}}$$

...is the Pratt-Arrow measure of relative risk aversion. Consumption Euler equation says that consumption growth is proportional to the gap between the real interest rate $i - \pi$ and the discount rate $\rho$; and inversely proportional to risk preferences.

a first-order condition describing the trade-off between consumption and labor supply...

$$ \frac{W}{P}\frac{\partial u}{\partial C} = -\frac{\partial u}{\partial N} $$

...and the budget constraint...

$$\dot{B}(t) = i(t)B(t) + W(t)N(t)L(t) - P(t)C(t)L(t).$$

```python

```

 Final goods producers

Competitive final goods firm produces consumption goods using a continuum of intermediate inputs...

$$ Y = \Bigg[\int_0^1 y_j^{\frac{\epsilon - 1}{\epsilon}}dj\Bigg]^{\frac{\epsilon}{\epsilon - 1}} $$

...final goods firm solves static cost minimization problem...

$$\min_{y_j} \int_0^1 p_jy_jdj$$

...subject to feasibility constraint...

$$ Y = \Bigg[\int_0^1 y_j^{\frac{\epsilon - 1}{\epsilon}}dj\Bigg]^{\frac{\epsilon}{\epsilon - 1}}. $$

 Solution to the firms problem

Form the Lagrangian...

$$ \mathcal{L} \equiv \int_0^1 p_jy_jdj + \lambda\Bigg(Y - \Bigg[\int_0^1 y_j^{\frac{\epsilon - 1}{\epsilon}}dj\Bigg]^{\frac{\epsilon}{\epsilon - 1}}\Bigg)$$

First-order conditions are...

$$ p_j - \lambda\frac{y_j^{-\frac{1}{\epsilon}}}{\int_0^1 y_j^{\frac{\epsilon - 1}{\epsilon}}dj}Y = 0\ \forall j$$

After quite a bit of algebra you can derive the firm's demand function for intermediate input $j$ as a function of its own price $p_j$ and the aggregate price level $P$ and output $Y$...

\begin{align}
    %\frac{p_i}{p_j} =& \frac{y_i^{-\frac{1}{\epsilon}}}{y_j^{-\frac{1}{\epsilon}}}
    %\frac{y_i}{y_j} =& \bigg(\frac{p_i}{p_j}\bigg)^{-\epsilon}
    %y_i =& \bigg(\frac{p_i}{p_j}\bigg)^{-\epsilon}y_j
    %p_iy_i =& p_i\bigg(\frac{p_i}{p_j}\bigg)^{-\epsilon}y_j
    %\int_0^1p_iy_idi =& \int_0^1p_i\bigg(\frac{p_i}{p_j}\bigg)^{-\epsilon}y_j di
    %\int_0^1p_iy_idi =& y_j\bigg(\frac{1}{p_j}\bigg)^{-\epsilon}\int_0^1p_i^{1-\epsilon} di
    %PY =& y_j\bigg(\frac{1}{p_j}\bigg)^{-\epsilon}\int_0^1p_i^{1-\epsilon} di
    y_j(p_j) =& \bigg(\frac{p_j}{P}\bigg)^{-\epsilon}Y
\end{align}

where

$$ P = \bigg[\int_0^1p_i^{1-\epsilon}\bigg]^{\frac{1}{1 - \epsilon}}. $$

$$0 = i^* + \phi_X X + \phi_{\pi} \pi - \pi - \rho - g$$
$$ 0 = \rho\pi - \frac{\epsilon - 1}{\theta} \bigg(X^{1 + \psi} - 1\bigg)$$

```python
from scipy import optimize

def nominal_interest_rate(X, pi, i_star, phi_X, phi_pi):
    """Nominal interest rate follows a Taylor rule."""
    return i_star + phi_X * np.log(X) + phi_pi * pi

def output_gap(X, pi, g, i_star, phi_X, phi_pi, rho):
    i = nominal_interest_rate(X, pi, i_star, phi_X, phi_pi)
    return (i - pi - rho - g) * X

def inflation(X, pi, epsilon, psi, rho, theta):
    return rho * pi - ((epsilon - 1) / theta) * (X**(1 + psi) - 1)

def basic_nk_model(t, X, pi, epsilon, g, i_star, phi_X, phi_pi, psi, rho, theta, **params):
    out = [output_gap(X, pi, g, i_star, phi_X, phi_pi, rho),
           inflation(X, pi, epsilon, psi, rho, theta)]
    return out

def terminal_condition(t, X, pi, initial_guess, **params):
    X_star, pi_star = find_equilibrium(initial_guess, params)
    out = [X - X_star, pi - pi_star]
    return out

def _equilibrium_system(vec, params):
    X, pi = vec
    return basic_nk_model(0.0, X, pi, **params)

def find_equilibrium(initial_guess, params):
    result = optimize.root(_equilibrium_system,
                           x0=initial_guess,
                           args=(params))
    if result.success:
        return result.x
    else:
        return result
```

```python
basic_nk_model(0.0, 1.0, 0.0, 0.1, 0.05, 0.1, 1.0, 0.5, 1.5, 0.05, 1.0)
```

    [0.0, 0.0]

To complete the model we need to define some parameter values.

```python
params = {'epsilon': 0.02, 'g': 0.05, 'i_star': 0.05, 'phi_X': 1.0, 'phi_pi': 0.5,
          'psi': 1.5, 'rho': 0.05, 'theta': 1.0, 'initial_guess': np.array([0.5, 0.5])}
```

```python
find_equilibrium(np.array([0.5, 0.5]), params)
```

    array([ 1.0019581 , -0.09608764])

Solving the model with pyCollocation

Defining a `pycollocation.TwoPointBVP` instance

```python
pycollocation.problems.TwoPointBVP?
```

```python
basic_nk_bvp = pycollocation.problems.TwoPointBVP(bcs_lower=None,
                                                  bcs_upper=terminal_condition,
                                                  number_bcs_lower=0,
                                                  number_odes=2,
                                                  params=params,
                                                  rhs=basic_nk_model,
                                                  )
```

### Finding a good initial guess for $k(t)$

Theory tells us that, starting from some initial condition $k_0$, the solution to the Solow model converges monotonically toward its long run equilibrium value $k^*$. Our initial guess for the solution should preserve this property...

```python
def initial_mesh(t, T, num, problem):
    # compute equilibrium values
    X_star, pi_star = find_equilibrium(initial_guess, problem.params)

    ts = np.linspace(t, T, num)
    Xs = X_star - (X_star - problem.params['k0']) * np.exp(-ts)
    pis = pi_star - (pi_star - problem.params['k0']) * np.exp(-ts)

    return ts, Xs, pis

```

### Solving the model

```python
pycollocation.solvers.Solver?
```

 Polynomial basis functions

```python
polynomial_basis = pycollocation.basis_functions.PolynomialBasis()
solver = pycollocation.solvers.Solver(polynomial_basis)

boundary_points = (0, 100)
ts, ks, cs = initial_mesh(*boundary_points, num=1000, problem=standard_ramsey_bvp)

basis_kwargs = {'kind': 'Chebyshev', 'domain': boundary_points, 'degree': 15}
k_poly = polynomial_basis.fit(ts, ks, **basis_kwargs)
c_poly = polynomial_basis.fit(ts, cs, **basis_kwargs)
initial_coefs = np.hstack([k_poly.coef, c_poly.coef])
nodes = polynomial_basis.roots(**basis_kwargs)

solution = solver.solve(basis_kwargs, boundary_points, initial_coefs,
                        nodes, standard_ramsey_bvp)

```

```python
ts, _, _ = initial_mesh(basis_kwargs['domain'], 1000, standard_ramsey_bvp)
k_soln, c_soln = solution.evaluate_solution(ts)
plt.plot(ts, k_soln)
plt.plot(ts, c_soln)
plt.show()
```

```python
k_resids, c_resids = solution.evaluate_residual(ts)
plt.plot(ts, k_resids)
plt.plot(ts, c_resids)

plt.show()
```

```python
k_normalized_resids, c_normalized_resids = solution.normalize_residuals(ts)
plt.plot(ts, np.abs(k_normalized_resids))
plt.plot(ts, np.abs(c_normalized_resids))
plt.yscale('log')
plt.show()
```

 B-spline basis functions

```python
bspline_basis = pycollocation.basis_functions.BSplineBasis()
solver = pycollocation.solvers.Solver(bspline_basis)

boundary_points = (0, 100)
ts, ks, cs = initial_mesh(*boundary_points, num=250, problem=standard_ramsey_bvp)

tck, u = bspline_basis.fit([ks, cs], u=ts, k=5, s=0)
knots, coefs, k = tck
initial_coefs = np.hstack(coefs)

basis_kwargs = {'knots': knots, 'degree': k, 'ext': 2}
nodes = np.linspace(*boundary_points, num=249)

solution = solver.solve(basis_kwargs, boundary_points, initial_coefs,
                        nodes, standard_ramsey_bvp)

```

```python
ts, _, _ = initial_mesh(*boundary_points, num=1000, problem=standard_ramsey_bvp)
k_soln, c_soln = solution.evaluate_solution(ts)
plt.plot(ts, k_soln)
plt.plot(ts, c_soln)
plt.show()
```

```python
k_resids, c_resids = solution.evaluate_residual(ts)
plt.plot(ts, k_resids)
plt.plot(ts, c_resids)

plt.show()
```

```python
k_normalized_resids, c_normalized_resids = solution.normalize_residuals(ts)
plt.plot(ts, np.abs(k_normalized_resids))
plt.plot(ts, np.abs(c_normalized_resids))
plt.yscale('log')
plt.show()
```

 Generic Ramsey-Cass-Koopmans model

Can we refactor the above code so that we can solve a Ramsey-Cass-Koopmans model for arbitrary $f$ and $u$? Yes!

```python
from pycollocation.tests import models
```

Example usage...

```python
def ces_output(k, alpha, sigma, **params):
    gamma = (sigma - 1) / sigma
    if gamma == 0:
        y = k**alpha
    else:
        y = (alpha * k**gamma + (1 - alpha))**(1 / gamma)
    return y

def ces_mpk(k, alpha, sigma, **params):
    y = ces_output(k, alpha, sigma)
    gamma = (sigma - 1) / sigma
    if gamma == 0:
        mpk = alpha * (y / k)
    else:
        mpk = alpha * k**(gamma - 1) * (y / (alpha * k**gamma + (1 - alpha)))
    return mpk

def crra_risk_aversion(t, c, theta, **params):
    return theta

def ces_equilibrium_capital(alpha, delta, g, n, rho, sigma, theta, **params):
    """Steady state value for capital stock (per unit effective labor)."""
    gamma = (sigma - 1) / sigma
    if gamma == 1:
        kss = (alpha / (delta + rho + theta * g))**(1 / (1 - alpha))
    else:
        kss = ((1 / (1 - alpha)) * (((delta + rho + theta * g) / alpha)**(gamma / (1 - gamma)) - alpha))**(-1 / gamma)
    return kss

ces_params = {'g': 0.02, 'theta': 1.0, 'n': 0.02, 'alpha': 0.15, 'delta': 0.04,
              'sigma': 0.5, 'rho': 0.02, 'k0': 1.0}
```

```python
generic_ramsey_bvp = models.RamseyCassKoopmansModel(crra_risk_aversion,
                                                    ces_output,
                                                    ces_equilibrium_capital,
                                                    ces_mpk,
                                                    ces_params)
```

```python
polynomial_basis = pycollocation.basis_functions.PolynomialBasis()
solver = pycollocation.solvers.Solver(polynomial_basis)

basis_kwargs = {'kind': 'Chebyshev', 'domain': [0, 100], 'degree': 15}

ts, ks, cs = initial_mesh(basis_kwargs['domain'], 1000, standard_ramsey_bvp)
k_poly = polynomial_basis.fit(ts, ks, **basis_kwargs)
c_poly = polynomial_basis.fit(ts, cs, **basis_kwargs)
initial_coefs = np.hstack([k_poly.coef, c_poly.coef])

solution = solver.solve(basis_kwargs, initial_coefs, standard_ramsey_bvp)

```

```python
k_soln, c_soln = solution.evaluate_solution(ts)
plt.plot(ts, k_soln)
plt.plot(ts, c_soln)
plt.show()
```

```python
k_normalized_resids, c_normalized_resids = solution.normalize_residuals(ts)
plt.plot(ts, np.abs(k_normalized_resids))
plt.plot(ts, np.abs(c_normalized_resids))
plt.yscale('log')
plt.show()
```

```python

```


```python
%matplotlib inline
%load_ext autoreload
%autoreload 2

import warnings
warnings.filterwarnings("ignore")
```

class: center, middle

# DEMOPS: Roll decay
## Status 2020-02-27
---

### The work has been focused on:
--

### Building a roll decay DB
* SQL database with MDL model tests 2007-2020

--

### Measure Rolldamping
* System identification of roll decay test
* Four methods have been tested

--

### Building roll damping DB
* System identification of roll decay DB

---
name: DB
## Building a roll decay DB
---
template: DB

```python
from sqlalchemy import create_engine
import data
from mdldb.mdl_db import MDLDataBase
from mdldb.tables import Base, Model, LoadingCondition, Run, RolldecayLinear, RolldecayDirect, RolldecayNorwegian
import pandas as pd
from sympy.physics.vector.printing import vpprint, vlatex
from IPython.display import display, Math, Latex, Markdown

engine = create_engine('sqlite:///' + data.mdl_db_path)
db = MDLDataBase(engine=engine)

df = pd.read_sql_table(table_name='run', con=engine, index_col='id')
```

---
template: DB

### MDL DB

```python
s = """The database from MDL currently contains *%i* tests conducted between **%i** and **%i**.
"""  % (len(df),
df['date'].min().year,
df['date'].max().year,)
Markdown(s)
```

The database from MDL currently contains *21243* tests conducted between **2005** and **2017**.

--

```python
exclude = ['smeg','prepare','rodergrundvinkel','rerun','unknown','none standard','teckenkoll']
emphase = ['roll decay']

test_type_s=''
for test_type in df['test_type'].unique():
    if test_type in exclude:
        continue

    if test_type in emphase:
        output = '**%s**' % test_type
    else:
        output = test_type

    test_type_s+=' %s,' % output

s = """<u>Test types:</u>%s
"""  % (test_type_s)
Markdown(s)
```

Test types: seakeeping, GM, reference speed, **roll decay**, zigzag, turning circle, u turn, spiral, bollard pull, captive, crash stop, wave calibration, pitch decay, surge decay, sway decay, softmooring, williamson turn, heave decay, yaw decay,

--

```python
number_of_rolldecay = (df['test_type']=='roll decay').sum()
s = """**%i** of the tests are roll decay tests """ % number_of_rolldecay
Markdown(s)
```

**407** of the tests are roll decay tests

```python
import sympy as sp
from rolldecayestimators.symbols import *
import rolldecayestimators.equations as equations
import rolldecayestimators.direct_estimator as direct_estimator
import rolldecayestimators.direct_linear_estimator as direct_linear_estimator
import rolldecayestimators.direct_estimator_improved as direct_estimator_improved

from latex_helpers import pylatex_extenders
import latex_helpers
import os.path
from rigidbodysimulator.substitute_dynamic_symbols import substitute_dynamic_symbols, find_name, find_derivative_name, lambdify, find_derivatives
import dill
```

---
name: equations
## Measure Rolldamping
---
template: equations

### General equation for roll decay motion:

```python
Math(vlatex(equations.roll_equation_general))
```

$\displaystyle B \dot{\phi} + I \ddot{\phi} + g m \operatorname{GZ}\left(\phi\right) = M_{x}$

--

During a roll decay test external moment is zero:

--

```python
Math(vlatex(equations.roll_decay_equation_general))
```

$\displaystyle B \dot{\phi} + I \ddot{\phi} + g m \operatorname{GZ}\left(\phi\right) = 0$

--

### Linear model

Linearizing the stiffness (ship static stability)

--

```python
latex = vlatex(equations.linear_stiffness_equation)
Math(latex)
```

$\displaystyle \operatorname{GZ}\left(\phi\right) = GM \phi$

--

Linear roll decay equation:

```python
Math(vlatex(equations.roll_decay_linear_equation))
```

$\displaystyle B \dot{\phi} + GM g m \phi + I \ddot{\phi} = 0$

---
template: equations

Damping term $ B $ can be linear:

```python
Math(vlatex(sp.Eq(B,zeta)))
```

$\displaystyle B = \zeta$

--

or quadratic:

```python
Math(vlatex(sp.Eq(B,zeta+d*sp.Abs(phi_dot))))
```

$\displaystyle B = d \left|{\dot{\phi}}\right| + \zeta$

---
name: nonlinearity
## Damping nonlinearity
---
template: nonlinearity
The linear model is sometimes too simple:

---
template: nonlinearity

```python
s="""Adding quadratic damping: $%s$""" % vlatex(sp.Eq(B,zeta+d*sp.Abs(phi_dot)))
Markdown(s)
```

Adding quadratic damping: $B = d \left|{\dot{\phi}}\right| + \zeta$

---
template: nonlinearity

--

* linear model has low score when including large angles

--
* linear model damping $\zeta$ increase for large angles

--
* for quadratic model $d$ increase instead

---
name: varying-stiffness
## Varying stiffness

---
template: varying-stiffness
The linear stiffness assumption is not valid for all ships at large roll angles:

---
template: varying-stiffness

```python
Math(vlatex(equations.linear_stiffness_equation))
```

$\displaystyle \operatorname{GZ}\left(\phi\right) = GM \phi$

---
template: varying-stiffness

### Natural frequency $ \omega_0 $ for each oscillation

---
template: varying-stiffness

### Adding a quadratic term to the stiffness:

```python
Math(vlatex(equations.quadratic_stiffness_equation))
```

$\displaystyle \operatorname{GZ}\left(\phi\right) = GM \phi + dGM \phi \left|{\phi}\right|$

---
name: building-roll-damping-DB
## Building roll damping DB
---

```python
plots = ['zeta','d','mean_damping','omega0','score']
s = ''
for plot in plots:
    code = """
template: building-roll-damping-DB
.right[]
---
""" % plot
    s+=code

Markdown(s)

```

template: building-roll-damping-DB
.right[]
---

template: building-roll-damping-DB
.right[]
---

template: building-roll-damping-DB
.right[]
---

template: building-roll-damping-DB
.right[]
---

template: building-roll-damping-DB
.right[]
---

## Next steps

--

### Regression on roll damping DB
--

### Ikeda method to predict roll damping
--

### Gather more meta data
* Bilge keels
* Inertia
* Hull form (probably takes too much time)

--

### Start writing paper

---
class: center, middle
## End