# Mass Definitions

``hmf``, as of v3.1, provides a simple way of converting between halo mass definitions, which serves two basic purposes:

1. the mass of a halo in one definition can be converted to its appropriate mass under another definition
2. the halo mass function can be converted between mass definitions.

## Introduction

By "mass definition" we mean the way the extent of a halo is defined. In ``hmf``, we support two main kinds of definition, which themselves can contain different models. In brief, ``hmf`` supports Friends-of-Friends (FoF) halos, which are defined via their linking length, $b$, and spherical-overdensity (SO) halos, which are defined by two criteria: an overdensity $\Delta_h$, and an indicator of what that overdensity is with respect to (usually mean background density $\rho_m$, or critical density $\rho_c$). In addition to being able to provide a precise overdensity for a SO halo, ``hmf`` provides a way to use the so-called "virial" overdensity, as defined in Bryan and Norman (1998).

Converting between SO mass definitions is relatively simple: given a halo profile and concentration for the given halo mass, determine the concentration required to make that profile contain the desired density, and then compute the mass of the halo under such a concentration.

There is no clear way to perform such a conversion for FoF halos. Nevertheless, if one assumes that all linked particles are the same mass, and that halos are spherical and singular isothermal spheres (cf. White (2001)), one can approximate an FoF halo by an SO halo of density $9 \rho_m /(2\pi b^3)$. ``hmf`` will make this approximation if a conversion between mass definitions is desired.

## Changing Mass Definitions

Most of the functionality concerning mass definitions is defined in the ``hmf.halos.mass_definitions`` module:

```python
import hmf
from hmf.halos import mass_definitions as md

print("Using hmf version v%s"%hmf.__version__)
```

    Using hmf version v3.0.2

While we're at it, import matplotlib and co:

```python
import matplotlib.pyplot as plt
%matplotlib inline
import inspect
```

Different mass definitions exist inside the ``mass_definitions`` module as ``Components``. All definitions are subclassed from the abstract ``MassDefinition`` class:

```python
[x[1] for x in inspect.getmembers(md, inspect.isclass) if issubclass(x[1], md.MassDefinition)]
```

    [hmf.halos.mass_definitions.FOF,
     hmf.halos.mass_definitions.MassDefinition,
     hmf.halos.mass_definitions.SOCritical,
     hmf.halos.mass_definitions.SOMean,
     hmf.halos.mass_definitions.SOVirial,
     hmf.halos.mass_definitions.SphericalOverdensity]

### The Mass Definition Component

To create an instance of any class, optional ``cosmo`` and ``z`` arguments can be specified. By default, these are the ``Planck15`` cosmology at redshift 0. We'll leave them as default for this example. Let's define two mass definitions, both spherical-overdensity definitions with respect to the mean background density:

```python
mdef_1 = md.SOMean(overdensity=200)
mdef_2 = md.SOMean(overdensity=500)
```

Each mass definition has its own ``model_parameters``, which define the exact overdensity. For both the ``SOMean`` and ``SOCritical`` definitions, the ``overdensity`` can be provided, as above (default is 200). This must be passed as a named argument. For the ``FOF`` definition, the ``linking_length`` can be passed (default 0.2). For the ``SOVirial``, no parameters are available. Available parameters and their defaults can be checked the same way as *any* ``Component`` within ``hmf``:

```python
md.SOMean._defaults
```

    {'overdensity': 200}

The explicit halo density for a given mass definition can be accessed:

```python
mdef_1.halo_density, mdef_1.halo_density/mdef_1.mean_density
```

    (17068502575484.857, 200.0)

### Converting Masses

To convert a mass in one definition to a mass in another, use the following:

```python
mnew, rnew, cnew = mdef_1.change_definition(m=1e12, mdef=mdef_2)
print("Mass in new definition is %.2f x 10^12 Msun / h"%(mnew/1e12))
print("Radius of halo in new definition is %.2f Mpc/h"%rnew)
print("Concentration of halo in new definition is %.2f"%cnew)
```

    Mass in new definition is 0.76 x 10^12 Msun / h
    Radius of halo in new definition is 0.16 Mpc/h
    Concentration of halo in new definition is 4.81

The input mass argument can be a list or array of masses also. To convert between masses, the concentration of the input halos, and their density profile, must be known. By default, an NFW profile is assumed, with a concentration-mass relation from Duffy et al. (2008).

One can alternatively pass a concentration directly:

```python
mnew, rnew, cnew = mdef_1.change_definition(m=1e12, mdef=mdef_2, c = 5.0)
print("Mass in new definition is %.2f x 10^12 Msun / h"%(mnew/1e12))
print("Radius of halo in new definition is %.2f Mpc/h"%rnew)
print("Concentration of halo in new definition is %.2f"%cnew)
```

    Mass in new definition is 0.72 x 10^12 Msun / h
    Radius of halo in new definition is 0.16 Mpc/h
    Concentration of halo in new definition is 3.31

If you have ``halomod`` installed, you can also pass any ``halomod.profiles.Profile`` instance (which itself includes a concentration-mass relation) as the ``profile`` argument.

### Converting Mass Functions

All halo mass function fits are measured using halos found using some halo definition. While some fits explicitly include a parameterization for the spherical overdensity of the halo (eg. ``Tinker08`` and ``Watson``), others do not.

By passing a mass definition to the ``MassFunction`` constructor, ``hmf`` will attempt to convert the mass function defined by the chosen fitting function to the appropriate mass definition, by solving

\begin{equation}
    \int_\infty^{m_{\rm old}} dm\ \  n(m) = \int_\infty^{m_{\rm new}(m_{\rm old})} dm\ \  n'(m)
\end{equation}

for $n'(m)$, resulting in

\begin{equation}
    n'(m) = n(m_{\rm old}(m_{\rm new}))\left(\frac{dm_{\rm new}}{d m_{\rm old}}\right)^{-1}.
\end{equation}

By default, the mass definition is ``None``, which turns off all mass conversion and uses whatever mass definition was employed by the chosen fit. This keeps all existing code running as it was previously. Nevertheless, care should be taken here: many of the different fits have different mass definitions, and should not be compared without some form of mass conversion. To see the mass definition intrinsic to the measurement of each fit, use the following:

```python
from hmf.mass_function.fitting_functions import SMT, Tinker08, Jenkins
```

```python
print(SMT.sim_definition.halo_finder_type, SMT.sim_definition.halo_overdensity)
print(Tinker08.sim_definition.halo_finder_type, Tinker08.sim_definition.halo_overdensity)
print(Jenkins.sim_definition.halo_finder_type, Jenkins.sim_definition.halo_overdensity)

```

    SO vir
    SO *
    FoF 0.2

Here "vir" corresponds to the virial definition of Bryan and Norman (1998), an asterisk indicates that the fit is itself parameterized for SO mass definitions, and 0.2 is the FoF linking length.

Let's convert the mass definition of ``Tinker08``, which has an intrinsic parameterization:

```python
# Default mass function object
mf = hmf.MassFunction()

dndm0 = mf.dndm

# Change the mass definition to 300rho_mean
mf.update(
    mdef_model  = "SOMean",
    mdef_params = {
        "overdensity": 300
    }
)

plt.plot(mf.m, mf.dndm/dndm0, label=r"300 $\rho_m$", lw=3)

# Change the mass definition to 500rho_crit
mf.update(
    mdef_model  = "SOCritical",
    mdef_params = {
        "overdensity": 500
    }
)

plt.plot(mf.m, mf.dndm/dndm0, label=r"500 $\rho_c$", lw=3)

plt.xscale('log')
plt.yscale('log')
plt.xlabel(r"Mass, $M_\odot/h$", fontsize=15)
plt.ylabel(r"Ratio of $dn/dm$ to $200 \rho_m$", fontsize=15)
plt.grid(True)
plt.legend();
```

This did not require any internal conversion. Let's try converting a fit that has no explicit parameterization for overdensity:

```python
# Default mass function object.
mf = hmf.MassFunction(hmf_model = "SMT", Mmax=15)
dndm0 = mf.dndm

# Change the mass definition to 300rho_mean
mf.update(
    mdef_model  = "SOMean",
    mdef_params = {
        "overdensity": 300
    }
)
plt.plot(mf.m, mf.dndm/dndm0, label=r"300 $\rho_m$", lw=3)

# Change the mass definition to 500rho_crit
mf.update(
    mdef_model  = "SOCritical",
    mdef_params = {
        "overdensity": 500
    }
)
plt.plot(mf.m, mf.dndm/dndm0, label=r"500 $\rho_c$", lw=3)

plt.xscale('log')
plt.yscale('log')
plt.xlabel(r"Mass, $M_\odot/h$", fontsize=15)
plt.ylabel(r"Ratio of $dn/dm$ to virial", fontsize=15)
plt.grid(True)
plt.legend();
```

Here, the measured mass function uses "virial" halos, which have an overdensity of ~330$\rho_m$, so that converting to 300$\rho_m$ is a down-conversion of density. Finally, we convert a FoF mass function:

```python
# Default mass function object.
mf = hmf.MassFunction(hmf_model = "Jenkins")
dndm0 = mf.dndm

# Change the mass definition to 300rho_mean
mf.update(
    mdef_model  = "FOF",
    mdef_params = {
        "linking_length": 0.3
    }
)
plt.plot(mf.m, mf.dndm/dndm0, label=r"FoF $b=0.3$", lw=3)

# Change the mass definition to 500rho_crit
mf.update(
    mdef_model  = "SOVirial",
    mdef_params = {}   # NOTE: we need to pass an empty dict here
                       # so that the "linking_length" argument is removed.
)
plt.plot(mf.m, mf.dndm/dndm0, label=r"SO Virial", lw=3)

plt.xscale('log')
plt.yscale('log')
plt.xlabel(r"Mass, $M_\odot/h$", fontsize=15)
plt.ylabel(r"Ratio of $dn/dm$ to FoF $b=0.2$", fontsize=15)
plt.grid(True)
plt.legend();
```


# Tutorial 1: Biological vs. Artificial Neural Networks

**Week 1, Day 3: Multi Layer Perceptrons**

**By Neuromatch Academy**

__Content creators:__ Arash Ash, Surya Ganguli

__Content reviewers:__ Saeed Salehi, Felix Bartsch, Yu-Fang Yang, Antoine De Comite, Melvin Selim Atay, Kelson Shilling-Scrivo

__Content editors:__ Gagana B, Kelson Shilling-Scrivo, Spiros Chavlis

__Production editors:__ Anoop Kulkarni, Kelson Shilling-Scrivo, Spiros Chavlis

---
# Tutorial objectives

In this tutorial, we will explore the Multi-layer Perceptrons (MLPs). MLPs are arguably one of the most tractable models (due to their flexibility) that we can use to study deep learning fundamentals. Here we will learn why they are:

* similar to biological networks
* good at function approximation
* implemented the way they are in PyTorch

```python
# @title Tutorial slides

# @markdown These are the slides for the videos in all tutorials today
from IPython.display import IFrame
IFrame(src=f"https://mfr.ca-1.osf.io/render?url=https://osf.io/4ye56/?direct%26mode=render%26action=download%26mode=render", width=854, height=480)
```

---
# Setup

This is a GPU free notebook!

```python
# @title Install dependencies
!pip install git+https://github.com/NeuromatchAcademy/evaltools --quiet
from evaltools.airtable import AirtableForm

# generate airtable form
atform = AirtableForm('appn7VdPRseSoMXEG','W1D3_T1','https://portal.neuromatchacademy.org/api/redirect/to/9c55f6cb-cdf9-4429-ac1c-ec44fe64c303')
```

      Building wheel for evaltools (setup.py) ... done

```python
# Imports
import random

import torch
import numpy as np
import matplotlib.pyplot as plt

import torch.nn as nn
import torch.optim as optim
from tqdm.auto import tqdm
from IPython.display import display
from torch.utils.data import DataLoader, TensorDataset
```

```python
# @title Figure settings
import ipywidgets as widgets       # interactive display
%config InlineBackend.figure_format = 'retina'
plt.style.use("https://raw.githubusercontent.com/NeuromatchAcademy/content-creation/main/nma.mplstyle")
```

```python
# @title Plotting functions

def imshow(img):
  img = img / 2 + 0.5     # unnormalize
  npimg = img.numpy()
  plt.imshow(np.transpose(npimg, (1, 2, 0)))
  plt.axis(False)
  plt.show()

def plot_function_approximation(x, relu_acts, y_hat):
  fig, axes = plt.subplots(2, 1)

  # Plot ReLU Activations
  axes[0].plot(x, relu_acts.T);
  axes[0].set(xlabel='x',
              ylabel='Activation',
              title='ReLU Activations - Basis Functions')
  labels = [f"ReLU {i + 1}" for i in range(relu_acts.shape[0])]
  axes[0].legend(labels, ncol = 2)

  # Plot function approximation
  axes[1].plot(x, torch.sin(x), label='truth')
  axes[1].plot(x, y_hat, label='estimated')
  axes[1].legend()
  axes[1].set(xlabel='x',
              ylabel='y(x)',
              title='Function Approximation')

  plt.tight_layout()
  plt.show()
```

```python
# @title Set random seed

# @markdown Executing `set_seed(seed=seed)` you are setting the seed

# for DL its critical to set the random seed so that students can have a
# baseline to compare their results to expected results.
# Read more here: https://pytorch.org/docs/stable/notes/randomness.html

# Call `set_seed` function in the exercises to ensure reproducibility.
import random
import torch

def set_seed(seed=None, seed_torch=True):
  if seed is None:
    seed = np.random.choice(2 ** 32)
  random.seed(seed)
  np.random.seed(seed)
  if seed_torch:
    torch.manual_seed(seed)
    torch.cuda.manual_seed_all(seed)
    torch.cuda.manual_seed(seed)
    torch.backends.cudnn.benchmark = False
    torch.backends.cudnn.deterministic = True

  print(f'Random seed {seed} has been set.')

# In case that `DataLoader` is used
def seed_worker(worker_id):
  worker_seed = torch.initial_seed() % 2**32
  np.random.seed(worker_seed)
  random.seed(worker_seed)
```

```python
# @title Set device (GPU or CPU). Execute `set_device()`
# especially if torch modules used.

# inform the user if the notebook uses GPU or CPU.

def set_device():
  device = "cuda" if torch.cuda.is_available() else "cpu"
  if device != "cuda":
    print("GPU is not enabled in this notebook. \n"
          "If you want to enable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `GPU` from the dropdown menu")
  else:
    print("GPU is enabled in this notebook. \n"
          "If you want to disable it, in the menu under `Runtime` -> \n"
          "`Hardware accelerator.` and select `None` from the dropdown menu")

  return device
```

```python
SEED = 2021
set_seed(seed=SEED)
DEVICE = set_device()
```

    Random seed 2021 has been set.
    GPU is not enabled in this notebook.
    If you want to enable it, in the menu under `Runtime` ->
    `Hardware accelerator.` and select `GPU` from the dropdown menu

---
# Section 0: Introduction to MLPs

```python
# @title Video 0: Introduction
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1E3411r7TL", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"Gh0KYl7ViAc", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

#add event to airtable
atform.add_event('Video 0: Introduction')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

---
# Section 1: The Need for MLPs

*Time estimate: ~35 mins*

```python
# @title Video 1: Universal Approximation Theorem
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1SP4y147Uv", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"tg8HHKo1aH4", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 1: Universal Approximation Theorem')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

## Coding Exercise 1: Function approximation with ReLU
We learned that one hidden layer MLPs are enough to approximate any smooth function! Now let's manually fit a sine function using ReLU activation.

We will approximate the sine function using a linear combination (a weighted sum) of ReLUs with slope 1. We need to determine the bias terms (which determines where the ReLU inflection point from 0 to linear occurs) and how to weight each ReLU. The idea is to set the weights iteratively so that the slope changes in the new sample's direction.

First, we generate our "training data" from a sine function using `torch.sine` function.

```python
>>> import torch
>>> torch.manual_seed(2021)
<torch._C.Generator object at 0x7f8734c83830>
>>> a = torch.randn(5)
>>> print(a)
tensor([ 2.2871,  0.6413, -0.8615, -0.3649, -0.6931])
>>> torch.sin(a)
tensor([ 0.7542,  0.5983, -0.7588, -0.3569, -0.6389])
```

These are the points we will use to learn how to approximate the function. We have 10 training data points so we will have 9 ReLUs (we don't need a ReLU for the last data point as we don't have anything to the right of it to model).

We first need to figure out the bias term for each ReLU and compute the activation of each ReLU where:

\begin{equation}
y(x) = \text{max}(0,x+b)
\end{equation}

We then need to figure out the correct weights on each ReLU so the linear combination approximates the desired function.

```python
def approximate_function(x_train, y_train):

  # Number of relus
  n_relus = x_train.shape[0] - 1

  # x axis points (more than x train)
  x = torch.linspace(torch.min(x_train), torch.max(x_train), 1000)

  ## COMPUTE RELU ACTIVATIONS

  # First determine what bias terms should be for each of `n_relus` ReLUs
  b = - x_train

  # Compute ReLU activations for each point along the x axis (x)
  relu_acts = torch.zeros((n_relus, x.shape[0]))

  for i_relu in range(n_relus):
    relu_acts[i_relu, :] = torch.relu(x + b[i_relu])

  ## COMBINE RELU ACTIVATIONS

  # Set up weights for weighted sum of ReLUs
  combination_weights = torch.zeros((n_relus, ))

  # Figure out weights on each ReLU
  prev_slope = 0
  for i in range(n_relus):
    delta_x = x_train[i+1] - x_train[i]
    slope = (y_train[i+1] - y_train[i]) / delta_x
    combination_weights[i] = slope - prev_slope
    prev_slope = slope

  # Get output of weighted sum of ReLU activations for every point along x axis
  y_hat = combination_weights @ relu_acts

  return y_hat, relu_acts, x

# add event to airtable
atform.add_event('Coding Exercise 1: Function approximation with ReLU')

# Make training data from sine function
N_train = 10
x_train = torch.linspace(0, 2*np.pi, N_train).view(-1, 1)
y_train = torch.sin(x_train)

## Uncomment the lines below to test your function approximation
y_hat, relu_acts, x = approximate_function(x_train, y_train)
plot_function_approximation(x, relu_acts, y_hat)
```

As you see in the top panel, we obtain 10 shifted ReLUs with the same slope. These are the basis functions that MLP uses to span the functional space, i.e., MLP finds a linear combination of these ReLUs.

---
# Section 2: MLPs in Pytorch

*Time estimate: ~1hr and 20 mins*

```python
# @title Video 2: Building MLPs in PyTorch
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1zh411z7LY", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"XtwLnaYJ7uc", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 2: Building MLPs in PyTorch')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

In the previous segment, we implemented a function to approximate any smooth function using MLPs. We saw that using Lipschitz continuity; we can prove that our approximation is mathematically correct. MLPs are fascinating, but before we get into the details on designing them, let's familiarize ourselves with some basic terminology of MLPs- layer, neuron, depth, width, weight, bias, and activation function. Armed with these ideas, we can now design an MLP given its input, hidden layers, and output size.

## Coding Exercise 2: Implement a general-purpose MLP in Pytorch
The objective is to design an MLP with these properties:
* works with any input (1D, 2D, etc.)
* construct any number of given hidden layers using `nn.Sequential()` and `add_module()` function
* use the same given activation function (i.e., [Leaky ReLU](https://pytorch.org/docs/stable/generated/torch.nn.LeakyReLU.html)) in all hidden layers.

**Leaky ReLU** is described by the following mathematical formula:

\begin{equation}
\text{LeakyReLU}(x) = \text{max}(0,x) + \text{negative_slope} \cdot \text{min}(0, x) =
\left\{
  \begin{array}{ll}
  x & ,\; \text{if} \; x \ge 0 \\
  \text{negative_slope} \cdot x & ,\;  \text{otherwise}
  \end{array}
\right.
\end{equation}

```python
class Net(nn.Module):
  def __init__(self, actv, input_feature_num, hidden_unit_nums, output_feature_num):
    super(Net, self).__init__()
    self.input_feature_num = input_feature_num # save the input size for reshapinng later
    self.mlp = nn.Sequential() # Initialize layers of MLP

    in_num = input_feature_num # initialize the temporary input feature to each layer
    for i in range(len(hidden_unit_nums)): # Loop over layers and create each one

      out_num = hidden_unit_nums[i] # assign the current layer hidden unit from list
      layer = nn.Linear(in_num, out_num) # use nn.Linear to define the layer
      in_num = out_num # assign next layer input using current layer output
      self.mlp.add_module('Linear_%d'%i, layer) # append layer to the model with a name

      actv_layer = eval('nn.%s'%actv) # Assign activation function (eval allows us to instantiate object from string)
      self.mlp.add_module('Activation_%d'%i, actv_layer) # append activation to the model with a name

    out_layer = nn.Linear(in_num, output_feature_num) # Create final layer
    self.mlp.add_module('Output_Linear', out_layer) # append the final layer

  def forward(self, x):
    # reshape inputs to (batch_size, input_feature_num)
    # just in case the input vector is not 2D, like an image!
    x = x.view(-1, self.input_feature_num)

    logits = self.mlp(x) # forward pass of MLP
    return logits

# add event to airtable
atform.add_event('Coding Exercise 2: Implement a general-purpose MLP in Pytorch')

input = torch.zeros((100, 2))
## Uncomment below to create network and test it on input
net = Net(actv='LeakyReLU(0.1)', input_feature_num=2, hidden_unit_nums=[100, 10, 5], output_feature_num=1).to(DEVICE)
y = net(input.to(DEVICE))
print(f'The output shape is {y.shape} for an input of shape {input.shape}')
```

    The output shape is torch.Size([100, 1]) for an input of shape torch.Size([100, 2])

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_b0675c60.py)

## Section 2.1: Classification with MLPs

```python
# @title Video 3: Cross Entropy
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1Ag41177mB", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"N8pVCbTlves", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 3: Cross Entropy')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

The main loss function we could use out of the box for multi-class classification for `N` samples and `C` number of classes is:
* CrossEntropyLoss:
This criterion expects a batch of predictions `x` with shape `(N, C)` and class index in the range $[0, C-1]$ as the target (label) for each `N` samples, hence a batch of `labels` with shape `(N, )`. There are other optional parameters like class weights and class ignores. Feel free to check the PyTorch documentation [here](https://pytorch.org/docs/stable/generated/torch.nn.CrossEntropyLoss.html) for more detail. Additionally, [here](https://sparrow.dev/cross-entropy-loss-in-pytorch/) you can learn where is appropriate to use the CrossEntropyLoss.

To get CrossEntropyLoss of a sample $i$, we could first calculate $-\log(\text{softmax(x}))$ and then take the element corresponding to $\text { labels }_i$ as the loss. However, due to numerical stability, we implement this more stable equivalent form,

\begin{equation}
\operatorname{loss}(x_i, \text { labels }_i)=-\log \left(\frac{\exp (x[\text { labels }_i])}{\sum_{j} \exp (x[j])}\right)=-x_i[\text { labels }_i]+\log \left(\sum_{j=1}^C \exp (x_i[j])\right)
\end{equation}

### Coding Exercise 2.1: Implement Batch Cross Entropy Loss

To recap, since we will be doing batch learning, we'd like a loss function that given:
* a batch of predictions `x` with shape `(N, C)`
* a batch of `labels` with shape `(N, )` that ranges from `0` to `C-1`

returns the average loss $L$ calculated according to:

\begin{align}
loss(x_i, \text { labels }_i) &= -x_i[\text { labels }_i]+\log \left(\sum_{j=1}^C \exp (x_i[j])\right) \\
L &= \frac{1}{N} \sum_{i=1}^{N}{loss(x_i, \text { labels }_i)}
\end{align}

Steps:

1.   Use indexing operation to get predictions of class corresponding to the labels (i.e., $x_i[\text { labels }_i]$)
2.   Compute $loss(x_i, \text { labels }_i)$ vector (`losses`) using `torch.log()` and `torch.exp()` without Loops!
3. Return the average of the loss vector

```python
def cross_entropy_loss(x, labels):

  # x is the model predictions we'd like to evaluate using lables
  x_of_labels = torch.zeros(len(labels))

  # 1. prediction for each class corresponding to the label
  for i, label in enumerate(labels):
    x_of_labels[i] = x[i, label]
  # 2. loss vector for the batch
  losses = - x_of_labels + torch.log(torch.sum(torch.exp(x), axis=1))
  # 3. Return the average of the loss vector
  avg_loss = losses.mean()

  return avg_loss

# add event to airtable
atform.add_event('Coding Exercise 2.1: Implement Batch Cross Entropy Loss')

labels = torch.tensor([0, 1])
x = torch.tensor([[10.0, 1.0, -1.0, -20.0],  # correctly classified
                  [10.0, 10.0, 2.0, -10.0]])  # Not correctly classified
CE = nn.CrossEntropyLoss()
pytorch_loss = CE(x, labels).item()

## Uncomment below to test your function
our_loss = cross_entropy_loss(x, labels).item()
print(f'Our CE loss: {our_loss:0.8f}, Pytorch CE loss: {pytorch_loss:0.8f}')
print(f'Difference: {np.abs(our_loss - pytorch_loss):0.8f}')
```

    Our CE loss: 0.34672737, Pytorch CE loss: 0.34672749
    Difference: 0.00000012

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_f3376d8f.py)

## Section 2.2: Spiral classification dataset
Before we could start optimizing these loss functions, we need a dataset!

Let's turn this fancy-looking equation into a classification dataset

\begin{equation}
\begin{array}{c}
X_{k}(t)=t\left(\begin{array}{c}
\sin \left[\frac{2 \pi}{K}\left(2 t+k-1\right)\right]+\mathcal{N}\left(0, \sigma\right) \\
\cos \left[\frac{2 \pi}{K}\left(2 t+k-1\right)\right]+\mathcal{N}\left(0, \sigma\right)
\end{array}\right)
\end{array}, \quad 0 \leq t \leq 1, \quad k=1, \ldots, K
\end{equation}

```python
def create_spiral_dataset(K, sigma, N):

  # Initialize t, X, y
  t = torch.linspace(0, 1, N)
  X = torch.zeros(K*N, 2)
  y = torch.zeros(K*N)

  # Create data
  for k in range(K):
    X[k*N:(k+1)*N, 0] = t*(torch.sin(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))
    X[k*N:(k+1)*N, 1] = t*(torch.cos(2*np.pi/K*(2*t+k)) + sigma*torch.randn(N))
    y[k*N:(k+1)*N] = k

  return X, y

# Set parameters
K = 4
sigma = 0.16
N = 1000

set_seed(seed=SEED)
X, y = create_spiral_dataset(K, sigma, N)
plt.scatter(X[:, 0], X[:, 1], c = y)
plt.show()
```

## Section 2.3: Training and Evaluation

```python
# @title Video 4: Training and Evaluating an MLP
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1QV411p7mF", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"DfXZhRfBEqQ", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 4: Training and Evaluating an MLP')

display(out)
```

    Tab(children=(Output(), Output()), _titles={'0': 'Youtube', '1': 'Bilibili'})

### Coding Exercise 2.3: Implement it for a classfication task
Now that we have the Spiral dataset and a loss function, it's your turn to implement a simple train/test split for training and validation.

Steps to follow:
  * Dataset shuffle
  * Train/Test split (20% for test)
  * Dataloader definition
  * Training and Evaluation

```python
def shuffle_and_split_data(X, y, seed):
  # set seed for reproducibility
  torch.manual_seed(seed)
  # Number of samples
  N = X.shape[0]

  # Shuffle data
  shuffled_indices = torch.randperm(N)   # get indices to shuffle data, could use torch.randperm
  X = X[shuffled_indices]
  y = y[shuffled_indices]

  # Split data into train/test
  test_size = int(N * 0.2)    # assign test datset size using 20% of samples
  X_test = X[:test_size]
  y_test = y[:test_size]
  X_train = X[test_size:]
  y_train = y[test_size:]

  return X_test, y_test, X_train, y_train

# add event to airtable
atform.add_event('Coding Exercise 2.3: Implement for a classfication task')

## Uncomment below to test your function
X_test, y_test, X_train, y_train = shuffle_and_split_data(X, y, seed=SEED)
plt.scatter(X_test[:, 0], X_test[:, 1], c=y_test)
plt.title('Test data')
plt.show()
```

And we need to make a Pytorch data loader out of it. Data loading in PyTorch can be separated in 2 parts:
* Data must be wrapped on a Dataset parent class where the methods __getitem__ and __len__ must be overrided. Note that at this point the data is not loaded on memory. PyTorch will only load what is needed to the memory. Here `TensorDataset` does this for us directly.
* Use a Dataloader that will actually read the data in batches and put into memory. Also, the option of `num_workers > 0` allows multithreading, which prepares multiple batches in the queue to speed things up.

```python
g_seed = torch.Generator()
g_seed.manual_seed(SEED)

batch_size = 128
test_data = TensorDataset(X_test, y_test)
test_loader = DataLoader(test_data, batch_size=batch_size,
                         shuffle=False, num_workers=2,
                         worker_init_fn=seed_worker,
                         generator=g_seed)

train_data = TensorDataset(X_train, y_train)
train_loader = DataLoader(train_data, batch_size=batch_size, drop_last=True,
                        shuffle=True, num_workers=2,
                         worker_init_fn=seed_worker,
                         generator=g_seed)
```

Let's write a general-purpose training and evaluation code and keep it in our pocket for next tutorial as well. So make sure you review it to see what it does.

Note that `model.train()` tells your model that you are training the model. So layers like dropout, batchnorm etc. which behave different on the train and test procedures know what is going on and hence can behave accordingly. And to turn off training mode we set `model.eval()`

```python
def train_test_classification(net, criterion, optimizer, train_loader,
                              test_loader, num_epochs=1, verbose=True,
                              training_plot=False, device='cpu'):

  net.train()

  training_losses = []

  for epoch in tqdm(range(num_epochs)):  # loop over the dataset multiple times

    running_loss = 0.0

    for i, data in enumerate(train_loader, 0):

      # get the inputs; data is a list of [inputs, labels]
      inputs, labels = data
      inputs = inputs.to(device).float()
      labels = labels.to(device).long()

      # zero the parameter gradients
      optimizer.zero_grad()

      # forward + backward + optimize
      outputs = net(inputs)

      loss = criterion(outputs, labels)
      loss.backward()
      optimizer.step()

      # print statistics
      if verbose:
        training_losses += [loss.item()]

  net.eval()

  def test(data_loader):
    correct = 0
    total = 0
    for data in data_loader:
      inputs, labels = data
      inputs = inputs.to(device).float()
      labels = labels.to(device).long()

      outputs = net(inputs)
      _, predicted = torch.max(outputs, 1)
      total += labels.size(0)
      correct += (predicted == labels).sum().item()

    acc = 100 * correct / total
    return total, acc

  train_total, train_acc = test(train_loader)
  test_total, test_acc = test(test_loader)

  if verbose:
    print(f"Accuracy on the {train_total} training samples: {train_acc:0.2f}")
    print(f"Accuracy on the {test_total} testing samples: {test_acc:0.2f}")

  if training_plot:
    plt.plot(training_losses)
    plt.xlabel('Batch')
    plt.ylabel('Training loss')
    plt.show()

  return train_acc, test_acc
```

### Think! 2.3.1: What's the point of .eval() and .train()?

Is it necessary to use `net.train()` and `net.eval()` for our MLP model? why?

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q1', text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

    Submission successful!
    Submission successful!

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_51988471.py)

Now let's put everything together and train your first deep-ish model!

```python
set_seed(SEED)
net = Net('ReLU()', X_train.shape[1], [128], K).to(DEVICE)
criterion = nn.CrossEntropyLoss()
optimizer = optim.Adam(net.parameters(), lr=1e-3)
num_epochs = 100

_, _ = train_test_classification(net, criterion, optimizer, train_loader,
                                 test_loader, num_epochs=num_epochs,
                                 training_plot=True, device=DEVICE)
```

And finally, let's visualize the learned decision-map. We know you're probably running out of time, so we won't make you write code now! But make sure you have reviewed it since we'll start with another visualization technique next time.

```python
def sample_grid(M=500, x_max=2.0):
  ii, jj = torch.meshgrid(torch.linspace(-x_max, x_max, M),
                          torch.linspace(-x_max, x_max, M))
  X_all = torch.cat([ii.unsqueeze(-1),
                     jj.unsqueeze(-1)],
                     dim=-1).view(-1, 2)
  return X_all

def plot_decision_map(X_all, y_pred, X_test, y_test,
                      M=500, x_max=2.0, eps=1e-3):
  decision_map = torch.argmax(y_pred, dim=1)

  for i in range(len(X_test)):
    indices = (X_all[:, 0] - X_test[i, 0])**2 + (X_all[:, 1] - X_test[i, 1])**2 < eps
    decision_map[indices] = (K + y_test[i]).long()

  decision_map = decision_map.view(M, M)
  plt.imshow(decision_map, extent=[-x_max, x_max, -x_max, x_max], cmap='jet')
  plt.show()
```

```python
X_all = sample_grid()
y_pred = net(X_all)
plot_decision_map(X_all, y_pred, X_test, y_test)
```

### Think! 2.3.2: Does it generalize well?
Do you think this model is performing well outside its training distribution? Why?

```python
# @title Student Response
from ipywidgets import widgets

text=widgets.Textarea(
   value='Type your answer here and click on `Submit!`',
   placeholder='Type something',
   description='',
   disabled=False
)

button = widgets.Button(description="Submit!")

display(text,button)

def on_button_clicked(b):
   atform.add_answer('q2' , text.value)
   print("Submission successful!")

button.on_click(on_button_clicked)
```

    Textarea(value='Type your answer here and click on `Submit!`', placeholder='Type something')

    Button(description='Submit!', style=ButtonStyle())

    Submission successful!
    Submission successful!

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_d90d9e4b.py)

What would be your suggestions to increase models ability to generalize? Think about it and discuss with your pod.

---
# Summary

In this tutorial we have explored the Multi-leayer Perceptrons (MLPs). More specifically, we have discuss the similarities of artificial and biological neural networks (for more information see the Bonus section), we have learned the Universal Approximation Theorem, and we have implemented MLPs in PyTorch.

```python
# @title Airtable Submission Link
from IPython import display as IPydisplay
IPydisplay.HTML(
   f"""
 <div>
   <a href= "{atform.url()}" target="_blank">
   </a>
   </div>""" )
```

---
# Bonus: Neuron Physiology and Motivation to Deep Learning

This section will motivate one of the most popular nonlinearities in deep learning, the ReLU nonlinearity, by starting from the biophysics of neurons and obtaining the ReLU nonlinearity through a sequence of approximations. We will also show that neuronal biophysics sets a time scale for signal propagation speed through the brain. This time scale implies that neural circuits underlying fast perceptual and motor processing in the brain may not be excessively deep.

This biological motivation for deep learning is not strictly necessary to follow the rest of this course, so this section can be safely skipped and **is a bonus section**.

```python
# @title Video 5: Biological to Artificial Neurons
from ipywidgets import widgets

out2 = widgets.Output()
with out2:
  from IPython.display import IFrame
  class BiliVideo(IFrame):
    def __init__(self, id, page=1, width=400, height=300, **kwargs):
      self.id=id
      src = "https://player.bilibili.com/player.html?bvid={0}&page={1}".format(id, page)
      super(BiliVideo, self).__init__(src, width, height, **kwargs)

  video = BiliVideo(id=f"BV1mf4y157vf", width=854, height=480, fs=1)
  print("Video available at https://www.bilibili.com/video/{0}".format(video.id))
  display(video)

out1 = widgets.Output()
with out1:
  from IPython.display import YouTubeVideo
  video = YouTubeVideo(id=f"ELAbflymSLo", width=854, height=480, fs=1, rel=0)
  print("Video available at https://youtube.com/watch?v=" + video.id)
  display(video)

out = widgets.Tab([out1, out2])
out.set_title(0, 'Youtube')
out.set_title(1, 'Bilibili')

# add event to airtable
atform.add_event('Video 5: Biological to Artificial Neurons')

display(out)
```

## Leaky Integrate-and-fire (LIF)
The basic idea of LIF neuron was proposed in 1907 by Louis Édouard Lapicque, long before we understood the electrophysiology of a neuron (see a translation of [Lapicque's paper](https://pubmed.ncbi.nlm.nih.gov/17968583/) ). More details of the model can be found in the book [**Theoretical neuroscience**](http://www.gatsby.ucl.ac.uk/~dayan/book/) by Peter Dayan and Laurence F. Abbott.

The model dynamics is defined with the following formula,

\begin{equation}
\frac{d V}{d t}=\left\{\begin{array}{cc}
\frac{1}{C}\left(-\frac{V}{R}+I \right) & t>t_{r e s t} \\
0 & \text { otherwise }
\end{array}\right.
\end{equation}

Note that $V$, $C$, and $R$ are the membrane voltage, capacitance, and resitance of the neuron respectively and $-\frac{V}{R}$ is the leakage current. When $I$ is sufficiently strong such that $V$ reaches a certain threshold value $V_{\rm th}$, it momentarily spikes and then $V$ is reset to $V_{\rm reset}< V_{\rm th}$, and voltage stays at $V_{\rm reset}$ for $\tau_{\rm ref}$ ms, mimicking the refractoriness of the neuron during an action potential (note that $V_{\rm reset}$ and $\tau_{\rm ref}$ is assumed to be zero in the lecture):

\begin{eqnarray}
V(t)=V_{\rm reset} \text{  for } t\in(t_{\text{sp}}, t_{\text{sp}} + \tau_{\text{ref}}]
\end{eqnarray}

where $t_{\rm sp}$ is the spike time when $V(t)$ just exceeded $V_{\rm th}$.

Thus, the LIF model captures the facts that a neuron:
- performs spatial and temporal integration of synaptic inputs
- generates a spike when the voltage reaches a certain threshold
- goes refractory during the action potential
- has a leaky membrane

For in-depth content on computational models of neurons, follow the [NMA](https://www.neuromatchacademy.org/) tutorial 1 of *Biological Neuron Models*. Specifically, for NMA-CN 2021 follow this [Tutorial](https://github.com/NeuromatchAcademy/course-content/blob/master/tutorials/W2D3_BiologicalNeuronModels/W2D3_Tutorial1.ipynb).

## Simulating an LIF Neuron

In the cell below is given a function for LIF neuron model with it's arguments described.

Note that we will use Euler's method to make a numerical approximation to a derivative. Hence we will use the following implementation of the model dynamics,

\begin{equation}
V_n=\left\{\begin{array}{cc}
V_{n-1} + \frac{1}{C}\left(-\frac{V}{R}+I \right) \Delta t & t>t_{r e s t} \\
0 & \text { otherwise }
\end{array}\right.
\end{equation}

```python
def run_LIF(I, T=50, dt=0.1, t_ref=10,
            Rm=1, Cm=10, Vth=1, V_spike=0.5):
  """
  Simulate the LIF dynamics with external input current

  Args:
    I          : input current (mA)
    T          : total time to simulate (msec)
    dt         : simulation time step (msec)
    t_ref      : refractory period (msec)
    Rm         : resistance (kOhm)
    Cm         : capacitance (uF)
    Vth        : spike threshold (V)
    V_spike    : spike delta (V)

  Returns:
    time       : time points
    Vm         : membrane potentials
  """

  # Set up array of time steps
  time = torch.arange(0, T+dt, dt)

  # Set up array for tracking Vm
  Vm = torch.zeros(len(time))

  # Iterate over each time step
  t_rest = 0
  for i, t in enumerate(time):

    # If t is after refractory period
    if t > t_rest:
      Vm[i] = Vm[i-1] + 1/Cm*(-Vm[i-1]/Rm + I)  * dt

    # If Vm is over the threshold
    if Vm[i] >= Vth:

      # Increase volatage by change due to spike
      Vm[i] += V_spike

      # Set up new refactory period
      t_rest = t + t_ref

  return time, Vm

sim_time, Vm = run_LIF(1.5)
# Plot the membrane voltage across time
plt.plot(sim_time, Vm)
plt.title('LIF Neuron Output')
plt.ylabel('Membrane Potential (V)')
plt.xlabel('Time (msec)')
plt.show()
```

### Interactive Demo: Neuron's transfer function explorer for different $R_m$ and $t_{ref}$
We know that real neurons communicate by modulating the spike count meaning that more input current causes a neuron to spike more often. Therefore, to find an input-output relationship, it makes sense to characterize their spike count as a function of input current. This is called the neuron's input-output transfer function. Let's plot the neuron's transfer function and see how it changes with respect to the **membrane resistance** and **refractory time**?

```python
# @title

# @markdown Make sure you execute this cell to enable the widget!
my_layout = widgets.Layout()

@widgets.interact(Rm=widgets.FloatSlider(1., min=1, max=100.,
                                         step=0.1, layout=my_layout),
                  t_ref=widgets.FloatSlider(1., min=1, max=100.,
                                            step=0.1, layout=my_layout)
                  )

def plot_IF_curve(Rm, t_ref):
  T = 1000 # total time to simulate (msec)
  dt = 1 # simulation time step (msec)
  Vth = 1 # spike threshold (V)
  Is_max = 2
  Is = torch.linspace(0, Is_max, 10)
  spike_counts = []
  for I in Is:
    _, Vm = run_LIF(I, T=T, dt=dt, Vth=Vth, Rm=Rm, t_ref=t_ref)
    spike_counts += [torch.sum(Vm > Vth)]

  plt.plot(Is, spike_counts)
  plt.title('LIF Neuron: Transfer Function')
  plt.ylabel('Spike count')
  plt.xlabel('I (mA)')
  plt.xlim(0, Is_max)
  plt.ylim(0, 80)
  plt.show()
```

### Think!: Real and Artificial neuron similarities

What happens at infinite membrane resistance ($R_m$) and small refactory time ($t_{ref}$)? Why?

Take 10 mins to discuss the similarity between a real neuron and an artificial one with your pod.

[*Click for solution*](https://github.com/NeuromatchAcademy/course-content-dl/tree/main//tutorials/W1D3_MultiLayerPerceptrons/solutions/W1D3_Tutorial1_Solution_d58d2933.py)


# **Save this file as studentid1_studentid2_lab#.ipynb**
(Your student-id is the number shown on your student card.)

E.g. if you work with 3 people, the notebook should be named:
12301230_3434343_1238938934_lab1.ipynb.

**This will be parsed by a regexp, so please double check your filename.**

Before you turn this problem in, please make sure everything runs correctly. First, **restart the kernel** (in the menubar, select Kernel$\rightarrow$Restart) and then **run all cells** (in the menubar, select Cell$\rightarrow$Run All).

**Make sure you fill in any place that says `YOUR CODE HERE` or "YOUR ANSWER HERE", as well as your names and email adresses below.**

```python
NAME = "Andrii Skliar"
NAME2 = "Gabriele Bani"
EMAIL = "andrii.skliar@student.uva.nl"
EMAIL2 = "bani.gabri@gmail.com"
```

---

# Lab 3: Gaussian Processes and Support Vector Machines

### Machine Learning 1, September 2017

Notes on implementation:

* You should write your code and answers in this IPython Notebook: http://ipython.org/notebook.html. If you have problems, please contact your teaching assistant.
* Please write your answers right below the questions.
* Among the first lines of your notebook should be "%pylab inline". This imports all required modules, and your plots will appear inline.
* Refer to last week's lab notes, i.e. http://docs.scipy.org/doc/, if you are unsure about what function to use. There are different correct ways to implement each problem!
* use the provided test boxes to check if your answers are correct

```python
%pylab inline
plt.rcParams["figure.figsize"] = [20,10]
```

    Populating the interactive namespace from numpy and matplotlib

```python
np.random.seed(1234)
```

## Part 1: Gaussian Processes

For part 1 we will be refer to Bishop sections 6.4.2 and 6.4.3. You may also want to refer to Rasmussen's Gaussian Process text which is available online at http://www.gaussianprocess.org/gpml/chapters/ and especially to the project found at https://www.automaticstatistician.com/index/ by Ghahramani for some intuition in GP.  To understand Gaussian processes, it is highly recommended understand how marginal, partitioned Gaussian distributions can be converted into conditional Gaussian distributions.  This is covered in Bishop 2.3 and summarized in Eqns 2.94-2.98.
$\newcommand{\bt}{\mathbf{t}}$
$\newcommand{\bx}{\mathbf{x}}$
$\newcommand{\by}{\mathbf{y}}$
$\newcommand{\bw}{\mathbf{w}}$
$\newcommand{\ba}{\mathbf{a}}$

### Periodic Data

We will use the same data generating function that we used previously for regression.

```python
def true_mean_function(x):
    return np.cos(2*pi*(x+1))

def add_noise(y, sigma):
    return y + sigma*np.random.randn(len(y))

def generate_t(x, sigma):
    return add_noise(true_mean_function(x), sigma)

```

```python
sigma = 0.2
beta  = 1.0 / pow(sigma, 2)
N_test = 100

x_test = np.linspace(-1, 1, N_test)
mu_test = np.zeros(N_test)
y_test = true_mean_function(x_test)
t_test = add_noise(y_test, sigma)

plt.plot( x_test, y_test, 'b-', lw=2)
plt.plot( x_test, t_test, 'go')
plt.show()
```

### 1. Sampling from the Gaussian process prior (30 points)
We will implement Gaussian process regression using the kernel function in Bishop Eqn. 6.63.

#### 1.1 k_n_m( xn, xm, thetas ) (5 points)
To start, implement function `k_n_m(xn, xm, thetas)` that takes scalars $x_n$ and $x_m$, and a vector of $4$ thetas, and computes the kernel function Bishop Eqn. 6.63 (10 points).  NB: usually the kernel function will take $D$ by $1$ vectors, but since we are using a univariate problem, this makes things easier.

```python
def k_n_m(xn, xm, thetas):
    k = thetas[0] * np.exp(-0.5*thetas[1]*(xn - xm)**2)+thetas[2]+thetas[3]*xn*xm
    return k
```

#### 1.2 computeK( X1, X2, thetas ) (10 points)
Eqn 6.60 is the marginal distribution of mean output of $N$ data vectors: $p(\mathbf{y}) = \mathcal{N}(0, \mathbf{K})$.  Notice that the expected mean function is $0$ at all locations, and that the covariance is a $N$ by $N$ kernel matrix $\mathbf{K}$.  Write a function `computeK(x1, x2, thetas)`
that computes the kernel matrix. Use k_n_m as part of an inner loop (of course, there are more efficient ways of computing the kernel function making better use of vectorization, but that is not necessary) (5 points).

```python
def computeK(x1, x2, thetas):
    K = np.array([[k_n_m(x1_i, x2_i, thetas) for x2_i in x2] for x1_i in x1])
    return K
```

```python
### Test your function
x1 = [0, 1, 2]
x2 = [1, 2, 3, 4]
thetas = [1, 2, 3, 4]
K = computeK(x1, x2, thetas)

assert K.shape == (len(x1), len(x2)), "the shape of K is incorrect"

```

#### 1.3 Plot function samples (15 points)
Now sample mean functions at the x_test locations for the theta values in Bishop Figure 6.5, make a figure with a 2 by 3 subplot and make sure the title reflects the theta values (make sure everything is legible).  In other words, sample $\by_i \sim \mathcal{N}(0, \mathbf{K}_{\theta})$.  Make use of numpy.random.multivariate_normal().  On your plots include the expected value of $\by$ with a dashed line and fill_between 2 standard deviations of the uncertainty due to $\mathbf{K}$ (the diagonal of $\mathbf{K}$ is the variance of the model uncertainty) (15 points).

```python
# function for correcting covariance matrix to make it positive semidefinite
def correct_psd(cov):
    min_eig = np.min(np.real(np.linalg.eigvals(cov)))
    if min_eig < 0:
        cov -= 10*min_eig * np.eye(cov.shape[0])
    return cov

thetas = [[1.00, 4.00, 0.00, 0.00],
         [9.00, 4.00, 0.00, 0.00],
         [1.00, 64.00, 0.00, 0.00],
         [1.00, 0.25, 0.00, 0.00],
         [1.00, 4.00, 10.00, 0.00],
         [1.00, 4.00, 0.00, 5.00]]

colors = ['red', 'blue', 'green', 'cyan', 'black']
_, plt_arr = plt.subplots(2, 3)

for n, theta in enumerate(thetas):

    # There was a warning about K_temp not being positive semidefinite due to the floating number operations
    # Therefore, to use proper covariance matrix, we add a very small number to correct "computation error"
    # Note, that it would still give a correct answer even using K_temp as a covariance matrix
    # However, to avoid warnings, we have found that solution to be quite robust
    K_temp = computeK(x_test, x_test, theta)
    K = correct_psd(K_temp)

    # used for checking if correcting covariance matrix to make it positive semidefinite
    # can actually have any influence on the covariance
    # if the matrices are close elementwise, covariance stays the same after correcting
    # and correction operation is correct
    assert np.sum(~np.isclose(K, K_temp, rtol = 1e-20, atol = 1e-18)) == 0,\
        "The corrected covariance matrix is different from the original one"

    (i, j) = (int(n/3), n%3)
    plt_arr[i,j].set_title("({0})".format(theta))

    mean = np.zeros(x_test.shape[0])
    std = np.sqrt(np.diagonal(K))

    plt_arr[i,j].fill_between(x_test, mean+2*std,\
                         mean-2*std, color='red', alpha=0.1)

    for color in colors:
        gauss = np.random.multivariate_normal(mean, K)
        plt_arr[i,j].plot(x_test, gauss, color = color)
        plt_arr[i,j].plot(x_test, mean, 'r--')
```

### 2. Predictive distribution (35 points)
So far we have sampled mean functions from the prior.  We can draw actual data $\bt$ two ways.  The first way is generatively, by first sampling $\by | \mathbf{K}$, then sampling $\bt | \by, \beta$ (Eqns 6.60 followed by 6.59).  The second way is to integrate over $\by$ (the mean draw) and directly sample $\bt | \mathbf{K}, \beta$ using Eqn 6.61.    This is the generative process for $\bt$.  Note that we have not specified a distribution over inputs $\bx$;  this is because Gaussian processes are conditional models.  Because of this we are free to generate locations $\bx$ when playing around with the GP; obviously a dataset will give us input-output pairs.

Once we have data, we are interested in the predictive distribution (note: the prior is the predictive distribution when there is no data).  Consider the joint distribution for $N+1$ targets, given by Eqn 6.64.  Its covariance matrix is composed of block components $C_N$, $\mathbf{k}$, and $c$.  The covariance matrix $C_N$ for $\bt_N$ is $C_N = \mathbf{K}_N + \beta^{-1}\mathbf{I}_N$.  We have just made explicit the size $N$ of the matrix; $N$ is the number of training points.  The kernel vector $\mathbf{k}$ is a $N$ by $1$ vector of kernel function evaluations between the training input data and the test input vector.  The scalar $c$ is a kernel evaluation at the test input.

#### 2.1 gp_predictive_distribution(...) (10 points)
Write a function `gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None)` that computes  Eqns 6.66 and 6.67, except allow for an arbitrary number of test points (not just one) and now the kernel matrix is for training data.  By having C as an optional parameter, we can avoid computing it more than once (for this problem it is unimportant, but for real problems this is an issue).  The function should compute $\mathbf{C}$, $\mathbf{k}$, and return the mean, variance and $\mathbf{C}$.  Do not forget: the computeK function computes $\mathbf{K}$, not $\mathbf{C}$! (10 points)

```python
def gp_predictive_distribution(x_train, t_train, x_test, theta, beta, C=None):
    K = computeK(x_train, x_train, theta)

    I_train = np.identity(x_train.shape[0])
    I_test = np.identity(x_test.shape[0])

    if C is None:
        C = K + 1/beta * I_train

    k = computeK(x_train, x_test, theta)
    c = (np.diag(computeK(x_test, x_test, theta)) + 1/beta) * I_test

    mean_test = k.T @ np.linalg.inv(C) @ t_train
    var_test = c - k.T @ np.linalg.inv(C) @ k

    return mean_test, var_test, C
```

```python
### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2*train_x
test_N = 3
test_x = np.linspace(-1, 1, test_N)
theta = [1, 2, 3, 4]
beta = 25
test_mean, test_var, C = gp_predictive_distribution(train_x, train_t, test_x, theta, beta, C=None)

assert test_mean.shape == (test_N,), "the shape of mean is incorrect"
assert test_var.shape == (test_N, test_N), "the shape of var is incorrect"
assert C.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, _, C_out = gp_predictive_distribution(train_x, train_t, test_x, theta, beta, C=C_in)

assert np.allclose(C_in, C_out), "C is not reused!"

```

#### 2.2 gp_log_likelihood(...) (10 points)
To learn the hyperparameters, we would need to compute the log-likelihood of the of the training data.  Implicitly, this is conditioned on the value setting for $\mathbf{\theta}$.  Write a function `gp_log_likelihood(x_train, t_train, theta, C=None, invC=None, beta=None)`, where C and invC can be stored and reused. It should return the log-likelihood, `C` and `invC`  (10 points)

```python
def gp_log_likelihood(x_train, t_train, theta, beta, C=None, invC=None):
    N = x_train.shape[0]

    if C is None:
        I_train = np.identity(N)
        K = computeK(x_train, x_train, theta)
        C = K + 1/beta * I_train

    if invC is None:
        invC = np.linalg.inv(C)

    lp = -1/2 * np.log(np.linalg.det(C))\
        - 1/2 * t_train.T @ invC @ t_train\
        - N/2 * np.log(2 * np.pi)

    return lp, C, invC
```

```python
### Test your function
N = 2
train_x = np.linspace(-1, 1, N)
train_t = 2 * train_x
theta = [1, 2, 3, 4]
beta = 25
lp, C, invC = gp_log_likelihood(train_x, train_t, theta, beta, C=None, invC=None)

assert lp < 0, "the log-likelihood should smaller than 0"
assert C.shape == (N, N), "the shape of var is incorrect"
assert invC.shape == (N, N), "the shape of C is incorrect"

C_in = np.array([[0.804, -0.098168436], [-0.098168436, 0.804]])
_, C_out, _ = gp_log_likelihood(train_x, train_t, theta, beta, C=C_in, invC=None)

assert np.allclose(C_in, C_out), "C is not reused!"

invC_in = np.array([[1.26260453, 0.15416407], [0.15416407, 1.26260453]])
_, _, invC_out = gp_log_likelihood(train_x, train_t, theta, beta, C=None, invC=invC_in)

assert np.allclose(invC_in, invC_out), "invC is not reused!"

```

#### 2.3 Plotting (10 points)
Repeat the 6 plots above, but this time conditioned on the training points.  Use the periodic data generator to create 2 training points where x is sampled uniformly between $-1$ and $1$.  For these plots, feel free to use the provided function "gp_plot".  Make sure you put the parameters in the title and this time also the log-likelihood. Try to understand the two types of uncertainty!  If you do not use `gp_plot(...)`, please add a fill between for the model and target noise. (10 points)

```python
def gp_plot( x_test, y_test, mean_test, var_test, x_train, t_train, theta, beta ):
    # x_test:
    # y_test:    the true function at x_test
    # mean_test: predictive mean at x_test
    # var_test:  predictive covariance at x_test
    # t_train:   the training values
    # theta:     the kernel parameters
    # beta:      the precision (known)

    # the reason for the manipulation is to allow plots separating model and data stddevs.
    std_total = np.sqrt(np.diag(var_test))       # includes all uncertainty, model and target noise
    std_model = np.sqrt(std_total**2 - 1.0/beta) # remove data noise to get model uncertainty in stddev
    std_combo = std_model + np.sqrt(1.0/beta)    # add stddev (note: not the same as full)

    plt.plot(x_test, y_test, 'b', lw=3)
    plt.plot(x_test, mean_test, 'k--', lw=2)
    plt.fill_between(x_test, mean_test+2*std_combo,mean_test-2*std_combo, color='k', alpha=0.25)
    plt.fill_between(x_test, mean_test+2*std_model,mean_test-2*std_model, color='r', alpha=0.25)
    plt.plot(x_train, t_train, 'ro', ms=10)

```

```python
N = 2
sigma = 0.2

x_train = np.random.uniform(-1, 1, N)
mu_train = np.zeros(N)
y_train = true_mean_function(x_train)
t_train = add_noise(y_train, sigma)

beta = 25

thetas = [[1.00, 4.00, 0.00, 0.00],
         [9.00, 4.00, 0.00, 0.00],
         [1.00, 64.00, 0.00, 0.00],
         [1.00, 0.25, 0.00, 0.00],
         [1.00, 4.00, 10.00, 0.00],
         [1.00, 4.00, 0.00, 5.00]]

colors = ['red', 'blue', 'green', 'cyan', 'pink']
_, axes = plt.subplots(2, 3)

for n, theta in enumerate(thetas):
    (i, j) = (int(n/3), n%3)
    plt.axes(axes[i,j])

    mean_test, var_test, C = gp_predictive_distribution(x_train, t_train, x_test, theta, beta)
    lp, _, _ = gp_log_likelihood(x_train, t_train, theta, beta, C=C)

    axes[i,j].set_title("(theta = {0}, lp = {1})".format(theta, np.round(lp,4)))

    gp_plot(x_test, y_test, mean_test, var_test, x_train, t_train, theta, beta)

```

#### 2.4 More plotting (5 points)
Repeat the 6 plots above, but this time conditioned a new set of 10 training points. (5 points)

```python
N = 10
sigma = 0.2

x_train = np.random.uniform(-1, 1, N)
mu_train = np.zeros(N)
y_train = true_mean_function(x_train)
t_train = add_noise(y_train, sigma)

beta = 25
thetas = [[1.00, 4.00, 0.00, 0.00],
         [9.00, 4.00, 0.00, 0.00],
         [1.00, 64.00, 0.00, 0.00],
         [1.00, 0.25, 0.00, 0.00],
         [1.00, 4.00, 10.00, 0.00],
         [1.00, 4.00, 0.00, 5.00]]

colors = ['red', 'blue', 'green', 'cyan', 'pink']
_, axes = plt.subplots(2, 3)

for n, theta in enumerate(thetas):

    (i, j) = (int(n/3), n%3)
    plt.axes(axes[i,j])

    mean_test, var_test, C = gp_predictive_distribution(x_train, t_train, x_test, theta, beta)
    lp, _, _ = gp_log_likelihood(x_train, t_train, theta, beta, C=C)

    # Note, that lp stands for log-probability
    axes[i,j].set_title("(theta = {0}, lp = {1})".format(theta, np.round(lp,4)))
    gp_plot(x_test, y_test, mean_test, var_test, x_train, t_train, theta, beta)
```

## Part 2: Support Vector Machines (45 points)
As seen in Part 1: Gaussian Processes, one of the significant limitations of many such algorithms is that the kernel function $k(\bx_n , \bx_m)$ must be evaluated for all possible pairs $\bx_n$ and $\bx_m$ of training points, which can be computationally infeasible during training and can lead to excessive computation times when making predictions for new data points.
In Part 2: Support Vector Machines, we shall look at kernel-based algorithms that have sparse solutions, so that predictions for new inputs depend only on the kernel function evaluated at a subset of the training data points.

### 2.1 Generating a linearly separable dataset (15 points)
a) (5 points) First of all, we are going to create our own 2D toy dataset $X$. The dataset will consists of two i.i.d. subsets $X_1$ and $X_2$, each of the subsets will be sampled from a multivariate Gaussian distribution,

\begin{align}
X_1 \sim &\mathcal{N}(\mu_1, \Sigma_1)\\
&\text{ and }\\
X_2 \sim &\mathcal{N}(\mu_2, \Sigma_2).
\end{align}

In the following, $X_1$ will have $N_1=20$ samples and a mean $\mu_1=(1,1)$. $X_2$ will have $N_2=30$ samples and a mean $\mu_2=(3,3)$.

Plot the two subsets in one figure, choose two colors to indicate which sample belongs to which subset. In addition you should choose, $\Sigma_1$ and $\Sigma_2$ in a way that the two subsets become linearly separable. (Hint: Which form has the covariance matrix for a i.i.d. dataset?)

```python
cov_1 = np.identity(2)/5
cov_2 = np.identity(2)/10

mean_1 = [1,1]
mean_2 = [3,3]

X1 = np.random.multivariate_normal(mean_1, cov_1, 20)
X2 = np.random.multivariate_normal(mean_2, cov_2, 30)

plt.scatter(X1[:, 0], X1[:, 1], color = 'red')
plt.scatter(X2[:, 0], X2[:, 1], color = 'blue')
```

b) (10 points) In the next step we will combine the two datasets X_1, X_2 and generate a vector `t` containing the labels. Write a function `create_X_and_t(X1, X2)` it should return the combined data set X and the corresponding target vector t.

```python
def create_X_and_t(X1, X2):
    t1 = np.array([-1.0 for x1_i in X1])
    t2 = np.array([1.0 for x2_i in X2])

    X = np.concatenate([X1, X2])
    t = np.concatenate([t1, t2])

    return X, t
```

```python
### Test your function
dim = 2
N1_test = 2
N2_test = 3
X1_test = np.arange(4).reshape((N1_test, dim))
X2_test = np.arange(6).reshape((N2_test, dim))
X_test, t_test = create_X_and_t(X1_test, X2_test)

assert X_test.shape == (N1_test + N2_test, dim), "the shape of X is incorrect"
assert t_test.shape == (N1_test + N2_test,), "the shape of t is incorrect"

```

### 2.2 Finding the support vectors (15 points)
Finally we going to use a SVM to obtain the decision boundary for which the margin is maximized. We have to solve the optimization problem

\begin{align}
\arg \min_{\bw, b} \frac{1}{2} \lVert \bw \rVert^2,
\end{align}

subject to the constraints

\begin{align}
t_n(\bw^T \phi(\bx_n) + b) \geq 1, n = 1,...,N.
\end{align}

In order to solve this constrained optimization problem, we introduce Lagrange multipliers $a_n \geq 0$. We obtain the dual
representation of the maximum margin problem in which we maximize

\begin{align}
\sum_{n=1}^N a_n - \frac{1}{2}\sum_{n=1}^N\sum_{m=1}^N a_n a_m t_n t_m k(\bx_n, \bx_m),
\end{align}

with respect to a subject to the constraints

\begin{align}
a_n &\geq 0, n=1,...,N,\\
\sum_{n=1}^N a_n t_n &= 0.
\end{align}

This takes the form of a quadratic programming problem in which we optimize a quadratic function of a subject to a set of inequality constraints.

a) (5 points) In this example we will use a linear kernel $k(\bx, \bx') = \bx^T\bx'$. Write a function `computeK(X)` that computes the kernel matrix $K$ for the 2D dataset $X$.

```python
def computeK(X):
    K = X @ X.T

    return K
```

```python
dim = 2
N_test = 3
X_test = np.arange(6).reshape((N_test, dim))
K_test = computeK(X_test)

assert K_test.shape == (N_test, N_test)

```

Next, we will rewrite the dual representation so that we can make use of computationally efficient vector-matrix multiplication. The objective becomes

\begin{align}
\min_{\ba} \frac{1}{2} \ba^T K' \ba - 1^T\ba,
\end{align}

subject to the constraints

\begin{align}
a_n &\geq 0, n=1,...,N,\\
\bt^T\ba &= 0.
\end{align}

Where
\begin{align}
K'_{nm} = t_n t_m k(\bx_n, \bx_m),
\end{align}
and in the special case of a linear kernel function,
\begin{align}
K'_{nm} = t_n t_m k(\bx_n, \bx_m) = k(t_n \bx_n, t_m \bx_m).
\end{align}

To solve the quadratic programming problem we will use a python module called cvxopt. You first have to install the module in your virtual environment (you have to activate it first), using the following command:

`conda install -c conda-forge cvxopt`

The quadratic programming solver can be called as

`cvxopt.solvers.qp(P, q[, G, h[, A, b[, solver[, initvals]]]])`

This solves the following problem,

\begin{align}
\min_{\bx} \frac{1}{2} \bx^T P \bx + q^T\bx,
\end{align}

subject to the constraints,

\begin{align}
G\bx &\leq h,\\
A\bx &= b.
\end{align}

All we need to do is to map our formulation to the cvxopt interface.

b) (10 points) Write a function `compute_multipliers(X, t)` that solves the quadratic programming problem using the cvxopt module and returns the lagrangian multiplier for every sample in the dataset.

```python
import cvxopt

def compute_multipliers(X, t):
    N = X.shape[0]
    X2 = X * t[np.newaxis].T
    P = cvxopt.matrix(computeK(X2))
    q = cvxopt.matrix(-np.ones((N, 1)))
    G = cvxopt.matrix(-np.identity(N))
    h = cvxopt.matrix(np.zeros((N, 1)))
    A = cvxopt.matrix(t[np.newaxis])
    b = cvxopt.matrix(np.zeros((1, 1)))

    sol = cvxopt.solvers.qp(P, q, G, h, A, b)
    a = np.array(sol['x'])
    return a
```

```python
### Test your function
dim = 2
N_test = 3
X_test = np.arange(6).reshape((N_test, dim))
t_test = np.array([-1., 1., 1.])
a_test = compute_multipliers(X_test, t_test)

assert a_test.shape == (N_test, 1)

```

         pcost       dcost       gap    pres   dres
     0: -7.2895e-01 -1.3626e+00  6e+00  2e+00  2e+00
     1: -3.0230e-01 -6.8816e-01  8e-01  1e-01  1e-01
     2: -2.3865e-01 -3.3686e-01  1e-01  1e-16  5e-15
     3: -2.4973e-01 -2.5198e-01  2e-03  6e-17  8e-16
     4: -2.5000e-01 -2.5002e-01  2e-05  6e-17  8e-16
     5: -2.5000e-01 -2.5000e-01  2e-07  6e-17  5e-16
    Optimal solution found.

### 2.3 Plot support vectors (5 points)
Now that we have obtained the lagrangian multipliers $\ba$, we use them to find our support vectors. Repeat the plot from 2.1, this time use a third color to indicate which samples are the support vectors.

```python
X, t = create_X_and_t(X1, X2)
a = compute_multipliers(X, t).squeeze()
# get indices of support vectors
indices = np.argwhere(~np.isclose(a, np.zeros(a.shape[0]), atol=1e-04))
support_vectors = X[indices].squeeze()
```

         pcost       dcost       gap    pres   dres
     0: -2.6751e+00 -4.5421e+00  1e+02  1e+01  2e+00
     1: -6.7742e-01 -1.5606e+00  7e+00  6e-01  8e-02
     2: -2.6240e-01 -1.0295e+00  8e-01  2e-16  3e-15
     3: -5.3042e-01 -6.6854e-01  1e-01  9e-17  1e-15
     4: -6.1201e-01 -6.6833e-01  6e-02  1e-16  2e-15
     5: -6.5360e-01 -6.6327e-01  1e-02  2e-16  2e-15
     6: -6.6211e-01 -6.6263e-01  5e-04  1e-16  2e-15
     7: -6.6256e-01 -6.6256e-01  5e-06  2e-16  2e-15
     8: -6.6256e-01 -6.6256e-01  5e-08  2e-16  2e-15
    Optimal solution found.

```python
plt.scatter(X1[:, 0], X1[:, 1], color = 'red')
plt.scatter(X2[:, 0], X2[:, 1], color = 'blue')
# green circles distinguish support vectors from all other points
plt.scatter(support_vectors[:, 0], support_vectors[:, 1], s=90, edgecolors='g', facecolors = 'none', lw=4)
```

### 2.4 Plot the decision boundary (10 Points)
The decision boundary is fully specified by a (usually very small) subset of training samples, the support vectors. Make use of

\begin{align}
\bw &= \sum_{n=1}^N a_n t_n \mathbf{\phi}(\bx_n)\\
b &= \frac{1}{N_S}\sum_{n \in S} (t_n - \sum_{m \in S} a_m t_m k(\bx_n, \bx_m)),
\end{align}

where $S$ denotes the set of indices of the support vectors, to calculate the slope and intercept of the decision boundary. Generate a last plot that contains the two subsets, support vectors and decision boundary.

```python
# Note, we are including parameter c for plotting margins
def f(x, w, b, c=0):
    return (w[0] * x + b - c) / -w[1]
```

```python
w = t*a @ X
b = np.sum([t[n] - np.sum([a[m] * t[m] * (X[n] @ X[m].T) for m in indices]) for n in indices]) / indices.shape[0]

x_min = np.min(X[:, 0])
x_max = np.max(X[:, 0])
x_plot = np.linspace(x_min, x_max, 100)
```

```python
plt.scatter(X1[:, 0], X1[:, 1], color = 'red')
plt.scatter(X2[:, 0], X2[:, 1], color = 'blue')
plt.scatter(support_vectors[:, 0], support_vectors[:, 1], s = 90, edgecolors = 'g', facecolors = 'none', lw = 4)

y_plot = f(x_plot, w, b).T
y_plotmax = f(x_plot, w, b, c=1).T
y_plotmin = f(x_plot, w, b, c=-1).T

# plot decision boundary
plt.plot(x_plot, y_plot, 'c')
# plot top margin
plt.plot(x_plot, y_plotmax, 'c--')
# plot lower margin
plt.plot(x_plot, y_plotmin, 'c--')
```