# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Definition

The [Laplace transform](https://en.wikipedia.org/wiki/Laplace_transform) constitutes an integral transform with the complex exponential function $e^{s t}$ as kernel. It maps a complex-valued signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$ into its complex-valued Laplace transform $X(s) \in \mathbb{C}$ with complex-valued independent variable $s \in \mathbb{C}$. The Laplace transform is named after its discoverer [Pierre-Simon Laplace](https://en.wikipedia.org/wiki/Pierre-Simon_Laplace).

Complex exponential functions constitute eigenfunctions of linear time-invariant (LTI) systems. The Laplace transform is therefore of special use in the context of signals and LTI systems. It is introduced in the following.

### Two-Sided Laplace Transform

The two-sided (or bilateral) Laplace transform is defined as

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ denotes the Laplace transform of $x(t)$. A complex signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$ is represented by its complex valued Laplace transform $X(s) \in \mathbb{C}$ with complex independent variable $s \in \mathbb{C}$. The variable $s$ can be interpreted as [complex frequency](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $s = \sigma + j \omega$ with $\sigma, \omega \in \mathbb{R}$.

Whether a Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ exists depends on the complex frequency $s$ and the signal $x(t)$ itself. All values $s$ for which the Laplace transform converges form a [*region of convergence*](https://en.wikipedia.org/wiki/Radius_of_convergence) (ROC). The Laplace transforms of two different signals may differ only with respect to their ROCs. Consequently, the ROC needs to be explicitly given for an unique inversion of the Laplace transform.

### One-Sided Laplace Transform

Causal signals play an important role in the theory of signals and system. For a causal signal $x(t) = 0$ for $t <0$, the relation $x(t) = x(t) \cdot \epsilon(t)$ holds. Introducing this into the definition of the two-sided Laplace transformation yields

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \cdot \epsilon(t) \, e^{- s t} \; dt = \int_{0}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

This motivates the definition of a one-sided Laplace transform

\begin{equation}
X(s) = \int_{0}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

In the literature both the one- and two-sided Laplace transform are termed as Laplace transform. Both give the same result for causal signals.

### Transformation of the Dirac Impulse

The transform $\mathcal{L} \{ \delta(t) \}$ of the [Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse) is derived by introducing $\delta(t)$ into the definition of the two-sided Laplace transform and exploiting the sifting property of the Dirac delta function

\begin{equation}
\mathcal{L} \{ \delta(t) \} = \int_{-\infty}^{\infty} \delta(t) \, e^{- s t} dt = e^{- s t} \Big\vert_{t=0}  =  1
\end{equation}
for $s \in \mathbb{C}$. The ROC covers the entire complex plane.

The transform of the Dirac impulse is equal to one. Hence, all complex frequencies $s$ are present with equal weight. This is an important property in the theory of signals and systems, since the Dirac impulse is used to characterize linear time-invariant (LTI) systems by their [impulse response](../systems_time_domain/impulse_response.ipynb) $h(t) = \mathcal{H} \{ \delta(t) \}$.

**Example - Transformation of the Dirac impulse**

The Laplace transform of the Dirac impulse can be derived in `SymPy` by direct evaluation of above integral

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t = sym.symbols('t', real=True)
s = sym.symbols('s', complex=True)

X = sym.integrate(sym.DiracDelta(t)*sym.exp(-s*t), (t, -sym.oo, sym.oo))
X
```

**Exercise**

* Derive the two-sided Laplace transform of a shifted Dirac impulse $\delta(t - \tau)$ either by manual evaluation of the integral or by modification of above example.

* Derive the one-sided Laplace transform of a shifted Dirac impulse $\delta(t - \tau)$. Hint: Differentiate between the cases $\tau < 0$ and $\tau > 0$.

### Transformation of the Causal Complex Exponential Signal

The transform $X(s) = \mathcal{L} \{ x(t) \}$ of the causal complex exponential signal

\begin{equation}
x(t) = \epsilon(t) \cdot e^{-s_0 t}
\end{equation}

with complex frequency $s_0 \in \mathbb{C}$ is derived by evaluation of the definition of the one-sided Laplace transform

\begin{equation}
\begin{split}
X(s) &= \int_{0}^{\infty} e^{-s_0 t} \cdot e^{-s t} \; dt = \frac{-e^{-(s+s_0)t}}{s+s_0} \, \bigg\vert_{0}^{\infty} = \frac{-1}{s+s_0} \left[ \lim_{t \to \infty} \left( e^{- (s+s_0) t} \right) -1 \right] \\
&= \frac{1}{s + s_0}
\end{split}
\end{equation}

As can be concluded from the limit case above, the Laplace transform converges only for $\Re \{ s + s_0 \} > 0$. Hence, the ROC is given as

\begin{equation}
\Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

Combining above findings, the transformation of the causal complex exponential signal reads

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

**Example - Transformation of exponential signal**

In `SymPy` the [Laplace transform](http://docs.sympy.org/latest/modules/integrals/integrals.html?highlight=laplace#sympy.integrals.transforms.laplace_transform) is defined as one-sided transformation. The respective function of `SymPy` is utilized in the following to calculate the transform of the causal complex exponential signal.

```python
s0 = sym.symbols('s0', complex=True)

X, a, cond = sym.laplace_transform(sym.exp(-s0*t), t, s)
X, a, cond
```

The function returns three results. The first result is the Laplace transform $X(s)$, the second result $a$ is the half-plane of convergence with $\Re \{ s \} > a$ and the third result represents an auxiliary convergence condition.

### Transformation of the Heaviside Signal

The Laplace transform of the [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) is derived from above transform of the causal complex exponential signal $\epsilon(t) \cdot e^{-s_0 t}$ by setting $s_0 = 0$

\begin{equation}
\mathcal{L} \{ \epsilon(t) \} = \frac{1}{s} \qquad \text{for } \Re \{ s \} > 0
\end{equation}

## Convergence

A sufficient but not necessary condition for the existence of the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of a given signal $x(t)$ is

\begin{equation}
\int_{-\infty}^{\infty} | x(t) \cdot e^{- \sigma t} | \; dt < \infty
\end{equation}

where $\sigma = \Re\{ s \}$ denotes the real part of the complex frequency $s = \sigma + j \omega$ of the Laplace transform. Formally the signal $x(t)$ has to be a [locally integrable function](https://en.wikipedia.org/wiki/Locally_integrable_function).

The imaginary part $\omega = \Im \{ s \}$ of the complex frequency $s$ has no effect in terms of convergence. Consequently, the ROC is determined solely by its real part $\sigma = \Re\{ s \}$. For a right-sided signal with $x(t) = 0$ for $t < T_0$, the ROC is given in the form of $\Re \{ s \} > a$ with $a \in \mathbb{R}$. This can be concluded from the decay of the exponential term $e^{- \sigma t}$ for $t \to \infty$, which ensures convergence for a given $a$. However this reasoning holds only for signals with [exponential growth](https://en.wikipedia.org/wiki/Exponential_growth). Please refer to the [Laplace transform of the causal exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal) for an example.

The same reasoning leads to the ROCs for a left-sided and two-sided signal. The resulting ROCs are shown in the following illustration

The gray areas denote the values $s$ for which the Laplace transformation converges. The borders $a$ of these areas (dashed lines) depend on the signal $x(t)$. A more detailed discussion of the ROCs for the Laplace transform can be found in the literature, e.g. [[Girod et al.](index.ipynb#Literature)]

## Representation

The Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ depends on the complex frequency $s \in \mathbb{C}$ and is in general complex valued $X(s) \in \mathbb{C}$. It can be illustrated by plotting its magnitude $|X(s)|$ and phase $\varphi (s)$ or real $\Re \{ X(s) \}$ and imaginary $\Im \{ X(s) \}$ part in the complex $s$-plane. The resulting three-dimensional plots are often not very illustrative.

However, many Laplace transforms of interest in the theory of signals and systems are [rational functions](https://en.wikipedia.org/wiki/Rational_function) in $s$. The polynomials of the numerator and denominator can be represented by their complex [roots](https://en.wikipedia.org/wiki/Zero_of_a_function#Polynomial_roots) and a constant factor. The roots of the numerator are termed as *zeros* while the roots of the denominator are termed as *poles* of $X(s)$. The polynomial and the zero/pole representation of a rational Laplace transform are equivalent

\begin{equation}
X(s) = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n} = K \cdot \frac{\prod_{\mu=0}^{Q} (s - s_{0 \mu})}{\prod_{\nu=0}^{P} (s - s_{\infty \nu})}
\end{equation}

where $M$ and $N$ denote the order of the numerator/denominator polynomial, $s_{0 \mu}$ and $s_{\infty \nu}$ the $\mu$-th zero/$\nu$-th pole of $X(s)$, and $Q = M-1$ and $P = N-1$ the total number of zeros and poles, respectively. For $M=N$ the factor $K = \frac{\beta_M}{\alpha_N}$. If $M > N$ at least one pole is located at $|s| = \infty$.

It is common to illustrate the poles and zeros in a [pole-zero plot](https://en.wikipedia.org/wiki/Pole%E2%80%93zero_plot). Here the locations of the complex poles and zeros, their degrees and the factor $K$ are plotted in the $s$-plane

The locations of the poles and zeros provide insights into the composition of a signal. For instance, the Laplace transform of the [complex exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal) has a pole at $s = - s_0$. A signal which is composed from a superposition of complex exponential signals will have multiple poles whose positions are related to the complex frequencies it is composed of.

**Exercise**

* Derive $X(s)$ together with its ROC from above pole-zero plot.

**Example - Illustration of Laplace transform of the exponential signal**

The Laplace transform of the causal complex exponential function

\begin{equation}
X(s) = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

is illustrated by plotting its magnitude $|X(s)|$ in the $s$-plane $s = \sigma + j \omega$ for $s_0 = 1 + j$

```python
sigma, omega = sym.symbols('sigma omega')

X = 1/(s + s0)
X1 = X.subs({s: sigma+sym.I*omega, s0: 1+sym.I})

sym.plotting.plot3d(abs(X1), (sigma, -2, 2), (omega, -2, 2),
                    xlabel=r'$\Re\{s\}$', ylabel=r'$\Im\{s\}$', title=r'$|X(s)|$');
```

Alternatively, the Laplace transform is illustrated by its pole-zero plot. First the poles

```python
poles = sym.roots(sym.denom(X), s)
poles
```

and zeros are computed

```python
zeros = sym.roots(sym.numer(X), s)
zeros
```

Above dictionaries are composed of the poles and zeros, and their degrees.

In order to illustrate the location of poles and zeros in the $s$-plane, a function is defined for a pole-zero plot using the Python library [`matplotlib`](http://matplotlib.org/)

```python
import matplotlib.pyplot as plt

def pole_zero_plot(poles, zeros, ax=None):

    for pole in poles:
        plt.plot(complex(pole).real, complex(pole).imag, 'rx', markersize=10)
    for zero in zeros:
        plt.plot(complex(zero).real, complex(zero).imag, 'bo', markersize=10)

    if ax is None:
        ax = plt.gca()
    ax.set_aspect(1)

    plt.axis([-2, 2, -2, 2])
    plt.xlabel(r'$\Re \{ s \}$')
    plt.ylabel(r'$\Im \{ s \}$')
    plt.grid()
```

The pole-zero plot is shown for $s_0 = 1 + j$

```python
X2 = X.subs(s0, 1+sym.I)
pole_zero_plot(sym.roots(sym.denom(X2), s), sym.roots(sym.numer(X2), s))
```

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Inductive bias in GCN: a spectral perspective

__author: [Marc Lelarge](https://www.di.ens.fr/~lelarge/), [code](https://github.com/dataflowr/notebooks/blob/master/graphs/GCN_inductivebias_spectral.ipynb), course: [dataflowr](https://dataflowr.github.io/website/)__

Here, we focus on Graph Convolution Networks (GCN) introduced by Kipf and Welling in their paper [Semi-Supervised Classification with Graph Convolutional Networks](https://arxiv.org/abs/1609.02907).
The GCN layer is one of the simplest Graph Neural Network layer defined by:
\begin{equation}
\label{eq:gcn_layer} h_i^{(\ell+1)} = \frac{1}{d_i+1}h_i^{(\ell)}W^{(\ell)} + \sum_{j\sim i} \frac{h_j^{(\ell)}W^{(\ell)}}{\sqrt{(d_i+1)(d_j+1)}},
\end{equation}
where $i\sim j$ means that nodes $i$ and $j$ are neighbors in the graph $G$, $d_i$ and $d_j$ are the respective degrees of nodes $i$ and $j$ (i.e. their number of neighbors in the graph) and $h_i^{(\ell)}$ is the embedding representation of node $i$ at layer $\ell$ and $W^{(\ell)}$ is a trainable weight matrix of shape `[size_input_feature, size_output_feature]`.

The [inductive bias](https://en.wikipedia.org/wiki/Inductive_bias) of a learning algorithm is the set of assumptions that the learner uses to predict outputs of given inputs that it has not encountered. For GCN, we argue that the inductive bias can be formulated as a simple spectral property of the algorithm: GCN acts as low-pass filters. This arguments follows from recent works [Simplifying Graph Convolutional Networks](http://proceedings.mlr.press/v97/wu19e.html) by Wu, Souza, Zhang, Fifty, Yu, Weinberger and [Revisiting Graph Neural Networks: All We Have is Low-Pass Filters](https://arxiv.org/abs/1905.09550) by NT and Maehara.

Here we will study a very simple case and relate the inductive bias of GCN to the property of the Fiedler vector of the graph. We'll consider the more general setting in a subsequent post.

## Notations

We consider undirected graphs $G=(V,E)$ with $n$ vertices denoted by $i,j \in [n]$. $i\sim j$ means that nodes $i$ and $j$ are neighbors in $G$, i.e. $\{i,j\}\in E$. We denote by $A$ its [adjacency matrix](https://en.wikipedia.org/wiki/Adjacency_matrix) and by $D$ the diagonal matrix of degrees. The vector of degrees is denoted by $d$ so that $d= A1$. The components of a vector $x\in \mathbb{R}^n$ are denoted $x_i$ but sometimes it is convenient to see the vector $x$ as a function from $V$ to $\mathbb{R}$ and use the notation $x(i)$ instead of $x_i$.

```python
# Install required packages.
!pip install -q torch-scatter -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html
!pip install -q torch-sparse -f https://pytorch-geometric.com/whl/torch-1.8.0+cu101.html
!pip install -q torch-geometric
```

```python
# Helper function for visualization.
%matplotlib inline
import torch
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np

from sklearn.metrics.cluster import normalized_mutual_info_score

def visualize(h, color, cmap="Set1"):
    plt.figure(figsize=(7,7))
    plt.xticks([])
    plt.yticks([])

    if torch.is_tensor(h):
        h = h.detach().cpu().numpy()
        plt.scatter(h[:, 0], h[:, 1], s=140, c=color, cmap=cmap)
        [m0,m1] = np.median(h,axis=0)
        [min0, min1] = np.min(h,axis=0)
        [max0, max1] = np.max(h,axis=0)
        plt.vlines(m0,min1,max1)
        plt.hlines(m1,min0,max0)
        for i in range(h.shape[0]):
            plt.text(h[i,0], h[i,1], str(i))

    else:
        nx.draw_networkx(G, pos=nx.spring_layout(G, seed=42), with_labels=True,
                         node_color=color, cmap=cmap)
    plt.show()
```

## Community detection in the Karate Club

We'll start with an unsupervised problem: given one graph, find a partition of its node in communities. In this case, we make the hypothesis that individuals tend to associate and bond with similar others, which is known as [homophily](https://en.wikipedia.org/wiki/Homophily).

To study this problem, we will focus on the [Zachary's karate club](https://en.wikipedia.org/wiki/Zachary%27s_karate_club) and try to recover the split of the club from the graph of connections. The [pytorch-geometric](https://pytorch-geometric.readthedocs.io/en/latest/#) library will be very convenient.

Note that GCN are not appropriate in an unsupervised setting as no learning is possible without any label on the vertices. However, this is not a problem here as we will not train the GCN! In more practical settings, GCN are used in a semi-supervised setting where a few labels are revealed for a few nodes (more on this in the section with the Cora dataset).

```python
from torch_geometric.datasets import KarateClub

dataset = KarateClub()
print(f'Dataset: {dataset}:')
print('======================')
print(f'Number of graphs: {len(dataset)}')
print(f'Number of features: {dataset.num_features}')
print(f'Number of classes: {dataset.num_classes}')
```

As shown above, the default number of classes (i.e. subgroups) in pytorch-geometric is 4, for simplicity, we'll focus on a partition in two groups only:

```python
data = dataset[0]
biclasses = [int(b) for b in ((data.y == data.y[0]) + (data.y==data.y[5]))]
```

We will use [networkx](https://networkx.org/) for drawing the graph. On the picture below, the color of each node is given by its "true" class.

```python
from torch_geometric.utils import to_networkx

G = to_networkx(data, to_undirected=True)
visualize(G, color=biclasses)
```

```python
def acc(predicitions, classes):
    n_tot = len(classes)
    acc = np.sum([int(pred)==cla for pred,cla in zip(predicitions,classes)])
    return max(acc, n_tot-acc), n_tot
```

The [Kernighan Lin algorithm](https://en.wikipedia.org/wiki/Kernighan%E2%80%93Lin_algorithm) is a heuristic algorithm for finding partitions of graphs and the results below show that it captures well our homophily assumption. Indeed the algorithm tries to minimize the number of crossing edges between the 2 communities.

```python
c1,c2 = nx.algorithms.community.kernighan_lin_bisection(G)
classes_kl = [0 if i in c1 else 1 for i in range(34)]
visualize(G, color=classes_kl, cmap="Set2")
```

```python
acc(classes_kl, biclasses)
```

```python
n_simu = 1000
all_acc = np.zeros(n_simu)
for i in range(n_simu):
    c1,c2 = nx.algorithms.community.kernighan_lin_bisection(G)
    classes_kl = [0 if i in c1 else 1 for i in range(34)]
    all_acc[i],_ = acc(classes_kl, biclasses)
```

The algorithm is not deterministic but performs poorly only a small fractions of the trials as shown below:

```python
bin_list = range(17,35)
_ = plt.hist(all_acc, bins=bin_list,rwidth=0.8)
```

## Inductive bias for GCN

To demonstrate the inductive bias for the GCN architecture, we consider a simple GCN with 3 layers and look at its performance without any training. To be more precise, the GCN takes as input the graph and outputs a vector $(x_i,y_i)\in \mathbb{R}^2$ for each node $i$.

```python
import torch
from torch.nn import Linear
from torch_geometric.nn import GCNConv
import torch.nn.functional as F

class GCN(torch.nn.Module):
    def __init__(self):
        super(GCN, self).__init__()
        self.conv1 = GCNConv(data.num_nodes, 4)# no feature...
        self.conv2 = GCNConv(4, 4)
        self.conv3 = GCNConv(4, 2)

    def forward(self, x, edge_index):
        h = self.conv1(x, edge_index)
        h = h.tanh()
        h = self.conv2(h, edge_index)
        h = h.tanh()
        h = self.conv3(h, edge_index)
        return h

torch.manual_seed(12345)
model = GCN()
print(model)
```

Below, we draw all the points $(x_i,y_i)$ for all nodes $i$ of the graph. The vertical and horizontal lines are the medians of the $x_i$'s and $y_i$'s respectively. The colors are the true classes. We see that __without any learning__ the points are almost separated in the lower-left and upper-right corners according to their community!

```python
h = model(data.x, data.edge_index)
visualize(h, color=biclasses)
```

```python
def color_from_vec(vec,m=None):
    if torch.is_tensor(vec):
        vec = vec.detach().cpu().numpy()
    if not(m):
        m = np.median(vec,axis=0)
    return np.array(vec < m)
```

Note that by drawing the medians above, we enforce a balanced partition of the graph. Below, we draw the original graph where the color for node $i$ depends if $x_i$ is larger or smaller than the median.

```python
color_out = color_from_vec(h[:,0])
visualize(G, color=color_out, cmap="Set2")
```

We made only a few errors without any training!

```python
acc(color_out,biclasses)
```

Our result might depend on the particular initialization, so we run a few more experiments below:

```python
all_acc = np.zeros(n_simu)
for i in range(n_simu):
    model = GCN()
    h = model(data.x, data.edge_index)
    color_out = color_from_vec(h[:,0])
    all_acc[i],_ = acc(color_out,biclasses)
```

```python
_ = plt.hist(all_acc, bins=bin_list,rwidth=0.8)
```

```python
np.mean(all_acc)
```

We see that on average, we have an accuracy over $24/34$ which is much better than chance!

We now explain why the GCN architecture with random initialization achieves such good results.

## Spectral analysis of GCN

We start by rewriting the equation \eqref{eq:gcn_layer} in matrix form:
$$
h^{(\ell+1)} = S h^{(\ell)}W^{(\ell)} ,
$$
where the scaled adjacency matrix $S\in\mathbb{R}^{n\times n}$ is defined by
$S_{ij} = \frac{1}{\sqrt{(d_i+1)(d_j+1)}}$ if $i\sim j$ or $i=j$ and $S_{ij}=0$ otherwise and $h^{(\ell)}\in \mathbb{R}^{n\times f^{(\ell)}}$ is the embedding representation of the nodes at layer $\ell$ and $W^{(\ell)}$ is the learnable weight matrix in $\mathbb{R}^{f^{(\ell)}\times f^{(\ell+1)}}$.

To simplify, we now ignore the $tanh$ non-linearities in our GCN above so that we get
$$
y =  S^3 W^{(1)}W^{(2)}W^{(3)},
$$
where $W^{(1)}\in \mathbb{R}^{n,4}$, $W^{(2)}\in \mathbb{R}^{4,4}$ and $W^{(3)}\in \mathbb{R}^{4,2}$ and $y\in \mathbb{R}^{n\times 2}$ is the output of the network (note that `data.x` is the identity matrix here).
The vector $W^{(1)}W^{(2)}W^{(3)}\in \mathbb{R}^{n\times 2}$ is a random vector with no particular structure so that to understand the inductive bias of our GCN, we need to understand the action of the matrix $S^3$.

The matrix $S$ is symmetric with eigenvalues $\nu_1\geq \nu_2\geq ...$ and associated eigenvectors $U_1,U_2,...$
We can show that indeed $1=\nu_1>\nu_2\geq ...\geq \nu_n\geq -1$ by applying Perron-Frobenius theorem. This is illustrated below.

```python
from numpy import linalg as LA

A = nx.adjacency_matrix(G).todense()
A_l = A + np.eye(A.shape[0],dtype=int)
deg_l = np.dot(A_l,np.ones(A.shape[0]))
scaling = np.dot(np.transpose(1/np.sqrt(deg_l)),(1/np.sqrt(deg_l)))
S = np.multiply(scaling,A_l)
eigen_values, eigen_vectors = LA.eigh(S)

_ = plt.hist(eigen_values, bins = 40)
```

But the most interesting fact for us here concerns the eigenvector $U_2$ associated with the second largest eigenvalues which is also known as the [Fiedler vector](https://en.wikipedia.org/wiki/Algebraic_connectivity).

A first result due to Fiedler tells us that the subgraph induced by $G$ on vertices with $U_2(i)\geq 0$ is connected. This is known as Fiedler’s Nodal Domain Theorem (see Chapter 24 in [Spectral and Algebraic Graph Theory](http://cs-www.cs.yale.edu/homes/spielman/sagt/) by Daniel Spielman). We check this fact below both on $U_2$ and $-U_2$ so that here we get a partition of our graph in 2 connected graphs (since we do not have any node $i$ with $U_2(i)=0$).

```python
fiedler = np.array(eigen_vectors[:,-2]).squeeze()
H1 = G.subgraph([i for (i,f) in enumerate(fiedler) if f>=0])
H2 = G.subgraph([i for (i,f) in enumerate(fiedler) if -f>=0])
H = nx.union(H1,H2)
plt.figure(figsize=(7,7))
plt.xticks([])
plt.yticks([])
nx.draw_networkx(H, pos=nx.spring_layout(G, seed=42), with_labels=True)
```

There are many possible partitions of our graph in 2 connected graphs and we see here that the Fiedler vector actually gives a very particular partition corresponding almost exactly to the true communities!

```python
visualize(G, color=[fiedler>=0], cmap="Set2")
```

There are actually very few errors made by Fiedler's vector. Another way to see the performance of the Fiedler's vector is to sort its entries and color each dot with its community label as done below:

```python
fiedler_c = np.sort([biclasses,fiedler], axis=1)
fiedler_1 = [v for (c,v) in np.transpose(fiedler_c) if c==1]
l1 = len(fiedler_1)
fiedler_0 = [v for (c,v) in np.transpose(fiedler_c) if c==0]
l0 = len(fiedler_0)
plt.plot(range(l0),fiedler_0,'o',color='red')
plt.plot(range(l0,l1+l0),fiedler_1,'o',color='grey')
plt.plot([0]*35);
```

To understand why the partition of Fiedler's vector is so good requires a bit of calculus. To simplify a bit, we will make a small modification about the matrix $S$ and define it to be $S_{ij} = \frac{1}{\sqrt{d_i d_j}}$ if $i\sim j$ or $i=j$ and $S_{ij}=0$ otherwise.

Define the (normalized) Laplacian $L=Id-S$ so that the eigenvalues of $L$ are $\lambda_i=1-\nu_i$ associated with the same eigenvector $U_i$ as for $S$. We also define the combinatorial [Laplacian](https://en.wikipedia.org/wiki/Laplacian_matrix) $L^* = D-A$.

We then have
\begin{equation}
\frac{x^TLx}{x^Tx} = \frac{x^TD^{-1/2}L^* D^{-1/2}x}{x^Tx}\\
= \frac{y^T L^* y}{y^TDy},
\end{equation}
where $y = D^{-1/2}x$. In particular, we get:
\begin{equation}
\lambda_2 = 1-\nu_2 = \min_{x\perp U_1}\frac{x^TLx}{x^Tx}\\
= \min_{y\perp d} \frac{y^T L^* y}{y^TDy},
\end{equation}
where $d$ is the vector of degrees.

Rewriting this last equation, we obtain
\begin{equation}
\label{eq:minlambda}\lambda_2 = \min \frac{\sum_{i\sim j}\left(y(i)-y(j)\right)^2}{\sum_i d_i y(i)^2},
\end{equation}
where the minimum is taken over vector $y$ such that $\sum_i d_i y_i =0$.

Now if $y^*$ is a vector achieving the minimum then we get the Fiedler vector (up to a sign) by $U_2 =  \frac{D^{1/2}y^*}{\|D^{1/2}y^*\|}$. In particular, we see that the sign of the elements of $U_2$ are the same as the elements of $y^*$.

To get an intuition about \eqref{eq:minlambda}, consider the same minimization but with the constraint that $y(i) \in \{-1,1\}$ with the meaning that if $y(i)=1$, then node $i$ is in community $0$ and if $y(i)=-1$ then node $i$ is in community $1$. In this case, we see that the numerator $\sum_{i\sim j}\left(y(i)-y(j)\right)^2$ is the number of edges between the two communities multiplied by 4 and the denominator $\sum_i d_i y(i)^2$ is twice the total number of edges in the graph. Hence the minimization problem is now a combinatorial problem asking for a graph partition $(P_1,P_2)$ of the graph under the constraint that $\sum_{i\in P_1}d_i= \sum_{j\in P_2} d_j$. This last condition is simply saying that the number of edges in the graph induced by $G$ on $P_1$ should be the same as the number of edges in the graph induced by $G$ on $P_2$ (note that this condition might not have a solution). Hence the minimization problem defining $y^*$ in \eqref{eq:minlambda} can be seen as a relaxation of this [bisection problem](https://en.wikipedia.org/wiki/Graph_partition#Spectral_partitioning_and_spectral_bisection). We can then expect the Fiedler vector to be close to $y^*$ at least the signs of its elements which would explain that the partition obtained thanks to the Fiedler vector is balanced and with a small cut, corresponding exactly to our goal here.

So now that we understand the Fiedler vector, we are ready to go back toi GCN. First, we check that the small simplification we made (removing non-linearities...) are really unimportant:

```python
torch.manual_seed(12345)
model = GCN()
W1 = model.conv1.weight.detach().numpy()
W2 = model.conv2.weight.detach().numpy()
W3 = model.conv3.weight.detach().numpy()

iteration = S**3*W1*W2*W3
visualize(torch.tensor(iteration), color=biclasses)
```

OK, we get the same embedding as with the untrained network but we now have a simpler math formula for the output:
$$
[Y_1,Y_2] = S^3 [R_1, R_2],
$$
where $R_1,R_2$ are $\mathbb{R}^n$ random vectors and $Y_1, Y_2$ are the output vectors in $\mathbb{R}^n$ used to do the scatter plot above.

But we can rewrite $S = \sum_{i}\nu_i U_i U_i^T$ so that we get $S^3 = \sum_{i}\nu_i^3 U_i U_i^T \approx U_1U_1^T + \nu_2^3 U_2U_2^T$ because all others $\nu_i<< \nu_2^3$. Hence, we get
\begin{equation}
Y_1 \approx U_1^T R_1 U_1 + \nu_2^3 U_2^T R_1 U_2 \\
Y_2 \approx U_1^T R_2 U_1 + \nu_2^3 U_2^T R_2 U_2
\end{equation}
Recall that the signal about the communities is in the $U_2$ vector so that we can rewrite it more explicitly as
\begin{equation}
Y_1(i) \approx a_1 + b_1 U_2(i)\\
Y_2(i) \approx a_2 + b_2 U_2(i),
\end{equation}
where $a_1,a_2,b_1,b_2$ are random numbers of the same magnitude. In other words, the points $(Y_1(i), Y_2(i))$ should be approximately aligned on a line and the two extremes of the corresponding segment correspond to the 2 communities $U_2(i)\geq 0$ or $U_2(i)\leq 0$.

```python
from sklearn import linear_model
from sklearn.metrics import mean_squared_error
regr = linear_model.LinearRegression()
regr.fit(iteration[:,0].reshape(-1, 1), iteration[:,1])
plt.figure(figsize=(7,7))
plt.xticks([])
plt.yticks([])
h = np.array(iteration)
plt.scatter(h[:, 0], h[:, 1], s=140, c=biclasses, cmap="Set1")
plt.plot(h[:, 0],regr.predict(iteration[:,0].reshape(-1, 1)))
```

```python
def glorot_normal(in_c,out_c):
    sigma = np.sqrt(2/(in_c+out_c))
    return sigma*np.random.randn(in_c,out_c)

```

```python
coef = np.zeros(n_simu)
base =  np.zeros(n_simu)
for i in range(n_simu):
    iteration = glorot_normal(34,4)@glorot_normal(4,4)@glorot_normal(4,2)
    regr.fit(iteration[:,0].reshape(-1, 1), iteration[:,1])
    base[i] = mean_squared_error(iteration[:,1],regr.predict(iteration[:,0].reshape(-1, 1)))
    iteration = np.array(S**3) @ iteration
    regr.fit(iteration[:,0].reshape(-1, 1), iteration[:,1])
    coef[i] = mean_squared_error(iteration[:,1],regr.predict(iteration[:,0].reshape(-1, 1)))
```

Below, we run a few simulations and compute the mean squared error between the points and the best interpolating line for the random input $[R_1,R_2]$ in blue and for the output $[Y_1, Y_2]$ in orange (that you can hardly see because the error is much smaller). Our theory seems to be nicely validated ;-)

```python
_ = plt.hist(base, bins = 34)
_ = plt.hist(coef, bins = 34)
```

Here we studied a very simple case but more general statements are possible as we will see in a subsequent post. To generalize the analysis made about Fiedler vector requires a little bit of spectral graph theory as explained in the module on spectral Graph Neural Networks, see [Deep Learning on graphs (2)](https://dataflowr.github.io/website/modules/graph2/)

Follow on [twitter](https://twitter.com/marc_lelarge)!

## Thanks for reading!


```python
import random
import math
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from scipy.integrate import odeint
import pandas as pd
from sympy import ordered
```

```python
def ode_model(contamination,t,r,C,m,g):

	Contamination = contamination;
	return(r*(1-Contamination/C)*Contamination-m*math.exp(-g*t)*Contamination)
	#return(r*(1-Contamination/C)-m*math.exp(-g*t)*Contamination)

# Extract specific time-ppints from ODE
def deterministic_run(precision,initial_contamination,r,C,m,g):
    tmax = 24

    time_space = np.linspace(0,tmax,precision+1)

    sim=odeint(ode_model,initial_contamination,time_space,args=(r,C,m,g))

    num_at_0=initial_contamination
    num_at_1=sim[int(precision*1.0/tmax)]
    num_at_2=sim[int(precision*2.0/tmax)]
    num_at_4=sim[int(precision*4.0/tmax)]
    num_at_8=sim[int(precision*8.0/tmax)]
    num_at_24=sim[int(precision*24.0/tmax)]

    return([num_at_0,num_at_1,num_at_2,num_at_4,num_at_8,num_at_24])
```

# Import the data

```python
df = pd.read_csv("../../Posterior_Beth_Expt.txt")
#create headers for the dataframe r', 'C', 'mu_detergent' 'gamma_detergent' 'mu_disinfectanct' 'gamma_disinfectanct' 'mu_distilled_water' 'gamma_distilled_water':
df.columns = ['r','C','mu_detergent','gamma_detergent','mu_disinfectant','gamma_disinfectant','mu_distilled_water','gamma_distilled_water']

```

```python
# Store Experimental data: Detergent, Disinfectant, Control and Distilled Water

Detergent_Means=[[np.zeros((1,6)) for i in range(1)] for j in range(1)] #surface, phase
Detergent_SD=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]

Disinfectant_Means=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]
Disinfectant_SD=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]

Control_Means=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]
Control_SD=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]

Distilled_Means=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]
Distilled_SD=[[np.zeros((1,6)) for i in range(1)] for j in range(1)]

# We store Detergent[Surface][Phase]

# Detergent for the Locker. Phases 1,2,3.
Detergent_Means[0][0] = [221.6,94.3,56.25,1.75,1.6,8.5]
Detergent_SD[0][0] = [76.4,86.9,89.4,0.5,2.3,4.04]

Disinfectant_Means[0][0] = [59.4,18.7,5.4,5.4,2.4,8.6]
Disinfectant_SD[0][0] = [91.8,26.2,2.30,4.67,4.34,4.28]

Distilled_Means[0][0] = [261.0,175.5,47,18.6,12.8,56]
Distilled_SD[0][0] = [31.5,61.7,9.0,13.2,78.2,76.4]

Control_Means[0][0] = [59.4,41,52,18.6,21,16.5]
Control_SD[0][0] = [23.5,9.9,21.7,13.2,16.2,6.54]

```

# Plot Detergent

```python
#Plot errorbars of experiments with predictions ontop

precision=5000
tmax = 24
time_space = np.linspace(0,tmax,precision+1)

#1. Detergent
initial_contamination=Detergent_Means[0][0][0]
P=odeint(ode_model,initial_contamination,time_space,args=(df["r"].median(),df["C"].median(),df["mu_detergent"].median(),df["gamma_detergent"].median()))
Pmin=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.05),df["C"].quantile(0.05),df["mu_detergent"].quantile(0.05),df["gamma_detergent"].quantile(0.05)))
Pmax=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.95),df["C"].quantile(0.95),df["mu_detergent"].quantile(0.95),df["gamma_detergent"].quantile(0.95)))

# create a vector of values between 0 and 6
x = np.array([0,1,2,4,8,24])

#Define new sd just for plotting to avoid SD value at 0
s = Detergent_SD[0][0]
measurement_data = Detergent_Means[0][0]#np.array([1200,134.0,202.0,294.0])

#Plot errobars of experimental data
plt.errorbar(x,measurement_data,yerr=s,fmt='o', color='black',label='Experimental data')

#Plot the model prediction
plt.plot(time_space,P,label="Model prediction",color='blue')

#Plot confidence intervals around the model prediction

plt.fill_between(time_space,np.concatenate(Pmin),np.concatenate(Pmax),alpha=0.2,color='blue')
#plt.plot(time_space,Pmin,label="Model prediction",color='red')
#plt.plot(time_space,Pmax,label="Model prediction",color='red')

#plt.fill_between(x, np.array(map(operator.sub, P["Contamination"], Pmin["Contamination"])), np.array(map(operator.add, P["Contamination"], Pmax["Contamination"])), color='b', alpha=.1)
plt.xlim(-1,25)
plt.ylim(0.1,500)
plt.ylabel("CFU recovered from coupon \n after t hours")
plt.yscale("log")
plt.xlabel("Hours after surface cleaning")
plt.legend(loc="upper right")
plt.title("Detergent")
#add parameter values to the plot
plt.text(0.5, 0.5, 'r = '+str(round(df["r"].median(),2))+'\nC = '+str(round(df["C"].median(),2))+'\nmu_detergent = '+str(round(df["mu_detergent"].median(),2))+'\ngamma_detergent = '+str(round(df["gamma_detergent"].median(),2)),
        horizontalalignment='center',
        verticalalignment='center',
        transform = plt.gca().transAxes)

#save the plot
plt.savefig("abc_prediction_BethExpt_detergent.png", dpi=600)

plt.show()
```

# Plot Disinfectant

```python
precision=5000
tmax = 24
time_space = np.linspace(0,tmax,precision+1)

#2. disinfectant
initial_contamination=Disinfectant_Means[0][0][0]
P=odeint(ode_model,initial_contamination,time_space,args=(df["r"].median(),df["C"].median(),df["mu_disinfectant"].median(),df["gamma_disinfectant"].median()))
Pmin=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.05),df["C"].quantile(0.05),df["mu_disinfectant"].quantile(0.05),df["gamma_disinfectant"].quantile(0.05)))
Pmax=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.95),df["C"].quantile(0.95),df["mu_disinfectant"].quantile(0.95),df["gamma_disinfectant"].quantile(0.95)))

# create a vector of values between 0 and 6
x = np.array([0,1,2,4,8,24])

#Define new sd just for plotting to avoid SD value at 0
s = Disinfectant_SD[0][0]
measurement_data = Disinfectant_Means[0][0]#np.array([1200,134.0,202.0,294.0])

#Plot errobars of experimental data
plt.errorbar(x,measurement_data,yerr=s,fmt='o', color='black',label='Experimental data')

#Plot the model prediction
plt.plot(time_space,P,label="Model prediction",color='blue')

#Plot confidence intervals around the model prediction

plt.fill_between(time_space,np.concatenate(Pmin),np.concatenate(Pmax),alpha=0.2,color='green')
#plt.plot(time_space,Pmin,label="Model prediction",color='red')
#plt.plot(time_space,Pmax,label="Model prediction",color='red')

#plt.fill_between(x, np.array(map(operator.sub, P["Contamination"], Pmin["Contamination"])), np.array(map(operator.add, P["Contamination"], Pmax["Contamination"])), color='b', alpha=.1)
plt.xlim(-1,25)
plt.ylim(1,125)
plt.ylabel("CFU recovered from coupon \n after t hours")
#plt.yscale("log")
plt.xlabel("Hours after surface cleaning")
plt.legend(loc="upper right")
plt.title("Disinfectant")
plt.text(0.5, 0.5, 'r = '+str(round(df["r"].median(),2))+'\nC = '+str(round(df["C"].median(),2))+'\nmu_disinfectant = '+str(round(df["mu_disinfectant"].median(),2))+'\ngamma_disinfectant = '+str(round(df["gamma_disinfectant"].median(),2)),
        horizontalalignment='center',
        verticalalignment='center',
        transform = plt.gca().transAxes)

#save the plot
plt.savefig("abc_prediction_BethExpt_disinfectant.png", dpi=600)

plt.show()

```

# Distilled water

```python
precision=5000
tmax = 24
time_space = np.linspace(0,tmax,precision+1)

#2. disinfectant
initial_contamination=Distilled_Means[0][0][0]
P=odeint(ode_model,initial_contamination,time_space,args=(df["r"].median(),df["C"].mean(),df["mu_distilled_water"].median(),df["gamma_distilled_water"].median()))
Pmin=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.05),df["C"].quantile(0.05),df["mu_distilled_water"].quantile(0.05),df["gamma_distilled_water"].quantile(0.05)))
Pmax=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.95),df["C"].quantile(0.95),df["mu_distilled_water"].quantile(0.95),df["gamma_distilled_water"].quantile(0.95)))

# create a vector of values between 0 and 6
x = np.array([0,1,2,4,8,24])

#Define new sd just for plotting to avoid SD value at 0
s = Distilled_SD[0][0]
measurement_data = Distilled_Means[0][0]#np.array([1200,134.0,202.0,294.0])

#Plot errobars of experimental data
plt.errorbar(x,measurement_data,yerr=s,fmt='o', color='black',label='Experimental data')

#Plot the model prediction
plt.plot(time_space,P,label="Model prediction",color='blue')

#Plot confidence intervals around the model prediction

plt.fill_between(time_space,np.concatenate(Pmin),np.concatenate(Pmax),alpha=0.2,color='purple')
#plt.plot(time_space,Pmin,label="Model prediction",color='red')
#plt.plot(time_space,Pmax,label="Model prediction",color='red')

#plt.fill_between(x, np.array(map(operator.sub, P["Contamination"], Pmin["Contamination"])), np.array(map(operator.add, P["Contamination"], Pmax["Contamination"])), color='b', alpha=.1)
plt.xlim(-1,25)
plt.ylim(1,300)
plt.ylabel("CFU recovered from coupon \n after t hours")
#plt.yscale("log")
plt.xlabel("Hours after surface cleaning")
plt.legend(loc="upper right")
plt.title("Distilled water")
plt.text(0.5, 0.5, 'r = '+str(round(df["r"].median(),2))+'\nC = '+str(round(df["C"].median(),2))+'\nmu_distilled_water = '+str(round(df["mu_distilled_water"].median(),2))+'\ngamma_distilled_water = '+str(round(df["gamma_distilled_water"].median(),2)),
        horizontalalignment='center',
        verticalalignment='center',
        transform = plt.gca().transAxes)
#save the plot
plt.savefig("abc_prediction_BethExpt_distilled_water.png", dpi=600)

plt.show()
```

# Control

```python
precision=5000
tmax = 24
time_space = np.linspace(0,tmax,precision+1)

#2. disinfectant
initial_contamination=Control_Means[0][0][0]
P=odeint(ode_model,initial_contamination,time_space,args=(df["r"].median(),df["C"].median(),0,0))
Pmin=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.05),df["C"].quantile(0.05),0,0))
Pmax=odeint(ode_model,initial_contamination,time_space,args=(df["r"].quantile(0.95),df["C"].quantile(0.95),0,0))

# create a vector of values between 0 and 6
x = np.array([0,1,2,4,8,24])

#Define new sd just for plotting to avoid SD value at 0
s = Control_Means[0][0]
measurement_data = Control_Means[0][0]#np.array([1200,134.0,202.0,294.0])

#Plot errobars of experimental data
plt.errorbar(x,measurement_data,yerr=s,fmt='o', color='black',label='Experimental data')

#Plot the model prediction
plt.plot(time_space,P,label="Model prediction",color='blue')

#Plot confidence intervals around the model prediction

plt.fill_between(time_space,np.concatenate(Pmin),np.concatenate(Pmax),alpha=0.2,color='orange')
#plt.plot(time_space,Pmin,label="Model prediction",color='red')
#plt.plot(time_space,Pmax,label="Model prediction",color='red')

#plt.fill_between(x, np.array(map(operator.sub, P["Contamination"], Pmin["Contamination"])), np.array(map(operator.add, P["Contamination"], Pmax["Contamination"])), color='b', alpha=.1)
plt.xlim(-1,25)
plt.ylim(10,125)
plt.ylabel("CFU recovered from coupon \n after t hours")
#plt.yscale("log")
plt.xlabel("Hours after surface cleaning")
plt.legend(loc="upper right")
plt.title("Control")
# add the values of the parameters to the plot
plt.text(0.5, 0.5, 'r = ' + str(round(df["r"].median(),2)) + '\n' + 'C = ' + str(round(df["C"].median(),2)),
        horizontalalignment='center',
        verticalalignment='center',
        transform=plt.gca().transAxes)

#save the plot
plt.savefig("abc_prediction_BethExpt_control.png", dpi=600)

plt.show()

```

```python

# Create pairplots of 50% of df
# make histograms in the diagonal with colour green
# make scatterplots in the upper triangle with colour blue
#g = sns.pairplot(df, diag_kind="kde",corner=True)
#g.map_lower(sns.kdeplot, levels=4, color=".2")

#Create a heatmap of correlations, anotate by p-value
corr = df.corr()
mask = np.zeros_like(corr, dtype=np.bool)
mask[np.triu_indices_from(mask)] = True
f, ax = plt.subplots(figsize=(11, 9))
cmap = sns.diverging_palette(220, 10, as_cmap=True)
sns.heatmap(corr, mask=mask, cmap=cmap, vmax=.3, center=0,
           square=True, linewidths=.5, cbar_kws={"shrink": .5})
plt.show()

# sns.heatmap(df.corr(),annot=True)
```

```python
df.describe()
```

      r
      C
      mu-detergent
      gamma-detergent
      mu-disinfectant
      gamma-disinfectant
      mu-distilled-water
      gamma-distilled-water
      die_off

      count
      999.000000
      999.000000
      999.000000
      999.000000
      999.000000
      999.000000
      999.000000
      999.000000
      999.000000

      mean
      0.591299
      40.341188
      0.505991
      0.508988
      0.514662
      0.514876
      0.462525
      0.537913
      0.348898

      std
      0.272396
      16.986121
      0.288530
      0.282117
      0.278590
      0.279990
      0.279929
      0.276786
      0.223340

      min
      0.001752
      1.087101
      0.010203
      0.008276
      0.010503
      0.000315
      0.010911
      0.006164
      0.001279

      25%
      0.374012
      27.092177
      0.263234
      0.261530
      0.287993
      0.290919
      0.218165
      0.308269
      0.171202

      50%
      0.638577
      40.524808
      0.501379
      0.511635
      0.526131
      0.507533
      0.448630
      0.547475
      0.317811

      75%
      0.830506
      54.584169
      0.747974
      0.749161
      0.751773
      0.746498
      0.687635
      0.775412
      0.495339

      max
      0.998731
      69.933323
      0.999445
      0.998863
      0.999629
      0.998386
      0.998954
      0.999705
      1.031695

```python
# Create histograms
#sns.distplot(df["r"])
#sns.distplot(df["C"])
#plot all mu_distilled_water, mu_detergent and mu_distilled_water together
sns.distplot(df["mu_distilled_water"])
sns.distplot(df["mu_detergent"])
sns.distplot(df["mu_disinfectant"])

```

```python
# df pairplot
#sns.pairplot(df, diag_kind="hist",corner=True)

```

```python
precision=5000
tmax = 24
time_space = np.linspace(0,tmax,precision+1)

initial_contamination=1200 #Distilled_Means[0][0][0]
P=odeint(ode_model,initial_contamination,time_space,args=(0.1,20,5,.05))
x = np.array([0,1,2,4,8,24]) #np.array([0,2,4,6,16,18,20,22,24]) #
#Define new sd just for plotting to avoid SD value at 0
s = Distilled_SD[0][0] #[385,94,86,162,117,124,397,629,1147]#
measurement_data = Distilled_Means[0][0]#[1200,134,202,294,400,644,1232,2044,2866]#np.array([1200,134.0,202.0,294.0])
#Plot errobars of experimental data
plt.errorbar(x,measurement_data,yerr=s,fmt='o', color='black',label='Experimental data')
#plt.yscale("log")
#plt.xlabel("Hours after surface cleaning")
plt.ylim(0,300)
#Plot the model prediction
plt.plot(time_space,P,label="Model prediction",color='blue')
plt.show()
```


```python
# Necessary imports
import numpy as np                                  # numpy (linear algebra and more)
import cvxpy as cp                                  # cvxpy (convex optimization solver)
from scipy.signal import butter, lfilter, freqz     # scipy.signal (low-pass filter)
import matplotlib.pyplot as plt                     # matplotlib (to plot stuff)
```

# Identification of linear systems

Identification of a linear system from input-output data -- an easy task (but we can make it difficult).

Topics addressed in this module:

- An overview of system identification
- Least-squares and reguarlized variants
- Model validation and selection
- Pre-filtering of I/O data

Recommended literature:

1. This is an excellent startin point: L. Ljung, *System Identification: Theory for the User*, Prentice-Hall, Upper Saddle River, New Jersey, 1999
2. An accessible introduction to system identification: K.J. Keesman, *System identification: an introduction*, Springer, 2011
3. A more modern approach to system identification using kernel methods: G. Pillonetto, F. Dinuzzo, T. Chen, G. De Nicolao and L. Ljung, *Kernel methods in system identification, machine learning and function estimation: a survey*, Automatica 50(1):657-682, 2014
4. P. Van den Hof, *Closed-loop issues in system identification*, Annual Reviews in Control 22:173-186, 1998

### System identification: objectives

The main objective of system identification is to infer information about the dynamics of a system using input-output data combined with, when available, prior information about the system.

Such prior information can be the knowledge that the system is linear (of known or unknown order), that it possesses a particular structure. As an example, assume we know that a system has the following structure
$$
\begin{align}
        x(t+1)
{}={}&
        \begin{bmatrix}
        \alpha & 1\\
        0 & 2\alpha
        \end{bmatrix}
        x(t)
        +
        \begin{bmatrix}
        1
        \\
        0
        \end{bmatrix}
        u(t)
        +
        w(t),
\\
        y(t)
{}={}&
        x(t),
\end{align}
$$
where $x$ is the state of the system, $u$ is the input and $w$ is an iid random noise.

Then, a typical system identification task would be to determine $\alpha$ using input-output data.

There exist cases in which we have no information about the system dynamics other than the fact that it is linear, that is,
$$
\begin{align}
        x(t+1)
{}={}&
        A
        x(t)
        +
        B
        u(t)
        +
        G
        w(t)
\\
        y(t)
{}={}&
        x(t),
\end{align}
$$
for matrices $A$, $B$ and $G$ of appropriate dimensions, where $w(t)$ is an iid random noise term. In such a case, our task consists in determining these matrices $A$, $B$ and $G$. Then our task would be to determine these matrices $A$, $B$ and $G$.

More generally, the system dynamics may by described by a set of nonlinear equations with either known or unknown structure. Here we focus on linear time-invariant dynamical systems.

### System dynamics

We have a system with structure
$$
    y(k)
{}={}
        \sum_{i=1}^{n}\alpha_i y(k-i)
{}+{}
        \sum_{i=1}^{m}\beta_{i} u(k-i)
{}+{}
        e(k),
$$
where $\alpha=(\alpha_1,\ldots, \alpha_n)$ and $\beta=(\beta_1,\ldots, \beta_m)$.

Define $\theta_\star=(\alpha^{\top}, \beta^{\top})^{\top}$ to be the vector of the true parameters of the system.

Then, for $k=\max(m,n),\ldots,N$ we have $y(k) = \begin{bmatrix}y(k-1)&\cdots&y(k-n)&u(k-1)&\cdots&u(k-m)\end{bmatrix}\theta$.

Therefore, assuming that $n>m$ -- as it is typically the case, we have

$$
    \begin{bmatrix}
    y(n)\\
    y(n+1)\\
    \vdots\\
    y(N)
    \end{bmatrix}
{}={}
    \underbrace{
    \begin{bmatrix}
    y(n-1) & \cdots & y(0) & u(n-1) & \cdots & u(n-m)\\
    y(n) & \cdots & y(1) & u(n) & \cdots & u(n-m+1)\\
    \vdots & \ddots & \vdots & \vdots & \ddots & \vdots\\
    y(N-1) & \cdots & y(N-n+1) & u(N) & \cdots & u(N-m+1)\\
    \end{bmatrix}}_{{}:={} \Phi_N}
    \theta_\star
{}+{}
    E,
$$
where $y(0), \ldots, y(n-1)$ are fixed and $E$ is the corresponding error term.

For the time being, we shall assume that the model structure, that is, the lengths $n$ and $m$ are known.

We now introduce the following class for linear systems:

```python
class linear_system:
    """Linear system class
    Defines a linear system of the form
         y(t) = a[0]*y(t-1) + ... + a[n]*y(t-n)
                + b[0]*u(t-1) + ... + b[m] * y(t-m)
                + e(t)
    with n > m, where
     y  :   system output
     u  :   input
     e  :   iid Gaussian noise with average 0 and variance `sigma`
    """

    def __init__(self, alpha, beta, sigma=0):
        """Constructor
        Syntax: ls = linear_system(alpha, beta, sigma),

        where
         alpha  :  coefficients of y
         beta   :  coefficients of u
         sigma  :  variance of e
        """
        self._alpha = alpha
        self._beta = beta
        self._theta_star = np.asmatrix(np.append(alpha,beta)).transpose();
        self._sigma = sigma

    def simulate(self, init_condition, u_input_signal):
        """Simulates the linear system
        Produces a simulation of the linear system using a
        given input signal
        """
        init_size = init_condition.size
        if (init_size < self.get_order()):
            raise Exception("Short initial condition!")
        num_data = u_input_signal.size
        y_output_signal = np.zeros(num_data+1)
        y_output_signal[:init_condition.size] = init_condition
        for t in range(1, num_data-init_size+2):
            y_t = 0.
            dt = t + init_size - 1
            for i in range(0, min(self._alpha.size, dt)):
                y_t += self._alpha[i]*y_output_signal[dt-i-1]
            for i in range(0, min(self._beta.size, dt)):
                y_t += self._beta[i]*u_input_signal[dt-i-1]

            y_output_signal[dt] = y_t + np.random.normal(0, self._sigma);
        return y_output_signal

    def get_phi(self):
        return self._phi

    def get_parameters(self):
        """System parameters
        Returns the system parameters `alpha` and `beta` in a
        vector theta = [alpha
                        beta ]
        """
        return self._theta_star

    def get_n(self):
        return self._alpha.size

    def get_m(self):
        return self._beta.size

    def get_order(self):
        return np.max([self.get_n(), self.get_m()])

```

```python
# -- Example: y[t] = 0.5 * y[t-1] - 2.0 * y[t-2] + u[t-1]
alpha_xmpl = np.array([0.5, -2.0])
beta_xmpl  = np.array([1.0])
mdl_xmpl   = linear_system(alpha_xmpl, beta_xmpl)

# Say we need to simulate the system with initial conditions y[0] = -4 and y[1] = 5
# and using the signal u[t] = sin(0.5 * t) for t = 0,...,T-1 with T=5
y_init_xmpl = np.array([-4, 5])
t_max_xmpl  = 5
time_xmpl   = np.linspace(0, t_max_xmpl, t_max_xmpl)
u_xmpl      = np.sin(0.5*time_xmpl[:-1])

Y_xmpl = mdl_xmpl.simulate(y_init_xmpl, u_xmpl)

print("excitation u = " + str(u_xmpl))
print("response   y = " + str(Y_xmpl))

plt.plot(time_xmpl[0:y_init_xmpl.size], Y_xmpl[0:y_init_xmpl.size], marker='x', color='r', markersize=12)
plt.plot(time_xmpl[y_init_xmpl.size:], Y_xmpl[y_init_xmpl.size:], color='b', marker='+', markersize=10)
plt.legend(['Initial Condition','Model prediction'])
plt.xlabel('Time')
plt.ylabel('System output y')
plt.grid()
plt.xticks(time_xmpl)
plt.show()
```

### Generation of I/O data

Let us generate data from that system using $e(t)\sim \mathcal{N}(0,\sigma^2)$ and the input signal $u(k) = c  \sin(k)$

```python
#### --- Construct a linear system
alpha = np.array([0.2,0.6,0.15]);
beta  = np.array([0.5, 1.2]);
sigma = 2.5
actual_system = linear_system(alpha, beta, sigma);

# --- Construct an input signal
num_data = 1000
time_span = np.linspace(0, num_data, num_data)
c = 10
u = c*(np.cos(0.3*time_span[:-1])                                       \
    + np.sin(0.08*time_span[:-1]**0.95))                                \
        *np.exp(-0.00015*time_span[:-1])

# --- Simulate the system
y0    = np.array([0,0,0])
y     = actual_system.simulate(y0, u)

# --- PLOT the input/output data
plt.figure(figsize=(15, 4))
plt.subplot(121)
plt.plot(time_span[:-1],u, linewidth=3, color='red')
plt.xlabel('time instant $k$')
plt.ylabel('input $u(k)$')
plt.subplot(122)
plt.plot(time_span,y, linewidth=3, color='green')
plt.xlabel('time instant $k$')
plt.ylabel('output $y(k)$')
plt.show()
```

### Construction of least-squares problem

We now solve the following regularized least-squares problem:

$$
\operatorname*{Minimize}_{\theta{}\in{}\mathbb{R}^{n+m}} \|\Phi\theta - Y\|^2 + R(\theta)
$$

Typical regularizers are:

- $R(\theta) = \lambda \theta^\top P^{-1} \theta$, for a matrix $P$. This is known as [Tikhonov regularization](https://en.wikipedia.org/wiki/Tikhonov_regularization) or ridge regression. It aims at avoiding multiple solutions of the non-regularized least-squares problem by favoring parameters with small norm.
- $R(\theta) = \lambda \|\theta\|_1$, which is used to enforce sparsity in $\theta$. This is often referred to as [LASSO](https://en.wikipedia.org/wiki/Tikhonov_regularization). The resulting vector $\theta^\star$ (the minimizer of the above regularized least-squares problem) will be sparser as we increase $\lambda$. A *debiasing* step is recommended after solving an $\|{}\cdot{}\|_1$-regularized least-squares problem:
- $R(\theta) = \lambda_1 \|\theta\|_1 + \lambda_2\|\theta\|_2^2$ is known as the [elastic net](https://en.wikipedia.org/wiki/Regularized_least_squares#Elastic_net) regularization. This regularizer generalizes both LASSO and ridge regression. The quadratic penalty term makes the loss function strictly convex, and it therefore has a unique minimum. The elastic net is used to select groups of correlated variables.

And many other...

The following class will be used to identify the given system from data

```python
class linear_system_identifier:
    """LTI system identification class
    This class is used to identify systems of a particular structure
    given input-output signals (u(t), y(t))
    """
    def __init__(self, input_data, output_data, n, m):
        """Constructor
        This is a constructor for `linear_system_identifier`
        Syntax:
        ssid = linear_system_identifier(input_signal, output_signal, n, m)

        where
        input_signal    :   input signal
        output_signal   :   output signal
        n               :   number of past y's affecting the output
        m               :   number of past u's affecting the output

        Parameters `n` and `m` determine the model order.

        Note: If `input_data` is of lenth `N`, output_data must be of
              length (at least) `N + 1`
        """
        self._u = input_data
        self._y = output_data[:input_data.size+1]
        self._n = n
        self._m = m

    def get_n(self):
        return self._n

    def get_m(self):
        return self._m

    def _inverse_range(self, low, high):
        return np.arange(high, low-1, -1)

    def construct_phi(self):
        num_data = self._u.size + 1
        num_rows_phi = num_data - self._n
        num_cols_phi = self._n + self._m
        phi = np.zeros([num_rows_phi, num_cols_phi])
        for i_row_phi in range(0, num_rows_phi):
            phi[i_row_phi][:self._n] =                                        \
                    self._y[                                                  \
                            self._inverse_range(                              \
                                    i_row_phi, i_row_phi + self._n - 1)       \
                            ]
            phi[i_row_phi][self._n:] =                                        \
                    self._u[                                                  \
                            self._inverse_range(                              \
                                    i_row_phi + 1, i_row_phi + self._m)       \
                            ]
        return np.matrix(phi)

    def estimate_params_rls(self, lam=0, verbose=False,                       \
                              regularizer=cp.sum_squares):
        """Estimates the system parameters by ReLS
        Solves a regularized least squares problem to determine the system
        parameters; the ReLS problem is of the form

        Minimize_theta |Phi * theta - Y|^2 + lambda * R(theta)

        Syntax:
        ssid.estimate_params_rls(lam=10, regularizer=cp.norm1, verbose=False)

        where
        ssid         :  object of type `linear_system_identifier`
        lam          :  regularization parameters (default = 0)
        regularizer  :  regularizer function
        verbose      :  whether to print out solver details (default=False)
        """
        phi      = self.construct_phi()
        theta    = cp.Variable(phi.shape[1])
        cost_fun = cp.sum_squares(phi*theta - self._y[self._n:])              \
                   + lam * regularizer(theta)
        prob  = cp.Problem(cp.Minimize(cost_fun))
        prob.solve(verbose=verbose, solver="SCS")
        alpha = theta.value[:self._n]
        beta = theta.value[self._n:]
        return linear_system(np.array(alpha), np.array(beta), 0)
```

```python
## --- Identify the system using I/O data
n_mdl = 3                                                       # model order n
m_mdl = 2                                                       # model order m
identifier     = linear_system_identifier(                      \
                                          u[:-1], y,            \
                                          m=m_mdl,              \
                                          n=n_mdl)              # system identifier
sys_identified = identifier.estimate_params_rls(lam=10)         # identify system
prms = sys_identified.get_parameters()                          # get system parameters
print("alpha :\n " + str(prms[:n_mdl]))                         # print alpha
print("beta  :\n " + str(prms[n_mdl:]))                         # print beta
```

    alpha :
     [[ 0.20024738]
     [ 0.56720056]
     [ 0.18143007]]
    beta  :
     [[ 0.43350534]
     [ 1.30088392]]

### Evaluation and Model Generalization Capacity

Several approaches exist in order to evaluate a model $M(p)$, where $p$ are its tuning parameters (e.g., above $p=(n,m, \lambda)$), in order to select the best model $M(p^\star)$, that is, the model that is likely to perform better than other models.

Criteria that are taken into account in model selection include, but are not limited to the following:

- Goodness of fit against a test set (which has not been used to train the model)
- (Occam's razor) Complexity (models of too high complexity are often undesired, although they might perform very well)
- Bias and variance of errors

Here we shall mainly focus on the first one - the goodness of fit against an external set of data. Assume the system has been actuated with a signal $(u^{\text{test}}(t))_{t=0}^{M}$ and has produced an output signal $(y^{\text{test}}(t))_{t=0}^{M}$.

Let us denote by $\hat{y}(t+k\mid t)$ the model prediction that time $t+k$ using information for $y$ up to time $k$.

The prediction error is

$$
e(t+k\mid t) = \hat{y}(t+k\mid t) - y^{\text{test}}(t+k).
$$

These are illustrated in the following figure:

Two very commonly used metrics are the *mean absolute error* (over the test set), that is

$$
\mathrm{MAE} = \mathrm{mean}(\{e(t+k\mid t)\}_{k=0}^{M})
$$

and the *root mean squared error*, that is

$$
\mathrm{RMS} = \mathrm{rms}(\{e(t+k\mid t)\}_{k=0}^{M})
$$

```python
## Model evaluation
num_data_test = 100
time_span_test = np.linspace(0, num_data_test, num_data_test)
u_test = 5 * np.sin(0.5*time_span_test)

init_condition = np.zeros(shape=(3))

def forecast_errors(u_test, actual_system, sys_identified, init_condition):
    y_actual_test = actual_system.simulate(init_condition, u_test)
    y_mdl_test    = sys_identified.simulate(init_condition, u_test)
    errors = y_actual_test - y_mdl_test
    return errors

errors = forecast_errors(u_test, actual_system, sys_identified, init_condition)

abs_err = np.abs(errors)
sqr_err = np.square(abs_err)

MAE  = np.sum(abs_err)
RMSE = np.sqrt(np.average(sqr_err))

print('MAE     = {0:.2f}'.format(MAE))
print('RMSE    = {0:.2f}'.format(RMSE))

```

    MAE     = 398.27
    RMSE    = 4.98

Additionally, we define the following index:

$$
        F_k
{}={}
        100
        \left(
            1
        {}-{}
            \sqrt{
                    \frac
                    {
                        \sum_{t} e(t+k\mid t)^2
                    }
                    {
                        \sum_{t} (y^{\text{test}}(t) - \bar{y}^{\text{test}} )^2
                    }
            }
        \right)
$$

$F_k$ concerns $k$-step-ahead predictions and in the numerator, the term $\sum_{t} e(t+k\mid t)^2$ quantifies the total $k$-step-ahead prediction error in the test set.

```python
num_data_test = 100
time_span_test = np.linspace(0, num_data_test, num_data_test)
U = 5 * np.sin(0.7 * time_span_test)
Y = actual_system.simulate(init_condition, u_test)

# -- we shall now evaluate our model using the test set (U, Y)
def predictor(model, U, Y, t, prediction_length):
    n = model.get_n()
    order = model.get_order()
    model.simulate()
```

### Model Selection

We may use the above statistics to evaluate a collection of models and choose the best among them.

As an exercise, for the above problem let us assume we know that $m=2$ and we need to determine the best value for $n$ and $\lambda$ using $\|{}\cdot{}\|_1$ as a regularizer.

```python
# Model selection
num_data_test = 200
time_span_test = np.linspace(0, num_data_test, num_data_test)
u_test = 5 * np.sin(0.5*time_span_test)

mdl_eval_cache = np.empty(shape=(0,3))
mdl_sys_cache  = np.empty(shape=(0,1))

for n_ in range(2,5):
    for l_ in np.logspace(0.1, 4.5, num=10):
        identifier = linear_system_identifier(u_test, y, m=2, n=n_)
        sys_identified = identifier.estimate_params_rls(                 \
                                    regularizer=cp.norm1,                \
                                    lam=l_,                              \
                                    verbose=False)
        errors = forecast_errors(u_test, actual_system, sys_identified, init_condition)
        abs_err = np.abs(errors)
        sqr_err = abs_err**2
        MAE  = np.sum(abs_err)
        RMSE = np.sqrt( np.average(sqr_err) )
        F = MAE
        F_prec = np.percentile(F, 95)
        eval_data_current = np.array([n_, l_, F_prec])
        mdl_sys_cache     = np.vstack([mdl_sys_cache, sys_identified]);
        mdl_eval_cache    = np.vstack([mdl_eval_cache, eval_data_current])

```

```python
from mpl_toolkits.mplot3d import Axes3D
fig = plt.figure(figsize=(7, 4))
ax = fig.add_subplot(111, projection='3d')

imax = np.argmin(mdl_eval_cache[:,2])
n_max = mdl_eval_cache[imax, 0]
l_max = mdl_eval_cache[imax, 1]
fit_max = mdl_eval_cache[imax, 2]

print('Best RMSE           = {0:.2f}'.format(fit_max))
print('n*                  = {0:.0f}'.format(n_max) )
print('lam*                = {0:.1f}'.format(l_max) )

ax.scatter(mdl_eval_cache[:,0], np.log10(mdl_eval_cache[:,1]), mdl_eval_cache[:,2])
ax.set_xlabel('n (model order)')
ax.set_ylabel('log(lambda)')
ax.set_zlabel('F')
plt.xticks(np.array([2,3,4]))
plt.yticks(np.array([0, 2, 4]))
plt.show()
```

### Improvement: filtering the I/O data

In order the improve the modelling performance, we may do [the following](https://people.mech.kuleuven.be/~jswevers/h04x3a/lecture_c2_c3.pdf):

- Excite the system only within the frequency band of interest
- Filter the output using a low-pass filter (e.g., a Butterworth LPF)

```python
def butterworth_lowpass(cutoff, fs, order=5):
    b, a = butter(order, 2 * cutoff / fs, btype='low', analog=False)
    return b, a

def butterworth_lowpass_filter(data, cutoff, fs, order=5):
    b, a = butterworth_lowpass(cutoff, fs, order=order)
    y = lfilter(b, a, data)
    return y

order  = 6      # filter order
fs     = 10     # sample rate, Hz
cutoff = 3.0    # desired cutoff frequency of the filter, Hz

# Obtain the filter coefficients
b, a = butterworth_lowpass(cutoff, fs, order)

# Filter y
y_filt = butterworth_lowpass_filter(y, cutoff, fs, order)

plot_time = np.min([1000, num_data-1])
plt.figure(figsize=(8, 4))
plt.plot(time_span[1:plot_time], y[1:plot_time], color='r')
plt.plot(time_span[1:plot_time], y_filt[1:plot_time], linewidth=2.5, color='b')
plt.legend(['original', 'filtered'])
plt.show()
```

### Statistical tests

Have we derived a model which performs well against test data (it has a high fit), we may/should ask whether this is simple enough (whether equally high performance may be obtained by a simpler model), whether the modeling residuals are normally distributed or what their distribution might be and whether the innovations $e(t)$ are independent.

```python
# Select model to test its residuals...
#best_model = mdl_sys_cache[imax,0]
best_model = actual_system

# Pick an input signal
num_data_test  = 1000
time_span_test = np.linspace(0, num_data_test, num_data_test)
u_test         = 5 * np.sin(0.5*time_span_test)

y_actual_test = actual_system.simulate(np.array([0,0,0]), u_test)
y_mdl_test    = best_model.simulate(np.array([0,0,0]), u_test)

residuals    = y_actual_test - y_mdl_test

plt.plot(time_span_test, residuals[:-1])
plt.xlabel('time')
plt.ylabel('residuals')
plt.show()

# --- QQ plot of modeling residuals
#     We want to see (quite literally) whether the residuals
#     come from a normal distribution
import pylab
import scipy.stats as stats
stats.probplot(residuals, dist="norm", plot=pylab)
pylab.show()

# --- Hypothesis testing: are the residuals normally distributed?
#     We use the Kolmogorov-Smirnov statistical test
print(stats.kstest(residuals, 'norm'))

```

## Endnotes

1. Not always is it the case that we are able to experiment (indefinitely) with a dynamical system. It is often the case that we may only obtain a certain amount of open-loop input-output data. In such cases, [cross-validation] approaches may be more fit (no pun intended).
2. Often in practice we encounter situations where a system cannot be excitated in open loop due to technical constraints and/or because it is open-loop unstable. In closed-loop settings, the excitation is a function of the system output.  These situations call for closed-loop system identification methodologies which are not covered here.

## Further reading

- AR(I)MA(X) models
- (Partial) autocorrelation functions - uncorrelatedness tests
- Maximum likelihood estimation
- Subspace identification


13.

Computer Lab – Chemical Kinetics 1

Goal: The purpose of this lab is to become familiar with modeling a simple ligand binding reaction.

Introduction

Consider a binding reaction where ligand ($L$) binds to a protein receptor ($R$) to form a loosely bound encounter complex ($L\cdot R$), and finally a tightly bound product ($P$) with the forward and backward rate constants $k_{f}$ and $k_{b}$, respectively.1

$$L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R \stackrel{k_{f2}}{\rightarrow} P \tag{1}$$

First, we need to write out the elementary rate equations:

\begin{align*}
    \\
    \frac{d[L]}{dt} &= \frac{d[R]}{dt} = -k_{f1}[L][R] + k_{b1}[L\cdot R] \tag{1.1}\\
    \frac{d[L\cdot R]}{dt} &= k_{f1}[L][R] -k_{b1}[L\cdot R] - k_{f2}[L\cdot R] \tag{1.2}\\
    \frac{d[P]}{dt} &=  k_{f2}[L\cdot R] \tag{1.3}
    \\
\end{align*}

Next, we apply a steady state approximation on $d[L\cdot R]/dt$:

\begin{equation}
k_{b1}[L\cdot R] = k_{f1}[L][R] - k_{f2}[L\cdot R]
 \implies \boxed{[L\cdot R] = \left( \frac{k_{f1}}{k_{b1}+k_{f2}} [L][R]\right)}
\end{equation}

Now, we take $[L\cdot R]$ and substitute into equations ${1.1}$ and ${1.3}$.

\begin{align}
    \frac{d[L]}{dt} &= \frac{d[R]}{dt} = -k_{f1}[L][R] + k_{b1}\left(\frac{k_{f1}}{k_{b1}+k_{f2}} [L][R] \right) \tag{1.1a}\\
    \frac{d[P]}{dt} &=  k_{f2} \left( \frac{k_{f1}}{k_{b1}+k_{f2}} [L][R]\right) \tag{1.3a}
\end{align}

Note that this last kinetic step is “one-way”, which violates thermodynamics (!). Don’t worry about this yet (we’re getting to kinetics soon in the lecture). For now, suffice it to say that here we invoke the assumption that the back rate is very small compared to forward rate.↩

Let’s walk through the steps of how to build a model of this receptor-ligand reaction using Python.

Procedure

Part I.

Set initial concentration of ligand to 10.0 and receptor to 5.0

Set rate_eqs=ODE_eqs_part_I

set “Forward Rate Parameter” ($k_{f1}$) value to 0.02

set “Reverse Rate Parameter” ($k_{b1}$) value to 0.01

Run the simulation and save the graph.

Part II.

Repeat simulation for following values:

Add "k-prod" ($k_{f2}$) and set value to 0.05.

Set rate_eqs=ODE_eqs_part_II

Run the simulation and save the graph.

Does the reaction reach a steady state?

If not, increase simulation time to 100

Repeat simulation for following values:

“Forward Rate Parameter” ($k_{f1}$): 0.01 , “Reverse Rate Parameter” ($k_{b1}$): 0.02 , “k-prod” ($k_{f2}$): 0.05

“Forward Rate Parameter” ($k_{f1}$): 0.02, “Reverse Rate Parameter” ($k_{b1}$): 0.01 , “k-prod” ($k_{f2}$): 0.5

```python
import numpy as np
import pandas as pd
```

```python
def ODE_eqs_part_I(C, t, k):
    r"""
    :math:`L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R`

    Args:
        C(list): collection of concentrations for each species [L, R, LR]
        t(float): iterate over some time-course
        k(list): collection of rate constants [kf1, kb1]
    """

    kf1, kb1 = k[0], k[1]
    dC1dt = -kf1*C[0]*C[1]+kb1*C[2]
    dC0dt = dC1dt
    dC2dt = -dC1dt
    return [dC0dt, dC1dt, dC2dt]

def ODE_eqs_part_II(C, t, k):
    r"""See eqs 1.1, 1.2 and 1.3
    :math:`L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R \stackrel{k_{f2}}{\rightarrow} P`

    Args:
        C(list): collection of concentrations for each species [L, R, LR, P]
        t(float): iterate over some time-course
        k(list): collection of rate constants [kf1, kb1, kf2]
    """

    kf1, kb1, kf2 = k[0], k[1], k[2]
    dC1dt = -kf1*C[0]*C[1]+kb1*C[2]
    dC0dt = dC1dt
    dC2dt = k[0]*C[0]*C[1]-k[1]*C[2]-k[2]*C[2]
    dC3dt = kf2*C[2]
    return [dC0dt, dC1dt, dC2dt, dC3dt]

def ODE_eqs_part_II_SS(C, t, k):
    r"""See eqs 1.2, 1.1a and 1.3a
    Using the steady state approximation on dLR/dt.
    :math:`L + R \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} L\cdot R \stackrel{k_{f2}}{\rightarrow} P`

    Args:
        C(list): collection of concentrations for each species [L, R, LR, P]
        t(float): iterate over some time-course
        k(list): collection of rate constants [kf1, kb1, kf2]
    """

    kf1, kb1, kf2 = k[0], k[1], k[2]
    dC1dt = -kf1*C[0]*C[1]+kb1*(kf1/(kb1+kf2)*C[0]*C[1])
    dC0dt = dC1dt
    dC2dt = k[0]*C[0]*C[1]-k[1]*C[2]-k[2]*C[2]
    dC3dt = kf2*(kf1/(kb1+kf2)*C[0]*C[1])
    return [dC0dt, dC1dt, dC2dt, dC3dt]

def simulate(steps, simulation_time, parameters, rate_eqs):
    """Method for integrating the ODEs with SciPy odeint
    https://docs.scipy.org/doc/scipy/reference/generated/scipy.integrate.odeint.html

    Args:
        steps(int): number of iterations
        simulation_time(int): duration of simulation (units: s)
        parameters(dict): {'species': ['L', 'R', 'LR'],\
                          'k': [kf1, kb1, kf2, kb2, ...],\  # rate constants
                          'InitConc': [L, R, P, ...]}       # initial concentrations (units: M)
        rate_eqs(function): method containing the ordinary differential equations
    Returns:
        Pandas DataFrame, plot
    """

    from scipy.integrate import odeint
    print(f"Number of species: {len(parameters['InitConc'])}")
    time = np.linspace(0, simulation_time, steps)
    C = odeint(rate_eqs, parameters["InitConc"], time, (parameters["k"],)).transpose()
    data = {"time": time, **{"%s"%s: C[i] for i,s in enumerate(parameters["species"])}}
    df = pd.DataFrame(data)
    plot = df.plot(x="time", y=parameters["species"], figsize=(10, 4),
                   xlim=(data["time"][0], data["time"][-1]))
    return df,plot

```

```python
parameters = {"species": ["L","R","LR"],
             "InitConc": [10, 5, 0],
              #          [kf1, kb1, kf2]
             "k":        [0.02, 0.01, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=10,
                   parameters=parameters, rate_eqs=ODE_eqs_part_I)
plot = plot.set_title(r"$L + R ⥂ L\cdot R$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
parameters = {"species": ["L","R", "LR", "P"],
             "InitConc": [10, 5, 0, 0],
             #            [kf1, kb1, kf2, ...]
             "k":        [0.02, 0.01, 0.05, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=100,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II)
plot = plot.set_title(r"$L + R ⥂ L\cdot R \rightarrow P$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
parameters = {"species": ["L","R", "LR", "P"],
             "InitConc": [10, 5, 0, 0],
             #            [kf1, kb1, kf2, ...]
             "k":        [0.02, 0.01, 0.05, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=100,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II_SS)
plot = plot.set_title(r"""Steady State
$L + R ⥂ L\cdot R \rightarrow P$""")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
#“Forward Rate Parameter” ( 𝑘𝑓1 ): 0.01 , “Reverse Rate Parameter” ( 𝑘𝑏1 ): 0.02 , “k-prod” ( 𝑘𝑓2 ): 0.05
parameters = {"species": ["L","R", "LR", "P"],
             "InitConc": [10, 5, 0, 0],
             #           [kf1, kb1, kf2, ...]
             "k":        [0.01, 0.02, 0.05, "NaN"]}
df,plot = simulate(steps=1000, simulation_time=100,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II)
plot = plot.set_title(r"$L + R ⥂ L\cdot R \rightarrow P$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
#“Forward Rate Parameter” ( 𝑘𝑓1 ): 0.02, “Reverse Rate Parameter” ( 𝑘𝑏1 ): 0.01 , “k-prod” ( 𝑘𝑓2 ): 0.5
parameters = {"species": ["L","R", "LR", "P"],
             "InitConc": [10, 5, 0, 0],
             "k":        [0.02, 0.01, 0.5, "NaN"]}
df,plot = simulate(steps=1000, simulation_time=100,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II)
plot = plot.set_title(r"$L + R ⥂ L\cdot R \rightarrow P$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

Bonus

This same reaction can be used to describe Michaelis-Menten enzyme kinetics, in which substrate (S) binds a protein enzyme (E) to form an encounter complex (E∙S), and then chemically converts to product (E + P).

$$E + S \underset{k_{b1}}{\stackrel{k_{f1}}{⥂}} E\cdot S \stackrel{k_{cat}}{\to} E + P$$

Suppose an enzyme has kf1 = 107 s-1, kb1 = 106 s-1 and kcat = 104 s-1. By adjusting parameters in the function above, which of these rates constants would you say most sensitively controls the overall rate of catalysis?

### Answer: $k_{cat}$. See plots below...

```python
parameters = {"species": ["E","S", "ES", "P"],
             "InitConc": [10, 5, 0, 0],
             #            [kf1, kb1, kf2, ...]
             "k":        [1e7, 1e6, 1e4, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=1e-7,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II)
plot = plot.set_title(r"$L + R ⥂ L\cdot R \rightarrow P$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
parameters = {"species": ["E","S", "ES", "P"],
             "InitConc": [10, 5, 0, 0],
             #            [kf1, kb1, kf2, ...]
             "k":        [1e7, 1e6, 1e8, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=1e-7,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II)
plot = plot.set_title(r"$L + R ⥂ L\cdot R \rightarrow P$")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python
parameters = {"species": ["E","S", "ES", "P"],
             "InitConc": [10, 5, 0, 0],
             #            [kf1, kb1, kf2, ...]
             "k":        [1e7, 1e6, 1e8, "NaN"]}
#print(pd.DataFrame(parameters))
df,plot = simulate(steps=1000, simulation_time=1e-7,
                   parameters=parameters, rate_eqs=ODE_eqs_part_II_SS)
plot = plot.set_title(r"""Steady State
$L + R ⥂ L\cdot R \rightarrow P$""")
fig = plot.get_figure()
#fig.savefig("figure.png")
df
```

```python

```