# CBE 60553, Fall 2017, Homework 1

## Problem 1: Choose your path wisely

A particular system has the equation of state $U = \frac{5}{2} PV + C$, where $C$ is an undetermined constant.

### 1. The system starts at state $A$, in which $P={0.2}\ {MPa}$ and $V = {0.01}\ {m^{3}}$. It is taken quasistatically along the path shown in the figure ($A \rightarrow B$, $B \rightarrow C$, $C \rightarrow A$ ).  Calculate the heat transferred from the surroundings, $q$, and the work done on the system, $w$, for each step along the path.

#### $i$) $A \rightarrow B$
$$U_{AB} = U_B - U_A = \frac{5}{2} (P_B V_B - P_A V_A) = {10000}\ J$$
$$W_{AB} = - \int_{V_A}^{V_B}P dV = -P(V_B - V_A) = -4000\ J$$
$$\therefore \ Q_{AB} = U_{AB} - W_{AB} = 14000\ J$$

#### $ii$) $B \rightarrow C$
Need pressure as a function of volume along this path. From the figure, the relationship is linear and given by

$$ P(V)= -15 \times 10^{6} V + 0.65 \times 10^{6} $$

Integrate to find the work
$$W_{BC} = - \int_{V_B}^{V_C}P dV = -\left[\frac{-15 \times 10^{6} V^{2}}{2} + 0.65 \times 10^{6} V \right]_{V_B}^{V_C} = 7000\ J$$

From our expression for U
$$U_{BC} = U_C - U_B = \frac{5}{2} (P_C V_C - P_B V_B) = -2500 \ J$$

$$\therefore \ Q_{BC} = U_{BC} - W_{BC} = -9500\ J$$

#### $iii$) $C \rightarrow A$
$$U_{CA} = U_A - U_C = \frac{5}{2} (P_A V_A - P_C V_C) = -7500\ J$$
Since volume is constant
$$W_{CA} = - \int_{V_C}^{V_A}P dV = 0 $$

$$\therefore \ Q_{CA} = U_{CA} - W_{CA} = -7500\ J$$

### 2. Calculate $q$ and $w$ for a quasistatic process starting at $A$ and ending at $B$ along the path $P=a + b(V-c)^{2}$, where $a = {0.1}\ {MPa}$, $b= 1 \times 10^{3}\ {MPa \cdot m^{-6}}$, and $c = {0.02}\ {m^{3}}$.

$A \rightarrow B$

Along the Parabola
$$ P = 10^{5} + 10^{9} \times (V-0.02)^{2} $$
the work can be found by integration

$$W_{AB} = - \int_{V_A}^{V_B}P dV = - \int_{V_A}^{V_B}\left[10^{5} + 10^{9} \times (V-0.02)^{2} \right] dV = -\left[10^{5} V + \frac {10^{9}} {3} (V-0.02)^{3} \right]_{0.01}^{0.03} = -2666.67\ J$$

Since
$$ U_{AB} = 10000\ J $$
then
$$ Q_{AB} = U_{AB} - W_{AB} = 10000\ J - (-2666.67\ J) = 12666.67\ J$$

### 3. The system exchanges both heat and work with its surroundings along the paths above. An /adiabat/ is a particular quasistatic path along which work is done but no heat is transferred.  Find the form of the adiabats $P=P(V)$ for the system described by $U = \frac{5}{2} PV +C$.  (Hint: If $\bar{d}q_\text{qs} = 0$, then $dU = \bar{d} w_\text{qs} = -PdV$.  What else does $dU$ equal?)

For an adiabatic system,
$$ dU = dQ - PdV = -PdV $$

and we can also write

$$ dU = \left. \frac{\partial U}{\partial V} \right|_{P} dV + \left. \frac{\partial U}{\partial P} \right|_{V} dP = 2.5PdV +
2.5VdP = -PdV $$

$$ \frac {7}{V} dV = - \frac{5}{P}dP $$

$$ \left. ln{V^{7}} \right|_{V_{0}}^{V} = -\left. ln{P^{5}} \right|_{P_{0}}^{P} $$

$$ ln{P^{5}V^{7}} = C'\ (C' = const)$$

$$ P^{5}V^{7} = C\ (C = const)$$

## Problem 2: Is it fundamental enough?

The following ten equations are purported to be fundamental equations for
various thermodynamic systems. Five, however, are inconsisent with the basic
  postulates of a fundamental equation and are thus unphysical.  For each, plot
  the relationship between $S$ and $U$ and identify the five that are
  unacceptable. $v_0$, $\theta$, and $R$ are all positive constants and, in the
  case of fractional exponents, the real positive root is to be implied.

$ (1)\ S = \left ( \frac{R^2}{v_0\theta} \right )^{1/3}\left ( NVU \right
)^{1/3}\hspace{20pt}
(2)\ S = \left ( \frac{R}{\theta^2} \right )^{1/3}\left ( \frac{NU}{V} \right)^{2/3} $

$ (3)\ S = \left ( \frac{R}{\theta} \right )^{1/2}\left ( NU + \frac{R\theta V^2}{v_0^2} \right)^{1/2}\hspace{20pt}
(4)\ S = \left ( \frac{R^2\theta}{v_0^3} \right ) \frac{V^3}{NU} $

$ (5)\ S = \left ( \frac{R^3}{v_0\theta^2} \right )^{1/5}\left ( N^2U^2V \right)^{1/5}\hspace{20pt}
(6)\ S = NR \ln \left ( \frac{UV}{N^2 R \theta v_0}  \right ) $

$ (7)\ S = \left ( \frac{NRU}{\theta} \right )^{1/2}\exp \left (-\frac{V^2}{2N^2v_0^2} \right )\hspace{20pt}
(8)\ S = \left ( \frac{NRU}{\theta} \right )^{1/2}\exp
\left (-\frac{UV}{NR\theta v_0} \right ) $

$ (9)\ U = \left ( \frac{NR\theta V}{v_0} \right ) \left ( 1+\frac{S}{NR} \right ) \exp \left (-S/NR \right)\hspace{20pt}
(10)\ U = \left ( \frac{v_0\theta}{R} \right ) \frac{S^2}{V} \exp\left ( S/NR \right) $

There are three postulates we are testing for

$(i)\  S(\lambda U,\lambda V, \lambda N) = \lambda S(U,V,N)$ : Postulate 3

$(ii)\  \frac{\partial S}{\partial U} > 0 $  : Postulate 2

$ (iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$ : Postulate 4

We assume $v_{0} = 1$, $R = 1$, $\theta = 1$, and $N$ and $V$ are constants.

$(1)\ S = \left ( \frac{R^2}{v_0\theta} \right )^{1/3}\left ( NVU \right)^{1/3} = \left (NVU \right)^{1/3}$

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = (\lambda^{3}NVU)^{1/3} = \lambda \cdot NVU = \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} > 0 $

$\hspace{10pt}$ $(iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(1)$ is acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0,2,100)
S = []
for u in U:
    s = u **(1./3)  # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(2)\ S = \left ( \frac{R}{\theta^2} \right )^{1/3}\left ( \frac{NU}{V} \right)^{2/3} = \left ( \frac{NU}{V} \right)^{2/3} $

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \left (\lambda \frac{NU}{V}\right)^{2/3} = \lambda^{2/3} \left(\frac{NU}{V}\right) ^{2/3} \neq \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} > 0 $

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(2)$ is not acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0,2,100)
S = []
for u in U:
    s = u **(2./3)  # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(3)\ S = \left ( \frac{R}{\theta} \right )^{1/2}\left ( NU + \frac{R\theta V^2}{v_0^2} \right)^{1/2} = \left ( NU + V^2 \right)^{1/2}$

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \left (\lambda^2 NU + \lambda^2 V^2\right)^{1/2} = \lambda \left(NU + V^2\right) ^{1/2} = \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} > 0 $

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(3)$ is acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(-1,2,100)
S = []
for u in U:
    s = (u + 1**2)**(1./2)  # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(4)\ S = \left ( \frac{R^2\theta}{v_0^3} \right ) \frac{V^3}{NU} = \frac{V^3}{NU} $

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \lambda \frac{V^3}{NU}
 = \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} < 0 $

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} \neq 0,\ as\ S \rightarrow 0$

$\therefore$ $(4)$ is not acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0.1,2,100)
S = []
for u in U:
    s = (1**3) / (1 * u)  # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(5)\ S = \left ( \frac{R^3}{v_0\theta^2} \right )^{1/5}\left ( N^2U^2V \right)^{1/5} = \left (N^2 U^2 V \right)^{1/5} $

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \lambda \left (N^2 U^2 V \right)^{1/5} = \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} > 0 $

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(5)$ is acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0,2,100)
S = []
for u in U:
    s = (u**2)**(1./5)  # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(6)\ S = NR \ln \left ( \frac{UV}{N^2 R \theta v_0} \right) = N \ln \left ( \frac{UV}{N^2} \right)$

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \lambda N \ln \left (\frac{UV}{N^2} \right) = \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} > 0 $

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} \neq 0,\ as\ S \rightarrow 0$

$\therefore$ $(6)$ is not acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0.01,2,100)
S = []
for u in U:
    s = np.log(u)  # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(7)\ S = \left ( \frac{NRU}{\theta} \right )^{1/2}\exp \left (-\frac{V^2}{2N^2v_0^2} \right) = \left (NU \right )^{1/2}\exp \left (-\frac{V^2}{2N^2} \right)$

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \lambda \left (NU \right )^{1/2}\exp \left (-\frac{V^2}{2N^2} \right)= \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U} > 0 $

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(7)$ is acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0,2,100)
S = []
for u in U:
    s = (u**(0.5)) * np.exp(-0.5) # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(8)\ S = \left ( \frac{NRU}{\theta} \right )^{1/2}\exp
\left (-\frac{UV}{NR\theta v_0} \right) = \left (NU \right )^{1/2}\exp \left (-\frac{UV}{N} \right)$

$\hspace{10pt}$ $(i)\ S(\lambda U,\lambda V, \lambda N) = \lambda \left (NU \right )^{1/2}\exp \left (-\lambda \frac{UV}{N} \right) \neq \lambda S(U,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U}$ is not monotonically increasing.

$\hspace{10pt}$ $(iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(8)$ is not acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

U = np.linspace(0,2,100)
S = []
for u in U:
    s = (u**(0.5)) * np.exp(-u) # assume N = 1 and V = 1
    S.append(s)

plt.plot(U, S,'-')
plt.xlabel('U')
plt.ylabel('S')
plt.show()
```

$(9)\ U = \left ( \frac{NR\theta V}{v_0} \right ) \left ( 1+\frac{S}{NR} \right ) \exp \left (-\frac{S}{NR} \right)= \left ( NV \right ) \left ( 1+\frac{S}{N} \right ) \exp \left (-\frac{S}{N} \right)$

$\hspace{10pt}$ $(i)\ U(\lambda S,\lambda V, \lambda N) = \lambda^{2} \left (NV \right ) \left(1 + \frac{S}{N}\right) \exp \left (-\frac{S}{N} \right) \neq \lambda U(S,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U}$ is not monotonically increasing.

$\hspace{10pt}$ $ (iii)$ assume $N = 1$ and $V = 1$,

$\hspace{26pt}$ then $ \frac{\partial U}{\partial S} = \exp \left(-S \right) - \exp \left(-S \right)(1 + S) $

$\hspace{26pt}$ thus, $ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(9)$ is not acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

S = np.linspace(0,2,100)
U = []
for s in S:
    u = (1 + s) * np.exp(-s) # assume N = 1 and V = 1
    U.append(u)

plt.plot(S, U,'-')
plt.xlabel('S')
plt.ylabel('U')
plt.xlim(0,2)
plt.ylim(0,1.2)
plt.show()
```

$(10)\ U = \left ( \frac{v_0\theta}{R} \right ) \frac{S^2}{V} \exp\left (\frac{S}{NR} \right) = \frac{S^2}{V} \exp\left (\frac{S}{N} \right) $

$\hspace{10pt}$ $(i)\ U(\lambda S,\lambda V, \lambda N) = \lambda \frac{S^2}{V}
\exp \left (\frac{S}{N} \right) = \lambda U(S,V,N) $

$\hspace{10pt}$ $(ii)\  \frac{\partial S}{\partial U}$ > 0.

$\hspace{10pt}$ $ (iii)\ \frac{\partial U}{\partial S} = 0,\ as\ S \rightarrow 0$

$\therefore$ $(10)$ is acceptable.

```python
import matplotlib.pyplot as plt
import numpy as np

S = np.linspace(0,2,100)
U = []
for s in S:
    u = (s**(2)) * np.exp(s) # assume N = 1 and V = 1
    U.append(u)

plt.plot(S, U,'-')
plt.xlabel('S')
plt.ylabel('U')
plt.xlim(0,2)
plt.show()
```

Therefore, (2),(4),(6),(8) and (9) are not physically permissible.

## Problem 3: Find your equilibrium

The fundamental equations of both systems $A$ and $B$ are

$$ S = \left (
\frac{R^2}{v_0\theta} \right )^{1/3} \left ( N V U \right )^{1/3} $$

The volume and mole number of system $A$ are $ 9 \times 10^{-6}\ m^3 $ and $3$ mol, respectively, and of system $B$ are $ 4 \times 10^{-6}\ m^3 $ and $2$ mol,
respectively.  First suppose $A$ and $B$ are completely isolated from one
another.  Plot the total entropy $S_A + S_B$ as function of $U_A/(U_A + U_B)$,
where $U_A + U_B = 80$ J. If $A$ and $B$ were connected by a diathermal wall and
the pair allowed to come to equilibrium, what would $U_A$ and $U_B$ be?

Call
$$ X = \frac{U_A}{U_A + U_B}$$

we know $U_A + U_B = 80$, therefore

$$ U_A = 80X,\hspace{20pt} U_B = 80(1 - X) $$

Then setting $R, v_0, \theta = 1 $ and plugging in $V_A$, $V_B$, $N_A$ and $N_B$.

$S = S_A + S_B = \left(3 \times 9 \times 10^{-6} \times 80X \right)^{1/3} + \left(2 \times 4 \times 10^{-6} \times 80(1-X)\right)^{1/3} = 0.086(1-X)^{1/3} + 0.129X^{1/3}$

Entropy is maximized when $X = 0.65$, which is where we would expect the system to go at equilibrium once the internal wall is made diathermal.

```python
import matplotlib.pyplot as plt
import numpy as np

X = np.linspace(0,1,100)
S = []
for x in X:
    s = 0.086 * (1 - x)**(1./3) + 0.129 * (x**(1./3))
    S.append(s)

plt.plot(X, S,'-')
plt.xlabel('X')
plt.ylabel('S')
plt.show()
```

From this graph, S is maximized when $X = 0.65$.

Therefore, $U_A = 80X = 52\ J$ and $U_B = 28\ J$.

An alternative non-graphical method is to solve for where

$$ \frac{\partial S}{\partial U} = 0 $$

```python
from sympy import *
X = Symbol('X', real = True)
S = 0.086 * (1 - X)**(1./3) + 0.129 * (X**(1./3))

Sprime =  S.diff(X) # differentiate S in terms of X

max = solve(Sprime, X) # solve Sprime =0 with respect to X

print 'X =', max[0]
print 'UA =', 80 * max[0]
print 'UB =', 80 * (1 - max[0])
```

    X = 0.647529554910575
    UA = 51.8023643928460
    UB = 28.1976356071540

## Problem 4: Exactly right

The Helmholtz energy $A$ is a thermodynamic state function.  Show that

$ \left (\frac{\partial A}{\partial V}\right )_T = - P $  and  $ \left(\frac{\partial A}{\partial T}\right )_V = - S\ $
implies $ \left (\frac{\partial S}{\partial V}\right )_T = \left
  (\frac{\partial P}{\partial T}\right )_V $

$$ dA = \left (\frac{\partial A}{\partial V}\right)_{T} dV + \left (\frac{\partial A}{\partial T}\right)_{V} dT $$

$$ \left. \frac {\partial}{\partial T} \left(\frac {\partial A}{\partial V} \right)_T \right|_V = \left. \frac {\partial}{\partial V} \left(\frac {\partial A}{\partial T} \right)_V \right|_T $$

$$ \therefore \ \left. \frac{\partial (-P)}{\partial T} \right |_{V} = \left. \frac{\partial (-S)}{\partial V} \right |_{T} $$

## Problem 5: A difference of degree

Determine whether the following five expressions are homogeneous and, if so, what their degree of homogeneity is:

$ (1)\ u=x^2y + xy^2 +3xyz $

$ (2)\ u=\sqrt{x+y} $

$ (3)\ u=\frac{x^3+x^2y+y^3}{x^2+xy+y^2} $

$ (4)\ u=e^{-y/x} $

$ (5)\ u=\frac{x^2+3xy+2y^3}{y^2} $

$(1)\ u(\lambda x,\lambda y, \lambda z) = \lambda^{3} \left(x^2y + xy^2 +3xyz \right) = \lambda^{3} u(x,y,z) $

$ \therefore$ $u$ is homogeneous and the degree of homogeneity is 3.

$(2)\ u(\lambda x,\lambda y, \lambda z) = \lambda^{1/2} \sqrt{x + y} = \lambda^{1/2} u(x,y,z) $

$ \therefore$ $u$ is homogeneous and the degree of homogeneity is 1/2.

$(3)\ u(\lambda x,\lambda y, \lambda z) = \lambda \frac{x^3 + x^2 y + y^3}{x^2 + xy + y^2} = \lambda u(x,y,z) $

$ \therefore$ $u$ is homogeneous and the degree of homogeneity is 1.

$(4)\ u(\lambda x,\lambda y, \lambda z) = e^{-y/x} = u(x,y,z) $

$ \therefore$ $u$ is homogeneous and the degree of homogeneity is 0.

$(5)\ u(\lambda x,\lambda y, \lambda z) = \frac{x^2+3xy+2\lambda y^3}{y^2} $

$ \therefore$ $u$ is not homogeneous.


# Typesetting Math in your book

Jupyter Book uses [MathJax](http://docs.mathjax.org/) for typesetting math in your
book. This allows you to have LaTeX-style mathematics in your online content.
This page shows you a few ways to control this.

For more information about equation numbering, see the
[MathJax equation numbering documentation](http://docs.mathjax.org/en/v2.7-latest/tex.html#automatic-equation-numbering).

# In-line math

To insert in-line math use the `$` symbol within a Markdown cell.
For example, the text `$this_{is}^{inline}$` will produce: $this_{is}^{inline}$.

# Math blocks

You can also include math blocks for separate equations. This allows you to focus attention
on more complex or longer equations, as well as link to them in your pages. To use a block
equation, wrap the equation in either `$$` or `\begin` statements.

For example,

```latex
\begin{equation}
  \int_0^\infty \frac{x^3}{e^x-1}\,dx = \frac{\pi^4}{15}
\end{equation}
```
results in

\begin{equation}
  \int_0^\infty \frac{x^3}{e^x-1}\,dx = \frac{\pi^4}{15}
\end{equation}

and

```latex
$$
  \int_0^\infty \frac{x^3}{e^x-1}\,dx = \frac{\pi^4}{15}
$$
```

results in

$$
  \int_0^\infty \frac{x^3}{e^x-1}\,dx = \frac{\pi^4}{15}
$$

## Numbering equations

If you'd like to number equations so that you can refer to them later, use the **math directive**.
It looks like this:

````
```{math}
:label: my_label
my_math
```
````

For example, the following code:

````
```{math}
:label: my_label
w_{t+1} = (1 + r_{t+1}) s(w_t) + y_{t+1}
```
````

Will generate this:

```{math}
:label: my_label
w_{t+1} = (1 + r_{t+1}) s(w_t) + y_{t+1}
```

## Linking to equations

If you've created an equation with a label, you can link to it from within your text
(and across pages!).

You can refer to the equation using the label that you've provided by using
the **`{eq}` role**. For example, putting `` {eq}`my_label` `` in-line will
result in this: {eq}`my_label`.

```python

```


Table of Contents

Introduction

Exploratory Data Analysis

Visualise the data

Supervised learning

Split the data into training and test set

Fit the model

Side note: intuition of the model

Fitting the model

Issues with plotting

Extra: verify the results of the model

# Introduction
I will explore the diabetes dataset from scikit-learn and see if we can apply any machine learning algorithms on the dataset. My aim is to be exposed to a range of different datasets from different sources and evaluate the merit in applying machine learning to these datasets.

```python
import numpy as np
import pandas as pd

import matplotlib.pyplot as plt

from sklearn import datasets, linear_model
from sklearn.model_selection import train_test_split

%matplotlib inline
plt.style.use('ggplot')
```

# Exploratory Data Analysis
We will look at the data initially to examine it and look at the input features, and the target variable.
Following this we should be in a better position to carry out some Machine Learning algorithm on the data, and evaluate its merit after doing so.

```python
diabetes_data = datasets.load_diabetes()

diabetes_data.keys()
```

    dict_keys(['data', 'target', 'DESCR', 'feature_names', 'data_filename', 'target_filename'])

```python
print(diabetes_data['DESCR'])
```

    .. _diabetes_dataset:

    Diabetes dataset
    ----------------

    Ten baseline variables, age, sex, body mass index, average blood
    pressure, and six blood serum measurements were obtained for each of n =
    442 diabetes patients, as well as the response of interest, a
    quantitative measure of disease progression one year after baseline.

    **Data Set Characteristics:**

      :Number of Instances: 442

      :Number of Attributes: First 10 columns are numeric predictive values

      :Target: Column 11 is a quantitative measure of disease progression one year after baseline

      :Attribute Information:
          - Age
          - Sex
          - Body mass index
          - Average blood pressure
          - S1
          - S2
          - S3
          - S4
          - S5
          - S6

    Note: Each of these 10 feature variables have been mean centered and scaled by the standard deviation times `n_samples` (i.e. the sum of squares of each column totals 1).

    Source URL:
    https://www4.stat.ncsu.edu/~boos/var.select/diabetes.html

    For more information see:
    Bradley Efron, Trevor Hastie, Iain Johnstone and Robert Tibshirani (2004) "Least Angle Regression," Annals of Statistics (with discussion), 407-499.
    (https://web.stanford.edu/~hastie/Papers/LARS/LeastAngle_2002.pdf)

We have already been told that the features have been scaled (normalised and standardised), therefore there is no need to feature scale this data.

```python
features_df = pd.DataFrame(
    data=diabetes_data['data'],
    columns=diabetes_data['feature_names'])
target = diabetes_data['target']
```

```python
features_df.head()
```

      age
      sex
      bmi
      bp
      s1
      s2
      s3
      s4
      s5
      s6

      0
      0.038076
      0.050680
      0.061696
      0.021872
      -0.044223
      -0.034821
      -0.043401
      -0.002592
      0.019908
      -0.017646

      1
      -0.001882
      -0.044642
      -0.051474
      -0.026328
      -0.008449
      -0.019163
      0.074412
      -0.039493
      -0.068330
      -0.092204

      2
      0.085299
      0.050680
      0.044451
      -0.005671
      -0.045599
      -0.034194
      -0.032356
      -0.002592
      0.002864
      -0.025930

      3
      -0.089063
      -0.044642
      -0.011595
      -0.036656
      0.012191
      0.024991
      -0.036038
      0.034309
      0.022692
      -0.009362

      4
      0.005383
      -0.044642
      -0.036385
      0.021872
      0.003935
      0.015596
      0.008142
      -0.002592
      -0.031991
      -0.046641

## Visualise the data
Before undertaking an machine learning project, it helps to visualise the target values, to get an intuition for the type of data we are looking at.

```python
plt.figure(figsize=(15,12))
plt.plot(target)
plt.ylabel("Target Variable")
plt.xlabel("Observation Number")
```

This line plot above isn't too intuitive about the nature of the data. It might be more helpful to plot a histogram to get an idea of the distribution of the data.

```python
plt.hist(target)
```

# Supervised learning
Since we have both the input data and the response variables, my first port of call would be to look at supervised learning, and see if we can fit a model to the data, to best predict whether or not the model works.

### Split the data into training and test set
In order to train a model we must split the dataset into a training set (train the model using this), and a test set (on which we can verify the accuracy of the model).
Therefore, as a rule we will use 80% of the data for training, and the other 20% for testing.

We will use the *train_test_split* method of *scikit-learn*, which ensures that the data is split randomly, to remove any bias that might exist in the ordering, and the split the sample into the training and testing set.

```python
X, y = diabetes_data['data'], diabetes_data['target']
# a, b = datasets.load_diabetes(return_X_y=True)
```

```python
X_train, X_test, y_train, y_test = train_test_split(
    X,
    y,
    test_size=0.3,
    random_state=10) # the random state will ensure we sample the same ones each time the notebook is run
```

```python
X_train.shape
```

    (309, 10)

So, we can see that we have 309 of the original observations in our training set, and complementing this will be the 309 response variables for those original observations.

### Fit the model

Now we will fit the model to the training set using linear regression.

```python
# create the linear regression model
reg = linear_model.LinearRegression()

# train the model on the training sets defined earlier
reg.fit(X_train, y_train)
```

    LinearRegression(copy_X=True, fit_intercept=True, n_jobs=None, normalize=False)

We can look at what the coefficients in our linear regression model are:

```python
reg.coef_
```

    array([  16.43761831, -223.55014379,  538.8442475 ,  274.22260478,
           -917.24014562,  378.94084512,  262.65687528,  451.75981325,
            711.38235372,  109.26789802])

So, the model looks roughly like: \
\begin{equation}
    response = 16.4 - 224 \cdot x_{1} + 539 \cdot x_{2} + 274 \cdot x_{3} + ...
\end{equation}
for $x_{1}$ as age, $x_2$ as sex, $x_3$ as body mass index.

Now we have trained the model using the test set, we want to test how good the model is at predicting the response variable, and we will do this using the test set.

### Side note: intuition of the model
Let's go into some detail about what the model is actually doing, and will do to predict the values of y in the test set.
From above, we see that the model has fitted certain values for the coefficients of each feature (from **reg.coef_**).
Define our hypothesis function with the parameter $\Theta$, where
\begin{align}
    \Theta &= \begin{bmatrix}
           \theta_{0} \\
           \theta_{2} \\
           \vdots \\
           \theta_{10}
         \end{bmatrix}
  \end{align}
  Each coefficient $\theta_{i}$ corresponds to a coefficient for one of the input features. Note that $\theta_{0}$ is included as the intercept term.

The prediction of the y_test values should be equal to:
\begin{align}
    y_{prediction} = (X_{test})^T \Theta
\end{align}

```python
reg.coef_.shape

```

    (10,)

```python
X_test.shape
```

    (133, 10)

```python
# numpy broadcasting to make matrix multiplication work
theta = reg.coef_[:, np.newaxis]
y_test_predictions = np.dot(X_test, theta)

 # this should be the same shape as the predictions using the LinearRegression class
y_test_predictions.shape
```

    (133, 1)

## Fitting the model
Now we will fit the model to the test data, using the coefficients from the linear regression, and assess their performance in the predictions. To do this we will make use of some of the methods in the metrics library of scikit-learn.

```python
y_predict = reg.predict(X_test)

from sklearn.metrics import mean_squared_error, r2_score

print(f"The mean squared error for the multiple linear regression is \t"
      f"{mean_squared_error(y_test, y_predict):.2f} \n \n"
      f"The r2_score, coefficient of determinations for the multiple linear regression is \t"
      f"{r2_score(y_test, y_predict):.2f}")
```

    The mean squared error for the multiple linear regression is 	3324.22

    The r2_score, coefficient of determinations for the multiple linear regression is 	0.47

We see that the mean_squared_error for the linear regression is quite high, indicating that the regression model is not a good one for predicting the response variable.

Further, we see that the R2 score, a measure of how well the model fits the data, is 0.47, which is quite low. Ideally from a ML model we would like to achieve an R2 score of around 0.8, and then we can confidently use the model to predict the response with a greater degree of confidence.

### Issues with plotting
It would be useful to see a visualisation of the output of the model we have fitted. However, since this is a multivariate linear regression model, we cannot plot the output of a 10-dimensional model, so we have to rely on the metrics of the regression to tell us how well the model has been fitted.

notebook-projects/Diabetes - Linear Regression.ipynb


# MODELING REACTION NETWORKS

```python
try:
    import controlSBML as ctl
except:
    !pip install controlSBML
    import controlSBML as ctl

import matplotlib.pyplot as plt
import pandas as pd
import numpy as np
import tellurium as te
```

    Collecting controlSBML
      Downloading controlSBML-0.1.6.tar.gz (21 kB)
    Requirement already satisfied: coverage in /usr/local/lib/python3.7/dist-packages (from controlSBML) (3.7.1)
    Collecting control
      Downloading control-0.9.1.tar.gz (357 kB)
         |████████████████████████████████| 357 kB 9.9 MB/s
    Collecting docstring-expander
      Downloading docstring_expander-0.23.tar.gz (4.0 kB)
    Collecting jupyterlab
      Downloading jupyterlab-3.3.2-py3-none-any.whl (8.7 MB)
         |████████████████████████████████| 8.7 MB 50.5 MB/s
    Requirement already satisfied: matplotlib in /usr/local/lib/python3.7/dist-packages (from controlSBML) (3.2.2)
    Collecting nose
      Downloading nose-1.3.7-py3-none-any.whl (154 kB)
         |████████████████████████████████| 154 kB 55.5 MB/s
    Requirement already satisfied: numpy in /usr/local/lib/python3.7/dist-packages (from controlSBML) (1.21.5)
    Requirement already satisfied: pandas in /usr/local/lib/python3.7/dist-packages (from controlSBML) (1.3.5)
    Requirement already satisfied: pip in /usr/local/lib/python3.7/dist-packages (from controlSBML) (21.1.3)
    Collecting pylint
      Downloading pylint-2.12.2-py3-none-any.whl (414 kB)
         |████████████████████████████████| 414 kB 54.2 MB/s
    Collecting python-libsbml
      Downloading python_libsbml-5.19.3-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (6.9 MB)
         |████████████████████████████████| 6.9 MB 35.4 MB/s
    Requirement already satisfied: seaborn in /usr/local/lib/python3.7/dist-packages (from controlSBML) (0.11.2)
    Collecting symmathSBML
      Downloading symmathSBML-0.2.1.tar.gz (19 kB)
    Requirement already satisfied: sympy in /usr/local/lib/python3.7/dist-packages (from controlSBML) (1.7.1)
    Collecting tellurium
      Downloading tellurium-2.2.2.1-py3-none-any.whl (121 kB)
         |████████████████████████████████| 121 kB 49.2 MB/s
    Collecting tk
      Downloading tk-0.1.0-py3-none-any.whl (3.9 kB)
    Requirement already satisfied: urllib3 in /usr/local/lib/python3.7/dist-packages (from controlSBML) (1.24.3)
    Requirement already satisfied: scipy in /usr/local/lib/python3.7/dist-packages (from control->controlSBML) (1.4.1)
    Collecting jupyter-server~=1.4
      Downloading jupyter_server-1.15.6-py3-none-any.whl (341 kB)
         |████████████████████████████████| 341 kB 50.5 MB/s
    Collecting tornado>=6.1.0
      Downloading tornado-6.1-cp37-cp37m-manylinux2010_x86_64.whl (428 kB)
         |████████████████████████████████| 428 kB 43.5 MB/s
    Collecting nbclassic~=0.2
      Downloading nbclassic-0.3.7-py3-none-any.whl (13 kB)
    Requirement already satisfied: ipython in /usr/local/lib/python3.7/dist-packages (from jupyterlab->controlSBML) (5.5.0)
    Requirement already satisfied: jinja2>=2.1 in /usr/local/lib/python3.7/dist-packages (from jupyterlab->controlSBML) (2.11.3)
    Requirement already satisfied: jupyter-core in /usr/local/lib/python3.7/dist-packages (from jupyterlab->controlSBML) (4.9.2)
    Requirement already satisfied: packaging in /usr/local/lib/python3.7/dist-packages (from jupyterlab->controlSBML) (21.3)
    Collecting jupyterlab-server~=2.10
      Downloading jupyterlab_server-2.11.2-py3-none-any.whl (50 kB)
         |████████████████████████████████| 50 kB 4.7 MB/s
    Requirement already satisfied: MarkupSafe>=0.23 in /usr/local/lib/python3.7/dist-packages (from jinja2>=2.1->jupyterlab->controlSBML) (2.0.1)
    Requirement already satisfied: argon2-cffi in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (21.3.0)
    Requirement already satisfied: prometheus-client in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (0.13.1)
    Collecting nbformat>=5.2.0
      Downloading nbformat-5.2.0-py3-none-any.whl (74 kB)
         |████████████████████████████████| 74 kB 2.9 MB/s
    Requirement already satisfied: nbconvert in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (5.6.1)
    Collecting jupyter-client>=6.1.1
      Downloading jupyter_client-7.1.2-py3-none-any.whl (130 kB)
         |████████████████████████████████| 130 kB 53.8 MB/s
    Requirement already satisfied: pyzmq>=17 in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (22.3.0)
    Requirement already satisfied: Send2Trash in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (1.8.0)
    Collecting websocket-client
      Downloading websocket_client-1.3.1-py3-none-any.whl (54 kB)
         |████████████████████████████████| 54 kB 2.8 MB/s
    Collecting anyio>=3.1.0
      Downloading anyio-3.5.0-py3-none-any.whl (79 kB)
         |████████████████████████████████| 79 kB 8.7 MB/s
    Requirement already satisfied: terminado>=0.8.3 in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (0.13.3)
    Requirement already satisfied: traitlets>=5 in /usr/local/lib/python3.7/dist-packages (from jupyter-server~=1.4->jupyterlab->controlSBML) (5.1.1)
    Requirement already satisfied: typing-extensions in /usr/local/lib/python3.7/dist-packages (from anyio>=3.1.0->jupyter-server~=1.4->jupyterlab->controlSBML) (3.10.0.2)
    Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.7/dist-packages (from anyio>=3.1.0->jupyter-server~=1.4->jupyterlab->controlSBML) (2.10)
    Collecting sniffio>=1.1
      Downloading sniffio-1.2.0-py3-none-any.whl (10 kB)
    Requirement already satisfied: nest-asyncio>=1.5 in /usr/local/lib/python3.7/dist-packages (from jupyter-client>=6.1.1->jupyter-server~=1.4->jupyterlab->controlSBML) (1.5.4)
    Requirement already satisfied: python-dateutil>=2.1 in /usr/local/lib/python3.7/dist-packages (from jupyter-client>=6.1.1->jupyter-server~=1.4->jupyterlab->controlSBML) (2.8.2)
    Requirement already satisfied: entrypoints in /usr/local/lib/python3.7/dist-packages (from jupyter-client>=6.1.1->jupyter-server~=1.4->jupyterlab->controlSBML) (0.4)
    Collecting jinja2>=2.1
      Downloading Jinja2-3.0.3-py3-none-any.whl (133 kB)
         |████████████████████████████████| 133 kB 59.6 MB/s
    Requirement already satisfied: requests in /usr/local/lib/python3.7/dist-packages (from jupyterlab-server~=2.10->jupyterlab->controlSBML) (2.23.0)
    Collecting json5
      Downloading json5-0.9.6-py2.py3-none-any.whl (18 kB)
    Requirement already satisfied: babel in /usr/local/lib/python3.7/dist-packages (from jupyterlab-server~=2.10->jupyterlab->controlSBML) (2.9.1)
    Requirement already satisfied: jsonschema>=3.0.1 in /usr/local/lib/python3.7/dist-packages (from jupyterlab-server~=2.10->jupyterlab->controlSBML) (4.3.3)
    Requirement already satisfied: pyrsistent!=0.17.0,!=0.17.1,!=0.17.2,>=0.14.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab->controlSBML) (0.18.1)
    Requirement already satisfied: attrs>=17.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab->controlSBML) (21.4.0)
    Requirement already satisfied: importlib-metadata in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab->controlSBML) (4.11.2)
    Requirement already satisfied: importlib-resources>=1.4.0 in /usr/local/lib/python3.7/dist-packages (from jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab->controlSBML) (5.4.0)
    Requirement already satisfied: zipp>=3.1.0 in /usr/local/lib/python3.7/dist-packages (from importlib-resources>=1.4.0->jsonschema>=3.0.1->jupyterlab-server~=2.10->jupyterlab->controlSBML) (3.7.0)
    Collecting notebook-shim>=0.1.0
      Downloading notebook_shim-0.1.0-py3-none-any.whl (13 kB)
    Requirement already satisfied: notebook<7 in /usr/local/lib/python3.7/dist-packages (from nbclassic~=0.2->jupyterlab->controlSBML) (5.3.1)
    Requirement already satisfied: ipykernel in /usr/local/lib/python3.7/dist-packages (from notebook<7->nbclassic~=0.2->jupyterlab->controlSBML) (4.10.1)
    Requirement already satisfied: ipython-genutils in /usr/local/lib/python3.7/dist-packages (from notebook<7->nbclassic~=0.2->jupyterlab->controlSBML) (0.2.0)
    Requirement already satisfied: six>=1.5 in /usr/local/lib/python3.7/dist-packages (from python-dateutil>=2.1->jupyter-client>=6.1.1->jupyter-server~=1.4->jupyterlab->controlSBML) (1.15.0)
    Requirement already satisfied: ptyprocess in /usr/local/lib/python3.7/dist-packages (from terminado>=0.8.3->jupyter-server~=1.4->jupyterlab->controlSBML) (0.7.0)
    Requirement already satisfied: argon2-cffi-bindings in /usr/local/lib/python3.7/dist-packages (from argon2-cffi->jupyter-server~=1.4->jupyterlab->controlSBML) (21.2.0)
    Requirement already satisfied: cffi>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from argon2-cffi-bindings->argon2-cffi->jupyter-server~=1.4->jupyterlab->controlSBML) (1.15.0)
    Requirement already satisfied: pycparser in /usr/local/lib/python3.7/dist-packages (from cffi>=1.0.1->argon2-cffi-bindings->argon2-cffi->jupyter-server~=1.4->jupyterlab->controlSBML) (2.21)
    Requirement already satisfied: pytz>=2015.7 in /usr/local/lib/python3.7/dist-packages (from babel->jupyterlab-server~=2.10->jupyterlab->controlSBML) (2018.9)
    Requirement already satisfied: decorator in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (4.4.2)
    Requirement already satisfied: pickleshare in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (0.7.5)
    Requirement already satisfied: setuptools>=18.5 in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (57.4.0)
    Requirement already satisfied: simplegeneric>0.8 in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (0.8.1)
    Requirement already satisfied: pygments in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (2.6.1)
    Requirement already satisfied: prompt-toolkit<2.0.0,>=1.0.4 in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (1.0.18)
    Requirement already satisfied: pexpect in /usr/local/lib/python3.7/dist-packages (from ipython->jupyterlab->controlSBML) (4.8.0)
    Requirement already satisfied: wcwidth in /usr/local/lib/python3.7/dist-packages (from prompt-toolkit<2.0.0,>=1.0.4->ipython->jupyterlab->controlSBML) (0.2.5)
    Requirement already satisfied: kiwisolver>=1.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->controlSBML) (1.3.2)
    Requirement already satisfied: cycler>=0.10 in /usr/local/lib/python3.7/dist-packages (from matplotlib->controlSBML) (0.11.0)
    Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in /usr/local/lib/python3.7/dist-packages (from matplotlib->controlSBML) (3.0.7)
    Requirement already satisfied: bleach in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter-server~=1.4->jupyterlab->controlSBML) (4.1.0)
    Requirement already satisfied: testpath in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter-server~=1.4->jupyterlab->controlSBML) (0.6.0)
    Requirement already satisfied: mistune<2,>=0.8.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter-server~=1.4->jupyterlab->controlSBML) (0.8.4)
    Requirement already satisfied: defusedxml in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter-server~=1.4->jupyterlab->controlSBML) (0.7.1)
    Requirement already satisfied: pandocfilters>=1.4.1 in /usr/local/lib/python3.7/dist-packages (from nbconvert->jupyter-server~=1.4->jupyterlab->controlSBML) (1.5.0)
    Requirement already satisfied: webencodings in /usr/local/lib/python3.7/dist-packages (from bleach->nbconvert->jupyter-server~=1.4->jupyterlab->controlSBML) (0.5.1)
    Collecting isort<6,>=4.2.5
      Downloading isort-5.10.1-py3-none-any.whl (103 kB)
         |████████████████████████████████| 103 kB 58.3 MB/s
    Collecting platformdirs>=2.2.0
      Downloading platformdirs-2.5.1-py3-none-any.whl (14 kB)
    Collecting mccabe<0.7,>=0.6
      Downloading mccabe-0.6.1-py2.py3-none-any.whl (8.6 kB)
    Collecting astroid<2.10,>=2.9.0
      Downloading astroid-2.9.3-py3-none-any.whl (254 kB)
         |████████████████████████████████| 254 kB 57.0 MB/s
    Collecting toml>=0.9.2
      Downloading toml-0.10.2-py2.py3-none-any.whl (16 kB)
    Collecting lazy-object-proxy>=1.4.0
      Downloading lazy_object_proxy-1.7.1-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (57 kB)
         |████████████████████████████████| 57 kB 5.9 MB/s
    Collecting typed-ast<2.0,>=1.4.0
      Downloading typed_ast-1.5.2-cp37-cp37m-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_12_x86_64.manylinux2010_x86_64.whl (843 kB)
         |████████████████████████████████| 843 kB 25.1 MB/s
    Requirement already satisfied: wrapt<1.14,>=1.11 in /usr/local/lib/python3.7/dist-packages (from astroid<2.10,>=2.9.0->pylint->controlSBML) (1.13.3)
    Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.7/dist-packages (from requests->jupyterlab-server~=2.10->jupyterlab->controlSBML) (2021.10.8)
    Requirement already satisfied: chardet<4,>=3.0.2 in /usr/local/lib/python3.7/dist-packages (from requests->jupyterlab-server~=2.10->jupyterlab->controlSBML) (3.0.4)
    Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.7/dist-packages (from sympy->controlSBML) (1.2.1)
    Requirement already satisfied: pytest in /usr/local/lib/python3.7/dist-packages (from tellurium->controlSBML) (3.6.4)
    Collecting libroadrunner>=2.0.3
      Downloading libroadrunner-2.2.0-cp37-cp37m-manylinux2014_x86_64.whl (54.8 MB)
         |████████████████████████████████| 54.8 MB 46 kB/s
    Collecting python-libnuml>=1.0.0
      Downloading python_libnuml-1.1.4-cp37-cp37m-manylinux1_x86_64.whl (2.0 MB)
         |████████████████████████████████| 2.0 MB 43.1 MB/s
    Collecting python-libcombine>=0.2.2
      Downloading python_libcombine-0.2.15-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (4.9 MB)
         |████████████████████████████████| 4.9 MB 49.8 MB/s
    Requirement already satisfied: appdirs>=1.4.3 in /usr/local/lib/python3.7/dist-packages (from tellurium->controlSBML) (1.4.4)
    Collecting scipy
      Downloading scipy-1.7.3-cp37-cp37m-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (38.1 MB)
         |████████████████████████████████| 38.1 MB 322 kB/s
    Collecting phrasedml>=1.0.9
      Downloading phrasedml-1.1.1-cp37-cp37m-manylinux2010_x86_64.whl (5.6 MB)
         |████████████████████████████████| 5.6 MB 36.9 MB/s
    Collecting antimony>=2.12.0
      Downloading antimony-2.12.0.2-py3-none-manylinux2010_x86_64.whl (5.8 MB)
         |████████████████████████████████| 5.8 MB 35.6 MB/s
    Collecting sbml2matlab>=0.9.1
      Downloading sbml2matlab-1.2.3-cp37-cp37m-manylinux1_x86_64.whl (3.2 MB)
         |████████████████████████████████| 3.2 MB 53.3 MB/s
    Collecting python-libsedml>=2.0.17
      Downloading python_libsedml-2.0.29-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.4 MB)
         |████████████████████████████████| 3.4 MB 3.7 MB/s
    Requirement already satisfied: plotly>=2.0.12 in /usr/local/lib/python3.7/dist-packages (from tellurium->controlSBML) (5.5.0)
    Collecting rrplugins>=2.0.3
      Downloading rrplugins-2.0.4-py3-none-manylinux2010_x86_64.whl (36.7 MB)
         |████████████████████████████████| 36.7 MB 101 kB/s
    Requirement already satisfied: tenacity>=6.2.0 in /usr/local/lib/python3.7/dist-packages (from plotly>=2.0.12->tellurium->controlSBML) (8.0.1)
    Requirement already satisfied: pluggy<0.8,>=0.5 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium->controlSBML) (0.7.1)
    Requirement already satisfied: atomicwrites>=1.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium->controlSBML) (1.4.0)
    Requirement already satisfied: py>=1.5.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium->controlSBML) (1.11.0)
    Requirement already satisfied: more-itertools>=4.0.0 in /usr/local/lib/python3.7/dist-packages (from pytest->tellurium->controlSBML) (8.12.0)
    Building wheels for collected packages: controlSBML, control, docstring-expander, symmathSBML
      Building wheel for controlSBML (setup.py) ... done
      Created wheel for controlSBML: filename=controlSBML-0.1.6-py3-none-any.whl size=28328 sha256=8976773aa8e290a9bcaf69f158a46a53113463755429b218bdddde7ac98c0393
      Stored in directory: /root/.cache/pip/wheels/b8/d2/c8/a0aa5c7a03fb9db9b14698e4bbf3323076e1bba4dccfe9bf9d
      Building wheel for control (setup.py) ... done
      Created wheel for control: filename=control-0.9.1-py2.py3-none-any.whl size=364703 sha256=3c6193a422979b614894bdcacd517a510d9d49618427a2bca3b1accafcc38272
      Stored in directory: /root/.cache/pip/wheels/b1/50/22/8005107713828997d1af664fbd629c7f935a1b87fc2f4ba6c1
      Building wheel for docstring-expander (setup.py) ... done
      Created wheel for docstring-expander: filename=docstring_expander-0.23-py3-none-any.whl size=4752 sha256=604f05e36a85f46f4c4c8aa7d8f833c597c0bd452c3296a3cd86f0e36a2180f2
      Stored in directory: /root/.cache/pip/wheels/3f/c6/1f/ac4967c9347ee35ea92c4ce8950ca99e1497c0bac38cdc4b29
      Building wheel for symmathSBML (setup.py) ... done
      Created wheel for symmathSBML: filename=symmathSBML-0.2.1-py3-none-any.whl size=24171 sha256=5dda83514313aa7783829026ead8129ee7503c9d6d76fcd68963949308472335
      Stored in directory: /root/.cache/pip/wheels/41/4f/5c/8a558bb489d276b7c5a3b519ef8009db1e87f89b32d3926c7d
    Successfully built controlSBML control docstring-expander symmathSBML
    Installing collected packages: tornado, sniffio, nbformat, jinja2, websocket-client, jupyter-client, anyio, jupyter-server, typed-ast, notebook-shim, lazy-object-proxy, json5, toml, scipy, sbml2matlab, rrplugins, python-libsedml, python-libsbml, python-libnuml, python-libcombine, platformdirs, phrasedml, nbclassic, mccabe, libroadrunner, jupyterlab-server, isort, astroid, antimony, tk, tellurium, pylint, nose, jupyterlab, control, symmathSBML, docstring-expander, controlSBML
      Attempting uninstall: tornado
        Found existing installation: tornado 5.1.1
        Uninstalling tornado-5.1.1:
          Successfully uninstalled tornado-5.1.1
      Attempting uninstall: nbformat
        Found existing installation: nbformat 5.1.3
        Uninstalling nbformat-5.1.3:
          Successfully uninstalled nbformat-5.1.3
      Attempting uninstall: jinja2
        Found existing installation: Jinja2 2.11.3
        Uninstalling Jinja2-2.11.3:
          Successfully uninstalled Jinja2-2.11.3
      Attempting uninstall: jupyter-client
        Found existing installation: jupyter-client 5.3.5
        Uninstalling jupyter-client-5.3.5:
          Successfully uninstalled jupyter-client-5.3.5
      Attempting uninstall: scipy
        Found existing installation: scipy 1.4.1
        Uninstalling scipy-1.4.1:
          Successfully uninstalled scipy-1.4.1
    ERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    google-colab 1.0.0 requires tornado~=5.1.0; python_version >= "3.0", but you have tornado 6.1 which is incompatible.
    flask 1.1.4 requires Jinja2<3.0,>=2.10.1, but you have jinja2 3.0.3 which is incompatible.
    datascience 0.10.6 requires folium==0.2.1, but you have folium 0.8.3 which is incompatible.
    albumentations 0.1.12 requires imgaug<0.2.7,>=0.2.5, but you have imgaug 0.2.9 which is incompatible.
    Successfully installed antimony-2.12.0.2 anyio-3.5.0 astroid-2.9.3 control-0.9.1 controlSBML-0.1.6 docstring-expander-0.23 isort-5.10.1 jinja2-3.0.3 json5-0.9.6 jupyter-client-7.1.2 jupyter-server-1.15.6 jupyterlab-3.3.2 jupyterlab-server-2.11.2 lazy-object-proxy-1.7.1 libroadrunner-2.2.0 mccabe-0.6.1 nbclassic-0.3.7 nbformat-5.2.0 nose-1.3.7 notebook-shim-0.1.0 phrasedml-1.1.1 platformdirs-2.5.1 pylint-2.12.2 python-libcombine-0.2.15 python-libnuml-1.1.4 python-libsbml-5.19.3 python-libsedml-2.0.29 rrplugins-2.0.4 sbml2matlab-1.2.3 scipy-1.7.3 sniffio-1.2.0 symmathSBML-0.2.1 tellurium-2.2.2.1 tk-0.1.0 toml-0.10.2 tornado-6.1 typed-ast-1.5.2 websocket-client-1.3.1

# Helpers

```python
def simulateAndPlot(roadrunner, start_time=0, end_time=5, num_point=50, variables=None, is_species=True, is_plot=True):
    """
    Runs a simulation and plots the variables.

    Parameters
    ----------
    roadrunner: ExtendedRoadrunner
    variables: list-str

    Returns
    -------
    Timeseries
    """
    if variables is None:
        if is_species:
            variables = ["time", "S1", "S2", "S3"]
        else:
            variables = ["time", "J0", "J1", "J2"]
    roadrunner.reset()
    ts = ctl.Timeseries(roadrunner.simulate(start_time, end_time, num_point, variables))
    if is_plot:
        ctl.plotOneTS(ts, figsize=(6, 4))
    return ts

# Tests
model = "S1->; S2*S3; S1=0; S2=0; S3=0"
rr = te.loada(model)
simulateAndPlot(rr, is_plot=False)
print("OK!")
```

    OK!

```python
def formatMat(mat, column_names=None, row_names=None):
    """
    Provides a nice format for a matrix.

    Parameters
    ----------
    mat: np.Array, NamedArray
    column_names: list-str
    row_names: list-str
    """
    if len(np.shape(mat)) == 1:
        mat = np.reshape(mat, (len(mat), 1))
    if column_names is None:
        if ("NamedArray" in str(type(mat))):
            column_names = mat.colnames
        else:
            column_names = range(np.shape(mat)[1])
    if row_names is None:
        if ("NamedArray" in str(type(mat))):
            row_names = mat.rownames
        else:
            row_names = range(np.shape(mat)[0])
    return pd.DataFrame(mat, columns=column_names, index=row_names)

# Tests
mat = np.array(range(10))
result1 = formatMat(mat, column_names=["a"])
mat = np.reshape(mat, (5,2))
result2 = formatMat(mat, column_names=["a", "b"])
#result1, result2
print("OK!")
```

    OK!

```python
len(np.shape(mat))
```

    1

# Sequential Pathway With Mass Action Kinetics, No Input Boundary

```python
SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY = """
J0: -> S1; k0
J1: S1->S2; k1*S1
J2: S2 -> S3; k2*S2

k0 = 0
k1 = 1
k2 = 1
S1 = 10
S2 = 0
S3 = 0
"""

RR_SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY = te.loada(SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY)
```

```python
_ = simulateAndPlot(RR_SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY)
```

```python
_ = simulateAndPlot(RR_SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY, variables=["time", "J0", "J1", "J2"])
```

What the species flux vector? ${\bf{v}_S} =$

The Jacobian of the species rate equations describes the flux values.
$J({\bf v}_S) = \frac{\partial {\bf v}_S}{\partial {\bf x}}$.

$\frac{\partial {\bf S1}}{\partial {S_1}} = $, $\frac{\partial {\bf S1}}{\partial {S_2}} = $

$\frac{\partial {\bf S2}}{\partial {S_1}} = $, $\frac{\partial {\bf S2}}{\partial {S_2}} = $

```python
formatMat(RR_SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY.getFullJacobian())

```

  <div id="df-32c3c3f4-4b14-4ab9-998a-3737d0682d47">
    <div class="colab-df-container">
      <div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>S1</th>
      <th>S2</th>
      <th>S3</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>S1</th>
      <td>-1.0</td>
      <td>0.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>S2</th>
      <td>1.0</td>
      <td>-1.0</td>
      <td>0.0</td>
    </tr>
    <tr>
      <th>S3</th>
      <td>0.0</td>
      <td>1.0</td>
      <td>0.0</td>
    </tr>
  </tbody>
</table>
</div>
      <button class="colab-df-convert" onclick="convertToInteractive('df-32c3c3f4-4b14-4ab9-998a-3737d0682d47')"
              title="Convert this dataframe to an interactive table."
              style="display:none;">

  <svg xmlns="http://www.w3.org/2000/svg" height="24px"viewBox="0 0 24 24"
       width="24px">
    <path d="M0 0h24v24H0V0z" fill="none"/>
    <path d="M18.56 5.44l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94zm-11 1L8.5 8.5l.94-2.06 2.06-.94-2.06-.94L8.5 2.5l-.94 2.06-2.06.94zm10 10l.94 2.06.94-2.06 2.06-.94-2.06-.94-.94-2.06-.94 2.06-2.06.94z"/><path d="M17.41 7.96l-1.37-1.37c-.4-.4-.92-.59-1.43-.59-.52 0-1.04.2-1.43.59L10.3 9.45l-7.72 7.72c-.78.78-.78 2.05 0 2.83L4 21.41c.39.39.9.59 1.41.59.51 0 1.02-.2 1.41-.59l7.78-7.78 2.81-2.81c.8-.78.8-2.07 0-2.86zM5.41 20L4 18.59l7.72-7.72 1.47 1.35L5.41 20z"/>
  </svg>
      </button>

  <style>
    .colab-df-container {
      display:flex;
      flex-wrap:wrap;
      gap: 12px;
    }

    .colab-df-convert {
      background-color: #E8F0FE;
      border: none;
      border-radius: 50%;
      cursor: pointer;
      display: none;
      fill: #1967D2;
      height: 32px;
      padding: 0 0 0 0;
      width: 32px;
    }

    .colab-df-convert:hover {
      background-color: #E2EBFA;
      box-shadow: 0px 1px 2px rgba(60, 64, 67, 0.3), 0px 1px 3px 1px rgba(60, 64, 67, 0.15);
      fill: #174EA6;
    }

    [theme=dark] .colab-df-convert {
      background-color: #3B4455;
      fill: #D2E3FC;
    }

    [theme=dark] .colab-df-convert:hover {
      background-color: #434B5C;
      box-shadow: 0px 1px 3px 1px rgba(0, 0, 0, 0.15);
      filter: drop-shadow(0px 1px 2px rgba(0, 0, 0, 0.3));
      fill: #FFFFFF;
    }
  </style>

    </div>
  </div>

# Sequential Pathway With Mass Action Kinetics, With Input Boundary

```python
SEQUENIAL_PATHWAY_MASS_ACTION_WITH_INPUT_BOUNDARY = """
J0: -> S1; k0
J1: S1->S2; k1*S1
J2: S2 -> S3; k2*S2

k0 = 1 # Created an input rate
k1 = 1
k2 = 1
S1 = 10
S2 = 0
S3 = 0
"""

RR_SEQUENIAL_PATHWAY_MASS_ACTION_WITH_INPUT_BOUNDARY = te.loada(SEQUENIAL_PATHWAY_MASS_ACTION_WITH_INPUT_BOUNDARY)
```

```python
ts1 = simulateAndPlot(RR_SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY, end_time=20, is_plot=False)
ts2 = simulateAndPlot(RR_SEQUENIAL_PATHWAY_MASS_ACTION_WITH_INPUT_BOUNDARY, end_time=20, is_plot=False)
ctl.plotManyTS(ts1, ts2, ncol=3, figsize=(10, 5), names=["no boundary", "with boundary"])
```

```python
ts1 = simulateAndPlot(RR_SEQUENIAL_PATHWAY_MASS_ACTION_NO_INPUT_BOUNDARY, is_species=False, end_time=20, is_plot=False)
ts2 = simulateAndPlot(RR_SEQUENIAL_PATHWAY_MASS_ACTION_WITH_INPUT_BOUNDARY, is_species=False, end_time=20, is_plot=False)
ctl.plotManyTS(ts1, ts2, ncol=3, figsize=(10, 5),  names=["no boundary", "with boundary"])
```

# mTOR


# From Second Quantization to Equation-of-Motion Coupled-Cluster using SymPy

## Table of contents
1. [Introduction](#Introduction)
2. [Second Quantization](#Second-Quantization)
3. [Normal product](#Normal-product)
4. [Contraction](#Contraction)
5. [Wicks theorem](#Wicks-theorem)
6. [Particle-Hole formalism](#Particle-Hole-formalism)
6. [Hartree-Fock](#Hartree-Fock)
7. [Baker-Campbell-Hausdorff](#Baker-Campbell-Hausdorff)
8. [Coupled-Cluster](#Coupled-Cluster)
9. [Equation-of-motion Coupled-Cluster](#Equation-of-motion-Coupled-Cluster)

### Introduction
We will have hands on tutorial for the derivation of EOM-CCSD amplitudes, one and two particle density matrix. I have developed a symbolic library using [SymPy](https://www.sympy.org/en/index.html) for deriving analytical expressions which can be easily extended to any operator.

First, we will derive the fermionic algebra from first quantization and study their properties.
$\newcommand{\ket}[1]{\left|{#1}\right\rangle}
\newcommand{\bra}[1]{\left\langle{#1}\right|}
\newcommand{\braket}[2]{\left\langle{#1}\middle|{#2}\right\rangle}$

### Second Quantization

In this section we will derive the fermionic algebra in second quantization representation from first quantization. Once the fermionic algebra is established, we will derive the relation between the its elements

$$ \ket{k} = {a^{+}_{k}}\ket{vac} $$

The opertor $a^{+}_{k}$ has created an electron on the vaccum state.
$$\phi_{k}(1) \longleftrightarrow {a^{+}_{k}}\ket{vac} $$
$\ket{vac}$ is an abstract vaccum state and $\braket{vac}{vac} = 1$
If there are two electrons the wavefunction in first quantization is
$$ \Phi(1,2) = \frac{1}{\sqrt{2}}
\begin{vmatrix}
\phi_{i}(1) & \phi_{k}(1) \\
\phi_{i}(2) & \phi_{k}(2)
\end{vmatrix} $$

The same in second quantization follows as (keeping normalization included)
$$\ket{ik} = {a^{+}_{i}a^{+}_{k}}\ket{vac} $$

As $\Phi(1,2)$ is an antisymmetric function therefore the creation operator in second quantization should respect this antisymmetry.

$$\Phi(2,1) = - \Phi(1,2)  \longleftrightarrow {a^{+}_{k}a^{+}_{i}}\ket{vac} = - {a^{+}_{i}a^{+}_{k}}\ket{vac} $$

One can see that the antisymmetry requirement is fulfilled by having the anticommutation relation between creation operators.

$$a^{+}_{k}a^{+}_{i} + a^{+}_{i}a^{+}_{k} = \{a^{+}_{k} , a^{+}_{i} \} = 0 \tag{QED 1} $$

Now we will repeat the same excercice for anhilation operators. Consider first one-electron system by creating an electron on $i$th orbital using $a_{i}^{+}$. We can define the anhilator operator $a_{i}$, which act on the former state and returns back the system to $\ket{vac}$.

$$a_{i} a_{i}^{+} \ket{vac} = \ket{vac}$$

When there is no electron in a orbital $i$, then it should result to zero.
$$a_{i} \ket{vac} = 0$$

In order to establish the anticommutation of anhilation operators we will utilize anticommutation relation of creation operators.

$$ a_{i} a_{k} \ket{ki} = \ket{vac} \ \ \  \& \ \ \  a_{k} a_{i} \ket{ik} = \ket{vac} = -a_{k} a_{i} \ket{ki}$$

Therefore,
$$ a_{i}a_{k} + a_{k}a_{i}  = \{a_{k}, a_{i}\} = 0 \tag{QED 2}$$

Now we already have two anticommutation relation and will try to derive the third, between creation and anhilation operators. Consider two orbitals $i \ \& \ k$, where $i\neq k$

$$a_{i}a_{k}^{+}\ket{i} = a_{i}\ket{ki} = - a_{i}\ket{ik} = - \ket{k}$$

Reversing the action,

$$a_{k}^{+} a_{i}\ket{i} = \ket{k}$$

Comparing both

$$a_{k}^{+} a_{i} + a_{i}a_{k}^{+} = \{ a_{i}, a_{k}^{+} \} = 0 $$

But when $i = k$, then

$$a_{i}^{+} a_{i} + a_{i}a_{i}^{+} = \{ a_{i}, a_{i}^{+} \} = 1 $$

Therefore the final expression which captures both sitation is

$$a_{i}^{+} a_{k} + a_{k}a_{i}^{+} = \{ a_{i}^{+}, a_{k} \} = \delta_{ik} \tag{QED 3}$$

Last, we need to establish the adjoint relation between anhilation and creation operator. Consider

From first quantization

$$\braket {\phi_{i}}{\phi_{k}} = \delta_{ik}$$

Using second quantization on right hand side where $\dagger$ indicates hermitian conjugate

$$\bra{vac}(a_{i}^{+})^{\dagger} a_{k}^{+}\ket{vac} = \delta_{ik}$$

It is only true if

$$(a_{i}^{+})^{\dagger} = a_{i}  \tag{QED 4}$$

Hence, $a_{i}^{+} = a_{i}^{\dagger}$ which means creation operator for $i$th orbital is hermitian conjugate of
anhilation operator for the same orbital. From now on for creation operator we will use  $\dagger$ as superscript instead of $+$

### Particle number operator

We succesfully showed the transformation of basis from first quantization to second quantization by creating fermionic algebra. Now we will do the same for operators. Lets look at number operator for $N$ fermions in $N$ orthogonal basis. The wavefuntion can be written as
$$\ket{\Psi_{N}} =  a^{\dagger}_{N}a^{\dagger}_{N-1} ... a^{\dagger}_{2} a^{\dagger}_{1}\ket{vac}$$
Define number operator for $k$th orbital
$$a^{\dagger}_{k}a_{k}  $$
Act this opeartor on $\ket{\Psi_{N}}$
$$a^{\dagger}_{k}a_{k} \ket{\Psi_{N}} = a^{\dagger}_{k}a_{k} (a^{\dagger}_{N}a^{\dagger}_{N-1} ... a^{\dagger}_{k} ...  a^{\dagger}_{2} a^{\dagger}_{1}\ket{vac})$$

Using eq.(1) and eq.(3), both $a^{\dagger}_{k}a_{k}$ can move next to $a^{\dagger}_{k}$ inside $\Psi_{N}$

$$a^{\dagger}_{k}a_{k} \ket{\Psi_{N}} = a^{\dagger}_{N}a^{\dagger}_{N-1} ...a^{\dagger}_{k}a_{k}a^{\dagger}_{k} ...  a^{\dagger}_{2} a^{\dagger}_{1}\ket{vac}  = a^{\dagger}_{N}a^{\dagger}_{N-1} ...a^{\dagger}_{k}...  a^{\dagger}_{2} a^{\dagger}_{1}\ket{vac} = +1\ket{\Psi_{N}}$$

#### Particle number operator for $N$ particles

$$\sum_{i=1}^{N} a^{\dagger}_{i}a_{i} \ket{\Psi_{N}} = N \ket{\Psi_{N}} $$

### Parity of permutation

$$p q r s \xrightarrow[]{(12)(34)} q p s r$$

The total number of transposition required to decompose any permutation $P$ is known as its parity, $\mathcal{p}$

If the total number is even then its called even parity

If the total number is odd then its called odd parity

Sign of permutation $P$ is defined as

$$sgn(P) =  -1^{p}$$

Hence,

$$sgn(P_{odd}) =  -1 \hspace{2cm} sgn(P_{even}) =  1$$

### Normal product
Given the product of operators, the normal product is defined as rearrangment of operators such that all the creation operator are moved to left and all the anhilation operator to right, then multiply it by sign of permutation.

Example: Consiper $Q = a_{p}^{\dagger}a_{q}a_{r}^{\dagger}$

Now, the in order to write normal product of $Q$, one has to permute $a_{q}a_{r}^{\dagger}$ and multiply thw whole by -1 ( $\because$ odd parity permutation).

$$n[a_{p}^{\dagger}a_{q}a_{r}^{\dagger}] = -a_{p}^{\dagger}a_{r}^{\dagger}a_{q} $$
Excercise: $Q =  a_{s}a_{p}^{\dagger}a_{r}a_{q}^{\dagger}$

1) $n[Q]$

2) $n[n[Q]]$ (nomal product is idempotent operator)

Solution: $Q =  a_{s}a_{p}^{\dagger}a_{r}a_{q}^{\dagger}$

1) $n[Q]=-a_{q}^{\dagger}a_{p}^{\dagger}a_{r}a_{s}=a_{p}^{\dagger}a_{q}^{\dagger}a_{r}a_{s}$

2) $n[n[Q]]=n[Q]$

Now, we will compute normal product using SymPy

```python
from secondquant_gen import AnnihilateFermion, CreateFermion, NO
from sympy import symbols

p,q,r,s = symbols('p,q,r,s')

Q = AnnihilateFermion(s)*CreateFermion(p)*AnnihilateFermion(r)*CreateFermion(q)

display(Q)

n = NO(Q)
nn = NO(n)

display(n,nn)

```

$\displaystyle a_{s} a^\dagger_{p} a_{r} a^\dagger_{q}$

$\displaystyle \left\{a^\dagger_{p} a^\dagger_{q} a_{r} a_{s}\right\}$

$\displaystyle \left\{a^\dagger_{p} a^\dagger_{q} a_{r} a_{s}\right\}$

### Contraction
Contraction between two operators is defined as itself minus its normal product. Symbolically

Example: Compute contraction of $Q = a_{p}a_{q}$

Excercise: Compute contraction of the following

1) $a_{p}a_{q}^{\dagger}$

2) $a_{p}^{\dagger}a_{q}$

3) $a_{p}^{\dagger}a_{q}^{\dagger}$

Solution:

Now, we will compute normal product using SymPy

```python
from secondquant_gen import contraction, F, Fd # F = AnnihilateFermion, Fd = CreateFermion
from sympy import symbols

p,q = symbols('p q')

C1 = contraction(F(p),F(q))
C2 = contraction(F(p),Fd(q))
C3 = contraction(Fd(p),F(q))
C4 = contraction(Fd(p),Fd(q))

display(C1,C2,C3,C4)
```

    0

$\displaystyle \delta_{p q}$

    0

    0

### Normal products with Contractions
Given normal product of operators with contraction of operators inside the normal product.

Then take all the contractions out in pairs and same order, leave uncontracted operators inside normal product and multiply sign of permutation.

Example:

Example:

### Wicks Theorem
Given a product of operators then it can be rewritten as normal product of the same plus normal product with all possible contractions inside.

Example:

Now, we will show use of wicks theorem using SymPy

```python
from secondquant_gen import wicks, F, Fd
from sympy import symbols, Dummy

p,q,r,s,t,u = symbols('p q r s t u')

E1 = wicks(F(p)*Fd(q)*Fd(r))
display(E1)
```

$\displaystyle \delta_{p q} a^\dagger_{r} - \delta_{p r} a^\dagger_{q} + \left\{a^\dagger_{q} a^\dagger_{r} a_{p}\right\}$

```python
E2 = wicks(F(p)*F(q)*Fd(r)*F(s)*Fd(t)*Fd(u))
display(E2)
```

$\displaystyle \delta_{p r} \delta_{q t} \delta_{s u} - \delta_{p r} \delta_{q t} \left\{a^\dagger_{u} a_{s}\right\} - \delta_{p r} \delta_{q u} \delta_{s t} + \delta_{p r} \delta_{q u} \left\{a^\dagger_{t} a_{s}\right\} + \delta_{p r} \delta_{s t} \left\{a^\dagger_{u} a_{q}\right\} - \delta_{p r} \delta_{s u} \left\{a^\dagger_{t} a_{q}\right\} - \delta_{p r} \left\{a^\dagger_{t} a^\dagger_{u} a_{q} a_{s}\right\} - \delta_{p t} \delta_{q r} \delta_{s u} + \delta_{p t} \delta_{q r} \left\{a^\dagger_{u} a_{s}\right\} - \delta_{p t} \delta_{q u} \left\{a^\dagger_{r} a_{s}\right\} + \delta_{p t} \delta_{s u} \left\{a^\dagger_{r} a_{q}\right\} + \delta_{p t} \left\{a^\dagger_{r} a^\dagger_{u} a_{q} a_{s}\right\} + \delta_{p u} \delta_{q r} \delta_{s t} - \delta_{p u} \delta_{q r} \left\{a^\dagger_{t} a_{s}\right\} + \delta_{p u} \delta_{q t} \left\{a^\dagger_{r} a_{s}\right\} - \delta_{p u} \delta_{s t} \left\{a^\dagger_{r} a_{q}\right\} - \delta_{p u} \left\{a^\dagger_{r} a^\dagger_{t} a_{q} a_{s}\right\} - \delta_{q r} \delta_{s t} \left\{a^\dagger_{u} a_{p}\right\} + \delta_{q r} \delta_{s u} \left\{a^\dagger_{t} a_{p}\right\} + \delta_{q r} \left\{a^\dagger_{t} a^\dagger_{u} a_{p} a_{s}\right\} - \delta_{q t} \delta_{s u} \left\{a^\dagger_{r} a_{p}\right\} - \delta_{q t} \left\{a^\dagger_{r} a^\dagger_{u} a_{p} a_{s}\right\} + \delta_{q u} \delta_{s t} \left\{a^\dagger_{r} a_{p}\right\} + \delta_{q u} \left\{a^\dagger_{r} a^\dagger_{t} a_{p} a_{s}\right\} + \delta_{s t} \left\{a^\dagger_{r} a^\dagger_{u} a_{p} a_{q}\right\} - \delta_{s u} \left\{a^\dagger_{r} a^\dagger_{t} a_{p} a_{q}\right\} + \left\{a^\dagger_{r} a^\dagger_{t} a^\dagger_{u} a_{p} a_{q} a_{s}\right\}$

### Expectation value with $\ket{vac}$

Let $Q = M_{1}M_{2}M_{3} ... M_{n-1}M_{n}$ product of $n$ operators

From this one can easily see the importance of Wick's theorem. Lets do some checks using SymPy

Exercise:

$Q = a_{p}a_{q}^{\dagger}a_{r}^{\dagger}$

$Q = a_{p}a_{q}a_{r}^{\dagger}a_{s}a_{t}^{\dagger}a_{u}^{\dagger}$

Compute $\bra{vac}Q\ket{vac}$?

Solution:

1. Conver $Q$ to normal order using wicks theorem.
2. Only terms which are fully contracted will result in non-zero

```python
from secondquant_gen import wicks, F, Fd
from sympy import symbols

p,q,r,s,t,u = symbols('p q r s t u')

E1 = wicks(F(p)*Fd(q)*Fd(r),keep_only_fully_contracted=True)
display(E1)
```

$\displaystyle 0$

```python
E2 = wicks(F(p)*F(q)*Fd(r)*F(s)*Fd(t)*Fd(u),keep_only_fully_contracted=True)
display(E2)
```

$\displaystyle \delta_{p r} \delta_{q t} \delta_{s u} - \delta_{p r} \delta_{q u} \delta_{s t} - \delta_{p t} \delta_{q r} \delta_{s u} + \delta_{p u} \delta_{q r} \delta_{s t}$

### Particle Hole formalism

This procedure makes the evaluation of certain matrix elements easier like

$$\bra{\Phi_{0}}\hat{H}\ket{\Phi_{0}}$$
$$\bra{\Phi_{i}^{a}}\hat{H}\ket{\Phi_{0}}$$
$$\bra{\Phi_{ij}^{ab}}\hat{H}\ket{\Phi_{0}}$$

where, $\Phi_{0}$ is Hartree-Fock determinant. Given a Hartree-Fock solution we get occupied and virtual orbitals. Lets assign symbols $(i,j,k,...)$ to occupied and $(a,b,c,...)$ to virtual.

We will define new set of operators
$$
b_{p} =
\left\{
	\begin{array}{ll}
		a_{p}^{\dagger}  & \mbox{if } p = i \\
		a_{p}            & \mbox{if } p = a
	\end{array}
\right.
$$

$$
b_{p}^\dagger =
\left\{
	\begin{array}{ll}
		a_{p}            & \mbox{if } p = i \\
		a_{p}^{\dagger}  & \mbox{if } p = a
	\end{array}
\right.
$$

This makes the HF determinant a Fermi vaccum

$$ b_{p}\ket{\Phi_{0}} = 0$$

This follows
$$ \{ b_{p},b_{q} \} = 0 \hspace{1cm} \{ b_{p}^{\dagger},b_{q}^{\dagger} \} = 0 \hspace{1cm} \{ b_{p},b_{q}^\dagger \}  = \delta_{pq}  $$

### Normal product of PHF
It is defined similarly, as rearrangment of operators ($b_{p}$ \& $b_{q}^{\dagger}$) such that all the creation operator are moved to left and all the anhilation operator to right, then multiply it by sign of permutation.

Example: Consiper $R = b_{p}^{\dagger}b_{q}b_{r}^{\dagger}$

Now, the in order to write normal product of $R$, one has to permute $b_{q}b_{r}^{\dagger}$ and multiply thw whole by -1 ( $\because$ odd parity permutation).

$$N[b_{p}^{\dagger}b_{q}b_{r}^{\dagger}] = -b_{p}^{\dagger}b_{r}^{\dagger}b_{q} $$

The thing one has to focus is this definition shows different result that $n[ ]$.
Lets do an excercise to explain this.

We are going to compute normal order of $a_{p}^{\dagger}a_{q}a_{r}^{\dagger}$ for $p=i,q=j,r=a$
$$N[a_{i}^{\dagger}a_{j}a_{a}^{\dagger}] = N[b_{i}b_{j}^{\dagger}b_{a}^{\dagger}] =
b_{j}^{\dagger}b_{a}^{\dagger}b_{i} = -b_{a}^{\dagger}b_{j}^{\dagger}b_{i}  =  -a_{a}^{\dagger}a_{j}a_{i}^{\dagger}  $$

Also compute
$$n[a_{i}^{\dagger}a_{j}a_{a}^{\dagger}] = -a_{i}^{\dagger}a_{a}^{\dagger}a_{j} = a_{a}^{\dagger}a_{i}^{\dagger}a_{j} = a_{a}^{\dagger}\delta_{ij} -  a_{a}^{\dagger}a_{j}^{\dagger}a_{i} $$

```python
from sympy.physics.secondquant import NO, F, Fd # F = AnnihilateFermion, Fd = CreateFermion
from sympy import symbols

i,j = symbols('i j', below_fermi=True)
a = symbols('a', above_fermi=True)

display(NO(Fd(i)*F(j)*Fd(a)))
```

$\displaystyle - \left\{a^\dagger_{a} a_{j} a^\dagger_{i}\right\}$

### Contractions in PHF

Same way they are defined but the operators are from PHF

Similarlry the contraction inside normal product, Wicks theorem are defined.

### Expectation value with $\ket{\Phi_{0}}$

Let $R = M_{1}M_{2}M_{3} ... M_{n-1}M_{n}$ product of $n$ operators

From this one can easily see the importance of Wick's theorem. Lets do some checks using SymPy and derive Hartree-Fock energy

### Electronic Hamiltonian

$$H_{el} = \sum_{pq} h_{pq} a_{p}^{\dagger} a_{q} + \frac{1}{4}\sum_{pqrs} v^{pq}_{rs} a_{p}^{\dagger}a_{q}^{\dagger}a_{s}a_{r}$$

where,

$$h_{pq} = \bra{\phi_{p} (x)}(-\frac{1}{2}\nabla^{2} -\sum_{\alpha} \frac{Z_{\alpha}}{r_{\alpha x}} )\ket{\phi_{q} (x)}$$

$$v_{pqrs} = \bra{\phi_{p} \phi_{q}}\ket{\phi_{r} \phi_{s}} = \braket{\phi_{p} \phi_{q}}{\phi_{r}\phi_{s}} - \braket{\phi_{p} \phi_{q}}{\phi_{s}\phi_{r}}$$

### Hartree-Fock
Lets find Hartree-Fock energy

```python
from sympy.physics.secondquant import wicks, F, Fd, NO  # F = AnnihilateFermion, Fd = CreateFermion
from sympy.physics.secondquant import substitute_dummies, contraction
from sympy.physics.secondquant import AntiSymmetricTensor, evaluate_deltas
from sympy import symbols, Rational, Dummy

p, q, r, s = symbols('p,q,r,s', cls=Dummy)

h = AntiSymmetricTensor('h', (p,), (q,))
pq = Fd(p)*F(q)
v = AntiSymmetricTensor('v', (p, q), (r, s))
pqsr = Fd(p)*Fd(q)*F(s)*F(r)

H = h*pq + Rational(1, 4)*v*pqsr
display(H) # Without summation

eq = wicks(H,keep_only_fully_contracted=True)
index_rule = {'below':  'ijklmno','above':  'abcde', 'general': 'pqrs'}
eq = substitute_dummies(eq, new_indices=True, pretty_indices=index_rule)
display(eq)

eq = evaluate_deltas(eq)
display(eq)
```

$\displaystyle h^{p}_{q} a^\dagger_{p} a_{q} + \frac{v^{pq}_{rs} a^\dagger_{p} a^\dagger_{q} a_{s} a_{r}}{4}$

$\displaystyle \delta_{i p} \delta_{p q} h^{q}_{p} + \frac{\delta_{i q} \delta_{j p} \delta_{p r} \delta_{q s} v^{rs}_{pq}}{2}$

$\displaystyle h^{i}_{i} + \frac{v^{ij}_{ij}}{2}$

### Normal ordered Hamiltonian

$$H_{N} = \sum_{pq} h_{pq} N[a_{p}^{\dagger} a_{q}] + \frac{1}{4}\sum_{pqrs} v^{pq}_{rs} N[a_{p}^{\dagger}a_{q}^{\dagger}a_{s}a_{r}]$$

One can show that

$$H_{el} = H_{N} + E_{HF}$$

From now on we will use $H_{N}$ or computing matrix elements, also $H_{N}$ measures the correlation effect

### Baker-Campbell-Hausdorff

$$e^{-B} A  e^{B} = A + B + [A,B] + \frac{1}{2!}[[A,B],B] + \frac{1}{3!}[[[A,B],B],B] +  ... $$

if $A$ is electronic Hamiltonian or normal ordered Hamiltonian and $B$ any excitation operator, then due to the stucture of Hamiltonian the series truncate after fourth expansion. This is the reason we call coupled cluster as coupled cluster.

### Coupled-Cluster

Coupled Cluster Singles and Doubles (CCSD)

$$\Psi_{CCSD} = e^{T} \ket{\Phi_{0}}$$
where,
$$ T_{1} = \sum_{ia}t_{i}^{a}a^{\dagger}i$$

$$ T_{2} = \frac{1}{4}\sum_{ijab}t_{ij}^{ab}a^{\dagger}b^{\dagger}ji$$

Equations which need to be solved
$$ E_{CCSD}^{cor} = \bra{\Phi_{0}}e^{-T} H_{N} e^{T} \ket{\Phi_{0}}   \tag{Correltion energy}$$

$$ 0 = \bra{\Phi_{i}^{a}}e^{-T} H_{N} e^{T} \ket{\Phi_{0}}   \tag{$T_{1}$ amplitude}$$

$$ 0 = \bra{\Phi_{ij}^{ab}}e^{-T} H_{N} e^{T} \ket{\Phi_{0}}   \tag{$T_{2}$ amplitude}$$

Lets see the power of SymPy now,

```python
from sympy.physics.secondquant import wicks, F, Fd, NO  # F = AnnihilateFermion, Fd = CreateFermion
from sympy.physics.secondquant import substitute_dummies, contraction
from sympy.physics.secondquant import AntiSymmetricTensor, evaluate_deltas
from sympy.physics.secondquant import PermutationOperator, simplify_index_permutations
from sympy import symbols, Rational, Dummy
import BCH

p, q, r, s = symbols('p,q,r,s', cls=Dummy)

i,j = symbols('i,j' , below_fermi=True)
a,b = symbols('a,b' , above_fermi=True)

f = AntiSymmetricTensor('f', (p,), (q,))
pq = NO(Fd(p)*F(q))
v = AntiSymmetricTensor('v', (p, q), (r, s))
pqsr = NO(Fd(p)*Fd(q)*F(s)*F(r))

H = f*pq + Rational(1, 4)*v*pqsr

ccsd = BCH.level(H,"SD")

eq = wicks(ccsd,simplify_kronecker_deltas=True,keep_only_fully_contracted=True)
index_rule = {'below':  'ijklmno','above':  'abcdef', 'general': 'pqrs'}
e_ccsd = substitute_dummies(eq, new_indices=True, pretty_indices=index_rule)
display(e_ccsd)
```

$\displaystyle f^{i}_{a} t^{a}_{i} - \frac{t^{a}_{j} t^{b}_{i} v^{ij}_{ab}}{2} + \frac{t^{ab}_{ij} v^{ij}_{ab}}{4}$

```python
eq = wicks(Fd(i)*F(a)*ccsd,simplify_kronecker_deltas=True,keep_only_fully_contracted=True)
index_rule = {'below':  'jklmno','above':  'bcdef', 'general': 'pqrs'}
t1 = (substitute_dummies(eq, new_indices=True, pretty_indices=index_rule))
display(t1)
```

$\displaystyle - f^{j}_{b} t^{b}_{i} t^{a}_{j} + f^{j}_{b} t^{ab}_{ij} - f^{j}_{i} t^{a}_{j} + f^{a}_{b} t^{b}_{i} + f^{a}_{i} - t^{b}_{j} t^{c}_{i} t^{a}_{k} v^{jk}_{bc} - t^{b}_{j} t^{c}_{i} v^{aj}_{bc} + t^{b}_{j} t^{a}_{k} v^{jk}_{ib} + t^{b}_{j} t^{ac}_{ik} v^{jk}_{bc} + t^{b}_{j} v^{aj}_{ib} - \frac{t^{b}_{i} t^{ac}_{jk} v^{jk}_{bc}}{2} + \frac{t^{a}_{k} t^{bc}_{ij} v^{jk}_{bc}}{2} + \frac{t^{bc}_{ij} v^{aj}_{bc}}{2} - \frac{t^{ab}_{jk} v^{jk}_{ib}}{2}$

```python
eq = wicks(Fd(i)*Fd(j)*F(b)*F(a)*ccsd,simplify_kronecker_deltas=True,keep_only_fully_contracted=True)
index_rule = {'below':  'klmno','above':  'cdef', 'general': 'pqrs'}
t2 = substitute_dummies(eq, new_indices=True, pretty_indices=index_rule)
P = PermutList = [PermutationOperator(i,j),PermutationOperator(a,b)]
t2 = simplify_index_permutations(t2,PermutList)
display(t2)
```

$\displaystyle f^{k}_{c} t^{c}_{i} t^{ab}_{jk} P(ij) + f^{k}_{c} t^{a}_{k} t^{bc}_{ij} P(ab) + f^{k}_{i} t^{ab}_{jk} P(ij) - f^{a}_{c} t^{bc}_{ij} P(ab) + t^{c}_{k} t^{d}_{i} t^{ab}_{jl} v^{kl}_{cd} P(ij) + t^{c}_{k} t^{a}_{l} t^{bd}_{ij} v^{kl}_{cd} P(ab) - t^{c}_{k} t^{ad}_{ij} v^{bk}_{cd} P(ab) + t^{c}_{k} t^{ab}_{il} v^{kl}_{jc} P(ij) + t^{c}_{i} t^{d}_{j} t^{a}_{k} t^{b}_{l} v^{kl}_{cd} + t^{c}_{i} t^{d}_{j} t^{a}_{k} v^{bk}_{cd} P(ab) + \frac{t^{c}_{i} t^{d}_{j} t^{ab}_{kl} v^{kl}_{cd}}{2} + t^{c}_{i} t^{d}_{j} v^{ab}_{cd} - t^{c}_{i} t^{a}_{k} t^{b}_{l} v^{kl}_{jc} P(ij) - t^{c}_{i} t^{a}_{k} t^{bd}_{jl} v^{kl}_{cd} P(ab) P(ij) - t^{c}_{i} t^{a}_{k} v^{bk}_{jc} P(ab) P(ij) - t^{c}_{i} t^{ad}_{jk} v^{bk}_{cd} P(ab) P(ij) - \frac{t^{c}_{i} t^{ab}_{kl} v^{kl}_{jc} P(ij)}{2} - t^{c}_{i} v^{ab}_{jc} P(ij) + \frac{t^{a}_{k} t^{b}_{l} t^{cd}_{ij} v^{kl}_{cd}}{2} + t^{a}_{k} t^{b}_{l} v^{kl}_{ij} + \frac{t^{a}_{k} t^{cd}_{ij} v^{bk}_{cd} P(ab)}{2} + t^{a}_{k} t^{bc}_{il} v^{kl}_{jc} P(ab) P(ij) + t^{a}_{k} v^{bk}_{ij} P(ab) + \frac{t^{cd}_{ij} t^{ab}_{kl} v^{kl}_{cd}}{4} + \frac{t^{cd}_{ij} v^{ab}_{cd}}{2} + \frac{t^{cd}_{jk} t^{ab}_{il} v^{kl}_{cd} P(ij)}{2} + t^{ac}_{ik} t^{bd}_{jl} v^{kl}_{cd} P(ij) + t^{ac}_{ik} v^{bk}_{jc} P(ab) P(ij) - \frac{t^{ac}_{ij} t^{bd}_{kl} v^{kl}_{cd} P(ab)}{2} + \frac{t^{ab}_{kl} v^{kl}_{ij}}{2} + v^{ab}_{ij}$

### Equation-of-motion Coupled-Cluster

EOM-CC allows to compute excited and open-shell character electronic states. There are many flavor depending on the target state from reference.

#### Energy and amplitude equations of EOM-CC

$$\bar{H} = e^{-T} H_{N} e^{T}$$

$$\bra{\Phi_{ij..}^{ab..}}\bar{H}\hat{R}\ket{\Phi_{0}} = E_{EOM} \bra{\Phi_{ij..}^{ab..}}\hat{R}\ket{\Phi_{0}}$$

$$\bra{\Phi_{ij..}^{ab..}}[\bar{H}-E_{cc},\hat{R}]\ket{\Phi_{0}} = \Delta E_{EOM} \bra{\Phi_{ij..}^{ab..}}\hat{R}\ket{\Phi_{0}}$$

Last equation is the one which we will derive using SymPy

Example - EOM-IP-CCSD using genralized Davidson method

$$\begin{pmatrix}
    \bar{H}_{SS} - E_{cc} & \bar{H}_{SD} \\
    \bar{H}_{DS}          &  \bar{H}_{DD} - E_{cc}
  \end{pmatrix}
  \begin{pmatrix}
    R_{1}\\
    R_{2}
  \end{pmatrix}=\omega
  \begin{pmatrix}
    R_{1}\\
    R_{2}
  \end{pmatrix}$$
  and

  $$\begin{pmatrix}
    L_{1} & &  L_{2}
  \end{pmatrix}
  \begin{pmatrix}
    \bar{H}_{SS}-E_{cc} & \bar{H}_{SD} \\
    \bar{H}_{DS} & \bar{H}_{DD}-E_{cc}
  \end{pmatrix}
  =
  \omega
  \begin{pmatrix}
    L_{1} & & L_{2},
  \end{pmatrix}    $$

Using matrix equations

Need right $\sigma$ amplitudes of EOM-IP-CCSD amplitudes and for this we need

$$\bra{\Phi_{i}}[\bar{H}-E_{cc},\hat{R_{1}}]\ket{\Phi_{0}} = ((\bar{H}_{SS} - E_{cc}) R_{1})$$

$$\bra{\Phi_{ij}^{a}}[\bar{H},\hat{R_{1}}]\ket{\Phi_{0}} = (\bar{H}_{DS} R_{1}) $$

$$\bra{\Phi_{i}}[\bar{H},\hat{R_{2}}]\ket{\Phi_{0}} = (\bar{H}_{SD} R_{2})$$

$$\bra{\Phi_{ij}^{a}}[\bar{H}-E_{cc},\hat{R_{2}}]\ket{\Phi_{0}} =((\bar{H}_{DD} - E_{cc}) R_{2}) $$

Therefore, trial vectors are defined as

$$\sigma_{1}=((\bar{H}_{SS}-E_{cc})R_{1})+(\bar{H}_{SD}R_{2})$$
$$\sigma_{2}=(\bar{H}_{DS}R_{1}) +((\bar{H}_{DD}-E_{cc})R_{2})$$

Similarly, left vectors are defined. Lets use power of SymPy for right $\sigma$ amplitudes of EOM-IP-CCSD

```python
import EOM,SIGMA
flavor1 = "IP"
R0_f1 = EOM.R0(flavor1)
R1_f1 = EOM.R1(flavor1)
R2_f1 = EOM.R2(flavor1)
Rf1  = R0_f1 + R1_f1 + R2_f1

SIGMA.RVECTORS(R0_f1,R1_f1,R2_f1,flavor1)

L0_f1 = EOM.L0(flavor1)
L1_f1 = EOM.L1(flavor1)
L2_f1 = EOM.L2(flavor1)
Lf1  = L0_f1 + L1_f1 + L2_f1

SIGMA.LVECTORS(L0_f1,L1_f1,L2_f1,flavor1)

```

 Computing right sigma amplitudes for IP (skipping summation for dummy variables)

$\displaystyle ((\overline{H}_{SS}-E_{cc})R_{1}) = - f^{j}_{a} r^{}_{j} t^{a}_{i} - f^{j}_{i} r^{}_{j} + r^{}_{j} t^{a}_{k} t^{b}_{i} v^{jk}_{ab} - r^{}_{j} t^{a}_{k} v^{jk}_{ia} - \frac{r^{}_{j} t^{ab}_{ik} v^{jk}_{ab}}{2}$

$\displaystyle (\overline{H}_{SD}R_{2}) = f^{j}_{a} r^{a}_{ij} + \frac{r^{a}_{jk} t^{b}_{i} v^{jk}_{ab}}{2} - \frac{r^{a}_{jk} v^{jk}_{ia}}{2} + r^{a}_{ij} t^{b}_{k} v^{jk}_{ab}$

$\displaystyle (\overline{H}_{DS}R_{1}) = f^{k}_{b} r^{}_{k} t^{ab}_{ij} - r^{}_{k} t^{b}_{l} t^{ac}_{ij} v^{kl}_{bc} + r^{}_{k} t^{b}_{i} t^{c}_{j} t^{a}_{l} v^{kl}_{bc} + r^{}_{k} t^{b}_{i} t^{c}_{j} v^{ak}_{bc} - r^{}_{k} t^{b}_{i} t^{a}_{l} v^{kl}_{jb} - r^{}_{k} t^{b}_{i} t^{ac}_{jl} v^{kl}_{bc} - r^{}_{k} t^{b}_{i} v^{ak}_{jb} + r^{}_{k} t^{b}_{j} t^{a}_{l} v^{kl}_{ib} + r^{}_{k} t^{b}_{j} t^{ac}_{il} v^{kl}_{bc} + r^{}_{k} t^{b}_{j} v^{ak}_{ib} + \frac{r^{}_{k} t^{a}_{l} t^{bc}_{ij} v^{kl}_{bc}}{2} + r^{}_{k} t^{a}_{l} v^{kl}_{ij} + \frac{r^{}_{k} t^{bc}_{ij} v^{ak}_{bc}}{2} + r^{}_{k} t^{ab}_{il} v^{kl}_{jb} - r^{}_{k} t^{ab}_{jl} v^{kl}_{ib} + r^{}_{k} v^{ak}_{ij}$

$\displaystyle ((\overline{H}_{DD}-E_{cc})R_{2}) = - f^{k}_{b} r^{b}_{ij} t^{a}_{k} - f^{k}_{b} r^{a}_{ik} t^{b}_{j} P(ij) + f^{k}_{i} r^{a}_{jk} P(ij) + f^{a}_{b} r^{b}_{ij} - \frac{r^{b}_{kl} t^{ac}_{ij} v^{kl}_{bc}}{2} - r^{b}_{ik} t^{c}_{j} t^{a}_{l} v^{kl}_{bc} P(ij) - r^{b}_{ik} t^{c}_{j} v^{ak}_{bc} P(ij) + r^{b}_{ik} t^{a}_{l} v^{kl}_{jb} P(ij) + r^{b}_{ik} t^{ac}_{jl} v^{kl}_{bc} P(ij) + r^{b}_{ik} v^{ak}_{jb} P(ij) + r^{b}_{ij} t^{c}_{k} t^{a}_{l} v^{kl}_{bc} + r^{b}_{ij} t^{c}_{k} v^{ak}_{bc} - \frac{r^{b}_{ij} t^{ac}_{kl} v^{kl}_{bc}}{2} + \frac{r^{a}_{kl} t^{b}_{i} t^{c}_{j} v^{kl}_{bc}}{2} - \frac{r^{a}_{kl} t^{b}_{i} v^{kl}_{jb} P(ij)}{2} + \frac{r^{a}_{kl} t^{bc}_{ij} v^{kl}_{bc}}{4} + \frac{r^{a}_{kl} v^{kl}_{ij}}{2} + r^{a}_{ik} t^{b}_{l} t^{c}_{j} v^{kl}_{bc} P(ij) - r^{a}_{ik} t^{b}_{l} v^{kl}_{jb} P(ij) - \frac{r^{a}_{ik} t^{bc}_{jl} v^{kl}_{bc} P(ij)}{2}$

 Computing left sigma amplitudes for IP (skipping summation for dummy variables)

$\displaystyle (L_{1}(\overline{H}_{SS}-E_{cc})) = - f^{i}_{a} l^{j}_{} t^{a}_{j} - f^{i}_{j} l^{j}_{} - l^{j}_{} t^{a}_{j} t^{b}_{k} v^{ik}_{ab} + l^{j}_{} t^{a}_{k} v^{ik}_{aj} - \frac{l^{j}_{} t^{ab}_{jk} v^{ik}_{ab}}{2}$

$\displaystyle (L_{2}\overline{H}_{DS}) = f^{a}_{b} l^{ij}_{a} t^{b}_{j} + f^{a}_{j} l^{ij}_{a} - f^{j}_{a} l^{ik}_{b} t^{a}_{k} t^{b}_{j} + f^{j}_{a} l^{ik}_{b} t^{ab}_{jk} - f^{k}_{j} l^{ij}_{a} t^{a}_{k} - \frac{f^{i}_{a} l^{jk}_{b} t^{ab}_{jk}}{2} + l^{jk}_{a} t^{a}_{l} t^{b}_{j} v^{il}_{bk} - \frac{l^{jk}_{a} t^{a}_{l} t^{b}_{k} t^{c}_{j} v^{il}_{bc}}{2} + \frac{l^{jk}_{a} t^{a}_{l} t^{bc}_{jk} v^{il}_{bc}}{4} + \frac{l^{jk}_{a} t^{a}_{l} v^{il}_{jk}}{2} - l^{jk}_{a} t^{b}_{j} v^{ia}_{bk} + \frac{l^{jk}_{a} t^{b}_{k} t^{c}_{j} v^{ia}_{bc}}{2} - l^{jk}_{a} t^{c}_{k} t^{ab}_{jl} v^{il}_{bc} + \frac{l^{jk}_{a} t^{c}_{l} t^{ab}_{jk} v^{il}_{bc}}{2} - l^{jk}_{a} t^{ab}_{jl} v^{il}_{bk} - \frac{l^{jk}_{a} t^{bc}_{jk} v^{ia}_{bc}}{4} - \frac{l^{jk}_{a} v^{ia}_{jk}}{2} + l^{ij}_{a} t^{a}_{k} t^{b}_{l} v^{kl}_{bj} - l^{ij}_{a} t^{a}_{l} t^{b}_{k} t^{c}_{j} v^{kl}_{bc} + \frac{l^{ij}_{a} t^{a}_{l} t^{bc}_{jk} v^{kl}_{bc}}{2} + l^{ij}_{a} t^{b}_{j} t^{c}_{k} v^{ak}_{bc} - l^{ij}_{a} t^{b}_{k} v^{ak}_{bj} + \frac{l^{ij}_{a} t^{c}_{j} t^{ab}_{kl} v^{kl}_{bc}}{2} + l^{ij}_{a} t^{c}_{l} t^{ab}_{jk} v^{kl}_{bc} + \frac{l^{ij}_{a} t^{ab}_{kl} v^{kl}_{bj}}{2} + \frac{l^{ij}_{a} t^{bc}_{jk} v^{ak}_{bc}}{2}$

$\displaystyle (L_{1}\overline{H}_{SD}) = - f^{i}_{a} l^{j}_{} + f^{j}_{a} l^{i}_{} + l^{k}_{} t^{b}_{k} v^{ij}_{ab} + l^{k}_{} v^{ij}_{ak} + l^{i}_{} t^{b}_{k} v^{jk}_{ab} - l^{j}_{} t^{b}_{k} v^{ik}_{ab}$

$\displaystyle (L_{2}(\overline{H}_{DD}-E_{cc})) = f^{b}_{a} l^{ij}_{b} - f^{k}_{a} l^{ij}_{b} t^{b}_{k} + f^{i}_{b} l^{jk}_{a} t^{b}_{k} P(ij) + f^{i}_{k} l^{jk}_{a} P(ij) - \frac{l^{kl}_{b} t^{bc}_{kl} v^{ij}_{ac}}{2} + l^{kl}_{a} t^{b}_{k} v^{ij}_{bl} - \frac{l^{kl}_{a} t^{b}_{l} t^{c}_{k} v^{ij}_{bc}}{2} + \frac{l^{kl}_{a} t^{bc}_{kl} v^{ij}_{bc}}{4} + \frac{l^{kl}_{a} v^{ij}_{kl}}{2} - l^{ik}_{b} t^{b}_{l} t^{c}_{k} v^{jl}_{ac} P(ij) - l^{ik}_{b} t^{b}_{l} v^{jl}_{ak} P(ij) + l^{ik}_{b} t^{c}_{k} v^{jb}_{ac} P(ij) + l^{ik}_{b} t^{bc}_{kl} v^{jl}_{ac} P(ij) + l^{ik}_{b} v^{jb}_{ak} P(ij) - l^{ik}_{a} t^{b}_{k} t^{c}_{l} v^{jl}_{bc} P(ij) + l^{ik}_{a} t^{b}_{l} v^{jl}_{bk} P(ij) - \frac{l^{ik}_{a} t^{bc}_{kl} v^{jl}_{bc} P(ij)}{2} - l^{ij}_{b} t^{b}_{k} t^{c}_{l} v^{kl}_{ac} + l^{ij}_{b} t^{c}_{k} v^{bk}_{ac} - \frac{l^{ij}_{b} t^{bc}_{kl} v^{kl}_{ac}}{2}$

### Properties

#### One particle density matrix (OPDM)
$$\gamma^I_{pq}= \bra{\Psi_{I}}p^{\dagger}q\ket{\Psi_{I}}$$

#### One particle transition density matrix (OPTDM)
$$\gamma^{IJ}_{pq}= \bra{\Psi_{I}}p^{\dagger}q\ket{\Psi_{J}}$$

```python
import EOM, DM, TDM
flavor1 = "IP"
R0_f1 = EOM.R0(flavor1)
R1_f1 = EOM.R1(flavor1)
R2_f1 = EOM.R2(flavor1)
Rf1  = R0_f1 + R1_f1 + R2_f1

L0_f1 = EOM.L0(flavor1)
L1_f1 = EOM.L1(flavor1)
L2_f1 = EOM.L2(flavor1)
Lf1  = L0_f1 + L1_f1 + L2_f1
DM.OPDM(Lf1,Rf1,flavor1)
```

 Computing OPDM for IP (skipping summation for dummy variables)

$\displaystyle \gamma_{ij} = \delta_{i j} l^{k}_{} r^{}_{k} + \frac{\delta_{i j} l^{kl}_{a} r^{a}_{kl}}{2} - l^{j}_{} r^{}_{i} + l^{jk}_{a} r^{}_{k} t^{a}_{i} - l^{jk}_{a} r^{a}_{ik}$

$\displaystyle \gamma_{ia} = l^{j}_{} r^{}_{j} t^{a}_{i} - l^{j}_{} r^{}_{i} t^{a}_{j} - l^{j}_{} r^{a}_{ij} - l^{jk}_{b} r^{}_{j} t^{b}_{i} t^{a}_{k} + l^{jk}_{b} r^{}_{j} t^{ab}_{ik} - \frac{l^{jk}_{b} r^{}_{i} t^{ab}_{jk}}{2} + \frac{l^{jk}_{b} r^{b}_{jk} t^{a}_{i}}{2} + l^{jk}_{b} r^{b}_{ij} t^{a}_{k} - \frac{l^{jk}_{b} r^{a}_{jk} t^{b}_{i}}{2}$

$\displaystyle \gamma_{ai} = - l^{ij}_{a} r^{}_{j}$

$\displaystyle \gamma_{ab} = l^{ij}_{a} r^{}_{i} t^{b}_{j} + \frac{l^{ij}_{a} r^{b}_{ij}}{2}$

```python
flavor2 = "CCSD"
R0_f2 = EOM.R0(flavor2)
R1_f2 = EOM.R1(flavor2)
R2_f2 = EOM.R2(flavor2)
Rf2  = R0_f2 + R1_f2 + R1_f2

L0_f2 = EOM.L0(flavor2)
L1_f2 = EOM.L1(flavor2)
L2_f2 = EOM.L2(flavor2)
Lf2  = L0_f2 + L1_f2 + L2_f2

TDM.OPTDM(Lf1,Rf1,Lf2,Rf2,flavor1,flavor2)
```

 Computing Dyson OPTDM between IP $\rightarrow$ CCSD (skipping summation for dummy variables)

$\displaystyle \gamma_i^{R} = - l^{j}_{b} r^{}_{j} t^{b}_{i} + l^{j}_{b} r^{b}_{ij} - \frac{l^{jk}_{bc} r^{}_{j} t^{bc}_{ik}}{2} + \frac{l^{jk}_{bc} r^{b}_{jk} t^{c}_{i}}{2} + r^{}_{i}$

$\displaystyle \gamma_a^{R} = l^{j}_{a} r^{}_{j} + \frac{l^{jk}_{ab} r^{b}_{jk}}{2}$

$\displaystyle \gamma_i^{L} = l^{i}_{}$

$\displaystyle \gamma_a^{L} = l^{i}_{} t^{a}_{i} + \frac{l^{ij}_{c} t^{ac}_{ij}}{2}$


```python
import numpy as np
import scipy.stats as spst
import scipy.sparse as spsp
from sif.models import BayesianLogisticRegression
from sif.models.sigmoid import sigmoid
from sif.samplers import multivariate_normal_sampler
```

```python
import matplotlib.pyplot as plt
from matplotlib import cm
%matplotlib inline
```

# Part One: Introduction

In this exposition I am considering a stochastic differential equation to draw samples from a normal distribution. The main tool of theoretical interest is the Fokker-Planck equation, which can describe the stationary distribution of the dynamics in infinite time. Consider the following stochastic differential equation,
\begin{align}
\mathop{d}X_t = -\nabla V(X_t) \mathop{d}t + \sqrt{2D}\mathop{d}B_t
\end{align}
where $V : \mathbb{R}\to\mathbb{R}$ is a potential. The object $\mathop{d}B_t$ refers to the change in Brownian motion over an infinitesimal period of time. (I have sometimes encountered the following notation for $\mathop{d}B_t$ which I find unusual and may just be a heuristic to aid intuition: $\mathop{d}B_t \sim\mathcal{N}(0, \mathop{d}t)$.) When these dynamics are run forward in time, they specify the random position of a particle at time $t$, denoted $X_t$. An object of interest for us is the probability density of being in position $x$ at time $t$, which I denote by $p_X(x, t)$. The Fokker-Planck equation says that these dynamics have a stationary distribution in the limit of infinite time and that this distribution has an explicit form as,
\begin{align}
p_s(x) &= \lim_{t\to\infty} p_X(x, t) \\
&\propto \exp\left(-V(x) / D\right)
\end{align}

The stationary distribution suggests a convenient mechanism for drawing samples from an arbitrary probability distribution $p_t(x)$:

1. Write the negative log-density of the target probability density and call this the potential $V(x)=-\log p_t(x)$.
2. Obtain the corresponding stochastic differential equation with $D=1$ and observe that its stationary distribution is $p_t(x)$.
3. Transform the continuous time stochastic differential equation into a discrete time difference equation via this recipe: $\mathop{d}X_t \to \Delta X_t = X_{t+1}-X_t$, $\mathop{d}t\to\Delta t$ (a small positive number representing discretization of time), and $\mathop{d}B_t\to \Delta B_t\sim \mathcal{N}(0, \Delta t)$.
4. Run the difference equation forward in time and hope that the discretization is fine enough to obtain samples from the target distribution.

To begin this notebook, I'll consider a specific example of a target distribution and apply this scheme for drawing samples from it. This distribution will be simple enough to directly observe convergence of the samples to the desired probability density. The potential is,
\begin{align}
V(x) &= \frac{(x-\mu)^2}{2\sigma^2} \\
\nabla_x V(x) &= \frac{x-\mu}{\sigma^2}
\end{align}
The corresponding stochastic differential equation and difference equation are,
\begin{align}
\mathop{d}X_t &= -\frac{X_t-\mu}{\sigma^2} \mathop{d}t + \sqrt{2}\mathop{d}B_t \\
\Delta X_t &= -\frac{X_t-\mu}{\sigma^2} \Delta t + \sqrt{2} \Delta B_t
\end{align}
Notice that by the Fokker-Planck equation we know the stationary distribution of these dynamics.
\begin{align}
p_s(x) &\propto \exp\left(-V(x)\right) \\
&= \exp\left(-\frac{(x-\mu)^2}{2\sigma^2}\right)
\end{align}
which we recognize as the kernel of a Gaussian distribution; hence, the stationary distribution is $\mathcal{N}(\mu,\sigma^2)$. As a specific example, consider $\mu=3$ and $\sigma^2=4$.

```python
mu = 3.
sigma = np.sqrt(4)
```

We need to set a discretization of time in order to run the stochastic dynamics. There is a trade-off here in the sense that smaller $\Delta t$ will produce a higher fidelity simulation of the continuous time dynamics but will also be slower to converge. For this example, I will discretely evolve the system every $1/10$ "units of time." I will evolve the system for a large number of iterations in order to approximate the stationary distribution of the dynamics.

```python
Delta_t = 1. / 10
n_iter = 1000000
```

This next portion of code implements these dynamics. One can see that the "recipe" is followed precisely. As a final step, I visualize a probability histogram of the discrete time iterates emerging from these dynamics and observe that they do indeed represent a large random sample from the desired target probability distribution.

```python
X = np.zeros((n_iter, ))
for i in range(1, n_iter):
    Delta_X = (
        -(X[i-1] - mu) / (sigma**2) * Delta_t +
        np.sqrt(2) * np.random.normal(scale=np.sqrt(Delta_t)))
    X[i] = X[i-1] + Delta_X
```

```python
r = np.linspace(X.min(), X.max(), 1000)
plt.figure(figsize=(17, 6))
plt.hist(X, bins=50, density=True, edgecolor="black", alpha=0.8,
         label="Empirical Distribution")
plt.plot(r, spst.norm.pdf(r, mu, sigma), linewidth=3,
         label="Target Probability Density")
plt.grid()
plt.legend(fontsize=16)
plt.xlabel("Particle Locations", fontsize=16)
plt.ylabel("Probability Density", fontsize=16)
plt.title("Stationary Distribution of the Iterates", fontsize=18)
plt.show()
```

# Part Two: Logistic Regression

In this section I will consider the problem of sampling from the posterior of a Bayesian logistic regression model. Suppose that we fix an input $\mathbf{x}\in\mathbb{R}^2$. we can define a generative model according to
\begin{align}
\theta&\sim\mathcal{N}(\mathbf{0}, \lambda^{-2}\mathbf{I}) \\
Y\vert\theta, \mathbf{x} &\sim \text{Bern}\left(\sigma(\mathbf{x}^\top \theta)\right)
\end{align}
The objective of Bayesian logistic regression is to infer a posterior distribution over $\theta$ given a collection of data points $\mathcal{D}=\left\{(\mathbf{x}_1, y_1),\ldots, (\mathbf{x}_n, y_n)\right\}$. If these observations are independent, we know that the posterior distribution will be proportional to,
\begin{align}
p(\theta\vert \mathcal{D}) \propto \left(\prod_{i=1}^n \sigma(\mathbf{x}_i^\top \theta)^{y_i} (1-\sigma(\mathbf{x}_i^\top \theta))^{1-y_i}\right) \left(\frac{\lambda^2}{\sqrt{2\pi}} \exp\left(-\lambda^2\frac{\theta^\top\theta}{2}\right)\right)
\end{align}
Some algebra and calculus shows that the derivative of the potential for this posterior distribution can be written as
\begin{align}
\nabla_\theta V(\theta) = -\sum_{i=1}^n \left(y - \sigma(\mathbf{x}_i^\top \theta)\right) \mathbf{x}_i + \lambda^{2} \theta
\end{align}
The corresponding stochastic differential equation to sample from this posterior is
\begin{align}
\mathop{d}\theta_t = -\nabla_\theta V(\theta_t) + \sqrt{2} \mathop{d}B_t
\end{align}

I begin by simulating a logistic regression problem with $n$ observations and $\mathbf{x}_{ij}\sim\mathcal{N}(0, 4)$. I set the true linear coefficients to be $\theta^\star = (2, -1)$.

```python
n = 500
X = np.random.normal(scale=2, size=(n, 2))
theta_star = np.array([2, -1])
y = np.random.binomial(1, sigmoid(X.dot(theta_star)))
```

The following code is a vectorized implementation of the potential, its gradient vector, and its Hessian matrix. One can see by inspection, however, that it is identical to the mathematical equation above.

```python
def V(theta, lambda_reg=1.):
    p = sigmoid(X.dot(theta))
    return (
        -np.sum(y*np.log(p) + (1-y)*np.log(1-p)) +
        lambda_reg / 2 * np.sum(theta**2))
```

```python
def grad_V(theta, lambda_reg=1.):
    return -X.T.dot(y - sigmoid(X.dot(theta))) + lambda_reg * theta
```

```python
def hess_V(theta, lambda_reg=1.):
    p = sigmoid(X.dot(theta))
    D = spsp.diags(p * (1. - p))
    return X.T.dot(D.dot(X)) + lambda_reg * np.eye(X.shape[1])
```

We need to set the amount of regularization to apply to the linear coefficients.

```python
lambda_reg = 0.5
```

In this problem I am using a discretization of time where I will update the dynamics of the particle every hundredth unit of time. I run the stochastic dynamics using the difference equation for a large number of iterations in order to obtain the stationary distribution. After running the dynamics for a large number of iterations, I discard all but the last $50,000$ iterations, which I treat as a sample from the posterior.

```python
Delta_t = 1. / 1000
n_iter = 100000
```

```python
Theta_samples = np.zeros((n_iter, len(theta_star)))
for i in range(1, n_iter):
    Delta_theta = (
        -grad_V(Theta_samples[i-1], lambda_reg=lambda_reg) * Delta_t +
        np.sqrt(2) * np.random.normal(scale=np.sqrt(Delta_t), size=(2, )))
    Theta_samples[i] = Theta_samples[i-1] + Delta_theta
```

```python
Theta = Theta_samples[-50000:]
```

Finally, I visualize the posterior distribution obtained from running the stochastic differential equation dynamics forward in time. I use a kernel density estimator in order to approximate the posterior from the samples. I also convert the posterior into "classification space" to show the posterior average decision boundary obtained by running the stochastic dynamics; notice that the decision boundary seems appropriate and believable.

We may also compare the samples generated by the stochastic dynamics to a Laplace approximation of the posterior using the usual approach of finding the posterior mode and computing the Hessian of the posterior at that point. This is the approach that my Sif library uses.

```python
lr = BayesianLogisticRegression(l2_reg=0.5, verbose=False)
lr.fit(X, y, use_bias=False)
S = lr.sample_parameters(10000)
```

```python
kernel = spst.gaussian_kde(Theta.T)
bounds_min, bounds_max = Theta.min(axis=0), Theta.max(axis=0)
n_grid = 30
Theta_1_grid, Theta_2_grid = np.meshgrid(
    np.linspace(bounds_min[0], bounds_max[0], n_grid),
    np.linspace(bounds_min[1], bounds_max[1], n_grid))
Theta_grid = np.vstack([Theta_1_grid.ravel(), Theta_2_grid.ravel()])
f_grid = np.reshape(kernel(Theta_grid), Theta_1_grid.shape)
```

```python
X_1_grid, X_2_grid = np.meshgrid(
    np.linspace(-6, 6, 100),
    np.linspace(-6, 6, 100))
X_grid = np.vstack([X_1_grid.ravel(), X_2_grid.ravel()])
p_grid = sigmoid(np.reshape(X_grid.T.dot(Theta.mean(axis=0)), X_1_grid.shape))
```

```python
plt.figure(figsize=(17, 12))

# Density plot.
plt.subplot(2, 2, 1)
plt.grid()
plt.contourf(Theta_1_grid, Theta_2_grid, f_grid, cmap='Blues')
cset = plt.contour(Theta_1_grid, Theta_2_grid, f_grid, colors="k")
plt.clabel(cset, inline=1, fontsize=14)
plt.xlabel("$\\theta_1$", fontsize=18)
plt.ylabel("$\\theta_2$", fontsize=18)
plt.title("Stationary Distribution of Stochastic Dynamics", fontsize=18)

# Decision plot.
plt.subplot(2, 2, 2)
plt.contourf(X_1_grid, X_2_grid, p_grid, cmap='coolwarm', alpha=0.7)
cset = plt.contour(X_1_grid, X_2_grid, p_grid, colors="k")
plt.clabel(cset, inline=1, fontsize=14)
plt.plot(X[y==0, 0], X[y==0, 1], "b.", label="Class Zero")
plt.plot(X[y==1, 0], X[y==1, 1], "r.", label="Class One")
plt.grid()
plt.legend(fontsize=16)
plt.xlabel("$X_1$", fontsize=16)
plt.ylabel("$X_2$", fontsize=16)
plt.title("Binary Classification Problem", fontsize=18)
plt.xlim((-6, 6))
plt.ylim((-6, 6))

# Quantile-Quantile plots.
bounds = [[1.4, 3.0], [-1.75, -0.5]]
for i in [0, 1]:
    plt.subplot(2, 2, i+3)
    quantiles = np.linspace(0, 1, 100)
    plt.plot(np.quantile(Theta[:, i], quantiles),
             np.quantile(S[:, i], quantiles), ".")
    plt.plot(bounds[i], bounds[i], "-")
    plt.grid()
    plt.xlabel("Quantiles of Stochastic Dynamics", fontsize=16)
    plt.ylabel("Quantiles of Laplace Approximation", fontsize=16)
    plt.title("Q-Q Plot for $\\theta_" + str(i+1) + "$", fontsize=18)

plt.tight_layout()
plt.show()
```