# Introduction to `esys.escript`

## Outline

This unit gives an introduction into solving partial differential equations (PDEs) in python. This
section assumed that you have a basic understanding how to work with python.

We are particularly looking at PDEs as they arise in geophysical problems. Of course it would take some work to build appropriate PDE solvers from scratch. To save this effort we use the python PDE solver module `esys.escript`. This section will give an introduction into the work with `esys.escript` to solve 2D problems. To illustrate the use we will look at the
calculation of gravity anomaly fields from subsurface density anomalies. The following section will then present some geophysical applications of PDEs. The the third section will discuss the use of `esys.escript` to solve seismic wave equations in time and in frequency domain.

We first present some details the PDEs for modeling gravity field anomalies before we start to
discuss how to use `esys.escript` to solve this PDE.

## Useful links:

- [esys.escript home page](https://launchpad.net/escript-finley)
- [Researchgate](https://www.researchgate.net/project/esys-escript)
- [user's guide](https://launchpad.net/escript-finley/3.0+/5.4/+download/docs.zip)
- [API documentation](https://esys-escript.readthedocs.io/en/latest/index.html)

## An example problem: Gravity Field Anomalies
The gravitational field $\mathbf{g}$ (also called gravitational acceleration) is a vector field defining the gravitational force experienced by a particle as its mass multiplied by the gravitational field at that point.
In Cartesian coordinates the gravitational field $\mathbf{g}$ is expressed in the form
\begin{equation}
\mathbf{g}=(g_x, g_y, g_z)
\end{equation}
The vector $\mathbf{g}$ needs to fulfill the Gauss's law which is a generalization of Newton's law introduced in The Gauss's law is stated in the following way:
\begin{equation}\label{EQGAUSSLAW}
 \frac{\partial g_x}{\partial x} +  \frac{\partial g_y}{\partial y} + \frac{\partial g_z}{\partial z} = - 4\pi G\rho
\end{equation}
where $G=6.67 \cdot 10^{-11}  \frac{m^3}{kg s^2}$ is the gravitational constant and $\rho$ is the density distribution.  Gauss's law also stated using the divergence operator $\mathbf{\nabla}^t$:
\begin{equation}
\mathbf{\nabla}^t \; \mathbf {g} =-4\pi G\rho
\end{equation}
The magnetic field $\mathbf{g}$ is obtained from its potential $u$
using the *Grad* operator $\mathbf{\nabla}$:
\begin{equation}
\mathbf{g} = -
\mathbf{\nabla} U
\end{equation}
with the gravity accelerations
\begin{equation}\label{EQGRADRULE}
g_x =  - \frac{\partial U}{\partial x}, g_y =  - \frac{\partial U}{ \partial y} \mbox{ and } g_z =  - \frac{\partial U}{\partial z},
\end{equation}
If $\rho$ in \eqref{EQGAUSSLAW} is selected as a density anomaly, ie. as the deviation from a constant background
density, then the Gauss's law \eqref{EQGAUSSLAW} and scalar potential definition \eqref{EQGRADRULE}
defines a PDE for the potential $u$ which gradient gives the gravity field anomaly $\mathbf{g}$
due to the density anomaly $\rho$. To model field observations we are interested in the
vertical component $g_z$ of $\mathbf{g}$.

## The PDE template
In `esys.escript` the PDE to be solved is defined trough a generic PDE template
in $x_0x_1$-coordinates.
which is provided through the `LinearSinglePDE` class. The
template fits nicely with the problem of finding the gravity potential. We will in the $x_0=x$
and $x_1=z$ coordinate system.

First we write `LinearSinglePDE` template down in an abstract formulation: When $u$ is the unknown
we define the so-called `flux` vector $\mathbf{F}$ which is in essence the negative gradient of the solution
times some matrix $\mathbf{A}$ plus some vector $X$:
\begin{equation} \label{EQFLUX}
\mathbf{F} = - \mathbf{A} \mathbf{\nabla} u +\mathbf{X}
\end{equation}
Ignoring the matrix $\mathbf{A}$ and setting $\mathbf{X}=0$
this already looks like the scalar potential definition \eqref{EQGRADRULE} when $u=U$ is the gravity potential and
$\mathbf{F}=\mathbf{g}$ is the gravity acceleration.

The flux vector $\mathbf{F}$ needs to fulfill the conservation equation :
\begin{equation}\label{EQCONSERVATION}
\mathbf{\nabla}^t \; \mathbf{F} + D \; u = Y
\end{equation}
where $D$ is a scalar and $Y$ is the right hand side. We can easily identify Gauss's law \eqref{EQGAUSSLAW}
when we choose $D=0$ and $Y=- 4\pi G\rho$.

Before we set this up in python we look at these equations in a bit more detail. For the 2D case the flux definition \eqref{EQFLUX} reads as
\begin{equation}\label{EQFLUX2}
\mathbf{F} =
\begin{bmatrix}
 F_0 \\
 F_1
\end{bmatrix}
= -
\begin{bmatrix}
A_{00} \frac{\partial u}{\partial  x_0} & + & A_{01} \frac{\partial  u}{\partial  x_1}\\
A_{10} \frac{\partial  u}{\partial  x_0} & + & A_{11} \frac{\partial  u}{\partial  x_1}
\end{bmatrix}
\end{equation}
Comparison with the grad rule \eqref{EQGRADRULE} shows that we need to choose
\begin{equation}
\begin{bmatrix}
A_{00} & A_{01} \\
A_{10}  & A_{11}
\end{bmatrix}=
\begin{bmatrix}
1 & 0  \\
0  & 1
\end{bmatrix}
\end{equation}
To define the PDE we want to solve we need to define the coefficients in the PDE template
through `LinearSinglePDE` instance. To get the solution $u$ (in fact a numerical approximation is calculated)
is then as easy as calling the `getSolution` method.
Before we can define the PDE we need to set up the region over which we would like to solve the PDE. In the
terminology of `esys.escript`  this is called a `Domain`.

## How to create a domain

The first step to set-up the problem is to define the region over which the problem is solved. Here we use
a rectangular grid of `NEx` elements in the horizonal x-direction and `NEz` elements in the vertical z-direction.
We use a grid line spacing of `dx`.

```python
#%matplotlib notebook
```

```python
NEx=100 # number of cells
NEz=100
dx=10. # in meter [m] grid spacing
Lx=dx*NEx
Lz=dx*NEz
print("domin extension = %s x %s"%(Lx,Lz))
print("grid = %s x %s"%(NEx,NEz))
```

    domin extension = 1000.0 x 1000.0
    grid = 100 x 100

We use the package `finley` which is part of the `esys.escript` distribution to set up the domain:

```python
from esys.escript import *
from esys.finley import Rectangle
domain=Rectangle(l0=Lx, l1=Lz, n0=NEx, n1=NEz)
print(type(domain))

```

    <class 'esys.finley.finleycpp.FinleyDomain'>

**Note** the following rules for `Rectangle`:
- the axis are labeled `x0` and `x1`
- the lower, left corner has the coordinates `(0.,0.)`.

Use `Brick` from `esys.finley` for 3D domains.
There are other domain packages available:
- `esys.finley` - general FEM solver solver also support unstructured meshes and contact elements
- `esys.ripley` - special solver for rectangular grids
- `esys.specley` - spectral element solver for wave problems (will discussed later)
- `esys.dudley` - FEM solver for tetrahedral and triangular meshes

## Setting up and solving a PDE

The first step to set up a PDE is to create an instance of the `LinearSinglePDE` class.
Here we call this instance `model` and attach it to the domain `domain` we have already created:

```python
from esys.escript.linearPDEs import LinearSinglePDE

model=LinearSinglePDE(domain)
print(model)
```

    <LinearPDE 140375848399536>

Now we need to set the coefficients $\mathbf{A}$ and $Y$. `esys-escript` will automatically assume that the other coefficients $\mathbf{X}$ and $D$ are zero. Let's start with coefficient $\mathbf{A}$:

Recall that for the Gauss' Law $\mathbf{A}$ is the identity matrix. `esys-escript`
provides the convenience function `identityTensor` that sets up the identity matrix.
Its return value can directly be passed on to `model` as coefficient `A`:

```python
model.setValue(A=identityTensor(domain))
```

**Note:**
`identityTensor` returns a $d \times d$ matrix where $d$ is chosen as the spatial
of its argument. As `domain` is two dimensional a $2 \times 2$ identity matrix is returned.
If the domain becomes 3D at a later point the identity matrix will be defines as $3 \times 3$. This allows writing code that is identity from the spatial dimension of the domain.

Next we define the PDE coefficient $Y$ which is given as $Y=- 4\pi G\rho$ and requires us
to define the density $\rho$.
As an example we assume that the density anomaly is a circle centered
at $\mathbf{c}=(c_0, c_1)=(\frac{Lx}{2}, \frac{Lx}{3})$ with radius $R_C=100$m.

To define $\rho$ we first need to calculate the distance of any point $\mathbf{x}$ in the
domain from the $\mathbf{c}$. Then $rho$ is set to $\rho_0=1000 \frac{kg}{m^3}$
at those points which distance is smaller than $R_C$. For the others  $\rho$ is set to zero.

First gets the coordinates of the points in the domain:

```python
x=domain.getX()
print(type(x))
```

    <class 'esys.escriptcore.escriptcpp.Data'>

`x` is a `Data` object which is giving locations in the domain. So `x` has two components:

```python
print("shape of data obect `x`",x.getShape())
```

    shape of data obect `x` (2,)

Components of `x` can be accessed by slicing:

```python
print("x0 coordinates = ",x[0])
print("x1 coordinates = ",x[1])
```

    x0 coordinates =  Summary: inf=0 sup=1000 data points=10201
    x1 coordinates =  Summary: inf=0 sup=1000 data points=10201

The statement `inf=0` gives the smallest value - in our case of the $x_0$ coordinate.
The maximal value is shown as `sup=1000` which gives the value of `Lx`. The number of
point being used is `10201`. The reason for this is that there is one value for each grid point which would be '(n0+1) x (n1+1)' values (Why?). In our case this would be
'(n0+1) x (n1+1) = (NEx+1) x (NEz+1) = 201 x 201 = 10201' values.

The distance of point $\mathbf{x}$ to point $\mathbf{c}$ can be calculated by
\begin{equation}\label{eq:distance}
d=\sqrt{ (x_0-c_0)^2 + (x_1-c_1)^2}
\end{equation}

```python
c=[Lx/2., Lz/3.]
RC=100.
```

```python
d=sqrt((x[0]-c[0])**2+(x[1]-c[1])**2)
print("distance to c = %s :"%c,d)
```

    distance to c = [500.0, 333.3333333333333] : Summary: inf=3.33333 sup=833.333 data points=10201

Notice that `d` is also a `Data` object as it has been  derived from
the `Data` object `x`. It can be seen as a function of the location `x`.

There is a more compact form to calculate `d` using the `length` function:

```python
d=length(x-c)
print("distance to c = %s :"%c,d)
```

    distance to c = [500.0, 333.3333333333333] : Summary: inf=3.33333 sup=833.333 data points=10201

We want have the anomaly distribution `rho` to be `rho0` where the distance `d` is smaller than `RC`
(or `d-RC<0`) and zero otherwise. We use the `whereNegative` function which returns one where its argument is negative and zero elsewhere:

```python
rho0=1000.
rho=rho0*whereNegative(d-RC)
```

Now we can set the right hand side `Y` of the PDE `model`:

```python
import numpy as np
G=6.67e-11  # m^3/kg/sec^2 gravity constant
model.setValue(Y=-4*G*np.pi*rho)
```

We are expecting the gravity to vanish at large distances away from the circle. For a bounded domain as we use in the gravity model here we enforce this condition on the boundary of the domain formed by the bottom and top face
$x_1=0$ and $x_1=L_z$. To tell this to `LinearSinglePDE` we need to set a mask `q` which marks the locations on the surface where we want the solution $u$ to be zero:
\begin{equation}\label{EQQ}
q(\mathbf{x}) = \begin{cases}
>0  & \mbox{ set } u(\mathbf{x})=0 \\
=0  & \mbox{ no constraint at } x
\end{cases}
\end{equation}
We use the `whereZero` function which takes a `Data` object. It returns
a `Data` object which has the value
**one** at location where the argument has the value **zero**. Otherwise zero is used.
It is applied to define a `q` for each face of the domain. These `q`s are then combined
by addition to define the final `q` which is has positive value at any point on the
surface of the domain but zero in the interior.

```python
x=domain.getX()
q_bottom=whereZero(x[1])     # 1 for face x_1=0
q_top=whereZero(x[1]-Lz)  # 1 for face x_1=Lz

model.setValue(q=q_bottom+q_top)
```

Question is: What boundary conditions are applied on the other two faces?
By default the so called *weak* boundary conditions
\begin{equation}\label{eq:weakBC}
\mathbf{F} \cdot \mathbf{n} = F_0 n_0 + F_1 n_1 = 0
\end{equation}
with outer normal field $\mathbf{n}=(n_0, n_1)$ are assumed.
As $\mathbf{n}=(-1,0)$ on the left face and
As $\mathbf{n}=(1,0)$ on the right face
of the domain the boundary condition there become
\begin{equation}\label{eq:weakBC1}
F_0= -\frac{\partial u}{\partial x_0} = 0
\end{equation}

Now we are are ready to get the solution of the PDE:

```python
u=model.getSolution()
```

**Note**

The `getSolution` call involves the solution of system of linear equation. The dimension
is the number of grid points. For large grids and for 3D problems this solve can take some time.

The `grad` function returns the gradient of the argument. The negative of the gradient
gives us the gravitational acceleration `g`:

```python
g=-grad(u)
print(g)
```

    Summary: inf=-3.85762e-05 sup=4.6591e-05 data points=40000

The gravitational acceleration is a vector:

```python
g.getShape()
```

    (2,)

Before we start looking into postprocessing techniques we need to understand better
what `esys.escript.Data` objects such as `d`, `u` and `g` are actually represent.

## What do the values in `Data` objects actually mean?

When we print `Data` objects this shows the minimum (`inf`) and maximum (`sup`) value but also
the number of `data points` being used. Let have a look at the `d`, `u` and `g` again:

```python
print("x :", x)
print("d :", d)
print("u :", u)
print("g :", g)
```

    x : Summary: inf=0 sup=1000 data points=10201
    d : Summary: inf=3.33333 sup=833.333 data points=10201
    u : Summary: inf=-0.00974452 sup=0 data points=10201
    g : Summary: inf=-3.85762e-05 sup=4.6591e-05 data points=40000

The number of data points for `x`, `d` and `u` are the same namely `10201`. As already explained this
comes from the fact that `d` holds the data on the grid nodes (intersects of grid lines).
The gradient `g` is stored cell (or element) based where in four integration points in each cell is used.
This explains the number of data points of `40000` as
\begin{equation}
NEx \times NEz \times \mbox{ # integration points } = 100  \times 100  \times 4 = 40000
\end{equation}
When handling `Data` objects it is obviously important to know which representation locations
a particular `Data` object is
using. For instance a `Data` object using grid nodes and a `Data` objects using cells cannot easily be added together. In order to handle this the `Data` object has an attribute that defines the way its values are handled.
This location attribute can be obtained by the `getFunctionSpace` method:

```python
print("d is stored at ", d.getFunctionSpace())
print("g is stored at ", g.getFunctionSpace())
```

    d is stored at  Finley_Nodes [ContinuousFunction(domain)] on FinleyMesh
    g is stored at  Finley_Elements [Function(domain)] on FinleyMesh

This a list of  location attributes are available:

- `ContinuousFunction(domain)`: mesh nodes
- `Solution(domain)`: solution of a PDE, typically also on mesh nodes
- `Function(domain)` : on integration points per element
- `ReducedFunction(domain)`: on element centers
- `DiracDeltaFunction(domain)`: point sources and sinks which we will use later

The locations where the values are hold can be obtained by the `getX` methof of a `FunctionSpace`:

```python
X=g.getFunctionSpace().getX()
```

One can test if the `FunctionSpace` attribute of two `Data` objects is the same:

```python
print("g and d on the same location?", g.getFunctionSpace() == d.getFunctionSpace())
print("X and g on the same location?",X.getFunctionSpace() == g.getFunctionSpace())
print("g is on integration points?",X.getFunctionSpace() == Function(domain))
```

    g and d on the same location? False
    X and g on the same location? True
    g is on integration points? True

Interpolation can be used to change the data location. We would like to have the
vertical gravity $g_z$ at element centers. Here we interpolate the `g[1]` which is the
vertical gravity at the integration points to the element centers:

```python
gz=interpolate(g[1], ReducedFunction(domain))
print(g)
print(gz)
```

    Summary: inf=-3.85762e-05 sup=4.6591e-05 data points=40000
    Summary: inf=-3.69606e-05 sup=4.65409e-05 data points=10000

**WARNING:** Not all interpolations work. This depends on the PDE model being used.
For instance interpolation from element centers to nodes is not supported for a `finley` domain:

```python
interpolate(gz, ContinuousFunction(domain))
```

## Visualization using `matplotlib`

`matplotlib` is a powerful and very versatile tool for plotting data in python.
Here we want to use it plot the distribution of the vertical gravity over the domain.
To hand `Data` objects to `matplotlib` they first need to converted into `numpy` arrays.
With `convertToNumpy` `esys.escript` provides an easy mechanism to do this.
To plot data we need not only the data but also the location at which they are position in order
to plot their distribution. So we not only convert the `Data` objects but also the corresponding
locations which we can get through the `FunctionSpace` attribute:

```python
gz_np=convertToNumpy(gz)
x_np=convertToNumpy(gz.getFunctionSpace().getX())
print(gz_np)
print(x_np)
```

We now can use the `tricontourf` to plot filled contours and `tricontour` to plot contour lines of the
the spatial distribution of vertical gravity `gz` using its `numpy` version `gz_np` and the corresponding data locations `x_np`.

** Note ** For more options see the documentaions of [tricontour](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.tricontour.html) and [tricontourf](https://matplotlib.org/3.1.1/api/_as_gen/matplotlib.pyplot.tricontourf.html).

```python
tomGal = 100000. # conversion from m/s**2 to mGal
```

```python
import matplotlib.pyplot as plt
tomGal = 100000

plt.figure(figsize=(7,7))
plt.clf()
contour=plt.tricontourf(x_np[0], x_np[1], gz_np[0]*tomGal, 15)
plt.tricontour(x_np[0], x_np[1], gz_np[0]*tomGal, 15, linewidths=0.8, colors='k')

plt.xlabel('$x_0$ [m]')
plt.ylabel('$x_1$ [m]')
plt.title("Vertical gravity $g_z$ [mGal] due to a circular anomaly")
plt.colorbar(contour)
plt.gca().set_aspect('equal')
```

We can also do this in 3D:

```python
if True:
    from mpl_toolkits.mplot3d import Axes3D
    fig = plt.figure()
    plt.clf()
    ax = fig.gca(projection='3d')
    contour=ax.plot_trisurf(x_np[0], x_np[1], gz_np[0]*tomGal , linewidth=0.2, antialiased=True, cmap=plt.cm.CMRmap)
    plt.xlabel('$x_0$ [m]')
    plt.ylabel('$x_1$ [m]')
    plt.title("Vertical gravity $g_z$ [mGal]")
    plt.colorbar(contour)
    plt.show()
```

**Note**:
For larger meshes with millions of element and also for 3D domains `matplotlib` is not really suitbale.
Firstly its is just too slow but it also does not provide appropriate functionalities to deal with these kind of data. Alternative packages are
- [mayavi](https://docs.enthought.com/mayavi/mayavi/)
- [paraview](https://www.paraview.org/)
- [visit](https://wci.llnl.gov/simulation/computer-codes/visit/)

Although these packages can also be used from python it is more appropriate to use them separately trough their respective GUIs. To hand data to them `Data` objects are written to external files preferable in the `VTK` format.
This can be done using the `saveVTK` function from the `esys.weipa` module, see the users guide for details.

## Value Picking

Visualization of the distribution is often not sufficient for instance if one wants to apply a more quantitative
analysis of the result; for instance comparison with a observations. There is a mechanism to pick values of `Data` objects at specific locations.

### Single Value
Here we want to obtain the vertical gravity at a point at a height $h=300$ m above the center of the circular anomaly located at $c$. The location of this point $\mathbf{p}$ is then
\begin{equation}\label{eq:P}
\mathbf{p}=(c_0, c_1+300)
\end{equation}
We define a so-called `Locator` that provides a mechanism to extract the value at this point $\mathbf{p}$.
The `Locator` is build for the use at all `Data` objects with a specific `FunctionSpace` attribute; in our case
that should be the same as for `gz`:

```python
from esys.escript.pdetools import Locator
p=[c[0], c[1]+300]
point_locator=Locator(where=ReducedFunction(domain), x=p)
```

Now we can easily get the value of `gz` at `p`:

```python
v=point_locator.getValue(gz)
print("value of gz @ %s is %s mGal. "%(point_locator.getX(), v*tomGal))
```

The `Locator` actually is not always using the specified point but picks the location
of the `FunctionSpace` nearest to the requested. In our case the point is actually moved by about $5$ m:

```python
print("target point was %s."%(p,))
```

    target point was [500.0, 633.3333333333333].

The `point_locator` can be reused to pick values from other `Data` objects as long as they can interpolated
to `ReducedFunction`:

```python
v2=point_locator.getValue(g)
print("value of g @ %s is %s."%(point_locator.getX(), v2))
```

    value of g @ [ 495.  635.] is [  1.67666900e-07  -1.32564753e-05].

### Along a line of points (transect)
One can also use the `Locator` to pick data for a set of points for instance along a horizontal transect.
First we define the $x_0$-coordinates of the points we would like to use:

```python
x0_transect=np.linspace(0., Lx, NEx)
```

We then add the $x_1$ coordinate as $c_1+300$ to define the locations in the transect in the 2D domain:

```python
h=300
x_transect=[ (x0, c[1]+h) for x0 in x0_transect]
print(c[1]+h-Lz/2)
```

    133.33333333333326

Now we can create new `Locator` named `transect_locator` using the points `x_transect`:

```python
transect_locator=Locator(where=ReducedFunction(domain), x=x_transect )
```

Then the vertical gravity across the transect can be picked from `gz`. We also get the true $x_0$ coordinates
of the points in the transect:

```python
gz_transect=transect_locator.getValue(gz*tomGal)
x0_transect=[ x[0] for x in transect_locator.getX()]
```

And finally we can plot the vertical gravity over the transect:

```python
plt.figure(figsize=(5,5))
plt.clf()
plt.plot(x0_transect, gz_transect)
plt.xlabel('offset [m]')
plt.ylabel('$g_z$ [mGal]')
plt.title("gravity anomaly over transect @ height %g"%(c[1]+h))
plt.show()
```


# Tutorial: optimal piecewise binning with binary target

## Basic

To get us started, let's load a well-known dataset from the UCI repository and transform the data into a ``pandas.DataFrame``.

```python
import numpy as np
import pandas as pd

from sklearn.datasets import load_breast_cancer
from sklearn.ensemble import GradientBoostingClassifier
```

```python
data = load_breast_cancer()
df = pd.DataFrame(data.data, columns=data.feature_names)
```

We choose a variable to discretize and the binary target.

```python
variable = "mean radius"
x = df[variable].values
y = data.target
```

Import and instantiate an ``OptimalPWBinning`` object class and we pass the variable name. The ``OptimalPWBinning`` can **ONLY** handle numerical variables. This differs from the ``OptimalBinning`` object class.

```python
from optbinning import OptimalPWBinning
```

```python
optb = OptimalPWBinning(name=variable)
```

We fit the optimal binning object with arrays ``x`` and ``y``. We set a minimum (lower bound) and maximum (upper bound) probability using arguments ``lb`` and ``ub``. Note that these bounds are optional, but guarantee an event rate between $[0, 1]$ as any probability distribution. As shown later, settings these bounds will increase the problem size thus leading to higher solution times. An alternative approach to bypass these bounds is implemented when transforming to event rate or WoE using functions ``fit`` and ``fit_transform``, where values outside the interval $[0, 1]$ are clipped.

Note that the Weight-of-Evidence is not defined for event rate equal 0 or 1. Therefore, for small datasets, we recommend modifying the interval $[0, 1]$ with a small value, thus using $[\epsilon, 1 - \epsilon]$.

```python
optb.fit(x, y, lb=0.001, ub=0.999)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='mean radius')

You can check if an optimal solution has been found via the ``status`` attribute:

```python
optb.status
```

    'OPTIMAL'

You can also retrieve the optimal split points via the ``splits`` attribute:

```python
optb.splits
```

    array([11.42500019, 12.32999992, 13.09499979, 13.70499992, 15.04500008,
           16.92500019])

#### The binning table

The optimal binning algorithms return a binning table; a binning table displays the binned data and several metrics for each bin. Class ``OptimalPWBinning`` returns an object ``PWBinningTable`` via the ``binning_table`` attribute.

```python
binning_table = optb.binning_table
```

```python
type(binning_table)
```

    optbinning.binning.piecewise.binning_statistics.PWBinningTable

The `binning_table` is instantiated, but not built. Therefore, the first step is to call the method `build`, which returns a ``pandas.DataFrame``.

```python
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 11.43)
      118
      0.207381
      3
      115
      1.032648
      -0.004820

      1
      [11.43, 12.33)
      79
      0.138840
      3
      76
      1.644837
      -0.058403

      2
      [12.33, 13.09)
      68
      0.119508
      7
      61
      2.140569
      -0.098609

      3
      [13.09, 13.70)
      49
      0.086116
      10
      39
      2.914961
      -0.157745

      4
      [13.70, 15.05)
      83
      0.145870
      28
      55
      4.105273
      -0.244597

      5
      [15.05, 16.93)
      54
      0.094903
      44
      10
      3.562494
      -0.208520

      6
      [16.93, inf)
      118
      0.207381
      117
      1
      0.082142
      -0.002887

      7
      Special
      0
      0.000000
      0
      0
      0.000000
      0.000000

      8
      Missing
      0
      0.000000
      0
      0
      0.000000
      0.000000

      Totals

      569
      1.000000
      212
      357
      -
      -

Let's describe the columns of this binning table:

- Bin: the intervals delimited by the optimal split points.
- Count: the number of records for each bin.
- Count (%): the percentage of records for each bin.
- Non-event: the number of non-event records $(y = 0)$ for each bin.
- Event: the number of event records $(y = 1)$ for each bin.
- $c_0$: the first coefficient of the event rate polynomial.
- $c_1$: the second coefficient of the event rate polynomial.

The event rate for bin $i$ is defined as $ER_i = c_0 + c_1 x_i$, where $x_i \in \text{Bin}_{i}$. In general,
\begin{equation}
ER_i = \sum_{j=0}^d c_j x_i^j,
\end{equation}
where $d$ is the degree of the event rate polynomial.

The last row shows the total number of records, non-event records, event records.

You can use the method ``plot`` to visualize the number of non-event and event records by bin, and WoE or event rate curve.

```python
binning_table.plot(metric="woe")
```

```python
binning_table.plot(metric="event_rate")
```

##### Event rate / WoE transformation

Now that we have checked the binned data, we can transform our original data into WoE or event rate values.

```python
x_transform_woe = optb.transform(x, metric="woe")
```

```python
x_transform_event_rate = optb.transform(x, metric="event_rate")
```

## Advanced

#### Optimal binning Information

The ``OptimalPWBinning`` can print overview information about the options settings, problem statistics, and the solution of the computation. By default, ``print_level=1``.

```python
optb = OptimalPWBinning(name=variable)
optb.fit(x, y)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='mean radius')

If ``print_level=0``, a minimal output including the header, variable name, status, and total time are printed.

```python
optb.information(print_level=0)
```

    optbinning (Version 0.13.0)
    Copyright (c) 2019-2021 Guillermo Navas-Palencia, Apache License 2.0

      Name    : mean radius
      Status  : OPTIMAL

      Time    : 0.1293  sec

If ``print_level>=1``, statistics on the pre-binning phase and the solver are printed. More detailed timing statistics are also included.

```python
optb.information(print_level=1)
```

    optbinning (Version 0.13.0)
    Copyright (c) 2019-2021 Guillermo Navas-Palencia, Apache License 2.0

      Name    : mean radius
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                         7

      Solver statistics
        Type                                auto
        Number of variables                   14
        Number of constraints                 20

      Timing
        Total time                          0.13 sec
        Pre-processing                      0.00 sec   (  0.55%)
        Estimator                           0.01 sec   ( 10.44%)
        Pre-binning                         0.07 sec   ( 50.95%)
        Solver                              0.03 sec   ( 26.36%)
        Post-processing                     0.01 sec   ( 10.95%)

If ``print_level=2``, the list of all options of the ``OptimalBinning`` are displayed.

```python
optb.information(print_level=2)
```

    optbinning (Version 0.13.0)
    Copyright (c) 2019-2021 Guillermo Navas-Palencia, Apache License 2.0

      Begin options
        name                         mean radius   * U
        estimator                            yes   * U
        objective                             l2   * d
        degree                                 1   * d
        continuous                          True   * d
        prebinning_method                   cart   * d
        max_n_prebins                         20   * d
        min_prebin_size                     0.05   * d
        min_n_bins                            no   * d
        max_n_bins                            no   * d
        min_bin_size                          no   * d
        max_bin_size                          no   * d
        monotonic_trend                     auto   * d
        n_subsamples                          no   * d
        max_pvalue                            no   * d
        max_pvalue_policy            consecutive   * d
        outlier_detector                      no   * d
        outlier_params                        no   * d
        user_splits                           no   * d
        special_codes                         no   * d
        split_digits                          no   * d
        solver                              auto   * d
        h_epsilon                           1.35   * d
        quantile                             0.5   * d
        regularization                        no   * d
        reg_l1                               1.0   * d
        reg_l2                               1.0   * d
        random_state                          no   * d
        verbose                            False   * d
      End options

      Name    : mean radius
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                         7

      Solver statistics
        Type                                auto
        Number of variables                   14
        Number of constraints                 20

      Timing
        Total time                          0.13 sec
        Pre-processing                      0.00 sec   (  0.55%)
        Estimator                           0.01 sec   ( 10.44%)
        Pre-binning                         0.07 sec   ( 50.95%)
        Solver                              0.03 sec   ( 26.36%)
        Post-processing                     0.01 sec   ( 10.95%)

#### Binning table statistical analysis

The ``analysis`` method performs a statistical analysis of the binning table, computing the statistics Gini index, Information Value (IV), Jensen-Shannon divergence, and the quality score. Additionally, several statistical significance tests between consecutive bins of the contingency table are performed: a frequentist test using the Chi-square test or the Fisher's exact test, and a Bayesian A/B test using the beta distribution as a conjugate prior of the Bernoulli distribution. The piecewise binning also includes two performance metrics: the average precision and the Brier score.

```python
binning_table.analysis(pvalue_test="chi2")
```

    ---------------------------------------------
    OptimalBinning: Binary Binning Table Analysis
    ---------------------------------------------

      General metrics

        Gini index               0.87503303
        IV (Jeffrey)             4.47808263
        JS (Jensen-Shannon)      0.37566002
        Hellinger                0.44267970
        Triangular               1.20862637
        KS                       0.72390466
        Avg precision            0.95577179
        Brier score              0.08956351
        HHI                      0.15727342
        HHI (normalized)         0.05193260
        Cramer's V               0.80066760
        Quality score            0.00000000

      Significance tests

        Bin A  Bin B  t-statistic      p-value  P[A > B]     P[B > A]
            0      1     0.289789 5.903557e-01  0.697563 3.024368e-01
            1      2     2.285355 1.306002e-01  0.941469 5.853092e-02
            2      3     2.317546 1.279217e-01  0.936661 6.333919e-02
            3      4     2.539025 1.110634e-01  0.950869 4.913141e-02
            4      5    29.418844 5.830790e-08  1.000000 1.898159e-11
            5      6    18.624518 1.591604e-05  0.999997 3.106835e-06

```python
binning_table.analysis(pvalue_test="fisher")
```

    ---------------------------------------------
    OptimalBinning: Binary Binning Table Analysis
    ---------------------------------------------

      General metrics

        Gini index               0.87503303
        IV (Jeffrey)             4.47808263
        JS (Jensen-Shannon)      0.37566002
        Hellinger                0.44267970
        Triangular               1.20862637
        KS                       0.72390466
        Avg precision            0.95577179
        Brier score              0.08956351
        HHI                      0.15727342
        HHI (normalized)         0.05193260
        Cramer's V               0.80066760
        Quality score            0.00000000

      Significance tests

        Bin A  Bin B  odd ratio      p-value  P[A > B]     P[B > A]
            0      1   0.660870 6.855009e-01  0.697563 3.024368e-01
            1      2   0.343985 1.879769e-01  0.941469 5.853092e-02
            2      3   0.447541 1.829850e-01  0.936661 6.333919e-02
            3      4   0.503663 1.153088e-01  0.950869 4.913141e-02
            4      5   0.115702 3.464724e-08  1.000000 1.898159e-11
            5      6   0.037607 4.144258e-05  0.999997 3.106835e-06

#### Event rate / WoE monotonicity

The ``monotonic_trend`` option permits forcing a monotonic trend to the event rate curve. The default setting "auto" should be the preferred option, however, some business constraints might require to impose different trends. The default setting "auto" chooses the monotonic trend most likely to maximize the information value from the options "ascending", "descending", "peak" and "valley" using a machine-learning-based classifier. For this example, we choose Gradient Boosting as an estimator.

```python
variable = "mean texture"
x = df[variable].values
y = data.target
```

```python
optb = OptimalPWBinning(name=variable, estimator=GradientBoostingClassifier())
optb.fit(x, y, lb=0.001, ub=0.999)
```

    OptimalPWBinning(estimator=GradientBoostingClassifier(), name='mean texture')

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 15.05)
      92
      0.161687
      4
      88
      0.931559
      -0.000000

      1
      [15.05, 16.39)
      61
      0.107206
      8
      53
      2.075863
      -0.076059

      2
      [16.39, 17.03)
      29
      0.050967
      6
      23
      2.149342
      -0.080541

      3
      [17.03, 18.46)
      79
      0.138840
      17
      62
      1.113793
      -0.019751

      4
      [18.46, 19.47)
      55
      0.096661
      20
      35
      4.139964
      -0.183682

      5
      [19.47, 20.20)
      36
      0.063269
      18
      18
      4.417902
      -0.197957

      6
      [20.20, 21.71)
      72
      0.126538
      43
      29
      0.933414
      -0.025415

      7
      [21.71, 22.74)
      40
      0.070299
      27
      13
      2.611206
      -0.102697

      8
      [22.74, 24.00)
      29
      0.050967
      24
      5
      0.275868
      -0.000000

      9
      [24.00, 26.98)
      43
      0.075571
      30
      13
      -1.111300
      0.057799

      10
      [26.98, inf)
      33
      0.057996
      15
      18
      0.448108
      0.000000

      11
      Special
      0
      0.000000
      0
      0
      0.000000
      0.000000

      12
      Missing
      0
      0.000000
      0
      0
      0.000000
      0.000000

      Totals

      569
      1.000000
      212
      357
      -
      -

```python
binning_table.plot(metric="event_rate")
```

For example, we can force the variable mean texture to be convex with respect to the probability of having breast cancer.

```python
optb = OptimalPWBinning(name=variable, estimator=GradientBoostingClassifier(),
                        monotonic_trend="convex")
optb.fit(x, y, lb=0.001, ub=0.999)
```

    OptimalPWBinning(estimator=GradientBoostingClassifier(),
                     monotonic_trend='convex', name='mean texture')

```python
binning_table = optb.binning_table
binning_table.build()
binning_table.plot(metric="event_rate")
```

#### User-defined split points

In some situations, we have defined split points or bins required to satisfy a priori belief,  knowledge or business constraint. The ``OptimalPWBinning`` permits to pass user-defined split points. The supplied information is used as a pre-binning, disallowing any pre-binning method set by the user. Furthermore, version 0.5.0 introduces ``user_splits_fixed`` parameter, to allow the user to fix some user-defined splits, so these must appear in the solution.

Example numerical variable:

```python
user_splits =       [  14,    15,    16,    17,    20,    21,    22,    27]
user_splits_fixed = [False, True,  True, False, False, False, False, False]
```

```python
optb = OptimalPWBinning(name=variable, user_splits=user_splits,
                        user_splits_fixed=user_splits_fixed)

optb.fit(x, y)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='mean texture',
                     user_splits=[14, 15, 16, 17, 20, 21, 22, 27],
                     user_splits_fixed=[False, True, True, False, False, False,
                                        False, False])

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 14.00)
      54
      0.094903
      2
      52
      1.153974
      -0.020565

      1
      [14.00, 15.00)
      37
      0.065026
      2
      35
      1.354515
      -0.034890

      2
      [15.00, 16.00)
      43
      0.075571
      7
      36
      1.244455
      -0.027552

      3
      [16.00, 20.00)
      210
      0.369069
      59
      151
      1.570900
      -0.047955

      4
      [20.00, 21.00)
      45
      0.079086
      26
      19
      1.874636
      -0.063142

      5
      [21.00, 22.00)
      49
      0.086116
      30
      19
      1.788369
      -0.059034

      6
      [22.00, 27.00)
      99
      0.173989
      71
      28
      1.708444
      -0.055401

      7
      [27.00, inf)
      32
      0.056239
      15
      17
      0.789092
      -0.021351

      8
      Special
      0
      0.000000
      0
      0
      0.000000
      0.000000

      9
      Missing
      0
      0.000000
      0
      0
      0.000000
      0.000000

      Totals

      569
      1.000000
      212
      357
      -
      -

```python
optb.information()
```

    optbinning (Version 0.13.0)
    Copyright (c) 2019-2021 Guillermo Navas-Palencia, Apache License 2.0

      Name    : mean texture
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                         8

      Solver statistics
        Type                                auto
        Number of variables                   16
        Number of constraints                 23

      Timing
        Total time                          0.16 sec
        Pre-processing                      0.00 sec   (  0.47%)
        Estimator                           0.02 sec   ( 11.20%)
        Pre-binning                         0.09 sec   ( 56.21%)
        Solver                              0.04 sec   ( 23.07%)
        Post-processing                     0.01 sec   (  8.35%)

#### Performance: choosing a solver

``OptimalPWBinning`` uses the RoPWR library to solving the piecewise regression problem. See https://github.com/guillermo-navas-palencia/ropwr for details.

#### Robustness: choosing an objective function and regularization

``OptimalPWBinning`` uses the RoPWR library to solving the piecewise regression problem. See https://github.com/guillermo-navas-palencia/ropwr for details.

#### Missing data and special codes

For this example, let's load data from the FICO Explainable Machine Learning Challenge: https://community.fico.com/s/explainable-machine-learning-challenge

```python
df = pd.read_csv("data/FICO_challenge/heloc_dataset_v1.csv", sep=",")
```

The data dictionary of this challenge includes three special values/codes:

* -9 No Bureau Record or No Investigation
* -8 No Usable/Valid Trades or Inquiries
* -7 Condition not Met (e.g. No Inquiries, No Delinquencies)

```python
special_codes = [-9, -8, -7]
```

```python
variable = "AverageMInFile"
x = df[variable].values
y = df.RiskPerformance.values
```

```python
df.RiskPerformance.unique()
```

    array(['Bad', 'Good'], dtype=object)

Target is a categorical dichotomic variable, which can be easily transform into numerical.

```python
mask = y == "Bad"
y[mask] = 1
y[~mask] = 0
y = y.astype(int)
```

For the sake of completeness, we include a few missing values

```python
idx = np.random.randint(0, len(x), 500)
x = x.astype(float)
x[idx] = np.nan
```

```python
optb = OptimalPWBinning(name=variable, special_codes=special_codes, solver="osqp")

optb.fit(x, y, lb=0, ub=1)
```

    OptimalPWBinning(estimator=LogisticRegression(), name='AverageMInFile',
                     solver='osqp', special_codes=[-9, -8, -7])

```python
optb.information(print_level=1)
```

    optbinning (Version 0.13.0)
    Copyright (c) 2019-2021 Guillermo Navas-Palencia, Apache License 2.0

      Name    : AverageMInFile
      Status  : OPTIMAL

      Pre-binning statistics
        Number of bins                        13

      Solver statistics
        Type                                osqp
        Number of variables                   26
        Number of constraints              18862

      Timing
        Total time                          0.44 sec
        Pre-processing                      0.00 sec   (  0.53%)
        Estimator                           0.04 sec   (  8.64%)
        Pre-binning                         0.12 sec   ( 27.57%)
        Solver                              0.26 sec   ( 59.10%)
        Post-processing                     0.02 sec   (  3.80%)

```python
binning_table = optb.binning_table
binning_table.build()
```

      Bin
      Count
      Count (%)
      Non-event
      Event
      c0
      c1

      0
      (-inf, 30.50)
      549
      0.052491
      99
      450
      0.791990
      -0.003080

      1
      [30.50, 41.50)
      545
      0.052108
      147
      398
      0.804033
      -0.003475

      2
      [41.50, 48.50)
      515
      0.049240
      145
      370
      0.808132
      -0.003574

      3
      [48.50, 54.50)
      520
      0.049718
      185
      335
      0.813688
      -0.003688

      4
      [54.50, 59.50)
      492
      0.047041
      198
      294
      0.817258
      -0.003754

      5
      [59.50, 64.50)
      603
      0.057654
      258
      345
      0.820850
      -0.003814

      6
      [64.50, 69.50)
      654
      0.062530
      300
      354
      0.822749
      -0.003844

      7
      [69.50, 75.50)
      808
      0.077254
      394
      414
      0.828317
      -0.003924

      8
      [75.50, 81.50)
      765
      0.073143
      400
      365
      0.819068
      -0.003801

      9
      [81.50, 95.50)
      1520
      0.145329
      863
      657
      0.841207
      -0.004073

      10
      [95.50, 101.50)
      477
      0.045607
      280
      197
      0.608803
      -0.001639

      11
      [101.50, 116.50)
      876
      0.083756
      538
      338
      1.161241
      -0.007082

      12
      [116.50, inf)
      1088
      0.104025
      709
      379
      0.483154
      -0.001261

      13
      Special
      560
      0.053542
      250
      310
      0.553571
      0.000000

      14
      Missing
      487
      0.046563
      234
      253
      0.519507
      0.000000

      Totals

      10459
      1.000000
      5000
      5459
      -
      -

Note the dashed bins 10 and 11, corresponding to the special codes bin and the missing bin, respectively.

```python
binning_table.plot(metric="event_rate")
```

#### Verbosity option

For debugging purposes, we can print information on each step of the computation by triggering the ``verbose`` option.

```python
optb = OptimalPWBinning(name=variable, solver="osqp", verbose=True)
optb.fit(x, y, lb=0, ub=1)
```

    2021-10-23 20:11:41,921 | INFO : Optimal piecewise binning started.
    2021-10-23 20:11:41,924 | INFO : Options: check parameters.
    2021-10-23 20:11:41,926 | INFO : Pre-processing started.
    2021-10-23 20:11:41,927 | INFO : Pre-processing: number of samples: 10459
    2021-10-23 20:11:41,929 | INFO : Pre-processing: number of clean samples: 9972
    2021-10-23 20:11:41,940 | INFO : Pre-processing: number of missing samples: 487
    2021-10-23 20:11:41,942 | INFO : Pre-processing: number of special samples: 0
    2021-10-23 20:11:41,944 | INFO : Pre-processing terminated. Time: 0.0005s
    2021-10-23 20:11:41,945 | INFO : Pre-binning: set logistic regression as an estimator.
    2021-10-23 20:11:41,947 | INFO : Pre-binning: estimator fitting started.
    2021-10-23 20:11:42,015 | INFO : Pre-binning: estimator terminated. Time 0.0694s.
    2021-10-23 20:11:42,016 | INFO : Pre-binning: optimal binning started.
    2021-10-23 20:11:42,139 | INFO : Pre-binning: number of splits: 9.
    2021-10-23 20:11:42,140 | INFO : Pre-binning: no need for subsamples.
    2021-10-23 20:11:42,141 | INFO : Pre-binning: optimal binning terminated. Time 0.1243s.
    2021-10-23 20:11:42,142 | INFO : Optimizer started.
    2021-10-23 20:11:42,145 | INFO : Optimizer: descending monotonic trend.
    ===============================================================================
                                         CVXPY
                                        v1.1.12
    ===============================================================================
    (CVXPY) Oct 23 08:11:42 PM: Your problem has 20 variables, 4 constraints, and 0 parameters.
    (CVXPY) Oct 23 08:11:42 PM: It is compliant with the following grammars: DCP, DQCP
    (CVXPY) Oct 23 08:11:42 PM: (If you need to solve this problem multiple times, but with different data, consider using parameters.)
    (CVXPY) Oct 23 08:11:42 PM: CVXPY will first compile your problem; then, it will invoke a numerical solver to obtain a solution.
    -------------------------------------------------------------------------------
                                      Compilation
    -------------------------------------------------------------------------------
    (CVXPY) Oct 23 08:11:42 PM: Compiling problem (target solver=OSQP).
    (CVXPY) Oct 23 08:11:42 PM: Reduction chain: CvxAttr2Constr -> Qp2SymbolicQp -> QpMatrixStuffing -> OSQP
    (CVXPY) Oct 23 08:11:42 PM: Applying reduction CvxAttr2Constr
    (CVXPY) Oct 23 08:11:42 PM: Applying reduction Qp2SymbolicQp
    (CVXPY) Oct 23 08:11:42 PM: Applying reduction QpMatrixStuffing
    (CVXPY) Oct 23 08:11:42 PM: Applying reduction OSQP
    (CVXPY) Oct 23 08:11:42 PM: Finished problem compilation (took 1.250e-01 seconds).
    -------------------------------------------------------------------------------
                                    Numerical solver
    -------------------------------------------------------------------------------
    (CVXPY) Oct 23 08:11:42 PM: Invoking solver OSQP  to obtain a solution.
    -----------------------------------------------------------------
               OSQP v0.6.0  -  Operator Splitting QP Solver
                  (c) Bartolomeo Stellato,  Goran Banjac
            University of Oxford  -  Stanford University 2019
    -----------------------------------------------------------------
    problem:  variables n = 20, constraints m = 19973
              nnz(P) + nnz(A) = 39964
    settings: linear system solver = qdldl,
              eps_abs = 1.0e-05, eps_rel = 1.0e-05,
              eps_prim_inf = 1.0e-04, eps_dual_inf = 1.0e-04,
              rho = 1.00e-01 (adaptive),
              sigma = 1.00e-06, alpha = 1.60, max_iter = 10000
              check_termination: on (interval 25),
              scaling: on, scaled_termination: off
              warm start: on, polish: on, time_limit: off

    iter   objective    pri res    dua res    rho        time
       1  -4.2858e+01   1.47e-03   2.00e+05   1.00e-01   2.28e-02s
     200  -2.3356e+03   4.66e-03   1.64e+04   1.97e-02   1.09e-01s
     400  -2.7738e+03   9.07e-03   5.72e+03   1.97e-02   1.88e-01s
     575  -2.8268e+03   8.00e-06   1.58e-01   1.57e-02   2.70e-01s
    plsh  -2.8268e+03   1.93e-16   4.42e-11   --------   2.71e-01s

    status:               solved
    solution polish:      successful
    number of iterations: 575
    optimal objective:    -2826.8492
    run time:             2.71e-01s
    optimal rho estimate: 2.01e-02

    -------------------------------------------------------------------------------
                                        Summary
    -------------------------------------------------------------------------------
    (CVXPY) Oct 23 08:11:42 PM: Problem status: optimal
    (CVXPY) Oct 23 08:11:42 PM: Optimal value: -2.827e+03
    (CVXPY) Oct 23 08:11:42 PM: Compilation took 1.250e-01 seconds
    (CVXPY) Oct 23 08:11:42 PM: Solver (including time spent in interface) took 2.794e-01 seconds
    2021-10-23 20:11:42,620 | INFO : Optimizer terminated. Time: 0.4740s
    2021-10-23 20:11:42,621 | INFO : Post-processing started.
    2021-10-23 20:11:42,622 | INFO : Post-processing: compute binning information.
    2021-10-23 20:11:42,632 | INFO : Post-processing: compute performance metrics.
    2021-10-23 20:11:42,652 | INFO : Post-processing terminated. Time: 0.0262s
    2021-10-23 20:11:42,656 | INFO : Optimal piecewise binning terminated. Status: OPTIMAL. Time: 0.7346s

    OptimalPWBinning(estimator=LogisticRegression(), name='AverageMInFile',
                     solver='osqp', verbose=True)


```python
%matplotlib inline

import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
import scipy

sns.set_context('notebook', font_scale=1.5)
```

```python
import warnings
warnings.simplefilter('ignore', FutureWarning)
```

**1**. (25 points)

In this exercise, we will practice using Pandas dataframes to explore and summarize a data set `heart`.

This data contains the survival time after receiving a heart transplant, the age of the patient and whether or not the survival time was censored

- Number of Observations - 69
- Number of Variables - 3

Variable name definitions::

- survival - Days after surgery until death
- censors - indicates if an observation is censored. 1 is uncensored
- age - age at the time of surgery

Answer the following questions (5 points each) with respect to the `heart` data set:

- How many patients were censored?
- What is the correlation coefficient between age and survival for uncensored patients?
- What is the average age for censored and uncensored patients?
- What is the average survival time for censored and uncensored patients under the age of 45?
- What is the survival time of the youngest and oldest uncensored patient?

```python
import statsmodels.api as sm
heart = sm.datasets.heart.load_pandas().data
heart.head(n=6)
```

      survival
      censors
      age

      0
      15.0
      1.0
      54.3

      1
      3.0
      1.0
      40.4

      2
      624.0
      1.0
      51.0

      3
      46.0
      1.0
      42.5

      4
      127.0
      1.0
      48.0

      5
      64.0
      1.0
      54.6

```python
#How many patients were censored?
heart[heart.censors==0].shape[0]
```

    24

24 patients were censored.

```python
#What is the correlation coefficient between age and survival for uncensored patients?
uncensored = heart[heart.censors==1]
scipy.stats.pearsonr(uncensored.age, uncensored.survival)[0]
```

    0.0032564992832119144

The correlation coefficient between age and survival for uncensored patients is 0.003.

```python
#What is the average age for censored and uncensored patients?
heart.groupby("censors").agg({'age':"mean"})
```

      age

      censors

      0.0
      41.729167

      1.0
      48.484444

The average age for censored patients is 48.48, for uncensored patients is 41.73.

```python
#What is the average survival time for censored and uncensored patients under the age of 45?
heart[heart.age<45].groupby("censors").agg({"survival":"mean"})
```

      survival

      censors

      0.0
      712.818182

      1.0
      169.909091

The average survival time for censored patients under 45 years old is 169.9 days.
The average survival time for uncensored patients under 45 years old is 712.8 days.

```python
#What is the survival time of the youngest and oldest uncensored patient?
survival_min = uncensored[uncensored.age == uncensored.age.min()].survival
survival_max = uncensored[uncensored.age == uncensored.age.max()].survival
```

```python
survival_min
```

    41    228.0
    Name: survival, dtype: float64

The survival time of the yougest uncensored patient is 228.

```python
survival_max
```

    17    60.0
    Name: survival, dtype: float64

The survival time of the oldest uncensored patient is 60.

**2**. (35 points)

- Consider a sequence of $n$ Bernoulli trials with success probabilty $p$ per trial. A string of consecutive successes is known as a success *run*. Write a function that returns the counts for runs of length $k$ for each $k$ observed in a dictionary. (10 points)

For example: if the trials were [0, 1, 0, 1, 1, 0, 0, 0, 0, 1], the function should return
```
{1: 2, 2: 1})
```

Test that it does so.

- What is the probability of observing at least one run of length 5 or more when $n=100$ and $p=0.5$?. Estimate this from 100,000 simulated experiments. Is this more, less or equally likely than finding runs of length 7 or more when $p=0.7$? (10 points)

- There is an exact solution

$$
s_n = \sum_{i=1}^n{f_i} \\
f_n = u_n - \sum_{i=1}^{n-1} {f_i u_{n-i}} \\
u_n = p^k - \sum_{i=1}^{k-1} u_{n-i} p^i
$$

Implement the exact solution using caching to avoid re-calculations and calculate the same two probabilities found by simulation. (15 points)

```python
def len_run(str_list):
    s_str = "".join([str(s) for s in str_list]).split("0")
    run = [s for s in s_str if s != ""]
    unique = set(run)
    l = [len(u) for u in unique]
    count = []
    for e in unique:
        count.append(run.count(e))

    return dict(zip(l, count))
```

```python
s = [0, 1, 0, 1, 1, 0, 0, 0, 0, 1]
len_run(s)
```

    {1: 2, 2: 1}

```python
n = 100
p = 0.5
N = 100000
result = []
for i in range(N):
    s = [np.random.binomial(1,p) for j in range(n)]
    run_len = len_run(s).keys()
    whether_5 = np.sum([e >= 5 for e in run_len])
    if whether_5>=1:
        result.append(1)
    else:
        result.append(0)
```

```python
np.mean(result)
```

    0.81115999999999999

```python
n = 100
p = 0.7
N = 100000
result = []
for i in range(N):
    s = [np.random.binomial(1,p) for j in range(n)]
    run_len = len_run(s).keys()
    whether_5 = np.sum([e >= 7 for e in run_len])
    if whether_5>=1:
        result.append(1)
    else:
        result.append(0)
```

```python
np.mean(result)
```

    0.94891999999999999

```python
from functools import lru_cache

@lru_cache()
def u(n,k,p):
    if n < k:
        return 0
    else:
        return p**k - sum(u(n-i, k, p)*p**i for i in range(1, k))

@lru_cache()
def f(n,k,p):
    return u(n,k,p) - sum(f(i,k,p)*u(n-i,k,p) for i in range(1,n))

@lru_cache()
def s(n,k,p):
    return sum(f(i,k,p) for i in range(1, n+1))
```

```python
s(100,5,0.5)
```

    0.8101095991963579

```python
s(100,7,0.7)
```

    0.9491817984156692

**3**. (40 points)

Given matrix $M$
```python
      [[7, 8, 8],
       [1, 3, 8],
       [9, 2, 1]]
```

- Normalize the given matrix $M$ so that all rows sum to 1.0.  (5 points)
- The normalized matrix can then be considered as a transition matrix $P$ for a Markov chain. Find the stationary distribution of this matrix in the following ways using `numpy` and `numpy.linalg` (or `scipy.linalg`):
    - By repeated matrix multiplication of a random probability vector $v$ (a row vector normalized to sum to 1.0) with $P$ using matrix multiplication with `np.dot`. (5 points)
    - By raising the matrix $P$ to some large power until it doesn't change with higher powers (see `np.linalg.matrix_power`) and then calculating $vP$ (10 points)
    - From the equation for stationarity $wP = w$, we can see that $w$ must be a left eigenvector of $P$ with eigenvalue $1$ (Note: np.linalg.eig returns the right eigenvectors, but the left eigenvector of a matrix is the right eigenvector of the transposed matrix). Use this to find $w$ using `np.linalg.eig`. (20 points)

Suppose $w = (w_1, w_2, w_3)$. Then from $wP = w$, we have:
\begin{align}
w_1 P_{11} + w_2 P_{21} + w_3 P_{31} &= w_1 \\
w_1 P_{12} + w_2 P_{22} + w_3 P_{32} &= w_2 \\
w_1 P_{13} + w_2 P_{23} + w_3 P_{331} &= w_3 \\
\end{align}
This is a singular system, but we also know that $w_1 + w_2 + w_3 = 1$. Use these facts to set up a linear system of equations that can be solved with `np.linalg.solve` to find $w$.

```python
M = np.array([[7,8,8],[1,3,8],[9,2,1]])
M
```

    array([[7, 8, 8],
           [1, 3, 8],
           [9, 2, 1]])

```python
sum_row = M.sum(axis=1)
normalized_M = M/sum_row[:,None]
normalized_M
```

    array([[ 0.30434783,  0.34782609,  0.34782609],
           [ 0.08333333,  0.25      ,  0.66666667],
           [ 0.75      ,  0.16666667,  0.08333333]])

```python
P = normalized_M
v = np.array([0.2, 0.7, 0.1])
```

```python
v_old = v
v_new = np.dot(v_old,P)
while any(v_old != v_new):
    v_old = v_new
    v_new = np.dot(v_old, P)
w1 = v_old
w1
```

    array([ 0.39862184,  0.2605972 ,  0.34078096])

```python
P_s = np.linalg.matrix_power(P,20)
P_s
```

    array([[ 0.39862184,  0.2605972 ,  0.34078096],
           [ 0.39862184,  0.2605972 ,  0.34078096],
           [ 0.39862184,  0.2605972 ,  0.34078096]])

```python
w2 = np.dot(v, P_s)
w2
```

    array([ 0.39862184,  0.2605972 ,  0.34078096])

```python
W,V = np.linalg.eig(P.T)
w3 = V[:, 0]/V[:, 0].sum()
w3
```

    array([ 0.39862184+0.j,  0.26059720+0.j,  0.34078096+0.j])

```python
A = np.r_[P.T-np.eye(3),[np.ones(3)]][1:,:]
b = np.array([0,0,1])
w4 = np.linalg.solve(A,b)
w4
```

    array([ 0.39862184,  0.2605972 ,  0.34078096])

```python

```


# The Implicit Crank-Nicolson Difference Equation for the Heat Equation
## The Heat Equation
The Heat Equation is the first order in time ($t$) and second order in space ($x$) Partial Differential Equation:
$$  \frac{\partial u}{\partial t} = \frac{\partial^2 u}{\partial x^2},$$
The equation describes heat transfer on a domain
$$ \Omega = \{ t \geq, 0\leq x \leq 1\}. $$
with an initial condition at time $t=0$ for all $x$ and boundary condition on the left ($x=0$) and right side ($x=1$).

## Crank-Nicolson Difference method
This note book will illustrate the Crank-Nicolson Difference method for the Heat Equation with the __initial conditions__
$$ u(x,0)=2x, \ \ 0 \leq x \leq \frac{1}{2}, $$
$$ u(x,0)=2(1-x), \ \ \frac{1}{2}  \leq x \leq 1, $$
and __boundary condition__
$$ u(0,t)=0,  u(1,t)=0. $$

```python
# LIBRARY
# vector manipulation
import numpy as np
# math functions
import math

# THIS IS FOR PLOTTING
%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import warnings
warnings.filterwarnings("ignore")
from IPython.display import HTML

HTML('''
<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>''')

```

<form action="javascript:code_toggle()"><input type="submit" value="Click here to toggle on/off the raw code."></form>

## Discete Grid
The region $\Omega$ is discretised into a uniform mesh $\Omega_h$. In the space $x$ direction into $N$ steps giving a stepsize of
$$ h=\frac{1-0}{N},$$
resulting in
$$x[i]=0+ih,$$
and into $N_t$ steps in the time $t$ direction giving a stepsize of
$$ k=\frac{1-0}{N}$$
resulting in
$$t[i]=0+ik.$$
The Figure below shows the discrete grid points for $N=10$ and $N_t=15$, the red dots are the unknown values, the green dots are the known boundary conditions and the blue dots are the known initial conditions of the Heat Equation.

```python
N=10
Nt=1000
h=1/N
k=1/Nt
r=k/(h*h)
time_steps=15
time=np.arange(0,(time_steps+.5)*k,k)
x=np.arange(0,1.0001,h)
X, Y = np.meshgrid(x, time)
fig = plt.figure()
plt.plot(X,Y,'ro');
plt.plot(x,0*x,'bo',label='Initial Condition');
plt.plot(np.ones(time_steps+1),time,'go',label='Boundary Condition');
plt.plot(x,0*x,'bo');
plt.plot(0*time,time,'go');
plt.xlim((-0.02,1.02))
plt.xlabel('x')
plt.ylabel('time (ms)')
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.title(r'Discrete Grid $\Omega_h,$ h= %s, k=%s'%(h,k),fontsize=24,y=1.08)
plt.show();
```

## Discrete Initial and Boundary Conditions

The discrete initial conditions are
$$ w[i,0]=2x[i], \ \ 0 \leq x[i] \leq \frac{1}{2} $$
$$ w[i,0]=2(1-x[i]), \ \ \frac{1}{2}  \leq x[i] \leq 1 $$
and the discete boundary conditions are
$$ w[0,j]=0,  w[10,j]=0, $$
where $w[i,j]$ is the numerical approximation of $U(x[i],t[j])$.

The Figure below plots the inital and boundary conditions for $t[0]=0.$

```python
w=np.zeros((N+1,time_steps+1))
b=np.zeros(N-1)
# Initial Condition
for i in range (1,N):
    w[i,0]=2*x[i]
    if x[i]>0.5:
        w[i,0]=2*(1-x[i])

# Boundary Condition
for k in range (0,time_steps):
    w[0,k]=0
    w[N,k]=0

fig = plt.figure(figsize=(8,4))
plt.plot(x,w[:,0],'o:',label='Initial Condition')
plt.plot(x[[0,N]],w[[0,N],0],'go',label='Boundary Condition t[0]=0')
#plt.plot(x[N],w[N,0],'go')
plt.xlim([-0.1,1.1])
plt.ylim([-0.1,1.1])
plt.title('Intitial and Boundary Condition',fontsize=24)
plt.xlabel('x')
plt.ylabel('w')
plt.legend(loc='best')
plt.show()
```

## The Implicit Crank-Nicolson Difference Equation
The implicit Crank-Nicolson difference equation of the Heat Equation is
\begin{equation*}
\frac{w_{ij+1}-w_{ij}}{k}=\frac{1}{2}\left(\frac{w_{i+1j+1}-2w_{ij+1}+w_{i-1j+1}}{h^2}+\frac{w_{i+1j}-2w_{ij}+w_{i-1j}}{h^2}\right)
\end{equation*}
Rearranging the equation we get
\begin{equation}
-rw_{i-1j+1}+(2+2r)w_{ij+1}-rw_{i+1j+1}=-rw_{i-1j}+(2-2r)w_{ij}+rw_{i+1j}
\end{equation}
for $i=1,...9$ where $r=\frac{k}{h^2}$.

This gives the formula for the unknown term $w_{ij+1}$ at the $(ij+1)$ mesh points
in terms of $x[i]$ along the jth time row.

Hence we can calculate the unknown pivotal values of $w$ along the first row of $j=1$ in terms of the known boundary conditions.

This can be written in matrix form
$$ A\mathbf{w}_{j+1}=B\mathbf{w}_{j} +\mathbf{b}_{j}+\mathbf{b}_{j+1} $$
for which $A$ is a $9\times9$ matrix:
$$
\left(\begin{array}{cccc cccc}
2+2r&-r& 0&0&0 &0&0&0\\
-r&2+2r&-r&0&0&0 &0&0&0\\
0&-r&2+2r &-r&0&0& 0&0&0\\
0&0&-r&2+2r &-r&0&0& 0&0\\
0&0&0&-r&2+2r &-r&0&0& 0\\
0&0&0&0&-r&2+2r &-r&0&0\\
0&0&0&0&0&-r&2+2r &-r&0\\
0&0&0&0&0&0&-r&2+2r&-r\\
0&0&0&0&0&0&0&-r&2+2r\\
\end{array}\right)
\left(\begin{array}{c}
w_{1j+1}\\
w_{2j+1}\\
w_{3j+1}\\
w_{4j+1}\\
w_{5j+1}\\
w_{6j+1}\\
w_{7j+1}\\
w_{8j+1}\\
w_{9j+1}\\
\end{array}\right)=
\left(\begin{array}{cccc cccc}
2-2r&r& 0&0&0 &0&0&0\\
r&2-2r&r&0&0&0 &0&0&0\\
0&r&2-2r &r&0&0& 0&0&0\\
0&0&r&2-2r &r&0&0& 0&0\\
0&0&0&r&2-2r &r&0&0& 0\\
0&0&0&0&r&2-2r &r&0&0\\
0&0&0&0&0&r&2-2r &r&0\\
0&0&0&0&0&0&r&2-2r&r\\
0&0&0&0&0&0&0&r&2-2r\\
\end{array}\right)
\left(\begin{array}{c}
w_{1j}\\
w_{2j}\\
w_{3j}\\
w_{4j}\\
w_{5j}\\
w_{6j}\\
w_{7j}\\
w_{8j}\\
w_{9j}\\
\end{array}\right)+
\left(\begin{array}{c}
rw_{0j}\\
0\\
0\\
0\\
0\\
0\\
0\\
0\\
rw_{10j}\\
\end{array}\right)+
\left(\begin{array}{c}
rw_{0j+1}\\
0\\
0\\
0\\
0\\
0\\
0\\
0\\
rw_{10j+1}\\
\end{array}\right).
$$
It is assumed that the boundary values $w_{0j}$ and $w_{10j}$ are known for $j=1,2,...$, and $w_{i0}$ for $i=0,...,10$ is the initial condition.

The Figure below shows the matrix $A$ and its inverse $A^{-1}$ in color plot form for $r$.

```python
A=np.zeros((N-1,N-1))
B=np.zeros((N-1,N-1))
for i in range (0,N-1):
    A[i,i]=2+2*r
    B[i,i]=2-2*r

for i in range (0,N-2):
    A[i+1,i]=-r
    A[i,i+1]=-r
    B[i+1,i]=r
    B[i,i+1]=r

Ainv=np.linalg.inv(A)
fig = plt.figure(figsize=(12,4));
plt.subplot(121)
plt.imshow(A,interpolation='none');
plt.xticks(np.arange(N-1), np.arange(1,N-0.9,1));
plt.yticks(np.arange(N-1), np.arange(1,N-0.9,1));
clb=plt.colorbar();
clb.set_label('Matrix elements values');
clb.set_clim((-1,1));
plt.title('Matrix A r=%s'%(np.round(r,3)),fontsize=24)

plt.subplot(122)
plt.imshow(B,interpolation='none');
plt.xticks(np.arange(N-1), np.arange(1,N-0.9,1));
plt.yticks(np.arange(N-1), np.arange(1,N-0.9,1));
clb=plt.colorbar();
clb.set_label('Matrix elements values');
clb.set_clim((-1,1));
plt.title(r'Matrix $B$ r=%s'%(np.round(r,3)),fontsize=24)
fig.tight_layout()
plt.show();
```

## Results
To numerically approximate the solution at $t[1]$ the matrix equation becomes
$$ \mathbf{w}_{1}=A^{-1}(B\mathbf{w}_{0} +\mathbf{b}_{1}+\mathbf{b}_{0}) $$
where all the right hand side is known.
To approximate solution at time $t[2]$ we use the matrix equation
$$ \mathbf{w}_{2}=A^{-1}(B\mathbf{w}_{1}+\mathbf{b}_{2} +\mathbf{b}_{1}). $$
Each set of numerical solutions $w[i,j]$ for all $i$ at the previous time step is used to approximate the solution $w[i,j+1]$.
The left and right plot below show the numerical approximation $w[i,j]$ of the Heat Equation using the BTCS method for $x[i]$ for $i=0,...,10$ and time steps $t[j]$ for $j=1,...,15$.

```python
fig = plt.figure(figsize=(12,6))
plt.subplot(121)
for j in range (1,time_steps+1):
    b[0]=r*w[0,j-1]+r*w[0,j]
    b[N-2]=r*w[N,j-1]+r*w[N,j]
    v=np.dot(B,w[1:(N),j-1])
    w[1:(N),j]=np.dot(Ainv,v+b)
    plt.plot(x,w[:,j],'o:',label='t[%s]=%s'%(j,time[j]))
plt.xlabel('x')
plt.ylabel('w')
#plt.legend(loc='bottom', bbox_to_anchor=(0.5, -0.1))
plt.legend(bbox_to_anchor=(-.4, 1), loc=2, borderaxespad=0.)

plt.subplot(122)
plt.imshow(w.transpose())
plt.xticks(np.arange(len(x)), x)
plt.yticks(np.arange(len(time)), time)
plt.xlabel('x')
plt.ylabel('time')
clb=plt.colorbar()
clb.set_label('Temperature (w)')
plt.suptitle('Numerical Solution of the  Heat Equation r=%s'%(np.round(r,3)),fontsize=24,y=1.08)
fig.tight_layout()
plt.show()
```

```python

```

Chapter 08 - Heat Equations/.ipynb_checkpoints/04_Heat Equation-CN-Different_Initial_Boundary-checkpoint.ipynb


# Jupyter like a pro

In this third notebook of the tutorial ["The World of Jupyter"](https://github.com/barbagroup/jupyter-tutorial/blob/master/World-of-Jupyter.md), we want to leave you with pro tips for using Jupyter in your future work.

## Importing libraries

First, a word on importing libraries. Previously, we used the following command to load all the functions in the **NumPy** library:
```python
import numpy
```
Once you execute that command in a code cell, you call any **NumPy** function by prepending the library name, e.g., `numpy.linspace()`, [`numpy.ones()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.ones.html#numpy.ones), [`numpy.zeros()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.zeros.html#numpy.zeros), [`numpy.empty()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.empty.html#numpy.empty), [`numpy.copy()`](https://docs.scipy.org/doc/numpy/reference/generated/numpy.copy.html#numpy.copy), and so on (explore the documentation for these very useful functions!).

But, you will find _a lot_ of sample code online that uses a different syntax for importing. They will do:
```python
import numpy as np
```
All this does is create an alias for `numpy` with the shorter string `np`, so you then would call a **NumPy** function like this: `np.linspace()`. This is just an alternative way of doing it, for lazy people that find it too long to type `numpy` and want to save 3 characters each time. For the not-lazy, typing `numpy` is more readable and beautiful. We like it better like this:

```python
import numpy
```

## Make your plots beautiful

When you make a plot using **Matplotlib**, you have many options to make your plots beautiful and publication-ready. Here are some of our favorite tricks.

First, let's load the `pyplot` module—and remember, `%matplotlib notebook` gets our plots inside the notebook (instead of a pop-up).

Our first trick is `rcparams`: we use it to customize the appearance of the plots. Here, we set the default font to a serif type of size 14 pt and make the size of the font for the axes labels 18 pt. Honestly, the default font is too small.

```python
from matplotlib import pyplot
%matplotlib notebook
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 14
pyplot.rcParams['axes.labelsize'] = 18
```

The following example is from a tutorial by [Dr. Justin Bois](http://www.justinbois.info), a lecturer in Biology and Biological Engineering at Caltech, for his class in [Data Analysis in the Biological Sciences](http://bebi103.caltech.edu/2015/) (2015). He has given us permission to use it.

```python
# Get an array of 100 evenly spaced points from 0 to 2*pi
x = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

# Make a pointwise function of x with exp(sin(x))
y = numpy.exp(numpy.sin(x))
```

Here, we added comments in the Python code with the `#` mark. Comments are often useful not only for others who read the code, but as a "note to self" for the future you!

Let's see how the plot looks with the new font settings we gave **Matplotlib**, and make the plot more friendly by adding axis labels. This is always a good idea!

```python
pyplot.figure()
pyplot.plot(x, y, color='k', linestyle='-')
pyplot.xlabel('$x$')
pyplot.ylabel('$\mathrm{e}^{\sin(x)}$')
pyplot.xlim(0.0, 2.0 * numpy.pi);
```

    <IPython.core.display.Javascript object>

Did you see how **Matplotlib** understands LaTeX mathematics? That is beautiful. The function `pyplot.xlim()` specifies the limits of the x-axis (you can also manually specify the y-axis, if the defaults are not good for you).

Continuing with the tutorial example by Justin Bois, let's have some mathematical fun and numerically compute the derivative of this function, using finite differences. We need to apply the following mathematical formula on all the discrete points of the `x` array:

\begin{equation}
\frac{\mathrm{d}y(x_i)}{\mathrm{d}x} \approx \frac{y(x_{i+1}) - y(x_i)}{x_{i+1} - x_i}.
\end{equation}

By the way, did you notice how we can typeset beautiful mathematics within a markdown cell? The Jupyter notebook is happy typesetting mathematics using LaTeX syntax.

Since this notebook is _"Jupyter like a pro,"_ we will define a custom Python function to compute the forward difference. It is good form to define custon functions to make your code modular and reusable.

```python
def forward_diff(y, x):
    """Compute derivative by forward differencing."""

    # Use numpy.empty to make an empty array to put our derivatives in
    deriv = numpy.empty(y.size - 1)

    # Use a for-loop to go through each point and compute the derivative.
    for i in range(deriv.size):
        deriv[i] = (y[i+1] - y[i]) / (x[i+1] - x[i])

    # Return the derivative (a NumPy array)
    return deriv

# Call the function to perform finite differencing
deriv = forward_diff(y, x)
```

Notice how we define a function with the `def` statement, followed by our custom name for the fuction, the function arguments in parenthesis, and ending the statement with a colon. The contents of the function are indicated by the indentation (four spaces, in this case), and the `return` statement indicates what the function returns to the code that called it (in this case, the contents of the variable `deriv`). Right after the function definition (in between triple quotes) is the _docstring_, a short text documenting what the function does. It is good form to always write docstrings for your functions!

In our custom `forward_diff()` function, we used `numpy.empty()` to create an empty array of length `y.size-1`, that is, one less than the length of the array `y`. Then, we start a for-loop that iterates over values of `i` using the [**`range()`**](https://docs.python.org/3/library/functions.html#func-range) function of Python. This is a very useful function that you should think about for a little bit. What it does is create a list of integers. If you give it just one argument, it's a _"stop"_ argument: `range(stop)` creates a list of integers from `0` to `stop-1`, i.e., the list has `stop` numbers in it because it always starts at zero. But you can also give it a _"start"_ and _"step"_ argument.

Experiment with this, if you need to. It's important that you internalize the way `range()` works. Go ahead and create a new code cell, and try things like:
```python
for i in range(5):
   print(i)
```
changing the arguments of `range()`. (Note how we end the `for` statement with a colon.) Now think for a bit: how many numbers does the list have in the case of our custom function `forward_diff()`?

Now, we will make a plot of the numerical derivative of $\exp(\sin(x))$. We can also compare with the analytical derivative:

\begin{equation}
\frac{\mathrm{d}y}{\mathrm{d}x} = \mathrm{e}^{\sin x}\,\cos x = y \cos x,
\end{equation}

```python
deriv_exact = y * numpy.cos(x) # analytical derivative

pyplot.figure()
pyplot.plot((x[1:] + x[:-1]) / 2.0, deriv,
            label='numerical',
            marker='.', color='gray',
            linestyle='None', markersize=10)
pyplot.plot(x, deriv_exact,
            label='analytical',
            color='k', linestyle='-') # analytical derivative in black line

pyplot.xlabel('$x$')
pyplot.ylabel('$\mathrm{d}y/\mathrm{d}x$')
pyplot.xlim(0.0, 2.0 * numpy.pi)
pyplot.legend(loc='upper center', numpoints=1);
```

    <IPython.core.display.Javascript object>

Stop for a bit and look at the first `pyplot.plot()` call above. The square brackets normally are how you access a particular element of an array via its index: `x[0]` is the first element of `x`, and `x[i+1]` is the `i`-th element. What's very cool is that you can also use _negative_ indices: they indicate counting backwards from the end of the array, so `x[-1]` is the last element of `x`.

A neat trick of arrays is called [_slicing_](http://docs.scipy.org/doc/numpy/reference/arrays.indexing.html): picking elements using the _colon notation_. Its general form is `x[start:stop:step]`. Note that, like the `range()` function, the `stop` index is _exclusive_, i.e., `x[stop]` is not included in the result.

For example, this code will give the odd numbers from `1` to `7`:
```python
x = numpy.array( [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] )
x[1:-1:2]
```
Try it! Remember, Python arrays are indexed from `0`, so `x[1]` is the second element. The end-point in the slice above is index `-1`, that's the last array element (not included in the result), and we're stepping by `2`, i.e., every other element. If the `step` is not given, it defaults to `1`. If `start` is not given, it defaults to the first array element, and if `stop` is not given, it defaults to the last element. Try several variations on the slice, until you're comfortable with it.

## There's a built-in for that

Here's another pro tip: whenever you find yourself writing a custom function for something that seems that a lot of people might use, find out first if there's a built-in for that. In this case, **NumPy** does indeed have a built-in for taking the numerical derivative by differencing! Check it out. We also use the function [`numpy.allclose()`](http://docs.scipy.org/doc/numpy-1.10.0/reference/generated/numpy.allclose.html) to check if the two results are close.

```python
numpy_deriv = numpy.diff(y) / numpy.diff(x)
print('Are the two results close? {}'.format(numpy.allclose(numpy_deriv, deriv)))
```

    Are the two results close? True

Not only is the code much more compact and easy to read with the built-in **NumPy** function for the numerical derivative ... it is also much faster:

```python
%timeit numpy_deriv = numpy.diff(y) / numpy.diff(x)
%timeit deriv = forward_diff(y, x)
```

    100000 loops, best of 3: 13.4 µs per loop
    10000 loops, best of 3: 75.2 µs per loop

**NumPy** functions will always be faster than equivalent code you write yourself because at the heart they use pre-compiled code and highly optimized numerical libraries, like BLAS and LAPACK.

## Do math like a pro

Do you want to compute the integral of $y(x) = \mathrm{e}^{\sin x}$? Of course you do. We find the analytical integral  using the integral formulas for modified Bessel functions:

\begin{equation}
\int_0^{2\pi}\mathrm{d} x\, \mathrm{e}^{\sin x} = 2\pi \,I_0(1),
\end{equation}

where $I_0$ is the modified Bessel function of the first kind. But if you don't have your special-functions handbook handy, we can find the integral with Python. We just need the right modules from the [**SciPy**](http://docs.scipy.org/doc/scipy/reference/) library. **SciPy** has a module of special functions, including Bessel functions, called `scipy.special`. Let's get that loaded, then use it to compute the exact integral:

```python
import scipy.special
exact_integral = 2.0 * numpy.pi * scipy.special.iv(0, 1.0)

print('Exact integral: {}'.format(exact_integral))
```

    Exact integral: 7.95492652101

Or instead, we may want to compute the integral numerically, via the trapezoid rule. The integral is over one period of a periodic function, so only the constant term of its Fourier series will contribute (the periodic terms integrate to zero). The constant Fourier term is the mean of the function over the interval, and the integral is the area of a rectangle: $2\pi \langle y(x)\rangle_x$. Sampling $y$ at $n$ evenly spaced points over the interval of length $2\pi$, we have:

\begin{align}
\int_0^{2\pi}\mathrm{d} x\, y(x) \approx \frac{2\pi}{n}\sum_{i=0}^{n} y(x_i),
\end{align}

**NumPy** gives as a `mean` method to quickly get the sum:

```python
approx_integral = 2.0 * numpy.pi * y[:-1].mean()

print('Approximate integral: {}'.format(approx_integral))
print('Error: {}'.format(exact_integral - approx_integral))
```

    Approximate integral: 7.95492652101
    Error: 0.0

```python
approx_integral = 2.0 * numpy.pi * numpy.mean(y[:-1])

print('Approximate integral: {}'.format(approx_integral))
print('Error: {}'.format(exact_integral - approx_integral))
```

    Approximate integral: 7.95492652101
    Error: 0.0

The syntax `y.mean()` applies the `mean()` **NumPy** method to the array `y`. Here, we apply the method to a _slice_ of `y` that does not include the last element (see discussion of _slicing_ above). We could have also done `numpy.mean(y[:-1])` (the function equivalent of the method `mean()` applied to an array); they give equivalent results and which one you choose is a matter of style.

## Beautiful interactive plots with Bokeh

**Matplotlib** will be your workhorse for creating plots in notebooks. But it's not the only game in town! A recent new player is [**Bokeh**](http://nbviewer.jupyter.org/github/bokeh/bokeh-notebooks/blob/master/index.ipynb), a visualization library to make amazing interactive plots and share them online. It can also handle very large data sets with excellent performance.

If you installed **Anaconda** in your system, you will probably already have **Bokeh**. You can check if it's there by running the `conda list` command. If you installed **Miniconda**, you will need to install it with `conda install bokeh`.

After installing **Bokeh**, we have many modules available: [`bokeh.plotting`](http://bokeh.pydata.org/en/latest/docs/reference/plotting.html#bokeh-plotting) gives you the ability to create interactive figures with zoom, pan, resize, save, and other tools.

```python
from bokeh import plotting as bplotting
```

**Bokeh** integrates with Jupyter notebooks by calling the output function, as follows:

```python
bplotting.output_notebook()
```

    BokehJS successfully loaded.

```python
# create a new Bokeh plot with axis labels, name it "bop"
bop = bplotting.figure(x_axis_label='x', y_axis_label='dy/dx')

# add a title, change the font
bop.title = "Derivative of exp(sin(x))"
bop.title_text_font = "palatino"

# add a line with legend and line thickness to "bop"
bop.line(x, deriv_exact, legend="analytical", line_width=2)

# add circle markers with legend, specify color
bop.circle((x[1:] + x[:-1]) / 2.0, deriv, legend="numerical", fill_color="gray", size=8, line_color=None)

bop.grid.grid_line_alpha=0.3

bplotting.show(bop);
```

**Note**—As of June 2016 (v.0.11.1), Bokeh does not support LaTeX on axis labels. This is an [issue](https://github.com/bokeh/bokeh/issues/647) they are working on, so stay tuned!

Look at the neat tools on the **Bokeh** figure: you can zoom in to any portion to explore the data, you can drag the plot area around, resize and finally save the figure to a file. You also have many beautiful [styling](http://bokeh.pydata.org/en/latest/docs/user_guide/styling.html#userguide-styling) options!

# Optional next step: get interactive with Lorenz

We found two really cool ways for you to get interactive with the Lorenz equations! Try out the interactive blog post by Tim Head on [Exploring the Lorenz equations](https://betatim.github.io/posts/interactive-posts/) (January 2016), and learn about IPython widgets. Or, check out the [Lorentz example on **Bokeh** plots](http://nbviewer.jupyter.org/github/bokeh/bokeh-notebooks/blob/master/gallery/lorenz.ipynb). Better yet, try them both.

---

(c) 2016 Lorena A. Barba. Free to use under Creative Commons Attribution CC-BY 4.0 License. This notebook was written for the tutorial "The world of Jupyter" at the Huazhong University of Science and Technology (HUST), Wuhan, China.

Example from Justin Bois (c) 2015 also under a CC-BY 4.0 License.

week1/jupyter-tutorial/3--Jupyter like a pro.ipynb


```julia
using MomentClosure, Latexify, OrdinaryDiffEq, Catalyst
```

    ┌ Info: Precompiling MomentClosure [01a1b25a-ecf0-48c5-ae58-55bfd5393600]
    └ @ Base loading.jl:1278

$$ G \stackrel{c_1}{\rightarrow} G+P, \\
   G^* \stackrel{c_2}{\rightarrow} G^*+P, \\
   P \stackrel{c_3}{\rightarrow} 0 \\
   G+P \underset{c_5}{\stackrel{c_4}{\rightleftharpoons}}  G^* $$

On/off gene states are merged into a Bernoulli variable $g(t)$ which can be either $1$ ($G$) or $0$ ($G^*$). The number of proteins in the system is given by $p(t)$.

### Using `ReactionSystemMod`

```julia
@parameters t, c₁, c₂, c₃, c₄, c₅
@variables p(t), g(t)

vars = [g, p]

ps = [c₁, c₂, c₃, c₄, c₅]

S = [0 0  0 -1 1;
     1 1 -1 -1 1]

as = [c₁*g,     # G -> G + P
      c₂*(1-g), # G* -> G* + P
      c₃*p,     # P -> 0
      c₄*g*p,   # G+P -> G*
      c₅*(1-g)] # G* -> G+P
# system size Ω can be included as an additional parameter if needed

binary_vars = [1]

rn = ReactionSystemMod(t, vars, ps, as, S)
```

    ReactionSystemMod(t, Term{Real,Nothing}[g(t), p(t)], Sym{ModelingToolkit.Parameter{Real},Nothing}[c₁, c₂, c₃, c₄, c₅], SymbolicUtils.Mul{Real,Int64,Dict{Any,Number},Nothing}[c₁*g(t), c₂*(1 - g(t)), c₃*p(t), c₄*g(t)*p(t), c₅*(1 - g(t))], [0 0 … -1 1; 1 1 … -1 1])

### Using Catalyst.jl `ReactionSystem`

* $\rightarrow$ indicates a reaction that follows the law of mass action (need to indicate only the reaction coefficient, full propensity function is constructed automatically)
* $\Rightarrow$ indicates a reaction that does not follow the law of mass action (need to define the full propensity function)

```julia
rn = @reaction_network begin
    (c₁), g → g+p
    (c₂*(1-g)), 0 ⇒ p
    (c₃), p → 0
    (c₄), g+p → 0
    (c₅*(1-g)), 0 ⇒ g+p
end c₁ c₂ c₃ c₄ c₅
```

\begin{align}
\require{mhchem}
\ce{ g &->[c{_1}] g + p}\\
\ce{ \varnothing &<=>[c{_2} \left( 1 - g\left( t \right) \right)][c{_3}] p}\\
\ce{ g + p &<=>[c{_4}][c{_5} \left( 1 - g\left( t \right) \right)] \varnothing}
\end{align}

identical stoichiometry matrix and propensity functions are recovered

```julia
propensities(rn, combinatoric_ratelaw=false)
```

    5-element Array{SymbolicUtils.Mul{Real,Int64,Dict{Any,Number},Nothing},1}:
     c₁*g(t)
     c₂*(1 - g(t))
     c₃*p(t)
     c₄*g(t)*p(t)
     c₅*(1 - g(t))

```julia
get_S_mat(rn)
```

    2×5 Array{Int64,2}:
     0  0   0  -1  1
     1  1  -1  -1  1

### Moment equations

Generate raw moment equations up to 3rd order.

The argument `combinatoric_ratelaw = false` indicates whether binomial coefficients are included when constructing the propensity functions for the reactions that follow the law of mass action (does not play a role in this specific scenarion)

Equivalently, central moment equations can be generated using `generate_central_moment_eqs(rn, 3, 5, combinatoric_ratelaw=false)`

```julia
raw_eqs = generate_raw_moment_eqs(rn, 3, combinatoric_ratelaw=false)
latexify(raw_eqs)
```

\begin{align*}
\frac{d\mu{_{10}}}{dt} =& c{_5} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} \\
\frac{d\mu{_{01}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} - c{_2} \mu{_{10}} - c{_3} \mu{_{01}} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} \\
\frac{d\mu{_{20}}}{dt} =& c{_5} + c{_4} \mu{_{11}} + c{_5} \mu{_{10}} - 2 c{_4} \mu{_{21}} - 2 c{_5} \mu{_{20}} \\
\frac{d\mu{_{11}}}{dt} =& c{_5} + c{_1} \mu{_{20}} + c{_2} \mu{_{10}} + c{_4} \mu{_{11}} + c{_5} \mu{_{01}} - c{_2} \mu{_{20}} - c{_3} \mu{_{11}} - c{_4} \mu{_{12}} - c{_4} \mu{_{21}} - c{_5} \mu{_{11}} - c{_5} \mu{_{20}} \\
\frac{d\mu{_{02}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} + c{_3} \mu{_{01}} + c{_4} \mu{_{11}} + 2 c{_1} \mu{_{11}} + 2 c{_2} \mu{_{01}} + 2 c{_5} \mu{_{01}} - c{_2} \mu{_{10}} - 2 c{_2} \mu{_{11}} - 2 c{_3} \mu{_{02}} - 2 c{_4} \mu{_{12}} - c{_5} \mu{_{10}} - 2 c{_5} \mu{_{11}} \\
\frac{d\mu{_{30}}}{dt} =& c{_5} + 3 c{_4} \mu{_{21}} + 2 c{_5} \mu{_{10}} - c{_4} \mu{_{11}} - 3 c{_4} \mu{_{31}} - 3 c{_5} \mu{_{30}} \\
\frac{d\mu{_{21}}}{dt} =& c{_5} + c{_1} \mu{_{30}} + c{_2} \mu{_{20}} + c{_4} \mu{_{12}} + c{_5} \mu{_{01}} + c{_5} \mu{_{10}} + c{_5} \mu{_{11}} + 2 c{_4} \mu{_{21}} - c{_2} \mu{_{30}} - c{_3} \mu{_{21}} - c{_4} \mu{_{11}} - 2 c{_4} \mu{_{22}} - c{_4} \mu{_{31}} - c{_5} \mu{_{20}} - 2 c{_5} \mu{_{21}} - c{_5} \mu{_{30}} \\
\frac{d\mu{_{12}}}{dt} =& c{_5} + c{_1} \mu{_{20}} + c{_2} \mu{_{10}} + c{_3} \mu{_{11}} + c{_4} \mu{_{21}} + c{_5} \mu{_{02}} + 2 c{_1} \mu{_{21}} + 2 c{_2} \mu{_{11}} + 2 c{_4} \mu{_{12}} + 2 c{_5} \mu{_{01}} - c{_2} \mu{_{20}} - 2 c{_2} \mu{_{21}} - 2 c{_3} \mu{_{12}} - c{_4} \mu{_{11}} - c{_4} \mu{_{13}} - 2 c{_4} \mu{_{22}} - c{_5} \mu{_{12}} - c{_5} \mu{_{20}} - 2 c{_5} \mu{_{21}} \\
\frac{d\mu{_{03}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} + 3 c{_1} \mu{_{11}} + 3 c{_1} \mu{_{12}} + 3 c{_2} \mu{_{01}} + 3 c{_2} \mu{_{02}} + 3 c{_3} \mu{_{02}} + 3 c{_4} \mu{_{12}} + 3 c{_5} \mu{_{01}} + 3 c{_5} \mu{_{02}} - c{_2} \mu{_{10}} - 3 c{_2} \mu{_{11}} - 3 c{_2} \mu{_{12}} - c{_3} \mu{_{01}} - 3 c{_3} \mu{_{03}} - c{_4} \mu{_{11}} - 3 c{_4} \mu{_{13}} - c{_5} \mu{_{10}} - 3 c{_5} \mu{_{11}} - 3 c{_5} \mu{_{12}}
\end{align*}

We are solving for moments up to `m_order = 3`, and in the equations encounter moments up to `exp_order = 5`.

Use the Bernoulli variable properties to eliminate redundant equations to see how they simplify:

```julia
bernoulli_eqs = bernoulli_moment_eqs(raw_eqs, binary_vars)
latexify(bernoulli_eqs)
```

\begin{align*}
\frac{d\mu{_{10}}}{dt} =& c{_5} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} \\
\frac{d\mu{_{01}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} - c{_2} \mu{_{10}} - c{_3} \mu{_{01}} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} \\
\frac{d\mu{_{11}}}{dt} =& c{_5} + c{_1} \mu{_{10}} + c{_5} \mu{_{01}} - c{_3} \mu{_{11}} - c{_4} \mu{_{12}} - c{_5} \mu{_{10}} - c{_5} \mu{_{11}} \\
\frac{d\mu{_{02}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} + c{_3} \mu{_{01}} + c{_4} \mu{_{11}} + 2 c{_1} \mu{_{11}} + 2 c{_2} \mu{_{01}} + 2 c{_5} \mu{_{01}} - c{_2} \mu{_{10}} - 2 c{_2} \mu{_{11}} - 2 c{_3} \mu{_{02}} - 2 c{_4} \mu{_{12}} - c{_5} \mu{_{10}} - 2 c{_5} \mu{_{11}} \\
\frac{d\mu{_{12}}}{dt} =& c{_5} + c{_1} \mu{_{10}} + c{_3} \mu{_{11}} + c{_5} \mu{_{02}} + 2 c{_1} \mu{_{11}} + 2 c{_5} \mu{_{01}} - 2 c{_3} \mu{_{12}} - c{_4} \mu{_{13}} - c{_5} \mu{_{10}} - 2 c{_5} \mu{_{11}} - c{_5} \mu{_{12}} \\
\frac{d\mu{_{03}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} + 3 c{_1} \mu{_{11}} + 3 c{_1} \mu{_{12}} + 3 c{_2} \mu{_{01}} + 3 c{_2} \mu{_{02}} + 3 c{_3} \mu{_{02}} + 3 c{_4} \mu{_{12}} + 3 c{_5} \mu{_{01}} + 3 c{_5} \mu{_{02}} - c{_2} \mu{_{10}} - 3 c{_2} \mu{_{11}} - 3 c{_2} \mu{_{12}} - c{_3} \mu{_{01}} - 3 c{_3} \mu{_{03}} - c{_4} \mu{_{11}} - 3 c{_4} \mu{_{13}} - c{_5} \mu{_{10}} - 3 c{_5} \mu{_{11}} - 3 c{_5} \mu{_{12}}
\end{align*}

### Closing the moment equations

Finally, we can apply the selected moment closure method on the system of raw moment equations:

```julia
closed_raw_eqs = moment_closure(raw_eqs, "conditional derivative matching", binary_vars)
latexify(closed_raw_eqs)
```

\begin{align*}
\frac{d\mu{_{10}}}{dt} =& c{_5} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} \\
\frac{d\mu{_{01}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} - c{_2} \mu{_{10}} - c{_3} \mu{_{01}} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} \\
\frac{d\mu{_{11}}}{dt} =& c{_5} + c{_1} \mu{_{10}} + c{_5} \mu{_{01}} - c{_3} \mu{_{11}} - c{_4} \mu{_{12}} - c{_5} \mu{_{10}} - c{_5} \mu{_{11}} \\
\frac{d\mu{_{02}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} + c{_3} \mu{_{01}} + c{_4} \mu{_{11}} + 2 c{_1} \mu{_{11}} + 2 c{_2} \mu{_{01}} + 2 c{_5} \mu{_{01}} - c{_2} \mu{_{10}} - 2 c{_2} \mu{_{11}} - 2 c{_3} \mu{_{02}} - 2 c{_4} \mu{_{12}} - c{_5} \mu{_{10}} - 2 c{_5} \mu{_{11}} \\
\frac{d\mu{_{12}}}{dt} =& c{_5} + c{_1} \mu{_{10}} + c{_3} \mu{_{11}} + c{_5} \mu{_{02}} + 2 c{_1} \mu{_{11}} + 2 c{_5} \mu{_{01}} - 2 c{_3} \mu{_{12}} - c{_5} \mu{_{10}} - 2 c{_5} \mu{_{11}} - c{_5} \mu{_{12}} - c{_4} \mu{_{10}} \mu{_{11}}^{-3} \mu{_{12}}^{3} \\
\frac{d\mu{_{03}}}{dt} =& c{_2} + c{_5} + c{_1} \mu{_{10}} + 3 c{_1} \mu{_{11}} + 3 c{_1} \mu{_{12}} + 3 c{_2} \mu{_{01}} + 3 c{_2} \mu{_{02}} + 3 c{_3} \mu{_{02}} + 3 c{_4} \mu{_{12}} + 3 c{_5} \mu{_{01}} + 3 c{_5} \mu{_{02}} - c{_2} \mu{_{10}} - 3 c{_2} \mu{_{11}} - 3 c{_2} \mu{_{12}} - c{_3} \mu{_{01}} - 3 c{_3} \mu{_{03}} - c{_4} \mu{_{11}} - c{_5} \mu{_{10}} - 3 c{_5} \mu{_{11}} - 3 c{_5} \mu{_{12}} - 3 c{_4} \mu{_{10}} \mu{_{11}}^{-3} \mu{_{12}}^{3}
\end{align*}

We can also print out the closure functions for each higher order moment:

```julia
latexify(closed_raw_eqs, :closure)
```

\begin{align*}
\mu{_{13}} =& \mu{_{10}} \mu{_{11}}^{-3} \mu{_{12}}^{3} \\
\mu{_{04}} =& \mu{_{01}}^{4} \mu{_{02}}^{-6} \mu{_{03}}^{4}
\end{align*}

### Numerical solution

The closed moment equations can be solved using DifferentialEquations.jl (or just OrdinaryDiffEq.jl which is more lightweight and sufficient for this particular case.

```julia
# PARAMETER INITIALISATION
pmap = [c₁ => 0.01,
        c₂ => 40,
        c₃ => 1,
        c₄ => 1,
        c₅ => 1]

# DETERMINISTIC INITIAL CONDITIONS
μ₀ = [1., 0.001]
u₀map = deterministic_IC(μ₀, closed_raw_eqs)

# time interval to solve on
tspan = (0., 1000.0)
dt = 1

@time oprob = ODEProblem(closed_raw_eqs, u₀map, tspan, pmap);
@time sol_CDM = solve(oprob, Tsit5(), saveat=dt);
```

     12.183509 seconds (17.83 M allocations: 933.692 MiB, 2.85% gc time)
      5.671233 seconds (12.59 M allocations: 608.756 MiB, 11.10% gc time)

```julia
using Plots

plot(sol_CDM.t, sol_CDM[1,:],
    label  = "CDM",
    legend = true,
    xlabel = "Time [s]",
    ylabel = "Mean gene number",
    lw=2,
    legendfontsize=8,
    xtickfontsize=10,
    ytickfontsize=10,
    dpi=100)
```

```julia
plot(sol_CDM.t, sol_CDM[2,:],
    label  = "CDM",
    legend = :bottomright,
    xlabel = "Time [s]",
    ylabel = "Mean protein number",
    lw=2,
    legendfontsize=8,
    xtickfontsize=10,
    ytickfontsize=10,
    dpi=100)
```

```julia
std_CDM = sqrt.(sol_CDM[4,2:end] .- sol_CDM[2,2:end].^2)
plot(sol_CDM.t[2:end], std_CDM,
    label  = "CDM",
    legend = true,
    xlabel = "Time [s]",
    ylabel = "standard deviation of the protein number",
    lw=2,
    legendfontsize=8,
    xtickfontsize=10,
    ytickfontsize=10,
    dpi=100)
```

### SSA

```julia
using DiffEqJump

# parameters [c₁, c₂, c₃, c₄, c₅]
p = [param[2] for param in pmap]

# initial conditions [g, p]
# NOTE: define as FLOATS as otherwise will encounter problems due to parsing from Int to Float
u₀ = [1., 0.]

# time interval to solve on
tspan = (0., 1000.)

# create a discrete problem to encode that our species are integer valued
dprob = DiscreteProblem(rn, u₀, tspan, p)

# create a JumpProblem and specify Gillespie's Direct Method as the solver:
jprob = JumpProblem(rn, dprob, Direct(), save_positions=(false, false))
# SET save_positions to (false, false) as otherwise time of each reaction occurence is saved

dt = 1 # time resolution at which numerical solution is saved

# solve and plot
ensembleprob  = EnsembleProblem(jprob)
@time sol_SSA = solve(ensembleprob, SSAStepper(), saveat=dt, trajectories=10000);
```

    205.463844 seconds (646.80 M allocations: 15.332 GiB, 56.95% gc time)

```julia
using DiffEqJump

# This is not equal to μ₀ as the initial values should be Int (not 0.001)
# However, these still must be defined as Floats, otherwise will encounter
# problems due to parsing from Int to Float using EnsembleAnalysis
u₀ = [1., 0.]
# create a discrete problem to encode that our species are integer valued
dprob = DiscreteProblem(rn, u₀, tspan, pmap)

# create a JumpProblem and specify Gillespie's Direct Method as the solver:
jprob = JumpProblem(rn, dprob, Direct(), save_positions=(false, false))
# SET save_positions to (false, false) as otherwise time of each reaction occurence is saved

dt = 1 # time resolution at which numerical solution is saved

# solve and plot
ensembleprob  = EnsembleProblem(jprob)
@time sol_SSA = solve(ensembleprob, SSAStepper(), saveat=dt, trajectories=10000);
```

    245.071448 seconds (645.00 M allocations: 15.269 GiB, 65.19% gc time)

Can compute all sample moments up to chosen order

```julia
@time SSA_μ = sample_raw_moments(sol_SSA, 2);
@time SSA_M = sample_central_moments(sol_SSA, 2);
```

      6.137514 seconds (2.82 M allocations: 1.325 GiB, 21.46% gc time)
      2.501856 seconds (2.79 M allocations: 744.488 MiB)

```julia
plot(sol_CDM.t, [sol_CDM[1,:], SSA_μ[1,0]],
    label  = ["CDM" "SSA"],
    legend = true,
    xlabel = "Time [s]",
    ylabel = "Mean gene number",
    lw=2,
    legendfontsize=8,
    xtickfontsize=10,
    ytickfontsize=10,
    dpi=100)
```

```julia
plot(sol_CDM.t, [sol_CDM[2,:], SSA_μ[0,1]],
    label  = ["CDM" "SSA"],
    legend = :bottomright,
    xlabel = "Time [s]",
    ylabel = "Mean protein number",
    lw=2,
    legendfontsize=8,
    xtickfontsize=10,
    ytickfontsize=10,
    dpi=100)
```

```julia
std_CDM = sqrt.(sol_CDM[4,2:end] .- sol_CDM[2,2:end].^2)
std_p_SSA = sqrt.(SSA_M[0,2][2:end])
plot(sol_CDM.t[2:end], [std_CDM, std_p_SSA],
    label  = ["CDM" "SSA"],
    legend = true,
    xlabel = "Time [s]",
    ylabel = "standard deviation of the protein number",
    lw=2,
    legendfontsize=8,
    xtickfontsize=10,
    ytickfontsize=10,
    dpi=100)
```