# Quasi Monte-Carlo integration

```R
# Load libraries
library(ggplot2, quietly=TRUE);
library(randtoolbox, quietly=TRUE);

# Decrease plot sizes
options(repr.plot.width=4, repr.plot.height=4);
```

## Table of contents

 * References
 * Monte Carlo integration (MC)
 * Sampling (QMC)
 * Low-discrepancy sequences
 * Sobol-sequences
 * High-dimensional Sobol-sequences and Brownian bridges

## References

 * Numerical Recipes. The Art of Scientific Computing, 3rd Edition, 2007
   Section 7.7, page 397-410
 * Christophe Dutang and Diethelm Wuertz - "A note on random number generation"
 https://cran.r-project.org/web/packages/randtoolbox/vignettes/fullpres.pdf https://github.com/cran/randtoolbox/blob/master/R/qmc.R
 * Xiaoqun Wang and Ian H. Sloan - Low Discrepancy Sequences in High Dimensions: How Well Are Their Projections Distributed?
   https://www.maths.unsw.edu.au/sites/default/files/amr06_8_0.pdf
 * Junyi Lin, Xiaoqun Wang - New Brownian bridge construction in quasi-Monte Carlo methods for computational finance http://dx.doi.org/10.1016/j.jco.2007.06.001
 * Jacques Du Toit - A High-performance Brownian Bridge for GPUs: Lessons for bandwidth bound applications http://www.nag.co.uk/doc/techrep/pdf/tr2_12.pdf

## Monte Carlo integration

Monte Carlo integration provides a general method of computing multivariate integrals through random sampling. Many problems can be transformed into the computation of the integral of some function $f$ on the d-dimensional unit hypercube $[0,1]^d$:

\begin{equation}
\int_{[0,1]^d} \! f(\mathbf{x}) \, \mathrm{d}\mathbf{x}. \quad \mathbf{x} = (x_1, \ldots, x_d)
\end{equation}

We can estimate this integral by picking $N$ uniformly distributed points in $[0,1]^d$ by averaging:

\begin{equation}
\frac{1}{n}\sum\limits_{i=1}^n\; f(\mathbf{x}_n),
\end{equation}

And as we increase N, we increase the precision of our approximation:

\begin{equation}
\lim_{n\to\infty}\; \frac{1}{n}\sum\limits_{i=1}^n\; f(\mathbf{x}_n) = \int_{[0,1]^s} \! f(\mathbf{x}) \, \mathrm{d}\mathbf{x}.
\end{equation}

If we select the points uniformly from the hypercube, the approximation will converge faster. Thus to improve efficiency, the main problem is how to select the sampling points.

## Sampling

We can use either of these approaches to generate points in the unit hypercube:

 * Uniform grid
 * Pseudo-random number generation (PRNG)
 * Quasi-random number generation (QRNG)
 * or a combination (e.g. using many PRNGs for different subintervals, this is called stratified sampling)

### Uniform grid
The most basic approach will be to generate a uniform grid:

```R
w <- 20
ggplot(data = data.frame(x=((0:(w*w-1))%/%w)/w,y=((0:(w*w-1))%%w)/w), aes(x,y))+ geom_point()
```

The problem is that we with this approach have to select the resolution of the grid in advance. We usually don't know how long to iterate in advance, and often we want to the algorithm to continue until we reach some precision.

### Pseudo-random number generation
With pseudo-random number generation we will be able to do this, but the points can clumb together or leave larger unfilled regions. Evaluating $f(\mathbf{x})$ at a point close to another one we have already used, will not give much new information and will thus not improve on our current approximation.

```R
n <- 400
set.generator("MersenneTwister", initialization="init2002", resolution=53, seed=12345)
mersenne_frame <- data.frame(x=runif(n), y=runif(n))
ggplot(data = mersenne_frame, aes(x,y))+ geom_point()
```

### Quasi-random sequences
In contrast, Quasi-random number generators does not aim to be truly random, they instead generate points in a more structured fashion, filling more and more of unit hypercube. After calculating 400 2D-points of the Torus algorithm (Kronecker sequences), we obtain the following pattern:

```R
n <- 400
torus_sequence <- torus(n, dim = 2)
ggplot(data = data.frame(x=torus_sequence[,1],y=torus_sequence[,2]), aes(x,y))+ geom_point()
```

But unlike uniform grid, we can continue to fill in more and more points in-between those already generated:

```R
n <- 2000
torus_sequence <- torus(n, dim = 2)
ggplot(data = data.frame(x=torus_sequence[,1],y=torus_sequence[,2]), aes(x,y))+ geom_point()
```

## Low-discrepancy sequences
Sequences, such as the one generated by the Torus algorithm above, are called *low-discrepancy sequences* and the algorithms for generating them are called *quasi-random number generators* (QRNGs). The basic idea for these algorithms is to generate proportionally equal amounts of samples in each subinterval, thus limiting the number of empty regions and regions with high sample density.

On Wikipedia we find the following definition of discrepancy:
<blockquote>
The *discrepancy* of a set $P = \{ x_1, \ldots, x_N\}$ is defined, using Niederreiter's notation, as

\begin{equation}
D_N(P) = \sup_{B\in J}
  \left|  \frac{A(B;P)}{N} - \lambda_s(B)  \right|
\end{equation}

where
$λ_s$ is the $s$-dimensional Lebesgue measure,
$A(B;P)$ is the number of points in $P$ that fall into $B$,
and $J$ is the set of $s$-dimensional intervals or boxes of the form

\begin{equation}
\prod_{i=1}^s [a_i, b_i) = \{ \mathbf{x} \in \mathbf{R}^s : a_i \le x_i < b_i \} \,
\end{equation}

where $0 \le a_i < b_i \le 1$.
</blockquote>

This can be understood as finding the region in the point set, with the largest difference between observed size $\left(\frac{A(B;P)}{N}\right)$ and actual size $\left(\lambda_s(B)\right)$.

## Sobol sequences
The Sobol-sequence is a method of generating quasi-random integers in a $D$-dimensional unit hypercube.

A very approachable description of Sobol generation algorithms is given by Bradley et al.
in their paper "Parallelisation Techniques for Random Number Generators"
https://people.maths.ox.ac.uk/gilesm/files/gems_rng.pdf
But lookout, they use both 0- and 1-indexing, which also have confused themselves, leading to an indexing error in equation (4), it should say $m_{q_n + 1}$.

We will now build a simple library for generating Sobol-numbers, to illustrate the different algorithms.

A $D$-dimensional Sobol-sequence consists of $D$ one-dimensional Sobol-sequences. Thus we can generate each dimension in parallel, and we only need to describe how to generate one-dimensional sequences.

However, to test our generators, we will generate two-dimensional sequences and use them to calculate an estimate for $\pi$.

### Direction vectors
To initialise each dimension, we need a so-called direction vector. We will not describe how these are generated (TODO: maybe we should actually do that at some point). We will use 30 direction numbers for each sequence:

\begin{equation}
m_0, m_1, \ldots m_{29}
\end{equation}

Each such sequence contain information to generate a Sobol-sequence of 30-bit quasi-random integers.

```R
sobol_bitwidth <- 30
```

The integers are later normalised into the hypercube by division with:

```R
sobol_divisor <- 2^30
```

We are going to use the following two direction vectors in our $\pi$ example later:

```R
sobol_directionVector <-
  rbind(c(536870912,268435456,134217728,67108864,33554432,
          16777216,8388608,4194304,2097152,1048576,524288,
          262144,131072,65536,32768,16384,8192,4096,2048,
          1024,512,256,128,64,32,16,8,4,2,1),
        c(536870912,805306368,671088640,1006632960,570425344,
          855638016,713031680,1069547520,538968064,808452096,
          673710080,1010565120,572653568,858980352,715816960,
          1073725440,536879104,805318656,671098880,1006648320,
          570434048,855651072,713042560,1069563840,538976288,
          808464432,673720360,1010580540,572662306,858993459))
```

### Independent formula

The simplest way of generating a Sobol-number is through the formula:

\begin{equation}
y_i = b_0m_0 \oplus b_2m_2 \oplus \ldots \oplus b_{29}m_{29}
\end{equation}

Where $y_i$ is the Sobol-number at index $i$, $b_0, b_1, \ldots b_{29}$ is the 30-bit binary representation of the index $i$ and $\oplus$ is the bitwise exclusive or operation.

In the recursive formulation, that we will discuss later, it is required that we instead of the binary representaion $i$, use the binary representation of *the gray code of $i$*. Gray codes are computed like this:

```R
grayCode <- function(i) (bitwXor(i, bitwShiftR(i, 1)))
```

Let $g_0, g_1, \ldots g_{29}$ be the binary representation fo the gray code, now the independent formula is:

\begin{equation}
y_n = g_0m_0 \oplus g_2m_2 \oplus \ldots \oplus g_{29}m_{29}
\end{equation}

Let us implement this in R. First we need a function for converting an integer $x$ into it's binary representation, $b_0, b_1, \ldots \b_{w}$ where $w$ is the bit width.

```R
# Returns TRUE if the n'th bit of x is set
testBit <- function(x,n) (bitwAnd(x, bitwShiftL(1, n)) != 0)

# This is similar to the intToBits(x) function in R, but is not
# restricted to 32-bit integers
bits <- function(x, bitwidth) {
     mapply(function (b) (testBit (x, b)), 0:(bitwidth-1))
}
```

The independent formula for generating Sobol-sequences can then be written as:

```R
sobol_independent <- function (i, m) {
  g <- grayCode(i)
  gbits <- bits(g, sobol_bitwidth)
  Reduce(bitwXor, m * gbits, 0, right = FALSE, accumulate = FALSE)
}
```

Let us test it:

```R
sobol_independent(5,sobol_directionVector[1,])
```

To generate the complete one-dimensional sequence, we just have to map this formula over every index into the sequence:

```R
sobol_independent_1D <- function(n, m) (mapply (function (i) (sobol_independent (i, m)), 0:(n-1)))
sobol_independent_1D(10, sobol_directionVector[1,])
```

Similarly, we can map it over the individual Sobol-vectors:

```R
sobol_independent_ND <- function (n, ms) {
    apply(ms, 1, function(m) (sobol_independent_1D(n,m)))
}
sobol_independent_ND(10, sobol_directionVector)
```

### Recursive formula
A much more efficient formulation was found by ... TODO

### Leap-frogging
To accomodate GPU computation, and even more efficient formulation was discovered by Bradley et al., which allows for coalesced writes.

TODO

## High-dimensional Sobol-sequences and Brownian bridge
Sobol sequences with many dimensions expose a different problem, that needs to be tackled. For the first couple of dimensions, Sobol numbers have good space coverage:

```R
n <- 16000
dimensions = 30
sobol_sequence <- sobol(n, dim = dimensions)
sobol_frame <- data.frame(x=sobol_sequence[,1],y=sobol_sequence[,2])
ggplot(data = sobol_frame[0:4000,], aes(x,y))+ geom_point() + xlab("dimension 1") + ylab("dimension 2")
```

But on the higher dimensions of the Sobol-sequence, the quality decreases significantly. We can illustrate by showing the 29th and 30th dimension of the same sequence as was plotted above:

```R
sobol_frame <- data.frame(x=sobol_sequence[,29],y=sobol_sequence[,30])
ggplot(data = sobol_frame[0:4000,], aes(x,y)) + geom_point() + xlab("dimension 29") + ylab("dimension 30")
```

Later in the sequence, the gaps disappear. Here we plot the last dimensions with 8000 and 16000 points.

```R
ggplot(data = sobol_frame[0:8000,], aes(x,y)) + geom_point() + xlab("dimension 29") + ylab("dimension 30")
ggplot(data = sobol_frame[0:16000,], aes(x,y)) + geom_point() + xlab("dimension 29") + ylab("dimension 30")
```

## Brownian bridge for path generation w. Sobol-numbers
When Sobol-sequences are used for path-generation, a strategy has been devised to tackle the above problem of little variation in the higher dimensions.

Instead of using each dimension in sequence to generate a path, the lower-dimension (higher quality) dimensions are used to generate the big steps of the sequence, while the latter dimensions are only used to "fill in the holes" (only taking small steps).

This can be done by using the Brownian bridge formula:

...

```R

```


## Integration over Polytopes

#### Extra dependencies : matplotlib (if using methods : plot_polytope and plot_polynomial)

```python
from sympy import sqrt
from sympy.abc import x, y, z
from sympy.geometry import *
from sympy.integrals.intpoly import *

```

## Methods :

### polytope_integrate(poly, expr, **kwargs) :
    Pre-processes the input data for integrating univariate/bivariate polynomials over 2-Polytopes.

    poly(Polygon) : 2-Polytope
    expr(SymPy expression) : uni/bi-variate polynomial

    Optional Parameters
    clockwise(Boolean) : If user is not sure about orientation of vertices and wants to clockwise sort the points.
    max_degree(Integer) : Maximum degree of any monomial of the input polynomial.

   #### Examples :

```python
triangle = Polygon(Point(0,0), Point(1,1), Point(1,0))
plot_polytope(triangle)
print("Area of Triangle with vertices : (0,0), (1,1), (1,0) : ", polytope_integrate(triangle, 1))
print("x*y integrated over Triangle with vertices : (0,0), (1,1), (1,0) : ", polytope_integrate(triangle, x*y),"\n")

hexagon = Polygon(Point(0, 0), Point(-sqrt(3) / 2, 0.5),
                  Point(-sqrt(3) / 2, 3 / 2), Point(0, 2),
                  Point(sqrt(3) / 2, 3 / 2), Point(sqrt(3) / 2, 0.5))
plot_polytope(hexagon)
print("Area of regular hexagon with unit side length  : ", polytope_integrate(hexagon, 1))
print("x + y**2 integrated over regular hexagon with unit side length  : ", polytope_integrate(hexagon, x + y**2))

polys = [1, x, y, x*y]
print("1, x, y, x*y integrated over hexagon : ", polytope_integrate(hexagon, polys, max_degree=2))
```

### integration_reduction(facets, index, a, b, expr, dims, degree) :
    facets : List of facets that decide the region enclose by 2-Polytope.
    index : The index of the facet with respect to which the integral is supposed to be found.
    a, b : Hyperplane parameters corresponding to facets.
    expr : Uni/Bi-variate Polynomial
    dims : List of symbols denoting axes
    degree : Degree of the homogeneous polynoimal(expr)

    This is a helper function for polytope_integrate. It relates the result of the integral of a polynomial over a
    d-dimensional entity to the result of the same integral of that polynomial over the (d - 1)-dimensional
    facet[index].

    For the 2D case, surface integral --> line integrals --> evaluation of polynomial at vertices of line segments
    For the 3D case, volume integral --> 2D use case

    The only minor limitation is that some lines of code are 2D specific, but that can be easily changed. Note that
    this function is a helper one and works for a facet which bounds the polytope(i.e. the intersection point with the
    other facets is required), not for an independent line.

   #### Examples:

```python
facets = [Segment2D(Point(0, 0), Point(1, 1)), Segment2D(Point(1, 1), Point(1, 0)), Segment2D(Point(0, 0), Point(1, 0))]
print(integration_reduction(facets, 0, (0, 1), 0, 1, [x, y], 0))
print(integration_reduction(facets, 1, (0, 1), 0, 1, [x, y], 0))
print(integration_reduction(facets, 2, (0, 1), 0, 1, [x, y], 0))
```

### hyperplane_parameters(poly) :
    poly : 2-Polytope

    Returns the list of hyperplane parameters for facets of the polygon.

    Limitation : 2D specific.
   #### Examples:

```python
triangle = Polygon(Point(0,0), Point(1,1), Point(1,0))
hyperplane_parameters(triangle)
```

### best_origin(a, b, lineseg, expr) :
    a, b : Line parameters of the line-segment
    expr : Uni/Bi-variate polynomial

    Returns a point on the lineseg whose vector inner product with the divergence of expr yields an expression with
    the least maximum total power. This is for reducing the number of computations in the integration reduction call.

    Limitation : 2D specific.

   #### Examples:

```python
print("Best origin for x**3*y on x + y = 3 : ", best_origin((1,1), 3, Segment2D(Point(0, 3), Point(3, 0)), x**3*y))
print("Best origin for x*y**3 on x + y = 3 : ",best_origin((1,1), 3, Segment2D(Point(0, 3), Point(3, 0)), x*y**3))
```

### decompose(expr, separate=False) :
    expr : Uni/Bi-variate polynomial.
    separate(default : False) : If separate is True then return list of constituting monomials.

    Returns a dictionary of the terms having same total power. This is done to get homogeneous polynomials of
    different degrees from the expression.

   #### Examples:

```python
print(decompose(1 + x + x**2 + x*y))
print(decompose(x**2 + x + y + 1 + x**3 + x**2*y + y**4 + x**3*y + y**2*x**2))
print(decompose(x**2 + x + y + 1 + x**3 + x**2*y + y**4 + x**3*y + y**2*x**2, 1))
```

### norm(expr) :

    point : Tuple/SymPy Point object/Dictionary

    Returns Euclidean norm of the point object.

   #### Examples:

```python
print(norm((1, 2)))
print(norm(Point(1, 2)))
print(norm({x: 3, y: 3, z: 1}))
```

### intersection(lineseg_1, lineseg_2) :

    lineseg_1, lineseg_2 : The input line segements whose intersection is to be found.

    Returns intersection point of two lines of which lineseg_1, lineseg_2 are part of. This function is
    called for adjacent line segments so the intersection point is always present with line segment boundaries.

   #### Examples:

```python
print(intersection(Segment2D(Point(0, 0), Point(2, 2)), Segment2D(Point(1, 0), Point(0, 1))))
print(intersection(Segment2D(Point(2, 0), Point(2, 2)), Segment2D(Point(0, 0), Point(4, 4))))
```

### is_vertex(ent) :

    ent : Geometrical entity to denote a vertex.

    Returns True if ent is a vertex. Currently tuples of length 2 or 3 and SymPy Point object are supported.
   #### Examples:

```python
print(is_vertex(Point(2, 8)))
print(is_vertex(Point(2, 8, 1)))
print(is_vertex((1, 1)))
print(is_vertex([2, 9]))
print(is_vertex(Polygon(Point(0, 0), Point(1, 1), Point(1, 0))))
```

### plot_polytope(poly) :

    poly : 2-Polytope

    Plots the 2-Polytope. Currently just defers it to plotting module in SymPy which in turn uses matplotlib.

   #### Examples:

```python
hexagon = Polygon(Point(0, 0), Point(-sqrt(3) / 2, 0.5),
                  Point(-sqrt(3) / 2, 3 / 2), Point(0, 2),
                  Point(sqrt(3) / 2, 3 / 2), Point(sqrt(3) / 2, 0.5))
plot_polytope(hexagon)

twist = Polygon(Point(-1, 1), Point(0, 0), Point(1, 1), Point(1, -1),
                Point(0, 0), Point(-1, -1))
plot_polytope(twist)

```

### plot_polynomial(expr) :

    expr : The uni/bi-variate polynomial to plot

    Plots the polynomial. Currently just defers it to plotting module in SymPy which in turn uses matplotlib.

   #### Examples:

```python
expr = x**2
plot_polynomial(expr)

expr = x*y
plot_polynomial(expr)
```

```python

```


# Solution {-}

Given the following spectral density function:

\begin{equation*}
  S(j\omega)=\delta(\omega)+\frac{1}{2}\delta(\omega-\omega_0)+\frac{1}{2}\delta(\omega+\omega_0)+\frac{2}{\omega^2+1}
\end{equation*}

1. Find the autocorrelation function

\begin{align*}
  R(\tau)=&\frac{1}{2\pi}+\frac{1}{4\pi}e^{j\omega_0\tau}+\frac{1}{4\pi}e^{-j\omega_0\tau}+e^{-|\tau|}\\
         =&\frac{1}{2\pi}+\frac{1}{4\pi}\cos \omega_0\tau+e^{-|\tau|}
\end{align*}

```python
from sympy import symbols, inverse_fourier_transform, DiracDelta, pi

t, w,  w0 = symbols('tau omega omega_0')

S = DiracDelta(w) + 1/2*DiracDelta(w - w0) + 1/2*DiracDelta(w + w0) + 2/(w**2 + 1)
S
```

$\displaystyle \delta\left(\omega\right) + 0.5 \delta\left(\omega - \omega_{0}\right) + 0.5 \delta\left(\omega + \omega_{0}\right) + \frac{2}{\omega^{2} + 1}$

```python
R = 1/(2*pi)*inverse_fourier_transform(S, w, t)
R
```

$\displaystyle \frac{0.5 e^{2 i \pi \omega_{0} \tau} + 1 + 0.5 e^{- 2 i \pi \omega_{0} \tau} + 2 \pi e^{- 2 \pi \tau}}{2 \pi}$


### Imports

```python
import numpy as np
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from IPython import display
plt.style.use('seaborn-white')
```

### Read and process data

```python
data = open('input.txt', 'r').read()
```

Process data and calculate indexes

```python
chars = list(set(data))
data_size, X_size = len(data), len(chars)
print("data has %d characters, %d unique" % (data_size, X_size))
char_to_idx = {ch:i for i,ch in enumerate(chars)}
idx_to_char = {i:ch for i,ch in enumerate(chars)}
```

    data has 1115402 characters, 65 unique

### Constants and Hyperparameters

```python
H_size = 100 # Size of the hidden layer
T_steps = 25 # Number of time steps (length of the sequence) used for training
learning_rate = 1e-1 # Learning rate
weight_sd = 0.1 # Standard deviation of weights for initialization
z_size = H_size + X_size # Size of concatenate(H, X) vector
```

### Activation Functions and Derivatives

#### Sigmoid

\begin{align}
\sigma(x) &= \frac{1}{1 + e^{-x}}\\
\frac{d\sigma(x)}{dx} &= \sigma(x) \cdot (1 - \sigma(x))
\end{align}

#### Tanh

\begin{align}
\frac{d\text{tanh}(x)}{dx} &= 1 - \text{tanh}^2(x)
\end{align}

```python
def sigmoid(x):
    return 1 / (1 + np.exp(-x))

def dsigmoid(y):
    return y * (1 - y)

def tanh(x):
    return np.tanh(x)

def dtanh(y):
    return 1 - y * y
```

### Parameters

```python
class Param:
    def __init__(self, name, value):
        self.name = name
        self.v = value #parameter value
        self.d = np.zeros_like(value) #derivative
        self.m = np.zeros_like(value) #momentum for AdaGrad
```

We use random weights with normal distribution (`0`, `weight_sd`) for $tanh$ activation function and (`0.5`, `weight_sd`) for $sigmoid$ activation function.

Biases are initialized to zeros.

```python
class Parameters:
    def __init__(self):
        self.W_f = Param('W_f',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_f = Param('b_f',
                         np.zeros((H_size, 1)))

        self.W_i = Param('W_i',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_i = Param('b_i',
                         np.zeros((H_size, 1)))

        self.W_C = Param('W_C',
                         np.random.randn(H_size, z_size) * weight_sd)
        self.b_C = Param('b_C',
                         np.zeros((H_size, 1)))

        self.W_o = Param('W_o',
                         np.random.randn(H_size, z_size) * weight_sd + 0.5)
        self.b_o = Param('b_o',
                         np.zeros((H_size, 1)))

        #For final layer to predict the next character
        self.W_v = Param('W_v',
                         np.random.randn(X_size, H_size) * weight_sd)
        self.b_v = Param('b_v',
                         np.zeros((X_size, 1)))

    def all(self):
        return [self.W_f, self.W_i, self.W_C, self.W_o, self.W_v,
               self.b_f, self.b_i, self.b_C, self.b_o, self.b_v]

parameters = Parameters()
```

### Forward pass

*Operation $z$ is the concatenation of $x$ and $h_{t-1}$*

#### Concatenation of $h_{t-1}$ and $x_t$
\begin{align}
z & = [h_{t-1}, x_t] \\
\end{align}

#### LSTM functions
\begin{align}
f_t & = \sigma(W_f \cdot z + b_f) \\
i_t & = \sigma(W_i \cdot z + b_i) \\
\bar{C}_t & = tanh(W_C \cdot z + b_C) \\
C_t & = f_t * C_{t-1} + i_t * \bar{C}_t \\
o_t & = \sigma(W_o \cdot z + b_t) \\
h_t &= o_t * tanh(C_t) \\
\end{align}

#### Logits
\begin{align}
v_t &= W_v \cdot h_t + b_v \\
\end{align}

#### Softmax
\begin{align}
\hat{y_t} &= \text{softmax}(v_t)
\end{align}

$\hat{y_t}$ is `y` in code and $y_t$ is `targets`.

```python
def forward(x, h_prev, C_prev, p = parameters):
    assert x.shape == (X_size, 1)
    assert h_prev.shape == (H_size, 1)
    assert C_prev.shape == (H_size, 1)

    z = np.row_stack((h_prev, x))
    f = sigmoid(np.dot(p.W_f.v, z) + p.b_f.v)
    i = sigmoid(np.dot(p.W_i.v, z) + p.b_i.v)
    C_bar = tanh(np.dot(p.W_C.v, z) + p.b_C.v)

    C = f * C_prev + i * C_bar
    o = sigmoid(np.dot(p.W_o.v, z) + p.b_o.v)
    h = o * tanh(C)

    v = np.dot(p.W_v.v, h) + p.b_v.v
    y = np.exp(v) / np.sum(np.exp(v)) #softmax

    return z, f, i, C_bar, C, o, h, v, y
```

### Backward pass

#### Loss

\begin{align}
L_k &= -\sum_{t=k}^T\sum_j y_{t,j} log \hat{y_{t,j}} \\
L &= L_1 \\
\end{align}

#### Gradients

\begin{align}
dv_t &= \hat{y_t} - y_t \\
dh_t &= dh'_t + W_y^T \cdot dv_t \\
do_t &= dh_t * \text{tanh}(C_t) \\
dC_t &= dC'_t + dh_t * o_t * (1 - \text{tanh}^2(C_t))\\
d\bar{C}_t &= dC_t * i_t \\
di_t &= dC_t * \bar{C}_t \\
df_t &= dC_t * C_{t-1} \\
\\
df'_t &= f_t * (1 - f_t) * df_t \\
di'_t &= i_t * (1 - i_t) * di_t \\
d\bar{C}'_{t-1} &= (1 - \bar{C}_t^2) * d\bar{C}_t \\
do'_t &= o_t * (1 - o_t) * do_t \\
dz_t &= W_f^T \cdot df'_t \\
     &+ W_i^T \cdot di_t \\
     &+ W_C^T \cdot d\bar{C}_t \\
     &+ W_o^T \cdot do_t \\
\\
[dh'_{t-1}, dx_t] &= dz_t \\
dC'_t &= f_t * dC_t
\end{align}

* $dC'_t = \frac{\partial L_{t+1}}{\partial C_t}$ and $dh'_t = \frac{\partial L_{t+1}}{\partial h_t}$
* $dC_t = \frac{\partial L}{\partial C_t} = \frac{\partial L_t}{\partial C_t}$ and $dh_t = \frac{\partial L}{\partial h_t} = \frac{\partial L_{t}}{\partial h_t}$
* All other derivatives are of $L$
* `target` is target character index $y_t$
* `dh_next` is $dh'_{t}$ (size H x 1)
* `dC_next` is $dC'_{t}$ (size H x 1)
* `C_prev` is $C_{t-1}$ (size H x 1)
* $df'_t$, $di'_t$, $d\bar{C}'_t$, and $do'_t$ are *also* assigned to `df`, `di`, `dC_bar`, and `do` in the **code**.
* *Returns* $dh_t$ and $dC_t$

#### Model parameter gradients

\begin{align}
dW_v &= dv_t \cdot h_t^T \\
db_v &= dv_t \\
\\
dW_f &= df'_t \cdot z^T \\
db_f &= df'_t \\
\\
dW_i &= di'_t \cdot z^T \\
db_i &= di'_t \\
\\
dW_C &= d\bar{C}'_t \cdot z^T \\
db_C &= d\bar{C}'_t \\
\\
dW_o &= do'_t \cdot z^T \\
db_o &= do'_t \\
\\
\end{align}

```python
def backward(target, dh_next, dC_next, C_prev,
             z, f, i, C_bar, C, o, h, v, y,
             p = parameters):

    assert z.shape == (X_size + H_size, 1)
    assert v.shape == (X_size, 1)
    assert y.shape == (X_size, 1)

    for param in [dh_next, dC_next, C_prev, f, i, C_bar, C, o, h]:
        assert param.shape == (H_size, 1)

    dv = np.copy(y)
    dv[target] -= 1

    p.W_v.d += np.dot(dv, h.T)
    p.b_v.d += dv

    dh = np.dot(p.W_v.v.T, dv)
    dh += dh_next
    do = dh * tanh(C)
    do = dsigmoid(o) * do
    p.W_o.d += np.dot(do, z.T)
    p.b_o.d += do

    dC = np.copy(dC_next)
    dC += dh * o * dtanh(tanh(C))
    dC_bar = dC * i
    dC_bar = dtanh(C_bar) * dC_bar
    p.W_C.d += np.dot(dC_bar, z.T)
    p.b_C.d += dC_bar

    di = dC * C_bar
    di = dsigmoid(i) * di
    p.W_i.d += np.dot(di, z.T)
    p.b_i.d += di

    df = dC * C_prev
    df = dsigmoid(f) * df
    p.W_f.d += np.dot(df, z.T)
    p.b_f.d += df

    dz = (np.dot(p.W_f.v.T, df)
         + np.dot(p.W_i.v.T, di)
         + np.dot(p.W_C.v.T, dC_bar)
         + np.dot(p.W_o.v.T, do))
    dh_prev = dz[:H_size, :]
    dC_prev = f * dC

    return dh_prev, dC_prev
```

### Forward Backward Pass

Clear gradients before each backward pass

```python
def clear_gradients(params = parameters):
    for p in params.all():
        p.d.fill(0)
```

Clip gradients to mitigate exploding gradients

```python
def clip_gradients(params = parameters):
    for p in params.all():
        np.clip(p.d, -1, 1, out=p.d)
```

Calculate and store the values in forward pass. Accumulate gradients in backward pass and clip gradients to avoid exploding gradients.

* `input`, `target` are list of integers, with character indexes.
* `h_prev` is the array of initial `h` at $h_{-1}$ (size H x 1)
* `C_prev` is the array of initial `C` at $C_{-1}$ (size H x 1)
* *Returns* loss, final $h_T$ and $C_T$

```python
def forward_backward(inputs, targets, h_prev, C_prev):
    global paramters

    # To store the values for each time step
    x_s, z_s, f_s, i_s,  = {}, {}, {}, {}
    C_bar_s, C_s, o_s, h_s = {}, {}, {}, {}
    v_s, y_s =  {}, {}

    # Values at t - 1
    h_s[-1] = np.copy(h_prev)
    C_s[-1] = np.copy(C_prev)

    loss = 0
    # Loop through time steps
    assert len(inputs) == T_steps
    for t in range(len(inputs)):
        x_s[t] = np.zeros((X_size, 1))
        x_s[t][inputs[t]] = 1 # Input character

        (z_s[t], f_s[t], i_s[t],
        C_bar_s[t], C_s[t], o_s[t], h_s[t],
        v_s[t], y_s[t]) = \
            forward(x_s[t], h_s[t - 1], C_s[t - 1]) # Forward pass

        loss += -np.log(y_s[t][targets[t], 0]) # Loss for at t

    clear_gradients()

    dh_next = np.zeros_like(h_s[0]) #dh from the next character
    dC_next = np.zeros_like(C_s[0]) #dh from the next character

    for t in reversed(range(len(inputs))):
        # Backward pass
        dh_next, dC_next = \
            backward(target = targets[t], dh_next = dh_next,
                     dC_next = dC_next, C_prev = C_s[t-1],
                     z = z_s[t], f = f_s[t], i = i_s[t], C_bar = C_bar_s[t],
                     C = C_s[t], o = o_s[t], h = h_s[t], v = v_s[t],
                     y = y_s[t])

    clip_gradients()

    return loss, h_s[len(inputs) - 1], C_s[len(inputs) - 1]
```

### Sample the next character

```python
def sample(h_prev, C_prev, first_char_idx, sentence_length):
    x = np.zeros((X_size, 1))
    x[first_char_idx] = 1

    h = h_prev
    C = C_prev

    indexes = []

    for t in range(sentence_length):
        _, _, _, _, C, _, h, _, p = forward(x, h, C)
        idx = np.random.choice(range(X_size), p=p.ravel())
        x = np.zeros((X_size, 1))
        x[idx] = 1
        indexes.append(idx)

    return indexes
```

## Training (Adagrad)

Update the graph and display a sample output

```python
def update_status(inputs, h_prev, C_prev):
    #initialized later
    global plot_iter, plot_loss
    global smooth_loss

    # Get predictions for 200 letters with current model

    sample_idx = sample(h_prev, C_prev, inputs[0], 200)
    txt = ''.join(idx_to_char[idx] for idx in sample_idx)

    # Clear and plot
    plt.plot(plot_iter, plot_loss)
    display.clear_output(wait=True)
    plt.show()

    #Print prediction and loss
    print("----\n %s \n----" % (txt, ))
    print("iter %d, loss %f" % (iteration, smooth_loss))
```

Update parameters

\begin{align}
\theta_i &= \theta_i - \eta\frac{d\theta_i}{\sum dw_{\tau}^2} \\
d\theta_i &= \frac{\partial L}{\partial \theta_i}
\end{align}

```python
def update_paramters(params = parameters):
    for p in params.all():
        p.m += p.d * p.d # Calculate sum of gradients
        #print(learning_rate * dparam)
        p.v += -(learning_rate * p.d / np.sqrt(p.m + 1e-8))
```

To delay the keyboard interrupt to prevent the training
from stopping in the middle of an iteration

```python
import signal

class DelayedKeyboardInterrupt(object):
    def __enter__(self):
        self.signal_received = False
        self.old_handler = signal.signal(signal.SIGINT, self.handler)

    def handler(self, sig, frame):
        self.signal_received = (sig, frame)
        print('SIGINT received. Delaying KeyboardInterrupt.')

    def __exit__(self, type, value, traceback):
        signal.signal(signal.SIGINT, self.old_handler)
        if self.signal_received:
            self.old_handler(*self.signal_received)
```

```python
# Exponential average of loss
# Initialize to a error of a random model
smooth_loss = -np.log(1.0 / X_size) * T_steps

iteration, pointer = 0, 0

# For the graph
plot_iter = np.zeros((0))
plot_loss = np.zeros((0))
```

Training loop

```python
while True:
    try:
        with DelayedKeyboardInterrupt():
            # Reset
            if pointer + T_steps >= len(data) or iteration == 0:
                g_h_prev = np.zeros((H_size, 1))
                g_C_prev = np.zeros((H_size, 1))
                pointer = 0

            inputs = ([char_to_idx[ch]
                       for ch in data[pointer: pointer + T_steps]])
            targets = ([char_to_idx[ch]
                        for ch in data[pointer + 1: pointer + T_steps + 1]])

            loss, g_h_prev, g_C_prev = \
                forward_backward(inputs, targets, g_h_prev, g_C_prev)
            smooth_loss = smooth_loss * 0.999 + loss * 0.001

            # Print every hundred steps
            if iteration % 100 == 0:
                update_status(inputs, g_h_prev, g_C_prev)

            update_paramters()

            plot_iter = np.append(plot_iter, [iteration])
            plot_loss = np.append(plot_loss, [loss])

            pointer += T_steps
            iteration += 1
    except KeyboardInterrupt:
        update_status(inputs, g_h_prev, g_C_prev)
        break
```

### Gradient Check

Approximate the numerical gradients by changing parameters and running the model. Check if the approximated gradients are equal to the computed analytical gradients (by backpropagation).

Try this on `num_checks` individual paramters picked randomly for each weight matrix and bias vector.

```python
from random import uniform
```

Calculate numerical gradient

```python
def calc_numerical_gradient(param, idx, delta, inputs, target, h_prev, C_prev):
    old_val = param.v.flat[idx]

    # evaluate loss at [x + delta] and [x - delta]
    param.v.flat[idx] = old_val + delta
    loss_plus_delta, _, _ = forward_backward(inputs, targets,
                                             h_prev, C_prev)
    param.v.flat[idx] = old_val - delta
    loss_mins_delta, _, _ = forward_backward(inputs, targets,
                                             h_prev, C_prev)

    param.v.flat[idx] = old_val #reset

    grad_numerical = (loss_plus_delta - loss_mins_delta) / (2 * delta)
    # Clip numerical error because analytical gradient is clipped
    [grad_numerical] = np.clip([grad_numerical], -1, 1)

    return grad_numerical
```

Check gradient of each paramter matrix/vector at `num_checks` individual values

```python
def gradient_check(num_checks, delta, inputs, target, h_prev, C_prev):
    global parameters

    # To calculate computed gradients
    _, _, _ =  forward_backward(inputs, targets, h_prev, C_prev)

    for param in parameters.all():
        #Make a copy because this will get modified
        d_copy = np.copy(param.d)

        # Test num_checks times
        for i in range(num_checks):
            # Pick a random index
            rnd_idx = int(uniform(0, param.v.size))

            grad_numerical = calc_numerical_gradient(param,
                                                     rnd_idx,
                                                     delta,
                                                     inputs,
                                                     target,
                                                     h_prev, C_prev)
            grad_analytical = d_copy.flat[rnd_idx]

            err_sum = abs(grad_numerical + grad_analytical) + 1e-09
            rel_error = abs(grad_analytical - grad_numerical) / err_sum

            # If relative error is greater than 1e-06
            if rel_error > 1e-06:
                print('%s (%e, %e) => %e'
                      % (param.name, grad_numerical, grad_analytical, rel_error))
```

```python
gradient_check(10, 1e-5, inputs, targets, g_h_prev, g_C_prev)
```


### Importing packages

#### Others

```python
# Used to read the data file
import pandas as pd

# Used in the definition of the model and its initial condition
import numpy as np

# Used for plots other than those generated by the parameter estimation module
import matplotlib.pyplot as plt
%matplotlib inline
```

#### The module itself

```python
import PDEparams as pde
```

### Defining the model

We use a classic two-species predator-prey model:

$$\begin{align}
\frac{\mathrm{d} x}{\mathrm{d} t} &= a\, x(1-x)-b\,x\,y\\
\frac{\mathrm{d} y}{\mathrm{d} t} &= b\,x\,y-y
\end{align}$$

Note that the arguments below do not include a spatial grid, since we are dealing with ODEs.

```python
def LotkaVolterra(z, t, a, b):
    '''The input z corresponds to the current state of the system, z = [x, y]. Since the input is in 1D, no
    pre-processing is needed.

    t is the current time.

    a and b correspond to the unknown parameters.
    '''

    x, y = z

    return [a*x*(1-x)-b*x*y, b*x*y-y]
```

Here we specify the functions that define the initial condition for each variable. Since there is no spatial dependence, the initial functions don't take any arguments and return a constant value. The initial condition used for data generation corresponds to $(x,y)=(0.3,0.5)$

```python
def initial_x():
    return 0.3

def initial_y():
    return 0.5
```

### Using `PDEparams` to estimate parameters

First, we load the data from the `.csv` file.

The data consist of 3 replicates, and have been generated using parameter values $a=3$, $b=1$.

The columns are, in order: $t$, $x$, $y$.

```python
df = pd.read_csv('LotkaVolterraData.csv')

df.head()
```

      0
      1
      2

      0
      0.060606
      0.325940
      0.469672

      1
      0.060606
      0.326474
      0.475169

      2
      0.060606
      0.317529
      0.486987

      3
      0.070707
      0.340549
      0.492719

      4
      0.070707
      0.324513
      0.470383

#### Constructing the `PDEmodel` object.

The inputs are

**Required:**
1. The data table `df`.
2. The model `LotkaVolterra`.
3. The list of initial condition functions.
4. The bounds for the parameter values.

**Optional:**
1. The parameter names.
2. The number of variables: 2. **(Default is 1, this needs to be provided in this case)**
3. The number of spatial dimensions: 0. **(Default is 1, this needs to be provided in this case)**
4. The number of replicates in the data: 3. **(Default is 1, this needs to be provided in this case)**
5. The indices of the measured variables. In this case, the default `None`, since we have data for all 2 variables.
6. The function to apply to the output. In this case, the default `None`, since our data is directly $x$ and $y$.

```python
my_model = pde.PDEmodel(df, LotkaVolterra, [initial_x, initial_y], bounds=[(2, 4), (0.5,2)],
                        param_names=[r'$a$', r'$b$'], nvars=2, ndims=0, nreplicates=3, obsidx=None, outfunc=None)
```

```python
# Printing the initial condition
my_model.initial_condition
```

    array([0.3, 0.5])

#### Finding the best fit parameters.

```python
%%time
my_model.fit()
```

            $a$       $b$
    0  2.993904  1.001379
    CPU times: user 279 ms, sys: 1.73 ms, total: 280 ms
    Wall time: 290 ms

```python
my_model.best_params
```

      $a$
      $b$

      0
      2.993904
      1.001379

```python
my_model.best_error
```

    9.413324634610873e-05

#### Likelihood profiles

We use the default grid of 100 points per parameter.

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model.likelihood_profiles()
```

    HBox(children=(IntProgress(value=0, description='parameters', max=2, style=ProgressStyle(description_width='in…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    CPU times: user 32.8 s, sys: 114 ms, total: 33 s
    Wall time: 36.3 s

The raw results

```python
my_model.result_profiles
```

      parameter
      value
      error

      0
      $a$
      2.000000
      0.003617

      1
      $a$
      2.020202
      0.003437

      2
      $a$
      2.040404
      0.003264

      3
      $a$
      2.060606
      0.003099

      4
      $a$
      2.080808
      0.002942

      5
      $a$
      2.101010
      0.002792

      6
      $a$
      2.121212
      0.002649

      7
      $a$
      2.141414
      0.002513

      8
      $a$
      2.161616
      0.002385

      9
      $a$
      2.181818
      0.002263

      10
      $a$
      2.202020
      0.002149

      11
      $a$
      2.222222
      0.002041

      12
      $a$
      2.242424
      0.001940

      13
      $a$
      2.262626
      0.001841

      14
      $a$
      2.282828
      0.001746

      15
      $a$
      2.303030
      0.001653

      16
      $a$
      2.323232
      0.001562

      17
      $a$
      2.343434
      0.001475

      18
      $a$
      2.363636
      0.001390

      19
      $a$
      2.383838
      0.001308

      20
      $a$
      2.404040
      0.001228

      21
      $a$
      2.424242
      0.001151

      22
      $a$
      2.444444
      0.001077

      23
      $a$
      2.464646
      0.001006

      24
      $a$
      2.484848
      0.000937

      25
      $a$
      2.505051
      0.000871

      26
      $a$
      2.525253
      0.000807

      27
      $a$
      2.545455
      0.000747

      28
      $a$
      2.565657
      0.000689

      29
      $a$
      2.585859
      0.000634

      ...
      ...
      ...
      ...

      170
      $b$
      1.560606
      0.002154

      171
      $b$
      1.575758
      0.002278

      172
      $b$
      1.590909
      0.002405

      173
      $b$
      1.606061
      0.002537

      174
      $b$
      1.621212
      0.002673

      175
      $b$
      1.636364
      0.002813

      176
      $b$
      1.651515
      0.002957

      177
      $b$
      1.666667
      0.003105

      178
      $b$
      1.681818
      0.003258

      179
      $b$
      1.696970
      0.003415

      180
      $b$
      1.712121
      0.003577

      181
      $b$
      1.727273
      0.003742

      182
      $b$
      1.742424
      0.003912

      183
      $b$
      1.757576
      0.004087

      184
      $b$
      1.772727
      0.004266

      185
      $b$
      1.787879
      0.004449

      186
      $b$
      1.803030
      0.004636

      187
      $b$
      1.818182
      0.004829

      188
      $b$
      1.833333
      0.005025

      189
      $b$
      1.848485
      0.005226

      190
      $b$
      1.863636
      0.005432

      191
      $b$
      1.878788
      0.005642

      192
      $b$
      1.893939
      0.005856

      193
      $b$
      1.909091
      0.006075

      194
      $b$
      1.924242
      0.006299

      195
      $b$
      1.939394
      0.006527

      196
      $b$
      1.954545
      0.006759

      197
      $b$
      1.969697
      0.006996

      198
      $b$
      1.984848
      0.007238

      199
      $b$
      2.000000
      0.007484

200 rows × 3 columns

Visualisation

```python
my_model.plot_profiles()
```

#### Bootstrapping

We use the default 100 repetitions.

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model.bootstrap()
```

    HBox(children=(IntProgress(value=0, description='runs', style=ProgressStyle(description_width='initial')), HTM…

                  $a$         $b$
    count  100.000000  100.000000
    mean     2.990633    0.999784
    std      0.010713    0.007454
    min      2.965644    0.982297
    25%      2.983347    0.995204
    50%      2.990437    0.999729
    75%      2.997088    1.003723
    max      3.025524    1.020776
    CPU times: user 26.9 s, sys: 46.2 ms, total: 27 s
    Wall time: 30 s

The summary

```python
my_model.bootstrap_summary
```

      $a$
      $b$

      count
      100.000000
      100.000000

      mean
      2.990633
      0.999784

      std
      0.010713
      0.007454

      min
      2.965644
      0.982297

      25%
      2.983347
      0.995204

      50%
      2.990437
      0.999729

      75%
      2.997088
      1.003723

      max
      3.025524
      1.020776

The raw results

```python
my_model.bootstrap_raw
```

      $a$
      $b$

      0
      3.003469
      1.018804

      1
      3.001168
      1.010144

      2
      2.987095
      0.992268

      3
      2.984709
      1.002661

      4
      2.978693
      0.997317

      5
      2.985765
      0.986396

      6
      2.987959
      1.001987

      7
      2.993803
      1.008941

      8
      2.999806
      0.995116

      9
      2.991132
      0.988603

      10
      2.996658
      1.003443

      11
      2.988690
      0.997330

      12
      3.006180
      1.008987

      13
      3.010555
      1.007097

      14
      2.996630
      0.999482

      15
      2.998953
      1.001399

      16
      2.988674
      1.000994

      17
      2.988828
      0.994609

      18
      2.991506
      1.002362

      19
      2.991680
      1.003272

      20
      2.981228
      0.997796

      21
      2.988050
      1.005017

      22
      2.974612
      0.991993

      23
      2.981773
      0.997162

      24
      3.003005
      1.006047

      25
      2.993998
      1.004142

      26
      3.002892
      1.001237

      27
      2.978464
      1.000084

      28
      2.983403
      1.006757

      29
      2.996396
      1.013988

      ...
      ...
      ...

      70
      2.974032
      0.995756

      71
      2.991096
      0.999210

      72
      2.983527
      1.002550

      73
      3.002846
      1.017781

      74
      2.985241
      1.001709

      75
      2.996998
      1.001349

      76
      2.991980
      0.986823

      77
      2.994040
      1.005094

      78
      2.988161
      1.006140

      79
      3.011394
      1.012344

      80
      2.965644
      0.982297

      81
      3.007071
      0.998107

      82
      2.992535
      0.996828

      83
      2.994667
      1.000332

      84
      3.002469
      1.002912

      85
      3.000906
      1.004076

      86
      2.976200
      0.985227

      87
      3.001998
      0.998558

      88
      2.983179
      0.998220

      89
      2.983757
      0.993848

      90
      2.984062
      0.996825

      91
      2.995231
      1.009268

      92
      2.982378
      0.995767

      93
      2.984077
      0.992630

      94
      2.992840
      0.995233

      95
      2.992151
      1.003035

      96
      3.000065
      1.003652

      97
      2.986510
      0.997617

      98
      2.977861
      0.997611

      99
      2.988477
      0.995422

100 rows × 2 columns

Visualisation

```python
my_model.plot_bootstrap()
```

### Only one observed variable

Let us suppose now that we only have data for $y$.

```python
df2 = df[df.columns[[0,2]]] # x, which we are ignoring, is column 1 in the data table

df2.head()
```

      0
      2

      0
      0.060606
      0.469672

      1
      0.060606
      0.475169

      2
      0.060606
      0.486987

      3
      0.070707
      0.492719

      4
      0.070707
      0.470383

#### Constructing the `PDEmodel` object.

The inputs are the same as before, except for the data—now `df2` instead of `df`—and `obsidx`, which specifies that we only have data for the 2nd variable in the system. Starting from zero, the corresponding index is 1. Since we only have one observed variable, `obsidx` can be either a number or a list of one element.

```python
my_model2 = pde.PDEmodel(df2, LotkaVolterra, [initial_x, initial_y], bounds=[(2, 4), (0.5,2)],
                         param_names=[r'$a$', r'$b$'], nvars=2, ndims=0, nreplicates=3, obsidx=[1], outfunc=None)
```

#### Finding the best fit parameters.

```python
%%time
my_model2.fit()
```

           $a$       $b$
    0  2.83873  1.029909
    CPU times: user 176 ms, sys: 25 µs, total: 176 ms
    Wall time: 179 ms

```python
my_model2.best_params
```

      $a$
      $b$

      0
      2.83873
      1.029909

```python
my_model2.best_error
```

    9.187040918800129e-05

#### Likelihood profiles

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model2.likelihood_profiles()
```

    HBox(children=(IntProgress(value=0, description='parameters', max=2, style=ProgressStyle(description_width='in…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    CPU times: user 43.7 s, sys: 104 ms, total: 43.8 s
    Wall time: 49.7 s

The raw results

```python
my_model2.result_profiles
```

      parameter
      value
      error

      0
      $a$
      2.000000
      0.000102

      1
      $a$
      2.020202
      0.000101

      2
      $a$
      2.040404
      0.000101

      3
      $a$
      2.060606
      0.000100

      4
      $a$
      2.080808
      0.000100

      5
      $a$
      2.101010
      0.000099

      6
      $a$
      2.121212
      0.000099

      7
      $a$
      2.141414
      0.000098

      8
      $a$
      2.161616
      0.000098

      9
      $a$
      2.181818
      0.000097

      10
      $a$
      2.202020
      0.000097

      11
      $a$
      2.222222
      0.000096

      12
      $a$
      2.242424
      0.000096

      13
      $a$
      2.262626
      0.000096

      14
      $a$
      2.282828
      0.000095

      15
      $a$
      2.303030
      0.000095

      16
      $a$
      2.323232
      0.000095

      17
      $a$
      2.343434
      0.000095

      18
      $a$
      2.363636
      0.000094

      19
      $a$
      2.383838
      0.000094

      20
      $a$
      2.404040
      0.000094

      21
      $a$
      2.424242
      0.000094

      22
      $a$
      2.444444
      0.000093

      23
      $a$
      2.464646
      0.000093

      24
      $a$
      2.484848
      0.000093

      25
      $a$
      2.505051
      0.000093

      26
      $a$
      2.525253
      0.000093

      27
      $a$
      2.545455
      0.000093

      28
      $a$
      2.565657
      0.000093

      29
      $a$
      2.585859
      0.000092

      ...
      ...
      ...
      ...

      170
      $b$
      1.560606
      0.000553

      171
      $b$
      1.575758
      0.000594

      172
      $b$
      1.590909
      0.000637

      173
      $b$
      1.606061
      0.000681

      174
      $b$
      1.621212
      0.000727

      175
      $b$
      1.636364
      0.000775

      176
      $b$
      1.651515
      0.000824

      177
      $b$
      1.666667
      0.000875

      178
      $b$
      1.681818
      0.000927

      179
      $b$
      1.696970
      0.000981

      180
      $b$
      1.712121
      0.001037

      181
      $b$
      1.727273
      0.001094

      182
      $b$
      1.742424
      0.001152

      183
      $b$
      1.757576
      0.001212

      184
      $b$
      1.772727
      0.001273

      185
      $b$
      1.787879
      0.001336

      186
      $b$
      1.803030
      0.001400

      187
      $b$
      1.818182
      0.001465

      188
      $b$
      1.833333
      0.001532

      189
      $b$
      1.848485
      0.001600

      190
      $b$
      1.863636
      0.001669

      191
      $b$
      1.878788
      0.001740

      192
      $b$
      1.893939
      0.001812

      193
      $b$
      1.909091
      0.001885

      194
      $b$
      1.924242
      0.001959

      195
      $b$
      1.939394
      0.002035

      196
      $b$
      1.954545
      0.002112

      197
      $b$
      1.969697
      0.002190

      198
      $b$
      1.984848
      0.002269

      199
      $b$
      2.000000
      0.002349

200 rows × 3 columns

Visualisation

```python
my_model2.plot_profiles()
```

#### Bootstrapping

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model2.bootstrap()
```

    HBox(children=(IntProgress(value=0, description='runs', style=ProgressStyle(description_width='initial')), HTM…

                  $a$         $b$
    count  100.000000  100.000000
    mean     2.928028    1.017756
    std      0.401623    0.070162
    min      2.151616    0.851199
    25%      2.641515    0.968889
    50%      2.817236    1.029071
    75%      3.133648    1.065381
    max      3.994846    1.196032
    CPU times: user 22.4 s, sys: 118 ms, total: 22.5 s
    Wall time: 25.2 s

The summary

```python
my_model2.bootstrap_summary
```

      $a$
      $b$

      count
      100.000000
      100.000000

      mean
      2.928028
      1.017756

      std
      0.401623
      0.070162

      min
      2.151616
      0.851199

      25%
      2.641515
      0.968889

      50%
      2.817236
      1.029071

      75%
      3.133648
      1.065381

      max
      3.994846
      1.196032

The raw results

```python
my_model2.bootstrap_raw
```

      $a$
      $b$

      0
      3.186450
      0.966881

      1
      2.994018
      0.992822

      2
      2.229250
      1.172231

      3
      2.736436
      1.045355

      4
      3.459128
      0.931309

      5
      2.771381
      1.040382

      6
      2.934312
      1.002232

      7
      2.734665
      1.056124

      8
      3.028957
      0.992229

      9
      2.466829
      1.098155

      10
      3.021764
      0.978904

      11
      2.605139
      1.077158

      12
      2.829591
      1.028945

      13
      2.829072
      1.036971

      14
      2.786716
      1.024529

      15
      2.410657
      1.125444

      16
      2.703599
      1.060549

      17
      2.887153
      1.011713

      18
      2.487482
      1.092487

      19
      2.551906
      1.073295

      20
      3.351057
      0.950278

      21
      3.196387
      0.963497

      22
      3.185130
      0.963274

      23
      3.989835
      0.860150

      24
      3.264633
      0.953557

      25
      2.269456
      1.175272

      26
      2.629859
      1.069511

      27
      2.619347
      1.070337

      28
      3.017502
      0.995884

      29
      2.633942
      1.060323

      ...
      ...
      ...

      70
      2.617034
      1.085760

      71
      2.513154
      1.086689

      72
      2.631748
      1.058986

      73
      2.473401
      1.110266

      74
      3.120630
      0.969463

      75
      3.085647
      0.967169

      76
      2.662882
      1.066937

      77
      2.660752
      1.054912

      78
      3.309790
      0.961033

      79
      3.911726
      0.873615

      80
      2.542773
      1.091506

      81
      2.151616
      1.196032

      82
      2.512438
      1.103099

      83
      2.879644
      1.023525

      84
      3.143682
      0.962638

      85
      2.503046
      1.110733

      86
      3.265422
      0.951855

      87
      3.529360
      0.915736

      88
      2.782934
      1.044810

      89
      2.946202
      1.009231

      90
      2.536247
      1.101490

      91
      3.074829
      0.979491

      92
      3.985203
      0.870064

      93
      3.135816
      0.976863

      94
      2.553888
      1.080913

      95
      2.823732
      1.032444

      96
      3.671526
      0.906530

      97
      2.674485
      1.060072

      98
      3.931037
      0.859171

      99
      2.771113
      1.050614

100 rows × 2 columns

Visualisation

```python
my_model2.plot_bootstrap()
```

### Only a function of the variables is observed

Finally, we consider the case when it's not the raw outputs of the system that are observed, but rather a function of them. In this case, let us assume that the observed quantity is

$$\mathcal{F}(x, y) = x\,y$$

```python
df3 = df[df.columns[[0]]] # we take only the time-points

# We add the new function
df3[r'$\mathcal{F}$'] = df['1']*df['2'] # x and y are, respectively, columns 1 and 2 in the data table

df3.head()
```

      0
      $\mathcal{F}$

      0
      0.060606
      0.153085

      1
      0.060606
      0.155130

      2
      0.060606
      0.154632

      3
      0.070707
      0.167795

      4
      0.070707
      0.152645

#### Constructing the `PDEmodel` object.

The inputs are the same as before, except for the data—now `df3`—and `outfunc`, which specifies the function to be applied to the outputs before computing the error.

```python
def F(z):
    x, y = z

    return x*y
```

```python
my_model3 = pde.PDEmodel(df3, LotkaVolterra, [initial_x, initial_y], bounds=[(2, 4), (0.5,2)],
                         param_names=[r'$a$', r'$b$'], nvars=2, ndims=0, nreplicates=3, obsidx=None, outfunc=F)
```

#### Finding the best fit parameters.

```python
%%time
my_model3.fit()
```

            $a$       $b$
    0  3.013358  0.962529
    CPU times: user 1.16 s, sys: 4.47 ms, total: 1.16 s
    Wall time: 1.47 s

```python
my_model3.best_params
```

      $a$
      $b$

      0
      3.013358
      0.962529

```python
my_model3.best_error
```

    4.443543597892799e-05

#### Likelihood profiles

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model3.likelihood_profiles()
```

    HBox(children=(IntProgress(value=0, description='parameters', max=2, style=ProgressStyle(description_width='in…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    HBox(children=(IntProgress(value=0, description='values within parameters', style=ProgressStyle(description_wi…

    CPU times: user 2min 5s, sys: 234 ms, total: 2min 5s
    Wall time: 2min 24s

The raw results

```python
my_model3.result_profiles
```

      parameter
      value
      error

      0
      $a$
      2.000000
      0.002442

      1
      $a$
      2.020202
      0.002340

      2
      $a$
      2.040404
      0.002239

      3
      $a$
      2.060606
      0.002140

      4
      $a$
      2.080808
      0.002043

      5
      $a$
      2.101010
      0.001948

      6
      $a$
      2.121212
      0.001854

      7
      $a$
      2.141414
      0.001763

      8
      $a$
      2.161616
      0.001674

      9
      $a$
      2.181818
      0.001587

      10
      $a$
      2.202020
      0.001503

      11
      $a$
      2.222222
      0.001420

      12
      $a$
      2.242424
      0.001340

      13
      $a$
      2.262626
      0.001261

      14
      $a$
      2.282828
      0.001185

      15
      $a$
      2.303030
      0.001112

      16
      $a$
      2.323232
      0.001040

      17
      $a$
      2.343434
      0.000971

      18
      $a$
      2.363636
      0.000904

      19
      $a$
      2.383838
      0.000840

      20
      $a$
      2.404040
      0.000778

      21
      $a$
      2.424242
      0.000719

      22
      $a$
      2.444444
      0.000662

      23
      $a$
      2.464646
      0.000608

      24
      $a$
      2.484848
      0.000556

      25
      $a$
      2.505051
      0.000507

      26
      $a$
      2.525253
      0.000460

      27
      $a$
      2.545455
      0.000416

      28
      $a$
      2.565657
      0.000375

      29
      $a$
      2.585859
      0.000336

      ...
      ...
      ...
      ...

      170
      $b$
      1.560606
      0.000092

      171
      $b$
      1.575758
      0.000094

      172
      $b$
      1.590909
      0.000095

      173
      $b$
      1.606061
      0.000096

      174
      $b$
      1.621212
      0.000098

      175
      $b$
      1.636364
      0.000099

      176
      $b$
      1.651515
      0.000100

      177
      $b$
      1.666667
      0.000102

      178
      $b$
      1.681818
      0.000103

      179
      $b$
      1.696970
      0.000104

      180
      $b$
      1.712121
      0.000106

      181
      $b$
      1.727273
      0.000107

      182
      $b$
      1.742424
      0.000108

      183
      $b$
      1.757576
      0.000109

      184
      $b$
      1.772727
      0.000110

      185
      $b$
      1.787879
      0.000111

      186
      $b$
      1.803030
      0.000112

      187
      $b$
      1.818182
      0.000113

      188
      $b$
      1.833333
      0.000114

      189
      $b$
      1.848485
      0.000115

      190
      $b$
      1.863636
      0.000116

      191
      $b$
      1.878788
      0.000117

      192
      $b$
      1.893939
      0.000118

      193
      $b$
      1.909091
      0.000119

      194
      $b$
      1.924242
      0.000120

      195
      $b$
      1.939394
      0.000121

      196
      $b$
      1.954545
      0.000121

      197
      $b$
      1.969697
      0.000122

      198
      $b$
      1.984848
      0.000123

      199
      $b$
      2.000000
      0.000123

200 rows × 3 columns

Visualisation

```python
my_model3.plot_profiles()
```

#### Bootstrapping

**Note:** if you see a "widget not found" message, just ignore it; a progress bar will appear when you run the cell below.

```python
%%time
my_model3.bootstrap()
```

    HBox(children=(IntProgress(value=0, description='runs', style=ProgressStyle(description_width='initial')), HTM…

                  $a$         $b$
    count  100.000000  100.000000
    mean     3.010285    0.965811
    std      0.019293    0.030561
    min      2.963211    0.892835
    25%      2.998660    0.946928
    50%      3.009469    0.967059
    75%      3.021914    0.985858
    max      3.060564    1.053666
    CPU times: user 46.2 s, sys: 71.1 ms, total: 46.3 s
    Wall time: 51.7 s

The summary

```python
my_model3.bootstrap_summary
```

      $a$
      $b$

      count
      100.000000
      100.000000

      mean
      3.010285
      0.965811

      std
      0.019293
      0.030561

      min
      2.963211
      0.892835

      25%
      2.998660
      0.946928

      50%
      3.009469
      0.967059

      75%
      3.021914
      0.985858

      max
      3.060564
      1.053666

The raw results

```python
my_model3.bootstrap_raw
```

      $a$
      $b$

      0
      3.004544
      0.975553

      1
      3.011460
      0.944719

      2
      3.022243
      0.956418

      3
      3.041521
      0.945383

      4
      3.033754
      0.943021

      5
      3.008854
      0.952098

      6
      3.004783
      0.966086

      7
      3.021300
      0.984676

      8
      2.993500
      0.981311

      9
      3.001813
      0.992382

      10
      3.026699
      0.948382

      11
      3.005744
      1.003509

      12
      3.019751
      0.951532

      13
      3.023140
      0.917367

      14
      2.992995
      1.005790

      15
      3.007374
      0.947082

      16
      3.017306
      0.943250

      17
      2.996537
      0.985509

      18
      2.993801
      0.976761

      19
      2.985710
      1.012736

      20
      3.007024
      0.983270

      21
      3.026094
      0.933456

      22
      2.998895
      0.997762

      23
      3.004754
      0.954400

      24
      2.994770
      0.949515

      25
      3.017222
      0.957733

      26
      3.009368
      0.949289

      27
      3.003725
      0.967571

      28
      3.001721
      1.000419

      29
      3.041532
      0.930961

      ...
      ...
      ...

      70
      3.011897
      0.981077

      71
      3.046875
      0.897459

      72
      3.018500
      0.953763

      73
      2.974220
      1.053666

      74
      2.977706
      1.004024

      75
      2.997120
      0.951527

      76
      2.986780
      1.023515

      77
      3.028154
      0.968776

      78
      3.007104
      0.940048

      79
      3.038248
      0.926103

      80
      2.997953
      0.957914

      81
      3.002111
      0.971376

      82
      2.986691
      0.967823

      83
      3.019109
      0.946467

      84
      3.013573
      0.983129

      85
      3.023152
      0.966546

      86
      3.000544
      0.965003

      87
      3.020180
      0.948670

      88
      3.006561
      0.965161

      89
      3.031220
      0.940459

      90
      2.997364
      0.986108

      91
      3.009976
      0.991857

      92
      3.037319
      0.954682

      93
      3.016250
      0.948644

      94
      3.016100
      0.927069

      95
      3.011795
      0.952902

      96
      3.060564
      0.893783

      97
      3.052337
      0.907510

      98
      2.979961
      0.988719

      99
      2.981551
      0.985775

100 rows × 2 columns

Visualisation

```python
my_model3.plot_bootstrap()
```


# Linear Regression

We will use a real-life dataset containing information about size and weight for several fish species.

1. species name of fish (Bream, Roach, Whitefish, Parkki, Perch, Pike, Smelt)
2. weight of fish in grams
3. vertical length in cm (`Length1`)
4. diagonal length in cm (`Length2`)
5. cross length in cm (`Length3`)
6. height in cm
7. diagonal width in cm

We would like to see if we can develop a way to predict the weight of a fish given its dimensions.

The relationship between total length (L, in inches) and total weight (W, in pounds) for nearly all species of fish can be expressed by an equation of the form

\begin{align}
W = cL^b
\end{align}

`W` is weight in pounds and `L` is length in inches. `b` is a constant that is close to 3.0 for all species and `c` is a constance that varies among species (For [northern pike](https://en.wikipedia.org/wiki/Northern_pike#Length_and_weight), for example, `b = 3.096` and `c = 0.000180`).

## Task

**Can we predict the weight of a fish given its dimensions and species?**

## Acquiring the data
First let's load the data into a [Pandas](https://mazelife.github.io/cmsc-291/lecture_10.html#pandas) dataframe:

```python
import pandas

fish_dataframe = pandas.read_csv("Fish.csv", dtype={"Species": "category"})
fish_dataframe
```

      Species
      Weight
      Length1
      Length2
      Length3
      Height
      Width

      0
      Bream
      242.0
      23.2
      25.4
      30.0
      11.5200
      4.0200

      1
      Bream
      290.0
      24.0
      26.3
      31.2
      12.4800
      4.3056

      2
      Bream
      340.0
      23.9
      26.5
      31.1
      12.3778
      4.6961

      3
      Bream
      363.0
      26.3
      29.0
      33.5
      12.7300
      4.4555

      4
      Bream
      430.0
      26.5
      29.0
      34.0
      12.4440
      5.1340

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      154
      Smelt
      12.2
      11.5
      12.2
      13.4
      2.0904
      1.3936

      155
      Smelt
      13.4
      11.7
      12.4
      13.5
      2.4300
      1.2690

      156
      Smelt
      12.2
      12.1
      13.0
      13.8
      2.2770
      1.2558

      157
      Smelt
      19.7
      13.2
      14.3
      15.2
      2.8728
      2.0672

      158
      Smelt
      19.9
      13.8
      15.0
      16.2
      2.9322
      1.8792

159 rows × 7 columns

```python
import seaborn
# Seaborn has various themes/styles. We'll pick a good default:
seaborn.set_theme(style="darkgrid")
seaborn.set_context("talk")

seaborn.relplot(x="Length1", y="Weight", hue="Species", data=fish_dataframe, height=8, aspect=1.5)
```

    <seaborn.axisgrid.FacetGrid at 0x7fdb387193d0>

## Simplifying the problem

For the moment, let's attempt to do univariate linear regression. Let's take *one* measure of dimension (length) and attempt to predict weight for just one species: Pike. We'll start with a scatter plot with just the Pike data:

```python
pike_dataframe = fish_dataframe[fish_dataframe.Species == "Pike"]
seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
```

Now we will train a linear regression model using the [scikit-Learn](https://scikit-learn.org/) Python library. One of the first things we will need to do is extract our `X` and `Y` values as individual arrays from the source dataframe, as scikit-learn does not work directly with Pandas. For our case, we want to extract a single DataFrame column for our X values and a single column for our Y values as a pair of arrays. The way to do this is as follows:

```
data_array = dataframe["column name"].values.reshape(-1, 1)
```

From there we will create an instance of the `LinearRegression` and then call `fit()` on it, which will train the model.

```python
from sklearn.linear_model import LinearRegression

x_values = pike_dataframe.Length1.values.reshape(-1, 1)
y_values = pike_dataframe.Weight.values.reshape(-1, 1)
pike_model = LinearRegression()
pike_model.fit(x_values, y_values)

x_values
```

    array([[30. ],
           [31.7],
           [32.7],
           [34.8],
           [35.5],
           [36. ],
           [40. ],
           [40. ],
           [40.1],
           [42. ],
           [43.2],
           [44.8],
           [48.3],
           [52. ],
           [56. ],
           [56. ],
           [59. ]])

Once we have fit the model, we can ask it to do predictions. As we talked about last week, a linear regression model gives us an equation that describes a line:

\begin{align}
y = mx + b \\
m = slope \\
b = intercept
\end{align}

The model tells us what `m` and `b` are (these are called the *coefficients*), so that when we plug in a value for `x`, we will have our prediction for `y`. We can extract those coefficients from the scikit-learn model we just trained.

```python
intercept = pike_model.intercept_[0]
slope = pike_model.coef_[0][0]
print(f"Intercept is: {intercept}")
print(f"Slope is: {slope}")
```

    Intercept is: -1540.824272231702
    Slope is: 53.19486584675106

We can also draw the line that our model has produced on top of the scatterplot to see how well it seems to fit:

```python
import matplotlib.pyplot as plt
import numpy

# Generate an array of 2 points at 27 and 60 on the x-axis:
x_pred = numpy.array([27, 60])
# Create the corresponding y values for the two X points:
y_pred = slope * x_pred + intercept

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_pred, y_pred)
```

## Evaluating the model

Remember **mean-squared error**?

Let's calculate it! Scikit-learn will help us here, we just need to give it two arrays: one of predicted y values and one of actual y values.

```python
from sklearn.metrics import mean_squared_error

predicted_y = pike_model.predict(x_values)
pike_mse = mean_squared_error(y_values, predicted_y)

print(f"MSE is {pike_mse:,.2f}")
pandas.DataFrame({'Actual': y_values.flatten(), 'Predicted': predicted_y.flatten()})

```

    MSE is 12,692.68

      Actual
      Predicted

      0
      200.0
      55.021703

      1
      300.0
      145.452975

      2
      300.0
      198.647841

      3
      300.0
      310.357059

      4
      430.0
      347.593465

      5
      345.0
      374.190898

      6
      456.0
      586.970362

      7
      510.0
      586.970362

      8
      540.0
      592.289848

      9
      500.0
      693.360093

      10
      567.0
      757.193932

      11
      770.0
      842.305718

      12
      950.0
      1028.487748

      13
      1250.0
      1225.308752

      14
      1600.0
      1438.088215

      15
      1550.0
      1438.088215

      16
      1650.0
      1597.672813

## Overfitting

The only problem here is we don't know if we've overfit.

A better way to measure how well our model is to split our data into training and test datasets. Scikit-learn will help us with this too:

```python
from sklearn.model_selection import train_test_split

# Split 20% of our data off into testing data:
x_training_data, x_test_data, y_training_data, y_test_data = train_test_split(x_values, y_values, test_size=0.2, random_state=0)

# Train our model as usual with the 80% of the data we reserved for training:
pike_model = LinearRegression()
pike_model.fit(x_training_data, y_training_data)

# Then have it predict X values
predicted_y = pike_model.predict(x_test_data)
print("Mean-squared error: {:,.2f}".format(mean_squared_error(y_test_data, predicted_y)))
pandas.DataFrame({'Actual': y_test_data.flatten(), 'Predicted': predicted_y.flatten()})
```

    Mean-squared error: 22,682.16

      Actual
      Predicted

      0
      300.0
      153.707099

      1
      456.0
      601.323002

      2
      540.0
      606.715965

      3
      500.0
      709.182256

## Polynomial Regression

Our model works...OK. But the MSE on "real-world" data is somewhat high. You could say that out model under-fits the data. To overcome under-fitting, we need to increase the complexity of the model. What if we try to fit this line with a curve?

```python
from sklearn.preprocessing import PolynomialFeatures

poly_reg = PolynomialFeatures(degree=2)
x_poly = poly_reg.fit_transform(x_values)
#poly_reg.fit(x_poly,y_values)
poly_pike_model = LinearRegression()
poly_pike_model.fit(x_poly,y_values)

```

    LinearRegression()

```python
# Generate an array of 2 points at 27 and 60 on the x-axis:
x_pred = numpy.array([27, 60])
# Create the corresponding y values for the two X points:
y_pred = slope * x_pred + intercept

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_pred, y_pred, color="red")

print("Mean-squared error: {}".format(mean_squared_error(y_test_data, predicted_y)))
pandas.DataFrame({'Actual': y_test_data.flatten(), 'Predicted': predicted_y.flatten()})
```

```python
# Generate an array of points:
x_grid = numpy.arange(min(x_values), max(x_values), 0.1)
x_grid = x_grid.reshape((len(x_grid), 1))

# Create the corresponding y values for the X points using a polynomial model:
y_grid = poly_pike_model.predict(poly_reg.fit_transform(x_grid))

seaborn.relplot(x="Length1", y="Weight", data=pike_dataframe, height=8, aspect=1.5)
plt.plot(x_grid, y_grid, color="red")

predicted_y = poly_pike_model.predict(poly_reg.fit_transform(x_values))
print("Mean-squared error: {}".format(mean_squared_error(y_values, predicted_y)))
```

```python
x_values
```

    array([[30. ],
           [31.7],
           [32.7],
           [34.8],
           [35.5],
           [36. ],
           [40. ],
           [40. ],
           [40.1],
           [42. ],
           [43.2],
           [44.8],
           [48.3],
           [52. ],
           [56. ],
           [56. ],
           [59. ]])

```python
x_poly
```

    array([[1.00000e+00, 3.00000e+01, 9.00000e+02],
           [1.00000e+00, 3.17000e+01, 1.00489e+03],
           [1.00000e+00, 3.27000e+01, 1.06929e+03],
           [1.00000e+00, 3.48000e+01, 1.21104e+03],
           [1.00000e+00, 3.55000e+01, 1.26025e+03],
           [1.00000e+00, 3.60000e+01, 1.29600e+03],
           [1.00000e+00, 4.00000e+01, 1.60000e+03],
           [1.00000e+00, 4.00000e+01, 1.60000e+03],
           [1.00000e+00, 4.01000e+01, 1.60801e+03],
           [1.00000e+00, 4.20000e+01, 1.76400e+03],
           [1.00000e+00, 4.32000e+01, 1.86624e+03],
           [1.00000e+00, 4.48000e+01, 2.00704e+03],
           [1.00000e+00, 4.83000e+01, 2.33289e+03],
           [1.00000e+00, 5.20000e+01, 2.70400e+03],
           [1.00000e+00, 5.60000e+01, 3.13600e+03],
           [1.00000e+00, 5.60000e+01, 3.13600e+03],
           [1.00000e+00, 5.90000e+01, 3.48100e+03]])

```python
simple_xs = numpy.array([[1], [2], [3]])
simple_xs
```

    array([[1],
           [2],
           [3]])

```python
PolynomialFeatures(degree=2).fit_transform(simple_xs)

```

    array([[1., 1., 1.],
           [1., 2., 4.],
           [1., 3., 9.]])

```python

```


# Riddle #1: An NEO with a peculiar orbit

**Author: Juan Luis Cano Rodríguez <hello@juanlu.space>**

## Statement

http://neo.ssa.esa.int/neocc-riddles

> The recently discovered asteroid 2020 HY5, firstly observed by Mt. Lem
mon Survey on 23 April 2020, has an interesting particularity: it roughly
passes half of its orbital period of about 387 days inside 1.3 au and the
other half outside. Such distance is used for the definition of NEOs: the
perihelion distance of an NEO must be below 1.3 au. 2020 HY5 actually
spends 192.0 days below 1.3 au and 194.9 days above that distance.
>
> And here is a riddle:
>
> - Assuming an NEO that spends exactly 50% of its time inside 1.3 au and
50% of its time outside 1.3 au, what would be the maximum aphelion
such an NEO could have?
> - As a bonus, would you be able to find similar cases in our database?
(Hint: you can use the advanced search functionality in our left menu)

## Solution

```python
import numpy as np

import matplotlib.pyplot as plt

from sympy import init_printing, symbols, solve, nsolve, diff, Eq, sin, cos, Function, pi
from sympy.abc import a, b, e, E, r, p, nu

init_printing()
```

Remembering the second Kepler law of planetary motion:

> The area of the sector traced by the radius vector from the sun, between any two positions of a planet in its orbit, is proportional to the time occupied in passing from one position to the other.
> (Battin, 1999)

Therefore, if the time inside the NEO distance $R_0$ is the same as the time outside, the subtended areas must be the same.

Area of the focal sector of the ellipse:

```python
A = (E - e * sin(E)) * a * b / 2
A
```

where $E$ is the eccentric anomaly, $a$ and $b$ are the semimajor and semiminor axis respectively, and $e$ is the eccentricity.

We now equate the area for $E = E_0$ to the remaining area, that is $A - A(E_0)$, where $E_0$ is the eccentric anomaly at which $r = R_0$:

```python
E0 = Function("E_0")(e)
E0
```

```python
areas_eq = Eq(A.subs(E, E0), A.subs(E, pi) - A.subs(E, E0))
areas_eq
```

```python
E0_eq = Eq(sin(E0), solve(areas_eq, sin(E0))[0])
E0_eq
```

This is the same as

$$E_0 - e \sin(E_0) = \frac{\pi}{2}$$

Now we use the polar equation of the ellipse:

```python
polar_eq = Eq(r, p / (1 + e * cos(nu)))
polar_eq
```

Where $p$ is the semilatus rectum and $nu$ is the true anomaly.

Using the relation between true anomaly $\nu$ and eccentric anomaly $E$:

$$\cos{\nu} = \frac{\cos{E} - e}{1 - e \cos{E}}$$

We can obtain an expression for $p$ in terms of $E$:

```python
R0 = symbols("R_0")
R0
```

```python
p_eq = solve(
    polar_eq.subs(cos(nu), (cos(E) - e) / (1 - e * cos(E))).subs(E, E0).subs(r, R0),
    p
)[0]
p_eq
```

And now we use the relation between the apocenter $r_a$ and the semilatus rectum $p$ to solve for our variable of interest:

```python
r_a = symbols("r_a")
r_a
```

```python
r_a_eq = solve(Eq(r_a * (1 - e), p_eq), r_a)[0]
r_a_eq
```

The problem statement suggests that the aphelion has a maximum. Is that true? Let's visualize it numerically first.

If we fix the eccentricity $e$, we get a value of the aphelion:

- $e = 0.1$
- $R_0 = 1.3$

```python
E0_ = nsolve((E - e * sin(E) - pi / 2).subs(e, 0.1), E, 1.5)
E0_
```

```python
r_a_ = r_a_eq.subs(E0, E0_).subs(e, 0.1).subs(R0, 1.3)
r_a_
```

Let's plot the aphelion $r_a$ versus the eccentricity $e$:

```python
def compute_r_a(e_, R0_=1.3):
    E0_ = nsolve((E - e * sin(E) - pi / 2).subs(e, e_), (pi / 2).evalf())
    return r_a_eq.subs(E0, E0_).subs(e, e_).subs(R0, R0_)
```

```python
e_arr = np.linspace(0, 1, 101)
r_a_arr = np.zeros_like(e_arr)

for ii, e_ in enumerate(e_arr):
    r_a_arr[ii] = float(compute_r_a(e_))
```

```python
fig, ax = plt.subplots()

l, = ax.plot(e_arr, r_a_arr)

ax.set_xlabel("Eccentricity $e$")
ax.set_ylabel("Aphelion $r_a$")

t = ax.set_title("Half-split distance equal to $R_0 = 1.3$ AU")
```

There is a maximum! Let's find it by equating the derivative of $r_a$ with respect to $e$ to zero:

```python
diff(r_a_eq, e)
```

We solve for $\frac{d}{de} E_0(e)$ by differentiating the implicit equation:

```python
E0_eq
```

```python
dE0_e = solve(
    Eq(
        diff(E0_eq.lhs, e),
        diff(E0_eq.rhs, e),
    ),
    diff(E0, e)
)[0]
dE0_e
```

And now replace:

```python
max_r_a_eq = Eq(diff(r_a_eq, e).subs(diff(E0, e), dE0_e), 0).simplify()
max_r_a_eq
```

```python
E0_eq
```

We have a system of two nonlinear equations for the eccentricy $e$ and the half-split eccentric anomaly $E_0$. One of them can be trivially solved for $e$, and we can replace that solution in the other one to end with a single equation in $E_0$:

```python
E0_ = symbols("E0_")
E0_
```

```python
e_ = solve(E0_eq.subs(E0, E0_), e)[0]
e_
```

```python
max_r_a_eq_E0 = max_r_a_eq.subs(E0, E0_).subs(e, e_)
max_r_a_eq_E0
```

And finally, we solve numerically:

```python
E0_max_r_a = nsolve(max_r_a_eq_E0.subs(R0, 1.3).lhs, 2)
E0_max_r_a
```

```python
e_max_r_a = e_.subs(E0_, E0_max_r_a).evalf()
e_max_r_a
```

Therefore, **the maximum aphelion this peculiar NEO can have is**:

```python
max_r_a = r_a_eq.subs(R0, 1.3).subs(E0, E0_max_r_a).subs(e, e_max_r_a)
max_r_a
```

■

This theoretical value is in fact very close to the actual aphelion of [2020 HY5](http://neo.ssa.esa.int/search-for-asteroids?sum=1&des=2020HY5): 1.609 AU.

To find other similar cases, we can pick pairs of (eccentricity, aphelion) that fulfill the condition we analyzed above:

```python
fig
```

Alternatively, we can compute the "time fraction" $\tau$: time spent inside $R_0$ over time spent outside:

$$\tau = \frac{T_i}{T_e} = \frac{2 n (E_0 - e \sin{E_0})}{2 n \pi - 2 n (E_0 - e \sin{E_0})} =
\frac{E_0 - e \sin{E_0}}{\pi - (E_0 - e \sin{E_0})}
$$

For that, we explore a grid of $(e, r_a)$ pairs, which give us the shape of the orbit, and compute $E_0$ as the intersection with $R_0$:

```python
E0_direct = solve(
    polar_eq.subs(r, R0).subs(p, r_a * (1 - e)).subs(cos(nu), (cos(E0_) - e) / (1 - e * cos(E0_))),
    E0_
)[-1]
E0_direct
```

which we can then use to calculate $\tau$:

```python
tau = (E0_ - e * sin(E0_)) / (pi - (E0_ - e * sin(E0_)))
tau
```

```python
e_arr_xx, r_a_arr_yy = np.meshgrid(
    np.linspace(0, 1, num=101),
    np.linspace(1.3, 1.65, num=101)
)

tau_xx_yy = np.zeros_like(e_arr_xx)

for ii in range(e_arr_xx.shape[0]):
    for jj in range(e_arr_xx.shape[1]):
        e_ii_jj = e_arr_xx[ii, jj]
        r_a_ii_jj = r_a_arr_yy[ii, jj]

        try:
            E0_ii_jj = float(E0_direct.subs(R0, 1.3).subs(e, e_ii_jj).subs(r_a, r_a_ii_jj))
        except TypeError:
            # Some solutions are complex
            E_ii_jj = float("nan")

        tau_xx_yy[ii, jj] = float(tau.subs(e, e_ii_jj).subs(E0_, E0_ii_jj))
```

And now we visualize it:

```python
fig, ax = plt.subplots(figsize=(12, 6))

c = ax.contourf(
    e_arr_xx, r_a_arr_yy, tau_xx_yy,
    levels=np.linspace(0, 2, 17), cmap=plt.cm.RdYlGn,
)
ax.plot(e_arr, r_a_arr, color="black", label=r"$\tau = 1$")

ax.plot(
    e_max_r_a, max_r_a, "x", markeredgewidth=2, color="red", markersize=10,
    label=r"Max $r_a$ with $\tau = 1$"
)

ax.text(0.34, 1.53, "More time\ninside $R_0$", {"fontsize": 16}, rotation=15)
ax.text(0.30, 1.59, "More time\noutside $R_0$", {"fontsize": 16}, rotation=15)

ax.legend()

ax.set_xlabel("Eccentricity $e$")
ax.set_ylabel("Aphelion $r_a$")

t = ax.set_title("Time fraction")

cb = fig.colorbar(c)
```

For example, the region with $e \in [0.4, 0.8]$ and $r_a \in [1.5, 1.6]$ will have a time fraction close to 1. Using these filters in the [ESA NEO Coordination Centre Advanced Search](http://neo.ssa.esa.int/advanced-search), we find about 150 examples, like [1865 Cerberus](http://neo.ssa.esa.int/search-for-asteroids?sum=1&des=1865%20Cerberus):

- Perihelion: 0.576 AU
- Aphelion: 1.584 AU
- Eccentricity: 0.467
- Period: 409.9 days

The eccentric anomaly of the intersection at $R_0$ is:

```python
E0_1865_Cerberus = E0_direct.subs(R0, 1.3).subs(e, 0.467).subs(r_a, 1.584)
E0_1865_Cerberus
```

And its time fraction is:

```python
tau_1865_Cerberus = tau.subs(e, 0.467).subs(E0_, E0_1865_Cerberus).evalf()
tau_1865_Cerberus
```

```python
tau_1865_Cerberus / (1 + tau_1865_Cerberus) * 409.9
```

Therefore, it spends about 209 days inside $R_0 = 1.3$ AU, and 200 days outside that radius.

On the other hand, [2005EP1](http://neo.ssa.esa.int/search-for-asteroids?sum=1&des=2005EP1) has this orbit:

- Perihelion: 0.205 AU
- Aphelion: 1.58 AU
- Eccentricity: 0.77
- Period: 307.9 days

```python
E0_2005EP1 = E0_direct.subs(R0, 1.3).subs(e, 0.77).subs(r_a, 1.58)
E0_2005EP1
```

```python
tau_2005EP1 = tau.subs(e, 0.77).subs(E0_, E0_2005EP1).evalf()
tau_2005EP1
```

```python
tau_2005EP1 / (1 + tau_2005EP1) * 307.9
```

This one spends 155 days inside, and 153 days outside.

## References

1. Battin, R. (1999). _An introduction to the mathematics and methods of astrodynamics, revised edition_. American Institute of Aeronautics and Astronautics.


# Example usage of rdsolver

(c) 2018 Justin Bois. This work is licensed under a [Creative Commons Attribution License CC-BY 4.0](https://creativecommons.org/licenses/by/4.0/). All code contained herein is licensed under an [MIT license](https://opensource.org/licenses/MIT).

`rdsolver` solves the following system of PDEs on a 2D Cartesian domain with periodic boundary conditions. The governing equations are

\begin{align}
\partial_t c_i = D_i(\partial_x^2 + \partial_y^2) c_i + \beta_i + \sum_j\gamma_{ij} c_j + f_i(\mathbf{c}),
\end{align}

where summation over like indices is not implied; all summation is explicit. The last term represents all nonlinear chemical reactions. The $\beta_i + \gamma_{ij}c_j$ terms are linear chemical dynamics. Note that we assume a diagonal constant diffusion tensor.

To specify the problem, the user needs to supply:

* The physical dimension of the system, which we will call $\mathbf{L} = (L_x, L_y)$
* The number of grid points $\mathbf{n} = (n_x, n_y)$
* The desired time points $t_0, t_1, \ldots$
* The initial concentration profiles of all species, $\mathbf{c}^0(x, y)$
* The values of the parameters $D_i$, $\beta_i$, and $\gamma_{ij}$.
* The function $f_i$ and any parametric arguments that need to be passed to it.

Here, I present an example of how to use `rdsolver`. To learn more about it and installation instructions, see the [README file](https://github.com/justinbois/rdsolver#reaction-diffusion-solver).

## Necessary imports

We work with `rdsolver` in a Jupyter notebook (recommended), you need to import `rdsolver` and `bokeh`, being sure to call `bokeh.io.output_notebook()` to enable interactive plotting in the Jupyter notebook. And it's pretty much automatic to import NumPy!

```python
%load_ext autoreload
%autoreload 2

import numpy as np
import numba

import rdsolver as rd

import bokeh
import bokeh.io
bokeh.io.output_notebook()
```

    Loading BokehJS ...

## The activator-substrate depletion model (ASDM)

The ASDM is a classic system that gives Turing patterns. In its simplest form, the dimensionless equations are

\begin{align}
\partial_t a &= d(\partial_x^2 + \partial_y^2)a + a^2s - a \\[0.5em]
\partial_t s &= (\partial_x^2 + \partial_y^2)s + \mu(1 - a^2s).
\end{align}

Thus, we have

\begin{align}
D &= (d, 1) \\[0.5em]
\beta &= (0, \mu) \\[0.5em]
\gamma &= \begin{pmatrix}
-1 & 0 \\
0 & 0
\end{pmatrix}.
\end{align}

We start by specifying the easy stuff, the physical size of the system, the number of grid points, and the time points we want. Because this particular system does not have very sharp gradients and we are not using a very large physical space, we do not need many grid points at all. Because `rdsolver` uses spectral methods, we can have very accurate calculations even with few grid points. We will use as 32 $\times$ 32 grid here.

```python
# Physical size of system
L = (10, 10)

# Number of grid points in x and y (can often be small with spec. meth.)
n = (32, 32)

# Specify times points we want
t = np.linspace(0, 250, 100)
```

Now, we need to define our parameters. We'll start with $D$, $\beta$, and $\gamma$, choosing $d = 0.05$ and $\mu = 1.4$.

```python
d = 0.05
mu = 1.4

D = (d, 1)
beta = (0, mu)
gamma = np.array([[-1, 0], [0, 0]])
```

Now, we need to define our nonlinear function $f$. This function has call signature `f(u, t, *f_args)`. The first argument, `u`, is an array containing the concentrations that can be unpacked as `a, s = u`. Note, however, that unpacking 3D arrays like this is not yet supported in Numba, so you should use the more verbose version below for Numba'd nonlinear functions.

Next, the function `f` takes the current time as an input. For the ASDM, and indeed for many R-D applications, the nonlinear chemical dynamics do not explicitly depend on time. Finally, `f_args` is a tuple containing any other arguments the function `f` needs.

The function must return an array the same shape as the input array `u` that gives the nonlinear terms in the dynamics. This is most easily accomplished using the `np.stack()` function.

```python
@numba.jit(nopython=True)
def f(u, t, mu):
    """Nonlinear terms for ASDM"""
    a = u[0,:,:]
    s = u[1,:,:]
    return np.stack((a**2 * s, -mu * a**2 * s))
```

We also have to specify the arguments that need to be passed to `f` as a tuple.

```python
# Specify the arguments that need to be passed to f
f_args = (mu, )
```

Now, we need to specify the initial conditions. The initial conditions must be a three-dimensional array of shape $(n_s, n_x, n_y)$, where $n_s$ is the number of chemical species. For convenience, you can specify a homogeneous steady state and use the `rd.initial_condition()` function that will generate an initial condition that is a small perturbation about the specified steady state.

```python
# Homogenous steady state in activator and substrate to perturb
uniform_conc = (1, 1)

# Generate perturbed initial condition
np.random.seed(42)
c0 = rd.initial_condition(uniform_conc=uniform_conc, n=n, L=L)

# Show the shape as a demonstration
c0.shape
```

    (2, 32, 32)

Now, everything is in place. We just have to solve. We do this by calling the `rd.solve()` function. The arguments are obvious from the call below.

```python
# Solve the system
c = rd.solve(c0, t, D=D, beta=beta, gamma=gamma, f=f, f_args=f_args, L=L)

# Look at the shape of the solution
c.shape
```

    /Users/bois/Dropbox/git/rdsolver/rdsolver/rd.py:172: NumbaPerformanceWarning: np.dot() is faster on contiguous arrays, called on (array(float64, 2d, C), array(float64, 1d, A))
      out[:,i,j] = np.dot(gamma, c[:,i,j])
      0%|          | 0/100 [00:00<?, ?it/s]/Users/bois/Dropbox/git/rdsolver/rdsolver/rd.py:497: NumbaPerformanceWarning: np.dot() is faster on contiguous arrays, called on (array(complex128, 2d, C), array(complex128, 1d, A))
      + np.dot(A_rhs, c_hat[:,i,j])
    /Users/bois/Dropbox/git/rdsolver/rdsolver/rd.py:497: NumbaPerformanceWarning: np.dot() is faster on contiguous arrays, called on (array(complex128, 2d, C), array(complex128, 1d, A))
      + np.dot(A_rhs, c_hat[:,i,j])
    100%|██████████| 100/100 [00:14<00:00,  7.00it/s]

    (2, 32, 32, 100)

We note that the solution is of the shape $(n_s, n_x, n_y, n_t)$, where $n_t$ is the number of time points we used. This structure is useful to know for slicing out species and time points of interest.

For plotting purposes, it is useful to interpolate the solution to have smooth concentration profiles. This is purely aesthetic; the solver will give a pixelated, but spectrally accurate, solution. We can use the `rd.viz.interpolate_concs()` function to get the interpolated concentration profiles.

```python
c_interp = rd.viz.interpolate_concs(c)
```

Finally, we are ready to display the solution. We use the `rd.viz.display_notebook()` function that will give a picture of the concentration field with a slider for adjusting the time. By default, for multiple species problems, (like the ASDM), up to three species are shown. The cyan channel is the first, magenta the second, and yellow is the third (though it is not present for the ASDM).

```python
# Use whatever your notebook URL is here
notebook_url = 'localhost:8889'

bokeh.io.show(rd.viz.display_notebook(t, c_interp), notebook_url=notebook_url)
```

If we like, we can look at a single species, in which case the colormap is Viridis.

```python
bokeh.io.show(rd.viz.display_notebook(t, c_interp[0]), notebook_url=notebook_url)
```

## The full ASDM model

We just solved a more simplified ASDM model, but we may consider a more complicated one, as proposed by Koch and Meinhardt (*Rev. Mod. Phys.*, 1994), in which the autocatalysis reactions can saturate.

\begin{align}
\partial_t a &= D_a (\partial_x^2 + \partial_y^2)a + \frac{\rho_a a^2 s}{1 + \kappa_a a^2} - \mu_a a + \sigma_a \\[0.5em]
\partial_t s &= D_s (\partial_x^2 + \partial_y^2)s - \frac{\rho_s a^2 s}{1 + \kappa_a a^2} + \sigma_s
\end{align}

For convenience, `rdsolver` has a growing set of models that you can pre-load. We can use `rd.models.asdm()` to load in the parameters, as well as the homogeneous steady state, for the more complete ASDM model.

```python
D, beta, gamma, f, f_args, homo_ss = rd.models.asdm()
```

The parameters in the above equations are inputted as keyword arguments, with the defaults set to what was used to generate Fig. 2 in the Koch and Meinhardt paper. The outputted function `f` is JITted for performance.

Note that if we wanted to recapitulate the simpler example we already did, we can call the function with the appropriate kwargs.

```python
d = 0.05
mu = 1.4
params = {'D_a': d,
          'D_s': 1,
          'rho_a': 1,
          'rho_s': mu,
          'sigma_a': 0,
          'sigma_s': mu,
          'mu_a': 1,
          'kappa_a': 0}
D, beta, gamma, f, f_args, homo_ss = rd.models.asdm(**params)
```

We already did that one, so we'll do the saturating model now. We'll now define the grid setup and the time points we want.

```python
n = (32, 32)
L = (50, 50)
t = np.linspace(0, 100000, 100)
```

Next, the initial condition, which will be a small perturbation about the steady state.

```python
np.random.seed(42)
c0 = rd.initial_condition(uniform_conc=homo_ss, n=n, L=L)
```

Now we can solve and interpolate....

```python
c = rd.solve(c0, t, D=D, beta=beta, gamma=gamma, f=f, f_args=f_args, L=L)
c_interp = rd.viz.interpolate_concs(c)
```

    100%|██████████| 100/100 [00:09<00:00, 10.46it/s]

And finally, we visualize:

```python
bokeh.io.show(rd.viz.display_notebook(t, c_interp), notebook_url=notebook_url)
```

Finally, if we want to display a single image, we can do so using `rd.viz.display_single_frame()`.

```python
bokeh.io.show(rd.viz.display_single_frame(c_interp, i=-1));
```