# Generative Learning Algorithms

## Discriminative & Generative

discriminative：try to learn $p(y|x)$ directly, such as logistic regression

or try to learn mappings from the space of inputs to the labels $\left \{ 0, 1\right \}$ directly, such as perceptron

generative：algorithms that try to model $p(x|y)$ and $p(y)$(called class prior), such as guassian discriminant analysis and naive bayes

when predicting, use bayes rule：

$$p(y|x)=\frac{p(x|y)p(y)}{p(x)}$$

then：

$$\hat{y}=\underset{y}{argmax}\ \frac{p(x|y)p(y)}{p(x)}= \underset{y}{argmax}\ {p(x|y)p(y)}$$

## Guassian Discriminant Analysis(GDA)

multivariant normal distribution

guassian distribution is parameterized by a mean vector $\mu \in \mathbb{R}^{d}$ and a covariance matrix $\Sigma \in \mathbb{R}^{d \times d}$, where $\Sigma >= 0$ is symmetric and positive semi-definite. also written $\mathcal{N}(\mu,\Sigma)$, it's density is given by：

$$p(x;\mu,\Sigma)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }exp\left ( -\frac{1}{2}(x-\mu)^{T}{\Sigma}^{-1}(x-\mu)\right )$$

unsurprisingly, for random variable $X \sim \mathcal{N}(\mu,\Sigma)$:

$$E[X] = \int_{x}xp(x;\mu, \Sigma)dx = \mu$$
$$Cov(X) = E[(X - E(X))(X - E(X))^{T}] = \Sigma$$

GDA:

when we have a classification problem in which the input features x are continous, we can use GDA, which model $p(x|y)$ using a multivariant normal distribution:

$$y\sim Bernoulli(\phi)$$
$$x | y=0 \sim \mathcal{N}(\mu_{0},\Sigma) $$
$$x | y=1 \sim \mathcal{N}(\mu_{1},\Sigma) $$

writing out the distribution:

$$p(y) = \phi^{y}(1 - \phi)^{1 - y}$$

$$p(x| y=0)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }exp\left (-\frac{1}{2}(x-\mu_{0})^{T}{\Sigma}^{-1}(x-\mu_{0})\right )$$

$$p(x| y=1)=\frac{1}{(2\pi)^{d/2}\left | \Sigma \right |^{1/2} }exp\left (-\frac{1}{2}(x-\mu_{1})^{T}{\Sigma}^{-1}(x-\mu_{1})\right )$$

the log-likelihood of the data is given by：

$$
\begin{equation}
\begin{split}
l(\phi,\mu_{0},\mu_{1},\Sigma) &= log\prod_{i=1}^{n}p(x^{(i)},y^{(i)};\phi,\mu_{0},\mu_{1},\Sigma) \\
&= log\prod_{i=1}^{n}p(x^{(i)}|y^{(i)};\mu_{0},\mu_{1},\Sigma)p(y^{(i)};\phi)
\end{split}
\end{equation}
$$

we find the maximum likelihood estimate of the parameters are：

$$\phi = \frac{1}{n}\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}$$

$$\mu_{0} = \frac{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}x^{(i)}  }{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}} $$

$$\mu_{1} = \frac{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}x^{(i)}  }{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}} $$

$$\Sigma=\frac{1}{n}\sum_{i=1}^{n}(x^{(i)} - \mu_{y^{(i)}})(x^{(i)} - \mu_{y^{(i)}})^{T}$$

as we wanted

## GDA and logistic regression

the GDA model has an interesting relationship to logistic regression.

if we view the quantity $p(y=1|x; \phi,\Sigma,\mu_{0},\mu_{1})$ as a function of $x$, we'll find that it can be expressed in the form:

$$
p(y=1|x; \phi,\Sigma,\mu_{0},\mu_{1}) = \frac{1}{1 + exp(-\theta^{T}x)}
$$

where $\theta$ is some appropriate function of $\phi,\Sigma,\mu_{0},\mu_{1}$.

proof:

$$
\begin{equation}
\begin{split}
p(y=1|x; \phi,\Sigma,\mu_{0},\mu_{1}) &= \frac{p(y=1, x)}{p(x)} \\
&=\frac{p(x|y=1)p(y=1)}{p(x|y=1)p(y=1) + p(x|y=1)p(y=1)} \\
&=\frac{\phi\cdot exp\left (-\frac{1}{2}(x-\mu_{1})^{T}{\Sigma}^{-1}(x-\mu_{1})\right )}{\phi\cdot exp\left (-\frac{1}{2}(x-\mu_{1})^{T}{\Sigma}^{-1}(x-\mu_{1})\right ) + (1 - \phi)\cdot exp\left (-\frac{1}{2}(x-\mu_{0})^{T}{\Sigma}^{-1}(x-\mu_{0})\right )} \\
&=\frac{1}{1 + \frac{1 - \phi}{\phi}exp\left(-\frac{1}{2}\left((x-\mu_{0})^{T}{\Sigma}^{-1}(x-\mu_{0}) - (x-\mu_{1})^{T}{\Sigma}^{-1}(x-\mu_{1})\right)\right)} \\
&=\frac{1}{1 + \frac{1 - \phi}{\phi}exp\left(-\frac{1}{2}\left((\mu_{1}^T -\mu_{0}^T)\Sigma^{-1}x + x^{T}\Sigma^{-1}(\mu_{1}-\mu_{0}) + (\mu_{0}^{T}\Sigma^{-1}\mu_{0} - \mu_{1}^{T}\Sigma^{-1}\mu_{1})\right)\right)}
\end{split}
\end{equation}
$$

because of $x^{T}a=a^{T}x$, we can express the above as:

$$\frac{1}{1 + exp(-\theta^{T}x)}$$

so the separating surface of GDA is $\theta^{T}x=0$.

as logistic regression, GDA can be interpreted by logistic model, but with different optimization strategy.

## Naive Bayes

in GDA，x is continous, when x is discrete, we can use naive bayes.

suppose $x=(x_{1}, x_{2},..., x_{d})$, for simplicity, we assume $x_{j}$ binary here, we of course have：

$$
\begin{equation}
\begin{split}
p(x|y) &= p(x_{1},x_{2},...,x_{d}|y) \\
&= p(x_{1}|y)p(x_{2}|y,x_{1})...p(x_{d}|y,x_{1},...,x_{d-1})
\end{split}
\end{equation}
$$

we will assume that $x_{j}$'s are conditionally independent given $y$. this assumption is called the naive bayes assumption, and the resulting algorithm is called the naive bayes classifier：

$$
\begin{equation}
\begin{split}
p(x|y) &= p(x_{1}|y)p(x_{2}|y,x_{1})...p(x_{d}|y,x_{1},...,x_{d-1}) \\
&= p(x_{1}|y)p(x_{2}|y)...p(x_{d}|y)
\end{split}
\end{equation}
$$

our model is parameterized by:

$$y\sim Bernoulli(\phi)$$
$$x_{j}|y=1 \sim Bernoulli(\phi_{j|y=1})$$
$$x_{j}|y=0 \sim Bernoulli(\phi_{j|y=0})$$

note:

$$k = \sum_{i=1}^{n}1\left\{y^{(i)}=1\right\}$$
$$s_{j} = \sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=1\right\}$$
$$t_{j} = \sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=0\right\}$$

we can write down the joint log likelihood of the data:

$$
\begin{equation}
\begin{split}
\mathcal{L}(\phi, \phi_{j|y=1}, \phi_{j|y=0}) &= log\prod_{i=1}^{n}p(x^{(i)},y^{(i)})\\
&=\sum_{i=1}^{n}log(p(x^{(i)}, y^{(i)})) \\
&=\sum_{i=1}^{n}log(p(y^{(i)})\prod_{j=1}^{d}p(x_{j}^{(i)}|y^{(i)})) \\
&=\sum_{y^{(i)}=1}(log(\phi) + \sum_{j=1}^{d}log(p(x_{j}^{(i)}|y=1))) + \sum_{y^{(i)}=0}(log(1 - \phi) + \sum_{j=1}^{d}log(p(x_{j}^{(i)}|y=0))) \\
&=k\ log(\phi) + (n-k)log(1 - \phi) + \sum_{j=1}^{d}(s_{j}\ log(\phi_{j|y=1}) + (k-s_{j})log(1 - \phi_{j|y=1}) + t_{j}\ log(\phi_{j|y=0}) + (n -k - t_{j})log(1 - \phi_{j|y=0})
\end{split}
\end{equation}
$$

maximizing this equal to maximize each part, we derive:

$$\phi=\frac{k}{n}$$
$$\phi_{j|y=1} = \frac{s_{j}}{k}$$
$$\phi_{j|y=0} = \frac{t_{j}}{n-k}$$

as expected

## Laplace Smoothing

there is problem with the naive bayes in its current form

if $x_{j}=1$ never occur in the training set, then $p(x_{j}=1|y=1)=0,p(x_{j}=1|y=0)=0$.

hence when predicting a sample $x$ with $x_{j}=1$, then:

$$
\begin{equation}
\begin{split}
p(y=1|x) &= \frac{\prod_{k=1}^{d}p(x_{k}|y=1)p(y=1)}{\prod_{k=1}^{d}p(x_{k}|y=1)p(y=1) + \prod_{k=1}^{d}p(x_{k}|y=0)p(y=0)} \\
&= \frac{0}{0}
\end{split}
\end{equation}
$$

does't know how to predict.

to fix this problem, instead of:

$$\phi_{j|y=1}=\frac{\sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=1\right\}}{\sum_{i=1}^{n}1\left\{y^{(i)}=1\right\}}$$

$$\phi_{j|y=0}=\frac{\sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=0\right\}}{\sum_{i=1}^{n}1\left\{y^{(i)}=0\right\}}$$

we add 1 to the numerator, add 2 to the denominator to:

1. avoid 0 in the parameter
2. $\phi_{j|y=1}$ and $\phi_{j|y=0}$ is still a probability distribution.

i.e:

$$\phi_{j|y=1}=\frac{1 + \sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=1\right\}}{2 + \sum_{i=1}^{n}1\left\{y^{(i)}=1\right\}}$$

$$\phi_{j|y=0}=\frac{1 + \sum_{i=1}^{n}1\left\{x_{j}^{(i)}=1 \wedge y^{(i)}=0\right\}}{2 + \sum_{i=1}^{n}1\left\{y^{(i)}=0\right\}}$$

## Text Classification

bernoulli event model:

first randomly determined whether a spammer or non-spammer

then runs through the dictionary deciding whether to include each word j.

multinomial event model:

first randomly determined whether a spammer or non-spammer

then each word in the email is generating from some same multinomial distribution independently.

using multinomial event model, if we have training set $\left \{ (x^{(1)},y^{(1)}),...,(x^{(n)},y^{(n)}) \right \}$ where $x^{(i)}=(x_{1}^{(i)},...,x_{d_{i}}^{(i)})\ $(here $d_{i}$ is the number of words in the i-th training example)

using maximum likelihood estimates of parameters like the above:

$$\phi = \frac{1}{n}\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}$$

$$\phi_{k|y=1}=\frac{\sum_{i=1}^{n}\sum_{j=1}^{d_{i}}1\left \{x_{j}^{(i)}=k\wedge y^{(i)}=1 \right \}}{\sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}d_{i}}$$

$$\phi_{k|y=0}=\frac{\sum_{i=1}^{n}\sum_{j=1}^{d_{i}}1\left \{x_{j}^{(i)}=k\wedge y^{(i)}=0 \right \}}{\sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}d_{i}}$$

add laplace smoothing with respect to multinomial event model:

$$\phi_{k|y=1}=\frac{1 + \sum_{i=1}^{n}\sum_{j=1}^{d_{i}}1\left \{x_{j}^{(i)}=k\wedge y^{(i)}=1 \right \}}{\left | V \right | + \sum_{i=1}^{n}1\left \{ y^{(i)}=1 \right \}d_{i}}$$

$$\phi_{k|y=0}=\frac{1 + \sum_{i=1}^{n}\sum_{j=1}^{d_{i}}1\left \{x_{j}^{(i)}=k\wedge y^{(i)}=0 \right \}}{\left | V \right | + \sum_{i=1}^{n}1\left \{ y^{(i)}=0 \right \}d_{i}}$$

$\left | V \right |$ is the size of the vocabulary

in short:

$$\phi_{k|y=1}=\frac{1 + number\ of\ words\ k\ occur\ in\ spammer}{\left | V \right | + number\ of\ words\ in\ spammer}$$

```python

```


# Nonlinear SVC

### Problem Formulation

The primal form for nonlinear SVC can be expressed as:

$$
\begin{align}
&\min_{\mathbf{w}, b, \mathbf{\xi}}  && \frac{1}{2}\mathbf{w}^{T}\mathbf{w} + C \sum_{i=1}^{m}\xi_{i}\\
&\text{subj. to}  && y_{i}\big(\mathbf{w}^{T}\phi(\mathbf{x}_{i}) + b\big) \ge 1 - \xi_{i}, \quad \xi_{i} \ge 0
\end{align}
$$

The Lagrangian function for this problem is:

$$
\mathcal{L}(\mathbf{w},b,\mathbf{\xi},\mathbf{\alpha}) = \frac{1}{2}\mathbf{w}^{T}\mathbf{w} + C \sum_{i=1}^{m}\xi_{i} + \sum_{i}^{m}\alpha_{i}\Big(1 - \xi_{i} - y_{i}\big(\mathbf{w}^{T}\phi(\mathbf{x}_{i}) + b\big) \Big)
$$

Stationarity KKT conditions:

$$
\begin{align}
&\nabla_{\mathbf{w}}\mathcal{L} = \mathbf{w} - \sum_{i=1}^{m}\alpha_{i}y_{i}\phi(\mathbf{x}_{i}) = 0\\
&\frac{\partial \mathcal{L}}{\partial b} = -\sum_{i=1}^{m}\alpha_{i}y_{i} = 0
\end{align}
$$

Substituting the stationarity conditions back into the Lagrangian eliminates $\mathbf{w}$ and $b$:

$$
\mathcal{L}(\mathbf{\alpha},\mathbf{\xi}) = \frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}\alpha_{i}\alpha_{j}y_{i}y_{j}\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j}) + \sum_{i=1}^{m}\alpha_{i} + \sum_{i=1}^{m}\xi_{i}(C - \alpha_{i})
$$

With the Lagrangian, the dual of the primal problem can be expressed as:

$$
\begin{align}
&\max_{\mathbf{\alpha}, \mathbf{\xi}} && -\frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}\alpha_{i}\alpha_{j}y_{i}y_{j}\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j}) + \sum_{i=1}^{m}\alpha_{i} + \sum_{i=1}^{m}\xi_{i}(C - \alpha_{i})\\
&\text{subj. to} && \mathbf{y}^{T}\mathbf{\alpha}=0, \quad \xi_{i} \ge 0, \quad \mathbf{\alpha} \ge 0
\end{align}
$$

Converting the problem to a minimization:

$$
\begin{align}
&\min_{\mathbf{\alpha}, \mathbf{\xi}} && \frac{1}{2}\sum_{i=1}^{m}\sum_{j=1}^{m}\alpha_{i}\alpha_{j}y_{i}y_{j}\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j}) - \sum_{i=1}^{m}\alpha_{i} - \sum_{i=1}^{m}\xi_{i}(C - \alpha_{i})\\
&\text{subj. to} && \mathbf{y}^{T}\mathbf{\alpha}=0, \quad \xi_{i} \ge 0, \quad \mathbf{\alpha} \ge 0
\end{align}
$$

Considering $\mathbf{\xi}$ as a Lagrange multiplier, this problem can be re-cast as:

$$
\begin{align}
&\min_{\mathbf{\alpha}, \mathbf{\xi}} && \frac{1}{2}\mathbf{\alpha}^{T}\mathbf{G}\mathbf{\alpha} - \mathbf{e}^{T}\mathbf{\alpha}\\
&\text{subj. to} && \mathbf{y}^{T}\mathbf{\alpha}=0, \quad 0 \le \alpha_{i} \le C
\end{align}
$$

where $\mathbf{G}_{ij} = y_{i}y_{j}\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j})$ is positive semidefinite.  This problem is a convex QP.  Note that the strength of the penalty, $C$, in the primal problem limits how large the dual variables, $\alpha_{i}$, can become.

The elements of $\mathbf{\alpha}$ which are greater than 0 can be used to determine the decision function to classify new instances of $x$.  First it is necessary to determine the bias term, $\hat{b}$, which can be obtained from the instances of $\mathbf{\alpha}$ where $0 < \alpha_{i} < C$.  These instances are dual variables of support vectors that lie on the margins, where $\xi_{i} = 0$.  Thus, $\hat{b}$ can be obtained from the active constraint $y_{i}\big(\hat{\mathbf{w}}^{T}\phi(\mathbf{x}_{i}) + \hat{b}\big)=1$.  Multiplying both sides by $y_{i}$ we have $\hat{\mathbf{w}}^{T}\phi(\mathbf{x}_{i}) + \hat{b}=y_{i}$.  We can now compute $\hat{b}$ by averaging over all the support vectors on the margin:

$$
\begin{gather}
\hat{b} = \frac{1}{n_{s}}\sum_{i=1}^{n_s}\Big(y_{i} - \hat{\mathbf{w}}^{T}\phi(\mathbf{x}_{i})\Big)\\
\hat{b} = \frac{1}{n_{s}}\sum_{i=1}^{n_s}\Big(y_{i} - \sum_{j=1}^{m}\alpha_{j}y_{j}\phi(\mathbf{x}_{i})^{T}\phi(\mathbf{x}_{j})\Big)
\end{gather}
$$

New instances, $\mathbf{x}^{\prime}$ can be classifed using the decision function $h(\mathbf{x}^{\prime})$:

$$
\begin{gather}
h(\mathbf{x}^{\prime}) = \hat{b} + \sum_{i=1}^{m}\alpha_{i}y_{i}\phi(\mathbf{x}^{\prime})^{T}\phi(\mathbf{x}_{i}),
\end{gather}
$$

where the summation occurs over all non-zero support vectors.

```python
import numpy as np
import numpy.linalg as LA
from sklearn import datasets
from sklearn.pipeline import Pipeline
from sklearn.preprocessing import StandardScaler
from sklearn.model_selection import train_test_split
from sklearn.svm import SVC
import pandas as pd
import cvxpy as cp
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
#Generate Data
X,y = datasets.make_moons(noise=0.2)

#split data into training and testing sets
X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=0)

#Scale the data
scalerX = StandardScaler().fit(X)
X_train_scaled = scalerX.transform(X_train)
X_test_scaled = scalerX.transform(X_test)
X_scaled = scalerX.transform(X)
```

```python
#SVM sklearn

#kernel
kernel_name = 'rbf'

C = 5
clf = SVC(kernel=kernel_name, degree=3, coef0=1, C=C)
clf.fit(X_train_scaled, y_train)

dual_coefs = clf.dual_coef_.flatten()
print("dual coefficients:")
print(dual_coefs)
print("\n-----------------\n")

support_vecs = clf.support_vectors_

intercept = clf.intercept_
print("Intercept:")
print(intercept)
print("\n-----------------\n")
```

    dual coefficients:
    [-1.8343627  -1.19186676 -5.         -5.         -5.         -5.
     -3.90822822 -0.75088879 -5.         -5.         -5.          5.
      1.6254994   2.96449742  5.          4.34676869  5.          5.
      5.          0.54678363  5.          3.20179734]

    -----------------

    Intercept:
    [-0.08985642]

    -----------------

```python
#Function Definitions

def setup_kernel(params):
    #polynomial kernel
    if params['kernel']=='poly':

        def _nonlin_kernel(x1,x2):
            _gamma = params['gamma']
            _degree = params['degree']
            _r = params['r']
            return (_gamma * np.dot(x1,x2) + _r)**_degree

        return _nonlin_kernel

    #gaussian RBF
    elif params['kernel']=='rbf':

        def _nonlin_kernel(x1,x2):
            _gamma = params['gamma']
            return np.exp(-_gamma * LA.norm(x1 - x2,2)**2)

        return _nonlin_kernel

def Gram_matrix(x,y,kernel):
    num_points,_ = x.shape
    G = np.zeros((num_points,num_points))
    for ii in range(0,num_points):
        for jj in range(ii,num_points):
            _g = y[ii]*y[jj]*kernel(x[ii],x[jj])
            G[ii,jj] = _g
            G[jj,ii] = _g
    return G

def compute_bhat(alpha,x,y,kernel,C):
    epsilon = 1e-10
    #find values of alpha on the margin (less than C and greater than 0)
    _alpha = alpha[alpha < (C - epsilon)]
    _x = x[alpha < (C - epsilon)]
    _y = y[alpha < (C - epsilon)]

    #sum over all support vectors on the margin
    ns = len(_alpha)
    bhat = 0
    for ii in range(0,ns):
        _bhat = 0
        #for vectors on the margin: y_i - theta^T * X_i
        for jj in range(0,len(alpha)):
            #theta^T * X_i
            _bhat += alpha[jj] * y[jj] * kernel(_x[ii],x[jj])
        bhat += _y[ii] - _bhat
    bhat = bhat/ns
    return bhat

def predict(x_new,x,y,alpha,bhat,kernel):
    y_pred = bhat
    for ii in range(len(alpha)):
        y_pred += alpha[ii] * y[ii] * kernel(x_new,x[ii])

    if y_pred > 0:
        return 1
    else:
        return 0
```

```python
#Solve from Scratch

#setup kernel and Gram matrix
gamma = 1/(X_train_scaled.shape[1] * X_train_scaled.var())
params = {'kernel':kernel_name,'gamma':gamma,'degree':3,'r':1}

kernel_func = setup_kernel(params)

y_ones = np.copy(y_train)
y_ones[y_train==0]=-1
G = Gram_matrix(X_train_scaled,y_ones,kernel_func)

#solve dual problem with CVX

n = len(X_train_scaled)

e = -np.ones(n)
lb = np.zeros(n)
ub = C*np.ones(n)

theta = cp.Variable(n)
prob = cp.Problem(cp.Minimize((1/2)*cp.quad_form(theta, G) + e.T @ theta),
                 [theta >= lb,
                  theta <= ub,
                  y_ones.T @ theta == 0])
prob.solve()
alpha_scratch = theta.value

#zero-out entries close to 0
epsilon = 1e-10
alpha_scratch[np.abs(alpha_scratch) < epsilon]=0

#get support vectors
X_support = X_train_scaled[alpha_scratch > 0,:]
y_support = y_ones[alpha_scratch > 0]

#remove alphas for inactive constraints (should be equal to 0)
alpha_support = alpha_scratch[alpha_scratch > 0]

print("scratch dual coefficients:")
print(alpha_support)
print("\n-----------------\n")

#compute intercept
bhat = compute_bhat(alpha_support,X_support,y_support,kernel_func,C)
print("scratch intercept:")
print(bhat)
print("\n-----------------\n")

#test = predict(np.array([2,2]),X_support,y_support,alpha_support,bhat,kernel_func)
```

    scratch dual coefficients:
    [5.         1.83564897 1.61377417 1.19156502 2.96311071 5.
     5.         5.         4.34602534 5.         5.         3.90781908
     0.74822945 5.         5.         5.         5.         5.
     0.55080362 5.         3.20954868 5.        ]

    -----------------

    scratch intercept:
    -0.089530709913457

    -----------------

```python
#sklearn - plot results

# create meshgrid for plotting
h = 0.02 #step size of mesh
x_min, x_max = X_scaled[:, 0].min() - 1, X_scaled[:, 0].max() + 1
y_min, y_max = X_scaled[:, 1].min() - 1, X_scaled[:, 1].max() + 1
xx, yy = np.meshgrid(np.arange(x_min, x_max, h),
                     np.arange(y_min, y_max, h))

#predict classes using sklearn kernel SVM classifier
Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

# contour plot
Z = Z.reshape(xx.shape)
plt.contourf(xx, yy, Z, cmap=plt.cm.coolwarm, alpha=0.8)

#testing data
X0 = X_test_scaled[y_test==0]
X1 = X_test_scaled[y_test==1]

# Plot the test data
plt.scatter(X0[:, 0], X0[:, 1], marker='v')
plt.scatter(X1[:, 0], X1[:, 1], marker='o')
plt.title('sklearn kernel SVM')
```

```python
#predict classes from scratch
Z_scratch = np.zeros(shape=xx.shape)
x_dim,y_dim = Z_scratch.shape

for ii in range(0,x_dim):
    for jj in range(0,y_dim):
        x_new = np.array([xx[ii,jj],yy[ii,jj]])
        Z_scratch[ii,jj] = predict(x_new,X_support,y_support,alpha_support,bhat,kernel_func)
```

```python
#scratch - plot results
plt.contourf(xx, yy, Z_scratch, cmap=plt.cm.coolwarm, alpha=0.8)
plt.scatter(X0[:, 0], X0[:, 1], marker='v')
plt.scatter(X1[:, 0], X1[:, 1], marker='o')
plt.title('scratch kernel SVM')
```


# Linear differential equations

MAT332, Fall 2020, Yury Kudryashov, Week 2, Lecture 1

```python
from sympy import *
init_printing()
```

## Dimension 1

Consider the equation $\dot x=ax$, where $a$ is a real number and $x$ is an unknown funnction $\mathbb R\to\mathbb R$. It is easy to see that $x(t)=e^{at}$ is a solution of this equation:
$$
\frac{d}{dt}e^{at}=e^{at}\cdot \frac{d(at)}{dt}=ae^{at}.
$$
Next, $x(t)=ce^{at}$ is a solution of this equation for any $c$. Are there any other solutions? The answer is NO: for any solution we have $x(t)=x(0)e^{at}$. Indeed, if $x$ is a solution, then
$$
\frac{d}{dt}\left(x(t)e^{-at}\right)=\dot x(t)e^{-at}+x(t)\cdot \frac{d}{dt}e^{-at}=ax(t)e^{-at}-ax(t)e^{-at}=0,
$$
hence $x(t)e^{-at}$ is a constant function, $x(t)e^{-at}=x(0)e^{-a\cdot 0}=x(0)$, thus $x(t)=x(0)e^{at}$.

We can also ask `Python` to verify that $x(t)=e^{at}$ is a solution.

```python
var('a t c')
x = c * exp(a * t)
simplify(x.diff(t) - a * x)
```

### Practice problems

1. Find all solutions of the equation $\dot x=2x$.
2. Find the solution of $\dot x=-x$ such that $x(1)=e$.

## Dimension 2

Now consider an equation of the form
$$
\dot x=Ax,
$$
where $A$ is a $(2\times 2)$-matrix and $x\colon\mathbb R\to\mathbb R^2$, $x(t)=\begin{pmatrix}x_1(t)\\x_2(t)\end{pmatrix}$ is an unknown vector-valued function.

Before discussing the general case, consider a few special cases.

### Diagonal matrix $A=\begin{pmatrix}\lambda&0\\0&\mu\end{pmatrix}$

In this case the system takes the form
$$
\begin{align}
\dot x_1&=\lambda x_1;\\
\dot x_2&=\mu x_2,
\end{align}
$$
and we can solve these two equations independently: $x_1(t)=x_1(0)e^{\lambda t}$, $x_2(t)=x_2(0)e^{\mu t}$. We can also write the answer as
$$
x(t)=\begin{pmatrix}e^{\lambda t}&0\\0&e^{\mu t}\end{pmatrix}x(0).
$$

Again, we can ask `Python` to double check our computation.

```python
var('λ μ t c0 c1')
c = Matrix([c0, c1])
A = Matrix([[λ, 0], [0, μ]])
M = Matrix([[exp(λ * t), 0], [0, exp(μ * t)]])
x = M * c
display(A) # display matrix `A`
display(M) # display the fundamental matrix of solutions
display(x) # display the solution `x`
display(x.diff(t) - A * x) # verify that `x` is a solution
display(x.subs(t, 0)) # verify that `x(0)=(c0, c1)`
```

### Jordan cell $A=\begin{pmatrix}\lambda&1\\0&\lambda\end{pmatrix}$

In this case the system takes the form
$$
\begin{align}
\dot x_1&=\lambda x_1+x_2;\\
\dot x_2&=\lambda x_2.
\end{align}
$$
While these equations are no longer independent, we can solve the second one, then substitute the solution in the first one. Then we get $x_2(t)=x_2(0)e^{\lambda t}$, $\dot x_1=\lambda x_1+x_2(0)e^{\lambda t}$. One can verify that $x_1(t)=x_2(0)te^{\lambda t}+x_1(0)e^{\lambda t}$ is a solution of this differential equation. As we will see later, this is the only solution of this equation with given $x_1(0)$. Again, we can write down the answer as
$$
x(t)=\begin{pmatrix}e^{\lambda t}&te^{\lambda t}\\0&e^{\lambda t}\end{pmatrix}x(0)
$$
and let `Python` verify the answer.

```python
A = Matrix([[λ, 1], [0, λ]])
M = Matrix([[exp(λ * t), t * exp(λ * t)], [0, exp(λ * t)]])
x = M * c
display(A) # display matrix `A`
display(M) # display the fundamental matrix of solutions
display(x) # display the solution `x`
display(x.diff(t) - A * x) # verify that `x` is a solution
display(simplify(x.diff(t) - A * x)) # verify that `x` is a solution
display(x.subs(t, 0)) # verify that `x(0)=(c0, c1)`
```

### Complex eigenvalues $A=\begin{pmatrix}a&-b\\b&a\end{pmatrix}$

In this case it is hard to guess the solution (at least without using complex exponentials, see the textbook) but it is easy to verify that
$$
\begin{align}
x_1(t)&=x_1(0)e^{at}\cos(bt)-x_2(0)e^{at}\sin(bt),\\
x_2(t)&=x_1(0)e^{at}\sin(bt)+x_2(0)e^{at}\cos(bt)
\end{align}
$$
is a solution.

```python
var('a b')
A = Matrix([[a, -b], [b, a]])
M = exp(a * t) * Matrix([[cos(b * t), -sin(b * t)], [sin(b * t), cos(b * t)]])
x = M * c
display(A) # display matrix `A`
display(M) # display the fundamental matrix of solutions
display(x) # display the solution `x`
display(x.diff(t) - A * x) # verify that `x` is a solution
display(simplify(x.diff(t) - A * x)) # verify that `x` is a solution
display(x.subs(t, 0)) # verify that `x(0)=(c0, c1)`
```

### Change of basis (a.k.a. conjugate matrix)

[Jordan normal form theorem](https://en.wikipedia.org/wiki/Jordan_normal_form) implies that any $(2\times 2)$-matrix is conjugate to one of the matrices we discussed above. Thus in order to solve $\dot x=Ax$ with any matrix $A$ it suffices to learn how to deal with each of these cases (done) and how to solve $\dot x=CAC^{-1}x$ provided that we know the solutions of $\dot y=Ay$. The answer is simple: for each solution $y(t)$ of $\dot y=Ay$, the function $x(t)=Cy(t)$ is a solution of $\dot x=CAC^{-1}x$.

Indeed,
$$
\frac{d}{dt}(Cy(t))=C\dot y(t)=CAy(t)=(CAC^{-1})Cy(t).
$$

## Exponential of a matrix

Given an $(n\times n)$-matrix $A$, one can define $\exp(A)=e^{A}$ as
$$e^{A}=\sum_{n=0}^{\infty}\frac{1}{n!}A^n=I+A+\frac{A^2}{2!}+\dots$$,
where $n!=1\times\dots\times n$ is the factorial of $n$. Then $x(t)=e^{At}x(0)$ is a solution of $\dot x=Ax$. Indeed,
$$\frac{d}{dt}(e^{At})=\frac{d}{dt}\left(I+tA+\frac{t^2}{2!}A^2+\frac{t^3}{3!}A^3+\dots+\frac{t^n}{n!}A^n+\dots\right)=0+A+tA^2+\frac{t^2}{2!}A^3+\dots+\frac{t^{n-1}}{(n-1)!}A^n+\dots=Ae^{At},$$
hence
$$\frac{d}{dt}(e^{At}x(0))=Ae^{At}x(0).$$

Note that computing $e^{At}$ is not a trivial task, so this formula gives us another way to think about the formulas for solutions we discussed above, not a simpler way to compute these solutions.


# Covid 19 Doubling Period Outside of China
Author: Scott Morgan

The following describes a Python program for calculating the Covid-19 doubling period using data from outside of China. Data for this project comes from the Johns Hopkins University Coronavirus Map (https://coronavirus.jhu.edu/map.html).
In the early stages of a pandemic, the number of infected increases exponentially and it is the exponential growth that we'll model. We'll create and exponential model containing two unknown parameters and use the exising number of infected over time to calculate the model parameters that minimize the square error between the model and the data.
## Math
### Model
$$\hat{y}=A2^{\frac{t}{\tau}}$$
$\tau$ is the doubling period and $\hat{y}$ is an estimate of the number of infected at time $t$, the number of days after January 19, 2020. $A$ is a constant representing the initial number of infected.
### Linearizing
$$\ln{\hat{y}}=\ln{A}+\frac{t}{\tau}\ln{2}$$
### System of equations from model
We can create a system of linear equations from the model output at each day.
$$\begin{matrix}\ln{\hat{y}_{1}}=\ln{A}+\frac{t_1}{\tau}\ln{2} \\ \vdots \\ \ln{\hat{y}_{n}}=\ln{A}+\frac{t_n}{\tau}\ln{2} \end{matrix}$$
### Matrix Representation
Now we can represent the system of linear equations in matrix form.
$$\begin{pmatrix}\hat{y}_1 \\ \vdots \\ \hat{y}_n \end{pmatrix}=\begin{bmatrix}t_1\ln{2} & 1 \\ \vdots & \vdots \\ t_n\ln{2} & 1\end{bmatrix}\begin{bmatrix}\frac{1}{\tau} \\ \ln{A} \\ \end{bmatrix}$$
Define additional matrices as
\begin{equation}
    \begin{aligned}
   \hat{Y} &= \begin{pmatrix}\hat{y}_1 \\ \vdots \\ \hat{y}_n \end{pmatrix} \\[5pt]
    H&=\begin{bmatrix}t_1\ln{2} & 1 \\ \vdots & \vdots \\ t_n\ln{2} & 1\end{bmatrix} \\[5pt]
  \Theta&=\begin{bmatrix}\frac{1}{\tau} \\ \ln{A} \end{bmatrix}=\begin{bmatrix}\theta_1 \\ \theta_2 \end{bmatrix} \\[5pt]
    \end{aligned}
\end{equation}
Then $$\hat{Y}=H\Theta$$
### Least Squares Solution
Let the vector of actual measurements of the infected for each day be
$$Y = \begin{pmatrix}y_1 \\ \vdots \\ y_n \end{pmatrix}$$
It is known that the parameters $\theta_1$ and $\theta_2$ associated with the least squares fit of the data to the model are given by
$$\Theta=[H^TH]^{-1}H^TY$$
So,
$$\theta_1 = \frac{1}{\tau}$$
and
$$\theta_2 = \ln{A}$$
Which gives the doubling period $\tau$ by
$$\tau=\frac{1}{\theta_1}$$
and A, the number of initial infected, as
$$A=e^{\theta_2}$$
## Further Reading
For more on the matematics of the Coronavirus, see Mathematics of the Corona outbreak – with professor Tom Britton (https://www.youtube.com/watch?v=gSqIwXl6IjQ).
## Code
The code for this project follows, along with a plot of the model and exising data.

```python
import numpy as np
import matplotlib.pyplot as plt
import math

from matplotlib import rcParams
rcParams.update({'figure.autolayout': True})

%matplotlib inline

# Source of data from outside of China: https://coronavirus.jhu.edu/map.html
Y = np.array([4, 6, 8 ,14, 25, 40, 57, 64, 87, 105, 118,
              153, 173, 183, 188, 212, 227, 265, 317, 343,
              361,457, 476, 523, 538, 595, 685, 780, 896,
              1000, 1100, 1200, 1400, 1700, 2000, 2400,
              2600, 3300, 4300, 5300, 6500, 8500, 10300,
              12700, 14900, 17500, 21200, 25200, 29100,
              32800, 37800, 44900, 47400, 63600, 75100,
              81700, 100500, 116100, 133800, 161600, 190900, 232200])

T = np.arange(1,Y.size + 1)
Tln2 = T*(np.log(2))
Ones = np.ones(Y.size)
H = np.concatenate([Tln2, Ones])
Ht = np.reshape(H, (2, Y.size))
H = np.transpose(Ht)
lnY = np.log(Y)
# m, c = np.linalg.lstsq(H, lnY, rcond=None)[0] # Alternate calculation
Hinv = np.linalg.pinv(H)
Theta = np.matmul(Hinv, lnY)
m = Theta[0]
c = Theta[1]
A = np.exp(c)
print("\nA")
print(f"{A:{4}.{3}}")
tau = 1/m
print("\ntau")
print(f"{tau:{4}.{3}}")
Ym = A*np.power(2, T/tau)
plt.plot(T, Y, label='actual')
plt.plot(T, Ym, label='model')
plt.title("Covid-19 Doubling Period Outside of China")
plt.xlabel("Time t (days after January 1\9, 2020)")
plt.ylabel("Number Infected I")
plt.annotate(f'Doubling period = {tau:{4}.{3}} days', xy=(0.25, 0.5), xycoords='figure fraction')
plt.annotate(f'Model: I = {A:{4}.{3}} x 2^(t/{tau:{4}.{3}})', xy=(0.25, 0.4), xycoords='figure fraction')
plt.legend()
plt.savefig('covid19.png')

time_to_million = (tau/math.log(2))*(math.log(1000000/A))

print(f"\nTime to 1,000,000 infected: {time_to_million:{4}.{3}} days after January 19")
```

```python

```

Covid-19 Doubling Period Outside of China.ipynb


In $d$ dimensions, with conventions \begin{align}\tilde\phi(k) &= \int d^dx\:e^{ik\cdot x}\phi(x),\\\phi(x) &= \frac{1}{(2\pi)^d}\int d^dk\:e^{-ik\cdot x}\tilde\phi(k),\\\left\langle\tilde\phi(k)\tilde\phi^*(k')\right\rangle&=(2\pi)^dP(k)\delta(k-k').\end{align} For white noise, $\langle\phi(x)\phi(x')\rangle = A\delta(x-x'),$ $$\left\langle\tilde\phi(k)\tilde\phi^*(k')\right\rangle=\int d^dx\: d^dx' e^{ik\cdot x -ik'\cdot x'}\langle\phi(x)\phi(x')\rangle=A\int d^dx\: e^{i(k-k')\cdot x}=A(2\pi)^d\delta(k'-k).$$ Thus $P(k)=A$, i.e. __is flat__. Now, the contribution to the pdf is $$\left\langle \phi(x)^2\right\rangle = \frac{1}{(2\pi)^{2d}}\int d^dk\:d^dk'\:e^{i(k+k')\cdot x}\left\langle \tilde\phi(k)\tilde\phi^*(-k')\right\rangle = \frac{1}{(2\pi)^d}\int d^dk\:P(k) = \frac{S_{d-1}}{(2\pi)^d} \int d|k|\: |k|^{d-1}P(|k|)=\frac{S_{d-1}}{(2\pi)^d}\int d\log|k|\: k^dP(|k|),$$ where $S_n$ is the surface area of the unit $n$-sphere. However this $k^d$ term is not part of the power spectrum. Maybe you're thinking of the scale-invariant power spectrum $P(k)=k^{-d}$?

If we used the convention $$\left\langle\tilde\phi(k)\tilde\phi^*(k')\right\rangle=\frac{(2\pi)^dP(k)\delta(k-k')}{k^d},$$ then white noise is $P(k)\propto k^d$ (__increasing__ with $k$) and the scale-invariant power spectrum is flat. Whichever convention we use, for white noise, the total power (summed over different $k$ with the same $|k|$) increases like $k^{d-1}$, and the average power (averaged over different $k$ with the same $|k|$) is flat.

```python
from numpy import *
from matplotlib import pyplot as plt
%matplotlib inline
```

```python
x = random.randn(256,256)
```

```python
plt.imshow(x)
plt.colorbar()
```

```python
f = fft.fft2(x)
```

```python
plt.imshow(fft.fftshift(abs(f)))
plt.colorbar()
```

```python

```

notebooks/white noise has a flat power spectrum.ipynb


```python
import sympy as sp
```

```python
sp.__version__
```

    '1.6.2'

### Use `dsolve` in `sympy` to solve differential equations

```python
from sympy.solvers.ode import dsolve
```

```python
t = sp.var('t')
f = sp.Function('f')
```

```python
f_ = sp.Derivative(f(t), t)
```

```python
f_
```

$\displaystyle \frac{d}{d t} f{\left(t \right)}$

```python
f__ = sp.Derivative(f(t),t,t)
```

```python
f__
```

$\displaystyle \frac{d^{2}}{d t^{2}} f{\left(t \right)}$

```python
ode = f__ - 4*f_ + 4
ode
```

$\displaystyle - 4 \frac{d}{d t} f{\left(t \right)} + \frac{d^{2}}{d t^{2}} f{\left(t \right)} + 4$

```python
dsolve(ode,f(t))
```

$\displaystyle f{\left(t \right)} = C_{1} + C_{2} e^{4 t} + t$

```python
dsolve(ode,f(t), ics={f(0):sp.E, f(4):sp.E})
```

$\displaystyle f{\left(t \right)} = t + \frac{4 e^{4 t}}{1 - e^{16}} - \frac{4}{1 - e^{16}} + e$


###Bayes Factor for Log Regression/

#####Is there really a relationship between failure and temperature?

An critism of our above analysis is that *assumed* that the relationship followed a logistic model, this we implictly assumed that the probabilities change over temperature. Let's look at the data again. (Top figure)

Could it be that infact this particular sequence of defects occured by chance?  After all, I can produce similar plots. (Bottom figure) This might explain the large overlap in temperatures.

```python
figsize(12.5, 6)
subplot(211)
plt.scatter(challenger_data[:, 0], challenger_data[:, 1], s=75,
            color="k", alpha=0.5)
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.title("(Real) Defects of the Space Shuttle O-Rings vs temperature")

subplot(212)
n = challenger_data.shape[0]
plt.scatter(challenger_data[:, 0], stats.bernoulli.rvs(0.6, size=n),
            s=75, color="k", alpha=0.5)
plt.yticks([0, 1])
plt.ylabel("Damage Incident?")
plt.xlabel("Outside temperature (Farhenhit)")
plt.title("(Artificial) Defects of the Space Shuttle O-Rings vs \
temperature")
```

#### Introducing the Bayes factor

The *Bayes factor* is a measure comparing two models. In our example, on the one hand we believe that temperature plays an important role in the probability of O-ring failures. On the other hand, we believe that there is no significant connection, and the pattern appears by coincidence. We can compare these model using the ratio of the *probabilities of observing the data, given the model*:

$$ \frac{ P( \text{observations} | \text{Model}_1 ) }{  P( \text{observations} | \text{Model}_2 ) }$$

Calculating this is not at all obvious. For simplicity, let's select a set of parameters from the posterior distribution (which is tantamount to selecting a particular model).

```python
alpha_hat = alpha_samples[0, 0]  # select the first alpha sample
beta_hat = beta_samples[0, 0]  # select the first beta sample

p_hat = logistic(temperature, beta_hat, alpha_hat)
print "estimates of probability at observed temperature, defects: "
print np.array(zip(p_hat, temperature, D))[:3, :]
print "..."
```

To calculate the numerator in the Bayes factor, we start by **assuming a model**, in our case assume `alpha_hat`, `beta_hat` are true, and calculate the probability of the defects we observe, equal to the product of:

$$ P(\; \text{Ber}(\; p(t_i \; | \; \hat{\beta}, \hat{\alpha} )\; ) = D_i \; ),\;\; i=1..N $$

For example, using the output above, the first computation, $i=0$, would look like

\begin{align}
& p = p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) = 0.667 \\\\
& d = D_0 = 0 \\\\
& \Rightarrow \; P( \; \text{Ber}(p) = d ) = ??
\end{align}

The probability of this ocurring is $(1-0.667) = 0.333\; $ (Recall the definition of a Bernoulli random variable $\text{Ber}(p)$: it is equal to $1$ with probability $p$ and equal to 0 with probability $1-p$). As each observation is independent, we multiply all the observations together. A clever way to do this for a specific $i$ is, recalling the $D_i$ can only be 1 or 0:

$$\left( p(t_1 \; | \; \hat{\beta}, \hat{\alpha} )\right)^{D_i} \left( 1 - p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) \right)^{(1-D_i)}$$

(it is possible that the quantity can overflow, so it is recommended to take the $\log$ of the above::

$$ D_i\log( p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) ) + (1-D_i)\log( 1- p(t_1 \; | \; \hat{\beta}, \hat{\alpha} ) ) $$

and sum the terms instead of multiplying. Be sure to use this transform for both models you are comparing.)

```python
print p_hat
_product = p_hat ** (D) * (1 - p_hat) ** (1 - D)
prob_given_model_1 = _product.prod()
print "probability of observations, model 1: %.10f" % prob_given_model_1
```

The second model we are comparing against is that $\beta=0$, i.e. there is no relationship between probabilites and temperature. We perform the same calculations as above. Notice that `beta=0` here in the below PyMC code:

```python
beta = 0
alpha = pm.Normal("alpha", 0, 0.001, value=0)

@pm.deterministic
def p(temp=temperature, alpha=alpha, beta=beta):
    return 1.0 / (1. + np.exp(beta * temperature + alpha))

observed = pm.Bernoulli("bernoulli_obs", p,
                        value=D, observed=True)

model = pm.Model({"observed": observed, "beta": beta, "alpha": alpha})

# mysterious code to be explained in Chapter 3
map_ = pm.MAP(model)
map_.fit()
mcmc = pm.MCMC(model)
mcmc.sample(260000, 220000, 3)
######

alpha_samples_model_2 = mcmc.trace("alpha")[:, None]
alpha_hat = alpha_samples_model_2[0]  # use the first 'model'
beta_hat = 0

p_hat = logistic(temperature, beta_hat, alpha_hat)
print "estimates of probability at observed temperature, defects: "
print np.array(zip(p_hat, temperature, D))[:3, :]
print
print "Notice the probability is constant for all temperatures?"
```

```python
# compute the probability of observations given the model_2

_product = p_hat ** (D) * (1 - p_hat) ** (1 - D)
prob_given_model_2 = _product.prod()
print "probability of observations, model 2: %.10f" % prob_given_model_2
```

```python
print "Bayes factor = %.3f" % (prob_given_model_1 / prob_given_model_2)
```

Is this good? Below is a chart that can be used to compare the computed Bayes factors to a degree of confidence in M1.

Bayes factor	Supporting M1

<1:1  	Negative (supports M2)

  1:1 to 3:1 	Barely worth mentioning

  3:1 to 10:1 	Substantial

  10:1 to 30:1 	Strong

  30:1 to 100:1	 Very strong

  > 100:1	 Decisive

We are not done yet. If you recall, we selected only one model, but recall we actually have a distribution of possible models (sequential pairs of ($\beta, \alpha$) from the posterior distributions). So to be correct we need to average over samples from the posterior:

```python
p = logistic(temperature[None, :], beta_samples, alpha_samples)
_product = p ** (D) * (1 - p) ** (1 - D)
prob_given_model_1 = _product.prod(axis=1).mean()
print "expected prob. of obs., given model 1: %.10f" % prob_given_model_1

p_model_2 = logistic(temperature[:, None],
                     np.zeros_like(temperature), alpha_samples_model_2)

_product = p_model_2 ** (D) * (1 - p_model_2) ** (1 - D)
prob_given_model_2 = _product.prod(axis=1).mean()
print "expected prob. of obs., given model 2: %.10f" % prob_given_model_2
print
print "Bayes factor: %.3f" % (prob_given_model_1 / prob_given_model_2)
```

It looks we have a pretty good model, and temperature *is* significant. Look at you, you're a rocket scientist now.

```python

```


```python
%matplotlib inline

import numpy as np
import scipy as sc
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sympy as sp

import itertools

sns.set();
```

```python
def extract(it):
    r"""
    Extract the values from a iterable of iterables.

    The function extracts the values from a iterable of iterables (eg. a list of tuples) to a list
    of coordinates. For example,

    [(1, 10), (2, 20), (3, 30), (4, 40)] -> [[1, 2, 3, 4], [10, 20, 30, 40]]

    If `it` is a list of M tuples each one with N elements, then `extract` returns
    a list of N lists each one with M elements.

    Parameters
    ----------
    it : iterable
        An iterable of iterables.

    Returns
    ------
    A list with the lists of first-elements, second-elements and so on.
    """

    return list(zip(*it))
```

# Runge-Kutta methods

The Runge-Kutta methods are in fact a family of methods designed to solve an ODE of the form:

$$y' = f(t, y)$$

with initial condition

$$y(t_{0}) = y_{0}$$

In other words, an initial condition problem.

## Two-stage Runge-Kutta methods

The so-called two-stage Runge-Kutta method has equations:

$$y_{k+1} = y_{k} + h\left(1 - \frac{1}{2\lambda}k_{1} + \frac{1}{2\lambda}k_{2}\right)$$

where

$$k_{1} = f(x_{k}, y_{k})$$

and

$$k_{2} = f(x_{k} + \lambda h, y_{k} + \lambda h k_{1})$$

The name "two-stage" comes from the fact that it is actually computed in two stages. First, we have to find $k_{1}$. Then we use that value to compute $k_{2}$.

Depending on the value given to $\lambda$, the method has different names. If $\lambda$ equals $1$, then it is called _improved Euler's method_. If $\lambda$ equals $2/3$, then it is called _Heun's method_.

### Improved Euler's method

By making $\lambda$ equals to $1$ in the general equation of the two-stage Runge-Kutta, we get the improved Euler's method. The equations are the following:

$$y_{k+1} = y_{k} + h\left(\frac{1}{2}k_{1} + \frac{1}{2}k_{2}\right)$$

with

$$k_{1} = f(x_{k}, y_{k})$$

and

$$k_{2} = f(x_{k} + h, y_{k} + hk_{1})$$

### Heun's method

As I said, the Heun's method comes from making $\lambda$ equals to $2/3$ in the general formula of the two-stage Runge-Kutta. The resulting equations are the following:

$$y_{k+1} = y_{k} + h\left(\frac{1}{4}k_{1} + \frac{3}{4}k_{2}\right)$$

where

$$k_{1} = f(x_{k}, y_{k})$$

and

$$k_{2} = f(x_{k} + \frac{2}{3}h, y_{k} + \frac{2}{3}hk_{1})$$

## Four-stage Runge-Kutta methods

### Classical Runge-Kutta method (RK4)

The method often referred as _classical Runge-Kutta method_ or simply _RK4_ is the Runge-Kutta method of 4 stages given by equations below:

$$y_{k+1} = y_{k} + \frac{h}{6}(k_{1} + 2k_{2} + 2k_{3} + k_{4}), n = 0, 1, 2, \dots$$

with

$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f\left(x_{k} + \frac{h}{2}, y_{k} + \frac{h}{2}k_{1}\right)$$
$$k_{3} = f\left(x_{k} + \frac{h}{2}, x_{k} + \frac{h}{2}k_{2}\right)$$
$$k_{4} = f(x_{k} + h, y_{k} + h k_{3})$$

### Variant of the classical Runge-Kutta method

There is a variation of the classical Runge-Kutta method (RK4) method. It is given by the following equations:

$$y_{k+1} = y_{k} + \frac{h}{8}(k_{1} + 3k_{2} + 3k_{3} + k_{4}), n = 0, 1, 2, \dots$$

with

$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f\left(x_{k} + \frac{h}{3}, y_{k} + \frac{h}{3}k_{1}\right)$$
$$k_{3} = f\left(x_{k} + \frac{2h}{3}, x_{k} + \frac{-h}{3}k_{1} + k_{2}\right)$$
$$k_{4} = f(x_{k} + h, y_{k} + hk_{1} - hk_{2} + hk_{3})$$

## General form and more theory

In general, the whole family of Runge-Kutta methods can be written as

$$y_{k+1} = y_{k} + h \sum_{i=1}^{s}b_{i}k_{i}$$

where

$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f(x_{k} + c_{2}h, y_{k} + h(a_{21}k_{1}))$$
$$k_{3} = f(x_{k} + c_{3}h, y_{k} + h(a_{31}k_{1} + a_{32}k_{2}))$$
$$\vdots$$
$$k_{s} = f(x_{k} + c_{s}h, y_{k} + h(a_{s1}k_{1} + a_{s2}k_{2} + \cdots + a_{ss-1}k_{s-1}))$$

A Runge-Kutta method is specified by

$s \doteq$ the number of stages, for $s \geq 1$,
$b_{i} \doteq$ the weights, for $i \in \{1, 2, \cdots, s\}$,
$c_{i} \doteq$ the loadings, for $i \in \{2, 2, \cdots, s\}$,
$a_{ij} \doteq$ the coefficients of $k_{j}$ in equation of $k_{i}$, for $1 \leq j < i \leq s$.

### Butcher tableau

A compact way of summarising these parameters is the Butcher tableau. Its general form is shown below.

|$0$      |         |         |         |           |        |
|:-------:|---------|---------|---------|-----------|--------|
|$c_{2}$  |$a_{21}$ |         |         |           |        |
|$c_{3}$  |$a_{31}$ |$a_{32}$ |         |           |        |
|$\vdots$ |$\vdots$ |$\vdots$ |$\ddots$ |           |        |
|$c_{s}$  |$a_{s1}$ |$a_{s2}$ |$\cdots$ |$a_{ss-1}$ |        |
|         |$b_{1}$  |$b_{2}$  |$\cdots$ |$b_{s-1}$  |$b_{s}$ |

### Method's order

A method is said to have order $p$ if the local truncation error is $O(h^{p+1})$. The minimum number of stages, $s$, required for a method to be of order $p$ until order 8 is given by the following table

|$p$     |1  |2  |3  |4  |5  |6  |7  |8  |
|:------:|---|---|---|---|---|---|---|---|
|min $s$ |1  |2  |3  |4  |6  |7  |9  |11 |

### Consistency

The method is said to be consistent if

$$\sum_{j=1}^{i-1}a_{ij} = c_{i}, \text{for } i = 2, \cdots, s$$

In other words, it is consistent if the sum of $a_{ij}$ on the $i$-th row is equal to the respective $c_{i}$.

### Runge-Kutta matrix

The $s$-by-$s$ Runge-Kutta matrix $M_{\mathit{RK}}$ is the lower triangular matrix defined by the coefficients $a_{ij}$ as shown below

$$M_{\mathit{RK}} = \begin{bmatrix}
       0 & 0 & \cdots & 0 & 0          \\[0.3em]
       a_{21} & 0 & \cdots & 0 & 0 \\[0.3em]
       a_{31} & a_{32} & \cdots & 0 & 0 \\[0.3em]
       \vdots & \vdots & \ddots & \vdots & \vdots \\[0.3em]
       a_{s1} & a_{s2} & \cdots & a_{ss-1} & 0
     \end{bmatrix}$$

## Butcher tableau examples

* Euler's method:

$$y_{n+1} = y_{n} + hf(x_{n}, y_{n})$$

|$0$ |    |
|:--:|----|
|    |$1$ |

* Improved Euler's method:

$$y_{k+1} = y_{k} + h\left(\frac{1}{2}k_{1} + \frac{1}{2}k_{2}\right)$$
$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f(x_{k} + h, y_{k} + hk_{1})$$

|$0$ |              |              |
|:--:|--------------|--------------|
|$1$ |$1$           |              |
|    |$\frac{1}{2}$ |$\frac{1}{2}$ |

* Heun's method:

$$y_{k+1} = y_{k} + h\left(\frac{1}{4}k_{1} + \frac{3}{4}k_{2}\right)$$
$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f(x_{k} + \frac{2}{3}h, y_{k} + \frac{2}{3}hk_{1})$$

|$0$           |              |              |
|:------------:|--------------|--------------|
|$\frac{2}{3}$ |$\frac{2}{3}$ |              |
|              |$\frac{1}{4}$ |$\frac{3}{4}$ |

* Classical Runge-Kutta (RK4):

$$y_{k+1} = y_{k} + \frac{h}{6}(k_{1} + 2k_{2} + 2k_{3} + k_{4}), n = 0, 1, 2, \dots$$
$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f\left(x_{k} + \frac{h}{2}, y_{k} + \frac{h}{2}k_{1}\right)$$
$$k_{3} = f\left(x_{k} + \frac{h}{2}, y_{k} + \frac{h}{2}k_{2}\right)$$
$$k_{4} = f(x_{k} + h, y_{k} + h k_{3})$$

|$0$           |              |              |              |              |
|:------------:|--------------|--------------|--------------|--------------|
|$\frac{1}{2}$ |$\frac{1}{2}$ |              |              |              |
|$\frac{1}{2}$ |$0$           |$\frac{1}{2}$ |              |              |
|$1$           |$0$           |$0$           |$1$           |              |
|              |$\frac{1}{6}$ |$\frac{1}{3}$ |$\frac{1}{3}$ |$\frac{1}{6}$ |

* Variant of the classical Runge-Kutta:

$$y_{k+1} = y_{k} + \frac{h}{8}(k_{1} + 3k_{2} + 3k_{3} + k_{4}), n = 0, 1, 2, \dots$$
$$k_{1} = f(x_{k}, y_{k})$$
$$k_{2} = f\left(x_{k} + \frac{h}{3}, y_{k} + \frac{h}{3}k_{1}\right)$$
$$k_{3} = f\left(x_{k} + \frac{2h}{3}, y_{k} - \frac{h}{3}k_{1} + k_{2}\right)$$
$$k_{4} = f(x_{k} + h, y_{k} + hk_{1} - hk_{2} + hk_{3})$$

|$0$           |               |              |              |              |
|:------------:|---------------|--------------|--------------|--------------|
|$\frac{1}{3}$ |$\frac{1}{3}$  |              |              |              |
|$\frac{2}{3}$ |$\frac{-1}{3}$ |$1$           |              |              |
|$1$           |$1$            |$-1$          |$1$           |              |
|              |$\frac{1}{8}$  |$\frac{3}{8}$ |$\frac{3}{8}$ |$\frac{1}{8}$ |

## Code

### Two-stage Runge-Kutta methods implementation

```python
def rk2(x_0, y_0, f, step=0.001, k_max=None, method='improved_euler'):
    r"""
    Two-stage Runge-Kutta method for solving first-order ODE.

    The function computes `k_max` iterations from the initial conditions `x_0` and `y_0` with
    steps of size `step`. It yields a total of `k_max` + 1 values. Being h_{k} the step at x_{k},
    the recorrent equation is:

    y_{k+1} = y_{k} + h_{k} * (1-(1/(2*lambda)) k_{1} + (1/(2*lambda)) k_{2})

    where

    k_{1} = f(x_{k}, y_{k})
    k_{2} = f(x_{k} + lambda * h_{k}, y_{k} + lambda * h_{k} * k_{1})

    When `method` is 'improved_euler', `lambda` is set to 1.
    When `method` is 'heun', `lambda` is set to 2/3.

    Parameters
    ----------
    x_0 : float
        The initial value for the independent variable.
    y_0 : array_like
        1-D array of initial values for the dependente variable evaluated at `x_0`.
    f : callable
        The function that represents the first derivative of y with respect to x.
        It must accept two arguments: the point x at which it will be evaluated and
        the value of y at this point.
    step : float, optional
        The step size between each iteration.
    k_max : number
        The maximum number of iterations.
    method : ["improved_euler", "heun"]
        The specific two-stage method to use.

    Yields
    ------
    x_k : float
        The point at which the function was evaluated in the last iteration.
    y_k : float
        The value of the function in the last iteration.

    Raises
    ------
    TypeError
        If the method argument is invalid or not supported.
    """
    if k_max is None: counter = itertools.count()
    else: counter = range(k_max)

    if method == 'improved_euler':
        b1, b2 = 1/2.0, 1/2.0
        c2 = 1
        a21 = 1
    elif method == 'heun':
        b1, b2 = 1/4.0, 3/4.0
        c2 = 2/3.0
        a21 = 2/3.0
    else:
        raise TypeError("The method {} is not valid or supported.".format(method))

    x_k = x_0
    y_k = y_0
    yield (x_k, y_k)
    for k in counter:
        k1 = f(x_k, y_k)
        k2 = f(x_k + c2 * step, y_k + a21 * step * k1)

        y_k = y_k + step * (b1 * k1 + b2 * k2)
        x_k = x_k + step
        yield (x_k, y_k)
```

### Four-stage Runge-Kutta methods implementation

```python
def rk4(x_0, y_0, f, step=0.001, k_max=None, method='classical'):
    r"""
    Four-stage Runge-Kutta methods for solving first-order ODE.

    The function computes `k_max` iterations from the initial conditions `x_0` and `y_0` with
    steps of size `step`. It yields a total of `k_max` + 1 values. We call h_{k} the step at x_{k}.

    Classical Runge-Kutta method (RK4):

    y_{k+1} = y_{k} + h/6 * (k_{1} + 2*k_{2} + 2*k_{3} + k_{4})

    where

    k_{1} = f(x_{k}, y_{k})
    k_{2} = f(x_{k} + h_{k}/2, y_{k} + h_{k}/2 * k_{1})
    k_{3} = f(x_{k} + h_{k}/2, y_{k} + h_{k}/2 * k_{2})
    k_{3} = f(x_{k} + h_{k}, y_{k} + h_{k} * k_{3})

    Variant of the classical Runge-Kutta method:

    y_{k+1} = y_{k} + h/8 * (k_{1} + 3*k_{2} + 3*k_{3} + k_{4})

    where

    k_{1} = f(x_{k}, y_{k})
    k_{2} = f(x_{k} + h_{k}/3, y_{k} + h_{k}/3 * k_{1})
    k_{3} = f(x_{k} + 2*h_{k}/3, y_{k} - h_{k}/3 * k_{1} + h_{k} * k_{2})
    k_{3} = f(x_{k} + h_{k}, y_{k} + h_{k} * k_{1} - h_{k} * k_{2} + h_{k} * k_{3})

    Parameters
    ----------
    x_0 : float
        The initial value for the independent variable.
    y_0 : array_like
        1-D array of initial values for the dependente variable evaluated at `x_0`.
    f : callable
        The function that represents the first derivative of y with respect to x.
        It must accept two arguments: the point x at which it will be evaluated and
        the value of y at this point.
    step : float, optional
        The step size between each iteration.
    k_max : number
        The maximum number of iterations.
    method : ["classical", "variant"]
        The specific four-stage method to use.

    Yields
    ------
    x_k : float
        The point at which the function was evaluated in the last iteration.
    y_k : float
        The value of the function in the last iteration.

    Raises
    ------
    TypeError
        If the method argument is invalid or not supported.
    """
    if k_max is None: counter = itertools.count()
    else: counter = range(k_max)

    if method == 'classical':
        b1, b2, b3, b4 = 1/6.0, 1/3.0, 1/3.0, 1/6.0
        c2, c3, c4 = 1/2.0, 1/2.0, 1
        a21, a31, a32, a41, a42, a43 = 1/2.0, 0, 1/2.0, 0, 0, 1
    elif method == 'variant':
        b1, b2, b3, b4 = 1/8.0, 3/8.0, 3/8.0, 1/8.0
        c2, c3, c4 = 1/3.0, 2/3.0, 1
        a21, a31, a32, a41, a42, a43 = 1/3.0, -1/3.0, 1, 1, -1, 1
    else:
        raise TypeError("The method {} is not valid or supported.".format(method))

    x_k = x_0
    y_k = y_0
    yield (x_k, y_k)
    for k in counter:
        k1 = f(x_k, y_k)
        k2 = f(x_k + c2 * step, y_k + a21 * step * k1)
        k3 = f(x_k + c3 * step, y_k + a31 * step * k1 + a32 * step * k2)
        k4 = f(x_k + c4 * step, y_k + a41 * step * k1 + a42 * step * k2 + a43 * step * k3)

        y_k = y_k + step * (b1 * k1 + b2 * k2 + b3 * k3 + b4 * k4)
        x_k = x_k + step
        yield (x_k, y_k)
```

## Examples

### Example 1: two-stage Runge-Kutta methods

Consider the following IVP:

$$y' = x^{2} + y^{2}$$

with

$$y(0) = 0$$

We will solve this IVP using the improved Euler's method and the Heun's method.

```python
def example1(x_k, y_k):
    return x_k**2 + y_k**2

results = rk2(x_0=0.0, y_0=0.0, f=example1, step=0.1, k_max=10, method='improved_euler')
x, y_improved_euler = extract(results)

results = rk2(x_0=0.0, y_0=0.0, f=example1, step=0.1, k_max=10, method='heun')
x, y_heun = extract(results)

df1 = pd.DataFrame({"x": x, "y_improved_euler": y_improved_euler, "y_heun": y_heun})

df1
```

      x
      y_heun
      y_improved_euler

      0
      0.0
      0.000000
      0.000000

      1
      0.1
      0.000333
      0.000500

      2
      0.2
      0.002667
      0.003000

      3
      0.3
      0.009002
      0.009503

      4
      0.4
      0.021355
      0.022025

      5
      0.5
      0.041776
      0.042621

      6
      0.6
      0.072411
      0.073442

      7
      0.7
      0.115577
      0.116817

      8
      0.8
      0.173913
      0.175396

      9
      0.9
      0.250586
      0.252374

      10
      1.0
      0.349640
      0.351830

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df1['x'], df1['y_improved_euler'], label='Improved Euler approximation with step 0.1', color='blue')
plt.plot(df1['x'], df1['y_heun'], label='Heun approximation with step 0.1', color='red')
plt.legend(loc='upper left', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Two-stage Runge-Kutta methods", xlabel="x", ylabel="y");
```

As we can see from the figure above, the solutions are nearly identical (we almost cannot distinguish between them).

### Example 2: four-stage Runge-Kutta methods

Consider the same IVP of example 1:

$$y' = x^{2} + y^{2}$$

with

$$y(0) = 0$$

We will solve this IVP using both the classical Runge-Kutta method (RK4) and its variant.

```python
def example2(x_k, y_k):
    return x_k**2 + y_k**2

results = rk4(x_0=0.0, y_0=0.0, f=example2, step=0.1, k_max=10, method='classical')
x, y_classical_rk4 = extract(results)

results = rk4(x_0=0.0, y_0=0.0, f=example2, step=0.1, k_max=10, method='variant')
x, y_variant_rk4 = extract(results)

df2 = pd.DataFrame({"x": x,
                    "y_classical_rk4": y_classical_rk4,
                    "y_variant_rk4": y_variant_rk4})

df2
```

      x
      y_classical_rk4
      y_variant_rk4

      0
      0.0
      0.000000
      0.000000

      1
      0.1
      0.000333
      0.000333

      2
      0.2
      0.002667
      0.002667

      3
      0.3
      0.009003
      0.009003

      4
      0.4
      0.021359
      0.021359

      5
      0.5
      0.041791
      0.041791

      6
      0.6
      0.072448
      0.072448

      7
      0.7
      0.115660
      0.115660

      8
      0.8
      0.174081
      0.174081

      9
      0.9
      0.250908
      0.250908

      10
      1.0
      0.350234
      0.350233

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df2['x'], df2['y_classical_rk4'],
         label='Classical Runge-Kutta approximation with step 0.1', color='blue')
plt.plot(df2['x'], df2['y_variant_rk4'],
         label='Variant of the classical Runge-Kutta approximation with step 0.1', color='red')
plt.legend(loc='upper left', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Four-stage Runge-Kutta methods", xlabel="x", ylabel="y");
```

As we can see from the figure above, the solutions are nearly identical (we almost cannot distinguish between them).

### Example 3

Consider the following IVP:

$$y' = tan(y) + 1$$

with

$$y(0) = 1$$

for $t \in [1, 1.1]$.

We will solve this IVP using Heun's method.

```python
def example3(x_k, y_k):
    return np.tan(y_k) + 1

results = rk2(x_0=1.0, y_0=1.0, f=example3, step=0.025, k_max=4, method='heun')
x, y_heun = extract(results)

df3 = pd.DataFrame({"x": x, "y_heun": y_heun})

df3
```

      x
      y_heun

      0
      1.000
      1.000000

      1
      1.025
      1.066869

      2
      1.050
      1.141332

      3
      1.075
      1.227418

      4
      1.100
      1.335079

### Example 4: Exercise 8 from section 8.6

The following example is actually the exercise 8 from section 8.6 of [Guidi].

Consider the following IVP:

$$y'' + (\exp(y') - 1) + y = -3\cos(t)$$

with

$$y(0) = y'(0) = 0$$

Find an approximation for the solution through the classical Runge-Kutta method for $t \in [0, 50]$ with $h = 0.01$. From the approximation obtained, find an estimative for the oscillation's amplitude for $t \in [43, 50]$ with 4 digits.

As this problem involves a second-order ODE, we must transform the variables so it becomes a system of first-order ODE:

$$u_{1} = y$$
$$u_{2} = y'$$

Since $u_{2}' = y''$, $y'' = f(t, y, y')$ becomes $u_{2}' = g(t, u_{1}, u_{2})$.

The resulting system of first-order ODE is

$$
  \begin{cases}
    u_{1}' = u_{2} \\
    u_{2}' = -3\cos(t) - \exp(u_{2}) + 1 - u_{1}
  \end{cases}
$$

```python
def example4(t_k, u_k):
    return np.array([u_k[1], -3*np.cos(t_k) - np.exp(u_k[1]) + 1 - u_k[0]])

results = rk4(x_0=0.0, y_0=np.array([0.0, 0.0]), f=example4, step=0.01, k_max=5000, method='classical')
t, ys = extract(results)
y_classical, dy_classical = extract(ys)

df4 = pd.DataFrame({"t": t, "y_classical": y_classical, "dy_classical": dy_classical})

t_interval = (df4.t > 43) & (df4.t < 50)
df4_interval = df4.loc[t_interval, ["t", "y_classical"]]

max_y = df4_interval.loc[:, "y_classical"].max()
min_y = df4_interval.loc[:, "y_classical"].min()

print("The amplitude of oscilattion for t in [43, 50] is {0:.3f}.".format(max_y - min_y))
```

    The amplitude of oscilattion for t in [43, 50] is 4.457.

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df4['t'], df4['y_classical'],
         label="Classical Runge-Kutta approximation with step 0.01", color='blue')
plt.plot(df4_interval['t'], df4_interval['y_classical'],
         label="Interval of interest, $t \in [43, 50]$", color='red')
plt.legend(loc='upper right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title=r"Solution of y'' + (exp(y') - 1) + y = -3cos(t)", xlabel="t", ylabel="y");
```

### Question 2

```python
def rk4_modified(x_0, y_0, f, step=0.001, k_max=None):
    if k_max is None: counter = itertools.count()
    else: counter = range(k_max)

    b1, b2, b3, b4, b5 = 1/6.0, 0.0, 0.0, 2/3.0, 1/6.0
    c2, c3, c4, c5 = 1/3.0, 1/3.0, 1/2.0, 1.0
    a21, a31, a32, a41, a42, a43, a51, a52, a53, a54 = 1/3.0, 1/6.0, 1/6.0, 1/8.0, 0.0, 3/8.0, 1/2.0, 0.0, -3/2.0, 2.0

    x_k = x_0
    y_k = y_0
    yield (x_k, y_k)
    for k in counter:
        k1 = f(x_k, y_k)
        k2 = f(x_k + c2 * step, y_k + a21 * step * k1)
        k3 = f(x_k + c3 * step, y_k + a31 * step * k1 + a32 * step * k2)
        k4 = f(x_k + c4 * step, y_k + a41 * step * k1 + a42 * step * k2 + a43 * step * k3)
        k5 = f(x_k + c5 * step, y_k + a51 * step * k1 + a52 * step * k2 + a53 * step * k3 + a54 * step * k4)

        y_k = y_k + step * (b1 * k1 + b2 * k2 + b3 * k3 + b4 * k4 + b5 * k5)
        x_k = x_k + step
        yield (x_k, y_k)
```

```python
def question2(t, u_k):
    return np.array([(4/5.0) * u_k[0] * u_k[1] - (1/4.0) * u_k[0], -(4/5.0) * u_k[0] * u_k[1]])

results = rk4_modified(x_0=0.0, y_0=np.array([0.005, 0.995]), f=question2, step=0.0125, k_max=800)
t, i_s = extract(results)
i, s = extract(i_s)
i, s = np.array(i), np.array(s)

df5 = pd.DataFrame({"t": t, "I": i, "S": s, "R": (1 - (i + s))})
df5 = df5[["t", "I", "S", "R"]]

print("Ratio I(10)/R(10) is {:.2f}.".format(df5["I"].iloc[-1]/df5["R"].iloc[-1]))
```

    Ratio I(10)/R(10) is 1.16.

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(df5['t'], df5['I'],
         label="$I(t)$: infected", color='blue')
plt.plot(df5['t'], df5['S'],
         label="$S(t)$: non-infected", color='green')
plt.plot(df5['t'], df5['R'],
         label="$R(t)$: recovered", color='red')
plt.legend(loc='upper right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title=r"Epidemic evolution: Kermack–McKendrick SIR model", xlabel="t", ylabel="y");
```

## References

* Guidi, L., Notas da disciplina Cálculo Numérico. Disponível em [Notas da disciplina Cálculo Numérico](http://www.mat.ufrgs.br/~guidi/grad/MAT01169/calculo_numerico.pdf).
* Heath, M. T., Scientific Computing: An Introductory Survey, 2nd Edition, McGraw Hill, 2002.
* Wikipedia, [Runge-Kutta methods](https://en.wikipedia.org/wiki/Runge%E2%80%93Kutta_methods).


# ATSC 507 Assignment VIII - Due March 3

```python
__author__ = 'Yingkai (Kyle) Sha'
__email__  = 'yingkai@eos.ubc.ca'
```

```python
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
from __future__ import division
% matplotlib inline
```

#Content
1. [**Known conditions**](#Known-conditions)
1. [**Numerical scheme**](#Numerical-scheme)
1. [**Results on the first timestep**](#Results-on-the-first-timestep)
1. [**Comparisons for 15 timesteps**](#Comparisons-for-15-timesteps)

# Known conditions

I repeat the problem to make it clear:

> Based on HW8 content, we are tring to solve $T(t)$ in:
$$
\frac{\partial T}{\partial t} = 3-2.25t-\frac{1.5T}{2-1.5t}
$$
And mean while we already know the analytical solution is:
$$
T = - 2.25t^2 + \left( 6 - 1.5T_0 \right)t + 2T_0 - 4
$$
with $t=m\Delta t$ and initial condition $T_0$.

> The problem requires to start with 0 dimension case $T_0 = 2^\circ C$ for 1 time step ($m=1$) in `Euler forward` and the family of `Runge-Kutta` methods.

```python
def analytical(t, T0):
    return (1.5*t+T0-2)*(2-1.5*t)
def slop_T(t, Tm):
    return 1.5*(2-1.5*t-(Tm/(2-1.5*t)))
```

# Numerical scheme

Here are 4 functions for testing:

  * Euler_forward: `Euler forward` temporal discretization
  * RK2: `2nd order Runge-Kutta`
  * RK3: `3rd order Runge-Kutta`
  * RK4: `4th order Runge-Kutta`

```python
# delta_t: step-length
# m: number of steps
# T0: Tref in Roland's equation
# Euler forward
def Euler_forward(delta_t, m, T0):
    time_steps = np.arange(0, (m+1)*delta_t, delta_t)
    T = np.empty(m); Tm = T0; delta_t = np.array(delta_t)
    for i in range(m):
        t = time_steps[i]
        T[i] = Tm + delta_t * slop_T(t, Tm)
        Tm = T[i]
    return T
# 2nd order Runge-Kutta
def RK2(delta_t, m, T0):
    time_steps = np.arange(0, (m+1)*delta_t, delta_t)
    T = np.empty(m); Tm = T0; delta_t = np.array(delta_t)
    for i in range(m):
        t = time_steps[i]
        T1 = Tm + 0.5*delta_t * slop_T(t, Tm)
        T[i] = Tm + delta_t * slop_T(t+0.5*delta_t, T1)
        Tm = T[i]
    return T
# 3rd order Runge-Kutta
def RK3(delta_t, m, T0):
    time_steps = np.arange(0, (m+1)*delta_t, delta_t)
    T = np.empty(m); Tm = T0; delta_t = np.array(delta_t)
    for i in range(m):
        t = time_steps[i]
        T1 = Tm + delta_t/3 * slop_T(t, Tm)
        T2 = Tm + delta_t/2 * slop_T(t+delta_t/3, T1)
        T[i] = Tm + delta_t * slop_T(t+0.5*delta_t, T2)
        Tm = T[i]
    return T
# 4th order Runge-Kutta
def RK4(delta_t, m, T0):
    time_steps = np.arange(0, (m+1)*delta_t, delta_t)
    T = np.empty(m); Tm = T0; delta_t = np.array(delta_t)
    for i in range(m):
        t = time_steps[i]
        C1 = slop_T(t, Tm)
        C2 = slop_T(t+0.5*delta_t, Tm+0.5*delta_t*C1)
        C3 = slop_T(t+0.5*delta_t, Tm+0.5*delta_t*C2)
        C4 = slop_T(t+delta_t, Tm+delta_t*C3)
        T[i] = Tm + delta_t * (C1+2*C2+2*C3+C4)/6
        Tm = T[i]
    return T
```

The analytical solution at the corresponds timestep for comparison.

```python
def analytical_sln(delta_t, m, T0):
    time_steps = np.arange(delta_t, (m+1)*delta_t, delta_t)
    return analytical(time_steps, T0)
```

# Results on the first timestep

Here I set $\Delta t = 1s$, $T_0 = 2^\circ C$ for the assignment.

```python
T0 = 2
delta_t = 1; m=1
```

```python
results = [Euler_forward(delta_t, m, T0), RK2(delta_t, m, T0), RK3(delta_t, m, T0), RK4(delta_t, m, T0)]
true_result = analytical_sln(delta_t, m, T0)
print('Euler_forward: {}\nRK2: {}\nRK3:{}\nRK4: {}\ntrue_result: {}'.format(results[0], results[1], results[2], results[3], true_result))
```

    Euler_forward: [ 3.5]
    RK2: [ 0.575]
    RK3:[ 1.625]
    RK4: [ 0.845]
    true_result: [ 0.75]

We see that `Euler forward` and `3rd order R.-K.`do not preforms well for its first step, meanwhile `2nd/4th order R.-K. methods` show a better guess for the first step.

# Comparisons for 15 timesteps

Equation in HW8 should be stable for all methods above. So here I integrate 15 timesteps to see which method approach to the analytical values faster.

`3rd R.-K. method` will not be applied, since for the second time step, $T^{**}$ is calculated by $t = 1+\frac13$, that makes term $\frac{1.5T}{2-1.5t}$ in `slop_T` encountered "divided by zero" problem.

```python
m = 15; t = range(1, m+1)
fig = plt.figure(figsize=(12, 6))
ax = fig.gca()
ax.plot(t, Euler_forward(delta_t, m, T0), linewidth=2, label='Euler forward')
ax.plot(t, RK2(delta_t, m, T0), linewidth=2, label='2nd order Runge-Kutta')
#ax.plot(t, RK3(delta_t, m, T0), linewidth=2, label='3rd order Runge-Kutta')
ax.plot(t, RK4(delta_t, m, T0), linewidth=2, label='4th order Runge-Kutta')
ax.plot(t, analytical_sln(delta_t, m, T0), color='k', linewidth=2, label='Analytical solution')
ax.legend(loc=3); ax.grid(); ax.set_xlim(1, m)
ax.set_xlabel('Timesteps ( m )', fontsize=12)
ax.set_title('Comparisons among Euler forward, and methods in R.-K. family', fontweight='bold', fontsize=14)
```

 We see that `Euler forward` do not perform well for their first step, but after the 2nd step, they do much better.