# Derivatives, Partial Derivatives and Gradient Descent

## Learning Objectives

By the end of this session you should be able to...

1. Compute the derivative of a function of one variable
2. Compute the partial derivatives of a function of many variables
3. Use the method of gradient descent to find the minimum of a function of one variable

## Motivation: Why are derivatives, partial derivatives and the method of gradient descent important? What do these things have to do with Data Science?
Derivatives and partial derivatives are used to implement the method of gradient descent.

Gradient Descent is
* A method for finding the minimum of a function
* Used to train logistic regression and other models in machine learning
* The key to training deep learning models, i.e. neural networks.

## 1. Introduction to Derivatives

The process of finding a derivative is called **Differentiation**, which is a technique used to calculate the *slope* of the *tangent line* of a function $y = f(x)$ at any point $x$. The derivative tells us how rapidly, and in what direction $y$ changes with respect to $x$

### 1A. Activity - Derivative Tutorial:

1. Go through this [Derivative tutorial from Math Is Fun](https://www.mathsisfun.com/calculus/derivatives-introduction.html) (15 min)
1. When you're done, talk with a partner about topics you still have questions on. See if you can answer each other's questions. (5 min)
1. We'll then go over questions on the tutorial as a class (10 min)

### 1B. Review Diagram

Review the below diagram as a class, and compare with what you just learned in the above `Derivative Tutorial`.

## 2. The Derivative Formula

$f^\prime(x) \equiv \frac{d}{dx}f(x) \equiv \frac{\Delta y}{\Delta x}  = \frac{f(x + \Delta x) - f(x)}{\Delta x}$

The symbol $\equiv$ is read as "is defined as".

The above formula is meant to be evaluated in the limit as $\Delta x$ approaches 0.

So we should choose a very small number for $\Delta x$, say 1.e-10

### 2A. Activity: Write a Python function to compute the gradient of any one-dimensional function

* NOTE: foe one dimensional function, the *gradient* is the same thing as the *derivative*

* Use the derivative formula, above

* Use your function to evaluate the gradient of $f(x)=x^2$at $x = 3$ and $x = -2$

```python
# define function we want to compute the gradient of
def f(x):
    return x**2

# function that computes the gradient of a one dimensional function, evaluated at a given position
def gradient(func,x):
    '''Inputs:
            func, the function to take the derivative of
            x, the position (value of x) where the derivative is to be evaluated
        Output:
            the gradient of func(x) evaluated at position x

    '''
    delta_x = 1e-6
    return (f(x + delta_x) - f(x)) / delta_x

x = 3
print(f'Gradient at x = {x} is {gradient(func=f,x=x):.3}')
x = -2
print(f'Gradient at x = {x} is {gradient(func=f,x=x):.3}')
```

    Gradient at x = 3 is 6.0
    Gradient at x = -2 is -4.0

Note that these values match the derivative that we compute from the calculus formula $\frac{d}{dx}x^n = nx^{n-1}$,
which is $\frac{d}{dx}x^2 = 2x^{2-1} = 2x^1 = 2x$

$2*3 = 6$

$2 * -2 = -4$

### 2B. Table of Derivatives of Common Functions

Computing derivatives of more complicated functions than $x^n$ is a source of much toil and busy work (not necessarily a bad thing) in introductory calculus classes. But now that you understand what a derivative is, it's not cheating to take a shortcut! The second page of this PDF lists formulas for derivatives of common functions. The first page summarizes the rules for computing derivatives. Utilize as a resource going forward!

- https://www.qc.edu.hk/math/Resource/AL/Derivative%20Table.pdf

## 3. Partial Derivatives of Functions of Two or More Variables

Now we know how to calculate a derivative for a function of one variable. But what if we have a function of $N$ variables where $N>1$?

We calculate the *partial derivatives* of the function.
Calculating the partial derivatives for a function of $N$ variables consists of calculating $N$ ordinary derivatives: one for each variable, where the other variables are held constant.

### 3A. Activity - Partial Derivative Video

Let's watch this video introduction to Partial Derivatives from **Khan Academy**: https://youtu.be/AXqhWeUEtQU

**Note:** Here are some derivative shortcuts that will help in the video:

$\frac{d}{dx}x^2 = 2x$

$\frac{d}{x}sin(x) = cos(x)$

$\frac{d}{dx}x = 1$

### 3B. The Symbolic Python library `sympy` computes derivatives and partial derivatives of functions
You can install this library from the command line via `conda install sympy` in an Anaconda shell.

```python
from sympy import symbols, diff
# initialize x and y to be symbols to use in a function
x, y = symbols('x y', real=True)
f = (x**2)/y
# Find the partial derivatives of x and y
partial_fx = diff(f, x, evaluate=True) # partial derivative of f(x,y) with respect to x
partial_fy = diff(f, y, evaluate=True) # partial derivative of f(x,y) with respect to y
print(f'The partial derivative of f(x,y) w.r.t. (with respect to) x is {partial_fx}')
print(f'The partial derivative of f(x,y) w.r.t. x is {partial_fy}')
print(f'The partial derivative of f(x,y) w.r.t. x evaluated at the point (2,1) is {partial_fx.evalf(subs={x: 2, y: 1}):.2}')
print(f'The partial derivative of f(x,y) w.r.t. y evaluated at the point (2,1) is {partial_fy.evalf(subs={x: 2, y: 1}):.2}')
```

    The partial derivative of f(x,y) w.r.t. (with respect to) x is 2*x/y
    The partial derivative of f(x,y) w.r.t. x is -x**2/y**2
    The partial derivative of f(x,y) w.r.t. x evaluated at the point (2,1) is 4.0
    The partial derivative of f(x,y) w.r.t. y evaluated at the point (2,1) is -4.0

## 4. Calculating a Function's Partial Derivatives and Gradient

### 4A. Example: calculate the partial derivatives for a function of two variables

```python
def f(x, y):
    return x**2/y

eps = 1e-6
x = 2
y = 1
print((f(x + eps, y) - f(x, y)) / eps)
print((f(x, y + eps) - f(x, y)) / eps)
```

    4.0000010006480125
    -3.9999959997594203

Looks about right! This works rather well, but it is just an approximation. Also, you need to call `f()` at least once per parameter (not twice, since we could compute `f(x, y)` just once). This makes this approach difficult to control for large systems (for example neural networks).

### 4B. Activity: compute the gradient of a function of two variables
Write a Python function that computes the *gradient* of any *two-dimensional* function $f(x,y)$ by **completing the code in the block below**

* The *gradient* of a function is the vector composed of all the function's *partial derivatives*

* The gradient operation is denoted by the *nabla* symbol $\nabla$
* The *gradient* of a two-dimensional function is thus
    $\nabla f(x,y) = \left(\frac {\partial{f(x,y)}}{\partial{x}}, \frac{\partial{f(x,y)}}{\partial{y}}\right)$
* Your `gradient()` function should calculate partial derivatives using the derivative formula from section **2**

* Use your `gradient()` function to evaluate the gradient of the function $f(x, y) = \frac{x^2}{y}$ at the point $(2,1)$

```python
# define the 2-dimensional function for which you want to compute the gradient
def f_xy(x,y):
    # your code here
    return

# function to compute gradient of any two-dimensional function
def gradient(func, x, y):
    '''
    inputs:
        func: a continuous function of two variables
        x, y: the values of the two variables at the point
            where the gradient is to be computed
    outputs:
        partial_x, partial_y: the partial derivatives of the input function,
            evaluated at specified point (x,y)

    '''
    delta = 1.e-10
    partial_x = None # your code here
    partial_y = None # your code here

    return partial_x, partial_y

# output
print(f'The gradient of f(x,y) = x**2/y at the point (2,1) is {gradient(func=f_xy, x=2, y=1)}')
```

    The gradient of f(x,y) = x**2/y at the point (2,1) is (None, None)

## 5. The Heart of the Matter: the Method of Gradient Descent

In many machine learning and deep learning applications, we need to find the minimum of a "cost function" (sometimes also called an "error function" or a "loss function").
Whatever we want to call it, the cost function measures the error of a model on some training data when we compare the predicted *targets* against the true *targets*.

The *targets* are what the model is trying to predict. Examples are **numbers** -- like housing prices, or **class labels** -- like "dog", "cat" or "mouse")

Why do we want to find the minimum for a cost function? Given that a cost function measures how wrong our model is, we want to minimize that error!

In Machine Learning cost functions measure how badly our models are performing. We want to adjust the parameters of the model (also known as **weights**) to minimize our cost function, therefore minimizing error of the model!

We find find the optimal weights that minimize the cost function by using the **Method of Gradient Descent**. The Method of Gradient Descent repeatedly makes small adjustments to the weights that make the cost function smaller, until the cost function reaches a minimum. We'll learn more about models in future lessons, but for now, let's review the Gradient Descent process with the below images, given a cost function $J(w_0,w_1)$ that is a function of weights $w_0$ and $w_1$:

Look at that bottom right image above. Looks like we're using partial derivatives to determine the optimal values of the weights $w_0$ and $w_1$ that minimize the cost function $J(w_0,w_1)$. And we know exactly how to do that!

### 5A. Finding the minimum of a function via Gradient Descent

Assume we want to minimize the function $J$ which has two weights $w_0$ and $w_1$

We have two options to find the minimum of $J(w_0, w_1)$:

1. Take partial derivatives of $J(w_0, w_1)$ with relation to $w_0$ and $w_1$:

$\frac{\partial J(w_0, w_1)}{\partial w_0}$

$\frac{\partial J(w_0, w_1)}{\partial w_1}$

And find the appropriate weights such that the partial derivatives equal 0:

$\frac{\partial J(w_0, w_1)}{\partial w_0} = 0$

$\frac{\partial J(w_0, w_1)}{\partial w_1} = 0$

In this approach we would solve the resulting system of linear or non-linear equations.

2. Use the Gradient Descent algorithm:

First we need to define three things:

- Arbitrary random initial values for $w_0$ and $w_1$.
- A step-size alpha ($\alpha$) -- also called the *learning rate*, a small number like $1.e-5$
- A *threshold* for the weight updates

Next we search for the optimal values of the weights $w_0$ and $w_1$ by using a loop to repeatedly update the weights until we minimize the cost function. We terminate the loop when the weight updates become negligibly small, i.e. below the threshold that we set. For example, if we adopt a threshold of 0.0001 and the weight updates -- the changes in the weight parameters from one iteration to the next -- are both less than 0.0001, then we stop the loop and declare victory, i.e. that we have found optimal values of the weights that (approximately) minimize the loss function!

Let's review some pseudocode for how to implement this algorithm:

```
# initialization
initialize the following:
    - starting weight values -- an initial guess, could be random
    - the learning rate (alpha), a small positive number (we'll choose 1.e-5)
    - the current weight update -- initialize to 1
    - the threshold -- set this to 1.e-4

# weight update loop
while the weight update is greater than the threshold:
    adjust the weights by adding the weight updates based on the algorithm

# return the final values of the weights
```

How do we compute the `weight updates based on the algorithm`?
The updates are the second term on the right hand side of the equations below: they are proportional to the partial derivatives and in the opposite direction.

$w_0 = w_0 - \alpha \frac{\partial J(w_0, w_1)}{\partial w_0}$

$w_1 = w_1 - \alpha \frac{\partial J(w_0, w_1)}{\partial w_1}$

### 5B. Gradient descent in one dimension

#### Let's define an interesting function of one variable to minimize:

$f(x) = \frac{x^4}{50} - 2x^2 + x + 1$
#### We can compute the derivative of this function using the formula from calculus

$\frac{d}{dx}(x^n) = nx^{n-1}$

So that

$f^{'}(x) = \frac{4}{50}x^3 - 4x + 1$

```python
import matplotlib.pyplot as plt
import numpy as np
np.random.seed(42)
eps = 1.e-10

# define an interesting function to minimize
x = np.linspace(-10,10, 100)
fx =lambda x: (1/50)*x**4 - 2*x**2 + x + 1

# compute derivative of x by applying the calculus formula d/dx(x**n) = n*x**(n-1)
df_dx = lambda x: (4/50)*x**3 - 4*x + 1

# plot the function, and its derivative
plt.figure(figsize=(10,8))
plt.plot(x,fx(x),label = 'function f(x)')
plt.plot(x,df_dx(x),'r--',label = 'derivative df/dx')
plt.xlabel('$x$',FontSize=18)
plt.ylabel('$f(x)$, $f\'(x)$',FontSize=18)
plt.title('The function $f(x)$ and its derivative $f\'(x)$', FontSize=18)

plt.legend(prop={'size': 18})
plt.grid()
```

#### Recall our `gradient()` function from section 2, above

```python
def gradient(func,x):
    delta_x = 1.e-10
    return (func(x + delta_x) - func(x))/delta_x
```

```python
# test the `gradient` code by choosing a few x values
#     and see if the answers make sense
x = -10
print(f'The gradient of f(x) at x = {x} is {gradient(func=fx,x=-10):.3}')
```

    The gradient of f(x) at x = -10 is -39.0

####  Write a `gradient descent` function for a function of one variable
Follow the pseudo-code in section **5A**

```python
def gradient_descent(func,x_init):

    '''
    Inputs:
        func: a function of one variable
        x_init: the starting value of the variable
    Outputs:
        x: the value of the variable at which func reached a minimum
        n_iter: the number of iterations of the weight update loop
            before it was terminated
        x_values: the list of updated weights
        '''

    # Initialize
    alpha = 1.e-4
    threshold = 1.e-4
    delta_x = 1.
    x = x_init
    n_iter = 0
    x_values = []

    # run the weight update loop until it terminates
    while np.abs(delta_x) > threshold: #and n_iter < max_iter:

        # compute the derivative of the function
        df_dx = gradient(func,x)
        # compute the weight update
        delta_x = -alpha*df_dx
        # apply the weight update
        x = x + delta_x
        # append to the running list of updated weights
        x_values.append(x)
        # increment the iteration count
        n_iter+=1

    return x, n_iter, x_values
```

#### Minimize the function $f(x) = \frac{x^4}{50} - 2x^2 + x + 1$
Choose an initial value for $x$

Then run your `gradient_descent` code to to find the value of $x$ that minimizes the function.

```python
# choose a starting point -- could be random
# np.random.seed(42)
# x_init = np.random.uniform(-10,10,1)
x_init = 2.5
x_star, n_iter, x_values = gradient_descent(func=fx,x_init=x_init)
print(f'Starting at the point x = {x_init}, \n    \
we found that the function reaches a minimum at x_star = \
{x_star:.3} \n    after {n_iter} iterations')
```

    Starting at the point x = 2.5,
        we found that the function reaches a minimum at x_star = 6.81
        after 6853 iterations

#### Check that the derivative is reasonably close to zero when evaluated at the `x_star` found by your `gradient_descent` function
Remember that the derivative ranges between $[-40,+40]$ according to the above graph.

```python
# derivative, from calculus formula
print(f'Derivative from the calculus formula is {df_dx(x_star)}')

# derivative, from definition
print(f'Derivative from the definition is  {gradient(func=fx,x=x_star)}')
```

    Derivative from the calculus formula is -0.9987282580894323
    Derivative from the definition is  -0.9987388693843968

#### Visualize how gradient descent finds the minimum of the function

```python
x_values = np.array(x_values)
x = np.linspace(-10,10, 100)

plt.figure(figsize=(12,8))
plt.plot(x,fx(x),'b-',label = 'function f(x)')
plt.plot(np.array(x_values),fx(x_values),'m.',markersize = 10, label='gradient descent updates')
plt.plot(x_init,fx(x_init),'k.',markersize = 20, label = '$x_{init}$, starting point')
plt.plot(x_star,fx(x_star),'k*',markersize = 20, label = '$x_{star}$, minimum of the function',)
plt.xlabel('x',FontSize = 18)
plt.ylabel('f(x)',FontSize = 18)
plt.title('Minimizing a function via gradient descent',FontSize = 18)
plt.grid()
plt.legend(loc='best',prop={'size': 18});
```

### 5C. Activity:
In the above section, our `gradient_descent()` function required almost 7000 iterations (steps) to find the minimum of the input function.
See if you can tune the value(s) of the *hyperparameters* $alpha$ and $threshhold$ in the `gradient_descent()` code to get reasonably close to the same answer(s) in fewer than 700 steps.

## Resources

- [Derivative tutorial from Math Is Fun](https://www.mathsisfun.com/calculus/derivatives-introduction.html)
- [Derivative Table](https://www.qc.edu.hk/math/Resource/AL/Derivative%20Table.pdf)
- [Khan Academy - Partial Derivatives video](https://www.youtube.com/watch?v=AXqhWeUEtQU&feature=youtu.be)
- [Towards Data Science - Machine Learning Fundamentals: cost functions and gradient Descent](https://towardsdatascience.com/machine-learning-fundamentals-via-linear-regression-41a5d11f5220)

```python

```


```python
%matplotlib inline

import os
import glob
from pathlib import Path
import numpy as np
import pandas as pd
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns

sns.set_context('notebook', font_scale=1.5)
```

Applications of Linear Alebra: PCA
====

We will explore 3 applications of linear algebra in data analysis - change of basis (for dimension reduction), projections (for solving linear systems) and the quadratic form (for optimization). The first application is the change of basis to the eigenvector basis that underlies Principal Components Analysis s(PCA).

We will review the following in class:

- The standard basis
- Orthonormal basis and orthgonal matrices
- Change of basis
- Similar matrices
- Eigendecomposition
- Sample covariance
- Covariance as a linear transform
- PCA and dimension reduction
- PCA and "explained variance"
- SVD

Variance and covariance
----

Remember the formula for covariance

$$
\text{Cov}(X, Y) = \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1}
$$

where $\text{Cov}(X, X)$ is the sample variance of $X$.

```python
def cov(x, y):
    """Returns covariance of vectors x and y)."""
    xbar = x.mean()
    ybar = y.mean()
    return np.sum((x - xbar)*(y - ybar))/(len(x) - 1)
```

```python
X = np.random.random(10)
Y = np.random.random(10)
```

```python
np.array([[cov(X, X), cov(X, Y)], [cov(Y, X), cov(Y,Y)]])
```

    array([[ 0.0495287 , -0.00474943],
           [-0.00474943,  0.10057312]])

```python
# This can of course be calculated using numpy's built in cov() function
np.cov(X, Y)
```

    array([[ 0.0495287 , -0.00474943],
           [-0.00474943,  0.10057312]])

```python
# Extension to more variables is done in a pair-wise way
Z = np.random.random(10)
np.cov([X, Y, Z])
```

    array([[ 0.0495287 , -0.00474943, -0.00618675],
           [-0.00474943,  0.10057312, -0.01860316],
           [-0.00618675, -0.01860316,  0.05552326]])

Eigendecomposition of the covariance matrix
----

```python
mu = [0,0]
sigma = [[0.6,0.2],[0.2,0.2]]
n = 1000
x = np.random.multivariate_normal(mu, sigma, n).T
```

```python
A = np.cov(x)
```

```python
m = np.array([[1,2,3],[6,5,4]])
ms = m - m.mean(1).reshape(2,1)
np.dot(ms, ms.T)/2
```

    array([[ 1., -1.],
           [-1.,  1.]])

```python
e, v = np.linalg.eig(A)
```

```python
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e, v.T):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3])
plt.title('Eigenvectors of covariance matrix scaled by eigenvalue.');
```

Covariance matrix as a linear transformation
----

The covariance matrix is a linear transformation that maps $\mathbb{R}^n$ in the direction of its eigenvectors with scaling factor given by the eigenvalues. Here we see it applied to a collection of random vectors in the box bounded by [-1, 1].

### We will assume we have a covariance matrix

```python
covx = np.array([[1,0.6],[0.6,1]])
```

### Create random vectors in a box

```python
u = np.random.uniform(-1, 1, (100, 2)).T
```

### Apply covariance matrix as linear transformation

```python
y = covx @ u
```

```python
e1, v1 = np.linalg.eig(covx)
```

### The linear transform maps the random vectors as described.

```python
plt.scatter(u[0], u[1], c='blue')
plt.scatter(y[0], y[1], c='orange')
for e_, v_ in zip(e1, v1.T):
    plt.plot([0, e_*v_[0]], [0, e_*v_[1]], 'r-', lw=2)
plt.xticks([])
plt.yticks([])
pass
```

PCA
----

Principal Components Analysis (PCA) basically means to find and rank all the eigenvalues and eigenvectors of a covariance matrix. This is useful because high-dimensional data (with $p$ features) may have nearly all their variation in a small number of dimensions $k$, i.e. in the subspace spanned by the eigenvectors of the covariance matrix that have the $k$ largest eigenvalues. If we project the original data into this subspace, we can have a dimension reduction (from $p$ to $k$) with hopefully little loss of information.

Numerically, PCA is typically done using SVD on the data matrix rather than eigendecomposition on the covariance matrix. The next section explains why this works.

### Data matrices that have zero mean for all feature vectors

\begin{align}
\text{Cov}(X, Y) &= \frac{\sum_{i=1}^n(X_i - \bar{X})(Y_i - \bar{Y})}{n-1} \\
  &= \frac{\sum_{i=1}^nX_iY_i}{n-1} \\
  &= \frac{XY^T}{n-1}
\end{align}

and so the covariance matrix for a data set X that has zero mean in each feature vector is just $XX^T/(n-1)$.

In other words, we can also get the eigendecomposition of the covariance matrix from the positive semi-definite matrix $XX^T$.

### Note that zeroing the feature vector does not affect the covariance matrix

```python
np.set_printoptions(precision=3)
```

```python
X = np.random.random((5,4))
X
```

    array([[0.855, 0.667, 0.536, 0.461],
           [0.146, 0.353, 0.268, 0.189],
           [0.824, 0.852, 0.85 , 0.165],
           [0.246, 0.469, 0.454, 0.218],
           [0.423, 0.35 , 0.367, 0.064]])

```python
### Subtract the row mean from each row
```

```python
Y = X - X.mean(1)[:, None]
```

```python
Y.mean(1)
```

    array([ 0.000e+00,  0.000e+00, -2.776e-17,  2.776e-17,  0.000e+00])

```python
Y
```

    array([[ 0.225,  0.037, -0.094, -0.169],
           [-0.093,  0.114,  0.029, -0.05 ],
           [ 0.151,  0.179,  0.177, -0.508],
           [-0.101,  0.122,  0.107, -0.129],
           [ 0.122,  0.049,  0.066, -0.237]])

```python
### Calculate the covariance
```

```python
np.cov(X)
```

    array([[ 0.03 , -0.004,  0.037, -0.002,  0.021],
           [-0.004,  0.008,  0.012,  0.011,  0.003],
           [ 0.037,  0.012,  0.115,  0.03 ,  0.053],
           [-0.002,  0.011,  0.03 ,  0.018,  0.01 ],
           [ 0.021,  0.003,  0.053,  0.01 ,  0.026]])

```python
np.cov(Y)
```

    array([[ 0.03 , -0.004,  0.037, -0.002,  0.021],
           [-0.004,  0.008,  0.012,  0.011,  0.003],
           [ 0.037,  0.012,  0.115,  0.03 ,  0.053],
           [-0.002,  0.011,  0.03 ,  0.018,  0.01 ],
           [ 0.021,  0.003,  0.053,  0.01 ,  0.026]])

Eigendecomposition of the covariance matrix
----

```python
e1, v1 = np.linalg.eig(np.dot(x, x.T)/(n-1))
```

```python
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e1, v1.T):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

Change of basis via PCA
----

### We can transform the original data set so that the eigenvectors are the basis vectors and find the new coordinates of the data points with respect to this new basis

This is the change of basis transformation covered in the Linear Alegebra module. First, note that the covariance matrix is a real symmetric matrix, and so the eigenvector matrix is an orthogonal matrix.

```python
e, v = np.linalg.eig(np.cov(x))
v.dot(v.T)
```

    array([[1., 0.],
           [0., 1.]])

### Linear algebra review for change of basis

Graphical illustration of change of basis
----

Suppose we have a vector $u$ in the standard basis $B$ , and a matrix $A$ that maps $u$ to $v$, also in $B$. We can use the eigenvalues of $A$ to form a new basis $B'$. As explained above, to bring a vector $u$ from $B$-space to a vector $u'$ in $B'$-space, we multiply it by $Q^{-1}$, the inverse of the matrix having the eigenvctors as column vectors. Now, in the eigenvector basis, the equivalent operation to $A$ is the diagonal matrix $\Lambda$ - this takes $u'$ to $v'$. Finally, we convert $v'$ back to a vector $v$ in the standard basis by multiplying with $Q$.

```python
ys = np.dot(v1.T, x)
```

#### Principal components

Principal components are simply the eigenvectors of the covariance matrix used as basis vectors. Each of the original data points is expressed as a linear combination of the principal components, giving rise to a new set of coordinates.

```python
plt.scatter(ys[0,:], ys[1,:], alpha=0.2)
for e_, v_ in zip(e1, np.eye(2)):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

For example, if we only use the first column of `ys`, we will have the projection of the data onto the first principal component, capturing the majority of the variance in the data with a single feature that is a linear combination of the original features.

#### Transform back to original coordinates

We may need to transform the (reduced) data set to the original feature coordinates for interpretation. This is simply another linear transform (matrix multiplication).

```python
zs = np.dot(v1, ys)
```

```python
plt.scatter(zs[0,:], zs[1,:], alpha=0.2)
for e_, v_ in zip(e1, v1.T):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

```python
u, s, v = np.linalg.svd(x)
u.dot(u.T)
```

    array([[1.00e+00, 1.11e-16],
           [1.11e-16, 1.00e+00]])

Dimension reduction via PCA
----

We have the sepctral decomposition of the covariance matrix

$$
A = Q^{-1}\Lambda Q
$$

Suppose $\Lambda$ is a rank $p$ matrix. To reduce the dimensionality to $k \le p$, we simply set all but the first $k$ values of the diagonal of $\Lambda$ to zero. This is equivalent to ignoring all except the first $k$ principal components.

What does this achieve? Recall that $A$ is a covariance matrix, and the trace of the matrix is the overall variability, since it is the sum of the variances.

```python
A
```

    array([[0.578, 0.199],
           [0.199, 0.212]])

```python
A.trace()
```

    0.7895579305747163

```python
e, v = np.linalg.eig(A)
D = np.diag(e)
D
```

    array([[0.665, 0.   ],
           [0.   , 0.124]])

```python
D.trace()
```

    0.7895579305747163

```python
D[0,0]/D.trace()
```

    0.8424777390334711

Since the trace is invariant under change of basis, the total variability is also unchanged by PCA. By keeping only the first $k$ principal components, we can still "explain" $\sum_{i=1}^k e[i]/\sum{e}$ of the total variability. Sometimes, the degree of dimension reduction is specified as keeping enough principal components so that (say) $90\%$ of the total variability is explained.

### Using Singular Value Decomposition (SVD) for PCA

SVD is a decomposition of the data matrix $X = U S V^T$ where $U$ and $V$ are orthogonal matrices and $S$ is a diagnonal matrix.

Recall that the transpose of an orthogonal matrix is also its inverse, so if we multiply on the right by $X^T$, we get the follwoing simplification

\begin{align}
X &= U S V^T \\
X X^T &= U S V^T (U S V^T)^T \\
 &= U S V^T V S U^T \\
 &= U S^2 U^T
\end{align}

Compare with the eigendecomposition of a matrix $A = W \Lambda W^{-1}$, we see that SVD gives us the eigendecomposition of the matrix $XX^T$, which as we have just seen, is basically a scaled version of the covariance for a data matrix with zero mean, with the eigenvectors given by $U$ and eigenvealuse by $S^2$ (scaled by $n-1$)..

```python
u, s, v = np.linalg.svd(x)
```

```python
e2 = s**2/(n-1)
v2 = u
plt.scatter(x[0,:], x[1,:], alpha=0.2)
for e_, v_ in zip(e2, v2):
    plt.plot([0, 3*e_*v_[0]], [0, 3*e_*v_[1]], 'r-', lw=2)
plt.axis([-3,3,-3,3]);
```

```python
v1 # from eigenvectors of covariance matrix
```

    array([[ 0.916, -0.402],
           [ 0.402,  0.916]])

```python
v2 # from SVD
```

    array([[-0.916, -0.402],
           [-0.402,  0.916]])

```python
e1 # from eigenvalues of covariance matrix
```

    array([0.665, 0.125])

```python
e2 # from SVD
```

    array([0.665, 0.125])

```python
a0 = np.random.normal(0,1,100)
a1 = a0 + np.random.normal(0,3,100)
a2 = 2*a0 + a1 + np.random.normal(5,0.01,100)
xs = np.vstack([a0, a1, a2])
xs.shape
```

    (3, 100)

```python
C = np.cov(xs)
```

```python
C
```

    array([[ 1.069,  1.632,  3.77 ],
           [ 1.632, 10.081, 13.345],
           [ 3.77 , 13.345, 20.884]])

```python
e, v = np.linalg.eig(C)
```

```python
v
```

    array([[-0.136, -0.816,  0.561],
           [-0.551, -0.408, -0.728],
           [-0.823,  0.408,  0.394]])

```python
U, s, V = np.linalg.svd(xs)
```

```python
(s**2)/(99)
```

    array([51.845,  5.777,  0.684])

```python
U
```

    array([[-0.085,  0.062,  0.994],
           [-0.307,  0.948, -0.086],
           [-0.948, -0.313, -0.061]])

```python
np.round(e/e.sum(), 4)
```

    array([0.95, 0.  , 0.05])

```python
ys = np.linalg.inv(v).dot(xs)
```

```python
plt.scatter(ys[0,:], ys[2,:])
pass
```

```python
zs = v[:, [0,2]].dot(ys[[0,2],:])
zs.shape
```

    (3, 100)

```python
v[:, [0,2]]
```

    array([[-0.136,  0.561],
           [-0.551, -0.728],
           [-0.823,  0.394]])

```python
plt.figure(figsize=(12.,4))
plt.subplot(1,3,1)
plt.scatter(zs[0,:], zs[1,:])
plt.subplot(1,3,2)
plt.scatter(zs[0,:], zs[2,:])
plt.subplot(1,3,3)
plt.scatter(zs[1,:], zs[2,:])
pass
```

```python
plt.figure(figsize=(12.,4))
plt.subplot(1,3,1)
plt.scatter(xs[0,:], xs[1,:])
plt.subplot(1,3,2)
plt.scatter(xs[0,:], xs[2,:])
plt.subplot(1,3,3)
plt.scatter(xs[1,:], xs[2,:])
pass
```

```python
plt.figure(figsize=(12.,4))
plt.subplot(1,3,1)
plt.scatter(ys[0,:], ys[1,:])
plt.subplot(1,3,2)
plt.scatter(ys[0,:], ys[2,:])
plt.subplot(1,3,3)
plt.scatter(ys[1,:], ys[2,:])
pass
```


# High order positivity preserving methods

 1. Chapter 6: stability and positivity preserving high order methods
    1. [Section 1.1: Modified Patankar schemes](#mP)
    1. [Section 1.2: Strong Stability Preserving RK](#SSPRK)

```python
# If you do not have numpy, matplotlib, scipy or nodepy, run this cell
!pip install numpy
# This is the basic package in python with all the numerical functions

!pip install scipy
# This package has some functions to deal with polynomials

!pip install matplotlib
# This package allows to  plot

!pip install nodepy
# This package has some interesting features for RK methods
```

```python
# We need a couple of packages in this chapter
import numpy as np
# This is the basic package in python with all the numerical functions

import matplotlib.pyplot as plt
# This package allows to  plot

from nodepy import rk
#This package already implemented some functions for Runge Kutta and multistep methods
```

```python
import numpy as np

## Linear scalar Dahlquist's equation
def linear_scalar_flux(u,t=0,k_coef=10):
    ff=np.zeros(np.shape(u))
    ff[0]= -k_coef*u[0]
    return ff

def linear_scalar_exact_solution(u0,t,k_coef=10):
    return np.array([np.exp(-k_coef*u0[0]*t)])

def linear_scalar_jacobian(u,t=0,k_coef=10):
    Jf=np.zeros((len(u),len(u)))
    Jf[0,0]=-k_coef
    return Jf

#nonlinear problem y'=-ky|y| +1
def nonlinear_scalar_flux(u,t=0,k_coef=10):
    ff=np.zeros(np.shape(u))
    ff[0]=-k_coef*abs(u[0])*u[0] +1
    return ff

def nonlinear_scalar_exact_solution(u0,t,k_coef = 10):
        sqrtk = np.sqrt(k_coef)
        ustar = 1 / sqrtk
        if u0[0] >= ustar:
            uex=np.array([1./np.tanh(sqrtk * t + np.arctanh(1/sqrtk /u0[0])) / sqrtk])
        elif u0[0] < 0 and t < - np.atan(sqrtk * u0[0]) / sqrtk:
            uex=np.array([np.tan(sqrtk * t + np.arctan(sqrtk * u0[0])) / sqrtk])
        else:
            uex=np.array([np.tanh(sqrtk * t + np.arctanh(sqrtk * u0[0])) / sqrtk])
        return uex

def nonlinear_scalar_jacobian(u,t=0,k_coef=10):
    Jf=np.zeros((len(u),len(u)))
    Jf[0,0]=-k_coef*abs(u[0])
    return Jf

# SYSTEMS

# linear systems
def linear_system2_flux(u,t=0):
    d=np.zeros(len(u))
    d[0]= -5*u[0] + u[1]
    d[1]= 5*u[0] -u[1]
    return d

def linear_system2_exact_solution(u0,t):
    A=np.array([[-5,1],[5,-1]])
    u_e=u0+(1-np.exp(-6*t))/6*np.dot(A,u0)
    return u_e

def linear_system2_jacobian(u,t=0):
    Jf=np.array([[-5,1],[5,-1]])
    return Jf

linear_system2_matrix = np.array([[-5,1],[5,-1]])

def linear_system2_production_destruction(u,t=0):
    p=np.zeros((len(u),len(u)))
    d=np.zeros((len(u),len(u)))
    p[0,1]=u[1]
    d[1,0]=u[1]
    p[1,0]=5*u[0]
    d[0,1]=5*u[0]
    return p,d

#lin system 3 x3

def linear_system3_flux(u,t=0):
    d=np.zeros(len(u))
    d[0]= -u[0] + 3*u[1]
    d[1]= -3*u[1] + 5*u[2]
    d[2]= -5*u[2]
    return d

def linear_system3_exact_solution(u0,t=0):
    u_e = np.zeros(len(u0))
    u_e[0] = 15.0/8.0*u0[2]*(np.exp(-5*t) - 2*np.exp(-3*t)+np.exp(-t))
    u_e[1] = 5.0/2.0*u0[2]*(-np.exp(-5*t) + np.exp(-3*t))
    u_e[2] = u0[2]*np.exp(-5*t)
    return u_e
def linear_system3_jacobian(u,t=0):
    Jf=np.zeros((len(u),len(u)))
    Jf[0,0]=-1.
    Jf[0,1]=3
    Jf[1,1] = -3
    Jf[1,2] = 5
    Jf[2,2] = -5
    return Jf

## Nonlinear 3x3 system production destruction
def nonlinear_system3_flux(u,t=0):
    ff=np.zeros(len(u))
    ff[0]= -u[0]*u[1]/(u[0]+1)
    ff[1]= u[0]*u[1]/(u[0]+1) -0.3*u[1]
    ff[2]= 0.3*u[1]
    return ff

def nonlinear_system3_production_destruction(u,t=0):
    p=np.zeros((len(u),len(u)))
    d=np.zeros((len(u),len(u)))
    p[1,0]=u[0]*u[1]/(u[0]+1)
    d[0,1]=p[1,0]
    p[2,1]=0.3*u[1]
    d[1,2]=p[2,1]
    return p,d

# SIR Model
def SIR_flux(u,t=0,beta=3,gamma=1):
    ff=np.zeros(len(u))
    N=np.sum(u)
    ff[0]=-beta*u[0]*u[1]/N
    ff[1]=+beta*u[0]*u[1]/N - gamma*u[1]
    ff[2]= gamma*u[1]
    return ff

def SIR_jacobian(u,t=0,beta=3,gamma=1):
    Jf=np.zeros((len(u),len(u)))
    N=np.sum(u)
    Jf[0,0]=-beta*u[1]/N
    Jf[0,1]=-beta*u[0]/N
    Jf[1,0]= beta*u[1]/N
    Jf[1,1]= beta*u[0]/N - gamma
    Jf[2,1] = gamma
    return Jf

def SIR_production_destruction(u,t=0,beta=3,gamma=1):
    p=np.zeros((len(u),len(u)))
    d=np.zeros((len(u),len(u)))
    N=np.sum(u)
    p[1,0]=beta*u[0]*u[1]/N
    d[0,1]=p[1,0]
    p[2,1]=gamma*u[1]
    d[1,2]=p[2,1]
    return p,d

# Nonlinear_oscillator
def nonLinearOscillator_flux(u,t=0,alpha=0.):
    ff=np.zeros(np.shape(u))
    n=np.sqrt(np.dot(u,u))
    ff[0]=-u[1]/n-alpha*u[0]/n
    ff[1]=u[0]/n - alpha*u[1]/n
    return ff

def nonLinearOscillator_exact_solution(u0,t):
    u_ex=np.zeros(np.shape(u0))
    n=np.sqrt(np.dot(u0,u0))
    u_ex[0]=np.cos(t/n)*u0[0]-np.sin(t/n)*u0[1]
    u_ex[1]=np.sin(t/n)*u0[0]+np.cos(t/n)*u0[1]
    return u_ex

# Non linear oscillator damped
def nonLinearOscillatorDamped_flux(u,t,alpha=0.01):
    ff=np.zeros(np.shape(u))
    n=np.sqrt(np.dot(u,u))
    ff[0]=-u[1]/n-alpha*u[0]/n
    ff[1]=u[0]/n - alpha*u[1]/n
    return ff

def nonLinearOscillatorDamped_exact_solution(u0,t,alpha=0.01):
    u_ex=np.zeros(np.shape(u0))
    n0=np.sqrt(np.dot(u0,u0))
    n=n0*np.exp(-alpha*t)
    u_ex[0]=n/n0*(np.cos(t/n)*u0[0]-np.sin(t/n)*u0[1])
    u_ex[1]=n/n0*(np.sin(t/n)*u0[0]+np.cos(t/n)*u0[1])
    return u_ex

# pendulum
def pendulum_flux(u,t=0):
    ff=np.zeros(np.shape(u))
    ff[0]=u[1]
    ff[1]=-np.sin(u[0])
    return ff

def pendulum_jacobian(u,t=0):
    Jf=np.zeros((2,2))
    Jf[0,1]=1.
    Jf[1,0]=np.cos(u[0])
    return Jf

def pendulum_entropy(u,t=0):
    return np.array(0.5*u[1]**2.-np.cos(u[0]), dtype=np.float)

def pendulum_entropy_variables(u,t=0):
    v=np.zeros(np.shape(u))
    v[0]=np.sin(u[0])
    v[1]=u[1]
    return v

# Robertson
def Robertson_flux(u,t=0,alpha=10**4,beta=0.04, gamma=3*10**7):
    ff=np.zeros(np.shape(u))
    ff[0] = alpha*u[1]*u[2]-beta*u[0]
    ff[1] = beta*u[0]-alpha*u[1]*u[2] - gamma*u[1]**2
    ff[2] = gamma*u[1]**2
    return ff

def Robertson_jacobian(u,t=0,alpha=10**4,beta=0.04, gamma=3*10**7):
    Jf=np.zeros((3,3))
    Jf[0,0]= -beta
    Jf[0,1]= alpha*u[2]
    Jf[0,2]= alpha*u[1]
    Jf[1,0]= beta
    Jf[1,1]= -alpha*u[2]-2*gamma*u[1]
    Jf[1,2]= -alpha*u[1]
    Jf[2,1] = 2*gamma*u[1]
    return Jf

def Robertson_production_destruction(u,t=0,alpha=10**4,beta=0.04, gamma=3*10**7):
    p=np.zeros((len(u),len(u)))
    d=np.zeros((len(u),len(u)))
    p[0,1]=alpha*u[1]*u[2]
    d[1,0]=p[0,1]
    p[1,0]=beta*u[0]
    d[0,1]=p[1,0]
    p[2,1]=gamma*u[1]**2
    d[1,2]=p[2,1]
    return p,d

# Lotka:
def lotka_flux(u,t=0,alpha=1,beta=0.2,delta=0.5,gamma=0.2):
    ff=np.zeros(np.shape(u))
    ff[0]=alpha*u[0]-beta*u[0]*u[1]
    ff[1]=delta*beta*u[0]*u[1]-gamma*u[1]
    return ff

def lotka_jacobian(u,t=0,alpha=1,beta=0.2,delta=0.5,gamma=0.2):
    Jf=np.zeros((2,2))
    Jf[0,0] = alpha -beta*u[1]
    Jf[0,1] = -beta*u[0]
    Jf[1,0] = delta*beta*u[1]
    Jf[1,1] = delta*beta*u[0] -gamma
    return Jf

#3 bodies problem in 2D: U=(x_1,x_2,v_1,v_2,y_1,y_2,w_1,w_2,z_1,z_2,s_1,s_2)
# where x is the 2D position of body1 and v is speed body1 sun
# y, w are position and velocity body2 earth
# z, s are position and velocity body3 mars

def threeBodies_flux(u,t=0):
    m1=1.98892*10**30
    m2=5.9722*10**24
    m3=6.4185*10**23
    G=6.67*10**(-11)
    f=np.zeros(np.shape(u))
    x=u[0:2]
    v=u[2:4]
    y=u[4:6]
    w=u[6:8]
    z=u[8:10]
    s=u[10:12]
    dxy3=np.linalg.norm(x-y)**3
    dxz3=np.linalg.norm(x-z)**3
    dyz3=np.linalg.norm(y-z)**3
    f[0:2]=v
    f[2:4]=-m2*G/dxy3*(x-y)-m3*G/dxz3*(x-z)
    f[4:6]=w
    f[6:8]=-m1*G/dxy3*(y-x)-m3*G/dyz3*(y-z)
    f[8:10]=s
    f[10:12]=-m1*G/dxz3*(z-x)-m2*G/dyz3*(z-y)
    return f

class ODEproblem:
    def __init__(self,name):
        self.name=name
        if self.name=="linear_scalar":
            self.u0 = np.array([1.])
            self.T_fin= 2.
            self.k_coef=10
            self.matrix=np.array([-self.k_coef])
        elif self.name=="nonlinear_scalar":
            self.k_coef=10
            self.u0 = np.array([1.1/np.sqrt(self.k_coef)])
            self.T_fin= 1.
        elif self.name=="linear_system2":
            self.u0 = np.array([0.9,0.1])
            self.T_fin= 1.
            self.matrix = np.array([[-5,1],[5,-1]])
        elif self.name=="linear_system3":
            self.u0 = np.array([0,0.,10.])
            self.T_fin= 10.
        elif self.name=="nonlinear_system3":
            self.u0 = np.array([9.98,0.01,0.01])
            self.T_fin= 30.
        elif self.name=="SIR":
            self.u0 = np.array([1000.,1,10**-20])
            self.T_fin= 10.
        elif self.name=="nonLinearOscillator":
            self.u0 = np.array([1.,0.])
            self.T_fin= 50
        elif self.name=="nonLinearOscillatorDamped":
            self.u0 = np.array([1.,0.])
            self.T_fin= 50
        elif self.name=="pendulum":
            self.u0 = np.array([2.,0.])
            self.T_fin= 50
        elif self.name=="Robertson":
            self.u0 = np.array([1.,10**-20,10**-20])
            self.T_fin= 10.**10.
        elif self.name=="lotka":
            self.u0 = np.array([1.,2.])
            self.T_fin= 100.
        elif self.name=="threeBodies":
            self.u0 = np.array([0,0,0,0,149*10**9,0,0,30*10**3,-226*10**9,0,0,-24.0*10**3])
            self.T_fin= 10.**8.
        else:
            raise ValueError("Problem not defined")

    def flux(self,u,t=0):
        if self.name=="linear_scalar":
            return linear_scalar_flux(u,t,self.k_coef)
        elif self.name=="nonlinear_scalar":
            return nonlinear_scalar_flux(u,t,self.k_coef)
        elif self.name=="linear_system2":
            return linear_system2_flux(u,t)
        elif self.name=="linear_system3":
            return linear_system3_flux(u,t)
        elif self.name=="nonlinear_system3":
            return nonlinear_system3_flux(u,t)
        elif self.name=="SIR":
            return SIR_flux(u,t)
        elif self.name=="nonLinearOscillator":
            return nonLinearOscillator_flux(u,t)
        elif self.name=="nonLinearOscillatorDamped":
            return nonLinearOscillatorDamped_flux(u,t)
        elif self.name=="pendulum":
            return pendulum_flux(u,t)
        elif self.name=="Robertson":
            return Robertson_flux(u,t)
        elif self.name=="lotka":
            return lotka_flux(u,t)
        elif self.name=="threeBodies":
            return threeBodies_flux(u,t)
        else:
            raise ValueError("Flux not defined for this problem")

    def jacobian(self,u,t=0):
        if self.name=="linear_scalar":
            return linear_scalar_jacobian(u,t,self.k_coef)
        elif self.name=="nonlinear_scalar":
            return nonlinear_scalar_jacobian(u,t,self.k_coef)
        elif self.name=="linear_system2":
            return linear_system2_jacobian(u,t)
        elif self.name=="linear_system3":
            return linear_system3_jacobian(u,t)
        elif self.name=="pendulum":
            return pendulum_jacobian(u,t)
        elif self.name=="SIR":
            return SIR_jacobian(u,t)
        elif self.name=="Robertson":
            return Robertson_jacobian(u,t)
        elif self.name=="lotka":
            return lotka_jacobian(u,t)
        else:
            raise ValueError("Jacobian not defined for this problem")

    def exact(self,u,t):
        if self.name=="linear_scalar":
            return linear_scalar_exact_solution(u,t,self.k_coef)
        elif self.name=="nonlinear_scalar":
            return nonlinear_scalar_exact_solution(u,t,self.k_coef)
        elif self.name=="linear_system2":
            return linear_system2_exact_solution(u,t)
        elif self.name=="linear_system3":
            return linear_system3_exact_solution(u,t)
        elif self.name=="nonLinearOscillator":
            return nonLinearOscillator_exact_solution(u,t)
        elif self.name=="nonLinearOscillatorDamped":
            return nonLinearOscillatorDamped_exact_solution(u,t)
        else:
            raise ValueError("Exact solution not defined for this problem")

    def exact_solution_times(self,u0,tt):
        exact_solution=np.zeros((len(u0),len(tt)))
        for it, t in enumerate(tt):
            exact_solution[:,it]=self.exact(u0,t)
        return exact_solution

    def prod_dest(self,u,t=0):
        if self.name=="linear_system2":
            return linear_system2_production_destruction(u,t)
        if self.name=="nonlinear_system3":
            return nonlinear_system3_production_destruction(u,t)
        elif self.name=="Robertson":
            return Robertson_production_destruction(u,t)
        elif self.name=="SIR":
            return SIR_production_destruction(u,t)
        else:
            raise ValueError("Prod Dest not defined for this problem")

```

## Modified Patankar scheme for production-destruction systems

Consider production-destruction systems (PDS)

\begin{equation}
\begin{cases}
d_t c_i = P_i(c ) - D_i(c ) , \quad  i=1,\dots,I,\quad & P_i(c) = \sum_{j=1}^I p_{i,j}(c),\\
c(t=0)=c_0,& D_i(c) = \sum_{j=1}^I d_{i,j}(c),
\end{cases}
\end{equation}

where

$$p_{i,j}(c) , d_{i,j} (c) \geq 0, \qquad \forall i,j \in I, \quad \forall c \in \mathbb R^{+,I}.$$

Applications: Chemical reactions, biological systems, population evolutions and PDEs.

Problems: linear_system2, nonlinear_system3, Robertson, SIR.

Example: SIR

\begin{equation*}
\begin{cases}
d_tS = - \beta \frac{SI}{N}\\
d_t I = \beta \frac{SI}{N} -\gamma I\\
d_t R = \gamma I
\end{cases}
\end{equation*}

Property 1: Conservation
	\begin{align*}
		&\sum_{i=1}^I c_i(0) = \sum_{i=1}^I c_i(t), \quad \forall t\geq 0 \\
		\Longleftrightarrow \quad &p_{i,j}(c)=d_{j,i} (c) , \qquad \forall i,j \in I, \quad \forall c \in \mathbb R^{+,I}. \end{align*}

Property 2: Positivity

\begin{align*}
&\text{If }P_i,D_i\text{ Lipschitz, and if when } c_i\to 0 \Rightarrow D_i(c)\to 0 \Longrightarrow \\
&c_i(0) > 0 \,   \forall i \in I \Longrightarrow c_i(t)>0 \,\forall i \in I\; \forall t >0.
\end{align*}

Goal:
* One step method
* Unconditionally positive
* Unconditionally conservative
* High order accurate

**Explicit Euler**
* $c^{n+1}= c^n +\Delta t \left( P(c^n) - D(c^n) \right)$
* **Conservative**
* First order
* Not unconditionally positive, if $\Delta t$ is too big...

Consider a conservative and positive PDS where we assume that
the right hand side is not identical zero. Then, there exists
a $c^n\geq0$ such that ${P}(c^n)-{D}(c^n)\neq0$. Since the PDS is conservative,
we can at least
find one constituent $i\in \lbrace  1,\dots, I \rbrace$, where $D_i(c^n)>P_i(c^n)\geq0$. Choosing
\begin{equation}
\Delta t >\frac{c_i^n}{D_i(c^n)-P_i(c^n)} > 0,
\end{equation}
we obtain
\begin{equation}
 c_i^{n+1}=c_i^{n} +\Delta t\left(P_i(c^n)-D_i(c^n)\right)<c_i^{n} +\frac{c_i^n}{D_i(c^n)-P_i(c^n)} \left(P_i(c^n)-D_i(c^n)\right)
 =c_i^{n}-c_i^{n}=0.
\end{equation}
This demonstrates the violation of the positivity for the explicit Euler method for unbounded timesteps $\Delta t$.

**Patankar's scheme** [Patankar's book 1980](https://books.google.it/books/about/Numerical_Heat_Transfer_and_Fluid_Flow.html?id=N2MVAQAAIAAJ&redir_esc=y)
* Unconditionally positive
* Weighting the destruction term in the original explicit Euler method

$$
c_i^{n+1}=c_i^n+\Delta t  \left( \sum_{j=1}^I p_{i,j}(c^n) -
\sum_{j=1}^I d_{i,j}(c^n) \frac{c^{n+1}_i}{c_i^n} \right), \quad i=1,\dots, I,\\
\Longleftrightarrow \left(1 +\Delta t \sum_{j=1}^I  \frac{d_{i,j}(c^n)}{c_i^n} \right) c_i^{n+1}=c_i^n+\Delta t  \left( \sum_{j=1}^I p_{i,j}(c^n) \right), \quad i=1,\dots, I,
$$

* conservation relation  is violated.

```python
# Exercise
## Patankar 1st order scheme
def patankar(prod_dest, tspan, u0):
    '''
    Input: prod_dest is the function that returns the matrices p_{i,j}(c) and d_{i,j}(c)
    tspan is the time vector
    u0 is the initial condition
    '''
    dim=len(u0)            # Dimension of the problem
    Nt=len(tspan)          # Length of time span
    U=np.zeros((dim,Nt))   # Solution vector
    p=np.zeros((dim,dim))  # Temporary production matrix
    d=np.zeros((dim,dim))  # Temporary destruction matrix
    U[:,0]=u0
    for it in range(1,Nt):   # Loop over timesteps
        dt=tspan[it]-tspan[it-1]
        p,d =prod_dest(U[:,it-1])   # Computing the production and destruction at the previous timestep
        for i in range(dim):        # Adding all the terms
            lhs = 1.           # Initializing the lhs coefficients
            rhs = U[i,it-1]           # Initializing the rhs
            for j in range(dim):
                lhs = lhs +  FILL IN THE LHS
                rhs = rhs +  FILL IN THE RHS
            U[i,it] = rhs/lhs    # Solve the final system
    return tspan, U
```

```python
# Test the method on the SIR problem, what do you observe?
pr=ODEproblem("SIR")
tt=np.linspace(0,pr.T_fin, 100)
tt,UU=patankar(pr.prod_dest,tt,pr.u0)
plt.plot(tt,UU[0,:], label="S")
plt.plot(tt,UU[1,:], label="I")
plt.plot(tt,UU[2,:], label="R")
plt.plot(tt,np.sum(UU,0), label="Total")
plt.legend()
```

**Modified Patankar Scheme** [H Burchard, E Deleersnijder, A Meister 2003](https://www.researchgate.net/profile/Andreas-Meister-2/publication/225796292_Application_of_modified_Patankar_schemes_to_stiff_biogeochemical_models_for_the_water_column/links/5538e85a0cf247b8587d7efb/Application-of-modified-Patankar-schemes-to-stiff-biogeochemical-models-for-the-water-column.pdf)
* Modification of Patankar scheme
* Unconditionally positive
* Conservative
* Linearly implicit (with a mass inversion)

\begin{equation}
c_i^{n+1}:=c_i^n+\Delta t  \left( \sum_{j=1}^I p_{i,j}(c^n) \frac{c^{n+1}_j}{c_j^n} - \sum_{j=1}^I d_{i,j}(c^n) \frac{c^{n+1}_i}{c_i^n} \right), \quad i=1,\dots, I.
\end{equation}

The scheme is implicit and can be solved inverting
the mass matrix $M$ in the system $Mc^{n+1}=c^n$ where $M$ is

\begin{equation}
M_{i,j}(c^n) =
\begin{cases}
1+\Delta t \sum_{l=1}^I \frac{d_{i,l}(c^n)}{c_i^n} ,  \quad & \text{if } i=j,\\
- \Delta t \frac{p_{i,j}(c^n)}{c_j^n} , \quad & \text{if } i\neq j.
\end{cases}
\end{equation}

The mass matrix
* has positive values on the diagonal
* has negative values on the off-diagonal
* is strictly diagonally dominant by columns
* the inverse is positive

#### Jacobi's iterative method
Let $M$ be a strictly diagonally dominant (by columns) matrix, with $M=D-L$ with $D>0$ diagonal matrix and $L>0$ matrix with 0 entries on the diagonal. Then, $M$ is invertible and $M^{-1}$ is positive.
##### Proof
Suppose we want to solve

$$
Mx=b
$$

for $b\geq 0$ a nonnegative vector. Proving that $M^{-1}$ is positive is equivalent to prove that $x$ is nonnegative for any $b\geq 0$.

To get to Jacobi's iteration let us rewrite the system as

$$
Dx=Lx+b \Longleftrightarrow x=D^{-1}(Lx+b)
$$

If $x$ is a solution of the orginal system, it is a solution of the previous one.

How to get to the solution $x$? Iterations

$$
x^{(k)}=D^{-1}(Lx^{(k-1)}+b).
$$

* Do the iterations converge?

$$
e^{(k)}:=x^{(k)}-x \\
e^{(k)}=x^{(k)}-x= D^{-1}(Lx^{(k-1)}+b)-D^{-1}(Lx+b) = D^{-1}L e^{(k-1)}.
$$

Now, the $\infty$-norm of $D^{-1}L$ is smaller than 1, because

$$
||D^{-1}L||_\infty = \max_i \sum_{j} \frac{|L_{ji}|}{|D_{ii}|}=\max_i  \frac{\sum_{j} L_{ji}}{D_{ii}} < 1.
$$

Hence,

$$
||e^{(k)}||_\infty \leq ||D^{-1}L||_\infty||e^{(k-1)}||_\infty < ||e^{(k-1)}||_\infty.
$$

* Is the solution $x$ positive?
Suppose we start from a positive guess $x^{(0)}$, then, by induction,

$$
x^{(k)}=\underbrace{D^{-1}L}_{\geq 0} \underbrace{x^{(k-1)}}_{\geq 0}+\underbrace{D^{-1}b}_{\geq 0}.
$$

So,

$$x=\lim_{k\to \infty} x^{(k)} \geq 0.$$

```python
# Exercise
## Modified Patankar 1st order scheme

def mPEuler(prod_dest, tspan, u0):
    '''
    Input: prod_dest is the function that returns the matrices p_{i,j}(c) and d_{i,j}(c)
    tspan is the time vector
    u0 is the initial condition
    '''
    dim=len(u0)            # Dimension of the problem
    Nt=len(tspan)          # Length of time span
    U=np.zeros((dim,Nt))   # Solution vector
    p=np.zeros((dim,dim))  # Temporary production matrix
    d=np.zeros((dim,dim))  # Temporary destruction matrix
    U[:,0]=u0
    for it in range(1,Nt):   # Loop over timesteps
        dt=tspan[it]-tspan[it-1]
        p,d =prod_dest(U[:,it-1])   # Computing the production and destruction at the previous timestep
        MM = np.eye(dim)            # Initializing the mass matrix
        for i in range(dim):        # Adding all the terms
            for j in range(dim):
                MM[i,j] = MM[i,j] + FILL IN THE OFF DIAGONAL TERMs
                MM[i,i] = MM[i,i] + FILL IN THE DIAGONAL TERMs
        U[:,it] = np.linalg.solve(MM,U[:,it-1])   # Solve the final system
    return tspan, U
```

```python
# TEST the modified Patankar. Is it better?
pr=ODEproblem("SIR")
tt=np.linspace(0,pr.T_fin, 100)
tt,UU=mPEuler(pr.prod_dest,tt,pr.u0)
plt.plot(tt,UU[0,:], label="S")
plt.plot(tt,UU[1,:], label="I")
plt.plot(tt,UU[2,:], label="R")
plt.plot(tt,np.sum(UU,0), label="Total")
plt.legend()
```

**High Order modified Patankar Runge--Kutta Methods**
* One mPRK22 [H Burchard, E Deleersnijder, A Meister 2003](https://www.researchgate.net/profile/Andreas-Meister-2/publication/225796292_Application_of_modified_Patankar_schemes_to_stiff_biogeochemical_models_for_the_water_column/links/5538e85a0cf247b8587d7efb/Application-of-modified-Patankar-schemes-to-stiff-biogeochemical-models-for-the-water-column.pdf)
* [S. Kopecz and A. Meister 2018](https://arxiv.org/abs/1702.04589) [2019](https://arxiv.org/abs/1703.05052) Categorization of families of mPRK22$(\alpha)$ and mPRK43$(\alpha,\beta)$
* [J. Huang, C-W. Shu 2018](https://www.brown.edu/research/projects/scientific-computing/sites/brown.edu.research.projects.scientific-computing/files/uploads/Positivity-preserving%20time%20discretizations.pdf) mPRK22$(\alpha, \beta)$
* [J. Huang, W. Zhao, C-W. Shu 2019](https://doi.org/10.1007/s10915-018-0881-9) Third order RK43
* [P. Öffner, D. Torlo 2020](https://arxiv.org/abs/1905.09237) Arbitrarily high order mPDeC

**Remark**: It has been proven that there are not modified Patankar RK schemes of order 3 with 3 stages.

### Modified Patankar Deferred Correction schemes

Blackboard, slides and [article](https://arxiv.org/abs/1905.09237)!

Final formulation:
at each correction $(k)$ at each subtimestep $m$

\begin{equation}
\begin{split}
&\mathcal L^{1,m}_i (\mathbf{c}^{(k)})-\mathcal L^{1,m}_i (\mathbf{c}^{(k-1)})+\mathcal L^{2,m}_i (\mathbf{c}^{(k)},\mathbf{c}^{(k-1)})=0\\
&c_i^{m,(k)}-c^0_i -\Delta t   \sum_{r=0}^M \theta_r^m \sum_{j=1}^I
\left( p_{i,j}(c^{r,(k-1)})
\frac{c^{m,(k)}_{\gamma(j,i, \theta_r^m)}}{c_{\gamma(j,i, \theta_r^m)}^{m,(k-1)}}
- d_{i,j}(c^{r,(k-1)})  \frac{c^{m,(k)}_{\gamma(i,j, \theta_r^m)}}{c_{\gamma(i,j, \theta_r^m)}^{m,(k-1)}} \right)=0.
\end{split}
\end{equation}

with
$$
\gamma(i,j,\theta^m_r):=\begin{cases}
i& \text{if } \theta_r^m\geq 0\\
j& \text{if } \theta_r^m<0
\end{cases}
$$

**CODE!**

```python
## Reminder: DeC code

from scipy.interpolate import lagrange
from numpy.polynomial.legendre import leggauss

def equispaced(order):
    '''
    Takes input d and returns the vector of d equispaced points in [-1,1]
    And the integral of the basis functions interpolated in those points
    '''
    nodes= np.linspace(-1,1,order)
    w= np.zeros(order)
    for k in range(order):
        yy= np.zeros(order)
        yy[k]=1.
        zz=lagrange(nodes,yy)
        pp=zz.integ()
        w[k]=pp(1)-pp(-1)

    return nodes, w

def lglnodes(n,eps=10**-15):
    '''
    Python translation of lglnodes.m

    Computes the Legendre-Gauss-Lobatto nodes, weights and the LGL Vandermonde
    matrix. The LGL nodes are the zeros of (1-x^2)*P'_N(x). Useful for numerical
    integration and spectral methods.

    Parameters
    ----------
    n : integer, requesting an nth-order Gauss-quadrature rule on [-1, 1]

    Returns
    -------
    (nodes, weights) : tuple, representing the quadrature nodes and weights.
                       Note: (n+1) nodes and weights are returned.

    Example
    -------
    >>> from lglnodes import *
    >>> (nodes, weights) = lglnodes(3)
    >>> print(str(nodes) + "   " + str(weights))
    [-1.        -0.4472136  0.4472136  1.       ]   [0.16666667 0.83333333 0.83333333 0.16666667]

    Notes
    -----

    Reference on LGL nodes and weights:
      C. Canuto, M. Y. Hussaini, A. Quarteroni, T. A. Tang, "Spectral Methods
      in Fluid Dynamics," Section 2.3. Springer-Verlag 1987

    Written by Greg von Winckel - 04/17/2004
        Contact: gregvw@chtm.unm.edu

    Translated and modified into Python by Jacob Schroder - 9/15/2018
    '''

    w = np.zeros((n+1,))
    x = np.zeros((n+1,))
    xold = np.zeros((n+1,))

    # The Legendre Vandermonde Matrix
    P = np.zeros((n+1,n+1))

    epss = eps

    # Use the Chebyshev-Gauss-Lobatto nodes as the first guess
    for i in range(n+1):
        x[i] = -np.cos(np.pi*i / n)

    # Compute P using the recursion relation
    # Compute its first and second derivatives and
    # update x using the Newton-Raphson method.

    xold = 2.0

    for i in range(100):
        xold = x

        P[:,0] = 1.0
        P[:,1] = x

        for k in range(2,n+1):
            P[:,k] = ( (2*k-1)*x*P[:,k-1] - (k-1)*P[:,k-2] ) / k

        x = xold - ( x*P[:,n] - P[:,n-1] )/( (n+1)*P[:,n])

        if (max(abs(x - xold).flatten()) < epss ):
            break

    w = 2.0 / ( (n*(n+1))*(P[:,n]**2))

    return x, w

def lagrange_basis(nodes,x,k):
    y=np.zeros(x.size)
    for ix, xi in enumerate(x):
        tmp=[(xi-nodes[j])/(nodes[k]-nodes[j])  for j in range(len(nodes)) if j!=k]
        y[ix]=np.prod(tmp)
    return y

def get_nodes(order,nodes_type):
    if nodes_type=="equispaced":
        nodes,w = equispaced(order)
    elif nodes_type == "gaussLegendre":
        nodes,w = leggauss(order)
    elif nodes_type == "gaussLobatto":
        nodes, w = lglnodes(order-1,10**-15)
    nodes=nodes*0.5+0.5
    w = w*0.5
    return nodes, w

def compute_theta_DeC(order, nodes_type):
    nodes, w = get_nodes(order,nodes_type)
    int_nodes, int_w = get_nodes(order,"gaussLobatto")
    # generate theta coefficients
    theta = np.zeros((order,order))
    beta = np.zeros(order)
    for m in range(order):
        beta[m] = nodes[m]
        nodes_m = int_nodes*(nodes[m])
        w_m = int_w*(nodes[m])
        for r in range(order):
            theta[r,m] = sum(lagrange_basis(nodes,nodes_m,r)*w_m)
    return theta, beta

def dec(func, tspan, y_0, M_sub, K_corr, distribution):
    N_time=len(tspan)
    dim=len(y_0)
    U=np.zeros((dim, N_time))
    u_p=np.zeros((dim, M_sub+1))
    u_a=np.zeros((dim, M_sub+1))
    rhs= np.zeros((dim,M_sub+1))
    Theta, beta = compute_theta_DeC(M_sub+1,distribution)
    U[:,0]=y_0
    for it in range(1, N_time):
        delta_t=(tspan[it]-tspan[it-1])
        for m in range(M_sub+1):
            u_a[:,m]=U[:,it-1]
            u_p[:,m]=U[:,it-1]
        for k in range(1,K_corr+1):
            u_p=np.copy(u_a)
            for r in range(M_sub+1):
                rhs[:,r]=func(u_p[:,r])
            for m in range(1,M_sub+1):
                u_a[:,m]= U[:,it-1]+delta_t*sum([Theta[r,m]*rhs[:,r] for r in range(M_sub+1)])
        U[:,it]=u_a[:,M_sub]
    return tspan, U
```

```python
# Modified Patankar code!
# One function decMPatankar for the main algorithm
# One function patankar_type_dec to form the Mass matrix and solve the system
def decMPatankar(prod_dest, tspan, y_0, M_sub, K_corr, distribution):
    '''
    Input: prod_dest is the function that returns the matrices p_{i,j}(c) and d_{i,j}(c)
    tspan is the time vector
    y_0 is the initial condition
    M_sub is the number of subtimesteps
    K_corr is the number of iterations
    distribution is the subtimestep distribution equispaced, gaussLobatto
    '''
    N_time=len(tspan)                       # number of timestep
    dim=len(y_0)                            # dimension of the problem
    U=np.zeros((dim, N_time))               # solution vector
    u_p=np.zeros((dim, M_sub+1))            # solution at the correction $(k-1)$
    u_a=np.zeros((dim, M_sub+1))            # solution at the correction $(k)$
    prod_p = np.zeros((dim,dim,M_sub+1))    # production matrix at corr $(k-1)$
    dest_p = np.zeros((dim,dim,M_sub+1))    # destruction matrix at corr $(k-1)$
    Theta, beta = compute_theta_DeC(M_sub+1,distribution)   #Theta and beta coefficients of the DeC algorithm
    U[:,0]=y_0                              # Initial solution
    for it in range(1, N_time):             # timestep loop
        delta_t=(tspan[it]-tspan[it-1])
        for m in range(M_sub+1):            # Initialization of ua, up
            u_a[:,m]=U[:,it-1]
            u_p[:,m]=U[:,it-1]
        for k in range(1,K_corr+1):         # Loop on the corrections
            u_p=np.copy(u_a)
            for r in range(M_sub+1):        # Computation of production and destruction
                prod_p[:,:,r], dest_p[:,:,r]=prod_dest(u_p[:,r])
            for m in range(1,M_sub+1):      # Loop on the subtimesteps
                #Solution of the system
                u_a[:,m]= patankar_type_dec(prod_p,dest_p,delta_t,m,Theta,u_p)
        U[:,it]=u_a[:,M_sub]
    return tspan, U

# Exercise
def patankar_type_dec(prod_p,dest_p,delta_t,m,Theta,u_p):
    '''
    Solution of the Modified Patankar DeC system
    First computing the mass matrix and then solving the system
    Input: prod_p, dest_p production and destruction matrices of up at all subtimesteps (tensors dim x dim x M_sub)
    delta_t time step
    m is the current subtimestep
    Theta: are the theta coefficients of DeC algorithm
    u_p is the solution at the previous correction
    '''
    dim=prod_p.shape[0]
    M_sub=prod_p.shape[2]-1
    mass= np.eye(dim)            # Initialization of the mass matrix as the identity
    RHS= u_p[:,0]                # RHS is simply the solution at time t^n
    for i in range(dim):         # Loop on i of mass matrix
        for r in range(M_sub+1): # Sum on subtimesteps r
            if Theta[r,m]>0:     # Check on the sign of theta
                for j in range(dim):   # Distribution on diagonal and off-diagonal of prod and dest terms
                    mass[i,j]=mass[i,j]  FILL IN THE OFF DIAGONAL TERMS WHEN Theta positive
                    mass[i,i]=mass[i,i]  FILL IN THE DIAGONAL TERMS WHEN Theta positive
            elif Theta[r,m]<0:
                for j in range(dim):
                    mass[i,i]=mass[i,i] FILL IN THE DIAGONAL TERMS WHEN Theta negative
                    mass[i,j]=mass[i,j] FILL IN THE OFF DIAGONAL TERMS WHEN Theta negative
    return np.linalg.solve(mass,RHS)    # Solution of the system

```

```python
# Test on SIR
pr=ODEproblem("SIR")
tt=np.linspace(0,pr.T_fin, 100)
order = 5
tt,UU=decMPatankar(pr.prod_dest,tt,pr.u0, order-1 , order , "gaussLobatto")
plt.plot(tt,UU[0,:], label="S")
plt.plot(tt,UU[1,:], label="I")
plt.plot(tt,UU[2,:], label="R")
plt.plot(tt,np.sum(UU,0), label="Total")
plt.legend()

print("The minimum value reached is %e"%(np.amin(UU)))
print("The conservation error is %e"%(np.max(np.sum(UU,0)-np.sum(UU[:,0])) ))
```

```python
# Ttest on more complicated problems
pr=ODEproblem("nonlinear_system3")
tt=np.linspace(0,pr.T_fin, 100)
order = 5
tt,UU=decMPatankar(pr.prod_dest,tt,pr.u0, order-1 , order , "equispaced")
plt.plot(tt,UU[0,:], label="c0")
plt.plot(tt,UU[1,:], label="c1")
plt.plot(tt,UU[2,:], label="c2")
plt.plot(tt,np.sum(UU,0), label="Total")
plt.legend()

print("The minimum value reached is %e"%(np.amin(UU)))
print("The conservation error is %e"%(np.max(np.sum(UU,0)-np.sum(UU[:,0])) ))
```

```python
#Test convergence accuracy
def compute_integral_error(c,c_exact):  # c is dim x times
    times=np.shape(c)[1]
    error=0.
    for t in range(times):
        error = error + np.linalg.norm(c[:,t]-c_exact[:,t],2)**2.
    error = np.sqrt(error/times)
    return error

pr = ODEproblem("linear_system2")

NN=4
dts=[pr.T_fin/2.0**k for k in range(4,NN+4)]
errorsmPDeC=np.zeros(len(dts))

for order in range(2,10):
    for k in range(NN):
        dt0=dts[k]
        tt=np.arange(0,pr.T_fin,dt0)
        u_exact=pr.exact_solution_times(pr.u0,tt)
        t2,U2=decMPatankar(pr.prod_dest, tt, pr.u0, order-1, order, "gaussLobatto")
        errorsmPDeC[k]=compute_integral_error(U2,u_exact)

    plt.loglog(dts,errorsmPDeC,"--",label="mPDeC%d"%(order))
    plt.loglog(dts,[dt**order*errorsmPDeC[2]/dts[2]**order for dt in dts],":",label="ref %d"%(order))

plt.title("mPDeC")
plt.legend()
#plt.savefig("convergence_RDeC.pdf")
plt.show()

```

```python
# Stiff problem: Robertson
pr = ODEproblem("Robertson")

Nt=40
order = 5  #2
tt = np.array([np.exp(k) for k in np.linspace(-14,np.log(pr.T_fin),Nt)])
tt,yy=decMPatankar(pr.prod_dest, tt, pr.u0, order-1,order,"gaussLobatto")

plt.semilogx(tt,yy[0,:])
plt.semilogx(tt,yy[1,:]*10**4)
plt.semilogx(tt,yy[2,:])

plt.semilogx(tt,np.sum(yy,0),label="Total")
plt.legend()

print("The minimum value reached is %e"%(np.amin(yy)))
print("The conservation error is %e"%(np.max(np.sum(yy,0)-np.sum(yy[:,0])) ))
```

## Strong Stability Preserving Runge Kutta schemes

References:
1. S. Gottlieb, C.-W. Shu. Total Variation Diminishing Runge--Kutta 1996 [ResearchGate link](https://www.researchgate.net/publication/24288196_Total_Variation_Diminishing_Runge-Kutta_Schemes)
1. S. Gottlieb, C.-W. Shu, E. Tadmor. 2000 [Brown page](https://www.brown.edu/research/projects/scientific-computing/sites/brown.edu.research.projects.scientific-computing/files/uploads/Strong%20Stability-Preserving%20High-Order%20Time%20Discretization%20Methods.pdf)

We have seen that under certain conditions the **explicit Euler method** can preserve **stability** properties of some PDEs.
For example, in hyperbolic scalar problems

$$u_t+f(u)_x=0,$$

where $f\in \mathcal C^1$, we can prove that

$$ TV(u(t))\leq TV(u(t^0)),\qquad t>t^0, \qquad \text{with  } TV(u) = \sum_j |u_{j+1}-u_j|. $$

This property can be preserved in the **explicit Euler** time discretization with the incremental form spatial discretization

$$
U^{n+1}_j=U^{n}_j +\Delta t \left [ C_{j+1/2}(U_{j+1}^{n}-U_{j}^{n})-D_{j-1/2}(U_{j}^{n}-U_{j-1}^{n}) \right]
$$

if $C_{j+1/2},D_{j+1/2}\geq 0$ for all $j$ and

$$
\Delta t |C_{j+1/2}+D_{j+1/2}|\leq 1,\qquad \text{for all }j.
$$

We can say that under this restriction the explicit Euler method is **total variation diminishing** (TVD).

These restrictions are the so-called CFL conditions, which are a restriction on the time step $\Delta t\leq \Delta t_{eE}$, which are the equivalent of the restrictions we observe on the time step for ODEs, where

$$
\Delta t \leq \frac{C}{L},
$$

with $C$ a constant and $L$ the Lipschitz continuity constant of $F$.

### Strong Stability Preserving (SSP) Runge Kutta methods
#### Explicit
Suppose that explicit Euler method is SSP (for example TVD, but not only) under the restriction

$$
\Delta t \leq \Delta t_{eE},
$$

how can we be sure that a RK method is as well SSP.

A general RK method is written as

$$
\begin{cases}
y^{(1)}=y^n
y^{(k)}=y^n + \Delta t \sum_{j=1}^{k-1} a_{kj} F(y^{(j)}), \quad k=1,\dots, S,\\
y^{n+1} = y^n+ \Delta t \sum_{j=1}^S b_{j} F(y^{(j)})
\end{cases}
$$

We can rewrite them in the SSP form

$$
\begin{cases}
y^{(0)} = y^n
y^{(k)}=\sum_{j=0}^{k-1} \alpha_{kj} y^{(j)} +\Delta t \beta_{kj} F(y^{(j)}), \quad k=1,\dots, S,\\
y^{n+1} = y^{(S)},
\end{cases}
$$

where $\sum_{j=1}^S\alpha_{kj}=1$ for all $k$ for consistency. Suppose, moreover, that we can find nonnegative $\alpha_{kj}, \beta_{kj}\geq 0$. Then, we can find similar restrictions on this form to preserve the strong stability belonging to the explicit Euler method.

Indeed, each step can be rewritten as a **convex combination** of explicit Euler steps weighted by some coefficients

$$
y^{(k)}=\sum_{j=0}^{k-1} \alpha_{kj} \left( y^{(j)} +\Delta t \frac{\beta_{kj}}{\alpha_{kj} } F(y^{(j)})\right), \quad k=1,\dots, S,
$$

hence, if

$$
\Delta t \frac{\beta_{kj}}{\alpha_{kj} } \leq \Delta t_{eE} \quad \forall j<k\leq S\\
\text{or } \Delta t \leq c \Delta t_{eE},  \qquad c:=\min_{j<k\leq S} \frac{\alpha_{kj} }{\beta_{kj}},
$$

then the scheme is **SSP**.

**Remark 1**: It is not always possible to find nonnegative $\beta_{kj}$, in those cases, one can proceed similarly, adding a stability hypothesis on the explicit Euler method for the backward in time problem ($\tilde F$). (Cost of computing the operators $\tilde F$ and $F$ rises as well as storing memory).
If you can, avoid the negative coefficient.

**Remark 2**: Multistep method can be rendered into SSP version with analogous arguments.

**Remark 3**: the goal is to optimize $c$ by finding the coefficients $\alpha,\beta$ keeping the order of accuracy required.
This can be done starting from a known RK method where the coefficients are half of the SSPRK method, hence there's a lot to optimize.

**Example**
Consider the RK22 method

$$
u^{(1)}=u^n\\
u^{(2)}=u^{(1)} +\Delta t F(u^{(1)})\\
u^{n+1}=u^n+\frac{1}{2} \left(F(u^{(1)}) + F(u^{(2)}) \right)
$$

and try to write it into a SSPRK method

$$
u^{(0)}=u^n\\
u^{(1)} = u^{(0)} +\Delta t F(u^{(0)})\\
u^{(2)} = \alpha_{2,0} u^{(0)} +\alpha_{2,1} u^{(1)} + \Delta t \left( \beta_{2,0} F(u^{(0)}) + \beta_{2,1} F(u^{(1)})\right) \\
u^{n+1}=u^{(2)}
$$

which, expanding becomes

$$
u^{n+1}= u^{(0)}  + \Delta t \left( \left( \beta_{2,0}+\alpha_{2,1}\right)  F(u^{(0)}) + \beta_{2,1} F(u^{(1)})\right),
$$

which, if we want to make the two method coincide, leads to the system

$$
\begin{cases}
\alpha_{2,0}+\alpha_{2,1}=1\\
\beta_{2,0}+\alpha_{2,1} =\frac{1}{2}\\
\beta_{2,1} = \frac{1}{2}.
\end{cases}
$$

#### Theorem 1
An $S$-stages, $S$-order SSPRK method has at most CFL $c\leq 1$.

#### Theorem 2
For **linear problems** ($F(y)=My$), there exist $S$-stages, $S$-order SSPRK methods with CFL $c=1$. (For nonlinear problems they are not order $S$).

#### Theorem 3
Optimal 2nd order 2 stages SSPRK with CFL $c=1$ is

$$
u^{(1)} = u^n +\Delta t F(u^n)\\
u^{n+1} = \frac{1}{2} u^n  + \frac{1}{2} u^{(1)} +\frac{1}{2} \Delta t F(u^{(1)}).
$$

Optimal 3nd order 3 stages SSPRK with CFL $c=1$ is

$$
u^{(1)} = u^n +\Delta t F(u^n)\\
u^{(2)} = \frac{3}{4} u^n  + \frac{1}{4} u^{(1)} +\frac{1}{4} \Delta t F(u^{(1)})\\
u^{n+1} = \frac{1}{3} u^n  + \frac{2}{3} u^{(2)} +\frac{2}{3} \Delta t F(u^{(2)}).
$$

There is no 4th order 4 stages with positive coefficients SSPRK.

```python
A=np.array([[0,0],[1,0]])
b=np.array([0.5,0.5])
rk2 = rk.ExplicitRungeKuttaMethod(A,b)

A=np.array([[0]])
b=np.array([1])
rk1 = rk.ExplicitRungeKuttaMethod(A,b)

print(rk2.optimal_shu_osher_form())

A=np.array([[0,0],[0.5,0]])
b=np.array([0,1])
rk2 = rk.ExplicitRungeKuttaMethod(A,b)

print(rk2.optimal_shu_osher_form())
```

#### Implicit RK
It is not so trivial to prove properties for implicit schemes, as a nonlinear solver is involved. When restricting to specific cases, it is possible to recast some results.

##### Theorem 1 (Harten)
Implicit Euler for incremental finite difference form is **unconditionally** TVD (SSP) (for any $\Delta t$).

*Proof in Chapter 2*

##### Theorem 2
It does not exists an implicit RK scheme of the form

$$
u^{(0)}=u^n\\
u^{(k)}=\sum_{j=0}^{k-1}\alpha_{kj}u^{(j)}+\Delta t \beta_k F(u^{(k)}),\qquad \alpha_{kj}\geq 0, \qquad k=1,\dots, S,\\
u^{n+1}=u^{(S)},
$$

with order $\geq 2$ that is also unconditionally stable.

### SSP multistep methods
For multistep methods we have analogous results, there are a series of optimal values for the coefficients of explicit SSP multistep methods, while there exists no high order unconditionally SSP implicit multistep method.

A collection of optimal values of SSPRK can be found at [Gottlieb's page at Brown University](http://www.cfm.brown.edu/people/sg/ssp.html)

More optimal values in [Ruuth Spiteri 2006](https://epubs.siam.org/doi/abs/10.1137/S0036142901389025?journalCode=sjnaam)

```python
#Few examples of order 3
ssprk33=rk.loadRKM("SSP33")
ssprk33.print_shu_osher()
print(ssprk33)
ssprk33.plot_stability_region()
plt.show()
```

```python
for method in ["SSP33","SSP43","SSP53","SSP63"]:
    ssprk=rk.loadRKM(method)
    ssprk.print_shu_osher()
    print(ssprk)
    ssprk.plot_stability_region()
    plt.show()

```

```python
#Few examples
ssprk104=rk.loadRKM("SSP104")
ssprk104.print_shu_osher()
print(ssprk104)
ssprk104.plot_stability_region()
plt.show()
```

#### Exercise
Test few methods with your favorite problems.

Chapter 6 Positivity preserving schemes.ipynb


```python
import MVG as MVG
import SuperMatExpr as SME
from sympy import *
import utils as utils

def getDet(cov):
     return -cov.shape[0]/2*ln(2*pi) - Rational(1,2)*ln(Determinant(cov))

D, N, Ns = symbols('D N Ns')
sig_y = symbols('\u03c3_y')

w = SME.SuperMatSymbol('w',D,1,'var')
y = SME.SuperMatSymbol('y',N,1,'var')
ys = SME.SuperMatSymbol('y_s',Ns,1,'var')
X = SME.SuperMatSymbol('X',D,N,'var')
Xs = SME.SuperMatSymbol('X_s',D,Ns,'var')

mu_w = SME.SuperMatSymbol('',D,1,'mean',dependent_vars=[w],expanded=ZeroMatrix(D,1))
mu_y = SME.SuperMatSymbol('',N,1,'mean',dependent_vars=[y],cond_vars=[w,X],expanded=X.T*w)
mu_ys = SME.SuperMatSymbol('',Ns,1,'mean',dependent_vars=[ys],cond_vars=[w,Xs],expanded=Xs.T*w)
S_ww = SME.SuperMatSymbol('',D,D,'covar',dependent_vars=[w],expanded=Identity(D))
S_yy = SME.SuperMatSymbol('',N,N,'covar',dependent_vars=[y],cond_vars=[w,X],expanded=sig_y**2*Identity(N))
S_ysys = SME.SuperMatSymbol('',Ns,Ns,'covar',dependent_vars=[ys],cond_vars=[w,Xs],expanded=sig_y**2*Identity(Ns))

p_w = MVG.MVG([w],moments=[mu_w,S_ww,getDet(S_ww)])
p_y = MVG.MVG([y],moments=[mu_y,S_yy,getDet(S_yy)],conditioned_vars=[w,X])
p_ys = MVG.MVG([ys],moments=[mu_ys,S_ysys,getDet(S_ysys)],conditioned_vars=[w,Xs])
p_w_y = p_w*p_y
p_w_post = p_w_y.condition([y])
p_w_ys = p_w_post*p_ys
p_ys_post = p_w_ys.marginalize([w])
print(utils.matLatex(p_ys_post.covar.expanded))
```

$I + -1 X \left(\sigma_y^{2} I + X^T X\right)^{-1} X^T$

$Xs^T X \left(\sigma_y^{2} I + X^T X\right)^{-1} y$

```python

```


# A Detailed RBC Model Example

Consider the equilibrium conditions for a basic RBC model without labor:

\begin{align}
C_t^{-\sigma} & = \beta E_t \left[C_{t+1}^{-\sigma}(\alpha A_{t+1} K_{t+1}^{\alpha-1} + 1 - \delta)\right]\\
Y_t & = A_t K_t^{\alpha}\\
I_t & = K_{t+1} - (1-\delta)K_t\\
Y_t & = C_t + I_t\\
\log A_t & = \rho_a \log A_{t-1} + \epsilon_t
\end{align}

In the nonstochastic steady state, we have:

\begin{align}
K & = \left(\frac{\alpha A}{1/\beta+\delta-1}\right)^{\frac{1}{1-\alpha}}\\
Y & = AK^{\alpha}\\
I & = \delta K\\
C & = Y - I
\end{align}

Given values for the parameters $\beta$, $\sigma$, $\alpha$, $\delta$, and $A$, steady state values of capital,  output, investment, and consumption are easily computed.

## Import requisite modules

```python
# Import numpy, pandas, linearsolve, matplotlib.pyplot
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

## Initializing the model in `linearsolve`

To initialize the model, we need to first set the model's parameters. We do this by creating a Pandas Series variable called `parameters`:

```python
# Input model parameters
parameters = pd.Series(dtype=float)
parameters['alpha']  = .35
parameters['beta']  = 0.99
parameters['delta']   = 0.025
parameters['rhoa'] = .9
parameters['sigma'] = 1.5
parameters['A'] = 1
```

Next, we need to define a function that returns the equilibrium conditions of the model. The function will take as inputs two vectors: one vector of "current" variables and another of "forward-looking" or one-period-ahead variables. The function will return an array that represents the equilibirum conditions of the model. We'll enter each equation with all variables moved to one side of the equals sign. For example, here's how we'll enter the produciton fucntion:

`production_function = technology_current*capital_current**alpha - output_curent`

Here the variable `production_function` stores the production function equation set equal to zero. We can enter the equations in almost any way we want. For example, we could also have entered the production function this way:

`production_function = 1 - output_curent/technology_current/capital_current**alpha`

One more thing to consider: the natural log in the equation describing the evolution of total factor productivity will create problems for the solution routine later on. So rewrite the equation as:

\begin{align}
A_{t+1} & =  A_{t}^{\rho_a}e^{\epsilon_{t+1}}\\
\end{align}

So the complete system of equations that we enter into the program looks like:

\begin{align}
C_t^{-\sigma} & = \beta E_t \left[C_{t+1}^{-\sigma}(\alpha Y_{t+1} /K_{t+1}+ 1 - \delta)\right]\\
Y_t & = A_t K_t^{\alpha}\\
I_t & = K_{t+1} - (1-\delta)K_t\\
Y_t & = C_t + I_t\\
A_{t+1} & =  A_{t}^{\rho_a}e^{\epsilon_{t+1}}
\end{align}

Now let's define the function that returns the equilibrium conditions:

```python
# Define function to compute equilibrium conditions
def equations(variables_forward,variables_current,parameters):

    # Parameters
    p = parameters

    # Variables
    fwd = variables_forward
    cur = variables_current

    # Household Euler equation
    euler_eqn = p.beta*fwd.c**-p.sigma*(p.alpha*fwd.y/fwd.k+1-p.delta) - cur.c**-p.sigma

    # Production function
    production_fuction =  cur.a*cur.k**p.alpha - cur.y

    # Capital evolution
    capital_evolution = fwd.k - (1-p.delta)*cur.k - cur.i

    # Goods market clearing
    market_clearing = cur.c + cur.i - cur.y

    # Exogenous technology
    technology_proc = cur.a**p.rhoa- fwd.a

    # Stack equilibrium conditions into a numpy array
    return np.array([
            euler_eqn,
            production_fuction,
            capital_evolution,
            market_clearing,
            technology_proc
        ])
```

Notice that inside the function we have to define the variables of the model form the elements of the input vectors `variables_forward` and `variables_current`.

## Initializing the model

To initialize the model, we need to specify the total number of state variables in the model, the number of state variables with exogenous shocks, the names of the endogenous variables, and the parameters of the model.

It is *essential* that the variable names are ordered in the following way: First the names of the endogenous variables with the state variables with exogenous shocks, then the state variables without shocks, and finally the control variables. Ordering within the groups doesn't matter.

```python
# Initialize the model
rbc = ls.model(equations = equations,
               n_states=2,
               n_exo_states=1,
               var_names=['a','k','c','y','i'],
               parameters=parameters)
```

## Steady state

Next, we need to compute the nonstochastic steady state of the model. The `.compute_ss()` method can be used to compute the steady state numerically. The method's default is to use scipy's `fsolve()` function, but other scipy root-finding functions can be used: `root`, `broyden1`, and `broyden2`. The optional argument `options` lets the user pass keywords directly to the optimization function. Check out the documentation for Scipy's nonlinear solvers here: http://docs.scipy.org/doc/scipy/reference/optimize.html

```python
# Compute the steady state numerically
guess = [1,1,1,1,1]
rbc.compute_ss(guess)
print(rbc.ss)
```

    a     1.000000
    k    34.398226
    c     2.589794
    y     3.449750
    i     0.859956
    dtype: float64

Note that the steady state is returned as a Pandas Series. Alternatively, you could compute the steady state directly and then sent the `rbc.ss` attribute:

```python
# Steady state solution
p = parameters
K = (p.alpha*p.A/(1/p.beta+p.delta-1))**(1/(1-p.alpha))
C = p.A*K**p.alpha - p.delta*K
Y = p.A*K**p.alpha
I = Y - C

rbc.set_ss([p.A,K,C,Y,I])
print(rbc.ss)
```

    a     1.000000
    k    34.398226
    c     2.589794
    y     3.449750
    i     0.859956
    dtype: float64

## Log-linearization and solution

Now we use the `.log_linear()` method to find the log-linear appxoximation to the model's equilibrium conditions. That is, we'll transform the nonlinear model into a linear model in which all variables are expressed as log-deviations from the steady state. Specifically, we'll compute the matrices $A$ and $B$ that satisfy:

\begin{align}
A E_t\left[ x_{t+1} \right] & = B x_t + \left[ \begin{array}{c} \epsilon_{t+1} \\ 0 \end{array} \right],
\end{align}

where the vector $x_{t}$ denotes the log deviation of the endogenous variables from their steady state values.

```python
# Find the log-linear approximation around the non-stochastic steady state
rbc.log_linear_approximation()

print('The matrix A:\n\n',np.around(rbc.a,4),'\n\n')
print('The matrix B:\n\n',np.around(rbc.b,4))
```

    The matrix A:

     [[ 0.00000e+00 -8.30000e-03 -3.59900e-01  8.30000e-03  0.00000e+00]
     [ 0.00000e+00  0.00000e+00  0.00000e+00  0.00000e+00  0.00000e+00]
     [ 0.00000e+00  3.43982e+01  0.00000e+00  0.00000e+00  0.00000e+00]
     [ 0.00000e+00  0.00000e+00  0.00000e+00  0.00000e+00  0.00000e+00]
     [-1.00000e+00  0.00000e+00  0.00000e+00  0.00000e+00  0.00000e+00]]

    The matrix B:

     [[-0.     -0.     -0.3599 -0.     -0.    ]
     [-3.4497 -1.2074 -0.      3.4497 -0.    ]
     [-0.     33.5383 -0.     -0.      0.86  ]
     [-0.     -0.     -2.5898  3.4497 -0.86  ]
     [-0.9    -0.     -0.     -0.     -0.    ]]

Finally, we need to obtain the *solution* to the log-linearized model. The solution is a pair of matrices $F$ and $P$ that specify:

1. The current values of the non-state variables $u_{t}$ as a linear function of the previous values of the state variables $s_t$.
1. The future values of the state variables $s_{t+1}$ as a linear function of the previous values of the state variables $s_t$ and the future realisation of the exogenous shock process $\epsilon_{t+1}$.

\begin{align}
u_t  &  = Fs_t\\
s_{t+1} & = Ps_t + \epsilon_{t+1}.
\end{align}

We use the `.klein()` method to find the solution.

```python
# Solve the model
rbc.solve_klein(rbc.a,rbc.b)

# Display the output
print('The matrix F:\n\n',np.around(rbc.f,4),'\n\n')
print('The matrix P:\n\n',np.around(rbc.p,4))
```

    The matrix F:

     [[ 0.2297  0.513 ]
     [ 1.      0.35  ]
     [ 3.3197 -0.1408]]

    The matrix P:

     [[0.9    0.    ]
     [0.083  0.9715]]

## Impulse responses

One the model is solved, use the `.impulse()` method to compute impulse responses to exogenous shocks to the state. The method creates the `.irs` attribute which is a dictionary with keys equal to the names of the exogenous shocks and the values are Pandas DataFrames with the computed impulse respones. You can supply your own values for the shocks, but the default is 0.01 for each exogenous shock.

```python
# Compute impulse responses and plot
rbc.impulse(T=41,t0=1,shocks=None,percent=True)

print('Impulse responses to a 0.01 unit shock to A:\n\n',rbc.irs['e_a'].head())
```

    Impulse responses to a 0.01 unit shock to A:

        e_a      a         k         c         y         i
    0  0.0  0.000  0.000000  0.000000  0.000000  0.000000
    1  1.0  1.000  0.000000  0.229718  1.000000  3.319739
    2  0.0  0.900  0.082993  0.249318  0.929048  2.976083
    3  0.0  0.810  0.155321  0.265744  0.864362  2.667127
    4  0.0  0.729  0.218116  0.279348  0.805341  2.389390

Plotting is easy.

```python
rbc.irs['e_a'][['a','k','c','y','i']].plot(lw='5',alpha=0.5,grid=True).legend(loc='upper right',ncol=2)
rbc.irs['e_a'][['e_a','a']].plot(lw='5',alpha=0.5,grid=True).legend(loc='upper right',ncol=2)
```

## Stochastic simulation

Creating a stochastic simulation of the model is straightforward with the `.stoch_sim()` method. In the following example, I create a 151 period (including t=0) simulation by first simulating the model for 251 periods and then dropping the first 100 values. The standard deviation of the shock to $A_t$ is set to 0.00763. The seed for the numpy random number generator is set to 0.

```python
rbc.stoch_sim(T=121,drop_first=100,cov_mat=np.array([0.00763**2]),seed=0,percent=True)
rbc.simulated[['k','c','y','i']].plot(lw='5',alpha=0.5,grid=True).legend(loc='upper right',ncol=4)
rbc.simulated[['a']].plot(lw='5',alpha=0.5,grid=True).legend(ncol=4)
rbc.simulated['e_a'].plot(lw='5',alpha=0.5,grid=True).legend(ncol=4)
```