 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  The presentation below largely follows part II in "Finite Difference Methods for Ordinary and Partial Differential Equations" by LeVeque (SIAM, 2007).

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Solution to ODE Initial Value Problems - Part 2

# Overview -- so far

So far we have discussed 3 basic techniques for integration of ODE IVP's

* **Single-Step Multi-Stage** schemes (explicit and implicit)
* **Taylor's Series** Methods
* **Linear Multi-Step** Schemes (explicit and implicit)

as well as
* **truncation error** of each method (and it's relation to step-error)
* **adaptive stepping**

# Overview -- to do

This notebook will take the analysis of these methods to discuss the other important issues of
* **Convergence** (Global Error) -- does the discrete solution converge to the continuous analytic solution?
* **Stability** -- Are there limits to how large a time-step we can take?
* **Stiff Equations**:  consequences of stability

# Convergence

We can think of an ODE method as providing a sequence of approximations $U_N$ where here $N$ is the number of time steps needed to reach the final time of interest $t_f$, in effect we are increasing the resolution of the method.  We then say that a method is convergent if this sequence converges to the true solution at the same time
$$
    \lim_{N\rightarrow \infty} U_N = u(t_f).
$$

We can also define this in a more familiar way in terms of $\Delta t$ such that
$$
    N \Delta t = t_f ~~~~ \Rightarrow ~~~~ N = \frac{t_f}{\Delta t}
$$
so that our definition of convergence becomes
$$
    \lim_{\Delta t \rightarrow 0} U_{\Delta t} = u(t_f).
$$

In general for a method to be convergent it must be
 - **consistent** which as before meant that the local truncation error $T = \mathcal{O}(\Delta t^p)$ where $p > 0$,
 - **zero-stable** which implies that the sum total of the errors as $\Delta t \rightarrow 0$ is bounded and has the same order as $T$ (the truncation error) which we know goes to zero as $\Delta t \rightarrow 0$.

put another way,  a method is convergent if the **global error**

$$
    E = | U_N(t_f) - u(t_f)| \rightarrow 0\quad\mathrm{as}\quad N\rightarrow\infty
$$

### Example:  Forward Euler on a Linear Problem

Consider the simple linear problem
$$
    \frac{\text{d}u}{\text{d}t} = \lambda u \quad \text{with} \quad u(0) = u_0
$$
which we know has the solution $u(t) = u_0 e^{\lambda t}$.  Applying Euler's method to this problem leads to
$$\begin{aligned}
    U_{n+1} &= U_n + \Delta t\lambda U_n \\
            &= (1 + \Delta t \lambda) U_n
\end{aligned}$$

We also know the local truncation error is defined by
$$\begin{aligned}
    T_n &= \frac{1}{\Delta t} \left[ U_{n+1} - u_{n+1} \right ] \\
        &= \frac{1}{\Delta t} \left[ (1 + \lambda \Delta t) u_n - u_{n+1} \right ]
\end{aligned}$$
which can be rearranged to find

$$
    u_{n+1} = (1 + \Delta t \lambda) u_n - \Delta t T_n.
$$

Note that all values here are in terms of the exact solution where as the application of Euler's method is in terms of the approximate solution.

Defining the global error as
$$
    E_{n} = u_n - U_n
$$

we can subtract the last two expressions to find

$$\begin{aligned}
    u_{n+1} - U_{n+1} &= \underbrace{(1 + \Delta t \lambda) u_n - \Delta t T_n}_{u_{n+1}} - \underbrace{(1 + \Delta t \lambda) U_n}_{U_{n+1}} \\
    E_{n+1} &= (1 + \Delta t \lambda) (u_n - U_n) - \Delta t T_n \\
            &= (1 + \Delta t \lambda) E_n - \Delta t T_n,
\end{aligned}$$

a recursive definition for the global error that connects it to the local truncation error.

Working backwards from the global error $E_n$ at maximum time step $n = t_f/\Delta t$,  the first few iterates of this look like

$$\begin{align}
    E_n &= (1 + \lambda \Delta t) E_{n - 1} - \Delta t T_{n-1} \\
    &= (1 + \lambda \Delta t) \left [(1 + \lambda \Delta t) E_{n - 2} - \Delta t T_{n-2} \right ] - \Delta t T_{n-1} \\
    &= (1 + \lambda \Delta t) \left [(1 + \lambda \Delta t) \left \{(1 + \lambda \Delta t) E_{n - 3} - \Delta t T_{n-3} \right \} - \Delta t T_{n-2} \right ] - \Delta t T_{n-1}
\end{align}$$

Which at this point looks like
$$
    E_n = (1 + \lambda \Delta t)^3E_{n-3} - \Delta t\sum_{i=1}^3 (1+\Delta t\lambda)^{i-1}T_{n-i}
$$

So taking it back $n$ steps ($3\rightarrow n$) gives
Which at this point looks like
$$
    E_n = (1 + \lambda \Delta t)^nE_0 - \Delta t\sum_{i=1}^n (1+\Delta t\lambda)^{i-1}T_{n-i}
$$

Expanding this expression out backwards in time to $n=0$ leads to
$$
    E_n = (1 + \Delta t \lambda)^n E_0 - \Delta t \sum^n_{i=1} (1 + \Delta t \lambda)^{i-1} T_{n - i}.
$$

Noting that $(1 + \Delta t \lambda)$ are the first two terms in the Taylor series for $e^{\lambda\Delta t}$ we will bound this term by

$$
    e^{\Delta t \lambda} \geq |1 + \Delta t \lambda|
$$

which then implies the term in the summation can be bounded by

$$
    |1 + \Delta t \lambda|^{i - 1} \leq e^{(n-1) \Delta t |\lambda|} \leq e^{n \Delta t |\lambda|} \leq e^{|\lambda| t_f}
$$

Using this expression in the expression for the global error we find
$$\begin{aligned}
    E_n &= (1 + \Delta t \lambda)^n E_0 - \Delta t \sum^n_{i=1} (1 + \Delta t \lambda)^{i-1} T_{n-i} \\
    |E_n| &\leq e^{|\lambda| \Delta t n} |E_0| + \Delta t \sum^n_{i=1} e^{|\lambda| t_f} |T_{n-i}| \\
          &\leq e^{|\lambda| t_f} \left(|E_0| + \Delta t \sum^n_{i=1} |T_{n - i}|\right) \\
          &\leq e^{|\lambda| t_f} \left(|E_0| + n \Delta t \max_{1 \leq i \leq n} |T_{n - i}|\right)
\end{aligned}$$

In other words the global error is bounded by the original global error and the maximum one-step error made multiplied by the number of time steps taken.  If $N = \frac{t_f}{\Delta t}$ as before and taking into account the local truncation error we can simplify this expression further to

$$
    |E_n| \leq e^{|\lambda| t_f} \left[|E_0| + t_f \left(\frac{1}{2} \Delta t |u''| + \mathcal{O}(\Delta t^2)\right ) \right]
$$

If we assume that we have used the correct initial condition $u_0$ then $E_0 \rightarrow 0$ as $\Delta t \rightarrow 0$ and we see that the global error $E_n$ is bounded by $\mathcal{O}(\Delta t)$, the same as the local truncation error:
$$
    |E_n| \leq e^{|\lambda| t_f} t_f \left(\frac{1}{2} \Delta t |u''| + \mathcal{O}(\Delta t^2)\right ) = \mathcal{O}(\Delta t).
$$

More generally,  it can be shown that for all single-step multi-stage schemes (of which Euler is the simplest), the Global error has the same order as the Truncation error, and therefore all of these schemes converge.

## Absolute Stability
Although zero-stability guarantees stability it is much more difficult to work with in general as the limit $\Delta t \rightarrow 0$ can be difficult to compute (and in general isn't ideal as it requires an infinite amount of computation).  Instead we often consider a finite $\Delta t$ and examine if the method is stable for this particular choice of $\Delta t$.  This has the practical upside that it will also tell us what particular $\Delta t$ will ensure that our method is indeed stable.

### Example
Consider the problem
$$
    u'(t) = \lambda (u - \cos t) - \sin t ~~~~ \text{with} ~~~~ u(0) = 1
$$

whose exact solution is
$$
    u(t) = \cos t.
$$

We can compute an estimate for what $\Delta t$ we need to use by examining the truncation error for Euler's method

$$\begin{aligned}
    T &= \frac{1}{2} \Delta t u''(t) + \mathcal{O}(\Delta t^2) \\
      &= -\frac{1}{2} \Delta t \cos t + \mathcal{O}(\Delta t^2)
\end{aligned}$$

and therefore
$$
    |E_n| \leq \Delta t \max_{0 \leq t \leq t_f} |\cos t| = \Delta t.
$$

If we want a solution where $|E_n| < 10^{-3}$ then $\Delta t \approx 10^{-3}$.  Turning to the application of Euler's method lets apply this to the case where $\lambda = -10$ and $\lambda = -2100$.

```python
# Implement vectorized Forward Euler
def euler(f, t_span, u0, N):
    """ simple implementation of constant step-size forward euler method for vector valued f
        This doc string should have so much more in it
    """
    t = numpy.linspace(t_span[0], t_span[1],N)
    if numpy.isscalar(u0):
        u = numpy.empty(t.shape)
    else:
        u = numpy.empty((len(t),len(u0)))
    u[0] = u0
    delta_t = t[1] - t[0]
    for (n, t_n) in enumerate(t[:-1]):
        K1 = delta_t * f(t_n, u[n])
        u[n + 1] = u[n] + K1
    return t, u.transpose()
```

```python
# Compare accuracy between Euler for different values of lambda
f = lambda t, u, lam: lam * (u - numpy.cos(t)) - numpy.sin(t)
u_exact = lambda t: numpy.cos(t)
u_0 = u_exact(0.)

t_span = [0., 2.]
num_steps = [2**n for n in range(4, 9)]
```

```python
#num_steps = [2**n for n in range(15,20)]
delta_t = numpy.empty(len(num_steps))
error_10 = numpy.empty(len(num_steps))
error_2100 = numpy.empty(len(num_steps))

t_f = t_span[1]
u_f = u_exact(t_f)

for (i, N) in enumerate(num_steps):
    # Compute Euler solution
    ff = lambda t, u: f(t, u, -10.)
    t, U_10 = euler(ff, t_span, u_0, N)
    delta_t[i] = t[1] - t[0]
    error_10[i] = numpy.abs(U_10[-1] - u_f) / numpy.abs(u_f)
    ff = lambda t, u: f(t, u, -2100.)
    t, U_2100 = euler(ff, t_span, u_0, N)
    error_2100[i] = numpy.abs(U_2100[-1] - u_f) / numpy.abs(u_f)

print('Error: lambda =   10: {}'.format(error_10))
print('Error: lambda = 2100: {}'.format(error_2100))

# Plot error vs. delta_t
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 2, 1)

axes.loglog(delta_t, error_10, 'bo', label='Forward Euler')

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[1], error_10[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")

axes.legend(loc=4)
axes.set_title("Error - $\lambda = 10$")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

axes = fig.add_subplot(1, 2, 2)

axes.loglog(delta_t, error_2100, 'bo', label='Forward Euler')
axes.loglog(delta_t, order_C(delta_t[1], error_2100[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")

axes.set_title("Error - $\lambda = 2100$")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

plt.show()
```

So what went wrong with $\lambda = -2100$, the global error should go as

$$
    E_n = \mathcal{O}(\Delta t)?
$$

If $\Delta t \approx 10^{-3}$ then for the case $\lambda = -10$ the previous **global error** is multiplied by $1+\lambda\Delta t$

$$1 + 10^{-3} \cdot -10 = 0.99$$

which means the contribution from $E^n$ will slowly decrease as we take more time steps.  For the other case we have

$$1 + 10^{-3} \cdot -2100 = -1.1$$

which means that for this $\Delta t$ the error made in previous time steps will grow!  For this not to happen we would have to have $\Delta t < 1 / 2100$ which would lead to convergence again.

### A simpler example
Consider our simplest test problem
$$
    u'(t) = \lambda u, \quad u(0) = u_0
$$

whose exact solution is
$$
    u(t) = u_0 e^{\lambda t}.
$$
which for real $\lambda < 0$ should decay to zero as $t\rightarrow\infty$

### Question?

What is the **largest** step-size $\Delta t$ for an Euler method, such that the discrete solution $U_N\rightarrow 0$ as $N\rightarrow\infty$?

Euler's method on this simplest problem can be written

$$
\begin{align}
    U_{n+1} &= U_n + \Delta t f(U_n) = U_n + \Delta t \lambda U_n\\
            &= (1 + \lambda\Delta t)U_n\\
\end{align}
$$

i.e. at each step the last solution is multiplied by the factor $(1 + \lambda\Delta t)$

After $N$ steps

$$
    U_N = (1 + \lambda\Delta t)^N U_0
$$

Assuming $\lambda < 0$ what is the largest step size $\Delta t$ such that $U_N\rightarrow 0$ as $N\rightarrow\infty$?

Hopefully it's clear that for this scheme to be stable requires

$$
    |1 + \lambda\Delta t| < 1
$$

The key parameter is actually the combination of $\lambda$ and the time step $z=\lambda\Delta t$, so stability requires that

$$
    | 1 + z | < 1
$$
or  𝑧∈[?,?]

```python
plt.figure(figsize=(8,6))
z = numpy.linspace(-3,2,200)
plt.plot(z,numpy.abs(1 + z))
plt.plot(z,numpy.ones(z.shape),'k--')
plt.plot([0.,0],[0.,3.])
plt.xlabel('$z$',fontsize=16)
plt.ylabel('$|1 + z|$',fontsize=16)
plt.grid()
plt.show()

```

Given $z\in[-2,0]$,  the  time step is simply

$$
    \Delta t = \frac{z}{\lambda}
$$
so the maximum time step we can take for $\lambda<0$ is

$$
    \Delta t = -\frac{2}{\lambda} > 0
$$

```python
# Compare accuracy between Euler for different values of time steps and $\lambda = -1$
lam = -1.
f = lambda t, u :  lam * u
u_exact = lambda t: numpy.exp(lam * t)
u_0 = u_exact(0.)

t_span = [0., 10.]

N = 21
N = 11
N = 6
N = 5
t, u = euler(f, t_span, u_0, N)

```

```python
tt = numpy.linspace(0.,t_span[1],100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.plot(tt, u_exact(tt),label='$u_{{exact}}$')
axes.plot(t, u, 'ro-', label='$u_{{euler}}$')
axes.set_xlabel('t')
axes.set_title('$\Delta t = {}$'.format(t[1]-t[0]), fontsize=18)
axes.grid()
axes.legend(loc='best',fontsize=14)
plt.show()

```

### Absolute Stability of the Forward Euler Method for complex $z$

The region $|1 + z| < 1$  is the **region of absolute stability**.  For more general problems $\lambda$ may be complex (stemming from eigenvalues of real problems),  so we generally consider the region of stability on the complex plane.

$$
    | 1 + z | < 1
$$
for $z\in\mathbb{C}$

```python
# Plot the region of absolute stability for Forward Euler
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

axes.fill(numpy.cos(t) - 1.0, numpy.sin(t), color=(255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
axes.set_xlim((-3, 3.0))
axes.set_ylim((-3,3))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Forward Euler")
plt.show()
```

### Why complex eigenvalues?  And why should we care?

#### A simple example:  The simple harmonic oscillator

The linear harmonic oscillator can be written as a 2nd-order ODE for position $x$ of a mass on a spring
$$
    \frac{d^2 x}{dt^2} + x = 0, \quad x(0)=x_0,\,  \frac{dx}{dt} = 0.
$$
Which has solutions
$$
    x(t) = x_0\cos(t)
$$

Alternatively, we can write this as a 2 dimensional system of first order equations by  defining the velocity $v= dx/dt$ and
$$
\begin{align}
    \frac{dx}{dt} &= v \\
    \frac{dv}{dt} &= -x \\
\end{align}
$$
or more cleanly as
$$
    \frac{d\mathbf{u}}{dt} = A\mathbf{u},\quad\mathbf{u}(0)=\mathbf{u_0}
$$
with
$$
\mathbf{u}=\begin{bmatrix} x\\ v\\\end{bmatrix}\quad A= \begin{bmatrix} 0 & 1 \\ -1 & 0 \\ \end{bmatrix}\quad\mathbf{u}_0=\begin{bmatrix} x_0\\ 0\\\end{bmatrix}
$$

All diagonalizable linear dynamical systems have solutions of the form

$$
    \mathbf{u}(t) = \sum_{i=1}^n c_i e^{\lambda_i t}\mathbf{s}_i
$$

where $\lambda_i$, $\mathbf{s}_i$ are corresponding eigenvalues and eigenvectors of $A$ and each eigenvector satisfies
$$
    \frac{d\mathbf{s}_i}{dt} = \lambda_i\mathbf{s}_i
$$
which is exactly our model problem.

In the case where
$$
A= \begin{bmatrix} 0 & 1 \\ -1 & 0 \\ \end{bmatrix}
$$

The eigenvalues are _ _ _ _ _ _ _ _ ?

Even with complex Eigenvalues and Eigenvectors, the solutions are real and can be shown to be perfect circles on the phase plane

$$
\begin{align}
    x(t) &= \cos(t)\\
    y(t) &=-\sin(t)\\
\end{align}
$$

However,  the plotting the eigenvalues on the stability diagram show's that Euler's method will be completely unstable for this problem for all time steps $\Delta t$

```python
# Plot the region of absolute stability for Forward Euler
x = numpy.linspace(-2, 2, 11)
v = numpy.linspace(-2, 2, 11)
X,V = numpy.meshgrid(x,v)
t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1,2,1)
axes.quiver(X, V, V, -X)
axes.plot(1.5*numpy.cos(t), -1.5*numpy.sin(t))
axes.plot([-2, 2],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-2, 2],'k--')
axes.set_xlabel('x',fontsize=16)
axes.set_ylabel('v',fontsize=16)

axes.set_title('Phase Plane: simple Harmonic oscillator',fontsize=16)
axes.set_aspect('equal')

axes = fig.add_subplot(1, 2, 2)
axes.fill(numpy.cos(t) - 1.0, numpy.sin(t), color=(255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
axes.plot([0., 0.0], [-1., 1.],'rx',markersize=10)
axes.set_xlim((-3, 3.0))
axes.set_ylim((-3,3))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Forward Euler",fontsize=16)
plt.show()
```

```python
f = lambda t, u : numpy.array([ u[1], -u[0]])
t_span = [0., 4*numpy.pi]
u_0 = numpy.array([1., 0.])
t, u = euler(f,t_span,u_0, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1,1,1)
axes.quiver(X, V, V, -X)
axes.plot(1.*numpy.cos(t), -numpy.sin(t))
axes.plot(u[0],u[1],'r')
axes.plot([-2, 2],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-2, 2],'k--')
axes.set_xlabel('x',fontsize=16)
axes.set_ylabel('v',fontsize=16)

axes.set_title("Euler's method: simple Harmonic oscillator", fontsize=16)
axes.set_aspect('equal')
```

### Absolute Stability of Single-Step Multi-Stage schemes -- The R method

This approach can be applied to **all** Single-Step Multi-Stage schemes including all the Runge-Kutta and Taylor Series methods.  The recipe is straightforward

*  Apply the stepping scheme for one step of the model problem $u' = \lambda u$, $u(0)=u_0$ assuming complex $\lambda$
*  This will result in a discrete approximation

$$
    U(\Delta t) = R(z) U_0
$$

where $z=\lambda\Delta t\in\mathbb{C}$, which will be a discrete approximation to the true solution

$$
    U(\Delta t) = e^{z} U_0
$$
* Absolute stability will require that $|R(z)| < 1$

### Absolute Stability of the backward Euler Method

Now try this on backward Euler.

$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}).
$$

which for our model problem is

$$
    U_{n+1} = U_n + \Delta t\lambda U_{n+1}
$$

or rearranging and solving for $U_{n+1}$ gives

$$
    U_{n+1} = \frac{1}{1-\Delta t\lambda} U_n
$$

so
$$
    R(z) = \frac{1}{1-z}
$$
and absolute stability requires

$$
    |R(z)| \leq 1 \leftrightarrow |1 - z| \geq 1$$

so in fact the stability region encompasses the entire complex plane except for a circle centered at $(1, 0)$ of radius 1 implying that the backward Euler method is in fact stable for any choice of $\Delta t$.

The backward version of Euler's method is defined as

$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}).
$$

If we again consider the test problem from before we find that

$$\begin{aligned}
    0 &= U_{n+1} (1 - \Delta t \lambda) - U_n \\
    &= \xi (1 - z) - 1
\end{aligned}$$

which has the root $\xi = \frac{1}{1 - z}$.  We then have

$$\left|\frac{1}{1-z}\right| \leq 1 \leftrightarrow |1 - z| \geq 1$$

so in fact the stability region encompasses the entire complex plane except for a circle centered at $(1, 0)$ of radius 1 implying that the backward Euler method is in fact stable for any choice of $\Delta t$.

```python
# Plot the region of absolute stability for Backward Euler
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

axes.set_facecolor((255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.fill(numpy.cos(t) + 1.0, numpy.sin(t), 'w')
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
# axes.set_xlim((-3, 1))
axes.set_ylim((-2,2))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Backward Euler")
plt.show()
```

## General Stability Regions for Linear Methods

We can also map out  stability regions for Linear Multi-step methods (although the analysis is slightly more complicated).

All of the Linear Multi-step methods from the previous lecture scan be written
$$
    \sum^r_{j=0} \alpha_j U_{n+j} = \Delta t \sum^r_{j=0} \beta_j f(t,U_{n+j})
$$

which for our model problem, $f(t,u) = \lambda u$ becomes
$$
    \sum^r_{j=0} \alpha_j U_{n+j} = \Delta t \sum^r_{j=0} \beta_j \lambda U_{n+j}
$$

or  rearranging and again defining $z=\Delta t\lambda$ becomes

$$
    \sum^r_{j=0} (\alpha_j - \beta_j z) U_{n+j} = 0.
$$
which takes the form of a "Linear Difference Equation"

### Characteristic Polynomials and Linear Difference Equations

To help us analyze stability regions we need to take a small aside and consider linear difference equations and their solutions.  Say we wanted to solve

$$\sum^r_{j=0} \alpha_j U_{n+j} = 0$$

given initial conditions $U_0, U_1, \ldots, U_{r-1}$.  This expression has a solution in the general form $U_n = \xi^n$.

Plugging this into the equation we have

$$\sum^r_{j=0} \alpha_j \xi^{n+j} = 0$$

which simplifies to

$$\sum^r_{j=0} \alpha_j \xi^j = 0 $$

by dividing by $\xi^n$.  If $\xi$ then is a root of the polynomial

$$\rho(\xi) \equiv \sum^r_{j=0} \alpha_j \xi^j$$

then $\xi$ solves the original difference equation.

### Example:  the Fibonacci Numbers

A classic example of  linear difference equation is the Fibonnacci sequence, $0,1,1,2,3,5,\ldots$.

Which are solutions to the two-term difference equation

$$
    U_{n+2} = U_{n+1} + U_n, \quad U_0=0,\, U_1 = 1
$$

or $\alpha = \begin{bmatrix} -1 & -1 & 1 \\\end{bmatrix}$

Substituting $U_n = \xi^n$ into
$$
U_{n+2} - U_{n+1} - U_n =0
$$
yields
$$
    \xi^n\left(\xi^2 - \xi -1\right) = 0
$$
which has solutions $\xi =  \frac{1\pm\sqrt{5}}{2}$.  Where the positive root is our favorite number $\phi=1.618033$ which is the golden ratio.  The negative root is $1-\phi$. Note, these roots are related to the eigenvalues of a related problem

Thus all Fibonacci numbers can be written as a linear combination of the two roots
$$
    U_n = A\phi^n + B(1-\phi)^n
$$
where $A$ and $B$ are determined by the first two terms in the sequence [0,1].

Note, in this case $\phi > 1$ (while $|(1-\phi)|<1$) thus $U_n\rightarrow\infty$ as $n\rightarrow\infty$.

### Stability for LMM's
Anyway,  Back to the Linear difference equations arising from applying any Linear multistep method to the model problem
$$
    \sum^r_{j=0} (\alpha_j - \beta_j z) U_{n+j} = 0.
$$

Letting
$$
    \rho(\xi) = \sum^r_{j=0} \alpha_j \xi^j, \quad \sigma(\xi) = \sum^r_{j=0} \beta_j \xi^j
$$
and substituting $U_n = \xi^n$ into the difference equations reduces to
$$
    \pi(\xi, z) = \rho(\xi) - z \sigma(\xi) = 0
$$
where $\pi(\xi, z)$ is the **stability polynomial** of the method, whose roots are solutions of the difference method.

### Absolute stability of the linear multi-step methods

For $U_n$ to stay bounded as $n\rightarrow\infty$,  it requires that all roots $\xi_i$ of $\pi(\xi, z)$ satisfy
$$
    |\xi_i| \leq 1
$$
If this is the case, then the multi-step method is **zero-stable**.  Roughly you can think of these roots as multiplying the past truncation errors so that if they are less than or equal to 1 they do not grow in magnitude.  We then define the region of **absolute stability** as the values for $z$ for which this is true.  This approach reduces to the $R$ method for  one-step methods as a special case.

### Example:  Forward Euler's Method

Examining forward Euler's method we have

$$\begin{aligned}
    0 &= U_{n+1} - U_n - \Delta t \lambda U_n \\
    &= U_{n+1} - U_n (1 + \Delta t \lambda)\\
\end{aligned}
$$

setting $U_n = \xi^n$ implies
$$
\begin{aligned}
   0 &= \xi^n\left(\xi -  (1 + z)\right)\\
\end{aligned}$$

or $\pi(\xi, z) = \xi -  (1 + z)$ whose root is $\xi = 1 + z$ and we have re-derived the stability region we had found before.

### Example:  Backwards Euler's Method

Similarly for backwards Euler

$$\begin{aligned}
    0 &= U_{n+1} - U_n - \Delta t \lambda U_{n+1} \\
    &= U_{n+1}(1 - \Delta t\lambda) - U_n \\
    &= \xi(1 - z)  - 1 \\
    &=\pi(\xi, z)
\end{aligned}$$

whose root is $\xi = 1/(1 - z)$

### Example: Adams-Bashforth 2

As an example consider the Adams-Bashforth 2-stage LMM
$$
    U_{n+2} = U_{n+1} + \frac{\Delta t}{2} (-f(U_n) + 3 f(U_{n+1}))
$$

Let $f(U) = \lambda U$,  $z = \Delta t \lambda$
$$
    U_{n+2} - U_{n+1} - \frac{1}{2} (-z U_n + 3 z U_{n+1}) = 0 \\
$$

substituting $U_n=\xi^n$, and factoring out $\xi^n$ yields
$$
\begin{aligned}
    \pi(\xi, z) &= \xi^{2} - \xi - \frac{1}{2} (-z  + 3 z \xi) \\
    &= 2 \xi^2 - (2 + 3z) \xi + z \\
\end{aligned}$$

## Plotting Stability Regions

Given a stability polynomial $\pi(\xi,z)$, absolute stability requires that

$$
    \max{|\xi_i|} <= 1
$$
where $\xi_i$ are the (generally complex) roots of $\pi(\xi, z)$ and therefore a function of $z$.  In the case of single step schemes, the problem reduces to finding the region of the complex plane where $|R(z)| < 1$.

Either way,  the simplest way to visualize the regions of the complex plane where the scheme is stable, is to calculate either $|R(z)|$ or $\max|\xi_i(z)|$ and plot the 1-contour and determine all regions that are $<1$.

### Some useful codes

```python
def stability_plot(X, Y, C, axes, title=None, continuous=True):
    """
    Utility function to make stability diagram given complex stability scalar C

    parameters:
    -----------

    X, Y:  numpy.meshgrids for complex plane
    C: numpy array
        Field to plot,  either |R(z)| for a single step scheme, or max(|xi_i(z)|) for a LMM scheme
    axes: matplotlib axes object
        subplot or plot to draw in.
    title: string
        subplot title if not None
    continuous: boolean
        if True, plot a continous coloring of C
        if False, plot Heaviside(C)
    """
    if  continuous:
        Ch = C
    else:
        Ch = numpy.heaviside(C-1,0.)
    pcolor_plot = axes.pcolor(X, Y, Ch, vmin=0, vmax=1, cmap=plt.get_cmap('Greens_r'))
    axes.contour(X, Y, C, 'k', levels=[1.0])
    fig = plt.gcf()
    fig.colorbar(pcolor_plot)
    axes.plot(x, numpy.zeros(x.shape),'k--')
    axes.plot(numpy.zeros(y.shape), y,'k--')

    axes.set_xlabel('Re', fontsize=16)
    axes.set_ylabel('Im', fontsize=16)
    if title is not None:
        axes.set_title(title, fontsize=16)

    axes.set_aspect('equal')

def plot_stability_ssms(R, x, y, axes=None, title=None):
    """
    plot stability regions for single-step multi-stage ODE schemes given the function R(z)
    such that U_{n+1} = R(z)U_n   and z is complex

    parameters:
    -----------

    R: calleable
        function of a complex variable z such that if |R|<=1, the scheme is absolutely stable
    x: numpy array
        array of values for the real axis
    y: numpy array
        values to plot for the imaginary axis
    axes: matplotlib axes object
        subplot or plot to draw in.  If axes=None create a new figure
    title: string
        subplot title if
    """

    X,Y = numpy.meshgrid(x,y)
    Z = X + 1j * Y
    if axes is None:
        fig = plt.figure(figsize=(8,6))
        axes = fig.add_subplot(1,1,1)

    abs_R = numpy.abs(R(Z))
    stability_plot(X, Y, numpy.abs(R(Z)), axes, title)

def plot_stability_lmm(pi_coeff, x, y, axes=None, title=None):
    """
    plot stability regions for linear multi-step  ODE schemes given the coefficients of the stability polynomial
    pi(xi, z)

    parameters:
    -----------

    pi_coeff: calleable (function of z)
        function that returns array of stability polynomial pi(z)
    x: numpy array
        array of values for the real axis
    y: numpy array
        values to plot for the imaginary axis
    axes: matplotlib axes object
        subplot or plot to draw in.  If axes=None create a new figure
    title: string
        subplot title if not None
    """

    X,Y = numpy.meshgrid(x,y)
    Z = X + 1j * Y
    if axes is None:
        fig = plt.figure(figsize=(8,6))
        axes = fig.add_subplot(1,1,1)

    norm_max = numpy.empty(Z.shape)
    for i,row in enumerate(Z):
        for j, z in enumerate(row):
            norm_max[i,j] = max(numpy.abs(numpy.roots(pi_coeff(z))))

    stability_plot(X, Y, norm_max, axes, title)

```

```python
x = numpy.linspace(-3,3,100)
y = numpy.linspace(-3,3,100)

R_euler = lambda z: 1 + z
R_beuler = lambda z: 1./(1. - z)
R_trap = lambda z: (1. + 0.5*z)/(1. - 0.5*z)
R_RK2 = lambda z: 1 + z + z**2/2
R_Taylor4 = lambda z: 1 + z + z**2/2 + z**3/6. + z**4/24.
```

```python
fig = plt.figure(figsize=(12,12))
axes = fig.add_subplot(2,2,1)
plot_stability_ssms(R_euler, x, y, axes=axes, title='Stability of Eulers method')

axes = fig.add_subplot(2,2,2)
plot_stability_ssms(R_beuler, x, y, axes=axes, title='Stability of Backwards Eulers')

axes = fig.add_subplot(2,2,3)
plot_stability_ssms(R_trap, x, y, axes=axes, title='Stability of Trapezoidal method')

axes = fig.add_subplot(2,2,4)
plot_stability_ssms(R_Taylor4, x, y, axes=axes, title='Stability of Taylor4 method')

```

```python
x = numpy.linspace(-3,3,100)
y = numpy.linspace(-3,3,100)

pi_euler = lambda z: numpy.array([1., -(1 + z)])
pi_AB2 = lambda z: numpy.array([2, -(2. + 3*z), z])
```

```python
fig = plt.figure(figsize=(12,6))
axes = fig.add_subplot(1,2,1)
plot_stability_lmm(pi_euler, x, y, axes=axes, title='Stability of Eulers method')
axes = fig.add_subplot(1,2,2)
plot_stability_lmm(pi_AB2, x, y, axes=axes, title='Stability of Adams-Bashforth2 method')
```

## Application to Stiff ODEs

Consider again the ODE we examined before

$$u'(t) = \lambda (u - \cos t) - \sin t$$

except this time with general initial condition $u(t_0) = u_0$.  What happens to solutions that are slightly different from $u_0 = 1$ or $t_0 = 0$?  The general solution of the ODE is

$$u(t) = e^{\lambda (t - t_0)} (u_0 - \cos t_0) + \cos t$$.

```python
# Plot "hairy" solutions to the ODE
u = lambda t_0, u0, lam, t: numpy.exp(lam * (t - t_0)) * (u0 - numpy.cos(t_0)) + numpy.cos(t)

fig = plt.figure(figsize=(12, 10))

for i,lam in enumerate([-1, -10, 1, 10]):
    axes = fig.add_subplot(2, 2, i+1)
    for u0 in numpy.linspace(-1, 1, 10):
        for t_0 in numpy.linspace(0.0, 9.0, 10):
            t = numpy.linspace(t_0,10.0,100)
            axes.plot(t, u(t_0, u0, lam, t),'b')
    t = numpy.linspace(0.0,10.0,100)
    axes.plot(t, numpy.cos(t), 'r', linewidth=5)
    axes.set_ylim((-2, 2))

    axes.set_title("Perturbed Solutions $\lambda = %s$" % lam)
    axes.set_xlabel('$t$')
    axes.set_ylabel('$u(t)$')
axes.set_ylim((-2, 2))
plt.show()

```

```python
# Plot "inverse hairy" solutions to the ODE
u = lambda t_0, u0, lam, t: numpy.exp(lam * (t - t_0)) * (u0 - numpy.cos(t_0)) + numpy.cos(t)

num_steps = 10
error = numpy.ones(num_steps) * 1.0
t_hat = numpy.linspace(0.0, 10.0, num_steps + 1)
t_whole = numpy.linspace(0.0, 10.0, 1000)

fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)
eta = 1.0
lam = 0.1

for n in range(1,num_steps):
    t = numpy.linspace(t_hat[n-1], t_hat[n], 100)
    U = u(t_hat[n-1], eta, lam, t)
    axes.plot(t, U, 'b')
    axes.plot(t_whole, u(t_hat[n-1], eta, lam, t_whole),'b--')
    axes.plot([t[-1], t[-1]], (U[-1], U[-1] + -1.0**n * error[n]), 'r')
    eta = U[-1] + -1.0**n * error[n]

t = numpy.linspace(0.0, 10.0, 100)
axes.plot(t, numpy.cos(t), 'g')

axes.set_title("Perturbed Solutions $\lambda = %s$" % lam)
axes.set_xlabel('$t$')
axes.set_ylabel('$u(t)$')
axes.set_ylim((-10,10))
plt.show()

```

### Example:  Chemical systems

Consider the transition of a chemical $A$ to a chemical $C$ through the process

$$A \overset{K_1}{\rightarrow} B \overset{K_2}{\rightarrow} C.$$

If we let

$$\vec{u} = \begin{bmatrix} [A] \\ [B] \\ [C] \end{bmatrix}$$

then we can model this simple chemical reaction with the system of ODEs

$$\frac{\text{d} \vec{u}}{\text{d} t} =
\begin{bmatrix}
    -K_1 & 0 & 0 \\
    K_1 & -K_2 & 0 \\
    0 & K_2 & 0
\end{bmatrix} \vec{u}$$

The solution of this system is of the form

$$u_j(t) = c_{j1} e^{-K_1 t} + c_{j2}e^{-K_2 t} + c_{j3}$$

```python
# Solve the chemical systems example using a Forward Euler scheme
# Problem parameters
K_1 = 3
K_2 = 1
N = 29
#K_1 = 30.0
#K_2 = 1.0

A = numpy.array([[-K_1, 0, 0], [K_1, -K_2, 0], [0, K_2, 0]])
f = lambda u: numpy.dot(A, u)
```

```python
t_span = [0, 2.]
u_0 = [2.5, 5.0, 2.0]
f = lambda t, u : A.dot(u)
t, U = euler(f, t_span, u_0, N)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t, U.transpose())
axes.legend(["[A]", "[B]", "[C]"])
axes.set_title("Chemical System")
axes.set_xlabel("$t$")
axes.set_title("$[A], [B], [C]$")
axes.set_ylim((0.0, 10.))
axes.set_xlim((0.0, 2.0))
axes.grid()

plt.show()
```

### What is stiffness?

In general a **stiff** ODE is one where $u'(t) \ll f'(t, u)$.  For linear dynamical systems like the chemical decay problem,  the **stiffness ratio**

$$\frac{\max_p |\lambda_p|}{\min_p |\lambda_p|}$$

can be used to characterize the stiffness of the system.  In our last example this ratio was $K_1 / K_2$ if $K_1 > K_2$.  As we increased this ratio we observed that the numerical method became unstable only a reduction in $\Delta t$ lead to stable solution again.  For explicit time step methods this is problematic as the reduction of the time step for only one of the species leads to very expensive evaluations.  For example, forward Euler has the stability criteria

$$|1 + \Delta t \lambda| < 1$$

where $\lambda$ will have to be the maximum eigenvalue of the system.

```python
# set the eigenvalues and time step
K_1 = 3.0
K_2 = 1.0
delta_t = 1.0
eigenvalues = [-K_1, -K_2]

```

```python
# Plot the region of absolute stability for Forward Euler
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

axes.fill(numpy.cos(t) - 1.0, numpy.sin(t), color=(255.0/255.0,145.0/255.0,0/255.0,1.0))
for lam in eigenvalues:
    print(lam * delta_t)
    axes.plot(lam * delta_t, 0.0, 'ko')
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
# axes.set_xlim((-3, 1))
axes.set_ylim((-2,2))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Forward Euler")
plt.show()
```

```python
# solve using backwards Euler
A = numpy.array([[-K_1, 0, 0], [K_1, -K_2, 0], [0, K_2, 0]])

N = 100
t = numpy.linspace(0.0, 2.0, N)
delta_t = t[1] - t[0]

U = numpy.empty((t.shape[0], 3))
U[0, :] = [2.5, 5.0, 2.0]

B = numpy.eye(3) - delta_t*A
for n in range(t.shape[0] - 1):
    # backwards Euler (originally wrong)
    #U[n+1, :] = U[n, :] / (1.0 - delta_t * f(U[n, :]))
    U[n+1, :] = numpy.linalg.solve(B, U[n])
```

```python

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t, U)
axes.legend(["[A]", "[B]", "[C]"])
axes.set_title("Chemical System")
axes.set_xlabel("$t$")
axes.set_title("$[A], [B], [C]$")
axes.set_ylim((0.0, 10.))
axes.set_xlim((0.0, 2.0))
axes.grid()

plt.show()
```

### A-Stability
What if we could expand the absolute stability region to encompass more of the left-half plane or even better, all of it.  A method that has this property is called **A-stable**.  We have already seen one example of this with backward Euler which as a stability region of

$$|1 - z| \geq 1$$

which covers the full left-half plane.  It turns out that for linear multi-step methods a theorem by Dahlquist proves that there are no LMMs that satisfies the A-stability criterion beyond second order (trapezoidal rule).  There are higher-order Runge-Kutta methods do however.

Perhaps this is too restrictive though.  Often large eigenvalues for systems (for instance coming from a PDE discretization for the heat equation) lie completely on the real line.  If the stability region can encompass as much of the real line as possible while leaving out the rest of the left-half plane we can possibly get a more efficient method.  There are a number of methods that can be constructed that have this property but are higher-order.

```python
# Plot the region of absolute stability for Backward Euler
fig = plt.figure()
axes = fig.add_subplot(1, 1, 1)

t = numpy.linspace(0.0, 2.0 * numpy.pi, 100)

K_1 = 3.0
K_2 = 1.0
delta_t = 1.0
eigenvalues = [-K_1, -K_2]

axes.set_facecolor((255.0/255.0,145.0/255.0,0/255.0,1.0))
axes.fill(numpy.cos(t) + 1.0, numpy.sin(t), 'w')
for lam in eigenvalues:
    print(lam * delta_t)
    axes.plot(lam * delta_t, 0.0, 'ko')
axes.plot([-3, 3],[0.0, 0.0],'k--')
axes.plot([0.0, 0.0],[-3, 3],'k--')
# axes.set_xlim((-3, 1))
axes.set_ylim((-2,2))
axes.set_aspect('equal')

axes.set_title("Absolute Stability Region for Backward Euler")
plt.show()
```

# L-Stability

It turns out not all A-stable methods are alike.  Consider the backward Euler method and the trapezoidal method defined by

$$\frac{U_{n+1} - U_n}{\Delta t} = \frac{1}{2}(f(U_n) + f(U_{n+1})$$

whose stability polynomial is

$$\begin{aligned}
    0 &= U_{n+1} - U_n - \Delta t \frac{1}{2} (\lambda U_n + \lambda U_{n+1}) \\
      &= U_{n+1}\left(1 - \frac{1}{2} \Delta t \lambda \right ) - U_n \left(1 + \frac{1}{2}\Delta t \lambda \right) \\
      &= \left(\xi - \frac{1 + \frac{1}{2}z}{1 - \frac{1}{2} z}\right) \left(1 - \frac{1}{2} z \right )\\
\end{aligned}$$
which shows that it is A-stable.  Lets apply both these methods to a problem we have seen before and see what happens.

```python
# Compare accuracy between Euler and Trapezoidal for the stiff ODE
f = lambda t, lam, u: lam * (u - numpy.cos(t)) - numpy.sin(t)
u_exact = lambda t_0, eta, lam, t: numpy.exp(lam * (t - t_0)) * (eta - numpy.cos(t_0)) + numpy.cos(t)

t_0 = 0.0
t_f = 1.0
eta = 2.0
lam = -1e6
```

```python
num_steps = [10, 20, 40, 50]
# num_steps = numpy.arange(100, 1000, 100)

delta_t = numpy.empty(len(num_steps))
error_euler = numpy.empty(len(num_steps))
error_trap = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(0, t_f, N)
    delta_t[i] = t[1] - t[0]
    u = u_exact(t_0, eta, lam, t_f)

    # Compute Euler solution
    U_euler = numpy.empty(t.shape)
    U_euler[0] = eta
    for (n, t_n) in enumerate(t[1:]):
        U_euler[n+1] = (U_euler[n] - lam * delta_t[i] * numpy.cos(t_n) - delta_t[i] * numpy.sin(t_n)) / (1.0 - lam * delta_t[i])
    error_euler[i] = numpy.abs(U_euler[-1] - u) / numpy.abs(u)

    # Compute using trapezoidal
    U_trap = numpy.empty(t.shape)
    U_trap[0] = eta
    for (n, t_n) in enumerate(t[1:]):
        U_trap[n+1] = (U_trap[n] + delta_t[i] * 0.5 * f(t_n, lam, U_trap[n]) - 0.5 * lam * delta_t[i] * numpy.cos(t_n) - 0.5 * delta_t[i] * numpy.sin(t_n)) / (1.0 - 0.5 * lam * delta_t[i])
    error_trap[i] = numpy.abs(U_trap[-1] - u)
```

```python

# Plot error vs. delta_t
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 2, 1)
axes.plot(t, U_euler, 'ro-')
axes.plot(t, u_exact(t_0, eta, lam, t),'k')

axes = fig.add_subplot(1, 2, 2)
axes.loglog(delta_t, error_euler, 'bo')
order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[1], error_euler[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")
axes.loglog(delta_t, order_C(delta_t[1], error_euler[1], 2.0) * delta_t**2.0, 'b--', label="2nd Order")

axes.legend(loc=4)
axes.set_title("Comparison of Errors for Backwards Euler")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

# Plots for trapezoid
fig = plt.figure()
fig.set_figwidth(fig.get_figwidth() * 2)
axes = fig.add_subplot(1, 2, 1)
axes.plot(t, U_trap, 'ro-')
axes.plot(t, u_exact(t_0, eta, lam, t),'k')

axes = fig.add_subplot(1, 2, 2)
axes.loglog(delta_t, error_trap, 'bo', label='Trapezoidal')
order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[1], error_trap[1], 1.0) * delta_t**1.0, 'r--', label="1st Order")
axes.loglog(delta_t, order_C(delta_t[1], error_trap[1], 2.0) * delta_t**2.0, 'b--', label="2nd Order")

axes.legend(loc=4)
axes.set_title("Comparison of Errors for Trapezoidal Rule")
axes.set_xlabel("$\Delta t$")
axes.set_ylabel("$|U(t_f) - u(t_f)|$")

plt.show()
```

It turns out that if we look at a one-step method and define the following ratio

$$U_{n+1} = R(z) U_n$$

we can define another form of stability, called **L-stable**, where we require that the method is A-stable and that

$$\lim_{z \rightarrow \infty} |R(z)| = 0.$$

Turns out that backwards Euler is L-stable while trapezoidal rule is not.

```python
R_beuler = lambda z: 1./(1. - z)
R_trap = lambda z: (1. + 0.5*z)/(1. - 0.5*z)
```

```python
fig = plt.figure(figsize=(12,6))
axes = fig.add_subplot(1,2,1)
plot_stability_ssms(R_beuler, x, y, axes=axes, title='Stability of Backwards Eulers')

axes = fig.add_subplot(1,2,2)
plot_stability_ssms(R_trap, x, y, axes=axes, title='Stability of Trapezoidal')
```

## Backward Differencing Formulas

A class of LMM methods that are useful for stiff ODE problems are the backward difference formula (BDF) methods which have the form

$$\alpha_0 U_n + \alpha_1 U_{n+1} + \cdots + \alpha_r U_{n+r} = \Delta \beta_r f(U_{n+r})$$

These methods can be derived directly from backwards finite differences from the point $U_{n+r}$ and the rest of the points back in time.  One can then derive r-step methods that are rth-order accurate this way.  Some of the methods are

$$\begin{aligned}
    r = 1:& & U_{n+1} - U_n = \Delta t f(U_{n+1}) \\
    r = 2:& &3 U_{n+2} - 4 U_{n+1} + U_n = 2 \Delta t f(U_{n+2}) \\
    r = 3:& &11U_{n+3} - 18U_{n+2} + 9U_{n+1} - 2 U_n = 6 \Delta t f(U_{n+3}) \\
    r = 4:& &25 U_{n+4} - 48 U_{n+3} +36 U_{n+2} -16 U_{n+1} +3 U_n = 12 \Delta t f(U_{n+4})
\end{aligned}$$

```python
pi_BDF1 = lambda z: numpy.array([ 1. - z, -1.])
pi_BDF2 = lambda z: numpy.array([ 3. - 2*z, -4. , 1.])
pi_BDF3 = lambda z: numpy.array([ 11 - 6.*z, -18., 9., -2.])
pi_BDF4 = lambda z: numpy.array([ 25. - 12*z, -48., 36, -16., 3])
```

```python
x = numpy.linspace(-10,10,50)
y = numpy.linspace(-10,10,50)
fig = plt.figure(figsize=(12,12))
axes = fig.add_subplot(2,2,1)
plot_stability_lmm(pi_BDF1, x, y, axes=axes, title='Stability of BDF1 method')

axes = fig.add_subplot(2,2,2)
plot_stability_lmm(pi_BDF2, x, y, axes=axes, title='Stability of BDF2 method')

axes = fig.add_subplot(2,2,3)
plot_stability_lmm(pi_BDF3, x, y, axes=axes, title='Stability of BDF3 method')

axes = fig.add_subplot(2,2,4)
plot_stability_lmm(pi_BDF4, x, y, axes=axes, title='Stability of BDF4 method')

```


```python
import sympy as sp
from pyodesys.native import native_sys
```

```python
x, y = sp.symbols('x y')
exprs = [sp.exp(y)*(3*y + 3*sp.sqrt(x+1)), sp.exp(y)*(5*y + 5*sp.sqrt(x+1))]
sp.cse(exprs)
```

```python
sp.cse(exprs, optimizations='basic')
```

```python
try:
    _ = sp.cse(exprs, ignore=(y,x))
except TypeError:
    print("Using an old version of SymPy (ignore not supported in cse())")
else:
    print(_)
```

```python
odesys = native_sys['cvode'].from_callback(lambda x, y, p, be: [-be.exp(p[0]+3)*y[0], be.exp(p[0]+3)*y[0]], 2, 1)
```

```python
path = next(filter(lambda x: x.endswith('.cpp'), odesys._native._written_files))
print(open(path).read())
```

```python
sp.cse(odesys.exprs)
```

```python

```


# Integrais de superficie

O cálculo de integrais de superficie é moi semellante ao cálculo feito para as integrais de liña, coa única diferenza de incorporar unha dimensión máis nos cálculos. Agora, o dominio de integración será unha porción dunha superficie no espazo tridimendional. Para realizar estes cálculos, seguiremos usando o módulo **Sympy**. Ademais, destes cálculos integrais, tamén revisaremos como representar graficamente unha superficie, estudo que xa foi realizado na *práctica 4*.

Nesta práctica volveremos a usar o módulo **Sympy** e **Matplotlib** tanto para calcular integrais como para representar as superficies que serán usadas como dominio de integración.

## Obxectivos

- Representación gráfica de superficies (**repaso**)
- Parametrización de superficies
- Cálculo de integrais de liña

# Superficies

Como xa fixemos en prácticas anteriores, para usar os módulos **Numpy**, **Matplotlib** e **Sympy**, o primeiro que debemos facer é importalos, e posto que usaremos este módulo usando a interface `pyplot` de **Matplotlib**, tamén haberá que importar esta.

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt
import sympy as sp
sp.init_printing(use_latex='mathjax')
%matplotlib inline
```

## Representación gráfica de superficies (repaso)
Un dos casos *particulares* de superficie no espazo tridimensional son aquelas superficies que se poden definir como:
$$
\text{Superficie}=\{(x,y,z)\in\mathbb{R}^3:\ z=f(x,y)\text{ con }(x,y)\in\mathcal{R}\},
$$
onde $\mathcal{R}$ é un recinto do plano $XY$ (que eventualmente podería ser todo $\mathbb{R}^2$).

Por exemplo, para representar o cilíndro $y^2+z^2=5^2$ no primeiro octante entre os planos $x=0$ e $x=4$ teríamos que primeiro despexar a coordenada $z$ como $z=+\sqrt{5^2-y^2}$ e polo tanto se representaría graficamente
$$
z=+\sqrt{5^2-y^2},\ (x,y)\in[0,4]\times[0,5]
$$
usando **Matplotlib**:

```python
# Definición da función en sympy
x,y,z = sp.symbols('x y z', real=True)
f = sp.Lambda((x,y),sp.sqrt(5**2-y**2))
f_np = sp.lambdify((x,y),f(x,y),"numpy")

N = 50  # 50 puntos en cada dirección dos eixes cartesianos
x = np.linspace(0.,4., N)
y = np.linspace(0., 5., N)

X, Y = np.meshgrid(x, y)
Z = f_np(X, Y)

from mpl_toolkits.mplot3d import axes3d
# Creación da figura 3D
fig = plt.figure()
ax = plt.axes(projection="3d")
# Representación da superficie z=f(x,y)
surf = ax.plot_surface(X, Y, Z)
# Etiquetas dos eixes
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
# Punto de vista do observador: elevación
ax.view_init(azim=45)
plt.show()
```

## Parametrización de superficies
De forma análoga a como se parametrizan curvas no espazo tridimensional, tamén se podne parametrizazr superficies tendo en conta que agora non soamente se da un único parámetro $t$ como nas curvas, senón que hai que dar dous novos parámetros $t$ e $s$:
$$
\text{Superficie}=\{(x,y,z)\in\mathbb{R}^3:\ x=\alpha(t,s),\ y=\beta(t,s),\ z=\gamma(t,s)\text{ con }t\in[t_0,t_1],\ s\in[s_0,s_1]\},
$$
onde $t$ e $s$ son os parámetros da representación da supercicie.

Por exemplo, para representar graficamente a sección do cilíndro antes descrita, poderíamos introducir como parámetros $t=\theta$ e $s=x$ onde $\theta$ é o ángulo formado polo vector posición do punto $(y,z)$ medido respecto ao eixo $y$. Deste xeito:
$$
x=f(t,s)=s,\qquad y=g(t,s)=5\cos(t),\qquad z=h(t,s)=5\sin(t)
$$
con $t\in[0,\pi/2]$ e $s\in[0,4]$. Para usar esta parametrización para realizar a representación gráfica, tamén usaríamos **Matplotlib**:

```python
# Definición da función en sympy
t, s = sp.symbols('t s', real=True)
alpha = sp.Lambda((t,s),s)
beta = sp.Lambda((t,s),5*sp.cos(t))
gamma = sp.Lambda((t,s),5*sp.sin(t))
alpha_np = sp.lambdify((t,s),alpha(t,s),"numpy")
beta_np = sp.lambdify((t,s),beta(t,s),"numpy")
gamma_np = sp.lambdify((t,s),gamma(t,s),"numpy")

N = 50  # 50 puntos en cada dirección dos eixes cartesianos
t = np.linspace(0., np.pi/2., N)
s = np.linspace(0., 4., N)

T, S = np.meshgrid(t, s)
X = alpha_np(T, S)
Y = beta_np(T, S)
Z = gamma_np(T, S)

from mpl_toolkits.mplot3d import axes3d
# Creación da figura 3D
fig = plt.figure()
ax = plt.axes(projection="3d")
# Representación da superficie parametrizada
surf = ax.plot_surface(X, Y, Z)
# Etiquetas dos eixes
ax.set_xlabel('x')
ax.set_ylabel('y')
ax.set_zlabel('z')
# Punto de vista do observador: elevación
ax.view_init(azim=45)
plt.show()
```

### Exercicio 12.1
Calcula con **Sympy** e representa graficamente con **Matplotlib** os vectores normais á porción de superficie do cilindro antes descrito usando:
- A descripción da superficie como $z=f(x,y)$
- A descripción paramétrica da superficie

```python
## O TEU CÓDIGO AQUÍ
```

# Cálculo de integrais de superficie

O cálculo dunha integral de superficie descrita por $z=f(x,y)$ non deixa de ser unha integral calculada sobre unha rexión $\mathcal{R}$ de dúas variables no plano $XY$, isto é,
$$
\int_{\mathcal{S}}F(x,y,z)dS=\int_{\mathcal{R}}F(x,y,f(x,y))||\vec{n}(x,y)||dxdy.
$$
onde $\vec{n}(x,y)$ é o vector normal oa superficie no punto $(x,y,f(x,y))\in\mathcal{S}$. Como xa estudamos no tema 3, o vector normal a unha superficie pode calcularse mediante o gradiente e resulta que
$$
||\vec{n}(x,y)||=\sqrt{1+\left(\frac{\partial f}{\partial x}(x,y)\right)^2+\left(\frac{\partial f}{\partial y}(x,y)\right)^2}.
$$

Vexamos cun exemplo como realizaríamos o cálculo da integral de superficie no caso de
$$
\int_{\mathcal{S}}(x+z)\,dS,
$$
onde $\mathcal{S}$ é a porción de cilíndro $y^2+z^2=5^2$ no primeiro octante e limitado polos planos $x=0$ e $x=4$.

```python
x, y, z = sp.symbols('x y z', real=True)
gradient = lambda f, v: sp.transpose(sp.Matrix([f]).jacobian(v))
F = sp.Lambda((x,y,z), y)
# Función que define a superficie como z=f(x,y)
f = sp.Lambda((x,y),sp.sqrt(y**2-5**2))
# Cálculo do vector normal
n = gradient(z-f(x,y),(x,y,z))
display('Módulo do vector normal:',n.norm())
n_norm = sp.sqrt(1+y**2/(y**2-25))
display('Módulo do vector normal simplificado:',n_norm)
# Calculo da integral de liña
I = sp.integrate(F(x,y,f(x,y))*n_norm,(x,0,4),(y,0,5))
display('Integral de superficie:', I)
```

    'Módulo do vector normal:'

$\displaystyle \sqrt{\left|{\frac{y}{\sqrt{y^{2} - 25}}}\right|^{2} + 1}$

    'Módulo do vector normal simplificado:'

$\displaystyle \sqrt{\frac{y^{2}}{y^{2} - 25} + 1}$

    'Integral de superficie:'

$\displaystyle 4 \int\limits_{0}^{5} y \sqrt{\frac{2 y^{2}}{y^{2} - 25} - \frac{25}{y^{2} - 25}}\, dy$

# Cálculo de integrais de superficie paramétricas

O cálculo dunha integral de superficie descrita de forma paramétrica por $x=\alpha(t,s)$, $y=\beta(t,s)$ e $z=\gamma(t,s)$ non deixa de ser unha integral calculada sobre unha rexión $(t,s)\in\mathcal{R}$ de dúas variables, isto é,
$$
\int_{\mathcal{S}}F(x,y,z)dS=\int_{\mathcal{R}}F(\alpha(t,s),\beta(t,s),\gamma(t,s))||\vec{n}(t,s)||dtds.
$$
onde $\vec{n}(t,s)$ é o vector normal oa superficie no punto $(\alpha(t,s),\beta(t,s),\gamma(t,s))\in\mathcal{S}$. Para calcular o vector normal a unha superficie, nos valemos do produto vectorial de dous vectores tanxentes á superficie
$$
\vec{\tau}_{t}(t,s)=
\displaystyle\begin{pmatrix}
\frac{\partial\alpha}{\partial t}\\
\frac{\partial\beta}{\partial t}\\
\frac{\partial\gamma}{\partial t}\\
\end{pmatrix},
\qquad
\vec{\tau}_{s}(t,s)=
\displaystyle\begin{pmatrix}
\frac{\partial\alpha}{\partial s}\\
\frac{\partial\beta}{\partial s}\\
\frac{\partial\gamma}{\partial s}\\
\end{pmatrix},
$$
co que resulta
$$
\vec{n}(t,s)=\vec{\tau}_{t}(t,s)\times\vec{\tau}_{s}(t,s).
$$

Vexamos cun exemplo como realizaríamos o cálculo da integral de superficie no caso de
$$
\int_{\mathcal{S}}(x+z)\,dS,
$$
onde $\mathcal{S}$ é a porción de cilíndro $y^2+z^2=5^2$ no primeiro octante e limitado polos planos $x=0$ e $x=4$.

```python
x, y, z, s, t = sp.symbols('x y z s t', real=True)
F = sp.Lambda((x,y,z), y)
# Función que define a superficie parametrica
S = sp.Lambda((t,s),sp.Matrix((s,5*sp.cos(t),5*sp.sin(t))))
# Cálculo do vector normal
tau_t = sp.diff(S(t,s),t)
tau_s = sp.diff(S(t,s),s)
n = tau_t.cross(tau_s)
display('Módulo do vector normal:',n.norm())
n_norm = sp.simplify(n.norm())
display('Módulo do vector normal simplificado:',n_norm)
# Calculo da integral de liña
I = sp.integrate(F(*S(t,s))*n_norm,(t,0,sp.pi/2),(s,0,4))
display('Integral de superficie:', I)
```

    'Módulo do vector normal:'

$$\sqrt{25 \sin^{2}{\left (t \right )} + 25 \cos^{2}{\left (t \right )}}$$

    'Módulo do vector normal simplificado:'

$$5$$

    'Integral de superficie:'

$$100$$

### Exercicio 12.2
Tendo en conta a porción do cilindro $\mathcal{S}$ descrita anteriormente, calcula a integral de superficie
$$
\int_{S}(x+z)dS,
$$
usando unha parametrización diferente á que aparece descrita nesta práctica.

```python
x, y, z, s, t = sp.symbols('x y z s t', real=True)
F = sp.Lambda((x,y,z), x + z)
# Función que define a superficie parametrica
S = sp.Lambda((t,s),sp.Matrix((2*s,5*sp.sin(3*t),5*sp.cos(3*t))))
# Cálculo do vector normal
tau_t = sp.diff(S(t,s),t)
tau_s = sp.diff(S(t,s),s)
n = tau_t.cross(tau_s)
display('Módulo do vector normal:',n.norm())
n_norm = sp.simplify(n.norm())
display('Módulo do vector normal simplificado:',n_norm)
# Calculo da integral de liña
I = sp.integrate(F(*S(t,s))*n_norm,(t,0,sp.pi/6),(s,0,2))
display('Integral de superficie:', I)
```

    'Módulo do vector normal:'

$$\sqrt{900 \sin^{2}{\left (3 t \right )} + 900 \cos^{2}{\left (3 t \right )}}$$

    'Módulo do vector normal simplificado:'

$$30$$

    'Integral de superficie:'

$$20 \pi + 100$$

```python

```


# Performing the Linear Noise Approximation (LNA) and conducting Approximate Bayesian Computation (ABC)

This notebook will guide the user through performing the linear noise approximation (LNA) on any system the user wishes, and through using the LNA when conducting ABC for parameter estimation.

## Toy Example: Stochastic three gene interaction network

Consider the Three Gene Example described in (Karlebach & Shamir, 2008) where Gene 1, 2 and 3 are represented by $g_1$, $g_2$ and $g_3$ respectively and $\textbf{g} = (g_1, g_2, g_3)^T$. The kinetic parameters of the system are represented through $k_*$. In this system, the levels of these genes vary and depend on each other, in a way that is determined by the equations below:

$$
\begin{align*}
\emptyset & \overset{\hphantom{k_{1}} \, k_{1,s} \, g_3 \hphantom{k}}{\longrightarrow}   g_1  \\
g_1 & \overset{\hphantom{k_{1,s}} \, k_{1,d} \,  \hphantom{k_{1,s}}}{\longrightarrow} \emptyset \\
g_1 & \overset{ \hphantom{k_{1}} \,k_{2,s} \, g_1 \hphantom{k_{1,s}}}{\longrightarrow} g_1 + g_2 \\
g_2 & \overset{\hphantom{k_{1,s}}\, k_{2,d} \, \hphantom{k_{1,s}}}{\longrightarrow} \emptyset \\
g_1 + g_2 & \overset{k_{3,s} \, k_{3,1} \, k_{3,2} }{\longrightarrow} g_1 + g_2 + g_3 \\
g_3 & \overset{ \hphantom{k_{1,s}} \, k_{3,d} \, \hphantom{k_{1,s}}}{\longrightarrow}  \emptyset
\end{align*}
$$

These equations are visualised in the below diagram:

The interactions between the genes can be seen: Gene 1 activates Gene 2 and Gene 3, Gene 2 activates Gene 3 and finally Gene 3 represses Gene 1 \[With the degradation of all the genes and the activation of Gene 1 from a constant source being supressed from the illustration\]. The reaction rates of this model are:

 $\textbf{a}= \begin{bmatrix}
      \frac{ k_{1,s}}{1+k_{1,3} \, \textbf{g}_3}, & k_{1,d} \, \textbf{g}_1, & \frac{k_{2,s} \, k_{2,1}\, \textbf{g}_1}{1+k_{2,1} \, \textbf{g}_1}, & k_{2,d} \, \textbf{g}_2, & \frac{k_{3,s} \, k_{3,1} \, \textbf{g}_1 \,  k_{3,2} \, \textbf{g}_2}{(1+k_{3,1} \, \textbf{g}_1)(1+k_{3,2} \, \textbf{g}_2)}, & k_{3,d} \, \textbf{g}_3
     \end{bmatrix} ^T$.

The Stoichiometry matrix of the system is:

$$\mathcal{S} =
 \begin{pmatrix}
  1 & -1 & 0 & 0 & 0 & 0 \\
  0 & 0 & 1 & -1 & 0 & 0 \\
  0 & 0 & 0 & 0 & 1 & -1
 \end{pmatrix} $$

In this example, the parameters $[k_{1,s}, k_{2,s}, k_{3,s}, k_{1,d}, k_{1,d}, k_{2,d}, k_{3,d}, k_{1,3}, k_{2,1}, k_{3,1}, k_{3,1}]$ have been chosen to take values $[2.0, 1.0, 15.0, 1.0, 1.0, 1.0, 100.0, 1.0, 1.0, 1.0]$ arbitrarily.

The user requires $\mathcal{S}$ and $\textbf{a}$ when constructing a stochastic model of a system. So, we construct $\mathcal{S}$ and $\textbf{a}$, where the reaction rates (reaction propensity vector), $\textbf{a}$, must be stated in a function.

### References

- Karlebach, G., & Shamir, R. (2008). Modelling and analysis of gene regulatory networks. Nature Reviews Molecular Cell Biology, 9(10), 770–780. https://doi.org/10.1038/nrm2503

# To run this notebook locally:
* Make sure that [Jupyter](https://jupyter.org/) is installed, along with [Julia](https://www.julialang.org/) and its [Jupyter Kernel](https://github.com/JuliaLang/IJulia.jl).
* Clone or download [GpABC.jl](https://github.com/tanhevg/GpABC.jl) to your machine.
* Run `jupyter` from the `examples` directory of the local copy of `GpABC.jl`:
```bash
$ cd GpABC.jl/examples
$ jupyter notebook lna-example.ipynb
```
* The first line of the cell below activates the local project environment and downloads all the dependencies.

```julia
import Pkg; Pkg.activate("."); Pkg.resolve()
using GpABC, OrdinaryDiffEq, Distances, Distributions, LinearAlgebra, StochasticDiffEq, Plots
pyplot()
```

```julia
#
# Stochastic Model
#
S = [1.0 -1.0  0.0  0.0  0.0  0.0;  # stoichometry matrix
     0.0  0.0  1.0 -1.0  0.0  0.0;
     0.0  0.0  0.0  0.0  1.0 -1.0]

params = [2.0, 1.0, 15.0, 1.0, 1.0, 1.0, 100.0, 1.0, 1.0, 1.0]  #parameters

reaction_rate_function = function(x,params)  #reaction rates of model (stated in a function)
    f = [params[1]/(1+params[7]*x[3]),
        params[4]*x[1],
        params[2]*params[8]*x[1]/(1+params[8]*x[1]),
        params[5]*x[2],
        params[3]*params[9]*x[1]*params[10]*x[2]/(1+params[9]*x[1])/(1+params[10]*x[2]),
        params[6]*x[3]]
    return f
end
```

    #3 (generic function with 1 method)

An Ordinary Differential Equation (ODE) model is given by the relationship $\dot{\textbf{g}} = \mathcal{S} \, \textbf{a} $.

So we define the ODE, `GeneReg`, by: $\mathcal{S} \, \textbf{a}$. (Giving the same example used in [`abc-examples.ipynb`](https://github.com/tanhevg/GpABC.jl/blob/master/examples/abc-example.ipynb)).

The initial conditions, timspan, solver and saveat are arbitrary and of the same form needed to pass to an [`OrdinaryDiffEq.jl`](https://github.com/JuliaDiffEq/OrdinaryDiffEq.jl) DE solver.

```julia
#
# ODE
#
GeneReg = function(params::AbstractArray{Float64,1},
    Tspan::Tuple{Float64,Float64}, x0::AbstractArray{Float64,1},
    solver::OrdinaryDiffEq.OrdinaryDiffEqAlgorithm, saveat::Float64)

  if size(params,1) != 10
    throw(ArgumentError("GeneReg needs 10 parameters, $(size(params,1)) were provided"))
  end

  function ODE_3GeneReg(dx, x, par, t)
    dx[1:3] = S*reaction_rate_function(x, par)
  end

  prob = ODEProblem(ODE_3GeneReg, x0 ,Tspan, params)
  Obs = solve(prob, solver, saveat=saveat)

  return Obs
end

#
# ODE settings
#
Tspan = (0., 10.) #timspan
x0 = [3.0, 2.0, 1.0] #initial conditions
solver = RK4()
saveat = 0.1

#
# Solve and plot the ODE
#
ODE_solution = GeneReg(params, Tspan, x0, solver, saveat)
plot(ODE_solution)
```

## Stochastic Example

Before calculating the LNA, let's view the stochastic simulation of `GeneReg`:

```julia
#
# SDE
#
StochGeneReg = function(params::AbstractArray{Float64,1},
    Tspan::Tuple{Float64,Float64}, x0::AbstractArray{Float64,1},
    solver::StochasticDiffEq.StochasticDiffEqAlgorithm, dt::Float64)

  if size(params,1) != 10
    throw(ArgumentError("GeneReg needs 10 parameters, $(size(params,1)) were provided"))
  end

  function sigma_3GeneReg(dx, x, par, t)
    dx[1] = 0.3
    dx[2] = 0.3
    dx[3] = 0.3
  end

  function ODE_3GeneReg(dx, x, par, t)
    dx[1:3] = S*reaction_rate_function(x, par)
  end

  prob = SDEProblem(ODE_3GeneReg, sigma_3GeneReg, x0 ,Tspan, params)
  Obs = solve(prob, solver, dt=dt)

  return Obs
end

#
# SDE settings
#
solver = EM()
dt = 0.1

#
# Solve and plot the SDE
#
SDE_solution = StochGeneReg(params, Tspan, x0, solver, saveat)
plot(SDE_solution)
```

## Computing the LNA

Now, if the user wishes to compute the LNA of the above stochastic problem, the process begins by setting up an [`LNAInput`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.LNAInput) structure:

```julia
volume = 100.0
input = LNAInput(params, S, reaction_rate_function, volume);
```

where `volume` is a parameter. The `volume` can be viewed as the volume that the reactions are taking place in, or molecule number. For example, if `volume` becomes very large the LNA will be very close to the deterministic equations of the system.

***
The [LNA](https://tanhevg.github.io/GpABC.jl/latest/overview-lna/) is the solution to the two ODEs, where $\Omega$ is the volume:

$$
\begin{align}
\frac{d\phi}{dt}& =\mathcal{S}\textbf{a}(\phi) \label{mean} \\
\frac{d\Sigma}{dt}&=\mathcal{A} \, \Sigma + \Sigma \, \mathcal{A}^T + \frac{1}{\sqrt{\Omega}} \, \mathcal{S} \, \text{diag}(\textbf{a}(\phi)) \, \mathcal{S}^T \label{covar}
\end{align}
$$
where, $\mathcal{A} = \mathcal{S} \cdot \mathcal{J}$ ($\mathcal{J}$ is the Jacobian), $\phi$(t) are the mean trajectories and $\Sigma$(t) is the covariance of the trajectories.

If the user wishes to see the results from the two ODEs, the user must provide the initial condtions of the ODEs in the form of (initial conditions of the species, initial covariance) and then call [`compute_LNA`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.compute_LNA).

This returns a [`LNA`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.LNA) structure that holds the solutions to the above ODEs. The means, $\phi$ can be called by `lna.traj_means` and the covariance $\Sigma$ can be called by `lna.traj_covars`. The time points the system of ODEs were solved for can be called by `lna.time_points`.

### References

- Komorowski, M., Finkenstädt, B., Harper, C.V., and Rand, D.A. (2009). Bayesian inference of biochemical kinetic parameters using the linear noise approximation. BMC Bioinformatics, 10:343. https://doi.org/10.1186/1471-2105-10-343

- Schnoerr, D., Sanguinetti, G., and Grima, R. (2017). Approximation and inference methods for stochastic biochemical kinetics—a tutorial review. Journal of Physics A: Mathematical and Theoretical, 50(9), 093001. https://doi.org/10.1088/1751-8121/aa54d9

```julia
x0 = ([3.0, 2.0, 1.0], Matrix(1.0I, 3,3)) #initial conditions
                                           #...in the form of
                                           #...(initial conditions of the species, initial covariance)
solver = RK4()
saveat = 0.1

lna = compute_LNA(input, x0, Tspan, saveat)
t = lna.time_points

#
# Calculate the variance of the trajectories for visualisation
#
variance = zeros(size(S)[1], length(t))
for (idx, elt) in enumerate(lna.traj_covars)
    variance[:, idx] = diag(elt)
end

#
# Plot the trajectories and confidence intervals
#
plot(t, lna.traj_means[1,:], ribbon=1.96*sqrt.(variance[1,:]), c=:blue, leg=false)
plot!(t, lna.traj_means[2,:], ribbon=1.96*sqrt.(variance[2,:]), c=:red, leg=false)
plot!(t, lna.traj_means[3,:], ribbon=1.96*sqrt.(variance[3,:]), c=:green, leg=false)
```

The three means are plotted with the shaded areas being the 95% confidence intervals.

## Sampling from the LNA

The LNA gives the mean of the trajectories and the covariance between them; hence a single trajectory can be sampled from a Multivariate Normal distribution, i.e. $\textbf{x}(t) \sim \mathcal{N}(\phi(t), \Sigma(t))$. The function [`sample_LNA_trajectories`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.sample_LNA_trajectories-Tuple{LNA,Int64}) automatically does this.

The user can also sample more than one trajectory, which are then averaged, by changing `n_samples`. The larger the number of samples that are averaged over, the less noisy the outputs are. The effect of varying `n_samples` is shown below:

```julia
n_samples = 1

@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)

plot(t, lna_trajectories[1,:])
plot!(t, lna_trajectories[2,:])
plot!(t, lna_trajectories[3,:])
```

```julia
n_samples = 10

@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)

plot(t, lna_trajectories[1,:])
plot!(t, lna_trajectories[2,:])
plot!(t, lna_trajectories[3,:])
```

```julia
n_samples = 100

@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)

plot(t, lna_trajectories[1,:])
plot!(t, lna_trajectories[2,:])
plot!(t, lna_trajectories[3,:])
```

## ABC using the LNA

The same examples that can be located in [`abc-examples.ipynb`](https://github.com/tanhevg/GpABC.jl/blob/master/examples/abc-example.ipynb) are shown, but using the LNA. The only difference is in the `simulator_function`.

Let's try and estimate the first three parameters of the model and compare simulation based ABC and emulation based ABC.

### Simulation-Based Rejection ABC

So, lets define the `simulator_function`. We need to create a function to be passed to ABC algortihms that takes a vector of the parameters we wish to estimate as an argument and outputs model results (as stated in the [documentation](https://tanhevg.github.io/GpABC.jl/latest/ref-abc/#abc_ref-1)).

So we define a function to create a `LNAInput` object and then call the function [`get_LNA_trajectories`](https://tanhevg.github.io/GpABC.jl/latest/ref-lna/#GpABC.get_LNA_trajectories) on this object. The output of `get_LNA_trajectories` conforms to the desired output of the `simulator_function`.

```julia
function simulator_function(var_params)
    input = LNAInput(vcat(var_params, params[n_var_params+1:end]), S, reaction_rate_function, volume)
    return get_LNA_trajectories(input, n_samples, x0, Tspan, saveat)
end
```

    simulator_function (generic function with 1 method)

Let's take the reference data to be `lna_trajectories` for the purpose of this example, calculated above using the following:
```
@time lna_trajectories = sample_LNA_trajectories(lna, n_samples)
```
This provides some fake data where we already know the nomimal parameters.

```julia
#
# SimulatedABCRejection arguments
#
priors = [Uniform(0., 5.), Uniform(0., 5.), Uniform(10., 20.)] #priors for the unknown parameters
n_var_params = length(priors) #no. of parameters to vary
n_particles = 500
threshold = 2.0
progress_every = 100
reference_data = lna_trajectories

sim_result = SimulatedABCRejection(reference_data,
                        simulator_function,
                        priors,
                        threshold,
                        n_particles;
                        write_progress = false
                        )

plot(sim_result)
```

Parameter 1, 2 and 3's nominal values were 2.0, 1.0 and 15.0; which haven't been uniquely returned following simulated-based ABC. However, a positive correlation between Parameter 2 and 3 has been recovered.

The above becomes our "gold-standard" that we should try to reproduce when conducting emulation-based rejection ABC.

### Emulation-Based Rejection ABC

The only new argument to sepcify is `n_design_points`; the number of design points (explained in [`abc-example.ipynb`](https://github.com/tanhevg/GpABC.jl/blob/master/examples/abc-example.ipynb)), when calling `EmulatedABCRejection`.

```julia
n_design_points = 200

emu_result = EmulatedABCRejection(reference_data,
                        simulator_function,
                        priors,
                        threshold,
                        n_particles,
                        n_design_points;
                        write_progress = false)

plot(emu_result)
```

The same information about Parameters 1, 2 and 3 can be visually deduced.

To show that the above still holds for [Simulation-Based](https://tanhevg.github.io/GpABC.jl/latest/overview-abc/#Simulation-based-ABC-SMC-1) and [Emulation Based](https://tanhevg.github.io/GpABC.jl/latest/overview-abc/#Emulation-based-ABC-SMC-1) ABC-SMC using the LNA, the same example as above is conducted below:

### Simulation-Based ABC-SMC

```julia
threshold_schedule = [4.0, 3.0, 2.0]

sim_abcsmc_res = SimulatedABCSMC(reference_data,
                        simulator_function,
                        priors,
                        threshold_schedule,
                        n_particles;
                        write_progress = false
                        )

population_colors=["#FF2F4E", "#A20018", "#800013"]
plot(sim_abcsmc_res, population_colors=population_colors)
```

### Emulation-Based ABC-SMC

```julia
emu_abcsmc_res = EmulatedABCSMC(reference_data,
                        simulator_function,
                        priors,
                        threshold_schedule,
                        n_particles,
                        n_design_points;
                        write_progress = false
                        )

plot(emu_abcsmc_res, population_colors=population_colors)
```

Again, the results are similar for emulation and simulation.


# 熟悉Jupyter notebook

* python 环境与版本
* 几个重要的工具：numpy, scipy, sklearn, pandas, keras, tensorflow
* 安装 tensorflow 与 keras
* jupyter notebook基本使用
* markdown的用法

## 1. python 环境与版本

python 语言是一种很简单易学的脚本语言。脚本语言就是不用编译的语言。C语言需要经历“编写、编译、链接、运行”（edit-compile-link-run）整个过程，而脚本语言只用“解释、运行”即可，不用“编译、链接”。

```python
print('Lalala')
```

    Lalala

经过广大开源社区大牛的不断开发与维护，python已经从2.7版本升级到3.6版本。

```python
print 'Hello'  # 2.7风格，但在3.6里面就不能用
```

```python
print('Hello') # 3.6风格，在2.7里面也能用
```

    Hello

众多的版本（解释器的版本）以及其配套的工具包与安装地址，可能造成了使用起来的混乱。比如说你在命令行里面给python3.6安装了numpy，但在python2.7里面想调用python3.6的numpy就不行，需要对应起来使用。Anaconda就提供了一个比较好的环境，将不同版本的python分开使用，这也是为什么我们选择用anaconda来作为讲解的工具。这里我们花点时间讲一讲ananconda的使用以及python环境与版本的管理。

在选择最新版本的Anaconda时，请选择python3.6版本。现在众多工具包已经宣布停止对python2.7版本的维护和升级，转向python3.6。我们也需要与时俱进。

```python

```

```python

```

```python

```

```python

```

```python

```

* 安装完成后，在命令行中敲击 jupyter notebook，就可以进入notebook的使用界面。

* 如何添加安装多个python版本,如何管理？

可以通过anaconda安装不同python环境，进行管理

方法：在命令行中执行以下代码 (相关阅读材料https://ipython.readthedocs.io/en/latest/install/kernel_install.html)

> conda create -n py27 python=2.7

Mac用户使用
> source activate py27

WINDOWS用户使用
> activate py27

激活python27环境

* 确认以下python的版本

* 如何加到可选的Kernel中？

> conda create -n env_name python=2.7

> activate env_name

mac user > source activate env_name

windows user > activate env_name

> pip install ipykernel

> sudo pip install ipykernel

> python -m ipykernel install --name Py27wanmen

my kernel space address is:

C:\ProgramData\jupyter\kernels\py27wanmen

关闭 python27环境

windows user>deactivate

mac user > source deactivate

查看已有kernel
> jupyter kernelspec list

分清python环境为什么重要？

* 避免**语法版本不一引起的错误**
* 避免**工具包安装与调用的混乱**

```python
print "?"   # py.27
```

    ?

```python
print("lalalallal")   # py.27 py.36
```

    lalalallal

### 安装决策树可视化工具 Graphviz

下载链接：https://graphviz.gitlab.io/download/
1. 下载并安装
2. 在windows下调整PATH(Win10, Anaconda3, Jupyter notebook) after "conda install graphviz" I have to add to the PATH: C:\Users\username\Anaconda3\Library\bin\graphviz

To modify PATH goto Control Panel > System and Security > System > Advanced System Settings > Environment Variables > Path > Edit > New

```python
import graphviz
```

```python
import sklearn.datasets as datasets
import pandas as pd
iris=datasets.load_iris()
df=pd.DataFrame(iris.data, columns=iris.feature_names)
y=iris.target
```

```python
from sklearn.tree import DecisionTreeClassifier
dtree=DecisionTreeClassifier()
dtree.fit(df,y)
```

    DecisionTreeClassifier(class_weight=None, criterion='gini', max_depth=None,
                max_features=None, max_leaf_nodes=None,
                min_impurity_decrease=0.0, min_impurity_split=None,
                min_samples_leaf=1, min_samples_split=2,
                min_weight_fraction_leaf=0.0, presort=False, random_state=None,
                splitter='best')

```python
from sklearn.externals.six import StringIO
from IPython.display import Image
from sklearn.tree import export_graphviz
import pydotplus
dot_data = StringIO()
export_graphviz(dtree, out_file=dot_data,
                filled=True, rounded=True,
                special_characters=True)
graph = pydotplus.graph_from_dot_data(dot_data.getvalue())
Image(graph.create_png())

```

```python

```

## 2. 几个重要的工具

> import numpy as np

> import scipy

> import pandas as pd

> import sklearn

> import keras.backend as K

> import tensorflow as tf

### numpy
numpy定义了python进行矩阵数值计算的基础
> np.add(A,B)

> np.subtract(A,B)

> np.dot(C,D)

> np.matmul(C,D)           ## C@D

> np.multiply(C,D)         ## C*D

```python
import numpy as np
A = np.array([1,2,3])
B = np.array([4,3,0])
print('A+B = ',np.add(A,B))
print('A-B = ',np.subtract(A,B))
```

    A+B =  [5 5 3]
    A-B =  [-3 -1  3]

```python
C = np.array([[1,2,3],
            [1,1,1]])
D = np.array([[4,3,9,9],
            [1,1,2,2,],
            [4,2,1,1]])
print('np.dot(C,D) = ',np.dot(C,D))
print('np.matmul(C,D) = ',np.matmul(C,D))
print('        C@D    = ',np.matmul(C,D))

print('np.multiply(D,D)=',np.multiply(D,D))
print('          D*D   = ', D*D)
```

    np.dot(C,D) =  [[18 11 16 16]
     [ 9  6 12 12]]
    np.matmul(C,D) =  [[18 11 16 16]
     [ 9  6 12 12]]
            C@D    =  [[18 11 16 16]
     [ 9  6 12 12]]
    np.multiply(D,D)= [[16  9 81 81]
     [ 1  1  4  4]
     [16  4  1  1]]
              D*D   =  [[16  9 81 81]
     [ 1  1  4  4]
     [16  4  1  1]]

```python

```

```python

```

```python

```

```python

```

### 重点看看 np.dot 和 np.matmul的区别

* 都是矩阵乘法
* np.matmul中，禁止矩阵与标量的乘法
* np.matmul中，在矢量乘矢量的内积运算中，matmul与dot没有差别
* np.matmul中，多维的矩阵，将前n-2维视为后2维的元素后，进行乘法操作

#### 矢量内积无区别，结果是标量

```python
t1 = np.array([1,2,3])
t2 = np.array([1,2,3])
print(np.dot(t1,t2))
print(np.matmul(t1,t2))
print(t1*t2) # 元素1 1 乘积
```

    14
    14
    [1 4 9]

#### 1维与2维矩阵的乘法

```python
a = np.array([1,2,3])
b = np.array([[1,1,1],
             [2,2,2],
             [3,3,3]])

```

```python
np.dot(a,b)
```

    array([14, 14, 14])

```python
np.matmul(a,b)
```

    array([14, 14, 14])

```python
#注意
a*b # 扩展维度后，元素乘积
```

    array([[1, 2, 3],
           [2, 4, 6],
           [3, 6, 9]])

```python

```

#### 多维矩阵乘积的比较

```python
m_4_4 = np.array([
                       [1,2,3,4],
                       [3,2,1,4],
                       [5,4,6,7],
                       [11,12,13,14]
                      ])

m_3_4_2 = np.array([
                             [[2,3],
                              [11,9],
                              [32,21],
                              [28,17]],

                             [[2,3],
                              [1,9],
                              [3,21],
                              [28,7]],

                             [[2,3],
                              [1,9],
                              [3,21],
                              [28,7]]
                            ])
```

```python
print(m_4_4.shape)
print(m_3_4_2.shape)

```

    (4, 4)
    (3, 4, 2)

```python
print('4x4*3x4x2 dot:\n {}\n'.format(np.dot(m_4_4,m_3_4_2)))
print('4x4*3x4x2 matmul:\n {}\n'.format(np.matmul(m_4_4,m_3_4_2)))
```

    4x4*3x4x2 dot:
     [[[232 152]
      [125 112]
      [125 112]]

     [[172 116]
      [123  76]
      [123  76]]

     [[442 296]
      [228 226]
      [228 226]]

     [[962 652]
      [465 512]
      [465 512]]]

    4x4*3x4x2 matmul:
     [[[232 152]
      [172 116]
      [442 296]
      [962 652]]

     [[125 112]
      [123  76]
      [228 226]
      [465 512]]

     [[125 112]
      [123  76]
      [228 226]
      [465 512]]]

```python
print('4 4 矩阵 与')
print('3 4 2 矩阵的乘积结果')
print('dot 结果的维度    =',np.dot(m_4_4,m_3_4_2).shape)
print('matmul 结果的维度 =',np.matmul(m_4_4,m_3_4_2).shape)

```

    4 4 矩阵 与
    3 4 2 矩阵的乘积结果
    dot 结果的维度    = (4, 3, 2)
    matmul 结果的维度 = (3, 4, 2)

**检验以下m_4_4与m_3_4_2最后两维的矩阵乘积结果**

```python
np.dot(m_4_4,m_3_4_2[0,:,:])
```

    array([[232, 152],
           [172, 116],
           [442, 296],
           [962, 652]])

```python
np.matmul(m_4_4,m_3_4_2[:,:,0].T)
```

    array([[232, 125, 125],
           [172, 123, 123],
           [442, 228, 228],
           [962, 465, 465]])

#### np.dot 定义：

> dot(a, b)[i,j,k,m] = sum(a[i,j,:] * b[k,:,m])

> import numpy as np

> **import scipy** 科学计算工具

> **import pandas as pd ** 大杀器

> **import sklearn ** 本课程前两周重点

> import keras.backend as K

> import tensorflow as tf

```python
import sklearn
```

```python
from sklearn import datasets
from sklearn.metrics import auc
from sklearn.model_selection import train_test_split
from sklearn.model_selection import cross_val_score
from sklearn.linear_model import LinearRegression
from sklearn.linear_model import Lasso
```

## 3. 安装Tensorflow 与 Keras

```python
import tensorflow as tf
```

理论上有很多种安装方式，这里我们用与anaconda相关的方式来安装，以方便以后的使用和管理。

> conda create -n tensorflow python=3.6

> activate tensorflow

> pip install --ignor_installed --upgrade tensorflow

## 4. Jupyter notebook的基本使用技巧

以下所有操作都需要在命令模式下完成（按ESC或者点击编辑框外的空白处）

* 向上增加空白cell, A
* 向下增加空白cell, B
* 运行cell，并将光标移动到下一个cell，shift+enter
* 运行cell，ctrl+enter
* 删除本cell，DD
* 剪切本cell, X
* 粘贴本cell, V

* 合并两个cell,A
* 将cell转为代码状态，Y
* 将cell转为markdown状态，M
* 将选中的几个cells合并，Shift+M
* 打开／关闭行号，L
* 恢复删除的最后一个cell，Z  只有最后一个啊
* 选中代码整体左移／右移, ctrl+[或者ctrl+]

```python

```

```python

```

```python

```

```python

```

```python

```

## 5. Markdown的基本技巧

[链接在这里哟](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

[小抄](https://guides.github.com/pdfs/markdown-cheatsheet-online.pdf)

```python

```

## 标题

#
##
###
####

```python

```

如何怎样怎么样**划重点**

```python

```

如何怎样怎么样*画斜体*

```python

```

引用一段话

> 于是我就念了一句诗

```python

```

* 列出几个观点
* 不带序号
    * 子观点

```python

```

1. 列出我的观点
2. 这次带序号
    * 子观点

```python

```

插入数学符号

$e^{i\pi}+1 = 0$

$ \sum_{\forall i}{x_i^{2}} $

\begin{align}
\dot{x} & = \sigma(y-x) \\
\dot{y} & = \rho x - y - xz \\
\dot{z} & = -\beta z + xy
\end{align}

\begin{equation*}
\left( \sum_{k=1}^n a_k b_k \right)^2 \leq \left( \sum_{k=1}^n a_k^2 \right) \left( \sum_{k=1}^n b_k^2 \right)
\end{equation*}

[更多小抄](https://reu.dimacs.rutgers.edu/Symbols.pdf)

[更多小抄2](http://jupyter-notebook.readthedocs.io/en/stable/examples/Notebook/Typesetting%20Equations.html)

```python

```

[引用外部链接](https://github.com/adam-p/markdown-here/wiki/Markdown-Cheatsheet)

```python

```

插入图片

```python

```

```python

```

```python

```

还有呢

```python

```

```python
from IPython.display import Image
```

```python
Image(filename='./spacex.jpg',width=200)
```

插入表格

| 第一栏        | 第二栏          | 第三栏  |
| ------------- |:-------------:| -----:|
| col 3 is      | right-aligned | 1600 |
| col 2 is      | centered      |   12 |
| zebra stripes | are neat      |    1 |

```python

```

```python

```

```python

```

```python

```


# 02 - Reverse Time Migration

This notebook is the second in a series of tutorial highlighting various aspects of seismic inversion based on Devito operators. In this second example we aim to highlight the core ideas behind seismic inversion, where we create an image of the subsurface from field recorded data. This tutorial follows on the modelling tutorial and will reuse the modelling operator and velocity model.

## Imaging requirement

Seismic imaging relies on two known parameters:

- **Field data** - or also called **recorded data**. This is a shot record corresponding to the true velocity model. In practice this data is acquired as described in the first tutorial. In order to simplify this tutorial we will generate synthetic field data by modelling it with the **true velocity model**.

- **Background velocity model**. This is a velocity model that has been obtained by processing and inverting the field data. We will look at this methods in the following tutorial as it relies on the method we are describing here. This velocity model is usually a **smooth version** of the true velocity model.

## Imaging computational setup

In this tutorial, we will introduce the back-propagation operator. This operator simulates the adjoint wave-equation, that is a wave-equation solved in a reversed time order. This time reversal led to the naming of the method we present here, called Reverse Time Migration. The notion of adjoint in exploration geophysics is fundamental as most of the wave-equation based imaging and inversion methods rely on adjoint based optimization methods.

## Notes on the operators

As we already describe the creation of a forward modelling operator, we will use a thin wrapper function instead. This wrapper is provided by a utility class called `AcousticWaveSolver`, which provides all the necessary operators for seismic modeling, imaging and inversion. The `AcousticWaveSolver` provides a more concise API for common wave propagation operators and caches the the Devito `Operator` objects to avoid unnecessary recompilation. However, any newly introduced operators will be fully described and only used from the wrapper in the next tutorials.

As before we initialize printing and import some utilities. We also raise the Devito log level to avoid excessive logging for repeated operator invocations.

```python
import numpy as np
%matplotlib inline

from devito import configuration
configuration['log_level'] = 'WARNING'
```

## Computational considerations

Seismic inversion algorithms are generally very computationally demanding and require a large amount of memory to store the forward wavefield. In order to keep this tutorial as light-weight as possible we are using a very simple
velocity model that requires low temporal and special resolution. For a more realistic model, a second set of preset parameters for a reduced version of the 2D Marmousi data set [1] is provided below in comments. This can be run to create some more realistic subsurface images. However, this second present is more computationally demanding and requires a slightly more powerful workstation.

```python
# Configure model presets
from examples.seismic import demo_model

# Enable model presets here:
preset = 'twolayer-isotropic'  # A simple but cheap model (recommended)
# preset = 'marmousi2d'  # A larger more realistic model

# Standard preset with a simple two-layer model
if preset == 'twolayer-isotropic':
    def create_model():
        return demo_model('twolayer-isotropic', origin=(0., 0.), shape=(101, 101),
                          spacing=(10., 10.), nbpml=20)
    filter_sigma = (1, 1)
    nshots = 21
    nreceivers = 101
    t0 = 0.
    tn = 1000.  # Simulation last 1 second (1000 ms)
    f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)

# A more computationally demanding preset based on the 2D Marmousi model
if preset == 'marmousi2d-isotropic':
    def create_model():
        return demo_model('marmousi2d-isotropic', data_path='../../../../opesci-data/')
    filter_sigma = (6, 6)
    nshots = 301  # Need good covergae in shots, one every two grid points
    nreceivers = 601  # One recevier every grid point
    t0 = 0.
    tn = 3500.  # Simulation last 3.5 second (3500 ms)
    f0 = 0.025  # Source peak frequency is 25Hz (0.025 kHz)
```

# True and smooth velocity models

First, we create the model data for the "true" model from a given demonstration preset. This model represents the subsurface topology for the purposes of this example and we will later use it to generate our synthetic data readings. We also generate a second model and apply a smoothing filter to it, which represents our initial model for the imaging algorithm. The perturbation between these two models can be thought of as the image we are trying to recover.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_velocity, plot_perturbation
from scipy import ndimage

# Create true model from a preset
model = create_model()

# Create initial model and smooth the boundaries
model0 = create_model()
model0.vp = ndimage.gaussian_filter(model0.vp, sigma=filter_sigma, order=0)

# Plot the true and initial model and the perturbation between them
plot_velocity(model)
plot_velocity(model0)
plot_perturbation(model0, model)
```

## Acquisition geometry

Next we define the positioning and the wave signal of our source and the location of our receivers. To generate the wavelet for our source we require the discretized values of time that we are going to use to model a single "shot",,
which again depends on the grid spacing used in our model. For consistency this initial setup will look exactly as in the previous modelling tutorial, although we will vary the position of our source later on during the actual imaging algorithm.

```python
#NBVAL_IGNORE_OUTPUT
# Define acquisition geometry: source
from examples.seismic import TimeAxis, RickerSource

# Define time discretization according to grid spacing
dt = model.critical_dt  # Time step from model grid spacing
time_range = TimeAxis(start=t0, stop=tn, step=dt)

src = RickerSource(name='src', grid=model.grid, f0=f0, time_range=time_range)

# First, position source centrally in all dimensions, then set depth
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 20.  # Depth is 20m

# We can plot the time signature to see the wavelet
src.show()
```

```python
# Define acquisition geometry: receivers
from examples.seismic import Receiver

# Initialize receivers for synthetic and imaging data
rec = Receiver(name='rec', grid=model.grid, npoint=nreceivers, time_range=time_range)
rec.coordinates.data[:, 0] = np.linspace(0, model.domain_size[0], num=nreceivers)
rec.coordinates.data[:, 1] = 30.
```

# True and smooth data

We can now generate the shot record (receiver readings) corresponding to our true and initial models. The difference between these two records will be the basis of the imaging procedure.

For this purpose we will use the same forward modelling operator that was introduced in the previous tutorial, provided by the `WaveSolver` utility class. This object instantiates a set of pre-defined operators according to an initial definition of the acquisition geometry, consisting of source and receiver symbols. The solver objects caches the individual operators and provides a slightly more high-level API that allows us to invoke the modelling modelling operators from the initial tutorial in a single line. In the following cells we use this to generate shot data by only specifying the respective model symbol `m` to use, and the solver will create and return a new `Receiver` object the represents the readings at the previously defined receiver coordinates.

```python
# Compute synthetic data with forward operator
from examples.seismic.acoustic import AcousticWaveSolver

solver = AcousticWaveSolver(model, src, rec, space_order=4)
true_d , _, _ = solver.forward(src=src, m=model.m)
```

```python
# Compute initial data with forward operator
smooth_d, _, _ = solver.forward(src=src, m=model0.m)
```

```python
#NBVAL_IGNORE_OUTPUT
# Plot shot record for true and smooth velocity model and the difference
from examples.seismic import plot_shotrecord

plot_shotrecord(true_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data, model, t0, tn)
plot_shotrecord(smooth_d.data - true_d.data, model, t0, tn)
```

# Imaging with back-propagation

As we explained in the introduction of this tutorial, this method is based on back-propagation.

## Adjoint wave equation

If we go back to the modelling part, we can rewrite the simulation as a linear system solve:

\begin{equation}
\mathbf{A}(\mathbf{m}) \mathbf{u} = \mathbf{q}
\end{equation}

where $\mathbf{m}$ is the discretized square slowness, $\mathbf{q}$ is the discretized source and $\mathbf{A}(\mathbf{m})$ is the discretized wave-equation. The discretized wave-equation matricial representation is a lower triangular matrix that can be solve with forward substitution. The pointwise writing or the forward substitution leads to the time-stepping stencil.

On a small problem one could form the matrix explicitly and transpose it to obtain the adjoint discrete wave-equation:

\begin{equation}
\mathbf{A}(\mathbf{m})^T \mathbf{v} = \delta \mathbf{d}
\end{equation}

where $\mathbf{v}$ is the discrete **adjoint wavefield** and  $\delta \mathbf{d}$ is the data residual defined as the difference between the field/observed data and the synthetic data $\mathbf{d}_s = \mathbf{P}_r \mathbf{u}$. In our case we derive the discrete adjoint wave-equation from the discrete forward wave-equation to get its stencil.

## Imaging

Wave-equation based imaging relies on one simple concept:

- If the background velocity model is cinematically correct, the forward wavefield $\mathbf{u}$ and the adjoint wavefield $\mathbf{v}$ meet at the reflectors position at zero time offset.

The sum over time of the zero time-offset correlation of these two fields then creates an image of the subsurface. Mathematically this leads to the simple imaging condition:

\begin{equation}
  \text{Image} = \sum_{t=1}^{n_t} \mathbf{u}[t] \mathbf{v}[t]
\end{equation}

In the following tutorials we will describe a more advanced imaging condition that produces shaper and more accurate results.

## Operator

We will now define the imaging operator that computes the adjoint wavefield $\mathbf{v}$ and correlates it with the forward wavefield $\mathbf{u}$. This operator essentially consist of three components:
* Stencil update of the adjoint wavefield `v`
* Injection of the data residual at the adjoint source (forward receiver) location
* Correlation of `u` and `v` to compute the image contribution at each timestep

```python
# Define gradient operator for imaging
from devito import TimeFunction, Operator
from examples.seismic import PointSource

from sympy import solve, Eq

def ImagingOperator(model, image):
    # Define the wavefield with the size of the model and the time dimension
    v = TimeFunction(name='v', grid=model.grid, time_order=2, space_order=4)

    u = TimeFunction(name='u', grid=model.grid, time_order=2, space_order=4,
                     save=time_range.num)

    # Define the wave equation, but with a negated damping term
    eqn = model.m * v.dt2 - v.laplace - model.damp * v.dt

    # Use SymPy to rearranged the equation into a stencil expression
    stencil = Eq(v.backward, solve(eqn, v.backward)[0])

    # Define residual injection at the location of the forward receivers
    dt = model.critical_dt
    residual = PointSource(name='residual', grid=model.grid,
                           time_range=time_range,
                           coordinates=rec.coordinates.data)
    res_term = residual.inject(field=v, expr=residual * dt**2 / model.m,
                               offset=model.nbpml)

    # Correlate u and v for the current time step and add it to the image
    image_update = Eq(image, image - u * v)

    return Operator([stencil] + res_term + [image_update],
                    subs=model.spacing_map)
```

## Implementation of the imaging loop

As we just explained, the forward wave-equation is solved forward in time while the adjoint wave-equation is solved in a reversed time order. Therefore, the correlation of these two fields over time requires to store one of the two fields. The computational procedure for imaging follows:

- Simulate the forward wave-equation with the background velocity model to get the synthetic data and save the full wavefield $\mathbf{u}$
- Compute the data residual
- Back-propagate the data residual and compute on the fly the image contribution at each time step.

This procedure is applied to multiple source positions (shots) and summed to obtain the full image of the subsurface. We can first visualize the varying locations of the sources that we will use.

```python
#NBVAL_IGNORE_OUTPUT

# Prepare the varying source locations
source_locations = np.empty((nshots, 2), dtype=np.float32)
source_locations[:, 0] = np.linspace(0., 1000, num=nshots)
source_locations[:, 1] = 30.

plot_velocity(model, source=source_locations)
```

```python
# Run imaging loop over shots
from devito import Function

# Create image symbol and instantiate the previously defined imaging operator
image = Function(name='image', grid=model.grid)
op_imaging = ImagingOperator(model, image)

for i in range(nshots):
    print('Imaging source %d out of %d' % (i+1, nshots))

    # Update source location
    src.coordinates.data[0, :] = source_locations[i, :]

    # Generate synthetic data from true model
    true_d, _, _ = solver.forward(src=src, m=model.m)

    # Compute smooth data and full forward wavefield u0
    smooth_d, u0, _ = solver.forward(src=src, m=model0.m, save=True)

    # Compute gradient from the data residual
    v = TimeFunction(name='v', grid=model.grid, time_order=2, space_order=4)
    residual = smooth_d.data - true_d.data
    op_imaging(u=u0, v=v, m=model0.m, dt=model0.critical_dt,
               residual=residual)
```

    Imaging source 1 out of 21
    Imaging source 2 out of 21
    Imaging source 3 out of 21
    Imaging source 4 out of 21
    Imaging source 5 out of 21
    Imaging source 6 out of 21
    Imaging source 7 out of 21
    Imaging source 8 out of 21
    Imaging source 9 out of 21
    Imaging source 10 out of 21
    Imaging source 11 out of 21
    Imaging source 12 out of 21
    Imaging source 13 out of 21
    Imaging source 14 out of 21
    Imaging source 15 out of 21
    Imaging source 16 out of 21
    Imaging source 17 out of 21
    Imaging source 18 out of 21
    Imaging source 19 out of 21
    Imaging source 20 out of 21
    Imaging source 21 out of 21

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_image

# Plot the inverted image
plot_image(np.diff(image.data, axis=1))
```

```python
assert np.isclose(np.linalg.norm(image.data), 1e6, rtol=1e1)
```

And we have an image of the subsurface with a strong reflector at the original location.

## References

[1] _Versteeg, R.J. & Grau, G. (eds.) (1991): The Marmousi experience. Proc. EAGE workshop on Practical Aspects of Seismic Data Inversion (Copenhagen, 1990), Eur. Assoc. Explor. Geophysicists, Zeist._