 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  The presentation below largely follows part II in "Finite Difference Methods for Ordinary and Partial Differential Equations" by LeVeque (SIAM, 2007).

```python

```

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Solution to ODE Initial Value Problems - Part 1

Many physical, biological, and societal systems can be written as a system of ordinary differential equations (ODEs).  In the case where the initial state (value) is know the problems can be written as

$$
    \frac{\text{d}\mathbf{u}}{\text{d}t} = \mathbf{f}(t, \mathbf{u}) \quad \mathbf{u}(0) = \mathbf{u}_0
$$

where
 - $\mathbf{\!u}(t)$ is the state vector
 - $\mathbf{\!f}(t, \mathbf{\!u})$ is a vector-valued function that controls the growth of $\mathbf{u}$ with time
 - $\mathbf{\!u}(0)$ is the initial condition at time $t = 0$

### The Example of our time:  A non-linear  model of Epidemics

Classical [Kermack and McKendrick (1927)](https://royalsocietypublishing.org/doi/10.1098/rspa.1927.0118) SIR model of epidemics (with reinfection)

$$
\begin{align}
    \frac{ds}{dt} &= -si + kr \\
    \frac{di}{dt} &= si -\sigma i \\
    \frac{dr}{dt} &= \sigma i - kr\\
\end{align}
$$

Where the variable $s$ represents the fraction of a population that is **Susceptible** to infection, $i$ is the proportion **Infected** and $r$ the fraction **Recovered**.  For this model $s+i+r =1$.  The parameters $\sigma, k \geq 0$ control the relative rates of infection and recovery.

For this problem
$$
\mathbf{u}(t) = \begin{bmatrix} s(t)\\ i(t)\\ r(t)\\\end{bmatrix},\quad\quad
\mathbf{f}(t,\mathbf{u}) = \begin{bmatrix} -si + kr \\ si -\sigma i \\ \sigma i - kr\\\end{bmatrix}
$$

### Numerical Solutions

```python
# Solve using SciPy's ODE integrator solve_ivp
from scipy.integrate import solve_ivp

# define the RHS of our system of ODE's
def f_sir(t, u, sigma, k):
    s,i,r = u
    return numpy.array([-s*i + k*r,
                        (s - sigma)*i,
                        sigma*i - k*r ])
```

```python
sigma = .2
k = 0.01
t_max = 100
u_0 = [0.999, 0.001, 0.]
sol = solve_ivp(f_sir, [0, t_max] , u_0, args=(sigma, k), rtol=1.e-6, atol=1.e-9,dense_output = True)
```

```python
t = numpy.linspace(0, t_max, 300)
z = sol.sol(t)

fig = plt.figure(figsize=(20,7))
axes = fig.add_subplot(1,2,1)
axes.plot(t,z[0],'r',label='s', linewidth=2)
axes.plot(t,z[1],'b',label='i', linewidth=2)
axes.plot(t,z[2],'g',label='r',  linewidth=2)
axes.plot(t,sigma*numpy.ones(t.shape),'k--',label='$\sigma$')

axes.legend(loc='best',shadow=True, fontsize=14)
axes.set_xlabel('Time',fontsize=16)
axes.set_ylabel('Population',fontsize=16)
axes.grid()
axes.set_title('SIR system: $\sigma={}$, $k={}$'.format(sigma,k),fontsize=18)
plt.show()
```

### Questions an epidemiologist might ask:
 - What are the dynamics of this system?  does it predict steady or oscillatory solutions?
 - Can we estimate critical parameters ($\sigma$, $k$) from data?
 - Can we reliably use this model to predict the future?
 - How do we evaluate whether this is a *useful* model?
 - How might I modify/improve this model.

### Questions a Computational Mathematician might ask:
 - Does a solution to the model even exist and is it unique?
 - Is our approximate numerical solution accurate?
 - What are the dynamics of this system?  does it predict steady or oscillatory solutions?
 - how do we understand the sensitivity to parameters?

### Existence and Uniqueness of solutions (n-D autonomous systems)

For proof see [Hirsch, Smale, Devaney, Dynamical Systems](https://www.amazon.com/Differential-Equations-Dynamical-Systems-Introduction/dp/0123820103)

#### Theorem: (Picard-Lindelhof)

Given an Autonomous, dynamical system

$$
    \frac{\text{d}\mathbf{u}}{\text{d}t} = \mathbf{f}(\mathbf{u}) \quad \mathbf{u}(0) = \mathbf{u}_0
$$

with $\mathbf{u}\in\mathbb{R}^n$ and $\mathbf{f}:\mathbb{R}^n\rightarrow\mathbb{R}^n$

If $\mathbf{f}$ is

* Bounded: $|\mathbf{f}| < M$
* Lipshitz Continuous: $$|\mathbf{f}(\mathbf{x}) - \mathbf{f}(\mathbf{y})| < K|\mathbf{x}- \mathbf{y} |$$

On a ball of radius $\rho$ around the initial condition $\mathbf{u}_0$.  Then a unique solution exists to the ODE IVP for some interval of time $t\in[-a,a]$ where $0 < a <  \min(\rho/M, 1/K)$

### Geometric Picture

        $$
    \frac{\text{d}\mathbf{u}}{\text{d}t} = \mathbf{f}(\mathbf{u}), \quad \mathbf{u}(0) = \mathbf{u}_0
$$

* **Short Version**: If $\mathbf{f}$ is sufficiently smooth, then a local solution to the ODE exists and is unique

* **Caveat**: The theorem itself gives *NO* constructive way to find that solution

#### Other Examples:  Simple radioactive decay
$$
    \mathbf{\!u} = [c], \quad \mathbf{f} = [-\lambda c]
$$

$$
    \frac{\text{d} c}{\text{d}t} = -\lambda c \quad c(0) = c_0
$$

which has solutions of the form $c(t) = c_0 e^{-\lambda t}$

```python
decay_constant = -numpy.log(2.)/1600.
t = numpy.linspace(0., 4800, 11)
y = numpy.linspace(0., 1.2, 11)
T, Y = numpy.meshgrid(t,y)
dt = numpy.ones(T.shape)
dy = -dt*Y

tp = numpy.linspace(0., 4800, 100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.quiver(T,Y, dt,dy,linewidth=0.1,color='gray')
axes.plot(tp,numpy.exp(decay_constant*tp),linewidth=3)
axes.plot(0.,1.,'ro', markersize=10)
axes.plot(1600., 0.5,'rx',markersize=10)
axes.grid()
axes.set_title("Radioactive Decay, $u' = - \lambda u$, $u(0)=1$, $t_{1/2}=1600$ yr", fontsize=18)
axes.set_xlabel('t (years)', fontsize=16)
axes.set_ylabel('u', fontsize=16)

axes.set_ylim((-.1,1.2))
plt.show()

```

#### Examples:  Complex radioactive decay (or chemical system).

Chain of decays from one species to another.

$$\begin{aligned}
    \frac{\text{d} c_1}{\text{d}t} &= -\lambda_1 c_1 \\
    \frac{\text{d} c_2}{\text{d}t} &= \lambda_1 c_1 - \lambda_2 c_2 \\
    \frac{\text{d} c_3}{\text{d}t} &= \lambda_2 c_2 - \lambda_3 c_3
\end{aligned}$$

$$\frac{\text{d} \mathbf{u}}{\text{d}t} = \frac{\text{d}}{\text{d}t}\begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix} =
\begin{bmatrix}
    -\lambda_1 & 0 & 0 \\
    \lambda_1 & -\lambda_2 & 0 \\
    0 & \lambda_2 & -\lambda_3
\end{bmatrix} \begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix}$$

$$\frac{\text{d} \mathbf{u}}{\text{d}t} = A \mathbf{u}$$

For systems of equations like this the general solution to the ODE is the matrix exponential:

$$\mathbf{u}(t) = \mathbf{u}_0 e^{A t}$$

which can be solved given the eigenvalues and eigenvectors of $A$.

#### Examples:  Particle tracking in a fluid

$$\frac{\text{d} \mathbf{X}}{\text{d}t} = \mathbf{V}(t, \mathbf{X})$$

In fact all ODE IVP systems can be thought of as tracking particles through a flow field (dynamical system).  In 1-dimension the flow "manifold" we are on is fixed by the initial condition.

```python
x = numpy.linspace(0., 1., 11)
y = numpy.linspace(0., 1., 11)
x_fine = numpy.linspace(0., 1.)
y_fine = numpy.linspace(0., 1.)

X, Y = numpy.meshgrid(x,y)
X_fine, Y_fine = numpy.meshgrid(x_fine, y_fine)

pi = numpy.pi
psi = numpy.sin(pi*X_fine)*numpy.sin(pi*Y_fine)
U = pi*numpy.sin(pi*X)*numpy.cos(pi*Y)
V = -pi*numpy.cos(pi*X)*numpy.sin(pi*Y)

x0 = 0.75
y0 = 0.75
psi0 = numpy.sin(pi*x0)*numpy.sin(pi*y0)

fig = plt.figure(figsize=(8,8))
axes = fig.add_subplot(1, 1, 1)
axes.quiver(X,Y, U, V)
axes.plot(.75, 0.75,'ro')
axes.contour(X_fine, Y_fine, psi, [ psi0 ])
axes.grid()
axes.set_title("Particle tracking", fontsize=18)
axes.set_xlabel('y', fontsize=16)
axes.set_ylabel('x', fontsize=16)

plt.show()
```

#### Examples: Van der Pol Oscillator

$$y'' - \mu (1 - y^2) y' + y = 0 \quad \quad \text{with} \quad \quad  y(0) = y_0, \quad y'(0) = v_0$$

$$\mathbf{u} = \begin{bmatrix} y \\ y' \end{bmatrix} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}$$

$$\frac{\text{d}}{\text{d}t} \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} = \begin{bmatrix} u_2 \\ \mu (1 - u_1^2) u_2 - u_1 \end{bmatrix} = \mathbf{f}(t, \mathbf{u})$$

```python
from scipy.integrate import solve_ivp

def f_vanderpol(t, u, mu=5.):
    return numpy.array([u[1], mu * (1.0 - u[0]**2) * u[1] - u[0]])

# N = 100
N = 500
t_span = (0., 200.)
u0 = [ 1., 10. ]
f = lambda t, u: f_vanderpol(t, u, mu=50.)
sol = solve_ivp(f, t_span, u0,method='BDF')
```

```python
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)

axes.plot(sol.t, sol.y[0])
axes.set_title("Solution to Van der Pol Oscillator", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("y(t)", fontsize=16)
axes.grid()

axes = fig.add_subplot(1, 2, 2)

axes.plot(sol.y[0],sol.y[1],'r')
axes.set_title("Phase Diagram for Van der Pol Oscillator", fontsize=18)
axes.set_xlabel("y(t)", fontsize=16)
axes.set_ylabel("y'(t)", fontsize=16)
axes.grid()
plt.show()
```

## The Big Questions

Given a RHS $\mathbf{f}(t,\mathbf{u})$ and initial condition $\mathbf{u}(t_0) \ldots$

*  How do you find a discrete numerical solution that approximates the trajectory $\mathbf{u}(t)$?
*  How do you control the accuracy of the approximation?
*  How do you improve the efficiency of the approximation?
*  How do you understand  stability and convergence?

## Some Notation:  Basic Stepping schemes

Introducing some notation to simplify things
$$\begin{aligned}
    t_0 &= 0 \\
    t_1 &= t_0 + \Delta t \\
    t_n &= t_{n-1} + \Delta t = n \Delta t + t_0 \\
    u_0 &= u(t_0) \approx U_0 \\
    u_1 &= u(t_1) \approx U_1 \\
    u_n &= u(t_n) \approx U_2 \\
\end{aligned}$$
where lower-case letters are "exact".

Looking back at our work on numerical differentiation why not approximate the derivative as a finite difference:

$$
    \frac{u(t + \Delta t) - u(t)}{\Delta t} = f(t, u)
$$

We still need to decide how to evaluate the $f(t, u)$ term however.

One obvious way to do this, is to just use $f(t, u(t))$ and write the update scheme as

$$
    u(t + \Delta t) = u(t) + \Delta t f(t,u(t))
$$

Which is our first integration scheme (which goes by the name Euler's method).  As usual, though the first scheme is often the worst scheme, but with a bit of understanding we can do much better with not a lot more work.

### Integral form of ODE IVP's: the  Relationship to quadrature

Euler's method is an example of a "Single Step, multi-stage" scheme of which there are many.  However, to derive them it is actually more instructive to work with the integral form of an ODE, which will put the equations in a form where we can use our ideas from quadrature to make progress

Given a a system of ODE's
$$
    \frac{d\mathbf{u}}{dt} = \mathbf{f}(t,\mathbf{u})
$$

We can integrate both sides
$$
    \int^{t + \Delta t}_t \frac{d\mathbf{u}}{d \tau} d\tau = \int^{t + \Delta t}_t \mathbf{f}(\tau, \mathbf{u}) d\tau
$$

which is equivalent to the differential form.  However using the fundamental theorem of calculus tells us that the LHS is $u(t + \Delta t) - u(t)$  so we can write the ODE as

$$
   \mathbf{u}(t + \Delta t) = \mathbf{u}(t) + \int^{t + \Delta t}_t \mathbf{f}(\tau, \mathbf{u}(\tau)) d\tau
$$

## Single-Step Multi-Stage Schemes

The integral form of an ODE initial value problem can be written

$$
    u(t + \Delta t) = u(t) + \int^{t + \Delta t}_t f(\tau, u(\tau)) d\tau
$$

Which says that our solution $u$, if it exists  at some time $\Delta t$ in the future,  is $u(t)$ plus a *number*

$$
    K = \int^{t + \Delta t}_t f(\tau, u(\tau) )d\tau
$$

which is a definite *line integral* (along an unknown solution).

An important class of ODE solvers are called *Single Step, Multi-stage schemes* which can be most easily understood as extensions of the  Newton-Cotes quadrature schemes for approximating $K$ (plus an error term that will scale as $\Delta t^p$)

### The Geometric picture

```python
t = numpy.linspace(0., 4800, 11)
y = numpy.linspace(0., 1.2, 11)
T, Y = numpy.meshgrid(t,y)
dt = numpy.ones(T.shape)
dy = -dt*Y

tK = 2000.
uK = numpy.exp(decay_constant*tK)
K = uK -1.
tp = numpy.linspace(0., 4800, 100)
tk = numpy.linspace(0., tK, 100)

fig = plt.figure(figsize=(10,8))
axes = fig.add_subplot(1, 1, 1)
axes.quiver(T,Y, dt,dy, color='gray')
axes.plot(tp,numpy.exp(decay_constant*tp))
axes.plot(0.,1.,'ro')
axes.plot(tk,numpy.exp(decay_constant*tk),'r--')
axes.plot(tK, uK, 'ro')
axes.plot([0.,0.], [1., uK], 'r--')
axes.text(10., 0.72, '$K$', fontsize=24, color='red')
axes.plot([0.,tK],[uK, uK], 'r--')
axes.text(900., uK - .1, '$\Delta t$', fontsize=24, color='red')
axes.text(-10, 1., '$U_0$', fontsize=24, color='blue')
axes.text(tK+10, uK, '$U_1$', fontsize=24, color='blue')

axes.grid()
axes.set_title("Direction Set, $u' = - \lambda u$, $u(0)=1$", fontsize=18)
axes.set_xlabel('t (years)', fontsize=16)
axes.set_ylabel('u', fontsize=16)

axes.set_ylim((-.1,1.2))
plt.show()

```

#### Forward Euler scheme
For example, if we approximate $K$ with a left-sided quadrature rule
$$
    K = \int^{t + \Delta t}_t f(\tau, u(\tau)) d\tau \approx \Delta t f(t, u(t))
$$

then our first ODE algorithm can be written
$$
u(t + \Delta t) =  u(t) + \Delta t f(t, u(t))
$$

Which is exactly Euler's method that we derived previously

in terms of our discrete approximation $U$
$$
\begin{align}
    K_1 &= \Delta t f(t_n, U_n)\\
    U_{n+1} &= U_n + K_1\\
\end{align}
$$

 known as the *forward Euler method*.  In essence we are approximating the derivative with the value of the function at the point we are at $t_n$.

```python
t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

# Euler step
dt = 1e3
u_np = c_0 + dt * (decay_constant * c_0)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., 1., 'ro')
axes.text(0., 1.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot((dt, dt), (u_np, c_0 * numpy.exp(decay_constant * dt)), 'k--')
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.text(10., 0.75, '$K_1$', fontsize=16)
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

```python
# Implement Forward Euler
def euler(f, t_span, u0, N):
    """ simple implementation of constant step-size forward euler method
        This doc string should have so much more in it
    """
    t = numpy.linspace(t_span[0], t_span[1],N)
    u = numpy.empty(t.shape)
    u[0] = u0
    delta_t = t[1] - t[0]
    for (n, t_n) in enumerate(t[:-1]):
        K1 = delta_t * f(t_n, u[n])
        u[n + 1] = u[n] + K1
    return t, u
```

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

t_span = [0.0, 1.6e3]
u0 = 1.
N = 40
t_euler, u_euler = euler(f, t_span, u0, N)

```

```python
t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = lambda t : c_0 * numpy.exp(decay_constant * t)

fig = plt.figure(figsize=(16, 6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(t_euler, u_euler, 'or', label="Euler")
axes.plot(t_exact, u_exact(t_exact), 'k--', label="True Solution")

axes.set_title("Forward Euler")
axes.set_xlabel("t (years)")
axes.set_ylabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.grid()
axes.legend()

abs_err = numpy.abs(u_euler - u_exact(t_euler))
rel_err = abs_err/u_exact(t_euler)
axes = fig.add_subplot(1, 2, 2)
axes.plot(t_euler,abs_err,'ro',label='absolute error')
axes.plot(t_euler,rel_err,'bo',label='relative error')
axes.set_xlabel("t (years)")
axes.set_ylabel("error")
axes.set_title('Error')
axes.legend(loc='best')
axes.grid()

plt.show()
```

### Backward's Euler

Similar to forward Euler is the *backward Euler* method which,   uses a right-rectangle rule to estimate $K$ given $f$ at a future time. i.e.

$$
    K\approx \Delta t f(t_{n+1}, U_{n+1})
$$

However, the update scheme now becomes
$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}).
$$

which requires a (usually non-linear) solve for $U_{n+1}$. Schemes where the function $f$ is evaluated at the unknown time are called *implicit methods*.

For some cases we can solve the equation by hand.  For instance in the case of our example problem, $f=\lambda U$, we have:

$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}) = U_n + \Delta t (\lambda U_{n+1})
$$

which can be solved for $U_{n+1}$ to find

$$\begin{aligned}
    U_{n+1} &= U_n + \Delta t (\lambda U_{n+1}) \\
    U_{n+1} \left[ 1 - \Delta t \lambda \right ] &= U_n \\
    U_{n+1} &= \frac{U_n}{1 - \Delta t \lambda}
\end{aligned}$$

```python
t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(decay_constant * t), label="True Solution")

# Plot Backwards Euler step
dt = 1e3
u_np = c_0 + dt * (decay_constant * c_0 * numpy.exp(decay_constant * dt))
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot(dt, u_np, 'ro')
axes.text(dt+ 10., u_np, '$U_1$', fontsize=16)
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)
axes.text(10., 0.85, '$K_1$', fontsize=16)

axes.grid()
axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
plt.show()
```

```python
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u
n_steps = 20
```

```python
t_exact = numpy.linspace(0.0, 1.6e3, 100)

# Implement backwards Euler
t_backwards = numpy.linspace(0.0, 1.6e3, n_steps)
delta_t = t_backwards[1] - t_backwards[0]
u_backwards = numpy.empty(t_backwards.shape)
u_backwards[0] = c_0
for n in range(0, t_backwards.shape[0] - 1):
    u_backwards[n + 1] = u_backwards[n] / (1.0 - decay_constant * delta_t)

fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(t_backwards, u_backwards, 'or', label="Backwards Euler")
axes.plot(t_exact, u_exact(t_exact), 'k--', label="True Solution")
axes.grid()
axes.set_title("Backwards Euler")
axes.set_xlabel("t (years)")
axes.set_ylabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()

abs_err = numpy.abs(u_backwards - u_exact(t_backwards))
rel_err = abs_err/u_exact(t_backwards)
axes = fig.add_subplot(1, 2, 2)
axes.plot(t_backwards,abs_err,'ro',label='absolute error')
axes.plot(t_backwards,rel_err,'bo',label='relative error')
axes.set_xlabel("t (years)")
axes.set_ylabel("error")
axes.set_title('Error')
axes.legend(loc='best')
axes.grid()

plt.show()
```

It's also useful to be able to do this in the case of systems of ODEs.  Let $f(U) = A U$, then

$$\begin{aligned}
    U_{n+1} &= U_n + \Delta t (A U_{n+1}) \\
     \left [ I - \Delta t A \right ]U_{n+1} &= U_n \\
    U_{n+1} &= \left [ I - \Delta t A \right]^{-1} U_n
\end{aligned}$$

In general however we are often not able to do this with arbitrary $f$.

Another simple implicit method is based on quadrature using the trapezoidal method.  The scheme is
$$
    \frac{U_{n+1} - U_{n}}{\Delta t} = \frac{1}{2} (f(U_n) + f(U_{n+1}))
$$

In this case what is the update scheme for $f(u) = \lambda u$?

$$\begin{aligned}
    U_{n+1} &= U_{n} + \frac{\Delta t}{2} (f(U_n) + f(U_{n+1})) \\
    U_{n+1} &= U_{n} + \frac{\Delta t}{2} (\lambda U_n + \lambda U_{n+1}) \\
    U_{n+1} \left[1 - \frac{\Delta t \lambda}{2}  \right] &= U_{n} \left[1 + \frac{\Delta t \lambda}{2} \right] \\
    U_{n+1} &= U_{n} \frac{1 + \frac{\Delta t \lambda}{2}}{1 - \frac{\Delta t \lambda}{2}} \\
\end{aligned}$$

```python
n_steps = 20
```

```python
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0
t_exact = numpy.linspace(0.0, 1.6e3, 100)

# Implement trapezoidal method
t = numpy.linspace(0.0, 1.6e3, n_steps)
delta_t = t[1] - t[0]
u = numpy.empty(t.shape)
u[0] = c_0
integration_constant = (1.0 + decay_constant * delta_t / 2.0) / (1.0 - decay_constant * delta_t / 2.0)
for n in range(t.shape[0] - 1):
    u[n + 1] = u[n] * integration_constant

fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(t, u, 'or', label="Trapezoidal")
axes.plot(t_exact, u_exact(t_exact), 'k--', label="True Solution")
axes.grid()

axes.set_title("Trapezoidal")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()

abs_err = numpy.abs(u  - u_exact(t))
rel_err = abs_err/u_exact(t)
axes = fig.add_subplot(1, 2, 2)
axes.plot(t,abs_err,'ro',label='absolute error')
axes.plot(t,rel_err,'bo',label='relative error')
axes.set_xlabel("t (years)")
axes.set_ylabel("error")
axes.set_title('Error')
axes.legend(loc='best')
axes.grid()

plt.show()
```

## Error Analysis of ODE Methods

At this point it is also helpful to introduce more notation to distinguish between the true solution to the ODE $u(t_n)$ and the approximated value which we will denote $U_n$.

**Definition:** We define the *truncation error* of a scheme by replacing the $U_n$ with the true solution $u(t_n)$ in the finite difference formula and looking at the difference from the exact solution.

For example we will use the difference form of forward Euler
$$
    \frac{U_{n+1} - U_n}{\Delta t} = f(t_n,U_n)
$$
and define the truncation error as
$$
    T(t, u; \Delta t) = \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - f(t_n, u(t_n)).
$$

**Definition:** A method is called *consistent* if
$$
    \lim_{\Delta t \rightarrow 0} T(t, u; \Delta t) = 0.
$$

**Definition:** We say that a method is *order* $p$ accurate if

$$
    \lVert T(t, u; \Delta t) \rVert \leq C \Delta t^p
$$

uniformally on $t \in [0, \tau]$.  This can also be written as $T(t, u; \Delta t) = \mathcal{O}(\Delta t^p)$.  Note that a method is consistent if $p > 0$.

### Error Analysis of Forward Euler

We can analyze the error and convergence order of forward Euler by considering the Taylor series centered at $t_n$:

$$
    u(t) = u(t_n) + (t - t_n) u'(t_n) + \frac{u''(t_n)}{2} (t - t_n)^2 + \mathcal{O}((t-t_n)^3)
$$

Evaluating this series at $t_{n+1}$ gives

$$\begin{aligned}
    u(t_{n+1}) &= u(t_n) + (t_{n+1} - t_n) u'(t_n) + \frac{u''(t_n)}{2} (t_{n+1} - t_n)^2 + \mathcal{O}((t_{n+1}-t_n)^3)\\
    &=u_n + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3)
\end{aligned}$$

From the definition of truncation error we can use our Taylor series expression and find the truncation error.  Take the finite difference form of forward Euler

$$
    \frac{U_{n+1} - U_n}{\Delta t} = f(t_n, U_n)
$$

and replacing the derivative formulation with $u(t_n)$ to find

$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - f(t_n, u_n) \\
\end{aligned}$$

Given the Taylor's series expansion for $u(t_{n+1})$

$$
u(t_{n+1}) =u(t_n) + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3)
$$

We substitute to find
$$
    T(t, u; \Delta t)  = \frac{u''(t_n)}{2} \Delta t + \mathcal{O}(\Delta t^2).
$$

This implies that forward Euler is first order accurate and therefore consistent.

Another equivalent definition of the truncation error uses the form
$$
    U_{n+1} = u(t_n) + \Delta t f(t_n)
$$
and the definition
$$
    T(t, u; \Delta t) = \frac{1}{\Delta t} \left [ U_{n+1} - u(t_{n+1}) \right]
$$

to find
$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} [U_{n+1} - u(t + \Delta t)] \\
    &= \frac{1}{\Delta t} \left[ \underbrace{u_n + \Delta t f(t_n, u_n)}_{U_{n+1}} - \underbrace{\left( u_n + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right )}_{u(t_{n+1})}\right ] \\
    &= \frac{1}{\Delta t} \left[ - \frac{u''(t_n)}{2} \Delta t^2 - \mathcal{O}(\Delta t^3) \right ] \\
    &= - \frac{u''(t_n)}{2} \Delta t - \mathcal{O}(\Delta t^2)
\end{aligned}$$

#### Truncation Error vs Step Error

Sometimes we will also consider the "Step Error"  which is the error that is introduced over one step

$$
    E_h = | U_{n+1} - u_{n+1} |
$$

This leads to an alternate definition of the truncation error as

$$
    T(t,u;\Delta t) = \frac{E_h}{\Delta t} = \frac{1}{\Delta t} [U_{n+1} - u_{n+1}]
$$

so if the Truncation error is $O(\Delta t^p)$ then the step error will be order $O(\Delta t^{p+1})$

So for Forward (or Backward's) Euler the step error

$$
    E_h = O(\Delta t^2)
$$

The step error can be very useful in *adaptive stepping* schemes

## Runge-Kutta Methods

One way to derive higher-order ODE solvers is to use higher order quadrature schemes that sample the function at a number of  intermediate stages to provide a more accurate estimate of $K$.  These are not *multi-step* methods as they still only require information from the current time step but they raise the order of accuracy by adding *stages*.  These types of methods are called **Runge-Kutta** methods.

### Example:  Two-stage Runge-Kutta Methods

The basic idea behind the simplest of the Runge-Kutta methods is to approximate $K$ using a mid-point scheme (which should be 2nd order accurate).  Unforrunately, we don't know the value of the mid-point.  However we can use an Euler step of size $\Delta t/2$ to estimate the mid-point.

We can write the algorithm as

$$\begin{aligned}
    K_1 &= \Delta t f(U_n, t_n) \\
    K_2 &= \Delta t f(U_n + K_1/2, t_n + \Delta t/2 )\\
    U_{n+1} &= U_n + K_2 \\
\end{aligned}$$

Where we now evaluate the function in two stages $K_1$ and $K_2$.

or for an autonomous ODE
$$
    U_{n+1} = U_n + \Delta t f(U_n + \frac{1}{2} \Delta t f(U_n))
$$

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# RK2 step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
Y1 = U0 + K1/2
K2 = dt * f(dt/2., Y1)
U1 = U0 + K2
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, U0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (U0, U0 + K1), 'k--')
axes.plot((0.0, dt/2.), (U0, U0 + K1/2.), 'k')

axes.plot(dt/2., U0 + K1/2, 'ro')
axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.text(10., 0.85, '$\\frac{K_1}{2}$', fontsize=18)
axes.plot((0.0, dt/2), (Y1, Y1), 'k--')
axes.text(250, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

axes.plot(dt, U1, 'go')
axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.text(10., 0.85, '$\\frac{K_1}{2}$', fontsize=18)
axes.plot((0.0, dt/2), (Y1, Y1), 'k--')
axes.text(250, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

axes.plot(dt, U1, 'go')
axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_1$', fontsize=18)
#axes.plot((0.0, 0.0), (U0, U1), 'g--')
#axes.plot((0.0, dt), (U1, U1), 'g--')

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

#### Error analysis RK2

The truncation error can be computed similarly to how we did so before but we do need to figure out how to compute the derivative inside of the function.  Note that due to
$$
    f(u(t_n)) = u'(t_n)
$$
that differentiating this with respect to $t$ leads to
$$
    f'(u(t_n)) u'(t_n) = u''(t_n)
$$
leading to
$$\begin{aligned}
    f\left(u(t_n) + \frac{1}{2} \Delta t f(u(t_n)) \right ) &= f\left(u(t_n) +\frac{1}{2} \Delta t u'(t_n) \right ) \\
    &= f(u(t_n)) + \frac{1}{2} \Delta t u'(t_n) f'(u(t_n)) + \frac{1}{8} \Delta t^2 (u'(t_n))^2 f''(u(t_n)) + \mathcal{O}(\Delta t^3) \\
    &=u'(t_n) + \frac{1}{2} \Delta t u''(t_n) + \mathcal{O}(\Delta t^2)
\end{aligned}$$

Using our alternative definition of the truncation error we have

$$
        T(t, u; \Delta t) = \frac{1}{\Delta t} \left[U_{n+1} - u_{n+1} \right]
$$
or

$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} \left[u_n + \Delta t f\left(u_n + \frac{1}{2} \Delta t f(u_n)\right) - \left(u_n + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right ) \right] \\
    &=\frac{1}{\Delta t} \left[\Delta t u'(t_n) + \frac{1}{2} \Delta t^2 u''(t_n) + \mathcal{O}(\Delta t^3) - \Delta t u'(t_n) - \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right] \\
    &= \mathcal{O}(\Delta t^2)
\end{aligned}$$

so this method is second order accurate.

### Example:  Improved Euler's method

The Improved Euler's method is another RK2 scheme but instead of approximating a mid-point quadrature rule, it approximates a trapezoidal rule.

We can write the algorithm as

$$\begin{aligned}
    K_1 &= \Delta t f(U_n, t_n) \\
    K_2 &= \Delta t f(U_n + K1, t_n + \Delta t )\\
    U_{n+1} &= U_n + \frac{1}{2}\left[K_1 +K_2\right] \\
\end{aligned}$$

Where we now use function evaluations at both the initial value, and at the euler point but take the average of those slopes.

Again, error analysis shows that this scheme also has a truncation error $T(t,u:\Delta t) = \mathcal{O}(\Delta t^2)$

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# Improved Euler step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
Y1 = U0 + K1
K2 = dt * f(dt, Y1)
U1 = U0 + 0.5*(K1 + K2)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, U0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (U0, U0 + K1), 'k--')
axes.plot((0.0, dt), (U0, U0 + K1), 'k')

axes.plot(dt, Y1, 'ro')
axes.text(dt+10, Y1, '$Y_1$', fontsize=18)
axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.plot((0.0, dt), (Y1, Y1), 'k--')
axes.text(350, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

axes.plot(dt, U1, 'go')
axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.plot(0., Y1, 'ko')
axes.text(10., Y1, '$K_1$', fontsize=18)

axes.plot((0., 0.), (U0, U0+K2),'b--')
axes.plot(0., U0+K2,'bo--')
axes.text(10., U0+K2, '$K_2$', fontsize=18)

axes.plot(0., U1,'gx', markersize=15)
axes.text(10., U1, '$0.5*(K_1 +K_2)$', fontsize=18)

axes.plot(dt, U1, 'go')
axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_1$', fontsize=18)
#axes.plot((0.0, 0.0), (U0, U1), 'g--')
#axes.plot((0.0, dt), (U1, U1), 'g--')

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

### Example:  4-stage Runge-Kutta Method

If RK2 is related to a Mid-point quadrature scheme,  then the classic 4-stage, 4th order Runge-Kutta scheme should be reminiscent of Simpson's Quadrature rule.  It requires 4 samples of $f(t,u)$ at the beginning of the step, two-samples in the middle and one at the end, then a linear combination of those samples

$$\begin{aligned}
    K_1 &= \Delta t f(t_n, U_n) \\
    K_2 &= \Delta t f(t_n + \Delta t/2, U_n + K_1/2) \\
    K_3 &= \Delta t f(t_n + \Delta t/2, U_n + K_2/2) \\
    K_4 &= \Delta t f(t_n + \Delta t, U_n + K_3) \\
        & \\
    U_{n+1} &= U_n + \frac{1}{6} \left [K_1 + 2(K_2 + K_3)  + K_4) \right ]
\end{aligned}$$

With truncation error $T = O(\Delta t^4)$

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# RK4 step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
K2 = dt * f(dt/2., U0 + K1/2)
K3 = dt * f(dt/2., U0 + K2/2)
K4 = dt * f(dt, U0 + K3)

U1 = U0 + 1./6. *( K1 + 2 * (K2 + K3) + K4)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, U0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0.-20, U0-.04, '$K_1$', color='red',fontsize=16)

axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt/2.), (U0, U0 + K1/2.), 'k--')
axes.plot(dt/2., U0 + K1/2, 'ro')
axes.text(dt/2-20, U0 + K1/2-.04, '$K_2$', color='red',fontsize=16)

axes.plot((0.0, dt/2.), (U0, U0 + K2/2.), 'k--')
axes.plot(dt/2., U0 + K2/2, 'ro')
axes.text(dt/2-20, U0 + K2/2+.02, '$K_3$', color='red',fontsize=16)

axes.plot((0.0, dt), (U0, U0 + K3), 'k--')
axes.plot(dt, U0 + K3, 'ro')
axes.text(dt-20, U0 + K3-.04, '$K_4$', color='red',fontsize=16)

axes.plot(dt, U1, 'go')
#axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_1$', fontsize=18)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

```python
def RK2(f, t_span, u0, N):
    """ implement constant step size 2 stage Runge-Kutta Method RK2"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[:-1]):
        K_1 = delta_t * f(t_n, u[n])
        K_2 = delta_t * f(t_n + delta_t/2., u[n] + K_1/2.)
        u[n+1] = u[n] + K_2
    return t, u

def improved_euler(f, t_span, u0, N):
    """ implement constant step size 2 stage Improved Euler Method trapezoidal rule"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[:-1]):
        K_1 = delta_t * f(t_n, u[n])
        K_2 = delta_t * f(t_n + delta_t, u[n] + K_1)
        u[n+1] = u[n] + 0.5 * (K_1 + K_2)
    return t, u
```

```python
def RK4(f, t_span, u0, N):
    """ implement constant step size 4 stage Runge-Kutta Method RK4"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[:-1]):
        K_1 = delta_t * f(t_n, u[n])
        K_2 = delta_t * f(t_n + delta_t/2., u[n] + K_1/2.)
        K_3 = delta_t * f(t_n + delta_t/2., u[n] + K_2/2.)
        K_4 = delta_t * f(t_n + delta_t, u[n] + K_3)
        u[n+1] = u[n] + 1./6. * (K_1 + 2.*( K_2 + K_3) + K_4)
    return t, u
```

```python
# Implement and compare the two-stage and 4-stage Runge-Kutta methods
f = lambda t, u: -u
N = 20
t_span = [ 0., 5.0 ]
u0 = 1.

u_exact = lambda t: u0*numpy.exp(-t)

t_exact = numpy.linspace(t_span[0], t_span[1], 100)
t_euler, u_euler = euler(f, t_span, u0, N)
t_ieuler, u_ieuler = improved_euler(f, t_span, u0, N)
t_RK2, u_RK2 = RK2(f, t_span, u0, N)
t_RK4, u_RK4 = RK4(f, t_span, u0, N)
```

```python
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)
axes.plot(t_exact,u_exact(t_exact),'k',label='exact')
axes.plot(t_euler, u_euler, 'ro', label='euler')
axes.plot(t_ieuler, u_ieuler, 'co', label='improved euler')
axes.plot(t_RK2, u_RK2, 'go', label='RK2')
axes.plot(t_RK4, u_RK4, 'bo', label='RK4')

axes.grid()
axes.set_xlabel('t', fontsize=16)
axes.set_ylabel('u', fontsize=16)
axes.legend(loc='best')

err = lambda u, t: numpy.abs(u - u_exact(t))/u_exact(t)

axes = fig.add_subplot(1, 2, 2)

axes.semilogy(t_euler,err(u_euler,t_euler),'ro',label='euler')
axes.semilogy(t_ieuler,err(u_ieuler,t_ieuler),'co',label='improved euler')
axes.semilogy(t_RK2,err(u_RK2,t_RK2),'go',label='RK2')
axes.semilogy(t_RK4,err(u_RK4,t_RK4),'bo',label='RK4')

axes.set_xlabel("t (years)")
axes.set_ylabel("Rel. error")
axes.set_title('Error')
axes.legend(loc='best')
axes.grid()
plt.show()

```

### Convergence of Single Step Multi-Stage schemes

All of the above schemes are consistent and have truncation errors $T\propto\Delta t^p$

```python
N = numpy.array([ 2**n for n in range(4,10)])
err_euler = numpy.zeros(len(N))
err_ieuler = numpy.zeros(len(N))
err_RK2 = numpy.zeros(len(N))
err_RK4 = numpy.zeros(len(N))

t_span = [ 0., 4.]
dt = t_span[1]/N

u0 = 1.
u_exact = u0*numpy.exp(-t_span[1])

for i, n in enumerate(N):
    t, u_euler = euler(f, t_span, u0, n)
    err_euler[i] = numpy.abs(u_euler[-1] - u_exact)
    t, u_ieuler = improved_euler(f, t_span, u0, n)
    err_ieuler[i] = numpy.abs(u_ieuler[-1] - u_exact)
    t, u_RK2 = RK2(f, t_span, u0, n)
    err_RK2[i] = numpy.abs(u_RK2[-1] - u_exact)
    t, u_RK4 = RK4(f, t_span, u0, n)
    err_RK4[i] = numpy.abs(u_RK4[-1] - u_exact)

err_fit = lambda dt, p: numpy.exp(p[1])*dt**p[0]
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

# Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_euler[2:]),1)
line = axes.loglog(dt, err_euler, 'o', label='euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# Improved Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_ieuler[2:]),1)
line = axes.loglog(dt, err_ieuler, 'o', label='improved euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# RK2
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK2[2:]),1)
line = axes.loglog(dt, err_RK2, 'o', label='rk2, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

#RK4
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK4[2:]),1)
line = axes.loglog(dt, err_RK4, 'o', label='rk4, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

axes.grid()
axes.set_xlabel('$\Delta t$', fontsize=16)
axes.set_ylabel('$Error$', fontsize=16)
axes.set_title('Convergence: Single Step Schemes', fontsize=18)
axes.legend(loc='best', fontsize=14)
plt.show()
```

## Summary: single-Step Multi-Stage Schemes

The integral form of an ODE initial value problem can be written

$$
    u(t + \Delta t) = u(t) + \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t})) d\tilde{\!t}
$$

Which says that our solution $u$, if it exists  at some time $\Delta t$ in the future,  is $u(t)$ plus a *number*

$$
    K = \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t}) )d\tilde{\!t}
$$

which is a definite *line integral* (along an unknown solution).

#### Single Step, Multi-stage schemes

are most easily understood as extensions of the  Newton-Cotes quadrature schemes for approximating $K$ (plus an error term that will scale as $\Delta t^p$)

**Explicit Schemes**

Name	 Stages	 "Quadrature"	$$T$$

Euler	 1	 Left-Rectangle	$$O(\Delta t)$$

Improved Euler	 2	 Trapezoidal	$$O(\Delta t^2)$$

RK2	 2	 Mid-Point	$$O(\Delta t^2)$$

RK4	 4	 Simpson	$$O(\Delta t^4)$$

**Implicit Schemes**

Name	 Stages	 "Quadrature"	$$T$$

Backwards-Euler	 1	 Right-Rectangle	$$O(\Delta t)$$

Trapezoidal	 2	 Trapezoidal	$$O(\Delta t^2)$$

```python
N = numpy.array([ 2**n for n in range(4,10)])
err_euler = numpy.zeros(len(N))
err_ieuler = numpy.zeros(len(N))
err_RK2 = numpy.zeros(len(N))
err_RK4 = numpy.zeros(len(N))

t_span = [ 0., 4.]
dt = t_span[1]/N

u0 = 1.
u_exact = u0*numpy.exp(-t_span[1])

for i, n in enumerate(N):
    t, u_euler = euler(f, t_span, u0, n)
    err_euler[i] = numpy.abs(u_euler[-1] - u_exact)
    t, u_ieuler = improved_euler(f, t_span, u0, n)
    err_ieuler[i] = numpy.abs(u_ieuler[-1] - u_exact)
    t, u_RK2 = RK2(f, t_span, u0, n)
    err_RK2[i] = numpy.abs(u_RK2[-1] - u_exact)
    t, u_RK4 = RK4(f, t_span, u0, n)
    err_RK4[i] = numpy.abs(u_RK4[-1] - u_exact)

err_fit = lambda dt, p: numpy.exp(p[1])*dt**p[0]
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

# Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_euler[2:]),1)
line = axes.loglog(dt, err_euler, 'o', label='euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# Improved Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_ieuler[2:]),1)
line = axes.loglog(dt, err_ieuler, 'o', label='improved euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# RK2
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK2[2:]),1)
line = axes.loglog(dt, err_RK2, 'o', label='rk2, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

#RK4
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK4[2:]),1)
line = axes.loglog(dt, err_RK4, 'o', label='rk4, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

axes.grid()
axes.set_xlabel('$\Delta t$', fontsize=16)
axes.set_ylabel('$Error$', fontsize=16)
axes.set_title('Convergence: Single Step Schemes', fontsize=18)
axes.legend(loc='best', fontsize=14)
plt.show()
```

## Adaptive Time Stepping

#### Why should we care about all of these schemes and their errors?

* Even though we know the formal error.  It is with respect to a true solution we don't know.
* In itself, the error estimates don't tell us how to choose a time step $\Delta t$ to keep the solution within a given tolerance
* However,  in combination, we can use multiple methods to control the error and provide **Adaptive** time stepping

#### Example:  Compare 1 step of Euler to one step of RK2

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# RK2 step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
Y1 = U0 + K1/2
K2 = dt * f(dt/2., Y1)
U1 = U0 + K2

t = numpy.linspace(U0, 1600.)
u_true = U0 * numpy.exp(decay_constant * t)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, u_true, label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (U0, U0 + K1), 'k')
#axes.plot((0.0, dt/2.), (U0, U0 + K1/2.), 'k')

#Euler step
axes.plot(dt, U0 + K1, 'ro')
#axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.text(dt + 10., U0 + K1, '$U_{euler}$', fontsize=18)

axes.plot(dt, U1, 'go')
#axes.plot((0.0, 0.0), (U0, Y1), 'k--')
#axes.text(10., 0.85, '$\\frac{K_1}{2}$', fontsize=18)
#axes.plot((0.0, dt/2), (Y1, Y1), 'k--')
#axes.text(250, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

# RK2 Step
axes.plot(dt, U1, 'go')
axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_{RK2}$', fontsize=18)
#axes.plot((0.0, 0.0), (U0, U1), 'g--')
#axes.plot((0.0, dt), (U1, U1), 'g--')

# difference
axes.plot((dt, dt), (U1, U0+K1),'k--')
axes.text(dt+40, 0.5*(U1 + U0+K1), '$\Delta\propto\Delta t^2$', fontsize=18)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.legend(loc='best')
axes.grid()
plt.show()
```

#### Relative Truncation Error

If we consider the *Step Error* for each of our schemes, we know that

$$
\begin{align}
    u_{n+1} &= U^{euler}_{n+1} + O(\Delta t^2)\\
    u_{n+1} &= U^{RK2}_{n+1} + O(\Delta t^3)\\
\end{align}
$$

Therefore we can compute the *relative truncation error* as

$$
    \Delta = | U^{euler}_{n+1} - U^{RK2}_{n+1} | = O(\Delta t^{?})
$$

* $\Delta$ is Computable!
* has a known dependence on step-size $\Delta t$

### Adaptive Time Stepping

Given the relative truncation error and its scaling with $\Delta t$, we can now use this to choose a single good time step.

#### Example:

Suppose we wanted our relative truncation error to be small relative to the solution or zero, we could set

$$
    \Delta_{target} = \mathtt{rtol}\,U^{RK2}_{n+1} + \mathtt{atol}
$$

where $\mathtt{rtol}$ and $\mathtt{atol}$ are relative and absolute tolerances (and we assume that $U^{RK2}_{n+1}$ is a reasonably good estimate of the true solution)

Moreover,  we know how the relative truncation error should scale with time step, i.e.

$$
    \Delta_{target} \propto \Delta t_{target}^2
$$

But our measured relative truncation error, $\Delta$ depends on  the step size we just took i.e

$$
    \Delta_{measured} \propto \Delta t_n^2
$$

### Adaptive Time Stepping

If we take the ratio of both relationships we get

$$
    \frac{\Delta_{target}}{\Delta_{measured}} = \left[\frac{\Delta t_{target}}{\Delta t_{n}}\right]^2
$$

or rearranging, our target step size is

$$
    \Delta t_{target} = \Delta t_{n}\left[\frac{\Delta_{target}}{\Delta_{measured}}\right]^{\frac{1}{2}}
$$

which tells us how to grow or shrink our time step to maintain accuracy.

In general,  if we have two methods with different step errors such that

$$
    \Delta \propto \Delta t^p
$$

then our adaptive stepper will look like

$$
    \Delta t_{target} = \Delta t_{n}\left[\frac{\Delta_{target}}{\Delta_{measured}}\right]^{1/p}
$$

This leads to all sorts of adaptive schemes most are included in standard libraries.

### Embedded Runge-Kutta Schemes

There are in fact a whole family of **Embedded RK** schemes which are $N$ stage schemes but can combine the $N$ function evaluations in two different ways to produce methods with different error estimates.

A popular one is **RK45** (available in `SciPy`) which is based on the [Dormand-Prince 5(4)](https://doi.org/10.1016/0771-050X(80)90013-3) pair which uses 6 function evaluations per step to produce a 4th order and 5th order scheme.  The 4th order scheme controls the time step, and the 5th order scheme actually is the solution.

```python
from scipy.integrate import solve_ivp

def f_vanderpol(t, u, mu=5):
    return numpy.array([u[1], mu * (1.0 - u[0]**2) * u[1] - u[0]])

t_span = (0., 50.)
u0 = [ 1., 0. ]
f = lambda t, u : f_vanderpol(t, u, mu=20)
sol = solve_ivp(f, t_span, u0, method='RK45',rtol=1.e-3,atol=1.e-8)
```

```python
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)

axes.plot(sol.t, sol.y[0],'o-')
axes.set_title("Solution to Van der Pol Oscillator", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("y(t)", fontsize=16)
axes.grid()

axes = fig.add_subplot(1, 2, 2)
delta_t = sol.t[1:] - sol.t[:-1]
axes.plot(sol.t[:-1], delta_t)
axes.grid()
axes.set_xlabel('$t$', fontsize=16)
axes.set_ylabel('$\Delta t$', fontsize=16)
axes.set_title('Timesteps, N = {}'.format(len(sol.t)), fontsize=18)

plt.show()
```

## Taylor Series Methods

A **Taylor series method** can be derived by direct substitution of the right-hand-side function $f(t, u)$ and its appropriate derivatives into the Taylor series expansion for $u(t_{n+1})$.  For a $p$th order method we would look at the Taylor series up to that order and replace all the derivatives of $u$ with derivatives of $f$ instead.

For the general case we have
$$\begin{align*}
    u(t_{n+1}) = u(t_n) + \Delta t u'(t_n) + \frac{\Delta t^2}{2} u''(t_n) + \frac{\Delta t^3}{6} u'''(t_n) + \cdots + \frac{\Delta t^p}{p!} u^{(p)}(t_n)
\end{align*}$$
which contains derivatives of $u$ up to $p$th order.

We then replace these derivatives with the appropriate derivative of $f$ which will always be one less than the derivative of $u$ (due to the original ODE)

$$
    u^{(p)}(t_n) = f^{(p-1)}(t_n, u(t_n))
$$

leading to the method

$$
\begin{align}
    u(t_{n+1}) &= u(t_n) + \Delta t f(t_n, u(t_n)) + \frac{\Delta t^2}{2} f'(t_n, u(t_n)) \\
    &+ \frac{\Delta t^3}{6} f''(t_n, u(t_n)) + \cdots + \frac{\Delta t^p}{p!} f^{(p-1)}(t_n, u(t_n)).
\end{align}
$$

### 2nd Order Taylor Series Method

We want terms up to second order so we need to take the derivative of $u' = f(t, u)$ once to find $u'' = f'(t, u)$ and therefore
$$\begin{align*}
    u(t_{n+1}) &= u(t_n) + \Delta t u'(t_n) + \frac{\Delta t^2}{2} u''(t_n) \\
    &=u(t_n) + \Delta t f(t_n, u(t_n)) + \frac{\Delta t^2}{2} f'(t_n, u(t_n)) ~~~ \text{or} \\
    U_{n+1} &= U_n + \Delta t f(t_n, U_n) + \frac{\Delta t^2}{2} f'(t_n, U_n).
\end{align*}$$

With Step error $O(\Delta t^3)$ and truncation error $T$, $O(\Delta t^2)$

### Example

Let's use our simplest problem $u'(t) = \lambda u$ with $f=\lambda u$.  Therefore

$$\begin{align*}
    f(t,u) &= \lambda u\\
    f'(t,u) &= \lambda u' = \lambda f = \lambda^2 u\\
    f''(t,u) &= \lambda^2 u' = \lambda^2 f = \lambda^3 u
\end{align*}$$

so a third order scheme would look like
$$
\begin{align}
    U(t_{n+1}) &= U(t_n)\left[ 1 + \lambda\Delta t  + \frac{(\lambda\Delta t)^2}{2} + \frac{(\lambda\Delta t)^3}{6}\right]+ O(\Delta t^4)
\end{align}
$$

```python
def Taylor_3_flambda_u(lamda, t_span, u0, N):
    """ implement constant step size  3rd order Taylor Series method for f(t,u) = \lambda u"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    lambda_dt = lamda*(t[1] - t[0])
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[:-1]):
        u[n+1] = u[n] * ( 1. + lambda_dt  + (lambda_dt**2)/2.  + (lambda_dt**3)/6.)
    return t, u

```

```python
lam = -1.
t_span = [0., 5.]
u0 = 1.

f = lambda t,u : -u
t_exact = numpy.linspace(t_span[0], t_span[1], 100)
u_exact = u0*numpy.exp(-t_exact)

N = 20
t_taylor, u_taylor = Taylor_3_flambda_u(lam, t_span, u0, N)
t_euler, u_euler = euler(f, t_span, u0, N)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_exact,u_exact,'k',label='exact')
axes.plot(t_euler, u_euler, 'ro', label='euler')
axes.plot(t_taylor, u_taylor, 'bo', label='Taylor3')

axes.grid()
axes.set_xlabel('t', fontsize=16)
axes.set_ylabel('u', fontsize=16)
axes.legend(loc='best')
plt.show()

```

### Some Drawbacks

**Taylor Series methods**
 - require differentiating the given equation which can be cumbersome and difficult to implement
 - require a new routine for every $f$

**General one-step/multi-stage methods**
  - higher order methods often require a large number of evaluations of $f$ per time step

# Overview -- so far

So far we have discussed 3 basic techniques for integration of ODE IVP's

* **Single-Step Multi-Stage** schemes (explicit and implicit)
* **Taylor's Series** Methods
* **Linear Multi-step** schemes (just started this)

as well as
* **truncation error** of each method (and it's relation to step-error)
* **adaptive stepping** for Single-Step Schemes

In general Single-Step Multi-Stage methods (e.g. Embedded RK schemes) plus adaptive time stepping make for a very robust family of solvers. However there are some other classical schemes worth mentioning that have some advantages

## Linear Multi-Step Methods

**Multi-step methods**  are ODE methods that
 - require only *one* new function evaluation per time step  to work.
 - reuse values and function evaluations at some number of previous time steps

**Disadvantages over single step methods**

 - Methods are not self-starting, i.e. they require other methods to find the initial values
 - Difficult to adapt. The time step  Δ𝑡  in one-step methods can be changed at any time while multi-step methods this is much more complex

### Simplest example:  The leap-frog method

The leap-frog method is similar to Euler's method in that it uses the information from two-previous time steps to advance the problem.  We can write the problem as a centered first-derivative centered   at time $t_{n+1}, U_{n+1}$ i.e.

$$\frac{U_{n+2} - U_{n}}{2\Delta t} = f(t_{n+1}, U_{n+1})$$

or

$$
    U_{n+2} = U_{n} + 2\Delta t\, f(t_{n+1}, U_{n+1})
$$

this method is known as the leap-frog method

```python
t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(decay_constant * t), label="True Solution")

# Plot Leap-Frog step
dt = 1e3
u1 = c_0 * numpy.exp(decay_constant * dt / 2.0)
u_np = c_0 + dt * (decay_constant * u1)
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot((dt, dt), (u_np, c_0 * numpy.exp(decay_constant * dt)), 'k--')
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)
axes.plot([0., dt/2, dt], [ c_0, u1, u_np],'ro')
axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.grid()
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.0))
plt.show()
```

```python
def leap_frog(f, t_span, u0, N, start=RK2):
    """ calculate fixed step with leap-frog iterator with a single step starter
    """
    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.zeros(t.shape)
    u[0] = u0
    # use a single-step multi-stage method to start

    t_start, u_start = start(f, (t[0],t[1]), u0, 2)
    u[1] = u_start[-1]
    for (n, t_np) in enumerate(t[1:-1]):
        u[n+2] = u[n] +  2 *delta_t * f(t_np, u[n+1])
    return t, u
```

```python
u0 = 1.0
t_span = (0., 1600.)
N = 7

# Stable example
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

t_exact = numpy.linspace(t_span[0], t_span[1], N)
u_exact = u0 * numpy.exp( decay_constant * t_exact)

t_leapfrog, u_leapfrog = leap_frog(f, t_span, u0, N, start=RK4)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_leapfrog, u_leapfrog, 'or-', label="Leap-Frog")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")
axes.grid()
axes.set_title("Leap-Frog", fontsize=18)
axes.set_xlabel("t (years)", fontsize=16)
axes.set_xlabel("$c(t)$", fontsize=16)
axes.legend(loc='best', fontsize=14)
plt.show()
```

### Error Analysis of Leap-Frog Method

To easily analyze this method we will expand the Taylor series around time $t_{n+1}$ to yield
$$\begin{aligned}
    u_{n+2} &= u_{n+1} + \Delta t f(t_{n+1},u_{n+1}) + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)
\end{aligned}$$

We need one more expansion however due to leap-frog.  Recall that leap-frog has the form
$$
    U_{n+2} = U_{n} + 2 \Delta t f(t_{n+1}, U_{n+1}).
$$
To handle the $U_{n}$ term we need to write this with relation to $u(t_{n+1})$.  Again we use the Taylor series
$$
    u(t_n) = u_{n+1} - \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  - \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)
$$

$$\begin{aligned}
    u(t_{n+2}) &= u_{n+1} + \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)\\
    u(t_{n}) &= u_{n+1} - \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  - \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)
\end{aligned}$$

Plugging these into our definition of the truncation error along with the leap-frog method definition leads to

$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} \left [\underbrace{U_{n} + 2 \Delta t f_{n+1}}_{U_{n+2}} - \underbrace{\left(u_{n+1} + \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4) \right )}_{u(t_{n+2})} \right ] \\
    &=\frac{1}{\Delta t} \left [ \underbrace{ \left(u_{n+1} - \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  - \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)\right)}_{u_{n}} + 2\Delta t f_n - \underbrace{\left(u_{n+1} + \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4) \right )}_{u(t_{n+2})} \right ] \\
    &=\frac{1}{\Delta t} \left [- \Delta t^3 \frac{u'''(t_n)}{3} + \mathcal{O}(\Delta t^4) \right ] \\
        &=- \Delta t^2 \frac{u'''(t_n)}{3} + \mathcal{O}(\Delta t^3)
\end{aligned}$$

Therefore the method is second order accurate and is consistent theoretically.  In practice it's a bit more complicated than that.

```python
# Compare accuracy between Euler, RK2 and Leap-Frog
f = lambda t, u: -u
u_exact = lambda t: numpy.exp(-t)
u_0 = 1.0

t_span = (0.0, 10.0)
num_steps = [2**n for n in range(4,11)]
```

```python
delta_t = numpy.empty(len(num_steps))
error_euler = numpy.empty(len(num_steps))
error_RK2 = numpy.empty(len(num_steps))
error_leapfrog = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(t_span[0], t_span[1], N)
    tt, u_euler = euler(f, t_span, u_0, N )
    tt, u_rk2 = RK2(f, t_span, u_0, N)
    tt, u_leapfrog = leap_frog(f, t_span, u_0, N, start=euler)

    delta_t[i] = t[1] - t[0]

    # Compute error for each
    error_euler[i] = numpy.linalg.norm(delta_t[i] * (u_euler - u_exact(t)), ord=1)
    error_RK2[i] = numpy.linalg.norm(delta_t[i] * (u_rk2 - u_exact(t)), ord=1)
    error_leapfrog[i] = numpy.linalg.norm(delta_t[i] * (u_leapfrog - u_exact(t)), ord=1)

# Plot error vs. delta_t
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, error_euler, 'bo', label='Forward Euler, $n=1$')
axes.loglog(delta_t, error_RK2, 'ro', label='RK2, $n=2$')
axes.loglog(delta_t, error_leapfrog, 'go', label="Leap-Frog, $n=2$")

axes.loglog(delta_t, order_C(delta_t[2], error_euler[2], 1.0) * delta_t**1.0, '--b')
axes.loglog(delta_t, order_C(delta_t[2], error_RK2[2], 2.0) * delta_t**2.0, '--r')
axes.loglog(delta_t, order_C(delta_t[2], error_leapfrog[2], 2.0) * delta_t**2.0, '--r')

axes.grid()
axes.legend(loc=2, fontsize=14)
axes.set_title("Comparison of Errors", fontsize=18)
axes.set_xlabel("$\Delta t$",fontsize=16)
axes.set_ylabel("$|U(t_f) - u(t_f)|$", fontsize=16)

plt.show()
```

### Look at the errors for Leap-Frog

They're actually quite large...If you make a quick plot of u_leapfrog vs $t$ you'll see what's happening (and is a good example of an issue we will need to address in future lectures)

```python
N= 100
t_leapfrog, u_leapfrog = leap_frog(f, t_span, u_0, N, start=euler)
## Your plotting code here
plt.figure()
plt.plot(t_leapfrog, u_leapfrog)
plt.grid()
plt.show()

```

### General Linear Multi-Step Methods

Leap-frog is perhaps the simplest of multi-step methods but all linear multi-step methods can be written as the linear combination of past, present and future solutions:
$$
    \sum^r_{j=0} \alpha_j U_{n+j} = \Delta t \sum^r_{j=0} \beta_j f(U_{n+j}, t_{n+j})
$$
If $\beta_r = 0$ then the method is explicit (only requires previous time steps).  Note that the coefficients are not unique as we can multiply both sides by a constant.  In practice a normalization of $\alpha_r = 1$ is used.

For example:  our Leap-frog method can be written using $r=2$, $\alpha = \begin{bmatrix} -1 & 0 & 1\\
\end{bmatrix}$, $\beta = \begin{bmatrix} 0 & 2 & 0 \\ \end{bmatrix}$

#### Example: Adams Methods

$$
    U_{n+r} = U_{n+r-1} + \Delta t \sum^r_{j=0} \beta_j f(U_{n+j}).
$$
All these methods have $\alpha_r = 1$, $\alpha_{r-1} = -1$ and $\alpha_j=0$ for $j < r - 1$.

### Adams-Bashforth Methods
The **Adams-Bashforth** methods are explicit solvers that maximize the order of accuracy given a number of steps $r$.  This is accomplished by looking at the Taylor series and picking the coefficients $\beta_j$ to eliminate as many terms in the Taylor series as possible.
$$\begin{aligned}
    \text{1-step:} & ~ & U_{n+1} &= U_n +\Delta t f(U_n) \\
    \text{2-step:} & ~ & U_{n+2} &= U_{n+1} + \frac{\Delta t}{2} (-f(U_n) + 3 f(U_{n+1})) \\
    \text{3-step:} & ~ & U_{n+3} &= U_{n+2} + \frac{\Delta t}{12} (5 f(U_n) - 16 f(U_{n+1}) + 23 f(U_{n+2})) \\
    \text{4-step:} & ~ & U_{n+4} &= U_{n+3} + \frac{\Delta t}{24} (-9 f(U_n) + 37 f(U_{n+1}) -59 f(U_{n+2}) + 55 f(U_{n+3}))
\end{aligned}$$

```python
def AB2(f, t_span, u0, N, start=RK2):
    """ calculate fixed step Adams-Bashforth 2-step method with a single step starter
        reuses previous function evaluations
    """
    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.zeros(t.shape)

    u[0] = u0
    # use a single-step multi-stage method to start
    t_start, u_start = start(f, (t[0],t[1]), u0, 2)
    u[1] = u_start[-1]

    # set initial function evaluations
    fn = f(t[0], u[0])
    fnp = f(t[1], u[1])
    for (n, t_np) in enumerate(t[2:]):
        u[n+2] = u[n + 1] + delta_t / 2.0 * (-fn + 3.0 * fnp)
        fn = fnp
        fnp = f(t_np, u[n+2])
    return t, u
```

```python
# Use 2-step Adams-Bashforth to compute solution
f = lambda t, u: -u
t_span = (0., 10.)
u0 = 1.0

N = 20
t, u_ab2 = AB2(f, t_span, u0, N, start=RK2)
```

```python
t_exact = numpy.linspace(t_span[0], t_span[1], 100)
u_exact = numpy.exp(-t_exact)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, u_ab2, 'ro', label="2-step A-B")

axes.set_title("Adams-Bashforth Method", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("u(t)",fontsize=16)
axes.legend(loc=1, fontsize=14)
axes.grid()

plt.show()
```

### Adams-Moulton Methods
The **Adams-Moulton** methods are the implicit versions of the Adams-Bashforth methods.  Since this gives one additional parameter to use $\beta_r$ these methods are generally one order of accuracy greater than their counterparts.
$$\begin{aligned}
    \text{1-step:} & ~ & U_{n+1} &= U_n + \frac{\Delta t}{2} (f(U_n) + f(U_{n+1})) \\
    \text{2-step:} & ~ & U_{n+2} &= U_{n+1} + \frac{\Delta t}{12} (-f(U_n) + 8f(U_{n+1}) + 5f(U_{n+2})) \\
    \text{3-step:} & ~ & U_{n+3} &= U_{n+2} + \frac{\Delta t}{24} (f(U_n) - 5f(U_{n+1}) + 19f(U_{n+2}) + 9f(U_{n+3})) \\
    \text{4-step:} & ~ & U_{n+4} &= U_{n+3} + \frac{\Delta t}{720}(-19 f(U_n) + 106 f(U_{n+1}) -264 f(U_{n+2}) + 646 f(U_{n+3}) + 251 f(U_{n+4}))
\end{aligned}$$

```python
# Use 2-step Adams-Moulton to compute solution
# u' = - decay u
decay_constant = 1.0
f = lambda t, u: -decay_constant * u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 20
# N = 10
# N = 5
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)
U[0] = 1.0
U[1] = U[0] + 0.5 * delta_t * f(t[0], U[0])
U[1] = U[0] + delta_t * f(t[0], U[1])
integration_constant = 1.0 / (1.0 + 5.0 * decay_constant * delta_t / 12.0)
for n in range(t.shape[0] - 2):
    U[n+2] = (U[n+1] + decay_constant * delta_t / 12.0 * (U[n] - 8.0 * U[n+1])) * integration_constant

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-M")

axes.set_title("Adams-Moulton Method ($f=-u$)", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("u(t)", fontsize=16)
axes.legend(loc=1, fontsize=14)
axes.grid()
plt.show()
```

### Truncation Error for Multi-Step Methods

We can again find the truncation error in general for linear multi-step methods:
$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} \left [\sum^r_{j=0} \alpha_j u_{n+j} - \Delta t \sum^r_{j=0} \beta_j f(u_{n+j}, t_{n+j}) \right ]
\end{aligned}$$

Using the general expansion and evaluation of the Taylor series about $t_n$ we have
$$\begin{aligned}
    u(t_{n+j}) &= u(t_n) + j \Delta t u'(t_n) + \frac{1}{2} (j \Delta t)^2 u''(t_n) + \mathcal{O}(\Delta t^3) \\
    u'(t_{n+j}) &= u'(t_n) + j \Delta t u''(t_n) + \frac{1}{2} (j \Delta t)^2 u'''(t_n) + \mathcal{O}(\Delta t^3)
\end{aligned}$$

collecting terms of order $u^{(p)}$

$$
\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t}\left( \sum^r_{j=0} \alpha_j\right) u(t_n) + \left(\sum^r_{j=0} (j\alpha_j - \beta_j)\right) u'(t_n) + \Delta t \left(\sum^r_{j=0} \left (\frac{1}{2}j^2 \alpha_j - j \beta_j \right) \right) u''(t_n) \\
& \quad \quad + \cdots + \Delta t^{q - 1} \left (\sum^r_{j=0} \left(\frac{1}{q!} j^q \alpha_j - \frac{1}{(q-1)!} j^{q-1} \beta_j \right) \right) u^{(q)}(t_n) + \cdots
\end{aligned}$$

The method is *consistent* if the first two terms of the expansion vanish, i.e. $\sum^r_{j=0} \alpha_j = 0$ and $\sum^r_{j=0} j \alpha_j = \sum^r_{j=0} \beta_j$.

```python
# Compare accuracy between RK-2, AB-2 and AM-2, RK-4
f = lambda t, u: -u
u_exact = lambda t: numpy.exp(-t)

t_f = 10.0
t_span = (0.0, t_f)

num_steps = [2**n for n in range(4,10)]
delta_t = numpy.empty(len(num_steps))
error_rk = numpy.empty(len(num_steps))
error_rk4 = numpy.empty(len(num_steps))
error_ab = numpy.empty(len(num_steps))
error_am = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(0, t_f, N)
    delta_t[i] = t[1] - t[0]

    # Compute RK2
    tt, U_rk  = RK2(f, t_span, u0, N)

    # Compute RK4
    tt, U_rk4  = RK4(f, t_span, u0, N)

    # Compute Adams-Bashforth 2-stage
    tt, U_ab  = AB2(f, t_span, u0, N)

    # Compute Adama-Moulton 2-stage
    U_am = numpy.empty(t.shape)
    U_am[:2] = U_rk[:2]
    decay_constant = 1.0
    integration_constant = 1.0 / (1.0 + 5.0 * decay_constant * delta_t[i] / 12.0)
    for n in range(t.shape[0] - 2):
        U_am[n+2] = (U_am[n+1] + decay_constant * delta_t[i] / 12.0 * (U_am[n] - 8.0 * U_am[n+1])) * integration_constant

    # Compute error for each
    error_rk[i] = numpy.linalg.norm(delta_t[i] * (U_rk - u_exact(t)), ord=1)
    error_rk4[i] = numpy.linalg.norm(delta_t[i] * (U_rk4 - u_exact(t)), ord=1)
    error_ab[i] = numpy.linalg.norm(delta_t[i] * (U_ab - u_exact(t)), ord=1)
    error_am[i] = numpy.linalg.norm(delta_t[i] * (U_am - u_exact(t)), ord=1)

# Plot error vs. delta_t
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.loglog(delta_t, error_rk, 'ko', label='RK-2')
axes.loglog(delta_t, error_ab, 'bo', label='AB-2')
axes.loglog(delta_t, error_am, 'go', label="AM-2")
axes.loglog(delta_t, error_rk4, 'co', label='RK-4')

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[0], error_ab[0], 1.0) * delta_t**1.0, '--r')
axes.loglog(delta_t, order_C(delta_t[1], error_ab[1], 2.0) * delta_t**2.0, '--b')
axes.loglog(delta_t, order_C(delta_t[1], error_am[1], 3.0) * delta_t**3.0, '--g')
axes.loglog(delta_t, order_C(delta_t[1], error_rk4[1], 4.0) * delta_t**4.0, '--c')

axes.legend(loc=4, fontsize=14)
axes.set_title("Comparison of Errors",fontsize=18)
axes.set_xlabel("$\Delta t$",fontsize=16)
axes.set_ylabel("$|U(t) - u(t)|$", fontsize=16)
axes.grid()

plt.show()
```

### Predictor-Corrector Methods

One way to simplify the Adams-Moulton methods so that implicit evaluations are not needed is by estimating the required implicit function evaluations with an explicit method.  These are often called **predictor-corrector** methods as the explicit method provides a *prediction* of what the solution might be and the now explicit *corrector* step works to make that estimate more accurate.

#### Example: One-Step Adams-Bashforth-Moulton

Use the One-step Adams-Bashforth method to predict the value of $U_{n+1}$ and then use the Adams-Moulton method to correct that value:
$$\begin{aligned}
    \hat{U}_{n+1} &= U_n + \Delta t f(U_n) \\
    U_{n+1} &= U_n + \frac{1}{2} \Delta t \left[f(U_n) + f(\hat{U}_{n+1}) \right]
\end{aligned}$$
leading to a second order accurate method.  Note this algorithm is identical to \_\_\_\_\_\_\_\_\_\_\_\__________?

```python
# One-step Adams-Bashforth-Moulton
f = lambda t, u: -u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 50
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)

U[0] = 1.0
for n in range(t.shape[0] - 1):
    U[n+1] = U[n] + delta_t * f(t[n], U[n])
    U[n+1] = U[n] + 0.5 * delta_t * (f(t[n], U[n]) + f(t[n+1], U[n+1]))

t_ie, u_ieuler = improved_euler(f, (0.0, 10.), 1., N)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-B")
axes.plot(t_ie, u_ieuler, 'bx', label="Improved Euler")

axes.set_title("Adams-Bashforth-Moulton P/C Method", fontsize=18)
axes.set_xlabel("t", fontsize=18)
axes.set_ylabel("u(t)", fontsize=18)
axes.legend(loc='best', fontsize=14)
axes.grid()

plt.show()
```


# Galaxy morphometry

Primarily, morphometrics can be used to address galaxy morphology. Different works use a non-parametric approach to measure a galaxy's shape characteristics. `Lotz et al. (2004)` characterize galaxies' Concentration, Asymmetry. Furthermore, `Ferrari et al. (2015)` include Shannon entropy (information entropy) to quantify pixel values distribution. More recently, `Rosa et al. (2018)` characterized a galaxy's morphology using the second moment of the gradient of the images through Gradient Pattern Analysis. One of the critical features of non-parametric morphology estimation is understanding how a given parameter can reliably separate elliptical and spiral galaxies. Each metric measures patterns and forms in the image. Concentration (C) measures how tightly pixels are distributed in the galaxy structure, Asymmetry (A) measures irregularity of the form of the galaxy's disk and bulge, Smoothness (S) measures the presence of the small structures in the galaxy's disk (star-forming regions), Entropy (H) is the measure of the heterogeneity of the pixel distribution and G2 analyses the asymmetric vector field (variation of flux intensity). When applied to an image with a clearly elliptical galaxy, usually, C will have high values, and A, S, H, and G2 will have low values. While in the case of a clearly spiral galaxy, it would be the opposite. Combining these values can provide a solid intuition of the class of unknown galaxies.

Besides using the morphometric parameters to study the actual physical
processes molding galaxies directly, they can also be used in Machine Learning (ML) methods
to discriminate ETGs from LTGs. `Barchi et al. (2020)` conducts a thorough
study of ML using the CASHG2 system as the primary input information and galaxy
Zoo 1, GZ1 `(LINTOTT et al., 2008; LINTOTT et al., 2011)` provides the "true" classification.
These are the main ingredients for the training step. Several ML algorithms
were tested: Decision Tree (DT); Support Vector Machine (SVM); and Multilayer
Perceptron (MLP). DT had a slightly better performance than the other
two, with an overall accuracy (OA) of 98.5%, when dealing only with bright galaxies.
When applying Deep Learning (DL) technique, they find only a tiny increase
in OA, 99.5%.

CyMorph can also be applied to:

 - X-Ray map tracing gas from the ICM. By combining the results of the galaxy's morphology and ICM hot gas X-Ray morphology properties, we plan to investigate how the cluster's ICM morphological properties (high asymmetry, for example) affect member galaxies' morphology.

 - Trace galaxy properties typical for $\gamma$-ray burst(GRB) hosts to prioritize targets with similar characteristics during gravitational wave optical follow-ups `(Santana-Silva et al. (in prep.)`.

In summary, CyMorph can extract five different metrics: Concentration (C), Asymmetry (A), Smoothness (S), Entropy (H), and Gradient Pattern Analysis (G2). Each of these metrics (except G2) ranges from 0 (min) to 1 (max) depending on the particular features of the content of the image. These metrics have a wide range of application in further resurch.

## Metrics

### Concentration

Concentration is straightforward to grasp intuitively; it simply measures how the flux is distributed on the galaxy profile, that is concentrated in the center (bulge) of the galaxy or distributed around the whole profile. The practical process to perform this kind of measurement consists of several steps. Literature has different approaches to calculate concentration `(BERSHADY et al., 2000; GRAHAM et al., 2001b; ABRAHAM et al., 1994).`. Here we fallow the method proposed in `Conselice (2003) and Lotz et al. (2004)`.

Figure 1 is showing the $R_p$ and $2*R_p$ (left panel) and $\eta$ profile (left panel) of a spiral galaxy

Figure 1

By definition, Concentration stands for a ratio of two portions of accumulated flux in two different fractions of the total flux of a galaxy. It is defined by: $C = log_{10}(R_1/R_2)$, where $R_1$ and $R_2$ are the fractions of the total flux. They also called the outer and inner radii enclosing a fraction of total flux. The values of $R_1$ and $R_2$ can vary arbitrary, in range of 100 to 0 (containing all flux and none of it). Several studies used different pairs of $R_1$ and $R_2$ `(LOTZ et al., 2004; FERRARI et al., 2015)`.

Panel A in Figure 2 shows the accumulated flux curve for the elliptical galaxy, panel B for the spiral galaxy. From the comparison, it could be seen that flux growth in the case of the elliptical galaxy slows considerably already at $R_2$, resulting in the highly concentrated center of this galaxy. In the case of a spiral galaxy, the accumulated flux keeps increasing more uniformly even after $R_2$.

Figure 2

The processing starts by calculating the accumulated flux intensity curve and $\eta$ profile on the cleaned image. Galaxies are resolved objects with poorly defined edges and do not all have the same radial surface brightness profile; some care is required to define the flux associated with each object. The $\eta$ profile is necessary to obtain the Petrosian radius ($R_p$) because the total flux of a galaxy is defined by the accumulated flux until $2*R_p$. The aperture $2*R_p$ is large enough to contain nearly all the flux for typical galaxy profiles but small enough that the sky noise impact is negligible `Blanton et al. (2001)`. Based on the $\eta$ profile, it is also largely insensitive to variations in the limiting surface brightness and redshift  (in the sense of distance), providing reliable results for the galaxies with a high signal-to-noise ratio.

We follow `Blanton et al. (2001)` and `Strateva et al. (2001)` and set $R_p$ on $\eta = 0.2$ as it is shown in the right panel in Figure 1. The left panel in Figure 1 shows the portions in $R_p$ and $2*R_p$. Finally, it becomes possible to set the radii that contain the fractions of total flux. Intuitively and as it could be seen in the Figure 2, elliptical galaxies have the flux concentrated in the center, and it falls almost entirely in the R2 radius, leaving R1 with almost equal to R2 and pushing the ratio up. The opposite occurs with spiral galaxies, the main portion of the flux is concentrated in the center, but the spiral arms increase the total flux located in R2 and drive the ratio down.

### Asymmetry

Asymmetry could be the most uncomplicated metric to extract, and the simplicity consists in the fact that it is not necessary to perform any additional operation on the $segmented\_image$. Tracing the asymmetry distribution in the galaxy profile can help to reveal dynamic processes in galaxies. This is especially true when we talk about collisionless stars and can track the matter distribution more precisely.  For instance, galaxies disturbed by interactions or mergers with another galaxy will tend to have high asymmetries `(CONSELICE et al., 2000)`.

Asymmetry, by definition, measures the degree of irregularity of the galaxy profile. To obtain it, CyMorph will rotate the segmented image by $180^{\circ}$ and run a $for$ loop on both of the images (segmented and rotated). This loop will compare each $[i,j]$ pixel of both of the images, and in case if both are not zero (containing flux counts), the non-zero pixels will be stored in $list1$ and $list2$—these lists of pixels for segmented and rotated images, respectively.

Figure 3 is showing how does correlation works when we collect the pixels on input (Panel A) and rotated (Panel B) images. Panel C shows the visual correlation of $list1$ and $list2$, rotated and original pixels respectively.

Figure 3

The next step would be the calculation of the correlation coefficient between the segmented image and the rotated image pixels lists. CyMorph will use Pearson and Spearman Ranks `(PRESS,2005)` coefficient functions to calculate the correlation coefficients between the two lists. In the case of an elliptical galaxy, the correlation coefficient will be high since the pixels of the elliptical galaxy are fairly heterogeneously distributed. In the case of a spiral galaxy, where pixels have a much higher gradient, it will mean that, after rotation, two pixels on the same position will have very different values.
The formula to calculate A is $A = 1 - spearmanr(list1, list2)$. When the correlation coefficient is high (meaning that there is no significant difference between pixel values on both images), asymmetry will be low (case of an elliptical galaxy). In case if the correlation coefficient is low, the asymmetry will be high (case of spiral galaxy).

As it could be seen in Figure 3, if one rotates a spiral galaxy, the correlation would be low since spiral arms and similar irregularities will contribute to the fact that the same pixels would have very different flux values. It would be the opposite if we rotated the elliptical one. The reason is that elliptical galaxies have nearly perfect flux distribution. So the correlation will be high.

### Smoothness

Smoothness (or clumpiness) is calculated very similarly with asymmetry. It calculates the Pearson and Spearman ranks correlation coefficients between the segmented image and its smoothed version `(ABRAHAM et al., 1996; CONSELICE, 2003; FERRARI et al., 2015)`. Instead of rotation, we are applying a second order of the Butter filter to smooth the original image. This filter provides the advantage of continuous adaptive control of the smoothing degree applied to the image `(KASZYNSKI; PISKOROWSKI, 2006; PEDRINI; SCHWARTZ, 2008; SAUTTER, 2018)`. By intuition, if one tries to smooth the elliptical galaxy, the result will be almost the same image because naturally, these galaxies are smooth. Scanning the images, storing the pixels to lists, and calculating the coefficient will result in a high correlation between the segmented and smoothed image. Spiral galaxies will produce the opposite result, for the correlation will be low between the images, and clumpiness will be high. The formula to obtain S is $S = 1 - spearmanr(list1, list2)$.

The nomenclature could bring confusion, but the logic behind this metric consists in the fact that spiral galaxies will present small structures inside the disk that will contribute to the high Clumpiness values, while elliptical galaxies, naturally smooth, will have high correlation and low Clumpiness value.

From Figure 4 we can see that smoothing the spiral galaxy produces a significant difference. In the case of an elliptical galaxy, it would be almost unnoticeable. The level of change in the image after the smoothing is the key factor in the metric calculation. Figure 4 is showing how does correlation works when we collect the pixels on input and smoothed image. Panel C is showing the visual correlation of $list1$ and $list2$, rotated and original pixels respectively.

Figure 4

### Entropy

Entropy (H) works very similarly to the concentration in the sense that it measures pixel density/frequency in a given number of bins. Entropy bins number is a parameter that can be tuned to adapt better to the data at hand, and it is responsible for how many bins the flux distribution will be split. In a simplified manner, H measures the distribution of pixel values in the image by dividing the image in an arbitrary number of bins. The process of the H extraction does not have any additional image manipulation. The input image pixels are raveled (converted to 1d array) and are used to calculate values of the histogram (frequency of the flux) and bin edges with $numpy.histogram$. The illustration of this process is shown in Figure 5.

It is worth noting how the flux of the elliptical galaxy is concentrated in a small area, while the spiral galaxy's flux occupies a broader area of flux distribution. The next step is the normalization of the frequency counts by maximum count and then calculating the entropy value with:

\begin{equation}
H(I)=-\sum_{k}^{K} p\left(I_{k}\right) \log \left[p\left(I_{k}\right)\right]
\end{equation}
Equation 1

where:

- $p(I_{k})$ - s the probability of occurrence of the value $I_{k}$
- $K$ - number of bins that data was split

For  discrete  variables, $H$ reaches  the maximum value for a uniform distribution, when $p(I_{k}) = 1/K$ for all $k$, resulting in $H_{max} = \log K$.  The minimum entropy is that of a delta function, for which $H = 0$. Hence, we can get the normalized entropy with:

\begin{equation}
\widetilde{H}(I)=\frac{H(I)}{H_{\max }} \quad 0 \leqslant \widetilde{H}(I) \leqslant 1
\end{equation}
Equation 2

Elliptical galaxies are expected to have low entropy (for having natural homogeneity of flux (pixel value) distribution), and spiral galaxies are expected to have high values of entropy (by presenting irregular structures and those having pixel value heterogeneity naturally) `(BISHIP, 2007; FERRARI et al., 2015)`.

The top row in the Figure 5 shows the flux distribution of an elliptical galaxy when the image is raveled and used as input for the $numpy.histogram$. The thin distribution limited to low counts with a fast decrease slope characterizes the flux concentration of the elliptical profile. We can see the opposite in the case of a spiral galaxy (middle row), where the flux is spread over a bigger area of the flux axis.

Figure 5

### Gradient Pattern Analysis (second moment)

The second moment of Gradient Pattern Analysis (GPA), in short G2, can be considered the most complex metric to be calculated. According to `Sautter (2018)`, GPA has four moments, but for the galaxy morphology classification, only the first and second are currently used. `Rosa et al. (2018)` showed that improved and revised version of the second moment (with due modifications) shows best results in galaxy separations when compared with classic CAS classification `(CONSELICE, 2003)`. We have performed a revision and optimization of the code to comply with the original definition.

#### Extraction process

The first step is to convert all zeros in the image to $numpy.nan$. This condition assures that the border pixels of the image will be ignored during the gradient field generation. Otherwise, they would contribute detrimentally to the final result. Then we are generating the gradient field with $numpy.gradient(segmented\_image)$.

In the next step, we will generate an asymmetric gradient field. This process consists of locating pairs of pixels located at the same distance from the center and comparing modulus (strength) and phase (direction) between them. If two pixels of the given pair have the same modulus but opposite phases, they will be considered symmetric and removed. This process will be performed on all the unique pairs of pixels (to exclude the repetition). The resulting lattice is called an asymmetry vector field because all symmetric pairs are removed. Next, we will obtain the count of asymmetric vectors, their sum, and their modulus sum. To determine if the vectors are aligned and have the same magnitude, we need to calculate $confluence$ using the equation 3:

\begin{equation}
confluence = \left(\frac{\left|\sum_{i}^{V_{a}} v_{a}^{i}\right|}{\sum_{i}^{V_{a}}\left|v_{a}^{i}\right|}\right)
\end{equation}
Equation 3

where:

- $v_{a}$ - list of asymmetrical vectors
- $V_{A}$ - count of asymmetric vectors

The final calculation of G2 value is obtained by the equation 4 `(ROSA et al., 1999;
RAMOS et al., 2000; ROSA et al., 2003; SAUTTER, 2018)`:

\begin{equation}
G_{2} = \frac{V_{A}} {V - V_{c}} (2-confluence)
\end{equation}
Equation 4

where:

- $V_{A}$ - total valid pixels
- $V_{c}$ - contour pixels
- $V_{A}$ - asymmetric pixels
- $2$ - normalization factor

Figure 6 shows results in two fringe cases: random noise, resulting in maximum G2 value (as there are very few pairs that end up canceled) and Gaussian noise, resulting in minimum G2 value (as there are all pixel pairs end up canceled).

Figure 6

It is possible to perform fine-tuning of the G2. It has two tolerances: $modulus\_tolerance$. $modulus\_tolerance$ is responsible for serving as the threshold of the minimum acceptable strength difference between two vectors. Worth noting that modulus are normalized by the maximum value during the calculation, so the $modulus\_tolerance$ can be influential even with low values, as it ranges from 0 (no tolerance at all) to 1 (any two pixels are considered to have same modulus). $phase\_tolerance$ serves as a threshold of minimal acceptable angle difference between two pixels (their vectors). It ranges from 0 (no tolerance) to 3.14 (any any two pixels are considered to have same angle)

The top row in Figure 6 shows the results expected with a simulated input image with Gaussian noise. In this case, almost all the vectors should cancel out, the asymmetric gradient field should be empty, and the resulting value of G2 should be near to 0 (minimum value of G2). The bottom row shows the results expected with a simulated input image with random noise. In this case, almost all the vectors  will be preserved, and the resulting value of G2 should be near to 2.0 (maximum value of G2).

Intuitively, elliptical galaxies will have a lot of the same vectors, being naturally smooth and round, and as a consequence, many vectors will be canceled. In the case of spiral galaxies, the internal structures distort the gradient field, resulting in many vectors being preserved. Figure 7 shows visualization of this behavior.

The top row shows the results expected with elliptical galaxies. Generally, most of the vectors should cancel out, resulting in a low value of G2 ($G2 < 0.5$). The bottom row shows the results expected with spiral galaxies. In this case, most of the vectors will be preserved, and the resulting value of G2 should be high ($G2 > 1.5$).

Figure 7