## Zadaca 1

```python
import sympy as sp
from sympy.physics import mechanics
import Resources.script as rt
mechanics.init_vprinting()
```

```python
m, g = sp.symbols('m, g')
y = mechanics.dynamicsymbols('y')
y
```

```python
vy = y.diff('t')
vy
```

```python
K = m * vy**2 / 2
K
```

```python
P = m * g * y
P
```

```python
L = K-P
L
```

```python
rt.lagrangian(L, [y])

```

$\displaystyle \left[\begin{matrix}m \left(g + \ddot{y}\right)\end{matrix}\right]$

## Zadaca 2

```python
m, g, l = sp.symbols('m, g, l')
theta = mechanics.dynamicsymbols('theta')
```

```python
x = l*sp.cos(theta)
x
```

```python
y = l*sp.sin(theta)
y
```

```python
K = m * (x.diff('t')**2 + y.diff('t')**2) / 2
K = sp.simplify(K)
K
```

```python
P = m*g*l*sp.sin(theta)
P
```

```python
L = K-P
L
```

```python
rt.lagrangian(L, [theta])
```

$\displaystyle \left[\begin{matrix}l m \left(g \cos{\left(\theta \right)} + l \ddot{\theta}\right)\end{matrix}\right]$

## Zadaca 3

```python
m1, m2, g = sp.symbols('m1, m2, g')
d1, d2 = mechanics.dynamicsymbols('d1, d2')
v1, v2 = d1.diff('t'), d2.diff('t')
```

```python
K1 = (m1*v1**2)/2
K2 = (m2*(v1**2+v2**2))/2
K = K1 + K2
K
```

```python
P1 = m1*g*d1
P2 = m2*g*d1
P = P1 + P2
P.simplify()
```

```python
L = K - P
L
```

```python
rt.lagrangian(L, [d1, d2])
```

$\displaystyle \left[\begin{matrix}g m_{1} + g m_{2} + m_{1} \ddot{d}_{1} + m_{2} \ddot{d}_{1}\\m_{2} \ddot{d}_{2}\end{matrix}\right]$

## Zacada 11

```python
m1, m2, g, l1, l2, lc1, lc2, I1, I2 = sp.symbols('m1, m2, g, l1, l2, l_c1, l_c2, I1, I2')
theta1, theta2 = mechanics.dynamicsymbols('theta1, theta2')
vtheta1, vtheta2 = theta1.diff('t'), theta2.diff('t')
lc1, lc2
```

```python
k1 = m1* lc1**2 *vtheta1**2 /2 + I1 * vtheta1**2 /2
k1.simplify()
```

```python
x2 = l1*sp.cos(theta1) + lc2*sp.cos(theta1+theta2)
y2 = l1*sp.sin(theta1) + lc2*sp.sin(theta1+theta2)
vx2 = x2.diff('t')
vy2 = y2.diff('t')
vx2,vy2
```

```python
k2 = m2 * (vx2**2 + vy2**2) / 2 + I2* (vtheta1**2 + vtheta2**2)/2
k2
```

```python
k = k1+k2
k
```

```python
p1 = m1*g*lc1*sp.sin(theta1)
p2 = m2*g*l1*sp.sin(theta1) + m2*g*lc2*sp.sin(theta1+theta2)
p = p1+p2
p
```

```python
L = k-p
rt.dynamic_model_with_3_matrices(L, [theta1, theta2])
```

$\displaystyle \left( \left[\begin{matrix}\left(I_{1} + I_{2} + l_{1}^{2} m_{2} + 2 l_{1} l_{c2} m_{2} \cos{\left(\theta_{2} \right)} + l_{c1}^{2} m_{1} + l_{c2}^{2} m_{2}\right) \ddot{\theta}_{1} & l_{c2} m_{2} \left(l_{1} \cos{\left(\theta_{2} \right)} + l_{c2}\right) \ddot{\theta}_{2}\\l_{c2} m_{2} \left(l_{1} \cos{\left(\theta_{2} \right)} + l_{c2}\right) \ddot{\theta}_{1} & \left(I_{2} + l_{c2}^{2} m_{2}\right) \ddot{\theta}_{2}\end{matrix}\right], \  \left[\begin{matrix}0 & - 2 l_{1} l_{c2} m_{2} \sin{\left(\theta_{2} \right)} \dot{\theta}_{1} \dot{\theta}_{2} & - l_{1} l_{c2} m_{2} \sin{\left(\theta_{2} \right)} \dot{\theta}_{2}^{2}\\l_{1} l_{c2} m_{2} \sin{\left(\theta_{2} \right)} \dot{\theta}_{1}^{2} & 0 & 0\end{matrix}\right], \  \left[\begin{matrix}g \left(l_{1} m_{2} \cos{\left(\theta_{1} \right)} + l_{c1} m_{1} \cos{\left(\theta_{1} \right)} + l_{c2} m_{2} \cos{\left(\theta_{1} + \theta_{2} \right)}\right)\\g l_{c2} m_{2} \cos{\left(\theta_{1} + \theta_{2} \right)}\end{matrix}\right]\right)$

## Zacada 12

```python
m1, m2, g, l1, l2, r1, r2 = sp.symbols('m1, m2, g, l1, l2, r1, r2')
theta1, theta2 = mechanics.dynamicsymbols('theta1, theta2')
vtheta1, vtheta2 = theta1.diff('t'), theta2.diff('t')
```

```python
Ix1, Iy1, Iz1, Ix2, Iy2, Iz2 = sp.symbols('I_x1, I_y1, I_z1, I_x2, I_y2, I_z2')
T1 = sp.Matrix([[Ix1, 0, 0], [0, Iy1, 0], [0, 0, Iz1]])
T2 = sp.Matrix([[Ix2, 0, 0], [0, Iy2, 0], [0, 0, Iz2]])
T1, T2
```

$\displaystyle \left( \left[\begin{matrix}I_{x1} & 0 & 0\\0 & I_{y1} & 0\\0 & 0 & I_{z1}\end{matrix}\right], \  \left[\begin{matrix}I_{x2} & 0 & 0\\0 & I_{y2} & 0\\0 & 0 & I_{z2}\end{matrix}\right]\right)$

```python
x1 = r1*sp.cos(theta1)
y1 = r1*sp.sin(theta1)
v1 = sp.Matrix([x1.diff('t'), y1.diff('t'), 0])
v1
```

$\displaystyle \left[\begin{matrix}- r_{1} \sin{\left(\theta_{1} \right)} \dot{\theta}_{1}\\r_{1} \cos{\left(\theta_{1} \right)} \dot{\theta}_{1}\\0\end{matrix}\right]$

```python
w1 = sp.Matrix([0, 0, theta1.diff('t')])
w1
```

$\displaystyle \left[\begin{matrix}0\\0\\\dot{\theta}_{1}\end{matrix}\right]$

```python
K1 = m1 * v1.T * v1 / 2  +  w1.T * T1 * w1 / 2
K1
```

$\displaystyle \left[\begin{matrix}\frac{I_{z1} \dot{\theta}_{1}^{2}}{2} + \frac{m_{1} r_{1}^{2} \sin^{2}{\left(\theta_{1} \right)} \dot{\theta}_{1}^{2}}{2} + \frac{m_{1} r_{1}^{2} \cos^{2}{\left(\theta_{1} \right)} \dot{\theta}_{1}^{2}}{2}\end{matrix}\right]$

```python
x2 = l1*sp.cos(theta1) + r2*sp.cos(theta1+theta2)
y2 = l1*sp.sin(theta1) + r2*sp.sin(theta1+theta2)
v2 = sp.Matrix([x2.diff('t'), y2.diff('t'), 0])
v2
```

$\displaystyle \left[\begin{matrix}- l_{1} \sin{\left(\theta_{1} \right)} \dot{\theta}_{1} - r_{2} \left(\dot{\theta}_{1} + \dot{\theta}_{2}\right) \sin{\left(\theta_{1} + \theta_{2} \right)}\\l_{1} \cos{\left(\theta_{1} \right)} \dot{\theta}_{1} + r_{2} \left(\dot{\theta}_{1} + \dot{\theta}_{2}\right) \cos{\left(\theta_{1} + \theta_{2} \right)}\\0\end{matrix}\right]$

```python
w2 = sp.Matrix([0, 0, theta2.diff('t')])
w2
```

$\displaystyle \left[\begin{matrix}0\\0\\\dot{\theta}_{2}\end{matrix}\right]$

```python
K2 = m2 * v2.T * v2 / 2  +  w2.T * T2 * w2 / 2
K2
```

$\displaystyle \left[\begin{matrix}\frac{I_{z2} \dot{\theta}_{2}^{2}}{2} + \frac{m_{2} \left(- l_{1} \sin{\left(\theta_{1} \right)} \dot{\theta}_{1} - r_{2} \left(\dot{\theta}_{1} + \dot{\theta}_{2}\right) \sin{\left(\theta_{1} + \theta_{2} \right)}\right)^{2}}{2} + \frac{m_{2} \left(l_{1} \cos{\left(\theta_{1} \right)} \dot{\theta}_{1} + r_{2} \left(\dot{\theta}_{1} + \dot{\theta}_{2}\right) \cos{\left(\theta_{1} + \theta_{2} \right)}\right)^{2}}{2}\end{matrix}\right]$

```python
P = m1 * g * r1 * sp.sin(theta1) + m2 * g * l1 * sp.sin(theta1) + m2 * g * r2 * sp.sin(theta1 + theta2)
K = K1+K2
K
```

$\displaystyle \left[\begin{matrix}\frac{I_{z1} \dot{\theta}_{1}^{2}}{2} + \frac{I_{z2} \dot{\theta}_{2}^{2}}{2} + \frac{m_{1} r_{1}^{2} \sin^{2}{\left(\theta_{1} \right)} \dot{\theta}_{1}^{2}}{2} + \frac{m_{1} r_{1}^{2} \cos^{2}{\left(\theta_{1} \right)} \dot{\theta}_{1}^{2}}{2} + \frac{m_{2} \left(- l_{1} \sin{\left(\theta_{1} \right)} \dot{\theta}_{1} - r_{2} \left(\dot{\theta}_{1} + \dot{\theta}_{2}\right) \sin{\left(\theta_{1} + \theta_{2} \right)}\right)^{2}}{2} + \frac{m_{2} \left(l_{1} \cos{\left(\theta_{1} \right)} \dot{\theta}_{1} + r_{2} \left(\dot{\theta}_{1} + \dot{\theta}_{2}\right) \cos{\left(\theta_{1} + \theta_{2} \right)}\right)^{2}}{2}\end{matrix}\right]$

```python
L = K[0] - P
L
```

```python
rt.lagrangian(L, [theta1, theta2])
```

$\displaystyle \left[\begin{matrix}I_{z1} \ddot{\theta}_{1} + g l_{1} m_{2} \cos{\left(\theta_{1} \right)} + g m_{1} r_{1} \cos{\left(\theta_{1} \right)} + g m_{2} r_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} + l_{1}^{2} m_{2} \ddot{\theta}_{1} - 2 l_{1} m_{2} r_{2} \sin{\left(\theta_{2} \right)} \dot{\theta}_{1} \dot{\theta}_{2} - l_{1} m_{2} r_{2} \sin{\left(\theta_{2} \right)} \dot{\theta}_{2}^{2} + 2 l_{1} m_{2} r_{2} \cos{\left(\theta_{2} \right)} \ddot{\theta}_{1} + l_{1} m_{2} r_{2} \cos{\left(\theta_{2} \right)} \ddot{\theta}_{2} + m_{1} r_{1}^{2} \ddot{\theta}_{1} + m_{2} r_{2}^{2} \ddot{\theta}_{1} + m_{2} r_{2}^{2} \ddot{\theta}_{2}\\I_{z2} \ddot{\theta}_{2} + g m_{2} r_{2} \cos{\left(\theta_{1} + \theta_{2} \right)} + l_{1} m_{2} r_{2} \sin{\left(\theta_{2} \right)} \dot{\theta}_{1}^{2} + l_{1} m_{2} r_{2} \cos{\left(\theta_{2} \right)} \ddot{\theta}_{1} + m_{2} r_{2}^{2} \ddot{\theta}_{1} + m_{2} r_{2}^{2} \ddot{\theta}_{2}\end{matrix}\right]$

## Zadaca 16

```python
m1, m2, g = sp.symbols('m1, m2, g')
l, r, theta = mechanics.dynamicsymbols('l, r, theta')
vl, vr, vtheta = l.diff('t'), r.diff('t'), theta.diff('t')
```

```python
k1 = m1 * vl**2 / 2
k1
```

```python
k2 = m2 * (vl**2 + vr**2) + m2*r**2*vtheta**2/2
k = k1+k2
k
```

```python
p = m1*g*l+m2*g*l
p.simplify()
```

```python
L = k-p
L
```

```python
rt.lagrangian(L, [l, r, theta])
```

$\displaystyle \left[\begin{matrix}g m_{1} + g m_{2} + m_{1} \ddot{l} + 2 m_{2} \ddot{l}\\m_{2} \left(- r \dot{\theta}^{2} + 2 \ddot{r}\right)\\m_{2} \left(r \ddot{\theta} + 2 \dot{r} \dot{\theta}\right) r\end{matrix}\right]$

## Zadaca 26

```python
H = sp.Matrix([
    [0.7071, 0.7071, 0, 2],
    [0, 0, 1, 1],
    [0.7071, -0.7071, 0, 5],
    [0, 0, 0, 1],
])
H
```

$\displaystyle \left[\begin{matrix}0.7071 & 0.7071 & 0 & 2\\0 & 0 & 1 & 1\\0.7071 & -0.7071 & 0 & 5\\0 & 0 & 0 & 1\end{matrix}\right]$

```python
R = H[:3, :3]
P = H[:3, 3]
R
```

$\displaystyle \left[\begin{matrix}0.7071 & 0.7071 & 0\\0 & 0 & 1\\0.7071 & -0.7071 & 0\end{matrix}\right]$

```python
F = sp.Matrix([10, 0, 5, 15, 20, 0])
F_wrt_U = F[:3,:]
M_wrt_U = F[3:,:]
```

```python
R.T * F_wrt_U,  R.T * (F_wrt_U.cross(P) + M_wrt_U)
```

$\displaystyle \left( \left[\begin{matrix}10.6065\\3.5355\\0\end{matrix}\right], \  \left[\begin{matrix}14.142\\0\\-20\end{matrix}\right]\right)$

```python
m1, m2, g = sp.symbols('m1, m2, g')
l, r, theta = mechanics.dynamicsymbols('l, r, theta')
robot = rt.SerialLinkRobot()
robot.add_revolute_joint(theta, 0, 0,0)
robot.add_prismatic_joint(0, l, 0, sp.pi/2)
robot.add_prismatic_joint(0, r, 0, 0)
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
x2, y2, z2 = robot.get_dh_matrix()[:3,3]
v2_squared = x2.diff('t')**2 + y2.diff('t')**2 + z2.diff('t')**2
K2 = m2 * v2_squared / 2
K2 = sp.simplify(K2)
K2
```

```python
x2, y2, z2
```

```python
l1, l2, l3 = sp.symbols('l1, l2, l3')
theta1, d2, theta3 = mechanics.dynamicsymbols('theta1, d2, theta3')
robot = rt.SerialLinkRobot()
robot.add_revolute_joint(theta1, 0, l1,-sp.pi/2)
robot.add_prismatic_joint(sp.pi/2, l2, 0, sp.pi/2)
robot.add_prismatic_joint(sp.pi/2, d2, 0, -sp.pi/2)
robot.add_revolute_joint(theta3, 0, l3, 0)
robot.add_subs([(l1,1),(l2,0.5),(l3,1)])
robot.interact()
```

    VBox(children=(HBox(children=(ToggleButton(value=True, description='Исклучи цртање'), Button(description='Пром…

```python
Jv = robot.linear_jacobian()
Jv
```

$\displaystyle \left[\begin{matrix}- l_{1} \sin{\left(\theta_{1} \right)} - l_{2} \cos{\left(\theta_{1} \right)} - l_{3} \cos{\left(\theta_{1} + \theta_{3} \right)} - d_{2} \sin{\left(\theta_{1} \right)} & - \sin{\left(\theta_{1} \right)} & \cos{\left(\theta_{1} \right)} & - l_{3} \cos{\left(\theta_{1} + \theta_{3} \right)}\\l_{1} \cos{\left(\theta_{1} \right)} - l_{2} \sin{\left(\theta_{1} \right)} - l_{3} \sin{\left(\theta_{1} + \theta_{3} \right)} + d_{2} \cos{\left(\theta_{1} \right)} & \cos{\left(\theta_{1} \right)} & \sin{\left(\theta_{1} \right)} & - l_{3} \sin{\left(\theta_{1} + \theta_{3} \right)}\\0 & 0 & 0 & 0\end{matrix}\right]$

```python
F = sp.Matrix([20, 10, 5])
F
```

$\displaystyle \left[\begin{matrix}20\\10\\5\end{matrix}\right]$

```python
T = Jv.T * F
T
```

$\displaystyle \left[\begin{matrix}- 20 l_{1} \sin{\left(\theta_{1} \right)} + 10 l_{1} \cos{\left(\theta_{1} \right)} - 10 l_{2} \sin{\left(\theta_{1} \right)} - 20 l_{2} \cos{\left(\theta_{1} \right)} - 10 l_{3} \sin{\left(\theta_{1} + \theta_{3} \right)} - 20 l_{3} \cos{\left(\theta_{1} + \theta_{3} \right)} - 20 d_{2} \sin{\left(\theta_{1} \right)} + 10 d_{2} \cos{\left(\theta_{1} \right)}\\- 20 \sin{\left(\theta_{1} \right)} + 10 \cos{\left(\theta_{1} \right)}\\10 \sin{\left(\theta_{1} \right)} + 20 \cos{\left(\theta_{1} \right)}\\10 l_{3} \sin{\left(\theta_{1} + \theta_{3} \right)} - 20 l_{3} \cos{\left(\theta_{1} + \theta_{3} \right)}\end{matrix}\right]$

```python
T.subs(theta1, sp.pi/6).subs(d2, 2).subs(theta3, sp.pi/4).subs(l1, 1).subs(l2, 0.5).subs(l3, 1).evalf()
```

$\displaystyle \left[\begin{matrix}-30.0151310892523\\-1.33974596215561\\22.3205080756888\\4.48287736084027\end{matrix}\right]$

```python

```


```python
%load_ext autoreload
%autoreload 2
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
import matplotlib.cm as cm
from IPython.display import display, Math, clear_output

import sympy  #Sympy1.2 is from conda-forge: [sudo conda install -c conda-forge sympy]
from sympy import *
from sympy.physics.vector import ReferenceFrame, CoordinateSym
from sympy.vector import CoordSys3D, divergence, curl

import ipyvolume as ipv
import time
from ipywidgets import Output

import helpers

np.seterr(divide='ignore', invalid='ignore');
```

## Symbolic Calculation of the Gradient

```python
#define symbolic vars, function
x,y,z =sympy.symbols('x y z')
fun = x**2

# evaluate the gradient symbolically
gradfun = [sympy.diff(fun, var) for var in (x,y,z)]
print('Gradient of: {}'.format(fun))
print(gradfun)
print(np.shape(gradfun))

#turn into a bivariate lambda for numpy
numgradfun = sympy.lambdify([x,y,z], gradfun)

x = np.linspace(-10, 10, 41)
y = np.linspace(-10, 10, 41)
z = np.linspace(-10, 10, 41)
X, Y, Z = np.meshgrid(x,y,z, indexing='ij')

gradient_eval = sum(numgradfun(X, Y, Z))

print('{}, {}, {}'.format(np.shape(X),np.shape(Y),np.shape(Z)))
print(np.shape(gradient_eval))
print(numgradfun(1,1,2))
print(sum(numgradfun(1,1,2)))
ipv.figure()
ipv.volshow(gradient_eval.T, opacity=0.03, level_width=0.1, data_min=0, data_max=1)
ipv.view(azimuth = 115, elevation = 90)
ipv.show()
```

    Gradient of: x**2
    [2*x, 0, 0]
    (3,)
    (41, 41, 41), (41, 41, 41), (41, 41, 41)
    (41, 41, 41)
    [2, 0, 0]
    2

    A Jupyter Widget


```julia
using PyPlot
using Dierckx
using PiecewiseInterpolation
```

Define function with first order discontinuity.
\begin{equation}
f(t) = \mathrm{e}^{-10|t|}
\end{equation}

```julia
f = t -> exp(-10abs(t))
t = linspace(-1, 1, 1000)
figure(figsize=(4, 3))
plot(t, f.(t))
times = collect(linspace(-1, 1, 30))
values = f.(times)
plot(times, values, "o", label="input data")
xlabel(L"t")
ylabel(L"f(t)")
```

Create naive and piecewise spline (with a single discontinuity at $t=0$):

```julia
ff = Spline1D(times, values, k=3)
p = PiecewiseSpline1D(times, values, [0.0])
;
```

Comparison of exact function, input data, and both interpolations.

```julia
figure(figsize=(4, 3))
plot(times, values, "o", label="input data")
new_times = linspace(-1, 1, 1001)
plot(new_times, f.(new_times), label="exact")
plot(new_times, ff.(new_times), label="naive")
plot(new_times, p.(new_times), "--", label="piecewise")
legend()
```

Absolute errors of both interpolations (vs exact function).

```julia
figure(figsize=(4, 3))
plot(new_times, ff.(new_times) - f.(new_times), label="naive")
plot(new_times, p.(new_times) - f.(new_times), label="piecewise")
ylabel("absolute error")
legend()
axhline(0.0, color="k", lw=0.5)
```

Relative errors of both interpolations (vs exact function).

```julia
figure(figsize=(4, 3))
plot(new_times, (ff.(new_times) - f.(new_times)) ./ f.(new_times), label="naive")
plot(new_times, (p.(new_times) - f.(new_times)) ./ f.(new_times), label="piecewise")
ylabel("relative error")
legend()
axhline(0.0, color="k", lw=0.5)
```

```julia

```


# Third Party Libraries With Rich Output

A number of third party libraries defined their own custom display logic. This gives their objcts rich output by default when used in the Notebook.

```python
from IPython.display import display
```

## Pandas

[Pandas](http://pandas.pydata.org/) is a data analysis library for Python. Its `DataFrame` objects have an HTML table representation in the Notebook.

```python
import pandas
```

Here is a small amount of stock data for APPL:

```python
%%writefile data.csv
Date,Open,High,Low,Close,Volume,Adj Close
2012-06-01,569.16,590.00,548.50,584.00,14077000,581.50
2012-05-01,584.90,596.76,522.18,577.73,18827900,575.26
2012-04-02,601.83,644.00,555.00,583.98,28759100,581.48
2012-03-01,548.17,621.45,516.22,599.55,26486000,596.99
2012-02-01,458.41,547.61,453.98,542.44,22001000,540.12
2012-01-03,409.40,458.24,409.00,456.48,12949100,454.53
```

Read this as into a `DataFrame`:

```python
df = pandas.read_csv('data.csv')
```

And view the HTML representation:

```python
df
```

## SymPy

[SymPy](http://sympy.org/) is a symbolic computing library for Python. Its equation objects have LaTeX representations that are rendered in the Notebook.

```python
from sympy.interactive.printing import init_printing
init_printing(use_latex='mathjax')
```

```python
from __future__ import division
import sympy as sym
from sympy import *
x, y, z = symbols("x y z")
k, m, n = symbols("k m n", integer=True)
f, g, h = map(Function, 'fgh')
```

```python
Rational(3,2)*pi + exp(I*x) / (x**2 + y)
```

```python
a = 1/x + (x*sin(x) - 1)/x
a
```

```python
(1/cos(x)).series(x, 0, 6)
```

## Vincent

[Vincent](https://vincent.readthedocs.io/en/latest/) is a visualization library that uses the [Vega](http://trifacta.github.io/vega/) visualization grammar to build [d3.js](http://d3js.org/) based visualizations in the Notebook and on http://nbviewer.ipython.org. `Visualization` objects in Vincetn have rich HTML and JavaSrcript representations.

```python
import vincent
import pandas as pd
```

```python
vincent.initialize_notebook()
```

```python
line = vincent.Line(df[['Open', 'High', 'Close']], width=600, height=300)
line.axis_titles(x='Date', y='Price')
line.legend(title='Ticker')
display(line)
```

## Cleanup

```python
!rm -f data.csv
```

```python

```

001-Jupyter/001-Tutorials/003-IPython-in-Depth/examples/IPython Kernel/Third Party Rich Output.ipynb


# Spectral Estimation of Random Signals

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## The Periodogram

The [periodogram](https://en.wikipedia.org/wiki/Spectral_density_estimation#Periodogram) is an estimator for the power spectral density (PSD) $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of a random signal $x[k]$. For the following it is assumed that $x[k]$ is drawn from a wide-sense ergodic real-valued random process.

### Definition

The PSD is defined as the [discrete-time Fourier transformation (DTFT) of the auto-correlation function (ACF)](../random_signals/power_spectral_densities.ipynb#Definition)

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathcal{F}_* \{ \varphi_{xx}[\kappa] \}
\end{equation}

Hence, the PSD can be computed from an estimate of the ACF. Let's assume that we want to estimate the PSD from $N$ samples of the random signal $x[k]$ by way of the ACF. The signal $x[k]$ is truncated to $N$ samples by multiplication (windowing) with the rectangular signal $\text{rect}_N[k]$ of length $N$

\begin{equation}
x_N[k] = x[k] \cdot \text{rect}_N[k]
\end{equation}

where $x_N[k]$ denotes the truncated signal.
The ACF is estimated by applying its definition in a straightforward manner. For a random signal $x_N[k]$ of finite length, the estimated ACF $\hat{\varphi}_{xx}[\kappa]$ can be expressed [in terms of a convolution](../random_signals/correlation_functions.ipynb#Definition)

\begin{equation}
\hat{\varphi}_{xx}[\kappa] = \frac{1}{N} \cdot x_N[k] * x_N[-k]
\end{equation}

Applying the DTFT to both sides and rearranging the terms yields

\begin{equation}
\hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{N} \, X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega})\, X_N(\mathrm{e}^{\,-\mathrm{j}\,\Omega}) =
\frac{1}{N} \, | X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) |^2
\end{equation}

where the intermediate and last equalities result from the symmetry relations of the DTFT. This estimate of the PSD is known as the periodogram. It can be computed directly from the DTFT

\begin{equation}
X_N(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k=0}^{N-1} x_N[k] \, \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k}
\end{equation}

of the truncated random signal.

### Example - Periodogram

The following example estimates the PSD of a random process which draws samples from normal distributed white noise with zero mean and unit variance. The true PSD is hence given as $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1$. In order to compute the periodogram by the discrete Fourier transformation (DFT), the signal $x[k]$ has to be zero-padded to ensure that squaring (multiplying) the spectra does not result in a circular convolution.

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

N = 128  # number of samples

# generate random signal
np.random.seed(5)
x = np.random.normal(size=N)

# compute magnitude of the periodogram
x = np.concatenate((x, np.zeros_like(x)))
X = np.fft.rfft(x)
Om = np.linspace(0, np.pi, len(X))
Sxx = 1/N * abs(X)**2

# plot results
plt.figure(figsize=(10,4))
plt.stem(Om, Sxx, 'C0', label=r'$|\hat{\Phi}_{xx}(e^{j \Omega})|$', use_line_collection=True)
plt.plot(Om, np.ones_like(Sxx), 'C1', label=r'$\Phi_{xx}(e^{j \Omega})$')
plt.title('Estimated and true PSD')
plt.xlabel(r'$\Omega$')
plt.axis([0, np.pi, 0, 4])
plt.legend()

# compute mean value of the periodogram
print('Mean value of the periodogram: {0:1.3f}'.format(np.mean(np.abs(Sxx))))
print('Variance of the periodogram: {0:1.3f}'.format(np.var(np.abs(Sxx))))
```

    Mean value of the periodogram: 1.024
    Variance of the periodogram: 0.791

**Exercise**

* What do you have to change to evaluate experimentally if the periodogram is a consistent estimator?
* Based on the results, is the periodogram a consistent estimator?

Solution: The conditions for consistency have to be checked for the limiting case of a infinitely long signal ($N \to \infty$). Increasing the length `N` of the random signal in above example reveals that the periodogram can be assumed to be bias free, $b_{\hat{\Phi}_{xx}} = 0$. However, its variance $\sigma^2_{\hat{\Phi}_{xx}}$ does not tend to zero. The reason for this is that by increasing the length $N$ of the random signal also the number of spectral coefficients is increased by the same amount.

### Evaluation

From above numerical example it should have become clear that the periodogram is no consistent estimator for the PSD $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. It can be shown that the estimator is asymptotically bias free for $N \to \infty$, hence

\begin{equation}
\lim_{N \to \infty} E\{ \hat{\Phi}_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})  \} = \Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

This is due to the [leakage effect](../spectral_analysis_deterministic_signals/leakage_effect.ipynb) which limits the spectral resolution for signals of finite length.

The variance of the estimator does not converge towards zero

\begin{equation}
\lim_{N \to \infty} \sigma^2_{\hat{\Phi}_{xx}} \neq 0
\end{equation}

This is due to the fact that by increasing $N$ also the number of independent frequencies $\Omega = \frac{2 \pi}{N} \mu$ for $\mu = 0,1,\dots,N-1$ increases.

The periodogram is the basis for a variety of advanced estimation techniques for the PSD. These techniques rely on averaging or smoothing of (overlapping) periodograms.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


```python
import sympy as sm
import sympy.physics.mechanics as me
me.init_vprinting()
```

```python
A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')

theta = me.dynamicsymbols('theta')

B.orient_axis(A, theta, A.y)
```

```python
Q = 3*me.outer(B.x, A.z) + 4*me.outer(A.x, A.x) + 6*me.outer(A.z, B.z)
```

```python
Q
```

```python
Q.express(A)
```

```python
Q.express(B)
```

```python
Q.to_matrix(A)
```

```python
Q.to_matrix(B)
```

```python
M1, M2, M3 = me.dynamicsymbols('M1, M2, M3')

A = me.ReferenceFrame('A')
B = me.ReferenceFrame('B')

w1, w2, w3 = me.dynamicsymbols('omega1, omega2, omega3')
```

```python
A_w_B = w1*B.x + w2*B.y + w3*B.z
A_w_B
```

```python
M = M1*B.x + M2*B.y + M3*B.z
M
```

```python
I11, I22, I33 = sm.symbols('I_{11}, I_{22}, I_{33}')
```

```python
I = I11*me.outer(B.x, B.x) + I22*me.outer(B.y, B.y) + I33*me.outer(B.z, B.z)
I
```

```python
H = me.dot(I, A_w_B)
H
```

```python
t = me.dynamicsymbols._t

Hdot = H.diff(t, B) + me.cross(A_w_B, H)
Hdot
```

```python
M - Hdot
```

```python
sm.Eq(M.to_matrix(B), Hdot.to_matrix(B))
```

```python
A_w_B.diff(t, B)
```

# Bicycle example

```python
B = me.ReferenceFrame('B')
H = me.ReferenceFrame('H')
```

```python
B_C_H = sm.Matrix([[0, -1, 0],
                   [-1, 0, 0],
                   [0, 0, -1]])
B_C_H
```

```python
B.orient_explicit(H, B_C_H.transpose())
```

```python
mH = 84
mB = 9.9
```

```python
r_O_Ho = 0.4*B.x - 1.1*B.z
r_O_Bo = 0.3*B.x - 0.5*B.z
```

```python
r_O_Fo = (mB*r_O_Bo + mH*r_O_Ho)/(mB + mH)
r_O_Fo
```

```python
I_H_Ho = me.inertia(H, 11.3, 11.0, 2.3, iyz=-1.7)
I_B_Bo = me.inertia(B, 0.5, 1.3, 0.8, izx=-0.1)
```

```python
I_H_Ho
```

```python
r_Fo_Ho = r_O_Fo - r_O_Ho
U = me.outer(H.x, H.x) + me.outer(H.y, H.y) + me.outer(H.z, H.z)
I_Ho_Fo = mH*(r_Fo_Ho.dot(r_Fo_Ho)*U -
              me.outer(r_Fo_Ho, r_Fo_Ho))
I_Ho_Fo
```

```python
r_Fo_Bo = r_O_Fo - r_O_Bo
I_Bo_Fo = mH*(r_Fo_Bo.dot(r_Fo_Bo)*U -
              me.outer(r_Fo_Bo, r_Fo_Bo))
I_Bo_Fo
```

```python
I_B_Fo = I_B_Bo + I_Bo_Fo
I_H_Fo = I_H_Ho + I_Ho_Fo
```

```python
I_F_Fo = I_B_Fo + I_H_Fo
I_F_Fo
```

```python
I_F_Fo.to_matrix(B)
```

```python

```


# 2017-09-15

Homework diode voltage temperature dependence.

Input our symbolic expression for $v_D$

```python
from sympy import *
init_printing()

T, iD, C, k, q, Eg = symbols('T iD C k q Eg')

vD = (k / q) * T * (ln(iD/C) - 4*ln(T)) + (Eg / q)
vD
```

Differentiate with respect to temperature:

```python
dvD = diff(vD, T)
dvD
```

Substitute specific values into this result and evaluate at $T = 300\,\mathrm{K}$:

```python
# replace with numbers

vals ={k: 1.38e-23,  # J / K
       q: 1.602e-19, # C
       Eg: 1.12 * 1.602e-19,  # J

       #hand computed from IS=1e-15 at T=300K
       C: 8.19e-7,  # A / T^4

       #set a specific current
       iD: 1e-3,  # A
      }

dvD_T = dvD.subs(vals)
dvD_T.subs(T, 300).evalf()
```

So, *this* silicon diode's forward voltage changes by $-1.7 \,\mathrm{mV / ^\circ C}$ from room temperature.

Look at this forward voltage and slope over the industrial temperature range $-40$ to $125\,\mathrm{^\circ C}$:

```python
# get vD as a function of T only
vD_T = vD.subs(vals)
```

```python
#setup numerical and plotting packages
import matplotlib as mpl
import matplotlib.pyplot as plt
import numpy as np

#plt.style.use('seaborn-bright')
mpl.rcParams['figure.figsize'] = (15.0, 8.0)
mpl.rcParams['font.size'] = 24
mpl.rcParams['lines.linewidth'] = 5
mpl.rcParams['lines.markersize'] = 5
```

```python
tempC = np.linspace(-40, 125, 1000)

#turn symbolic expression into a numpy function of one argument
# result is equivalent to:
#    def vD_func(T):
#        ...
vD_func = lambdify(T, vD_T, 'numpy')
```

```python
#plot!

fig, ax1 = plt.subplots()

ax1.plot(tempC, vD_func(tempC + 273.14), 'r')  # careful of K and degC!

ax1.set_xlim([min(tempC), max(tempC)])
ax1.set_xlabel('Industrial temp ($^\circ C$)')
ax1.set_ylabel('$v_D$ ($\mathrm{V}$)')

ax1.set_ylim([0.5, 0.9])

plt.grid(True)
plt.show()
```

With this plot range, there is no curvature!  Expand the range to see the curve:

```python
t = np.linspace(-100, 400, 1000)
plt.plot(t, vD_func(t + 273.14))  # careful of K and degC!
plt.plot(tempC, vD_func(tempC + 273.14), 'r')  # careful of K and degC!

plt.xlim([min(t), max(t)])
plt.xlabel('Industrial temp ($^\circ C$)')
plt.ylabel('$v_D$ ($\mathrm{V}$)')

plt.grid(True)
plt.show()
```

We can conclude with *high confidence* that it is a very good approximation to say that for a constant current, a diode's forward voltage drops linearly by $1.7 \,\mathrm{mV / ^\circ C}$.

```python

```


## Tutorial 5: Newton-Raphson method
Calculating equilibrium composition from given free energies.

```python
import numpy as np
import matplotlib.pyplot as plt
import sympy as sym
```

### Free energies
$$ g_s = (c_s - 0.5)^2 + 0.5c_s - 0.1 $$
$$ g_l = (c_l - 0.1)^2 $$

```python
c_s = sym.Symbol('c_s')
c_l = sym.Symbol('c_l')
```

```python
g_s = (c_s - 0.5)**2 + 0.5*c_s - 0.1
g_l = (c_l - 0.1)**2
print("g_s = ", g_s)
print("g_l = ", g_l)
print("g_s = ", g_s.expand())
print("g_l = ", g_l.expand())
```

    g_s =  0.5*c_s + (c_s - 0.5)**2 - 0.1
    g_l =  (c_l - 0.1)**2
    g_s =  c_s**2 - 0.5*c_s + 0.15
    g_l =  c_l**2 - 0.2*c_l + 0.01

### Plot free energies

```python
c = np.linspace(0.01, 0.99, 100)
# converting symbolic expression to function, just for plotting
g_s_func = sym.lambdify(c_s, g_s)
g_l_func = sym.lambdify(c_l, g_l)
plt.plot(c, g_s_func(c), label='g_s')
plt.plot(c, g_l_func(c), label='g_l')
plt.legend()
plt.show()
```

### Common tangent construction
To get equilibrium concentrations $ (c_s^{eq}, c_l^{eq}) $:
1. Equate the slopes:
$$ f_1 = \Big(\frac{\partial g_s}{\partial c}\Big)_{c_s^{eq}} - \Big(\frac{\partial g_l}{\partial c}\Big)_{c_l^{eq}} = 0 $$
2. Equate the intercept:
$$ f_2 = \Big[g_s - c_s^{eq}\Big(\frac{\partial g_s}{\partial c}\Big)_{c_s^{eq}}\Big] - \Big[g_l - \Big(c_l^{eq}\frac{\partial g_l}{\partial c}\Big)_{c_l^{eq}}\Big] = 0 $$

```python
df_s = sym.diff(g_s, c_s)
df_l = sym.diff(g_l, c_l)
int_s = g_s - c_s*df_s
int_l = g_l - c_l*df_l
f1 = df_s - df_l
f2 = int_s - int_l
```

```python
print("f1 = ", f1)
print("f1 = ", f1.simplify())
print("f2 = ", f2)
print("f2 = ", f2.simplify())
```

    f1 =  -2*c_l + 2*c_s - 0.3
    f1 =  -2*c_l + 2*c_s - 0.3
    f2 =  c_l*(2*c_l - 0.2) - c_s*(2*c_s - 0.5) + 0.5*c_s - (c_l - 0.1)**2 + (c_s - 0.5)**2 - 0.1
    f2 =  1.0*c_l**2 - 1.0*c_s**2 + 0.14

```python
sym.diff(f2, c_l)
```

    2*c_l

### Newton Raphson iterations
$$ \begin{bmatrix} c_s \\ c_l \end{bmatrix}_{i+1} = \begin{bmatrix} c_s \\ c_l \end{bmatrix}_{i} -  \begin{bmatrix} \dfrac{\partial f_1}{\partial c_s} & \dfrac{\partial f_1}{\partial c_l} \\ \dfrac{\partial f_2}{\partial c_s} & \dfrac{\partial f_2}{\partial c_l} \end{bmatrix}^{-1} \begin{bmatrix} f_1 \\ f_2 \end{bmatrix} $$

```python
total_iterations = 10
c_guess = np.array( [0.1, 0.11] ) # (c_s, c_l)
c = c_guess
jacobian = np.array( [ [ sym.diff(f1, c_s), sym.diff(f1, c_l) ], [ sym.diff(f2, c_s), sym.diff(f2, c_l)] ] )
print("jacobian = \n", jacobian)

jacobian = np.array( [ [ sym.diff(f1, c_s).evalf(subs={ c_s:c[0], c_l:c[1] }),
                            sym.diff(f1, c_l).evalf(subs={ c_s:c[0], c_l:c[1] }) ],
                        [ sym.diff(f2, c_s).evalf(subs={ c_s:c[0], c_l:c[1] }) ,
                         sym.diff(f2, c_l).evalf(subs={ c_s:c[0], c_l:c[1] })] ] )
```

    jacobian =
     [[2 -2]
     [-2*c_s 2*c_l]]

```python
for i in range(total_iterations):
    jacobian = np.array( [ [ sym.diff(f1, c_s).evalf(subs={ c_s:c[0], c_l:c[1] }),
                            sym.diff(f1, c_l).evalf(subs={ c_s:c[0], c_l:c[1] }) ],
                        [ sym.diff(f2, c_s).evalf(subs={ c_s:c[0], c_l:c[1] }) ,
                         sym.diff(f2, c_l).evalf(subs={ c_s:c[0], c_l:c[1] })] ] )
    jacobian = jacobian.astype(np.float64) # converting datatype to float64 format
    inv_jacobian = np.linalg.inv(jacobian)
    c = c - np.dot(inv_jacobian, np.array( [f1.evalf(subs={ c_s:c[0], c_l:c[1] }) ,
                                            f2.evalf(subs={ c_s:c[0], c_l:c[1] })  ] ) )
    print(i+1, c)
```

    1 [-5.24500000000001 -5.39500000000001]
    2 [0.541666666666726 0.391666666666726]
    3 [0.541666666666667 0.391666666666667]
    4 [0.541666666666667 0.391666666666667]
    5 [0.541666666666667 0.391666666666667]
    6 [0.541666666666667 0.391666666666667]
    7 [0.541666666666667 0.391666666666667]
    8 [0.541666666666667 0.391666666666667]
    9 [0.541666666666667 0.391666666666667]
    10 [0.541666666666667 0.391666666666667]

### Using built in non-linear solver fsolve

```python
from scipy.optimize import fsolve
def equations(c):
    return ( f1.evalf(subs={ c_s:c[0], c_l:c[1] }), f2.evalf(subs={ c_s:c[0], c_l:c[1] }) )

c_eqbm = fsolve(equations, c_guess )
print(c_eqbm)
```

    [0.54166667 0.39166667]


# 2.4 Statistical Decision Theory

#### Loss function

The most common is *squared error loss*: $L(Y, f(X)) = (Y - f(X))^2$

#### Expected prediction Error
(2.9, 2.10) This leads us to a criterion for choosing $f$,

$$EPE(f) = E(Y - f(X))^2 = \int{[y - f(x)]^2}Pr(dx, dy)$$

(2.11) By conditioning on X, we can write EPE as:

$$EPE(f) = E_X E_{Y|X} ([Y - f(X)]^2|X) $$

Proof:

$$
\begin{equation}
 \int{[y - f(x)]^2}Pr(dx, dy) \\
 = \int{[y - f(x)]^2}p(x, y)dxdy \\
 = \int{[y - f(x)]^2}p(x)p(y | x)dxdy \\
 = \int_x { \left( \int_y {[y - f(x)]^2p(y | x)dy} \right)p(x)dx } \\
 = E_X E_{Y|X} ([Y - f(X)]^2|X)
\end{equation}
$$

(2.12) It suffices to minimize EPE pointwise: $$f(x) = argmin_c E_{Y|X} ( [Y - c]^2 | X = x)$$

(2.13) The solution is : $$f(x)=E(Y | X = x)$$

The **nearest-neighbor** methods attempt to directly implement this recipe
using the training data. Since there is typically at most one observation
at any point x, we settle for:  $$\hat{f}(x) = \text{Ave}(y_i|x_i \in N_k(x))$$

For large training sample size N, the points in the neighborhood are likely
to be close to x, and as k gets large the average will get more stable.

How does **linear regression fit** into this framework ?

$$f(x) \approx x^T\beta$$

Plugging this into EPE (2.9) and differentiating
we can solve for β:

$$\beta= [E(XX^T)]^{-1}E(XY)$$

Proof:

$$
\begin{equation}
 \int{[y - x^T\beta]^2\text{ Pr}(dx, dy)} \\
 = \int{[y^2 - 2yx^T+(x^T\beta)^2]\text{ Pr}(dx, dy)} \\
\end{equation}
$$

Differentiating w.r.t $\beta$:

$$
\begin{equation}
 2\int{xx^T\beta\text{ Pr}(dx, dy)} -  2\int{xy\text{ Pr}(dx, dy)} = 0 \\
 2 \times (E(XX^T\beta) - E(XY)) = 0 \\
 E(XX^T)\beta = E(XY) \\
 \beta = [E(XX^T)]^{-1}E(XY)
\end{equation}
$$

The least squares solution $\hat{\beta}=(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$   amounts to replacing the expectation in $\beta = [E(XX^T)]^{-1}E(XY)$ by averages over the training data. **Note:** Each expectation produces N times more than average, however the constant (i.e 1/N) in two expectations cancel out each other.

#### Bayes Classifier

```python
%matplotlib inline

import random
import numpy as np
import matplotlib.pyplot as plt

sample_size = 100

def generate_data(means, size):
    return np.array([
        np.random.multivariate_normal(random.choice(means), np.eye(2) / 5)
        for _ in range(size)
    ])

def plot_data(orange_data, blue_data):
    axes.plot(orange_data[:, 0], orange_data[:, 1], 'o', color='orange')
    axes.plot(blue_data[:, 0], blue_data[:, 1], 'o', color='blue')

blue_means = np.random.multivariate_normal([1, 0], np.identity(2), 10)
orange_means = np.random.multivariate_normal([0, 1], np.identity(2), 10)

blue_data = generate_data(blue_means, sample_size)
orange_data = generate_data(orange_means, sample_size)

# plotting
fig = plt.figure(figsize = (8, 8))
axes = fig.add_subplot(1, 1, 1)
plot_data(orange_data, blue_data)

plt.show()
```

```python
from itertools import filterfalse, product
from scipy.stats import multivariate_normal

def plot_grid(orange_grid, blue_grid):
    axes.plot(orange_grid[:, 0], orange_grid[:, 1], '.', zorder = 0.001,
          color='orange', alpha = 0.25, scalex = False, scaley = False)

    axes.plot(blue_grid[:, 0], blue_grid[:, 1], '.', zorder = 0.001,
          color='blue', alpha = 0.25, scalex = False, scaley = False)

def pdf(x, means):
    return np.mean([multivariate_normal.pdf(x, mean = m, cov = np.eye(2) / 5)
                    for m in means], axis = 0)

plot_xlim = axes.get_xlim()
plot_ylim = axes.get_ylim()

grid = np.array([*product(np.linspace(*plot_xlim, 50), np.linspace(*plot_ylim, 50))])

orange_pdf = pdf(grid, orange_means)
blue_pdf = pdf(grid, blue_means)

# Plotting
axes.clear()
axes.set_title("Bayes Optimal Classifier")

plot_data(orange_data, blue_data)
plot_grid(grid[orange_pdf >= blue_pdf], grid[orange_pdf < blue_pdf])

fig
```


 Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Kyle T. Mandli

Note:  The presentation below largely follows part II in "Finite Difference Methods for Ordinary and Partial Differential Equations" by LeVeque (SIAM, 2007).

```python
from __future__ import print_function

%matplotlib inline
import numpy
import matplotlib.pyplot as plt
```

# Numerical Solution to ODE Initial Value Problems - Part 1

Many physical, biological, and societal systems can be written as a system of ordinary differential equations (ODEs).  In the case where the initial state (value) is know the problems can be written as

$$
    \frac{\text{d}\mathbf{u}}{\text{d}t} = \mathbf{f}(t, \mathbf{u}) \quad \mathbf{u}(0) = \mathbf{u}_0
$$

where
 - $\mathbf{\!u}(t)$ is the state vector
 - $\mathbf{\!f}(t, \mathbf{\!u})$ is a vector-valued function that controls the growth of $\mathbf{u}$ with time
 - $\mathbf{\!u}(0)$ is the initial condition at time $t = 0$

#### Examples:  Simple radioactive decay
$$
    \mathbf{\!u} = [c]
$$

$$
    \frac{\text{d} c}{\text{d}t} = \lambda c \quad c(0) = c_0
$$

which has solutions of the form $c(t) = c_0 e^{\lambda t}$

```python
t = numpy.linspace(0.0, 3200., 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, 1.0 * numpy.exp(decay_constant * t))
axes.plot(1600.,0.5,'ro')
axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years", fontsize=18)
axes.set_xlabel('t (years)', fontsize=16)
axes.set_ylabel('$c$', fontsize=16)
axes.set_xlim((0.0, t[-1]))
axes.set_ylim((0.0, 1.1))
axes.grid()
plt.show()
```

#### Examples:  Complex radioactive decay (or chemical system).

Chain of decays from one species to another.

$$\begin{aligned}
    \frac{\text{d} c_1}{\text{d}t} &= -\lambda_1 c_1 \\
    \frac{\text{d} c_2}{\text{d}t} &= \lambda_1 c_1 - \lambda_2 c_2 \\
    \frac{\text{d} c_3}{\text{d}t} &= \lambda_2 c_3 - \lambda_3 c_3
\end{aligned}$$

$$\frac{\text{d} \mathbf{u}}{\text{d}t} = \frac{\text{d}}{\text{d}t}\begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix} =
\begin{bmatrix}
    -\lambda_1 & 0 & 0 \\
    \lambda_1 & -\lambda_2 & 0 \\
    0 & \lambda_2 & -\lambda_3
\end{bmatrix} \begin{bmatrix} c_1 \\ c_2 \\ c_3 \end{bmatrix}$$

$$\frac{\text{d} \mathbf{u}}{\text{d}t} = A \mathbf{u}$$

For systems of equations like this the general solution to the ODE is the matrix exponential:

$$\mathbf{u}(t) = \mathbf{u}_0 e^{A t}$$

#### Examples:  Particle tracking in a fluid

$$\frac{\text{d} \mathbf{X}}{\text{d}t} = \mathbf{V}(t, \mathbf{X})$$

In fact all ODE IVP systems can be thought of as tracking particles through a flow field (dynamical system).  In 1-dimension the flow "manifold" we are on is fixed by the initial condition.

```python
x = numpy.linspace(0., 1., 11)
y = numpy.linspace(0., 1., 11)
x_fine = numpy.linspace(0., 1.)
y_fine = numpy.linspace(0., 1.)

X, Y = numpy.meshgrid(x,y)
X_fine, Y_fine = numpy.meshgrid(x_fine, y_fine)

pi = numpy.pi
psi = numpy.sin(pi*X_fine)*numpy.sin(pi*Y_fine)
U = pi*numpy.sin(pi*X)*numpy.cos(pi*Y)
V = -pi*numpy.cos(pi*X)*numpy.sin(pi*Y)

x0 = 0.75
y0 = 0.75
psi0 = numpy.sin(pi*x0)*numpy.sin(pi*y0)

fig = plt.figure(figsize=(6,6))
axes = fig.add_subplot(1, 1, 1)
axes.quiver(X,Y, U, V)
axes.plot(.75, 0.75,'ro')
axes.contour(X_fine, Y_fine, psi, [ psi0 ])
axes.grid()
axes.set_title("Particle tracking", fontsize=18)
axes.set_xlabel('y', fontsize=16)
axes.set_ylabel('x', fontsize=16)

plt.show()
```

#### Examples: Van der Pol Oscillator

$$y'' - \mu (1 - y^2) y' + y = 0 \quad \quad \text{with} \quad \quad  y(0) = y_0, \quad y'(0) = v_0$$

$$\mathbf{u} = \begin{bmatrix} y \\ y' \end{bmatrix} = \begin{bmatrix} u_1 \\ u_2 \end{bmatrix}$$

$$\frac{\text{d}}{\text{d}t} \begin{bmatrix} u_1 \\ u_2 \end{bmatrix} = \begin{bmatrix} u_2 \\ \mu (1 - u_1^2) u_2 - u_1 \end{bmatrix} = \mathbf{f}(t, \mathbf{u})$$

```python
from scipy.integrate import solve_ivp

def f_vanderpol(t, u, mu=5):
    return numpy.array([u[1], mu * (1.0 - u[0]**2) * u[1] - u[0]])

# N = 100
N = 500
t_span = (0., 200.)
u0 = [ 1., 0. ]
f = lambda t, u : f_vanderpol(t, u, mu=50)
sol = solve_ivp(f, t_span, u0, method='BDF')
```

```python
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)

axes.plot(sol.t, sol.y[0])
axes.set_title("Solution to Van der Pol Oscillator", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("y(t)", fontsize=16)
axes.grid()

axes = fig.add_subplot(1, 2, 2)

axes.plot(sol.y[0],sol.y[1],'r')
axes.set_title("Phase Diagram for Van der Pol Oscillator", fontsize=18)
axes.set_xlabel("y(t)", fontsize=16)
axes.set_ylabel("y'(t)", fontsize=16)
axes.grid()
plt.show()
```

## Basic Stepping Schemes

Introducing some notation to simplify things
$$\begin{aligned}
    t_0 &= 0 \\
    t_1 &= t_0 + \Delta t \\
    t_n &= t_{n-1} + \Delta t = n \Delta t + t_0 \\
    u_0 &= u(t_0) \approx U_0 \\
    u_1 &= u(t_1) \approx U_1 \\
    u_n &= u(t_n) \approx U_2 \\
\end{aligned}$$
where lower-case letters are "exact".

Looking back at our work on numerical differentiation why not approximate the derivative as a finite difference:

$$
    \frac{u(t + \Delta t) - u(t)}{\Delta t} = f(t, u)
$$

We still need to decide how to evaluate the $f(t, u)$ term however.

#### Relationship to quadrature

Actually it is often more instructive to work with the integral form of an ODE

$$
    \int^{t + \Delta t}_t \frac{\text{d} u}{\text{d}\tilde{\!t}} d\tilde{\!t} = \int^{t + \Delta t}_t f(t, u) d\tilde{\!t}
$$

which is equivalent to the differential form.  However using the fundamental theorem of calculus tells us that the LHS is $u(t + \Delta t) - u(t)$  so we can write the ODE as

$$
    u(t + \Delta t) = u(t) + \int^{t + \Delta t}_t f(\tilde{\!t}, u(\tilde{\!t})) d\tilde{\!t}
$$

#### The Geometric picture

$$
    \frac{\text{d} u}{\text{d} t} = f(t,u)
$$
says $f$ is a slope field that is tangent to any solution $u(t)$ that passes through $u_0$ at $t=t_0$

```python
t = numpy.linspace(0., 4800, 11)
y = numpy.linspace(0., 1.2, 11)
T, Y = numpy.meshgrid(t,y)
dt = numpy.ones(T.shape)
dy = -dt*Y

tp = numpy.linspace(0., 4800, 100)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.quiver(T,Y, dt,dy)
axes.plot(tp,numpy.exp(decay_constant*tp))
axes.plot(0.,1.,'ro')
axes.grid()
axes.set_title("Direction Set, $u' = - \lambda u$, $u(0)=1$", fontsize=18)
axes.set_xlabel('t (years)', fontsize=16)
axes.set_ylabel('u', fontsize=16)

axes.set_ylim((-.1,1.2))
plt.show()

```

Alternatively

$$
    u(t + \Delta t) = u(t) + \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t})) d\tilde{\!t}
$$
Implies, that $u$ at some time $\Delta t$ in the future,  is $u(t)$ plus a *number*
$$
    K = \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t}) )d\tilde{\!t}
$$
which is a definite *line integral* (along an unknown solution).  As we will see many of the algorithms for integrating ODE's can be related to simple quadrature rules for approximating $K$

## Single-Step Multi-Stage Schemes

The integral form of an ODE initial value problem can be written

$$
    u(t + \Delta t) = u(t) + \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t})) d\tilde{\!t}
$$

Which says that our solution $u$, if it exists  at some time $\Delta t$ in the future,  is $u(t)$ plus a *number*
$$
    K = \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t}) )d\tilde{\!t}
$$
which is a definite *line integral* (along an unknown solution).

An important class of ODE solvers are called *Single Step, Multi-stage schemes* which can be most easily understood as extensions of the  Newton-Cotes quadrature schemes for approximating $K$ (plus an error term that will scale as $\Delta t^p$)

### The Geometric picture

```python
t = numpy.linspace(0., 4800, 11)
y = numpy.linspace(0., 1.2, 11)
T, Y = numpy.meshgrid(t,y)
dt = numpy.ones(T.shape)
dy = -dt*Y

tK = 2000.
uK = numpy.exp(decay_constant*tK)
K = uK -1.
tp = numpy.linspace(0., 4800, 100)
tk = numpy.linspace(0., tK, 100)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.quiver(T,Y, dt,dy)
axes.plot(tp,numpy.exp(decay_constant*tp))
axes.plot(0.,1.,'ro')
axes.plot(tk,numpy.exp(decay_constant*tk),'r--')
axes.plot(tK, uK, 'ro')
axes.plot([0.,0.], [1., uK], 'r--')
axes.text(10., 0.72, '$K$', fontsize=24, color='red')
axes.plot([0.,tK],[uK, uK], 'r--')
axes.text(900., uK - .1, '$\Delta t$', fontsize=24, color='red')
axes.text(-10, 1., '$U_0$', fontsize=24, color='blue')
axes.text(tK+10, uK, '$U_1$', fontsize=24, color='blue')

axes.grid()
axes.set_title("Direction Set, $u' = - \lambda u$, $u(0)=1$", fontsize=18)
axes.set_xlabel('t (years)', fontsize=16)
axes.set_ylabel('u', fontsize=16)

axes.set_ylim((-.1,1.2))
plt.show()

```

#### Forward Euler scheme
For example, if we approximate $K$ with a left-sided quadrature rule
$$
    K = \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t})) d\tilde{\!t} \approx \Delta t f(t, u(t))
$$

then our first ODE algorithm can be written
$$
u(t + \Delta t) =  u(t) + \Delta t f(t, u(t))
$$

or in terms of our discrete approximation $U$
$$
\begin{align}
    K_1 &= \Delta t f(t_n, U_n)\\
    U_{n+1} &= U_n + K_1\\
\end{align}
$$

which is known as the *forward Euler method*.  In essence we are approximating the derivative with the value of the function at the point we are at $t_n$.

```python
t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

# Euler step
dt = 1e3
u_np = c_0 + dt * (decay_constant * c_0)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., 1., 'ro')
axes.text(0., 1.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot((dt, dt), (u_np, c_0 * numpy.exp(decay_constant * dt)), 'k--')
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.text(10., 0.75, '$K_1$', fontsize=16)
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

```python
# Implement Forward Euler
def euler(f, t_span, u0, N):
    """ simple implementation of constant step-size forward euler method
        This doc string should have so much more in it
    """
    t = numpy.linspace(t_span[0], t_span[1],N)
    u = numpy.empty(t.shape)
    u[0] = u0
    delta_t = t[1] - t[0]
    for (n, t_n) in enumerate(t[:-1]):
        K1 = delta_t * f(t_n, u[n])
        u[n + 1] = u[n] + K1
    return t, u
```

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

t_span = [0.0, 1.6e3]
u0 = 1.
N = 20
t_euler, u_euler = euler(f, t_span, u0, N)

```

```python
t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = c_0 * numpy.exp(decay_constant * t_exact)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_euler, u_euler, 'or', label="Euler")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")

axes.set_title("Forward Euler")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.grid()
axes.legend()
plt.show()
```

### Backward's Euler

Similar to forward Euler is the *backward Euler* method which,   uses a right-rectangle rule to estimate $K$ given $f$ at a future time. i.e.

$$
    K\approx \Delta t f(t_{n+1}, U_{n+1})
$$

However, the update scheme now becomes
$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}).
$$

which requires a (usually non-linear) solve for $U_{n+1}$. Schemes where the function $f$ is evaluated at the unknown time are called *implicit methods*.

For some cases we can solve the equation by hand.  For instance in the case of our example problem we have:

$$
    U_{n+1} = U_n + \Delta t f(t_{n+1}, U_{n+1}) = U_n + \Delta t (\lambda U_{n+1})
$$

which can be solved for $U_{n+1}$ to find

$$\begin{aligned}
    U_{n+1} &= U_n + \Delta t (\lambda U_{n+1}) \\
    U_{n+1} \left[ 1 - \Delta t \lambda \right ] &= U_n \\
    U_{n+1} &= \frac{U_n}{1 - \Delta t \lambda}
\end{aligned}$$

```python
t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(decay_constant * t), label="True Solution")

# Plot Backwards Euler step
dt = 1e3
u_np = c_0 + dt * (decay_constant * c_0 * numpy.exp(decay_constant * dt))
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot(dt, u_np, 'ro')
axes.text(dt+ 10., u_np, '$U_1$', fontsize=16)
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)
axes.text(10., 0.85, '$K_1$', fontsize=16)

axes.grid()
axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
plt.show()
```

```python
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = c_0 * numpy.exp(decay_constant * t_exact)

# Implement backwards Euler
t_backwards = numpy.linspace(0.0, 1.6e3, 10)
delta_t = t_backwards[1] - t_backwards[0]
u_backwards = numpy.empty(t_backwards.shape)
u_backwards[0] = c_0
for n in range(0, t_backwards.shape[0] - 1):
    u_backwards[n + 1] = u_backwards[n] / (1.0 - decay_constant * delta_t)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_backwards, u_backwards, 'or', label="Backwards Euler")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")
axes.grid()
axes.set_title("Backwards Euler")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()
plt.show()
```

It's also useful to be able to do this in the case of systems of ODEs.  Let $f(U) = A U$, then

$$\begin{aligned}
    U_{n+1} &= U_n + \Delta t (A U_{n+1}) \\
    U_{n+1} \left [ I - \Delta t A \right ] &= U_n \\
    U_{n+1} &= \left [ I - \Delta t A \right]^{-1} U_n
\end{aligned}$$

In general however we are often not able to do this with arbitrary $f$.

Another simple implicit method is based on quadrature using the trapezoidal method.  The scheme is
$$
    \frac{U_{n+1} - U_{n}}{\Delta t} = \frac{1}{2} (f(U_n) + f(U_{n+1}))
$$

In this case what is the update scheme?

$$\begin{aligned}
    U_{n+1} &= U_{n} + \frac{\Delta t}{2} (f(U_n) + f(U_{n+1})) \\
    U_{n+1} &= U_{n} + \frac{\Delta t}{2} (\lambda U_n + \lambda U_{n+1}) \\
    U_{n+1} \left[1 - \frac{\Delta t \lambda}{2}  \right] &= U_{n} \left[1 + \frac{\Delta t \lambda}{2} \right] \\
    U_{n+1} &= U_{n} \frac{1 + \frac{\Delta t \lambda}{2}}{1 - \frac{\Delta t \lambda}{2}} \\
\end{aligned}$$

```python
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0
t_exact = numpy.linspace(0.0, 1.6e3, 100)
u_exact = c_0 * numpy.exp(decay_constant * t_exact)

# Implement trapezoidal method
t = numpy.linspace(0.0, 1.6e3, 10)
delta_t = t[1] - t[0]
u = numpy.empty(t.shape)
u[0] = c_0
integration_constant = (1.0 + decay_constant * delta_t / 2.0) / (1.0 - decay_constant * delta_t / 2.0)
for n in range(t.shape[0] - 1):
    u[n + 1] = u[n] * integration_constant

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, u, 'or', label="Trapezoidal")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")
axes.grid()

axes.set_title("Trapezoidal")
axes.set_xlabel("t (years)")
axes.set_xlabel("$c(t)$")
axes.set_ylim((0.4,1.1))
axes.legend()
plt.show()
```

## Error Analysis of ODE Methods

At this point it is also helpful to introduce more notation to distinguish between the true solution to the ODE $u(t_n)$ and the approximated value which we will denote $U_n$.

**Definition:** We define the *truncation error* of a scheme by replacing the $U_n$ with the true solution $u(t_n)$ in the finite difference formula and looking at the difference from the exact solution.

For example we will use the difference form of forward Euler
$$
    \frac{U_{n+1} - U_n}{\Delta t} = f(t_n)
$$
and define the truncation error as
$$
    T(t, u; \Delta t) = \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - f(t_n, u(t_n)).
$$

**Definition:** A method is called *consistent* if
$$
    \lim_{\Delta t \rightarrow 0} T(t, u; \Delta t) = 0.
$$

**Definition:** We say that a method is *order* $p$ accurate if

$$
    \lVert T(t, u; \Delta t) \rVert \leq C \Delta t^p
$$

uniformally on $t \in [0, T]$.  This can also be written as $T(t, u; \Delta t) = \mathcal{O}(\Delta t^p)$.  Note that a method is consistent if $p > 0$.

### Error Analysis of Forward Euler

We can analyze the error and convergence order of forward Euler by considering the Taylor series centered at $t_n$:

$$
    u(t) = u(t_n) + (t - t_n) u'(t_n) + \frac{u''(t_n)}{2} (t - t_n)^2 + \mathcal{O}((t-t_n)^3)
$$

Evaluating this series at $t_{n+1}$ gives

$$\begin{aligned}
    u(t_{n+1}) &= u(t_n) + (t_{n+1} - t_n) u'(t_n) + \frac{u''(t_n)}{2} (t_{n+1} - t_n)^2 + \mathcal{O}((t_{n+1}-t_n)^3)\\
    &=u_n + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3)
\end{aligned}$$

From the definition of truncation error we can use our Taylor series expression and find the truncation error.  Take the finite difference form of forward Euler

$$
    \frac{U_{n+1} - U_n}{\Delta t} = f(t_n)
$$

and replacing the derivative formulation with $u(t_n)$ to find

$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - f(t_n) \\
    &= \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - u'(t_n).
\end{aligned}$$

From here we use the Taylor series centered at $t_n$ and evaluated at $t_{n+1}$ to find

$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{u(t_{n+1}) - u(t_n)}{\Delta t} - u'(t_n) \\
    &= \frac{1}{\Delta t} \left[ u(t_n) + u'(t_n) (t - t_n) + \frac{u''(t_n)}{2} (t - t_n)^2 + \mathcal{O}((t-t_n)^3) - u(t_n) \right] - u'(t_n) \\
    &=  u'(t_n) + \frac{u''(t_n)}{2} \Delta t + \mathcal{O}(\Delta t^2) - u'(t_n) \\
    &= \frac{u''(t_n)}{2} \Delta t + \mathcal{O}(\Delta t^2).
\end{aligned}$$

This implies that forward Euler is first order accurate and therefore consistent.

Another equivalent definition of the truncation error uses the form
$$
    U_{n+1} = u(t_n) + \Delta t f(t_n)
$$
and the definition
$$
    T(t, u; \Delta t) = \frac{1}{\Delta t} \left [ U_{n+1} - u(t_{n+1}) \right]
$$

to find
$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} [U_{n+1} - u(t + \Delta t)] \\
    &= \frac{1}{\Delta t} \left[ \underbrace{u_n + \Delta t f(t_n, u_n)}_{U_{n+1}} - \underbrace{\left( u_n + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right )}_{u(t_{n+1})}\right ] \\
    &= \frac{1}{\Delta t} \left[ - \frac{u''(t_n)}{2} \Delta t^2 - \mathcal{O}(\Delta t^3) \right ] \\
    &= - \frac{u''(t_n)}{2} \Delta t - \mathcal{O}(\Delta t^2)
\end{aligned}$$

#### Truncation Error vs Step Error

Sometimes we will also consider the "Step Error"  which is the error that is introduced over one step

$$
    E_h = | U_{n+1} - u_{n+1} |
$$

which in general will be of order $O(\Delta t^{p+1})$ if the Truncation error is $O(\Delta t^p)$.  So for Forward (or Backward's) Euler the step error

$$
    E_h = O(\Delta t^2)
$$

The step error can be very useful in *adaptive stepping* schemes

## Runge-Kutta Methods

One way to derive higher-order ODE solvers is to use higher order quadrature schemes that sample the function at a number of  intermediate stages to provide a more accurate estimate of $K$.  These are not *multi-step* methods as they still only require information from the current time step but they raise the order of accuracy by adding *stages*.  These types of methods are called **Runge-Kutta** methods.

### Example:  Two-stage Runge-Kutta Methods

The basic idea behind the simplest of the Runge-Kutta methods is to approximate $K$ using a mid-point scheme (which should be 2nd order accurate.  Unforunately, we don't know the value of the mid-point.  However we can use an Euler step of size $\Delta_t/2$ to estimate the mid-point.

We can write the algorithm as

$$\begin{aligned}
    K_1 &= \Delta t f(U_n, t_n) \\
    K_2 &= \Delta t f(U_n + K_1/2, t_n + \Delta/2, )\\
    U_{n+1} &= U_n + K_2 \\
\end{aligned}$$

Where we now evaluate the function in two stages $K_1$ and $K_2$.

or for an autonomous ODE
$$
    U_{n+1} = U_n + \Delta t f(U_n + \frac{1}{2} \Delta t f(U_n))
$$

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# RK2 step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
Y1 = U0 + K1/2
K2 = dt * f(dt/2., Y1)
U1 = U0 + K2

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, U0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (U0, U0 + K1), 'k--')
axes.plot((0.0, dt/2.), (U0, U0 + K1/2.), 'k')

axes.plot(dt/2., U0 + K1/2, 'ro')
axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.text(10., 0.85, '$\\frac{K_1}{2}$', fontsize=18)
axes.plot((0.0, dt/2), (Y1, Y1), 'k--')
axes.text(250, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

axes.plot(dt, U1, 'go')
axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.text(10., 0.85, '$\\frac{K_1}{2}$', fontsize=18)
axes.plot((0.0, dt/2), (Y1, Y1), 'k--')
axes.text(250, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

axes.plot(dt, U1, 'go')
axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_1$', fontsize=18)
#axes.plot((0.0, 0.0), (U0, U1), 'g--')
#axes.plot((0.0, dt), (U1, U1), 'g--')

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

#### Error analysis RK2

The truncation error can be computed similarly to how we did so before but we do need to figure out how to compute the derivative inside of the function.  Note that due to
$$
    f(u(t_n)) = u'(t_n)
$$
that differentiating this leads to
$$
    f'(u(t_n)) u'(t_n) = u''(t_n)
$$
leading to
$$\begin{aligned}
    f\left(u(t_n) + \frac{1}{2} \Delta t f(u(t_n)) \right ) &= f\left(u(t_n) +\frac{1}{2} \Delta t u'(t_n) \right ) \\
    &= f(u(t_n)) + \frac{1}{2} \Delta t u'(t_n) f'(u(t_n)) + \frac{1}{8} \Delta t^2 (u'(t_n))^2 f''(u(t_n)) + \mathcal{O}(\Delta t^3) \\
    &=u'(t_n) + \frac{1}{2} \Delta t u''(t_n) + \mathcal{O}(\Delta t^2)
\end{aligned}$$

Going back to the truncation error we have
$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} \left[u_n + \Delta t f\left(u_n + \frac{1}{2} \Delta t f(u_n)\right) - \left(u_n + \Delta t f(t_n, u_n) + \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right ) \right] \\
    &=\frac{1}{\Delta t} \left[\Delta t u'(t_n) + \frac{1}{2} \Delta t^2 u''(t_n) + \mathcal{O}(\Delta t^3) - \Delta t u'(t_n) - \frac{u''(t_n)}{2} \Delta t^2 + \mathcal{O}(\Delta t^3) \right] \\
    &= \mathcal{O}(\Delta t^2)
\end{aligned}$$

so this method is second order accurate.

### Example:  4-stage Runge-Kutta Method

If RK2 is related to a Mid-point quadrature scheme,  then the classic 4-stage, 4th order Runge-Kutta scheme should be reminiscent of Simpson's Quadrature rule.  It requires 4 samples of $f(t,u)$ at the beginning of the step, two-samples in the middle and one at the end, then a linear combination of those samples

$$\begin{aligned}
    K_1 &= \Delta t f(t_n, U_n) \\
    K_2 &= \Delta t f(t_n + \Delta t/2, U_n + K_1/2) \\
    K_3 &= \Delta t f(t_n + \Delta t/2, U_n + K_2/2) \\
    K_4 &= \Delta t f(t_n + \Delta t, U_n + K_3) \\
        & \\
    U_{n+1} &= U_n + \frac{1}{6} \left [K_1 + 2(K_2 + K_3)  + K_4) \right ]
\end{aligned}$$

With truncation error $T = O(\Delta t^4)$

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# RK4 step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
K2 = dt * f(dt/2., U0 + K1/2)
K3 = dt * f(dt/2., U0 + K2/2)
K4 = dt * f(dt, U0 + K3)

U1 = U0 + 1./6. *( K1 + 2 * (K2 + K3) + K4)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, U0 * numpy.exp(decay_constant * t), label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0.-20, U0-.04, '$K_1$', color='red',fontsize=16)

axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt/2.), (U0, U0 + K1/2.), 'k--')
axes.plot(dt/2., U0 + K1/2, 'ro')
axes.text(dt/2-20, U0 + K1/2-.04, '$K_2$', color='red',fontsize=16)

axes.plot((0.0, dt/2.), (U0, U0 + K2/2.), 'k--')
axes.plot(dt/2., U0 + K2/2, 'ro')
axes.text(dt/2-20, U0 + K2/2+.02, '$K_3$', color='red',fontsize=16)

axes.plot((0.0, dt), (U0, U0 + K3), 'k--')
axes.plot(dt, U0 + K3, 'ro')
axes.text(dt-20, U0 + K3-.04, '$K_4$', color='red',fontsize=16)

axes.plot(dt, U1, 'go')
#axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_1$', fontsize=18)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.grid()
plt.show()
```

```python
def RK2(f, t_span, u0, N):
    """ implement constant step size 2 stage Runge-Kutta Method RK2"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[1:]):
        K_1 = delta_t * f(t_n, u[n])
        K_2 = delta_t * f(t_n + delta_t/2., u[n] + K_1/2.)
        u[n+1] = u[n] + K_2
    return t, u

def improved_euler(f, t_span, u0, N):
    """ implement constant step size 2 stage Improved Euler Method trapezoidal rule"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[1:]):
        K_1 = delta_t * f(t_n, u[n])
        K_2 = delta_t * f(t_n + delta_t, u[n] + K_1)
        u[n+1] = u[n] + 0.5 * (K_1 + K_2)
    return t, u
```

```python
def RK4(f, t_span, u0, N):
    """ implement constant step size 4 stage Runge-Kutta Method RK4"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[1:]):
        K_1 = delta_t * f(t_n, u[n])
        K_2 = delta_t * f(t_n + delta_t/2., u[n] + K_1/2.)
        K_3 = delta_t * f(t_n + delta_t/2., u[n] + K_2/2.)
        K_4 = delta_t * f(t_n + delta_t, u[n] + K_3)
        u[n+1] = u[n] + 1./6. * (K_1 + 2.*( K_2 + K_3) + K_4)
    return t, u
```

```python
# Implement and compare the two-stage and 4-stage Runge-Kutta methods
f = lambda t, u: -u

N = 20
t_span = [ 0., 5.0 ]
u0 = 1.

t_exact = numpy.linspace(t_span[0], t_span[1], 100)
u_exact = u0*numpy.exp(-t_exact)
t_euler, u_euler = euler(f, t_span, u0, N)
t_ieuler, u_ieuler = improved_euler(f, t_span, u0, N)
t_RK2, u_RK2 = RK2(f, t_span, u0, N)
t_RK4, u_RK4 = RK4(f, t_span, u0, N)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_exact,u_exact,'k',label='exact')
axes.plot(t_euler, u_euler, 'ro', label='euler')
axes.plot(t_ieuler, u_ieuler, 'co', label='improved euler')
axes.plot(t_RK2, u_RK2, 'go', label='RK2')
axes.plot(t_RK4, u_RK4, 'bo', label='RK4')

axes.grid()
axes.set_xlabel('t', fontsize=16)
axes.set_ylabel('u', fontsize=16)
axes.legend(loc='best')
plt.show()

```

### Convergence of Single Step Multi-Stage schemes

All of the above schemes are consistent and have truncation errors $T\propto\Delta t^p$

```python
N = numpy.array([ 2**n for n in range(4,10)])
err_euler = numpy.zeros(len(N))
err_ieuler = numpy.zeros(len(N))
err_RK2 = numpy.zeros(len(N))
err_RK4 = numpy.zeros(len(N))

t_span = [ 0., 4.]
dt = t_span[1]/N

u0 = 1.
u_exact = u0*numpy.exp(-t_span[1])

for i, n in enumerate(N):
    t, u_euler = euler(f, t_span, u0, n)
    err_euler[i] = numpy.abs(u_euler[-1] - u_exact)
    t, u_ieuler = improved_euler(f, t_span, u0, n)
    err_ieuler[i] = numpy.abs(u_ieuler[-1] - u_exact)
    t, u_RK2 = RK2(f, t_span, u0, n)
    err_RK2[i] = numpy.abs(u_RK2[-1] - u_exact)
    t, u_RK4 = RK4(f, t_span, u0, n)
    err_RK4[i] = numpy.abs(u_RK4[-1] - u_exact)

err_fit = lambda dt, p: numpy.exp(p[1])*dt**p[0]
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

# Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_euler[2:]),1)
line = axes.loglog(dt, err_euler, 'o', label='euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# Improved Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_ieuler[2:]),1)
line = axes.loglog(dt, err_ieuler, 'o', label='improved euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# RK2
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK2[2:]),1)
line = axes.loglog(dt, err_RK2, 'o', label='rk2, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

#RK4
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK4[2:]),1)
line = axes.loglog(dt, err_RK4, 'o', label='rk4, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

axes.grid()
axes.set_xlabel('$\Delta t$', fontsize=16)
axes.set_ylabel('$Error$', fontsize=16)
axes.set_title('Convergence: Single Step Schemes', fontsize=18)
axes.legend(loc='best', fontsize=14)
plt.show()
```

## Summary: single-Step Multi-Stage Schemes

The integral form of an ODE initial value problem can be written

$$
    u(t + \Delta t) = u(t) + \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t})) d\tilde{\!t}
$$

Which says that our solution $u$, if it exists  at some time $\Delta t$ in the future,  is $u(t)$ plus a *number*
$$
    K = \int^{t + \Delta t}_t f(\tilde{t}, u(\tilde{t}) )d\tilde{\!t}
$$
which is a definite *line integral* (along an unknown solution).

#### Single Step, Multi-stage schemes

are most easily understood as extensions of the  Newton-Cotes quadrature schemes for approximating $K$ (plus an error term that will scale as $\Delta t^p$)

**Explicit Schemes**

Name	 Stages	 "Quadrature"	$$T$$

Euler	 1	 Left-Rectangle	$$O(\Delta t)$$

Improved Euler	 2	 Trapezoidal	$$O(\Delta t^2)$$

RK2	 2	 Mid-Point	$$O(\Delta t^2)$$

RK4	 4	 Simpson	$$O(\Delta t^4)$$

**Implicit Schemes**

Name	 Stages	 "Quadrature"	$$T$$

Backwards-Euler	 1	 Right-Rectangle	$$O(\Delta t)$$

Trapezoidal	 2	 Trapezoidal	$$O(\Delta t^2)$$

```python
N = numpy.array([ 2**n for n in range(4,10)])
err_euler = numpy.zeros(len(N))
err_ieuler = numpy.zeros(len(N))
err_RK2 = numpy.zeros(len(N))
err_RK4 = numpy.zeros(len(N))

t_span = [ 0., 4.]
dt = t_span[1]/N

u0 = 1.
u_exact = u0*numpy.exp(-t_span[1])

for i, n in enumerate(N):
    t, u_euler = euler(f, t_span, u0, n)
    err_euler[i] = numpy.abs(u_euler[-1] - u_exact)
    t, u_ieuler = improved_euler(f, t_span, u0, n)
    err_ieuler[i] = numpy.abs(u_ieuler[-1] - u_exact)
    t, u_RK2 = RK2(f, t_span, u0, n)
    err_RK2[i] = numpy.abs(u_RK2[-1] - u_exact)
    t, u_RK4 = RK4(f, t_span, u0, n)
    err_RK4[i] = numpy.abs(u_RK4[-1] - u_exact)

err_fit = lambda dt, p: numpy.exp(p[1])*dt**p[0]
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

# Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_euler[2:]),1)
line = axes.loglog(dt, err_euler, 'o', label='euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# Improved Euler
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_ieuler[2:]),1)
line = axes.loglog(dt, err_ieuler, 'o', label='improved euler, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

# RK2
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK2[2:]),1)
line = axes.loglog(dt, err_RK2, 'o', label='rk2, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

#RK4
p = numpy.polyfit(numpy.log(dt[2:]), numpy.log(err_RK4[2:]),1)
line = axes.loglog(dt, err_RK4, 'o', label='rk4, p={:3.2f}'.format(p[0]))
axes.loglog(dt, err_fit(dt,p),'--', color=line[0].get_color())

axes.grid()
axes.set_xlabel('$\Delta t$', fontsize=16)
axes.set_ylabel('$Error$', fontsize=16)
axes.set_title('Convergence: Single Step Schemes', fontsize=18)
axes.legend(loc='best', fontsize=14)
plt.show()
```

## Adaptive Time Stepping

#### Why should we care about all of these schemes and their errors?

* Even though we know the formal error.  It is with respect to a true solution we don't know.
* In itself, the error estimates don't tell us how to choose a time step $\Delta t$ to keep the solution within a given tolerance
* However,  in combination, we can use multiple methods to control the error and provide **Adaptive** time stepping

#### Example:  Compare 1 step of Euler to one step of RK2

```python
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

# RK2 step
dt = 1e3
U0 = 1.0
K1 = dt * f(0., U0)
Y1 = U0 + K1/2
K2 = dt * f(dt/2., Y1)
U1 = U0 + K2

t = numpy.linspace(U0, 1600.)
u_true = U0 * numpy.exp(decay_constant * t)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, u_true, label="True Solution")
axes.plot(0., U0, 'ro')
axes.text(0., U0+.01, '$U_0$', fontsize=16)
axes.plot((0.0, dt), (U0, U0 + K1), 'k')
#axes.plot((0.0, dt/2.), (U0, U0 + K1/2.), 'k')

#Euler step
axes.plot(dt, U0 + K1, 'ro')
#axes.plot((0.0, 0.0), (U0, Y1), 'k--')
axes.text(dt + 10., U0 + K1, '$U_{euler}$', fontsize=18)

axes.plot(dt, U1, 'go')
#axes.plot((0.0, 0.0), (U0, Y1), 'k--')
#axes.text(10., 0.85, '$\\frac{K_1}{2}$', fontsize=18)
#axes.plot((0.0, dt/2), (Y1, Y1), 'k--')
#axes.text(250, Y1 - 0.05, '$\\frac{\Delta t}{2}$', fontsize=18)

# RK2 Step
axes.plot(dt, U1, 'go')
axes.plot((0., dt), (U0, U1), 'k')
axes.text(dt+20, U1, '$U_{RK2}$', fontsize=18)
#axes.plot((0.0, 0.0), (U0, U1), 'g--')
#axes.plot((0.0, dt), (U1, U1), 'g--')

# difference
axes.plot((dt, dt), (U1, U0+K1),'k--')
axes.text(dt+40, 0.5*(U1 + U0+K1), '$\Delta\propto\Delta t^2$', fontsize=18)

axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.05))
axes.legend(loc='best')
axes.grid()
plt.show()
```

#### Relative Truncation Error

If we consider the *Step Error* for each of our schemes, we know that

$$
\begin{align}
    u_{n+1} &= U^{euler}_{n+1} + O(\Delta t^2)\\
    u_{n+1} &= U^{RK2}_{n+1} + O(\Delta t^3)\\
\end{align}
$$

Therefore we can compute the *relative truncation error* as

$$
    \Delta = | U^{euler}_{n+1} - U^{RK2}_{n+1} | = O(\Delta t^{?})
$$

* $\Delta$ is Computable!
* has a known dependence on step-size $\Delta t$

### Adaptive Time Stepping

Given the relative truncation error and its scaling with $\Delta t$, we can now use this to choose a single good time step.

#### Example:

Suppose we wanted our relative truncation error to be small relative to the solution or zero, we could set

$$
    \Delta_{target} = \mathtt{rtol}\,U^{RK2}_{n+1} + \mathtt{atol}
$$

where $\mathtt{rtol}$ and $\mathtt{atol}$ are relative and absolute tolerances (and we assume that $U^{RK2}_{n+1}$ is a reasonably good estimate of the true solution)

But our measured relative truncation error, $\Delta$ depends on  the step size we just took i.e

$$
    \Delta_{measured} \propto \Delta t_n^2
$$

### Adaptive Time Stepping

If we take the ratio of both relationships we get

$$
    \frac{\Delta_{target}}{\Delta_{measured}} = \left[\frac{\Delta t_{target}}{\Delta t_{n}}\right]^2
$$

or rearranging, our target step size is

$$
    \Delta t_{target} = \Delta t_{n}\left[\frac{\Delta_{target}}{\Delta_{measured}}\right]^{\frac{1}{2}}
$$

which tells us how to grow or shrink our time step to maintain accuracy.

In general,  if we have two methods with different step errors such that

$$
    \Delta \propto \Delta t^p
$$

then our adaptive stepper will look like

$$
    \Delta t_{target} = \Delta t_{n}\left[\frac{\Delta_{target}}{\Delta_{measured}}\right]^{1/p}
$$

This leads to all sorts of adaptive schemes most are included in standard libraries.

### Embedded Runge-Kutta Schemes

There are in fact a whole family of **Embedded RK** schemes which are $N$ stage schemes but can combine the $N$ function evaluations in two different ways to produce methods with different error estimates.

A popular one is **RK45** (available in `SciPy`) which is based on the [Dormand-Prince 5(4)](https://doi.org/10.1016/0771-050X(80)90013-3) pair which uses 6 function evaluations per step to produce a 4th order and 5th order scheme.  The 4th order scheme controls the time step, and the 5th order scheme actually is the solution.

```python
from scipy.integrate import solve_ivp

def f_vanderpol(t, u, mu=5):
    return numpy.array([u[1], mu * (1.0 - u[0]**2) * u[1] - u[0]])

t_span = (0., 50.)
u0 = [ 1., 0. ]
f = lambda t, u : f_vanderpol(t, u, mu=20)
sol = solve_ivp(f, t_span, u0, method='RK45',rtol=1.e-3,atol=1.e-6)
```

```python
fig = plt.figure(figsize=(16,6))
axes = fig.add_subplot(1, 2, 1)

axes.plot(sol.t, sol.y[0],'o-')
axes.set_title("Solution to Van der Pol Oscillator", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("y(t)", fontsize=16)
axes.grid()

axes = fig.add_subplot(1, 2, 2)
delta_t = sol.t[1:] - sol.t[:-1]
axes.plot(sol.t[:-1], delta_t)
axes.grid()
axes.set_xlabel('$t$', fontsize=16)
axes.set_ylabel('$\Delta t$', fontsize=16)
axes.set_title('Timesteps, N = {}'.format(len(sol.t)), fontsize=18)

plt.show()
```

## Taylor Series Methods

A **Taylor series method** can be derived by direct substitution of the right-hand-side function $f(t, u)$ and its appropriate derivatives into the Taylor series expansion for $u(t_{n+1})$.  For a $p$th order method we would look at the Taylor series up to that order and replace all the derivatives of $u$ with derivatives of $f$ instead.

For the general case we have
$$\begin{align*}
    u(t_{n+1}) = u(t_n) + \Delta t u'(t_n) + \frac{\Delta t^2}{2} u''(t_n) + \frac{\Delta t^3}{6} u'''(t_n) + \cdots + \frac{\Delta t^p}{p!} u^{(p)}(t_n)
\end{align*}$$
which contains derivatives of $u$ up to $p$th order.

We then replace these derivatives with the appropriate derivative of $f$ which will always be one less than the derivative of $u$ (due to the original ODE)

$$
    u^{(p)}(t_n) = f^{(p-1)}(t_n, u(t_n))
$$

leading to the method

$$
\begin{align}
    u(t_{n+1}) &= u(t_n) + \Delta t f(t_n, u(t_n)) + \frac{\Delta t^2}{2} f'(t_n, u(t_n)) \\
    &+ \frac{\Delta t^3}{6} f''(t_n, u(t_n)) + \cdots + \frac{\Delta t^p}{p!} f^{(p-1)}(t_n, u(t_n)).
\end{align}
$$

### 2nd Order Taylor Series Method

We want terms up to second order so we need to take the derivative of $u' = f(t, u)$ once to find $u'' = f'(t, u)$ and therefore
$$\begin{align*}
    u(t_{n+1}) &= u(t_n) + \Delta t u'(t_n) + \frac{\Delta t^2}{2} u''(t_n) \\
    &=u(t_n) + \Delta t f(t_n, u(t_n)) + \frac{\Delta t^2}{2} f'(t_n, u(t_n)) ~~~ \text{or} \\
    U_{n+1} &= U_n + \Delta t f(t_n, U_n) + \frac{\Delta t^2}{2} f'(t_n, U_n).
\end{align*}$$

With Step error $O(\Delta t^3)$ and truncation error $T$, $O(\Delta t^2)$

### Example

Let's use our simplest problem $u'(t) = \lambda u$ with $f=\lambda u$.  Therefore

$$\begin{align*}
    f(t,u) &= \lambda u\\
    f'(t,u) &= \lambda u' = \lambda f = \lambda^2 u\\
    f''(t,u) &= \lambda^2 u' = \lambda^2 f = \lambda^3 u
\end{align*}$$

so a third order scheme would look like
$$
\begin{align}
    U(t_{n+1}) &= U(t_n)\left[ 1 + \lambda\Delta t  + \frac{(\lambda\Delta t)^2}{2} + \frac{(\lambda\Delta t)^3}{6}\right]+ O(\Delta t^4)
\end{align}
$$

```python
def Taylor_3_flambda_u(lamda, t_span, u0, N):
    """ implement constant step size  3rd order Taylor Series method for f(t,u) = \lambda u"""

    t = numpy.linspace(t_span[0], t_span[1], N)
    lambda_dt = lamda*(t[1] - t[0])
    u = numpy.empty(t.shape)
    u[0] = u0
    for (n, t_n) in enumerate(t[1:]):
        u[n+1] = u[n] * ( 1. + lambda_dt  + (lambda_dt**2)/2.  + (lambda_dt**3)/6.)
    return t, u

```

```python
lam = -1.
t_span = [0., 5.]
u0 = 1.

f = lambda t,u : -u
t_exact = numpy.linspace(t_span[0], t_span[1], 100)
u_exact = u0*numpy.exp(-t_exact)

N = 20.
t_taylor, u_taylor = Taylor_3_flambda_u(lam, t_span, u0, N)
t_euler, u_euler = euler(f, t_span, u0, N)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_exact,u_exact,'k',label='exact')
axes.plot(t_euler, u_euler, 'ro', label='euler')
axes.plot(t_taylor, u_taylor, 'bo', label='Taylor3')

axes.grid()
axes.set_xlabel('t', fontsize=16)
axes.set_ylabel('u', fontsize=16)
axes.legend(loc='best')
plt.show()

```

### Some Drawbacks

**Taylor Series methods**
 - require differentiating the given equation which can be cumbersome and difficult to implement
 - require a new routine for every $f$

**General one-step/multi-stage methods**
  - higher order methods often require a large number of evaluations of $f$ per time step

## Linear Multi-Step Methods

**Multi-step methods**  are ODE methods that
 - require only *one* new function evaluation per time step  to work.
 - reuse values and function evaluations at some number of previous time steps

**Disadvantages over single step methods**

 - Methods are not self-starting, i.e. they require other methods to find the initial values
 - Difficult to adapt. The time step  Δ𝑡  in one-step methods can be changed at any time while multi-step methods this is much more complex

### Simplest example:  The leap-frog method

The leap-frog method is similar to Euler's method in that it uses the information from two-previous time steps to advance the problem.  We can write the problem as a centered first-derivative centered   at time $t_{n+1}, U_{n+1}$ i.e.

$$\frac{U_{n+2} - U_{n}}{2\Delta t} = f(t_{n+1}, U_{n+1})$$

or

$$
    U_{n+2} = U_{n} + 2\Delta t\, f(t_{n+1}, U_{n+1})
$$

this method is known as the leap-frog method

```python
t = numpy.linspace(0.0, 1.6e3, 100)
c_0 = 1.0
decay_constant = -numpy.log(2.0) / 1600.0

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t, c_0 * numpy.exp(decay_constant * t), label="True Solution")

# Plot Leap-Frog step
dt = 1e3
u1 = c_0 * numpy.exp(decay_constant * dt / 2.0)
u_np = c_0 + dt * (decay_constant * u1)
axes.plot((0.0, dt), (c_0, u_np), 'k')
axes.plot((dt, dt), (u_np, c_0 * numpy.exp(decay_constant * dt)), 'k--')
axes.plot((0.0, 0.0), (c_0, u_np), 'k--')
axes.plot((0.0, dt), (u_np, u_np), 'k--')
axes.text(400, u_np - 0.05, '$\Delta t$', fontsize=16)
axes.plot([0., dt/2, dt], [ c_0, u1, u_np],'ro')
axes.set_title("Radioactive Decay with $t_{1/2} = 1600$ years")
axes.set_xlabel('t (years)')
axes.set_ylabel('$c$')
axes.grid()
axes.set_xlim(-1e2, 1.6e3)
axes.set_ylim((0.5,1.0))
plt.show()
```

```python
def leap_frog(f, t_span, u0, N, start=RK2):
    """ calculate fixed step with leap-frog iterator with a single step starter
    """
    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.zeros(t.shape)
    u[0] = u0
    # use a single-step multi-stage method to start

    t_start, u_start = start(f, (t[0],t[1]), u0, 2)
    u[1] = u_start[-1]
    for (n, t_np) in enumerate(t[1:-1]):
        u[n+2] = u[n] +  2 *delta_t * f(t_np, u[n+1])
    return t, u
```

```python
u0 = 1.0
t_span = (0., 1600.)
N = 7

# Stable example
decay_constant = -numpy.log(2.0) / 1600.0
f = lambda t, u: decay_constant * u

t_exact = numpy.linspace(t_span[0], t_span[1], N)
u_exact = u0 * numpy.exp( decay_constant * t_exact)

t_leapfrog, u_leapfrog = leap_frog(f, t_span, u0, N, start=RK2)
```

```python
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)
axes.plot(t_leapfrog, u_leapfrog, 'or-', label="Leap-Frog")
axes.plot(t_exact, u_exact, 'k--', label="True Solution")
axes.grid()
axes.set_title("Leap-Frog", fontsize=18)
axes.set_xlabel("t (years)", fontsize=16)
axes.set_xlabel("$c(t)$", fontsize=16)
axes.legend(loc='best', fontsize=14)
plt.show()
```

### Error Analysis of Leap-Frog Method

To easily analyze this method we will expand the Taylor series around time $t_{n+1}$ to yield
$$\begin{aligned}
    u(t_{n+2}) &= u_{n+1} + \Delta t f(t_{n+1},u_{n+1}) + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)
\end{aligned}$$

We need one more expansion however due to leap-frog.  Recall that leap-frog has the form
$$
    U_{n+2} = U_{n} + 2 \Delta t f(t_{n+1}, U_{n+1}).
$$
To handle the $U_{n}$ term we need to write this with relation to $u(t_{n+1})$.  Again we use the Taylor series
$$
    u(t_n) = u_{n+1} - \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  - \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)
$$

$$\begin{aligned}
    u(t_{n+2}) &= u_{n+1} + \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)\\
    u(t_{n}) &= u_{n+1} - \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  - \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)
\end{aligned}$$

Plugging these into our definition of the truncation error along with the leap-frog method definition leads to

$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} \left [\underbrace{U_{n} + 2 \Delta t f_{n+1}}_{U_{n+2}} - \underbrace{\left(u_{n+1} + \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4) \right )}_{u(t_{n+2})} \right ] \\
    &=\frac{1}{\Delta t} \left [ \underbrace{ \left(u_{n+1} - \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  - \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4)\right)}_{U_{n}} + 2\Delta t f_n - \underbrace{\left(u_{n+1} + \Delta t f_{n+1} + \Delta t^2 \frac{u''(t_{n+1})}{2}  + \Delta t^3 \frac{u'''(t_{n+1})}{6} + \mathcal{O}(\Delta t^4) \right )}_{u(t_{n+2})} \right ] \\
    &=\frac{1}{\Delta t} \left [- \Delta t^3 \frac{u'''(t_n)}{3} + \mathcal{O}(\Delta t^4) \right ] \\
        &=- \Delta t^2 \frac{u'''(t_n)}{3} + \mathcal{O}(\Delta t^3)
\end{aligned}$$

Therefore the method is second order accurate and is consistent theoretically.  In practice it's a bit more complicated than that.

```python
# Compare accuracy between Euler, RK2 and Leap-Frog
f = lambda t, u: -u
u_exact = lambda t: numpy.exp(-t)
u_0 = 1.0

t_span = (0.0, 10.0)
num_steps = [2**n for n in range(4,11)]
```

```python
delta_t = numpy.empty(len(num_steps))
error_euler = numpy.empty(len(num_steps))
error_RK2 = numpy.empty(len(num_steps))
error_leapfrog = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(t_span[0], t_span[1], N)
    tt, u_euler = euler(f, t_span, u_0, N )
    tt, u_rk2 = RK2(f, t_span, u_0, N)
    tt, u_leapfrog = leap_frog(f, t_span, u_0, N, start=euler)

    delta_t[i] = t[1] - t[0]

    # Compute error for each
    error_euler[i] = numpy.linalg.norm(delta_t[i] * (u_euler - u_exact(t)), ord=1)
    error_RK2[i] = numpy.linalg.norm(delta_t[i] * (u_rk2 - u_exact(t)), ord=1)
    error_leapfrog[i] = numpy.linalg.norm(delta_t[i] * (u_leapfrog - u_exact(t)), ord=1)

# Plot error vs. delta_t
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, error_euler, 'bo', label='Forward Euler')
axes.loglog(delta_t, error_RK2, 'ro', label='RK2')
axes.loglog(delta_t, error_leapfrog, 'go', label="Leap-Frog")

axes.loglog(delta_t, order_C(delta_t[2], error_euler[2], 1.0) * delta_t**1.0, '--b')
axes.loglog(delta_t, order_C(delta_t[2], error_RK2[2], 2.0) * delta_t**2.0, '--r')
axes.loglog(delta_t, order_C(delta_t[2], error_leapfrog[2], 2.0) * delta_t**2.0, '--r')

axes.grid()
axes.legend(loc=2, fontsize=14)
axes.set_title("Comparison of Errors", fontsize=18)
axes.set_xlabel("$\Delta t$",fontsize=16)
axes.set_ylabel("$|U(t_f) - u(t_f)|$", fontsize=16)

plt.show()
```

### Look at the errors for Leap-Frog

They're actually quite large...If you make a quick plot of u_leapfrog vs $t$ you'll see what's happening (and is a good example of an issue we will need to address in future lectures)

```python
N= 1000
t_leapfrog, u_leapfrog = leap_frog(f, t_span, u_0, N, start=euler)
## Your plotting code here
plt.figure()
plt.plot(t_leapfrog, u_leapfrog)
plt.grid()
plt.show()

```

### General Linear Multi-Step Methods

Leap-frog is perhaps the simplest of multi-step methods but all linear multi-step methods can be written as the linear combination of past, present and future solutions:

Want to get from time $U_n$ to time $U_{n+r}$ by taking steps $j$:

$$
    \sum^r_{j=0} \alpha_j U_{n+j} = \Delta t \sum^r_{j=0} \beta_j f(U_{n+j}, t_{n+j})
$$
If $\beta_r = 0$ then the method is explicit (only requires previous time steps).  Note that the coefficients are not unique as we can multiply both sides by a constant.  In practice a normalization of $\alpha_r = 1$ is used.

For example:  our Leap-frog method can be written using $r=2$, $\alpha = \begin{bmatrix} -1 & 0 & 1\\
\end{bmatrix}$, $\beta = \begin{bmatrix} 0 & 2 & 0 \\ \end{bmatrix}$

#### Example: Adams Methods

$$
    U_{n+r} = U_{n+r-1} + \Delta t \sum^r_{j=0} \beta_j f(U_{n+j}).
$$
All these methods have $\alpha_r = 1$, $\alpha_{r-1} = -1$ and $\alpha_j=0$ for $j < r - 1$.

### Adams-Bashforth Methods
The **Adams-Bashforth** methods are explicit solvers that maximize the order of accuracy given a number of steps $r$.  This is accomplished by looking at the Taylor series and picking the coefficients $\beta_j$ to eliminate as many terms in the Taylor series as possible. AB is initialized with forward Euler.
$$\begin{aligned}
    \text{1-step:} & ~ & U_{n+1} &= U_n +\Delta t f(U_n) \\
    \text{2-step:} & ~ & U_{n+2} &= U_{n+1} + \frac{\Delta t}{2} (-f(U_n) + 3 f(U_{n+1})) \\
    \text{3-step:} & ~ & U_{n+3} &= U_{n+2} + \frac{\Delta t}{12} (5 f(U_n) - 16 f(U_{n+1}) + 23 f(U_{n+2})) \\
    \text{4-step:} & ~ & U_{n+4} &= U_{n+3} + \frac{\Delta t}{24} (-9 f(U_n) + 37 f(U_{n+1}) -59 f(U_{n+2}) + 55 f(U_{n+3}))
\end{aligned}$$

```python
def AB2(f, t_span, u0, N, start=RK2):
    """ calculate fixed step Adams-Bashforth 2-step method with a single step starter
        reuses previous function evaluations
    """
    t = numpy.linspace(t_span[0], t_span[1], N)
    delta_t = t[1] - t[0]
    u = numpy.zeros(t.shape)

    u[0] = u0
    # use a single-step multi-stage method to start
    t_start, u_start = start(f, (t[0],t[1]), u0, 2)
    u[1] = u_start[-1]

    # set initial function evaluations
    fn = f(t[0], u[0])
    fnp = f(t[1], u[1])
    for (n, t_np) in enumerate(t[2:]):
        u[n+2] = u[n + 1] + delta_t / 2.0 * (-fn + 3.0 * fnp)
        fn = fnp
        fnp = f(t_np, u[n+2])
    return t, u
```

```python
# Use 2-step Adams-Bashforth to compute solution
f = lambda t, u: -u
t_span = (0., 10.)
u0 = 1.0

N = 20
t, u_ab2 = AB2(f, t_span, u0, N, start=RK2)
```

```python
t_exact = numpy.linspace(t_span[0], t_span[1], 100)
u_exact = numpy.exp(-t_exact)

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, u_ab2, 'ro', label="2-step A-B")

axes.set_title("Adams-Bashforth Method", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("u(t)",fontsize=16)
axes.legend(loc=1, fontsize=14)
axes.grid()

plt.show()
```

### Adams-Moulton Methods
The **Adams-Moulton** methods are the implicit versions of the Adams-Bashforth methods.  Since this gives one additional parameter to use $\beta_r$ these methods are generally one order of accuracy greater than their counterparts. AM is initialized with trapezoidal scheme.
$$\begin{aligned}
    \text{1-step:} & ~ & U_{n+1} &= U_n + \frac{\Delta t}{2} (f(U_n) + f(U_{n+1})) \\
    \text{2-step:} & ~ & U_{n+2} &= U_{n+1} + \frac{\Delta t}{12} (-f(U_n) + 8f(U_{n+1}) + 5f(U_{n+2})) \\
    \text{3-step:} & ~ & U_{n+3} &= U_{n+2} + \frac{\Delta t}{24} (f(U_n) - 5f(U_{n+1}) + 19f(U_{n+2}) + 9f(U_{n+3})) \\
    \text{4-step:} & ~ & U_{n+4} &= U_{n+3} + \frac{\Delta t}{720}(-19 f(U_n) + 106 f(U_{n+1}) -264 f(U_{n+2}) + 646 f(U_{n+3}) + 251 f(U_{n+4}))
\end{aligned}$$

```python
# Use 2-step Adams-Moulton to compute solution
# u' = - decay u
decay_constant = 1.0
f = lambda t, u: -decay_constant * u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 20
# N = 10
# N = 5
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)
U[0] = 1.0
U[1] = U[0] + 0.5 * delta_t * f(t[0], U[0])
U[1] = U[0] + delta_t * f(t[0], U[1])
integration_constant = 1.0 / (1.0 + 5.0 * decay_constant * delta_t / 12.0)
for n in range(t.shape[0] - 2):
    U[n+2] = (U[n+1] + decay_constant * delta_t / 12.0 * (U[n] - 8.0 * U[n+1])) * integration_constant

fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-M")

axes.set_title("Adams-Moulton Method ($f=-u$)", fontsize=18)
axes.set_xlabel("t", fontsize=16)
axes.set_ylabel("u(t)", fontsize=16)
axes.legend(loc=1, fontsize=14)
axes.grid()
plt.show()
```

### Truncation Error for Multi-Step Methods

We can again find the truncation error in general for linear multi-step methods:
$$\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t} \left [\sum^r_{j=0} \alpha_j u_{n+j} - \Delta t \sum^r_{j=0} \beta_j f(u_{n+j}, t_{n+j}) \right ]
\end{aligned}$$

Using the general expansion and evaluation of the Taylor series about $t_n$ we have
$$\begin{aligned}
    u(t_{n+j}) &= u(t_n) + j \Delta t u'(t_n) + \frac{1}{2} (j \Delta t)^2 u''(t_n) + \mathcal{O}(\Delta t^3) \\
    u'(t_{n+j}) &= u'(t_n) + j \Delta t u''(t_n) + \frac{1}{2} (j \Delta t)^2 u'''(t_n) + \mathcal{O}(\Delta t^3)
\end{aligned}$$

collecting terms of order $u^{(p)}$

$$
\begin{aligned}
    T(t, u; \Delta t) &= \frac{1}{\Delta t}\left( \sum^r_{j=0} \alpha_j\right) u(t_n) + \left(\sum^r_{j=0} (j\alpha_j - \beta_j)\right) u'(t_n) + \Delta t \left(\sum^r_{j=0} \left (\frac{1}{2}j^2 \alpha_j - j \beta_j \right) \right) u''(t_n) \\
& \quad \quad + \cdots + \Delta t^{q - 1} \left (\sum^r_{j=0} \left(\frac{1}{q!} j^q \alpha_j - \frac{1}{(q-1)!} j^{q-1} \beta_j \right) \right) u^{(q)}(t_n) + \cdots
\end{aligned}$$

The method is *consistent* if the first two terms of the expansion vanish, i.e. $\sum^r_{j=0} \alpha_j = 0$ and $\sum^r_{j=0} j \alpha_j = \sum^r_{j=0} \beta_j$.

```python
# Compare accuracy between RK-2, AB-2 and AM-2, RK-4
f = lambda t, u: -u
u_exact = lambda t: numpy.exp(-t)

t_f = 10.0
t_span = (0.0, t_f)

num_steps = [2**n for n in range(4,10)]
delta_t = numpy.empty(len(num_steps))
error_rk = numpy.empty(len(num_steps))
error_rk4 = numpy.empty(len(num_steps))
error_ab = numpy.empty(len(num_steps))
error_am = numpy.empty(len(num_steps))

for (i, N) in enumerate(num_steps):
    t = numpy.linspace(0, t_f, N)
    delta_t[i] = t[1] - t[0]

    # Compute RK2
    tt, U_rk  = RK2(f, t_span, u0, N)

    # Compute RK4
    tt, U_rk4  = RK4(f, t_span, u0, N)

    # Compute Adams-Bashforth 2-stage
    tt, U_ab  = AB2(f, t_span, u0, N)

    # Compute Adama-Moulton 2-stage
    U_am = numpy.empty(t.shape)
    U_am[:2] = U_rk[:2]
    decay_constant = 1.0
    integration_constant = 1.0 / (1.0 + 5.0 * decay_constant * delta_t[i] / 12.0)
    for n in range(t.shape[0] - 2):
        U_am[n+2] = (U_am[n+1] + decay_constant * delta_t[i] / 12.0 * (U_am[n] - 8.0 * U_am[n+1])) * integration_constant

    # Compute error for each
    error_rk[i] = numpy.linalg.norm(delta_t[i] * (U_rk - u_exact(t)), ord=1)
    error_rk4[i] = numpy.linalg.norm(delta_t[i] * (U_rk4 - u_exact(t)), ord=1)
    error_ab[i] = numpy.linalg.norm(delta_t[i] * (U_ab - u_exact(t)), ord=1)
    error_am[i] = numpy.linalg.norm(delta_t[i] * (U_am - u_exact(t)), ord=1)

# Plot error vs. delta_t
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.loglog(delta_t, error_rk, 'ko', label='RK-2')
axes.loglog(delta_t, error_ab, 'bo', label='AB-2')
axes.loglog(delta_t, error_am, 'go', label="AM-2")
axes.loglog(delta_t, error_rk4, 'co', label='RK-4')

order_C = lambda delta_x, error, order: numpy.exp(numpy.log(error) - order * numpy.log(delta_x))
axes.loglog(delta_t, order_C(delta_t[0], error_ab[0], 1.0) * delta_t**1.0, '--r')
axes.loglog(delta_t, order_C(delta_t[1], error_ab[1], 2.0) * delta_t**2.0, '--b')
axes.loglog(delta_t, order_C(delta_t[1], error_am[1], 3.0) * delta_t**3.0, '--g')
axes.loglog(delta_t, order_C(delta_t[1], error_rk4[1], 4.0) * delta_t**4.0, '--c')

axes.legend(loc=4, fontsize=14)
axes.set_title("Comparison of Errors",fontsize=18)
axes.set_xlabel("$\Delta t$",fontsize=16)
axes.set_ylabel("$|U(t) - u(t)|$", fontsize=16)
axes.grid()

plt.show()
```

### Predictor-Corrector Methods

One way to simplify the Adams-Moulton methods so that implicit evaluations are not needed is by estimating the required implicit function evaluations with an explicit method.  These are often called **predictor-corrector** methods as the explicit method provides a *prediction* of what the solution might be and the now explicit *corrector* step works to make that estimate more accurate.

#### Example: One-Step Adams-Bashforth-Moulton

Use the One-step Adams-Bashforth method to predict the value of $U_{n+1}$ and then use the Adams-Moulton method to correct that value:
$$\begin{aligned}
    \hat{U}_{n+1} &= U_n + \Delta t f(U_n) \\
    U_{n+1} &= U_n + \frac{1}{2} \Delta t \left[f(U_n) + f(\hat{U}_{n+1}) \right]
\end{aligned}$$
leading to a second order accurate method.  Note this algorithm is identical to \_\_\_\_\_\_\_\_\_\_\_\__________?

```python
# One-step Adams-Bashforth-Moulton
f = lambda t, u: -u

t_exact = numpy.linspace(0.0, 10.0, 100)
u_exact = numpy.exp(-t_exact)

N = 50
t = numpy.linspace(0, 10.0, N)
delta_t = t[1] - t[0]
U = numpy.empty(t.shape)

U[0] = 1.0
for n in range(t.shape[0] - 1):
    U[n+1] = U[n] + delta_t * f(t[n], U[n])
    U[n+1] = U[n] + 0.5 * delta_t * (f(t[n], U[n]) + f(t[n+1], U[n+1]))

t_ie, u_ieuler = improved_euler(f, (0.0, 10.), 1., N)
fig = plt.figure(figsize=(8,6))
axes = fig.add_subplot(1, 1, 1)

axes.plot(t_exact, u_exact, 'k', label="True")
axes.plot(t, U, 'ro', label="2-step A-B")
axes.plot(t_ie, u_ieuler, 'bx', label="Improved Euler")

axes.set_title("Adams-Bashforth-Moulton P/C Method", fontsize=18)
axes.set_xlabel("t", fontsize=18)
axes.set_ylabel("u(t)", fontsize=18)
axes.legend(loc='best', fontsize=14)
axes.grid()

plt.show()
```