# The Runge-Kutta Method, Higher-Order ODEs and Multistep Methods

By Thomas P Ogden (<t.p.ogden@durham.ac.uk>)

_Here we introduce the classical Runge-Kutta method, go beyond first-order ODEs, and take a first look at multistep methods._

```
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

In Part 1 we looked at the Explicit Euler method, and checked that it had a global order of accuracy $\mathcal{O}(h^1)$. Now we'll again take an ODE $y' = f(t,y(t))$ with an initial condition $y(t_0) = y_0$ and look at more advanced numerical methods for solving the problem.

## The Runge-Kutta Method

For the Explicit Euler method we truncated the Taylor expansion after the linear term. It's clear that including more terms before truncation will give us a higher-order approximation. The classical Runge-Kutta method does just this, up to an order of accuracy $\mathcal{O}(h^4)$ — 3 orders higher than Explicit Euler! We're not going to derive the approximation here but you can look it up if you're interested. The finite difference step is given by

$$
y_{n+1} = y_n + \tfrac{h}{6} \left( k_1 + 2k_2 + 2k_3 + k_4 \right),
$$

where

$$
\begin{align}
k_1 &= f(t_n, y_n), \\
k_2 &= f(t_n + \tfrac{h}{2}, y_n + \tfrac{h}{2}k_1), \\
k_3 &= f(t_n + \tfrac{h}{2}, y_n + \tfrac{h}{2}k_2), \\
k_4 &= f(t_n + h, y_n + h k_3).
\end{align}
$$

### Implementing the Method in Python

Just like we did with the Explicit Euler method, we'll define a function to implement the Runge-Kutta method for a first-order ODE system.

```
def ode_int_rk(func, y_0, t, args={}):
    """ Classical Runge-Kutta (RK4) approximation to a first-order ODE system
    with initial conditions.

    Args:
        func: (callable) The first-order ODE system to be approximated.
        y_0: (array) The initial condition.
        t: (array) A sequence of time points for which to solve for y.
        args: (dict) Extra arguments to pass to function.

    Out:
        y: (array) the approximated solution of the system at each time in t,
            with the initial value y_0 in the first row.
    """

    # Initialise the approximation array
    y = np.zeros([len(t), len(y_0)])
    y[0] = y_0

    # Loop through the time steps, approximating this step from the prev step
    for i, t_i in enumerate(t[:-1]):

        h = t[i+1] - t_i # size of the step

        k_1 = func(t_i, y[i], args)
        k_2 = func(t_i+h/2., y[i]+h/2.*k_1, args)
        k_3 = func(t_i+h/2., y[i]+h/2.*k_2, args)
        k_4 = func(t_i+h, y[i]+h*k_3, args)

        y[i+1] = y[i] + h/6.*(k_1 + 2.*k_2 + 2.*k_3 + k_4) # RK4 step

    return y
```

### Checking Accuracy

We'll define our test function `exp` just as we did in Part 1.

```
from exp import exp # same function we defined in Part 1.
```

And just as we did for the Explicit Euler, we'll use this test function to check the order of accuracy of the method by looking over a wide range of stepsizes.

```
y_0 = np.array([1.]) # Initial condition

solve_args = {}
solve_args['a'] = 1.

t_max = 5.

# Range of stepsizes
max_N = 12
N = 2**np.arange(2, max_N) # N = 2, 4, 8, ..., 2^max_N

order_check = 4 # for visual check of the order of accuracy

y_end = np.zeros(len(N)) # array to fill with the final values
stepsize = np.zeros(len(N)) # array to fill with the stepsizes

for i, N_i in enumerate(N): # loop over different numbers of steps

    t = np.linspace(0., t_max, N_i+1)
    y_end[i] = ode_int_rk(exp, y_0, t, solve_args)[-1]

    stepsize[i] = t_max/N_i

plt.loglog(stepsize, abs(y_end - np.exp(solve_args['a']*t_max)),
           'b-o', label='Global error')
plt.loglog(stepsize, stepsize**order_check,'k--', label=r'$h^4$')
plt.xlabel(r'$h$')
plt.legend(loc=2)
```

So we've confirmed that the Runge-Kutta method is $\mathcal{O}(h^4)$ – so that the global error gets smaller much quicker with increased number of steps than the Explicit Euler.

## Example 2: A Forced and Damped Pendulum

The next problem we're going to look at is a second-order ODE, but crucially _any higher-order ODE can be rewritten as a set of coupled first-order equations_, such that we can apply the solvers we've already designed. We'll see how to do that with this example.

### The Physical Problem

Take an idealised pendulum: a weightless string of length $\ell$, fixed at one end with a mass $m$ at the other. The pendulum is free to swing in a plane subject to gravity, friction proportional to its velocity $v$, and may be driven by an external periodic force $F_d \cos {\omega_d t}$.

We start with Newton's second law applied to the horizontal displacement $x$ for the unforced and undamped pendulum,

$$m\ddot{x} = -mg \sin{\theta}$$

where $g$ is the local acceleration due to gravity and $\theta{(t)}$ is the angle of displacement of the string from vertical at time $t$. We now apply a frictional force proportional to the translational velocity and rearrange to obtain a second-order homogeneous differential equation,

$$ m \ddot{x} + k \ell \dot{x} + mg \sin{\theta} = 0 $$

where $k$ is the coefficient of friction.

We want to consider angular displacement, so substitute $\dot{x} = \ell \dot{\theta}$ and $\ddot{x} = \ell \ddot{\theta}$, and now apply the driving force

$$ m \ell \ddot{\theta} + k \ell \dot{\theta} + mg \sin{\theta} = F_d \cos {\omega_d t} $$

The equation is nonlinear (due to the sine function), so finding an analytic solution is going to be difficult. Numerical methods are the next line of attack, so we'll try our Runge-Kutta method on the problem. First we must rewrite the second-order equation as a set of coupled first-order equations. Let $y_0 = \theta$, $y_1 = \dot{\theta}$ and $y_2 = \ddot{\theta}$. Then

$$
\begin{align*}
y_0' &= y_1 = \dot{\theta} \\
y_1' &= y_2 = \ddot{\theta} = -\frac{k}{m}\dot{\theta} - \frac{g}{\ell} \sin{\theta} + \frac{F_d}{m \ell} cos{\omega_d t}
\end{align*}
$$

is the system we want. We make a final tidying of the parameters by letting $\alpha = g/\ell$, $\beta = k/m$ and $\gamma = F/m\ell$

$$
\begin{align*}
y_0' &= y_1 \\
y_1' &= -\alpha \sin{y_0} -\beta y_1 + \gamma \cos{\omega t}
\end{align*}
$$

### Setting up the Problem in Python

We write this pair of euqations up as a Python function to be passed to our ODE integrator:

```
def pendulum(t, y, args):
    """ A damped and forced pendulum, described as set of two first-order ODEs.

    Args:
        t: Time
        y: Pendulum system vector [angle, angular velocity]
        args['alpha']: gravity_acc/length_of_pendulum
        args['beta']: friction_constant/mass_pendulum
        args['gamma']: driving_force/mass_pendulum/length_pendulum
        args['omega']: driving_freq

    Returns:
        dydt: ODE vector
    """

    dydt = np.zeros(2)

    dydt[0] = y[1]
    dydt[1] = (-args['alpha']*np.sin(y[0]) - args['beta']*y[1] +
                args['gamma']*np.cos(args['omega']*t))

    return dydt
```

Now we're ready to select some parameters. First we'll remove friction ($k = 0$) and apply no driving force ($F_d = 0$), under which conditions we expect to see simple harmonic motion of the pendulum back and forth.

```
### Parameters

gravity_acc = 10. # [m /s2]

length_pendulum = 1. # [m]
mass_pendulum = 1. # [kg]
friction_constant = 0. # [kg /m /s]

driving_force = 0. # [N]
driving_freq = 0. # [2π /s]

N = 200

t = np.linspace(0., 10., N+1) # [s] an array of time steps
```

We're going to pass the arguments to the solver as a dict, so let's make that now.

```
solve_args = {}
solve_args['alpha'] = gravity_acc/length_pendulum
solve_args['beta'] = friction_constant/mass_pendulum
solve_args['gamma'] = driving_force/mass_pendulum/length_pendulum
solve_args['omega'] = driving_freq
```

Next we need some initial conditions. We'll pick $\theta_0 = \tfrac{\pi}{8}$ so that we can check our numerical result with the **small angle approximation**,

$$\sin \theta \approx \theta.$$

With this approximation, the undamped and unforced solution can be found analytically: $y = y_0 \cos (\sqrt{\alpha} t)$ (you can check this by substituting it into the original ODE).

```
initial_ang = np.pi/8 # [rad]
initial_ang_vel = 0. # [rad /s]

initial_cond = np.array([initial_ang, initial_ang_vel])
```

### Solving the Problem

Now we can solve the ODE system using the Runge-Kutta method we made above and plot the angle $y_0$ over time.

```
from scipy.integrate import odeint

# y = odeint(pendulum, initial_cond, t, args=(solve_args,))

# Solve Pendulum ODE with RK4
y = ode_int_rk(pendulum, initial_cond, t, solve_args)

y_small_ang = initial_ang*np.cos(np.sqrt(solve_args['alpha'])*t)

plt.plot(t, y[:,0], c='b', label='Angle')
plt.plot(t, y_small_ang, 'r--', label='Small angle approx.')
plt.xlabel('Time (s)')
plt.ylabel('Angle (rad)')
plt.legend(loc=2)
```

So we see that the pendulum oscillates as expected and the frequency matches the known small angle result reasonably well for a few swings. (Subject to the error in the small angle approximation. Try making `initial_ang` bigger or smaller to see where the approximation works well.)

Other perspectives we might be interested in are the the pendulum's trajectory in phase space, which we plot in a **phase diagram** and its spectral properties, via a **power spectrum**. We'll plot those now.

```
def plot_pendulum(t,y):
    """ Plot Angle, Phase Diagram, FFT. """

    fig = plt.figure()

    # Plot Angle
    ax_1 = fig.add_subplot(211)
    ax_1.plot(t, y[:,0], c='b')
    ax_1.set_xlabel('Time (s)')
    ax_1.set_ylabel('Angle (rad)')

    # Plot Phase Diagram
    ax_2 = fig.add_subplot(223)
    ax_2.plot(y[:,0], y[:,1], c='g')
    ax_2.set_xlabel('Angle (rad)')
    ax_2.set_ylabel('Angular Velocity (rad /s)')

    # Fourier Transform
    f_fft = np.fft.fftfreq(len(t), t[1]-t[0])
    y_fft = np.fft.fft(y[:,0])/np.sqrt(2*len(t))

    # Plot Power Spectrum
    ax_3 = fig.add_subplot(224)
    ax_3.plot(f_fft[:N/2]*2*np.pi, abs(y_fft[:N/2]), c='r')
    ax_3.set_xlim([0, 30])
    ax_3.set_xlabel('Ang Freq ($2 \pi$ Hz)')
    ax_3.set_ylabel('Power')
```

```
plot_pendulum(t,y)
```

The phase diagram (green, bottom left) shows each possible physical state for the system. We see the undamped, unforced pendulum follows a regular orbit. The power spectrum (red, bottom right) is found by taking a discrete Fourier transform using NumPy's `fftpack` module. We see a single narrow peak at $\omega \approx \sqrt{\alpha}$, the pendulum's natural frequency.

### Adding Friction

To better model a physical pendulum we might add a friction constant $k$, and see what happens when we solve the system with this included.

```
friction_constant = .5 # [kg /m /s]
solve_args['beta'] = friction_constant/mass_pendulum

# Solve Pendulum ODE with RK4
y = ode_int_rk(pendulum, initial_cond, t, solve_args)

plot_pendulum(t,y)
```

We see that over time the friction attenuates the pendulum's swing. The phase space trajectory no longer orbits consistently but spirals in, and the spectral peak has been widened out to include a spread of lower frequency components.

### Adding a Driving Force

```
driving_force = 10. # [N]
driving_freq = 10. # [2π /s]

solve_args['gamma'] = driving_force/mass_pendulum/length_pendulum
solve_args['omega'] = driving_freq

# Solve Pendulum ODE with RK4
y = ode_int_rk(pendulum, initial_cond, t, solve_args)

plot_pendulum(t,y)
```

The driven system is of course more interesting! The phase space trajectory looks chaotic, but the pendulum's behaviour is clear in the power spectrum, where we see we now have an additional peak at $\omega_F$.

Anyway, there's plenty interesting to explore in the damped forced pendulum system, but as we're just interested in the integration methods here, we'll move on. The takeaway is: by converting our second-order ODE into a system of first-order ODEs, we are able to solve it using the methods we've already written. And crucially, this can be done for any higher-order ODE.

## The Two-Step Adams-Bashforth Method

Remember that with these different methods, we're always looking to do the same thing: to choose a numerical approximation to the integral in

$$ y(t_{n+1}) = y(t_n) + \int_{t_n}^{t_{n+1}} \! f(t,y(t)) \, \mathrm{d}t. $$

So far, we've looked at **single-step** methods where we only use one known point to estimate the next one. But one way to improve the accuracy of our method is to use more known points. Such approximations are called **multistep methods**, and we'll look at an example now.

Where the Expliit Euler method takes the slope $f$ to be a constant on the interval $[t_n, t_{n+1}]$, the idea behind _Adams-Bashforth_ methods is to approxmiate $f$ by a [Lagrange interpolating polynomial](lagrange):

$$ P(t) = \sum_{j=1}^{m}{P_j(t)}   $$

where

$$ P_j(t) = y_j \prod_{\substack{k=1 \\ k \ne j}}^{m}{ \frac{t - t_k}{t_j - t_k} }. $$

Here $P(t)$ is the polynomial of degree $\le (m-1)$ that passes through the $m$ points $(t_1, y_1 = f(t_1))$, $(t_2, y_2 = f(t_2))$ $\dots$ $(t_m, y_m = f(t_m))$. We'll take the linear $(m = 2)$ interpolant on the point $t_{n}$ and an earlier point $t_{n-1}$, so we have

$$ P(t) = f(t_n, y_n)\frac{t - t_{n-1}}{t_n - t_{n-1}} + f(t_{n-1}, y_{n-1})\frac{t - t_{n}}{t_{n-1} - t_n}. $$

Now if we put this approximating polynomial into the integral of, we find

\begin{align}
\int_{t_n}^{t_{n+1}} \! f(t,y(t)) \, \mathrm{d}t \approx \int_{t_n}^{t_{n+1}} \! P(t) \, \mathrm{d}t &= \int_{t_n}^{t_{n+1}} \! \left[ f(t_n, y_n)\frac{t - t_{n-1}}{t_n - t_{n-1}} + f(t_{n-1}, y_{n-1})\frac{t - t_{n}}{t_{n-1} - t_n} \right] \mathrm{d}t \\
&= \frac{(t_n - t_{n+1})}{2(t_{n-1}-t_n)} \left[ f(t_n,y_n)(t_n + t_{n+1} - 2t_{n-1}) - f(t_{n-1},y_{n-1})(t_n - t_{n+1}) \right]
\end{align}

### Step Sizes

If we let $h_1 := t_n - t_{n-1}$ and $h_2 := t_{n+1} - t_n$ then

$$ \int_{t_n}^{t_{n+1}} \! P(t) \, \mathrm{d}t = \frac{h_2}{2 h_1} \left[ (2 h_1 + h_2) f(t_n,y_n)  - h_2 f(t_{n-1},y_{n-1}) \right]. $$

Putting this back into $\color{grey}{[1]}$, we get

$$ y(t_{n+1}) \approx y(t_{n}) + \frac{h_2}{2 h_1} \left[ (2 h_1 + h_2) f(t_n,y_n)  - h_2 f(t_{n-1},y_{n-1}) \right] $$

and our sequence of approximation points $y_n$ is calculated as

$$ y_{n+1} = y_n + \frac{h_2}{2 h_1} \left[ (2 h_1 + h_2) f(t_n,y_n)  - h_2 f(t_{n-1},y_{n-1}) \right] $$

for $n = 1, 2, \dots N$.

**If the steps are of equal size**, i.e. $h := h_1 = h_2$ we find

$$ y_{n+1} = y_n + \frac{3}{2} h f(t_n,y_n) - \frac{1}{2} h f(t_{n-1}, t_{n-1}) $$

which is the [standard two-step Adams-Bashforth method][multistep].

[lagrange]: http://mathworld.wolfram.com/LagrangeInterpolatingPolynomial.html
[multistep]: http://en.wikipedia.org/wiki/Linear_multistep_method#Families_of_multistep_methods

### Accuracy

Replacing $f(t,y(t))$ with the interpolant $P(t)$ [incurs a global error][wiki_order_cite] of order $\mathcal{O}(h^m)$, so in the case of the two-step method we have $\mathcal{O}(h^2)$.

Note that if you follow the same derivation with $m = 1$ you get the Euler method — so the Euler method is also in fact the one-step Adams-Bashforth method.

[wiki_order_cite]: http://en.wikipedia.org/wiki/Linear_multistep_method#CITEREFIserles1996

## Next

In [Part 3][part3] we'll look at an example of a **stiff** problem, which resists solution by the Explicit Euler method, and so discover another measure of a numerical method to go with its accuracy: the numerical **stability**. We'll then introduce **implicit methods**, which allow us to solve stiff problems.

[part3]: ./8_Stiff-Problems-Implicit-Methods-and-Computational-Cost.ipynb


# HEWL S-SAD Merging Statistics

Merging statistics are a useful means to assess data quality in crystallography. However, each statistic has inherent shortcomings. For example, R-merge will appear inflated if the multiplicity is high, and the Pearson correlation coefficients used for $CC_{1/2}$ are very sensitive to outliers.

Most scaling and merging programs output multiple merging statistics to get around these shortcomings. However, one can imagine that it could also be useful to customize certain parameters, such as how many resolution bins are used. Or, perhaps a better statistic will be developed that is worth implementing.

In this notebook, ``reciprocalspaceship`` is used to compute half-dataset correlation coefficients ($CC_{1/2}$ and $CC_{anom}$) for a dataset collected from a tetragonal hen egg-white lysozyme (HEWL) crystal at 6550 eV.
These data are unmerged, but were scaled in AIMLESS.
They contain sufficient sulfur anomalous signal to determine a solution by the SAD method.
This illustrates the use of ``rs`` to implement a merging routine, create a custom analysis, and could be useful as a template for other exploratory crystallographic data analyses.
As an example, we will compare half-dataset correlations computed using both Pearson and Spearman correlation coefficients.

```python
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_context("notebook", font_scale=1.3)
import numpy as np
```

```python
import reciprocalspaceship as rs
```

```python
print(rs.__version__)
```

    0.9.4

---
### Load scaled, unmerged data

This data has been scaled in AIMLESS. The data includes the image number and the scaled **I** and **SIGI** values.

```python
hewl = rs.read_mtz("data/HEWL_unmerged.mtz")
```

```python
hewl.head()
```

      BATCH
      I
      SIGI
      PARTIAL

      H
      K
      L

      0
      0
      4
      137
      696.5212
      87.83294
      False

      4
      520
      710.6812
      88.107025
      False

      4
      856
      672.05634
      87.75671
      False

      4
      1239
      642.47485
      87.90302
      False

      4
      2160
      655.71783
      87.74394
      False

```python
print(f"Number of observed reflections: {len(hewl)}")
```

    Number of observed reflections: 816804

---
### Merging with Inverse-Variance Weights

Since the input data are unmerged, we will implement the inverse-variance weighting scheme used by AIMLESS to merge the observations.
The weighted average is a better estimator of the true mean than the raw average, and this weighting scheme corresponds to the maximum likelihood estimator of the true mean if we assume that the observations are normally-distributed about the true mean.

The merged intensity for each reflection, $I_h$, can be determined from the observed intensities, $I_{h,i}$, and error estimates, $\sigma_{h,i}$, as follows:

\begin{equation}
I_h = \frac{\sum_{i}w_{h,i} I_{h,i}}{\sum_{i} w_{h,i}}
\end{equation}

where the weight for each observation, $w_{h,i}$ is given by:

\begin{equation}
w_{h,i} = \frac{1}{(\sigma_{h,i})^2}
\end{equation}

The updated estimate of the uncertainty, $\sigma_{h}$, is given by:

\begin{equation}
\sigma_{h} = \sqrt{\frac{1}{\sum_{i} w_{h,i}}}
\end{equation}

Let's start by implementing the above equations in a function that will compute the merged $I_h$ and $\sigma_h$. We will use the [Pandas groupby](https://pandas.pydata.org/pandas-docs/stable/user_guide/groupby.html) methods to apply this function across the unique Miller indices in the ``DataSet``. If we group Friedel pairs together, we will refer to the quantity as **IMEAN**, and if we keep the Friedel pairs separate we will refer to the quantities as **I(+)**  and **I(-)**.

```python
def merge(dataset, anomalous=False):
    """
    Merge dataset using inverse-variance weights.

    Parameters
    ----------
    dataset : rs.DataSet
        DataSet to be merged containing scaled I and SIGI
    anomalous : bool
        If True, I(+) and I(-) will be reported. If False,
        IMEAN will be reported

    Returns
    -------
    rs.DataSet
        Merged DataSet object
    """
    ds = dataset.hkl_to_asu(anomalous=anomalous)
    ds["w"]  = ds['SIGI']**-2
    ds["wI"] = ds["I"] * ds["w"]
    g = ds.groupby(["H", "K", "L"])

    result = g[["w", "wI"]].sum()
    result["I"] = result["wI"] / result["w"]
    result["SIGI"] = np.sqrt(1 / result["w"])
    result = result.loc[:, ["I", "SIGI"]]
    result.merged = True

    if anomalous:
        result = result.unstack_anomalous()

    return result
```

Using `anomalous=False`, this function can be used to compute **IMEAN** and **SIGIMEAN** by including both Friedel pairs:

```python
result1 = merge(hewl, anomalous=False)
```

```python
result1.sample(5)
```

      I
      SIGI

      H
      K
      L

      34
      14
      9
      238.14276
      5.412156

      5
      5
      10
      2619.72
      43.076794

      43
      8
      1
      45.92524
      2.000772

      22
      2
      7
      92.24401
      1.229599

      42
      10
      0
      53.67026
      3.489320

Using `anomalous=True`, this function can be used to compute **I(+)**, **SIGI(+)**, **I(-)**, and **SIGI(-)** by separating Friedel pairs:

```python
result2 = merge(hewl, anomalous=True)
```

```python
result2.sample(5)
```

      I(+)
      SIGI(+)
      I(-)
      SIGI(-)

      H
      K
      L

      11
      7
      16
      188.28104
      6.185037
      173.23112
      6.336241

      24
      24
      5
      17.531303
      1.161394
      17.531303
      1.161394

      36
      11
      2
      39.90348
      1.855634
      46.158234
      1.342185

      27
      11
      11
      310.0335
      8.241861
      327.22
      7.975232

      13
      13
      12
      124.78872
      2.328268
      124.78872
      2.328268

A variant of the above function is implemented in `rs.algorithms`, and we will use that implementation in the next section for computing merging statistics. This function computes **IMEAN**, **I(+)**, **I(-)**, and associated uncertainties for each unique Miller index.

```python
result3 = rs.algorithms.merge(hewl)
```

```python
result3.sample(5)
```

      IMEAN
      SIGIMEAN
      I(+)
      SIGI(+)
      I(-)
      SIGI(-)
      N(+)
      N(-)

      H
      K
      L

      11
      5
      9
      445.99692
      5.5288916
      446.19846
      7.563336
      445.76562
      8.102551
      64
      56

      30
      8
      3
      81.085785
      1.0904284
      72.2529
      1.5436918
      89.8823
      1.5405104
      60
      64

      15
      1
      5
      2664.3076
      32.183956
      2670.4834
      47.109356
      2658.9023
      44.07226
      56
      64

      8
      5
      0
      917.98047
      15.161337
      917.98047
      15.161337
      917.98047
      15.161337
      64
      64

      34
      27
      0
      50.074245
      4.3588166
      50.074245
      4.3588166
      50.074245
      4.3588166
      4
      4

---
### Merging with 2-fold Cross-Validation

To compute correlation coefficients we will repeatedly split our data into half-datasets. We will do this by randomly splitting the data using the image number. These half-datasets will be merged independently and used to determine uncertainties in the correlation coefficients. We will first write a method to randomly split our data, and then we will then write a method that automates the sampling and merging of multiple half-datasets.

```python
def sample_halfdatasets(data):
    """Randomly split DataSet into two equal halves by BATCH"""
    batch = data.BATCH.unique().to_numpy(dtype=int)
    np.random.shuffle(batch)
    halfbatch1, halfbatch2 = np.array_split(batch, 2)
    half1 = data.loc[data.BATCH.isin(halfbatch1)]
    half2 = data.loc[data.BATCH.isin(halfbatch2)]
    return half1, half2
```

```python
def merge_dataset(dataset, nsamples):
    """
    Merge DataSet using inverse-variance weighting scheme. This represents the
    maximum-likelihood estimator of the mean of the observed intensities assuming
    they are independent and normally distributed with the same mean.

    Sample means across half-datasets can be used to compute the merging statistics CC1/2 and CCanom.
    """
    dataset = dataset.copy()
    samples = []
    for n in range(nsamples):
        half1, half2 = sample_halfdatasets(dataset)
        mergedhalf1 = rs.algorithms.merge(half1)
        mergedhalf2 = rs.algorithms.merge(half2)
        result = mergedhalf1.merge(mergedhalf2, left_index=True, right_index=True, suffixes=(1, 2))
        result["sample"] = n
        samples.append(result)
    return rs.concat(samples).sort_index()
```

---
### Merge HEWL data

We will now merge the HEWL data, repeatedly sampling across half-datasets in order to assess the distribution of correlation coefficients.

```python
# This cell takes a few minutes with nsamples=15
merged = merge_dataset(hewl, 15)
```

```python
merged
```

      IMEAN1
      SIGIMEAN1
      I(+)1
      SIGI(+)1
      I(-)1
      SIGI(-)1
      N(+)1
      N(-)1
      IMEAN2
      SIGIMEAN2
      I(+)2
      SIGI(+)2
      I(-)2
      SIGI(-)2
      N(+)2
      N(-)2
      sample

      H
      K
      L

      0
      0
      4
      662.71674
      27.763823
      662.71674
      27.76382
      662.71674
      27.76382
      10
      10
      658.93616
      35.859802
      658.93616
      35.859802
      658.93616
      35.859802
      6
      6
      0

      4
      673.3035
      39.26171
      673.3035
      39.26171
      673.3035
      39.26171
      5
      5
      655.83997
      26.47925
      655.83997
      26.47925
      655.83997
      26.47925
      11
      11
      1

      4
      652.56683
      29.268377
      652.56683
      29.268377
      652.56683
      29.268377
      9
      9
      672.53235
      33.193497
      672.53235
      33.193497
      672.53235
      33.193497
      7
      7
      2

      4
      651.18304
      29.259443
      651.18304
      29.259443
      651.18304
      29.259443
      9
      9
      674.3303
      33.20654
      674.3303
      33.20654
      674.3303
      33.20654
      7
      7
      3

      4
      664.65784
      29.259375
      664.65784
      29.259375
      664.65784
      29.259375
      9
      9
      656.9748
      33.206642
      656.9748
      33.206642
      656.9748
      33.206642
      7
      7
      4

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      45
      10
      2
      7.669444
      6.6297607
      7.669444
      6.6297607
      NaN
      NaN
      1
      0
      22.89468
      4.1084785
      22.89468
      4.1084785
      NaN
      NaN
      3
      0
      10

      2
      14.465271
      4.4560146
      14.465271
      4.4560146
      NaN
      NaN
      2
      0
      25.364758
      5.6226034
      25.364758
      5.6226034
      NaN
      NaN
      2
      0
      11

      2
      18.59031
      8.728156
      18.59031
      8.728156
      NaN
      NaN
      1
      0
      18.685297
      3.8105915
      18.685297
      3.8105915
      NaN
      NaN
      3
      0
      12

      2
      14.465271
      4.4560146
      14.465271
      4.4560146
      NaN
      NaN
      2
      0
      25.364758
      5.6226034
      25.364758
      5.6226034
      NaN
      NaN
      2
      0
      13

      2
      15.318142
      3.9687188
      15.318142
      3.9687188
      NaN
      NaN
      3
      0
      30.170218
      7.3511124
      30.170218
      7.3511124
      NaN
      NaN
      1
      0
      14

187328 rows × 17 columns

---
### Compute $CC_{1/2}$ and $CC_{anom}$

We will first assign each reflection to a resolution bin and then we will compute the correlation coefficients

```python
merged, labels = merged.assign_resolution_bins(bins=15)
```

```python
groupby1 = merged.groupby(["sample", "bin"])[["IMEAN1", "IMEAN2"]]
pearson1 = groupby1.corr(method="pearson").unstack().loc[:, ("IMEAN1", "IMEAN2")]
pearson1.name = "Pearson"
spearman1 = groupby1.corr(method="spearman").unstack().loc[:, ("IMEAN1", "IMEAN2")]
spearman1.name = "Spearman"
results1 = rs.concat([pearson1, spearman1], axis=1)
results1 = results1.groupby("bin").agg(["mean", "std"])
```

```python
results1.head()
```

      Pearson
      Spearman

      mean
      std
      mean
      std

      bin

      0
      0.997269
      0.000416
      0.998856
      0.000104

      1
      0.998025
      0.000334
      0.999361
      0.000059

      2
      0.998973
      0.000313
      0.999649
      0.000054

      3
      0.999317
      0.000113
      0.999661
      0.000027

      4
      0.999386
      0.000075
      0.999680
      0.000015

```python
plt.figure(figsize=(8, 4))
plt.errorbar(results1.index, results1[("Pearson", "mean")],
             yerr=results1[("Pearson", "std")],
             color="#1b9e77",
             label=r"$CC_{1/2}$ (Pearson)")
plt.errorbar(results1.index, results1[("Spearman", "mean")],
             yerr=results1[("Spearman", "std")],
             color="#d95f02",
             label=r"$CC_{1/2}$ (Spearman)")
plt.xticks(results1.index, labels, rotation=45, ha='right', rotation_mode='anchor')
plt.ylabel("Correlation Coefficient")
plt.xlabel(r"Resolution Bin ($\AA$)")
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.grid(axis="y", linestyle='--')
plt.show()
```

It is important to note the scale on the y-axis -- this dataset is edge-limited, and as such the $CC_{1/2}$ is very high across all resolution bins. The Spearman CC appears higher across all resolution bins except at high resolution, and overall has a lower standard deviation among samples.
This is consistent with our expectation that Spearman CCs are a more robust estimator of correlation than Pearson CCs.

Let's now repeat this for the anomalous data, computing $CC_{anom}$:

```python
merged["ANOM1"] = merged["I(+)1"] - merged["I(-)1"]
merged["ANOM2"] = merged["I(+)2"] - merged["I(-)2"]

# Similar to CChalf, but we will only look at acentric reflections
groupby2 = merged.acentrics.groupby(["sample", "bin"])[["ANOM1", "ANOM2"]]
pearson2 = groupby2.corr(method="pearson").unstack().loc[:, ("ANOM1", "ANOM2")]
pearson2.name = "Pearson"
spearman2 = groupby2.corr(method="spearman").unstack().loc[:, ("ANOM1", "ANOM2")]
spearman2.name = "Spearman"
results2 = rs.concat([pearson2, spearman2], axis=1)
results2 = results2.groupby("bin").agg(["mean", "std"])
```

```python
results2.head()
```

      Pearson
      Spearman

      mean
      std
      mean
      std

      bin

      0
      0.405502
      0.048131
      0.492206
      0.029448

      1
      0.255115
      0.047082
      0.321767
      0.030535

      2
      0.343341
      0.052632
      0.432287
      0.020717

      3
      0.470596
      0.051916
      0.551680
      0.025049

      4
      0.586280
      0.043609
      0.615225
      0.024101

```python
plt.figure(figsize=(8, 4))
plt.errorbar(results2.index, results2[("Pearson", "mean")],
             yerr=results2[("Pearson", "std")],
             color="#1b9e77",
             label=r"$CC_{anom}$ (Pearson)")
plt.errorbar(results2.index, results2[("Spearman", "mean")],
             yerr=results2[("Spearman", "std")],
             color="#d95f02",
             label=r"$CC_{anom}$ (Spearman)")
plt.xticks(results2.index, labels, rotation=45, ha='right', rotation_mode='anchor')
plt.ylabel("Correlation Coefficient")
plt.xlabel(r"Resolution Bin ($\AA$)")
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.grid(axis="y", linestyle='--')
plt.show()
```

There is significant anomalous signal across all but the highest resolution bins. The Spearman CCs have smaller error bars than the corresponding Pearson CCs and the Spearman CCs are also higher in most bins. These differences highlight the influence of outlier measurements on the different correlation coefficients.

---
### Summary

We have used `reciprocalspaceship` to merge a dataset using inverse-variance weights. As part of this analysis, we performed repeated 2-fold cross-validation to compute Pearson and Spearman correlation coefficients and associated uncertainties. This relatively simple procedure to obtain uncertainty estimates for correlation coefficients is seldom done when analyzing merging quality. However, we can see that the standard deviation for computed correlation coefficients can be nearly $\pm0.1$ for quantities such as $CC_{anom}$. This is worth keeping in mind when analyzing SAD experiments because this dataset is very high quality (edge-limited) when one considers the $CC_{1/2}$. Putting these two correlation coefficients on the same axes emphasizes this point:

```python
plt.figure(figsize=(9, 6))
plt.errorbar(results1.index, results1[("Pearson", "mean")],
             yerr=results1[("Pearson", "std")],
             color='#1b9e77',
             label=r"$CC_{1/2}$ (Pearson)")
plt.errorbar(results1.index, results1[("Spearman", "mean")],
             yerr=results1[("Spearman", "std")],
             color='#d95f02',
             label=r"$CC_{1/2}$ (Spearman)")
plt.errorbar(results2.index, results2[("Pearson", "mean")],
             yerr=results2[("Pearson", "std")],
             color='#1b9e77',
             linestyle="--",
             label=r"$CC_{anom}$ (Pearson)")
plt.errorbar(results2.index, results2[("Spearman", "mean")],
             yerr=results2[("Spearman", "std")],
             color='#d95f02',
             linestyle="--",
             label=r"$CC_{anom}$ (Spearman)")
plt.xticks(results1.index, labels, rotation=45, ha='right', rotation_mode='anchor')
plt.ylabel(r"Correlation Coefficient")
plt.xlabel(r"Resolution Bin ($\AA$)")
plt.legend(loc='center left', bbox_to_anchor=(1, 0.5))
plt.grid(axis="y", linestyle='--')
plt.tight_layout()
```

Even a simple change to the procedure for computing merging statistics, such as substituting a Spearman correlation coefficient for a Pearson one, can alter the apparent quality of a dataset. By lowering the barrier to implementing new analyses, we hope that `reciprocalspaceship` can encourage the development of more robust indicators of crystallographic data quality.


# Lecture 6: More About Filters
Audio Processing, MED4, Aalborg University, 2021

Jesper Kjær Nielsen (jekjaer@gmail.com) and Cumhur Erkut (cer@create.aau.dk), Aalborg University

## A generalised phasor and correlation
In the next 20 minutes, you will learn
- the behaviour of the signal (a generalised phasor)
$$
    z^{n} = r^{n}\mathrm{e}^{j\omega n}
$$
- what correlation is
- how the simililarity between signals can be measured using correlation

### A generalised phasor
In the first lecture, we saw that a **complex-number** $z$ can be written as
$$
    z = a+jb = r\mathrm{e}^{j\omega}
$$
where $a$, $b$, $r$, and $\omega$ are the **real part**, the **imaginary part**, the **magnitude**, and the **phase** of $z$.

A very important signal is a complex number raised to **minus** the sampling index, i.e.,
$$
    z^{-n} = \left(r\mathrm{e}^{j\omega}\right)^{-n} = r^{-n}\mathrm{e}^{-j\omega n}
$$
which simplifies to a **phasor** for $r=1$. Thus, we can refer to $z^{-n}$ as a **generalised phasor**.

From Euler's formula, we can also have
\begin{align}
    z^{-n} &= r^{-n}\left[\cos(\omega n)-j\sin(\omega n)\right]\\
    &= r^{-n}\cos(\omega n)-jr^{n}\sin(\omega n)\ .
\end{align}

#### Examples of $z^{-n}$ for different values of $z$
Recall thath $z^{-n}$ is defined as
$$
    z^{-n} = r^{-n}\mathrm{e}^{-j\omega n}\ .
$$

Thus, $z^{-n}$ is a product of the two signals $r^{-n}$ and $\mathrm{e}^{-j\omega n}$ where the latter can also be written as (from Euler's formula)
$$
    \mathrm{e}^{-j\omega n} = \cos(\omega n)-j\sin(\omega n)\ .
$$

Let us see how these two signals behave for different values of $r$ and $\omega$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
nData = 50
samplingIndices = np.arange(nData)
magnitude = 1. # any nonnegative number
plt.figure(figsize=(10,6))
plt.plot(samplingIndices, magnitude**(-samplingIndices), linewidth=2), plt.grid(True)
plt.xlim((samplingIndices[0],samplingIndices[nData-1])), plt.xlabel('n'), plt.ylabel('$r^{-n}$');
```

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
nData = 50
samplingIndices = np.arange(nData)
digitalFreq = 10*np.pi/180 # radians/sample
phasor = np.exp(-1j*digitalFreq*samplingIndices)
plt.figure(figsize=(10,6))
plt.plot(samplingIndices, np.real(phasor), linewidth=2, label='$\cos(\omega n)$)')
plt.plot(samplingIndices, np.imag(phasor), linewidth=2, label='$-\sin(\omega n)$)'), plt.grid(True)
plt.xlim((samplingIndices[0],samplingIndices[nData-1])), plt.xlabel('n'), plt.legend();
```

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

def computeGenPhasor(complexNumber, samplingIndices):
    '''Compute the signal samples of a generalised phasor'''
    genPhasor = complexNumber**(-samplingIndices)
    nData = np.size(samplingIndices)
    overSamplingFactor = 50
    overSamplingIndices = np.arange(nData*overSamplingFactor)/overSamplingFactor
    overSampledGenPhasor = complexNumber**(-overSamplingIndices)
    return genPhasor, overSampledGenPhasor, overSamplingIndices

nData = 35
samplingIndices = np.arange(nData)

unitCircle = np.exp(1j*2*np.pi*np.arange(1000)/1000)
```

```python
magnitude = 0.9 # any nonnegative number
digitalFreq = 30*np.pi/180 # radians/sample
complexNumber = magnitude*np.exp(1j*digitalFreq)
genPhasor, overSampledGenPhasor, overSamplingIndices = computeGenPhasor(complexNumber, samplingIndices)

plt.figure(figsize=(10,6)), plt.subplot(2,1,1).axis('equal')
plt.plot(np.real(unitCircle), np.imag(unitCircle), linewidth=2, label='Unit cirle')
plt.plot(np.real(complexNumber), np.imag(complexNumber),'o', markersize=10, label='Complex number'), plt.grid(True)
plt.xlabel('Real part'), plt.ylabel('Imaginary part'), plt.legend(), plt.subplot(2,1,2)
plt.plot(overSamplingIndices, np.real(overSampledGenPhasor), 'b')
plt.plot(samplingIndices, np.real(genPhasor), 'bo', label='Real($z^{n}$)')
plt.plot(overSamplingIndices, np.imag(overSampledGenPhasor), 'r')
plt.plot(samplingIndices, np.imag(genPhasor), 'ro', label='Imag($z^{n}$)')
plt.xlim((samplingIndices[0],samplingIndices[nData-1])), plt.grid(True), plt.xlabel('n'), plt.legend();
```

### Correlation
How do you rank the signals in terms of similarity to $x_n$?

We can make a ompute compute the correlation between two signals $x_n$ and $y_n$ as
$$
    c_m = \sum_{n=-\infty}^\infty x_ny_{n+m}\ .
$$
This is called the **cross-correlation function** and useful in many audio applications such as
- time-difference of arrival (TDOA) estimation (to estimate the direction of a source using a microphone array)
- pitch/fundamental frequency estimation (lecture 12)
- **interpretation** of the Z-transform and Fourier transform (lecture 7)

An important special case of the cross-correlation function is for zero lag (i.e., $m=0$) where we obtain
$$
    c_0 = \sum_{n=-\infty}^\infty x_ny_{n}\ .
$$
Thus, we here **multiply** two functions and then **sum** the computed product.

#### Example: correlation of two simple signals
Consider the two signals
\begin{align}
    x_n &= 1\delta_n -2\delta_{n-1}+0.5\delta_{n-2}-1\delta_{n-3}\\
    y_n &= 1
\end{align}
where $\delta_n$ is our well-known **impulse** which is zero for all $n$ except for $n=0$ where it is 1.

To compute the correlation $c_0$ between these two signals, we first compute their product which is
$$
    x_ny_n = 1\delta_n -2\delta_{n-1}+0.5\delta_{n-2}-1\delta_{n-3}\ .
$$

Finally, we sum the product of these two signals and obtain
$$
    c_0 = \sum_{n=-\infty}^\infty x_ny_{n} = \sum_{n=0}^3 x_ny_{n} = -1.5\ .
$$

#### Example: correlation of a signal with a phasor
Let us consider the phasor
$$
    f_n = \mathrm{e}^{-j\omega n}
$$
and correlate that with a signal $x_n$. The cross-correlation $c_0$ is now
$$
    c_0 = \sum_{n=-\infty}^\infty x_n f_n = \sum_{n=-\infty}^\infty x_n \mathrm{e}^{-j\omega n}\ .
$$

When viewed as a **function of $\omega$**, this correlation is typically denoted $c_0 = H(\omega)$ and called the **discrete-time Fourier transform** (DTFT) which we will talk much more about in lecture 7.

Consider the case where the signal is a real-valued sinusoid given by
$$
    x_n = \cos(\omega_0 n)
$$
for $n=0,1,2,\ldots, N-1$.

We wish to compute the correlation $c_0 = H(\omega)$ for different values of $\omega$.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt

nData = 100
samplingIndices = np.arange(nData)
signalFreq = 2*np.pi*10/nData # radians/sample
signal = np.cos(signalFreq*samplingIndices)
```

```python
freqGrid = 2*np.pi*np.arange(nData)/nData
correlation = np.zeros(nData)+1j*np.zeros(nData)
for ii in np.arange(nData):
    phasor = np.exp(-1j*freqGrid[ii]*samplingIndices)
    correlation[ii] = np.sum(signal*phasor)
```

```python
plt.figure(figsize=(10,6))
plt.plot(freqGrid, np.abs(correlation), lineWidth=2, label='$|H(\omega)|$')
plt.plot(np.array([signalFreq,signalFreq]),np.array([0,nData/2]), '--', lineWidth=2, label='$\omega_0$')
plt.grid(True), plt.xlabel('$\omega$'), plt.ylabel('$|H(\omega)|$')
plt.xlim((0,np.pi)), plt.legend();
```

### Summary
1. We have looked at the generalised phasor
$$
    z^{-n} = r^{-n}\mathrm{e}^{-j\omega n}
$$
which is a product of two function
 1. $r^{-n}$: is either exponentially decaying or increasing in $n$
 2. $\mathrm{e}^{-j\omega n}$ is a phasor
2. We can compute the correlation between two signals as
$$
    c_0 = \sum_{n=-\infty}^\infty x_ny_{n}\ .
$$
The scalar $c_0$ will measure how simililar these two signals are.

### Active 5 minutes break
Consider a the simple filter
$$
    y_n = x_n + ay_{n-1}\ .
$$
1. Is this a feedforward or a feedback filter?
2. Sketch this filter using a multiplication, delay, and summation blocks.
3. Compute the impulse reponse of the filter for $a=0.5$ and $a=2$ by sending and impulse $\delta_n$ through the filter. What do you see?

## The Z-transform
In the next 20 minutes, you will learn
- how the Z-transform is defined
- what a transfer function is
- what the Z-transform is used for

### The definition of the Z-transform
For a general signal $x_n$, the Z-transform is defined as
$$
    X(z) = \mathcal{Z}\left\{x_n\right\} = \sum_{n=-\infty}^\infty x_nz^{-n}\ .
$$
where $z=r\mathrm{e}^{j\omega}$ .

Note that
- the Z-transform is essentially a correlation between the signal $x_n$ and our generalised phasor $z^{-n}$
- the Z-transform is very useful for analysing filters!
- if we set $r=1$, the Z-transform simplifies to the **discrete-time Fourier transform** (much more on this in lecture 7)

We can also write the Z-transform of $x_n$ as
$$
    X(z) = \mathcal{Z}\left\{x_n\right\} = \sum_{n=-\infty}^\infty x_nz^{-n} = \sum_{n=-\infty}^\infty \left[x_nr^{-n}\right]\mathrm{e}^{-j\omega n}\ .
$$

Thus, we can interpret the Z-transform as an operation doing two things:
1. multiply the signal $x_n$ by an exponentially increasing ($0<r<1$), decreasing ($r>1$), or constant ($r=1$) function $r^{-n}$
2. correlated the the weighted signal in 1. by phasors of different frequencies

#### Example: Z-transform of an impulse

An impulse is given by
$$
    \delta_n = \begin{cases}
        1 & n=0\\
        0 & \text{otherwise}
    \end{cases}\ .
$$

Therefore, the Z-transform of it is
$$
    \mathcal{Z}\left\{\delta_n\right\} = \sum_{n=-\infty}^{\infty} \delta_n z^{-n} = z^0 = 1\ .
$$

#### Example: Z-transform of a delay
The Z-transform of a delayed signal $x_{n-D}$
$$
    \mathcal{Z}\left\{x_{n-D}\right\} = \sum_{n=-\infty}^\infty x_{n-D} z^{-n}\ .
$$
We now replace $n$ by $m+D$, where $m$ is another sampling index, so that
\begin{align}
    \mathcal{Z}\left\{x_{n-D}\right\} &= \sum_{m+D=-\infty}^\infty x_{m+D-D} z^{-(m+D)} = \sum_{m=-\infty}^\infty x_{m} z^{-m} z^{-D} \\
    &= z^{-D}\sum_{m=-\infty}^\infty x_{m} z^{-m} = z^{-D}X(z)\ .
\end{align}

#### Example: Z-transform of a feedforward comb-filter
Recall from lecture 5 that a **feedforward comb-filter** is given by
$$
    y_n = x_n+bx_{n-D}\ .
$$

The Z-transform of this filter is
\begin{align}
    Y(z) &= \sum_{n=-\infty}^\infty y_nz^{-n} = \sum_{n=-\infty}^\infty x_nz^{-n} + b \sum_{n=-\infty}^\infty x_{n-D}z^{-n}\\
    &= X(z) + b\sum_{n=-\infty}^\infty x_{n}z^{-(n+D)} = X(z) + bX(z) z^{-D}\\
    &= (1+bz^{-D})X(z)\ .
\end{align}

### The transfer function
In the previous example, we saw that taking Z-transform of the output of a feedforward comb filter resulted in
$$
    Y(z) = (1+bz^{-D})X(z)\ .
$$

In another example, we sat that the Z-transform of an impulse was
$$
    \mathcal{Z}\left\{\delta_n\right\} = 1\ .
$$

Thus, if we set the input signal equal to an impulse (i.e., $x_n=\delta_n$), we get
$$
    Y(z) = (1+bz^{-D})\mathcal{Z}\left\{\delta_n\right\} = 1+bz^{-D}
$$
which is called the **transfer function** and typically denoted $H(z)$.

Since the filter output $y_n$ is the **impulse response** $h_n$ of the filter when $x_n=\delta_n$, we also see that the transfer function must be the Z-transform of the impulse response, i.e.,
$$
    H(z) = \sum_{n=-\infty}^\infty h_n z^{-n}\ .
$$

Note that the transfer function reduces to the **frequency response** when we $r=1$.

#### Example: Z-transform of a feedback comb-filter
Recall from lecture 5 that a **feedback comb-filter** is given by
$$
    y_n = x_n+ay_{n-D}\ .
$$

The Z-transform of this filter is
\begin{align}
    Y(z) &= \sum_{n=-\infty}^\infty y_nz^{-n} = \sum_{n=-\infty}^\infty x_nz^{-n} + a \sum_{n=-\infty}^\infty y_{n-D}z^{-n}\\
    &= X(z) + a\sum_{n=-\infty}^\infty y_{n}z^{-(n+D)} = X(z) + aY(z) z^{-D}\ .
\end{align}
If we now isolate $Y(z)$, we obtain
$$
    Y(z) = \frac{1}{1-az^{-D}}X(z) = H(z)X(z)
$$
where $H(z)$ is the **transfer function** of the filter.

#### Filtering and the Z-transform

Until now, we have represented the filtering operation using a difference equation, i.e.,
$$
    y_n = \sum_{m=0}^q b_m x_{n-m} + \sum_{m=1}^p a_m y_{n-m}
$$
where $\{a_m\}_{m=1}^p$ and $\{b_m\}_{m=0}^q$ are the **filter coefficients**.

Using the Z-transform, we can instead represent it as
$$
    Y(z) = H(z)X(z)
$$
where
$$
    H(z) = \frac{B(z)}{A(z)} = \frac{b_0+b_1z^{-1}+\cdots+b_qz^{-q}}{1-a_1z^{-1}-\cdots-a_pz^{-p}}
$$
is the **transfer function** of the filter.

The transfer function is useful for two things:
1. **Stability**: does the filter output go to zero as $n$ increases
2. **Frequency response:** how the filter modifies the frequency components in the input signal

### Summary
1. The Z-transform is given by
$$
    X(z) = \mathcal{Z}\left\{x_n\right\} = \sum_{n=-\infty}^\infty x_nz^{-n} = \sum_{n=-\infty}^\infty \left[x_nr^{-n}\right]\mathrm{e}^{-j\omega n}\ .
$$
Thus, we can interpret the Z-transform as an operation doing two things:
 1. multiply the signal $x_n$ by an exponentially increasing ($0<r<1$), decreasing ($r>1$), or constant ($r=1$) function $r^{-n}$
 2. correlated the the weighted signal in 1. by phasors of different frequencies
2. In the Z-domain, filtering can be written as multiplication, i.e.,
$$
    Y(z) = H(z)X(z)
$$
where $H(z)$ is the **transfer function**.
3. The transfer function $H(z)$ is the Z-transform of the filter's impulse response $h_n$ and important for analysing the filter.

## Poles and zeros
In the next 20 minutes, you will learn
- what roots and zeros are
- how roots can say something about the stability of the filter

Consider the simple **feedback** filter
$$
    y_n = x_n + ay_{n-1}
$$
which has the impulse response
$$
    h_n = \begin{cases}
        a^n & n\geq0\\
        0 & \text{otherwise}
    \end{cases}
$$
and the Z-transform
$$
    H(z) = \sum_{n=-\infty}^\infty h_nz^{-n} = \sum_{n=0}^\infty a^nz^{-n} = \frac{1}{1-az^{-1}}
$$
where the last equality follows from the **geometric series**.

Alternatively, and simpler, we can also derive the Z-transform directly from the difference equation as
$$
    \mathcal{Z}\{y_n\} = Y(z) = X(z) + az^{-1}Y(z)\ .
$$

If we now solve this for $Y(z)$, we obtain
$$
    Y(z) = \frac{1}{1-az^{-1}}X(z)
$$
from which we see that
$$
    H(z) = \frac{1}{1-az^{-1}}\ .
$$

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D

def computeAbsTransferFunction(filterCoef, xGrid, yGrid):
    transferFunction = np.zeros((nGridPoints,nGridPoints))+1j*np.zeros((nGridPoints,nGridPoints))
    for ii in np.arange(nGridPoints):
        for jj in np.arange(nGridPoints):
            z = xGrid[jj]+1j*yGrid[ii]
            transferFunction[ii,jj] = z/(z-filterCoef)
    absTransferFunction = np.abs(transferFunction)
    maxAbsTransferFunction = np.max(absTransferFunction)
    # limit the dynamic range
    return np.clip(absTransferFunction,maxAbsTransferFunction/1e12,maxAbsTransferFunction)
```

```python
filterCoef = 1.
nData = 25
samplingIndices = np.arange(nData)
impulseResponse = filterCoef**samplingIndices

nGridPoints = 200
xGrid = 3*np.arange(nGridPoints)/nGridPoints-1.5
yGrid = xGrid
absTransferFunction = computeAbsTransferFunction(filterCoef, xGrid, yGrid)

unitCircle = np.exp(1j*2*np.pi*np.arange(nGridPoints)/nGridPoints)
```

```python
plt.figure(figsize=(12,8))
plt.subplot(2,1,1)
plt.plot(samplingIndices, impulseResponse)
plt.xlabel('$n$'), plt.ylabel('$h_n$'), plt.xlim((samplingIndices[0],samplingIndices[-1]))
plt.subplot(2,1,2)
plt.imshow(absTransferFunction, extent=[xGrid[0],xGrid[-1],yGrid[0],yGrid[-1]], aspect='equal',\
           vmin=np.min(absTransferFunction), vmax=np.max(absTransferFunction), cmap='terrain'), plt.colorbar()
plt.plot(np.real(unitCircle),np.imag(unitCircle), 'r', linewidth=2, label='Unit circle')
plt.xlabel('Real'), plt.ylabel('Imag'), plt.title('$|H(z)|$'), plt.legend();
```

### Poles and zeros
We can write the transfer function $H(z)$ above as the ratio between two polynomials, i.e.,
$$
    H(z) = \frac{1}{1-az^{-1}} = \frac{z}{z-a} = \frac{B(z)}{A(z)}\ .
$$

Note that
- **zeros**: the roots of the equation $B(z) = 0$ are called **zeros**
- **poles**: the roots of the equation $A(z) = 0$ are called **poles**

For the above example, $z=0$ is, therefore, a zero, and $z=a$ is a pole.

As we saw previously, a general difference equation (i.e., filtering in the time-domain)
$$
    y_n = \sum_{m=0}^q b_m x_{n-m} + \sum_{m=1}^p a_m y_{n-m}
$$
can be written in the Z-domain as
$$
    Y(z) = H(z)X(z)
$$
where
\begin{align}
    H(z) &= \frac{B(z)}{A(z)} = \frac{b_0+b_1z^{-1}+\cdots+b_qz^{-q}}{1-a_1z^{-1}-\cdots-a_pz^{-p}}\\
    &= \frac{z^{p}}{z^{q}}\frac{b_0z^q+b_1z^{q-1}+\cdots+b_q}{z^p-a_1z^{p-1}-\cdots-a_p}\ .
\end{align}

To compute the **zeros** of $H(z)$, we have to solve
$$
    B(z) = 0\ .
$$

To compute the **poles** of $H(z)$, we have to solve
$$
    A(z) = 0\ .
$$

#### Pole-zero map
A pole-zero map is the complex plane containing
- the unit circle
- all zeros as a **circle** ('o')
- all poles as a **cross** ('x')

#### Filter stability
- **Time-domain**: A filter is said to be **stable** when its impulse response goes to zero for an increasing $n$
- **Z-domain**: A filter is said to be **stable** if **all poles are within the unit circle**!

Which one of these filters are stable?

### Summary
1. The transfer function of a filter can be written as
$$
    H(z) = \frac{B(z)}{A(z)} = \frac{b_0+b_1z^{-1}+\cdots+b_qz^{-q}}{1-a_1z^{-1}-\cdots-a_pz^{-p}}\ .
$$
2. The solutions to the equation $B(z) = 0$ are called **zeros**
3. The solutions to the equation $A(z) = 0$ are called **poles**
4. A pole-zero plot shows graphically the position of the zeros and poles in the complex plane
5. If all roots are inside the unit circle, the filter is stable

### Active 5 minutes break
Consider the filter with difference equation
$$
    y_n = 2x_n-x_{n-1}  + 2y_{n-1}-2y_{n-2}\ .
$$
1. Is the filter a feedforward or a feedback filter?
2. Compute the transfer function $H(z)$ of the filter.
3. Compute the poles and zeros and sketch them in a pole-zero plot. Is the filter stable?

---

Tip: A transfer function can be written as
$$
    H(z) = \frac{B(z)}{A(z)} = \frac{b_0+b_1z^{-1}+\cdots+b_qz^{-q}}{1-a_1z^{-1}-\cdots-a_pz^{-p}}\ .
$$

## Filter properties and types
In the next 20 minutes, you will learn
- how you can rearrange filters
- the four basic filter types

### Linearity of filters
Filters are so-called **linear tranforms** since they satisfy the following two properties
1. Homogenity
2. Additivity

#### Homogenity
In a homogeneous system, we obtain the same result by scaling a signal before or after we filter it, i.e.,
$$
    (aX(z))H(z) = aY(z)
$$
if $Y(z) = X(z)H(z)$.

#### Additivity
In a additive system, we obtain the same filter output by adding two input signals before filtering or by summing the filtered versions of these two input signals, i.e.,
\begin{align}
    Y(z) &= H(z) X(z) = H(z)(X_1(z)+X_2(z)) = H(z)X_1(z)+H(z)X_2(z)
\end{align}
if $X(z) = X_1(z)+X_2(z)$.

### Filters in series
The filter order does not matter since
$$
    Y(z) = F(z)G(z)X(z) = G(z)F(z)X(z)\ .
$$

### Filters in parallel
If we are filtering the same filter through two different **parallel** filters, we obtain
$$
    Y(z) = F(z)X(z) + G(z)X(z) = (G(z)+F(z))X(z)\ .
$$

### Basic filter types
There are **four** basic, ideal filter types
1. **Lowpass filter**: only low frequencies will pass the filter
2. **Highpass filter**: only high frequencies will pass the filter
3. **Bandpass filter**: only frequencies in a certain band will pass the filter
4. **Bandstop filter**: only frequenceies outside a certain bain will pass the filter (also sometimes called a **notch filter**)

### Summary
1. Filters are linear
2. The order in which you filter things does not matter
3. The Z-transform makes it easy to analyse what happens when we combine filters in different filters
4. There are four main filter types which are the
 1. **lowpass filter**
 2. **highpass filter**
 3. **bandpass filter**
 4. **bandstop filter** (also sometimes called a **notch filter**)


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.6 (Python 3.7.3-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at https://docs.sympy.org/1.6/

```python
a, b, g, d, k, l0 = symbols('a b g d k l0', real=True, positive=True)
```

```python
l0, a, b, g, d
ksq = k**2
A, B, G, D = a**2, b**2, g**2, d**2
TA = 2*A
A2B = A*A + B
C = TA + G*A2B
f1 = ksq + TA*k + A2B
f2 = 1 + G*k + D*ksq
den = A2B + C*k
f = l0 * f1 * f2 / den
f
```

```python
diff(f,l0)
```

```python
da=diff(f,a)
da.factor()
```

```python
db=diff(f,b)
db.factor()
```

```python
dg=diff(f,g)
dg.factor()
```

```python
dd=diff(f,d)
dd.factor()
```

```python

```


If the data shows a curvy trend, then linear regression will not produce very accurate results when compared to a non-linear regression because, as the name implies, linear regression presumes that the data is linear.
Let's learn about non linear regressions and apply an example on python. In this notebook, we fit a non-linear model to the datapoints corrensponding to China's GDP from 1960 to 2014.

Importing required libraries

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

Though Linear regression is very good to solve many problems, it cannot be used for all datasets. First recall how linear regression, could model a dataset. It models a linear relation between a dependent variable y and independent variable x. It had a simple equation, of degree 1, for example y = $2x$ + 3.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 2*(x) + 3
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
#plt.figure(figsize=(8,6))
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

Non-linear regressions are a relationship between independent variables $x$ and a dependent variable $y$ which result in a non-linear function modeled data. Essentially any relationship that is not linear can be termed as non-linear, and is usually represented by the polynomial of $k$ degrees (maximum power of $x$).

$$ \ y = a x^3 + b x^2 + c x + d \ $$

Non-linear functions can have elements like exponentials, logarithms, fractions, and others. For example: $$ y = \log(x)$$

Or even, more complicated such as :
$$ y = \log(a x^3 + b x^2 + c x + d)$$

Let's take a look at a cubic function's graph.

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph
y = 1*(x**3) + 1*(x**2) + 1*x + 3
y_noise = 20 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

As you can see, this function has $x^3$ and $x^2$ as independent variables. Also, the graphic of this function is not a straight line over the 2D plane. So this is a non-linear function.

Some other types of non-linear functions are:

### Quadratic

$$ Y = X^2 $$

```python
x = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

y = np.power(x,2)
y_noise = 2 * np.random.normal(size=x.size)
ydata = y + y_noise
plt.plot(x, ydata,  'bo')
plt.plot(x,y, 'r')
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Exponential

An exponential function with base c is defined by $$ Y = a + b c^X$$ where b ≠0, c > 0 , c ≠1, and x is any real number. The base, c, is constant and the exponent, x, is a variable.

```python
X = np.arange(-5.0, 5.0, 0.1)

##You can adjust the slope and intercept to verify the changes in the graph

Y= np.exp(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Logarithmic

The response $y$ is a results of applying logarithmic map from input $x$'s to output variable $y$. It is one of the simplest form of __log()__: i.e. $$ y = \log(x)$$

Please consider that instead of $x$, we can use $X$, which can be polynomial representation of the $x$'s. In general form it would be written as
\begin{equation}
y = \log(X)
\end{equation}

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = np.log(X)

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

### Sigmoidal/Logistic

$$ Y = a + \frac{b}{1+ c^{(X-d)}}$$

```python
X = np.arange(-5.0, 5.0, 0.1)

Y = 1-4/(1+np.power(3, X-2))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

# Non-Linear Regression example

For an example, we're going to try and fit a non-linear model to the datapoints corresponding to China's GDP from 1960 to 2014. We download a dataset with two columns, the first, a year between 1960 and 2014, the second, China's corresponding annual gross domestic income in US dollars for that year.

```python
import numpy as np
import pandas as pd

#downloading dataset
!wget -nv -O china_gdp.csv https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv

df = pd.read_csv("china_gdp.csv")
df.head(10)
```

    2019-08-11 16:41:36 URL:https://s3-api.us-geo.objectstorage.softlayer.net/cf-courses-data/CognitiveClass/ML0101ENv3/labs/china_gdp.csv [1218/1218] -> "china_gdp.csv" [1]

      Year
      Value

      0
      1960
      5.918412e+10

      1
      1961
      4.955705e+10

      2
      1962
      4.668518e+10

      3
      1963
      5.009730e+10

      4
      1964
      5.906225e+10

      5
      1965
      6.970915e+10

      6
      1966
      7.587943e+10

      7
      1967
      7.205703e+10

      8
      1968
      6.999350e+10

      9
      1969
      7.871882e+10

__Did you know?__ When it comes to Machine Learning, you will likely be working with large datasets. As a business, where can you host your data? IBM is offering a unique opportunity for businesses, with 10 Tb of IBM Cloud Object Storage: [Sign up now for free](http://cocl.us/ML0101EN-IBM-Offer-CC)

### Plotting the Dataset ###
This is what the datapoints look like. It kind of looks like an either logistic or exponential function. The growth starts off slow, then from 2005 on forward, the growth is very significant. And finally, it decelerate slightly in the 2010s.

```python
plt.figure(figsize=(8,5))
x_data, y_data = (df["Year"].values, df["Value"].values)
plt.plot(x_data, y_data, 'ro')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

### Choosing a model ###

From an initial look at the plot, we determine that the logistic function could be a good approximation,
since it has the property of starting with a slow growth, increasing growth in the middle, and then decreasing again at the end; as illustrated below:

```python
X = np.arange(-5.0, 5.0, 0.1)
Y = 1.0 / (1.0 + np.exp(-X))

plt.plot(X,Y)
plt.ylabel('Dependent Variable')
plt.xlabel('Indepdendent Variable')
plt.show()
```

The formula for the logistic function is the following:

$$ \hat{Y} = \frac1{1+e^{\beta_1(X-\beta_2)}}$$

$\beta_1$: Controls the curve's steepness,

$\beta_2$: Slides the curve on the x-axis.

### Building The Model ###
Now, let's build our regression model and initialize its parameters.

```python
def sigmoid(x, Beta_1, Beta_2):
     y = 1 / (1 + np.exp(-Beta_1*(x-Beta_2)))
     return y
```

Lets look at a sample sigmoid line that might fit with the data:

```python
beta_1 = 0.10
beta_2 = 1990.0

#logistic function
Y_pred = sigmoid(x_data, beta_1 , beta_2)

#plot initial prediction against datapoints
plt.plot(x_data, Y_pred*15000000000000.)
plt.plot(x_data, y_data, 'ro')
```

Our task here is to find the best parameters for our model. Lets first normalize our x and y:

```python
# Lets normalize our data
xdata =x_data/max(x_data)
ydata =y_data/max(y_data)
```

#### How we find the best parameters for our fit line?
we can use __curve_fit__ which uses non-linear least squares to fit our sigmoid function, to data. Optimal values for the parameters so that the sum of the squared residuals of sigmoid(xdata, *popt) - ydata is minimized.

popt are our optimized parameters.

```python
from scipy.optimize import curve_fit
popt, pcov = curve_fit(sigmoid, xdata, ydata)
#print the final parameters
print(" beta_1 = %f, beta_2 = %f" % (popt[0], popt[1]))
```

     beta_1 = 690.453018, beta_2 = 0.997207

Now we plot our resulting regression model.

```python
x = np.linspace(1960, 2015, 55)
x = x/max(x)
plt.figure(figsize=(8,5))
y = sigmoid(x, *popt)
plt.plot(xdata, ydata, 'ro', label='data')
plt.plot(x,y, linewidth=3.0, label='fit')
plt.legend(loc='best')
plt.ylabel('GDP')
plt.xlabel('Year')
plt.show()
```

## Practice
Can you calculate what is the accuracy of our model?

```python
# write your code here

# split data into train/test
msk = np.random.rand(len(df)) < 0.8
train_x = xdata[msk]
test_x = xdata[~msk]
train_y = ydata[msk]
test_y = ydata[~msk]

# build the model using train set
popt, pcov = curve_fit(sigmoid, train_x, train_y)

# predict using test set
y_hat = sigmoid(test_x, *popt)

# evaluation
print("Mean absolute error: %.2f" % np.mean(np.absolute(y_hat - test_y)))
print("Residual sum of squares (MSE): %.2f" % np.mean((y_hat - test_y) ** 2))
from sklearn.metrics import r2_score
print("R2-score: %.2f" % r2_score(y_hat , test_y) )

```

    Mean absolute error: 0.04
    Residual sum of squares (MSE): 0.00
    R2-score: 0.91

Double-click __here__ for the solution.

Want to learn more?

IBM SPSS Modeler is a comprehensive analytics platform that has many machine learning algorithms. It has been designed to bring predictive intelligence to decisions made by individuals, by groups, by systems – by your enterprise as a whole. A free trial is available through this course, available here: SPSS Modeler

Also, you can use Watson Studio to run these notebooks faster with bigger datasets. Watson Studio is IBM's leading cloud solution for data scientists, built by data scientists. With Jupyter notebooks, RStudio, Apache Spark and popular libraries pre-packaged in the cloud, Watson Studio enables data scientists to collaborate on their projects without having to install anything. Join the fast-growing community of Watson Studio users today with a free account at Watson Studio

Thanks for completing this lesson!

Author:  Saeed Aghabozorgi

Saeed Aghabozorgi, PhD is a Data Scientist in IBM with a track record of developing enterprise level applications that substantially increases clients’ ability to turn data into actionable knowledge. He is a researcher in data mining field and expert in developing advanced analytic methods like machine learning and statistical modelling on large datasets.

Copyright © 2018 Cognitive Class. This notebook and its source code are released under the terms of the MIT License.


```python
import numpy as np
import matplotlib.pyplot as plt
import seaborn as sns
import sstools as ss
%matplotlib inline
sns.set_style('darkgrid')
```

# Let's generate some data from a state space model

```python
np.random.seed(12345)  # set random seed
```

We'll generate data according to a simple state space model and then recover the underlying model via the Kalman filter/smoother. We follow the conventions of [Durbin and Koopman (2012)](http://www.ssfpack.com/DKbook.html).

### Observation model

$$\begin{align}
y_t &= Z_t \alpha_t + \varepsilon_t \\
\varepsilon_t &\sim \mathcal{N}(0, H_t)
\end{align}$$

```python
Z = np.array([[1, -1], [0, 1], [1, 0]])  # observation matrix
H = 0.1 * np.array([[1, 0, 0], [0, 1, 0.2], [0, 0.2, 1]])

Np, Nm = Z.shape
```

```python
assert(H.shape == (Np, Np))
```

### State evolution model

$$\begin{align}
\alpha_{t+1} &= T_t \alpha_t + R_t \eta_t \\
\eta_t &\sim \mathcal{N}(0, Q_t)
\end{align}$$

```python
th = 10 * np.pi / 180  # rotation angle
T = np.array([[np.cos(th), np.sin(th)], [-np.sin(th), np.cos(th)]])  # rotation matrix
R = np.array([1.1, 1])[:, np.newaxis]
Q = np.array([0.05])[:, np.newaxis]

_, Nr = R.shape
```

```python
assert(T.shape == (Nm, Nm))
assert(R.shape[0] == Nm)
assert(Q.shape == (Nr, Nr))
```

### Prior

$$
\alpha_1 \sim \mathcal{N}(a_1, P_1)
$$

```python
a_init = np.zeros(Nm)
P_init = 5 * np.eye(Nm)
```

```python
assert(P_init.shape == (Nm, Nm))
```

## Now generate some data

```python
Ny = 100  # number of samples
```

Preallocate data arrays:

```python
alpha = np.empty((Ny, Nm))
y = np.empty((Ny, Np))
eps = np.empty((Ny, Np))
eta = np.empty((Ny, Nr))
```

Initialize:

```python
alpha[0] = np.random.multivariate_normal(a_init, P_init)
```

```python
for t in range(Ny):
    eps[t] = np.random.multivariate_normal(np.zeros(Np), H)
    eta[t] = np.random.multivariate_normal(np.zeros(Nr), Q)
    y[t] = Z.dot(alpha[t]) + eps[t]
    if t + 1 < Ny:
        alpha[t + 1] = T.dot(alpha[t]) + R.dot(eta[t])
```

```python
plt.figure(figsize=(10, 4))
plt.plot(y);
```

# Kalman filter (forward)

Now filter! In DK conventions, this is
$$\begin{align}
v_t &= y_t - Z_t a_t  \\
F_t &= Z_t P_t Z_t^\top + H_t  \\
aa \equiv \mathbb{E}(\alpha_t\vert y_{1:t}) &= a_t + P_t Z_t^\top F_t^{-1} v_t  \\
PP \equiv \mathrm{var}(\alpha_t \vert y_{1:t}) &= P_t - P_t Z_t^\top F_t^{-1} Z_t P_t  \\
K_t &= T_t P_t Z_t^\top F_t^{-1}  \\
L_t &= T_t - K_t Z_t  \\
a_{t + 1} &= T_t a_t + K_t v_t  \\
P_{t + 1} &= T_t P_t(T_t - K_t Z_t)^\top + R_t Q_t R_t^\top
\end{align}$$

Preallocate some arrays for the filter.

```python
a = np.empty((Ny, Nm))
P = np.empty((Ny, Nm, Nm))
aa = np.empty((Ny, Nm))
PP = np.empty((Ny, Nm, Nm))
v = np.empty((Ny, Np))
F = np.empty((Ny, Np, Np))
K = np.empty((Ny, Nm, Np))
L = np.empty((Ny, Nm, Nm))
```

Set some priors:

```python
a[0] = a_init
P[0] = P_init
```

```python
for t in range(Ny):
    v[t] = y[t] - Z.dot(a[t])
    F[t] = Z.dot(P[t]).dot(Z.T) + H
    Finv = np.linalg.inv(F[t])
    aa[t] = a[t] + P[t].dot(Z.T).dot(Finv).dot(v[t])
    PP[t] = P[t] - P[t].dot(Z.T).dot(Finv).dot(Z).dot(P[t])
    K[t] = T.dot(P[t]).dot(Z.T).dot(Finv)
    L[t] = T - K[t].dot(Z)
    if t + 1 < Ny:
        a[t + 1] = T.dot(a[t]) + K[t].dot(v[t])
        P[t + 1] = T.dot(P[t]).dot(L[t].T) + R.dot(Q).dot(R.T)
```

```python
plt.figure(figsize=(10, 4))
plt.plot(aa)
plt.plot(alpha)
plt.title('Actual and recovered states')
pass
```

# Kalman smoother (backward)

First define the posterior of the state vector given all the data as
$$\begin{align}
p(\alpha_t \vert y_{1:N}) = \mathcal{N}(\hat{\alpha}_t, V_t)
\end{align}$$

The recursions for the smoothed state $\hat{\alpha}$ are then

$$\begin{align}
r_n &= 0  \\
N_n &= 0  \\
r_{t - 1} &= Z_t^\top F_t^{-1} v_t + L_t^\top r_t  \\
N_{t - 1} &= Z_t^\top F_t^{-1} Z_t + L_t^\top N_t L_t  \\
\hat{\alpha}_t &= a_t + P_t r_{t - 1}  \\
V_t &= P_t - P_t N_{t - 1} P_t
\end{align}$$

Now, preallocate some arrays and initialize:

```python
r = np.empty((Ny, Nm))
ahat = np.empty((Ny, Nm))
N = np.empty((Ny, Nm, Nm))
V = np.empty((Ny, Nm, Nm))

ahat[0] = a_init
V[0] = P_init
r[-1] = 0
N[-1] = 0
```

```python
for t in range(Ny - 1, 0, -1):
    Finv = np.linalg.inv(F[t])

    r[t - 1] = Z.T.dot(Finv).dot(v[t]) + L[t].T.dot(r[t])
    N[t - 1] = Z.T.dot(Finv).dot(Z) + L[t].T.dot(N[t]).dot(L[t])
    ahat[t] = a[t] + P[t].dot(r[t - 1])
    V[t] = P[t] - P[t].dot(N[t - 1]).dot(P[t])
```

```python
plt.figure(figsize=(10, 4))
plt.plot(ahat)
plt.plot(alpha)
plt.title('Actual and recovered states')
pass
```

# Disturbance smoother/fast state smoother

As an alternative to the Kalman smoother, Durbin and Koopman recommend a disturbance smoother for the noise terms $\varepsilon$ and $\eta$. This recursion has lower memory requirements (using only $v_t$, $F_t$, and $K_t$) and does not require multiplication by the often dense $a_t$ and $P_t$.

This system of recursions (for the posterior means) is given by

$$\begin{align}
u_t &= F_t^{-1} v_t - K_t^\top r_t  \\
\hat{\varepsilon}_t &= H_t u_t  \\
\hat{\eta}_t &= Q_t R_t^\top r_t  \\
r_{t - 1} &= Z_t^\top u_t + T_t^\top r_t
\end{align}$$

with additional recursions for the variances (cf. Durbin and Koopman 4.5.3).

These recursions provide a particularly efficient means of calculating the smoothed state estimates $\hat{\alpha}_t$:

$$\begin{align}
\hat{\alpha}_1 &= a_1 + P_1 r_0  \\
\hat{\alpha}_{t + 1} &= T_t \hat{\alpha}_t + R_t Q_t R_t^\top r_t
\end{align}$$

This takes advantage of the typical sparsity of $T$ and $RQR$. However, it does not lend itself to calculating the posterior smoothed variances $V$.

Thus, if we only care about the posterior smoothed state means, we can do so very efficiently by saving only $F$, $K$,  and $v$ from the filtering step, calculate $r$ from the recursion using $u$ above, and run the model forward in time to get $\hat{\alpha}$.

### Implementation of the fast state smoother

First, we initialize $r$ and $\hat{\alpha}$:

```python
# initialize r
r = np.empty((Ny, Nm))
r[-1] = 0
alphat = np.empty((Ny, Nm))
```

Then we recursively calculate $r$, working backward in time:

```python
# calculate r
for t in range(Ny - 1, -1, -1):
    Finv = np.linalg.inv(F[t])

    u = Finv.dot(v[t]) - K[t].T.dot(r[t])
    thisr = Z.T.dot(u) + T.T.dot(r[t])
    if t > 0:
        r[t - 1] = thisr
    else:
        r_init = thisr

```

Finally, we run the state space model forward in time

```python
# run model forward
alphat[0] = a_init + P_init.dot(r_init)
RQR = R.dot(Q).dot(R.T)

for t in range(0, Ny - 1):
    alphat[t + 1] = T.dot(alphat[t]) + RQR.dot(r[t])
```

```python
plt.figure(figsize=(10, 4))
plt.plot(alphat)
plt.plot(alpha)
plt.title('Actual and recovered states')
pass
```

```python
plt.figure(figsize=(10, 4))
plt.plot(alphat - ahat)
plt.title('Difference between fast and standard state smoothers')
pass
```

# Drawing posterior samples (simulation smoothing)

We want to be able to sample from the smoothed posterior for the states, $p(\alpha_t|y_{1:N})$. We can do this using the technique of [Durbin and Koopman (2002)](http://biomet.oxfordjournals.org/content/89/3/603.short), which involves following steps:

1. Draw disturbance terms ($\varepsilon$ and $\eta$) from the *prior*, denoted $w^+$.
1. Simulate the model using these disturbances, generating a state sequence, $\alpha^+$ and observations $y^+$.
1. Using the Kalman filter and smoother, calculate smoothed estimates of the state sequence conditioned
    on the actual and the simulated data, $\hat{\alpha} = \mathbb{E}(\alpha|y_{1:n})$ and
    $\hat{\alpha}^+ = \mathbb{E}(\alpha|y^+_{1:n})$.
1. Combine these estimates with the simulated state sequence. The result,
    $\alpha^+ - \hat{\alpha}^+ + \hat{\alpha}$, is a draw from the posterior.

## Implementing the posterior sampler

Preallocate arrays:

```python
alpha_plus = np.empty((Ny, Nm))
y_plus = np.empty((Ny, Np))
```

### Draw disturbances

```python
eps_sim = np.random.multivariate_normal(np.zeros(Np), H, Ny)
eta_sim = np.random.multivariate_normal(np.zeros(Nr), Q, Ny)
```

### Simulate model

```python
alpha_plus[0] = np.random.multivariate_normal(a_init, P_init)

for t in range(Ny):
    y_plus[t] = Z.dot(alpha_plus[t]) + eps_sim[t]
    if t + 1 < Ny:
        alpha_plus[t + 1] = T.dot(alpha_plus[t]) + R.dot(eta_sim[t])
```

### Get smoothed means

Now run the data through:

```python
vv, KK, FFinv = ss.kalman_filter(y, a_init, P_init, Z, H, T, R, Q)
alpha_hat = ss.fast_state_smoother(vv, KK, FFinv, a_init, P_init, Z, T, R, Q)
```

```python
vv, KK, FFinv = ss.kalman_filter(y_plus, a_init, P_init, Z, H, T, R, Q)
alpha_hat_plus = ss.fast_state_smoother(vv, KK, FFinv, a_init, P_init, Z, T, R, Q)
```

### And combine...

```python
alpha_draw = alpha_plus - alpha_hat_plus + alpha_hat
```

```python
plt.plot(alpha_draw)
pass
```

### Let's pull this into a function:

```python
adraw = ss.simulate(y, a_init, P_init, Z, H, T, R, Q)
plt.plot(adraw)
pass
```

# Let's make a posterior

```python
Ns = 1000  # number of samples

alpha_samples = np.empty((Ns, Ny, Nm))
```

```python
for idx in range(Ns):
    alpha_samples[idx] = ss.simulate(y, a_init, P_init, Z, H, T, R, Q)
```

```python
trans = 5 / Ns
plt.figure(figsize=(10, 8))

for idx in range(Nm):
    plt.subplot(Nm, 1, idx + 1)
    plt.plot(alpha_samples[..., idx].T, color=sns.color_palette()[idx], alpha=trans)
    plt.plot(alpha[..., idx], color=sns.color_palette()[idx], linestyle='--', label='Actual')
    plt.plot(alphat[..., idx], color=sns.color_palette()[idx], label='Posterior')
    plt.legend()
pass
```

```python

```


# Van der Pol oscillator
We will look at the second order differentual equation (see https://en.wikipedia.org/wiki/Van_der_Pol_oscillator):

$$
{d^2y_0 \over dx^2}-\mu(1-y_0^2){dy_0 \over dx}+y_0= 0
$$

```python
from __future__ import division, print_function
import itertools
import numpy as np
import sympy as sp
import matplotlib.pyplot as plt
#from pyodesys.native.gsl import NativeGSLSys as SymbolicSys
from pyodesys.native.cvode import NativeCvodeSys as SymbolicSys
sp.init_printing()
%matplotlib inline
print(sp.__version__)
```

Note that we imported ``NativeCvodeSys`` as ``SymbolicSys``, this speed up the time of integration by more than an order of magnitude due to using compiled C++ code for our mathematical expressions.

One way to reduce the order of our second order differential equation is to formulate a system of first order ODEs, using:

$$ y_1 = \dot y_0 $$

which gives us:

$$
\begin{cases}
\dot y_0 = y_1 \\
\dot y_1 = \mu(1-y_0^2) y_1-y_0
\end{cases}
$$

Let's call this system of ordinary differential equations vdp1:

```python
vdp1 = lambda x, y, p: [y[1], -y[0] + p[0]*y[1]*(1 - y[0]**2)]
mu_val = 2.5
y0_1 = [0.0, 1.0]
y0_1, (y0_1[0], vdp1(0, y0_1, [mu_val])[0])
```

An alternative would be to use use the Liénard transformation:

$$ y = x - x^3/3 - \dot x/\mu $$

```python
transf = lambda y, dydx, p: [y[0], y[0] - y[0]**3/3 - dydx[0]/p[0]]
x, mu = sp.symbols('x mu', real=True)
y = [yi(x) for yi in sp.symbols('y:2', cls=sp.Function)]
dydx = [yi.diff(x) for yi in y]
[sp.Eq(yi, expr, evaluate=False) for yi, expr in zip(y, transf(y, dydx, [mu]))]  # Just for displaying
```

which gives us (we could generate this result using SymPy):
$$
\begin{cases}
\dot y_0 = \mu \left(y_0-\frac{1}{3}y_0^3-y_1\right) \\
\dot y_1 = \frac{1}{\mu} y_0
\end{cases}
$$

```python
vdp2 = lambda x, y, p: [p[0]*(y[0] - y[0]**3/3 - y[1]), y[0]/p[0]]
calc_y0_2 = lambda y0, mu: transf(y0, vdp1(0, y0, [mu]), [mu])
y0_2 = calc_y0_2(y0_1, mu_val)
(y0_2, y0_2[0], vdp2(0, y0_2, [mu_val])[0])
```

```python
def solve_and_plot(odesys, y0, tout, mu, indices=None, integrator='native', **kwargs):
    plt.figure(figsize=(16, 4))
    xout, yout, info = odesys.integrate(tout, y0, [mu], integrator=integrator, **kwargs)
    plt.subplot(1, 2, 1)
    odesys.plot_result(indices=indices, ls=('-',), c=('k', 'r'))
    plt.legend(loc='best')
    plt.subplot(1, 2, 2)
    odesys.plot_phase_plane()
    info.pop('internal_xout')  # too much output
    info.pop('internal_yout')
    return len(xout), info
```

```python
tend = 25
```

```python
odesys1 = SymbolicSys.from_callback(vdp1, 2, 1, names='y0 y1'.split())
odesys1.exprs
```

```python
for mu in [0, 3, 9]:
    solve_and_plot(odesys1, y0_1, np.linspace(0, tend, 500), mu)
```

As we see, the period ($\tau$) varies with $\mu$, in 1952 Mary Cartwright derived an approximate formula for $\tau$ (valid for large $\mu$):

$$
\tau  = (3 - 2 \ln 2)\mu + 2 \alpha \mu^{-1/3}
$$

where $\alpha \approx 2.338$

```python
tau = lambda mu: 1.6137056388801094*mu + 4.676*mu**(-1./3)
for mu in [20, 40, 60]:
    solve_and_plot(odesys1, y0_1, np.linspace(0, 5*tau(mu), 500), mu)
```

For larger values of $\mu$ we run into trouble (the numerical solver fails).
The phase portrait is not well resolved due to rapid variations in y1.

Let us look at our alternative formulation:

```python
odesys2 = SymbolicSys.from_callback(vdp2, 2, 1, names='y0 y1'.split())
odesys2.exprs
```

```python
solve_and_plot(odesys2, y0_2, tend, mu_val, nsteps=2000)
```

This looks much better. Let's see if the solver has an easier time dealing with this formulation of y2 for large values of $\mu$:

```python
ls = itertools.cycle(('-', '--', ':'))
for mu in [84, 160, 320]:
    y0_2 = calc_y0_2(y0_1, mu)
    print(y0_2)
    solve_and_plot(odesys2, y0_2, np.linspace(0, 5*tau(mu), 500), mu)
```

Indeed it has.

## Stiffness

Let us compare the performance of explicit and implicit steppers (Adams and BDF respectivecly) for varying values of $\mu$

```python
solve_and_plot(odesys2, calc_y0_2(y0_1, mu_val), tend, mu_val, nsteps=2000)
J = odesys2.get_jac()
J
```

For this simple system we can afford calculating the eigenvalues analytically

```python
odesys2._NativeCode._written_files
```

```python
symbs = odesys2.dep + tuple(odesys2.params)
symbs
```

```python
Jeig = J.eigenvals().keys()
eig_cbs = [sp.lambdify(symbs, eig, modules='numpy') for eig in Jeig]
Jeig
```

```python
eigvals = np.array([(eig_cbs[0](*(tuple(yvals)+(mu_val + 0j,))),
                     eig_cbs[1](*(tuple(yvals)+(mu_val + 0j,)))) for yvals in odesys2._internal[1]])
```

```python
plt.plot(odesys2._internal[0], odesys2.stiffness(), label='from SVD')
plt.plot(odesys2._internal[0], np.abs(eigvals[:,0])/np.abs(eigvals[:,1]), label='analytic')
plt.legend()
```

## Audio
Plotting is instructive from a mathematical standpoint, but these equations were often investigated by to *listening* to audio amplified by electrical circuits modelled by the equation. So let us generate some audio.

```python
def arr_to_wav(arr, rate=44100):
    from IPython.display import Audio
    from scipy.io.wavfile import write
    scaled = np.int16(arr/np.max(np.abs(arr)) * 32767)
    write('test.wav', rate, scaled)
    return Audio('test.wav')
```

```python
xout, yout, info = odesys2.integrate(np.linspace(0, 500*tau(40.0), 2*44100), y0_1, [40.0], integrator='native')
arr_to_wav(yout[:, 0])
```

```python
def overlay(tend_mu, odesys=odesys2, time=3, rate=44100, plot=False):
    yout_tot = None
    for tend, mu in tend_mu:
        xout, yout, info = odesys.integrate(np.linspace(0, tend*tau(mu[0]), time*rate), y0_1, mu, integrator='native')
        print(tend, mu, tend*tau(mu[0]))
        if yout_tot is None:
            yout_tot = yout[:, 0]
        else:
            yout_tot += yout[:, 0]
    if plot:
        plt.figure(figsize=(16,4))
        plt.plot(yout_tot[slice(None) if plot is True else slice(0, plot)])
    return arr_to_wav(yout_tot, rate=rate)
```

```python
overlay([
    (400, [2.0]),
    (410, [2.1]),
], plot=10000)
```

## Forced van der pol oscillator

```python
vdp_forced = lambda x, y, p: [y[1], p[1]*sp.sin(p[2]*x) - y[0] + p[0]*y[1]*(1 - y[0]**2)]
odesys_forced = SymbolicSys.from_callback(vdp_forced, 2, 3)
overlay([(700, [8, 1, 0.5])], odesys_forced, plot=5000)  # Non-chaotic behavior
```

```python
overlay([(700, [8, 1.2, 0.6])], odesys_forced, plot=5000)  # Chaotic behavior
```

### Transient $\mu$

```python
vdp_transient = lambda x, y, p: [y[1], - y[0] + p[0]*sp.exp(-p[1]*x)*y[1]*(1 - y[0]**2)]
odesys_transient = SymbolicSys.from_callback(vdp_transient, 2, 2)
odesys_transient.exprs
```

```python
overlay([
    (440, [0.1, 1/2500.]),
    (445, [0.5, 1/1000.]),
    (890, [0.1, 2/2500.]),
    (896, [0.5, 2/1000.]),
], odesys_transient, plot=-1)
```

```python
odesys2._native._written_files
```

```python

```