# Realization of Recursive Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Quantization of Variables and Operations

As for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-Effects), the practical realization of recursive filters may suffer from the quantization of variables and algebraic operations. The effects of [coefficient quantization](quantization_of_coefficients.ipynb) were already discussed. This section takes a look at the quantization of variables.  We limit the investigations to the recursive part of a second-order section (SOS), since any recursive filter of order $N \geq 2$ can be [decomposed into SOSs](cascaded_structures.ipynb).

The computation of the output signal $y[k] = \mathcal{H}\{ x[k] \}$ by a difference equation involves a number of multiplications and additions. As discussed already for [non-recursive filters](../nonrecursive_filters/quantization_effects.ipynb#Quantization-of-Signals-and-Operations), multiplying two numbers in a binary representation (e.g. [two's complement](https://en.wikipedia.org/wiki/Two's_complement) or [floating point](https://en.wikipedia.org/wiki/Floating_point)) requires requantization of the result to keep the word length constant. The addition of two numbers may fall outside the maximum/minimum values of the representation and may suffer from clipping.

The resulting round-off and clipping errors depend on the number and sequence of algebraic operations. These depend on the structure used for implementation of the SOSs. For ease of illustration we limit our discussion to the [direct form I and II](direct_forms.ipynb). Similar insights can be achieved in a similar manner for other structures.

### Analysis of Round-Off Errors

Round-off errors are a consequence of reducing the word length after a multiplication. In order to investigate the influence of these errors on a recursive filter, the statistical model for [round-off errors in multipliers](../nonrecursive_filters/quantization_effects.ipynb#Model-for-round-off-errors-in-multipliers) as introduced for non-recursive filters is used. We furthermore neglect clipping.

The difference equation for the recursive part of a SOS realized in direct form I or II is given as

\begin{equation}
y[k] =  x[k] - a_1 \, y[k-1] - a_2 \, y[k-2]
\end{equation}

where $a_0 = 1$, $a_1$ and $a_2$ denote the coefficients of the recursive part. Introducing the requantization after the multipliers into the difference equation yields the output signal $y_Q[k]$

\begin{equation}
y_Q[k] = x[k] - \mathcal{Q} \{ a_{1} \, y[k-1] \} - \mathcal{Q} \{ a_{2} \, y[k-2] \}
\end{equation}

where $\mathcal{Q} \{ \cdot \}$ denotes the requantizer. Requantization is a non-linear process which results in a requantization error. If the value to be requantized is much larger that the quantization step $Q$, the average statistical properties of this error can be modeled as additive uncorrelated white noise. Introducing the error into above difference equation gives

\begin{equation}
y_Q[k] = x[k] - a_1 \, y[k-1] - e_1[k] - a_2 \, y[k-2] - e_2[k]
\end{equation}

where the two white noise sources $e_1[k]$ and $e_2[k]$ are assumed to be uncorrelated to each other. This difference equation can be split into a set of two difference equations

\begin{align}
y_Q[k] &= y[k] + e[k] \\
y[k] &=  x[k] - a_1 \, y[k-1] - a_2 \, y[k-2] \\
e[k] &= - e_1[k] - e_2[k] - a_1 \, e[k-1] - a_2 \, e[k-2]
\end{align}

The first difference equation computes the desired output signal $y[k]$ as a result of the input signal $x[k]$. The second one the additive error $e[k]$ due to requantization as a result of the requantization error $- (e_1[k] + e_2[k])$ injected into the recursive filter.
The power spectral density (PSD) $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the error $e[k]$ is then given as

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = | H(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 \cdot (\Phi_{e_1 e_1}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + \Phi_{e_2 e_2}(\mathrm{e}^{\,\mathrm{j}\,\Omega}))
\end{equation}

According to the model for the requantization errors, their PSDs are given as $\Phi_{e_1 e_1}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{e_2 e_2}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{Q^2}{12}$. Introducing this together with the transfer function of the SOS yields

\begin{equation}
\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \left| \frac{1}{1 + a_1 \, \mathrm{e}^{\,-\mathrm{j}\,\Omega} + a_2 \, \mathrm{e}^{\,-\mathrm{j}\,2\,\Omega}} \right|^2 \cdot \frac{Q^2}{6}
\end{equation}

#### Example - Round-off error of a SOS

The following example evaluates the error $e[k] = y_Q[k] - y[k]$ for a SOS which only consists of a recursive part. The desired system response $y[k]$ is computed numerically by floating point operations with double precision, $y_Q[k]$ is computed by applying a uniform midtread quantizer after the multiplications. The system is excited by uniformly distributed white noise. Besides the PSD $\Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, the signal-to-noise ratio (SNR) $10 \cdot \log_{10} \left( \frac{\sigma_y^2}{\sigma_e^2} \right)$ in dB of the filter is evaluated.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 8192  # length of signals
w = 8  # wordlength for requantization of multiplications

def uniform_midtread_quantizer(x):
    '''Uniform mid-tread quantizer w/o limiter.'''
    return Q * np.floor(x/Q + 1/2)

def no_quantizer(x):
    '''Dummy quantizer.'''
    return x

def sos_df1(x, a, requantize=None):
    '''Realization of a recursive SOS with round-off of multiplications.'''
    y = np.zeros(len(x)+2)  # initial value appended
    for k in range(len(x)):
        y[k] = x[k] - requantize(a[1]*y[k-1]) - requantize(a[2]*y[k-2])

    return y[0:-2]

# cofficients of the SOS
p = 0.90*np.array([np.exp(1j*np.pi/3), np.exp(-1j*np.pi/3)])
a = np.poly(p)
# quantization step
Q = 1/(2**(w-1))

# compute input signal
x = np.random.uniform(low=-1, high=1, size=N)
# compute output signals w and w/o requantization
yQ = sos_df1(x, a, requantize=uniform_midtread_quantizer)
y = sos_df1(x, a, requantize=no_quantizer)
# compute requantization error
e = yQ-y
# Signal-to-noise ratio
SNR = 10*np.log10(np.var(y)/np.var(e))
print('SNR due to requantization: %f dB' % SNR)

# estimate PSD of requantization error
nf, Pxx = sig.welch(e, window='hamming', nperseg=256, noverlap=128)
Pxx = .5*Pxx  # due to normalization in scipy.signal
Om = 2*np.pi*nf
# compute frequency response of system
w, H = sig.freqz([1, 0, 0], a)

# plot results
plt.figure(figsize=(10, 4))
plt.plot(Om, Pxx/Q**2 * 12, 'b', label=r'$|\hat{\Phi}_{ee}(e^{j \Omega})|$')
plt.plot(w, np.abs(H)**2 * 2, 'g', label=r'$|H(e^{j \Omega})|^2$')
plt.title('Estimated PSD and transfer function of requantization noise')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$Q^2/12$')
plt.axis([0, np.pi, 0, 100])
plt.legend()
plt.grid()
```

    SNR due to requantization: 44.983949 dB

### Small Limit Cycles

Besides the requantization noise, recursive filters may be subject to periodic oscillations present at the output. These undesired oscillations are termed *limit cycles*. Small limit cycles emerge from the additive round-off noise due to requantization after a multiplication. The feedback in a recursive filter leads to a feedback of the requantization noise. This may lead to a periodic output signal with an amplitude range of some quantization steps $Q$, even after the input signal is zero. The presence, amplitude and frequency of small limit cycles depends on the location of poles and the structure of the filter. A detailed treatment of this phenomenon is beyond the scope of this notebook and can be found in the literature.

#### Example - Small limit cycles of a SOS

The following example illustrates small limit cycles for the system investigated in the previous example. The input signal is uniformly distributed white noise till time-index $k=256$ and zero for the remainder.

```python
# compute input signal
x = np.random.uniform(low=-1, high=1, size=256)
x = np.concatenate((x, np.zeros(1024)))
# compute output signal
yQ = sos_df1(x, a, requantize=uniform_midtread_quantizer)

# plot results
np.seterr(divide='ignore')
plt.figure(figsize=(10, 3))
plt.plot(20*np.log10(np.abs(yQ)))
plt.title('Level of output signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$|y_Q[k]|$ in dB')
plt.grid()

plt.figure(figsize=(10, 3))
k = np.arange(1000, 1050)
plt.stem(k, yQ[k]/Q, use_line_collection=True)
plt.title('Output signal for zero input')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_Q[k] / Q$ ')
plt.axis([k[0], k[-1], -3, 3])
plt.grid()
```

**Exercise**

* Estimate the period of the small limit cycles. How is it related to the poles of the system?
* What amplitude range is spanned?

Solution: The period of the small limit cycles can be estimated from the second illustration as $P = 6$. The normalized frequency of a harmonic exponential signal with the same periodicity is given as $\Omega_0 = \frac{2 \pi}{P} = \frac{\pi}{3}$. The poles of the system can be extracted from the code of the first example as $z_{\infty 0,1} = 0.9 \cdot e^{\pm j \frac{\pi}{3}}$. The periodicity of the small limit cycles is hence linked to the normalized frequency of the poles. The amplitude range spanned by the small limit cycles is $\pm 2 Q$.

### Large Limit Cycles

Large limit cycles are periodic oscillations of a recursive filter due to overflows in the multiplications/additions. As for small limit cycles, large limit cycles may be present even after the input signal is zero. Their level is typically in the range of the minimum/maximum value of the requantizer. Large limit cycles should therefore be avoided in a practical implementation. The presence of large limit cycles depends on the scaling of input signal and coefficients, as well as the strategy used to cope for clipping. Amongst others, they can be avoided by proper scaling of the coefficients to prevent overflow. Again, a detailed treatment of this phenomenon is beyond the scope of this notebook and can be found in the literature.

#### Example - Large limit cycles of a SOS

The following example illustrates large limit cycles for the system investigated in the first example. In order to trigger large limit cycles, the coefficients of the filter have been doubled. The input signal is uniformly distributed white noise till time-index $k=256$ and zero for the remainder.

```python
def uniform_midtread_quantizer(x, xmin=1):
    '''Uniform mid-tread quantizer with limiter.'''
    # limiter
    x = np.copy(x)
    if x <= -xmin:
        x = -1
    if x > xmin - Q:
        x = 1 - Q
    # linear uniform quantization
    xQ = Q * np.floor(x/Q + 1/2)

    return xQ

# compute input signal
x = np.random.uniform(low=-1, high=1, size=256)
x = np.concatenate((x, np.zeros(1024)))
# compute output signal
yQ = sos_df1(x, 2*a, requantize=uniform_midtread_quantizer)

# plot results
plt.figure(figsize=(10, 3))
plt.plot(20*np.log10(np.abs(yQ)))
plt.title('Level of output signal')
plt.xlabel(r'$k$')
plt.ylabel(r'$|y_Q[k]|$ in dB')
plt.grid()

plt.figure(figsize=(10, 3))
k = np.arange(1000, 1050)
plt.stem(k, yQ[k], use_line_collection=True)
plt.title('Output signal for zero input')
plt.xlabel(r'$k$')
plt.ylabel(r'$y_Q[k]$ ')
plt.grid()
```

**Exercise**

* Determine the period of the large limit cycles. How is it related to the poles of the system?

Solution: The period of the large limit cycles can be estimated from the second illustration as $P = 6$. The normalized frequency of a harmonic exponential signal with the same periodicity is given as $\Omega_0 = \frac{2 \pi}{P} = \frac{\pi}{3}$. The poles of the system can be extracted from the code of the first example as $z_{\infty 0,1} = 0.9 \cdot e^{\pm j \frac{\pi}{3}}$. The periodicity of the large limit cycles is hence linked to the normalized frequency of the poles.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


```python
# %load /Users/facai/Study/book_notes/preconfig.py
%matplotlib inline

import matplotlib.pyplot as plt
import seaborn as sns
sns.set(color_codes=True)
#sns.set(font='SimHei')
plt.rcParams['axes.grid'] = False

import numpy as np

#from IPython.display import SVG
def show_image(filename, figsize=None, res_dir=True):
    if figsize:
        plt.figure(figsize=figsize)

    if res_dir:
        filename = './res/{}'.format(filename)

    plt.imshow(plt.imread(filename))
```

Chapter 9 Convolutional Networks
================================

Convolution is a specialized kind of linear operation.

### 9.1 The Convolution Operation

\begin{align}
    s(t) &= \int x(a) w(t-a) \mathrm{d}a \\
         &= (x \ast w)(t)
\end{align}
where $x$ is the input, $w$ is the kernel, and $s(t)$ is referred to as the feature map.

Since convolution is commutative,

\begin{align}
    S(i, j) &= (I \ast K)(i, j) &= \sum_m \sum_n I(m, n) K(i - m, j - n) \\
            &= (K \ast I)(i, j) &= \sum_m \sum_n I(i - m, j - n) K(m, n)
\end{align}

Many machine learning libraries implement cross-correlation but call it convolution.

Discrete convolution can be viewed as multiplication by a matrix.

```python
show_image("fig9_1.png", figsize=(8, 8))
```

### 9.2 Motivation

Convolution leverages three important ideas:

+ sparse interactions: fewer parameters
+ parameter sharing: tied weights
+ equivariant representations:
  a function $f(x)$ is equivariant to a funtion $g$ if $f(g(x)) = g(f(x))$.

```python
show_image("fig9_5.png", figsize=(8, 5))
```

### 9.3 Pooling

A pooling function replaces the output of the new at a certain location with a summary statistic of the nearby outputs.

popular pooling functions:

+ max
+ average
+ L2 norm
+ weighted average

strong prior: function must be invariant to small translations.

```python
show_image("fig9_7.png", figsize=(10, 8))
```

```python
show_image("fig9_9.png", figsize=(10, 8))
```

### 9.4 Convolution and Pooling as an Infinitely Strong Prior

Prior: weak or strong <== how concentrated the probability density in the prior is.

We can image a convolutional net as being similar to a fully connected net, but with an infinitely strong prior over its weights: the weights for one hidden unit must be identical to the weights of its neighbor, but shifted in space.

convolution and pooling can cause underfitting.

### 9.5 Variants of the Basic Convolution Function

4-D tensors: (batch_size, height, width, channels)

Three zero-padding:

+ valid convolution: m - k +1 = m - (k - 1)
+ same convolution: m，补零至核中心
+ full convolution: m + k - 1，补零至核边角

see [details below](https://cn.mathworks.com/help/matlab/ref/conv2.html).

```python
show_image("matlab_conv_2d.png", figsize=(12, 8))
```

```python
A = np.random.rand(3, 3)
A
```

    array([[ 0.35212801,  0.37922789,  0.65163256],
           [ 0.59534513,  0.74098237,  0.42971692],
           [ 0.73934346,  0.73590775,  0.10633755]])

```python
B = np.random.rand(4, 4)
B
```

    array([[ 0.20413973,  0.24286469,  0.83290138,  0.30052197],
           [ 0.93643405,  0.41111035,  0.60742698,  0.0043744 ],
           [ 0.20255814,  0.3365866 ,  0.26242548,  0.85468478],
           [ 0.64287861,  0.38078144,  0.22940042,  0.62623427]])

```python
def gen_kernel_fn(kernel):
    def kernel_fn(a, x_start, y_start):
        x_size, y_size = kernel.shape
        a_slice = a[x_start:x_start+x_size, y_start:y_start+y_size]
        return (a_slice * kernel).sum()
    return kernel_fn

def calc_conv2d_res_size(a, kernel):
    res_x_size = a.shape[0] - kernel.shape[0] + 1
    res_y_size = a.shape[1] - kernel.shape[1] + 1
    return res_x_size, res_y_size

def conv2d(a, kernel):
    kernel_fn = gen_kernel_fn(kernel)

    res_x_size, res_y_size = calc_conv2d_res_size(a, kernel)
    res = np.zeros((res_x_size, res_y_size))
    for x in range(res_x_size):
        for y in range(res_y_size):
            res[x, y] = kernel_fn(a, x, y)
    return res
```

```python
# valid convolution
conv2d(B, A)
```

    array([[ 2.25524107,  1.82679291],
           [ 2.14415575,  1.6570156 ]])

```python
def calc_2d_pad_width(target_size, real_size):
    pad_x_width = (target_size[0] - real_size[0]) / 2
    pad_y_width = (target_size[1] - real_size[1]) / 2
    return np.array([[pad_x_width] * 2, [pad_y_width] * 2], dtype=np.int)
```

```python
def zero_pad_and_conv2d(a, kernel, target_size):
    res_size = calc_conv2d_res_size(a, kernel)
    pad_width = calc_2d_pad_width(target_size, res_size)
    a_pad = np.pad(a, pad_width, 'constant', constant_values=0)

    return conv2d(a_pad, kernel)
```

```python
# same convolution
same_conv_size = B.shape
zero_pad_and_conv2d(B, A, same_conv_size)
```

    array([[ 0.98847255,  1.71888185,  1.64232006,  1.17086158],
           [ 1.29107212,  2.25524107,  1.82679291,  1.59511597],
           [ 1.43133448,  2.14415575,  1.6570156 ,  1.63554738],
           [ 0.93613643,  1.13343863,  1.44076255,  1.01712864]])

```python
# full convolution
full_conv_size = [x1 + x2 for (x1, x2) in zip(B.shape, A.shape)]
print("full convolution size: {}".format(full_conv_size))
zero_pad_and_conv2d(B, A, full_conv_size)
```

    full convolution size: [7, 7]

    array([[ 0.02170772,  0.17605365,  0.41822408,  0.82445577,  0.83695663,
             0.22218895],
           [ 0.1873004 ,  0.98847255,  1.71888185,  1.64232006,  1.17086158,
             0.18214847],
           [ 0.55696519,  1.29107212,  2.25524107,  1.82679291,  1.59511597,
             0.74033208],
           [ 0.76561571,  1.43133448,  2.14415575,  1.6570156 ,  1.63554738,
             0.97337498],
           [ 0.40824929,  0.93613643,  1.13343863,  1.44076255,  1.01712864,
             0.67378397],
           [ 0.41892063,  0.49192708,  0.52026328,  0.62915348,  0.31826381,
             0.22051462]])

##### Comparison of local connections, convolution, and full connections.

convolutional layers $\to$ tiled convolution $\to$ locally connected layer

```python
show_image("fig9_16.png", figsize=(10, 8))
```

compute gradients: See Goodfellow (2010) for a full derivation of the equations in the fully general multi-dimensional, multi-example case.

### 9.6 Structured Outputs

Convolutional networks => output a high-dimensional, structured object. For example, to label every pixel in an image.

### 9.7 Data Types

Convolutional networks can porcess inputs with varying spatial extents which is the same kind of observations.

### 9.8 Efficient Convolution Algorithms

+ Fourier transform: point-wise multiplication in frequency domain.
+ Sometimes d-dimensional kernel is *separable*.

### 9.9 Random or Unsupervised Features

Today, most convolutinoal networks are trained in a purely supervised fashion.

### 9.10 The Neuroscientific Basis for Convolutional Networks

### 9.11 Convolutional Networks and the History of Deep Learning

```python

```


# Simulation Study for Goodness-of-Fit Tests via Persistence Diagrams

In this notebook, we illustrate how to devise goodness-of-fit tests for point patterns based on statistics derived from the persistence diagram. In a first step, this concerns deviation tests based on scalar quantities that are asymptotically normal. In the second step, the persistence diagram serves as functional summary statistic for a global envelope test.

```R
rm(list = ls())
library(GET)
library(ggplot2)
library(grid)
library(gridExtra)
library(parallel)
library(reshape)
library(spatstat)
library(TDA)
```

    Loading required package: spatstat.data

    Loading required package: nlme

    Loading required package: rpart

    spatstat 1.62-2       (nickname: ‘Shape-shifting lizard’)
    For an introduction to spatstat, type ‘beginner’

For the entire simulation study, the null model is a homogeneous Poisson point process with intensity $2$ in a $10 \times 10$ observation window.

```R
win = owin(c(0,10), c(0,10))
```

## Deviation Test for Complete Spatial Randomness

As a first step, we apply the central limit theorem of the persistence diagram in large domains to derive a deviation test against complete spatial randomness via two TDA-based summary statistics.

As a test statistic based on clusters, we use the integral over the number of cluster deaths in a finite time interval $[0, r_{\mathsf c}]$ with $r_{\mathsf c} \le r_{\mathsf f}$, i.e.,
$$\int_0^{r_{\mathsf c}} \mathsf{PD}^0(\mathcal P_n)([0, d]) {\rm d}d.$$
After subtracting the mean, this test statistic becomes reminiscent of the classical Cram\'er-von-Mises statistic except that we do not consider squared deviations. Although squaring would make it easier to detect two-sided deviations, it would also require knowledge of quantiles of the square integral of a centered Gaussian process. Albeit possible, this incurs substantial computational expenses.  Hence, our simpler alternative has the appeal that as an integral of a Gaussian process, the test statisticis asymptotically normal and therefore characterized by its mean and variance.

However, this statistic requires that the intensity is known in advance, whereas in practice we need to estimate it from data. To arrive at an intensity-adapted version, we know that for every $a > 0$,
$$\mathsf{PD}^0((a\mathcal P)_{an})([0, ad]) = \mathsf{PD}^0( \mathcal P_n)([0, d]).$$
Moreover, if $\mathcal P$ is a Poisson point process with intensity $\lambda_0$, then the scaled process $a\mathcal P$ is a Poisson point process with intensity $a^{-2}\lambda_0$. Noting that also the Lebesgue integral produces a factor $a$ through a change of variable, we arrive at the test statistc

Hence, after taking these scalings into account, we obtain the intensity-dependent test statistic:
\begin{align}
        \label{t0Def}
        T_{\mathsf C} = \frac1{\sqrt{\lambda}|W|}\int_0^{r_{\mathsf C} / \sqrt\lambda} \mathsf{PD}^0(\mathcal P_n)([0, d]) {\rm d}d.
\end{align}

To compute the persistence diagram, we rely on the package [TDA](https://cran.r-project.org/web/packages/TDA/index.html).

```R
t_vals0 <- seq(0, .299, by = 1e-3)

tda0_raw <- function(coords){
    #' Death times of a point pattern
    #'
    #' @param coords coordinates of point pattern

    diag <- alphaComplexDiag(coords)$diagram
    diag_df <-  data.frame(dimension = diag[,1],
                           death = diag[,3])
    sort(diag_df[diag_df$dimension == 0, 'death'])
}

tda0 <- function(X,
                 r = 0,
                 tx = t_vals0,
                 correction = NULL){
    #' Adapter of death times for spatstat scaled by number of points
    #'
    #' @param X point pattern
    #' @param r time point
    #' @param t_vals time values for which the accumulated death count is computed
    #' @param correction edge correction
    dt <- tda0_raw(cbind(X$x, X$y))

    #intensity
    intens <- X$n / area.owin(win)

    #rescale integration domain
    tx <- tx * sqrt(2 / intens)

    fv(data.frame(r = tx,
                  trans = findInterval(tx, dt) / (sqrt(intens) * area.owin(win))),
       valu = 'trans',
       labl = c('r', 'lab'),
       fname='Diag')
}
```

As a test statistic based on loops, we use the accumulated persistence function, which aggregates the life times of all loops with birth times in a time interval $[0, r_{\mathsf L}]$ with $r_{\mathsf L} \le r_{\mathsf f}$. That is,
$$\int_{[0, r_{\mathsf L}] \times [0, r_{\mathsf f}]} (d - b) \mathsf{PD}^1(\mathcal P_n)({\rm d}b, {\rm d}d).$$
After centering and rescaling, this statisitc converges in the large-volume limit to a normal random variable.
As in the cluster-based test, we adapt to the intensity, to obtain
\begin{align}
        \label{t1Def}
        T_{\mathsf L} =  \frac1{\sqrt{\lambda}|W|}\int_{\big[0, r_{\mathsf L} / \sqrt \lambda\big] \times \big[0, r_{\mathsf f} / \sqrt \lambda\big]} (d - b) \mathsf{PD}^1(\mathcal P_n)({\rm d}b, {\rm d}d)
\end{align}

The statistics $T_{\mathsf C}$ and $T_{\mathsf L}$ are specific possibilities to define scalar characteristics from the persistence diagram. Depending on the application context other choices, such as $\mathsf{APF}^0$ instead of $T_{\mathsf C}$ could be useful. However, in the simulation study below, we found the weighting by life times of clusters to be detrimental.

```R
t_vals1 <- seq(0, .499, by=5e-3)
ty1 <- seq(0, .299, by=1e-2)

tda1_raw <- function(coords,
                     max_bt = 10){
    #' accumulated persistence function of a point pattern
    #'
    #' @param coords coordinates of point pattern
    #' @param max_bt add maximum birth time to avoid out of bound error

    diag <- alphaComplexDiag(coords)$diagram
    diag_df <- data.frame(dimension = diag[,1],
                           birth = diag[,2],
                           life = (diag[,3] - diag[,2]))

    #only 1-dimensional features
    diag_df <- diag_df[diag_df$dimension == 1, 2:3]
    diag_df <- diag_df[order(diag_df$birth),]
    rownames(diag_df) <- c()
    rbind(diag_df, c(max_bt, max_bt))
}

tda1 <- function(X,
                 r = 0,
                 tx = t_vals1,
                 correction = NULL){
    #' Adapter of the accumulated persistence function for spatstat
    #'
    #' @param X point pattern
    #' @param r time point
    #' @param tx times at which the APF is computed

    tda1_vals <- tda1_raw(cbind(X$x, X$y))
    tda1_vals <- cbind(tda1_vals[,1], cumsum(tda1_vals$life))
    birth_locs = findInterval(tx, tda1_vals[,1])

    #intensity
    intens <- X$n / area.owin(win)

    #rescale integration domain
    tx <- tx * sqrt(2 / intens)

    fv(data.frame(r = tx,
                  trans = tda1_vals[1 + birth_locs, 2] / (sqrt(intens) * area.owin(win))),
      valu = 'trans',
      labl = c('r', 'lab'),
      fname='Diag')
}
```

Finally, we compare the TDA-based statistics to a quantity derived from the classical Ripley L-function. More precisely, we let
$$T_{\mathsf {Rip}} = \int_0^{r_{\mathsf {Rip}}} \hat L(r){\rm d}r$$
denote the integral of the estimated Ripley's L-function until a radius $r_{\mathsf{Rip}} > 0$. We note that there is no need for a normalization as estimating the L-function already takes into account an estimated intensity. By existing [FCLTs for Ripley's K-function](https://link.springer.com/article/10.1007/s10986-015-9266-z), we expect that $T_{\mathsf{Rip}}$ is also asymptotically normal.

```R
Lstat <- function(X,
                  rmax = .5){
    #' Statistics based on Ripley's L-function.
    #'
    #' @param X point pattern
    #' @param rmax Maximum desired value of the radius
    L <- Lest(X,
              rmax = rmax)

    res <-fv(data.frame(r = L$r,
                    trans = L$trans),
       valu = 'trans',
       labl = c('r', 'lab'),
       fname='Diag')
}
```

### Exploratory Analysis

In the simulation study, we explore how reliably the cluster- and loop-based tests can detect deviations from the null model of complete spatial randomness. As alternatives to the null hypothesis, we consider Matérn cluster process as an example for an attractive point pattern. Then, we take the Strauss processes and the Ginibre point process as examples for repulsive point patterns. Finally, we take the Baddeley-Silverman process as a point process with complex higher-order interactions.
More precisely, the Matérn cluster process $\mathsf{MatC}(2, 0.5, 1)$ features a Poisson parent process with intensity 2 and generates a $\mathsf{Poi}(1)$ number of offspring uniformly in a disk of radius $0.1$ around each parent. The Strauss process $\mathsf{Str}(4.0, 0.6, 0.5)$ has interaction parameter $0.6$ and interaction radius $0.5$, respectively.

```R
ginibre <- function(W = win,
                    n = 300,
                    offset = 17.75,
                    bbox = 1000
                   ){
    #' Ginibre point process in a window
    #'
    #' @param W sampling window
    #' @param n number of points in larger window
    #' @param offset offset for plus sampling
    #' @param bbox bounding box size

    #find eigenvalues of random matrix
    ginibs <- matrix(rnorm(n * n) + rnorm(n * n) * 1i,
                      ncol = n)
    evs <- eigen(ginibs)$values

    #shift and scale points to window size
    evs_shift <- evs + (offset + offset * 1i)
    evs_scaled <- evs_shift * (W$xrange)[2] / (2. * offset)

    #restrict to sampling window
    X <- ppp(x = Re(evs_scaled),
             y = Im(evs_scaled),
             window = owin(c(-bbox, bbox),
                           c(-bbox, bbox)))
    X[W]
}
```

```R
seed <- 45
set.seed(seed)
nsim <- 1e3

time <- Sys.time()

ppps_val <- rpoispp(2,
                    win = win,
                    nsim = nsim)

mcps <- rMatClust(2, .5, 1,
                  win = win,
                  nsim = nsim)

strs <- rStrauss(4, .6, .5,
                 W = win,
                 nsim = nsim)

basi <- rcell(win,
              dx = .67,
              nsim = nsim)

ginibs <- mclapply(1:nsim,
                   function(i) ginibre(win),
                   mc.cores = 36)
Sys.time() - time
```

    Time difference of 3.621955 mins

```R
width = 20
height = 20
options(repr.plot.width = width,
        repr.plot.height = height)

real <- 5

samples <-list(ppps_val[[real]],
               mcps[[real]],
               strs[[real]],
               basi[[real]],
               ginibs[[real]])

plots <- lapply(samples,
                function(pp) qplot(x = pp$x,
                                   y = pp$y,
                                   size = I(2.5)) +
                    theme_void() +
                    theme(plot.margin = margin(10, 10, 10, 10)) +
                    theme(panel.border = element_rect(fill = NA))
                )
grid.arrange(plots[[1]],
             plots[[2]],
             plots[[3]],
             plots[[4]],
             clip = F,
             nrow = 4, ncol = 4)
```

Next, we compare the persistence diagrams.

```R
library(grid)

#0-features
plots0 <- lapply( samples,
                function(pp){
                    pd <- alphaComplexDiag(cbind(pp$x, pp$y))$diag
                            ggplot(data.frame(pd[pd[,'dimension'] == 0,
                                                 2:3][-1,]),
                            aes(x = Birth, y = Death)) +
                            geom_violin() +
                            geom_point(size = 1) +
                            scale_y_continuous(limits = c(0, 0.5)) +
                            theme_bw()
                    })

#1-features
plots1 <- lapply(samples,
                function(pp){
                    pd <- alphaComplexDiag(cbind(pp$x, pp$y))$diag
                            ggplot(data.frame(pd[pd[,'dimension'] == 1, 2:3]),
                            aes(x = Birth, y = Death)) +
                            geom_point(size = 1) +
                            scale_x_continuous(limits = c(0, .8)) +
                            scale_y_continuous(limits = c(0, 1.3)) +
                            theme_bw()
                    })

options(warn=-1)
grid.arrange(grobs = list(plots0[[1]], plots0[[2]], plots0[[3]],plots0[[4]],
                           plots1[[1]] ,plots1[[2]],plots1[[3]],plots1[[4]]),
            ncol = 4, nrow = 4)
options(warn=0)
```

Finally, we illustrate the corresponding L-functions.

```R
Lests <- lapply(samples, Lest)
plots <- lapply(Lests,
                function(l) qplot(x = l$r,
                                  y = l$iso,
                                  xlim = c(0, .9),
                                  ylim = c(0, 1.0),
                                  ylab = "L(r)",
                                  xlab = "r",
                                  geom = 'line'
                                 ) +
                geom_line() +
                geom_abline(intercept = 0,
                            slope = 1,
                            linetype = "dashed"
                           ) +
                theme_bw())

options(warn=-1)
grid.arrange(plots[[1]],
             plots[[2]],
             plots[[3]],
             plots[[4]],
             nrow = 4, ncol = 4)
options(warn=0)
```

## Simulations under Complete Spatial Randomness

First, we discuss the test statistics with complete spatial randomness as null hypothesis.

### Means and Variances of Test Statistics under the Null Model

To determine the means and variances of the test statistics under the null model, we compute the death times and accumulated life times for 10,000 independent samples of a Poisson point process.

```R
seed <- 42
set.seed(seed)
nsim <- 1e4

ppps <- rpoispp(2,
                win = win,
                nsim = nsim)
```

Now, we collect the TDA-based statistics and Ripley's $L$-function into one list. Additionally, we include a linear combination of cluster-a

```R
funs_sum <- list(function(X) mean((tda0(X)$trans[1:100])),
                 function(X) mean(tda1(X)$trans),
                 function(X) mean((tda0(X)$trans[1:100])) - 10 * mean(tda1(X)$trans),
                 function(X) mean(Lstat(X)$trans))
```

```R
null_stats <- lapply(funs_sum,
                    function(f) as.numeric(mclapply(ppps,
                                         function(X) f(X),
                                                    mc.cores = 36))
                    )
null_means <- lapply(null_stats, mean)
null_sds <- lapply(null_stats, sd)
```

```R
null_means
null_sds
```

0.619130699156496

0.02757224280391

0.343408271117396

0.247929906570714

0.0473230093006575

0.00354181995445526

0.056886975096815

0.0145117176176012

Now, we compute the means and variances for the three statistics. For the loops and for Ripley's L-function, we can choose the entire relevant range of radii, so that $t_{\mathsf L} = t_{\mathsf{Ripley}} = 0.5$. For the clusters, it would be unreasonable, as for large values of the radius, we simply obtain the number of points in the observation window, which is not discriminative. Hence, we choose a smaller bound $t_{\mathsf C} = 0.1$.

### Test errors of Type I & II

Since the test statistics is asymptotically normal, knowledge of means and variances under the null model allows us to assess the test errors of type I and II.

```R
p_val <- function(X,
                  stat,
                  null_mean,
                  null_sd){
    #' p-value for deviation test
    #'
    #' @param pp point pattern to be tested
    #' @param stat test statistics
    #' @param null_mean mean under the null model
    #' @param null_sd standard deviation under the null model
    z_val <- abs(stat(X) - null_mean) / null_sd
    2 * (1 - pnorm(z_val))
}
```

To analyze the type I and II errors, we draw 1,000 realizations from the null model and from the alternatives, respectively. Under the null model the rejection rates are close to the nominal 5\%-level, thereby illustrating that already for moderately large point patterns the approximation by the Gaussian limit is accurate.

Using the mean and variance from the null model, we now compute the test powers for the alternatives. Using the $L$-function based statistics $T_{\mathsf{Ripley}}$, in the Mat\'ern-cluster and Strauss process alternatives the null hypothesis is rejected in $92.6\%$ and $86.3\%$ of the cases. Using the TDA-based statistics $T_{\mathsf C}$ and $T_{\mathsf L}$ alone yields smaller rejection rates. However, the linear combination $T_{\mathsf C} - 10 T_{\mathsf L}$ leads to comparable rejection rates as the $L$-function.

Finally, the Baddeley-Silverman process showcases that for patterns with complex interactions, the $L$-function statistics may exhibit low power. In this setting, all three TDA-based statistics outperform the $L$-function
statistics.

```R
alpha <- .05
pps_list <- list(ppps_val, mcps, strs, basi, ginibs)

do.call(rbind.data.frame,
        lapply(seq(length(funs_sum)),
               function(i) sapply(pps_list[1:4],
                                  function(pp_list) sum(
                                      mclapply(pp_list,
                                               function(pp) p_val(pp,
                                                                  funs_sum[[i]],
                                                                  null_means[[i]],
                                                                  null_sds[[i]]),
                                               mc.cores = 36
                                              ) < alpha)
                                  )
               )
       )
```

A data.frame: 4 × 4

c.55L..45L..47L..56L.	c.542L..614L..847L..910L.	c.556L..541L..905L..869L.	c.672L..836L..887L..603L.

<int>	<int>	<int>	<int>

55	542	556	672

45	614	541	836

47	847	905	887

56	910	869	603

## Simulations under a Cluster Process

So far, we have discussed how to use the cluster- and loop-based tests for testing the null hypothesis of complete spatial randomness. In the present section, we illustrate at the hand of a Matérn cluster process that it is possible to use for testing clustering or regularity of the point patterns. In other words, the null hypothesis is now a Mat\'ern cluster process

Similarly to the Poisson null model, in practice the intensity of the Matérn cluster process is not known but must be estimated from data. However, for the Matérn process the issue is more severe, since it is described by three parameter that all need to be estimated. For this purpose, we resort to the [minimum contrast method](https://www.crcpress.com/Statistical-Inference-and-Simulation-for-Spatial-Point-Processes/Moller-Waagepetersen/p/book/9781584882657).

```R
seed <- 42
set.seed(seed)
nsim <- 1e3
nsim_nest <-1e2

ppps <- rpoispp(2,
                win = win,
                nsim = nsim)

mcps <- rMatClust(2, .5, 1,
                  win = win,
                  nsim = nsim)
```

  In addition, now that the null model does not depend any longer on a single parameter, we cannot apply a simple rescaling to arrive at an intensity-adapted version of the estimators. Therefore, we need to resort to nested Monte Carlo simulations. That is, once we estimated the parameters of the Mat\'ern process, we determine the mean and variance of the test statistics based on $n_{\mathsf{nest}} = 100$ independent samples. Due to the complexity of the parameter estimation, we should expect substantial effects on the empirical results.

In the simulation study of 1,000 simulation runs, we use complete spatial randomness as alternative to the null hypothesis of clustering. First, all tests are conservative as the rejection rates under the null hypothesis are far below the nominal 5\%-level. This is particularly pronounced for $T_{\mathsf C}$ and $T_{\mathsf{Ripley}}$ where the null-hypothesis is never rejected. Regarding the type II error, although the linearly combined TDA-based test statistic $T_{\mathsf C} - 10 T_{\mathsf L}$ is outperformed by $T_{\mathsf{Ripley}}$, it still exhibits a reasonable rejection rate of $71.2\%$.

```R
pps_list <- list(mcps, ppps)

set.seed(seed)

time <- Sys.time()

p_vals <- lapply(1:nsim,
       function(i){
    #estimate parameters
    pars <- as.numeric((matclust.estpcf(mcps[[i]])$modelpar))

    #simulate null model
    mcps_nest <-rMatClust(pars[1], pars[2], pars[3],
                          win = win,
                          nsim = nsim_nest)

    #compute mean and variance under null
    null_stats <- lapply(funs_sum,
                        function(f) as.numeric(mclapply(mcps_nest,
                                                      f,
                                                      mc.cores = 36
                                                     ))
                        )
    null_means <- lapply(null_stats, mean)
    null_sds <- lapply(null_stats, sd)

    #compute p-value
     do.call(cbind,
             lapply(pps_list,
                  function(pp_list)
                      sapply(1:4,
                             function(j)
                                 p_val(pp_list[[i]],
                                       funs_sum[[j]],
                                       null_means[[j]],
                                       null_sds[[j]])
                             )
                    )
              )
           }
       )
Sys.time() - time
```

    Warning message:
    “Nelder-Mead simplex was degenerate”
    Warning message:
    “Nelder-Mead simplex was degenerate”

    Time difference of 24.75055 mins

```R
acc <- 0
alpha <- .05
for(i in 1:nsim)
    acc <- acc + (p_vals[[i]] < alpha)
acc
```

A matrix: 4 × 2 of type dbl

 0	391

26	511

28	712

 0	827

### Visualization

Finally, we visualize the average value of the statistics for the considered point processes.

The mean curves for the death times of clusters match up with the intuition about clustering and repulsion. For large radii, they all approach a common value, namely the expected number of points in the observation window. However, for the Matérn model, the slope is far steeper for small distances, caused by the merging of components of points within clusters. In contrast, for the Strauss process the slope is at first much smaller than in the Poisson model, thereby reflecting the repulsive nature of the Gibbs potential.

It is convenient to organize both the functional summary statistics in data frames.

```R
tda_df <- function(pps, stat){
    #' return statistic in data frame
    #'
    #' @param pps point processes for statistics are computed
    #' @param stat summary statistic
    tda_list <- mclapply(pps,
                function(X) stat(X)$trans,
                mc.cores = 36
               )
    do.call('rbind', tda_list)
}
```

```R
compare_df <- data.frame(
  t = t_vals0,
  ppp = colMeans(tda_df(ppps_val, tda0)),
  mcp = colMeans(tda_df(mcps, tda0)),
  str = colMeans(tda_df(strs, tda0)),
  bas = colMeans(tda_df(basi, tda0))
)

pl_1 <- ggplot(compare_df, aes(t)) +
  geom_line(aes(y = ppp, color = "a")) +
  geom_line(aes(y = mcp, color = "b")) +
  geom_line(aes(y = str, color = "c")) +
  geom_line(aes(y = bas, color = "d")) +
  scale_color_discrete(name = "Model",
                       labels = c("Poisson",
                                  "Matérn",
                                  "Strauss",
                                  "Badd.-Silv."
                                 )) +
  xlab('t') +
  ylab('number of deaths') +
  theme_bw() +
  theme(text = element_text(size=font_size))
```

When comparing the shapes for the loop-based features, a radically different picture emerges. Here, the APF for the repulsive Strauss process lies above the accumulated life times of the null model. Although the average lifetime of the features in the Strauss process is smaller than in the Poisson model, there are substantially more births. Taken together, these competing effects still lead to a net increase of the accumulated life times in the Strauss model compared to the Poisson setting.

```R
font_size <- 25

compare_df <- data.frame(
    t = t_vals1,
    ppp = colMeans(tda_df(ppps_val, tda1)),
    mcp = colMeans(tda_df(mcps, tda1)),
    str = colMeans(tda_df(strs, tda1)),
    bas = colMeans(tda_df(basi, tda1))
)
options(warn=-1)
pl_2 <- ggplot(compare_df, aes(t)) +
    geom_line(aes(y = ppp, color = "a")) +
    geom_line(aes(y = mcp, color = "b")) +
    geom_line(aes(y = str, color = "c")) +
    geom_line(aes(y = bas, color = "d")) +
    scale_color_discrete(name = "Model",
                         labels = c("Poisson",
                                    "Matérn",
                                    "Strauss",
                                    "Bad.-Silv."
                                   )) +
    xlim(c(0, .4)) +
    ylim(c(0,.065)) +
    xlab('t') +
    ylab('accumulated life times') +
    theme_bw() +
    theme(text = element_text(size=font_size))
grid.arrange(pl_1, pl_2,
             nrow = 4, ncol = 2)
options(warn=0)
```

## Envelope Tests for Complete Spatial Randomness

Leveraging the functional CLT allowed us to deduce that the deviation statistics considered above is asymptotically normal. Hence, we could easily extract an estimate for the asymptotic variance under the null model via simulation.  A caveat of the above analysis is that the clustering and repulsion effects were very strong.

Recently, [global rank-envelope tests](https://rss.onlinelibrary.wiley.com/doi/full/10.1111/rssb.12172) are becoming widely adopted, as they are both powerful and provide additional information as to why a null hypothesis is rejected. The global rank envelope tests are fundamentally Monte Carlo-based tests and therefore do not relate directly to the functional CLT. However, they also rely on a functional summary statistics as input. In most of the application cases in spatial statistics, one of the distance-based second-order functionals such as Ripley's K-function or L-function are used. In this section, we analyze the effect of replacing them by TDA-based statistics.

Since envelope tests are typically able to detect very subtle changes from the null model, we consider now a parameter set-up, where both the Matérn cluster as well as the Strauss process are substantially more similar to the Poisson point process.

As a first step, we generate $s = 999$ realizations of the null model.

```R
seed <- 43
set.seed(seed)

s <- 1e3 - 1

ppps_env <- rpoispp(2,
                    win = win,
                    nsim = s)
```

First, we extract the p-value from a functional test statistic.

```R
p_val_env <- function(X,
                      stat){
    #' p-value for deviation test
    #'
    #' @param X point pattern to be tested
    #' @param stat functional test statistic
    #' @param null_mean mean under the null model
    #' @param null_sd standard deviation under the null model

    env <- envelope(X,
                fun = stat,
                savefuns = TRUE,
                verbose = FALSE,
                correction = "translate",
                nsim = length(ppps_env),
                simulate = ppps_env)

    #p-value of rank-envelope
    test <- global_envelope_test(env, type="erl")
    attr(test, 'p')
}
```

We also introduce the linearly combined TDA-estimator.

```R
tda01 <- function(X,
                  r = 0,
                  correction = "translate"
                 ){
    #' Linearly combined TDA-estimator
    #'
    #' @param X point pattern
    #' @param r time point
    #' @param correction edge correction

    t0 <- tda0(X)
    t1 <- tda1(X)
    res <- t0$trans - 10 * t1$trans

    fv(data.frame(r = t0$r,
                  trans = res),
       valu = 'trans',
       labl = c('r', 'lab'),
       fname='Diag')
}
```

```R
funs_sum <- lapply(list(tda0, tda1, tda01, Lest),
                   function(f)
                       function(X) p_val_env(X, f)
                       )
```

Now, we extract the p-values from the global envelope test.

```R
time <- Sys.time()
do.call(rbind.data.frame,
        lapply(funs_sum,
               function(f) sapply(pps_list[1:4],
                                  function(pp_list) sum(
                                      mclapply(pp_list,
                                               f,
                                               mc.cores = 36
                                              ) < alpha)
                                  )
))
Sys.time() - time
```

A data.frame: 4 × 4

c.60L..33L..54L..47L.	c.690L..471L..750L..928L.	c.462L..247L..821L..976L.	c.655L..606L..928L..585L.

<int>	<int>	<int>	<int>

60	690	462	655

33	471	247	606

54	750	821	928

47	928	976	585

    Time difference of 1.586437 hours


```python
# HIDDEN
from datascience import *
from prob140 import *
import numpy as np
import matplotlib.pyplot as plt
plt.style.use('fivethirtyeight')
%matplotlib inline
import math
from scipy import stats
from sympy import *
init_printing()
```

### Beta Densities with Integer Parameters ###
In the previous section we learned how to work with joint densities, but many of the joint density functions seemed to appear out of nowhere. For example, we checked that the function

$$
f(x, y) = 120x(y-x)(1-y), ~~~~ 0 < x < y < 1
$$

is a joint density, but there was no clue where it came from. In this section we will find its origin and go on to develop an important family of densities on the unit interval.

### Order Statistics of IID Uniform $(0, 1)$ Variables ###
Let $U_1, U_2, \ldots, U_n$ be i.i.d. uniform on $(0, 1)$. Imagine each $U_i$ as the position of a dart thrown at the unit interval. The graph below shows the positions of five such darts, each shown as a star.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
y = 1 - np.ones(5)
x = stats.uniform.rvs(size=5)
order_stats = np.sort(x)
plt.scatter(x, y, marker='*', color='r', s=100)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('Five IID Uniform (0, 1) Variables');
```

Based on the graph above, can you tell which star corresponds to $U_1$? You can't, because $U_1$ could be any of the five stars. So also you can't identify any of the five variables $U_1, U_2, U_3, U_4, U_5$.

What you *can* see, however, is the list of $U_i$'s *sorted in increasing order*. You can see the value of the minimum, the second on the sorted list, the third, the fourth, and finally the fifth which is the maximum.

These are called the *order statistics* of $U_1, U_2, U_3, U_4, U_5$, and are denoted $U_{(1)}, U_{(2)}, U_{(3)}, U_{(4)}, U_{(5)}$.

Remember that because the $U_i$'s are independent random variables with densities, there can't be ties: the chance that two of them are equal is 0.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
order_stats = np.sort(x)
plt.scatter(x, y, marker='*', color='r', s=100)
u_labels = make_array('$U_{(1)}$', '$U_{(2)}$', '$U_{(3)}$', '$U_{(4)}$', '$U_{(5)}$')
for i in range(5):
    plt.text(order_stats[i], -0.0007, u_labels[i], size=16)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('Order Statistics of the Five IID Uniform (0, 1) Variables');
```

In general for $1 \le k \le n$, the *$k$th order statistic* of $U_1, U_2, \ldots, U_n$ is the $k$th value when the $U_i$'s are sorted in increasing order. This can also be thought of as the $k$th *ranked* value when the minimum has rank 1. It is denoted $U_{(k)}$.

### Joint Density of Two Order Statistics ###
Let $n = 5$ as above and let's try to work out the joint density of $U_{(2)}$ and $U_{(4)}$. That's the joint density of the second and fourth values on the sorted list.

The graph below shows the event $\{U_{(2)} \in dx, U_{(4)} \in dy\}$ for values $x$ and $y$ such that $0 < x < y < 1$.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
y = 1 - np.ones(5)
x = make_array(0.1, 0.3, 0.45, 0.7, 0.9)
plt.scatter(x, y, marker='*', color='r', s=100)
plt.plot([0.28, 0.32], [0, 0], color='gold', lw=2)
plt.text(0.28, -0.0007, r'$dx$', size=16)
plt.plot([0.68, 0.72], [0, 0], color='gold', lw=2)
plt.text(0.68, -0.0007, r'$dy$', size=16)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('$n = 5$; $\{ U_{(2)} \in dx, U_{(4)} \in dy \}$');
```

To find $P(U_{(2)} \in dx, U_{(4)} \in dy)$, notice that:
- One of $U_1, U_2, U_3, U_4, U_5$ must be in $dx$; there are 5 ways to choose this one.
- Of the remaining 4 variables, one of them must be in $dy$; there are 4 ways to choose this one.
- Of the remaining 3, one must be in $(0, x)$.
- Of the remaining 2, one must be in $(x, y)$.
- The last one must be in $(y, 1)$.

Thus
$$
P(U_{(2)} \in dx, U_{(4)} \in dy) ~ \sim ~ 5(1dx) \cdot 4(1dy) \cdot 3x \cdot 2(y-x) \cdot 1(1-y)
~ = ~ 120x(y-x)(1-y)dxdy
$$

and therefore the joint density of $U_{(2)}$ and $U_{(4)}$ is given by

$$
f(x, y) = 120x(y-x)(1-y), ~~~ 0 < x < y < 1
$$

This solves the mystery of how the formula arises.

But it also does much more. The *marginal* densities of the order statistics of i.i.d. uniform $(0, 1)$ variables form a family that is important in data science.

### The Density of $U_{(k)}$ ###
Let $U_{(k)}$ be the $k$th order statistic of $U_1, U_2, \ldots, U_n$. We will find the density of $U_{(k)}$ by following the same general process that we followed to find the joint density above.

The graph below displays the event $\{ U_{(k)} \in dx \}$. For the event to occur,
- One of the variables $U_1, U_2, \ldots, U_n$ has to be in $dx$.
- Of the remaining $n-1$ variables, $k-1$ must have values in $(0, x)$ and the rest in $(x, 1)$.

```python
# HIDDEN

plt.plot([0, 1], [0, 0], color='k', lw=2)
plt.scatter(0.4, 0, marker='*', color='r', s=100)
plt.plot([0.38, 0.42], [0, 0], color='gold', lw=2)
plt.text(0.38, -0.0007, r'$dx$', size=16)
plt.text(0.1, 0.001, '$k-1$ stars', size=16)
plt.text(0.1, 0.0005, 'in $(0, x)$', size=16)
plt.text(0.6, 0.001, '$n-k$ stars', size=16)
plt.text(0.6, 0.0005, 'in $(x, 1)$', size=16)
plt.text(0, -0.0007, r'0', size=16)
plt.text(0.98, -0.0007, r'1', size=16)
plt.xlim(0, 1)
plt.yticks([])
plt.xticks([])
plt.title('$\{ U_{(k)} \in dx \}$');
```

There are $n$ ways to choose the variable that lands in $dx$. Once that is chosen, $k-1$ of the remaining $n-1$ variables must have values in $(0, x)$. Thus

$$
P(U_{(k)} \in dx) \sim n \cdot 1 dx \cdot \binom{n-1}{k-1}x^{k-1}(1-x)^{n-k}
$$

by the binomial formula.

Therefore the density of $U_{(k)}$ is given by

$$
f_{U_{(k)}} (x) = \frac{n!}{(k-1)!(n-k)!} x^{k-1}(1-x)^{n-k}, ~~~ 0 < x < 1
$$

Let's rewrite the exponents slightly:
$$
f_{U_{(k)}} (x) = \frac{n!}{(k-1)!((n-k+1)-1)!} x^{k-1}(1-x)^{(n-k+1)-1}, ~~~ 0 < x < 1
$$

Because $1 \le k \le n$, we know that $n-k+1$ is a positive integer. Since $n$ is an arbitrary positive integer, so is $n-k+1$.

### Beta Densities ###
Thus we have shown that if $r$ and $s$ are any two positive integers, then the function

$$
f(x) ~ = ~ \frac{(r+s-1)!}{(r-1)!(s-1)!} x^{r-1}(1-x)^{s-1}, ~~~ 0 < x < 1
$$

is a probability density function. This is called the *beta density with parameters $r$ and $s$*.

Thus $U_{(k)}$ has the beta density with parameters $k$ and $n-k+1$.

The shape of the density is determined by the factors that involve $x$ and $1-x$. All the factorials are just parts of the constant that make the density integrate to 1.

Notice that the uniform $(0, 1)$ density is the same as the beta density with parameters $r = 1$ and $s = 1$. The uniform $(0, 1)$ density is a member of the *beta family*.

The graph below shows some beta density curves. As you would expect, the beta $(3, 3)$ density is symmetric about 0.5.

```python
x = np.arange(0, 1.01, 0.01)
for i in np.arange(1, 7, 1):
    plt.plot(x, stats.beta.pdf(x, i, 6-i), lw=2)
plt.title('Beta $(i, 6-i)$ densities for $1 \leq i \leq 5$');
```

By choosing the parameters appropriately, you can create beta densities that put much of their mass near a prescribed value. That is one of the reasons beta densities are used to model *random proportions*. For example, if you think that the probability that an email is spam is most likely in the 60% to 90% range, but might be lower, you might model your belief by choosing the density that peaks at around 0.75 in the graph above.

The calculation below shows you how to get started on the process of picking parameters so that the beta density with those parameters has properties that reflect your beliefs.

### The Beta Integral ###
The beta density integrates to 1, and hence for all positive integers $r$ and $s$ we have

$$
\int_0^1 x^{r-1}(1-x)^{s-1}dx ~ = ~ \frac{(r-1)!(s-1)!}{(r+s-1)!}
$$

Thus probability theory makes short work of an otherwise laborious integral. Also, we can now find the expectation of a random variable with a beta density.

Let $X$ have the beta $(r, s)$ density for two positive integer parameters $r$ and $s$. Then

\begin{align*}
E(X) &= \int_0^1 x \frac{(r+s-1)!}{(r-1)!(s-1)!} x^{r-1}(1-x)^{s-1}dx \\
&= \frac{(r+s-1)!}{(r-1)!(s-1)!} \int_0^1 x^r(1-x)^{s-1}dx \\
&= \frac{(r+s-1)!}{(r-1)!(s-1)!} \cdot \frac{r!(s-1)!}{(r+s)!} ~~~~ \text{(beta integral for parameters } r+1 \text{and } s\text{)}\\
&= \frac{r}{r+s}
\end{align*}

You can follow the same method to find $E(X^2)$ and hence $Var(X)$.

The formula for the expectation allows you to pick parameters corresponding to your belief about the random proportion being modeled by $X$. For example, if you think the proportion is likely to be somewhere around 0.4, you might start by trying out a beta prior with $r = 2$ and $s = 3$.

You will have noticed that the form of the beta density looks rather like the binomial formula. Indeed, we used the binomial formula to derive the beta density. Later in the course you will see another close relation between the beta and the binomial. These properties make the beta family one of the most widely used families of densities in machine learning.

```python

```


Solution to: [Day 5: Normal Distribution I](https://www.hackerrank.com/challenges/s10-normal-distribution-1/tutorial)

Table of Contents

- Table of Contents
- Notes
    - Normal Distribution
    - Standard Normal Distribution
    - Cumulative Probability
- Solution
    - Imports
    - Constants
    - Input
    - Cumulative Probability function
    - Main

```javascript
%%javascript
$.getScript('https://kmahelona.github.io/ipython_notebook_goodies/ipython_notebook_toc.js')
```

    <IPython.core.display.Javascript object>

# Notes

## Normal Distribution
The probability density of normal distribution is:
\begin{equation}
\large
\mathcal{N} = \frac
{1}
{\sigma * 2 \pi}
e^
{\frac
{(x - \mu)^{2}}
{2\sigma^{2}}
}
\end{equation}

Here,
- μ is the mean (or expectation) of the distribution.
    - It can also be the median & mode.
- $\sigma^{2}$ is the variance.
- $\sigma$ is the standard deviation.

## Standard Normal Distribution
If μ = 0 and σ = 1, then the normal distribution is known as standard normal distribution:

\begin{equation}
\large
\phi(x) =
\frac
{e^{-\frac{x^{2}}{2}}}
{\pi^{\frac{1}{2}}}
\end{equation}

Every normal distribution can be represented as standard normal distribution:
\begin{equation}
\large
\mathcal{N} (\mu\sigma^{2}) =
\frac{1}{\sigma}
\Phi
\frac
{x - \mu}
{\sigma}
\end{equation}

## Cumulative Probability
Consider a real-valued random variable, X. The **cumulative distribution function** of X (or just the distribution function of X) evaluated at x is the probability that X will take a value less than or equal to x:

\begin{equation}
\large
F_{x}(x) = P(X \leq x)
\end{equation}

Also,
\begin{equation}
\large
P(A \leq X \leq B) = P(A < B) = F_{x}(B) - F_{x}(A)
\end{equation}

The **cumulative distribution function** for a function with normal distribution is:

\begin{equation}
\large
\Phi(x) =
\frac{1}{2}(1 + erf(
\frac
{x - mu}
{\sigma * 2^{\frac{1}{2}}}
))
\end{equation}

Where erf is the **error function**:

\begin{equation}
\large
(z) = \frac
{2}{\pi^{\frac{1}{2}}}
\sum_{0}^{z} e^{-x^{2}} * dx
\end{equation}

# Solution

## Imports

```python
from typing import Tuple
```

## Constants

```python
import math
```

## Input

```python
def get_input() -> Tuple[int, int, float, ]:
	"""Returns input for Day 5: Normal Distribution I

	Returns:
		Tuple[int, int, float, int]: Represents: mean, sd, less than value, and interval
	"""
	return (20, 2, 19.5, 20, 22)

	mean, sd = (int(x) for x in input().split())
	less_than = float(input())
	interval_1, interval_2 = (int(x) for x in input().split())
	return mean, sd, less_than, interval_1, interval_2
```

## Cumulative Probability function

```python
def custom_erf(z: int) -> float:
	"""Returns error from error function for normal distribution

	Args:
		z (int): error function value

	Returns:
		float: error amount

	NOTE:
		- math module has an inbuilt erf function
		- dt represents an infinitely small value from a Maclaurin series
		- formula taken from: https://www.johndcook.com/blog/python_erf/
	"""
	## Vars
	constants = (
		1.061405429,
		-1.453152027,
		1.421413741,
		-0.284496736,
		0.254829592,
	)
	p = 0.3275911
	sign = 1 if z >= 0 else -1
	z = abs(z)
	t = 1 / ( 1 + p * z)

	## Calc
	y = 0
	for c in constants: y = (y + c) * t
	y = 1 - (y * math.e ** (-z * z))

	return y * sign
```

```python
def calc_cum_distr(x: int, mean: float,	sd: float) -> float:
	"""returns cumulative probability for val in normal distribution.

	Args:
		x (int): value to find cum probability for

	Returns:
		float: cumulative probability of x
	"""
	z = (x - mean) / (sd * 2 ** (1/2))
	return 1/2 * (1 + custom_erf(z))
```

```python
def format_scale(num: float) -> float:
	"""Formats number to scale

	Args:
		num (float): number

	Returns:
		float: formatted number
	"""
	return f'{num :.3f}'
```

## Main

```python
def main():
	"""Runs script
	"""
	## Input
	mean, sd, less_than, interval_1, interval_2 = get_input()

	## Cum prob for less than
	cum_prob_less_than = calc_cum_distr(less_than, mean, sd)
	print( format_scale(cum_prob_less_than))

	## Interval cum prob
	int1_cum_prob = calc_cum_distr(interval_1, mean, sd)
	int2_cum_prob = calc_cum_distr(interval_2, mean, sd)
	interval_cum_prob = int2_cum_prob - int1_cum_prob
	print( format_scale(interval_cum_prob))
```

```python
if __name__ == "__main__":
	main()
```

    0.401
    0.341


```python
import numpy as np
import matplotlib.pyplot as plt
np.set_printoptions(linewidth=150)
```

```python
class rb():
    position = 10
    def __init__(self, position):
        self.position = position
    def apply(self, systemmatrix, rhs, delta_x):
        pass
class neumann(rb):
    waermestrom = 5
    def __init__(self, position, waermestrom):
        super(neumann,self).__init__(position)
        self.waermestrom = waermestrom
    def apply(self, systemmatrix, rhs, delta_x):
        if (self.position == 0):
            # Vorwärtsdifferenz
            systemmatrix[self.position, 0:2] = (1, -1)
        elif (self.position < len(rhs)):
            # Rückwärtsdifferenz
            systemmatrix[self.position, self.position-1:self.position+1] = (-1, 1)
        else:
            raise ValueError("Out of bounds")
        rhs[self.position] = rhs[self.position] + self.waermestrom * delta_x
class dirichlet(rb):
    temperatur = 5
    def __init__(self, position, temperatur):
        super(dirichlet,self).__init__(position)
        self.temperatur = temperatur
    def apply(self, systemmatrix, rhs, delta_x):
        if (self.position > len(rhs)):
            raise ValueError("Out of bounds")
        systemmatrix[self.position, self.position] = 1
        rhs[self.position] = rhs[self.position] + self.temperatur
```

```python
def assemble(knoten, l, rben):
    delta_x = l / (knoten-1)
    systemmatrix = np.zeros((knoten, knoten))
    rhs = np.zeros((knoten))
    for i in range(1, knoten-1):
        systemmatrix[i, i-1:i+2] = (-1, 2, -1)
    for rb in rben:
        rb.apply(systemmatrix, rhs, delta_x)
    return systemmatrix, rhs
def solve(knoten, l, rben):
    systemmatrix, rhs = assemble(knoten, l, rben)
    solution = np.linalg.solve(systemmatrix, rhs)
    return np.linspace(0, l, knoten), solution, systemmatrix, rhs
```

### Aufgabe 2.3.1
_2x Neumann_

```python
knoten = 10
rb_vorne = dirichlet(0, 50)
rb_hinten = dirichlet(knoten-1, 200)
l = 10
x, T1, s, r = solve(knoten, l, [rb_vorne, rb_hinten])
plt.plot(x, T1)
plt.title("vorne und hinten Neumann")
plt.xlabel("Weg [m]")
plt.ylabel("Temperatur [K]")
plt.show()
```

_Neumann und Dirichlet_

```python
knoten = 10
rb_vorne = dirichlet(0, 50)
rb_hinten = neumann(knoten-1, 50)
l = 10
x, T2, s, r = solve(knoten, l, [rb_vorne, rb_hinten])
plt.plot(x, T2)
plt.title("vorne Neumann, hinten Dirichlet")
plt.xlabel("Weg [m]")
plt.ylabel("Temperatur [K]")
plt.show()
```

### Aufgabe 2.3.2
Analytisch: es gibt keine "absolute" Randbedingung, die Lösung kann beliebig nach oben und unten verschoben werden.
Numerisch: Die Matrix hat nicht mehr vollen Rang, ein Eigenwert verschwindet.

```python
knoten = 3
rb_vorne = neumann(0, 50)
rb_hinten = neumann(knoten-1, 50)
l = 10
s, r = assemble(knoten, l, [rb_vorne, rb_hinten])
print(s)
print("Eigenwerte:")
print(np.linalg.eig(s)[0])
```

    [[ 1. -1.  0.]
     [-1.  2. -1.]
     [ 0. -1.  1.]]
    Eigenwerte:
    [ 3.00000000e+00  1.00000000e+00 -3.36770206e-17]

### Aufgabe 2.3.3
Analytische Lösungen

Für $0 = \kappa \dfrac{\partial^2 T}{\partial x^2}$ ist die allgemeine Lösung $T = c_1 \cdot x + c_2$.

Für zwei Neumann-RBen folgt:
$T_1 = T(x = 0) = c_2$, $T_2 = T(x=L) = c_1 \cdot L + c_2 = c_1 \cdot L + T_1 \rightarrow c_1 = \dfrac{T_2 - T_1}{L}$

und damit die analytische Lösung für die gegebenen Werte: $T(x) = 15 \frac{K}{m} \cdot x + 50 K$

Für Neumann vorne und Dirichlet hinten:
$T_1 = T(x = 0) = c_2$, $ \left.\frac{d T}{d x}\right|_L = c_1$

und damit für die gegebenen Werte: $T(x) = 50 \frac{K}{m} + 50 K$

```python
exakt_1 = [15 * xi + 50 for xi in x]
exakt_2 = [50 * xi + 50 for xi in x]
plt.plot(x, exakt_1, label="$T_1$ exakt")
plt.plot(x, exakt_2, label="$T_2$ exakt")
plt.plot(x, T1, ".", label="$T_1$ numerisch")
plt.plot(x, T2, ".", label="$T_2$ numerisch")
plt.legend()
```

_Netzstudie_

Für 6 und 11 Punkte liegt die numerische Lösung exakt auf der analytischen Lösung.

```python
result = {}
x = np.linspace(0,10,10)
plt.plot(x, exakt_1, label="$T_1$ exakt")
plt.plot(x, exakt_2, label="$T_2$ exakt")
for knoten in [6, 11]:
    rb_vorne = dirichlet(0, 50)
    rb_hinten2 = neumann(knoten-1, 50)
    rb_hinten1 = dirichlet(knoten-1, 200)
    x, T1, s, r = solve(knoten, 10, [rb_vorne, rb_hinten1])
    x, T2, s, r = solve(knoten, 10, [rb_vorne, rb_hinten2])
    plt.plot(x, T1, "." if knoten == 11 else "x", label="$T_1$, {} Knoten".format(knoten))
    plt.plot(x, T2, "." if knoten == 11 else "x", label="$T_2$, {} Knoten".format(knoten))
    result[knoten] = [T1, T2]
plt.legend()
plt.show()
```

### Aufgabe 2.3.4 / 2.3.5

Das ist erst interessant, sobald wir Wärmequellen vorgeben, da erst dann die Lösung nicht mehr linear ist.

Wir wählen folgende Funktion für die Wärmequellen (mit $s_0 = 200 \frac{K}{s}$):
$$s(x) = \begin{cases} 0 & x < \frac{L}{5} \\ s_0 & \frac{L}{5} \leq x < \frac{2L}{5} \\ 0 & \mathrm{sonst}\end{cases}$$
Das liefert die folgende analytische Lösung:
$$T(x) = - \frac 12 \frac{s(x)}{\kappa}x^2 + c_1 \cdot x + c_2 \rightarrow T(x) = \begin{cases} c_1 x + c_2 & x < \frac{L}{5} \\ -\frac{s_0}{2\kappa}x^2 + c_3 x + c_4 & \frac{L}{5} \leq x < \frac{2L}{5} \\ c_5 x + c_6 & \mathrm{sonst}\end{cases}$$
Als Randbedingungen wählen wir eine Dirichlet-RB bei $x=0$: $T = 300K = c_2$ und eine Neumann-RB bei $x=L$: adiabater Rand, d.h. $\left.\frac{dT}{dx}\right|_L = 0 \rightarrow c_5 = 0$. Außerdem ist der Temperaturverlauf stetig und sprungfrei.
Das liefert: \begin{equation}c_1 \cdot \frac{L}{5} + c_2 = -\frac{s_0}{2\kappa} \frac{L^2}{25} + c_3 \frac{L}{5} + c_4\end{equation}
$$c_1 = -\frac{s_0}{\kappa} \frac{L}{5} + c_3$$
$$-\frac{s_0}{2\kappa} \frac{4L^2}{25} + c_3 \frac{2L}{5} + c_4 = c_6$$
$$-\frac{s_0}{\kappa} \frac{2L}{5} + c_3 = 0 \rightarrow c_3 = \frac{s_0}{\kappa} \frac{2L}{5} = 80 \frac{K}{m}$$
Einsetzen in Gl. 2
$$c_1 = -\frac{s_0}{\kappa} \frac{L}{5} + \frac{s_0}{\kappa} \frac{2L}{5} = \frac{s_0}{\kappa} \frac{L}{5} = 40 \frac{K}{m}$$
Beides in Gl. 1:
$$\frac{s_0}{\kappa} \frac{L^2}{25} + c_2 = -\frac{s_0}{2\kappa} \frac{L^2}{25} + \frac{s_0}{\kappa} \frac{2L^2}{25} + c_4 \rightarrow c_2 - \frac{s_0}{\kappa} \frac{L^2}{50} = c_4 = 260 K$$

$$-\frac{s_0}{\kappa} \frac{4L^2}{50} + \frac{s_0}{\kappa} \frac{4L}{25} +c_2 - \frac{s_0}{\kappa} \frac{L^2}{50} = c_6 \rightarrow c_6 = c_2 + \frac{s_0}{\kappa}\frac{3L^2}{50} = 420 K$$

```python
x1 = np.linspace(0, 2, 100)
x2 = np.linspace(2,4,100)
x3 = np.linspace(4,10,100)
y1 = [40 * xv + 300 for xv in x1]
y2 = [-200/20 * xv**2 + 80 * xv + 260 for xv in x2]
y3 = [420 for xv in x3]
xan = np.concatenate([x1, x2, x3])
yan = np.concatenate([y1, y2, y3])
plt.plot(x1, y1, x2, y2, x3, y3)
plt.show()
```

```python
class waermequelle(rb):
    staerke = 5
    kappa = 10
    def __init__(self, position, staerke):
        super(waermequelle,self).__init__(position)
        self.staerke = staerke
    def apply(self, systemmatrix, rhs, delta_x):
        rhs[self.position] = rhs[self.position] + self.staerke * delta_x**2 / self.kappa
```

```python
result = {}
knoten = 1001
for knoten in [6, 11, 21, 41, 61, 101, 201, 501, 1001]:
    wq_start = int((knoten-1)/10*2)
    wq_ende = int((knoten-1)/10*4)
    rb_vorne = dirichlet(0, 300)
    rb_hinten = neumann(knoten-1, 0)
    wqs = [waermequelle(i, 200) for i in range(wq_start, wq_ende)]
    l = 10
    x, T, s, r = solve(knoten, l, [rb_vorne, rb_hinten] + wqs)
    if (knoten == 6 or knoten == 11 or knoten== 61 or knoten == 1001):
        plt.plot(x, T, "." if knoten != 1001 else "-", label="$T$, {} Knoten".format(knoten))
    result[knoten] = [x, T]

plt.title("Netzstudie für Wärmequelle zwischen x=2m und x=4m mit Intensität 300 K/s")
plt.xlabel("Weg [m]")
plt.ylabel("Temperatur [K]")

plt.plot(xan, yan, label="analytisch")
plt.legend()
plt.show()

```

```python
errordict = {}
for key, values in result.items():
    errors = []
    for value in zip(values[0], values[1]):
        position_in_exact = (np.abs(xan-value[0])).argmin()
        exact = yan[position_in_exact]
        numeric = value[1]
        error = exact - numeric
        errors += [error]
    errordict[key] = [values[0], errors]
```

```python
plt.figure(figsize=(12, 6))
plt.subplot(1, 2, 1)

meanerrors = {}
for key, values in errordict.items():
    meanerrors[key] = sum(np.abs(values[1]))/key
    plt.plot(values[0], values[1], ".", label="{} Knoten".format(key))
plt.title("Netzstudie für Wärmequelle - Fehler")
plt.xlabel("Weg [m]")
plt.ylabel("Temperaturdifferenz zu absoluter Lösung [K]")
plt.legend()
ax = plt.subplot(1, 2, 2)
plt.xlabel("Anzahl Knoten")
plt.ylabel("Mean Error")
plt.title("Mittlerer Fehler für verschiedene Knotenanzahlen")
plt.plot(meanerrors.keys(), list(meanerrors.values()))
plt.show()
```

Es ist gut erkennbar, dass dies ein Verfahren zweiter Ordnung ist.

```python

```


```python
from IPython.display import SVG
SVG(filename='n-pendulum-with-cart.svg')
```

```python
import sympy as sp
import sympy.physics.mechanics as mech
```

```python
n = 5
```

```python
q = mech.dynamicsymbols('q:{}'.format(n + 1))  # Generalized coordinates
qd = mech.dynamicsymbols('q:{}'.format(n + 1),1)  # Generalized coordinates
f = mech.dynamicsymbols('f')                   # Force applied to the cart

m = sp.symbols('m:{}'.format(n + 1))         # Mass of each bob
l = sp.symbols('l:{}'.format(n))             # Length of each link
g, t = sp.symbols('g t')                     # Gravity and time
```

```python
N = mech.ReferenceFrame('N')  # Inertial reference frame
pointN = mech.Point('N*')           # Origin point
pointN.set_vel(N, 0)             # Origin's velocity is zero
```

```python
points = []
particles = []
forces = []
```

```python
points = [pointN.locatenew('point_0', l[0]*(sp.sin(q[0])*N.x-sp.cos(q[0])*N.y))]
points[0].set_vel(N, points[0].pos_from(pointN).dt(N))
particles = [mech.Particle('particle_0',points[0],m[0])]
forces = [(points[0],-particles[0].mass*g*N.y)]

for i in range(1,n):
    points.append(points[i-1].locatenew('point_'+str(i), l[i]*(sp.sin(q[i])*N.x-sp.cos(q[i])*N.y)))
    points[i].set_vel(N, points[i].pos_from(pointN).dt(N))
    particles.append(mech.Particle('particle_'+str(i),pointi,m[i]))
    forces.append((points[i],-particles[i].mass*g*N.y))
```

```python
# Construct the Lagrangian
L = mech.Lagrangian(N, *particles)
L
```

$\displaystyle \frac{l_{0}^{2} m_{0} \sin^{2}{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{0}}{\left(t \right)}\right)^{2}}{2} + \frac{l_{0}^{2} m_{0} \cos^{2}{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{0}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{1} \left(l_{0} \sin{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \sin{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \sin{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{1} \left(l_{0} \cos{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{2} \left(l_{0} \sin{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \sin{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \sin{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{2} \left(l_{0} \cos{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{3} \left(l_{0} \sin{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \sin{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \sin{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{3} \left(l_{0} \cos{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{4} \left(l_{0} \sin{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \sin{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \sin{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2} + \frac{m_{4} \left(l_{0} \cos{\left(\operatorname{q_{0}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{0}}{\left(t \right)} + l_{1} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{1}}{\left(t \right)} + l_{2} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{2}}{\left(t \right)} + l_{3} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{3}}{\left(t \right)} + l_{4} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}}{2}$

```python
# Create the LagrangesMethod object
LM = mech.LagrangesMethod(L, points, hol_coneqs=None, forcelist=forces, frame=N)
```

```python
LM.form_lagranges_equations()
```

```python
# Form Lagranges Equations
ELeqns = LM.form_lagranges_equations()
```

```python
sp.trigsimp(LM.mass_matrix)
```

$\displaystyle \left[\begin{matrix}m_{0} + m_{1} + m_{2} + m_{3} + m_{4} + m_{5} & - l_{0} \left(m_{1} + m_{2} + m_{3} + m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & - l_{1} \left(m_{2} + m_{3} + m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & - l_{2} \left(m_{3} + m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{3}}{\left(t \right)} \right)} & - l_{3} \left(m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{4}}{\left(t \right)} \right)} & - l_{4} m_{5} \sin{\left(\operatorname{q_{5}}{\left(t \right)} \right)}\\- l_{0} \left(m_{1} + m_{2} + m_{3} + m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{1}}{\left(t \right)} \right)} & l_{0}^{2} m_{1} + l_{0}^{2} m_{2} + l_{0}^{2} m_{3} + l_{0}^{2} m_{4} + l_{0}^{2} m_{5} & l_{0} l_{1} \left(m_{2} + m_{3} + m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} & l_{0} l_{2} \left(m_{3} + m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} & l_{0} l_{3} \left(m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} & l_{0} l_{4} m_{5} \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)}\\- l_{1} \left(m_{2} + m_{3} + m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{2}}{\left(t \right)} \right)} & l_{0} l_{1} \left(m_{2} + m_{3} + m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} & l_{1}^{2} m_{2} + l_{1}^{2} m_{3} + l_{1}^{2} m_{4} + l_{1}^{2} m_{5} & l_{1} l_{2} \left(m_{3} + m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} & l_{1} l_{3} \left(m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} & l_{1} l_{4} m_{5} \cos{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)}\\- l_{2} \left(m_{3} + m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{3}}{\left(t \right)} \right)} & l_{0} l_{2} \left(m_{3} + m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} & l_{1} l_{2} \left(m_{3} + m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} & l_{2}^{2} m_{3} + l_{2}^{2} m_{4} + l_{2}^{2} m_{5} & l_{2} l_{3} \left(m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} & l_{2} l_{4} m_{5} \cos{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)}\\- l_{3} \left(m_{4} + m_{5}\right) \sin{\left(\operatorname{q_{4}}{\left(t \right)} \right)} & l_{0} l_{3} \left(m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} & l_{1} l_{3} \left(m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} & l_{2} l_{3} \left(m_{4} + m_{5}\right) \cos{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} & l_{3}^{2} m_{4} + l_{3}^{2} m_{5} & l_{3} l_{4} m_{5} \cos{\left(\operatorname{q_{4}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)}\\- l_{4} m_{5} \sin{\left(\operatorname{q_{5}}{\left(t \right)} \right)} & l_{0} l_{4} m_{5} \cos{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} & l_{1} l_{4} m_{5} \cos{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} & l_{2} l_{4} m_{5} \cos{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} & l_{3} l_{4} m_{5} \cos{\left(\operatorname{q_{4}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} & l_{4}^{2} m_{5}\end{matrix}\right]$

```python
mech.find_dynamicsymbols(LM.mass_matrix)
```

    {q1(t), q2(t), q3(t), q4(t), q5(t)}

```python
sp.trigsimp(LM.forcing)
```

$\displaystyle \left[\begin{matrix}l_{0} m_{1} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{2} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{3} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{4} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{5} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{1} m_{2} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{3} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{4} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{5} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{2} m_{3} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{2} m_{4} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{2} m_{5} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{3} m_{4} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} + l_{3} m_{5} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} + l_{4} m_{5} \cos{\left(\operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{5}}{\left(t \right)}\right)^{2} + f{\left(t \right)}\\- l_{0} \left(g m_{1} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + g m_{2} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + g m_{3} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + g m_{4} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + g m_{5} \cos{\left(\operatorname{q_{1}}{\left(t \right)} \right)} + l_{1} m_{2} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{3} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{4} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{2} m_{3} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{2} m_{4} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{2} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{3} m_{4} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} + l_{3} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} + l_{4} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{5}}{\left(t \right)}\right)^{2}\right)\\l_{1} \left(- g m_{2} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} - g m_{3} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} - g m_{4} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} - g m_{5} \cos{\left(\operatorname{q_{2}}{\left(t \right)} \right)} + l_{0} m_{2} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{3} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{4} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{2}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} - l_{2} m_{3} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} - l_{2} m_{4} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} - l_{2} m_{5} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} - l_{3} m_{4} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} - l_{3} m_{5} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} - l_{4} m_{5} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{5}}{\left(t \right)}\right)^{2}\right)\\l_{2} \left(- g m_{3} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} - g m_{4} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} - g m_{5} \cos{\left(\operatorname{q_{3}}{\left(t \right)} \right)} + l_{0} m_{3} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{4} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{1} m_{3} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{4} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{5} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{3}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} - l_{3} m_{4} \sin{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} - l_{3} m_{5} \sin{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2} - l_{4} m_{5} \sin{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{5}}{\left(t \right)}\right)^{2}\right)\\l_{3} \left(- g m_{4} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} - g m_{5} \cos{\left(\operatorname{q_{4}}{\left(t \right)} \right)} + l_{0} m_{4} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{0} m_{5} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{1} m_{4} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{1} m_{5} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{2} m_{4} \sin{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{2} m_{5} \sin{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{4}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} - l_{4} m_{5} \sin{\left(\operatorname{q_{4}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{5}}{\left(t \right)}\right)^{2}\right)\\l_{4} m_{5} \left(- g \cos{\left(\operatorname{q_{5}}{\left(t \right)} \right)} + l_{0} \sin{\left(\operatorname{q_{1}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{1}}{\left(t \right)}\right)^{2} + l_{1} \sin{\left(\operatorname{q_{2}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{2}}{\left(t \right)}\right)^{2} + l_{2} \sin{\left(\operatorname{q_{3}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{3}}{\left(t \right)}\right)^{2} + l_{3} \sin{\left(\operatorname{q_{4}}{\left(t \right)} - \operatorname{q_{5}}{\left(t \right)} \right)} \left(\frac{d}{d t} \operatorname{q_{4}}{\left(t \right)}\right)^{2}\right)\end{matrix}\right]$

```python
mech.find_dynamicsymbols(LM.forcing)
```

    {Derivative(q1(t), t),
     Derivative(q2(t), t),
     Derivative(q3(t), t),
     Derivative(q4(t), t),
     Derivative(q5(t), t),
     f(t),
     q1(t),
     q2(t),
     q3(t),
     q4(t),
     q5(t)}

Simulation
==========

Now that the symbolic equations of motion are available we can simulate the pendulum's motion. We will need some more SymPy functionality and several NumPy functions, and most importantly the integration function from SciPy, `odeint`.

```python
import numpy as np
from numpy.linalg import solve
from scipy.integrate import odeint
```

First, define some numeric values for all of the constant parameters in the problem.

```python
arm_length = 1. / n                          # The maximum length of the pendulum is 1 meter
bob_mass = 0.01 / n                          # The maximum mass of the bobs is 10 grams
parameters = [g, m[0]]                       # Parameter definitions starting with gravity and the first bob
parameter_vals = [9.81, 0.01 / n]            # Numerical values for the first two
for i in range(n):                           # Then each mass and length
    parameters += [l[i], m[i + 1]]
    parameter_vals += [arm_length, bob_mass]
```

Mathematica has a really nice `NDSolve` function for quickly integrating their symbolic differential equations. We make use of SymPy's lambdify function to do something similar, i.e. to create functions that will evaluate the "full" mass matrix, $M$, and "full" forcing vector, $f$ from $M\dot{x} = f(x, r, t)$ as a NumPy function.

```python
dynamic = q + qd    # Make a list of the states
dynamic.append(f)  # Add the input force

M_func = sp.lambdify(dynamic + parameters, LM.mass_matrix_full) # Create a callable function to evaluate the mass matrix
f_func = sp.lambdify(dynamic + parameters, LM.forcing_full)     # Create a callable function to evaluate the forcing vector
```

To integrate the ODE's we need to define a function that returns the derivatives of the states given the current state and time.

```python
def right_hand_side(x, t, args):
    """Returns the derivatives of the states.

    Parameters
    ----------
    x : ndarray, shape(2 * (n + 1))
        The current state vector.
    t : float
        The current time.
    args : ndarray
        The constants.

    Returns
    -------
    dx : ndarray, shape(2 * (n + 1))
        The derivative of the state.

    """
    r = 0.0                                  # The input force is always zero
    arguments = np.hstack((x, r, args))      # States, input, and parameters
    dx = np.array(solve(M_func(*arguments),  # Solving for the derivatives
                  f_func(*arguments))).T[0]

    return dx
```

Now that we have the right hand side function, the initial conditions are set such that the pendulum is in the vertical equilibrium and a slight initial rate is set for each speed to ensure the pendulum falls. The equations can then be integrated with SciPy's `odeint` function given a time series.

```python
x0 = np.hstack((np.pi / 2 * np.ones(len(q)),  # q0...qn+1
                1e-3 * np.ones(len(qd))))          # qd0...un+1
```

```python
t = np.linspace(0.0, 10.0, num=500)  # Time vector
```

```python
x = odeint(right_hand_side, x0, t, args=(parameter_vals,))  # Numerical integration
```

Plotting
========

The results of the simulation can be plotted with matplotlib. First, load the plotting functionality.

```python
import matplotlib.pyplot as plt
```

```python
%matplotlib inline
```

```python
from IPython.core.pylabtools import figsize
figsize(8.0, 6.0)
```

The coordinate trajectories are plotted below.

```python
lines = plt.plot(t, x[:, :x.shape[1] // 2])
lab = plt.xlabel('Time [sec]')
leg = plt.legend(dynamic[:x.shape[1] // 2])
```

And the generalized speed trajectories.

```python
lines = plt.plot(t, x[:, x.shape[1] // 2:])
lab = plt.xlabel('Time [sec]')
leg = plt.legend(dynamic[x.shape[1] // 2:])
```

Animation
=========

matplotlib now includes very nice animation functions for animating matplotlib plots. First we import the necessary functions for creating the animation.

```python
from matplotlib import animation
```

```python
def animate_pendulum(t, states, length, filename=None):
    """Animates the n-pendulum and optionally saves it to file.

    Parameters
    ----------
    t : ndarray, shape(m)
        Time array.
    states: ndarray, shape(m,p)
        State time history.
    length: float
        The length of the pendulum links.
    filename: string or None, optional
        If true a movie file will be saved of the animation. This may take some time.

    Returns
    -------
    fig : matplotlib.Figure
        The figure.
    anim : matplotlib.FuncAnimation
        The animation.

    """
    # the number of pendulum bobs
    numpoints = states.shape[1] // 2

    # first set up the figure, the axis, and the plot elements we want to animate
    fig = plt.figure()

    # set the limits based on the motion
    xmin = -numpoints*length
    xmax = numpoints*length

    # create the axes
    ax = plt.axes(xlim=(xmin, xmax), ylim=(-1.1, 1.1), aspect='equal')

    # display the current time
    time_text = ax.text(0.04, 0.9, '', transform=ax.transAxes)

    # blank line for the pendulum
    line, = ax.plot([], [], lw=2, marker='o', markersize=6)

    # initialization function: plot the background of each frame
    def init():
        time_text.set_text('')
        line.set_data([], [])
        return time_text, line,

    # animation function: update the objects
    def animate(i):
        time_text.set_text('time = {:2.2f}'.format(t[i]))
        x = np.hstack((states[i, 0], np.zeros((numpoints))))
        y = np.hstack((states[i, 0], np.zeros((numpoints))))

        for j in np.arange(1, numpoints):
            x[j] = x[j - 1] + length * np.cos(states[i, j])
            y[j] = y[j - 1] + length * np.sin(states[i, j])
        line.set_data(x, y)
        return time_text, line,

    # call the animator function
    anim = animation.FuncAnimation(fig, animate, frames=len(t), init_func=init,
            interval=t[-1] / len(t) * 1000, blit=True, repeat=False)

    # save the animation if a filename is given
    if filename is not None:
        anim.save(filename, fps=30, codec='libx264')
```

```python
animate_pendulum(t, x, arm_length, filename="open-loop.mp4")
```

```python
from IPython.display import HTML
html = \
"""

"""
HTML(html)
```

```python

```