Godfrey Beddard 'Applying Maths in the Chemical & Biomolecular Sciences an example-based approach' Chapter 9

```python
# import all python add-ons etc that will be needed later on
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from sympy import *
init_printing()                         # allows printing of SymPy results in typeset maths format
plt.rcParams.update({'font.size': 14})  # set font size for plots
```

# Chapter 9 Fourier Series and Transforms

This chapter describes several related topics: the Fourier series and Fourier transforms, as well as autocorrelations, convolutions, and their numerical calculation.

Fourier series are used in solving partial differential equations, as explained in Chapter 10.5, for example, the diffusion and Schroedinger equation. Fourier transforms are used to produce every NMR spectrum, MRI scan, X-ray structure, and IR spectrum recorded; a good grasp of this topic is therefore essential for the molecular scientist.

In many experiments the measuring instrument distorts the data and, in this case, what is measured is convoluted with the response of the instrument. Here Fourier transforms can unravel the data to produce the true response. At the end of the chapter a different type of transform, the Hadamard transform is also described and some examples of its use in time-resolved measurements, such as time-resolved kinetics and x-ray diffraction, given.

## 1 Motivation and concept

The Taylor and Maclaurin series reconstruct functions as an infinite series in the powers $x^n$, and the coefficients needed to do this are the derivatives of the function. These series have rather tight restrictions placed upon them; the function must be differentiable $n$ times over and the remainder must approach zero. These are described in Chapter 5. In a Fourier series, the expansion is performed instead, as trigonometric series in sines and cosines, with two sets of coefficients, $a_n$ and $b_n$, to describe the $n^\text{th}$ term, and which are evaluated by integration. The series formally extends to infinity, but in practice, at most only a few tens of terms are needed to replicate most functions to an acceptable level of approximation. The advantage of using a Fourier rather than a Taylor/Maclaurin series is that a wide class of functions can be described by the series, including discontinuous ones. However, by their very nature, Fourier series can only represent _periodic functions_, and this must not be forgotten. Periodic means that the function repeats itself; the repeat interval is normally taken to be $-\pi$ to $+\pi$ but can be extended to the range $-L$ to $L$ and $L$ can even be made infinite.

To illustrate that it is possible to make an arbitrary shaped function by adding a number of sine and cosine waves, four such waves are shown in Fig.1. It is possible to imagine, without any difficulty, that waveforms that are more complex can be formed by using more sine or cosine terms. The lowest and most complicated waveform is simply the sum of the individual waves, and repeats itself with a period of 2$\pi$. This waveform could be the signal that is measured on an oscilloscope or spectrometer and recorded on a computer. It might alternatively, be part of an image that is formed from adjacent columns of different waveforms. Periodic components present in a signal represent information in the waveform or image can be retrieved by performing a Fourier transform. This unravelling process is discussed later on but now the Fourier series is considered.

**Fig 1**. A complex and _periodic_ waveform or function is constructed out of the sum of sine and/or cosine waves. The complicated waveform, repeats itself with a period of 2$\pi$. In the Fourier series, the reconstruction of this waveform will require many more sine and cosine terms to reconstruct its form than are used to generate it, because the Fourier series only represents a function _exactly_ when an infinite number of terms are included in the summation.

## 1.1 The Fourier Series.

A Fourier series aims to reconstruct a function, such as a waveform, using the weighted sums of sine and cosine functions or their exponential representations. The great importance and usefulness of the Fourier series is that they represents the best fit, in a least-squares way, to a function $f(x)$ because $\displaystyle \int_{-L}^{L}[f(x)-g(x)]^2dx$ is minimized when $g(x)$ is the Fourier series expansion of $f(x)$.

Because the Fourier series can extend to infinity, many more sine and cosine terms can be used to exactly reproduce a waveform, such as that shown in Fig. 1, than were used to produce it in the first place. The Fourier series cannot somehow magically pick out just those sine and cosine waves used to make the original function, therefore very many terms each with a unique frequency are needed to faithfully reconstruct a waveform.

If the waveform shown in Fig. 1 were truncated so that it was zero outside $x = \pm \pi$, for example, then its Fourier series would have to match this new waveform which suddenly becomes zero and do so a series must contain a sufficiently large number terms  not only to cancel out to zero outside the range $x = \pm \pi$, but also to reproduce the function inside the range.

Suppose that, over the range $\pm\pi$, a Fourier series $g(x)$ approximates a 'victim' or target function $f(x)$, which might be $x^3$ or $\displaystyle (e^{-x} - 1)^2$ or any 'normal' function, then the Fourier series $g(x)$ that approximates the true function $f(x)$ can be written down in general and in quite a straightforward manner and is

$$\displaystyle  g(x) = \frac{a_0}{2}+ \sum\limits_{n=1}^\infty a_n\cos(nx) + \sum\limits_{n=1}^\infty b_n\sin(nx)   \tag{1}$$

where $g(x)$ approximates $f(x)$. The summations start at index $n$ = 1, and $n$ is a positive integer. The $a_n$ and $b_n$ coefficients are the integrals

$$\displaystyle   a_n= \frac{1}{\pi}\int\limits_{-\pi}^{+\pi}f(x)\cos(nx)dx \qquad  (n \ge 0) \tag{2} $$
$$\displaystyle   b_n= \frac{1}{\pi}\int\limits_{-\pi}^{+\pi}f(x)\sin(nx)dx \qquad  (n \gt 0) \tag{3} $$

which are normalized by 1/$\pi$.

When the number of terms in the series is large, then $g(x) \rightarrow f (x)$, and when infinite, $g(x) = f (x)$. At each value of $x$, the Fourier series consists of a constant term, $a_0$/2 plus the sum of an infinite number of oscillating terms in integer multiples of $x$. Notice, that the target function $f(x)$ appears as part of the expansion coefficients only, and must, therefore, be capable of being integrated. The target function $f(x)$ determines the weighting to be placed on each term in the expansion, and this is how information about the shape of $f(x)$ is included in the expansion.

If $f(x)$ is periodic in time, then the variable $x$ would normally be changed to $\omega t$ where $\omega = 2\pi\nu$ is the angular frequency in units of radian s$^{-1}$, and $\nu$ is the frequency in s$^{-1}$. If the dimension is spatial, then $x$ is often replaced with $2\pi x/L$ of which $2\pi/L$ can be interpreted as a spatial frequency, with units of radians m${-1}$ by analogy with 'normal' frequencies. Often this spatial frequency is called the wavevector and given the symbol $k$.

Before a worked example, it is worthwhile examining limits other than $\pm\pi$, describe the exponential form of the series and also simplifying some series using symmetry. The $a_n$ and $b_n$ constants are also derived.

### 1.2 Series limits from -$L$ to $L$

Over the range $−L$ to $L$, the equations to use for $f(x)$ are similar to equations (1)–(3) but $x$ is
changed to $\pi x/L$. The series is

$$\displaystyle  g(x) = \frac{a_0}{2}+ \sum\limits_{n=1}^\infty a_n\cos \left(\frac{n\pi x}{L}\right) + \sum\limits_{n=1}^\infty b_n\sin\left(\frac{n\pi x}{L}\right)   \tag{4}$$

and the coefficients are changed similarly,

$$\displaystyle   a_n= \frac{1}{L}\int\limits_{-L}^{L}f(x)\cos \left(\frac{n\pi x}{L}\right)dx \qquad  (n \ge 0) \tag{5} $$
$$\displaystyle   b_n= \frac{1}{L}\int\limits_{-L}^{L}f(x)\sin \left(\frac{n\pi x}{L}\right)dx \qquad  (n \gt 0) \tag{6} $$

Notice that the arguments, limits, and normalization are each changed compared to those when the range is $\pm \pi$. The integral now has limits $\pm L$ instead of $\pm \pi$, and normalization 1/$L$ rather than 1/$\pi$.

### 1.3 Exponential representation

Because the sine and cosine functions can be represented as sums and differences of complex exponential terms, for example, $\displaystyle \cos(x) = (e^{ix} + e^{-ix})/2$, the most general way of describing the Fourier series is to use the complex exponential form;

$$\displaystyle g(x) = \sum\limits_{n=-\infty}^\infty c_ne^{+i\, n\pi x/L}    \tag{7}$$

and the coefficients become

$$\displaystyle c_n=\frac{1}{2L}\int\limits_{-L}^L f(x)e^{-i\,n\pi x/L}dx    \tag{8} $$

Note the change in sign and limits in the second exponential, and that $i=\sqrt{-1}$. The set of coefficients $c_n$, are sometimes called the _amplitude spectrum_ of the transform. Note also that for practical purposes the expansion $g(x)$, and the true function $f(x)$  are considered equivalent.

### 1.4 Deriving the integral describing the $a$ and $b$ coefficients

The integrals describing the coefficients $a_n$ are obtained by multiplying each term in the Fourier series by $\cos(mx)$, where $m$ is an integer, then integrating term by term using the orthogonality of sine and cosine integrals as necessary. The coefficients $b_n$ can be obtained similarly by multiplying the series by $\sin(mx)$ and integrating.

First, the form of the cosine integrals is examined. The integrals containing a product of sine and cosine are all zero for any $n$ and $m$ because the integral has 'odd' symmetry because of the sine, and the limits are symmetrical;

$$\displaystyle  \int\limits_{-\infty}^{+\infty}\cos(mx)\sin(nx)dx = 0  \tag{9}$$

To convince you that the symmetry makes the integral zero, which is the area under the curve, a curve is plotted below, see figure 2. The curve, while complicated, is inverted about $x$ = 0 so that the area $\gt$ 0 is equal and opposite to that $\lt$ 0. More details on odd and even functions is given in the next section.

Figure 2. An odd function such as shown above has zero integral when evaluated over a symmetrical region about zero.
____

The product of two cosines makes the integral

$$\displaystyle \int\limits_{-\pi}^\pi \cos(mx)\cos(nx)dx = \pi\delta_{n,m}   \tag{10}$$

where $m$ and $n$ are integers and the (Kronecker) delta function $\delta_{n,m}$ is zero only if $n \ne m$, and is $1$ if $n = m$.

To calculate the $a_n$ coefficients, $f(x)$ is represented as the Fourier series $g(x)$

$$\displaystyle  f(x) \approx g(x) = \frac{a_0}{2}+ \sum\limits_{n=1}^\infty a_n\cos(nx) + \sum\limits_{n=1}^\infty b_n\sin(nx) \tag{11}$$

and this is now  multiplied by $\cos(mx)$ and integrated. The cosine integrals with $m\gt$ 0 are,

$$\int\limits_{-\pi}^\pi f(x) \cos(mx)dx = \frac{a_0}{2}\int\limits_{-\pi}^\pi \cos(mx)dx +\int\limits_{-\pi}^\pi \left [\sum\limits_{n=1} a_n\cos(nx)+ \sum\limits_{n=1} b_n\sin(nx ) \right] \cos(mx)dx \tag{12}$$

These integrals can be easily evaluated: The first is zero by symmetry, and by direct integration, $\displaystyle \int_{-\pi}^\pi \cos(mx)dx=\left. \frac{\sin(mx)}{m}\right|_{-\pi}^\pi =0$. The integral containing $b_n$ is given by eqn. 10 and is zero also when $n \ne m$, therefore eqn. 12 becomes

$$\displaystyle \int\limits_{-\pi}^\pi f(x) \cos(mx)dx =\pi\sum\limits_{n=1}a_n\delta_{n,m} = \pi a_n$$

because the delta function picks out just one term from the summation. Rearranging gives

$$\displaystyle a_n = \frac{1}{\pi}\int\limits_{-\pi}^\pi f(x) \cos(mx)dx   \quad (n \gt 0) $$

The index has been changed to $n$ from $m$ because both $n$ and $m$ are integers and only one index is needed; equation (11) was written initially with $n$, so this is again chosen. When $m = n =$ 0, equation (12) becomes

$$\displaystyle \int\limits_{-\pi}^\pi f(x)dx =\frac{a_0}{2}\int\limits_{-\pi}^\pi 1dx=\pi a_0 $$

making $\displaystyle a_0= \pi^{-1}\int\limits_{-\pi}^\pi f(x)dx$. Similar arguments lead to the calculation of $b_n$.

### 1.5 Odd and even functions

When functions are either even or odd, the Fourier series is simplified to cosine or sine series respectively. An even function has the property $f (-x) = f (x)$ and the sine integration (eqn. 6) producing the Fourier coefficient $b_n$ is odd and therefore $b_n$ = 0, but only because the integration limits are symmetrical. The cosine integral (eqn. 5) is not zero, and can be written as

$$\displaystyle a_n=\frac{2}{\pi}\int\limits_0^\pi f(x)\cos(nx)dx   \tag{13}$$

When $f(x)$ is odd $f(-x) = -f(x)$, the opposite situation arises; $a_n$ = 0 and only the sine terms remain;

$$\displaystyle b_n=\frac{2}{\pi}\int\limits_0^\pi f(x)\sin(nx)dx   \tag{14}$$

In cases when $f(x)$ is neither odd nor even, for example $f(x) = \pi/2 - x$, both coefficients have to be evaluated.

As an example, consider calculating the series of $f(x) = x^2$ over the range $\pm \pi$. Because this is an even function, the sine integral should be zero and $b_n$ = 0. The $a$ coefficients are

$$\displaystyle a_n=\frac{2}{\pi}\int\limits_0^\pi x^2\cos(nx)dx  $$

and when $n$ = 0,

$$\displaystyle a_0=\frac{2}{\pi}\int\limits_0^\pi x^2dx =\frac{2}{3}\pi^2 $$

When $n\gt$ 0, $a_n$ can be evaluated using integration by parts.

```python
#using SymPy to do the indefinite integration
x ,n = symbols('x n', positive = True)
func = x**2*cos(n*x)/pi
an = integrate(func,x, conds='none')      # coefficients a_n
an
```

Adding the limits $\pm \pi$ produces zero for the sine terms and the remaining cosine produces $\displaystyle a_n=(-1)^n\frac{4}{n^3}$. All the $b_n$=0 thus using eqn. 4. the Fourier series for $x^2$ is

$$\displaystyle  x^2 \approx g(x)= \frac{\pi^2}{3}-4\cos(x)+\cos(2x)-\frac{4}{9}\cos(3x) \cdots \ = \frac{\pi^2}{3}+4\sum\limits_{n=1}^\infty \frac{(-1)^n}{n^2}\cos(nx)     \tag{15} $$

which, as shown in the next figure, is a good representation of $x^2$. The following Python code is included as an example of a Fourier series calculation

```python
fig1= plt.figure(figsize=(12.5, 5.0))
plt.rcParams.update({'font.size': 16})  # set font size for plots
ax0 = fig1.add_subplot(1,3,1)
ax1 = fig1.add_subplot(1,3,2)
ax2 = fig1.add_subplot(1,3,3)

gx = lambda x, k : (np.pi**2)/3.0 + 4.0*sum( (-1)**n *np.cos(n*x)/n**2 for n in range(1,k)) # series summation

x = np.linspace(-2*np.pi,2*np.pi,200)   # define x values, -pi to pi with 200 data points

for i,ax in enumerate([ax0,ax1]):
    j = 20
    if i == 0 : j = 3      # number of terms in summation
    ax.plot(x/np.pi, gx(x,j),linestyle='dashed',color='black',label='max n = '+str(j)) # plot as x/pi , 4 terms
    ax.plot(x/np.pi,x**2,color='red')
    ax.set_xlabel(r'$x/\pi$')
    ax.set_ylim([-1,15])
    ax.axhline(0)
    ax.legend()

kk = 10
nvals = [i for i in range(kk)]   # make list of values 0, 1, 2,...
fn = lambda  n : (np.pi**2)/3.0 if n ==0 else  (-1)**n/n**2 if n >0 else 0
an = [fn(i) for i in range(kk)]
ax2.axhline(0,linewidth=1, color='grey')
ax2.bar(nvals,an,label=r'$a_n$ coefficients')
ax2.set_ylim([-2,4])
ax2.set_xlabel('n')
ax2.legend()
plt.tight_layout()
plt.show()
```

**Figure 3**. Left: Plot of $x^2$ and its Fourier series to $n$ = 3 showing a poor fit to the true function. More terms produce a better fit but only over the range $\pm \pi$ as shown in the centre panel where 20 terms are included in the summation. This plot also shows how the fit is only over the range $\pm \pi$ and then it repeats itself. On the right are shown the coefficients $a_n$ and shows how rapidly their value decreases as $n$ increases.

```python

```


```python
%matplotlib inline

import numpy as np
import scipy as sc
import pandas as pd
import matplotlib.pyplot as plt
import seaborn as sns
import sympy as sp

import itertools
import math

sns.set();
```

# Dormand-Prince Method

> Disclaimer: the original code (written in Scilab) and example below are due to Leonardo Guidi.

```python
def dopri(x0, xf, y0, f, rtol, atol):
    r"""
    Dormand-Prince method for solving first-order ODE.

    This function implements the Dormand-Prince5(4) method
    with adaptive step through the estimation of the error
    tolerance (with norm sup) from the approximations of
    global error O(h**5) and O(h**4).

    It solves the following IVP:

    y'=f(x,y), x > x0,
    y(x0) = y0.

    The initial step is computed from the algorithm proposed in
    Hairer,E; Nørsett,S.P.; Wanner,G. "Solving Ordinary
    Differential Equations I. Nonstiff Problems" 2nd rev.ed. (1993),
    pg. 169.

    The estimatives for the computation of the step follow the ideas
    exposed in this very same reference in pages 167 and 168.

    Parameters
    ----------
    x0 : float
        The initial value for the independent variable.
    xf : float
        The final value for the independent variable.
    y0 : array_like
        1-D array of initial values for the dependente variable evaluated at `x_0`.
    f : callable
        The function that represents the first derivative of y with respect to x.
        It must accept two arguments: the point x at which it will be evaluated and
        the value of y at this point.

    Returns
    ------
    x : numpy.ndarray
        The point at which the function was evaluated.
    y : numpy.ndarray
        The approximation of the solution.
    hc : numpy.ndarray
        Steps used along the computations.
    rp : list
        Iterations where rejection of the step size occurred.
    """

    xi = x0
    yi = y0
    fac = 0.8
    facmax = 5
    facmin = 0.2

    a = np.zeros((7, 7))
    a[1, 0] = 1/5
    a[2, 0:2] = np.array([3/40, 9/40])
    a[3, 0:3] = np.array([44/45, -56/15, 32/9])
    a[4, 0:4] = np.array([19372/6561, -25360/2187, 64448/6561, -212/729])
    a[5, 0:5] = np.array([9017/3168, -355/33, 46732/5247, 49/176, -5103/18656])
    a[6, 0:6] = np.array([35/384, 0.0, 500/1113, 125/192, -2187/6784, 11/84])

    b = np.array([5179/57600, 0.0, 7571/16695, 393/640, -92097/339200, 187/2100, 1/40])

    c = np.array([0.0, 1/5, 3/10, 4/5, 8/9, 1.0, 1.0])

    i = 0
    r = 0

    x = []
    y = []
    hc = []
    rp = []

    done = False

    d0 = np.max(np.abs(y0))
    d1 = np.max(np.abs(f(x0, y0)))

    h0 = 1.0e-6 if np.min([d0, d1]) < 1.0e-5 else (0.01 * (d0/d1))

    d2 = np.max(np.abs(f(x0 + h0, y0 + h0 * f(x0, y0)) - f(x0, y0)))/h0

    h1 = np.max(1.0e-6, h0/1000) if np.max([d1, d2]) < 1.0e-15 else (0.01/np.max([d1, d2]))**(1/5)

    h = np.min([100 * h0, h1])

    while not done:
        k1 = f(xi, yi)
        k2 = f(xi + h * c[1], yi + h * (a[1, 0] * k1))
        k3 = f(xi + h * c[2], yi + h * (a[2, 0] * k1 + a[2, 1] * k2))
        k4 = f(xi + h * c[3], yi + h * (a[3, 0] * k1 + a[3, 1] * k2 + a[3, 2] * k3))
        k5 = f(xi + h * c[4], yi + h * (a[4, 0] * k1 + a[4, 1] * k2 + a[4, 2] * k3 + a[4, 3] * k4))
        k6 = f(xi + h * c[5], yi + h * (a[5, 0] * k1 + a[5, 1] * k2 + a[5, 2] * k3 + a[5, 3] * k4 + a[5, 4] * k5))

        y4 = yi + h * (a[6, 0] * k1 + a[6, 2] * k3 + a[6, 3] * k4 + a[6, 4] * k5 + a[6, 5] * k6)

        k7 = f(xi + h * c[6], y4)

        y5 = yi + h * (b[0] * k1 + b[2] * k3 + b[3] * k4 + b[4] * k5 + b[5] * k6 + b[6] * k7)

        sc = atol + np.max([np.abs(y5), np.abs(yi)]) * rtol
        err = np.max(np.abs(y5 - y4))/sc
        hnew = h * np.min([facmax, np.max([facmin, fac * (1/err)**(1/5)])])

        if err <= 1:
            if (xi + h) <= xf:
                i += 1
                xi += h
                yi = y5
                x.append(xi)
                y.append(yi)
                hc.append(h)
                h = hnew
            else:
                h = xf - xi
                r += 1
                rp.append(i)
        else:
            h = hnew
            r += 1
            rp.append(i)

        done = xi >= xf

    return (np.array(x), np.array(y), np.array(hc), rp)
```

## Example 1

```python
def F(t, z):
    return np.array([z[1], z[0]**(-13) - z[0]**(-7)])

t, sol, hi, rejp = dopri(0.0, 100.0, np.array([2.0, 0.0]), F, atol=1.0e-8, rtol=1.0e-8)

q = np.insert(sol[:, 0], 0, 2)
p = np.insert(sol[:, 1], 0, 0)

H = (1/2) * (p**2) + (1/12) * (q**(-12) - 2 * q**(-6))
```

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(q, p)
plt.scatter(q[0], p[0], marker='o', color='red')
plt.scatter(q[-1], p[-1], marker='x', color='black')
ax.set(title="Phase space", xlabel="$q$", ylabel="$p$");
```

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(np.abs(H/H[0] - 1))
ax.set(title="Hamiltonian's relative error along the appproximations", xlabel="t", ylabel="$H$");
```

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot((q**-13 - q**-7)**2, label=r"$(dV/dt)^{2}$", color='red')
plt.plot((p**2) * (13 * q**-14 - 7 * q**-8), label=r"$d^{2}V/dt^{2}$", color='blue')
plt.legend(loc='upper right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title=r"Graph of $(dV/dt)^{2}$ and $d^{2}V/dt^{2}$", xlabel="t");
```

## Question 1

```python
def F(t):
    return 1/2 + sum(5/n * math.sin(n*math.pi/10) * math.cos(n*math.sqrt(10)*t) for n in range(1, 6))

def f(t, u_k):
    return np.array([u_k[1], F(t) * math.cos(u_k[0]) - (1/10) * u_k[1] - 10 * math.sin(u_k[0])])

def mechanical_energy(theta, dtheta):
    return (1/2) * dtheta**2 - 10 * np.cos(theta)

t, sol, hi, rejp = dopri(0.0, 20.0, np.array([0.0, 0.0]), f, atol=1.0e-10, rtol=1.0e-10)
theta = sol[:, 0]
dtheta = sol[:, 1]

energy = mechanical_energy(theta, dtheta)

max_energy_index = np.argmax(energy)
t_max_energy = t[np.argmax(energy)]
max_energy = energy[np.argmax(energy)]

print("Mechanical energy in t = {:.4} is {:.4} J.".format(t_max_energy, max_energy))
```

    Mechanical energy in t = 6.145 is -0.7452 J.

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(t, theta, color='blue', label=r"$\theta$")
plt.plot(t, dtheta, color='red', label=r"$d\theta/dt$")
plt.legend(loc='upper right', fancybox=True, framealpha=1, shadow=True, borderpad=1, frameon=True)
ax.set(title="Solution of the PVI", xlabel="Time", ylabel="Angle and its derivative");
```

```python
fig, ax = plt.subplots(figsize=(13, 8))
plt.plot(t, energy, color='blue')
plt.scatter(t_max_energy, max_energy, color='red', marker='x', s=100)
plt.text(0.0, max_energy, "Maximum energy in t = {:.2} is {:.2} J.".format(t_max_energy, max_energy))
ax.set(title="Mechanical energy", xlabel="Time", ylabel="Energy");
```


```
!pip install pystan
```

    Collecting pystan
      Downloading https://files.pythonhosted.org/packages/49/f9/106f3db81cf4da1be937a11b63f400d072718662f3fd1349ea7f0458aa0f/pystan-2.18.0.0-cp36-cp36m-manylinux1_x86_64.whl (50.0MB)
        100% |████████████████████████████████| 50.0MB 762kB/s
    Collecting Cython!=0.25.1,>=0.22 (from pystan)
      Downloading https://files.pythonhosted.org/packages/64/3f/cac281f3f019b825bbc03fa8cb7eb03d9c355f4aa9eef978279a4966cb21/Cython-0.29-cp36-cp36m-manylinux1_x86_64.whl (2.1MB)
        100% |████████████████████████████████| 2.1MB 13.4MB/s
    Requirement already satisfied: numpy>=1.7 in /usr/local/lib/python3.6/dist-packages (from pystan) (1.14.6)
    Installing collected packages: Cython, pystan
    Successfully installed Cython-0.29 pystan-2.18.0.0

```
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
import pystan
import seaborn
from scipy.stats import mstats
%matplotlib inline
```

### ガウシアンプロセスサンプラー
下記の式に従ってサンプリングを行うガウシアンプロセスサンプラー。

#### 共分散関数

$$
{K(x|α,ρ)_{i,j} =α^2\exp\Big(−\frac{1}{2ρ^2} \sum_{d=1}^D(x_{i,d} −x_{j,d})^2\Big)
}
$$

において、$a = 1$, $\rho=1$ を適用し、

$$
{K(x|1,1)_{i,j} =\exp\Big(−\frac{1}{2} \sum_{d=1}^D(x_{i,d} −x_{j,d})^2\Big)
}
$$

とした共分散関数を利用します。ここで $i, j$ はそれぞれデータ数方向のindexであり、$d$ はベクトル型データの次元方向のindexです。

#### 平均関数
平均関数は $ 0$ ベクトルを利用します。

#### ガウシアンプロセスサンプラー
下記のガウシアンプロセスサンプラーを用います。

$$
\begin{align}
f(x) & =\mathcal {GP} ({0}, K(x | \alpha, \rho)) \\
y &\sim \mathcal N(f(x), 0.3)
\end{align}
$$

ちなみに実際にサンプリングするときは有限の数のデータ $x$ を利用するので $\mathcal f(x) = {GP}$ を二番目の式に代入してしまって、確率分布の計算を考えた場合には

$$
y \sim \mathcal N({0}, K(x | \alpha, \rho) + 0.3I)
$$

となることろ利用します。$I$ は単位行列で、元々の二番目の式の分だけ分散が大きくなる（共分散は変わらない）という雰囲気だけ…。

```
stan_model = """
data {
  int<lower=1> N;
  real x[N];
}

transformed data {
  matrix[N, N] K = cov_exp_quad(x, 1.0, 1.0);
  vector[N] mu = rep_vector(0, N);
  for (n in 1:N)
    K[n, n] = K[n, n] + 0.3;
}

parameters {
  vector[N] y;
}

model {
  y ~ multi_normal(mu, K);
}
"""
```

```
sm = pystan.StanModel(model_code=stan_model)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_66b01db10e2bf62035a80c81f7695a54 NOW.
    /usr/local/lib/python3.6/dist-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/tmptyp_p_up/stanfit4anon_model_66b01db10e2bf62035a80c81f7695a54_4329639226961591462.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)

```
x_data = np.linspace(-5, 5, 50)
N_data = x_data.shape[0]
y_sample = sm.sampling(iter=500, warmup=100, chains=3, seed=1992,
                       data={'N': N_data, 'x': x_data})
```

```
y_data = y_sample.extract("y")["y"]
```

#### サンプルを可視化
合計で1200個のサンプルが得られました。

適当に何個かを選んで見てみましょう。

```
y_data.shape

```

    (1200, 50)

```
plt.figure(figsize=(16, 8))
for i in range(4):
    plt.subplot(2, 2, i+1)
    plt.scatter(x_data, y_data[i, :])
```

### ガウシアンプロセス回帰
上記のガウシアンプロセスサンプラーでは勝手に共分散関数 $K(x|\alpha, \rho)$ を勝手に決めて、サンプリングを行いました。実際のデータの生成プロセスは、まさかこんな再生核ヒルベルト空間を介して出てきているわけがありませんので、これは単にガウシアンプロセスを見るための例題です。もう一度式を見ましょう。

 $$
\begin{align}
f(x) & =\mathcal {GP} ({0}, K(x | \alpha, \rho)) \\
y &\sim \mathcal N(f(x), 1)
\end{align}
$$

 重要なのは、何らかの生成プロセスを通して観測したデータ $(x, y)^N$ に対して、ガウシアンプロセスで回帰を行う場合は、共分散関数 $\alpha, \rho$ を上手く調整ししてやることになります（なぜならガウシアンプロセスサンプラーが発生するデータは、これらのパラメータに依存しているため）。

 より一般的には、観測時のノイズ もパラメータ化して

 $$
\begin{align}
f(x) & =\mathcal {GP} ({0}, K(x | \alpha, \rho)) \\
y &\sim \mathcal N(f(x), \sigma^2)
\end{align}
$$

 とします。更に事前分布としてパラメータには下記のような分布を敷いておきましょう。特に何も仮定したくなければ無情報事前分布を敷いておけばいいです。

 $$
 \begin{align}
 \rho &\sim \mathcal {InvGamma} (1, 1) \\
 \alpha & \sim \mathcal N(0, 1) \\
 \rho & \sim \mathcal N(0, 1) \\
 \end{align}
 $$

```
stan_model_fit = """
data {
  int<lower=1> N;
  real x[N];
  vector[N] y;
}

transformed data {
  vector[N] mu = rep_vector(0, N);
}

parameters {
  real<lower=0> rho;
  real<lower=0> alpha;
  real<lower=0> sigma;
}

model {
  matrix[N, N] L_K;
  matrix[N, N] K = cov_exp_quad(x, alpha, rho);
  real sq_sigma = square(sigma);

  // diagonal elements
  for (n in 1:N)
    K[n, n] = K[n, n] + sq_sigma;

  L_K = cholesky_decompose(K);

  rho ~ inv_gamma(5, 5);
  alpha ~ normal(0, 1);
  sigma ~ normal(0, 1);

  y ~ multi_normal_cholesky(mu, L_K);
}
"""
```

#### 補足事項
気づいた方もいるかもしれませんが、`cholesky_decompose`という文言が入っています。これは、共分散関数（有限データに対して分散共分散行列）は対称行列ですので、実際 $N \times N$ の行列を保持する意味はありません。兎にも角にも、メモリや計算量を節約しているという認識でいいでしょう。`multi_normal_cholesky`の部分ではコレスキー分解された共分散関数を渡すという約束です（こいつも計算量を節約したガウス分布と思って構わない）。

### パラメータの事後分布表示

```
sm_fit = pystan.StanModel(model_code=stan_model_fit)
```

    INFO:pystan:COMPILING THE C++ CODE FOR MODEL anon_model_57d3fcd935a2af9ccbdd75d872c99b40 NOW.
    /usr/local/lib/python3.6/dist-packages/Cython/Compiler/Main.py:367: FutureWarning: Cython directive 'language_level' not set, using 2 for now (Py2). This will change in a later release! File: /tmp/tmp81r22676/stanfit4anon_model_57d3fcd935a2af9ccbdd75d872c99b40_5820035884268912099.pyx
      tree = Parsing.p_module(s, pxd, full_module_name)

```
fit = sm_fit.sampling(iter=2000, warmup=500, chains=3, seed=1992,
                           data={'N': N_data, 'x': x_data, 'y': y_data[0, :] })
```

```
rho = fit.extract('rho')['rho']
alpha = fit.extract('alpha')['alpha']
sigma = fit.extract('sigma')['sigma']
```

#### $\rho$ の事後分布
データを生成したガウシアンプロセスサンプラーは $1.0$ を設定していたのでした。

```
plt.hist(rho, bins=20)
```

#### $\alpha$ の事後分布
データを生成したガウシアンプロセスサンプラーは $1.0$ を設定していたのでした。

```
plt.hist(alpha, bins=20)
```

#### $\sigma$ の事後分布
データを生成したガウシアンプロセスサンプラーは $0.3$ を設定していたのでした。
少し高めに出ているように見えますね。

```
plt.hist(sigma, bins=20)
```

```

```


# piston example with Gauss-Legendre collocation

lumped moving mass + two lumped compartments (ideal gas of fixed mass)
state-space
$
\begin{bmatrix}
    p & s_1 & v_1 & s_2 & v_2
\end{bmatrix}^\mathrm{T}
\: \in \:
\mathcal{X} \subset \mathbb{R}^5
$

When joining the piston and the two compartments
a constraint between the state variables is introduced, namely
$v_1 + v_2 = A \left( l - w \right)$.

In piston_animation.ipynb
this has been eliminated, thereby transforming the DAE system into an ODE system.
This lead to the state space
$
\begin{bmatrix}
    q & p & s_1 & s_2
\end{bmatrix}^\mathrm{T}
\: \in \:
\mathcal{X} \subset \mathbb{R}^4
$.

```python
import matplotlib
import matplotlib.pyplot as plt
import matplotlib.animation as anim

import sympy
sympy.init_printing()

from IPython.display import display

import numpy

import sys
sys.path.insert(0, './code')

from gauss_legendre import gauss_legendre
from symbolic import eval_expr
from evaluate_functional import evaluate_functional
import ideal_gas_lumped
```

### macroscopic state

```python
p = sympy.Symbol('p')
s_1 = sympy.Symbol('s_1')
v_1 = sympy.Symbol('v_1')
s_2 = sympy.Symbol('s_2')
v_2 = sympy.Symbol('v_2')

x = [p, s_1, v_1, s_2, v_2]
x
```

### parameters

```python
l = sympy.Symbol('l')
r = sympy.Symbol('r')
w = sympy.Symbol('w')
A = sympy.Symbol('A')

m_Al = sympy.Symbol('m_Al')
m_Cu = sympy.Symbol('m_Cu')
m = sympy.Symbol('m')

κ_Al = sympy.Symbol('κ_Al')
κ_Cu = sympy.Symbol('κ_Cu')
α = sympy.Symbol('α')

d = sympy.Symbol('d')

params = {
    # length of cylinder (m)
    l: 0.1,

    # radius of cylinder / piston (m)
    r: 0.05,

    # length of piston (m)
    w: 0.006,

    # cross-sectional area of piston (m**2)
    A: r**2 * sympy.pi,

    # density of piston material (kg/m**3)
    m_Al: 2700.0,
    m_Cu: 8960.0,

    # mass of piston (kg)
    m: m_Cu * A * w,

    # thermal conductivity piston material (W/(m*K))
    κ_Al: 237.0,
    κ_Cu: 401.0,

    # thermal conduction coefficient through piston (W/K)
    α: κ_Cu * A / w,

    # friction coefficient between piston and cylinder (N*s/m)
    d: 1.0,
}

params = {**params, **ideal_gas_lumped.params}
```

### functionals

```python
functionals = {}

υ = sympy.Symbol('υ')
functionals[υ] = p / m

U_1 = sympy.Symbol('U_1') # internal energy
m_1 = sympy.Symbol('m_1') # fixed mass (kg)
θ_1 = sympy.Symbol('θ_1') # temperature (K)
π_1 = sympy.Symbol('π_1') # pressure (Pa)
ideal_gas_lumped.add_functionals(functionals, U=U_1, s=s_1, v=v_1, m=m_1, m_a=ideal_gas_lumped.m_Ar, θ=θ_1, π=π_1)

U_2 = sympy.Symbol('U_2')
m_2 = sympy.Symbol('m_2')
θ_2 = sympy.Symbol('θ_2')
π_2 = sympy.Symbol('π_2')
ideal_gas_lumped.add_functionals(functionals, U=U_2, s=s_2, v=v_2, m=m_2, m_a=ideal_gas_lumped.m_Ar, θ=θ_2, π=π_2)

q = sympy.Symbol('q')
functionals[q] = v_1/A + w/2
q_2 = sympy.Symbol('q_2')
functionals[q_2] = l - v_2/A - w/2

E = sympy.Symbol('E') # total energy (J)
functionals[E] = sympy.Rational(1,2) * p**2 / m + U_1 + U_2

S = sympy.Symbol('S')
functionals[S] = s_1 + s_2
```

```python
eval_expr(S, functionals)
```

```python
eval_expr(q - q_2, functionals) # should be zero
```

constraint
$v_1 + v_2 = A \left( l - w \right)$

### initial conditions

determine $m_1$, $m_2$ and $s_1(0)$, $s_2(0)$

```python
# wanted conditions
v_10 = float(eval_expr((A*l - A*w) / 2, params=params))
v_20 = v_10

θ_10 = 273.15 + 25.0
π_10 = 1.5 * 1e5

θ_20 = 273.15 + 20.0
π_20 = 1.0 * 1e5
```

```python
import ideal_gas
from scipy.optimize import fsolve
```

```python
n_10 = fsolve(lambda n : ideal_gas.S_π(ideal_gas.U2(θ_10, n), v_10, n) - π_10, x0=2e22)[0]
s_10 = ideal_gas.S(ideal_gas.U2(θ_10, n_10), v_10, n_10)
# relative to Standard conditions for Temperature and Pressure (STP)
print(f"n = {n_10}")
print(f"s = {s_10}")
print(f"θ = {ideal_gas.U_θ(s_10, v_10, n_10) - 273.15} °C")
print(f"π = {ideal_gas.U_π(s_10, v_10, n_10) * 1e-5} bar")
print(f"u = {ideal_gas.U(s_10, v_10, n_10)}")
```

    n = 1.345119603317771e+22
    s = 3.3828942208847232
    θ = 25.00000000000159 °C
    π = 1.500000000000008 bar
    u = 83.05585577928

```python
n_20 = fsolve(lambda n : ideal_gas.S_π(ideal_gas.U2(θ_20, n), v_20, n) - π_20, x0=2e22)[0]
s_20 = ideal_gas.S(ideal_gas.U2(θ_20, n_20), v_20, n_20)
print(f"n = {n_20}")
print(f"s = {s_20}")
print(f"θ = {ideal_gas.U_θ(s_20, v_20, n_20) - 273.15} °C")
print(f"π = {ideal_gas.U_π(s_20, v_20, n_20) * 1e-5} bar")
print(f"u = {ideal_gas.U(s_20, v_20, n_20)}")
```

    n = 9.12041411630436e+21
    s = 2.3394613409617664
    θ = 20.000000000004206 °C
    π = 1.0000000000000147 bar
    u = 55.37057051952051

```python
x_0 = [0, s_10, v_10, s_20, v_20]
params[m_1] = n_10 * ideal_gas_lumped.m_Ar
params[m_2] = n_20 * ideal_gas_lumped.m_Ar
x_0
```

### dynamics

```python
J_1 = sympy.Matrix([
    [0, -A, A, 1],
    [A, 0, 0, 0],
    [-A, 0, 0, 0],
    [-1, 0, 0, 0]
])

J_2 = sympy.Matrix([
    [0, 1, 1],
    [-1, 0, 0],
    [-1, 0, 0]
])

J_3 = J_2

J = sympy.BlockDiagMatrix(J_1, J_2, J_3).as_explicit()
J
```

```python
θ_0 = sympy.Symbol("θ_0")
```

```python
e_1 = sympy.Matrix([
    [υ],
    [-π_1],
    [-π_2],
    [-d*υ]
])

e_2 = sympy.Matrix([
    [θ_1 - θ_0],
    [1/θ_1 * α*(θ_2 - θ_1)],
    [1/θ_1 * d*υ**2/2]
])

e_3 = sympy.Matrix([
    [θ_2 - θ_0],
    [1/θ_2 * α*(θ_1 - θ_2)],
    [1/θ_2 * d*υ**2/2]
])

e = sympy.BlockMatrix([[e_1], [e_2], [e_3]]).as_explicit()
```

```python
f = -(J @ e).simplify()
f
```

$$\left[\begin{matrix}- A π_{1} + A π_{2} + d υ\\- A υ\\A υ\\υ\\- \frac{\frac{d υ^{2}}{2} - α \left(θ_{1} - θ_{2}\right)}{θ_{1}}\\- θ_{0} + θ_{1}\\- θ_{0} + θ_{1}\\- \frac{\frac{d υ^{2}}{2} + α \left(θ_{1} - θ_{2}\right)}{θ_{2}}\\- θ_{0} + θ_{2}\\- θ_{0} + θ_{2}\end{matrix}\right]$$

```python
x
```

```python
xdot = [-f[0], -f[4], -f[1], -f[7], -f[2]]
xdot
```

```python
constraints = [v_1 + v_2 - A * (l - w)]
```

```python
t_f = 0.8
dt = 10e-4
s = 2
print(f"K = {int(t_f // dt)}")
```

    K = 800

```python
class Logger:
    def __init__(self):
        self.log = []

    def info(self, message):
        self.log.append(message)
```

```python
logger = Logger()
%time time, solution = gauss_legendre(x, xdot, x_0, t_f, dt, s, functionals, params, tol=1e-9, logger=logger, constraints=constraints)
```

    CPU times: user 1.74 s, sys: 9.73 ms, total: 1.75 s
    Wall time: 1.76 s

```python
logger.log[0:5]
```

    ['converged after 3 steps',
     'converged after 3 steps',
     'converged after 3 steps',
     'converged after 3 steps',
     'converged after 3 steps']

```python
time2, solution2 = time.copy(), solution.copy()
```

```python
fig, ax = plt.subplots(dpi=200)
ax.plot(time, solution[:, 2], label="v_1");
#ax.plot(time, solution[:, 4], label="v_2");
#ax.legend();
ax.plot(time2, solution2[:, 2], '--');
```

```python
import matplotlib.ticker as mticker

class MathTextSciFormatter(mticker.Formatter):
    def __init__(self, fmt="%1.2e"):
        self.fmt = fmt
    def __call__(self, x, pos=None):
        s = self.fmt % x
        decimal_point = '.'
        positive_sign = '+'
        tup = s.split('e')
        significand = tup[0].rstrip(decimal_point)
        sign = tup[1][0].replace(positive_sign, '')
        exponent = tup[1][1:].lstrip('0')
        if exponent:
            exponent = '10^{%s%s}' % (sign, exponent)
        if significand and exponent:
            s =  r'%s{\times}%s' % (significand, exponent)
        else:
            s =  r'%s%s' % (significand, exponent)
        return "${}$".format(s)

fig, axes = plt.subplots(nrows=3, sharex=True, dpi=200)
fig.tight_layout(pad=1.5)

# piston position q
axes[0].set_title(r"piston position $q \: (\mathrm{m})$")
axes[0].plot(time, evaluate_functional(x, q, solution, functionals, params))
#axes[0].set_ylabel(r"$q \: (\mathrm{m})$")
axes[0].ticklabel_format(style='sci', axis='y', scilimits=(0,0))
axes[0].xaxis.major.formatter._useMathText = True
#axes[0].yaxis.set_major_formatter(MathTextSciFormatter("%1.2e"))

# total energy E
axes[1].set_title(r"total energy $E \: (\mathrm{J})$")
axes[1].plot(time, evaluate_functional(x, E, solution, functionals, params))
#axes[1].set_ylabel(r"$E \: (\mathrm{J})$")
axes[1].ticklabel_format(style='sci', axis='y', scilimits=(0,0))
#axes[1].yaxis.set_major_formatter(MathTextSciFormatter("%1.2e"))

# total entropy S
S = s_1 + s_2
axes[2].set_title(r"total entropy $S \: (\mathrm{J}/\mathrm{K})$")
axes[2].plot(time, evaluate_functional(x, S, solution, functionals, params))
axes[2].set_xlabel(r"time $t \: (\mathrm{s})$")
#axes[2].set_ylabel(r"$S \: (\mathrm{J}/\mathrm{K})$");
#axes[2].ticklabel_format(style='sci', axis='y', scilimits=(0,0))
axes[2].yaxis.set_major_formatter(MathTextSciFormatter("%1.2e"))

# Format with 2 decimal places
#plt.gca().yaxis.set_major_formatter(MathTextSciFormatter("%1.2e"))

# error on hidden constraint
# V = v_1 + v_2
# V_tot = A * (l - w)
# axes[3].plot(time, evaluate_functional(x, V - V_tot, solution, functionals, params))
# axes[3].set_xlabel(r"$t \: (\mathrm{s})$")
# axes[3].set_ylabel(r"$\tilde{V} \: (\mathrm{m^3})$");
# axes[3].ticklabel_format(style='sci', axis='y', scilimits=(0,0))

font = {
    'family' : 'Calibri',
    'weight' : 'normal',
    'size'   : 12
}
matplotlib.rc('font', **font)
matplotlib.rc('text', usetex=True)

#fig.savefig("simulation.pdf")
```

```python
# relative error for energy conservation
energy = evaluate_functional(x, E, solution, functionals, params)
abs(energy[0] - energy[-1]) / energy[0]
```

```python
import matplotlib
import matplotlib.animation as anim

def animate(solution, functionals, params, file):
    plt.ioff()

    r2 = 2 * params[r]
    w2 = params[w] / 2
    l2 = params[l] - w2

    cmap = matplotlib.cm.get_cmap('YlOrRd')

    fig, ax = plt.subplots(dpi=200)
    ax.set_xlim(left=0, right=params[l])
    ax.set_ylim(bottom=-params[r], top=params[r])

    q0 = params[l]/2
    vol1 = plt.Rectangle((0, -params[r]), q0-w2, r2, fc='b')
    ax.add_patch(vol1)
    pist = plt.Rectangle((q0-w2, -params[r]), params[w], r2, fc='#C84843')
    ax.add_patch(pist)
    vol2 = plt.Rectangle((q0+w2, -params[r]), l2-q0, r2, fc='g')
    ax.add_patch(vol2)

    sol_q = evaluate_functional(x, q, solution, functionals, params)
    sol_θ_1 = evaluate_functional(x, θ_1, solution, functionals, params)
    sol_θ_2 = evaluate_functional(x, θ_2, solution, functionals, params)

    θ_min = numpy.min([numpy.min(sol_θ_1), numpy.min(sol_θ_2)])
    θ_max = numpy.max([numpy.max(sol_θ_1), numpy.max(sol_θ_2)])
    θ_swing = θ_max - θ_min

    data = numpy.block([sol_q.reshape(-1,1), sol_θ_1.reshape(-1,1), sol_θ_2.reshape(-1,1)])

    def animate(datum):
        q, θ_1, θ_2 = datum

        l_1 = q - w2
        l_2 = l2 - q

        vol1.set_width(l_1)
        pist.set_x(l_1)
        vol2.set_x(l_1 + params[w])
        vol2.set_width(l_2)

        vol1.set_fc(cmap(0.8 * ((θ_1-θ_min) / θ_swing)))
        vol2.set_fc(cmap(0.8 * ((θ_2-θ_min) / θ_swing)))

        return (vol1, pist, vol2)

    animation = anim.FuncAnimation(fig, animate, frames=data, blit=True, repeat=False)
    animation.save(file, fps=20, extra_args=['-vcodec', 'libx264'])
    plt.close(fig)
```

```python
%time animate(solution[:4000], functionals, params, 'piston.mp4')
```

    CPU times: user 24.4 s, sys: 1.47 s, total: 25.9 s
    Wall time: 27.2 s