##### Last modified: March 5th 2021 (Opal)

# Cross-Correlation

### Background
Cross-correlation is a mathematical operation that is commonly used in signal processing and pattern recognition to measure similarity of two functions as they are displaced from one another. For two finite discrete signals $f, g \in \mathbb{C}^{n}$, the uni-variate discrete *circular cross-correlation* ($\star$) is defined as

\begin{equation}\label{cc-dis-circular-definition}
    (f \star g)[\tau] \equiv \sum_{j=0}^{n-1} \overline{f[j]} g[(j + \tau)_{\text{mod} n}]
\end{equation}

where $\overline{f}$ denotes the complex conjugate of $f$, the bracket $[j]$ denotes the $j^{th}$ element of the signal, and $\tau \in \mathbb{Z}$ is the discrete displacement. In general, discrete circular cross-correlation can be extended to the multi-variate case for snapshots with $k \in \mathbb{Z}$ variables:

\begin{equation}\label{cc-dis-definition-multi-variate}
(f \star \overset {k}{\cdots } \star g)[\overline{\tau}] \equiv \sum_{j_{1}=-\infty}^{\infty} ... \sum_{j_{k}=-\infty}^{\infty}
\overline{f[j_{1}, ..., j_{k}]} g[(j_{1}+ \tau_{1})_{\text{mod} n_{1}}, (j_{2}+ \tau_{2})_{\text{mod} n_{2}}, ..., (j_{k} + \tau_{k})_{\text{mod} n_{k}}]
\end{equation}

where $\overline{\tau}= [\tau_{1}, \tau_{2},... ,\tau_{k}] \in \mathbb{Z}^{k}$ is the multi-variate discrete displacement. The optimal shift between the two discrete signals, $\bar{\tau}^{*}$, is found by the maximum of the cross-correlation, such that

\begin{equation}\label{cc-max}
    \bar{\tau}^{*} = {\text{argmax}_{\bar{\tau} \in \mathbb{Z}^{k}}} {(f \star \overset {k}{\cdots } \star g)[\overline{\tau}]}
\end{equation}

Once the optimal cross-correlation is found for all training snapshots by applying circular discrete cross-correlation between each snapshot and the initial condition, the shift function $\mathbf{c}(t)$ is found via polynomial curve fitting between the time increments $\left[t_{0}, t_{1}, ..., t_{f} \right]$ and corresponding optimal shift $\left[\tau^{\star}_{0}, \tau^{\star}_{1}, ..., \tau^{\star}_{f} \right]$.  The wave speed at the testing regime is approximated by extrapolating $\mathbf{c}(t)$ beyond the training regime.

### Motivation to use to find spatial shifting in convective-dominated problems
When the functions correlate, the value of $(f \ast g)$ is maximized. This is because when peaks and troughs align, they make a positive contribution to the finite sum.

### Circular convolution theorem and cross-correlation theorem
Given sequences $x$ and $y_{N}$, where $y_{N}$ is N-periodic, then the cross-correlation of $x$ and $y_{N}$ is given by
$$
x \ast y_{N} \equiv \mathcal{F}^{-1}\{ \overline{\mathcal{F}\{x\}} \cdot \mathcal{F}{\{y\}\}}
$$

where $\mathcal{F}$ denotes the Fourier transform, and $\overline{\mathcal{F}\{x\}}$ indicates the complex conjugate of $\mathcal{F}\{x\}$. This is analogous to the convolution theorem.

### In this notebook, we apply the circular discrete cross-correlation to:
1. MHD 1D SLICES. $\checkmark$
2. MHD 2D SLICES. $\checkmark$
3. HUX 1D SLICES. (A comparison of technqiues) $\checkmark$

```python
import os
curr_dir = os.getcwd()
os.chdir("../")
print("current directory path = ", os.getcwd())
```

    current directory path =  /Users/oissan/PycharmProjects/Space-Weather-ROM2

```python
import numpy as np
import scipy.linalg as la
import scipy.sparse as sparse
from scipy import signal
import scipy
from scipy.ndimage import correlate
from numpy.fft import fft, ifft
from scipy.interpolate import interp1d
from scipy.integrate import solve_ivp
from tools.MASweb import get_mas_path
from psipy.model import MASOutput
from psipy.data import sample_data
from psipy.model.variable import Variable
import rom_operator_inference as roi
from scipy import stats
import numpy as np
import time
import astropy.units as u
```

```python
os.chdir(os.getcwd() + "/code/")
from numerical_methods import apply_numerical_method
from hux_propagation import apply_hux_f_model
os.chdir("../")
```

```python
import matplotlib.pyplot as plt
import matplotlib
font = {'family' : 'serif',
        'size'   : 15}

matplotlib.rc('font', **font)
matplotlib.rc('xtick', labelsize=15)
matplotlib.rc('ytick', labelsize=15)
```

# Import MHD dataset

```python
# mas output - br, rho, vr
mas_path = get_mas_path(cr="2210")
model = MASOutput(mas_path)
print(model.variables)
```

    ['br', 'rho', 'vr']

```python
# read in MHD velocity results.
vr_model = model['vr']
```

```python
# save MHD mesh coordinates Heliographic (rotating) Coordinate System (HG)
# phi - (0, 2pi)
p = vr_model.phi_coords
# sin(theta) - (-pi/2, pi/2)
t = vr_model.theta_coords
# 30 solar radii to approximately 1 AU
r = (vr_model.r_coords * u.solRad).to(u.AU)
# velocity profile
f = vr_model.data
print(f.shape)
```

    (128, 111, 140)

```python
# force periodicity on f
# since the last phi index is less than 2*pi, then we will append 2*pi to phi scale.
p = np.append(p, 2*np.pi)

# append the first slice p=0 to velocity matrix.
f = np.append(f,[f[0,:, :]],axis=0)
```

```python
r1, r2 = 0, 125
v1 = f[:, 55, r1]
v2 = f[:, 55, r2]
```

```python
fig, ax = plt.subplots(figsize=(15, 5))
_ = ax.plot(180*p/np.pi, v1, ls= "-", label="$v_{1}(\phi, $ r = "+ str(round(r[r1].value, 3)) + "AU)")
_ = ax.plot(180*p/np.pi, v2, ls="-.", label="$v_{2}(\phi$, r = " + str(round(r[r2].value, 3)) + "AU)")
_ = ax.set_xticks([0, 90, 180, 270, 360])
_ = ax.set_xlabel("$\phi$ (Deg.)")
_ = ax.set_ylabel("$v_{r}$(km/s)")
_ = plt.legend()
```

# Circular discrete cross-correlation

```python
def normalize(x):
    return (x - np.mean(x))/np.std(x)
```

```python
def periodic_corr_fft(x, y):
    """Periodic correlation, implemented using the FFT.
    x and y must be real sequences with the same length.
    """
    return ifft(fft(x).conj() * fft(y)).real

def periodic_corr_np(x, y):
    """Periodic correlation, implemented using np.correlate.

    x and y must be real sequences with the same length.
    """
    return np.correlate(x, np.hstack((y[1:], y)), mode='valid')
```

```python
res2 = periodic_corr_np(normalize(v1), normalize(v2))
res2 /= np.max(res2)
```

```python
fig, (ax_v1, ax_v2, ax_corr, ax_lap) = plt.subplots(4, 1, figsize=(10, 10))
ax_v1.plot(180*p/np.pi, v1)
ax_v1.set_title('$v_{1}$')
ax_v1.set_xlabel('Carrington Longitude (Deg.)')
ax_v1.set_ylabel('$v_{r}$ (km/s)')
ax_v1.set_xticks([0, 90, 180, 270, 360])

ax_v2.plot(180*p/np.pi, v2)
ax_v2.set_title('$v_{2}$')
ax_v2.set_xlabel('Carrington Longitude (Deg.)')
ax_v2.set_ylabel('$v_{r}$ (km/s)')
ax_v2.set_xticks([0, 90, 180, 270, 360])

ax_corr.scatter(180*p/np.pi, res2)
ax_corr.scatter(p[np.argmax(res2)]*180/np.pi, np.max(res2), c="r")
ax_corr.axvline(x=p[np.argmax(res2)]*180/np.pi, linestyle="--", c="r")

ax_corr.set_xticks([360, 270, 180, 90, 0, round(p[np.argmax(res2)]*180/np.pi)])
ax_corr.set_title('Cross-correlated signal')
ax_corr.set_xlabel('Lag Carrington Longitude (Deg.)')
ax_corr.set_ylabel("ZNCC")

if np.argmax(res2) < len(p):
    index_roll = len(p) - np.argmax(res2)
else:
    index_roll = np.argmax(res2) - len(p)
ax_lap.plot(180/np.pi*p, v1, c="r", linestyle="--", label="$v_{1}$")
ax_lap.plot(180/np.pi*p, v2, c="g", linestyle="-.", label="$v_{2}$")
ax_lap.scatter(180/np.pi*np.roll(p, shift=index_roll), v2, c="b", s=2, label="shifted $v_{2}$")

ax_lap.set_xticks([0, 90, 180, 270, 360])
ax_lap.set_title('shifted $v_{2}$')
ax_lap.set_xlabel('Carrington Longitude (Deg.)')
ax_lap.set_ylabel("$v_{r}$ (km/s)")

plt.legend()
fig.tight_layout()
```

# Apply cross-correlation to all Equator velocity profiles

```python
v1 = f[:, 55, 0]
shift_vec = [0]
for ii in range(1, len(r)):
    v2 = f[:, 55, ii]
    corr = periodic_corr_np(normalize(v1), normalize(v2))
    corr = corr/np.max(corr)
    shift_vec.append(p[np.argmax(corr)]*180/np.pi)
```

```python
fig, ax = plt.subplots(figsize=(7, 3))
# plot the shifted data.
_ = ax.plot(r, shift_vec, ls="-.")

# given only 50%. of the training data- can we extrapolate?
n = int(len(r)*0.5)
m, b = np.polyfit(r[:n].value, shift_vec[:n], 1)
_ = ax.plot(r, m*r.value + b, label="linear-fit")

_ = ax.set_xlabel("Heliocentric Distance (AU)")
_ = ax.set_ylabel("Longitude Shift (Deg.)")
_ = ax.set_title("Cross-Correlation Shifting \n Linear-fit: y = " + str(round(m, 2)) + "x" + str(round(b, 2)))
_ = plt.legend()
_ = plt.tight_layout()

plt.savefig("CrossCorrelation/figs/cross-correlation-shift-128np.png")
```

```python
V_new = np.zeros((len(p), len(r)))
V_new[:, 0] = f[:, 55, 0]

for jj in range(1, len(r)):
    f_interp = interp1d(p*180/np.pi, f[:,55, jj], fill_value="extrapolate", bounds_error=False)
    x_new = (p*180/np.pi - shift_vec[jj]) % (360)
    V_new[:, jj] = f_interp(x_new)
```

```python
def plot_heat_data(Z, title, sample_columns=np.arange(0, len(r), 15), variable="v", xlabel="$\phi$ (Deg.)", ylabel="$v_{r}$ ( km/s)",
                  save_fig=False, file_name=None):

    fig, ax = plt.subplots(figsize=(14, 4))
    color = iter(plt.cm.viridis_r(np.linspace(0, 1, len(sample_columns))))

    for j in sample_columns:
        x_all = np.concatenate([[0], Z[:,j], [0]])  # Pad results with boundary conditions.
        ax.plot(p*(180/np.pi), Z[:,j], color=next(color), label=str(variable) + rf"$(\phi,r$" + "=" +
                str(round(r[j].to(u.AU).value,2)) + " AU)")

    ax.set_xlabel(xlabel, fontsize=15)
    ax.set_xticks([0, 90, 180, 270, 360])
    ax.set_ylabel(ylabel, fontsize=15)
    ax.legend(loc=(1.05, .05))
    ax.spines["right"].set_visible(False)
    ax.spines["top"].set_visible(False)
    ax.tick_params(axis='both', which='major', labelsize=12)
    fig.suptitle(title, fontsize=15)

    if save_fig:
        fig.savefig(file_name, dpi=300, bbox_inches='tight')
```

```python
plot_heat_data(V_new, title="Shifted MHD at the Equator")
```

# Apply circular Cross-Correlation to 2D MHD Data

```python
v1 = f[:, :, r1]
v2 = f[:, :, r2]
```

```python
def periodic_corr_2d_longitude_only(x, y):
    """Periodic correlation, implemented using np.correlate.

    x and y must be real sequences with the same length.
    """
    return scipy.signal.correlate2d(x, np.concatenate((y[1:, :], y)), mode='valid')
```

```python
def periodic_corr_2d(x, y):
    """Periodic correlation, implemented using np.correlate.

    x and y must be real sequences with the same length.
    """
    return scipy.signal.correlate2d(x, np.concatenate((y[1:, :], y)), boundary="wrap", mode='same')
```

```python
res = periodic_corr_2d(normalize(v1), normalize(v2))
res/= np.max(res)
```

```python
res.shape
```

    (129, 111)

```python
indices = np.where(res == res.max())
```

```python
print("latitude shift = ", t[indices[1][0]])
```

    latitude shift =  -4.371139006309477e-08

```python
print("longitude shift = ", p[indices[0][0]]*180/np.pi)
```

    longitude shift =  45.00000125223908

```python
fig, ax = plt.subplots(nrows=3, sharex=True, figsize=(6, 9))

ax[0].imshow(v1.T,
                extent=[p[0]*180/np.pi, p[-1]*180/np.pi, t[0]*180/np.pi, t[-1]*180/np.pi],
                vmin = 300, vmax = 700, cmap="plasma", aspect="auto", origin="lower")
ax[0].set_title("(a) $V_{30R_{S}}$")
im = ax[1].imshow(v2.T,
                extent=[p[0]*180/np.pi, p[-1]*180/np.pi, t[0]*180/np.pi, t[-1]*180/np.pi],
                vmin = 300, vmax = 700, cmap="plasma", aspect="auto", origin="lower")

ax[1].set_title("(b) $V_{1 AU}$")

img = ax[2].imshow(res.T, extent=[p[0]*180/np.pi, p[-1]*180/np.pi, t[0]*180/np.pi, t[-1]*180/np.pi], origin="lower")
ax[2].scatter(p[indices[0][0]]*180/np.pi, t[indices[1][0]], c="r")

fig.subplots_adjust(right=0.8)
cbar_ax = fig.add_axes([0.85, 0.13, 0.03, 0.21])
plt.colorbar(img, cax=cbar_ax)
_ = cbar_ax.set_ylabel('ZNCC', rotation=90)

ax[2].set_xticks([360, 270, 180, 90, 0, round(p[indices[0][0]]*180/np.pi)])
ax[2].set_title("(c) $V_{30R_{S}} \star V_{1 AU}$")

ax[2].set_xlabel(r"Longitude (Deg.)", fontsize=14)
ax[0].set_ylabel(r"$\theta$ (Deg.)", fontsize=14)
ax[1].set_ylabel(r"$\theta$ (Deg.)", fontsize=14)
ax[2].set_ylabel(r"$\theta$ (Deg.)", fontsize=14)
ax[2].set_ylabel(r"$\theta$ (Deg.)", fontsize=14)
ax[0].set_yticks([-90, -45, 0, 45, 90])
ax[1].set_yticks([-90, -45, 0, 45, 90])
ax[2].set_yticks([-90, -45, 0, 45, 90])

fig.subplots_adjust(right=0.8)
cbar_ax = fig.add_axes([0.85, 0.45, 0.03, 0.4])
fig.colorbar(im, cax=cbar_ax)

_ = cbar_ax.set_ylabel('km/s', rotation=90)

plt.savefig("CrossCorrelation/figs/2d-cc.png" , bbox_inches="tight", dpi=600)
```

```python
fig, ax = plt.subplots(figsize=(5, 3))
pos = ax.imshow(v1.T,
                extent=[p[0]*180/np.pi, p[-1]*180/np.pi, t[0]*180/np.pi, t[-1]*180/np.pi],
                vmin = 300, vmax = 700, cmap="plasma", aspect="auto", origin="lower")
#ax.set_title("(a) $V_{30R_{S}}$")

cbar = fig.colorbar(pos, ax=ax)
cbar.ax.set_ylabel('km/s', rotation=90)
ax.set_xticks([360, 270, 180, 90, 0])
ax.set_xlabel(r"Longitude (Deg.)", fontsize=14)
ax.set_ylabel(r"$\theta$ (Deg.)", fontsize=14)
ax.set_yticks([90, 45, 0, -45, -90])

plt.savefig("CrossCorrelation/figs/cc_30rs.png" , bbox_inches="tight", dpi=500)

```

```python
fig, ax = plt.subplots(figsize=(5, 3))

pos = ax.imshow(v2.T,
                extent=[p[0]*180/np.pi, p[-1]*180/np.pi, t[0]*180/np.pi, t[-1]*180/np.pi],
                vmin = 300, vmax = 700, cmap="plasma", aspect="auto", origin="lower")

cbar = fig.colorbar(pos, ax=ax)
cbar.ax.set_ylabel('km/s', rotation=90)
ax.set_xticks([360, 270, 180, 90, 0])
ax.set_yticks([-90, -45, 0, 45, 90])
ax.set_xlabel(r"Longitude (Deg.)", fontsize=14)
ax.set_ylabel(r"$\theta$ (Deg.)", fontsize=14)

plt.savefig("CrossCorrelation/figs/cc_1AU.png" , bbox_inches="tight", dpi=500)
```

```python
fig, ax = plt.subplots(figsize=(5, 3))

pos = ax.imshow(res.T, extent=[p[0]*180/np.pi, p[-1]*180/np.pi, t[0]*180/np.pi, t[-1]*180/np.pi], aspect="auto", origin="lower")
ax.scatter(p[indices[0][0]]*180/np.pi, t[indices[1][0]], marker="x", c="r", label="maximum")

cbar = fig.colorbar(pos, ax=ax)
cbar.ax.set_ylabel('ZNCC', rotation=90)

ax.set_xticks([360, 270, 180, 90, 0, round(p[indices[0][0]]*180/np.pi)])
ax.set_ylabel(r"$\theta$ (Deg.)", fontsize=14)

ax.set_yticks([-90, -45, 0, 45, 90])
ax.set_xlabel(r"Longitude (Deg.)", fontsize=14)

ax.set_yticks([-90, -45, 0, 45, 90])

plt.savefig("CrossCorrelation/figs/cc_subfigure.png" , bbox_inches="tight", dpi=500)
```

# Apply Cross-Correlation to all 2D slices

```python
v1 = f[:, :, 0]

shift_vec_2d = np.zeros((2, len(r)))

for ii in range(1, len(r)):
    v2 = f[:, :, ii]
    corr = periodic_corr_2d(normalize(v1), normalize(v2))
    corr = corr/np.max(corr)
    indices = np.where(corr == corr.max())
    shift_vec_2d[:, ii] = [p[indices[0][0]]*180/np.pi, t[indices[1][0]]*180/np.pi]
```

```python
fig, ax = plt.subplots(figsize=(10, 3))
ax.plot(r, shift_vec_2d[1, :], label="latitude-lag")
ax.plot(r, shift_vec_2d[0, :],  label="longitude-lag")
ax.set_ylabel("Lag (Deg.)")
ax.set_xlabel("Heliocentric Distance (AU)")
_ = ax.legend()
```

```python
fig, ax = plt.subplots(figsize=(7, 3))
# plot the shifted data.
_ = ax.plot(r, shift_vec_2d[0, :], ls="-.", label="2D")
_ = ax.plot(r, shift_vec, ls="-.", label="1D")

# given only 50%. of the training data- can we extrapolate?
n = int(len(r)*0.5)
m, b = np.polyfit(r[:n].value, shift_vec_2d[0, :n], 1)
_ = ax.plot(r, m*r.value + b, label="linear-fit (2D)")

m, b = np.polyfit(r[:n].value, shift_vec[:n], 1)
_ = ax.plot(r, m*r.value + b, label="linear-fit (1D)")

_ = ax.axvline(r[n].value, c="k")

_ = ax.set_xlabel("Heliocentric Distance (AU)")
_ = ax.set_ylabel("Longitude Shift (Deg.)")
_ = ax.set_title("Cross-Correlation Shifting (MHD)")
_ = plt.legend()
```

# Cross-Correlation to HUX dataset (compare with method of characteristics)

```python
HUX_V = apply_hux_f_model(f[:, 55, 0], (r[1:]-r[:-1]).to(u.km).value, p[1:] -p[:-1]).T
```

```python
plot_heat_data(HUX_V, title="HUX")
```

# Cross-Correlation for all slices in comparison to the initial condition

```python
v1 = HUX_V[:, 0]
shift_vec_hux = [0]
for ii in range(1, len(r)):
    v2 = HUX_V[:, ii]
    corr = periodic_corr_np(normalize(v1), normalize(v2))
    corr = corr/np.max(corr)
    shift_vec_hux.append(p[np.argmax(corr)]*180/np.pi)
```

```python
fig, ax = plt.subplots(figsize=(7, 3))
# plot the shifted data.
_ = ax.plot(r, shift_vec_hux, ls="-.", label="HUX")
_ = ax.plot(r, shift_vec, ls="-.", label="MHD-1D")

# given only 50%. of the training data- can we extrapolate?
n = int(len(r)*0.5)
m, b = np.polyfit(r[:n].value, shift_vec_hux[:n], 1)
_ = ax.plot(r, m*r.value + b, label="linear-fit (HUX)")

_ = ax.axvline(r[n].value, c="k")

# given only 50%. of the training data- can we extrapolate?
n = int(len(r)*0.5)
m, b = np.polyfit(r[:n].value, shift_vec[:n], 1)
_ = ax.plot(r, m*r.value + b, label="linear-fit (MHD-1D)")

_ = ax.set_xlabel("Heliocentric Distance (AU)")
_ = ax.set_ylabel("Longitude Shift (Deg.)")
_ = ax.set_title("Cross-Correlation Shifting")
_ = plt.legend()
```

# Method of Characteristics vs. Cross-Correlation

```python
def compute_phi_shift_forward(p, r, v, omega=(2 * np.pi) / (25.38 * 86400), method=None):
    # initialize phi shift matrix.
    phi_shift_mat = np.zeros((len(r), len(p)))

    # phi at index 0 is original phi grid
    phi_shift_mat[0, :] = np.zeros(len(p))

    # delta r.
    dr = np.mean(r[1:] - r[:-1])

    # compute the phi shift for each idx in r.
    for ii in range(len(r)-1):
        if method == "ballistic":
            phi_shift = -(omega/v[:, 0])*dr
        else:
            phi_shift = -(omega/v[:, ii])*dr
        phi_shift_mat[ii+1, :] = phi_shift_mat[ii, :] + phi_shift

    return phi_shift_mat
```

```python
phi_shift = compute_phi_shift_forward(p, r.to(u.km).value, HUX_V, method="ballistic")
```

```python
peak_b = 60#85
peak_e = 110#110
```

```python
print(p[peak_b]*180/np.pi)
print(p[peak_e]*180/np.pi)
```

    168.75000042702834
    309.375003059615

```python
avg_shift_vec = np.zeros(len(r))

V_new[:, 0] = HUX_V[:, 0]
for jj in range(1, len(r)):
    avg_shift = np.mean(phi_shift[jj, peak_b: peak_e]) #+np.std(phi_shift[jj, :])
    avg_shift_vec[jj] = avg_shift
```

```python
fig, ax = plt.subplots(figsize =(10, 3))
ax.plot(r.to(u.AU), -avg_shift_vec*180/np.pi, c="g", ls="--", label="Method of Characteristics")
ax.scatter(r.to(u.AU), shift_vec_hux, s=2, c="r", label="Cross-Correlation Extrapolation ")

# given only 70%. of the training data- can we extrapolate?
n = int(len(r)*0.7)
m, b = np.polyfit(r[:n].value, shift_vec_hux[:n], 1)
print(m, b)
_ = ax.plot(r, m*r.value + b,  c="r", label="CC linear-fit")

_ = ax.axvline(r[n].value, c="k")

_ = ax.set_xlabel("Heliocentric Distance (AU)")
_ = ax.set_ylabel("Longitude Shift (Deg.)")
_ = ax.set_title("A Comparison of Techniques")
_ = plt.legend()
_ = plt.tight_layout()

plt.savefig("CrossCorrelation/figs/comparison-of-techniques-cc-np128.png")
```

```python

```

```python

```


## Workshop 8: Bayesian Inference with Regression

*Note:* Chrome displays a vertical bar after all maths expressions. This is not intentional, it's a display bug. If this annoys you, please use a different browser.

In this workshop we will be looking at Bayesian linear regression. Briefly, this involves learning a linear regression model from a training set of *(x, y)* pairs, where *x* is a vector representing a data point and *y* is a real-valued response variable. Earlier we looked at ridge regression, which involved:
1. assuming a linear relationship between inputs and outputs, i.e., $y \approx \mathbf{x}' \mathbf{w}$ for all pairs
1. minimising the *residual sum of squares error*, that is finding the parameters $\hat{\mathbf{w}}$ that give the best fit to the training responses (subject to added *regularisation* term)
1. using $\hat{\mathbf{w}}$ to make test inferences

Today we will look at *Bayesian* inference for the above model. In this case we don't follow steps 2 and 3 above, but rather formulate the *posterior* over the parameters, and make test inferences using *all settings of the parameters* weighted by their posterior probability. These operations can be solved exactly, using linear algebra.

### (Desperately) seeking posterior

Before diving into the implementation, let's recap the maths behind Bayesian linear regression. We assume the data was generated from a Normal distribution, with its mean a linear function of the input vector and constant variance $\sigma^2$ (assumed known, herein).

$$ \newcommand\ys{\mathbf{y}}
\newcommand\xs{\mathbf{x}}
\newcommand\Xs{\mathbf{X}}
\newcommand\ws{\mathbf{w}}
\newcommand\Vs{\mathbf{V}}
\newcommand\Is{\mathbf{I}}
\begin{align*}
y &\sim \textrm{Normal}(\xs' \ws, \sigma^2) & \mbox{Likelihood}\\
\ws &\sim \textrm{Normal}(\mathbf{0}, \gamma^2 \mathbf{I}_D) & \mbox{Prior}
\end{align*}$$

The prior over the weights encourages low-magnitude weights, with parameter $\gamma^2$ controlling the variance.

Given this formulation, we next step is to find an expression for the posterior

$$
\begin{align*}
p(\ws | \Xs, \ys, \sigma^2) = \frac{ p(\ys | \Xs, \ws, \sigma^2) p(\ws) }{ p(\ys | \Xs) } \propto p(\ys | \Xs, \ws, \sigma^2) p(\ws)
\end{align*}
$$

where $\Xs$ are the training points (a matrix) and $\ys$ are a vector of the response values for each training point.

Recall that this can be solved, resulting in a Normal distribution for the posterior. We'll come back to this later on.

## Data set

We'll generate a simple synthetic dataset with a tiny handful of points drawn from a simple quadratic function,
$$ y = 5(x-\frac{1}{2})^2 $$
with a small amount of added noise.

```python
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```python
# generate a few points
N = 5
# each x is a scalar, uniformly drawn between [0, 1]
X = np.sort(np.random.random(size=N))
# reshape X into a matrix, such that each instance is a row
X = X[:,np.newaxis]

#Fixed Run#3 for more clear uncertainty plot
#X2 = [ [0.18615756], [0.59675711] ,[0.67832224] ,[0.70492722] ,[0.89709031]]
#X = np.array(X2)

print(X)

# generate the target response values using the quadratic function
# and additive noise
sigma2 = 0.1**2
y = 5*(X-0.5)**2 + sigma2**0.5 * np.random.normal(size=(N,1))

# plot the training data
plt.plot(X, y, 'bo')

# and plot the true function (without noise)
Xp = np.arange(-0.2, 1.2, 0.01)
Xp = Xp[:,np.newaxis]
yp_gold = 5*(Xp-0.5)**2
plt.plot(Xp, yp_gold, 'g')
```

### Polynomial basis functions

We'll fit this data with a polynomial model, considering a few different orders. Recall the basis function trick whereby you can augment the features of the input with new columns to implement a richer model class. Here we'll create a *basis matrix* for a $d^{th}$ order polynomial as follows

$$ \Phi = \begin{bmatrix}
    1 & x_1 & x_1^2 & x_1^3 & \ldots & x_1^d \\
    1 & x_2 & x_2^2 & x_2^3 & \ldots & x_2^d \\
    \vdots &    \vdots &    \vdots &\vdots &    \ddots & \vdots \\
    1 & x_n & x_n^2 & x_n^3 & \ldots & x_n^d \\
\end{bmatrix} $$

Observe how each row is a training point raised to a different power, from *0* to *d*. This means that when we take a row from $\Phi$ and take the dot product with $\mathbf{w}$, a parameter vector of size *d+1*, we obtain

$$ \begin{align}
\Phi_{i} \mathbf{w} &= \sum_{j=0}^d \Phi_{ij} w_j \\
&= 1 \times w_0 + x_i \times w_1 + x_i^2 \times w_2 + \ldots + x_i^d \times w_d
\end{align}$$

I.e., a $d^{th}$ order polynomial expression with $\mathbf{w}$ giving the coefficients.

```python
def polynomial_basis(X, Xtest, d):
    # ... Over to you

    return Phi, Phi_test

Phi, Phi_p = polynomial_basis(X, Xp, 9)
print (Phi.shape)
```

**Discussion**: How does this basis trick relate to kernel methods?

### Computing the posterior

Returning to the maths, the next step is to formulate the posterior. We plug in the Normal defintions of the likelihood and prior to solve for the unnormalised posterior

$$
\begin{align*}
p(\ws | \Xs, \ys, \sigma^2) & \propto \textrm{Normal}(\ws | \mathbf{0}, \gamma^2 \mathbf{I}_D)
\textrm{Normal}(\ys | \Xs \ws, \sigma^2 \mathbf{I}_N) \\
& \propto  \textrm{Normal}(\ws |  \ws_N, \mathbf{V}_N) \\
\mbox{where} ~ \ws_N &= \frac{1}{\sigma^2} \Vs_N \Phi' \ys \\
\Vs_N &= \sigma^2 ( \Phi' \Phi + \frac{\sigma^2}{\gamma^2} \Is_D )^{-1}
\end{align*}
$$

which is (proportional to) a Normal distribution. As we know the normalising term for a Gaussian [the denominator of the PDF](https://en.wikipedia.org/wiki/Multivariate_normal_distribution#Non-degenerate_case), this means we have an exact normalised solution for the posterior, i.e.,  $p(\ws | \Xs, \ys, \sigma^2) = \textrm{Normal}(\ws |  \ws_N, \mathbf{V}_N)$.

Let's compute the mean vector and covariance matrix for the posterior based on the above expression.

```python
gamma2 = 100 # large number = more permissive, prior is broader
VN = # ... Over to you
wN = # ... Over to you
print(wN.shape, VN.shape)
```

Next, we can plot the prior and posterior to see how they differ; given we have several weights, we'll just look at a couple of these, $w_0$ and $w_1$.

```python
# set up a 2d plot mesh
plt.figure(1)
delta = 0.05
xpl = np.arange(-3, 3, delta)
ypl = np.arange(-3, 3, delta)
Xpl, Ypl = np.meshgrid(xpl, ypl)

# plot a bivariate normal for the prior
Zpl = mlab.bivariate_normal(Xpl, Ypl, gamma2 ** 0.5,gamma2 ** 0.5, 0, 0, 0)
CS = plt.contour(Xpl, Ypl, Zpl)
plt.clabel(CS, inline=1, fontsize=10)
plt.title('Prior p(w) indices 0,1')

# plot a bivariate normal for the posterior
plt.figure(2)
Zpl = mlab.bivariate_normal(Xpl, Ypl, VN[0,0] ** 0.5, VN[1,1] ** 0.5, wN[0], wN[1], VN[0,1])
CS = plt.contour(Xpl, Ypl, Zpl) #, levels=[0.001, 0.1, 0.25, 0.5, 0.75, 1, 1.5, 2, 2.5])
plt.clabel(CS, inline=1, fontsize=10)
plt.title('Posterior p(w|X,y) indices 0,1')
plt.plot(wN[0], wN[1], 'rx')
```

**Discussion question**: Can you explain why the prior and the posterior are so different? How is this related to the dataset? Why are the elipses in the posterior not aligned to the axes? *You might want to change the parameter indices from 0,1 to other pairs to get a better idea of the full posterior.*

## Tim: 'w's are not independent anymore??

### Bayesian inference

The conceptually simplest means of doing inference is to draw a few parameter vectors from the posterior (sampling from a Gaussian). Let's do this 10 times and plot the results.

```python
# plot the data points
plt.plot(X, y, 'o')
# plot the mean prediction
yp = np.dot(Phi_p, wN)
plt.plot(Xp, yp, 'g-')

# generate some samples from the posterior
samples = []
ps = []
for s in range(10):
    # draw a weight vector
    w_sampled = np.random.multivariate_normal(wN.flatten(), VN, 1).flatten()
    # plot the predictions for this weight vector
    yp_sampled = np.dot(Phi_p, w_sampled)
    p = plt.plot(Xp.flatten(), yp_sampled.flatten(), ':', lw=3)
    samples.append(w_sampled)
    ps.append(p[0])

plt.ylim(-1,2)
```

It's interesting to see what happens near the data points, and away from them, in particular the edges of the plot. We'll come back to this.

But there's a more elegant solution, as the predictive distribution can be found in closed form. Namely

$$
\begin{align*}
p(y_{*} | \xs_{*}, \Xs, \ys, \sigma^2) &= \textrm{Normal}(y_{*} | \xs_{*}'\ws_N, \sigma^2_N(\xs_{*})) \\
\sigma^2_N(\xs_{*}) & = \sigma^2 + \xs_{*}' \Vs_N \xs_{*}
\end{align*}
$$

Note that the predictive mean is a simple application of the posterior mean to the data point, but the predictive variance is  a bit more complicated. Let's compute the mean and variance on the test points, and visualise the results.

```python
# compute the mean from above
yp = # ... Over to you

# compute the variance using formula above
s2s = np.zeros(Xp.shape[0])
for i in range(Xp.shape[0]):
    s2s[i] = # ... Over to you

# plot the data
plt.plot(X, y, 'o')
# plot the 95% confidence interval
plt.fill_between(Xp.flatten(), yp+2*s2s ** 0.5, yp-2*s2s ** 0.5, facecolor='blue', alpha=0.1)
# plot the mean as a green dotted line
plt.plot(Xp, yp, 'g:')
# plot the ground truth in red
plt.plot(Xp, yp_gold, 'r')

plt.xlabel('x')
plt.ylabel('y')
plt.ylim(-2,5)
```

**Discussion**: How does the uncertainty plot compare to the samples above? How does the uncertainty change relative to the distance from training points? Can you explain why?

**Practical**: How does the setting of *gamma2* affect the fit? How about the number of data points, *N*. Try some other values and see what happens.

**Discussion**: Is a 9th order polynomial a good choice for this problem? Based on the results above, would you recommend this model, or make a different choice?

### Bayesian model selection

Now let's revisit the assumption of having a 9th order polynomial. The evidence gives us a good way of evaluating the quality of the fit.

We can compute the evidence, $p(\ys|\Xs, \sigma^2, \gamma^2)$ <s>based on slide 31 of deck 14 (note the notation key, for mapping between Bishop to Murphy notation)</s>. This is also known as the *marginal likelihood* and is given in Bishop 3.5.1, p167 as equation 3.86. We won't delve into the details, but broadly speaking the $|\mathbf{A}|$ term penalises model complexity, while the $E$ term ($E(\mathbf{m}_N)$ in Bishop) measures the quality of the fit to the data.

```python
def log_evidence(m, Phi, sigma2, gamma2):
    N, M = Phi.shape
    alpha, beta = 1/float(gamma2), 1/float(sigma2)
    rss = np.sum((y - np.dot(Phi, m)) ** 2)
    wpen = np.dot(m.T, m)
    E = beta/2.0 * rss + alpha/2.0 * wpen
    A = alpha * np.eye(M) + beta * np.dot(Phi.T, Phi)
    lE = M/2.0 * np.log(alpha) + N/2.0 * np.log(beta) - E \
        - 0.5 * np.log(np.linalg.det(A)) - N/2.0 * np.log(2.0 * np.pi)

    # return both the evidence, and the RSS term (the raw quality of fit)
    return lE.flatten()[0], rss

# what's the evidence for our 9th order model?
print(log_evidence(wN, Phi, sigma2, gamma2))
```

So what happens if we use a lower order model, e.g., a 3rd order model?

```python
Phi3, Phi_p3 = polynomial_basis(X, Xp, 3)
VN3 = # ... Copy from above
wN3 = # ... Copy from above

yp3 = # ... Copy from above
s2s = np.zeros(Phi_p3.shape[0])
for i in range(Phi_p3.shape[0]):

    s2s[i] = # ... Copy from above

plt.plot(X, y, 'o')
plt.fill_between(Xp.flatten(), yp3+2*s2s ** 0.5, yp3-2*s2s ** 0.5, facecolor='blue', alpha=0.1)
plt.plot(Xp, yp3, 'g:')
plt.plot(Xp, yp_gold, 'r')
plt.xlabel('x')
plt.ylabel('y')
```

**Discussion**: does that look like a better fit to you? Consider both the area [0,1] near the training points, and those outside this range.

Let's see what the *evidence* says, and compare this to the above result:

```python
# ... Over to you
```

The RSS has barely changed, but the evidence is much higher. We can look at various model orders to see which has the best *evidence* to perform Bayesian model selection:

```python
evs = []
gamma2 = 1000
for order in range(0,10):
    Phio, Phipo = polynomial_basis(X, Xp, order)
    Vo = # ... Copy from above
    wo = # ... Copy from above
    lEo = # ... Copy from above
    print('order', order, 'log evidence', lEo[0], 'rss', lEo[1])
    evs.append((order,) + lEo)
```

```python
#plot the above log evidence values against the model order
evs = np.vstack(evs)
plt.plot(evs[:,0], evs[:,1])
#plt.ylim(-100, -5)
```

**Discussion**: So which model class will be chosen? Is this a reasonable situation?

**Practical**: Rerun the code with a new random training set, or different values of *N*, such as 2 or 3 points or 20; the results may be different. Can you explain why the outcome might be different?


# Raumluftqualität 5.0

Für einen belüfteten Raum ist

$$
  k(t) = k_\infty + (k_0-k_\infty)\,\rm{e}^{-\beta\,t}
$$

Dieser exponentielle Verlauf nähert sich dem Wert $k_\infty$ immer stärker an, kann ihn theoretisch aber nicht erreichen. Das hat seinen Grund darin, dass die Exponentialfunktion (theoretisch) niemals den Wert 0 annehmen kann.

Beispiel: für $k_0=400\,\rm ppM$, $k_\infty=1000\,\rm ppM$, $\beta = 0.5$ und $t_0=0$ ergibt sich:

Die Luftwechselzahl ist $\beta = 0.5\,\rm\dfrac{1}{h}$. Der Kehrwert dieser Zahl wird als Zeitkonstante bezeichnet, es ist

$$
  t_\rm{k} = \dfrac{1}{\beta} = 2\,\rm{h}
$$

Wird die Zeitkonstante $t_\rm{k}$ in die Funktion $k(t)$ eingesetzt, so ergibt sich

$$
  k(t_\rm{k}) = k_\infty + (k_0-k_\infty)\,\rm{e}^{-1}
$$

Entsprechend ergibt sich nach $2, 3, \ldots n$ Zeitkonstanten:

$$
  k(n\,t_\rm{k}) = k_\infty + (k_0-k_\infty)\,\rm{e}^{-n}
$$

Hierfür schreibt man auch

$$
  \dfrac{k(n\,t_\rm{k}) -k_\infty}{k_0-k_\infty} = \rm{e}^{-n}
$$

Nach einer Zeitkonstanten ist die Differenz $k(t) -k_\infty$ auf den Wert $\frac{1}{e}\approx 0.368 = 36.8\%$ abgesunken.

Nach drei Zeitkonstanten sind es noch etwa $5\,\%$ und nach fünf Zeitkonstanten etwa 6.74 Promille.

```python
from sympy import *
init_printing()

from matplotlib import pyplot as plt, ticker as tk
%config InlineBackend.figure_format='retina' # nur für Macbook Pro nötig

import numpy as np
import pandas as pd

from fun_expr import Function_from_Expression as FE
```

```python
# Die Funktionsvariable definieren
t = Symbol('t')

# bekannte Konstanten
k_zul = 1000e-6
k_0 = k_au = 400e-6

beta = 0.5
t_0 = 0

# Die Funktion k(t) definieren
k = FE(t, k_zul + (k_0-k_zul)*exp(-beta*(t-t_0)),name='k')
k.equation()
```

$\displaystyle k\left(t\right) = 0.001 - 0.0006 e^{- 0.5 t}$

```python
lt = np.linspace(0,10) # 10 Stunden

df = pd.DataFrame(
    {
        't': lt,
        'k': 1e6*k.lambdified(lt), # k in ppM
    }
)

df.head().T
```

      0
      1
      2
      3
      4

      t
      0.0
      0.204082
      0.408163
      0.612245
      0.816327

      k
      400.0
      458.204384
      510.762517
      558.222127
      601.077808

```python
ax = df.plot(x='t',y='k',label='$k=k(t)$')
ax.axhline(1e6*k_zul,c='r')

ax.set(
    xlim=(0,10),xlabel='Zeit $t$ in Stunden',
    ylim=(0,1200),ylabel='Konzentration $k$ in ppM'
)

delta_t = 2 # 1/beta = 2
for i in range(0,10,delta_t):
    ax.plot([i,i+delta_t,i+delta_t],[1e6*v for v in [k(i),k_zul,k(i+delta_t)]],c='r',marker='o')

ax.grid();
plt.savefig('Nb_5_fig_1.pdf',bbox_inches='tight')
```


Solutions to the Introduction to Composite Materials module of MIT's Open Course Mechanics of Materials.

Other material properties listed here.

David Roylance. 3.11 Mechanics of Materials. Fall 1999. Massachusetts Institute of Technology: MIT OpenCourseWare, https://ocw.mit.edu. License: Creative Commons BY-NC-SA.

```python
import numpy as np
import sympy as sp
from sympy import init_printing
init_printing(use_latex=True)
import matplotlib.pyplot as plt
```

For problem #1, the longitudinal and transverse stiffnesses are calculated using the equations from the slab model.

$E_1 = V_f*E_f + V_m*E_m$

$\frac{1}{E_2} = \frac{V_f}{E_f} + \frac{V_m}{E_m}$

```python
# Problem 1
# for S-glass fibers
Ef = 85.5  # GPa
Vf = 0.7

# for the epoxy
Em = 3.5  # GPa
Vm = 1 - Vf

# from the slab model, the composite stiffnesses are
E1 = Vf*Ef + Vm*Em
E2 = 1/(Vf/Ef + Vm/Em)

print(f"The longitudinal stiffness: E1 = {E1:.1f} GPa")
print(f"The transverse stiffness: E2 = {E2:.1f} GPa")
```

    The longitudinal stiffness: E1 = 60.9 GPa
    The transverse stiffness: E2 = 10.6 GPa

In problem #2, the longitudinal stiffness of an E-glass nylon composite is plotted over a range of fiber volumes, $V_f$.

```python
# Problem 2
# for E-glass fibers
Ef = 72.4  # GPa
# for the nylon
Em = 3.0  # GPa

Vf = np.linspace(0, 1, 100, endpoint=True)
Vm = 1 - Vf
E1 = Vf*Ef + Vm*Em

plt.plot(Vf, E1)
plt.xlabel(r"$V_f$")
plt.ylabel(r"$E_1, GPa$")
plt.grid(True)
plt.show()
```

In problem #3, the longitudinal breaking tensile stress of an E-glass epoxy composite is plotted over a range of fiber volumes, $V_f$. Breaking stress is determined mostly by the fiber strength. Using the fiber breaking strain and composite stiffness we have:

$\sigma_b = \epsilon_{fb}*E_1 = \epsilon_{fb}*(V_f*E_f + V_m*E_m)$

At low fiber volumes its possible for the fibers to break and the matrix to hold the entire load, so the breaking stress in this region is descibed as:

$\sigma_b = V_m*\sigma_{mb}$

```python
# Problem 3
# for E-glass fibers
Ef = 72.4  # GPa
sigma_fb = 2.4  # Gpa, fiber breaking stress
epsilon_fb = 0.026  # breaking strain of the fiber

# for the epoxy
Em = 3.5  # GPa
sigma_mb = 0.045  # Gpa, matrix breaking stress
epsilon_mb = 0.04  # breaking strain of the matrix

Vf = np.linspace(0, 1, 100, endpoint=True)
Vm = 1 - Vf
E1 = Vf*Ef + Vm*Em
sigma1 = epsilon_fb*E1
sigma2 = Vm*sigma_mb
sigma = [max(s1, s2) for s1, s2 in zip(sigma1, sigma2)]

plt.plot(Vf, sigma1)
plt.plot(Vf, sigma2)
plt.xlabel(r"$V_f$")
plt.ylabel(r"$\sigma_b, GPa$")
plt.grid(True)
plt.show()
```

After plotting both breaking stress equations, it is cleat the breaking stress is only determined by the first equation.

Problem #4 asks for the greatest fiber packing volume fraction given optimal fiber packing. And assuming that the optimal packing is hexagonal packing, the fiber volume fraction is determined with the following equation:

$V_f = \left(\frac{\pi}{2\sqrt{3}}\right)*\left(\frac{r}{R}\right)^2$

Where $r$ is the fiber radius and $2*R$ is the spacing between fiber centers, which in an optimal pattern: $2*R = 2*r$ and the last term drops out of the equation.

```python
#Problem 4
Vf = (np.pi/(2*np.sqrt(3)))
print(f"The max fiber volume fraction = {Vf:.3f}")
```

    The max fiber volume fraction = 0.907

Problem #5 asks to show how the slab model is used to calculate the transverse stiffness of the composite: $\frac{1}{E_2} = \frac{Vf}{E_f} + \frac{V_m}{E_m}$

Some assumptions need to be made to reach this equation: first, the stress in the fiber and matrix are the same; and second, the deformation of the slab in the transverse direction is the sum of the fiber and matrix deformations:

$\epsilon_2*1 = \epsilon_f*V_f + \epsilon_m*V_m$

Deformation is $strain*length$, and length in the transverse direction of a unit slab is 1, and lengths for the fiber and matrix are equal to their volume fraction. See Figure 3 from the composites module, shown below, for how the the volume fractions add up to the unit length, $V_f + V_m = 1$.

The stress-strain relationship $\epsilon = \frac{\sigma}{E}$ is substituted into the equation, resulting in:

$\frac{\sigma_2}{E_2}*1 = \frac{\sigma_f}{E_f}*V_f + \frac{\sigma_m}{E_m}*V_m$

The first assumption that fiber and matrix stresses are equal to the composite transverse stress, $\sigma_2 = \sigma_f = \sigma_m$, allow for all the $\sigma$ terms to cancel out, resulting in the transverse stiffness equation.

$\frac{1}{E_2} = \frac{Vf}{E_f} + \frac{V_m}{E_m}$

```python

```


Table of Contents

1  First Foray Into Discrete/Fast Fourier Transformation

1.1  Correlation

1.2  Fourier Transformation

1.3  DFT In Action

1.4  Fast Fourier Transformation (FFT)

2  Reference

```python
# code for loading the format for the notebook
import os

# path : store the current path to convert back to it later
path = os.getcwd()
os.chdir(os.path.join('..', '..', 'notebook_format'))

from formats import load_style
load_style(css_style='custom2.css', plot_style=False)
```

```python
os.chdir(path)

# 1. magic for inline plot
# 2. magic to print version
# 3. magic so that the notebook will reload external python modules
# 4. magic to enable retina (high resolution) plots
# https://gist.github.com/minrk/3301035
%matplotlib inline
%load_ext watermark
%load_ext autoreload
%autoreload 2
%config InlineBackend.figure_format='retina'

import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from sklearn.model_selection import train_test_split

%watermark -a 'Ethen' -d -t -v -p numpy,pandas,sklearn,matplotlib
```

    Ethen 2018-12-13 16:10:17

    CPython 3.6.4
    IPython 6.4.0

    numpy 1.14.2
    pandas 0.23.0
    sklearn 0.19.1
    matplotlib 2.2.2

# First Foray Into Discrete/Fast Fourier Transformation

In many real-world applications, signals are typically represented as a sequence of numbers that are time dependent. For example, digital audio signal would be one common example, or the hourly temperature in California would be another one. In order to extract meaningful characteristics from these kind of data, many different transformation techniques have been developed to decompose it into simpler individual pieces that are much easier and compact to reason with.

**Discrete Fourier Transformation (DFT)** is one of these algorithms that takes a signal as an input and breaks it down into many individual frequency components. Giving us, the end-user, easier pieces to work with. For the digital audio signal, applying DFT gives us what tones are represented in the sount and at what energies.

Some basics of digital signal processing is assumed. The following link contains an excellent primer to get people up to speed. I feverishly recommend going through all of it if the reader is not pressed with time. [Blog: Seeing Circles, Sines, And Signals a Compact Primer On Digital Signal Processing](https://jackschaedler.github.io/circles-sines-signals/index.html)

## Correlation

Correlation is a widely used concept in signal processing. It must be noted that the definition of correlation here is slightly different from the definition we encounter in statistics. In the context of signal processing, correlation measures how similar two signals are by computing the dot product between the two. i.e. given two signals $x$ and $y$, the correlation of the two signal can be computed using:

\begin{align}
\sum_{n=0}^N x_n \cdot y_n
\end{align}

The intuition behind this is that if the two signals are indeed similar, then whenever $x_n$ is positive/negative then $y_n$ should also be positive/negative. Hence when two signals' sign often matches, the resulting correlation number will also be large, indicating that the two signals are similar to one another. It is worth noting that correlation can also take on negative values, a large negative correlation means that the signal is also similar to each other, but one is inverted with respect to the other.

```python
# create examples of two signals that are dissimilar
# and two that are similar to illustrate the concept

def create_signal(sample_duration, sample_freq, signal_type, signal_freq):
    """
    Create some signals to work with, e.g. if we were to sample at 100 Hz
    (100 times per second) and collect the data for 10 seconds, resulting
    in 1000 samples in total. Then we would specify sample_duration = 10,
    sample_freq = 100.

    Apart from that, we will also give the option of generating sine or cosine
    wave and the frequencies of these signals
    """
    raw_value = 2 * np.pi * signal_freq * np.arange(0, sample_duration, 1. / sample_freq)
    if signal_type == 'cos':
        return np.cos(raw_value)
    elif signal_type == 'sin':
        return np.sin(raw_value)
```

```python
# change default style figure and font size
plt.rcParams['figure.figsize'] = 8, 6
plt.rcParams['font.size'] = 12
plt.style.use('fivethirtyeight')

# dissimilar signals have low correlation
signal1 = create_signal(10, 100, 'sin', 0.1)
signal2 = create_signal(10, 100, 'cos', 0.1)
plt.plot(signal1, label='Sine')
plt.plot(signal2, label='Cosine')
plt.title('Correlation={:.1f}'.format(np.dot(signal1, signal2)))
plt.legend()
plt.show()
```

```python
# similar signals have high correlation
signal1 = create_signal(10, 100, 'sin', 0.1)
signal2 = create_signal(10, 100, 'sin', 0.1)
plt.plot(signal1, label='Sine 1')
plt.plot(signal2, label='Sine 2', linestyle='--')
plt.title('Correlation={}'.format(np.dot(signal1, signal2)))
plt.legend()
plt.show()
```

Correlation is one of the key concept behind DFT, because as we'll soon see, in DFT, our goal is to find frequencies that gives a high correlation with the signal at hand and a high amplitude of this correlation indicates the presence of this frequency in our signal.

## Fourier Transformation

Fourier Transformation takes a time-based signal as an input, measures every possible cycle and returns the overall cycle components (by cycle, we're essentially preferring to circles). Each cycle components stores information such as for each cycle:

- **Amplitude:** how big is the circle?
- **Frequency:** How fast is it moving? The faster the cycle component is moving, the higher the frequency of the wave.
- **Phase:** Where does it start, or what angle does it start?

This cycle component is also referred to as **phasor**. The following gif aims to make this seemingly abstract description into a concrete process that we can visualize.

After applying DFT to our signal shown on the right, we realized that it can be decomposed into five different phasors. Here, the center of the first phasor/cycle component is placed at the origin, and the center of each subsequent phasor is "attached" to the tip of the previous phasor. Once the chain of phasors is built, we begin rotating the phasor. We can then reconstruct the time domain signal by tracing the vertical distance from the origin to the tip of the last phasor.

Let's now take a look at DFT's formula:

\begin{align}
X_k = \sum_{n=0}^{N-1} x_n \cdot e^{ -\varphi \mathrm{i} }
\end{align}

- $x_n$: The signal's value at time $n$.
- $e^{-\varphi\mathrm{i}}$: Is a compact way of describing a pair of sine and cosine waves.
- $\varphi = \frac{n}{N} 2\pi k$: Records phase and frequency of our cycle components. Where $N$ is the number of samples we have. $n$ the current sample we're considering. $k$ the currenct frequency we're considering. The $2\pi k$ part represents the cycle component's speed measured in radians and $n / N$ measures the percentage of time that our cycle component has traveled.
- $X_k$ Amount of cycle component with frequency $k$.

> Side Note: If the readers are a bit rusty with trigonometry (related to sine and cosine) and complex numbers. e.g. There're already many excellent materials out there that covers these concepts. [Blog: Trigonometry Review](https://jackschaedler.github.io/circles-sines-signals/trig_review.html) and [Blog: Complex Numbers](https://jackschaedler.github.io/circles-sines-signals/complex.html)

From the formula, we notice that it's taking the dot product between the original signal $x_n$ and $e^{ -\varphi \mathrm{i} }$. If we expand $e^{ -\varphi \mathrm{i} }$ using the Euler's formula. $e^{ -\varphi \mathrm{i} } = cos(\varphi) - sin(\varphi)i$, we end up with the formula:

\begin{align}
X_k &= \sum_{n=0}^{N-1} x_n \cdot \big( cos(\varphi) - sin(\varphi)i \big) \\
    &= \sum_{n=0}^{N-1} x_n \cdot cos(\varphi) - i \sum_{n=0}^{N-1} x_n \cdot sin(\varphi)
\end{align}

By breaking down the formula a little bit, we can see that underneath the hood, what fourier transformation is doing is taking the input signal and doing 2 correlation calculations, one with the sine wave (it will give us the y coordinates of the circle) and one with the cosine wave (which will give us the x coordinates or the circle). And the following succinct one-sentence colour-coded explanation is also a great reference that we can use for quick reference.

## DFT In Action

To see DFT in action, we will create a dummy signal that will be composed of four sinusoidal waves of different frequencies. 0, 10, 2 and 0.5 Hz respectively.

```python
# reminder:
# sample_duration means we're collecting the data for x seconds
# sample_freq means we're sampling x times per second
sample_duration = 10
sample_freq = 100
signal_type = 'sin'
num_samples = sample_freq * sample_duration
num_components = 4

components = np.zeros((num_components, num_samples))
components[0] = np.ones(num_samples)
components[1] = create_signal(sample_duration, sample_freq, signal_type, 10)
components[2] = create_signal(sample_duration, sample_freq, signal_type, 2)
components[3] = create_signal(sample_duration, sample_freq, signal_type, 0.5)

fig, ax = plt.subplots(nrows=num_components, sharex=True, figsize=(12,8))
for i in range(num_components):
    ax[i].plot(components[i])
    ax[i].set_ylim((-1.1, 1.1))
    ax[i].set_title('Component {}'.format(i))
    ax[i].set_ylabel('Amplitude')

ax[num_components - 1].set_xlabel('Samples')
plt.tight_layout()
```

Then we will combine these individual signals together with some weights assigned to each signal.

```python
signal = -0.5 * components[0] + 0.1 * components[1] + 0.2 * components[2] - 0.6 * components[3]

plt.plot(signal)
plt.xlabel('Samples')
plt.ylabel('Amplitude')
plt.show()
```

By looking at the dummy signal we've created visually, we might be able to notice the presence of a signal which shows 5 periods in the sampling duration of 10 seconds. In other words, after applying DFT to our signal, we should expect the presence a signal with the frequency of 0.5 HZ.

Here, we will leverage numpy's implementation to check whether the result makes intuitive sense or not. The implementation is called `fft`, but let's not worry about that for the moment.

```python
fft_result = np.fft.fft(signal)
print('length of fft result: ', len(fft_result))
fft_result[:5]
```

    length of fft result:  1000

    array([-5.00000000e+02+0.00000000e+00j,  2.25951276e-13+8.46545056e-15j,
            1.28375093e-13+1.50003997e-13j,  5.19653030e-14+1.99840144e-15j,
           -4.90882687e-15+1.16994724e-13j])

The `fft` routine returns an array of length 1000 which is equivalent to the number of samples. If we look at each individual element in the array, we'll notice that these are the DFT coefficients. It has two components, the real number corresponds to the cosine waves and the imaginary number that comes from the sine waves. In general though, we don't really care if there's a cosine or sine wave present, as we are only concerned which frequency pattern has a higher correlation with our original signal. This can be done by considering the absolute value of these coefficients.

```python
plt.plot(np.abs(fft_result))
plt.xlim((-5, 120))  # notice that we limited the x-axis to 120 to focus on the interesting part
plt.ylim((-5, 520))
plt.xlabel('K')
plt.ylabel('|DFT(K)|')
plt.show()
```

If we plot the absolute values of the `fft` result, we can clearly see a spike at K=0, 5, 20, 100 in the graph above. However, we are often times more interested in the energy of of each frequency. **Frequency Resolution** is the distance in Hz between two adjacent data points in DFT, which is defined as:

\begin{align}
\Delta f = \frac{f_s}{N}
\end{align}

Where $f_s$ is the sampling rate and $N$ is the number of data points. The denominator can be expressed in terms of sampling rate and time, $N = f_s \cdot t$. Looking closely at the formula, it is telling us the only thing that increases frequency resolution is time.

In our case, the `sample_duration` we've specified above was 10, thus the frequencies corresponding to these K are: 0 Hz, 0.5 Hz, 2 Hz and 10 Hz respectively (remember that these frequencies were the components that was used in the dummy signal that we've created). And based on the graph depicted below, we can see that by passing our signal to a DFT, we were able to retrieve its underlying frequency information.

```python
t = np.linspace(0, sample_freq, len(fft_result))
plt.plot(t, np.abs(fft_result))
plt.xlim((-1, 15))
plt.ylim((-5, 520))
plt.xlabel('K')
plt.ylabel('|DFT(K)|')
plt.show()
```

## Fast Fourier Transformation (FFT)

Recall that the formula for Discrete Fourier Transformation was:

\begin{align}
X_k = \sum_{n=0}^{N-1} x_n \cdot e^{ -\frac{n}{N} 2\pi k \mathrm{i} }
\end{align}

Since we now know that it's computing the dot product between the original signal and a cycle component at every frequency, we can implement this ourselves.

```python
def dft(x):
    """Compute the Discrete Fourier Transform of the 1d ndarray x."""
    N = x.size
    n = np.arange(N)
    k = n.reshape((N, 1))

    # complex number in python are denoted by the j symbol,
    # instead of i that we're showing in the formula
    e = np.exp(-2j * np.pi * k * n / N)
    return np.dot(e, x)
```

```python
# apply dft to our original signal and confirm
# the results looks the same
dft_result = dft(signal)
print('result matches:', np.allclose(dft_result, fft_result))

plt.plot(np.abs(dft_result))
plt.xlim((-5, 120))
plt.ylim((-5, 520))
plt.xlabel('K')
plt.ylabel('|DFT(K)|')
plt.show()
```

However, if we compare the timing between our simplistic implementation versus the one from numpy, we can see a dramatic time difference.

```python
%timeit dft(signal)
%timeit np.fft.fft(signal)
```

    34.1 ms ± 898 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)
    24.8 µs ± 535 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

If we leave aside the fact that one is implemented using Python's numpy and one is most likely implemented in optimized C++, the time difference actually comes from the fact that in practice, people uses a more optimized version of Fourier Transformation called **Fast Fourier Transformation** (how unexpected ...) to perform the calculation. The algorithm accomplish significant speedup by exploiting symmetry property. i.e. if we devise a hypothetical algorithm which can decompose a 1024-point DFT into two 512-point DFTs, then we are essentially halving our computational cost. Let's take a look at how we can achieve this by looking at an example with 8 data points.

\begin{align}
X_k = x_0 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 0 } + x_1 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 } + \dots + x_7 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 7 }
\end{align}

Our goal is to examine the possibility of rewriting this eight-point DFT in terms of two DFTs of smaller length. Let's first examine choosing all the terms with an even sample index, i.e. $x_0$, $x_2$, $x_4$, and $x_6$. Giving us:

\begin{align}
G_k &= x_0 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 0 } + x_2 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 2 } + x_4 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 4 } + x_6 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 6 } \\
    &= x_0 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 0 } + x_2 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 1 } + x_4 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 2 } + x_6 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 3 }
\end{align}

After plugging the values for the even sample index and simplifying the fractions in the complex exponentials, we can observe that our $G_k$ is a 4 samples DFT with $x_0$, $x_2$, $x_4$, $x_6$ as our input signal. Now that we've shown that we can decompose the even index samples, let's see if we can simplify the remaining terms, the odd-index samples, are given by:

\begin{align}
Q_k &= x_1 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 } + x_3 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 3 } + x_5 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 5 } + x_7 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 7 } \\
    &= e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 } \cdot \big( x_1 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 0 } + x_3 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 2 } + x_5 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 4 } + x_7 \cdot e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 6 } \big) \\
    &= e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 } \cdot \big( x_1 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 0 } + x_3 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 1 } + x_5 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 2 } + x_7 \cdot e^{ -\mathrm{i} \frac{2\pi}{4} k ~\times~ 3 } \big) \\
    &= e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 } \cdot H_k
\end{align}

After the derivation, we can see our $Q_k$ is obtained by multiplying $e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 }$ by the four point DFT with the odd index samples of $x_1$, $x_3$, $x_5$, $x_7$, which we'll denote as $H_k$. Hence, we have achieved the goal of decomposing an eight-point DFT into two four-point ones:

\begin{align}
X_k &= G_k + e^{ -\mathrm{i} \frac{2\pi}{8} k ~\times~ 1 } \cdot H_k
\end{align}

We have only worked through rearranging the terms a bit, next we'll introduce a symmetric trick that allows us to compute the sub-result only once and save computational cost.

The question that we'll be asking ourselves is what is the value of $X_{N+k}$ is. From our above expression:

\begin{align}
X_{N + k} &= \sum_{n=0}^{N-1} x_n \cdot e^{-i~2\pi~(N + k)~n~/~N}\\
          &= \sum_{n=0}^{N-1} x_n \cdot e^{- i~2\pi~n} \cdot e^{-i~2\pi~k~n~/~N}\\
          &= \sum_{n=0}^{N-1} x_n \cdot e^{-i~2\pi~k~n~/~N}
\end{align}

Here we've used the property that $exp[2\pi~i~n] = 1$ for any integer $n$, since $exp[2\pi~i]$ means that we're going 1 full circle, and multiplying that number by any integer $n$ means we're spinning for $n$ circles. The last line shows a nice symmetry property of the DFT: $X_{N+k}=X_k$. This means when we break our eight-point DFT into two four-point DFTs, it allows us to re-use a lot of the results for both $X_k$ and $X_{k + 4}$ and significantly reduce the number of calculations through the symmetric property:

\begin{align}
X_{k + 4} &= G_{k + 4} + e^{ -\mathrm{i} \frac{2\pi}{8} (k + 4) ~\times~ 1 } \cdot H_{k + 4} \\
          &= G_k + e^{ -\mathrm{i} \frac{2\pi}{8} (k + 4) ~\times~ 1 } \cdot H_k
\end{align}

We saw that the starting point of the algorithm was that the DFT length $N$ was even and we were able to decrease the computation by splitting it into two DFTS of length $N/2$, following this procedure we can again decompose each of the $N/2$ DFTs into two $N/4$ DFTs. This property turns the original $\mathcal{O}[N^2]$ DFT computation into a $\mathcal{O}[N\log N]$ algorithm to compute DFT.

```python
def fft(x):
    N = x.shape[0]

    if N % 2 > 0:
        raise ValueError('size of x must be a power of 2')
    elif N <= 32:  # this cutoff should be enough to start using the non-recursive version
        return dft(x)
    else:
        fft_even = fft(x[0::2])
        fft_odd = fft(x[1::2])
        factor = np.exp(-2j * np.pi * np.arange(N) / N)
        return np.concatenate([fft_even + factor[:N // 2] * fft_odd,
                               fft_even + factor[N // 2:] * fft_odd])
```

```python
# here, we assume the input data length is a power of two
# if it doesn't, we can choose to zero-pad the input signal
x = np.random.random(1024)
np.allclose(fft(x), np.fft.fft(x))
```

    True

```python
%timeit dft(x)
%timeit fft(x)
%timeit np.fft.fft(x)
```

    36.7 ms ± 1.71 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)
    1.62 ms ± 32.9 µs per loop (mean ± std. dev. of 7 runs, 1000 loops each)
    27.6 µs ± 711 ns per loop (mean ± std. dev. of 7 runs, 10000 loops each)

By re-vising our algorithm that we use to compute DFT, we have improved our implementation by an order of magnitude! The algorithm we've implemented here is also referred to as the [radix-2 Cooley-Tukey FFT](https://en.wikipedia.org/wiki/Cooley%E2%80%93Tukey_FFT_algorithm).

Though the pure-Python functions are probably not useful in practice, as due to the importance of the FFT in so many applications, Both NumPy, `numpy.fft`, and SciPy, `scipy.fftpack` have wrappers of the extremely well-tested FFTPACK library. But I still believe that by spending the time to get an intuition of what's happening underneath the hood of these tools, the better practitioners we'll be.

Application of DFT in the context of equipment condition monitoring. [Blog: Fast Fourier Transforms in Python](https://ericstrong.org/fast-fourier-transforms-in-python/)

# Reference

- [Blog: Understanding the FFT Algorithm](https://jakevdp.github.io/blog/2013/08/28/understanding-the-fft/)
- [Blog: Dummies guide to Fourier Transform](https://nipunbatra.github.io/blog/2016/FT.html)
- [Blog: An Introduction to the Fast Fourier Transform](https://www.allaboutcircuits.com/technical-articles/an-introduction-to-the-fast-fourier-transform/)
- [Blog: An Intuitive Discrete Fourier Transform Tutorial](http://practicalcryptography.com/miscellaneous/machine-learning/intuitive-guide-discrete-fourier-transform/)
- [Blog: Understanding the Fourier Transform by example](https://www.ritchievink.com/blog/2017/04/23/understanding-the-fourier-transform-by-example/)
- [Blog: The Fourier Transform, explained in one sentence](https://blog.revolutionanalytics.com/2014/01/the-fourier-transform-explained-in-one-sentence.html)
- [Blog: Seeing Circles, Sines, And Signals a Compact Primer On Digital Signal Processing](https://jackschaedler.github.io/circles-sines-signals/index.html)


# 1st vs 2nd order Taylor methods

## Intial Value Poblem
The general form of the population growth differential equation
\begin{equation} y^{'}=t-y, \ \ (0 \leq t \leq 4), \end{equation}
with the initial condition
\begin{equation}x(0)=1, \end{equation}
For N=4
with the analytic (exact) solution
\begin{equation} y= 2e^{-t}+t+1. \end{equation}

## Taylor Solution

\begin{equation} f(t,y)=t-y,  \end{equation}
differentiate with respect to $t$,
\begin{equation} f'(t,y)=1-y'=1-t+y,  \end{equation}
This gives the first order Taylor,
\begin{equation}T^1(t_i,w,i)=f(t_i,w_i)=t_i-w_i, \end{equation}
and the second order Taylor,
\begin{equation}
T^2(t_i,w,i)=f(t_i,w_i)+\frac{h}{2}f'(t_i,w_i)=t_i-w_i+\frac{h}{2}(1-t_i+w_i).\end{equation}

The first order Taylor difference equation, which is identical to the Euler method, is

\begin{equation}
w_{i+1}=w_i+h(t_i-w_i).  \end{equation}
The second order Taylor difference equation is
\begin{equation}
w_{i+1}=w_i+h(t_i-w_i+\frac{h}{2}(1-t_i+w_i)).  \end{equation}

```python
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings

warnings.filterwarnings("ignore")

```

```python
def Second_order_taylor(N,IC):
    x_end=4
    x_start=0

    INTITIAL_CONDITION=IC
    h=x_end/(N)
    N=N+1;
    Numerical_Solution=np.zeros(N)
    Numerical_Solution_first=np.zeros(N)
    t=np.zeros(N)
    Analytic_Solution=np.zeros(N)
    Upper_bound=np.zeros(N)

    t[0]=x_start
    Numerical_Solution[0]=INTITIAL_CONDITION
    Numerical_Solution_first[0]=INTITIAL_CONDITION
    Analytic_Solution[0]=INTITIAL_CONDITION
    for i in range (1,N):
        Numerical_Solution_first[i]=Numerical_Solution_first[i-1]+h*(t[i-1]-Numerical_Solution_first[i-1])
        Numerical_Solution[i]=Numerical_Solution[i-1]+h*(t[i-1]-Numerical_Solution[i-1]+h/2*(1-t[i-1]+Numerical_Solution[i-1]))
        t[i]=t[i-1]+h
        Analytic_Solution[i]=2*math.exp(-t[i])+t[i]-1

    fig = plt.figure(figsize=(10,4))
    # --- left hand plot
    ax = fig.add_subplot(1,3,1)
    plt.plot(t,Numerical_Solution,color='blue',label='Second Order')
    plt.plot(t,Numerical_Solution_first,color='red',label='First Order')
    plt.legend(loc='best')
    plt.title('Numerical Solution h=%s'%(h))

    # --- right hand plot
    ax = fig.add_subplot(1,3,2)
    plt.plot(t,Analytic_Solution,color='blue')
    plt.title('Analytic Solution')

    #ax.legend(loc='best')
    ax = fig.add_subplot(1,3,3)
    plt.plot(t,np.abs(Analytic_Solution-Numerical_Solution),color='blue',label='Second Order Error')
    plt.plot(t,np.abs(Analytic_Solution-Numerical_Solution_first),color='red',label='First Order Error')
    plt.title('Error')
    plt.legend(loc='best')
    # --- title, explanatory text and save

    # --- title, explanatory text and save
    fig.suptitle(r"$y'=y-t$", fontsize=20)
    plt.tight_layout()
    plt.subplots_adjust(top=0.85)
```

```python
Second_order_taylor(40,1)
```

```python

```

Chapter 02 - Higher Order Methods/202_Taylor Method Error Example.ipynb


```python
%matplotlib inline
```

# Volume exclusion

This model is introduced in

S. Motsch, D. Peurichard, From short-range repulsion to Hele-Shaw problem in a model of tumor growth, *J. Math. Biology*, Vol. 76, No. 1, 2017.

First of all, some standard imports.

```python
import os
import sys
import time
import math
import torch
import numpy as np
from matplotlib import pyplot as plt
import sisyphe.models as models
from sisyphe.display import scatter_particles

use_cuda = torch.cuda.is_available()
dtype = torch.cuda.FloatTensor if use_cuda else torch.FloatTensor
```

## Repulsion force

Each particle is a disk with a (fixed) random radius. The particles repel each other when they overlap. The force exerted by a particle located at $x_j$ with radius $R_j$ on a particle located at $x_i$ with radius $R_i$ is

\begin{align}F = -\frac{\alpha}{R_i} \nabla_{x_i} U\left(\frac{|x_i - x_j|^2}{(R_i + R_j)^2}\right),\end{align}

where the potential is

\begin{align}U(s) = -\log(s) + s - 1\,\,\text{for}\,\, s<1 \,\,\text{and}\,\, U(s) = 0\,\, \text{for}\,\, s>1.\end{align}

Initially, the particles are clustered in a small region with a strong overlapping.

```python
N = 10000
rmin = .1
rmax = 1.
R = (rmax-rmin)*torch.rand(N).type(dtype)+rmin
L = 100.
D0 = 20.
pos = (D0*torch.rand((N,2)).type(dtype)-D0/2)+torch.tensor([L/2,L/2]).type(dtype)

dt = .1

simu = models.VolumeExclusion(pos=pos,
                 interaction_radius=R,
                 box_size=L,
                 alpha=2.5,
                 division_rate=0.,
                 death_rate=0.,
                 dt=dt)
```

Run the simulation over 200 units of time using an adaptive time-step which ensures that the energy :attr:`E` of the system decreases.

```python
# sphinx_gallery_thumbnail_number = 13

frames = [0,1,2,3,4,5,10,30,50,75,100,150,200]

s = time.time()
scatter_particles(simu,frames)
e = time.time()
```

Print the total simulation time and the average time per iteration.

```python
print('Total time: '+str(e-s)+' seconds')
print('Average time per iteration: '+str((e-s)/simu.iteration)+' seconds')
```

Note this funny behaviour: the particles are clustered by size!

## Repulsion force, random births and random deaths

Same system but this time, particles die at a constant rate and give birth to new particles at the same rate. A new particle is added next to its parent and has the same radius.

```python
N = 10000
rmin = .1
rmax = 1.
R = (rmax-rmin)*torch.rand(N).type(dtype)+rmin
L = 100.
D0 = 20.
pos = (D0*torch.rand((N,2)).type(dtype)-D0/2)+torch.tensor([L/2,L/2]).type(dtype)

dt = .1

simu = models.VolumeExclusion(pos=pos,
                 interaction_radius=R,
                 box_size=L,
                 alpha=2.5,
                 division_rate=.3,
                 death_rate=.3,
                 dt=dt,
                 Nmax = 20000)
```

Run the simulation over 200 units of time using an adaptive time-step which ensures that the energy :attr:`E <sisyphe.models.VolumeExclusion.E>` of the system decreases.

```python
frames = [0,1,2,3,4,5,10,30,50,75,100,150,200]

s = time.time()
scatter_particles(simu,frames)
e = time.time()
```

Print the total simulation time and the average time per iteration.

```python
print('Total time: '+str(e-s)+' seconds')
print('Average time per iteration: '+str((e-s)/simu.iteration)+' seconds')
```


# Nonlinear Equations

We want to find a root of the nonlinear function $f$ using different methods.

1. Bisection method
2. Newton method
3. Chord method
4. Secant method
5. Fixed point iterations

```python
%matplotlib inline
from numpy import *
from matplotlib.pyplot import *
import sympy as sym

```

```python
t = sym.symbols('t')

f_sym = t/8. * (63.*t**4 - 70.*t**2. +15.) # Legendre polynomial of order 5

f_prime_sym = sym.diff(f_sym,t)

f = sym.lambdify(t, f_sym, 'numpy')
f_prime = sym.lambdify(t,f_prime_sym, 'numpy')

phi = lambda x : 63./70.*x**3 + 15./(70.*x)
#phi = lambda x : 70.0/15.0*x**3 - 63.0/15.0*x**5
#phi = lambda x : sqrt((63.*x**4 + 15.0)/70.)

# Let's plot
n = 1025

x = linspace(-1,1,n)
c = zeros_like(x)

_ = plot(x,f(x))
_ = plot(x,c)
_ = grid()

```

```python
# Initial data for the variuos algorithms

# interval in which we seek the solution
a = 0.7
b = 1.

# initial points
x0 = (a+b)/2.0
x00 = b

```

```python
# stopping criteria
eps = 1e-10
n_max = 1000
```

## Bisection method

$$
x^k = \frac{a^k+b^k}{2}
$$
```
                   if (f(a_k) * f(x_k)) < 0:
                      b_k1 = x_k
                      a_k1 = a_k
                   else:
                      a_k1 = x_k
                      b_k1 = b_k
```

```python
def bisect(f,a,b,eps,n_max):
    assert f(a)*f(b) < 0
    a_new = a
    b_new = b
    x = mean([a,b])
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        if ( f(a_new) * f(x) < 0 ):
            # root in (a_new,x)
            b_new = x
        else:
            # root in (x,b_new)
            a_new = x

        x_new = mean([a_new,b_new])

        #err = 0.5 *(b_new -a_new)
        err = abs(f(x_new))
        #err = abs(x-x_new)

        errors.append(err)
        x = x_new
        it += 1

    semilogy(errors)
    print(it)
    print(x)
    print(err)
    return errors

errors_bisect = bisect(f,a,b,eps,n_max)

```

```python
# is the number of iterations coherent with the theoretical estimation?
```

In order to find out other methods for solving non-linear equations, let's compute the Taylor's series of $f(x^k)$ up to the first order

$$
f(x^k) \simeq f(x^k) + (x-x^k)f^{\prime}(x^k)
$$
which suggests the following iterative scheme
$$
x^{k+1} = x^k - \frac{f(x^k)}{f^{\prime}(x^k)}
$$

The following methods are obtained applying the above scheme where

$$
f^{\prime}(x^k) \approx q^k
$$

## Newton's method
$$
q^k = f^{\prime}(x^k)
$$

$$
x^{k+1} = x^k - \frac{f(x^k)}{q^k}
$$

```python
def newton(f,f_prime,x0,eps,n_max):
    err = abs(f(x0))
    errors = [err]
    it = 0
    x = x0
    while (err > eps and it < n_max):
        qk = f_prime(x)
        if abs(qk) < 1e-12:
            raise RuntimeError("f_prime(x) is close to zero")
        x_new = x - f(x)/qk
        err = abs(f(x_new))
        x = x_new
        errors.append(err)
        it += 1

    #print x
    #print err
    #print it
    semilogy(errors)
    return errors

%time errors_newton = newton(f,f_prime,1.0,eps,n_max)
```

## Chord method

$$
q^k \equiv q = \frac{f(b)-f(a)}{b-a}
$$

$$
x^{k+1} = x^k - \frac{f(x^k)}{q}
$$

```python
def chord(f,a,b,x0,eps,n_max):
    q = (f(b) - f(a))/(b-a)
    err = eps +1.
    errors = [err]
    it = 0
    x = x0
    while (err > eps and it <n_max):
        x_new = x - f(x)/q
        err = abs(x_new - x)
        errors.append(err)
        x = x_new
        it += 1

    print(x)
    print(err)
    print(it)
    semilogy(errors)
    return errors

errors_chord = chord (f,a,b,x0,eps,n_max)
```

## Secant method

$$
q^k = \frac{f(x^k)-f(x^{k-1})}{x^k - x^{k-1}}
$$

$$
x^{k+1} = x^k - \frac{f(x^k)}{q^k}
$$

Note that this algorithm requirs **two** initial points

```python
def secant(f,x0,x00,eps,n_max):
    err = eps +1.
    errors = [err]

    it = 0

    xk = x0
    xkk = x00
    while (err > eps and it < n_max):
        qk = (f(xk) - f(xkk))/(xk - xkk)
        x_new = xk - f(xk)/qk
        err = abs (x_new - xk)
        xkk = xk
        xk = x_new
        it += 1
        errors.append(err)

    print(xk)
    print(err)
    print(it)
    semilogy(errors)
    return errors

errors_secant = secant(f,x0,x00,eps,n_max)
```

## Fixed point iterations

$$
f(x)=0 \to x-\phi(x)=0
$$

$$
x^{k+1} = \phi(x^k)
$$

```python
def fixed_point(phi,x0,eps,n_max):
    x = x0
    err = eps + 1.
    errors = [err]
    it = 0
    while (err > eps and it < n_max):
        x_new = phi (x)
        err = abs (x_new -x)
        x = x_new
        it += 1
        errors.append(err)

    print(x)
    print(err)
    print(it)
    semilogy(errors)
    return errors

errors_fixed = fixed_point(phi,0.3,eps,n_max)

```

## Comparison

```python
# plot the error convergence for the methods
loglog(errors_bisect, label='bisect')
loglog(errors_chord, label='chord')
loglog(errors_secant, label='secant')
loglog(errors_newton, label ='newton')
loglog(errors_fixed, label ='fixed')
_ = legend()
```

```python
# Let's compare the scipy implmentation of Newton's method with our..
```

```python
import scipy.optimize as opt
%time opt.newton(f, 1.0, f_prime, tol = eps)
```

    CPU times: user 170 µs, sys: 6 µs, total: 176 µs
    Wall time: 179 µs

    0.90617984593866396

```python

```


```python
import sympy as sp
```

```python
p, rho, K, rho0 = sp.symbols("p rho K rho0")
```

```python
sp.diff(K * rho**sp.Rational("5/3")*(1 - (rho0/rho)**sp.Rational("1/3")), rho).simplify()
```

$\displaystyle \frac{K \rho^{\frac{2}{3}} \left(5 - 4 \sqrt[3]{\frac{\rho_{0}}{\rho}}\right)}{3}$

```python

```


# Hypersolvers for Optimal Control - Direct Optimal Control

```python
import sys; sys.path.append(2*'../') # go n dirs back
from src import *
```

```python
import time
import torch
import torch.nn as nn
import matplotlib.pyplot as plt

from torchdyn.core import NeuralODE
from torchdyn.datasets import *
from torchdyn.numerics import odeint, Euler, HyperEuler
```

## Optimal Control

We want to control an inverted pendulum and stabilize it in the upright position. The equations in Hamiltonian form describing an inverted pendulum with a torsional spring are as following:

$$\begin{equation}
    \begin{bmatrix} \dot{q}\\ \dot{p}\\ \end{bmatrix} =
    \begin{bmatrix}
    0& 1/m \\
    -k& -\beta/m\\
    \end{bmatrix}
    \begin{bmatrix} q\\ p\\ \end{bmatrix} -
    \begin{bmatrix}
    0\\
    mgl \sin{q}\\
    \end{bmatrix}+
    \begin{bmatrix}
    0\\
    1\\
    \end{bmatrix} u
\end{equation}$$

```python
class ControlledPendulum(nn.Module):
    """
    Inverted pendulum with torsional spring
    """
    def __init__(self, u, m=1., k=.5, l=1., qr=0., β=.01, g=9.81):
        super().__init__()
        self.u = u # controller (nn.Module)
        self.nfe = 0 # number of function evaluations
        self.cur_f = None # current function evaluation
        self.cur_u = None # current controller evaluation
        self.m, self.k, self.l, self.qr, self.β, self.g = m, k, l, qr, β, g # physics

    def forward(self, t, x):
        self.nfe += 1
        q, p = x[..., :1], x[..., 1:]
        self.cur_u = self.u(t, x)
        dq = p/self.m
        dp = -self.k*(q - self.qr) - self.m*self.g*self.l*torch.sin(q) \
            -self.β*p/self.m + self.cur_u
        self.cur_f = torch.cat([dq, dp], -1)
        return self.cur_f
```

In order to control the pendulum, we have to define a proper _integral cost function_ which will be our loss to be minimized during training. In a general form, it can be defined as:

\begin{equation}
        J = x^\top(t_f)\mathbf{P} x(t_f) + \int_{t_0}^{t_f} \left[ x^\top(t) \mathbf{Q} x(t) + u^\top(t) \mathbf{R} u(t) \right] dt
\end{equation}

where $ x = \begin{bmatrix} q\\ p\\ \end{bmatrix}$ is the state and $\mathbf{u}$ is the controller and matrices $\mathbf{P},~\mathbf{Q}, ~ \mathbf{R}$ are weights for controlling the performance.

```python
class IntegralCost(nn.Module):
    '''Integral cost function
    Args:
        x_star: torch.tensor, target position
        P: float, terminal cost weights
        Q: float, state weights
        R: float, controller regulator weights
    '''
    def __init__(self, x_star, P=0, Q=1, R=0):
        super().__init__()
        self.x_star = x_star
        self.P, self.Q, self.R, = P, Q, R

    def forward(self, x, u=torch.Tensor([0.])):
        """
        x: trajectory
        u: control input
        """
        cost = self.P*torch.norm(x[-1] - self.x_star, p=2, dim=-1).mean()
        cost += self.Q*torch.norm(x - self.x_star, p=2, dim=-1).mean()
        cost += self.R*torch.norm(u - 0, p=2).mean()
        return cost
```

```python
# Change device according to your configuration
# device = torch.device('cuda:1') if torch.cuda.is_available() else torch.device('cpu')
device = torch.device('cpu') # feel free to change :)
```

```python
# The controller is a simple MLP with one hidden layer with bounded output
class NeuralController(nn.Module):
    def __init__(self, model, u_min=-20, u_max=20):
        super().__init__()
        self.model = model
        self.u_min, self.u_max = u_min, u_max

    def forward(self, t, x):
        x = self.model(x)
        return torch.clamp(x, self.u_min, self.u_max)

model = nn.Sequential(nn.Linear(2, 32), nn.Tanh(), nn.Linear(32, 1)).to(device)
u = NeuralController(model)
for p in u.model[-1].parameters(): torch.nn.init.zeros_(p)

# Controlled system
sys = ControlledPendulum(u).to(device)
```

```python
from math import pi as π

# Loss function declaration
x_star = torch.Tensor([0., 0.]).to(device)
cost_func = IntegralCost(x_star)

# Time span
dt = 0.2
t0, tf = 0, 3 # initial and final time for controlling the system
steps = int((tf - t0)/dt) + 1 # so we have a time step of 0.2s
t_span = torch.linspace(t0, tf, steps).to(device)

# Initial distribution
x0 = π # limit of the state distribution (in rads and rads/second)
init_dist = torch.distributions.Uniform(torch.Tensor([-x0, -x0]), torch.Tensor([x0, x0]))
```

```python
# We consider the controller fixed during each solver step
class RandConstController(nn.Module):
    def __init__(self):
        super().__init__()
        self.u0 = torch.Tensor(1024, 1).uniform_(-10,10).to(device)

    def forward(self, t, x):
        return self.u0

# Save previously learned controller
u_no_hypersolver = sys.u
sys.u = RandConstController() # modify controller for training
```

```python
class VanillaHyperNet(nn.Module):
    """Simple hypernetwork for controlled systems
    Input: current x, f and u from the controlled system
    Output: p-th order residuals"""
    def __init__(self, net):
        super().__init__()
        self.net = net

    def forward(self, t, x):
        xfu = torch.cat([x, sys.cur_f, sys.cur_u], -1)
        return self.net(xfu)

net = nn.Sequential(nn.Linear(5, 32), nn.Softplus(), nn.Linear(32, 32), nn.Tanh(), nn.Linear(32, 2))
hypersolver = HyperEuler(VanillaHyperNet(net))
# model = nn.DataParallel(hypersolver, device_ids=[1]) # feel free to change here according to your setup and GPU available.
# model = model.to(device)
```

```python
model = torch.load('saved_models/hs_torchdyn.pt').to(device)
hypersolver = model.module
```

## Optimal Control with Hypersolvers
We can use the trained hypernetwork to generate trajectories of the pendulum: then, we cast the same optimal control problem defined at the beginning of the notebook and train the neural controller.

```python
sys = ControlledPendulum(None)
```

```python
# Reinstantiate controller
# net = nn.Sequential(nn.Linear(2, 64), nn.Softplus(), nn.Linear(64, 64), nn.Tanh(), nn.Linear(64, 1)).to(device)
# u = NeuralController(net)
# for p in u.model[-1].parameters(): torch.nn.init.zeros_(p)
u = BoxConstrainedController(2, 1, constrained=True, num_layers=3, output_scaling=torch.Tensor([-5, 5]).to(device))
sys.u = u

# Time span
t0, tf = 0, 3 # initial and final time for controlling the system
steps = int((tf - t0)/dt) + 1 # so we have a time step of 0.2s
t_span = torch.linspace(t0, tf, steps).to(device)

# Initial distribution
x0 = π # limit of the state distribution (in rads and rads/second)
init_dist = torch.distributions.Uniform(torch.Tensor([-x0, -x0]), torch.Tensor([x0, x0]))
```

```python

u_min, u_max = -5, 5 # constraints for the controller
cost_func = IntegralCost(x_star, P=1, Q=1) # final position is more important

# Hyperparameters
lr = 3e-3
epochs = 1000
bs = 1024
```

### Euler

```python
# Reinstantiate controller
# sys = ControlledPendulum(u)
u = BoxConstrainedController(2, 1, constrained=True, num_layers=3, output_scaling=torch.Tensor([-5, 5]).to(device))
sys.u = u.to(device)
opt = torch.optim.Adam(u.parameters(), lr=lr)

# Training loop
t0 = time.time(); losses=[]
for e in range(epochs):
    x0 = init_dist.sample((bs,)).to(device)
    _, trajectory = odeint(sys, x0, t_span, solver='euler', atol=1e-7, rtol=1e-7)
    loss = cost_func(trajectory); losses.append(loss.detach().cpu().item())
    loss.backward(); opt.step(); opt.zero_grad()
    print('Loss {:.4f} , epoch {}'.format(loss.item(), e), end='\r')
timing = time.time() - t0; print('\nTraining time: {:.4f} s'.format(timing))

u_euler = sys.u
```

    /home/botu/anaconda3/lib/python3.8/site-packages/torchdyn/numerics/odeint.py:82: UserWarning: Setting tolerances has no effect on fixed-step methods
      warn("Setting tolerances has no effect on fixed-step methods")

    Loss 5.2740 , epoch 999
    Training time: 28.8481 s

### HyperEuler

```python
# Reinstantiate controller
u = BoxConstrainedController(2, 1, constrained=True, num_layers=3, output_scaling=torch.Tensor([-5, 5]).to(device))
sys.u = u.to(device)
opt = torch.optim.Adam(u.parameters(), lr=lr)

# Training loop
# Time should be taken with gpu, neural network inference is slower on cpu
t0 = time.time(); losses=[]
for e in range(epochs):
    x0 = init_dist.sample((bs,)).to(device)
    _, trajectory = odeint(sys, x0, t_span, solver=hypersolver)
    loss = cost_func(trajectory); losses.append(loss.detach().cpu().item())
    loss.backward(); opt.step(); opt.zero_grad()
    print('Loss {:.4f} , epoch {}'.format(loss.item(), e), end='\r')
timing = time.time() - t0; print('\nTraining time: {:.4f} s'.format(timing))

u_hyper = sys.u
```

    Loss 1.1993 , epoch 999
    Training time: 40.9267 s

### Midpoint

```python
# Reinstantiate controller
u = BoxConstrainedController(2, 1, constrained=True, num_layers=3, output_scaling=torch.Tensor([-5, 5]).to(device))
sys.u = u.to(device)
opt = torch.optim.Adam(u.parameters(), lr=lr)

# Training loop
t0 = time.time(); losses=[]
for e in range(epochs):
    x0 = init_dist.sample((bs,)).to(device)
    _, trajectory = odeint(sys, x0, t_span, solver='midpoint')
    loss = cost_func(trajectory); losses.append(loss.detach().cpu().item())
    loss.backward(); opt.step(); opt.zero_grad()
    print('Loss {:.4f} , epoch {}'.format(loss.item(), e), end='\r')
timing = time.time() - t0; print('\nTraining time: {:.4f} s'.format(timing))

u_mp = sys.u
```

    Loss 1.1765 , epoch 999
    Training time: 54.4334 s

### RK4

```python
# Reinstantiate controller
u = BoxConstrainedController(2, 1, constrained=True, num_layers=3, output_scaling=torch.Tensor([-5, 5]).to(device))
sys.u = u.to(device)
opt = torch.optim.Adam(u.parameters(), lr=lr)

# Training loop
t0 = time.time(); losses=[]
for e in range(epochs):
    x0 = init_dist.sample((bs,)).to(device)
    _, trajectory = odeint(sys, x0, t_span, solver='rk4')
    loss = cost_func(trajectory); losses.append(loss.detach().cpu().item())
    loss.backward(); opt.step(); opt.zero_grad()
    print('Loss {:.4f} , epoch {}'.format(loss.item(), e), end='\r')
timing = time.time() - t0; print('\nTraining time: {:.4f} s'.format(timing))

u_rk4 = sys.u
```

    Loss 1.0977 , epoch 999
    Training time: 109.5863 s

```python
# Saving learned controllers
torch.save(u_euler, 'saved_models/u_euler.pt')
torch.save(u_hyper, 'saved_models/u_hyper.pt')
torch.save(u_mp, 'saved_models/u_mp.pt')
torch.save(u_rk4, 'saved_models/u_rk4.pt')
```


```python
import random
import sympy as sm
import numpy as np
import matplotlib.pyplot as plt
import ipywidgets as widgets
from scipy import optimize

#setting nice printing and seed
sm.init_printing(use_unicode=True)
random.seed(12848174)
```

```python
rho = sm.symbols('rho')
beta = sm.symbols('beta')
gamma = sm.symbols('gamma')
w = sm.symbols('w')
b = sm.symbols('b')
delta = sm.symbols('Delta')
h1 = sm.symbols('h_1')
h2 = sm.symbols('h_2')
c1 = sm.symbols('c1')
c2 = sm.symbols('c2')
l1 = sm.symbols('l1')
l2 = sm.symbols('l2')
exp = sm.symbols('E[v_2(h_2)]')
u = sm.symbols("u")
```

```python
u2 = (c2**(1-rho))/(1-rho)
dis_u2 = gamma*l2
total_u2 = u2 - dis_u2
h2_f = (h1 + l1)
c2_f = w*h2*l2
total_u2
```

```python
total_u2_sub2 = total_u2.subs(c2,c2_f)
#foc2 = sm.diff(total_u2_sub2,l2)
#foc2
total_u2_sub2
```

```python
#we should not solve for l2 rigth? this is just a dummy of work or not
#sol2 = sm.solve(sm.Eq(foc2,0),l2)[0]
#sol2
h2_sol = sm.solve(sm.Eq(total_u2_sub2,u),h2)[0]
h2_sol
```

```python
def utility_work(h2,l2,w,rho,gamma):
    solution = (((h2*l2*w)**(1-rho))/(1-rho))-gamma*l2
    return solution

def utility_notwork(benefits,rho,gamma,l2):

    b = ((benefits**(1-rho))/(1-rho))-gamma*l2
    return b
def hum_cap_u(l2,gamma,b,w,rho):
    # i use h2_sol but replace u with the "cutoff"-value, that is the value of unemployment benefits
    nw_u = utility_notwork(b,rho,gamma,0)
    hum_cap = ((-gamma*l2*rho+gamma*l2-rho*nw_u+nw_u)**-(1/(rho-1)))/(l2*w)

    return hum_cap
```

```python
rho = 2
beta = 0.96
gamma = 0.1
w = 2
b = 1
delta = 0.1
```

```python
#I calculatet the humam capital given b
hum_cap_u(1,gamma,b,w,rho)
```

```python
def plot_func(h2,w,rho,gamma,b):

    work = utility_work(h2,1,w,rho,gamma)
    not_work = utility_notwork(b,rho,gamma,0)
    cutoff = hum_cap_u(1,gamma,b,w,rho)

    plt.plot(h2,work,label = "Utility of working")
    plt.plot(h2,not_work,color="red",label="Utility of not working")
    plt.legend()

    plt.xlabel("Human capital")
    plt.ylabel("Utility")

    plt.show

    print("The worker will supply labour if human capital is above",round(cutoff[0],2), "and no labour if it is below" )
    return
```

```python
h_vec = np.linspace(0.1,1.5,100)
b_vec = np.linspace(1,1,100)
plot_func(h_vec,w,rho,gamma,b_vec)
```

```python
#Another way of showing this is using a binary plot. This requires us to define
#each level om human capital as either a capital where the worker provies labour
# or a level where she does not - the cutoff is as shown before
cutoff = hum_cap_u(1,gamma,b,w,rho)
print(round(cutoff,2),"human capital")
```

    0.56 human capital

```python
#We define a list
willwork = list(range(100))

#And fill it with either success (1) if the human capital i above the cutoff or
#or falier (0) if it is on the cutoff or below
for i in range(len(h_vec)):
    if h_vec[i] > cutoff :
        willwork[i] = 1
    else :
        willwork[i] = 0
```

```python
cutoff_vec = np.linspace(cutoff,cutoff,100)
plt.plot(h_vec,willwork)
plt.plot(cutoff_vec,willwork,linestyle="dashed",color="red",label=("Cutoff human capital level"))
plt.legend()
plt.xlabel("Human capital")
plt.ylabel("Labour supply")

plt.show
```

some text here

```python
#the labour supply as a function of human captal is then:
def labour_supply(hum_cap,w,rho,gamma,b):
    cutoff = hum_cap_u(1,gamma,b,w,rho)

    if hum_cap > cutoff :
        print("with a human capital level of",hum_cap,"the worker wil supply 1 labour (l = 1)")
        l = 1
    else :
        print("with a human capital level of",hum_cap,"the worker will supply 0 labour (l=0)")
        l = 0
    return l
```

```python
human_capital = widgets.FloatSlider(min=0.1,max=1.5,value=0.1,step=0.01)
widgets.interact(labour_supply,hum_cap=human_capital,w=widgets.fixed(w),rho=widgets.fixed(rho),gamma=widgets.fixed(gamma),b=widgets.fixed(b))
```

    interactive(children=(FloatSlider(value=0.1, description='hum_cap', max=1.5, min=0.1, step=0.01), Output()), _…

    <function __main__.labour_supply(hum_cap, w, rho, gamma, b)>

Interactive solution to question 1

```python

```

## Question 2

```python
rho = sm.symbols('rho')
beta = sm.symbols('beta')
gamma = sm.symbols('gamma')
w = sm.symbols('w')
b = sm.symbols('b')
delta = sm.symbols('Delta')
h1 = sm.symbols('h_1')
h2 = sm.symbols('h_2')
c1 = sm.symbols('c1')
c2 = sm.symbols('c2')
l1 = sm.symbols('l1')
l2 = sm.symbols('l2')
exp = sm.symbols('E[v_2(h_2)]')
u = sm.symbols("u")
```

```python
u1 = (c1**(1-rho))/(1-rho)
dis_u1 = gamma*l1
total_u1 = u1 - dis_u1 + beta*exp
total_u1
```

```python
# I define funcitons for period 2
def utility_2(h1,w=1,l1=1,rho=2,gamma=0.1,beta=0.96,delta=0.1,b=1):
    if l1 == 1:
        h2 = h1+1 +(0.5*delta)
        c = w*h1*1

        u_p2 = utility_work(h2,1,w,rho,gamma)

        solution = ((c**(1-rho))/(1-rho))-gamma*1 +beta*u_p2
    else :
        h2 = h1 + 0 + (0.5*delta)
        c = b

        u_nw_p2 = utility_notwork(b,rho,gamma,0)
        solution = ((c**(1-rho))/(1-rho))-gamma*0 +beta*u_nw_p2

    return solution

```

```python
h1 = 0.1
utility_2(h1,w,1,rho,gamma,beta,delta,b)
```

```python
rho = 2
beta = 0.96
gamma = 0.1
w = 2
b = 1
delta = 0.1
h_vec = np.linspace(0.1,1.5,100)

```

```python
b_vec = np.linspace(1,1,100)
x = utility_2(h_vec,w,1,rho,gamma,beta,delta,b)
y = utility_2(h_vec,w,0,rho,gamma,beta,delta,b_vec)

```

```python
#The utility of not working is
y[0]
```

```python
#The cutof point can be found using a numerical optimizer

rho = 2
beta = 0.96
gamma = 0.1
w = 2
b = 1
delta = 0.1

#because it is negative values we minimize (and not -minimize)
def objective(h):
    return utility_2(h,w,1,rho,gamma,beta,delta,b)
def constraint(h):
    return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
```

```python
cons = ({"type": "ineq","fun":constraint})
h_init = 0.1

```

```python
sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
sol
```

         fun: -1.9599999941585289
         jac: array([4.28598835])
     message: 'Optimization terminated successfully.'
        nfev: 21
         nit: 7
        njev: 7
      status: 0
     success: True
           x: array([0.35172508])

```python
#The equilibrium value is
sol.x[0]
```

```python
cutoff = np.linspace(sol.x[0],sol.x[0],100)
plt.plot(h_vec,x, label = "Utility of working",alpha= 0.8)
plt.plot(h_vec,y, color="green", label = "Utility of not working",alpha= 0.8)
plt.plot(cutoff,x,color = "red", linestyle = "dashed",label="Labour cutoff",alpha= 0.4)
plt.scatter(cutoff[0],y[0],color="purple",alpha = 1)
plt.xlabel("Human capital level")
plt.ylabel("Utility")
plt.legend()
plt.show()
```

```python
willwork_p1 = list(range(100))

#And fill it with either success (1) if the human capital i above the cutoff or
#or falier (0) if it is on the cutoff or below
for i in range(len(h_vec)):
    if utility_2(h_vec[i],w,1,rho,gamma,beta,delta,1) > utility_2(h_vec[i],w,0,rho,gamma,beta,delta,1) :
        willwork_p1[i] = 1
    else :
        willwork_p1[i] = 0
```

```python
plt.plot(h_vec,willwork_p1)
plt.plot(cutoff,np.linspace(0,1,100),linestyle = "dashed",color="red")
```

This means that the worker will supply all her labour (1) if her human capital level i above 0.35, but 0 if her human capital level is below this.

## Question 3

```python

```

```python
#itinerary
## make st of w's with 0.1,0.2 and so on til 0.9
### calculate utility for those
#### see if any of them are higher that b

```

We will examine if workers will ever work if their wage is lower than the unemployment benefits. In this case the unemployment benefits are given as 1. It is theoretically posible that a worker would chose to work, if the acumulation of human capital can more than make up for the loss in utility.

```python
#We define a list of different wage levels to examnie
wages = np.linspace(0,0.99,100)
#We have chosen these for the sake of interpertation
```

```python
rho = 2
beta = 0.96
gamma = 0.1
b = 1
delta = 0.1
h_vec = np.linspace(0.1,1.5,100)

```

```python
#This for loop test if the utility gained from working, even if wages are lover than b, is higher than -1.96 - the utility of
#not working in both periods
for wage in wages:
    name = "wage_" + str(wage)
    cutoff_w = utility_2(h1 = h_vec, w = wage, l1 = 1, rho = 2, gamma = 0.1, beta= 0.96, delta = 0.1, b = 1)
    if max(cutoff_w) > -1.96:
        print(name)
        #This will be the lowest wage where it is profitable to work
        break
```

    wage_0.6

    C:\Users\augus\Anaconda3\lib\site-packages\ipykernel_launcher.py:2: RuntimeWarning: divide by zero encountered in reciprocal

    C:\Users\augus\Anaconda3\lib\site-packages\ipykernel_launcher.py:9: RuntimeWarning: divide by zero encountered in reciprocal
      if __name__ == '__main__':

```python
#It tursout that it is utility profitable for the worker to work even if the wage is 0.6 while the unemployment benefits are 1
## Now we will show this
```

```python
cutoff = np.linspace(sol.x[0],sol.x[0],100)
plt.plot(h_vec,x, label = "Wage = 2",alpha= 0.6)
plt.plot(h_vec,y, color = "red", linestyle = "dashed",alpha= 0.5, label = "Utility of not working")
plt.plot(h_vec,cutoff_w, color = "green", label = "Wage = 0.6")
plt.xlabel("Human capital level")
plt.ylabel("Utility")
plt.ylim(-5,0)
plt.legend()
plt.show()
```

```python
#It is evident that at very high levels of human capital, it can be rational to work, even if the wage is lower than the
#unemployment benefits. From our loop before we know that it is the lowest wage where this is true is 0.6. Therefore we
# use optimization to find the human capital level of at which i makes sense to work if your wage is 0.6.

```

```python

```

```python
rho = 2
beta = 0.96
gamma = 0.1
w = 0.6
b = 1
delta = 0.1

#because it is negative values we minimize (and not -minimize)
def objective(h):
    return utility_2(h,w,1,rho,gamma,beta,delta,b)
def constraint(h):
    return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
```

```python
cons = ({"type": "ineq","fun":constraint})
h_init = 0.1

```

```python
sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
sol
```

         fun: -1.960000000389105
         jac: array([1.01749761])
     message: 'Optimization terminated successfully.'
        nfev: 27
         nit: 9
        njev: 9
      status: 0
     success: True
           x: array([1.47461354])

```python
sol.x[0]
```

```python
#I test that there is no wage value below 0.6 where the worker would work
w = 0.59

#because it is negative values we minimize (and not -minimize)
def objective(h):
    return utility_2(h,w,1,rho,gamma,beta,delta,b)
def constraint(h):
    return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
cons = ({"type": "ineq","fun":constraint})
h_init = 0.1
sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
if sol.x[0] > 1.5:
    print("No level of human capital where the worker would work, at human capital level of",str(w))
else:
    print("The worker til work if she has a human capital level of", str(round(sol.x[0],2)))
```

    No level of human capital where the worker would work, at human capital level of 0.59

```python
#So the worker will work at a wage of 0.6, if she has a human capital level og 1.47 or higher
```

```python
#Define wages to plot
wages = [0.6,0.7,0.8,0.9,0.99]
#And the parameters again
rho = 2
beta = 0.96
gamma = 0.1
b = 1
delta = 0.1
h_vec = np.linspace(0.1,1.5,100)
```

```python
for wage in wages:
    wage_vec = utility_2(h1 = h_vec, w = wage, l1 = 1, rho = 2, gamma = 0.1, beta= 0.96, delta = 0.1, b = 1)
    plt.plot(h_vec,wage_vec, label = "Wage ="+str(wage))

plt.plot(h_vec,x, label = "Wage = 2",alpha= 0.6)
plt.plot(h_vec,y, color = "red", linestyle = "dashed",alpha= 0.5, label = "Utility of not working")
plt.xlabel("Human capital level")
plt.ylabel("Utility")
plt.ylim(-5,-0.5)
plt.legend()
plt.show()
```

```python
#The respective "cutoff"-levels of human capital can be calculated using numerical optimazation
```

```python
for wage in wages:
    w = wage
    def objective(h):
        return utility_2(h,w,1,rho,gamma,beta,delta,b)
    def constraint(h):
        return (1.96 + utility_2(h,w,1,rho,gamma,beta,delta,b))
    cons = ({"type": "ineq","fun":constraint})
    h_init = 0.1

    sol = optimize.minimize(objective,h_init,method="SLSQP",constraints=cons)
    print("Human capital level =",round(sol.x[0],2),"with wages =", wage)
```

    Human capital level = 1.47 with wages = 0.6
    Human capital level = 1.23 with wages = 0.7
    Human capital level = 1.05 with wages = 0.8
    Human capital level = 0.91 with wages = 0.9
    Human capital level = 0.81 with wages = 0.99

```python
#We could make this interactiv - it will take some time though
```

```python
#Explain why people chose to work even if the wage is lowwer than b
```

## Assigmnet 2

```python
def corr_func_opt(phi,v_m=0, s_m=0):

    np.random.seed(1)
    pi_m = 0
    yn = 0
    y_m = 0
    xt = np.random.normal(loc=0,scale=pms['sigma_x'],size=1000)
    ct = np.random.normal(loc=0,scale=pms['sigma_c'],size=1000)

    list_output = list(range(1000))
    list_inflation = list(range(1000))
    for i in range(1000):
        if i == 0 :

            y = equili_output_ar(v_m,pi_m,s_m,y_m,ct[i],xt[i])
            list_output[i] = y
            yn = y
            p = equili_pi_phi(y_m,yn,pi_m,s_m,ct[i],phi)
            list_inflation[i] = p
            pi_m = p
            y_m = y
            v_m = pms["delta"] * v_m + xt[i]
            s_m = pms["omega"] * s_m + ct [i]
        else :
            y = equili_output_ar(v_m,pi_m,s_m,y_m,ct[i],xt[i])

            list_output[i] = y
            yn = y
            p = equili_pi_phi(y_m,yn,pi_m,s_m,ct[i],phi)
            list_inflation[i] = p
            pi_m = p
            y_m = y
            v_m = pms["delta"] * v_m + xt[i]
            s_m = pms["omega"] * s_m + ct[i]

    correlation = np.corrcoef(list_inflation,list_output)[0,1]

    return correlation

```

```python
v_m = 0
s_m = 0
def objective(phi):
    return corr_func_opt(phi, v_m, s_m)
def constraint(phi):
    return (0.31 - corr_func_opt(phi, v_m, s_m))
```

```python

```

```python
constraint(0.3)
```

```python
cons = ({"type": "ineq","fun":constraint})
phi_init = 0.1

```

```python
sol = optimize.minimize(objective,phi_init,method="SLSQP",constraints=cons)
sol
```

```python

```