# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of a Passive Electrical Network

[Electrical networks](https://en.wikipedia.org/wiki/Electrical_network) composed of linear passive elements, like resistors, capacitors and inductors can be described by linear ordinary differential equations (ODEs) with constant coefficients. Hence, in view of the theory of signals and systems they can be interpreted as a linear time-invariant (LTI) system. The different ways to characterize the properties of an LTI system introduced before are illustrated at the example of a second-order analog [low-pass filter](https://en.wikipedia.org/wiki/Low-pass_filter).

It is assumed that no energy is stored in the capacitor and inductor for $t<0$. It is furthermore assumed that $x(t) = 0$ for $t<0$. Hence $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$. For illustration, the normalized values $L = .5$, $R = 1$, $C = .4$ are used for the elements of the electrical network.

### Differential Equation

The differential equation describing the input/output relation of the electrical network is derived by applying [Kirchhoff's circuit laws](https://en.wikipedia.org/wiki/Kirchhoff's_circuit_laws) to the network. This results in the following ODE

\begin{equation}
C L \frac{d^2 y(t)}{dt^2} + C R \frac{d y(t)}{dt} + y(t) = x(t)
\end{equation}

This ODE is defined in `SymPy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, L, R, C = sym.symbols('t L R C', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(L*C*y.diff(t, 2) + R*C*y.diff(t) + y, x)
ode
```

The normalized values of the network elements are stored in a dictionary for substitution

```python
RLC = {R: 1, L: sym.Rational('.5'), C: sym.Rational('.4')}
RLC
```

### Impulse Response

The passive electrical network and the ODE describing its input/output relation can be interpreted as an LTI system. Hence, the system can be characterized by its [impulse response](impulse_response.ipynb) $h(t)$ which is defined as the output of the system for a Dirac impulse $x(t) = \delta(t)$ at the input. For the given system, the impulse response is calculated by explicit solution of the ODE

```python
solution_h = sym.dsolve(ode.subs(x, sym.DiracDelta(t)).subs(y, sym.Function('h')(t)))
solution_h
```

The integration constants $C_1$ and $C_2$ have to be determined from the initial conditions $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

```python
integration_constants = sym.solve( (solution_h.rhs.limit(t, 0, '-'), solution_h.rhs.diff(t).limit(t, 0, '-')), ['C1', 'C2'] )
integration_constants
```

Substitution of the values for the integration constants $C_1$ and $C_2$ into the result from above yields the impulse response of the low-pass

```python
h = solution_h.subs(integration_constants).rhs
h
```

The impulse response is plotted for the values of $R$, $L$ and $C$ given above

```python
sym.plot(h.subs(RLC), (t,0,10), ylabel=r'h(t)');
```

### Step Response

The [step response](step_response.ipynb) is derived by integrating over the impulse response $h(t)$. For ease of illustration this is performed for the specific values of the elements given above

```python
tau = sym.symbols('tau')

he = sym.integrate(h.subs(RLC).subs(t,tau), (tau,0,t))
he
```

Let's plot the step response

```python
sym.plot(he, (t,0,10), ylabel=r'$h_\epsilon(t)$');
```

### Transfer Function

For an exponential input signal $x(t) = e^{s t}$, the [transfer function](eigenfunctions.ipynb#Transfer-Function) $H(s)$ represents the weight of the exponential output signal $y(t) = H(s) \cdot e^{s t}$. The transfer function is derived by introducing $x(t)$ and $y(t)$ into the ODE and solving for $H(s)$

```python
s = sym.symbols('s')
H = sym.Function('H')(s)

H, = sym.solve(ode.subs(x, sym.exp(s*t)).subs(y, H*sym.exp(s*t)).doit(), H)
H
```

The transfer characteristics of an LTI system for harmonic exponential signals $e^{j \omega} = \cos(\omega t) + j \sin(\omega t)$ are of special interest in the analysis of electrical circuits. It can be derived from $H(s)$ by substituting the complex frequency $s$ with $s = j \omega$. The resulting transfer function $H(j \omega)$ provides the attenuation and phase the system adds to a harmonic input signal.

```python
w = sym.symbols('omega')

Hjw = H.subs(s, sym.I * w)
Hjw
```

The magnitude of the transfer function $|H(j \omega)|$ is plotted for illustration for the specific values of the elements given above

```python
sym.plot(abs(Hjw.subs(RLC)), (w, -10, 10), ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

It is obvious, that the low frequencies pass through the system and that high frequencies are attenuated. This motivates the term 'low-pass' for such systems.

As alternative to the solution of the ODE, the transfer function $H(s)$ is [computed from the impulse response](eigenfunctions.ipynb#Link-between-Transfer-Function-and-Impulse-Response) and plotted for the specific values of the elements given above

```python
H2 = sym.integrate(h.subs(RLC)*sym.exp(-s*t), (t,0,sym.oo), conds='none')
H2 = H2.simplify()
H2
```

```python
sym.plot(abs(H2.subs(s, sym.I*w)), (w, -10, 10), ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


```python
from sympy import sqrt, symbols, Eq, init_printing, cos, pi, solveset, S
from sympy.plotting import plot
import numpy as np
import matplotlib.pyplot as plt
from matplotlib.patches import Ellipse
from matplotlib.ticker import (MultipleLocator, FormatStrFormatter,
                               AutoMinorLocator)
from scipy.spatial.distance import euclidean as dist

init_printing()
```

```python
R1, R2, e, a, b, alpha = symbols("R1, R2, e, a, b, alpha")

eq_e1 = Eq(e, sqrt(R1 ** 2 + R2 ** 2 - 2 * R1 * R2 * cos(alpha)) / (R1 + R2))
eq_e2 = Eq(e, sqrt(1 - b ** 2 / a ** 2))

x, y = symbols("x, y")
eq_ells = Eq((x / a) ** 2 + (y / b) ** 2, 1)
eq_ells

eq_e1, eq_e2, eq_ells
```

```python
f1 = 15.96
f2 = 6.29
angle = pi / 2
dia = 9

temp_a = float((f1 + f2) / 2)
temp_e = float(eq_e1.subs({R1: f1, R2: f2, alpha: angle}).evalf().rhs)
temp_b = float(solveset(eq_e2.subs({a: (f1 + f2) / 2, e: temp_e}), b).args[-1])
temp_A0 = 2 * temp_e * temp_a

temp_a, temp_a*2, temp_e, temp_b, temp_b*2, temp_A0
```

```python
fig, ax = plt.subplots(figsize=(10, 6), dpi=160)

###########################################################################################################
# Defining ellipse
w, h = 2 * temp_a, 2 * temp_b
origin = 0
ells = Ellipse(
    xy=[origin, origin], width=w, height=h, edgecolor="C0", lw=1, facecolor="none"
)

ells.set_clip_box(ax.bbox)
ax.add_artist(ells)
ax.plot(origin, origin, "o", label=f"Ellipse's Origin (0, 0)")

extra_space = 5

ax.hlines(
    0,
    xmin=origin - w / 2,
    xmax=origin + w / 2,
    linestyles="--",
    label=f"Major axis(2a): {w:.2f} cm",
)
ax.vlines(
    0,
    ymin=origin - h / 2,
    ymax=origin + h / 2,
    linestyles="-",
    label=f"Minor axis(2b): {h:.2f} cm",
)

###########################################################################################################

###########################################################################################################
# Distance b/w focii (A0)
ax.hlines(
    0,
    xmin=-temp_A0 / 2,
    xmax=temp_A0 / 2,
    edgecolor="C1",
    label=f"$A_0: ${temp_A0:.2f} cm",
)
ax.plot([-temp_A0 / 2, temp_A0 / 2], [0, 0], "oC1", label=f"Focii: ({f1}, {f2})")

ax.annotate("f1", xy=(-temp_A0 / 2, 0), xytext=(-temp_A0 / 2, 1))
ax.annotate("f2", xy=(temp_A0 / 2, 0), xytext=(temp_A0 / 2, 1))
ax.annotate(f"{temp_A0:.2f}", xy=(0, 0), xytext=(1, -1))

ax.annotate(
    f"({(-temp_A0/2):.2f}, 0)",
    xy=(-temp_A0 / 2, 0),
    xytext=(-temp_A0 / 2, -1),
    ha="center",
)
ax.annotate(
    f"({(temp_A0/2):.2f}, 0)",
    xy=(temp_A0 / 2, 0),
    xytext=(temp_A0 / 2, -1),
    ha="center",
)

###########################################################################################################

###########################################################################################################
# Mirror Portion with beams from focii
temp_x = (f1 ** 2 - f2 ** 2) / (2 * temp_A0)
temp_y = np.sqrt(f1 ** 2 - (temp_x + temp_A0 / 2) ** 2)
ax.plot(temp_x, temp_y, "oC2", label=f"Mirror's origin ({temp_x:.2f}, {temp_y:.2f})")

ax.plot([-temp_A0 / 2, temp_x], [0, temp_y], "--C2")
ax.plot([temp_A0 / 2, temp_x], [0, temp_y], "--C2")

ax.annotate(
    f"{f1}",
    xy=(-temp_A0 / 2, 0),
    xytext=(-temp_A0 / 2 + 5, 3),
    rotation=12,
    ha="center",
    va="center",
)
ax.annotate(
    f"{f2}", xy=(temp_A0 / 2, 0), xytext=(temp_A0 / 2 - 2, 3), ha="center", va="center"
)

ax.annotate(
    f"({temp_x:.2f}, {temp_y:.2f})", xy=(temp_x, temp_y), xytext=(temp_x, temp_y+1), ha="left", va="center"
)
###########################################################################################################

###########################################################################################################
# Mirror portion (with diameter dia)
temp_x2 = np.linspace(temp_x, temp_x + dia / 2)
temp_y2 = np.array(
    [
        float(solveset(eq_ells.subs({x: i, a: temp_a, b: temp_b}), y).args[-1])
        for i in temp_x2
    ]
)

temp_x3 = np.linspace(temp_x, temp_x - dia / 2)
temp_y3 = np.array(
    [
        float(solveset(eq_ells.subs({x: i, a: temp_a, b: temp_b}), y).args[-1])
        for i in temp_x3
    ]
)

ax.plot(temp_x2, temp_y2, "--C3", lw=2)
ax.plot(temp_x3, temp_y3, "--C3", lw=2)
ax.plot(
    [temp_x + dia / 2, temp_x - dia / 2],
    [temp_y2[-1], temp_y3[-1]],
    "oC3",
    lw=2,
    label=f"Mirror of dia: {dia} cm",
)

###########################################################################################################

ax.text(
    -10,
    9,
    f"e={temp_e:.2f} cm; "
    + r"$\alpha=\frac{\pi^0}{2}$"
    + "\nOptimized for Freq: [325-1100]GHz",
)

ax.xaxis.set_minor_locator(AutoMinorLocator(5))
ax.yaxis.set_minor_locator(AutoMinorLocator(5))

ax.set(
    xlim=[origin - w / 2 - extra_space, origin + w / 2 + extra_space],
    ylim=[origin - h / 2 - extra_space, origin + h / 2 + extra_space],
    xlabel="X (cm)",
    ylabel="Y (cm)",
)

ax.grid(alpha=0.25, which="minor")
ax.grid(alpha=0.9, which="major")

ax.legend(fontsize=6, ncol=2)
ax.axis('equal')

plt.savefig(f"./Simulations/Mirror-[{dia}]-[325-1100]GHz.pdf")
plt.savefig(f"./Simulations/Mirror-[{dia}]-[325-1100]GHz.png")

plt.show()
```

```python
eq_ells, solveset(eq_ells.subs({y:0, a:temp_a, b:temp_b}), x)
```

```python
11.125*2
```

```python
y0 = 5.28
solveset(eq_ells.subs({y:y0, a:temp_a, b:temp_b}), x)
```

```python
temp_y/2
```

```python
np.pi, np.rad2deg(np.pi), np.inf
```

```python
np.rad2deg(np.arccos(0)), np.cos(np.pi), np.rad2deg(np.arctan(np.inf))
```

```python
temp_a, temp_b
```

```python
r1 = 20
r2 = 8.960156084249821
a1 = (r1+r2)/2
e1 = eq_e1.subs({R1:r1, R2:r2, alpha:np.pi/2}).rhs
b1 = solveset(eq_e2.subs({a:a1, e:e1}), b).args[-1]
a1, b1, e1
```

```python
fig, ax = plt.subplots()

###########################################################################################################
# Defining ellipse
w, h = 2 * temp_a, 2 * temp_b
origin = 0
ells = Ellipse(
    xy=[origin, origin], width=w, height=h, edgecolor="C0", lw=1, facecolor="none"
)

ells.set_clip_box(ax.bbox)
ax.add_artist(ells)
ax.set(
    xlim=[origin - w / 2 - extra_space, origin + w / 2 + extra_space],
    ylim=[origin - h / 2 - extra_space, origin + h / 2 + extra_space],
    xlabel="X (cm)",
    ylabel="Y (cm)",
)
ax.grid()
plt.show()
```

```python
slope = (temp_y-0)/(temp_x-(-temp_A0/2))
slope
```

```python
m, c = symbols("m, c")
eq_y = Eq(y, m*x+c)
eq_y
```

```python

```

```python

```


# Integrating out the noise in Gaussian likelihood problems

A property of Gaussian likelihoods and certain priors is that the joint posterior distribution $p(\theta,\sigma)$ can be analytically integrated with respect to $\sigma$ to yield the marginal distribution $p(\theta|X)$ up to a proportion,

$\begin{align} p(\theta|X) &= \int_{0}^{\infty} p(\theta, \sigma|X) \mathrm{d}\sigma\\ &\propto \int_{0}^{\infty} p(X|\theta, \sigma) p(\theta, \sigma) \mathrm{d}\sigma,\end{align}$

For example, for Gaussian log-likelihood models where we assume a prior, $\sigma\sim U(a, b),$ then the marginal posterior is given by,

$p(\theta|X) \propto \frac{\pi ^{-n/2} \text{sse}(\theta)^{\frac{1}{2}-\frac{n}{2}} \left[\Gamma
   \left(\frac{n-1}{2},\frac{\text{sse}(\theta)}{2 b^2}\right)-\Gamma
   \left(\frac{n-1}{2},\frac{\text{sse}(\theta)}{2 a^2}\right)\right]}{2 \sqrt{2} (b - a)}$,

where $\text{sse}(\theta) = \sum_{i=1}^n(f_i(\theta) - y_i)^2,$ is the sum of square errors and $\Gamma(u,v)$ is the upper incomplete gamma function.

In this notebook, we illustrate how using the posterior geometry where the noise parameter $\sigma$ is integrated out can lead to faster convergence of the posterior distribution than the circumstance where $\sigma$ is estimated.

An issue with using the integrated log-likelihood is that the estimated model is no longer generative and so posterior predictive checking is harder. Whether this is more than compensated for by the speed up of sampling from a posterior of lower dimensionality is problem specific although unlikely to considerably speed up sampling.

## Single output problem - logistic model
Here we illustrate how a posterior distribution for the logistic model growth rate and carrying capacity parameters can be estimated using both full posterior and the marginal posterior geometries.

```python
from __future__ import print_function
import pints
import pints.toy as toy
import pints.plot
import numpy as np
import matplotlib.pyplot as plt
import scipy.stats

# Load a forward model
model = toy.LogisticModel()

# Create some toy data
real_parameters = [0.015, 500]
times = np.linspace(0, 1000, 100)
signal_values = model.simulate(real_parameters, times)

# Add noise
nu = 2
sigma = 10
observed_values_norm = signal_values + scipy.stats.norm.rvs(loc=0, scale=sigma, size=signal_values.shape)

plt.figure(figsize=(14, 6))
plt.xlabel('Time')
plt.ylabel('Values')
plt.plot(times, observed_values_norm)
plt.plot(times, signal_values)
plt.show()

# Create an object with links to the model and time series
problem = pints.SingleOutputProblem(model, times, observed_values_norm)
```

Fit the model using a Gaussian likelihood where we estimate the noise parameter $\sigma$.

```python
# Create a log-likelihood function (adds an extra parameter!)
log_likelihood = pints.GaussianLogLikelihood(problem)

# Create a uniform prior over both the parameters and the new noise variable
log_prior = pints.UniformLogPrior(
    [0.01, 400, 1],
    [0.02, 600, 100]
)

# Create a posterior log-likelihood (log(likelihood * prior))
log_posterior = pints.LogPosterior(log_likelihood, log_prior)

# Choose starting points for 3 mcmc chains
real_parameters1 = np.array(real_parameters + [sigma])
xs = [
    real_parameters1 * 1.1,
    real_parameters1 * 0.9,
    real_parameters1 * 1.15,
]

# Create mcmc routine
mcmc = pints.MCMCController(log_posterior, 3, xs, method=pints.AdaptiveCovarianceMCMC)

# Add stopping criterion
mcmc.set_max_iterations(4000)

# Start adapting after 1000 iterations
mcmc.set_initial_phase_iterations(1000)

# Disable logging mode
mcmc.set_log_to_screen(False)

# Run!
print('Running...')
chains = mcmc.run()
print('Done!')

# Show traces and histograms
pints.plot.trace(chains)

# Discard warm up
chains = chains[:, 2000:, :]
chains_gaussian = chains

# Check convergence using rhat criterion
print('R-hat:')
print(pints.rhat_all_params(chains))

# Show graphs
plt.show()
```

Now fitting the same model using a log-likelihood where the noise parameter $\sigma$ has been integrated out. Here the convergence to the stationary distribution is quicker.

```python
# Create a log-likelihood function with lower and upper values on uniform prior for sigma
lower = 1
upper = 100
log_likelihood = pints.GaussianIntegratedUniformLogLikelihood(problem, lower, upper)

# Create a uniform prior over both the parameters and the new noise variable
log_prior = pints.UniformLogPrior(
    [0.01, 400],
    [0.02, 600]
)

# Create a posterior log-likelihood (log(likelihood * prior))
log_posterior = pints.LogPosterior(log_likelihood, log_prior)

# Choose starting points for 3 mcmc chains
real_parameters = np.array(real_parameters)
xs = [
    real_parameters * 1.1,
    real_parameters * 0.9,
    real_parameters * 1.0,
]

# Create mcmc routine
mcmc = pints.MCMCController(log_posterior, 3, xs, method=pints.AdaptiveCovarianceMCMC)

# Add stopping criterion
mcmc.set_max_iterations(4000)

# Start adapting after 1000 iterations
mcmc.set_initial_phase_iterations(250)

# Disable logging mode
mcmc.set_log_to_screen(False)

# Run!
print('Running...')
chains = mcmc.run()
print('Done!')

# Show traces and histograms
pints.plot.trace(chains)

# Discard warm up
chains = chains[:, 2000:, :]
chains_integrated = chains

# Check convergence using rhat criterion
print('R-hat:')
print(pints.rhat_all_params(chains))

# Show graphs
plt.show()
```

Comparing the marginal distributions for $r$ and $\kappa$ for the two models, they look similar.

```python
chains_gaussian1 = np.vstack(chains_gaussian)
chains_integrated1 = np.vstack(chains_integrated)

# plot
plt.figure(figsize=(15, 6))
plt.subplot(1, 2, 1)
bins = np.linspace(0.0147, 0.0153, 20)
plt.hist(chains_gaussian1[:, 0], bins, alpha=0.5, color="blue", label='Estimate sigma')
plt.hist(chains_integrated1[:, 0], bins, alpha=0.5, color="orange", label='Integrate out sigma')
plt.title("Growth rate")

plt.subplot(1, 2, 2)
bins = np.linspace(490, 510, 20)
plt.hist(chains_gaussian1[:, 1], bins, alpha=0.5, color="blue", label='Estimate sigma')
plt.hist(chains_integrated1[:, 1], bins, alpha=0.5, color="orange", label='Integrate out sigma')
plt.title("Carrying capacity")

plt.rc('font', size=14)
plt.legend(loc="upper left", bbox_to_anchor=(0.5, -0.1))
plt.show()
```

## Multiple output problem - Fitzhugh-Nagumo model

Now we illustrate how to fit a Gaussian model to a problem with two outputs, where the noise has been integrated out of the likelihood.

```python
# Create a model
model = pints.toy.FitzhughNagumoModel()

# Run a simulation
parameters = [0.1, 0.5, 3]
times = np.linspace(0, 20, 200)
values = model.simulate(parameters, times)

# First add some noise
sigma = 0.5
noisy = values + np.random.normal(0, sigma, values.shape)

# Plot the results
plt.figure(figsize=(14, 6))
plt.xlabel('Time')
plt.ylabel('Noisy values')
plt.plot(times, noisy)
plt.show()

# Define a multiple output problem
problem = pints.MultiOutputProblem(model, times, noisy)
```

```python
# Create a log-likelihood function (adds an extra parameter!)
log_likelihood = pints.GaussianLogLikelihood(problem)

# Create a uniform prior over both the parameters and the new noise variable
log_prior = pints.UniformLogPrior(
    [0, 0, 0, 0, 0],
    [10, 10, 10, 20, 20]
)

# Create a posterior log-likelihood (log(likelihood * prior))
log_posterior = pints.LogPosterior(log_likelihood, log_prior)

# Choose starting points for 3 mcmc chains
real_parameters1 = np.array(parameters + [sigma, sigma])
xs = [
    real_parameters1 * 1.1,
    real_parameters1 * 0.9,
    real_parameters1 * 1.15,
]

# Create mcmc routine
mcmc = pints.MCMCController(log_posterior, 3, xs, method=pints.AdaptiveCovarianceMCMC)

# Add stopping criterion
mcmc.set_max_iterations(4000)

# Start adapting after 1000 iterations
mcmc.set_initial_phase_iterations(1000)

# Disable logging mode
mcmc.set_log_to_screen(False)

# Run!
print('Running...')
chains = mcmc.run()
print('Done!')

# Show traces and histograms
pints.plot.trace(chains)

# Discard warm up
chains = chains[:, 2000:, :]
chains_gaussian = chains

# Check convergence using rhat criterion
print('R-hat:')
print(pints.rhat_all_params(chains))
chains_full = chains

# Show graphs
plt.show()
```

Now estimating the same model where the two noise parameters are integrated out of the posterior. Again, the convergence rate tends to be faster and we see better chain mixing although this varies from run to run.

```python
# Create a log-likelihood function (assumed U(0, 20) priors on both sigmas)
log_likelihood = pints.GaussianIntegratedUniformLogLikelihood(problem, 0, 20)

# Create a uniform prior over both the parameters and the new noise variable
log_prior = pints.UniformLogPrior(
    [0, 0, 0],
    [10, 10, 10]
)

# Create a posterior log-likelihood (log(likelihood * prior))
log_posterior = pints.LogPosterior(log_likelihood, log_prior)

# Choose starting points for 3 mcmc chains
real_parameters1 = np.array(parameters)
xs = [
    real_parameters1 * 1.1,
    real_parameters1 * 0.9,
    real_parameters1 * 1.15,
]

# Create mcmc routine
mcmc = pints.MCMCController(log_posterior, 3, xs, method=pints.AdaptiveCovarianceMCMC)

# Add stopping criterion
mcmc.set_max_iterations(4000)

# Start adapting after 1000 iterations
mcmc.set_initial_phase_iterations(1000)

# Disable logging mode
mcmc.set_log_to_screen(False)

# Run!
print('Running...')
chains = mcmc.run()
print('Done!')

# Show traces and histograms
pints.plot.trace(chains)

# Discard warm up
chains = chains[:, 2000:, :]
chains_gaussian = chains

# Check convergence using rhat criterion
print('R-hat:')
print(pints.rhat_all_params(chains))
chains_marginal=chains

# Show graphs
plt.show()
```

Comparing the effective sample size between the posterior geometry where we sample from the full posterior versus the marginal version we see that we have much larger effective sample sizes using the marginalised density.

```python
full = pints.effective_sample_size(np.vstack(chains_full))
marginal = pints.effective_sample_size(np.vstack(chains_marginal))
print('Effective sample size of model parameters for full posterior: ' + str(full[0:2]))
print('Effective sample size of model parameters for marginal posterior: ' + str(marginal[0:2]))
```

    Effective sample size of model parameters for full posterior: [347.53874375248461, 247.12936497020999]
    Effective sample size of model parameters for marginal posterior: [488.53736925661275, 488.66809520505473]


.. _nb_subset_selection:

## Subset Selection Problem

A genetic algorithm can be used to approach subset selection problems by defining custom operators. In general, a metaheuristic algorithm might not be the ultimate goal to implement in a real-world scenario; however, it might be useful to investigate patterns or characteristics of possible well-performing subsets.
Let us consider a simple toy problem where we have to select numbers from a list. For every solution, exactly ten numbers have to be selected that their sum is minimized.
For the subset selection problem, a binary encoding can be used where **one** indicates a number is picked. In our problem formulation, the list of numbers is represented by $L$ and the binary encoded variable by $x$.

\begin{align}
\begin{split}
\min f(x) & = & \sum_{k=1}^{n} L_k \cdot x_k\\[2mm]
\text{s.t.} \quad g(x)  & = & (\sum_{k=1}^{n} x_k - 10)^2\\[2mm]
\end{split}
\end{align}

As shown above, the equality constraint is handled by ensuring $g(x)$ can only be zero if exactly ten numbers are chosen.
The problem can be implemented as follows:

```python
import numpy as np
from pymoo.model.problem import Problem

class SubsetProblem(Problem):
    def __init__(self,
                 L,
                 n_max
                 ):
        super().__init__(n_var=len(L), n_obj=1, n_constr=1, elementwise_evaluation=True)
        self.L = L
        self.n_max = n_max

    def _evaluate(self, x, out, *args, **kwargs):
        out["F"] = np.sum(self.L[x])
        out["G"] = (self.n_max - np.sum(x)) ** 2

# create the actual problem to be solved
np.random.seed(1)
L = np.array([np.random.randint(100) for _ in range(100)])
n_max = 10
problem = SubsetProblem(L, n_max)

```

The customization requires writing custom operators in order to solve this problem efficiently. We recommend considering the feasibility directly in the evolutionary operators because otherwise, most of the time, infeasible solutions will be processed.
The sampling creates a random solution where the subset constraint will always be satisfied.
The mutation randomly removes a number and chooses another one. The crossover takes the values of both parents and then randomly picks either the one from the first or from the second parent until enough numbers are picked.

```python
from pymoo.model.crossover import Crossover
from pymoo.model.mutation import Mutation
from pymoo.model.sampling import Sampling

class MySampling(Sampling):

    def _do(self, problem, n_samples, **kwargs):
        X = np.full((n_samples, problem.n_var), False, dtype=np.bool)

        for k in range(n_samples):
            I = np.random.permutation(problem.n_var)[:problem.n_max]
            X[k, I] = True

        return X

class BinaryCrossover(Crossover):
    def __init__(self):
        super().__init__(2, 1)

    def _do(self, problem, X, **kwargs):
        n_parents, n_matings, n_var = X.shape

        _X = np.full((self.n_offsprings, n_matings, problem.n_var), False)

        for k in range(n_matings):
            p1, p2 = X[0, k], X[1, k]

            both_are_true = np.logical_and(p1, p2)
            _X[0, k, both_are_true] = True

            n_remaining = problem.n_max - np.sum(both_are_true)

            I = np.where(np.logical_xor(p1, p2))[0]

            S = I[np.random.permutation(len(I))][:n_remaining]
            _X[0, k, S] = True

        return _X

class MyMutation(Mutation):
    def _do(self, problem, X, **kwargs):
        for i in range(X.shape[0]):
            X[i, :] = X[i, :]
            is_false = np.where(np.logical_not(X[i, :]))[0]
            is_true = np.where(X[i, :])[0]
            X[i, np.random.choice(is_false)] = True
            X[i, np.random.choice(is_true)] = False

        return X
```

After having defined the operators a genetic algorithm can be initialized.

```python
from pymoo.algorithms.so_genetic_algorithm import GA
from pymoo.optimize import minimize

algorithm = GA(
    pop_size=100,
    sampling=MySampling(),
    crossover=BinaryCrossover(),
    mutation=MyMutation(),
    eliminate_duplicates=True)

res = minimize(problem,
               algorithm,
               ('n_gen', 60),
               seed=1,
               verbose=True)

print("Function value: %s" % res.F[0])
print("Subset:", np.where(res.X)[0])

```

    ===========================================================================
    n_gen |  n_eval |   cv (min)   |   cv (avg)   |     fopt     |     favg
    ===========================================================================
        1 |     100 |  0.00000E+00 |  0.00000E+00 |          258 |  4.43940E+02
        2 |     200 |  0.00000E+00 |  0.00000E+00 |          205 |  3.54340E+02
        3 |     300 |  0.00000E+00 |  0.00000E+00 |          175 |  3.00390E+02
        4 |     400 |  0.00000E+00 |  0.00000E+00 |          161 |  2.52240E+02
        5 |     500 |  0.00000E+00 |  0.00000E+00 |          124 |  2.18240E+02
        6 |     600 |  0.00000E+00 |  0.00000E+00 |          118 |  1.91860E+02
        7 |     700 |  0.00000E+00 |  0.00000E+00 |          118 |  1.70050E+02
        8 |     800 |  0.00000E+00 |  0.00000E+00 |           85 |  1.49630E+02
        9 |     900 |  0.00000E+00 |  0.00000E+00 |           85 |  1.39550E+02
       10 |    1000 |  0.00000E+00 |  0.00000E+00 |           85 |  1.30580E+02
       11 |    1100 |  0.00000E+00 |  0.00000E+00 |           74 |  1.19930E+02
       12 |    1200 |  0.00000E+00 |  0.00000E+00 |           74 |  1.11800E+02
       13 |    1300 |  0.00000E+00 |  0.00000E+00 |           74 |  1.06050E+02
       14 |    1400 |  0.00000E+00 |  0.00000E+00 |           68 |  1.00100E+02
       15 |    1500 |  0.00000E+00 |  0.00000E+00 |           68 |  9.46300E+01
       16 |    1600 |  0.00000E+00 |  0.00000E+00 |           52 |  8.97600E+01
       17 |    1700 |  0.00000E+00 |  0.00000E+00 |           50 |  8.45100E+01
       18 |    1800 |  0.00000E+00 |  0.00000E+00 |           50 |  8.09000E+01
       19 |    1900 |  0.00000E+00 |  0.00000E+00 |           50 |  7.66900E+01
       20 |    2000 |  0.00000E+00 |  0.00000E+00 |           50 |  7.33700E+01
       21 |    2100 |  0.00000E+00 |  0.00000E+00 |           45 |  7.00100E+01
       22 |    2200 |  0.00000E+00 |  0.00000E+00 |           45 |  6.78900E+01
       23 |    2300 |  0.00000E+00 |  0.00000E+00 |           45 |  6.60300E+01
       24 |    2400 |  0.00000E+00 |  0.00000E+00 |           45 |  6.43500E+01
       25 |    2500 |  0.00000E+00 |  0.00000E+00 |           45 |  6.36500E+01
       26 |    2600 |  0.00000E+00 |  0.00000E+00 |           45 |  6.16100E+01
       27 |    2700 |  0.00000E+00 |  0.00000E+00 |           45 |  6.03300E+01
       28 |    2800 |  0.00000E+00 |  0.00000E+00 |           43 |  5.94900E+01
       29 |    2900 |  0.00000E+00 |  0.00000E+00 |           43 |  5.76600E+01
       30 |    3000 |  0.00000E+00 |  0.00000E+00 |           43 |  5.65100E+01
       31 |    3100 |  0.00000E+00 |  0.00000E+00 |           42 |  5.54700E+01
       32 |    3200 |  0.00000E+00 |  0.00000E+00 |           42 |  5.42800E+01
       33 |    3300 |  0.00000E+00 |  0.00000E+00 |           42 |  5.36700E+01
       34 |    3400 |  0.00000E+00 |  0.00000E+00 |           42 |  5.27000E+01
       35 |    3500 |  0.00000E+00 |  0.00000E+00 |           42 |  5.21800E+01
       36 |    3600 |  0.00000E+00 |  0.00000E+00 |           42 |  5.19700E+01
       37 |    3700 |  0.00000E+00 |  0.00000E+00 |           39 |  5.14400E+01
       38 |    3800 |  0.00000E+00 |  0.00000E+00 |           39 |  5.10500E+01
       39 |    3900 |  0.00000E+00 |  0.00000E+00 |           39 |  5.08500E+01
       40 |    4000 |  0.00000E+00 |  0.00000E+00 |           39 |  5.05500E+01
       41 |    4100 |  0.00000E+00 |  0.00000E+00 |           39 |  5.01200E+01
       42 |    4200 |  0.00000E+00 |  0.00000E+00 |           39 |  4.93800E+01
       43 |    4300 |  0.00000E+00 |  0.00000E+00 |           39 |  4.88000E+01
       44 |    4400 |  0.00000E+00 |  0.00000E+00 |           39 |  4.85700E+01
       45 |    4500 |  0.00000E+00 |  0.00000E+00 |           39 |  4.81400E+01
       46 |    4600 |  0.00000E+00 |  0.00000E+00 |           38 |  4.76700E+01
       47 |    4700 |  0.00000E+00 |  0.00000E+00 |           38 |  4.71800E+01
       48 |    4800 |  0.00000E+00 |  0.00000E+00 |           37 |  4.66100E+01
       49 |    4900 |  0.00000E+00 |  0.00000E+00 |           37 |  4.62700E+01
       50 |    5000 |  0.00000E+00 |  0.00000E+00 |           37 |  4.62000E+01
       51 |    5100 |  0.00000E+00 |  0.00000E+00 |           37 |  4.57600E+01
       52 |    5200 |  0.00000E+00 |  0.00000E+00 |           37 |  4.56000E+01
       53 |    5300 |  0.00000E+00 |  0.00000E+00 |           37 |  4.53400E+01
       54 |    5400 |  0.00000E+00 |  0.00000E+00 |           36 |  4.49200E+01
       55 |    5500 |  0.00000E+00 |  0.00000E+00 |           36 |  4.46700E+01
       56 |    5600 |  0.00000E+00 |  0.00000E+00 |           36 |  4.45100E+01
       57 |    5700 |  0.00000E+00 |  0.00000E+00 |           36 |  4.43800E+01
       58 |    5800 |  0.00000E+00 |  0.00000E+00 |           36 |  4.42300E+01
       59 |    5900 |  0.00000E+00 |  0.00000E+00 |           36 |  4.41400E+01
       60 |    6000 |  0.00000E+00 |  0.00000E+00 |           36 |  4.38100E+01
    Function value: 36
    Subset: [ 5  9 12 31 36 37 47 52 68 99]

Finally, we can compare the found subset with the optimum known simply through sorting:

```python
opt = np.sort(np.argsort(L)[:n_max])
print("Optimal Subset:", opt)
print("Optimal Function Value: %s" % L[opt].sum())
```

    Optimal Subset: [ 5  9 12 31 36 37 47 52 68 99]
    Optimal Function Value: 36


# Theory: Gravitational restricted three-body system

## Equation of motion

The motion of the planets can computed from Newton's equations of motion. We consider two dimensional motion. For a three-body system, a set of coupled differential equations can be used to express the system.They can be expressed using vector notation. Let ${\bf{r}} = x \hat{i} + y \hat{j} $. Where, $\hat{i}$ and $\hat{j}$ are unit vectors along x and y axis. So, the differential equation of motion of a planet is:

\begin{equation}
	\label{e1}
	 \frac{d^2 \bf{r}}{d t^2} = \frac{\bf{F}}{M_E}                   \hspace{5cm}     (1)
\end{equation}

## Dimensionless form of the differential equation

We also define the following dimensionless variables

\begin{eqnarray}
	\nonumber
	\hat{t} &=& \frac{t}{T}  \\ \nonumber
	\hat{\bf{r}} &=& \frac{\bf{r}}{R}  \\ \nonumber
	\hat{M_E} &=& \frac{M_E}{M}  \\ \nonumber
	\hat{M_S} &=& \frac{M_S}{M}  \\ \nonumber
	\hat{\bf{F}} &=& \frac{\bf{F}}{F^*}
\end{eqnarray}

We note that the derivatives can be written in terms of the dimensionless quantities as:

\begin{eqnarray}
    \nonumber
	\frac{d \bf{r}}{d t} &=& \frac{R}{T}\frac{d \bf{\hat{r}}}{d \hat{t}}     \hspace{6cm} (2) \\  \nonumber
    \frac{d^2 \bf{r}}{d t^2} &=& \frac{R}{T^2} \frac{d^2 \bf{\hat{r}}}{d \hat{t}^2} \hspace{5.5cm} (3)
\end{eqnarray}

Substituting these values in Eq. (1), we get:

\begin{equation}
    \nonumber
	\frac{R}{T^2} \frac{d^2 \bf{\hat{r}}}{d \hat{t}^2} = \frac{F^*\bf{\hat{F}}}{\hat{M_E} M}  	\hspace{5cm} (4) \\
\end{equation}

\begin{equation}
    \nonumber
	\frac{d^2 \bf{\hat{r}}}{d \hat{t}^2} = \frac{\bf{\hat{F}}}{\hat{M_E} } \frac{F^*T^2}{RM}         \hspace{5cm} (5)
\end{equation}

Let,

\begin{equation}
     \nonumber
	\frac{F^*T^2}{RM} = 1
\end{equation}

\begin{equation}
	F^* = \frac{RM}{T^2}
\end{equation}

So, Eq. (5) becomes:

\begin{equation}
	\frac{d^2 \bf{\hat{r}}}{d \hat{t}^2} = \frac{\bf{\hat{F}}}{\hat{M_E} } \hspace{5cm} (6)
\end{equation}

Eq. (6) is dimensionless. It can be broken down into two coupled first order equations:

\begin{equation}
    \nonumber
	\frac{d \bf{\hat{r}}}{d \hat{t}} = \hat{\bf{v}} \hspace{5cm} (7)
\end{equation}

\begin{equation}
	\frac{d \bf{\hat{v}}}{d \hat{t}} = \frac{\bf{\hat{F}}}{\hat{M_E} } \hspace{5cm} (8)
\end{equation}

Eq. (7) and Eq. (8) are the required dimensionless coupled first order ODEs.

The force is given by Newton's law:
\begin{equation}
     \nonumber
	|{\bf{F}}| = \frac{GM_EM_S}{|\bf{r}|^2} = \frac{\hat{M}_E \hat{M}_S}{|\hat{\bf{r}}|^2}\frac{GM^2}{R^2}
\end{equation}

So,

\begin{equation}
    \nonumber
	|{\bf{\hat{F}}}| = \frac{\hat{M}_E \hat{M}_S}{|\hat{\bf{r}}|^2}\frac{GM^2}{R^2} \frac{1}{F^*} 	=\frac{\hat{M}_E \hat{M}_S}{|\hat{\bf{r}}|^2}\frac{GM^2}{R^2} \frac{T^2}{RM} =\frac{\hat{M}_E \hat{M}_S}{|\hat{\bf{r}}|^2}\frac{GMT^2}{R^3} =\frac{\hat{G} \hat{M}_E \hat{M}_S}{|\hat{\bf{r}}|^2}
\end{equation}

where,
\begin{equation}
	\hat{G} = \frac{GMT^2}{R^3}
\end{equation}

Finally, considering the direction of the force:

\begin{equation}
    \nonumber
	\bf{F} = |\bf{F}|\cos(\theta) \hat{\bf{i}}  + |\bf{F}|\sin(\theta) \hat{\bf{j}}
\end{equation}

Here,
\begin{equation}
	\theta = \tan^{-1}\frac{y}{x}
\end{equation}

The value of the normalizing constants are taken in astronomical units:

\begin{equation}
	\nonumber
	T = 365 \times 24\times 60 \times 60 \,seconds = 1.0 \, year
\end{equation}
\begin{equation}
	R = 1.5 \times 10^{11} \,m = 1.0 \, AU
\end{equation}
\begin{equation}
	M = 6 \times 10^{24} \,kg = 1.0 \, M_E
\end{equation}

The sun is positioned at the origin and assumed to have no motion. Earth's initial position is taken as x = 1 AU and y = 0. The initial velocity is taken to be 30 km/s.

```python

```


[Home Page](../../Start_Here.ipynb)

[Previous Notebook](../diffusion_1d/Diffusion_Problem_Notebook.ipynb)

[1](../introduction/Introductory_Notebook.ipynb)
[2](../diffusion_1d/Diffusion_Problem_Notebook.ipynb)
[3]
[4](../chip_2d/Challenge_CFD_Problem_Notebook.ipynb)

[Next Notebook](../chip_2d/Challenge_CFD_Problem_Notebook.ipynb)

# Solving Ordinary Differential Equations using PINNs

Now that we have familirized ourself with the SimNet APIs and solved the 1-dimensional diffusion equation in a single simulation and parameterized setting, we move to the next application of PINNs. In this notebook we will solve the system of differential equations that define a coupled spring mass porblem. In this tutorial we will see how to solve the transient problems over small time intervals easily by treating time as a continuous variable. We will also use PINNs to solve the problem in an inverse setting where we have the solution data and we use PINN to find the coefficient of the ODE.

You can refer to the *SimNet User Guide Chapter 4 and Chpater 12* for more examples on solving different types of time domain problems and inverse problems respectively. Also, for more information about the SimNet APIs you can refer the *SimNet Source Code Documentation*.

### Learning Outcomes
1. How to use SimNet to simulate transient problems using PINNs by treating time as a continuous variable
2. How to use SimNet to solve a inverse problem

## Problem Description

In this tutorial, we will solve a simple spring mass system as shown in figure below. The systems shows three masses attached to each other by four springs. The springs slide along a friction-less horizontal surface. The masses are assumed to be point masses and the springs are mass-less.

The model's equations are given as below:
$$
\begin{equation}\label{ode_eqn}
\begin{split}
m_1 x_1''(t) &= -k_1 x_1(t) + k_2(x_2(t) - x_1(t)),\\
m_2 x_2''(t) &= -k_2 (x_2(t) - x_1(t))+ k_3(x_3(t) - x_2(t)),\\
m_3 x_3''(t) &= -k_3 (x_3(t) - x_2(t)) - k_4 x_3(t). \end{split}
\end{equation}
$$

Where, $x_1(t), x_2(t), \text{and } x_3(t)$ denote the mass positions along the horizontal surface measured from their equilibrium position, plus right and minus left. As shown in the figure, first and the last spring are fixed to the walls.

For the first part of the tutorial, we will assume the following conditions:
$$
\begin{equation}\label{ode_IC}
\begin{split}
[m_1, m_2, m_3] &= [1, 1, 1],\\
[k_1, k_2, k_3, k_4] &= [2, 1, 1, 2],\\
[x_1(0), x_2(0), x_3(0)] &= [1, 0, 0],\\
[x_1'(0), x_2'(0), x_3'(0)] &= [0, 0, 0].
\end{split}
\end{equation}
$$

## Case Setup

Now that we have our problem defined, let's take a look at the code required to solve it using SimNet.

## Defining the differential equations for the problem

This process is similar to the previous tutorial. We will write each parameter ($k's \text{ and } m's$) as function and substitute it as a number if its constant. This will allow usto parameterize any of this constants by passing them as a string. This will also allow us to solve the inverse problem where either of these quantitity is unknown and can be predicted by the neural network.

```python
from sympy import Symbol, Function, Number
from simnet.pdes import PDES
from simnet.variables import Variables

class SpringMass(PDES):
  name= 'SpringMass'

  def __init__(self, k=(2, 1, 1, 2), m=(1, 1, 1)):

    self.k  = k
    self.m  = m

    k1 = k[0]
    k2 = k[1]
    k3 = k[2]
    k4 = k[3]
    m1 = m[0]
    m2 = m[1]
    m3 = m[2]

    t=Symbol('t')
    input_variables = {'t':t}

    x1 = Function('x1')(*input_variables)
    x2 = Function('x2')(*input_variables)
    x3 = Function('x3')(*input_variables)

    if type(k1) is str:
        k1 = Function(k1)(*input_variables)
    elif type(k1) in [float, int]:
        k1 = Number(k1)
    if type(k2) is str:
        k2 = Function(k2)(*input_variables)
    elif type(k2) in [float, int]:
        k2 = Number(k2)
    if type(k3) is str:
        k3 = Function(k3)(*input_variables)
    elif type(k3) in [float, int]:
        k3 = Number(k3)
    if type(k4) is str:
        k4 = Function(k4)(*input_variables)
    elif type(k4) in [float, int]:
        k4 = Number(k4)

    if type(m1) is str:
        m1 = Function(m1)(*input_variables)
    elif type(m1) in [float, int]:
        m1 = Number(m1)
    if type(m2) is str:
        m2 = Function(m2)(*input_variables)
    elif type(m2) in [float, int]:
        m2 = Number(m2)
    if type(m3) is str:
        m3 = Function(m3)(*input_variables)
    elif type(m3) in [float, int]:
        m3 = Number(m3)

    self.equations = Variables()
    self.equations['ode_x1'] = m1*(x1.diff(t)).diff(t) + k1*x1 - k2*(x2 - x1)
    self.equations['ode_x2'] = m2*(x2.diff(t)).diff(t) + k2*(x2 - x1) - k3*(x3 - x2)
    self.equations['ode_x3'] = m3*(x3.diff(t)).diff(t) + k3*(x3 - x2) + k4*x3
```

## Creating the geometry

Once we have the ODEs defined, we can easily form the train domain as seen in earlier tutorials. In this example, we will use `Point1D` geometry to create the point mass. We will also define the time range of the solution and create symbol for time ($t$) to define the initial condition etc. in train domain. The below code shows the geometry definition for this problem. The strategy of solving transient problem using time as a continuous variable is same as solving a parameterized problem, i.e. we will create a symbolic variable for time `t` and use the time range as an input to the `param_ranges` attribute for the boundary and interior sampling

```python
from sympy import Symbol, Eq
import numpy as np
from simnet.solver import Solver
from simnet.dataset import TrainDomain, ValidationDomain
from simnet.data import Validation
from simnet.sympy_utils.geometry_1d import Point1D
from simnet.controller import SimNetController
from simnet.plot_utils.vtk import var_to_vtk
from spring_mass_ode import SpringMass

# define time variable and range
t_max = 10.0
t_symbol = Symbol('t')
x = Symbol('x')
time_range = {t_symbol: (0, t_max)}

geo = Point1D(0)
```

## Creating Train Domain: Assigning the initial conditions and equations

Next, we will define the train domain using the 1d point we just defined. Please note that we would not be using the x-coordinate ($x$) information of the point, and it is only used to sample a point in space. We will assign it different values for variable ($t$) only (initial conditions and ODEs over the time-range).

```python
class SpringMassTrain(TrainDomain):
  def __init__(self, **config):
    super(SpringMassTrain, self).__init__()

    # initial conditions
    IC = geo.boundary_bc(outvar_sympy={'x1': 1.,
                                       'x2': 0,
                                       'x3': 0,
                                       'x1__t': 0,
                                       'x2__t': 0,
                                       'x3__t': 0},
                         param_ranges={t_symbol: 0},
                         batch_size_per_area=1)
    self.add(IC, name="IC")

    # solve over given time period
    interior = geo.boundary_bc(outvar_sympy={'ode_x1': 0.0,
                                             'ode_x2': 0.0,
                                             'ode_x3': 0.0},
                               param_ranges=time_range,
                               batch_size_per_area=500)
    self.add(interior, name="Interior")
```

## Creating Validation Domain

Next we will define the validation data for this problem. The solution of this problem can be obtained analytically and the expression can be coded into dictionaries of numpy arrays for `x1`, `x2`, and `x3`. This part of the code is similar to the previous diffusion example.

```python
class SpringMassVal(ValidationDomain):
   def __init__(self, **config):
     super(SpringMassVal, self).__init__()
     deltaT = 0.001
     t = np.arange(0, t_max, deltaT)
     t = np.expand_dims(t, axis=-1)
     invar_numpy = {'t': t}
     outvar_numpy = {'x1': (1/6)*np.cos(t) + (1/2)*np.cos(np.sqrt(3)*t) + (1/3)*np.cos(2*t),
                     'x2': (2/6)*np.cos(t) + (0/2)*np.cos(np.sqrt(3)*t) - (1/3)*np.cos(2*t),
                     'x3': (1/6)*np.cos(t) - (1/2)*np.cos(np.sqrt(3)*t) + (1/3)*np.cos(2*t)}
     val = Validation.from_numpy(invar_numpy, outvar_numpy)
     self.add(val, name="Val")
```

## Creating the Neural Network Solver

Now that we have the train domain and other validation and monitor domains defined, we can prepare the neural network solver and run the problem. We can see that since this is problem involves only time dimension, we will creat the network with only `t` as input.

```python
class SpringMassSolver(Solver):
  train_domain = SpringMassTrain
  val_domain = SpringMassVal

  def __init__(self, **config):
    super(SpringMassSolver, self).__init__(**config)

    self.equations = SpringMass(k=(2, 1, 1, 2), m=(1, 1, 1)).make_node()

    spring_net = self.arch.make_node(name='spring_net',
                                   inputs=['t'],
                                   outputs=['x1','x2','x3'])
    self.nets = [spring_net]

  @classmethod # Explain This
  def update_defaults(cls, defaults):
    defaults.update({
        'network_dir': './network_checkpoint_spring_mass',
        'max_steps': 10000,
        'decay_steps': 100,
        'nr_layers': 6,
        'layer_size': 256,
        'xla': True,
        })
```

Awesome! We have just completed the file set up for the problem. We are now ready to solve the sytem of ODEs using Neural Networks!

```python
import os
import sys

sys.path.append('../../source_code/spring_mass')
!python ../../source_code/spring_mass/spring_mass_solver.py
```

## Visualizing the solution

The .npz arrays can be plotted to visualize the output of the simulation. The .npz files that are created are found in the `network_checkpoint*` directory. The below figure shows the comparison of the neural network solution and the analytical solution. Again, we have a very good agreement in both the results.

The below script shows an example of how the npz arrays can be plotted.

```python
%%capture
import sys
!{sys.executable} -m pip install ipympl
%matplotlib inline
import matplotlib.pyplot as plt

base_dir = './network_checkpoint_spring_mass/val_domain/results/'
# plot in 1d
predicted_data = np.load(base_dir + 'Val_pred.npz', allow_pickle=True)
true_data = np.load(base_dir + 'Val_true.npz', allow_pickle=True)
predicted_data = np.atleast_1d(predicted_data.f.arr_0)[0]
true_data = np.atleast_1d(true_data.f.arr_0)[0]

plt.plot(true_data['t'], true_data['x1'], label='True x1')
plt.plot(true_data['t'], true_data['x2'], label='True x2')
plt.plot(true_data['t'], true_data['x3'], label='True x3')
plt.plot(predicted_data['t'], predicted_data['x1'], label='Pred x1')
plt.plot(predicted_data['t'], predicted_data['x2'], label='Pred x2')
plt.plot(predicted_data['t'], predicted_data['x3'], label='Pred x3')
plt.xlabel("Time")
plt.ylabel("Displacement")
plt.legend()
plt.savefig('comparison_spring_mass.png')
```

```python
from IPython.display import Image
Image(filename='comparison_spring_mass.png')
```

# Solving the Inverse Problem

Another important advantage of a neural network solver over traditional numerical methods is its ability to solve inverse problems. In an inverse problem, we start with a set of observations and then use those observations to calculate the causal factors that produced them. To demonstrate this concept we will solve the same spring mass system in an inverse setting. Specifically, we will assume that the data for the displacements of the 3 masses $x_1, x_2 \text{ and } x_3$ is available to us through analytical solution and the first mass ($m_1$) and the last spring coefficient ($k_4$) is unknown. We will use the neural network to assimilate the displacements from the analytical solution and use it to invert out the unknown quantites from ODEs.

## Case Setup

The equation file again remains same for this part. Here, we will use the `BC` class to assimilate the solution in the training data. Then we will make a network to memorize the three displacements by developing a mapping between $t$ and $(x_1, x_2, x_3)$. The second network will be trained to invert out the desired quantites viz. $(m_1, k_4)$.

## Creating Train Domain

For this problem, we will use the solution coming from analytical solution in from of numpy arrays. We will use the `BC` class to handle such input data. The `BC` class takes in separate dictionaries for input variables and output variables. These dictionaries have a key for each variable and a numpy array of values associated to the key. The `BC` class can be used to assimilate more data even in the forms of a text, CSV, etc. file formats. All we need to do is to convert them into pairs of key and variable and form a dictionary. The `BC` class is very versatile and forms the parent class for the `interior_bc` and `boundary_bc` methods that we have used in the previous tutorials. More information on how to use the `BC` class can be found in the *SimNet User Guide Chapter 12*.

Since part of the problem involves memorizing the given flow field, we will have `['t']` as input keys and `['x_1', 'x_2', 'x_3', 'ode_x1', 'ode_x2', 'ode_x3']` as the output keys. Setting `['x_1', 'x_2', 'x_3']` as input values from analytical data, we are essentially making the network assimilate the analytical distribution of these variables in the selected domain. Setting `['ode_x1', 'ode_x2', 'ode_x3']` equal to 0, we also inform the network to satisfy the ODE losses at those sampled points. Now assuming that we also know the initial conditions of the solution, except the `'m_1'` and `'k_4'`, all the variables in these ODEs are known. Thus the network can use this information to invert out the unknowns. Lets only look at the major additions/changes to the code. The full script can be referred in `spring_mass_inverse.py`.

```python
# define time variable and range
t_max = 10.0
t_symbol = Symbol('t')
x = Symbol('x')
time_range = {t_symbol: (0, t_max)}

geo = Point1D(0)

deltaT = 0.01
t = np.arange(0, t_max, deltaT)
t = np.expand_dims(t, axis=-1)

invar_numpy = {'t': t}
outvar_numpy = {'x1': (1/6)*np.cos(t) + (1/2)*np.cos(np.sqrt(3)*t) + (1/3)*np.cos(2*t),
                'x2': (2/6)*np.cos(t) + (0/2)*np.cos(np.sqrt(3)*t) - (1/3)*np.cos(2*t),
                'x3': (1/6)*np.cos(t) - (1/2)*np.cos(np.sqrt(3)*t) + (1/3)*np.cos(2*t)}
outvar_numpy.update({'ode_x1': np.full_like(invar_numpy['t'], 0)})
outvar_numpy.update({'ode_x2': np.full_like(invar_numpy['t'], 0)})
outvar_numpy.update({'ode_x3': np.full_like(invar_numpy['t'], 0)})

class SpringMassTrain(TrainDomain):
  def __init__(self, **config):
    super(SpringMassTrain, self).__init__()

    # initial conditions
    IC = geo.boundary_bc(outvar_sympy={'x1': 1.,
                                       'x2': 0,
                                       'x3': 0,
                                       'x1__t': 0,
                                       'x2__t': 0,
                                       'x3__t': 0},
                         param_ranges={t_symbol: 0},
                         batch_size_per_area=1)
    self.add(IC, name="IC")

    # data
    data =  BC.from_numpy(invar_numpy, outvar_numpy)

    self.add(data, name="Data")
```

## Creating Monitor Domain

For this tutorial, we will create `MonitorDomain` to monitor the convergence of average `'m_1'` and `'k_4'` inside the domain as the solution progresses. Once we find that the average value of these quantities have reached a steady value, we can end the simulation. The code to generate the `MonitorDomain` can be found below

```python
from simnet.dataset import MonitorDomain
class SpringMassMonitor(MonitorDomain):
  def __init__(self, **config):
    super(SpringMassMonitor, self).__init__()

    global_monitor = Monitor(invar_numpy, {'average_m1': lambda var: tf.reduce_mean(var['m1']),
                                           'average_k4': lambda var: tf.reduce_mean(var['k4'])})
    self.add(global_monitor, 'GlobalMonitor')
```

## Creating the Neural Network Solver

The process of creating a neural network for an inverse problem is similar to other problems we have seen in previous tutorials. However, as the information for the displacements, and in turn their gradients (i.e. velocities and accelerations), is already present (from analytical data) for the network to memorize, we will stop the gradient calls on each of these variables in their respective equations. This means that only the networks predicting `'m1'` and `'k4'` will be optimized to minimize the equation residuals. The displacements, velocities and accelerations are treated as ground truth data.

Also, note that the the mass and spring constant are passed in as Symbolic variables (`'m1'` and `'k4'` respectively) to the
equations as they are unknowns in this problem.

```python
class SpringMassSolver(Solver):
  train_domain = SpringMassTrain
  monitor_domain = SpringMassMonitor

  def __init__(self, **config):
    super(SpringMassSolver, self).__init__(**config)

    self.equations = SpringMass(k=(2, 1, 1, 'k4'), m=('m1', 1, 1)).make_node(stop_gradients=['x1', 'x1__t', 'x1__t__t',
                                                                                'x2', 'x2__t', 'x2__t__t',
                                                                                'x3', 'x3__t', 'x3__t__t'])

    spring_net = self.arch.make_node(name='spring_net',
                                   inputs=['t'],
                                   outputs=['x1','x2','x3'])
    invert_net = self.arch.make_node(name='invert_net',
                                   inputs=['t'],
                                   outputs=['m1','k4'])

    self.nets = [spring_net, invert_net]

  @classmethod # Explain This
  def update_defaults(cls, defaults):
    defaults.update({
        'network_dir': './network_checkpoint_spring_mass_inverse',
        'max_steps': 10000,
        'decay_steps': 100,
        'nr_layers': 6,
        'layer_size': 256,
        'xla': True,
        })
```

## Visualizing the solution

We can monitor the Tensorboard plots to see the convergence of the simulation and the final values of the $m_1$ and $k_4$. We see that after sufficient training, we get the $m_1$ and $k_4$ as 1 and 2 respectively. These were the same values that we used to create the analytical solution.

# Licensing
This material is released by NVIDIA Corporation under the Creative Commons Attribution 4.0 International (CC BY 4.0)

[Home Page](../../Start_Here.ipynb)

[Previous Notebook](../diffusion_1d/Diffusion_Problem_Notebook.ipynb)

[1](../introduction/Introductory_Notebook.ipynb)
[2](../diffusion_1d/Diffusion_Problem_Notebook.ipynb)
[3]
[4](../chip_2d/Challenge_CFD_Problem_Notebook.ipynb)

[Next Notebook](../chip_2d/Challenge_CFD_Problem_Notebook.ipynb)


```python
import torch
import torch.nn as nn
from torch.autograd import Variable
import matplotlib.pyplot as plt
import numpy as np
```

# Pytorch basics

## Define tensors

```python
# random Tensor
X = torch.rand(3, 5)
```

```python
X
```

    tensor([[0.0924, 0.2911, 0.1212, 0.1868, 0.5374],
            [0.3645, 0.5727, 0.5804, 0.6288, 0.0055],
            [0.4104, 0.7801, 0.1654, 0.3685, 0.6012]])

```python
# shape of tensor
X.shape
```

    torch.Size([3, 5])

```python
# type of tensor
X.dtype
```

    torch.float32

```python
# create a tensor from Python
vector = torch.FloatTensor([1, 2, 3, 4, 5])
```

```python
# you can apply operations similar to how Numpy works
print(vector)
print(vector*2)
```

    tensor([1., 2., 3., 4., 5.])
    tensor([ 2.,  4.,  6.,  8., 10.])

```python
integer_vector = torch.LongTensor([1,2, 3, 4, 5])
```

```python
print(integer_vector)
print(integer_vector/2)
```

    tensor([1, 2, 3, 4, 5])
    tensor([0, 1, 1, 2, 2])

# Autograd

Pytorch keep tracks of the gradients automatically

If I wrap a tensor in a Variable object, then Pytorch automatically tracks its gradients:

$$
\begin{align}
b &= w_1 a \\
c &= w_2 a \\
d &= w_3 b + w_4 c \\
L &= 10 - d
\end{align}
$$

```python
a = Variable(torch.FloatTensor([1]), requires_grad=True)
w1, w2, w3, w4 = [Variable(torch.FloatTensor([1]), requires_grad=True),
                  Variable(torch.FloatTensor([2]), requires_grad=True),
                  Variable(torch.FloatTensor([-1]), requires_grad=True),
                  Variable(torch.FloatTensor([4]), requires_grad=True)
                 ]
b = w1*a
c = w2*a
d = w3*b + w4*c
L = 10 - d
```

Let's compute
$$\frac{dd}{dw_4} = c = 2$$

```python
d.backward()
```

```python
w4.grad
```

    tensor([2.])

```python
a = Variable(torch.FloatTensor([1]), requires_grad=True)
w1, w2, w3, w4 = [Variable(torch.FloatTensor([1]), requires_grad=True),
                  Variable(torch.FloatTensor([2]), requires_grad=True),
                  Variable(torch.FloatTensor([-1]), requires_grad=True),
                  Variable(torch.FloatTensor([4]), requires_grad=True)
                 ]
b = w1*a
c = w2*a
d = w3*b + w4*c
L = 10 - d
```

```python
L.backward()
```

Now
$$
\begin{align}
\frac{dL}{da} &= \frac{dL}{dd} \frac{dd}{db} \frac{db}{da} + \frac{dL}{dd} \frac{dd}{dc} \frac{dc}{da}\\
&= -w_3 w_1 -w_4 w_2\\
&= -7
\end{align}
$$

```python
-w3*w1-w4*w2
```

    tensor([-7.], grad_fn=<ThSubBackward>)

```python
a.grad
```

    tensor([-7.])

# Regression

```python
# generate fake data
from sklearn.datasets import make_regression
```

```python
X, y = make_regression(n_features=1, n_informative=1, bias=2, noise=20, random_state=0)
```

```python
X.shape
```

    (100, 1)

```python
y.shape
```

    (100,)

```python
plt.scatter(X, y);
plt.xlabel('x')
plt.ylabel('y');
```

A simple regression model
$$y = w_1b+w_0$$

```python
from sklearn.linear_model import LinearRegression
```

```python
w0_best, w1_best = [LinearRegression().fit(X, y).intercept_, LinearRegression().fit(X, y).coef_[0]]
```

Let's find it through autograd

```python
Xt = torch.from_numpy(X).type(torch.float32)
yt = torch.from_numpy(y.reshape(-1, 1)).type(torch.float32)
w = torch.randn(2, 1, requires_grad=True)
```

```python
learning_rate = 0.001
for i in range(5000): # 10 iterations of gradient descent
    # forward pass
    yp = Xt*w[1] + w[0]
    MSE = ((yp-yt)**2).mean()
    MSE.backward()
    # don't update the gradient
    with torch.no_grad():
        w -= learning_rate * w.grad
        if i % 500 == 0:
            print("MSE = ", MSE)

        # Manually zero the gradients after running the backward pass
        w.grad.zero_()
        w.grad.zero_()
print("Final solution")
print(w[0])
print(w[1])
print("Solution found by scikit-learn")
print(w0_best)
print(w1_best)
```

    MSE =  tensor(2230.8096, grad_fn=<MeanBackward1>)
    MSE =  tensor(686.4489, grad_fn=<MeanBackward1>)
    MSE =  tensor(486.8759, grad_fn=<MeanBackward1>)
    MSE =  tensor(460.7115, grad_fn=<MeanBackward1>)
    MSE =  tensor(457.2306, grad_fn=<MeanBackward1>)
    MSE =  tensor(456.7607, grad_fn=<MeanBackward1>)
    MSE =  tensor(456.6963, grad_fn=<MeanBackward1>)
    MSE =  tensor(456.6875, grad_fn=<MeanBackward1>)
    MSE =  tensor(456.6862, grad_fn=<MeanBackward1>)
    MSE =  tensor(456.6860, grad_fn=<MeanBackward1>)
    Final solution
    tensor([0.3727], grad_fn=<SelectBackward>)
    tensor([42.8514], grad_fn=<SelectBackward>)
    Solution found by scikit-learn
    0.3716363459385481
    42.85335572691592

# Higher level API: nn

The subpackage `nn` allows you to define *layers* that are connected sequentially or in any other fashion

```python
# let's define a complex regression
X = np.linspace(-100, 100, num=500).reshape(-1, 1)
y = np.sin(0.1*X) + np.random.randn(len(X), 1)*0.4
```

```python
plt.scatter(X, y)
```

```python
model = nn.Sequential(
    nn.Linear(1, 10), # 10 hidden units
    nn.ReLU(), # sigmoid activation
    nn.Linear(10, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 1) # final prediction
)
```

```python
Xt = torch.from_numpy(X).type(torch.float32)
yt = torch.from_numpy(y).type(torch.float32)
```

```python
# initial prediction
plt.scatter(X, y)
plt.scatter(X, model.forward(Xt).data.numpy())
plt.legend(['real', 'predicted'])
```

```python
# nn comes with several predefined packages
loss_function = nn.MSELoss(reduction='sum')
learning_rate = 0.00001
```

```python
for i in range(5000):
    yp = model.forward(Xt)
    loss = loss_function(yp, yt)
    # reset gradients
    model.zero_grad()

    loss.backward()

    if i % 500 == 0:
        print(loss)
    # don't grow the graph
    with torch.no_grad():
        for param in model.parameters():
            param.data -= learning_rate * param.grad
```

    tensor(333.6453, grad_fn=<MseLossBackward>)
    tensor(292.4708, grad_fn=<MseLossBackward>)
    tensor(290.1729, grad_fn=<MseLossBackward>)
    tensor(287.1142, grad_fn=<MseLossBackward>)
    tensor(284.0723, grad_fn=<MseLossBackward>)
    tensor(280.1725, grad_fn=<MseLossBackward>)
    tensor(274.9623, grad_fn=<MseLossBackward>)
    tensor(267.6495, grad_fn=<MseLossBackward>)
    tensor(257.6697, grad_fn=<MseLossBackward>)
    tensor(244.2865, grad_fn=<MseLossBackward>)

```python
# initial prediction
plt.scatter(X, y)
plt.scatter(X, model.forward(Xt).data.numpy())
plt.legend(['real', 'predicted'])
```

# Package optim

There are several algorithms better than stochastic gradient descent

```python
model = nn.Sequential(
    nn.Linear(1, 10), # 10 hidden units
    nn.ReLU(), # sigmoid activation
    nn.Linear(10, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 20), # 20 hidden units
    nn.ReLU(),
    nn.Linear(20, 1) # final prediction
)
```

```python
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for i in range(5000):
    yp = model.forward(Xt)
    loss = loss_function(yp, yt)
    # reset gradients
    optimizer.zero_grad()

    loss.backward()

    if i % 500 == 0:
        print(loss)
    optimizer.step()
```

    tensor(315.0819, grad_fn=<MseLossBackward>)
    tensor(299.9580, grad_fn=<MseLossBackward>)
    tensor(294.2961, grad_fn=<MseLossBackward>)
    tensor(291.4652, grad_fn=<MseLossBackward>)
    tensor(289.1334, grad_fn=<MseLossBackward>)
    tensor(286.6578, grad_fn=<MseLossBackward>)
    tensor(283.8851, grad_fn=<MseLossBackward>)
    tensor(280.9672, grad_fn=<MseLossBackward>)
    tensor(277.7617, grad_fn=<MseLossBackward>)
    tensor(274.0918, grad_fn=<MseLossBackward>)

```python
# initial prediction
plt.scatter(X, y)
plt.scatter(X, model.forward(Xt).data.numpy())
plt.legend(['real', 'predicted'])
```

# Custom modules

```python
class TwoLayerNet(torch.nn.Module):
    def __init__(self, D_in, H, D_out):
        """
        In the constructor we instantiate two nn.Linear modules and assign them as
        member variables.
        """
        super(TwoLayerNet, self).__init__()
        self.linear1 = torch.nn.Linear(D_in, H)
        self.linear2 = torch.nn.Linear(H, D_out)

    def forward(self, x):
        """
        In the forward function we accept a Tensor of input data and we must return
        a Tensor of output data. We can use Modules defined in the constructor as
        well as arbitrary (differentiable) operations on Tensors.
        """
        h_relu = self.linear1(x).clamp(min=0)
        y_pred = self.linear2(h_relu)
        return y_pred
```

```python
model = nn.Sequential(
    nn.Linear(1, 10), # 10 hidden units
    nn.ReLU(), # sigmoid activation
    TwoLayerNet(10, 20, 20),
    nn.Linear(20, 1) # final prediction
)
```

```python
optimizer = torch.optim.Adam(model.parameters(), lr=learning_rate)

for i in range(5000):
    yp = model.forward(Xt)
    loss = loss_function(yp, yt)
    # reset gradients
    optimizer.zero_grad()

    loss.backward()

    if i % 500 == 0:
        print(loss)
    optimizer.step()
```

    tensor(1161.2627, grad_fn=<MseLossBackward>)
    tensor(293.3873, grad_fn=<MseLossBackward>)
    tensor(292.4538, grad_fn=<MseLossBackward>)
    tensor(292.3284, grad_fn=<MseLossBackward>)
    tensor(292.1781, grad_fn=<MseLossBackward>)
    tensor(291.9982, grad_fn=<MseLossBackward>)
    tensor(291.7816, grad_fn=<MseLossBackward>)
    tensor(291.5215, grad_fn=<MseLossBackward>)
    tensor(291.2095, grad_fn=<MseLossBackward>)
    tensor(290.8332, grad_fn=<MseLossBackward>)

# Fancy dataset loading

```python
from torch.utils.data import TensorDataset, DataLoader
```

```python
Xt.shape
```

    torch.Size([500, 1])

```python
# randomize the data
dataloader = DataLoader(TensorDataset(Xt, yt), batch_size=250,
                        shuffle=True)
```

```python
for epoch in range(1000):
    for i_batch, sample_batched in enumerate(dataloader):
        yp = model.forward(Xt)
        loss = loss_function(yp, yt)
        # reset gradients
        optimizer.zero_grad()

        loss.backward()

        optimizer.step()
    if epoch % 500 == 0:
        print(loss)
```

    tensor(255.5187, grad_fn=<MseLossBackward>)
    tensor(248.4363, grad_fn=<MseLossBackward>)


---

# Natural Coordinate: Shape Funtion and its derivative

Pramod Kumar Yadav (@iampramodyadav)

---

- This programme calculates the values of shape functions and their derivatives.
- plot the shape funtions

## Function for shape function
- [nptel reference](https://nptel.ac.in/content/storage2/courses/112104116/lecture-11/11_10.htm)
- [SymPy Documentation](https://docs.sympy.org/latest/tutorial/basic_operations.html)

```python
from sympy import *
```

```python
z=Symbol('z')
```

```python
def SHAPE(p,z):
  '''
  SHAPE(p,z)
  p: order (p) of approximation
  z:value of natural coordinate
  This function return shape funtions values at given x
  '''
  z=Symbol('z')
  n=[]

  for i in range(0, p+1):
    point=-1
    point=point+2*i/p
    n.append(point)

  shape=[1]*(p+1)
  for i in range(0,p+1):
    for j in range(0,p+1):
      if i!=j:
        shape[i]=shape[i]*((z-n[j])/(n[i]-n[j]))
  return shape
```

```python
print(SHAPE.__doc__)
```

      SHAPE(p,z)
      p: order (p) of approximation
      z:value of natural coordinate
      This function return shape funtions values at given x

## Function for derivative of shape function
- [SymPy Documentation](https://docs.sympy.org/latest/tutorial/calculus.html)
- [Reference](https://stackoverflow.com/questions/47998930/differentiation-of-a-list-of-functions-with-respect-to-a-list-of-variables)

```python
def dSHAPE(p,z):
  '''
  dSHAPE(p,z)
  p: order (p) of approximation
  z:value of natural coordinate
  This function return derivative of shape funtions values at given x
  '''
  z=Symbol('z')
  f=SHAPE(p,z)
  z=symbols('z')
  return [simplify(diff(f1, z)) for f1 in f]
```

```python
print(dSHAPE.__doc__)
```

      dSHAPE(p,z)
      p: order (p) of approximation
      z:value of natural coordinate
      This function return derivative of shape funtions values at given x

---

```python
df=dSHAPE(3,z)
print('Derivative of shape function:\n',df)
```

    Derivative of shape function:
     [-1.6875*z**2 + 1.125*z + 0.0625000000000001, 5.0625*z**2 - 1.125*z - 1.6875, -5.0625*z**2 - 1.125*z + 1.6875, 1.6875*z**2 + 1.125*z - 0.0624999999999999]

```python
f=SHAPE(3,z)
print('Shape funtion:\n',f ,'\n')

print('Simplified shape funtion:\n',[simplify(f1) for f1 in f])
```

    Shape funtion:
     [(0.25 - 0.75*z)*(0.5 - 0.5*z)*(-1.5*z - 0.5), (0.5 - 1.5*z)*(0.75 - 0.75*z)*(1.5*z + 1.5), (1.5 - 1.5*z)*(0.75*z + 0.75)*(1.5*z + 0.5), (0.5*z + 0.5)*(0.75*z + 0.25)*(1.5*z - 0.5)]

    Simplified shape funtion:
     [-0.5625*z**3 + 0.5625*z**2 + 0.0625*z - 0.0625, 1.125*(z - 1)*(z + 1)*(1.5*z - 0.5), -1.125*(z - 1)*(z + 1)*(1.5*z + 0.5), 0.5*(0.75*z + 0.25)*(z + 1)*(1.5*z - 0.5)]

```python
simplify(f[0]) #first term
```

$\displaystyle - 0.5625 z^{3} + 0.5625 z^{2} + 0.0625 z - 0.0625$

```python
df[0] #deruvative of first term
```

$\displaystyle - 1.6875 z^{2} + 1.125 z + 0.0625000000000001$

---

## Visualization: Plotting shape functions

```python
import seaborn as sns
```

```python
def plotSHAPE(p,z):
  '''
  plotSHAPE(p,z)
  p: order (p) of approximation
  z:value of natural coordinate
  This function return plot of shape funtions
  '''
  sns.set()
  sns.set_style("whitegrid", {'grid.linestyle': '--'})

  z=Symbol('z')
  f=SHAPE(p,z)
  p1=plot(0,(z,-1,1),show=False)
  for i in f:
    p2=plot(i,(z,-1,1),show=False)
    p1.append(p2[0])

  return p1.show()
```

```python
print(plotSHAPE.__doc__)
```

      plotSHAPE(p,z)
      p: order (p) of approximation
      z:value of natural coordinate
      This function return plot of shape funtions

```python
plotSHAPE(1,z)
```

```python
plotSHAPE(2,z)
```

```python
plotSHAPE(3,z)
```

```python
plotSHAPE(4,z)
```

```python
import matplotlib.pyplot as plt
%matplotlib inline
plt.rcParams['figure.figsize'] = 14, 8

plotSHAPE(5,z)
```

---


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c)2014 L.A. Barba, G.F. Forsyth. Partly based on David Ketcheson's pendulum lesson, also under CC-BY.

# Phugoid Oscillation

Welcome back! This is the second IPython Notebook of the series _"The phugoid model of glider flight"_, the first learning module of the course [**"Practical Numerical Methods with Python."**](http://openedx.seas.gwu.edu/courses/GW/MAE6286/2014_fall/about)

In the first notebook, _"Phugoid Motion"_, we described the physics of an aircraft's oscillatory trajectory, seen as an exchange of kinetic and potential energy. This analysis goes back to Frederick Lanchester, who published his book _"Aerodonetics"_ on aircraft stability in 1909. We concluded that first exposure to our problem of interest by plotting the flight paths predicted by Lanchester's analysis, known as _phugoids_.

Here, we will look at the situation when an aircraft is initially moving on the straight-line phugoid (obtained with the parameters $C=2/3$, $\cos\theta=1$, and $z=z_t$ in the previous analysis), and experiences a small upset, a wind gust that slightly perturbs its path. It will then enter into a gentle oscillation around the previous straight-line path: a _phugoid oscillation_.

If the aircraft experiences an upward acceleration of $-d^2z/dt^2$, and we assume that the perturbation is small, then $\cos\theta=1$ is a good approximation and Newton's second law in the vertical direction is:

$$\begin{equation}
L - W = - \frac{W}{g}\frac{d^2 z}{dt^2}
\end{equation}$$

In the previous notebook, we saw that the following relation holds for the ratio of lift to weight, in terms of the trim velocity $v_t$:

$$\begin{equation}
\frac{L}{W}=\frac{v^2}{v_t^2}
\end{equation}$$

This will be useful: we can divide Equation (1) by the weight and use Equation (2) to replace $L/W$. Another useful relation from the previous notebook expressed the conservation of energy (per unit mass) as $v^2 = 2 gz$. With this, Equation (1) is rearranged as:

$$\begin{equation}
\frac{d^2z}{dt^2} + \frac{gz}{z_t} = g
\end{equation}$$

Look at Equation (3) for a moment. Does it ring a bell? Do you recognize it?

If you remember from your physics courses the equation for _simple harmonic motion_, you should see the similarity!

Take the case of a simple spring. Hooke's law is $F=-kx$, where $F$ is a restoring force, $x$ the displacement from a position of equilibrium and $k$ the spring constant. This results in the following ordinary differential equation for the displacement:

$$\begin{equation}
 \frac{d^2 x}{dt^2}= -\frac{k}{m}x
\end{equation}$$

which has the solution $x(t) = A \cos(\omega t- \phi)$, representing simple harmonic motion with an angular frequency $\omega=\sqrt{k/m}=2\pi f$ and phase angle $\phi$.

Now look back at Equation (3): it has the same form and it represents simple harmonic motion with angular frequency $\omega=\sqrt{g/z_t}$.

Think about this for a moment ... we can immediately say what the period of the oscillation is: exactly $2 \pi \sqrt{z_t/g}$ — or, in terms of the trim velocity, $\pi \sqrt{2} v_t/g$.

_This is a remarkable result!_ Think about it: we know nothing about the aircraft, or the flight altitude, yet we can obtain the period of the phugoid oscillation simply as a function of the trim velocity. For example, if trim velocity is 200 knots, we get a phugoid period of about 47 seconds—over that time, you really would not notice anything if you were flying in that aircraft.

Next, we want to be able to compute the trajectory of the aircraft for a given initial perturbance. We will do this by numerically integrating the equation of motion.

## Prepare to integrate

We want to integrate the differential equation and plot the trajectory of the aircraft. Are you ready?

The equation for the phugoid oscillation is a second-order, ordinary differential equation (ODE). Let's represent the time derivative with a prime, and write it like this:

$$\begin{equation}
z(t)'' + \frac{g \,z(t)}{z_t}=g
\end{equation}$$

There's a convenient trick when we work with ODEs: we can turn this 2nd-order equation into a system of two 1st-order equations. Like this:

$$\begin{eqnarray}
z'(t) &=& v(t)\\
v'(t) &=& g\left(1-\frac{z(t)}{z_t}\right)
\end{eqnarray}$$

Are you following? Make sure you are following the derivations, even if it means writing the equations down in your own notes! (Yes, the old-fashioned paper way.)

Another way to look at a system of two 1st-order ODEs is by using vectors. You can make a vector with your two independent variables,

$$\begin{equation}
\vec{u}  = \begin{pmatrix} z \\ v \end{pmatrix}
\end{equation}$$

and write the differential system as a single vector equation:

\begin{equation}
\vec{u}'(t)  = \begin{pmatrix} v \\ g-g\frac{z(t)}{z_t} \end{pmatrix}
\end{equation}

If you call the right-hand-side $\vec{f}(\vec{u})$, then the equation is very short: $\vec{u}'(t) = \vec{f}(\vec{u})$—but let's drop those arrows to denote vectors from now on, as they are a bit cumbersome: just remember that $u$ and $f$ are vectors in the phugoid equation of motion.

Next, we'll prepare to solve this problem numerically.

## Initial value problems

Let's step back for a moment. Suppose we have a first-order ODE $u'=f(u)$. You know that if we were to integrate this, there would be an arbitrary constant of integration. To find its value, we do need to know one point on the curve $(t, u)$. When the derivative in the ODE is with respect to time, we call that point the _initial value_ and write something like this:

$$u(t=0)=u_0$$

In the case of a second-order ODE, we already saw how to write it as a system of first-order ODEs, and we would need an initial value for each equation: two conditions are needed to determine our constants of integration. The same applies for higher-order ODEs: if it is of order $n$, we can write it as $n$ first-order equations, and we need $n$ known values. If we have that data, we call the problem an _initial value problem_.

Remember the definition of a derivative? The derivative represents the slope of the tangent at a point of the curve $u=u(t)$, and the definition of the derivative $u'$ for a function is:

$$u'(t) = \lim_{\Delta t\rightarrow 0} \frac{u(t+\Delta t)-u(t)}{\Delta t}$$

If the step $\Delta t$ is already very small, we can _approximate_ the derivative by dropping the limit. We can write:

$$\begin{equation}
u(t+\Delta t) \approx u(t) + u'(t) \Delta t
\end{equation}$$

With this equation, and because we know $u'(t)=f(u)$, if we have an initial value, we can step by $\Delta t$ and find the value of $u(t+\Delta t)$, then we can take this value, and find $u(t+2\Delta t)$, and so on: we say that we _step in time_, numerically finding the solution $u(t)$ for a range of values: $t_1, t_2, t_3 \cdots$, each separated by $\Delta t$. The numerical solution of the ODE is simply the table of values $t_i, u_i$ that results from this process.

## Discretization

In order to execute the process described above and find the numerical solution of the ODE, we start by choosing the values $t_1,t_2,t_3 \cdots t_n$—we call these values our *grid* in time. The first point of the grid is given by our _initial value_, and the small difference between two consecutive times is called the _time step_, denoted by $\Delta t$.  The solution value at time $t_n$ is denoted by $u_n$.

Let's build a time grid for our problem. We first choose a final time $T$ and the time step $\Delta t$. In code, we'll use readily identifiable variable names: `T` and `dt`, respectively. With those values set, we can calculate the number of time steps that will be needed to reach the final time; we call that variable `N`.

Let's write some code. The first thing we do in Python is load our favorite libraries: NumPy for array operations, and the Pyplot module in Matplotlib, to later on be able to plot the numerical solution. The line `%matplotlib inline` tells IPython Notebook to show the plots inline.

```
import numpy
import matplotlib.pyplot as plt
%matplotlib inline
```

Now, initialize `T` and `dt`, calculate `N` and build a NumPy array with all the values of time that make up the grid.

```
T = 100.0
dt = 0.01
N = int(T/dt)+1
t = numpy.linspace(0.0, T, N)
```

We have our grid! Now it's time to apply the numerical time stepping represented by Equation (10).

##### Challenge!

* Write the code above using the NumPy function `arange()` instead of `linspace()`. If you need to, read the documentation for these functions.

##### Pro tip:

Enter a question mark followed by any function, e.g., `?numpy.linspace`, into a code cell and execute it, to get a help pane on the notebook.

## Euler's method

The approximate solution at time $t_n$ is $u_n$, and the numerical solution of the differential equation consists of computing a sequence of approximate solutions by the following formula, based on Equation (10):

$$u_{n+1} = u_n + \Delta t \,f(u_n).$$

This formula is called **Euler's method**.

For the equations of the phugoid oscillation, Euler's method gives the following algorithm that we need to implement in code:

\begin{align}
z_{n+1} & = z_n + \Delta t \, v_n \\
v_{n+1} & = v_n + \Delta t \left(g - \frac{g}{z_t} \, z_n \right).
\end{align}

### And solve!

To apply the numerical solution method, we need to set things up in code: define the parameter values needed in the model, initialize a NumPy array to hold the discrete solution values, and initialize another array for the angular values.

```
# initial conditions
z0 = 100.  #altitude
v  = 10   #upward velocity resulting from gust
zt = 100.
g  = 9.81

u = numpy.array([z0, v])

# initialize an array to hold the changing angle values
z = numpy.zeros(N)
z[0] = z0
```

You should pay attention to a couple of things: (1) See how there is a dot after the numbers used to define our parameters? We just want to be explicit (as a good habit) that these variables are real numbers, called "floats." (2) We both _created_ and _initialized_ with zeros everywhere the solution vector `z`. Look up the documentation for the handy NumPy function `zeros()`, if you need to. (3) In the last line above, we assign the _initial value_ to the first element of the solution vector: `z[0]`.

Now we can step in time using Euler's method. Notice how we are time stepping the two independend variables at once in the time iterations.

```
# time-loop using Euler's method
for n in range(1,N):
    u = u + dt*numpy.array([u[1], g*(1-u[0]/zt)])
    z[n] = u[0]
```

Make sure you understand what this code is doing. This is a basic pattern in numerical methods: iterations in a time variable that apply a numerical scheme at each step.

## Plot the solution

If the code is correct, we have stored in the array `z` the position of the glider at each time. Let's use Matplotlib to examine the flight path of the arcraft.

You should explore the [Matplotlib tutorial](http://matplotlib.org/users/pyplot_tutorial.html) (if you need to) and familiarize yourself with the command-style functions that control the size, labels, line style, and so on. Creating good plots is a useful skill: it is about communicating your results effectively.

Here, we set the figure size, the limits of the vertical axis, the format of tick-marks, and axis labels. The final line actually produces the plot, with our chosen line style (continuous black line).

```
plt.figure(figsize=(10,4))   #set plot size
plt.ylim(40,160)             #y-axis plot limits
plt.tick_params(axis='both', labelsize=14) #increase font size for ticks
plt.xlabel('t', fontsize=14) #x label
plt.ylabel('z', fontsize=14) #y label
plt.plot(t,z, 'k-');
```

##### Explore and think

Try changing the value of `v` in the initial conditions.

* What happens when you have a larger gust?
* What about a smaller gust?
* What happens if there isn't a gust (`v = 0`)?

## Exact solution

The equation for phugoid oscillations is a 2nd-order, linear ODE and it has an exact solution of the following form:

$$
\begin{equation}
z(t) = A \sin \left(\sqrt{\frac{g}{z_t}} t \right) + B \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}
$$

where $A$ and $B$ are constants that we solve for using initial conditions.

Our numerical solution used the initial conditions:

$$\begin{eqnarray}
z(0) = z_0 \\
v(0) = v_0
\end{eqnarray}$$

Applying these to the exact solution and solving for $A$ and $B$, we get:

$$\begin{equation}
z(t) = v_0 \sqrt{\frac{z_t}{g}} \sin \left(\sqrt{\frac{g}{z_t}} t \right) + (z_0-z_t) \cos \left(\sqrt{\frac{g}{z_t}} t \right) + z_t
\end{equation}$$

We already defined all of these variables for our numerical solution, so we can immediately compute the exact solution.

##### Pro tip:

The expression is a  bit long —if you don't feel like scrolling left and right, you can insert a line break in the code using a backslash '\' _(without any trailing spaces)_ and Python will treat the next line as a continuation of the first.

```
z_exact = v*(zt/g)**.5*numpy.sin((g/zt)**.5*t)+\
            (z0-zt)*numpy.cos((g/zt)**.5*t)+zt
```

## Plot the exact solution

Now we can plot our exact solution!  Even better, we can plot _both_ the numerical solution *and* the exact solution to see how well Euler's method approximated the phugoid oscillations.

To add another curve to a plot, simply type a second `plt.plot()` statement.  We also added a legend using the format

```Python
plt.legend(["line 1","line 2"])
```

```
plt.figure(figsize=(10,4))
plt.ylim(40,160)             #y-axis plot limits
plt.tick_params(axis='both', labelsize=14) #increase font size for ticks
plt.xlabel('t', fontsize=14) #x label
plt.ylabel('z', fontsize=14) #y label
plt.plot(t,z)
plt.plot(t, z_exact)
plt.legend(['Numerical Solution','Analytical Solution']);
```

That looks like pretty good agreement, but what's happening towards the end? We'll come back to this.

Euler's method, like all numerical methods, introduces some errors.  If the method is *convergent*, the approximation will get closer and closer to the exact solution as we reduce the size of the step, $\Delta t$. The error in the numerical method should tend to zero, in fact, when $\Delta t\rightarrow 0$—when this happens, we call the method _consistent_. We'll define these terms more carefully in the theory components of this course.

## Convergence

To compare the two solutions, we need to use a **norm** of the difference, like the $L_1$ norm, for example.

$$ E = \Delta t \sum_{n=0}^N \left|z(t_n) - z_n\right|$$

The $L_1$ norm is the sum of the individual differences between the exact and the numerical solutions, at each mesh point.

We check for convergence by calculating the numerical solution using progressively smaller values of `dt`. We already have most of the code that we need.  We just need to add an extra loop and an array of different $\Delta t$ values to iterate through.

You should read the documentation for the function [enumerate](https://docs.python.org/2/library/functions.html#enumerate) and make sure you understand how we're using it below.

##### Warning

The cell below can take a little while to finish (the last $\Delta t$ value alone requires 1 million iterations!).  If the cell is still running, the input label will say `In [*]`.  When it finishes, the `*` will be replaced by a number.

```
# time-increment array
dt_values = numpy.array([0.1, 0.05, 0.01, 0.005, 0.001, 0.0001])

# array that will contain solution of each grid
z_values = numpy.empty_like(dt_values, dtype=numpy.ndarray)

for i, dt in enumerate(dt_values):
    N = int(T/dt)+1    # number of time-steps
    ### discretize the time using numpy.linspace() ###
    t = numpy.linspace(0.0, T, N)

    # initial conditions
    u = numpy.array([z0, v])
    z = numpy.empty_like(t)
    z[0] = z0

    # time loop - Euler method
    for n in range(1,N):
        ### compute next solution using Euler method ###
        u = u + dt*numpy.array([u[1], g*(1-u[0]/zt)])
        z[n] = u[0]   # store the elevation at time-step n+1

    z_values[i] = z.copy()    # store the total elevation calculation grid i
```

### Calculate the error

We now have numerical solutions for each $\Delta t$ in the array `z_values`.  To calculate the error corresponding to each $\Delta t$, we can write a function!

```
def get_error(z, dt):
    """Returns the error relative to analytical solution using L-1 norm.

    Parameters
    ----------
    z : array of float
        numerical solution.
    dt : float
        time increment.

    Returns
    -------
    err : float
        L_{1} norm of the error with respect to the exact solution.
    """
    N = len(z)
    t = numpy.linspace(0.0, T, N)

    z_exact = v*(zt/g)**.5*numpy.sin((g/zt)**.5*t)+\
                (z0-zt)*numpy.cos((g/zt)**.5*t)+zt

    return dt * numpy.sum(numpy.abs(z-z_exact))
```

**Note**: in the last line of the function, we perform an 'array operation':

`z - z_exact`

We are *not* subtracting one value from another.  Instead, we are taking the difference between elements at each corresponding index in both arrays.  Here is a quick example:

```
a = numpy.array([1, 2, 3])
b = numpy.array([4, 4, 4])

b - a
```

    array([3, 2, 1])

Now, we iterate through each $\Delta t$ value and calculate the corresponding error.

```
error_values = numpy.empty_like(dt_values)

for i, dt in enumerate(dt_values):
    ### call the function get_error() ###
    error_values[i] = get_error(z_values[i], dt)
```

Remember, *if* the method is convergent then the error should get smaller as  $\Delta t$ gets smaller.  To visualize this, let's plot $\Delta t$ vs. error.  If you use `plt.plot` you won't get a very useful result.  Instead, use `plt.loglog` to create the same plot with a log-log scale.  This is what we do almost always to assess the errors of a numerical scheme graphically.

```
plt.figure(figsize=(10, 6))
plt.tick_params(axis='both', labelsize=14) #increase tick font size
plt.grid(True)                         #turn on grid lines
plt.xlabel('$\Delta t$', fontsize=16)  #x label
plt.ylabel('Error', fontsize=16)       #y label
plt.loglog(dt_values, error_values, 'ko-')  #log-log plot
plt.axis('equal')                      #make axes scale equally;
```

This is the kind of result we like to see!  As $\Delta t$ shrinks (towards the left), the error gets smaller and smaller, like it should.

##### Challenge!

We calculated the error for several different timestep sizes using two nested `for` loops.  That worked, but whenever possible, we like to re-use code (and not just copy and paste it!).

Create a function that implements Euler's method and re-write the error analysis cell to use your function.

---

######The cell below loads the style of this notebook.

```
from IPython.core.display import HTML
css_file = '../../styles/numericalmoocstyle.css'
HTML(open(css_file, "r").read())
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 750px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.8em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Alegreya Sans' sans-serif;
    line-height: 140%;
    font-size: 125%;
    font-weight: 400;
    width:600px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Nixie One', serif;
    font-style:regular;
    font-weight: 400;
    font-size: 45pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Nixie One', serif;
    font-weight: 400;
    font-size: 30pt;
    line-height: 100%;
    color: rgb(0,51,102);
    margin-bottom: 0.1em;
    margin-top: 0.3em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Nixie One', serif;
    margin-top:16px;
	font-size: 22pt;
    font-weight: 600;
    margin-bottom: 3px;
    font-style: regular;
    color: rgb(102,102,0);
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Nixie One', serif;
    font-size: 14pt;
    text-align: center;
    margin-top: 0em;
    margin-bottom: 2em;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Nixie One', sans-serif;
    font-weight: 400;
    font-size: 16pt;
    color: rgb(163,0,0);
    font-style: italic;
    margin-bottom: .1em;
    margin-top: 0.8em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'PT Mono', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

.CodeMirror{
        font-family: "PT Mono";
        font-size: 90%;
}

</style>


# Simple pendulum using Lagrange's equation

Defines a LagrangianPendulum class that is used to generate basic pendulum plots from solving Lagrange's equations.

* Last revised 17-Mar-2019 by Dick Furnstahl (furnstahl.1@osu.edu).

## Euler-Lagrange equation

For a simple pendulum, the Lagrangian with generalized coordinate $\phi$ is

$\begin{align}
  \mathcal{L} = \frac12 m L^2 \dot\phi^2 - mgL(1 - \cos\phi)
\end{align}$

The Euler-Lagrange equation is

$\begin{align}
 \frac{d}{dt}\frac{\partial\mathcal{L}}{\partial \dot\phi} = \frac{\partial\mathcal L}{\partial\phi}
 \quad\Longrightarrow\quad
 m L^2 \ddot \phi = -mgL\sin\phi
  \ \mbox{or}\ \ddot\phi = - \omega_0^2\sin\phi = 0
  \;.
\end{align}$

## Hamilton's equations

The generalized momentum corresponding to $\phi$ is

$\begin{align}
   \frac{\partial\mathcal{L}}{\partial \dot\phi} = m L^2 \dot\phi \equiv p_\phi
   \;.
\end{align}$

We can invert this equation to find $\dot\phi = p_\phi / m L^2$.
Constructing the Hamiltonian by Legendre transformation we find

$\begin{align}
  \mathcal{H} &= \dot\phi p_\phi  - \mathcal{L} \\
              &= \frac{p_\phi^2}{m L^2} - \frac12 m L^2 \dot\phi^2 + mgL(1 - \cos\phi) \\
              &= \frac{p_\phi^2}{2 m L^2} + mgL(1 - \cos\phi)
              \;.
\end{align}$

Thus $\mathcal{H}$ is simply $T + V$.  Hamilton's equations are

$\begin{align}
  \dot\phi &= \frac{\partial\mathcal{H}}{\partial p_\phi} = \frac{p_\phi}{m L^2} \\
  \dot p_\phi &= -\frac{\partial\mathcal{H}}{\partial \phi} = -mgL \sin\phi
  \;.
\end{align}$

```python
%matplotlib inline
```

```python
import numpy as np
from scipy.integrate import odeint, solve_ivp

import matplotlib.pyplot as plt
```

```python
# The dpi (dots-per-inch) setting will affect the resolution and how large
#  the plots appear on screen and printed.  So you may want/need to adjust
#  the figsize when creating the figure.
plt.rcParams['figure.dpi'] = 100.    # this is the default for notebook

# Change the common font size (smaller when higher dpi)
font_size = 12
plt.rcParams.update({'font.size': font_size})
```

## Pendulum class and utility functions

```python
class LagrangianPendulum():
    """
    Pendulum class implements the parameters and Lagrange's equations for
     a simple pendulum (no driving or damping).

    Parameters
    ----------
    L : float
        length of the simple pendulum
    g : float
        gravitational acceleration at the earth's surface
    omega_0 : float
        natural frequency of the pendulum (\sqrt{g/l} where l is the
        pendulum length)
    mass : float
        mass of pendulum

    Methods
    -------
    dy_dt(t, y)
        Returns the right side of the differential equation in vector y,
        given time t and the corresponding value of y.
    """
    def __init__(self, L=1., mass=1., g=1.
                ):
        self.L = L
        self.g = g
        self.omega_0 = np.sqrt(g/L)
        self.mass = mass

    def dy_dt(self, t, y):
        """
        This function returns the right-hand side of the diffeq:
        [dphi/dt d^2phi/dt^2]

        Parameters
        ----------
        t : float
            time
        y : float
            A 2-component vector with y[0] = phi(t) and y[1] = dphi/dt

        Returns
        -------

        """
        return [y[1], -self.omega_0**2 * np.sin(y[0]) ]

    def solve_ode(self, t_pts, phi_0, phi_dot_0,
                  abserr=1.0e-9, relerr=1.0e-9):
        """
        Solve the ODE given initial conditions.
        Specify smaller abserr and relerr to get more precision.
        """
        y = [phi_0, phi_dot_0]
        solution = solve_ivp(self.dy_dt, (t_pts[0], t_pts[-1]),
                             y, t_eval=t_pts,
                             atol=abserr, rtol=relerr)
        phi, phi_dot = solution.y

        return phi, phi_dot
```

```python
def plot_y_vs_x(x, y, axis_labels=None, label=None, title=None,
                color=None, linestyle=None, semilogy=False, loglog=False,
                ax=None):
    """
    Generic plotting function: return a figure axis with a plot of y vs. x,
    with line color and style, title, axis labels, and line label
    """
    if ax is None:        # if the axis object doesn't exist, make one
        ax = plt.gca()

    if (semilogy):
        line, = ax.semilogy(x, y, label=label,
                            color=color, linestyle=linestyle)
    elif (loglog):
        line, = ax.loglog(x, y, label=label,
                          color=color, linestyle=linestyle)
    else:
        line, = ax.plot(x, y, label=label,
                    color=color, linestyle=linestyle)

    if label is not None:    # if a label if passed, show the legend
        ax.legend()
    if title is not None:    # set a title if one if passed
        ax.set_title(title)
    if axis_labels is not None:  # set x-axis and y-axis labels if passed
        ax.set_xlabel(axis_labels[0])
        ax.set_ylabel(axis_labels[1])

    return ax, line
```

```python
def start_stop_indices(t_pts, plot_start, plot_stop):
    start_index = (np.fabs(t_pts-plot_start)).argmin()  # index in t_pts array
    stop_index = (np.fabs(t_pts-plot_stop)).argmin()  # index in t_pts array
    return start_index, stop_index
```

```python

```

## Make simple pendulum plots

```python
# Labels for individual plot axes
phi_vs_time_labels = (r'$t$', r'$\phi(t)$')
phi_dot_vs_time_labels = (r'$t$', r'$d\phi/dt(t)$')
state_space_labels = (r'$\phi$', r'$d\phi/dt$')

# Common plotting time (generate the full time then use slices)
t_start = 0.
t_end = 50.
delta_t = 0.001

t_pts = np.arange(t_start, t_end+delta_t, delta_t)

L = 1.
g = 1.
mass = 1.

# Instantiate a pendulum
p1 = LagrangianPendulum(L=L, g=g, mass=mass)

```

```python
# both plots: same initial conditions
phi_0 = (3./4.)*np.pi
phi_dot_0 = 0.
phi, phi_dot = p1.solve_ode(t_pts, phi_0, phi_dot_0)

# start the plot!
fig = plt.figure(figsize=(15,5))
overall_title = 'Simple pendulum from Lagrangian:  ' + \
                rf' $\omega_0 = {p1.omega_0:.2f},$' + \
                rf'  $\phi_0 = {phi_0:.2f},$' + \
                rf' $\dot\phi_0 = {phi_dot_0:.2f}$' + \
                '\n'     # \n means a new line (adds some space here)
fig.suptitle(overall_title, va='baseline')

# first plot: phi plot
ax_a = fig.add_subplot(1,3,1)

start, stop = start_stop_indices(t_pts, t_start, t_end)
plot_y_vs_x(t_pts[start : stop], phi[start : stop],
            axis_labels=phi_vs_time_labels,
            color='blue',
            label=None,
            title=r'$\phi(t)$',
            ax=ax_a)

# second plot: phi_dot plot
ax_b = fig.add_subplot(1,3,2)

start, stop = start_stop_indices(t_pts, t_start, t_end)
plot_y_vs_x(t_pts[start : stop], phi_dot[start : stop],
            axis_labels=phi_dot_vs_time_labels,
            color='blue',
            label=None,
            title=r'$\dot\phi(t)$',
            ax=ax_b)

# third plot: state space plot from t=30 to t=50
ax_c = fig.add_subplot(1,3,3)

start, stop = start_stop_indices(t_pts, t_start, t_end)
plot_y_vs_x(phi[start : stop], phi_dot[start : stop],
            axis_labels=state_space_labels,
            color='blue',
            label=None,
            title='State space',
            ax=ax_c)

fig.tight_layout()
fig.savefig('simple_pendulum_Lagrange.png', bbox_inches='tight')
```

Now trying the power spectrum, plotting only positive frequencies and cutting off the lower peaks:

```python
start, stop = start_stop_indices(t_pts, t_start, t_end)
signal = phi[start:stop]
power_spectrum = np.abs(np.fft.fft(signal))**2
freqs = np.fft.fftfreq(signal.size, delta_t)
idx = np.argsort(freqs)

fig_ps = plt.figure(figsize=(5,5))
ax_ps = fig_ps.add_subplot(1,1,1)
ax_ps.semilogy(freqs[idx], power_spectrum[idx], color='blue')
ax_ps.set_xlim(0, 1.)
ax_ps.set_ylim(1.e5, 1.e11)
ax_ps.set_xlabel('frequency')
ax_ps.set_title('Power Spectrum')

fig_ps.tight_layout()
```

```python

```