# 17 PDEs: Solution with Time Stepping

## Heat Equation
The **heat equation** can be derived from Fourier's law and energy conservation (see the [lecture notes on the heat equation (PDF)](https://github.com/ASU-CompMethodsPhysics-PHY494/PHY494-resources/blob/master/17_PDEs/17_PDEs_LectureNotes_HeatEquation.pdf))

$$
\frac{\partial T(\mathbf{x}, t)}{\partial t} = \frac{K}{C\rho} \nabla^2 T(\mathbf{x}, t),
$$

## Problem: insulated metal bar (1D heat equation)
A metal bar of length $L$ is insulated along it lengths and held at 0ºC at its ends. Initially, the whole bar is at 100ºC. Calculate $T(x, t)$ for $t>0$.

### Analytic solution
Solve by separation of variables and power series: The general solution that obeys the boundary conditions $T(0, t) = T(L, t) = 0$ is

$$
T(x, t) = \sum_{n=1}^{+\infty} A_n \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right), \quad k_n = \frac{n\pi}{L}
$$

The specific solution that satisfies $T(x, 0) = T_0 = 100^\circ\text{C}$ leads to $A_n = 4 T_0/n\pi$ for $n$ odd:

$$
T(x, t) = \sum_{n=1,3,5,\dots}^{+\infty} \frac{4 T_0}{n \pi} \sin(k_n x)\, \exp\left(-\frac{k_n^2 K t}{C\rho}\right)
$$

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
plt.style.use('ggplot')
```

```python
def T_bar(x, t, T0, L, K=237, C=900, rho=2700, nmax=1000):
    T = np.zeros_like(x)
    eta = K / (C*rho)
    for n in range(1, nmax, 2):
        kn = n*np.pi/L
        T += 4*T0/(np.pi * n) * np.sin(kn*x) * np.exp(-kn*kn * eta * t)
    return T
```

```python
T0 = 100.
L = 1.0
X = np.linspace(0, L, 100)
for t in np.linspace(0, 3000, 50):
    plt.plot(X, T_bar(X, t, T0, L))
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ ($^\circ$C)");
```

### Numerical solution: Leap frog
Discretize (finite difference):

For the time domain we only have the initial values so we use a simple forward difference for the time derivative:

$$
\frac{\partial T(x,t)}{\partial t} \approx \frac{T(x, t+\Delta t) - T(x, t)}{\Delta t}
$$

For the spatial derivative we have initially all values so we can use the more accurate central difference approximation:

$$
\frac{\partial^2 T(x, t)}{\partial x^2} \approx \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

Thus, the heat equation can be written as the finite difference equation

$$
\frac{T(x, t+\Delta t) - T(x, t)}{\Delta t} = \frac{K}{C\rho} \frac{T(x+\Delta x, t) + T(x-\Delta x, t) - 2 T(x, t)}{\Delta x^2}
$$

which can be reordered so that the RHS contains only known terms and the LHS future terms. Index $i$ is the spatial index, and $j$ the time index: $x = x_0 + i \Delta x$, $t = t_0 + j \Delta t$.

$$
T_{i, j+1} = (1 - 2\eta) T_{i,j} + \eta(T_{i+1,j} + T_{i-1, j}), \quad \eta := \frac{K \Delta t}{C \rho \Delta x^2}
$$

Thus we can step forward in time ("leap frog"), using only known values.

### Solve the 1D heat equation numerically for an iron bar
* $K = 237$ W/mK
* $C = 900$ J/K
* $\rho = 2700$ kg/m3
* $L = 1$ m
* $T_0 = 373$ K and $T_b = 273$ K
* $T(x, 0) = T_0$ and $T(0, t) = T(L, t) = T_b$

#### Key considerations

The key line is the computation of the new temperature field at time step $j+1$ from the temperature distribution at time step $j$. It can be written purely with numpy array operations (see last lecture!):

```python
T[1:-1] = (1 - 2*eta) * T[1:-1] + eta * (T[2:] + T[:-2])
```

Note that the range operator `T[start:end]` *excludes* `end`, so in order to include `T[1], T[2], ..., T[-2]` (but not the rightmost `T[-1]`) we have to use `T[1:-1]`.

The *boundary conditions* are fixed for all times:
```python
T[0] = T[-1] = Tb
```

The *initial conditions* (at time step `j=0`)
```python
T[1:-1] = T0
```
are only used to compute the distribution of temperatures at the next step `j=1`.

#### Solution

```python
import numpy as np

import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
%matplotlib notebook
```

For HTML/nbviewer output, use inline:

```python
%matplotlib inline
```

```python
L_rod = 1.    # m
t_max = 3000. # s

Dx = 0.02   # m
Dt = 2    # s

Nx = int(L_rod // Dx)
Nt = int(t_max // Dt)

Kappa = 237 # W/(m K)
CHeat = 900 # J/K
rho = 2700  # kg/m^3

T0 = 373 # K
Tb = 273 # K

eta = Kappa * Dt / (CHeat * rho * Dx**2)
eta2 = 1 - 2*eta

step = 20  # plot solution every n steps

print("Nx = {0}, Nt = {1}".format(Nx, Nt))
print("eta = {0}".format(eta))

T = np.zeros(Nx)
T_plot = np.zeros((Nt//step + 1, Nx))

# initial conditions
T[1:-1] = T0
# boundary conditions
T[0] = T[-1] = Tb

t_index = 0
T_plot[t_index, :] = T
for jt in range(1, Nt):
    T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        T_plot[t_index, :] = T
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
```

    Nx = 49, Nt = 1500
    eta = 0.4876543209876543
    Completed  1499 iterations: t=2998 s

#### Visualization
Visualize (you can use the code as is).

Note how we are making the plot use proper units by mutiplying with `Dt * step` and `Dx`.

```python
X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
Z = T_plot[X, Y]
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
ax.set_xlabel(r"time $t$ (s)")
ax.set_ylabel(r"position $x$ (m)")
ax.set_zlabel(r"temperature $T$ (K)")
fig.tight_layout()
```

2D as above for the analytical solution…

```python
X = Dx * np.arange(T_plot.shape[1])
plt.plot(X, T_plot.T)
plt.xlabel(r"$x$ (m)")
plt.ylabel(r"$T$ (K)");
```

#### Slower solution

I benchmarked this slow solution at 89.7 ms and the fast solution at 14.8 ms (commented out all `print`) so the explicit loop is not that much worse (probably because the overhead on array copying etc is high).

```python
L_rod = 1.    # m
t_max = 3000. # s

Dx = 0.02   # m
Dt = 2    # s

Nx = int(L_rod // Dx)
Nt = int(t_max // Dt)

Kappa = 237 # W/(m K)
CHeat = 900 # J/K
rho = 2700  # kg/m^3

T0 = 373 # K
Tb = 273 # K

eta = Kappa * Dt / (CHeat * rho * Dx**2)
eta2 = 1 - 2*eta

step = 20  # plot solution every n steps

print("Nx = {0}, Nt = {1}".format(Nx, Nt))
print("eta = {0}".format(eta))

T = np.zeros(Nx)
T_new = np.zeros_like(T)
T_plot = np.zeros((int(np.ceil(Nt/step)) + 1, Nx))

# initial conditions
T[1:-1] = T0
# boundary conditions
T[0] = T[-1] = Tb

T_new[:] = T

t_index = 0
T_plot[t_index, :] = T
for jt in range(1, Nt):
    # T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
    for ix in range(1, Nx-1):
        T_new[ix] = eta2 * T[ix] + eta*(T[ix+1] + T[ix-1])
    T[:] = T_new
    if jt % step == 0 or jt == Nt-1:
        t_index += 1
        T_plot[t_index, :] = T
        print("Iteration {0:5d}".format(jt), end="\r")
else:
    print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
```

```python
X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
Z = T_plot[X, Y]
fig = plt.figure()
ax = fig.add_subplot(111, projection="3d")
ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
ax.set_xlabel(r"time $t$ (s)")
ax.set_ylabel(r"position $x$ (m)")
ax.set_zlabel(r"temperature $T$ (K)")
fig.tight_layout()
```

## Stability of the solution

### Empirical investigation of the stability
Investigate the solution for different values of `Dt` and `Dx`. Can you discern patters for stable/unstable solutions?

Report `Dt`, `Dx`, and `eta`
* for 3 stable solutions
* for 3 unstable solutions

```python
def calculate_T(L_rod=1, t_max=3000, Dx=0.02, Dt=2, T0=373, Tb=273,
               step=20):
    Nx = int(L_rod // Dx)
    Nt = int(t_max // Dt)

    Kappa = 237 # W/(m K)
    CHeat = 900 # J/K
    rho = 2700  # kg/m^3

    eta = Kappa * Dt / (CHeat * rho * Dx**2)
    eta2 = 1 - 2*eta

    print("Nx = {0}, Nt = {1}".format(Nx, Nt))
    print("eta = {0}".format(eta))

    T = np.zeros(Nx)
    T_plot = np.zeros((int(np.ceil(Nt/step)) + 1, Nx))

    # initial conditions
    T[1:-1] = T0
    # boundary conditions
    T[0] = T[-1] = Tb

    t_index = 0
    T_plot[t_index, :] = T
    for jt in range(1, Nt):
        T[1:-1] = eta2 * T[1:-1] + eta*(T[2:] + T[:-2])
        if jt % step == 0 or jt == Nt-1:
            t_index += 1
            T_plot[t_index, :] = T
            print("Iteration {0:5d}".format(jt), end="\r")
    else:
        print("Completed {0:5d} iterations: t={1} s".format(jt, jt*Dt))
    return T_plot

def plot_T(T_plot, Dx, Dt, step):
    X, Y = np.meshgrid(range(T_plot.shape[0]), range(T_plot.shape[1]))
    Z = T_plot[X, Y]
    fig = plt.figure()
    ax = fig.add_subplot(111, projection="3d")
    ax.plot_wireframe(X*Dt*step, Y*Dx, Z)
    ax.set_xlabel(r"time $t$ (s)")
    ax.set_ylabel(r"position $x$ (m)")
    ax.set_zlabel(r"temperature $T$ (K)")
    fig.tight_layout()
    return ax
```

```python
Dx, Dt, step = 0.01, 2, 20
T_plot = calculate_T(Dx=Dx, Dt=Dt, step=step)
plot_T(T_plot, Dx, Dt, step)
```

Note that *decreasing* the value of $\Delta x$ made the solution *unstable*. This is strange, we have gotten used to the idea that working on a finer mesh will increase the detail (until we hit round-off error) and just become computationally more expensive. But here the algorithm suddenly becomes unstable (and it is not just round-off).

For certain combination of values of $\Delta t$ and $\Delta x$ the solution become unstable. Empirically, bigger $\eta$ leads to instability. (In fact, $\eta \geq \frac{1}{2}$ is unstable for the leapfrog algorithm as we will see.)

### Von Neumann stability analysis

If the difference equation solution diverges then we *know* that we have a bad approximation to the original PDE.

Von Neumann stability analysis starts from the assumption that *eigenmodes* of the difference equation can be written as

$$
T_{m,j} = \xi(k)^j e^{ikm\Delta x}, \quad t=j\Delta t,\ x=m\Delta x
$$

with the unknown wave vectors $k=2\pi/\lambda$ and unknown complex functions – the *amplification factors* – $\xi(k)$.

Solutions of the difference equation can be written as linear superpositions of these basis functions. But they are only stable if the eigenmodes are stable, i.e., will not grow in time (with $j$). This is the case when

$$
|\xi(k)| < 1
$$

for all $k$.

Insert the eigenmodes into the finite difference equation

$$
T_{m, j+1} = (1 - 2\eta) T_{m,j} + \eta(T_{m+1,j} + T_{m-1, j})
$$

to obtain

\begin{align}
\xi(k)^{j+1} e^{ikm\Delta x} &= (1 - 2\eta) \xi(k)^{j} e^{ikm\Delta x}
    + \eta(\xi(k)^{j} e^{ik(m+1)\Delta x} + \xi(k)^{j} e^{ik(m-1)\Delta x})\\
\xi(k) &= (1 - 2\eta) + \eta(e^{ik\Delta x} + e^{-ik\Delta x})\\
\xi(k) &= 1 - 2\eta + 2\eta \cos k\Delta x\\
\xi(k) &= 1 + 2\eta\big(\cos k\Delta x - 1\big)
\end{align}

For $|\xi(k)| < 1$ (and all possible $k$):

\begin{align}
|\xi(k)| < 1 \quad &\Leftrightarrow \quad \xi^2(k) < 1\\
(1 + 2y)^2 = 1 + 4y + 4y^2 &< 1 \quad \text{with}\ \  y = \eta(\cos k\Delta x - 1)\\
y(1 + y) &< 0 \quad \Leftrightarrow \quad -1 < y < 0\\
\eta(\cos k\Delta x - 1) &\leq 0 \quad \forall k \quad (\eta > 0, -1 \leq \cos x \leq 1)\\
\eta(\cos k\Delta x - 1) &> -1\\
\eta &< \frac{1}{1 - \cos k\Delta x}\\
\eta = \frac{K \Delta t}{C \rho \Delta x^2} &< \frac{1}{2} \le \frac{1}{1 - \cos k\Delta x}
\end{align}

Thus, solutions are only stable for $\eta < 1/2$. In particular, decreasing $\Delta t$ will always improve stability, But decreasing $\Delta x$ requires an quadratic *increase* in $\Delta t$!

Note
* Perform von Neumann stability analysis when possible (depends on PDE and the specific discretization).
* Test different combinations of $\Delta t$ and $\Delta x$.
* Not guarantee that decreasing both will lead to more stable solutions!

Check my inputs:

This was stable and it conforms to the stability criterion:

```python
Dt = 2
Dx = 0.02
eta = Kappa * Dt /(CHeat * rho * Dx*Dx)
print(eta)
```

... and this was unstable, despite a seemingly small change:

```python
Dt = 2
Dx = 0.01
eta = Kappa * Dt /(CHeat * rho * Dx*Dx)
print(eta)
```


# Introduction

We now have defined all of the necessary components to derive the equations of motion of the system using Kane's Method:

- Linear velocity of each mass center.
- Angular velocity of each rigid body.
- All contributing (external) forces.

We will use the `KanesMethod` class which provides an automated computation of the first order ordinary differential equations given the above quantities. (Any method can be used, such as Newton-Euler or Hamilton's method, but there are only automated methods for Kane's equations and Lagrangian dynamics at the moment).

# Setup

Import the solutions from the previous notebooks:

```python
from __future__ import print_function, division
from solution.kinetics import *
```

We will need the `KanesMethod` class and `trigsimp` to give a reasonably compact result.

```python
from sympy import trigsimp
from sympy.physics.mechanics import KanesMethod
```

Enable nice math printing:

```python
from sympy.physics.vector import init_vprinting
init_vprinting(use_latex='mathjax', pretty_print=False)
```

```python
KanesMethod?
```

# Equations of Motion

At the bare minimum for unconstrained systems, the `KanesMethod` class needs to know the generalized coordinates, the generalized speeds, the kinematical differential equations, the loads, the bodies, and a Newtonian reference frame. First, make a list of the generalized coordinates, i.e. the three joint angles:

```python
coordinates = [theta1, theta2, theta3]
coordinates
```

Now list the generalized speeds, i.e. the joint rates:

```python
speeds = [omega1, omega2, omega3]
speeds
```

The `KanesMethod` object can now be created with the inertial reference frame, coordinates, speeds, and kinematical differential equations:

```python
kinematical_differential_equations
```

```python
kane = KanesMethod(inertial_frame, coordinates, speeds, kinematical_differential_equations)
```

Now list the six loads we defined and the three bodies:

```python
loads = [lower_leg_grav_force,
         upper_leg_grav_force,
         torso_grav_force,
         lower_leg_torque,
         upper_leg_torque,
         torso_torque]
loads
```

```python
bodies = [lower_leg, upper_leg, torso]
bodies
```

The equations of motion can now be computed using the `kanes_equations` method, which takes the list of loads and bodies. It returns the equations of motion (i.e. first order ordinary differential equations) in Kane's form:

$$ F_r + F_r^* = 0$$

which is essentially equivalent to the classic Newton_Euler form:

$$ \mathbf{F} = \mathbf{m}\mathbf{a} $$

$$ \mathbf{T} = \mathbf{I} \mathbf{\alpha} $$

```python
fr, frstar = kane.kanes_equations(bodies, loads)
```

```python
trigsimp(fr + frstar)
```

Keep in mind that out utlimate goal is to have the equations of motion in first order form:

$$ \dot{\mathbf{x}} = \mathbf{g}(\mathbf{x}, t) $$

The equations of motion are linear in terms of the derivatives of the generalized speeds and the `KanesMethod` class automatically puts the equations in a more useful form for the next step of numerical simulation:

$$ \mathbf{M}(\mathbf{x}, t)\dot{\mathbf{x}} = \mathbf{f}(\mathbf{x}, t) $$

Note that

$$ \mathbf{g} = \mathbf{M}^{-1}(\mathbf{x}, t) \mathbf{f}(\mathbf{x}, t) $$

and that $\mathbf{g}$ can be computed analytically but for non-toy problems, it is best to do this numerically. So we will simply generate the $\mathbf{M}$ and $\mathbf{f}$ matrices for later use.

The mass matrix, $\mathbf{M}$, can be accessed with the `mass_matrix` method (use `mass_matrix_full` to include the kinematical differential equations too. We can use `trigsimp` again to make this relatively compact:

```python
mass_matrix = trigsimp(kane.mass_matrix_full)
mass_matrix
```

The right hand side, $\mathbf{f}$, is a vector function of all the non-inertial forces (gyroscopic, external, coriolis, etc):

```python
forcing_vector = trigsimp(kane.forcing_full)
forcing_vector
```


## Constrained Optimization Approach to Predictability Analysis in
 Bio-Mathematical Modeling

Ivan Borisov,

Developer, InSysBio LLC

### Agenda

- Bio-Mathematical Modeling
- Validation of a Systems Biology model
- LikelihoodProfiler
- Taxol treatment model case study: assessing identifability and predictability

## A Systems Biology Model

A general kinetic systems biology (SB) model:
\begin{align}
\frac{dx(t)}{dt} & = f(x(t),\theta) \\
x(0) & = x_0 \\
y(t,\theta) & = g(x(t),\theta)+\epsilon(t)
\end{align}

- $x(t)$ - model's state variables
- $y(t, \theta)$ - the observables (output, experimentally observed quantities)
- $\epsilon(t)$ - the measurement error
- {$\theta, x_0$} - set of both known and unknown model's parameters

### Assessing model's credibility:

- Which parameters are structurally identifiable?
- Is experimental data sufficient to practically identify all the unknown parameters?
- What are the ranges for parameters variations?
- How reliable are the model's predictions?

## Identifiability Analysis with LikelihoodProfiler

LikelihoodProfiler (LP) https://github.com/insysbio/LikelihoodProfiler.jl
- Is a free software package for **Identifiability analysis** based on **Profile Likelihood** method and written in **Julia** language
- Transforms confidence intervals estimation into **“single-pass” constrained optimization problem**
- Is **fast** and **efficient** even for non-differentiable likelihood functions
- Implements **Confidence Intervals by Constrained Optimization** algorithm proposed in [Borisov I, Metelkin E (2020) Confidence intervals by constrained optimization—An algorithm and software package for practical identifiability analysis in systems biology](https://doi.org/10.1371/journal.pcbi.1008495)

## Cancer Taxol Treatment Model
Identifiability of this model was analyzed in [Marisa C.Eisenberg, Harsh V.Jain. A confidence building exercise in data and identifiability](https://www.sciencedirect.com/science/article/pii/S0022519317303454)

```julia
using DiffEqBase, OrdinaryDiffEq, Plots, NLopt, DataFrames, CSV, Distributions, LikelihoodProfiler
import Statistics: mean

const K   = 10.515*100
const V0  = 1.3907*K
const lam = 9.5722
const theta = 10.
const aRP  = 20.
const arstexp = 3.
const adthexp = 4.;
```

```julia
# https://github.com/marisae/cancer-chemo-identifiability/blob/master/Profile%20Likelihood/testa0_de.m
function ode_func(du, u, θ, t, drug)
    (a0, ka, r0, d0, kd) = (θ[1],θ[2],θ[3],θ[4],θ[5])

    Ncel = u[1] + u[2] + u[3]
    Lfac = ((K-Ncel)^theta)/((V0^theta) + ((K-Ncel)^theta))
    arst = a0*(drug^arstexp)/(ka^arstexp + (drug^arstexp))
    adth = d0*(drug^adthexp)/(kd^adthexp + (drug^adthexp))
    arcv = r0

    # The differential equations
    du[1] = -lam*u[1] + aRP*u[2]*Lfac - arst*u[1] + arcv*u[3]
    du[2] = 2*lam*u[1] - aRP*u[2]*Lfac
    du[3] = arst*u[1] - adth*u[3] - arcv*u[3]
end;
```

Let's load experimental data

```julia
df = CSV.read("taxol_data.csv", DataFrame)
dose = [5., 10., 40., 100.]

times = df.times
C005 = mean(df.Cell005)
C010 = mean(df.Cell010)
C040 = mean(df.Cell040)
C100 = mean(df.Cell100)

data = [df.Cell005/C005, df.Cell010/C010, df.Cell040/C040, df.Cell100/C100]
datamean = [C005, C010, C040, C100];
```

... and define the loss function
$$ \Lambda(\theta) = \sum_{i=1}^n \left( y(t_i,θ) - data_i \right)^2 $$

```julia
function lsq_func(θ)
    loss = 0.
    for (i,d) in enumerate(dose)
       prob = ODEProblem((du,u,θ,t)->ode_func(du,u,θ,t,d), u0, tspan, θ)
       sol = solve(prob, AutoTsit5(Rosenbrock23()), saveat=times)

       sim = (sol[1,:] + sol[2,:] + sol[3,:])/datamean[i]
       loss += sum((sim-data[i]).^2)
    end
    return loss
end;
```

Let's assume we have fitted the model to experimental data...

```julia
# best fit values
u0 = [7.2700, 2.5490, 0.]
θ0 = [8.3170, 8.0959, 0.0582, 1.3307, 119.1363]
tspan = (0.,15.);
```

```julia
prob = ODEProblem((du,u,θ,t)->ode_func(du,u,θ,t,5.0), u0, tspan, θ0)
sol = solve(prob, AutoTsit5(Rosenbrock23()), saveat=times)

plot(sol.t, (sol[1,:]+sol[2,:]+sol[3,:]), xlabel = "t", ylabel = "Ncell", label="Ncell(t) for dose=5", line=3, fmt = :png);
scatter!(times, df.Cell005, label = "Experimental data")
```

Now it's time to check parameters **practical identifiability**.
- Parameter $\theta_i$ is said to be practically identifiable if it's confidence interval (CI) is finite
- Parameters' confidence interval can be obtained with **profile likelihood** which  implies constructing likelihood-based CIs by exploring $\Lambda(\theta_i)$ as a function of a single parameter $\theta_i$ : $PL(\theta_i) = \min_{j\neq i}(\Lambda(\theta))$
- The confidence interval to a given confidence level $\alpha$ are $\theta_i$ values with $PL(\theta_i)$ below the threshold ($\alpha$-quantile of a $\chi^2$-distribution)

#### LikelihoodProfiler (LP)
- Allows the users to identify the **endpoints of confidence intervals** without recovering the whole profile
- Obtaines the **endpoints of confidence intervals** as solutions to the constrained optimization problem

```julia
# likelihood threashold
sigmasq = (mean([(df.Cerr005/C005); (df.Cerr010/C010); (df.Cerr040/C040); (df.Cerr100/C100)]))^2
threshold = sigmasq*cquantile(Chisq(5), 0.05)

α = lsq_func(θ0) + threshold;
```

Let's estimate the confidence intervall for *a0* parameter

```julia
using LikelihoodProfiler

a0_ci = get_interval(θ0, 1, lsq_func,:CICO_ONE_PASS,
        loss_crit = α, scan_bounds = (1e-3,1e3), scale = fill(:log,length(θ0)))
plot(a0_ci, fmt = :png, legend=:bottomright)
```

We can enrich the profile with intermediate points to make smooth plot

```julia
update_profile_points!(a0_ci)
plot(a0_ci, fmt = :png, legend=:bottomright)
```

**LikelihoodProfiler** usage can be extended to **confidence bands** estimation
- Confidence bands are **generalizations of confidence intervals** for arbitrary observables
- Construction of a confidence band implies estimating confidence intervals of an observable as function of all parameters $A(\theta)$ at given timepoints
- The confidence band to a given confidence level $\alpha$ are $A(\theta)$ values with $PL(\theta) = \Lambda(\theta)$ below the threshold ($\alpha$-quantile of a $\chi^2$-distribution)

Let's estimate confidence bands for *Ncell* observable and *dose=5*

```julia
function bands_func(parampampam, i, dose=5.0)
    prob = ODEProblem((du,u,θ,t)->ode_func(du,u,θ,t,dose), u0, tspan, parampampam)
    sol = solve(
        prob,
        AutoTsit5(Rosenbrock23()),
        saveat=times
    )
    return sol[1,i]+sol[2,i]+sol[3,i]
end;
```

```julia
intervals = Vector{ParamInterval}(undef,length(times))
for i in eachindex(times)
    @time intervals[i] = get_interval(θ0, θ->bands_func(θ,i), lsq_func, :CICO_ONE_PASS,
        loss_crit = α, scan_bounds = (1e-3,1e3), theta_bounds = fill((1e-3,1e3),length(θ0)),scale = fill(:log,length(θ0)))
end
```

      8.951276 seconds (9.89 M allocations: 654.626 MiB, 1.76% gc time, 96.90% compilation time)
     10.238887 seconds (18.00 M allocations: 1.366 GiB, 4.03% gc time)
     12.926473 seconds (22.93 M allocations: 1.738 GiB, 3.77% gc time)
     12.068383 seconds (21.98 M allocations: 1.663 GiB, 3.99% gc time)
     10.283547 seconds (18.50 M allocations: 1.404 GiB, 3.74% gc time)
     12.025768 seconds (21.85 M allocations: 1.650 GiB, 4.01% gc time)

Let' plot **confidence bands** for *Ncell* observable

```julia
lbands = [iv.result[1].value for iv in intervals]
ubands = [iv.result[2].value for iv in intervals]
u = (sol[1,:]+sol[2,:]+sol[3,:])
plot(sol.t, u, xlabel = "t", ylabel = "Ncell", label="Ncell(t) for dose=5", line=3,
    ribbon = (u-lbands, ubands-u), fc=:orange, fa=0.3, fmt = :png, legend=:bottomright);
scatter!(times, df.Cell005, label = "Experimental data")
```

### Results

- **LikelihoodProfiler** application can be extended to predictability analysis and confidence bands estimation
- The efficacy of endpoint estimation without profile recovery becames espessially vivid in the case of **confidence bands**

### Links

- LikelihoodProfiler package https://github.com/insysbio/LikelihoodProfiler.jl
- Use-case and benchmark models https://github.com/insysbio/likelihoodprofiler-cases

### Roadmap

- Adapt the algorithm to other credability measurements, including predictive and validation bands
- Improve underlying optimization routines
- Contributions are welcome :)

##   Thanks for your attention!


# Design of Digital Filters

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Design of Non-Recursive Filters using the Frequency Sampling Method

For some applications, the desired frequency response is not given at all frequencies but rather at a number of discrete frequencies. For this case, the frequency sampling method provides a solution for the design of non-recursive filters.

### The Frequency Sampling Method

Let's assume that the desired transfer function $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is specified at a set of $N$ equally spaced frequencies $\Omega_\mu = \frac{2 \pi}{N} \mu$

\begin{equation}
H_\text{d}[\mu] = H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\frac{2 \pi}{N} \mu})
\end{equation}

for $\mu = 0, 1, \dots, N-1$. The coefficients of a non-recursive filter with finite impulse response (FIR) can then be computed by inverse discrete Fourier transformation (DFT) of $H_\text{d}[\mu]$

\begin{equation}
h[k] = \text{DFT}_N^{-1} \{ H_\text{d}[\mu] \} = \frac{1}{N} \sum_{\mu = 0}^{N-1} H_\text{d}[\mu] \; \mathrm{e}^{\,\mathrm{j}\,\frac{2 \pi}{N} \mu\,k}
\end{equation}

for $k = 0,1, \dots, N-1$.

In order to investigate the properties of the designed filter, its transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is computed. It is given by discrete-time Fourier transformation (DTFT) of its impulse response $h[k]$

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{k = 0}^{N-1} h[k] \; \mathrm{e}^{\,-\mathrm{j}\,\Omega\,k} =
\sum_{\mu = 0}^{N-1} H_\text{d}[\mu] \cdot \frac{1}{N} \sum_{k = 0}^{N-1} \mathrm{e}^{\,-\mathrm{j}\,k\,(\Omega - \frac{2 \pi}{N}\,\mu)}
\end{equation}

When comparing this result with the [interpolation of a DFT](../spectral_analysis_deterministic_signals/zero_padding.ipynb#Interpolation-of-the-Discrete-Fourier-Transformation), it can be concluded that $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is yielded by interpolation of the desired transfer function $H_\text{d}[\mu]$

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \sum_{\mu=0}^{N-1} H_\text{d}[\mu] \cdot \mathrm{e}^{-\,\mathrm{j}\, \frac{( \Omega - \frac{2 \pi}{N} \mu ) (N-1)}{2}} \cdot \text{psinc}_N ( \Omega - \frac{2 \pi}{N} \mu)
\end{equation}

where $\text{psinc}_N(\cdot)$ denotes the $N$-th order periodic sinc function.

Both the transfer function of the filter $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the desired transfer function $H_\text{d}[\mu]$ are equal at the specified frequencies $\Omega_\mu = \frac{2 \pi}{N} \mu$. Values in between adjacent $\Omega_\mu$ are interpolated by the periodic sinc function. This is illustrated in the following.

#### Example: Approximation of an ideal low-pass

The design of an ideal low-pass filter using the frequency sampling method is considered. For $|\Omega| < \pi$ the transfer function of the ideal low-pass is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

where $\Omega_\text{c}$ denotes its corner frequency. The desired transfer function $H_\text{d}[\mu]$ for the frequency sampling method is derived by sampling $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. Note that for sampling on the unit circle with $0 \leq \Omega < 2 \pi$, the periodicity $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = H_\text{d}(\mathrm{e}^{\,\mathrm{j}(\Omega + n 2 \pi)})$ for $n \in \mathbb{Z}$ has to be considered.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig

N = 32  # length of filter
Omc = np.pi/3  # corner frequency of low-pass

# specify desired frequency response
Ommu = 2*np.pi/N*np.arange(N)
Hd = np.zeros(N)
Hd[Ommu <= Omc] = 1
Hd[Ommu >= (2*np.pi-Omc)] = 1

# compute impulse response of filter
h = np.fft.ifft(Hd)
h = np.real(h)  # due to round-off errors
# compute frequency response of filter
Om, H = sig.freqz(h, worN=8192)

# plot impulse response
plt.figure(figsize = (10,3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot transfer functions
plt.figure(figsize = (10,3))
plt.plot(Om, np.abs(H), 'b-', label=r'designed $|H(e^{j \Omega})|$')
plt.stem(Ommu, np.abs(Hd), 'g', label=r'desired $|H_d[\mu]|$')
plt.plot([0, Omc, Omc], [1, 1, 0], 'r--')
plt.title('Magnitude response of desired/designed filter')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -0.05, 1.5])
# plot phase
plt.figure(figsize = (10,3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi(\Omega)$')
plt.grid()
```

**Exercises**

* What phase behavior does the designed filter have?
* Increase the length `N` of the filter. Does the attenuation in the stop-band improve?

The reason for the poor perfomance of the designed filter is the zero-phase of the desired transfer function which cannot be realized by a causal non-recursive system. This was [already discussed for the window method](../filter_design/window_method.ipynb#Zero-Phase-Filters). In comparison to the window method, the freqeuncy sampling method suffers from addtional time-domain aliasing due to the periodicy of the DFT. Again a linear-phase design is better in such situations.

### Design of Linear-Phase Filters

The design of non-recursive FIR filters with a linear phase is often desired due to their constant group delay. As for the [window method](../filter_design/window_method.ipynb), the design of a digital filter with a generalized linear phase is considered in the following. For $|\Omega| < \pi$ its transfer function is given as

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot \mathrm{e}^{\,-\mathrm{j} \alpha \Omega + \mathrm{j} \beta}
\end{equation}

where $A(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \in \mathbb{R}$ denotes the amplitude of the filter, $-\alpha\,\Omega$ its linear phase and $\beta$ a constant phase offset. The impulse response $h[k]$ of a linear-phase filter shows specific symmetries which have already been discussed for the [design of linear-phase filters using the window method](../filter_design/window_method.ipynb#Causal-Linear-Phase-Filters). For the resulting four types of linear-phase FIR filters, the properties of $A(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the values of $\alpha$ and $\beta$ have to be chosen accordingly for the formulation of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $H_d[\mu]$, respectively. This is illustrated in the following for the design of a low-pass filter.

#### Example: Linear-phase approximation of an ideal low-pass

We aim at the approximation of an ideal low-pass as a linear-phase non-recursive FIR filter. For the sake of comparison with a similar [example for the window method](../filter_design/window_method.ipynb#Example:-Causal-linear-phase-approximation-of-ideal-low-pass), we choose a type 1 filter with odd filter length $N$, $\alpha = \frac{N-1}{2}$ and $\beta = 0$. The desired frequency response $H_\text{d}[\mu]$ is given by sampling

\begin{equation}
H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \mathrm{e}^{\,-\mathrm{j} \frac{N-1}{2} \Omega} \cdot \begin{cases}
1 & \text{for } |\Omega| \leq \Omega_\text{c} \\
0 & \text{otherwise}
\end{cases}
\end{equation}

which is defined for $|\Omega| < \pi$. Note that for sampling on the unit circle with $0 \leq \Omega < 2 \pi$, the periodicity $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = H_\text{d}(\mathrm{e}^{\,\mathrm{j}(\Omega+ n 2 \pi)})$ for $n \in \mathbb{Z}$ has to be considered.

```python
N = 33  # length of filter
Omc = np.pi/3  # corner frequency of low-pass

# specify desired frequency response
Ommu = 2*np.pi/N*np.arange(N)
Hd = np.zeros(N)
Hd[Ommu <= Omc] = 1
Hd[Ommu >= (2*np.pi-Omc)] = 1
Hd = Hd * np.exp(-1j*Ommu*(N-1)/2)

# compute impulse response of filter
h = np.fft.ifft(Hd)
h = np.real(h)  # due to round-off errors
# compute frequency response of filter
Om, H = sig.freqz(h, worN=8192)

# plot impulse response
plt.figure(figsize = (10,3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot frequency response
plt.figure(figsize = (10,3))
plt.plot(Om, np.abs(H), 'b-', label=r'designed $|H(e^{j \Omega})|$')
plt.stem(Ommu, np.abs(Hd), 'g', label=r'desired $|H_d[\mu]|$')
plt.plot([0, Omc, Omc], [1, 1, 0], 'r--')
plt.title('Magnitude response of desired/designed filter')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -0.05, 1.5])
# plot phase
plt.figure(figsize = (10,3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi(\Omega)$')
plt.grid()
```

**Exercises**

* Does the designed filter have the desired linear phase?
* Increase the length `N` of the filter. What is different to the previous example?
* How could the method be modified to change the properties of the frequency response?

### Comparison to Window Method

For a comparison of the frequency sampling to the [window method](../filter_design/window_method.ipynb) it is assumed that the desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is given. For both methods, the coefficients $h[k]$ of an FIR approximation are computed as follows

1. Window Method
\begin{align}
h_\text{d}[k] &= \mathcal{F}_{*}^{-1} \{ H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \} \\
h[k] &= h_\text{d}[k] \cdot w[k]
\end{align}

2. Frequency Sampling Method
\begin{align}
H_\text{d}[\mu] &= H_\text{d}(\mathrm{e}^{\,\mathrm{j} \frac{2 \pi}{N} \mu}) \\
h[k] &= \text{DFT}_N^{-1} \{ H_\text{d}[\mu] \}
\end{align}

For finite lengths $N$, the difference between both methods is related to the periodicity of the DFT. For a desired frequency response $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ which does not result in a FIR $h_\text{d}[k]$ of length $N$, the inverse DFT in the frequency sampling method will suffer from time-domain aliasing. In the general case, filter coefficients computed by the window and frequency sampling method will hence differ.

However, for a rectangular window $w[k]$ and $N \to \infty$ both methods will become equivalent. This reasoning motivates an oversampled frequency sampling method, where $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is sampled at $M \gg N$ points in order to derive an approximation of $h_\text{d}[k]$ which is then windowed to the target length $N$. The method is beneficial in cases where a closed-form inverse DTFT of $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, as required for the window method, cannot be found.

#### Example: Oversampled frequency sampling method

We consider the design of a linear-phase approximation of an ideal low-pass filter using the oversampled frequency sampling method. For the sake of comparison, the parameters have been chosen in accordance to a [similar example using the window method](../filter_design/window_method.ipynb#Example:-Causal-linear-phase-approximation-of-ideal-low-pass). Using $H_\text{d}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ from the previous example in this section, the filter is computed by

1. (Over)-Sampling the desired response at $M$ frequencies
\begin{equation}
H_\text{d}[\mu] = H_\text{d}(\mathrm{e}^{\,\mathrm{j} \frac{2 \pi}{M} \mu})
\end{equation}

2. Inverse DFT of length $M$
\begin{equation}
h[k] = \text{DFT}_M^{-1} \{ H_\text{d}[\mu] \}
\end{equation}

3. Windowing to desired length $N$
\begin{equation}
h[k] = h_\text{d}[k] \cdot w[k]
\end{equation}

```python
N = 33  # length of filter
M = 8192  #
Omc = np.pi/2  # corner frequency of low-pass

# specify desired frequency response
Ommu = 2*np.pi/M*np.arange(M)
Hd = np.zeros(M)
Hd[Ommu <= Omc] = 1
Hd[Ommu >= (2*np.pi-Omc)] = 1
Hd = Hd * np.exp(-1j*Ommu*(N-1)/2)

# compute impulse response of filter
h = np.fft.ifft(Hd)
h = np.real(h)  # due to round-off errors
h = h[0:N]  # rectangular window
# compute frequency response of filter
Om, H = sig.freqz(h, worN=8192)

# plot impulse response
plt.figure(figsize = (10,3))
plt.stem(h)
plt.title('Impulse response')
plt.xlabel(r'$k$')
plt.ylabel(r'$h[k]$')
# plot frequency response
plt.figure(figsize = (10,3))
plt.plot(Om, 20 * np.log10(abs(H)), label='rectangular window')
plt.plot([0, Omc, Omc], [0, 0, -100], 'r--')
plt.title('Magnitude response of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$|H(e^{j \Omega})|$ in dB')
plt.axis([0, np.pi, -100, 3])
plt.legend()
plt.grid()
# plot phase
plt.figure(figsize = (10,3))
plt.plot(Om, np.unwrap(np.angle(H)))
plt.title('Phase of designed filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$\varphi(\Omega)$')
plt.grid()
```

**Exercises**

* Compare the designed filter and its properties to the [same design using the window method](../filter_design/window_method.ipynb#Example:-Causal-linear-phase-approximation-of-ideal-low-pass)
* Change the number of samples `M` used for sampling the desired response. What changes if you increase/decrease `M`?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples, 2016-2017*.


## Reducing Order for Accelerated Analysis:  *One Guy’s Take on SVD, POD, DMD, and Their Use for Nuclear Engineering*

Jeremy Roberts, Associate Professor \
Alan Levin Department of Mechanical and Nuclear Engineering \
Kansas State University

Wednesday, January 27th, 2021

Rothrock Lecture Series \
Department of Nuclear Engineering \
Texas A&M University

## I Spotted a Bear that Changed My World

Karhunen and Loève said a stochastic process $X_t$  with covariance $K_X(t, t')$ can be represented exactly (or approximately) as an infinite (or a finite), weighted sum of the (time-dependent) eigenvectors of a certain functional of $K_{X}$.

Let $\mathbf{x}_j$ be a column of an image (i.e., matrix) $\mathbf{A}$.

Then

$$
\mathbf{C}_{ij} = E[(\mathbf{x}_i-\mathbf{m})^T (\mathbf{x}_j-\mathbf{m})] \, ,
$$

for $\mathbf{m} = E[\mathbf{x}]$, and eigendecomposition leads to

$$
\mathbf{C} = \mathbf{W}\boldsymbol{\Lambda}\mathbf{W}^T \, .
$$

Define $\mathbf{y} = \mathbf{W}^T\mathbf{x}$ (or $\mathbf{B} = \mathbf{W}^T\mathbf{A}$).  By construction, $\mathbf{W}^T$ rotates the columns $\mathbf{x}$ so that the result is decorrelated, i.e., diagonalized.  Moreover, the columns of $\mathbf{W}$ with the largest eigenvalues $\lambda$ preserve the most “energy” of the initial system upon the inverse (or a "best" picture).

This is (often, but maybe not correctly) called the Karhunen-Loève Transform.

 (See R.D. Dony, "Karhunen-Loève Transform". *The Transform and Data Compression Handbook*
Ed. K. R. Rao and P.C. Yip. Boca Raton, CRC Press LLC, 2001.)

## Whoa, slow down.  Show me the SVD.

First, what does "best" mean?  Least squares?  Minimax?

Given $\mathbf{A} \in R^{m\times n}$ of rank $\min(m, n)$, can we find $\tilde{\mathbf{A}}  \in R^{m\times n}$ of rank $r < \min(m, n)$ that satisfies

$$
 \min_{\tilde{A}} \sqrt{\sum_j \sum_i (A_{ij} - \tilde{A}_{ij})^2} \qquad \text{least-square pixel error}
$$

Equivalently, where $\mathbf{x}$ is a column of $\mathbf{A}$, find $\tilde{\mathbf{A}}$ that satisfies

$$
 \min_{\tilde{A}} \sqrt{\sum_j ||\mathbf{x}_i - \tilde{\mathbf{x}}_i||_2^2}
 \qquad \text{root-mean square column error}
$$

Both are useful ways to think of the problem since applications are often about "all the pixels" or "all the columns."

The solution, of course, is the singular value decomposition (SVD), or

$$
  \mathbf{A} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T \, ,
$$

where $\mathbf{U} \in R^{m, n}$, $\boldsymbol{\Sigma} \in R^{n, n}$ is a diagonal matrix of strictly nonnegative *singular* values $\sigma_i,\, i = 1\ldots n$ such that $\sigma_i \geq \sigma_{i+1}$, and $\mathbf{V} \in R^{n, n}$. Both $\mathbf{U}$ and $\mathbf{V}$ are orthogonal matrices (i.e., $\mathbf{U}^T \mathbf{U} = \mathbf{I}$).

Proof is left to the viewer... but the *truncated* SVD  $\mathbf{U}$ yields the approximation $\tilde{\mathbf{A}} = \mathbf{U}_r \boldsymbol{\Sigma}_r \mathbf{V}_r^T$ that uses the first $r$ columns $\mathbf{U}$ and minimizes

$$
  \sqrt{\sum_j \sum_i (A_{ij} - \tilde{A}_{ij})^2} = || \tilde{\mathbf{A}} - \mathbf{A} ||_F  \, .
$$

among all possible rank-$r$ approximations $\mathbf{A}$.

```python
import urllib
url = 'https://unsplash.com/photos/f1q4NlVRYSc/download?force=true&w=2400'
urllib.request.urlretrieve(url , 'snake.jpg')
import matplotlib.pyplot as plt
A_rgb = plt.imread('snake.jpg')   # 1737x2400 matrix of RGB tuples
plt.imshow(A_rgb)
```

    <matplotlib.image.AxesImage at 0x7fb62ea3b278>

```python
import numpy as np
A = np.array(A_rgb.reshape((A_rgb.shape[0], A_rgb.shape[1]*A_rgb.shape[2])), dtype='float')
U, sigma, V = np.linalg.svd(A, compute_uv=True) # sigma is a 1-d array
```

```python
import matplotlib.pyplot as plt
plt.semilogy(sigma, 'go', mfc='w')
plt.semilogy(10, sigma[10], 'rs', 50, sigma[50], 'b^', 100, sigma[100], 'kh', mfc='None', ms=15)
plt.xlabel('$i$'); plt.ylabel('$\sigma_i$'); plt.title("Most information in first ~100 singular values.")
```

```python
A_r = []
for r in [10, 50, 100]:
    Sigma = np.diag(sigma) # make Sigma a diagonal matrix
    A_r.append(U[:, :r]@(Sigma[:r, :r]@V[:r, :])) # one can also try sklearn.decomposition.TruncatedSVD
    A_r[-1] = np.array(A_r[-1].reshape((1737, 2400, 3)), dtype='i')
    A_r[-1][A_r[-1]<0]=0; A_r[-1][A_r[-1]>255]=255;
fig, axes = plt.subplots(nrows=2, ncols=2, figsize=(10,8))
axes[0, 0].imshow(A_rgb); axes[0,0].set_title('original')
tmpl = 'n={},  {:.1f}% compression'
comp = lambda n: (1-(U.shape[0]+V.shape[0])*n/A.shape[0]/A.shape[1])*100
axes[1, 0].imshow(A_r[0]); axes[1,0].set_title(tmpl.format(10, comp(10)))
axes[0, 1].imshow(A_r[1]); axes[0,1].set_title(tmpl.format(50, comp(50)))
axes[1, 1].imshow(A_r[2]); axes[1,1].set_title(tmpl.format(100, comp(100)))
```

# Um, the Bear?

Forget about the mean $\mathbf{m} = E[\mathbf{x}]$ (or assume zero, etc.).  Then

$$
  \mathbf{W}\boldsymbol{\Lambda}\mathbf{W}^T =  \mathbf{C}
     = \mathbf{A}^T \mathbf{A} = (\mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T)^T \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T = \mathbf{V}\boldsymbol{\Sigma}^T \mathbf{U}^T \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T
     = \mathbf{V}\boldsymbol{\Sigma}^2 \mathbf{V}^T \, .
$$

Similarly,

$$
 (\mathbf{C}^T \neq ) \quad \mathbf{A} \mathbf{A}^T  = \mathbf{U}\boldsymbol{\Sigma}^2 \mathbf{U}^T \, .
$$

So, our original projection $\mathbf{y} = \mathbf{W}^T \mathbf{x}$ can be viewed as a projection of $\mathbf{x}$ onto the right singular vectors $\mathbf{v}$ of $\mathbf{A}$ or, equivalently, the row space of $\mathbf{A}$.

With SVD, it's easy to see more clearly what we have:

$$
\mathbf{A} \approx  \sigma_1 \mathbf{u}_1 \mathbf{v}^T_1 + \sigma_2 \mathbf{u}_2 \mathbf{v}^T_2 + \ldots \,
$$

In other words, we construct $\mathbf{A}$ by adding rank-1 matrices (outer products) weighted by the singular values.

# SVD to POD

For simplicity, consider an "expensive" model that has already been discretized in space, e.g.,

$$
  \frac{d\mathbf{x}}{dt} = \mathbf{L} \mathbf{x} + \mathbf{q} \, ,
$$

where $x_i(t) \equiv x(r_i, t)$,  $\mathbf{x},\mathbf{q} \in R^{m\times 1}$, and $\mathbf{L} \in R^{m\times m}$.  Then

  1. Chop up time and compute the "snapshots" $\mathbf{x}_0, \mathbf{x}_1, \ldots$ for $t = t_0, t_1, \ldots$.

 2. Place the snapshot into the matrix $\mathbf{X}$.

  3. Compute the SVD of $\mathbf{X}$ to determine the principle "directions" (or "components") of the observed field, i.e., the first $r$ left singular vectors $\mathbf{U} \in R^{m\times r}$.

4. At this point, the data (whether from our expensive model or, perhaps, from experiment) can be projected onto the reduced space $\mathbf{U}^{T}\mathbf{x} = \mathbf{y}$ and recovered (approximately) through  $\mathbf{x} = \mathbf{Uy}$.  This application, strongly rooted in statistics, was first(?) reported in 1967 by Lumley for analysis of *experimental* data.

5. Substitute $\mathbf{x} = \mathbf{Uy}$  into our model to find

$$
\frac{d}{dt}\mathbf{Uy} = \mathbf{LUy} + \mathbf{q} \, .
$$

Think carefully about the emphasis here on $\mathbf{U}$!

6.  Multiply this result equation on the left by $\mathbf{U}^T$ to find

$$
  \mathbf{U}^T \frac{d}{dt} \mathbf{Uy} = \mathbf{U}^{T} \mathbf{LUy} + \mathbf{U}^T \mathbf{q} \, .
$$

or, because $\mathbf{U}^T = \mathbf{U}^{-1}$,

$$
   \frac{d\mathbf{y}}{dt} = \tilde{\mathbf{L}}\mathbf{y} + \tilde{\mathbf{q}} \, .
$$

This is, generically, the Galerkin-POD method.  Here, Galerkin indicates $\mathbf{U}$ is used for both representing $\mathbf{x}$ (test) and diagonalizing the system via projection (trial).

If $r \ll m$, this system is *way easier to solve*.  The "coefficients" $\mathbf{y}$ can be found, and the original field reconstructed via $\mathbf{x} = \mathbf{Uy}$.

6. I'm totally skipping boundary conditions, nonlinearities, blah blah.  The point is that POD is use of SVD to shrink the system.  The orthogonality leaves a single derivative on the left, i.e., a system that looks mightily unchanged.

# POD for Multigroup Models

Some of my work has dealt with the $k$-eigenvalue problem:

$$
  \boldsymbol{\hat{\Omega}}  \cdot \nabla \psi_{c,g}(\boldsymbol{\hat{\Omega}}) +
    \Sigma^{t}_{c,g} \psi_{c,g}(\boldsymbol{\hat{\Omega}}) =
    \frac{1}{4\pi} \sum\limits^{N_g}_{g'=1} \left(\Sigma^{s}_{c,g\leftarrow g'}  + \frac{\chi_{c,g}}{k} \nu\Sigma^{f}_{c,g'}\right) \phi_{c,g'}\, .
$$

The notation is largely standard, but the $c$ indicates a spatial unknown and the $g$ indicates group.

Two questions with evolving answers that have driven my interest in POD-like things:

  - What's the best way to represent boundary conditions in so-called response-matrix methods?
  - **Can the generalized multigroup methods of Rahnema and Forget ever be of practical use?**

See, e.g., these:

Rahnema, Farzad, Steven Douglass, and Benoit Forget. "Generalized energy condensation theory." Nuclear Science and Engineering 160.1 (2008): 41-58.

Zhu, Lei, and Benoit Forget. "A discrete generalized multigroup energy expansion theory." Nuclear Science and Engineering 166.3 (2010): 239-253.

## The Discrete-Generalized Multigroup Method

Start by collecting fine groups (indexed by $g$) into coarse groups (indexed by $G$):

\begin{equation}
  \boldsymbol{\hat{\Omega}}  \cdot \nabla \psi_{c,g}(\boldsymbol{\hat{\Omega}}) +
    \Sigma^{t}_{c,g} \psi_{c,g}(\boldsymbol{\hat{\Omega}}) =
    \frac{1}{4\pi} \sum\limits^{N_G}_{G'=1} \sum\limits_{g'\in G'} \left (\Sigma^{s}_{c,g\leftarrow g'} +
    \frac{\chi_{c,g}}{k} \nu\Sigma^{f}_{c,g'} \right ) \phi_{c,g'} \, .
\end{equation}

Suppose we have a set of vectors $P^G_{i,g}$ that satisfy $P^G_{0,g} = \alpha \mathbf{1}$ and

\begin{equation}
 \sum\limits_{g\in G} P^G_{i,g} P^G_{j,g} = \delta_{ij} \, .
\end{equation}

Then let
\begin{equation}
 \psi_{c,G,i} (\boldsymbol{\hat{\Omega}})  \equiv \sum\limits_{g\in G} P^G_{i,g} \psi_{c,g} (\boldsymbol{\hat{\Omega}})
 \qquad \text{and} \qquad
 \psi_{c,g}(\boldsymbol{\hat{\Omega}})  = \sum\limits^{N^G_i}_{i=0} P^G_{i,g} \psi_{c,G,i} (\boldsymbol{\hat{\Omega}})\, .
\end{equation}

We play the same POD game (substitution and multiply on the left):

\begin{equation}
 \boldsymbol{\hat{\Omega}}  \cdot \nabla \psi_{c,G,i}(\boldsymbol{\hat{\Omega}}) +
    \Sigma^{t}_{c,G,0} \psi_{c,G,i}(\boldsymbol{\hat{\Omega}}) + \delta_{c,G,i}(\boldsymbol{\hat{\Omega}}) \psi_{c,G,0}(\boldsymbol{\hat{\Omega}}) =
    \frac{1}{4\pi} \sum\limits^{N_G}_{G'=1} \Sigma^{s}_{c,G\gets G',i} \phi_{c,G',0} +
    \frac{\chi_{c,G,i}}{4\pi k} \sum\limits^{N_G}_{G'=1} \nu\Sigma^{f}_{c,G'} \phi_{c,G',0} \, ,
\end{equation}

where, e.g.,

\begin{equation}
  \Sigma^{t}_{c,G,0} \equiv
      \frac{\sum\limits_{g\in G} \Sigma^{t}_{c,g} \phi_{c,g} }
           {\sum\limits_{g\in G} \phi_{c,g}} \, ,
 \quad
 \delta_{c,G,i}(\boldsymbol{\hat{\Omega}}) \equiv
      \frac{\sum\limits_{g\in G} P^G_{i,g} \left(\Sigma^{t}_{c,g} - \Sigma^{t}_{c,G,0} \right ) \psi_{c,g}(\boldsymbol{\hat{\Omega}})}
           {\psi_{c,G,0}(\boldsymbol{\hat{\Omega}})} \, , \quad \text{and} \quad
  \Sigma^{s}_{c,G\gets G',i} \equiv
      \frac{\sum\limits_{g\in G} P^G_{i,g} \sum\limits_{g'\in G'} \Sigma^{s}_{c,g\leftarrow g'} \phi_{c,g'} }
           {\sum\limits_{g'\in G} \phi_{c,g'}} \, ,
\end{equation}

Some points:

 - The $i = 0$ equation is almost normal.
 - The "cross sections" can largely be precomputed (so fine-group data not needed downstream)
 - We've formally done nothing useful yet
 - If we truncate, the problem is cheaper, but is accuracy any good?

## SPH-esque Correction

Good reactor modeling is often focused on preserving reaction rates.  In general, if we simplify our phase space (average over space, energy, reduce the order, whatever) or "homogenize", we have this sad problem:

$$
\bar{\Sigma}^\text{ref}_{r,g}\bar{\phi}^\text{ref}_{r,g} = \bar{R}^\text{ref}_{r,g}
  \neq  \bar{R}^\text{homog}_{r,g} = \bar{\Sigma}^\text{homog}_{r,g}\bar{\phi}^\text{homog}_{r,g}\, ,
$$

In other words, the simple model (or data, or method, etc.) doesn't match the fancy one.

To balance the books, let's require that

\begin{equation}
    \bar{\Sigma}^\text{homog}_{r,g} = \bar{\Sigma}^\text{ref}_{r,g}\frac{\bar{\phi}^\text{ref}_{r,g}}{\bar{\phi}^\text{homog}_{r,g}}
    =\omega_{r,g}\bar{\Sigma}^\text{ref}_{r,g}\, ,
    \label{eq:sph}
\end{equation}

where $\omega_{r,g}$ is the SPH factor.

Does it work for the DGM "modes"?

To conclude:

  - DGM with SPH *may* provide a method that is more problem independent than traditional multigroup approaches (for equivalent "sizes").
  - POD smokes DLP

R. L. Reed, J. A. Roberts, *Effectiveness of the discrete generalized multi- group method based on truncated, pod-driven basis sets*, Annals of Nuclear Energy 126 (2019) 253–261.

R. L. Reed, J. A. Roberts, *Extension of the discrete generalized multigroup method using SPH factors*, under review

R. L. Reed, *Improvements to the discrete generalized multigroup method based on proper orthogonal decomposition and SPH factors*, PhD Thesis, K-State, 2019

# From SVD to DMD

Remember our snapshot matrix $\mathbf{X}$ from above generated by solving (or watching) a system described by $\mathbf{x}_t = \mathbf{f}(\mathbf{x}, t)$.

For convenience, let

$$
  \mathbf{X_+} = [\mathbf{x}_1, \mathbf{x}_2, \ldots, \mathbf{x}_n]
  \quad \text{and} \quad
  \mathbf{X_-} = [\mathbf{x}_0, \mathbf{x}_1, \ldots, \mathbf{x}_{n-1}]
$$

If $\mathbf{f}(\mathbf{x}, t) = \mathcal{A}\mathbf{x}$, then $\mathbf{x}(t) = e^{\mathcal{A}t} \mathbf{x}_0$ is known, and $\mathbf{X_+} = \mathbf{A}\mathbf{X_-}$ for $\mathbf{A} = e^{\mathcal{A} \Delta}$.

For other systems, we just hope for a good approximation to $\mathbf{A}$.

First, define

$$
  \mathbf{A} = \mathbf{X}_{+} \mathbf{X}_{-}^{\dagger} = \mathbf{X}_{+} \mathbf{V}\boldsymbol{\Sigma}^{-1}\mathbf{U}^T
$$

where $\dagger$ indicates the pseudo inverse and $\mathbf{X}_{-} = \mathbf{U}\boldsymbol{\Sigma}\mathbf{V}^T$ is the (rank-$r$, truncated) SVD.

Because $\mathbf{A}$ is large, let's define a rank-$r$ operator and its eigendecomposition:

$$
  \tilde{A} = \mathbf{U}^T \mathbf{A} \mathbf{U} \quad \longrightarrow \quad \tilde{A} = \mathbf{W}\boldsymbol{\Lambda} \mathbf{W}^T \, .
$$

It turns out that

  1.  The $\lambda$'s are eigenvalues of $\mathbf{A}$.
  2.  The vectors $ \boldsymbol{\Phi} = \mathbf{X}_+ \mathbf{V}\boldsymbol{\Sigma}^{-1}\mathbf{W}$ are
      eigenvectors of $\mathbf{A}$ and called "DMD modes."
  3.  Schmid's original work proposed $\boldsymbol{\Phi} = \mathbf{U}\mathbf{W}$.
  4.  If $\mathbf{z}$ satisfies $\mathbf{\tilde{A}}^T \mathbf{z} = \lambda \mathbf{z}$, then
      $\mathbf{U}\mathbf{z}$ is an eigenvector of $\mathbf{A}^T$.  (And suddenly, approximate, data-driven adjoints seem so....close...)

- For an entire overview of DMD, see Kutz, J. Nathan, et al. *Dynamic mode decomposition: data-driven modeling of complex systems*. Society for Industrial and Applied Mathematics, 2016.

- Close to the first paper (but easier to get) is Schmid, Peter J. "Application of the dynamic mode decomposition to experimental data." *Experiments in fluids* 50.4 (2011): 1123-1130.

With the spectrum of $\mathbf{A}$ approximated, it follows that

$$
  \mathbf{x}(t) \approx \sum_i \mathbf{\phi}_i e^{\lambda_i t/\Delta} b_i \, ,
$$

where a common approach sets $\mathbf{b} = \boldsymbol{\Phi}^{\dagger} \mathbf{x}_0$.

# DMD for Reactor "Transients"

Three things we've tried with DMD:

  - **Approximating isotopic evolution in multi-material systems (i.e., space, time, and species)**
  - Approximating local and integral flux/power/etc. for fast(er) transients via "fitting"
  - Doing the same via projection (scoping not too successful)
  - **Pretending power (and related) iteration is a real transient headed toward steady conditions.**

Abdo, Mohammad, Rabab Elzohery, and Jeremy A. Roberts. "Modeling isotopic evolution with surrogates based on dynamic mode decomposition." Annals of Nuclear Energy 129 (2019): 280-288.

Abdo, et al. Analysis of the LRA Reactor Benchmark Using Dynamic Mode Decomposition. Trans ANS. 119.

Elzohery, R. and Jeremy Roberts. "Exploring Transient, Neutronic, Reduced-Order Models using DMD/POD-Galerkin and Data-Driven DMD."  PHYSOR 2020.

Roberts, Jeremy A., et al. "Acceleration of the power method with dynamic mode decomposition." Nuclear Science and Engineering 193.12 (2019): 1371-1378.

# Isotopics

The application was motivated by some scoping UQ done for the K-State TRIGA Reactor.

For the analysis, nuclides were tracked in each fuel element over a set period at full power given initial loadings (and spread about those nominal values).

# The Power Method

Given a matrix $\mathbf{A}$, find its largest eigenvalue $\lambda$ and corresponding eigenvector $\mathbf{v}$ by

  1. Guess $\mathbf{v}$ (normalize to 1)
  2. $\mathbf{v}' = \mathbf{Av}$
  3. $\lambda = |\mathbf{v}'|$
  4. $\mathbf{v} = \mathbf{v}' / |\mathbf{v}'|$
  5. repeat 2-4.

For reactor models, this sort of mimics reality: perturbed initial state dies off to the fundamental mode.  Maybe DMD can help?  The idea:

  1.  Do several PM iterations
  2.  Do DMD on those "snap shots"
  3.  Extrapolate in time and cross your fingers.

So much more to chat about, but we're done for today.

Thanks - jaroberts@ksu.edu


# Simple Linear Dynamical System Demo

Linear Dynamical Systems (LDS) are used to model data that evolve in principled ways. They are widely used in control systems, signal processing, neuroscience, and other fields. Our focus in on learning and inference for LDS, and a good reference for this is Chapter 13.3 of [Pattern Recognition and Machine Learning](http://users.isr.ist.utl.pt/~wurmd/Livros/school/Bishop%20-%20Pattern%20Recognition%20And%20Machine%20Learning%20-%20Springer%20%202006.pdf) by Christopher Bishop.

For LDS with Poisson observations, especially as they are used in neuroscience, [Estimating State and parameters in state space models of spike trains](https://www.cambridge.org/core/services/aop-cambridge-core/content/view/FAB8634C2790F3461E3E86BB632EAE6F/9781139941433c6_p137-159_CBO.pdf/estimating_state_and_parameters_in_state_space_models_of_spike_trains.pdf), by J.H Macke _et al_ is a good reference.

## 1. Setup
The line `import ssm` imports the package for use. Here, we have also imported a few other packages for plotting.

**Note**: We recently added Seaborn to the list of dependencies. If you have an older installation of SSM, you may need to manually install Seaborn using `pip install seaborn` or `conda install seaborn`.

```python
import autograd.numpy as np
import autograd.numpy.random as npr
npr.seed(0)

import matplotlib.pyplot as plt
%matplotlib inline

from matplotlib.gridspec import GridSpec

import seaborn as sns
sns.set_style("white")
sns.set_context("talk")

color_names = ["windows blue",
               "red",
               "amber",
               "faded green",
               "dusty purple",
               "orange",
               "clay",
               "pink",
               "greyish",
               "mint",
               "light cyan",
               "steel blue",
               "forest green",
               "pastel purple",
               "salmon",
               "dark brown"]

colors = sns.xkcd_palette(color_names)

import ssm
from ssm.util import random_rotation
from ssm.plots import plot_dynamics_2d

# Specify whether or not to save figures
save_figures = False
```

## 2. Create an LDS Object

A Linear Dynamical System consists of a continuous state variable, $x \in \mathbb{R}^D$, which evolves according to linear dynamics.
At time-step $t$, we observe an emission (sometimes called an observation) $y_t \in \mathbb{R}^N$. Similarly, we use $x_t$ to denote the state at time $t$.

The state dynamics obey the following relation (Bishop 2006):

$$
\begin{align}
x_t &= A x_{t-1} + V u_{t} + b + w_t \\
\end{align}
$$

The vector $b$ is an offset vector, which can drive the dynamics in a particular direction.
The terms $w$ and $v$ are noise terms, which perturb the states and observations.
Most commonly these are modeled as zero-mean multivariate Gaussians,
but one nice feature of SSM is that it supports many distributions for these noise terms.

We support the following dynamics models, which can be specified by passing an argument to the constructor the LDS class, e.g `dynamics="gaussian"`:

1. **`none`**: Model has no dynamics. At each time-step, the state is drawn independently from a Gaussian. This is the equivalent of filling the dynamics matrix $A$ with zeros.
1. **`gaussian`**: Model has linear-gaussian dynamics. $w \sim \mathcal{N}(0, \Sigma_w )$.
1. **`diagonal_gaussian`**: This is a special case of the Gaussian dynamics model where the noise term has diagonal covariance.
1. **`studentst`**: $w \sim t(0, \Sigma_w, \nu_w)$.
1. **`diagonal_studentst`**: Constrains covariance matrix to be diagonal.

To specify an LDS, we also need to specify how the emissions $y_t$ are generated (the emissions are what we observe).
The most common emission model is the simple linear-Gaussian emission model:
$$
y_t = Cx_t + d + F u_t + v_t
$$
In this case, $C$ is called the measurement matrix, $d$ is an offset or bias term, and $F$ is called the feedthrough matrix or passthrough matrix (it passes the input directly to the emission).
$v_t \sim \mathcal{N}(0, \Sigma_v)$ is a noise term, which in the simplest case is Gaussian, but can also be modeled to come from other distributions.
We support the following emission models.

1. **`gaussian`**: The most common emission model. $y_t \sim \mathcal{N}(C x_t + d + F u_t, \Sigma_v)$
1. **`studentst`**: Same as above, but uses the Students-T distribution instead of the normal distribution.
1. **`poisson`**: Useful for modeling count or event data. Under this model, the observations are nonnegative integers drawn from a poisson distribution (Macke 2011).
The latent state controls the rate parameter, in a non-linear fashion as follows. Let $y_{i,t}$ be the $i$-th dimension of our observation at time $t$.
We'll also make use of an excitation vector $g_t = C x_t + d + F u_t$, and additionally a mean function $\eta(\bullet)$.
The observations are drawn from a poisson distribution at rates specified by the excitation vector:
$$P( y_{i,t} \mid g_{i,t}) = \frac{1}{y_{i,t}!} \eta(g_{i,t})^{y_{i,t}}\exp{(-\eta(g_{i,t}))}.$$ By default, the mean function is set to $\exp(\bullet)$.

1. **`bernoulli`**:
As in the Poisson case, we have an excitation vector $g_t = C x_t + d + F u_t$, and a mean function $\eta(\bullet)$. In this case, though, the mean function is always the logistic function, becuase its outputs need to be between zero and one. The observations are drawn from a bernoulli distribution where $y_{i,t}$ is 1 with probability $\eta(g_{i,t})$.

For all of the above we support orthogonal variants, in which the measurement matrix $C$ is constrained to be orthogonal.

_Note: SSM supports many other emission models. We are in the process of creating full-standalone documentation to describe them. For now, the best way to learn about SSM's other functionality is look at the source code. The emission models are described in emissions.py._

The below line creates an instance of the SSM's `LDS` class:
`true_lds = ssm.LDS(obs_dim, state_dim, dynamics="gaussian", emissions="gaussian")`

Following that, we modify the dynamics matrix (the matrix $A$ in the above equations) so that our system will have some interesting behavior.

The lines most relevant for a user of SSM are these:
`true_lds.dynamics.A = A
true_lds.dynamics.b = b`

The first line sets the dynamics matrix $A$ to be the one which we create. The second line sets the offset vector $b$.

```python
# Set the parameters of the LDS
time_bins = 200   # number of time bins
state_dim = 2     # number of latent dimensions
obs_dim = 10      # number of observed dimensions
```

```python
# Make an LDS with somewhat interesting dynamics parameters
true_lds = ssm.LDS(obs_dim, state_dim, emissions="gaussian_orthog")

A0 = .99 * random_rotation(state_dim, theta=np.pi/20)
S = np.arange(1, state_dim+1)
R = np.linalg.svd(npr.randn(state_dim, state_dim))[0] * S
A = R.dot(A0).dot(np.linalg.inv(R))
b = npr.randn(state_dim)

# Set the dynamics matrix (A) and the
true_lds.dynamics.A = A
true_lds.dynamics.b = b
```

## 3. Sample from the LDS and plot
Since our state variable $x$ is two-dimensional, we can plot its path over time in the 2D plane. Below, we plot the trajectory of the state, along with a vector field showing the dynamics of the system. The small blue arrows show the direction that the dynamics of the system will "push" the state at each point.

Though it's a bit beyond the scope of this notebook, a lot can be said about how the matrix $A$ determins the dynamics of the system. For example, if the spectral radius of $A$ is larger than 1, the system will be unstable, and the state will blow up as time goes to infinity (the spectral radius is the largest singular value).
We've been hand-wavy in defining stability and instability.
In our case, we have explicitly constructed a stable system, and so we see that the dynamics tend towards a fixed point.

We see that the trajectory (shown in black) follows the blue arrows quite closely, spiraling towards the center. Why doesn't the trajectory follow the arrows exactly? This is because we have noise on the dynamics equation (the $w_t$s in the equations above).

```python
# Sample from the LDS object.
states, emissions = true_lds.sample(time_bins)

# Plot the dynamics vector field
q = plot_dynamics_2d(A,
                     bias_vector=b,
                     mins=states.min(axis=0),
                     maxs=states.max(axis=0),
                     color=colors[0])

plt.plot(states[:,0], states[:,1], '-k', lw=3)
plt.xlabel("$x_1$")
plt.ylabel("$x_2$")
plt.title("Simulated Latent States")
plt.tight_layout()

if save_figures:
    plt.savefig("lds_1.pdf")
```

### Exercise 3.1: Modify trajectory
See if you can modify the LDS object created in Section 2 so that the system will be unstable. In other words, you want to set the matrix $A$ so that the largest singular value is greater than 1.

One way of doing this is by starting with the matrix $A$ we used previously, and taking the SVD of that matrix. After taking the SVD, you can modify the largest singular value, and then re-form the dynamics matrix.
After sampling from the system and plotting, do you notice anythign different?

### Exercise 3.2: Advancing $t$ timesteps.
Say the LDS starts at state $x_0$ at time zero. Can you derive an expression for the mode of $p(x_t \mid x_0)$, where $t$ is some number of timesteps in the future?
In other words, what is $ \mathbb{E} \left[x_t \mid x_0 \right]$ ?

## 4. Visualize States and Observations
In the above section, we were able to plot a trajectory for the latent state over time. Since our emissions are 10-dimensional, we can't show a trajectory for them in a 10-d plane (matplotlib doesn't support that yet). Instead, we'll plot each emission coordinate $y_i$ as a function over time. We do the same for the simulated latent states.

```python
plt.figure(figsize=(8, 6))
gs = GridSpec(2, 1, height_ratios=(1, obs_dim/state_dim))

# Plot the continuous latent states
lim = abs(states).max()
plt.subplot(gs[0])
for d in range(state_dim):
    plt.plot(states[:, d] + lim * d, '-k')
plt.yticks(np.arange(state_dim) * lim, ["$x_{}$".format(d+1) for d in range(state_dim)])
plt.xticks([])
plt.xlim(0, time_bins)
plt.title("Simulated Latent States")

lim = abs(emissions).max()
plt.subplot(gs[1])
for n in range(obs_dim):
    plt.plot(emissions[:, n] - lim * n, '-k')
plt.yticks(-np.arange(obs_dim) * lim, ["$y_{{ {} }}$".format(n+1) for n in range(obs_dim)])
plt.xlabel("time")
plt.xlim(0, time_bins)

plt.title("Simulated emissions")
plt.tight_layout()

if save_figures:
    plt.savefig("lds_2.pdf")
```

## 5. Fit an LDS to Data
Simulating a Linear-Gaussian LDS is all fine, but you're probably interested in fitting an LDS to your own datasets. This is useful, for instance, if you believe that your data are well described by linear dynamical system, and you're interested in examining the dynamics of that system.

In the below cells, we'll treat the variable `emissions` which we generated above as data which comes from an unknown LDS.
The line `data = emissions` is just to emphasize that our new LDS will be fit from data.

1. First, we make a new LDS object:
`lds = ssm.LDS(obs_dim, state_dim, emissions="gaussian")`

2. Then we fit the LDS:
`
elbos, q = lds.fit(data, method="laplace_em", num_iters=10)
`

3. Next, we get the posterior of the latent states with:
`state_means = q.mean_continuous_states[0]
`

4. Finally smooth the observations using our dynamics model:
`smoothed_obs = lds.smooth(state_means, data)`

**`LDS.fit`** returns a tuple of `posterior, elbos`. `posterior` is an object representing the posterior of the data given the observations, $p(x \mid y)$.
By default, SSM parameterizes the posterior as a Gaussian with a block-tridiagional precision matrix.
Of most use will probably be `posterior.mean_continuous_states[0]` which returns $\mathbb{E}q(x \mid y)$.
This is our best estimate of the state trajectory given our observations.
By default, calling `LDS.fit` will use the Laplace-EM algorithm, which is a variational inference method. In the case of Gaussian dynamics and Gaussian dynamics, Laplace-EM reduces to the normal EM update rules.
Other fitting methods, like `"svi"` (stochastic variational inference) are also supported, but don't tend to work as well in practice.
We'd recommend sticking to the default fitting methods, at least for now.

**`elbos`** stands for "evidence lower bound" and it is a lower bound on the log-likelihood of the data. It is used to monitor convergence of the EM algorithm

**Note 1**: You'll notice when you call `fit` that you actually see two progress bars, one labeled "LP" and one labeled "ELBO". This is because internally, the fitting is done in two steps. In order to get a good initialization of the paramters, we first initialize using an autoregressive HMM.
Then, we obtain the final parameters using the Laplace-EM algorithm. The first progress bar is showing progress for the initialization (fitting an ar-HMM).

**Note 2**: We realize some of the APIs are quite weird. For example, `mean_continuous_states[0]` is a bit strange. This is because the APIs are currently shared between vanilla LDS and switching LDS. We plan on fixing this soon, and will update the notebook.

```python
data = emissions

print("Fitting LDS with Laplace-EM using structured variational posterior")
lds = ssm.LDS(obs_dim, state_dim, emissions="gaussian")
elbos, q = lds.fit(data, method="laplace_em", num_iters=10)

# Get the posterior mean of the continuous states
state_means = q.mean_continuous_states[0]

# Smooth the data under the variational posterior
smoothed_obs = lds.smooth(state_means, data)
```

    Fitting LDS with Laplace-EM using structured variational posterior
    Initializing with an ARHMM using 25 steps of EM.

    HBox(children=(IntProgress(value=0, max=25), HTML(value='')))

    HBox(children=(IntProgress(value=0, max=10), HTML(value='')))

Below, we plot the ELBOs which we got from the `fit` function to check that our fitting procedure converged.

```python
# Plot the ELBOs
plt.plot(elbos, label="Laplace-EM")
plt.xlabel("Iteration")
plt.ylabel("ELBO")
plt.legend()
```

## 6. Visualize Model Fit
Below, we show the true and estimated states over time. Although there is an offset between the true state and our estimate, we see that the dynamics of the system seem to match pretty well: our estimate follows a similar trajectory as the true state.

We also use our model to "smooth" the emissions by removing noise. Once we have learned the dynamics of the system, we see that we can remove some of the jitter from the emissions.

```python
plt.figure(figsize=(8,4))
for d in range(state_dim):
    plt.plot(states[:,d] + 4 * d, '-k', label="True States" if d==0 else None)
    plt.plot(state_means[:,d] + 4 * d, '--', color=colors[0], label="Estimated States" if d==0 else None)
plt.ylabel("$x$")
plt.xlabel("time")
plt.legend(loc='upper right')
plt.title("True and Estimated States")
plt.show()
```

```python
# Plot the smoothed emissions
plt.figure(figsize=(8,4))
for n in range(obs_dim):
    plt.plot(emissions[:, n] + 4 * n, '-k', label="Original" if n == 0 else None)
    plt.plot(smoothed_obs[:, n] + 4 * n, '--', color=colors[0], label="Smoothed" if n == 0 else None)
plt.legend()
plt.xlabel("time")
plt.title("Original and Smoothed Observations")
plt.show()
```

Below, we visualize the true dynamics, as well as the dynamics of the system we learned. We would hope that the overall features would be the same, though the overall dynamics might be rotated.

```python
# Extract dynamics matrix from the true model
A_true = true_lds.dynamics.A
b_true = true_lds.dynamics.b

A_est = lds.dynamics.A
b_est = lds.dynamics.b

f, ax = plt.subplots(1, 2, figsize=(10, 4))
plot_dynamics_2d(A_true, b_true, npts=10, axis=ax[0], color=colors[0])
ax[0].set_xlabel("$x_1$")
ax[0].set_ylabel("$x_2$")
ax[0].set_title("True Dynamics")

plot_dynamics_2d(A_est, b_est, npts=10, axis=ax[1], color=colors[1])
ax[1].set_xlabel("$x_1$")
ax[1].set_ylabel("$x_2$")
ax[1].set_title("Inferred Dynamics")

plt.tight_layout()
```

### Exercise 6.1: Mis-specified Dynamics Model
What you expect to happen if you fit an HMM to your data when the true dataset was generated by an LDS? Assume that in both cases, the observation model is Gaussian, so that only the dynamics model is misspecified.

How would the performance of your model change as you increase the number of discrete states in the HMM?

## 7. Simulate Poisson data from a Poisson LDS with the same dynamics

In the below cells, we create an LDS with poisson observations that shares the dynamics of the LDS above.
The line:

`plds = ssm.LDS(obs_dim, state_dim, emissions="poisson_orthog", emission_kwargs=dict(link="softplus"))
`

creates an LDS object with Poisson observations.
Specifying `link="softplus"` specifes that the link function will be the softplus function.
The term `link` refers to the link a generalized linear model, which controls how the linear latent variable, in this case the state $x_t$ controls the emission distribution.
When creating a Poisson or Bernoulli LDS, the user specifies the link function, which allows in-turn sets the mean function of the emissions.

For Poisson LDS, SSM supports two link functions:
1. `link="softplus"`: The mean function $\eta(x)$ is set to the inverse softplus function.
2. `link="log"`: The mean function  $\eta(x)$ is set to the exponential function.

Below, we copy the parameters from our prior LDS to the new object:
`plds.dynamics.params = copy.deepcopy(true_lds.dynamics.params)
`

We use `copy.deepcopy` to make sure we don't accidentally change the parameters in the other LDS object.
The Poisson emissions are composed of a linear mapping from the continuous states:
$$z_t = C x_t + d + F u$$
In the below line, we set the bias vector $d$ in the above equation to zeros:
`
plds.emissions.ds = 0 * np.ones(obs_dim)
`

After sampling from our Poisson LDS, we sample and show the trajectory, as before. We then also visualize our Poisson observations. Since Poisson observations are nonnegative integers, we plot them as a color density.

```python
import copy

plds = ssm.LDS(obs_dim, state_dim, emissions="poisson_orthog", emission_kwargs=dict(link="softplus"))
plds.dynamics.params = copy.deepcopy(true_lds.dynamics.params)
plds.emissions.ds = 0 * np.ones(obs_dim)
states_plds, obs_plds = plds.sample(time_bins)
```

```python
# Plot the dynamics vector field
plot_dynamics_2d(plds.dynamics.A,
                 plds.dynamics.b,
                 mins=states_plds.min(axis=0),
                 maxs=states_plds.max(axis=0),
                 color=colors[0])
plt.plot(states_plds[:,0], states_plds[:,1], '-k', lw=3)
plt.xlabel("$x_1$")
plt.ylabel("$x_2$")
plt.title("Simulated Latent States")
plt.tight_layout()

if save_figures:
    plt.savefig("lds_6.pdf")
```

```python
# Plot the dynamics vector field
plt.figure(figsize=(8, 6))
gs = GridSpec(2, 1, height_ratios=(1, obs_dim/state_dim))

# Plot the continuous latent states
lim = abs(states_plds).max()
plt.subplot(gs[0])
for d in range(state_dim):
    plt.plot(states_plds[:, d] + lim * d, '-k')
plt.yticks(np.arange(state_dim) * lim, ["$x_{}$".format(d+1) for d in range(state_dim)])
plt.xticks([])
plt.xlim(0, time_bins)
plt.title("Simulated Latent States")

lim = abs(obs_plds).max()
plt.subplot(gs[1])
plt.imshow(obs_plds.T, cmap="Greys", aspect="auto")
plt.yticks(np.arange(obs_dim), ["$y_{{ {} }}$".format(n+1) for n in range(obs_dim)])
plt.xlabel("time")
plt.xlim(0, time_bins)
plt.title("Simulated Poisson Observations")
plt.colorbar()

plt.tight_layout()

if save_figures:
    plt.savefig("lds_7.pdf")
```

### Exercise 7.1: Mis-specified Observation Model
What you expect to happen if you fit an LDS with Gaussian observations to a system where the true observation model is Poisson?

You can try this by following the steps in Section 5 using the data we generated in Section 7.

notebooks/1b Simple Linear Dynamical System.ipynb


```python
%matplotlib inline
import matplotlib.pyplot as plt
import seaborn as sns
plt.rcParams["figure.figsize"] = (16, 9)
import sympy
import numpy as np
from fenics import *

import sympy2fenics as sf
from periodic_bc import FlatTorus3D
from time_stepping import leapfrog, theta_method
```

    /usr/local/lib/python3.5/dist-packages/matplotlib/__init__.py:1405: UserWarning:
    This call to matplotlib.use() has no effect because the backend has already
    been chosen; matplotlib.use() must be called *before* pylab, matplotlib.pyplot,
    or matplotlib.backends is imported for the first time.

      warnings.warn(_use_error_msg)

```python
"""Chosen exact solution for testing."""
u_ = sf.str2sympy("(sin(z-t), sin(x-t), sin(y-t))")
# Initial data
a = Expression(sf.sympy2exp(u_), t=0.0, degree=3)
# Initial velocity
b = Expression(sf.sympy2exp(sympy.diff(u_, "t")), t=0.0, degree=3)
# Exact solution
u = lambda t: Expression(sf.sympy2exp(u_), t=t, degree=3)
# curl(u x curl(u)) for use later
g = lambda t: Expression(sf.sympy2exp(sf.curl(u_.cross(sf.curl(u_)))), t=t, degree=3)
```

```python
"""Basic setup"""
# Domain
L = pi * 2.0
bc = FlatTorus3D(L, L, L)
# Mesh
N = 8
mesh = UnitCubeMesh(N, N, N)
coor = mesh.coordinates()
coor[:, :] = coor * L
mesh.bounding_box_tree().build(mesh)
# Function space
V = FunctionSpace(mesh, 'N1curl', 1, constrained_domain = bc)
```

```python
"""
Maxwell wave equation: for divergence free a, b, and f, find u such that
    u'' + curl curl u = f,
    u(0) = a, u'(0) = b.
For our chosen exact solution, f=(0,0,0).
"""
# Right-hand side
f = lambda t: Expression(("0.0", "0.0", "0.0"), degree=2)
# Solve
ah = interpolate(a, V)
bh = interpolate(b, V)
v = TestFunction(V)
A = lambda u, v: dot(curl(u), curl(v))
fh = lambda t, v: dot(f(t), v)
```

```python
# Crank-Nicolson solve
(ts, uh) = theta_method(A, fh, ah, bh, 0.5, 0.1, 200)
```

    Progress: 0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..done

```python
# Auxiliary functions
def proj_divfree(u):
    """Decompose a discrete vector field into the gradient and the div-free part."""
    # Solve the equation: -Δw=div(u)
    W = FunctionSpace(mesh, "CG", 1, constrained_domain = bc)
    w = TrialFunction(W)
    y = TestFunction(W)
    q = Function(W)
    problem = LinearVariationalProblem(dot(grad(w), grad(y)) * dx, dot(u, grad(y)) * dx, q)
    solver = LinearVariationalSolver(problem)
    solver.parameters["linear_solver"] = "minres"
    solver.parameters["preconditioner"] = "hypre_amg"
    solver.solve()
    # remove the gradient part
    g = project(grad(q), u.function_space())
    h = Function(u.function_space())
    h.vector()[:] = u.vector() - g.vector()
    return (h, g)

def eval_avg(u):
    """Compute the total integral of a vector field."""
    return np.array([assemble(u[i] * dx) for i in range(3)])

def l2_norm(u):
    """Compute the L2 norm of a Function."""
    return np.sqrt(assemble(dot(u, u) * dx))

def proj_imcurl(u):
    """Project a discrete vector field to its im(curl) part."""
    divfree = proj_divfree(u)[0]
    avg = eval_avg(divfree)
    return project(divfree - Constant(avg), u.function_space())

def plot_x_component_at_a_point(ts, uh, u, pt=(0.0, 0.0, 0.0), name="test"):
    """Plot the value of the x-component of the numerical and exact solution at one point."""
    peval = lambda u: u((pt))[0]
    up = []
    uhp = []
    for (i, t) in enumerate(ts):
        up.append(peval(u(t)))
        uhp.append(peval(uh[i]))
    plt.plot(ts, uhp, label="Numerical solution")
    plt.plot(ts, up, label="Exact solution")
    plt.legend()
    plt.xlabel("Time")
    plt.ylabel("u(t, (0,0,0))")
    plt.savefig("{}.svg".format(name), transparent=True, bbox_inches='tight', pad_inches=0)

def plot_violation(ts, uh, name="test"):
    """Plot the magnitude of the average and the L2 norm of the gradient part of the numerical solution."""
    avgs = [np.linalg.norm(eval_avg(uh_)) for uh_ in uh]
    grads = [l2_norm(proj_divfree(uh_)[1]) for uh_ in uh]
    plt.plot(ts, avgs, label="Magnitude of the averge")
    plt.plot(ts, grads, label="$L^2$-norm of gradient part")
    plt.legend()
    plt.xlabel("Time")
    plt.savefig("{}.svg".format(name), transparent=True, bbox_inches='tight', pad_inches=0)
```

```python
plot_x_component_at_a_point(ts, uh, u, name="linear_orig_sol")
```

```python
plot_violation(ts, uh, name="linear_orig_violation")
```

```python
# Compute with initial data in the correct discrete space
ah0 = proj_imcurl(ah)
bh0 = proj_imcurl(bh)
# Crank-Nicolson solve
(ts, uh) = theta_method(A, fh, ah0, bh0, 0.5, 0.1, 200)
```

    Progress: 0%..10%..20%..30%..40%..50%..60%..70%..80%..90%..done

```python
plot_x_component_at_a_point(ts, uh, u, name="linear_projected_sol")
```

```python
plot_violation(ts, uh, name="linear_projected_violation")
```

```python
"""
Nonlinear problem:
    u'' + curl (1 + εu ×) curl u = f
This particular problem: u is the same as before but
    f = εu × curl u
"""
ε = 0.1
ah = interpolate(a, V)
bh = interpolate(b, V)
A = lambda u, v: dot(curl(u) + ε * cross(u, curl(u)), curl(v))
fh = lambda t, v: ε * dot(g(t), v)
```

```python
# Crank-Nicolson solve
(ts, uh) = theta_method(A, fh, ah, bh, 0.5, 0.1, 200)
```

    Progress: 0%..10%..20%..30%..blowup at t=71.30000000000022.

```python
plot_x_component_at_a_point(ts, uh, u, name="nonlinear_orig_sol")
```

```python
plot_violation(ts, uh, name="nonlinear_orig_violation")
```

```python
# Compute with initial data in the correct discrete space
ah0 = proj_imcurl(ah)
bh0 = proj_imcurl(bh)
# Crank-Nicolson solve
(ts, uh) = theta_method(A, fh, ah0, bh0, 0.5, 0.1, 200)
```

    Progress: 0%..10%..20%..30%..40%..50%..blowup at t=100.69999999999855.

```python
plot_x_component_at_a_point(ts, uh, u, name="nonlinear_projected_sol")
```

```python
plot_violation(ts, uh, name="nonlinear_projected_violation")
```


$Un$ $sistema$ $de$ $ecuaciones$ $lineales$ $2x2$ $de$ $la$ $forma:$

$$\left\{\begin{matrix}
 & ax+bx=c \\
 & dx+ex=f
\end{matrix}\right.$$

$Llamamos$ $arreglo$ $rectangular$ $del$ $sistema,$ $al$ $arreglo$ $de$ $números$ $(coeficientes)$ $de$ $la$ $siguiente$ $manera:$

$$\begin{bmatrix}
a &b  &c \\
 d& e &f
\end{bmatrix}$$

$El$ $siguiente$ $programa$ $te$ $permite$ $introducir$ $los$ $datos$ $para$  $un$ $sistema$ $2x2,$ $introduce$ $lo$ $que$ $se$ $te$ $pide:$

```python
#importamos las bibliotecas que utilizaremos
%matplotlib inline
from matplotlib import pyplot as plt
from IPython.display import display, Math, Latex, Markdown
from sympy.interactive import printing
from sympy import Eq, solve_linear_system, Matrix,Symbol, init_printing, init_session
from numpy import linalg
import numpy as np
import sympy as sp
printing.init_printing(use_latex=True)
init_printing(use_latex='mathjax')

#Definimos las variables que se utilizarán guardar datos

def printmd(string):
    display(Markdown(string))

#Definimos las variables para los coeficientes

x, y=sp.symbols ('x,y')
a,b,c,d,e,f,g,h,r,j,k,l,m,n,a1,b1,c1=sp.symbols ('a,b,c,d,e,f,g,h,r,j,k,l,m,n,a1,b1,c1')

to= r'$$Introduce\ los\ coeficientes\ de\ tu\ primera\ ecuación\ lineal,\ de\ la\ forma\ ax+by=c$$'

printmd(to)

#Para que Python interprete la entrada como un número entero, se debe utilizar la función int() para decimal se usa float()

a= int(input("Cuál es el coeficiente de x: "))
a
b=int(input("Cuál es el coeficiente de y: "))
b
c=int(input("Cuál es el término constante: "))

#
printmd('$$Tu\ ecuación\ 1\ es:$$')

ec1= sp.Function('ec1')
ec2= sp.Function('ec2')

ec1= Eq(a*x + b*y, c)

display(ec1)

ta= r'$$Introduce\ los\ datos\ de\ la\ segunda\ ecuación\ de\ la\ forma\ dx+ey=f:$$'

printmd(ta)

d= int(input("Cuál es el coeficiente de x: "))
d
e=int(input("Cuál es el coficiente de y: "))
e
f=int(input("Cuál es el termino constante: "))

printmd('$$Tu\ ecuación\ 2\ es:$$')

ec2= Eq(d*x + e*y, f)

display(ec2)

printmd ('$$Por\ lo\ tanto\ tu\ sistema\ es:$$')

display(ec1, ec2)

printmd('$$Por\ lo\ tanto\ el\ arreglo\ rectangular\ de\ tu\ sistema\ es:$$')

row1=[a,b,c]
row2=[d,e,f]

system = Matrix ((row1,row2))

display (system)

#Gráficamos el sistema

printmd ('$$La\ gráfica\ del\ sistema\ es:$$')

def y1(x):
    return (c - a*x)/b
# Función lineal.

def y2(x):
    return  (f - d*x)/f
# Valores del eje X que toma el gráfico.
x = range(-10, 15)
# Graficar ambas funciones.
plt.plot(x, [y1(i) for i in x])
plt.plot(x, [y2(i) for i in x])
# Establecer el color de los ejes.
plt.axhline(0, color="black")
plt.axvline(0, color="black")
# Limitar los valores de los ejes.
plt.xlim(-15, 15)
plt.ylim(-15, 15)
# Guardar gráfico como imágen PNG.
plt.savefig("output.png")
# Mostrarlo.
plt.show()
```

$Tomando$ $en$ $cuenta$ $el$ $sistema$ $que$ $introdujiste$ $anteriormente:$

$-Intercambia$ $un$ $coeficiente$ $a$ $la$ $vez.$

$-Observa$ $cómo$ $cambia$ $el$ $sistema$ $respecto$ $al$ $original.$

```python

to= r'$$Introduce\ los\ coeficientes\ de\ tu\ primera\ ecuación\ lineal,\ de\ la\ forma\ ax+by=c$$'

printmd(to)

x,y = sp.symbols('x,y')

#Para que Python interprete la entrada como un número entero, se debe utilizar la función int() para decimal se usa float()

g= int(input("Cuál es el coeficiente de x: "))
g
h=int(input("Cuál es el coeficiente de y: "))
h
r=int(input("Cuál es el término constante: "))

#
printmd('$$Tu\ ecuación\ 1\ es:$$')

eq1 = sp.Function('eq1')

eq1 = Eq(g*x+h*y, r)

display(eq1)

ta= r'$$Introduce\ los\ datos\ de\ la\ segunda\ ecuación\ de\ la\ forma\ dx+ey=f:$$'

printmd(ta)

j= int(input("Cuál es el coeficiente de x: "))
j
k=int(input("Cuál es el coficiente de y: "))
k
l=int(input("Cuál es el termino constante: "))

printmd('$$Tu\ ecuación\ 2\ es:$$')

eq2 = sp.Function('eq2')
eq2 =Eq(j*x+k*y, l)

display(eq2)

printmd ('$$Por\ lo\ tanto\ tu\ sistema\ es:$$')

display(eq1, eq2)

printmd('$$Por\ lo\ tanto\ el\ arreglo\ rectangular\ de\ tu\ sistema\ es:$$')

row3=[g,h,r]
row4=[j,k,l]

system2 = Matrix ((row3,row4))

display (system2)

#Gráficamos el sistema

printmd ('$$La\ gráfica\ del\ sistema\ es:$$')

def y3(x):
    return (r - g*x)/h
# Función lineal.

def y4(x):
    return  (l - j*x)/k
# Valores del eje X que toma el gráfico.
x = range(-10, 15)
# Graficar ambas funciones.
plt.plot(x, [y3(i) for i in x])
plt.plot(x, [y4(i) for i in x])
# Establecer el color de los ejes.
plt.axhline(0, color="black")
plt.axvline(0, color="black")
# Limitar los valores de los ejes.
plt.xlim(-15, 15)
plt.ylim(-15, 15)
# Guardar gráfico como imágen PNG.
plt.savefig("output.png")
# Mostrarlo.
plt.show()
```

$Tomando$ $en$ $cuenta$ $el$ $sistema$ $que$ $introduciste$ $al$ $inicio:$

$-Multiplica$ $una$ $Ecuación$ $por$ $un$ $número$

$-Observa$ $cómo$ $cambia$ $el$ $sistema$ $respecto$ $al$ $original.$

```python
m=int(input('Qué ecuación quieres multiplicar por un número (introduce 1 o 2 según la ecuación que desees multiplicar): '))
m
if m == 1:
    n=int(input('Por qué número deseeas multiplicar: '))
    n

    a1= int(a*n)
    a1
    b1=int(b*n)
    b1
    c1= int(c*n)

    printmd ('$$Tu\ nuevo\ sistema\ es:$$ ')

    eq3 = sp.Function('eq3')
    eq3 = ec1*n

    display(eq3)

```

Sistemas de Ecuaciones cambio de coeficientes .ipynb


```python
import numpy as np
import math
import matplotlib.pyplot as plt
import random
import sympy as sy
import sympy.physics.units as units
```

```python
k = random.randint(1,9)
end = (2*math.pi*k)/(1.5)
x = np.arange(0, end, end/1000)
n = x.shape[0]

print(n)

X = sy.Symbol('X')

f = X + 2*sy.sin(1.5*X)

y = np.zeros(n)

for i in range(n):
  y[i] = f.subs(X, x[i])

x_modified = np.c_[np.ones((len(x),1)), x]
true_theta = np.matmul(np.linalg.pinv(x_modified), y)
print(true_theta)
```

    1000
    [0.31795423 0.97467268]

```python
def train_data(x_train, y_train):
  theta = np.matmul(np.linalg.pinv(x_train), y_train)
  theta.reshape(x_train.shape[1])
  #print(theta.shape)
  return theta
```

```python
# for simplehypo
x_train = np.c_[np.ones((len(x),1)), x]

# for complex hypo
#x_train = np.c_[np.ones((len(x),1)), x, pow(x,2), pow(x,3), pow(x,4)]#, pow(x,5), pow(x,6), pow(x,7)]

m = x_train.shape[1]
print(m)
theta = np.zeros((100,m))

for i in range(100):

  y = np.zeros(len(x))
  for j in range(len(x)):
    y[j] = f.subs(X, x[j]) + np.random.normal(0, np.sqrt(2))

  theta[i, :] = train_data(x_train, y)

bias = true_theta - np.mean(theta, axis = 0)
variance = np.mean(np.square(theta - np.mean(theta)), axis = 0)
mse = np.mean(np.square(true_theta - theta), axis = 0)
cal_mse = np.square(bias) + variance

#print(true_theta - theta)
#print(theta)

print("bias: " + str(bias))
print("variance: " + str(variance))
print("Empirical MSE: " + str(mse))
print("Calculated MSE: " + str(cal_mse))
print("Difference b/w MSE: " + str(cal_mse-mse))
```

    2
    bias: [ 0.00320818 -0.00012217]
    variance: [0.11628647 0.10895396]
    Empirical MSE: [7.38065858e-03 3.78741600e-05]
    Calculated MSE: [0.11629676 0.10895398]
    Difference b/w MSE: [0.1089161 0.1089161]

```python
x_modified = np.c_[np.ones((len(x),1)), x, pow(x,2), pow(x,3), pow(x,4), pow(x,5), pow(x,6), pow(x,7)]
true_theta = np.matmul(np.linalg.pinv(x_modified), y)
print(true_theta)

# for complex hypothesis consider following x
x_train = np.c_[np.ones((len(x),1)), x, pow(x,2), pow(x,3), pow(x,4), pow(x,5), pow(x,6), pow(x,7)]

m = x_train.shape[1]
print(m)
theta = np.zeros((100,m))

for i in range(100):

  y = np.zeros(len(x))
  for j in range(len(x)):
    y[j] = f.subs(X, x[j]) + np.random.normal(0, np.sqrt(2))

  theta[i, :] = train_data(x_train, y)

bias = true_theta - np.mean(theta, axis = 0)
variance = np.mean(np.square(theta - np.mean(theta)), axis = 0)
mse = np.mean(np.square(true_theta - theta), axis = 0)
cal_mse = np.square(bias) + variance

#print(true_theta - theta)
#print(theta)

print("bias: " + str(bias))
print("variance: " + str(variance))
print("Empirical MSE: " + str(mse))
print("Calculated MSE: " + str(cal_mse))
print("Difference b/w MSE: " + str(cal_mse-mse))

```

    [ 2.02431694e+00 -8.79001214e-01  5.55916372e-01 -7.66133162e-02
      5.64622204e-03 -2.33729755e-04  5.28104876e-06 -5.23776946e-08]
    8
    bias: [ 6.81695669e-02 -2.86205100e-01  1.14067585e-01 -1.69830302e-02
      1.11701687e-03 -2.91325148e-05  1.78298994e-08  7.34981128e-09]
    variance: [3.13513635 0.92067975 0.10931252 0.08009378 0.04591451 0.04793571
     0.04784377 0.0478461 ]
    Empirical MSE: [1.21190201e-01 3.44006390e-01 7.25452633e-02 2.89361308e-03
     3.05861294e-05 9.36047331e-08 7.48036034e-11 9.58488507e-15]
    Calculated MSE: [3.13978344 1.00259311 0.12232393 0.08038221 0.04591576 0.04793571
     0.04784377 0.0478461 ]
    Difference b/w MSE: [3.01859324 0.65858672 0.04977867 0.07748859 0.04588517 0.04793562
     0.04784377 0.0478461 ]


# Computing Pi with Colliding Blocks

This notebook attempts to use colliding blocks to calculate the digits of pi, based off a really cool video by 3Blue1Brown

```python
%%html

```

and the paper [Playing Pool with $\pi$](https://www.maths.tcd.ie/~lebed/Galperin.%20Playing%20pool%20with%20pi.pdf) by G. Galperin,

in which a simple problem is described: a block of mass $M$ moves at a constant speed $V$ on a frictionless surface towards a vertical wall. Closer to the wall there is another block of mass $m$ at rest. After the collision, the small block will hit the wall and be reflected back so that the blocks collide again.

If the ratio between the block masses is a power of 100, and the collisions are perfectly elastic, then the total number of collisions $N_\text{collisions}$ (i.e. collisions + reflections) is equal to certain number of digits of $\pi$. Basically, if the masses satisfy the relation $\frac{M}{m} = 100^{d-1}$, then the number of clacks is equal to the first $d$ digits of $\pi$.

## 1. Below is the javascript i attempted to use to get a nice display for the computation simulation

Only problem is that I wanted to try and use a physics engine for the collision calculations. The first one I used is matter.js, but I learned that they don't have continuous collision detection, so when setting the mass of the larger block > 1000kg, the inner mass goes so fast that the engine no longer detects its collisions, and the block just dissapears... I tried again with planck.js, which is a port of box2d and features ccd, however I get a similar problem, where the inner block goes so fast and begins to clip into the larger block at around 1000kg, throwing off the computation.

```python
%%html

```

## 2. Equations and definitions, according to the paper

### Definitions/Conditions
- when the two blocks hit each other we count it as a ***collision***
- when the inner block hits the wall we count it as a ***reflection***
- both collisions and reflections are elastic, so energy and momentum are conserved in these processes

We are interested in the total number of "clacks" given by

\begin{equation}
N_\text{clacks} = N_\text{collisions} + N_\text{reflections} \tag*{(1)}
\end{equation}

**Pre-condition:**
- We initialize the simulation with the large block (of mass $M$) moving at some constant speed $V_0$ before colliding with the small block (of mass $m$) initially at rest.

**Post-condition:**
- The final state of the simulation will have the two blocks moving independently away from the wall, with the large block moving faster than the small block, guaranteeing no further collisions.

### Equations
**Reflections:** before hitting the wall, the small block moves with positive velocity, which is fully reversed after the collision. This can be proven in general by writting the conservation of total energy and momentum:

\begin{eqnarray}
\frac12 mv^2_\text{before} &=& \frac12 mv^2_\text{after}, \tag*{(2)}\\
mv_\text{before} &=& mv_\text{after}, \tag*{(3)}\\
\end{eqnarray}

and velocity is reversed, so
\begin{equation}
v_\text{before} = -v_\text{after} \tag*{(4)}
\end{equation}

**Collisions:** when the blocks collide with one another, their collision is perfectly elastic, so both momentum and energy is conserved, leading to the following:

\begin{align*}
\text{conservation of energy  :} \qquad & \frac12 MV^2_k + \frac12 mv^2_k  = \frac12 MV^2_{k+1} + \frac12 mv^2_{k+1}, \tag*{(5)}\\
\text{conservation of momentum  :} \qquad & MV_k - mv_k = MV_{k+1} + mv_{k+1}. \tag*{(6)}\\
\end{align*}

Doing some work, getting lazy, and using wolfram alpha magic to solve the rest for $V_{k+1}$ and $v_{k+1}$, we get
\begin{align}
v_{k+1} &= \frac{m-M}{m+M}v_k + \frac{2M}{m+M}V_k \tag*{(7)}, \\
V_{k+1} &= \frac{2m}{m+M}v_k + \frac{M-m}{m+M}V_k \tag*{(8)}
\end{align}

which leads to a pretty nice computation for our simulation

## 3. Calculations

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# if collision is between two blocks...
def collision(m, M, vec):
    msum = (m + M)
    v_1 = (m - M)/float(msum)
    v_2 = 2*M/float(msum)
    V_1 = 2*m/float(msum)
    V_2 = (M - m)/float(msum)

    v = v_1*vec[0] + v_2*vec[1] # (eq #7)
    V = V_1*vec[0] + V_2*vec[1] # (eq #8)
    return v,V
```

```python
# if collision is a reflection...
def reflection(vec):
    v = (-1)*vec[0] # reverse velocity of small block (eq #4)
    V = vec[1]
    return [v, V]
```

```python
# checks whether large block is moving away faster than smaller block, which is our end condition
def checkspeeds(speed1, speed2):
    if (speed1 <= 0 and speed2 <=0):
        if(abs(speed2) >= abs(speed1)):
            return False
    return True
```

```python
def run_sim(num_digits):
    M = 100**(num_digits-1)
    currentVector = [0,1]
    ctype = 'C' # the large block will always initially collide with the small block
    clackCount = 0
    v, V = [], []
    while(checkspeeds(currentVector[0], currentVector[1])):
        if (ctype == 'C'):
            currentVector = collision(1, M, currentVector)
            v.append(currentVector[0])
            V.append(currentVector[1])
            clackCount += 1 # increment counter on collision
            ctype = 'R'
        elif (ctype == 'R'):
            currentVector = reflection(currentVector)
            v.append(currentVector[0])
            V.append(currentVector[1])
            clackCount += 1 # increment counter on collision
            ctype = 'C'

    return clackCount, v, V
```

```python
print(run_sim(6)[0])
```

    314159

```python
print(np.pi)
```

    3.141592653589793

it works!
### now to plot the velocities

```python
def plot_velocities(V, v, num_digits, clack_count, linemarkers=True):
    M = 100**(num_digits-1)
    x = np.arange(0, len(V), 1)
    fig, ax1 = plt.subplots()
    ax1.set_title('Velocity of each block ($m=1, M={}$)'.format(M), fontsize=16)

    colorV = 'g'
    colorv = 'b'

    ax2 = ax1.twinx()
    if linemarkers:
        ax1.plot(x, V, 'o-', color=colorV)
        ax2.plot(x, v, 'o-', color=colorv)
    else:
        ax1.plot(x, V, '-', color=colorV)
        ax2.plot(x, v, '-', color=colorv)
    ax1.grid(False)
    ax2.grid(False)
    ax1.set_xlabel(r'$k$ (number of collisions)', fontsize=16)
    ax1.set_ylabel('velocity large block', fontsize=16, color=colorV)
    ax2.set_ylabel('velocity small block', fontsize=16, color=colorv)
    Vmax = 1.5
    vmax = max(v)*1.1
    xmax = max(x)*1.1
    ax1.plot([0, xmax], [0, 0], 'k-', lw=1)  # y=0 line
    ax1.plot([0, 0], [-2*Vmax, 2*Vmax], 'k-', lw=1)  # x=0 line
    ax1.text(xmax*7/9, 1.3, 'clacks: {}'.format(clack_count), fontsize=16)
    ax1.xaxis.set_ticks(np.arange(0, len(V), np.max([1, int(1/7*len(V))])))
    ax1.set_xlim(-xmax/50, xmax)
    ax1.set_ylim(-Vmax, Vmax)
    ax2.set_ylim(-vmax, vmax);
```

```python
clacks, v, V = run_sim(2)
v_1 = np.asarray(v)
V_1 = np.asarray(V)
```

```python
plot_velocities(V_1, v_1,
                num_digits=2,
                clack_count=clacks,
                linemarkers=True)
```

So you can sort of see the makings of a velocity phase diagram, and I probably could have made one if I figured out a way to stretch the axes enough.

## 4. Conclusion

Overall, a pretty cool problem. I learned a lot about elastic collisions (I also did some research on relativistic elastic collisions and learned that the equations I used break down when the speed of the colliding bodies is close to the speed of light), physics engines and their pitfalls, the fact that you can model interesting physics phenomenon with geometry, and also the sheer amount of stuff I can cram into a jupyter notebook.