# Time Evolution: Split Operator Method

### Category: Prerequisites

### Physics Prerequisites: Quantum Mechanics

### Quantum Time Evolution Series
* [Quantum Harmonic Osscillator](../Prerequisites/QHO.ipynb)
* [Quantum Time Evolution](../Prerequisites/Time-Evolution.ipynb)
* [Quantum Ground States: Imaginary Time](../Prerequisites/Imaginary-Time-Evolution.ipynb)

When cleaning my apartment, sometimes I just grab the nearest dirty thing to me and try to do something to it.  But that is not the most efficient way to get things done.  If I'm planning, I'll first dedicate my attention to one problem, like putting clothing away, then rotate my attention to something else, like dirty dishes.  I can keep focused on just one task and do it well.  Each problem I solve optimally in shorter intervals instead of tackling everything at once.

That same principle applies to solving partial differential equations.  [1] called this principle one of the big ideas of numerical computation.  In numerics, we call it Strang splitting.

We will be applying Strang splitting to solve the Schrondinger equation, but people use the same idea to a variety of problems, like ones with different timescales, length scales, or physical processes.  We will be using it to seperate out terms diagonal in position space from terms diagonal in momentum space.

We can describe a class of general partial differential equations by
\begin{equation}
\frac{\partial}{\partial t}y = L_1(y,t)+L_2(y,t)
\end{equation}

Over a small time step, the following approximation holds
\begin{equation}
y(\delta t)= e^{L_1(y,0) \delta t+L_2(y,0) \delta t}y(0)
\end{equation}

For Strang splitting, instead of applying both operators together, we break them up into two.  I'll discuss non-commutativity later.
\begin{equation}
y(\delta t)= e^{L_1 (y,0) \delta t} e^{L_2(y,0) \delta t} y(0) = U_1 U_2 y(0)
\end{equation}
$U_1$ and $U_2$ are evolution operators. We can define
\begin{equation}
\tilde{y(0)}= U_2 y(0)
\end{equation}
so that
\begin{equation}
y (\delta t) = U_1 \tilde{y}(0)
\end{equation}

### Applying to Quantum Mechanics
Now let's take a look at the Schrodinger Equation:
\begin{equation}
i \hbar \frac{\partial}{\partial t} | \Psi \rangle = \mathcal{H} | \Psi \rangle
=
\left[ \frac{\hat{p}^2}{2m} + V(x) \right] | \Psi \rangle
=
\left[ \mathcal{H}_p + \mathcal{H}_x \right] | \Psi \rangle
\end{equation}
The Hamiltonian gets seperated into position terms and momentum terms.  For ease, let's define our unitary evolution operators,
\begin{equation}
U_p(\delta t)=e^{-\frac{i}{\hbar}\mathcal{H}_p \delta t}
\;\;\;\;\;
U_x (\delta t)= e^{-\frac{i}{\hbar}\mathcal{H}_x \delta t}
\end{equation}

I mentioned earlier that I would discuss non-commutativity.  We need to do that now.  We can't simply separate the evolution operator for the full Hamiltonian into two parts, because we would introduce terms proportional to the commutator.
\begin{equation}
e^{A+B}=e^{A}e^{B}e^{[A,B]}
\end{equation}
$e^{A+B}$ expanded has terms that look like $AB$ and $BA$, whereas $e^{A}e^{B}$ only has terms that look like $AB$.  We lose the symmetry of the expression.  We can gain back an order of accuracy by symmetrizing our formula, calculating a time step by
\begin{equation}
|\Psi (\delta t) \rangle =
U_x (\delta t/2) U_p (\delta t) U_x (\delta t/2)
\end{equation}
But the next step will the start with $U_x (\delta t/2)$ !

\begin{equation}
|\Psi (2 \delta t) \rangle = \left(
U_x (\frac{\delta t}{2}) U_p (\delta t) U_x (\frac{\delta t}{2})\right)\left( U_x (\frac{\delta t}{2}) U_p (\delta t) U_x (\frac{\delta t}{2}) \right)
\end{equation}
\begin{equation}
= U_x (\frac{\delta t}{2}) U_p (\delta t) U_x (\delta t)  U_p (\delta t) U_x (\frac{\delta t}{2})
\end{equation}

All we need to do to add an order of accuracy is start the simulation with $U_x(\delta t/2)$ and end it with $U_x(\delta t/2)$, leaving everything else the same.  Pretty remarkable you can get that much of an improvement for that little work.  Once we apply this to a bunch of time steps, we get
\begin{equation}
U_x (\frac{\delta t}{2}) U_p (\delta t) \left( \prod_{n-1} U_x(\delta t) U_p (\delta t) \right) U_x (\frac{\delta t}{2}).
\end{equation}
We have to apply a few operators before starting the loop. Between the loop and a measurement, we have to apply an additional operator.

In the spatial domain, the momentum operator involves derivatives and is rather icky.  But in the momentum domain, we only have to multiply by $k^2/2m$.  Thanks to some nicely optimized libraries, we can just transform into the momentum domain with `fft`, solve the momentum problem there, and transform back with `ifft`.

## Rabi Oscillations

To demonstrate time evolution in a simple system with interesting physics, I chose to apply the split operator to Rabi Oscillations between two harmonic oscillators.

To get an idea of what will happen, we will use a qualitative model of two states weakly coupled to each other by a parameter $\epsilon$.  If we have the two minima sufficifiently seperated from each other, tunneling will happen slowly and will not significantly affect the shape of the eigenfunctions and their energies $E_0$.  Instead of of solving for the shape of the wavefunction, we solve a two-state Hamiltonian that looks like this,

\begin{equation}i \hbar \frac{\partial}{\partial t}
\begin{bmatrix}
| \phi_r \rangle  \\
| \phi_l \rangle
\end{bmatrix}
= \begin{bmatrix}
E_0  & \epsilon \\
ϵ & E_0 \\
\end{bmatrix}
\begin{bmatrix}
| \phi_r \rangle \\
| \phi_l \rangle
\end{bmatrix}
\end{equation}
The eigenvalues and corresponding eigenvectors of the matrix are,
\begin{equation}
\lambda_1 = E_0 + \epsilon \;\;\;\;\;\;
\lambda_2 = E_0 - \epsilon
\end{equation}

\begin{equation}
\vec{v}_{1} = \begin{bmatrix}
| \phi_r \rangle \\
| \phi_l \rangle
\end{bmatrix}
\;\;\;\;\;
\vec{v}_2 = \begin{bmatrix}
| \phi_r \rangle \\
- |\phi_l \rangle
\end{bmatrix}
\end{equation}

If a wavefunction starts purely in the right state, we want to choose a combination of our eigenvectors that sets the left state to zero at $t=0$.  The resulting wavefunction will evolve as,
\begin{equation}
| \Psi (t) \rangle = \frac{1}{\sqrt{2}}e^{-\frac{i}{\hbar} E_0 t } \left(
e^{-\frac{i}{\hbar} \epsilon t} \begin{bmatrix}
| \phi_r \rangle \\
| \phi_l \rangle
\end{bmatrix}
+ e^{\frac{i}{\hbar} \epsilon t} \begin{bmatrix}
| \phi_r \rangle \\
-| \phi_l \rangle
\end{bmatrix}
\right)
\end{equation}

\begin{equation}
= \sqrt{2} e^{-\frac{i}{\hbar} E_0 t } \begin{bmatrix}
\cos \frac{\epsilon t}{\hbar} | \phi_r \rangle \\
i \sin \frac{\epsilon t}{\hbar} | \phi_l \rangle
\end{bmatrix}
\end{equation}

Thus this simple phenomological model shows us how we can expect the wavefunction to move back and forth between the two wells in a cyclical manner, with a period proportional to the rate of tunneling.

```julia
using Plots
using FFTW
pyplot()
```

    Plots.PyPlotBackend()

## Input Parameters

```julia
# Set Time Parameters
t0=0
tf=40000
dt=.1

# Set Space Grid Parameters
dx=.1
xmax=8
# xmin will be -xmax.  Making the situation symmetric

# How far seperated are the potential minima
seperation=6;
# minima at seperation/2 and -seperation/2

# How often we measure occupation and view the state
nmeasure=1000;
```

## Automatic Evaluation Parameters

Given the parameters above, we can calculate the following variables that we will use in the code.

Note: `k` Gave me a bit of a headache. The algorithm depends quite a bit on the conventions `fft` decides to use ;(
Currently, I'm using odd `N`.  You'll have to change the formula if you use even `N`.

```julia
t=collect(t0:dt:tf)
x=collect(-xmax:dx:xmax)

nt=length(t)
N=length(x)

k = [ collect(0:((N-1)/2)) ; collect(-(N-1)/2:-1) ] *2*π/(N*dx);

occupation=zeros(Complex{Float64},floor(Int,nt/nmeasure),2);
```

## The Potentials and Evolution Operators

```julia
Vx=.5*(abs.(x).-seperation/2).^2;
Vk=k.^2/2

Uxh=exp.(-im*Vx*dt/2);
Ux=exp.(-im*Vx*dt);
Uf=exp.(-im*Vk*dt);
"potentials and evolvers defined"
```

    "potentials and evolvers defined"

```julia
plot(x, Vx)

plot!(xlabel="x", ylabel="V(x)",
plot_title="Double Well Potential")
```

## The Unperturbed Wavefunctions

The ground state for a harmonic oscillator is a Gaussian
\begin{equation}
\langle x | \phi \rangle= \phi (x) = \frac{1}{\pi^{1/4}} e^{-\frac{x^2}{2}}
\end{equation}
We assume $\omega = \hbar = m = 1$ for sake of convenience.

```julia
ϕ(x)=π^(-.25)*exp(-x.^2/2)
```

    ϕ (generic function with 1 method)

```julia
ϕl=ϕ.(x.+seperation/2);
ϕr=ϕ.(x.-seperation/2);
Ψ0=convert(Array{Complex{Float64},1},ϕl);
```

```julia
plot(x,abs2.(ϕl),label="Left")
plot!(x,abs2.(ϕr),label="Right")

plot!(xlabel="x", ylabel="⟨ ϕ| ϕ ⟩",
    title="Left and Right Wavefunctions")
```

## FFT's

This algorithm runs a large number of Fast Fourier Transforms and Inverse Fast Fourier Transforms.  To speed the process, we can tell the computer to spend some time, in the beginning, allocating the right amount of space and optimizing the routine for the particular size and type of array we are passing it.

The next cell does this, by using `plan_fft` and `plan_ifft` to generate objects that can act on our arrays as operators.

```julia
ft=plan_fft(Ψ0);
Ψf=ft*Ψ0;
ift=plan_ifft(Ψf);
```

## Occupancy of each state

To measure the occupancy of the total wavefunction in either the left or right well groundstate, I compute the value
\begin{equation}
c_r=\langle \Psi | \phi_r \rangle  = \int \Psi^* (x) \phi_r(x) dx \;\;\;\;\;\;\; p_r=c_r^2
\end{equation}
\begin{equation}
c_l = \langle \Psi | \phi_l \rangle = \int \Psi^* (x) \phi_l (x) dx\;\;\;\;\;\;\; p_l = c_l^2
\end{equation}
The probability of being in the state is the coefficient squared.

Though in theory these values will always be real, numerical errors introduce some errors, and Julia will assume that the answer is complex.  Therefore, we need to apply `abs` to make the numbers `Float64` instead of `Complex{Float64}`.

```julia
nmeas=1000
c=zeros(Float64,floor(Int,nt/nmeas),2);
```

```julia
# Uncomment the # lines to generate a gif.  Note: It takes a long time

Ψ=Ψ0;
jj=1

# The operators we have to start off with
Ψ=Ψ.*Uxh

Psif=ft*Ψ
Psif=Psif.*Uf
Ψ=ift*Psif

#@gif for ii in 1:nt
for ii in 1:nt

    Ψ=Ψ.*Ux

    Psif=ft*Ψ
    Psif=Psif.*Uf
    Ψ=ift*Psif

    if ii%nmeas == 0

        # Every time we measure, we have to finish with a U_x half time step
        Ψt=Ψ.*Uxh

        c[jj,1]=abs(sum( conj(Ψt).*ϕl )) *dx
        c[jj,2]=abs(sum( conj(Ψt).*ϕr )) *dx

        jj+=1
    end

    #plot(x[21:141],Vx[21:141]/6, label="Vx scaled")
    #plot!(x,abs(conj(Ψt).*Ψt), label="Wavefunction")
    #plot!(xlabel="x", ylabel="Ψ",
    #    plot_title="Wavefunction evolution")

end
#end every nmeas

Ψ=Ψ.*Uxh;
```

```julia
plot(c[:,1].^2,label="Left Well")
plot!(c[:,2].^2, label="Right well")
plot!(xlabel="steps",ylabel="⟨ Ψ(t) | ϕ ⟩",
    title="Rabi Oscillations for a Double Harmonic Oscillator")
```

PyPlot generated png's strung together:

Plots generated gif:

The faster one can perform Fourier Transforms, the faster one can perform this algorithm.  Therefore, scientists, such as [2], will use multiple cores or GPU's.

In addition to real time evolution, algorithms like this can determine the ground state of an arbitrary system by imaginary time evolution.  Soon, I will take the work covered here and look at this aspect of the algorithm.

[1] Glowinski, Roland, Stanley J. Osher, and Wotao Yin, eds. Splitting Methods in Communication, Imaging, Science, and Engineering. Springer, 2017.

[2] Heiko Bauke and Christoph H. Keitel. Accelerating the Fourier split operator method via graphics processing unit. Computer Physics Communications, 182(12):2454–2463 (2011)

```julia

```


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Definition

The [Laplace transform](https://en.wikipedia.org/wiki/Laplace_transform) is an integral transform which represents a signal $x(t)$ in the spectral domain. It features the complex exponential function $e^{s t}$ with $s \in \mathbb{C}$ as kernel. Note, the same functions constitute eigenfunctions of linear time-invariant (LTI) systems. The Laplace transform is named after its discoverer [Pierre-Simon Laplace](https://en.wikipedia.org/wiki/Pierre-Simon_Laplace).

### Two-Sided Laplace Transform

The two-sided (or bilateral) Laplace transform is defined as

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ denotes the Laplace transform of $x(t)$. A complex signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$ is represented by its complex valued Laplace transform $X(s) \in \mathbb{C}$ with complex independent variable $s \in \mathbb{C}$. The variable $s$ can be interpreted as [complex frequency](../continuous_signals/standard_signals.ipynb#Complex-Exponential-Signal) $s = \sigma + j \omega$ with $\sigma, \omega \in \mathbb{R}$.

Whether a Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ exists depends on the complex frequency $s$ and the signal $x(t)$ itself. All values $s$ for which the Laplace transform converges form a [*region of convergence*](https://en.wikipedia.org/wiki/Radius_of_convergence) (ROC). The Laplace transforms of two different signals may differ only with respect to their ROCs. Consequently, the ROC needs to be explicitly given for an unique inversion of the Laplace transform.

### One-Sided Laplace Transform

Causal signals play an important role in the theory of signals and system. For a causal signal $x(t) = 0$ for $t <0$, the relation $x(t) = x(t) \cdot \epsilon(t)$ holds. Introducing this into the definition of the two-sided Laplace transformation results in

\begin{equation}
X(s) = \int_{-\infty}^{\infty} x(t) \cdot \epsilon(t) \, e^{- s t} \; dt = \int_{0}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

This motivates the definition of the one-sided Laplace transform

\begin{equation}
X(s) = \int_{0}^{\infty} x(t) \, e^{- s t} \; dt
\end{equation}

In the literature both the one- and two-sided Laplace transform are termed as Laplace transform. For causal signals both give the same result.

### Transformation of the Dirac Impulse

The transform $\mathcal{L} \{ \delta(t) \}$ of the [Dirac impulse](../continuous_signals/standard_signals.ipynb#Dirac-Impulse) is derived by introducing $\delta(t)$ into the definition of the two-sided Laplace transform and exploiting the sifting property of the Dirac delta function

\begin{equation}
\mathcal{L} \{ \delta(t) \} = \int_{-\infty}^{\infty} \delta(t) \, e^{- s t} dt = 1
\end{equation}
for $s \in \mathbb{C}$. The ROC covers the entire complex plane.

The transform of the Dirac impulse is equal to one. Hence, all complex frequencies $s$ are present with equal weight. This is an important property in the theory of signals and systems, since the Dirac impulse is used to characterize linear time-invariant (LTI) systems by their [impulse response](../systems_time_domain/impulse_response.ipynb) $h(t) = \mathcal{H} \{ \delta(t) \}$.

**Example**

The Laplace transform of the Dirac impulse can be derived in `SymPy` by direct evaluation of above integral

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t = sym.symbols('t', real=True)
s = sym.symbols('s', complex=True)

X = sym.integrate(sym.DiracDelta(t)*sym.exp(-s*t), (t, -sym.oo, sym.oo))
X
```

**Exercise**

* Derive the two-sided Laplace transform of a shifted Dirac impulse $\delta(t - \tau)$ either by manual evaluation of the integral or by modification of above example.

* Derive the one-sided Laplace transform of a shifted Dirac impulse $\delta(t - \tau)$. Hint: Differentiate between the cases $\tau < 0$ and $\tau > 0$.

### Transformation of the Causal Complex Exponential Signal

The transform $X(s) = \mathcal{L} \{ x(t) \}$ of the causal complex exponential signal

\begin{equation}
x(t) = \epsilon(t) \cdot e^{-s_0 t}
\end{equation}

with complex frequency $s_0 \in \mathbb{C}$ is derived by evaluation of the definition of the one-sided Laplace transform

\begin{align}
X(s) &= \int_{0}^{\infty} e^{-s_0 t} \cdot e^{-s t} \; dt = \frac{-e^{-(s+s_0)t}}{s+s_0} \, \bigg\vert_{0}^{\infty} = \frac{-1}{s+s_0} \left[ \lim_{t \to \infty} \left( e^{- (s+s_0) t} \right) -1 \right] \\
&= \frac{1}{s + s_0}
\end{align}

As can be concluded from the limit case above, the Laplace transform converges only for $\Re \{ s + s_0 \} > 0$. Hence, the ROC is given as

\begin{equation}
\Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

Combining above results, the transformation of the causal complex exponential signal reads

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

**Example**

In `SymPy` the [Laplace transform](http://docs.sympy.org/latest/modules/integrals/integrals.html?highlight=laplace#sympy.integrals.transforms.laplace_transform) is defined as one-sided transformation. The function is utilized in the following to calculate the transform of the causal complex exponential signal.

```python
s0 = sym.symbols('s0', complex=True)

X, a, cond = sym.laplace_transform(sym.exp(-s0*t), t, s)
X, a, cond
```

The function returns three results. Its first element is the Laplace transform $X(s)$, the second element $a$ is the half-plane of convergence with $\Re \{ s \} > a$ the third element is an auxiliary convergence condition.

### Transformation of the Heaviside Signal

The Laplace transform of the [Heaviside signal](../continuous_signals/standard_signals.ipynb#Heaviside-Signal) is derived from above transform of the causal complex exponential signal $\epsilon(t) \cdot e^{-s_0 t}$ by setting $s_0 = 0$

\begin{equation}
\mathcal{L} \{ \epsilon(t) \} = \frac{1}{s} \qquad \text{for } \Re \{ s \} > 0
\end{equation}

## Convergence

A sufficient but not necessary condition for the existence of the Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ of a given signal $x(t)$ is

\begin{equation}
\int_{-\infty}^{\infty} | x(t) \cdot e^{- \sigma t} | \; dt < \infty
\end{equation}

where $\sigma = \Re\{ s \}$ denotes the real part of the complex frequency $s = \sigma + j \omega$ of the Laplace transform. Formally the signal $x(t)$ has to be a [locally integrable function](https://en.wikipedia.org/wiki/Locally_integrable_function).

The imaginary part $\omega = \Im \{ s \}$ of the complex frequency $s$ has no effect in terms of convergence. Consequently, the ROC is determined solely by its real part $\sigma = \Re\{ s \}$. For a right-sided signal with $x(t) = 0$ for $t < T_0$, the ROC is given in the form of $\Re \{ s \} > a$ with $a \in \mathbb{R}$. This can be concluded from the decay of the exponential term $e^{- \sigma t}$ for $t \to \infty$, which ensures convergence for a given $a$. However this reasoning holds only for signals with [exponential growth](https://en.wikipedia.org/wiki/Exponential_growth). Please refer to the [Laplace transform of the causal exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal) for an example.

The same reasoning leads to the ROCs for a left-sided and two-sided signal. The resulting ROCs are shown in the following illustration

The gray areas denote the values $s$ for which the Laplace transformation converges. The borders $a$ of these areas (dashed lines) depend on the signal $s(t)$. A more detailed discussion of the ROCs for the Laplace transform can be found in the literature, e.g. [[Girod et al.](index.ipynb#Literature)]

## Representation

The Laplace transform $X(s) = \mathcal{L} \{ x(t) \}$ depends on the complex frequency $s \in \mathbb{C}$ and is in general complex valued $X(s) \in \mathbb{C}$. It can be illustrated by plotting its magnitude $|X(s)|$ and phase $\varphi (s)$ or real $\Re \{ X(s) \}$ and imaginary $\Im \{ X(s) \}$ part in the complex $s$-plane. The resulting three-dimensional plots are often not very illustrative.

However, many Laplace transforms of interest in the theory of signals and systems are [rational functions](https://en.wikipedia.org/wiki/Rational_function) in $s$. The polynomials of the numerator and denominator can be represented by their complex [roots](https://en.wikipedia.org/wiki/Zero_of_a_function#Polynomial_roots) and a constant factor. The roots of the numerator are termed as *zeros* while the roots of the denominator a termed as *poles* of $X(s)$. The polynomial and the zero/pole representation of a rational Laplace transform are equivalent

\begin{equation}
X(s) = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n} = K \cdot \frac{\prod_{\mu=0}^{Q} (s - s_{0 \mu})}{\prod_{\nu=0}^{P} (s - s_{\infty \nu})}
\end{equation}

where $M$ and $N$ denote the order of the numerator/denominator polynomial, $s_{0 \mu}$ and $s_{\infty \nu}$ the $\mu$-th zero/$\nu$-th pole of $X(s)$, and $Q = M-1$ and $P = N-1$ the total number of zeros and poles, respectively. For $M=N$ the factor $K = \frac{\beta_M}{\alpha_N}$. If $M > N$ at least one pole is located at $|s| = \infty$.

It is common to illustrate the poles and zeros in a [pole-zero plot](https://en.wikipedia.org/wiki/Pole%E2%80%93zero_plot). Here the locations of the complex poles and zeros, their degrees and the factor $K$ are plotted in the $s$-plane

The locations of the poles and zeros provide insights into the composition of a signal. For instance, the Laplace transform of the [complex exponential signal](#Transformation-of-the-Causal-Complex-Exponential-Signal) has a pole at $s = - s_0$. A signal which is composed from a superposition of complex exponential signals will have multiple poles whose positions are related to the complex frequencies of the signals it is composed of.

**Exercise**

* Derive $X(s)$ together with its ROC from above pole-zero plot.

**Example**

The Laplace transform of the causal complex exponential function

\begin{equation}
X(s) = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

is illustrated by plotting its magnitude $|X(s)|$ in the $s$-plane $s = \sigma + j \omega$ for $s_0 = 1 + j$

```python
sigma, omega = sym.symbols('sigma omega')

X = 1/(s + s0)
X1 = X.subs({s: sigma+sym.I*omega, s0: 1+sym.I})

sym.plotting.plot3d(abs(X1), (sigma, -2, 2), (omega, -2, 2),
                    xlabel=r'$\Re\{s\}$', ylabel=r'$\Im\{s\}$', title=r'$|X(s)|$');
```

Alternatively, the Laplace transform is illustrated by its pole-zero plot. First the poles

```python
poles = sym.roots(sym.denom(X), s)
poles
```

and zeros are computed

```python
zeros = sym.roots(sym.numer(X), s)
zeros
```

Above dictionaries are composed of the poles and zeros, and their degrees.

In order to illustrate the location of poles and zeros in the $s$-plane, a function is defined for a pole-zero plot using the Python library [`matplotlib`](http://matplotlib.org/)

```python
import matplotlib.pyplot as plt

def pole_zero_plot(poles, zeros, ax=None):

    for pole in poles:
        plt.plot(complex(pole).real, complex(pole).imag, 'rx', markersize=10)
    for zero in zeros:
        plt.plot(complex(zero).real, complex(zero).imag, 'bo', markersize=10)

    if ax is None:
        ax = plt.gca()
    ax.set_aspect(1)

    plt.axis([-2, 2, -2, 2])
    plt.xlabel(r'$\Re \{ s \}$')
    plt.ylabel(r'$\Im \{ s \}$')
    plt.grid()
```

The pole-zero plot is shown for $s_0 = 1 + j$

```python
X2 = X.subs(s0, 1+sym.I)
pole_zero_plot(sym.roots(sym.denom(X2), s), sym.roots(sym.numer(X2), s))
```

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# SEIR model example

\begin{align}
\dot{S} & = - \alpha\beta\frac{SI}{N}\\
\dot{E} & =  \alpha\beta\frac{SI}{N} - E\frac{1}{\tau_{EI}}\\
\dot{I} & =  E\frac{1}{\tau_{EI}} - I\frac{1}{\tau_{IR}}\\
\dot{R} & =  I\frac{1}{\tau_{IR}} \\
\end{align}

# SEIR model example
\begin{align}
\dot{S} & = S_f - \alpha\beta\frac{SI}{N} + \rho_{RS} R + \phi_S\\
\dot{E} & = E_f + \alpha\beta\frac{SI}{N} - E\frac{1}{\tau_{EI}} + \phi_E\\
\dot{I} & = I_f + E\frac{1}{\tau_{EI}} - I\frac{1}{\tau_{IR}} + \phi_I\\
\dot{R} & = R_f + I\frac{1}{\tau_{IR}} - \rho_{IR} R + \phi_R\\
\dot{\Phi} & = \sum_{i} (\phi_{i}) \\
N & = N_0 + \Phi \\
N_0 & = S_0+E_0+I_0+R_0  \\
\end{align}

Where:
* $S:$ Susceptible Individuals
* $E:$ Exposed Individuals
* $I:$ Infectious Individuals
* $R:$ Removed Individuals
* $\alpha:$ Average number of contacts per person per time
* $\beta:$ Probability of disease transmission in a contact between a susceptible and an infectious subject
* $N:$ Total population
* $\tau_{EI}:$ Transition time between exposed and infectious
* $\tau_{IR}:$ Transition time between infectious and recovered
* $\rho_{RS}:$ Immunity loss rate. Inverse of Immunity loss rate medium time ($\frac{1}{t_{RS}}$)
* $\phi_S,\phi_E,\phi_I,\phi_R:$ External flux per compartment
* $\Phi$: Total external population flux

```python
# Util libraries
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
from datetime import datetime

# cv19 libraries
from cv19gm.models.seir import SEIR
import cv19gm.utils.cv19functions as cv19functions
```

```python
# For pop-up plots execute this code (optional)
import platform
OS = platform.system()

if OS == 'Linux':
    %matplotlib tk
    print('Linux')
elif OS == 'Windows':
    %matplotlib qt
    print('Windows')
elif OS == 'Darwin':
    %matplotlib tk
    print('Mac (Funciona?)')
```

    Linux

# Build a basic SEIR model using a configuration file

## Constant parameters

```python
# Input configuration file
config = 'cfg/SEIR.toml'
```

```python
# Build simulation object
model = SEIR(config = config,beta=0.3)
```

```python
%%capture
# Simulate (solve ODE)
model.solve()
```

```python
# Show results
model.results
```

      t
      dates
      S
      E
      E_d
      I
      I_d
      R
      R_d
      Flux
      E_ac
      I_ac
      R_ac
      I_det
      I_d_det
      I_ac_det
      prevalence_total
      prevalence_susc
      prevalence_det

      0
      0
      None
      986000
      3000
      1500
      6000
      3000
      5000
      0
      0.0
      1500
      18000
      0
      6000
      3000
      18000
      0.018000
      0.018000
      0.018000

      1
      1
      None
      984216
      4073
      1681
      6106
      1563
      5603
      382
      0.0
      3181
      19563
      382
      6106
      1563
      19563
      0.019564
      0.019564
      0.019564

      2
      2
      None
      982376
      5003
      1785
      6392
      1159
      6227
      536
      0.0
      4966
      20723
      918
      6392
      1159
      20723
      0.020724
      0.020724
      0.020724

      3
      3
      None
      980434
      5858
      1890
      6820
      1122
      6886
      616
      0.0
      6857
      21846
      1534
      6820
      1122
      21846
      0.021846
      0.021846
      0.021846

      4
      4
      None
      978352
      6685
      2019
      7366
      1214
      7595
      677
      0.0
      8876
      23060
      2212
      7366
      1214
      23060
      0.023061
      0.023061
      0.023061

      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...
      ...

      495
      495
      None
      59570
      0
      0
      0
      0
      940429
      0
      0.0
      928717
      949156
      935385
      0
      0
      949156
      0.949156
      0.949156
      0.949156

      496
      496
      None
      59570
      0
      0
      0
      0
      940429
      0
      0.0
      928717
      949156
      935385
      0
      0
      949156
      0.949156
      0.949156
      0.949156

      497
      497
      None
      59570
      0
      0
      0
      0
      940429
      0
      0.0
      928717
      949156
      935385
      0
      0
      949156
      0.949156
      0.949156
      0.949156

      498
      498
      None
      59570
      0
      0
      0
      0
      940429
      0
      0.0
      928717
      949156
      935385
      0
      0
      949156
      0.949156
      0.949156
      0.949156

      499
      499
      None
      59570
      0
      0
      0
      0
      940429
      0
      0.0
      928717
      949156
      935385
      0
      0
      949156
      0.949156
      0.949156
      0.949156

500 rows × 19 columns

```python
plt.plot(model.t,model.S+model.E+model.I+model.R)
```

    [<matplotlib.lines.Line2D at 0x7fdf4ccb6b50>]

```python
# Plot matplotlib
plt.plot(model.t,model.S,label='S')
plt.plot(model.t,model.E,label='E')
plt.plot(model.t,model.I,label='I')
plt.plot(model.t,model.R,label='R')
plt.xlim(0,300)
plt.legend(loc=0)
plt.show()
```

```python
# Plot with pandas
ax = model.results.plot(x='t',y=['S','E','I','R'])
ax.set_xlim(0,300)
ax.set_title('SEIR simulation plot')
```

## Variable parameters
Create a function with time dependent parameters. Check covid19functions notebook for more information on this.

```python
config2 = 'cfg/SEIR_v2.toml'
```

```python
model2 = SEIR(config = config2)
```

    input dict
    input dict

```python
%%capture
model2.solve()
```

```python
plt.plot(model2.t,model2.S,label='S')
plt.plot(model2.t,model2.E,label='E')
plt.plot(model2.t,model2.I,label='I')
plt.plot(model2.t,model2.R,label='R')
plt.xlim(0,300)
plt.legend(loc=0)
plt.show()
```

# SEIR desde cmd

```python
beta = 0.2
population = 10000
I = 100
```

```python
model = SEIR(config = config)
```

```python
model.solve()
```

### Variable parameters

```python
t = np.linspace(0,6,1000)
```

```python
beta = lambda t: np.sin(t)
```

```python
plt.plot(t,beta(t))
```

    [<matplotlib.lines.Line2D at 0x7f667035d730>]

```python
model = SEIR(config = config,beta=beta)
```

```python
model.solve()
```

```python
plt.plot(model.t,model.I)
plt.title('Active Infected')
plt.xlim(0,50)
plt.show()
```

```python

```

```python
beta2 = cv19functions.events(days=[[0,10],[10,60],[60,100]],values=[0.12,0.2,0.3],default = 0.1)
```

```python
t = np.linspace(0,200,10000)
```

```python
plt.plot(t,beta2(t))
```

    [<matplotlib.lines.Line2D at 0x7f667029c580>]

```python
model2 = SEIR(config = config,beta=beta2)
```

```python
model2.solve()
```

```python
plt.plot(model2.t,model2.I)
```

    [<matplotlib.lines.Line2D at 0x7f667029cd60>]

# Study

```python
# Input configuration file
config = 'cfg/SEIR.toml'
```

```python
# Build simulation object
model1 = SEIR(config = config, I_d_det = 600, population = 1000000)
model2 = SEIR(config = config, I_d_det = 600, population = 10000000)
```

```python
%%capture
# Simulate (solve ODE)
model1.solve()
model2.solve()
```

```python
model1.I_d[0]
```

```python
# Plot matplotlib
#plt.plot(model.t,model1.S,label='S')
#plt.plot(model.t,model1.E,label='E')
plt.plot(model.t,model1.I_d,label='I_1')
plt.plot(model.t,model2.I_d,label='I_2')
plt.xlim(0,300)
plt.yscale('log')
plt.legend(loc=0)
plt.show()
```

```python

```


```
%pylab inline
```

    Populating the interactive namespace from numpy and matplotlib

```
from sympy import *
from pylab import *
```

```
import matplotlib.pyplot as plt
```

```
from sympy.interactive import printing
printing.init_printing(use_latex=True)
from IPython.display import display
a,x,Z = symbols('a,x,Z')
```

## Problem 1

### 1D

The partition function for 1D system is

$Z = \sum_i e^{-\beta E_i}$

The probability on a state $E_i$

$$P(E_i) = \frac{e^{-\beta E_i}}{Z}$$

In our case, there are only two possible energy states, $E_1 = -\mu B$, $E_2 = \mu B$. So the magnetization is

$$ M =  \mu N \left( e^{\beta \mu B} - e^{-\beta \mu B} \right)/Z$$

Write magnetization as a function of $T$ and $T$.

$$M(T,B) = \mu N \frac{ \left( e^{\mu B /(k_B T)} - e^{- \mu B/(k_B T)} \right)  }  {  \left(  e^{\mu B /(k_B T)} + e^{- \mu B/(k_B T)}  \right)  } = \mu N \tanh(\mu B /(k_B T))$$

###### $M$ vs $T$ Plots

To plot $M~T$, we need to make a quantity with dimension of $T$ . $\bar T = \mu B/k_B$ will do the work.

$$\frac{M(T)}{\bar M} = e^{\bar T/T} - e^{-\bar T/T} $$

in which, $\bar M = \mu N$ .

At $T \rightarrow \infty $ limit, we have the magnetization $M$ is 0. This is because high temperature distroys the alignment of dipoles.

```
tanh(0)
```

At $T\rightarrow \infty$, we have $\frac{M}{\bar M}$ 1.

```
tanh(inf)
```

At very low temperature, the dipoles have no random motion so all dipoles aligned together thus $M$ is 1.

Now plot out this result.

```
t = linspace(0, 100, 1000)
mt = tanh(1/t)

fig11 = plt.figure()

axes = fig11.add_axes([0.1, 0.1, 0.8, 0.8]) # left, bottom, width, height (range 0 to 1)

axes.plot(t, mt, 'r')

axes.set_xlabel('T bar')
axes.set_ylabel('M bar')
axes.set_title('M ~ T');
```

This plot agrees with our results of the limits.

###### $M$ vs $B$ Plots

Define a unit magnetic induction $\bar B = k_B T/\mu$. Write the magnetization as a function of $B$.

$$ \frac{M(B)}{\bar M} = \tanh(B/\bar B) $$

At the limit of $B\rightarrow 0$, we have $M\rightarrow 0$, which is because no magnetic field is there to align the dipoles. (No interactions between dipoles.)

At the limit of $B\rightarrow \infty$, we have $M\rightarrow 1$, which is because infinite magnetic field can align all the dipoles to one direction no matter how strong the thermal random motion is.

Plot the result.

```
b = linspace(0, 5, 100)
mb = tanh(b)

fig12 = plt.figure()

axes = fig12.add_axes([0.1, 0.1, 0.8, 0.8]) # left, bottom, width, height (range 0 to 1)

axes.plot(b, mb, 'r')

axes.set_xlabel('B/Bbar')
axes.set_ylabel('M/Mbar')
axes.set_title('M ~ B');
```

###### Heat Capacity

Energy of the system is

$$ E = - \mu B N \tanh(\mu B /(k_B T)) $$

```
t13e = linspace(0, 5, 100)
e1 = -tanh(t13e)

fig13e = plt.figure()

axes = fig13e.add_axes([0.1, 0.1, 0.8, 0.8]) # left, bottom, width, height (range 0 to 1)

axes.plot(t13e, e1, 'r')

axes.set_xlabel('T/Tbar')
axes.set_ylabel('E/Ebar')
axes.set_title('E ~ T');
```

Heat capacity is

$$ C = \frac{\partial}{\partial T} E = \frac{\mu^2 B^2 N}{k_B} \frac{ (1 - \tanh^2(\mu B/(k_B T)) }{T^2} $$

```
t13 = linspace(0, 5, 100)
c1 = tanh(t13)

fig13 = plt.figure()

axes = fig13.add_axes([0.1, 0.1, 0.8, 0.8]) # left, bottom, width, height (range 0 to 1)

axes.plot(t13, c1, 'r')

axes.set_xlabel('T/Tbar')
axes.set_ylabel('C/Cbar')
axes.set_title('C ~ T');
```

### 2D

The energy of a dipole can be written as $E(\theta) = -\mu B \cos\theta$, in which $\theta$ is the angle between the dipole and magnetic field.

Partition function is

$$ Z = \int _ \theta 2 e^{-\beta E(\theta)} \mathrm d \theta  = 2\pi \mathrm {BesselI}(0, \mu B \beta) $$

where $\mathrm{BesselI}$ is the modified Bessel function of the first kind.

Then we can calculate magnetization, which is given by

$$ M = \frac{1}{Z}\int _ \theta 2N\mu \cos\theta e^{-\beta E(\theta)} d \theta  $$

The result given by Mathematica is (so bad that I can't make sympy to deal with such integrations.)

$$ M = \mu N \frac{ \mathrm{BesselI}(1, B \beta \mu)  }{ \mathrm{BesselI}(0, B \beta \mu) }$$

Energy should be

$$ E = - \mu B N \frac{ \mathrm{BesselI}(1, B \beta \mu)  }{ \mathrm{BesselI}(0, B \beta \mu) }$$

Heat capacity is the derivitive of energy over temperature.

$$ C = B^2 \mu^2 N  \frac{-2 \mathrm{BesselI}(0, B \beta \mu)^2 +  \mathrm{BesselI}(0,\mu\beta B)(  \mathrm{BesselI}(0,\mu\beta B) + \mathrm{BesselI}(2,\mu\beta B)   ) }{2 k T^2 \mathrm{BesselI}(0, B \beta \mu)^2} $$

Plot out these results using Mathematica.

### 3D

Energy of a dipole is $E(\theta) = -\mu B \cos\theta$.

Then we can calculate the partition function, which is

$$ Z = \int_\theta \int _ \phi N e^{-\beta E(\theta)}\sin\theta d\theta d \phi = 4\pi N\frac{ \sinh(\mu B \beta)}{\mu B \beta} $$

Magnetization is

$$ M =  -\frac{\mu N}{\mu B \beta} + \mu N \coth(\mu B \beta) = \mu N \left(  - \frac{\bar B}{B} + \coth \left(\frac{B}{\bar B}\right) \right) $$

where $\bar B = \frac{1}{\mu \beta}$. Define $\bar M = \mu N$ .Then we can write down the dimensionless equation, which is

$$ \frac{M}{\bar M} =  - \frac{\bar B}{B} + \coth \left(\frac{B}{\bar B}\right) $$

Total energy is

$$ E = \frac{1}{Z} \int _ \phi \int _ \theta (-\mu B \cos\theta N) e^{-\mu E(\theta)}\sin\theta d \theta d\phi = - N \left( \frac{1}{\beta} - B \mu \coth(\mu \beta B) \right)  $$

Take the derivative with respect to temperature, we have the heat capacity.

$$ C = N \left( 1 - \left(\frac{\bar T}{T} \right)^2 \mathrm{csch}^2 \left( \frac{\bar T}{T} \right) \right) $$

Plot out the results.

## Problem 2

#### First law of thermodynamics

Energy can be transfered from one form to another, or from one system to another, but can never be disapear. The total energy of a closed system is conserved. For any infinitesimal process, the change of internal energy is equivalent to the sum of work done to other systems and heat obsorbed from other systems, which is to say, in the language of mathematiccs,

#### $$ \mathrm d U = \mathrm d Q + \mathrm d W $$

#### Entropy

Basicly, at high temperature and in large systems, entropy is the dominant of all marco properties of a system. Entropy indicates the reversibility of a process, defines Kelvin and restricts us from reaching absolute zero.

The most important property of entropy is that it is a function that only depends on state of the system not the path. This property can lead us to many interesting results.

#### Free energy

As I have learned in undergrad stat mech, free energy is a quantity defined to make our life easier when dealing with systems which has fixed temperature and fixed volume. In that case, free energy can never grow larger. Mathematically,

$$ \mathrm d F \leq 0 $$

Suppose we have a gas system going through a reversible process, free energy

$$ \mathrm d F = - S\mathrm d T - p \mathrm d V $$

For fixed $T$ and $V$, we have $\mathrm d F = 0$. For irreversible process, it is less than 0.

#### Chemical potential

Chemical potential is the increased Gibbs free energy when one mole particle is added to the system.


Have you ever wondered, how the control of the water heater in your home works? In Germany, there are two types of water heaters: 1) flow heaters (Durchlauferhitzer), where water will be heated up as it flows out of the water tap, or 2) water boilers (Warmwasserspeicher), where warm water is stored in a tank continuously.

Out of curiousity, I want to present the functionality and the power consumption of such water heaters in the following simulation. As a model, I will take the water boiler and consider the following process. The tank of the water boiler has a volume of $V = 10\,L$. The heating process is electrical with the heat flow $Q$. We will assume that the water is used continuously, meaning the inflow and outflow of the water is constant with $F = 0.15\,\frac{L}{s}$. The temperature of the water outlet should be at $42^\circ{}C$. At the beginning, the tank is already filled with water with the same temperature as the inlet $T_\alpha = 20^\circ{}C$.

#### Nomeclature
| Symbol | |
|---|---|
| $U$ | Internal energy |
| $H$ | Enthalpy flow rate |
| $F$ | Water flow rate in $m^3\,s^{-1}$ |
| $V$ | Tank volume in $m^3$ |
| $T$ | Temperature in $K$ |
| $Q$ | Heat flow rate in $W$ |
| $\rho$ | Water density in $kg\,m^{-3}$ |
| $c_p$ | Heat capacity of water in $J\,kg^{-1}\,K^{-1}$ |

# Balance Equation

The first step in a simulation is the formation of a model equation. Thus, you should dig up your old lecture notes on the subject of heat and mass transfer and construct the heat balance equation. We will assume an ideal mixing inside the tank, so that the outlet temperature is identical to the temperature at any point inside the tank.

The energy balance states that the changes of energy inside the tank is equal to the amount of energy that goes in and out of the system.

\begin{equation} \label{eq:heatBalance}
    \frac{dU}{dt} = H_{\alpha} - H_{\omega} + Q
\end{equation}

The change in internal energy can be represented as $\frac{dU}{dt} = V\,\rho\,c_v\,\frac{dT_R}{dt}$. Further, we assume that water is incompressible, hence $c_v = c_p$. The enthalpy stream can be described with $\dot{H}_i = F\,\rho\,c_p\,T_i$, With the assumption of ideal mixing, the outflow temperature $T_\omega$ is equal to the temperature of the tank $T_R$. The result is the following first-order ordinary differential equation:

\begin{align} \label{eq:tempOde}
    \frac{d T_R}{dt} = \frac{F}{V}(T_{\alpha} - T_R) + \frac{Q}{V\,\rho\,c_p}
\end{align}

The balance equation gives us an initial-value problem with $T_R(t=0) = T_\alpha$.

# Numerical Methods: Runge-Kutta-Method

Despite the fact that the above equation can be analitically solved for a constant heat flow rate $Q$, we will use a numerical method to solve the differential equation, the classical Runge-Kutta-Method. Suppose we have the value of $T_{R,i} \equiv y_i$ at a specific time $t_i$, the value of $y(t_{i+1})$ for the time $t_{i+1} = t_i + \Delta{}t$ can be approximated with the weighted sums of the differential itself. With $f(t, y) = \frac{dT_R}{dt}$:

| | | | |
|-|-|-|-|
| $t_1 = t_i$ | $y_1 = y_i$ | $p_1 = f(t_1, y_1)$ | $k_1 = \Delta{}t\,p_1$ |
| $t_2 = t_i + \frac{\Delta{}t}{2}$ | $y_1 = y_i + \frac{k_1}{2}$ | $p_2 = f(t_2, y_2)$ | $k_2 = \Delta{}t\,p_2$ |
| $t_3 = t_i + \frac{\Delta{}t}{2}$ | $y_3 = y_i + \frac{k_2}{2}$ | $p_3 = f(t_3, y_3)$ | $k_3 = \Delta{}t\,p_3$ |
| $t_4 = t_i + \Delta{}t$ | $y_4 = y_i + k_3$ | $p_4 = f(t_4, y_4)$ | $k_4 = \Delta{}t\,p_4$ |

Consequently, the value of $y(t_{i+1})$ will be
\begin{align} \label{eq:RungeKutta}
    y(t_{i+1}) = y_i + \frac{k_1 + 2k_2 + 2k_3 + k_4}{6}
\end{align}

# Implementation with Python

It is time to implement the simulation in python. We will also use the numpy library for mathematical computing and matplotlib library for plotting. First, we import these libraries and define the differential equation as a function.

```python
import numpy as np
import matplotlib.pyplot as plt
plt.rcParams.update({'font.size': 14})

def temperature_ode(t: float, T_R: float, params: dict):
    '''Differential equation of the heat balance'''
    T_in = params['T_in']
    F = params['F']
    V = params['V']
    rho = params['rho']
    c_p = params['c_p']
    Q = params['Q']

    return (F/V*(T_in - T_R)) + (Q/(V*rho*c_p))
```

Next, we define the parameters as python variables. You might have noticed the `params` argument in the `temperature_ode` function. All of the following parameters will be then passed to the function as one dictionary, a data-type in python. I find this method to be quite convenient.

```python
F = 0.15e-3    # m^3 s^-1
V = 0.01       # m^3
T_in = 293.15  # K
T_R = T_in     # K
Q = 600        # W; initial heat flow rate
rho = 997      # kg m^-3
c_p = 4.186    # J kg^-1
T_set = 315.15 # K

dt = 0.1       # s, time step for numerically solving the differential equation

params = {'F': F, 'V': V, 'T_in': T_in, 'T_R': T_R, 'Q': Q, 'rho': rho,
          'c_p': c_p, 'T_set': T_set, 'dt': dt}
```

In the following function, we define one step of a Runge-Cutta computation.

```python
def runge_kutta_step(fun: callable, t_0: float, y_0: float, dt: float, *args):
    '''Perform one step of classical Runge-Kutta Solver for initial-value problem'''

    t_1 = t_0
    y_1 = y_0
    k_1 = dt * fun(t_1, y_1, *args)

    t_2 = t_1 + 0.5*dt
    y_2 = y_1 + 0.5*k_1
    k_2 = dt * fun(t_2, y_2, *args)

    t_3 = t_2
    y_3 = y_1 + 0.5*k_2
    k_3 = dt * fun(t_3, y_3, *args)

    t_4 = t_1 + dt
    y_4 = y_1 + k_3
    k_4 = dt * fun(t_4, y_4, *args)

    t = t_0 + dt
    k = k_1/6 + k_2/3 + k_3/3 + k_4/6
    y = y_0 + k

    return (t, y)
```

And now all is set. We can start the simulation. As the temperature output of the tank should be $T_\omega = 42^\circ{}C$, we have to calculate the specific heat flow rate needed to reach this end temperature. For this purpose, we will set the balance equation to its stationary state, meaning every time-derivative (in this case $\frac{dT_R}{dt}$) is set to zero. The resulting equation can be solved to the required heat flow rate

\begin{equation}\label{eq:stationaryQ}
    Q = F\,\rho\,c_p\,(T_R - T_{\alpha})
\end{equation}

```python
Q_stat = F*rho*c_p*(T_set - T_in)
params['Q'] = Q_stat

print(f'For an output temperature of {T_set:.2f} K,' \
      f' the required heat flow rate is {Q_stat:.2f} W.')
```

    For an output temperature of 315.15 K, the required heat flow rate is 13.77 W.

Thus we will use this heat flow rate in the simulation. To give an idea, how $T_R$ varies with $Q$, we will also simulate the temperature for different $Q$s.

```python
params_1 = params.copy()
params_1['Q'] = Q_stat / 2
params_2 = params.copy()
params_3 = params.copy()
params_3['Q'] = Q_stat * 2

t_start = 0
t_end = 400

time = np.arange(t_start, t_end+dt*0.1, dt)
temp_1 = np.zeros(len(time))
temp_2 = np.zeros(len(time))
temp_3 = np.zeros(len(time))

for idx, t in enumerate(time):
    if idx == 0:
        # Set the initial values
        temp_1[idx] = T_R
        temp_2[idx] = T_R
        temp_3[idx] = T_R
    else:
        _, temp_1[idx] = runge_kutta_step(temperature_ode, t,
                                          temp_1[idx-1], dt, params_1)
        _, temp_2[idx] = runge_kutta_step(temperature_ode, t,
                                          temp_2[idx-1], dt, params_2)
        _, temp_3[idx] = runge_kutta_step(temperature_ode, t,
                                          temp_3[idx-1], dt, params_3)

# Plot result
fig, ax = plt.subplots(figsize=(8, 6))
ax.plot(time, temp_1-273.15, label=f'$Q = {Q_stat/2:.2f}\,W$')
ax.plot(time, temp_2-273.15, '-g', label=f'$Q = {Q_stat:.2f}\,W$')
ax.plot(time, temp_3-273.15, label=f'$Q = {Q_stat*2:.2f}\,W$')
ax.plot([min(time), max(time)], [42, 42], '--')
ax.grid('on')
ax.legend()
ax.set_xlabel('Time $t$ in s')
ax.set_ylabel('Temperature $T_R$ in $^\circ{}C$')
ax.set_xlim([time[0], time[-1]])
fig.tight_layout()
```

And the simulation is done. As you can see, our water heater would need roughly $350\,s$ under the given conditions before reaching the desired temperature. You probably wouldn't want to wait so long (especially in the morning) to get the pleasant warm water. And that is where the concept of control comes in.

# Temperature Control

In the previous figure, we have seen that the stationary outlet temperature $T_R$ depends on the heat input $Q$, where a higher heat rate leads to a higher temperature. If we look at the first 50 seconds, we also see that the higher heat rate results in a faster rise in outlet temperature. Therefore, one idea of control is to change an actuator (in this case the heat flow rate) over the course of time to reach the desired output as quickly as possible.

For our case, we will use a feedback controller with classical tuning rules. As a first step, we will take the balance equation:

\begin{align}
    \frac{d T_R}{dt} = \frac{F}{V}(T_{\alpha} - T_R) + \frac{Q}{V\,\rho\,c_p}
\end{align}

and perform a Laplace-Transformation on it. Suppose that the laplace transformation of the controlled variable is $\mathcal{L}\{T_R\} = Y(s)$ and the control signal is $\mathcal{L}\{Q\} = U(s)$:

\begin{alignat}{3}
    &\quad& s\,Y(s) - T_R(t=0) &= \frac{F}{V}\left(s\,T_{\alpha} - Y(s)\right) + \frac{1}{V\,\rho\,c_p}U(s) \\
    \Leftrightarrow&\quad& \left(s + \frac{F}{V}\right)\,Y(s) &= \frac{1}{V\,\rho\,c_p}U(s) + \frac{s\,F\,T_\alpha}{V} + T_R(t=0) \\
    \Leftrightarrow&\quad& Y(s) &= \frac{1}{V\,\rho\,c_p\,\left(s + \frac{F}{V}\right)}U(s) + \frac{s\,F\,T_\alpha}{V\,\left(s + \frac{F}{V}\right)} + \frac{T_R(t=0)}{\left(s + \frac{F}{V}\right)}
\end{alignat}

In this frequency domain, the system is defined as the transfer function of the control signal $U(s)$ to the controlled variable $Y(s)$.

\begin{align}
    G(s) &= \frac{1}{V\,\rho\,c_p\,\left(s + \frac{F}{V}\right)} = \frac{1}{V\,\rho\,c_p\,s + F\,\rho\,c_p} \nonumber \\
    &= \frac{\frac{1}{F\,\rho\,c_p}}{\frac{V}{F}\,s + 1} \equiv \frac{K}{T_1\,s + 1}
\end{align}

In the classical tuning rules by **Ziegler** and **Nichols**, a PI-Controller for the so called PT$_1$-Model above is obtained via the rule $K_P = \frac{1}{K}$ and $T_I = 0.07\,T_1$.

```python
# Define controller parameter
K = 1 / (F * c_p * rho)
T_1 = V / F

K_P = 1 / K
T_I = T_1 * 0.7

print(f"The obtained controller parameters are: K_P = {K_P:.2f}, T_I = {T_I:.2f}")
```

    The obtained controller parameters are: K_P = 0.63, T_I = 46.67

We can now simulate these parameters together with the system differential equation from above. To do this, we would also need the following PI-controller equation

\begin{align}
    Q(t) = K_P\,e(t) + \frac{K_P}{T_I}\int_0^t e(\tau)\,d\tau
\end{align}

The argument $e(t)$ in this integral refers to the error of the controlled variable to its reference signal $e(t) = T_{set} - T_R(t)$. In the implementation, we would approximate the integral with the sum.

```python
def pi_controller(epsilon: np.ndarray, K_P, T_I, dt):
    '''
    Implementation of PI-Controller

    epsilon is the error of the controlled variable to its reference, saved as a list.
    '''
    return K_P * epsilon[-1] + (K_P/T_I)*np.sum(np.multiply(epsilon, dt))
```

All is set, we can now begin the simulation.

```python
t_start = 0
t_end = 400

time = np.arange(t_start, t_end+dt*0.1, dt)
temp = np.zeros(len(time))
Q = np.zeros(len(time))
epsilon = np.array([])

for idx, t in enumerate(time):
    if idx == 0:
        # Set the initial values
        temp[idx] = T_R
    else:
        epsilon = np.append(epsilon, (T_set - temp[idx-1]))
        Q[idx] = pi_controller(epsilon, K_P, T_I, dt)
        params['Q'] = Q[idx]
        _, temp[idx] = runge_kutta_step(temperature_ode, t, temp[idx-1], dt, params)

# Plot result
fig, ax = plt.subplots(figsize=(8, 6))

ax.plot(time, temp-273.15, 'g')
ax.plot([min(time), max(time)], [42, 42], '--')
ax.grid('on')
ax.set_xlabel('Time $t$ in s')
ax.set_ylabel('Temperature $T_R$ in K')
ax.set_xlim([time[0], time[-1]])
ax.yaxis.label.set_color('g')
ax.tick_params(axis='y', colors='g')

ax2 = ax.twinx()
ax2.plot(time, Q, 'r')
ax2.set_ylabel('Heat flow rate $Q$ in W')
ax2.yaxis.label.set_color('r')
ax2.tick_params(axis='y', colors='r')

fig.tight_layout()
```

As you can see, the temperature of the water flowing out of the heater would now reach the desired temperature within roughly 150 s, instead of 300 s as shown in the previous figure. The controller would be able to reduce your waiting time by 50%! On the downside, notice that there are ranges where the water temperature would have exceeded the $42 ^\circ C$ line. While you probably wouldn't mind a temperature a little bit higher than what you would have set, such oscillating behaviour is quite often undesirable, if not harmful in some occasions. In any cases, an iteration or two of controller's optimization should be done and controllers from a quick tuning rule are far from optimized. Regardless, you should have gotten an idea, how such control system works in general.

### The end

You can find this notebook in this [link](https://github.com/FelixFebrian/FelixFebrian.github.io/blob/master/notebooks/2020-07-02-Temperature-Control-of-Water-Heater.ipynb). Let me know that you like (or don't like) this article and any topic ideas are warmly welcomed.

```python

```


###### Content provided under a Creative Commons Attribution license, CC-BY 4.0; code under MIT License. (c)2014 [David I. Ketcheson](http://davidketcheson.info)

##### version 0.1 - May 2014

# Hyperbolic Conservation Laws

\begin{equation*}
\newcommand{Dx}{\Delta x}
\newcommand{Dt}{\Delta t}
\newcommand{imh}{{i-1/2}}
\newcommand{iph}{{i+1/2}}
\end{equation*}
Many models of wave phenomena are governed by *hyperbolic conservation laws*.  In this short course, we will learn about hyperbolic conservation laws and their numerical solution.

## Conservation of mass

Imagine a fluid flowing in a narrow tube.  We'll use $q$ to indicate the density of the fluid and $u$ to indicate its velocity.  Both of these are functions of space and time: $q = q(x,t)$; $u=u(x,t)$.  The total mass in the section of tube $[x_1,x_2]$ is

\begin{equation}
\int_{x_1}^{x_2} q(x,t) dx.
\end{equation}

This total mass can change in time due to fluid flowing in or out of this section of the tube.  We call the rate of flow the *flux*, and represent it with the function $f(q)$.  Thus the net rate of flow of mass into (or out of) the interval $[x_1,x_2]$ at time $t$ is

$$f(q(x_1,t)) - f(q(x_2,t)).$$

We just said that this rate of flow must equal the time rate of change of total mass; i.e.

$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = f(q(x_1,t)) - f(q(x_2,t)).$$

Now since $\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx = f(q(x_2,t)) - f(q(x_1,t))$, we can rewrite this as

$$\frac{d}{dt} \int_{x_1}^{x_2} q(x,t) dx = -\int_{x_1}^{x_2} \frac{\partial}{\partial x} f(q) dx.$$

Under certain smoothness assumptions on $q$, we can move the time derivative inside the integral.  We'll also put everything on the left side, to obtain

$$\int_{x_1}^{x_2} \left(\frac{\partial}{\partial t}q(x,t) + \frac{\partial}{\partial x} f(q)\right) dx = 0.$$

Since this integral is zero for *any* choice of $x_1,x_2$, it must be that the integrand (the expression in parentheses) is actually zero *everywhere*!  Therefore we can write the **differential conservation law**

$$q_t + f_x = 0.$$

Here and throughout the course, we use subscripts to denote partial derivatives.
This equation expresses the fact that the total mass is conserved -- since locally the mass can change only due to a net inflow or outflow.

## Advection

In order to solve the conservation law above, we need an expression for the flux, $f$.  The rate of flow is just mass times velocity: $f=u q$.  Thus we obtain the **continuity equation**

$$q_t + (uq)_x = 0.$$

In general, we need another equation to determine the velocity $u(x,t)$.  In [Lesson 4](Lesson_04_Fluid_dynamics.ipynb) we'll look at the full equations of fluid dynamics, but for now let's consider the simplest case, in which all of the fluid flows at a single, constant velocity $u(x,t)=a$.  Then the continuity equation becomes the **advection equation**

$$q_t + a q_x = 0.$$

This equation has a very simple solution.  If we are given the density $q(x,0)=q_0(x)$ at time zero, then the solution is just

$$q(x,t) = q_0(x-at).$$

Let's plot the solution of the advection equation on the interval $[0,1]$ for the initial condition
$$q_0(x) = e^{-2(x-1/2)^2}.$$

First, let's import all the modules we'll need.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from matplotlib import animation
```

```python
from IPython.display import HTML

def display_animation(anim):
    plt.close(anim._fig)
    return HTML(anim.to_jshtml())
```

Next, we'll set up a grid and the initial condition:

```python
x = np.linspace(0,1,1000)  # Spatial grid
t = np.linspace(0,1)       # Temporal grid
a = 1.0                    # Advection speed

def q_0(x):                # Initial condition
    return np.exp(-200.*(x-0.2)**2)
```

Finally, let's make an animation of the solution.  It will take a few moments to run this code.  For now, you don't need to worry about understanding all of the plotting code below.  Just play with the animation until you have a feel for how the solution behaves.

```python
fig = plt.figure(figsize=(8,4))      # Create an empty figure
ax  = plt.axes()
line, = ax.plot([], [],linewidth=2)  # Create an empty line plot
plt.axis((0,1,-0.1,1.1))             # Set the bounds of the plot

def plot_q(t):
    line.set_data(x,q_0(x-a*t))  # Replace the line plot with the solution at time t

anim = animation.FuncAnimation(fig, plot_q, frames=t)  # Animate the solution

display_animation(anim)
```

<link rel="stylesheet"
href="https://maxcdn.bootstrapcdn.com/font-awesome/4.4.0/
css/font-awesome.min.css">

    <input id="_anim_slider7bd28756ab9441edb958a3c7bfd3052d" type="range" style="width:350px"
           name="points" min="0" max="1" step="1" value="0"
           onchange="anim7bd28756ab9441edb958a3c7bfd3052d.set_frame(parseInt(this.value));"></input>

    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.slower()"></button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.first_frame()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.previous_frame()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.reverse_animation()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.pause_animation()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.play_animation()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.next_frame()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.last_frame()">
        </button>
    <button onclick="anim7bd28756ab9441edb958a3c7bfd3052d.faster()"></button>
  <form action="#n" name="_anim_loop_select7bd28756ab9441edb958a3c7bfd3052d" class="anim_control">
    <input type="radio" name="state"
           value="once" > Once </input>
    <input type="radio" name="state"
           value="loop" checked> Loop </input>
    <input type="radio" name="state"
           value="reflect" > Reflect </input>
  </form>

As you can see, the initial pulse just moves to the right at speed $a$ as time advances.  This isn't very interesting, but it captures the most important feature of hyperbolic equations: waves travel at finite speed.

## Characteristics

Notice that the solution value is constant along the line $x-at=x_0$, in the $x-t$ plane, for each value of $x_0$.  These lines are called **characteristics**; they are the trajectories along which solution information is transmitted.  The value $a$ is referred to as the **characteristic velocity**.  The code below plots some of these characteristics.

When we learn about more complicated conservation laws, we'll see that information still travels along characteristics, but those characteristics aren't necessarily straight lines.

```python
fig = plt.figure(figsize=(8,4))
ax  = plt.axes()

for x_0 in np.linspace(0,1,10):
    ax.plot(x,(x-x_0)/a,'-k')
plt.ylim(0,1)
```

## A finite volume method for advection

We can easily solve the advection equation exactly.  But the advection equation is a prototype for more complicated conservation laws that we will only be able to solve approximately by using numerical methods.  In order to better understand these methods, we will discuss them first in the context of the advection equation.

For simplicity, we'll suppose that we wish to solve the advection equation on the interval $[0,1]$.  We introduce a set of equally spaced *grid cells* of width $\Dx$, and write $x_i$ to mean the center of cell $i$.  Thus the first cell is the interval $[0,\Dx]$ and $x_1=\Dx/2$.  We will also write $x_\imh$ or $x_\iph$ to denote the left or right boundary of cell $i$, respectively.

We write $Q_i$ to denote the *average* value of the solution over cell $i$:

$$Q_i = \frac{1}{\Dx} \int_{x_\imh}^{x_\iph} q \ dx.$$

The simplest finite volume method is obtained by supposing that the solution is actually *equal* to $Q_i$ over all of cell $i$.

Suppose $a>0$.  Then the flux into cell $i$ from the left is $a Q_{i-1}$ and the flux out of cell $i$ to the right is $a Q_i$.  Then our integral conservation law reads

$$Q_i'(t) = -\frac{a}{\Dx}\left(Q_i - Q_{i-1}\right).$$

Applying a forward difference in time we obtain the *upwind method*

$$Q^{n+1}_i = Q^n_i -\frac{a}{\Dx}\left(Q_i - Q_{i-1}\right).$$

We call this the upwind method because the solution behaves as if it were being blown by a wind to the right, and the method uses the value $Q_{i-1}$ from the upwind direction.

Here is a bit of Python code to solve the advection equation using the upwind method.

```python
a = 1.0     # advection speed

m = 50      # number of cells
dx = 1./m   # Size of 1 grid cell
x = np.arange(-dx/2, 1.+dx/2, dx)  # Cell centers, including ghost cells

t = 0.      # Initial time
T = 0.5     # Final time
dt = 0.8 * dx / a  # Time step

Q = np.exp(-200*(x-0.2)**2)    # Initial data
Qnew = np.empty(Q.shape)

while t < T:

    # Extrapolation at boundaries:
    Qnew[0] = Q[1]
    Qnew[-1] = Q[-2]

    for i in range(1,len(x)):
        Qnew[i] = Q[i] - a*dt/dx * (Q[i]-Q[i-1])

    Q = Qnew.copy()
    t = t + dt

plt.plot(x,Q,linewidth = 2)
plt.title('t = '+str(t));
```

Notice how we set up a grid that contains an extra cell at each end, outside of the problem domain $[0,1]$.  These are called **ghost cells** and are often useful in handling the solution at the grid boundaries.

  The technique we have used to set the ghost cell values above, by copying the last value inside the grid to the ghost cells, is known as **zero-order extrapolation**.  It is useful for allowing waves to pass out of the domain (so-called *non-reflecting* boundaries).  Note that we don't actually need the ghost cell at the right end for the upwind method, but for other methods we will.

The upwind method is simple, but it is not very accurate.  Notice how the computed solution becomes wider and shorter over time.  This behavior is referred to as *dissipation*.

### Exercise

Now do the following with the code above:

1. Set $m=1000$ or more and notice that it takes some time to compute the solution.  Rewrite the inner loop (over $i$) as a single line with no loop, using numpy slicing.  For large values of $m$, the code with slicing is much faster.
1. Notice that the last step of the simulation goes past time $T$.  Modify the code so that the last step is adjusted to exactly reach $T$.
2. Change the code so that animation of the solution versus time is plotted.  You will want to accumulate frames of the solution in a list and then use the same kind of code we used above to animate the exact solution.
3. Add some code to plot the exact solution.

*Extra credit*: change the left boundary condition so that there is a sinusoidal wave coming in from the left:
$$u(0,t) = \sin(20 \pi t).$$
What do you notice about the sinusoid as it moves into the domain?

```python

```

After making it through the exercise above, you should feel pretty comfortable with the basics of scientific programming in Python.

## The CFL condition

Take a look at the line of code that sets the time step:
```python
    dt = 0.8 * dx / a
```
You might be wondering where that formula came from.  Rearranging that equation, we have
$$a \frac{\Delta t}{\Delta x} = 0.8.$$
The quantity $\nu = a \frac{\Delta t}{\Delta x}$ is the distance the exact solution moves during each time step, in units of grid cells.  It is referred to as the *CFL number* or just the *Courant number* after the authors Courant, Friedrichs and Lewy who [established its importance](http://www.stat.uchicago.edu/~lekheng/courses/302/classics/courant-friedrichs-lewy.pdf).  Try the following values of $\nu$ in the code above, and compare the results with those you obtained already using $\nu=0.8$.
1. $\nu = 1.0$
2. $\nu = 1.5$
3. $\nu = 0.1$

Finally, try setting $a$ to a negative value.  What happens?

The results you have observed can be explained as follows.  Over a time step of size $\Dt$, the solution moves by an amount $a \Dt$.  So $q(x_i,t_n)$ should be given exactly by $q(x_i-a\Dt,t_{n-1})$.  This is referred to as the *domain of dependence* of the solution.

The upwind method uses the values $Q_i^{n-1}$ and $Q_{i-1}^{n-1}$ to compute $Q_i^n$.  The points $(x_{i-1},t_n)$ and $(x_i,t_n)$ (as well as the locations of solution values they depend, and the ones those depend on, and so forth) are the *numerical domain of dependence*.

For this to work, the true domain of dependence $x_i-a\Dt$ must lie within the numerical domain of dependence, as $\Dt,\Dx \to 0$.  This is known as the [CFL condition](http://en.wikipedia.org/wiki/Courant%E2%80%93Friedrichs%E2%80%93Lewy_condition).

For the upwind method, that means that we must have
$$x_i - \Dx \le x_i - a \Dt \le x_i$$
or in other words
$$0 \le a \frac{\Dt}{\Dx} \le 1.$$

If the CFL condition is violated, the information that is used by the numerical method doesn't include the true information that influences the exact solution, so the numerical solution cannot be convergent.

## The Lax-Friedrichs method

The upwind method gets its name from the fact that it uses the value $U_{i-1}$ and not $U_{i+1}$.  Assuming $a>0$, the correct solution value $U_i^{n+1}$ should come from a point to the left of $x_i$ at time $t_n$ (i.e., the wind blows to the right, so $x_{i-1}$ is *upwind* of $x_i$).

This bias is fine for the advection equation, where we know everything moves in the same direction.  But for more complicated conservation laws, things may move in either direction.  It will be useful to have a method that uses information from both directions.  The simplest such method is known as the **Lax-Friedrichs** method.  For the conservation law $q_t + f(q)_x$, this method is

$$Q_i^{n+1} = \frac{1}{2}(Q_{i-1}^n + Q_{i+1}^n) - \frac{\Dt}{2\Dx}\left(f(Q_{i+1}^n) - f(Q_{i-1}^n)\right).$$

Notice that the flux difference term clearly approximates $f(q)_x$.  Meanwhile, the value of $q$ itself is approximated by taking the average of two neighboring values.  This average makes this method dissipative too (but it ensures that the solution is stable).

### Exercise

1. What does the CFL condition imply for the time step when using the Lax-Friedrichs method?

2. In the cell below, implement the Lax-Friedrichs method for advection.

```python

```

*Extra credit*: Compute the norm of the difference between the approximate and exact solution.  How does it change if you decrease $\Dx$?

## Accuracy

The methods we have used so far (i.e., the *upwind method* and the *Lax-Friedrichs method*) are both dissipative.  Furthermore, both of these methods are only *first order accurate*, meaning that if we reduce the values of $\Dt$ and $\Dx$ by a factor of two, the overall error decreases only by a factor of two.  In [Lesson 3](Lesson_03_High-resolution_methods.ipynb), we will learn about more accurate methods.  But first, in [Lesson 2](Lesson_02_Traffic.ipynb) we'll look at a model for traffic flow.


# Physics 256
## Chaos in the Driven Dissipative Pendulum

```python
import style
style._set_css_style('../include/bootstrap.css')

from IPython.display import YouTubeVideo
# A driven double pendulum
YouTubeVideo('7DK1Eayyj_c')
```

## Last Time
### [Notebook Link: 16_SimpleHarmonicMotion](./16_SimpleHarmonicMotion)
- simple harmonic motion
- instability of the Euler method
- effects of non-linearities

## Today

- The driven dissipative  pendulum

## Setting up the Notebook

```python
import matplotlib.pyplot as plt
import numpy as np
%matplotlib inline
plt.style.use('../include/notebook.mplstyle');
%config InlineBackend.figure_format = 'svg'
colors = plt.rcParams['axes.prop_cycle'].by_key()['color']
```

## Dissipative Driven Pendulum

## Equation of Motion

In the presence of external forces (including those that lead to non-conservation of energy) due to:

 * Gravity: $\;m g \hat{y}$
 * Dissipation: $\;-\gamma\; \vec{v}$
 * Motor: $\; F_\mathrm{D}(t) = F_\mathrm{D} \sin (\Omega_\mathrm{D} t) \hat{\theta}$

We can write down the equation of motion from Newton's law: $\; m \frac{d \vec{v}}{d t^2} = \vec{F} $

Since $x^2 + y^2 = \ell^2$ it is convenient to write everything in terms of the angle $\theta$:

$$ y = \ell \cos \theta $$
$$ x = \ell \sin \theta $$

In plane polar coordinates $\vec{r} = (r,\theta)$ where

\begin{align}
\hat{r} &= \cos \theta \hat{x} + \sin\theta \hat{y} \\
\hat{\theta} &= -\sin\theta \hat{x} + \cos\theta \hat{y}
\end{align}

and for $d r / dt = 0$ we have:

$$ \vec{v} = \ell \dot{\theta} \hat{\theta}$$

$$ \dot{\vec{v}} = -\ell \dot{\theta}^2 \hat{r} + \ell \ddot{\theta} \hat{\theta}$$

$$ \hat{y} = \cos \theta\; \hat{r} - \sin \theta\; \hat{\theta} $$

and measuring $\gamma$ and $F_D$ in terms of the mass $m$ the equation of motion in the $\hat{\theta}$ direction is:

\begin{equation}
\frac{d^2 \theta}{d t^2} = -\frac{g}{\ell}\sin\theta - \gamma \frac{d\theta}{d t} + F_\mathrm{D} \sin \Omega_\mathrm{D}t
\end{equation}

Last time, we solved this EOM using the Euler-Cromer method for the case of $\gamma = F_{\rm D} = 0$ and compared to the exact solution.

\begin{eqnarray}
\theta(t) &=& 2 \sin^{-1} \left\{ k\, \mathrm{sn}\left[K(k^2)-\sqrt{\frac{g}{\ell}} t; k^2\right] \right\} \newline
k &=& \sin \frac{\theta_0}{2} \newline
K(m) &=& \int_0^1 \frac{d z}{\sqrt{(1-z^2)(1-m z^2)}}
\end{eqnarray}

```python
def non_linear_θ(ℓ,θ0,t):
    '''The solution for θ for the non-linear pendulum.'''
    # use special functions
    from scipy import special
    k = np.sin(θ0/2.0)
    K = special.ellipk(k*k)
    (sn,cn,dn,ph) = special.ellipj(K-np.sqrt(g/l)*t,k*k)
    return 2.0*np.arcsin(k*sn)
```

```python
from scipy.constants import pi as π
from scipy.constants import g

# constants and intitial conditions
ℓ = 0.25 # m
Δt = 0.001 # s

t = np.arange(0.0,4.0,Δt)
θ,ω = np.zeros_like(t),np.zeros_like(t)
θ[0] = π/4.0 # rad

for n in range(t.size-1):
    θ[n+1] = θ[n] + ω[n]*Δt
    ω[n+1] = ω[n] -(g/ℓ)*np.sin(θ[n+1])*Δt

# the exact solution
plt.plot(t,non_linear_θ(ℓ,θ[0],t), color=colors[0],label='Exact')

# the Euler-Cromer method
plt.plot(t[::20],θ[::20], 'o', mfc='None', markersize = 6, mec=colors[0],label='Euler Cromer method')
plt.legend(loc='lower left',frameon=True)

plt.xlabel('Time [s]')
plt.ylabel('θ(t) [rad]')
```

    Text(0, 0.5, 'θ(t) [rad]')

## Adding Dissipation and the Driving Force

We are unable to find an analytical solution for the general case.  However, we can easily use the semi-implicit Euler method to compute an approximate solution.

\begin{eqnarray}
\frac{d \omega}{d t} &=& -\frac{g}{\ell}\sin\theta - \gamma \frac{d\theta}{d t} + F_\mathrm{D} \sin \Omega_\mathrm{D}t \newline
\frac{d\theta}{d t} &=& \omega.
\end{eqnarray}

For a time-step $\Delta t \ll 1$ we can approximate the continuous derviative using a discrete forward approximation:

\begin{eqnarray}
\omega_{n+1} &=& \omega_n - \frac{g}{\ell} \sin[\theta_n] \Delta t
- \gamma \omega_n \Delta t + F_D \sin[\Omega_D t] \Delta t \newline
\theta_{n+1} &=& \theta_n + \omega_{n+1} \Delta t
\end{eqnarray}

```python
from scipy.constants import g
from scipy.constants import pi as π
def euler(t,FD,ℓ,θ0,ω0,γ,ΩD):
    ''' Semi-implicit Euler Method for the non-linear, dissipative, driven pendulum.'''

    Δt = t[1]-t[0]
    ω,θ = np.zeros_like(t),np.zeros_like(t)
    θ[0],ω[0] = θ0,ω0

    # perform the numerical integration
    for n in range(t.size-1):
        ω[n+1] = ω[n] + (-(g/ℓ)*np.sin(θ[n]) - γ*ω[n] + FD*np.sin(ΩD*t[n]))*Δt
        θ[n+1] = θ[n] + ω[n+1]*Δt

        # keep theta in [-pi,pi)
        if θ[n+1] < -π: θ[n+1] += 2.0*π
        if θ[n+1] >= π: θ[n+1] -= 2.0*π

    return θ,ω
```

### Determine how the driving force effects the time evolution

```python
params = ℓ,θ0,ω0,γ,ΩD = g, 0.2, 0.0, 0.5, 2.0/3.0
FD = [0.0,0.5,0.75,1.0,1.25,1.5]
Δt = 0.04
t = np.arange(0.0,60,Δt)
```

#### No driving force: $F_{\mathrm{D}} = 0$

```python
θ,ω = euler(t,0.0,*params)

plt.plot(t,θ)
plt.title(r'$F_\mathrm{D} = %3.1f\; \mathrm{s}^{-2}$' % 0.0)
plt.xlabel('Time [s]')
plt.ylabel('θ(t) [rad]')
```

#### Weak driving force

```python
params = g, 0.2, 0.0, 0.5, 2.0/3.0
θ,ω = euler(t,0.2,*params)

plt.plot(t,θ)
plt.title(r'$F_\mathrm{D} = %3.1f\; \mathrm{s}^{-2}$' % 0.2)
plt.xlabel('Time [s]')
plt.ylabel('θ(t) [rad]')
```

    Text(0, 0.5, 'θ(t) [rad]')

### Let's investigate the explicit $F_\mathrm{D}$ dependence

```python
F = [0.0,0.5,0.75,1.0,1.25,1.5]

# create a subplot array
fig, axes = plt.subplots(2,3, figsize=(12,10), sharex=True)
fig.subplots_adjust(wspace=0.3)
for i, ax in enumerate(axes.flat):
    θ,ω = euler(t,F[i],*params)
    ax.plot(t,θ, label=r'$F_\mathrm{D} = %3.1f\; \mathrm{s}^{-2}$' % F[i])
    ax.legend(frameon=True, loc='lower right')

# set axis labels
[ax.set_ylabel('θ(t) [rad]') for ax in axes[:,0]]
[ax.set_xlabel('Time [s]') for ax in axes[-1,:]]
```

    [Text(0.5, 0, 'Time [s]'), Text(0.5, 0, 'Time [s]'), Text(0.5, 0, 'Time [s]')]

## Understanding Chaos

 * What does **chaos** mean in this context?
 * It appears that when the driving force is strong, the angle as a function of time does not follow any predictable pattern.
 * However, we **CAN** predict it, as we have just calculated θ(t) via a numerical integration of the equation of motion.
 * This apparent contradiction is referred to as *deterministic chaos* and is common for non-linear ODE’s

This type of chaos is most easily understood in terms of the sensitivity of the trajectory on the initial conditions.

```python
θ0 = [0.2,0.21,0.22] #  initial conditions
F = [0.5,1.2] # driving force

# compare the initial conditions
fig, axes = plt.subplots(2,1,sharex=True,sharey=False, figsize=(6,8))
for i, ax in enumerate(axes.flat):
    for j,cθ in enumerate(θ0):
        label = r'$\theta_0 = %4.2f\,\mathrm{rad}$' % cθ
        params = ℓ,cθ,ω0,γ,ΩD
        θ,ω = euler(t,F[i],*params)
        ax.plot(t,θ,label=label)
    ax.legend(frameon=True, loc='lower left')
    ax.text(70,0.0,r'$F_\mathrm{D} = %3.1f\; \mathrm{s}^{-2}$' % F[i],fontsize=30)

    # set axis labels
    ax.set_ylabel('θ(t) [rad]')

axes[-1].set_xlabel('Time [s]')
```

    Text(0.5, 0, 'Time [s]')

## Quantifying this Effect

 * We can try to quantify this effect by looking at the difference in angle at long times for two pendulums with nearly identical initial conditions.
 * Start them off with the same initial velocity, but with an initial angle difference of $\Delta \theta_0 = 0.0001$ radians.

```python
F = [0.5,1.2] # driving force [1/s^2]
θ0 = 0.2 # initial angle [rad]
params = [(ℓ,θ0,ω0,γ,ΩD),(ℓ,θ0+1.0E-4,ω0,γ,ΩD)]

Δθ = []
fig, axes = plt.subplots(2,1,sharex=True, sharey=False, squeeze=False, figsize=(6,8))
for i, ax in enumerate(axes.flat):
    label = r'$F_\mathrm{D} = %3.1f\; \mathrm{s}^{-2}$' % F[i]

    θ1,ω = euler(t,F[i],*params[0])
    θ2,ω = euler(t,F[i],*params[1])
    Δθ.append(np.abs(θ1-θ2))

    ax.semilogy(t, Δθ[i], label=label)
    ax.legend(loc='best', frameon=False)

    # set axis labels
    ax.set_ylabel('|Δθ(t)| [rad]')

axes[1,0].set_xlabel('Time [s]')
```

    Text(0.5, 0, 'Time [s]')

Note that the two regimes differ by the slope of the general linear trend on a log scale.

$$ \log \Delta \theta \sim \lambda t $$
$$ \Rightarrow \Delta \theta \propto \mathrm{e}^{\lambda t} $$

$\lambda$ is known as the Lyapunov exponent after Russian mathematician Aleksandr Lyapunov

and can be used to characterize the sensitivity of a dynamic system to initial conditions.

$ \lambda > 0 \Rightarrow $ chaotic

$ \lambda < 0 \Rightarrow $ non-chaotic

Let's quantify this by trying to extract numerical values for the Lyapunov exponent.

We will use two methods from ```scipy```:

1. [scipy.signal.argrelextrema](https://docs.scipy.org/doc/scipy/reference/generated/scipy.signal.argrelextrema.html):  Can find local minima and maxima.
2. [scipy.optimize.curve_fit](https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.curve_fit.html):  General curve-fitting.

```python
from scipy.signal import argrelextrema
from scipy.optimize import curve_fit
F = [0.5,1.2]

# Linear fitting function
def linear(x,a0,a1):
    return a0 + a1*x

# find the local maxima and perform the fit
popt = [0,0]
for i,cF in enumerate(F):
    ind = argrelextrema(np.log(Δθ[i]),np.greater)[0]
    popt[i], pcov = curve_fit(linear,t[ind],np.log(Δθ[i][ind]))
```

#### Plot the results of the fit

```python
fig, axes = plt.subplots(2,1,sharex=True, sharey=False, squeeze=False, figsize=(6,8))
for i, ax in enumerate(axes.flat):
    labellam = r'$\lambda = %4.2f\; \mathrm{s}^{-1}$' % popt[i][1]

    ax.semilogy(t, Δθ[i], ',', markeredgewidth=0.0)
    ax.semilogy(t, np.exp(linear(t,*popt[i])), linewidth=3.0, label=labellam, color=colors[0])

    # set labels and legend
    ax.set_ylabel('|Δθ(t)| [rad]')
    ax.text(70,1.0E-6,r'$F_\mathrm{D} = %3.1f\; \mathrm{s}^{-2}$' % F[i],fontsize=30)
    ax.legend()
axes[1,0].set_xlabel('Time [s]')
```

    Text(0.5, 0, 'Time [s]')

### Code Re-Use

We keep needing to generate a label based on the driving force.  Let's write a function to do this:

```python
def FD_label(Fd):
    '''Create a label based on the value of FD'''
    return f'$F_\mathrm{{D}} = {F[i]:G}\; \mathrm{{s}}^{{-2}}$'
```

## Phase Space Plots

So far we have only analyzed the angle $\theta$ as a function of time, but there is information encoded in the angular velocity $\omega$ as well.  We can observe evidence of chaotic motion by looking at plots of $\theta$ vs $\omega$

   We are using ax.text and transform=ax.transAxes here to label our plots.

```python
params = ℓ,θ0,ω0,γ,ΩD
F = [0.5,1.2]

fig, axes = plt.subplots(2,1,sharex=True, sharey=False, squeeze=False, figsize=(6,8))
for i, ax in enumerate(axes.flat):
    θ,ω = euler(t,F[i],*params)

    ax.plot(θ,ω,marker='o', linestyle='None', ms=1)

    # set axis labels and legends
    ax.set_ylabel('ω [rad/s]')
    ax.set_xlim(-π,π)
    ax.text(1.05,0.5,FD_label(F[i]),fontsize=30,transform=ax.transAxes)

axes[1,0].set_xlabel('θ(t) [rad]')
```

 * For a small driving force, we observe transients then relaxation into periodic motion.
 * For large driving force, the trajectory has many nearly closed orbits.

## Poincaré Section

These nearly closed orbits are related to so-called **Strange Attractors** which can be found by looking at a phase space plot that is **in phase** with the driving force.  This is equivalent ot making a *stroboscopic* measurement.

$$ \Omega_\mathrm{D} t = 2 n \pi $$

where $n \in \mathbb{Z}$.  We can code this by creating a stride $m$ that is equivalent to the length of the period $T = 2\pi/\Omega_D$:

$$ p_{\rm step} = \left\lfloor \frac{T}{\Delta t} + \frac{1}{2} \right\rfloor $$

This is equivalent to only selecting those time steps when:
\begin{equation}
\left | t[n] - \frac{2 n \pi}{\Omega_D} \right| < \frac{\Delta t}{2} .
\end{equation}

### Generate a longer solution to include more phase-space data

```python
Δt = 2.0*π/(ΩD*100)
longt = np.arange(0,10000,Δt)

F = [0.5,1.2]
θ,ω = [],[]
for cF in F:
    th,w = euler(longt,cF,*params)
    θ.append(th)
    ω.append(w)
```

### Get the in-phase time slices

 Programming challenge

    Determine the stride value pstep corresponding to the in-phase times.

```python

```

### Now we plot the actual Poincaré section

```python
pcol = colors[1],colors[-2]
plt.figure(figsize=(8,8))
for i in range(2):
    plt.plot(θ[i][::pstep], ω[i][::pstep], marker='o', color=pcol[i], ms=3, linestyle='None', label=FD_label(F[i]), alpha=0.5)

plt.title('Strange Attractors')
plt.legend(fontsize=16, handletextpad=0.2, markerscale=2)
plt.xlabel('θ [rad]')
plt.ylabel('ω [rad/s]')
plt.xlim(-π,π);
```

The Poincare section looks very different for the chaotic vs non-chaotic regime.

 * Two points in non-chaotic regime:
    * Original periodic motion
    * Periodic motion, parallel to the driver
 * In chaotic regime, we get a surface.
    * _Fuzziness_ not due to numerics: property of the system.
    * Largely independent of details of initial conditions.
    * Surface has a fractal dimension

### Let's take a look at how the chaotic behavior actuall emerges in this system.

    We are using not flatenning our plot axes here and thus axes is a $3 \times 1$ matrix.

```python
F = [1.35, 1.44, 1.465]
θ0 = 0.2
params = ℓ,θ0,ω0,γ,ΩD

fig, axes = plt.subplots(3,1,sharex=True, sharey=True, squeeze=False, figsize=(6,10))
for i, ax in enumerate(axes.flat):
        θ,ω = euler(t,F[i],*params)
        ax.plot(t, θ, label=FD_label(F[i]))
        ax.legend(loc="lower left", frameon=True, prop={'size':16})

        # set axis labels
        ax.set_ylabel('θ(t) [rad]')

axes[-1,0].set_xlabel('Time [s]')
```

 * As we increase the driving force, we observe that for special values of $F_\mathrm{D}$, the period doubles.
 * Periodic motion with frequencies $\Omega_\mathrm{D}/2, \Omega_\mathrm{D}/4 \ldots$
 * Striking phenomenon! Usually in non-linear systems we expect the appearance of higher harmonics: frequencies that are integer multiples of the driving frequency.

We can systematicaly study this effect via a **Bifurcation Diagram**.

 1. Study the angle $\theta$ as a function of time.
 2. After waiting until all transients have died away, we plot $\theta$ at times that are in phase with the driving force.
 3. Repeat for different values of the driving force.

## Bifurcation Diagram

We want to analyze the long-time behavior of the system after all trasients have died out as a function of $F_\mathrm{D}$

```python
run = False
F = np.arange(0,1.5,0.0025)
if run:
    θ = np.zeros([len(longt[10*pstep::pstep]),len(F)])
    for i,cF in enumerate(F):
        th,ω = euler(longt,cF,*params)
        θ[:,i] = th[10*pstep::pstep]
    np.savetxt('../data/theta.dat',θ)
else:
    θ = np.loadtxt('../data/theta.dat')
```

```python
plt.figure(figsize=(7,7))
pF = np.ones_like(θ[:,0])
for i,cF in enumerate(F):
    u = np.unique(θ[:,i])
    pF = cF*np.ones_like(u)
    plt.plot(pF, u, marker='o', ms=3, linestyle='None', mfc=colors[1], mew=0, mec='None', alpha=1,rasterized=True)

plt.ylabel('θ [rad]')
plt.xlabel(r'$F_\mathrm{D}\; [\mathrm{s}^{-2}]$')
plt.xlim(1.34,1.5);
plt.ylim(0,π);
```

### The Onset of Chaos

 * The bifurcation diagram gives us a feeling for how chaotic behavior is approached: a cascade of period doublings.
 * Is this behavior universal? We have yet to discuss why this doubling happens.
 * Define $F_n$ as the value of some parameter where the $n^{\mathrm{th}}$ period doubling occurs.

$$\delta_n = \frac{F_n - F_{n-1}}{F_{n+1}-F_n}$$

What happens when $n\to\infty$? We find a number called the **Universal Feigenbaum-$\delta$**

$$\lim_{n\to\infty} \delta_n \to 4.699\ldots$$

```python

```


# Computational Astrophysics
## Linear Systems of Equations

---
## Eduard Larrañaga

Observatorio Astronómico Nacional\
Facultad de Ciencias\
Universidad Nacional de Colombia

---

### About this notebook

In this notebook we present some of the techniques to find the roots of a function.

---

A system of linear equations can be written in matrix form

\begin{equation}
A \mathbf{x} = \mathbf{b}
\end{equation}

where, $A$ is a real $n\times n$ matrix with coefficients $a_{ij}$.
$\mathbf{b}$ is a given real vector and $\mathbf{x}$ is a vector with
$n$ unknown values.

* If $\det{A} = |A| \ne 0$ and $\mathbf{b} \ne 0$, the  LSE has a unique solution
\begin{equation}
\mathbf{x} = A^{-1} \mathbf{b}\,\,,
\end{equation}
where $A^{-1}$ is the inverse of $A$ with $AA^{-1}=A^{-1}
A=\mathbf{I}$.

* If $\det{A} = 0$, the equations either
have no solution or an infinite number of solutions.

---
## Matrix Inversion Method

The inverse of a matrix $A$ is given by

\begin{equation}
A^{-1} = \frac{1}{|A|} \,\, \underbrace{\mathrm{adj}{A}}_\text{adjugate}\,\,.
\end{equation}

the adjugate of $A$ is the transpose of $A$'s cofactor matrix $C$:

\begin{equation}
\mathrm{adj}{A} = C^T\,\,.
\end{equation}

So the problem becomes finding $C$ and $\det A$.

This method scales in complexity with $n!$, where n is the number of rows/columns of $A$.

---
## Upper-diagonal Matrix

Consider a linear system with an upper-diagonal matrix,

\begin{equation}
A\mathbf{x} = \begin{pmatrix}
a_{11}&a_{12}&a_{13}\\
0 &a_{22}&a_{23}\\
0 &0 &a_{33} \end{pmatrix}\begin{pmatrix}x_1\\x_2\\x_3\end{pmatrix}
= \begin{pmatrix}b_1\\b_2\\b_3\end{pmatrix}\,\,.
\end{equation}

This LSE is solved trivially by simple back-substitution:

\begin{equation}
x_3 = \frac{b_3}{a_{33}},\hspace{0.3cm}x_2 = \frac{1}{a_{22}}(b_2 - a_{23} x_3), \hspace{0.3cm} x_1 = \frac{1}{a_{11}} ( b_1 - a_{12} x_2 - a_{13} x_3).
\end{equation}

This is implemented in a simple way for a $n\times n$ system,

```python
import numpy as np

def backsubs(A,b):
    '''
    ------------------------------------------
    backsubs(A,b)
    ------------------------------------------
    Returns the solution fo the Linear System
    A x = b
    where
    A: nxn upper diagonal matrix
    b: n vector

    Arguments:
    A: upper-diagonal numpy array of size nxn
    b: numpy array of size n
    ------------------------------------------
    '''
    n = len(b)
    m = n-1
    # Create an empty vector for the solution
    x = np.zeros(n)

    # Main loop for back-substitution
    for i in range(m,-1,-1):
        xterms = 0
        for j in range(i+1,n):
            xterms = xterms + A[i,j]*x[j]
        x[i] = 1/A[i,i]*(b[i] - xterms)
    return x

```

**Example**

We will solve the LSE

\begin{equation}
\begin{pmatrix}
5 & 3 & 4 & 2\\
0 & -1 & 7 & 3 \\
0 & 0 & 5 & 4 \\
0 & 0 & 0 & 3
\end{pmatrix}
\begin{pmatrix}
x_1 \\
x_2 \\
x_3 \\
x_4
\end{pmatrix} =
\begin{pmatrix}
3 \\
14 \\
12 \\
6
\end{pmatrix}
\end{equation}

```python
A = np.array([[5., 3., 4., 2.],
              [0., -1., 7., 3.],
              [0., 0., 5., 4.],
              [0., 0., 0., 3.]])

b = np.array([3., 14., 12., 6.])

x = backsubs(A,b)
x
```

    array([ 0.6, -2.4,  0.8,  2. ])

Lets probe that the solution is correct,

```python
np.dot(A,x)
```

    array([ 3., 14., 12.,  6.])

---
## Cramer's Rule

According to Cramer's rule the solution to the LSE is

\begin{equation}
x_i = \frac{\det A_i}{\det A}\,\,,
\end{equation}

where $A_i$ is the matrix formed from $A$ by replacing its $i$-th
column by the column vector $\mathbf{b}$.

Cramer's rule is more efficient than matrix inversion because it scales with $n^3$, so is more efficient
for large matrixes and has comparable efficiency to direct methods
such as Gauss Elimination.

### Example.

Consider the system

\begin{equation}
\begin{pmatrix}
5&3&4\\
2&1&5\\
5&4&1
\end{pmatrix}
\begin{pmatrix}
x_1\\ x_2\\ x_3
\end{pmatrix}
= \begin{pmatrix}3\\4\\2\end{pmatrix}\,\,.
\end{equation}

The determinant of the matrix $A$ is

\begin{equation}
\det A = -14.
\end{equation}

Therefore we will use Cramer's Rule to solve. The solution for variable $x_1$ is given by

\begin{equation}
x_1 = \frac{\det A_1}{\det A}\,\,,
\end{equation}

where $A_1$ is the matrix formed from $A$ by replacing its first
column by the column vector $\mathbf{b}$, i.e.

\begin{equation}
A_1 = \begin{pmatrix}
3&3&4\\
4&1&5\\
2&4&1
\end{pmatrix}.
\end{equation}

This matrix has $\det A_1 = 17$ and therefore

\begin{equation}
x_1 =- \frac{17}{14}.
\end{equation}

The solution for variable $x_2$ is given by

\begin{equation}
x_2 = \frac{\det A_2}{\det A}\,\,,
\end{equation}

where $A_2$ is

\begin{equation}
A_2 = \begin{pmatrix}
5&3&4\\
2&4&5\\
5&2&1
\end{pmatrix}.
\end{equation}

This matrix has $\det A_2 = -25$ and therefore

\begin{equation}
x_2 = \frac{25}{14}.
\end{equation}

The solution for variable $x_3$ is given by

\begin{equation}
x_3 = \frac{\det A_3}{\det A}\,\,,
\end{equation}

where $A_3$ is

\begin{equation}
A_3 = \begin{pmatrix}
5&3&3\\
2&1&4\\
5&4&2
\end{pmatrix}.
\end{equation}

This matrix has $\det A_3 = -13$ and therefore

\begin{equation}
x_3 = \frac{13}{14}.
\end{equation}

The complete solution is

\begin{equation}
\textbf{x}
= \frac{1}{14} \begin{pmatrix} -17 \\ 25 \\ 13 \end{pmatrix}\,\,.
\end{equation}

---

Now lets implement this example computationally.

```python
import numpy as np

# Defning matrix A
A = np.array([[5., 3., 4.],
              [2., 1., 5.],
              [5., 4., 1.]])

A
```

    array([[5., 3., 4.],
           [2., 1., 5.],
           [5., 4., 1.]])

We will use the function `numpy.linalg.det()` to calculate the determinants.

```python
detA = np.linalg.det(A)
detA
```

    -13.999999999999996

Since this determinant is non-zero, we define the vector $\textbf{b}$ and implement Cramer's rule,

```python
import CramersRule as cr
import numpy as np

A = np.array([[5., 3., 4.],
              [2., 1., 5.],
              [5., 4., 1.]])
b = np.array([3., 4., 2.])

x = cr.CramersRule(A,b)
x
```

    array([-1.21428571,  1.78571429,  0.92857143])

When the matrix is very large or when the values of the coefficients are very small, the function `numpy.linalg.det()` may give overflow or underflow errors. In these cases, it may be possible to calculate the determinant of the matrix using the function `numpy.linalg.slogdet()`. More information about this function at https://numpy.org/doc/stable/reference/generated/numpy.linalg.slogdet.html#numpy.linalg.slogdet

---
## Gauss Elimination

The Gauss algorithm consists of a series of steps to bring a $n \times n$ matrix into the  upper triangular form.

1. Sort the rows of $A$ so that the diagonal coefficient $a_{ii}$
  (called \emph{the pivot}) of row $i$ (for all $i$) is non-zero. If this
  is not possible, the LSE cannot be solved.

2. Replace the $j$-th equation with

\begin{equation}
-\frac{a_{j1}}{a_{11}} \times (\text{1-st equation}) + (j\text{-th equation})\,\,,
\end{equation}

where $j$ runs from $2$ to $n$. This will zero-out column 1 for $i>1$.

3. Repeat the previous step, but starting with the next row down and with
$j >$ (current row number). The current row be row $k$. Then we must replace rows $j$, $j > k$, with

  \begin{equation}
    -\frac{a_{jk}}{a_{kk}} \times (k\text{-th equation}) + (j\text{-th equation})\,\,,
  \end{equation}

where $k < j \le n$.

4. Repeat (3) until all rows have been reduced and the matrix is in upper
triangular form.

5. Back-substitute to find $\mathbf{x}$.

```python
import numpy as np
from GaussElimination import *

def backsubs(A,b):
    '''
    ------------------------------------------
    backsubs(A,b)
    ------------------------------------------
    Returns the solution fo the Linear System
    A x = b
    where
    A: nxn upper diagonal matrix
    b: n vector

    Arguments:
    A: upper-diagonal numpy array of size nxn
    b: numpy array of size n
    ------------------------------------------
    '''
    n = len(b)
    m = n-1
    # Create an empty vector for the solution
    x = np.zeros(n)

    # Main loop for back-substitution
    for i in range(m,-1,-1):
        xterms = 0
        for j in range(i+1,n):
            xterms = xterms + A[i,j]*x[j]
        x[i] = 1/A[i,i]*(b[i] - xterms)
    return x

A = np.array([[5., 3., 4.], [2., 1., 5.], [5., 4., 1.]])
b = np.array([3., 4., 2.])

A,b = GaussElim(A,b)

x = backsubs(A,b)
x
```

    array([-1.21428571,  1.78571429,  0.92857143])

```python
GaussElim?
```

---
## Tri-Diagonal Systems

Consider the following $4 \times 4 $ LSE, represented by a tri-diagonal matrix,

\begin{equation}
\begin{pmatrix}
b_1&c_1&0&0\\
a_1&b_2&c_2&0\\
0&a_2&b_3&c_3\\
0&0&a_3&b_4
\end{pmatrix}
\begin{pmatrix}
x_1\\ x_2\\ x_3 \\ x_4
\end{pmatrix}
= \begin{pmatrix}f_1\\f_2\\f_3\\f_4\end{pmatrix}\,\,.
\end{equation}

Tri-diagonal LSEs like this one are simple to solve using Gaussian elimination. When applying a forward elimination procedure, we obtain

\begin{equation}
\begin{pmatrix}
1&c_1/d_1&0&0\\
0&1&c_2/d_2&0\\
0&0&1&c_3/d_3\\
0&0&0&1
\end{pmatrix}
\begin{pmatrix}
x_1\\ x_2\\ x_3 \\ x_4
\end{pmatrix}
= \begin{pmatrix}y_1\\y_2\\y_3\\y_4\end{pmatrix}\,\,,
\end{equation}

where

\begin{equation}
\begin{array}{lcl}
d_1 &=& b_1\,,\\
d_2 &=& b_2 - a_{1} c_{1}/d_{1}\,,\\
d_3 &=& b_3 - a_{2} c_{2}/d_{2}\,,\\
d_4 &=& b_4 - a_{3} c_{3}/d_{3}\,,\\
\end{array}\hspace{2cm}
\begin{array}{lcl}
y_1 &=& f_1/d_1\,,\\
y_2 &=& (f_2 - y_1 a_1)/d_2\,,\\
y_3 &=& (f_3 - y_2 a_2)/d_3\,,\\
y_4 &=& (f_4 - y_3 a_3)/d_4\,.\\
\end{array}
\end{equation}

Then, using back-substitution, we obtain the solution

\begin{equation}
\begin{array}{lcl}
x_1 &=& y_1 - x_2 c_1 / d_1\,,\\
x_2 &=& y_2 - x_3 c_2 / d_2\,,\\
x_3 &=& y_3 - x_4 c_3 / d_3\,,\\
x_4 &=& y_4\,.
\end{array}
\end{equation}

---

In a general case, for a $n\times n$ system, we can write the solving algorithm as

1. **Forward Elimination.**
 * At the first step: $d_1 = b_1$ and $y_1 = f_1 / d_1$.

 * At the $k$-th step:
  \begin{equation*}
    \begin{aligned}
      d_k & = b_k - a_{k-1} c_{k-1} / d_{k-1}\,\,,\\
      y_k & = (f_k - y_{k-1} a_{k-1}) / d_k\,\,.
    \end{aligned}
  \end{equation*}

2. **Backward Substitution.** The $x_k$ are given by

  \begin{equation*}
    \begin{aligned}
      x_n &= y_n\,\,,\\
      x_{k-1} &= y_{k-1}- x_k c_{k-1} / d_{k-1}\,\,.
    \end{aligned}
  \end{equation*}

```python

```


# Eigenvalues and eigenfunctions

Computing the eigenvectors $\psi(x)$ and eigenvalues $E$ of some Hamiltonian $H$ belongs to the key problems of quantum mechanics. For the one-dimensional Schrödinger equation we have to solve
\begin{equation}
E\psi(x) = H\psi(x) = -\frac{1}{2}\frac{\mathrm{d}^2\psi(x)}{\mathrm{d} x^2} + V(x)\psi(x)
\end{equation}for some given potential $V(x)$. Analytical solutions, however, are scare goods. Thus, one has to rely on approximations or numerical methods.

In principle, numerical approaches are straight forward. Just replace the Hermitian Hamiltonian by some Hermitian matrix by discretizing the differential operators, e.g., by finite differences or pseudospectral methods. Then utilize the computational routines of your favorite computational linear algebra package to solve the eigenvalue problem. Often  these packages are equipped with special routines that can take advantage of hermiticity. Below I give an example, which calculates the eigenvalues and the eigenvectors of the harmonic oscillator with $V(x)=x^2/2$ in Python. Differentials are approximated via 2nd order finite differences. Numerical results for the discrete Hamiltonian matrix are compared to the known eigenvalues and eigenvectors. One clearly sees that the small eigenvalues are well reproduced. The discrete Hamiltonian features, however, also large eigenvalues and eigenstates which are very different from the eigenvalues and eigenvectors of the related contiguous problem.

Our naive approach works reasonably well. It has, however, a serious drawback. It becomes computationally very demanding as soon as one tries to tackle two- or three-dimensional problems because the matrices become easily very large. So large that one can not compute its eigenvalues and eigenvectors in a reasonable amount of time. Even worse, the matrices may become so large that they no longer fit in the computers main memory. Fortunately, one is usually not interested in all eigenvalues, knowing some, e.g., a few of the smallest ones, might be sufficient. In this case the Lanczos algorithm is often the method of choice. This algorithm basically constructs from the original large Hermitian matrix a smaller tridiagonal symmetric matrix that has eigenvalues that are close to some eigenvalues of the larger original matrix. A further nice feature of the Lanczos algorithm is that it does not require to store the matrix explicitly in memory, a routine that computes matrix-vector products of the matrix is sufficient.

```python
#!/usr/bin/env python

# import useful modules
import matplotlib
from math import factorial
from numpy import *
from pylab import *
from numpy.polynomial.hermite import *

# use LaTeX, choose nice some looking fonts and tweak some settings
matplotlib.rc('font', family='serif')
matplotlib.rc('font', size=16)
matplotlib.rc('legend', fontsize=16)
matplotlib.rc('legend', numpoints=1)
matplotlib.rc('legend', handlelength=1.5)
matplotlib.rc('legend', frameon=False)
matplotlib.rc('xtick.major', pad=7)
matplotlib.rc('xtick.minor', pad=7)
matplotlib.rc('lines', lw=1.5)
matplotlib.rc('text', usetex=True)
matplotlib.rc('text.latex',
              preamble=[r'\usepackage[T1]{fontenc}',
                        r'\usepackage{amsmath}',
                        r'\usepackage{txfonts}',
                        r'\usepackage{textcomp}'])

def V(x):
    return 0.5*x**2

close('all')
figure(figsize=(6, 4.5))

# define grid
N=512                   # number of grid points
x0, x1=-10., 10.        # grid boundaries
dx=(x1-x0)/(N-1)        # grid spacing
x=linspace(x0, x1, N)   # grid points
# setup Hamiltonian
H_kin=zeros([N, N])
for k in range(0, N):
    if k>0:
        H_kin[k-1, k]=-0.5/dx**2
    H_kin[k, k]=1./dx**2
    if k+1<N:
        H_kin[k+1, k]=-0.5/dx**2
H_pot=diag(V(x))
H=H_kin+H_pot
# compute eigenvalues
E=eigvalsh(H)
# plot first eigenvalues
n=100
plot(array(range(0, n)), array(range(0, n))+0.5, 'rx', mew=1.25, ms=8,
     label='harmonic oscillator eigenvalues')
plot(array(range(0, n)), sort(E)[0:n], '.', mew=1.25, ms=8, mfc='none', mec='b',
     label='eigenvalues of the finite differences Hamiltonian')
xlabel('index')
ylabel('energy')
legend(bbox_to_anchor=(0., 1.02, 1., .102), loc=3,
       borderaxespad=0.)
gca().set_position((0.15, 0.15, 0.8, 0.625))
show()
savefig('eigenvalues.png')

# compute eigenvectors
E, psi_E=eigh(H)
# plot some eigenvectors
clf()
for n in [0, 1, 2, 20]:
    c=zeros(n+1)
    c[n]=1
    plot(x, psi_E[:, n]/sqrt(dx),
         label='$n=%i$' % n)
legend(ncol=2)
xlabel(r'$x$')
ylabel(r'$\Psi_n(x)$')
gca().set_ylim(-0.8, 1.2)
gca().set_position((0.15, 0.15, 0.8, 0.75))
show()
savefig('eigenvectors.png')

# plot error of some eigenvectors
clf()
for n in [0, 1, 2, 20]:
    c=zeros(n+1)
    c[n]=1
    plot(x,
         (psi_E[:, n]/sqrt(dx))**2 -
         (hermval(x, c)*exp(-0.5*x**2)/sqrt(sqrt(pi)*2**n*factorial(n)))**2,
         label='$n=%i$' % n)
legend(ncol=2)
xlabel(r'$x$')
ylabel(r'$|\Psi_n(x)|^2-|\Psi_{n,\mathrm{exact}}(x)|^2$')
gca().set_ylim(-0.004, 0.006)
gca().set_position((0.2, 0.15, 0.75, 0.75))
show()
```

Errors of the squared absolute value of some eigenvectors of the harmonic oszillator.

```python

```