# Theano 实例：更复杂的网络

```python
import theano
import theano.tensor as T
import numpy as np
from load import mnist
from theano.sandbox.rng_mrg import MRG_RandomStreams as RandomStreams

srng = RandomStreams()

def floatX(X):
    return np.asarray(X, dtype=theano.config.floatX)
```

    Using gpu device 1: Tesla C2075 (CNMeM is disabled)

上一节我们用了一个简单的神经网络来训练 MNIST 数据，这次我们使用更复杂的网络来进行训练，同时加入 `dropout` 机制，防止过拟合。

这里采用比较简单的 `dropout` 机制，即将输入值按照一定的概率随机置零。

```python
def dropout(X, prob=0.):
    if prob > 0:
        X *= srng.binomial(X.shape, p=1-prob, dtype = theano.config.floatX)
        X /= 1 - prob
    return X
```

之前我们采用的的激活函数是 `sigmoid`，现在我们使用 `rectify` 激活函数。

这可以使用 `T.nnet.relu(x, alpha=0)` 来实现，它本质上相当于：`T.switch(x > 0, x, alpha * x)`，而 `rectify` 函数的定义为：

$$
\text{rectify}(x) = \left\{
\begin{aligned}
x, & \ x > 0 \\
0, & \ x < 0
\end{aligned}\right.
$$

之前我们构造的是一个单隐层的神经网络结构，现在我们构造一个双隐层的结构即“输入-隐层1-隐层2-输出”的全连接结构。

$$
\begin{aligned}
& h_1 =  \text{rectify}(W_{h_1} \ x) \\
& h_2 =  \text{rectify}(W_{h_2} \ h_1) \\
& o =  \text{softmax}(W_o h_2)
\end{aligned}
$$

`Theano` 自带的 `T.nnet.softmax()` 的 GPU 实现目前似乎有 bug 会导致梯度溢出的问题，因此自定义了 `softmax` 函数：

```python
def softmax(X):
    e_x = T.exp(X - X.max(axis=1).dimshuffle(0, 'x'))
    return e_x / e_x.sum(axis=1).dimshuffle(0, 'x')

def model(X, w_h1, w_h2, w_o, p_drop_input, p_drop_hidden):
    """
    input:
        X:             input data
        w_h1:          weights input layer to hidden layer 1
        w_h2:          weights hidden layer 1 to hidden layer 2
        w_o:           weights hidden layer 2 to output layer
        p_drop_input:  dropout rate for input layer
        p_drop_hidden: dropout rate for hidden layer
    output:
        h1:    hidden layer 1
        h2:    hidden layer 2
        py_x:  output layer
    """
    X = dropout(X, p_drop_input)
    h1 = T.nnet.relu(T.dot(X, w_h1))

    h1 = dropout(h1, p_drop_hidden)
    h2 = T.nnet.relu(T.dot(h1, w_h2))

    h2 = dropout(h2, p_drop_hidden)
    py_x = softmax(T.dot(h2, w_o))
    return h1, h2, py_x
```

随机初始化权重矩阵：

```python
def init_weights(shape):
    return theano.shared(floatX(np.random.randn(*shape) * 0.01))

w_h1 = init_weights((784, 625))
w_h2 = init_weights((625, 625))
w_o = init_weights((625, 10))
```

定义变量：

```python
X = T.matrix()
Y = T.matrix()
```

定义更新的规则，之前我们使用的是简单的 SGD，这次我们使用 RMSprop 来更新，其规则为：
$$
\begin{align}
MS(w, t) & = \rho MS(w, t-1) + (1-\rho) \left(\left.\frac{\partial E}{\partial w}\right|_{w(t-1)}\right)^2 \\
w(t) & = w(t-1) - \alpha \left.\frac{\partial E}{\partial w}\right|_{w(t-1)} / \sqrt{MS(w, t)}
\end{align}
$$

```python
def RMSprop(cost, params, accs, lr=0.001, rho=0.9, epsilon=1e-6):
    grads = T.grad(cost=cost, wrt=params)
    updates = []
    for p, g, acc in zip(params, grads, accs):
        acc_new = rho * acc + (1 - rho) * g ** 2
        gradient_scaling = T.sqrt(acc_new + epsilon)
        g = g / gradient_scaling
        updates.append((acc, acc_new))
        updates.append((p, p - lr * g))
    return updates
```

训练函数：

```python
# 有 dropout，用来训练
noise_h1, noise_h2, noise_py_x = model(X, w_h1, w_h2, w_o, 0.2, 0.5)
cost = T.mean(T.nnet.categorical_crossentropy(noise_py_x, Y))
params = [w_h1, w_h2, w_o]
accs = [theano.shared(p.get_value() * 0.) for p in params]
updates = RMSprop(cost, params, accs, lr=0.001)
# 训练函数
train = theano.function(inputs=[X, Y], outputs=cost, updates=updates, allow_input_downcast=True)
```

预测函数：

```python
# 没有 dropout，用来预测
h1, h2, py_x = model(X, w_h1, w_h2, w_o, 0., 0.)
# 预测的结果
y_x = T.argmax(py_x, axis=1)
predict = theano.function(inputs=[X], outputs=y_x, allow_input_downcast=True)
```

训练：

```python
trX, teX, trY, teY = mnist(onehot=True)

for i in range(50):
    for start, end in zip(range(0, len(trX), 128), range(128, len(trX), 128)):
        cost = train(trX[start:end], trY[start:end])
    print "iter {:03d} accuracy:".format(i + 1), np.mean(np.argmax(teY, axis=1) == predict(teX))
```

    iter 001 accuracy: 0.943
    iter 002 accuracy: 0.9665
    iter 003 accuracy: 0.9732
    iter 004 accuracy: 0.9763
    iter 005 accuracy: 0.9767
    iter 006 accuracy: 0.9802
    iter 007 accuracy: 0.9795
    iter 008 accuracy: 0.979
    iter 009 accuracy: 0.9807
    iter 010 accuracy: 0.9805
    iter 011 accuracy: 0.9824
    iter 012 accuracy: 0.9816
    iter 013 accuracy: 0.9838
    iter 014 accuracy: 0.9846
    iter 015 accuracy: 0.983
    iter 016 accuracy: 0.9837
    iter 017 accuracy: 0.9841
    iter 018 accuracy: 0.9837
    iter 019 accuracy: 0.9835
    iter 020 accuracy: 0.9844
    iter 021 accuracy: 0.9837
    iter 022 accuracy: 0.9839
    iter 023 accuracy: 0.984
    iter 024 accuracy: 0.9851
    iter 025 accuracy: 0.985
    iter 026 accuracy: 0.9847
    iter 027 accuracy: 0.9851
    iter 028 accuracy: 0.9846
    iter 029 accuracy: 0.9846
    iter 030 accuracy: 0.9853
    iter 031 accuracy: 0.985
    iter 032 accuracy: 0.9844
    iter 033 accuracy: 0.9849
    iter 034 accuracy: 0.9845
    iter 035 accuracy: 0.9848
    iter 036 accuracy: 0.9868
    iter 037 accuracy: 0.9864
    iter 038 accuracy: 0.9866
    iter 039 accuracy: 0.9859
    iter 040 accuracy: 0.9857
    iter 041 accuracy: 0.9853
    iter 042 accuracy: 0.9855
    iter 043 accuracy: 0.9861
    iter 044 accuracy: 0.9865
    iter 045 accuracy: 0.9872
    iter 046 accuracy: 0.9867
    iter 047 accuracy: 0.9868
    iter 048 accuracy: 0.9863
    iter 049 accuracy: 0.9862
    iter 050 accuracy: 0.9856


# Problema prototipo de cuarto orden

#### Alejandro Alonso Membrilla y Pilar Navarro Ramírez

```python
import numpy as np # Importamos el módulo NumPy con el pseudónimo np
import sympy as sp # Importamos el módulo SymPy con el pseudónimo sp
import math
from sympy import sin, cos
%matplotlib inline
import matplotlib.pyplot as plt  # plt será el pseudónimo correspondiente

t,z,xL,xR,yL,yR = sp.symbols('t,z,xL,xR,yL,yR')
```

```python
# Base de elementos finitos de Hermite
def lh1(t):
    """primera func. de base del E.F. de Hermite P3 en 1D"""
    return (1+2*t) * (t-1)**2

def lh2(t):
    """segunda func. de base del E.F. de Hermite P3 en 1D"""
    return t * (t-1)**2

def lh3(t):
    """tercera func. de base del E.F. de Hermite P3 en 1D"""
    return -t**2 * (2*t-3)

def lh4(t):
    """cuarta func. de base del E.F. de Hermite P3 en 1D"""
    return t**2 * (t-1)

# def dlh1(t):
#     return 2*(t - 1)**2 + (2*t - 2)*(2*t + 1)

# def dlh2(t):
#     return t*(2*t - 2) + (t - 1)**2

# def dlh3(t):
#     return -2*t**2 - 2*t*(2*t - 3)

# def dlh4(t):
#     return t**2 + 2*t*(t - 1)

def Finv(z,a,b):
    """afinidad entre cada subintervalo de la partición
        y el intervalo unidad"""
    return (z-a)/(b-a)

def dFinv(z,a,b):
    """ derivada de la afinidad Finv"""
    return 1/(b-a)
```

```python
def wi0(z,x,i):
    """funciones de base del E.F. de Hermite P3 unidimensional
    que controlan el valor de la función"""
    if (x[i-1]<=z)*(z<=x[i]):
        valor = lh1(Finv(z,x[i-1],x[i]))
    elif (x[i]<=z)*(z<=x[i+1]):
        valor = lh3(Finv(z,x[i-1],x[i]))
    return valor

def wi1(z,x,i):
    """funciones de base del E.F. de Hermite P3 unidimensional
    que controlan la derivada"""
    if (x[i-1]<=z)*(z<=x[i]):
        valor = lh2(Finv(z,x[i-1],x[i]))*(x[i]-x[i-1])
    elif (x[i]<=z)*(z<=x[i+1]):
        valor = lh4(Finv(z,x[i-1],x[i]))*(x[i]-x[i-1])
    return valor

def w00(z,x,i):
    """función de base del E.F. de Hermite P3 unidimensional
    para el valor de la función en el extremo izquierdo"""
    if (x[0]<=z)*(z<=x[1]):
        return lh3(Finv(z,x[i-1],x[i]))
    else:
        return 0

def w01(z,x,i):
    """función de base del E.F. de Hermite P3 unidimensional
    para la derivada de la función en el extremo izquierdo"""
    if (x[0]<=z)*(z<=x[1]):
        return lh4(Finv(z,x[i-1],x[i]))*(x[i]-x[i-1])
    else:
        return 0

def wn0(z,x,i):
    """funciones de base del E.F. de Hermite P3 unidimensional
    para el valor de la función en el extremo derecho"""
    if (x[i-1]<=z)*(z<=x[i]):
        lh1(Finv(z,x[i-1],x[i]))
    else:
        return 0

def wn1(z,x,i):
    """funciones de base del E.F. de Hermite P3 unidimensional
    para la derivada de la función en el extremo derecho"""
    if (x[i-1]<=z)*(z<=x[i]):
        lh2(Finv(z,x[i-1],x[i]))
    else:
        return 0
```

```python
a = 0;   b = 10;

nx= 10;      x = np.linspace(a,b,nx+1);
nxx = 1000;  xx = np.linspace(a,b,nxx);
```

## Ejercicio 2. Matriz de rigidez

```python
func_base = [lh1,lh2,lh3,lh4]
Rk = [
        [ sp.integrate(sp.diff(p_i(Finv(z,xL,xR)),z,2)*sp.diff(p_j(Finv(z,xL,xR)),z,2),[z,xL,xR])
         for p_j in func_base ]
        for p_i in func_base
    ]

Rk = sp.simplify( sp.Matrix(Rk) )
Rk.subs({xL:0,xR:sp.Symbol('h_k')})
```

$\displaystyle \left[\begin{matrix}\frac{12}{h_{k}^{3}} & \frac{6}{h_{k}^{3}} & - \frac{12}{h_{k}^{3}} & \frac{6}{h_{k}^{3}}\\\frac{6}{h_{k}^{3}} & \frac{4}{h_{k}^{3}} & - \frac{6}{h_{k}^{3}} & \frac{2}{h_{k}^{3}}\\- \frac{12}{h_{k}^{3}} & - \frac{6}{h_{k}^{3}} & \frac{12}{h_{k}^{3}} & - \frac{6}{h_{k}^{3}}\\\frac{6}{h_{k}^{3}} & \frac{2}{h_{k}^{3}} & - \frac{6}{h_{k}^{3}} & \frac{4}{h_{k}^{3}}\end{matrix}\right]$

## Ejercicio 3. Vector de cargas

```python
C = sp.Symbol('C')
dk = sp.simplify(sp.Matrix([ C*sp.integrate(p_i(Finv(z,xL,xR)),[z,xL,xR])
                             for p_i in func_base]
                          ))
dk.subs({xL:0,xR:sp.Symbol('h_k')})
```

$\displaystyle \left[\begin{matrix}\frac{C h_{k}}{2}\\\frac{C h_{k}}{12}\\\frac{C h_{k}}{2}\\- \frac{C h_{k}}{12}\end{matrix}\right]$


# Tutorial 03: Instationary Parabolic Equations

In this tutorial we extend the elliptic problem from tutorial 01 to the
time dependent case. Following the method of lines a general approach to time-dependent
problems is presented.
This tutorial depends on tutorial 01.

# PDE Problem

In this tutorial we consider the following problem:
\begin{align}\label{equ:problem}
\partial_t u -\Delta u + q(u) &= f &&\text{in $\Omega\times\Sigma$},\\
u &= g &&\text{on $\Gamma_D\subseteq\partial\Omega$},\\
-\nabla u\cdot \nu &= j &&\text{on $\Gamma_N=\partial\Omega\setminus\Gamma_D$},\\
u &= u_0 &&\text{at $t=0$}. \label{equ:problemEnd}
\end{align}
This problem is a straightforward extension of the stationary
problem solved in tutorial 01.
The parameter functions $f$, $g$, $j$ may now also depend on time
and (with some restrictions) the subdivision into Dirichlet and Neumann boundary
can be time-dependent as well. The initial
condition $u_0$ is a function of $x\in\Omega$.

Multiplying with a test function and integrating in space
results in the following weak formulation [[8]](#ern):
Find $u\in L_2(t_0,t_0+T;u_g+V(t))$:
\begin{equation}
\frac{d}{dt} \int_\Omega u v \,dx+ \int_\Omega \nabla u \cdot \nabla v
+ q(u) v - f v \, dx + \int_{\Gamma_N} jv \, ds = 0 \qquad
\begin{array}{l}
\forall v \in V(t),\\
t \in \Sigma,
\end{array}
\label{eq:WeakForm}
\end{equation}
where $V(t) = \{v\in H^1(\Omega)\,:\, \text{$v=0$ on $\Gamma_D(t)$}\}$
and $H^1(\Omega)\ni u_g(t)|_{\Gamma_D}=g$. This can be written in a more compact way
with residual forms as follows:
\begin{equation*}
\frac{d}{dt} m^{\text{L2}}(u,v) + r^{\text{NLP}}(u,v) = 0 \quad \forall v \in V(t), t \in \Sigma.
\end{equation*}
where
\begin{equation*}
m^{\text{L2}}(u,v) = \int_\Omega u v \,dx
\end{equation*}
is the new temporal residual form, actually the $L_2$ inner product, and
\begin{equation*}
r^{\text{NLP}}(u,v) = \int_\Omega \nabla u \cdot \nabla v + (q(u)-f)v\,dx + \int_{\Gamma_N} jv\,ds ,
\end{equation*}
is the spatial residual form known from tutorial 01.
Under suitable assumptions it can be shown that problem \eqref{eq:WeakForm} has
a unique solution, [[8]](#ern) for the linear case.

# Finite Element Method

In order to arrive at a fully discrete formulation we follow the method of lines
paradigm:

1. Choose a finite-dimensional test space $V_h(t)\subset V(t)$. Then \eqref{eq:WeakForm} results in a system of ordinary differential equations for the coefficients $z_j(t)$ in the expansion of $u_h(t)=\sum_{j=1}^n (z_j(t))_j \phi_j$.
2. Choose an appropriate method to integrate the system of ordinary differential equations (ODEs).

The finite-dimensional space we choose here is just the conforming
finite element space $V_h^{k,d}(\mathcal{T}_h,t)$ introduced in tutorial 01.
It may now depend also on time due to the time-dependent splitting of the boundary
in Dirichlet and Neumann part. Also the function $u_{h,g}(t)$ depends on time
and we have $u_h(t)\in U_h(t) = u_{h,g}(t) + V_h(t)$.

For the integration of the system of ODEs, subdivide the time interval into
not necessarily equidistant subintervals:
\begin{equation*}
\overline{\Sigma} = \{t^{0}\} \cup (t^0,t^1] \cup \ldots \cup (t^{N-1},t^N]
\end{equation*}
with $t^0=t_0$, $t^N=t_0+T$, $t^{k-1}<t^k$ for $1\leq k\leq N$ and
set  the time step to $\Delta t^k=t^{k+1}-t^k$.

One of the simplest ODE integrators is
the one-step-$\theta$ rule which reads:
\begin{equation}
\label{Eq:InstationaryProblem}
\begin{split}
&\text{Find $u_h^{k+1}\in U_h(t^{k+1})$ s.t.:}
\ \frac{1}{\Delta t_k}(m_h^\text{L2}(u_h^{k+1},v;t^{k+1})-m_h^\text{L2}(u_h^{k},v;t^{k})) + \\
&\hspace{25mm}\theta r_h^\text{NLP}(u_h^{k+1},v;t^{k+1}) + (1-\theta) r_h^\text{NLP}(u_h^{k},v;t^{k}) = 0
\quad \forall v\in V_h(t^{k+1}).
\end{split}
\end{equation}

Reordering terms shows that this method results in the solution
of a nonlinear system per time step which has the same structure as before:
\begin{equation*}
\text{Find $u_h^{k+1}\in U_h(t^{k+1})$ s.t.:}
\quad r_h^{\theta,k} (u_h^{k+1},v) + s_h^{\theta,k}(v) = 0
\quad \forall v\in V_h(t^{k+1}).
\end{equation*}
where
\begin{align*}
r^{\theta,k}_h(u,v) &= m_h^\text{L2}(u,v;t^{k+1})+\Delta t^k \theta r_h^\text{NLP}(u,v;t^{k+1}) ,\\
s^{\theta,k}_h(v) &= -m_h^\text{L2}(u^k_h,v;t^k) + \Delta t^k (1-\theta) r_h^\text{NLP}(u_h^k,v;t^k) .
\end{align*}
Implementation-wise the new residual form is comprised of a linear combination
of temporal and spatial residual forms.

The one step $\theta$ method results in the implicit Euler method for $\theta=1$,
the Crank-Nicolson method for $\theta=1/2$ and the explicit Euler method for $\theta=0$.
A large number of alternative methods are possible. In particular the class of
Runge-Kutta methods is introduced below.

## Runge-Kutta Methods in Shu-Osher Form

For the temporal and spatial residual forms $m_h$ and $r_h$, Runge-Kutta
methods can be written in the Shu-Osher form ([[9]](#ShuOsher88),[[10]](#gottlieb11)):

1. $u_h^{(0)} = u_h^{k}$.
2. For $i=1,\ldots,s\in\mathbb{N}$, find $u_h^{(i)}\in u_{h,g}(t^k+d_i \Delta t^k)
    + V_h(t^{k+1})$:
\begin{equation*}
\begin{split}
\sum\limits_{j=0}^{s} \biggl[a_{ij} m_h&\left(u_h^{(j)},v;t^k+d_j \Delta t^k\right) \\
&\qquad + b_{ij} \Delta t^k r_h\left(u_h^{(j)}, v;t^k+d_j \Delta t^k\right) \biggr] = 0
\qquad \forall v\in V_h(t^{k+1}).
\end{split}
\end{equation*}
3. $u_h^{k+1} = u_h^{(s)}$.

Here we assume that the same type of boundary condition
holds through the interval $(t^k,t^{k+1}]$. The parameter $s$ denotes
the number of stages of the scheme.
An $s$-stage scheme is given by the parameters
\begin{equation*}
A = \left[\begin{array}{ccc}
a_{10} & \ldots & a_{1s}\\
\vdots &  & \vdots\\
a_{s0} & \ldots & a_{ss}
\end{array}\right],
\quad B = \left[\begin{array}{ccc}
b_{10} & \ldots & b_{1s}\\
\vdots &  & \vdots\\
b_{s0} & \ldots & b_{ss}
\end{array}\right],
\quad d = \left(
d_{0}, \ldots, d_{s}
\right)^T.
\end{equation*}
*Explicit* schemes are characterized by $a_{ij} = 0$ for $j>i$ and $b_{ij}=0$ for $j\geq i$.
*Diagonally implicit* schemes are characterized by $a_{ij} = b_{ij}= 0$ for $j>i$.
Fully implicit schemes where $A$ and $B$ are full are not considered in PDELab.
Without loss of generality it can be assumed that $a_{ii}=1$. Moreover,
some diagonally implicit schemes, in particular all those implemented in PDELab right
now, satisfy $b_{ii}=b\,\forall i$. This means that in case of a linear problem the matrix is the
same in all steps and needs to be assembled only once.
PDELab allows you to add new schemes by specifying $A$, $B$ and $d$.
All explicit Runge-Kutta methods and most implicit Runge-Kutta methods
can be brought to the Shu-Osher form.
Some examples are:

 - One step $\theta$ scheme (introduced above):
\begin{equation*}
A = \left[\begin{array}{cc}
-1 & 1
\end{array}\right],
\quad B = \left[\begin{array}{cc}
1-\theta & \theta
\end{array}\right],
\quad d = \left(
0, 1
\right)^T.
\end{equation*}
Explicit/implicit Euler ($\theta\in\{0,1\}$), Crank-Nicolson ($\theta=1/2$).
 -  Heun's second order explicit method
\begin{equation*}
A = \left[\begin{array}{ccc}
-1 & 1 & 0\\
-1/2 & -1/2 & 1\\
\end{array}\right],
\quad B = \left[\begin{array}{ccc}
1 & 0 & 0\\
0 & 1/2 & 0\\
\end{array}\right],
\quad d = \left(
0, 1, 1
\right)^T.
\end{equation*}
 -  Alexander's two-stage second order strongly S-stable method \cite{alexander:77}:
\begin{equation*}
A = \left[\begin{array}{ccc}
-1 & 1 & 0\\
-1 & 0 & 1\\
\end{array}\right],
\quad B = \left[\begin{array}{ccc}
0 & \alpha     & 0\\
0 & 1-\alpha & \alpha\\
\end{array}\right],
\quad d = \left(
0, \alpha, 1
\right)^T
\end{equation*}
with $\alpha=1-\sqrt{2}/{2}$.
 -  Fractional step $\theta$ \cite{Meidner201545}, three stage second order strongly A-stable scheme:
\begin{equation*}
A = \left[\begin{array}{rrrr}
-1 & 1 & 0 & 0\\
0  & -1 & 1 & 0\\
0  & 0 & -1 & 1\\
\end{array}\right],
\quad B = \left[\begin{array}{rrrr}
\theta (1-\alpha) & \theta\alpha & 0 & 0\\
0 & \theta'\alpha & \theta' (1-\alpha) & 0\\
0 & 0 & \theta(1-\alpha) & \theta\alpha
\end{array}\right],
\quad d = \left(
0, \theta, 1-\theta, 1
\right)^T
\end{equation*}
with $\theta=1-\sqrt{2}/{2}$, $\alpha=2\theta$, $\theta' = 1-2\theta = 1-\alpha = \sqrt{2}-1$.
Note also that $\theta\alpha= \theta' (1-\alpha) = 2\theta^2$.

## Explicit Time Stepping Schemes

Considering the case of the explicit Euler method ($\theta=0$) in \eqref{Eq:InstationaryProblem}
results in the problem: Find $u_h^{k+1}\in U_h(t^{k+1})$ s.t.:
\begin{equation*}
 m_h^\text{L2}(u_h^{k+1},v;t)-m_h^\text{L2}(u_h^{k},v;t) +
\Delta t^k r_h^\text{NLP}(u_h^{k},v;t) = 0
\quad \forall v\in V_h(t^{k+1}).
\end{equation*}
For certain spatial schemes, e.g. finite volume or discontinuous Galerkin,
and exploiting that $m_h^\text{L2}$ is bilinear, the corresponding algebraic system
to be solved is (block-) diagonal:
\begin{equation}
Dz^{k+1} = s^k - \Delta t^k q^k.
\end{equation}
Moreover, a stability condition restricting the time step $\Delta t^k$
has to be obeyed. The maximum allowable time step can be computed
explicitly for the simplest schemes and depends on the mesh $\mathcal{T}_h$.
For explicit time-stepping schemes therefore the following algorithm is employed:
1. While traversing the mesh assemble the vectors $s^k$ and
$q^k$ separately and compute the maximum time step $\Delta t^k$.
2. Form the right hand side $b^k=s^k - \Delta t^k q^k$ and ``solve'' the
diagonal system $Dz^{k+1} = b^k$ (can be done in one step).

This procedure can be applied also to more general time-stepping schemes
such as strong stability preserving Runge-Kutta methods [[16]](#Shu).

# Realization in PDELab

## Parameter Class in `problem.hh`

The parameter class provides all data of the PDE problem: Coefficient functions,
boundary and initial conditions. They all may depend on time now in comparison to tutorial 01.
In order to pass the time argument there are at least two options:

1. Extend the interface of all methods by an additional time
argument.
2. Pass the evaluation time via a separate function and store the
time in a private data member for subsequent spatial evaluations.

The second approach is taken in PDELab as it allows the reuse
of classes from the stationary case. Interfaces of methods are not changed
and only an additional method needs to be provided.
This new method on the class `Problem` has the following implementation:
```c++
//! Set time in instationary case
void setTime (Number t_)
{
  t = t_;
}
```
The method just stores the given time in an internal data member.

The use of the time is shown by the Dirichlet boundary condition
extension method implementing the function
$u_g(x,t) = \sin(2\pi t)*\prod_{i=1}^{d-1} \sin^2(\pi x_i)\sin^2(10\pi x_i)$:

```c++
template<typename E, typename X>
Number g (const E& e, const X& x) const
{
  auto global = e.geometry().global(x);
  Number s=sin(2.0*M_PI*t);
  for (std::size_t i=1; i<global.size(); i++)
    s*=sin(global[i]*M_PI)*sin(global[i]*M_PI);
  for (std::size_t i=1; i<global.size(); i++)
    s*=sin(10*global[i]*M_PI)*sin(10*global[i]*M_PI);
  return s;
 }
```
Note that there is no extra method for the initial condition. The method `g` is assumed to provide the initial condition for the initial time.

## Instantiating grid

This is done as in the previous tutorials.

```c++
#include <dune/jupyter.hh>
#include "nonlinearheatfem.hh"
#include "problem.hh"
```

```c++
const int dim = 2;
const int degree = 1;
using RF = double;
```

```c++
Dune::ParameterTree ptree;
Dune::ParameterTreeParser ptreeparser;
ptreeparser.readINITree("tutorial03.ini",ptree);

// read ini file
const int refinement = ptree.get<int>("grid.refinement");
```

```c++
//using grid factory
using Grid = Dune::YaspGrid<dim>;
using DF = Grid::ctype ;
// define the extensions of the domain
Dune::FieldVector<double,dim> lowerleft(0.0);
Dune::FieldVector<double,dim> upperright;
upperright[0] = ptree.get("grid.structured.LX",(double)1.0);
upperright[1] = ptree.get("grid.structured.LZ",(double)1.0);

std::array< unsigned int, dim> N;
N[0] = ptree.get("grid.structured.NX",(int)10);
N[1] = ptree.get("grid.structured.NY",(int)10);

// build a structured grid
auto grid = Dune::StructuredGridFactory<Grid>::createCubeGrid(lowerleft, upperright, N);
```

```c++
grid->globalRefine(refinement);
using GV = Grid::LeafGridView ;
GV gv=grid->leafGridView();
using FEM = Dune::PDELab::QkLocalFiniteElementMap<GV,DF,double,degree>;
FEM fem(gv);
```

## Changes due to Instationarity

We now go through the changes which are due to instationarity. The first change concerns the initialization of simulation time and set-up of the parameter object:

The first change concerns the initialization of simulation time and set-up of the parameter object:

```c++
// make user functions and set initial time
RF time = 0.0;
RF eta = ptree.get("problem.eta",(RF)1.0);
Problem<RF> problem(eta);
problem.setTime(time);
```

Now a PDELab grid function can be constructed:

```c++
auto g = Dune::PDELab::
makeInstationaryGridFunctionFromCallable(
    gv,
    [&](const auto& e, const auto& x){
        return problem.g(e,x);},
    problem);;
```

A new function `makeInstationaryGridFunctionFromCallable` is provided since the grid function produced by it now also provides a method `setTime` to pass the time argument.
Note that the lambda closure and the parameter object need to be
provided as the `setTime` method is only on the parameter class
and not on the lambda closure.

```c++
auto b = Dune::PDELab::
makeBoundaryConditionFromCallable(
    gv,
    [&](const auto& i, const auto& x){
        return problem.b(i,x);}
);;
```

Setting up the grid function space and interpolating the initial condition is the same as before.

```c++
// Make grid function space
using CON = Dune::PDELab::ConformingDirichletConstraints;
using VBE = Dune::PDELab::ISTL::VectorBackend<>;
using GFS = Dune::PDELab::GridFunctionSpace<GV,FEM,CON,VBE>;
GFS gfs(gv,fem);
gfs.name("Vh");

// Assemble constraints
using CC = typename GFS::template
ConstraintsContainer<RF>::Type;
CC cc;
Dune::PDELab::constraints(b,gfs,cc); // assemble constraints
std::cout << "constrained dofs=" << cc.size() << " of "
        << gfs.globalSize() << std::endl;

// A coefficient vector
using Z = Dune::PDELab::Backend::Vector<GFS,RF>;
Z z(gfs); // initial value

// Make a grid function out of it
using ZDGF = Dune::PDELab::DiscreteGridFunction<GFS,Z>;
ZDGF zdgf(gfs,z);

// initialize simulation time, the coefficient vector
Dune::PDELab::interpolate(g,gfs,z);
```

Then the grid operator for the spatial part can be set up as in the stationary problem tutorial 01. We just use the new local operator `NonlinearHeatFEM`:

```c++
// Make instationary grid operator
using LOP = NonlinearHeatFEM<Problem<RF>,FEM>;
LOP lop(problem);
using MBE = Dune::PDELab::ISTL::BCRSMatrixBackend<>;
int degree = ptree.get("fem.degree",(int)1);
MBE mbe((int)pow(1+2*degree,dim));
using GO0 = Dune::PDELab::GridOperator<GFS,GFS,LOP,MBE,
                                 RF,RF,RF,CC,CC>;
GO0 go0(gfs,cc,gfs,cc,lop,mbe);
```

The temporal part is done similarly and just employs another local operator
`L2`:

```c++
using TLOP = L2<FEM>;
TLOP tlop;
using GO1 = Dune::PDELab::GridOperator<GFS,GFS,TLOP,MBE,
                                 RF,RF,RF,CC,CC>;
GO1 go1(gfs,cc,gfs,cc,tlop,mbe);
```

Spatial and temporal part are now combined using the new class `OneStepGridOperator` which can assemble the linear combinations needed in the Runge-Kutta methods:

```c++
using IGO = Dune::PDELab::OneStepGridOperator<GO0,GO1>;
IGO igo(go0,go1);
```

Linear solver backend and Newton method can be set up in the same way as before, just using the `OneStepGridOperator`.

```c++
// Select a linear solver backend
using LS = Dune::PDELab::ISTLBackend_SEQ_CG_AMG_SSOR<IGO>;
LS ls(100,0);

// solve nonlinear problem
using PDESOLVER = Dune::PDELab::Newton<IGO,LS,Z>;
PDESOLVER pdesolver(igo,z,ls);
pdesolver.setReassembleThreshold(0.0);
pdesolver.setVerbosityLevel(2);
pdesolver.setReduction(1e-8);
pdesolver.setMinLinearReduction(1e-4);
pdesolver.setMaxIterations(25);
pdesolver.setLineSearchMaxIterations(10);
```

Then a time-stepping scheme in the form of the matrices $A$, $B$ and the vector $d$ needs to be selected.
Several classes are already provided and it is very easy to write your own class:

```c++
Dune::PDELab::OneStepThetaParameter<RF> method1(1.0);
Dune::PDELab::Alexander2Parameter<RF> method2;
Dune::PDELab::Alexander3Parameter<RF> method3;
```

The selection mechanism for the time-stepping scheme is implemented by setting a pointer to an appropriate method:

```c++
int torder = ptree.get("fem.torder",(int)1);
Dune::PDELab::TimeSteppingParameterInterface<RF>*
  pmethod=&method1;
if (torder==1) pmethod = &method1;
if (torder==2) pmethod = &method2;
if (torder==3) pmethod = &method3;
if (torder<1||torder>3)
  std::cout<<"torder not in [1,3]"<<std::endl;
```

Note that`TimeSteppingParameterInterface` is the interface from which all time stepping parameter classes derive.

Then an object of class`OneStepMethod` can be instantiated. This class is able to compute a single time step of a general Runge-Kutta one step method.

```c++
using OSM = Dune::PDELab::OneStepMethod<RF,IGO,PDESOLVER,Z,Z>;
OSM  osm(*pmethod,igo,pdesolver);
osm.setVerbosityLevel(2);
```

Before entering the time loop we set up VTK output for the instationary case.
Paraview has several ways to handle sequences of output files. Here we use
an advanced method that writes a special \lstinline{pvd}-file providing the
names of the files for each individual step and the associated absolute time. This
is particularly useful when non-equidistant time steps are used. All the
data files for the time steps are put in a subdirectory in order not to clog your file system.
The following code creates a directory (the name is read from the ini-file):

```c++
// prepare VTK writer and write first file
std::string filename=ptree.get("output.filename","output");

//write filename to .txt file
std::ofstream out("name.txt");
out << filename;
out.close();

std::filesystem::create_directory(filename);
```

Now a `SubsamplingVTKWriter` and a `VTKSequenceWriter` can be instantiated:

```c++
int subsampling=ptree.get("output.subsampling",(int)1);
using VTKWRITER = Dune::SubsamplingVTKWriter<GV>;
VTKWRITER vtkwriter(gv,Dune::refinementIntervals(subsampling));
using VTKSEQUENCEWRITER = Dune::VTKSequenceWriter<GV>;
VTKSEQUENCEWRITER vtkSequenceWriter(
  std::make_shared<VTKWRITER>(vtkwriter),filename,filename,"");
```

filled with data

```c++
using VTKF = Dune::PDELab::VTKGridFunctionAdapter<ZDGF>;
vtkSequenceWriter.addVertexData(std::shared_ptr<VTKF>(
                           new VTKF(zdgf,"solution")));
```

and the first file is written

```c++
vtkSequenceWriter.write(time,Dune::VTK::appendedraw);
```

Now the time loop is under user control:

```c++
RF T = ptree.get("problem.T",(RF)1.0);
RF dt = ptree.get("fem.dt",(RF)0.1);
while (time<T-1e-8)
  {
    // assemble constraints for new time step
    problem.setTime(time+dt);
    Dune::PDELab::constraints(b,gfs,cc);

    // do time step
    Z znew(z);
    osm.apply(time,dt,z,g,znew);

    // accept time step
    z = znew;
    time+=dt;

    // output to VTK file
    vtkSequenceWriter.write(time,Dune::VTK::appendedraw);
  }
```

```c++
vtkSequenceWriter
```

First the final time and time step are read from the parameter file. Then, for each time step the constraints are reassembled. Note that the boundary condition type is not supposed to change *within* a time step, so all stages of the Runge-Kutta Method
will use the same constraints (but not necessarily the same values at the
Dirichlet boundary).

Then, the `apply` method of the one step method is used to advance one step in time. Its arguments are the current `time`, the size of the time step `dt`, the coefficient vector `z` of the oldtime step, the boundary condition function `g` used to interpolate
boundary conditions for the stages and, as last argument, the new coefficient
vector`znew` to be computed as a result.

In order to advance to the next time step the coefficient vector is copied and the
time variable is advanced. Finally, the output file for the time step is written.

Intentionally, the time loop is under user control to allow other things to be done,
e.g. choose adaptive time step size, proceeding to important absolute times,
writing only  every $n$th output file and so on.

## Spatial Local Operator

The file `nonlinearheatfem.hh` contains the two local operators
for the spatial and the temporal part. The class `NonlinearHeatFEM` implements the
spatial part. It is actually very easy since the main part can be reused
from the `NonlinearPoissonFEM` local operator  implemented in tutorial 01 through
public inheritance:

```c++
template<typename Param, typename FEM>
class NonlinearHeatFEM :
  public NonlinearPoissonFEM<Param,FEM>,
  public Dune::PDELab::
    InstationaryLocalOperatorDefaultMethods<double>
{
  Param& param;
public:
  //! Pass on constructor
  NonlinearHeatFEM (Param& param_, int incrementorder_=0)
    : NonlinearPoissonFEM<Param,FEM>(param_,incrementorder_),
    param(param_)
  {}
  //! set time for subsequent evaluation
  void setTime (double t) {
    param.setTime(t);
  }
};
```

Just a few methods need to be added to the local operator.
Default implementations can be inherited from `InstationaryLocalOperatorDefaultMethods`. Here only the method `setTime` needs to be implemented to pass on the time to the parameter class for subsequent evaluations.

The following methods need to be provided by a local operator for instationary computations:

```c++
void setTime (R t_)
```
to set the time for all subsequent method calls.

```c++
R getTime () const
```
to get the time set previously.

```c++
void preStep (RealType time, RealType dt, int stages)
```
This method is called at the beginning of a time step, before all stages.

```c++
void postStep ()
```
This method is called at the end of a time step, after all stages.

```c++
void preStage (RealType time, int r)
```
This method is called at the beginning of a Runge-Kutta stage, where $r$ is the number of the stage (starting with $r=1$).

```c++
int getStage () const
```
Return the number of the current stage.

```c++
void postStage ()
```
Called after the intermediate result of a stage has been computed.

```c++
RealType suggestTimestep (RealType dt) const
```
In an explicit scheme this method can be called at the end of stage 1 to suggest a stable time step for the explicit scheme.

## Temporal Local Operator

The temporal local operator needs to implement the residual form related to the $L_2$-inner product. It could also be used to compute an $L_2$ projection of some function.

Like in the spatial local operator we provide a finite element map as template parameter and derive from some helpful classes to provide default implementations of some methods:
```c++
template<typename FEM>
class L2
  : public Dune::PDELab::FullVolumePattern,
    public Dune::PDELab::LocalOperatorDefaultFlags,
    public Dune::PDELab::
  InstationaryLocalOperatorDefaultMethods<double>
```

The following flag indicates that we provide a `pattern_volume` method
specifying the sparsity pattern. Actually, this method is provided from the base class
`FullVolumePatter`:
```c++
enum { doPatternVolume = true };
```

The next flag indicates that just volume integrals depending on trial and test functions are required:
```c++
enum { doAlphaVolume = true };
```

The implementation of `alpha_volume` is very similar to the reaction term in tutorial 01. For efficiency reasons also `jacobian_volume` is provided as well as the `*_apply_*` variants for matrix-free operation (which might be very useful here since the mass matrix is well conditioned).

# Exercise

The code of `tutorial03` solves the problem described in (\ref{equ:problem}) - (\ref{equ:problemEnd}), with the following choices applied:

  \begin{align} \label{ch1:first}
    q(u) &= \eta u^2 \\
    f &= 0 \\
    \Gamma_D &= \{x \in\partial\Omega \mid x_0 = 0 \} \\
    g(x,t) &= \sin(2\pi t) \prod_{i=1}^{d-1} \sin(\pi x_i)^2
    \sin(10\pi x_i)^2\\
    j(x,t) &= 0 \\
    u_0(x) &= g(x,0) = 0 \label{ch1:last}\\
    t_0&=0 .
  \end{align}

## Getting to know the Code

 As in the previous exercises you can control most of the settings through the ini-file `tutorial03.ini` or directly within the notebook. Get an overview of the configurable settings and run the code.

The program writes output with the extension `pvd`. This is one of several ways to write VTK output for the instationary case, c.f. the documentation of the `tutorial03`. The `pvd`-file can be visualized by ParaView and it consists of a collection of the corresponding `vtu`-files.
One big advantage of this approach is that the physical time can be printed out. This can be achieved by using the ``AnnotateTimeFilter`` in ParaView.

 - *the video below shows the solution for the nonlinear heat equation*

<video controls src="videos/ex01nonLin.ogv" width="70%"/>

## Making Discretizations easily exchangable

**Step 1: Switching to the linear heat equation**
 For the rest of the exercise we want to consider the linear heat equation. Therefore the reaction term has to be set to $q =  0$. Recompile and rerun `exercise03.cc` and investigate the difference to the nonlinear reaction term.

 - *example video linear heat equation*

<video controls src="videos/ex02lin.ogv" width="70%"/>

<video controls src="videos/ex02diff.ogv" width="70%"/>

**Hint for the rest of the exercise:** For different runs of the simulation you can change the output filename in `tutorial03.ini`.

Since the initial problem was nonlinear, Newton's method is used to solve the discretized equations. For the linear case it is sufficient to use the class `StationaryLinearProblemSolver`. Search in the notebook for the lines starting with

```c++
using PDESOLVER = Dune::PDELab::Newton<IGO,LS,Z>;
PDESOLVER pdesolver(igo,z,ls);
```

and change to the `StationaryLinearProblemSolver`.

It has the same template arguments as the Newton solver. Give the instance of the class `StationaryLinearProblemSolver` also the name `pdesolver`. If you have problems with the construction of this solver consider for example the code in the driver of `tutorial00`. Compile and run again. The program reports the status of the solver. Get used to these different two outputs.

```c++
//using PDESOLVER = Dune::PDELab::Newton<IGO,LS,Z>;
//PDESOLVER pdesolver(igo,z,ls);

using PDESOLVER = Dune::PDELab::StationaryLinearProblemSolver<IGO,LS,Z>;
PDESOLVER pdesolver(igo,ls,z,1e-10);
```

As a next step we want to use two spatial discretizations, i.e. $\mathcal{Q}_1$ and $\mathcal{Q}_2$ elements. The degree of the spatial discretization can be changed in the ini-file. Currently $\mathcal{Q}_1$ elements are used. Please change to $\mathcal{Q}_2$ elements and rerun the simulation.

- *the video shows a comparison between $\mathcal{Q}_1$ (left) and $\mathcal{Q}_2$ (right) elements*

<video controls src="videos/ex02deg2.ogv" width="70%"/>

**Step 2: Arbitrary one-step schemes** We want to examine the numerical solution under three different time
  discretization schemes -- Implicit Euler, Crank-Nicolson and Fractional-Step-$\theta$. In order to change the time discretization scheme you will have to go to the<font color = 'red'> file </font> `driver.hh` and search for the line
```c++
    Dune::PDELab::Alexander2Parameter<RF> pmethod;
```

Change this to use the`Dune::PDELab::ImplicitEulerParameter<RF>`, compile and rerun the simulation. The program reports the progress of the time stepping and the method used. Convince yourself that you are using indeed the Implicit Euler. The other two time stepping methods can be applied similarly.

  **Note that** there is no special one step parameter class for Crank-Nicolson.  Crank-Nicolson is however the special case of the one step $\theta$ scheme with $\theta=0.5$.  So you can create a parameter
  object for Crank-Nicolson with `Dune::PDELab::OneStepThetaParameter<RF>(0.5)`.  A parameter object for the Fractional Step $\theta$ scheme can be created with `Dune::PDELab::FractionalStepParameter<RF>()`.

 - Implicit Euler method
 ```c++
 Dune::PDELab::ImplicitEulerParameter<RF> pmethod;
 ```
 - Crank-Nicolson method
 ```c++
Dune::PDELab::OneStepThetaParameter<RF> pmethod(0.5);
```
 - Fractional-Step-$\theta$
```c++
 Dune::PDELab::FractionalStepParameter<RF> pmethod;
```

**Step 3: Different Initial and Boundary Conditions**
Consider the initial and boundary conditions
  (\ref{ch1:first})--(\ref{ch1:last}) modified as follows:
  \begin{align}
    \label{ch2:first}
    \Gamma_D&=\emptyset \\
    u_0(x) &= g(x, 0) = \prod_{i=0}^{d-1}\min\{1,\max\{0,\tilde{f}(x_i)\}\} \\
        & \qquad\qquad \tilde{f}(\xi):=0.5-8(|\xi-0.5|-0.25) \nonumber \\
    \label{ch2:last}
    j(x,t)&=0 .
  \end{align}
The initial condition given by $u_0$ models a block of constant 1 concentration in the middle, constant 0 concentration at the border and some linear decrease in between.

On a $16\times16$ or finer grid the initial values can be represented exactly by $\mathcal{Q}_1$ and $\mathcal{Q}_2$ finite elements. The exact solution will instantly become smooth and tend toward the mean over time.  A computed solution is only an approximation, and may show different behavior.  Most often it may
take a long time for the solution to become smooth and, depending on the time stepping scheme used, there are spikes oscillating from one time step to the next.

Please implement the initial and boundary conditions (\ref{ch2:first})--(\ref{ch2:last}). When implementing $u_0$ you may be tempted to use the function `abs()`. This is wrong, `abs()` is a function from the C-library to compute absolute values for *integers*. If a floating point value is plugged in, it is truncated to an integer. The correct way is to use `std::abs()` instead.

Compile and run your program. Remember that the grid needs $16\times16$ elements for the $\mathcal{Q}_1$/$\mathcal{Q}_2$ elements to resolve the initial condition exactly. What happens to the interpolated initial condition if you use a coarser mesh?

<video controls src="videos/ex03BC.ogv" width="70%"/>

 - *initial condition for grid with $8\times8$ elements, using $\mathcal{Q}_1$ elements*

**Step 4: Investigate Maximum Principle**

With these preparations done, it is now time to actually check how the different discretizations perform.  Run your program to produce some output that you can examine in ParaView. Change the settings in the ini-file to a $64\times 64$ grid and the time step size `<dt>`$\,=1/64=0.015625$. Run the simulation until `<tend>`$\,=4 \cdot\text{dt}$.
 When examining the solution in ParaView, apply the "Warp by Scalar" filter to get an image distorted into the third dimension according to the values of the solution.

After one time step, the solution computed by $\mathcal{Q}_1$ finite elements with Implicit Euler time stepping should be completely smooth. The same goes for $\mathcal{Q}_2$ with Implicit Euler.

With both Crank-Nicolson and Fractional Step $\theta$, both with $\mathcal{Q}_1$ and $\mathcal{Q}_2$ the solution should be quite non-smooth, i.e. there should be some edges visible. The Fractional Step $\theta$ scheme should be smoother than Crank-Nicolson.

Try to run the simulation with smaller time steps.  How small do you need to make the time steps to get smooth solutions with Fractional Step $\theta$?

Can you get the Crank-Nicolson scheme to produce smooth solutions as well?

## Time Dependent $g$ and $j$

Consider now the initial and boundary conditions (\ref{ch1:first})--(\ref{ch1:last}) time dependent:
\begin{align}
  \label{ch3:first}
  \Gamma_D &= \{ x\in\partial\Omega \mid x_0 = 0 \} \\
  g(x,t) &= t/10 \\
  u_0(x) &= g(x,0) \\
  \label{ch3:last}
  j(x,t) &= -(0.5 + \cos(t)/2) .
  \end{align}
Incorporating the time dependence into the functions $g$ and $j$ is easy even if they don't have the time variable as an argument. The problem parameter class possesses the member variable `t` and the member function
 ```c++
    void setTime (Number t_)
    {
      t = t_;
    }
 ```
by means of which the correct time is always available. Please implement the conditions (\ref{ch3:first})--(\ref{ch3:last}), compile and rerun the program. You might also want to increase the final time of the simulation. Examine the results in ParaView with the "Warp by Scalar" filter.

**A short note on time dependent $\Gamma_D$ and $\Gamma_N$:** In principle it is possible to implement time dependent $\Gamma_D$ and $\Gamma_N$ the same way as for $g$ and $j$. But for conforming spatial discretizations there is an important limiting assumption, namely that the type of boundary conditions do not change over a time step.

 - *solution for g and j*

```c++
template<typename E, typename X>
Number g (const E& e, const X& x) const
{
    return t/10.;
}

template<typename I, typename X>
Number j (const I& i, const X& x) const
{
  double result = -(0.5 + cos(t)/2.);
  return result;
}
```

<video controls src="videos/ex03tBC.ogv" width="70%"/>

# Bibliographie

[1] Alexander, R. *Diagonally Implicit Runge-Kutta Methods for Stiff O. D. E.'s*. SIAM Journal on Numerical Analysis. Vol. 14. 1977.

[2] Bastian, P. *Lecture Notes on Scientific Computing with Partial Differential Equations*. 2014. http://conan.iwr.uni-heidelberg.de/teaching/numerik2_ss2014/num2.pdf.

[3] Braess, D. *Finite Elemente*. Springer. 2003.

[4] Brenner, S. C. and Scott, L.R. *The mathematical theory of finite element methods*. Springer. 1994.

[5] Ciarlet, P. G. *The finite element method for elliptic problems*. SIAM. Classics in Applied Mathematics. 2002.

[6] Elman, H., Silvester, D. and Wathen, A. *Finite Elements and Fast Iterative Solvers*. Oxford University Press. 2005.

[7] Eriksson, K. ,Estep,D. Hansbo, P. and Johnson, C. *Computational Differential Equations*. Cambridge University Press. 1996. http://www.csc.kth.se/~jjan/private/cde.pdf.

[8] Ern, A. and Guermond, J.-L. *Theory and practice of finite element methods*. Springer. 2004.
<a id="ern"> </a>

 [9] Shu, C. W. and Osher, S. *Efficient implementation of essentially non-oscillatory shock-capturing schemes*. J. Comput. Phys. Vol. 77. 1988.
 <a id="ShuOsher88"> </a>

[10] Gottlieb, S., Ketcheson, D.I. and Shu, C.W. *Strong Stability Preserving Runge-Kutta and Multistep Time Discretizations*. World Scientific. 2011.
<a id="gottlieb11"> </a>

[11] Geuzaine, C. and Remacle, J.-F. *Gmsh: A 3-D finite element mesh generator with built-in pre- and post-processing facilities*. International Journal for Numerical Methods in Engineering. Vol. 79. 2009.

[12] Großmann, C. and Roos, H.-G. *Numerische Behandlung partieller Differentialgleichungen*. Teubner. 2006.

[13] Hackbusch, W. *Theorie und Numerik elliptischer Differentialgleichungen*. Teubner. 1986. http://www.mis.mpg.de/preprints/ln/lecturenote-2805.pdf.

[14] Meidner, D. and Richter, T. *A posteriori error estimation for the fractional step theta discretization of the incompressible Navier–Stokes equations*. Computer Methods in Applied Mechanics and Engineering. 2015.

[15] Rannacher, R. *Einführung in die Numerische Mathematik II (Numerik partieller Differentialgleichungen)*. 2006. http://numerik.iwr.uni-heidelberg.de/~lehre/notes.

[16] Shu, C.W. *Total-variation-diminishing time discretizations*. SIAM J. Sci. Stat. Comput. Vol. 9. 1988.
<a id = "Shu"> </a>


```python
import cirq
```

We have already seen how we can use a quantum circuit to send an entire quantum state just using two bits of classical information. This leads to a natural follow up question, namely, can we use a quantum channel to transport classical information efficiently.

This can be done by something called superdense Coding.
Here Alice sends one qubit through a quantum channel and thereby communicates two bits of classical information to Bob.

Reading: Ch 5, Phys. Rev. Lett. 69, 2881

Again Bob and Alice start out with shared Bell state $|\beta_{00}\rangle$.
The idea here is even simpler than the one used for quantum teleportation.

It is now easy to see that with the help of a suitable (application of a unitary Transformation only acting on the subspace of the first qubit) single-qubit gate Alice can transform $|\beta_{00}\rangle$ into any other Bell-state.

For example:
\begin{equation}
Z \otimes  I |\beta_{00}\rangle = \frac{1}{\sqrt{2}}\left(|00\rangle-|11\rangle \right) = |\beta_{10}\rangle
\end{equation}
So if Alice now sends her qubit to Bob and Bob performs a Bell measurement, he will know both classical bits. Fundamentally this protocol exploits the nature of entanglement.

```python
from gates import make_bell_state, bell_measurement
def superedense(b1,b2):
    #creat two qubuits
    a, b =cirq.LineQubit.range(2)
    #dictornary how to send bits
    to_send = {(0,0):[], (0,1):[cirq.X(a)],(1,0):[cirq.Z(a)], (1,1):[cirq.X(a),cirq.Z(a)]}
    c = cirq.Circuit()
    #create bell state \beta_{00}
    c.append(make_bell_state(a,b))
    #Alice transforms 00 to required bell state
    c.append(to_send[(b1,b2)])
    #Alice send state to Bob who performs bell measurment
    bell_measurement(c,a,b)
    print("Circuit to send {}{}".format(b1,b2))
    print(c)
    return c
```

Let's see if it all worked out

```python
for b in [(0,0), (0,1), (1,0), (1,1)]:
    c = superedense(*b)
    s=cirq.Simulator()
    results=s.simulate(c)
    print(results)
    print()
```

    Circuit to send 00
    0: ───H───@───@───H───M───
              │   │       │
    1: ───────X───X───────M───
    measurements: 0,1=00
    output vector: |00⟩

    Circuit to send 01
    0: ───H───@───X───@───H───M───
              │       │       │
    1: ───────X───────X───────M───
    measurements: 0,1=01
    output vector: |01⟩

    Circuit to send 10
    0: ───H───@───Z───@───H───M───
              │       │       │
    1: ───────X───────X───────M───
    measurements: 0,1=10
    output vector: |10⟩

    Circuit to send 11
    0: ───H───@───X───Z───@───H───M───
              │           │       │
    1: ───────X───────────X───────M───
    measurements: 0,1=11
    output vector: |11⟩


# Practical 1a: Machine Learning Fundamentals

© Deep Learning Indaba. Apache License 2.0.

## Introduction
In this practical, we introduce the idea of classification (sorting things into categories) using a machine-learning model. We explore the relationship between a classifier's parameters and the decision boundary (a line that separates categories) and also introduce the idea of a loss function. Finally, we briefly introduce Tensorflow.

## Learning Objectives
* Understand the idea of **classification**
* Understand the concept of (linear) **separability** of a dataset.
* Understand what the **parameters** of a classifier are and how they relate to the **decision boundary**
* Be able to briefly explain what **Tensorflow** is.

**Imports**

```
!pip install -q moviepy > /dev/null 2>&1
!pip install -q imageio > /dev/null 2>&1
!pip install -q tensorflow==2.0.0-beta0 > /dev/null 2>&1

import tensorflow as tf
import numpy as np                 # Numpy is an efficient linear algebra library.
import matplotlib.pyplot as plt    # Matplotlib is used to generate plots of data.
from matplotlib import animation, rc

from IPython import display

try:
  tf.executing_eagerly()
  print("TensorFlow executing eagerly: {}".format(tf.executing_eagerly()))
except ValueError:
  print('Already running eagerly')

```

Make sure to run the imports cell above, otherwise the rest of the cells will fail when you try to run them. (To run a cell press `shift` + `enter` with your mouse cursor in the cell or press the play button in the top right of the cell.)

## Outline
In this practical, we tackle the task of **classification** of a simple, synthetic dataset. Classification in machine learning involves learning a labelling of examples into one (or more) discrete categories. This differs from another common task in machine learning called **regression**, which involves learning a mapping from inputs to a continuous-valued output.

1. We begin by introducing a synthetic dataset of red and blue points which we want to separate
2. We introduce and explore the idea of **linear separability**
3. We define a **loss** as a measure of how good of a seperator a particular line is
4. We briefly introduce TensorFlow and show how it can be used to automatically find the minimum of a loss function.

**Helper Functions (RUN ME)**

```
# this is a helper function to assist with plotting the dataset below
def plot_dataset(inputs, labels):
  # Plot the given 2D inputs and labels using Matplotlib.
  plt.scatter(
      inputs[:, 0], inputs[:, 1],
      c=['red' if label > 0 else 'blue' for label in labels])

  plt.axis('equal')

  plt.xlabel('x1')
  plt.ylabel('x2')
```

## The Data

Run the code in the cell below, and look at the resulting plot. It should produce a simple 2-D data set consisting of 2 classes of points, the classes are represented by colours blue and red. Our task is to build a **binary classifier** that can distinguish between red and blue points (red and blue are referred to as the **classes** of the points), using only the 2-D coordinates of a point. In other words, we want a function that takes as input a 2-D vector representing the coordinates of a point and returns a value of 1 or 0 indicating whether the point is red or blue. Here we have **encoded** the colours red and blue into the numbers 1 and 0 (which make it easier to work with in maths and code!)

**Note:** we have arbitrarily encoded red as 1 and blue as 0, you could do it the other way around too as long as you're consistent!

```
#@title Generate the Dataset  {run: "auto"}
# Define the centre(s) of the points
centre = 1    #@param {type:"slider", min:0, max:2, step:0.1}

points_in_class = 20  # How many points we want per class

# A fixed random seed is a common "trick" used in ML that allows us to recreate
# the same data when there is a random element involved.
np.random.seed(0)

# Generate random points in the "red" class
red_inputs = np.random.normal(loc=centre, scale=1.0, size=[points_in_class, 2])
# Generate random points in the "blue" class
blue_inputs = np.random.normal(loc=-centre, scale=1.0, size=[points_in_class, 2])
# Put these together
inputs = np.concatenate((red_inputs, blue_inputs), axis=0)

# The class (label) is 1 for red or 0 for blue
red_labels = np.ones(points_in_class)
blue_labels = np.zeros(points_in_class)
labels = np.concatenate((red_labels, blue_labels), axis=0)

# num_data_points is the total data set size
num_data_points = 2 * points_in_class

plot_dataset(inputs, labels)
```

###What does the data look like?
The inputs are 2-dimensional vectors (points in a 2-D space). Here are the coordinates of 4 points, which we've deliberately chosen so that points 1 and 2 are "red" and points 3 and 4 are "blue".

```
print('Input 1:\t', inputs[0])
print('Input 2:\t', inputs[1])

print('Input 3:\t', inputs[-1])
print('Input 4:\t', inputs[-2])
```

The labels are either 0 (blue) or 1(red). Here are the labels corresponding to the points above:

```
print('Label 1:\t', labels[0])
print('Label 2:\t', labels[1])

print('Label 3:\t', labels[-1])
print('Label 4:\t', labels[-2])
```

### Other Examples of Binary Classification Problems

In this practical, we are using a synthetic dataset where we have 2 classes of 2-D points that come from different distributions, distinguised by the colours, red and blue. To make this more concrete, here are some examples of more real-world binary classification problems.

* Determine whether an email message (input) is SPAM or NOT SPAM (label)
* Determine whether an image, represented by its encoded pixel values (input) is a picture of a DOG or a CAT (label)
* Determine whether energy usage of a building will go UP or DOWN (label) next month, using a time series of past energy usage values (input)

## Linear separability

For a 2-dimensional dataset with 2 classes, we can say that the dataset is **linearly separable** if it is possible to draw a (1-D) line separating the examples of each class. In this case, we would want a line between the red and blue points such that all of the red points lie on one side of the line and all of the blue points on the other. Linear separability of a D-dimensional dataset with 2 classes means that there exists a single (D-1)-dimensional (hyper-)plane that separates the classes (a hyperplane is a generalisation of a straight line to many dimensions).

### Exploratory Task
In the code cell under the heading "The Data", change the slider for the ```centre``` value. This will automatically update the value in the code and will redraw the plot.

* At what value of centre does the dataset become linearly separable?

### Question for discussion
Can you think of some 2-D, 2-class datasets, similar to the one above, that are separable (the points from the 2 classes don't overlap each other), but are not **linearly** separable? Draw some examples on paper or plot them using Matplotlib and discuss this with your neighbour and tutors.

## Drawing the line

As you may recall from school, a line in 2 dimensions, with coordinate axes $x_1$ and $x_2$, which passes through the origin (0, 0) can be represented by the equation $w_1x_1 + w_2x_2 = 0$

We can also write this in vector form as: $\mathbf{w}^T\mathbf{x} = 0$, where $\mathbf{w}^T = [w_1, w_2]$ and $\mathbf{x}^T = [x_1, x_2]$.

When a line (or more generally, a hyperplane) is defined this way, we call the **parameters** $\mathbf{w} = (w_1, w_2)$ a **normal vector** for the line. The normal vector is orthogonal (perpendicular) to the line. We want to construct such a line that separates red and blue  points, which we will call a **decision boundary**. The name "decision boundary" comes from the fact that the line describes the boundary in the input space where the model's output (decision) changes from one class to another.

In the following cell, we plot our dataset along with a normal vector $\mathbf{w}$ and decision boundary. You can adjust the values of $w_1$ and $w_2$ by using the sliders on the right. Observe the effect that the values have on the normal vector drawn in *red* and decision boundary in *black*. Adjust the values so that the black line separates the blue and red points (i.e. red points on one side and blue on the other). Your line should also have the normal vector pointing in the direction of the red points. The reason that direction is significant is that we want to eventually **classify** points on one side of the line (decision boundary) as being red and the other as being blue.

Is it possible to find a line through the origin that perfectly separates the points?

**Note**: Each of our inputs is a 2-D vector, made up of two coordinate values. We refer to these 2 coordinate axes as $x_1$ and $x_2$. For example, if we have an input $(1, 2)$, then we would say $x_1 = 1$ and $x_2 = 2$ for that point.

```
#@title Effect of parameters {run: "auto"}

# Define the parameters
w1 = -1. #@param { type: "slider", min: -5, max: 5, step: 0.1 }
w2 = -1.6 #@param { type: "slider", min: -5, max: 5, step: 0.1 }

plot_dataset(inputs, labels)

# Add the weight vector to the plot. We plot it in red, as it has to "point"
# in the direction of the red points.
ax = plt.axes()
ax.arrow(0, 0, w1, w2, head_width=0.3, head_length=0.3, fc='r', ec='r')

# Plot part of the decision boundary in black. It is orthogonal to the weight
# vector.
t = 2 # this is how long the line should be
plt.plot([-t * w2, t * w2], [t * w1, -t * w1], 'k-')

plt.xlim([-4, 4])
plt.ylim([-4, 4])

plt.show()
```

## Classification
Given a normal vector $\mathbf{w}$, we can evaluate which side of the decision boundary a particular point $\mathbf{x_i} = (x_{i,1}, x_{i, 2})$ lies by evaluating $\mathbf{w^Tx_i}$. If $\mathbf{w^Tx_i} > 0$, the point $\mathbf{x_i}$ lies to one side of the boundary (in the direction of the normal vector), and we can classify that point as belonging to class 1 (in our case, "red"). If $\mathbf{w^Tx_i} < 0$ the point lies on the other side and can be classified as class 0 (in our case, "blue"). Finally if $\mathbf{w^Tx_i} = 0$ the point lies on the decision boundary and we can decide whether to classify it as either 0 or 1, or ignore it.

## How "good" is the line?

If you've played around with the above code, you may have developed some intuition around how different settings of the parameters influence the final placement of the decision boundary. The purpose of machine learning is to *automatically* adjust the values of $w_1$ and $w_2$ to find a suitable decision boundary! But to do this, we need to mathematically specify some **loss** or **objective** function. The loss is a function of the parameters $w_1$ and $w_2$ and tells us how good a certain configuration of the parameter values are at classifying the data. This function is defined such that it reaches its optimum setting when it is minimised, i.e. the *smaller* its value, the *better* the separation between the classes. An additional property a loss function can have that is often crucial for machine learning is being *differentiable*. A differentiable loss function allows us to use *gradient-based optimisation* to find its minimum and the corresponding optimal values of $w_1$ and $w_2$.

For this classification problem, we consider the **binary cross-entropy** loss function to measure how good the model's predictions are.  This loss function compares the model's prediction for each example, $\mathbf{x_i}$ to the true **target** $y_i$ (we often refer to the true label associated with an input as the "target"). It then applies the non-linear log function to penalise the model for being further from the true class.

The binary cross entropy function uses an operation called a  $\operatorname{sigmoid}$ function. This functions allows our classifier to output any real value. The binary cross entropy loss function, however, expects the predictions made by a classifier to be between $0$ and $1$. The sigmoid function "squashes" any real number inputs to lie in the interval $(0, 1)$.

For those of you who are more mathematically inclined, the equation for the binary cross entropy loss, on a dataset with $N$ points is defined as follows:

\begin{align}
l(\mathbf{w}; \mathbf{\hat{y}}, \mathbf{y}) = -\frac{1}{N}\sum_{i=1}^N y_i log(\hat{y}_i) + (1-y_i)log(1-\hat{y}_i)
\end{align}

where $\hat{y}_i = \operatorname{sigmoid}(\mathbf{w}^T\mathbf{x_i})$ and the $\operatorname{sigmoid}$ function is defined as:

$$
\mathrm{sigmoid}(a) = \frac{1}{1 + e^{-a}} .
$$

If you don't understand all of the math, don't stress too much, but if you want like to learn more about the binary cross-entropy loss, check out this [blog](https://towardsdatascience.com/understanding-binary-cross-entropy-log-loss-a-visual-explanation-a3ac6025181a).

Let's now wrap this in a Python function so that we can compute the loss for any values of $w_1$ and $w_2$:

```
def compute_loss(w1, w2, inputs, labels):

  total_log_likelihood = 0
  N = len(inputs)

  # Add the contribution of each datapoint to the loss
  for (x1, x2), target in zip(inputs, labels):
    # As our targets are 0 or 1, our prediction function must output a value between 0 and 1.
    # The sigmoid function 'squashes' any value to lie between 0 and 1:
    prediction = tf.sigmoid(w1*x1 + w2*x2)

    # Compute the local loss term
    # We add 1e-10 to make the log operations numerically stable (i.e. avoid taking the log of 0.)
    log_likelihood = target * tf.math.log(prediction + 1e-10) + (1.-target)*tf.math.log(1.-prediction + 1e-10)
    total_log_likelihood += log_likelihood

  loss = -total_log_likelihood
  average_loss = loss / N
  return average_loss
```

### More on the sigmoid function

**Question:** Discuss with your neighbour what you think is important when designing a good loss function i.e. what should the loss function do when the model produces a good/bad prediction?

If you are comfortable with the mathematics, show how the **sigmoid** function (and the binary cross-entropy function) is good for evaluating the quality of predictions by using the following questions:

As noted earlier, the sigmoid function is defined as
$$
\mathrm{sigmoid}(a) = \frac{1}{1 + e^{-a}} .
$$
Can you show that
$$
1 - \mathrm{sigmoid}(a) = \frac{1}{1 + e^{a}} ,
$$
and draw both of these on a sheet of paper?

* What is its value when $a = \mathbf{w}^{T}\mathbf{x}$ is positive? negative? and zero?
* What happends to its value when  $a = \mathbf{w}^{T}\mathbf{x}$ becomes larger?
* What is the value of $\mathrm{sigmoid}(\mathbf{w^Tx})$ when $\mathbf{w}^T\mathbf{x} = 0$? How does this change how we classify points on either side of the decision boundary?

**HINT**: Remember the idea of the loss function is to return small values when the classifier makes good predictions and large values when the classifier makes bad predictions.

*If you are not comfortable with the maths*, remember the main aim of this question to **highlight the importance of the binary cross-entropy loss** and **NOT** the math so focus on the concepts!

### Bonus Question
We derived the `compute_loss()` function above based on minimising the log-loss of the prediction error. This is related to a concept called 'cross-entropy'. But another way of deriving exactly the same loss function is by maximising the likelihood of the data under the model $P(y | x, w_1, w_2)$. If you are familiar with this concept (eg. from statistics), see if you can derive it this way as well.

### Optional Further Reading
More information on the [cross-entropy loss](http://ml-cheatsheet.readthedocs.io/en/latest/loss_functions.html) and another interesting connection to [information theory](https://rdipietro.github.io/friendly-intro-to-cross-entropy-loss/).

## Loss value for your chosen $w_1$ and $w_2$

The following line of code computes the loss value for your chosen values of $w_1$ and $w_2$. Try changing the values of $w_1$ and $w_2$ using the sliders. Can you see how a better separation results in a lower loss?

**Note:** If you've used an older version of TensorFlow before, it might be confusing how this code cell works! We explain more about this later...

```
#@title Effect of parameters on the loss {run: "auto"}

# Define the parameters
w1 = -1. #@param { type: "slider", min: -5, max: 5, step: 0.1 }
w2 = -1.8 #@param { type: "slider", min: -5, max: 5, step: 0.1 }

plot_dataset(inputs, labels)

# Add the weight vector to the plot. We plot it in red, as it has to "point"
# in the direction of the red points.
ax = plt.axes()
ax.arrow(0, 0, w1, w2, head_width=0.3, head_length=0.3, fc='r', ec='r')

# Plot part of the decision boundary in black. It is orthogonal to the weight
# vector.
t = 2 # this is how long the line should be
plt.plot([-t * w2, t * w2], [t * w1, -t * w1], 'k-')

plt.xlim([-4, 4])
plt.ylim([-4, 4])

plt.show()

compute_loss(w1, w2, inputs, labels).numpy()
```

## Visualising the loss function

We can visualise the loss function for our dataset by plotting its value at every point in a whole grid of $w_1$ and $w_2$ parameter values. We do this using a **contour plot**, which is a technique for visualising a 3-D function on a 2-D plot by letting colour represent the third dimension. All of the points with the same colour have the same loss value.

```
# We define a function so we can re-use this code later
def plot_contours():
  # Generate a whole bunch of (w1, w2) points in a grid
  ind = np.linspace(-5, 5, 50)
  w1grid, w2grid = np.meshgrid(ind, ind)

  # Compute the loss for each point in the grid
  losses = []
  for w1s, w2s in zip(w1grid, w2grid):
    loss = compute_loss(w1s, w2s, inputs, labels)
    losses.append(loss)

  # Pack the loss values for every value of w1 & w2 into one (50,50) array
  losses_array = np.concatenate(losses).reshape(50,50)

  # Now plot the resulting loss function as a contour plot over the whole grid of (w1, w2) values.
  fig = plt.figure()
  plt.contourf(w1grid, w2grid, losses_array, 20, cmap=plt.cm.jet)
  cbar = plt.colorbar()
  cbar.ax.set_ylabel('Binary cross-entropy loss value')
  plt.xlabel('w1 value')
  plt.ylabel('w2 value')
  plt.title('Total loss for different values of w1 and w2')

plot_contours()
```

As you can see by the bar on the right, as the colour goes from red to blue the loss value gets lower and lower until it reaches the smallest value (dark blue). Since we want the loss to be as small as possible, we want our values of $w_1$ and $w_2$ to produce a loss in the dark blue area of the contour plot. This is achieved within a certain range of values for $w_1$ and $w_2$.

**Question:** Can you read which values of $w_1$ and $w_2$ will give you the smallest loss? Are these values similar to the ones you found earlier to linearly separate the data?

**Question:** Depending on the center point you chose, the smallest value for the loss may not be 0. Under what circumstances is this the case?

## Optimising the loss using TensorFlow

Now that we have a function that gives us the loss for different values of $w_1$ and $w_2$, we want an automated method to find the values that minimise the loss function. This is where optimisation by **gradient descent** comes in. The idea is that for each (batch of) data points, we compute the loss using the current values of $w_1$ and $w_2$ on the data. We then compute the **gradient** (or derivative) of the loss function at the current values of $w_1$ and $w_2$. The negative of the gradient points in the direction of *steepest descent* along the loss function. By adjusting the values of $w_1$ and $w_2$ in the direction of the negative gradient, we move closer towards the minimum of the loss function (provided the loss function is "well behaved"). How big of a step we take is mediated by the **learning rate**. To do this more easily, we will use TensorFlow.

Don't worry if you don't understand what's going on here, you will see this in a lot more detail during the **Mathematics for Machine Learning** lectures!

### Optional extra reading: TensorFlow

TensorFlow (TF) is an open source software library for numerical computation using the concept of Tensors. You can think of Tensors as being a generalisation of matrices to higher dimensions, or roughly equivalent to multi-dimensional arrays. Scalars are 0-dimensional tensors, vectors are 1-dimensional, standard matrices are 2-dimensional, and higher-dimensional tensors have 3 or more dimensions. You can think of dimensions as representing groups of numbers that mean the same thing. For example, for images, we often use 3-dimensional tensors where the first dimension represents the red, green, and blue color channels of the image, and the next two are the columns and rows of pixels of the image.

**Note**: Don't be confused when people say "2-D vector" or "3-D vector", which refers to a 1-dimensional tensor that has size 2 or 3.

The major advantage of using TensorFlow is that it can automatically derive the gradients of many mathematical expressions involving tensors. It achieves this through a process called "automatic differentiation". Tensorflow also supports multiple "kernels", allowing you to easily run your code on normal processors (CPUs), graphics cards (GPUs) and other more exotic hardware accelerators like Google's Tensor Processing Units (TPUs)

Tensorflow actually provides **two modes of operation**, the first, called "graph mode", builds a computation graph upfront and then feeds data into the graph. By building the graph upfront, Tensorflow can apply optimisations to the graph that allow it to extract peak performance from the hardware you're running on. You will have encountered this mode if you used Tensorflow before or attended the Indaba last year! The second mode, called ["Eager-mode"](https://www.tensorflow.org/guide/eager), is a lot newer and evaluates Tensor operations imperatively (in the order you write them), similar to NumPy and PyTorch. Eager-mode is slightly less performant but a lot more intuitive, especially if you've never used a "define-and-run" programming style (like graph mode) before, and is therefore the mode we will use in these practicals.

### Using Tensorflow to optimise the loss
We use TensorFlow to optimise the parameters of the model with gradient descent. We loop over the dataset multiple times (called "epochs") and plot the final decision boundary along with a plot showing how the parameters and loss changed over the epochs.

**Note**: TensorFlow is probably overkill for this example, because the gradient is very easy to calculate, but we introduce it here because it will become essential to calculate the gradients of more complex models in later practicals!

```
lr = 0.25  # The learning rate

# Initialise Tensorflow variables representing our parameters.
# We need to use TensorFlow variables here rather than Numpy or Python ones so
# that TensorFlow is able to compute gradients.
w1 = tf.Variable(-2.0)
w2 = tf.Variable(-4.0)

plot_contours()

# Loop over the dataset multiple times
parameter_values = []
for epoch in range(70):
  plt.scatter(w1.numpy(), w2.numpy(), marker='o', color='black')

  # The GradientTape is how TF keeps track of gradients in Eager-mode
  with tf.GradientTape() as tape:
    loss = compute_loss(w1, w2, inputs, labels)

  # Now we take a step in parameter space in the direction of the gradient to move the parameters closer (hopefully!) to their optimum
  dw1, dw2 = tape.gradient(loss, [w1, w2])

  # Step 'lr units' in the direction of the negative gradient
  # We achieve this by subtracting lr * dw1 and lr * dw2 from the w1 and w2 variables
  w1.assign_sub(lr*dw1)
  w2.assign_sub(lr*dw2)

print('Finished optimisation, the final values of w1 and w2 are:')
print(w1.numpy(), w2.numpy())

# Plot the final point on the loss surface.
plt.scatter(w1.numpy(), w2.numpy(), marker='x', color='red')
plt.show()

# Plot the final decision boundary
plot_dataset(inputs, labels)
ax = plt.axes()
ax.arrow(0, 0, w1.numpy(), w2.numpy(), head_width=0.3, head_length=0.3, fc='r', ec='r')
plt.plot([-2 * w2.numpy(), 2 * w2.numpy()], [2 * w1.numpy(), -2 * w1.numpy()], 'k-')

plt.xlim([-4, 4])
plt.ylim([-4, 4])

plt.show()
```

Can you see how the model steps across the contour plot until it minimises the loss?

How did the final values of $w_1$ and $w_2$ found by Tensorflow correspond to the ones you found manually? If they aren't, can you explain why?

## Optional Tasks
If you've worked through this practical, answered all the questions and feel you have a good understanding of what's going on, try the following tasks:

1. Add a **bias** parameter to the equation for the decision boundary and visualise how that changes the decision boundary, the loss and the ultimate solution found by Tensorflow.
2. Add a **regulariser**, for example, the L2 regulariser (see the appendix below for more information) - how does it affect the contour plot of the parameters vs the loss? How does changing the strength of regularisation affect the loss?

Note: The benefit of using regularisation will be discussed in the next practical!

# Next Steps
Have a look at [2017's practical,](https://github.com/deep-learning-indaba/practicals2017/blob/master/practical1.ipynb) which takes a more "bottom-up" approach, covers more detail on how gradients are computed and also looks at a multi-class classification problem with a non-linear decision boundary.

Note: 2017's practicals use Tensorflow's "graph mode" as opposed to "Eager mode" that we use here.

# Appendix

### L1 and L2 Regularization

Two of the most simple regularization methods are L1 and L2 regularization (you may have heard of them as _Lasso_ and _Ridge_ regression if you've used linear regression before). Both of these methods regularize the model by adding a term to the loss that penalizes the model if it becomes too complex.
L1 regularization adds a term based on the L1 norm:

$loss_{L1} = loss + \lambda \sum_i |w_i|$

where $\lambda$ is a parameter that controls the amount of regularization, and $w_i$ are the parameters of the model. L1 regularization has the effect of forcing some parameters to shrink to 0, effectively removing them from the model.

L2 regularization similarly adds a term based on the L2 norm:

$loss_{L2} = loss + \lambda \sum_i w_i^2$.

L2 regularization has the effect of preventing any of the parameters from becoming too large and _overpowering_ the others.

In some cases it can work well to use both L1 and L2 regularization.

For more information see the articles [here](http://enhancedatascience.com/2017/07/04/machine-learning-explained-regularization/) and [here](https://towardsdatascience.com/l1-and-l2-regularization-methods-ce25e7fc831c).


# Coarrays, MUSIC, and the Cramér-Rao bound

This notebook contains scripts for analyzing the performance of SS-MUSIC and DA-MUSIC for various sparse linear arrays. You can use these scripts to produce figures similar to those in the following papers:

```
@article{wang_coarrays_2017,
    title = {Coarrays, {MUSIC}, and the {Cram\'{e}r}-{Rao} Bound},
    volume = {65},
    issn = {1053-587X},
    doi = {10.1109/TSP.2016.2626255},
    number = {4},
    journal = {IEEE Trans. Signal Process.},
    author = {Wang, M. and Nehorai, A.},
    month = feb,
    year = {2017},
    pages = {933--946}
}
@inproceedings{wang_performance_2017,
    title = {Performance analysis of coarray-based {MUSIC} and the {Cram\'{e}r}-{Rao} bound},
    doi = {10.1109/ICASSP.2017.7952719},
    booktitle = {2017 {IEEE} {International} {Conference} on {Acoustics}, {Speech} and {Signal} {Processing} ({ICASSP})},
    author = {Wang, M. and Zhang, Z. and Nehorai, A.},
    month = mar,
    year = {2017},
    pages = {3061--3065}
}
```

```python
import numpy as np
import doatools.model as model
import doatools.estimation as estimation
import doatools.performance as perf
import matplotlib.pyplot as plt
import itertools
from tqdm import tqdm
%matplotlib inline
```

```python
# Common parameters
wavelength = 1.0 # normalized
d0 = wavelength / 2.0

# Arrays
arrays = [
    model.CoPrimeArray(3, 5, d0, '2m', 'Co-prime Array'),
    model.NestedArray(4, 6, d0, 'Nested Array'),
    model.MinimumRedundancyLinearArray(10, d0, 'MRA')
]
n_arrays = len(arrays)
```

## Numerical demonstration of the analytical MSE

We compare the analytical MSE,
\begin{equation}
\mathrm{MSE}_{\mathrm{an}} = \frac{1}{K} \sum_{k=1}^K \epsilon(\theta_k),
\end{equation}
with the empirical MSE
\begin{equation}
\mathrm{MSE}_{\mathrm{em}} = \frac{1}{KL} \sum_{l=1}^L\sum_{k=1}^K(\hat{\theta}_k^{(l)} - \theta_k^{(l)})^2,
\end{equation}
where $\epsilon(\theta_k)$ is defined in (11), $\theta_k^{(l)}$ is the $k$-th DOA in the $l$-th trial, and $\theta_k^{(l)}$ is the corresponding estimate.

```python
# Setup
sources = model.FarField1DSourcePlacement(np.linspace(-np.pi*6/16, np.pi*5/16, 11))
power_source = 1.0

n_param_n_snapshots = 20
n_param_snr = 20

min_n_snapshots = 10
max_n_snapshots = 1000
min_snr = -20.0
max_snr = 20.0

params_n_snapshots = np.round(np.linspace(min_n_snapshots, max_n_snapshots, n_param_n_snapshots)).astype(np.int32)
params_snr = np.linspace(min_snr, max_snr, n_param_snr)
rv_modes = ['da', 'ss']
```

Here we set `n_repeats` to 100, which means each point in the final plots is averaged from 100 trials. The simulation will take 30-60 minutes depending on your CPU. The resulting plots will be noiser compared with Fig. 2 in the paper.

You need to increase `n_repeats` to at least 5000 to obtain plots as clean as those in Fig. 2. The original scripts were written in MATLAB and utilized `parfor`, which were scalable on clusters. However, implementing `parfor` is not an easy task in Python, and running the full-scale simulations with a single CPU can be very time consuming.

```python
# Run Monte Carlo simulations.
n_repeats = 100

results_em = np.zeros((n_arrays, 2, n_param_n_snapshots, n_param_snr))
results_an = np.zeros((n_arrays, 2, n_param_n_snapshots, n_param_snr))

source_signal = model.ComplexStochasticSignal(sources.size, power_source)
coarray_acm_builders = [estimation.CoarrayACMBuilder1D(arr) for arr in arrays]
root_music = estimation.RootMUSIC1D(wavelength)

# We have 4 varying parameters here. Pardon us for these for loops :)
with tqdm(total=results_em.size) as pbar:
    for dd, array in enumerate(arrays):
        for ss, rv_mode in enumerate(rv_modes):
            for ii, n_snapshots in enumerate(params_n_snapshots):
                for jj, snr in enumerate(params_snr):
                    power_noise = 10**(-snr/10) * np.min(power_source)
                    noise_signal = model.ComplexStochasticSignal(array.size, power_noise)
                    cur_mse = 0
                    for rr in range(n_repeats):
                        _, R = model.get_narrowband_snapshots(array, sources, wavelength, source_signal, noise_signal,
                                                              n_snapshots, return_covariance=True)
                        Rv = coarray_acm_builders[dd](R, rv_mode)
                        _, estimates = root_music.estimate(Rv, sources.size, d0)
                        cur_mse += np.sum((estimates.locations - sources.locations)**2)
                    results_em[dd, ss, ii, jj] = cur_mse / (n_repeats * sources.size)
                    C_an = perf.ecov_coarray_music_1d(array, sources, wavelength, power_source, power_noise, n_snapshots)
                    results_an[dd, ss, ii, jj] = np.mean(np.diag(C_an))
                    pbar.update(1)

rel_errors = np.abs(results_an - results_em) / results_em
```

    100%|██████████████████████████████████████████████████████████| 2400/2400 [27:27<00:00,  1.17s/it]

```python
# Plot the results.
plt.figure(figsize=(7, 8))

bounds = (params_snr[0], params_snr[-1], params_n_snapshots[-1], params_n_snapshots[0])
for dd, array in enumerate(arrays):
    # Left
    ax = plt.subplot(n_arrays, 2, dd * 2 + 1)
    plt.imshow(rel_errors[dd, 0, :, :], extent=bounds, aspect='auto',
               cmap='gray_r', vmin=0.0, vmax=1.0)
    ax.invert_yaxis()
    plt.colorbar()
    if dd == n_arrays - 1:
        plt.xlabel('SNR (dB)')
    plt.ylabel('Number of snapshots')
    plt.title(arrays[dd].name + ' (DA-MUSIC)')

    # Right
    ax = plt.subplot(n_arrays, 2, dd * 2 + 2);
    plt.imshow(rel_errors[dd, 1, :, :], extent=bounds, aspect='auto',
               cmap='gray_r', vmin=0.0, vmax=1.0)
    ax.invert_yaxis()
    plt.colorbar()
    if dd == n_arrays - 1:
        plt.xlabel('SNR (dB)')
    plt.ylabel('Number of snapshots')
    plt.title(arrays[dd].name + ' (SS-MUSIC)')

plt.tight_layout()
plt.show()
```

## Prediction of resolvability

For illustration purpose, we analytically predict the resolvability of the two sources via the following simple criterion:

\begin{equation}
    \epsilon(\theta_1) + \epsilon(\theta_2)
    \underset{\mathrm{Resolvable}}{\overset{\mathrm{Unresovalble}}{\gtreqless}} \Delta\theta.
\end{equation}

```python
# Setups
power_source = 1.0
power_noise = 1.0 # SNR = 0 dB
n_snapshots = 500
n_repeats = 500

theta_0 = np.pi/6
delta_thetas = np.linspace(np.pi/60, np.pi/600, 20);
```

```python
# Helper functions.
def check_doa_correctness(actual, est, tolerance=np.Inf):
    '''Checks if estimated doa is reasonably close to the actual ones.

    Input must be in ascending order within range of (-pi/2, pi/2).

    More precisely, suppose the true DOAs are x_1, x_2, ..., x_k, the estimated
    DOAs must fall within the following regions:
    (-pi/2, x_1 + (x_2 - x_1) / 2),
    (x_1 + (x_2 - x_1) / 2, x_2 + (x_3 - x_2) / 2), ...
    (x_k - (x_k - x_{k-1}) / 2, pi/2)
    If tolerance is not inf, the estimated DOAs must also fall within the following
    regions:
    (x_1 - tolerance, x_1 + tolerance), ...
    (x_k - tolerance, x_k + tolerance)
    '''
    n = actual.size
    # Size mismatches.
    if n != est.size:
        return False
    # Special case: one source only.
    if n == 1:
        return np.abs(actual[0] - est[0]) < tolerance
    # General case
    spaces = np.diff(actual)
    max_deviations = np.minimum(0.5 * spaces, tolerance)
    # First
    if est[0] < max(-np.pi / 2, actual[0] - tolerance) or \
       est[0] > actual[0] + max_deviations[0]:
        return False
    # Last
    if est[-1] > min(np.pi / 2, actual[-1] + tolerance) or \
       est[-1] < actual[-1] - max_deviations[-1]:
        return False
    # Middle parts
    for i in range(1, n - 1):
        if est[i] < actual[i] - max_deviations[i - 1] or \
           est[i] > actual[i] + max_deviations[i]:
            return False
    return True

def check_resolution_ana(array, wavelength, doa1, doa2, power_source, power_noise, n_snapshots, l=1.0):
    '''Checks if two sources are resolvable analytically with SS-MUSIC
    (or DA-MUSIC) using the analytical asymptotic MSE we derived in (11)
    after Theorem 1.
    '''
    # Ensure doa1 < doa2
    if doa1 > doa2:
        doa1, doa2 = doa2, doa1
    sources = model.FarField1DSourcePlacement([doa1, doa2])
    C = perf.ecov_coarray_music_1d(array, sources, wavelength, power_source, power_noise, n_snapshots)
    # Apply the resolution criterion
    return doa2 - doa1 > l * (np.sqrt(C[0, 0]) + np.sqrt(C[1, 1]))
```

```python
# Run Monte Carlo simulations.
resolution_success_rate = np.zeros((n_arrays, delta_thetas.size))
resolution_threshold_ana = np.zeros((n_arrays,))

coarray_acm_builders = [estimation.CoarrayACMBuilder1D(arr) for arr in arrays]
root_music = estimation.RootMUSIC1D(wavelength)

with tqdm(total=resolution_success_rate.size) as pbar:
    for dd, array in enumerate(arrays):
        source_signal = model.ComplexStochasticSignal(2, power_source)
        noise_signal = model.ComplexStochasticSignal(array.size, power_noise)
        # Marks unresolvable separations predicted by the analytical results.
        unresolvable_flags = np.zeros((delta_thetas.size,))
        for ii, delta_theta in enumerate(delta_thetas):
            theta_1 = theta_0 - 0.5 * delta_theta
            theta_2 = theta_0 + 0.5 * delta_theta
            # Analytically check the minimal resolvable delta.
            unresolvable_flags[ii] = not check_resolution_ana(array, wavelength, theta_1, theta_2,
                                                              power_source, power_noise, n_snapshots)
            # Collect the empirical success rate.
            n_successes = 0
            sources = model.FarField1DSourcePlacement([theta_1, theta_2])
            for rr in range(n_repeats):
                _, R = model.get_narrowband_snapshots(array, sources, wavelength, source_signal, noise_signal,
                                                      n_snapshots, return_covariance=True)
                Rv = coarray_acm_builders[dd](R, 'ss')
                _, estimates = root_music.estimate(Rv, 2, array.d0)
                # Requires accurate resolution.
                if check_doa_correctness(sources.locations, estimates.locations, 0.5 * delta_theta):
                    n_successes += 1
            resolution_success_rate[dd, ii] = n_successes / n_repeats
            pbar.update(1)
        min_delta_ana_idx = np.argmax(unresolvable_flags > 0)
        if unresolvable_flags[min_delta_ana_idx] == 0:
            resolution_threshold_ana[dd] = delta_thetas[0]
        else:
            resolution_threshold_ana[dd] = delta_thetas[min_delta_ana_idx]
```

    100%|██████████████████████████████████████████████████████████████| 60/60 [03:22<00:00,  4.13s/it]

```python
# Plot the results.
plt.figure()
markers = ['+', 'o', 'd', '*']
for dd, array in enumerate(arrays):
    plt.plot(np.rad2deg(delta_thetas), resolution_success_rate[dd,:], '-' + markers[dd],
                 label=array.name, color='C' + str(dd))
    plt.gca().set_ylim((0, 1))
    ana_threshold = np.rad2deg(resolution_threshold_ana[dd])
    plt.plot([ana_threshold, ana_threshold], [0, 1], '--',  color='C' + str(dd),
             label=array.name + ' Predicted')
plt.xlabel('Separation (degree)')
plt.ylabel('Rate of success')
plt.title('Probability of resolution')
plt.legend()
plt.grid(True)
plt.show()
```

## Efficiency analysis

We study the asymptotic statistical efficiency of DA-MUSIC and SS-MUSIC under different array geometries and parameter settings. We define their average efficiency as

\begin{equation}
    \kappa = \frac{\mathrm{tr}(\mathrm{CRB}_{\boldsymbol{\theta}})}{\sum_{k=1}^K \epsilon(\theta_k)}.
\end{equation}

For efficient estimators we expect $\kappa = 1$, while for inefficient estimators we expect $0 \leq \kappa < 1$.

```python
# Setup
n_snapshots = 1 # normalized
power_source = 1.0
snrs = np.linspace(-10, 20, 30)
n_snrs = snrs.size

source_numbers = [1, 6, 12]
n_experiments = len(source_numbers)
```

```python
# Evaluate the statistical efficiency under difference SNRs.
e1 = np.zeros((n_arrays, n_snrs, n_experiments))
e2 = np.zeros((n_arrays, n_snrs, n_experiments))
for kk, n_sources in enumerate(source_numbers):
    sources = model.FarField1DSourcePlacement(
        np.linspace(-np.pi / 3, np.pi / 3, n_sources))
    for dd, array in enumerate(arrays):
        for ii, snr in enumerate(snrs):
            power_noise = power_source * 10**(-snr/10)
            # Asymptotic covariance matrix
            C = perf.ecov_coarray_music_1d(array, sources, wavelength, power_source,
                                           power_noise, n_snapshots)
            # CRB
            B = perf.crb_stouc_farfield_1d(array, sources, wavelength, power_source,
                                           power_noise, n_snapshots)
            e1[dd, ii, kk] = np.sum(np.diag(C))
            e2[dd, ii, kk] = np.sum(np.diag(B))

efficiency = e2 / e1
```

```python
# Plot
plt.figure(figsize=(6, 10))
for kk, n_sources in enumerate(source_numbers):
    plt.subplot(n_experiments, 1, kk + 1)
    markers = ['+', 'o', 'd', '*']
    for dd, array in enumerate(arrays):
        hp = plt.plot(snrs, efficiency[dd, :, kk], '-' + markers[dd], label=array.name)
    plt.xlabel('SNR')
    plt.ylabel('Average efficiency')
    plt.axis([-10, 20, 0, 1])
    plt.title('K = {0:d}'.format(n_sources))
    plt.legend()
plt.tight_layout()
plt.show()
```


# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Inverse Transform

So far only the [(forward) Laplace transform](definition.ipynb) has been introduced. The Laplace transform features also an [inverse Laplace transform](https://en.wikipedia.org/wiki/Inverse_Laplace_transform). It can be shown that the inverse Laplace transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ is uniquely determined for practically relevant signals. The Laplace transform maps a complex-valued signal $x(t) \in \mathbb{C}$ with real-valued independent variable $t \in \mathbb{R}$ into its complex-valued Laplace transform $X(s) \in \mathbb{C}$ with complex-valued independent variable $s \in \mathbb{C}$. This section discusses two techniques to compute the inverse Laplace transform.

### Integral Formula

Using results from complex analysis, the inverse Laplace transform is given by the following complex line integral

\begin{equation}
x(t) = \frac{1}{2 \pi j} \int_{\sigma - j \infty}^{\sigma + j \infty} X(s) \, e^{s t} \; ds
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$ is assumed to be analytic in its simply connected region of convergence (ROC). The notation $\sigma \mp j \infty$ for the lower/upper integration limit denotes an arbitrary integration path which lies in the ROC and ranges from $\Im \{s\} = - \infty$ to $\Im \{s\} = + \infty$. The integration path can be chosen parallel to the imaginary axis but also all other paths in the ROC are possible. This results from [Cauchy's integral theorem](https://en.wikipedia.org/wiki/Cauchy's_integral_theorem). Two equivalent paths are shown in the following illustration

where the blue line indicates the integration path and the gray area the ROC.

### Rational Laplace Transforms

Computing the inverse Laplace transform by above integral formula can be challenging. For Laplace transforms $X(s) = \mathcal{L} \{ x(t) \}$ which are given as rational functions, the [Cauchy residue theorem](https://en.wikipedia.org/wiki/Residue_theorem) provides a practically tractable solution. It states that the value of a line integral of an holomorphic function over a closed contour is given by summing up its [residues](https://en.wikipedia.org/wiki/Residue_theorem). The residue is the value of the line integral for a path enclosing a singularity. Consequently, the inverse Laplace transform of a rational Laplace transform can be computed by summing up the individual contributions from its poles. This procedure is detailed in the following.

#### Basic Procedure

A rational Laplace transform $X(s)$ can be written in terms of its numerator and denominator polynomial

\begin{equation}
X(s) = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n}
\end{equation}

where $M$, $N$ denote the order of the numerator and denominator polynomial and $\beta_m$, $\alpha_n$ their coefficients, respectively. It is assumed that $\alpha_N \neq 0$ and that $M \leq N$. If $M > N$, $X(s)$ can be decomposed by [polynomial division](https://en.wikipedia.org/wiki/Polynomial_long_division) into a sum of powers of $s$ and a rational function fulfilling $M \leq N$.

Now a [partial fraction decomposition](https://en.wikipedia.org/wiki/Partial_fraction_decomposition) of $X(s)$ is performed resulting in

\begin{equation}
X(s) = A_0 + \sum_{\mu = 1}^{L} \sum_{\nu = 1}^{R_\mu} \frac{A_{\mu \nu}}{(s - s_{\infty \mu})^\nu}
\end{equation}

where $s_{\infty \mu}$ denotes the $\mu$-th unique pole of $X(s)$, $R_\mu$ its degree and $L$ the total number of different poles $\mu = 1 \dots L$. Using the known Laplace transforms ([cf. table of selected transforms](table_theorems_transforms.ipynb#Selected-Transforms))

\begin{equation}
\mathcal{L} \{ t^n e^{-s_0 t} \epsilon(t) \} = \frac{n!}{(s + s_0)^{n+1}} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

and $\mathcal{L} \{ \delta(t) \} = 1$ together with the linearity of the Laplace transform yields a generic result for the inverse Laplace transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of a right-sided signal

\begin{equation}
x(t) = A_0 \cdot \delta(t) + \epsilon(t) \sum_{\mu = 1}^{L} e^{s_{\infty \mu} t} \sum_{\nu = 1}^{R_\mu} \frac{A_{\mu \nu} \, t^{\mu - 1}}{(\nu -1)!}
\end{equation}

It remains to compute the coefficients $A_0$ and $A_{\mu \nu}$ of the partial fraction decomposition. The constant coefficient $A_0$ is given as

\begin{equation}
A_0 = \lim_{s \to \infty} X(s)
\end{equation}

For a pole $s_{\infty \mu}$ with degree $R_\mu = 1$, the coefficient $A_{\mu 1}$ reads

\begin{equation}
A_{\mu 1} = \lim_{s \to s_{\infty \mu}} \left( X(s) \cdot (s - s_{\infty \mu}) \right)
\end{equation}

For a pole $s_{\infty \mu}$ of higher degree $R_\mu > 1$, the coefficients $A_{\mu \nu}$ are given as

\begin{equation}
A_{\mu \nu} = \frac{1}{(R_\mu - \nu)!} \lim_{s \to s_{\infty \mu}} \frac{d^{R_\mu - \nu}}{d s^{R_\mu - \nu}} \left( X(s) \cdot (s - s_{\infty \mu})^{R_\mu} \right)
\end{equation}

#### Classification of Poles

Above procedure allows to compute the inverse Laplace transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of a rational Laplace transform $X(s)$ in a systematic way. It is well suited for an algorithmic realization. However, for manual calculus it is useful to classify the poles with respect to their location in the $s$-plane and their symmetries. The classification can then be used to formulate a modified partial fraction decomposition which limits the need for later algebraic simplification of the inverse Laplace transform. Three classes of poles are typically considered

| Type | Pole-Zero Diagramm | $X(s)$ | $x(t) = \mathcal{L}^{-1} \{ X(s) \} \qquad \qquad$ |
|---|:---:|:---:|:---:|
| Single complex pole |  | $\frac{n!}{(s + s_0)^{n+1}}$ | $t^n e^{-s_0 t} \epsilon(t)$ |
| Conjugated imaginary poles |  | $\frac{A s + B}{s^2 + \omega_0^2}$ | $\begin{cases} \sin(\omega_0 t) \epsilon(t) \\ \cos(\omega_0 t) \epsilon(t) \end{cases}$ |
| Conjugated complex poles |  | $\frac{A s + B}{(s + \sigma_0)^2 + \omega_0^2}$ | $\begin{cases} e^{-\sigma_0 t} \sin(\omega_0 t) \epsilon(t) \\ e^{-\sigma_0 t} \cos(\omega_0 t) \epsilon(t) \end{cases}$ |

where $s_0 \in \mathbb{C}$ and $\omega_0, \sigma_0 \in \mathbb{R}$. The expansion coefficients $A, B \in \mathbb{R}$ can be derived by comparison of coefficients. Whether $x(t)$ contains a sine or cosine depends on the coefficient $A$. If $A \neq 0$ then $x(t)$ contains a cosine.

#### Example

The inverse transform $x(t) = \mathcal{L}^{-1} \{ X(s) \}$ of

\begin{equation}
H(s) = \frac{2 s^2 + 14 s + 124}{s^3 + 8 s^2 + 46 s + 68} \qquad \text{for } \Re \{s \} > -2
\end{equation}

is computed. First the function $X(s)$ is defined in `SymPy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

s = sym.symbols('s', complex=True)
t = sym.symbols('t', real=True)

X = (2*s**2 + 14*s + 124)/(s**3 + 8 * s**2 + 46*s + 68)
X
```

The poles of $X(s)$ are derived by computing the roots of the denominator polynomial

```python
poles = sym.roots(sym.denom(X))
poles
```

The result is a real-valued pole and a conjugate complex pair of poles. According to above introduced classification of poles, the following ansatz is chosen for the partial fraction decomposition of the Laplace transform

\begin{equation}
X_p(s) = \frac{A}{s + 2} + \frac{B s + C}{s^2 + 6s + 34}
\end{equation}

The coefficients $A, B, C \in \mathbb{R}$ are derived by equating coefficients with $X(s)$

```python
A, B, C = sym.symbols('A B C', real=True)

Xp = A / (s+2) + (B*s + C)/(s**2 + 6*s + 34)
coeffs = sym.solve(sym.Eq(X, Xp), (A, B, C))
coeffs
```

Introducing the coefficients into $X_p(s)$ yields

```python
Xp = Xp.subs(coeffs)
Xp
```

The first fraction belongs to the complex conjugate poles. Applying [completion of the square](https://en.wikipedia.org/wiki/Completing_the_square) to the denominator, its inverse can be identified in the [table of Laplace transforms](table_theorems_transforms.ipynb#Transforms) as exponentially decaying cosine signal. Performing the inverse Laplace transform with `SymPy` yields

```python
x1 = sym.inverse_laplace_transform(Xp.args[1], s, t)
x1
```

The second fraction belongs to a real-valued pole of first degree. Its inverse Laplace transform can be looked-up directly in the [table of Laplace transforms](table_theorems_transforms.ipynb#Transforms) as exponentially decaying signal. Performing the inverse Laplace transform again with `SymPy` yields

```python
x2 = sym.inverse_laplace_transform(Xp.args[0], s, t)
x2
```

The inverse Laplace transform of $X(s)$ is given by summing up these two parts

```python
x = x1 + x2
x
```

The resulting signal is plotted for illustration

```python
sym.plot(x, (t, -0.1, 4), xlabel='$t$', ylabel='$x(t)$');
```

The same result can be derived directly from $X(s)$ by using the inverse Laplace transform of `SymPy`

```python
sym.inverse_laplace_transform(X, s, t).simplify()
```

**Exercise**

* Derive the inverse Laplace transform of $X(s)$ by manual calculation.
* Derive the inverse Laplace transform of
    \begin{equation}
    X(s) = \frac{1}{(s+1)(s+2)^2} \qquad \text{for } \Re \{ s \} \geq -1
    \end{equation}
    by manual calculation. Check your results with `SymPy`.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.