# Занятие 10
# Прикладная алгебра и численные методы
## Матричные нормы. Приближение матриц меньшего ранга

```
import numpy as np
import sympy
import scipy.linalg
import numpy.linalg
import pandas as pd
import matplotlib.pyplot as plt
from google.colab import files
```

```
#!python -m pip install scipy --upgrade
```

```
sympy.__version__, scipy.__version__, np.__version__
```

    ('1.7.1', '1.6.1', '1.19.5')

## Подчиненные (операторные) матричные нормы.

https://numpy.org/doc/stable/reference/generated/numpy.linalg.norm.html

numpy.linalg.norm(x, ord=None, axis=None, keepdims=False)

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.norm.html

scipy.linalg.norm(a, ord=None, axis=None, keepdims=False, check_finite=True)

если check_finite=True, проверяется, все ли элементы матрицы конечны

#### Возможные значения параметра ord
одинаково в numpy и scipy
$$
\begin{matrix}
ord &	norm\ for\ matrices\\
None &		Frobenius\ norm\\
‘fro’	 &	Frobenius\ norm\\
‘nuc’ &		nuclear\ norm\\
inf &		max(sum(abs(x),\ axis=1))\\
-inf &		min(sum(abs(x),\ axis=1))\\
0	 &	–\\
1	 &	max(sum(abs(x),\ axis=0))\\
-1 &		min(sum(abs(x),\ axis=0))\\
2	 &	2-norm\ (largest\ singular\ value)\\
-2 &		smallest\ singular\ value\\
other &		–
\end{matrix}
$$

Матричная норма
$ \|A\|_{1}=\max \limits _{1\leq j\leq n}\sum _{i=1}^{m}|a_{ij}|$, подчинённая векторной норме$ \|x\|_{1}=\sum _{i=1}^{n}|x_{i}|$.

Матричная норма $\|A\|_{\infty }=\max \limits _{1\leq i\leq m}\sum _{j=1}^{n}|a_{ij}|$, подчинённая векторной норме $\|x\|_{\infty }=\max \limits _{1\leq i\leq n}|x_{i}|$.

Спектральная норма $\|A\|_{2}=\sup \limits _{\|x\|_{2}=1}\|Ax\|_{2}=\sup \limits _{(x,x)=1}{\sqrt {(Ax,Ax)}}={\sqrt {\lambda _{max}(A^{*}A)}}$, подчиненная векторной норме $\|x\|_{2}={\sqrt {\sum _{i=1}^{n}|x_{i}|^{2}}}$.

Норма Фробениуса $ \|A\|_{F} = \sqrt{\sum _{i, j}|a_{ij}|^2} = {\rm tr}(A^*A) = \sqrt{\sum _{i=1}^{n}\sigma_{i}^2}$

https://docs.sympy.org/latest/modules/matrices/matrices.html#sympy.matrices.matrices.MatrixBase.norm

В sympy реализованы нормы с такими же значениями ord, как в scipy и numpy:

norm(ord=None)

это метод в классе матриц.

## Задание 1.
Вычислить матричные нормы $||M||_1$, $||M||_2$, $||M||_\infty$ и $||M||_F$  для матрицы из файла sem10task1.xlsx.

Использовать scipy, numpy для всех норм, sympy для $||M||_1$ и $||M||_\infty$.

```
uploaded = files.upload()

for fn in uploaded.keys():

  print('User uploaded file "{name}" with length {length} bytes'.format(name=fn, length=len(uploaded[fn])))
```

<input type="file" id="files-8ccc24fe-9084-43c1-9b07-f71055313401" name="files[]" multiple disabled
   style="border:none" />
<output id="result-8ccc24fe-9084-43c1-9b07-f71055313401">
 Upload widget is only available when the cell has been executed in the
 current browser session. Please rerun this cell to enable.
 </output>

    Saving sem10task1.xlsx to sem10task1.xlsx
    User uploaded file "sem10task1.xlsx" with length 10145 bytes

```

```

## Спектр матрицы - множество ее собственных значений.
https://numpy.org/doc/stable/reference/generated/numpy.linalg.eigvals.html

numpy.linalg.eigvals(a)

https://docs.scipy.org/doc/scipy/reference/generated/scipy.linalg.eigvals.html

scipy.linalg.eigvals(a, b=None, overwrite_a=False, check_finite=True, homogeneous_eigvals=False)
## Задание 2
Пусть $A_2$ - матрица, состоящая из первых десяти строк и первых десяти столбцов матрицы Задания 1.
С помощью разложения SVD (scipy.linalg.svdvals) получить сингулярные числа матрицы $A_2$. Изобразить на комплексной плоскости сингулярные числа, спектр матрицы $A_2$ и окружность с радиусом, равным спектральному радиусу матрицы $A_2$.

Вывести на экран спектр, спектральный радиус и сингулярные числа.

```

```

## Задание 3
Пусть $A_3$ - матрица, составленная из четных строк и нечетных столбцов матрицы $A_1$ (считаем, что нумерация с единицы!).

Проверить утверждение $||A||_F = \sqrt{\sigma_1^2 + \sigma_1^2 + ... + \sigma_1^n}$ на матрицах $A_1$, $A_2$ и $A_3$.

Использовать цикл, не копировать код!

```

```

## Приближенное разложение меньшего ранга
$$
A_{n\times n} = A_{n\times r} \times  A_{r\times n},\quad
r - \mbox{ ранг матрицы А}.
$$
Пусть SVD-разложение матрицы $A = U\Sigma V^*$, где
$$
\Sigma =
\left(
\begin{matrix}
\sigma_1 & 0 & 0 & ... & 0\\
0 & \sigma_2 & 0 & ... & 0 \\
0 &0 & ...& ... & 0\\
0 & ... & ...& ... & 0 \\
0 &0 & ... & ... & \sigma_n
\end{matrix}
\right),
$$
тогда $A_r = U\Sigma_r V^*$, где
$$
\Sigma_r  =
\left(
\begin{matrix}
\sigma_1 & 0 & 0 & ... & ... & 0\\
0 & \sigma_2 & 0 & ... & ... &0 \\
0 &0 & ...& ... & ... & 0\\
0 & ... & ...& \sigma_r & ... & 0 \\
0 &0 & ... & ... &... & 0\\
 ...  & ...  & ... & ... &... &  ... \\
0 &0 & ... & ... &... & 0\\
\end{matrix}
\right),
$$
## Задание 4
Построить приближение ранга 7 для матрицы А2 в спектральной норме, найти норму разности матриц.

```

```


# Demo - 1D Poisson's equation

**Mikael Mortensen** (email: `mikaem@math.uio.no`), Department of Mathematics, University of Oslo.

Date: **April 13, 2018**

**Summary.** This is a demonstration of how the Python module [shenfun](https://github.com/spectralDNS/shenfun) can be used to solve Poisson's
equation with Dirichlet boundary conditions in one dimension. Spectral convergence, as
shown in the figure below, is demonstrated.
The demo is implemented in
a single Python file [dirichlet_poisson1D.py](https://github.com/spectralDNS/shenfun/blob/master/demo/dirichlet_poisson1D.py), and
the numerical method is is described in more detail by J. Shen [[shen1]](#shen1) and [[shen95]](#shen95).

Convergence of 1D Poisson solvers for both Legendre and Chebyshev modified basis function.

## Poisson's equation

Poisson's equation is given as

$$
\begin{equation}
\nabla^2 u(x) = f(x) \quad \text{for }\, x \in [-1, 1], \label{eq:poisson} \tag{1}
\end{equation}
$$

$$
\begin{equation}
u(-1)=a, u(1)=b, \notag
\label{_auto1} \tag{2}
\end{equation}
$$

where $u(x)$ is the solution, $f(x)$ is a function and $a, b$ are two possibly
non-zero constants.

To solve Eq. ([1](#eq:poisson)) with the Galerkin method we need smooth continuously
differentiable basis functions, $v_k$, that satisfy the given boundary conditions.
And then we look for solutions like

$$
\begin{equation}
u(x) = \sum_{k=0}^{N-1} \hat{u}_k v_k(x), \label{eq:u} \tag{3}
\end{equation}
$$

where $N$ is the size of the discretized problem,
$\hat{\mathbf{u}} = \{\hat{u}_k\}_{k=0}^{N-1}$ are the unknown expansion
coefficients, and the function space is $\text{span}\{v_k\}_{k=0}^{N-1}$.

The basis functions of the function space can, for example,  be constructed from
[Chebyshev](https://en.wikipedia.org/wiki/Chebyshev_polynomials), $T_k(x)$, or
[Legendre](https://en.wikipedia.org/wiki/Legendre_polynomials), $L_k(x)$, polynomials
and we use the common notation $\phi_k(x)$ to represent either one of them. It turns out that
it is easiest to use basis functions with homogeneous Dirichlet boundary conditions

$$
\begin{equation}
v_k(x) = \phi_k(x) - \phi_{k+2}(x),
\label{_auto2} \tag{4}
\end{equation}
$$

for $k=0, 1, \ldots N-3$. This gives the function space
$V^N_0 = \text{span}\{v_k(x)\}_{k=0}^{N-3}$.
We can then add two more linear basis functions (that belong to the kernel of Poisson's equation)

$$
\begin{equation}
v_{N-2} = \frac{1}{2}(\phi_0 - \phi_1),
\label{_auto3} \tag{5}
\end{equation}
$$

$$
\begin{equation}
v_{N-1} = \frac{1}{2}(\phi_0 + \phi_1).
\label{_auto4} \tag{6}
\end{equation}
$$

which gives the inhomogeneous space $V^N = \text{span}\{v_k\}_{k=0}^{N-1}$.
With the two linear basis functions it is easy to see that the last two degrees
of freedom, $\hat{u}_{N-2}$ and $\hat{u}_{N-1}$, now are given as

$$
\begin{equation}
u(-1) = \sum_{k=0}^{N-1} \hat{u}_k v_k(-1) = \hat{u}_{N-2} = a,
\label{eq:dirichleta} \tag{7}
\end{equation}
$$

$$
\begin{equation}
u(+1) = \sum_{k=0}^{N-1} \hat{u}_k v_k(+1) = \hat{u}_{N-1} = b,
\label{eq:dirichletb} \tag{8}
\end{equation}
$$

and, as such, we only have to solve for $\{\hat{u}_k\}_{k=0}^{N-3}$, just like
for a problem with homogeneous boundary conditions (for homogeneous boundary condition
we simply have $\hat{u}_{N-2} = \hat{u}_{N-1} = 0$).
We now formulate a variational problem using the Galerkin method: Find $u \in V^N$ such that

$$
\begin{equation}
\int_{-1}^1 \nabla^2 u \, v \, w\, dx = \int_{-1}^1 f \, v\, w\, dx \quad \forall v \, \in \, V^N_0. \label{eq:varform} \tag{9}
\end{equation}
$$

Note that since we only have $N-3$ unknowns we are only using the homogeneous test
functions from $V^N_0$.

The weighted integrals, weighted by $w(x)$, are called inner products, and a
common notation is

$$
\begin{equation}
\int_{-1}^1 u \, v \, w\, dx = \left( u, v\right)_w.
\label{_auto5} \tag{10}
\end{equation}
$$

The integral can either be computed exactly, or with quadrature. The advantage
of the latter is that it is generally faster, and that non-linear terms may be
computed just as quickly as linear. For a linear problem, it does not make much
of a difference, if any at all. Approximating the integral with quadrature, we
obtain

$$
\begin{align*}
\int_{-1}^1 u \, v \, w\, dx &\approx \left( u, v \right)_w^N, \\
&\approx \sum_{j=0}^{N-1} u(x_j) v(x_j) w(x_j),
\end{align*}
$$

where $\{w(x_j)\}_{j=0}^{N-1}$ are quadrature weights.
The quadrature points $\{x_j\}_{j=0}^{N-1}$
are specific to the chosen basis, and even within basis there are two different
choices based on which quadrature rule is selected, either Gauss or Gauss-Lobatto.

Inserting for test and trialfunctions, we get the following bilinear form and
matrix $A\in\mathbb{R}^{N-2\times N-2}$ for the Laplacian (using the
summation convention in step 2)

$$
\begin{align*}
\left( \nabla^2u, v \right)_w^N &= \left( \nabla^2\sum_{k=0}^{N-3}\hat{u}_k v_{k}, v_j \right)_w^N, \quad j=0,1,\ldots, N-3\\
    &= \left(\nabla^2 v_{k}, v_j \right)_w^N \hat{u}_k, \\
    &= a_{jk} \hat{u}_k.
\end{align*}
$$

Note that the sum in $a_{jk} \hat{u}_{k}$ runs over $k=0, 1, \ldots, N-3$ since
the second derivatives of $v_{N-1}$ and $v_{N}$ are zero.
The right hand side linear form and vector is computed as $\tilde{f}_j = (f,
v_j)_w^N$, for $j=0,1,\ldots, N-3$, where a tilde is used because this is not
a complete transform of the function $f$, but only an inner product.

The linear system of equations to solve for the expansion coefficients
of $u(x)$ is given as

$$
\begin{equation}
A \hat{\mathbf{u}} = \tilde{\mathbf{f}}.
\label{_auto6} \tag{11}
\end{equation}
$$

Now, when the expansion coefficients $\hat{\mathbf{u}}$ are found by
solving this linear system, they may be
transformed to real space $u(x)$ using ([3](#eq:u)), and here the contributions
from $\hat{u}_{N-2}$ and $\hat{u}_{N-1}$ must be accounted for. Note that the matrix
$A$ (different for Legendre or Chebyshev) has a very special structure that
allows for a solution to be found very efficiently in order of $\mathcal{O}(N)$
operations, see [[shen1]](#shen1) and [[shen95]](#shen95). These solvers are implemented in
shenfun for both bases.

## Method of manufactured solutions

In this demo we will use the method of manufactured
solutions to demonstrate spectral accuracy of the `shenfun` Dirichlet bases. To
this end we choose an analytical function that satisfies the given boundary
conditions:

$$
\begin{equation}
u_e(x) = \sin(k\pi x)(1-x^2) + a(1-x)/2 + b(1+x)/2, \label{eq:u_e} \tag{12}
\end{equation}
$$

where $k$ is an integer and $a$ and $b$ are constants. Now, feeding $u_e$ through
the Laplace operator, we see that the last two linear terms disappear, whereas the
first term results in

$$
\begin{equation}
 \nabla^2 u_e(x) = \frac{d^2 u_e}{dx^2},
\label{_auto7} \tag{13}
\end{equation}
$$

$$
\begin{equation}
                  = -4k \pi x \cos(k\pi x) - 2\sin(k\pi x) - k^2 \pi^2 (1 -
x^2) \sin(k \pi x). \label{eq:solution} \tag{14}
\end{equation}
$$

Now, setting $f_e(x) = \nabla^2 u_e(x)$ and solving for $\nabla^2 u(x) = f_e(x)$,
we can compare the numerical solution $u(x)$ with the analytical solution $u_e(x)$
and compute error norms.

## Implementation

### Preamble

We will solve Poisson's equation using the [shenfun](https://github.com/spectralDNS/shenfun) Python module. The first thing needed
is then to import some of this module's functionality
plus some other helper modules, like [Numpy](https://numpy.org) and [Sympy](https://sympy.org):

```
from shenfun import inner, div, grad, TestFunction, TrialFunction, Function, \
    project, Dx, Array, FunctionSpace, dx
import numpy as np
from sympy import symbols, cos, sin, exp, lambdify
```

We use `Sympy` for the manufactured solution and `Numpy` for testing.

### Manufactured solution

The exact solution $u_e(x)$ and the right hand side $f_e(x)$ are created using
`Sympy` as follows

```
a = -1
b = 1
k = 4
x = symbols("x")
ue = sin(k*np.pi*x)*(1-x**2) + a*(1 - x)/2. + b*(1 + x)/2.
fe = ue.diff(x, 2)
```

These solutions are now valid for a continuous domain. The next step is thus to
discretize, using a discrete mesh $\{x_j\}_{j=0}^{N-1}$ and a finite number of
basis functions.

Note that it is not mandatory to use `Sympy` for the manufactured solution. Since the
solution is known ([14](#eq:solution)), we could just as well simply use `Numpy`
to compute $f_e$ at $\{x_j\}_{j=0}^{N-1}$. However, with `Sympy` it is much
easier to experiment and quickly change the solution.

### Discretization

We create a basis with a given number of basis functions, and extract the computational
mesh from the basis itself

```
N = 32
SD = FunctionSpace(N, 'Chebyshev', bc=(a, b))
#SD = FunctionSpace(N, 'Legendre', bc=(a, b))
```

Note that we can either choose a Legendre or a Chebyshev basis.

### Variational formulation

The variational problem ([9](#eq:varform)) can be assembled using `shenfun`'s
[TrialFunction](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.TrialFunction), [TestFunction](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.TestFunction) and [inner()](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.inner.inner) functions.

```
u = TrialFunction(SD)
v = TestFunction(SD)
# Assemble left hand side matrix
A = inner(v, div(grad(u)))
# Assemble right hand side
fj = Array(SD, buffer=fe)
f_hat = Function(SD)
f_hat = inner(v, fj, output_array=f_hat)
```

Note that the `sympy` function `fe` can be used to initialize the [Array](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Array)
`fj`. We wrap this Numpy array in an [Array](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.arguments.Array) class
(`fj = Array(SD, buffer=fe)`), because an Array
is required as input to the [inner()](https://shenfun.readthedocs.io/en/latest/shenfun.forms.html#shenfun.forms.inner.inner) function.

### Solve linear equations

Finally, solve linear equation system and transform solution from spectral
$\{\hat{u}_k\}_{k=0}^{N-1}$ vector to the real space $\{u(x_j)\}_{j=0}^{N-1}$
and then check how the solution corresponds with the exact solution $u_e$.
To this end we compute the $L_2$-errornorm using the `shenfun` function
[dx()](https://shenfun.readthedocs.io/en/latest/shenfun.utilities.html#shenfun.utilities.dx)

```
u_hat = A.solve(f_hat)
uj = SD.backward(u_hat)
ua = Array(SD, buffer=ue)

print("Error=%2.16e" %(np.sqrt(dx((uj-ua)**2))))
```

### Convergence test

To do a convergence test we will now create a function `main`, that takes the
number of quadrature points as parameter, and prints out
the error.

```
def main(N, family='Chebyshev'):
    SD = FunctionSpace(N, family=family, bc=(a, b))
    u = TrialFunction(SD)
    v = TestFunction(SD)

    # Get f on quad points
    fj = Array(SD, buffer=fe)

    # Compute right hand side of Poisson's equation
    f_hat = Function(SD)
    f_hat = inner(v, fj, output_array=f_hat)

    # Get left hand side of Poisson's equation
    A = inner(v, div(grad(u)))

    f_hat = A.solve(f_hat)
    uj = SD.backward(f_hat)

    # Compare with analytical solution
    ua = Array(SD, buffer=ue)
    l2_error = np.linalg.norm(uj-ua)
    return l2_error
```

For example, we find the error of a Chebyshev discretization
using 12 quadrature points as

```
main(12, 'Chebyshev')
```

To get the convergence we call `main` for a list
of $N=[12, 16, \ldots, 48]$, and collect the errornorms in
arrays to be plotted. The error can be plotted using
[matplotlib](https://matplotlib.org), and the generated
figure is also shown in this demos summary.

```
%matplotlib inline

import matplotlib.pyplot as plt

N = range(12, 50, 4)
error = {}
for basis in ('legendre', 'chebyshev'):
    error[basis] = []
    for i in range(len(N)):
        errN = main(N[i], basis)
        error[basis].append(errN)

plt.figure(figsize=(6, 4))
for basis, col in zip(('legendre', 'chebyshev'), ('r', 'b')):
    plt.semilogy(N, error[basis], col, linewidth=2)
plt.title('Convergence of Poisson solvers 1D')
plt.xlabel('N')
plt.ylabel('Error norm')
plt.legend(('Legendre', 'Chebyshev'))
plt.show()
```

The spectral convergence is evident and we can see that
after $N=40$ roundoff errors dominate as the errornorm trails off around $10^{-14}$.

## Complete solver

A complete solver, that can use either Legendre or Chebyshev bases, chosen as a
command-line argument, can also be found [here](https://github.com/spectralDNS/shenfun/blob/master/demo/dirichlet_poisson1D.py).

1.  **J. Shen**.
    Efficient Spectral-Galerkin Method I. Direct Solvers of Second- and Fourth-Order Equations Using Legendre Polynomials,
    *SIAM Journal on Scientific Computing*,
    15(6),
    pp. 1489-1505,
    [doi: 10.1137/0915089](http://dx.doi.org/10.1137/0915089),
    1994.

2.  **J. Shen**.
    Efficient Spectral-Galerkin Method II. Direct Solvers of Second- and Fourth-Order Equations Using Chebyshev Polynomials,
    *SIAM Journal on Scientific Computing*,
    16(1),
    pp. 74-87,
    [doi: 10.1137/0916006](http://dx.doi.org/10.1137/0916006),
    1995.


```python
import numpy as np
import pandas as pd
import linearsolve as ls
import matplotlib.pyplot as plt
plt.style.use('classic')
%matplotlib inline
```

# Homework 10

**Instructions:** Complete the notebook below. Download the completed notebook in HTML format. Upload assignment using Canvas.

**Due:** Mar. 16 at **2pm.**

## Exercise: The New-Keynesian Model with Optimal Monetary Policy

Consider the New-Keynesian model opptimal monetary policy:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\pi_t & = \beta E_t\pi_{t+1}  + \kappa y_t+ u_t\\
\kappa\pi & = -\lambda y
\end{align}

where $\bar{y}$ and $\pi^T$ have been set to zero. We can interpret $y$ as the output gap (log difference of output from steady state). The system above has four equations and four endogenous variables: $y_t$, $\pi_t$, $i_t$, and $r_t$ ($u_t$ and $g_t$ are exogenous to the system).

### Exogenous Shocks and Parameters

As usual, the variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}.
\end{align}

Use the following parameterization in the simulations below:

| $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\rho_g$$ | $$\rho_u$$ | $$\lambda$$ |
|-----------|----------------|------------|------------|------------|-------------|
| 0.995     | $$-\log\beta$$ | 0.25       | 0.5        | 0.5        | 0.1         |

The shock covariance matrix for the model is:

\begin{align}
\text{Covariance matrix} & = \left[\begin{array}{cc}\sigma_g^2 & 0\\ 0 & \sigma_u^2\end{array}\right]
\end{align}

Use the following values for $\sigma_g$ and $\sigma_u$ in the simulations:

| $$\sigma_g$$ | $$\sigma_u$$ |
|--------------|--------------|
| 0.015        | 0.003        |

```python
# Create a variable called 'parameters' that stores the model parameter values (except lambda) in a Pandas Series. PROVIDED
parameters = pd.Series({'beta':0.995,'r_bar':-np.log(0.995),'kappa':0.25,'rho_g':0.5,'rho_u':0.5})

# Add the value for lambda as 'lam' to 'parameters'

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first. PROVIDED
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable. PROVIDED
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # IS equation.

    # Fisher_equation.

    # Monetary policy. Variable name: 'monetary_policy'

    # Phillips curve.

    # Demand process.

    # Inflation process.

    # Stack equilibrium conditions into a numpy array. PROVIDED
    return np.array([is_equation,fisher_equation,monetary_policy,phillips_curve,demand_process,inflation_process])

# Initialize the model into a variable named 'model'. PROVIDED
nk_model = ls.model(equations = equilibrium_equations,n_states=2,var_names=var_names,shock_names=shock_names,parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model. PROVIDED
nk_model.compute_ss([0,0,0,0.01,0.01,0.01])
```

### Simulation with $\lambda = 0$

You will:

1. Compute a 401 period simulation of the model.
2. Construct a plot of simulated output and inflation.
3. Compute standard deviations of simulated output and inflation.

```python
# Set the value for 'lam' in nk_model.parameters to 0

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False because the model is already log-linear.

# Compute simulation with seed equal to 126 and T equal to 401

```

```python
# Create a figure with dimensions 12x8. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output (times 100)

# Plot inflation (times 400)

# Set axis title

# Set y-axis label to "Percent"

# Construct legends PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python
# Print standard deviation of output (times 100)

```

```python
# print standard deviation of inflation (times 400)

```

### Simulation with $\lambda = 1$

You will:

1. Compute a 401 period simulation of the model.
2. Construct a plot of simulated output and inflation.
3. Compute standard deviations of simulated output and inflation.

```python
# Set the value for 'lam' in nk_model.parameters to 1

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False because the model is already log-linear.

# Compute simulation with seed equal to 126 and T equal to 401

```

```python
# Create a figure with dimensions 12x8. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output (times 100)

# Plot inflation (times 400)

# Set axis title

# Set y-axis label to "Percent"

# Construct legends PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python
# Print standard deviation of output (times 100)

```

```python
# print standard deviation of inflation (times 400)

```

### Simulation with $\lambda = 10$

You will:

1. Compute a 401 period simulation of the model.
2. Construct a plot of simulated output and inflation.
3. Compute standard deviations of simulated output and inflation.

```python
# Set the value for 'lam' in nk_model.parameters to 10

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model
# set argumement 'log_linear' to False because the model is already log-linear.

# Compute simulation with seed equal to 126 and T equal to 401

```

```python
# Create a figure with dimensions 12x8. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create axis. PROVIDED
ax = fig.add_subplot(1,1,1)

# Plot output (times 100)

# Plot inflation (times 400)

# Set axis title

# Set y-axis label to "Percent"

# Construct legends PROVIDED
ax.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

```python
# Print standard deviation of output (times 100)

```

```python
# print standard deviation of inflation (times 400)

```

**Questions**

1. In terms of a central bank's preference for stabilizing output and inflation, explain in words what it means for $\lambda$ to increase.
2. Which value for $\lambda$ produced the smallest fluctuations in output?
3. Which value for $\lambda$ produced the smallest fluctuations in inflation?

**Answers**

1.

2.

3.

## Exercise: Optimal Monetary Policy v. Taylor's Rule

In this exercise, you will compute and compare impulse responses to supply and demand shocks for the new Keynesian model under two assumptions about monetary policy:

1. The central bank follows optimal monetary policy.
2. The central bank follows Taylor's simple rule.

### NK Model with Optimal Policy

The New-Keynesian model opptimal monetary policy:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\pi_t & = \beta E_t\pi_{t+1}  + \kappa y_t+ u_t\\
\kappa\pi & = -\lambda y
\end{align}

The system above has four equations and four endogenous variables: $y_t$, $\pi_t$, $i_t$, and $r_t$ ($u_t$ and $g_t$ are exogenous to the system).

### Exogenous Shocks and Parameters

As usual, the variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}.
\end{align}

Use the following parameterization in the simulations:

| $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\rho_g$$ | $$\rho_u$$ | $$\lambda$$ |
|-----------|----------------|------------|------------|------------|-------------|
| 0.995     | $$-\log\beta$$ | 0.25       | 0.5        | 0.5        | 0.1         |

```python
# Create a variable called 'parameters' that stores the model parameter values (except lambda) in a Pandas Series. PROVIDED
parameters = pd.Series({'beta':0.995,'r_bar':-np.log(0.995),'kappa':0.25,'rho_g':0.5,'rho_u':0.5})

# Add the value for lambda as 'lam' to 'parameters'

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first. PROVIDED
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable. PROVIDED
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # IS equation. PROVIDED
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation. PROVIDED
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy. Variable name: 'monetary_policy'

    # Phillips curve. PROVIDED
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + cur.u - cur.pi

    # Demand process. PROVIDED
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process. PROVIDED
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array. PROVIDED
    return np.array([is_equation,fisher_equation,monetary_policy,phillips_curve,demand_process,inflation_process])

# Initialize the model into a variable named 'nk_model_optimal'. PROVIDED
nk_model_optimal = ls.model(equations = equilibrium_equations,n_states=2,var_names=var_names,shock_names=shock_names,parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model_optimal. PROVIDED
nk_model_optimal.compute_ss([0,0,0,0.01,0.01,0.01])

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model_optimal
# set argumement 'log_linear' to False because the model is already log-linear. PROVIDED
nk_model_optimal.approximate_and_solve(log_linear = False)
```

### NK Model with Taylor's Rule

The New-Keynesian model opptimal monetary policy:

\begin{align}
y_t & = E_t y_{t+1}  - \left( r_{t} - \bar{r}\right) + g_t\\
i_{t} & = r_{t} + E_t \pi_{t+1}\\
\pi_t & = \beta E_t\pi_{t+1}  + \kappa y_t+ u_t\\
i_{t} & = \bar{r} + \pi^T + \phi_{\pi}\big(\pi_t - \pi^T\big) + \phi_{y}\big(y_t - \bar{y}\big)
\end{align}

The system above has four equations and four endogenous variables: $y_t$, $\pi_t$, $i_t$, and $r_t$ ($u_t$ and $g_t$ are exogenous to the system).

### Exogenous Shocks and Parameters

As usual, the variables $g_t$ and $u_t$ represent exogenous shocks to aggregate demand and inflation. They follow AR(1) processes:

\begin{align}
g_{t+1} & = \rho_g g_{t} + \epsilon^g_{t+1}\\
u_{t+1} & = \rho_u u_{t} + \epsilon^u_{t+1}.
\end{align}

Use the following parameterization in the simulations:

| $$\beta$$ | $$\bar{r}$$    | $$\kappa$$ | $$\rho_g$$ | $$\rho_u$$ | $$\pi^T$$ | $$\phi_{\pi}$$ | $\phi_{y}$ | $$\bar{y}$$ |
|-----------|----------------|------------|------------|------------|-----------|----------------|------------|-------------------|
| 0.995     | $$-\log\beta$$ | 0.25       | 0.5        | 0.5        | 0.02/4    | 1.5            | 0.5/4      | 0                 |

```python
# Create a variable called 'parameters' that stores the model parameter values (except lambda) in a Pandas Series. PROVIDED
parameters = pd.Series({'beta':0.995,'r_bar':-np.log(0.995),'kappa':0.25,'rho_g':0.5,'rho_u':0.5})

# Add the value for lambda as 'pi_T' to 'parameters'

# Add the value for lambda as 'phi_pi' to 'parameters'

# Add the value for lambda as 'phi_y' to 'parameters'

# Create variable called 'var_names' that stores the variable names in a list with state variables ordered first. PROVIDED
var_names = ['g','u','y','pi','i','r']

# Create variable called 'shock_names' that stores an exogenous shock name for each state variable. PROVIDED
shock_names = ['e_g','e_u']

# Define a function that evaluates the equilibrium conditions of the model solved for zero. PROVIDED
def equilibrium_equations(variables_forward,variables_current,parameters):

    # Parameters. PROVIDED
    p = parameters

    # Current variables. PROVIDED
    cur = variables_current

    # Forward variables. PROVIDED
    fwd = variables_forward

    # IS equation. . PROVIDED
    is_equation = fwd.y - (cur.r -p.r_bar) + cur.g - cur.y

    # Fisher_equation. . PROVIDED
    fisher_equation = cur.r + fwd.pi - cur.i

    # Monetary policy. Variable name: 'monetary_policy'

    # Phillips curve. . PROVIDED
    phillips_curve = p.beta*fwd.pi + p.kappa*cur.y + cur.u - cur.pi

    # Demand process. . PROVIDED
    demand_process = p.rho_g*cur.g - fwd.g

    # Inflation process. . PROVIDED
    inflation_process = p.rho_u*cur.u - fwd.u

    # Stack equilibrium conditions into a numpy array. PROVIDED
    return np.array([is_equation,fisher_equation,monetary_policy,phillips_curve,demand_process,inflation_process])

# Initialize the model into a variable named 'nk_model_taylor'. PROVIDED
nk_model_taylor = ls.model(equations = equilibrium_equations,n_states=2,var_names=var_names,shock_names=shock_names,parameters = parameters)

# Compute the steady state numerically using .compute_ss() method of nk_model_taylor. PROVIDED
nk_model_taylor.compute_ss([0,0,0,0.01,0.01,0.01])

# Find the log-linear approximation around the non-stochastic steady state and solve using .approximate_and_solve() method of nk_model_taylor
# set argumement 'log_linear' to False because the model is already log-linear. PROVIDED
nk_model_taylor.approximate_and_solve(log_linear = False)
```

### Impulse Responses

Compute a 21 period impulse responses of the each model's variables to a 0.01 unit shock to the exogenous component of  demand ($g_t$) and a 0.01/4 unit shock to the exogenous component of inflation ($u_t$) in period 5.

```python
# Compute the desired impulse responses for the model with optimal monetary policy

# Compute the desired impulse responses for the model with Taylor's rule

```

#### Demand Shocks

Plot the impulse responses of output and inflation to the demand shock under each monetary policy specification. Multiply output by 100 and multiply inflation by 400 (to annualize the values).

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create left subplot. PROVIDED
ax1 = fig.add_subplot(1,2,1)
# Create right subplot. PROVIDED
ax2 = fig.add_subplot(1,2,2)

# Left subplot title. PROVIDED
ax1.set_title('Output $y_t$')
# Right subplot title. PROVIDED
ax2.set_title('Inflation $\pi_t$')

# Left subplot y-axis label. PROVIDED
ax1.set_ylabel('Percent')
# Right subplot y-axis label. PROVIDED
ax2.set_ylabel('Percent')

# Plot inflation (multiplied by 100) under both monetary policies on right axis.

# Plot inflation (multiplied by 400) under both monetary policies on right axis. Add labels for the lines.

# Construct legends PROVIDED
ax2.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

#### Demand Shocks

Plot the impulse responses of output and inflation to the inflation shock under each monetary policy specification. Multiply output by 100 and multiply inflation by 400 (to annualize the values).

```python
# Create figure. PROVIDED
fig = plt.figure(figsize=(12,4))

# Create left subplot. PROVIDED
ax1 = fig.add_subplot(1,2,1)
# Create right subplot. PROVIDED
ax2 = fig.add_subplot(1,2,2)

# Left subplot title. PROVIDED
ax1.set_title('Output $y_t$')
# Right subplot title. PROVIDED
ax2.set_title('Inflation $\pi_t$')

# Left subplot y-axis label. PROVIDED
ax1.set_ylabel('Percent')
# Right subplot y-axis label. PROVIDED
ax2.set_ylabel('Percent')

# Plot output (multiplied by 100) under both monetary policies on left axis.

# Plot inflation (multiplied by 400) under both monetary policies on right axis. Add labels for the lines.

# Construct legends PROVIDED
ax2.legend(loc='center left', bbox_to_anchor=(1, 0.5))
```

**Questions**

1. Does Taylor's rule appear to do a better job matching the optimal response to demand or inflation shocks? Explain

**Answers**

1.


O. Pannekoucke$^{1,2,3}$

$^1$ INPT-ENM, Toulouse, France

$^2$ CNRM, Université de Toulouse, Météo-France, CNRS, Toulouse, France

$^3$ CERFACS, Toulouse, France

(olivier.pannekoucke@meteo.fr)

**Abstract**

This notebook illustrates the use of sympkf to build and handle the PKF dynamics associated with the Burgers' equation given by the non-linear advection plus a diffusion
$$\partial_t u + u\partial_x u = \kappa \partial_x^2 u.$$
For this dynamics, the resulting PKF system is not closed, and we show how to introduce a closure at a symbolic level. The notebook ends with a numerical simulation, which illustrates how a numerical code can be automaticaly generated by using SympPKF, then used to perform a numerical simulation.

# Definition of the Burgers' dynamics

```python
# Import of libraries
import sympy
from sympy import init_printing, symbols, Function, Derivative, Eq, Integer, Rational
init_printing()

from sympkf import Expectation, PDESystem, SymbolicPKF, FDModelBuilder, t
```

**Definition of the dynamics from sympy tools**

```python
# Set the spatial coordinate system
x = symbols('x')
# Set the constants
kappa = symbols('kappa')
# Define the spatio-temporal scalar field
u = Function('u')(t,x)
```

```python
# Definition of the Burgers dynamics
burgers_equation = Eq(Derivative(u,t),
    -u*Derivative(u,x)+kappa*Derivative(u,x,2))
burgers_equation
```

```python
# Processing of the PDE system
burgers = PDESystem( burgers_equation )
burgers
```

    PDE System :
            prognostic functions : u(t, x)
            constant functions   :
            exogeneous functions :
            constants            : kappa

# Computation of the PKF dynamics by using SymPKF

## Direct computation of the PKF dynamics

```python
pkf_burgers = SymbolicPKF(burgers)
```

Computation of the PKF dynamics as given in metric and in aspect tensor form.

```python
import time
start = time.time()
pkf_burgers.in_metric
pkf_burgers.in_aspect
end = time.time()
print(end - start)
```

    1.5819108486175537

```python
pkf_burgers = SymbolicPKF(burgers)
```

```python
for equation in pkf_burgers.in_metric: display(equation)
```

```python
for equation in pkf_burgers.in_aspect: display(equation)
```

It follows that the PKF dynamics is unclosed since the term $\mathbb{E}[\varepsilon\partial_x^4\varepsilon]$ is not given as a function of the prognostic variables $u$,$V_u$ and $g_{u,xx}$ (or $s_{u,xx}$ when the dynamics is given in aspect tensor form).

## Sequential computation of the PKF dynamics using the splitting strategy

The computation of the dynamics can be considered in two steps thanks to the splitting dynamics described in Pannekoucke and Arbogast (2021).

```python
# PKF for the non-linear advection
nladvection_pkf = SymbolicPKF(
    Eq(Derivative(u,t), -u*Derivative(u,x))
)

for equation in nladvection_pkf.in_aspect: display(equation)
```

```python
# PKF for the diffusion
diffusion_pkf = SymbolicPKF(
    Eq(Derivative(u,t), kappa*Derivative(u,x,2))
)

for equation in diffusion_pkf.in_aspect: display(equation)
```

Merging the resulting above dynamics provide the PKF dynamics of the full Burgers' equation.

## Exploration of the substitution tree

Note that the computation of the PKF dynamics relies on the transformation of terms in $\mathbb{E}[\partial^\alpha\varepsilon\partial^\beta\varepsilon]$ into terms in $\mathbb{E}[\varepsilon\partial^\gamma\varepsilon]$ where $\gamma\leq\alpha+\beta$.

This transformation is handled in SymPKF thanks to a substitution dictionnary, which can be displayed as follows

```python
for key, value in pkf_burgers.subs_tree.items():
    display({key:value})
```

## Itermediate conclusion

Since the PKF equation are not closed. We now introduce an illustration of a closure resulting from the work of [Pannekoucke et al. (2018)](https://doi.org/10.5194/npg-2018-10)

# Introduction of a closure for the PKF dynamics applied for the Burgers' equation

First, the unclosed terms can be obtained from the instance of the class `SymbolicPKF`.

```python
for term in pkf_burgers.unclosed_terms:
    display(term)
```

```python
unclosed_term = list(pkf_burgers.unclosed_terms)[0]
```

## Data-driven exploration of closure

[Pannekoucke and Fablet (2020)](https://doi.org/10.5194/gmd-13-3373-2020) has proposed a data-driven approach for closing a PKF dynamics. This relies on the proposal of closure. An approach for such proposals can rely of approximation of correlations.

For instance, if a correlation function is approximated by a "quasi"-gaussian function
$$\rho(x,x+\delta x) \approx \exp\left({-\frac{\delta x^2}{s(x)+s(x+\delta x)}}\right),$$
then the identification in the Taylor expansion of the correlation
\begin{equation}
\rho(x,x+\delta x) = \sum_k \frac{1}{k!}\mathbb{E}\left(\varepsilon(x)\partial_x^k \varepsilon( x)\right)\delta x^k,
\end{equation}
can leads to proposal closure for terms $\mathbb{E}\left(\varepsilon \partial_x^k \varepsilon\right)$.

```python
from sympkf import remove_eval_derivative

s = pkf_burgers.fields[u].aspect[0]
eps = pkf_burgers.fields[u].epsilon
dx = sympy.Symbol('\delta x')

# definition of the correlation
rho = sympy.exp(- dx*dx / ( s + s.subs(x,x+dx) ) )

# Taylor expansion of the correlation with respect to dx, at order 6
taylor_order = 6
taylor = rho.series(dx,0,taylor_order)
taylor = remove_eval_derivative(taylor.removeO())

# Definition of pattern for the design of closure proposals
a = sympy.Wild("a",properties=[lambda k: k.is_Rational])
b = sympy.Wild("b")

for order in [4,5]:
    # Extract the term of a given order
    expr = taylor.coeff(dx,order)*sympy.factorial(order)
    if order==4: display(expr)
    # Create a proposal for the term at the order from pattern matching
    expr = sum([sympy.symbols(f'a_{k}^{order}')*term.match(a*b)[b]
                for k,term in enumerate(expr.args)] )
    display({Expectation(eps*Derivative(eps,x,order)):expr})
```

In the above results, the terms $\mathbf{a}=(a_0^4,a_1^4,a_2^4)$ denotes unknown scalar
that remains to estimate from a machine learning approach, e.g. the method proposed by [Pannekoucke and Fablet (2020)](https://doi.org/10.5194/gmd-13-3373-2020) which relies on an automatic generation of neural-network architecture.

While interesting, this kind of approach is not detailed here, and we now introduce a known closure for the PKF applied to the Burgers' equation, namely the closure proposed by [Pannekoucke et al. (2018)](https://doi.org/10.5194/npg-2018-10).

## Introduction of a the closure proposed by Pannekoucket et al. (2018)

The closure proposed in [Pannekoucke et al. (2018)](https://doi.org/10.5194/npg-2018-10) reads as
$$\mathbb{E}[\varepsilon\partial_x^4\varepsilon]\sim 3g_{u,xx}^2 -2\partial_x^2 g_{u,xx},$$
where $\sim$ means that this is not an equality (since we do not know the term), but a proposal of closure.
This closure can be introduced as follows.

### Definition of the closure

```python
from sympy import Integer
g = pkf_burgers.fields[u].metric[0] # metric tensor
s = pkf_burgers.fields[u].aspect[0] # aspect tensorc
```

```python
P18_closure = Integer(3)*g**Integer(2)-Integer(2)*Derivative(g,x,2)
P18_closure = P18_closure.subs(g,1/s).doit().expand()
display(P18_closure)
```

### Introduction of the closure in the PKF dynamics

```python
for equation in pkf_burgers.in_aspect:
    display(equation.subs(unclosed_term,P18_closure).expand())
```

## Itermediate conclusion

Now, the PKF dynamics being closed, it is possible to perform numerical simulation so to assess the ability of the PKF dynamics to reproduce the covariance dynamics in the Burgers' equation

# Numerical experiment to assess the skill of the closed PKF dynamics

In the numerical example, we consider the dynamics written in aspect tensor form.

## Automatic code generation from the closed PKF system

SymPKF comes with a python numerical code generator which translate a system of partial differential equation into a python's code using `numpy` and where the partial derivative with respect to spatial coordinates are approximated thanks to a finite difference approach, consistent at the second order.

In what follows, we illustrate another way to introduce a closure.

```python
from sympy import Integer
from sympkf import FDModelBuilder

g = pkf_burgers.fields[u].metric[0] # metric tensor
s = pkf_burgers.fields[u].aspect[0] # aspect tensor

# loc. Gaussian closure of P18 in metric form, then in aspect form
P18_closure = Integer(3)*g**Integer(2)-Integer(2)*Derivative(g,x,2)
P18_closure = P18_closure.subs(g,1/s).doit().expand()

# Introduction of the closure as a dictionnary
unclosed_term = list(pkf_burgers.unclosed_terms)[0]
pkf_burgers.set_closure({unclosed_term:P18_closure})

# Build a numerical code at runtime
exec(FDModelBuilder(pkf_burgers.in_aspect, class_name='ClosedPKFBurgers').code)
closed_pkf_burgers = ClosedPKFBurgers(shape=(241,))
```

    Warning: constant `kappa` has to be set

**Sample of the numerical code generated in the class `ClosedPKFBurgers`**

```python
[..]
# Compute derivatives
#-----------------------
DV_u_x_o1 = (-V_u[np.ix_(self.index('x',-1))] + V_u[np.ix_(self.index('x',1))])/(2*self.dx[self.coordinates.index('x')])

[..]
# Implementation of the trend
#--------------------------------
du[:] = -DV_u_x_o1/2 - Du_x_o1*u + Du_x_o2*kappa

dV_u[:] = -DV_u_x_o1**2*kappa/(2*V_u) - DV_u_x_o1*u + DV_u_x_o2*kappa - 2*Du_x_o1*V_u - 2*V_u*kappa/s_u_xx

ds_u_xx[:] = 2*DV_u_x_o1**2*s_u_xx*kappa/V_u**2 + DV_u_x_o1*Ds_u_xx_x_o1*kappa/V_u - 2*DV_u_x_o2*s_u_xx*kappa/V_u - 2*Ds_u_xx_x_o1**2*kappa/s_u_xx - Ds_u_xx_x_o1*u + Ds_u_xx_x_o2*kappa + 2*Du_x_o1*s_u_xx + 4*kappa
[..]
```

In the last lines,
```python
exec(FDModelBuilder(pkf_burgers.in_aspect, class_name='ClosedPKFBurgers').code)
closed_pkf_burgers = ClosedPKFBurgers(shape=(241,))
```
first a class `ClosedPKFBurgers` is created at runtime, then an instance of this class, that is `closed_pkf_burgers`.
The discretization is made with $241$ grid points, while the length of the domain is $1$.

```python
# Uncomment the following line if you want to see the code generated by sympkf
#print(FDModelBuilder(closed_pkf, class_name='ClosedPKFBurgers').code)
```

## Numerical experiment

```python
import matplotlib.pyplot as plt
```

```python
def plot_results(data, label=None, labelx=True, title=None, save_file=None, normalisation=None,
                 selected_times=None,style=None, name=None, alpha=1., bolds=[0., 1.]):

    normalisation = 1. if normalisation is None else normalisation

    selected_times = [time for time in data] if selected_times is None else selected_times

    style = 'k' if style is None else style

    for time in selected_times:
        lalpha = alpha if time in bolds else 0.2
        lname = name if time==selected_times[-1] else None
        plt.plot(domain.x[0],data[time]/normalisation, style, alpha = lalpha, label=lname)

    if labelx:
        plt.xlabel('$x/D$', fontsize=15)
    if label:
        plt.ylabel(label, fontsize=15)
    if title:
        plt.title(title)
    if save_file:
        plt.savefig(save_file)
```

**Definition of the physical domain of computation**

```python
domain = closed_pkf_burgers
```

```python
domain.shape, domain.lengths
```

**Set initial fields**

```python
import numpy as np
```

```python
# Set initial condition for U (ensemble averaged velocity field)
Umax = 0.5
U0=Umax*( 1+np.cos(2*np.pi/ domain.lengths[0]  *(domain.x[0]-0.25)) ) *0.5

# Set initial condition for V (variance)
V0 = (0.01*Umax)**2 + np.zeros(domain.shape[0])

# Set initial condition for s (aspect tensor)
lh = 0.02*domain.lengths[0]
s0 = lh**2 + + np.zeros(domain.shape[0])

# build the initial state vector as used for the prediction in the numerical code
state0 = np.asarray([U0, V0, s0])
normalization = {
                '(a) Velocity':U0.max(),
                '(b) Variance':V0.max(),
                '(c) Length-scale':lh
                }
```

**Set the parameters of simulation**

```python
# Set the diffusion coefficient $\kappa$.

dt_front = 0.005
kappa = Umax*dt_front

closed_pkf_burgers.kappa = kappa
```

```python
kappa
```

```python
dt = 0.002               # default time step
refine_time = int(1)    # used to increase the time resolution (reduce the time step)
time_simulation = 1.
closed_pkf_burgers.set_dt(dt/refine_time)
times = closed_pkf_burgers.window(time_simulation)
saved_times = times[::50*refine_time]
```

```python
closed_pkf_burgers._dt
```

**Computation of a short forecast**

```python
pkf_trajectory = closed_pkf_burgers.forecast(times, state0, saved_times)
```

**Plot results**

```python
length_scale = lambda s: np.sqrt(s)

def plot_vlatcov_stat(trajectory, name, style='k', subplot=False):
    if not subplot: plt.figure(figsize=(14,4))
    for k,field,label in zip(range(3),normalization,"abc"):
        if field=='Length-scale':
            data = {time:length_scale(trajectory[time][k]) for time in trajectory}
        else:
                data = {time:trajectory[time][k] for time in trajectory}
        plt.subplot(131+k)
        plot_results(data,style=style,label=field, name=name,normalisation=normalization[field])
        #plt.title(f'({label})')
        if k==0:
            plt.legend()
```

```python
plot_vlatcov_stat(pkf_trajectory, name='P18')
```

## Ensemble validation of the PKF dynamics

### Generation and diagnosis of an initial ensemble of forecast error

**Generation of an ensemble of initial error**

We consider that the initial covariance matrix $\mathbf{P}^f_0$ is so that the correlation functions are the Gaussian
functions
$$\rho_h(x,y) = e^{-\frac{(x-y)^2}{2l_h^2}},$$
where $l_h$ denotes the correlation length-scale.

Hence, any Gaussian sample $\varepsilon^f\sim\mathcal{N}(0,\mathbf{P}^f_0)$ from this covariance matrix is defined from the transformation
$$\varepsilon^f = {\mathbf{P}^f_0}^{1/2}\zeta,$$
where $\zeta\sim\mathcal{N}(0,\mathbf{I})$ is a sample of a nomalized Gaussian of zero mean and covariance matrix the identity matrix $\mathbf{I}$.

Since the correlation function is the same at each point of the domain (the correlation is homogeneous), the covariance matrix $\mathbf{P}^f_0$ is diagonal in Fourier's space (spectral space). In this illustration we build forecast-error sample from the Fourier's space.

```python
# construction of the initial covariance matrix $P_0$
# 1) Definition of the correlation function on the circle
gauss = lambda dx : np.exp(-0.5*dx**2/lh**2) # lh has been previously specified
correlation = gauss(domain.x[0]-domain.x[0][domain.shape[0]//2])

spectrum = np.abs(np.fft.fft(correlation))

# 2) Computation of the square root matrix
std_spectrum = np.sqrt(spectrum)

# 3) Routine to peform a Gaussian sample from the homogeneous correlation of the given sqrt-spectrum
def make_sample():
    zeta = np.random.normal(size=domain.shape)
    zeta = np.fft.fft(zeta)
    ef = np.fft.ifft(std_spectrum * zeta)
    ef = np.real(ef)
    return ef

# 4) Routine to perfrom an ensemble of error of given size
def make_init_ensemble(mean, variance, Ne):
    """ Generate an ensemble of forecast-error """
    return np.array([mean+np.sqrt(variance)*make_sample() for k in range(Ne)])
```

```python
plt.figure(figsize=(12,5))
plt.subplot(121)
plt.plot(domain.x[0], correlation)
plt.title('Homogenous correlation function');
plt.subplot(122)
for k in range(10):
    plt.plot(domain.x[0], make_sample())
plt.title("Example of sample errors");
```

**Diagnosis of the covariance, variance, anisotropy from an ensemble**

```python
def estimate_covariance(ensemble):
    """ Estimate the covariance function from an ensemble
    ensemble : array
    """
    mean = ensemble.mean(0)
    error = (ensemble - mean)/np.sqrt(len(ensemble))
    return error.T @ error

class EnsembleDiagnosis(object):
    """ Diagnosis tool of the variance and of the anisotropy for a covariance
    model parameterized from this two features (VLATcov model)
    """
    def __init__(self, ensemble, base_space):
        self.base_space = base_space

        if isinstance(ensemble, list):
            ensemble = np.array(ensemble)

        if len(ensemble.shape)==3:
            ensemble = np.array([elm[0] for elm in ensemble])

        # 1) Computation of the mean
        self.mean = ensemble.mean(axis=0)

        # 2) Computation of the variance
        self.std = ensemble.std(axis=0)
        self.variance = self.std*self.std

        # 3) Computation of the metric terms
        #  we use the formula g_ij = E[(D_i eps)(D_j eps)]

        #  a) Computation of the normalized error
        epsilon = (ensemble-self.mean)/self.std

        #  b) Computation of derivatives (uses centered second order finite difference)
        n = self.base_space.shape[0]
        K = np.arange(n)
        kp = (K+1)%n
        km = (K-1)%n
        dx = self.base_space.dx[0]
        Depsilon = np.array([(eps[kp]-eps[km])/(2*dx) for eps in epsilon])
        self.metric = (Depsilon*Depsilon).mean(axis=0)     # see Pannekoucke et al. (2018) for details

        # Computation of the diffusion tensor
        self.aspect = 1/self.metric
        self.length_scale = np.sqrt(self.aspect)
```

**Validation of the diagnosis tool on an initial ensemble**

```python
Ne = 1600

ensemble = make_init_ensemble(state0, V0.max(), Ne)

mean = ensemble.mean(axis=0)
std = ensemble.std(axis=0)

print(f"Validation of the mean (=0): {mean.mean()} +/- {mean.std()}" )
print(f"Validation of the standard-deviation (=1): {std.mean()} +/- {std.std()}" )

ens_diagnosis = EnsembleDiagnosis(ensemble, domain)

plt.figure(figsize=(15,5))

plt.subplot(131)
plt.plot(ens_diagnosis.mean)
plt.title('Ensemble mean')

plt.subplot(132)
plt.plot(ens_diagnosis.variance/V0.max())
plt.title('Variance (normalized by the specified variance)')

plt.subplot(133)
plt.plot(ens_diagnosis.length_scale/lh)
plt.title('Length-scale (normalized by $l_h$)')

alert_msg = "Departure between the ensemble estimation of the length-scale and its theoretical value"
assert np.abs(ens_diagnosis.length_scale.mean()/lh-1)<0.1, alert_msg
```

### Computation of an ensemble of forecast

**Automatic generation of a numerical code solving the Burgers equation**

```python
# Build a numerical code at runtime
exec(FDModelBuilder(burgers_equation, class_name='Burgers').code)
num_burgers = Burgers(shape=(241,), kappa=kappa)
```

**Generation fo the ensemble of forecast**

The computation of the ensemble of forecast relies on the parallel computation of the time integration of each state of the ensemble

```python
# Build a numerical code at runtime
ensemble_forecast = num_burgers.ensemble_forecast(times, ensemble, saved_times)
```

```python
# Compute diagnosis from ensemble
ensemble_traj = {}
for time in saved_times:
    diagnosis = EnsembleDiagnosis(ensemble_forecast[time], domain)
    ensemble_traj[time] = [diagnosis.mean, diagnosis.variance, diagnosis.aspect]
```

```python
plot_vlatcov_stat(ensemble_traj, name='Ens. validation', style='k--')
plot_vlatcov_stat(pkf_trajectory, name='PKF P18-closure', subplot=True)
#plt.savefig("./figures/sympkf_burgers-simulation.pdf")
```

We observe that the ensemble estimations of the variance and of the anisotropy (here featured by the length-scale) are in accordance with the dynamics predicted by the PKF closed following P18.

# Conclusion

In this notebook we showed that SymPKF is able to compute the PKF dynamics associated to a given dynamics, by considering the Burgers' equation.

The PKF system for the Burgers' equation being unclosed, we showed how to introduce as closure.

So to perform a numerical simulation, the automatic code generator provided with SymPKF has been used to render a numerical code for the PKF system.


```python
%load_ext autoreload
%autoreload 2
```

```python
# Importing packages and own module
import numpy as np
from scipy import optimize
import matplotlib.pyplot as plt
from types import SimpleNamespace
import inauguralproject as ip
```

# Question 1

This question investigates what level of insurance an agent would choose, if the insurance company are not trying to create any profit, but simply make sure that the premiums covers the cost. In that case, the premium, $\pi$, is a function of the size of the coverage, $q$, and the probality of the loss, $p$,:

\begin{equation}
\pi(p,q)= pq.
\end{equation}

The expected utility of the agent is

\begin{equation}
V(q;\pi) = pu\left(y-x+q-\pi(p,q)\right) + (1-p) u\left(y-\pi(p,q)\right).
\end{equation}

where $y$ is the initial assets hold by the agent, $x$ is the amount of loss, and $u(\cdot)$ is a CRRA utility function.
As the coverage can not exceed the loss, the solution to the question is given by

\begin{equation}
q^\ast = argmax_{q\in \left[0,x\right]}V(q;\pi)
\end{equation}

```python
# a. initial parameters
par = SimpleNamespace()
par.theta = -2
par.p = 0.2
par.y = 1

# b. creating vector of x's and for the optimal q's
N = 1000
x_vec = np.linspace(0.01,0.9,N)
optimal_q_vec = np.empty(N)

# c. solving the problem
for i, x in enumerate(x_vec):
    optimal_q_vec[i] = ip.optimal_q_function(x,par)

# d. plotting the solution
plt.style.use('seaborn-whitegrid')

fig = plt.figure(dpi=100)
ax = fig.add_subplot(1,1,1)

ax.plot(x_vec, optimal_q_vec, label = '$q^*$')

ax.set_xlabel('Loss, $x$')
ax.set_ylabel('Coverage, $q$')
ax.set_title('Optimal Coverage for the Agent')
ax.set_xlim(0,0.9)
ax.set_ylim(0,0.9)
legend = ax.legend(loc=2, frameon=True, framealpha=1)

plt.show()
```

As shown in the figure, agent would choose full coverage.

# Question 2

The agent is willing to pay the premium of an insurance contract, as long as the expected utility of taking the contract is lower or equal to the expected utility of not taking the contract. Here, I will cover which contracts that will be acceptable for both the agent and the ensurer, in the case where the potential loss is $0.6$. The results is shown in the figure.

```python
# a. Creating a vector of q's and a vector of minimum premiums
x = 0.6
q_vec = np.linspace(0.01,x,10)
pi = q_vec*par.p

# b. Solving for the maximum premium
premium_vec = np.empty(len(q_vec))
for i, q in enumerate(q_vec):
    premium_vec[i] = ip.optimal_premium(x,q,par)

# c. Plotting graph
fig = plt.figure(dpi=111)
ax = fig.add_subplot(1,1,1)

ax.plot(q_vec,premium_vec, label = 'Maximimum premium for agent', color='red')
ax.plot(q_vec, pi, label = 'Minimum premium for insurance company', color='blue')
ax.fill_between(q_vec, pi, premium_vec, color= 'grey', alpha=0.3, label= 'Acceptable for both parties')
ax.set_title('Area of Feasible Contracts')

ax.set_xlim(0,x)
ax.set_ylim(0,0.3)
ax.set_xlabel('Insurance Coverage')
ax.set_ylabel('Premium')
legend = ax.legend(loc=2, frameon=True, framealpha=1)

plt.show()
```

In the case of a market with a lot of insurance companies, the premium would be near the blue line, but not at it, as the companies have to make some amount of contribution margin, to run the companies. In a case with monopoly the premium would at the red line line.

# Question 3

Instead of the loss being fixed and either occur or not, I now consider the case where the loss is drawn from a random beta distribution and where the coverage is a fraction, $\gamma$, of the loss. Thus, the coverage and the loss can be expessed by

\begin{equation}
q = \gamma x \:\:\: \gamma\in[0,1]
\end{equation}

and

\begin{equation}
x\sim Beta(\alpha, \beta), \:\:where\:\: \alpha = 2, \beta = 7
\end{equation}

The agents utility, if they take on a insurance contract, is

\begin{equation}
V(\gamma, \pi) = \int_0^1 u(y-(1-\gamma)x-\pi)f(x)dx
\end{equation}

and if they do not take on a contract, it is

\begin{equation}
V_{NI} = \int_0^1 u(y-x)f(x)dx.
\end{equation}

By performing a Monte Carlo integration, I solve for the expected value of the agents utility, given different values of $\gamma$ and $\pi$

```python
# a. Drawing random numbers from beta-distribuition
np.random.seed(118)
N = 10000
par.X = np.random.beta(a=2,b=7,size = N)

# b. Solving for different values of gamma and pi
gamma1 = par.gamma = 0.9
pi1 = par.pi = 0.2
expected_value1 = ip.expected_value(par)

gamma2 =par.gamma = 0.45
pi2 = par.pi = 0.1
expected_value2 = ip.expected_value(par)

# c. Printing results
print(f'When coverage is {gamma1} and the premium is {pi1} the expected utility is {expected_value1:.3f}')
print(f'When coverage is {gamma2} and the premium is {pi2} the expected utility is {expected_value2:.3f}')

if expected_value1 > expected_value2:
    print(f'Therefore, the agent prefers a coverage of {gamma1} with a premium of {pi1}')
else:
    print(f'Therefore, the agent prefers a coverage of {gamma2} and a premium of {pi2}')
```

    When coverage is 0.9 and the premium is 0.2 the expected utility is -1.286
    When coverage is 0.45 and the premium is 0.1 the expected utility is -1.297
    Therefore, the agent prefers a coverage of 0.9 with a premium of 0.2

# Question 4

For the insurance company's viewpoint, they want to maximize their profit. If they are a monopoly, they will take the maximum premium that the agent is willing to accept. Therefore, they have to solve

$$
\begin{aligned}
\pi^{\ast} = argmax_\pi & V(\gamma, \pi)\\
&\text{s.t.}\\
V(\gamma, \pi)& \ge V_{NI}
\end{aligned}
$$

I solve it for a coverage ratio of $\gamma = 0.95$.

```python
# a. initial parameters
gamma = par.gamma = 0.95
initial_guess = 0.2

# b. Solving for optimal premium
sol = optimize.root(
    ip.objective_for_maximizing_profit,
    initial_guess,
    args = (par),
    )

solution = float(sol.x)

# c. Print solution
print(f'If the customer wants a coverage of {gamma} their are willing to accept a premium up to {solution:.4f}.')
```

    If the customer wants a coverage of 0.95 their are willing to accept a premium up to 0.2369.


This notebook is part of the $\omega radlib$ documentation: http://wradlib.org/wradlib-docs.

Copyright (c) 2016, $\omega radlib$ developers.
Distributed under the MIT License. See LICENSE.txt for more info.

# RADOLAN Grid

## Polar Stereographic Projection

The projected composite raster is equidistant with a grid-spacing of 1.0 km in most cases. There are composites which have 2.0 km grid-spacing (e.g. PC).

There are three different grid sizes, the well-known 900 rows by 900 columns (normal), 1500 rows by 1400 columns (extended, european) and 460 rows by 460 columns (small).

Common to all is that the plane of projection intersects the earth sphere at $\phi_0 = 60.0^{\circ}N$. The cartesian co-ordinate system is aligned parallel to the $\lambda_0 = 10.0^{\circ}E$ meridian.

The reference point ($\lambda_m$, $\phi_m$) is $9.0^{\circ}E$ and $51.0^{\circ}N$, which is the center of the two smaller grids. The extended grid has an offset in respect to this reference point of 350km by 150km.

The earth as sphere with an radius of 6370.04 km is used for all calculations.

With formulas (1), (2) and (3) the geographic reference points ($\lambda$, $\phi$) can be converted to projected cartesian coordinates. The calculated (x y) is the distance vector to the origign of the cartesian coordinate system (north pole).

$\begin{equation}
x = R * M(\phi) * cos(\phi) * sin(\lambda - \lambda_0)
\tag{1}
\end{equation}$

$\begin{equation}
y = -R * M(\phi) * cos(\phi) * cos(\lambda - \lambda_0)
\tag{2}
\end{equation}$

$\begin{equation}
M(\phi) =  \frac {1 + sin(\phi_0)} {1 + sin(\phi)}
\tag{3}
\end{equation}$

Assumed the point ($10.0^{\circ}E$, $90.0^{\circ}N$) is defined as coordinate system origin. Then all ccordinates can be calculated with the known grid-spacing d as:

$\begin{equation}
x = x_0 + d * (j - j_0)
\tag{4}
\end{equation}$

$\begin{equation}
y = y_0 + d * (i - i_0)
\tag{5}
\end{equation}$

with i, j as cartesian indices.

$\omega radlib$ provides the convenience function [wradlib.georef.get_radolan_grid()](http://wradlib.org/wradlib-docs/latest/generated/wradlib.georef.get_radolan_grid.html) which returns the radolan grid for further processing. It takes `nrows` and `ncols` as parameters and returns the projected cartesian coordinates or the wgs84 coordinates (keyword arg wgs84=True) as numpy ndarray (nrows x ncols x 2).

```python
import wradlib as wrl
import matplotlib.pyplot as pl
import warnings
warnings.filterwarnings('ignore')
try:
    get_ipython().magic("matplotlib inline")
except:
    pl.ion()
import numpy as np
```

```python
radolan_grid_xy = wrl.georef.get_radolan_grid(900,900)
print("{0}, ({1:.4f}, {2:.4f})".format(radolan_grid_xy.shape, *radolan_grid_xy[0,0,:]))
```

```python
radolan_grid_ll = wrl.georef.get_radolan_grid(900,900, wgs84=True)
print("{0}, ({1:.4f}, {2:.4f})".format(radolan_grid_ll.shape, *radolan_grid_ll[0,0,:]))
```

## Inverse Polar Stereographic Projection

The geographic coordinates of specific datapoints can be calculated by using the cartesian coordinates (x,y) and the following formulas:

$\begin{equation}
\lambda = \arctan\left(\frac {-x} {y}\right) + \lambda_0
\tag{6}
\end{equation}$

$\begin{equation}
\phi = \arcsin\left(\frac {R^2 * \left(1 + \sin\phi_0\right)^2 - \left(x^2 + y^2\right)} {R^2 * \left(1 + \sin\phi_0\right)^2 + \left(x^2 + y^2\right)}\right)
\tag{7}
\end{equation}$

## Standard Formats

### WKT-String

The German Weather Service provides a [WKT-string](https://kunden.dwd.de/geoserver/web/?wicket:bookmarkablePage=:org.geoserver.web.demo.SRSDescriptionPage&code=EPSG:1000001). This WKT (well known text) is used to create the osr-object representation of the radolan projection.

For the scale_factor the intersection of the projection plane with the earth sphere at $60.0^{\circ}N$ has to be taken into account:

$\begin{equation}
scale\_factor = \frac {1 + \sin\left(60.^{\circ}\right)} {1 + \sin\left(90.^{\circ}\right)} = 0.93301270189
\tag{8}
\end{equation}$

Also, the `PROJECTION["Stereographic_North_Pole"]` isn't known within GDAL/OSR. It has to be changed to the known `PROJECTION["polar_stereographic"]`.

With these adaptions we finally yield the Radolan Projection as WKT-string. This WKT-string is used within $\omega radlib$ to create the osr-object by using the helper-function [wradlib.georef.create_osr()](http://wradlib.org/wradlib-docs/latest/generated/wradlib.georef.create_osr.html).

```python
proj_stereo = wrl.georef.create_osr("dwd-radolan")
print(proj_stereo)
```

### PROJ.4

Using the above WKT-String the PROJ.4 representation can be derived as:

```python
    +proj=stere +lat_0=90 +lat_ts=90 +lon_0=10 +k=0.93301270189
    +x_0=0 +y_0=0 +a=6370040 +b=6370040 +to_meter=1000 +no_defs
```

This PROJ.4-string can also be used to create the osr-object by using the helper-function [wradlib.georef.proj4_to_osr()](http://wradlib.org/wradlib-docs/latest/generated/wradlib.georef.proj4_to_osr.html):

```python
# create radolan projection osr object
dwd_string = ('+proj=stere +lat_0=90 +lat_ts=90 +lon_0=10 +k=0.93301270189'
              '+x_0=0 +y_0=0 +a=6370040 +b=6370040 +to_meter=1000 +no_defs')
proj_stereo = wrl.georef.proj4_to_osr(dwd_string)
print(proj_stereo)
```

## Grid Reprojection

Within $\omega radlib$ the [wradlib.georef.reproject()](http://wradlib.org/wradlib-docs/latest/generated/wradlib.georef.reproject.html) function can be used to convert the radolan grid data from xy-space to lonlat-space and back. First, we need to create the necessary Spatial Reference Objects for the RADOLAN-projection and wgs84.

```python
from osgeo import osr
proj_stereo = wrl.georef.create_osr("dwd-radolan")
proj_wgs = osr.SpatialReference()
proj_wgs.ImportFromEPSG(4326)
print(proj_wgs)
```

Then, we call `reproject` with the osr-objects as `projection_source` and `projection_target` parameters.

```python
radolan_grid_ll = wrl.georef.reproject(radolan_grid_xy, projection_source=proj_stereo, projection_target=proj_wgs)
print("{0}, ({1:.4f}, {2:.4f})".format(radolan_grid_ll.shape, *radolan_grid_ll[0,0,:]))
```

And the other way round.

```python
radolan_grid_xy = wrl.georef.reproject(radolan_grid_ll, projection_source=proj_wgs, projection_target=proj_stereo)
print("{0}, ({1:.4f}, {2:.4f})".format(radolan_grid_xy.shape, *radolan_grid_xy[0,0,:]))
```

In the following example the RADOLAN grid is projected to wgs84 and GaussKrüger Zone3.

```python
# create Gauss Krueger zone 3 projection osr object
proj_gk3 = osr.SpatialReference()
proj_gk3.ImportFromEPSG(31467)

# transform radolan polar stereographic projection to wgs84 and then to gk3
radolan_grid_ll = wrl.georef.reproject(radolan_grid_xy,
                                       projection_source=proj_stereo,
                                       projection_target=proj_wgs)
radolan_grid_gk = wrl.georef.reproject(radolan_grid_ll,
                                       projection_source=proj_wgs,
                                       projection_target=proj_gk3)

lon_wgs0 = radolan_grid_ll[:, :, 0]
lat_wgs0 = radolan_grid_ll[:, :, 1]

x_gk3 = radolan_grid_gk[:, :, 0]
y_gk3 = radolan_grid_gk[:, :, 1]

x_rad = radolan_grid_xy[:, :, 0]
y_rad = radolan_grid_xy[:, :, 1]

print("\n------------------------------")
print("source radolan x,y-coordinates")
print(u"       {0}      {1} ".format('x [km]', 'y [km]'))
print("ll: {:10.4f} {:10.3f} ".format(x_rad[0, 0], y_rad[0, 0]))
print("lr: {:10.4f} {:10.3f} ".format(x_rad[0, -1], y_rad[0, -1]))
print("ur: {:10.4f} {:10.3f} ".format(x_rad[-1, -1], y_rad[-1, -1]))
print("ul: {:10.4f} {:10.3f} ".format(x_rad[-1, 0], y_rad[-1, 0]))
print("\n--------------------------------------")
print("transformed radolan lonlat-coordinates")
print(u"      {0}  {1} ".format('lon [degE]', 'lat [degN]'))
print("ll: {:10.4f}  {:10.4f} ".format(lon_wgs0[0, 0], lat_wgs0[0, 0]))
print("lr: {:10.4f}  {:10.4f} ".format(lon_wgs0[0, -1], lat_wgs0[0, -1]))
print("ur: {:10.4f}  {:10.4f} ".format(lon_wgs0[-1, -1], lat_wgs0[-1, -1]))
print("ul: {:10.4f}  {:10.4f} ".format(lon_wgs0[-1, 0], lat_wgs0[-1, 0]))
print("\n-----------------------------------")
print("transformed radolan gk3-coordinates")
print(u"     {0}   {1} ".format('easting [m]', 'northing [m]'))
print("ll: {:10.0f}   {:10.0f} ".format(x_gk3[0, 0], y_gk3[0, 0]))
print("lr: {:10.0f}   {:10.0f} ".format(x_gk3[0, -1], y_gk3[0, -1]))
print("ur: {:10.0f}   {:10.0f} ".format(x_gk3[-1, -1], y_gk3[-1, -1]))
print("ul: {:10.0f}   {:10.0f} ".format(x_gk3[-1, 0], y_gk3[-1, 0]))
```


```python
from EDO import EDO
from sympy import var, Lambda, cos, sin, tan, exp, log, ln, sqrt, solve
x = var('x')
y = var('y')
z = var('z')
```

```python
# Questão 01
x0 = 1
y0 = -1
dy = Lambda((x, y), x**-2 - y/x - y**2)
print('f(x, y) = ', dy(x,y))
I = (1, 1.6)
N = 4
Y = EDO(dy, y0, x0)
y1, y_steps, x_steps = Y.runge_kutta2(I, N, dp=6)
print(f'Solução Aproximada: y(1.6) = {y1}')
```

    f(x, y) =  -y**2 - y/x + x**(-2)
    Solução Aproximada: y(1.6) = -0.618013

```python
# Questão 02
x0 = 1
y0 = 2
dy = Lambda((x, y), 1 + y/x)
print('f(x, y) = ', dy(x,y))
I = (1, 2)
N = 5
Y = EDO(dy, y0, x0)
y1, y_steps, x_steps = Y.runge_kutta2(I, N, dp=6)
print(f'Solução Aproximada: y(2) = {y1}')
```

    f(x, y) =  1 + y/x
    Solução Aproximada: y(2) = 5.377426

```python
# Questão 3
x0 = 1
y0 = 1
z0 = -1
dy = Lambda((x, y, z), z)
dz = Lambda((x, y, z), 2*y**3)
print('f(x, y, z) = ', dy)
print('g(x, y, z) = ', dz)
I = (1, 1.2)
N = 2
Y = EDO(dy, y0, x0)
res = Y.runge_kutta4z(dz, z0, I, N, dp=6)
print(f'Solução Aproximada: y({I[1]}) = {res[0]}, com z({I[1]}) = {res[1]}')
```

    f(x, y, z) =  Lambda((x, y, z), z)
    g(x, y, z) =  Lambda((x, y, z), 2*y**3)
    Solução Aproximada: y(1.2) = 0.83334, com z(1.2) = -0.694436

```python
# Questão 4
x0 = 0
y0 = 1
z0 = 2
dy = Lambda((x, y, z), z)
dz = Lambda((x, y, z), exp(x) - 2*y - 3*z)
print('f(x, y, z) = ', dy)
print('g(x, y, z) = ', dz)
I = (0, 0.6)
N = 3
Y = EDO(dy, y0, x0)
res = Y.runge_kutta2z(dz, z0, I, N, dp=6)
print(f'Solução Aproximada: y({I[1]}) = {res[0]}, com z({I[1]}) = {res[1]}')
```

    f(x, y, z) =  Lambda((x, y, z), z)
    g(x, y, z) =  Lambda((x, y, z), -2*y - 3*z + exp(x))
    Solução Aproximada: y(0.6) = 1.394886, com z(0.6) = 0.05563

```python
# Questão 05
x0 = 0
y0 = 1
dy = Lambda((x, y), x * y**(1/3))
print('f(x, y) = ', dy(x,y))
I = (0, 1)
N = 4
Y = EDO(dy, y0, x0)
y1, y_steps, x_steps = Y.runge_kutta4(I, N, dp=6)
print(f'Solução Aproximada: y(1) = {y1}')
```

    f(x, y) =  x*y**0.333333333333333
    Solução Aproximada: y(1) = 1.539597


```python
import psi4
from psi4 import *
from psi4.core import *
import numpy as np
import os
sys.path.append('os.getcwd()')
from opt_helper import stre, bend, intcosMisc, linearAlgebra
```

## Rational Function Optimization

Rational Function Optimization (RFO) is the method of choice for minimizations. This tutorial will walk through the basic theory of RFO and show a sample calculation.  The method was introduced for geometry optimizations by A. Banerjee, N. Adams, J. Simons, and R. Shepard in _J. Phys. Chem._ 89, 52 (1985).

In the Newton-Raphson method, the potential energy surface is approximated by the truncated Taylor expansion in internal coordinates $q$ and gradient $g$ (where vectors are interpreted as columns).

$$ \epsilon = E(q) - E_0 = g^T \Delta q + \frac{1}{2}(\Delta q)^T \mathbf{H} \Delta q $$

An extension is to express the potential via a [2/2] Pade approximation, where __S__ is the scaling matrix.  If __S__ were zero, then the harmonic approximation would be obtained.

$$ \epsilon = \frac{g^T\Delta q + \frac{1}{2} (\Delta q)^T \textbf{H} \Delta q}{1 + \Delta q^T \textbf{S} \Delta q} $$

The relative energy expression can be rewritten in the form of $N+1$ dimensional vectors, where $N$ is the number of coordinates.

$$ \epsilon = \frac{ \frac{1}{2}\begin{pmatrix} \Delta{q^T} & 1\end{pmatrix}\begin{pmatrix} {\textbf H} & g \\ g^T & 0  \end{pmatrix}\begin{pmatrix}\Delta{q} \\ 1\end{pmatrix} }{ \begin{pmatrix} \Delta q^T & 1 \end{pmatrix} \begin{pmatrix} \textbf S & 0  \\ 0 & 1 \end{pmatrix} \begin{pmatrix} \Delta q \\ 1 \end{pmatrix}}$$

since the right-hand side is
\begin{align}
  &= \frac{ \frac{1}{2}\begin{pmatrix} \Delta{q^T} & 1\end{pmatrix} \begin{pmatrix} \textbf{H} \Delta q  + g \\ g^T \Delta q + 0 \end{pmatrix}}{\begin{pmatrix} \Delta q^T & 1 \end{pmatrix} \begin{pmatrix} \textbf S \Delta q + 0 \\ 0 + 1 \end{pmatrix}} \\
  \\
 &= \frac { \frac{1}{2} \Delta q^T \textbf H \Delta q + \frac{1}{2} \Delta q^T g + \frac{1}{2} g^T \Delta q} { \Delta q^T\textbf S \Delta q + 1}
 \\
\end{align}
which is equivalent to the expression above.

Making the stationary point assumption that
$$\frac{\partial \epsilon}{\partial q } = 0 $$
we can derive the expression for the step.

\begin{align}
\frac{\partial \epsilon}{\partial q} &= \frac {g + \mathbf{H} \Delta q}{1 + \Delta q^T \textbf S \Delta q} - \frac{ g \Delta q^T + \frac{1}{2} \Delta q^T \textbf H \Delta q}{ 1 + \Delta q^T \textbf S \Delta q} \Big( \frac{ 2 \textbf S \Delta q}{1 + \Delta q^T \textbf S \Delta q}\Big) \\
\\
\frac{\partial \epsilon}{\partial q} &= \frac {g + \textbf{H}\Delta q}{1 + \Delta q^T \textbf S \Delta q} - \epsilon \Big( \frac{ 2 \textbf S \Delta q}{1 + \Delta q^T \textbf S \Delta q} \Big) \\
\\
0 &= \frac{ g + \textbf{H} \Delta q - 2 \epsilon \textbf S \Delta q}{1 + \Delta q^T \textbf S \Delta q}\\
\\
0 &=  g + \textbf{H} \Delta q - 2 \epsilon \textbf S \Delta q \\
\\
g + H\Delta q &= 2 \epsilon \textbf S \Delta q = \lambda \textbf S \Delta q\\
\end{align}

where $\lambda$ is defined as $2\epsilon$.  It can be shown that
$$\lambda = g^T \Delta q$$

which allows the stationarity condition to be written as follows.

\begin{align}
\begin{pmatrix} \textbf H \Delta q \\ g^T \Delta q \end{pmatrix} + \begin{pmatrix} g \\ 0 \end{pmatrix} &= \begin{pmatrix} \lambda \textbf S \Delta q \\ \lambda \end{pmatrix} \\
\textrm{or} \\
\begin{pmatrix} \textbf H & g \\ g^T & 0 \end{pmatrix} \begin{pmatrix} \Delta q \\ 1 \end{pmatrix} &= \lambda \begin{pmatrix} {\textbf{S} \Delta q} \\ 1 \end{pmatrix} \\
\end{align}

The matrix __S__ is usually taken to be the identify matrix.  The result is an eigenvalue equation with $\lambda$ as the eigenvalue!

$$
\begin{pmatrix} \textbf H & g \\ g^T & 0 \end{pmatrix}
\begin{pmatrix} \Delta q \\ 1 \end{pmatrix} = \lambda \begin{pmatrix} { \Delta q} \\ 1 \end{pmatrix}
$$

The $N+1$ dimensional matrix on the left is called the "RFO matrix".  We build it from our Hessian and our gradient in internal coordinates.  Then the eigenvectors and eigenvalues of this matrix are determined.  For minimum-energy searches, we usually choose the eigenvector with the lowest value of $\lambda$.  This value is hopefully negative, and is 2 times the  energy change anticipated from the step.  This eigenvector is intermediate-normalized by scaling the last element to 1.  The rest of the vector is the desired RFO step, or displacement in internal coordinates.

Occasionally, this eigenvector has a very small final element, and cannot be intermediate normalized.  This comes about, for example, when the step breaks molecular symmetry and the projected energy change may be zero.  Also, the algorithm is numerically problematic if the gradients are very small.

In practice, RFO performs better than ordinary Newton-Raphson steps if the energy surface being explored is not very harmonic, or if the optimization is begun far from a minimum.  However, if the potential surface is nearly flat (e.g., methane dimer), then RFO like ordinary N-R, will perform poorly.

## Demonstration
Now we prepare a water molecule and show an RFO step computation.

```python
mol = psi4.geometry("""
O
H 1 0.9
H 1 0.9 2 104
""")
# We'll use cc-pVDZ RHF.
psi4.set_options({"basis": "cc-pvdz"})
mol.update_geometry()

# Generate the internal coordinates manually.
intcos = [stre.STRE(0,1), stre.STRE(0,2), bend.BEND(1,0,2)]
for intco in intcos:
    print(intco)

# Handy variables for later.
Natom = mol.natom()
Nintco = len(intcos)
Z = [int(mol.Z(i)) for i in range(Natom)]
```

Now we guess the Hessian and compute the gradient in internal coordinates

```python
# Compute initial guess Hessian.
xyz = np.array(mol.geometry())

H = np.zeros((Nintco,Nintco), float)
for i,intco in enumerate(intcos):
    H[i,i] = intco.diagonalHessianGuess(xyz, Z, guessType="SCHLEGEL")

print("\n Schlegel Guess Hessian for Water (in au)")
print(H)

g_x = np.reshape( np.array( psi4.gradient('scf')), (3*Natom))
g_q = -1 * intcosMisc.qForces(intcos, xyz, g_x)

print("Gradient in internal coordinates (au)")
print(g_q)
```

Now we build the RFO matrix and diagonalize it.

```python
dim = Nintco + 1
RFOmat = np.zeros( (dim, dim), float)
RFOmat[0:-1,0:-1] = H
RFOmat[-1,0:-1] = g_q
RFOmat[0:-1,-1] = g_q
print("RFO matrix")
print(RFOmat)

evals, evects = linearAlgebra.symmMatEig(RFOmat)
print("Eigenvalues")
print(evals)
```

Happily, there is in this case only 1 direction with a negative predicted energy change.  We choose the corresponding eigenvector, and then intermediate normalize it.  That is our desired RFO step in internal coordinates!

```python
RFOdq = evects[0]
print("RFO eigenvector")
print(RFOdq)
RFOdq[:] = RFOdq / RFOdq[-1]
print("RFO intermediate normalized eigenvector")
print(RFOdq)
# Now drop the 1 at the end.
print("RFO step in internal coordinates")
RFOdq = RFOdq[0:-1]
print(RFOdq)
```

This is the desired step in internal coordinates.  We can see that the bond lengths and the bond angle will all increase.

## Extensions
The Partioned-RFO method is an extension of the RFO method, and may be used to seek non-minimum stationary points such as transition states.  In the P-RFO method, there are two RFO matrices.  One is used for maximization along 1 or more degrees of freedom while the second one is for minimization along the others.  For the P-RFO method to be effective the starting geometry must be reasonable, and the curvature of the surface in different directions must be correctly represented by the Hessian.  Thus, it is usually necessary to compute the Hessian for a transition-state optimization so that the algorithm begin maximizing in the right direction.

An extended scheme to limit the RFO step size called the restricted-step for RS-RFO method may be found in E. Besalu and J.M. Bofill, _Theor. Chem. Acc._ __100__, 265 (1998).  This involves an interation over repeated RFO matrix diagonalizations and does not always converge.  However, if it fails one simply resorts to more simple step-size scaling.


# Random Signals and LTI-Systems

*This jupyter notebook is part of a [collection of notebooks](../index.ipynb) on various topics of Digital Signal Processing. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## The Wiener Filter

The [Wiener filter](https://en.wikipedia.org/wiki/Wiener_filter), named after [*Nobert Wiener*](https://en.wikipedia.org/wiki/Norbert_Wiener), aims at estimating an unknown random signal by filtering a noisy distorted observation of the signal. It has a wide range of applications in signal processing. For instance, noise reduction, system identification, deconvolution and signal detection. The Wiener filter is frequently used as basic building block for algorithms that denoise audio signals, like speech, or remove noise from a image.

### Signal Model

The following signal model is underlying the Wiener filter

The random signal $s[k]$ is subject to distortion by the linear time-invariant (LTI) system $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and additive noise $n[k]$, resulting in the observed signal $x[k] = s[k] * g[k] + n[k]$. The additive noise $n[k]$ is assumed to be uncorrelated from $s[k]$. It is furthermore assumed that all random signals are wide-sense stationary (WSS). This distortion model holds for many practical problems, like e.g. the measurement of a physical quantity by a sensor.

The basic concept of the Wiener filter is to apply the LTI system $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ to the observed signal $x[k]$ such that the output signal $y[k] = x[k] * h[k]$ matches $s[k]$ as close as possible. In order to quantify the deviation between $y[k]$ and $s[k]$, the error signal

\begin{equation}
e[k] = y[k] - s[k]
\end{equation}

is introduced. The error $e[k]$ equals zero, if the output signal $y[k]$ perfectly matches $s[k]$. In general, this goal cannot be achieved in a strict sense. As alternative one could aim at minimizing the linear average of the error $e[k]$. However, this quantity is not very well suited for optimization since it is signed. Instead, the quadratic average of the error $e[k]$ is used in the Wiener filter and other techniques. This measure is known as [*mean squared error*](https://en.wikipedia.org/wiki/Mean_squared_error) (MSE). It is defined as

\begin{equation}
E \{ |e[k]|^2 \} = \frac{1}{2 \pi} \int\limits_{-\pi}^{\pi} \Phi_{ee}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \,\mathrm{d}\Omega = \varphi_{ee}[\kappa] \Big\vert_{\kappa = 0}
\end{equation}

the equalities are deduced from the properties of the power spectral density (PSD) and the auto-correlation function (ACF). Above equation is referred to as [*cost function*](https://en.wikipedia.org/wiki/Loss_function). We aim at minimizing the cost function, hence minimizing the MSE between the original signal $s[k]$ and its estimate $y[k]$. The solution is referred to as [minimum mean squared error](https://en.wikipedia.org/wiki/Minimum_mean_square_error) (MMSE) solution.

### Transfer Function of the Wiener Filter

At first, the Wiener filter shall only have access to the observed signal $x[k]$ and selected statistical measures. It is assumed that the cross-power spectral density $\Phi_{xs}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ between the observed signal $x[k]$ and the original signal $s[k]$, and the PSD of the observed signal $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ are known. This knowledge can either be gained by estimating both from measurements taken at an actual system or by using suitable statistical models.

The optimal filter is found by minimizing the MSE $E \{ |e[k]|^2 \}$ with respect to the transfer function $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$. The solution of this optimization problem goes beyond the scope of this notebook and can be found in the literature, e.g. [[Girod et. al](../index.ipynb#Literature)]. The transfer function of the Wiener filter is given as

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})} = \frac{\Phi_{xs}(\mathrm{e}^{\,-\mathrm{j}\,\Omega})}{\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}
\end{equation}

for $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \neq 0$.
No knowledge on the actual distortion process is required besides the assumption of an LTI system and additive noise. Only the PSDs $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ have to be known in order to estimate $s[k]$ from $x[k]$ in the MMSE sense. Care has to be taken that the filter $H(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ is causal and stable in practical applications.

**Example - Denoising of an Deterministic Signal**

The following example considers the estimation of the original signal from a distorted observation. It is assumed that the original signal is a deterministic signal $s[k] = \sin[\Omega_0\,k]$ which is distorted by an LTI system and additive wide-sense ergodic normal distributed zero-mean white noise with PSD $N_0 = 0.1$. The PSDs $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ are estimated from $s[k]$ and $x[k]$ using the [Welch technique](../spectral_estimation_random_signals/welch_method.ipynb). The Wiener filter is applied to the observation $x[k]$ in order to compute the estimate $y[k]$ of $s[k]$. Note that the discrete signals have been illustrated by continuous lines for ease of illustration.

```python
import numpy as np
import matplotlib.pyplot as plt
import scipy.signal as sig
%matplotlib inline

N = 2**14  # number of samples
M = 256  # length of Wiener filter
Om0 = 0.1*np.pi  # frequency of original signal
N0 = 0.1  # PSD of additive white noise

# generate original signal
s = np.cos(Om0 * np.arange(N))
# generate observed signal
g = 1/20*np.asarray([1, 2, 3, 4, 5, 4, 3, 2, 1])
np.random.seed(1)
n = np.random.normal(size=N, scale=np.sqrt(N0))
x = np.convolve(s, g, mode='same') + n
# estimate (cross) PSDs using Welch technique
f, Pxx = sig.csd(x, x, nperseg=M)
f, Psx = sig.csd(s, x, nperseg=M)
# compute Wiener filter
H = Psx/Pxx
H = H * np.exp(-1j*2*np.pi/len(H)*np.arange(len(H))*(len(H)//2))  # shift for causal filter
h = np.fft.irfft(H)
# apply Wiener filter to observation
y = np.convolve(x, h, mode='same')

# plot (cross) PSDs
Om = np.linspace(0, np.pi, num=len(H))

plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.plot(Om, 20*np.log10(np.abs(.5*Pxx)), label=r'$| \Phi_{xx}(e^{j \Omega}) |$ in dB')
plt.plot(Om, 20*np.log10(np.abs(.5*Psx)), label=r'$| \Phi_{sx}(e^{j \Omega}) |$ in dB')
plt.title('(Cross) PSDs')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -60, 40])
plt.grid()

# plot transfer function of Wiener filter
plt.subplot(122)
plt.plot(Om, 20*np.log10(np.abs(H)))
plt.title('Transfer function of Wiener filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$| H(e^{j \Omega}) |$ in dB')
plt.axis([0, np.pi, -150, 3])
plt.grid()
plt.tight_layout()

# plot signals
idx = np.arange(500, 600)
plt.figure(figsize=(10, 4))
plt.plot(idx, x[idx], label=r'observed signal $x[k]$')
plt.plot(idx, s[idx], label=r'original signal $s[k]$')
plt.plot(idx, y[idx], label=r'estimated signal $y[k]$')
plt.title('Signals')
plt.xlabel(r'$k$')
plt.axis([idx[0], idx[-1], -1.5, 1.5])
plt.legend()
plt.grid()
```

Let's listen to the observed signal $x[k]$ and the output of the Wiener filter $y[k]$

```python
from scipy.io import wavfile

fs = 8000
wavfile.write('Wiener_observed_signal.wav', fs, np.int16(x*16384))
wavfile.write('Wiener_output.wav', fs, np.int16(y*16384))
```

**Observed signal**

<audio src="./Wiener_observed_signal.wav" controls>Your browser does not support the audio element.</audio>[./Wiener_observed_signal.wav](./Wiener_observed_signal.wav)

**Output of Wiener filter**

<audio src="./Wiener_output.wav" controls>Your browser does not support the audio element.</audio>[./Wiener_output.wav](./Wiener_output.wav)

**Exercise**

* Take a look at the PSDs and the resulting transfer function of the Wiener filter. How does the Wiener filter remove the noise from the observed signal?
* Change the frequency `Om0` of the original signal $s[k]$ and the noise power `N0` of the additive noise. What changes?

Solution: The cross-PSD $\Phi_{sx}(e^{j \Omega})$ has its maximum at the frequency $\Omega_0$ of the original signal $s[k]$, besides this frequency the PSD $\Phi_{xx}(e^{j \Omega})$ has approximately the value of the PSD $\Phi_{nn}(e^{j \Omega}) = N_0$ of the additive noise. The resulting Wiener filter $G(e^{j \Omega})$ has an attenuation of 0 dB at $\Omega_0$ which increases off this frequency. It has the characteristics of a band-pass filter centered at $\Omega_0$. By filtering the observed signal $x[k]$ with the Wiener filter, the major portion of the additive white noise is removed. However, around $\Omega_0$ both the original signal $s[k]$ and noise pass the filter. This explains the remaining small deviations between the estimated signal $y[k]$ and the original signal $y[k]$. The adaption of the Wiener filter to the original signal and additive noise changes when changing its frequency and PSD.

### Wiener Deconvolution

As discussed above, the general formulation of the Wiener filter is based on the knowledge of the PSDs $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and $\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ characterizing the distortion process and the observed signal, respectively. These PSDs can be derived from the PSDs of the original signal $\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ and the noise $\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$, and the transfer function $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the distorting system.

Under the assumption that $n[k]$ is uncorrelated from $s[k]$, the PSD $\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ can be derived as

\begin{equation}
\Phi_{sx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot G(\mathrm{e}^{\,-\mathrm{j}\,\Omega})
\end{equation}

and

\begin{equation}
\Phi_{xx}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot  |G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})
\end{equation}

Introducing these results into the general formulation of the Wiener filter yields

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot G(\mathrm{e}^{\,-\mathrm{j}\,\Omega})}{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) \cdot  |G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 + \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}
\end{equation}

This specialization is also known as [*Wiener deconvolution filter*](https://en.wikipedia.org/wiki/Wiener_deconvolution). The filter can be derived from the PSDs of the original signal $s[k]$ and the noise $n[k]$, and the transfer function $G(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ of the distorting system. This form is especially useful when the PSDs can be modeled reasonably well by analytic probabilty density functions (PSDs). In many cases, the additive noise can be modeled as white noise $\Phi_{nn}(e^{j \Omega}) = N_0$.

### Interpretation

The Wiener deconvolution filter derived above is also useful for an interpretation of the operation of the Wiener filter. It can be rewritten by introducing the frequency dependent [signal-to-noise ratio](https://en.wikipedia.org/wiki/Signal-to-noise_ratio) $\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$ between the original signal and the noise as

\begin{equation}
H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{G(\mathrm{e}^{\,\mathrm{j}\,\Omega})} \cdot \left(   \frac{|G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2}{|G(\mathrm{e}^{\,\mathrm{j}\,\Omega})|^2 + \frac{1}{\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}} \right)
\end{equation}

Now two special cases are discussed:

1. If there is no additive noise $\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 0$, the bracketed expression is equal to 1. Hence, the Wiener filter is simply given as the inverse system to the distorting system $$H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{1}{G(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$$

2. If the distorting system is just a pass-through $G(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = 1$, the Wiener filter is given as

    $$H(\mathrm{e}^{\,\mathrm{j}\,\Omega}) = \frac{\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega}) + 1}=\frac{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}{\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})+\Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})}$$
    Two extreme cases are considered
    * for $\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\gg \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ (high $\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$) at a given frequency $\Omega$ the transfer function approaches 1
    * for $\Phi_{ss}(\mathrm{e}^{\,\mathrm{j}\,\Omega})\ll \Phi_{nn}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$ (low $\text{SNR}(\mathrm{e}^{\,\mathrm{j}\,\Omega})$) at a given frequency $\Omega$ the transfer function approaches 0

The discussed cases explain the operation of the Wiener filter for specific situations, the general operation is a combination of these.

**Example - Denoising of a deterministic signal with the Wiener deconvolution filter**

The preceding example of the general Wiener filter will now be reevaluated with the Wiener deconvolution filter.

```python
N = 2**14  # number of samples
M = 256  # length of Wiener filter
Om0 = 0.1*np.pi  # frequency of original signal
N0 = .1  # PSD of additive white noise

# generate original signal
s = np.cos(Om0 * np.arange(N))
# generate observed signal
g = 1/20*np.asarray([1, 2, 3, 4, 5, 4, 3, 2, 1])
np.random.seed(1)
n = np.random.normal(size=N, scale=np.sqrt(N0))
x = np.convolve(s, g, mode='same') + n
# estimate PSD
f, Pss = sig.csd(s, s, nperseg=M)
f, Pnn = sig.csd(n, n, nperseg=M)
# compute Wiener filter
G = np.fft.rfft(g, M)
H = 1/G * (np.abs(G)**2 / (np.abs(G)**2 + N0/Pss))
H = H * np.exp(-1j*2*np.pi/len(H)*np.arange(len(H)) *
               (len(H)//2-8))  # shift for causal filter
h = np.fft.irfft(H)
# apply Wiener filter to observation
y = np.convolve(x, h, mode='same')

# plot (cross) PSDs
Om = np.linspace(0, np.pi, num=len(H))

plt.figure(figsize=(10, 4))
plt.subplot(121)
plt.plot(Om, 20*np.log10(np.abs(.5*Pss)),
         label=r'$| \Phi_{ss}(e^{j \Omega}) |$ in dB')
plt.plot(Om, 20*np.log10(np.abs(.5*Pnn)),
         label=r'$| \Phi_{nn}(e^{j \Omega}) |$ in dB')
plt.title('PSDs')
plt.xlabel(r'$\Omega$')
plt.legend()
plt.axis([0, np.pi, -60, 40])
plt.grid()

# plot transfer function of Wiener filter
plt.subplot(122)
plt.plot(Om, 20*np.log10(np.abs(H)))
plt.title('Transfer function of Wiener filter')
plt.xlabel(r'$\Omega$')
plt.ylabel(r'$| H(e^{j \Omega}) |$ in dB')
plt.axis([0, np.pi, -150, 10])
plt.grid()
plt.tight_layout()

# plot signals
idx = np.arange(500, 600)
plt.figure(figsize=(10, 4))
plt.plot(idx, x[idx], label=r'observed signal $x[k]$')
plt.plot(idx, s[idx], label=r'original signal $s[k]$')
plt.plot(idx, y[idx], label=r'estimated signal $y[k]$')
plt.title('Signals')
plt.xlabel(r'$k$')
plt.axis([idx[0], idx[-1], -1.5, 1.5])
plt.legend()
plt.grid()
```

**Exercise**

* Compare the transfer function of the Wiener deconvolution filter to the transfer function of the general Wiener filter from the preceding example.
* What is different compared to the general Wiener filter? Why?

Solution: The transfer function $H(e^{j \Omega})$ of the Wiener deconvolution filter is much smoother than the transfer function of the general Wiener filter from the preceding example. This is due to the knowledge of the transfer function $G(e^{j \Omega})$ of the disturbing system. The general Wiener filter inherently estimates this from the noisy observations.

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Digital Signal Processing - Lecture notes featuring computational examples*.


# Boundary Value Problem Example - Heated Rod
## CH EN 2450 - Numerical Methods
**Prof. Tony Saad (www.tsaad.net)
Department of Chemical Engineering
University of Utah**

```python
%matplotlib inline
%config InlineBackend.figure_format = 'svg'
import matplotlib.pyplot as plt
import numpy as np
```

# Dirichlet Boundary Conditions

The coefficient matrix looks like
\begin{equation}
\left[ {\begin{array}{*{20}{c}}
1&0&0& \cdots &0\\
1& - \left( 2 + \Delta {x^2}H \right)&1&0& \vdots \\
0& \ddots & \ddots & \ddots &0\\
 \vdots &0&1& - \left( 2 + \Delta x^2H \right)&1\\
0& \cdots &0&0&1
\end{array}} \right]
\left( {\begin{array}{*{20}{c}}
T_1\\
\begin{array}{l}
T_2\\
T_3
\end{array}\\
\begin{array}{l}
 \vdots \\
 \vdots
\end{array}\\
T_{n - 1}\\
T_n
\end{array}} \right)
= \left( {\begin{array}{*{20}{c}}
T_\rm{left}\\
- T_\infty \Delta {x^2}H\\
 \vdots \\
- T_\infty\Delta {x^2}H\\
T_\rm{right}
\end{array}} \right)
\end{equation}

```python
n = 200 # number of desired grid points
L = 1.0 # rod length
Tleft = 500
Tright = 499
Tinf = 6000
h = 0.0001 # heat transfer coefficient, due to advection of surrounding air

x = np.linspace(0, L, n) # create an array for the grid points x_i
dx = x[1] - x[0] # calculate grid spacing (grid spacing is constant here)

# create lower diagonal vector
ld = np.ones(n-1)
ld[-1] = 0.0

# create main diagonal vector
d = -(2.0 + dx*dx * h) * np.ones(n)
d[0] = 1.0
d[-1] = 1.0

# create upper diagonal vector
ud = np.ones(n-1)
ud[0] = 0.0

# combine those into a matrix
A = np.diag(ld,-1) + np.diag(d, 0) + np.diag(ud, 1)

# build the RHS array
rhs = - Tinf * dx*dx * h * np.ones(n)
rhs[0] = Tleft
rhs[-1] = Tright

# solve the system of equations
sol = np.linalg.solve(A, rhs)

plt.plot(x, sol)
plt.ylim(min(sol), max(sol))
plt.grid()
```

```python
temperature = np.tile(sol,(40,1))
plt.imshow(temperature,cmap='jet')
```

    <matplotlib.image.AxesImage at 0x119ecc910>

# Neumann Boundary Conditions

```python
def solve_heat_eq_neumann(H, L, Tleft, β, Tinf, n):
    x = np.linspace(0, L, n)
    dx = x[1] - x[0]

    ld = np.ones(n-1)
    ld[-1] = 2.0

    d = -(2.0 + dx*dx * H) * np.ones(n)
    d[0] = 1.0

    ud = np.ones(n-1)
    ud[0] = 0.0

    A = np.diag(ld,-1) + np.diag(d, 0) + np.diag(ud, 1)

    rhs = - Tinf * dx*dx * H * np.ones(n)
    rhs[0] = Tleft
    rhs[-1] -= 2.0*β*dx

    sol = np.linalg.solve(A, rhs)
    return x, sol
```

```python
n = 100
L = 1.0
Tleft = 300
β = -500.0
Tinf = 200
H = 10.0
x, sol = solve_heat_eq_neumann(H, L, Tleft, β, Tinf, n)
plt.plot(x, sol)
plt.grid()
```

```python

```

topics/boundary value problems/Boundary Value Problem Example - Heated Rod.ipynb