# The Laplace Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Properties

The Laplace transform has a number of specific properties which can be concluded directly from its definition. The most important ones in the context of signals and systems are reviewed in the following.

### Linearity

As can be concluded from its definition, the Laplace transform is a linear operation. For two signals $x_1(t)$ and $x_2(t)$ with Laplace transforms $X_1(s) = \mathcal{L} \{ x_1(t) \}$ and $X_2(s) = \mathcal{L} \{ x_2(t) \}$ the following holds

\begin{equation}
\mathcal{L} \{ A \cdot x_1(t) + B \cdot x_2(t) \} = A \cdot X_1(s) + B \cdot X_2(s)
\end{equation}

with $A, B \in \mathbb{C}$. The region of convergence (ROC) of the superposition $A x_1(t) + B x_2(t)$ is as least as large as the intersection of the ROCs of $x_1(t)$ and $x_2(t)$

\begin{equation}
\text{ROC} \{ A \cdot x_1(t) + B \cdot x_2(t) \} \supseteq \text{ROC} \{ x_1(t) \} \cap \text{ROC} \{ x_2(t) \}
\end{equation}

The Laplace transform of a weighted superposition of signals is equal to the weighted superposition of the individual Laplace transforms. This property is useful to derive the Laplace transform of signals that can be expressed as superposition of other signals for which the Laplace transforms are known or can be calculated easier.

#### Transformation of the cosine and sine signal

The Laplace transform of the causal cosine $\epsilon(t) \cdot \cos(\omega_0 t)$ and sine $\epsilon(t) \cdot \sin(\omega_0 t)$ signal is derived by expressing the cosine/sine as harmonic exponential signals using [Euler's formula](https://en.wikipedia.org/wiki/Euler's_formula)

\begin{align}
\cos(\omega_0 t) &= \frac{1}{2} \left( e^{j \omega_0 t} + e^{-j \omega_0 t} \right) \\
\sin(\omega_0 t) &= \frac{1}{2j} \left( e^{j \omega_0 t} - e^{-j \omega_0 t} \right)
\end{align}

The Laplace transform of the [causal complex exponential signal](definition.ipynb#Transformation-of-the-Causal-Complex-Exponential-Signal) reads

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot e^{-s_0 t} \} = \frac{1}{s + s_0} \qquad \text{for } \Re \{ s \} > \Re \{ - s_0 \}
\end{equation}

Using the linearity of the Laplace transform yields

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot \cos(\omega_0 t) \} =
\frac{1}{2} \left( \frac{1}{s - j \omega_0} + \frac{1}{s + j \omega_0} \right) = \frac{s}{s^2 + \omega_0^2}
\qquad \text{for } \Re \{ s \} > 0
\end{equation}

and

\begin{equation}
\mathcal{L} \{ \epsilon(t) \cdot \sin(\omega_0 t) \} =
\frac{1}{2j} \left( \frac{1}{s - j \omega_0} - \frac{1}{s + j \omega_0} \right) = \frac{\omega_0}{s^2 + \omega_0^2}
\qquad \text{for } \Re \{ s \} > 0
\end{equation}

**Example**

The Laplace transform of the causal cosine signal is calculated in `Sympy`

```python
%matplotlib inline
import sympy as sym
sym.init_printing()

t, w0 = sym.symbols('t omega0', real=True)
s = sym.symbols('s', complex=True)

x = sym.sin(w0*t)
X, a, cond = sym.laplace_transform(x, t, s)
X, a, cond
```

**Exercise**

* Determine the zeros and poles of the Laplace transform $\mathcal{L} \{ \epsilon(t) \cdot \sin(\omega_0 t) \}$ by manual calculus and by extending above example. What specific symmetries do the zeros and poles show in the $s$-plane?

### Symmetry for Real-Valued Signals

The Laplace transform of the complex conjugate $x^*(t)$ of a signal $x(t)$ is given as

\begin{equation}
\mathcal{L} \{ x^*(t) \} = \int_{-\infty}^{\infty} x^*(t) \, e^{-s t} dt
= \left( \int_{-\infty}^{\infty} x(t) \, e^{-s^* t} dt \right)^*
= X^*(s^*)
\end{equation}

where $X(s) = \mathcal{L} \{ x(t) \}$. The ROC for $x^*(t)$ is equal to the ROC for $x(t)$. Above result is used to derive the Laplace transform of the real part of the signal $x(t)$

\begin{equation}
\mathcal{L} \left\{ \Re \{ x(t) \} \right\} = \mathcal{L} \left\{ \frac{1}{2} [ x(t) + x^*(t) ] \right\} = \frac{1}{2} \left[ X(s) + X^*(s^*) \right]
\end{equation}

and imaginary part

\begin{equation}
\mathcal{L} \left\{ \Im \{ x(t) \} \right\} = \mathcal{L} \left\{ \frac{1}{2j} [ x(t) - x^*(t) ] \right\} = \frac{1}{2j} \left[ X(s) - X^*(s^*) \right]
\end{equation}

For a real-valued signal $x(t) \in \mathbb{R}$ with $\Im \{ x(t) \} = 0$ it follows that

\begin{equation}
X(s) = X^*(s^*)
\end{equation}

It can be concluded, that the Laplace transform of a real valued signal shows complex conjugate symmetry. The real/imaginary part of the Laplace transform show even/odd symmetry

\begin{align}
\Re \{ X(s) \} &= \Re \{ X(-s) \} \\
\Im \{ X(s) \} &= - \Im \{ X(-s) \}
\end{align}

and accordingly its magnitude/phase show even/odd symmetry

\begin{align}
|X(s)| &= |X(-s)| \\
\varphi(s) &= - \varphi(-s)
\end{align}

where $\varphi(s) = \arg \{ H(s) \}$.

The poles and zeros of a [Laplace transform which can be expressed as a rational function](definition.ipynb#Representation) in $s$

\begin{equation}
X(s) = \frac{\sum_{m=0}^{M} \beta_m s^m}{\sum_{n=0}^{N} \alpha_n s^n} = K \cdot \frac{\prod_{\mu=0}^{Q} (s - s_{0 \mu})}{\prod_{\nu=0}^{P} (s - s_{\infty \nu})}
\end{equation}

with $Q=M-1$ and $P=N-1$ show also specific symmetries. From above condition for the Laplace transform of a real-valued signal $x(t) \in \mathbb{R}$, it can be concluded that the coefficients $\alpha_n, \beta_m$ of the numerator and denominator are real-valued. The [complex conjugate root theorem](https://en.wikipedia.org/wiki/Complex_conjugate_root_theorem) states that the roots of a polynomial with real valued coefficients are either real-valued or complex conjugate pairs. Hence, the zeros and poles of the Laplace transform $X(s)$ of a real-valued signal $x(t)$ are either real-valued or constitute complex conjugate pairs.

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


# Measuring qubits in qoqo

This notebook is designed to demonstrate the use of measurements in qoqo. We will look at several examples of measuring qubits, from single and multi-qubit registers. To learn about the effect of measurement, we will look at the state vectors before and after measurement.

```python
from qoqo_quest import Backend
from qoqo import Circuit
from qoqo import operations as ops
```

## Measuring a single qubit

Here we first prepare the qubit in a superposition state,
\begin{equation}
|+ \rangle = \frac{1}{\sqrt{2}} \big ( |0 \rangle + |1 \rangle \big ).
\end{equation}
We look at the state after preparation, then do a measurement in the Z basis, and finally look again at the state after measurement.

We see that the state after measurement has been projected into the state either $|0>$ or $|1>$, consistently with the measurement outcome. Running this code many times should result in a random distribution of 'True' and 'False' outcomes.

```python
state_init = Circuit()
state_init += ops.Hadamard(qubit=0)  # prepare |+> state

# write state before measuring to readout register 'psi_in'
read_input = Circuit()
read_input += ops.DefinitionComplex(name='psi_in', length=2, is_output=True)
read_input += ops.PragmaGetStateVector(readout='psi_in', circuit=Circuit())

# measure qubit in Z basis and write result to classical register 'M1'
meas_circ = Circuit()
meas_circ += ops.DefinitionBit(name='M1', length=1, is_output=True)
meas_circ += ops.MeasureQubit(qubit=0,readout='M1',readout_index=0)

# write state after measuring to readout register 'psi_out'
read_output = Circuit()
read_output += ops.DefinitionComplex(name='psi_out', length=2, is_output=True)
read_output += ops.PragmaGetStateVector(readout='psi_out', circuit=Circuit())

# put each step of the circuit together
circuit = state_init + read_input + meas_circ + read_output

# run the circuit and collect output
backend = Backend(number_qubits=1)
(result_bit_registers, result_float_registers, result_complex_registers) = backend.run_circuit(circuit)

print('Input state: \n', result_complex_registers['psi_in'][0], '\n')
print('Measurement result: ', result_bit_registers['M1'][0][0], '\n')
print('State after measurement: \n', result_complex_registers['psi_out'][0])
```

    Input state:
     [(0.7071067811865475+0j), (0.7071067811865475+0j)]

    Measurement result:  False

    State after measurement:
     [(1+0j), 0j]

## Measuring a single qubit in the X basis

Instead of measuring in the Z basis, we can measure the qubit in the X basis by performing a Hadamard operator before the measurement.

This time we see that the measurement result is always 'False', since we are measuring the $|+ \rangle$ state in the X basis, and it is an X eigenvector of the X operator.

```python
# add Hadamard operator to change from Z to X basis
meas_X_circ = Circuit()
meas_X_circ += ops.DefinitionBit(name='M1', length=1, is_output=True)
meas_X_circ += ops.Hadamard(qubit=0)
meas_X_circ += ops.MeasureQubit(qubit=0,readout='M1',readout_index=0)

# perform additional Hadamard after measurement to readout in Z basis
read_output = Circuit()
read_output += ops.DefinitionComplex(name='psi_out', length=2, is_output=True)
read_output += ops.Hadamard(qubit=0)
read_output += ops.PragmaGetStateVector(readout='psi_out', circuit=Circuit())

circuit = state_init + read_input + meas_X_circ + read_output

# run the circuit and collect output
backend = Backend(number_qubits=1)
(result_bit_registers, result_float_registers, result_complex_registers) \
        = backend.run_circuit(circuit)

print('Input state: \n', result_complex_registers['psi_in'][0], '\n')
print('Measurement result: ', result_bit_registers['M1'][0][0], '\n')
print('State after measurement: \n', result_complex_registers['psi_out'][0])
```

    Input state:
     [(0.7071067811865475+0j), (0.7071067811865475+0j)]

    Measurement result:  False

    State after measurement:
     [(0.7071067811865475+0j), (0.7071067811865475+0j)]

## Measuring a multi-qubit register

Here we first prepare a multi-qubit register and demonstrate how it is possible to measure the entire register. As an example we prepare the multi-qubit register in the state,
\begin{equation}
|\psi \rangle = \frac{1}{\sqrt{2}} |010 \rangle + \frac{i}{\sqrt{2}} |101 \rangle.
\end{equation}

After preparation we read out the simulated state, before measurement. Next we measure each qubit of the state, and finally we readout out the post-measurement state.

```python
number_of_qubits = 3

state_init = Circuit()
state_init += ops.PauliX(qubit=1)
state_init += ops.Hadamard(qubit=0)
state_init += ops.CNOT(control=0, target=1)
state_init += ops.CNOT(control=0, target=2)
state_init += ops.SGate(qubit=0)

# write state before measuring to readout register 'psi_in'
read_input = Circuit()
read_input += ops.DefinitionComplex(name='psi_in', length=2**number_of_qubits,
                                    is_output=True)
read_input += ops.PragmaGetStateVector(readout='psi_in', circuit=Circuit())

# measure qubits in Z basis and write result to classical register 'M1M2M3'
meas_circ = Circuit()
meas_circ += ops.DefinitionBit(name='M1M2M3', length=3, is_output=True)
meas_circ += ops.MeasureQubit(qubit=0,readout='M1M2M3',readout_index=0)
meas_circ += ops.MeasureQubit(qubit=1,readout='M1M2M3',readout_index=1)
meas_circ += ops.MeasureQubit(qubit=2,readout='M1M2M3',readout_index=2)

# write state after measuring to readout register 'psi_out'
read_output = Circuit()
read_output += ops.DefinitionComplex(name='psi_out', length=2**number_of_qubits,
                                    is_output=True)
read_output += ops.PragmaGetStateVector(readout='psi_out', circuit=Circuit())

circuit = state_init + read_input + meas_circ + read_output

# run the circuit and collect output
backend = Backend(number_qubits=number_of_qubits)
(result_bit_registers, result_float_registers, result_complex_registers) \
        = backend.run_circuit(circuit)

print('Input state: \n', result_complex_registers['psi_in'][0], '\n')
print('Measurement results: ', result_bit_registers['M1M2M3'][0], '\n')
print('State after measurement: \n', result_complex_registers['psi_out'][0])
```

    Input state:
     [0j, 0j, (0.7071067811865475+0j), 0j, 0j, 0.7071067811865475j, 0j, 0j]

    Measurement results:  [False, True, False]

    State after measurement:
     [0j, 0j, (1+0j), 0j, 0j, 0j, 0j, 0j]

## Measuring one qubit from a multi-qubit register

Measuring only a single qubit from a multi-qubit register is an almost identical process to measuring the entire register, except we only add a single measurement in this case.

Here we again prepare the input state,
\begin{equation}
|\psi \rangle = \frac{1}{\sqrt{2}} |010 \rangle + \frac{i}{\sqrt{2}} |101 \rangle.
\end{equation}

After preparation we read out the simulated state, before measurement. Next we measure the first qubit of the state, and finally we readout out the post-measurement state.

```python
number_of_qubits = 3

state_init = Circuit()
state_init += ops.PauliX(qubit=1)
state_init += ops.Hadamard(qubit=0)
state_init += ops.CNOT(control=0, target=1)
state_init += ops.CNOT(control=0, target=2)
state_init += ops.SGate(qubit=0)

# write state before measuring to readout register 'psi_in'
read_input = Circuit()
read_input += ops.DefinitionComplex(name='psi_in', length=2**number_of_qubits,
                                    is_output=True)
read_input += ops.PragmaGetStateVector(readout='psi_in', circuit=Circuit())

# measure qubit in Z basis and write result to classical register 'M1'
meas_circ = Circuit()
meas_circ += ops.DefinitionBit(name='M1', length=1, is_output=True)
meas_circ += ops.MeasureQubit(qubit=0,readout='M1',readout_index=0)

# write state after measuring to readout register 'psi_out'
read_output = Circuit()
read_output += ops.DefinitionComplex(name='psi_out', length=2**number_of_qubits,
                                    is_output=True)
read_output += ops.PragmaGetStateVector(readout='psi_out', circuit=Circuit())

circuit = state_init + read_input + meas_circ + read_output

# run the circuit and collect output
backend = Backend(number_qubits=number_of_qubits)
(result_bit_registers, result_float_registers, result_complex_registers) \
        = backend.run_circuit(circuit)

print('Input state: \n', result_complex_registers['psi_in'][0], '\n')
print('Measurement results: ', result_bit_registers['M1'][0], '\n')
print('State after measurement: \n', result_complex_registers['psi_out'][0])
```

    Input state:
     [0j, 0j, (0.7071067811865475+0j), 0j, 0j, 0.7071067811865475j, 0j, 0j]

    Measurement results:  [True]

    State after measurement:
     [0j, 0j, 0j, 0j, 0j, 0.9999999999999998j, 0j, 0j]


## Cox Process

A Cox process [1] is closely related to a Poisson process [2] commonly used to model event arrivals described by a certain rate $\lambda > 0$. A discrete random variable $X$ is Poisson distributed if the number of events or counts $k$, has the following form:

\begin{equation}
Poiss(k; \lambda) = Pr\{X=k\} = \frac{\lambda^{k}}{k!}\exp(-\lambda)
\end{equation}

Note that the mean of the Poisson RV is equal to its rate as well as its variance: $E[X] = VAR[X] = \lambda$. In addition, the rates are additive. If $X_i$ are iid Poisson random variables with rate $\lambda_i$, then

\begin{equation}
    S = \sum_{i=i}^{n} X_i \sim Poiss\bigg(\sum_{i=1}^{n} \lambda_i \bigg)
\end{equation}

A Poisson process is defined as random countable subset of a space $S$ such that for any disjoint subsets $A_1,...,A_n$ of $S$, the number of points in each subset $N(A_1),...,N(A_n)$ are independent random variables that are Poisson distributed: $N(A_k) \sim Poiss(\mu(A_k))$, where the rate is defined by a mean measure $\mu(A)$ often expressed in terms of the rate function $\lambda(x): S \rightarrow \mathbb{R}^{+}$:

\begin{equation}
    \mu(A) = \int_A \lambda(x) dx < \infty
\end{equation}

If the rate function $\lambda(x)$ is constant, we have a *homogeneous* Poisson process, where $\mu(A) = \lambda |A|$, and $|A|$ denotes the measure of $A$, e.g. area in $\mathbb{R}^{2}$. If the rate function is spatially varying, then we have an *inhomogeneous* Poisson process [2].

A Cox Process is a Poisson process in which the rate function is itself a stochastic process [1]. In particular, a Log Gaussian Cox Process (LGCP) models the rate function $\lambda(x)$ as a transformation of a Gaussian Process (GP) [3] that guarantees positivity. Common transformations include exponential, sigmoid and square functions. We can summarize the Cox Process as follows:

\begin{eqnarray}
    f(x) &\sim& GP\big(m(x), K(x,x^{\prime})\big) \\
    \lambda(x) &=& \exp\{f(x)\} \\
    y(A) &\sim& Poiss\bigg(\int_A \lambda(x) dx \bigg)
\end{eqnarray}

Let's implement the Cox Process in PyMC! We'll use the redwoods dataset which is provided as part of spatstat R package. It contains locations of California redwood trees in a square sampling region.

```python
%matplotlib inline
import numpy as np
import pandas as pd

import seaborn as sns
import matplotlib.pyplot as plt

import pymc3 as pm

import theano
import theano.tensor as tt

import warnings
warnings.filterwarnings('ignore')
```

```python
#load data
print "loading data..."
train_df = pd.read_csv('./data/redwoods.csv')
train_df.head()

```

    loading data...

      redwoodfull.x
      redwoodfull.y

      0
      0.931481
      0.817680

      1
      0.938889
      0.764273

      2
      0.935185
      0.721915

      3
      0.979630
      0.664825

      4
      0.787037
      0.661142

```python
plt.figure()
plt.scatter(train_df['redwoodfull.x'], train_df['redwoodfull.y'])
plt.title('California Redwoods')
plt.xlabel('X1')
plt.ylabel('X2')
plt.savefig('./figures/redwoods_dataset.png')
```

```python
#discretize spatial data
D = 2 #dimension
num_bins = 64
hist, xedges, yedges = np.histogram2d(train_df['redwoodfull.x'].values, train_df['redwoodfull.y'].values, bins=num_bins)
xcenters = xedges[:-1] + 0.5 * (xedges[1:] - xedges[:-1])
ycenters = yedges[:-1] + 0.5 * (yedges[1:] - yedges[:-1])
```

```python
plt.figure()
plt.imshow(hist, cmap='hot')
plt.savefig('./figures/redwoods_hist.png')
```

```python
#form input arrays
y_data = np.zeros(num_bins * num_bins)  #observations (counts)
x_data = np.zeros((num_bins * num_bins, 2)) #locations

cnt = 0
for i in range(num_bins):
    for j in range(num_bins):
        y_data[cnt] = hist[i][j]
        x_data[cnt,:] = np.hstack((xcenters[i], ycenters[j]))
        cnt += 1
    #end for
#end for
```

```python
print "creating the model..."
with pm.Model() as model:

    #hyper-parameter priors
    #l = pm.Gamma('l', alpha=2, beta=2)
    #eta = pm.HalfCauchy('eta', beta=3)
    #cov_func = eta**2 * pm.gp.cov.ExpQuad(D, ls=l*np.ones(D))

    #Gaussian Process
    cov_func = pm.gp.cov.ExpQuad(D, ls=0.1*np.ones(D))
    gp = pm.gp.Latent(cov_func=cov_func)
    f = gp.prior('f', X=x_data)

    obs = pm.Poisson('obs', mu=tt.exp(f), observed=y_data)
```

    creating the model...

```python
print "model inference..."
with model:
    trace = pm.sample(1000, burn=500, thin=2)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...

    model inference...

    100%|██████████| 1500/1500 [64:46:58<00:00, 155.48s/it]

```python
#generate plots
plt.figure()
pm.traceplot(trace, varnames=['f'])
plt.savefig('./figures/redwoods_trace.png')
```

```python
ftrace = np.mean(trace['f'][500::2,:], axis=0)
ftrace = np.reshape(ftrace, (num_bins, num_bins))
latent_rate = np.exp(ftrace)
```

```python
plt.figure()
plt.imshow(latent_rate)
plt.savefig('./figures/redwoods_latent_rate.png')

```

We can see that the heatmap of the latent rate function above matches the histogram plot, i.e. we can see higher concentration of trees along the main diagonal as well as patches above and below the diagonal. The latent rate function can give insight into land features that cause high tree growth.

### References

[1] D. Cox, "Some statistical methods connected with series of events", JRSS, 1955
[2] J. Kingman, "Poisson Processes", Oxford Studies in Probability, Clarendon Press, 1993
[3] C. Rasmussen and C. Williams, "Guassian Processes for Machine Learning", The MIT Press, 2006


# 1D Reaction-Diffusion problem

## General formulation

This is a well known problem to FEM with Continuous Galerkin approximation. Spurious oscillations rise when certain (not rare) conditions are met. The general problem consists in:

Find $u \in \mathbb{C}^2(\overline{\Omega})$ in a closed domain $\overline{\Omega}\subset \mathbb{R}^n\,(n=1,2,3)$  such that:

\begin{equation}
\left\{
\begin{aligned}
- &k \Delta u + \sigma u = f(x), \quad \forall x \in \Omega \\
&\left. u \right|_{\partial \Omega} = g(x)
\end{aligned}
\right.
\end{equation}
where $\overline{\Omega} := \Omega \cup \partial \Omega$ denotes the closed domain, $\Omega$ is the open domain and $\partial \Omega$ is the domain's boundary (the closure). Note that $\Omega \cap \partial \Omega = \emptyset$. The solution is a scalar field such that $u: \overline{\Omega} \to \mathbb{R}$. Also, we have the "source term" that is a scalar field of the same form, $f: \overline{\Omega} \to \mathbb{R}$, but this function is a given data to the (direct) problem. The $g(x)$ is a prescribed function applied to the boundary, which is also a given data to the problem. The parameter $k$ (here a scalar field like $u$) sometimes is named as "diffusivity coefficient" related to the quantity $u$, in a physical context (transport of a quantity, look for Reynolds' Theorem for further clarifications). In some contexts, parameter $k$ must be a 2-rank tensor, as in anisotropic Porous Media flows. Analagously, we have the parameter $\sigma: \overline{\Omega} \to \mathbb{R}$, which is denoted (in some contexts) as the "reaction coefficient" related to the proportional rate that $u$ is changing due to generation/consumption. Parameters $k$ and $\sigma$ are given data to the (direct) problem too.

P.S.: I will not discuss the ill problems formulation, neither others mathematical requirements to assert the well-posedness of the problem, uniqueness and so on.

## 1D simplification

In some scenarios, the mathematical modeling approach can be simplified to the 1D case, which is written as follows:

Find $u \in \mathbb{C}^2$ such that:

\begin{equation}
\left\{
\begin{aligned}
- &k u''(x) + \sigma u(x) = f(x), \quad \forall x \in ((x_1, x_2) \subset \mathbb{R}) \\
&u(x_1) = u_1 \\
&u(x_2) = u_2
\end{aligned}
\right.
\end{equation}

where $u_1, u_2 \in \mathbb{R}$ are the prescribed boundary condition values, $u'(x) \equiv \dfrac{d u}{d x}$ and thus $u''(x) \equiv \dfrac{d^2 u}{d x^2}$. For the sake of simplicity, I will omit the argument of $u$ function. Such aforementioned problems are known as Two-Point Boundary Value problems.

### Variational formulation and Galerkin approximation

One can find the derivation of the weak form, but I will just give its result below.

Given the space of admissible solution:

\begin{equation}
\mathcal{U} (\overline{\Omega}) := \left\{ u \in H^1 (\overline{\Omega}) \left| \,u(x_1) = u_1, u(x_2) = u_2 \right. \right\}
\end{equation}

and the space of suitable variations as

\begin{equation}
\mathcal{V} (\overline{\Omega}) := \left\{ v \in H^1 (\overline{\Omega}) \left| \,u(x_1) = u(x_2) = 0 \right. \right\}
\end{equation}

The discretization of the above spaces is performed by Galerkin approximation with Lagrangean function space $\mathbb{P}$ of order $k$ defined over $\overline{\Omega}$. Thus we define here:

\begin{equation}
            \mathcal{S}_h^k(\overline{\Omega}) := \left\{ \varphi_h \in \mathcal{C}(\Omega^e): \left.\varphi\right|_{\Omega^e}
            \in \mathbb{P}_k(\Omega^e), \forall \Omega^e \in \mathcal{T}_h \right\}
\end{equation}

the space of Lagrange polynomials subject to a domain partition $\mathcal{T}_h:=\cup \Omega^e \approx \overline{\Omega}$ and $\cap \Omega^e = \emptyset$ (non-overlapping elements). Thus

\begin{equation}
    \mathcal{U}_h := \mathcal{U} \cap \mathcal{S}_h^k \quad \text{and} \quad
    \mathcal{V}_h := \mathcal{V} \cap \mathcal{S}_h^k
\end{equation}

we have the discretized spaces.

Now, we can stated out our "discretized" Variational Formulation as:

Find $u_h \in \mathcal{U}_h$ such that

\begin{equation}
a(u_h, v_h) = F(v_h), \quad \forall v_h \in \mathcal{V}_h
\end{equation}

where

\begin{align}
a(u, v) &:= \int_{\Omega} u' v' dx
          + \int_{\Omega} \sigma u v dx\\
F(v) &:= \int_{\Omega} f v dx
\end{align}

and the domain is $\overline{\Omega} \equiv [x_1, x_2]$.

## A practical FEniCS example

Solve this in FEniCS is very straightforward compared to classical FEM codes. The FEniCS framework provides a high-level interface which eases the pain at most.

Just for learning purpose, let the given data be setted as:

\begin{align*}
&f(x) \equiv f = 1 & \\
&u_1 = u_2 = 0 & \\
&x_1 = 0, &x_2 = 1 \\
&k = 10^{-8}, &\sigma = 1
\end{align*}

With the source term as $f(x) = 1$, the exact solution can be easily obtained:

\begin{equation}
u(x) \approx
\left\{
\begin{aligned}
&u(x) = 1, &\text{if } x \in (x_1, x_2) \\
&u(x) = 0, &\text{if } x = x_1 \text{ or } x = x_2
\end{aligned}
\right.
\end{equation}

for small values of $k$, say $k = 10^{-8}$.

So, how to solve the problem with FEniCS? We will construct the procedures stepwisely.

* Importing all the libs we'll need.

```python
from fenics import *  # all the FEniCS namespace (not a recommended Python practice)
import matplotlib.pyplot as plt
from matplotlib import rc
import numpy as np
```

* Defining the domain and related mesh

```python
x_left = 0.0
x_right = 1.0
numel = 15
mesh = IntervalMesh(numel, x_left, x_right)  # IntervalMesh(num_of_elements, inf_interval, sup_interval)
mesh_ref = IntervalMesh(100, x_left, x_right)
```

* Setting the degree of the functions in the Continuous Galerkin method and the variation space. Additionaly, we define here a space to be employed in the projection of the reference analytical solution

```python
p = 1
V = FunctionSpace(mesh, "CG", p)  # "CG" stands for Continuous Galerkin, p is the degree
Vref = FunctionSpace(mesh_ref, "CG", p)
```

* Defining a Python function which marks the boundaries

```python
def left(x, on_boundary):
    return x < 0+DOLFIN_EPS
def right(x, on_boundary):
    return x > 1-DOLFIN_EPS
```

* Setting the prescribed boundary values

```python
u1, u2 = 0.0, 0.0
g_left = Constant(u1)
g_right = Constant(u2)
```

* Now we modify the spaces, as expected

```python
bc_left = DirichletBC(V, g_left, left)
bc_right = DirichletBC(V, g_right, right)
dirichlet_condition = [bc_left, bc_right]
```

* Here we define the source function over the domain

```python
f = Expression("1.0", domain=mesh, degree=p)
```

* Now comes the good part. We set up the Trial and Test functions from the admissible space

```python
u_h = TrialFunction(V)
v_h = TestFunction(V)
```

* The parameters of the problem

```python
k = Constant(1e-8)
sigma = Constant(1.0)
```

* Then we write the unsymmetric form, very much like it is written in mathematical form

```python
a = k*inner(grad(v_h), grad(u_h))*dx + sigma*inner(u_h, v_h)*dx
```

* Also we define the associated linear form

```python
L = f*v_h*dx
```

* Now we declare the solution variable

```python
u_sol = Function(V)
```

* Thus we set the discretized variational problem to be solved

```python
problem = LinearVariationalProblem(a, L, u_sol, dirichlet_condition)
```

* And we go further and solve it with no difficulties!

```python
solver = LinearVariationalSolver(problem)
solver.solve()
```

* But we will need to check if the solution is "good enough". So we compare with the available exact solution

```python
sol_exact = Expression(
    'x[0]<=0+tol || x[0]>=1 - tol ? 0 : 1',
    degree=p+1,
    tol=DOLFIN_EPS
)
u_e = interpolate(sol_exact, Vref)
u_e = interpolate(sol_exact, Vref)
```

* Now, lets plot our results!

```python
plot(u_sol, marker='x', label='Approx')
plot(u_e, label='Exact')
# Setting the font
plt.rc('text',usetex=True)
plt.rc('font', size=14)
# Plotting
plt.xlim(x_left, x_right)   # Limites do eixo x
plt.ylim(np.min(u_sol.vector().get_local()), 1.02*np.max(u_sol.vector().get_local()))  # Limites do eixo y
plt.grid(True, linestyle='--')  # Ativa o grid do grafico
plt.xlabel(r'$x$')  # Legenda do eixo x
plt.ylabel(r'$u(x)$')   # Legenda do eixo y
plt.legend(loc='best',borderpad=0.5)    # Ativa legenda no grafico e diz para se posicionar na melhor localizacao detectada
plt.show()  # Exibe o grafico em tela
```

## Something seems not good... (Batman?)

Terrible, it isn't? This is due the high Damköhler number of the problem $(Da >> 1)$. The Damköhler number is conceptually defined as

\begin{equation}
Da := \frac{\text{reaction rate of a physical quantity}}{\text{diffusion rate of a physical quantity}}
\end{equation}

In Finite Element context, we evaluate the Damköhler number in an element-wise manner, because this is how we (normally) stabilize the method and vanish the spurious oscillations. The local (within each element, valid for 1D and 2D... I need to confirm for 3D) Damköhler number is denote as

\begin{equation}
Da_K \equiv \frac{\sigma h_k^2}{6k}
\end{equation}

where $h_K$ is the mesh parameter, related to some characteristic lenght of the element. For further details about what I stated above and for the method I will summarize below, check the paper "The Galerkin Gradient Least-Squares Method" (Franca and Do Carmo, 1989). Highly recommend reading for who want to dive in FEM Stabilization methods.

So, where all this stuff will be applied?

## GGLS stabilization

As commented above, this method was proposed by "The Galerkin Gradient Least-Squares Method" (Franca and Do Carmo, 1989) aiming to solve singular diffusion problems in reaction dominated cases ($Da >> 1$). The principle is similar to GLS method, but control is gained in $H^1$-seminorm, while classical Galerkin (and GLS) controls over $L^2$-norm.

With the modification, the variational problem is rewritten as follows:

\begin{equation}
B(u,v) = F(v)
\end{equation}

and

\begin{align}
&B(u,v) := a(u,v) + R(u,v) \\
&F(v) := L(v) + S(v)
\end{align}

with

\begin{align}
&R(u,v) := \sum_{K}\tau_k \left(\nabla\left(\sigma u -k \Delta u\right),
\nabla\left(\sigma v -k \Delta v\right)\right)_K\\
&S(v) := \sum_{K}\tau_k \left(f, \nabla\left(\sigma v -k \Delta v\right) \right)_K
\end{align}

where

\begin{equation}
(u, v)_{K} := \int_{\Omega^K} u v dx
\end{equation}

denotes the $L^2$ inner product restricted to the element $K$.

Note the presence of a stabilizing parameter $\tau_k$. By this parameter, we control how much we want to add to an element local contribution aiming to stabilize in the local formulation. This parameter is computed locally as follows:

\begin{equation}
\tau_K := \frac{h_K^2}{6 \sigma} \widetilde{\xi}(Da_K)
\end{equation}

where

\begin{equation}
\xi(Da_K) :=
\left\{
\begin{aligned}
&1, &Da_K \geq 8, \\
&0.064 Da_K + 0.49, &1 \leq Da_K < 8 \\
&0, &Da_K < 1
\end{aligned}
\right.
\end{equation}

The above expression is an asymptotic approximation to $\widetilde{\xi}$. See the aforementioned paper for further details.

### 1D GGLS

To simplify to 1D, just remember that the gradient reduce to $\nabla (\bullet) \equiv \dfrac{d (\bullet)}{d x} = u'$. Thus,

\begin{align}
&R(u,v) := \sum_{K}\tau_k \left(\sigma u' -k u''', \sigma v' -k v''' \right)_K \\
&S(v) := \sum_{K}\tau_k \left(f, \sigma u' -k v''' \right)_K
\end{align}

## Revisiting the FEniCS implementation

Finally, we can stabilize with GGLS the previous Continuous Galerkin FEM approach.

* First, we need to define the stabilizing parameter and its related quantities

```python
h = CellDiameter(mesh)
Da_k = (sigma*pow(h,2))/(6.0*k)
eps = conditional(ge(Da_k,8),1,conditional(ge(Da_k,1),0.064*Da_k+0.49,0))
tau = (eps*(pow(h,2)))/(6.0*sigma)
```

* Now we add the stabilizing terms to the classical Galerkin formulation

```python
a += inner(grad(sigma*u_h - k*div(grad(u_h))), tau*grad(sigma*v_h - k*div(grad(v_h))))*dx
L += inner(grad(f), tau*grad(sigma*v_h - k*div(grad(v_h))))*dx
```

* Redefining the problem

```python
u_sol = Function(V)
problem = LinearVariationalProblem(a, L, u_sol, dirichlet_condition)
```

* Solving the stabilized problem

```python
solver = LinearVariationalSolver(problem)
solver.solve()
```

* Now we plot the results to check

```python
plot(u_sol, marker='x', label='Approx')
plot(u_e, label='Exact')
# Configuracoes de fontes no grafico
plt.rc('text',usetex=True)
plt.rc('font', size=14)
# Plotando
plt.xlim(x_left, x_right)   # Limites do eixo x
plt.ylim(np.min(u_sol.vector().get_local()), 1.05*np.max(u_e.vector().get_local()))  # Limites do eixo y
plt.grid(True, linestyle='--')  # Ativa o grid do grafico
plt.xlabel(r'$x$')  # Legenda do eixo x
plt.ylabel(r'$u(x)$')   # Legenda do eixo y
plt.legend(loc='best',borderpad=0.5)    # Ativa legenda no grafico e diz para se posicionar na melhor localizacao detectada
plt.show()  # Exibe o grafico em tela
```

Now things look right.

## Another stabilization: Galerkin Least Squares (GLS)

An alternative stabilization is the very well known method named Galerkin Least Squares, or GLS for short.
This methods was introduced by Hughes, Franca and Hulbert (see the paper "A new finite element formulation for computational fluid dynamics: VIII. The Galerkin/least-squares method for advective–diffusive equations" for further details). The formulation to reaction-diffusion problem was based on a remark provide in the Franca and Do Carmo GGLS paper.

It's very similar approach to GGLS method in the modification sense related to the classical Galerkin formulation. Now, we have the following terms to add:

\begin{align}
&R_{GLS}(u,v) := \sum_{K}\tau_k \left(\sigma u -k \Delta u, \sigma v -k \Delta v \right)_K \\
&S_{GLS}(v) := \sum_{K}\tau_k \left(f, \sigma v -k \Delta v \right)_K
\end{align}

which trivially is extended to 1D as

\begin{align}
&R_{GLS}(u,v) := \sum_{K}\tau_k \left(\sigma u -k u'', \sigma v -k v''\right)_K \\
&S_{GLS}(v) := \sum_{K}\tau_k \left(f, \sigma v -k v'' \right)_K
\end{align}

### 1D GLS implementation in FEniCS

For instance, the stabilization parameter and its parameters can be pretty much the same as in GLS, just for testing purpose. Minor changes are necessary, but we will solve within a different variational problem just to compare the methods.

* The classical Continuous Galerkin terms

```python
a_GLS = k*inner(grad(v_h), grad(u_h))*dx + sigma*inner(u_h, v_h)*dx
L_GLS = f*v_h*dx
```

* Adding the GLS terms

```python
a_GLS += tau*inner((sigma*u_h - k*div(grad(u_h))), sigma*v_h - k*div(grad(v_h)))*dx
L_GLS += tau*inner(f, sigma*v_h - k*div(grad(v_h)))*dx
```

* Now we solve the problem with GLS

```python
u_sol_gls = Function(V)
problem = LinearVariationalProblem(a_GLS, L_GLS, u_sol_gls, dirichlet_condition)
solver = LinearVariationalSolver(problem)
solver.solve()
```

* Then we plot:

```python
plot(u_sol, marker='.', label='GGLS')
plot(u_sol_gls, marker='x', label='GLS')
plot(u_e, label='Exact')
# Configuracoes de fontes no grafico
plt.rc('text',usetex=True)
plt.rc('font', size=14)
# Plotando
plt.xlim(x_left, x_right)   # Limites do eixo x
plt.ylim(np.min(u_sol.vector().get_local()), 1.02*np.max(u_sol_gls.vector().get_local()))  # Limites do eixo y
plt.grid(True, linestyle='--')  # Ativa o grid do grafico
plt.xlabel(r'$x$')  # Legenda do eixo x
plt.ylabel(r'$u(x)$')   # Legenda do eixo y
plt.legend(loc='best',borderpad=0.5)    # Ativa legenda no grafico e diz para se posicionar na melhor localizacao detectada
plt.show()  # Exibe o grafico em tela
```

The above result confirms what we expected: GLS can not control spurious oscillation with $Da>>1$ and $p = 1$ while GGLS achieve success controlling $H^1$-seminorm.


# The Discrete Fourier Transform

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Comunications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Summary of Properties, Theorems and Transforms

The [properties](properties.ipynb), [theorems](theorems.ipynb) and transforms of the discrete Fourier transform (DFT) as derived in the previous sections are compiled in the following. The corresponding tables serve as a reference for the application of the DFT in the theory of signals and systems. Please refer to the respective sections for details.

### Definition

The DFT of length $N$ and its inverse are defined as

\begin{align}
X[\mu] &= \sum_{k = 0}^{N-1} x[k] \, e^{- j \mu \frac{2 \pi}{N} k} \\
x[k] &= \sum_{k = 0}^{N-1} X[\mu] \, e^{j \mu \frac{2 \pi}{N} k}
\end{align}

for $k, \mu \in \mathbb{Z}$.

### Properties and Theorems

Under the assumption of causal signals of finite length $N$ or peridic signals of period $N$, the properties and theorems of the DTFT are given as

|  | $x[k] \qquad \qquad \qquad \qquad$ | $X[\mu] = \text{DFT}_N \{ x[k] \} \qquad \qquad \qquad$ |
|:---|:---:|:---:|:---|
| [Linearity](properties.ipynb#Linearity) | $A \, x_1[k] + B \, x_2[k]$ | $A \, X_1[\mu] + B \, X_2[\mu]$ |
| [Real-valued signal](properties.ipynb#Real-valued-signals) | $x^*[k]$ | $X^*[N-\mu]$ |
| [Convolution](theorems.ipynb#Convolution-Theorem) | $x[k] \circledast_N h[k]$ | $X[\mu] \cdot H[\mu]$ |
| [Periodic shift](theorems.ipynb#Shift-Theorem) | $x[k - \kappa]$ | $e^{-j \mu \frac{2 \pi}{N} \kappa} \cdot X[\mu]$ |
| [Multiplication](theorems.ipynb#Multiplication-Theorem) | $x[k] \cdot h[k]$ | $\frac{1}{N} X[\mu] \circledast_{N} H[\mu]$ |
| [Modulation](theorems.ipynb#Modulation-Theorem) | $e^{j M \frac{2 \pi}{N} k} \cdot x[k]$ | $X[\mu - M]$ |
| [Parseval's Theorem](theorems.ipynb#Parseval's-Theorem) | $\sum_{k = 0}^{N-1} \lvert x[k] \rvert^2$ | $\frac{1}{N} \sum_{\mu = 0}^{N-1} \lvert X[\mu] \rvert^2$ |

where $A, B \in \mathbb{C}$ and $\kappa, M \in \mathbb{Z}$.

### Selected Transforms

DFTs which are frequently used are given as

| $x[k] \qquad \qquad \qquad \qquad$ | $X[\mu] = \text{DFT}_N \{ x[k] \} \qquad \qquad \qquad \qquad \qquad \qquad$ |
|:---:|:---:|:---|
| $\delta[k]$ | $1$ |
| $1$ | $N \cdot \delta[\mu]$ |
| $e^{j \Omega_0 k} $ | $e^{-j (\Omega_0 - \mu \frac{2 \pi}{N} ) \frac{N-1}{2} } \cdot \frac{\sin \left( \frac{N (\Omega_0 - \mu \frac{2\pi}{N})}{2} \right)}{\sin \left( \frac{ \Omega_0 - \mu \frac{2\pi}{N}}{2} \right)}$ |
| $\text{rect}_M[k]$ | $e^{-j \pi \mu \frac{M-1}{N} } \cdot \frac{\sin \left( \frac{M \pi \mu}{N} \right)}{\sin \left( \frac{ \pi \mu}{N} \right)}$ |

where $0 \leq \Omega_0 < 2 \pi$ and $0 < M \leq N$. Additional transforms may be found in the literature or [online](https://en.wikipedia.org/wiki/Discrete_Fourier_transform#Some_discrete_Fourier_transform_pairs).

**Copyright**

The notebooks are provided as [Open Educational Resource](https://de.wikipedia.org/wiki/Open_Educational_Resources). Feel free to use the notebooks for your own educational purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Lecture Notes on Signals and Systems* by Sascha Spors.


Working with Gaussians
=======

### Preliminaries

- Goal
  - Review of information processing with Gaussian distributions in linear systems
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp. 85-93
    - [MacKay - 2006 - The Humble Gaussian Distribution](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Mackay-2006-The-humble-Gaussian-distribution.pdf) (highly recommended!)
    - [Ariel Caticha - 2012 - Entropic Inference and the Foundations of Physics](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Caticha-2012-Entropic-Inference-and-the-Foundations-of-Physics.pdf), pp.30-34, section 2.8, the Gaussian distribution

### Example Problem

Consider a set of observations $D=\{x_1,…,x_N\}$ in the 2-dimensional plane (see Figure). All observations were generated by the same process. We now draw an extra observation $x_\bullet = (a,b)$ from the same data generating process. What is the probability that $x_\bullet$ lies within the shaded rectangle $S$?

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate();
IJulia.clear_output();
```

```julia
using Distributions, PyPlot
N = 100
generative_dist = MvNormal([0,1.], [0.8 0.5; 0.5 1.0])
function plotObservations(obs::Matrix)
    plot(obs[1,:], obs[2,:], "kx", zorder=3)
    fill_between([0., 2.], 1., 2., color="k", alpha=0.4, zorder=2) # Shaded area
    text(2.05, 1.8, "S", fontsize=12)
    xlim([-3,3]); ylim([-2,4]); xlabel("a"); ylabel("b")
end
D = rand(generative_dist, N) # Generate observations from generative_dist
plotObservations(D)
x_dot = rand(generative_dist) # Generate x∙
plot(x_dot[1], x_dot[2], "ro");
```

### The Gaussian Distribution

- Consider a random (vector) variable $x \in \mathbb{R}^M$ that is "normally" (i.e., Gaussian) distributed. The _moment_ parameterization of the Gaussian distribution is completely specified by its _mean_ $\mu$ and _variance_ $\Sigma$ and given by
$$
p(x | \mu, \Sigma) = \mathcal{N}(x|\mu,\Sigma) \triangleq |2 \pi \Sigma |^{-\frac{1}{2}} \,\exp\left\{-\frac{1}{2}(x-\mu)^T \Sigma^{-1} (x-\mu) \right\}\,.
$$

- Alternatively, the _canonical_ (a.k.a. _natural_  or _information_ ) parameterization of the Gaussian distribution is given by
$$\begin{equation*}
p(x | \eta, \Lambda) = \mathcal{N}_c(x|\eta,\Lambda)  = \exp\left\{ a + \eta^T x - \frac{1}{2}x^T \Lambda x \right\}\,.
\end{equation*}$$
  - $a = -\frac{1}{2} \left( M \log(2 \pi) - \log |\Lambda| + \eta^T \Lambda \eta\right)$ is the normalizing constant that ensures that $\int p(x)\mathrm{d}x = 1$.
  - $\Lambda = \Sigma^{-1}$ is called the *precision matrix*.
  - $\eta = \Sigma^{-1} \mu$ is the _natural_ mean or for clarity often called the *precision-weighted mean*.

### Why the Gaussian?

- Why is the Gaussian distribution so ubiquitously used in science and engineering? (see [Jaynes, pg.220](https://archive.org/details/ProbabilityTheoryTheLogicOfScience/page/n249)).

- (1) Operations on probability distributions tend lead to Gaussian distributions:
  - The [Gaussian distribution has higher entropy](https://en.wikipedia.org/wiki/Differential_entropy#Maximization_in_the_normal_distribution) than any other with the same variance.
    - Therefore any operation on a probability distribution that discards information but preserves variance gets us closer to a Gaussian.
  - Any smooth function with single rounded maximum, if raised to higher and higher powers, goes into a Gaussian function. (Example: sequential Bayesian inference).

- (2) Once the Gaussian has been attained, this form tends to be preserved. e.g.,
  - The convolution of two Gaussian functions is another Gaussian function (useful in sum of 2 variables and linear transformations)
  - The product of two Gaussian functions is another Gaussian function (useful in Bayes rule).
  - The Fourier transform of a Gaussian function is another Gaussian function.

###  Transformations and Sums of Gaussian Variables

- A **linear transformation** $z=Ax+b$ of a Gaussian variable $\mathcal{N}(x|\mu,\Sigma)$ is Gaussian distributed as

$$
p(z) = \mathcal{N} \left(z \,|\, A\mu+b, A\Sigma A^T \right) \tag{SRG-4a}
$$

- The **sum of two independent Gaussian variables** is also Gaussian distributed. Specifically, if $x \sim \mathcal{N} \left(x|\mu_x, \Sigma_x \right)$ and $y \sim \mathcal{N} \left(y|\mu_y, \Sigma_y \right)$, then the PDF for $z=x+y$ is given by
$$\begin{align}
p(z) &= \mathcal{N}(x\,|\,\mu_x,\Sigma_x) \ast \mathcal{N}(y\,|\,\mu_y,\Sigma_y) \notag\\
  &= \mathcal{N} \left(z\,|\,\mu_x+\mu_y, \Sigma_x +\Sigma_y \right) \tag{SRG-8}
\end{align}$$

  - The sum of two Gaussian _distributions_ is NOT a Gaussian distribution. Why not?

### Example: Gaussian Signals in a Linear System

- Given independent variables
$x \sim \mathcal{N}(\mu_x,\sigma_y^2)$ and $y \sim \mathcal{N}(\mu_y,\sigma_y^2)$, what is the PDF for $z = A\cdot(x -y) + b$ ? (see [Exercises](https://nbviewer.jupyter.org/github/bertdv/BMLIP/blob/master/lessons/exercises/Exercises.ipynb))

- Think about the role of the Gaussian distribution for stochastic linear systems in relation to what sinusoidals mean for deterministic linear system analysis.

### Bayesian Inference for the Gaussian

- Let's estimate a constant $\theta$ from one 'noisy' measurement $x$ about that constant.

- We assume the following measurement equations (the tilde $\sim$ means: 'is distributed as'):
$$\begin{align*}
x &= \theta + \epsilon \\
\epsilon &\sim \mathcal{N}(0,\sigma^2)
\end{align*}$$

- Also, let's assume a Gaussian prior for $\theta$
$$\begin{align*}
\theta &\sim \mathcal{N}(\mu_0,\sigma_0^2) \\
\end{align*}$$

##### Model specification

- Note that you can rewrite these specifications in probabilistic notation as follows:

$$\begin{align}
    p(x|\theta) &=  \mathcal{N}(x|\theta,\sigma^2) \tag{likelihood}\\
    p(\theta) &=\mathcal{N}(\theta|\mu_0,\sigma_0^2) \tag{prior}
\end{align}$$

- (**Notational convention**). Note that we write $\epsilon \sim \mathcal{N}(0,\sigma^2)$ but not $\epsilon \sim \mathcal{N}(\epsilon | 0,\sigma^2)$, and we write  $p(\theta) =\mathcal{N}(\theta|\mu_0,\sigma_0^2)$ but not $p(\theta) =\mathcal{N}(\mu_0,\sigma_0^2)$.

##### Inference

- For simplicity, we assume that the variance $\sigma^2$ is given and will proceed to derive a Bayesian posterior for the mean $\theta$. The case for Bayesian inference for $\sigma^2$ with a given mean is [discussed later](#inference-for-precision).

- Let's do Bayes rule for the posterior PDF $p(\theta|x)$.
$$\begin{align*}
p(\theta|x)  &= \frac{p(x|\theta) p(\theta)}{p(x)} \propto p(x|\theta) p(\theta) \notag \\
    &= \mathcal{N}(x|\theta,\sigma^2) \mathcal{N}(\theta|\mu_0,\sigma_0^2)   \notag \\
    &\propto \exp \left\{   -\frac{(x-\theta)^2}{2\sigma^2} - \frac{(\theta-\mu_0)^2}{2\sigma_0^2} \right\} \notag \\
    &\propto \exp \left\{ \theta^2 \cdot \left( -\frac{1}{2 \sigma_0^2} - \frac{1}{2\sigma^2}  \right)  + \theta \cdot  \left( \frac{\mu_0}{\sigma_0^2} + \frac{x}{\sigma^2}\right)   \right\} \notag \\
    &= \exp\left\{ -\frac{\sigma_0^2 + \sigma^2}{2 \sigma_0^2 \sigma^2} \left( \theta - \frac{\sigma_0^2 x +  \sigma^2 \mu_0}{\sigma^2 + \sigma_0^2}\right)^2  \right\}
\end{align*}$$
which we recognize as a Gaussian distribution w.r.t. $\theta$.

- (Just as an aside,) this computational 'trick' for multiplying two Gaussians is called **completing the square**. The procedure makes use of the equality $$ax^2+bx+c_1 = a\left(x+\frac{b}{2a}\right)^2+c_2$$

- In particular, it follows that the posterior for $\theta$ is
$$\begin{equation*}
    p(\theta|x) = \mathcal{N} (\theta |\, \mu_1, \sigma_1^2)
\end{equation*}$$
where
$$\begin{align*}
  \frac{1}{\sigma_1^2}  &= \frac{\sigma_0^2 + \sigma^2}{\sigma^2 \sigma_0^2} = \frac{1}{\sigma_0^2} + \frac{1}{\sigma^2}  \\
  \mu_1   &= \frac{\sigma_0^2 x +  \sigma^2 \mu_0}{\sigma^2 + \sigma_0^2} = \sigma_1^2 \, \left(  \frac{1}{\sigma_0^2} \mu_0 + \frac{1}{\sigma^2} x \right)
\end{align*}$$

### (Multivariate) Gaussian Multiplication

- So, multiplication of two Gaussian distributions yields another (unnormalized) Gaussian with
  - posterior precision equals **sum of prior precisions**
  - posterior precision-weighted mean equals **sum of prior precision-weighted means**

- As we just saw, great application to Bayesian inference!

$$\begin{equation*}
\underbrace{\text{Gaussian}}_{\text{posterior}}
 \propto \underbrace{\text{Gaussian}}_{\text{likelihood}} \times \underbrace{\text{Gaussian}}_{\text{prior}}
\end{equation*}$$

- In general, the multiplication of two multi-variate Gaussians yields an (unnormalized) Gaussian:
$$\begin{equation*}
\boxed{\mathcal{N}(x|\mu_a,\Sigma_a) \cdot \mathcal{N}(x|\mu_b,\Sigma_b) = \underbrace{\mathcal{N}(\mu_a|\, \mu_b, \Sigma_a + \Sigma_b)}_{\text{normalization constant}} \cdot \mathcal{N}(x|\mu_c,\Sigma_c)} \tag{SRG-6}
\end{equation*}$$
where
$$\begin{align*}
\Sigma_c^{-1} &= \Sigma_a^{-1} + \Sigma_b^{-1} \\
\Sigma_c^{-1} \mu_c &= \Sigma_a^{-1}\mu_a + \Sigma_b^{-1}\mu_b
\end{align*}$$

- $\Rightarrow$ Note that Bayesian inference is trivial in the _canonical_ parameterization of the Gaussian, where we would get
$$\begin{align*}
 \Lambda_c &= \Lambda_a + \Lambda_b  \quad &&\text{(precisions add)}\\
 \eta_c &= \eta_a + \eta_b \quad &&\text{(precision-weighted means add)}
\end{align*}$$

### CODE EXAMPLE

Let's plot the exact product of two Gaussian PDFs as well as the normalized product according to the above derivation.

```julia
using PyPlot, Distributions
d1 = Normal(0, 1) # μ=0, σ^2=1
d2 = Normal(3, 2) # μ=3, σ^2=4

# Calculate the parameters of the product d1*d2
s2_prod = (d1.σ^-2 + d2.σ^-2)^-1
m_prod = s2_prod * ((d1.σ^-2)*d1.μ + (d2.σ^-2)*d2.μ)
d_prod = Normal(m_prod, sqrt(s2_prod)) # Note that we neglect the normalization constant.

# Plot stuff
x = range(-4, stop=8, length=100)
plot(x, pdf.(d1,x), "k")
plot(x, pdf.(d2,x), "b")
plot(x, pdf.(d1,x) .* pdf.(d2,x), "r-") # Plot the exact product
plot(x, pdf.(d_prod,x), "r:")          # Plot the normalized Gaussian product
legend([L"\mathcal{N}(0,1)",
        L"\mathcal{N}(3,4)",
        L"\mathcal{N}(0,1) \mathcal{N}(3,4)",
        L"Z^{-1} \mathcal{N}(0,1) \mathcal{N}(3,4)"]);
```

The solid and dotted red curves are identical up to a scaling factor $Z$.

### Bayesian Inference with multiple Observations

- Now consider that we measure a data set $D = \{x_1, x_2, \ldots, x_N\}$, with measurements
  $$\begin{align*}
x_n &= \theta + \epsilon_n \\
\epsilon_n &\sim \mathcal{N}(0,\sigma^2)
\end{align*}$$
and the same prior for $\theta$:
$$\begin{align*}
\theta &\sim \mathcal{N}(\mu_0,\sigma_0^2) \\
\end{align*}$$

- Let's derive a distribution for the next sample $x_{N+1}$.

##### inference

- Clearly, the posterior for $\theta$ is now
$$\begin{align*}
p(\theta|D) \propto  \underbrace{\mathcal{N}(\theta|\mu_0,\sigma_0^2)}_{\text{prior}} \cdot \underbrace{\prod_{n=1}^N \mathcal{N}(x_n|\theta,\sigma^2)}_{\text{likelihood}}
\end{align*}$$
which is a multiplication of $N+1$ Gaussians and is therefore also Gaussian distributed.

- Using the property that precisions and precision-weighted means add when Gaussians are multiplied, we can immediately write the posterior $$p(\theta|D) = \mathcal{N} (\theta |\, \mu_N, \sigma_N^2)$$ as
$$\begin{align*}
  \frac{1}{\sigma_N^2}  &= \frac{1}{\sigma_0^2} + \sum_n  \frac{1}{\sigma^2} \tag{B-2.142}\\
  \mu_N   &= \sigma_N^2 \, \left( \frac{1}{\sigma_0^2} \mu_0 + \sum_n \frac{1}{\sigma^2} x_n  \right) \tag{B-2.141}
\end{align*}$$

##### application: prediction of future sample

- We now have a posterior for the model parameters. Let's write down what we know about the next sample $x_{N+1}$.

$$\begin{align*}
p(x_{N+1}|D_N) &= \int p(x_{N+1}|\theta) p(\theta|D_{N})\mathrm{d}\theta \\
  &= \int \mathcal{N}(x_{N+1}|\theta,\sigma^2) \mathcal{N}(\theta|\mu_N,\sigma^2_N) \mathrm{d}\theta \\
  &= \mathcal{N}(x_{N+1}|\mu_N, \sigma^2_N +\sigma^2 ) \tag{use SRG-6}
\end{align*}$$

- Uncertainty about $x_{N+1}$ comprises both uncertainty about the parameter ($\sigma_N^2$) and observation noise $\sigma^2$.

### Maximum Likelihood Estimation for the Gaussian

- In order to determine the _maximum likelihood_ estimate of $\theta$, we let $\sigma_0^2 \rightarrow \infty$ (leads to uniform prior for $\theta$), yielding $ \frac{1}{\sigma_N^2} = \frac{N}{\sigma^2}$ and consequently
$$\begin{align*}
  \mu_{\text{ML}}  = \left.\mu_N\right\vert_{\sigma_0^2 \rightarrow \infty} = \sigma_N^2 \, \left(   \frac{1}{\sigma^2}\sum_n  x_n  \right) = \frac{1}{N} \sum_{n=1}^N x_n
  \end{align*}$$

- As expected, having an expression for the maximum likelihood estimate, it is now possible to rewrite the (Bayesian) posterior mean for $\theta$ as (exercise)
$$\begin{align*}
  \underbrace{\mu_N}_{\text{posterior}}   &= \frac{\sigma^2}{N} \, \left( \frac{1}{\sigma_0^2} \mu_0 + \sum_n \frac{1}{\sigma^2} x_n  \right) \\
  &= \underbrace{\mu_0}_{\text{prior}} + \underbrace{\underbrace{\frac{N \sigma_0^2}{N \sigma_0^2 + \sigma^2}}_{\text{gain}}\cdot \underbrace{\left(\mu_{\text{ML}} - \mu_0 \right)}_{\text{prediction error}}}_{\text{correction}}\tag{B-2.141}
\end{align*}$$

- Hence, the posterior mean always lies somewhere between the prior mean $\mu_0$ and the maximum likelihood estimate (the "data" mean) $\mu_{\text{ML}}$.

### Conditioning and Marginalization of a Gaussian

- Let $z = \begin{bmatrix} x \\ y \end{bmatrix}$ be jointly normal distributed as

$$\begin{align*}
p(z) &= \mathcal{N}(z | \mu, \Sigma)
  =\mathcal{N} \left( \begin{bmatrix} x \\ y \end{bmatrix} \left| \begin{bmatrix} \mu_x \\ \mu_y \end{bmatrix},
  \begin{bmatrix} \Sigma_x & \Sigma_{xy} \\ \Sigma_{yx} & \Sigma_y \end{bmatrix} \right. \right)
\end{align*}$$

- Since covariance matrices are by definition symmetric, it follows that $\Sigma_x$ and $\Sigma_y$ are symmetric and $\Sigma_{xy} = \Sigma_{yx}^T$.

- Let's factorize $p(x,y)$ into $p(y|x)\cdot p(x)$ through conditioning and marginalization.

##### Conditioning

$$\begin{align*}
p(y|x) &\triangleq \frac{p(x,y)}{p(x)} \\
 &= \mathcal{N}\left(y\,|\,\mu_y + \Sigma_{yx}\Sigma_x^{-1}(x-\mu_x),\, \Sigma_y - \Sigma_{yx}\Sigma_x^{-1}\Sigma_{xy} \right)\\
\end{align*}$$

##### Marginalization

$$\begin{align*}
p(x) &\triangleq \int p(x,y)\,\mathrm{d}y \\
 &= \mathcal{N}\left( x|\mu_x, \Sigma_x \right)
\end{align*}$$

- **proof**: in Bishop pp.87-89

- Useful for applications to Bayesian inference in jointly Gaussian systems.

#### CODE EXAMPLE

Plot of the joint, marginal, and conditional distributions.

```julia
using PyPlot, Distributions

μ = [1.0; 2.0]
Σ = [0.3 0.7;
     0.7 2.0]
joint = MvNormal(μ,Σ)
marginal_x = Normal(μ[1], sqrt(Σ[1,1]))

#Plot p(x,y)
subplot(221)
x_range = y_range = range(-2,stop=5,length=1000)
joint_pdf = [ pdf(joint, [x_range[i];y_range[j]]) for  j=1:length(y_range), i=1:length(x_range)]
imshow(joint_pdf, origin="lower", extent=[x_range[1], x_range[end], y_range[1], y_range[end]])
grid(); xlabel("x"); ylabel("y"); title("p(x,y)"); tight_layout()

# Plot p(x)
subplot(222)
plot(range(-2,stop=5,length=1000), pdf.(marginal_x, range(-2,stop=5,length=1000)))
grid(); xlabel("x"); ylabel("p(x)"); title("Marginal distribution p(x)"); tight_layout()

# Plot p(y|x)
x = 0.1
conditional_y_m = μ[2]+Σ[2,1]*inv(Σ[1,1])*(x-μ[1])
conditional_y_s2 = Σ[2,2] - Σ[2,1]*inv(Σ[1,1])*Σ[1,2]
conditional_y = Normal(conditional_y_m, sqrt.(conditional_y_s2))
subplot(223)
plot(range(-2,stop=5,length=1000), pdf.(conditional_y, range(-2,stop=5,length=1000)))
grid(); xlabel("y"); ylabel("p(y|x)"); title("Conditional distribution p(y|x)"); tight_layout()
```

As is clear from the plots, the conditional distribution is a renormalized slice from the joint distribution.

### Example: Conditioning of Gaussian

- Consider (again) the system
$$\begin{align*}
p(x\,|\,\theta) &= \mathcal{N}(x\,|\,\theta,\sigma^2)
\end{align*}$$
with a Gaussian prior for $\theta$:
$$
p(\theta) = \mathcal{N}(\theta\,|\,\mu_0,\sigma_0^2)
$$

- This system is equivalent to (Exercise)

$$
p(x,\theta) = \mathcal{N} \left( \begin{bmatrix} x\\
  \theta  \end{bmatrix}
  \,\left|\, \begin{bmatrix} \mu_0\\
  \mu_0\end{bmatrix},
         \begin{bmatrix} \sigma_0^2+\sigma^2  & \sigma_0^2\\
         \sigma_0^2 &\sigma_0^2
  \end{bmatrix}
  \right. \right)
$$

- Direct substitution of the rule for Gaussian conditioning leads to the posterior (derivation as an Exercise):
$$\begin{align*}
p(\theta|x) &= \mathcal{N} \left( \theta\,|\,\mu_1, \sigma_1^2 \right)\,,
\end{align*}$$
with
$$\begin{align*}
K &= \frac{\sigma_0^2}{\sigma_0^2+\sigma^2} \qquad \text{($K$ is called: Kalman gain)}\\
\mu_1 &= \mu_0 + K \cdot (x-\mu_0)\\
\sigma_1^2 &= \left( 1-K \right) \sigma_0^2
\end{align*}$$

- $\Rightarrow$ Moral: For jointly Gaussian systems, we can do inference simply in one step by using the formulas for conditioning and marginalization.

### Recursive Bayesian Estimation

- Consider the signal $x_t=\theta+\epsilon_t$, where $D_t= \left\{x_1,\ldots,x_t\right\}$ is observed _sequentially_ (over time).

- **Problem**: Derive a recursive algorithm for $p(\theta|D_t)$, i.e., an update rule for (posterior) $p(\theta|D_t)$ based on (prior) $p(\theta|D_{t-1})$ and (new observation) $x_t$.

##### Model specification
- Let's define the estimate after $t$ observations (i.e., our _solution_ ) as $p(\theta|D_t) = \mathcal{N}(\theta\,|\,\mu_t,\sigma_t^2)$.

- We define the joint distribution for $\theta$ and $x_t$, given background $D_{t-1}$, by

$$\begin{align*} p(x_t,\theta \,|\, D_{t-1}) &= p(x_t|\theta) \, p(\theta|D_{t-1}) \\
  &= \underbrace{\mathcal{N}(x_t\,|\, \theta,\sigma^2)}_{\text{likelihood}} \, \underbrace{\mathcal{N}(\theta\,|\,\mu_{t-1},\sigma_{t-1}^2)}_{\text{prior}}
\end{align*}$$

##### Inference

- Use Bayes rule,
$$\begin{align*}
p(\theta|D_t) &= p(\theta|x_t,D_{t-1}) \\
  &\propto p(x_t,\theta | D_{t-1}) \\
  &= p(x_t|\theta) \, p(\theta|D_{t-1}) \\
  &= \mathcal{N}(x_t|\theta,\sigma^2) \, \mathcal{N}(\theta\,|\,\mu_{t-1},\sigma_{t-1}^2) \\
  &= \mathcal{N}(\theta|x_t,\sigma^2) \, \mathcal{N}(\theta\,|\,\mu_{t-1},\sigma_{t-1}^2) \;\;\text{(note this trick)}\\
  &= \mathcal{N}(\theta|\mu_t,\sigma_t^2) \;\;\text{(use Gaussian multiplication formula SRG-6)}
\end{align*}$$
with
$$\begin{align*}
K_t &= \frac{\sigma_{t-1}^2}{\sigma_{t-1}^2+\sigma^2} \qquad \text{(Kalman gain)}\\
\mu_t &= \mu_{t-1} + K_t \cdot (x_t-\mu_{t-1})\\
\sigma_t^2 &= \left( 1-K_t \right) \sigma_{t-1}^2
\end{align*}$$

- This linear _sequential_ estimator of mean and variance in Gaussian observations is called a **Kalman Filter**.

- Note that the uncertainty about $\theta$ decreases over time (since $0<(1-K_t)<1$). Since we assume that the statistics of the system do not change (stationarity), each new sample provides new information.

- Recursive Bayesian estimation is the basis for **adaptive signal processing** algorithms such as Least Mean Squares (LMS) and Recursive Least Squares (RLS).

#### CODE EXAMPLE

Let's implement the Kalman filter described above. We'll use it to recursively estimate the value of $\theta$ based on noisy observations.

```julia
using PyPlot

N = 50                         # Number of observations
θ = 2.0                        # True value of the variable we want to estimate
σ_ϵ2 = 0.25                    # Observation noise variance
x = sqrt(σ_ϵ2) * randn(N) .+ θ # Generate N noisy observations of θ

t = 0
μ = fill!(Vector{Float64}(undef,N), NaN)    # Means of p(θ|D) over time
σ_μ2 = fill!(Vector{Float64}(undef,N), NaN) # Variances of p(θ|D) over time

function performKalmanStep()
    # Perform a Kalman filter step, update t, μ, σ_μ2
    global t += 1
    if t>1 # Use posterior from prev. step as prior
        K = σ_μ2[t-1] / (σ_ϵ2 + σ_μ2[t-1]) # Kalman gain
        μ[t] = μ[t-1] + K*(x[t] - μ[t-1])  # Update mean using (1)
        σ_μ2[t] = σ_μ2[t-1] * (1.0-K)      # Update variance using (2)
    elseif t==1 # Use prior
        # Prior p(θ) = N(0,1000)
        K = 1000.0 / (σ_ϵ2 + 1000.0) # Kalman gain
        μ[t] = 0 + K*(x[t] - 0)      # Update mean using (1)
        σ_μ2[t] = 1000 * (1.0-K)     # Update variance using (2)
    end
end

while t<N
    performKalmanStep()
end

# Plot the 'true' value of θ, noisy observations x, and the recursively updated posterior p(θ|D)
t = collect(1:N)
plot(t, θ*ones(N), "k--")
plot(t, x, "rx")
plot(t, μ, "b-")
fill_between(t, μ-sqrt.(σ_μ2), μ+sqrt.(σ_μ2), color="b", alpha=0.3)
legend([L"\theta", L"x[t]", L"\mu[t]"])
xlim((1, N)); xlabel(L"t"); grid()
```

The shaded area represents 2 standard deviations of posterior $p(\theta|D)$. The variance of the posterior is guaranteed to decrease monotonically for the standard Kalman filter.

###  Product of Normally Distributed Variables
- (We've seen that) the sum of two Gausssian distributed variables is also Gaussian distributed.

- Has the _product_ of two Gaussian distributed variables also a Gaussian distribution?

- **No**! In general this is a difficult computation. As an example, let's compute $p(z)$ for $Z=XY$ for the special case that $X\sim \mathcal{N}(0,1)$ and $Y\sim \mathcal{N}(0,1)$.
$$\begin{align*}
p(z) &= \int_{X,Y} p(z|x,y)\,p(x,y)\,\mathrm{d}x\mathrm{d}y \\
  &= \frac{1}{2 \pi}\int  \delta(z-xy) \, e^{-(x^2+y^2)/2} \, \mathrm{d}x\mathrm{d}y \\
  &=  \frac{1}{\pi} \int_0^\infty \frac{1}{x} e^{-(x^2+z^2/x^2)/2} \, \mathrm{d}x \\
  &= \frac{1}{\pi} \mathrm{K}_0( \lvert z\rvert )\,.
\end{align*}$$
where  $\mathrm{K}_n(z)$ is a [modified Bessel function of the second kind](http://mathworld.wolfram.com/ModifiedBesselFunctionoftheSecondKind.html).

#### CODE EXAMPLE

We plot $p(Z)$ to give an idea of what this distribution looks like.

```julia
using PyPlot, Distributions, SpecialFunctions
X = Normal(0,1)
pdf_product_std_normals(z::Vector) = (besselk.(0, abs.(z))./π)
range1 = collect(range(-4,stop=4,length=100))
plot(range1, pdf.(X, range1))
plot(range1, pdf_product_std_normals(range1))
legend([L"p(X)=p(Y)=\mathcal{N}(0,1)", L"p(Z)"]); grid()
```

### Solution to Example Problem

We apply maximum likelihood estimation to fit a 2-dimensional Gaussian model ($m$) to data set $D$. Next, we evaluate $p(x_\bullet \in S | m)$ by (numerical) integration of the Gaussian pdf over $S$: $p(x_\bullet \in S | m) = \int_S p(x|m) \mathrm{d}x$.

```julia
using HCubature, LinearAlgebra# Numerical integration package
# Maximum likelihood estimation of 2D Gaussian
N = length(sum(D,dims=1))
μ = 1/N * sum(D,dims=2)[:,1]
D_min_μ = D - repeat(μ, 1, N)
Σ = Hermitian(1/N * D_min_μ*D_min_μ')
m = MvNormal(μ, convert(Matrix, Σ));

# # Contour plot of estimated Gaussian density
A = Matrix{Float64}(undef,100,100); B = Matrix{Float64}(undef,100,100)
density = Matrix{Float64}(undef,100,100)
for i=1:100
    for j=1:100
        A[i,j] = a = (i-1)*6/100 .- 2
        B[i,j] = b = (j-1)*6/100 .- 3
        density[i,j] = pdf(m, [a,b])
    end
end
c = contour(A, B, density, 6, zorder=1)
PyPlot.set_cmap("cool")
clabel(c, inline=1, fontsize=10)

# Plot observations, x∙, and the countours of the estimated Gausian density
plotObservations(D)
plot(x_dot[1], x_dot[2], "ro")

# Numerical integration of p(x|m) over S:
(val,err) = hcubature((x)->pdf(m,x), [0., 1.], [2., 2.])
println("p(x⋅∈S|m) ≈ $(val)")
```

##  OPTIONAL SLIDES

### Inference for the Precision Parameter of the Gaussian

- Again, we consider an observed data set $D = \{x_1, x_2, \ldots, x_N\}$ and try to explain these data by a Gaussian distribution.

- We discussed earlier Bayesian inference for the mean with a given variance. Now we will derive a posterior for the variance if the mean is given. (Technically, we will do the derivation for a precision parameter $\lambda = \sigma^{-2}$, since the discussion is a bit more straightforward for the precision parameter).

##### model specification
- The likelihood for the precision parameter is
$$\begin{align*}
p(D|\lambda) &= \prod_{n=1}^N \mathcal{N}\left(x_n \,|\, \mu, \lambda^{-1} \right) \\
  &\propto \lambda^{N/2} \exp\left\{ -\frac{\lambda}{2}\sum_{n=1}^N \left(x_n - \mu \right)^2\right\} \tag{B-2.145}
\end{align*}$$

- The conjugate distribution for this function of $\lambda$ is the [_Gamma_ distribution](https://en.wikipedia.org/wiki/Gamma_distribution), given by
$$
p(\lambda\,|\,a,b) = \mathrm{Gam}\left( \lambda\,|\,a,b \right) \triangleq \frac{1}{\Gamma(a)} b^{a} \lambda^{a-1} \exp\left\{ -b \lambda\right\}\,, \tag{B-2.146}
$$
where $a>0$ and $b>0$ are known as the _shape_ and _rate_ parameters, respectively.

- (Bishop fig.2.13). Plots of the Gamma distribution $\mathrm{Gam}\left( \lambda\,|\,a,b \right) $ for different values of $a$ and $b$.

- The mean and variance of the Gamma distribution evaluate to $\mathrm{E}\left( \lambda\right) = \frac{a}{b}$ and $\mathrm{var}\left[\lambda\right] = \frac{a}{b^2}$.

##### inference

- We will consider a prior $p(\lambda) = \mathrm{Gam}\left( \lambda\,|\,a_0, b_0\right)$, which leads by Bayes rule to the posterior
$$\begin{align*}
p(\lambda\,|\,D) &\propto \underbrace{\lambda^{N/2} \exp\left\{ -\frac{\lambda}{2}\sum_{n=1}^N \left(x_n - \mu \right)^2\right\} }_{\text{likelihood}} \cdot \underbrace{\frac{1}{\Gamma(a_0)} b_0^{a_0} \lambda^{a_0-1} \exp\left\{ -b_0 \lambda\right\}}_{\text{prior}} \\
  &\propto \mathrm{Gam}\left( \lambda\,|\,a_N,b_N \right)
\end{align*}$$
with
$$\begin{align*}
a_N &= a_0 + \frac{N}{2} \tag{B-2.150} \\
b_N &= b_0 + \frac{1}{2}\sum_n \left( x_n-\mu\right)^2 \tag{B-2.151}
\end{align*}$$

- Hence the **posterior is again a Gamma distribution**. By inspection of B-2.150 and B-2.151, we deduce that we can interpret $2a_0$ as the number of a priori (pseudo-)observations.

- Since the most uninformative prior is given by $a_0=b_0 \rightarrow 0$, we can derive the **maximum likelihood estimate** for the precision as
$$
\lambda_{\text{ML}} = \left.\mathrm{E}\left[ \lambda\right]\right\vert_{a_0=b_0\rightarrow 0} = \left. \frac{a_N}{b_N}\right\vert_{a_0=b_0\rightarrow 0} = \frac{N}{\sum_{n=1}^N \left(x_n-\mu \right)^2}
$$

- In short, if we do density estimation with a Gaussian distribution $\mathcal{N}\left(x_n\,|\,\mu,\sigma^2 \right)$ for an observed data set $D = \{x_1, x_2, \ldots, x_N\}$, the maximum likelihood estimates for $\mu$ and $\sigma^2$ are given by
$$\begin{align*}
\mu_{\text{ML}} &= \frac{1}{N} \sum_{n=1}^N x_n \tag{B-2.121} \\
\sigma^2_{\text{ML}} &= \frac{1}{N} \sum_{n=1}^N \left(x_n - \mu_{\text{ML}} \right)^2 \tag{B-2.122}
\end{align*}$$

- These estimates are also known as the _sample mean_ and _sample variance_ respectively.

### Some Useful Matrix Calculus

Aside from working with Gaussians, it will be helpful for the next lessons to be familiar with some matrix calculus. We shortly recapitulate used formulas here.

- We define the **gradient** of a scalar function $f(A)$ w.r.t. an $n \times k$ matrix $A$ as
$$
\nabla_A f \triangleq
    \begin{bmatrix}
\frac{\partial{f}}{\partial a_{11}} & \frac{\partial{f}}{\partial a_{12}} & \cdots & \frac{\partial{f}}{\partial a_{1k}}\\
\frac{\partial{f}}{\partial a_{21}} & \frac{\partial{f}}{\partial a_{22}} & \cdots & \frac{\partial{f}}{\partial a_{2k}}\\
\vdots & \vdots & \cdots & \vdots\\
\frac{\partial{f}}{\partial a_{n1}} & \frac{\partial{f}}{\partial a_{n2}} & \cdots & \frac{\partial{f}}{\partial a_{nk}}
    \end{bmatrix}
$$

- The following formulas are useful (see Bishop App.-C)
$$\begin{align*}
|A^{-1}|&=|A|^{-1} \tag{B-C.4} \\
\nabla_A \log |A| &= (A^{T})^{-1} = (A^{-1})^T \tag{B-C.28} \\
\mathrm{Tr}[ABC]&= \mathrm{Tr}[CAB] = \mathrm{Tr}[BCA] \tag{B-C.9} \\
\nabla_A \mathrm{Tr}[AB] &=\nabla_A \mathrm{Tr}[BA]= B^T \tag{B-C.25} \\
\nabla_A \mathrm{Tr}[ABA^T] &= A(B+B^T)  \tag{B-C.27}\\
 \nabla_x x^TAx &= (A+A^T)x \tag{from B-C.27}\\
\nabla_X a^TXb &= \nabla_X \mathrm{Tr}[ba^TX] = ab^T \notag
\end{align*}$$

```julia
open("../../styles/aipstyle.html") do f
    display("text/html", read(f, String))
end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```


#Asoociazione A.L. -> Matrice
Data l'a.l.
$$
f:R^3 \to R^2; f(x, y, z) = (ax+by+cz, dx+ey+fz)
$$
Posso identificare una matrice che la rappresenti
$$
M_f =
\begin{pmatrix}
a & b & c \\
d & e & f
\end{pmatrix} \\
$$

Pongo per ogni riga i coefficenti di ogni componente dello spazio vettoriale di arrivo.
Dato che stiamo operando da $R^3 \to R^2$ prendiamo i versori fondamentali di R^3
$$
e_1 = (1, 0, 0) \\
e_2 = (0, 1, 0) \\
e_3 = (0, 0, 1) \\
$$

E proviamo ad applicare $f$ ai tre versori
$$
f(e_1) = (a, d) \\
f(e_2) = (b, e) \\
f(e_3) = (c, f) \\
$$

Queste sono esattamente le 3 colonne della mia matrice $M_f$
Quindi sia le righe(coefficenti) che le colonne(applicazione ai versori) portano un significato.

##Costruzione della matrice
Data quindi una funzione generica
$$
f:R^n \to R^m \\
$$
Applichiamola ai versori di R^n e otteniamo le colonne
$$
e_1 = (1, 0, \dots, 0) \to^f (a_{11}, a_{21}, \dots, a_{m1}) \\
e_2 = (0, 1, \dots, 0) \to^f (a_{21}, a_{22}, \dots, a_{m2}) \\
e_n = (0, 0, \dots, 1) \to^f (a_{1n}, a_{2n}, \dots, a_{mn}) \\
\begin{align}
M_f = &
\begin{pmatrix}
a_{11} & a_{12} & \dots & a_{1n} & \\
a_{21} & a_{22} & \dots & a_{2n} & \\
\dots  & \dots  & \dots  & \dots &  \\
a_{m1} & a_{m2} & \dots & a_{mn} &
\end{pmatrix} \\
&
\begin{array}{cccc}
\downarrow & \downarrow & & \downarrow \\
f(e_1) & f(e_2) & & f(e_n)
\end{array}
\end{align}
$$

Posso vedere che la mia applicazione mappa a $R^m$ e che la mia matrice ha proprio $m$ righe.
Infatti ogni riga contiene i coefficenti da applicare alla n_upla in ingresso per avere l'elemento della emmesimo della ennumpla d'uscita
$$
f(x_1, x_2, \dots, x_n) = (a_1, a_2, \dots, a_m) \\
a_1 = x_1a_{11} + x_2a_{12} + \dots + x_na_{1n} \\
a_2 = x_1a_{21} + x_2a_{22} + \dots + x_na_{2n} \\
\dots \\
a_m = x_1a_{m1} + x_2a_{m2} + \dots + x_na_{mn} \\
$$
Quindi posso notare che
$$
f:R^n \to R^m, M_f \in R^{mn}
$$
E posso notare che ogni coponente di $f(x_1, x_2, \dots, x_n)$ è una combinazione lineare (polinomio omogenuo di primo grado) della n_upla $(x_1, x_2, \dots, x_n)$ con coefficenti derivati dalla matrice.

#Associazione Matrice -> A.L.
Assegnata una matrice questa rappresenta sempre un'applicazione lineare.
###Esempio
$$
A_f =
\begin{pmatrix}
1 & -1 & 4 \\
2 & 2 &  0
\end{pmatrix}\\
$$
Provo a definire f
$$
f: R^3 \to R^32; f(x, y , z) = (x-y+4z, 2x+2y)
$$
Esiste quindi una **corrispondenza biunivoca** tra una qualunque matrice $R^{mn}$ e un applicazione lineare $f:R^n \to R^m$

#Studio di A.L. tramite matrici
Data questa corrispondenza biunivoca è possibile studiare le matrici per analizzare la relativa applicazione lineare.
Partendo dalla matrice di un a.l. posso quindi capire se l'a.l. è:
 1. suriettiva
 2. iniettiva
 3. biiettiva

#Matrice di un A.L. Suriettiva
L'applicazione lineare è suriettiva se genera tutto lo spazio vettoriale $R^m$.
Per controllarlo guardando la matrice devo valutare se lo spazio vettoriale generato dalla colonna delle matrice corrisponde a $m$.
Per farlo è sufficente calcolare il rango della matrice, dato che il rango della matrice corrisponde alla dimensione dello spazio vettoriale delle colonne.
Se $\rho(M_f) = m; f$ è suriettiva.
Dato che se è un applicazione lineare il rango è pari al numero delle righe allora devo giusto controllare che il numero delle righe sia $m$

#Matrice di una A.L. Iniettiva
Un a.l. è iniettiva se $\ker f = \{0_{R^n}\}$
Ma
$$
\ker f = m - \dim{\operatorname{Im}f} = n - \rho{M_f} \\
$$
Quindi la a.l. è iniettiva quando $n$ è uguale al rango della matrice, cioè quando è uguale al numero delle colonne.

##Riassumendo
 * **$\rho(M_f) = m$**     l'a.l. è **Suriettiva**
 * **$\rho(M_f) = n$**     l'a.l. è **Iniettiva**
 * **$\rho(M_f) = m = n$** l'a.l. è **Biiettiva/Invertibile**

#Matrice di un A.L. Invertibile
Se
$$
f: R^n \to R^n; \text{Invertibile} \\
M_{f^{-1}} = (M_f)^{-1} \\
$$
Quindi per trovare l'applicazione lineare inversa di un applicazione lineare $f$ invertibile mi basta calcolare l'inversa della sua matrice.

#Matrice di una composizione di A.L.
Dati
$$
f: R^n \to R^m \\
g: R^m \to R^z \\
h = g \circ f \\
M_{h} = M_gM_f
$$
Quindi per calcolare la composizione di due a.l. posso moltiplicare le relative matrici.

Algebra 24 Applicazioni Lineare e Matrici.ipynb


#  Fixed particle number and mixed states

The main advantage of this package is the ability to work efficiently in the fixed particles subspace. In most situations, we are interested in fermionic states with a fixed number of particles. This could happen, for instance, when dealing with the eigenstates of the most common Hamiltonians. Given that situation, we can reduce the basis of our system by a lot. More exactly, if d is the dimension and n the number of particles, we can go from $2^d\times 2^d$ to $\binom{d}{n}\times \binom{d}{n}$ systems, which is an enormous speedup. This can be very useful for diagonalizing Hamiltonians, or many other situations. Working in the fixed particle subspace is one of the strenghts of this package.

Another common situation is dealing with states with no fixed particle number. In this case, the best description is given by the $\rho^{\rm qsp}$:

$\begin{equation}
\rho^{\rm qsp} =
\begin{pmatrix}
\rho^{\rm sp} & \kappa\\
-\kappa^* & I-\rho^{\rm sp}
\end{pmatrix}
\end{equation}$

which has eigenvalues 0 or 1 iff the state is a quasiparticles vacuum. We will learn how to compute this matrix.

Finally, we are going to cover a general mixed state, that is to say a convex combination of pure fermionic states. We will find the resulting one body matrix.

1. [Operators with fixed number](#opfix)
2. [States with fixed number](#statefix)
3. [States with non fixed number](#nonfix)
4. [Mixed states](#mixed)

```julia
using Fermionic
```

## Operators with fixed number

When working with fixed particle number, we are mainly interested in operators conserving the fermion number, i.e. $a_i^\dagger a_j$, $a_i a_j^\dagger$ and their products. We can access these operators in the following way:

```julia
# d = dimension
d = 4
# n = number of fixed particles
n = 2
# i is the mode to be created
i = 2
# j is the mode to be destroyed
j = 4
# this is the c_i^dagger c_j operator
ada(d,n,i,j)
```

    6×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 2 stored entries:
      [3, 1]  =  -1.0
      [6, 4]  =  1.0

These operators live in the subspace with fixed particle number, which scales as $\binom{d}{n}$ instead of the infamous $2^d$, with $d$ the dimension and $n$ the number of fixed particles. This basis can be explicitly obtained

```julia
b, index = basis_m(d,n);
Matrix(b)
```

    6×4 Array{Float64,2}:
     0.0  0.0  1.0  1.0
     0.0  1.0  0.0  1.0
     0.0  1.0  1.0  0.0
     1.0  0.0  0.0  1.0
     1.0  0.0  1.0  0.0
     1.0  1.0  0.0  0.0

So the operator connects:

The second output form basis_m(), which is called index, are the positions of each element from the basis b in the whole $2^n$ basis. It can be useful for mapping between basis.

```julia
println(Matrix(b)[3,:])
println(Matrix(b)[1,:])
println("--------------------")
println(Matrix(b)[6,:])
println(Matrix(b)[4,:])
```

    [0.0, 1.0, 1.0, 0.0]
    [0.0, 0.0, 1.0, 1.0]
    --------------------
    [1.0, 1.0, 0.0, 0.0]
    [1.0, 0.0, 0.0, 1.0]

You can also compute the complex conjugate aad().
In its core, the funcion ada() computes the basis of this space. If we wish to compute a combination of these operators, we can directly input the basis to avoid innecesary definitions:

```julia
b, index = basis_m(d,n);
ada(b, index, i, j)
```

    6×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 2 stored entries:
      [3, 1]  =  -1.0
      [6, 4]  =  1.0

```julia
ada(b, index, i, j)*aad(b, index, i, j)
```

    6×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 2 stored entries:
      [3, 3]  =  -1.0
      [6, 6]  =  -1.0

#### Op_fixed()

As we did with the operators, it is possible to initialize all the possible one body operators through the type Op_fixed.
This is useful when working with Hamiltonians which iterate many times over the same operators (think of two body operators summing over all modes). In order to initalize these operators, we input the dimension and the number of particles.

```julia
o = Op_fixed(d,n);
```

We can then acces the one body operators as follows:

```julia
ada(o, i, j)
```

    6×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 2 stored entries:
      [3, 1]  =  -1.0
      [6, 4]  =  1.0

We also gain access to several methods, as we did with Op objects. Thw whole list can be found in the index.

#### Creation and destuction operators in fixed space

Even though we are working in the fixed particle subspace, we can define operators $a$ and $a^\dagger$ which takes a fermionic state in the subspace with $N$ particles to the corresponding state with $N-1$ or $N+1$ particles respectivelly.

In order to do so, we define a(d,n,i) and ad(d,n,i) with d the dimension, n the number of fixed particles and i the mode. If the creation operator is applied outside the original subspace, we add an ancilla in the chosen mode.

```julia
ad(4,2,1)
```

    4×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 3 stored entries:
      [2, 1]  =  1.0
      [3, 2]  =  1.0
      [4, 3]  =  1.0

Naturally, we can apply this operator to a state in the fixed space

```julia
s=st(fixed_state(State([0,0,0,0,0,0,0,0,0,0,0,0,1,0,0,0],Op(4)),2))
println("State in fixed basis ",s)
b,_=basis_m(4,2)
println("Which corresponds to ", Array(b[6,:]))
println("If we apply the creation operator we get ", ad(4,2,4)*s)
b2,_=basis_m(4,3)
println("which corresponds to ", Array(b2[3,:]))
println("If we apply the creation on an outside mode, we add an ancilla ", ad(4,2,5)*s)
b3,_=basis_m(5,3)
println("which corresponds to ", Array(b3[8,:]))
```

    State in fixed basis [0, 0, 0, 0, 0, 1]
    Which corresponds to [1.0, 1.0, 0.0, 0.0]
    If we apply the creation operator we get [0.0, 0.0, 1.0, 0.0]
    which corresponds to [1.0, 1.0, 0.0, 1.0]
    If we apply the creation on an outside mode, we add an ancilla [0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 0.0, 1.0, 0.0, 0.0]
    which corresponds to [1.0, 1.0, 0.0, 0.0, 1.0]

It is important to remark that succesive applications have to take into account the change in the number of starting particles:

```julia
println(ad(4,1,4)*a(4,2,2)*s)
b4,_=basis_m(4,2);Matrix(b)
println("Which is ", Array(b4[4,:]))
```

    [0.0, 0.0, 0.0, 1.0, 0.0, 0.0]
    Which is [1.0, 0.0, 0.0, 1.0]

```julia
ada(4,2,2,4) == ad(4,1,2)*a(4,2,4)
```

    true

This can be simplified if we input an array of modes to the operators

```julia
ad(6,2,[2,1])
```

    15×15 SparseArrays.SparseMatrixCSC{Float64,Int64} with 6 stored entries:
      [10, 1]  =  1.0
      [11, 2]  =  1.0
      [12, 3]  =  1.0
      [13, 4]  =  1.0
      [14, 5]  =  1.0
      [15, 6]  =  1.0

```julia
ad(6,2,[2,1]) == ad(6,3,1)*ad(6,2,2)
```

    true

We can also initialize all creation and destruction operators for each mode by inputing only the dimension and the starting number of particles. For that, we use af(d,m) and adf(d,m):

```julia
atot = af(4,2)
```

    4-element Array{SparseArrays.SparseMatrixCSC{Float64,Int64},1}:

      [1, 4]  =  1.0
      [2, 5]  =  1.0
      [3, 6]  =  1.0

      [1, 2]  =  1.0
      [2, 3]  =  1.0
      [4, 6]  =  -1.0

      [1, 1]  =  1.0
      [3, 3]  =  -1.0
      [4, 5]  =  -1.0

      [2, 1]  =  -1.0
      [3, 2]  =  -1.0
      [4, 4]  =  -1.0

We got an array where each elements corresponds to a destruction operator for mode equal to the number of element. When we evaluate, it is of course equivalent to what we made before.

```julia
atot[1] == a(4,2,1)
```

    true

The same can be done for creation operators:

```julia
adtot = adf(4,2)
```

    4-element Array{SparseArrays.SparseMatrixCSC{Float64,Int64},1}:

      [2, 1]  =  1.0
      [3, 2]  =  1.0
      [4, 3]  =  1.0

      [1, 1]  =  1.0
      [3, 4]  =  -1.0
      [4, 5]  =  -1.0

      [1, 2]  =  -1.0
      [2, 4]  =  -1.0
      [4, 6]  =  1.0

      [1, 3]  =  1.0
      [2, 5]  =  1.0
      [3, 6]  =  1.0

```julia
adtot[3] == ad(4,2,3)
```

    true

### Alternative path
If we have been working in the whole space, but wish to look at the reduced subspace, we will use a function called *fixed()* which takes 2 arguments: the operator we want to reduce and the number of particles we want to fix to. For operators that preserve particle number, this will result in the elimination of empty rows and colums, allowing us to do matrix operations in smaller dimensions.

```julia
o = Op(4)
ada(o,1,2)
```

    16×16 SparseArrays.SparseMatrixCSC{Float64,Int64} with 4 stored entries:
      [9 , 5]  =  1.0
      [10, 6]  =  1.0
      [11, 7]  =  1.0
      [12, 8]  =  1.0

Notice we are using a number-preserving operation. We will now fix this operation for 2 particles states.

```julia
fixed(ada(o,1,2), 2)
```

    6×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 2 stored entries:
      [4, 2]  =  1.0
      [5, 3]  =  1.0

Which is the same as computing directly in the fixed space:

```julia
ada(4,2,1,2)
```

    6×6 SparseArrays.SparseMatrixCSC{Float64,Int64} with 2 stored entries:
      [4, 2]  =  1.0
      [5, 3]  =  1.0

Notice the dimension has dropped from 16x16 to 6x6 (in general, from $2^d\times 2^d$ to $\binom{d}{n}\times \binom{d}{n}$). This new state is written in a new reduced basis. We can always access the new basis, with fixed number parity with the function *basis_m()* which also takes two arguments: the operators Op() and the number of particles.

```julia
b, _ = basis_m(4,2);
Matrix(b)
```

    6×4 Array{Float64,2}:
     0.0  0.0  1.0  1.0
     0.0  1.0  0.0  1.0
     0.0  1.0  1.0  0.0
     1.0  0.0  0.0  1.0
     1.0  0.0  1.0  0.0
     1.0  1.0  0.0  0.0

So the operator $a_1^\dagger a_2$ connects these two states

```julia
println(Matrix(b)[4,:])
println(Matrix(b)[2,:])
println("--------------------")
println(Matrix(b)[5,:])
println(Matrix(b)[3,:])
```

    [1.0, 0.0, 0.0, 1.0]
    [0.0, 1.0, 0.0, 1.0]
    --------------------
    [1.0, 0.0, 1.0, 0.0]
    [0.0, 1.0, 1.0, 0.0]

We can do this for every operator, but it only really makes sense for those that are number preseving.
Some really nice examples are the superconducting and the Lipkin hamiltonians, which are also solved in the /examples folder.

## States with fixed number

We already shown how to work with states in order to access some properties, such as the one body matrix and its corresponding entropy. If we are working with fixed number, we can do the same thing. There is a new type now called **State_fixed**.

-  **State_fixed(s::AbstractVector, o::Op_fixed, n::Int64)**: initializes a state from a vector **s** (for example an Array{Float64,1}/Array{Complex{Float64},1} or a SparseVector{Float64,Int64}/SparseVector{Complex{Float64},Int64}) with fixed particle number **n**, using the fermionic operators defined by **o**. Once initizialized, functions below are made avaiable.

These states are written in the basis we obtained with basis_m() and should be normalized for proper results. The arguments for these types are the array/sparse array, the operators and the number of fixed particles.

It is really important that the length matches the length from basis_m(). You can create the state manually as a vector, or use the function *fixed_state()* on a state obtained in the complete basis (for example if you obtained the state through the application of creation operators over the vacuum).

```julia
#If we manually create the vector:
stat = zeros(binomial(4,2));
stat[1] = 1;
stat[6] = 1;
stat = stat/sqrt(stat'*stat)
```

    6-element Array{Float64,1}:
     0.7071067811865475
     0.0
     0.0
     0.0
     0.0
     0.7071067811865475

```julia
o = Op_fixed(4,2)
state = State_fixed(stat,o);
rhosp(state)
```

    4×4 SparseArrays.SparseMatrixCSC{Float64,Int64} with 4 stored entries:
      [1, 1]  =  0.5
      [2, 2]  =  0.5
      [3, 3]  =  0.5
      [4, 4]  =  0.5

```julia
eigensp(state)
```

    4-element Array{Float64,1}:
     0.5
     0.5
     0.5
     0.5

```julia
ssp(state)
```

    1.0

Lets project a vector in the fulled basis to the reduced basis (with fixed particle number) using fixed_state()

```julia
using Fermionic
o = Op(4)
s = (ad(o,1)*ad(o,2) + ad(o,3)*ad(o,4))*vacuum(o);
s = s/sqrt(s'*s)
```

    16-element SparseArrays.SparseVector{Float64,Int64} with 2 stored entries:
      [4 ]  =  0.707107
      [13]  =  0.707107

```julia
sf = fixed_state(s, 2) #second argument is the number of particles we want to fix
```

    6-element SparseArrays.SparseVector{Float64,Int64} with 2 stored entries:
      [1]  =  0.707107
      [6]  =  0.707107

```julia
of = Op_fixed(4,2)

state = State_fixed(sf,of);
rhosp(state)
```

    4×4 SparseArrays.SparseMatrixCSC{Float64,Int64} with 4 stored entries:
      [1, 1]  =  0.5
      [2, 2]  =  0.5
      [3, 3]  =  0.5
      [4, 4]  =  0.5

We can do the  same for already defined State and output the corresponding State_fixed

```julia
o = Op(4)
s = (ad(o,1)*ad(o,2) + ad(o,3)*ad(o,4))*vacuum(o);
s = s/sqrt(s'*s)
s = State(s,o);

sf = fixed_state(s, 2) #second argument is the number of particles we want to fix
typeof(sf)
```

    State_fixed{SparseArrays.SparseVector{Float64,Int64}}

We can also do the opposite operation, for going from the fixed to the unfixed basis. In order to do that, we use unfixed_state(). It works both for vectors, inputing unfixed_state(s::AbstractVector, n::Int64, num::Int64) with n the dimension and num the number of particles, and for State_fixed, by just inputing the state unfixed_state(s::State_fixed).

```julia
o = Op(4)
s = (ad(o,1)*ad(o,2) + ad(o,3)*ad(o,4))*vacuum(o);
s = s/sqrt(s'*s)
s = State(s,o);

sf = fixed_state(s, 2) #second argument is the number of particles we want to fix
suf = unfixed_state(sf)
typeof(suf)
st(suf) == st(s)
```

    true

## Non fixed particle number

A general fermionic state has fixed parity but no fixed number. The former formalism can also be used for working with this type of states. We have also defined the $\rho^{\rm qsp}$, which is the matrix defined as followes

$\begin{equation}
\rho^{\rm qsp} =
\begin{pmatrix}
\rho^{\rm sp} & \kappa\\
-\kappa^* & I-\rho^{\rm sp}
\end{pmatrix}
\end{equation}$

where $\kappa_{ij} = \langle a_j a_i\rangle$ and $-\kappa_{ij}^* = \langle a_j^\dagger a_i^\dagger\rangle$ compose an antisymmetric matrices. The matrix $\rho^{\rm qsp}$ can be diagonalized with Bogoliubov transformations. It is of course invisible to particle-hole transformations.

```julia
op4 = Op(4)
ad1 = ad(op4,1)
ad2 = ad(op4,2)
ad3 = ad(op4,3)
ad4 = ad(op4,4)
vac = vacuum(op4);
```

```julia
#initialize a state with no fixed fermionic number and real coefficients
state_qsp1 = (ad1*ad2+ad3*ad4+ad1*ad2*ad3*ad4)*vac + vac
state_qsp1 = state_qsp1/sqrt(state_qsp1'*state_qsp1)
state_qsp1 = State(state_qsp1, op4);
```

```julia
Matrix(rhoqsp(state_qsp1))
```

    8×8 Array{Float64,2}:
     0.5   0.0  0.0   0.0   0.0  0.5   0.0  0.0
     0.0   0.5  0.0   0.0  -0.5  0.0   0.0  0.0
     0.0   0.0  0.5   0.0   0.0  0.0   0.0  0.5
     0.0   0.0  0.0   0.5   0.0  0.0  -0.5  0.0
     0.0  -0.5  0.0   0.0   0.5  0.0   0.0  0.0
     0.5   0.0  0.0   0.0   0.0  0.5   0.0  0.0
     0.0   0.0  0.0  -0.5   0.0  0.0   0.5  0.0
     0.0   0.0  0.5   0.0   0.0  0.0   0.0  0.5

```julia
#initialize a state with no fixed fermionic number and complex coefficients
state_qsp2 = (ad1*ad2+im*ad1*ad4+ad1*ad2*ad3*ad4)*vac
state_qsp2 = state_qsp2/sqrt(state_qsp2'*state_qsp2)
state_qsp2 = State(state_qsp2, op4);
```

```julia
Matrix(rhoqsp(state_qsp2))
```

    8×8 Array{Complex{Float64},2}:
     1.0+0.0im       0.0+0.0im            0.0+0.0im       …       0.0+0.0im
     0.0+0.0im  0.666667+0.0im            0.0+0.0im               0.0+0.0im
     0.0+0.0im       0.0+0.0im       0.333333+0.0im          0.333333+0.0im
     0.0+0.0im       0.0-0.333333im       0.0+0.0im               0.0+0.0im
     0.0+0.0im       0.0+0.0im            0.0+0.0im               0.0+0.0im
     0.0+0.0im       0.0+0.0im            0.0-0.333333im  …      -0.0-0.333333im
     0.0+0.0im       0.0+0.333333im       0.0+0.0im               0.0+0.0im
     0.0+0.0im       0.0+0.0im       0.333333-0.0im          0.333333+0.0im

It is Hermitian

```julia
Matrix(rhoqsp(state_qsp1))' == Matrix(rhoqsp(state_qsp1))
```

    true

We can also check the average number of particles:

```julia
n_avg(state_qsp2)
```

    2.666666666666667

## Mixed States

So far, we have only worked with pure states. Nontheless, the most general state is a convex combination of pure states. The one body matrix of a mixed state will be obtained by combining the one body matrices of the pure states involved in the decomposition like so:

$\rho^{\rm sp} = \sum_i p_i \rho_i^{\rm sp}$

We can build this matrix by calling rhosp_mixed(). The argument are two vectors of the same length, the first one containing the probabilities and the second the respective pure states.

```julia
p1 = 0.25
p2 = 0.75
p = [p1, p2]
```

    2-element Array{Float64,1}:
     0.25
     0.75

```julia
using Fermionic

o = Op(4);

state1 = ad(o,2)*ad(o,1)*vacuum(o);
state1 = state1/sqrt(state1'*state1)
state2 = (ad(o,3)*ad(o,4)-im*ad(o,1)*ad(o,2))*vacuum(o);
state2 = state2/sqrt(state2'*state2);

s1 = State(state1,o);
s2 = State(state2,o);

s = [s1, s2];

Matrix(rhosp_mixed(p, s))
```

    4×4 Array{Complex{Float64},2}:
     0.625+0.0im    0.0+0.0im    0.0+0.0im    0.0+0.0im
       0.0+0.0im  0.625+0.0im    0.0+0.0im    0.0+0.0im
       0.0+0.0im    0.0+0.0im  0.375+0.0im    0.0+0.0im
       0.0+0.0im    0.0+0.0im    0.0+0.0im  0.375+0.0im

```julia
eigensp_mixed(p,s)
```

    4-element Array{Float64,1}:
     0.625
     0.625
     0.375
     0.375

We can also fix the number, as we just learned

```julia
using Fermionic

o = Op(4)
nume = 2
of = Op_fixed(4,2)

p1 = 0.25
p2 = 0.75
p = [p1, p2]

state1 = ad(o,2)*ad(o,1)*vacuum(o);
state1 = state1/sqrt(state1'*state1)
state2 = (ad(o,3)*ad(o,4)-im*ad(o,1)*ad(o,2))*vacuum(o);
state2 = state2/sqrt(state2'*state2);

s1 = fixed_state(state1, nume);
s2 = fixed_state(state2, nume);

s1f = State_fixed(s1, of); #Both states must be the same type!
s2f = State_fixed(s2, of);

s = [s1f, s2f];

Matrix(rhosp_mixed(p, s))
```

    4×4 Array{Complex{Float64},2}:
     0.625+0.0im    0.0+0.0im    0.0+0.0im    0.0+0.0im
       0.0+0.0im  0.625+0.0im    0.0+0.0im    0.0+0.0im
       0.0+0.0im    0.0+0.0im  0.375+0.0im    0.0+0.0im
       0.0+0.0im    0.0+0.0im    0.0+0.0im  0.375+0.0im

```julia
eigensp_mixed(p,s)
```

    4-element Array{Float64,1}:
     0.625
     0.625
     0.375
     0.375

examples/3. Fixed particle number and mixed states (Tutorial).ipynb