#
 Variational Linear Systems: Simple Example

 Ryan LaRose

This notebook briefly demonstrates the current state of the Variational Linear Systems (VLS) code. All code is contained in `vls_pauli.py`, which defines a `PauliSystem` class.

```python
# =======
# imports
# =======
import time

import numpy as np

from vls_pauli import PauliSystem
from cirq import ParamResolver, Symbol, ops, Circuit, LineQubit
```

# Creating a Linear System of Equations

A `PauliSystem` consists of a matrix of the form
\begin{equation}
A = \sum_{k = 1}^{K} c_k \sigma_k
\end{equation}
where $c_k$ are complex coefficients and $\sigma_k$ are strings of Pauli operators. In code, we represent the matrix $A$ as arrays of strings corresponding to Pauli operators. For example, to represent the Pauli operators
\begin{align}
\sigma_1 &= \sigma_I \otimes \sigma_x \otimes \sigma_Y \otimes \sigma_Z
\end{align}
we would write:

```python
# specify the pauli operators of the matrix
Amat_ops = np.array([["I", "X", "Y", "Z"]])
```

To store more terms, we simply append more lists of Pauli operators (string keys) to the operator matrix above. Coefficients $c_k$ are stored similarly as arrays of complex values:

```python
# specify the coefficients of each term in the matrix
Amat_coeffs = np.array([1-0j])
```

Finally, the solution vector
\begin{equation}
|b\rangle = U |0\rangle
\end{equation}
is represented by the unitary $U$ that (efficiently) prepares $|b\rangle$ from the ground state. For example, the unitary $U$ could be
\begin{equation}
U = \sigma_I \otimes \sigma_X \otimes \sigma_Y \otimes \sigma_Z,
\end{equation}
which we would represent in code as:

```python
# specify the unitary that prepares the solution vector b
Umat_ops = np.array(["I", "X", "Y", "Z"])
```

To create `PauliSystem`, we can then simply feed in `Amat_coeffs`, `Amat_ops`, and `Umat_ops`.

```python
# create a linear system of equations
system = PauliSystem(Amat_coeffs, Amat_ops, Umat_ops)
```

# Working with a `PauliSystem`

The `PauliSystem` class can tell basic information about the system:

```python
print("Number of qubits in system:", system.num_qubits())
print("Size of matrix:", system.size())
```

    Number of qubits in system: 4
    Size of matrix: (16, 16)

To see the actual matrix representation of the system (in the computational basis), we can do:

```python
# get the matrix of the system
matrix = system.matrix()
print(matrix)
```

    [[0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.-1.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+1.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.-1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.-1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.-1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.-1.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+1.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+1.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.-1.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.-1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j
      0.+0.j 0.+1.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+1.j 0.+0.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]
     [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.-1.j
      0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]]

We can also see the solution vector $|b\rangle$ by doing:

```python
b = system.vector()
print(b)
```

    [0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+1.j 0.+0.j 0.+0.j 0.+0.j
     0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j 0.+0.j]

# Creating an Ansatz

Initially, the `PauliSystem` ansatz for $V$ is an empty circuit:

```python
# print out the initial (empty) ansatz
system.ansatz
```

We are free to pick whatever ansatz we wish. Here, we will start with the two-qubit alternating ansatz and simplify it to single qubit rotations. The two-qubit alternating ansatz is built-in to the `PauliSystem` class and can be easily created by doing:

```python
system.make_ansatz_circuit()
system.ansatz
```

1: ───X^Symbol("0")────Y^Symbol("1")────Z^Symbol("2")────@───X^Symbol("6")────Y^Symbol("7")────Z^Symbol("8")────@───X^Symbol("39")───Y^Symbol("40")───Z^Symbol("41")───────X───X^Symbol("45")───Y^Symbol("46")───Z^Symbol("47")───────X───
                                                         │                                                      │                                                          │                                                          │
2: ───X^Symbol("3")────Y^Symbol("4")────Z^Symbol("5")────X───X^Symbol("9")────Y^Symbol("10")───Z^Symbol("11")───X───X^Symbol("24")───Y^Symbol("25")───Z^Symbol("26")───@───┼───X^Symbol("30")───Y^Symbol("31")───Z^Symbol("32")───@───┼───
                                                                                │   │                                                      │   │
3: ───X^Symbol("12")───Y^Symbol("13")───Z^Symbol("14")───@───X^Symbol("18")───Y^Symbol("19")───Z^Symbol("20")───@───X^Symbol("27")───Y^Symbol("28")───Z^Symbol("29")───X───┼───X^Symbol("33")───Y^Symbol("34")───Z^Symbol("35")───X───┼───
                                                         │                                                      │                                                          │                                                          │
4: ───X^Symbol("15")───Y^Symbol("16")───Z^Symbol("17")───X───X^Symbol("21")───Y^Symbol("22")───Z^Symbol("23")───X───X^Symbol("36")───Y^Symbol("37")───Z^Symbol("38")───────@───X^Symbol("42")───Y^Symbol("43")───Z^Symbol("44")───────@───

This circuit contains 48 parameters (4 qubits x 2 "gates" / qubit x 6 parameters / gate). (Note that printing the circuit gets cut off in the notebook, scroll side to side to see the entire circuit.) For our simple example, we will chop off some of the gates to make the optimization easier:

```python
# remove some of the gates and print it out
system.ansatz = system.ansatz[:-13]
system.ansatz
```

1: ───X^Symbol("0")────Y^Symbol("1")────Z^Symbol("2")────

2: ───X^Symbol("3")────Y^Symbol("4")────Z^Symbol("5")────

3: ───X^Symbol("12")───Y^Symbol("13")───Z^Symbol("14")───

4: ───X^Symbol("15")───Y^Symbol("16")───Z^Symbol("17")───

# Computing the Cost

The local cost function is computed via the Hadamard Test. The local cost function can be written
\begin{equation}
C_1 = 1 - \frac{1}{n} \sum_{k = 1}^{K} \sum_{l \geq k}^{K} \frac{w_{k, l} c_k c_l^*}{\langle 0 | V^\dagger A_k^\dagger A_l V | 0 \rangle} \sum_{j = 1}^{n} \text{Re} \, \langle V_{k, l}^{(j)} \rangle
\end{equation}
where
\begin{equation}
\langle V_{k, l}^{(j)} \rangle := \langle0^{\otimes n}| V^\dagger A_k^\dagger U P_j U^\dagger A_l V |0^{\otimes n}\rangle
\end{equation}
and
\begin{equation}
    w_{k, l} = \begin{cases}
    1 \qquad \text{if } k = l\\
    2 \qquad \text{otherwise}
    \end{cases} .
\end{equation}
Thus we have $(n + 1) K^2$ different circuits to run in order to compute the cost. For this simple example, $n = 4$ and $K = 1$, so we only have five circuits to run. The circuit for computing $\langle V_{1, 1}^{(1)} \rangle$ is shown below:

```python
system.make_hadamard_test_circuit(system.ops[0], system.ops[0], 0, "real")
```

0: ───H──────────────────────────────────────────────────────@───@───@───@───────@───@───@───H───M('z')───
                                                             │   │   │   │       │   │   │
1: ───────X^Symbol("0")────Y^Symbol("1")────Z^Symbol("2")────┼───┼───┼───@───────┼───┼───┼────────────────
                                                             │   │   │           │   │   │
2: ───────X^Symbol("3")────Y^Symbol("4")────Z^Symbol("5")────X───┼───┼───X───X───X───┼───┼────────────────
                                                                 │   │               │   │
3: ───────X^Symbol("12")───Y^Symbol("13")───Z^Symbol("14")───────Y───┼───Y───Y───────Y───┼────────────────
                                                                     │                   │
4: ───────X^Symbol("15")───Y^Symbol("16")───Z^Symbol("17")───────────Z───Z───Z───────────Z────────────────

The circuit for computing the norm
\begin{equation}
\langle 0 | V^\dagger A_k^\dagger A_l V | 0 \rangle = \langle \psi | A_k^\dagger A_l | \psi \rangle
\end{equation}
for the example $k = 0$, $l = 0$ is shown below:

```python
system.make_norm_circuit(system.ops[0], system.ops[0], "real")
```

0: ───H──────────────────────────────────────────────────────@───@───@───@───@───@───H───M('z')───
                                                             │   │   │   │   │   │
1: ───────X^Symbol("0")────Y^Symbol("1")────Z^Symbol("2")────┼───┼───┼───┼───┼───┼────────────────
                                                             │   │   │   │   │   │
2: ───────X^Symbol("3")────Y^Symbol("4")────Z^Symbol("5")────X───┼───┼───X───┼───┼────────────────
                                                                 │   │       │   │
3: ───────X^Symbol("12")───Y^Symbol("13")───Z^Symbol("14")───────Y───┼───────Y───┼────────────────
                                                                     │           │
4: ───────X^Symbol("15")───Y^Symbol("16")───Z^Symbol("17")───────────Z───────────Z────────────────

To compute the cost, we can call `PauliSystem.cost` or `PauliSystem.eff_cost` (the latter exploits symmetries to compute the cost more efficiently) and pass in a set of angles to the ansatz gates:

```python
# =======================================
# compute the cost for some set of angles
# =======================================

# normalize the coefficients
system.normalize_coeffs()

# get some angles
angles = np.random.randn(18)

# compute the cost and time it
start = time.time()
cost = system.eff_cost(angles)
end = time.time() - start

# print out the results
print("Local cost C_1 =", cost)
print("Time to compute cost =", end, "seconds")
```

    0.93245
    Local cost C_1 = 0.93245
    Time to compute cost = 0.28073954582214355 seconds

# Solving the System

To solve the system, we minimize the cost function. We'll do this below with the Powell optimization algorithm.

```python
# ===============================================
# minimize the cost (prints each cost evaluation)
# ===============================================
start = time.time()
out = system.solve(x0=angles, opt_method="COBYLA")
end = time.time() - start
```

    0.9289499999999999
    0.5724
    0.9228500000000001
    0.5689500000000001
    0.60015
    0.59275
    0.5699000000000001
    0.5736
    0.5724
    0.5871
    0.57345
    0.58565
    0.56935
    0.8321000000000001
    0.8319
    0.5731999999999999
    0.78175
    0.7933
    0.5726
    1.1881
    0.90985
    0.7093499999999999
    0.68825
    0.5679
    0.486
    0.46304999999999996
    0.5261
    0.4115
    0.40464999999999995
    0.40035
    0.32825000000000004
    0.21225000000000005
    0.17710000000000004
    0.1372
    0.1362
    0.12850000000000006
    0.1290499999999999
    0.14834999999999998
    0.13065000000000004
    0.1499999999999999
    0.13560000000000005
    0.15510000000000002
    0.13044999999999995
    0.15080000000000005
    0.18009999999999993
    0.1735
    0.07740000000000002
    0.08599999999999997
    0.11825000000000008
    0.14174999999999993
    0.07645000000000002
    0.06230000000000002
    0.06555
    0.052000000000000046
    0.05315000000000003
    0.11620000000000008
    0.05710000000000004
    0.054750000000000076
    0.05630000000000002
    0.05095000000000005
    0.0616000000000001
    0.05689999999999995
    0.05625000000000002
    0.041749999999999954
    0.06655
    0.03964999999999996
    0.042200000000000015
    0.04239999999999999
    0.04544999999999999
    0.06269999999999998
    0.020950000000000024
    0.02429999999999999
    0.014599999999999946
    0.01330000000000009
    0.01905000000000001
    0.012249999999999983
    0.023499999999999965
    0.013699999999999934
    0.011299999999999977
    0.024249999999999994
    0.012049999999999894
    0.026499999999999968
    0.010050000000000114
    0.011050000000000004
    0.00824999999999998
    0.00934999999999997
    0.009550000000000058
    0.02639999999999998
    0.00924999999999998
    0.016650000000000054
    0.012899999999999912
    0.020100000000000007
    0.00824999999999998
    0.011050000000000004
    0.007800000000000029
    0.00924999999999998
    0.012199999999999989
    0.009449999999999958
    0.013100000000000112
    0.009299999999999975
    0.009700000000000042
    0.008249999999999869
    0.011150000000000104
    0.011249999999999982
    0.006050000000000111
    0.00934999999999997
    0.0034999999999999476
    0.0033999999999999586
    0.0034999999999999476
    0.003549999999999942
    0.0040999999999999925
    0.0048000000000000265
    0.0024999999999999467
    0.00495000000000001
    0.0021500000000000963
    0.0041999999999999815
    0.0026999999999999247
    0.0028000000000000247
    0.0020499999999999963
    0.0024999999999999467
    0.0019499999999998963
    0.004450000000000065
    0.003450000000000064
    0.0014499999999999513
    0.0017000000000000348
    0.0005500000000000504
    0.0007499999999999174
    0.0011999999999999789
    0.0012499999999999734
    0.0004999999999999449
    0.0010000000000000009
    0.0009000000000000119
    0.0017000000000000348
    0.0008000000000000229
    0.0009000000000000119
    0.0008500000000000174
    0.0007499999999999174
    0.0006500000000000394
    0.0007500000000000284
    0.0007500000000000284
    0.0011499999999999844
    0.0005499999999999394
    0.0005499999999999394
    0.00034999999999996145
    0.0005499999999999394
    0.0004999999999999449
    0.0006500000000000394
    0.00019999999999997797
    0.0004999999999999449
    0.00029999999999996696
    0.00044999999999995044
    0.00034999999999996145
    0.0004999999999999449
    0.00029999999999996696
    0.0007000000000000339
    0.00029999999999996696
    0.00039999999999995595
    0.0004999999999999449
    0.0003500000000000725
    0.00039999999999995595
    0.00034999999999996145
    0.0006500000000000394
    0.0005500000000000504
    0.00039999999999995595
    0.00034999999999996145
    0.0005499999999999394
    0.0006000000000000449
    0.00039999999999995595
    0.00024999999999997247
    0.00039999999999995595
    0.00014999999999998348
    0.00029999999999996696
    0.00029999999999996696
    0.00044999999999995044
    0.0006000000000000449
    0.0006000000000000449
    0.0005500000000000504
    0.00044999999999995044
    0.00019999999999997797
    0.00024999999999997247
    0.00019999999999997797
    0.00039999999999995595
    0.00029999999999996696
    0.00039999999999995595
    0.00024999999999997247
    0.0005999999999999339
    0.0005499999999999394
    0.0004999999999999449
    0.00039999999999995595
    0.00044999999999995044
    0.00034999999999996145
    0.00044999999999995044
    0.0006000000000000449
    0.00034999999999996145
    0.00034999999999996145
    0.00029999999999996696
    0.0006499999999999284
    0.00014999999999998348
    0.00039999999999995595
    0.00019999999999997797
    0.00029999999999996696
    0.00034999999999996145
    0.0005499999999999394
    0.00034999999999996145
    0.00034999999999996145
    0.0005999999999999339
    0.0007000000000000339
    0.00019999999999997797
    0.0003500000000000725
    0.00039999999999995595
    0.0006000000000000449
    0.0004999999999999449
    0.00029999999999996696
    0.00029999999999996696
    0.00040000000000006697
    0.0006000000000000449
    0.00039999999999995595
    0.00034999999999996145
    0.00024999999999997247
    0.00039999999999995595
    0.00034999999999996145
    0.00024999999999997247
    0.00024999999999997247
    0.00014999999999998348
    0.00019999999999997797
    0.00029999999999996696
    0.0007000000000000339

```python
print("It took {} minute(s) to solve the system.".format(round(end / 60)))
#print("Number of iterations of optimization method:", out["nit"])
print("Number of function evaluations:", out["nfev"])
```

    It took 1 minute(s) to solve the system.
    Number of function evaluations: 227

# Comparing the Estimated and Exact Solutions

Below we print out the cost at the optimal angles found for the ansatz and print out the ansatz circuit with the optimal angles.

```python
# get the optimal angles
opt_angles = out["x"]

# evaluate the cost at the optimal angles found
system.eff_cost(opt_angles)

# get a param resolver
param_resolver = ParamResolver(
    {str(ii) : opt_angles[ii] for ii in range(len(opt_angles))}
)

sol_circ = system.ansatz.with_parameters_resolved_by(param_resolver)
sol_circ
```

    0.00040000000000006697

1: ───X^-0.994─────Y^-0.991─────Z^0.915────

2: ───X^-0.00312───Y^0.0113─────Z^-0.588───

3: ───X^0.0075─────Y^0.000161───Z^0.944────

4: ───X^0.00105────Y^-0.00422───Z^-0.38────

Next we convert the circuit to a unitary matrix and get the first column to check our solution with the actual solution.

```python
# get the approximate solution and compute "bhat"
xhat = sol_circ.to_unitary_matrix()[:, 0]
bhat = np.dot(matrix, xhat)

# print out the overlap between "bhat" and the actual solution vector b
print("overlap of computed and exact solution =", np.dot(b.conj().T, bhat))
```

    overlap of computed and exact solution = (0.04257143659529625-0.9986949908283624j)

```python
# make sure both vectors are normalized
print("<bhat|bhat> =", np.dot(bhat.conj().T, bhat))
print("<b|b> =", np.dot(b.conj().T, b))
```

    <bhat|bhat> = (1.0000000000000004+0j)
     = (1+0j)

# Future Work

* Better optimization methods.
    * Optimize over a subset of the parameters at a time, then loop through (and reoptimize).
    * Add random gates using simulated annealing.
    * Compute all $nK^2$ circuits in parallel.
* Compute expectations of local observables at each cost iteration.
* Allow for arbitrary unitaries (not just Paulis)


## Classification and density estimation

In density estimation we estimate joint probability distributions from multivariate data sets to identify the inherent clustering. This is essentially  unsupervised classification

If we have labels for some of these data points (e.g., an object is tall, short, red, or blue) we can develop a relationship between the label and the properties of a source. This is  supervised classification

Classification, regression, and density estimation are all related. For example, the regression function $\hat{y} = f(y|\vec{x})$ is the best estimated value of $y$ given a value of $\vec{x}$. In classification $y$ is categorical and $f(y|\vec{x})$ the called the _discriminant function_

- Using density estimation for classification is referred to as _generative classification_ (we have a full model of the density for each class or we have a model which describes how data could be generated from each class).

- Classification that finds the decision boundary that separates classes is called  _discriminative classification_

Both have their place in astrophysical classification

## Classification loss: how well are we doing

The first question we need to address is how we score (defined the success of our classification)

We can define a _loss function_. A zero-one loss function assigns a value of one for a misclassification and zero for a correct classification (i.e. we will want to minimize the loss).

If $\hat{y}$ is the best guess value of $y$, the classification loss, $L(y,\widehat{y})$, is

> $L(y,\widehat{y}) = \delta(y \neq \widehat{y})$

which means

>$\begin{eqnarray} L(y,\hat{y}) & = & \left\{ \begin{array}{cl} 1 & \mbox{if $y\neq\hat{y}$}, \\  0 & \mbox{otherwise.}       	           \end{array} \right. \end{eqnarray}$

The expectation (mean) value of the loss $\mathbb{E} \left[ L(y,\hat{y}) \right] = p(y\neq \hat{y})$  is called the classification risk

This is related to regression loss functions: $L(y, \hat{y}) = (y - \hat{y})^2$ and risk $\mathbb{E}[(y - \hat{y})^2]$.

We can then define:

> $ {\rm completeness} = \frac{\rm true\ positives}
  {\rm true\ positives + false\ negatives}
$

> $  {\rm contamination} = \frac{\rm false\ positives}
  {\rm true\ positives + false\ positives}
$

or

> $ {\rm true\ positive\ rate} = \frac{\rm true\ positives}
  {\rm true\ positives + false\ negatives}
$

> $  {\rm false\ positive\ rate} = \frac{\rm false\ positives}
  {\rm true\ negatives + false\ positives}
$

## Comparing the performance of classifiers

Best performance is a bit of a subjective topic (e.g. star-galaxy separation for correlation function studies or Galactic streams studies). We trade contamination as a function of completeness and this is science dependent.

**ROC curves: Receiver Operating Characteristic curves**

- Plot the true-positive vs the false-positive rate

- Initially used to analyze radar results in WWII (a very productive era for statistics...).

- One concern about ROC curves is that they are sensitive to the relative sample sizes (if there are many more background events than source events small false positive results can dominate a signal). For these cases we we can plot efficiency (1 - contamination) vs completeness

```python
%matplotlib inline

# Suppress deprecation warnings
import warnings
warnings.filterwarnings("ignore")

# Author: Jake VanderPlas
# License: BSD
import numpy as np
from matplotlib import pyplot as plt

from sklearn.naive_bayes import GaussianNB
from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA
from sklearn.linear_model import LogisticRegression
from sklearn.neighbors import KNeighborsClassifier
from sklearn.tree import DecisionTreeClassifier
from astroML.classification import GMMBayes

from sklearn.metrics import precision_recall_curve, roc_curve

from astroML.utils import split_samples, completeness_contamination
from astroML.datasets import fetch_rrlyrae_combined

from astroML.plotting import setup_text_plots
setup_text_plots(fontsize=12, usetex=True)

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
y = y.astype(int)
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25], random_state=0)

#------------------------------------------------------------
# Fit all the models to the training data
def compute_models(*args):
    names = []
    probs = []
    for classifier, kwargs in args:
        clf = classifier(**kwargs)
        clf.fit(X_train, y_train)
        y_probs = clf.predict_proba(X_test)[:, 1]

        names.append(classifier.__name__)
        probs.append(y_probs)

    return names, probs

names, probs = compute_models((GaussianNB, {}),
                              (LDA, {}),
                              (QDA, {}),
                              (LogisticRegression,
                               dict(class_weight='balanced')),
                              (KNeighborsClassifier,
                               dict(n_neighbors=10)),
                              (DecisionTreeClassifier,
                               dict(random_state=0, max_depth=12,
                                    criterion='entropy')),
                              (GMMBayes, dict(n_components=3, min_covar=1E-5,
                                              covariance_type='full')))

#------------------------------------------------------------
# Plot ROC curves and completeness/efficiency
fig = plt.figure(figsize=(10, 5))
fig.subplots_adjust(left=0.1, right=0.95, bottom=0.15, top=0.9, wspace=0.25)

# ax2 will show roc curves
ax1 = plt.subplot(121)

# ax1 will show completeness/efficiency
ax2 = plt.subplot(122)

labels = dict(GaussianNB='GNB',
              LinearDiscriminantAnalysis='LDA',
              QuadraticDiscriminantAnalysis='QDA',
              KNeighborsClassifier='KNN',
              DecisionTreeClassifier='DT',
              GMMBayes='GMMB',
              LogisticRegression='LR')

thresholds = np.linspace(0, 1, 1001)[:-1]

# iterate through and show results
for name, y_prob in zip(names, probs):
    fpr, tpr, thresh = roc_curve(y_test, y_prob)

    # add (0, 0) as first point
    fpr = np.concatenate([[0], fpr])
    tpr = np.concatenate([[0], tpr])

    ax1.plot(fpr, tpr, label=labels[name])

    comp = np.zeros_like(thresholds)
    cont = np.zeros_like(thresholds)
    for i, t in enumerate(thresholds):
        y_pred = (y_prob >= t)
        comp[i], cont[i] = completeness_contamination(y_pred, y_test)
    ax2.plot(1 - cont, comp, label=labels[name])

ax1.set_xlim(0, 0.04)
ax1.set_ylim(0, 1.02)
ax1.xaxis.set_major_locator(plt.MaxNLocator(5))
ax1.set_xlabel('false positive rate')
ax1.set_ylabel('true positive rate')
ax1.legend(loc=4)

ax2.set_xlabel('efficiency')
ax2.set_ylabel('completeness')
ax2.set_xlim(0, 1.0)
ax2.set_ylim(0.2, 1.02)

plt.savefig('figures/ROC.png')
plt.show()
```

## Generative Classification

In generative classifiers we model class-conditional densities $p_k(\vec{x})$ given $p(\vec{x}|y=y_k)$

$p(y=y_k)$, or $\pi_k$ for short, is the probability of any point having class $k$ (equivalent to the prior probability of the class $k$).

Our goal is to estimate the $p_k$'s

 The discriminative function

$\hat{y} = f(y|\vec{x})$ represents the best guess of $y$ given a value of $\vec{x}$.

$f(y|\vec{x})$ is the _discriminant function_

For a simple 2-class example

> $\begin{eqnarray}
g(\vec{x}) & = &  \int y \, p(y|\vec{x}) \, dy \\
%    & = & \int y p(y|\vec{x}) \, dy \\
       & = & 1 \cdot p(y=1 | \vec{x}) + 0 \cdot p(y=0 | \vec{x}) = p(y=1 | \vec{x}).
%     & = & p(y=1 | \vec{x})
\end{eqnarray}
$

From Bayes rule

> $\begin{eqnarray} g(\vec{x})     & = & \frac{p(\vec{x}|y=1) \, p(y=1)}{p(\vec{x}|y=1) \, p(y=1)  + p(\vec{x}|y=0) \, p(y=0)} \\         & = & \frac{\pi_1 p_1(\vec{x})}{\pi_1 p_1(\vec{x}) + \pi_0 p_0(\vec{x})} \end{eqnarray}$

 Bayes Classifier

The discriminant function gives a binary predictor called a Bayes classifier

>$\begin{eqnarray} \widehat{y} & = & \left\{ \begin{array}{cl}       	           1 & \mbox{if $g(\vec{x}) > 1/2$}, \\       	           0 & \mbox{otherwise,}       	           \end{array}       	   \right. \\     & = & \left\{ \begin{array}{cl}               1 & \mbox{if $p(y=1|\vec{x}) > p(y=0|\vec{x})$}, \\               0 & \mbox{otherwise,}               \end{array}       \right. \\     & = & \left\{ \begin{array}{cl}               1 & \mbox{if $\pi_1 p_1(\vec{x}) > \pi_0 p_0(\vec{x})$}, \\               0 & \mbox{otherwise.}               \end{array}       \right.\end{eqnarray}$

 Decision Boundary

A set of $x$ values at which each class is equally likely;

>$
\pi_1 p_1(\vec{x}) = \pi_2 p_2(\vec{x});
$

> $g_1(\vec{x}) = g_2(\vec{x})$; $g_1(\vec{x}) - g_2(\vec{x}) = 0$;  $g(\vec{x}) = 1/2$; in a two-class problem

## Simplest classifier: Naive Bayes

We want $p(x_1,x_2,x_3...x_n|y)$ but if we assume that all attributes are conditionally independent this simplifies to

> $ p(x^i,x^j|y_k) = p(x^i|y)p(x^j|y_k)$

which can be written as

> $ p({x^0,x^1,x^2,\ldots,x^N}|y_k) = \prod_i p(x^i|y_k)$

From Bayes' rule and conditional independence we get

> $
  p(y_k | {x^0,x^1,\ldots,x^N}) =
  \frac{\prod_i p(x^i|y_k) p(y_k)}
       {\sum_j \prod_i p(x^i|y_j) p(y_j)}.
$

We calculate the most likely value of $y$ by maximizing over $y_k$,

> $
\widehat{y} = \arg \max_{y_k} \frac{\prod_i p(x^i|y_k) p(y_k)}
        {\sum_j \prod_i p(x^i|y_j) p(y_j)},
$

or

> $
\widehat{y} = \arg \max_{y_k} \frac{\prod_i p_k(x^i) \pi_k}
        {\sum_j \prod_i p_j(x^i) \pi_j}.
$

## The rest of the class is now just estimating densities

>  $p(\vec{x}|y=y_k)$ and $\pi_k$ are learned from a set of training data.
- $\pi_k$ is just the frequency of the class $k$ in the training set
- $p(\vec{x}|y=y_k)$ is just the density (probability) of a object with class $k$ having the attributes $x$

If the training set does not cover the full parameter space $p_k(x^i)$ can be $0$ for some value of $y_k$ and $x^i$.  The posterior probability is then $p(y_k|\{x^i\}) = 0/0$ which is unfortunate! The trick is _Laplace smoothing_ : an offset $\alpha$ is added to the probability of each bin $p(\vec{x}|y=y_k)$ for all $i, k$ (equivalent to the addition of a Bayesian prior to the naive Bayes classifier).

## Gaussian Naive Bayes

In Gaussian naive Bayes $p_k(x^i)$ are modeled as one-dimensional normal distributions, with means $\mu^i_k$ and widths $\sigma^i_k$. The naive Bayes estimator is then

>$\hat{y} = \arg\max_{y_k}\left[\ln \pi_k - \frac{1}{2}\sum_{i=1}^N\left(2\pi(\sigma^i_k)^2 + \frac{(x^i - \mu^i_k)^2}{(\sigma^i_k)^2} \right) \right]$

 Note: this is the log of the Bayes criterion with no normalization constant

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import colors

from astroML.datasets import fetch_imaging_sample

from astroML.plotting.tools import draw_ellipse
from sklearn.naive_bayes import GaussianNB

def get_stars_and_galaxies(Nstars=10000, Ngals=10000):
    """Get the subset of star/galaxy data to plot"""
    data = fetch_imaging_sample()

    objtype = data['type']

    stars = data[objtype == 6][:Nstars]

    galaxies = data[objtype == 3][:Ngals]

    return np.concatenate([stars,galaxies]), np.concatenate([np.zeros(len(stars)),np.ones(len(galaxies))])

data, y = get_stars_and_galaxies(Nstars=10000, Ngals=10000)

# select r model mag and psf - model mag as columns
X = np.column_stack((data['rRaw'], data['rRawPSF'] - data['rRaw']))

#------------------------------------------------------------
# Fit the Naive Bayes classifier
clf = GaussianNB()
clf.fit(X, y)

# predict the classification probabilities on a grid
xlim = (15, 25)
ylim = (-5, 5)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71),
                     np.linspace(ylim[0], ylim[1], 81))
Z = clf.predict_proba(np.c_[xx.ravel(), yy.ravel()])
Z = Z[:, 1].reshape(xx.shape)

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(10,10))
ax = fig.add_subplot(111)
ax.scatter(X[:, 0], X[:, 1], c=y, cmap=plt.cm.Oranges, zorder=2)

ax.contour(xx, yy, Z, [0.5], linewidths=2., colors='blue')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$x$')
ax.set_ylabel('$y$')

plt.show()

```

## Linear discriminant analysis

Linear discriminant analysis (LDA) assumes  the class distributions have identical
covariances for all $k$ classes (all classes are a set of shifted Gaussians). The
optimal classifier is derived from the log of the class
posteriors

> $g_k(\vec{x}) = \vec{x}^T \Sigma^{-1} \vec{\mu_k} - \frac{1}{2}\vec{\mu_k}^T \Sigma^{-1} \vec{\mu_k} + \log \pi_k,
$

with $\vec{\mu_k}$ the mean of class $k$ and $\Sigma$ the covariance of the
Gaussians. The class dependent covariances that would normally give rise to a quadratic dependence on
$\vec{x}$ cancel out if they are assumed to be constant. The Bayes classifier is, therefore, linear with respect to $\vec{x}$.

The discriminant boundary between classes is the line that minimizes
the overlap between Gaussians

> $  g_k(\vec{x}) - g_\ell(\vec{x}) = \vec{x}^T \Sigma^{-1} (\mu_k-\mu_\ell)  - \frac{1}{2}(\mu_k - \mu_\ell)^T \Sigma^{-1}(\mu_k -\mu_\ell)  + \log (\frac{\pi_k}{\pi_\ell}) = 0. $

Relaxing the requirement that the covariances of the
Gaussians are constant, the discriminant function
becomes quadratic in $x$:

> $ g(\vec{x}) = -\frac{1}{2} \log | \Sigma_k |   - \frac{1}{2}(\vec{x}-\mu_k)^T C^{-1}(\vec{x}-\mu_k) + \log \pi_k. $

This is sometimes known as _quadratic discriminant analysis_ (QDA)

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import colors

from sklearn.discriminant_analysis import LinearDiscriminantAnalysis as LDA
from sklearn.discriminant_analysis import QuadraticDiscriminantAnalysis as QDA

from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# perform LDA
clf = LDA()
clf.fit(X_train[:, :2], y_train)
y_pred = clf.predict(X_test[:, :2])

# perform QDA
qlf = QDA()
qlf.fit(X_train[:, :2], y_train)
qy_pred = qlf.predict(X_test[:, :2])

completeness, contamination = completeness_contamination(y_pred, y_test)
qcompleteness, qcontamination = completeness_contamination(qy_pred, y_test)

print ("completeness", completeness, qcompleteness)
print ("contamination", contamination, qcontamination)

#------------------------------------------------------------
# Compute the decision boundary
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71),
                     np.linspace(ylim[0], ylim[1], 81))

Z = clf.predict_proba(np.c_[yy.ravel(), xx.ravel()])
Z = Z[:, 1].reshape(xx.shape)

QZ = qlf.predict_proba(np.c_[yy.ravel(), xx.ravel()])
QZ = QZ[:, 1].reshape(xx.shape)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.Oranges, zorder=2)
im.set_clim(-0.5, 1)

im = ax.imshow(Z, origin='lower', aspect='auto',
               cmap=plt.cm.binary, zorder=1,
               extent=xlim + ylim)
im.set_clim(0, 1.5)

ax.contour(xx, yy, Z, [0.5], linewidths=2., colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# right plot: qda
ax = fig.add_subplot(122)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.Oranges, zorder=2)
im.set_clim(-0.5, 1)

im = ax.imshow(QZ, origin='lower', aspect='auto',
               cmap=plt.cm.binary, zorder=1,
               extent=xlim + ylim)
im.set_clim(0, 1.5)

ax.contour(xx, yy, QZ, [0.5], linewidths=2., colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

plt.show()
```

## GMM and Bayes classification

The natural extension to the Gaussian assumptions is to use GMM's to learn the density distribution.

The number of Gaussian components $K$ must be chosen for each class independently

```python
import numpy as np
from matplotlib import pyplot as plt

from astroML.classification import GMMBayes
from astroML.decorators import pickle_results
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

# Suppress deprecation warnings
import warnings
warnings.filterwarnings("ignore")

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results

# GMM-bayes takes several minutes to run, and is order[N^2]
#  truncating the dataset can be useful for experimentation.
#X = X[::10]
#y = y[::10]

(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)
N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# perform GMM Bayes
Ncolors = np.arange(1, X.shape[1] + 1)
Ncomp = [1, 5]

@pickle_results('GMMbayes_rrlyrae.pkl')
def compute_GMMbayes(Ncolors, Ncomp):
    classifiers = []
    predictions = []

    for ncm in Ncomp:
        classifiers.append([])
        predictions.append([])
        for nc in Ncolors:
            clf = GMMBayes(ncm, min_covar=1E-5, covariance_type='full')
            clf.fit(X_train[:, :nc], y_train)
            y_pred = clf.predict(X_test[:, :nc])

            classifiers[-1].append(clf)
            predictions[-1].append(y_pred)

    return classifiers, predictions

classifiers, predictions = compute_GMMbayes(Ncolors, Ncomp)
completeness, contamination = completeness_contamination(predictions, y_test)

print ("completeness", completeness[0])
print ("contamination", contamination[0])

#------------------------------------------------------------
# Compute the decision boundary
clf = classifiers[1][1]
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71),
                     np.linspace(ylim[0], ylim[1], 81))

Z = clf.predict_proba(np.c_[yy.ravel(), xx.ravel()])
Z = Z[:, 1].reshape(xx.shape)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.binary, zorder=2)
im.set_clim(-0.5, 1)

im = ax.imshow(Z, origin='lower', aspect='auto',
               cmap=plt.cm.Oranges, zorder=1,
               extent=xlim + ylim)
im.set_clim(0, 1.5)

ax.contour(xx, yy, Z, [0.5], linewidths=2., colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness[0], '^--k', c='k', label='N=%i' % Ncomp[0])
ax.plot(Ncolors, completeness[1], 'o-k', c='k', label='N=%i' % Ncomp[1])

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination[0], '^--k', c='k', label='N=%i' % Ncomp[0])
ax.plot(Ncolors, contamination[1], 'o-k', c='k', label='N=%i' % Ncomp[1])
ax.legend(prop=dict(size=12),
          loc='lower right',
          bbox_to_anchor=(1.0, 0.78))

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

## K-nearest neighbours

As with density estimation (and kernel density estimation)  the intuitive justification is that $p(y|x) \approx p(y|x')$ if $x'$ is very close to $x$.

The number of neighbors, $K$, regulates the complexity of the classification. In simplest form, a majority rule classification is adopted, where each of the $K$ points votes on the classification. Increasing $K$ decreases the variance in the classification but at the expense of an increase in the bias.

Weights can be assigned to  individual votes by weighting the vote by the distance to the nearest point.

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib import colors

from sklearn.neighbors import KNeighborsClassifier
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# perform Classification
classifiers = []
predictions = []
Ncolors = np.arange(1, X.shape[1] + 1)
kvals = [1, 2, 5, 10, 40]

for k in kvals:
    classifiers.append([])
    predictions.append([])
    for nc in Ncolors:
        clf = KNeighborsClassifier(n_neighbors=k)
        clf.fit(X_train[:, :nc], y_train)
        y_pred = clf.predict(X_test[:, :nc])

        classifiers[-1].append(clf)
        predictions[-1].append(y_pred)

completeness, contamination = completeness_contamination(predictions, y_test)

print ("completeness (as a fn of neighbors and colors)", completeness)
print ("contamination", contamination)

#------------------------------------------------------------
# Compute the decision boundary
clf = classifiers[1][1]
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 71),
                     np.linspace(ylim[0], ylim[1], 81))

Z = clf.predict(np.c_[yy.ravel(), xx.ravel()])
Z = Z.reshape(xx.shape)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.Oranges, zorder=2)
im.set_clim(-0.5, 1)

im = ax.imshow(Z, origin='lower', aspect='auto',
               cmap=plt.cm.binary, zorder=1,
               extent=xlim + ylim)
im.set_clim(0, 2)

ax.contour(xx, yy, Z, [0.5], linewidths=2., colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

ax.text(0.02, 0.02, "k = %i" % kvals[1],
        transform=ax.transAxes)

# plot completeness vs Ncolors
ax = fig.add_subplot(222)

ax.plot(Ncolors, completeness[0], 'o-k', label='k=%i' % kvals[0])
ax.plot(Ncolors, completeness[1], '^--k', label='k=%i' % kvals[1])

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination[0], 'o-k', label='k=%i' % kvals[0])
ax.plot(Ncolors, contamination[1], '^--k', label='k=%i' % kvals[1])
ax.legend(prop=dict(size=12),
          loc='lower right',
          bbox_to_anchor=(1.0, 0.79))

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))
ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

## Discriminative classification

Rather than a probabilistc description we directly model the decision boundary between two or more classes

For a two class example $\{0,1\}$, the discriminant function is given by $g(x) =  p(y=1 | x)$. Once we have it we can use the rule

>$ \widehat{y} = \left\{ \begin{array}{cl}       	           1 & \mbox{if $g(x) > 1/2$}, \\       	           0 & \mbox{otherwise,}       	           \end{array}       	   \right.$

to perform classification.

## Logistic regression

Assuming a binomial logistic regression and consider the linear model

> $
\begin{eqnarray}
  p(y=1|x) &=& \frac{\exp\left[\sum_j \theta_j x^j\right]}
  {1 + \exp\left[\sum_j \theta_j x^j\right]}\nonumber\\
  &=& p({\theta}),
\end{eqnarray}
$

where we define

> $
\mbox{logit}(p_i) = \log\left(\frac{p_i}{1-p_i}\right)
= \sum_j \theta_j x_i^j.
$

The name logistic regression comes from the fact that the function $e^x/(1+e^x)$ is called the logistic function.
- useful for categorical regression as values cannot go above or below 1 and 0.

Because $y$ is binary, it can be modeled as a Bernoulli distribution with (conditional) likelihood function

> $
L(\beta) = \prod_{i=1}^N p_i(\beta)^{y_i} (1-p_i(\beta))^{1-y_i}.
$

logistic regression is related to linear discriminant analysis (LDA). In LDA

>\begin{eqnarray}
\log\left( \frac{p(y=1|x)}{p(y=0|x)} \right) & = &
  - \frac{1}{2}(\mu_0+\mu_1)^T \Sigma^{-1} (\mu_1-\mu_0) \\ \nonumber
  & + & \log\left( \frac{\pi_0}{\pi_1} \right) + x^T \Sigma^{-1} (\mu_1-\mu_0) \\ \nonumber
  & = & \alpha_0 + \alpha^T x.
\end{eqnarray}

In logistic regression the model is by assumption

> $
\log\left( \frac{p(y=1|x)}{p(y=0|x)} \right) = \beta_0 + \beta^T x.
$

Logistic regression minimizes classification error rather than density estimation error.

## Support Vector Machines

Find the hyperplane that maximizes the distance of the closest point from either class. This distance is the margin (width of the line before it hits a point). We want the line that maximizes the margin (m).

The points on the margin are called _support vectors_

If we assume $y \in \{-1,1\}$, (+1 is maximum margin, -1 is minimum, 0 is the decision boundary)

The maximum is then just when $\beta_0 + \beta^T x_i = 1$ etc

The hyperplane which maximizes the margin is given by finding

> \begin{equation}
\max_{\beta_0,\beta}(m) \;\;\;
  \mbox{subject to} \;\;\; \frac{1}{||\beta||} y_i ( \beta_0 + \beta^T x_i )
  \geq m \,\,\, \forall \, i.
\end{equation}

The constraints can be written as $y_i ( \beta_0 + \beta^T x_i ) \geq m ||\beta|| $.

Thus the optimization problem is equivalent to minimizing
> \begin{equation}
\frac{1}{2} ||\beta|| \;\;\; \mbox{subject to} \;\;\; y_i
  ( \beta_0 + \beta^T x_i ) \geq 1 \,\,\, \forall \, i.
\end{equation}

This optimization  is a _quadratic programming_ problem (quadratic objective function with linear constraints)

Note that because SVM uses a metric which maximizes the margin rather than a measure over all points in the data sets, it is similar in spirit to the rank-based estimators

- The median of a distribution is unaffected by even large perturbations of outlying points, as long as those perturbations do not cross the boundary.
- In the same way, once the support vectors are determined, changes to the positions or numbers of points beyond the margin will not change the decision boundary.  For this reason, SVM can be a very powerful tool for discriminative classification.

- This is why there is a high completeness compared to the other methods: it does not matter that the background sources outnumber the RR Lyrae stars by a factor of $\sim$200 to 1. It simply determines the best boundary between the small RR Lyrae clump and the large background clump.
- This completeness, however, comes at the cost of a relatively large contamination level.

- SVM is not scale invariant so it often worth rescaling the data to [0,1] or to whiten it to have a mean of 0 and variance 1 (remember to do this to the test data as well!)
- The data dont need to be separable (we can put a constraint in minimizing the number of "failures")

```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

from sklearn.svm import SVC
from astroML.decorators import pickle_results
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination

#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results

# SVM takes several minutes to run, and is order[N^2]
#  truncating the dataset can be useful for experimentation.
#X = X[::5]
#y = y[::5]

(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)

N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# Fit SVM
Ncolors = np.arange(1, X.shape[1] + 1)

@pickle_results('SVM_rrlyrae.pkl')
def compute_SVM(Ncolors):
    classifiers = []
    predictions = []

    for nc in Ncolors:
        # perform support vector classification
        clf = SVC(kernel='linear', class_weight='balanced')
        clf.fit(X_train[:, :nc], y_train)
        y_pred = clf.predict(X_test[:, :nc])

        classifiers.append(clf)
        predictions.append(y_pred)

    return classifiers, predictions

classifiers, predictions = compute_SVM(Ncolors)

completeness, contamination = completeness_contamination(predictions, y_test)

print ("completeness", completeness)
print ("contamination", contamination)

#------------------------------------------------------------
# compute the decision boundary
clf = classifiers[1]
w = clf.coef_[0]
a = -w[0] / w[1]
yy = np.linspace(-0.1, 0.4)
xx = a * yy - clf.intercept_[0] / w[1]

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
ax.plot(xx, yy, '-k')
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.Oranges, zorder=2)
im.set_clim(-0.5, 1)

ax.set_xlim(0.7, 1.35)
ax.set_ylim(-0.15, 0.4)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness, 'o-k')

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination, 'o-k')

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

## Decision Trees

The hierarchical application of decision boundaries lead to _decision trees_

Tree structure:
- top node contains the entire data set
- at each branch the data are subdivided into two child nodes
- split is based on a predefined decision boundary (usually axis aligned)
- splitting repeats, recursively, until we reach a predefined stopping criteria

The "leaf nodes" record the fraction of points that have one classification or the other

Application of the tree to classification is simple (a series of binary decisions). The  fraction of points from the training set classified as one class or the other (in the leaf node) defines the class associated with that leaf node.

Decision trees are simple to interpret (a set of questions)

## Splitting Criteria

In order to build a decision tree we must choose the feature and
value on which we wish to split the data.

Information content or entropy of the data

> $
    E(x) = -\sum_i p_i(x) \ln (p_i(x)),
$

where $i$ is the class and $p_i(x)$ is the probability of that class
given the training data.

Information gain is the reduction in entropy due to the partitioning of the data

For a binary split $IG(x)$ is

> $ IG(x|x_i) = E(x) - \sum_{i=0}^{1} \frac{N_i}{N}  E(x_i), $

where $N_i$ is the number of points, $x_i$, in the $i$th class,
and $E(x_i)$ is the entropy associated with that class

Search for the split is undertaken in a greedy fashion (one attribute at a time) where we sort the data on feature $i$ and maximize the information gain for a given split point, $s$,

>$
IG(x|s) = E(x) - \arg\max_s \left( \frac{N(x|x<s)}{N}  E(x|x<s) -
  \frac{N(x|x\ge s)}{N}  E(x|x\ge s)  \right ).
$

To determine the depth of the tree we use cross validation

```python
%matplotlib inline
import numpy as np
from matplotlib import pyplot as plt

from sklearn.tree import DecisionTreeClassifier
from astroML.datasets import fetch_rrlyrae_combined
from astroML.utils import split_samples
from astroML.utils import completeness_contamination
#----------------------------------------------------------------------
# get data and split into training & testing sets
X, y = fetch_rrlyrae_combined()
X = X[:, [1, 0, 2, 3]]  # rearrange columns for better 1-color results
(X_train, X_test), (y_train, y_test) = split_samples(X, y, [0.75, 0.25],
                                                     random_state=0)
N_tot = len(y)
N_st = np.sum(y == 0)
N_rr = N_tot - N_st
N_train = len(y_train)
N_test = len(y_test)
N_plot = 5000 + N_rr

#----------------------------------------------------------------------
# Fit Decision tree
Ncolors = np.arange(1, X.shape[1] + 1)

classifiers = []
predictions = []
Ncolors = np.arange(1, X.shape[1] + 1)
depths = [5,15]

for depth in depths:
    classifiers.append([])
    predictions.append([])
    for nc in Ncolors:
        clf = DecisionTreeClassifier(random_state=0, max_depth=depth,
                                     criterion='entropy')
        clf.fit(X_train[:, :nc], y_train)
        y_pred = clf.predict(X_test[:, :nc])

        classifiers[-1].append(clf)
        predictions[-1].append(y_pred)

completeness, contamination = completeness_contamination(predictions, y_test)

print ("completeness as a function of tree depth and number of colors", completeness)
print ("contamination", contamination)

#------------------------------------------------------------
# compute the decision boundary

clf = classifiers[1][1]
xlim = (0.7, 1.35)
ylim = (-0.15, 0.4)

xx, yy = np.meshgrid(np.linspace(xlim[0], xlim[1], 101),
                     np.linspace(ylim[0], ylim[1], 101))

Z = clf.predict(np.c_[yy.ravel(), xx.ravel()])
Z = Z.reshape(xx.shape)

#----------------------------------------------------------------------
# plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(bottom=0.15, top=0.95, hspace=0.0,
                    left=0.1, right=0.95, wspace=0.2)

# left plot: data and decision boundary
ax = fig.add_subplot(121)
im = ax.scatter(X[-N_plot:, 1], X[-N_plot:, 0], c=y[-N_plot:],
                s=4, lw=0, cmap=plt.cm.Oranges, zorder=2)
im.set_clim(-0.5, 1)

ax.contour(xx, yy, Z, [0.5], linewidths=2., colors='k')

ax.set_xlim(xlim)
ax.set_ylim(ylim)

ax.set_xlabel('$u-g$')
ax.set_ylabel('$g-r$')

ax.text(0.02, 0.02, "depth = %i" % depths[1],
        transform=ax.transAxes)

# plot completeness vs Ncolors
ax = fig.add_subplot(222)
ax.plot(Ncolors, completeness[0], 'o-k', label="depth=%i" % depths[0])
ax.plot(Ncolors, completeness[1], '^--k', label="depth=%i" % depths[1])

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.NullFormatter())

ax.set_ylabel('completeness')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

# plot contamination vs Ncolors
ax = fig.add_subplot(224)
ax.plot(Ncolors, contamination[0], 'o-k', label="depth=%i" % depths[0])
ax.plot(Ncolors, contamination[1], '^--k', label="depth=%i" % depths[1])
ax.legend(prop=dict(size=12),
          loc='lower right',
          bbox_to_anchor=(1.0, 0.79))

ax.xaxis.set_major_locator(plt.MultipleLocator(1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.2))
ax.xaxis.set_major_formatter(plt.FormatStrFormatter('%i'))

ax.set_xlabel('N colors')
ax.set_ylabel('contamination')
ax.set_xlim(0.5, 4.5)
ax.set_ylim(-0.1, 1.1)
ax.grid(True)

plt.show()
```

## Bagging and Random Forests

We can improve the performance of decisions trees (especially when there are many attributes) by _bagging_ (bootstrap aggregation). This averages the predictive results of a series of bootstrap samples

For a sample of $N$ points in a training set, bagging generates $K$ equally sized bootstrap samples from which to estimate the function $f_i(x)$. The final estimator, defined by bagging, is then
>$
f(x) = \frac{1}{K} \sum_i^K f_i(x).
$

Random forests extend bagging by generating decision trees from the bootstrap samples.  The features on which to generate the tree are selected at random from the full set of features in the data (the number of features selected per split level is typically the square root of the total number of attributes).  The final classification from
the random forest is based on the averaging of the classifications of each of the individual decision trees.

This limits over-fitting and helps with axis aligned cuts

```python
import numpy as np
from matplotlib import pyplot as plt

from sklearn.ensemble import RandomForestRegressor
from astroML.datasets import fetch_sdss_specgals
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Fetch and prepare the data
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 distinct points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the results
#  This is a long computation, so we'll save the results to a pickle.
#@pickle_results('photoz_forest.pkl')
def compute_photoz_forest(depth):
    rms_test = np.zeros(len(depth))
    rms_train = np.zeros(len(depth))
    i_best = 0
    z_fit_best = None

    for i, d in enumerate(depth):
        clf = RandomForestRegressor(n_estimators=10,max_features=3,
                                    max_depth=d, random_state=0)
        clf.fit(mag_train, z_train)

        z_fit_train = clf.predict(mag_train)
        z_fit = clf.predict(mag_test)
        rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
        rms_test[i] = np.mean(np.sqrt((z_fit - z_test) ** 2))

        if rms_test[i] <= rms_test[i_best]:
            i_best = i
            z_fit_best = z_fit

    return rms_test, rms_train, i_best, z_fit_best

depth = np.arange(1, 20)
rms_test, rms_train, i_best, z_fit_best = compute_photoz_forest(depth)
best_depth = depth[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# left panel: plot cross-validation results
ax = fig.add_subplot(121)
ax.plot(depth, rms_test, '-k', label='cross-validation')
ax.plot(depth, rms_train, '--k', label='training set')
ax.legend(loc=1, prop=dict(size=13))

ax.set_xlabel('depth of tree')
ax.set_ylabel('rms error')

ax.set_xlim(0, 21)
ax.set_ylim(0.009,  0.04)
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))

# right panel: plot best fit
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.03, 0.97, "depth = %i\nrms = %.3f" % (best_depth, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)

ax.set_xlabel(r'$\rm z_{true}$', fontsize=16)
ax.set_ylabel(r'$\rm z_{fit}$', fontsize=16)

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

## Boosting classification

Boosting is an ensemble approach motivated by the idea that combining many weak classifiers can result in an improved
classification.   Boosting creates models to attempt to correct the errors of the ensemble so far. At the heart of boosting is the idea that we reweight the data based on how incorrectly the data were classified in the previous
iteration.

We run the classification multiple times and each time reweight the data based on the previous performance of the classifier. At the end of this procedure we allow the classifiers to vote on the final classification. The most popular form of boosting is that of adaptive boosting

A fundamental limitation of the boosted decision tree is the computation time for large data sets (they rely on a chain of classifiers which are each dependent on the last).

```python
import numpy as np
from matplotlib import pyplot as plt

from sklearn.ensemble import GradientBoostingRegressor
from astroML.datasets import fetch_sdss_specgals
from astroML.decorators import pickle_results

#------------------------------------------------------------
# Fetch and prepare the data
data = fetch_sdss_specgals()

# put magnitudes in a matrix
mag = np.vstack([data['modelMag_%s' % f] for f in 'ugriz']).T
z = data['z']

# train on ~60,000 points
mag_train = mag[::10]
z_train = z[::10]

# test on ~6,000 distinct points
mag_test = mag[1::100]
z_test = z[1::100]

#------------------------------------------------------------
# Compute the results
#  This is a long computation, so we'll save the results to a pickle.
@pickle_results('photoz_boosting.pkl')
def compute_photoz_forest(N_boosts):
    rms_test = np.zeros(len(N_boosts))
    rms_train = np.zeros(len(N_boosts))
    i_best = 0
    z_fit_best = None

    for i, Nb in enumerate(N_boosts):
        clf = GradientBoostingRegressor(n_estimators=Nb, learning_rate=0.1,
                                        max_depth=3, random_state=0)
        clf.fit(mag_train, z_train)

        z_fit_train = clf.predict(mag_train)
        z_fit = clf.predict(mag_test)
        rms_train[i] = np.mean(np.sqrt((z_fit_train - z_train) ** 2))
        rms_test[i] = np.mean(np.sqrt((z_fit - z_test) ** 2))

        if rms_test[i] <= rms_test[i_best]:
            i_best = i
            z_fit_best = z_fit

    return rms_test, rms_train, i_best, z_fit_best

N_boosts = (10, 100, 200, 300, 400, 500)
rms_test, rms_train, i_best, z_fit_best = compute_photoz_forest(N_boosts)
best_N = N_boosts[i_best]

#------------------------------------------------------------
# Plot the results
fig = plt.figure(figsize=(8, 4))
fig.subplots_adjust(wspace=0.25,
                    left=0.1, right=0.95,
                    bottom=0.15, top=0.9)

# left panel: plot cross-validation results
ax = fig.add_subplot(121)
ax.plot(N_boosts, rms_test, '-k', label='cross-validation')
ax.plot(N_boosts, rms_train, '--k', label='training set')
ax.legend(loc=1, prop=dict(size=13))

ax.set_xlabel('number of boosts')
ax.set_ylabel('rms error')
ax.set_xlim(0, 510)
ax.set_ylim(0.009,  0.032)
ax.yaxis.set_major_locator(plt.MultipleLocator(0.01))

ax.text(0.03, 0.03, "Tree depth: 3",
        ha='left', va='bottom', transform=ax.transAxes)

# right panel: plot best fit
ax = fig.add_subplot(122)
ax.scatter(z_test, z_fit_best, s=1, lw=0, c='k')
ax.plot([-0.1, 0.4], [-0.1, 0.4], ':k')
ax.text(0.03, 0.97, "N = %i\nrms = %.3f" % (best_N, rms_test[i_best]),
        ha='left', va='top', transform=ax.transAxes)

ax.set_xlabel(r'$\rm z_{true}$', fontsize=16)
ax.set_ylabel(r'$\rm z_{fit}$', fontsize=16)

ax.set_xlim(-0.02, 0.4001)
ax.set_ylim(-0.02, 0.4001)
ax.xaxis.set_major_locator(plt.MultipleLocator(0.1))
ax.yaxis.set_major_locator(plt.MultipleLocator(0.1))

plt.show()
```

## Alright, alright but what the @#%! should I use?

A convenient cop-out: no single model can be known in advance to be the best classifier!

In general the level of accuracy increases for parametric models as:
- naive Bayes,
- linear discriminant analysis (LDA),
- logistic regression,
- linear support vector machines,
- quadratic discriminant analysis (QDA),
- linear ensembles of linear models.

For non-parametric models accuracy increases as:
- decision trees
- $K$-nearest-neighbor,
- neural networks
- kernel discriminant analysis,
-  kernelized support vector machines
-  random forests
- boosting

Naive Bayes and its variants are by far the easiest to compute. Linear support vector machines are more expensive, though several fast algorithms exist. Random forests can be easily parallelized.


[Sascha Spors](https://orcid.org/0000-0001-7225-9992),
Professorship Signal Theory and Digital Signal Processing,
[Institute of Communications Engineering (INT)](https://www.int.uni-rostock.de/),
Faculty of Computer Science and Electrical Engineering (IEF),
[University of Rostock, Germany](https://www.uni-rostock.de/en/)

# Tutorial Signals and Systems (Signal- und Systemtheorie)

Summer Semester 2021 (Bachelor Course #24015)

- lecture: https://github.com/spatialaudio/signals-and-systems-lecture
- tutorial: https://github.com/spatialaudio/signals-and-systems-exercises

WIP...
The project is currently under heavy development while adding new material for the summer semester 2021

Feel free to contact lecturer [frank.schultz@uni-rostock.de](https://orcid.org/0000-0002-3010-0294)

## Zeitkontinuierliche Elementarsignale

```python
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
import numpy as np

def plot_exp_osc(s, t):

    x = np.exp(s*t)
    Rex = x.real
    Imx = x.imag

    fig = plt.figure(figsize=(10, 10))
    ax = fig.add_subplot(2, 2, 1, projection='3d')
    ax.plot3D(t, Rex, Imx)
    ax.set_xlabel('t / s')
    ax.set_ylabel(r'$\Re\{x(t)\}$')
    ax.set_zlabel(r'$\Im\{x(t)\}$')
    ax.set_xticks(np.arange(tend+1))
    ax.set_yticks(np.linspace(-1, 1, 9, endpoint=True))
    ax.set_zticks(np.linspace(-1, 1, 9, endpoint=True))
    ax.set_ylim(-1, 1)
    ax.set_zlim(-1, 1)
    ax.view_init(45, -45)

    ax = fig.add_subplot(2, 2, 2, projection='3d')
    ax.plot3D(t, Rex, Imx)
    ax.set_xlabel('t / s')
    ax.set_ylabel(r'$\Re\{x(t)\}$')
    ax.set_zlabel(r'$\Im\{x(t)\}$')
    ax.set_xticks(np.arange(tend+1))
    ax.set_yticks(np.linspace(-1, 1, 9, endpoint=True))
    ax.set_zticks(np.linspace(-1, 1, 9, endpoint=True))
    ax.set_ylim(-1, 1)
    ax.set_zlim(-1, 1)
    ax.view_init(0, 0)

    ax = fig.add_subplot(2, 2, 3)
    ax.plot(t, Rex)
    ax.set_xlabel('t / s')
    ax.set_ylabel(r'$\Re\{x(t)\}$')
    ax.set_title(r'$\Re\{x(t)\}=\Re\{exp(s_0 \cdot t)\}$ mit $s_0$=' +
                 str(s.real)+r'+j$\cdot$'+str(s.imag/np.pi)+r'$\pi$')
    ax.set_xticks(np.arange(tend+1))
    ax.set_yticks(np.linspace(-2, 2, 17, endpoint=True))
    ax.set_xlim(0, tend)
    ax.set_ylim(-2, 2)
    ax.grid(True)

    ax = fig.add_subplot(2, 2, 4)
    ax.plot(t, Imx)
    ax.set_xlabel('t / s')
    ax.set_ylabel(r'$\Im\{x(t)\}$')
    ax.set_title(r'$\Im\{x(t)\}=\Im\{exp(s_0 \cdot t)\}$ mit $s_0$=' +
                 str(s.real)+r'+j$\cdot$'+str(s.imag/np.pi)+r'$\pi$')
    ax.set_xticks(np.arange(tend+1))
    ax.set_yticks(np.linspace(-2, 2, 17, endpoint=True))
    ax.set_xlim(0, tend)
    ax.set_ylim(-2, 2)
    ax.grid(True)

tend = 5
fs = 100
f0 = 1
om0 = 2*np.pi*f0
N = fs*tend
t = np.arange(N)/fs
```

# Mathematisch positiv drehende, abklingende, komplexe Schwingung

$f_0 = 1$ Hz, daher $\omega_0=2\pi$ rad/s

$\sigma_0 = -0.25$ /s

$x(t) = \mathrm{e}^{s_0 \cdot t}$ mit $s = \sigma_0 + \mathrm{j}\omega_0$

```python
sigma0 = -0.25
s0 = sigma0 + 1j*om0
plot_exp_osc(s0, t)
```

# Mathematisch negativ drehende, abklingende, komplexe Schwingung

$f_0 = 1$ Hz, daher $\omega_0=2\pi$ rad/s

$\sigma_0 = -0.25$ /s

$x(t) = \mathrm{e}^{s_0 \cdot t}$ mit $s_0 = \sigma_0 - \mathrm{j}\omega_0$, negativ drehend weil **Minuszeichen** in $s$! Kann interpretiert werden als negative Kreisfrequenz $-\omega_0$.

```python
sigma0 = -0.25
s0 = sigma0 - 1j*om0
plot_exp_osc(s0, t)
```

# Mathematisch positiv drehende, aufschwingende, komplexe Schwingung

$f_0 = 1$ Hz, daher $\omega_0=2\pi$ rad/s

$\sigma_0 = +0.1$ /s (aufschwingend, weil hier positives Vorzeichen und $\omega_0 > 0$)

$x(t) = \mathrm{e}^{s_0 \cdot t}$ mit $s_0 = \sigma_0 + \mathrm{j}\omega_0$

```python
sigma0 = +0.15
s0 = sigma0 + 1j*om0
plot_exp_osc(s0, t)
```

# Mathematisch positiv drehende, harmonische, komplexe Schwingung

$f_0 = 1$ Hz, daher $\omega_0=2\pi$ rad/s

$\sigma_0 = 0$ /s (ungedämpft)

$x(t) = \mathrm{e}^{s_0 \cdot t}$ mit $s_0 = \sigma_0 + \mathrm{j}\omega_0$, also hier speziell $x(t)=\mathrm{e}^{\mathrm{j} \omega_0 \cdot t}$

Im Bild oben, rechts erkennt man nicht, ob positiv oder negativ drehend, daher brauchen wir zusätzlich den Bezug zwischen Real- und Imaginärteil über die Zeit.

```python
sigma0 = 0
s0 = sigma0 + 1j*om0
plot_exp_osc(s0, t)
```

# Exponentiell abklingend

$f_0 = 0$ Hz, daher $\omega_0=0$ rad/s

$\sigma_0 = -1$ /s (negatives Vorzeichen ist abklingend)

$x(t) = \mathrm{e}^{s_0 \cdot t}$ mit $s_0 = \sigma_0 + \mathrm{j}\omega_0$

Wir erhalten eine reelle Funktion x(t), ähnelt einer Entladekurve des Kondensators mit $\tau=1$ s, hat damit tatsächlich auch viel zu tun.

```python
sigma0 = -1
s0 = sigma0 + 1j*0
plot_exp_osc(s0, t)
```

# Exponentiell ansteigend

$f_0 = 0$ Hz, daher $\omega_0=0$ rad/s

$\sigma_0 = 0.125$ /s (positives Vorzeichen ist ansteigend)

$x(t) = \mathrm{e}^{s_0 \cdot t}$ mit $s_0 = \sigma + \mathrm{j}\omega_0$

Wir erhalten eine reelle Funktion x(t).

```python
sigma0 = +0.125
s0 = sigma0 + 1j*0
plot_exp_osc(s0, t)
```

# Rechteck-Signal

\begin{equation}
\text{rect}(t) = \begin{cases} 1 & |t| < \frac{1}{2} \\ \frac{1}{2} & |t| = \frac{1}{2} \\ 0 & |t| > \frac{1}{2} \end{cases}
\end{equation}

Nach Definition hat rect(t) Fläche 1.

```python
plt.figure(figsize=(5, 3))
eps = np.finfo(np.float32).eps
plt.plot([-2, -1/2-eps, -1/2, -1/2+eps, +1/2-eps, +1/2, +1/2+eps, 2],
         [0, 0, 1/2, 1, 1, 1/2, 0, 0])
plt.xlabel('t')
plt.ylabel('rect(t)')
plt.xlim(-2, 2)
plt.grid(True)
```

# Dreieck-Signal

\begin{equation}
\text{tri}(t) = \begin{cases} + t + 1 & -1 \leq t \leq 0 \\ - t +1 & 0 \leq t \leq +1 \\ 0 & \mathrm{sonst} \end{cases}
\end{equation}

Nach Definition hat tri(t) Fläche 1.

```python
plt.figure(figsize=(5, 3))
t = np.linspace(-2, 0, 100)
x = t + 1
x[t < -1] = 0
plt.plot(t, x, 'C0')
t = np.linspace(0, 2, 100)
x = - t + 1
x[t > +1] = 0
plt.plot(t, x, 'C0')
plt.xlabel('t')
plt.ylabel('tri(t)')
plt.xlim(-2, 2)
plt.grid(True)
```

# Spaltfunktion

Sinc-Funktion

\begin{equation}
\text{sinc}(t) := \begin{cases} \frac{\sin(t)}{t} & t \neq 0 \\ 1 & t = 0 \end{cases}
\end{equation}

```python
plt.figure(figsize=(5, 3))
t = np.linspace(-20, 20, 200)
x = np.sinc(t / np.pi)  # numpy's sinc is defined as sin(pi t) / (pi t) !!!
plt.plot(t, x)
plt.xlabel('t')
plt.ylabel('sinc(t)')
plt.xlim(-20, +20)
plt.grid(True)
```

# Sprungfunktion

Einheitssprung, Heaviside-Funktion, Heaviside step function

\begin{equation}
\epsilon(t) = \begin{cases} 0 & t<0 \\  \frac{1}{2} & t=0 \\ 1 & t > 0 \end{cases}
\end{equation}

```python
plt.figure(figsize=(5, 3))
eps = np.finfo(np.float32).eps
plt.plot([-2, -eps, 0, +eps, 2],
         [0, 0, 1/2, 1, 1])
plt.xlabel('t')
plt.ylabel(r'$\epsilon$(t)')
plt.xlim(-2, 2)
plt.grid(True)
```

# Vorzeichenfunktion

sign function, signum function

\begin{equation}
\text{sgn}(t) = \begin{cases} 1 & t>0 \\ 0 & t=0 \\ -1 & t < 0  \end{cases}
\end{equation}

Wichtig: wir sollten die Sprungfunktion und Vorzeichenfunktion gut zu unterscheiden wissen.

```python
plt.figure(figsize=(5, 3))
eps = np.finfo(np.float32).eps
plt.plot([-2, -eps], [-1, -1], 'C0')
plt.plot([+eps, 2], [1, 1], 'C0')
plt.plot([0], [0], 'C0o')
plt.xlabel('t')
plt.ylabel(r'sgn(t)')
plt.xlim(-2, 2)
plt.grid(True)
```

# Dirac Impuls

Es wird definiert
\begin{equation}
\int_{-\infty}^{\infty} x(t) \cdot \delta(t) := x(t=0)
\end{equation}

Speziell für $x(t)=1$ gilt
\begin{equation}
\int_{-\infty}^{\infty} \delta(t) = 1
\end{equation}

Wir zeichnen den Dirac Impuls als Pfeil und seinen Vorfaktor/Gewicht in Klammer an die Pfeilspitze. Streng genommen hat der Pfeil keine Amplitude, aber beim Malen müssen wir ihm eine Höhe geben, es hat sich bewährt, die Höhe des Vorfaktors zu benutzen, speziell dann, wenn in das Diagramm auch andere Signale eingezeichnet werden.

```python
plt.figure(figsize=(5, 3))
ax = plt.axes()
ax.arrow(0, 0, 0, 1, head_width=0.05, head_length=0.1, fc='C0', ec='C0')
plt.text(0.1,1.1,'(1)')
plt.xlim(-2,2)
plt.yticks([0, 10])
plt.ylim(0,1.5)
plt.xlabel('t')
plt.ylabel(r'$\delta$(t)');
```

# Dirac Impuls-Kamm

Dirac-Kamm

Wir definieren einen verschobenen Dirac Impuls um Einheitszeiten $\mu\in\mathbb{Z}$

\begin{equation}
{\bot \!\! \bot \!\! \bot}(t) = \sum_{\mu = - \infty}^{\infty} \delta(t - \mu)
\end{equation}

Hier malen wir die Pfeile für die Dirac Impulse mit Amplitude was seinem Vorfaktor/Gewicht, also

```python
plt.figure(figsize=(5, 3))
ax = plt.axes()
for mu in np.arange(-5, 6):
    ax.arrow(mu, 0, 0, 0.9, head_width=0.2, head_length=0.1, fc='C0', ec='C0')
    plt.text(mu+0.05, 1.05, '(1)')
plt.text(5.1, 0.5, '.....')
plt.text(-5.7, 0.5, '.....')
plt.xticks(np.arange(-5, 6))
plt.xlim(-5.5, 5.5)
plt.yticks([0, 1, 10])
plt.ylim(0, 1.25)
plt.xlabel('t')
plt.ylabel(r'${\bot \!\! \bot \!\! \bot}(t) $')
plt.grid(True)
```

## Copyright

This tutorial is provided as Open Educational Resource (OER), to be found at
https://github.com/spatialaudio/signals-and-systems-exercises
accompanying the OER lecture
https://github.com/spatialaudio/signals-and-systems-lecture.
Both are licensed under a) the Creative Commons Attribution 4.0 International
License for text and graphics and b) the MIT License for source code.
Please attribute material from the tutorial as *Frank Schultz,
Continuous- and Discrete-Time Signals and Systems - A Tutorial Featuring
Computational Examples, University of Rostock* with
``main file, github URL, commit number and/or version tag, year``.


###### Content under Creative Commons Attribution license CC-BY 4.0, code under MIT license (c) 2019 Daniel Koehn, based on (c)2018 L.A. Barba, G.F. Forsyth [CFD Python](https://github.com/barbagroup/CFDPython#cfd-python), (c)2014 L.A. Barba, I. Hawke, B. Knaepen [Practical Numerical Methods with Python](https://github.com/numerical-mooc/numerical-mooc#practical-numerical-methods-with-python), also under CC-BY.

```python
# Execute this cell to load the notebook's style sheet, then ignore it
from IPython.core.display import HTML
css_file = '../style/custom.css'
HTML(open(css_file, "r").read())
```

<link href="https://fonts.googleapis.com/css?family=Merriweather:300,300i,400,400i,700,700i,900,900i" rel='stylesheet' >
<link href="https://fonts.googleapis.com/css?family=Source+Sans+Pro:300,300i,400,400i,700,700i" rel='stylesheet' >
<link href='http://fonts.googleapis.com/css?family=Source+Code+Pro:300,400' rel='stylesheet' >
<style>

@font-face {
    font-family: "Computer Modern";
    src: url('http://mirrors.ctan.org/fonts/cm-unicode/fonts/otf/cmunss.otf');
}

#notebook_panel { /* main background */
    background: rgb(245,245,245);
}

div.cell { /* set cell width */
    width: 800px;
}

div #notebook { /* centre the content */
    background: #fff; /* white background for content */
    width: 1000px;
    margin: auto;
    padding-left: 0em;
}

#notebook li { /* More space between bullet points */
margin-top:0.5em;
}

/* draw border around running cells */
div.cell.border-box-sizing.code_cell.running {
    border: 1px solid #111;
}

/* Put a solid color box around each cell and its output, visually linking them*/
div.cell.code_cell {
    background-color: rgb(256,256,256);
    border-radius: 0px;
    padding: 0.5em;
    margin-left:1em;
    margin-top: 1em;
}

div.text_cell_render{
    font-family: 'Source Sans Pro', sans-serif;
    line-height: 140%;
    font-size: 110%;
    width:680px;
    margin-left:auto;
    margin-right:auto;
}

/* Formatting for header cells */
.text_cell_render h1 {
    font-family: 'Merriweather', serif;
    font-style:regular;
    font-weight: bold;
    font-size: 250%;
    line-height: 100%;
    color: #004065;
    margin-bottom: 1em;
    margin-top: 0.5em;
    display: block;
}
.text_cell_render h2 {
    font-family: 'Merriweather', serif;
    font-weight: bold;
    font-size: 180%;
    line-height: 100%;
    color: #0096d6;
    margin-bottom: 0.5em;
    margin-top: 0.5em;
    display: block;
}

.text_cell_render h3 {
    font-family: 'Merriweather', serif;
	font-size: 150%;
    margin-top:12px;
    margin-bottom: 3px;
    font-style: regular;
    color: #008367;
}

.text_cell_render h4 {    /*Use this for captions*/
    font-family: 'Merriweather', serif;
    font-weight: 300;
    font-size: 100%;
    line-height: 120%;
    text-align: left;
    width:500px;
    margin-top: 1em;
    margin-bottom: 2em;
    margin-left: 80pt;
    font-style: regular;
}

.text_cell_render h5 {  /*Use this for small titles*/
    font-family: 'Source Sans Pro', sans-serif;
    font-weight: regular;
    font-size: 130%;
    color: #e31937;
    font-style: italic;
    margin-bottom: .5em;
    margin-top: 1em;
    display: block;
}

.text_cell_render h6 { /*use this for copyright note*/
    font-family: 'Source Code Pro', sans-serif;
    font-weight: 300;
    font-size: 9pt;
    line-height: 100%;
    color: grey;
    margin-bottom: 1px;
    margin-top: 1px;
}

    .CodeMirror{
            font-family: "Source Code Pro";
			font-size: 90%;
    }
/*    .prompt{
        display: None;
    }*/

    .warning{
        color: rgb( 240, 20, 20 )
        }
</style>

# 2D Linear Advection Equation

In order to solve the 2D thermal mantle convection in the next class, we first have to deal with the 2D advection problem. To mitigate the effect of numerical diffusion, we solve the problem by implementing a 2D Marker-in-Cell code based on our 1D implementation from [this notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/04_Advection_1D/03_Numerical_Diffusion.ipynb).

## 2D Linear Advection Problem

The linear 2D heat advection equation reads:

\begin{equation}
\frac{\partial T}{\partial t} + v_x \frac{\partial T}{\partial x} + v_z \frac{\partial T}{\partial z}= 0, \tag{1}
\end{equation}

where $T$ denotes the 2D temperature field and $v_x, v_z$ the components of the velocity field in x- and z-direction, respectively.

As we have discussed in [this exercise](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/04_Advection_1D/03_Numerical_Diffusion.ipynb) almost all finite-difference based approaches to solve the advection problem suffer either from numerical instability or numerical diffusion. Therefore, we used the **Marker-in-Cell method** to avoid these problems. The basic 1D Marker-in-Cell code can be reused for the 2D advection problem. However, we have to consider some problems and improvements in our 2D Marker-in-Cell code:

For each marker $i$ at position $x_i$ and time $t^n$:

1. Save the current marker position $x^n,\; z^n$ at time step $n$.

2. Estimate velocity field ${\bf{V^n}} = (V_x^n,\;V_z^n)^T$ at current marker positions $x^n$ using linear interpolation.

3. Estimate temperature field $T$ at current marker positions using cubic interpolation.

4. Update marker positions via $x^{n+1} = x^n + dt\; v(t^{n+1},x^n)$

5. Estimate new velocity field ${\bf{V^{n+1}}} = (V_x^{n+1},\;V_z^{n+1})^T$ at marker positions $x^{n+1}$ using linear interpolation.

6. Estimate an average velocity $\bf{V_{avg}} = \frac{{\bf{V^{n+1}}}+{\bf{V^{n}}}}{2}$

7. Starting again from point $x^n$ and update the marker position via $x^{n+1} = x^n + dt\; V_{avg}$

5. Use cubic interpolation to estimate the temperature on the 2D grid using the temperature of the markers.

Some care has to be taken if point $x^n$ is outside of the computational domain, since
Python will return NaN for the velocity (or temperature) of this point. If no extrapolation is desired, use the velocity $V(t^{n}, x_i)$ in this case.

Let 's implement this **midpoint scheme** in Python:

```python
# Import libraries
import numpy
from matplotlib import pyplot
%matplotlib inline
from scipy import interpolate
```

```python
# Set the font family and size to use for Matplotlib figures.
pyplot.rcParams['font.family'] = 'serif'
pyplot.rcParams['font.size'] = 16
```

```python
# Marker-in-cell code to solve the 2D advection equation
# ------------------------------------------------------
def Adv_2D(T0, nt, dt, dx, dz, Vx, Vz, Lx1, Lx2, Lz1, Lz2, x, z, X, Z):
    """
    Computes and returns the temperature distribution
    after a given number of time steps for the 2D advection
    problem. A marker-in-cell approach with Dirichlet conditions
    on all boundaries is used in order to mitigate the effect of
    numerical diffusion.

    Parameters
    ----------
    T0 : numpy.ndarray
        The initial temperature distribution as a 2D array of floats.
    nt : integer
        Maximum number of time steps to compute.
    dt : float
        Time-step size.
    dx : float
        Grid spacing in the x direction.
    dz : float
        Grid spacing in the z direction.
    Vx : float
        x-component of the velocity field.
    Vz : float
        z-component of the velocity field.
    Lx1, Lx2 : float
        Model extension from Lx1 - Lx2.
    Lz1, Lz2 : float
        Model extension from Lz1 - Lz2.
    x, z : float
        Model coordinates as 1D arrays.
    X, Z : float
        Model coordinates as 2D arrays.

    Returns
    -------
    T : numpy.ndarray
        The temperature distribution as a 2D array of floats.
    """

    # Integrate in time.
    T = T0.copy()

    # Estimate number of grid points in x- and z-direction
    nz, nx = T.shape

    # Define number of markers and initial marker positions
    nx_mark = 4 * nx  # number of markers in x-direction
    nz_mark = 4 * nz  # number of markers in z-direction

    # Time loop
    for n in range(nt):

        # initial marker positions
        x_mark = numpy.linspace(Lx1, Lx2, num=nx_mark)
        z_mark = numpy.linspace(Lz1, Lz2, num=nz_mark)
        X_mark, Z_mark = numpy.meshgrid(x_mark,z_mark)

        # Interpolate velocities from grid to marker position at timestep n
        f = interpolate.interp2d(x, z, Vx, kind='linear')
        vx_mark_n = f(x_mark, z_mark)

        f = interpolate.interp2d(x, z, Vz, kind='linear')
        vz_mark_n = f(x_mark, z_mark)

        # Interpolate temperature from grid to marker position at timestep n
        f = interpolate.interp2d(x, z, T, kind='cubic')
        T_mark = f(x_mark, z_mark)

        # Save current marker positions
        X0 = X_mark
        Z0 = Z_mark

        # Update marker position
        X_mark = X_mark + vx_mark_n * dt
        Z_mark = Z_mark + vz_mark_n * dt

        # Interpolate velocities from grid to marker position at timestep n+1
        vx_mark_n1 = interpolate.griddata((X.flatten(), Z.flatten()), Vx.flatten(), (X_mark, Z_mark), method='linear')
        vz_mark_n1 = interpolate.griddata((X.flatten(), Z.flatten()), Vz.flatten(), (X_mark, Z_mark), method='linear')

        # Replace Nan values
        mask = numpy.where(numpy.isnan(vx_mark_n1))
        vx_mark_n1[mask] = 0

        mask = numpy.where(numpy.isnan(vz_mark_n1))
        vz_mark_n1[mask] = 0

        # Update marker position with midpoint velocity
        X_mark = X0 + dt * (vx_mark_n + vx_mark_n1) / 2.
        Z_mark = Z0 + dt * (vz_mark_n + vz_mark_n1) / 2.

        # Interpolate temperature field from marker to grid positions
        T = interpolate.griddata((X_mark.flatten(), Z_mark.flatten()), T_mark.flatten(), (X, Z), method='cubic')

        # Replace Nan-values by old temperature field
        mask = numpy.where(numpy.isnan(T))
        T[mask] = T0[mask]

    return T
```

## Temperature Advection in boring velocity field $V = (Z,-X)$

After implementing the 2D Marker-in-Cell code to solve the 2D linear advection problem, let 's define some modelling parameters and initial spatial Gaussian temperature distribution

\begin{equation}
T_0 = A_0\; exp\biggl(-\frac{(x+x_{\text{shift}})^2+z^2}{\sigma^2}\biggr) \notag
\end{equation}

on the domain $x \in \{-0.5,0.5\},\; z \in \{-0.5,0.5\}$ with amplitude $A_0 = 2$, shift in x-direction $x_{\text{shift}} = 0.25$ and half-width $\sigma = 0.1$.

To advect the temperature, a velocity field is required. We start with the "boring" velocity field $V = (Z,-X)$ from the [How to sail without wind exercise](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/02_finite_difference_intro/3_fd_ODE_example_sailing_wo_wind.ipynb):

```python
# Definition of modelling parameters
# ----------------------------------
Lx = 0.5  # half-length of the model in x direction [m]
Lz = 0.5  # half-height of the model in z direction [m]
nx = 41   # number of points in the x direction
nz = 41   # number of points in the z direction
dx = 2 * Lx / (nx - 1)  # grid spacing in the x direction
dz = 2 * Lz / (nz - 1)  # grid spacing in the z direction

# Define the locations along a gridline.
x = numpy.linspace(-Lx, Lx, num=nx)
z = numpy.linspace(-Lz, Lz, num=nz)

# Define the initial temperature distribution as Gaussian function
X, Z = numpy.meshgrid(x,z) # coordinates X,Z required to define T0

sigma = 0.1
amp = 2.
xshift = 0.25

# initial temperature distribution
T0 = amp * numpy.exp(-((X + xshift)**2 + Z**2) / sigma**2.)

# Define velocity field (Vx,Vz)
Vx = Z
Vz = - X
```

Let's check if the initial temperature distribution and velocity field are correctly implemented.

```python
# Plot the initial temperature distribution and streamlines
pyplot.figure(figsize=(10.0, 7))

extent = [numpy.min(x), numpy.max(x),numpy.min(z), numpy.max(z)]
cmap = 'gist_heat'
im = pyplot.imshow(numpy.flipud(T0), extent=extent, interpolation='spline36', cmap=cmap)
stream = pyplot.streamplot(X,Z,Vx,Vz,color='w')

pyplot.xlabel('x [m]')
pyplot.ylabel('z [m]')
cbar = pyplot.colorbar(im)
cbar.set_label('Initial Temperature [°C]')
```

Looks correct, so finally we define the timestep `dt` and the number of time steps `nt` you want to advect the temperature field and run the `Adv_2D` code

```python
# Set the time-step size
dt = 5e-1  # time-step size [s]
nt = 10   # number of time steps to compute

# Compute the temperature distribution after nt timesteps
T = Adv_2D(T0, nt, dt, dx, dz, Vx, Vz, -Lx, Lx, -Lz, Lz, x, z, X, Z)
```

```python
# Plot the temperature distribution and streamlines
pyplot.figure(figsize=(10.0, 7))

extent = [numpy.min(x), numpy.max(x),numpy.min(z), numpy.max(z)]
cmap = 'gist_heat'
im = pyplot.imshow(numpy.flipud(T), extent=extent, interpolation='spline36', cmap=cmap, aspect=1)
stream = pyplot.streamplot(X,Z,Vx,Vz,color='w')

pyplot.xlabel('x [m]')
pyplot.ylabel('z [m]')
cbar = pyplot.colorbar(im)
cbar.set_label('Temperature [°C]')
```

As we have seen in the [1D advection marker-in-cell notebook](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/04_Advection_1D/03_Numerical_Diffusion.ipynb) the marker-in-cell method shows no numerical diffusion. As you can see, this is also true for the 2D implementation. This becomes more obvious when visualizing the 2D advection in the following movie:

```python
from IPython.display import YouTubeVideo
YouTubeVideo('tBd88w_tTF0')
```

Let's play around with another velocity field ...

## Temperature advection in a more complex velocity field

Because we want to use the marker-in-cell code to model mantle convection in the next lecture, let's study advection in the velocity field from the [Lorenz Equations](https://nbviewer.jupyter.org/github/daniel-koehn/Differential-equations-earth-system/blob/master/03_Lorenz_equations/01_LorenzEquations_intro.ipynb):

\begin{equation}
\begin{split}
V_x &= - 2 \pi\; sin(\pi x)\; cos(\pi z)\\
V_z &= 2 \pi\; cos(\pi x)\; sin(\pi z)
\end{split}
\notag
\end{equation}

First, we define the modelling parameters ...

```python
# Definition of modelling parameters
# ----------------------------------
Lx = 1.    # half-length of the model in x direction [m]
Lz = 1.    # half-height of the model in z direction [m]
nx = 41    # number of points in the x direction
nz = 41    # number of points in the z direction
dx = Lx / (nx - 1)  # grid spacing in the x direction
dz = Lz / (nz - 1)  # grid spacing in the z direction

# Define the locations along a gridline.
x = numpy.linspace(0, Lx, num=nx)
z = numpy.linspace(0, Lz, num=nz)

# Define the initial temperature distribution as Gaussian function
X, Z = numpy.meshgrid(x,z) # coordinates X,Z required to define T0

sigma = 0.1
amp = 2.
xshift = -0.25
zshift = -0.25

# initial temperature distribution
T0 = amp * numpy.exp(-((X + xshift)**2 + (Z+zshift)**2) / sigma**2.)

# Define velocity field (Vx,Vz)
Vx = - 2. * numpy.pi * numpy.sin(numpy.pi*X) * numpy.cos(numpy.pi*Z)
Vz = 2. * numpy.pi * numpy.cos(numpy.pi*X) * numpy.sin(numpy.pi*Z)
```

... check if the initial temperature distribution and velocity field are correctly implemented ...

```python
# Plot the initial temperature distribution and streamlines
pyplot.figure(figsize=(16., 7.))

extent = [numpy.min(x), numpy.max(x),numpy.min(z), numpy.max(z)]
cmap = 'gist_heat'
im = pyplot.imshow(numpy.flipud(T0), extent=extent, interpolation='spline36', cmap=cmap)
stream = pyplot.streamplot(X,Z,Vx,Vz,color='w')

pyplot.xlabel('x [m]')
pyplot.ylabel('z [m]')
cbar = pyplot.colorbar(im)
cbar.set_label('Initial Temperature [°C]')
pyplot.xlim(numpy.min(x), numpy.max(x))
pyplot.ylim(numpy.min(z), numpy.max(z))
```

... and run the advection code ...

```python
# Set the time-step size
dt = 3e-2  # time-step size [s]
nt = 8    # number of time steps to compute

# Compute the temperature distribution after nt timesteps
T = Adv_2D(T0, nt, dt, dx, dz, Vx, Vz, 0, Lx, 0, Lz, x, z, X, Z)
```

```python
# Plot the temperature distribution and streamlines
pyplot.figure(figsize=(10.0, 7))

extent = [numpy.min(x), numpy.max(x),numpy.min(z), numpy.max(z)]
cmap = 'gist_heat'
im = pyplot.imshow(numpy.flipud(T), extent=extent, interpolation='spline36', cmap=cmap, aspect=1)
stream = pyplot.streamplot(X,Z,Vx,Vz,color='w')

pyplot.xlabel('x [m]')
pyplot.ylabel('z [m]')
cbar = pyplot.colorbar(im)
cbar.set_label('Temperature [°C]')
pyplot.xlim(numpy.min(x), numpy.max(x))
pyplot.ylim(numpy.min(z), numpy.max(z))
```

As you can see, the spherical 2°C temperature blob get 's quite distorted by the convective velocity field, the longer it is advected in the complex velocity field. Play around with the number of time steps `nt`. A movie of the more complex advection example can be found here:

```python
from IPython.display import YouTubeVideo
YouTubeVideo('GfLyEv9qYUU')
```

## What we learned:

* How to implement the 2D Marker-in-Cell code in Python

* Computed 2D advection of a Gaussian temperature distribution in a simple velocity field, where the shape is conserved

* However, when using the velocity field from the Lorenz problem, the temperature distribution is significantly distorted over time.