```python
import sympy
sympy.init_printing()

u_max,u_star,rho_max,rho_star,A,B=sympy.symbols('u_max u_star rho_max rho_star A B')
```

```python
eq1=sympy.Eq(0,u_max*rho_max*(1-A*rho_max-B*rho_max**2))
eq2=sympy.Eq(0,u_max*(1 - 2*A*rho_star-3*B*rho_star**2))
eq3=sympy.Eq(u_star,u_max*(1-A*rho_star-B*rho_star**2))
eq4=sympy.Eq(eq2.lhs-3*eq3.lhs,eq2.rhs-3*eq3.rhs)

rho_sol=sympy.solve(eq4,rho_star)[0]
B_sol=sympy.solve(eq1,B)[0]
quadA=eq2.subs([(rho_star,rho_sol),(B,B_sol)])
A_sol=sympy.solve(quadA,A)
```

```python
aval=A_sol[0].evalf(subs={u_star: 1.5,u_max:2.0,rho_max:15.0})
bval=B_sol.evalf(subs={rho_max:15.0,A:aval})
```

```python
aval
```

```python
bval
```

```python
sympy.init_printing(use_latex=False)
u_max=2.0
rho_max=15.0
u_star=1.5
rho_star_1=-12.6369
rho_star_2=6.32057

```

    0.0

```python

```

```python

```


# Differential calculus

```python
from sympy import Symbol
from sympy.core.containers import Tuple
from sympy import symbols
from sympy import IndexedBase
from sympy import Matrix
from sympy import Function
from sympy import pi, cos, sin
from sympy import srepr
from sympy import expand
from sympy import Function
from sympy import Integer, Float, Rational
from sympy.physics.quantum import TensorProduct

from sympde.core import Constant
from sympde.calculus import grad, dot, inner, outer, cross, rot, curl, div
from sympde.calculus import laplace, hessian, bracket, convect, D, conv
from sympde.calculus import ArgumentTypeError
from sympde.calculus import jump, avg, Dn, minus, plus
from sympde.topology import Domain
from sympde.topology import ScalarFunctionSpace, VectorFunctionSpace
from sympde.topology import ProductSpace
from sympde.topology import H1Space, HcurlSpace, HdivSpace, L2Space, UndefinedSpace
from sympde.topology import TestFunction, ScalarTestFunction, VectorTestFunction
from sympde.topology import ScalarField, VectorField
from sympde.topology import element_of, elements_of
```

```python
domain = Domain('Omega', dim=3)

V = ScalarFunctionSpace('V', domain)
W = VectorFunctionSpace('W', domain)

alpha, beta, gamma = [Constant(i) for i in ['alpha','beta','gamma']]

f,g,h = [element_of(V, name=i) for i in ['f','g','h']]
F,G,H = [element_of(W, i) for i in ['F','G','H']]
```

### Gradient properties

$ \nabla (f+g) = \nabla f + \nabla g $

$ \nabla (\alpha f) = \alpha \nabla f$

$ \nabla (\alpha f+ \beta g) = \alpha \nabla f + \beta \nabla g $

$ \nabla (fg) = f\nabla g + g\nabla f $

$ \nabla (\frac{f}{g}) = -\frac{f}{g^2} \nabla g + \frac{1}{g} \nabla f$

```python
assert( grad(f+g) == grad(f) + grad(g) )
assert( grad(alpha*f) == alpha*grad(f) )
assert( grad(alpha*f + beta*g) == alpha*grad(f) + beta*grad(g)  )

assert( grad(f*g) == f*grad(g) + g*grad(f) )
assert( grad(f/g) == -f*grad(g)/g**2 + grad(f)/g )

assert( expand(grad(f*g*h)) == f*g*grad(h) + f*h*grad(g) + g*h*grad(f) )
```

### Curl properties

```python
assert( curl(F+G) == curl(F) + curl(G) )
assert( curl(alpha*F) == alpha*curl(F) )
assert( curl(alpha*F + beta*G) == alpha*curl(F) + beta*curl(G)  )

assert( curl(cross(F,G)) == F*div(G) - G*div(F) - convect(F, G) + convect(G, F) )
assert( curl(f*F) == f*curl(F) + cross(grad(f), F) )
```

### Laplace properties

```python
assert( laplace(f+g) == laplace(f) + laplace(g) )
assert( laplace(alpha*f) == alpha*laplace(f) )
assert( laplace(alpha*f + beta*g) == alpha*laplace(f) + beta*laplace(g)  )
assert( laplace(f*g) == f*laplace(g) + g*laplace(f) + 2*dot(grad(f), grad(g)) )
```

### Divergence properties

```python
assert( div(F+G) == div(F) + div(G) )
assert( div(alpha*F) == alpha*div(F) )
assert( div(alpha*F + beta*G) == alpha*div(F) + beta*div(G)  )

assert( div(cross(F,G)) == -dot(F, curl(G)) + dot(G, curl(F)) )
assert( div(f*F) == f*div(F) + dot(F, grad(f)))
```

### Other properties

```python
assert( curl(grad(f)) == 0 )
assert( div(curl(F)) == 0 )
assert( div(cross(grad(f), grad(g))) == 0 )
assert( curl(curl(F)) == grad(div(F)) - laplace(F))
assert( curl(f*grad(g)) == cross(grad(f), grad(g)) )
```

```python

```


# Clase 4

## Overfitting y underfitting

```python
%matplotlib inline
%config InlineBackend.figure_format = 'retina'

import matplotlib.pyplot as plt
import numpy as np
import seaborn as sns
```

```python
sns.set(rc={'figure.figsize': (12, 8)}, style='white')
np.random.seed(42)
```

## Simulación con un modelo lineal

Generamos datos de acuerdo al siguiente modelo:

\begin{align}
y &= \alpha_1 x + \alpha_0 + \epsilon \\
\alpha_1 &= 1.5 \\
\alpha_0 &= 2 \\
\epsilon &\sim \mathcal{N}(0, 1)\\
\end{align}

```python
X = np.linspace(0, 10, 30, endpoint=False)

α_1 = 1.5
α_0 = 2
ϵ = np.random.normal(0, 1, size=len(X))
Y = α_1 * X + α_0 + ϵ
```

```python
sns.scatterplot(X, Y)
```

## Regresión lineal

Ahora hacemos un fit con un modelo lineal.

$$ \hat{y} = \alpha_1 x + \alpha_0$$

```python
modelo_lineal = np.polyfit(X, Y, deg=1)
```

```python
modelo_lineal
```

    array([1.39458098, 2.32137836])

```python
Y_predicted = np.polyval(modelo_lineal, X)

ax = sns.scatterplot(X, Y, label='observado')
sns.lineplot(X, Y_predicted, color='r', label='modelo')
```

A continuación, simulamos nuevos datos, usando el modelo inicial.

```python
X_nuevos = np.linspace(10, 15, 15)
Y_nuevos = α_1 * X_nuevos + α_0 + np.random.normal(0, 1, size=len(X_nuevos))
```

```python
Y_predicted_nuevos = np.polyval(modelo_lineal, X_nuevos)

X_todos = np.append(X, X_nuevos)
Y_todos = np.append(Y, Y_nuevos)
Y_predicted_todos = np.append(Y_predicted, Y_predicted_nuevos)

ax = sns.scatterplot(X, Y, label='observado')
sns.scatterplot(X_nuevos, Y_nuevos, label='nuevo', color='orange', ax=ax)
sns.lineplot(X_todos, Y_predicted_todos, color='r', label='modelo', alpha=0.3, ax=ax)
```

Para ver cuán bueno fue el fit, ploteamos los residuos.

$$e = y - \hat{y}$$

Dónde $y$ son los valores observados y $\hat{y}$ las predicciones de nuestro modelo.

```python
def plot_residuals(x, y, y_pred):
    residuals = y - y_pred
    _fig, (ax1, ax2) = plt.subplots(1, 2)

    sns.scatterplot(x, residuals, ax=ax1)
    ax1.axhline(0, linestyle='--', color='grey', alpha=0.6)
    sns.distplot(residuals, ax=ax2)
```

```python
plot_residuals(X, Y, Y_predicted)
```

También podemos calcular el [coeficiente de determinación](https://es.wikipedia.org/wiki/Coeficiente_de_determinación), conocido como $R^2$, que determina que proporción de la varianza de los datos explica el modelo.

$$R^2 := 1 - \frac{RSS}{TSS} \\
RSS = \sum (y - \hat{y})^2 \\
TSS = \sum (y - \bar{y})^2$$

Donde $RSS$ es la [suma del cuadrado de los residuos](https://en.wikipedia.org/wiki/Residual_sum_of_squares), $TSS$ es la [suma total de los cuadrados](https://en.wikipedia.org/wiki/Total_sum_of_squares) y $\bar{y}$ es la media lo los valores observados de $y$.

```python
def r_squared(y, y_pred):
    RSS = ((y - y_pred) ** 2).sum()
    TSS = ((y - y.mean()) ** 2).sum()

    return 1 - RSS/TSS
```

```python
r_squared(Y, Y_predicted)
```

    0.9590924499139448

## Regresión polinómica

Probamos ahora fitteando una [regresión polinomial](https://es.wikipedia.org/wiki/Regresión_no_lineal#Regresión_polinomial) de 4to grado.

$$ \hat{y} = \sum_{i=0}^{4} \alpha_i x^i$$

```python
modelo_polinomico = np.polyfit(X, Y, deg=4)
modelo_polinomico
```

    array([-0.0047652 ,  0.09426085, -0.56455497,  2.36610021,  2.24718869])

```python
Y_predicted = np.polyval(modelo_polinomico, X)

ax = sns.scatterplot(X, Y, label='observado')
sns.lineplot(X, Y_predicted, color='r', label='modelo')
```

```python
r_squared(Y, Y_predicted)
```

    0.9656338527552731

```python
Y_predicted_nuevos = np.polyval(modelo_polinomico, X_nuevos)
Y_predicted_todos = np.append(Y_predicted, Y_predicted_nuevos)

ax = sns.scatterplot(X, Y, label='observado')
sns.scatterplot(X_nuevos, Y_nuevos, label='nuevo', color='orange', ax=ax)
sns.lineplot(X_todos, Y_predicted_todos, color='r', label='modelo', alpha=0.3, ax=ax)
```

## Simulación con un modelo cuadrático

\begin{align}
y &= \alpha_2 x^2 + \alpha_1 x + \alpha_0 + \epsilon \\
\alpha_2 &= 2 \\
\alpha_1 &= 1.5 \\
\alpha_0 &= 2 \\
\epsilon &\sim \mathcal{N}(0, 1)\\
\end{align}

```python
X = np.linspace(-3, 3, 30, endpoint=False)

α_2 = 2
α_1 = 1.5
α_0 = 2
ϵ = np.random.normal(0, 1, size=len(X))
Y = α_2 * X * X + α_1 * X + α_0 + ϵ
```

```python
sns.scatterplot(X, Y)
```

```python
modelo_lineal = np.polyfit(X, Y, deg=1)

Y_predicted = np.polyval(modelo_lineal, X)

ax = sns.scatterplot(X, Y, label='observado')
sns.lineplot(X, Y_predicted, color='r', label='modelo')
```

```python
r_squared(Y, Y_predicted)
```

    0.12354571557502825

```python
plot_residuals(X, Y, Y_predicted)
```

En este caso, podemos observar que nuestro modelo no captura correctamente la estructura de los datos.

## Ejercicio

Ajustar un modelo cuadrático y calcular el coeficiente de determinación.


# Reconstruction of a complex wavefunction

In this tutorial, a walkthrough of how to reconstruct a **complex** wavefunction via training a *Restricted Boltzmann Machine* (RBM), the neural network behind qucumber, will be presented.

## The wavefunction to be reconstructed
The simple wavefunction below describing two qubits (coefficients stored in *qubits_psi.txt*) will be reconstructed.

\begin{equation}
            \vert\psi \rangle = \alpha \vert00\rangle + \beta \vert 01\rangle + \gamma \vert10\rangle + \delta \vert11\rangle
\end{equation}

where the exact values of $\alpha, \beta, \gamma$ and $\delta$ used for this tutorial are

\begin{align}
\alpha & = 0.2861  + 0.0539 i \\
\beta &  = 0.3687 - 0.3023 i \\
\gamma & = -0.1672 - 0.3529 i \\
\delta & = -0.5659 - 0.4639 i.
\end{align}

The example dataset, *qubits_train.txt*, comprises of 500 $\sigma$ measurements made in various bases (X, Y and Z). A corresponding file containing the bases for each data point in *qubits_train.txt*, *qubits_train_bases.txt*, is also required. As per convention, spins are represented in binary notation with zero and one denoting spin-down and spin-up, respectively.

## Using qucumber to reconstruct the wavefunction

### Imports
To begin the tutorial, first import the required Python packages.

```python
import numpy as np
import torch
import matplotlib.pyplot as plt

from qucumber.nn_states import ComplexWaveFunction

from qucumber.callbacks import MetricEvaluator

import qucumber.utils.unitaries as unitaries
import qucumber.utils.cplx as cplx

import qucumber.utils.training_statistics as ts
import qucumber.utils.data as data
```

The Python class *ComplexWaveFunction* contains generic properties of a RBM meant to reconstruct a complex wavefunction, the most notable one being the gradient function required for stochastic gradient descent.

To instantiate a *ComplexWaveFunction* object, one needs to specify the number of visible and hidden units in the RBM. The number of visible units, *num_visible*, is given by the size of the physical system, i.e. the number of spins or qubits (2 in this case), while the number of hidden units, *num_hidden*, can be varied to change the expressiveness of the neural network.

**Note:** The optimal *num_hidden* : *num_visible* ratio will depend on the system. For the two-qubit wavefunction described above, good results are yielded when this ratio is 1.

On top of needing the number of visible and hidden units, a *ComplexWaveFunction* object requires the user to input a dictionary containing the unitary operators (2x2) that will be used to rotate the qubits in and out of the computational basis, Z, during the training process. The *unitaries* utility will take care of creating this dictionary.

The *MetricEvaluator* class and *training_statistics* utility are built-in amenities that will allow the user to evaluate the training in real time.

Lastly, the *cplx* utility allows qucumber to be able to handle complex numbers. Currently, Pytorch does not support complex numbers.

### Training
To evaluate the training in real time, the fidelity between the true wavefunction of the system and the wavefunction that qucumber reconstructs, $\vert\langle\psi\vert\psi_{RBM}\rangle\vert^2$, will be calculated along with the Kullback-Leibler (KL) divergence (the RBM's cost function). First, the training data and the true wavefunction of this system need to be loaded using the *data* utility.

```python
train_path = "qubits_train.txt"
train_bases_path = "qubits_train_bases.txt"
psi_path = "qubits_psi.txt"
bases_path = "qubits_bases.txt"

train_samples, true_psi, train_bases, bases = data.load_data(
    train_path, psi_path, train_bases_path, bases_path
)
```

The file *qubits_bases.txt* contains every unique basis in the *qubits_train_bases.txt* file. Calculation of the full KL divergence in every basis requires the user to specify each unique basis.

As previouosly mentioned, a *ComplexWaveFunction* object requires a dictionary that contains the unitariy operators that will be used to rotate the qubits in and out of the computational basis, Z, during the training process. In the case of the provided dataset, the unitaries required are the well-known $H$, and $K$ gates. The dictionary needed can be created with the following command.

```python
unitary_dict = unitaries.create_dict()
# unitary_dict = unitaries.create_dict(unitary_name=torch.tensor([[real part],
#                                                                 [imaginary part]],
#                                                                 dtype=torch.double)
```

If the user wishes to add their own unitary operators from their experiment to *unitary_dict*, uncomment the block above. When *unitaries.create_dict()* is called, it will contain the identity and the $H$ and $K$ gates by default with the keys "Z", "X" and "Y", respectively.

The number of visible units in the RBM is equal to the number of qubits. The number of hidden units will also be taken to be the number of visible units.

```python
nv = train_samples.shape[-1]
nh = nv

nn_state = ComplexWaveFunction(
    num_visible=nv, num_hidden=nh, unitary_dict=unitary_dict, gpu=False
)
# nn_state = ComplexWaveFunction(num_visible=nv, num_hidden=nh, unitary_dict=unitary_dict)
```

By default, qucumber will attempt to run on a GPU if one is available (if one is not available, qucumber will default to CPU). If one wishes to run qucumber on a CPU, add the flag "gpu = False" in the *ComplexWaveFunction* object instantiation. Uncomment the line above to run this tutorial on a GPU.

Now the hyperparameters of the training process can be specified.

1. **epochs**: the total number of training cycles that will be performed (default = 100)
2. **pos_batch_size**: the number of data points used in the positive phase of the gradient (default = 100)
3. **neg_batch_size**: the number of data points used in the negative phase of the gradient (default = *pos_batch_size*)
4. **k**: the number of contrastive divergence steps (default = 1)
5. **lr**: the learning rate (default = 0.001)

    **Note:** For more information on the hyperparameters above, it is strongly encouraged that the user to read through the brief, but thorough theory document on RBMs. One does not have to specify these hyperparameters, as their default values will be used without the user overwriting them. It is recommended to keep with the default values until the user has a stronger grasp on what these hyperparameters mean. The quality and the computational efficiency of the training will highly depend on the choice of hyperparameters. As such, playing around with the hyperparameters is almost always necessary.

The two-qubit example in this tutorial should be extremely easy to train, regardless of the choice of hyperparameters. However, the hyperparameters below will be used.

```python
epochs = 100
pbs = 50  # pos_batch_size
nbs = 50  # neg_batch_size
lr = 0.1
k = 5
```

For evaluating the training in real time, the *MetricEvaluator* will be called to calculate the training evaluators every 10 epochs. The *MetricEvaluator* requires the following arguments.

1. **log_every**: the frequency of the training evaluators being calculated is controlled by the *log_every* argument (e.g. *log_every* = 200 means that the *MetricEvaluator* will update the user every 200 epochs)
2. A dictionary of functions you would like to reference to evaluate the training (arguments required for these functions are keyword arguments placed after the dictionary)

The following additional arguments are needed to calculate the fidelity and KL divergence in the *training_statistics* utility.

- **target_psi** (the true wavefunction of the system)
- **space** (the hilbert space of the system)

The training evaluators can be printed out via the *verbose=True* statement.

Although the fidelity and KL divergence are excellent training evaluators, they are not practical to calculate in most cases; the user may not have access to the target wavefunction of the system, nor may generating the hilbert space of the system be computationally feasible. However, evaluating the training in real time is extremely convenient.

Any custom function that the user would like to use to evaluate the training can be given to the *MetricEvaluator*, thus avoiding having to calculate fidelity and/or KL divergence. As an example, functions that calculate the the norm of each of the reconstructed wavefunction's coefficients are presented. Any custom function given to *MetricEvaluator* must take the neural-network state (in this case, the *ComplexWaveFunction* object) and keyword arguments. Although the given example requires the hilbert space to be computed, the scope of the *MetricEvaluator*'s ability to be able to handle any function should still be evident.

```python
def alpha(nn_state, space, **kwargs):
    rbm_psi = nn_state.psi(space)
    normalization = nn_state.compute_normalization(space).sqrt_()
    alpha_ = cplx.norm(
        torch.tensor([rbm_psi[0][0], rbm_psi[1][0]], device=nn_state.device)
        / normalization
    )

    return alpha_

def beta(nn_state, space, **kwargs):
    rbm_psi = nn_state.psi(space)
    normalization = nn_state.compute_normalization(space).sqrt_()
    beta_ = cplx.norm(
        torch.tensor([rbm_psi[0][1], rbm_psi[1][1]], device=nn_state.device)
        / normalization
    )

    return beta_

def gamma(nn_state, space, **kwargs):
    rbm_psi = nn_state.psi(space)
    normalization = nn_state.compute_normalization(space).sqrt_()
    gamma_ = cplx.norm(
        torch.tensor([rbm_psi[0][2], rbm_psi[1][2]], device=nn_state.device)
        / normalization
    )

    return gamma_

def delta(nn_state, space, **kwargs):
    rbm_psi = nn_state.psi(space)
    normalization = nn_state.compute_normalization(space).sqrt_()
    delta_ = cplx.norm(
        torch.tensor([rbm_psi[0][3], rbm_psi[1][3]], device=nn_state.device)
        / normalization
    )

    return delta_
```

Now the hilbert space of the system must be generated for the fidelity and KL divergence and the dictionary of functions the user would like to compute every "*log_every*" epochs must be given to the *MetricEvaluator*.

```python
log_every = 2
space = nn_state.generate_hilbert_space(nv)

callbacks = [
    MetricEvaluator(
        log_every,
        {
            "Fidelity": ts.fidelity,
            "KL": ts.KL,
            "normα": alpha,
#             "normβ": beta,
#             "normγ": gamma,
#             "normδ": delta,
        },
        target_psi=true_psi,
        bases=bases,
        verbose=True,
        space=space,
    )
]
```

Now the training can begin. The *ComplexWaveFunction* object has a property called *fit* which takes care of this.

```python
nn_state.fit(
    train_samples,
    epochs=epochs,
    pos_batch_size=pbs,
    neg_batch_size=nbs,
    lr=lr,
    k=k,
    input_bases=train_bases,
    callbacks=callbacks,
)
```

All of these training evaluators can be accessed after the training has completed, as well. The code below shows this, along with plots of each training evaluator versus the training cycle number (epoch).

```python
fidelities = callbacks[0].Fidelity
KLs = callbacks[0].KL
coeffs = callbacks[0].normα
# Please note that the key given to the *MetricEvaluator* must be what comes after callbacks[0].
epoch = np.arange(log_every, epochs + 1, log_every)
```

```python
# Some parameters to make the plots look nice
params = {'text.usetex': True,
            'font.family': 'serif',
            'legend.fontsize': 14,
            'figure.figsize': (10, 3),
            'axes.labelsize': 16,
            'xtick.labelsize':14,
            'ytick.labelsize':14,
            'lines.linewidth':2,
            'lines.markeredgewidth': 0.8,
            'lines.markersize': 5,
            'lines.marker': "o",
            "patch.edgecolor": "black"
         }
plt.rcParams.update(params)
plt.style.use('seaborn-deep')
```

```python
fig, axs = plt.subplots(nrows=1, ncols=3, figsize=(14, 3))
ax = axs[0]
ax.plot(epoch, fidelities, "o", color = "C0", markeredgecolor="black")
ax.set_ylabel(r'Fidelity')
ax.set_xlabel(r'Epoch')

ax = axs[1]
ax.plot(epoch, KLs, "o", color = "C1", markeredgecolor="black")
ax.set_ylabel(r'KL Divergence')
ax.set_xlabel(r'Epoch')

ax = axs[2]
ax.plot(epoch, coeffs, "o", color = "C2", markeredgecolor="black")
ax.set_ylabel(r'$\vert\alpha\vert$')
ax.set_xlabel(r'Epoch')

plt.tight_layout()
plt.savefig("complex_fid_KL.pdf")
plt.show()
```

It should be noted that one could have just ran *nn_state.fit(train_samples)* and just used the default hyperparameters and no training evaluators.

At the end of the training process, the network parameters (the weights, visible biases and hidden biases) are stored in the *ComplexWaveFunction* object. One can save them to a pickle file, which will be called *saved_params.pt*, with the following command.

```python
nn_state.save("saved_params.pt")
```

This saves the weights, visible biases and hidden biases as torch tensors with the following keys: "weights", "visible_bias", "hidden_bias".


Fill in any place that says `# YOUR CODE HERE` or YOUR ANSWER HERE, as well as your name and collaborators below.
Grading for pre-lecture assignments is all or nothing. Partial credit is available for in-class assignments and checkpoints, but **only when code is commented**.

```python
NAME = ""
COLLABORATORS = ""
```

---

```python
import grading_helper as _test
```

```python
# Space for imports, utility functions, etc.

# YOUR CODE HERE
```

# Double Pendulum

(Adapted from textbook exercise 8.15)

A double pendulum consists of a normal pendulum with another pendulum hanging from its end. For simplicity let us ignore friction, and assume that both pendulums have bobs of the same mass $m$ and massless arms of the same length $\ell$. Thus the setup looks like this:

The position of the arms at any moment in time is uniquely specified by the two angles $\theta_1$ and $\theta_2$.  The equations of motion for the angles are most easily derived using the Lagrangian formalism. The resulting pair of equations are

$$\begin{align}
2\ddot\theta_1 + \ddot\theta_2 \cos(\theta_1-\theta_2) &+ \dot\theta_2^2 \sin(\theta_1-\theta_2) + 2{g\over\ell} \sin\theta_1 = 0\,,\\
\ddot\theta_2 + \ddot\theta_1 \cos(\theta_1-\theta_2) &- \dot\theta_1^2 \sin(\theta_1-\theta_2) + {g\over\ell} \sin\theta_2 = 0\,.
\end{align}$$

These are second-order equations, but we can convert them into first-order ones by the usual method, defining two new variables, $\omega_1$ and $\omega_2$, thus:

$$\dot\theta_1 = \omega_1,\qquad \dot\theta_2 = \omega_2\,.$$

In terms of these variables our equations of motion become

$$\begin{align}
2\dot\omega_1 + \dot\omega_2 \cos(\theta_1-\theta_2) &+ \omega_2^2 \sin(\theta_1-\theta_2) + 2{g\over\ell} \sin\theta_1 = 0\,,\\
\dot\omega_2 + \dot\omega_1 \cos(\theta_1-\theta_2) &- \omega_1^2 \sin(\theta_1-\theta_2) + {g\over\ell} \sin\theta_2 = 0\,.
\end{align}$$

Finally we have to rearrange these into the standard form with a single derivative on the left-hand side of each one, which gives

$$\begin{align*}
\dot\omega_1 &= - {\omega_1^2\sin(2\theta_1-2\theta_2) + 2\omega_2^2\sin(\theta_1-\theta_2) + (g/\ell) \bigl[ \sin(\theta_1-2\theta_2) + 3 \sin\theta_1 \bigr]\over 3 - \cos(2\theta_1-2\theta_2)}, \\ \\
\dot\omega_2 &= {4\omega_1^2\sin(\theta_1-\theta_2) + \omega_2^2\sin(2\theta_1-2\theta_2) + 2(g/\ell) \bigl[ \sin(2\theta_1-\theta_2) - \sin\theta_2 \bigr]\over 3 - \cos(2\theta_1-2\theta_2)}.
\end{align*}$$

(If you want to try deriving it yourself, this last step involves some trigonometric identities.)

These two equations, along with the equations $\dot{\theta}_1=\omega_1$ and $\dot{\theta}_2=\omega_2$, give us four first-order equations which between them define the motion of the double pendulum.

1. Write a program to solve the equations of motion from $t=0$ to $t=60$ seconds for the case where $\ell=0.4$ meters, with the initial conditions $\theta_1=\theta_2=\pi$ rad, $\omega_1=0$, and $\omega_2=0.001$ rad/s.

2. The mechanical energy of the system is given by

  $$E = m\ell^2 \bigl[ \omega_1^2 + \tfrac12 \omega_2^2 + \omega_1\omega_2 \cos(\theta_1-\theta_2) \bigr] - mg\ell(2\cos\theta_1 + \cos\theta_2)\,.$$

  Calculate the total energy of the system assuming that the mass $m=1$ kg, and make a graph of energy as a function of time from $t=0$ to $t=60$ seconds. Store the energy in an array named `E`. Before you move on, make sure that energy is conserved.

```python
%%graded # 10 points

# YOUR CODE HERE
```

```python
%%tests

_test.similar(E, 11.76*np.ones(len(E))) # E should be constant
```

Make an animation of the motion of the double pendulum that moves in real time (i.e. your animation should be 60 seconds long).

> Make sure that every time step in your animation is the same size. The motion of the pendulum should be smooth.

```python
%%graded # 10 points

# YOUR CODE HERE
```

Assignments/10.5 Checkpoint - Double Pendulum.ipynb


# quant-econ Solutions: Finite Markov Chains

Solutions for

* http://quant-econ.net/py/linear_algebra.html

* http://quant-econ.net/jl/linear_algebra.html

Thanks to Willem Hekman and Guanlong Ren for providing this solution.

## Solution to Exercise 1

We have an optimization problem:

$$ v(x) = \max_{y,u} \{ -y'Py - u'Qu \} $$

s.t.

$$ y = Ax + Bu $$

with primitives

- $P$ be a symmetric and positive semidefinite $n \times n$ matrix.

- $Q$ be a symmetric and positive semidefinite $m \times m$ matrix.

- $A$ an $n \times n$ matrix.

- $B$ an $n \times m$ matrix.

The associated Lagrangian is :

$$L = -y'Py - u'Qu + \lambda' \lbrack Ax + Bu - y \rbrack$$

### 1.

Differentiating Lagrangian equation w.r.t y and setting its derivative equal to zero yields

$$ \frac{ \partial L}{\partial y} = - (P + P') y - \lambda = - 2 P y - \lambda = 0 \:,$$
since P is symmetric.

Accordingly, the first-oder condition for maximizing L w.r.t. y implies

$$ \lambda = -2 Py \:.$$

### 2.

Differentiating Lagrangian equation w.r.t. u and setting its derivative equal to zero yields

$$ \frac{ \partial L}{\partial u} = - (Q + Q') u - B'\lambda = - 2Qu + B'\lambda = 0 \:.$$

Substituting $\lambda = -2 P y$ gives

$$ Qu + B'Py = 0 \:.$$

Substituting the linear constraint $y = Ax + Bu$ into above equation gives

$$ Qu + B'P(Ax + Bu) = 0  $$

$$ (Q + B'PB)u + B'PAx = 0 $$
which is the first-oder condition for maximizing L w.r.t. u.

Thus, the optimial choice of u must satisfy

$$ u = -(Q + B'PB)^{-1}B'PAx \:,$$

which follows from the definition of the first-oder conditions for Lagrangian equation.

### 3.

Rewriting our problem by substituting the constraint into the objective function, we get

$$ v(x) = \max_{u} \{ -(Ax+ Bu)'P(Ax+Bu) - u'Qu \} \:.$$

Since we know the optimial choice of u satisfies $ u = -(Q + B'PB)^{-1}B'PAx $, then

$$ v(x) =  -(Ax+ B u)'P(Ax+B u) - u'Q u  \,\,\,\, with \,\,\,\, u = -(Q + B'PB)^{-1}B'PAx $$

To evaluate the function

\begin{align}
v(x) &=  -(Ax+ B u)'P(Ax+Bu) - u'Q u \\
&= -(x'A' + u'B')P(Ax+Bu) - u'Q u \\
&= - x'A'PAx - u'B'PAx - x'A'PBu - u'B'PBu - u'Qu \\
&= - x'A'PAx - 2u'B'PAx - u'(Q + B'PB) u
\end{align}

For simplicity, denote by $S := (Q + B'PB)^{-1} B'PA$,
then $ u = -Sx$.

Regarding the second term $- 2u'B'PAx$,

\begin{align}
- 2u'B'PAx &= -2 x'S'B'PAx  \\
& = 2 x'A'PB( Q + B'PB)^{-1} B'PAx
\end{align}
Notice that the term $(Q + B'PB)^{-1}$ is symmetic as both P and Q are symmetric.

Regarding the third term $- u'(Q + B'PB) u$,

\begin{align}
- u'(Q + B'PB) u &= - x'S' (Q + B'PB)Sx \\
&= -x'A'PB(Q + B'PB)^{-1}B'PAx
\end{align}

Hence, the summation of second and third terms is $x'A'PB(Q + B'PB)^{-1}B'PAx$.

This implies that

\begin{align}
 v(x) &= - x'A'PAx - 2u'B'PAx - u'(Q + B'PB) u\\
 &= - x'A'PAx + x'A'PB(Q + B'PB)^{-1}B'PAx \\
 &= -x'[A'PA - A'PB(Q + B'PB)^{-1}B'PA] x
\end{align}

Therefore, the solution to the optimization problem $v(x) = -x' \tilde{P}x$ follows the above result by denoting $\tilde{P} := A'PA - A'PB(Q + B'PB)^{-1}B'PA$.

```python

```


**Exercise set 6**
==============

>In connection with experimental design, we have seen two approaches
>for checking if determined effects are important or not. These two
>approaches are based on creating a probability plot and performing ANOVA.
>The goal of this exercise is to learn how we use these two approaches in
>practice.

**Exercise 6.1**

In this part of the exercise, we will deal with the technical aspects of
creating a normal probability plot. Our final aim is here
to check if the data for some measured quantities (given in
the data files:
[data1.txt](Data/data1.txt) (located at `Data/data1.txt`),
[data2.txt](Data/data2.txt) (located at `Data/data2.txt`),
[data3.txt](Data/data3.txt) (located at `Data/data3.txt`), and
[data4.txt](Data/data4.txt) (located at `Data/data4.txt`))
comes
from a normal distribution.

**(a)**  Before we begin creating normal probability plots, we should
inspect the raw data. Plot histograms for the raw data. Based
on this, would you say that any of the data files contain numbers
that might come from a normal distribution?
You can also, for each data set, try to compare directly with
a normal distribution. For each data set, you can obtain
the mean and standard deviation, and you can plot
a normal distribution with these values in the same figure
where you have the histograms.

```python
# Your code here
```

**Your answer to question 6.1(a):** *Double click here*

**(b)**  For creating the normal probability plot, we need to be able
to obtain certain parameters for the standard normal distribution.
Here, we will investigate some of the functions that can give us
such parameters.

The standard normal probability density function ($\operatorname{PDF}$) is given by,

$\operatorname{PDF}(x) = \frac{1}{\sqrt{2 \pi}} \operatorname{e}^{-\tfrac{x^2}{2}},$

and the cumulative distribution function (CDF) is,

$
\operatorname{CDF}(x) = \int_{-\infty}^{x} \operatorname{PDF} (t)\,\operatorname{d}t =
\frac{1}{2} \left[ 1 + \operatorname{erf} \left( \frac{x}{\sqrt{2}} \right) \right],
$

where $\operatorname{erf}(\ldots)$ is the error function.
The cumulative distribution function gives the probability of observing a
value less than or equal to $x$: $P(X \leq x) = \operatorname{CDF}(x)$.

We can also turn this equation around: Given a probability $P$ what is
the value of $x$ that gives this probability? To answer this question,
we need the quantile function (also known as the percent-point function),
which is the inverse of the cumulative distribution function.
For the standard normal distribution, the percent-point
function ($\operatorname{PPF}$),
is given by,

$
\operatorname{PPF}(P) = \sqrt{2} \operatorname{erf}^{-1}(2P - 1).
$

If we make use of the `scipy` package, all these functions
are available to us:
```python
from scipy.stats import norm
import numpy as np

x = np.linspace(-2, 2, 100)
pdf = norm.pdf(x)  # Probability density function.
cdf = norm.cdf(x)  # Cumulative probability density function.
ppf = norm.ppf(x)  # Quantile function/percent-point function.
```

Use these methods to answer the following questions for the standard normal distribution:

* (i)  What is the probability of observing a $x \leq 1$?

* (ii)  What is the probability of observing a $x \leq 0$?

* (iii)  What is the probability of observing a $x \leq -2$?

* (iv)  Given that the probability of observing $x \leq \alpha$ is
$10$\%, what is $\alpha$?

* (v)  Given that the probability of observing $x \leq \alpha$ is
$90$\%, what is $\alpha$?

* (vi)  Given that the probability of observing $x \leq \alpha$ is
$99$\%, what is $\alpha$?

```python
# Your code here
```

**Your answer to question 6.1(b):** *Double click here*

**(c)**  To construct the normal probability plot, we will make use of
the $\operatorname{PPF}$.
If the data we are to investigate contain $n$ points, then
we need to figure out how these $n$ points would be placed
in the distribution we are going to compare with (here: the standard normal distribution).
In the following, we will call the $n$ points we have measured
for $y_1$, $y_2$, $\ldots$, $y_n$, and we assume that we have
sorted them so that $y_1 \leq y_2 \leq \ldots \leq y_n$.

We now need to check
how $n$ points drawn from a normal distribution would
be distributed, and compare this with how our measured data is distributed.
One way of doing that is to find the most probable location ($x_1$) of the
smallest value, the most probable location ($x_2$) of the second smallest
value, and so on, up to the most probable location ($x_n$) for the
largest value. There is no simple formula for finding $x_i$ and we have
to rely on a result from statistics:  These locations, the so-called
order statistic medians, for the normal distribution are exactly
related to order statistic medians from a *uniform distribution*, $m_i$, by,

\begin{equation}
x_i = \operatorname{PPF}(m_i) .
\label{eq:orderstat}
\tag{1}
\end{equation}

Thus we can potentially find $x_i$ by first obtaining the corresponding
$m_i$. Unfortunately, no analytical expression for $m_i$ exist and
we have to rely on approximate estimates. One such approximation
was
suggested by [Filliben](https://doi.org/10.1080/00401706.1975.10489279),

\begin{equation}
m_i =
\begin{cases}
1 - 0.5^{1/n} & \text{if } i = 1, \\
\frac{i - 0.3175}{n + 0.365} & \text{if } i = 2, 3, \ldots, n-1, \\
0.5^{1/n} & \text{if } i=n,
\end{cases}
\label{eq:uniformorderstat}
\tag{2}
\end{equation}

Thus, in summary, to create the normal probability plot we do the following:

* (i)  We sort our original data ($y_1$, $y_2$, $\ldots$, $y_n$).

* (ii)  For each sorted data point, we calculate its
uniform order statistic median, $m_i$,
using Eq. \eqref{eq:uniformorderstat}.

* (iii)  For each sorted data point, we calculate its most
probable location, $x_i$, in a normal distribution using
Eq. \eqref{eq:orderstat} and
the $m_i$ value we found in the previous step.

* (iv)  We plot the sorted data against the most probable locations
found in the previous step. That is, we plot the pairs ($x_i$, $y_i$),
and if the data is from a normal distribution, we expect that these
points fall on a straight line.

Create the normal probability plots for the four data sets given
in [data1.txt](Data/data1.txt) (located at `Data/data1.txt`),
[data2.txt](Data/data2.txt) (located at `Data/data2.txt`),
[data3.txt](Data/data3.txt) (located at `Data/data3.txt`), and
[data4.txt](Data/data4.txt) (located at `Data/data4.txt`). Which
of these would you say are numbers that could originate from a
normal distribution?

```python
# Your code here
```

**Your answer to question 6.1(c):** *Double click here*

**(d)**  The method we have described above works for any distribution,
not just the normal distribution. We can create similar plots
for other distributions by changing the $\operatorname{PPF}$ function
in Eq. [(1)](#mjx-eqn-eq:orderstat)
to the corresponding function for the distribution we wish to check for.
Repeat the previous step, but use the `Gumbel distribution`
(in `scipy` this is available by
`from scipy.stats import gumbel_r`) in place of the
normal distribution. Based on the plots you now create, would you say
that any of the data sets may contain numbers from a Gumbel distribution?

```python
# Your code here
```

**Your answer to question 6.1(d):** *Double click here*

**Exercise 6.2**

After running a set of experiments, you determine the effects
given in Table 1 for $4$ factors: A, B, C, and D.
Use a normal probability plot to identify the important effects among
the ones listed in this table. (Note: These numbers were also used in lecture $6$.)

|**Factor** | **Effect** |
|:---------:|:----------:|
|A          |  -8.00     |
|B          |  24.00     |
|C          |  -2.25     |
|D          |  -5.50     |
|AB         |   1.00     |
|AC         |   0.75     |
|AD         |   0.00     |
|BC         |  -1.25     |
|BD         |   4.50     |
|CD         |  -0.25     |
|ABC        |  -0.75     |
|ABD        |   0.50     |
|ACD        |  -0.25     |
|BCD        |  -0.75     |
|ABCD       |  -0.25     |

| |
|---|
|**Table 1:** *Effects determined in a set of experiments.*|

```python
# Your code here
```

**Your answer to question 6.2:** *Double click here*

**Exercise 6.3**

From a $2^2$ factorial experiment replicated three times you have obtained
the data given in Table 3. We use here a short-hand notation
for the $4$ possible combinations of the variables: $(1)$, $a$, $b$, and $ab$.
In this notation $(1)$ is the experiment where all factors were at their low levels. For the
other cases, the absence of a letter means that the corresponding factor was at a low level, and
the presence of a letter means that the corresponding factor was at a high level (e.g. "$a$" is the
same as saying that factor A was at the high level and B at the low level).

|**Experiment** | **Replicate 1** | **Replicate 2** | **Replicate 2** |
|:---:|:---:|:---:|:---:|
|$(1)$ | $9$  | $10$ | $11$ |
|$a$   | $30$ | $31$ | $29$ |
|$b$   | $19$ | $20$ | $21$ |
|$ab$  | $5$  | $6$  | $4$  |

| |
|---|
|**Table 2:** *Results from a $2^2$ factorial experiment, repeated $3$ times.*|

**(a)**  Calculate the effects (A, B, and AB).

```python
# Your code here
```

**Your answer to question 6.3(a):** *Double click here*

**(b)**  Use ANOVA to investigate which effects are important in this case.
Use a significance level of $\alpha = 0.01$. For
a significance level of $\alpha = 0.01$, the relevant critical
$f$-value is $f_{\alpha=0.01}(1, 8) = 11.259$ with $1$ and $8$ degrees
of freedom. (Note: These numbers in Table 2
are the same as for the example on
page $96$ in the textbook.)

```python
# Your code here
```

**Your answer to question 6.3(b):** *Double click here*


# Partial Differential Equations

In our study of numerical methods, we start with the Euler equations. By investigating the mathematical form of these equations, we seek to simplify the Euler equations to arrive at a simplistic model equation we can start using to study the behavior of various numerical methods.

What do we mean by investigating the mathematical form of the Euler equations?

   - From the study of partial differential equations, we know that we can classify a partial differential equation according to its **order**, wether it is **linear** or **non-linear**, homogeneous or non-homogeneous, and by its type, wether it is **hyperbolic**, **parabolic**, or **elliptic**.

Suppose we have a second-order, partial differential equation with the following form:

$$
a(x,t) \frac{\partial^2 u}{\partial x^2} + b(x,t) \frac{\partial^2 u}{\partial x\partial t} +
    c(x,t) \frac{\partial^2 u}{\partial t^2} + d(x,t) \frac{\partial u}{\partial x} +
    e(x,t) \frac{\partial u}{\partial t} + f(x,t) u = g(x,t)
$$

The above partial differential equations is:

   - **elliptic** if $b^2 - 4ac < 0$
   - **parabolic** if $b^2 - 4ac = 0$
   - **hyperbolic** if $b^2 - 4ac > 0$

A full study of PDEs is beyond the scope of this course. What is important to know is that the characteristics of the PDE heavily influences our choice and construction of the numerical methods we use to find solutions to the PDE.

**The goal is therefore to find a "model" PDE for the Euler equations with similar type and order that we can use to begin our study of numerical methods.** The hope is to find a model equation that is simplier, but still representative of the Euler equations.

## Euler Equations

Before introducing a model equation for the Euler equations, let us review when the Euler equations are a good representation of a gas flow. The end goal is still to develop a model equation for the Euler equations, but since this set of equations is a simplification of the Navier-Stokes equations, we should know what terms we are ignoring in Euler equations. This is easily done by non-dimensionalizing the Navier-Stokes equations,

$$
\begin{align}
\textrm{Length} & : x^* = \frac{x}{L}  \\ \\
\textrm{Time} & : t^* = \frac{t U_0}{L} \\ \\
\textrm{Velocity} & : u^* = \frac{u}{U_0} \\ \\
\textrm{Pressure} & : p^* = \frac{p}{p_0} \\ \\
\textrm{Spatial derivative} & : \frac{\partial}{\partial x} = \frac{1}{L} \frac{\partial}{\partial x^*} \\ \\
\textrm{Time derivative} & : \frac{\partial}{\partial t} = \frac{U_0}{L} \frac{\partial}{\partial t^*}
\end{align}
$$

Using these definitions, substitute them into the incompressible form of the Navier-Stokes equations:

$$
\begin{align}
& \frac{U_0}{L} \left( \frac{\partial u^*}{\partial x^*} + \frac{\partial v^*}{\partial y^*}  + \frac{\partial w^*}{\partial z^*} \right) = 0 \\
&\frac{U_0^2}{L} \frac{\partial u^*} {\partial t^*} + \frac{U_0^2}{L} \left(u^* \frac{\partial u^*}{\partial x^*} +
                                                                           v^* \frac{\partial u^*}{\partial y^*} +
                                                                           w^* \frac{\partial u^*}{\partial z^*} \right) = -\frac{p_0}{\rho U_0^2} \frac{\partial p^*}{\partial x^*} + \frac{\nu}{U_0 L} \left( \frac{\partial^2 u^*}{\partial x^{*2}}  +  \frac{\partial^2 v^*}{\partial y^{*2}} +  \frac{\partial^2 w^*}{\partial z^{*2}} \right)
\end{align}
$$

Removing the $()^*$ notation and reverting back to the vector form, we get

$$
\begin{align}
& \nabla \cdot \mathbf{u} = 0 \\
& \frac{\partial \mathbf{u}} {\partial t} + \mathbf{u} \cdot \nabla \mathbf{u} = -\frac{p_0}{\rho U_0^2} \nabla p + \frac{\nu}{U_0 L} \nabla^2 \mathbf{u}
\end{align}
$$

From inspection, we can see that a convenient definition for the pressure scale would be to use $p_0 = \rho U_0^2$. Also, the non-dimensional term in front of the viscous term is defined as the Reynolds number,

$$
\textrm{Re} = \frac{\textrm{inertial forces}}{\textrm{viscous forces}} = \frac{ U_0^2 / L }{ \nu U_0 / L^2 } = \frac{U_0 L}{\nu}
$$

The non-dimensional form of the incompressible Navier-Stokes equations is

$$
\begin{align}
& \frac{\partial \mathbf{u}} {\partial t} + \mathbf{u} \cdot \nabla \mathbf{u} = -\nabla p + \frac{1}{\textrm{Re}} \nabla^2 \mathbf{u}
\end{align}
$$

Defining the **material derivative** as

$$
\begin{align}
\frac{\textrm{D}\mathbf{u}}{\textrm{D} t} & = \frac{\partial \mathbf{u}}{\partial t} + \mathbf{u} \cdot \nabla \mathbf{u} \\
\textrm{Lagrangian Acceleration} & = \textrm{Eulerian Acceleration} + \textrm{Convective Acceleration}
\end{align}
$$

WE can write the Euler equations as a set of partial differential equations describing the conservation of mass, momentum, and energy:

$$
\begin{align}
\textrm{mass} & : \frac{\textrm{D} \rho}{\textrm{D} t} = -\rho \nabla \cdot \mathbf{u} \\
\textrm{momentum} & :\frac{\textrm{D} \mathbf{u}}{\textrm{D} t} = -\frac{\nabla \mathbf{p}}{\rho} \\
\textrm{energy} & : \frac{\textrm{D} e}{\textrm{D} t} = -\frac{p}{\rho} \nabla \cdot \mathbf{u}
\end{align}
$$

and we see that in the limit that the Reynolds number goes to infinity, the Navier-Stokes equations simplify to the Euler equations. When the Reynolds number is large, we the affect of the viscous terms to be contained to a small region of the flow near the walls, e.g., the  boundary layer.

We should observe that each PDE in this set of conservation equation is similar. Let's focus on the momentum equation, which we can write as:

$$
\frac{\textrm{D} \mathbf{u}}{\textrm{D} t} = \frac{\partial \mathbf{u}}{\partial t} + \mathbf{u} \cdot \nabla \mathbf{u} = -\frac{\nabla \mathbf{p}}{\rho}
$$

In one-dimensional where the velocity vector, $\mathbf{u}(\mathbf{x},t) = u(x,t)$, we have

$$
\frac{\partial u}{\partial t} + u \frac{\partial u }{\partial x} = -\frac{\nabla \mathbf{p}}{\rho}
$$

where the x-velocity, $u$, is a function of the time, $t$, and the Cartesian coordinate, $x$. If we assume the pressure gradient is zero (*not the best assumption, but useful*), then we get the non-linear PDE,

$$
\frac{\partial u}{\partial t} + u \frac{\partial u }{\partial x} = 0
$$

To generalize this PDE, let us write the following

$$
\frac{\partial u}{\partial t} + c(u) \frac{\partial u }{\partial x} = 0
$$

where we have assume the coefficient, $c$ is a function of the variable $u$. Note that at this point, the variable $u(x,t)$ does not necessarily need to represent the velocity. Both the mass and energy conservation equations can be simplified in a similar manner. We previously stated that we can classify a PDE as **hyperbolic**, **parabolic**, or **elliptic**. The PDE described above is a non-linear, hyperbolic PDE. It is non-linear because the coefficient, $c$, is a function of $u$. Let us simplify this even further and assume that $c$ is constant.

## Hyperbolic model equation

If $c$ is constant, and let us assume $c > 0$, the we arrive at the model hyperbolic PDE. This equation is often referred to as the linear convection (or advection) PDE,

$$
\frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} = 0
$$

In order to find a solution to this PDE, we must define the initial condition for $u(x,t)$ at $t=t_0$ as well as the boundary condition at $x=0$. Let the initial condition be defined as

$$
u(x,t_0) = g(x)
$$

and the boundary condition at $x=0$,

$$
u(0,t) = u_0
$$

There is an exact solution to this PDE, which can be found using the method of characteristics. If $c > 0$, then we only need the boundary condition at $x=0$. *(Note that if $c < 0$, then we need the boundary condition at $x=L$.)* Note that the boundary condition is informed by the initial condition; as a result, we often call solutions to linear convection PDE as solutions to the initial value problem.

The method of characteristics can be used to find an analytical solution to the
above equation. Along a characteristic line, we have

$$
\frac{\textrm{d}x}{\textrm{d} t} = c.
$$

As a result, along a particular curve $x = x(t)$, we have

$$
\frac{\textrm{d}}{\textrm{d} t}\Big[u(x(t),t)\Big] = \frac{\partial u}{\partial t} + \frac{\textrm{d} x}{\textrm{d} t} \frac{\partial u}{\partial x}
 =  \frac{\partial u}{\partial t} + c \frac{\partial u}{\partial x} .
$$

The value $u$ is therefore constant along the characteristics (the material derivative $du/dt$ is zero, thus u is constant along lines $x-t$ lines).
We also know that these characteristic lines are straight, since the curvature (second-derivative) of the characteristics is zero,

$$
\frac{\textrm{d}^2 x}{\textrm{d}t^2} = \frac{\textrm{d}}{\textrm{d}t}\left(\frac{\textrm{d} x}{\textrm{d}t}\right) = \frac{\textrm{d}c}{\textrm{d}t} = 0
$$

Knowing that the value of $u$ is constant along a characteristic, we can define a series of parametric lines that only depend on the
initial value of $u$ at a point emanating from  $(\eta,0)$. Direct integration of

$$
\frac{\textrm{d}x}{\textrm{d} t} = c.
$$

results in the characteristic line
$$
x = c t + \xi
$$

or

$$
\xi= x - c t
$$

where the solution of $u(x,t)$ of the initial value problem is given by the

$$
u(x,t) = u(\xi, 0) = u_0(\xi) = u_0(x - ct)
$$

The exact solution is

$$
u(x,t) = g(x - c(t - t_0))
$$

## Analytical Solutions to the Linear Convection PDE

Let us assume, $c = 1$, with an initial condition given by the Gaussian distribution,

$$
u(x,0) = g(x) = \frac{1}{\sigma \sqrt{2 \pi}} \exp\left( -\frac{(x - \mu)^2}{2\sigma^2}\right)
$$

where $\sigma$ is the standard deviation, and $\mu$ is the mean. The boundary condition is provided by the initial condition,

$$
u(0,t) = g(0)
$$

```python
import numpy as np
import matplotlib
import matplotlib.pyplot as plt

%matplotlib inline
```

```python
def gaussian(x, mu, sigma):
    '''
    Return a Gaussian distribution.

    Args:
        x (np.array) : Values of x
        mu (float) : Mean
        sigma (float) : Standard deviation
    Returns:
        f (np.array) : Gaussian distribution as a function of x
    '''
    f = np.zeros_like(x)
    for i in range(len(x)):
        f[i] = 1.0 / sigma / np.sqrt(2.0 * np.pi) * np.exp(- (x[i] - mu)**2 / 2 / sigma**2)
    return f
```

```python
# Define the mean and standard deviation of the distribution
mean = 0.0
std = 0.1

# Create a mesh of 100 points
x = np.linspace(-1.0,1.0, 100)

# Define the initial condition
u0 = gaussian(x, mean, std)

c = 1.0 # Wave speed
t = 0.5 # Time

# Compute the solution at t = 0.5
un = gaussian(x - c * t, mean, std)
```

```python
# Plot the solution

fig = plt.figure(figsize=(9,6))
ax = fig.add_subplot(111)

line = ax.plot(x, u0, lw=1, c='k', marker='o', ms=2, label=r't = 0')
line = ax.plot(x, un, lw=1, c='b', marker='o', ms=2, label=r't = 0.5 s')

ax.legend()
ax.set_xlim((-1,1))
```


# Characterization of Systems in the Time Domain

*This Jupyter notebook is part of a [collection of notebooks](../index.ipynb) in the bachelors module Signals and Systems, Communications Engineering, Universität Rostock. Please direct questions and suggestions to [Sascha.Spors@uni-rostock.de](mailto:Sascha.Spors@uni-rostock.de).*

## Analysis of a Damped Spring Pendulum

The damped [spring pendulum](https://en.wikipedia.org/wiki/Spring_pendulum) is an example for a mechanical system that can be modeled by a linear ordinary differential equation (ODE) with constant coefficients. In view of the theory of signals and systems it hence can be interpreted as a linear time-invariant (LTI) system. The mechanical properties of the damped spring pendulum are analyzed by using the theory of LTI systems. The underlying mechanical setup is depicted in the following

A compact mass $m$ is mounted on a spring with stiffness $k$ which is connected to the ground. A damper with viscous damping coefficient $c$ is mounted parallel to the spring to model the friction present in the system. It is assumed that the movement of the mass over time is restricted to the vertical axis, here denoted by $y$. It is further assumed that the mass is in its idle position for $t<0$. The pretension of the spring by the mass can be neglected this way. It is additionally assumed that the mass is not moving for $t<0$. Denoting the displacement of the mass over time with $y(t)$, these initial conditions are formulated as $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

The normalized values $m = 0.1$, $c = 0.1$, $k = 2.5$ are used for illustration in the following.

### Differential Equation

The differential equation of the mechanical system is derived by considering the force equilibrium at the mass

\begin{equation}
F_\text{S}(t) + F_\text{F}(t) + F_\text{I}(t) = F_\text{E}(t)
\end{equation}

where $F_\text{E}(t)$ denotes an external force acting onto the mass, the other forces are derived in the sequel. The force $F_\text{S}(t)$ induced by the spring is given by [Hooke's law](https://en.wikipedia.org/wiki/Hooke%27s_law)

\begin{equation}
F_\text{S}(t) = k y(t)
\end{equation}

Its common to model the frictional force $F_\text{F}(t)$ induced by the damper as being proportional to the velocity of the mass

\begin{equation}
F_\text{F}(t) = c \frac{d y(t)}{dt}
\end{equation}

The inertial force $F_\text{I}(t)$ due to the acceleration of the mass is given as

\begin{equation}
F_\text{I}(t) = m \frac{d^2 y(t)}{dt^2}
\end{equation}

Introducing the forces into the force equilibrium yields the differential equation describing the displacement of the damped spring pendulum

\begin{equation}
m \frac{d^2 y(t)}{dt^2} + c \frac{d y(t)}{dt} + k y(t) = F_\text{E}(t)
\end{equation}

as a consequence of the external force.
Above equation constitutes an ODE with constant coefficients. It can be interpreted as an LTI system with the external force as input signal $x(t) = F_\text{E}(t)$ and the displacement of the mass as output signal $y(t)$.

### Comparison to Passive Electrical Networks

Comparing the ODEs of the damped spring pendulum and the [second-order analog low-pass](http://localhost:8888/notebooks/systems_time_domain/network_analysis.ipynb#Differential-Equation) yields that both constitute second-order ODEs with constant coefficients. Dividing the ODE of the second-order analog low pass by $C$ results in

\begin{equation}
L \frac{d^2 u_\text{o}(t)}{dt^2} + R \frac{d u_\text{o}(t)}{dt} + \frac{1}{C} u_\text{o}(t) = \frac{1}{C} u_\text{i}(t)
\end{equation}

where $u_\text{i}(t)$ and $u_\text{o}(t)$ denote the in- and output voltage of the analog circuit.
Comparison with above ODE of the spring pendulum yields the [equivalence of both systems](https://en.wikipedia.org/wiki/System_equivalence) for

| | 2nd-order low-pass | spring pendulum |
|:---|:---|:---|
| input signal $x(t)$ | $u_\text{i}(t) = F_\text{E}(t) C$ | $F_\text{E}(t) = \frac{u_\text{i}(t)}{C}$ |
| output signal $y(t)$ | $u_\text{o}(t)$ | $y(t)$ |
|  | $L = m$ | $m = L$ |
|  | $R = c$ | $c = R$ |
|  | $C = \frac{1}{k}$ | $k = \frac{1}{C}$ |

Note, the equivalence between mechanical systems described by ODEs with constant coefficients and analog circuits was used to simulate such systems by [analog computers](https://en.wikipedia.org/wiki/Analog_computer).

### Impulse Response

The LTI system corresponding to the pendulum can be characterized by its [impulse response](impulse_response.ipynb) $h(t)$. It is defined as the output of the system for a Dirac Delta impulse $x(t) = \delta(t)$ at the input. Physically this can be approximated by hitting the mass (very shortly and forceful). The impulse response characterizes the movement $y(t)$ of the mass after such an event.

First the ODE of the spring pendulum is defined in `SymPy`

```python
import sympy as sym
sym.init_printing()

t, m, c, k = sym.symbols('t m c k', real=True)
x = sym.Function('x')(t)
y = sym.Function('y')(t)

ode = sym.Eq(m*y.diff(t, 2) + c*y.diff(t) + k*y, x)
ode
```

The normalized values of the physical constants are stored in a dictionary for ease of later substitution

```python
mck = {m: 0.1, c: sym.Rational('.1'), k: sym.Rational('2.5')}
mck
```

The impulse response is calculated by explicit solution of the ODE.

```python
solution_h = sym.dsolve(
    ode.subs(x, sym.DiracDelta(t)).subs(y, sym.Function('h')(t)))
solution_h
```

The integration constants $C_1$ and $C_2$ have to be determined from the initial conditions $y(t) = 0$ and $\frac{d y(t)}{dt} = 0$ for $t<0$.

```python
integration_constants = sym.solve((solution_h.rhs.limit(
    t, 0, '-'), solution_h.rhs.diff(t).limit(t, 0, '-')), ['C1', 'C2'])
integration_constants
```

Substitution of the values for the integration constants $C_1$ and $C_2$ into the result from above yields the impulse response of the spring pendulum

```python
h = solution_h.subs(integration_constants)
h
```

The impulse response is plotted for the specific values of $m$, $c$ and $k$ given above

```python
sym.plot(h.rhs.subs(mck), (t, 0, 12), ylabel=r'h(t)');
```

### Transfer Function

For an exponential input signal $x(t) = e^{s t}$, the [transfer function](eigenfunctions.ipynb#Transfer-Function) $H(s)$ represents the weight of the exponential output signal $y(t) = H(s) \cdot e^{s t}$. The transfer function is derived by introducing $x(t)$ and $y(t)$ into the ODE and solving for $H(s)$

```python
s = sym.symbols('s')
H = sym.Function('H')(s)

H, = sym.solve(ode.subs(x, sym.exp(s*t)).subs(y, H*sym.exp(s*t)).doit(), H)
H
```

The transfer characteristic of an LTI system for harmonic exponential signals $e^{j \omega t} = \cos(\omega t) + j \sin(\omega t)$ is of special interest in the analysis of resonating systems. It can be derived from $H(s)$ by substituting the complex frequency $s$ with $s = j \omega$. The resulting transfer function $H(j \omega)$ provides the attenuation and phase the system adds to an harmonic input signal.

```python
w = sym.symbols('omega', real=True)

Hjw = H.subs(s, sym.I * w)
Hjw
```

The magnitude of the transfer function $|H(j \omega)|$ is plotted for the specific values of the elements given above

```python
sym.plot(abs(Hjw.subs(mck)), (w, -15, 15),
         ylabel=r'$|H(j \omega)|$', xlabel=r'$\omega$');
```

When inspecting the magnitude of the transfer function it becomes evident that the damped spring pendulum shows resonances (maxima) for two specific angular frequencies. These resonance frequencies $\omega_0$ are calculated by inspecting the extreme values of $|H(j \omega)|$. First the derivative of $|H(j \omega)|$ with respect to $\omega$ is computed and set to zero

```python
extrema = sym.solve(sym.Eq(sym.diff(abs(Hjw), w), 0), w)
extrema
```

For the maxima of the transfer function only the 2nd and 3rd extrema are of interest

```python
w0 = extrema[1:3]
w0
```

The resonance frequencies are computed for the specific values of $m$, $c$ and $k$ given above

```python
[w00.subs(mck) for w00 in w0]
```

The phase of the transfer function $\varphi(j \omega)$ is computed and plotted for the specific values of the elements given above

```python
phi = sym.arg(Hjw)
sym.plot(phi.subs(mck), (w, -15, 15),
         ylabel=r'$\varphi(j \omega)$', xlabel=r'$\omega$');
```

**Exercise**

* Change the viscous damping coefficient $c$ of the spring pendulum and investigate how the magnitude and phase of the transfer function $H(j \omega)$ changes.

* How does the frequency of the damped harmonic oscillation in the impulse response relates to the resonance frequency?

### Application: Vibration Isolation

An application of above example is the design of [vibration isolation](https://en.wikipedia.org/wiki/Vibration_isolation) by a damped spring pendulum. A typical example is a rotating machinery with mass $m$ which has some sort of imbalance. Assuming that the imbalance can be modeled as a rotating mass, the external force $F_\text{E}(t)$ is given by the vertical component of its [centrifugal force](https://en.wikipedia.org/wiki/Centrifugal_force)

\begin{equation}
F_\text{E}(t) = F_0 \sin(\omega t) = F_0 \cdot \Im \{e^{j \omega t} \}
\end{equation}

where $\omega$ denotes the angular frequency of the rotating machinery and

\begin{equation}
F_0 = m_\text{I} r \omega^2
\end{equation}

the amplitude of the force with $m_\text{I}$ denoting the mass of the imbalance and $r$ the radius of its circular orbit. Since $e^{j \omega t}$ is an eigenfunction of the LTI system, the resulting displacement is then given as

\begin{equation}
y(t) = F_0 \cdot \Im \{e^{j \omega t} H(j \omega) \}
\end{equation}

The aim of vibration isolation is to keep the magnitude of the displacement as low as possible.

**Exercise**

* Compute and plot the displacement for given $m_\text{I}$ and $r$.

* For which angular frequencies $\omega$ is the magnitude of the displacement largest? How is the phase relation between the external force $F_\text{E}(t)$ and displacement $y(t)$ at these frequencies?

* How should the resonance frequencies $\omega_0$ of the spring pendulum be chosen in order to get a good vibration isolation for a machine rotating with angular frequency $\omega$? How is the phase relation between the external force $F_\text{E}(t)$ and displacement $y(t)$ at this frequency?

**Copyright**

This notebook is provided as [Open Educational Resource](https://en.wikipedia.org/wiki/Open_educational_resources). Feel free to use the notebook for your own purposes. The text is licensed under [Creative Commons Attribution 4.0](https://creativecommons.org/licenses/by/4.0/), the code of the IPython examples under the [MIT license](https://opensource.org/licenses/MIT). Please attribute the work as follows: *Sascha Spors, Continuous- and Discrete-Time Signals and Systems - Theory and Computational Examples*.


# Solvers

Boilerplate to make the doctester work.

```
import sys
import os
sys.path.insert(1, os.path.join(os.path.pardir, "ipython_doctester"))
from sympy import *
from ipython_doctester import test
# Work around a bug in IPython. This will disable the ability to paste things with >>>
def notransform(line): return line
from IPython.core import inputsplitter
inputsplitter.transform_classic_prompt = notransform
init_printing()
```

For each exercise, fill in the function according to its docstring. Execute the cell to see if you did it right.

```
a, b, c, d, x, y, z, t = symbols('a b c d x y z t')
f, g, h = symbols('f g h', cls=Function)
```

## Algebraic Equations

Write a function that computes the [quadratic equation](http://en.wikipedia.org/wiki/Quadratic_equation).

```
def quadratic():
    return solve(a*x**2 + b*x + c, x)
quadratic()
```

$$\begin{bmatrix}\frac{- b + \sqrt{- 4 a c + b^{2}}}{2 a}, & - \frac{b + \sqrt{- 4 a c + b^{2}}}{2 a}\end{bmatrix}$$

Write a function that computes the general solution to the cubic $x^3 + ax^2 + bx + c$.

```
def cubic():
    return solve(x**3 + a*x**2 + b*x + c, x)
cubic()
```

$$\begin{bmatrix}- \frac{1}{3} a + \frac{- \frac{1}{9} a^{2} + \frac{1}{3} b}{\sqrt[3]{\frac{1}{27} a^{3} - \frac{1}{6} a b + \frac{1}{2} c + \sqrt{\left(- \frac{1}{9} a^{2} + \frac{1}{3} b\right)^{3} + \frac{1}{4} \left(\frac{2}{27} a^{3} - \frac{1}{3} a b + c\right)^{2}}}} - \sqrt[3]{\frac{1}{27} a^{3} - \frac{1}{6} a b + \frac{1}{2} c + \sqrt{\left(- \frac{1}{9} a^{2} + \frac{1}{3} b\right)^{3} + \frac{1}{4} \left(\frac{2}{27} a^{3} - \frac{1}{3} a b + c\right)^{2}}}, & - \frac{1}{3} a + \frac{- \frac{1}{9} a^{2} + \frac{1}{3} b}{\left(- \frac{1}{2} - \frac{1}{2} \sqrt{3} \mathbf{\imath}\right) \sqrt[3]{\frac{1}{27} a^{3} - \frac{1}{6} a b + \frac{1}{2} c + \sqrt{\left(- \frac{1}{9} a^{2} + \frac{1}{3} b\right)^{3} + \frac{1}{4} \left(\frac{2}{27} a^{3} - \frac{1}{3} a b + c\right)^{2}}}} - \left(- \frac{1}{2} - \frac{1}{2} \sqrt{3} \mathbf{\imath}\right) \sqrt[3]{\frac{1}{27} a^{3} - \frac{1}{6} a b + \frac{1}{2} c + \sqrt{\left(- \frac{1}{9} a^{2} + \frac{1}{3} b\right)^{3} + \frac{1}{4} \left(\frac{2}{27} a^{3} - \frac{1}{3} a b + c\right)^{2}}}, & - \frac{1}{3} a + \frac{- \frac{1}{9} a^{2} + \frac{1}{3} b}{\left(- \frac{1}{2} + \frac{1}{2} \sqrt{3} \mathbf{\imath}\right) \sqrt[3]{\frac{1}{27} a^{3} - \frac{1}{6} a b + \frac{1}{2} c + \sqrt{\left(- \frac{1}{9} a^{2} + \frac{1}{3} b\right)^{3} + \frac{1}{4} \left(\frac{2}{27} a^{3} - \frac{1}{3} a b + c\right)^{2}}}} - \left(- \frac{1}{2} + \frac{1}{2} \sqrt{3} \mathbf{\imath}\right) \sqrt[3]{\frac{1}{27} a^{3} - \frac{1}{6} a b + \frac{1}{2} c + \sqrt{\left(- \frac{1}{9} a^{2} + \frac{1}{3} b\right)^{3} + \frac{1}{4} \left(\frac{2}{27} a^{3} - \frac{1}{3} a b + c\right)^{2}}}\end{bmatrix}$$

## Differential Equations

A population that grows without bound is modeled by the differential equation

$$f'(t)=af(t)$$

Solve this differential equation using SymPy.

```
dsolve(f(t).diff(t) - a*f(t), f(t))
```

If the population growth is bounded, it is modeled by

$$f'(t) = f(t)(1 - f(t))$$

```
dsolve(f(t).diff(t) - f(t)*(1 - f(t)), f(t))
```


0 to 9 pandigital number (e.g. $1406357289$)

$
\begin{align}
d_2 d_3 d_4=406 \ &\vdots \ 2 \\
d_3 d_4 d_5=063 \ &\vdots \ 3 \\
d_4 d_5 d_6=635 \ &\vdots \ 5 \\
d_5 d_6 d_7=357 \ &\vdots \ 7 \\
d_6 d_7 d_8=572 \ &\vdots \ 11 \\
d_7 d_8 d_9=728 \ &\vdots \ 13 \\
d_8 d_9 d_{10}=289 \ &\vdots \ 17
\end{align}
$

find the sum of all 0 to 9 pandigital numbers with this property

```python
%%time

from itertools import permutations

nums = []

for d in permutations(range(10)):
    if d[0] == 0: continue
    if d[5] not in {0, 5}: continue
    if d[3] not in {0, 2, 4, 6, 8}: continue
    if (d[2]*100 + d[3]*10 + d[4]) %  3 > 0: continue
    if (d[4]*100 + d[5]*10 + d[6]) %  7 > 0: continue
    if (d[5]*100 + d[6]*10 + d[7]) % 11 > 0: continue
    if (d[6]*100 + d[7]*10 + d[8]) % 13 > 0: continue
    if (d[7]*100 + d[8]*10 + d[9]) % 17 > 0: continue

    nums += [int(''.join(map(str, d)))]

print(nums)
print(f'sum: {sum(nums)}')
```


# K-Nearest Neighbors Algorithm

* Last class, we introduced the probabilistic generative classifier.
* As discussed, the probabilistic generative classifier requires us to assume a parametric form for each class (e.g., each class is represented by a multi-variate Gaussian distribution, etc..).  Because of this, the probabilistic generative classifier is a *parametric* approach
* Parametric approaches have the drawback that the functional parametric form needs to be decided/assumed in advance and, if chosen poorly, might be a poor model of the distribution that generates the data resulting in poor performance.
* Non-parametric methods are those that do not assume a particular generating distribution for the data.  The $K$-nearest neighbors algorithm is one example of a non-parametric classifier.

* Nearest neighbor methods compare test point to the $k$ nearest training data points and then estimate an output value based on the desired/true output values of the $k$ nearest training points
* Essentially, there is no ``training'' other than storing the training data points and their desired outputs
* In test, you need to: (1) determine which $k$ training data points are closest to the test point; and (2) determine the output value for the test point
* In order to find the $k$ nearest neighbors in the training data, you need to define a *similarity measure* or a *dissimilarity measure*.  The most common dissimilarity measure is Euclidean distance.
    * Euclidean distance: $d_E = \sqrt{\left(\mathbf{x}_1-\mathbf{x}_2\right)^T\left(\mathbf{x}_1-\mathbf{x}_2\right)}$
    * City block distance: $d_C = \sum_{i=1}^d \left| x_{1i} - x_{2i} \right|$
    * Mahalanobis distance: $\left(\mathbf{x}_1-\mathbf{x}_2\right)^T\Sigma^{-1}\left(\mathbf{x}_1-\mathbf{x}_2\right)$
    * Geodesic distance
    * Cosine angle similarity: $\cos \theta = \frac{\mathbf{x}_1^T\mathbf{x}_2}{\left\|\mathbf{x}_1\right\|_2\left\|\mathbf{x}_2\right\|_2}$
    * and many more...

* If you are doing classification, once you find the $k$ nearest neighbors to your test point in the training data, then you can determine the class label of your test point using (most commonly) *majority vote*
* If there are ties, they can be broken randomly or using schemes like applying the label of the closest data point in the neighborhood
*  Of course, there are MANY modifications to you can make to this.  A common one is to weight the votes of each of the nearest neighbors by their distance/similarity measure value.  If they are closer, they get more weight.

```python
import numpy as np
from matplotlib import pyplot as plt
from matplotlib.colors import ListedColormap
from sklearn.model_selection import train_test_split
from sklearn.datasets import make_moons, make_circles, make_classification
from sklearn import neighbors
%matplotlib inline

#figure params
h = .02  # step size in the mesh
figure = plt.figure(figsize=(17, 9))

#set up classifiers
n_neighbors = 3
classifiers = []
classifiers.append(neighbors.KNeighborsClassifier(n_neighbors, weights='uniform'))
classifiers.append(neighbors.KNeighborsClassifier(n_neighbors, weights='distance'))
names = ['K-NN_Uniform', 'K-NN_Weighted']

#Put together datasets
n_samples = 300
X, y = make_classification(n_samples, n_features=2, n_redundant=0, n_informative=2,
                           random_state=0, n_clusters_per_class=1)
rng = np.random.RandomState(2)
X += 2 * rng.uniform(size=X.shape)
linearly_separable = (X, y)

datasets = [make_moons(n_samples, noise=0.3, random_state=0),
            make_circles(n_samples,  noise=0.2, factor=0.5, random_state=1),
            linearly_separable]

i = 1
# iterate over datasets
for X, y in datasets:
    # preprocess dataset, split into training and test part
    X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=.8)  #split into train/test folds

    #set up meshgrid for figure
    x_min, x_max = X[:, 0].min() - .5, X[:, 0].max() + .5
    y_min, y_max = X[:, 1].min() - .5, X[:, 1].max() + .5
    xx, yy = np.meshgrid(np.arange(x_min, x_max, h), np.arange(y_min, y_max, h))

    # just plot the dataset first
    cm = plt.cm.RdBu
    cm_bright = ListedColormap(['#FF0000', '#0000FF'])
    ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
    # Plot the training points
    ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
    # and testing points
    ax.scatter(X_test[:, 0], X_test[:, 1], marker='+', c=y_test, cmap=cm_bright, alpha=0.6)
    ax.set_xlim(xx.min(), xx.max())
    ax.set_ylim(yy.min(), yy.max())
    ax.set_xticks(())
    ax.set_yticks(())
    i += 1

    # iterate over classifiers
    for name, clf in zip(names, classifiers):
        ax = plt.subplot(len(datasets), len(classifiers) + 1, i)
        clf.fit(X_train, y_train)
        score = clf.score(X_test, y_test)

        # Plot the decision boundary. For that, we will assign a color to each
        # point in the mesh [x_min, x_max]x[y_min, y_max].
        Z = clf.predict(np.c_[xx.ravel(), yy.ravel()])

        # Put the result into a color plot
        Z = Z.reshape(xx.shape)
        ax.contourf(xx, yy, Z, cmap=cm, alpha=.8)

        # Plot also the training points
        ax.scatter(X_train[:, 0], X_train[:, 1], c=y_train, cmap=cm_bright)
        # and testing points
        ax.scatter(X_test[:, 0], X_test[:, 1], marker='+', c=y_test, cmap=cm_bright,
                   alpha=0.4)

        ax.set_xlim(xx.min(), xx.max())
        ax.set_ylim(yy.min(), yy.max())
        ax.set_xticks(())
        ax.set_yticks(())
        ax.set_title(name)
        ax.text(xx.max() - .3, yy.min() + .3, ('%.2f' % score).lstrip('0'),
                size=15, horizontalalignment='right')
        i += 1

figure.subplots_adjust(left=.02, right=.98)
plt.show()
```

# Error and Evaluation Metrics

* A key step in machine learning algorithm development and testing is determining a good error and evaluation metric.

* Evaluation metrics help us to estimate how well our model is trained and it is important to pick a metric that matches our overall goal for the system.

* Some common evaluation metrics include precision, recall, receiver operating curves, and confusion matrices.

### Classification Accuracy and Error

* Classification accuracy is defined as the number of correctly classified samples divided by all samples:

\begin{equation}
\text{accuracy} = \frac{N_{cor}}{N}
\end{equation}
where $N_{cor}$ is the number of correct classified samples and $N$ is the total number of samples.

* Classification error is defined as the number of incorrectly classified samples divided by all samples:

\begin{equation}
\text{error} = \frac{N_{mis}}{N}
\end{equation}
where $N_{mis}$ is the number of misclassified samples and $N$ is the total number of samples.

* Suppose there is a 3-class classification problem, in which we would like to classify each training sample (a fish) to one of the three classes (A = salmon or B = sea bass or C = cod).

* Let's assume there are 150 samples, including 50 salmon, 50 sea bass and 50 cod.  Suppose our model misclassifies 3 salmon, 2 sea bass and 4 cod.

* Prediction accuracy of our binary classification model is calculated as:

\begin{equation}
\text{accuracy} = \frac{47+48+46}{50+50+50} = \frac{47}{50}
\end{equation}

* Prediction error is calculated as:

\begin{equation}
\text{error} = \frac{N_{mis}}{N} = \frac{3+2+4}{50+50+50} = \frac{3}{50}
\end{equation}

### Confusion Matrices

* A confusion matrix summarizes the classification accuracy across several classes. It shows the ways in which our classification model is confused when it makes predictions, allowing visualization of the performance of our algorithm. Generally, each row represents the instances of a actual class while each column represents the instances in an predicted class.

* If our classifier is trained to distinguish between salmon, sea bass and cod. We can summarize the prediction result in the confusion matrix as follows:

| Actual/Predicted | Salmon | Sea bass | Cod  |
| --- | --- | --- | --- |
| Salmon | 47 | 2 | 1 |
| Sea Bass | 2 | 48 | 0 |
| Cod | 0 | 0 | 50 |

* In this confusion matrix, of the 50 actual salmon, the classifier predicted that 2 are sea bass, 1 is cod incorrectly and 47 are labeled salmon correctly. All correct predictions are located in the diagonal of the table. So it is easy to visually inspect the table for prediction errors, as they will be represented by values outside the diagonal.

### TP, FP, TN, and FN

* True positive (TP): correctly predicting event values
* False positive (FP): incorrectly calling non-events as an event
* True negative (TN): correctly predicting non-event values
* False negative (FN): incorrectly labeling events as non-event

* Precision is also called positive predictive value.

\begin{equation}
\text{Precision} = \frac{\text{TP}}{\text{TP}+\text{FP}}
\end{equation}

* Recall is also called true positive rate, probability of detection

\begin{equation}
\text{Recall} = \frac{\text{TP}}{\text{TP}+\text{FN}}
\end{equation}

* Fall-out is also called false positive rate, probability of false alarm.

\begin{equation}
\text{Fall-out} = \frac{\text{FP}}{\text{N}}= \frac{\text{FP}}{\text{FP}+\text{TN}}
\end{equation}

* *Consider the salmon/non-salmon classification problem, what are the TP, FP, TN, FN values?*

| Actual/Predicted | Salmon | Non-Salmon  |
| --- | --- | --- |
| Salmon | 47 | 3 |
| Non-Salmon | 2 | 98 |

### ROC curves

* The Receiver Operating Characteristic (ROC) curve is a plot between the true positive rate (TPR) and the false positive rate (FPR), where the TPR is defined on the $x$-axis and FPR is defined on the $y$-axis.

* $TPR = TP/(TP+FN)$ is defined as ratio between true positive prediction and all real positive samples. The definition used for $FPR$ in a ROC curve is often problem dependent.  For example, for detection of targets in an area, FPR may be defined as the ratio between the number of false alarms per unit area ($FA/m^2$).  In another example, if you have a set number of images and you are looking for targets in these collection of images, FPR may be defined as the number of false alarms per image.  In some cases, it may make the most sense to simply use the Fall-out or false positive rate.

* Given a binary classifier and its threshold, the (x,y) coordinates of ROC space can be calculated from all the prediction result.  You trace out a ROC curve by varying the threshold to get all of the points on the ROC.

* The diagonal between (0,0) and (1,1) separates the ROC space into two areas, which are left up area and right bottom area. The points above the diagonal represent good classification (better than random guess) which below the diagonal represent bad classification (worse than random guess).

* *What is the perfect prediction point in a ROC curve?*

### MSE and MAE

* *Mean Square Error* (MSE) is the average of the squared error between prediction and actual observation.

* For each sample $\mathbf{x}_i$, the prediction value is $y_i$ and the actual output is $d_i$. The MSE is

\begin{equation}
MSE = \sum_{i=1}^n \frac{(d_i - y_i)^2}{n}
\end{equation}

* *Root Mean Square Error* (RMSE) is simply the square root the MSE.

\begin{equation}
RMSE = \sqrt{MSE}
\end{equation}

* *Mean Absolute Error* (MAE) is the average of the absolute error.
\begin{equation}
MAE = \frac{1}{n} \sum_{i=1}^n \lvert d_i - y_i \rvert
\end{equation}

```python

```

Lecture07_KNN/Lecture 07 Non-parametric K-Nearest Neighbors Algorithm.ipynb


# Lecture 13:  Integral Transforms, D/FFT and Electron Microscopy
----

## Reading and Reference

* Advanced engineering Mathematics, E. Kreyszig, John wiley and Sons, 2010
* Numerical Recipes, W. Press, Cambridge University Press, 1986
* M. De Graef and M. McHenry, Structure of Materials, Cambridge University Press, 2nd ed.
* C. Hammond, The Basics of Crystallography and Diffraction, Oxford Science Publications, 4th ed.

## What to Learn?

* The definition of an integral transform
* The algorithm for computing the discrete Fourier transform
* How diffraction patterns can be used to create phase contrast images in electron microscopy

## What to Do?

* Compute the Fourier transform of different aperture functions.
* Select different regions of a Fourier transform to reconstruct a simulated TEM image and an image of your choosing.

### Introduction to Integral Transforms
----

An integral transform maps a function of one independent variable into a function of another independant variable using a _kernel_:

$$g(\alpha) = \int_{a}^{b} f(t) K(\alpha,t) dt $$

The function $f(t)$ is transformed to a new function $g(\alpha)$ through the definate integral.  A similarity to the dot product of functions is evident in this form and this operation can be thought of as a mapping or projection of $f(t)$ into a different independent variable $\alpha$.  Existence, integrability and inversion of integral transform operations are important in the study of this topic, although not covered in these notes.

Two examples of integral transforms, the Laplace and Fourier, are discussed in this lecture.  It is typical to use the Laplace transform to remove the time dependence from Fick's second law in diffusion problems.  The Fourier transform is used in the study of diffraction under certain conditions.

To assist in this lecture some special symbols in `Python` and `sympy` are reviewed:

```python
import sympy as sp
sp.init_printing(use_latex=True)
```

```python
# symbols we will need below
x,y,z,t,c = sp.symbols('x y z t c')
# note the special declaration that omega is a positive number
omega = sp.symbols('omega', positive=True)
```

###  Complex Number Review
----

A reminder that $i$ is the square root of negative one and this is how you specify $i$ in `Sympy` and that is different than the complex data type in `Python`.

```python
sp.I**2
```

The natural logarithm of $e$ is $1$:

```python
sp.log(sp.E)
```

In SymPy there are two ways to deal with integration.  If you would like to represent an unevaluated integral, you can use the `Integral` function.  If you want to compute the integration of an expression you can use the `integrate` function.

```python
sp.Integral(sp.E**(sp.I*omega*t),t)
```

```python
# 'omega', positive=True
sp.integrate(sp.E**(sp.I*omega*t),t)
```

Where we assume there is no zero frequency (as we are dividing by $\omega$) - hence the assumption `positive=True` in the symbol definition above.  (Try replacing $\omega$ with $y$ and inspect the expression returned by `integrate`.)

### The Fourier Transform
----

As the domain of the periodicity increases, the frequency spectrum required required to represent the function becomes more finely divided.  Recall the argument of the trigonometric terms in the functions of the Fourier series:

$$ \frac{n \pi (\omega +c)}{d} $$

where n is the order of the frequency component, c the offset relative to the origin, and d the domain width.  If we let the domain width go to infinity (implying that the function is not periodic) then an integral sum is required rather than a discrete summation.  The, infinte, non-periodic function and its frequency spectrum are related by the Fourier transform defined by:

$$ \hat{f}(\omega) = \sqrt{\frac{1}{2\pi}} \int^{+\infty}_{-\infty} f(t) \exp[-i \omega t] dt $$

This results in a mapping of the function f(t) into frequency space.

The real or complex and even or odd nature of the function $f(t)$ determines if the transformed function is even, odd, real, or complex.  For the purposes of materials crystal structures in this lecture we will be using even and real functions.

### Diffraction from An Aperture
----

A useful physical problem requiring use of the Fourier transform is diffraction.  In this problem we will use a top-hat function to represent the location of an infinity of wave sources from an aperture.  We use the `sp.Piecewise` function to generate a "tophat" function for the Fourier transform.

```python
%matplotlib inline
import matplotlib.pyplot as plt
plt.rcParams['figure.figsize'] = 8, 4

p = sp.Piecewise((0,x<-1),(1,x<1),(0,True))
sp.plot(p);
```

At some distance from the aperture we place a detector that measures the combined intensity of all the wave sources, however due to the finite width of the slit each wave travels a different distance to the detector.  The phase difference between the waves at the detector is given by the Fourier transform of the aperture function when the [Fraunhofer](https://en.wikipedia.org/wiki/Fraunhofer_diffraction_equation) approximation is valid.

This aperture function is even and real so we expect our transformed function to also be even and real.  We use the definition of the integral transform above to write an explicit integral statement of the Fourier transform of the top-hat function above.  The integral is $1$ between $c$ and $-c$ and zero elsewhere - so we can integrate **just the non-zero part**.  This is integrated as:

```python
sp.Integral(1*sp.exp(-sp.I*2*omega*x),(x,-c,c))
```

Calling explicitly for the integration and assigning the result to `a`:

```python
a = sp.sqrt(1/(2*sp.pi))*sp.integrate(1*sp.exp(-sp.I*2*omega*x),(x,-c,c))
a
```

This does not (at first glance) appear to be a real function due to the two exponential terms, but we can use some of the algebraic methods built into `SymPy` to help.  We can ask for this form using sines and cosines with the `rewrite` method.  Furthermore - we can simplify it further with the expand function.  Trial and error may be required to determine the best combination and ordering of algebraic manipulations.

```python
solution = sp.expand(a.rewrite(sp.sin))
solution
```

Here we can use the `subs` (substitution) method to set the value of `c`.  I plotted the square of the function since the intensity of a diffracted wave is related to the time averaged energy transferred by the wave.  This is proportional to the amplitude squared.  As our function is real valued, we can just plot the square.

```python
sp.plot(solution.subs(c,1));
```

```python
sp.plot(solution.subs(c,1)**2);
```

### Diffraction from Two Apertures
----

We could perform the same integration over two top-hat functions and plot those results.

```python
compositeIntegral = sp.sqrt(1/(2*sp.pi))*sp.Integral(1*sp.exp(-sp.I*2*omega*x),(x,1,2)) + \
sp.sqrt(1/(2*sp.pi))*sp.Integral(1*sp.exp(-sp.I*2*omega*x),(x,-2,-1))
compositeIntegral
```

```python
om = compositeIntegral.doit()
om
```

The diffracted intensity from this pair of slits would appear as:

```python
sp.plot(om.rewrite(sp.sin).expand()**2)
```

Or we could functionalize our function to explore other parameters:

```python
def diffractionFunction(d=4.0, w=1.0):
    result = sp.sqrt(1/(2*sp.pi))*sp.Integral(1*sp.exp(-sp.I*2*omega*x),\
                                     (x,-(d+w),-(d-w))) + \
    sp.sqrt(1/(2*sp.pi))*sp.Integral(1*sp.exp(-sp.I*2*omega*x),\
                                     (x,(d-w),(d+w)))
    return result.doit()
```

```python
sp.expand(diffractionFunction(10.,2.).rewrite(sp.sin))
```

### DIY:  Complex Numbers
----

Perform the Fourier transformation on an odd or complex valued function.  Plot the real and imaginary parts of both the target function and the transformed functions.

### DIY:  The Airy Disk
----

Solve for the diffracted intensity in two dimensions from a circular aperture. It may be easier to do this as a discrete problem using the DFT below.

### The Discrete Fourier Transform
----

The discrete Fourier Transform is defined [here](http://en.wikipedia.org/wiki/Discrete_Fourier_transform) and is regarded as one of the most important advances in computing science in the 20th century.  Other resources such as Numerical Recipes, the Python help files and many other websites detail the calculation and implementations.

It is often instructive to review other implementations of the DFT to help you gain experience.  I will be modeling this implementation after Jake Vanderplas' blog article [here](http://jakevdp.github.io/blog/2013/08/28/understanding-the-fft/).  Following the notion in the blog article:

Forward DFT:

$$X_k = \sum_{n=0}^{N-1} x_n \cdot e^{-i~2\pi~k~n~/~N}$$

Inverse DFT:

$$x_n = \frac{1}{N}\sum_{k=0}^{N-1} X_k e^{i~2\pi~k~n~/~N}$$

In this section of the notebook, we use Vanderplas' description and implementation.

----
For simplicity, we'll concern ourself only with the forward transform, as the inverse transform can be implemented in a very similar manner.  Taking a look at the DFT expression above, we see that it is nothing more than a straightforward linear operation: a matrix-vector multiplication of $\vec{x}$,

$$\vec{X} = M \cdot \vec{x}$$

with the matrix $M$ given by

$$M_{kn} = e^{-i~2\pi~k~n~/~N}$$

With this in mind, we can compute the DFT using simple matrix multiplication as follows:

```python
import numpy as np
def DFT_slow(x):
    """Compute the discrete Fourier Transform of the 1D array x"""
    x = np.asarray(x, dtype=float)
    N = x.shape[0]
    n = np.arange(N)
    k = n.reshape((N, 1))
    M = np.exp(-2j * np.pi * k * n / N)
    return np.dot(M, x)
```

We can use the "all close" function to check if the result from `DFT_slow` and `Numpy` are close:

```python
x_signal = np.random.random(1024)
np.allclose(DFT_slow(x_signal), np.fft.fft(x_signal))
```

----

I think it would be instructive to symbolically expand the matrix above so that it is clear how `n*k` leads to a two dimensional matrix.  Switching to `sympy` symbols to expose the details we can do the following:

```python
import sympy as sp
from sympy import Matrix
import numpy as np
sp.init_printing()
```

* `x` is the input vector.
* `k` is the wavenumber or frequency.
* `n` is the component of the input vector.

```python
x = sp.Matrix(sp.symbols('x0:5'))
n = sp.Matrix(sp.symbols('n0:5')).T
k = sp.Matrix(sp.symbols('k0:5'))
N = sp.symbols('N')
M = (-sp.I*2*sp.pi*k*n/N).applyfunc(sp.exp)
```

```python
M*x
```

Each frequency element is projected into each point of the input vector - the matrix links `k` and `n`.  So - the contribution at each point is a sum of each frequency contribution, similar to the dot product of functions.

### DFT with Numpy Functions
----

In this section we use the `FFT` submodule of `numpy` to help in the computation of the DFT.

```python
?np.fft # This gives us information on the conventions used in the return values of the functions.
```

```python
?np.fft.fft # This is the main DFT function we will use.
```

```python
?np.fft.fftfreq  # This is a helper function to prepare a vector of frequencies.
```

```python
?np.arange  # Points in an evenly spaced interval.
```

This approach is derived from a nice discussion on FFT found on the blog Glowing Python.

First we will divide up time into `samplingInterval` sized chunks between 0 and 1.  This will aid in getting the x-axis scaled correctly so that frequency can be read directly off the DFT result.  You can take `samplingInterval` in seconds putting samplingRate in Hz.  Notice the approach here - we could have done this all in one line, but, by intelligently naming our variables and exposing the details of our thoughts the code is more readable:

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np

samplingRate = 150.0
samplingInterval = 1.0/samplingRate
timeVector = np.arange(0, 1, samplingInterval)

# Print out the first few elements so you can see what is going on:
timeVector[0:10:]
```

Next we decide on the frequency of our signal and create a list to have a signal to work with.

```python
signalFrequency = 10.0;
ourSignal = np.sin(2*np.pi*signalFrequency*timeVector) + 0.5*np.sin(2*np.pi*(2*signalFrequency)*timeVector)
```

Plotting the input function for clarity:

```python
fig = plt.figure()

axes = fig.add_axes([0.1, 0.1, 0.8, 0.8])
axes.plot(timeVector, ourSignal, 'r')
axes.set_xlabel('Time')
axes.set_ylabel('Signal')
axes.set_title('Our Modest Signal');
```

Using `numpy` to compute the DFT:

```python
n = ourSignal.size
frequencies = np.fft.fftfreq(n, d=1.0/samplingRate)
spectrum = np.abs(np.fft.fft(ourSignal))

fig = plt.figure()

axes = fig.add_axes([0.1, 0.1, 0.8, 0.8])

axes.scatter(frequencies, spectrum, c='r', marker='s', alpha=0.4)

axes.set_xlabel('Frequency')
axes.set_ylabel('Amplitude')
axes.set_title('Our Amplitude Spectrum');
```

### Interactive Microscopy Demonstration (Optional)
Original developed by C. Carter, translated to Python by D. Lewis

---

Transmission electron microscopy utilizes diffraction to determine crystal structures and develop contrast in images.  In this section of the lecture we will simulate the diffraction pattern of an atomic structure.  Using this diffraction pattern we will simulate using a diffraction aperture to reconstruct a phase contrast image.

```python
%matplotlib inline
import numpy as np
import matplotlib as mpl
import matplotlib.pyplot as plt
from numpy.fft import *

def atomic_func(x,y):
    param = 64.0
    return (1+np.sin(4*(x+y)*2*np.pi/param))*(1+np.sin(2*(x-2*y)*2*np.pi/param))/4

def aperture(X, Y, xoffset, yoffset, size):
    return (X-xoffset)**2+(Y-yoffset)**2 > size**2
```

We define two functions above:

* `atomic_func` is used to provide an image function periodic in two dimensions from which the diffraction pattern will be constructed.  This can be thought of as the density of electrons in a solid that is used to approximate a crystal structure.
* `aperture` returns a Boolean array that will be used to mask the diffraction pattern so that individual frequencies can be selected for image reconstruction.  `aperture` will return `True` or `False`.

```python
x = np.arange(0.0,256.0,1.0)
y = np.arange(0.0,256.0,1.0)
X,Y = np.meshgrid(x, y)

Z = atomic_func(X,Y)
```

The `Z` array holds the atomic image function.

```python
P = np.zeros(Z.shape,dtype=complex)
K = np.zeros(Z.shape,dtype=complex)

K = fftshift(fft2(Z, norm='ortho'))

P = np.copy(K)
P[np.where(aperture(X, Y, 128, 128, 3) & aperture(X, Y, 150, 128, 3))] = 0
```

The `P` array holds the processed Fourier spectrum.  The values of `P` are set to zero when they are outside the aperture.  We use the `K` array to hold a opy of the image

In this cell we create two more `numpy` arrays (there are other ways to do this) that have the same shape as Z. The `P` array we use to hold the processed Fourier spectrum.  The processing uses `numpy`'s Boolean indexing to set values in P equal to zero when they are "outside" the aperture.  When we get to the images below you'll see what is meant.

Because Python passes by reference we need to call for a copy of K so that we can modify one without changing the other.

From this processed spectrum we will create an image.  The K array holds the whole Fourier spectrum.

```python
Im = fftshift(ifft2(P))
```

Above we reprocess `P` into the image `Im`.

```python
fig, axes = plt.subplots(nrows=1, ncols=3, figsize=(30,9))
axes[0].imshow(Z, origin='lower')
axes[1].imshow(abs(K),origin='lower', cmap=plt.get_cmap('pink'))
aperture1 = plt.Circle((128,128),3**2,color='r', fill = False)
aperture2 = plt.Circle((150,128),3**2,color='y', fill = False)
axes[1].add_artist(aperture1)
axes[1].add_artist(aperture2)
axes[2].imshow(abs(Im)**2, origin='lower')
plt.show()
```

### Homework
----

Apply the DFT to an image of your choosing.  Select the low frequency part of the DFT and regenerate the image (i.e. take the inverse FFT) from only these selected frequencies.  Use a Boolean selection to zero out parts of the frequency spectrum before you convert back.  To read an image in from disk, use the `ndimage` function from SciPy:

```python
from scipy.ndimage import imread
img = imread('./images/pattern2.jpg', mode='L')
```

checking the data type of `img` will prove helpful.

### Summary
----

* Integral transforms map one function space into another function space.  You can find books that include tables of Laplace and Fourier transforms.  Many other transforms exist - but the principle is the same.
* The DFT organizes amplitude information in predictable yet non-intuitive ways.  Read the documentation for the functions you use!
* Integral transforms are a means for reducing the complexity of certain ODEs and PDEs.
* Diffraction and diffusion are two example applications where integral transforms can be employed.

### Reading Assignments and Practice
----

* Pam Champness' book on electron diffraction is a (relatively) easy read on diffraction.  You can always have a look at Cullity, Hammond, or any other book on structure and X-ray/electron characterization.
* Practice taking the FFT of signals you construct by hand.  This is a good step when you are debugging a problem.  You should always have a test case available to determine if your work is correct.