# Learning Theory

## bias and variance

we define bias of a model to be the expected generalization error even if we were to fit it to a very large (say, infinity) training set.

variance is about the expected distance of a model from this setting's optimal model.

(hoeffding inequality) let $Z_{1},...,Z_{n}$ be $n$ iid random variables draw from Bernoulli ($\phi$) distribution, let $\hat{\phi} = \frac{1}{n}\sum_{i=1}^{n}Z_{i}$ be the mean of these random variables, let $\gamma > 0$ be fixed, then:

$$P(|\phi - \hat{\phi}| > \gamma) \le 2exp(-2\gamma^{2}n)$$

proof t.b.c.

## errors

to simplify our exposition, we restrict to binary classification problem.

assume training set $S = \left\{(x^{(i)}, y^{(i)})\right\}_{i=1}^{n}$ draw iid from some distribution $\mathcal{D}$.

then for a hypothesis $h$, we define the training error (also called the empirical risk or empirical error) to be:

$$\hat{\epsilon}(h) = \frac{1}{n}\sum_{i=1}^{n}1\left\{h(x^{(i)} \ne y^{(i)})\right\}$$

we define the generalization error to be:

$$\epsilon(h) = P_{(x,y)\sim\mathcal{D}}(h(x) \ne y)$$

our goal is to minimize the generalization error.

while we can only minimize training error in practice, our training process can often be formalize as the process of empirical risk minimization (ERM):

$$\hat{\theta} = \underset{\theta}{argmin}\ \hat{\epsilon}(h_{\theta})$$

we denote hypothesis class $\mathcal{H}$ to be the set of all classifier considered by a learning algorithm, thus ERM can be rewritten as:

$$\hat{h} = \underset{h \in \mathcal{H}}{argmin}\ \hat{\epsilon}(h)$$

the optimal model is:

$$h^{\ast} = \underset{h \in \mathcal{H}}{argmin}\ {\epsilon}(h)$$

we want to give guarantee on the approximation of $h^{\ast}$ by $\hat{h}$.

this is no easy, we consider it in two cases.

## the case of finite hypothesis class

take any fixed $h_{i} \in H$

define a bernoulli random variable $Z = 1\left\{h_{i}(x)\ne y\right\}$

similarily $Z_{j} = 1\left\{h_{i}(x^{(j)})\ne y^{(j)}\right\}$, then $Z$ and $Z_{j}$ have the same distribution.

$$\epsilon(h_{i}) = E(Z) = E(Z_{j})$$

and

$$\hat\epsilon(h_{i}) = \frac{1}{n}\sum_{j=1}^{n}Z_{j}$$

we can then apply the hoeffding inequality:

$$P(|\epsilon(h_{i}) - \hat\epsilon(h_{i})| > \gamma) \le 2exp(-2\gamma^{2}n)$$

this shows that for particular $h_{i}$, training error will be close to generalization error with high probability if $n$ is large.

further, use the finiteness of $\mathcal{H}$, we derive the uniform convergence result:

$$P(\exists h\in\mathcal{H}.|\epsilon(h_{i}) - \hat\epsilon(h_{i})| > \gamma) \le 2k\ exp(-2\gamma^{2}n)$$

that is:

$$P(\forall h\in\mathcal{H}.|\epsilon(h_{i}) - \hat\epsilon(h_{i})| \le \gamma) > 1 - 2k\ exp(-2\gamma^{2}n)$$

what we did was, for particular values of $n$ and $\gamma$, give a lower bound on the probability that for all $h\in \mathcal{H}$, $|\epsilon(h) - \hat\epsilon(h)| \le \gamma$.

our estimate can be represented by $(n, \gamma, \delta)$, where $\delta = 2k\ exp(-2\gamma^{2}n)$.

now with the probability at least $1 - \delta$, we have:

$$
\begin{equation}
\begin{split}
\epsilon{(\hat{h})}\ \le& \ \hat\epsilon{(\hat{h})} + \gamma\\
\le& \ \hat\epsilon{(h^{\ast})} + \gamma\\
\le& \ \epsilon{(h^{\ast})} + 2\gamma
\end{split}
\end{equation}
$$

this is what we want.

theorem. let $|\mathcal{H}| = k$, and let $n, \delta$ be fixed. then with the probabilty at least $1 - \delta$, we have:

$$\epsilon(\hat{h})\ \le\ \epsilon(h^{\ast}) + 2\sqrt{\frac{1}{2n}log\frac{2k}{\delta}}$$

in other words, fix $\gamma, \delta$. to guarantee the results, it suffices that:

$$n \ge \frac{1}{2\gamma^{2}}log\frac{2k}{\delta} = O(\frac{1}{\gamma^{2}}log\frac{k}{\delta})$$

## the case of infinite hypothesis class

intuitive argument:

suppose $\mathcal{H}$ is parameterized by $d$ real numbers. IEEE double-precision uses 64 bits to represent a floating point number.

now each model in $\mathcal{H}$ is parameterized by $64d$ bits, our hypothesis class really consist of at most $k=2^{64d}$ different hypotheses.

use the finite case result, to make the guarantee, it suffices:

$$n \ge O(\frac{1}{\gamma^{2}}log\frac{2^{64d}}{\delta}) = O(\frac{d}{\gamma^{2}}log\frac{1}{\delta}) = O_{\gamma, \delta}(d)$$

that is, to learn "well" using a hypothesis class of $d$ parameters, generally we are going to need $O(d)$ training examples.

but $d$ can be different for the same $\mathcal{H}$, for example:

$$h_{\theta}(x) = 1\left\{\theta_{0} + \theta_{1}x_{1} + ... + \theta_{d}x_{d} \ge 0\right\}$$

$$h_{u, v}(x) = 1\left\{(u_{0}^2 - v_{0}^2) + (u_{1}^2 - v_{1}^2)x_{1} + ... + (u_{d}^2 - v_{d}^2)x_{d} \ge 0\right\}$$

they both represent the class of linear classifiers in $\mathbb{R}^{d}$, but one represent by $d + 1$ parameters, the other by $2d + 2$ parameters.

now we give some definitions to clear it out.

given a set $S = \left\{x^{(1)}, ..., x^{(D)}\right\}$ of points $x^{(i)} \in \mathcal{X}$, we say that $\mathcal{H}$ shatters $S$ if $\mathcal{H}$ can realize any labeling on $S$.

we define the vapnik-chervonenkis dimension of $\mathcal{H}$, written $VC(\mathcal{H})$, to be the size of the largest set that is shattered by $\mathcal{H}$.

for approporiate representation, $VC(\mathcal{H})$ is roughly linear in the number of parameters.

theorem. let $\mathcal{H}$ be given, $D = VC(\mathcal{H})$, then with probability at least $1 - \delta$, we have that for all $h \in \mathcal{H}$:

$$|\epsilon(h) - \hat\epsilon(h)| \le O\left(\sqrt{\frac{D}{n}log\frac{n}{D} + \frac{1}{n}log\frac{1}{\delta}}\right)$$

thus, we also have:

$$\epsilon(\hat{h}) \le \epsilon(h^{\ast}) + O\left(\sqrt{\frac{D}{n}log\frac{n}{D} + \frac{1}{n}log\frac{1}{\delta}}\right)$$

proof t.b.c

```python

```


# Moran Processes

The evolutionary models discussed in the previous chapters assume an infinite population that can be divided in to infinitessimal parts. Finite populations can also be studied using a model called a Moran Process (first described in 1958).

---

## Moran process with neutral drift

[Video](https://youtu.be/OeMku85hwEc?list=PLnC5h3PY-znxMsG0TRYGOyrnEO-QhVwLb)

Consider a population of two types of fixed size $N$. This can be represented as a vector of the form: $(i, N-i)$ where $i\geq 0$ represents the number of individuals of the first type.

The term **neutral** drift refers to the fact that the two types reproduce at the same rate.

The Moran process is as follows:

- At a given time step: select a random individual for reproduction and a random individual for elimination
- The eliminated individual is replaced by a new individual of the same type as the individual chosen for reproduction.
- Proceed to the next time step.
- The process terminates when there is only one type of individual in the population.

---

Here is some simple Python code that simulates such a Process assuming an initial population of $(3, 3)$:

```python
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline

def neutral_moran(N, i=1, seed=0):
    """
    Return the population counts for the Moran process with neutral drift.
    """

    population = [0 for _ in range(i)] + [1 for _ in range(N - i)]
    counts = [(population.count(0), population.count(1))]
    np.random.seed(seed)
    while len(set(population)) == 2:
        reproduce_index = np.random.randint(N)
        eliminate_index = np.random.randint(N)
        population[eliminate_index] = population[reproduce_index]
        counts.append((population.count(0), population.count(1)))
    return counts

N = 6
plt.plot(neutral_moran(N=N, i=3, seed=6));
```

For different seeds we see we obtain different results. What becomes of interest is not the path but the end result: which strategy overcomes the presence of the other?

```python
def neutral_fixation(N, i=None, repetitions=10):
    """
    Repeat the neutral Moran process and calculate the fixation probability
    """
    fixation_count = 0
    for seed in range(repetitions):
        final_counts = neutral_moran(N=N, i=i, seed=seed)
        if final_counts[-1][0] > 0:
            fixation_count += 1

    return  fixation_count / repetitions
```

Let us take a look at probability of the first strategy taking over for different starting populations:

```python
probabilities = [neutral_fixation(N, i=i, repetitions=500) for i in range(1, N)]
plt.scatter(range(1, N), probabilities)
plt.xlabel("$i$")
plt.ylabel("$x_i$");
```

We see that as the initial population starts with more of a given type, the chance that that type "takes over" (becomes fixed) grows.

This Moran Process is a specific case of a Markov Process:

- A given state of the system can be described by a single integer $0\leq i\leq N$;
- The state to state transition probabilities are given by:

  $$
  \begin{aligned}
  p_{i, i-1}&=\frac{i(N - i)}{N^2}\\
  p_{i, i+1}&=\frac{i(N - i)}{N^2}\\
  p_{i, i}&=1 - p_{i, i-1} - p_{i, i+1}
  \end{aligned}
  $$

  We also have two absorbing states (when the Moran process ends):

  $$p_{00}=1\qquad p_{0i}=0\text{ for all }i>0$$

  $$
  p_{NN}=1\qquad p_{Ni}=0\text{ for all } N>i
  $$

  these transitions can be represented as a matrix. Here for example is the matrix for $N=6$:

```python
N = 6
p = np.zeros((N + 1, N + 1))
p[0, 0] = 1
p[N, N] = 1
for i in range(1, N):
    for j in [i - 1, i + 1]:
        p[i, j] = i * (N - i) / (N ** 2)
    p[i, i] = 1 - sum(p[i, :])
p.round(2)
```

    array([[1.  , 0.  , 0.  , 0.  , 0.  , 0.  , 0.  ],
           [0.14, 0.72, 0.14, 0.  , 0.  , 0.  , 0.  ],
           [0.  , 0.22, 0.56, 0.22, 0.  , 0.  , 0.  ],
           [0.  , 0.  , 0.25, 0.5 , 0.25, 0.  , 0.  ],
           [0.  , 0.  , 0.  , 0.22, 0.56, 0.22, 0.  ],
           [0.  , 0.  , 0.  , 0.  , 0.14, 0.72, 0.14],
           [0.  , 0.  , 0.  , 0.  , 0.  , 0.  , 1.  ]])

The above corresponds to a particular type of Markov process called a Birth-Death process

---

## Birth death process

[Video](https://youtu.be/zJQQF2tq9AA?list=PLnC5h3PY-znxMsG0TRYGOyrnEO-QhVwLb)

A birth death process is a Markov process with the following properties:

- $p_{i,i+1}+p_{i,i-1}\leq 1$
- $p_{ii}=1-p_{i,i+1}-p_{i,i-1}$
- $p_{00}=1$ and $p_{NN}=1$

---

Thus we have two absorbing states: $\{0, N\}$. Let us denote by $x_i$ the probability of being in $state$ $i$ and eventually reaching state $N$.

We have the following linear system:

\begin{align}
    x_0&=0\\
    x_i&=p_{i,i-1}x_{i-1}+p_{ii}x_i+p_{i,i+1}x_{i+1}\text{ for all }0< i< N-1\\
    x_N&=1\\
\end{align}

---

## Theorem: Fixation probabilities for the birth death process

Given a birth death process as defined above, the fixation probability $x_i$ is given by:

$$x_i=\frac{1+\sum_{j=1}^{i-1}\prod_{k=1}^j\gamma_k}{1+\sum_{j=1}^{N-1}\prod_{k=1}^j\gamma_k}$$

where:

$$
\gamma_k = \frac{p_{k,k-1}}{p_{k,k+1}}
$$

### Proof

We have:

$$
\begin{aligned}
    p_{i,i+1}x_{i+1} & = -p_{i,i-1}x_{i-1} + x_i(1 - p_{ii}) \\
    p_{i,i+1}x_{i+1} & = p_{i,i-1}(x_{i} - x_{i-1}) + x_ip_{i,i+1} \\
    x_{i+1} - x_i    & = \frac{p_{i, i-1}}{p_{i, i+1}}(x_i-x_{i-1})=\gamma_i(x_i-x_{i-1})
\end{aligned}
$$

We observe that:

$$
\begin{aligned}
    x_2 - x_1 &= \gamma_1(x_1-x_{0})=\gamma_1x_1\\
    x_3 - x_2 &= \gamma_2(x_2-x_1)=\gamma_2\gamma_1x_1\\
    x_4 - x_3 &= \gamma_3(x_3-x_2)=\gamma_3\gamma_2\gamma_1x_1\\
              &\; \vdots & \\
    x_{i+1} - x_i &= \gamma_i(x_i-x_{i-1})=\prod_{k=1}^i\gamma_kx_1\\
               &\; \vdots & \\
    x_{N} - x_{N-1} &= \gamma_{N-1}(x_{N-1}-x_{N-2})=\prod_{k=1}^{N-1}\gamma_kx_1\\
\end{aligned}
$$

thus we have:

$$x_i=\sum_{j=0}^{i-1}x_{j+1}-x_j=\left(1+\sum_{j=1}^{i-1}\prod_{k=1}^j\gamma_k\right)x_1$$

we complete the proof by solving the following equation to obtain $x_1$:

$$x_N=1=\left(1+\sum_{j=1}^{N-1}\prod_{k=1}^j\gamma_k\right)x_1$$

---

In the case of neutral drift (considered above) we have:

$$p_{i,i-1}=p_{i,i+1}$$

thus:

$$
\gamma_i=1
$$

so:

$$
x_i=\frac{1+\sum_{j=1}^{i-1}\prod_{k=1}^j\gamma_k}{1+\sum_{j=1}^{N-1}\prod_{k=1}^j\gamma_k}=\frac{1+i-1}{1+N-1}=\frac{i}{N}
$$

```python
probabilities = [neutral_fixation(N, i=i, repetitions=500) for i in range(1, N)]
plt.scatter(range(1, N), probabilities, label="Simulated")
plt.plot(range(1, N), [i / N for i in range(1, N)], label="Theoretic: $i/N$", linestyle="dashed")
plt.xlabel("$i$")
plt.ylabel("$x_i$")
plt.legend();
```

---

## Fixation probability

The fixation probability in a Moran process is the probability that a give type starting with $i=1$ individuals takes over an entire population. We denote the fixation probabilities of the first/second type as $\rho_1$ and $\rho_2$ respectively and we have:

$$
\rho_1=x_1
$$

$$
\rho_2=1-x_{N-1}
$$

---

We will now consider a Moran process on a game:

---

## Moran process on a game

[Video](https://www.youtube.com/watch?v=TpqVoF1fBF8&index=43&list=PLnC5h3PY-znxMsG0TRYGOyrnEO-QhVwLb)

Consider a matrix $A\in\mathbb{R}^{m\times n}$ representing a game with two strategies.

$$
A=
\begin{pmatrix}
    a & b\\
    c & d
\end{pmatrix}
$$

The Moran process is as follows:

- At a given time step: all individuals play all other individuals.
- Obtain their fitness as given by the game.
- Randomly select an individual proportional to their fitness as an individual to be reproduced
- Uniformly select an individual to be replaced
- Proceed to the next time step.
- The process terminates when there is only one type of individual in the population.

Assuming $i$ individuals of the first type, the fitness of both types is given respectively by:

$$f_{1i}=\frac{a(i-1)+b(N-i)}{N-1}$$
$$f_{2i}=\frac{c(i)+d(N-i-1)}{N-1}$$

The transition probabilities are then given by:

$$p_{i,i+1}=\frac{if_{1i}}{if_{1i} + (N-i)f_{2i}}\frac{N-i}{N}$$
$$p_{i,i-1}=\frac{(N-i)f_{2i}}{if_{1i} + (N-i)f_{2i}}\frac{i}{N}$$

which gives:

$$\gamma_i=\frac{f_{2i}}{f_{1i}}$$

thus:

$$
x_i=\frac{1+\sum_{j=1}^{i-1}\prod_{k=1}^j\gamma_k}{1+\sum_{j=1}^{N-1}\prod_{k=1}^j\gamma_k}
$$

---

Here is some code to carry out this calculation:

```python
def theoretic_fixation(N, game, i=1):
    """
    Calculate x_i as given by the above formula
    """
    f_ones = np.array([(game[0, 0] * (i - 1) + game[0, 1] * (N - i)) / (N - 1) for i in range(1, N)])
    f_twos = np.array([(game[1, 0] * i + game[1, 1] * (N - i - 1)) / (N - 1) for i in range(1, N)])
    gammas = f_twos / f_ones
    return (1 + np.sum(np.cumprod(gammas[:i-1]))) / (1 + np.sum(np.cumprod(gammas)))
```

Here is an example of calculating $x_1$ for the following game for $N=4$:

[Video](https://www.youtube.com/watch?v=3sBVrnQhemE&index=44&list=PLnC5h3PY-znxMsG0TRYGOyrnEO-QhVwLb)

$$
A =
\begin{pmatrix}
    4 & 1\\
    1 & 4
\end{pmatrix}
$$

```python
A = np.array([[4, 1],
              [1, 4]])
theoretic_fixation(N=4, i=1, game=A)
```

    0.125

Applying the theorem gives:

$$
\begin{aligned}
f_{1i}&=\frac{4(i - 1) + 4 - i}{3} = \frac{4i-4+4-i}{3}=i\\
f_{2i}&=\frac{i + 4(3 - i)}{3} = \frac{12-3i}{3}=4-i
\end{aligned}
$$

$$
\gamma_i = \frac{f_{2i}}{f_{1i}}=\frac{4-i}{i}=\frac{4}{i}-1
$$

Thus:

$$
\begin{aligned}
x_1 & =\frac{1 + \sum_{j=1}^{0}\prod_{k=1}^{j}\gamma_k}{1 + \sum_{j=1}^{4 - 1}\prod_{k=1}^{j}\gamma_k}\\
    & =\frac{1}{1 + \sum_{j=1}^{3}\prod_{k=1}^{j}\gamma_k}\\
    & =\frac{1}{1 + \gamma_1 + \gamma_1\times \gamma_2 + \gamma_1 \times \gamma_2 \times \gamma_3}\\
    & =\frac{1}{1+3+3\times 1 + 3 \times 1\times \frac{1}{3}} = \frac{1}{1 + 3 + 3 + 1}=\frac{1}{8}\\
\end{aligned}
$$

Nashpy has the ability to run a single Moran process:

```python
import nashpy as nash

game = nash.Game(A)
initial_population = np.array((0, 1, 1, 1))
np.random.seed(0)
generations = game.moran_process(
    initial_population=initial_population
)
for population in generations:
    print(population)
```

    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [0 1 1 1]
    [1 1 1 1]

We see there that in a population of 4 individuals, a single individual of the first type (`0`) does not become fixed. That is just for a single run, to be able to approximate the fixation probability the process needs to be repeated, this can be done in Nashpy:

```python
import collections

def approximate_fixation(N, A, i=None, repetitions=10):
    """
    Repeat the Moran process and calculate the fixation probability

    This is done by carrying out the following steps:

    1. Creating a game
    2. Building an initial population with i individuals
       of the first type
    3. Getting the fixation probabilities of both types
    4. Returning the probability of the first type
    """
    game = nash.Game(A)
    initial_population = i * [0] + (N - i) * [1]
    probabilities = game.fixation_probabilities(
        initial_population=initial_population,
        repetitions=repetitions
    )

    return probabilities[0]
```

Here is how the fixation probabilities vary for different initial populations:

```python
probabilities = [approximate_fixation(N, i=i, A=A, repetitions=500) for i in range(1, N)]
plt.scatter(range(1, N), probabilities, label="Simulated")
plt.plot(range(1, N), [i / N for i in range(1, N)], label="Neutral: $i/N$", linestyle="dashed")
plt.plot(range(1, N), [theoretic_fixation(N=N, i=i, game=A) for i in range(1, N)], label="Theoretic")
plt.xlabel("$i$")
plt.ylabel("$x_i$")
plt.legend();
```


# Problem Sheet 2 Question 2b - 3rd Order Taylor

The general form of the population growth differential equation
\begin{equation} y^{'}=y-t, \ \ (0 \leq t \leq 2), \end{equation}
with the initial condition
\begin{equation}y(0)=2.\end{equation}
For N=4
with the analytic (exact) solution
\begin{equation} y= e^{t}+t+1.\end{equation}

Apply __3rd Order Taylor__ to approximate the solution of the given initial value problems using the indicated number of time steps. Compare the approximate solution with the given exact solution, and compare the actual error with the theoretical error.

```python
## Library
import numpy as np
import math

%matplotlib inline
import matplotlib.pyplot as plt # side-stepping mpl backend
import matplotlib.gridspec as gridspec # subplots
import warnings
import pandas as pd

warnings.filterwarnings("ignore")
```

## General Discrete Interval
The continuous time $a\leq t \leq b $ is discretised into $N$ points seperated by a constant stepsize
\begin{equation} h=\frac{b-a}{N}.\end{equation}
## Specific Discrete Interval
Here the interval is $0\leq t \leq 2$ with $N=4$
\begin{equation} h=\frac{2-0}{4}=0.5.\end{equation}
This gives the 5 discrete points with stepsize h=0.5:
\begin{equation} t_0=0, \ t_1=0.5, \ ... t_{4}=2. \end{equation}
This is generalised to
\begin{equation} t_i=0+i0.5, \ \ \ i=0,1,...,4.\end{equation}
The plot below illustrates the discrete time steps from 0 to 2.

```python
### Setting up time
t_end=2
t_start=0
N=4
h=(t_end-t_start)/(N)
t=np.arange(t_start,t_end+0.01,h)
fig = plt.figure(figsize=(10,4))
plt.plot(t,0*t,'o:',color='red')
plt.plot(t[0],0*t[0],'o',color='green')

plt.title('Illustration of discrete time points for h=%s'%(h))
plt.plot();
```

## 3rd Order Taylor Solution
The 3rd Order Taylor difference equation is given by
\begin{equation} w_{i+1}=w_i+h\left[f(t_i,w_i)+\frac{h}{2}f'(t_i,w_i)+\frac{h^2}{6}f''(t_i,w_i)\right],\end{equation}
where
\begin{equation}f(t_i,w_i)=w_i-t_i,\end{equation}
\begin{equation} f'(t_i,w_i)=w_i-t_i-1,\end{equation}
and
\begin{equation} f''(t_i,w_i)=w_i-t_i-1,\end{equation}
which gives
\begin{equation} w_{i+1}=w_i+h(w_i-t_i)+\frac{h^2}{2}(w_i-t_i-1)+\frac{h^3}{6}(w_i-t_i-1) \end{equation}

for $i=0,1,2,3$
with
$w_0=2.$

```python
IC=2
w=np.zeros(N+1)
w[0]=IC
for i in range (0,N):
    w[i+1]=w[i]+h*(w[i]-t[i])+h*h/2*(w[i]-t[i]-1)+h*h*h/6*(w[i]-t[i]-1)
```

```python
y=np.exp(t)+t+1 # Exact Solution
```

## Global Error
The upper bound of the global error is:
\begin{equation} |y(t_i)-w_i|\leq\frac{Mh^3}{24}\frac{1}{L} |e^{Lt_i}-1| \end{equation}
where
$|y'''(t)|\leq M $ for $t \in (0,2)$ and $L$ is the Lipschitz constant.
Generally we do not have access to $M$ or $L$, but as we have the exact solution we can find $M$ and $L$.
### Local Error
The global error consistent of an exponential term and the local error.
The local error for the Euler method is
\begin{equation} |y(t_1)-w_1|<\tau=\frac{Mh^3}{24} \end{equation}

by differentiation the exact solution
$ y= y= e^{t}+t+1$
four times
\begin{equation}y'= e^{t}+1,\end{equation}
\begin{equation}y''=e^{t},\end{equation}
and
\begin{equation}y''''=e^{t}.\end{equation}
Then we choose the largest possible value of $y''''$ on the interval of interest $(0,2)$, which gives,

\begin{equation}|y''(t)|\leq e^{2}=7.38<8, \end{equation}
therefore
\begin{equation}M=8.\end{equation}
This gives the local error
\begin{equation} |y(t_1)-w_1|<\tau=\frac{Mh^3}{24}=\frac{8h^3}{24}.\end{equation}
### Lipschitz constant
The Lipschitz constant $L$ is from the Lipschitz condition,
\begin{equation}\left| \frac{\partial f(t,y)}{\partial t}\right|\leq L. \end{equation}
The constant can be found by taking partical derivative of $f(t,y)=y-t$ with respect to $y$
\begin{equation}\frac{\partial f(t,y)}{\partial y}=1,\end{equation}
\begin{equation}L=1.\end{equation}

### Specific Global Upper Bound Formula
Puting these values into the upper bound gives:
\begin{equation} |y(t_i)-w_i|\leq\frac{8h^3}{24} |e^{t_i}-1|. \end{equation}

```python
Upper_bound=8*h*h*h/(24*1)*(np.exp(t)-1) # Upper Bound
```

## Results

```python
fig = plt.figure(figsize=(12,5))
# --- left hand plot
ax = fig.add_subplot(1,4,1)
plt.plot(t,w,'o:',color='k')
plt.plot(t[0],w[0],'o',color='green')

#ax.legend(loc='best')
plt.title('Numerical Solution (w) h=%s'%(h))

# --- right hand plot
ax = fig.add_subplot(1,4,2)
plt.plot(t,y,color='blue')
plt.title('Analytic Solution (y)')

#ax.legend(loc='best')
ax = fig.add_subplot(1,4,3)
plt.plot(t,np.abs(w-y),'o:',color='red')
plt.title('Exact Error |w-y|')

# --- title, explanatory text and save

ax = fig.add_subplot(1,4,4)
plt.plot(t,np.abs(w-y),'o:',color='red')
plt.plot(t,Upper_bound,color='purple')
plt.title('Upper Bound')

# --- title, explanatory text and save
fig.suptitle(r"$y'=y-t$", fontsize=20)
plt.tight_layout()
plt.subplots_adjust(top=0.85)
```

```python
d = {'time t_i': t,    '3rd Order Taylor (w_i) ':w,    'Exact (y_i) ':y,    'Exact Error( |y_i-w_i|) ':np.abs(y-w),'Upper Bound':Upper_bound}
df = pd.DataFrame(data=d)
df
```

      time t_i
      3rd Order Taylor (w_i)
      Exact (y_i)
      Exact Error( |y_i-w_i|)
      Upper Bound

      0
      0.0
      2.000000
      2.000000
      0.000000
      0.000000

      1
      0.5
      3.145833
      3.148721
      0.002888
      0.027030

      2
      1.0
      4.708767
      4.718282
      0.009514
      0.071595

      3
      1.5
      6.958180
      6.981689
      0.023509
      0.145070

      4
      2.0
      10.337421
      10.389056
      0.051635
      0.266211

```python

```

Chapter 02 - Higher Order Methods/2_Problem Sheet/202b_Problem_Sheet.ipynb


# Plotting 3D Search Spaces

As I was making progress on my Master's thesis, I needed to figure a way to show visualizations of functions with 3 variables, so I created a Python script that allows me to visualize it using contour surfaces. This is a tutorial showing how I implemented this Python script and how it helped me gain insights about a function that would otherwise be extremely hard to visualize.

## Introduction

Python has some great packages for plotting functions. The most widely used is [`matplotlib`](https://matplotlib.org/), which has some awesome graphing features to create plots pretty enough to meet scientific journals standards. In fact, I challenged myself to plot all my thesis results using exclusively `matplotlib`.

When I was taking optimization classes, plotting functions was extremely important to visualize the search space (domain) I was dealing with, and choose the most adequate algorithm accordingly.

Most books and lectures depicts optimization algorithms on functions with 1 or 2 variables. This is actually reasonable since from 2 variables and beyond, the algorithms work the same way.

Nonetheless, I still wanted to visualize the search space of functions with 3 variables, and then, I created a Python function that plots a 3D search space using `matplotlib`. There are other tools that allows you to visualize 3D search spaces such as [`mayavi`](http://code.enthought.com/projects/mayavi/) and [`skimage`](http://scikit-image.org/docs/dev/auto_examples/edges/plot_marching_cubes.html), but I found an implemenation on `matplotlib` to be the most appropriate, since all my plots were made using this package.

## Importing the libraries

The libraries used in this tutorial are:

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
```

## Plotting One Variable Functions

Starting with the simplest case, one variable functions have 1D search space. This means that the search space is a line, which would be very boring to visualize.

In this case, besides the search space, a one variable function plot also contains the image of the function.

Considering the following quadractic funtion:

\begin{equation}
    f(x)=x^2
\end{equation}

The Python script that plots this function is:

```python
x = np.linspace(-5,5,50)
fun = x**2

plt.grid()
plt.plot(x,fun)
```

## Plotting Two Variables Functions

Moving on to two variables functions, we end up with 2D search spaces, which turns out to be a surface. This makes things a bit more interesting since now we can either have a 2D representation of the function using contour lines, or plot the search space along with the image of the function through a 3D plot.

Considering the 2D case for a quadractic function:

\begin{equation}
    f(x_1,x_2)=x_1^2+x_2^2
\end{equation}

The Python script that plot both styles is:

```python
x = np.linspace(-5, 5, 100)
x1, x2 = np.meshgrid(x, x)
fun = x1**2 + x2**2

fig = plt.figure(figsize=(12,3))

ax = fig.add_subplot(1, 2, 1)
ax.contour(x1,x2,fun,cmap='jet')

ax = fig.add_subplot(1, 2, 2, projection='3d')
ax.plot_surface(x1, x2, fun, cmap='jet')
```

These plots are as far as most articles about optimization go.

Indeed, it is not that necessary to visualize 3D seach spaces since most algorithms generalize well for higher dimension, but visualizing these search spaces can still bring insights about the domain that would otherwise be extremely difficult to grasp.

## Plotting Three Variable Functions

Finally, it is time to visualize 3D search spaces. We've reached a point in which a plot containing the domain and image of a function requires a fourth dimension. This means that at this point we can only visualize a function through its search space, which, fortunately, is enough to understand what's going on.

Instead of dealing with contour lines, as in two variables functions, three variables use contour surfaces, which is composed of [isosurfaces](https://en.wikipedia.org/wiki/Isosurface).

Contour surfaces creates a volume of data points, meaning the we can't see what's "inside" this volume unless we plot a cross section of it. This is where my idea to create this Python script came from.

The idea is simple: instead of plotting the whole volume, it will plot only the surfaces that are visible. After joining these surfaces as a cube, we have a perspective which feels three dimensional.

I created a Python function for this and named it `cross_section_3d()`. It takes some arguments, but for now let's just define and visualize its results.

First, let's consider consider a 3D quadractic function:

\begin{equation}
    f(x_1,x_2,x_3)=x_1^2+x_2^2+x_3^2
\end{equation}

In Python we can define this function as:

```python
def fun(x1,x2,x3):
   return x1**2+x2**2+x3**2
```

**Fun fact:** This function is also called the [Sphere Function](https://www.sfu.ca/~ssurjano/spheref.html) because the search space is a circle in 2 dimensions, a sphere in 3 and a hypersphere in higher dimensions.

Next, we define `cross_section_3d()`.

### `cross_section_3d()`

```python
def cross_section_3d(ax,fun,levels,xlim,ylim,zlim,mid,style='contour',colors='gray',res=61):

    levels = np.linspace(levels[0],levels[1],levels[2]).tolist()
    cset = [[],[],[],[]]

    range_ = [[xlim[0],mid[0],ylim[0],mid[1],mid[2]],[mid[0],xlim[1],ylim[0],mid[1],zlim[1]],
             [xlim[0],mid[0],mid[1],ylim[1],zlim[1]],[mid[0],xlim[1],mid[1],ylim[1],zlim[1]]]
    for key, i in enumerate(range_):
        x1 = np.linspace(i[0],i[1],int((float(i[1]-i[0]))/(xlim[1]-xlim[0])*res))
        x2 = np.linspace(i[2],i[3],int((float(i[3]-i[2]))/(ylim[1]-ylim[0])*res))
        X1,X2 = np.meshgrid(x1,x2)
        y = fun(X1,X2,i[4])
        if style == 'contour':
            cset[key] = ax.contour(X1, X2, y, zdir='z',levels=levels, offset=i[4],cmap=colors,
                                                              linewidths=1,linestyles='solid')
        else:
            cset[key] = ax.contourf(X1, X2, y, zdir='z',levels=levels, offset=i[4],cmap=colors)

    range_ = [[ylim[0],mid[1],mid[2],zlim[1],mid[0]],[mid[1],ylim[1],zlim[0],mid[2],xlim[0]],
             [ylim[0],mid[1],zlim[0],mid[2],xlim[0]],[mid[1],ylim[1],mid[2],zlim[1],xlim[0]]]
    for key, i in enumerate(range_):
        x2 = np.linspace(i[0],i[1],int((float(i[3]-i[2]))/(ylim[1]-ylim[0])*res))
        x3 = np.linspace(i[2],i[3],int((float(i[3]-i[2]))/(zlim[1]-zlim[0])*res))
        X2,X3 = np.meshgrid(x2,x3)
        y = fun(i[4],X2,X3)
        if style == 'contour':
            cset[key] = ax.contour(y, X2, X3, zdir='x',levels=levels, offset=i[4],cmap=colors,
                                                              linewidths=1,linestyles='solid')
        else:
            cset[key] = ax.contourf(y, X2, X3, zdir='x',levels=levels, offset=i[4],cmap=colors)

    range_ = [[xlim[0],mid[0],mid[2],zlim[1],mid[1]],[mid[0],xlim[1],mid[2],zlim[1],ylim[0]],
             [xlim[0],mid[0],zlim[0],mid[2],ylim[0]],[mid[0],xlim[1],zlim[0],mid[2],ylim[0]]]
    for key, i in enumerate(range_):
        x1 = np.linspace(i[0],i[1],int((float(i[1]-i[0]))/(xlim[1]-xlim[0])*res))
        x3 = np.linspace(i[2],i[3],int((float(i[3]-i[2]))/(zlim[1]-zlim[0])*res))
        X1,X3 = np.meshgrid(x1,x3)
        y = fun(X1,i[4],X3)
        if style == 'contour':
            cset[key] = ax.contour(X1, y, X3, zdir='y',levels=levels, offset=i[4],cmap=colors,
                                                              linewidths=1,linestyles='solid')
        else:
            cset[key] = ax.contourf(X1, y, X3, zdir='y',levels=levels, offset=i[4],cmap=colors)

    #z-axis lines
    ax.plot([xlim[0],xlim[0]],[ylim[0],ylim[0]],[zlim[0],mid[2]],c='k',lw=1,zorder=1e+99)
    ax.plot([mid[0],mid[0]],[ylim[0],ylim[0]],[mid[2],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[1],xlim[1]],[ylim[0],ylim[0]],[zlim[0],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],xlim[0]],[mid[1],mid[1]],[mid[2],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],xlim[0]],[ylim[1],ylim[1]],[zlim[0],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([mid[0],mid[0]],[mid[1],mid[1]],[mid[2],zlim[1]],c='k',lw=1,zorder=1e+99)

    #x-axis lines
    ax.plot([xlim[0],mid[0]],[ylim[0],ylim[0]],[mid[2],mid[2]],c='k',lw=1,zorder=1e+99)
    ax.plot([mid[0],xlim[1]],[ylim[0],ylim[0]],[zlim[1],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],xlim[1]],[ylim[0],ylim[0]],[zlim[0],zlim[0]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],xlim[1]],[ylim[1],ylim[1]],[zlim[1],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],mid[0]],[mid[1],mid[1]],[zlim[1],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],mid[0]],[mid[1],mid[1]],[mid[2],mid[2]],c='k',lw=1,zorder=1e+99)

    #y-axis lines
    ax.plot([xlim[0],xlim[0]],[ylim[0],mid[1]],[mid[2],mid[2]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],xlim[0]],[mid[1],ylim[1]],[zlim[1],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[0],xlim[0]],[ylim[0],ylim[1]],[zlim[0],zlim[0]],c='k',lw=1,zorder=1e+99)
    ax.plot([xlim[1],xlim[1]],[ylim[0],ylim[1]],[zlim[1],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([mid[0],mid[0]],[ylim[0],mid[1]],[zlim[1],zlim[1]],c='k',lw=1,zorder=1e+99)
    ax.plot([mid[0],mid[0]],[ylim[0],mid[1]],[mid[2],mid[2]],c='k',lw=1,zorder=1e+99)

    ax.set_xlim3d(xlim[0]*.97,xlim[1]*.97)
    ax.set_ylim3d(ylim[0]*.97,ylim[1]*.97)
    ax.set_zlim3d(zlim[0]*.97,zlim[1]*.97)

    ax.set_xlabel('x1')
    ax.set_ylabel('x2')
    ax.set_zlabel('x3')

    ax.w_xaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))
    ax.w_yaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))
    ax.w_zaxis.set_pane_color((1.0, 1.0, 1.0, 0.0))

    ax.w_xaxis.line.set_color((1.0, 1.0, 1.0, 0.0))
    ax.w_yaxis.line.set_color((1.0, 1.0, 1.0, 0.0))
    ax.w_zaxis.line.set_color((1.0, 1.0, 1.0, 0.0))

    ax.grid(False)
```

Finally, we are ready to visualize the 3D seach space for this function:

```python
fig = plt.figure()
ax = Axes3D(fig, azim = -120, elev = 40)

cross_section_3d(ax,fun,levels=[0,75,10],xlim=[-5,5],ylim=[-5,5],zlim=[-5,5],mid=[0,0,0],
                 style='contourf',colors='jet',res=61)
```

Now that we know the result, let's go through the parameters:

* `ax`: A `matplotlib` figure object;

* `fun`: A three variable function that you want to analyse;

* `levels`: A list containing the minimun, maximum and number of levels for coloring;

* `xlim`, `ylim`, `zlim`: The boundaries for plotting;

* `mid`: A list containing the coordinates where the cross section occurs;

* `style`: The style of plot. Could either be `contour` or `contourf`;

* `colors`: The coloring option for the plot;

* `res`: Amount of values calculated.

Looking at the parameters, we see that most of them are aesthetic. The important parameter here to explore is `mid`.

This parameters allows to explore new cross sections and see hidden features in the search space.

Let's try some random values for `mid`:

```python
fig = plt.figure(figsize=(20,15))

for i in range(9):
    ax = fig.add_subplot(3,3,i+1, projection = '3d', azim = -120, elev = 40)

    mid = [np.random.rand()*4,np.random.rand()*4,np.random.rand()*-4]
    cross_section_3d(ax,fun,levels=[0,75,10],xlim=[-5,5],ylim=[-5,5],zlim=[-5,5],mid=mid,
                       style='contourf',colors='jet',res=61)
```

## Extra

It's time to try a different function, say:

\begin{equation}
    f(x_1,x_2,x_3)= cos(x_1)+cos(x_2)+cos(x_3)
\end{equation}

In Python:

```python
def fun2(x, y, z):
    return np.cos(x) + np.cos(y) + np.cos(z)
```

Playing with the parameters it's possible to get different results:

```python
fig = plt.figure(figsize=(20,15))

for i in range(9):
    ax = fig.add_subplot(3,3,i+1, projection = '3d', azim = -120, elev = 40)

    mid = [np.random.rand()*4,np.random.rand()*4,np.random.rand()*-4]
    cross_section_3d(ax,fun2,levels=[-3,3,11],xlim=[-5,5],ylim=[-5,5],zlim=[-5,5],mid=mid,
                       style='contourf',colors='jet',res=61)
```


# Lab 08 - Unsupervised learning

(Adapted from [one of the 2019 ICFA tutorial notebooks](https://github.com/radiasoft/icfa_ml/blob/master/tutorials_2019/unsupervized_learning/unsupervised_learning.ipynb))

## Homework
- 8-1: K-means clustering comparison
- 8-2 B: DBSCAN algorithm

# Environment

```python
%reset -f

# This will import np, plt
%pylab --no-import-all inline

# For high DPI displays
%config InlineBackend.figure_format = 'retina'
```

    Populating the interactive namespace from numpy and matplotlib

# Introduction

As opposed to supervised learning, in unsupervised learning the goal is to infer a model which describes an unknown structure from unlabeled data. We'll consider two classes of unsupervised learning algorithms: clustering and anomaly detection.

We'll also see how unsupervised techniques can be used to reduce the dimensionality of the input feature set.

# 1. Clustering

Clustering is the task of grouping a set of items together in such a way that items in one group (a cluster) are more similar to each other than to those in other groups. There are several types of clustering algorithms:

*   Hierarchical clustering (e.g. Linkage clustering)
*   Centroid-based clustering (e.g. k-means)
*   Distribution-based clustering (e.g. Expectation-Maximization)
*   Density-based clustering (e.g. DBSCAN)

We will look at one of the more simpler approaches: centroid-based clustering.

## K-Means Algorithm

Suppose we have a dataset {$x_1$, $x_2$, $x_3$, ..., $x_N$} consisting of $N$ observations of $D$ dimensional vectors **x** (i.e. $D$ features).

The goal is to partition the dataset into $K$ clusters. Therefore, the number of clusters in our dataset needs to be known a priori.

A cluster is a group of data points whose distances between one another in $D$-dimensional space are small compared to points outside the cluster.

This can be formalized by introducing a $D$-dimensional mean vector $\mu_k$, where $k = 1,2,3,...K$. This represents the center of the cluster.

The K-means clustering algorithm assigns a vector $x_{i,j}$ to the cluster which minimizes the following distortion measure: $J_k = ||x_{i,j} - \mu_k||^2$

The mean vector is then updated by computing the mean intensity value of the considered cluster such that

\begin{equation}
\mu_k = \frac{\sum\limits_i \sum\limits_j r_{i,j,k}x_{i,j}}{\sum\limits_i \sum\limits_j r_{i,j}}
\end{equation}

where

\begin{equation}
r_{i,j,k}=\begin{cases}
    1, & \text{if $k = \arg_k \min(||x_{i,j} - \mu_k||^2)$}\\
    0, & \text{otherwise}.
  \end{cases}
\end{equation}

(see also Lecture 13.2 — Clustering | KMeans Algorithm — [ Machine Learning | Andrew Ng ] https://www.youtube.com/watch?v=hDmNF9JG3lo for more details)

## Dataset

Let's generate a two-dimensional dataset that we can easily visualize.

Note that the first 7 points are similar to each other (therefore they would belong to cluster 0) while the next 7 points are also similar to each other (they would belong to cluster 1)

```python
# Generate a dataset
data = np.array([[2,2], [2,3], [3,2], [3,3], [5,4], [1,1], [6,4], [15,12], [14,14], [18,14], [25,30], [21,17], [18,16], [12,18]])

# Inspect the shape
data.shape
```

    (14, 2)

```python
# Plot the data
## YOUR CODE
plt.scatter(data[:,0], data[:,1])
```

## Clustering by distances

**Task** Complete the implementation of the functions `distance`, `predict_cluster` and `predict_clusters`, which are used within the k-means algorithm.

```python
# Define a distance function as the Euclidean distance
def distance(pt1, pt2):

    ## YOUR CODE
    return np.sqrt(np.sum((pt1 - pt2)**2))

# Test
distance(data[0], data[1])
```

    1.0

```python
# Return a cluster index of a single data point, given a list of cluster centers
def predict_cluster(pt1, centers):

    ## YOUR CODE
    dist = [distance(pt1, mu) for mu in centers]

    ix = np.argmin(dist)

    return ix

# Test. This should return 0
predict_cluster(data[0], [ np.array([1,2]), np.array([20,20]) ] )
```

    0

```python
# Return a list of cluster indexes (cluster_labels)
def predict_clusters(data, centers):
    ## YOUR CODE
    return np.array([predict_cluster(pt, centers) for pt in data])

# Test. This should return an array of labels: array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1])
predict_clusters(data, [ np.array([1,2]), np.array([20, 20]) ] )
```

    array([0, 0, 0, 0, 0, 0, 0, 1, 1, 1, 1, 1, 1, 1])

## Visualization

The code below assigns colors to each cluster: Cluster 0 will be blue, Cluster 1 will be red, etc.

```python
def plot_clusters(data, centers):

    # Get the cluster indices
    cluster_labels = predict_clusters(data, centers)

    fig, ax = plt.subplots()
    colors = ['blue', 'red', 'green', 'orange']
    for ix, center in enumerate(centers):
        cdata = data[cluster_labels == ix]
        ax.scatter(cdata[:,0], cdata[:,1], color=colors[ix], label=f'Cluster {ix}')
        ax.scatter(center[0], center[1], marker='x', color=colors[ix], label=f'Cluster center {ix}')
    ax.legend()

plot_clusters(data,  [ np.array([4,2]), np.array([20, 20]) ] )
```

## Algorithm

**Task**

Complete the implementation of the function below, which performs one iteration of k-means, i.e.:

- Finding the center for each data point
- Calculating the mean of the data points associated with each cluster.

```python
# This will perform one iteration of the algorithm
def calculate_new_centers(data, centers):
    """
    Args:
        data (array-like): points with shape (N points, D features)
        centers (list): initial cluster centers

    Returns:
        new_centers (list): new centers

    """

    ## YOUR CODE

    # List of indices
    cluster_labels = predict_clusters(data, centers)

    # Create new centers based on the means of the clusters
    new_centers = []
    for ix in range(len(centers)):
        cdata = data[cluster_labels == ix]
        mean = np.mean(cdata, axis=0)
        new_centers.append(mean)

    return new_centers

calculate_new_centers(data, data[2:4])
```

    [array([2.        , 1.66666667]), array([12.63636364, 12.27272727])]

We'll iterate the algorithm 4 times

```python
# Randomly initialize the means to some value (x1, x2) within our two-dimensional space
means0 = np.array([[np.random.random()*30,np.random.random()*30], [np.random.random()*30,np.random.random()*30]])

for it in range(4):
    # Plot the current solution
    plot_clusters(data, means0)
    plt.title(f'Iteration: {it}')

    # Next solution
    means0 = calculate_new_centers(data, means0)
```

```python
# Final result
means0
```

    [array([17.57142857, 17.28571429]), array([3.14285714, 2.71428571])]

```python
# Final cluster assignments
predict_clusters(data, means0)
```

    array([1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0])

**Task**

Repeat the above iterations a few times. What are potential problems with the algorithm?

## K-Means with sklearn

In practice we often use algorithms from well-developed packages.

```python
# We now import the KMeans module in sklearn
from sklearn.cluster import KMeans
```

```python
# We can use a one-liner to fit our data using 2 clusters
kmeans = KMeans(n_clusters=2, random_state=0).fit(data)
```

```python
# The final means of each cluster
kmeans.cluster_centers_
```

    array([[17.57142857, 17.28571429],
           [ 3.14285714,  2.71428571]])

```python
# Label per data point indicating the final cluster it has been assigned to:
kmeans.labels_
```

    array([1, 1, 1, 1, 1, 1, 1, 0, 0, 0, 0, 0, 0, 0], dtype=int32)

```python
# We can predict new values
kmeans.predict([[0, 0], [17, 15]] )
```

    array([1, 0], dtype=int32)

```python
# Use the same plotting function as before
plot_clusters(data, kmeans.cluster_centers_)
```

**Task**

Try K-means on a smoother dataset.

```python
data2 = np.random.multivariate_normal([5, 5], np.array([[2, 1], [1,2]]), size=1000)
plt.scatter(data2[:,0], data2[:,1])
```

```python
## YOUR CODE
kmeans2 = KMeans(n_clusters=3, random_state=0).fit(data2)
```

```python
# Use the same plotting function as before
plot_clusters(data2, kmeans2.cluster_centers_)
```

# 2. Anomaly Detection

Anomaly detection is the identification of outliers, which do not resemble the bulk of the dataset. Several techniques are used in anomaly detection, including:

*   k Nearest Neighbours
*   DBSCAN
*   Isolation trees

## k Nearest Neighbours Distance

The k Nearest Neighbours distance anomaly detection algorithm assigns a score to each point in the dataset based on its distance from its kth nearest neighbour.

The larger the distance score, the more likely the point is an anomaly.

```python
# Randomly generate points
data = np.random.randint(10, size=(15,2)) + 10

# Add outliers
data = np.vstack((data, np.array([5,5])))

# Inspect the shape
data.shape
```

    (16, 2)

```python
# Visualize the data
plt.scatter(data[:,0], data[:,1])
```

**Task**

Complete the implementation of the function `compute_distances` below.

```python
# Compute distances of each point wrt all other points

def compute_distances(data):
    """"
    Input: array of shape (N points, D features)
    Output: array of shape (N,N)
    """

    ## YOUR CODE
    return np.array([[distance(pt1, pt2) for pt1 in data] for pt2 in data])

distances = compute_distances(data)

#visalize the distances
plt.matshow(distances)
plt.colorbar()
```

```python
# For each point, sort the distances and check which is the (k + 1)th nearest neighbour

def generate_knn_distances(k, distances):

    knn_distances = []

    for i in range(len(data)):

        sorted_distances = np.sort(distances[i,:])

        knn_distances.append(sorted_distances[k+1])

    return np.array(knn_distances)
```

```python
# Plot scatter plot with heat map

knn_distances = generate_knn_distances(3, distances)

plt.scatter(data[:,0], data[:,1], c=knn_distances, edgecolors='black')
plt.colorbar(label='Outlier Score')
```

### Try on some other data

```python
# Randomly generate points

data = np.random.randint(10, size=(15,2)) + 10

# Add outliers

data = np.vstack((data, np.array([[5,5], [6,6]])))

distances = compute_distances(data)

knn_distances = generate_knn_distances(3, distances)

plt.scatter(data[:,0], data[:,1], c=knn_distances, edgecolors='black')
plt.colorbar(label='Outlier Score')
```

```python
# Randomly generate points

data_1 = np.random.randint(10, size=(15,2)) + 10

data_2 = np.random.randint(10, size=(15,2)) + 20

# Add outliers

data = np.vstack((data_1, data_2))

data = np.vstack((data, np.array([[5,5], [6,6]])))

distances = compute_distances(data)

knn_distances = generate_knn_distances(3, distances)

plt.scatter(data[:,0], data[:,1], c=knn_distances, edgecolors='black')
plt.colorbar(label='Outlier Score')
```

# 3. Feature dimensionality reduction

Principal Components Analysis (PCA) is an unsupervised learning technique which reduces the dimensionality of the features.

```python
import pandas as pd
import seaborn as sns
```

```python
# load make_blobs to simulate data
from sklearn.datasets import make_blobs
```

```python
X1, Y1 = make_blobs(n_features=10,
         n_samples=100,
         centers=4, random_state=4,
         cluster_std=2)

# Inspect the shape
X1.shape, Y1.shape
```

    ((100, 10), (100,))

```python
# Check the labels
Y1
```

    array([2, 1, 0, 3, 0, 2, 1, 2, 3, 3, 1, 3, 0, 2, 0, 0, 3, 2, 3, 2, 1, 0,
           0, 1, 0, 0, 2, 0, 0, 2, 0, 1, 2, 3, 3, 0, 2, 1, 3, 2, 1, 3, 1, 2,
           2, 0, 0, 1, 1, 0, 2, 0, 1, 0, 3, 2, 3, 1, 2, 3, 2, 1, 3, 3, 0, 1,
           3, 3, 1, 2, 3, 2, 3, 0, 3, 3, 1, 1, 3, 1, 3, 1, 1, 0, 2, 1, 0, 2,
           0, 2, 1, 2, 0, 3, 2, 0, 1, 3, 1, 2])

```python
# These are the unique ones
set(Y1)
```

    {0, 1, 2, 3}

# Try to visualize
Visualizing this 10-dimensional space is difficult. Try various 2D and 3D projections

```python
# Plot setup
from matplotlib import colors
cmap = plt.cm.rainbow
norm = colors.BoundaryNorm(np.arange(-.5, 4, 1), cmap.N)
```

**Task:**
   Try various projections: 0,1,2, or 3,6,9

```python
plt.scatter(X1[:,2], X1[:,4], c=Y1, cmap=cmap, norm=norm)
plt.colorbar(ticks=[0,1,2,3,4])
```

# PCA decomposition

Principal Component Analysis (see https://en.wikipedia.org/wiki/Principal_component_analysis) can be useful to project a high-dimensional space onto a lower dimensional space.

```python
from sklearn import decomposition
```

```python
# Get the documentation
# ?decomposition.PCA
```

```python
pca = decomposition.PCA(n_components=4)

pc = pca.fit_transform(X1)

pc.shape
```

    (100, 4)

```python
# Organize the data with pandas

pc_df = pd.DataFrame(data = pc,
        columns = ['PC1', 'PC2','PC3','PC4'])
pc_df['Cluster'] = Y1
pc_df
```

      PC1
      PC2
      PC3
      PC4
      Cluster

      0
      -8.133443
      -0.302138
      9.984672
      0.423878
      2

      1
      18.931381
      0.193838
      0.045462
      -3.894042
      1

      2
      -6.571463
      -14.188490
      -3.120115
      1.278588
      0

      3
      -7.533948
      14.439427
      -5.443487
      3.358252
      3

      4
      -4.591760
      -11.315284
      -9.130630
      -1.420151
      0

      ...
      ...
      ...
      ...
      ...
      ...

      95
      -4.734792
      -15.677706
      -4.013717
      1.638843
      0

      96
      14.474724
      -1.170496
      1.650520
      -6.619600
      1

      97
      -6.305432
      13.098273
      -3.843662
      3.060215
      3

      98
      22.169496
      -3.863984
      1.128840
      -0.480688
      1

      99
      -8.071378
      -1.455127
      9.638617
      -2.078444
      2

100 rows × 5 columns

```python
pca.explained_variance_ratio_
```

    array([0.41594854, 0.3391866 , 0.1600729 , 0.02016822])

```python
df = pd.DataFrame({'var':pca.explained_variance_ratio_,
             'PC':['PC1','PC2','PC3','PC4']})
sns.barplot(x='PC',y="var",
           data=df, color="c");
```

```python
sns.lmplot( x="PC1", y="PC2",
  data=pc_df,
  fit_reg=False,
  hue='Cluster', # color by cluster
  legend=True,
  scatter_kws={"s": 80}) # specify the point size
```

```python
from mpl_toolkits.mplot3d import Axes3D

fig = plt.figure()
ax = fig.add_subplot(111, projection='3d')
ax.scatter(pc_df['PC1'], pc_df['PC2'], pc_df['PC3'], c=pc_df['Cluster'])
#ax.legend()
ax.set_xlabel("PC1")
ax.set_ylabel("PC2")
ax.set_zlabel("PC3")
```

# Homework

**Homework 8-1**

Perform K-means clustering on both the original and the PCA-reduced blob dataset. Do you get the same results?

Note: the specific cluster index assigned by K-means to a given cluster (e.g. `0` or `1` or `2`) may differ (even between repeated calls of k-means on the same dataset), but the value of these specific indices do not carry any meaning: the important point is whether K-means finds the same separate clusters. (Irrespective of how they are labeled.)

```python
## YOUR CODE
```

**Homework 8-2**

In K-means, we need to know the number of clusters in advance. Try using the DBSCAN algorithm instead, once again on both the original and the PCA-reduced dataset, and see if you get the same results.

(Note: DBSCAN requires the eps parameter to be set, i.e. the maximum distance between two samples for them to be considered as in the same neighborhood)

```python
## YOUR CODE
```


```python
import numpy as np
from scipy.fftpack import dct, idct

import matplotlib
import seaborn as sns
import matplotlib.pyplot as plt
import matplotlib.gridspec as gridspec
from music21 import converter
matplotlib.style.use('styles.mplstyle')

from helpers import cm2inch, title
```

## Complex roots of unity

```python
def omega(N, n):
    return np.exp(n * 2*np.pi*(0+1j) / N)
```

```python
N = 5
roots = np.array([ omega(N, k) for k in range(N)])

plt.figure(figsize=cm2inch(8.2, 7))
circle = plt.Circle((0,0), 1, ec='.7', fc='w', ls='--', lw=.5)
plt.gca().add_artist(circle)

for k, r in enumerate(roots):
    x, y = np.real(r), np.imag(r)
    plt.plot([0, x], [0, y], 'k--', lw=.5, alpha=.3)
    plt.plot(x, y, 'o', c='C3', ms=5)
    plt.annotate(f'$\omega_{N}^{k} = e^\\frac{{{k} \cdot 2\pi i}}{{{N}}}$',
                 xy=(x,y), xytext=(10,0), textcoords='offset points')

plt.axis('equal');
plt.xlim(-1.4, 1.4)
plt.ylim(-1.4, 1.4)
plt.ylabel('imaginary part')
plt.xlabel('real part')
title(f'Complex {N}-th roots of unity')
plt.tight_layout()
plt.savefig('../figures/suppl-S3/fig-roots-of-unity.pdf')
```

## Cosine eigenvectors

```python
def show_eigenvector(N, n):
    colors = ['.5'] + [f'C{i}' for i in range(9)]
    cos = lambda xs: np.cos(xs * np.pi * n / N)
    xs = np.arange(0, N)
    plt.plot(xs, cos(xs), '.-', lw=.5, c=colors[n])

    xs = np.array([N-1, N])
    plt.plot(xs, cos(xs), '.:', c=colors[n], lw=.5)

    plt.axhline(0, ls='--', c='.75', zorder=-1, lw=.5)

plt.figure(figsize=cm2inch(8.2, 3.5))
show_eigenvector(10, 2)
```

```python
fig = plt.figure(figsize=cm2inch(8.2, 6), tight_layout=True)
max_n = 5
gs = gridspec.GridSpec(max_n, 2)

for n in range(max_n):
    fig.add_subplot(gs[n, 0])
    show_eigenvector(10, n)
    plt.ylim(-1.1, 1.1)
    plt.ylabel(f'$\mathbf{{v}}_{n}$')

    if n < max_n-1: plt.xticks([])
    if n == 0: title('$N = 10$')
    if n == max_n - 1: plt.xlabel('time step')

for n in range(max_n):
    fig.add_subplot(gs[n, 1])
    show_eigenvector(30, n)
    plt.yticks([])
    plt.ylim(-1.1, 1.1)

    if n < max_n-1: plt.xticks([])
    if n == 0: title('$N = 30$')
    if n == max_n - 1: plt.xlabel('time step')

plt.tight_layout(pad=0)
plt.savefig('../figures/suppl-S3/fig-cosines.pdf')
```

## Autoregressive process AR(1)

We look at the (unnormalized) autocovariance matrix
\begin{align}
[\Psi(\rho)]_{ij} := \rho^{|j-i|}
\end{align}
from Ahmed et al (1974). These are Toeplitz matrices

```python
def get_psi(N, rho):
    psi = np.eye(N)
    for i, j in zip(*np.triu_indices(N)):
        psi[i, j] = rho**(j-i)
        psi[j, i] = rho**(j-i)
    return psi

# get_psi(5, .99)
```

```python
fig = plt.figure(figsize=cm2inch(8.2, 6))
gs = gridspec.GridSpec(3, 2)

def show_col(col, rho, N=200):
    ax1 = fig.add_subplot(gs[:2, col])
    psi = get_psi(N, rho)
    plt.imshow(psi)
    title(f'Covariance $\\rho = {rho}$')

    ax1 = fig.add_subplot(gs[2, col])
    lam, V = np.linalg.eig(psi)
    top_lambdas = lam.argsort()[::-1][1:4]
    plt.plot(V[:, top_lambdas])
    title('Eigenvectors')

# show_col(0, .1)
# show_col(1, .5)
show_col(0, .95)
show_col(1, .999)

plt.tight_layout()
plt.savefig('../figures/suppl-S3/fig-ar1.pdf')
```

```python

```


# A CKAR(2) model
Here we demonstrate the package's utilities in the context of a simple censored and kinked AR(2) model with two regimes.

Consider the model
$$\begin{align}
y_t^*&=\phi_1^*y_{t-1}^*+\phi_1y_{t-1}+\phi_2^*y_{t-1}^*+\phi_2y_{t-1}+\epsilon_t,\\
y_t&=\max\{y_t^*,0\}.
\end{align}$$

This features a 'positive regime' ($y^*\geq0$) and a 'negative regime' ($y^*<0$).

Noting $y_t=\mathbf{1}\{y_t^*\geq0\}y_t^*$, we can rewrite the equation for $y_t^*$ as

$$y_t^*=(\phi_1^*+\phi_1\mathbf{1}\{y_{t-1}\geq0\})y_{t-1}^*+(\phi_2^*+\phi_2\mathbf{1}\{y_{t-2}\geq0\})y_{t-2}^*+\epsilon_t,$$

where $\mathbf{1}\{y\geq0\}$ is an indicator variable.

```julia
using ThresholdStability

function AR2_to_TAR(ϕs, ϕ_stars)
    ϕ1, ϕ2 = ϕs
    ϕ1_star, ϕ2_star = ϕ_stars
    Σ = []
    vals = [1, -1]
    for val1 in vals
        for val2 in vals
            A = zeros(2, 2); A[2, 1] = 1
            A[1, 1] = ϕ1_star + ϕ1 * indicator(val1, 0)
            A[1, 2] = ϕ2_star + ϕ2 * indicator(val2, 0)
            push!(Σ, A)
        end
    end
    return Vector{Array{Float64, 2}}(Σ)
    # NOTE Σ is of form where Σ[1] is with yₜ₋₁,yₜ₋₂ ≥ 0, Σ[2] is with yₜ₋₁ ≥ 0 and yₜ₋₂ < 0, Σ[3] is with yₜ₋₁ < 0 and yₜ₋₂ ≥ 0
    # and Σ[4] is with yₜ₋₁,yₜ₋₂ < 0.
end
```

    AR2_to_TAR (generic function with 1 method)

This yields a set of four $2\times2$ matrices, with each matrix corresponding to a different regime pair (positive-positive, positive-negative, negative-positive and negative-negative.)

For exposition, consider $\phi_1 = 0.4$, $\phi_1^* = 0.2$, $\phi_2=0.2$, and $\phi_2^*=0.1$:

```julia
Σ4 = AR2_to_TAR([0.4, 0.2], [0.2, 0.1])
```

    4-element Array{Array{Float64,2},1}:
     [0.6000000000000001 0.30000000000000004; 1.0 0.0]
     [0.6000000000000001 0.1; 1.0 0.0]
     [0.2 0.30000000000000004; 1.0 0.0]
     [0.2 0.1; 1.0 0.0]

The state space constraints for this model are given by `X4`:

```julia
E1, E2, E3, E4 = [1 0.; 0 1.], [1 0.; 0 -1.], [-1 0.; 0 1.], [-1 0.; 0 -1.]
D1 = zeros(1,2); D2, D3, D4 = copy(D1), copy(D1), copy(D1)
X4 = [[E1, D1], [E2, D2], [E3, D3], [E4, D4]]
```

    4-element Array{Array{Array{Float64,2},1},1}:
     [[1.0 0.0; 0.0 1.0], [0.0 0.0]]
     [[1.0 0.0; 0.0 -1.0], [0.0 0.0]]
     [[-1.0 0.0; 0.0 1.0], [0.0 0.0]]
     [[-1.0 0.0; 0.0 -1.0], [0.0 0.0]]

and the automaton can be constructed using

```julia
G = automaton_constructor(Σ4)
```

    LightAutomaton{LightGraphs.SimpleGraphs.SimpleDiGraph{Int64},LightGraphs.SimpleGraphs.SimpleEdge{Int64}}({4, 8} directed simple Int64 graph, Dict{LightGraphs.SimpleGraphs.SimpleEdge{Int64},Dict{Int64,Int64}}(Edge 4 => 2 => Dict(7 => 4),Edge 1 => 3 => Dict(2 => 1),Edge 3 => 4 => Dict(6 => 3),Edge 1 => 1 => Dict(1 => 1),Edge 2 => 1 => Dict(3 => 2),Edge 2 => 3 => Dict(4 => 2),Edge 3 => 2 => Dict(5 => 3),Edge 4 => 4 => Dict(8 => 4)), 8, 8)

We then have the discrete system

```julia
s4 = discreteswitchedsystem(Σ4, G, X4)
```

    Hybrid System with automaton LightAutomaton{LightGraphs.SimpleGraphs.SimpleDiGraph{Int64},LightGraphs.SimpleGraphs.SimpleEdge{Int64}}({4, 8} directed simple Int64 graph, Dict{LightGraphs.SimpleGraphs.SimpleEdge{Int64},Dict{Int64,Int64}}(Edge 4 => 2 => Dict(7 => 4),Edge 1 => 3 => Dict(2 => 1),Edge 3 => 4 => Dict(6 => 3),Edge 1 => 1 => Dict(1 => 1),Edge 2 => 1 => Dict(3 => 2),Edge 2 => 3 => Dict(4 => 2),Edge 3 => 2 => Dict(5 => 3),Edge 4 => 4 => Dict(8 => 4)), 8, 8)

Alternatively, by substituting for $y_t$ on the first lag only we can produce a set of two $3\times3$ matrices:

```julia
function AR2_to_companion(ϕs, ϕ_stars)
    ϕ1, ϕ2 = ϕs
    ϕ1_star, ϕ2_star = ϕ_stars
    Σ = []
    vals = [1, -1]
    for val in vals
        A = zeros(3, 3); A[2, 1] = 1; A[1, 2] = ϕ2_star; A[1, 3] = ϕ2
        A[1, 1] = ϕ1_star + ϕ1 * indicator(val, 0)
        A[3, 1] = indicator(val, 0)
        push!(Σ, A)
    end
    return Vector{Array{Float64, 2}}(Σ)

    # NOTE Σ has form s.t. Σ[1] is with yₜ₋₁ ≥ 0 and Σ[2] is with yₜ₋₁ < 0
end

Σ2 = AR2_to_companion([0.4, 0.2], [0.2, 0.1])
Σ2st = [Σ2[1], Σ2[2], Σ2[1], Σ2[2]]
```

    4-element Array{Array{Float64,2},1}:
     [0.6000000000000001 0.1 0.2; 1.0 0.0 0.0; 1.0 0.0 0.0]
     [0.2 0.1 0.2; 1.0 0.0 0.0; 0.0 0.0 0.0]
     [0.6000000000000001 0.1 0.2; 1.0 0.0 0.0; 1.0 0.0 0.0]
     [0.2 0.1 0.2; 1.0 0.0 0.0; 0.0 0.0 0.0]

`Σ2` consists of the two matrices and `Σ2st` corresponds to the same partitioning of the state space as for `Σ4`.

In this latter case, the automaton is again `G` but the state space constraints are now given by `X2`:

```julia
E1, E2, E3, E4 = [1 0 0.; 0 1 0.], [1 0 0.; 0 -1 0.], [-1 0 0.; 0 1 0.], [-1 0 0.; 0 -1 0.]
D1, D3 = [0 1 -1.], [0 1 -1.]; D2, D4 = [0 0 1.], [0 0 1.]
X2 = [[E1, D1], [E2, D2], [E3, D3], [E4, D4]]
```

    4-element Array{Array{Array{Float64,2},1},1}:
     [[1.0 0.0 0.0; 0.0 1.0 0.0], [0.0 1.0 -1.0]]
     [[1.0 0.0 0.0; 0.0 -1.0 0.0], [0.0 0.0 1.0]]
     [[-1.0 0.0 0.0; 0.0 1.0 0.0], [0.0 1.0 -1.0]]
     [[-1.0 0.0 0.0; 0.0 -1.0 0.0], [0.0 0.0 1.0]]

and we construct the discrete switched systems

```julia
s2 = discreteswitchedsystem(Σ2)
s2st = discreteswitchedsystem(Σ2st, G, X2)
```

    Hybrid System with automaton LightAutomaton{LightGraphs.SimpleGraphs.SimpleDiGraph{Int64},LightGraphs.SimpleGraphs.SimpleEdge{Int64}}({4, 8} directed simple Int64 graph, Dict{LightGraphs.SimpleGraphs.SimpleEdge{Int64},Dict{Int64,Int64}}(Edge 4 => 2 => Dict(7 => 4),Edge 1 => 3 => Dict(2 => 1),Edge 3 => 4 => Dict(6 => 3),Edge 1 => 1 => Dict(1 => 1),Edge 2 => 1 => Dict(3 => 2),Edge 2 => 3 => Dict(4 => 2),Edge 3 => 2 => Dict(5 => 3),Edge 4 => 4 => Dict(8 => 4)), 8, 8)

For our choice of parameter values, this model is stable. In particular, the upper bounds on the joint spectral radius (JSR) of `Σ2`, the constrained joint spectral radius (CJSR) of `(Σ4, G)`, and the state-constrained joint spectral radii (SCJSR) of `(Σ4, G, X4)` and `(Σ2st, G, X2)` all agree in value, up to 4 significant figures:

```julia
@show γ_jsr2 = jsr(s2)
@show γ_cjsr4 = cjsr(s4)
@show γ_scjsr4 = sosbound_γ(s4, 2)
@show γ_scjsr2 = sosbound_γ(s2st, 2)
```

    γ_jsr2 = jsr(s2) = 0.9245022503731286
    γ_cjsr4 = cjsr(s4) = 0.9245031320476007
    γ_scjsr4 = sosbound_γ(s4, 2) = 0.9245058059692384
    γ_scjsr2 = sosbound_γ(s2st, 2) = 0.9245089530944824

    0.9245089530944824

Indeed, if $\phi_i,\phi_i^*\geq0$ for $i=1,2$, then the model will be stable if and only if
$\sum_{i=1}^2(\phi_i+\phi_i^*)<1$.

As expected, the model appears stable when plotted:

```julia
using Plots, Distributions
using LaTeXStrings
pyplot()
function simulate_AR2(y0, Σ, T, σ)  # for companion form
    y = zeros(3, T)
    y[:, 1] = y0
    for t in 1:T-1
        ϵ_t = [rand(Normal(0, σ)), 0., 0.]
        if y[1, t] ≥ 0.
            y[:, t+1] = Σ[1]*y[:, t] + ϵ_t
        else
            y[:, t+1] = Σ[2]*y[:, t] + ϵ_t
        end
    end
    return y
end

function plot_AR2(y0, Σ, T, σ; N = 20, row=1)
    ys = []
    ens_means = zeros(T)
    for i in 1:N
        y = simulate_AR2(y0, Σ, T, σ)
        y = y[row,:]
        push!(ys, y)
        ens_means .+= y
    end
    ens_means ./= N  # ensemble means

    E_y = simulate_AR2(y0, Σ, T, 0.)  # calculating deterministic results
    E_y = E_y[row,:]

    plot(ys, color = :grey, alpha = 0.1, label = "")
    plot!(ens_means, color = :grey, linewidth = 2, label = "Ensemble mean")
    plot!(E_y, color = :blue, linewidth = 2, linestyle = :dash, label = "Deterministic")
    plot!(xlabel=L"t", legend=:topright)
end

plot_AR2(3*ones(3), Σ2, 200, 1)
plot!(ylabel=L"y^*", yguidefontrotation=-90)
```

Introducing negative parameters, we see that the different upper bounds can diverge.

Consider $\phi_1=0.5$, $\phi_1^*=0.5$, $\phi_2=-0.47$, $\phi_2^*=-0.5$. When plotted, the system appears stable:

```julia
Σ = AR2_to_companion([0.5, -0.47], [0.5, -0.5])
plot_AR2(3*ones(3), Σ, 200, 1, row=1)
plot!(ylabel=L"y^*", yguidefontrotation=-90)
```

However, this is a case where the conservatism of (the upper bounds on) the JSR and CJSR would prevent us from concluding the system is stable.

```julia
Σ = AR2_to_TAR([0.5, -0.47], [0.5, -0.5])
E1, E2, E3, E4 = [1 0.; 0 1.], [1 0.; 0 -1.], [-1 0.; 0 1.], [-1 0.; 0 -1.]
D1 = zeros(1,2); D2, D3, D4 = copy(D1), copy(D1), copy(D1)
X = [[E1, D1], [E2, D2], [E3, D3], [E4, D4]]  # state space constraints
G = automaton_constructor(Σ)
s = discreteswitchedsystem(Σ, G, X)
@show γ_jsr = jsr(s)  # > 1
@show γ_cjsr = cjsr(s)  # > 1
```

    γ_jsr = jsr(s) = 1.1051277501200645
    γ_cjsr = cjsr(s) = 1.0010692986341705

    1.0010692986341705

However, the upper bound on the SCJSR is less conservative, yielding a bound below 1. We can thus conclude the system is stable.

```julia
@show γ_scjsr = sosbound_γ(s, 2)  # < 1
```

    γ_scjsr = sosbound_γ(s, 2) = 0.9848907470703125

    0.9848907470703125

Note that (the upper bound on) the SCJSR remains conservative, even if less so than the CJSR or JSR.

For example, consider $\phi_1=\phi_1^*=0.6$ and $\phi_2=\phi_2^*=-0.6$. This appears stable:

```julia
Σ = AR2_to_companion([0.6, -0.6], [0.6, -0.6])
plot_AR2(3*ones(3), Σ, 200, 1, row=1)
plot!(ylabel=L"y^*", yguidefontrotation=-90)
```

Plots for other initial values $y_0$ follow a similar pattern.

However, the bound on the SCJSR exceeds 1 in this case (as does the bound for the CJSR)

```julia
Σ = AR2_to_TAR([0.6, -0.6], [0.6, -0.6])
s = discreteswitchedsystem(Σ, G, X)
@show γ_cjsr = cjsr(s)
@show γ_scjsr = sosbound_γ(s, 2)
```

    γ_cjsr = cjsr(s) = 1.118374537305998
    γ_scjsr = sosbound_γ(s, 2) = 1.0954513549804688

    1.0954513549804688


# 01 - Introduction to seismic modelling

This notebook is the first in a series of tutorials highlighting various aspects of seismic inversion based on Devito operators. In this first example we aim to highlight the core ideas behind seismic modelling, where we create a numerical model that captures the processes involved in a seismic survey. This forward model will then form the basis for further tutorials on the implementation of inversion processes using Devito operators.

## Modelling workflow

The core process we are aiming to model is a seismic survey, which consists of two main components:

- **Source** - A source is positioned at a single or a few physical locations where artificial pressure is injected into the domain we want to model. In the case of land survey, it is usually dynamite blowing up at a given location, or a vibroseis (a vibrating engine generating continuous sound waves). For a marine survey, the source is an air gun sending a bubble of compressed air into the water that will expand and generate a seismic wave.
- **Receiver** - A set of microphones or hydrophones are used to measure the resulting wave and create a set of measurements called a *Shot Record*. These measurements are recorded at multiple locations, and usually at the surface of the domain or at the bottom of the ocean in some marine cases.

In order to create a numerical model of a seismic survey, we need to solve the wave equation and implement source and receiver interpolation to inject the source and record the seismic wave at sparse point locations in the grid.

## The acoustic seismic wave equation
The acoustic wave equation for the square slowness $m$, defined as $m=\frac{1}{c^2}$, where $c$ is the speed of sound in the given physical media, and a source $q$ is given by:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) = q \ \text{in } \Omega \\
 &u(.,t=0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

with the zero initial conditions to guarantee unicity of the solution.
The boundary conditions are Dirichlet conditions:
\begin{equation}
 u(x,t)|_\delta\Omega = 0
\end{equation}

where $\delta\Omega$ is the surface of the boundary of the model $\Omega$.

# Finite domains

The last piece of the puzzle is the computational limitation. In the field, the seismic wave propagates in every direction to an "infinite" distance. However, solving the wave equation in a mathematically/discrete infinite domain is not feasible. In order to compensate, Absorbing Boundary Conditions (ABC) or Perfectly Matched Layers (PML) are required to mimic an infinite domain. These two methods allow to approximate an infinite media by damping and absorbing the waves at the limit of the domain to avoid reflections.

The simplest of these methods is the absorbing damping mask. The core idea is to extend the physical domain and to add a Sponge mask in this extension that will absorb the incident waves. The acoustic wave equation with this damping mask can be rewritten as:

\begin{cases}
 &m \frac{d^2 u(x,t)}{dt^2} - \nabla^2 u(x,t) + \eta \frac{d u(x,t)}{dt}=q  \ \text{in } \Omega \\
 &u(.,0) = 0 \\
 &\frac{d u(x,t)}{dt}|_{t=0} = 0
\end{cases}

where $\eta$ is the damping mask equal to $0$ inside the physical domain and increasing inside the sponge layer. Multiple choice of profile can be chosen for $\eta$ from linear to exponential.

# Seismic modelling with devito

We describe here a step by step setup of seismic modelling with Devito in a simple 2D case. We will create a physical model of our domain and define a single source and an according set of receivers to model for the forward model. But first, we initialize some basic utilities.

```python
#NBVAL_IGNORE_OUTPUT
# Adding ignore due to (probably an np notebook magic) bug
import numpy as np
%matplotlib inline
```

## Define the physical problem

The first step is to define the physical model:

- What are the physical dimensions of interest
- What is the velocity profile of this physical domain

We will create a simple velocity model here by hand for demonstration purposes. This model essentially consists of two layers, each with a different velocity: $1.5km/s$ in the top layer and $2.5km/s$ in the bottom layer. We will use this simple model a lot in the following tutorials, so we will rely on a utility function to create it again later.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Model, plot_velocity, demo_model

# Define a physical size
shape = (101, 101)  # Number of grid point (nx, nz)
spacing = (10., 10.)  # Grid spacing in m. The domain size is now 1km by 1km
origin = (0., 0.)  # What is the location of the top left corner. This is necessary to define
# the absolute location of the source and receivers

# Define a velocity profile. The velocity is in km/s
v = np.empty(shape, dtype=np.float32)
v[:, :41] = 2.5
v[:, 41:61] = 2.5
v[:, 61:] = 2.5

# With the velocity and model size defined, we can create the seismic model that
# encapsulates this properties. We also define the size of the absorbing layer as 10 grid points
#model = Model(vp=v, origin=origin, shape=shape, spacing=spacing,
#              space_order=2, nbl=10, bcs="damp")

#Preset models change the shapes!!
model = demo_model('marmousi2d-isotropic', data_path="/media/pedrosp/Data/Simulations/devitocodes/data")
field = np.transpose(model.vp.data)

#print(field)
np.save("/media/pedrosp/Data/Simulations/devitocodes/devito/examples/seismic/cemeai/model_marmousi2d-isotropic",field)
plot_velocity(model)
```

# Acquisition geometry

To fully define our problem setup we also need to define the source that injects the wave to model and the set of receiver locations at which to sample the wavefield. The source time signature will be modelled using a Ricker wavelet defined as

\begin{equation}
  q(t) = (1-2\pi^2 f_0^2 (t - \frac{1}{f_0})^2 )e^{- \pi^2 f_0^2 (t - \frac{1}{f_0})}
\end{equation}

To fully define the source signature we first need to define the time duration for our model and the timestep size, which is dictated by the CFL condition and our grid spacing. Luckily, our `Model` utility provides us with the critical timestep size, so we can fully discretize our model time axis as an array:

```python
from examples.seismic import TimeAxis

t0 = 0.  # Simulation starts a t=0
tn = 1000.  # Simulation last 1 second (1000 ms)
dt = model.critical_dt  # Time step from model grid spacing

time_range = TimeAxis(start=t0, stop=tn, step=dt)
```

The source is positioned at a $20m$ depth and at the middle of the $x$ axis ($x_{src}=500m$), with a peak wavelet frequency of $10Hz$.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import RickerSource

f0 = 0.010  # Source peak frequency is 10Hz (0.010 kHz)
src = RickerSource(name='src', grid=model.grid, f0=f0,
                   npoint=1, time_range=time_range)

# First, position source centrally in all dimensions, then set depth
src.coordinates.data[0, :] = np.array(model.domain_size) * .5
src.coordinates.data[0, -1] = 20.  # Depth is 20m

# We can plot the time signature to see the wavelet
src.show()
```

Similarly to our source object, we can now define our receiver geometry as a symbol of type `Receiver`. It is worth noting here that both utility classes, `RickerSource` and `Receiver` are thin wrappers around the Devito's `SparseTimeFunction` type, which encapsulates sparse point data and allows us to inject and interpolate values into and out of the computational grid. As we have already seen, both types provide a `.coordinates` property to define the position within the domain of all points encapsulated by that symbol.

In this example we will position receivers at the same depth as the source, every $10m$ along the x axis. The `rec.data` property will be initialized, but left empty, as we will compute the receiver readings during the simulation.

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import Receiver

# Create symbol for 101 receivers
rec = Receiver(name='rec', grid=model.grid, npoint=101, time_range=time_range)

# Prescribe even spacing for receivers along the x-axis
rec.coordinates.data[:, 0] = np.linspace(0, model.domain_size[0], num=101)
rec.coordinates.data[:, 1] = 20.  # Depth is 20m

# We can now show the source and receivers within our domain:
# Red dot: Source location
# Green dots: Receiver locations (every 4th point)
plot_velocity(model, source=src.coordinates.data,
              receiver=rec.coordinates.data[::4, :])
```

# Finite-difference discretization

Devito is a finite-difference DSL that solves the discretized wave-equation on a Cartesian grid. The finite-difference approximation is derived from Taylor expansions of the continuous field after removing the error term.

## Time discretization

We only consider the second order time discretization for now. From the Taylor expansion, the second order discrete approximation of the second order time derivative is:
\begin{equation}
\begin{aligned}
 \frac{d^2 u(x,t)}{dt^2} = \frac{\mathbf{u}(\mathbf{x},\mathbf{t+\Delta t}) - 2 \mathbf{u}(\mathbf{x},\mathbf{t}) + \mathbf{u}(\mathbf{x},\mathbf{t-\Delta t})}{\mathbf{\Delta t}^2} + O(\mathbf{\Delta t}^2).
\end{aligned}
\end{equation}

where $\mathbf{u}$ is the discrete wavefield, $\mathbf{\Delta t}$ is the discrete
time-step (distance between two consecutive discrete time points) and $O(\mathbf{\Delta
  t}^2)$ is the discretization error term. The discretized approximation of the
second order time derivative is then given by dropping the error term. This derivative is represented in Devito by `u.dt2` where u is a `TimeFunction` object.

## Spatial discretization

We define the discrete Laplacian as the sum of the second order spatial
derivatives in the three dimensions:
\begin{equation}
\begin{aligned}
\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t})= \sum_{j=1}^{j=\frac{k}{2}} \Bigg[\alpha_j \Bigg(&
\mathbf{u}(\mathbf{x+jdx},\mathbf{y},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x-jdx},\mathbf{y},\mathbf{z},\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y+jdy},\mathbf{z},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y-jdy},\mathbf{z}\mathbf{t}) + \\
&\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z+jdz},\mathbf{t})+\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z-jdz},\mathbf{t})\Bigg) \Bigg] + \\
&3\alpha_0 \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}).
\end{aligned}
\end{equation}

This derivative is represented in Devito by `u.laplace` where u is a `TimeFunction` object.

## Wave equation

With the space and time discretization defined, we can fully discretize the wave-equation with the combination of time and space discretizations and obtain the following second order in time and $k^{th}$ order in space discrete stencil to update one grid point at position $\mathbf{x}, \mathbf{y},\mathbf{z}$ at time $\mathbf{t}$, i.e.
\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

```python
# In order to represent the wavefield u and the square slowness we need symbolic objects
# corresponding to time-space-varying field (u, TimeFunction) and
# space-varying field (m, Function)
from devito import TimeFunction

# Define the wavefield with the size of the model and the time dimension
u = TimeFunction(name="u", grid=model.grid, time_order=2, space_order=2)

# We can now write the PDE
pde = model.m * u.dt2 - u.laplace + model.damp * u.dt

# The PDE representation is as on paper
pde
```

$\displaystyle \operatorname{damp}{\left(x,y \right)} \frac{\partial}{\partial t} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial x^{2}} u{\left(t,x,y \right)} - \frac{\partial^{2}}{\partial y^{2}} u{\left(t,x,y \right)} + \frac{\frac{\partial^{2}}{\partial t^{2}} u{\left(t,x,y \right)}}{\operatorname{vp}^{2}{\left(x,y \right)}}$

```python
# This discrete PDE can be solved in a time-marching way updating u(t+dt) from the previous time step
# Devito as a shortcut for u(t+dt) which is u.forward. We can then rewrite the PDE as
# a time marching updating equation known as a stencil using customized SymPy functions
from devito import Eq, solve

stencil = Eq(u.forward, solve(pde, u.forward))
stencil
```

$\displaystyle u{\left(t + dt,x,y \right)} = \frac{- \frac{- \frac{2.0 u{\left(t,x,y \right)}}{dt^{2}} + \frac{u{\left(t - dt,x,y \right)}}{dt^{2}}}{\operatorname{vp}^{2}{\left(x,y \right)}} - \frac{2.0 u{\left(t,x,y \right)}}{h_{y}^{2}} + \frac{u{\left(t,x,y - h_{y} \right)}}{h_{y}^{2}} + \frac{u{\left(t,x,y + h_{y} \right)}}{h_{y}^{2}} - \frac{2.0 u{\left(t,x,y \right)}}{h_{x}^{2}} + \frac{u{\left(t,x - h_{x},y \right)}}{h_{x}^{2}} + \frac{u{\left(t,x + h_{x},y \right)}}{h_{x}^{2}} + \frac{\operatorname{damp}{\left(x,y \right)} u{\left(t,x,y \right)}}{dt}}{\frac{\operatorname{damp}{\left(x,y \right)}}{dt} + \frac{1}{dt^{2} \operatorname{vp}^{2}{\left(x,y \right)}}}$

# Source injection and receiver interpolation

With a numerical scheme to solve the homogenous wave equation, we need to add the source to introduce seismic waves and to implement the measurement operator, and interpolation operator. This operation is linked to the discrete scheme and needs to be done at the proper time step. The semi-discretized in time wave equation with a source reads:

\begin{equation}
\begin{aligned}
\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t+\Delta t}) = &2\mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) - \mathbf{u}(\mathbf{x},\mathbf{y}, \mathbf{z},\mathbf{t-\Delta t}) +\\
& \frac{\mathbf{\Delta t}^2}{\mathbf{m(\mathbf{x},\mathbf{y},\mathbf{z})}} \Big(\Delta \mathbf{u}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) + \mathbf{q}(\mathbf{x},\mathbf{y},\mathbf{z},\mathbf{t}) \Big).
\end{aligned}
\end{equation}

It shows that in order to update $\mathbf{u}$ at time $\mathbf{t+\Delta t}$ we have to inject the value of the source term $\mathbf{q}$ of time $\mathbf{t}$. In Devito, it corresponds the update of $u$ at index $t+1$ (t = time implicitly) with the source of time $t$.
On the receiver side, the problem is either as it only requires to record the data at the given time step $t$ for the receiver at time $time=t$.

```python
# Finally we define the source injection and receiver read function to generate the corresponding code
src_term = src.inject(field=u.forward, expr=src * dt**2 / model.m)

# Create interpolation expression for receivers
rec_term = rec.interpolate(expr=u.forward)
```

# Devito operator and solve
After constructing all the necessary expressions for updating the wavefield, injecting the source term and interpolating onto the receiver points, we can now create the Devito operator that will generate the C code at runtime. When creating the operator, Devito's two optimization engines will log which performance optimizations have been performed:
* **DSE:** The Devito Symbolics Engine will attempt to reduce the number of operations required by the kernel.
* **DLE:** The Devito Loop Engine will perform various loop-level optimizations to improve runtime performance.

**Note**: The argument `subs=model.spacing_map` causes the operator to substitute values for our current grid spacing into the expressions before code generation. This reduces the number of floating point operations executed by the kernel by pre-evaluating certain coefficients.

```python
#NBVAL_IGNORE_OUTPUT
from devito import Operator

op = Operator([stencil] + src_term + rec_term, subs=model.spacing_map)
print(op)
```

    #define _POSIX_C_SOURCE 200809L
    #define START_TIMER(S) struct timeval start_ ## S , end_ ## S ; gettimeofday(&start_ ## S , NULL);
    #define STOP_TIMER(S,T) gettimeofday(&end_ ## S, NULL); T->S += (double)(end_ ## S .tv_sec-start_ ## S.tv_sec)+(double)(end_ ## S .tv_usec-start_ ## S .tv_usec)/1000000;

    #include "stdlib.h"
    #include "math.h"
    #include "sys/time.h"
    #include "xmmintrin.h"
    #include "pmmintrin.h"

    struct dataobj
    {
      void *restrict data;
      int * size;
      int * npsize;
      int * dsize;
      int * hsize;
      int * hofs;
      int * oofs;
    } ;

    struct profiler
    {
      double section0;
      double section1;
      double section2;
    } ;

    int Kernel(struct dataobj *restrict damp_vec, const float dt, const float o_x, const float o_y, struct dataobj *restrict rec_vec, struct dataobj *restrict rec_coords_vec, struct dataobj *restrict src_vec, struct dataobj *restrict src_coords_vec, struct dataobj *restrict u_vec, struct dataobj *restrict vp_vec, const int x_M, const int x_m, const int y_M, const int y_m, const int p_rec_M, const int p_rec_m, const int p_src_M, const int p_src_m, const int time_M, const int time_m, struct profiler * timers)
    {
      float (*restrict damp)[damp_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[damp_vec->size[1]]) damp_vec->data;
      float (*restrict rec)[rec_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[rec_vec->size[1]]) rec_vec->data;
      float (*restrict rec_coords)[rec_coords_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[rec_coords_vec->size[1]]) rec_coords_vec->data;
      float (*restrict src)[src_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[src_vec->size[1]]) src_vec->data;
      float (*restrict src_coords)[src_coords_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[src_coords_vec->size[1]]) src_coords_vec->data;
      float (*restrict u)[u_vec->size[1]][u_vec->size[2]] __attribute__ ((aligned (64))) = (float (*)[u_vec->size[1]][u_vec->size[2]]) u_vec->data;
      float (*restrict vp)[vp_vec->size[1]] __attribute__ ((aligned (64))) = (float (*)[vp_vec->size[1]]) vp_vec->data;

      /* Flush denormal numbers to zero in hardware */
      _MM_SET_DENORMALS_ZERO_MODE(_MM_DENORMALS_ZERO_ON);
      _MM_SET_FLUSH_ZERO_MODE(_MM_FLUSH_ZERO_ON);

      for (int time = time_m, t0 = (time)%(3), t1 = (time + 2)%(3), t2 = (time + 1)%(3); time <= time_M; time += 1, t0 = (time)%(3), t1 = (time + 2)%(3), t2 = (time + 1)%(3))
      {
        /* Begin section0 */
        START_TIMER(section0)
        for (int x = x_m; x <= x_M; x += 1)
        {
          #pragma omp simd aligned(damp,u,vp:32)
          for (int y = y_m; y <= y_M; y += 1)
          {
            float r12 = 1.0/dt;
            float r11 = 1.0/(dt*dt);
            float r10 = 1.0/(vp[x + 2][y + 2]*vp[x + 2][y + 2]);
            u[t2][x + 2][y + 2] = (r10*(-r11*(-2.0F*u[t0][x + 2][y + 2] + u[t1][x + 2][y + 2])) + r12*(damp[x + 1][y + 1]*u[t0][x + 2][y + 2]) + 1.0e-2F*(u[t0][x + 1][y + 2] + u[t0][x + 2][y + 1] + u[t0][x + 2][y + 3] + u[t0][x + 3][y + 2]) - 3.99999991e-2F*u[t0][x + 2][y + 2])/(r10*r11 + r12*damp[x + 1][y + 1]);
          }
        }
        STOP_TIMER(section0,timers)
        /* End section0 */
        /* Begin section1 */
        START_TIMER(section1)
        for (int p_src = p_src_m; p_src <= p_src_M; p_src += 1)
        {
          float posx = -o_x + src_coords[p_src][0];
          float posy = -o_y + src_coords[p_src][1];
          int ii_src_0 = (int)(floor(1.0e-1*posx));
          int ii_src_1 = (int)(floor(1.0e-1*posy));
          int ii_src_2 = (int)(floor(1.0e-1*posy)) + 1;
          int ii_src_3 = (int)(floor(1.0e-1*posx)) + 1;
          float px = (float)(posx - 1.0e+1F*(int)(floor(1.0e-1F*posx)));
          float py = (float)(posy - 1.0e+1F*(int)(floor(1.0e-1F*posy)));
          if (ii_src_0 >= x_m - 1 && ii_src_1 >= y_m - 1 && ii_src_0 <= x_M + 1 && ii_src_1 <= y_M + 1)
          {
            float r1 = 5.99760042078401F*(vp[ii_src_0 + 2][ii_src_1 + 2]*vp[ii_src_0 + 2][ii_src_1 + 2])*(1.0e-2F*px*py - 1.0e-1F*px - 1.0e-1F*py + 1)*src[time][p_src];
            u[t2][ii_src_0 + 2][ii_src_1 + 2] += r1;
          }
          if (ii_src_0 >= x_m - 1 && ii_src_2 >= y_m - 1 && ii_src_0 <= x_M + 1 && ii_src_2 <= y_M + 1)
          {
            float r2 = 5.99760042078401F*(vp[ii_src_0 + 2][ii_src_2 + 2]*vp[ii_src_0 + 2][ii_src_2 + 2])*(-1.0e-2F*px*py + 1.0e-1F*py)*src[time][p_src];
            u[t2][ii_src_0 + 2][ii_src_2 + 2] += r2;
          }
          if (ii_src_1 >= y_m - 1 && ii_src_3 >= x_m - 1 && ii_src_1 <= y_M + 1 && ii_src_3 <= x_M + 1)
          {
            float r3 = 5.99760042078401F*(vp[ii_src_3 + 2][ii_src_1 + 2]*vp[ii_src_3 + 2][ii_src_1 + 2])*(-1.0e-2F*px*py + 1.0e-1F*px)*src[time][p_src];
            u[t2][ii_src_3 + 2][ii_src_1 + 2] += r3;
          }
          if (ii_src_2 >= y_m - 1 && ii_src_3 >= x_m - 1 && ii_src_2 <= y_M + 1 && ii_src_3 <= x_M + 1)
          {
            float r4 = 5.99760084529726e-2F*px*py*(vp[ii_src_3 + 2][ii_src_2 + 2]*vp[ii_src_3 + 2][ii_src_2 + 2])*src[time][p_src];
            u[t2][ii_src_3 + 2][ii_src_2 + 2] += r4;
          }
        }
        STOP_TIMER(section1,timers)
        /* End section1 */
        /* Begin section2 */
        START_TIMER(section2)
        for (int p_rec = p_rec_m; p_rec <= p_rec_M; p_rec += 1)
        {
          float posx = -o_x + rec_coords[p_rec][0];
          float posy = -o_y + rec_coords[p_rec][1];
          int ii_rec_0 = (int)(floor(1.0e-1*posx));
          int ii_rec_1 = (int)(floor(1.0e-1*posy));
          int ii_rec_2 = (int)(floor(1.0e-1*posy)) + 1;
          int ii_rec_3 = (int)(floor(1.0e-1*posx)) + 1;
          float px = (float)(posx - 1.0e+1F*(int)(floor(1.0e-1F*posx)));
          float py = (float)(posy - 1.0e+1F*(int)(floor(1.0e-1F*posy)));
          float sum = 0.0F;
          if (ii_rec_0 >= x_m - 1 && ii_rec_1 >= y_m - 1 && ii_rec_0 <= x_M + 1 && ii_rec_1 <= y_M + 1)
          {
            sum += (1.0e-2F*px*py - 1.0e-1F*px - 1.0e-1F*py + 1)*u[t2][ii_rec_0 + 2][ii_rec_1 + 2];
          }
          if (ii_rec_0 >= x_m - 1 && ii_rec_2 >= y_m - 1 && ii_rec_0 <= x_M + 1 && ii_rec_2 <= y_M + 1)
          {
            sum += (-1.0e-2F*px*py + 1.0e-1F*py)*u[t2][ii_rec_0 + 2][ii_rec_2 + 2];
          }
          if (ii_rec_1 >= y_m - 1 && ii_rec_3 >= x_m - 1 && ii_rec_1 <= y_M + 1 && ii_rec_3 <= x_M + 1)
          {
            sum += (-1.0e-2F*px*py + 1.0e-1F*px)*u[t2][ii_rec_3 + 2][ii_rec_1 + 2];
          }
          if (ii_rec_2 >= y_m - 1 && ii_rec_3 >= x_m - 1 && ii_rec_2 <= y_M + 1 && ii_rec_3 <= x_M + 1)
          {
            sum += 1.0e-2F*px*py*u[t2][ii_rec_3 + 2][ii_rec_2 + 2];
          }
          rec[time][p_rec] = sum;
        }
        STOP_TIMER(section2,timers)
        /* End section2 */
      }

      return 0;
    }

Now we can execute the create operator for a number of timesteps. We specify the number of timesteps to compute with the keyword `time` and the timestep size with `dt`.

```python
#NBVAL_IGNORE_OUTPUT
op(time=time_range.num-1, dt=model.critical_dt)
```

    Operator `Kernel` ran in 0.01 s

    PerformanceSummary([(PerfKey(name='section0', rank=None),
                         PerfEntry(time=0.003076000000000018, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section1', rank=None),
                         PerfEntry(time=1.5000000000000004e-05, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[])),
                        (PerfKey(name='section2', rank=None),
                         PerfEntry(time=0.0004410000000000036, gflopss=0.0, gpointss=0.0, oi=0.0, ops=0, itershapes=[]))])

After running our operator kernel, the data associated with the receiver symbol `rec.data` has now been populated due to the interpolation expression we inserted into the operator. This allows us the visualize the shot record:

```python
#NBVAL_IGNORE_OUTPUT
from examples.seismic import plot_shotrecord

plot_shotrecord(rec.data, model, t0, tn)
```

```python
assert np.isclose(np.linalg.norm(rec.data), 370, rtol=1)
```


## Install R Markdown Packages
install.packages("rmarkdown")

##Basic mathematics
A \cup B	:	$$A \cup B$$
A \cap B	:	$$A \cap B$$
x \in A	:	$$x \in A$$
5 \pm 2	:	$$5 \pm 2$$
\log(x)	:	$$\log(x)$$
\sin(x)	:	$$\sin(x)$$
\sqrt{27}	:	$$\sqrt{27}$$
\overline{x}	:	$$\overline{x}$$
\bar{x}	:	$$\bar{x}$$
\hat{x}	:	$$\hat{x}$$
\frac{k}{n}	:	$$\frac{k}{n}$$
\binom{k}{n}	:	$$\binom{k}{n}$$
\frac{\partial f}{\partial x} :	$$\frac{\partial f}{\partial x}$$

##Equtations
x^2	:	$$x^2$$
x_2	:	$$x_2$$
\lim_{x\to\infty}	:	$$\lim_{x\to\infty}$$
\displaystyle \lim_{x\to\infty}	:	$$\displaystyle \lim_{x\to\infty}$$
\int_0^{\infty} f(x) \; dx	:	$$\int_0^{\infty} f(x) \; dx$$
\displaystyle \int_0^{\infty} f(x) \; dx	:	$$\displaystyle \int_0^{\infty} f(x) \; dx$$

\pm : $$\pm$$
\mp : $$\mp$$
\times : $$\times$$
\div : $$\div$$
\ast : $$\ast$$
\cap : $$\cap$$
\cup : $$\cup$$
\leq : $$\leq$$
\geq : $$\geq$$
\neq : $$\neq$$

## The Greek Alphabet
\alpha A	:	$$\alpha A$$
\beta B	:	$$\beta B$$
\gamma \Gamma	:	$$\gamma \Gamma$$
\delta \Delta	:	$$\delta \Delta$$
\epsilon E	:	$$\epsilon E$$
\varepsilon	:	$$\varepsilon$$
\zeta Z	:	$$\zeta Z$$
\eta  :	$$\eta$$
\theta \Theta	:	$$\theta \Theta$$
\iota I	:	$$\iota I$$
\kappa K	:	$$\kappa K$$
\lambda \Lambda	:	$$\lambda \Lambda$$
\mu M	:	$$\mu M$$
\nu N	:	$$\nu N$$
\xi \Xi	:	$$\xi \Xi$$
\omicron O	:	$$\omicron O$$
\pi \Pi	:	$$\pi \Pi$$
\rho P	:	$$\rho P$$
\sigma \Sigma	:	$$\sigma \Sigma$$
\tau 	:	$$\tau$
\upsilon Y	:	$$\upsilon Y$$
\phi \Phi	:	$$\phi \Phi$$
\varphi	:	$$\varphi$$
\chi X	:	$$\chi X$$
\psi \Psi	:	$$\psi \Psi$$
\omega \Omega	:	$$\omega \Omega$$

## Example
X-Bar ~ N(u, σ/sqrt(n)) :
$$\bar{X} \sim N(\mu, \frac{\sigma}{\sqrt(n)})$$

x=y+3  :
$$\begin{equation} x=y+3
\end{equation}$$

{n+1\choose k} = {n\choose k} + {n \choose k-1} :
$$
{n+1\choose k} = {n\choose k} + {n \choose k-1}
$$

$$
|x| = \left\{ \begin{array}{rl}
 -x &\mbox{ if $x<0$} \\
  x &\mbox{ otherwise}
       \end{array} \right.
$$

```R

```


## Final exam fall 2019
Inverted pendulum

```python
import numpy as np
import sympy as sy
import control.matlab as cm
sy.init_printing(use_latex='mathjax', order='lex')
```

```python
h,w0 = sy.symbols('h,omega0', real=True, positive=True)
s,z = sy.symbols('s,z')
t = sy.symbols('t', real=True)
```

```python
# Zero-order hold sampling
G = w0**2/(s**2 - w0**2)
ystep = sy.inverse_laplace_transform(G/s, s, t)
sy.expand(ystep)
```

$$\frac{\theta\left(t\right)}{2} e^{\omega_{0} t} - \theta\left(t\right) + \frac{\theta\left(t\right)}{2} e^{- \omega_{0} t}$$

```python
Yz = w0**2/(2*w0**2) * ( z/(z-sy.exp(w0*h)) - 2*z/(z-1) + z/(z-sy.exp(-w0*h)))
Uz = z/(z-1)
H = sy.simplify(Yz/Uz)
sy.cancel(H)
```

$$\frac{z e^{2 h \omega_{0}} - 2 z e^{h \omega_{0}} + z + e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1}{2 z^{2} e^{h \omega_{0}} - 2 z e^{2 h \omega_{0}} - 2 z + 2 e^{h \omega_{0}}}$$

```python
num, den = sy.fraction(sy.cancel(H))
den
```

$$2 z^{2} e^{h \omega_{0}} - 2 z e^{2 h \omega_{0}} - 2 z + 2 e^{h \omega_{0}}$$

```python
num = sy.simplify(num/(2*sy.exp(h*w0)))
den = sy.simplify(den/(2*sy.exp(h*w0)))
den, num
```

$$\left ( z^{2} - z e^{h \omega_{0}} - \frac{z}{e^{h \omega_{0}}} + 1, \quad \frac{z}{2} e^{h \omega_{0}} - z + \frac{z}{2 e^{h \omega_{0}}} + \frac{e^{h \omega_{0}}}{2} - 1 + \frac{1}{2 e^{h \omega_{0}}}\right )$$

```python
# Coefficients
dencoeffs = sy.Poly(den, z).all_coeffs()
numcoeffs = sy.Poly(num, z).all_coeffs()
dencoeffs, numcoeffs
```

$$\left ( \left [ 1, \quad \frac{1}{e^{h \omega_{0}}} \left(- e^{2 h \omega_{0}} - 1\right), \quad 1\right ], \quad \left [ \frac{1}{2 e^{h \omega_{0}}} \left(e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1\right), \quad \frac{1}{2 e^{h \omega_{0}}} \left(e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1\right)\right ]\right )$$

```python
# Observable canonical form
Phi = sy.simplify(sy.Matrix([[-dencoeffs[1], 1],[-dencoeffs[2], 0]]))
Gamma2 = sy.trigsimp(sy.cancel(sy.Matrix([[numcoeffs[0]],[numcoeffs[1]]])))
C = sy.Matrix([[1, 0]])
Htest = sy.simplify(C*(z*sy.eye(2)-Phi).inv()*Gamma2)
sy.cancel(H), Htest[0]
```

$$\left ( \frac{z e^{2 h \omega_{0}} - 2 z e^{h \omega_{0}} + z + e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1}{2 z^{2} e^{h \omega_{0}} - 2 z e^{2 h \omega_{0}} - 2 z + 2 e^{h \omega_{0}}}, \quad \frac{z e^{2 h \omega_{0}} - 2 z e^{h \omega_{0}} + z + e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1}{2 \left(z^{2} - 2 z \cosh{\left (h \omega_{0} \right )} + 1\right) e^{h \omega_{0}}}\right )$$

```python
Phi, Gamma2
```

$$\left ( \left[\begin{matrix}2 \cosh{\left (h \omega_{0} \right )} & 1\\-1 & 0\end{matrix}\right], \quad \left[\begin{matrix}\frac{1}{2 e^{h \omega_{0}}} \left(e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1\right)\\\frac{1}{2 e^{h \omega_{0}}} \left(e^{2 h \omega_{0}} - 2 e^{h \omega_{0}} + 1\right)\end{matrix}\right]\right )$$

```python
Gamma = sy.Matrix([[sy.cosh(h*w0)-1],[sy.cosh(h*w0)-1]])
sy.simplify(Gamma2-Gamma)
```

$$\left[\begin{matrix}0\\0\end{matrix}\right]$$

```python
Phi,Gamma
```

$$\left ( \left[\begin{matrix}2 \cosh{\left (h \omega_{0} \right )} & 1\\-1 & 0\end{matrix}\right], \quad \left[\begin{matrix}\cosh{\left (h \omega_{0} \right )} - 1\\\cosh{\left (h \omega_{0} \right )} - 1\end{matrix}\right]\right )$$

```python
# Some numerical values
w0n = 1.0
hn = 0.2/w0n
Phi_n = np.asarray(Phi.subs({w0:w0n, h:hn})).astype(np.float64)
Gamma_n = np.asarray(Gamma.subs({w0:w0n, h:hn})).astype(np.float64)
Phi_n, Gamma_n
```

    (array([[ 2.04013351,  1.        ],
            [-1.        ,  0.        ]]), array([[0.02006676],
            [0.02006676]]))

```python
C = np.array([1., 0])
D = np.array([[0]])
sys = cm.ss(Phi_n, Gamma_n, C, D, hn)
cm.pole(sys)
```

    array([1.22140276, 0.81873075])

```python
# Desired poles: double pole in -2w0 (continuous time)
pd = np.exp(-2*w0n*hn)
L = cm.acker(A=Phi_n, B=Gamma_n,poles = [pd,pd])
L
```

    matrix([[19.43797021, 15.42035131]])

```python
L2 = np.array([[19,15]])
np.linalg.eig(Phi_n - np.dot(Gamma_n, L2))
```

    (array([0.67893191+0.07235727j, 0.67893191-0.07235727j]),
     array([[-0.57809328-0.04268582j, -0.57809328+0.04268582j],
            [ 0.81485341+0.j        ,  0.81485341-0.j        ]]))

```python
# Closed-loop system
Phic = Phi_n - np.dot(Gamma_n, L2)
sys_c = cm.ss(Phic, Gamma_n, C, D, hn)
# Steady-state solution
cm.dcgain(sys_c)
```

    array(0.37050778)

```python
l0=1/0.3705
```

```python
import matplotlib.pyplot as plt
tvec = np.arange(0,40)*hn
yout, tout =  cm.step(sys_c*2.7, tvec)
plt.figure(figsize=(13,4))
plt.plot(tout, yout)
```

```python
cm.tf(sys_c*2.7)
```

      0.05418 z + 0.05418
    ----------------------
    z^2 - 1.358 z + 0.4662

    dt = 0.2

```python
cm.pole(sys_c)
```

    array([0.67893191+0.07235727j, 0.67893191-0.07235727j])

```python
# Finding the deadbeat observer
CT = C.copy()
CT.shape = (2,1)
KT = cm.acker(Phi_n.T, CT, [0,0])
K = KT.T
K
```

    matrix([[ 2.04013351],
            [-1.        ]])

```python
# Verify
sys_o = cm.ss(Phi_n - np.dot(K, CT.T), Gamma_n, C, D, hn)
cm.pole(sys_o)
```

    array([0., 0.])

### From output feedback with observer to 2dof controller

```python
D_o = np.array([[l0, 0]])
sys_o = cm.ss(Phi_n - np.dot(K, CT.T)-np.dot(L2, Gamma_n),
              np.hstack((l0*Gamma_n,K)), -L, D_o, hn)
```

```python
cm.tf(sys_o)
```

    Input 1 to output 1:
    2.699 z^2 + 1.795 z + 0.7887
    ----------------------------
       z^2 + 1.365 z + 0.6823

    Input 2 to output 1:
        -24.24 z + 11.1
    ----------------------
    z^2 + 1.365 z + 0.6823

    dt = 0.2

```python
K
```

    matrix([[ 2.04013351],
            [-1.        ]])

```python
np.hstack((l0*Gamma_n, K))
```

    matrix([[ 0.05416128,  2.04013351],
            [ 0.05416128, -1.        ]])

```python

```


```python
from sympy import init_session
init_session()
```

    IPython console for SymPy 1.3 (Python 3.7.4-64-bit) (ground types: python)

    These commands were executed:
    >>> from __future__ import division
    >>> from sympy import *
    >>> x, y, z, t = symbols('x y z t')
    >>> k, m, n = symbols('k m n', integer=True)
    >>> f, g, h = symbols('f g h', cls=Function)
    >>> init_printing()

    Documentation can be found at http://docs.sympy.org/1.3/

```python
a1, a3 = symbols("a_1 a_3", positive=True)
lam = symbols("\lambda")
```

```python
f = x**2/(a1**2 + lam) + z**2/(a3**2 + lam) - 1
```

```python
f
```

```python
r = solve(f, lam)
```

```python
simplify(r[0])
```

```python
simplify(r[1])
```

```python

```


```python
from sympy import *
from sympy.abc import x, y, m, n, z, r
```

```python
dis = (x**2 - x)/(4 + x) >= 3
dis
```

$\displaystyle \frac{x^{2} - x}{x + 4} \geq 3$

```python
3/x
```

$\displaystyle \frac{3}{x}$

```python
x/(sqrt(sqrt(y + 45/S(16)*(y**2 + 5))))
```

$\displaystyle \frac{x}{\sqrt[4]{\frac{45 y^{2}}{16} + y + \frac{225}{16}}}$

```python
Eq(x, m*y + r, evaluate=False)
```

$\displaystyle x = m y + r$

```python
S(15)/7
```

$\displaystyle \frac{15}{7}$

```python
eq = Eq(15, m*7 + r, evaluate=False)
eq
```

$\displaystyle 15 = 7 m + r$

```python
eq.subs({m:2, r:1})
```

$\displaystyle \text{True}$

$${{15}\over{0}} \quad\leftrightarrow\quad 15 = 0m + r =  r \quad m, r\in\mathbb{Z} \wedge r < 15$$
Se scelgo
$$m=1, r=0 \quad\rightarrow\quad 15 \not = 0*1 + 0 = 0$$

Ritornando al nostro problema iniziale, il denominatore $x+4\not=0$ per ogni $x\in\mathbb{R}$, quindi $x\not=-4$.

```python
dis
```

$\displaystyle \frac{x^{2} - x}{x + 4} \geq 3$

**Intermezzo**, questo che abbiamo scritto e' la condizione di esistenza rispetto alla divisione. Ce ne possono essere altre, ad esempio:

```python
sqrt(x+1)
```

$\displaystyle \sqrt{x + 1}$

La scrittura $\sqrt{y}$ significa che devo trovare un numero $z\in\mathbb{R}$ tale che $y = z^2$. Quindi, ha senso $\sqrt{-3}$? No, perche' non esiste $z\in\mathbb{R}$ tale che $-3=z^2$. Tornando alla scrittura sopra, devo trovare un numero $z\in\mathbb{R}$ tale che $x+1 = z^2$. Qual e' la condizione per cui riesco effettivamente a trovare questo $z$? Quando $x + 1\geq 0$, ovvero $x \geq -1$.

```python
solve(dis)
```

$\displaystyle \left(6 \leq x \wedge x < \infty\right) \vee \left(x \leq -2 \wedge -4 < x\right)$

---

```python
x
```

$\displaystyle x$

```python
expr = abs((x**2 - 9)/(x-1))
expr
```

$\displaystyle \left|{\frac{x^{2} - 9}{x - 1}}\right|$

```python
from sympy.plotting import plot
```

```python
plot(expr,)
```

```python
solve(expr > 5)
```

$\displaystyle \left(-\infty < x \wedge x < -7\right) \vee \left(1 < x \wedge x < 2\right) \vee \left(x < 1 \wedge \frac{5}{2} - \frac{\sqrt{41}}{2} < x\right) \vee \left(x < \infty \wedge \frac{5}{2} + \frac{\sqrt{41}}{2} < x\right)$