# 3.4.1 Ridge Regression

Ridge regression shrinks the regression coefficients by imposing a penalty on their size. The ridge coefficients minimize a penalized RSS (3.41):
$$
\hat{\beta}^{ridge} = \underset{\beta}{argmin} \left\{
\sum_{i=1}^N \left( y_i-\beta_0-\sum_{j=1}^p{x_{ij}\beta_j} \right)^2
+\gamma\sum_{j=1}^p{\beta_j^2}
\right\}
$$
Here $\gamma \ge 0$ is a complexity parameter: $\gamma \rightarrow \infty$, the coefficients are shrunk toward zero (and each other). This idea also used in neural networks (known as *weight decay*).

An equivalent way to write the ridge problem is (3.42):
$$
\begin{equation}
\hat{\beta}^{ridge} = \underset{\beta}{argmin} \left\{
\sum_{i=1}^N \left( y_i-\beta_0-\sum_{j=1}^p{x_{ij}\beta_j} \right)^2
\right\},\\
\text{subject to } \sum_{j=1}^p {\beta_j^2 \le t}
\end{equation}
$$

There is a one-to-one correspondence between $\gamma \text{ and } t$. A large positive coefficient on one variable can be canceled by a similarly large negative coefficient on its correlated cousin. By imposing constraints as in (3.42), this problem is alleviated.

The ridge solutions are not equivariant under scaling of the inputs, so one normally standardizes the inputs before solving (3.41).

Notice that the intercepts has been left out of the penalty term: because, adding a constant to each target $y_i$ would not simply result in a shift of the prediction by the same amount. It can be shown that the solution to (3.41) can be separated into two parts, after reparametrization using *centered* inputs, i.e replacing $x_{ij} \text{ by } x_{ij}-\overline{x}_j$:
1. we estimate $\beta_0 \text{ as } \overline{y}$,
2. The remaining coefficients get estimated by a ridge regression.

*Proof*:

$$
\begin{equation}
\hat{\beta}^{ridge} = \underset{\beta}{argmin} \left\{
\sum_{i=1}^N \left( y_i- \beta_0-\sum_{j=1}^p\overline{x}_j\beta_j -\sum_{j=1}^p{(x_{ij} - \overline{x}_j)\beta_j} \right)^2
+\gamma\sum_{j=1}^p{\beta_j^2}
\right\}\\
\hat{\beta}^{ridge} = \underset{\beta}{argmin} \left\{
\sum_{i=1}^N \left( y_i- \beta_0^C  -\sum_{j=1}^p{(x_{ij} - \overline{x}_j)\beta_j} \right)^2
+\gamma\sum_{j=1}^p{\beta_j^2}
\right\}
\end{equation}
$$

where $\beta_0^C = \beta_0-\sum_{j=1}^p\overline{x}_j\beta_j$. Now the stationary point w.r.t $\beta_0^C$ will be:
$$
\begin{align}
\beta_0^C &= \frac{1}{N}(\sum_{i=1}^N y_i- \sum_{i=1}^N\sum_{j=1}^p{x_{ij}\beta_j +
\sum_{i=1}^N\sum_{j=1}^p\overline{x}_j\beta_j})
\\
&=\frac{1}{N}(\sum_{i=1}^N y_i- \sum_{j=1}^p\beta_j\sum_{i=1}^N x_{ij} +
N\sum_{j=1}^p\overline{x}_j\beta_j)
\\
&=\frac{1}{N}(\sum_{i=1}^N y_i- \sum_{j=1}^p\beta_j\overline{x}_jN +
N\sum_{j=1}^p\overline{x}_j\beta_j)
\\
&=\overline{y}
\end{align}
$$

This completes proof.

We assume that the centering has been done, so that the matrix **X** has p columns (3.43):

$$
RSS(\gamma)=(\mathbf{y} - \mathbf{X}\beta)^T(\mathbf{y} - \mathbf{X}\beta) + \gamma\beta^T\beta
$$

the ridge regression solutions are easily seen to be:
$$
\hat{\beta}^{ridge}=(\mathbf{X}^T\mathbf{X}+\gamma\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}
$$

where **I** is the $p\times p$ identity matrix. Notice that the solution is again a linear function of **y**.

*Proof*:

$$
\begin{align}
\cfrac{\partial{RSS(\gamma)}}{\partial{\beta}}
&=\cfrac{\partial{((\mathbf{y} - \mathbf{X}\beta)^T(\mathbf{y} - \mathbf{X}\beta) + \gamma\beta^T\beta)}}{\partial{\beta}}\\
&=\cfrac{\partial{\left(
  \mathbf{y}^T\mathbf{y} - 2\mathbf{y}^T\mathbf{X}\beta
  + (\mathbf{X}\beta)^T\mathbf{X}\beta
  + \gamma\beta^T\beta
\right)}}{\partial{\beta}}\\
&=-2\mathbf{y}^T\mathbf{X}+2\beta^T\mathbf{X}^T\mathbf{X}+2\gamma\beta^T
\end{align}
$$

We set the first derivative to zero:

$$
\begin{equation}
\cfrac{\partial{RSS(\gamma)}}{\partial{\beta}} = 0\\
-\mathbf{y}^T\mathbf{X}+\beta^T\mathbf{X}^T\mathbf{X}+\gamma\beta^T=0\\
\beta^T(\mathbf{X}^T\mathbf{X}+\gamma\mathbf{I})=\mathbf{y}^T\mathbf{X}\\
\beta^T=\mathbf{y}^T\mathbf{X}(\mathbf{X}^T\mathbf{X}+\gamma\mathbf{I})^{-1}\\
\beta=(\mathbf{X}^T\mathbf{X}+\gamma\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}
\end{equation}
$$

Figure 3.8 shows the ridge coefficient estimates for the prostate cancer example, plotted as functions of $df(\gamma)$, the *effective degrees of freedom* implied by the penalty $\gamma$. In the case of orthonormal inputs, the ridge estimates are: $\hat{\beta}^{ridge}=\hat{\beta}/(1+\gamma)$

**TODO:** Implement Figure 3.8.

Ridge expression can be also derived as the mean or mode of a posterior distribution, with a prior distribution. Suppose, $y_i \sim N(\beta_0+x_i^T\beta, \sigma^2)$ and the $\beta_j \sim N(0, \mathcal{T}^2)$, independently of one another. Then the log-posterior density of $\beta$, is equal to the expression in (3.41), with $\gamma=\sigma^2/\mathcal{T}^2$.

**TODO**: proof.

The *singular value decomposition* (SVD) of the centered input matrix **X** gives us some insight into the nature of ridge regression. The SVD of the $N \times p$ matrix **X** (3.45):
$$
\mathbf{X}=\mathbf{UD}\mathbf{V}^T
$$
- **U** - $N \times p$ orthogonal matrix, the columns of **U** span the column space of **X**.
- **V** - $N \times p$ orthogonal matrix, the columns of **V** span the row space of **X**.
- **D** - $p \times p$ diagonal matrix, with diagonal entries $d_1 \ge d_2 \ge ... \ge d_p \ge 0$, called singular values of **X**. If one or more values $d_j = 0$, **X** is singular.

Using SVD we can write the least squares fitted vector as (3.46):

$$
\begin{align}
\mathbf{X}\hat{\beta}^{ls} &= \mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}\\
&= \mathbf{UD}\mathbf{V}^T(\mathbf{VD}\mathbf{U}^T\mathbf{UD}\mathbf{V}^T)^{-1}\mathbf{VD}\mathbf{U}^T\mathbf{y}\\
&= \mathbf{UD}\mathbf{V}^T(\mathbf{VD}\mathbf{D}\mathbf{V}^T)^{-1}\mathbf{VD}\mathbf{U}^T\mathbf{y}\\
&= \mathbf{UD}\mathbf{V}^T[\mathbf{V}^T]^{-1}\mathbf{D}^{-1}\mathbf{D}^{-1}\mathbf{V}^{-1}\mathbf{VD}\mathbf{U}^T\mathbf{y}\\
&= \mathbf{UD}\mathbf{D}^{-1}\mathbf{D}^{-1}\mathbf{D}\mathbf{U}^T\mathbf{y}\\
&= \mathbf{U}\mathbf{U}^T\mathbf{y}
\end{align}
$$

Now the ridge solutions are (3.47):

$$
\begin{align}
\mathbf{X}\hat{\beta}^{ridge}&=\mathbf{X}(\mathbf{X}^T\mathbf{X}+\gamma\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}\\
&=\mathbf{UD}\mathbf{V}^T
(\mathbf{V}\mathbf{D}^2\mathbf{V}^T+\gamma\mathbf{V}\mathbf{V}^T)^{-1}
\mathbf{VD}\mathbf{U}^T\mathbf{y}\\
&=\mathbf{UD}\mathbf{V}^T
(\mathbf{V}(\mathbf{D}^2+\gamma\mathbf{I})\mathbf{V}^T)^{-1}
\mathbf{VD}\mathbf{U}^T\mathbf{y}\\
&=\mathbf{UD}\mathbf{V}^T
\mathbf{V}(\mathbf{D}^2+\gamma\mathbf{I})^{-1}\mathbf{V}^T
\mathbf{VD}\mathbf{U}^T\mathbf{y}\\
&=\mathbf{UD}(\mathbf{D}^2+\gamma\mathbf{I})^{-1}\mathbf{D}\mathbf{U}^T\mathbf{y}\\
&=\sum_{j=1}^p\mathbf{u}_j\cfrac{d_j^2}{d_j^2+\gamma}\mathbf{u}_j^T\mathbf{y}
\end{align}
$$
- Computes the coordinates of **y** w.r.t the orthonormal basis **U**.
- Then shrinks these coordinates of **y** by the factors  $d_j^2/(d_j^2+\gamma)$.
- A greater amount of shrinkage is applied to the coordinates of basis vectors with smaller $d_j^2$.

**Principal Components**

What does a small value of $d_j^2$ mean? The sample covariance matrix is $\mathbf{S}=\mathbf{X}^T\mathbf{X}/N$ (3.46):
$$
\mathbf{X}^T\mathbf{X} = \mathbf{VD}^2\mathbf{V}^T
$$

which is *eigen decomposition* of $\mathbf{X}^T\mathbf{X}$, aka the *principal components* directions of **X**.

The first principal component (PC) direction $v_1$ has the largest variance, that is:
$$
\begin{align}
Var(\mathbf{z}_1) &= Var(\mathbf{X}v_1)\\
&= \frac{1}{N} v_1^T\mathbf{VD}^2\mathbf{V}^Tv_1\\
&= \frac{1}{N}
   \begin{pmatrix} 1&0 & \dots & 0 \end{pmatrix}
   \mathbf{D}^2
   \begin{pmatrix} 1 \\ 0 \\ \vdots \\ 0 \end{pmatrix}\\
&=\frac{d_1^2}{N}
\end{align}
$$

and in fact $\mathbf{z}_1 = \mathbf{X}v_1 = \mathbf{UD}\mathbf{V}^{T}v_1= \mathbf{u}_1d_1$ and it is called the first PC. Ridge regression shrinks  PC directions having small variance the most.

**Effective degrees of freedom**

In Figure 3.7 we have plotted the estimated prediction error versus the quantity:
$$
\begin{align}
df(\gamma) &= tr[\mathbf{X}(\mathbf{X}^T\mathbf{X}+\gamma\mathbf{I})^{-1}\mathbf{X}^T]\\
&= tr(\mathbf{H_{\gamma}})\\
&= tr[\mathbf{UD}(\mathbf{D}^2+\gamma\mathbf{I})^{-1}\mathbf{D}\mathbf{U}^T]\\
&= \sum_{j=1}^p \cfrac{d_j^2}{d_j^2+\gamma}
\end{align}
$$

This monotone decreasing function of $\gamma$ is the *effective degrees of freedom* of the ridge regression fit. Note that:
- $\gamma = 0 \text{  then  } df(\gamma)=p$
- $\gamma \rightarrow \infty \text{  then  } df(\gamma) \rightarrow 0$
- Although all p coefficients in a ridge fit will be non-zero, they are fit in a restricted fashion controlled by $\gamma$.


# Astrofizyka - praca domowa 1
### Dominik Stańczak

```python
%matplotlib inline
import matplotlib.pyplot as plt
# from sympy.abc import *
import numpy as np
import sympy
sympy.init_printing()
```

```python
N_gwiazd = 1e6
```

$$ f(M) = dN/dM = A_1 M^{-1.3} $$

od 0.08 do 0.5 mas słońca

$$ f(M) = dN/dM = A_2 M^{-2.2} $$

od 0.5 do 1 mas słońca

$$ f(M) = dN/dM = A_3 M^{-2.5} $$

od 1 do 150 mas słońca

Ile spodziewamy się gwiazd neutronowych (powstają z gwiazd o masie początkowej 8-20 mas słońca) i czarnych dziur (20 mas słońca)?

*przyjmuję masę Słońca równą 1.*

```python
A1, A2, A3, M, U = sympy.symbols('A_1, A_2, A_3, M, U', positive=True)
f1 = A1 * M**-1.3
f2 = A2 * M**-2.2
f3 = A3 * M**-2.5

f = sympy.Piecewise((0, M < 0.08),
                    (f1, M <= 0.5),
                    (f2, M <= 1),
                    (f3, M <= 150),
                    (0, True))
f
```

```python
ciągłość = [sympy.Eq(f1, f2).subs({M:0.5}),
            sympy.Eq(f2, f3).subs({M:1})
           ]
ciągłość
```

```python
warunek_normalizacji = sympy.Eq(sympy.Integral(f, (M, 0.08, 150)), 1)
# normalizuję do 1, do prawdopodobienstw
warunek_normalizacji
```

```python
wartości_stałych = sympy.solve(ciągłość + [warunek_normalizacji.doit()], (A1, A2, A3))
wartości_stałych
```

```python
rozkład_prawdopodobieństwa = f.subs(wartości_stałych)
rozkład_prawdopodobieństwa
```

```python
dystrybuanta = rozkład_prawdopodobieństwa.integrate()
równanie = sympy.Eq(U, dystrybuanta) # U ma rozkład jednorodny z zakresu (0, 1)
równanie
```

```python
rozwiązanie = sympy.solve(równanie, M, rational=False)
# rozwiązanie ma dość paskudną formę przez trzy niezerowe przypadki

def squash_solution(solved_eq, x):
    solutions = [sympy.lambdify(U, out, "numpy")(x) for out in solved_eq]
    y = np.zeros_like(x)
    for sol in solutions:
        indices = ~np.isnan(sol)
        y[indices] = sol[indices]
    return y

masy = squash_solution(rozwiązanie, np.random.random(int(N_gwiazd)))
masy
```

    array([1.42039344, 0.22285203, 1.7941953 , ..., 0.36699205, 1.04696562,
           0.31514545])

```python
plt.style.use('grayscale')
plt.figure(figsize=(10,7))
plt.hist(masy, bins=100, log=True);
plt.ylabel("Liczba gwiazd (całość = $10^6$)");
plt.xlabel("Masa gwiazdy (w jednostkach masy słonecznej)");
```

```python
print(f"Średnia masa wylosowanej planety to {np.mean(masy):.3f} ± {np.std(masy):.3f} mas Słońca.")
print(f"{np.sum(masy > 8)} planet ma masę powyżej 8 mas Słońca, a powyżej 20 - {np.sum(masy > 20)}.")
```

    Średnia masa wylosowanej planety to 0.510 ± 1.690 mas Słońca.
    3940 planet ma masę powyżej 8 mas Słońca, a powyżej 20 - 995.

```python
plt.figure(figsize=(10,7))
plt.hist(masy, bins=100, density=True, label="Otrzymane wyniki");
plt.yscale('log')
plt.xlabel("Masa gwiazdy (w jednostkach masy słonecznej)");
plt.ylabel("Rozkład prawdopodobieństwa mas")
x = np.linspace(0.08, 150, 300)
y = sympy.lambdify(M, rozkład_prawdopodobieństwa, 'numpy')(x)
plt.plot(x, y, label="Spodziewana wartość teoretyczna")
plt.legend(loc="upper right")
```


## Statement

$P(\text{score}|\text{dist}=x) = 1 - \dfrac{x}{q}, \ q > 50$

- during each practice run, shoot from distances $x =\overline{1 \dots 50}$
- precisely $2 \%$ chance to score a total of exactly $20$ points
- **find $q$, rounded to $10$ decimal places**

## Solution
The underlying distribution counting $k$ successes from $n$ trials, where the trials are not necessarily identically distributed, is the [Poisson binomial distribution](https://en.wikipedia.org/wiki/Poisson_binomial_distribution).

However, computing $\Pr(K=20)$ using the pmf is infeasible.

### Dynamic programming
Let $P_n(k)$ denote the probability of obtaining $k$ successes in $n$ trials:
- $P_1(0) = 1-p_1$
- $P_1(1) = p_1$
- $P_n(0) = P_{n-1}(0) \cdot (1-p_n)$ (only failures in all trials)
- $P_n(n) = P_{n-1}(n-1) \cdot p_n$ (only successes in all trials)

The general formula is:
$P_n(k) = P_{n-1}(k-1) \cdot p_n + P_{n-1}(k) \cdot (1 - p_n)$

That is, there are two cases:
- get $k-1$ successes in the last $n-1$ trials, and get a final success $p_n$
- get $k$ successes in the last $n-1$ trials, and get a final failure $(1 - p_n)$

In our case: $p_x = 1-\dfrac{x}{q}$ and $P_{50}(20) = 0.02$.

## References
- https://stats.stackexchange.com/questions/177199/success-of-bernoulli-trials-with-different-probabilities
- http://csg.sph.umich.edu/abecasis/class/2006/615.05.pdf

```python
import numpy as np
from sympy import *
```

```python
q = symbols('q')
p = lambda x: 1 - x / q

n = 50 # trials
k = 20 # successes

P = [[0 for _ in range(n + 1)] for _ in range(n + 1)]

P[1][0] = 1 - p(1)
P[1][1] = p(1)

for cur_n in range(2, n + 1):
    P[cur_n][0] = P[cur_n - 1][0] * (1 - p(cur_n))

for cur_n in range(2, n + 1):
    P[cur_n][cur_n] = P[cur_n - 1][cur_n - 1] * p(cur_n)

for cur_n in range(2, n + 1):
    for cur_k in range(1, min(k + 1, cur_n + 1)):
        P[cur_n][cur_k] = expand(
            P[cur_n - 1][cur_k - 1] * p(cur_n) +
            P[cur_n - 1][cur_k] * (1 - p(cur_n))
        )
```

```python
eq = lambda val: P[n][k].evalf(subs={q: val})
```

```python
# q is between these 2:
lo = 50
hi = 53
target = 0.02 # P[n][k] must equal this

steps = 0
precision = 1e-16

while not np.isclose(lo - hi, 0, rtol=0, atol=precision):
    steps += 1
    mid = lo + (hi - lo) / 2
    val = float(eq(mid))

    if np.isclose(val, target, rtol=0, atol=precision):
        print(f"{val=}")
        print(f"{round(mid, 10)=}")
        print(f"{steps=}")
        break

    elif val > 0.02:
        lo = mid #+ delta

    elif val < 0.02:
        hi = mid #- delta

    else:
        assert False, "unreachable"
```

```python

```


```python
%config InlineBackend.figure_format = 'retina'
import numpy as np
import matplotlib.pyplot as plt
import pandas as pd
import seaborn as sns
```

# Product of 4 consecutive numbers is always 1 less than a perfect square

Shubhanshu Mishra (shubhanshu.com)

For every $n \in \mathbb{Z}$, we can have 4 consecutive numbers as follows:

$
n, n+1, n+2, n+3
$

We can complete the proof, if we can show that there exists a $k \in \mathbb{Z}$, such that the following equation holds:

$
\begin{equation}
n*(n+1)*(n+2)*(n+3) = (k^2 - 1)
\end{equation}
$

```python
i_max = 4
nums = np.arange(0, 50)+1
consecutive_nums = np.stack([
    np.roll(nums, -i)
    for i in range(i_max)
], axis=1)[:-i_max+1]
n_prods = consecutive_nums.prod(axis=1)
df = pd.DataFrame(consecutive_nums, columns=[f"n{i+1}" for i in range(i_max)])
df["prod"] = n_prods
df["k"] = np.sqrt(n_prods+1).astype(int)
df["k^2"] = df["k"]**2
df["k^2 - 1"] = df["k^2"] - 1
df
```

      n1
      n2
      n3
      n4
      prod
      k
      k^2
      k^2 - 1

      0
      1
      2
      3
      4
      24
      5
      25
      24

      1
      2
      3
      4
      5
      120
      11
      121
      120

      2
      3
      4
      5
      6
      360
      19
      361
      360

      3
      4
      5
      6
      7
      840
      29
      841
      840

      4
      5
      6
      7
      8
      1680
      41
      1681
      1680

      5
      6
      7
      8
      9
      3024
      55
      3025
      3024

      6
      7
      8
      9
      10
      5040
      71
      5041
      5040

      7
      8
      9
      10
      11
      7920
      89
      7921
      7920

      8
      9
      10
      11
      12
      11880
      109
      11881
      11880

      9
      10
      11
      12
      13
      17160
      131
      17161
      17160

      10
      11
      12
      13
      14
      24024
      155
      24025
      24024

      11
      12
      13
      14
      15
      32760
      181
      32761
      32760

      12
      13
      14
      15
      16
      43680
      209
      43681
      43680

      13
      14
      15
      16
      17
      57120
      239
      57121
      57120

      14
      15
      16
      17
      18
      73440
      271
      73441
      73440

      15
      16
      17
      18
      19
      93024
      305
      93025
      93024

      16
      17
      18
      19
      20
      116280
      341
      116281
      116280

      17
      18
      19
      20
      21
      143640
      379
      143641
      143640

      18
      19
      20
      21
      22
      175560
      419
      175561
      175560

      19
      20
      21
      22
      23
      212520
      461
      212521
      212520

      20
      21
      22
      23
      24
      255024
      505
      255025
      255024

      21
      22
      23
      24
      25
      303600
      551
      303601
      303600

      22
      23
      24
      25
      26
      358800
      599
      358801
      358800

      23
      24
      25
      26
      27
      421200
      649
      421201
      421200

      24
      25
      26
      27
      28
      491400
      701
      491401
      491400

      25
      26
      27
      28
      29
      570024
      755
      570025
      570024

      26
      27
      28
      29
      30
      657720
      811
      657721
      657720

      27
      28
      29
      30
      31
      755160
      869
      755161
      755160

      28
      29
      30
      31
      32
      863040
      929
      863041
      863040

      29
      30
      31
      32
      33
      982080
      991
      982081
      982080

      30
      31
      32
      33
      34
      1113024
      1055
      1113025
      1113024

      31
      32
      33
      34
      35
      1256640
      1121
      1256641
      1256640

      32
      33
      34
      35
      36
      1413720
      1189
      1413721
      1413720

      33
      34
      35
      36
      37
      1585080
      1259
      1585081
      1585080

      34
      35
      36
      37
      38
      1771560
      1331
      1771561
      1771560

      35
      36
      37
      38
      39
      1974024
      1405
      1974025
      1974024

      36
      37
      38
      39
      40
      2193360
      1481
      2193361
      2193360

      37
      38
      39
      40
      41
      2430480
      1559
      2430481
      2430480

      38
      39
      40
      41
      42
      2686320
      1639
      2686321
      2686320

      39
      40
      41
      42
      43
      2961840
      1721
      2961841
      2961840

      40
      41
      42
      43
      44
      3258024
      1805
      3258025
      3258024

      41
      42
      43
      44
      45
      3575880
      1891
      3575881
      3575880

      42
      43
      44
      45
      46
      3916440
      1979
      3916441
      3916440

      43
      44
      45
      46
      47
      4280760
      2069
      4280761
      4280760

      44
      45
      46
      47
      48
      4669920
      2161
      4669921
      4669920

      45
      46
      47
      48
      49
      5085024
      2255
      5085025
      5085024

      46
      47
      48
      49
      50
      5527200
      2351
      5527201
      5527200

```python
fig, ax = plt.subplots(1,3, figsize=(18, 6))

ax[0].plot("n1", "prod", "bo-", data=df)
ax[0].set_xlabel("n", fontsize=20)
ax[0].set_ylabel(f"$y = \prod_{{i=0}}^{{i={i_max}}} (n+i)$", fontsize=20)

ax[1].plot(df["k"], df["prod"], "ko-")
ax[1].set_xlabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[1].set_title("$y = k^2 - 1$", fontsize=20)

ax[2].plot(df["n1"], df["k"], "ko-")
ax[2].set_ylabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[2].set_xlabel("$n$", fontsize=20)

fig.tight_layout()
```

Let us look at the right hand side of the equation first, i.e. $k^2 - 1$.

This can be rewritten as $\textbf{(k-1)*(k+1)}$

**Now, this is where a hint lies.**

What the right hand side means that it is a product of two integers ($k-1$ and $k+1$) which differ by 2.

We can see that this is the case:

$
\begin{equation}
(k+1) - (k-1) \\
= k + 1 - k - (-1) \\
= k - k + 1 - (-1) \\
= 0 + 1 + 1 \\
= 2 \\
\end{equation}
$

So, if we can somehow show that the left hand side of the original equation, i.e. $n*(n+1)*(n+2)*(n+3)$:

* can be represented as a product of two numbers which differ by 2, then we are done,
* as these numbers can then be mapped to $k-1$ and $k+1$ for some $k \in \mathbb{Z}$.

We can group the numbers $\textbf{n, n+1, n+2, n+3}$ into pairs, with the hope of getting $k-1$ and $k+1$.

We can utilize following facts to choose the two pairs:

* The difference of the products should be constant, and hence independent of $n$
* Knowing that product of two factors of type $(n+i)*(n+j) = n^2 + (i+j)*n + i*j$,
* We can observe that $i+j$ will be same for numbers which are equidistant from the middle of all numbers.

Now we can select our pair of numbers.

* The first pair is $n$ and $(n+3)$,
    * and their product is $\textbf{n * (n+3)}$
    * which can be expanded as $\color{red}{\textbf{n^2 + 3n}}$

* And, the second pair $(n+1)$ and $(n+2)$,
    * and their product is $\textbf{(n+1)*(n+2)}$
    * which can be expanded as $\color{red}{\textbf{n^2 + 3n}} + \textbf{2}$

Based on the above pairing we can immediately see that the difference of these pair products is as follows:

$
\begin{equation}
[(n+1)*(n+2)] - [n * (n+3)]\\
= [\color{red}{n^2 + 3n} + 2] - [\color{red}{n^2 + 3n}]\\
= n^2 + 3n + 2 - n^2 - 3n\\
= (n^2 -n^2) + (3n - 3n) + 2\\
= 0 + 0 + 2\\
= 2
\end{equation}
$

Hence, based on the above simplification, we can map:

* $(\color{red}{n^2 + 3n} + 2) \rightarrow (k+1)$, and
* $(\color{red}{n^2 + 3n}) \rightarrow (k-1)$.

Now, if we choose $\color{blue}{\textbf{k = (n^2 + 3n + 1)}}$, the following equations hold:

* $n^2 + 3n + 2 = \color{blue}{(n^2 + 3n + 1)} + 1 = \color{blue}{k} + 1$

* $n^2 + 3n = \color{blue}{(n^2 + 3n + 1)} - 1 = \color{blue}{k} - 1$

Hence, we have proved the following:

$
\begin{equation}
\forall n \in \mathbb{Z}, \\
\exists k \in \mathbb{Z} \\
n*(n+1)*(n+2)*(n+3) \\
= [(n+3)*n]*[(n+1)*(n+2)]\\
= [\color{red}{n^2 + 3n}]*[\color{red}{n^2 + 3n} + 2]\\
= [\color{blue}{(n^2 + 3n + 1)} - 1]*[\color{blue}{(n^2 + 3n + 1)} + 1]\\
= [\color{blue}{k} - 1]*[\color{blue}{k} + 1]\\
= (k^2 - 1)
\end{equation}
$

And this equation can be solved by choosing $\color{blue}{\textbf{k = (n^2 + 3n + 1)}}$.

Hence, proved.

```python
df["k = n^2 + 3n + 1"] = (df["n1"]**2 + 3*df["n1"] + 1)
df
```

      n1
      n2
      n3
      n4
      prod
      k
      k^2
      k^2 - 1
      k = n^2 + 3n + 1

      0
      1
      2
      3
      4
      24
      5
      25
      24
      5

      1
      2
      3
      4
      5
      120
      11
      121
      120
      11

      2
      3
      4
      5
      6
      360
      19
      361
      360
      19

      3
      4
      5
      6
      7
      840
      29
      841
      840
      29

      4
      5
      6
      7
      8
      1680
      41
      1681
      1680
      41

      5
      6
      7
      8
      9
      3024
      55
      3025
      3024
      55

      6
      7
      8
      9
      10
      5040
      71
      5041
      5040
      71

      7
      8
      9
      10
      11
      7920
      89
      7921
      7920
      89

      8
      9
      10
      11
      12
      11880
      109
      11881
      11880
      109

      9
      10
      11
      12
      13
      17160
      131
      17161
      17160
      131

      10
      11
      12
      13
      14
      24024
      155
      24025
      24024
      155

      11
      12
      13
      14
      15
      32760
      181
      32761
      32760
      181

      12
      13
      14
      15
      16
      43680
      209
      43681
      43680
      209

      13
      14
      15
      16
      17
      57120
      239
      57121
      57120
      239

      14
      15
      16
      17
      18
      73440
      271
      73441
      73440
      271

      15
      16
      17
      18
      19
      93024
      305
      93025
      93024
      305

      16
      17
      18
      19
      20
      116280
      341
      116281
      116280
      341

      17
      18
      19
      20
      21
      143640
      379
      143641
      143640
      379

      18
      19
      20
      21
      22
      175560
      419
      175561
      175560
      419

      19
      20
      21
      22
      23
      212520
      461
      212521
      212520
      461

      20
      21
      22
      23
      24
      255024
      505
      255025
      255024
      505

      21
      22
      23
      24
      25
      303600
      551
      303601
      303600
      551

      22
      23
      24
      25
      26
      358800
      599
      358801
      358800
      599

      23
      24
      25
      26
      27
      421200
      649
      421201
      421200
      649

      24
      25
      26
      27
      28
      491400
      701
      491401
      491400
      701

      25
      26
      27
      28
      29
      570024
      755
      570025
      570024
      755

      26
      27
      28
      29
      30
      657720
      811
      657721
      657720
      811

      27
      28
      29
      30
      31
      755160
      869
      755161
      755160
      869

      28
      29
      30
      31
      32
      863040
      929
      863041
      863040
      929

      29
      30
      31
      32
      33
      982080
      991
      982081
      982080
      991

      30
      31
      32
      33
      34
      1113024
      1055
      1113025
      1113024
      1055

      31
      32
      33
      34
      35
      1256640
      1121
      1256641
      1256640
      1121

      32
      33
      34
      35
      36
      1413720
      1189
      1413721
      1413720
      1189

      33
      34
      35
      36
      37
      1585080
      1259
      1585081
      1585080
      1259

      34
      35
      36
      37
      38
      1771560
      1331
      1771561
      1771560
      1331

      35
      36
      37
      38
      39
      1974024
      1405
      1974025
      1974024
      1405

      36
      37
      38
      39
      40
      2193360
      1481
      2193361
      2193360
      1481

      37
      38
      39
      40
      41
      2430480
      1559
      2430481
      2430480
      1559

      38
      39
      40
      41
      42
      2686320
      1639
      2686321
      2686320
      1639

      39
      40
      41
      42
      43
      2961840
      1721
      2961841
      2961840
      1721

      40
      41
      42
      43
      44
      3258024
      1805
      3258025
      3258024
      1805

      41
      42
      43
      44
      45
      3575880
      1891
      3575881
      3575880
      1891

      42
      43
      44
      45
      46
      3916440
      1979
      3916441
      3916440
      1979

      43
      44
      45
      46
      47
      4280760
      2069
      4280761
      4280760
      2069

      44
      45
      46
      47
      48
      4669920
      2161
      4669921
      4669920
      2161

      45
      46
      47
      48
      49
      5085024
      2255
      5085025
      5085024
      2255

      46
      47
      48
      49
      50
      5527200
      2351
      5527201
      5527200
      2351

```python
fig, ax = plt.subplots(1,3, figsize=(12, 6))

ax[0].plot("n1", "prod", "bo-", data=df)
ax[0].set_xlabel("n", fontsize=20)
ax[0].set_ylabel(f"$y = \prod_{{i=0}}^{{i={i_max}}} (n+i)$", fontsize=20)

ax[1].plot(df["k"], df["prod"], "ko-")
ax[1].set_xlabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[1].set_title("$y = k^2 - 1$", fontsize=20)

ax[2].plot(df["n1"], df["k"], "ko-", label="$k = \sqrt{y + 1}$")
ax[2].plot(df["n1"], df["k = n^2 + 3n + 1"], "r--", label="$k = n^2 + 3n + 1$")
ax[2].legend(fontsize=14)

ax[2].set_ylabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[2].set_xlabel("$n$", fontsize=20)

fig.tight_layout()
```

# More videos to come

Shubhanshu Mishra (shubhanshu.com)

```python
fig, ax = plt.subplots(1,3, figsize=(12, 6))
fig.patch.set_facecolor('white')

ax[0].plot("n1", "prod", "bo-", data=df)
ax[0].set_xlabel("n", fontsize=20)
ax[0].set_ylabel(f"$y = \prod_{{i=0}}^{{i={i_max}}} (n+i)$", fontsize=20)

ax[1].plot(df["k"], df["prod"], "ko-")
ax[1].set_xlabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[1].set_title("$y = k^2 - 1$", fontsize=20)

ax[2].plot(df["n1"], df["k"], "ko-", label="$k = \sqrt{y + 1}$")
ax[2].plot(df["n1"], df["k = n^2 + 3n + 1"], "r--", label="$k = n^2 + 3n + 1$")
ax[2].legend(fontsize=14)

ax[2].set_ylabel("$k = \sqrt{y + 1}$", fontsize=20)
ax[2].set_xlabel("$n$", fontsize=20)

fig.suptitle(f"Product of 4 consecutive integers is 1 less than a perfect square.", fontsize=20)

fig.tight_layout()
```

## Related works

* P. Erdös. J. L. Selfridge. "The product of consecutive integers is never a power." Illinois J. Math. 19 (2) 292 - 301, June 1975. https://doi.org/10.1215/ijm/1256050816

## Visual Proof

```python
nums = np.arange(10,10+4)
A = np.zeros((nums[0], nums[-1]))
A[:, nums[0]:] = 1
sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)
```

```python
nums = np.arange(10,10+4)
A = np.zeros((nums[1], nums[2]))
A[:, nums[0]:] = 2
A[nums[0]:, :] = 3
A[nums[0]:, nums[0]:] = 1

sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)
```

```python
import matplotlib.animation as animation
from IPython.display import HTML
```

```python
fig, ax = plt.subplots(1,1)
frames = []

nums = np.arange(10,10+4)
A = np.zeros((nums[1], nums[-1]))

im = ax.pcolormesh(A, cmap="inferno", vmin=0, vmax=4)
title = ax.set_title(f"Start")
ax.invert_yaxis()
ax.set_xticks(np.arange(A.shape[1]))
ax.set_yticks(np.arange(A.shape[0]))
ax.grid(which="major", color="w", linestyle='-', linewidth=3)

def init():
    im.set_array(A)
    title.set_text("")
    return im, title

def animate(i):
    text = ""
    if i == 0:
        A[:, nums[0]:] = 4
        A[nums[0]:, :] = 4
        text = "$n * n$"
    if i == 1:
        A[:, nums[0]:] = 2
        A[nums[0]:, ] = 4
        text = "$n * (n+3)$"
    if i == 2:
        A[:, nums[0]:] = 2
        A[:, nums[2]:] = 3
        A[nums[0]:, ] = 4
        text = "$n * (n+3)$"
    if i == 3:
        A[:, nums[2]:] = 4
        A[nums[0]:, :] = 3
        A[nums[0]:, nums[0]:] = 4
        A[nums[0]:, nums[0]:nums[2]] = 4
        text = "$(n+1) * (n+2)$"
    if i == 4:
        A[nums[0]:, nums[0]:nums[2]] = 1
        text = "$n * (n+3) = (n+1)*(n+2) - 2$"
#     print(A)
    im.set_array(A)
    title.set_text(f"Step: {i} | {text}")
    return im, title

# ax = sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)

fig.tight_layout()

ani = animation.FuncAnimation(fig,animate,frames=5,interval=2000,blit=True,repeat=True)

HTML(ani.to_html5_video())
# frames
```

```python
# ax.cla()
nums = np.arange(10,10+4)
A = np.zeros((nums[1], nums[-1]))
A[:, nums[0]:] = 2
A[nums[0]:, ] = 4
sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)
plt.show()
# plt.pause(1)

A[:, nums[2]:] = 4
sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)
plt.show()
# plt.pause(1)

A[nums[0]:, :] = 2
A[nums[0]:, nums[0]:] = 4
A[nums[0]:, nums[0]:nums[2]] = 1
sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)
plt.show()
# plt.pause(1)
```

```python
nums = np.arange(10,10+4)
A = np.zeros((nums[1], nums[-1]))
A[:, nums[0]:] = 2
A[nums[0]:, :] = 3
A[nums[0]:, nums[0]:] = 1
A[:, nums[2]:] = 4
sns.heatmap(A, linewidth=2, cbar=False, vmin=0, vmax=4)
```

```python

```

Slide Notebooks/Product of consecutive numbers.ipynb


# AMath 583 Lab 5

## Where to find this notebook:

* This notebook will be in \$UWHPSC/labs/lab5/lab5a.ipynb if you have cloned the class repository.  You may need to "git pull" to update.
* Or you can bring it down to your computer (or SageMathCloud project) by typing this in a terminal shell:
  *   wget http://faculty.washington.edu/rjl/classes/am583s2014/lab5a.ipynb

## Announcements:

* Please sit at assigned tables.
* If you're having problems with notebooks in SMC, note that you can also use IPython from a terminal.

* Homework 2 will be posted by Thursday.  Part of the homework will be to read this paper:
  "Best Practices for Scientific Computing, by G. Wilson, D. A. Aruliah", by C. T. Brown, et. al.
  that can be found at <http://www.plosbiology.org/article/info%3Adoi%2F10.1371%2Fjournal.pbio.1001745>

```
%pylab inline
```

## SymPy

SymPy is a package of Python tools for symbolic calculations in algebra and calculus.  This is notebook has a very brief demo of some features, but you can learn much more about it from various on-line resources such as:

* <http://docs.sympy.org/latest/tutorial/intro.html#what-is-symbolic-computation>
* <http://nbviewer.ipython.org/github/jrjohansson/scientific-python-lectures/blob/master/Lecture-5-Sympy.ipynb>

You can do "from sympy import *" to get all of sympy in your namespace, but note that this will overwrite standard numpy variables such as pi and functions such as sin, cos, exp, sqrt, with sympy versions.  Since we may want to mix symbolic and numerical computing, we will be explicit about what's coming from SymPy:

```
import sympy as S  # shorthand
```

```
S.init_printing()   # so symbolic math is printed nicely
```

### Define a polynomial and factor it:

```
x = S.symbols('x')  # defines a symbol
f = x**3 - 3*x**2 - 18*x + 40   # a new symbolic expression
f                   # print it nicely
```

```
S.factor(f)
```

### We can also differentiate:

```
S.diff(f,x)  # differentiate f with respect to x
```

### A messier function and its derivative:

```
f = (x**3 * S.exp(5*x)*S.cos(S.pi*x)) / (1 + S.sqrt(x))
f
```

```
g = S.diff(f,x,n=3)
g
```

Note that if you "print g" it does not come out so pretty, but this might be more useful if you want to cut and paste this into a Fortran program, for example:

```
print g
```

### Evaluate this derivative at the point $x = 0.2$:

```
g2 = g.subs(x, 0.2)
g2
```

Note this substituted 0.2 for $x$, but the special symbol $\pi$ has not be converted to a number.  The SymPy function N converts special constants like $\pi$ and $e$ to numerical values:

```
S.N(g2)
```

You can specify how many digits to use when it does this substitution, e.g.

```
S.N(S.pi,n=50)
```

## Symbolic differentiation for a Newton iteration.

In the last two labs you implemented Newton's method for different functions and probably had to compute the derivatives by hand.  Here's a way to write and fvals function using SymPy:

```
def fvals(x, debug=False):
    from sympy import symbols,diff,sqrt
    # First specify f symbolically and differentiate using SymPy:
    xs = symbols('xs')
    fs = xs**2 - 2.
    fprimes = diff(fs, xs)

    # Now evaluate numerically at the value x passed in to this function:
    f = fs.subs(xs, x)
    fprime = fprimes.subs(xs, x)

    # The next lines are just for illustrating that this is working:
    if debug:
        print "fs = ",fs
        print "fprimes = ",fprimes
        print "x = ",x

    return f, fprime

# Try it out:
fv = fvals(3., debug=True)
print "fvals returns: ", fv
```

## Try this out with Newton's method:

```
def newton(fvals, x0, tol):
    xk = x0
    kmax = 30   # maximum number of iterations
    print "   k               xk                      f(xk)"
    for k in range(kmax):

        fxk, fpxk = fvals(xk)  # evaluate f(x) and f'(x)
        print "%4i   %22.15f  %22.15f"  % (k, xk, fxk)

        if abs(fxk) < tol:
            break  #leave the loop if tolerance satisfied

        xk = xk - fxk / fpxk  # update xk using Newton's method

    return xk
```

```
newton(fvals, 2, 1e-10)
```

## Exercises:

* Use Newton's method to find a root of $f(x) = (x^2 - 2)\exp(-0.1 x^2) + 0.5$ from Lab 4.
* Let $f(x) = \sqrt{\cos(2\pi x) e^x}$.  Compute $f''(0.1)$, the second derivative evaluated at $x=0.1$.  This value is needed for the Lab 5 Quiz.

Remember you can get help in IPython by adding a ? to the end of an object and running the cell...

```
S.integrate?
```

```
S.diff?
```


# $k$ Nearest Neighbors

## Illustration of Nearest Neighbors

Arguably one of the simplest classification algorithm is $k$ nearest neighbors (KNN). Below figure portrays the algorithm's simplicity. We see a training set of five red and five blue dots, representing some label 1 and 0, respectively. The two axes represent two features, e.g. income and credit card balance. If we now add a new test data point $x_0$ (green dot), KNN will label this test data according to its $k$ closest neighbors. In below figure we set $k=3$. The three closest neighbors are: one blue dot and two red dots, resulting in estimated probabilities of 2/3 for the red class and 1/3 for the blue class. Hence KNN will predict that the new data point will belong to the red class. More so, based on the training set the algorithm is able to draw a decision boundary. (Note: In a classification settings with $j$ classes a *decision boundary* is a hyperplane that partitions the underlying vector space into $j$ sets, one for each class.). This is shown with the jagged line that separates background colors cyan (blue label) and light blue (red label). Given any possible pair of feature values, KNN labels the response along the drawn decision boundary. With $k=1$ the boundary line is very jagged. Increasing the number of $k$ will smoothen the decision boundary. This tells us that small values of $k$ will produce large variance but low bias, meaning that each new added training point might change the decision boundary line significantly but the decision boundary separates the training set (almost) correctly. As $k$ increases, variance decreases but bias increases. This is a manifestation of the Bias-Variance Trade-Off as discussed in the script ([Fortmann-Roe (2012)](http://scott.fortmann-roe.com/docs/BiasVariance.html)) and highlights the importance of selecting an adequate value of $k$ - a topic we will pick up in a future chapter.

## Mathematical Description of KNN

Having introduced KNN illustratively, let us now define this in mathematical terms. Let our data set be $\{(x_1, y_1), (x_2, y_2), \ldots (x_n, y_n)\}$ with $x_i \in \mathbb{R}^p$ and $y_i \in \{0, 1\}\; \forall i \in \{1, 2, \ldots, n\}$. Based on the $k$ neighbors, KNN estimates the conditional probability for class $j$ as the fraction of points in $\mathcal{N}(k, x_0)$ (the set of the $k$ closest neighbors of $x_0$) whose response values equals $j$ (Russell and Norvig (2009)):

$$\begin{equation}
\Pr(Y = j | X = x_0) = \frac{1}{k} \sum_{i \in \mathcal{N}(k, x_0)} \mathbb{I}(y_i = j).
\end{equation}$$

Once the probability for each class $j$ is calculated, the  KNN classifier predicts a class label $\hat{y}_0$ for the new data point $x_0$ by maximizing the conditional probability (Batista and Silva (2009)).

$$\begin{equation}
\hat{y}_0 = \arg \max_{j} \frac{1}{k} \sum_{i \in \mathcal{N}(k,x_0)} \mathbb{I}(y_i = j)
\end{equation}$$

Selecting a new data point $x_0$'s nearest neighbors requires some notion of **distance measure**. Most researchers chose Minkowski's distance, which is often referred to as $L^m$ norm (Guggenbuehler (2015)). The distance between points $x_a$ and $x_b$ in $\mathbb{R}^p$ is then defined as follows (Russell and Norvig (2009)):

$$\begin{equation}
L^m (x_a, x_b) = \left(\sum_{i=1}^p |x_{a, i} - x_{b, i}|^m \right)^{1/m}
\end{equation}$$

Using $m=2$, above equation simplifies to the well known Euclidean distance and $m=1$ yields the Manhattten distance. Python's `sklearn` package, short for scikit-learn, offers [several other options](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.DistanceMetric.html) that we will not discuss.

Beyond the pure distance measure, it is also possible to weight training data points relative to their distance from a certain point. In above figure distance is weighted uniformly. Alternatively one could weight points by the inverse of their distance (closer neighbors of a query point will have a greater influence than neighbors which are further away) or any other user-defined weighting function. For [further details check `sklearn`'s documentation](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html#sklearn.neighbors.KNeighborsClassifier) for details).

## Application: Predicting Share Price Movement

### Loading Data

The application of KNN is shown using simple stock market data. The idea is to predict a stock's movement based on simple features such as:
* `Lag1, Lag2`: log returns of the two previous trading days
* `SMI`: SMI log return of the previous day

The response is a binary variable: if a stock closed above the previous day's closing price it equals 1, and 0 if it fell. We start by loading the necessary packages and stock data from a csv - a procedure we are well acquainted with by now. Thus comments are held short.

```python
%matplotlib inline
import numpy as np
import pandas as pd
import matplotlib.pyplot as plt
plt.style.use('seaborn-whitegrid')
plt.rcParams['font.size'] = 14
```

```python
# Import daily shares prices and select window
shsPr = pd.read_csv('Data/SMIDataDaily.csv', sep=',',
                     parse_dates=['Date'], dayfirst=True,
                     index_col=['Date'])
shsPr = shsPr['2017-06-30':'2012-06-01']
shsPr.head()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      ...
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK
      SMI

      Date

      2017-06-30
      23.68
      72.90
      50.45
      79.00
      13.86
      447.2
      1918.0
      54.90
      207.3
      83.45
      ...
      244.2
      462.7
      2322.0
      323.6
      87.65
      16.24
      354.1
      279.1
      6160.0
      8906.89

      2017-06-29
      23.63
      72.85
      51.30
      78.60
      14.10
      446.0
      1913.0
      55.00
      204.7
      83.65
      ...
      246.9
      463.9
      2310.0
      323.3
      87.80
      16.36
      353.4
      276.3
      6145.0
      8944.04

      2017-06-28
      24.07
      73.55
      51.65
      80.00
      13.82
      452.5
      1927.0
      56.35
      207.5
      85.40
      ...
      251.2
      464.9
      2342.0
      325.3
      88.55
      16.29
      360.0
      278.0
      6195.0
      9076.73

      2017-06-27
      24.40
      73.70
      51.65
      80.20
      13.74
      453.8
      1931.0
      56.40
      207.6
      84.30
      ...
      252.0
      465.0
      2355.0
      322.7
      88.80
      16.00
      362.0
      281.7
      6280.0
      9072.92

      2017-06-26
      24.63
      73.85
      51.35
      80.45
      13.49
      457.0
      1947.0
      56.25
      208.2
      85.65
      ...
      251.7
      469.3
      2375.0
      323.9
      89.40
      15.71
      367.5
      284.4
      6310.0
      9121.22

5 rows × 21 columns

Having the data in a proper dataframe, we are now in a position to create the features and response values.

```python
# Calculate log-returns and label responses:
# 'direction' equals 1 if stock closed above
# previous day and 0 if it fell.
today = np.log(shsPr / shsPr.shift(-1))
direction = np.where(today >= 0, 1, 0)

# Convert 'direction' to dataframe
direction = pd.DataFrame(direction, index=today.index, columns=today.columns)

# Lag1, 2: t-1 and t-2 returns; excl. smi (in last column)
Lag1 = np.log(shsPr.iloc[:, :-1].shift(-1) / shsPr.iloc[:, :-1].shift(-2))
Lag2 = np.log(shsPr.iloc[:, :-1].shift(-2) / shsPr.iloc[:, :-1].shift(-3))

# Previous day return for SMI index
smi  = np.log(shsPr.iloc[:, -1].shift(-1) / shsPr.iloc[:, -1].shift(-2))
```

### KNN Algorithm Applied

Now comes the difficult part. What we want to achieve is to run the KNN algorithm for every stock and for different hyperparameter $k$ and see how it performs. For this we do the following steps:

1. Create a feature matrix `X` containing `Lag1`, `Lag2` and `SMI` data for share i
2. Create a response vector `y` with binary direction values
3. Split data to training (before 2016-06-30) and test set (after 2016-06-30)
4. Run KNN for different values of $k$ (loop)
5. Write test score for given $k$ to matrix `scr`
6. Once we have run through all $k$'s we proceed with step 1. with share i+1

This means we need two loops. The first corresponds to the share (e.g. ABB, Adecco, etc.), the second runs the KNN algorithm for different values of $k$.

The reason for this approach is that we are interested in finding any pattern/structure that would provide a successful trading strategy. There is obviously no free lunch. Predicting share price direction is by no means an easy task and we must be well aware that we are in for a difficult job here. If it were simple, neither one of us would be sitting here but run his own fund. But nonetheless, let us see how KNN performs and how homogeneous (or heterogeneous) the results are.

As usual our first step is to prepare the ground by loading the necessary package and defining some auxiliary variables. The KNN function we will be using is available through the `sklearn` (short for Scikit-learn) package. We only load the `neighbor` sublibrary which contains the needed KNN function called `KNeigborsClassifier()`. KNN is applied with the default distance metric: Euclidean distance (Minkowski's distance with $m=2$). If we would prefer another distance metric we would have to specify it ([see documentation](http://scikit-learn.org/stable/modules/generated/sklearn.neighbors.KNeighborsClassifier.html)).

```python
# Import relevant functions
from sklearn import neighbors

# k = {1, 3, ..., 200}
k = np.arange(1, 200, 2)

# Array to store results in. Dimension is [k x m]
# with m=20 for the 20 companies (excl. SMI)
scr = np.empty(shape=(len(k), len(shsPr.columns)-1))
```

```python
for i in range(len(shsPr.columns)-1):

    # 1) Create matrix with feature values of stock i
    X = pd.concat([Lag1.iloc[:, i], Lag2.iloc[:, i], smi], axis=1)
    X = X[:-3]  # Drop last three rows with NaN (due to lag)

    # 2) Remove last three rows of response dataframe
    #    to have equal no. of rows for features and response
    y = direction.iloc[:, i]
    y = y[:-3]

    # 3) Split data into training set...
    X_train = X['2016-06-30':]
    y_train = y['2016-06-30':]
    # ...and test set.
    X_test = X[:'2016-07-01']
    y_test = y[:'2016-07-01']

    # Convert responses to 1xN array (with .ravel() function)
    y_train = y_train.values.ravel()
    y_test  = y_test.values.ravel()

    for j in range(len(k)):

        # 4) Run KNN
        # Instantiate KNN class
        knn = neighbors.KNeighborsClassifier(n_neighbors=k[j])
        # Fit KNN classifier using training set
        knn = knn.fit(X_train, y_train)

        # 5) Extract test score for k[j]
        scr[j, i] = knn.score(X_test, y_test)
```

```python
# Convert data to pandas dataframe
tickers = shsPr.columns
scr = pd.DataFrame(scr, index=k, columns=tickers[:-1])
```

```python
scr.head()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      1
      0.462451
      0.533597
      0.521739
      0.513834
      0.509881
      0.533597
      0.498024
      0.498024
      0.505929
      0.411067
      0.549407
      0.553360
      0.513834
      0.521739
      0.509881
      0.470356
      0.490119
      0.549407
      0.498024
      0.529644

      3
      0.478261
      0.470356
      0.529644
      0.486166
      0.577075
      0.533597
      0.525692
      0.482213
      0.525692
      0.486166
      0.517787
      0.573123
      0.486166
      0.517787
      0.541502
      0.450593
      0.490119
      0.557312
      0.517787
      0.513834

      5
      0.498024
      0.494071
      0.537549
      0.494071
      0.525692
      0.573123
      0.537549
      0.474308
      0.501976
      0.486166
      0.525692
      0.513834
      0.466403
      0.501976
      0.521739
      0.482213
      0.494071
      0.553360
      0.533597
      0.509881

      7
      0.517787
      0.486166
      0.505929
      0.498024
      0.505929
      0.553360
      0.525692
      0.517787
      0.474308
      0.509881
      0.513834
      0.482213
      0.470356
      0.509881
      0.525692
      0.466403
      0.490119
      0.561265
      0.533597
      0.509881

      9
      0.553360
      0.478261
      0.505929
      0.470356
      0.486166
      0.533597
      0.505929
      0.513834
      0.458498
      0.462451
      0.545455
      0.486166
      0.462451
      0.501976
      0.501976
      0.470356
      0.490119
      0.517787
      0.482213
      0.470356

### Results & Analysis

Now let's see the results in an overview.

```python
scr.describe()
```

      ABBN
      ADEN
      BAER
      CFR
      CSGN
      GEBN
      GIVN
      LHN
      LONN
      NESN
      NOVN
      ROG
      SCMN
      SGSN
      SLHN
      SREN
      UBSG
      UHR
      ZURN
      SIK

      count
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000
      100.000000

      mean
      0.553202
      0.506798
      0.515968
      0.508577
      0.517431
      0.499763
      0.533320
      0.492016
      0.537510
      0.519960
      0.546877
      0.513043
      0.479051
      0.516996
      0.490514
      0.520791
      0.489289
      0.507352
      0.553557
      0.530040

      std
      0.031183
      0.010579
      0.023187
      0.024694
      0.016666
      0.020266
      0.013498
      0.019546
      0.027272
      0.020842
      0.018939
      0.016882
      0.019902
      0.020634
      0.020150
      0.017837
      0.014962
      0.027736
      0.020282
      0.028188

      min
      0.462451
      0.470356
      0.462451
      0.454545
      0.446640
      0.458498
      0.498024
      0.450593
      0.458498
      0.411067
      0.505929
      0.474308
      0.446640
      0.474308
      0.450593
      0.450593
      0.438735
      0.454545
      0.482213
      0.466403

      25%
      0.529644
      0.500988
      0.501976
      0.493083
      0.509881
      0.486166
      0.525692
      0.477273
      0.529644
      0.509881
      0.533597
      0.501976
      0.462451
      0.501976
      0.474308
      0.516798
      0.482213
      0.486166
      0.545455
      0.512846

      50%
      0.557312
      0.505929
      0.517787
      0.509881
      0.517787
      0.498024
      0.533597
      0.490119
      0.547431
      0.525692
      0.545455
      0.513834
      0.476285
      0.521739
      0.494071
      0.525692
      0.490119
      0.501976
      0.557312
      0.525692

      75%
      0.581028
      0.513834
      0.533597
      0.525692
      0.525692
      0.513834
      0.537549
      0.505929
      0.553360
      0.533597
      0.561265
      0.525692
      0.490119
      0.533597
      0.501976
      0.533597
      0.498024
      0.525692
      0.569170
      0.546443

      max
      0.600791
      0.533597
      0.569170
      0.557312
      0.577075
      0.573123
      0.573123
      0.541502
      0.577075
      0.549407
      0.592885
      0.573123
      0.541502
      0.561265
      0.541502
      0.545455
      0.521739
      0.573123
      0.592885
      0.581028

Following finance theory, returns should be distributed symmetrically. Thus the simplest guess would be to expect a share price to increase on 50% of the days and to decrease on the remaining 50%. Similar to guessing a coin flip, if we would guess an 'up' movement for every day, we obviously would - in the long run - be correct 50% of the times. This would make for a score of 50%.

Looking in that light at the above summary, we see some very interesting results. For 13 out of 20 stocks KNN produces test scores of > 50% for even the 0.25th percentile. Let's plot the ones with the highest test-scores (ABBN, NOVN, SIK and ZURN) to see at what value of $k$ the best test-score is achieved.

```python
nms = ['ABBN', 'NOVN', 'SIK', 'ZURN']

plt.figure(figsize=(12, 8))
for col in nms:
    scr[col].plot(legend=True)
plt.axhline(0.50, c='k', ls='--');
```

For ABB and Novartis the max. score is around $k=100$ while for Sika and Zurich it is between 177 - 185. Furthermore, it seems interesting that for ABB, Novartis and Zurich the test score was barely below 50%. If this is indeed a pattern we would have found a trading strategy, wouldn't we?

To further assess our results we look into KNN's prediction of ABB stock movements. For this we rerun our KNN classifier algorithm for ABB as before.

```python
# 1) Create matrix with feature values of stock i
X = pd.concat([Lag1['ABBN'], Lag2['ABBN'], smi], axis=1)
X = X[:-3]  # Drop last three rows with NaN (due to lag)
```

```python
# 2) Remove last three rows of response dataframe
#    to have equal no. of rows for features and response
y = direction['ABBN']
y = y[:-3]
```

```python
# 3) Split data into training set...
X_train = X['2016-06-30':]
y_train = y['2016-06-30':]
# ...and test set.
X_test = X[:'2016-07-01']
y_test = y[:'2016-07-01']

# Convert responses to 1xN array (with .ravel() function)
y_train = y_train.values.ravel()
y_test  = y_test.values.ravel()
```

For ABB the maximum score is reached where $k=119$. You can check this with the `scr['ABBN'].idxmax()` command, which provides the index of the maximum value of the selected column. In our case, the index is equivalent to the value of $k$. Thus we run KNN with $k=119$.

```python
# 4) Run KNN
# Instantiate KNN class for ABB with k=119
knn = neighbors.KNeighborsClassifier(n_neighbors=119)
# Fit KNN classifier using training set
knn = knn.fit(X_train, y_train)

# 5) Extract test score for ABB
scr_ABB = knn.score(X_test, y_test)
scr_ABB
```

    0.6007905138339921

The score of 60.08% is the very same as what we have seen in above summary statistic. Nothing new so far. (Recall that the score is the total of correctly predicted outcomes.)

However, the alert reader should by now raise some questions regarding our assumption that 50% of the returns should have been positive. In the long run, this might be true. But our training sample contained only 1'018 records and of these 535 were positive.

```python
# Percentage of 'up' days in training set
y_train.sum() / y_train.size
```

    0.5255402750491159

Therefore, if we would guess 'up' for every day of our test set and **given the distribution of classes in the test set is exactly as in our training set**, then we would predict the correct movement in 52.55% of the cases. So in that light, the predictive power of our KNN algorithm has to be put in perspective to the 52.55%.

In summary, our KNN algorithm has a score of 60.08%. Our best guess (based on the training set) would yield a score of 52.55%. This still displays that overall our KNN algorithm outperforms our best guess. Nonetheless, the margin is smaller than initially thought.

### Confusion Matrix

There are more tools to assess the accuracy of an algorithm. We postpone the discussion of these tools to a later chapter and at this stage restrict ourselves to the discussion of a tool called "confusion matrix".

A confusion matrix is a convenient way of displaying how our classifier performs. In binary classification (with e.g. response $y \in \{0, 1\}$) there are four prediction categories possible (Ting (2011)):

* **True positive**: True response value is 1, predicted value is 1 ("hit")
* **True negative**: True response value is 0, predicted value is 0 ("correct rejection")
* **False positive**: True response value is 0, predicted value is 1 ("False alarm", Type 1 error)
* **False negative**: True response value is 1, predicted value is 0 ("Miss", Type 2 error)

These information help us to understand how our (KNN) algorithm performed. There are different two ways of arranging confusion matrix. James et al. (2013) follow the convention that column labels indicate the true class label and rows the predicted response class. Others have it transposed such that column labels indicate predicted classes and row labels show true values. We will use the latter approach as it is more common.

To run this in Python, we first predict the response value for each data entry in our test matrix `X_test`. Then we arrange the data in a suitable manner.

```python
# Predict 'up' (=1) or 'down' (=0) for test set
pred = knn.predict(X_test)
```

```python
# Store data in DataFrame
cfm = pd.DataFrame({'True direction': y_test,
                    'Predicted direction': pred})
cfm.replace(to_replace={0:'Down', 1:'Up'}, inplace=True)

# Arrange data to confusion matrix
print(cfm.groupby(['Predicted direction','True direction']) \
      .size().unstack('Predicted direction'))
```

    Predicted direction  Down   Up
    True direction
    Down                   31   83
    Up                     18  121

As mentioned before, rows represent the true outcome and columns show what class KNN predicted. In 31 cases, the test set's true response was 'down' (in our case represented by 0) and KNN correctly predicted 'down'. 121 times KNN was correct in predicting an 'up' (=1) movement. 18 returns in the test set were positive but KNN predicted a negative return. And in 83 out of 253 cases KNN predicted an 'up' movement whereas in reality the stock price decreased. The KNN score of 60.08% for ABB is the sum of true positive and negative (31 + 121) in relation to the total number of predictions (253 = 31 + 18 + 83 + 121). The error rate is 1 - score or (18 + 83)/253.

Class-specific performance is also helpful to better understand results. The related terms are **sensitivity** and **specifity**. In the above case, sensitivity is the percentage of true 'up' movements that are identified. A good 88.1% (= 121 / (18 + 121)). The specifity is the percentage of 'down' movements that are correctly identified, here a poor 27.2% (= 31 / (31 + 83)). More on this in the next chapter.

Because confusion matrices are important to analyze results, `Scikit-learn` has its own [command to generate it](http://scikit-learn.org/stable/modules/generated/sklearn.metrics.confusion_matrix.html). It is part of the `metrics` sublibrary. The difficulty is that in contrast to above (manually generated) table, the function's output provides no labels. Therefore one must be sure to know which value are where. Here's the code to generate the confusion matrix.

```python
from sklearn.metrics import confusion_matrix

# Confusion matrix
confusion_matrix(y_test, pred)
```

    array([[ 31,  83],
           [ 18, 121]], dtype=int64)

In general it is tremendously helpful to visualize results. At the time of writing in February 2020, Anaconda shipped with `Sklearn` version 0.21.3. This `Sklearn` version unfortunately doesn't have a specific plotting function for the confusion matrix. However, on the package's website a [function code is provided](http://scikit-learn.org/stable/auto_examples/model_selection/plot_confusion_matrix.html) that does exactly this. Below their code is applied to our KNN results on ABB's stock price movement. (**See Post Script at the end for further details**)

```python
import itertools
plt.style.use('default')

def plot_confusion_matrix(cm, classes,
                          normalize=False,
                          title='Confusion matrix',
                          cmap=plt.cm.Blues):
    """
    This function prints and plots the confusion matrix.
    Normalization can be applied by setting `normalize=True`.
    """
    if normalize:
        cm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]
        print("Normalized confusion matrix")
    else:
        print('Confusion matrix, without normalization')

    print(cm)

    plt.imshow(cm, interpolation='nearest', cmap=cmap)
    plt.title(title)
    plt.colorbar()
    tick_marks = np.arange(len(classes))
    plt.xticks(tick_marks, classes, rotation=45)
    plt.yticks(tick_marks, classes)

    fmt = '.2f' if normalize else 'd'
    thresh = cm.max() / 2.
    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):
        plt.text(j, i, format(cm[i, j], fmt),
                 horizontalalignment="center",
                 color="white" if cm[i, j] > thresh else "black")

    plt.tight_layout()
    plt.ylabel('True label')
    plt.xlabel('Predicted label')
```

```python
#plt.style.use('default')
# Compute confusion matrix
cfm_matrix = confusion_matrix(y_test, pred)
np.set_printoptions(precision=2)

# Plot non-normalized confusion matrix
plt.figure()
plot_confusion_matrix(cfm_matrix, classes=['Down', 'Up'],
                      title='Confusion matrix, without normalization');
```

```python
# Plot normalized confusion matrix
plt.figure()
plot_confusion_matrix(cfm_matrix, classes=['Down', 'Up'], normalize=True,
                      title='Normalized confusion matrix');
```

# Further Ressources

In writing this notebook, many ressources were consulted. For internet ressources the links are provided within the textflow above and will therefore not be listed again. Beyond these links, the following ressources were consulted and are recommended as further reading on the discussed topics:

* Batista, Gustavo, and Diego Furtado Silva, 2009, How k-nearest neighbor parameters affect its performance, in *Argentine Symposium on Artificial Intelligence*, 1–12, sn.
* Fortmann-Roe, Scott, 2012, Understanding the Bias-Variance Tradeoff from website, http://scott.fortmann-roe.com/docs/BiasVariance.html, 08/15/17.
* Guggenbuehler, Jan P., 2015, Predicting net new money using machine learning algorithms and newspaper articles, Technical report, University of Zurich, Zurich.
* James, Gareth, Daniela Witten, Trevor Hastie, and Robert Tibshirani, 2013, *An Introduction to Statistical Learning: With Applications in R* (Springer Science & Business Media, New York, NY).
* Müller, Andreas C., and Sarah Guido, 2017, *Introduction to Machine Learning with Python* (O’Reilly Media, Sebastopol, CA).
* Russell, Stuart, and Peter Norvig, 2009, *Artificial Intelligence: A Modern Approach* (Prentice Hall Press, Upper Saddle River, NJ).
* Ting, Kai Ming, 2011, Confusion matrix, in Claude Sammut, and Geoffrey I. Webb, eds., *Encyclopedia of Machine Learning* (Springer Science & Business Media, New York, NY).

# Post Script

Note that as of Scikit-Learn version 0.22.1 the confusion matrix plot was integrated into the sklearn distribution. Note however, that older Anaconda distributions might contain sklearn versions <= 0.22.1. Therefore in order to make use of it please check your sklearn version and update it if necessary (with `conda update scikit-learn` or `pip install --upgrade scikit-learn` in your shell). [Details see here](https://scikit-learn.org/stable/install.html#install-by-distribution).

```python
import sklearn
sklearn.show_versions()
```

    System:
        python: 3.6.10 |Anaconda, Inc.| (default, Jan  7 2020, 15:18:16) [MSC v.1916 64 bit (AMD64)]
    executable: C:\Anaconda\python.exe
       machine: Windows-10-10.0.19041-SP0

    Python dependencies:
           pip: 20.0.2
    setuptools: 45.1.0.post20200127
       sklearn: 0.22.1
         numpy: 1.18.1
         scipy: 1.4.1
        Cython: 0.29.14
        pandas: 1.0.0
    matplotlib: 3.1.2
        joblib: 0.14.1

    Built with OpenMP: True

For details on what parameter can be adjusted see here. [Details see here](https://scikit-learn.org/stable/modules/generated/sklearn.metrics.plot_confusion_matrix.html#sklearn.metrics.plot_confusion_matrix)

```python
from sklearn.metrics import plot_confusion_matrix

# To use same colors use cmap = plt.cm.Blues
# To plot the normalized figures, add normalize = 'true', 'pred', or 'all'
plot_confusion_matrix(estimator = knn,
                      X = X_test, y_true = y_test,
                      display_labels = ['Down', 'Up'],
                      values_format = '.0f');
```


# Spatial discretisation

So far, we've seen time derivatives and ordinary differential equations of the form

$$
\dot{u} = f(t, u).
$$

Most problems one encounters in the real world have spatial as well as time derivatives. Our first example is the [*Poisson equation*](https://en.wikipedia.org/wiki/Poisson%27s_equation):

$$
\begin{align}
-\frac{\text{d}^2 u}{\text{d} x^2} &= f(x) \quad x \in \Omega = (-1, 1)\\
 u(-1) &= a\\
 \frac{\text{d} u}{\text{d} x}(1) &= b\\
 \end{align}.
$$

This is termed a *boundary value problem* (BVP), as opposed to the ODEs which were *initial value problems*, since we do not specify an initial condition, but rather a condition on the boundary of the domain.

This equation appears in a remarkably large number of places. As one example, it models the equilibrium temperature profile of a thermally conducting material maintained at constant temperature $a$ on the left and cooled at constant rate $b$ on the right.

To solve this problem numerically we must make a number of choices:

- how to represent the solution $u$;
- how to compute its derivatives;
- how to enforce the boundary conditions;
- how and where to evaluate $f$;
- in what sense we would like our solution to satisfy the equation.

[Iserles' book](http://www.damtp.cam.ac.uk/user/ai/Arieh_Iserles/Textbook.html) contains a quite mathematical treatment of finite differences. I also like [Randy LeVeque's *Finite difference methods for ordinary and partial differential equations*](http://staff.washington.edu/rjl/fdmbook/).

## Finite difference framework

We will focus on _finite difference_ methods here, which make the following choices in answer to the questions above:

- The solution $u(x)$ is represented by _pointwise_ values $u_i = u(x_i)$ at some discrete set of points $-1 = x_0 < x_1 < \dots < x_N = 1$. Importantly, the framework _does not_ specify the value of $u$ outside of these points;
- derivatives of $u$ at points $x_i$ are approximated using differencing formulae that utilise a finite number of neighbouring points (independent of $N$);
- boundary conditions are either enforced pointwise (e.g. the $u(-1)$ case above), or (when constraining derivatives) with one-sided differencing formulae;
- $f$ is evaluated pointwise at each $x_i$;
- we require that our finite difference method satisfies the equation pointwise at each $x_i$ in the interior of the domain.

### Differencing formulae

Our starting point is the definition of a derivative:

$$
\frac{d u(x)}{d x} = \lim_{\epsilon \to 0} \frac{u(x + \epsilon) - u(x)}{\epsilon}.
$$

If we wish to approximate this in our finite difference framework, where we only have point values, we can do so using neighbouring values. Writing $x_{i+1} - x_i = h$ for convenience, we can write

$$
\frac{d u(x_i)}{d x} \approx \frac{u(x_{i+1}) - u(x_i)}{h} = \frac{u_{i+1} - u_i}{h} =: D_+ u_i.
$$

This is a _one-sided_ approximation: we only use $u_i$ and $u_{i+1}$. Another one-sided approximation would be to offset in the other direction

$$
D_{-} u_i := \frac{u_i - u_{i-1}}{h}.
$$

Finally, we can also use a _centred_ approximation by averaging the two one-sided approximations:

$$
D_0 u_i := \frac{u_{i+1} - u_{i-1}}{2h} = \frac{1}{2} (D_+ + D_-) u_i.
$$

Let's have a picture

```python
%matplotlib notebook

import numpy
from matplotlib import pyplot
import matplotlib.lines as mlines
pyplot.style.use('ggplot')

n = 200
h = 2/(n-1)
x = numpy.linspace(1,2.5,n)
pyplot.plot(x, numpy.sin(x));

def newline(p1, p2, **kwargs):
    ax = pyplot.gca()
    xmin, xmax = ax.get_xbound()

    if(p2[0] == p1[0]):
        xmin = xmax = p1[0]
        ymin, ymax = ax.get_ybound()
    else:
        ymax = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmax-p1[0])
        ymin = p1[1]+(p2[1]-p1[1])/(p2[0]-p1[0])*(xmin-p1[0])

    l = mlines.Line2D([xmin,xmax], [ymin,ymax], **kwargs)
    ax.add_line(l)
    return l

h = 0.25
xi = 1.6
ximinus = xi - h
xiplus = xi + h

pyplot.plot([ximinus, xi, xiplus], numpy.sin([ximinus, xi, xiplus]), marker="o", linestyle="none")

newline((xi, numpy.sin(xi)), (xiplus, numpy.sin(xiplus)), linestyle="dashed", label="$D_+ u(x)$")
newline((xi, numpy.sin(xi)), (ximinus, numpy.sin(ximinus)), linestyle="dotted", label="$D_- u(x)$")
newline((ximinus, numpy.sin(ximinus)), (xiplus, numpy.sin(xiplus)), linestyle="-.", label="$D_0 u(x)$")

newline((xi, numpy.sin(xi)), (xiplus, numpy.sin(xi) + h*numpy.cos(xi)), color="black", label="$u'(x)$")

pyplot.legend();
```

Let's look at the full approximate derivative too.

```python
n = 10
h = 2/(n-1)
x = numpy.linspace(-1,1,n)
u = numpy.sin(x)
pyplot.figure()
pyplot.plot(x, numpy.cos(x), label="$u'$");
pyplot.plot(x[1:], (u[1:] - u[:-1])/h, label="$D_-$", marker="o", linestyle="none")
pyplot.legend();
```

## Accuracy

Certainly from the picture of the slope above, it appears that the centered difference formula is more accurate than the one-sided approximations. Can we formalise this at all?

To do so, we turn to the favourite tool of the budding numericist, the *taylor expansion*.

### Recap, Taylor expansions

For a sufficiently smooth function $u$, given a point $x$, we can represent the function at a new point $x + h$ by its Taylor expansion

$$
u(x + h) = u(x) + u'(x) h + \frac{1}{2} u''(x) h^2 + \frac{1}{6} u'''(x) h^3 + \dots = \sum_{n=0}^\infty \frac{1}{n!} h^n u^{(n)}(x)
$$

where the notation $u'$ is shorthand for $\frac{\text{d} u}{\text{d} x}$ and $u^{(n)}(x) = \frac{\text{d}^n u}{\text{d} x^n}$.

Let's look at how this works for a sample function.

```python
from functools import reduce
from operator import mul

def u(x, n=0):
    factor = (-1)**(n // 2)
    if n % 2 == 0:
        u_ = numpy.sin
    else:
        u_ = numpy.cos
    return factor * u_(x)

pyplot.figure()

x = numpy.linspace(0.5, 1.75, 500)

pyplot.plot(x, u(x));

x0 = 0.6
h = 0.8

def fac(n):
    return reduce(mul, range(1, n+1), 1)

def taylor(u, x0, h, n):
    return u(x0) + sum(h**i/fac(i) * u(x0, i) for i in range(1, n))

xs = numpy.linspace(x0, x0+h, 20)
for n in range(1, 5):
    pyplot.plot(xs, [taylor(u, x0, x - x0, n) for x in xs], marker="o", label=r"$\tilde{u} + \mathcal{O}(h^%d)$" % n)

pyplot.legend();
```

If we chop off the series at some finite $n$ we write

$$
u(x + h) = u(x) + u'(x) h + \frac{1}{2} u''(x) h^2 + \mathcal{O}(h^3)
$$

with $h$ sufficiently small.

To determine the order of a method, we substitute the Taylor expansion into the differencing expression and calculate.

As an example, let us consider the one-sided differencing operator $D_+$. To simplify notation, we will choose $x = 0$, and we have

$$
\begin{align}
u'(0) &\approx \frac{u(h) - u(0)}{h} \quad \text{ definition of } D_+\\
      &= h^{-1}(\underbrace{u(0) + u'(0) h + \frac{1}{2} u''(0) h^2 + \mathcal{O}(h^3)}_{u(h)} - u(0)) \\
      &= u'(0) + \frac{1}{2} u''(0) h + \mathcal{O}(h^2)\\
\end{align}.
$$

The leading-order error term in the right hand side is $\mathcal{O}(h)$, and so we say that this is a _first-order_ method. Derivation that the operator $D_-$ is also first-order proceeds identically.

#### Questions

1. Show that the centered difference operator $D_0$ computes a second-order accurate derivative.

```python

```

## Stability

We will postpone mathematical discussion of stability for a while, and give an intuition for some potential problems. Let us first check that our implementation of differencing operators provides us with the expected (mathematical) convergence orders for a smooth function.

```python
def dplus(x, u):
    return x[:-1], (u[1:] - u[:-1])/(x[1:] - x[:-1])

def dminus(x, u):
    return x[1:], (u[1:] - u[:-1])/(x[1:] - x[:-1])

def center(x, u):
    return x[1:-1], (u[2:] - u[:-2])/(x[2:] - x[:-2])
```

```python
grids = 2**numpy.arange(3, 10)

def error(f, df, op):
    for n in grids:
        x = numpy.linspace(-1, 1, n)
        x, y = op(x, f(x))
        yield numpy.linalg.norm(y - df(x), numpy.inf)

pyplot.figure()
for op in [dplus, dminus, center]:
    pyplot.loglog(1/grids, list(error(numpy.sin, numpy.cos, op)), marker="o", linestyle="none", label=op.__name__)

pyplot.xlabel("Resolution ($h$)")
pyplot.ylabel("$l_\infty$ error in derivative")

pyplot.loglog(1/grids, 1/grids, label="$h$")
pyplot.loglog(1/grids, 1/grids**2, label="$h^2$")
pyplot.legend();
```

So both the one-sided differences are first-order accurate, whereas the centered difference is second-order accurate, as expected. One thing to be wary of, however, is using some of these approximations for functions that are "rough" on the grid scale.

We can make this question more precise by asking whether there are functions whose derivatives are non-zero, but for which our numerical approximations compute $u'(x_i) = 0$.

Let's try and contrive an example:

```python
x = numpy.linspace(-1, 1, 9)
xf = numpy.linspace(-1, 1, 100)

def f(x):
    return numpy.cos(1/2 + 4*numpy.pi*x)

def df(x):
    return -4*numpy.pi*numpy.sin(1/2 + 4*numpy.pi*x)
```

```python
pyplot.figure()
pyplot.plot(x, f(x), marker="o", label="coarse")
pyplot.plot(xf, f(xf), "-", label="fine")
pyplot.legend();
```

What about the derivatives?

```python
pyplot.figure()
for op in [dplus, dminus, center]:
    x_, y = op(x, f(x))
    pyplot.plot(x_, y, "o-", label=op.__name__)
pyplot.plot(xf, df(xf), "-", label="Exact")
pyplot.legend();
```

The centered difference approximation produces a _zero_ derivative for this function. Hence if we have a solution $u(x)$, we can (at least to the numerical operator) construct a new solution $\tilde{u}(x) = u(x) + f(x)$.

Suddenly, even if our actual equation has a unique solution, the numerical solution is no longer unique. This turns out to cause all kinds of terrible issues with numerical algorithms and must be avoided at all costs.

## Higher-order derivatives

We can compute high-order derivatives by repeatedly applying differencing operators for lower-order derivatives.

For example, the second derivative

$$
\frac{\text{d}^2 u}{\text{d} x^2} \approx D^2 u_i = D_+ D_- u_i = \frac{1}{h^2}\left(u_{i+1} - 2 u_i + u_{i-1}\right) = D_- D_+ u_i.
$$

### Questions

1. Show that this is a _second-order_ accurate approximation of the second derivative
2. We could also use $D^2 u_i = D_0 D_0 u_i$, derive the stencil for this case.
3. Finally, show that if we define a "half-step" centered difference operator

$$
\hat{D}_0 u = \frac{1}{h}\left[u\left(x + \frac{h}{2}\right) - u\left(x - \frac{h}{2}\right)\right]
$$
then we have

$$
D^2 = \hat{D}_0 \circ \hat{D}_0 = D_+ \circ D_-
$$

```python

```

## Boundary conditions

The final missing piece required before we can solve the our first PDE is to figure out how we will treat boundary conditions. To do this, we will first recast the differencing operators as matrices. it is then somewhat easier to see what is going on.

We can think of the differencing operator acting on an entire vector

$$
U = \begin{bmatrix}
u_0\\
u_1\\
\vdots\\
u_N
\end{bmatrix}
$$

at once. For example, we can write

$$
D_+ = \frac{1}{h} \begin{bmatrix}
-1 & 1 & 0 & \dots & 0\\
0 & -1 & 1 & \dots & 0\\
\vdots & \ddots & \ddots & \vdots & \vdots\\
0 & \dots & 0 & -1 & 1\\
0 & \dots & 0 & 0 & -1\\
\end{bmatrix}
$$

and

$$
D^2 = \frac{1}{h^2} \begin{bmatrix}
-2 & 1 & 0 &  \dots & 0\\
1 & -2 & 1 &  \dots & 0\\
\vdots & \ddots & \ddots & \vdots & \vdots\\
0 & \dots & \dots 1 & -2 & 1\\
0 & \dots & 0 & 1 & -2\\
\end{bmatrix}.
$$

Recall, our problem was to find $u \in (-1, 1)$ satisfying

$$
\begin{align}
\frac{\text{d}^2 u}{\text{d} x^2} u &= f\\
u(-1) &= a\\
\frac{\text{d} u}{\text{d} x}(1) &= b
\end{align}
$$

In matrix form, this becomes

$$
\underbrace{D^2}_{A} \underbrace{\begin{bmatrix}
u_0\\
u_1\\
\vdots\\
u_N
\end{bmatrix}}_{U} = \underbrace{\begin{bmatrix} f_0\\f_1\\\vdots\\f_N\end{bmatrix}}_{F}.
$$

This works perfectly in the interior of the domain, but we need to figure out what to do at the boundaries. For example, we can't use the standard differencing operator for $D^2$ on $u_N$, because $u_{N+1}$ does not exist.
Fortunately, our boundary conditions inform how to modify the matrix appropriately.

### Dirichlet conditions

These conditions, of the form

$$
u(-1) = a
$$

specify the _value_ of the solution at a particular point (or set of points). This means, that rather than solving a small equation to determine the value at this point, we _already know_ and can instead replace the relevant rows of the matrix. Let us suppose we have ordered our points such that $u(-1)$ corresponds to $u_0$. Then we have

$$
\begin{bmatrix}
1 & 0 & \dots & 0\\
 & & & \\
 & & A_{1:,:} & \\
 & & &\\
\end{bmatrix}
U = \begin{bmatrix} a\\ \\ F_{1:} \\ \\ \end{bmatrix}.
$$

In general, if we have a boundary value $\alpha_i$ that constrains $u_i$ then we replace the $i$th row with the identity, and the $i$th value in the right hand side with $\alpha_i$.

This modification destroys any symmetry that might have existed in the matrix $A$, since we have zeroed rows, but not the corresponding columns. If we write the linear system in block form, we can, however, see a way around this:

$$
\begin{bmatrix}
I & 0\\
A_{10} & A_{11}
\end{bmatrix}
\begin{bmatrix}
U_0\\
U_1
\end{bmatrix}
=
\begin{bmatrix}
F_0\\
F_1
\end{bmatrix}
$$

Since we know $U_0$ (they are just $F_0$), we can forward-substitute and move the lower-left block of the matrix onto the right hand side, to produce

$$
\begin{bmatrix}
I & 0\\
0 & A_{11}
\end{bmatrix}
\begin{bmatrix}
U_0\\
U_1
\end{bmatrix}
=
\begin{bmatrix}
F_0\\
F_1 - A_{10}F_0
\end{bmatrix}.
$$

This is often a convenient form to work with.

Alternately, since the equations for $U_0$ are just the identity, we can write our solver to just handle

$$
A_{11} U_1 = F_1 - A_{10}F_0
$$

and insert the boundary values into a big vector whenever we need to visualise it.

## Neumann conditions

We can now treat boundary conditions that constrain the value of the solution, but recall that the condition at $x=1$ instead constrains gradient of the solution. We cannot do this by setting values, but must instead form an equation for the boundary value.

### One-sided difference

There are typically two ways to do this. We either come up with a one-sided differencing formula for the derivative directly. For example, recalling the one-sided difference we might replace the boundary term

$$
\frac{\text{d} u}{\text{d} x}(1) = b
$$

by

$$
\frac{u_n - u_{n-1}}{h} = b.
$$

This is simple but has some potential drawbacks

1. We need to make a different choice for the discretisation on the boundary to that in the interior
2. This choice may not have the same order of accuracy
3. It may destroy symmetry that previously existed in the problem.

### Ghost values

An alternate option is to introduce a (or possibly more than one) _ghost value_ outside of the domain such that we can then just use our interior discretisation. We then define the value of this ghost point to be the reflection (possibly weighted by the boundary value) of the interior point.

That is, we introduce $u_{n+1} = u(x_{n+1})$ and set

$$
u_{n+1} = u_{n-1} + 2b(\underbrace{x_n - x_{n-1}}_{h}).
$$

Now we can use our interior discretisation

$$
\frac{-u_{n-1} + 2u_n - u_{n+1}}{h^2} = f(x_n)
$$

substituting in the definition of $u_{n+1}$ we obtain

$$
\begin{align}
\frac{-u_{n-1} + 2u_n - (u_{n-1} + 2bh)}{h^2} &= f(x_n)\\
\frac{2(u_n - u_{n-1})}{h^2} &= f(x_n) + \frac{2b}{h}\\
\frac{u_n - u_{n-1}}{h^2} &= \frac{f(x_n)}{2} + \frac{b}{h}
\end{align}.
$$

Let's compare these approaches.

```python
def laplacian(N, rhsfunc):
    x = numpy.linspace(0, 1, N+1)
    h = 1/N
    rhs = rhsfunc(x)
    e = numpy.ones(N)
    # interior discretisation
    L = (2*numpy.eye(N+1) - numpy.diag(e, 1) - numpy.diag(e, -1)) / h**2
    return x, L, rhs, h

def apply_dirichlet(L, rhs, h, vals, indices):
    N, _ = L.shape
    diag = numpy.eye(1, N)
    bcmask = numpy.zeros(N, dtype=bool)
    bcmask[indices] = True
    # Dirichlet rows
    L[numpy.ix_(bcmask)] = numpy.vstack([numpy.roll(diag, i) for i in indices])
    rhs[numpy.ix_(bcmask)] = vals
    # Forward substitute
    rhs[numpy.ix_(~bcmask)] -= L[numpy.ix_(~bcmask, bcmask)] @ vals
    L[numpy.ix_(~bcmask, bcmask)] = 0
    return L, rhs, h

def apply_neumann_oneside(L, rhs, h, b, index):
    N, _ = L.shape
    assert index == N - 1
    L[index, :] = 0
    L[index, index] = 1/h
    L[index, index - 1] = -1/h
    rhs[index] = b
    return L, rhs, h

def apply_neumann_ghost(L, rhs, h, b, index):
    N, _ = L.shape
    L[index, index] /= 2
    rhs[index] = b/h + rhs[index]/2
    return L, rhs, h
```

We'll solve

$$
\begin{align}
-\frac{\text{d}^2 u}{\text{d} x^2} &= e^x \text{ in } (0, 1)\\
u(0) &= e^{0}\\
\frac{\text{d} u}{\text{d} x} &= e^{1}\\
\end{align}
$$

with convenient exact solution $u(x) = e^{x}$.

```python
N = 10
rhsfunc = lambda x: -numpy.exp(x)
exact = lambda x: numpy.exp(x)
x, L, rhs, h = laplacian(N, rhsfunc)
L, rhs, h = apply_dirichlet(L, rhs, h, [exact(0)], [0])
L, rhs, h = apply_neumann_oneside(L, rhs, h, exact(1), N)
uoneside = numpy.linalg.solve(L, rhs)

x, L, rhs, h = laplacian(N, rhsfunc)
L, rhs, h = apply_dirichlet(L, rhs, h, [exact(0)], [0])
L, rhs, h = apply_neumann_ghost(L, rhs, h, exact(1), N)
ughost = numpy.linalg.solve(L, rhs)
```

```python
pyplot.figure()
pyplot.plot(x, uoneside, label="Computed Oneside")
pyplot.plot(x, ughost, label="Computed Ghost")
pyplot.plot(x, exact(x), label="Exact")
pyplot.legend();
```

## Observations

Perhaps unsurprisingly, the one-sided application of the Neumann conditions performs worse than the ghost version. Interestingly, they are effectively the *same* discretisation in the matrix, the only difference is that in the ghost version, we corrected the right hand side we're solving for by a small amount to take into account the issues in the one-sided discretisation.

It looks like we have a lower-order scheme. Let's check by performing an MMS test.

To do so, we have to introduce how to measure errors. Since our discrete solution $u_i$ is supposed to approximate $u(x_i)$ it is natural to consider the pointwise errors $u_i - u(x_i)$. Let us now consider how to measure the size of the error vector (or indeed any vector).

$$
E = \begin{bmatrix} u_0 - u(x_0)\\
\vdots\\
u_n - u(x_n)
\end{bmatrix}
$$

So far we have been using the $\infty$-norm or $\max$-norm.

$$
\|E\|_\infty := \max_{0 \le i \le n} |E_i| = \max_{0 \le i \le n} |u_i - u(x_i)|
$$

which measures the largest pointwise error over the interval.

Other common norms are the $1$-norm

$$
\|E\|_1 = h \sum_{i=0}^n |E_i|
$$

and the $2$-norm

$$
\|E\|_2 = \left(h \sum_{i=0}^n |E_i|^2 \right)^{1/2}.
$$

Notice the factor of $h$ appearing in these definitions. This is needed so the norm does not spuriously grow when we add more points.

### Aside

These are special cases of $l_p$ norms

$$
\|E\|_p = \left(h \sum_{i=0}^n |E_i|^p\right)^{1/p}.
$$

```python
# Here we use the 2-norm
def error(u, exact, h):
    return numpy.sqrt(h)*numpy.linalg.norm(u - exact)
```

```python
def mms_errors(neumann):
    errors = []
    Ns = numpy.asarray(list(2**i for i in range(4, 11)))
    rhsfunc = lambda x: -numpy.exp(x)
    exact = lambda x: numpy.exp(x)
    for N in Ns:
        x, L, rhs, h = laplacian(N, rhsfunc)
        L, rhs, h = apply_dirichlet(L, rhs, h, [exact(0)], [0])
        L, rhs, h = neumann(L, rhs, h, exact(1), N)
        u = numpy.linalg.solve(L, rhs)
        errors.append(error(u, exact(x), 1/N))
    return 1/Ns, numpy.asarray(errors)
```

```python
_, oneside = mms_errors(apply_neumann_oneside)
hs, ghost = mms_errors(apply_neumann_ghost)
pyplot.figure()
pyplot.loglog(hs, oneside, "o", label="Oneside");
pyplot.loglog(hs, ghost, "x", label="Ghost");
pyplot.loglog(hs, hs, label="$\mathcal{O}(h)$");
pyplot.loglog(hs, hs**2, label="$\mathcal{O}(h^2)$");
pyplot.legend();
```

This confirms our suspicion that the one-sided differencing for the Neumann condition is only first-order accurate.

An alternative approach to obtaining a second-order scheme (rather than the ghost method above) is to try and determine a second-order accurate one-sided difference approximation to the first derivative. We will state an example first, and then see where it comes from.

A second-order accurate one-sided approximation to the first derivative is obtained with

$$
\frac{\text{d} u}{\text{d} x} \approx \frac{1}{h}\left(\frac{3}{2} u_i - 2 u_{i-1} + \frac{1}{2} u_{i-2}\right)
$$

```python
def apply_neumann_oneside_second(L, rhs, h, b, index):
    N, _ = L.shape
    assert index == N - 1
    L[index, :] = 0
    L[index, index] = 3/(2*h)
    L[index, index - 1] = -2/h
    L[index, index - 2] = 1/(2*h)
    rhs[index] = b
    return L, rhs, h
```

```python
_, oneside = mms_errors(apply_neumann_oneside)
_, second = mms_errors(apply_neumann_oneside_second)
hs, ghost = mms_errors(apply_neumann_ghost)
pyplot.figure()
pyplot.loglog(hs, oneside, "o", label="Oneside");
pyplot.loglog(hs, second, "s", label="Oneside second order");
pyplot.loglog(hs, ghost, "x", label="Ghost");
pyplot.loglog(hs, hs, label="$\mathcal{O}(h)$");
pyplot.loglog(hs, hs**2, label="$\mathcal{O}(h^2)$");
pyplot.legend();
```

This converges at second order as, perhaps, expected. The absolute error is a little worse than the ghosted version. However, this approach is sometimes more convenient, especially on irregularly spaced meshes.

## Deriving high-order finite difference stencils

Where did the approximation

$$
\frac{\text{d} u}{\text{d} x} \approx \frac{1}{h}\left(\frac{3}{2} u_i - 2 u_{i-1} + \frac{1}{2} u_{i-2}\right)
$$

come from? Given some points at which we're allowed to evaluate $u$, we can derive an appropriate formula from the Taylor series using the *method of undetermined coefficients*. This works in a very similar way to determining the truncation error for a given expansion.

For the example above, we want to approximation $u'(x)$ and we are given $u_i = u(x)$, $u_{i-1} = u(x - h)$, and $u_{i-2} = u(x - 2h)$. We can write our differencing operator as a *linear combination* of the provided points

$$
D_2 u(x) = a u(x) + bu(x-h) + c u(x - 2h)
$$

where our goal is to determine $a$, $b$, and $c$ to minimise the truncation error (that is, give the best possible accuracy).

Let's Taylor-expand on the right hand side

$$
D_2 u(x) = a u(x) + b \overbrace{\left(u(x) - hu'(x) + \frac{h^2}{2} u''(x) - \frac{h^3}{6} u'''(x)\right)}^{u(x - h)} + c \overbrace{\left(u(x) - 2h u'(x) + \frac{4h^2}{2}u''(x) - \frac{8 h^3}{6} u'''(x)\right)}^{u(x-2h)} + \mathcal{O}(h^4)
$$

gathering terms we have

$$
D_2 u(x) = (a + b + c)u(x) - (b + 2c) h u'(x) + \frac{1}{2}(b + 4c)h^2 u''(x) - \frac{1}{6}(b + 8c) h^3 u'''(x) + \mathcal{O}(h^4).
$$

To maximise the accuracy of agreement with $u'(x)$ we need

$$
\begin{aligned}
a + b + c &= 0 && \text{zeroing the $h^0 u(x)$ term}\\
b + 2c &= -\frac{1}{h} && \text{ensuring that we have a $u'(x)$ term}\\
b + 4c &= 0 && \text{zeroing the $h^2 u''(x)$ term}\\
b + 8c &= 0 && \text{zeroing the $h^3 u'''(x)$ term}.
\end{aligned}
$$

Since we have only three unknowns, we can only satisfy three equations. To maxmimise the accuracy, we'll choose to zero the $h^2$ and $h^0$ terms, and live with the $h^3$ term. We therefore need to solve the linear system

$$
\begin{bmatrix}
1 & 1 & 1\\
0 & 1 & 2\\
0 & 1 & 4
\end{bmatrix}
\begin{bmatrix}
a \\ b \\ c
\end{bmatrix}
=
\begin{bmatrix}
0 \\ -\frac{1}{h} \\ 0
\end{bmatrix}.
$$

```python
import numpy
A = numpy.asarray([[1, 1, 1],
                   [0, 1, 2],
                   [0, 1, 4]])
b = numpy.asarray([0, -1, 0])

numpy.linalg.solve(A, b)
```

So we have

$$
\begin{bmatrix}
a \\ b \\ c
\end{bmatrix} =
\frac{1}{2h}
\begin{bmatrix}
3\\
-4\\
1
\end{bmatrix}
$$

and hence our optimal formula is

$$
D_2 u(x) = \frac{1}{2h}(3 u_i - 4 u_{i-1} + u_{i-2})
$$

as advertised. We can immediately determine the accuracy of this approximation since we know the first term we did not manage to match exactly is

$$
- \frac{1}{6}(b + 8c) h^3 u'''(x)
$$

substituting in the values for $b$ and $c$ we have

$$
\begin{aligned}
D_2 u(x) - u'(x) &= -\frac{1}{6}\left(\frac{-2}{h} + \frac{8}{2h}\right) h^3 u'''(x) + \mathcal{O}(h^4)\\
                 &= -\frac{1}{3} h^2 u'''(x) + \mathcal{O}(h^4)
\end{aligned}
$$

so this approximation is second order accurate.

```python
def dtwo(x, u):
    h = x[2:] - x[1:-1]
    du = 1/(2*h) * (3 * u[2:] - 4*u[1:-1] + u[:-2])
    return x[2:], du

grids = 2**numpy.arange(3, 10)

def error(f, df, op):
    for n in grids:
        x = numpy.linspace(-1, 1, n)
        x, y = op(x, f(x))
        yield numpy.sqrt(1/n)*numpy.linalg.norm(y - df(x), None)

pyplot.figure()
pyplot.loglog(1/grids, list(error(numpy.sin, numpy.cos, dtwo)), marker="o", linestyle="none", label=dtwo.__name__)

pyplot.xlabel("Resolution ($h$)")
pyplot.ylabel("$l_2$ error in derivative")

pyplot.loglog(1/grids, 1/grids**2, label="$h^2$")
pyplot.loglog(1/grids, 1/grids**2, label="$h^2$")
pyplot.legend();
```

```python
numpy.linalg.norm?
```

## Questions

1. Use this technique to derive a third-order accurate one-sided differencing operator for $\frac{\text{d}}{\text{d} x}$ using 4 points $u(x)$, $u(x - h)$, $u(x - 2h)$, $u(x - 3h)$.

```python

```

## Differencing for advection

For centered difference approximations, there is no asymmetry in the stencil. For one-sided approximations, however, there is. If we just consider a three-point region centered around a point $i$, we can write the three approximations to $\frac{\text{d}}{\text{d} x}$ we have seen as stencils:

$$
\begin{aligned}
D_+ &= \frac{1}{h}\begin{bmatrix}0 & -1 &1\end{bmatrix}\\
D_- &= \frac{1}{h}\begin{bmatrix}-1 &  1 & 0\end{bmatrix}\\
D_0 &= \frac{1}{2h}\begin{bmatrix}-1 & 0 & 1\end{bmatrix}\\
\end{aligned}.
$$

Recall that we noticed that the centered difference approximation sometimes gave catastrophic results (all zero derivatives) for very rough functions. The question therefore might arise how to pick between $D_+$ and $D_-$. We will study this using the linear *advection equation* as a prototype.

This equation models the transport of some material by a bulk motion. This (especially when talking about fluid flow) is called convection. As usual [wikipedia has lots of information](https://en.wikipedia.org/wiki/Advection).

It looks remarkably benign, find $u(x, t)$ satisfying

$$
\partial_t u + c \cdot \nabla u = f(t, x)
$$

Where $c$ is the advecting velocity and

$$
\nabla u = \partial_x u
$$
in one dimension, and
$$
\nabla u = \begin{bmatrix} \partial_x u\\ \partial_y u\end{bmatrix}
$$
in two dimensions.

This is a first-order PDE, for which we need to supply one boundary condition (to pin down the spatial derivative) and one initial condition (to start everything off). The boundary condition, it turns out, has to be at the *inflow* boundary.

Let's try and solve this equation with an explicit Euler time integration scheme on the interval $[0, 5]$ and look at the effect of the different differencing operators. If we write this out we have (using superscripts for time points and subscripts for spatial points)

$$
u^{n+1}_i = u^n_i + \Delta t (f^n_i - c D u^n_i).
$$

We'll set the forcing function $f$ to be zero and pick boundary condition

$$
u(t, 0) = 0
$$

and initial condition

$$
u(0, x) = e^{-2(x - 2.5)^2}
$$

```python
L = 30
nx = 50
x = numpy.linspace(0, L, nx + 1)
h = L/nx
u = numpy.exp(-2*(x - L/2)**2)
uhat = numpy.zeros_like(x)
uhat[nx//3:2*nx//3] = 1

def Aupwind(nx, h, c):
    A = numpy.zeros((nx+1, nx+1), dtype=float)
    for i in range(1, nx+1):
        A[i, i-1] = c/h
        A[i, i] = -c/h
    # Boundary condition (will be fixed later)
    A[0, 0] = 0
    return A

def Adownwind(nx, h, c):
    A = numpy.zeros((nx+1, nx+1), dtype=float)
    for i in range(1, nx):
        A[i, i] = c/h
        A[i, i+1] = -c/h
    # Boundary condition (will be fixed later)
    A[0, 0] = 0
    A[nx, nx] = c/h
    return A

t = 0
tfinal = 10
c = 1
dt = 2/(nx + 1)

Id = numpy.eye(nx+1)

downwind = Id + dt*Adownwind(nx, h, c)
upwind = Id + dt*Aupwind(nx, h, c)

u = uhat
hist = [(t, u)]
while t < tfinal:
    u = upwind @ u
    hist.append((t, u))
    t += dt
hist = numpy.asarray(hist)
```

```python
pyplot.figure()
for t, hist_ in hist[::len(hist)//10]:
    pyplot.plot(x, hist_, label=f"$t = {t:3.1f}$")
pyplot.legend();
```

## Observations

1. The "downwind" discretisation is unstable (blowing up near the boundary). The upwind version is stable, if we have a small enough time step. This makes physical sense, because the downwind discretisation is trying to obtain information "from the future".

2. The upwind discretisation is *dissipative*: the correct physical solution is just to transport the initial condition to the right, but we see that the peak spreads and flattens.

3. This also occurs with sharp fronts (e.g. transporting a hat function)

4. Some of this gets easier if we naturally have some diffusion (a second derivative) in the equation (as in the coursework).

```python

```