```python
from resources.workspace import *
from IPython.display import display
from scipy.integrate import odeint
import copy

%matplotlib inline
```

# Lyapunov exponents and eigenvalues

A **Lypunov exponent** can be understood loosely as a kind of generalized eigenvalue for time-depenent linear transformations, or for the linearization of a nonlinear evolution.

What do eigenvalues tell us about a matrix and why might the above results seem intuitive?

Consider the equation for the evolution of the pertubations $\boldsymbol{\delta}^i_k$.  We can write,

$$\begin{align}
& \boldsymbol{\delta}_k^i = \mathbf{x}_k^c - \mathbf{x}_k^i \\
\Rightarrow & \dot{\boldsymbol{\delta}}_k^i = f(\mathbf{x}_k^c) - f(\mathbf{x}_k^i).
\end{align}$$

But for small perturbations, we can reasonably make an approximation with a Taylor expansion,

$$\begin{align}
 f(\mathbf{x}_k^c) - f(\mathbf{x}_k^i) \approx \nabla f\rvert_{\mathbf{x}c}  \boldsymbol{\delta}^i_k, & &
\end{align}$$

where the term,

$$f\rvert_{\mathbf{x}c}$$

is the gradient with respect to the state variables, i.e., the **[Jacobian matrix](https://en.wikipedia.org/wiki/Jacobian_matrix_and_determinant)**, evaluated at the control trajectory.

This means that for small perturbations, the evolution is well approximated by the linear Jacobian equations, and we can think of these linear equations having some kind of generalized eigenvalues, describing the invariant (exponential) growth and decay rates for the system.

#### The power method

The method of breeding errors above is conceptually very similar to the classical [power method](https://en.wikipedia.org/wiki/Power_iteration) for finding the leading eigenvalue of a diagonalizable matrix:

* Suppose $\mathbf{M}\in\mathbb{R}^{n\times n}$ is a diagonalizable matrix, with eigenvalues,

$$
\rvert \mu_1 \rvert > \rvert\mu_2\rvert \geq \cdots \geq \rvert\mu_n\rvert,
$$

i.e., $\mathbf{M}$ has a single eigenvalue of magnitude greather than all its others.

* Let $\mathbf{v}_0 \in \mathbb{R}^n$ be a randomly selected vector, with respect to the Gaussian distribution on $\mathbb{R}^n$.

* We define the algorithm,

$$\begin{align}
\mathbf{v}_{k+1} \triangleq \frac{\mathbf{M} \mathbf{v}_k}{ \left\rvert \mathbf{M} \mathbf{v}_k\right\rvert} & &
\widehat{\mu}_{k+1} \triangleq \mathbf{v}_{k+1}^{\rm T} \mathbf{M} \mathbf{v}_{k+1}
\end{align}$$

as the power method.

It is easy to verify that with probability one, the sequence $\widehat{\mu}_k$ converges to the dominant eigenvalue, $\mu_1$, and $\mathbf{v}_k$ converges to an eigenvector for the dominant eigenvalue.

**Exc 4.20**: Fill in the code below to write an algorithm for the power method.

```python
def power_method(M, v, number_iterations):
    """takes a diagonalizable matrix M and returns approximations for the leading eigenvector/eigenvalue"""

    for i in range(number_iterations):
        ### fill in missing lines here

    return v, mu
```

```python
# Example solution

# show_answer('power_method')
```

**Exc 4.22**: Test your solution to **Exc 4.20**.  Use the code and slider below to study the rate of convergence.  In this case, the matrix will have eigenvalues

$$\begin{align}
\left\{r^i : \hspace{2mm} i =0, 1, 2, \hspace{2mm} \text{and} \hspace{2mm} r\in(1,2]\right\}
\end{align}$$

The parameter $k$ defines how many iterations of the power method are computed.  How does the value $r$ affect the number of iterations necessary to reach convergence?

```python
def animate_power_convergence_rate(k=1, r=1.5):

    # We define a well conditioned matrix M, depending on the ratio of the eigenvalues
    M = array([r ** i for i in range(3)])
    M = np.diag(M)
    e_3 = array([0, 0, 1])

    # define a random initial condition
    np.random.seed(0)
    v = randn(3)
    v = v / sqrt(v.T @ v)

    # and storage for the series of approximations
    v_hist = zeros(k+1)
    v_hist[0] = e_3.T @ v

    mu_hist = zeros(k+1)
    mu_hist[0] = v.T @ M @ v

    # for the number of iterations k, return the power method approximation
    for it in range(1,k+1):
        np.random.seed(0)
        v, mu = power_method(M, v, it)
        v_hist[it] = np.arccos(e_3.T @ v)
        mu_hist[it] = mu

    # PLOTTING
    fig = plt.figure(figsize=(16,8))
    ax1 = plt.subplot(121)
    ax2 = plt.subplot(122)
    ax1.plot(range(0,k+1), v_hist)
    ax2.plot(range(0,k+1), mu_hist)
    ax1.set_ybound([0,1.05])
    ax2.set_ybound([.9,4])

    t_scl = np.floor_divide(k+1, 10)

    ax1.set_xticks(range(0, k+1, t_scl + 1))
    ax2.set_xticks(range(0, k+1, t_scl + 1))

    ax1.text(0, 1.07, r'Angle between $\mathbf{v}_k$ and eigenvector', size=20)
    ax2.text(0, 4.05, r'Value of $\mu_k$', size=20)
    ax1.tick_params(
        labelsize=20)
    ax2.tick_params(
        labelsize=20)

    plt.show()

w = interactive(animate_power_convergence_rate, k=(1,15), r=(1.05,2, .05))
w
```

Exc 4.24.a : Suppose the power method is performed on a generic diagonalizable matrix $\mathbf{M}\in\mathbb{R}^{n\times n}$, with eigenvalues

$$\begin{align}
\rvert \mu_1 \rvert > \rvert\mu_2 \rvert\geq \cdots \geq \rvert\mu_n \rvert,
\end{align}$$

with a randomly selected initial vector $\mathbf{v}_0$, with respect to the Gaussian distribution on $\mathbb{R}^n$.

Can you conjecture what is the order of convegence for the sequences $\mathbf{v}_k$ and $\widehat{\mu}_k$?

**Hint**: the rate depends on the eigenvalues.

**Exc 4.42.b***: Prove the rate of convergence.

```python
# Answer

# show_answer('power_method_convergence_rate')
```

Exc 4.28* : We have brushed over why the algorithm described above converges with *probability one*, can you prove why this is the case?

```python
# Answer

# show_answer('probability_one')
```

Exc 4.30.a : Let $\widehat{\mu}_k$ be defined as in **Exc 4.24**.  Suppose we define a sequence of values,

$$\begin{align}
\widehat{\lambda}_T = \frac{1}{T} \sum_{k=1}^T\log\left(\rvert \widehat{\mu}_k\right \rvert).
\end{align}$$

Answer the following:

 Can you conjecture what $\widehat{\lambda}_T$ converges to as $T \rightarrow \infty$?

  **Hint**: Use the fact that $\widehat{\mu}_k \rightarrow \mu_1$ as $k \rightarrow \infty$

 Suppose we define the Lyapunov exponents as the log-average growth rates of the matrix $\mathbf{M}$.  What can you guess about the relationship between the eigenvalues and the Lyapunov exponents of the matrix $\mathbf{M}$?

Exc 4.30.b*: Prove that the limit

$$\begin{align}
\lim_{T \rightarrow \infty} \widehat{\lambda}_T
\end{align}$$

exists, and what quantity it converges to.

```python
# Answers

# show_answer('lyapunov_exp_power_method')
```

#### The QR algorithm

The power method is an intuitive method for finding the dominant eigenvalue for a special class of matrices.  However, we generally want to find directions that may also be growing, though more slowly than the dominant direction.

Intuitively, if we are tracking a control trajectory with data assimilation and we corrected the forecast errors only in the direction of dominant error growth, we may still lose track of the control trajectory, only it would be more slowly than the dominant rate of growth.

There is a simple generalization of the power method for finding higher dimensional subspaces.  We may consider *separating* perturbations into directions that grow at different rates.  One easy way to perform this is to construct a *moving frame* in the span of the perturbations.  If there is only one perturbation, then the power method constructs precisely a 1-dimensional moving frame, with a vector that is always of norm 1.

If there are two perturbations we can construct a moving frame in the span of the perturbations with a [Gram-Schmidt](https://en.wikipedia.org/wiki/Gram%E2%80%93Schmidt_process) step.  A visualization of the Gram-Schmidt process for three vectors is picuted in the visualization below.

**By Lucas V. Barbosa [Public domain], from Wikimedia Commons**

In our case, suppose we have two initial, orthogonal vectors

$$
\mathbf{x}_0^1, \mathbf{x}_0^2
$$

which we will propagate forward.  We define for each $j=1,2$,

$$
\widehat{\mathbf{x}}^j_1 \triangleq \mathbf{M} \mathbf{x}^j_0.
$$

The first vector will follow the usual power method, i.e.,

$$
\mathbf{x}^1_1 \triangleq \frac{\widehat{\mathbf{x}}_1^1}{\left\rvert \widehat{\mathbf{x}}_1^1\right\rvert},
$$

However, we want to separate the second vector $\widehat{\mathbf{x}}_1^2$ so the new perturbations don't align.  We thus remove the components in the direction of $\mathbf{x}_1^1$, before we normalize $\widehat{\mathbf{x}}_1^2$.

$$\begin{align}
\mathbf{y}^2_1 &\triangleq \widehat{\mathbf{x}}_1^2- \langle \mathbf{x}_1^1,  \widehat{\mathbf{x}}^2_1\rangle \mathbf{x}_1^1 \\
\mathbf{x}^2_1 & \triangleq \frac{\mathbf{y}_1^2}{\left\rvert \mathbf{y}_1^2 \right\rvert}
\end{align}$$

It is easy to see by definition that $\mathbf{x}_1^1, \mathbf{x}_1^2$ are orthogonal, but we can also show an important dynamical property with this transformation.  Define the following coefficients,

$$
\begin{align}
U^{11}_1 &=\left\rvert \widehat{\mathbf{x}}_1^1\right\rvert \\
U^{22}_1 &=\left\rvert \mathbf{y}_1^2 \right\rvert \\
U^{12}_1 &= \langle \mathbf{x}^1_1, \widehat{\mathbf{x}}_1^2\rangle
\end{align}
$$

**Exc 4.32**: Can you write the recursion for the vectors $\mathbf{x}_0^1, \mathbf{x}_0^2$ transformed into $\mathbf{x}_1^1,\mathbf{x}_1^2$ with the coefficients $U^{ij}_1$ defined above in matrix form?  Can you write the recursion for an arbitrary number of steps $k\in\{1,2,\cdots\}$?

```python
# Answer

# show_answer('gram-schmidt')
```

The above procedure defines the *naive* QR algorithm --- one should note that there are more computationally efficient versions of this algorithm utilized in standard linear algebra software libraries.  However, this simple intuition forms the basis for many powerful theoretical results.

The QR algorithm (in its refined version) is the standard method for computing the [Schur decomposition](https://en.wikipedia.org/wiki/Schur_decomposition) for a matrix, which is used for many purposes as it is a numerically stable alternative to the [Jordan Cannonical Form](https://en.wikipedia.org/wiki/Jordan_normal_form), pictued below:

**By Jakob.scholbach [CC BY-SA 3.0 or GFDL], from Wikimedia Commons**

The Jordan Canonical form is highly appealing as it is the diagonal or "almost-diagonal" form of a matrix.  However, this is highly unstable to compute in most applications.

The Schur decomposition relaxes this further, from "almost-diagonal" to upper triangular, another useful form for a matrix.  In particular, the Schur decomposition is one approach to find **all eigenvalues** for a matrix, separated into a **chain of descending growth and decay rates**.

Exc 4.34: Suppose a matrix $\mathbf{M}$ has a Schur decomposition, given as,

 $$ \begin{align}
\mathbf{M} = \mathbf{Q} \mathbf{U} \mathbf{Q}^{\rm T},
\end{align}$$

where $\mathbf{U}$ is strictly upper triangular, and $\mathbf{Q}$ is orthogonal such that $\mathbf{Q}^{\rm T} = \mathbf{Q}^{-1}$.  Can you prove that the eigenvalues of $\mathbf{M}$ are the diagonal elements of $\mathbf{U}$?

If $\mathbf{Q}^j$ is the $j$-th column of $\mathbf{Q}^j$, what does the product

$$\begin{align}
\left(\mathbf{Q}^j\right)^{\rm T} \mathbf{M} \mathbf{Q}^j
\end{align}$$

equal in terms of the earlier quantities? Hint: how does this relate to the power method?

```python
# Answer

# show_answer('schur_decomposition')
```

Exc 4.36: Can you conjecture what the Schur decomposition will take in the case that the matrix $\mathbf{M}$ has complex eigenvalues?

```python
# Answer

# show_answer('real_schur')
```

### Next: [Lyapunov vectors and ensemble based covariances](T4 - Lyapunov vectors and ensemble based covariances.ipynb)

tutorials/DA and the Dynamics of Ensemble Based Forecasting/T3 - Lyapunov exponents and eigenvalues.ipynb


```python
import os
import sympy
from sympy.parsing.sympy_parser import parse_expr
from sympy.polys.polytools import degree
import random
from examgen.lib.helper import alpha, digits_nozero, get_coefficients, render, shuffle
```

```python
import wolframalpha
client = wolframalpha.Client("WR2K94-3VL28J9KAK")
```

```python
def poly1(x):
    vals = sum([k*x**i for i,k in enumerate(reversed(get_coefficients(2)))])
    return vals

def poly2(x):
    vals = sum([k*x**i for i,k in enumerate(reversed(get_coefficients(3)))])
    return vals

def poly3(x):
    vals = sum([k*x**i for i,k in enumerate(reversed(get_coefficients(4)))])
    return vals

def polyn(x, order=3):
    vals = sum([k*x**i for i,k in enumerate(reversed(get_coefficients(order)))])
    return vals

_functions = [sympy.sin, sympy.cos, sympy.tan, sympy.ln, sympy.sqrt, sympy.exp,
              lambda a: a, poly1, poly2, poly3]
```

```python
def make_int_pow_prob_wa(var="x", order=3):
    """
    Generates a n-order polynomial to be integrated.

    x : charector for the variable to be solved for. defaults to "x".
                            OR
        a list of possible charectors. A random selection will be made from them.
    n : order of the polynomial or a list of possible orders. Defaults to 3.
        A random selection will be made from them.

    """
    if isinstance(var, str):
        var = sympy.Symbol(var, positive=True)
    elif isinstance(var, list):
        var = sympy.Symbol(random.choice(var), positive=True)
    if isinstance(order, list):
        order = random.choice(order)
    eq = 0;
    sol = 0;
    length = random.randint(1,3)
    for i in range(1,length+1):
        a = random.randint(1,10)*random.choice([-1,1])
        b = random.randint(1,10)
        n = random.randint(1,5)
        while (not n%order):
            n = random.randint(1,5)
        if (True):
            sign = random.choice([-1,1])
            eq += sympy.Rational(a,b)*(var**sympy.Rational(n,order))**sign
            sol+= sympy.Rational(a*n,b*order)*(var**(sympy.Rational(n*sign,order)-1))
        else:
            eq += (sympy.Rational(a,b)*sympy.root(var**n,order))**random.choice([-1,1])

    sol = sympy.latex(sympy.powsimp(sympy.integrate(eq, var), combine='all',force=True))
    eq = sympy.latex(sympy.Integral(eq, var))
    eq = 'd'.join(eq.split("\\partial"))
    eq = "$$" + eq + "$$"
    sol = "$$" + sol + " + C $$"
    return eq, sol
```

```python
eq,sol=make_int_pow_prob_wa()
```

```python
res = client.query(eq)
```

```python
print(next(res.results).text)
```

    -(20 x^(5/3))/7 - 10/(7 x^(2/3))

```python
eq
```

    '$$\\int \\left(- \\frac{5 x^{\\frac{5}{3}}}{2} + \\frac{5}{2 x^{\\frac{5}{3}}}\\right)\\, dx$$'

```python
sol
```

    '$$- \\frac{15 x^{\\frac{8}{3}}}{16} - \\frac{15}{4 x^{\\frac{2}{3}}} + C $$'

```python

```


# Week 6 worksheet 1: Numerical differentiation and differential equations

This notebook is modified from one created by Charlotte Desvages.

Another branch of numerical calculus is the computation of approximations to **derivatives**. This week, we introduce **finite difference** approximations and their order of accuracy to estimate derivatives numerically, and we explore their applications in computing numerical solutions to ordinary differential equations.

The best way to learn programming is to write code. Don't hesitate to edit the code in the example cells, or add your own code, to test your understanding. You will find practice exercises throughout the notebook, denoted by 🚩 ***Exercise $x$:***.

#### Displaying solutions

Solutions will be released one week after the worksheets are released, as a new `.txt` file in the same GitHub repository. After pulling the file to your workspace, run the following cell to create clickable buttons under each exercise, which will allow you to reveal the solutions.

```python
%run scripts/create_widgets.py W06-W1
```

---
### 📚 Book sections

- **ASC**: sections 3.2, 7.1
- **PCP**: sections 8.1, 8.2

---
## 1. Finite difference approximations to derivatives

The lecture introduced two different approximations to the first derivative $F'(x)$ of a function $F(x)$: the forward difference $D_1(x)$ and the centred difference $D_C(x)$. Both of these approximations, and many more, can be derived using **Taylor expansion**.

As a reminder from your Calculus courses, the Taylor expansion of a *sufficiently regular*$^\ast$ function $F(x)$ around a point $x = x_0$ is given by

$$
F(x) = \sum_{p=0}^\infty \frac{(x - x_0)^p}{p!} F^{(p)}(x_0),
$$

where $F^{(p)}(x_0)$ is the $p$th derivative of $F$ evaluated at $x = x_0$, i.e. $F^{(p)}(x_0) = \left. \frac{d^p F}{dx^p}\right|_{x = x_0}$.

---

$^\ast$ Here, *sufficiently regular* or *sufficiently smooth* means that the function is continuously differentiable in a neighbourhood of $x$ at least as many times as is necessary for any required derivatives to be well-defined in that neighbourhood.

---

### 1.1. Derivation of the forward difference approximation

For a sufficiently regular function $F$ and a small enough $\Delta x$, consider the Taylor expansion of $F(x + \Delta x)$ about $x$:

$$
  F \left( x + \Delta x \right) = F \left( x \right) + \Delta x F' \left( x \right) + \frac{\Delta x^2}{2} F'' \left( x \right) + \frac{\Delta x^3}{6} F'''\left(x\right) + \dots
$$

Since we are seeking an approximation to $F'(x)$, we can rearrange the above to make $F'(x)$ the subject:

\begin{align}
F'(x) &= \frac{1}{\Delta x}\left( F(x + \Delta x) - F(x) - \frac{\Delta x^2}{2}  F'' \left( x \right) - \frac{\Delta x^3}{6} F'''\left(x\right) - \dots \right) \\
&= \underbrace{\frac{F(x + \Delta x) - F(x)}{\Delta x}}_{:= D_1(x)} - \frac{\Delta x}{2}  F'' \left( x \right) - \frac{\Delta x^2}{6} F'''\left(x\right) - \dots
\end{align}

With this, we have derived our **forward difference** approximation $D_1(x)$ -- for sufficiently small $\Delta x$, truncating the Taylor series expansion to the 1st Taylor polynomial gives $F'(x) \approx D_1(x)$.

### 1.2. Order of accuracy of a finite difference approximation

In fact, the equation above also gives us the **error** between this approximation and the exact derivative:

\begin{equation}
  D_1(x) - F' \left( x \right)
  = \frac{\Delta x}{2}  F'' \left( x \right) + \frac{\Delta x^2}{6} F'''\left(x\right) + \dots
\end{equation}

Consider using this approximation to estimate the derivative of $F$ at a given point $x$. The value of $F(x)$ and its derivatives at that point are fixed -- in the expression above, the **accuracy** of the approximation $D_1(x)$ depends entirely on the choice of **step size** $\Delta x$. The question is now: how does the error increase or decrease depending on the value of $\Delta x$?

Here, for **sufficiently small** $\Delta x$ (and if $F'' \left( x \right) \ne 0$), the first term on the right-hand side dominates the infinite sum. The error magnitude scales approximately **linearly** with the step size $\Delta x$. We can write

$$
D_1(x) - F' \left( x \right) = O(\Delta x).
$$

That is, for sufficiently small $\Delta x$, further halving $\Delta x$ will approximately divide the error magnitude by two. The approximation is said to be **first order accurate**.

---
#### 🚩 Definition: Order of accuracy of finite difference approximations

Let $F(x)$ be a sufficiently smooth function, and $D(x)$ be a finite difference approximation of its $m$th derivative $F^{(m)}(x)$, with step size $\Delta x$. We say that $D(x)$ is **$k$th order accurate** if, for sufficiently small $\Delta x$,

$$
D(x) - F^{(m)}(x) = O(\Delta x^k).
$$

---
🚩 ***Exercise 1***: Following the same method as above, using the Taylor expansion of $F(x - \Delta x)$ about $F(x)$, derive the backward difference approximation $D_{-1}(x)$ to the first derivative $F'(x)$, which should take the form

$$
D_{-1}(x) = \frac{F(x) - F(x - \Delta x)}{\Delta x}.
$$

What is its order of accuracy?

```python
%run scripts/show_solutions.py W06-W1_ex1
```

---
🚩 ***Exercise 2***: Using the Taylor expansions of both $F(x + \Delta x)$ and $F(x - \Delta x)$ about $F(x)$, derive the centred difference approximation $D_C(x)$ to the first derivative $F'(x)$, and show that it is second order accurate.

```python
%run scripts/show_solutions.py W06-W1_ex2
```

---
🚩 ***Exercise 3***: The **forward** difference approximation $D_1(x)$ is *exact* only for a particular set of (continuous) functions -- what are these functions?

In other words, find conditions on $F(x)$ such that $\forall x$ and $\forall \Delta x \neq 0$, we have $D_1(x) - F'(x) = 0$.

```python
%run scripts/show_solutions.py W06-W1_ex3
```

---
### 1.3. Numerical investigation of the order of accuracy

Similarly to what we did for investigating the degree of precision and rate of convergence of a quadrature rule, we can also perform numerical experiments to provide evidence for the order of accuracy of a finite difference approximation.

---
🚩 ***Exercise 4***: Consider the function

\begin{equation}
  F \left( x \right) = e^{-x^2}.
\end{equation}

Find the first derivative of $F$, analytically (by hand). Evaluate this derivative at $x = -\tfrac{1}{2}$.

Then, use the backward difference approximation $D_{-1}(x)$ to find approximations for $F' \left( -\tfrac{1}{2} \right)$ for each $\Delta x \in \left\{ 0.04, 0.02, 0.01, 0.005 \right\}$. Plot the magnitude of the error in the derivative approximation against $\Delta x$.

Why do these error values suggest that the backward difference approximation for the derivative is first order accurate? What might you expect to happen if the forward difference approximation were to be replaced with the centred difference approximation?

```python

```

```python
%run scripts/show_solutions.py W06-W1_ex4
```

---
### 1.4. Higher derivatives

We now seek a finite difference approximation for the second derivative $F''(x)$. Consider, for instance, the forward difference approximation $D_1(x)$ for $F'(x)$. We have

$$
F'(x) \approx D_1(x) = \frac{F(x + \Delta x) - F(x)}{\Delta x}.
$$

Taking the derivative on both sides allows us to obtain an approximation for $F''(x)$:

$$
F''(x) \approx \frac{F'(x + \Delta x) - F'(x)}{\Delta x}.
\qquad \color{red}{(\ast)}
$$

Now, let us use $D_1(x)$ again to approximate $F'(x + \Delta x)$ and $F'(x)$, using the same step size:

$$
F''(x) \approx \frac{\frac{F(x + 2\Delta x) - F(x + \Delta x)}{\Delta x} - \frac{F(x + \Delta x) - F(x)}{\Delta x}}{\Delta x}
= \frac{F(x + 2\Delta x) - 2F(x + \Delta x) + F(x)}{\Delta x^2}
:= D^{(2)}_1(x).
$$

This final expression gives us a forward difference approximation of the second derivative $F''(x)$.

---

To investigate its **accuracy**, we use the Taylor expansions of $F(x + 2\Delta x)$ and $F(x + \Delta x)$ about $F(x)$:

\begin{align}
F \left( x + \Delta x \right) &= F \left( x \right) + \Delta x F' \left( x \right) + \frac{\Delta x^2}{2} F'' \left( x \right) + \frac{\Delta x^3}{6} F'''\left(x\right) + \dots \\
F \left( x + 2\Delta x \right) &= F \left( x \right) + (2\Delta x) F' \left( x \right) + \frac{(2\Delta x)^2}{2} F'' \left( x \right) + \frac{(2\Delta x)^3}{6} F'''\left(x\right) + \dots
\end{align}

Substituting these into $D^{(2)}_1(x)$ to calculate the error, we obtain

$$
D^{(2)}_1(x) - F''(x)
= \Delta x F'''(x) + \frac{7\Delta x^2}{12} F''''(x) + \dots
= O(\Delta x),
$$

therefore this approximation is first order accurate. (The detailed derivation will be shown in a video this week.)

---
🚩 ***Exercise 5***: Instead of using $D_1(x)$ again to approximate $F'(x + \Delta x)$ and $F'(x)$ in $\color{red}{(\ast)}$, we can use the **backward** difference approximation $D_{-1}(x)$, to obtain a *centred* difference approximation $D^{(2)}_C(x)$ to the *second* derivative $F''(x)$.

Derive the expression for $D^{(2)}_C(x)$, and using the same procedure as above, show (analytically) that it is second order accurate.

```python
%run scripts/show_solutions.py W06-W1_ex5
```

---
🚩 ***Exercise 6***: Consider, again, the function

\begin{equation}
  F \left( x \right) = e^{-x^2}.
\end{equation}

Find the **second** derivative of $F$, analytically (by hand). Evaluate this second derivative at $x = -\tfrac{1}{2}$.

Then, using the **centred** difference approximation $D^{(2)}_C(x)$ to find approximations for $F'' \left( -\tfrac{1}{2} \right)$ for several carefully chosen values of $\Delta x$, perform similar numerical experiments as in Exercise 4 to confirm that the order of accuracy of $D^{(2)}_C(x)$ is indeed 2.

```python

```

```python
%run scripts/show_solutions.py W06-W1_ex6
```

---
## 2. Computing solutions of ODEs

Finite difference approximations of derivatives can be used to compute approximate solutions of differential equations -- this is in fact a widely used method in many applications in mathematical modelling.

---

Consider the first order ordinary differential equation (ODE)

\begin{equation}\label{eqn:ode}
  \frac{\mathrm{d} \Psi}{\mathrm{d} t} = -\gamma \Psi \quad \textrm{ for } t > 0,
\end{equation}

where $\gamma$ is a real and non-negative constant, and subject to an initial condition $\Psi \left( t = 0 \right) = \Psi_0$. The ODE together with this initial condition form an *initial value problem* (IVP). The exact solution is

\begin{equation}
  \Psi \left( t \right) = \Psi_0 e^{-\gamma t},
\end{equation}

as can, for example, be verified via direct substitution and checking that the initial condition is satisfied.

---

Of course, for this particular IVP, the analytical solution is straightforward to compute -- we don't really need a numerical method! This example is just to demonstrate the general process and principles used to solve differential equations numerically -- for many real-life problems with more complex models, numerical methods are often the only way to obtain a useful solution.

---

Consider the replacement of the derivative with a discrete approximation, for example via

\begin{align}
  \psi_0 & = \Psi_0, \\
  \frac{\psi_{n + 1} - \psi_{n}}{\Delta t} & = -\gamma \psi_n \quad \textrm{ for } n = 0, 1, 2, \ldots, \qquad \color{blue}{(\ast)}
\end{align}

with $\Delta t > 0$. Here $\psi_n$ is a *numerical approximation* for $\Psi$ at $t = n \Delta t$, in other words we have $\psi_n \approx \Psi(n \Delta t)$. We also used a *forward difference* approximation of the derivative on the LHS:

$$
\left. \frac{\mathrm{d} \Psi}{\mathrm{d} t} \right|_{t = n \Delta t}
\approx \frac{\Psi(t + \Delta t) - \Psi(t)}{\Delta t}
\approx \frac{\psi_{n + 1} - \psi_{n}}{\Delta t}.
$$

$\color{blue}{(\ast)}$ is known as a "forward Euler" discretisation of the original differential equation. Such discretisations are often generally referred to as *finite difference schemes*.

- Since the initial value $\psi_0  = \Psi_0$ is known (we impose it), for $n = 0$, the only unknown in $\color{blue}{(\ast)}$ is $\psi_1$, and we can re-arrange the expression to express the "new" value $\psi_{1}$ in terms of the "old" value $\psi_0$.
- With this, we can compute $\psi_1$.
- Now, at $n=1$, the only unknown in $\color{blue}{(\ast)}$ is $\psi_2$; we can compute it following the same process we used for $\psi_1$.
- We can continue computing any "new" value $\psi_{n+1}$ **iteratively**, using the previously computed value $\psi_n$. Rearranging $\color{blue}{(\ast)}$ accordingly gives

\begin{equation}
  \psi_{n + 1} = \left( 1 - \gamma \Delta t \right) \psi_n \quad \textrm{ for } n = 0, 1, 2, \ldots
\end{equation}

---
🚩 ***Exercise 7***: Compute and plot the numerical approximation for the solution to the initial value problem, obtained by using the forward Euler scheme with $\psi_0 = 1$, with $100$ steps of size $\Delta t = 0.1$, and with $\gamma = 1$.

```python

```

```python
%run scripts/show_solutions.py W06-W1_ex7
```

---
🚩 ***Exercise 8***: Experiment with different values of $\gamma$ and $\Delta t$. What happens if $\gamma \Delta t$ becomes too large?

```python

```

```python
%run scripts/show_solutions.py W06-W1_ex8
```

---
🚩 ***Exercise 9*** (exploration): We used a forward difference approximation for the derivative; can you find other discretisations for the problem?

```python

```

```python
%run scripts/show_solutions.py W06-W1_ex9
```


# 局部线性嵌入(Lcally Linear Embedding, LLE)

## 符号定义

|符号|含义|
|:-:|:-:|
|$\pmb{x}$|样本点|
|$X_i$|样本点$\pmb{x_i}$的k近邻点集合|
|$N$|总样本数|
|$N_i$|样本点$\pmb{x_i}$的k近邻点集合的样本总数|
|$\pmb{X_i^{(j)}}$|样本点$\pmb{x_i}$的k近邻点集合中的第j个样本点|
|$\pmb{w}$|线性组合的权重向量|
|$\pmb{y}$|样本点降维后坐标|

## 概念

LLE属于流形学习。流形学习假设所有数据点分布在嵌入于外维欧式空间的一个潜在的流形体上。LLE的核心思想是：流形在局部领域上是线性的，即任意一个点都可以由其周围的点在最小二乘意义下进行线性表示；降维后，任意一个点依然可以由其周围的点在最小二乘意义下进行线性表示，并且线性表示的系数不变。

## 原理

由于LLE假设局部是线性的，因此对于数据点$\pmb{x}$，其可以由其k-近邻数据点的线性组合进行表示

$$
\begin{equation}
    \pmb{x_i} = \sum_{j=1}^{N_i}w_j\pmb{x_j}
\end{equation}
$$

由于LLE假设降维后依然可以用相同的系数对数据点$\pmb{x}$进行表示，所以有

$$
\begin{equation}
    \pmb{y_i} = \sum_{j=1}^{N_i}w_j\pmb{y_j}
\end{equation}
$$

由上分析可以发现LLE有三步：
* 第一步为求得任意数据点$\pmb{x}$的k-近邻数据点集合；
* 第二步为求得任意数据点$\pmb{x}$的k-近邻数据点线性表示，并得到对应的权重向量$\pmb{w}$；
* 第三步为求得符合权重向量$\pmb{w}$的低维表示

## 推导

### 线性组合权重系数求解

对于线性表示的权重系数有如下的损失函数

$$
\begin{equation}
    \mathcal{L} = \sum_{i=1}^N||\pmb{x_i}-\sum_{j=1}^{N_i}w_{ij}\pmb{X_i^{(j)}}||_2^2, s.t. \sum_{j=1}^{N_i}w_{ij}=1
\end{equation}
$$

最佳的权重系数显然是上述的损失最小，即$\arg\min\limits_{W}\mathcal{L}$，其中$W$为由所有样本点的权重系数组成的矩阵

有
$$
\begin{equation}
   \begin{split}
   \arg\min\limits_{W}\mathcal{L} &=  \arg\min\limits_{W}\sum_{i=1}^N||\pmb{x_i}-\sum_{j=1}^{N_i}w_{ij}\pmb{X_i^{(j)}}||_2^2 \\
   &= \arg\min\limits_{W}\sum_{i=1}^N||\sum_{j=1}^{N_i}(\pmb{x_i}-\pmb{X_i^{(j)}})w_{ij}||_2^2 \\
   &= \arg\min\limits_{W}\sum_{i=1}^N||([\underbrace{\pmb{x_i},\pmb{x_i}, \cdots, \pmb{x_i}}_{N_i}]-X_i)\pmb{w_i}||_2^2 \\
   &= \arg\min\limits_{W}\sum_{i=1}^N\pmb{w_i}^T([\underbrace{\pmb{x_i},\pmb{x_i}, \cdots, \pmb{x_i}}_{N_i}]-X_i)^T([\underbrace{\pmb{x_i},\pmb{x_i}, \cdots, \pmb{x_i}}_{N_i}]-X_i)\pmb{w_i}
   \end{split}, s.t. \sum_{j=1}^{N_i}w_{ij}=\pmb{w_i}^T[1, 1, \cdots, 1]_k^T = 1
\end{equation}
$$

即求解如下存在约束的优化问题
$$
\begin{equation}
    \left\{
    \begin{split}
    & \min\mathcal{L}, \\
    & s.t. \pmb{w_i}^T[1, 1, \cdots, 1]_k^T - 1 = 0
    \end{split}
    \right .
\end{equation}
$$

考虑运用拉格朗日乘子法
即
$$
\begin{equation}
    \left\{
    \begin{split}
    &\frac{\partial{\mathcal{L}}}{\partial{\pmb{w_i}}}-\lambda\frac{\partial{(\pmb{w_i}^T[1, 1, \cdots, 1]_k^T - 1)}}{\partial{\pmb{w_i}}} = \pmb{0} \\
    & \pmb{w_i}^T[1, 1, \cdots, 1]_k^T - 1 = 0
    \end{split}
    \right .
\end{equation}
$$

记$S_i=([\underbrace{\pmb{x_i},\pmb{x_i}, \cdots, \pmb{x_i}}_{N_i}]-X_i)^T([\underbrace{\pmb{x_i},\pmb{x_i}, \cdots, \pmb{x_i}}_{N_i}]-X_i)$

同时记$\pmb{1_k} = [1, 1, \cdots, 1]_k^T$

得到
$$
\begin{equation}
    \begin{split}
        &2S_i\pmb{w_i} - \lambda\pmb{1_k} = 0\\
        \Rightarrow &\pmb{w_i} = \frac{1}{2}S_i^{-1}\lambda\pmb{1_k}
    \end{split}
\end{equation}
$$

又因为$\pmb{w_i}^T\pmb{1_k} - 1 = 0$

可以得到
$$
\begin{equation}
    \begin{split}
        & \pmb{1_k}^T \frac{1}{2}S_k^{-1}\lambda\pmb{1_k} = 1 \\
        \Rightarrow &\lambda = \frac{1}{\pmb{1_k}^T \frac{1}{2}S_i^{-1}\pmb{1_k}}
    \end{split}
\end{equation}
$$

所以得到

$$
\begin{equation}
    \pmb{w_i} = \frac{S_i^{-1}\pmb{1_k}}{\pmb{1_k}^TS_i^{-1}\pmb{1_k}}, i=1, 2, \cdots, N
\end{equation}
$$

应用上式即可求得所有数据点由k-近邻数据点线性表示的权重向量

## 低维表示求解

考虑到低维表示下，依然需要满足上一步得到的线性组合结果，因此有如下的损失函数

$$
\begin{equation}
    \mathcal{L_2} = \sum_{i=1}^N||\pmb{y_i}-\sum_{j=1}^{N_i}w_{ij}\pmb{Y_i^{(j)}}||_2^2, s.t. \sum_{i=1}^N\pmb{y_i}=0, \sum_{i=1}^N\pmb{y_i}\pmb{y_i}^T=N\pmb{I_{d\times d}}
\end{equation}
$$

上式不同于第二步。第二步是线性组合的权重未知，但是数据点已知。但是在第三步是线性组合的权重已知，而降维后的坐标未知。

定义如下的矩阵

$$
\begin{equation}
    V = [v_{ji}]_{N\times N},
    v_{ji} =
    \left\{
    \begin{split}
    & w_{ij}, 数据点j是数据点i的近邻点 \\
    & 0, else
    \end{split}
    \right .
\end{equation}
$$

因此有下式

$$
\begin{equation}
    \sum_{j=1}^{N_i}w_{ij}\pmb{Y_i^{(j)}} = \sum_{j=1}^Nv_{ji}\pmb{Y_i^{(j)}} = Y\pmb{V_i}
\end{equation}
$$

其中$\pmb{V_i}$是矩阵$V$的第$i$列

$$
\begin{equation}
    \begin{split}
    \mathcal{L_2}
    &= \sum_{i=1}^N||\pmb{y_i} - Y\pmb{V_i}||_2^2 \\
    &= \sum_{i=1}^N||Y(\pmb{I_i} - \pmb{V_i})||_2^2
    \end{split}
\end{equation}
$$

其中$\pmb{I_i}$是单位矩阵的第$i$列，上式中的$Y(\pmb{I_i} - \pmb{V_i})$可以被视为一个$N\times N$矩阵的第$i$列

由于对于任意矩阵$A=[\pmb{a_1}, \pmb{a_2}, \cdots, \pmb{a_{N}}]$，有
$$
\begin{equation}
    \sum_{i=1}^N||\pmb{a_i}||_2^2 = \sum_{i=1}^N\pmb{a_i}^T\pmb{a_i} = tr(AA^T)
\end{equation}
$$

因此可以得到
$$
\begin{equation}
    \mathcal{L_2}=tr(Y(I-V)(I-V)^TY^T)
\end{equation}
$$

令上式中的$(I-V)(I-V)^T=M=M^T$
有
$$
\begin{equation}
    \mathcal{L_2}=tr(YMY^T)
\end{equation}
$$

第三步为如下的优化问题

$$
\begin{equation}
    \left\{
    \begin{split}
    & \min\limits_{Y} \mathcal{L_2} \\
    & YY^T = N\pmb{I_{d\times d}}
    \end{split}
    \right .
\end{equation}
$$

同样的，使用拉格朗日乘子法可得
$$
\begin{equation}
    \left\{
    \begin{split}
    &\frac{\partial{\mathcal{L_2}}}{\partial{Y}}-\lambda\frac{\partial({YY^T-N\pmb{I_{d\times d}}})}{\partial{Y}} = 0 \\
    & YY^T = N\pmb{I_{d\times d}}
    \end{split}
    \right.
\end{equation}
$$

由$\frac{\partial{\mathcal{L_2}}}{\partial{Y}}-\lambda\frac{\partial({YY^T-N\pmb{I_{d\times d}}})}{\partial{Y}} = 0$可得

$$
\begin{equation}
    \begin{split}
    & YM+YM^T-2\lambda Y=0 \\
    \Rightarrow &YM^T=\lambda Y \\
    \Rightarrow &MY^T=\lambda Y^T
    \end{split}
\end{equation}
$$

显然，$Y^T$为矩阵$M$特征向量构成的矩阵

由此可得
$$
\begin{equation}
    \mathcal{L_2} = tr(YMY^T) = tr(\lambda YY^T)=tr(\lambda N\pmb{I}_{d\times d})=\lambda Nd
\end{equation}
$$

因此为了使得损失$\mathcal{L_2}$尽可能小，应当选择矩阵$M$前d个最小特征值对应的特征向量。

又因为$M$为对称非负定矩阵，且$(I-V)\pmb{1_N}=0$，因此$M$的最小特征值为0。在实际使用时，通常排除0这一最小特征值对应的特征向量，而选用其后的d个特征值对应的特征向量

## 步骤
* 定义数据集$X=\{\pmb{x_1}, \pmb{x_2}, \cdots, \pmb{x_N}\}$、邻域大小$k$以及权系数矩阵$V$
* 使用k-近邻算法得到所有样本点的k个最近邻点
* 对每一个样本点利用下式求解线性组合权重并赋值到权系数矩阵$V$的相应位置
$$
    \pmb{w_i} = \frac{S_i^{-1}\pmb{1_k}}{\pmb{1_k}^TS_i^{-1}\pmb{1_k}}, i=1, 2, \cdots, N
$$
* 计算下示矩阵的前$d+1$个最小特征值对应的特征向量，去掉最小特征值对应的特征向量后，转置得到LLE结果
$$
    M=(I-V)(I-V)^T
$$

```python
import numpy as np
from scipy import linalg
import matplotlib.pyplot as plt
from sklearn import manifold, datasets
from sklearn.neighbors import NearestNeighbors
```

```python
class MyLLE(object):

    def __init__(self, input_data, reduction_dims, n_neighbors):

        self.input_data = np.array(input_data)
        self.reduction_dims = reduction_dims
        self.n_neighbors = n_neighbors

        self.samples_num, self.input_dims = self.input_data.shape

        # knn result
        self.knn_mask = np.zeros((self.samples_num, self.samples_num))

        # weight matrix
        self.weight_matrix = np.zeros((self.samples_num, self.samples_num))

        # reduction result
        self.output_mat = np.zeros((self.samples_num, reduction_dims))

    def __call__(self):
        self.cal_knn_result()
        self.cal_linear_weight()
        self.cal_reduction_result()

        return np.array(self.output_mat)

    def cal_knn_result(self):

        # calculate the k nearest neighbors of each samples
        nbrs = NearestNeighbors(n_neighbors=self.n_neighbors+1).fit(self.input_data)
        self.knn_mask = nbrs.kneighbors_graph(self.input_data).toarray() - np.eye(self.samples_num)

    def cal_linear_weight(self):
        for i in range(self.samples_num):
            # expand current sample vector
            expand_sample_mat = np.repeat(self.input_data[i].reshape(1, -1), self.n_neighbors, axis=0).T
            # obtain neighbors marix
            neighbors_mat = self.input_data[np.argwhere(self.knn_mask[i]!=0).reshape(-1), :].T
            # cal s_matrix
            temp_mat = expand_sample_mat - neighbors_mat
            s_mat = np.dot(temp_mat.T, temp_mat)

            # --------------------------------
            # amount of regularization to add for the problem to be
            # well-posed in the case of n_neighbors > n_dim
            # this method comes from the source code of LLE in sklearn
            trace = np.trace(s_mat)
            if trace > 0:
                R = 1e-3 * trace
            else:
                R = 1e-3

            s_mat.flat[::self.n_neighbors + 1] += R
            s_mat_invert = linalg.pinv(s_mat)

            # all-ones vector
            ones_vector = np.ones((self.n_neighbors, 1))
            # cal linear weight
            linear_weight_vector = np.matmul(s_mat_invert, ones_vector)/np.sum(s_mat_invert)

            self.weight_matrix[np.argwhere(self.knn_mask[i]!=0), i] = linear_weight_vector

    def cal_reduction_result(self):

        # M matrix
        temp_matrix = np.eye(self.samples_num)-self.weight_matrix
        m_mat = np.matmul(temp_matrix, temp_matrix.T)

        # cal eigenvalue and eigenvector
        _, topk_e_vecs = self._topk_eigen(m_mat, self.reduction_dims)
        self.output_mat = topk_e_vecs.copy()

    def _topk_eigen(self, mat, k, skip_num=1):
        e_vals, e_vecs = np.linalg.eig(mat)
        sorted_ind = np.argsort(e_vals)

        topk_e_vals = e_vals[sorted_ind[skip_num:k+skip_num]]
        topk_e_vecs = e_vecs[:, sorted_ind[skip_num:k+skip_num]]
        return topk_e_vals, topk_e_vecs
```

```python
# create data
raw_samples, samples_color = datasets.make_swiss_roll(n_samples=1500, random_state=1000)

# LLE, using sklearn
sklearn_lle, _ = manifold.locally_linear_embedding(raw_samples, n_neighbors=20, n_components=2)

# My LLE
my_lle = MyLLE(raw_samples, 2, 20)
my_lle_result = my_lle()

# ----------------------
# draw func
fig = plt.figure(figsize=(20, 20))

ax = fig.add_subplot(221, projection="3d")
ax.set_title("Original data", fontsize=15)
ax.scatter(raw_samples[:, 0], raw_samples[:, 1], raw_samples[:, 2], c=samples_color, cmap=plt.cm.Spectral)

ax = fig.add_subplot(222)
plt.xticks([])
plt.yticks([])
plt.title("Projection of data using Sklearn LLE", fontsize=15)
ax.scatter(sklearn_lle[:, 0], sklearn_lle[:, 1], c=samples_color, cmap=plt.cm.Spectral)

ax = fig.add_subplot(223, projection="3d")
ax.set_title("Original data", fontsize=15)
ax.scatter(raw_samples[:, 0], raw_samples[:, 1], raw_samples[:, 2], c=samples_color, cmap=plt.cm.Spectral)

ax = fig.add_subplot(224)
plt.xticks([])
plt.yticks([])
plt.title("Projection of data using My LLE", fontsize=15)
ax.scatter(my_lle_result[:, 0], my_lle_result[:, 1], c=samples_color, cmap=plt.cm.Spectral)

plt.show()
```


# Mutual Information Estimation

$\def\abs#1{\left\lvert #1 \right\rvert}
\def\Set#1{\left\{ #1 \right\}}
\def\mc#1{\mathcal{#1}}
\def\M#1{\boldsymbol{#1}}
\def\R#1{\mathsf{#1}}
\def\RM#1{\boldsymbol{\mathsf{#1}}}
\def\op#1{\operatorname{#1}}
\def\E{\op{E}}
\def\d{\mathrm{\mathstrut d}}$

```python
# init
import ipywidgets as widgets
import matplotlib.pyplot as plt
import numpy as np
import pandas as pd
import seaborn as sns
import tensorboard as tb
import torch
import torch.optim as optim
from IPython import display
from torch import Tensor, nn
from torch.nn import functional as F
from torch.utils.tensorboard import SummaryWriter

%load_ext tensorboard
%matplotlib inline

def plot_samples_with_kde(df, **kwargs):
    p = sns.PairGrid(df, **kwargs)
    p.map_lower(sns.scatterplot, s=2)  # scatter plot of samples
    p.map_upper(sns.kdeplot)  # kernel density estimate for pXY
    p.map_diag(sns.kdeplot)  # kde for pX and pY
    return p

SEED = 0

# set device
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# create samples
XY_rng = np.random.default_rng(SEED)
rho = 1 - 0.19 * XY_rng.random()
mean, cov, n = [0, 0], [[1, rho], [rho, 1]], 1000
XY = XY_rng.multivariate_normal(mean, cov, n)

XY_ref_rng = np.random.default_rng(SEED)
cov_ref, n_ = [[1, 0], [0, 1]], n
XY_ref = XY_ref_rng.multivariate_normal(mean, cov_ref, n_)

Z = Tensor(XY).to(DEVICE)
Z_ref = Tensor(XY_ref).to(DEVICE)

ground_truth = -0.5 * np.log(1 - rho ** 2)
```

    The tensorboard extension is already loaded. To reload it, use:
      %reload_ext tensorboard

**How to estimate MI via KL divergence?**

In this notebook, we will introduce a few methods of estimating the mutual information ({prf:ref}`mi-estimation`) via KL divergence.

We first introduce the Mutual Information Neural Estimation (MINE) method in {cite}`belghazi2018mine`.

## MINE

The idea is to obtain MI {eq}`MI` from KL divergence {eq}`D` as follows:

$$
\begin{align*}
I(\R{X}\wedge \R{Y}) = D(\underbrace{P_{\R{X},\R{Y}}}_{P_{\R{Z}}}\| \underbrace{P_{\R{X}}\times P_{\R{Y}}}_{P_{\R{Z}'}}).
\end{align*}
$$

and then apply the DV formula {eq}`avg-DV` to estimate the divergence:

---

**Definition** MINE
:label: MINE

The idea of mutual information neural estimation (MINE) is to estimate $I(\R{X}\wedge\R{Y})$ by

$$
\begin{align}
\sup_{t_{\theta}: \mc{Z} \to \mathbb{R}} \overbrace{\frac1n \sum_{i\in [n]} t_{\theta}(\R{X}_i,\R{Y}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\theta}(\R{X}'_i,\R{Y}'_i)}}^{\R{I}_{\text{MINE}}(\theta):=}
\end{align}
$$ (MINE)

where

- the supremum is over $t_{\theta}$ representable by a neural network with trainable/optimizable parameters $\theta$,
- $P_{\R{X}',\R{Y}'}:=P_{\R{X}}\times P_{\R{Y}}$, and
- $(\R{X}'_i,\R{Y}'_i\mid i\in [n'])$ is the sequence of i.i.d. samples of $P_{\R{X}',\R{Y}'}$.

---

The above is not a complete description of MINE because there are some implementation details yet to be filled in.

### Resampling

**How to obtain the reference samples ${\R{Z}'}^{n'}$, i.e., ${\R{X}'}^{n'}$ and ${\R{Y}'}^{n'}$?**

We can approximate the i.i.d. sampling of $P_{\R{X}}\times P_{\R{Y}}$ using samples from $P_{\R{X},\R{Y}}$ by a re-sampling trick:

$$
\begin{align}
P_{\R{Z}'^{n'}} &\approx P_{((\R{X}_{\R{J}_i},\R{Y}_{\R{K}_i})\mid i \in [n'])}
\end{align}
$$ (resample)

where $\R{J}_i$ and $\R{K}_i$ for $i\in [n']$ are independent and uniformly random indices

$$
P_{\R{J},\R{K}} = \op{Uniform}_{[n]\times [n]}
$$

and $[n]:=\Set{1,\dots,n}$.

MINE {cite}`belghazi2018mine` uses the following implementation that samples $(\R{J},\R{K})$ but without replacement. You can change $n'$ using the slider for `n_`.

```python
rng = np.random.default_rng(SEED)

def resample(data, size, replace=False):
    index = rng.choice(range(data.shape[0]), size=size, replace=replace)
    return data[index]

@widgets.interact(n_=widgets.IntSlider(n, 2, n, continuous_update=False))
def plot_resampled_data_without_replacement(n_):
    XY_ = np.block([resample(XY[:, [0]], n_), resample(XY[:, [1]], n_)])
    resampled_data = pd.DataFrame(XY_, columns=["X'", "Y'"])
    plot_samples_with_kde(resampled_data)
    plt.show()
```

    interactive(children=(IntSlider(value=1000, continuous_update=False, description='n_', max=1000, min=2), Outpu…

In the above, the function `resample`
- uses `choice` to uniformly randomly select a choice
- from a `range` of integers going from `0` to
- the size of the first dimension of the `data`.

Note however that the sampling is without replacement.

**Is it a good idea to sample without replacement?**

**Exercise**

Sampling without replacement has an important restriction $n'\leq n$. Why?

**Solution**

To allow $n>n'$, at least one sample $(\R{X}_i,\R{Y}_i)$ needs to be sampled more than once.

**Exercise** To allow $n>n'$, complete the following code to sample with replacement and observe what happens when $n \gg n'$.

```python
@widgets.interact(n_=widgets.IntSlider(n, 2, 10 * n, continuous_update=False))
def plot_resampled_data_with_replacement(n_):
    ### BEGIN SOLUTION
    XY_ = np.block(
        [resample(XY[:, [0]], n_, replace=True), resample(XY[:, [1]], n_, replace=True)]
    )
    ### END SOLUTION
    resampled_data = pd.DataFrame(XY_, columns=["X'", "Y'"])
    plot_samples_with_kde(resampled_data)
    plt.show()
```

    interactive(children=(IntSlider(value=1000, continuous_update=False, description='n_', max=10000, min=2), Outp…

**Exercise**

Explain whether the resampling trick gives i.i.d. samples $(\R{X}_{\R{J}_i},\R{Y}_{\R{K}_i})$ for the cases with replacement and without replacement respectively?

**Solution**

The samples are identically distributed. However, they are not independent except in the trivial case $n=1$ or $n'=1$, regardless of whether the sample is with replacement or not. Consider $n=1$ and $n'=2$ as an example.

### Smoothing

To improve the stability of the training, MINE applies additional smoothing to the gradient calculation.

**How to train the neural network?**

{eq}`MINE` can be solved iteratively by minibatch gradient descent using the loss function:

$$
\begin{align}
\R{L}_{\text{MINE}}(\theta) &:= \overbrace{- \frac{1}{\abs{\R{B}}} \sum_{i\in \R{B}}  t_{\theta} (\R{X}_i, \R{Y}_i) }^{\R{L}(\theta):=} + \log \overbrace{\frac{1}{\abs{\R{B}'}} \sum_{i\in \R{B}'}  e^{t_{\theta} (\R{X}'_i, \R{Y}'_i)}}^{\R{L}'(\theta):=}
\end{align}
$$

where $\R{B}$ and $\R{B}'$ are batches of uniformly randomly chosen indices from $[n]$ and $[n']$ respectively.

The gradient expressed in terms of $\R{L}$ and $\R{L}'$ is:

$$
\begin{align}
\nabla \R{L}_{\text{MINE}}(\theta) &= \nabla \R{L}(\theta) + \frac{\nabla \R{L}'(\theta)}{\R{L}'(\theta)}.
\end{align}
$$

Variation in $\nabla \R{L}'(\theta)$ leads to the variation of the overall gradient especially when $\R{L}'(\theta)$ is small. With minibatch gradient descent, the sample average is over a small batch and so the variance can be quite large.

**How to reduce the variation in the gradient approximation?**

To alleviate the variation, MINE replaces the denominator $\R{L}'(\theta)$ by its moving average:

$$
\theta_{j+1} := \theta_j - s_j \underbrace{\left(\nabla \R{L}_j (\theta_j) + \frac{\nabla \R{L}'_j(\theta_j)}{\overline{\R{L}'}_j}\right)}_{\text{(*)}}
$$ (MINE:update)

where $\R{L}_j$ and $\R{L}'_j$ are the

$$
\overline{\R{L}'}_j =  \beta \overline{\R{L}'}_{j-1} + (1-\beta) \R{L}'(\theta_j)
$$ (MINE:L2bar)

for some smoothing factor $\beta\in [0,1]$.

**How to implement the smoothing?**

Instead of implementing a new optimizer, a simpler way is to redefine the loss at each step $i$ as follows {cite}`belghazi2018mine`:

$$
\R{L}_{\text{MINE},j}(\theta) = \R{L}_j(\theta) + \frac{\R{L}'_j(\theta)}{\overline{\R{L}'}_j}
$$ (MINE:mv)

where $\overline{\R{L}'}_j$ in {eq}`MINE:L2bar` is regarded as a constant in calculating the gradient. It is easy to verify that $\nabla \R{L}_{\text{MINE},j}(\theta_j)$ gives the desired gradient (*) in {eq}`MINE:update`.

In summary:

---

**Definition**

MINE estimates the mutual information $I(\R{X}\wedge\R{Y})$ as $\R{I}_{\text{MINE}}(\theta_j)$ {eq}`MINE` where $\theta_j$ is updated by descending along the gradient of $\R{L}_{\text{MINE},j}$ {eq}`MINE:mv` iteratively after $j$ steps.

---

### Implementation

Consider implementing MINE for the jointly gaussian $\R{X}$ and $\R{Y}$:

```python
SEED = 0

# set device
DEVICE = "cuda" if torch.cuda.is_available() else "cpu"

# create samples
XY_rng = np.random.default_rng(SEED)
rho = 1 - 0.19 * XY_rng.random()
mean, cov, n = [0, 0], [[1, rho], [rho, 1]], 1000
XY = XY_rng.multivariate_normal(mean, cov, n)

X = Tensor(XY[:, [0]]).to(DEVICE)
Y = Tensor(XY[:, [1]]).to(DEVICE)
ground_truth = -0.5 * np.log(1 - rho ** 2)
X.shape, Y.shape, ground_truth
```

    (torch.Size([1000, 1]), torch.Size([1000, 1]), 0.7405246745135301)

The tensors `X` and `Y` correspond to samples of $\R{X}$ and $\R{Y}$ respectively. The first dimension indices the different samples. The `ground_truth` is the actual mutual information $I(\R{X}\wedge\R{Y})$.

**Exercise**

Complete the definition of `forward` so that the neural network $t_{\theta}$ is a vectorized function that takes samples `x` and `y` of $\R{X}$ and $\R{Y}$ as input and approximates the density ratio $\frac{P_{\R{X},\R{Y}}}{P_{\R{X}}\times P_{\R{Y}}}$ at `(x, y)`:

- Use `torch.cat` to concatenate tensors `x` and `y` in the last dimension.
- Use `F.elu` for the activation function $\sigma$.

```python
class Net(nn.Module):
    def __init__(self, input_size=2, hidden_size=100, sigma=0.02):
        super().__init__()
        # fully-connected (fc) layers
        self.fc1 = nn.Linear(input_size, hidden_size)
        self.fc2 = nn.Linear(hidden_size, hidden_size)  # layer 2
        self.fc3 = nn.Linear(hidden_size, 1)  # layer 3
        nn.init.normal_(self.fc1.weight, std=sigma)  #
        nn.init.constant_(self.fc1.bias, 0)
        nn.init.normal_(self.fc2.weight, std=sigma)
        nn.init.constant_(self.fc2.bias, 0)
        nn.init.normal_(self.fc3.weight, std=sigma)
        nn.init.constant_(self.fc3.bias, 0)

    def forward(self, x, y):
        """
        Vectorized function that implements the neural network t(x,y).

        Parameters:
        -----------
        x, y: 2D Tensors where the first dimensions index different samples.
        """
        ### BEGIN SOLUTION
        a1 = F.elu(self.fc1(torch.cat([x, y], dim=-1)))
        a2 = F.elu(self.fc2(a1))
        t = self.fc3(a2)
        ### END SOLUTION
        return t

    def plot(self, xmin=-5, xmax=5, ymin=-5, ymax=5, xgrids=50, ygrids=50, ax=None):
        """Plot a heat map of a neural network net. net can only have two inputs."""
        x, y = np.mgrid[xmin : xmax : xgrids * 1j, ymin : ymax : ygrids * 1j]
        with torch.no_grad():
            z = (
                net(
                    Tensor(x.reshape(-1, 1)).to(DEVICE),
                    Tensor(y.reshape(-1, 1)).to(DEVICE),
                )
                .reshape(x.shape)
                .cpu()
            )
        if ax is None:
            ax = plt.gca()
        im = ax.pcolormesh(x, y, z, cmap="RdBu_r", shading="auto")
        ax.figure.colorbar(im)
        ax.set(xlabel=r"$x$", ylabel=r"$y$", title=r"Heatmap of $t(x,y)$")
```

To create and plot the neural network:

```python
torch.manual_seed(SEED)
net = Net().to(DEVICE)
net.plot()
```

**Exercise**

To implement a neural network trainer for MINE similar to `DVTrainer`, completes the definition of `loss` according to {eq}`MINE:mv`:

```python
class MINETrainer:
    def __init__(
        self, X, Y, net, n_iters_per_epoch, writer_params={}, m=1, beta=0.1, **kwargs
    ):
        """
        Neural estimator for Mutual Information based on MINE.

        Estimate I(X;Y) using samples of X and Y by training a network t to maximize
            avg(t(X,Y)) - avg(e^t(X',Y')) / m
        where samples of (X',Y') approximates P_X * P_Y using the resampling trick, and
        m is obtained by smoothing avg(e^t(X',Y')) with the factor beta.

        parameters:
        -----------

        X, Y : Tensors with first dimensions of the same size indicing the samples.
        net  : The neural network t that takes X and Y as input and output a real number for each a real number for each sample.
        n_iters_per_epoch : Number of iterations per epoch.
        m    : initial value for the moving average
        beta : Smoothing/forgetting factor between [0,1]
        writer_params     : Parameters to be passed to SummaryWriter for logging.
        """
        self.X = X
        self.Y = Y
        self.n = min(X.shape[0], Y.shape[0])
        self.beta = beta
        self.m = m
        self.net = net
        self.n_iters_per_epoch = n_iters_per_epoch

        # set optimizer
        self.optimizer = optim.Adam(self.net.parameters(), **kwargs)

        # logging
        self.writer = SummaryWriter(**writer_params)
        self.n_iter = self.n_epoch = 0

    def step(self, epochs=1):
        """
        Carries out the gradient descend for a number of epochs and returns the MI estimate evaluated over the entire data.

        Loss for each epoch is recorded into the log, but only one MI estimate is computed/logged using the entire dataset.
        Rerun the method to continue to train the neural network and log the results.

        Parameters:
        -----------
        epochs : number of epochs
        """
        for i in range(epochs):
            self.n_epoch += 1

            # random indices for selecting samples for all batches in one epoch
            idx = torch.randperm(self.n)

            # resampling to approximate the sampling of (X',Y')
            idx_X = torch.randperm(self.n)
            idx_Y = torch.randperm(self.n)

            for j in range(self.n_iters_per_epoch):  # loop through multiple batches
                self.n_iter += 1
                self.optimizer.zero_grad()

                # obtain a random batch of samples
                batch_X = self.X[idx[j : self.n : self.n_iters_per_epoch]]
                batch_Y = self.Y[idx[j : self.n : self.n_iters_per_epoch]]
                batch_X_ref = self.X[idx_X[j : self.n : self.n_iters_per_epoch]]
                batch_Y_ref = self.Y[idx_Y[j : self.n : self.n_iters_per_epoch]]

                # define the loss
                # BEGIN SOLUTION
                L = -self.net(batch_X, batch_Y).mean()
                L_ = self.net(batch_X_ref, batch_Y_ref).exp().mean()
                self.m = (1 - self.beta) * L_.detach() + self.beta * self.m
                loss = L + L_ / self.m
                # END SOLUTION
                loss.backward()
                self.optimizer.step()

            self.writer.add_scalar("Loss/train", loss.item(), global_step=self.n_iter)

        with torch.no_grad():
            idx_X = torch.randperm(self.n)
            idx_Y = torch.randperm(self.n)
            X_ref = self.X[idx_X]
            Y_ref = self.Y[idx_Y]
            estimate = (
                self.net(self.X, self.Y).mean()
                - self.net(X_ref, Y_ref).logsumexp(0)
                + np.log(self.n)
            ).item()
            self.writer.add_scalar("Estimate", estimate, global_step=self.n_epoch)
            return estimate
```

To create the trainer for MINE:

```python
trainer = MINETrainer(X, Y, net, n_iters_per_epoch=10)
```

To train the neural network:

```python
if input("Train? [Y/n]").lower() != "n":
    for i in range(10):
        print('MI estimate:', trainer.step(10))
    net.plot()
```

```python
%tensorboard --logdir=runs
```

    Reusing TensorBoard on port 6006 (pid 5413), started 0:01:34 ago. (Use '!kill 5413' to kill it.)

**Exercise**

See if you can get an estimate close to the ground truth by training the neural network repeated as shown below.

To clean-up:

```python
if input('Delete logs? [y/N]').lower() == 'y':
    !rm -rf ./runs
```

    Delete logs? [y/N] y

```python
tb.notebook.list() # list all the running TensorBoard notebooks.
while (pid := input('pid to kill? (press enter to exit)')):
    !kill {pid}
```

    Known TensorBoard instances:
      - port 6007: logdir runs (started 0:18:01 ago; pid 5113)
      - port 6006: logdir runs (started 1:17:51 ago; pid 4485)

    pid to kill? (press enter to exit) 5113
    pid to kill? (press enter to exit) 4485
    pid to kill? (press enter to exit)

## MI-NEE

**Is it possible to generate i.i.d. samples for ${\R{Z}'}^{n'}$?**

Consider another formula for mutual information:

---

**Proposition**
:label: MI-3D

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'}) - D(P_{\R{X}}\|P_{\R{X}'}) - D(P_{\R{Y}}\|P_{\R{Y}'})
\end{align}
$$ (MI-3D)

for any product reference distribution $P_{\R{X}'}\times P_{\R{Y}'}$ for which the divergences are finite.

---

---

**Corollary**
:label: MI-ub

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= \inf_{\substack{P_{\R{X}'}\in \mc{P}(\mc{X})\\ P_{\R{Y}'}\in \mc{P}(\mc{Y})}} D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'}).
\end{align}
$$ (MI-ub)

where the optimal solution is $P_{\R{X}'}\times P_{\R{Y}'}=P_{\R{X}}\times P_{\R{Y}}$, the product of marginal distributions of $\R{X}$ and $\R{Y}$.

---

---

**Proof**

{eq}`MI-ub` follows from {eq}`MI-3D` directly since the divergences are non-negative. To prove the proposition:

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &= H(\R{X}) + H(\R{Y}) - H(\R{X},\R{Y})\\
&= E\left[-\log dP_{\R{X}'}(\R{X})\right] - D(P_{\R{X}}\|P_{\R{X}'})\\
&\quad+E\left[-\log dP_{\R{Y}'}(\R{Y})\right] - D(P_{\R{Y}}\|P_{\R{Y}'})\\
&\quad-E\left[-\log d(P_{\R{X}'}\times P_{\R{Y}'})(\R{X},\R{Y})\right] + D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'})\\
&= D(P_{\R{X},\R{Y}}\|P_{\R{X}'}\times P_{\R{Y}'}) - D(P_{\R{X}}\|P_{\R{X}'}) - D(P_{\R{Y}}\|P_{\R{Y}'})
\end{align}
$$

---

*Mutual Information Neural Entropic Estimation (MI-NEE)* {cite}`chan2019neural` uses {eq}`MI-3D` to estimate MI by estimating the three divergences.

Applying {eq}`avg-DV` to each divergence in {eq}`MI-3D`:

$$
\begin{align}
I(\R{X}\wedge \R{Y}) &\approx \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t_{\R{X},\R{Y}}(\R{X}_i,\R{Y}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\R{X},\R{Y}}(\R{X}'_i,\R{Y}'_i)}\\
&\quad - \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t_{\R{X}}(\R{X}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\R{X}}(\R{X}'_i)} \\
&\quad - \sup_{t: \mc{Z} \to \mathbb{R}} \frac1n \sum_{i\in [n]} t_{\R{Y}}(\R{Y}_i) - \frac1{n'}\sum_{i\in [n']} e^{t_{\R{Y}}(\R{Y}'_i)}
\end{align}
$$ (MI-NEE)

$P_{\R{X}'}$ and $P_{\R{Y}'}$ are known distributions and so arbitrarily many i.i.d. samples can be drawn from them directly without using the resampling trick {eq}`resample`.

Indeed, since the choice of $P_{\R{X}'}$ and $P_{\R{Y}'}$ are arbitrary, we can also also train neural networks to optimize them. In particular, {eq}`MI-ub` is a special case where we can train neural networks to approximate $P_{\R{X}}$ and $P_{\R{Y}}$.


# The soil production function

This lesson produced by Simon M Mudd and Fiona J Clubb. Last update (13/09/2021)

Back in the late 1800s, people (including G.K. Gilbert) were speculating about the rates at which soil was formed. This might depend on things like the number of burrowing animals, the rock type, the number of plants, and other factors.

The soil is "produced" from somewhere, and usually it is produced from some combination of conversion of rock to sediments, addition of organic matter, and deposition of dust. But we are going to focus on the conversion of rock material to sediment that can move.

Gilbert suggested that the rate soil was produced (from underlying rocks) depended on the thickness of the soil. We can make a prediction about the relationship between soil thickness and the rate soil is produced, and we call this the *soil production function*.

This function has proposed to have a few different forms, which we will explore below.

## Exponential Soil Production

In lectures we identified that the rate of weathering on a hillslope could be described as an exponential function that depends on soil depth, with weathering rates declining as soil gets deeper (Heimsath et al., 1997):

$p = W_0 e^{-\frac{h}{\gamma}}$

where $W_0$ is the soil production rate with no soil, and $\gamma$ is a length scale that determines how quickly soil production falls off with depth.

Typical values for $W_0$ are in the range 0.01-1 mm/yr [(Perron, 2017)](http://www.annualreviews.org/doi/abs/10.1146/annurev-earth-060614-105405). Note that when you're doing numerical calculations you have to be consistent with units. We will always do calculations in length units of ***metres*** (m), time units of ***years*** (y) and mass units of ***kilograms*** (kg). However we might convert to other units for the purposes of plotting sensible numbers (e.g. Weathering rates in mm/y = m/y $\times$ 1000).

Let's take a look at what this function looks like by plotting it with python:

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
```

```python
h_locs = np.arange(0,2,0.1)
```

We define the soil production function:

```python
def soil_prod_function(h_locs, W_0 = 0.0001, gamma = 0.4):
    P = np.multiply(W_0,  np.exp( - np.divide(h_locs,gamma)  )  )
    return P
```

Now lets plot the function and see what it looks like.

In the code below there are two soil production functions that you can compare. For example if you make `W_0` twice as much as `W_02` that means the second soil production function produces soil twice as fast as the first when there is no soil.

```python
plt.rcParams['figure.figsize'] = [10, 4]
plt.clf()

# TRY CHANGING THE FOUR PARAMETER BELOW
# These two are for the first soil production function
W_0 = 0.0001
gamma = 0.4

# These two are for the second soil production function
W_02 = 0.0002
gamma2 = 0.4

# This bit calculates the functions
P = soil_prod_function(h_locs, W_0 = W_0, gamma = gamma)
P2 = soil_prod_function(h_locs, W_0 = W_02, gamma = gamma2)

# The rest of this stuff makes the figure
f, ax = plt.subplots(1, 1)
ax.plot(h_locs, P*1000,label="P1")
ax.plot(h_locs, P2*1000,label="P2")
ax.set_xlabel("Soil thickness ($m$)")
ax.set_ylabel("Soil production (mm/yr)")
plt.title("Two soil production function. Try playing with the parameters!")
ax.legend()
plt.tight_layout()
```

## The peaked soil production function

We also discussed in the lecture an alternative way in which soil may be produced: where there are very slow rates of soil production where there is bare bedrock, then soil peaks at some intermediate thickness before decreasing exponentially with increasing soil thickness. This model dates back to Gilbert (1877), and makes intuitive sense: water is needed for weathering processes as we discussed today. If there is bare bedrock, water is quickly transported through overland flow and little weathering can take place. If there is too much soil, then it's unlikely to be fully saturated down to the bedrock--soil interface.

In this section, we will make some plots of a hypothetical peaked (or humped) soil production function.

We will use the theoretical formulation from [Cox (1980)](https://onlinelibrary.wiley.com/doi/abs/10.1002/esp.3760050305) to calculate the weathering rate for a range of soil depths. This is a bit more complicated than the exponential function and has a bigger range of parameters:

\begin{equation}
W = W_0 (\alpha e^{-kh}) + (1 - \alpha)f \\
f = \Bigg(1 + c\frac{h}{h_c} - \frac{h^2}{{h_c}^2}\Bigg)
\end{equation}

You should recognise some of these parameters from the exponential equation. The first part of the equation is the exponential function multiplied by a coefficient, $\alpha$. $W$ is still the weathering rate, $W_0$ is the inital rate of soil production where there is no soil, and $h$ is soil depth. There are two new parameters: $h_c$ is a critical soil depth (m), and $c$ is an empirical constant. Anhert (1977) suggests that $c$ might vary between 1.7 - 2.3, $h_c$ might vary between 0.6 - 1.5, and $\alpha$ between 0 - 1. If $\alpha = 1$, then the relationship is simply the exponential function.

```python
# first, let's clear the original figure
plt.clf()
# make a new figure
fig, ax = plt.subplots()

k = 1

# define the critical depth for soil production
h_c = 0.5 #metres

# define the initial rate of soil production
W_0 = 0.0005 #m/year

# define the constant c
c = 2 #dimensionless

# define alpha
alpha = 0.2

# calculate the weathering rate for the range of soil depths, h
f = (1 + c*(h_locs/h_c) - h_locs**2/h_c**2)
W = W_0 * (alpha * np.exp(-k*h_locs) + (1 - alpha)*f)

# plot the new result with a blue dashed line
ax.plot(h_locs,W*1000.,'--', color='blue', label = 'Peaked function')

# add a legend
plt.legend(loc = 'upper right')

# set the y limit of the humped function to 0 (so we don't get negative weathering rates), and set the axis labels
ax.set_ylim(0,)
plt.xlabel("Soil Depth (m)")
plt.ylabel("Weathering Rate (mm/y)")
plt.title("The peaked soil production function")
plt.tight_layout()
```

Optional Exercise 1
---

1. Have a play around and try to change some of the parameters in the peaked function (simply modify in the code block above). How does this affect the curve?

2. Try to make a plot with the exponential and peaked functions on the same set of axes, so you can compare them (HINT - copy the line that creates the exponential soil production function into the code block above, and  then give it a different colour).

---

---
## Optional Exercise 2
<p>Create a figure from the practical today that shows the difference between the peaked and exponential soil production functions for different initial soil production rates. You should write a figure caption that annotates what your soil production plot is showing. The caption should be a paragraph of text that describes each line, and the parameters that have been used to create them, and offers a brief explanation of how the parameters used influence rates of soil production. For an indication of the level of detail required, you could look at examples of captions to figures in journal articles, such as Figure 3 in
[Heimsath et al. (2012)](https://www.nature.com/ngeo/journal/v5/n3/pdf/ngeo1380.pdf). You can use any program you like, such as Microsoft Word, to create your figure.

**Note**: the exercises from the practicals in this module will not be marked, but they are all teaching you important skills that will be used in the summative assessment. I would therefore really encourage you to engage with them. I will go over the answers and discuss the exercises at the start of the next session.

For your independent project, you will be expected to present 5 figures with captions, so this is a good chance to practice how to write a good figure caption!

```python

```


# LU Factorisation

**ENGSCI233: Computational Techniques and Computer Systems**

*Department of Engineering Science, University of Auckland*

In the lab, you will be implementing an LU factorisation scheme. You first encountered LU factorisation in ENGSCI 111 and you will again in ENGSCI 211. This section serves as revision of the core concepts, and then extending to introduce the concept of **partial pivoting**.

***If the matrices below are not rendering properly...***

- Double click in the cell to open the markdown editor.
- Ctrl+Enter to recompile

## 1 The basics

The matrix $A$ is decomposed into **lower** and **upper triangular** matrices, $L$ and $U$, where (assuming zero indexing)

\begin{equation}
A=LU,\quad\quad L=\begin{bmatrix}
1 &0 &0 &\cdots &0 &0 \\
l_{10} &1 &0 &\cdots &0 &0 \\
l_{20} &l_{21} &1 &\cdots &0 &0 \\
\vdots &\vdots &\vdots &\ddots &\vdots  &\vdots \\
l_{n-2,0} &l_{n-2,1} &l_{n-2,2} &\cdots &1 &0 \\
l_{n-1,0} &l_{n-1,1} &l_{n-1,2} &\cdots &l_{n-1,n-2} &1\end{bmatrix},\quad\quad U=\begin{bmatrix}
u_{00} & u_{01} & u_{02} & \cdots & u_{0,n-2} & u_{0,n-1} \\
0 & u_{11} & u_{12} & \cdots & u_{1,n-2} & u_{1,n-1} \\
0 & 0 & u_{22} & \cdots & u_{2,n-2} & u_{2,n-1} \\
\vdots & \vdots & \vdots & \ddots & \vdots  & \vdots \\
0 & 0 & 0 & \cdots & u_{n-2,n-2} & u_{n-2,n-1} \\
0 & 0 & 0 & \cdots & 0 & u_{n-1,n-1}\end{bmatrix},
\end{equation}

The matrices $L$ and $U$ are constructed by applying **Gaussian elimination** to reduce $A$ to an upper triangular form and saving the **subtraction factors** as entries in the $L$ matrix.

The matrix equation $A\mathbf{x}=\mathbf{b}$ is then solved by a two step process of **backward** and **forward substitution**. Suppose we have:

\begin{equation}
A = \begin{bmatrix} \hphantom{-}2 & \hphantom{-}3 & -4 & \hphantom{-}2 \\ -4 & -5 & \hphantom{-}6 & -3 \\ \hphantom{-}2 & \hphantom{-}2 & \hphantom{-}1 & \hphantom{-}0 \\ -6 & -7 & \hphantom{-}14 & -4 \end{bmatrix},\quad\quad \mathbf{b} = \begin{bmatrix} \hphantom{-}4 \\ -8 \\ \hphantom{-}9 \\ \hphantom{-}6 \end{bmatrix}
\end{equation}

1. Performing the LU factorisation we obtain.

  \begin{equation}
  L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 \\ -2 & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ \hphantom{-}1 & -1 & \hphantom{-}1 & \hphantom{-}0 \\ -3 & \hphantom{-}2 & \hphantom{-}2 & \hphantom{-}1 \end{bmatrix},\quad\quad U = \begin{bmatrix} \hphantom{-}2 & \hphantom{-}3 & -4 & \hphantom{-}2 \\ \hphantom{-}0 & \hphantom{-}1 & -2 & \hphantom{-}1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}3 & -1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}2 \end{bmatrix}
  \end{equation}

  Then, substituting $A=LU$ into the matrix equation $A\mathbf{x}=\mathbf{b}$ and writing $U\mathbf{x}=\mathbf{y}$:

2. First, we solve the matrix equation $L\mathbf{y} = \mathbf{b}$ for the intermediate vector $\mathbf{y}$ using forward substitution.

  \begin{equation}
  \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 \\ -2 & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ \hphantom{-}1 & -1 & \hphantom{-}1 & \hphantom{-}0 \\ -3 & \hphantom{-}2 & \hphantom{-}2 & \hphantom{-}1 \end{bmatrix}\begin{bmatrix} y_0 \\ y_1 \\ y_2 \\ y_3 \end{bmatrix} = \begin{bmatrix} \hphantom{-}4 \\ -8 \\ \hphantom{-}9 \\ \hphantom{-}6 \end{bmatrix} \quad\quad\Rightarrow\quad\quad \mathbf{y} = \begin{bmatrix} 4 \\ 0 \\ 5 \\ 8 \end{bmatrix}
  \end{equation}

3. Then, we solve the matrix equation $U\mathbf{x} = \mathbf{y}$ for the solution vector $\mathbf{x}$ using backward substitution.

  \begin{equation}
  \begin{bmatrix} \hphantom{-}2 & \hphantom{-}3 & -4 & \hphantom{-}2 \\ \hphantom{-}0 & \hphantom{-}1 & -2 & \hphantom{-}1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}3 & -1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}2 \end{bmatrix}\begin{bmatrix} x_0 \\ x_1 \\ x_2 \\ x_3 \end{bmatrix} = \begin{bmatrix} 4 \\ 0 \\ 5 \\ 8 \end{bmatrix} \quad\quad\Rightarrow\quad\quad \mathbf{x} = \begin{bmatrix} 1 \\ 2 \\ 3 \\ 4 \end{bmatrix}
  \end{equation}

You will be required to code up steps 1-3 as the first part of the lab.

## 2 Partial pivoting

The potential for numerical error can be reduced by using **partial pivoting**. During Gaussian Elimination, instead of using the current row as the pivot, we swap the rows of $A$ **before** the elimination step to ensure the pivot has the largest magnitude of all coefficients in the column below it.

We need to store the pivot strategy in a vector, $\mathbf{p}$, so that we can later apply the same swaps to $\mathbf{b}$.

Applying Gaussian elimination with partial pivoting to $A$ above:

First, we swap the top and bottom rows, because $-6$ is the largest **magnitude** entry in the **zeroth** column. We record $p_0=3$ indicating we have swapped to use the third row (assuming zero indexing).

\begin{equation}
A = \begin{bmatrix} \hphantom{-}2 & \hphantom{-}3 & -4 & \hphantom{-}2 \\ -4 & -5 & \hphantom{-}6 & -3 \\ \hphantom{-}2 & \hphantom{-}2 & \hphantom{-}1 & \hphantom{-}0 \\ -6 & -7 & \hphantom{-}14 & -4 \end{bmatrix}\quad\quad\Rightarrow\quad\quad A = \begin{bmatrix}  -6 & -7 & \hphantom{-}14 & -4 \\ -4 & -5 & \hphantom{-}6 & -3 \\ \hphantom{-}2 & \hphantom{-}2 & \hphantom{-}1 & \hphantom{-}0 \\ \hphantom{-}2 & \hphantom{-}3 & -4 & \hphantom{-}2 \end{bmatrix}, \quad p_0=3
\end{equation}

apply row reduction operations and record subtraction factors

\begin{equation}
\rightarrow\quad\begin{bmatrix}  -6 & -7 & \hphantom{-}14 & -4 \\ \hphantom{-}0 & -\frac{1}{3} & -\frac{10}{3} & -\frac{1}{3} \\ \hphantom{-}0 & -\frac{1}{3} & \hphantom{-}\frac{17}{3} & -\frac{4}{3} \\ \hphantom{-}0 & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \end{bmatrix},\quad\quad L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0\\ \hphantom{-}\frac{2}{3} & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ -\frac{1}{3} & \hphantom{-}* & \hphantom{-}1 & \hphantom{-}0 \\ -\frac{1}{3} & \hphantom{-}* & \hphantom{-}* & \hphantom{-}1 \end{bmatrix}
\end{equation}

Choosing the next pivot row using values in the **first** column, we see the magnitude of entry in row 3 is the highest. Therefore, we again swap to use the third row and record $p_1=3$.

\begin{equation}
\rightarrow\quad\begin{bmatrix}  -6 & -7 & \hphantom{-}14 & -4 \\ \hphantom{-}0 & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \\ \hphantom{-}0 & -\frac{1}{3} & \hphantom{-}\frac{17}{3} & -\frac{4}{3} \\ \hphantom{-}0 & -\frac{1}{3} & -\frac{10}{3} & -\frac{1}{3} \end{bmatrix},\quad\quad L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0\\ -\frac{1}{3} & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ -\frac{1}{3} & \hphantom{-}* & \hphantom{-}1 & \hphantom{-}0 \\ \frac{2}{3} & \hphantom{-}* & \hphantom{-}* & \hphantom{-}1 \end{bmatrix},\quad p_1=3
\end{equation}

Note that, in performing this row swap, **we also swapped the entries of $L$**. Performing the next row reduction step.

\begin{equation}
\rightarrow\quad\begin{bmatrix}  -6 & -7 & \hphantom{-}14 & -4 \\ \hphantom{-}0 & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}6 & -1 \\ \hphantom{-}0 & \hphantom{-}0 & -3 & \hphantom{-}0 \end{bmatrix},\quad\quad L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0\\ -\frac{1}{3} & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ -\frac{1}{3} & -\frac{1}{2} & \hphantom{-}1 & \hphantom{-}0 \\ \hphantom{-}\frac{2}{3} & -\frac{1}{2} & \hphantom{-}* & \hphantom{-}1 \end{bmatrix},\quad p_1=3
\end{equation}

Choosing the next pivot row using values in the **second** column, we see the magnitude of entry in row 2 is the highest. There is no need for a swap in this case, and so we record $p_2=2$. Performing the row reductions

\begin{equation}
\rightarrow\quad \begin{bmatrix} -6 & -7 & \hphantom{-}14 & -4 \\ \hphantom{-}0 & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}6 & -1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 & -\frac{1}{2} \end{bmatrix},\quad\quad L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0\\ -\frac{1}{3} & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ -\frac{1}{3} & -\frac{1}{2} & \hphantom{-}1 & \hphantom{-}0 \\ \hphantom{-}\frac{2}{3} & -\frac{1}{2} & -\frac{1}{2} & \hphantom{-}1 \end{bmatrix},\quad p_2=2
\end{equation}

The final entry of $\mathbf{p}$ is trivial, we don't have a choice about the last pivot row, it must be $3$, and there are no additional row reductions to make. So finally

\begin{equation}
U = \begin{bmatrix} -6 & -7 & \hphantom{-}14 & -4 \\ \hphantom{-}0 & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}6 & -1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 & -\frac{1}{2} \end{bmatrix},\quad\quad L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0\\ -\frac{1}{3} & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ -\frac{1}{3} & -\frac{1}{2} & \hphantom{-}1 & \hphantom{-}0 \\ \hphantom{-}\frac{2}{3} & -\frac{1}{2} & -\frac{1}{2} & \hphantom{-}1 \end{bmatrix}, \quad\quad \mathbf{p} = \begin{bmatrix} 3 \\ 3 \\ 2\\ 3\end{bmatrix}
\end{equation}

Before solving the matrix equation (by forward and backward substitution), we must first apply the row swaps to $\mathbf{b}$

\begin{equation}
\mathbf{b}=\begin{bmatrix} \hphantom{-}4 \\ -8 \\ \hphantom{-}9 \\ \hphantom{-}6 \end{bmatrix} \quad\begin{matrix}\rightarrow \\ p_0=3\end{matrix}\quad \begin{bmatrix} \hphantom{-}6 \\ -8 \\ \hphantom{-}9 \\ \hphantom{-}4 \end{bmatrix} \quad\begin{matrix}\rightarrow \\ p_1=3\end{matrix}\quad \begin{bmatrix} \hphantom{-}6 \\ \hphantom{-}4 \\ \hphantom{-}9 \\ -8  \end{bmatrix}\quad\begin{matrix}\rightarrow \\ p_2=2\end{matrix}\quad\begin{bmatrix} \hphantom{-}6 \\ \hphantom{-}4 \\ \hphantom{-}9 \\ -8  \end{bmatrix} = \mathbf{b}'
\end{equation}

## 3 Matrix storage

The triangular matrices $L$ and $U$ conform to certain rules on structure. For example, $L\,$ **must have** 1's along the diagonal, and $U$ **must have** 0's in its lower triangle. We can use this information to store the matrices $L$ and $U$ more compactly:

\begin{equation}
L = \begin{bmatrix} \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0\\ -\frac{1}{3} & \hphantom{-}1 & \hphantom{-}0 & \hphantom{-}0 \\ -\frac{1}{3} & -\frac{1}{2} & \hphantom{-}1 & \hphantom{-}0 \\ \hphantom{-}\frac{2}{3} & -\frac{1}{2} & -\frac{1}{2} & \hphantom{-}1 \end{bmatrix}\quad\text{and}\quad U = \begin{bmatrix} -6 & -7 & \hphantom{-}14 & -4 \\ \hphantom{-}0 & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}6 & -1 \\ \hphantom{-}0 & \hphantom{-}0 & \hphantom{-}0 & -\frac{1}{2} \end{bmatrix} \quad\Rightarrow\quad LU = \begin{bmatrix} -6 & -7 & \hphantom{-}14 & -4 \\ -\frac{1}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} & \hphantom{-}\frac{2}{3} \\ -\frac{1}{3} & -\frac{1}{2} & \hphantom{-}6 & -1 \\ \hphantom{-}\frac{2}{3} & -\frac{1}{2} & -\frac{1}{2} & -\frac{1}{2} \end{bmatrix}
\end{equation}

You will be expected to implement this storage system in the lab.


```python
import networkx as nx
import matplotlib.pyplot as plt
import numpy as np
```

```python
V = np.array([65,75])
Vc = np.cumsum(V)
max_B = 10000 # maximum budget
no_runs = 100 # no. of runs to average
K = 20 # controls number of restarts
T = Vc[-1] # Total nodes - 1
num_edges = 20
```

```python
G = nx.Graph()

total = 0
for i in range(len(V)):
    H = nx.gnp_random_graph(V[i],0.85)
    H = nx.convert_node_labels_to_integers(H, first_label=total, ordering='default', label_attribute=None)
    G = nx.compose(H,G)
    total+= V[i]
```

```python
pairs = [(np.random.randint(0,V[0]),np.random.randint(V[0],T)) for i in list(range(num_edges))]
G.add_edges_from(pairs)
G_no_edges=G.number_of_edges()
G_no_nodes=G.number_of_nodes()
print(G_no_nodes,G_no_edges)
```

    140 4148

```python
print(pairs)
```

    [(47, 80), (20, 120), (52, 110), (44, 73), (57, 107), (27, 87), (40, 129), (31, 137), (13, 85), (7, 79), (0, 91), (35, 128), (53, 106), (30, 74), (19, 116), (3, 109), (0, 115), (8, 111), (50, 72), (52, 121)]

```python
options = {
    "node_color": "black",
    "node_size": 50,
    "linewidths": 0,
    "width": 0.1,
}
'''
nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)
nx.draw_networkx_edges(G, pos=pos, alpha=0.2)
for x in range(len(pos)):
    pos[x] = pos[x]+np.array([0.02,0])
labels = dict([(n, n) for n in G.nodes()]) #add labels
_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color='#000000', font_size=15) #draw labels
'''

plt.figure(1,figsize=(10,10))
pos = nx.spring_layout(G)  # Seed for reproducible layout
nx.draw(G, pos, **options)
for x in range(len(pos)):
    pos[x] = pos[x]+np.array([0.02,0])
labels = dict([(n, n) for n in G.nodes()]) #add labels
_ = nx.draw_networkx_labels(G, pos, labels=labels, font_color='#000000', font_size=15) #draw labels
plt.show()
```

```python
from louvain import detect_communities, modularity
def draw_communities(G, node_size=90, alpha=1, k=None, randomized=False):
    partition = detect_communities(G, randomized=randomized)
    print("Modularity for best partition:", modularity(G, partition))
    community_map = {}
    for community, nodes in enumerate(partition):
        for node in nodes:
            community_map[node] = community

    cmap = plt.get_cmap("jet")
    plt.figure(figsize=(15, 15))
    pos = nx.spring_layout(G, k=k)
    indexed = [community_map.get(node) for node in G]
    plt.axis("off")
    nx.draw_networkx_nodes(G, pos=pos, cmap=cmap, node_color=indexed, node_size=node_size, alpha=alpha)
    nx.draw_networkx_edges(G, pos=pos, alpha=0.2)
    for x in range(len(pos)):
        pos[x] = pos[x]+np.array([0.02,0])
    labels = dict([(n, n) for n in G.nodes()]) #add labels
    _ = nx.draw_networkx_labels(G, pos, labels=labels, font_color='#000000', font_size=15) #draw labels
draw_communities(G)
```

```python
partition = detect_communities(G, randomized=False)
community_map = {}
node_map = {}
```

```python
for community, nodes in enumerate(partition):
    for node in nodes:
        node_map[node] = community
    community_map[community] = nodes
```

```python
def node_fn(node):
    return int(node_map[node]==1)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.5357142857142857

```python
def MH_sampling(G,B):
    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        neighbors = list(nx.neighbors(G,sample))
        sample_t = np.random.choice(neighbors)
        if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
            sample = sample_t
        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)

def RDS_sampling(G,B):
    est_RW = []
    est_RW_t1 = 0
    est_RW_t2 = 0
    sample = np.random.choice(G.nodes())
    deg_pr_sent = G.degree(sample)
    est_RW_t1 += node_fn(sample)/deg_pr_sent
    est_RW_t2 += 1/deg_pr_sent
    est_RW.append(est_RW_t1/est_RW_t2)
    for ii in range(2,B+1):
        neighbors = list(nx.neighbors(G,sample))
        sample = np.random.choice(neighbors)
        deg_pr_sent = G.degree(sample)
        est_RW_t1 += node_fn(sample)/deg_pr_sent
        est_RW_t2 += 1/deg_pr_sent
        est_RW.append(est_RW_t1/est_RW_t2)
    return np.array(est_RW)
```

```python
def RDSRR_sampling(G,B,U=None):
    #M = np.power(B,2/3)
    #restart_ind = np.power(np.arange(1,M),1.5)
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(K*np.log(t)))
        t+=1

    est_RW = []
    est_RW_t1 = 0
    est_RW_t2 = 0
    sample = np.random.choice(G.nodes())
    deg_pr_sent = G.degree(sample)
    est_RW_t1 += node_fn(sample)/deg_pr_sent
    est_RW_t2 += 1/deg_pr_sent
    est_RW.append(est_RW_t1/est_RW_t2)
    for ii in range(2,B+1):
        if ii in restart_ind:
            #sample = np.random.choice(community_map[np.random.randint(len(community_map))])
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample = np.random.choice(neighbors)

        deg_pr_sent = G.degree(sample)
        est_RW_t1 += node_fn(sample)/deg_pr_sent
        est_RW_t2 += 1/deg_pr_sent
        est_RW.append(est_RW_t1/est_RW_t2)
    return np.array(est_RW)

def MHRR_sampling(G,B,U=None):
    #M = np.power(B,2/3)
    #restart_ind = np.power(np.arange(1,M),1.5)
    restart_ind = [10]
    t=20
    while restart_ind[-1]<B:
        restart_ind.append(restart_ind[-1]+int(K*np.log(t)))
        t+=1

    est_MH= []
    est_MH_t = 0
    sample = np.random.choice(G.nodes())
    est_MH_t += node_fn(sample)
    est_MH.append(est_MH_t)
    for ii in range(2,B+1):
        if ii in restart_ind:
            #sample_t = np.random.choice(community_map[np.random.randint(len(community_map))])
            if U is None:
                sample = np.random.choice(community_map[np.random.choice([x for x in list(community_map.keys()) if x!=node_map[sample]])])
            else:
                z = [node_map[x] for x in U].index(node_map[sample])
                sample = np.random.choice(U[:z]+U[z+1:])
        else:
            neighbors = list(nx.neighbors(G,sample))
            sample_t = np.random.choice(neighbors)
            if np.random.rand() <= (G.degree(sample)/G.degree(sample_t)):
                sample = sample_t

        est_MH_t += node_fn(sample)
        est_MH.append(est_MH_t/ii)
    return np.array(est_MH)
```

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(3,figsize=(20,16))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5)
plt.grid()
```

```python
def node_fn(node):
    return int(G.degree(node)>70)
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.007142857142857143

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(4,figsize=(20,16))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5)
plt.ylim(0,1)
plt.grid()
```

```python
def node_fn(node):
    return int((G.degree(node)%2==0))
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.4714285714285714

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(5,figsize=(20,16))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5)
plt.grid()
```

```python
import sympy

def node_fn(node):
    return int(sympy.isprime(G.degree(node)))
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.15714285714285714

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(6,figsize=(20,16))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5)
plt.ylim(0,1)
plt.grid()
```

```python
def node_fn(node):
    return int(sympy.isprime(node))
F_org = sum([node_fn(i) for i in G.nodes()])/G_no_nodes
print(F_org)
```

    0.24285714285714285

```python
MSE_MH_t = 0
for ii in range(1,no_runs+1):
    MSE_MH_t += (MH_sampling(G,max_B)-F_org)**2
MSE_MH = MSE_MH_t/(no_runs)
MSE_MH = np.sqrt(MSE_MH)/F_org

MSE_rds_t = 0
for ii in range(1,no_runs+1):
    MSE_rds_t += (RDS_sampling(G,max_B)-F_org)**2
MSE_rds = MSE_rds_t/(no_runs)
MSE_rds = np.sqrt(MSE_rds)/F_org

MSE_rdsrr_t = 0
for ii in range(1,no_runs+1):
    MSE_rdsrr_t += (RDSRR_sampling(G,max_B)-F_org)**2
MSE_rdsrr = MSE_rdsrr_t/(no_runs)
MSE_rdsrr = np.sqrt(MSE_rdsrr)/F_org

MSE_mhrr_t = 0
for ii in range(1,no_runs+1):
    MSE_mhrr_t += (MHRR_sampling(G,max_B)-F_org)**2
MSE_mhrr = MSE_mhrr_t/(no_runs)
MSE_mhrr = np.sqrt(MSE_mhrr)/F_org

plt.figure(7,figsize=(20,16))
plt.plot(np.array(list(range(len(MSE_MH)))),MSE_MH,color='red',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_mhrr)))),MSE_mhrr,color='blue',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rds)))),MSE_rds,color='black',linewidth=1.5)
plt.plot(np.array(list(range(len(MSE_rdsrr)))),MSE_rdsrr,color='purple',linewidth=1.5)
plt.ylim(0,1)
plt.grid()
```

```python

```


###  ¿ Cómo se mueve un péndulo ?

> Se dice que un sistema cualquiera, mecánico, eléctrico, neumático, etc., es un oscilador armónico si, cuando se deja en libertad fuera de su posición de equilibrio, vuelve hacia ella describiendo oscilaciones sinusoidales, o sinusoidales amortiguadas en torno a dicha posición estable.
- https://es.wikipedia.org/wiki/Oscilador_armónico

Referencias:
   - http://matplotlib.org
   - https://seaborn.pydata.org
   - http://www.numpy.org
   - http://ipywidgets.readthedocs.io/en/latest/index.html

**En realidad esto es el estudio de oscilaciones. **
___

Los sistemas mas sencillos a estudiar en oscilaciones son el sistema ` masa-resorte` y el `péndulo simple`.

\begin{align}
\ddot{x} +  \omega_{0}^2 x &= 0, \quad \omega_{0} = \sqrt{\frac{k}{m}}\notag\\
\ddot{\theta} + \omega_{0}^{2}\, \theta &= 0, \quad\mbox{donde}\quad \omega_{0}^2 = \frac{g}{l}
\end{align}

### Sistema `masa-resorte`

La solución a este sistema `masa-resorte`se explica en términos de la segunda ley de Newton. Para este caso, si la masa permanece constante y solo consideramos la dirección en $x$. Entonces,

\begin{equation}
F = m\ddot{x} = m \frac{d\dot{x}}{dt} = m \frac{d^2x}{dt^2}
\end{equation}

¿ Cuál es la fuerza? ** Ley de Hooke! **
\begin{equation}
F = -k x, \quad k > 0
\end{equation}
Vemos que la fuerza se opone al desplazamiento y su intensidad es proporcional al mismo.  Y $k$ es la constante elástica o recuperadora del resorte.

Cuya solución se escribe como
\begin{equation}
x(t) = A \cos(\omega_{o} t) + B \sin(\omega_{o} t)
\end{equation}
Y su primera derivada (velocidad) sería
\begin{equation}
\dot{x}(t) = \omega_{0}[- A \sin(\omega_{0} t) + B\cos(\omega_{0}t)]
\end{equation}

- **¿Cómo se ven las gráficas de `x vs t`, tanto para la posición y velocidad?**

_Esta instrucción es para que las gráficas aparezcan dentro de este entorno._

```python
%matplotlib inline
```

_Esta es la librería con todas las instrucciones para realizar gráficos. _

```python
import matplotlib.pyplot as plt
```

```python
import matplotlib as mpl
label_size = 14
mpl.rcParams['xtick.labelsize'] = label_size
mpl.rcParams['ytick.labelsize'] = label_size
```

 _Y esta es la librería con todas las funciones matemáticas necesarias._

```python
import numpy as np
```

```python
x = np.arange(0,10,20)
x
```

    array([0])

```python
t = np.linspace(0, 50, 100)
plt.figure(figsize = (7, 4))
plt.plot(t, .5*np.cos(.5 *t) + .1 * np.sin(.5 *t), '-', lw = 1, ms = 4,
         label = '$x(t)$')
plt.plot(t, .5*.1*np.cos(.5*t) - .5*.5 * np.sin(.5*t), 'ro-', lw = 1, ms = 4,
         label = r'$\dot{x(t)}$')
plt.xlabel('$t$', fontsize = 20)
plt.show()
```

```python
plt.figure(figsize = (7, 4))
plt.scatter(t, .5*np.cos(.5 *t) + .1 * np.sin(.5 *t), lw = 0, c = 'red',
            label = '$x(t)$')
plt.plot(t, .5*np.cos(.5 *t) + .1 * np.sin(.5 *t), 'r-', lw = 1)
plt.scatter(t, .5*.1*np.cos(.5*t) - .5*.5 * np.sin(.5*t), lw = 0, c = 'b',
            label = r'$\dot{x(t)}$')
plt.plot(t, .5*.1*np.cos(.5*t) - .5*.5 * np.sin(.5*t), 'b-', lw = 1)
plt.xlabel('$t$', fontsize = 20)
plt.legend(loc = 'best')
plt.show()
```

Y si consideramos un conjunto de frecuencias de oscilación, entonces

```python
frecuencias = np.array([.1, .2 , .5, .6])
plt.figure(figsize = (7, 4))
for f in frecuencias:
    plt.plot(t, .5 * np.cos(f * t) + .1 * np.sin(f * t), '*-')
plt.xlabel('$t$', fontsize = 16)
plt.ylabel('$x(t)$', fontsize = 16)
plt.title('Oscilaciones', fontsize = 16)
plt.show()
```

Estos colores, son el default de `matplotlib`, sin embargo existe otra librería dedicada, entre otras cosas, a la presentación de gráficos.

```python
import seaborn as sns
sns.set(style='ticks', palette='Set2')
```

```python
frecuencias = np.array([.1, .2 , .5, .6])
plt.figure(figsize = (7, 4))
for f in frecuencias:
    plt.plot(t, .5 * np.cos(f * t) + .1 * np.sin(f * t), 'o-',
             label = '$\omega_0 = %s$'%f)
plt.xlabel('$t$', fontsize = 16)
plt.ylabel('$x(t)$', fontsize = 16)
plt.title('Oscilaciones', fontsize = 16)
plt.legend(loc='center left', bbox_to_anchor=(1.05, 0.5), prop={'size': 14})
plt.show()
```

Si queremos tener manipular un poco mas las cosas, hacemos uso de lo siguiente:

```python
from ipywidgets import *
```

```python
def masa_resorte(t = 0):
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.plot(.5 * np.cos(.5 * t) + .1 * np.sin(.5 * t), [0],  'ko', ms = 10)
    ax.set_xlim(xmin = -0.6, xmax = .6)
    ax.axvline(x=0, color = 'r')
    ax.axhline(y=0, color = 'grey', lw = 1)
    fig.canvas.draw()

interact(masa_resorte, t = (0, 50,.01));
```

    A Jupyter Widget

La opción de arriba generalmente será lenta, así que lo recomendable es usar `interact_manual`.

```python
def masa_resorte(t = 0):
    fig = plt.figure()
    ax = fig.add_subplot(1, 1, 1)
    ax.plot(.5 * np.cos(.5 * t) + .1 * np.sin(.5 * t), [0],  'ko', ms = 10)
    ax.set_xlim(xmin = -0.6, xmax = .6)
    ax.axvline(x=0, color = 'r')
    ax.axhline(y=0, color = 'grey', lw = 1)
    fig.canvas.draw()
interact_manual(masa_resorte, t = (0, 50,.01));
```

    A Jupyter Widget

### Péndulo simple

Ahora, si fijamos nuestra atención al movimiento de un péndulo simple _(oscilaciones pequeñas)_. La ecuación diferencial a resolver tiene la misma forma. La diferencia más evidente es como hemos definido a $\omega_{0}$. Esto quiere decir que,

\begin{equation}
\theta(t) = A\cos(\omega_{0} t) + B\sin(\omega_{0}t)
\end{equation}

Si graficamos la ecuación de arriba vamos a encontrar un comportamiento muy similar al ya discutido anteriormente. Es por ello que ahora veremos el movimiento en el plano $xy$. Es decir,

\begin{align}
x &= l \sin(\theta), \quad
y = l \cos(\theta)
\end{align}

```python
def theta_t(a, b, g, l, t):
    omega_0 = np.sqrt(g/l)
    return a * np.cos(omega_0 * t) + b * np.sin(omega_0 * t)
```

```python
def pendulo_simple(t = 0):
    fig = plt.figure(figsize = (5,5))
    ax = fig.add_subplot(1, 1, 1)
    x = 2 * np.sin(theta_t(.4, .6, 9.8, 2, t))
    y =  - 2 * np.cos(theta_t(.4, .6, 9.8, 2, t))
    ax.plot(x, y, 'ko', ms = 10)
    ax.plot([0], [0], 'rD')
    ax.plot([0, x ], [0, y], 'k-', lw = 1)
    ax.set_xlim(xmin = -2.2, xmax = 2.2)
    ax.set_ylim(ymin = -2.2, ymax = .2)
    fig.canvas.draw()
interact_manual(pendulo_simple, t = (0, 10,.01));
```

    A Jupyter Widget

### Condiciones iniciales

Realmente lo que se tiene que resolver es,

\begin{equation}
\theta(t) = \theta(0) \cos(\omega_{0} t) + \frac{\dot{\theta}(0)}{\omega_{0}} \sin(\omega_{0} t)
\end{equation}

> **Actividad.** Modificar el programa anterior para incorporar las condiciones iniciales.

```python
# Solución:
def theta_t():

    return
```

```python
def pendulo_simple(t = 0):
    fig = plt.figure(figsize = (5,5))
    ax = fig.add_subplot(1, 1, 1)
    x = 2 * np.sin(theta_t( , t))
    y =  - 2 * np.cos(theta_t(, t))
    ax.plot(x, y, 'ko', ms = 10)
    ax.plot([0], [0], 'rD')
    ax.plot([0, x ], [0, y], 'k-', lw = 1)
    ax.set_xlim(xmin = -2.2, xmax = 2.2)
    ax.set_ylim(ymin = -2.2, ymax = .2)
    fig.canvas.draw()
interact_manual(pendulo_simple, t = (0, 10,.01));
```

### Espacio fase  $(q, p)$

Recuerden el momento lineal se define como
$
 p = m v,
$
donde $m$ es la masa y $v$ la velocidad. Y la posición y velocidad para el sistema `masa-resorte` se escriben como:

\begin{align}
x(t) &= x(0) \cos(\omega_{o} t) + \frac{\dot{x}(0)}{\omega_{0}} \sin(\omega_{o} t)\\
\dot{x}(t) &= -\omega_{0}x(0) \sin(\omega_{0} t) + \dot{x}(0)\cos(\omega_{0}t)]
\end{align}

```python
k = 3  #constante elástica [N]/[m]
m = 1 # [kg]
omega_0 = np.sqrt(k/m)
x_0 = .5
x_0_dot = .1
```

```python
t = np.linspace(0, 50, 300)
```

```python
x_t = x_0 *np.cos(omega_0 *t) + (x_0_dot/omega_0) * np.sin(omega_0 *t)
x_t_dot = -omega_0 * x_0 * np.sin(omega_0 * t) + x_0_dot * np.cos(omega_0 * t)
```

```python
plt.figure(figsize = (7, 4))
plt.plot(t, x_t, label = '$x(t)$', lw = 1)
plt.plot(t, x_t_dot, label = '$\dot{x}(t)$', lw = 1)
plt.legend(loc='center left', bbox_to_anchor=(1.01, 0.5), prop={'size': 14})
plt.xlabel('$t$', fontsize = 18)
plt.show()
```

```python
plt.figure(figsize = (5, 5))
plt.plot(x_t, x_t_dot/omega_0, 'ro', ms = 2)
plt.xlabel('$x(t)$', fontsize = 18)
plt.ylabel('$\dot{x}(t)/\omega_0$', fontsize = 18)
plt.show()
```

```python
plt.figure(figsize = (5, 5))
plt.scatter(x_t,  x_t_dot/omega_0, cmap = 'viridis', c = x_t_dot, s = 8, lw = 0)
plt.xlabel('$x(t)$', fontsize = 18)
plt.ylabel('$\dot{x}(t)/\omega_0$', fontsize = 18)
plt.show()
```

#### Multiples condiciones iniciales

```python
k = 3  #constante elástica [N]/[m]
m = 1 # [kg]
omega_0 = np.sqrt(k/m)
```

```python
t = np.linspace(0, 50, 50)
```

```python
x_0s = np.array([.7, .5, .25, .1])
x_0s_dot = np.array([.2, .1, .05, .01])
cmaps = np.array(['viridis', 'inferno', 'magma', 'plasma'])
```

```python
plt.figure(figsize = (6, 6))
for indx, x_0 in enumerate(x_0s):
    x_t = x_0 *np.cos(omega_0 *t) + (x_0s_dot[indx]/omega_0) * np.sin(omega_0 *t)
    x_t_dot = -omega_0 * x_0 * np.sin(omega_0 * t) + x_0s_dot[indx] * np.cos(omega_0 * t)
    plt.scatter(x_t,  x_t_dot/omega_0, cmap = cmaps[indx],
                c = x_t_dot, s = 10,
                lw = 0)
    plt.xlabel('$x(t)$', fontsize = 18)
    plt.ylabel('$\dot{x}(t)/\omega_0$', fontsize = 18)
    #plt.legend(loc='center left', bbox_to_anchor=(1.05, 0.5))
```

Trayectorias del oscilador armónico simple en el espacio fase $(x,\, \dot{x}\,/\omega_0)$ para diferentes valores de la energía.

Created with Jupyter by Lázaro Alonso.

```python
!dir
```

     El volumen de la unidad C es Windows
     El número de serie del volumen es: B683-E7AA

     Directorio de C:\Users\alvar\1clase simulacion\Simulacion2018\Modulo1

    23/01/2018  12:19    <DIR>          .
    23/01/2018  12:19    <DIR>          ..
    23/01/2018  11:56    <DIR>          .ipynb_checkpoints
    23/01/2018  11:56    <DIR>          circular_maps
    23/01/2018  11:29             7.316 Clase0_GuiaSimulacionM.ipynb
    23/01/2018  11:29            10.851 Clase1_IntroSim.ipynb
    23/01/2018  11:29            22.773 Clase2_GitTutorial.ipynb
    23/01/2018  11:29             2.901 Clase3_GitHubTutorial.ipynb
    23/01/2018  12:19           426.207 Clase4_OsciladorArmonico.ipynb
    23/01/2018  11:29           316.852 Clase5_OsciladorAmortiguado.ipynb
    23/01/2018  11:29         2.507.209 Clase6_MapaLogistico.ipynb
    23/01/2018  11:29            80.843 Clase7_ModeloAhorro.ipynb
    23/01/2018  11:29           498.908 Clase8_MembranaCircular.ipynb
    23/01/2018  11:29               354 filename.py
    23/01/2018  11:29            37.039 logohithub.jpg
    23/01/2018  11:29            63.443 poblacion.jpg
    23/01/2018  11:29             1.464 welcome.py
                  13 archivos      3.976.160 bytes
                   4 dirs  851.959.349.248 bytes libres