```python
%matplotlib inline

import matplotlib.pyplot as plt
import numpy as np
from scipy import stats
```

# Monte Carlo integration

## Simple Monte Carlo integration

The basic idea of Monte Carlo integration is very simple and only requires elementary statistics. Suppose we want to find the value of
$$
I = \int_a^b f(x) dx
$$
in some region with volume $V$. Monte Carlo integration estimates this integral by estimating the fraction of random points that fall below $f(x)$ multiplied by $V$.

In a statistical context, we use Monte Carlo integration to estimate the expectation
$$
E[g(X)] = \int_X g(x) p(x) dx
$$

with

$$
\bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$
where $x_i \sim p$ is a draw from the density $p$.

We can estimate the Monte Carlo variance of the approximation as
$$
v_n = \frac{1}{n^2} \sum_{o=1}^n (g(x_i) - \bar{g_n})^2)
$$

Also, from the Central Limit Theorem,

$$
\frac{\bar{g_n} - E[g(X)]}{\sqrt{v_n}} \sim \mathcal{N}(0, 1)
$$

The convergence of Monte Carlo integration is $\mathcal{0}(n^{1/2})$ and independent of the dimensionality. Hence Monte Carlo integration generally beats numerical integration for moderate- and high-dimensional integration since numerical integration (quadrature) converges as $\mathcal{0}(n^{d})$. Even for low dimensional problems, Monte Carlo integration may have an advantage when the volume to be integrated is concentrated in a very small region and we can use information from the distribution to draw samples more often in the region of importance.

An elementary, readable description of Monte Carlo integration and variance reduction techniques can be found [here](https://www.cs.dartmouth.edu/~wjarosz/publications/dissertation/appendixA.pdf).

## Intuition behind Monte Carlo integration

We want to find some integral

$$I = \int{f(x)} \, dx$$

Consider the expectation of a function $g(x)$ with respect to some distribution $p(x)$. By definition, we have

$$
E[g(x)] = \int{g(x) \, p(x) \, dx}
$$

If we choose $g(x) = f(x)/p(x)$, then we have

$$
\begin{align}
E[g(x)] &= \int{\frac{f(x}{p(x)} \, p(x) \, dx} \\
&= \int{f(x) dx} \\
&= I
\end{align}
$$

By the law of large numbers, the average converges on the expectation, so we have

$$
I \approx \bar{g_n} = \frac{1}{n} \sum_{i=1}^n g(x_i)
$$

If $f(x)$ is a proper integral (i.e. bounded), and $p(x)$ is the uniform distribution, then $g(x) = f(x)$ and this is known as ordinary Monte Carlo. If the integral of $f(x)$ is improper, then we need to use another distribution with the same support as $f(x)$.

**Example: Estimating $\pi$**

We have a function

$$
f(x, y) =
\begin{cases}
1 & \text{if}\ x^2 + y^2 \le 1 \\
0 & \text{otherwise}
\end{cases}
$$

whose integral is
$$
I = \int_{-1}^{1} \int_{-1}^{1} f(x,y) dx dy = \pi
$$

So a Monte Carlo estimate of $\pi$ is
$$
Q = 4 \sum_{i=1}^{N} f(x, y)
$$

if we sample $p$ from the standard uniform distribution in $\mathbb{R}^2$.

```python
x = np.linspace(-3,3,100)
dist = stats.norm(0,1)
a = -2
b = 0
plt.plot(x, dist.pdf(x))
plt.fill_between(np.linspace(a,b,100), dist.pdf(np.linspace(a,b,100)), alpha=0.5)
plt.text(b+0.1, 0.1, 'p=%.4f' % (dist.cdf(b) - dist.cdf(a)), fontsize=14)
pass
```

#### Using quadrature

```python
from scipy.integrate import quad
```

```python
y, err = quad(dist.pdf, a, b)
y
```

    0.47724986805182085

#### Simple Monte Carlo integration

If we can sample directly from the target distribution $N(0,1)$

```python
n = 1000000
x = dist.rvs(n)
np.sum((a < x) & (x < b))/n
```

    0.477918

If we cannot sample directly from the target distribution $N(0,1)$ but can evaluate it at any point.

Recall that $g(x) = \frac{f(x)}{p(x)}$. Since $p(x)$ is $U(a, b)$, $p(x) = \frac{1}{b-a}$. So we want to calculate

$$
\frac{1}{n} \sum_{i=1}^n (b-a) f(x)
$$

```python
n = 10000
x = np.random.uniform(a, b, n)
np.mean((b-a)*dist.pdf(x))
```

    0.48069595176138846

## Intuition for error rate

We will just work this out for a proper integral $f(x)$ defined in the unit cube and bounded by $|f(x)| \le 1$. Draw a random uniform vector $x$ in the unit cube. Then

$$
\begin{align}
E[f(x_i)] &= \int{f(x) p(x) dx} = I \\
\text{Var}[f(x_i)] &= \int{(f(x_i) - I )^2 p(x) \, dx} \\
&= \int{f(x)^2 \, p(x) \, dx} - 2I \int(f(x) \, p(x) \, dx + I^2 \int{p(x) \, dx} \\
&= \int{f(x)^2 \, p(x) \, dx}  + I^2 \\
& \le \int{f(x)^2 \, p(x) \, dx} \\
& \le \int{p(x) \, dx} = 1
\end{align}
$$

Now consider summing over many such IID draws $S_n = f(x_1) + f(x_2) + \cdots + f(x_n)$. We have

$$
\begin{align}
E[S_n] &= nI \\
\text{Var}[S_n] & \le n
\end{align}
$$

and as expected, we see that $I \approx S_n/n$. From Chebyshev's inequality,

$$
\begin{align}
P \left( \left| \frac{s_n}{n} - I \right| \ge \epsilon \right)  &=
P \left( \left| s_n - nI \right| \ge n \epsilon \right) & \le \frac{\text{Var}[s_n]}{n^2 \epsilon^2} & \le
\frac{1}{n \epsilon^2} = \delta
\end{align}
$$

Suppose we want 1% accuracy and 99% confidence - i.e. set $\epsilon = \delta = 0.01$. The above inequality tells us that we can achieve this with just $n = 1/(\delta \epsilon^2) = 1,000,000$ samples, regardless of the data dimensionality.

### Example

We want to estimate the following integral $\int_0^1 e^x dx$.

```python
x = np.linspace(0, 1, 100)
plt.plot(x, np.exp(x))
plt.xlim([0,1])
plt.ylim([0, np.exp(1)])
pass
```

#### Analytic solution

```python
from sympy import symbols, integrate, exp

x = symbols('x')
expr = integrate(exp(x), (x,0,1))
expr.evalf()
```

    1.71828182845905

#### Using quadrature

```python
from scipy import integrate

y, err = integrate.quad(exp, 0, 1)
y
```

    1.7182818284590453

#### Monte Carlo integration

```python
for n in 10**np.array([1,2,3,4,5,6,7,8]):
    x = np.random.uniform(0, 1, n)
    sol = np.mean(np.exp(x))
    print('%10d %.6f' % (n, sol))
```

            10 1.847075
           100 1.845910
          1000 1.731000
         10000 1.727204
        100000 1.719337
       1000000 1.718142
      10000000 1.718240
     100000000 1.718388

### Monitoring variance in Monte Carlo integration

We are often interested in knowing how many iterations it takes for Monte Carlo integration to "converge". To do this, we would like some estimate of the variance, and it is useful to inspect such plots. One simple way to get confidence intervals for the plot of Monte Carlo estimate against number of iterations is simply to do many such simulations.

For the example, we will try to estimate the function (again)

$$
f(x) = x \cos 71 x + \sin 13x, \ \  0 \le x \le 1
$$

```python
def f(x):
    return x * np.cos(71*x) + np.sin(13*x)
```

```python
x = np.linspace(0, 1, 100)
plt.plot(x, f(x))
pass
```

#### Single MC integration estimate

```python
n = 100
x = f(np.random.random(n))
y = 1.0/n * np.sum(x)
y
```

    0.03550964669105388

#### Using multiple independent sequences to monitor convergence

We vary the sample size from 1 to 100 and calculate the value of $y = \sum{x}/n$ for 1000 replicates. We then plot the 2.5th and 97.5th percentile of the 1000 values of $y$ to see how the variation in $y$ changes with sample size. The blue lines indicate the 2.5th and 97.5th percentiles, and the red line a sample path.

```python
n = 100
reps = 1000

x = f(np.random.random((n, reps)))
y = 1/np.arange(1, n+1)[:, None] * np.cumsum(x, axis=0)
upper, lower = np.percentile(y, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1), y, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), y[:, 0], c='red', linewidth=1);
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

#### Using bootstrap to monitor convergence

If it is too expensive to do 1000 replicates, we can use a bootstrap instead.

```python
xb = np.random.choice(x[:,0], (n, reps), replace=True)
yb = 1/np.arange(1, n+1)[:, None] * np.cumsum(xb, axis=0)
upper, lower = np.percentile(yb, [2.5, 97.5], axis=1)
```

```python
plt.plot(np.arange(1, n+1)[:, None], yb, c='grey', alpha=0.02)
plt.plot(np.arange(1, n+1), yb[:, 0], c='red', linewidth=1)
plt.plot(np.arange(1, n+1), upper, 'b', np.arange(1, n+1), lower, 'b')
pass
```

## Variance Reduction

With independent samples, the variance of the Monte Carlo estimate is

$$
\begin{align}
\text{Var}[\bar{g_n}] &= \text{Var} \left[ \frac{1}{N}\sum_{i=1}^{N} \frac{f(x_i)}{p(x_i)} \right] \\
&= \frac{1}{N^2} \sum_{i=1}^{N}  \text{Var} \left[ \frac{f(x_i)}{p(x_i)} \right] \\
&= \frac{1}{N^2} \sum_{i=1}^{N} \text{Var}[Y_i] \\
&= \frac{1}{N} \text{Var}[Y_i]
\end{align}
$$

where $Y_i = f(x_i)/p(x_i)$. In general, we want to make $\text{Var}[\bar{g_n}]$ as small as possible for the same number of samples. There are several variance reduction techniques (also colorfully known as Monte Carlo swindles) that have been described - we illustrate the change of variables and importance sampling techniques here.

### Change of variables

The Cauchy distribution is given by
$$
f(x) = \frac{1}{\pi (1 + x^2)}, \ \ -\infty \lt x \lt \infty
$$

Suppose we want to integrate the tail probability $P(X > 3)$ using Monte Carlo. One way to do this is to draw many samples form a Cauchy distribution, and count how many of them are greater than 3, but this is extremely inefficient.

#### Only 10% of samples will be used

```python
import scipy.stats as stats

h_true = 1 - stats.cauchy().cdf(3)
h_true
```

    0.10241638234956674

```python
n = 100

x = stats.cauchy().rvs(n)
h_mc = 1.0/n * np.sum(x > 3)
h_mc, np.abs(h_mc - h_true)/h_true
```

    (0.13, 0.26932817794994063)

#### A change of variables lets us use 100% of draws

We are trying to estimate the quantity

$$
\int_3^\infty \frac{1}{\pi (1 + x^2)} dx
$$

Using the substitution $y = 3/x$ (and a little algebra), we get

$$
\int_0^1 \frac{3}{\pi(9 + y^2)} dy
$$

Hence, a much more efficient MC estimator is

$$
\frac{1}{n} \sum_{i=1}^n \frac{3}{\pi(9 + y_i^2)}
$$

where $y_i \sim \mathcal{U}(0, 1)$.

```python
y = stats.uniform().rvs(n)
h_cv = 1.0/n * np.sum(3.0/(np.pi * (9 + y**2)))
h_cv, np.abs(h_cv - h_true)/h_true
```

    (0.10219440906830025, 0.002167361082027339)

### Importance sampling

Suppose we want to evaluate

$$
I = \int{h(x)\,p(x) \, dx}
$$

where $h(x)$ is some function and $p(x)$ is the PDF of $y$. If it is hard to sample directly from $p$, we can introduce a new density function $q(x)$ that is easy to sample from, and write

$$
I = \int{h(x)\, p(x)\, dx} = \int{h(x)\, \frac{p(x)}{q(x)} \, q(x) \, dx}
$$

In other words, we sample from $h(y)$ where $y \sim q$ and weight it by the likelihood ratio $\frac{p(y)}{q(y)}$, estimating the integral as

$$
\frac{1}{n}\sum_{i=1}^n \frac{p(y_i)}{q(y_i)} h(y_i)
$$

Sometimes, even if we can sample from $p$ directly, it is more efficient to use another distribution.

#### Example

Suppose we want to estimate the tail probability of $\mathcal{N}(0, 1)$ for $P(X > 5)$. Regular MC integration using samples from $\mathcal{N}(0, 1)$ is hopeless since nearly all samples will be rejected. However, we can use the exponential density truncated at 5 as the importance function and use importance sampling. Note that $h$ here is simply the identify function.

```python
x = np.linspace(4, 10, 100)
plt.plot(x, stats.expon(5).pdf(x))
plt.plot(x, stats.norm().pdf(x))
pass
```

#### Expected answer

We expect about 3 draws out of 10,000,000 from $\mathcal{N}(0, 1)$ to have a value greater than 5. Hence simply sampling from $\mathcal{N}(0, 1)$ is hopelessly inefficient for Monte Carlo integration.

```python
%precision 10
```

    '%.10f'

```python
v_true = 1 - stats.norm().cdf(5)
v_true
```

    2.866515719235352e-07

#### Using direct Monte Carlo integration

```python
n = 10000
y = stats.norm().rvs(n)
v_mc = 1.0/n * np.sum(y > 5)
# estimate and relative error
v_mc, np.abs(v_mc - v_true)/v_true
```

    (0.0, 1.0)

#### Using importance sampling

```python
n = 10000
y = stats.expon(loc=5).rvs(n)
v_is = 1.0/n * np.sum(stats.norm().pdf(y)/stats.expon(loc=5).pdf(y))
# estimate and relative error
v_is, np.abs(v_is- v_true)/v_true
```

    (2.8290057563382236e-07, 0.01308555981236137)

```python

```


Un sistema de orden $4$ se puede escribir en su forma controlador de la siguiente manera:

$$
\frac{d}{dt} x_c =
\begin{pmatrix}
0 & 1 & 0 & 0 \\
0 & 0 & 1 & 0 \\
0 & 0 & 0 & 1 \\
-a_4 & -a_3 & -a_2 & -a_1
\end{pmatrix} x_c +
\begin{pmatrix}
0 \\
0 \\
0 \\
1
\end{pmatrix} u
$$

```
from sympy import init_printing
init_printing()
```

```
from sympy import Matrix, var, eye, collect, expand
```

```
var("s, a1, a2, a3, a4, p1, p2, p3, p4")
```

```
Ac = Matrix([[0, 1, 0, 0], [0, 0, 1, 0], [0, 0, 0, 1], [-a4, -a3, -a2, -a1]])
bc = Matrix([[0],[0],[0],[1]])
```

```
Ac
```

```
(s*eye(4) - Ac).det()
```

Este es el polinomio caracteristico del sistema:

$$
\det{(sI - A_c)} = s^4 + a_1 s^3 + a_2 s^2 + a_3 s + a_4
$$

Tambien podemos definir una retroalimentación de estado:

$$
u = f_c x_c + v
$$

en donde

$$
f_c =
\begin{pmatrix}
a_4 - \bar{a}_4 & a_3 - \bar{a}_3 & a_2 - \bar{a}_2 & a_1 - \bar{a}_1
\end{pmatrix}
$$

La ecuación de estado del sistema retroalimentada quedaría:

$$
\frac{d}{dt} x_c = A_{f_c} x_c + b_c v
$$

donde $A_{f_c} = A_c + b_c f_c$

```
fc = Matrix([[a4-p4, a3-p3, a2-p2, a1-p1]])
```

```
Afc = Ac + bc*fc
```

```
Afc
```

y su polinomio caracteristico es:

```
(s*eye(4) - Afc).det()
```

La matriz de controlabilidad del par $(A_c, b_c)$ es:

```
CAcbc = ((bc.row_join(Ac*bc)).row_join(Ac*Ac*bc)).row_join(Ac*Ac*Ac*bc)
```

```
CAcbc
```

y su determinante es:

```
CAcbc.det()
```

Si ahora analizamos la matriz de controlabilidad del sistema retroalimentado:

```
CAfcbc = ((bc.row_join(Afc*bc)).row_join(Afc*Afc*bc)).row_join(Afc*Afc*Afc*bc)
```

```
CAfcbc
```

Pero en general, podemos demostrar que la matriz de un sistema retroalimentado nos es mas que:

$$
C_{(A_f, b)} =
\begin{pmatrix}
b & A_f b & \dots & A_f^{n-1} b
\end{pmatrix} =
\begin{pmatrix}
b & \left( A + b f^T \right) b & \dots & \left( A + b f^T \right)^{n-1} b
\end{pmatrix}
$$

en donde los terminos van siendo:

```
var("A, b, f")
```

```
b
```

```
expand((A + b*f)*b)
```

```
expand((A + b*f)**2*b)
```

```
expand((A + b*f)**3*b)
```

Lo que en general, podemos ver como:

$$
C_{(A_f, b)} =
\begin{pmatrix}
b & A b & \dots & A^{n-1} b
\end{pmatrix} X
$$

donde $X$ tiene que ser de la forma:

$$
X =
\begin{pmatrix}
1 & b f & b^2 f^2 & b^3 f^3 \\
0 & 1 & b f & b^2 f^2\\
0 & 0 & 1 & b f \\
0 & 0 & 0 & 1
\end{pmatrix}
$$

```
X = Matrix([[1, b*f, b**2*f**2 , b**3*f**3], [0, 1, b*f, b**2*f**2], [0, 0, 1, b*f], [0, 0, 0, 1]])
```

```
X
```

```
CAb = Matrix([[b, A*b, A**2*b, A**3*b]])
```

```
CAb
```

```
CAb*X
```

Y como habiamos visto anteriormente, el determinante de esta matriz, no cambiará:

```
CAfcbc.det()
```

IPythonNotebooks/Teoria de Control I/Forma controlador.ipynb


```python
%matplotlib notebook
import numpy as np
import matplotlib.pyplot as pl
import matplotlib as mpl
```

# Simulating Reaction-Diffusion Systems

In this notebook we want to show step by step how to simulate a reaction-diffusion system in Python. Here, we decided for the [Gray-Scott model](http://www.karlsims.com/rd.html) because of the variety of shapes it can produce.

In this models, we have two chemicals $A$ and $B$ which are distributed across a grid of size $N$. The symbol $A_{ij}$ represents the concentration of chemical $A$ at grid coordinates $(i,j)$ (similar for $B$).

The discrete reaction-diffusion update equations are

$$
\begin{align}
A_{ij}(t+1) &= A_{ij}(t) + \Big[D_A (\nabla^2 A)_{ij} - A_{ij}B_{ij}^2 + f(1-A_{ij}) \Big]\times\Delta t\\
B_{ij}(t+1) &= B_{ij}(t) + \Big[D_B (\nabla^2 B)_{ij} + A_{ij}B_{ij}^2 - (k+f)B_{ij} \Big]\times\Delta t
\end{align}
$$

The single terms are explained in the tutorial linked above.

First, we need to take care of the discretized Laplacian terms.

## Discrete Laplacian
As explained on [Wikipedia](https://en.wikipedia.org/wiki/Discrete_Laplace_operator#Implementation_via_operator_discretization), the discretized Laplacian of a grid cell $(i,j)$ can be computed by summing over neighboring cells and subtract the value of the original cell with the total weight. One possible implementation is to only recognize direct neighbors of grid difference $\Delta=1$, i.e. at $(i,j-1)$, $(i,j+1)$, $(i-1,j)$, and $(i+1,j)$.

The whole update formula is

$$
(\nabla^2 A)_{ij} = A_{i,j-1} + A_{i,j+1} + A_{i-1,j} + A_{i+1,j} - 4A_{ij}
$$

How can we do this efficiently in `numpy`? Let's first define a small concentration matrix `A`.

```python
A = np.ones((3,3))
A[1,1] = 0
A
```

    array([[1., 1., 1.],
           [1., 0., 1.],
           [1., 1., 1.]])

Now for each cell, we want to add its right neighbor. We can easily access this value in a matrix sense by doing a `numpy.roll`, which shifts all elements in a certain direction, with periodic boundary conditions.

```python
right_neighbor = np.roll(A, # the matrix to permute
                         (0,-1), # we want the right neighbor, so we shift the whole matrix -1 in the x-direction)
                         (0,1), # apply this in directions (y,x)
                        )
right_neighbor
```

    array([[1., 1., 1.],
           [0., 1., 1.],
           [1., 1., 1.]])

So to compute the discrete Laplacian of a matrix $M$, one could use the following function.

```python
def discrete_laplacian(M):
    """Get the discrete Laplacian of matrix M"""
    L = -4*M
    L += np.roll(M, (0,-1), (0,1)) # right neighbor
    L += np.roll(M, (0,+1), (0,1)) # left neighbor
    L += np.roll(M, (-1,0), (0,1)) # top neighbor
    L += np.roll(M, (+1,0), (0,1)) # bottom neighbor

    return L
```

Let's test this with our example matrix

```python
discrete_laplacian(A)
```

    array([[ 0., -1.,  0.],
           [-1.,  4., -1.],
           [ 0., -1.,  0.]])

Seems like it worked! Note that periodic boundary conditions were used, too.

## Implement update formula

Computing the Laplacian was the hardest part. The other parts are simple. Just take the concentration matrices and apply the update formula.

```python
def gray_scott_update(A, B, DA, DB, f, k, delta_t):
    """
    Updates a concentration configuration according to a Gray-Scott model
    with diffusion coefficients DA and DB, as well as feed rate f and
    kill rate k.
    """

    # Let's get the discrete Laplacians first
    LA = discrete_laplacian(A)
    LB = discrete_laplacian(B)

    # Now apply the update formula
    diff_A = (DA*LA - A*B**2 + f*(1-A)) * delta_t
    diff_B = (DB*LB + A*B**2 - (k+f)*B) * delta_t

    A += diff_A
    B += diff_B

    return A, B
```

## Choosing initial conditions

The initial conditions are very important in the Gray-Scott model. If you just randomize the initial conditions it might happen that everything just dies out. It seems to be a good idea to assume a homogeneous distribution of chemicals with a small disturbance which can then produce some patterns. We can also add a bit of noise. We can do the same decisions as Rajesh Singh in his [somewhat more elaborate version](https://rajeshrinet.github.io/blog/2016/gray-scott/) and disturb with a square in the center of the grid. Let's do the following.

```python
def get_initial_configuration(N, random_influence=0.2):
    """
    Initialize a concentration configuration. N is the side length
    of the (N x N)-sized grid.
    `random_influence` describes how much noise is added.
    """

    # We start with a configuration where on every grid cell
    # there's a lot of chemical A, so the concentration is high
    A = (1-random_influence) * np.ones((N,N)) + random_influence * np.random.random((N,N))

    # Let's assume there's only a bit of B everywhere
    B = random_influence * np.random.random((N,N))

    # Now let's add a disturbance in the center
    N2 = N//2
    radius = r = int(N/10.0)

    A[N2-r:N2+r, N2-r:N2+r] = 0.50
    B[N2-r:N2+r, N2-r:N2+r] = 0.25

    return A, B
```

Let's also add a function which makes nice drawings and then draw some initial configurations.

```python
def draw(A,B):
    """draw the concentrations"""
    fig, ax = pl.subplots(1,2,figsize=(5.65,4))
    ax[0].imshow(A, cmap='Greys')
    ax[1].imshow(B, cmap='Greys')
    ax[0].set_title('A')
    ax[1].set_title('B')
    ax[0].axis('off')
    ax[1].axis('off')
```

```python
A, B = get_initial_configuration(200)
draw(A,B)
```

Now we can simulate! We should first decide for some parameter choices. Let's stick with  [Rajesh's choices](https://rajeshrinet.github.io/blog/2016/gray-scott/).

```python
# update in time
delta_t = 1.0

# Diffusion coefficients
DA = 0.16
DB = 0.08

# define feed/kill rates
f = 0.060
k = 0.062

# grid size
N = 200

# simulation steps
N_simulation_steps = 10000
```

And for the simulation we simply update the concentrations for `N_simulation_steps` time steps.

```python
A, B = get_initial_configuration(200)

for t in range(N_simulation_steps):
    A, B = gray_scott_update(A, B, DA, DB, f, k, delta_t)

draw(A,B)
```

Isn't this nice? You might also want to try the following values (directly taken from Rajesh's version above):

```python
DA, DB, f, k = 0.14, 0.06, 0.035, 0.065 # bacteria
A, B = get_initial_configuration(200)

for t in range(N_simulation_steps):
    A, B = gray_scott_update(A, B, DA, DB, f, k, delta_t)

draw(A,B)
```


# Rotation Curve of The Solar System

We learned in Module1/BonusChallenge1 about Kepler's laws of planetary motion, and how you can use the known periods and distances of satellites to calculate the mass of the body they are orbiting using Kepler's third law:

\begin{equation}
P^2 = \frac{4 \pi^2}{G \, M} a^3
\end{equation}

Where $M$ is the mass of the central object. Another way to write that equation is in terms of the $speed$ a satellite is traveling around the object they are orbiting:

\begin{equation}
v = \sqrt{\frac{G \, M}{R}}
\end{equation}

Where $G$ is the gravitational constant, $M$ is the mass of the central object, and $R$ is the distance to the satellite. (__Bonus:__ Can you demonstrate this?)

Because velocities are relatively easy to calculate using the $Doppler$ $shift$ of light, it is often convenient to use the latter equation. Let's make a "rotation curve" for the Solar System.

First, let's calculate the expected rotation curve for our Solar System using the equations above. Let's start be defining a function that calculates the expected velocity of a planet in the Solar System at a given radius:

```python
#FILL IN CODE
def velocity(#FILL IN):
    #FILL IN CODE
```

Now create an array of radii that you want to calculate the velocity at. Then, calculate the expected velocities at all of those radii:

```python
# FILL IN CODE
R = # FILL IN CODE

# USE THIS ARRAY AND YOUR VELOCITY FUNCTION TO CALCULATE VELOCITY:

```

Next, let's make a plot of $v$ vs. $R$ for the Solar System. Make sure the axes are labelled properly! Also be sure to check your units throughout.

```python

```

Now that we've done that, let's see how the expected values for the velocities of the planets in our Solar System compare with the actual values. Collect both $v$ and $R$ into lists or arrays below:

```python
#FILL IN CODE
v_planets = # FILL IN CODE
R_planets = # FILL IN CODE
```

Add those values to the plot you created above:

```python

```

How do the actual values compare with the measured values? Hopefully they match well! What do you notice about how the velocity changes with radius?

```python

```


# Astroinformatics "Machine Learning Basics"
## Class 3:
In this tutorial, we'll see basics concepts of machine learning. (We will not see classification yet, but these concepts applies to those problems too). All this concepts are very well explained in the [Deep Learning Book, Chapter 5](http://www.deeplearningbook.org/contents/ml.html)

First a brief discussion about the frequentist and bayesians approach of machine learning. Then a basic problem of linear regression in order to give some insight about the approach of frequentist and bayesians of this problem and its connection. Then we'll see the concept of capacity, overfitting and underfitting and how to solve it using regularization (explained from a frequentist and bayesian point of view). Finally, we use cross validation to select the hyperparameters of our optimization problem.

# Frequentiest and Bayesians

Please read the discussion in [this great book](http://nbviewer.jupyter.org/github/CamDavidsonPilon/Probabilistic-Programming-and-Bayesian-Methods-for-Hackers/blob/master/Chapter1_Introduction/Ch1_Introduction_PyMC3.ipynb), It'll give you a great insight about the difference between the two approaches. I also recommend  [this reading](https://ocw.mit.edu/courses/mathematics/18-05-introduction-to-probability-and-statistics-spring-2014/readings/MIT18_05S14_Reading20.pdf) from an MIT class

```python
import numpy as np
import matplotlib.pyplot as plt
from numpy.linalg import inv
from mpl_toolkits.mplot3d import axes3d
from matplotlib import cm
```

# Linear regression least square

Given some data $(x_{i}, y_{i})_{i=1}^{N}$, we want to find the best affine transformation of $x$ that better predicts $y$, the affine model is $\hat{y} = w^{T}x$ where we add a 1 to $x$ in order to have an offset of the linear model (affine transformation). Let's define $(X,Y)$ as the dataset where each row of $X$ is $x_{i}^{T}$ and $Y$ has $y_{i}$ as components. Now, we define the mean squared error ($MSE$) as our measure of performance of the model:

$$ MSE = \frac{1}{N}\| \hat{Y}-Y \|^{2}_{2} $$

where $\hat{Y}$ is the estimated values of the linear model. In order to find the best model in the least square sense, we can minimize MSE by gradient:

\begin{equation}
\nabla_{w} MSE = 0 \\
\nabla_{w} \frac{1}{N} \| \hat{Y} - Y \|^{2}_{2} = 0 \\
\frac{1}{N} \nabla_{w} \| Xw - Y \|^{2}_{2} = 0 \\
\nabla_{w} (Xw-Y)^{T}(Xw-Y) = 0 \\
\nabla_{w} (w^{T}X^{T}Xw-2w^{T}X^{T}Y-Y^{T}Y) = 0 \\
2X^{T}Xw-2X^{T}Y=0 \\
w = (X^{T}X)^{-1}X^{T}Y
\end{equation}

We know that this is the solution for the linear regression because the $MSE$ is a convex function of $w$m we can check this by taking the second derivative, $\nabla_{w}^{2}MSE = 2X^{T}X$ which is a positive semi definite matrix (the eigenvalues are bigger or equal to zero). In this case, we need all the eigenvalues bigger than zero, if there is at least one eigenvalue equal to zero, that means that the value of the MSE is "flat" in the direction of that eigenvector, when the eigenvalue is "close" to zero, it can produce numerical instability on the optimization (this can be fixed with regularization). Now let's make an example:

```python
def linear_function(w, x):
    return np.dot(x, w)

w = np.array([0.7, 0.3])[...,np.newaxis]
print(w.shape)
noise = 0.10
n_points = 20
np.random.seed(500)

# we add an extra dimension to make it a column vector
x_samples = np.linspace(3, 5, n_points)[..., np.newaxis]
# then we add a column of ones in order to have the constant term a*x + b*1 = y
augmented_x = np.concatenate([x_samples, np.ones(shape=(n_points,1))], axis=1)
print("samples shape: "+str(augmented_x.shape))
# adding gaussian noise to the data
y_samples = linear_function(w, augmented_x) + np.random.normal(loc=0.0, scale=noise, size=(n_points,1))
print("target shape: "+str(y_samples.shape))
fig, ax = plt.subplots(figsize=(12,7))
ax.plot(x_samples, linear_function(w, augmented_x), label="Real solution")
ax.scatter(x_samples, y_samples, label="Samples", s=70)
ax.legend(fontsize=14)
ax.set_xlabel("x", fontsize=14)
ax.set_ylabel("y", fontsize=14)
plt.show()
```

```python
# Least square solution
estimated_w = inv(augmented_x.T @ augmented_x) @ augmented_x.T @ y_samples
# MSE
error = np.linalg.norm(y_samples - linear_function(estimated_w, augmented_x))**2/len(y_samples)
# eigenvectors and eigenvalues of the covariance matrix
eg_values, eg_vectors = np.linalg.eig(augmented_x.T @ augmented_x)
print("estimated w:" +str(estimated_w))
print("mean squared error: "+str(error))
print("eigenvalues: "+str(eg_values))
print("eigenvectos: "+str(eg_vectors))
```

    estimated w:[[ 0.68565289]
     [ 0.33056476]]
    mean squared error: 0.0101285040086
    eigenvalues: [ 346.94365923    0.42476182]
    eigenvectos: [[ 0.97134386 -0.23767859]
     [ 0.23767859  0.97134386]]

```python
# making error maningfold
X_array = np.arange(-1, 2.5, 0.05)
Y_array = np.arange(-1, 2.5, 0.05)
X, Y = np.meshgrid(X_array, Y_array)
Z = np.zeros(shape=(len(X_array), len(Y_array)))

for i, x in enumerate(X_array):
    for j, y in enumerate(Y_array):
        w_loop = np.array([x, y])[..., np.newaxis]
        Z[i, j] = np.linalg.norm(y_samples - linear_function(w_loop, augmented_x))**2/len(y_samples)
```

```python
fig, (ax, ax2) = plt.subplots(1, 2, figsize=(15,7))
ax.plot(x_samples, linear_function(w, augmented_x), label="Real solution")
ax.scatter(x_samples, y_samples, label="Samples", s=70)
ax.plot(x_samples, linear_function(estimated_w, augmented_x), label="Estimated solution")
ax.legend(fontsize=14)
ax.set_xlabel("x", fontsize=14)
ax.set_ylabel("y", fontsize=14)

levels = np.linspace(0, np.amax(Z), 100)
cont = ax2.contourf(X, Y, Z, levels = levels)#,cmap="inferno")
soa = np.concatenate([np.roll(np.repeat(estimated_w.T, 2, axis=0), shift=1), eg_vectors], axis=1)*1.0
X2, Y2, U, V = zip(*soa)
ax2.quiver(X2, Y2, U, V, angles='xy', scale_units='xy', scale=1,
           color="y", label="eigen vectors of covariance matrix")
ax2.legend(fontsize=14)
ax2.set_xlabel("MSE for each w", fontsize=14)
plt.show()
```

```python
fig = plt.figure(figsize=(12, 7))
ax2 = fig.add_subplot(1, 1, 1, projection='3d')
surf = ax2.plot_surface(X, Y, Z, rstride=1, cstride=1, cmap=cm.coolwarm,
                       linewidth=0, antialiased=False)
ax2.set_ylabel("w[1]", fontsize=14)
ax2.set_xlabel("x[0]", fontsize=14)
plt.show()
```

As we can see here, there is one direction where the MSE is relatively flat (small eigenvalue), that is why we see little variation in that direction by changing w, but it is still convex enough to have a unique solution. This is a frequentist way to solve a linear regression, is just a data driven solution with the only assumption of the model shape and that the MSE is a good way to measure performance, let's see now what is the bayesian way, which involves some distribution assumptions expressed using probabilities.

# Linear Regression Maximum Likelihood Estimation
Now, we define our model in a probabilistic way, that is, for example:

$$ \hat{y} = w^{T}x + \epsilon \hspace{0.5cm} \text{with} \hspace{0.5cm} \epsilon \sim \mathcal{N}(0, \sigma) $$

This means that we are assuming that the data has gaussian noise of a given variance $\sigma^{2}$. Now, we can write what is the probability of occurrence of the data Y, for a given input X and a model:

$$ P(Y \mid X, w, \sigma) $$

We call this "Likelihood". Now, it is very intuitive that if a particular value of w produces a high probability of seeing the data (high likelihood value), then w is a good choice for our model, so we define the maximum likelihood estimation of w as:

$$ \hat{w} = \text{argmax}_{w}  P(Y \mid X, w, \sigma) = \text{argmax}_{w} \prod_{i}^{N}P(y_{i}, x_{i},w,\sigma) = \text{argmax}_{w} \prod_{i}^{N}\mathcal{N}(y_{i}; wx_{i}, \sigma) $$

where the last equality comes from the assumption of i.i.d samples. Now, this product over many probabilities can produce numerical underflow, so we can obtain a more convenient optimization problem. Instead of maximizing the likelihood, we minimize the "negative log likelihood" which is $NLL = -\log(P(Y \mid X, w, \sigma))$, so the optimization problem is:

$$ \hat{w} = \text{argmin} -\log(P(Y \mid X, w, \sigma)) $$

This expression to minimize arise naturally when we start by the optimization by minimizing the Kullback Leibler divergence between the estimated distribution of the data and the distribution produced by the model (we'll skip this for now). Something interesting happens when we work a little bit the expression of the NLL

\begin{equation}
\begin{split}
NLL = & -\log(\prod_{i}^{N}\mathcal{N}(y_{i}; wx_{i}, \sigma)) \\
= & -\sum_{i=1}^{N}\log(\mathcal{N}(y_{i}; wx_{i}, \sigma))
= & -\sum_{i=1}^{N}\left [ \log \frac{1}{\sqrt{2\pi}\sigma} - \frac{(y_{i}-wx_{i})^{2}}{2 \sigma^{2}} \right ] \propto  \| \hat{Y} - Y \|^{2}_{2}
\end{split}
\end{equation}

Least Mean Squared Error is equivalent to the Maximum Likelihood Estimation when gaussian noise in the data is assumed!!

# Capacity, overfitting and Underfitting

Generally, in machine learning, the purpose of fitting a model to a dataset, is to evaluate the model on new data, not just the one that we use to train the model. The ability to perform well on new data is called generalization.

In order to measure the generalization capacity of a trained model, we subtract a subset of the original dataset which is not used to train the model but to test it. Those sets are called train set (used to find the parameters) and test set (used to measure the generalization performance), then we can estimate the generalization ability of our model by measuring the error (or another metric of performance) on the test set. Here we are assuming that both sets, train and test sets are generated by the same data-generating process, that means i.i.d assumptions over every sample (as we did before in the maximum likelihood case)

The objective of the optimization process is to make the training error small enough and make the gap between the training and test error small. The following concepts are useful to understand this process:

- Capacity: ability of the model to fit a wide variety of functions (like relations between inputs and outputs)
- Overfitting: Occurs when the gap between the training error and test error is too large, when the Capacity of the model is too high compared with the real complexity of the data-generating process
- Underfitting: Happens when the model is not able to obtain a sufficiently low error value on the training set, may occur because the Capacity of the model is not enough to encode the complexity of the data-generating process

Let's understand this with an example using a polinomial fitting over data. In the following example, the true complexity of the data is a quadratic polynomial. We fit models with low and high capacities compare with the real complexity

```python
w = np.array([-2, 0.6, 0.7])[...,np.newaxis]
noise = 1.2
n_points = 20
train_size = 10
test_size = n_points - train_size

np.random.seed(0)

x_samples = np.linspace(-2, 2, n_points)[..., np.newaxis]
# Making quadratic polinomial
augmented_x = np.concatenate([x_samples**2, x_samples, x_samples**0], axis=1)
y_samples = linear_function(w, augmented_x) + np.random.normal(loc=0.0, scale=noise, size=(n_points,1))
x_plot = np.linspace(-2,2,100)[..., np.newaxis]
aug_x_plot = np.concatenate([x_plot**2, x_plot, x_plot**0], axis=1)

# Dividing in train and test set
indexes = np.arange(start=0, stop=n_points,step=1)
np.random.shuffle(indexes)
train_index = indexes[:train_size]
test_index = indexes[train_size:]
x_train = x_samples[train_index, ...]
aug_x_train = augmented_x[train_index, ...]
y_train = y_samples[train_index, ...]
x_test = x_samples[test_index, ...]
aug_x_test = augmented_x[test_index, ...]
y_test = y_samples[test_index, ...]

fig, ax = plt.subplots(figsize=(12,7))
ax.plot(x_plot, linear_function(w, aug_x_plot), label="Real solution")
ax.scatter(x_train, y_train, label="train samples", s=70)
ax.scatter(x_test, y_test, label="test samples", s=70)
ax.legend(fontsize=14)
ax.set_xlabel("x", fontsize=14)
ax.set_ylabel("y", fontsize=14)
plt.show()
```

```python
# Linear, Quadratic 5 and 10 degree polynomial fit.
linear_coef = np.polyfit(x_train[:, 0], y_train[:, 0], deg=1, full=True)
qd_coef = np.polyfit(x_train[:, 0], y_train[:, 0], deg=2,full=True)
deg5_coef = np.polyfit(x_train[:, 0], y_train[:, 0], deg=5, full=True)
deg10_coef = np.polyfit(x_train[:, 0], y_train[:, 0], deg=10, full=True)
p1 = np.poly1d(linear_coef[0])
p2 = np.poly1d(qd_coef[0])
p3 = np.poly1d(deg5_coef[0])
p4 = np.poly1d(deg10_coef[0])
error1 = np.linalg.norm(y_test[:, 0] - p1(x_test[:, 0]))**2/len(y_test)
error2 = np.linalg.norm(y_test[:, 0] - p2(x_test[:, 0]))**2/len(y_test)
error3 = np.linalg.norm(y_test[:, 0] - p3(x_test[:, 0]))**2/len(y_test)
error4 = np.linalg.norm(y_test[:, 0] - p4(x_test[:, 0]))**2/len(y_test)
```

```python
print("Generalization errors")
print("linear: "+str(error1))
print("quadratic: "+str(error2))
print("deg5: "+str(error3))
print("deg10: "+str(error4))
fig, ax = plt.subplots(figsize=(12,7))
ax.plot(x_plot, linear_function(w, aug_x_plot), label="Real solution", lw=3)
ax.scatter(x_train, y_train, label="train samples", s=70)
ax.scatter(x_test, y_test, label="test samples", s=70)
ax.plot(x_plot, p1(x_plot),'--' ,label="linear (Underfitting)")
ax.plot(x_plot, p2(x_plot),'--' , label="quadratic (Appropiate Capacity)")
ax.plot(x_plot, p3(x_plot),'--' , label="5 deg (Not too overfitted)" )
ax.plot(x_plot, p4(x_plot),'--' , label="10 deg (Overfitting)")
ax.legend(fontsize=14)
ax.set_xlabel("x", fontsize=14)
ax.set_ylabel("y", fontsize=14)
ax.set_ylim([-15, 5])
plt.show()
```

In the last plot, we can see that the linear model does not have enough capacity to express the relation between x and y. Quadratic and deg 5 polynomial are good enough to find the relation. Some times, a high capacity model with a large family of functions that can represent (this is representational capacity), does not find the best solution during the optimization process, these additional limitations reduce the capacity of the actual solution, this is called the effective capacity. In the case of deg 10 polynomial, the capacity of the model is too high, so fits perfectly the train data but has a poor generalization ability.

The capacity of the model can be modified by modifying the model or changing the effective capacity by adding restrictions to the loss function. This is called Regularization.

# Regularization
## Regularized least squares
In this example, we'll use the weight decay regularization, which tends to choose parameters on the solution space that are close to the origin (small euclidean norm). We just need to modify the loss function by adding one term:

\begin{equation}
\begin{split}
 J(w) = & MSE + R \\
 = & \frac{1}{N} \| \hat{Y} - Y \|^{2}_{2} + \lambda \| w \|^{2}_{2}
\end{split}
\end{equation}

Let's see how the solution looks by taking the gradient of J with respect to w

\begin{equation}
\nabla_{w} J(w) = 0 \\
2X^{T}Xw-2X^{T}Y + 2\lambda w=0 \\
w = (X^{T}X + \lambda I)^{-1}X^{T}Y
\end{equation}

As we can see in the last expression, now we take the inverse of the correlation matrix plus the identity ponderated by $\lambda$. This means that we are adding convexity to the problem because the eigenvalues of this new matrix are going to be bigger if we increase $\lambda$ (we are making the matrix less singular), the convexity of this new manifold as a combination of a parabola because of the regularization term and the convexity of the original problem without the regularization. The optimization will tend to choose small norm w if we increase $\lambda$

Now, let's see an example by fitting a high capacity model but with this regularization term in the loss function

```python
# Same model as before
w = np.array([-2, 0.6, 0.7])[...,np.newaxis]
noise = 1.2
n_points = 20
train_size = 10
test_size = n_points - train_size
np.random.seed(0)

x_samples = np.linspace(-2, 2, n_points)[..., np.newaxis]
augmented_x = np.concatenate([x_samples**2, x_samples, x_samples**0], axis=1)
y_samples = linear_function(w, augmented_x) + np.random.normal(loc=0.0, scale=noise, size=(n_points,1))
x_plot = np.linspace(-2,2,100)[..., np.newaxis]
aug_x_plot = np.concatenate([x_plot**2, x_plot, x_plot**0], axis=1)

indexes = np.arange(start=0, stop=n_points,step=1)
np.random.shuffle(indexes)
train_index = indexes[:train_size]
test_index = indexes[train_size:]
x_train = x_samples[train_index, ...]
aug_x_train = augmented_x[train_index, ...]
y_train = y_samples[train_index, ...]
x_test = x_samples[test_index, ...]
aug_x_test = augmented_x[test_index, ...]
y_test = y_samples[test_index, ...]

# Now we do it for high capacity model
deg = 10
x_deg = []
for i in range(deg+1):
    x_deg.append(x_samples**(deg-i))
x_deg = np.concatenate(x_deg, axis=1)
x_deg_train = x_deg[train_index, ...]
x_deg_test = x_deg[test_index, ...]
```

```python
# Least square solution
reg_values = [10**7, 0.5, 0]
labels = ["Too large lambda (Underfitting)", "appropiate lambda", "no regularization (Overfitting)"]
reg_w = []
solution = []
for i, lam in enumerate(reg_values):
    # we save the regularized solution for each lambda
    reg_w.append(inv(x_deg_train.T @ x_deg_train + lam*np.identity(deg+1)) @ x_deg_train.T @ y_train)
    solution.append(np.poly1d(reg_w[-1][:,0]))
```

```python
fig, ax_array = plt.subplots(1,3,figsize=(15,5))
for i, lam in enumerate(reg_values):
    ax_array[i].plot(x_plot, linear_function(w, aug_x_plot), label="Real solution", lw=3)
    ax_array[i].scatter(x_train, y_train, label="train samples", s=70)
    ax_array[i].scatter(x_test, y_test, label="test samples", s=70)
    p = solution[i]
    ax_array[i].plot(x_plot, p(x_plot), label="Estimated solution")
    ax_array[i].set_ylim([-10, 5])
    ax_array[i].set_title(labels[i], fontsize=14)
    ax_array[i].legend()

plt.show()
```

The last plot shows how the regularization term modifies the effective capacity of the model. For very high $\lambda$ (left plot), the solutions are reduced to a very small region of the original space, so the capacity of the model is reduced too much and produce underfitting on the data. For very small $\lambda$ (right plot), there is no penalization for the size of the weights and the model is able to look for solutions using its original capacity, so the model overfits. For a medium $\lambda$ (middle plot), the effective capacity is probably close to the necessary one to find the correct function of the data.

## Probabilistic Perspective of Regularization, Maximum a Posteriori

In this case, we will add information about the distribution of the parameters p(w) as a prior knowledge, by using Bayes' theorem to modify the likelihood in the following way:

\begin{equation}
\begin{split}
P(\theta \mid D) = & \frac{P(D \mid \theta) P(\theta)}{P(D)} \\
\propto & P(D \mid w) P(w)
\end{split}
\end{equation}

Where $D$ is the data and $\theta$ the parameters of the model. $P(\theta)$ is called "prior" since it is prior knowledge added to the model about how the parameters distribute, in some application, the designer of the model might have some idea where to look for the parameters for a particular problem. $P(D \mid \theta)$ is the likelihood as we already know. $P(\theta \mid D)$ is called "posterior" probability, it is the distribution of the parameters for a given data, basically  an update of our prior after seeing evidence of the real process (samples from the data-generating process)

Let's consider our previous model and assume a gaussian distribution for the prior of the parameters

$$ \hat{y} = w^{T}x + \epsilon \hspace{0.5cm} \text{with} \hspace{0.5cm} \epsilon \sim \mathcal{N}(0, \sigma) \hspace{0.5cm} \text{with} \hspace{0.5cm} p(w) \sim \mathcal{N(0, \tau)} $$

Then, the posterior probability of the parameters is proportional to:
\begin{equation}
\begin{split}
P(w \mid Y,X,\sigma , \tau) \propto P(Y \mid X, w, \sigma) P(w \mid \tau)
\end{split}
\end{equation}

If we find $w$ where the posterior probability is maximized, it means that for the given dataset and the prior knowledge, there is a high probability that the model with that value of $w$ is the one that produce the data. So the solution to the maximum a posteriori (MAP) is:

\begin{equation}
\begin{split}
\hat{w} = & \text{argmax}_{w}  P(w \mid Y, X, \sigma, \tau) \\
= & \text{argmin}_{w} -\log(P(w \mid Y, X, \sigma, \tau))
\end{split}
\end{equation}

Something interesting happens (again) when we work a little bit this expression

\begin{equation}
\begin{split}
-\log(P(w \mid Y, X, \sigma, \tau)) = & -\sum_{i=1}^{N}\log \mathcal{N}(y_{i}, wx_{i}, \sigma)-\log \mathcal{N}(w; 0, \tau) \\
= & n \log \sqrt{2 \pi} \sigma + \sum_{i=1}^{N} \left ( \frac{(y_{i}-wx_{i})^{2}}{2\sigma^{2}} \right ) + n\log \sqrt{2 \pi \tau} + \sum_{i=1}^{N}\left ( \frac{w^{2}}{2 \tau^{2}} \right ) \\
\propto & \| \hat{Y} - Y \|^{2}_{2} + \frac{N \sigma^{2}}{2 \tau^{2}} \| w \|^{2}_{2} \hspace{0.2cm} \text{check this math please}
\end{split}
\end{equation}

Regularized least mean squared is the same as maximum a posteriori with gaussian noise and gaussian prior! The amount of regularization, in this case, is controlled by the width of the gaussian prior $\tau$

# Hyperparameters and Cross-validation

Many models and regularization factors are subject to hyperparameters that must be chosen by the designer. With hyperparameters I mean, for example, the lambda for regularization, the degree of the polynomial, number of layer and neurons in a neural network,  gamma coefficient for support vector machines, etc.

We should choose the hyperparameters that produce a better model to generalize over new data (test set). I good way to do this is used cross-validation, which is a procedure to have a better estimation of the generalization performance. Some times we do not have too many examples, so the random choice for the test set could be very sensitive to the realization, and of course, the generalization performance estimation too, cross-validation try to fix this problem by doing the following:

#### K-fold cross-validation

For a given dataset $D$, performance metric F and number of subsets k, we do:
- Split D into k mutually exclusive subsets $D_{i}$ with $\bigcup_{i=1}^{K} D_{i} = D$
- For i from 1 to k:
    - train the model with $D\backslash D_{i}$
    - Compute performance F over $D_{i}$
- end for
- Return performance

```python
def cross_validation(lam, x_subsets, y_subsets):
    train_error = []
    test_error = []
    for i, x_test in enumerate(x_subsets):
        x_train = np.concatenate([x for j, x in enumerate(x_subsets) if j!=i], axis=0)
        y_train = np.concatenate([y for j, y in enumerate(y_subsets) if j!=i], axis=0)
        y_test = y_subsets[i]
        w = inv(x_train.T @ x_train + lam*np.identity(x_train.shape[1])) @ x_train.T @ y_train
        p = np.poly1d(w[:,0])
        test_error.append(np.linalg.norm(y_test[:, 0] - p(x_test[:, -2]))**2/len(y_test))
        train_error.append(np.linalg.norm(y_train[:, 0] - p(x_train[:, -2]))**2/len(y_train))
    return np.array(train_error), np.array(test_error)

def kfold_cv(x_data, y_data, lam_array, kfold=4):
    x_subsets = np.split(x_data, kfold)
    y_subsets = np.split(y_data, kfold)

    train_error_mean = []
    test_error_mean = []
    train_error_std = []
    test_error_std = []
    for j, lam in enumerate(lam_array):
        print('\r{}'.format(float(j/len(lam_array))*100), end='')
        train_error, test_error = cross_validation(lam, x_subsets, y_subsets)
        train_error_mean.append(np.mean(train_error))
        train_error_std.append(np.std(train_error))
        test_error_mean.append(np.mean(test_error))
        test_error_std.append(np.std(test_error))

    return [np.array(train_error_mean),
           np.array(train_error_std),
           np.array(test_error_mean),
           np.array(test_error_std)]
```

```python
lam_array = np.linspace(0.01, 10**8, 10000)
one_over_lambda = 1.0/lam_array
train_error_mean, train_error_std, test_error_mean, test_error_std = kfold_cv(x_deg, y_samples, lam_array)
optimal_lambda = lam_array[np.where(test_error_mean==np.amin(test_error_mean))[0]]
```

    99.9900000000000143

```python
fig, ax = plt.subplots(figsize=(12,7))
ax.plot(lam_array, test_error_mean, label="test error")
ax.plot(lam_array, train_error_mean, label="train error")
ax.set_xscale("log")
ax.set_yscale("log")
ax.set_xlabel("lambda (Less capacity <--- ---> More capacity)", fontsize=14)
ax.set_ylabel("errors log scale", fontsize=14)
ax.set_title("Cross validation results", fontsize=14)
ax.set_xlim([np.amax(lam_array), np.amin(lam_array)])
ax.axvline(x=optimal_lambda, color='r', linestyle='--',
           lw=4, label="Optimal lambda = "+str(optimal_lambda))
ax.legend(fontsize=14)
plt.show()
```

It is easy to see how the parameter $\lambda$ change the effective capacity and produce a smooth transition between underfitting (left side of the optimal $\lambda$, too large $\lambda$) and overfitting (right side of the optimal $\lambda$, too small $\lambda$) regime.

```python
deg_w = (inv(x_deg_train.T @ x_deg_train + optimal_lambda*np.identity(deg+1)) @ x_deg_train.T @ y_train)
deg_p = np.poly1d(w[:,0])
```

```python
fig, ax = plt.subplots(figsize=(12,7))
ax.plot(x_plot, linear_function(w, aug_x_plot), label="Real solution", lw=4)
ax.scatter(x_train, y_train, label="train samples", s=70)
ax.scatter(x_test, y_test, label="test samples", s=70)
ax.plot(x_plot, deg_p(x_plot),'-o' ,label="regularized high capacity model", lw=1,ms=4)
ax.legend(fontsize=14)
ax.set_xlabel("x", fontsize=14)
ax.set_ylabel("y", fontsize=14)
ax.set_ylim([-15, 5])
plt.show()
```


```python
# Question 33, Assignment 1
# firstorder partial derivatives of f(x,y)
from sympy import plot_implicit, Eq
from sympy.abc import x, y
p=plot_implicit (Eq(4*y*y*y+2*x,0) ,(x, -2, 2), (y, -2, 2),show=False, line_color='r')
p2=plot_implicit (Eq(2*y+2*x,0) ,(x, -2, 2), (y, -2, 2),show=False, line_color='b')
p.extend(p2)
p.show()
```

```python
# Question 33, Assignment 1
# 3D Graph of f(x,y)
from mpl_toolkits import mplot3d

import numpy as np
import matplotlib.pyplot as plt

def z_function(x, y):
    return x**2+2*x*y+y**4

x = np.linspace(-1, 1, 100)
y = np.linspace(-1, 1, 100)

X, Y = np.meshgrid(x, y)
Z = z_function(X, Y)

fig = plt.figure(figsize=(12,12))
ax = plt.axes(projection="3d")
ax.plot_surface(X, Y, Z, rstride=1, cstride=1,
                cmap='coolwarm')
ax.set_xlabel('X')
ax.set_ylabel('Y')
ax.set_zlabel('f(x,y)')
ax.set_title('f(x,y)=x**2+2*x*y+y**4')
ax.set_zlim3d(-0.5, 2)
ax.view_init(-0,105)
```

```python

```


# Iterative Solvers 3 - The Conjugate Gradient Method

## Symmetric positive definite matrices

A very frequent type of matrices are symmetric positive definite matrices. Let $A\in\mathbb{R}^{n\times n}$ be a symmetric matrix (that is $A^T=A$). $A$ is called symmetric positive definite if

$$
x^TAx > 0, \forall x\neq 0.
$$

This is equivalent to the condition that all eigenvalues of $A$ are larger than zero (remember that symmetric matrices only have real eigenvalues).

One application of symmetric positive definite matrices are energy functionals. The expression $x^TAx$ arises when discretising functional involving kinetic energies (e.g. energies of the from $E = \frac{1}{2}m|\nabla f|^2$ for f a given function).

For linear systems involving symmetric positive definite matrices we can derive a special algorithm, namely the Method of Conjugate Gradients (CG).

## Lanczos - Arnoldi for symmetric matrices

Let us start with the Arnoldi recurrence relation

$$
AV_m = V_mH_m + h_{m+1,m}v_{m+1}e_m^T
$$

We know that $H_m$ is an upper Hessenberg matrix (i.e. the upper triangular part plus the first lower triangular diagonal can only be nonzero). Also, we know from the orthogonality of the $v_k$ vectors that

$$
V_m^TAV_m = H_m.
$$

Let $A$ now be symmetric. From the symmetry of $A$ an even nicer structure for $H_m$ arises. $H_m$ is upper Hessenberg, but now it is also symmetric. The only possible type of matrices to satisfy this condition are tridional matrices. These are matrices, where only the diagonal and the first upper and lower super/subdiagonals are nonzero.

Let us test this out. Below you find our simple implementation of Arnoldi's method. We then plot the resulting matrix $H_m$.

```python
import numpy as np

def arnoldi(A, r0, m):
    """Perform m-1 step of the Arnoldi method."""
    n = A.shape[0]

    V = np.empty((n, m + 1), dtype=np.float64)
    H = np.zeros((m+1, m), dtype=np.float64)

    V[:, 0] = r0 / np.linalg.norm(r0)

    for index in range(m):
        # Multiply the previous vector with A
        tmp = A @ V[:, index]
        # Now orthogonalise against the previous basis vectors
        h = V[:, :index + 1].T @ tmp # h contains all inner products against previous vectors
        H[:index + 1, index] = h
        w = tmp - V[:, :index + 1] @ h # Subtract the components in the directions of the previous vectors
        # Normalise and store
        H[index + 1, index] = np.linalg.norm(w)
        V[:, index + 1] = w[:] / H[index + 1, index]

    return V, H
```

The following code creates a random symmetric positive definite matrix.

```python
from numpy.random import RandomState

n = 500

rand = RandomState(0)
Q, _ = np.linalg.qr(rand.randn(n, n))
D = np.diag(rand.rand(n))
A = Q.T @ D @ Q
```

Now let's run Arnoldi's method and plot the matrix H. We are adding some artificial noise so as to ensure for the log-plot that all values are nonzero. The colorscale shows the logarithm of the magnitude of the entries.

```python
%matplotlib inline
from matplotlib import pyplot as plt

m = 50
r0 = rand.randn(n)
V, H = arnoldi(A, r0, m)

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)

im = ax.imshow(np.log10(1E-15 + np.abs(H)))
fig.colorbar(im)
```

It is clearly visible that only the main diagonal and the first upper and lower off-diagonal are nonzero, as expected. This hugely simplifies the Arnoldi iteration. Instead of orthogonalising $Av_m$ against all previous vectors we only need to orthogonalise against $v_m$ and $v_{m-1}$. All other inner products are already zero. Hence, the main orthogonalisation step now takes the form

$$
w = Av_m - (v_m^TAv_m)v_m - (v_{m-1}^TAv_m)v_{m-1}.
$$

Since the new vector $w$ is composed of only 3 vectors. This is also called a 3-term recurrence. The big advantage is that in addition to $Av_m$ we only need to keep $v_m$ and $v_{m-1}$ in memory. Hence, no matter how many iterations we do, the memory requirement remains constant, in contrast to Arnoldi for nonsymmetric matrices, where we need to keep all previous vectors in memory.

Arnoldi with a short recurrence relation for symmetric matrices has a special name. It is called **Lanczos method**.

## Solving linear systems of equations with Lanczos

We can now proceed exactly as in the Full orthogonalisation method and arrive at the linear system of equations

$$
T_my_m = \|r_0\|_2e_1,
$$

where $x_m = x_0 + V_my_m$ and $T_m = V_m^TAV_m$ is the tridiagonal matrix obtained from the Lanczos method.

The conjugate gradient method is an implementation of this approach. A very good derivation from Lanczos to CG is obtained in the beautiful book by Yousef Saad "[Iterative Methods for Sparse Linear Systems](https://www-users.cs.umn.edu/~saad/IterMethBook_2ndEd.pdf)", which is available online for free. Here, we will briefly motivate another approach to CG, which is a bit more intuitive and reveals more about the structure of the method, namely CG as an optimisation algorithm for a quadratic minimisation problem. One of the most beautiful summaries of this approach is contained in the paper [An introduction to the Conjugate Gradient Method Without the Agonizing Pain](https://www.cs.cmu.edu/~quake-papers/painless-conjugate-gradient.pdf) by Jonathan Shewchuk.

## A quadratic optimisation problem

We consider the quadratic minimisation problem

$$
\min_{x\in\mathbb{R}^n} f(x)
$$

with $f(x)=\frac{1}{2}x^TAx - b^Tx$. We have

$$
\nabla f(x) = Ax - b
$$

and hence the only stationary point is the solution of the linear system $Ax=b$. Furthermore, it is really a minimiser since $f''(x) > 0$ for all $x\in\mathbb{R}^n$ as $A$ is positive definite.

## The Method of Steepest Descent

Our first idea is the method of steepest descent. Remember that the negative gradient is a descent direction. Given a point $x_k$. We have

$$
-\nabla f(x_k) = b - Ax_k := r_k.
$$

The negative gradient is hence just the residual. Hence, we need to minimise along the direction of the residual, that is we will have
$x_{k+1} = x_k + \alpha_k r_k$ for some value $\alpha_k$. To compute $\alpha_k$ we just solve

$$
\frac{d}{d\alpha}f(x_k + \alpha r_k) = 0
$$

Since $\frac{d}{d\alpha}f(x_k + \alpha r_k) = r_{k+1}^Tr_k$ we just need to choose $\alpha_k$ such that $r_{k+1}$ is orthogonal to $r_k$. The solution is given by $\alpha_k = \frac{r_k^Tr_k}{r_k^TAr_k}$. This gives us a complete method consisting of three steps to get from $x_k$ to $x_{k+1}$.

$$
\begin{align}
r_k &= b - Ax_k\nonumber\\
\alpha_k &= \frac{r_k^Tr_k}{r_k^TAr_k}\nonumber\\
x_{k+1} &= x_k + \alpha_k r_k
\end{align}
$$

We are not going to derive the complete convergence analysis here but only state the final result. Let $\kappa := \frac{\lambda_{max}}{\lambda_{min}}$, where $\lambda_{max}$ and $\lambda_{min}$ are the largest, respectively smallest eigenvalue of $A$ (remember that all eigenvalues are positive since $A$ is symmetric positive definite). The number $\kappa$ is called the condition number of $A$. Let $e_k = x_k - x^*$ be the difference of the exact solution $x^*$ satisfying $Ax^*=b$ and our current iterate $x_k$. Note that $r_k = -Ae_k$.

We now have that

$$
\|e_k\|_A\leq \left(\frac{\kappa - 1}{\kappa + 1}\right)^k\|e_0\|_A,
$$

where $\|e_k\|_A := \left(e_k^TAe_k\right)^{1/2}$.

This is an extremely slow rate of convergence. Let $\kappa=10$, which is a fairly small number. Then the error reduces in each step only by a factor of $\frac{9}{11}\approx 0.81$ and we need 11 iterations for each digit of accuracy.

## The method of conjugate directions

The steepest descent approach was not bad. But we want to improve on it. The problem with the steepest descent method is that we have no guarantee that we are reducing the error $e_{k+1}$ as much as possible along our current direction $r_k$ when we minimize. But we can fix this.

Let us pick a set of directions $d_0, d_1, \dots, d_{n-1}$, which are mutually orthogonal, that is $d_i^Td_j =0$ for $i\neq j$. We now want to enforce the condition that

$$
e_{k+1}^Td_k = 0.
$$

This means that the remaining error is orthogonal to $d_k$ and hence is a linear combination of all the other search directions. We have therefore exhausted all the information from $d_k$. Let's play this through.

We know that $e_{k+1} = x_{k+1} - x^* = x_k -x^* + \alpha_k d_k = e_k + \alpha_kd_k$.

It follows that

$$
\begin{align}
e_{k+1}^Td_k &= d_k^T(e_k + \alpha_kd_k)\nonumber\\
             &= d_k^Te_k + \alpha_kd_k^Td_k = 0\nonumber
\end{align}
$$

and therefore $\alpha_k = -\frac{d_k^Te_k}{d_k^Td_k}$.

Unfortunately, this does not quite work in practice as we don't know $e_k$. But there is a solution. Remember that $r_k = -Ae_k$. We just need an $A$ in the right place. To achieve this we choose **conjugate directions**, that is we impose the condition that

$$
d_i^TAd_j = 0
$$

for $i\neq j$. We also impose the condition that $e_{k+1}^TAd_k = 0$. Writing this out we obtain

$$
\alpha_k = \frac{d_k^Tr_k}{d_k^TAd_k}.
$$

This expression is computable if we have a suitable set of conjugate directions $d_k$. Moreoever, it guarantees that the method converges in at most $n$ steps since in every iteration we are annihiliating the error in the direction of $d_k$ and there are only $n$ different directions.

## Conjugate Gradients - Mixing steepest descent with conjugate directions

The idea of conjugate gradients is to obtain the conjugate directions $d_i$ by taking the $r_i$ (the gradients) and to $A$-orthogonalise (conjugate) them against the previous directions. We are leaving out the details of the derivation and refer to the Shewchuk paper. But the final algorithm now takes the following form.

$$
\begin{align}
d_0 &= r_0 = b - Ax_0\nonumber\\
\alpha_i &= \frac{r_i^Tr_i}{d_i^TAd_i}\nonumber\\
x_{i+1} &=x_i + \alpha_id_i\nonumber\\
r_{i+1} &= r_i - \alpha_i Ad_i\nonumber\\
\beta_{i+1} &= \frac{r_{i+1}^Tr_{i+1}}{r_i^Tr_i}\nonumber\\
d_{i+1} &= r_{i+1} + \beta_{i+1}d_i\nonumber
\end{align}
$$

Conjugate Gradients has a much more favourable convergence bound than steepest descent. One can derive that

$$
\|e_i\|_A\leq 2\left(\frac{\sqrt{\kappa} - 1}{\sqrt{\kappa} + 1}\right)^i\|e_0\|_A.
$$

If we choose again the example that $\kappa=10$ we obtain

$$
\|e_i\|A\lessapprox 0.52^i\|e_0\|_A.
$$

Hence, we need around $4$ iterations for each digits of accuracy instead of 11 for the method of steepest descent.

## A numerical example

The following code creates a symmetric positive definite matrix.

```python
from scipy.sparse import diags

n = 10000

data = [2.1 * np.ones(n),
        -1. * np.ones(n - 1),
        -1. * np.ones(n - 1)]

offsets = [0, 1, -1]

A = diags(data, offsets=offsets, shape=(n, n), format='csr')
```

We now solve the associated linear system with CG and plot the convergence.

```python
from scipy.sparse.linalg import cg

b = rand.randn(n)
residuals = []

callback = lambda x: residuals.append(np.linalg.norm(b - A @ x) / np.linalg.norm(b))

sol, _ = cg(A, b, tol=1E-6, callback=callback, maxiter=1000)

fig = plt.figure(figsize=(8, 8))
ax = fig.add_subplot(111)
ax.semilogy(1 + np.arange(len(residuals)), residuals, 'k--')
ax.set_title('CG Convergence')
ax.set_xlabel('Iteration Step')
_ = ax.set_ylabel('$\|Ax-b\|_2 / \|b\|_2$')
```


# Solving Ax = b

This work by Jephian Lin is licensed under a [Creative Commons Attribution 4.0 International License](http://creativecommons.org/licenses/by/4.0/).

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
import sympy
```

## Main idea

Let $A$ be an $m\times n$ matrix.  The solution set of $A{\bf x} = {\bf b}$ is the intersection of the affine planes $\langle {\bf r}_i, {\bf x}\rangle = b_i$ for all $i=1,\ldots,m$, where ${\bf r}_i$ is the $i$-th row of $A$ and $b_i$ is the $i$-th entry of ${\bf b}$.
Therefore, the solution set of $A$ is an affine space (shifted space).

The solutions set of $A{\bf x} = {\bf b}$ is of the form:

    general solutions = particular solution + homogeneous solutions
    (a shifted space)      (a vector)              (a space)

Here "general solution" stands for all solutions of $A{\bf x} = {\bf b}$,
"particular solution" stands for one arbitrary solution of $A{\bf x} = {\bf b}$, and
"homogeneous solutions" stands for all solutions of $A{\bf x} = {\bf 0}$.

Every matrix lead to its **reduced echelon form** after some **row operations**.  If $\left[\begin{array}{cc}R | {\bf r}\end{array}\right]$ is the reduced echelon form of $\left[\begin{array}{cc}A | {\bf b}\end{array}\right]$, then $A{\bf x} = {\bf b} = R{\bf x} = {\bf r}$.

## Side stories

- $A{\bf x} = {\bf b} \iff {\bf b}\in\operatorname{Col}(A)$
- matrix inverse

## Experiments

###### Exercise 1
This exercise helps you to visualize the affine space $A{\bf x} = {\bf b}$.
Let
```python
A = np.array([[1,1,1],
              [1,1,1]])
b = np.array([5,5])
```

###### 1(a)
Use the techniques you learned in Lesson 2 to draw some random solutions of $A{\bf x} = {\bf b}$.
What is the nullity of $A$?  What is the "dimension" of the affine space?
Hint:
```python
xs = 5*np.random.randn(3,10000)
mask = (np.abs(b[:,np.newaxis] - A.dot(xs)) < 0.1).all(axis = 0)
```

```python
### your answer here
```

###### 1(b)
It is known that
```python
p = np.array([5,0,0])
```
is a particular solution of $A{\bf x} = {\bf b}$.
Add a vector of `p` upon your previous drawing.

```python
### your answer here
```

###### 1(c)
Do the same for
```python
b = np.array([5,6])
```
How many solutions are there?

```python
### your answer here
```

###### Exercise 2
This exercise helps you to visualize the affine space $A{\bf x} = {\bf b}$.
Let
```python
A = np.array([[1,1,1],
              [1,1,1]])
b = np.array([5,5])
```

###### 2(a)
Draw the grid using the columns of $A$ and draw a vector for $b$.
Is $b$ in the column space of $A$?

```python
### your answer here
```

###### 2(b)
Do the same for
```python
b = np.array([5,6])
```
Is $b$ in the column space of $A$?

```python
### your answer here
```

#### Remark
Whether a particular solution exists depends only on whether ${\bf b}$ is in the column space of $A$ or not.
We say a equation $A{\bf x} = {\bf b}$ is **consistent** if it has at least a particular solution.

Whether the homogeneous solutions contains only the trivial solution ${\bf 0}$ depends only on $A$.

This table summarize the number of solutions of $A{\bf x} = {\bf b}$.

 hom \ par | consistent | inconsistent
 --------- | ---------- | ------------
 trivial | one | none
 nontrivial | infinite | none

## Exercises

##### Exercise 3
Let
```python
A = sympy.Matrix([[1,1],
                  [-1,0],
                  [0,-1]])
b = sympy.Matrix([3,-2,-1])
Ab = A.col_insert(2,b)
```

###### 3(a)
Calculate the reduced echelon form of `Ab` .
Can you tell if  `b` is in the column space of `A` ?

```python
### your answer here
```

###### 3(b)
Let
```python
b = sympy.Matrix([1,2,3])
```
and update `Ab` .
Can you tell if `b` is in the column space of `A` ?

```python
### your answer here
```

##### Exercise 4
Let
```python
A = sympy.Matrix([[1,1,1],
                  [1,2,4],
                  [1,3,9]])
b1 = sympy.Matrix([1,0,0])
```

###### 4(a)
If a matrix has no free variable, then the homogeneous solution is trivial.
Find the unique solution of $A{\bf x} = {\bf b}_1$.

```python
### your answer here
```

###### 4(b)
Let
```python
b2 = sympy.Matrix([0,1,0])
Ab = A.col_insert(3,b1)
Abb = Ab.col_insert(4,b2)
```
Can you use `Abb` to solve the solutions of $A{\bf x} = {\bf b}_1$ and $A{\bf x} = {\bf b}_2$ at once?

```python
### your answer here
```

###### 4(c)
Let
```python
b3 = sympy.Matrix([0,0,1])
```
Solve the solutions of $A{\bf x} = {\bf b}_1$,  $A{\bf x} = {\bf b}_2$, and $A{\bf x} = {\bf b}_3$ at once.

```python
### your answer here
```

###### 4(d)
Let
$$ B = \begin{bmatrix}
 | & ~ & | \\
 {\bf b}_1 & \cdots & {\bf b}_3 \\
 | & ~ & |
 \end{bmatrix}.$$
 Find a matrix $X$ such that $AX = B$.
 When $B$ is the identity matrix
$$ I_n = \begin{bmatrix}
 1 & ~ & ~ \\
 ~ & \ddots & ~ \\
 ~ & ~ & 1
 \end{bmatrix},$$
 the matrix $X$ with $AX = I_n$ is called the **inverse** of $A$, denoted by $A^{-1}$.

```python
### your answer here
```

###### 4(e)
Compare your answer in 4(d) with the output of `np.linalg.inv(A)` .

```python
### your answer here
```

##### Exercise 5
Let
```python
A = sympy.Matrix([[1,3,3,18],
                  [5,15,16,95],
                   [-5,-15,-15,-90]])
R,pvts = A.rref()
```

###### 5(a)
Let $B$ be the matrix whose columns are the columns of $A$ the corresponding to leading variables.
Pick a column of $A$ corresponding a free variable.
Check that the column is in the column space of $B$.
(If yes, this means this column is redundant for generating the column space of $A$.)

```python
### your answer here
```

###### 5(b)
Check if $B$ itself has any redundant column.

```python
### your answer here
```

#### Remark
Let $S = \{{\bf u}_1, \ldots, {\bf u}_n\}$ be a collection of vectors and $A$ the matrix whose columns are $S$.
We say $S$ is **linearly independent** if one of the following equivalent condition holds:
- $c_1{\bf u}_1 + \cdots + c_n{\bf u}_n = {\bf 0}$ only have trivial solution $c_1 = \cdots = c_n = 0$.
- $A{\bf x} = {\bf 0}$ only have trivial solution ${\bf x} = 0$.
- $A$ has no free variable.

Moreover, if a space $V$ is equal to$\operatorname{span}(S)$ and $S$ is linearly independent, then we say $S$ is a **basis** of the the space $V$.

##### Exercise 6
Let
```python
A = sympy.Matrix([[1,1,1],
                  [-1,0,0],
                  [0,-1,0],
                  [0,0,-1]])
```
Check if the columns of $A$ form a linearly independent set.

```python
### your answer here
```

##### Exercise 7
```python
A = sympy.Matrix([[1,3,3,18],
                  [5,15,16,95],
                  [-5,-15,-15,-90]])
R,pvts = A.rref()
```
Check what is `A.nullspace()`, `A.rowspace()`, and `A.columnspace()` and think about their meaning.

```python
### your answer here
```

#### Remark
Since it is impossible to output a space, the three commands in Exercise 7 in fact outputs the basis of the space only, which is enough.

**Nullspace**: its basis consists of ${\bf h}$'s in the previous lesson.
**Rowspace**: its basis consists of the rows of $R$ corresponding to the pivots.
**Columnspace**: its basis consists of the columns of $A$ corresponding to the pivots.

Mathematics/Linear Algebra/0.0b Supplement of MIT Course - Concept Explanations with Problems_and_Questions to Solve/06-Solving-Ax-=-b.ipynb


## 2 Analyzing the Data

```python
import numpy as np
import pandas as pd
import scipy.stats as st
import statsmodels.api as sm
import seaborn as sns
from statsmodels.regression.rolling import RollingOLS
import warnings
warnings.filterwarnings('ignore')
```

```python
df = pd.read_excel("proshares_analysis_data.xlsx",sheet_name="hedge_fund_series",index_col="date")
df.head()
```

      HFRIFWI Index
      MLEIFCTR Index
      MLEIFCTX Index
      HDG US Equity
      QAI US Equity

      date

      2011-08-31
      -0.032149
      -0.025588
      -0.025689
      -0.027035
      -0.006491

      2011-09-30
      -0.038903
      -0.032414
      -0.032593
      -0.032466
      -0.022142

      2011-10-31
      0.026858
      0.043593
      0.043320
      0.050532
      0.025241

      2011-11-30
      -0.013453
      -0.012142
      -0.012431
      -0.028608
      -0.007965

      2011-12-31
      -0.004479
      0.001938
      0.001796
      0.012875
      0.001854

### 1. For the series in the “hedge fund series” tab, report the following summary statistics:

(a) mean

(b) volatility

(c) Sharpe ratio

Annualize these statistics.

```python
def summary(df):
    '''
    Given a pandas DataFrame of time series with monthly data, returns annualized mean, volatility and Sharpe Ratio
    '''
    mean = df.mean()*12
    volatility = df.std()*np.sqrt(12)
    out = pd.DataFrame({'mean':mean,'volatility':volatility})
    out['SharpeRatio'] = out['mean']/out['volatility']
    return out
summary(df)
```

      mean
      volatility
      SharpeRatio

      HFRIFWI Index
      0.050784
      0.061507
      0.825665

      MLEIFCTR Index
      0.038821
      0.053848
      0.720933

      MLEIFCTX Index
      0.037330
      0.053682
      0.695382

      HDG US Equity
      0.028100
      0.056380
      0.498407

      QAI US Equity
      0.025491
      0.045484
      0.560434

### 2. For the series in the “hedge fund series” tab, , calculate the following statistics related to tailrisk.

(a) Skewness

(b) Excess Kurtosis (in excess of 3)

(c) VaR (.05) - the fifth quantile of historic returns

(d) CVaR (.05) - the mean of the returns at or below the fifth quantile

(e) Maximum drawdown - include the dates of the max/min/recovery within the max drawdown period.

There is no need to annualize any of these statistics

```python
def tailRisks(df):
    '''
    Given a pandas DataFrame of time series, returns Skewness, Excess Kurtosis, 5% VaR & CVar, Maximum Drawdown
    '''

    sk = df.skew()
    #kt = st.kurtosis(df,fisher=True)
    kt = df.kurtosis()
    VaR = df.quantile(0.05)
    CVaR = df[df < VaR].mean()

    levels = (1+df).cumprod()
    high = levels.cummax()
    drawdown = (levels - high)/high
    maxDrawdown = drawdown.min()

    bottom = pd.Series(pd.to_datetime([drawdown[col].idxmin() for col in drawdown]),index=df.columns)
    peak = pd.Series(pd.to_datetime([(levels[col][:bottom[col]].idxmax()) for col in levels]),index=df.columns)

    peakLevels = pd.Series([levels[col].loc[peak.loc[col]] for col in levels],index=df.columns)

    recovered = []
    for col in levels:
        for lev in levels[col][bottom[col]:]:
            if lev >= peakLevels[col]:
                recovered.append(levels.index[levels[col] == lev][0])
                break

    recovered = pd.Series(recovered,index=df.columns)

    drawdown.plot(figsize=(9,6), title = 'Drawdown')

    return pd.DataFrame({'Skewness':sk,'Excess Kurtosis':kt,'5% VaR':VaR,'5% CVaR':CVaR,'Max Drawdown':maxDrawdown,\
                        'Peak':peak,'Bottom':bottom,'Recovered':recovered})

tailRisks(df)
```

## 3. For the series in the “hedge fund series” tab, run a regression of each against SPY (found in the “merrill factors” tab.) Include an intercept. Report the following regression-based statistics:

(a) Market Beta

(b) Treynor Ratio

(c) Information ratio

Annualize these three statistics as appropriate.

```python
merrill_factors = pd.read_excel("proshares_analysis_data.xlsx",sheet_name= "merrill_factors",index_col="date")
spy = merrill_factors['SPY US Equity']
merrill_factors.head()
```

      SPY US Equity
      USGG3M Index
      EEM US Equity
      EFA US Equity
      EUO US Equity
      IWM US Equity

      date

      2011-08-31
      -0.054976
      0.000009
      -0.092549
      -0.087549
      -0.005889
      -0.088913

      2011-09-30
      -0.069449
      0.000017
      -0.179064
      -0.108083
      0.142180
      -0.111541

      2011-10-31
      0.109147
      -0.000013
      0.162986
      0.096275
      -0.069502
      0.151012

      2011-11-30
      -0.004064
      0.000000
      -0.019723
      -0.021764
      0.054627
      -0.003783

      2011-12-31
      0.010440
      0.000009
      -0.043017
      -0.022139
      0.075581
      0.005114

```python
def performanceMeasures(seriesY,seriesX):

    mean =seriesY.mean()*12
    sharpe = mean/(seriesY.std()*(12**0.5))
    model = sm.OLS(seriesY,sm.add_constant(seriesX)).fit()
    rsq = model.rsquared
    beta = model.params[1]
    treynor = mean/beta
    information = model.params[0]/(model.resid.std()*np.sqrt(12))

    return pd.DataFrame({'Mean Return':mean,'Sharpe Ratio':sharpe,'R Squared':rsq, 'Beta':beta,\
                         'Treynor Ratio':treynor, 'Information Ratio':information},index= [seriesY.name])
```

```python
frames = []
for col in df:
    p = performanceMeasures(df[col],spy)
    frames.append(p)
pd.concat(frames)
```

      Mean Return
      Sharpe Ratio
      R Squared
      Beta
      Treynor Ratio
      Information Ratio

      HFRIFWI Index
      0.050784
      0.825665
      0.753090
      0.394320
      0.128789
      -0.020169

      MLEIFCTR Index
      0.038821
      0.720933
      0.816158
      0.359382
      0.108021
      -0.051272

      MLEIFCTX Index
      0.037330
      0.695382
      0.815047
      0.358034
      0.104263
      -0.055939

      HDG US Equity
      0.028100
      0.498407
      0.785733
      0.369199
      0.076111
      -0.084218

      QAI US Equity
      0.025491
      0.560434
      0.719379
      0.284993
      0.089443
      -0.057274

### 4. Relative Performance

Discuss the previous statistics, and what they tell us about...

(a) the differences between SPY and the hedge-fund series?

The large part of most Hedge Fund series can be explained using SPY as the R Squared from Regrssening Hedge Fund series on SPY high (greater than 70% for each series.

(b) which performs better between HDG and QAI.

HDG has higher Mean Returns than QAI but has lower Sharpe Ratio, Treynor Ratio and Information Ratios. Thus higher returns have come with higher volaility in HDG. From a mean variance optimized standpoint, QAI has performed better.

(c) whether HDG and the ML series capture the most notable properties of HFRI.

We note (from question 5) that both HDG and ML are highly correlated with HFRI with a correlation of about 90%. However, HFRI's excess kurtosis of 6.73 is substantially higher than the excess kurtosis of HDG and ML (about 2.4-2.5). Thus HDG and ML would fail to capture the very high tail-risk of HFRI.

Furhter, HDG and ML series are highly correlated (~98% correlation). Regressing HFRI on HDG and ML together would thus suffer from a high multi-collinearity.

## 5. Report the correlation matrix for these assets.
(a) Show the correlations as a heat map.
(b) Which series have the highest and lowest correlations?

```python
correlations = df.corr()
correlations
```

      HFRIFWI Index
      MLEIFCTR Index
      MLEIFCTX Index
      HDG US Equity
      QAI US Equity

      HFRIFWI Index
      1.000000
      0.910392
      0.910063
      0.897653
      0.866714

      MLEIFCTR Index
      0.910392
      1.000000
      0.999939
      0.984621
      0.864431

      MLEIFCTX Index
      0.910063
      0.999939
      1.000000
      0.984471
      0.863734

      HDG US Equity
      0.897653
      0.984621
      0.984471
      1.000000
      0.847597

      QAI US Equity
      0.866714
      0.864431
      0.863734
      0.847597
      1.000000

```python
sns.heatmap(correlations)
```

```python
def findHighestLowestCorrelations(correlationsMatrix):
    high, low = [],[]
    for col in correlationsMatrix.columns:

        val_high = correlationsMatrix[col][(correlationsMatrix[col] == correlationsMatrix[col].nlargest(2).iloc[1])]
        high.append((val_high.index[0],val_high.name,val_high[0]))

        val_low = correlationsMatrix[col][(correlationsMatrix[col] == min(correlationsMatrix[col]))]
        low.append((val_low.index[0],val_low.name,val_low[0]))

    print("Highest correlation pair = ",max(high,key = lambda x:x[2]))
    print("Lowest correlation pair = ",min(low,key = lambda x:x[2]))

findHighestLowestCorrelations(correlations)
```

    Highest correlation pair =  ('MLEIFCTX Index', 'MLEIFCTR Index', 0.9999391791142366)
    Lowest correlation pair =  ('QAI US Equity', 'HDG US Equity', 0.8475965688236916)

## 6. Replicate HFRI with the six factors listed on the “merrill factors” tab. Include a constant, and run the unrestricted regression,

  \begin{align}
  r^{hfri}_{t} = \alpha^{merr} + x^{merr}_{t} \beta^{merr} + \epsilon^{merr}_{t} (1)
  \end{align}

  \begin{align}
  \hat r^{hfri}_{t} = \hat\alpha^{merr} + x^{merr}_{t} \hat \beta^{merr} (2)
  \end{align}

Note that the second equation is just our notation for the fitted replication.

(a) Report the intercept and betas.

(b) Are the betas realistic position sizes, or do they require huge long-short positions?

(c) Report the R-squared.

(d) Report the volatility of $\epsilon^{merr}$, (the tracking error.)

```python
model = sm.OLS(df['HFRIFWI Index'],sm.add_constant(merrill_factors)).fit()
model.params
```

    const            0.001147
    SPY US Equity    0.072022
    USGG3M Index    -0.400591
    EEM US Equity    0.072159
    EFA US Equity    0.106318
    EUO US Equity    0.022431
    IWM US Equity    0.130892
    dtype: float64

The Betas are realistic position sizes except for USGG3M Index

```python
print("R-squared of the Model =",round(model.rsquared,6))
```

    R-squared of the Model = 0.855695

```python
print("Volatility of Error term (annualized) =",round(model.resid.std()*(12**0.5),6))
```

    Volatility of Error term (annualized) = 0.023365

### 7. Let’s examine the replication out-of-sample.
Starting with t = 61 month of the sample, do the following:

• Use the previous 60 months of data to estimate the regression equation, (1). This gives time-t estimates of the regression parameters, $\tilde \alpha^{merr}_{t}$ and $\tilde \beta^{merr}_{t}$

• Use the estimated regression parameters, along with the time-t regressor values, $x^{merr}_{t}$, to calculate the time-t replication value that is, with respect to the regression estimate, built

\begin{align}
\tilde r^{merr}_{t} \equiv \tilde \alpha^{merr}_{t} + (x^{merr}_{t})\prime \tilde \beta^{merr}_{t}
\end{align}

• Step forward to t = 62, and now use t = 2 through t = 61 for the estimation. Re-run the
steps above, and continue this process throughout the data series. Thus, we are running a
rolling, 60-month regression for each point-in-time.
How well does the out-of-sample replication perform with respect to the target?

```python
def get_est_coef(Y, X):

    X_add_const = sm.add_constant(X)

    model = sm.OLS(Y, X_add_const)

    res = model.fit()

    return res.params[0], res.params[1:]

fitted_values = []
window_size = 60

for i in range(df.shape[0] - window_size - 1):

    alpha, beta = get_est_coef(df['HFRIFWI Index'][i : i+window_size], merrill_factors[i : i+window_size])

    fitted_values.append((alpha + np.dot(merrill_factors[i+window_size+1 : i+window_size+2], beta).item()))

table_7 = pd.DataFrame({'True Values': df['HFRIFWI Index'][61:], 'Fitted Values': fitted_values})

table_7.plot(figsize=(13,8))

```

```python
table_7.corr()
```

      True Values
      Fitted Values

      True Values
      1.000000
      0.943363

      Fitted Values
      0.943363
      1.000000

We observe that the predicted value is 94.34% correlated with actual performance. Thus out-of-sample replication performs very well. The same can also be visualized in the chart below:

## 8. We estimated the replications using an intercept. Try the full-sample estimation, but this time without an intercept.

 \begin{align}
  r^{hfri}_{t} = x^{merr}_{t} \beta^{merr} + \epsilon^{merr}_{t}
  \end{align}

  \begin{align}
  \check r^{hfri}_{t} = x^{merr}_{t} \check \beta^{merr}
  \end{align}

Report

(a) the regression beta. How does it compare to the estimated beta with an intercept, $\hat \beta^{merr}$?

```python
model_without_intercept = sm.OLS(df['HFRIFWI Index'],merrill_factors).fit()
regressionBetas = pd.DataFrame({"With Intercept":model.params,"Without Intercept":model_without_intercept.params})
regressionBetas
```

      With Intercept
      Without Intercept

      EEM US Equity
      0.072159
      0.069936

      EFA US Equity
      0.106318
      0.101524

      EUO US Equity
      0.022431
      0.023479

      IWM US Equity
      0.130892
      0.128999

      SPY US Equity
      0.072022
      0.087505

      USGG3M Index
      -0.400591
      0.334503

      const
      0.001147
      NaN

The Regression Betas are mostly similar in both the regressions except for USGG3M Index Beta. In the absence of intercept, there is a much higher weightage of USGG3M which could be thought of as a proxy of Risk Free Interest Rates

(b) the mean of the fitted value, $\check r^{hfri}_{t}$. How does it compare to the mean of the HFRI?

```python
pd.Series([df['HFRIFWI Index'].mean()*12,model_without_intercept.predict(merrill_factors).mean()*12],\
          index=['Mean of HFRI','Mean of Fitted Values'])
```

    Mean of HFRI             0.050784
    Mean of Fitted Values    0.042888
    dtype: float64

The mean of the Fitted Values is lower than the mean of HFRI

(c) the correlations of the fitted values, $\check r^{hfri}_{t}$ to the HFRI. How does the correlation compare to that of the fitted values with an intercept, $\hat r^{hfri}_{t}$?

```python
HFRIpredictions = pd.DataFrame({'HFRI Index':df['HFRIFWI Index'],\
                                'Predict with intercept':model.predict(sm.add_constant(merrill_factors)),\
                               'Predict without intercept':model_without_intercept.predict(merrill_factors)})
HFRIpredictions.corr()
```

      HFRI Index
      Predict with intercept
      Predict without intercept

      HFRI Index
      1.000000
      0.925038
      0.924517

      Predict with intercept
      0.925038
      1.000000
      0.999437

      Predict without intercept
      0.924517
      0.999437
      1.000000

The correlations of HFRI in both the cases is almost equally good. 92.45% correlatation between HFRI Index and Prediction without intercept. 92.50% correlation between HFRI and prediction with intercept. The predicted values with and without intercept are highly correlated. Thusincluding the intercept or not doesn't affect much for the prediction. If including the intercept, the intercept part would be non-tradable asset. Without intercept, it's in accordance with our purpose of replicating(using all tradable assets).

Do you think Merrill and ProShares fit their replicators with an intercept or not?

We think Merrill and ProShares would use the replicators  without the Intercept  as the purpose is to replicate the entire HFRI Index and not just mimic the returns from HFRI Index.


```python
from sympy import Matrix
```

```python
A = Matrix([[0,2,0,1],[2,2,3,2],[4,-3,0,1],[6,1,-6,-5]])
```

```python
A
```

$\displaystyle \left[\begin{matrix}0 & 2 & 0 & 1\\2 & 2 & 3 & 2\\4 & -3 & 0 & 1\\6 & 1 & -6 & -5\end{matrix}\right]$

```python
B=Matrix([[0,-2,-7,6]])
```

```python
B = B.transpose()
```

```python
B
```

$\displaystyle \left[\begin{matrix}0\\-2\\-7\\6\end{matrix}\right]$

```python
A.gauss_jordan_solve(B)
```

    (Matrix([
     [-1/2],
     [   1],
     [ 1/3],
     [  -2]]),
     Matrix(0, 1, []))

<b>x1=-0.05 , x2=1.0,  x3=0.33, x4=-2.0</b>


# UMAP on sparse data

Sometimes datasets get very large, and potentially very very high dimensional. In many such cases, however, the data itself is sparse -- that is, while there are many many features, any given sample has only a small number of non-zero features observed. In such cases the data can be represented much more efficiently in terms of memory usage by a sparse matrix data structure. It can be hard to find dimension reduction techniques that work directly on such sparse data -- often one applies a basic linear technique such as ``TruncatedSVD`` from sklearn (which does accept sparse matrix input) to get the data in a format amenable to other more advanced dimension reduction techniques. In the case of UMAP this is not necessary -- UMAP can run directly on sparse matrix input. This tutorial will walk through a couple of examples of doing this. First we'll need some libraries loaded. We need ``numpy`` obviously, but we'll also make use of ``scipy.sparse`` which provides sparse matrix data structures. One of our examples will be purely mathematical, and we'll make use of ``sympy`` for that; the other example is test based and we'll use sklearn for that (specifically ``sklearn.feature_extraction.text``). Beyond that we'll need umap, and plotting tools.

```python
import numpy as np
import scipy.sparse
import sympy
import sklearn.datasets
import sklearn.feature_extraction.text
import umap
import umap.plot
import matplotlib.pyplot as plt
%matplotlib inline
```

    /Users/leland/anaconda3/envs/umap_0.4dev/lib/python3.7/site-packages/datashader/transfer_functions.py:21: FutureWarning: xarray subclass Image should explicitly define __slots__
      class Image(xr.DataArray):

## A mathematical example

Our first example constructs a sparse matrix of data out of pure math. This example is inspired by the work of [John Williamson](https://johnhw.github.io/umap_primes/index.md.html), and if you haven't looked at that work you are strongly encouraged to do so. The dataset under consideration will be the integers. We will represent each integer by a vector of its divisibility by distinct primes. Thus our feature space is the space of prime numbers (less than or equal to the largest integer we will be considering) -- potentially very high dimensional. In practice a given integer is divisible by only a small number of distinct primes, so each sample will be mostly made up of zeros (all the primes that the number is not divisible by), and thus we will have a very sparse dataset.

To get started we'll need a list of all the primes. Fortunately we have ``sympy`` at our disposal and we can quickly get that information with a single call to ``primerange``. We'll also need a dictionary mapping the different primes to the column number they correspond to in our data structure; effectively we'll just be enumerating the primes.

```python
primes = list(sympy.primerange(2, 110000))
prime_to_column = {p:i for i, p in enumerate(primes)}
```

Now we need to construct our data in a format we can put into a sparse matrix easily. At this point a little background on sparse matrix data structures is useful. For this purpose we'll be using the so called ["LIL" format](https://scipy-lectures.org/advanced/scipy_sparse/lil_matrix.html). LIL is short for "List of Lists", since that is how the data is internally stored. There is a list of all the rows, and each row is stored as a list giving the column indices of the non-zero entries. To store the data values there is a parallel structure containing the value of the entry corresponding to a given row and column.

To put the data together in this sort of format we need to construct such a list of lists. We can do that by iterating over all the integers up to a fixed bound, and for each integer (i.e. each row in our dataset) generating the list of column indices which will be non-zero. The column indices will simply be the indices corresponding to the primes that divide the number. Since ``sympy`` has a function ``primefactors`` which returns a list of the unique prime factors of any integer we simply need to map those through our dictionary to covert the primes into column numbers.

Parallel to that we'll construct the corresponding structure of values to insert into a matrix. Since we are only concerned with divisibility this will simply be a one in every non-zero entry, so we can just add a list of ones of the appropriate length for each row.

```python
%%time
lil_matrix_rows = []
lil_matrix_data = []
for n in range(100000):
    prime_factors = sympy.primefactors(n)
    lil_matrix_rows.append([prime_to_column[p] for p in prime_factors])
    lil_matrix_data.append([1] * len(prime_factors))
```

    CPU times: user 1.9 s, sys: 24.2 ms, total: 1.93 s
    Wall time: 1.93 s

Now we need to get that into a sparse matrix. Fortunately the ``scipy.sparse`` package makes this easy, and we've already built the data in a fairly useful structure. First we create a sparse matrix of the correct format (LIL) and the right shape (as many rows as we have generated, and as many columns as there are primes). This is essentially just an empty matrix however. We can fix that by setting the ``rows`` attribute to be the rows we have generated, and the ``data`` attribute to be the corresponding structure of values (all ones). The result is a sparse matrix data structure which can then be easily manipulated and converted into other sparse matrix formats easily.

```python
factor_matrix = scipy.sparse.lil_matrix((len(lil_matrix_rows), len(primes)), dtype=np.float32)
factor_matrix.rows = np.array(lil_matrix_rows)
factor_matrix.data = np.array(lil_matrix_data)
factor_matrix
```

    <100000x10453 sparse matrix of type '<class 'numpy.float32'>'
    	with 266398 stored elements in LInked List format>

As you can see we have a matrix with 100000 rows and over 10000 columns. If we were storing that as a numpy array it would take a great deal of memory. In practice, however, there are only 260000 or so entries that are not zero, and that's all we really need to store, making it much more compact.

The question now is how can we feed that sparse matrix structure into UMAP to have it learn an embedding. The answer is surprisingly straightforward -- we just hand it directly to the fit method. Just like other sklearn estimators that can handle sparse input UMAP will detect the sparse matrix and just do the right thing.

```python
%%time
mapper = umap.UMAP(metric='cosine', random_state=42, low_memory=True).fit(factor_matrix)
```

That was easy! But is it really working? We can easily plot the results:

```python
umap.plot.points(mapper, values=np.arange(100000), theme='viridis')
```

And this looks very much in line with the results [John Williamson got](https://johnhw.github.io/umap_primes/index.md.html) with the proviso that we only used 100,000 integers instead of 1,000,000 to ensure that most users should be able to run this example (the full million may require a large memory compute node). So it seems like this is working well. The next question is whether we can use the ``transform`` functionality to map new data into this space. To test that we'll need some more data. Fortunately there are more integers. We'll grab the next 10,000 and put them in a sparse matrix, much as we did for the first 100,000.

```python
%%time
lil_matrix_rows = []
lil_matrix_data = []
for n in range(100000, 110000):
    prime_factors = sympy.primefactors(n)
    lil_matrix_rows.append([prime_to_column[p] for p in prime_factors])
    lil_matrix_data.append([1] * len(prime_factors))
```

```python
new_data = scipy.sparse.lil_matrix((len(lil_matrix_rows), len(primes)), dtype=np.float32)
new_data.rows = np.array(lil_matrix_rows)
new_data.data = np.array(lil_matrix_data)
new_data
```

To map the new data we generated we can simply hand it to the ``transform`` method of our trained model. This is a little slow, but it does work.

```python
new_data_embedding = mapper.transform(new_data)
```

And we can plot the results. Since we just got the locations of the points this time (rather than a model) we'll have to resort to matplotlib for plotting.

```python
fig = plt.figure(figsize=(12,12))
ax = fig.add_subplot(111)
plt.scatter(new_data_embedding[:, 0], new_data_embedding[:, 1], s=0.1, c=np.arange(10000), cmap='viridis')
ax.set(xticks=[], yticks=[], facecolor='black');
```

The color scale is different in this case, but you can see that the data has been mapped into locations corresponding to the various structures seen in the original embedding. Thus, even with large sparse data we can embed the data, and even add new data to the embedding.

## A text analysis example

Let's look at a more classical machine learning example of working with sparse high dimensional data -- working with text documents. Machine learning on text is hard, and there is a great deal of literature on the subject, but for now we'll just consider a basic approach. Part of the difficulty of machine learning with text is turning language into numbers, since numbers are really all most machine learning algorithms understand (at heart anyway). One of the most straightforward ways to do this for documents is what is known as the ["bag-of-words" model](https://en.wikipedia.org/wiki/Bag-of-words_model). In this model we view a document as simply a multi-set of the words contained in it -- we completely ignore word order. The result can be viewed as a matrix of data by setting the feature space to be the set of all words that appear in any document, and a document is represented by a vector where the value of the *i*th entry is the number of times the *i*th word occurs in that document. This is a very common approach, and is what you will get if you apply sklearn's ``CountVectorizer`` to a text dataset for example. The catch with this approach is that the feature space is often *very* large, since we have a feature for each and every word that ever occurs in the entire corpus of documents. The data is sparse however, since most documents only use a small portion of the total possible vocabulary. Thus the default output format of ``CountVectorizer`` (and other similar feature extraction tools in sklearn) is a ``scipy.sparse`` format matrix.

For this example we'll make use of the classic 20newsgroups dataset, a sampling of newsgroup messages from the old NNTP newsgroup system covering 20 different newsgroups. The ``sklearn.datasets`` module can easily fetch the data, and, in fact, we can fetch a pre-vectorized version to save us the trouble of running ``CountVectorizer`` ourselves. We'll grab both the training set, and the test set for later use.

```python
news_train = sklearn.datasets.fetch_20newsgroups_vectorized(subset='train')
news_test = sklearn.datasets.fetch_20newsgroups_vectorized(subset='test')
```

If we look at the actual data we have pulled back, we'll see that sklearn has run a ``CountVectorizer`` and produced the data is sparse matrix format.

```python
news_train.data
```

    <11314x130107 sparse matrix of type '<class 'numpy.float64'>'
    	with 1787565 stored elements in Compressed Sparse Row format>

The value of the sparse matrix format is immediately obvious in this case; while there are only 11,000 samples there are 130,000 features! If the data were stored in a standard ``numpy`` array we would be using up 10GB of memory! And most of that memory would simply be storing the number zero, over and over again. In sparse matrix format it easily fits in memory on most machines. This sort of dimensionality of data is very common with text workloads.

The raw counts are, however, not ideal since common words the "the" and "and" will dominate the counts for most documents, while contributing very little information about the actual content of the document. We can correct for this by using a ``TfidfTransformer`` from sklearn, which will convert the data into [TF-IDF format](https://en.wikipedia.org/wiki/Tf%E2%80%93idf). There are lots of ways to think about the transformation done by TF-IDF, but I like to think of it intuitively as follows. The information content of a word can be thought of as (roughly) proportional to the negative log of the frequency of the word; the more often a word is used, the less information it tends to carry, and infrequent words carry more information. What TF-IDF is going to do can be thought of as akin to re-weighting the columns according to the information content of the word associated to that column. Thus the common words like "the" and "and" will get down-weighted, as carrying less information about the document, while infrequent words will be deemed more imporant and have their associated columns up-weighted. We can apply this transformation to both the train and test sets (using the same transformer trained on the training set).

```python
tfidf = sklearn.feature_extraction.text.TfidfTransformer(norm='l1').fit(news_train.data)
train_data = tfidf.transform(news_train.data)
test_data = tfidf.transform(news_test.data)
```

The result is still a sparse matrix, since TF-IDF doesn't change the zero elements at all, nor the number of features.

```python
train_data
```

    <11314x130107 sparse matrix of type '<class 'numpy.float64'>'
    	with 1787565 stored elements in Compressed Sparse Row format>

Now we need to pass this very high dimensional data to UMAP. Unlike some other non-linear dimension reduction techniques we don't need to apply PCA first to get the data down to a reasonable dimensionality; nor do we need to use other techniques to reduce the data to be able to be represented as a dense ``numpy`` array; we can work directly on the 130,000 dimensional sparse matrix.

```python
%%time
mapper = umap.UMAP(metric='hellinger', random_state=42).fit(train_data)
```

    CPU times: user 8min 40s, sys: 3.07 s, total: 8min 44s
    Wall time: 8min 43s

Now we can plot the results, with labels according to the target variable of the data -- which newsgroup the posting was drawn from.

```python
umap.plot.points(mapper, labels=news_train.target)
```

We can see that even going directly from a 130,000 dimensional space down to only 2 dimensions UMAP has done a decent job of separating out many of the different newsgroups.

We can now attempt to add the test data to the same space using the ``transform`` method.

```python
test_embedding = mapper.transform(test_data)
```

While this is somewhat expensive computationally, it does work, and we can plot the end result:

```python
fig = plt.figure(figsize=(12,12))
ax = fig.add_subplot(111)
plt.scatter(test_embedding[:, 0], test_embedding[:, 1], s=1, c=news_test.target, cmap='Spectral')
ax.set(xticks=[], yticks=[]);
```

```python

```