# COMP 135 day04 Lab: Dirichlet distributions and the Dirichlet-Categorical model for discrete outcomes

## Outline

* **Part 1: Dirichlet Distribution visualization**
* * Examine several ways to visualize samples from a Dirichlet.
* * Exercises 1a-1e: Write code to draw samples from several possible Dirichlets.
* * Exercises 1f-1g: Create and interpret pairplot and triangle plot visuals of Dirichlet distributions

* **Part 2: Dirichlet-Categorical Posteriors**
* * Exercise 2a: Write function to determine the posterior parameters after seeing many observations
* * Exercise 2b: Show the posterior after seeing 1, 10, 100, 1000, 10000 samples, with a uniform prior
* * Exercise 2d: Show the posterior after seeing 1, 10, 100, 1000, 10000 samples, with a different (misinformed) prior

## Takeaways

* To draw samples:
* * [`scipy.stats.dirichlet(hyperparameters).rvs(size=(nrows,ncols), random_state=SEED)`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.dirichlet.html)
* * OR [`np.random.dirichlet(hyperparameters)`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.dirichlet.html)
* To evaluate the PDF:
* * Evaluate pdf with `scipy.stats.<distribution>(hyperparameters).pdf(values)` (can use the `logpdf` instead of `pdf`)

* Visualizing V-ary dirichlets (with 3 categories) can be done with a triangle view (since the actual space of allowed parameters has 2 effective dimensions on one plane)

```python
import numpy as np
import pandas as pd
import scipy.stats
```

```python
np.set_printoptions(precision=3, suppress=False)
```

```python
pd.options.display.float_format = '{:,.3g}'.format  # show 4 digits of precision
```

```python
import matplotlib.pyplot as plt
import seaborn as sns
sns.set_style("whitegrid")
sns.set_context("notebook", font_scale=1.25)
```

```python

```

# Part 1: Dirichlet Distribution, Visualized

## Dirichlet distribution background

Suppose we have $V$ possible categories, known in advance.

We want a distribution over possible *V-dimensional probability vectors*

We can think of this as a "distribution over distributions".

#### Random Variable: Probability vector $\mu \in \Delta^{V}$

This is a vector of *non-negative entries that sum to one*:

$$
\mathbf{\mu} = [\mu_1, \mu_2, \ldots \mu_V ]
$$

where $\mu_v \geq 0$ and $\sum_{v=1}^V \mu_v = 1$

#### Parameters: Pseudocount vector $\alpha \in \mathbb{R}_+^V$

This is a vector of all positive numbers:

$$
\mathbf{\alpha} = [\alpha_1, \alpha_2, \ldots \alpha_V ]
$$

Where $\alpha_1 > 0$, $\alpha_2 > 0$, etc.

#### Formal definition of PDF:

$$
\text{DirPDF}\left( [\mu_1 \ldots \mu_V] \mid [\alpha_1 \ldots \alpha_V] \right)
    = c(\alpha) \cdot \prod_{v=1}^V \mu_v^{\alpha_v-1}
$$

with normalizing "constant" (a term constant wrt $\mu$):

$$
c(\alpha) = \frac
    {\Gamma( \sum_{v=1}^V \alpha_v )}
    {\prod_{v=1}^V \Gamma(\alpha_v)}
$$

#### Standard Numerical implementations

You can use the built-in functions from SciPy to sample from this distribution:

* To draw samples:
* * [`scipy.stats.dirichlet(hyperparameters).rvs(size=(nrows,ncols), random_state=SEED)`](https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.dirichlet.html)
* * OR [`np.random.dirichlet(hyperparameters)`](https://numpy.org/doc/stable/reference/random/generated/numpy.random.dirichlet.html)

See <https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.dirichlet.html>

Set our random state so things are reproducible.

```python
prng = np.random.RandomState(0)
```

## Drawing samples with V=2

The code below prints out vectors $\mu$ sampled from a Dirichlet with particular $\alpha$

#### First try with symmetric parameter $\alpha = [1, 1]$

```python
V = 2

ones_alpha_V = np.ones(V)
```

```python
prng.dirichlet(ones_alpha_V)
```

    array([0.388, 0.612])

```python
prng.dirichlet(ones_alpha_V)
```

    array([0.54, 0.46])

```python
prng.dirichlet(ones_alpha_V)
```

    array([0.347, 0.653])

#### Now try with symmetric parameter $\alpha = [0.1, 0.1]$

```python
point1_alpha_V = 0.1 * np.ones(V)
```

```python
prng.dirichlet(point1_alpha_V)
```

    array([2.542e-04, 9.997e-01])

```python
prng.dirichlet(point1_alpha_V)
```

    array([0.965, 0.035])

```python
prng.dirichlet(point1_alpha_V)
```

    array([1.000e+00, 3.489e-06])

## Discussion 1a: What do you notice when you compare the $\alpha = [1, 1]$ to $\alpha = [0.1, 0.1]$?

TODO discuss

## Discussion 1b: What do you expect to happen for symmetric $\alpha = [0.001, 0.001]$? What about symmetric $\alpha = [33, 33]$?

TODO discuss

## Exercise 1c: Write code to draw 3 samples from $\alpha = [0.001, 0.001]$ with V=2

TODO write code (hint: see lines above)

## Exercise 1d: Write code to draw 3 samples from $\alpha = [33, 33, 33]$ with V=3

```python
# TODO write code (hint: see lines above for how to call .dirichlet() )
```

## Exercise 1e: Write code to draw 3 samples from $\alpha = [70, 20, 10]$ with V=3

```python
# TODO write code (hint: see lines above)
```

```python

```

# Setup Visualization: Plotting V=3 samples on the 'triangle'

This code below just defines a function to help us visualize a provided array of samples of vector mu.

Don't worry too much about understanding the details, so feel free to skip ahead.

```python
def show_samples_on_V3_triangle(mu_S3=None, ax_handle=None, transparency=1.0):
    ''' Show samples from a Dirichlet on the triangle view of the V=3 simplex

    Create a triangle where each corner represents one extreme one-hot mu value
    Then plot each sampled mu vector (in V=3 dim space) as a point on this triangle

    Args
    ----
    mu_S3 : None or 2D array, shape (n_samples, n_categories) = (S, V)
        Provide many samples of 3-dimensional probability vectors
        Each row of this array should sum to one.
        Each sample vector will be visualized as a point on the triangle.
        If None, just plot the triangle with no samples.
    ax_handle : None or matplotlib axis handle
        If an axis handle is provided, will draw triangle in this handle
        If None, will create a new figure
    transparency : float, between 0.0 and 1.0
        Specifies the transparency of the samples.
        When number of samples is small, defaults to 1.0 so each sample easy to see.
        When number of samples is large, useful to set small (like 0.05) so density is visible.

    References
    ----------
    Many thanks to code here that inspired this
    http://y-okamoto-psy1949.la.coocan.jp/Python/en1/Dirichlet/
    '''
    # Create new figure if needed
    if ax_handle is None:
        plt.figure(figsize = (5, 5))
        ax_handle = plt.gca()
    else:
        plt.sca(ax_handle)

    # Create 3 vertices of an equilateral triangle
    P1_12 = np.asarray([0.0, (3.0**0.5) - 1.0]).reshape((1,2))
    P2_12 = np.array([-1.0, -1.0]).reshape((1,2))
    P3_12 = np.array([1.0, -1.0]).reshape((1,2))
    corners_32 = np.vstack([P1_12, P2_12, P3_12])

    # Draw the triangle
    # be sure to complete outline by going back to first point
    corners_42 = np.vstack([corners_32, P1_12])
    ax_handle.plot(corners_42[:,0], corners_42[:,1])

    # Label each vertex of the triangle
    B = 0.2
    ax_handle.text(P1_12[0,0] + B/2, P1_12[0,1] - B/3,
                   '$\mu = [1, 0, 0]$', fontsize = 12)
    ax_handle.text(P2_12[0,0], P2_12[0,1] - B/2,
                   '$\mu = [0, 1, 0]$', fontsize = 12)
    ax_handle.text(P3_12[0,0] - 3*B, P3_12[0,1] - B/2,
                   '$\mu = [0, 0, 1]$', fontsize = 12)

    # Make plot pretty
    ax_handle.set_yticks([]);
    ax_handle.set_xticks([]);
    sns.despine(top=True, right=True, left=True, bottom=True);
    ax_handle.set_aspect('equal');

    # Show provided sample points as points within the triangle
    if mu_S3 is not None:
        assert mu_S3.ndim == 2
        S, V = mu_S3.shape
        assert V == 3

        points_S2 = np.dot(mu_S3, corners_32)
        ax_handle.plot(points_S2[:,0], points_S2[:,1],
                       '.', color='b', linewidth=0, alpha=transparency)
```

# Show empty triangle visual

The triangle's corners represent the 3 "extreme" possible values of $\mu$

```python
show_samples_on_V3_triangle()
```

# Show an interior value of mu

We show a particular $\mu$ vector on the triangle plot, marked by the blue dot

```python
mu_1V = np.asarray([[0.05, 0.15, 0.8]])

print("Plotting mu_1V")
print("-------------")
print(mu_1V)

show_samples_on_V3_triangle(mu_1V)
```

## Discussion 1f: Does the location of the plotted mu as a dot within the triangle make sense?

TODO discuss

## Example 1(i): Generate Samples from Dirichlet(alpha = [70, 20, 10])

Let's generate many samples from a specific Dirichlet distribution, and practice visualizing it

```python
V = 3
alpha_V = np.asarray([70, 20, 10])
```

```python
# First, make a 2D array that is (n_samples, n_categories) = (S, V)
S = 10000
mu_SV = np.zeros((S, V))

# Fill in this array with samples from a dirichlet over V categories
for s in range(S):
    mu_SV[s] = prng.dirichlet(alpha_V)

# Convert to a pandas data frame (easy to use with seaborn visualization library)
mu_df = pd.DataFrame(mu_SV, columns=['$\mu_1$', '$\mu_2$', '$\mu_3$'])

# Print out the first 5 rows
mu_df.head()
```

      $\mu_1$
      $\mu_2$
      $\mu_3$

      0
      0.669
      0.193
      0.138

      1
      0.754
      0.115
      0.131

      2
      0.756
      0.116
      0.128

      3
      0.8
      0.155
      0.0457

      4
      0.714
      0.226
      0.06

## Pair plot for samples from Dirichlet( alpha = [70, 20, 10] )

```python
fig_handle = sns.pairplot(
    mu_df,
    aspect=1.0,
    corner=True, # only lower triangle
    diag_kws={"color": 'b', "alpha": 0.8, "bins":np.linspace(0, 1, 21)},
    plot_kws={"color": 'b', "alpha": 0.05, "s": 3, 'marker': '.', 'edgecolor':'none'})
# ^- alpha here means transparency for the 'alpha' channel of the RGB-A scheme

B = 0.003
fig_handle.set(
    xlim=(-B, 1+B), xticks=[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
    ylim = (-B, 1+B), yticks=[0.0, 0.2, 0.4, 0.6, 0.8, 1.0]);
plt.suptitle("alpha = %s" % str(alpha_V));
plt.tight_layout(); # improve spread and visibility of subplots

```

## Triangle plot for samples from Dir( alpha = [70, 20, 10] )

```python
show_samples_on_V3_triangle(mu_df.values, transparency=0.05)
```

## Example 1(ii): Generate Samples alpha = [0.9, 0.5, 0.1]

```python
V = 3
alpha_V = np.asarray([0.9, 0.5, 0.1])
```

```python
# First, make a 2D array that is (n_samples, n_categories) = (S, V)
S = 10000
mu_SV = np.zeros((S, V))

# Fill in this array with samples from a dirichlet over V categories
for s in range(S):
    mu_SV[s] = prng.dirichlet(alpha_V)

mu_df = pd.DataFrame(mu_SV, columns=['$\mu_1$', '$\mu_2$', '$\mu_3$'])
mu_df.head()
```

      $\mu_1$
      $\mu_2$
      $\mu_3$

      0
      0.918
      0.0816
      0.00018

      1
      0.998
      0.00196
      1.78e-07

      2
      0.326
      0.666
      0.00768

      3
      0.906
      0.0938
      1.1e-17

      4
      0.931
      0.0693
      5.68e-05

## Pair plot visual with alpha = [0.9, 0.5, 0.1]

```python
fig_handle = sns.pairplot(
    mu_df,
    aspect=1.0,
    corner=True, # only lower triangle
    diag_kws={"color": 'b', "alpha": 0.8, "bins":np.linspace(0, 1, 21)},
    plot_kws={"color": 'b', "alpha": 0.05, "size": 3, 'marker': '.', 'edgecolor':'none'})
# ^- alpha here means transparency for the 'alpha' channel of the RGB-A scheme

B = 0.003
fig_handle.set(
    xlim=(-B, 1+B), xticks=[0.0, 0.2, 0.4, 0.6, 0.8, 1.0],
    ylim = (-B, 1+B), yticks=[0.0, 0.2, 0.4, 0.6, 0.8, 1.0]);
plt.suptitle("alpha = %s" % str(alpha_V));
plt.tight_layout(); # improve spread and visibility of subplots

```

# Triangle visualization of alpha = [0.9, 0.5, 0.1]

```python
show_samples_on_V3_triangle(mu_df.values, transparency=0.05)
```

## Exercise 1g: Create visuals for alpha = [333, 33, 3]

```python
# TODO write code to sample mu
```

```python
# TODO write code to make pairplot
```

```python
# TODO write code to make triangle plot
```

```python

```

```python

```

# Part 2: Posterior visualization in the Dir-Cat model

We'll assume the Dir-Cat model discussed in class

Our random variables are:

\begin{align}
\text{probability vector} \quad & \mu \in \Delta^V
\\
\text{observed category indicators} \quad & x_n \in \{1, \ldots V\}
\end{align}

\begin{align}
p(\mu) &= \text{Dir}( \mu | \alpha)
\\
p(x_1, \ldots x_N | \mu) &= \prod_{n=1}^N \text{CatPMF}( x_n | \mu)
\end{align}

We are interested in the posterior:

$$
p( \mu | x_1, \ldots x_N)
$$

## Exercise 2a: Calculating posterior parameters

Using the formulas discussed in the textbook / video lecture, complete the function below to determine the parameters of the Dirichlet posterior distribution after seeing N one-hot encoded outcomes in array `x_NV`

Resource: page 8-9 of the day04 notes: https://www.cs.tufts.edu/comp/136/2021s/notes/day04.pdf#page=08

**TODO** edit the function below.

```python
def calc_dirichlet_posterior_parameters(x_NV, alpha_V):
    ''' Compute Dirichlet posterior parameters

    Args
    ----
    x_NV : 2D array, shape (n_examples, n_categories) = (N, V)
        Each row is a one-hot vector of size V indicating a single categorical outcome
    alpha_V : 1D array, shape (n_categories,) = (V,)
        Prior parameter vector, "pseudocounts" for each category
        All entries must be positive

    Returns
    -------
    alphahat_V : 1D array, shape (n_categories,) = (V,)
        Posterior parameter vector, "pseudocounts" for each category
        All entries must be positive
    '''
    alphahat_V = alpha_V # TODO fix me
    return alphahat_V
```

## Draw 'fake' categorical data X  from model given parameter mu

```python
# Define a random generator object, with specific seed for reproducibility
prng = np.random.RandomState(101)
```

```python
# Pick a "true" probability vector: 30%, 50%, 20%
true_mu_V = np.asarray([0.3, 0.5, 0.2])

print("true_mu_V")
print("---------")
print(true_mu_V)
```

    true_mu_V
    ---------
    [0.3 0.5 0.2]

```python
# Settings
N = 10000                 # num data examples to observe

# Sample N examples from Multinomial with 1 trial (aka Categorical) given mu
x_NV = scipy.stats.multinomial(1, true_mu_V).rvs(size=N, random_state=prng)

print("x_NV with shape " + str(x_NV.shape))
print("---------")
print(x_NV[:5])
print("...")
```

    x_NV with shape (10000, 3)
    ---------
    [[0 1 0]
     [0 1 0]
     [0 0 1]
     [0 0 1]
     [1 0 0]]
    ...

## Exercise 2b: Visualizing Posterior given a *uniform* prior

Complete the code below, to visualize the posterior over mu for our "fake data" `x_N`

```python
alpha_V = np.ones(3)
```

```python
print("Assuming prior is Dir with parameter")
print("alpha_V")
print("-------")
print(alpha_V)
```

    Assuming prior is Dir with parameter
    alpha_V
    -------
    [1. 1. 1.]

```python
Npartial_list = [0, 10, 100, 1000, 10000]
L = len(Npartial_list)

fig, axgrid = plt.subplots(
    nrows=1, ncols=L,
    sharex=True, sharey=False,
    figsize=(5 * L, 5))

# Define a grid of mu values at which we'll evaluate the PDF
G = 201
mu_grid_G = np.linspace(0.0001, 0.9999, G)
for aa, Npartial in enumerate(Npartial_list):

    # Compute the parameters of the Dirichlet posterior distribiution, using only first Npartial observations
    alphahat_V = calc_dirichlet_posterior_parameters(x_NV[:Npartial], alpha_V)

    # Show S samples from the posterior
    S = 1000
    # TODO draw samples from your Dirichlet posterior with parameters alphahat_V
    # musamp_SV will collect all your S samples, each one a V-dim array
    # For now, we'll just fill it with all [1/3, 1/3, 1/3] so this code runs
    musamp_SV = np.ones((S, V)) / 3

    show_samples_on_V3_triangle(musamp_SV, ax_handle=axgrid[aa], transparency=0.05)
    axgrid[aa].set_title("after N = %d" % Npartial, fontsize=24)

plt.tight_layout();
```

## Discussion 2c : As N gets larger, what happens to the posterior? How does this compare to what happens to the ML estimator? How does this relate to the "true" mu value that generated the data?

TODO discuss with your group!

## Exercise 2d: Visualizing Posterior given a misinformed prior

Repeat Exercise 2b using the *same* fake data, but act as if your prior was instead

$$
alpha_V = [1, 9, 99]
$$

(in other words, quite confident in prior belief that the outcome would be in the third category.

Visualize the posterior over mu for the same "fake data" `x_NV`, after seeing each of the Npartial examples as in 2b.

```python
alpha_V = np.asarray([1, 9, 99.0])
```

```python
print("Assuming prior is Dir with parameter")
print("alpha_V")
print("-------")
print(alpha_V)
```

    Assuming prior is Dir with parameter
    alpha_V
    -------
    [ 1.  9. 99.]

```python
# TODO modify code from 2b, to make triangle plots for N = 0, 10, 100, ... 10000
```

## Discussion 2e: Does having a misinformed prior matter? Discuss how your answer might depend on the amount of available data.

```python
# TODO discuss
```

## Exercise 2f: Repeat 2d, but for each plot show the location of the *MAP* estimate for $\mu$ on the triangle plot.

```python
# TODO this is a bonus
```

```python

```


```python
import sympy as sym
from sympy.vector import CoordSys3D
from sympy import Matrix, powdenest
from sympy.physics.mechanics import dynamicsymbols, init_vprinting
from sympy.solvers.ode.systems import dsolve_system
init_vprinting(use_latex=True)
```

```python
# u = sym.Symbol('u')
# v = sym.Symbol('v')
u, v = dynamicsymbols('u v')
ref = CoordSys3D('ref')

# sigma = (sym.cos(u)*sym.cos(v), sym.cos(u)*sym.sin(v), sym.sin(u))
# sigma = sym.cos(u)*sym.cos(v)*ref.i + sym.cos(u)*sym.sin(v)*ref.j + sym.sin(u)*ref.k
# sigma = u*ref.i + v*ref.j + u*v*ref.k
sigma = (u**3 + 1)*sym.cos(v)*ref.i + (u**3 + 1)*sym.sin(v)*ref.j + u*ref.k
# print(sym.diff(sigma, u))
# sym.diff(sigma, u).to_matrix(ref)
sigma
```

```python
def tangent_plane_basis(sigma):
    return sym.diff(sigma, u).simplify(), sym.diff(sigma, v).simplify()

def first_fundamental_form(sigma):
    sigma_u, sigma_v = tangent_plane_basis(sigma)
    E = sigma_u.dot(sigma_u).simplify()
    F = sigma_u.dot(sigma_v).simplify()
    G = sigma_v.dot(sigma_v).simplify()
    I = Matrix([[E, F], [F, G]])
    return I

def normal_vector(sigma):
    sigma_u, sigma_v = tangent_plane_basis(sigma)
    N = sigma_u.cross(sigma_v).simplify().normalize().simplify()
    return N

def second_fundamental_form(sigma):
    sigma_u, sigma_v = tangent_plane_basis(sigma)
    sigma_uu = sym.diff(sigma_u, u).simplify()
    sigma_uv = sym.diff(sigma_u, v).simplify()
    assert sym.diff(sigma_v, u).simplify() == sigma_uv
    sigma_vv = sym.diff(sigma_v, v).simplify()
    N = normal_vector(sigma)
    e = sigma_uu.dot(N).simplify()
    f = sigma_uv.dot(N).simplify()
    g = sigma_vv.dot(N).simplify()
    II = Matrix([[e, f], [f, g]])
    return II

def christoffel_symbols(sigma):
    I = first_fundamental_form(sigma)
    E_u, E_v = sym.diff(I[0,0], u).simplify(), sym.diff(I[0,0], v).simplify()
    F_u, F_v = sym.diff(I[0,1], u).simplify(), sym.diff(I[0,1], v).simplify()
    G_u, G_v = sym.diff(I[1,1], u).simplify(), sym.diff(I[1,1], v).simplify()
    system_intercepts = Matrix(
        [
            [E_u/2, E_v/2, F_v - G_u/2],
            [F_u - E_v/2, G_u/2, G_v/2]
        ]
    )
    result = ((I**(-1))*system_intercepts)
    return result

def gauss_equations(sigma):
    I  = first_fundamental_form(sigma)
    # II = second_fundamental_form(sigma)
    # K  = (II.det()/I.det()).simplify()
    E, F, G = I[0,0], I[0,1], I[1,1]
    # E, F, G, K = sym.symbols('E,F,G,K')
    K = sym.Symbol('K')
    ch = christoffel_symbols(sigma)

    eq1 = sym.Eq(E*K, sym.diff(ch[0,0], v) - sym.diff(ch[1,1], u) + ch[0,0]*ch[1,1] - ch[0,1]*ch[1,0] + ch[1,0]*ch[1,2] - ch[1,1]*ch[1,1])
    eq2 = sym.Eq(F*K, sym.diff(ch[0,1], u) - sym.diff(ch[0,0], v) + ch[1,1]*ch[0,1] - ch[1,0]*ch[0,2])
    eq3 = sym.Eq(G*K, sym.diff(ch[0,2], u) - sym.diff(ch[0,1], v) + ch[0,0]*ch[0,2] + ch[0,1]*ch[1,2] - ch[0,1]*ch[0,1] - ch[1,1]*ch[0,2])
    return eq1, eq2, eq3

def gauss_curvature(sigma, mode='extrinsic'):
    if mode == 'extrinsic':
        I  = first_fundamental_form(sigma)
        II = second_fundamental_form(sigma)
        K  = (II.det()/I.det()).simplify()
        K_sym = sym.Symbol('K')
        return sym.Eq(K_sym, K)
    elif mode == 'intrinsic':
        eqns = gauss_equations(sigma)
        K_sym = sym.Symbol('K')
        for eq in eqns:
            if len(eq.find(0)) == 0:
                sol = sym.solve(eq, K_sym)
                return sym.Eq(K_sym, sol[0])

def geodesics_equations(sigma):
    ch = christoffel_symbols(sigma)
    return sym.Eq(sym.diff(sym.diff(u)) + \
        (Matrix([sym.diff(u), sym.diff(v)]).T * Matrix([[ch[0,0], ch[0,1]], [ch[0,1], ch[0,2]]]) * Matrix([sym.diff(u), sym.diff(v)]))[0].simplify(), 0), \
           sym.Eq(sym.diff(sym.diff(v)) + \
        (Matrix([sym.diff(u), sym.diff(v)]).T * Matrix([[ch[1,0], ch[1,1]], [ch[1,1], ch[1,2]]]) * Matrix([sym.diff(u), sym.diff(v)]))[0].simplify(), 0)

def find_geodesics(sigma):
    eqns = geodesics_equations(sigma)
    geo = dsolve_system(eqns)
    # geo1 = sym.dsolve(eqns[1])
    return geo

def covariant_derivative(sigma, field, dir):
    # primer calculem la derivada "total"
    sigma_u, sigma_v = tangent_plane_basis(sigma)
    total_derivative_along_dir = ...
    # llavors li restem la component normal
    N = normal_vector(sigma)
    covariant_derivative = total_derivative_along_dir - total_derivative_along_dir.dot(N).simplify()*N
    # et voilà! tenim la derivada covariant
    covariant_derivative = covariant_derivative.simplify()
    return covariant_derivative
```

```python
first_fundamental_form(sigma)
```

$\displaystyle \left[\begin{matrix}9 u^{4} + 1 & 0\\0 & \left(u^{3} + 1\right)^{2}\end{matrix}\right]$

```python
second_fundamental_form(sigma)
```

$\displaystyle \left[\begin{matrix}- \frac{6 \left(u^{3} + 1\right) u}{\sqrt{\left(u^{3} + 1\right)^{2} \left(9 u^{4} + 1\right)}} & 0\\0 & \frac{\left(u^{3} + 1\right)^{2}}{\sqrt{\left(u^{3} + 1\right)^{2} \left(9 u^{4} + 1\right)}}\end{matrix}\right]$

```python
sym.factor(christoffel_symbols(sigma)).cancel()
```

$\displaystyle \left[\begin{matrix}\frac{18 u^{3}}{9 u^{4} + 1} & 0 & \frac{- 3 u^{5} - 3 u^{2}}{9 u^{4} + 1}\\0 & \frac{3 u^{2}}{u^{3} + 1} & 0\end{matrix}\right]$

```python
geodesics_equations(sigma)
```

```python
sym.factor(gauss_curvature(sigma, mode='intrinsic'))
```

```python
sym.factor(gauss_curvature(sigma, mode='extrinsic'))
```

```python
sym.cancel(sym.factor(gauss_equations(sigma)))
```

$\displaystyle \left( K \left(9 u^{4} + 1\right) = - \frac{6 u}{\left(u + 1\right) \left(9 u^{4} + 1\right) \left(u^{2} - u + 1\right)}, \  \text{True}, \  K \left(u + 1\right)^{2} \left(u^{2} - u + 1\right)^{2} = - \frac{6 \left(u + 1\right) \left(u^{2} - u + 1\right) u}{\left(9 u^{4} + 1\right)^{2}}\right)$

```python
# find_geodesics(sigma) # Does not work!
```


# Gaussian Process Regression

________
###  Table of content

[1. Regression model](#model)

[2. Prediction](#Prediction)

[3. Finding the best kernel parameters](#kernel)

_________

**Dataset :**

Let $(X_i)_{i\in[\![1,n]\!]}$ be i.i.d. random variables in $\mathbb{R}^d$ and consider the matrix $X \in \mathbb{R}^{n . d}$ such that the i-th row of $X$ is the observation $X_i^T$. For all $1 \leq i \leq n$, $X_i$ is an individual which has been associated with the label $Y_i \in \mathbb{R}$, and consider the matrix $Y = [Y_1, Y_2, ..., Y_n]^T \in \mathbb{R}^n$.

## 1. Regression model

In the Gaussian process regression model, $Y = f(X) + \epsilon$.

The $(\epsilon_i)_{i\in[\![1,n]\!]} \in \mathbb{R}^n$ are the i.i.d. noise variables with independant normal distributions, so that $\epsilon = [\epsilon_1, \epsilon_2, ... \epsilon_n]^T \sim N(0, \sigma ^2 I_n)$.
$ f$ is a Gaussian Process, i.e. an n-dimensional vector, defined by :

$$
f(.) \sim \mathcal{GP}(0, k_{\gamma}(. , .))
$$

where $k_{\gamma}(. , .)$ is a valid covariance function and $\gamma$ is the parameter to optimize.

A commonly used kernel function is the squared exponential or radial basis function (RBF) kernel, defined as follows:

$$k_\gamma (z, z') = \exp(-\frac{\parallel z - z' \parallel ^2}{2 \gamma^2})$$

## 2. Prediction

Gaussian Process Regression is a nonparametric model. Therein, prediction will be directly performed using the conditionnal gaussian distribution.

Given a dataset of observed outputs $ \lbrace (x_i, y_i) \rbrace_{1 \leq i \leq n}$, we want to predict the output $Y_{test}$ of a test set $X_{test}$ drawn from the same distribution.

With $Y_0 = [Y_{test}, Y]^T$, $X_0 = [X_{test}, X]^T$ and $\epsilon_0 = [0, \epsilon]^T$, the model is thus :

$$Y_0 = f(X_0) + \epsilon_0 \sim N(0, K_0 + \sigma^2I_n)$$

where
$$K_0 = \begin{pmatrix}
          K_{aa} & K_{ab} \\
          K_{ba} & K_{bb} \\
         \end{pmatrix}$$ and
$$\begin{equation}
    \begin{cases}
      K_{aa} = (k_\gamma(X_i, X_j))\\
      K_{ab} = (k_\gamma(X_i, X_{test, j}))\\
      K_{ba} = (k_\gamma(X_{test, i}, X_j))\\
      K_{bb} = (k_\gamma(X_{test, i}, X_{test, j}))
    \end{cases}
\end{equation}$$

We can then compute the conditional distribution $(Y_{test}|X_{test}, X, Y) \sim \mathcal{N}(m, D)$ by using the conditional Gaussian distribution formulas :

$$\begin{equation}
    \begin{cases}
      m = K_{ab}K_{bb}^{-1}Y\\
      D = (K_{aa} + \sigma^2I_n) - K_{ab}K_{bb}^{-1}K_{ba}
    \end{cases}
\end{equation}$$

## 3. Finding the best kernel parameters

With $X$, $Y$ and $\epsilon$ as previously defined, let's apply the model to the dataset : $Y = f(X) + \epsilon$.

As f(.) is a Gaussian Process, $f(X) \sim N(0, K_\gamma)$ where $K_\gamma = (k_\gamma (X_i, X_j))_{i,j \in [\![1,n]\!]}$, and since $\epsilon \sim N(0, \sigma ^2 I_n)$, it implies that $ (Y | X; \gamma) \sim N(0, K_\gamma + \sigma ^2 I_n)$.

Consequently, the probability distribution is:
$$P(Y=y | X; \gamma) = \frac{1}{(2\pi)^{n/2} det(K_\gamma + \sigma^2 I_n)^{1/2}}exp(-\frac{1}{2} y^T (K_\gamma + \sigma^2 I_n)^{-1} y)$$

The likelihood of the model, given the observed data, is defined as:
$$L(\gamma) = P(Y=y | X; \gamma)$$

The aim of the training is to find the parameters $\gamma$ which maximizes the likelihood function (Maximum Likelihood Estimation), which is the same as minimizing the negative log-likelihood. To unify with the neural network architecture used later, we chose to minimize the negative log-likelihood :
$$l(\gamma) = - \log (P(Y=y | X; \gamma))$$

The resulting equation is thus:
$$l(\gamma) = \frac{1}{2} ( n\log(2\pi) + \log det(K_\gamma + \sigma^2 I_n) + y^T (K_\gamma + \sigma^2 I_n)^{-1} y )$$

To simplify this, we can rewrite $K_\gamma + \sigma ^2 I_n$ as $K_\gamma$.

Again, to unify this with the neural network approach (based on the chain rule), we need to compute the derivative of the negative log-likelihood with respect to $K_\gamma$ :
$$\frac {\partial l}{\partial K_\gamma} = \frac{1}{2} ( \frac {\partial n\log(2\pi)}{\partial K_\gamma} + \frac {\partial \log det K_\gamma }{\partial K_\gamma} + \frac {\partial y^T K_\gamma^{-1} y )}{\partial K_\gamma})$$

Pre-requisites :
$$\begin{equation}
    \begin{cases}
      \frac {\partial \log \det A}{\partial A} = A^{-1}\\
      \frac {\partial A^{-1}}{\partial A} = -(A^{-1}) (A^{-1})\\
    \end{cases}
\end{equation}$$

Consequently :
$$\begin{equation}
    \begin{cases}
      \frac {\partial n\log(2\pi)}{\partial K_\gamma} = 0\\
      \frac {\partial \log det K_\gamma}{\partial K_\gamma} = K_\gamma ^{-1}\\
      \frac {\partial y^T K_\gamma^{-1} y}{\partial K_\gamma} = - K_\gamma^{-1} y y^T K_\gamma^{-1}
    \end{cases}
\end{equation}$$

> PROOF for $\frac {\partial y^T K_\gamma^{-1} y}{\partial K_\gamma} = - K_\gamma^{-1} y y^T K_\gamma^{-1}$ :
>
> As $y^T K_\gamma^{-1} y = \sum_{i,j} y_i y_j (K_\gamma^{-1})_{i,j}$,
> $$\frac {\partial y^T K^{-1} y}{\partial K_{k,l}} = \sum_{i,j} y_i y_j (\frac {\partial K^{-1}} {\partial K_{k,l}})_{i,j}$$
>
> Given that $\frac {\partial K^{-1}} {\partial K_{k,l}} =  - K^{-1} \frac {\partial K}{\partial K_{k,l}} K^{-1}$  and that the derivative with respect to the coordinate (k,l) is equal to a (n,n) null matrix with a 1 at position (k,l),  it ends up with the following equation:
> $$\frac {\partial K^{-1}} {\partial K_{k,l}} =  - K^{-1} \frac {\partial K}{\partial K_{k,l}} K^{-1} = - [(K^{-1}_{ik} K^{-1}_{lj})_{i,j}]$$
> Consequently :
$$(\frac {\partial K^{-1}} {\partial K_{k,l}})_{i,j} = - K^{-1}_{ik} K^{-1}_{lj}$$
Finally, the derivative equals :
$$\frac {\partial y^T K^{-1} y}{\partial K_{k,l}} = \sum_{i,j} - y_i y_j K^{-1}_{ik} K^{-1}_{lj}.$$
>
> This may be written as :
> $$\frac {\partial y^T K^{-1} y}{\partial K_\gamma}  = - K_\gamma^{-1} y y^T K_\gamma^{-1}$$

Finally :
$$\frac {\partial l}{\partial K_\gamma} = \frac{1}{2} (K_\gamma ^{-1} - K_\gamma^{-1} y y^T K_\gamma^{-1})$$

In Deep Kernel Learning, we have $K_{\gamma, w} = (k_\gamma (h_w(X_i), h_w(X_j))_{i,j \in [|1,n|]})$ where $h_w(.)$ represents the Neural Network.

During the backpropagation, we use the chain rule to compute $\frac {\partial K_{\gamma, w}}{\partial \gamma}$ and $\frac {\partial K_{\gamma, w}}{\partial w}$ in the following way :

$$\begin{equation}
    \begin{cases}
      \frac {\partial l}{\partial \gamma} = \frac {\partial l}{\partial K_{\gamma, w}} \frac {\partial K_{\gamma, w}}{\partial \gamma}\\
      \frac {\partial l}{\partial w} = \frac {\partial l}{\partial K_{\gamma, w}} \frac {\partial K_{\gamma, w}}{\partial w}
    \end{cases}
\end{equation}$$

# Implementation

## Data

```python
import pandas as pd
from sklearn.model_selection import train_test_split
import numpy as np
import seaborn as sns
import matplotlib.pyplot as plt
from mpl_toolkits import mplot3d
```

```python
df_rwanda = pd.read_csv('./data/rwanda')
df_rwanda.head()
```

      Unnamed: 0
      cluster
      latitude
      longitude
      wealth_index
      mean_light
      median_light
      std_light
      min_light
      max_light

      0
      0
      1
      -2.532818
      29.684726
      -0.531405
      0.06
      0.0
      0.596992
      0.0
      6.0

      1
      1
      2
      -1.833858
      30.310689
      -0.409830
      0.00
      0.0
      0.000000
      0.0
      0.0

      2
      2
      3
      -1.888155
      29.478298
      -0.478115
      0.00
      0.0
      0.000000
      0.0
      0.0

      3
      3
      4
      -2.366763
      30.521692
      -0.435960
      0.00
      0.0
      0.000000
      0.0
      0.0

      4
      4
      5
      -2.171266
      30.018541
      -0.449480
      0.00
      0.0
      0.000000
      0.0
      0.0

```python
fig = plt.figure(figsize=(20, 18))
ax = fig.gca(projection='3d')
ax.view_init(-90, 0)
ax.set_facecolor('black')
X = df_rwanda['latitude']
Y = df_rwanda['longitude']
Z = df_rwanda['mean_light']
ax.scatter3D(X, Y, Z, c=Z, cmap='Reds', s=100, marker='+')
```

```python
fig = plt.figure(figsize=(20, 18))
ax = fig.gca(projection='3d')
ax.view_init(-90, 0)
ax.set_facecolor('black')
X = df_rwanda['latitude']
Y = df_rwanda['longitude']
Z = df_rwanda['wealth_index']
ax.scatter3D(X, Y, Z, c=Z, cmap='Greens', s=100, marker='+')
```

## Model

```python
from scipy.linalg import cholesky, cho_solve, inv, det, norm

class GaussianProcessRegression:

    def __init__(self, sigma=0.01, gamma=None):
        self.sigma = sigma
        self.gamma = gamma

        self.X = None
        self.y = None

        self.L = None
        self.alpha = None

    def kernel(self, x, y):
        return np.exp((-0.5 / self.gamma ** 2) * norm(x - y) ** 2)

    def loss(self):
        n, _ = np.shape(self.X)

        K_bb = None
        K_bb = [[self.kernel(self.X[i], self.X[j]) for i in range(n)] for j in range(n)]
        K_bb += self.sigma**2 * np.identity(n)
        K_bb = np.array(K_bb)
        K_bb_inv = inv(K_bb)

        l = n * np.log(2 * np.pi) + det(K_bb) + np.dot(self.y.T, K_bb_inv).dot(self.y)
        l *= 0.5

        tmp = [[(norm(self.X[i] - self.X[j]) ** 2) for i in range(n)] for j in range(n)]
        tmp = np.array(tmp)
        tmp /= np.abs(self.gamma ** 3)
        K_grad = np.dot(tmp, K_bb)
        dgamma = np.trace(np.dot(K_bb_inv, K_grad)) - np.dot(np.dot(self.y.T, K_bb_inv), K_grad).dot(np.dot(K_bb_inv, self.y))
        dgamma *= 0.5

        return l, dgamma

    def train(self, X_train, y_train):
        self.X = X_train
        self.y = y_train

        if self.gamma == None:
            # Choose hyperparameters based on maximizing the log-marginal likelihood
            learning_rate = 1e4
            self.gamma = np.random.randn()
            tmp = 1
            print('init :', self.gamma)
            #while np.abs(tmp) > 1e-3:
            for k in range(10):
                l, dgamma = self.loss()
                self.gamma = self.gamma - (learning_rate / dgamma[0][0] ** 2) * dgamma[0][0]
                tmp = (learning_rate / dgamma[0][0] ** 2) * dgamma[0][0]
                print(tmp)
                print('gamma :', self.gamma, 'dgamma', dgamma[0][0], 'loss :', l[0][0])

        # Precompute quantities required for predictions which are independent of actual query points
        n, _ = np.shape(self.X)

        K_bb = [[self.kernel(self.X[i], self.X[j]) for i in range(n)] for j in range(n)]
        K_bb += self.sigma**2 * np.identity(n)
        K_bb = np.array(K_bb)

        self.L = cholesky(K_bb, lower=True)
        self.alpha = cho_solve((self.L, True), self.y)
        #self.alpha = cho_solve((self.L.T, True), self.alpha)

    def predict(self, X_test):
        n, _ = np.shape(self.X)
        p, _ = np.shape(X_test)

        K_aa = [[self.kernel(X_test[i], X_test[j]) for j in range(p)] for i in range(p)]
        K_ab = [[self.kernel(X_test[i], self.X[j]) for j in range(n)] for i in range(p)]

        K_aa = np.array(K_aa)
        K_ab = np.array(K_ab)
        K_ba = K_ab.T

        v = cho_solve((self.L, True), K_ba)
        cov = (K_aa + self.sigma**2 * np.identity(p)) - np.dot(v.T, v)

        m_test = np.dot(K_ab, self.alpha)

        return m_test, cov
```

# Training

## without nightlight intensities

```python
X = df_rwanda[['latitude', 'longitude']]
y = df_rwanda['wealth_index']

X = np.resize(X, (X.shape[0], X.shape[1]))
y = np.resize(y, (y.shape[0], 1))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.20, random_state=42)
```

```python
model = GaussianProcessRegression(gamma=0.30)
model.train(X_train, y_train)

preds = model.predict(X)[0]
test_preds = model.predict(X_test)[0]
train_preds = model.predict(X_train)[0]

c1 = np.linalg.norm(y_test - test_preds) / y_test.shape[0]
c2 = np.linalg.norm(y_train - train_preds) / y_train.shape[0]
print('train cost :', c2)
print('test cost :', c1)
```

    train cost : 0.02167035547366247
    test cost : 0.05568564478405448

```python
lats = X[:, 0]
longs = X[:, 1]
preds = preds.reshape((X.shape[0],))

df_train = pd.DataFrame({'latitude':lats, 'longitude':longs, 'wealth_index_predicted':preds})
df_train.head()
```

      latitude
      longitude
      wealth_index_predicted

      0
      -2.532818
      29.684726
      -0.265882

      1
      -1.833858
      30.310689
      -0.448226

      2
      -1.888155
      29.478298
      -0.266708

      3
      -2.366763
      30.521692
      -0.440272

      4
      -2.171266
      30.018541
      -0.577882

```python
fig = plt.figure(figsize=(20, 18))
ax = fig.gca(projection='3d')
ax.view_init(-90, 0)
ax.set_facecolor('black')
X = df_train['latitude']
Y = df_train['longitude']
Z = df_train['wealth_index_predicted']
ax.scatter3D(X, Y, Z, c=Z, cmap='Greens', s=100, marker='+')
```

## without nightlight intensities

```python
X = df_rwanda[['latitude', 'longitude']]
y = df_rwanda['wealth_index']

X = np.resize(X, (X.shape[0], X.shape[1]))
y = np.resize(y, (y.shape[0], 1))

X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.40, random_state=42)
```

```python
model = GaussianProcessRegression(gamma=0.245)
model.train(X_train, y_train)

preds = model.predict(X)[0]
test_preds = model.predict(X_test)[0]
train_preds = model.predict(X_train)[0]

c1 = np.sqrt(np.linalg.norm(y_test - test_preds) ** 2 / y_test.shape[0])
c2 = np.sqrt(np.linalg.norm(y_train - train_preds) ** 2 / y_train.shape[0])
print('train cost :', c2)
print('test cost :', c1)
```

    train cost : 0.3884810252335164
    test cost : 0.5246826512985179

```python
lats = X[:, 0]
longs = X[:, 1]
preds = preds.reshape((X.shape[0],))

df_train = pd.DataFrame({'latitude':lats, 'longitude':longs, 'wealth_index_predicted':preds})
df_train.head()
```

      latitude
      longitude
      wealth_index_predicted

      0
      -2.532818
      29.684726
      -0.142270

      1
      -1.833858
      30.310689
      -0.465138

      2
      -1.888155
      29.478298
      -0.331629

      3
      -2.366763
      30.521692
      -0.423667

      4
      -2.171266
      30.018541
      -0.443740

```python
fig = plt.figure(figsize=(20, 18))
ax = fig.gca(projection='3d')
ax.view_init(-90, 0)
ax.set_facecolor('black')
X = df_train['latitude']
Y = df_train['longitude']
Z = df_train['wealth_index_predicted']
ax.scatter3D(X, Y, Z, c=Z, cmap='Greens', s=100, marker='+')
```

# Bibliographie

Gaussian Process theory : http://www.gaussianprocess.org/gpml/chapters/RW.pdf

```python

```


# Computer Simulation of Epidemics with the SIR Model

In this notebook we will explore a simple numerical simulation of how an epidemic spreads through a population. The model is called "SIR", because every individual in the population is assigned to one of three *compartments*:
\begin{align}
S &= \textrm{Susceptible} \\
I &= \textrm{Infected} \\
R &= \textrm{Removed (or Recovered)}
\end{align}

The code snippets in this notebook are written in the [Julia programming language](https://julialang.org/). You can run each cell by placing your cursor inside the cell and either clicking the "Run" button at the top of the page, or typing `Shift+Enter`. Make sure you run the code cells in order, as some of the later cells depend on the earlier ones.

## Numerical Models

Before we begin modelling the epidemic, let's look at a simpler numerical model for the motion of an object that is accelerating. The system has a number of properties of interest:

- $t$, the time (in seconds) since the object started moving
- $x$, the distance (in metres) of the object from its starting point
- $v$, the object's velocity
- $a$, the object's acceleration

These properties are related by the following system of equations:

\begin{align}
v &= \frac{dx}{dt} & \text{velocity is equal to the rate of change of distance with respect to time}\\
a &= \frac{dv}{dt} & \text{acceleration is equal to the rate of change of velocity with respect to time}\\
\text{or equivalently, }
a &= \frac{d^2x}{dt^2}
\end{align}

Given the four properties $[t,x,v,a]$, we can work out what the system will look like at some later time $(t+\Delta t)$ by updating each property according to these equations. We read "$\Delta t$" as "delta t" and it means "a small change in time".

The following function updates distance and velocity according to the above equations, given an acceleration and the timestep $\Delta t$. Note that the update uses a midpoint formula based on half of the "old" velocity (the velocity at the beginning of the timestep) and half of the "new" velocity (the velocity at the end of the timestep).

```julia
function update(x, velocity, acceleration, delta_t)
    new_v = velocity + acceleration * delta_t
    new_x = x + 0.5(velocity * delta_t) + 0.5(new_v * delta_t)
    return new_x, new_v
end
```

    update (generic function with 1 method)

We can use this update function to simulate the motion in the $x$ direction of an object that accelerates from a standing start at 1 metre per second per second*, for example, a shopping trolley rolling downhill along a street where the distance along the street is $x$.

\* That wasn't a stutter. An acceleration of one metre per second per second ($1 \text{ms}^{-2}$) means that every second the trolley increases its speed by one metre per second.

```julia
delta_t = 1.0 # time step is one second
nsteps = 10 # total time ten seconds

### TRY CHANGING THESE VALUES! ###
x = 0.0 # starting position (a distance of 0)
v = 0.0 # starting velocity (0 means object is stationary at time t=0)
a = 1.0 # accelerating at 1m/s^2

distance = Array{Float64}(undef, nsteps) # how far has the object travelled at each timestep?
for t = 1:nsteps
    x, v = update(x, v, a, delta_t)
    distance[t] = x
end

using Plots
gr()
plot(distance[1:nsteps], xlabel="time", ylabel="distance", legend=false)
```

## Computing the SIR Model

Back to the SIR model: the proportion of individuals in each compartment of the population changes as the epidemic spreads, according to a system of differential equations:

\begin{align}
\frac{dS}{dt} &= -\beta S(t) I(t) & \text{rate of change in susceptible } S \\
\frac{dI}{dt} &= \beta S(t) I(t) - \gamma I(t)  & \text{rate of change in infected } I \\
\frac{dR}{dt} &= \gamma I(t) & \text{rate of change in removed } R
\end{align}

The model parameters $\beta$ (beta) and $\gamma$ (gamma) represent the infection rate and the recovery rate, respectively. Note that $\frac{dS}{dt} + \frac{dI}{dt} + \frac{dR}{dt} = 0$.

There's one other number that is not a parameter to this model, but is often mentioned in discussing epidemics: the *basic reproduction number* $R_0$, which represents the expected number of people who would be directly infected by a single infected person.

We can simulate the change in each compartment over time by repeatedly updating each compartment based on the corresponding differential equation. In the SIR model, this can be calculated as:

\begin{align}
R_0 &= \frac{\beta N}{\gamma},
\end{align}

where $N = S+I+R$ is the total size of the population.

Here is update function to calculate the new proportion of individuals in each of the compartments $[S,I,R]$ after a timestep $\Delta t$:

```julia
function update_SIR(s, i, r, beta, gam, delta_t)
    new_s = s - (beta * s * i * delta_t)
    new_i = i + (beta * s * i * delta_t) - (gam * i * delta_t)
    new_r = r + (gam * i * delta_t)
    return [new_s, new_i, new_r]
end
```

    update_SIR (generic function with 1 method)

We can use this function to calculate the change in each group over time for an imaginary outbreak in a group of 100 people over a period of six months. Initially only one person is infected. Try changing the initial numbers in each compartment, as well as the model parameters beta and gamma, and see how it changes the spread over time.

```julia
num_days = 180
delta_t = 1.0 # timestep of one day
n_steps = round(Int64, num_days / delta_t)
result = Array{Float64}(undef, n_steps, 3)

### TRY CHANGING THESE VALUES! ###
s = 99 # susceptible
i =  1 # infectious
r =  0 # removed
beta = 0.0021 # probability of a infection
gam = 0.1 # 1/recovery time e.g. average 10 days to recover

time = 1:n_steps
for step = time
    result[step,:] = [s, i, r]
    s, i, r = update_SIR(s, i, r, beta, gam, delta_t)
end

N = s+i+r
R0 = beta * N / gam

using Printf
plot(time*delta_t, result[1:n_steps,:], title="Example SIR results", xlabel="Day", ylabel="Number of cases", labels = ["Susceptible" "Infected" "Removed"])
annotate!(150, 50, @sprintf("R0 = %.1f", R0), :blue)
```

## Validating Predictions Against Real-World Data

We can also look at real world data and see how well the actual progression of COVID-19 is simulated by our SIR model. The file [au_covid.csv](au_covid.csv) contains data for the first 230 days of the COVID-19 outbreak in Australia, taken from the first 230 days of the [2019 Novel Coronavirus COVID-19 (2019-nCoV) Data Repository by Johns Hopkins CSSE](https://github.com/CSSEGISandData/COVID-19/).

```julia
using CSV
x = CSV.read("au_covid.csv", transpose=true)
states = convert(Matrix, x)
state_names = reshape(String.(names(x)), 1, :)
plot(states, title="COVID-19 Cases per State or Territory", xlabel="Day", ylabel="Number of cases", labels=state_names, legend=:topleft)
```

We'll just look at the national total, as the numbers for most states are too small to accurately model.

```julia
national=sum(states,dims=2)
plot(national,title="COVID-19 Cases in Australia", xlabel="Day", ylabel="Number of cases", legend=false)
```

For a population of the size of Australia, we can choose arbitrary values of beta and gamma to see how closely the SIR model matches the observed data. Try changing the values in the code below to see how it affects the fit.

```julia
popsize=24600000 # population of Australia
num_days = size(states,1)
delta_t = 1.0 # timestep of one day
n_steps = round(Int64, num_days / delta_t)
result = Array{Float64}(undef, n_steps, 3)

### TRY CHANGING THESE VALUES! ###
infected = 4
beta = 3.5e-9
gam = 0.05 # 20-day infectious period

s = popsize-infected
i = infected
r = 0
time = 1:n_steps
for step = time
    result[step,:] = [s,i,r]
    s, i, r = update_SIR(s, i, r, beta, gam, delta_t)
end
predicted=result[time,2]+result[time,3]

R0 = beta * popsize / gam

plot(time*delta_t, [national[time],predicted], title="Actual vs. Observed Cases", xlabel="Day", ylabel="Number of Cases", labels = ["Actual" "Predicted"])
annotate!(50, 10000, @sprintf("R0 = %.1f", R0), :blue)
```

Rather than guessing the values of $\beta$ and $\gamma$, it would be better to fit them to the data to minimize the mean squared error between the predicted and actual values for each day. The [DifferentialEquations.jl](https://docs.sciml.ai/v1.6/analysis/parameter_estimation.html) package allows us to perform this fitting in Julia - but that's a subject for a future notebook.


# 2021-09-01 Boundaries and Fourier methods

## Last time
* High order on arbitrary grids
* Method of manufactured solutions
* More satisfying approach to boundary conditions

## Today
* More on boundary conditions
* Fourier methods for analyzing finite difference methods
* Discussion

```julia
using Plots
using LinearAlgebra
```

# Interpolation by Vandermonde matrices

We can compute a polynomial

$$ p(x) = c_0 + c_1 x + c_2 x^2 + \dotsb $$

that assumes function values $p(x_i) = u_i$ by solving a linear system with the Vandermonde matrix.

$$ \underbrace{\begin{bmatrix} 1 & x_0 & x_0^2 & \dotsb \\
    1 & x_1 & x_1^2 & \dotsb \\
    1 & x_2 & x_2^2 & \dotsb \\
    \vdots & & & \ddots \end{bmatrix}}_V \begin{bmatrix} c_0 \\ c_1 \\ c_2 \\ \vdots \end{bmatrix} = \begin{bmatrix} u_0 \\ u_1 \\ u_2 \\ \vdots \end{bmatrix} .$$

```julia
function vander(x, k=nothing)
    if k === nothing
        k = length(x)
    end
    V = ones(length(x), k)
    for j = 2:k
        V[:, j] = V[:, j-1] .* x
    end
    V
end
```

    vander (generic function with 2 methods)

```julia
cond(vander(LinRange(-1, 1, 5) .+ 1))
#cond(vander([-1,-.99,-.98,-.97, 1]))
```

    743.2516243247674

# Arbitrary order derivatives

```julia
function fdstencil(source, target, k)
    "kth derivative stencil from source to target"
    x = source .- target
    V = vander(x)
    rhs = zero(x)'
    rhs[k+1] = factorial(k)
    rhs / V
end
x = LinRange(-2, 2, 5)
u = sin.(x)
fdstencil(x, 0.5, 2)

```

    1×5 adjoint(::Vector{Float64}) with eltype Float64:
     -0.208333  1.33333  -1.75  0.333333  0.291667

```julia
plot([z -> fdstencil(x, z, 2) * u,
        z -> -sin(z)])
```

# High order discretization of the Laplacian
## The Poisson problem $-u_{xx} = f$ with boundary conditions

```julia
function poisson(x, spoints, forcing; left=(0, zero), right=(0, zero))
    n = length(x)
    L = zeros(n, n)
    rhs = forcing.(x)
    for i in 2:n-1
        jleft = min(max(1, i-spoints÷2), n-spoints+1)
        js = jleft : jleft + spoints - 1
        L[i, js] = -fdstencil(x[js], x[i], 2)
    end
    L[1,1:spoints] = fdstencil(x[1:spoints], x[1], left[1])
    L[n,n-spoints+1:n] = fdstencil(x[n-spoints+1:n], x[n], right[1])
    rhs[1] = left[2](x[1])
    rhs[n] = right[2](x[n])
    L, rhs
end
```

    poisson (generic function with 1 method)

```julia
L, b = poisson(LinRange(-1, 1, 6), 3, zero, left=(1, zero))
L
```

    6×6 Matrix{Float64}:
     -3.75   5.0   -1.25   0.0           0.0           0.0
     -6.25  12.5   -6.25   0.0           0.0           0.0
      0.0   -6.25  12.5   -6.25          0.0           0.0
      0.0    0.0   -6.25  12.5          -6.25          0.0
      0.0    0.0    0.0   -6.25         12.5          -6.25
      0.0    0.0    0.0    9.25186e-18  -2.31296e-16   1.0

# Method of manufactured solutions

## Problem: analytic solutions to PDEs are hard to find

Let's choose a smooth function with rich derivatives,
$$ u(x) = \tanh(x) . $$
Then $$ u'(x) = \cosh^{-2}(x) $$ and $$ u''(x) = -2 \tanh(x) \cosh^{-2}(x) . $$

* This works for nonlinear too.

```julia
x = LinRange(-2, 2, 10)
L, rhs = poisson(x, 5,
    x -> 2 * tanh(x) / cosh(x)^2,
    left=(0, tanh),
    right=(1, x -> cosh(x)^-2))
u = L \ rhs
plot(x, u, marker=:circle, legend=:topleft)
plot!(tanh)
```

# Convergence rate

```julia
ns = 2 .^ (3:10)
hs = 1 ./ ns
function poisson_error(n, spoints)
    x = LinRange(-2, 2, n)
    L, rhs = poisson(x, spoints, x -> 2 * tanh(x) / cosh(x)^2,
        left = (0, tanh),
        right = (1, x -> cosh(x)^-2))
    u = L \ rhs
    norm(u - tanh.(x), Inf)
end
```

    poisson_error (generic function with 2 methods)

```julia
plot(hs, [poisson_error(n, 5) for n in ns], marker=:circle)
plot!(hs, [poisson_error(n, 3) for n in ns], marker=:square)
plot!([h -> h^2, h -> h^4], xscale=:log10, yscale=:log10, legend=:topleft)
```

# Symmetry in boundary conditions: Dirichlet

We have implemented Dirichlet conditions by modifying the first row of the matrix,
\begin{align}
\begin{bmatrix} 1 & 0 & 0 & 0 & 0 \\ \\ & & A_{1:,:} & & \\ \\ \end{bmatrix} \begin{bmatrix} u_0 \\ \\ u_{1:} \\ \\ \end{bmatrix} = \begin{bmatrix} a \\ \\ f_{1:} \\ \\ \end{bmatrix} .
\end{align}

* This matrix is not symmetric even if $A$ is.
* We can eliminate $u_0$ and create a reduced system for $u_{1:}$.
* Generalize: consider a $2\times 2$ block system
\begin{align*}
\begin{bmatrix} I & 0 \\ A_{10} & A_{11} \end{bmatrix} \begin{bmatrix} u_0 \\ u_1 \end{bmatrix} = \begin{bmatrix} f_0 \\ f_1 \end{bmatrix}.
\end{align*}

We can rearrange as
$$A_{11} u_1 = f_1 - A_{10} f_0,$$
which is symmetric if $A_{11}$ is.
* This is called "lifting" and is often done implicitly in the mathematics literature.  It is convenient for linear solvers and eigenvalue solvers, but inconvenient for IO and postprocessing, as well as some nonlinear problems.
* Convenient alternative: write
$$\begin{bmatrix} I & 0 \\ 0 & A_{11} \end{bmatrix} \begin{bmatrix} u_0 \\ u_1 \end{bmatrix} = \begin{bmatrix} f_0 \\ f_1 - A_{10} f_0 \end{bmatrix},$$
which is symmetric and decouples the degrees of freedom associated with the boundary. This method applies cleanly to nonlinear problems.
* Optionally scale the identity by some scalar related to the norm of $A_{11}$.

# Symmetry in boundary conditions: Neumann

Consider FD discretization of the Neumann boundary condition
$$
\frac{du}{dx}(1) = b .
$$
1. Use a one-sided difference formula as in
$$ \frac{u_n - u_{n-1}}{h} = b . $$
  * an extra discretization choice
  * may reduce order of accuracy compared to interior discretization, lose symmetry.
2. Temporarily introduce a ghost value $u_{n+1} = u(x_{n+1} = 1 + h)$ (possibly more) and define it to be a reflection of the values from inside the domain.  In the case $b=0$, this reflection is $u_{n+i} = u_{n-i}$.  More generally,
$$ u_{n+i} = u_{n-i} + 2b(x_n - x_{n-i}) . $$

After this definition of ghost values, we apply the interior discretization at the boundary. For our reference equation, we would write

$$ \frac{-u_{n-1} + 2 u_n - u_{n+1}}{h^2} = f(x_n) $$

which simplifies to $$ \frac{u_n - u_{n-1}}{h^2} = f(x_n)/2 + b/h $$
after dividing by 2 and moving the boundary term to the right hand side.


```python
import matplotlib.pyplot as plt
import numpy as np
import sympy as sym
```

```python
%matplotlib notebook
```

```python
def average_fecundity(U, T, R, P, S):
    return 2 * ((R + P - T - S) * U**2 + (T + S - 2 * P) * U + P)

def equilibrium_coordinators_share(U, T, R, P, S):
    F = average_fecundity(U, T, R, P, S)
    return 2 * ((R - S) * U**2 + S * U) / F
```

```python
U = sym.symbols('U', real=True, nonnegative=True)
T, R, P, S = sym.symbols('T, R, P, S', real=True, positive=True)
x = equilibrium_coordinators_share(U, T, R, P, S)
```

```python
first_derivative = sym.lambdify((U, T, R, P, S), sym.diff(x, U, 1), modules=["numpy"])
second_derivative = sym.lambdify((U, T, R, P, S), sym.diff(x, U, 2), modules=["numpy"])
```

```python
_, ax = plt.subplots(1,1)
Us = np.logspace(-6, 0, 1000)
payoffs = [6, 2, 4, 1]
xs = equilibrium_coordinators_share(Us, *payoffs)
x_primes = first_derivative(Us, *payoffs)
x_prime_primes = second_derivative(Us, *payoffs)
ax.plot([0,1], [0,1], "k--")
ax.plot(xs, Us)
ax.plot(xs, x_primes, label=r"$\frac{\partial U}{\partial x}$")
ax.plot(xs, x_prime_primes, label=r"$\frac{\partial^2 U}{\partial x^2}$")
ax.set_xlabel(r"$x^*$")
ax.set_ylabel(r"$U(x^*)$", rotation="horizontal")
ax.set_title("T={}, R={}, P={}, S={}".format(*payoffs))
ax.legend()

```

    <IPython.core.display.Javascript object>

    <matplotlib.legend.Legend at 0x14e991e80>

```python
first_derivative(0, *payoffs), first_derivative(1, *payoffs)
```

    (0.5, 1.3333333333333335)

```python
# slope of the L-locus at x=0, U=0
1 / sym.diff(x, U).subs({U: 0})
```

    P/S

```python
# slope of the L-locus at x=1, U=1
1 / sym.together(sym.diff(x, U).subs({U: 1}))
```

    R/T

```python
sym.factor(sym.diff(x, U), U)
```

    -(-P*S + U**2*(2*P*R - P*S - R*T) + U*(-2*P*R + 2*P*S))/(P + U**2*(P + R - S - T) + U*(-2*P + S + T))**2

```python
numerator, denominator = sym.fraction(sym.factor(sym.diff(x, U), U))
```

```python
# always positive!
denominator
```

    (P + U**2*(P + R - S - T) + U*(-2*P + S + T))**2

```python
numerator
```

    P*S - U**2*(2*P*R - P*S - R*T) - U*(-2*P*R + 2*P*S)

Numerator polynomial opens up $\iff PS + RT > 2 PR$

```python
numerator_poly = sym.poly(numerator, U)
```

```python
sym.factor(numerator_poly.discriminant())
```

    4*P*R*(P*R - S*T)

Discriminant of the numerator polynomial will be negative (implying polynomial has imaginary roots!) $\iff$ $\frac{P}{S} < \frac{T}{R}$

```python
sym.solve(numerator_poly, U)
```

    [(sqrt(P)*sqrt(R)*sqrt(P*R - S*T) - P*(R - S))/(-2*P*R + P*S + R*T),
     -(sqrt(P)*sqrt(R)*sqrt(P*R - S*T) + P*(R - S))/(-2*P*R + P*S + R*T)]

Sufficient conditions for first derivative to be strictly positive are that numerator polynomial opens up and discriminant of the polynomial is negative.

```python

```

```python
second_derivative(0, *payoffs), second_derivative(1, *payoffs)
```

    (1.5, 0.4444444444444445)

$$ (-U*(S + U*(R - S))*(-2*P + S + T + 2*U*(P + R - S - T)) + (S + 2*U*(R - S))*(P + U**2*(P + R - S - T) + U*(-2*P + S + T))) > 0$$

```python
sym.simplify(sym.diff(x, U, 2).subs({U: 0}))
```

    (2*P*(R - S) - 2*S*(-2*P + S + T))/P**2

```python
sym.simplify(sym.diff(x, U, 2).subs({U: 1}))
```

    (-2*P*R - 2*R*T + 2*S*T + 2*T**2)/R**2

```python
sym.together(sym.diff(x, U, 2))
```

    2*(R*(P + U**2*(P + R - S - T) + U*(-2*P + S + T))**2 - S*(P + U**2*(P + R - S - T) + U*(-2*P + S + T))**2 - U*(S + U*(R - S))*(P + U**2*(P + R - S - T) + U*(-2*P + S + T))*(P + R - S - T) + U*(S + U*(R - S))*(-2*P + S + T + 2*U*(P + R - S - T))**2 - (S + 2*U*(R - S))*(P + U**2*(P + R - S - T) + U*(-2*P + S + T))*(-2*P + S + T + 2*U*(P + R - S - T)))/(P + U**2*(P + R - S - T) + U*(-2*P + S + T))**3

```python
numerator, denominator = sym.fraction(sym.simplify(sym.diff(x, U, 2)))
```

```python
# always positive!
denominator
```

    (P + U**2*(P + R - S - T) + U*(-2*P + S + T))**3

```python
numerator_poly = sym.poly(numerator, U)
```

```python
sym.factor(numerator_poly.discriminant())
```

    432*P**2*R**2*(P*R - S*T)**2*(P + R - S - T)**2*(4*P*R - S**2 - 2*S*T - T**2)

```python
sym.solve(numerator_poly, U)
```

    [-(-3*P*R + 3*P*S)/(3*(2*P*R - P*S - R*T)) - (9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)/(3*(27*P*S*(-3*P*R + 3*P*S)/(2*(2*P*R - P*S - R*T)**2) + (-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + sqrt(-4*(9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)**3 + (27*P*S*(-3*P*R + 3*P*S)/(2*P*R - P*S - R*T)**2 + 2*(-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2))**2)/2 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2)))**(1/3)) - (27*P*S*(-3*P*R + 3*P*S)/(2*(2*P*R - P*S - R*T)**2) + (-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + sqrt(-4*(9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)**3 + (27*P*S*(-3*P*R + 3*P*S)/(2*P*R - P*S - R*T)**2 + 2*(-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2))**2)/2 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2)))**(1/3)/3,
     -(-3*P*R + 3*P*S)/(3*(2*P*R - P*S - R*T)) - (9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)/(3*(-1/2 - sqrt(3)*I/2)*(27*P*S*(-3*P*R + 3*P*S)/(2*(2*P*R - P*S - R*T)**2) + (-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + sqrt(-4*(9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)**3 + (27*P*S*(-3*P*R + 3*P*S)/(2*P*R - P*S - R*T)**2 + 2*(-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2))**2)/2 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2)))**(1/3)) - (-1/2 - sqrt(3)*I/2)*(27*P*S*(-3*P*R + 3*P*S)/(2*(2*P*R - P*S - R*T)**2) + (-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + sqrt(-4*(9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)**3 + (27*P*S*(-3*P*R + 3*P*S)/(2*P*R - P*S - R*T)**2 + 2*(-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2))**2)/2 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2)))**(1/3)/3,
     -(-3*P*R + 3*P*S)/(3*(2*P*R - P*S - R*T)) - (9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)/(3*(-1/2 + sqrt(3)*I/2)*(27*P*S*(-3*P*R + 3*P*S)/(2*(2*P*R - P*S - R*T)**2) + (-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + sqrt(-4*(9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)**3 + (27*P*S*(-3*P*R + 3*P*S)/(2*P*R - P*S - R*T)**2 + 2*(-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2))**2)/2 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2)))**(1/3)) - (-1/2 + sqrt(3)*I/2)*(27*P*S*(-3*P*R + 3*P*S)/(2*(2*P*R - P*S - R*T)**2) + (-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + sqrt(-4*(9*P*S/(2*P*R - P*S - R*T) + (-3*P*R + 3*P*S)**2/(2*P*R - P*S - R*T)**2)**3 + (27*P*S*(-3*P*R + 3*P*S)/(2*P*R - P*S - R*T)**2 + 2*(-3*P*R + 3*P*S)**3/(2*P*R - P*S - R*T)**3 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2))**2)/2 + 27*(P**2*R + P**2*S - P*S**2 - P*S*T)/(2*(2*P**2*R - P**2*S + 2*P*R**2 - 3*P*R*S - 3*P*R*T + P*S**2 + P*S*T - R**2*T + R*S*T + R*T**2)))**(1/3)/3]

```python

```


##A Single Neuron with 3 inputs (Vanilla Network)
As in most cases, when you initialize parameters in neural networks, our network will have weights initialized randomly, and biases set as zero to start or 1 but in this case we set up at 2

```python
inputs = [1, 2, 3]
weights = [0.2, 0.8, -0.5]
bias = 2

output = (inputs[0]*weights[0] + inputs[1]*weights[1]+inputs[2]*weights[2] + bias)
print(output)
```

    2.3

What might we need to change if we have 4 inputs, rather than the 3

```python
inputs = [ 1.0 , 2.0 , 3.0 , 2.5 ]
weights = [ 0.2 , 0.8 , - 0.5 , 1.0 ]
bias = 2.0

output = (inputs[0]*weights[0] + inputs[1]*weights[1]+inputs[2]*weights[2] + inputs[3]*weights[3] + bias)
print(output)

# Alternative for above

# output = 0
# for i in range(len(inputs)):

#   output = inputs[i] * weights[i] + output

# output += bias

# print(output)
```

    4.8

#A layer of Neurons
Neural networks typically have layers that consist of more than one neuron. Layers are nothing more than groups of neurons. Each neuron in a layer takes exactly the same input — the input given to the layer (which can be either the training data or the output from the previous layer) but contains its own set of weights and its own bias, producing its own unique output. For example Let’s say we have a scenario with 3 neurons in a layer and 4 inputs:

we have three sets of weights and three biases, which define three neurons

```python
inputs = [ 1, 2, 3, 2.5 ]
weights1 = [ 0.2 , 0.8 , - 0.5 , 1 ]
weights2 = [ 0.5 , - 0.91 , 0.26 , - 0.5 ]
weights3 = [ - 0.26 , - 0.27 , 0.17 , 0.87 ]

bias1 = 2
bias2 = 3
bias3 = 0.5

output1 = 0
output2 = 0
output3 = 0

# for i in range(len(inputs)):
#   output1 = inputs[i] * weights1[i] + output1
#   output2 = inputs[i] * weights2[i] + output2
#   output3 = inputs[i] * weights3[i] + output3

# weighted1 = output1 + bias1
# weighted2 = output2 + bias2
# weighted3 = output3 + bias3

# print([weighted1, weighted2, weighted3])

outputs = [
# Neuron 1:
inputs[0] * weights1[0] +
inputs[1] * weights1[1] +
inputs[2] * weights1[2] +
inputs[3] * weights1[3] + bias1,
# Neuron 2:
inputs[0] * weights2[0] +
inputs[1] * weights2[1] +
inputs[2] * weights2[2] +
inputs[3] * weights2[3] + bias2,
# Neuron 3:
inputs[0] * weights3[0] +
inputs[1] * weights3[1] +
inputs[2] * weights3[2] +
inputs[3] * weights3[3] + bias3]

print(outputs)

```

    [4.8, 1.21, 2.385]

```python
#Alternative for above

inputs = [1, 2, 3, 2.5]
weights = [[0.2, 0.8, - 0.5, 1 ],
           [0.5, - 0.91, 0.26, - 0.5 ],
           [-0.26, - 0.27, 0.17, 0.87 ]]
biases = [2, 3, 0.5 ]

# Output of current layer
layer_outputs = []

for neuron_weights, neuron_biases in zip(weights, biases):
  #output of given neuron
  neuron_output = 0

  for n_input, n_weights in zip (inputs, neuron_weights):
    neuron_output = n_input * n_weights + neuron_output
  ## Add bias
  neuron_output += neuron_biases
  layer_outputs.append(neuron_output)

print(layer_outputs)
```

    [4.8, 1.21, 2.385]

#Dot Product and Vector Addition in Python

\begin{equation}
\vec{a} \cdot \vec{b}=\sum_{i=1}^{n} a_{i} b_{i}=a_{1} b_{1}+a_{2} b_{2}+\ldots+a_{n} b_{n}
\end{equation}

\begin{equation}
\vec{a}+\vec{b}=\left[a_{1}+b_{1}, a_{2}+b_{2}, \ldots, a_{n}+b_{n}\right]
\end{equation}

```python
a = [1, 2, 3]
b = [2, 3, 4]

dot_product = a[0]*b[0] + a[1]*b[1] + a[2]*b[2]
print(dot_product)
```

    20

#Succinct way to perform in numpy with dot operator

```python
import numpy as np
print(np.dot(a, b))
```

    20

#A Single Neuron with NumPy

```python
inputs = [ 1.0 , 2.0 , 3.0 , 2.5 ]
weights = [ 0.2 , 0.8 , - 0.5 , 1.0 ]
bias = 2.0

output = np.dot(inputs, weights) + bias
print(output)
```

    4.8

#A Layer of Neurons with NumPy

```python
inputs = np.array([1.0, 2.0, 3.0, 2.5])
weights = np.array([[0.2, 0.8, -0.5, 1],
                [0.5, - 0.91, 0.26, -0.5],
                [- 0.26, - 0.27 , 0.17 , 0.87]])
biases = [2.0, 3.0, 0.5]

output = np.dot(weights, inputs) + biases
print(output)
```

    [4.8   1.21  2.385]

#Matrix Product

\begin{equation}
c_{i j}=a_{i 1} b_{1 j}+a_{i 2} b_{2 j}+\cdots+a_{i n} b_{n j}=\sum_{k=1}^{n} a_{i k} b_{k j}
\end{equation}

#Transposition of the Matrix product

\begin{equation}
\vec{a}.\vec{b}= a.b^{T}
\end{equation}

Transposition simply modifies a matrix in a way that its rows become columns and columns become rows.

```python
#To transform a list into a matrix containing a single row (perform an equivalent operation of turning a vector into row vector)
(perform an equivalenarray of shape (n, 1):
a = [1, 2, 3]
np.expand_dims(np.array(a), axis = 0)
```

    array([[1, 2, 3]])

#A Layer of Neurons & Batch of Data with Numpy

```python
inputs = [[1.0, 2.0, 3.0, 2.5 ],
          [2.0, 5.0, - 1.0, 2.0],
           [-1.5, 2.7, 3.3, - 0.8]]

weights = np.array([[0.2, 0.8, - 0.5, 1.0],
                    [0.5, - 0.91, 0.26, - 0.5],
                    [-0.26, -0.27, 0.17, 0.87]]).T

biases = [2.0 , 3.0 , 0.5 ]
#np.dot(inputs, np.array(weights).T) + biases
output_layer = np.dot(inputs, weights) + biases
print(output_layer)
```

    [[ 4.8    1.21   2.385]
     [ 8.9   -1.81   0.2  ]
     [ 1.41   1.051  0.026]]


# The Poisson-Bolzman Distribution

In order to understand lubrication better, we simulate thin layers of lubricant on a metallic surface, solvated in water.
Different structures of lubricant films are created by varying parameters like their concentration and the charge of the surface.
The lubricant is somewhat solvable in water, thus parts of the film will diffuse into the bulk water.
Lubricant molecules are charged, and their distribution is roughly exponential.

As simplification, we  first create a solution of ions (Na+, purple; Cl-, green) in water (not shown).

Close to the positively charged metallic surface, the electric potential (red) will be highest, falling off exponentially when further away.
This potential attracts negatively charged Chlorine ions, and pushes positively charged Natrium ions away, resulting in a higher (lower) concentration of Clorine (Natrium) near the surface.

```python
import numpy as np
import scipy.constants as sc
import matplotlib.pyplot as plt

# Sampling and plotting
from generate_structure import generate_structure
from generate_structure import plot_dist
from generate_structure import get_histogram
from IPython.display import Image

# Physical distribution generation
from poisson_bolzmann_distribution import debye
from poisson_bolzmann_distribution import gamma
from poisson_bolzmann_distribution import charge_density
from poisson_bolzmann_distribution import potential

np.random.seed(74)
```

To calculate this, we first need to find out how ions are distributed in solution.
A good description of the concentration of our ion species, $c_{Na^+}$, is given by the solution to the Poisson-Bolzmann equation:

$
\begin{align}
\rho_{Na^+}(z) &= \rho_{Na^+}(\infty) e^{-e \Psi(z)/k_B T}\\
\Psi(z) &= \frac{2k_B T}{e} \log\Big(\frac{1 + \gamma e^{-\kappa z}}{1- \gamma e^{-\kappa z}}\Big)
        \approx \frac{4k_B T}{e} \gamma e^{-\kappa z} \\
\gamma &= \tanh(\frac{e\Psi(0)}{4k_B T})\\
\kappa &= 1/\lambda_D\\
\lambda_D &= \Big(\frac{\epsilon \epsilon_0 k_B T}{\sum_{i} \rho_i(\infty) e^2 z_i^2} \Big)^\frac{1}{2} [m^{-1}]
\end{align}
$

With:
* $z$: Distance from the double layer
* $\Psi(0)$: Potential at the surface
* $\Psi(z)$: Potential in the solution
* $k_B$: Bolzmann Constant
* $T$: Temperature [Kelvin]
* $e$: Elemental Charge (or Euler's constant when exponentiated)
* $\gamma$: Term from Gouy-Chapmann theory
    * $\gamma \rightarrow 1$ for high potentials
    * $\Psi(z) \approx \Psi_0 e^{-\kappa z}$ for low potentials $\Psi(0) \approx 0$
* $\lambda_D$: Debye Length (34.0 nm for NaCl, 10^-4 M, 25°C)
* $\rho_{Na^+}$: Concentration of Natrium ions
* $\rho_{Na^+}(\infty)$: Bulk Natrium concentration (at infinity, where the solution is homogeneous)
* $\epsilon$: Permittivity of the solution
* $\epsilon_0$: Electric constant aka Vacuum permittivity
* $z_i$: Charge of species i

These equations are implemented in `poisson_bolzmann_distribution.py`

## Debye length
We start with the expression for the Debye length, which describes the thickness of the ionic double layer, or the distance at which the surface charge is mostly screened off.

```python
default_density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': default_density, 'Cl': default_density}
charge = {'Na': 1, 'Cl': -1}
deb = debye(rho_bulk=rho, charge=charge) * 10**9
print('Debye Length of 10^-4 M saltwater: {} nm (Target: 30.4 nm)'.format(round(deb, 2)))

density = np.logspace(-6, 0, 50) * sc.Avogadro * 1000
debyes = [debye(rho_bulk={'Na': d, 'Cl': d}, charge=charge) * 10**9 for d in density]

plt.xlabel('Density [1/m^3]')
plt.ylabel('Debye length at 25° [nm]')
plt.semilogx(density, debyes, marker='.')
plt.show()
```

The debye length depends on the concentration of ions in solution, at low concentrations it becomes large. We can reproduce literature debye lengths with our function, so everything looks good.

## Gamma Function

Next we calculate the gamma function $\gamma = \tanh(\frac{e\Psi(0)}{4k_B T})$

```python
x = np.linspace(12, 16, 40)
gammas = [gamma(10 ** i, 300) for i in x]
plt.xlabel('Potential')
plt.ylabel('Gamma at 300K')
plt.plot(x, gammas, marker='o')
plt.show()
```

Which looks as expected, but we have no values to compare it against.

## Potential

We plug these two functions into the expression for the potential

$\Psi(z) = \frac{2k_B T}{e} \log\Big(\frac{1 + \gamma e^{-\kappa z}}{1- \gamma e^{-\kappa z}}\Big)
        \approx \frac{4k_B T}{e} \gamma e^{-\kappa z}$

```python
z = np.linspace(0, 2*10**-7, 10000)
density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': density, 'Cl':density}
charge = {'Na': 1, 'Cl': -1}
pot_0 = 100
psi = [potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]
plt.xlabel('z [nm]')
plt.ylabel('Potential [V]')
plt.plot(z*10**9, psi, marker='')
plt.show()
```

The potential is smooth and looks roughly exponential. Everything good so far.

## Charge density

Now we obtain the charge density $\rho$ from the potential $\Psi$ via

$\rho_{Na^+}(z) = \rho_{Na^+}(\infty) e^{-e \Psi(z)/k_B T}$

```python
z = np.linspace(0, 100*10**-9, 2000)
density = sc.Avogadro * 1000 * 10**-4
rho = {'Na': density, 'Cl':density}
charge = {'Na': 1, 'Cl': -1}
pot_0 = 0.05  # Breaks if > 1

psi = [potential(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0) for loc in z]

rho_na = np.array([charge_density(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0, species='Na') for loc in z])
rho_cl = np.array([charge_density(location=loc, rho_bulk=rho, charge=charge, surface_potential=pot_0, species='Cl') for loc in z])

deb = debye(rho_bulk=rho, charge=charge) * 10**9

fig, ax1 = plt.subplots(figsize=[16,5])
ax1.set_xlabel('z [nm]')
ax1.plot(z*10**9, psi, marker='', color='red', label='Potential', linewidth=1, linestyle='--')
ax1.set_ylabel('Potential')
ax1.axvline(x=deb, label='Debye Length', color='orange')

ax2 = ax1.twinx()
ax2.plot(z*10**9, [density]*len(z), label='Bulk concentration', color='grey')
ax2.plot(z*10**9, rho_na, marker='', color='green', label='Na+ ions')
ax2.plot(z*10**9, rho_cl, marker='', color='blue', label='Cl- ions')
ax2.set_ylabel('Density')

#fig.legend(loc='center')
ax2.legend(loc='best', fontsize=15)
ax1.legend(loc='upper center', fontsize=15)
fig.tight_layout()
plt.show()
```

The charge density behaves as expected, it interpolates between low (high) concentration and the bulk concentration within the first few debye lengths.

## Sampling
Now let's see if we can just plug our new distribution in our existing framework.

First, we need to convert the physical distribution to the format we were using so far:

```python
def wrap_distribution(x, species):
    """Wrapper for na+ ions."""
    density = sc.Avogadro * 1000 * 10**-4
    rho = {'Na': density, 'Cl':density}
    charge = {'Na': 1, 'Cl': -1}
    pot_0 = 0.05  # Breaks if > 1

    def call_distri(loc):
        distri = charge_density(location=loc, rho_bulk=rho,
                                charge=charge, surface_potential=pot_0, species=species)
        return float(distri)

    if not np.isscalar(x):
        y = []
        for i in range(0, len(x)):
            val = call_distri(x[i])

            # Normalize to be 1 at x=0
            val /= call_distri(0)
            # Scale distribution to have values in [0, 0.1] for ease of sampling
            val /= 10
            y += [val]
        return np.array(y)

    # If we have only a point estimate
    val = call_distri(x)
    # Normalize to be 1 at x=0
    val /= call_distri(0)
    # Scale distribution to have values in [0, 0.1] for ease of sampling
    val /= 10
    return val

def cl_distribution(x):
    return wrap_distribution(x, species='Cl')

def na_distribution(x):
    return wrap_distribution(x, species='Na')
```

Now we can sample the distribution of our $Na^+$ ions in z-direction.

```python
x = 50 * 10**-9
z = 100 * 10**-9
box = np.array([x, x, z])
sample_size = 10000
```

```python
struc = generate_structure(distribution=na_distribution, box=box, atom_count=sample_size)
histx, histy, histz = get_histogram(struc, box=box, n_bins=51)
plot_dist(histz, 'Distribution of Na+ ions in z-direction', reference_distribution=na_distribution)
```

```python
struc = generate_structure(distribution=cl_distribution, box=box, atom_count=sample_size)
histx, histy, histz = get_histogram(struc, box=box, n_bins=51)
plot_dist(histz, 'Distribution of Cl- ions in z-direction', reference_distribution=cl_distribution)
```

## Write to file
To visualize our structure, we export it to the .xyz file format, which is basically

```
ATOM_NUMBER
Optional comment
atom_type x y z
atom_type x y z
```

Avogadro expects x, y, z to be in units of $10^{-9}~m$, so we convert our salt "solution" to this unit.

Also, we use fewer atoms, just 100 for the 100x50x50 box - this will increase fluctuations with respect to the correct distribution though.

```python
from generate_structure import concat_names_structs
from generate_structure import export_named_struc

cl_struc = generate_structure(distribution=cl_distribution, box=box, atom_count=100)
na_struc = generate_structure(distribution=na_distribution, box=box, atom_count=100)

concat_list = concat_names_structs(struc_list=[cl_struc, na_struc], name_list=['Cl', 'Na'])
rescaled_list = []
for line in concat_list:
    name, x, y, z = line
    x = float(x) * 10**9
    y = float(y) * 10**9
    z = float(z) * 10**9
    rescaled_list += [[name, x, y, z]]

rescaled_list = np.array(rescaled_list)
export_named_struc(rescaled_list)

histx, histy, histz = get_histogram(cl_struc, box=box, n_bins=21)
plot_dist(histz, 'Distribution of Cl- ions in z-direction', reference_distribution=cl_distribution)

histx, histy, histz = get_histogram(na_struc, box=box, n_bins=21)
plot_dist(histz, 'Distribution of Na+ ions in z-direction', reference_distribution=na_distribution)
```

Now we can import the xyz file into avogadro, the visualization should look like this:

```python
Image(filename='distributed_atom_structure.png')
```


# Polynomial Constrained Boolean Optimization (PCBO) problems with qubovert

*qubovert* must be pip installed.

Import `boolean_var` from `qubovert`. This will create Polynomial Constrained Boolean Optimization (PCBO) objects.

```python
from qubovert import boolean_var
```

Let's try to encode the optimization problem of finding the minimum of $C$, where

$$C = x_0 x_1 - 2x_1 + x_1 x_2 x_3.$$

```python
# create the variables
x = [boolean_var("x%d" % i) for i in range(4)]

C = x[0] * x[1] - 2 * x[1] + x[1] * x[2] * x[3]

print(C)
```

    {('x1', 'x0'): 1, ('x1',): -2, ('x1', 'x3', 'x2'): 1}

And we subject $C$ to the constraints

$$x_0 \oplus x_1 = x_3, \quad 3x_0 + 2x_1 + 4x_3 < 7, \quad {\rm and} \quad x_0 x_1 = x_3 - x_2.$$

To enforce the constraints, we will pick a symbol $\lambda$ that we can tune as we like later.

```python
#!pip install sympy
from sympy import Symbol

lam = Symbol("lam", real=True, positive=True)

C.add_constraint_eq_XOR(
    x[3], x[0], x[1],
    lam=lam
).add_constraint_lt_zero(
    3 * x[0] + 2 * x[1] + 4 * x[3] - 7,
    lam=lam
).add_constraint_eq_zero(
    x[0] * x[1] - x[3] + x[2],
    lam=lam
)

print(C)
```

    {('x1', 'x0'): 11*lam + 1, ('x1',): -19*lam - 2, ('x1', 'x3', 'x2'): 1, ('x0',): -26*lam, ('x3', 'x0'): 22*lam, ('x1', 'x3', 'x0'): 2*lam, ('x1', 'x3'): 14*lam, ('x3',): -30*lam, ('__a0', 'x0'): 6*lam, ('__a1', 'x0'): 12*lam, ('__a2', 'x0'): 24*lam, ('x1', '__a0'): 4*lam, ('x1', '__a1'): 8*lam, ('x1', '__a2'): 16*lam, ('x3', '__a0'): 8*lam, ('x3', '__a1'): 16*lam, ('x3', '__a2'): 32*lam, (): 36*lam, ('__a0',): -11*lam, ('__a1',): -20*lam, ('__a2',): -32*lam, ('__a0', '__a1'): 4*lam, ('__a2', '__a0'): 8*lam, ('__a2', '__a1'): 16*lam, ('x1', 'x2', 'x0'): 2*lam, ('x3', 'x2'): -2*lam, ('x2',): lam}

Notice that the constraints are automatically added to the objective function, and in particular the inequality constraint introduced some ancilla bits (labeled `'_a0'`, `'_a1'`, and `'_a2'`). Let's look at the constraints.

```python
print(C.constraints)
```

    {'eq': [{('x0',): 1, ('x1', 'x0'): -2, ('x1',): 1, ('x3',): -1}, {('x1', 'x0'): 1, ('x3',): -1, ('x2',): 1}], 'lt': [{('x0',): 3, ('x1',): 2, ('x3',): 4, (): -7}]}

Notice there are two equality constraints and one inequality constraint. The first equality one comes from enforcing the XOR constraint, and the second comes from enforcing the other equality constraint. The `'eq'` key of the constraints dictionary indicates that the quantity equals zero, and the `'lt'` key of the constraints dictionary indicates that the quantity is less than zero. Other possible keys are `'le'`, `'gt'`, `'ge'`, and `'ne'`. See the docstrings for `PCBO.add_constraint_eq_zero`, `PCBO.add_constraint_lt_zero`, `PCBO.add_constraint_le_zero`, `PCBO.add_constraint_gt_zero`, `PCBO.add_constraint_ge_zero`, and `PCBO.add_constraint_ne_zero` for info.

For testing purposes, let's solve this bruteforce to make sure everything is working.

```python
solutions = C.solve_bruteforce(all_solutions=True)
print(solutions)
```

    [{'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1, '__a0': 0, '__a1': 0, '__a2': 0}]

Notice that there is one unique solution that minimizes the objective function and obeys all the constraints. We can get rid of the ancilla information with ``C.remove_ancilla_from_solution``.

```python
solution = solutions[0]
C.remove_ancilla_from_solution(solution)
```

    {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1}

Now let's solve this problem with a generic QUBO solver. Notice that the degree of problem is more than two, making `C` not a natural Quadratic Unconstrained Boolean Optimization Problem (QUBO).

```python
C.degree
```

    3

We can convert it to a QUBO (note that there are some options for the reduction from PUBO to QUBO, see the `C.to_qubo` method for details). Ancilla bits will need to be added, and bit labels are mapped to integers.

```python
Q = C.to_qubo()
print("num PUBO variables", C.num_binary_variables)
print("num QUBO variables", Q.num_binary_variables)
print()
print(Q)
```

    num PUBO variables 7
    num QUBO variables 9

    {(0, 1): 13*lam + 2, (0,): -19*lam - 2, (7,): 6*lam + 9, (0, 2): 16*lam + 3, (0, 7): -4*lam - 6, (2, 7): -4*lam - 6, (3, 7): 1, (1,): -26*lam, (1, 2): 22*lam, (1, 7): 2*lam, (2,): -30*lam, (1, 4): 6*lam, (1, 5): 12*lam, (1, 6): 24*lam, (0, 4): 4*lam, (0, 5): 8*lam, (0, 6): 16*lam, (2, 4): 8*lam, (2, 5): 16*lam, (2, 6): 32*lam, (4,): -11*lam, (5,): -20*lam, (6,): -32*lam, (4, 5): 4*lam, (4, 6): 8*lam, (5, 6): 16*lam, (8,): 6*lam + 3, (0, 8): -4*lam - 2, (1, 8): -4*lam - 2, (3, 8): 2*lam, (2, 3): -2*lam, (3,): lam, (): 36*lam}

For testing purposes, let's solve this with bruteforce to see what the proper value of $\lambda$ should be to enforce the constraints. Notice how we remap the QUBO solution to the PCBO solution with `C.convert_solution(x)`.

```python
for l in (1, 2, 3):
    Q_temp = Q.subs({lam: l})
    solutions = Q_temp.solve_bruteforce(all_solutions=True)
    solutions = [C.convert_solution(x) for x in solutions]
    print('lam', l)
    for s in solutions:
        print("\t", s, "is", "valid" if C.is_solution_valid(s) else "invalid")
    print()
```

    lam 1
    	 {'x1': 1, 'x0': 0, 'x3': 0, 'x2': 0, '__a0': 0, '__a1': 0, '__a2': 1} is invalid
    	 {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 0, '__a0': 0, '__a1': 0, '__a2': 0} is invalid
    	 {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1, '__a0': 0, '__a1': 0, '__a2': 0} is valid

    lam 2
    	 {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1, '__a0': 0, '__a1': 0, '__a2': 0} is valid

    lam 3
    	 {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1, '__a0': 0, '__a1': 0, '__a2': 0} is valid

We see that $\lambda = 2$ is sufficient to enforce the constraints. So let's update our QUBO.

```python
Q_good = Q.subs({lam: 2})
```

Now let's solve the QUBO with D'Wave's simulated annealer.

```python
#!pip install dwave-neal
from neal import SimulatedAnnealingSampler

sampler = SimulatedAnnealingSampler()
```

Note that their software package takes in a specific form for QUBOs, namely, the keys of the dictionary must be two element tuples. This form can be accessed from `Q` and `Q_good` with `Q.Q` or `Q_good.Q`.

```python
qubo_sample = sampler.sample_qubo(Q_good.Q, num_reads=100)
print("objective function:", qubo_sample.first.energy + Q_good.offset, "\n")

qubo_solution = qubo_sample.first.sample
print("qubo solution:", qubo_solution, "\n")

solution = C.convert_solution(qubo_solution)
print("pcbo solution:", solution)
print("objective function:", C.value(solution), "\n")

print("The solution is", "valid" if C.is_solution_valid(solution) else "invalid")
```

    objective function: -1.0

    qubo solution: {0: 1, 1: 0, 2: 1, 3: 1, 4: 0, 5: 0, 6: 0, 7: 1, 8: 0}

    pcbo solution: {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1, '__a0': 0, '__a1': 0, '__a2': 0}
    objective function: -1

    The solution is valid

This matches the result of `C.solve_bruteforce()`. Now we'll solve an QUSO formulation of our problem. Again we'll take $\lambda = 2$.

```python
L = C.to_quso().subs({lam: 2})
# note that we cannot do C.subs({lam: 2}).to_quso()!! This is because C.subs({lam: 2})
# creates a new PCBO object, and it's mapping from variables labels to integers may be
# different than C's mapping. For example, try C.mapping == C.subs({lam: 2}).mapping a
# few times. They will often be different.
print("num PUBO variables", C.num_binary_variables)
print("num QUSO variables", L.num_binary_variables)
print()
print(L)
```

    num PUBO variables 7
    num QUSO variables 9

    {(0, 1): 7.0, (0,): -3.75, (1,): -11.5, (): 45.0, (7,): -4.75, (0, 2): 8.75, (2,): -13.25, (0, 7): -3.5, (2, 7): -3.5, (3, 7): 0.25, (3,): -1.25, (1, 2): 11.0, (1, 7): 1.0, (1, 4): 3.0, (1, 5): 6.0, (1, 6): 12.0, (6,): -16.0, (0, 4): 2.0, (0, 5): 4.0, (0, 6): 8.0, (2, 4): 4.0, (2, 5): 8.0, (2, 6): 16.0, (4, 5): 2.0, (4, 6): 4.0, (4,): -4.0, (5, 6): 8.0, (5,): -8.0, (8,): -3.5, (0, 8): -2.5, (1, 8): -2.5, (3, 8): 1.0, (2, 3): -1.0}

Similar to their QUBO solver, D'Wave's QUSO solver accepts a specific form for QUSO models, namely a linear term dictionary and a quadratic term dictionary. These can be accessed with `L.h` and `L.J`.

```python
quso_sample = sampler.sample_ising(L.h, L.J, num_reads=100)
print("objective function:", quso_sample.first.energy + L.offset, "\n")

quso_solution = quso_sample.first.sample
print("quso solution:", quso_solution, "\n")

solution = C.convert_solution(quso_solution)
print("pcbo solution:", solution)
print("objective function:", C.value(solution), "\n")

print("The solution is", "valid" if C.is_solution_valid(solution) else "invalid")
```

    objective function: -1.0

    quso solution: {0: -1, 1: 1, 2: -1, 3: -1, 4: 1, 5: 1, 6: 1, 7: -1, 8: 1}

    pcbo solution: {'x1': 1, 'x0': 0, 'x3': 1, 'x2': 1, '__a0': 0, '__a1': 0, '__a2': 0}
    objective function: -1

    The solution is valid

Again this matches the result of `C.solve_bruteforce()`.

Much of the functionality from above can also be done with Polynomial Constrained Spin Optimization (PCSO). See ``help(qubovert.PCSO)`` and ``help(qubovert.spin_var)``.


# Homework 4 Programming Supplement
## Name

This is the programming supplement to Homework 4.  Here, you will make synthetic Rayleigh wave seismograms.  In practice this is more complicated and we would typically use a normal mode summation or compute the response in a layer-cake model but the salient ideas are here.

```python
from numpy import pi
from numpy import ones
from numpy import zeros
from numpy import asarray
from numpy import cos
from numpy import sin
from numpy import amax
from numpy import linspace
from matplotlib import pyplot as plt
from scipy.signal import hann
from scipy.signal import iirfilter, lfilter, filtfilt
%matplotlib inline
```

```python
# utility function
def deg2km(gcarc):
    """
    Converts great circle distance in degress to kilometers.

    Parameters
    ----------
    gcarc : float
         Great circle distance in degrees.

    Returns
    -------
    distkm : float
         The corresponding great circle distance in kilometers.
    """
    distkm = 111.195*gcarc
    return distkm
```

# Shearer Section 8.7 Question 4a

Here we assume that the Rayleigh wave phase velocity at periods between $50$ seconds to $500$ seconds can be approximated by the polynomial

\begin{equation}
   c(T) = 4.020 - 1.839 \times 10^{-3} T + 3.071 \times 10^{-5} T^2 - 3.549 \times 10^{-8} T^3
\end{equation}
where $T$ is the period in seconds and $c$ has units of km/s.

The group velocity, in km/s, was shown in the homework to be
\begin{equation}
   U = \frac{c}{1 + \frac{T}{c} \frac{dc}{dT} }
\end{equation}

where the derivative is given by
\begin{equation}
   \frac{dc}{dT}
  =-1.839 \times 10^{-3} + 2 \times 3.071 \times 10^{-5} T - 3 \times 3.549 \times 10^{-8} T^2
\end{equation}

Given this, fill in the following code segments

```python
def compute_C(T):
    """
    Computes the phase velocity as a function of period.

    Parameters
    ----------
    T : array_like
        The period in seconds at which to compute the phase velocities.

    Returns
    -------
    c : array_like
        The corresponding phase velocities in km/s.
    """
    c = 4.02 - 1.839e-3*T + 3.071e-5*T**2 - 3.549e-8*T**3
    return c

def compute_dcdT(T):
    """
    Computes the derivative of the phase velocity with respect to the period.

    Parameters
    ----------
    T : array_like
        The period in seconds at which to compute dc/dT

    Returns
    -------
    dcdT : array_like
        The corresponding derivatives of the phase velocity in km/s/s.
    """
    dcdT = #Put your code here
    return dcdT

def compute_U(T):
    """
    Computes the group velocity for the given period.

    Parameters
    ----------
    T : array_like
        The period in seconds at which to compute the group velocity.

    Returns
    -------
    U : array_like
        The corresponding group velocity in km/s.
    """
    c = # Compute phase velocity
    dcdT = # Compute derivative of phase velocity
    U = # Compute group velocity
    return U

def unit_test():
    assert( abs(compute_C(250) - 4.92509) < 1.e-2), "Phase velocity is wrong"
    assert( abs(compute_U(250) - 3.65281) < 1.e-2), "Group velocity is wrong"
    return True

lpass = unit_test()
if (lpass):
    print("Passed unit test")
else:
    print("Failed unit tests")

```

    Passed unit test

```python
# Plot the result
T = linspace(50,500,451)
c = compute_C(T) # Phase velocity
U = compute_U(T) # Group velocity
plt.figure(figsize=(8,6))
plt.plot(T, c, label='Phase')
plt.plot(T, U, label='Group')
plt.ylabel('Velocity (km/s)')
plt.xlabel('Period (s)')
plt.title("Phase and Group Velocity")
plt.xlim(50,500)
plt.grid(True)
plt.legend()
plt.show()
```

# Question: Is the phase or group velocity faster?

PUT YOUR ANSWER HERE

# Simplified Surface Wave Synthetics

In class we derived a form for Rayleigh waves at the free surface

\begin{equation}
  u_x(t, x, z=0) = 0.42 A k_x \sin (\omega t - k_x x_1)
\end{equation}

\begin{equation}
  u_z(t, x, z=0) = 0.62 A k_x \cos (\omega t - k_x x_1)
\end{equation}

where a Poisson solid was assumed.  Here, we'll set the amplitude to $1$.  Additionally, in our derivation we assumed that the medium was a halfspace.  Here, we will allow the wavenumber to vary with frequency.  In practice Rayleigh waves are disperisive so this is an appropriate thing to do.  The phase velocities were previously defined.

Of course, each solution is monochromatic.  Hence, we have to compute the superposition of all frequencies, i.e.,

\begin{equation}
  u_x(t, x, 0) = \sum_{\omega=\omega_0}^{\omega_1}  0.42 \frac{\omega}{c} \sin \left ( \omega t - \frac{\omega}{c} x_1 \right )
\end{equation}

\begin{equation}
  u_x(t, x, 0) = \sum_{\omega=\omega_0}^{\omega_1} 0.62 \frac{\omega}{c} \cos \left ( \omega t - \frac{\omega}{c} x_1 \right )
\end{equation}

where $\omega_0$ and $\omega_1$ are the angular cutoff frequencies.  It remains an open question what the frequency sampling should be.  From Fourier analysis one can show that the temporal duration is simply related to the frequency resolution is related the temporal sampling period by $ \Delta f = \frac{1}{N \Delta T}$ or

\begin{equation}
  \Delta \omega = \frac{2 \pi }{N \Delta T}
\end{equation}

where $N$ is the number of samles and $\Delta T$ the sampling period.  The number of frequencies is then found with

\begin{equation}
  N_\omega =\operatorname{int} \left \{ \frac{\omega_1 - \omega_0}{\Delta \omega} \right \} + 1
\end{equation}

## Implementation 1

Therefore, we can rewrite the summations as

\begin{equation}
  u_x(t, x, 0) = 0.42 \sum_{j=0}^{N_\omega-1} \frac{(\omega_0 + j \Delta \omega)}{c}
                 \sin \left ((\omega_0 + k \Delta \omega) \cdot \left (t - \frac{1}{c(T)} x_1  \right )\right )
\end{equation}

\begin{equation}
  u_x(t, x, 0) = 0.62 \sum_{j=0}^{N_\omega-1} \frac{(\omega_0 + j \Delta \omega)}{c}
                 \cos \left ((\omega_0 + k \Delta \omega) \cdot \left (t - \frac{1}{c(T)} x_1  \right )\right )
\end{equation}

where $c$ is the apparent velocity in $x$ which is a function of period, and by extension, angular frequency.

## Implementation 2

Such an implementation can yield seismograms that ring.  This is because we are effectively applying a bandpass filter with sharp cutoff frequencies.  To mitigate this ringing we can apply a window function which tapers to $0$ at the cutoff frequencies.  To do this introduce a Hann window function defined by

\begin{equation}
  W(\omega) = \sin^2 \left (\pi \frac{\omega - \omega_0}{\omega_2 - \omega_1} \right )
\end{equation}

Notice, that this is $0$ at $\omega = \omega_0$ and $\omega = \omega_1$.  It is therefore sufficient to use a discrete version

\begin{equation}
  W_j = \frac{1}{2} - \frac{1}{2} \cos \left ( \frac{2\pi j}{N_f - 1} \right ); \, \qquad j=0,1, \cdots N_f - 1
\end{equation}

which is the Hann window defined in Scipy.  Thus, we have a second implementation

\begin{equation}
  u_x(t, x, 0) = 0.42 \sum_{j=0}^{N_\omega-1} \frac{(\omega_0 + j \Delta \omega)}{c}
                 \sin \left ((\omega_0 + k \Delta \omega) \cdot \left (t - \frac{1}{c(T)} x_1  \right )\right ) W_j
\end{equation}

\begin{equation}
  u_x(t, x, 0) = 0.62 \sum_{j=0}^{N_\omega-1} \frac{(\omega_0 + j \Delta \omega)}{c}
                 \cos \left ((\omega_0 + k \Delta \omega) \cdot \left (t - \frac{1}{c(T)} x_1  \right )\right ) W_j
\end{equation}

## Your Turn:

Implement a function that computes Rayleigh wave synthetics on both the vertical and radial channels given the phase velocity curve defined in the previous problem.

```python
def rayleigh(gcarc=asarray([0, 30, 90, 150]), Tmax=1.5*3600, dt=5,
             f0=0.002, f1=0.02, df=None, useWindow=True):
    """
    Computes the radial and vertical Rayleigh waves at the free surface.  Here, the
    phase velocity is modeled by compute_C which assumes:
    c(T) = 4.020 - 1.839 \times 10^{-3} T + 3.071 \times 10^{-5} T^2 - 3.549 \times 10^{-8} T^3

    Parameters
    ----------
    gcarc : array_like
        The receiver distances in degrees at which the seismograms
        are to be computed.  This has dimension equal to the number of receivers.
    Tmax : float
        The max modeling time in seconds.
    dt : float
        The sampling period of the time domain seismograms in seconds.
    f0 : float
        The low cutoff frequency in Hz.
    f1 : float
        The high cutoff frequency in Hz.
    df : float
        The desired sampling frequency in Hz.  If not specified the default will be
        analytically computed.  Choosing a small sampling frequency can mitigate
        wraparound.
    useWindow : bool
        If True then use a Hann window function to reduce ringing in time domain.

    Returns
    -------
    times : array_like
        An array of times in seconds at which the synthetics are defined.  This
        has dimension [nt] where nt is the number of time samples.
    gatherr : matrix
        A matrix of radial channel seismograms tabulated at distances specified with
        gcarc with dimension [nrec x nt] where nrec is the number of receivers
        and nt is the number of time samples.
    gatherv : matrix
        A matrix of vertical channel seismograms tabulated at distances specified with
        gcarc with dimension [nrec x nt] where nrec is the number of receivers
        and nt is the number of time samples.
    """
    # Warn on aliasing
    if (f1 > 1./(2.0*dt)):
        print("Warning - signal will be aliased")
    # Initialize parameters for modeling
    nrec = len(gcarc)                      # Number of receivers
    x = deg2km(gcarc)                      # Convert the great circle distances to km
    nt = int((Tmax - 0)/dt) + 1            # Compute the number of samples in the seismogram
    times = linspace(0, Tmax, nt)          # Compute the times at which to tabulate the response
    omega0 =                               # Compute the lowcut angular frequency
    omega1 =                               # Compute the highcut angular frequency
    # If df is not given then estimate it
    if (df is None):
        domega = 2.0*pi/(nt*dt)
    else:
        domega = 2.0*pi*df                 # Convert to angular frequency
    nf = int((omega1 - omega0)/domega) + 1 # Number of modeling frequencies
    omega =                            )   # Compute the modeling frequencies with linspace
    # Precompute the phase velocities and wavenumbers
    T =                                    # Compute the periods at which to evaluate c(T)
    c =                                    # Compute the phase velocities for each frequency with compute_C(T)
    k =                                    # Compute wavenumber in x from omega and c
    p =                                    # Compute the apparent slowness in x from c
    # Handle window function
    if (useWindow):
        w = hann(len(omega))               # Have SciPy calculate the Hann window
    else:
        w = ones(len(omega))               # Dirty trick to write same code for Implementaiton 1 and 2
    # Set space for the gather
    gatherr = zeros([nrec, nt])
    gatherz = zeros([nrec, nt])
    # Model each seismogram
    for irec in range(nrec):
        seisr = zeros(nt) # Allocate space to radial resonse
        seisz = zeros(nt) # Allocate space to vertical response
        # Loop on each temporal index
        for it in range(nt):
            seisr[it] =   # Compute radial synthetic at time[it]
            seisz[it] =   # Compute vertical synthetic at time [it]
        gatherr[irec,:] = seisr[:] # Copy radial repsonse to gather
        gatherz[irec,:] = seisz[:] # Copy vertical response to gather
    # End loop on receivers
    return times, gatherr, gatherz

```

```python
# Compute the rayleigh wave response on the vertical component
gcarc = asarray([0, 30, 90, 150])
times, gatherr, gatherz = rayleigh(Tmax=6500, useWindow=True)
nrec = gatherr.shape[0]
# Plot the gather
maxresp = max(amax(abs(gatherr)), amax(abs(gatherz)))
yticks = []
ylabels = []
plt.figure(figsize=(12,8))
for irec in range(nrec):
    yshift = irec*(2*maxresp)
    plt.plot(times, gatherz[irec,:] + yshift, color='black')
    #plt.plot(times, gatherr[irec,:] + yshift, color='red',   linewidth=1)
    yticks.append(yshift)
    ylabels.append("%s"%str(gcarc[irec]))
plt.yticks(yticks, ylabels)
plt.xlabel("Time (s)")
plt.ylabel("Distance (Degrees)")
plt.title("Vertical Component Rayleigh Waves As A Function of Distance")
plt.xlim(0, min(5500, max(times)))
plt.grid(True)
plt.show()

plt.figure(figsize=(12,4))
plt.plot(times, gatherz[2,:], color='black', label='Vertical')
plt.plot(times, gatherr[2,:], color='red',   label='Radial')
plt.xlim(2200, 3000)
plt.legend()
plt.grid(True)
plt.ylabel("Response")
plt.xlabel("Time (s)")
plt.title("Vertical and Radial Channel at Distance %.0f (deg)"%(gcarc[2]))
plt.show()
```

# Question: Is this Rayleigh wave dispersive?

PUT YOUR ANSWER HERE

Now we put the synthetics through a series of narrow band filters to estimate the Rayleigh wave onset time.  This is how the group velocity could be computed in practice.

```python
# Next lets pass this through some narrow band filters
resp = zeros(len(times))
resp[:] = gatherz[2,:]

# Compute nyquist
fnyq = 1.0/(2*(times[1] - times[0]))
# Filter design  IIR bandpass filters
W = [1./60., 1./40]/fnyq
b50,  a50  = iirfilter(N=2, Wn=W, btype='band', output='ba')
W = [1./80., 1./60]/fnyq
b70,  a70  = iirfilter(N=2, Wn=W, btype='band', output='ba')
W = [1./110, 1./90]/fnyq
b100, a100 = iirfilter(N=2, Wn=W, btype='band', output='ba')
W = [1./160, 1./140]/fnyq
b150, a150 = iirfilter(N=2, Wn=W, btype='band', output='ba')
W = [1./210, 1./190]/fnyq
b200, a200 = iirfilter(N=2, Wn=W, btype='band', output='ba')
W = [1./410, 1./390]/fnyq
b400, a400 = iirfilter(N=2, Wn=W, btype='band', output='ba')

# Apply the filters
resp50  = lfilter(b50,  a50,  resp)
resp70  = lfilter(b70,  a70,  resp)
resp100 = lfilter(b100, a100, resp)
resp150 = lfilter(b150, a150, resp)
resp200 = lfilter(b200, a200, resp)
resp400 = lfilter(b400, a400, resp)

# Rescale so everyone is the same size
resp50  = resp50/max(abs(resp50))
resp70  = resp70/max(abs(resp70))
resp100 = resp100/max(abs(resp100))
resp150 = resp150/max(abs(resp150))
resp200 = resp200/max(abs(resp200))
resp400 = resp400/max(abs(resp400))

# Plot
plt.figure(figsize=(12,6))
plt.ylim(-2, 12)
plt.yticks([0, 2, 4, 6, 8, 10], ["50s", "70s", "100s", "150s", "200s", "400s"])
plt.plot(times, resp50,    label='50s')
plt.plot(times, resp70 +2,    label='70s')
plt.plot(times, resp100+4, label='100s')
plt.plot(times, resp150+6, label='150s')
plt.plot(times, resp200+8, label='200s')
plt.plot(times, resp400+10,label='400s')
plt.grid(True)
plt.xlim(2200, 3500)
plt.xlabel("Time (s)")
plt.ylabel("Synthetic Dominant Period")
plt.title("Synthetics Separated By Dominant Period")
plt.show()
```

# Question: Based on your plot of phase velocity and group velocity, are  seismograms better representations of group velocity or phase velocity?

PUT YOUR ANSWER HERE

```python

```