# Information Retrieval in High Dimensional Data

# Assignment #2, 17.11.2017

# Principal Component Analysis
Group Number: G10

Group Members:
- Achtner Martin
- Arifi Ridon
- Ehrhardt Daniel
- Fichtner Lukas

## Task 1

Let $\mathbf{X} \in \mathbb{R}^{pxN}, p < N$ be a centered matrix of $p$ $N$-dimensional data samples with the full-size SVD $\mathbf{X} = \mathbf{U\Sigma} \mathbf{V}^T$. Assume that the singular values are sorted in a descending manner, i.e. $\sigma_{1,1} \geq \dots \geq \sigma_{p,p}$.

+ Provide a normalized vector $\hat{\mathbf{s}} \in \mathbb{R}^p$, such that

\begin{equation}\hat{\mathbf{s}} = \arg\max\limits_{\mathbf{s} \text{ s.t.} \|\mathbf{s}\|=1}\,\mathbf{s}^T\mathbf{\Sigma\Sigma}^T\mathbf{s}.\end{equation}

$ \rightarrow \mathbf{\Sigma\Sigma^T} = \mathbf{\Lambda}$ is a diagonal matrix with the biggest eigenvalue ${\sigma_{1,1}}^2$ in the upper left corner.
To maximize the equation $\mathbf{s}^T\mathbf{\Sigma\Sigma}^T\mathbf{s}$ we have to weigh the biggest eigenvalue with the maximum weight possible (which in this case is 1), so $\hat{\mathbf{s}}$ has ${\sigma_{1,1}}^2$ as its first entry, and the remaining entries are equal to zero.

$ \rightarrow \hat{\mathbf{s}} = (s_1, s_2\dots,s_p) = (1,0,\dots,0)$

$(s_k = 1$ if $k=1$, else $s_k = 0)$

```python

```

+ Show that the empirical variance of the inner products of the columns of $\mathbf{X}$ with a normalized vector $\mathbf{a}$,

\begin{equation}\frac{1}{N}\sum\limits_{i=1}^N (\mathbf{a}^T \mathbf{x}_i)^2 = \frac{1}{N}\mathbf{a}^T\mathbf{XX}^T\mathbf{a},\end{equation}

is maximized when $\mathbf{a}$ is set to the first column of $\mathbf{U}$, i.e. $\mathbf{a}=\mathbf{u}_1$ (note that $\|\mathbf{a}\| = 1$).
Hint: Write $\mathbf{a}$ as a linear combination of the columns of $\mathbf{U}$. Verify that such a representation does not affect the norm constraint.

$\rightarrow$ Let's represent $\mathbf{a}$ as a linear combination:

\begin{equation} \mathbf{a} = a_1 e_1 + a_2 e_2 + \dots + a_p e_p \end{equation}

We denote $e_k$ as the k-th euclidean basis vector. They form an orthonormal basis of $\mathbb{R}^p$.

$\| \mathbf{a}\| = 1$, if the coefficients $a_i$ satisfy the condition:
$\sqrt{a_1^2 + a_2^2 + \dots + a_p^2} = 1$.

We now conduct a change of basis, s.t. $\mathbb{R}^p$ is represented by another set of orthonormal vectors, namely the column vectors of $\mathbf{U}$.

\begin{equation} \mathbf{a} = \alpha_1 u_1 + \alpha_2 u_2 + \dots + \alpha_k u_k  \end{equation}

ist again has length 1 if the coefficients $\alpha_i$ are chosen like this:
\begin{equation}\|\mathbf{a}\| = \sqrt{\alpha_1^2 + \alpha_2^2 + \dots + \alpha_k^2} = 1
\end{equation}

\begin{equation}\mathbf{X} = \mathbf{U \Sigma V}^T \end{equation}

\begin{equation}\mathbf{X}^T = \mathbf{V \Sigma}^T \mathbf{U}^T \end{equation}

\begin{equation}\mathbf{XX}^T = \mathbf{U \Sigma V}^T \mathbf{V \Sigma}^T \mathbf{U}^T \end{equation}

with $\mathbf{V}^T \mathbf{V} = \mathbf{I}$:

\begin{equation}= \mathbf{U \Sigma \Sigma}^T \mathbf{U}^T \end{equation}

\begin{equation}= \mathbf{U \Lambda U}^T \end{equation}

\begin{equation}\frac{1}{N}\sum\limits_{i=1}^N (\mathbf{a}^T \mathbf{x}_i)^2 = \frac{1}{N}\mathbf{a}^T\mathbf{(U \cdot \Lambda \cdot U^T)}^T\mathbf{a},\end{equation}

If we denote the products $\mathbf{a}^T \mathbf{U} = \mathbf{s}^T$ and $\mathbf{U}^T \mathbf{a} = \mathbf{s}$ as vectors, we can reduce the problem to the problem of the task before, which has been already solved.

In order to get a 1 as the first entry of $\mathbf{s}$, $\mathbf{a}$ has to be set to $\mathbf{u}_1$ (the first column vector of $\mathbf{U}$), because $\mathbf{U}$ is a orthonormal matrix.

Hence $\mathbf{u}_i^T \mathbf{u}_j = 1$, if $i = j$ and 0 else.

## Task 2

For this task download the modified version of the <i>Yale Face Database B</i> provided on Moodle (task2_data.zip). The Yale Face Database B consists of single light source images of 10 subjects, each seen in different poses and illumination conditions. In the provided form the database is divided into 5 subsets. In subset 0 the subject is illuminated by an almost frontal light source, while for the subset 1-4 the light source is gradually moved along the horizon. Subset 0 will serve as training set, while subsets 1-4 are used for testing.

+ Write a function that takes as an input matrix $\mathbf{T}$ of vectorized images from subset 0. The ouput of this function are the 20 first singular vectors $\mathbf{U}[:, 1],\dots,\mathbf{U}[:, 20]$. Display the first 3 vectors as images, i.e., reshape them to size 50 x 50 and display them.

```python
# Import libs
import imageio
import matplotlib.pyplot as plt
import numpy as np
import os
```

```python
# Check for number and size of images
path = 'yaleBfaces/subset0/'
filenames = sorted((fn for fn in os.listdir(path) if fn.endswith('.png')))
im0 = imageio.imread(path + filenames[0])
print('Number of image files in subset 0: {}'.format(len(filenames)))
print('Size of a picture: {}'.format(im0.shape))
```

    Number of image files in subset 0: 70
    Size of a picture: (50, 50)

```python
# Create Matrix T
T = np.zeros((2500, 70))
for idx, fn in enumerate(filenames):
    im = imageio.imread(path + fn)
    T[:, idx] = np.reshape(im, 2500)
```

```python
# Center Matrix T
mu = np.mean(T, axis=1)
mu = np.expand_dims(mu, axis=1)
T = T - mu
```

```python
# Function that returns 20 vectors of Matrix U of SVD of T
def get_singular_vectors(T):
    U, s, V = np.linalg.svd(T, compute_uv=True)
    return U[:, :20]
```

```python
# Compute SVD of T
singular_vectors = get_singular_vectors(T)
```

```python
# Display the first three singular vectors as images
u0 = np.reshape(singular_vectors[:, 0], (50, 50))
u1 = np.reshape(singular_vectors[:, 1], (50, 50))
u2 = np.reshape(singular_vectors[:, 2], (50, 50))

ax1 = plt.subplot(131)
plt.imshow(u0)
ax2 = plt.subplot(132)
plt.imshow(u1)
ax3 = plt.subplot(133)
plt.imshow(u2)
plt.show()
```

+ Write a function that takes as an input the training set $\mathbf{T}$ composed of vectorized pictures from subset 0), a vector containing the labels of the training set (i.e., if the $i$-th sample belongs to class $j$, the $i$-th entry of the label vector is $j$. In this exercise $j$ is an integer between 1 and 10), the test samples $S$ (a matrix composed of vectorized pictures from subsets 1-4) and the corresponding labels (in a separate vector), the 20 singular vectors from the first step, and the parameter $k$ that denotes how many of the PCs are used. Use the euclidian distance to classify each sample image based on its three nearest neighbors. (This is done by comparing the test samples with the training samples in the reduced space.) As an output give the fraction of images from $S$ that were misclassified, i.e., the error rate. Repeat this for subsets 1 through 4 and for $k= 1,\dots,20$. Plot the error rate for each subset.

```python
def missclassified(T, training_labels, S, test_labels, singular_vectors, k):
    # Transform data via Transformation matrix U_k
    reduced_training_data = np.dot(singular_vectors[:, :k].T, T)
    reduced_test_data = np.dot(singular_vectors[:, :k].T, S)

    assigned_labels = np.zeros(len(test_labels))
    for i in range(len(test_labels)):
        testset = reduced_test_data[:, i]
        euclidian_distances = np.zeros(len(training_labels))
        for j in range(len(training_labels)):
            trainingset = reduced_training_data[:, j]
            # Save euclidian distances to every other image
            euclidian_distances[j] = np.linalg.norm(trainingset-testset)
        # Get indices of nearest neighbours
        nearest_indices = np.argsort(euclidian_distances)[:3]
        #print(nearest_indices)
        nearest_labels = np.array(training_labels[nearest_indices], dtype='int64')
        counts = np.bincount(nearest_labels)
        #print(counts)
        #print(test_labels[i], np.argsort(counts)[-1])
        assigned_labels[i] = np.argsort(counts)[-1]
        equals = assigned_labels != test_labels
    return np.sum(equals)/len(test_labels)
```

```python
# Assign training set labels
path = 'yaleBfaces/subset0/'
filenames = sorted((fn for fn in os.listdir(path) if fn.endswith('.png')))
training_labels = np.zeros(len(filenames), dtype='int64')

for idx, fn in enumerate(filenames):
    training_labels[idx] = int(fn[6:8])
#training_labels
```

```python
# Read image set and compute error rates
error_rates = np.zeros((4, 20))
for i in range(1, 5):
    path = 'yaleBfaces/subset{}/'.format(i)
    filenames = sorted((fn for fn in os.listdir(path) if fn.endswith('.png')))
    test_labels = np.zeros(len(filenames), dtype='int64')
    S = np.zeros((2500, len(filenames)))
    for idx, fn in enumerate(filenames):
        test_labels[idx] = int(fn[6:8])
        im = imageio.imread(path + fn)
        S[:, idx] = np.reshape(im, 2500)
    # Center test matrix S with mu from before
    S = S - mu
    for k in range(1, 21):
        error_rates[i-1, k-1] = missclassified(T, training_labels, S, test_labels , singular_vectors, k)
```

```python
fig = plt.figure(figsize=(12, 10))
for i in range(4):
    plt.plot(range(1, 21),error_rates[i, :])

#plt.legend(('Subset1', 'Subset2', 'Subset3', 'Subset4'))
plt.show()
```

+ Repeat the above experiment without using the first three singualr vectors, i.e., use $k = 1,\dots,17$ singular vectors starting from the 4-th. Plot the error rate as before. How do you explain the difference in recognition rate?

```python
# Read image set and compute error rates
error_rates = np.zeros((4, 17))
for i in range(1, 5):
    path = 'yaleBfaces/subset{}/'.format(i)
    filenames = sorted((fn for fn in os.listdir(path) if fn.endswith('.png')))
    test_labels = np.zeros(len(filenames), dtype='int64')
    S = np.zeros((2500, len(filenames)))
    for idx, fn in enumerate(filenames):
        test_labels[idx] = int(fn[6:8])
        im = imageio.imread(path + fn)
        S[:, idx] = np.reshape(im, 2500)
    # Center test matrix S with mu from before
    S = S - mu
    for k in range(1, 18):
        error_rates[i-1, k-1] = missclassified(T, training_labels, S, test_labels , singular_vectors[:, 3:20], k)
```

```python
fig = plt.figure(figsize=(12, 10))
for i in range(4):
    plt.plot(range(1, 18),error_rates[i, :])

plt.legend(('Subset1', 'Subset2', 'Subset3', 'Subset4'))
plt.show()
```

$\rightarrow$ The recognition rate is much better now because the first singular vectors correspond to the differences in the lighting conditions (which we want to ignore) and not in the differences of the peoples' faces we are interested in.


Водопьян А.О. Хабибуллин Р.А. 2019 г.

## Объемный коэффициент нефти

### Корреляция при давлении выше давления насыщения

Корреляция для объемного коэффициента нефти при давлении выше давления насыщения в некоторых источниках указывается, что она принадлежит Стендингу, в некоторых Vasquez & Beggs. На самом деле это не корреляция, так как природа ее происхождения не статистическая, а вполне себе физическое уравнение.

$$ b_o = b_{ob} \cdot \exp(c_o(p_b - p)) $$

где:

$ b_o $ - объемный коэффициент нефти при давлении p, м3/м3

$ b_{ob} $ - объемный коэффициент нефти при давлении насыщения, м3/м3

$ c_o $ - сжимаемость нефти, 1/МПа

$ p $ - давление, МПа

$ p_b $ - давление насыщения, МПа

```python
import matplotlib.pyplot as plt
import matplotlib as mpl
import numpy as np
from sympy import *
init_printing()
import sys
sys.path.append('../')
import uniflocpy.uPVT.PVT_correlations as PVT
```

```python
# Уравнение для объемного коэффициента нефти реализовано
# в виде функции unf_fvf_VB_m3m3_above в модуле PVT_correlations.
# Подробные данные по функции включая исходный код приведены ниже

PVT.unf_fvf_VB_m3m3_above??
```

```python
# параметры определяющие диапазоны значений для построения графиков
p_set=np.arange(8,11,0.25)
co_set=np.arange(1 * 10 ** (-3),4 * 10 ** (-3),10 ** (-3))
bob = 1.2
pb = 8
# функция для автоматизации построения графиков по давлению насыщения
def prep_plot(func,p_set,co_set,pb,bob,plot_title,plot_xlab,plot_ylab):
    for co in co_set:
        b_o_set=[]
        for p in p_set:
            b_o_set.append(func(bob,co,pb,p))
        plt.plot(p_set, b_o_set, label='co ={}'.format(co))
    plt.title(plot_title)
    plt.ylabel(plot_ylab, color = 'black')
    plt.xlabel(plot_xlab, color = 'black')
    plt.legend()
```

```python
# код для построения графиков
plt.figure(figsize=(15,8))
f = PVT.unf_fvf_VB_m3m3_above
prep_plot(f,p_set,co_set,pb,bob,
              'Объемный коэффициент нефти от давления (выше давления насыщения)',
               '$P, MPa$',
               '$b_o, м^3/м^3$')

# рисуем все
plt.grid()
plt.show()
```

### Корреляция Маккейна при давлении меньше или равном давлению насыщения

Уравнение выводится из материального баланса и не является корреляцией.

$$ b_o = \left( \frac{ \rho_{STO} + 0.01357 R_s \gamma_g}{\rho_{or}}\right) $$

где:

$ b_o $ - объемный коэффициент нефти при давлении p, м3/м3

$ \rho_{STO} $ - плотность дегазированной нефти, фунт/фт3 (кг/м3)

$ R_s $ - газосодержание при давлении p, фт3/баррель (м3/м3)

$ \gamma_g $ - плотность газа относительно воздуха

$ \rho_{or} $ - плотность пластовой нефти, фунт/фт3 (кг/м3)

#### Внутри функции уже реализован перевод величин, единицы измерения в скобках - входные параметры в функцию

```python
# Уравнение для объемного коэффициента нефти реализовано
# в виде функции unf_fvf_Mccain_m3m3_below в модуле PVT_correlations.
# Подробные данные по функции включая исходный код приведены ниже

PVT.unf_fvf_Mccain_m3m3_below??
```

```python
# параметры определяющие диапазоны значений для построения графиков
rs_set=np.arange(0,300,25)
rho_set=np.arange(600,850,50)
rho_sto = 800
gamma_gas = 0.8
# функция для автоматизации построения графиков по давлению насыщения
def prep_plot(func,rs_set,rho_set,gamma_gas,rho_sto,plot_title,plot_xlab,plot_ylab):
    for rho in rho_set:
        b_o_set=[]
        for rs in rs_set:
            b_o_set.append(func(rho_sto,rs,rho,gamma_gas))
        plt.plot(rs_set, b_o_set, label='rho ={}'.format(rho))
    plt.title(plot_title)
    plt.ylabel(plot_ylab, color = 'black')
    plt.xlabel(plot_xlab, color = 'black')
    plt.legend()
```

```python
# код для построения графиков
plt.figure(figsize=(15,8))
f = PVT.unf_fvf_Mccain_m3m3_below
prep_plot(f,rs_set,rho_set,gamma_gas,rho_sto,
              'Объемный коэффициент нефти от давления (ниже давления насыщения)',
               '$Rs, м^3/м^3$',
               '$b_o, м^3/м^3$')

# рисуем все
plt.grid()
plt.show()
```

Этот график вообще говоря неверный,он построен для понимания, потому что при разном Rs будет разная плотность и график будет нелинейным.

notebooks/uPVT. Oil formation volume factor.ipynb


# High-Order Example

[](https://mybinder.org/v2/gh/teseoch/fem-intro/master?filepath=fem-intro-high-order.ipynb)

Run it with binder!

```python
import numpy as np
import scipy.sparse as spr
from scipy.sparse.linalg import spsolve

import plotly.graph_objects as go
```

As in the linear case, the domain is $\Omega = [0, 1]$, which we discretize with $n_{el}$ segments (or elements) $s_i$.

Now we also create the high-order nodes.

Note that we append them at the end.

```python
#domain
omega = np.array([0, 1])

#number of bases and elements
n_el = 10

#Regular nodes, as before
s = np.linspace(omega[0], omega[1], num=n_el+1)
# s = np.cumsum(np.random.rand(n_elements+1))
s = (s-s[0])/(s[-1]-s[0])
```

```python
# we now pick the order
order = 3

nodes = s

#more bases
n_bases = n_el + 1 + n_el*(order-1)

#create the nodes for the plots
for e in range(n_el):
    #For every segment, we create order + 1 new high-order nodes
    tmp = np.linspace(s[e], s[e+1], num=order+1)
    #exclude the first and last since they already exists
    tmp = tmp[1:-1]

    #and append at the end of nodes
    nodes = np.append(nodes, tmp)
```

Plot, in orange the high-order nodes, in blue the linear nodes.

```python
go.Figure(data=[
    go.Scatter(x=s, y=np.zeros(s.shape), mode='lines+markers'),
    go.Scatter(x=nodes[n_el+1:], y=np.zeros(nodes.shape), mode='markers')
])
```

# Local bases

As in the linear case, we define the **reference element** $\hat s= [0, 1]$, a segment of unit length.

On each element we now have `order+1` (e.g., 2 for linear, 3 for quadratic) **non-zero** local bases.

We define their "piece" on $\hat s$.

It is important to respect the order of the nodes: the first 2 bases are always for the endpoints, and the others are ordered left to right.

Definition of linear bases, same as before

```python
def hat_phi_1_0(x):
    return 1-x
def hat_phi_1_1(x):
    return x
```

Definition of quadratic bases

```python

def hat_phi_2_0(x):
    return 2*(x-0.5)*(x-1)
def hat_phi_2_1(x):
    return 2*(x-0)*(x-0.5)
def hat_phi_2_2(x):
    return -4*(x-0.5)**2+1
```

Definition of cubic bases

```python
def hat_phi_3_0(x):
    return -9/2*(x-1/3)*(x-2/3)*(x-1)
def hat_phi_3_1(x):
    return 9/2*(x-0)*(x-1/3)*(x-2/3)
def hat_phi_3_2(x):
    return 27/2*(x-0)*(x-2/3)*(x-1)
def hat_phi_3_3(x):
    return -27/2*(x-0)*(x-1/3)*(x-1)
```

Utility function to return the list of functions

```python
def hat_phis(order):
    if order == 1:
        return [hat_phi_1_0, hat_phi_1_1]
    elif order == 2:
        return [hat_phi_2_0, hat_phi_2_1, hat_phi_2_2]
    elif order == 3:
        return [hat_phi_3_0, hat_phi_3_1, hat_phi_3_2, hat_phi_3_3]
```

We can now plot the `order+1` local bases, same code as before.

Note that the first two bases correspond to the end-points, and the others are ordered.

```python
x = np.linspace(0, 1)

data = []

tmp = hat_phis(order)

for o in range(order+1):
    data.append(go.Scatter(x=x, y=tmp[o](x), mode='lines', name="$\hat\phi_{}$".format(o)))

go.Figure(data=data)
```

We use `sympy` to compute the gradients of the local bases.

```python
import sympy as sp

xsym = sp.Symbol('x')

def grad_hat_phis(order):
    #For linear we need to get the correct size
    if order == 1:
        return [lambda x : -np.ones(x.shape), lambda x : np.ones(x.shape)]

    res = []
    tmp = hat_phis(order)

    for fun in tmp:
        res.append(sp.lambdify(xsym, fun(xsym).diff(xsym)))
    return res
```

Plotting gradients

```python
x = np.linspace(0, 1)

data = []

tmp = grad_hat_phis(order)

for o in range(order+1):
    data.append(go.Scatter(x=x, y=tmp[o](x), mode='lines', name="$\hat\phi_{}$".format(o)))

go.Figure(data=data)
```

# Basis construction

This code is exacly as before.

The only difficulty is the local to global mapping:
- the first 2 nodes are always the same

$$g_e^0 = e \qquad\mathrm{and}\qquad g_e^1=g+1$$
- the others are

$$g_e^i = n_{el} + e (\mathrm{order}-1) + i.$$

```python
elements = []
for e in range(n_el):
    el = {}

    el["n_bases"] = order+1

    #2 bases
    el["phi"] = hat_phis(order)
    el["grad_phi"] = grad_hat_phis(order)

    #local to global mapping
    high_order_nodes = list(range(n_el + 1 + e*(order-1), n_el + e*(order-1) + order))
    el["loc_2_glob"] = [e, e+1] + high_order_nodes

    #geometric mapping
    el["gmapping"] = lambda x, e=e : s[e] + x*(s[e+1]-s[e])
    el["grad_gmapping"] = lambda x : (s[e+1]-s[e])

    elements.append(el)
```

We define a function to interpolate the vector $\vec{u}$ using the local to global, geometric mapping, and local bases to interpolate the data, as before.

```python
def interpolate(u):
    uinterp = np.array([])
    x = np.array([])

    xhat = np.linspace(0, 1)

    for e in range(n_el):
        el = elements[e]

        uloc = np.zeros(xhat.shape)

        for i in range(el["n_bases"]):
            glob_node = el["loc_2_glob"][i]
            loc_base = el["phi"][i]

            uloc += u[glob_node] * loc_base(xhat)

        uinterp = np.append(uinterp, uloc)
        x = np.append(x, el["gmapping"](xhat))

    return x, uinterp
```

We can generate a random vector $\vec{u}$ and use the previous function. This will interpolate all nodes.

```python
u = np.random.rand(n_bases)

x, uinterp = interpolate(u)

go.Figure(data=[
    go.Scatter(x=x, y=uinterp, mode='lines'),
    go.Scatter(x=nodes, y=u, mode='markers'),
])
```

# Assembly

We are now ready the assemble the global stiffness matrix, which is exacly as before.

```python
import quadpy
scheme = quadpy.line_segment.gauss_patterson(5)

rows = []
cols = []
vals = []

for e in range(n_el):
    el = elements[e]

    for i in range(el["n_bases"]):
        for j in range(el["n_bases"]):
            val = scheme.integrate(
                lambda x:
                el["grad_phi"][i](x) * el["grad_phi"][j](x) / el["grad_gmapping"](x),
                [0.0, 1.0])

            rows.append(el["loc_2_glob"][i])
            cols.append(el["loc_2_glob"][j])
            vals.append(val)

rows = np.array(rows)
cols = np.array(cols)
vals = np.array(vals)

L = spr.coo_matrix((vals, (rows, cols)))
L = spr.csr_matrix(L)
```

We set the rows `0` and `n_el` to identity for the boundary conditions.

```python
for bc in [0, n_el]:
    _, nnz = L[bc,:].nonzero()
    for j in nnz:
        if j != bc:
            L[bc, j] = 0.0
    L[bc, bc] = 1.0
```

We set the right-hand side to zero, and set the two boundary conditions to 1 and 4.

```python
f = np.zeros((n_bases, 1))
f[0] = 1
f[n_el] = 4
```

We now solve $L\vec{u}=f$ for $\vec{u}$.

```python
u = spsolve(L, f)
```

We now plot the solution. We expect a line, independently of `order`!

```python
x, uinterp = interpolate(u)

go.Figure(data=[
    go.Scatter(x=x, y=uinterp, mode='lines', name="solution"),
    go.Scatter(x=nodes, y=u, mode='markers', name="$u$"),
])
```

# Mass Matrix

This is exactly as before!

```python
rows = []
cols = []
vals = []

for e in range(n_el):
    el = elements[e]

    for i in range(el["n_bases"]):
        for j in range(el["n_bases"]):
            val = scheme.integrate(
                lambda x:
                el["phi"][i](x) * el["phi"][j](x) * el["grad_gmapping"](x),
                [0.0, 1.0])

            rows.append(el["loc_2_glob"][i])
            cols.append(el["loc_2_glob"][j])
            vals.append(val)

rows = np.array(rows)
cols = np.array(cols)
vals = np.array(vals)

M = spr.coo_matrix((vals, (rows, cols)))
M = spr.csr_matrix(M)
```

Now we set $\vec{f}=4$ and zero boundary conditions.

```python
f = 4*np.ones((n_bases, 1))
f = M*f

f[0] = 0
f[n_el] = 0
```

We now solve $L\vec{u}=M\vec{f}$ for $\vec{u}$

```python
u = spsolve(L, f)
```

```python
x, uinterp = interpolate(u)

go.Figure(data=[
    go.Scatter(x=x, y=uinterp, mode='lines', name="solution"),
    go.Scatter(x=nodes, y=u, mode='markers', name="$u$"),
])
```

```python

```


```python
%matplotlib widget
```

# Introduction to Jupyter Notebooks and Sympy (symbolic math package)

## Python Package Imports

```python
%load_ext autoreload
%autoreload 2
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import axes3d
import matplotlib.cm as cm
from IPython.display import display, Math, clear_output

import sympy  #Sympy1.2 is from conda-forge: [sudo conda install -c conda-forge sympy]
from sympy import *
from sympy.physics.vector import ReferenceFrame, CoordinateSym
from sympy.vector import CoordSys3D, divergence, curl

import ipyvolume as ipv
import time
from ipywidgets import Output

import helpers

np.seterr(divide='ignore', invalid='ignore');

import json
import matplotlib
s = json.load(open("styles/bmh_matplotlibrc.json"))
matplotlib.rcParams.update(s)
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload

    /Applications/anaconda3/anaconda3/lib/python3.6/site-packages/matplotlib/__init__.py:800: MatplotlibDeprecationWarning: axes.color_cycle is deprecated and replaced with axes.prop_cycle; please use the latter.
      mplDeprecation)

### Check versions of important packages
(Need Sympy Version >=1.2 from conda-forge)

```python
print('Ipyvolume version = {}'.format(ipv.__version__))
print('Sympy version = {}'.format(sympy.__version__))
print('IPyVolume version = {}'.format(ipv.__version__))
```

    Ipyvolume version = 0.4.6
    Sympy version = 1.2
    IPyVolume version = 0.4.6

## Let's check that plotting and the plotting widget are correctly installed

The plot may be hidden with only a "power" button in the upper-right. Click this button to toggle the display of the plot. Note the widgets below the plot that allow you to: 1) 'home' the view; 2) toggle the view forward or backwards; 3) pan the axes; 4) zoom to a rectangle and; 5) save/download the figure.

```python
init_printing()
plt.figure()
plt.plot([1,2,3])
plt.show()
```

    A Jupyter Widget

## Sympy is a Python module for symbolic mathematics.

#### The basics of vectors and fields in Sympy are:
1. Unit vectors are indicated by a . (For example, R.x is the unit vector in the x direction).
2. Scalars are indicated by [] (For example, R[0] is x)

#### Sympy functions and capabilities to be aware of:
From the [**Sympy.vector**](http://docs.sympy.org/latest/modules/vector/index.html) module:

Vector calculus examples specific to electrodynamics include:

1. [divergence](http://docs.sympy.org/latest/modules/vector/api/vectorfunctions.html#divergence)
2. [curl](http://docs.sympy.org/latest/modules/vector/api/vectorfunctions.html#curl)
3. [gradient](http://docs.sympy.org/latest/modules/vector/api/vectorfunctions.html#gradient)
4. [scalar_potential](http://docs.sympy.org/latest/modules/vector/api/vectorfunctions.html#scalar_potential)
5. [scalar_potential_difference](http://docs.sympy.org/latest/modules/vector/api/vectorfunctions.html#potential_difference)
6. [is_conservative](http://docs.sympy.org/latest/modules/vector/api/vectorfunctions.html#is_conservative)

#### Sympy can evaluate integrals and derivates symbolically and numerically:

1. [integrals](http://docs.sympy.org/latest/tutorial/calculus.html?highlight=derivatives#integrals)
2. [derivatives](http://docs.sympy.org/latest/tutorial/calculus.html?highlight=derivatives#derivatives)

#### Sympy supports non-Cartesian coordinate systems:
1. The coordinate system can be represented in non-Cartesian systems, such as spherical or cylindrical:
`A = CoordSys3D('A', transformation='spherical')  # (transformation parameter requires Sympy 1.2)`
[coordinate transformations](http://docs.sympy.org/latest/modules/vector/coordsys.html#transforming-new-system)


# Centripetal Acceleration
> ### $
c^2 = a^2 + b^2 + 2ab\cos\theta = a^2 + b^2 + 2(a \cdot b)
$

```python
import matplotlib.pyplot as plt
import numpy as np
import sympy as sm
fig = plt.figure(figsize=(7,7))
ax = fig.add_subplot()
ax.spines['left'].set_position('center')
ax.spines['bottom'].set_position('center')
ax.set_aspect('equal',adjustable='box')
ax.set_xlim(xmin=-8,xmax=8)
ax.set_ylim(ymin=-8,ymax=8)
ax.set_xticks(np.arange(-8,8))
ax.set_yticks(np.arange(-8,8))
theta = np.linspace(-2*np.pi,2*np.pi,100)
ax.plot(4*np.cos(theta),4*np.sin(theta))

x,b = sm.symbols('x b')
a = -1/sm.tan(1/3)*4
eq = a*x + b
eq1 = sm.Eq((eq).subs(x,sm.cos(1/3)), 4*sm.sin(1/3))
d = sm.solve(eq1,b)[0]
eq1 = eq.subs(b,d)
y = -1/np.tan(1/3)*theta + d
ax.scatter(4*np.cos(1/3),4*np.sin(1/3))
ax.plot([0,4*np.cos(1/3)],[0,4*np.sin(1/3)])
ax.plot(theta,y)
a = -1/sm.tan(1.3)*4
eq = a*x + b
eq1 = sm.Eq((eq).subs(x,sm.cos(1.3)), 4*sm.sin(1.3))
d = sm.solve(eq1,b)[0]
eq2 = eq.subs(b,d)
y = -1/np.tan(1.3)*theta + d
ax.scatter(4*np.cos(1.3),4*np.sin(1.3))
ax.plot([0,4*np.cos(1.3)],[0,4*np.sin(1.3)])
ax.plot(theta,y)
```

> ### $
\Delta{v}^2 = v^2 + v^2 - 2v^2 cos\theta
\\
\Delta{v} = \sqrt{2v^2(1-cost\theta)} = \sqrt{2}\,v \,\sqrt{1-cos\theta}\\
\frac{\Delta{v}}{\Delta{t}} = \frac{\sqrt{2}\,v \,\sqrt{1-cos\theta}}{t}
\frac{\Delta{v}}{\Delta{t}} = \frac{\sqrt{2}\,v\omega \,\sqrt{1-cos\theta}}{t}
$

> ### $
\displaystyle \lim_{t\to 0} \frac{\Delta{v}}{\Delta{t}} = \frac{\sqrt{2}\,v \,\sqrt{1-cos\theta}}{\frac{\theta}{\omega}}
= \displaystyle \lim_{\theta \to 0}\frac{\sqrt{2}\,v\omega \,\sqrt{1-cos\theta}}{\theta}
= \displaystyle \lim_{\theta \to 0} \frac{\sqrt{2}\,v\omega \,\sqrt{1-cos\theta}}{\theta}
\Big(\frac{\sqrt{1+cos\theta}}{\sqrt{1+cos\theta}} \Big)
\\
\therefore \displaystyle \lim_{\theta \to 0} \frac{ \sqrt{2}\,v\omega \,\sin\theta}
{\theta \sqrt{1+cos\theta}}
= \displaystyle \lim_{\theta \to 0} \frac{\sin\theta}{\theta}
\displaystyle \lim_{\theta \to 0} \frac{\sqrt{2} v \omega}{\sqrt{1+cos\theta}}
= v \omega \\
\because \omega = \frac{v}{r}, v = r\omega \quad a = \frac{v^2}{r} = r \omega^2
\therefore F_{(구심력)}\; = m\frac{v^2}{r} = mr\omega^2
$

> ### $v = r\omega = r \frac{\theta}{t} \\
\dot{v} = r\frac{\dot{\theta t} - \theta}{t^2} \\
\dot{v} = r\frac{\omega}{t} - r\frac{\omega}{t}$

```python

```


# EEP/IAS 118 - Problem Set 4

## Due __Tuesday, July 27__ at 11:59pm.

Submit materials (a copy of this notebook and any additional typed work/written scans) as one combined pdf on __Gradescope__. All work can be completed in this notebook. Make sure to run (`shift` + `enter`) all your answer cells before submission to make sure all your output is displayed.

If you do not want to use this notebook, __your submitted PDF must include R code for all requested summary statistics/regression output AND the desired output itself.__ For example, if a question asks you to run a regression, I expect to see the code used to estimate the regression __and__ the output of the regression itself. You can download a compiled report from RStudio by going to __File > Compile Report__.

Answers that only provide an estimated coefficient values without the code/output from which it was obtained will lose points.

## R Tips

* You can type `?function()` for a given function and execute the cell to receive a popup of documentation on that function.
* If used after a regression with the `lm()` command that you name `reg_name`, running `reg_name$fitted.values` will output a vector of fitted values ($\hat y$) that is in the same order as the observations in your dataset (i.e. the first fitted value corresponds to the first observation in your data).
* The function `lm.beta()` in teh package __lm.beta__ will let us run standardized regressions. First, run the regression per usual using `lm()`, saving it to memory as `reg_name`. Then, run`lm.beta(reg_name)` to get the standardized version. If you execute it without saving to memory, you will obtain a regression table for the standardized variables, and if you save it to memory as `reg_name_beta` you can access the coefficients with `reg_name_beta$standardized.coefficients`.
* `cor(var1, var2)` prints the matrix of correlation between the two variables `var1` and `var2`.

# Exercise 1. Fuel Consumption

## Guidelines

This exercise uses the `Cars_PS4.dta` data, which contains observations for 200 cars on their miles per gallon $(mpg)$, their number of cylinders $(cyl)$, their engine displacement (some technical parameter of the car related to the size of the piston and the number of cylinder), horsepower $(hp)$, the vehicle weight in pounds $(wght)$, and the price in dollars $(price)$.

## Preamble

When writing **R** code (and especially once we move over to RStudio), it can be helpful to include a preamble in your script where you load in your data and all the packages you'll use throughout the entire problem set. Go ahead and use the below code cell to load in your dataset (assign it any name you'd like), and load the packages you'll be using (we'll at least need __tidyverse__ and __haven__).

```R
library(tidyverse)
library(haven)

cars <- read_dta("Cars_PS4.dta")
head(cars)
getwd()
```

## Question 1.

### Estimate the model:

$$ mpg= \beta_0 + \beta_1wght + \beta_2cyl + u $$

#### (a)  Print the regression summary table to the notebook and write out the estimated equation.

```R
# Insert your code for 1a here.
```

Estimated equation is above.

#### (b) Comment on your results (SSS for both slope parameters).

Add your written answers for 1b here.

#### (c) Compare the relative effect of weight and cylinder in determining fuel efficiency (hint: think standardizing).

```R
# Insert your code for 1c here.
```

Add any written answer for 1c here.

#### (d)  Without re-estimating the model above, what is the effect of vehicle weight on fuel efficiency when vehicle weight is measures in TONS (where 1 ton$=$2000 pounds)?  Answer this question by commenting on SSS (and when discussing significance please explicitly mention why significance does or does not change when vehicle weight is converted to tons).

Add your written answers for 1d here.

## Question 2.

### Use your results above to:

#### (a) Predict the average fuel efficiency (mpg) of the Mazda Tribute SUVs that weighs 2500lbs and has 6 cylinders.

```R
# Include your code for Question 2a here.

```

Add your written answers for Question 2a here.

#### (b) Run a regression that allows you to put a 90\% confidence interval around this predicted value.

```R
# Include your code for Question 2b here.

```

Add your written answers for Question 2b here.

## Question 3.

###  Let $mpg^0$ be the unknown fuel efficiency of one particular Mazda Tribute SUV that weighs 2500lbs and has 6 cylinders.

#### (a)  Find a 90\% confidence interval for $mpg^0$.

```R
# Include your code for Question 3a here.

```

Add your written answers for Question 3a here.

#### (b)  Compare the width of this confidence interval with what you found in the previous question. Explain in no more than 3 sentences.

```R
# Include your code for Question 3b here.

```

Add your written answers for Question 3b here.

## Question 4.

### Plot the graph of fuel efficiency $(mpg)$ against $(wght)$. What functional form might be more appropriate than a linear form for explaining fuel efficiency?

```R
# Include your code for Question 4 here

```

Add your written answers for Question 4 here.

## Question 5.

### Now estimate the following model

$$ ln(mpg)=\beta_0+\beta_1wght  +\beta_1cyl+ u  $$

### Print the regression summary table to the notebook and write out the estimated equation.

```R
# Type your code for Question 5 here

```

Add your written discussion for Question 5 here.

## Question 6.

### For explaining variation in $mpg$, decide whether you prefer the model from question 1 or the model from question 5.

Add your written discussion for Question 6 here.

## Question 7.

###  Let's think about adding horsepower into the linear model, and estimate the following model:

$$ mpg= \beta_0 + \beta_1wght + \beta_2cyl + \beta_3 hp+ u $$

### Comment on what happens to the significance of $cyl$. What accounts for the change in the standard errors of $cyl$? (No more than 3 sentences).

```R
# Type your code for Question 7 here

```

Add your written discussion for Question 7 here.

## Question 8.

###  Finally let's think about what determines car price by estimating the following two equations:

$$ price = \beta_0 + \beta_1mpg+ u$$

$$ price = \beta_0 + \beta_1mpg + \beta_2wght+u$$

#### (a) Write out the two estimated equations

```R
# Type your code for part (a) here

```

Add your written discussion for part (a) here.

#### (b)  Comment on how the coefficient on $mpg$ changes when we add in the variable $wgt$.  What does this tell us about the correlation between $mpg$ and $wght$? (No more than 3 sentences).

Add your written discussion for part (b) here.

# Exercise 2:  Do Legislators with More Daughters Vote More Liberally on Women's Issues?

## Background

The data for this exercise were used in Ebonya Washington's paper: "Female Socialization: How Daughters Affect Their Legislator Fathers' Voting on Women's Issues."  published in the American Economic Review in 2008. The paper asks whether having daughters influences the voting behavior of members of the US Congress. The hypothesis is that having (more) daughters makes legislators more likely to vote liberally on issues concerning women.

For this exercise, we will focus on votes that took place in the 108th Congress, which held session in 2003/04. As a measure of a liberal voting record, we use scores assigned by the American Association of University Women (AAUW), a liberal group that concerns itself with issues of interest to women. For the 108th Congress, the AAUW selected 9 pieces of legislation in the areas of education, equality and reproductive rights. The AAUW then assigned a score to each member of Congress. The scores range from 0 to 100 and measure the percentage of times the legislator voted in favor of the position held by the AAUW.

 The dataset `Legislators_PS4.dta` contains the following characteristics for 320 members of the 108th Congress:

 * $ngirls$ number of daughters
 * $totchi$ number of children
 * $age$ Age
 * $female$ indicator for being female
 * $repub$ indicator for being a Republican
 *  $moredef$ proportion of people in the legislator's district who are in favor of "more spending on
defense"
 *  $aauw$ AAUW score

### Question 1. Estimate and report results for the following regression models:

\begin{align}
aauw&=\beta_0 +\beta_1ngirls+u  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1) \\
aauw&=\beta_0 +\beta_1ngirls+\beta_2totchi+u ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~(2)  \\
aauw&=\beta_0 +\beta_1ngirls+\beta_2totchi+\beta_3female+\beta_4repub+u\ \ \ \ ~~\ \ \ (3)
\end{align}

```R
# Add Code for Question 1 here.
```

Add your written answer for Question 1 here.

### Question 2. Look at the results of the above regressions.

#### (a) Briefly interpret $\beta_1$ in the first model (remember SSS).

Add your written answer for part (a) here.

#### (b) Discuss how the coefficient on $\beta_1$ changes across the three models.  What do you think explains the difference in $\beta_1$ between the first and second model?

Add your written answer for Question (b) here.

### Question 3. Estimate the following quadratic regression of voting scores on age:

$$aauw = \beta_0 + \beta_1 ngirls+ \beta_2 totchi  + \beta_3 age + \beta_4 age^2 + u \ \ \ \ \ \ \ (4) $$

#### (a) Comment on the sign of $\beta_3$ and $\beta_4$.  What does this tell us about the relationship between age and $aauw$?

```R
# Add your code for part (a) here.
```

Add your written answer for part (a) here.

#### (b) What is the marginal effect of age on $aauw$ (provide the mathematical formula).

```R
# Add your code for part (b) here.
```

Add your written answer for part (b) here.

#### (c)  Evaluate the marginal effect of age on $aauw$ for someone with the mean age (53.79614)

```R
# Add your code for part (c) here.
```

Add your written answer for part (c) here.

#### (d)  At what age does the marginal effect of $age$ on $aauw$  become positive?

```R
# Add your code for part (d) here.
```

Add your written answer for part (d) here.

## Question 4.

### Compare equations (3) to equation (4).  In no more than 3 sentences please explain why you would choose one model over the other?

Add your written answer for Question 4 here.

## Downloading your Notebook

Download a PDF copy of your notebook by using __File > Download as > PDF via Chrome (.pdf)__. It does not matter what browser you are using, you can use this command to download a copy of your notebook while on Chrome/Firefox/Opera/Edge/Safari/heck probably even Internet Explorer or Netscape Navigator.

If you have a problem downloading your notebook or combining PDFs, or if you are using RStudio on your own computer and are unsure what sort of output I want, __please email me__.

Problem Sets/Problem Set 4/Problem Set 4.ipynb


```python
from sympy import *
from sympy.abc import r,x,y,z
from sympy import init_printing
from cubature import cubature
import sympy.printing.llvmjitcode as jit
from timeit import default_timer as timer
import matplotlib.pyplot as plt
%matplotlib inline
init_printing()
```

# Energy of a Helium atom

The Hamiltonian for this system is
$$-\frac{1}{2} (\nabla_1^2 + \nabla_2^2) - \frac{2}{r_1} - \frac{2}{r_2} + \frac{1}{r_{12}}$$

The first two terms are the kinetic energy, the next two are the electron-nuclear Coulomb interaction, and the final term is the electron-electron Coulomb interaction.

```python
r1 = Symbol('r1')    # magnitude of vector r1 (electron 1 to nucleus distance)
r2 = Symbol('r2')    # magnitude of vector r2 (electron 2 to nucleus distance)
r12 = Symbol('r12') # |r2-r1| (magnitude of vector r2-r1)

```

### Simple trial wavefunction

This trial wavefunction uses a Slater form for the electon-nuclear interactions.  The value in the exponent (-2) is fixed by the cusp condition.   The electron-electron interaction uses a simple Pade form.   There is one adjustable parameter, $\beta$.   The integration occurs over the spatial coordinates of each electron, giving a 6 dimensional integral.

In the QMCPACK distribution, the example input file in `examples/molecules/He/simple_he.xml` has the same wavefunction.

```python
beta = Symbol('beta')
R1 = exp(-2*r1)
R2 = exp(-2*r2)
G = exp(r12/2/(1 + beta*r12))
R1*R2*G
```

```python
def del_spherical(e, r):
    """Compute Laplacian for expression e with respect to symbol r.
        Currently works only with radial dependence"""
    t1 = r*r*diff(e, r)
    t2 = diff(t1, r)/(r*r)
    return simplify(t2)

def grad_spherical(e, r, rhat):
    """Compute gradient for expresion e with respect to symbol r.
       The symbol rhat is the unit vector in the radial direction."""
    t1 = diff(e, r)
    t2 = simplify(t1)
    return t2*rhat

```

```python
# Wavefunction has the form F1 * F2 * G
# grad (FG) = G grad(F) + F grad(G)

def grad_combo(F, gradF, G, gradG):
    t = F*gradG + G*gradF
    return simplify(t)

# del^2 (FG) = G del^2(F0 + F del^2(G) + 2 grad(F) * grad(G)
def del_combo(F, gradF, delsqF, G, gradG, delsqG):
    t = G * delsqF + F * delsqG + 2*gradF*gradG
    t2 = t.subs(rhat1*rhat1, 1)
    t3 = t2.subs(rhat2*rhat2, 1)
    t4 = t3.subs(rhat12*rhat12, 1)
    t5 = t4.subs(rhat1*rhat12, rhat1_12)
    t6 = t5.subs(rhat2*rhat12, rhat2_12)
    return simplify(t6)

```

```python
# Unit vectors
rhat1 = Symbol('rhat1')
rhat2 = Symbol('rhat2')
rhat12 = Symbol('rhat12')
rhat1_12 = Symbol('rhat1_12')
rhat2_12 = Symbol('rhat2_12')
```

```python
# TODO: Explain this derivation
grad_1 = grad_spherical(R1, r1, rhat1)
del2_1 = del_spherical(R1, r1)

grad_2 = grad_spherical(R2, r2, rhat2)
del2_2 = del_spherical(R2, r2)

gradG_1 = grad_spherical(G, r12, rhat12)
del2G_1 = del_spherical(G, r12)

grad_R1G = grad_combo(R1, grad_1, G, -gradG_1)

delsq_R1G = del_combo(R1, grad_1, del2_1, G, -gradG_1, del2G_1)

grad_R2G = grad_combo(R2, grad_2, G, gradG_1)
delsq_R2G = del_combo(R2, grad_2, del2_2, G, gradG_1, del2G_1)

# This is the integrand for the denominator
psi2 = R1*R1*R2*R2*G*G

h1 = -1/S(2) * (R2*R2) * R1 * G * delsq_R1G - 2*psi2/r1
h2 = -1/S(2) * (R1*R1) * R2 * G * delsq_R2G - 2*psi2/r2
h = h1 + h2 + psi2/r12
h = simplify(h)
# This is the integrand for the numerator
h
```

```python
# Symbols for performing the integration
x1,y1,z1 = symbols('x1 y1 z1')
x2,y2,z2 = symbols('x2 y2 z2')
# Add a very tiny softening to keep the evaluation of the integrand from blowing up at certain points.
r1_c = sqrt(x1*x1 + y1*y1 + z1*z1 + 1e-20)
r2_c = sqrt(x2*x2 + y2*y2 + z2*z2 + 1e-20)
r12_c = sqrt((x2-x1)*(x2-x1) + (y2-y1)*(y2-y1) + (z2-z1)*(z2-z1) + 1e-20)
rhat1_12_c = (x1*(x2-x1) + y1*(y2-y1) + z1*(z2-z1))/r1_c/r12_c
rhat2_12_c = (x2*(x2-x1) + y2*(y2-y1) + z2*(z2-z1))/r2_c/r12_c
```

```python
# Perform 6D integral. Using scipy.integrate.nquad will take too long.
# Use Cubature and Sympy LLVM JIT
#  TODO: insert link to explain different integrators

beta_val = 1.0
num = tmp.subs(beta, beta_val)
denom = simplify(tmp2.subs(beta, beta_val))

ndim = 6
lim_min = [-30.0]*ndim
lim_max = [30.0]*ndim

repls = [(r1, r1_c), (r2, r2_c), (r12, r12_c), (rhat1_12, rhat1_12_c), (rhat2_12, rhat2_12_c)]
repls_n = [a for a in repls]
tmp_h = h.subs(beta, beta_val)
tmp_d = psi2.subs(beta, beta_val)

# Use CSE (Common Subexpression Elimination) to speed up the integrand evaluation
cse_h = cse([tmp_h, tmp_d])
repls.extend(cse_h[0])
expr_h = [repls, cse_h[1]]

jit_expr = jit.llvm_callable( [x1, y1, z1, x2, y2, z2], expr_h, callback_type='cubature')

epsrel = 1e-4
start = timer()
res = cubature(jit_expr, ndim=ndim, fdim=2, xmin=lim_min, xmax=lim_max, relerr=epsrel)
end = timer()
E = res[0][0]/res[0][1]
print("For beta = ",beta_val," Energy = ",E)
print("Time: %.2f seconds"%(end-start))
```

    For beta =  1.0  Energy =  -2.82611479297
    Time: 96.18 seconds

```python
# Collect computation of energy into a single function
def compute_energy(beta_val):

    repls = [(r1, r1_c), (r2, r2_c), (r12, r12_c), (rhat1_12, rhat1_12_c), (rhat2_12, rhat2_12_c)]
    repls_n = [a for a in repls]
    tmp_h = h.subs(beta, beta_val)
    tmp_d = psi2.subs(beta, beta_val)

    cse_h = cse([tmp_h,tmp_d])
    repls.extend(cse_h[0])
    expr_h = [repls, cse_h[1]]

    jit_expr = jit.llvm_callable( [x1, y1, z1, x2, y2, z2], expr_h, callback_type='cubature')

    epsrel = 1e-4
    res = cubature(jit_expr, ndim=ndim, fdim=2, xmin=lim_min, xmax=lim_max, relerr=epsrel)
    E = res[0][0]/res[0][1]

    return E
```

```python
# Compute energies over a range of variational parameters to find the minimum.
energies = []
betas = []
for i in range(10):
    beta_val = i*.02 + 0.01
    e = compute_energy(beta_val)
    print(beta_val, e)
    betas.append(beta_val)
    energies.append(e)
```

    0.01 -2.86001117585
    0.03 -2.86676650391
    0.05 -2.87135272931
    0.06999999999999999 -2.87442400954
    0.09 -2.87639789443
    0.11 -2.87755791558
    0.13 -2.8781022511
    0.15000000000000002 -2.87817447072
    0.17 -2.87788106408
    0.19 -2.87730296758

```python
fig = plt.plot(betas, energies)
fig[0].axes.set_xlabel(r"$\beta$")
fig[0].axes.set_ylabel("Energy")
```

```python

```


# Monetary Policy Model

**Randall Romero Aguilar, PhD**

This demo is based on the original Matlab demo accompanying the  Computational Economics and Finance 2001 textbook by Mario Miranda and Paul Fackler.

Original (Matlab) CompEcon file: **demdp11.m**

Running this file requires the Python version of CompEcon. This can be installed with pip by running

    !pip install compecon --upgrade

Last updated: 2021-Oct-01

## About

A central bank must set nominal interest rate so as to minimize deviations of inflation rate and GDP gap from established targets.

A monetary authority wishes to control the nominal interest rate $x$ in order to minimize the variation of the inflation rate $s_1$ and the gross domestic product (GDP) gap $s_2$ around specified targets $s^∗_1$ and $s^∗_2$, respectively. Specifically, the authority wishes to minimize expected discounted stream of weighted squared deviations

\begin{equation}
L(s) = \frac{1}{2}(s − s^∗)'\Omega(s − s^∗)
\end{equation}

where $s$ is a $2\times 1$ vector containing the inflation rate and the GDP gap, $s^∗$ is a $2\times 1$ vector of  targets, and $\Omega$ is a $2 \times 2$ constant positive definite matrix of preference weights. The inflation rate and the GDP gap are a joint controlled exogenous linear Markov process

\begin{equation}
s_{t+1} = \alpha + \beta s_t + \gamma x_t + \epsilon_{t+1}
\end{equation}

where $\alpha$ and $\gamma$ are $2 \times 1$ constant vectors, $\beta$ is a $2 \times 2$ constant matrix, and $\epsilon$ is a $2 \times 1$ random vector with mean zero. For institutional reasons, the nominal interest rate $x$ cannot be negative. **What monetary policy minimizes the sum of current and expected future losses?**

This is an infinite horizon, stochastic model with time $t$ measured in years. The state vector $s \in \mathbb{R}^2$
contains the inflation rate and the GDP gap. The action variable $x \in [0,\infty)$ is the nominal interest rate. The state transition function is $g(s, x, \epsilon) = \alpha + \beta s + \gamma x + \epsilon$

In order to formulate this problem as a maximization problem, one posits a reward function that equals the negative of the loss function $f(s,x) = −L(s)$

The sum of current and expected future rewards satisfies the Bellman equation

\begin{equation}
V(s) = \max_{0\leq x}\left\{-L(s) + \delta + E_\epsilon V\left(g(s,x,\epsilon)\right)\right\}
\end{equation}

Given the structure of the model, one cannot preclude the possibility that the nonnegativity constraint on the optimal nominal interest rate will be binding in certain states. Accordingly, the shadow-price function $\lambda(s)$ is characterized by the Euler conditions

\begin{align}
    \delta\gamma'E_\epsilon \lambda\left(g(s,x,\epsilon)\right) &= \mu \\
    \lambda(s) &= -\Omega(s-s^*) + \delta\beta'E_\epsilon \lambda\left(g(s,x,\epsilon)\right)
\end{align}

where the nominal interest rate $x$ and the long-run marginal reward $\mu$ from increasing the nominal interest rate must satisfy the complementarity condition
\begin{equation}
x \geq 0, \qquad \mu \leq 0, \qquad x > 0 \Rightarrow \mu = 0
\end{equation}

It follows that along the optimal path

\begin{align}
    \delta\gamma'E_\epsilon \lambda_{t+1} &= \mu_t \\
    \lambda_t &= -\Omega(s_t-s^*) + \delta\beta'E_\epsilon \lambda_{t+1}\\
    x \geq 0, \qquad \mu \leq 0, &\qquad x > 0 \Rightarrow \mu = 0
\end{align}

Thus, in any period, the nominal interest rate is reduced until either the long-run marginal reward or the nominal interest rate is driven to zero.

```python
import numpy as np
import matplotlib.pyplot as plt
from mpl_toolkits.mplot3d import Axes3D
from matplotlib import cm
from compecon import BasisChebyshev, DPmodel, BasisSpline, qnwnorm, demo
import pandas as pd
pd.set_option('display.float_format',lambda x: f'{x:.3f}')
```

### Model Parameters

```python
α   = np.array([[0.9, -0.1]]).T             # transition function constant coefficients
β    = np.array([[-0.5, 0.2], [0.3, -0.4]])  # transition function state coefficients
γ   = np.array([[-0.1, 0.0]]).T             # transition function action coefficients
Ω   = np.identity(2)                        # central banker's preference weights
ξ   = np.array([[1, 0]]).T                  # equilibrium targets
μ   = np.zeros(2)                           # shock mean
σ   = 0.08 * np.identity(2),                # shock covariance matrix
δ   = 0.9                                   # discount factor
```

### State Space

There are two state variables: 'GDP gap' = $s_0\in[-2,2]$ and 'inflation'=$s_1\in[-3,3]$.

```python
n = 21
smin = [-2, -3]
smax = [ 2,  3]

basis = BasisChebyshev(n, smin, smax, method='complete',
                       labels=['GDP gap', 'inflation'])
```

### Action space

There is only one action variable x: the nominal interest rate, which must be nonnegative.

```python
def bounds(s, i, j):
    lb  = np.zeros_like(s[0])
    ub  = np.full(lb.shape, np.inf)
    return lb, ub
```

### Reward Function

```python
def reward(s, x, i, j):
    s = s - ξ
    f = np.zeros_like(s[0])
    for ii in range(2):
        for jj in range(2):
            f -= 0.5 * Ω[ii, jj] * s[ii] * s[jj]
    fx = np.zeros_like(x)
    fxx = np.zeros_like(x)
    return f, fx, fxx
```

### State Transition Function

```python
def transition(s, x, i, j, in_, e):
    g = α + β @ s + γ @ x + e
    gx = np.tile(γ, (1, x.size))
    gxx = np.zeros_like(s)
    return g, gx, gxx
```

The continuous shock must be discretized. Here we use Gauss-Legendre quadrature to obtain nodes and weights defining a discrete distribution that matches the first 6 moments of the Normal distribution (this is achieved with m=3 nodes and weights) for each of the state variables.

```python
m   = [3, 3]
[e,w] = qnwnorm(m,μ,σ)
```

### Model structure

```python
bank = DPmodel(basis, reward, transition, bounds,
               x=['interest'], discount=δ, e=e, w=w)
```

Compute Unconstrained Deterministic Steady-State

```python
bank_lq = bank.lqapprox(ξ,0)

sstar = bank_lq.steady['s']
xstar = bank_lq.steady['x']
```

If Nonnegativity Constraint Violated, Re-Compute Deterministic Steady-State

```python
if xstar < 0:
    I = np.identity(2)
    xstar = 0.0
    sstar = np.linalg.solve(np.identity(2) - β, α)

frmt = '\t%-21s = %5.2f'
print('Deterministic Steady-State')
print(frmt % ('GDP Gap', sstar[0]))
print(frmt % ('Inflation Rate', sstar[1]))
print(frmt % ('Nominal Interest Rate', xstar))
```

    Deterministic Steady-State
    	GDP Gap               =  0.61
    	Inflation Rate        =  0.06
    	Nominal Interest Rate =  0.00

### Solve the model
We solve the model by calling the `solve` method in `bank`. On return, `sol` is a pandas dataframe with columns *GDP gap*, *inflation*, *value*, *interest*, and *resid*. We set a refined grid `nr=5` for this output.

```python
S = bank.solve(nr=5)
```

    Solving infinite-horizon model collocation equation by Newton's method
    iter change       time
    ------------------------------
       0       6.7e+00    0.4658
       1       9.9e-01    1.2961
       2       9.1e-04    2.1418
       3       8.3e-07    2.6914
       4       7.7e-09    3.2808
    Elapsed Time =    3.28 Seconds

To make the 3D plots, we need to reshape the columns of `sol`.

```python
S3d = {x: S[x].values.reshape((5*n,5*n)) for x in S.columns}
```

This function will make all plots

```python
def makeplot(series,zlabel,zticks,title):
    fig = plt.figure(figsize=[8,5])
    ax = fig.gca(projection='3d')
    ax.plot_surface(S3d['GDP gap'], S3d['inflation'], S3d[series], cmap=cm.coolwarm)
    ax.set_xlabel('GDP gap')
    ax.set_ylabel('Inflation')
    ax.set_zlabel(zlabel)
    ax.set_xticks(np.arange(-2,3))
    ax.set_yticks(np.arange(-3,4))
    ax.set_zticks(zticks)
    ax.set_title(title)
```

###  Optimal policy

```python
fig1 = makeplot('interest', 'Nomianal Interest Rate',
               np.arange(0,21,5),'Optimal Monetary Policy')
```

### Value function

```python
fig2 = makeplot('value','Value',
                np.arange(-12,S['value'].max(),4),'Value Function')
```

### Residuals

```python
fig3 = makeplot('resid','Residual',
                [-1.5e-3, 0, 1.5e3],'Bellman Equation Residual')
plt.ticklabel_format(style='sci', axis='z', scilimits=(-1,1))
```

## Simulating the model

We simulate 21 periods of the model starting from $s=s_{\min}$, 10000 repetitions.

```python
T = 21
nrep = 10_000
data = bank.simulate(T, np.tile(np.atleast_2d(smax).T,nrep))
```

```python
subdata = data[data['time']==T][['GDP gap', 'inflation', 'interest']]
stats =pd.DataFrame({'Deterministic Steady-State': [*sstar.flatten(), xstar],
              'Ergodic Means': subdata.mean(),
              'Ergodic Standard Deviations': subdata.std()})
stats.T
```

      GDP gap
      inflation
      interest

      Deterministic Steady-State
      0.608
      0.059
      0.000

      Ergodic Means
      0.588
      0.056
      0.306

      Ergodic Standard Deviations
      0.316
      0.335
      0.716

### Simulated State and Policy Paths

```python
subdata = data[data['_rep'].isin(range(3))]
opts = dict(spec='r*', offset=(0, -15), fs=11, ha='right')
gdpstar, infstar, intstar = stats['Ergodic Means']

def simplot(series,ylabel,yticks,steady):
    fig = demo.figure('Simulated and Expected ' + ylabel,'Period', ylabel,[0, T + 0.5])
    plt.plot(data[['time',series]].groupby('time').mean())
    plt.plot(subdata.pivot('time','_rep',series),lw=1)
    plt.xticks(np.arange(0,24,4))
    plt.yticks(yticks)
    demo.annotate(T, steady, f'Expected {series}\n = {steady:.2f}', **opts)
    return fig
```

```python
fig4 = simplot('GDP gap','GDP gap',np.arange(smin[0],smax[0]+1),gdpstar)
```

```python
fig5 = simplot('inflation', 'Inflation Rate',np.arange(smin[1],smax[1]+1),infstar)
```

```python
fig6 = simplot('interest','Nominal Interest Rate',np.arange(-2,5),intstar)
```

```python
#demo.savefig([fig1,fig2,fig3,fig4,fig5,fig6])
```

_build/jupyter_execute/notebooks/dp/11 Monetary Policy Model.ipynb


# <center/>  Optimal Transport in Computational Neuroscience
<center/> Lang Liu and Alec Greaves-Tunnell and Zaid Harchaoui

<center/> Department of Statistics

<center/> University of Washington

<center/> {liu16, alecgt, zaid}@uw.edu

<center/> July 30, 2020

This notebook comprises the lab portion of the Neurohackademy 2020 lecture on Optimal Transport (OT). It is intended to be a hands-on tour of the concepts introduced in the lecture, with an emphasis on bridging the gap between theoretical exposition and the application of OT tools in the modern practice of computational neuroscience.

The lab is divided into three sections:

- __Part I__ reviews optimal transport and entropic regularization, and implements some toy examples.

- __Part II__ takes a more detailed look at the computation of Wasserstein barycenters.

- __Part III__ explores the application of these concepts to source imaging for MEG data.

A final section contains reference materials and resources for further study.

```python
import os
import sys

from numba import jit

import numpy as np
import matplotlib.pyplot as plt
import imageio
import pickle

from ot.unbalanced import barycenter_unbalanced
from mtw import MTW

import mne
import nilearn
from nilearn import plotting

sys.path.insert(0,os.path.abspath('..'))
from src.utils import get_surf_dist

import warnings
warnings.filterwarnings('ignore')
```

## Part I: Optimal transport and entropic regularization

### 1.1 Review

Let us recall the optimal transport (OT) problem.
In the discrete case, the probability measures can be represented as vectors $a \in \Sigma_n$, $b \in \Sigma_m$, where

$$ \Sigma_n = \left\{ x \in \mathbb{R}_+^n : \sum_{i=1}^n x_i = 1\right\}$$

denotes the $n$-dimensional simplex.
Note that $a$ and $b$ are not required to have the same dimension.
The cost matrix $C \in \mathbb{R}_+^{n \times m}$ contains the pairwise costs $C_{ij}$ of transporting a unit of mass from bin $i$ in $a$ to bin $j$ in $b$.
Then the OT problem reads

\begin{equation}
L_C(a,b) = \min_{P \in U(a,b)} \langle C, P \rangle,
\end{equation}

where the feasible set $U(a,b)$ consists of all matrices in $\mathbb{R}_+^{n \times m}$ whose rows sum to $a$ and columns sum to $b$ (that is, all valid transportation plans between discrete measures).

As the name suggests, entropy-regularized optimal transport proposes to augment the original OT optimization problem with a regularization term given by the entropy of the transport plan.

The entropy $H(P)$ of a transport plan $P$ is defined as
$$
H(P) = - \sum_{i,j} P_{ij} \log P_{ij},
$$

so that the entropy-regularized OT problem is

$$
L_C^\varepsilon(a,b) = \min_{P \in U(a,b)} \langle C, P \rangle - \varepsilon H(P).
$$

The parameter $\varepsilon > 0$ controls the tradeoff between the transportation cost $\langle C, P \rangle$ and entropy $H(P)$ terms.

There are several consequences of entropic regularization that make it highly advantageous in practice:

- The entropy-regularized problem is strongly convex in $P$; it therefore admits a unique solution.

- The entropy term encourages non-degeneracy in the solution, which may result in more appropriate or realistic transportation plans in certain applications.

- The strong convexity of the objective accelerates convergence in both the computational and statistical senses.

- The algorithm to solve the entropy-regularized problem is amenable to GPU hardware acceleration.

__Remark.__ There are other choices of regularizations. You may find more details in the extended version of this notebook.

### 1.2 Algorithm

The entropy-regularized OT problem is solved by means of an iterative procedure known as Sinkhorn's algorithm. As seen in the lecture, the optimality condition for the entropy-regularized OT problem implies that the solution will have the form

$$ P_\varepsilon = \text{diag}(u) K \text{diag}(v),$$

where $u \in \mathbb{R}_+^n$ and $v \in \mathbb{R}_+^m$ are unknown positive scaling vectors and $K$ is again the Gibbs kernel $K = \exp(- C / \varepsilon)$.

The scaling vectors $u$ and $v$ must be such that $P_\varepsilon$ satisfies the appropriate row and column sum constraints for a transport plan between $a$ and $b$. Specifically, the rows of $P_\varepsilon$ must sum to $a$, and the columns must sum to $b$:
\begin{align*}
\sum_{j = 1}^m (P_\varepsilon)_{ij} &= a_i, \ \ \text{ for every } i=1, ..., n \\
\sum_{i = 1}^n (P_\varepsilon)_{ij} &= b_j, \ \ \text{ for every } j=1, ..., m. \\
\end{align*}

We can write these constraints in matrix notation and relate them to the identity $P_\varepsilon = \text{diag}(u) K \text{diag}(v)$ to obtain

\begin{align*}
P_\varepsilon 1_m &= u \odot (Kv) = a \\
P_\varepsilon^T 1_n &= v \odot (K^T u) = b, \end{align*}

where $x \odot y$ denotes the element-wise product between vectors $x$ and $y$.

This constraint means that $u = a / (Kv)$ and $v = b / (K^T u)$, where the division is understood to be taken component-wise.

Sinkhorn's algorithm therefore simply alternates between the updates

\begin{align*}
u \leftarrow \frac{a}{Kv} \\
v \leftarrow \frac{b}{K^T u} \end{align*}
after initializing $v = 1_m$.
When $m = n$, Sinkhorn's algorithm converges in $O(n^2 \log{n})$ complexity assuming appropriate conditions.

```python
'''
In this block we implement a single iteration of the Sinkhorn update.

The boolean parameter log_errs determines whether we compute row / column sum constraint violations after each update.
'''
def Sinkhorn_update(v, a, b, K, log_errs):
    row_err = col_err = None

    u = a / np.dot(K, v)
    if log_errs:
        col_err = np.linalg.norm((v * np.dot(K.T, u) - b), 1)

    v = b / np.dot(K.T, u)
    if log_errs:
        row_err = np.linalg.norm((u * np.dot(K, v) - a), 1)

    return u, v, row_err, col_err
```

The full algorithm requires an initial condition for $v$ and a criterion for termination of the algorithm. For simplicity, we will simply fix the number of iterations (what other conditions could we use?).

Of course, we must also specify the two target measures $a$ and $b$, along with the Gibbs kernel $K$.

```python
'''
Basic implementation of Sinkhorn algorithm for entropy-regularized OT.
'''
def Sinkhorn(a, b, K, max_iter = 2000, log_errs = True):
    v = np.ones(len(b)) # initialize v
    errs = []

    for i in range(max_iter):
        u, v, row_err, col_err = Sinkhorn_update(v, a, b, K, log_errs)
        errs += [(row_err, col_err)]
    return u, v, errs
```

### 1.3 Examples

We'll illustrate the application of Sinkhorn's algorithm on two simple examples: histograms and point clouds.

#### 1.3.1 Histograms

A histogram is a set of non-negative weights that sum to one, defined on a regular grid. Often, histograms are used to summarize a dataset in terms of the frequency with which the response is observed in each bin.

We begin by defining and viewing the target histograms $a$ and $b$. In this example, we'll generate the histograms from two 1-D Gaussian random variables with different means:

```python
'''
Generate and plot two histograms.
'''
N = 200 # size of histogram
t = np.arange(0, N) / N # histogram bins

Gaussian = lambda t0, sigma: np.exp(-(t-t0)**2/(2*sigma**2))
normalize = lambda p: p/np.sum(p)

sigma = .06
vmin = .02
a = Gaussian(.25,sigma) # generate histograms
b = Gaussian(.8,sigma)

a = normalize( a+np.max(a)*vmin)
b = normalize( b+np.max(b)*vmin)

plt.figure(figsize = (10,7)) # plot

plt.subplot(2, 1, 1)
plt.bar(t, a, width = 1/len(t), color = "darkblue")
plt.title('a')
plt.subplot(2, 1, 2)
plt.bar(t, b, width = 1/len(t), color = "darkblue")
plt.title('b')

plt.show()
```

Next we'll compute the cost matrix $C$ and corresponding Gibbs kernel $K = \exp(- C/\varepsilon)$.

We'll use the squared distance as the cost of transport between bins $i$ and $j$

$$ C_{ij} = \left(\frac{i}{N} - \frac{j}{N} \right)^2,$$

so that the Gibbs kernel is a Gaussian convolution:

$$ K_{ij} = \exp\left(-\frac{\left( i/N - j/N \right)^2}{ \varepsilon}\right). $$

```python
'''
Set regularization parameter epsilon. Compute cost matrix and Gibbs kernel.
'''
epsilon = 0.0009

[Y,X] = np.meshgrid(t,t)

C = (X-Y)**2
K = np.exp(-C/epsilon)
```

We run the algorithm for a fixed number of iterations.

```python
u, v, errs = Sinkhorn(a, b, K, max_iter = 2000)
```

One way to track the progress of the algorithm as a function of iteration number is to quantify how closely the candidate solution at iteration $t$ satisfies the row and column sum constraints of the entropy-regularized OT problem:

```python
'''
Plot the constraint violations in log scale.
'''
plt.figure(figsize=(9,5))

plt.subplot(2,1,1)
plt.plot([np.log(x[0]) for x in errs])
plt.title('Row sum constraint violation (log scale)')
plt.ylabel("$\log \ \ ||P 1 - a||_1$")
plt.xlabel('Sinkhorn iteration')

plt.subplot(2,1,2)
plt.plot([np.log(x[1]) for x in errs])
plt.title('Column sum constraint violation (log scale)')
plt.ylabel("$\log \ \ ||P^T 1 - b||_1$")
plt.xlabel('Sinkhorn iteration')

plt.tight_layout()
plt.show()
```

Finally, we can view the optimal transport plan $P_\varepsilon$ as a heatmap in which each tile shows the optimal allocation of mass from bin $i$ of histogram $a$ to bin $j$ of histogram $b$.

```python
'''
Plot entropy-regularized OT solution.
'''
P = np.dot(np.dot(np.diag(u),K),np.diag(v)) # optimal transport plan
plt.figure(figsize=(5,5))
plt.imshow(np.log(P+1e-5))
plt.axis('off')
```

__Exercise__: How does our choice of the regularization parameter $\varepsilon$ affect the optimal transport plan $P_\varepsilon$? Experiment with different values for $\varepsilon$ and plot the resulting $P_\varepsilon$. Mathematically, what should we expect as $\varepsilon$ grows either very small or very large?

#### 1.3.2 Point clouds

We'll also consider an example of entropy-regularized optimal transport on _point clouds_.

```python
'''
Generate and plot two point clouds.
'''
# experiment parameters
N = [30, 50]
d = 2
epsilon = .01

# point cloud a: uniform draws from unit square
x = np.random.rand(2,N[0])-.5

# point cloud b: uniform draws from annulus surrounding circle
theta = 2*np.pi*np.random.rand(1,N[1])
r = .8 + .2*np.random.rand(1,N[1])
y = np.vstack((np.cos(theta)*r,np.sin(theta)*r))

# plot
def plotp(x, col):
    plt.scatter(x[0,:], x[1,:], s=200, edgecolors='k', c=col, linewidths=2)

plt.figure(figsize=(6,6))

plotp(x, 'blue')
plotp(y, 'red')

plt.axis("off")
plt.xlim(np.min(y[0,:])-.1,np.max(y[0,:])+.1)
plt.ylim(np.min(y[1,:])-.1,np.max(y[1,:])+.1)

plt.show()
```

Again, we'll use the squared 2-norm (this time on a 2-D space) for the cost function:

$$ C(x,y) = \vert \vert x - y \vert \vert^2.$$

We'll set the target densities to be uniform weightings on the observed points. That is, to each observation $x_i$ we assign the weight $a_i = \frac{1}{N_x}$ and to each observation $y_j$ we assign the weight $b_j = \frac{1}{N_y}$, where $N_x$ and $N_y$ are the number of points in the point clouds $x$ and $y$, respectively.

```python
'''
Define cost matrix, Gibbs kernel, and target histograms.
'''
# cost function
x2 = np.sum(x**2,0)
y2 = np.sum(y**2,0)
C = np.tile(y2,(N[0],1)) + np.tile(x2[:,np.newaxis],(1,N[1])) - 2*np.dot(np.transpose(x),y)

# Gibbs kernel
K = np.exp(-C / epsilon)

# target densities
a = np.ones(N[0])/N[0]
b = np.ones(N[1])/N[1]
```

We're now prepared to run Sinkhorn's algorithm.

```python
'''
Compute optimal transport plan.
'''
u, v, errs = Sinkhorn(a, b, K, max_iter=2000)
```

As above, one way to track the progress of the algorithm is in terms of the marginal constraint violations of the candidate optimal transport plan $P_\varepsilon$.

```python
'''
Plot the constraint violations in log scale.
'''
plt.figure(figsize=(9,5))

plt.subplot(2,1,1)
plt.plot([np.log(x[0]) for x in errs])
plt.title('Row sum constraint violation (log scale)')
plt.ylabel("$\log \ \ ||P 1 - a||_1$")
plt.xlabel('Sinkhorn iteration')

plt.subplot(2,1,2)
plt.plot([np.log(x[1]) for x in errs])
plt.title('Column sum constraint violation (log scale)')
plt.ylabel("$\log \ \ ||P^T 1 - b||_1$")
plt.xlabel('Sinkhorn iteration')

plt.tight_layout()
plt.show()
```

Finally, we'll consider some ways to visualize the results. Below we plot the largest individual components of the optimal transport plan as lines between elements of the two point clouds, with more solid lines indicating greater mass transport between the corresponding elements.

```python
'''
Visualization: mass movements between point clouds.
'''
plt.figure(figsize=(10,10))

plotp(x, 'b')
plotp(y, 'r')

P = np.dot(np.dot(np.diag(u),K),np.diag(v)) # compute transport plan

weights = np.arange(0.1,1,0.3)
for w in weights:
    A = P * (P > np.max(P)*w)
    i,j = np.where(A != 0)
    plt.plot([x[0,i],y[0,j]], [x[1,i],y[1,j]], 'k', lw = 1, alpha=w)

plt.axis("off")
plt.xlim(np.min(y[0,:])-.1,np.max(y[0,:])+.1)
plt.ylim(np.min(y[1,:])-.1,np.max(y[1,:])+.1)

plt.show()
```

```python
'''
Visualization: optimal transport plan as heat map.
'''
P = np.dot(np.dot(np.diag(u),K),np.diag(v)) # compute optimal transport plan

plt.figure(figsize=(4,6))
plt.imshow(P)
plt.axis('off')
```

__Exercise__: There are quite a few parameters involved in the experiment presented in this section. If you have some extra time, try to build some intuition around this example by playing with these. For example:

- What happens when the point clouds have a highly unequal number of observations?

- What happens when the histogram weights aren't uniform?

- How does the strength of entropy regularization affect the optimal transport plan?

## Part II: Wasserstein barycenters

### 2.1 Intro

A key perspective in OT is that it is not only a method to compute minimial-cost transport plans between probabilities, but also it provides a mathematically well-defined notion of __distance__ between probabilities whenever we can specify a unit cost of transport between two points in the space on which they are defined.

This perspective motivates the concept of a *Wasserstein barycenter*, which solves

$$ \min_{b \in \Sigma_m} \frac{1}{S} \sum_{s=1}^S L_C(a_s, b),$$

where $\{a_s\}_{s=1}^S$ are a collection of histograms. A Wasserstein barycenter can thus be viewed as an average between observed histograms, where the average is taken over the optimal transport costs between each observation and the barycenter.

__Exercise:__ Show that the arithmetic mean of observations $\{a_s\}_{s=1}^S$ corresponds to the Wasserstein barycenter when $L_C(a_s, b) = \lVert a_s-b \rVert_2^2$.

### 2.2 Algorithm

As in the previous section, we approximate the solution to the Wasserstein barycenter problem above using an entropy-regularized version of the problem:

$$ \min_{b \in \Sigma_m} \frac{1}{S} \sum_{s=1}^S L^\varepsilon_C(a_s, b).$$

A key advantage of this formulation is that it can be re-written as an optimization problem over $S$ transportation plans $P_s$, $s = 1, ..., S$, such that each $P_s$ has observation $a_s$ as its row sum and all transportation plans share the barycenter $b$ as their column sum:

$$ \sum_{i=1}^{n_1} (P_1)_{ij} = ... = \sum_{i=1}^{n_S} (P_S)_{ij} = b_j, \ \ \text{ for every } j=1, ..., m,$$

or, in matrix notation,

$$ b = P_1^\top 1 = P_2^\top 1 = \ \ ... \ \ = P_S^\top 1.$$

This problem in turn can be solved by an extension of the same Sinkhorn algorithm used in the previous section, where now have $S$ cost matrices $C_s$ and are required to find $S$ sets of scaling vectors $(u_s, v_s)$.

The algorithm proceeds as follows:

After first initializing

\begin{align*}
u_s = v_s = 1, \ \ s = 1, ..., S
\end{align*}

we iteratively update

\begin{align}
u_s &\leftarrow \frac{a_s}{K_s v_s} \\
b &\leftarrow \prod_{s=1}^S (v_s \odot (K_s^\top u_s))^{1/S} \\
v_s &\leftarrow \frac{b}{K_s^T u_s}
\end{align}

The barycenter problem can be extended to _weighted_ averages between histograms by replacing the equal weighting $1/S$ with any collection of weights $\lambda_1, ..., \lambda_S$ that are non-negative and sum to one. We will implement this slightly more general form and visualize barycenters with unequal weighting at the end of this section.

### 2.3 Example

#### 2.3.1 2-D histograms

We'll show an example using image data from MNIST. An image can be considered as a 2-D histogram, with bins corresponding to pixels, after normalizing pixel intensities.

```python
'''
Load, normalize, and plot two 2D histograms (i.e. images).
'''
# MNIST examples
zero_png = imageio.imread('../../../data/otml_neuro/zero.png')
eight_png = imageio.imread('../../../data/otml_neuro/eight.png')

# scale and normalize
rescale = lambda x: (x - x.min()) / (x.max() - x.min())

zero_png = normalize(rescale(zero_png) + .01)
eight_png = normalize(rescale(eight_png) + .01)

# plot
plt.subplot(1,2,1)
plt.imshow(zero_png)
plt.axis("off")

plt.subplot(1,2,2)
plt.imshow(eight_png)
plt.axis("off")

plt.show()
```

The algorithm is implemented in the code block below. It takes advantage of a fast convolutional implementation of kernel multiplication - see [[5]](https://hal.archives-ouvertes.fr/hal-01188953/document) for details.

```python
'''
Implement fast kernel multiplication and generalized Sinkhorn for entropic approximation of Wasserstein barycenters.
'''
epsilon = .04**2
N = zero_png.shape[0]
t = np.linspace(0,1,N)
[Y,X] = np.meshgrid(t,t)
K1 = np.exp(-(X-Y)**2/epsilon)

def K(x): # fast kernel implementation
    return np.dot(np.dot(K1,x),K1)

def barycenter_2d(A, K, lambd, n_iter = 4000, log_errs = True):
    v = np.ones([N,N,S])
    u = np.copy(v)
    Err_q = np.zeros(n_iter)

    for i in range(n_iter):

        for s in range(S):
            if log_errs:
                Err_q[i] = Err_q[i] + np.linalg.norm(u[:,:,s]*K(v[:,:,s]) - A[:,:,s], 1)
            u[:,:,s] = A[:,:,s]/K(v[:,:,s])

        b = np.zeros(N)
        for s in range(S):
            b = b + lambd[s] * np.log(np.maximum(1e-19*np.ones(len(v[:,:,s])), v[:,:,s]*K(u[:,:,s])))
        b = np.exp(b) # update of barycenter in log-space

        for s in range(S):
            v[:,:,s] = b/K(u[:,:,s])

    return Err_q, b
```

We'll first compute the Wasserstein barycenter using equal weights:

```python
S = 2
A = np.stack((zero_png, eight_png), axis=2)
lambd = np.ones(S) / S # equal weighting of histograms
err, b = barycenter_2d(A, K, lambd)
```

As above, we can chart the progress of the algorithm by plotting the marginal constraint violation - in this case, we keep track of the sum of constraint violations for each of the input histograms.

```python
'''
Plot constraint violation vs. iteration number.
'''
plt.figure(figsize=(7,5))
plt.plot(np.log(err),linewidth = 2)
plt.show()
```

And of course we can view the barycenter, which is itself a histogram.

```python
'''
Visualize barycenter.
'''
plt.imshow(b)
plt.axis("off");
```

For a more thorough view of how the Wasserstein barycenter interpolates between the two histograms, we can compute it for a sequence of weightings and view the resulting histograms:

```python
'''
Show sequence of barycenters corresponding to different weightings.
'''
plt.figure(figsize=(16,4))

plt.subplot(1,6,1)
plt.axis("off")
plt.imshow(zero_png)
plt.title("$\lambda_a = 1; \lambda_b = 0$")

ix = 1
for lam in np.arange(0.2,1.0,0.2):
    lambd = [1-lam, lam]
    _, b = barycenter_2d(A, K, lambd)

    plt.subplot(1,6,1+ix)
    plt.axis("off")
    plt.imshow(b)
    plt.title('$\lambda_a = $'+str(np.around(lambd[0], 1))+'; '+'$\lambda_b = $'+str(np.around(lambd[1], 1)))
    ix +=1

plt.subplot(1,6,6)
plt.axis("off")
plt.title("$\lambda_a = 0; \lambda_b = 1$")
plt.imshow(eight_png)

plt.show()
```

__Exercise:__ How do these results compare to what we would find if we simply took the pixel-wise arithmetic (weighted) mean between the two points? Please fill in the code block below.

```python
'''
Show sequence of pixel-wise arithmetic mean corresponding to different weightings.
'''
plt.figure(figsize=(16,4))

plt.subplot(1,6,1)
plt.axis("off")
plt.imshow(zero_png)
plt.title("$\lambda_a = 1; \lambda_b = 0$")

ix = 1
for lam in np.arange(0.2,1.0,0.2):
    lambd = [1-lam, lam]
    b = None  # add your code here to replace None

    plt.subplot(1,6,1+ix)
    plt.axis("off")
    plt.imshow(b)
    plt.title('$\lambda_a = $'+str(np.around(lambd[0], 1))+'; '+'$\lambda_b = $'+str(np.around(lambd[1], 1)))
    ix +=1

plt.subplot(1,6,6)
plt.axis("off")
plt.title("$\lambda_a = 0; \lambda_b = 1$")
plt.imshow(eight_png)

plt.show()
```

## Part III: Optimal transport in computational neuroscience

### 3.1 Intro

Finally, some neuroscience!

The goal of this section is to show how the foregoing concepts can be adapted to the solution of some modern problems in computational neuroscience. We will focus on the analysis of MEG data, which is represented as a collection of real, scalar, non-negative valued measurements at each vertex of a mesh uniformly covering the cortical surface. The measurements correspond to the amplitude of current dipoles with orientation normal to the cortical surface at each vertex location.

From the OT perspective, these measurements can be viewed as histograms, with each vertex corresponding to a unique histogram bin. The cost matrix will reflect the _geodesic distance_ between vertices, that is, the $(i,j)$ entry of the cost matrix represents the spatial distance between vertices $i$ and $j$, taking into account the curved shape of the cortical surface.

We'll start by loading some data and visualizing both an example signal and the cost matrix.

```python
'''
Load some example data and use provided code to compute a cost matrix on the brain surface mesh.
'''
# locate data
data_path = '../../../data/otml_neuro/mne_data/'
subjects_dir = os.path.join(data_path, "subjects")

resolution = 4
spacing = 'ico%d' % resolution

# get surface mesh and compute distances between vertices
surf, dist = get_surf_dist(spacing, subjects_dir)
n_features = len(dist)

# summarize result
print('Pairwise vertex distance matrix has size: {}'.format(dist.shape))
print('Maximum distance between vertices: {:.2f} cm'.format(100 * dist.max()))

# generate surface map
bg_map = os.path.join(subjects_dir, "fsaverage/surf/lh.sulc")
bg_map = nilearn.surface.load_surf_data(bg_map)[:n_features]
```

We use the _fsaverage_ surface map as our reference and compute geodesic distances using the [Floyd-Warshall algorithm](https://en.wikipedia.org/wiki/Floyd%E2%80%93Warshall_algorithm) given the points and triangles of the surface mesh. The Floyd-Warshall algorithm is an efficient method to compute the shortest path between each pair of nodes in a network; in this application, the nodes are vertices of the brain mesh, and the network edges contain information on the distance between the connected vertices.

Next we simulate a signal on a subset of surface mesh vertices and plot the result.

```python
'''
Get vertex labels and simulate a source signal.
'''
label_name = 'lh.BA45'
fname_label = os.path.join(subjects_dir,
                           'fsaverage/label/%s.label' % label_name)

label = mne.read_label(fname_label, subject="fsaverage")
label = label.morph(subject_to="fsaverage", grade=resolution,
                    subjects_dir=subjects_dir)

# simulate signal on subset of label vertices
seed = 42
rng = np.random.RandomState()

activation_size = 20
activations = np.zeros((n_features, 1))

picks = rng.choice(label.vertices, activation_size)
signal = rng.rand(activation_size)
activations[picks, 0] = signal / np.sum(signal) # signal is a valid histogram
```

```python
'''
View the simulated signal.
'''
view = plotting.plot_surf_stat_map(surf, activations, colorbar=True,
                                   alpha=1., threshold=0.01, cmap="viridis",
                                   bg_map=bg_map, vmax=activations.max() / 2)
```

We can also visualize the cost matrix by plotting a heatmap over the surface mesh showing distance from a fixed reference point. We choose the reference point to be roughly at the center of the simulated signal, but feel free to change it to view distances from other locations.

```python
'''
Visualize cost as distance from a fixed point on the cortical surface.
'''
reference_vertex = 120 # reference vertex chosen to be roughly near simulated signal

view = plotting.plot_surf_roi(surf, 100 * dist[reference_vertex], colorbar=True)
ax, colorbar_ax = view.get_axes()
_ = colorbar_ax.set_ylabel("Geodesic distance (cm)")
```

### 3.2 Wasserstein barycenters of brains

One recent application for OT in computational neuroscience [[1](https://arxiv.org/pdf/1902.04812.pdf),[2](https://arxiv.org/pdf/1805.07833.pdf)] is motivated by the problem of pooling information about the spatial localization of signals across many recordings, as might be obtained, for example, in a group study involving several individuals. The challenge is to identify a principled method to summarize a collection of signals. In this section, we illustrate an OT approach that computes the Wasserstein barycenter of a collection of source signals on a human brain surface.

We'll start by simulating a collection of signals in source space, corresponding to the measurements that might be obtained from different subjects.

```python
'''
Simulate and visualize a collection of signals in source space.
'''
# simulate activations
n_subjects = 6
activation_size = 3
activations = np.zeros((n_features, n_subjects))
for i in range(n_subjects):
    picks = rng.choice(label.vertices, activation_size)
    activations[picks, i] = rng.rand(activation_size) + 3

# plot
f, axes = plt.subplots(n_subjects // 2, 2, subplot_kw={'projection': '3d'}, figsize=(18, 10))
axes = axes.ravel()
for i in range(n_subjects):
    view = plotting.plot_surf_stat_map(surf, activations[:, i], colorbar=True,
                                       alpha=1., threshold=0.01, cmap="viridis",
                                       bg_map=bg_map, axes=axes[i],
                                       vmax=activations[:, i].max() / 2)

    axes[i].set_title("Subject %d" % (i + 1))
```

To summarize information across these signals, we compute the Wasserstein barycenter of the simulated signals, using the distance matrix computed above as the ground cost of transport between points on the cortical surface.

*Technical note*: Strictly speaking, the Wasserstein barycenter is only defined on histograms, and (as would also be the case with real MEG source signals) the data simulated above are not histograms because they are not normalized. Fortunately, the same approach generalizes to this unnormalized case, via a formulation known as "unbalanced optimal transport" - this is what is implemented below (see references for more details).

```python
'''
Compute and plot the Wasserstein barycenter.
'''
M = dist / np.median(dist)  # this normalization by the median works well in practice
epsilon = 1. / n_features
alpha = 10.  # regularization for unbalanced OT
barycenter = barycenter_unbalanced(abs(activations) + 1e-10, M, epsilon, alpha) # compute barycenter

view = plotting.plot_surf_stat_map(surf, barycenter, colorbar=True, cmap="viridis",
                                   alpha=1., threshold=0.01,
                                   bg_map=bg_map, vmax=barycenter.max() / 2,
                                   title='Wasserstein barycenter')
```

Excercise: Summarize information across these signals using the arithmetic mean. Please fill in the code block below.

```python
'''
Compute and plot the naive (Euclidean) average.
'''
euclidean_mean = None  # add your code here to replace None
view = plotting.plot_surf_stat_map(surf, euclidean_mean, colorbar=True, cmap="viridis",
                                   alpha=1., threshold=0.01,
                                   bg_map=bg_map, vmax=euclidean_mean.max() / 2,
                                   title='Euclidean averaged signal')
```

### 3.3 Optimal transport regularization for MEG source imaging

#### 3.3.1 Intro

Finally, we discuss the application of these OT tools to a problem of scientific interest: group-level source imaging with MEG data. Source imaging is an inverse problem in which a lower-dimensional observed signal is mapped to a high-dimensional but unobserved source space.

In a group study, signals are recorded from multiple individuals under the same experimental condition, and the objective is to infer a source signal for each individual. This is known as the "multi-task" setting in machine learning. The multi-task problem can be written as

$$ \min_{x_1, ..., x_S \in \mathbb{R}^p} \frac{1}{2n} \sum_{s=1}^S || y_s - L_s x_s ||_2^2,$$

where $y_s \in \mathbb{R}^n$ is the observed signal, $L_s \in \mathbb{R}^{n \times p}$ is the known linear transformation from source to sensor space, and $s \in \{1, ..., S\}$ indexes the individuals in the group study. The goal is to find the source signals $x_1, ..., x_S$ that minimize the reconstruction error of the observed signal in sensor space.

#### 3.3.2 Regularization via the Wasserstein barycenter

In general, inverse problems of the kind described above are "ill-posed" in the sense that they do not admit a unique solution; there are many possible source signals that may have given rise to the observed sensor measurements. The problem must therefore be constrained by some additional component that helps us select between possible explanations for the data - in optimization, this is known as _regularization_.

Recently, OT has been applied to the group-level source imaging problem as a means to define a regularization term, which yields the _multi-task Wasserstein_ (MTW) model [[2]](https://arxiv.org/pdf/1805.07833.pdf). The idea is to add a term of the form

$$ \min_{b \in \mathbb{R}^p} \frac{1}{S} \sum_{s=1}^S L^\varepsilon_C(b, x_s)$$

to the multi-task problem. The result is that the estimated signals $x_1, ..., x_S$ are encouraged to be "close" to each other in the sense of minimizing distance to the (equal-weight) Wasserstein barycenter. This reflects the expectation that there will be some similarity across individuals in the study.

The full MTW problem is given by

$$ \min_{x_1, ..., x_S \in \mathbb{R}^p} \overbrace{\frac{1}{2n} \sum_{s=1}^S || y_s - L_s x_s ||_2^2}^{\text{fit to observed signal}} + \overbrace{\alpha \min_{b \in \mathbb{R}^p} \frac{1}{S} \sum_{s=1}^S L^\varepsilon_C(b, x_s)}^{\text{distance to source barycenter}} + \underbrace{\beta ||x_s||_1}_{\text{sparsity}},$$

where the additional $||x_s||_1$ term enforces sparsity in the estimated signal. The parameters of the model are thus $\varepsilon$ (entropy smoothing), $\alpha$ (strength of Wasserstein penalty), and $\beta$ (strength of sparsity penalty).

#### 3.3.3 Example

We include a small example using MEG recordings from two individuals. The data are obtained from [[4]](http://doi.org/10.5281/zenodo.889235); they have already been preprocessed, so that for each individual we have the corresponding gain matrix and sensor measurements.

```python
'''
Load preprocessed MEG data, consisting of observed measurements and gain matrices for each individual.
'''
data_file = '../../../data/otml_neuro/MEG_preprocessed.pkl'

with open(data_file, 'rb') as f:
    data = pickle.load(f)

gains = data['gain']
Ms = data['M']

print("Gain matrices : {} x {} x {} (n_subject x n_sensor x n_source)".format(*gains.shape))
print("Observed signal : {} x {} x {} (n_subject x n_sensor x time)".format(*Ms.shape))
```

Next we define an MTW model, which requires setting the parameters $\varepsilon$, $\alpha$, $\beta$ defined above.

The model is subsequently fit to the preprocessed data. We limit the optimization routine to maintain reasonable computational requirements for this notebook; this should still be enough to give an idea of the result, though of course the optimization would be run to convergence in a real study.

```python
'''
Instantiate and fit a MTW model.
'''
M = Ms[:,:,20] # infer signal at time point 20

epsilon = 100. / n_features # set model parameters
alpha = 10.0
beta = 2.0
sigma0 = 0.01

mtw = MTW(M=dist, epsilon=epsilon, alpha=alpha, beta=beta, sigma0=sigma0) # instantiate MTW object

mtw.maxiter = 20 # reduce iteration / tolerance settings to reduce computational burden
mtw.tol_cd = 1e-2
mtw.tol_ot = 1e-3

mtw = mtw.fit(gains, M) # fit to MEG data
```

The results are stored in the MTW attribute coefs_: they have shape $(\text{n_source} \times \text{n_subject})$, corresponding to an estimated source signal for each subject.

```python
coefs = mtw.coefs_
coefs.shape
```

The distribution of weights can be viewed by plotting histograms for each subject. Note the sparsity of the result, which is due to the $\ell1$-norm penalty in the objective function.

```python
'''
Plot histograms of estimated source space weights.
'''

plt.figure(figsize=(10,4))

plt.subplot(1,2,1)
plt.hist(coefs[:,0])
plt.title('Subject 1')

plt.subplot(1,2,2)
plt.hist(coefs[:,1])
plt.title('Subject 2')

plt.tight_layout()
plt.show()
```

Finally, we can view the estimated signals in the source space.

```python
'''
Visualize estimated source signals.
'''
f, axes = plt.subplots(1, 2, subplot_kw={'projection': '3d'}, figsize=(16, 6))
axes = axes.ravel()

for i in range(2):
    view = plotting.plot_surf_stat_map(surf, coefs[:,i], colorbar=True,
                                   alpha=1., threshold=2e-6, cmap="viridis", axes=axes[i],
                                   bg_map=bg_map, vmax=coefs[:,i].max() / 2)

    axes[i].set_title("Subject %d" % (i + 1))
```

We have given a demonstration of the method on real data, but of course important scientific questions remain about its application. For a more thorough validation of the MTW approach to group-level source imaging, we refer to the experiments in original paper [[1]](https://arxiv.org/pdf/1902.04812.pdf).

## Acknowlegments

L. Liu gratefully acknowledges support from NSF DMS grant #1839291. A. Greaves-Tunnell gratefully acknowledges support from NSF DMS grant #1839291 and NIH Big Data for Genomics and Neuroscience Training Grant #8T32LM012419. Z. Harchaoui gratefully acknowledges support from NSF DMS grant #1839291.

L. Liu, A. Greaves-Tunnell and Z. Harchaoui would like to warmly thank H. Janati, A. Gramfort, N. Courty, M. Cuturi, G. Peyré, and R. Flamary for generously sharing their codes and notebooks. The neuroscience component of the lab is largely based on the companion codes to the papers [[1]](https://arxiv.org/pdf/1902.04812.pdf) and [[2]](https://arxiv.org/pdf/1805.07833.pdf), while exposition of OT concepts is based on G. Peyré's excellent [Numerical Tours](https://www.numerical-tours.com/python/), particularly the [notebook on entropic regularization](https://nbviewer.jupyter.org/github/gpeyre/numerical-tours/blob/master/python/optimaltransp_5_entropic.ipynb).

## References and further resources

[[1]] Janati, H., Bazeille, T., Thirion, B., Cuturi, M., & Gramfort, A. [Group level MEG/EEG source imaging via optimal transport: minimum Wasserstein estimates.](https://arxiv.org/pdf/1902.04812.pdf) _In International Conference on Information Processing in Medical Imaging_. (2019)

[[2]] Janati, H., Cuturi, M., & Gramfort, A. [Wasserstein regularization for sparse multi-task regression.](https://arxiv.org/pdf/1805.07833.pdf) _AISTATS_. (2019)

[[3]] Leonard, C. [From the Schrodinger problem to the Monge-Kantorovich problem.](https://arxiv.org/abs/1011.2564) _Journal of Functional Analysis_. (2012)

[[4]] Nurminen J., Paananen H., Mäkelä J. [High frequency somatosensory MEG: evoked responses, FreeSurfer reconstruction](http://doi.org/10.5281/zenodo.889235). (2017)

[[5]] Solomon, J., De Goes, F., Peyré, G., Cuturi, M., Butscher, A., Nguyen, A., Du, T. and Guibas, L. [Convolutional wasserstein distances: Efficient optimal transportation on geometric domains.](https://hal.archives-ouvertes.fr/hal-01188953/document) _ACM Transactions on Graphics (TOG), 34(4), p.66._ (2015)

The section on Wasserstein barycenters for brains is based on the example notebook in H. Janati's [GitHub repository](https://github.com/hichamjanati/OT-tutorials).

For an extensive introduction to the theory, computational aspects, and applications of optimal transport, we refer interested readers to the textbook of Cuturi and Peyré:

- Peyré, G., & Cuturi, M. (2019). [Computational Optimal Transport.](https://arxiv.org/pdf/1803.00567.pdf) Foundations and Trends in Machine Learning, 11(5-6), 355-607.