This is the practical component of the [Machine Learning Summer School, Moscow 2019](https://mlss2019.skoltech.ru/) session on kernels, focusing on hypothesis testing with kernel statistics.

The materials here are most recently by
[Dougal Sutherland](http://www.gatsby.ucl.ac.uk/~dougals/)
with consultation from [Arthur Gretton](http://www.gatsby.ucl.ac.uk/~gretton/),
updated from [a previous course](https://github.com/dougalsutherland/ds3-kernels/),
and based in large part on [earlier materials](https://github.com/karlnapf/ds3_kernel_testing)
by [Heiko Strathmann](http://herrstrathmann.de/).

We'll cover, in varying levels of detail, the following topics:

- Two-sample testing with the kernel Maximum Mean Discrepancy (MMD).
  - Basic concepts of hypothesis testing, including permutation tests.
  - Computing kernel values.
  - Estimators for the MMD.
  - Learning an appropriate kernel function.
- Independence testing with the Hilbert-Schmidt Independence Criterion.

## Dependencies

### Colab

If you open this notebook in Google Colab, the only thing you need to do is to intall package by using the code below. Everything will be running on cloud resources, so this is probably the easiest option if you trust that your network connection is going to be reasonably reliable. Make a copy to your own Google Drive to save your progress, and to use a GPU, click Runtime -> Change runtime type -> Hardware accelerator -> GPU. Everything you need is already installed on Colab; use a Python 3 notebook.

### Local setup

#### Python version
This notebook requires Python 3.6+. Python 3.0 was released in 2008, and it's time to stop living in the past; most importart Python projects [are dropping support for Python 2 this year](https://python3statement.org/). If you've never used Python 3 before, don't worry! It's almost the same; for the purposes of this notebook, you probably only need to know that you should write `print("hi")` since it's a function call now, and you can write `A @ B` instead of `A.dot(B)`.

#### Python packages

The main thing we use is PyTorch and Jupyter. If you already have those set up, you should be fine; just additionally make sure you also have (with `conda install` or `pip install`) `seaborn`, `tqdm`, and `sckit-learn`. We import everything right at the start, so if that runs you shouldn't hit any surprises later on.

If you don't already have a setup you're happy with, we recommend the `conda` package manager - start by installing [miniconda](https://docs.conda.io/en/latest/miniconda.html). Then you can create an environment with everything you need as:

```bash
conda create --name mlss-testing --override-channels -c pytorch -c defaults --strict-channel-priority python=3 notebook ipywidgets numpy scipy scikit-learn pytorch=1.1 torchvision matplotlib seaborn tqdm
conda activate mlss-testing

pip install --upgrade git+https://github.com/mlss-skoltech/tutorials.git

jupyter notebook
```

(If you have an old conda setup, you can use `source activate` instead of `conda activate`, but it's better to [switch to the new style of activation](https://conda.io/projects/conda/en/latest/release-notes.html#recommended-change-to-enable-conda-in-your-shell). This won't matter for this tutorial, but it's general good practice.)

(You can make your life easier when using jupyter notebooks with multiple kernels by installing `nb_conda_kernels`, but as long as you install and run `jupyter` from inside the env it will also be fine.)

## PyTorch

We're going to use PyTorch in this tutorial, even though we're not doing a ton of "deep learning." (The CPU version will be fine, though a GPU might let you get slightly better performance in some of the "advanced" sections.)

If you haven't used PyTorch before, don't worry! The API is unfortunately a little different from NumPy (and TensorFlow), but it's pretty easy to get used to; you can refer to [a cheat sheet vs NumPy](https://github.com/wkentaro/pytorch-for-numpy-users/blob/master/README.md) as well as the docs: [tensor methods](https://pytorch.org/docs/stable/tensors.html) and [the `torch` namespace](https://pytorch.org/docs/stable/torch.html#torch.eq). Feel free to ask if you have trouble figuring something out.

You can convert a `torch.Tensor` to a `numpy.ndarray` with [`t.numpy()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.numpy), and vice versa with [`torch.as_tensor()`](https://pytorch.org/docs/stable/torch.html#torch.as_tensor). (These share data when possible.) Doing this breaks PyTorch's ability to track gradients through these objects, but it's okay for things we won't need to take derivatives of. If you have a one-element tensor, you can get a regular Python number out of it with [`t.item()`](https://pytorch.org/docs/stable/tensors.html#torch.Tensor.item).

$
\DeclareMathOperator{\E}{\mathbb E}
\DeclareMathOperator{\Var}{Var}
\DeclareMathOperator{\Tr}{Tr}
\DeclareMathOperator{\MMD}{MMD}
\DeclareMathOperator{\MMDhat}{\widehat{MMD}}
\newcommand{\PP}{\mathbb{P}}
\newcommand{\QQ}{\mathbb{Q}}
\newcommand{\h}{\mathcal H}
$

```python
!pip install --upgrade git+https://github.com/mlss-skoltech/tutorials.git#subdirectory=kernels
```

```python
import os
import pkg_resources

DATA_PATH = pkg_resources.resource_filename('probability_testing', 'data/')

%matplotlib inline
import matplotlib as mpl
import matplotlib.pyplot as plt
import seaborn as sns
sns.set(context='notebook')
from IPython.display import display, Markdown, HTML

from tqdm import tqdm_notebook as tqdm
# from tqdm import tqdm  # if you're in JupyterLab/etc and tqdm_notebook doesn't work well

import functools

import numpy as np
import sklearn
from sklearn import model_selection
import torch

from probability_testing import support
from probability_testing.support import as_tensors, maybe_squeeze
```

## A note

Please ask for help if you get stuck! Whether it's with a code thing or something conceptual, your instructors are happy to help you try to work through things. Solutions are also available in [`solutions-testing.ipynb`](solutions-testing.ipynb), but you're probably better off not rushing straight to those.

## Starting out with two-sample testing

```python
with np.load(os.path.join(DATA_PATH, 'simple.npz')) as data:
    X, Y = as_tensors(data['X'], data['Y'])

plt.hist(X.numpy(), alpha=.5, bins='auto', histtype='stepfilled')
plt.hist(Y.numpy(), alpha=.5, bins='auto', histtype='stepfilled');
```

Question: do $X$ and $Y$ come from the same distribution?

## Comparing means

Let's start by comparing the means of the samples.
If the distributions are the same, their means are the same,
so the expectation of this statistic would be zero.

```python
def mean_difference(X, Y, squared=False):
    # make sure X and Y are 1d tensors:
    # the data files have them as [n, 1] to go with Python convention
    # (and for us to use later)
    X, Y = [maybe_squeeze(t, 1) for t in as_tensors(X, Y)]
    assert len(X.shape) == len(Y.shape) == 1

    # TODO: compute mean difference of X and Y in `result`

    return (result * result) if squared else result

mean_difference(X, Y).item()
```

Well, it's not zero...but of course we were never going to get *exactly* zero.

Where do we draw the line?

The classical statistical way to do this is to assume that $P_X = P_Y$ (the null hypothesis), and consider what the distribution of the test statistic would be in that case (the null distribution). If the number we observe would be very unlikely under the null distribution, then we "reject the null" and say that the two samples are different.

One way to do this is called _permutation testing_.

### Permutation testing (distribution of test statistic under $P_X = P_Y$)

This is based on the observation that if $P_X = P_Y$, then we can shuffle the samples together, and that won't change the distribution of our shuffled $X'$ or $Y'$ (under the null hypothesis). We can then compute what our test statistic would be if we did this a bunch of times, and this will give us an estimate of what the true null distribution is. If we do a lot of permutation samples, and our test statistic landed above, say, 99% of them, then we can reject the null hypothesis.

Permutation test procedure
1. Concatenate `X` and `Y` into a new set `Z`
2. Shuffle samples in `Z`
3. Split `Z` into two sets `X'` and `Y'` of sizes $n_X$ and $n_Y$
4. Calculate test statistics using `X'` and `Y'`
5. Repeat steps 1-4 several times

**Exercise:** fill in the following method for implementing two-sample tests. Here `test_statistic` is a variable representing a function, like `mean_difference`, that will compute some statistic given two inputs.

```python
def two_sample_permutation_test(
    test_statistic, X, Y, num_permutations=500, progress=True
):
    X, Y = as_tensors(X, Y)
    assert X.shape[1:] == Y.shape[1:]  # check same dimensionality

    orig_stat = test_statistic(X, Y)

    range_ = range(num_permutations)
    if progress:
        range_ = tqdm(range_)

    stats = []
    for i in range_:
        # TODO: compute test statistic on permuted samples, in `this_stat`
        stats.append(this_stat)

    return orig_stat, torch.stack(stats)
```

Here's a helper function I'll give you to visualize the results of the permutation:

```python
def plot_null_samples(
    statistic,        # the value of the actual test statistic
    null_samples,     # an array of samples from the statistic's null distribution
    ax=None,          # a matplotlib axis object; none for the default one
    from_zero=False,  # indicate it's a one-sided test whose statistic is always >= 0
    one_sided=True,   # indicate it's a one-sided test for high values of the statistic
    alpha=1,          # transparency of the histogram for the null samples
    level=0.05,       # level of the test, e.g. 0.05 for 5% false rejection rate
):
    null_samples = np.asarray(null_samples)
    if ax is None:
        ax = plt.gca()

    ax.hist(
        null_samples,
        bins="auto",
        histtype="stepfilled",
        label="Permutation samples",
        alpha=alpha,
        density=True,
    )

    if from_zero:
        lo = 0
        hi = np.percentile(null_samples, 100 * (1 - level))
    elif one_sided:
        lo = np.min(null_samples) - 0.02 * (np.max(null_samples) - np.min(null_samples))
        # should be -inf, but axvspan doesn't support that
        hi = np.percentile(null_samples, 100 * (1 - level))
    else:
        lo = np.percentile(null_samples, 100 * level / 2)
        hi = np.percentile(null_samples, 100 * (1 - level / 2))

    ax.axvspan(lo, hi, fc="b", alpha=0.25, label=f"{1 - level:.1%} region")

    ax.axvline(x=statistic, c="r", lw=2, label="Actual statistic")
    ax.legend(loc="best")

    ax.set_xlabel("Test statistic value")
    ax.set_ylabel("Density")

    if from_zero:
        ax.set_xlim(0, ax.get_xlim()[1])
    elif one_sided:
        ax.set_xlim(lo, ax.get_xlim()[1])
```

```python
plot_null_samples(
    *two_sample_permutation_test(mean_difference, X, Y),
    one_sided=False)

# The * here is a bit of Python helper syntax. If you haven't seen it before,
# fn(*it) calls fn like fn(it[0], it[1], ...) for the length of it,
# so here it's like doing
#   stat, nulls = two_sample_permutation_test(...)
#   plot_null_samples(stat, nulls)
```

Depending on the test statistic, there might be many other ways to compute the null distribution.
Not all of them work via sampling it directly like this – which can be pretty expensive, e.g. 500 times as expensive as the test in the first place....

For example, for our mean difference test statistic, if we assume that $X$ and $Y$ are each Gaussian, the distribution of the statistic is also Gaussian – as we can see, at least roughly, in this plot.
We might be even able to analytically work out the parameters of this distribution.

If you haven't realized, this would be basically a [t-test](https://en.wikipedia.org/wiki/Student%27s_t-test), where we assume that we know that both distributions have the same variance so we can distinguish them via solely looking at their mean. (In this particular case, that's true....)

The advantage of permutation tests, though, is that you don't have to make strong parametric assumptions like this.

As it will be useful for kernel based test statistics later, we will also look at the distribution of a squared test statistic, in order to make it strictly positive, but still tend to zero if the two distributions are the same.

```python
squared_mean_diff = functools.partial(mean_difference, squared=True)
# squared_mean_diff is a new function object that just calls mean_difference,
# but defaults the `squared` argument to True

plot_null_samples(
    *two_sample_permutation_test(squared_mean_diff, X, Y),
    from_zero=True)
```

### Alternative distribution via explicit simulation (distribution of test statistic under $P_X \neq P_Y$)

The test statistic also has a distribution under the alternative hypothesis ($P_X \neq P_Y$); our computed test statistic is only a single sample from this distribution.

Unfortunately, it's not easy to look at this alternative distribution in practice,
unless we have a way to generate more data.
But let's take a look in a synthetic case where we know the true distribution.

**Exercise:** sample from the alternative to make the comparison plot below.

```python
np.random.seed(0); torch.manual_seed(5)

# This is a case when the null hypothesis is false:
N = 200
sample_X = lambda: torch.randn(N, 1)
sample_Y = lambda: torch.randn(N, 1) + 0.3

# Do the normal testing thing, where we have one sample from each.
X = sample_X()
Y = sample_Y()

# single sample from the alternative + null estimates
stat, perm_samples = two_sample_permutation_test(squared_mean_diff, X, Y, num_permutations=500)

statistics_alt = np.zeros(500)
# TODO: fill statistics_alt with samples from the alternative distribution

fig, ax = plt.subplots(figsize=(12, 4))
ax.hist(statistics_alt, alpha=0.5, bins='auto', density=True,
        label='Alternative samples', histtype='stepfilled',
        color=sns.color_palette()[1])
plot_null_samples(stat, perm_samples, ax=ax, from_zero=True, alpha=.5)
```

As you can see, even though most of the alternative distribution is far larger than the null, we can be unlucky: there are a significant portion of datasets for which the test would not reject the null simply by chance.
This particular seed was one of the unlucky ones: most of the time we'd reject, but this time we didn't. The probability with which the test rejects when the alternative is false is called the *power*.

**Optional:** Play with the sample size, or the amount of difference between $X$ and $Y$, and see how this impacts the position of the alternative distribution (and as such the test power).

**Optional:** Actually generate more data from the null distribution, also. How well does the permutation distribution approximate the true null? How does this change with smaller / larger `N`?

### Another dataset

```python
with np.load(os.path.join(DATA_PATH, 'almost_simple.npz')) as d:
    X, Y = as_tensors(d['X'], d['Y'])

plt.hist(X.numpy(), alpha=.5, bins='auto', histtype='stepfilled')
plt.hist(Y.numpy(), alpha=.5, bins='auto', histtype='stepfilled');
```

Looks like the mean won't cut it this time. Maybe the standard deviation?

**Exercise:** Implement a two-sample test statistic based on both the mean and the standard deviation, e.g. $(\mu_X - \mu_Y)^2 + (\sigma_X - \sigma_Y)^2$.

```python
def mean_std_stat(X, Y):
    # make data 1d
    X, Y = [maybe_squeeze(t, 1) for t in as_tensors(X, Y)]
    assert len(X.shape) == len(Y.shape) == 1

    # TODO: return some statistic based on the mean and std dev

plot_null_samples(
    *two_sample_permutation_test(mean_std_stat, X, Y),
    from_zero=True)  # if your statistic can be negative, remove the from_zero argument...
```

At least for the statistic I picked, this was comfortably inside the rejection region, even though visually the samples are quite different.

Of course, it's possible we were just unlucky (as we saw could happen above) – but actually, the distributions I generated this data from have exactly the same means and standard deviations. Clearly, this test is never going to be able to tell such data apart, so it's time to move onto something better. We could keep adding more and more moments, but those get hard to estimate pretty quickly, so let's remember what we talked about in the lectures instead.

## Maximum Mean Discrepancy

As we saw in the lectures, the MMD can be thought of as either
$$
\MMD(\PP, \QQ)
= \sup_{f \in \h : \lVert f \rVert_\h \le 1} \E_{X \sim \PP}[f(X)] - \E_{Y \sim \QQ}[ f(Y) ]
$$
or, more relevantly right now,
$$
\MMD(\PP, \QQ)
=
\lVert
\E_X[ \varphi(X) ]
- \E_Y[ \varphi(Y) ]
\rVert_\h
$$
where $\varphi : \mathcal X \to \h$ is the feature map of the kernel $k(x, y) = \langle \varphi(x), \varphi(y) \rangle_\h$.

If we take $\varphi(x) = x$, corresponding to a linear kernel, then our squared difference in means is exactly the squared MMD. But we can do a lot better by picking a different kernel.

**Exercise:** does the difference in means + standard deviations you used above correspond to an MMD with some kernel? If so, what kernel? If not, is there a kernel MMD that can distinguish the same set of distributions as that can?

## Computing kernels

To help organize computing kernels, I've put some infrastructure in the `LazyKernel` class (in `probability_testing.support.kernels`, if you want to look at it). It handles a bunch of gruntwork that you won't want to deal with later on. Here's an example of how to use it to implement a kernel:

```python
from probability_testing.support import LazyKernel

class LinearKernel(LazyKernel):
    def _compute(self, X, Y):
        return X @ Y.t()
```

Here, the `_compute` method computes the kernel between two matrices of inputs, `X` and `Y`, of shape `[n_X, dim]` and `[n_Y, dim]`; it returns a kernel matrix of shape `[n_X, n_Y]`. (Here, `.t()` is PyTorch syntax for a transpose, and `@` is the nifty Python 3.6+ syntax for matrix multiplication.)

The `LazyKernel` base class lets us use this in various ways. First, to find the kernel from one set of points to another:

```python
K = LinearKernel(X, Y)
```

To get the kernel matrices X-to-Y, X-to-X or Y-to-Y you can use `K.XY`, `K.XX` and `K.YY` repsectively
*(which are the result of `_compute(X, X)` and `_compute(Y, Y)`. These aren't computed until you need them, but then they're cached after you use them the first time; this is why it's a `LazyKernel`.)*

```python
K.XX, K.XY, K.YY
```

If you only want the kernel matrix for a dataset to itself, you can just not pass the second argument. Then `K.XY` won't exist.

```python
LinearKernel(X)
```

You can also pass three arguments; then there'll be `XZ`, etc. You can also access them with e.g. `K[0, 2]`.

Alternatively, you can pass `None`, which is a special value meaning "use the first one." Then `XY` and so on will exist, but it knows to cache them appropriately.

```python
K = LinearKernel(X, None, Y)
print(K)
K.YZ
```

There are also some useful methods in `LazyKernel` that you can find in the source code for it.

**Exercise:** Remember from lecture that a *characteristic* kernel can distinguish *any* pair of distributions (with enough samples). None of the kernels we've implemented so far are characteristic. Your job is to implement one: the kernel that goes by names including exponentiated quadratic, squared exponential, Gaussian RBF, and probably more:

$$k(X_i, X_j) = \exp\left( -\frac{1}{2 \sigma^2} \lVert X_i - X_j \rVert^2 \right).$$

It might be helpful to recall that
$$\lVert X_i - X_j \rVert^2 = \lVert X_i \rVert^2 + \lVert X_j \rVert^2 - 2 X_i^T X_j;$$
you should probably convert that into a matrix form to implement it in `_compute`.

Alternatively, [`torch.pdist`](https://pytorch.org/docs/stable/nn.html#torch.nn.functional.pdist) computes these distances, but it only gives you the upper triangle of the matrix so you'll have to turn it into a full symmetric matrix yourself.

```python
class ExpQuadKernel(LazyKernel):
    def __init__(self, *parts, sigma=1):
        super().__init__(*parts)
        self.sigma = sigma
        self.const_diagonal = 1  # Says that k(x, x) = 1 for any x.
                                 # Just a slight optimization; not really necessary.

    # TODO: implement _compute

```

You can check your implementation against scikit-learn's implementation (but it doesn't work in PyTorch, so don't just use it directly):

```python
sigma = np.random.lognormal()
K = ExpQuadKernel(X, Y, sigma=sigma)

from sklearn.metrics.pairwise import rbf_kernel
gamma = 1 / (2 * sigma**2)  # sklearn uses this parameterization
assert np.allclose(K.XX.numpy(), rbf_kernel(X, gamma=gamma))
assert np.allclose(K.XY.numpy(), rbf_kernel(X, Y, gamma=gamma))
assert np.allclose(K.YY.numpy(), rbf_kernel(Y, gamma=gamma))
del rbf_kernel
```

### `Matrix` classes

Okay, one more handy thing about these `LazyKernel` classes that might be useful here. (You don't have to use them, but you can.)

if you do `K.XY_m` (or `K.YY_m` or `K.matrix(0, 1)`, etc), then you get a special `support.kernels.Matrix` subclass. This implements – and caches – various operations you might need. Let's take a look:

```python
K = ExpQuadKernel(X, Y)
```

```python
K.XY_m, K.XX_m
```

There are several offdiagonal operations that you may need later:

```python
print('  \n'.join(m for m in dir(K.XY_m) if not m.startswith('_') and 'offdiag' in m))
```

```python
K.XX_m.mean(), K.XX_m.offdiag_mean()
```

Check out the second half of `probability_testing.support.kernels` to see how they're implemented / what options there are.

### MMD estimators

Okay, enough admiring my beautiful code. Remember that we have
\begin{align}
\MMD^2(\PP, \QQ)
  &= \lVert \E_X[ \varphi(X) ] - \E_Y[ \varphi(Y) ] \rVert_\h^2
\\&= \langle
        \E_X[ \varphi(X) ] - \E_Y[ \varphi(Y) ],
        \E_X[ \varphi(X) ] - \E_Y[ \varphi(Y) ]
     \rangle_\h
\\&= \langle \E_X[ \varphi(X) ], \E_X[ \varphi(X) \rangle_\h
   + \langle \E_Y[ \varphi(Y) ], \E_Y[ \varphi(Y) \rangle_\h
   - 2 \langle \E_X[ \varphi(X) ], \E_Y[ \varphi(Y) ] \rangle_\h
\\&= \E_{X, X', Y, Y'}\left[
     \langle \varphi(X), \varphi(X') \rangle_\h
   + \langle \varphi(Y), \varphi(Y') \rangle_\h
   - 2 \langle \varphi(X), \varphi(Y) \rangle_\h
   \right]
\\&= \E_{X, X', Y, Y'}\left[
     k(X, X')
   + k(Y, Y')
   - 2 k(X, Y)
   \right]
.\end{align}

This gives a pretty natural idea for how to estimate the MMD. Well, three ideas of varying amounts of naturalness:

The *biased* estimator (exactly the MMD between the empirical distributions) is, if we have $m$ samples from $X$ and $n$ from $Y$:
$$
\MMDhat_b^2(X, Y)
= \frac{1}{m^2} \sum_{i=1}^m \sum_{j=1}^m k(X_i, X_j)
+ \frac{1}{n^2} \sum_{i=1}^n \sum_{j=1}^n k(Y_i, Y_j)
- \frac{2}{m n} \sum_{i=1}^m \sum_{j=1}^n k(X_i, Y_j)
.$$
But this term has a bias due to the $k(X_i, X_i)$ and $k(Y_i, Y_i)$ terms.

(You can tell that it's biased because if $X$ and $Y$ are each samples from the same distribution, the true MMD value is $0$, usually $\MMDhat^2(X, Y) > 0$, and it's not possible to have $\MMDhat^2(X, Y) < 0$ – thus $\E \MMDhat^2(X, Y) > 0$.)

The *unbiased* estimator gets rid of these terms:
$$
\MMDhat_u^2(X, Y)
= \frac{1}{m (m-1)} \sum_{i \ne j}^m k(X_i, X_j)
+ \frac{1}{n (n-1)} \sum_{i \ne j}^n k(Y_i, Y_j)
- \frac{2}{m n} \sum_{i=1}^m \sum_{j=1}^n k(X_i, Y_j)
.$$
This makes it unbiased; in fact, it's the minimum variance unbiased estimator.

The $U$-statistic estimator only works when $n = m$, and also takes out the $k(X_i, Y_i)$ terms, which gives you a slightly worse estimator:
$$
\MMDhat_U^2(X, Y)
= \frac{1}{m (m-1)} \sum_{i \ne j}^m \left(
    k(X_i, X_j) + k(Y_i, Y_j) - 2 k(X_i, Y_j)
  \right)
.$$
The advantage is that $U$-statistics have been studied pretty thoroughly by statisticians, so we know things about their variance and asymptotic distributions and whatnot.

**Exercise:** Implement at least one of these estimators, as functions of a `LazyKernel(X, Y)`. Using the `Matrix` helpers, each can be literally one line, but you can implement it however you feel like. You don't necessarily have to do all three; do at least one of them, but the others aren't too much extra work on top of the first one.

```python
def mmd2_biased(K):
    pass  # TODO: implement, using K.XX_m, K.XY_m, K.YY_m (or K.XX, K.XY, K.YY)

def mmd2_unbiased(K):
    pass  # TODO: implement

def mmd2_u_stat(K):
    assert K.ns[0] == K.ns[1]
    # TODO: implement

```

```python
plot_null_samples(
    *two_sample_permutation_test(lambda X, Y: mmd2_biased(ExpQuadKernel(X, Y)), X, Y),
    from_zero=True)
```

```python
plot_null_samples(
    *two_sample_permutation_test(lambda X, Y: mmd2_unbiased(ExpQuadKernel(X, Y)), X, Y))
```

```python
plot_null_samples(
    *two_sample_permutation_test(lambda X, Y: mmd2_u_stat(ExpQuadKernel(X, Y)), X, Y))
```

They're all pretty much the same; when you're just doing permutation testing, the differences between them aren't super important.

**Exercise:** Run the test with a `LinearKernel`. Which of our estimators, if any, corresponds to the `mean_difference` statistic from before? (Do you understand why?)

**Optional:** We're working on small datasets here so far, so this implementation of permutations is fine. But notice that we're recomputing the kernel matrix for each permutation, even though the matrices actually have all the same elements (just in jumbled-up order). Implement a faster way that doesn't involve this recomputation.

Hint: this is easiest for the biased estimator, which you can write in a way amenable to [this kind of approach](https://pytorch.org/docs/stable/torch.html#torch.einsum). (The `LazyKernel.joint()` method, which concatenates all the kernel matrices together, might be useful.) Doing some algebra to work out a few slightly annoying correction terms, you can also do it similarly for the U-statistic estimator.

If you're really, really careful, you can also make the unbiased estimator really fast (exploiting cache locality and things like that). We wrote about this in [this paper](https://arxiv.org/abs/1611.04488) (Section 3), and it's implemented in [Shogun](https://www.shogun-toolbox.org/). (Example usage [here](http://shogun.ml/examples/latest/examples/statistical_testing/quadratic_time_mmd.html) or in in [this notebook](https://nbviewer.jupyter.org/github/shogun-toolbox/shogun/blob/develop/doc/ipython-notebooks/statistical_testing/mmd_two_sample_testing.ipynb).)

_Extra credit:_ empirically compare how our silly implementation here, your faster implementation, and the Shogun implementation scale as the dataset size increases.

```python
# Override this function if you implement a better one yourself,
# and the stuff later will run faster....
# This just establishes an API for passing in a kernel directly,
# which is a little clumsy to implement, but it's not any faster.
def mmd2_permutations_slow(K, use_biased=False, num_permutations=1000, progress=True):
    # Some fiddling to be able to use the same kernel with "new" data.
    assert K.n_parts == 2
    from copy import copy
    K_copy = copy(K)
    def mmd2_with_K(X, Y):
        K_copy.change_part(0, X)
        K_copy.change_part(1, Y)
        return (mmd2_biased if use_biased else mmd2_unbiased)(K_copy)

    return two_sample_permutation_test(
        mmd2_with_K, K.X, K.Y,
        num_permutations=num_permutations, progress=progress)

mmd2_permutations = mmd2_permutations_slow
```

```python
K = ExpQuadKernel(X, Y, sigma=1)
plot_null_samples(*mmd2_permutations_slow(K))
```

### Alternative methods to estimate the null distribution

There are many other way to get our hands on the distribution of the MMD test statistic under the null hypothesis.

We know that asymptotically, the distribution is an infinite sum of Chi-square variables – but it's an infinite sum, and we don't even know what sum it is, so that doesn't help us too much.
Techniques to approximate it include:

* moment matching using a Gamma distribution: fast, but doesn't result in a consistent test.
* a spectral approximation: using eigenvalues of the kernel matrix, we can estimate that infinite sum. Costly (cubic!) for large sample sets.
* wild bootstrap: a technique used for correlated samples that is similar to permuting. (It's useful for e.g. testing an MCMC chain, though.)
* linear time statistics: for those, one can often show that the null distribution is Gaussian, with a variance you can estimate in closed form.

We won't cover any of these here, since they require some more details.
The permutation test, while it can potentially be slow, is easy to understand and works for most applications (if implemented well).

[This old Shogun notebook](https://nbviewer.jupyter.org/github/shogun-toolbox/shogun/blob/develop/doc/ipython-notebooks/statistical_testing/mmd_two_sample_testing.ipynb) contains many null approximation methods.

## MMD witness function in the RKHS

One nice feature of the MMD is that we can see where the density functions are different by looking at the (empirical) MMD witness function.
Remember that
\begin{align}
\MMD(\PP, \QQ)
  &= \sup_{f : \lVert f \rVert_\h \le 1} \E_{X \sim \PP} f(X) - \E_{Y \sim \QQ} f(Y)
\\&= \sup_{f : \lVert f \rVert_\h \le 1} \langle f, \E_{X \sim \PP}[\varphi(X)] - \E_{Y \sim \QQ}[\varphi(Y)] \rangle_\h
\end{align}
and so
$$
f^* \propto \E_{X \sim \PP}[\varphi(X)] - \E_{Y \sim \QQ}[\varphi(Y)]
,$$
which means (using $f^*(t) = \langle f^*, \varphi(t) \rangle_\h$)
$$
f^*(t) \propto \E_{X \sim \PP} k(X, t) - \E_{Y \sim \QQ} k(Y, t)
.$$

We can estimate $f^*$ with empirical averages.

**Thought exercises:**
- What's the proportionality constant hidden by $\propto$?
- Does the constant matter?
- Which MMD estimator, if any, does directly estimating $f^*$ correspond to?

The points where $\lvert f^*(t) \rvert$ are large are where the MMD test considers $\PP$ and $\QQ$ to be the "most different."
Let's define a grid and evaluate the witness function on it to see where that is.

**Exercise:** Finish the `mmd_witness` function to evaluate the witness function.

```python
def mmd_witness(K, eval_pts):
    assert K.n_parts == 2
    K.append_part(eval_pts[:, None])  # now K.XZ compares X to eval_pts

    witness = 0
    # TODO: estimate MMD witness function on grid_pts

    K.drop_last_part()
    return witness

def plot_mmd_witness_1d(K, ax=None, grid_num=1000):
    X, Y = [t.squeeze(1) for t in K.parts]
    assert len(X.shape) == 1

    if ax is None:
        ax = plt.gca()
    ax.hist(X.numpy(), alpha=.5, density=True, histtype='stepfilled', bins='auto')
    ax.hist(Y.numpy(), alpha=.5, density=True, histtype='stepfilled', bins='auto')
    ax.grid(False)
    ax.set_ylabel("Density")

    ax2 = ax.twinx()
    lo, hi = ax2.get_xlim()
    grid_pts = torch.linspace(lo, hi, grid_num)
    ax2.plot(grid_pts.numpy(), mmd_witness(K, grid_pts).numpy(), color='k', lw=2)
    ax2.set_xlim(lo, hi)
    ax2.grid(False)
    ax2.axhline(0, color='k', lw=0.5)
    ax2.set_ylabel("Witness value")

plot_mmd_witness_1d(ExpQuadKernel(X, Y))
```

As we can see, the witness function is positive where X as a higher density than Y, and negative vice versa.
It is zero where both densities match.
Intuitively, the RKHS norm of this function can only be zero if the densities match everywhere, and it grows as the densities differ on more and more points in their support.
Of course, this kind of visualization only works in low dimensions.

## Choosing a kernel

So far, we've been using `ExpQuadKernel` with its default `sigma=1`. That's a pretty arbitrary choice; what if we tried some different values?

**Thought exercise:** What do you expect the test results and witness functions to look like if we try larger or smaller sigmas (as the next block does)?

```python
for sig in [.1, .001, 10]:
    K = ExpQuadKernel(X, Y, sigma=sig)
    fig, (a1, a2) = plt.subplots(ncols=2, figsize=(12, 4))
    plot_null_samples(*mmd2_permutations(K, use_biased=False), ax=a1)
    plot_mmd_witness_1d(K, ax=a2)
    fig.suptitle(f"$\sigma$ = {sig}")
```

Make sure you intuitively understand what happened here (especially the witness plots).

Anyway, bandwidth 1 is a totally arbitrary choice that depends on the scaling of the data. We're going to talk about how to make actual, data-informed choices in a minute. But first, it's a good idea to start with a heuristic that usually works:

[](https://twitter.com/dril/status/484722159462260736)

In our case, the choice we can try evbery time is the "median heuristic": set $\sigma$ to the median of the distance between data points.

**Exercise:** compute the median distance between training data points. You can either compute this yourself, or use predefined functions, whichever you'd prefer. If it's too slow to compute, you can subsample the data points.

```python
def median_distance(Z):
    # TODO: return the median distance among the stacked samples Z
    # If you want to be fancy, add options to optionally subset if Z is big.

med = median_distance(torch.cat([X, Y], 0))
K = ExpQuadKernel(X, Y, sigma=med)

fig, (a1, a2) = plt.subplots(ncols=2, figsize=(12, 4))
plot_null_samples(*mmd2_permutations(K), ax=a1, one_sided=True)
plot_mmd_witness_1d(K, ax=a2)

med.item()
```

So the default 1 wasn't too far off, actually. The median heuristic worked great!

But it doesn't always.

### A dataset where the median heuristic doesn't work

```python
fig, (ax1, ax2) = plt.subplots(ncols=2, figsize=(12, 4))

with np.load(os.path.join(DATA_PATH, 'blobs.npz')) as d:
    X, Y = as_tensors(d['X'], d['Y'])
ax1.scatter(*X.t(), alpha=.5, label='X')
ax1.scatter(*Y.t(), alpha=.5, label='Y')
# ax1.legend()
ax1.set_title("Blobs")

with np.load(os.path.join(DATA_PATH, 'blobs_single.npz')) as d:
    ax2.scatter(*d['X'].T, alpha=0.5, label='X')
    ax2.scatter(*d['Y'].T, alpha=0.5, label='Y')
ax2.legend()
ax2.set_title("One blob (zoomed in)");
```

These two distributions are clearly very different. But the scale at which they're different is much smaller than the median distance, which looks mostly at global structure.

```python
med = median_distance(torch.cat([X, Y], 0))
K = ExpQuadKernel(X, Y, sigma=med)

plot_null_samples(*mmd2_permutations(K))
med.item()
```

## Learning kernels by maximizing (a proxy for) test power

So, we want some way to find the best test, where "best" means "most powerful." That is, we want some criterion where we can get a sense of how powerful a test is – preferably without having to run the test lots of times, since that'll take lots of data and computation.

There are a few possible schemes to do this. Here we're going to consider one that depends on a little math.

We need to maximize the test power – the probability of rejecting when $\PP \ne \QQ$:

$$
     \Pr\left( m \MMDhat_U^2(X, Y) > \hat c_\alpha \right)
$$

It turns out that under the alternative $\PP \ne \QQ$,
$\MMDhat_U^2$ is asymptotically normal,
$$
\frac{\MMDhat_U^2(X, Y) - \MMD^2(\PP, \QQ)}{\sqrt{\Var_{X' \sim \PP^m, Y' \sim \QQ^m}\left[ \MMDhat_U^2(X', Y') \right]}}
\stackrel{D}{\to}
\mathcal N(0, 1)
.$$
Call that thing in the denominator,
which is the variance of $\MMDhat_U^2$ given samples of size $m$ from $\PP$ and $\QQ$,
$V_m(\PP, \QQ)$.

Under the null hypothesis $\PP = \QQ$,
we have that $m \MMDhat_U^2(X, Y)$ converges to some gross distribution (an infinite mixture of chi-squares).
Let $c_\alpha$ be the $(1-\alpha)$th quantile of that distribution,
the rejection threshold;
we don't know that exactly,
but we can estimate it with permutation testing as $\hat c_\alpha$.
In this framework, note that $c_\alpha$ depends on $\PP$ and $\QQ$ (and the kernel in the MMD)
but it *doesn't* depend on $m$,
and in the usual setting we have
$\hat c_\alpha \to c_\alpha$.

So, for the test power we get

\begin{align}
     \Pr\left( m \MMDhat_U^2(X, Y) > \hat c_\alpha \right)
  &= \Pr\left(
       \frac{\MMDhat_U^2(X, Y) - \MMD^2(\PP, \QQ)}{\sqrt{V_m(\PP, \QQ)}}
     > \frac{\hat c_\alpha / m - \MMD^2(\PP, \QQ)}{\sqrt{V_m(\PP, \QQ)}}
   \right)
\\&\to 1 - \Phi\left(
        \frac{c_\alpha / m - \MMD^2(\PP, \QQ)}{\sqrt{V_m(\PP, \QQ)}}
    \right)
\\&= \Phi\left(
      \frac{\MMD^2(\PP, \QQ)}{\sqrt{V_m(\PP, \QQ)}}
    - \frac{c_\alpha}{m \sqrt{V_m(\PP, \QQ)}}
    \right)
\end{align}

We can then maximize the power of the test by maximizing the thing inside the parentheses.

Now, it turns out that for a fixed kernel, $V_m$ is $\mathcal{O}(1 / m)$, and $\MMD^2$ and $c_\alpha$ are constants. So the first term is $\mathcal{O}\left(\sqrt{m}\right)$ and the second is $\mathcal{O}\left(1 / \sqrt{m}\right)$ – which means we should be able to just ignore the second term when $m$ is reasonably large.

That means we need to choose $k$ to maximize
$$t := \frac{\MMD^2(\PP, \QQ)}{\sqrt{V_m(\PP, \QQ)}}.$$
This is a function of the kernel and $\PP$ and $\QQ$, *not* our particular sample from it, but we can estimate $t$ based on samples. We know how to estimate $\MMD^2$, and it turns out you can also estimate $V_m$ in the same quadratic time: see [this note](https://arxiv.org/abs/1906.02104). Then you can just divide the estimators.
That note is 12 pages of dense equations, so...I'll be here when you're done.

(Don't actually read it.)

**Bonus points:** Find a mistake in the note. I checked it three times, but I wouldn't be at all surprised if there are still mistakes!

**Big bonus points:** Figure out a way to simplify the derivation. Seriously, huge bonus points.

Anyway, even the _expression_ for the estimator is too disgusting to write here. But I implemented it for you, in `probability_testing.support.mmd2`. (*That's* what the `Matrix` subclasses are really for.)

```python
from probability_testing.support import mmd2_u_stat_variance
# mmd2_u_stat_variance(K) estimates V_m based on K.XX, K.XY, K.YY
```

To do permutation testing with kernel selection, we have two choices:

- Do permutations of the *whole* test procedure, including picking a kernel. This is valid, but usually super computationally expensive.

- Fit the kernel on a training set, and then run the final test on a held-out test set. (If we pick the kernel on the same set we test on, then we're likely to overfit: to pick a kernel that happens to work well on this particular dataset, but not the whole distribution.)

In real life, if `X` and `Y` were all we had, we'd have to split them up (and guess at how much to split them by...) into a "training" and a "testing" set.

**Quick thought exercise:** What are the tradeoffs when picking a bigger training set versus a bigger testing set? When would one be a better choice? Can you try multiple values and see?

Because I'm generous, I'm going to provide you with another set of samples to learn a kernel on. Enjoy!

```python
with np.load(os.path.join(DATA_PATH, 'blobs2.npz')) as d:
    X_train, Y_train = as_tensors(d['X'], d['Y'])
```

**Exercise:** Compare a fixed set of kernels, the different bandwidths on the blobs data, using this criterion.

This is an unbiased estimator, so it might be negative, or zero! Make sure you don't divide by zero, or take the square root of a negative number.

```python
def criterion_est(K, eps=1e-6):
    # TODO: return an estimate of the power criterion above
    # You can use eps to avoid dividing by zero, as you please.
    eps, = K.as_tensors(eps)

```

```python
# check the function runs
criterion_est(ExpQuadKernel(X_train, Y_train, sigma=1))
```

```python
# evaluate criterion for a bunch of sigmas
sigmas = np.logspace(-1.5, 2.5, num=50)
t_ests = [criterion_est(ExpQuadKernel(X_train, Y_train, sigma=s)) for s in tqdm(sigmas)]

plt.plot(sigmas, t_ests, marker='o')
plt.xscale('log')
plt.axvline(med, color='r')
```

**Optional:** We know how to estimate $c_\alpha$ – with permutation testing – so we could optimize the more-complete criterion, too. Does that give you different answers?

([`torch.kthvalue`](https://pytorch.org/docs/stable/torch.html#torch.kthvalue) might be useful. Also, remember that $c_\alpha$ is the permutation threshold for the distribution of $m \, \MMDhat^2$, not just plain $\MMDhat^2$.)

### Gradient descent to optimize the power criterion

What if we have more than one or two kernel parameters? What if we have dozens, or even (if we have really big datasets) thousands, or millions? Then trying out a grid of different combinations isn't going to work well. Instead, we can be good deep learners and do gradient descent.

(This is why we did everything in PyTorch!)

**Exercise:** Instead of doing a grid search, optimize $\sigma$ with gradient descent.

It's probably a better idea to parameterize it with $\log \sigma$ instead of $\sigma$ directly.

Does it always go to the same place, or can you get stuck in local minima?

```python
log_sigma = torch.tensor(np.log(1), requires_grad=True)
opt = torch.optim.SGD([log_sigma], lr=.1)  # could fiddle with optimizer if you want

trace = []
with tqdm(range(500)) as bar:  # <- feel free to increase the number of iterations
    for i in bar:
        opt.zero_grad()  # reset gradient state before computing things

        sigma = torch.exp(log_sigma)

        # TODO: compute into `loss` here, as the thing you want to *minimize*

        loss.backward()  # compute the gradients
        opt.step()       # update log_sigma along its gradient

        sig_val = sigma.cpu().item()
        loss_val = loss.cpu().item()

        trace.append((i, sig_val, loss_val))
        bar.set_postfix(sigma=f"{sig_val:.4}", loss=f"{loss_val:.4}")

fig, (a1, a2) = plt.subplots(ncols=2, figsize=(12, 4), constrained_layout=True)
inds, sigs, losses = np.asarray(trace).T

a1.plot(inds, losses)
a1.set_title("loss estimate")
a1.set_xlabel("iteration")

a2.plot(inds, sigs)
a2.set_title(r"$\sigma$")
a2.set_xlabel("iteration")
```

Let's check how our test works now

```python
K = ExpQuadKernel(X, Y, sigma=sigma.detach())

plot_null_samples(*mmd2_permutations(K, num_permutations=500))
```

# Independence testing

So far, we've been asking whether $\PP = \QQ$ for two different distributions. Sometimes we'd rather do a related problem, *independence testing*. That is, we have paired data $\{ (X_i, Y_i) \}_{i=1}^n \sim \PP_{XY}^n$, and we want to know whether $X$ and $Y$ are independent or not.

By definition, this is the same thing as asking whether $\PP_{XY} = \PP_X \times \PP_Y$, the product of the marginal distributions. But we know how to do that: if we use a characteristic kernel on the joint space, they're independent if and only if $\MMD(\PP_{XY}, \PP_{X} \times \PP_Y) = 0$.
The quantity $\MMD(\PP_{XY}, \PP_{X} \times \PP_Y)^2$  is also called the HSIC ("Hilbert-Schmidt Independence Criterion").

Usually we choose a kernel $k$ for $X$ and a kernel $\ell$ for $Y$, and use the _product kernel_ on the joint space:
$$
  (k \times \ell)( (x, y), (x', y') )
  = k(x, x') \ell(y, y')
.$$
It turns out you can estimate it like this:
$$
\frac{1}{n^2} \Tr\left( H K H L \right)
,$$
where $K$ is the kernel matrix on the $X$ samples,
$L$ is the kernel matrix on the $L$ samples,
and $H = I - \frac1n \mathbf{1} \mathbf{1}^T$ is the _centering matrix_.
(Here $I$ is the identity matrix, and $\mathbf 1$ is a vector of all $1$s.)

**Exercise:** Implement the HSIC estimator. You can do it literally as written here, with an H matrix and taking matrix products and so on. Or, if you want a more efficient implementation – $\mathcal{O}(n^2)$ instead of $\mathcal{O}(n^3)$ – you can notice a few things:

- If you expand out what $H K H$ does, it can be done in quadratic time without actually constructing $H$.
- $\Tr(A B) = \sum_i ( A B )_{ii} = \sum_i \sum_j A_{ij} B_{ji}$, which you can easily implement in quadratic time.

```python
def _as_kernel_matrices(K, L):
    if isinstance(K, LazyKernel):
        K = K.XX
    if isinstance(L, LazyKernel):
        L = L.XX
    K, L = as_tensors(K, L)
    assert len(K.shape) == len(L.shape) == 2
    assert K.shape[0] == K.shape[1] == L.shape[0] == L.shape[1]
    return K, L
```

```python
def hsic_est(K, L):
    K, L = _as_kernel_matrices(K, L)

    # TODO: implement the HSIC estimator

    return hsic
```

To run a test, we'll also need to do permutations. Note that permutations are slightly different from before: we'll need to permute $K$ and $L$ *separately*, so that $\{(x_i, y_i)\}$ becomes $\{(x_i, y_j)\}$, and any possible dependence between them is broken.

Here's some framework for computing permuations for the HSIC estimator. If you want to do it a faster way, go ahead.

```python
def hsic_permutations(K, L, num_permutations=1000, progress=True):
    K, L = _as_kernel_matrices(K, L)

    est = hsic_est(K, L)

    range_ = range(num_permutations)
    if progress:
        range_ = tqdm(range_)

    stats = []
    for i in range_:
        pass # TODO: permute the entries of K and L and call hsic_est

    return est, torch.stack(stats)
```

Let's try it on some simple data.

```python
with np.load(os.path.join(DATA_PATH, 'hsic.npz')) as d:
    X, Y = as_tensors(d['X'], d['Y'])
```

```python
fig, ax = plt.subplots()
ax.scatter(X, Y)
```

Clearly, there is some very simple dependence going on.
The first thing a statistician would do is of course to check for correlation.

```python
print("Correlation between X and Y:", np.corrcoef(X.squeeze(), Y.squeeze())[0, 1])
```

The correlation is very low.
A kernel test for independence is more powerful, even with a bandwidth parameter chosen by the median heuristic.

**Exercise:** Implement and visualize this test.

Pretty resounding rejection of the null hypothesis.

**Optional**: Do we already have the code for doing an actual hypothesis test based on the correlation coefficient? Implement it and run the test to verify that it is not very powerful here.

## European parliament documents translations and string kernels

We will now do a slightly more elaborate test, that involves some very mild NLP:
we'll analyze dependence between documents.
We will use transcripts of the Canadian parliament's house debates, downloaded from [here](https://www.isi.edu/natural-language/download/hansard/).
Those consist of pairs of French and English transcripts. Here's the end of two corresponding documents:

> d until tomorrow at 2 p.m., pursuant to Standing Order 24(1).
> (The House adjourned at 6.41 p.m.)

> main, à 10 heures, conformément à l'article 24(1) du Règlement.
> (La séance est levée à 18 h 20.)

Our question here is whether we can detect this supposedly strong dependence structure using the kernel HSIC.
Note that this approach does not rely on attempting to translate the documents, but rather on comparing within-document structure.
HSIC compares the self-similarity within the English documents with self-similarity of the French ones.

We're going to construct a string kernel, a  "bag of words" kernel between documents $s$ and $t$,
$$
k(s,t) = \phi(s)^\top \phi(t)
$$
where $\mathcal W$ is the set of all words that we consider,
and each element of $\phi(x)\in\mathbb{N}^{|\mathcal{W}|}$ contains the number of times that a particular word $w\in\mathcal{W}$ appears in $x$.
The kernel value will be larger if a word appears in both documents many times.
We're going to keep things simple and actually construct the representation $\phi$ for each document.

There are also better choices for string kernels, that actually take into account the structure of a word.
(Shogun has efficient implementations of many classical ones, usually based on low-level dynamic programming concepts;
you can also define them based on modern word embeddings.)
But simple word counts will be enough for us here.

```python
# stopwords, from nltk.corpus.stopwords:
# words like "the", "our" (in English) or "le", "avec" (in French)
# that we don't want overwhelming the meaningful words, so we just skip
with open(os.path.join(DATA_PATH, 'stopwords-english.txt')) as f:
    en_stop = {x.strip() for x in f}
with open(os.path.join(DATA_PATH, 'stopwords-french.txt')) as f:
    fr_stop = {x.strip() for x in f}
```

To compute $\phi$, we would normally need to do a first pass to explicitly keep track of all the words $\mathcal W$. This can be kind of slow, though, so instead we're going to use the "hashing trick." Rather than keeping track of a vocabulary, we use a hash function ([MurmurHash](https://en.wikipedia.org/wiki/MurmurHash)) to map each word to a (probably) unique index. To avoid hash collisions, we make $\phi$ very high-dimensional ($2^{20} \approx 1,000,000$), but use a sparse representation so that doesn't take much memory.

```python
import re
import tarfile
from scipy import sparse

en_feats = {}
fr_feats = {}

from sklearn.feature_extraction.text import HashingVectorizer
en_vec = HashingVectorizer(
    analyzer="word", decode_error='replace', stop_words=en_stop,
)
fr_vec = HashingVectorizer(
    analyzer="word", decode_error='replace', stop_words=fr_stop,
)

with tarfile.open('data/transcripts.tar.bz2') as tar:
    for info in tqdm(tar, total=627):
        fname = info.name
        is_french = fname.endswith("f")
        is_english = fname.endswith("e")
        if not (is_english or is_french):
            continue

        with tar.extractfile(info) as f:
            if is_english:
                en_feats[fname] = en_vec.transform([f.read()])
            else:
                fr_feats[fname] = fr_vec.transform([f.read()])

assert len(en_feats) == len(fr_feats)
names = {k[:-1] for k in en_feats}
X = sparse.vstack([en_feats[k + 'e'] for k in names])
Y = sparse.vstack([fr_feats[k + 'f'] for k in names])
```

Our features are enormous (and enormously sparse) matrices.

```python
X
```

**Exercise:** Construct linear kernel matrices for the French and English data.

Now, our PyTorch kernel infrastructure isn't going to love trying to construct a matrix with a million columns. We're only going to use a linear kernel, though, so we can just compute the kernel ourselves. (You can convert from these sparse matrix classes to numpy arrays with `toarray()`, but do it *after* computing the kernel matrix to avoid trying to allocate an enormous dense array:

```python
# TODO: compute kernels
K_XX = (
    # ...
).toarray().astype(np.float32)
K_YY = (
    # ...
).toarray().astype(np.float32)
```

**Exercise:** Run the HSIC test.

We can clearly see that the English and French documents are not independent.

Machine Learning Summer School 2019 (Moscow, Russia)/tutorials/kernels/testing.ipynb


```python
%matplotlib inline
```

# CSA Histogram Approach

Static Chemical Shift Powder Pattern using a Histogram Approach

The equation for the static powder pattern for a simple chemical shift anisotropy interation is given by the following equation

\begin{align}H = \delta_{iso} + \frac {1}{2} \delta \left ( 3 \cos^2 \theta - 1 \right ) - \delta \eta \sin^2 \theta \cos 2 \phi\end{align}

There are a number of conventions for the assignment of $\eta$ and $\delta$, we have used Haeberlen's convention.

if $\sigma_{xx}$, $\sigma_{yy}$ and $\sigma_{zz}$ are the principal components of the chemical shielding tensor then they must have the following order.

\begin{align}\left | \sigma_{zz} - \sigma_{iso} \right | \ge  \left | \sigma_{xx} - \sigma_{iso} \right | \ge  \left | \sigma_{yy} - \sigma_{iso} \right |\end{align}
where

\begin{align}\sigma_{iso} = \frac {1}{3} \left ( \sigma_{xx} + \sigma_{yy} + \sigma_{zz} \right )\end{align}
and then $\delta$ and $\eta$ ared defined as follows

\begin{align}\delta = \sigma_{zz} - \sigma_{iso}\end{align}
and

\begin{align}\eta = \frac {\sigma_{xx}-\sigma_{yy}}{\sigma_{zz}-\sigma_{iso}}\end{align}

References
~~~~~~~~~~

- U. Haeberlen, In Advances in Magnetic Resonance; Suppl. 1; J. S. Waugh, Ed.; Academic Press, New York, 1976.

```python
import numpy as np
from matplotlib import pyplot as plt
import sys

def return_Hammersley_points_array( l, n, p):
    """
    l is the power of x to go out to p^m
    n is the maximun number of points
    p is the order of the Hammersley point, 1,2,3,4,... etc

    returns
    --------
    np.array of double

    """

    vvv = np.zeros(n)

    for m in range(n):
        m1=1*m
        if p == 1:
            vvv[m] =  m1/n
        else:
            v = 0.0

            for j in range(l,-1,-1):
                num = m1//p**j

                if num > 0:
                    m1 -= num*p**j
                    v  += num / ( p ** (j+1) )

            vvv[m]=v

    return(vvv)

def omega_cs( theta, phi, iso_cs=0.0, asymm_cs=100, eta_cs=1.0):

    return (iso_cs +0.5* asymm_cs*(3.0 * (np.cos(theta)**2) -1.0 - eta_cs*(np.sin(theta)**2)*np.cos( 2.0 * phi ))), np.sin(theta)

if __name__ == "__main__":

    # Define CSA powder pattern

    # Principal components of the chemical shift shielding tensor

    s_zz = -120.0
    s_yy = -50.0
    s_xx =  100.0

    # Check for Haeberlens convention

    iso_cs =(s_zz+s_yy+s_xx)/3.

    if abs(s_zz-iso_cs) >= abs(s_xx-iso_cs) and abs(s_xx-iso_cs) >= abs(s_yy-iso_cs):
        h_zz = s_zz
        h_yy = s_yy
        h_xx = s_xx

    elif abs(s_zz-iso_cs) < abs(s_xx-iso_cs) and abs(s_xx-iso_cs) >= abs(s_yy-iso_cs):
        h_zz = s_xx
        h_yy = s_yy
        h_xx = s_zz

    else:
        print("problem with assignment of cs tensors")
        sys.exit()

    asymm_cs = h_zz-iso_cs
    eta_cs   = (h_xx-h_yy)/(h_zz-iso_cs)

    # Calculate Hammersley Points and Powder pattern
    N_particles = 2**17

    theta = return_Hammersley_points_array(22, N_particles, 2)
    phi   = return_Hammersley_points_array(22, N_particles, 3)

    omega, solid_angle = omega_cs(theta*np.pi,2*np.pi*phi, eta_cs=eta_cs, iso_cs=iso_cs, asymm_cs=asymm_cs)

    # Plot Powder pattern and use sin(theta) solid angle weighting

    plt.hist(omega, bins = 200, weights=solid_angle, density=True);
    plt.xlim(250.0, -250.0)
    ax = plt.gca()
    ax.spines['top'].set_visible(False)
    ax.spines['left'].set_visible(False)
    ax.spines['right'].set_visible(False)
    plt.yticks([])
    plt.xlabel('Hz', fontsize=14)

    ax.annotate('$\sigma_{xx}$',
            xy=(s_xx+5, 0.0030), xycoords='data',
            xytext=(-50, 30), textcoords='offset points',
            arrowprops=dict(arrowstyle="->"),fontsize=14)

    ax.annotate('$\sigma_{yy}$',
            xy=(s_yy+5, 0.012), xycoords='data',
            xytext=(-50, 00), textcoords='offset points',
            arrowprops=dict(arrowstyle="->"),fontsize=14)

    ax.annotate('$\sigma_{zz}$',
            xy=(s_zz-5, 0.0044), xycoords='data',
            xytext=(50, 30), textcoords='offset points',
            arrowprops=dict(arrowstyle="->"),fontsize=14)

    plt.title(f"{N_particles} Hammersley Pts CSA Calculated Histogram", fontsize=14);

    plt.show()
```


# Time Series Analysis
In this lab you will run a simple time series analysis for the concentration of CO2 in Earth's atmosphere recorded at the [Mauna Loa Observatory](https://www.esrl.noaa.gov/gmd/obop/mlo/) on Hawaii (run by the National Oceanic and Atmospheric Administration in the USA). The analysis has been written for you in the Python computer programming language, but **you don't need any prior knowledge of computer programming**. The aims of this lab are:
* Understand what a time series is
* Familiarise yourself with important concepts in time series analysis
* Examine the changing CO2 concentrations in the Earth's atmosphere through time
* Get an introduction to the use of progamming languages such as Python for performing data analysis

### Python
The programming language we are using in this lab is called Python.  No prior knowledge of programming is required for this lab. **Learning how to be a programmer is not the aim!** However, this sort of scientific computing is becoming more common place in research and consultancy, so it won't do you any harm to see it in action. Python is a multifunctional computer language, for example it can interface with ArcGIS and QGIS (the software used in the previous lab) to automate workflows.

The code for this lab has all been written for you in this notebook, you just need to execute it.
**To run a code block, click in a cell, hold down shift, and press enter.**
An asterisk in square brackets `In [*]:` will appear while the code is being executed, and this will change to a number `In [1]:` when the code is finished. *The order in which you execute the code blocks matters, they must be run in sequence.*

Inside blocks of python code there are comments indicated by lines that start with `#`. These lines are not computer code but rather comments providing information about what the code is doing to help you follow along. Before we get started we need to tell python which tools we want to use (these are called modules):

```python
# import modules for numerical and statistical calculations and for plotting data
import urllib
import numpy as np
import scipy.stats as stat
import matplotlib.pyplot as plt

# tell python to allow plotting to occur within the page
%matplotlib inline
```

### Access timeseries data
The time series data of mean monthly CO2 concentrations recorded at Mauna Loa can be access [here](https://www.esrl.noaa.gov/gmd/ccgg/trends/data.html). However, by providing a direct link to the data stored online, we can just open it directly in python:

**N.B. The direct link to the Mauna Loa observatory seems to be blocked on student accounts and therefore the data file has been provided on Moodle. PLEASE DOWNLOAD THE DATA FROM MOODLE**

```python
# download the Mauna Loa CO2 data, we are only going to use two columns,
# the decimal date and the monthly average CO2 concentrations,
# so we unpack the results into two variables, date and Co2 concentration

# load the data from the file
Filename = "co2_mm_mlo.txt"
DecimalDate, CO2 = np.loadtxt(Filename, usecols=(2,3), unpack=True)

# let's print the first five values of each to get an idea what the data looks like
print(DecimalDate[0:5])
print(CO2[0:5])
```

### Plot time series of mean monthly CO2

First thing we might like to do is just to plot the data and see what it looks like. This is likely a plot you have seen before in climate change lectures and in media articles about climate change. This plot is known as the "Keeling Curve" after geochemist Charles David Keeling, who designed and built the Mauna Loa CO2 monitoring station in 1957 ([Harris, 2010](https://pubs.acs.org/doi/abs/10.1021/ac1001492)).

Note that we need to consider the units we are working in. Time is in years and CO2 concentration is in parts per million (ppm).

So let's make the plot by creating a figure, plotting the data, adding a legend and labelling our axes:

```python
# plot the time series
FigWidth = 10.
fig1 = plt.figure(1,figsize=(FigWidth,FigWidth*9./16.))
ax1 = fig1.add_subplot(111)
ax1.plot(DecimalDate,CO2,'k-',label="Monthly Mean CO$_2$ Data")
plt.xlabel("Time (year)")
plt.ylabel("CO$_2$ (ppm)")
# add the legend
ax1.legend()
plt.savefig("MaunaLoa_CO2_TimeSeries.png",dpi=600)
```

Note that the last line saves the figure as an image file on your computer so you could put it in a report or essay.

## Decomposition

This time series can be thought of as being made up of a number of components, each of which can be analysed in turn to get a full statistical description of how the variable is changing through time. There is an overall **trend** whereby CO2 concentrations in the atmosphere are increasing through time. There is then some regularity to the fluctuations about this trend, some **seasonality** in the data. These components can each be described individually using mathematical descriptions, and the resulting mathematical model can be used for prediction.

First let's look at the components in a bit more detail by zooming in on a part of the plot, changing the axis limits:

```python
# change the x and y axis scale to show seasonality
ax1.set_xlim(2000,2010)
ax1.set_ylim(360,400)
plt.savefig("MaunaLoa_CO2_TimeSeries_Seasonal.png",dpi=600)
fig1
```

```python
# reset axis scales back to normal
ax1.autoscale()
fig1
```

### Trend

There is clearly an upward trend in the CO2 concentrations in the atmosphere through time. The simplest way to analyse this trend would be through performing linear regression analysis between time and CO2 concentrations, just like you have done using Minitab in previous labs. This can tell us mathetmatically the nature of the trend, plus give us an idea of how much of the variance in CO2 concentrations can be explained by this trend. Thus we can start with a straight line equation:

$$
\begin{equation}
{Y_t = C_1 + C_2 X_t}
\end{equation}
$$

In this equation $X_t$ is time, and $Y_t$ is CO2 concentration (the subscript $_t$ has been used to indicate we are considering the **trend** component), $C_1$ and $C_2$ are fitted coefficients corresponding to the intercept and slope of a fitted line respectively.

The statistics packages within python have a linear regression tool we can use, this works in exactly the same way as the linear regression tool you have been using in Minitab. So let's run it and print the results to screen:

```python
# perform linear regression
slope, intercept, r_value, p_value, std_err = stat.linregress(DecimalDate,CO2)

# set p if very low
if (p_value<0.001):
    p_value = 0.001

print("C1 = ", intercept)
print("C2 = ", slope)
print("R^2 = ", r_value**2.)
print("p = ", p_value)
```

Now we can plot the results of our regression, label the regression line and add our equation and R$^2$ value:

```python
# make a string of the equation so that we can add
eq = "$Y_t=$"+'{0:.2f}'.format(intercept)+" + "+'{0:.2f}'.format(slope)+"$ X_t$"

# plot the result
ax1.plot(DecimalDate, intercept+slope*DecimalDate,'r--',label=eq)

# add R2 and p value
ax1.text(2000,310,"R$^2$ = "+'{0:.2f}'.format(r_value**2.)+", $p <$  "+'{0:.3f}'.format(p_value),color="r")

# add the legend
ax1.legend()

# save output
fig1.savefig("MaunaLoa_CO2_TimeSeries_LinearTrend.png",dpi=600)

#display the updated figure
fig1
```

Wow, this linear regression can explain 98% of the variance in CO2 concentrations through time. Not bad! But as with all regression applications we need to take a look at our residuals to see whether our model is consistent.

We can take a look at the residuals between the model fit and the raw data to have an idea of how well our regression model is performing. There are three plots we could work with, a histogram of the residuals, a normal probability plot, and a plot of the residuals vs the modelled values:

```python
# calculate the residuals
linear_model = intercept+slope*DecimalDate
residuals = CO2-linear_model

# plot the residuals on a new figure
fig2 = plt.figure(2,figsize=(4,4))
ax2 = fig2.add_subplot(111)
ax2.hist(residuals)
plt.xlabel("Residuals in CO$_2$ (ppm)")
plt.ylabel("Frequency")
plt.xlim(-10,10)
plt.tight_layout()
fig2.savefig("MaunaLoa_CO2_TimeSeries_LinearTrend_Residuals_Hist.png", dpi=600)
```

```python
# create another plot for normal probability plot
fig3 = plt.figure(2,figsize=(4,4))
ax3 = fig3.add_subplot(111)
stat.probplot(residuals,plot=ax3)

# customise the colours
ax3.get_lines()[0].set_marker('o')
ax3.get_lines()[0].set_markerfacecolor([0.1,0.3,0.65])
ax3.get_lines()[0].set_markeredgecolor([0.1,0.3,0.65])
plt.tight_layout()
fig3.savefig("MaunaLoa_CO2_TimeSeries_LinearTrend_ProbPlot.png", dpi=600)
```

```python
# create a third plot for model vs residuals
fig4 = plt.figure(2,figsize=(6,4))
ax4 = fig4.add_subplot(111)
ax4.plot(linear_model,residuals,'ko')
ax4.plot([linear_model.min(),linear_model.max()],[0,0],'r--')
plt.xlabel("Model fit CO$_2$ (ppm)")
plt.ylabel("Residuals in CO$_2$ (ppm)")
plt.tight_layout()
fig4.savefig("MaunaLoa_CO2_TimeSeries_LinearTrend_Residuals.png", dpi=600)
```

So our residuals are not normally distributed and there is *clearly* heteroscedasticity. Measured CO2 concentrations are higher than our best fit line towards the beginning and end of the record, and lower in the middle. Instead of transforming any variables (which I can tell you will not work in this case), we can try fitting a non-linear (curvy) model, or more specifically a **quadratic** model with an extra coefficient ($C_3$) of the form:

$$
\begin{equation}
{Y_t = C_1 + C_2 X_t + C_3 {X_t}^2}
\end{equation}
$$

To do this we will use a polynomial regression function, rather than linear regression function. Nmuerical python has a *"polyfit"* tool do this, and we have to tell it that the order of the polynomial is 2 (i.e. X$^2$ is the largest power in the equation):

```python
# try a 2nd order polynomial fit instead
coefs = np.polyfit(DecimalDate,CO2,2)
fit = np.poly1d(coefs)
print("C1 = ",coefs[2])
print("C2 = ",coefs[1])
print("C3 = ",coefs[0])
```

Now annoyingly, this function doesnt explicitly calculate R$^2$ for us. We need to do it ourselves.
So first let's subtract the model from data to get our residuals:

```python
# calculate the residuals
residuals = CO2-fit(DecimalDate)
```

The coefficient of determination (R$^2$) is calculated from the total sum of squared differences ($SS_{tot}$) between the observations and the mean. This sum of squares is proporational to the total variance. We also need the sum of squares of our residuals relative to our best fit model ($SS_{res}$):

$$
\begin{equation}
R^2 = 1 - {{SS_{res}}\over{SS_{tot}}}
\end{equation}
$$

```python
# calculate total sum of squares relative to to the mean
SS_tot = np.sum((CO2-np.mean(CO2))**2.)

# calculate total sum of squared residuals
SS_res = np.sum(residuals**2.)

# calculate R2 and print to screen
R_2 = 1-(SS_res/SS_tot)
print("R^2 = ", R_2)
```

Now we can plot our new model fit on our existing graph and report the equation and $R^2$:

```python
# make a string of the equation
eq2 = "$Y_t=$"+'{0:.3f}'.format(coefs[0])+"${X_t}^2$ - "+'{0:.2f}'.format(-coefs[1])+"$ X_t$ + "+'{0:.1f}'.format(coefs[2])

# plot the result
ax1.plot(DecimalDate, fit(DecimalDate),'b-',label=eq2)

# do chi_dquared test to check significance of relationship
chi_sq, p_value = stat.chisquare(CO2,fit(DecimalDate))
p_value = 1-p_value

# set p if very low
if (p_value<0.001):
    p_value = 0.001

ax1.text(2000,320,"$R^2$ = "+'{0:.2f}'.format(R_2)+", $p <$ "+'{0:.3f}'.format(p_value), color="b")

# add the legend
ax1.legend()

# save figure
fig1.savefig("MaunaLoa_CO2_TimeSeries_QuadraticTrend.png", dpi=600)

fig1
```

This looks to be doing a much better job of representing the trend in CO2 concentrations. Let's check the residuals:

```python
# plot the residuals
fig5 = plt.figure(5,figsize=(4,4))
ax5 = fig5.add_subplot(111)
ax5.hist(residuals)
plt.xlabel("Residuals in CO$_2$ (ppm)")
plt.ylabel("Frequency")
plt.xlim(-10,10)
plt.tight_layout()
fig5.savefig("MaunaLoa_CO2_TimeSeries_QuadTrend_Residuals.png", dpi=600)
```

```python
# create a second plot axis for normal probability plot
fig6 = plt.figure(6,figsize=(4,4))
ax6 = fig6.add_subplot(111)
stat.probplot(residuals,plot=ax6)

# customise the colours
ax6.get_lines()[0].set_marker('o')
ax6.get_lines()[0].set_markerfacecolor([0.1,0.3,0.65])
ax6.get_lines()[0].set_markeredgecolor([0.1,0.3,0.65])
plt.tight_layout()
fig6.savefig("MaunaLoa_CO2_TimeSeries_QuadTrend_ProbPlot.png", dpi=600)
```

```python
# create a third plot for model vs residuals
Model = fit(DecimalDate)
fig7 = plt.figure(7,figsize=(6,4))
ax7 = fig7.add_subplot(111)
ax7.plot(Model,residuals,'ko')
ax7.plot([Model.min(),Model.max()],[0,0],'r--')
plt.xlabel("Model fit CO$_2$ (ppm)")
plt.ylabel("Residuals in CO$_2$ (ppm)")
plt.tight_layout()
fig7.savefig("MaunaLoa_CO2_TimeSeries_QuadTrend_Resis.png", dpi=600)
```

OK that looks like an improvement. We could now go on and try higher order polynomial fits or try and fit other types of functions such as power laws or exponential functions, particularly if the quadratic was not able to offer a reasonable fit ot the data.

## Timeseries Filters

An alternative approach to identifying the trend component in a timeseries is to run a filter across the data. A filter acts to smooth out regular and irregular noise and isolate the trend. The advantage of a filter is that it makes no *a priori* assumptions about the nature of the trend (e.g. linear, quadratic, power law). The disadvantage is that the trend is not described mathematically and thus cannot be extrapolated to make future predictions.

The simplest form of filter is a running average (running mean). This means for each data point in our timeseries we take an average $\bar{Y_t}$ of all the nearby values $Y_t$, so the overbar indicates an average value:

$$
\begin{equation}
\bar{Y_t} = {{1}\over{m}}\sum_{j=-k}^{k} Y_t+j
\end{equation}
$$

Where $m$ is the size of the moving window (how many values to calculate the mean from), and $k$ is a index referencing the nearest values to $Y_t$ within the moving window. Below we define a function that calculates the running mean, and use a 24 month (2 year) window size to smooth our data:

```python
# filter
def running_mean(x,N):
    cumsum = np.cumsum(np.insert(x, 0, 0))
    return (cumsum[N:] - cumsum[:-N]) / float(N)

# number of timesteps to average over = 24 months/ 2 years
Nt = 12
CO2_filter = running_mean(CO2,Nt)

# set up filtered dates and raw CO2 data
# we need to trim the ends of the filter since we have no computations until we are half a window in to the dataset
DecimalDate_f = DecimalDate[int(Nt/2):-int((Nt/2)-1)]
CO2_f = CO2[int(Nt/2):-int((Nt/2)-1)]

FigWidth = 10.
fig8 = plt.figure(8,figsize=(FigWidth,FigWidth*9./16.))
ax8 = fig8.add_subplot(111)
ax8.plot(DecimalDate,CO2,'k-',label="Mean Monthly CO$_2$")
ax8.plot(DecimalDate_f,CO2_filter,'r--',lw=2,label="Moving Average")
plt.xlabel("Time (year)")
plt.ylabel("CO$_2$ (ppm)")
ax8.set_xlim(2000,2010)
ax8.legend()
fig8.savefig("MaunaLoa_CO2_TimeSeries_MovingAverage.png", dpi=600)
```

TASK

 Explore the effect of window size on the degree of filtering by changing the size of the moving average window at the line  `Nt = 24` in the code above. Then rerun the last block of code. You will need to use an even number. I'd suggest trying 6 months, and 96 months.

If you want to zoom in to be able to look a little closer at the effects, comment/uncomment the line `#ax8.set_xlim(2000,2010)` by adding/deleting the `#` symbol and rerun the code block.

## Seasonality

Now that the trend has been identified and modelled, you can explore the seasonal signal in the residuals. First let's plot the residuals through time to observe the seasonal signal, corrected for our filtered running mean.

```python
# Get seasonal component
SeasonalCO2 = CO2_f-CO2_filter

# plot the time series
FigWidth = 12.
fig9 = plt.figure(9,figsize=(FigWidth,FigWidth*1./4.))
ax9 = fig9.add_subplot(111)
ax9.plot(DecimalDate_f,SeasonalCO2,'k-',label="Monthly Mean CO$_2$ Data")
plt.xlabel("Time (year)")
plt.ylabel("Seasonal CO$_2$ (ppm)")
plt.tight_layout()
plt.savefig("MaunaLoa_CO2_TimeSeries_Seasonal.png",dpi=600)
```

We can zoom in a little bit to see the seasonal pattern more clearly.

```python
ax9.set_xlim(2000,2010)
fig9.savefig("MaunaLoa_CO2_TimeSeries_Seasonal10.png",dpi=600)
fig9
```

To model the seasonal signal we will use a harmonic function (e.g. a sine wave). There are other ways we could approach this but this is a simple btu effective approach. The mathematical equation for a sine wave is:

$$
\begin{equation}
{S_t = A\:\sin\:(\:f\:X_t + \phi) + D}
\end{equation}
$$

where $A$ is the amplitude of the wave, $f$ is the wave frequency, $\phi$ is the phase (where in the cycle of one wavelength the signal starts), and $D$ is the offset.

We can fit a seasonal signal to the decomposed data by fitting a sine wave of this form to the data using a least squares optimisation function (i.e. minimising the square of our residuals).

```python
#import optimisation routine for fitting sine curve to the data
from scipy.optimize import leastsq, minimize

# create a sine wave function for optimising to our seasonal data
# minimising the difference to the data
optimize_func = lambda x: x[0]*np.sin(x[1]*DecimalDate_f+x[2]) + x[3] - SeasonalCO2

# take an initial guess at the parameters
guess_mean = np.mean(SeasonalCO2)
guess_phase = 1.6
guess_freq = 6.3
guess_amp = 3.

# run the least squared analysis to optimise
est_amp, est_freq, est_phase, est_mean = leastsq(optimize_func, [guess_amp, guess_freq, guess_phase, guess_mean])[0]

# recreate the fitted curve using the optimized parameters
sine_fit = est_amp*np.sin(est_freq*DecimalDate_f+est_phase) + est_mean

ax9.plot(DecimalDate_f,sine_fit,'r--')
fig9.savefig("MaunaLoa_CO2_TimeSeries_Fit.png",dpi=600)
fig9
```

### Random component / residuals

The remaining difference (residuals) between the decomposed seasonal data and the sine wave model fit through it is the last component of our timeseries. This may still contain some signal which could be further decomposed if desired.

```python
# get residuals from seasonal signal
Residuals = SeasonalCO2 - sine_fit

# plot residuals
fig10 = plt.figure(9,figsize=(FigWidth,FigWidth*1./4.))
ax10 = fig10.add_subplot(111)
ax10.plot(DecimalDate_f,Residuals,'k-',label="Monthly Mean CO$_2$ Data")
plt.xlabel("Time (year)")
plt.ylabel("Residual CO$_2$ (ppm)")
ax10.set_xlim(2000,2010)
plt.tight_layout()
plt.savefig("MaunaLoa_CO2_TimeSeries_Residuals.png",dpi=600)
```

```python
# plot the residuals
fig11 = plt.figure(10,figsize=(4,4))
ax11 = fig11.add_subplot(111)
ax11.hist(Residuals)
plt.xlabel("Residuals in CO$_2$ (ppm)")
plt.ylabel("Frequency")
#plt.xlim(-10,10)
plt.tight_layout()
fig11.savefig("MaunaLoa_CO2_TimeSeries_Final_Residuals_Hist.png", dpi=600)
```

```python
# create a second plot axis for normal probability plot
fig12 = plt.figure(1,figsize=(4,4))
ax12 = fig12.add_subplot(111)
stat.probplot(Residuals,plot=ax12)

# customise the colours
ax12.get_lines()[0].set_marker('o')
ax12.get_lines()[0].set_markerfacecolor([0.1,0.3,0.65])
ax12.get_lines()[0].set_markeredgecolor([0.1,0.3,0.65])
plt.tight_layout()
fig12.savefig("MaunaLoa_CO2_TimeSeries_Final_ProbPlot.png", dpi=600)
```

```python
# create a third plot for model vs residuals
Model = fit(DecimalDate_f)
fig12 = plt.figure(12,figsize=(6,4))
ax12 = fig12.add_subplot(111)
ax12.plot(Model,Residuals,'ko')
ax12.plot([Model.min(),Model.max()],[0,0],'r--')
plt.xlabel("Model fit Seasonal CO$_2$ (ppm)")
plt.ylabel("Residuals in CO$_2$ (ppm)")
plt.tight_layout()
fig12.savefig("MaunaLoa_CO2_TimeSeries_Final_Resis.png", dpi=600)
```

### Final decomposition

So now we have our final decomposition, let's plot the results all together.

```python
# Create decomposition plot
FigWidth=10
fig10 = plt.figure(10,figsize=(FigWidth,FigWidth))

# plot data
ax10a = fig10.add_subplot(411)
ax10a.plot(DecimalDate,CO2,'k-')
ax10a.set_ylabel("CO$_2$ (ppm)")
ax10a.xaxis.set_ticklabels([])
plt.text(1960,400,"Raw Data")

#plot trend
ax10b = fig10.add_subplot(412)
ax10b.plot(DecimalDate,fit(DecimalDate),'k-')
ax10b.set_ylabel("CO$_2$ (ppm)")
ax10b.xaxis.set_ticklabels([])
plt.text(1960,400,"Trend")

#plot seasonal
ax10c = fig10.add_subplot(413)
ax10c.plot(DecimalDate_f,sine_fit,'k-')
ax10c.set_ylabel("CO$_2$ (ppm)")
ax10c.xaxis.set_ticklabels([])
ax10c.set_ylim(-5,5)
plt.text(1960,3.8,"Seasonal")

#plot noise
ax10d = fig10.add_subplot(414)
ax10d.plot(DecimalDate_f,Residuals,'k-')
ax10d.set_ylabel("CO$_2$ (ppm)")
plt.text(1960,1.3,"Residuals")
plt.xlabel("Date (years)")

plt.tight_layout()
plt.savefig("MaunaLoa_CO2_TimeSeries_Decomposition.png",dpi=600)
```

```python
# recreate the fitted curve using the optimized parameters
X = np.arange(0,2*np.pi,0.01)
sine_wave = 1.*np.sin(4.*X+np.pi/2) + 0.5
plt.figure(11,figsize=(12,4))
plt.plot(X,sine_wave)
plt.xlabel("X")
plt.ylabel("Y")
plt.tight_layout()
plt.savefig("sinewave.png",dpi=600)
```

```python
Time = np.arange(1957,2100,0.1)

Prediction = coefs[2]+ coefs[1]*Time+coefs[0]*Time**2. + est_amp*np.sin(est_freq*Time+est_phase) + est_mean
```

```python
FigWidth = 10.
fig99 = plt.figure(99,figsize=(FigWidth,FigWidth*9./16.))
ax99 = fig99.add_subplot(111)
ax99.plot(DecimalDate,CO2,'k-',label="Mean Monthly CO$_2$")
ax99.plot(Time,Prediction,'r-',label="Predicted CO$_2$")
plt.xlabel("Time")
plt.ylabel("CO2")
plt.tight_layout()
plt.savefig("prediction.png",dpi=600)
```


# Weighing Baby Solar Systems with the Very Large Array

Leader: Patrick Sheehan

### Background

Stars form from clouds of gas and dust that exist within our galaxy. This is a picture of the Orion Nebula, one of the biggest nearby star forming regions:

If you zoom way in on a single forming star in this region, you see a dark, dusty disk silhoutted agains the bright background of the Orion Nebula:

It's in these "protoplanetary disks" that planets form, so how much material is in them, i.e. their "mass", tells you what sort of planetary system they are capable of forming. It turns out that radio telescopes, like the VLA:

take pictures that are sensitive to the amount of material that is present in these disks. I have collected VLA observations of 27 "protoplanetary disks" in the Taurus Molecular Cloud Complex, another young "star forming region", that are all <1 million years old, so particularly young. We would like to know how much material is present in those systems, so that we can understand the types of planetary systems that they can form.

### Measuring the masses of protoplanetary disks in Taurus

To get started, let's download the data for one of the 27 objects. I have put all of the images into a Google Drive folder, when you are ready, ask Patrick to share that folder with you. Go to that folder, pick out one of the objects, and download the FITS image for that source. If doing this via Google Colab, you should be able to load Google Drive and read the data directly from there (*Hint:* See the AccessingGoogleDrive notebook):

```python

```

Also, load in the data from the FITS file (*Hint: See the Astropy-images tutorial for further details*):

```python

```

Now, use matplotlib to make a plot showing an image of that FITS file that you just downloaded. You might want to change the stretch to see if you can bring out any faint features:

```python

```

It turns out that how much material is in a protoplanetary disk is proportional to the brightness of the disk in this image, or the "flux". What is the highest value you can find for one of the pixels in the image? That should be pretty close to the total flux of the source. Record it here:

```python
flux =
```

The formula for calculating the mass of a disk from a flux is:

\begin{equation}
M_d = \frac{F_{\nu} \, d^2}{B_{\nu}(T) \, \kappa_{\nu}}
\end{equation}

where $F_{\nu}$ is the flux, $d$ is the distance to the source, $T$ is the temperature of the disk, and $\kappa_{\nu}$ is the "opacity". Disks typically have a temperature of $\sim20$ K, so that is probably appropriate to use here. A typical value for $\kappa_{\nu}$ is $\sim 0.1$ cm$^2$/g, so that is also a reasonable value to use. $\nu$ is the frequency of light that the picture was taken with, which happens to be 33 GHz. You'll need to look up the distance to the Taurus Molecular Cloud Complex. $B_{\nu}(T)$ is the "blackbody function". Rather than trying to program this yourself (it's complicated), you can use the blackbody_nu function in the astropy package: https://docs.astropy.org/en/stable/api/astropy.modeling.blackbody.blackbody_nu.html#astropy.modeling.blackbody.blackbody_nu.

```python
from astropy.modeling.blackbody import blackbody_nu
```

Use the blackbody function and the information above to calculate mass of this protoplanetary disk:

```python
mass =
```

The Earth has a mass of $M_{\oplus} = 5.972 \times 10^{27}$ grams. How many Earths could be formed out of this disk?

```python

```

Now that we've done this calculation for a single disk, can you write a loop to do the calculation for all 27 of the disks? One thing to keep in mind - there might be some sources that are binary or multiple systems. Can you make your code report all of the sources? Also, do any of the images have any weird features in them? They may not, but one thing to keep an eye out for are  strange rays or striping in the image that could be caused by issues with the observations. If so, report to Patrick so he can fix them!

```python

```

Finally, now that we have masses for all of the disks, can you make a plot of the distribution of the masses of all of the disks? $Hint$: You can use the matplotlib "hist" function: https://matplotlib.org/3.2.1/api/_as_gen/matplotlib.pyplot.hist.html. Also consider plotting it as a cumulative distribution, also using the hist function.

```python

```

### Comparing with Protoplanetary Disks in Other Star Forming Regions

Taurus isn't the only star forming region, but it is one of the youngest, with disks that are $<1$ millions years old. Other scientists have done calculations similar to the ones you just did to calculate the masses of protoplanetary disks in older star forming regions. Two of those that we want to compare to are the Chameleon star forming region ($\sim2-3$ million years old) and the Upper Sco region ($\sim5-10$ million years old). The data for the masses is available here: https://github.com/psheehan/CIERA-HS-Program/tree/master/Projects/WeighingBabySolarSystems in usc_d.txt and cham_d.txt. Download those data here, and then the following lines will read them in.

Download the data:

```python

```

Now, this code will read them in:

```python
upper_sco_masses = numpy.loadtxt("UpperSco.txt")
chameleon_masses = numpy.loadtxt("Chameleon.txt")
```

Make a histogram showing a comparison of the disk masses in Taurus with the disk masses in Chameleon and Upper Sco. Make sure they are properly labeled so that we can tell which line is which. Again, you might consider plotting a cumulative distribution and see how that looks for comparison.

```python

```

What can you see about the disk masses in the different star forming regions? Which has the most massive disks? Is there a trend with age?

### A More Accurate Mass Measurement

So far we've been using the peak flux as the measurement of how bright a disk is, but in reality there are more pixels than just the single peak pixel. Here we are going to write some code to do a more realistic flux measurement. This will be particularly useful for sources that are clearly "resolved", i.e. they are extended and look like more than just a single, compact circular blob.

There are a few steps we will want to take. First, we want to have a quantitative assessment of which emission is "real", and not just noise. To do this, lets calculate the standard deviation of the image (*Hint:* See the numpy "std" function):

```python

```

We will only want to consider emission greater that 3$\times$ the standard deviation of the image. Write some code that returns the locations of the image where emission is greater than 3$\times$ the value you just calculated:

```python

```

Now can you put this together and create a function that adds up all of the emission above 3$\times$ the standard deviation? Be careful of binaries, you might need to add additional criteria!

```python

```

The last thing we need to do is to adjust for the effects of the "beam". The flux values reported by the image are actually reported in units of Jy/beam, where the "beam" is basically how much the image is blurred by the telescope. The size of the beam is encoded in two values in the header of the FITS file: "BMAJ" and "BMIN", and the area of the beam can be calculated as $\pi$BMAJ$\times$BMIN/4. These are also recorded in units of degrees, but we care about the size of the beam in pixels. You should be able to find the size of the pixels, in degrees, in the header value "CDELT1". Calculate the area of the beam in pixels, using this information:

```python

```

To correct your function to calculate the total flux that you wrote above for the beam, re-write that function so that it now divides by the area of the beam, in pixels, below:

```python

```

Now, apply this new function for calculating the flux of the protoplanetary disks to all 27 sources:

```python

```

Finally, re-make the histogram plots from above, comparing the masses you calculate here to the masses of protoplanetary disks in other star forming regions. Does this change any of your conclusions?

```python

```

### Still looking for more to do?

Talk to Patrick, and he'll be more than happy to give you more to do! In particular, looking into fitting "Gaussian" functions to the sources to measure their fluxes and sizes in a more rigorous way.

```python

```


```python
from decodes.core import *
from decodes.io.jupyter_out import JupyterOut
import math

out = JupyterOut.unit_square( )
```

    http://decod.es/	v0.2.3
    io loaded

# A Parametric Representation

The parametric representation of surfaces has much in common with that of curves.

Where a parameterized curve is described as a function that maps a single numeric interval onto points in space, a parameterized surface can be similarly expressed as ***a function that maps a domain in two dimensions onto points in space***.

In other words, while each may be understood as ***a machine for the production of three-dimensional points***, a curve does so using a single parameter $t$, while a surface requires a pair of parameters $u,v$.

In mathematic notation, a parametric surface is denoted by a function $\Phi$ that takes in two parameters $u$ and $v$ and produces a collection of points, denoted by $\Phi(u,v)$, as the parameter pair $(u,v)$ varies in the two-dimensional domain.

Just as we saw for a curve, three elements – *** a function, a domain, and a resulting set of points *** – constitute a parameterized surface. Expanded out in coordinates, any point on this surface can be written as:

\begin{align}
\Phi(u,v) = (x(u,v),y(u,v),z(u,v))
\end{align}

One important simplification that we make in moving towards expressing a surface in code concerns the domain.

Generally speaking, surface domains can take on arbitrary shapes in two dimensions. To simplify our implementation in code, ***we will restrict surface domains to be rectangular***.

Working from this assumption, ***the domain may be represented by two intervals, one in $u$ and one in $v$***.

The complete mathematical description of a parameterized surface can be written as:

\begin{align*}
x = &\ldots \\
y = &\ldots \\
z = &\ldots \\
u\colon u_{0} &\rightarrow u_{1} \\
v\colon v_{0} &\rightarrow v_{1}
\end{align*}

In code, this may be expressed as:

```python
def evaluate(u,v):
    x = ...
    y = ...
    z = ...
    return Point(x,y,z)
```

```python
for u in Interval(u0,u1)/divs_u:
    for v in Interval(v0,v1)/divs_v:
        pts.append(evaluate(u,v))
```

## Surface Objects in Decod.es

Just as the basic elements of a surface are the same as a curve, so the basic attributes of a Decod.es Surface class parallel those exhibited by a Curve.

Distinctions between these two are primarily a result of the pair of parameters called for by Surface evaluation, which precipitates the need for a pair of domain intervals and a pair of tolerance values.

```python
"""
Surface Class Construction
Like a Curve, a Surface is constructed of an evaluation function and two tolerance values. The
domain is expressed as a tuple of two Intervals. Note that in the Decod.es library, the tolerance
arguments are optional, and default to fixed ration of the u and v domains.
"""
class Surface():
    def __init__(self, function,  dom_u, dom_v, tol_u, tol_v):
        self._func = function
        self._dom = dom_u, dom_v
        self._tol = tol_u, tol_v
```

Surface initialization closely resembles the initialization of a Curve.

        *Basic Surface Members and Methods*

        `srf._func`
`srf.func`
        Function
        A function that, given two parameters `u` and `v`, returns a Point in space.

        `srf._dom`
`srf.domain_u`
`srf.domain_v`
        (Interval,Interval)
        A pair of Intervals that describe valid `u` and `v` values to evaluate.

        `srf._tol`
`srf.tol_u`
`srf.tol_v`
        (Float,Float)
        A pair of values that describe the tolerance of the curve expressed in terms of the `u` and `v` domain Intervals.

        `srf.deval(u,v)`
        Point
        Given parameters `u` and `v`, returns a Point that falls on this Surface within the defined domain.

        `srf.eval(u,v)`
        Point
        Given parameters `u` and `v`, returns a Point that falls on this Surface within a normalized domain.

        *Basic Surface Properties*

        `srf.surrogate`
        Mesh
        Returns a Mesh representation of this Surface constructed from a sub-sampling of Points determined by `surf.tol_u`, `surf.tol_v`.

        `srf.u0`
`srf.u1`
`srf.v0`
`srf.v1`
        Float
        Returns the upper or lower bounds of the related `u` or `v` domain Interval.

```python
"""
Surface Evaluation
Evaluates this Surface and returns a Point. u and v are float values that fall within the defined
domain of this Surface.
"""
def deval(self,u,v):
    return Point(self.func(u,v))
```

```python
"""
Surface Evaluation
Evaluates this Surface and returns a Point. u and v are normalized float values (0-\>1) which will
be remapped to the domain defined by this Surface.
"""
def eval(self,u,v):
    return self.deval(
        Interval.remap(u,Interval(),self.domain_u),
        Interval.remap(v,Interval(),self.domain_v)
        )
```

```python
"""
Isoparms
Isoparametric Curves and PLines may be extracted from Surfaces using the methods seen below. Keyword
arguments are passed to define the value of the fixed parameter, such that a u-isoparm is constructed
with a fixed value for v, and vice-versa.
"""
# a curve for which u varies and v is fixed
iso_u = surf.isocurve( v_val = 0.5 )

# a polyine for which u is fixed and v varies.
iso_v = surf.isopolyline( u_val = 0.25, res = 20 )
```

```python
"""
Surface Mesh
"""
v_vals = Interval().divide(res_v,True)
u_vals = Interval().divide(res_u,True)
pts = [surf.eval(u,v) for v in v_vals for u in u_vals]
msh = Mesh(pts)

for j in range(res_v):
    row = j*(res_u+1)
    for i in range(res_u):
        pi_0 = row+i
        pi_1 = row+i+1
        pi_2 = row+i+res_u+2
        pi_3 = row+i+res_u+1
        msh.add_face(pi_0,pi_1,pi_2,pi_3)
```

## Crafting Parametric Surfaces

Having defined the basic elements of a parameterized surface in code, we have all we need to construct our own Surfaces.

As with Curves, all that this requires is the definition of a coordinate function and the specification of the appropriate pair of domain intervals.

Such a format should ring familiar, since it recalls not only the format of a Curve, but also the Mathematical Monsters series of examples that we introduced earlier.

```python
"""
A Monstrous Surface
The construction of Surfaces reflect the format we employed in an earlier series of examples. Here,
a flower-like surface is defined. On its own, the Surface itself is difficult to visualize without
extracting some form of discrete representation.
"""
a,b = 1,1

ival_u, ival_v = Interval.twopi(), Interval.twopi()
def func(u,v):
    x = ( (a+1)*math.cos(u) + math.cos(u*(a+1)) ) * (v + 1)
    y = ( (a+1)*math.sin(u) + math.sin(u*(a+1)) ) * (v + 1)
    z = 3.0*math.sin(v*b)
    return Point (x,y,z)

srf = Surface(func,ival_u, ival_v)
```

The code below plots a Mesh along this Surface with an arbitrary density of faces.

```python
"""
Mesh Extraction from Surface
A discrete Mesh may be extracted from a Surface as a surrogate. Here, we construct a Mesh version
of the monstrous Surface defined above.
"""
srf.tol_u = srf.domain_u.delta / count_u
srf.tol_v = srf.domain_v.delta / count_v
msh = srf.surrogate
```

```python
"""
Orecchiette Surface Parametrization
Based on work by George L. Legendre in Pasta by Design
"""
ival_u, ival_v = Interval(0,15), Interval(0,150)
def func (u, v):
    upi,vpi = u*pi,v*pi
    x = 2/3*v*cos(upi/75) + 0.3*cos(2/15*upi)
    y = 10*sin(upi/75)
    z = 0.1*cos(upi/3) +
        5*(cos(vpi/30)**2) * (0.5 + 0.5*(cos(2*upi/75)))**4  +
        1.5*(0.5 + 0.5*cos(2*upi/75))**5 * sin(vpi/30)**10
    return Point(x,y,z)
```

While appropriating existing surface parameterizations is a straightforward exploratory exercise, these methods on their own do not help us to compose surfaces that meet specific aims, or to craft our own parameterizations with precision.

We must be able to discern the meaning of those parameters that drive known surfaces in order to effectively guide their modification, and should be able to craft our own parameterizations such that a desired set of properties is produced. Nothing develops this skill as effectively as familiarity with a wide variety of precedents.

With this aim in mind, here we turn to examining some mathematical surfaces represented in a more familiar way, starting with a surface represented by a function $z = f(x,y)$ which maps a point in the x-y plane to a point above or below in space.

When the given point is restricted to a rectangle, a mathematician would call such a surface a ***graph of the function***.

This form can be easily parameterized by letting the domain be the rectangle in the plane, which is also the surface as seen from above. We may express a graph surface in the more familiar format of a parameterization function and domain by setting $u = x$ and $v = y$:

\begin{align*}
x &= u\\
y &= v\\
z &= f(u,v)\\
u\colon a &\rightarrow b \\
v\colon c &\rightarrow d
\end{align*}

In this format, the evaluation function for any graph surface produces a Point with `x` and `y` coordinates that match the given `u` and `v` arguments, and has a height given by the graph function.

```python
"""
A Graph Surface
Graph surfaces exhibit a direct relationship between their domains, which may be understood as
rectangular patches on the x-y plane, and the resulting form, which conforms to this shape when
viewed from above. Here we see the graph surface related to the "product of lines" function.
"""
ival_u, ival_v = Interval(a,b), Interval(c,d)
def func(u,v):
    z = (u - a)*(u - b)*(v - c)*(v - d)
    return Point(u,v,z)
```

The graph surface is not the limit of what can be achieved within the constraints of a rectangular domain.

***A rectangular domain need not produce a rectangular surface***.

To see this clearly, we may consider how the alternate coordinate geometries allow us to describe a number of useful surfaces by parameterizations on a rectangular domain.

### Parameterization Using Alternate Coordinate Geometries

Many of the most elemental parameterizations with which we should
be familiar employ some modification of cylindrical and spherical
coordinates.

A basic literacy in
these alternate ways of representing points in space enables access
to a rich world of already-discovered surface parameterizations.

#### Constant Surfaces
The most immediate surfaces associated
with these alternate systems of coordinates are the ***constant
surfaces*** that result from ***setting one parameter to a constant value,
while allowing the other two to vary.***

```python
"""
A Cylindrical Surface
Rather than defining a Point directly in Cartesian coordinates, we may construct a cylindrical
surface by plotting Points in cylindrical space.
"""
ival_u, ival_v = Interval.twopi(), Interval(0,h)
def func(u,v):
    return CS().eval_cyl(const, u, v)
```

```python
"""
Spherical Patch Surface
By employing spherical coordinates, we can adjust the boundary parallels and meridians of a
spherical patch.
"""
ival_u, ival_v = Interval(theta_0, theta_1), Interval(phi_0,phi_1)
def func(u,v):
    rho = const
    theta, phi = u,v
    return CS().eval_sph(rho,phi,theta)
```

#### Modifications of Constant Surfaces

Consider the considerable range of shapes that are
made possible by making controlled modifications to the parameterizations of constant surfaces.

#### Helicoid

```python
"""
Helicoid
Constructs a helix-like Surface given parameters r1, r2 that define the inner and outer radii,
n that defines the number of turns, and b that relates to the overall height.
"""
ival_u, ival_v = Interval(0, 2*pi*n), Interval(r1,r2)
def func(u,v):
    return CS().eval_cyl( v, u, b*u )
```

#### Catenoid

```python
"""
Catenoid
Constructs an hourglass-like Surface given parameters h that defines the height, and c that defines
the curvature and horizontal extents.
"""
ival_u, ival_v = Interval.twopi(), Interval(-h/2,h/2)
def func(u,v):
    return CS().eval_cyl( cosh(v/c), u, v )
```

#### Torus

```python
"""
Torus
Constructs an doughnut-like Surface given parameters r1 (shown as R in equation) that defines the
primary radius and r2 (shown as r) that defines the secondary radius.
"""
ival_u, ival_v = Interval(theta0,theta1), Interval.twopi()
def func(u,v):
    x = (r1 + r2*cos(v))*cos(u)
    y = (r1 + r2*cos(v))*sin(u)
    z = r2*sin(v)
    return Point(x,y,z)
```

#### Superellipsoid

```python
"""
Superellipsoid
Constructs a pillow-like Surface given parameters dim_x, dim_y, and dim_z that describe the
dimensions of the bounding box, and parameters m and n that describe the 'puffiness' of the
resulting form.
"""
ival_u, ival_v = Interval.twopi(), Interval(0, pi)

def sgn_pow(x, alpha):
    if x == 0: return 0
    return abs(x)**alpha * (x/abs(x))

def func(u,v):
    x = sgn_pow(sin(v),m) * sgn_pow(cos(u),n)
    y = sgn_pow(sin(v),m) * sgn_pow(sin(u),n)
    z = sgn_pow(cos(v),m)
    return Point(dim_x * x, dim_y * y, dim_z * z)
```

110 - Surfaces/357 - A Parametric Representation.ipynb