# Model Checking

After running an MCMC simulation, `sample` returns a `MultiTrace` object containing the samples for all the stochastic and deterministic random variables. The final step in Bayesian computation is model checking, in order to ensure that inferences derived from your sample are valid. There are two components to model checking:

1. Convergence diagnostics
2. Goodness of fit

Convergence diagnostics are intended to detect lack of convergence in the Markov chain Monte Carlo sample; it is used to ensure that you have not halted your sampling too early. However, a converged model is not guaranteed to be a good model. The second component of model checking, goodness of fit, is used to check the internal validity of the model, by comparing predictions from the model to the data used to fit the model.

## Convergence Diagnostics

Valid inferences from sequences of MCMC samples are based on the
assumption that the samples are derived from the true posterior
distribution of interest. Theory guarantees this condition as the number
of iterations approaches infinity. It is important, therefore, to
determine the **minimum number of samples** required to ensure a reasonable
approximation to the target posterior density. Unfortunately, no
universal threshold exists across all problems, so convergence must be
assessed independently each time MCMC estimation is performed. The
procedures for verifying convergence are collectively known as
*convergence diagnostics*.

One approach to analyzing convergence is **analytical**, whereby the
variance of the sample at different sections of the chain are compared
to that of the limiting distribution. These methods use distance metrics
to analyze convergence, or place theoretical bounds on the sample
variance, and though they are promising, they are generally difficult to
use and are not prominent in the MCMC literature. More common is a
**statistical** approach to assessing convergence. With this approach,
rather than considering the properties of the theoretical target
distribution, only the statistical properties of the observed chain are
analyzed. Reliance on the sample alone restricts such convergence
criteria to **heuristics**. As a result, convergence cannot be guaranteed.
Although evidence for lack of convergence using statistical convergence
diagnostics will correctly imply lack of convergence in the chain, the
absence of such evidence will not *guarantee* convergence in the chain.
Nevertheless, negative results for one or more criteria may provide some
measure of assurance to users that their sample will provide valid
inferences.

For most simple models, convergence will occur quickly, sometimes within
a the first several hundred iterations, after which all remaining
samples of the chain may be used to calculate posterior quantities. For
more complex models, convergence requires a significantly longer burn-in
period; sometimes orders of magnitude more samples are needed.
Frequently, lack of convergence will be caused by **poor mixing**.
Recall that *mixing* refers to the degree to which the Markov
chain explores the support of the posterior distribution. Poor mixing
may stem from inappropriate proposals (if one is using the
Metropolis-Hastings sampler) or from attempting to estimate models with
highly correlated variables.

```python
%matplotlib inline
import numpy as np
import seaborn as sns; sns.set_context('notebook')
import warnings
warnings.filterwarnings("ignore", category=UserWarning)
```

```python
from pymc3 import Normal, Binomial, sample, Model
from pymc3.math import invlogit

# Samples for each dose level
n = 5 * np.ones(4, dtype=int)
# Log-dose
dose = np.array([-.86, -.3, -.05, .73])
deaths = np.array([0, 1, 3, 5])

with Model() as bioassay_model:

    # Logit-linear model parameters
    alpha = Normal('alpha', 0, sd=100)
    beta = Normal('beta', 0, sd=100)

    # Calculate probabilities of death
    theta = invlogit(alpha + beta * dose)

    # Data likelihood
    obs_deaths = Binomial('obs_deaths', n=n, p=theta, observed=deaths)
```

```python
with bioassay_model:
    bioassay_trace = sample(1000)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [beta, alpha]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:03<00:00, 875.72draws/s]
    The acceptance probability does not match the target. It is 0.8964481369333642, but should be close to 0.8. Try to increase the number of tuning steps.

```python
from pymc3 import traceplot

traceplot(bioassay_trace, varnames=['alpha'])
```

### Informal Methods

The most straightforward approach for assessing convergence is based on
simply **plotting and inspecting traces and histograms** of the observed
MCMC sample. If the trace of values for each of the stochastics exhibits
asymptotic behavior over the last $m$ iterations, this may be
satisfactory evidence for convergence.

```python
traceplot(bioassay_trace, varnames=['beta'])
```

A similar approach involves
plotting a histogram for every set of $k$ iterations (perhaps 50-100)
beyond some burn in threshold $n$; if the histograms are not visibly
different among the sample intervals, this may be considered some evidence for
convergence. Note that such diagnostics should be carried out for each
stochastic estimated by the MCMC algorithm, because convergent behavior
by one variable does not imply evidence for convergence for other
variables in the analysis.

```python
import matplotlib.pyplot as plt

beta_trace = bioassay_trace['beta']

fig, axes = plt.subplots(2, 5, figsize=(14,6))
axes = axes.ravel()
for i in range(10):
    axes[i].hist(beta_trace[100*i:100*(i+1)])
plt.tight_layout()
```

An extension of this approach can be taken
when multiple parallel chains are run, rather than just a single, long
chain. In this case, the final values of $c$ chains run for $n$
iterations are plotted in a histogram; just as above, this is repeated
every $k$ iterations thereafter, and the histograms of the endpoints are
plotted again and compared to the previous histogram. This is repeated
until consecutive histograms are indistinguishable.

Another *ad hoc* method for detecting lack of convergence is to examine
the traces of several MCMC chains initialized with different starting
values. Overlaying these traces on the same set of axes should (if
convergence has occurred) show each chain tending toward the same
equilibrium value, with approximately the same variance. Recall that the
tendency for some Markov chains to converge to the true (unknown) value
from diverse initial values is called *ergodicity*. This property is
guaranteed by the reversible chains constructed using MCMC, and should
be observable using this technique. Again, however, this approach is
only a heuristic method, and cannot always detect lack of convergence,
even though chains may appear ergodic.

```python
with bioassay_model:

    bioassay_trace = sample(1000, chains=2, start=[{'alpha':0.5}, {'alpha':5}])
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [beta, alpha]
    Sampling 2 chains: 100%|██████████| 3000/3000 [00:03<00:00, 917.28draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
bioassay_trace.get_values('alpha', chains=0)[0]
```

    1.2628710304833797

```python
plt.plot(bioassay_trace.get_values('alpha', chains=0)[:200], 'r--')
plt.plot(bioassay_trace.get_values('alpha', chains=1)[:200], 'k--')
```

A principal reason that evidence from informal techniques cannot
guarantee convergence is a phenomenon called ***metastability***. Chains may
appear to have converged to the true equilibrium value, displaying
excellent qualities by any of the methods described above. However,
after some period of stability around this value, the chain may suddenly
move to another region of the parameter space. This period
of metastability can sometimes be very long, and therefore escape
detection by these convergence diagnostics. Unfortunately, there is no
statistical technique available for detecting metastability.

### Formal Methods

Along with the *ad hoc* techniques described above, a number of more
formal methods exist which are prevalent in the literature. These are
considered more formal because they are based on existing statistical
methods, such as time series analysis.

PyMC currently includes three formal convergence diagnostic methods. The
first, proposed by [Geweke (1992)](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1177011446), is a time-series approach that
compares the mean and variance of segments from the beginning and end of
a single chain.

$$z = \frac{\bar{\theta}_a - \bar{\theta}_b}{\sqrt{S_a(0) + S_b(0)}}$$

where $a$ is the early interval and $b$ the late interval, and $S_i(0)$ is the spectral density estimate at zero frequency for chain segment $i$. If the
z-scores (theoretically distributed as standard normal variates) of
these two segments are similar, it can provide evidence for convergence.
PyMC calculates z-scores of the difference between various initial
segments along the chain, and the last 50% of the remaining chain. If
the chain has converged, the majority of points should fall within 2
standard deviations of zero.

In PyMC, diagnostic z-scores can be obtained by calling the `geweke` function. It
accepts either (1) a single trace, (2) a Node or Stochastic object, or
(4) an entire Model object:

```python
from pymc3 import geweke

with bioassay_model:
    tr = sample(2000, tune=1000)

z = geweke(tr, intervals=15)
```

    Auto-assigning NUTS sampler...
    Initializing NUTS using jitter+adapt_diag...
    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [beta, alpha]
    Sampling 2 chains: 100%|██████████| 6000/6000 [00:08<00:00, 706.98draws/s]
    There were 1 divergences after tuning. Increase `target_accept` or reparameterize.
    The number of effective samples is smaller than 25% for some parameters.

```python
plt.scatter(*z[0]['alpha'].T)
plt.hlines([-1,1], 0, 1000, linestyles='dotted')
plt.xlim(0, 1000)
```

The arguments expected are the following:

- `x` : The trace of a variable.
- `first` : The fraction of series at the beginning of the trace.
- `last` : The fraction of series at the end to be compared with the section at the beginning.
- `intervals` : The number of segments.

Plotting the output displays the scores in series, making it is easy to
see departures from the standard normal assumption.

A second convergence diagnostic provided by PyMC is the Gelman-Rubin
statistic [Gelman and Rubin (1992)](http://projecteuclid.org/DPubS?service=UI&version=1.0&verb=Display&handle=euclid.ss/1177011136). This diagnostic uses multiple chains to
check for lack of convergence, and is based on the notion that if
multiple chains have converged, by definition they should appear very
similar to one another; if not, one or more of the chains has failed to
converge.

The Gelman-Rubin diagnostic uses an analysis of variance approach to
assessing convergence. That is, it calculates both the between-chain
varaince (B) and within-chain varaince (W), and assesses whether they
are different enough to worry about convergence. Assuming $m$ chains,
each of length $n$, quantities are calculated by:

$$\begin{align}B &= \frac{n}{m-1} \sum_{j=1}^m (\bar{\theta}_{.j} - \bar{\theta}_{..})^2 \\
W &= \frac{1}{m} \sum_{j=1}^m \left[ \frac{1}{n-1} \sum_{i=1}^n (\theta_{ij} - \bar{\theta}_{.j})^2 \right]
\end{align}$$

for each scalar estimand $\theta$. Using these values, an estimate of
the marginal posterior variance of $\theta$ can be calculated:

$$\hat{\text{Var}}(\theta | y) = \frac{n-1}{n} W + \frac{1}{n} B$$

Assuming $\theta$ was initialized to arbitrary starting points in each
chain, this quantity will overestimate the true marginal posterior
variance. At the same time, $W$ will tend to underestimate the
within-chain variance early in the sampling run. However, in the limit
as $n \rightarrow
\infty$, both quantities will converge to the true variance of $\theta$.
In light of this, the Gelman-Rubin statistic monitors convergence using
the ratio:

$$\hat{R} = \sqrt{\frac{\hat{\text{Var}}(\theta | y)}{W}}$$

This is called the potential scale reduction, since it is an estimate of
the potential reduction in the scale of $\theta$ as the number of
simulations tends to infinity. In practice, we look for values of
$\hat{R}$ close to one (say, less than 1.1) to be confident that a
particular estimand has converged. In PyMC, the function
`gelman_rubin` will calculate $\hat{R}$ for each stochastic node in
the passed model:

```python
from pymc3 import gelman_rubin

gelman_rubin(bioassay_trace)
```

    {'alpha': 1.0014649509860685, 'beta': 1.0031377765269096}

For the best results, each chain should be initialized to highly
dispersed starting values for each stochastic node.

By default, when calling the `forestplot` function using nodes with
multiple chains, the $\hat{R}$ values will be plotted alongside the
posterior intervals.

```python
from pymc3 import forestplot

forestplot(bioassay_trace)
```

## Autocorrelation

In general, samples drawn from MCMC algorithms will be autocorrelated. This is not a big deal, other than the fact that autocorrelated chains may require longer sampling in order to adequately characterize posterior quantities of interest. The calculation of autocorrelation is performed for each lag $i=1,2,\ldots,k$ (the correlation at lag 0 is, of course, 1) by:

$$\hat{\rho}_i = 1 - \frac{V_i}{2\hat{\text{Var}}(\theta | y)}$$

where $\hat{\text{Var}}(\theta | y)$ is the same estimated variance as calculated for the Gelman-Rubin statistic, and $V_i$ is the variogram at lag $i$ for $\theta$:

$$\text{V}_i = \frac{1}{m(n-i)}\sum_{j=1}^m \sum_{k=i+1}^n (\theta_{jk} - \theta_{j(k-i)})^2$$

This autocorrelation can be visualized using the `autocorrplot` function in PyMC3:

```python
from pymc3 import autocorrplot

autocorrplot(tr);
```

### Effective sample size

The effective sample size is estimated using the partial sum:

$$\hat{n}_{eff} = \frac{mn}{1 + 2\sum_{i=1}^T \hat{\rho}_i}$$

where $T$ is the first odd integer such that $\hat{\rho}_{T+1} + \hat{\rho}_{T+2}$ is negative.

The issue here is related to the fact that we are **estimating** the effective sample size from the fit output. Values of $n_{eff} / n_{iter} < 0.001$ indicate a biased estimator, resulting in an overestimate of the true effective sample size.

```python
from pymc3 import effective_n

effective_n(bioassay_trace)
```

    {'alpha': 450.4086151352394, 'beta': 355.07144137202664}

Both low $n_{eff}$ and high $\hat{R}$ indicate **poor mixing**.

It is tempting to want to **thin** the chain to eliminate the autocorrelation (*e.g.* taking every 20th sample from the traces above), but this is a waste of time. Since thinning deliberately throws out the majority of the samples, no efficiency is gained; you ultimately require more samples to achive a particular desired sample size.

## Diagnostics for Gradient-based Samplers

Hamiltonian Monte Carlo is a powerful and efficient MCMC sampler when set up appropriately. However, this typically requires carefull tuning of the sampler parameters, such as tree depth, leapfrog step size and target acceptance rate. Fortunately, the NUTS algorithm takes care of some of this for us. Nevertheless, tuning must be carefully monitored for failures that frequently arise. This is particularly the case when fitting challenging models, such as those with high curvature or heavy tails.

Fortunately, however, gradient-based sampling provides the ability to diagnose these pathologies. PyMC makes several diagnostic statistics available as attributes of the `MultiTrace` object returned by the `sample` function.

```python
bioassay_trace.stat_names
```

    {'depth',
     'diverging',
     'energy',
     'energy_error',
     'max_energy_error',
     'mean_tree_accept',
     'step_size',
     'step_size_bar',
     'tree_size',
     'tune'}

- `mean_tree_accept`: The mean acceptance probability for the tree that generated this sample. The mean of these values across all samples but the burn-in should be approximately `target_accept` (the default for this is 0.8).
- `diverging`: Whether the trajectory for this sample diverged. If there are many diverging samples, this usually indicates that a region of the posterior has high curvature. Reparametrization can often help, but you can also try to increase `target_accept` to something like 0.9 or 0.95.
- `energy`: The energy at the point in phase-space where the sample was accepted. This can be used to identify posteriors with problematically long tails. See below for an example.
- `energy_error`: The difference in energy between the start and the end of the trajectory. For a perfect integrator this would always be zero.
- `max_energy_error`: The maximum difference in energy along the whole trajectory.
- `depth`: The depth of the tree that was used to generate this sample
- `tree_size`: The number of leafs of the sampling tree, when the sample was accepted. This is usually a bit less than $2 ^ \text{depth}$. If the tree size is large, the sampler is using a lot of leapfrog steps to find the next sample. This can for example happen if there are strong correlations in the posterior, if the posterior has long tails, if there are regions of high curvature ("funnels"), or if the variance estimates in the mass matrix are inaccurate. Reparametrisation of the model or estimating the posterior variances from past samples might help.
- `tune`: This is `True`, if step size adaptation was turned on when this sample was generated.
- `step_size`: The step size used for this sample.
- `step_size_bar`: The current best known step-size. After the tuning samples, the step size is set to this value. This should converge during tuning.

If the name of the statistic does not clash with the name of one of the variables, we can use indexing to get the values. The values for the chains will be concatenated.

We can see that the step sizes converged after the 2000 tuning samples for both chains to about the same value. The first 3000 values are from chain 1, the second from chain 2.

```python
with bioassay_model:
    trace = sample(1000, tune=2000, init=None, chains=2, discard_tuned_samples=False)
```

    Multiprocess sampling (2 chains in 2 jobs)
    NUTS: [beta, alpha]
    Sampling 2 chains: 100%|██████████| 6000/6000 [00:06<00:00, 940.06draws/s]
    The number of effective samples is smaller than 25% for some parameters.

```python
plt.plot(trace['step_size_bar'])
```

The `get_sampler_stats` method provides more control over which values should be returned, and it also works if the name of the statistic is the same as the name of one of the variables. We can use the `chains` option, to control values from which chain should be returned, or we can set `combine=False` to get the values for the individual chains:

The `NUTS` step method has a maximum tree depth parameter so that infinite loops (which can occur for non-identified models) are avoided. When the maximum tree depth is reached (the default value is 10), the trajectory is stopped. However complex (but identifiable) models can saturate this threshold, which reduces sampling efficiency.

The `MultiTrace` stores the tree depth for each iteration, so inspecting these traces can reveal saturation if it is occurring.

```python
sizes1, sizes2 = trace.get_sampler_stats('depth', combine=False)
fig, (ax1, ax2) = plt.subplots(2, 1, sharex=True, sharey=True)
ax1.plot(sizes1)
ax2.plot(sizes2)
```

We can also check the acceptance for the trees that generated this sample. The mean of these values across all samples (except the tuning stage) is expected to be the same as `target_accept`, which is 0.8 by default.

```python
accept = trace.get_sampler_stats('mean_tree_accept', burn=1000)
sns.distplot(accept, kde=False)
```

### Divergent transitions

Recall that simulating Hamiltonian dynamics via a symplectic integrator uses a discrete approximation of a continuous function. This is only a reasonable approximation when the step sizes of the integrator are suitably small. A divergent transition may indicate that the approximation is poor.

If there are too many divergent transitions, then samples are not being drawn from the full posterior, and inferences based on the resulting sample will be biased

If there are diverging transitions, PyMC3 will issue warnings indicating how many were discovered. We can obtain the indices of them from the trace.

```python
trace['diverging'].nonzero()
```

    (array([  53,  135, 1779, 3001, 3014, 3063, 3089, 3192, 3820, 4816]),)

### Bayesian Fraction of Missing Information

The Bayesian fraction of missing information (BFMI) is a measure of how hard it is to
sample level sets of the posterior at each iteration. Specifically, it quantifies how well momentum resampling matches the marginal energy distribution. A small value indicates that the adaptation phase of the sampler was unsuccessful, and invoking the central limit theorem may not be valid. It indicates whether the sampler is able to adequately explore the posterior distribution.

Though there is not an established rule of thumb for an adequate threshold, values close to one are optimal. Reparameterizing the model is sometimes helpful for improving this statistic.

```python
from pymc3 import bfmi
```

```python
bfmi(trace)
```

    0.9616500794573204

Another way of diagnosting this phenomenon is by comparing the overall distribution of
energy levels with the *change* of energy between successive samples. Ideally, they should be very similar.

If the distribution of energy transitions is narrow relative to the marginal energy distribution, this is a sign of inefficient sampling, as many transitions are required to completely explore the posterior. On the other hand, if the energy transition distribution is similar to that of the marginal energy, this is evidence of efficient sampling, resulting in near-independent samples from the posterior.

```python
energy = trace['energy']
energy_diff = np.diff(energy)
sns.distplot(energy - energy.mean(), label='energy')
sns.distplot(energy_diff, label='energy diff')
plt.legend()
```

If the overall distribution of energy levels has longer tails, the efficiency of the sampler will deteriorate quickly.

## Goodness of Fit

Checking for model convergence is only the first step in the evaluation
of MCMC model outputs. It is possible for an entirely unsuitable model
to converge, so additional steps are needed to ensure that the estimated
model adequately fits the data. One intuitive way of evaluating model
fit is to compare model predictions with the observations used to fit
the model. In other words, the fitted model can be used to simulate
data, and the distribution of the simulated data should resemble the
distribution of the actual data.

Fortunately, simulating data from the model is a natural component of
the Bayesian modelling framework. Recall, from the discussion on
imputation of missing data, the posterior predictive distribution:

$$p(\tilde{y}|y) = \int p(\tilde{y}|\theta) f(\theta|y) d\theta$$

Here, $\tilde{y}$ represents some hypothetical new data that would be
expected, taking into account the posterior uncertainty in the model
parameters.

Sampling from the posterior predictive distribution is easy
in PyMC. The `sample_ppc` function draws posterior predictive checks from all of the data likelhioods. Consider the `gelman_bioassay` example,
where deaths are modeled as a binomial random variable for which
the probability of death is a logit-linear function of the dose of a
particular drug.

The posterior predictive distribution of deaths uses the same functional
form as the data likelihood, in this case a binomial stochastic. Here is
the corresponding sample from the posterior predictive distribution (we typically need very few samples relative to the MCMC sample):

```python
from pymc3 import sample_ppc

with bioassay_model:
    deaths_sim = sample_ppc(bioassay_trace, samples=500)
```

    100%|██████████| 500/500 [00:00<00:00, 1371.23it/s]

The degree to which simulated data correspond to observations can be evaluated in at least two ways. First, these quantities can simply be compared visually. This allows for a qualitative comparison of model-based replicates and observations. If there is poor fit, the true value of the data may appear in the tails of the histogram of replicated data, while a good fit will tend to show the true data in high-probability regions of the posterior predictive distribution. The Matplot package in PyMC provides an easy way of producing such plots, via the `gof_plot` function.

```python
fig, axes = plt.subplots(1, 4, figsize=(14, 4))

for obs, sim, ax in zip(deaths, deaths_sim['obs_deaths'].T, axes):
    ax.hist(sim, bins=range(7))
    ax.plot(obs+0.5, 1, 'ro')
```

## Exercise: Meta-analysis of beta blocker effectiveness

Carlin (1992) considers a Bayesian approach to meta-analysis, and includes the following examples of 22 trials of beta-blockers to prevent mortality after myocardial infarction.

In a random effects meta-analysis we assume the true effect (on a log-odds scale) $d_i$ in a trial $i$
is drawn from some population distribution. Let $r^C_i$ denote number of events in the control group in trial $i$,
and $r^T_i$ denote events under active treatment in trial $i$. Our model is:

$$\begin{aligned}
r^C_i &\sim \text{Binomial}\left(p^C_i, n^C_i\right) \\
r^T_i &\sim \text{Binomial}\left(p^T_i, n^T_i\right) \\
\text{logit}\left(p^C_i\right) &= \mu_i \\
\text{logit}\left(p^T_i\right) &= \mu_i + \delta_i \\
\delta_i &\sim \text{Normal}(d, t) \\
\mu_i &\sim \text{Normal}(m, s)
\end{aligned}$$

We want to make inferences about the population effect $d$, and the predictive distribution for the effect $\delta_{\text{new}}$ in a new trial. Build a model to estimate these quantities in PyMC, and (1) use convergence diagnostics to check for convergence and (2) use posterior predictive checks to assess goodness-of-fit.

Here are the data:

```python
r_t_obs = [3, 7, 5, 102, 28, 4, 98, 60, 25, 138, 64, 45, 9, 57,
           25, 33, 28, 8, 6, 32, 27, 22]
n_t_obs = [38, 114, 69, 1533, 355, 59, 945, 632, 278,1916, 873, 263,
           291, 858, 154, 207, 251, 151, 174, 209, 391, 680]
r_c_obs = [3, 14, 11, 127, 27, 6, 152, 48, 37, 188, 52, 47, 16, 45,
           31, 38, 12, 6, 3, 40, 43, 39]
n_c_obs = [39, 116, 93, 1520, 365, 52, 939, 471, 282, 1921, 583, 266,
           293, 883, 147, 213, 122, 154, 134, 218, 364, 674]
N = len(n_c_obs)
```

```python
# Write your answer here
```

## References

Gelman, A., & Rubin, D. B. (1992). Inference from iterative simulation using multiple sequences. Statistical Science. A Review Journal of the Institute of Mathematical Statistics, 457–472.

Geweke, J., Berger, J. O., & Dawid, A. P. (1992). Evaluating the accuracy of sampling-based approaches to the calculation of posterior moments. In Bayesian Statistics 4.

Brooks, S. P., Catchpole, E. A., & Morgan, B. J. T. (2000). Bayesian Animal Survival Estimation. Statistical Science. A Review Journal of the Institute of Mathematical Statistics, 15(4), 357–376. doi:10.1214/ss/1177010123

Gelman, A., Meng, X., & Stern, H. (1996). Posterior predicitive assessment of model fitness via realized discrepencies with discussion. Statistica Sinica, 6, 733–807.

Betancourt, M. (2017). A Conceptual Introduction to Hamiltonian Monte Carlo. arXiv.org.


# Spherical Distribution Models

Histology studies show that axons within one white matter bundle are not organized parallel to each other, but rather that their orientations are dispersed around the central bundle direction *(Leergaard et al. 2010)*.
The diffusion signal originating from tissue with dispersed axons can be modeled as the spherical convolution of a spherical distribution $G:\mathbb{S}^2\rightarrow[0,\infty]$ with a convolution kernel $K(\textbf{n})$ with $\textbf{n}\in\mathbb{S}^2$, where the kernel describes the diffusion signal of a single axon micro-environment.
Formally, this convolution can be written as
\begin{equation}
 E(\textbf{n})=\int_{\mathbb{S}^2}F(\textbf{n}-\textbf{g})K(\textbf{g})d\textbf{g}=(\operatorname{F}\,*_{\mathbb{S}^2}\,K)(\textbf{n})
\end{equation}
where $\textbf{g}\in\mathbb{S}^2$ is an integration variable, and we will use the second shorthand for spherical convolution in the rest of this article.
Given an axon bundle along $\textbf{n}_\parallel$, then the "sharpness" of $F$ along $\textbf{n}_\parallel$ describes the "spread", i.e. the dispersion of single axon micro-environments around $\textbf{n}_\parallel$.
The chosen shape of $K(\textbf{n})$ varies between microstructure models and can be Sticks, Cylinders, Zeppelins or combinations thereof.
But once $K(\textbf{n})$ is known, $G$ can be recovered by spherical deconvolution of the signal with the kernel as $G(\textbf{n})=(E\,*^{-1}_{\mathbb{S}^2}\,K)(\textbf{n})$ with $*^{-1}_{\mathbb{S}^2}$ the deconvolution operator, or estimated using parametric distributions. In Dmipy, we do the latter with Watson or Bingham distributions.

## Watson and Bingham Distribution (SD1 and SD2)

To provide a more straightforward, but less general way to quantify axon dispersion, parameteric representations for $F$ such as the Bingham and Watson distribution have been proposed *(Kaden et al. 2007)*.
We show schematic representations of both distributions the figure below.
The Bingham distribution $B(\textbf{n}|\boldsymbol{\mu},\kappa_1,\kappa_2)$ is an antipodally symmetric distribution, centered around direction $\boldsymbol{\mu}$, describing a possibly anisotropic density with concentration parameters $\kappa_1$ and $\kappa_2$ *(Bingham et al. 1974)*.
More formally, its probability density along normalized unit vector $\textbf{n}$ is given as
\begin{equation}
 \operatorname{B}(\textbf{n}|\boldsymbol{\mu},\kappa_1,\kappa_2)=\frac{\exp(\textbf{n}^T\textbf{Bn})}{4\pi\,_1F_1(1/2;3/2;\textbf{B})}\quad\textrm{with}\quad \textbf{B}=\textbf{R}^T\textbf{B}_{\textrm{diag}}\textbf{R}
\end{equation}

```python
from IPython.display import Image
Image("dispersion_distributions.png", width=600)
```

with $_1F_1$ the confluent hypergeometric function, $\textbf{R}$ a rotation matrix that aligns the distribution with $\boldsymbol{\mu}$ and $\textbf{B}_{\textrm{diag}}=\textrm{Diag}(\kappa_1,\kappa_2,0)$. Note that concentration parameters $\kappa_1,\kappa_2$ are inversely related to dispersion. The Watson distribution $W(\textbf{n}|\boldsymbol{\mu},\kappa)$ is a special case of Bingham when $\kappa=\kappa_1=\kappa_2$, meaning $W(\textbf{n}|\boldsymbol{\mu},\kappa)=B(\textbf{n}|\boldsymbol{\mu},\kappa,\kappa)$.

\begin{equation}
 \operatorname{W}(\textbf{n}|\boldsymbol{\mu},\kappa)=\frac{\exp\left(\kappa(\boldsymbol{\mu}\cdot\textbf{n})^2\right)}{4\pi\,_1F_1(1/2;3/2;\kappa)}
\end{equation}

### Example and Intuition of Watson Distribution

A Watson distribution with concentration $\kappa$ is related to a spherical Gaussian distributions with standard deviation $\sigma_\phi$. To illustrate this, plot a spherical arc of a Watson distribution for different $\kappa$ and fit it to an explicit Gaussian distribution.

```python
import numpy as np
def gauss_(x, amplitude, sigma):
    return amplitude * np.exp(-(x ** 2) / (2 * sigma ** 2))
```

```python
from dipy.core.geometry import sphere2cart
mu = np.r_[0, 0]
theta_range = np.linspace(-np.pi / 2, np.pi / 2, 191)
degree_range = 180 * theta_range / np.pi
x_range, y_range, z_range = sphere2cart(np.ones(191), theta_range, np.zeros(191))
n = np.c_[x_range, y_range, z_range]
```

```python
from dmipy.distributions.distributions import SD1Watson, kappa2odi
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit
import seaborn as sns
%matplotlib inline
sns.set_style("whitegrid")

odi = kappa2odi(np.r_[16, 9, 3])
watson = SD1Watson(mu=[0, 0])
y1 = watson(n, odi=odi[0])
_, sig1 = curve_fit(gauss_, degree_range, y1)[0]
y2 = watson(n, odi=odi[1])
_, sig2 = curve_fit(gauss_, degree_range, y2)[0]
y3 = watson(n, odi=odi[2])
_, sig3 = curve_fit(gauss_, degree_range, y3)[0]

plt.figure(figsize=[7, 4])
plt.plot(degree_range, y1, c='r', label=r'$\kappa=16$')
plt.plot(degree_range, y2, c='g', label=r'$\kappa=9$')
plt.plot(degree_range, y3, c='b', label=r'$\kappa=3$')
plt.xlabel('Deviation from $\mu$ [$\phi$ in degrees]', fontsize=20)
plt.ylabel('Watson Density W(n|$\mu$,$\kappa$)', fontsize=20)
plt.xlim(-60, 60)
plt.legend(fontsize=20, loc='upper left')
plt.text(10, 1.7, r'$\sigma_\phi\approx ' + str(round(sig1, 1)) + '^o$', fontsize=20, color='r')
plt.text(22, .5, r'$\sigma_\phi\approx ' + str(round(sig2, 1)) + '^o$', fontsize=20, color='g')
plt.text(35, .18, r'$\sigma_\phi\approx ' + str(round(sig3, 1)) + '^o$', fontsize=20, color='b');
```

We can see that larger $\kappa$ corresponds to a smaller $\sigma_\phi$. In literature, *(Ronen et al. 2013)* studied axon dispersion in the Corpus Callosum, and found that typical values in $\sigma_\phi$ are between 10 and 25 degrees. Plotting $\kappa$ and Orientation Dispersion Index (ODI) as a function of $\sigma_\phi$, we can get a feeling for what values to expect when fitting Watson distributions to real data.

```python
length = 1000
sigmas = np.zeros(length)
kappas = np.linspace(1., 300, length)
ODIs = kappa2odi(kappas)
for i, odi_ in enumerate(ODIs):
    y = watson(n, mu=mu, odi=odi_)
    _, sigmas[i] = curve_fit(gauss_, degree_range, y)[0]
plt.figure(figsize=[9, 7])
plt.semilogy(sigmas, kappas, label='Concentration ($\kappa$)')
plt.semilogy(sigmas, ODIs, label='ODI=(2/$\pi$)arctan(1/$\kappa$)')
plt.xlabel('Dispersion $\sigma_\phi$ [degrees]', fontsize=20)
plt.ylabel('Dispersion Parameter', fontsize=20)
plt.legend(fontsize=18)
plt.axvline(10, ymin=0, ymax=100, ls='--', color='k')
plt.axvline(25, ymin=0, ymax=100, ls='--', color='k')
plt.text(10.7, 250, 'Corpus Callosum', fontsize=12)
plt.text(12, 115, '[Ronen 2013]', fontsize=12)
plt.text(10.5, 17, r'$\kappa\approx16$', fontsize=18)
plt.text(25.5, 3.9, r'$\kappa\approx3$', fontsize=18)
plt.text(25.5, .09, r'ODI$\approx0.2$', fontsize=18)
plt.text(10.5, .021, r'ODI$\approx0.04$', fontsize=18);
```

As you can see, realistic values (in the Corpus Callosum) for $\kappa$ range between 3 and 16, which corresponds in ODI to values between 0.02 and 0.2.

### Example and Intuition of Bingham Distribution

A Bingham distribution is a generalization of the Watson by allowing a secondary dispersion parameter $\beta$, which enables dispersion anisotropy in secondary orientation $\psi$. To illustrate what this means, we reproduce Fig 3. from (*Tariq et al. 2016)*'s Bingham–NODDI model.

```python
from dmipy.distributions.distributions import SD2Bingham
from scipy.interpolate import griddata

theta_range = np.linspace(0, np.pi / 2., 91)
phi_range = np.linspace(0, 2 * np.pi, 360)
theta_grid, phi_grid = np.meshgrid(theta_range, phi_range)

sampling_points = np.reshape(np.concatenate((np.array([theta_grid]), np.array([phi_grid]))),(2, -1)).T
theta_, phi_ = np.reshape(np.concatenate((np.array([theta_grid]), np.array([phi_grid]))),(2, -1))

x_, y_, z_ = sphere2cart(np.ones_like(phi_), theta_, phi_)
n = np.c_[x_, y_, z_]
bingam = SD2Bingham(mu=mu)

odi = kappa2odi(16)
beta_fraction = [0, 14 / 16., 1.]

data_k16_b0_psi0 = bingam(n, psi=0, odi=odi, beta_fraction=beta_fraction[0])
data_k16_b14_psi0 = bingam(n, psi=0, odi=odi, beta_fraction=beta_fraction[1])
data_k16_b16_psi0 = bingam(n, psi=0, odi=odi, beta_fraction=beta_fraction[2])
data_grid_k16_b0_psi0 = griddata(sampling_points, data_k16_b0_psi0, (theta_grid, phi_grid), method='nearest',fill_value=0)
data_grid_k16_b14_psi0 = griddata(sampling_points, data_k16_b14_psi0, (theta_grid, phi_grid), method='nearest',fill_value=0)
data_grid_k16_b16_psi0 = griddata(sampling_points, data_k16_b16_psi0, (theta_grid, phi_grid), method='nearest',fill_value=0)

data_k16_b0_psi1 = bingam(n, psi=np.pi/3, odi=odi, beta_fraction=beta_fraction[0])
data_k16_b14_psi1 = bingam(n, psi=np.pi/3, odi=odi, beta_fraction=beta_fraction[1])
data_k16_b16_psi1 = bingam(n, psi=np.pi/3, odi=odi, beta_fraction=beta_fraction[2])
data_grid_k16_b0_psi1 = griddata(sampling_points, data_k16_b0_psi1, (theta_grid, phi_grid), method='nearest',fill_value=0)
data_grid_k16_b14_psi1 = griddata(sampling_points, data_k16_b14_psi1, (theta_grid, phi_grid), method='nearest',fill_value=0)
data_grid_k16_b16_psi1 = griddata(sampling_points, data_k16_b16_psi1, (theta_grid, phi_grid), method='nearest',fill_value=0)

data_k16_b0_psi2 = bingam(n, psi=2*np.pi/3, odi=odi, beta_fraction=beta_fraction[0])
data_k16_b14_psi2 = bingam(n, psi=2*np.pi/3, odi=odi, beta_fraction=beta_fraction[1])
data_k16_b16_psi2 = bingam(n, psi=2*np.pi/3, odi=odi, beta_fraction=beta_fraction[2])
data_grid_k16_b0_psi2 = griddata(sampling_points, data_k16_b0_psi2, (theta_grid, phi_grid), method='nearest',fill_value=0)
data_grid_k16_b14_psi2 = griddata(sampling_points, data_k16_b14_psi2, (theta_grid, phi_grid), method='nearest',fill_value=0)
data_grid_k16_b16_psi2 = griddata(sampling_points, data_k16_b16_psi2, (theta_grid, phi_grid), method='nearest',fill_value=0)
```

```python
plt.figure(figsize=[10, 10])
ax = plt.subplot(331, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b0_psi0.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=0')
ax = plt.subplot(332, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b14_psi0.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=14')
ax = plt.subplot(333, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b16_psi0.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=16')

ax = plt.subplot(334, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b0_psi1.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=0')
ax = plt.subplot(335, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b14_psi1.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=14')
ax = plt.subplot(336, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b16_psi1.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=16')

ax = plt.subplot(337, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b0_psi2.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=0')
ax = plt.subplot(338, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b14_psi2.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=14')
ax = plt.subplot(339, projection="polar")
ax.pcolormesh(phi_range, theta_range, data_grid_k16_b16_psi2.T, cmap = 'jet')
ax.set_title(r'$\kappa$=16, $\beta$=16');
```

It can be seen that when $\kappa=\beta$, then the distribution corresponds to a narrow spherical band, but when $\kappa>\beta$ the Bingham distribution corresponds to an anisotropic Gaussian distribution on the sphere.

## References
- Leergaard, Trygve B., et al. "Quantitative histological validation of diffusion MRI fiber orientation distributions in the rat brain." PloS one 5.1 (2010): e8595.
- Kaden, Enrico, Thomas R. Knösche, and Alfred Anwander. "Parametric spherical deconvolution: Inferring anatomical connectivity using diffusion MR imaging." NeuroImage 37.2 (2007): 474-488.
- Bingham, Christopher. "An antipodally symmetric distribution on the sphere." The Annals of Statistics (1974): 1201-1225.
- Ronen, Itamar, et al. "Microstructural organization of axons in the human corpus callosum quantified by diffusion-weighted magnetic resonance spectroscopy of N-acetylaspartate and post-mortem histology." Brain Structure and Function 219.5 (2014): 1773-1785.
- Tariq, Maira, et al. "Bingham–noddi: Mapping anisotropic orientation dispersion of neurites using diffusion mri." NeuroImage 133 (2016): 207-223.


# Gaussian Mixture Model

```{note}
GMM assumes instances were generated from a mixture of several Gaussian distributions.<br/>
GMM uses the EM algorithm, E-step estimates which Gaussian, M-step maximizes parameters.<br/>
We can use BIC and AIC to select the number of distributions.
```

The simplest GMM variant which implemented in *GaussianMixture* must know in advance the number of Gaussian distributions $k$.

GMM assumes instances are generated by two steps:

$$z^{(i)} \sim \mbox{Multinomial}(\phi)$$

$$x^{(i)}|z^{(i)}=j \sim \mathcal{N}(\mu_{j}, \Sigma_{j})$$

## Estimate

The parameters of our model are $\phi$, $\mu$ and $\Sigma$. To estimate them, we can write down the likelihood of our data:

$$
\begin{equation}
\begin{split}
\mathcal{l}(\phi, \mu, \Sigma) &= \sum_{i=1}^{n}\log{p(x^{(i)};\phi,\mu,\Sigma)}\\
&= \sum_{i=1}^{n}\log\sum_{z^{(i)}=1}^{k}p(x^{(i)}|z^{(i)}; \mu,\Sigma)p(z^{(i)}; \phi)
\end{split}
\end{equation}
$$

No closed form solution to this MLE problem, while if we knew what  $z^{(i)}$'s were, the problem would have been easy.

The EM algorithm iteratively estimates $z^{(i)}$ and maximizes parameters:

1. the E-step tries to estimate $z^{(i)}$, more precisely:

$$
w_{j}^{(i)} := p(z^{(i)}=j|x^{(i)}; \phi,\mu,\Sigma)
$$

2. the M-step maximizes the parameters of based on our guesses:

$$
\phi_{j} = \frac{1}{n}\sum_{i=1}^{n}w_{j}^{(i)}
$$
$$
\mu_{j} = \frac{\sum_{i=1}^{n}w_{j}^{(i)}x^{(i)}}{\sum_{i=1}^{n}w_{j}^{(i)}}
$$
$$
\Sigma_{j} = \frac{\sum_{i=1}^{n}w_{j}^{(i)}(x^{(i)} - \mu_{j})(x^{(i)} - \mu_{j})^{T}}{\sum_{i=1}^{n}w_{j}^{(i)}}
$$

```{image} ../images/gmm.png
:alt: gmm
:width: 500px
:align: center
```

## Examples

```python
import numpy as np
from sklearn.mixture import GaussianMixture

X = np.array([[1, 2], [1, 4], [1, 0], [10, 2], [10, 4], [10, 0]])
# need to assign n_components
gm = GaussianMixture(n_components=2, random_state=0).fit(X)
```

```python
# phi
gm.weights_
```

    array([0.5, 0.5])

```python
# mu
gm.means_
```

    array([[10.,  2.],
           [ 1.,  2.]])

```python
# Sigma
gm.covariances_
```

    array([[[1.00000000e-06, 1.15699599e-29],
            [1.15699599e-29, 2.66666767e+00]],

           [[1.00000000e-06, 1.24902977e-30],
            [1.24902977e-30, 2.66666767e+00]]])

### predict

```python
X_predict = np.array([[0, 1], [11, 5]])
gm.predict(X_predict)
```

    array([1, 0])

```python
gm.predict_proba(X_predict)
```

    array([[0., 1.],
           [1., 0.]])

### sample

```python
# as a generative model, we can sample new instances from it
X_new, y_new = gm.sample(7)
X_new, y_new
```

    (array([[ 9.99844709,  3.21101447],
            [ 9.99866645, -1.70416514],
            [ 9.99803008,  0.62383712],
            [ 1.00050588,  4.06756329],
            [ 0.99891919,  6.15629625],
            [ 0.99942086,  2.79087858],
            [ 0.9985898 ,  1.70347684]]),
     array([0, 0, 0, 1, 1, 1, 1]))

```python
# log of probability density function(PDF)
gm.score_samples(X_new)
```

    array([2.40556655, 0.42448816, 1.59092314, 2.95683262, 0.06321552,
           3.60133524, 2.87549155])

## Selecting the Number of Distributions

GMM use metrics such as *Bayesian information criterion* (BIC) or the *Akaike information criterion* (AIC):

$$
\begin{equation}
\begin{split}
BIC &= \log(n)p - 2\log(\hat{L}) \\
AIC &= 2p - 2\log(\hat{L})
\end{split}
\end{equation}
$$

where $n$ is the number of instances, $p$ is the number of parameters, $\hat{L}$ is the likelihood.

Both BIC and AIC penalize models that have more parameters to learn and reward models that fit the data well.

```python
"""
select k that minimize bic or the elbow of bic.
"""
gm.bic(X)
```

    -20.92644270457478

```python
gm.aic(X)
```

    -18.635796866083382


# Announcements
- Problem Set 6 will be on finite difference methods, due Nov 2 (to be posted on D2L).
- __Outlook__: Next week, we'll move on to Monte Carlo Markov Chain methods.

*Note:  The presentation below largely follows part I in "Finite Difference Methods for Ordinary and Partial Differential Equations" by LeVeque (SIAM, 2007).*

# The Heat Equation
Numerical solutions to partial differential equations (PDEs) are a large subfield of applied math, and we won't be able to cover a full survey in this course. Today, we'll solve the heat equation
$$
\partial_{t}u(\vec x,t) = \alpha \vec \nabla^2 u(\vec x,t)\,,
$$
which describes how a quantity $u$ (such as heat) diffuses through a medium, and
with $\alpha$ a positive constant, called _diffusivity_ of the medium. The heat equation is among the most widely studied PDEs, and a prototype for parabolic PDEs.

Today we will numerically solve an example in 1D, applying the finite difference method from Lecture 23 and 24 for the spatial solution, combined with a forward Euler step for the time solution.

## Example Problem
Let consider the 1D heat equation on a wire with extend $x\in[0,1]$,
\begin{equation}
    \partial_{t}u = \partial^2_{x}u , \quad 0 < x < 1, \quad t > 0 \\
\end{equation}
with boundary conditions
\begin{equation}
    u(0,t) = 0, \quad {u}(1,t) = 0\,,\\
\end{equation}
which correspond to endpoints held at constant temperature, and inital temperature distribution
\begin{equation}
    u(x, 0) = 10\sin(\pi x)\,.
\end{equation}

# Discretization
Lecture 23 introduced the _Finite Difference_ method and we used it to solve a boundary value problem with Dirichlet conditions as a linear system.

Let's use the subscript $i$ to indicate the spatial discretization, and subscript $J$ for the time discretization, so that $U_{i,J} = u(x_i,t_J)$.
Today we will incorporate the Dirichlet conditions as ghost cells, $U_{0,J}$ and $U_{N+1,J}$, corresponding to the boundary points.

Then the spatial discretization (at one slice in time) becomes
$$
    A = \frac{1}{\Delta x^2} \begin{bmatrix}
        \Delta x^2 & 0  \\
        1 & -2 & 1 \\
          &  1 & -2 & 1 \\
          &    & \ddots & \ddots & \ddots \\
          &    &        &      1 &     -2 & 1 \\
          &    &        &        &      1 & -2 & 1 \\
          &    &        &        &        &  0 & \Delta x^2
    \end{bmatrix} \quad \quad U_J = \begin{bmatrix}
        U_{0,J} \\ U_{1,J} \\ \vdots \\ U_{N,J} \\ U_{N+1,J}
    \end{bmatrix} \quad \quad b_J = \begin{bmatrix}
        u(0,t_J)=0 \\ f(x_1,t_J) \\ \vdots \\ f(x_{N},t_J) \\ u(1,t_J) =0\,.
    \end{bmatrix}
$$
_Note that $A$ is time independent!_

The next step is to discretize in time
$$
\frac{U_{i,J+1} -U_{i,J}}{\Delta t} = U^{\prime\prime}_{i,J} = \frac{U_{i-1,J}-2U_{i,J}+U_{i+1,j}}{\Delta x^2}\,,
$$

 **Solve today's Example Problem via the construction of a linear system of equations and time stepping.**

 Your expression for $U_{i,J+1}$ in terms of elements of $U_J$:

$$U_{i,J+1} = U_{i,J} +\frac{\Delta t}{\Delta x^2}\left(U_{i-1,J}-2U_{i,J}+U_{i+1,j}\right)$$

_The code below is adapted from Lecture 23. The missing parts need to be adapted to today's problem, but you may find looking back at Lecture 23 helpful._

```python
import numpy
import matplotlib.pyplot as plt
# Problem setup
f_t0 = lambda x: 10.*numpy.sin(numpy.pi*x)
#spatial discretization
a = 0.0
b = 1.0
u_a = 0.0
u_b = 0.0

# Spatial Discretization
N_x = 10
#keep boundary points
x = numpy.linspace(a, b, N_x + 2)
delta_x = (b - a) / (N_x + 1)

#time discretization
t_0 = 0
t_final = 0.3
N_t = 100
delta_t = (t_final-t_0)/(N_t+1)

# Construct matrix A
A = numpy.zeros((N_x + 2, N_x + 2))
diagonal = numpy.ones(N_x + 2) / delta_x**2
A += numpy.diag(diagonal * -2.0, 0)
A += numpy.diag(diagonal[:-1], 1)
A += numpy.diag(diagonal[:-1], -1)

# Now, add boundary conditions for ghost cells to A
#left boundary
A[0,1] = 0
A[0,0] = 1
A[-1,-2] = 0
A[-1,-1] =1
# Construct RHS - without boundary conditions
b = f_t0(x)
# add boundary conditions to b
b[0] = u_a
b[-1] = u_b

#U[J,:] will contain solution at time J
U = numpy.empty((N_t+2,N_x + 2))
#spatial solution
U[0,:] = numpy.linalg.solve(A, b)

#now do time iteration
for J in range(0,N_t+1):
    U[J,0] = 0
    U[J,-1] = 0
    #boundary conditions
    for i in range (1,N_x+1):
        U[J+1,i] = U[J,i] +delta_t/delta_x**2*(U[J,i-1]-2*U[J,i]+U[J,i+1])

for i in range(0,N_t+2,10):
    plt.plot(x,U[i,:],label ="t=%.2f"%(i*delta_t))
plt.legend()
```

```python

```


# Polynomials in SymPy & NumPy

## SymPy

### Refererences

[SymPy documentation](https://docs.sympy.org/latest/index.html)

* [Polynomial Manipulation](https://docs.sympy.org/latest/modules/polys/index.html)

```python
from sympy import *
```

Initialize the pretty-printer depending on the environment.

```python
init_printing(use_unicode=False, wrap_line=False)
```

Create some *symbols* (sympy.core.symbol.Symbol).

```python
x, y, z = symbols('x,y,z')
```

### SymPy Polynomials

```python
e1 = (x + y)*(y - 2*z)
e1
```

```python
e1.as_poly()
```

```python
e2 = (3*x/2 + y)*(z - 1)
e2
```

```python
e2.as_poly()
```

```python
e3 = (x + 2*pi)*y
e3
```

```python
e3.as_poly()
```

```python
e3.as_poly(x, y)
```

```python
e3.as_poly(domain=ZZ[pi])
```

```python
e4 = x + sqrt(2)
e4
```

```python
e4.as_poly()
```

```python
e4.as_poly(extension=True)
```

```python

```

```python

```

```python

```


# Poiseuille flow

This notebook will use Dedalus to create a minimal working example of the solution to planar Poiseuille flow
\begin{align}
\partial_{x}^2 u - \frac{1}{\varepsilon^2}\Gamma u &= -2
\end{align}
where the penalty mask $\Gamma$ can be chosen by shifting and rescaling optimal masks $\Gamma^*$ for the inner problem
\begin{align}
\Gamma(x) &= \Gamma^*\left(\frac{x - \varepsilon \ell}{\varepsilon \delta}\right)
\end{align}
and the physical solution $u$ satisfies the boundary conditions
\begin{align}
x &= 1 & u &= 0\\
x &\to -\infty & \partial_x u &\to 0
\end{align}

# Imports

```python
import numpy as np
import matplotlib.pyplot as plt
import dedalus.public as de

from matplotlib import rc
rc('font',**{'family':'serif','serif':['Computer Modern Roman']})
rc('text', usetex=True)
```

# Optimized discontinuous mask

We will solve the problem numerically for a standard discontinuous mask, with finite boundary conditions at distance -5. We enforce Robin boundary conditions within the solid to match onto the analytical exponential behaviour.

One can change the mask function to calculate different solutions.
Masks chosen according to the optimal criterion will achieve $\mathcal{O}(\varepsilon^2)$ error from the reference solution $u_0 = x(1-x)$.

```python
# Calculate penalized solution
ϵ = 0.1
Nx = [128,128]

xb0 = de.Chebyshev('x0',Nx[0],interval=(-5,0))
xb1 = de.Chebyshev('x1',Nx[1],interval=(0,1))
xbasis = de.Compound('x',[xb0,xb1])
domain = de.Domain([xbasis], grid_dtype=np.float64)
x = xbasis.grid(*domain.dealias)

Γ = domain.new_field(name='Γ',scales=domain.dealias)
Γ['g'] = 1.*(x<ϵ)

poiseuille = de.LBVP(domain, variables=['u','ux'])
poiseuille.meta[:]['x']['dirichlet'] = True
poiseuille.parameters['Γ'] = Γ
poiseuille.parameters['ε'] = ϵ
poiseuille.add_equation("dx(ux) - (Γ/ε**2)*u = -2")
poiseuille.add_equation("ux - dx(u) = 0")
poiseuille.add_bc("left(ux) = 0")
poiseuille.add_bc("right(u) = 0")

poiseuille_solver = poiseuille.build_solver()
poiseuille_solver.solve()

u, ux = poiseuille_solver.state['u'], poiseuille_solver.state['ux']
```

```python
# Plot penalized and reference solutions
fig, ax = plt.subplots()
ax.plot(x,u['g'],color='C1',label='Penalized')
ax.plot(x[x>0],x[x>0]*(1-x[x>0]),'k--',label='Reference')
ax.fill_between(x[x<0],0,10,color='lightgray')
ax.set(aspect=1,xlim=[-1,1],ylim=[0,1],xlabel='$x$',ylabel='$u$')
ax.spines['top'].set_visible(False)
ax.spines['right'].set_visible(False)
ax.legend()
```


```python
import numpy as np
from sympy import *
init_printing(use_latex='mathjax')
```

```python
def get_jacobian(expressions, symbols):
    rows = len(expressions)
    columns = len(symbols)

#     print("Expressions:")
#     for expression in expressions:
#         display(expression)

    results = [[0 for x in range(columns)] for y in range(rows)]
    for row, expression in enumerate(expressions):
        for column, symbol in enumerate(symbols):
#             print('Row %d, column %d, expression: %s, symbol: %s' % (row, column, expression, symbol))
            df = diff(expression, symbol)
#             print("DF: %s" % df)
            results[row][column] = df
    return results
```

```python
def get_hessians(jacobian, symbols):
    jacobian = np.array(jacobian)
    rows, columns = jacobian.shape
    hessians = [[[None for z in range(len(symbols))] for x in range(columns)] for y in range(rows)]
    for row in range(rows):
        for column in range(columns):
            for index, symbol in enumerate(symbols):
                df = diff(jacobian[row, column], symbol)
                hessians[row][column][index] = df

    return hessians
```

```python
def evaluate_at(exprs, symbols):
    results = []
    def process_list(exprs, symbols):
        r = []
        for expr in exprs:
            if isinstance(expr, list):
                r.append(process_list(expr, symbols))
            else:
                r.append(expr.subs(symbols))
        return r

    results = []
    if isinstance(exprs, list):
        results.append(process_list(exprs, symbols))
    else:
        results.append(exprs.subs(symbols))
    return results
```

```python
x, y,z = symbols('x y z')
s = [x, y, z]
expr = x ** 2 * cos(y) + exp(z) * sin(y)
j = get_jacobian([expr], s)
r = evaluate_at(j, {x: pi, y: pi, z: 1})
r
```

    Expressions:

$$x^{2} \cos{\left (y \right )} + e^{z} \sin{\left (y \right )}$$

$$\left [ \left [ \left [ - 2 \pi, \quad - e, \quad 0\right ]\right ]\right ]$$

```python
x, y = symbols('x y')
s = [x, y]
j = get_jacobian([
    x**2 * y - cos(x) * sin(y),
    exp(x + y)
], s)
r = evaluate_at(j, {x: 0, y: pi})
r
```

    Expressions:

$$x^{2} y - \sin{\left (y \right )} \cos{\left (x \right )}$$

$$e^{x + y}$$

$$\left [ \left [ \left [ 0, \quad 1\right ], \quad \left [ e^{\pi}, \quad e^{\pi}\right ]\right ]\right ]$$

```python
x, y = symbols('x y')
s = [x, y]
j = get_jacobian([
    x ** 3 * cos(y) - x * sin(y)
], s)
h = get_hessians(j, s)
h
```

    Expressions:

$$x^{3} \cos{\left (y \right )} - x \sin{\left (y \right )}$$

$$\left [ \left [ \left [ 6 x \cos{\left (y \right )}, \quad - 3 x^{2} \sin{\left (y \right )} - \cos{\left (y \right )}\right ], \quad \left [ - 3 x^{2} \sin{\left (y \right )} - \cos{\left (y \right )}, \quad - x^{3} \cos{\left (y \right )} + x \sin{\left (y \right )}\right ]\right ]\right ]$$

```python
x, y, z = symbols('x y z')
s = [x, y, z]
j = get_jacobian([
    x * y + sin(y) * sin(z) + z ** 3 * exp(x)
], s)
h = get_hessians(j, s)
h
```

    Expressions:

$$x y + z^{3} e^{x} + \sin{\left (y \right )} \sin{\left (z \right )}$$

$$\left [ \left [ \left [ z^{3} e^{x}, \quad 1, \quad 3 z^{2} e^{x}\right ], \quad \left [ 1, \quad - \sin{\left (y \right )} \sin{\left (z \right )}, \quad \cos{\left (y \right )} \cos{\left (z \right )}\right ], \quad \left [ 3 z^{2} e^{x}, \quad \cos{\left (y \right )} \cos{\left (z \right )}, \quad 6 z e^{x} - \sin{\left (y \right )} \sin{\left (z \right )}\right ]\right ]\right ]$$

```python
x, y, z = symbols('x y z')
s = [x, y, z]
j = get_jacobian([
    x * y *  cos(z) - sin(x) * exp(y) * z ** 3
], s)
h = get_hessians(j, s)
evaluate_at(h, {x: 0, y: 0, z: 0 })
```

    Expressions:

$$x y \cos{\left (z \right )} - z^{3} e^{y} \sin{\left (x \right )}$$

$$\left [ \left [ \left [ \left [ 0, \quad 1, \quad 0\right ], \quad \left [ 1, \quad 0, \quad 0\right ], \quad \left [ 0, \quad 0, \quad 0\right ]\right ]\right ]\right ]$$

```python

```

Certification 2/Week2.5 - Jacobian and Hessians - final.ipynb


# Spectral Expansions for Asian Options

## Distributional properties

We assume that  under the risk-neutral probability measure, the underlying asset price follows a geometric Brownian motion process:
$$S_t = S_0 e^{\sigma B_t + (r-q-\sigma^2 /2)t}$$

Let $\mathscr{A}_t$ be the continuously sampled arithmetic average price:
$$\mathscr{A}_t=\frac{1}{t}\int_0^t S_u du$$

From the general form of option pricing formula $e^{-rT}\mathbb{E}[(K-\mathscr{A}_T)^+]$, make the following derivation:
$$\left(K-\frac{1}{T}\int^T_0S_udu\right)^+=\frac{4S_0}{\sigma^2T}\left(k-\frac{\sigma^2}{4S_0}\int^{\tau}_0S_0e^{\sigma B_u+(r-q-\sigma^2/2)u}d\left(\frac{\sigma^2 u}{4}\right)\right)^+\\
=\frac{4S_0}{\sigma^2T}\left(k-\int^{\tau}_0e^{2(B_t+\nu t)}dt\right)$$

Define *Brownian exponential functional* $A_{\tau}^{(\nu)}$:
$$A_{\tau}^{(\nu)}:=\int^{\tau}_0e^{2(B_t+\nu t)}dt$$

From the invariance by time reversal of the distribution of a Levy process, the following process $X_t$ has an identity in law, $X_t \overset{(law)}{=}A_{t}^{(\nu)} $:
$$X_t = \int_0^T \exp{\left(2(B_t-B_u)+2\nu (t-u)\right)}du, X_0=0$$

From Ito's fomula, we express the SDE of $X_t$ as:
$$dX_t = \left[2(\nu+1)X_t+1\right]+2X_tdB_t$$
This $X_t$ is a one-dimensional diffusion process on $[0,\infty)$ started at $X_0 = 0$ and with the infinitesimal diffusion $a(x)=2x$, infinitesimal drift $b(x)=2(\nu+1)x+1$, and infinitesimal generator
$$
\begin{align}
(\mathscr{G}f)(x)&=\frac{1}{2}a^2(x)f''(x)+b(x)f'(x)\\&=2x^2f''(x)+[2(\nu+1)+1]f'(x)
\end{align}
$$
This diffusion has scale and speed densities:
$$\mathscr{s}(x)=\exp{\left(-\int\frac{2b(x)}{a^2(x)}dx\right)}=x^{-\nu-1}e^{1/2x}$$
$$\mathscr{m}(x)=\frac{2}{a^2(x)\mathscr{s}(x)}=\frac{1}{2}x^{\nu-1}e^{-1/2x}$$

In addition, this paper also consider *up-and-out puts* on the diffusion X. For $b>k$, consider functions
$$P^{(\nu)}(k,\tau):=\mathbb{E}\left[(k-X_t)^+\right],P_b^{(\nu)}(k,\tau):=\mathbb{E}\left[1_{\{\mathscr{T}_b>\tau\}}(k-X_\tau)^+\right]$$
where $\mathscr{T}_b:=inf\{t\ge0:X_t=b\}$ is the first hitting time of $b$ and $1_{\{\mathscr{T}_b>\tau\}}=1$ if $\mathscr{T}_b>\tau$. Their strategy is to first calculate the function $P_b^{(\nu)}(k,\tau)$ and then take the limit $b\uparrow\infty$ to recover the function $P^{(\nu)}(k,\tau)$

## Whittaker functions

The Whittaker functions are solutions to the ODE
$$\frac{d^2w}{dz^2}+\left(-\frac{1}{4}+\frac{\kappa}{z}+\frac{1/4-\mu^2}{z^2}\right)w=0$$

There are two linearly independent solutions to the ODE:
$$M_{\kappa,\mu}=\exp{(-z/2)}z^{\mu+1/2}M\left(\mu-\kappa+\frac{1}{2},1+2\mu;z\right),$$
$$W_{\kappa,\mu}=\exp{(-z/2)}z^{\mu+1/2}U\left(\mu-\kappa+\frac{1}{2},1+2\mu;z\right)$$
where
$$M(a,b,z)=\sum_{k=0}^\infty\frac{(a)_k}{(b)_k}\frac{z^k}{k!}$$
$$U(a,b,z)=\frac{\pi}{\sin{b\pi}}\left(\frac{M(a,b,z)}{\Gamma(1+a-b)\Gamma(b)}-z^{1-b}\frac{M(1+a-b,2-b,z)}{\Gamma(a)\Gamma(2-b)}\right)$$

## Up-and-out puts price derivation

From the last section, $P_b^{(\nu)}(k,\tau)$ is what we care the most, we need the following integral to calculate:
$$P_b^{(\nu)}(k,\tau)=\int_0^k (k-y)p_b(\tau;0,y)\mathscr{m}(y)dy$$
where $p_b(t;x,y)$ is the transition probability density w.r.t. the speed measure $\mathscr{m}(y)dy$ of the infinitesimal generator $(\mathscr{G}f)(x)$ started at $x\in[0,b)$ and killed at $b$.

This prob density admit a spectral representation
$$p_b(t;x,y)=\sum_{n=1}^\infty e^{-\lambda_n \tau}\varphi_n(x)\varphi_n(y)$$
where $\{\lambda_n,\varphi_n\}_{n=1}^\infty$ are eigenvalues and (normalized) eigenfunctions of the Strum-Liouville problem:
$$-(\mathscr{G}u)(x)\equiv \lambda u(x)$$
with the boundary conditions at 0 and $b$:
$$\underset{x\downarrow0}{lim}\frac{u'(x)}{\mathscr{s}(x)}=0, u(b)=0$$

This linear ODE turns out to be the solution of *Whittaker's form of the confluent hypergeometric equation*, with the solution:
$$\psi(x,\lambda)=(2x)^{(1-\nu)/2}e^{1/4x}W_{(1-\nu)/2,(1/2)\sqrt{\nu^2-2\lambda}}\left(\frac{1}{2x}\right)$$

From Theorem 4.1 in Kent (1980), for fixed $x > 0$ the zeros of $\psi(x,\lambda)$ are simple and positive, it can be written as a canonical product
$$\psi(x,\lambda)=\prod_{n=1}^\infty\left(1-\frac{\lambda}{\lambda_{n,x}}\right)$$
by finding root $\psi(x,\lambda)=0$, we solve the eigenvalues $\lambda_n=\lambda_{n,b}$ as the zeros of $\psi(b,\lambda)$. Consider the two cases: $0<\lambda\le\nu^2/2$ and $\lambda>\nu^2/2$

### Approximation for $0<\lambda\le\nu^2/2$

By setting $\lambda = (\nu^2-q^2)/2$, we are equivalently solving $q\in[0,|\nu|)$. To get precise numerical values of $q_{n,b}$, we need to find the roots of $W_{(1-\nu)/2,q/2}\left(\frac{1}{2x}\right)$ numerically. However, for large $b$ we can get estimates by using the Whittaker function
asymptotics for $\mu>0$ and $z > 0$:
$$W_{\kappa,\mu}(z)\sim\frac{\Gamma(2\mu)}{\Gamma(1/2+\mu-\kappa)}z^{1/2-\mu}e^{-z/2}$$

This gives
$$\psi(b,(\nu^2-q^2)/2)\sim\frac{\Gamma(q)}{\Gamma((\nu+q)/2)}\left(\frac{1}{2b}\right)^{(\nu-q)/2}$$

For $\nu<0$,  the reciprocal of the gamma function $1/\Gamma((\nu+q)/2)$ has zeros $(\nu+q)/2=-n+1, n = 1,...,[|\nu|/2]+1$. Thus, for $\nu < 0$ and *large enough* b, the total number of zeros in $[0,|\nu|)$ is equal to $N_\nu(b)=[|\nu|/2]+1$ and
$$q_{n,b}=(|\nu|-2n+2), n=1,...,[|\nu|/2]+1$$

For $\nu>0$, $N_\nu(b)\equiv0, \forall b>0$

Then the non-normalized eigenfunctions are:
$$\psi(x,\lambda_{n,b})=(2x)^{(1-\nu)/2}e^{1/4x}W_{(1-\nu)/2,q_{n,b}/2}\left(\frac{1}{2x}\right)$$

### Approximation for $\lambda>\nu^2/2$

Setting $\lambda = (\nu^2+p^2)/2$. We can obtain estimates that improve with increasing n by using the following estimate of the Whittaker function with purely imaginary second index $\mu=i\rho, \rho >0, \kappa\in\mathbb{R}$ and fixed $z>0$:
$$W_{\kappa,i\rho}=\sqrt{2z}e^{\pi\rho/2}\rho^{\kappa-1/2}\cos{\left(\rho\ln{\left(\frac{z}{4\rho}\right)}+\rho-\left(\kappa-\frac{1}{2}\right)\frac{\pi}{2}\right)}\times[1+O(\rho^{-1})]$$

In this case, the root is related to the cosine function, let
$$\frac{p}{2}\left(-\ln{(4bp)}+1\right)+\frac{1}{4}\pi\nu=\frac{\pi}{2}+n\pi $$
then the solution of following equation gives the estimate of $p_{n,b}$
$$p_{n,b}[\ln{(4bp_{n,b})}-1]=2\pi\left(n+\frac{\nu}{4}-\frac{1}{2}\right)$$

Then the non-normalized eigenfunctions are:
$$\psi(x,\lambda_{N_\nu(b)+n,b})=(2x)^{(1-\nu)/2}e^{1/4x}W_{(1-\nu)/2,ip_{n,b}/2}\left(\frac{1}{2x}\right)$$

### Normalization of the eigenfunctions

The normalizing factor is related to the other linearly independent solution to the ODE. By calculating the Wronskian, define
$$\xi_{n,b}^{(\nu)}:=\frac{\partial}{\partial p}\left.W_{(1-\nu)/2,ip_{n,b}/2}\left(\frac{1}{2b}\right)\right|_{p=p_{n,b}}$$
$$\eta_{n,b}^{(\nu)}:=-\frac{\partial}{\partial q}\left.W_{(1-\nu)/2,q_{n,b}/2}\left(\frac{1}{2b}\right)\right|_{q=q_{n,b}}$$

The norms of the non-normalized eigenfunctions are:

For $\lambda\le\nu^2/2,\nu<0$ and $N_\nu>0$, the norms are given by:
$$\frac{1}{||\psi(\bullet,\lambda_{n,b})||^2}=2^\nu q_{n,b}\frac{\Gamma(\tfrac{1}{2}(\nu+q_{n,b}))}{\eta_{n,b}^{(\nu)}\Gamma(1+q_{n,b})}M_{(1-\nu)/2,q_{n,b}/2}\left(\frac{1}{2b}\right), n=1,...,N_\nu(b)$$

For $\lambda\in(\nu^2/2,\infty)$, the norms are given by:
$$\frac{1}{||\psi(\bullet,\lambda_{N_\nu(b)+n,b})||^2}=2^\nu p_{n,b}\frac{\Gamma(\tfrac{1}{2}(\nu+ip_{n,b}))}{\xi_{n,b}^{(\nu)}\Gamma(1+ip_{n,b})}M_{(1-\nu)/2,ip_{n,b}/2}\left(\frac{1}{2b}\right),n=1,2,...$$

### Spectral representation for $P_b^{(\nu)}(k,\tau)$

The transition probability density takes the form
$$p_b(t;x,y)=\sum_{n=1}^\infty e^{-t\lambda_{n,b}}\frac{\psi(x,\lambda_{n,b})\psi(y,\lambda_{n,b})}{||\psi(\bullet,\lambda_{n,b})||^2}$$

Define $c(\lambda)$ as
$$
\begin{align}
c(\lambda)&=\int_0^k(k-y)\psi(y,\lambda)\mathscr{m}(y)dy\\
&=2^{-(1+\nu)/2}k^{(\nu+3)/2}e^{-1/4k}W_{-(\nu+3)/2,\sqrt(\nu^2-2\lambda)/2}\left(\frac{1}{2k}\right)
\end{align}$$

Then the transformed payoff $P_b^{(\nu)}(k,\tau)$ have the spectral representation:
$$P_b^{(\nu)}(k,\tau)=\sum_{n=1}^\infty e^{-\tau\lambda_{n,b}}\frac{c(\lambda_{n,b})}{||\psi(\bullet,\lambda_{n,b})||^2}$$

## Proposition 1

The function $P_b^{(\nu)}(k,\tau)$ is given by the following series:
$$\begin{align}
P_b^{(\nu)}(k,\tau)\\
=&\sum_{n=1}^\infty e^{-\tau(\nu^2+p^2_{n,b})/2}p_{n,b}\frac{\Gamma(\tfrac{1}{2}(\nu+ip_{n,b}))}{4\xi_{n,b}^{(\nu)}\Gamma(1+ip_{n,b})}\\
&\times(2k)^{(\nu+3)/2}e^{-1/4k}W_{-(\nu+3)/2,(ip_{n,b})/2}\left(\frac{1}{2k}\right)M_{(1-\nu)/2,ip_{n,b}/2}\left(\frac{1}{2b}\right)\\
&+\sum_{n=1}^{N_\nu(b)} e^{-\tau(\nu^2-q^2_{n,b})/2}q_{n,b}\frac{\Gamma(\tfrac{1}{2}(\nu+q_{n,b}))}{4\eta_{n,b}^{(\nu)}\Gamma(1+q_{n,b})}\\
&\times(2k)^{(\nu+3)/2}e^{-1/4k}W_{-(\nu+3)/2,q_{n,b}/2}\left(\frac{1}{2k}\right)M_{(1-\nu)/2,q_{n,b}/2}\left(\frac{1}{2b}\right)
\end{align}$$

The final barrier put price can be solved by:
$$e^{-rT}\mathbb{E}[(K-\mathscr{A}_T)^+]=e^{-rT}\left(\frac{4S_0}{\sigma^2 T}\right)P_b^{(\nu)}(k,\tau)$$

By put-call-parity, we have the asian call price to compare:
$$e^{-rT}\mathbb{E}[(\mathscr{A}_T-K)^+]=e^{-rT}\mathbb{E}[(K-\mathscr{A}_T)^+]+\frac{e^{-qT}-e^{-rT}}{(r-q)T}S_0 - e^{-rT}K$$

```python
import ExactAsian as ea
```

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=2.0,
                            vol=0.5,
                            texp=2,
                            intr=0.05,
                            divr=0.0,
                            b=1.0,
                            call=True)
c7 = c.exact_asian_price(n_eig=13)
c7
```

    0.3541364096758034

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=2.1,
                            vol=0.5,
                            texp=1,
                            intr=0.05,
                            divr=0.0,
                            b=1.0,
                            call=True)
c6 = c.exact_asian_price(n_eig=23)
c6
```

    0.3069431156763319

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=2.0,
                            vol=0.5,
                            texp=1,
                            intr=0.05,
                            divr=0.0,
                            b=1.0,
                            call=True)
c5 = c.exact_asian_price(n_eig=23)
c5
```

    0.24734959809175927

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=1.9,
                            vol=0.5,
                            texp=1,
                            intr=0.05,
                            divr=0.0,
                            b=1.0,
                            call=True)
c4 = c.exact_asian_price(n_eig=24)
c4
```

    0.19437963526095092

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=2.0,
                            vol=0.25,
                            texp=2,
                            intr=0.0125,
                            divr=0.0,
                            b=1.0,
                            call=True)
c3 = c.exact_asian_price(n_eig=41)
c3
```

    0.17279198671749574

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=2.0,
                            vol=0.3,
                            texp=1,
                            intr=0.18,
                            divr=0.0,
                            b=1.0,
                            call=True)
c2 = c.exact_asian_price(n_eig=57)
c2
```

    0.21956906427076406

```python
c = ea.BsmAsianLinetsky2004(strike = 2.0,
                            spot=2.0,
                            vol=0.1,
                            texp=1,
                            intr=0.02,
                            divr=0.0,
                            b=1.0,
                            call=True)
c1 = c.exact_asian_price(n_eig=500)
c1 # this converges but gives out an elusive value
```

    -256.22228220272973

We summarize the numerical results as follows:

| Case | Our Replication       | The Paper    |
| ---- | --------------------- | ------------ |
| 1    | ?                     | 0.0559860415 |
| 2    | 0.21*956* | 0.2183875466 |
| 3    | 0.172*791* | 0.1722687410 |
| 4    | 0.19*437* | 0.1931737903 |
| 5    | 0.24*734* | 0.2464156905 |
| 6    | 0.306*943* | 0.3062203648 |
| 7    | 0.35*413*  | 0.3500952190 |

## Analysis

We would like to make some analysis based on our codes and results, hopefully it will be helpful for implementation and further improvement.

### Accuracy

In our `ExactAsian.py` file, we tear the series summation into 5 parts: `p1`, `p2`, `const`, `w1` and `m1` with following expression (for the case with pure imaginary second index in the Whittaker function, the infinite summation part of Equation (15) in the paper)

$$p_1 = e^{-(\nu^2+p_{n,b}^2)\tau/2}$$

$$p_2 = \frac{p_{n,b}\Gamma((\nu+ip_{n,b})/2)}{4\xi_{n,b}\Gamma(1+ip_{n,b})}$$

$$const = 2k^{(\nu+3)/2}\times e^{-1/(4k)}$$

$$w_1 = W_{-(\nu+3)/2,ip_{n,b}/2}\left(\frac{1}{2k}\right)$$

$$m_1 = M_{(1-\nu)/2,ip_{n,b}/2}\left(\frac{1}{2b}\right)$$

Comparing the results given by the same function in `Mathematica`, `p2` mostly contributes the error. Further looking into `p2`, we think there are some possible aspects to concern:

#### Eigenvalues

We find the exactly same values for $p_{n,b}$ comparing to Table 2 in the paper.

```python
c.find_zeros_imag(n=11,nu=3)
```

    array([ 4.2721541 ,  6.33345718,  8.19966721,  9.94974532, 11.61929302,
           13.22821883, 14.7890835 , 16.31045757, 17.79852112, 19.2579178 ])

```python
c.find_zeros_imag(n=6,nu=-0.6)
```

    array([ 2.01930274,  4.49156487,  6.52694892,  8.3791226 , 10.11991323])

So it's not likely that the error is from $p_{n,b}$.

#### Gamma Function

We also compared the results of Gamma function in `mpmath` package and `Mathematica`, the results are the same.

```python
import mpmath as m
```

```python
p_vec = c.find_zeros_imag(n=11,nu=3)
```

```python
nu = 3
m.gamma(complex(nu/2,p_vec[0]/2))
```

    mpc(real='0.12726000595038786', imag='0.14363804118134285')

```python
m.gamma(complex(nu/2,p_vec[9]/2))
```

    mpc(real='2.7518238561063786e-6', imag='5.914875567216349e-6')

`Mathematica` give the result `0.12726 + 0.143638*I` and `2.75182*10^-6 + 5.91488*10^-6*I` for the first and 10th $p_{n,b}$. So it's not likely that the error is from Gamma function.

#### Derivative w.r.t. the second index of Whittaker function

In the `.py` file, we use `scipy.misc.derivative` to numerically calculate the value, while the author used $HypergeometricU^{(0,1,0)}[a,b,z]$ in `Mathematica`, here are the comparison:

```python
c.xi_p(nu=3, eigenval=p_vec[0], b=1)
```

    (-0.010576392192596096+6.0084762587507305e-12j)

```python
c.xi_p(nu=3, eigenval=p_vec[9], b=1)
```

    (1.8960220410634224e-08-1.7625992449015035e-17j)

`Mathematica` gives the result `-0.0105754 + 1.92279*10^-17*I` and `1.89922*10^-8 + 1.18019*10^-23 I` for the first and 10th $p_{n,b}$. The error is more observable in this comparison. We guess if there are closed form formula for the derivative $WhittakerW^{(0,1,0)}[a,b,z]$ in python, we may have a chance to further reduce the deviation.

#### The overall error of `p2`

Let's calculate `p2` and make comparison:

```python
p_vec[0]*m.gamma(complex(nu/2,p_vec[0]/2))/(4*c.xi_p(nu=3, eigenval=p_vec[0], b=1)*m.gamma(complex(1,p_vec[0])))
```

    mpc(real='851.13685258067608', imag='2951.0811552690107')

```python
p_vec[9]*m.gamma(complex(nu/2,p_vec[9]/2))/(4*c.xi_p(nu=3, eigenval=p_vec[9], b=1)*m.gamma(complex(1,p_vec[9])))
```

    mpc(real='1943124015951449.8', imag='704626259404885.13')

`Mathematica` gives the result `851.215 + 2951.35*I` and `1.93985*10^15 + 7.03439*10^14*I` for the first and 10th $p_{n,b}$. The author highligted the remarkable capabilities of `Mathematica` in hadling special functions with arbitrary precision arithmetics.

The error is more observable in this comparison. The results given by `Mathematica` and our code consistently deviates at the third or forth digit both in `p2` and final price. Thus, we believe that a more accurate derivative contributes to a more accurate result.

### Speed

This approach reduces the *path generating* process in MC to a *root finding* problem. We use `sympy.solve` to calculate the exact value.

Convergence is related to the $\tau=\sigma^2 T/4$ parameter. We recommend to use this approach when $\tau$ is large because this parameter controls the numerical convergence of the series: higher order terms are suppressed with $e^{-p_{n,b}^2\tau /2}$. Thus, larger $\tau$ requires less eigenvalues to sum.

For smaller values of $\tau$, we need to sum up more terms in a nearly alternating signs series until $e^{-p_{n,b}^2\tau /2}$ suppressed the large-n terms, so the accuracy is also affected. In the $\tau = 0.0025$ case of our implementation, we even get an elusive negative value, while others are fine.

### Miscellaneous

#### Tests for $\nu=-6$

In the paper's setting, the second summation is not tested, for the settings $\nu=3, -0.6$ give $N_\nu(b)=0$.

We intended to test it for $\nu=-6$, and encounted the following problem:

```python
import ExactAsian as ea
c = ea.BsmAsianLinetsky2004(strike = 90.0,
                            spot=100.0,
                            vol=0.1,
                            texp=1,
                            intr=0.0,
                            divr=0.025,
                            b=1.0,
                            call=True)
```

```python
c.exact_asian_price(n_eig=50)
```

We looked into the case where $\nu<0$ and $\lambda\in(0,\nu^2/2]$. The zeros are related to the reciprocal of the gamma function. Recalling the approximation (Equation 27 in the paper):

$$\psi(b,(\nu^2-q^2)/2)\sim\frac{\Gamma(q)}{\Gamma((\nu+q)/2)}\left(\frac{1}{2b}\right)^{(\nu-q)/2}$$

It requires the $q_{n,b}$ value to satisfy:
$$q_{n,b}=(|\nu|-2n+2), n=1,...,[|\nu|/2]+1$$

Here are the $q_{n,b}$ values in the $\nu=-6$ case

```python
c.find_zeros_real(nu=-6)
```

    array([4, 2, 0])

The summation part includes the term $\Gamma(\tfrac{1}{2}(\nu+q_{n,b}))$. With the $q_{n,b}$ values above, we have $\Gamma(-1),\Gamma(-2),\Gamma(-3)$, and this is the reason for the `ValueError: gamma function pole`. The derivation given by the author seems convincing, so we don't know where it goes wrong.

#### Vectorization

Unfortunately, all our functions are wriitten in `for` loop. The `mpmath.hyper()` is the function $\,_n F_n$, it recognizes the vector as the type of the hypergeometric function. Here is an example of $\,_3 F_2$:

```python
import mpmath as m
m.hyper([1,3,5],[2,4],0.5j)
```

    mpc(real='0.51200000000000001', imag='0.56599999999999995')


# Diversity inducing distributions

To approaches to select $k$ diverse elements from a larger set, we look into two approaches:

- [MRFs](#Repulsive-Markov-Random-Fields)
- [DPPs](#Determinantal-Point-Processes)

```python
# from chmp.ipynb import notebook_preamble; notebook_preamble()
%matplotlib inline
# disable rescaling the figure, to gain tighter control over the result
%config InlineBackend.print_figure_kwargs = {'bbox_inches':None}
```

```python
import collections
import itertools as it
import math
import random

import matplotlib.pyplot as plt
import numba
import numpy as np
import pandas as pd

from chmp.ds import mpl_set
```

```python
def random_nonzero_choice(a, n_nonzero=-1):
    if n_nonzero <= 0:
        n_nonzero = np.sum(a)

    return _random_nonzero_choice(np.asarray(a, bool), np.int64(n_nonzero))

@numba.jit('int64(boolean[:], int64)', nopython=True, nogil=True)
def _random_nonzero_choice(a, n_nonzero):
    j = random.randrange(0, n_nonzero)
    i_max = -1

    for i, v in enumerate(a):
        if v:
            j -= 1
            i_max = i

        if j < 0:
            return i

    return i_max

def random_weighted_zero_choice(a, p):
    return _random_weighted_zero_choice(np.asarray(a, np.bool), np.asarray(p, np.float64))

@numba.jit('int64(boolean[:], float64[:])', nopython=True, nogil=True)
def _random_weighted_zero_choice(a, p):
    threshold = 0.0
    for q in p:
        threshold += q

    threshold = threshold * random.random()

    pi = 0

    i_max = -1

    for i, v in enumerate(a):
        if v:
            continue

        i_max = i
        threshold -= p[pi]
        pi += 1

        if threshold <= 0:
            return i

    return i_max

def det_change(D, B, L_sub_inv, p_out=None):
    if p_out is None:
        p_out = np.zeros_like(D, dtype=np.float64)

    else:
        p_out = np.asarray(p_out, np.float64)

    _det_change(
        np.asarray(D, np.float64),
        np.asarray(B, np.float64),
        np.asarray(L_sub_inv, np.float64),
        p_out,
    )

    return p_out

@numba.jit('void(float64[:], float64[:, :], float64[:, :], float64[:])', nogil=True, nopython=True)
def _det_change(D, B, L_sub_inv, p_out):
    # p = D - np.diag(B.T @ L_sub_inv @ B)
    for k in range(len(p_out)):
        tmp = 0.0

        for i in range(L_sub_inv.shape[0]):
            for j in range(L_sub_inv.shape[1]):
                tmp += L_sub_inv[i, j] * B[i, k] * B[j, k]

        p_out[k] = D[k] - tmp
```

```python
param_n = 6
param_k = 3
phi = np.linspace(0, 2 * np.pi, param_n + 1)[:param_n]

states = [
    [int(i in active) for i in range(param_n)]
    for active in it.combinations(range(param_n), param_k)
]

euclidean_distance = 2 * (1 - np.cos(phi[:, None] - phi[None, :]))
```

```python
states[:10]
```

    [[1, 1, 1, 0, 0, 0],
     [1, 1, 0, 1, 0, 0],
     [1, 1, 0, 0, 1, 0],
     [1, 1, 0, 0, 0, 1],
     [1, 0, 1, 1, 0, 0],
     [1, 0, 1, 0, 1, 0],
     [1, 0, 1, 0, 0, 1],
     [1, 0, 0, 1, 1, 0],
     [1, 0, 0, 1, 0, 1],
     [1, 0, 0, 0, 1, 1]]

```python
euclidean_distance
```

    array([[ 0.,  1.,  3.,  4.,  3.,  1.],
           [ 1.,  0.,  1.,  3.,  4.,  3.],
           [ 3.,  1.,  0.,  1.,  3.,  4.],
           [ 4.,  3.,  1.,  0.,  1.,  3.],
           [ 3.,  4.,  3.,  1.,  0.,  1.],
           [ 1.,  3.,  4.,  3.,  1.,  0.]])

## Repulsive Markov Random Fields

The inclusion of item $i$ is encoded by the binary indicator variable $s_i \in \{0, 1\}$. We assume the distribution of the $s_i$ varibales is given by

$$
\begin{align}
    p(\{s_i\})
    &= \frac{1}{Z} \exp\left[ - E(\{s_i\}) \right]
\\
    &= \frac{1}{Z} \exp\left[ - \sum_{ij} s_i w_{ij} s_j \right],
\end{align}
$$

with an interaction energy $w_{ij}$. Higher values of $w_{ij}$ lead to repulsion between items $i$ and $j$. The diagonal $w_{ii}$ encodes the prevalance of each item without interactions, since the variables are assumed to be binary.

Use Gibbs sampling to draw samples from this distribution. Change in energy, when adding item $k$, is given by

$$
\begin{align}
    E_{+k}
        &= w_{kk} + \sum_{i  \in \mathrm{active}} (w_{ik} + w_{ki})
\\
        &= w_{kk} + 2 \sum_{i \in \mathrm{active}} w_{ik}
\end{align}
$$

```python
pairwise_interaction = -euclidean_distance
```

```python
mrf_expected_energy = np.asarray([s @ pairwise_interaction @ s for s in states])
mrf_expected_energy = mrf_expected_energy - np.min(mrf_expected_energy)
```

```python
mrf_expected_p = np.exp(-mrf_expected_energy)
mrf_expected_p = mrf_expected_p / np.sum(mrf_expected_p)
mrf_expected_p = {tuple(s): p for s, p in zip(states, mrf_expected_p)}
```

```python
stats = collections.Counter()
active = np.asarray(states[0], dtype=bool)

idx = np.arange(len(active))
for iteration in range(10000):
    # remove one active state
    i = random_nonzero_choice(active)
    active[i] = False
    inactive = ~active

    delta_e = (
        # \Delta E_k = w_{kk}
        pairwise_interaction[inactive, inactive] +

        # \Delta E_k = 2 \sum_{j \in \mathrm{active}} w_{ik}
        2 * pairwise_interaction[inactive, :] @ active
    )
    delta_e = delta_e - np.min(delta_e)

    p = np.exp(-delta_e)
    p = p / np.sum(p)

    j = random_weighted_zero_choice(active, p)
    active[j] = True

    if iteration > 100:
        stats[tuple(int(i) for i in active)] += 1
```

```python
mrf_actual_Z = sum(stats.values())
mrf_actual_p = {k: p / mrf_actual_Z for k, p in stats.items()}
```

```python
def plot_actual_and_expected(actual_p, expected_p, title=None):
    df = pd.DataFrame({
        'actual': pd.Series(actual_p),
        'expected': pd.Series(expected_p),
    })
    df['cluster_id'] = (100_000 * df['expected']).astype(int)
    df = df.sort_values('expected')
    df['pos'] = np.arange(len(df))

    plt.figure(figsize=(8, 4))
    df[['actual', 'expected']].plot.bar(rot=60, ax=plt.gca())
    mpl_set(
        bottom=0.275, left=0.1, right=0.9,
        xticks=[], xlim=(-0.5, len(df) - 0.5),
        title=title,
    )

    for _, pos, state in df_illustration.itertuples():
        plt.axes([0.1 + (pos + 0.5) / len(df)  * (0.9 - 0.1) - 0.05, 0.05, 0.1, 0.2])
        state = np.asarray(state, dtype=bool)
        plt.plot(np.cos(phi[~state]), np.sin(phi[~state]), '.b')
        plt.plot(np.cos(phi[state]), np.sin(phi[state]), '.r')
        mpl_set(xticks=[], yticks=[], ylim=(-1.1, 1.1), xlim=(-1.1, 1.1))
        plt.box(False)
```

```python
plot_actual_and_expected(mrf_actual_p, mrf_expected_p, title='MRF: actual vs. expected')
```

## Determinantal Point Processes

The probability of chosing set $Y$ is given by

$$
    P(Y) = \det(L_Y),
$$

where the kernel matrix $L_{Y,ij}$ is specified in terms of the kernel $L(x_i, x_j) = q(x_i) k(x_i, x_j) q(x_j)$. When $q$ is chosen as $q(x) = \sqrt{p(x)}$, the determinant can be interpreted as

$$
\begin{align}
    P(Y) \propto \det(L_Y) = \det(K_Y) \prod_{i \in Y} p(x_i)
\end{align}
$$

The interactions between items, is encoded by the kernel $k$. For $P$ to be a valid probability distribution, $k$ must be [positive-definite kernel](https://en.wikipedia.org/wiki/Positive-definite_kernel). Examples, according to wikipedia, are:

- linear kernel: $k(x, y) = x \cdot y$
- polynomial kernel $k(x, y) = (x \cdot y + r)^n$
- gaussian kernel: $k(x, y) = \exp [-(x - y)^2 / 2 \sigma^2 ]$
- laplacian kernel: $k(x, y) = \exp [-\alpha (x - y) ]$

Here, we are looking at $k-DPPs$ with a fixed number of items. To draw samples, we use Gibbs sampling. With active
elements $Y$, the probability to add candidate element $i$ to the set of active elements is determined by

$$
\begin{align}
    P(Y \cup \{ i \}) &= \det(L_Y) \det(L_{ii} - L_{iY} L_Y^{-1} L_{Yi}),
\end{align}
$$

with $L_{ii} = e_i^T L\,e_i$, $L_{iY} = e_i\,L\,P_Y$, $L_{iY}^T = L_{Yi}$, and $L_Y = P_Y\,L\,P_Y$. Here we used the unit vectors $e_i$ and the projectors $P_Y$ onto the $Y$-subspace.

```python
# gaussian kernel
L = np.exp(-euclidean_distance / 10)

# laplace kernel
#L = np.exp(-np.sqrt(euclidean_distance) / 1e3)

dpp_expected_p = {
    tuple(s): np.linalg.det(L[s, :][:, s])
    for s in states
    for s in [np.asarray(s, bool)]
}
dpp_expected_z = sum(dpp_expected_p.values())
dpp_expected_p = {k: p / dpp_expected_z for k, p in dpp_expected_p.items()}
```

```python
stats = collections.Counter()

state = np.asarray(states[0], dtype=np.bool)
inv_state = ~state

for iteration in range(10000):
    # remove point
    i = random_nonzero_choice(state)
    state[i] = False
    inv_state[i] = True

    # decompose the matrix
    L_sub = L[state, :][:, state]
    B = L[state, :][:, inv_state]
    D = L[inv_state, inv_state]

    # invert the sub matrix
    L_sub_inv = np.linalg.inv(L_sub)

    # calculate the determinant changes
    p = det_change(D, B, L_sub_inv)

    j = random_weighted_zero_choice(state, p=p)
    state[j] = True
    inv_state[j] = False

    if iteration > 100:
        stats[tuple(state.astype(int))] += 1
```

```python
dpp_actual_Z = sum(stats.values())
dpp_actual_p = {k: p / dpp_actual_Z for k, p in stats.items()}
```

```python
plot_actual_and_expected(dpp_actual_p, dpp_expected_p, title='DPP: actual vs. expected')
```

```python

```