# Bias in the profile Poisson likelihood

> You can find an interactive version of this Jupyter notebook in my [github](https://github.com/giacomov/giacomov.github.io/tree/master/notebooks)

A common method used when modeling spectra in the X-rays and gamma-rays community involves the maximization of a profile likelihood commonly called CSTAT.

## CSTAT derivation

We start from an observed spectrum accumulated in a source region (a time interval with a signal for a time series or a circular region containing a source for a 2d dataset), and a background spectrum obtained by counting events in a "background region", usually an off-source interval for 1-d data or a control region like an annulus around the source of interest in 2-d data. Let's assume both spectra have the same number of bins $N$.

Following the same notation as in the [XSpec manual](https://heasarc.gsfc.nasa.gov/xanadu/xspec/manual/XSappendixStatistics.html), let's consider the i-th bin in the spectra, with these definitions:

* $t_s$ is the exposure for the on-source spectrum
* $t_b$ is the exposure for the background spectrum
* $S_i$, $B_i$ are the number of counts respectively in the on-source and in the off-source spectrum
* $m_i$ is the expected *rate* of events from the source model under consideration (i.e., a spectral model convolved with the response of the instrument)
* $b_i$ is the expected *rate* of events from the background model under consideration

Both the source and background counts are Poisson random variables. Let's derive the formula for CSTAT using the [sympy](http://www.sympy.org/en/index.html) package for symbolic computation:

```python
from sympy import *
from IPython.display import display
init_printing()

# define symbols
ts,mi,bi,Si,tb,bi,Bi = symbols("t_s m_i b_i S_i t_b b_i B_i",
                               real=True,
                               nonnegative=True)
```

The expected number of counts from the source model is $t_s (m_i + b_i)$, so the Poisson distribution for the source spectrum is:

```python
# Source Poisson distribution
P1 = (ts*(mi+bi))**Si * exp(-(ts*(mi+bi))) / factorial(Si)

display(P1)
```

The expected number of counts for the background is $t_b b_i$:

```python
P2 = (tb * bi)**Bi * exp(-(tb*bi)) / factorial(Bi)

display(P2)
```

The joint probability of observing $S_i$ counts in the source spectrum and $B_i$ counts in the background spectrum is $P1 \times P2$. Let's then take the logarithm of it, which is our [log(likelihood)](https://en.wikipedia.org/wiki/Likelihood_function#Log-likelihood) function:

```python
P = P1 * P2

L = log(P).expand().simplify()

display(L)
```

And here is the important bit: we now assume we do not have any information on the background model, so we consider each expectation in the background spectrum $b_i$ as a parameter free to vary. In other words, we use a background model with $N$ parameters (as many as the number of bins in the spectra) all free to vary independently of each other. This is the [maximum variance model](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff). We can then consider the [profile likelihood](https://en.wikipedia.org/wiki/Likelihood_function#Profile_likelihood) obtained by profiling out (i..e, maximizing the likelihood with respect to) the $b_i$. In order to do so, we take the partial derivative with respect to the $b_i$ and solve the equation $\delta L / \delta b_i = 0$ (because at the maximum of L all partial derivatives must be zero):

```python
dLdb = diff(L, bi)

res = solve([dLdb], [bi])

display(res)
```

Of the two solutions, we take the second one which is positive (the expected background cannot be negative):

```python
bi_mle = res[1][0]

bi_mle.factor().simplify()
```

We can now substitute it into the equation of the log likelihood. However, it becomes a very long expression:

```python
L.subs(bi, bi_mle)
```

Instead, we will simply compute the numerical value for $b_i$ first and then plug that into the log likelihood.

> All the code used in the following is implemented in the last cell of this notebook. Execute that first if you are running the notebook interactively.

## Bias in CSTAT

Let's now test the formula.

In order to avoid unnecessary complications, we consider a very simple case where we have a source with a constant spectrum and a background that is constant as well. We also assume an exposure of 1 second for both the source and the background spectrum and an ideal instrument (no need to convolve by the effective area). This allows us to focus on the problem at hand, instead of the details of an X-ray or gamma-ray analysis.

In a real situation we do not know the true models. We will assume a constant for the source, and we will use the CSTAT trick for the background (i.e., a stepwise model with $N$ parameters free to vary independently).

Let's simulate a dataset and fit it using CSTAT. This is achieved with my `fit_one_realization` function:

```python
source_rate = 30.0
background_rate = 50.0
number_of_bins = 10
# This is to guarantee repeatibility
random_seed = 0

res = fit_one_realization(random_seed, number_of_bins,
                          source_rate, background_rate,
                          plot=True)
print("Reconstructed value: %.2f" % res)
```

The method worked. Now, let's observe carefully the plot and understand exactly what is happening.

The red data points are the counts in the background spectrum, and the red model is the maximum variance model used by CSTAT, where every $b_i$ is given by the solution we called `bi_mle` above. As you can see, even though the true model is a constant, the background model jumps up and down quite a lot. This is the effect of considering as many parameters as there are bins.

The total model is given by the source model (a constant) plus the background model. This is why the total model is not a flat line anymore.

Now let's consider the point around 5 keV. There is an upward fluctuation in the total model (remember that the true model is actually a constant at 30 + 50 = 80 counts per keV). Thus, in the background model, the parameter $b_i$ for that particular bin is pulled upward. At the opposite, around 7.5 keV there is a underflucutation in the total model, and the background model is pulled downward.

The problems start showing up when the number of bins in the spectra increases so that many of them are either 0 or 1. I will show this by performing many simulations (16000) of the same situation (source rate = 30 counts/keV, background rate = 50 counts/keV) but varying the number of bins in the spectrum:

```python
# Let's make a bunch of simulations

source_rate = 30.0
background_rate = 50.0

number_of_bins = [50, 100, 200, 400, 500, 1600]
n_realizations = 16000

fig, subs = plt.subplots(3,2, figsize=(8, 8))

for i, (sub, n_bins) in enumerate(zip(subs.flatten(),
                                      number_of_bins)):

    # No rebinning
    res = fit_many_realizations(n_realizations, n_bins,
                                source_rate,
                                background_rate,
                                fit_one_realization,
                                rebin=0.0)

    _ = sub.hist(res, 80)

    sub.set_xlabel("Rate")

    if (i+1) % 2 != 0:

        sub.set_ylabel("# of realizations")

fig.tight_layout()
```

We can clearly see that while in the first two cases in the upper panels the method returns an estimate for the rate that is centered on the true value (30), for the case in the second rows there is a clear bias towards larger rates. Moreover, when the number of bins increases even more (third row) we have a bias and a lot of very wrong results (look at the left tail of the distribution on the left) until we get to the point where all fits return an estimated rate of zero (lower right panel).

This problem have to do with two things: the number of parameters in the background model for the last two cases is really large compared to the information contained in the data. Essentially we are modeling a constant with 500 and 1600 independent parameters!

Moreover, when the counts in the background $B_i$ are zero, then $b_i$ is zero. Indeed we have:

```python
bi_mle.subs(Bi, 0).simplify()
```

Even though sympy is not smart enough to recognize it, the expression in the square root is actually equal to $(m_i t_b + m_i t_s - S_i)^2$:

```python
((mi * tb + mi * ts - Si)**2).expand()
```

so if $B_{i}=0$ then $b_i=0$ independently of $m_i$ and $S_i$. This is unphysical (the background cannot be zero, and also violates the assumptions we started from because the rate parameter of a Poisson distribution cannot be zero). Essentially, there is little to no information about the background when this happens and the formulation of the problem breaks down. As we have just seen, this introduces a bias which becomes more and more severe as the number of bins where this happens increase.

If this is the case, then a simple countermeasure is to rebin the spectra so that there is at least some background counts in each bin. Let's see if this solves the problem:

```python
fig, subs = plt.subplots(3,2, figsize=(8, 8), sharex=True)

for i, (sub, n_bins) in enumerate(zip(subs.flatten(),
                                      number_of_bins)):

    # min counts in background = 1
    res1 = fit_many_realizations(n_realizations, n_bins,
                                 source_rate,
                                 background_rate,
                                 fit_one_realization,
                                 rebin=1.0)

    res3 = fit_many_realizations(n_realizations, n_bins,
                                 source_rate,
                                 background_rate,
                                 fit_one_realization,
                                 rebin=3.0)

    res5 = fit_many_realizations(n_realizations, n_bins,
                                 source_rate,
                                 background_rate,
                                 fit_one_realization,
                                 rebin=5.0)

    _ = sub.hist(res1, 80, label='Rebin: 1')
    _ = sub.hist(res3, 80, label='Rebin: 3', alpha=0.5)
    _ = sub.hist(res5, 80, label='Rebin: 5', alpha=0.5)

    sub.set_xlabel("Rate")

    if (i+1) % 2 != 0:

        sub.set_ylabel("Fraction of realizations")

#_ = subs[-1][-1].set_xlim([10, 50])
_ = subs[-1][-1].set_ylim([0, 650])

_ = subs[0][0].legend(loc=0)
```

It looks like rebinning with 1 counts correct in large part but not completely the problem, but using 3 or 5 is much better.

> NOTE: here we are dealing with a simple illustrative problem. In a real situation, you should be a little conservative with the number of counts you require for each bin in the background.

# An alternative: model the background directly

An even better solution to the problem is to avoid the use of the profile likelihood alltogether, and instead model simultaneously the background and the data with two explicit models (one for the source, one for the background).

We can then use the normal Poisson log-likelihood (aka Cash statistic) for both separately and then sum them and maximize the sum varying the parameters of both the source and the background model.

This has many advantages beyond solving the problem with the bias outlined above:

1. We use a much better trade off between variance and bias in the background
2. When computing errors on parameters we can take into account the covariance between parameters in the background and in the source model
3. The background model is explicit and can be tested

For the simple example treated here, this is very straightforward. I coded this in the `fit_one_realization_no_profiling` function, let's see how it behaves:

```python
fig, subs = plt.subplots(3,2, figsize=(8, 8), sharex=True, sharey=True)

for i, (sub, n_bins) in enumerate(zip(subs.flatten(),
                                      number_of_bins)):

    # Fit
    resf = fit_many_realizations(n_realizations, n_bins,
                                 source_rate,
                                 background_rate,
                                 fit_one_realization_no_profiling,
                                 rebin=0.0)

    _ = sub.hist(resf, 60, label='Bkg fit')

    sub.set_xlabel("Rate")

    if (i+1) % 2 != 0:

        sub.set_ylabel("# of realizations")

_ = subs[0][0].legend(loc=0)
```

# Code

```python
import numpy as np
import scipy.optimize
import matplotlib.pyplot as plt
import multiprocessing
import functools
import sys
%matplotlib inline
from numba import jit

def generate_data(n_bins, src_rate, bkg_rate):
    start = 1.0  # keV
    stop = 10.0  # keV

    dE = (stop - start) / n_bins

    edges = np.linspace(start, stop, n_bins + 1)

    energy_centers = (edges[1:] + edges[:-1]) / 2.0

    observation = np.random.poisson((src_rate + bkg_rate) * dE, n_bins).astype(float)

    background = np.random.poisson(bkg_rate * dE, n_bins).astype(float)

    return energy_centers, dE, observation, background

# This is a translation in python of the method
# StdCstat::specificPerformB in CstatVariants.cxx
# from the XSpec source code
@jit(nopython=True)
def cstat_xspec(S, B, expos_ratio, y, rebin=0.0):

    assert len(S) == len(B)
    assert len(y) == len(S)

    if rebin > 0.0:

        newS = np.zeros_like(S)
        newB = np.zeros_like(B)
        newy = np.zeros_like(y)

        m = 0
        j = 0
        k = 0

        for i, bi in enumerate(B):

            m += bi

            if i == len(B) - 1:

                if m >= rebin:

                    newS[k] = np.sum(S[j: i + 1])
                    newB[k] = np.sum(B[j: i + 1])
                    newy[k] = np.sum(y[j: i + 1])

                    k += 1

                else:

                    # We reached the end but we don't have
                    # enough counts in the last bin. Let's
                    # incorporate these in the last bin
                    # of the rebinned one
                    newS[k - 1] += np.sum(S[j:])
                    newB[k - 1] += np.sum(B[j:])
                    newy[k - 1] += np.sum(y[j:])

                break

            if m >= rebin:
                newS[k] = np.sum(S[j: i + 1])
                newB[k] = np.sum(B[j: i + 1])
                newy[k] = np.sum(y[j: i + 1])

                m = 0
                j = i + 1
                k += 1

        assert np.sum(S) == np.sum(newS[:k]), "Sum of S is different"
        assert np.sum(B) == np.sum(newB[:k]), "Sum of B is different"
        assert abs(np.sum(newy[:k]) - np.sum(y)) < 0.01, "Sum of y is different"

        S = newS[:k]
        B = newB[:k]
        y = newy[:k]

        assert np.all(B >= rebin)

    tsi = 1.0
    tbi = 1.0

    FLOOR = 1.0e-5
    stat = 0

    F = np.zeros_like(S)

    #   for (size_t i=0; i<N; i++) {
    for i in range(len(S)):
        #       Real si = sbin[i];
        si = S[i]
        #       Real bi = bbin[i];
        bi = B[i]

        # NOTE: for simplicity tsi and tbi are 1 for all bins
        #       Real tsi = tsbin[i];
        #       Real tbi = tbbin[i];
        # tsi = ts[i]
        # tbi = tb[i]

        #       Real yi = ybin[i];
        yi = y[i]

        #       Real ti = tsi + tbi;
        ti = tsi + tbi

        #       yi = std::max(yi, FLOOR/tsi);
        yi = max(yi, FLOOR / tsi)

        #     if (si == 0.0) {
        if si == 0.0:

            stat += tsi * yi - bi * np.log(tbi / ti)
        #       } else {
        else:
            #           if (bi == 0.0) {
            if bi == 0.0:
                # 	  stat += tsi*yi + si*(std::log(si)-std::log(tsi*yi)-1);
                # 	}
                #               if ( yi <= si/ti ) {
                if yi <= si / ti:
                    #                   stat += -tbi*yi - si*std::log(tsi/ti);
                    stat += -tbi * yi - si * np.log(tsi / ti)
                #               } else {
                else:
                    #                   stat += tsi*yi + si*(std::log(si)-std::log(tsi*yi)-1);
                    stat += tsi * yi + si * (np.log(si) - np.log(tsi * yi) - 1)
            else:

                #               // now the main case where both data and background !=0
                #               // Solve quadratic equation for f. Use the positive root to ensure
                #               // that f > 0.
                #               Real a = ti;
                a = ti
                #               Real b = ti*yi - si - bi;
                b = ti * yi - si - bi
                #               Real c = -bi*yi;
                c = -bi * yi
                #               Real d = sqrt(b*b - 4.0*a*c);
                d = np.sqrt(b * b - 4.0 * a * c)
                #               if ( b >= 0.0 ) {
                if b >= 0:
                    #                   f = -2*c / (b + d);
                    f = -2 * c / (b + d)
                #               } else {
                else:
                    #                   f = -(b - d) / (2*a);
                    f = -(b - d) / (2 * a)
                # This is to keep track of the profiled out background parameter
                F[i] = f

                #               stat += tsi*yi + ti*f - si*std::log(tsi*yi+tsi*f)
                #                       - bi*std::log(tbi*f) - si*(1-std::log(si))
                #                       - bi*(1-std::log(bi));
                stat += tsi * yi + ti * f - si * np.log(tsi * yi + tsi * f) \
                        - bi * np.log(tbi * f) - si * (1 - np.log(si)) \
                        - bi * (1 - np.log(bi))

    return stat, F

# Wrapper than can be used in scipy.minimize
def cstat_wrapper(rate, obs, bkg, dE, rebin, full=False, cstat_function=cstat_xspec):
    # Only one parameter, the rate

    # Compute expected model counts
    model_counts = np.repeat(rate * dE, obs.shape[0])

    if full:

        return cstat_function(obs, bkg, 1.0,
                              model_counts, rebin)

    else:

        return cstat_function(obs, bkg, 1.0,
                              model_counts, rebin)[0]

def fit(obs, bkg, dE, x0, rebin=0.0):

    results = scipy.optimize.minimize_scalar(cstat_wrapper, bounds=(0.0, x0 * 10.0),
                                             args=(obs, bkg, dE, rebin),
                                             method='Bounded')

    if not results.success:
        raise RuntimeError("Fit failed with reason %s" % results.message)

    return results.x

def fit_one_realization(seed,
                        n_bins,
                        true_src_rate,
                        true_bkg_rate,
                        plot=False,
                        rebin=0.0):

    np.random.seed(seed)

    energies, dE, obs, bkg = generate_data(n_bins,
                                           true_src_rate,
                                           true_bkg_rate)

    # print(np.average(obs - bkg) / dE)

    # We start the fit from very close to the true value so that if the fit
    # converges somewhere else it is something intrinsic to the
    # likelihood function, and not a minimization problem
    mle_est = fit(obs, bkg, dE, true_src_rate, rebin=rebin)

    if plot:

        # Get the values for the nuisance parameters
        _, bkg_nuisance = cstat_wrapper(mle_est, obs, bkg, dE,
                                        rebin=rebin,
                                        full=True)

        _ = plt.errorbar(energies, obs, xerr=dE / 2.0,
                         yerr=np.sqrt(obs), fmt='.')

        _ = plt.errorbar(energies, bkg, xerr=dE / 2.0,
                         yerr=np.sqrt(bkg),
                         color='red', alpha=0.5, fmt='.')

        if rebin == 0:

            _ = plt.step(energies, bkg_nuisance,
                         color='red', where='mid',
                         label='Background model')

            _ = plt.step(energies,
                         mle_est * dE + bkg_nuisance,
                         color='blue', where='mid',
                         label='Total model')

            _ = plt.legend(loc=0)

        _ = plt.xlabel("Energy (keV)")
        _ = plt.ylabel("Counts per bin")

    return mle_est

# Wrapper than can be used in scipy.minimize
# This implements the cstat when the background model is
# a constant (no profiling)
def cstat_no_profiling(parameters, obs, bkg, dE, rebin=None, full=False):
    src_rate, bkg_rate = parameters

    # Assume exposure=1 for simplicity
    exposure = 1.0

    # Compute expected model counts
    src_model_counts = np.repeat((src_rate + bkg_rate) * dE * exposure, obs.shape[0])
    bkg_model_counts = np.repeat(bkg_rate * dE * exposure, obs.shape[0])

    cstat_src = -np.sum(obs * np.log(src_model_counts) - src_model_counts)
    cstat_bkg = -np.sum(bkg * np.log(bkg_model_counts) - bkg_model_counts)

    cstat = cstat_src + cstat_bkg

    if full:

        return cstat, bkg_model_counts

    else:

        return cstat

def fit_no_profiling(obs, bkg, dE, x0):
    results = scipy.optimize.minimize(cstat_no_profiling, x0,
                                      args=(obs, bkg, dE),
                                      tol=1e-3)

    if not results.success:
        raise RuntimeError("Fit failed with reason %s" % results.message)

    return results.x

def fit_one_realization_no_profiling(seed,
                                     n_bins,
                                     true_src_rate,
                                     true_bkg_rate,
                                     plot=False,
                                     rebin=None):
    np.random.seed(seed)

    energies, dE, obs, bkg = generate_data(n_bins,
                                           true_src_rate,
                                           true_bkg_rate)

    # print(np.average(obs - bkg) / dE)

    # We start the fit from very close to the true value so that if the fit
    # converges somewhere else it is something intrinsic to the
    # likelihood function, and not a minimization problem
    mle_est = fit_no_profiling(obs, bkg, dE, [true_src_rate,
                                              true_bkg_rate])

    exposure = 1.0

    if plot:
        _ = plt.errorbar(energies, obs, xerr=dE / 2.0,
                         yerr=np.sqrt(obs), fmt='.')

        _ = plt.axhline((mle_est[0] + mle_est[1]) * dE * exposure,
                        color='blue',
                        label='Total model')

        _ = plt.errorbar(energies, bkg, xerr=dE / 2.0,
                         yerr=np.sqrt(bkg),
                         color='red', alpha=0.5, fmt='.')

        _ = plt.axhline(mle_est[1] * dE * exposure,
                        color='red',
                        label='Background model')

        _ = plt.xlabel("Energy (keV)")
        _ = plt.ylabel("Counts per bin")
        _ = plt.legend(loc=0)

    return mle_est[0]

def fit_many_realizations(n_realizations,
                          n_bins,
                          true_src_rate,
                          true_bkg_rate,
                          fit_function=fit_one_realization,
                          rebin=None,
                          parallel=True):

    mle_estimates = np.empty(n_realizations)

    worker = functools.partial(fit_function,
                               n_bins=n_bins,
                               true_src_rate=true_src_rate,
                               true_bkg_rate=true_bkg_rate,
                               plot=False,
                               rebin=rebin)
    if parallel:

        pool = multiprocessing.Pool()

        try:

            for i, res in enumerate(pool.imap(worker,
                                              range(n_realizations),
                                              chunksize=1000)):
                mle_estimates[i] = res

                sys.stderr.write("\r%i of %i completed" % (i + 1, n_realizations))

        except:

            raise

        finally:

            pool.close()
            pool.join()

        sys.stderr.write("\n")

    else:

        # This is for testing purposes, avoiding the parallel execution

        mle_estimates = map(worker, range(n_realizations))

    return mle_estimates

```

```python

```


# Regression

In this notebook, we calculate the Swiss Army infinitsimal jackknife error bounds for linear regression.

This example illustrates at least one way in which dimension dependence can occur implicitly in the error bound of Theorem 1.

```python
import autograd
import autograd.numpy as np
import scipy as sp
import paragami
import vittles

from tqdm import tqdm
import time

import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# A convenience function to visually compare two variables.
def comparison_plot(x, y):
    plt.plot(x, x, 'k')
    plt.plot(x, y, 'r.')
```

## Generate data and calculate IJ.

First, generate some data according to a linear regression model.

```python
def generate_data(num_obs, dim, sigma):
    """ Generate linear regression data.

    Args
    ------
    num_obs: The number of data points
    dim:     The dimension of the regressors
    sigma:   The standard deviation of the residuals

    Returns
    --------
    x, y, theta_true: Data y and x for y ~ theta_true * x.
    """
    theta_true = np.linspace(0, 1, dim)
    x = np.random.random((num_obs, dim)) - 0.5
    y = np.random.normal(x @ theta_true, sigma, num_obs)
    return x, y, theta_true

dim = 5
sigma = 0.5
num_obs = dim * 100

x, y, theta_true = generate_data(num_obs, dim, sigma)
```

First, we specify the weighted regression loss function.  In this case, the optimum is available in closed form.  For convenience we implement the closed form optimum and test that it matches the optimum of the loss function.

```python
def eval_log_loss(theta, w, x, y):
    resid = y - x @ theta
    log_lik = -0.5 * np.sum(w * (resid ** 2)) / num_obs
    return -1 * log_lik

def get_optimum(w, x, y):
    xtx = (w[:, None] * x).T @ x
    xty = (w[:, None] * x).T @ y
    return np.linalg.solve(xtx, xty)

# Test the loss function and optimum with random weights.
w = np.ones(num_obs) + 0.1 * (np.random.random(num_obs) - 0.5)

log_loss_objective = paragami.OptimizationObjective(
    lambda theta: eval_log_loss(theta, w, x, y))

opt = sp.optimize.minimize(
    fun=log_loss_objective.f,
    jac=log_loss_objective.grad,
    hess=log_loss_objective.hessian,
    method='trust-exact',
    x0=theta_true,
    options={'gtol': 1e-12})
print(opt.message)

assert(np.linalg.norm(opt.x - get_optimum(w, x, y)) / dim < 1e-8)
```

    Iter 0: f = 0.11733794
    Iter 1: f = 0.11585070
    Optimization terminated successfully.

Get the initial optimum at the vector of unity weights.

```python
w1 = np.ones(num_obs)
theta_opt = get_optimum(w1, x, y)
```

Define an IJ predictor using ``vittles``.

```python
ij_predictor = \
    vittles.HyperparameterSensitivityLinearApproximation(
        lambda theta, w: eval_log_loss(theta, w, x, y),
        opt_par_value=theta_opt,
        hyper_par_value=w1)
```

Get the IJ predictions and exact CV for a range of weights.

```python
theta_ij = np.full((num_obs, dim), float('nan'))
theta_cv = np.full((num_obs, dim), float('nan'))

ij_time = 0
cv_time = 0

w_loo = np.ones(num_obs)
for n in tqdm(range(num_obs)):
    w_loo[n] = 0
    tic = time.time()
    theta_ij[n, :] = ij_predictor.predict_opt_par_from_hyper_par(w_loo)
    ij_time = ij_time + time.time() - tic

    tic = time.time()
    theta_cv[n, :] = get_optimum(w_loo, x, y)
    cv_time = cv_time + time.time() - tic

    w_loo[n] = 1

comparison_plot(theta_cv - theta_opt[None, :], theta_ij - theta_opt[None, :])
```

The IJ is faster even than the closed form optimum, though, to be fair, the closed form optimum could be made more efficient by caching $(X^T X)^{-1}$ and using the Woodbury matrix identity.

```python
print('IJ time: {}'.format(ij_time))
print('CV time: {}'.format(cv_time))
```

    IJ time: 0.004000186920166016
    CV time: 0.025417566299438477

## Error bound calculation.

Now we calculate bounds on the error of the IJ approximation.  The objective function and its derivatives are

$$
\begin{align}
\ell\left(x_{n},\theta\right)	&=-\frac{1}{2}\left(y_{n}-\theta^{T}x_{n}\right)^{2} \\
g_{n}\left(\theta\right)	&=\left(y_{n}-\theta^{T}x_{n}\right)x_{n} \\
h_{n}\left(\theta\right)	&=-x_{n}x_{n}^{T}. \\
\end{align}
$$

We will define $\Omega_\theta$ as a ball of radius $b$ centered at $\hat\theta$.  The letter $b$ is for "bound".

$$
\Omega_{\theta} =\left\{ \theta: \left\Vert \theta-\hat{\theta} \right\Vert_{2}\le b\right\}.
$$

For notational convenience, we additionally define the following quantities.

$$
\begin{align}
R	=& \frac{1}{N}\sum_{n=1}^{N}y_{n}x_{n}\\
S	=& \frac{1}{N}\sum_{n=1}^{N}x_{n}x_{n}^{T}.
\end{align}
$$

In this notation,

$$
\begin{align}
G\left(\theta\right)	&=
    \frac{1}{N}\sum_{n=1}^{N}\left(y_{n}-\theta^{T}x_{n}\right)x_{n} \\
	&=R-S\theta\\
\end{align}
$$

and

### Assumption 1.

Assumption 1 is certainly satisfied -- indeed, $g_n(\theta)$ is linear.

### Assumption 2.

We have

$$
H\left(\theta\right)	=-S.
$$

Consequently,

$$
C_{op} =1/\lambda_{min}\left(S\right).
$$

In this case, $C_{op}$ is a function of the data alone, and not of the parameter $\theta$.  It can be readily calculated from the data.

### Assumption 3.

We will not require the constants $C_g$ and $C_h$ from assumption 3 here.  As shown below, instead of using Corollary 1, it will be more convienct to control the supremum in Condition 1 directly.

### Assumption 4.

Because $h_n(\theta) = - x_n x_n ^T$ does not depend on $\theta$, we can effectively take $\Delta_\theta = \infty$ and $L_h = 0$.

### Assumption 5.

Because $L_h = 0$, we will not need $C_w$ in this case.

### Condition 1.

We will calculate bounds separately for the gradient and Hessian complexity, called $\delta_g$ and $\delta_h$ respectively, and take

$$
\delta = \max\{\delta_g, \delta_g\}.
$$

### Condition 1 for the Hessian.

We are considering leave one out, so

$$
\begin{align}
\sup_{w \in W_\delta} \sup_{\theta\in\Omega_{\theta}}
\left \Vert
\frac{1}{N} (w_n - 1) h_n(\theta)
\right \Vert_1
 = \frac{1}{N} \sup_{n \in [N]} \sup_{\theta\in\Omega_{\theta}}\left\Vert h_{n}\left(\theta\right) \right\Vert_{1}.
\end{align}
$$

Now,

$$
\begin{align}
\sup_{\theta\in\Omega_{\theta}}\left\Vert h_{n}\left(\theta\right) \right\Vert_{1}
    &=\sup_{\theta\in\Omega_{\theta}}\left\Vert x_{n}x_{n}^{T} \right\Vert_{1} \\
	&=\sum_{d_{1},d_{2}}\left|x_{nd_{1}}x_{nd_{2}}\right| \\
	&=\sum_{d_{1}}\sum_{d_{2}}\left|x_{nd_{1}}\right|\left|x_{nd_{2}}\right| \\
	&=\left\Vert x_{n} \right\Vert_{1}^{2}.
\end{align}
$$

so we define

$$
\delta_h = \frac{1}{N} \sup_{n \in [N]} \left\Vert x_{n} \right\Vert_{1}^{2}
$$

If we were using leave-k-out, one could apply the triangle inequalty to get an upper bound of the sum of the top k values of $\left\Vert x_{n} \right\Vert_{1}^{2}$.

### Condition 1 for the gradient.

As with the Hessian,

$$
\begin{align}
\sup_{w \in W_\delta} \sup_{\theta\in\Omega_{\theta}}
\left \Vert
\frac{1}{N} (w_n - 1) g_n(\theta)
\right \Vert_1
 = \frac{1}{N} \sup_{n \in [N]} \sup_{\theta\in\Omega_{\theta}}\left\Vert g_{n}\left(\theta\right) \right\Vert_{1}.
\end{align}
$$

Now,

$$
\begin{align}
\sup_{\theta\in\Omega_{\theta}} \left\Vert g_{n}\left(\theta\right) \right\Vert_{1}
    &=\sup_{\theta\in\Omega_{\theta}}
    \left\Vert \left(y_{n} - \theta^{T}x_{n}\right)x_{n} \right\Vert_{1} \\
    &= \left\Vert x_{n} \right \Vert_{1} \sup_{\theta\in\Omega_{\theta}}\left|y_{n}-x_{n}^{T}\theta\right|.
\end{align}
$$

To control this, we need to control each terms' largest residual for $\theta\in\Omega_{\theta}$.  We can get a bound that takes advantage of the fact that $\Omega_\theta$ is centered at $\hat\theta$ as follows:

$$
\begin{align}
\sup_{\theta\in\Omega_{\theta}}\left|y_{n}-x_{n}^{T}\theta\right|	&=
    \sup_{\theta\in\Omega_{\theta}}
        \left|y_{n}-x_{n}^{T}\left(\left(\theta-\hat{\theta}\right)+\hat{\theta}\right)\right|  \\
&\le \left|y_{n}-x_{n}^{T} \hat{\theta}\right| +
    \sup_{\theta\in\Omega_{\theta}}\left|x_{n}^{T}\left(\theta-\hat{\theta}\right)\right| \\
& \le \left|y_{n}-x_{n}^{T}\hat{\theta}\right| +
     \left\Vert x_{n} \right\Vert_{2}
         \sup_{\theta\in\Omega_{\theta}} \left\Vert \theta-\hat{\theta} \right\Vert_{2} \\
& = \left|y_{n}-x_{n}^{T}\hat{\theta}\right| + \left\Vert x_{n} \right\Vert_{2}b.
\end{align}
$$

Putting this together,

$$
\delta_g = \frac{1}{N} \max_{n \in [N]}
    \left\Vert x_{n} \right \Vert_{1}
        \left( \left|y_{n}-x_{n}^{T}\hat{\theta}\right| + \left\Vert x_{n} \right\Vert_{2}b \right)
$$

Again, if we were using leave-k-out, the triangle inequality would give as a bound the sum of the top k such values.

### Theorem 1.

In this case, with $L_h = 0$ the error bound reduces to

$$
\begin{align}
\delta \le & \frac{1}{2} C_{op}^{-1} \Rightarrow \\
\max_{w \in W_\delta} \left\Vert
    \hat\theta_{IJ}(w) - \hat\theta(w) \right\Vert_2 \le &
2 C_{op}^2 \delta^2
\end{align}
$$

## A note on dimension dependence in Theorem 1.

In order for Theorem 1 to apply, we must have $\delta \le \Delta_\delta$.  In this case, that means that

$$
\begin{align}
\delta &\le \frac{1}{2} C_{op}^{-1} \Leftrightarrow \\
\frac{1}{N} \max_{n \in [N]} \left( \max \{
    \left\Vert x_{n} \right \Vert_{1}^2 \} ,
    \left\Vert x_{n} \right \Vert_{1}
        \left( \left|y_{n}-x_{n}^{T}\hat{\theta}\right| + \left\Vert x_{n} \right\Vert_{2}b \right)
    \right)
&\le \frac{\lambda_{min}\left(S\right)}{2}
\end{align}
$$

Depending on the values of $x_n$, this condition is not guaranteed to hold.  In particular, the matrix $S$ may be very nearly singular due to colinearity, e.g., due to having two nearly identical regressors.  In this case, the right hand side would be very small, but the left hand side may not necessarily be small.  And, indeed, in such a case we may not expect the IJ to work well.

More generally, we note that the bound on $\delta$ increases  with the dimension $D$ (roughly linearly if the elements of $x_n$ are IID), but that $C_{op}^{-1}$ may plausibly remain the same or even decrease as dimension increases.  Consequently, for a fixed $N$, as $D$ increases the bound of Theorem 1  eventually may not apply in general.

Even when $\delta \le \Delta_\delta$ and Theorem 1 applies, we certainly expect the bound $C_{op} ^ 2 \delta^2$ to increase as the dimension increases with all else fixed.

# Calculating and checking the bounds.

We now define a functions that calculate and report the error bounds.

```python
def print_error_bounds(ij_error_terms):
    for k, v in ij_error_terms.items():
        print('{:>15} = {}'.format(k, v))

def get_ij_error_bounds(x, y, theta_width):
    num_obs = x.shape[0]
    d = x.shape[1]

    s_mat = x.T @ x / num_obs
    c_op = 1 / np.min(np.linalg.eigvals(s_mat))

    # Because L_h = 0, C_{ij} has a simple form.
    c_ij = 1.0

    x_l1_vec = np.sum(np.abs(x), axis=1)
    x_l2_vec = np.linalg.norm(x, axis=1)
    resid = y - x @ theta_opt

    delta_g = x_l1_vec * (np.abs(resid) + theta_width * x_l2_vec) / num_obs
    delta_h = (x_l1_vec ** 2)  / num_obs

    # It is interesting to know whether the Hessian complexity or
    # gradient complexity is bounding.  This is particuarly true
    # because one could work harder to get a better bound on the gradient
    # complexity if necessary.
    print('Proportion where delta_h dominates: ', np.sum(delta_h > delta_g) / num_obs)

    delta = np.max(np.vstack([delta_g, delta_h]))

    delta_delta = 0.5 / (c_ij * c_op)
    err_bound = 2 * (c_op ** 2) * c_ij * delta ** 2

    ij_error_terms = dict()

    ij_error_terms['theta_width'] = theta_width
    ij_error_terms['c_op'] = c_op
    ij_error_terms['d'] = d
    ij_error_terms['delta_h'] = np.max(delta_h)
    ij_error_terms['delta_g'] = np.max(delta_g)
    ij_error_terms['delta'] = delta
    ij_error_terms['delta_delta'] = delta_delta
    ij_error_terms['c_ij'] = c_ij
    ij_error_terms['err_bound'] = err_bound

    if delta > delta_delta:
        print('\n\n#############\nWarning: delta is not small enough to apply theorem 1.\n')
        ij_error_terms['valid'] = False
    else:
        ij_error_terms['valid'] = True

    return ij_error_terms
```

Let's take a guess at a size for $\Omega_\theta$ based on the asymptotic covariance.

```python
obj_hessian = autograd.hessian(lambda theta: eval_log_loss(theta, w1, x, y))(theta_opt)
theta_cov = np.linalg.inv(num_obs * obj_hessian)
theta_width = 2 * np.max(np.sqrt(np.diag(theta_cov)))

print('Using b = {}'.format(theta_width))
```

    Using b = 0.3288812145414719

```python
# theta_width is b in the derivation.
# Note that, in the regime where the Hessian complexity dominates, changing
# theta_width has no effect on the error bounds.
theta_width = 5.0

ij_error_terms = get_ij_error_bounds(x, y, theta_width)
print_error_bounds(ij_error_terms)
```

    Proportion where delta_h dominates:  0.0
                  d = 5
            delta_h = 0.008995287741236152
               c_op = 14.424082493007873
               c_ij = 1.0
            delta_g = 0.02227996563744187
              valid = True
              delta = 0.02227996563744187
          err_bound = 0.20655486292756872
        theta_width = 5.0
        delta_delta = 0.03466424989196899

First, confirm that $\Omega_\theta$ was chosen to be large enough.

```python
err_bound = ij_error_terms['err_bound']
theta_ij_diff = theta_ij - theta_opt
theta_diff_norm = np.linalg.norm(theta_ij_diff + np.sign(theta_ij_diff) * err_bound, axis=1)
plt.hist(theta_diff_norm / theta_width);
if np.max(theta_diff_norm / theta_width) > 1:
    print('###############\n Warning: \\Omega_\\theta is too small.  The results may not be valid.')
    print('Increase ``theta_width`` and try again.')
else:
    print('\\Omega_{\\theta} is big enough to capture the observed changes.')
```

We confirm that the error bounds capture the actual errors.  Ideally, the actual errors would be an appreciable proprotion of the error bounds, though this is not always the case.

```python
err_bound = ij_error_terms['err_bound']
err = theta_cv - theta_ij

plt.figure(figsize=(15, 5))

if ij_error_terms['valid']:
    plt.subplot(1, 2, 1)
    plt.hist(err / err_bound);
    plt.title('Error bound ratios\n(should be less than one)')
    print('Error bound ratio (should be less than one): ', np.max(np.abs(err / err_bound)))
    plt.subplot(1, 2, 2)
    plt.hist(theta_ij / err_bound);
    plt.title('Estimate over error')
else:
    print('Error bounds are invalid.')
    print('Error bound ratio (should be less than one): ', np.max(np.abs(err / err_bound)))

```


## A Quick Tour of DifferentialEquations.jl

DifferentialEquations.jl is a metapackage for solving differential equations in Julia. The basic workflow is:

- Define a problem
- Solve a problem
- Plot the solution

The API between different types of differential equations is unified through multiple dispatch

## Example: Lotka-Volterra ODE

$$\begin{align}
x' &= ax - bxy\\
y' &= -cy + dxy
\end{align}$$

```julia
using DifferentialEquations
# Define a problem
const a = 1.0; const b = 2.0
const c = 1.5; const d = 1.25
f = function (t,u,du) # Define f as an in-place update into du
    du[1] = a*u[1] - b*u[1]*u[2]
    du[2] = -c*u[2]+ d*u[1]*u[2]
end
u0 = [1.0;1.0]; tspan = (0.0,10.0)
prob = ODEProblem(f,u0,tspan)
```

    DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,##1#2}(#1,[1.0,1.0],(0.0,10.0))

```julia
# Solve the problem
sol = solve(prob)
```

    DiffEqBase.ODESolution{Array{Array{Float64,1},1},Array{Float64,1},Array{Array{Array{Float64,1},1},1},DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,##1#2},OrdinaryDiffEq.Tsit5}(Array{Float64,1}[[1.0,1.0],[0.912378,0.971564],[0.768635,0.887673],[0.640215,0.739539],[0.579038,0.586645],[0.575609,0.43638],[0.647248,0.312288],[0.858273,0.222465],[1.35784,0.199293],[1.9923,0.308114],[2.0708,0.692973],[1.38472,1.00669],[0.749332,0.868987],[0.585712,0.607774],[0.61316,0.352207],[0.830329,0.229312],[1.6363,0.225058],[2.17455,0.525744],[1.95406,0.781683]],[0.0,0.0942336,0.292772,0.583403,0.894442,1.27134,1.72127,2.31275,3.08354,3.81497,4.51466,5.0524,5.69616,6.21559,6.92158,7.61688,8.7721,9.6451,10.0],Array{Array{Float64,1},1}[Array{Float64,1}[[Inf,Inf]],Array{Float64,1}[[-1.0,-0.25],[-0.977358,-0.267944],[-0.953858,-0.285244],[-0.874675,-0.340554],[-0.864124,-0.347634],[-0.861484,-0.349373],[-0.860489,-0.349304]],Array{Float64,1}[[-0.860489,-0.349304],[-0.814788,-0.37831],[-0.768366,-0.40298],[-0.623447,-0.471329],[-0.607034,-0.479105],[-0.602901,-0.480922],[-0.595958,-0.478639]],Array{Float64,1}[[-0.595958,-0.478639],[-0.541156,-0.496724],[-0.488396,-0.507047],[-0.336725,-0.524479],[-0.322681,-0.527544],[-0.319131,-0.528077],[-0.306713,-0.517479]],Array{Float64,1}[[-0.306713,-0.517479],[-0.266969,-0.513047],[-0.229776,-0.504214],[-0.120563,-0.467931],[-0.109463,-0.464552],[-0.106698,-0.463573],[-0.100342,-0.455355]],Array{Float64,1}[[-0.100342,-0.455355],[-0.0676245,-0.438163],[-0.0373299,-0.418962],[0.0560904,-0.354913],[0.066499,-0.347975],[0.0690868,-0.34618],[0.0732407,-0.34059]],Array{Float64,1}[[0.0732407,-0.34059],[0.10258,-0.318603],[0.130465,-0.296665],[0.225873,-0.227795],[0.238624,-0.219607],[0.241807,-0.217569],[0.242992,-0.215772]],Array{Float64,1}[[0.242992,-0.215772],[0.279229,-0.193137],[0.315581,-0.17141],[0.452284,-0.105553],[0.473176,-0.097418],[0.478416,-0.0954156],[0.476402,-0.0950276]],Array{Float64,1}[[0.476402,-0.0950276],[0.530856,-0.0744214],[0.587835,-0.0528887],[0.785114,0.023796],[0.810624,0.0365301],[0.816915,0.0398332],[0.816626,0.0393212]],Array{Float64,1}[[0.816626,0.0393212],[0.860996,0.0647479],[0.891706,0.0978093],[0.796413,0.267322],[0.733842,0.304354],[0.717618,0.314065],[0.764589,0.305151]],Array{Float64,1}[[0.764589,0.305151],[0.654751,0.376068],[0.451512,0.464349],[-0.684867,0.676435],[-0.917973,0.625448],[-0.972402,0.610386],[-0.799214,0.754298]],Array{Float64,1}[[-0.799214,0.754298],[-1.03394,0.759797],[-1.23319,0.710792],[-1.28411,0.314734],[-1.13229,0.233961],[-1.0974,0.214075],[-1.40325,0.232449]],Array{Float64,1}[[-1.40325,0.232449],[-1.31558,0.050614],[-1.1445,-0.116552],[-0.762234,-0.295025],[-0.787111,-0.167891],[-0.786233,-0.140416],[-0.552988,-0.489531]],Array{Float64,1}[[-0.552988,-0.489531],[-0.461295,-0.514335],[-0.378013,-0.516403],[-0.188398,-0.508778],[-0.191578,-0.523974],[-0.191915,-0.526953],[-0.126249,-0.466685]],Array{Float64,1}[[-0.126249,-0.466685],[-0.0625391,-0.435904],[-0.00837493,-0.398468],[0.136143,-0.295743],[0.13908,-0.295662],[0.140036,-0.295226],[0.181242,-0.258361]],Array{Float64,1}[[0.181242,-0.258361],[0.223879,-0.228947],[0.265291,-0.201101],[0.421627,-0.119131],[0.445462,-0.109591],[0.451447,-0.107246],[0.44952,-0.105962]],Array{Float64,1}[[0.44952,-0.105962],[0.530807,-0.0749502],[0.616297,-0.0423637],[0.888188,0.0902778],[0.920606,0.118146],[0.927983,0.125535],[0.899776,0.122741]],Array{Float64,1}[[0.899776,0.122741],[0.908496,0.170455],[0.856593,0.238345],[0.0134149,0.577832],[-0.254365,0.634424],[-0.320807,0.648238],[-0.111962,0.640453]],Array{Float64,1}[[-0.111962,0.640453],[-0.270317,0.680535],[-0.449287,0.71273],[-0.993857,0.727265],[-1.0439,0.700943],[-1.05652,0.694192],[-1.10085,0.736798]]],DiffEqBase.ODEProblem{Array{Float64,1},Float64,true,##1#2}(#1,[1.0,1.0],(0.0,10.0)),OrdinaryDiffEq.Tsit5
      order: 5
      adaptiveorder: 4,OrdinaryDiffEq.#124,true,0)

```julia
# Plot the solution using the plot recipe
using Plots; gr() # Using the GR Backend
plot(sol,title="All Plots.jl Attributes are Available")
```

<?xml version="1.0" encoding="utf-8"?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="600" height="400" viewBox="0 0 600 400">
<defs>
  <clipPath id="clip00">
    <rect x="0" y="0" width="600" height="400"/>
  </clipPath>
</defs>
<polygon clip-path="url(#clip00)" points="
0,400 600,400 600,0 0,0
  " fill="#ffffff" fill-opacity="1"/>
<defs>
  <clipPath id="clip01">
    <rect x="120" y="0" width="421" height="400"/>
  </clipPath>
</defs>
<polygon clip-path="url(#clip00)" points="
30.6037,384.952 596.063,384.952 596.063,23.3815 30.6037,23.3815
  " fill="#ffffff" fill-opacity="1"/>
<defs>
  <clipPath id="clip02">
    <rect x="30" y="23" width="566" height="362"/>
  </clipPath>
</defs>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  30.6037,379.528 30.6037,28.805
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  143.696,379.528 143.696,28.805
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  256.787,379.528 256.787,28.805
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  369.879,379.528 369.879,28.805
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  482.971,379.528 482.971,28.805
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  596.063,379.528 596.063,28.805
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  39.0856,329.504 587.581,329.504
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  39.0856,238.421 587.581,238.421
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  39.0856,147.339 587.581,147.339
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  39.0856,56.2558 587.581,56.2558
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,384.952 596.063,384.952
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,384.952 30.6037,379.528
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  143.696,384.952 143.696,379.528
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  256.787,384.952 256.787,379.528
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  369.879,384.952 369.879,379.528
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  482.971,384.952 482.971,379.528
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  596.063,384.952 596.063,379.528
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,384.952 30.6037,23.3815
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,329.504 39.0856,329.504
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,238.421 39.0856,238.421
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,147.339 39.0856,147.339
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  30.6037,56.2558 39.0856,56.2558
  "/>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 30.6037, 396.952)" x="30.6037" y="396.952">0</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 143.696, 396.952)" x="143.696" y="396.952">2</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 256.787, 396.952)" x="256.787" y="396.952">4</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 369.879, 396.952)" x="369.879" y="396.952">6</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 482.971, 396.952)" x="482.971" y="396.952">8</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 596.063, 396.952)" x="596.063" y="396.952">10</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 29.4037, 334.004)" x="29.4037" y="334.004">0.5</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 29.4037, 242.921)" x="29.4037" y="242.921">1.0</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 29.4037, 151.839)" x="29.4037" y="151.839">1.5</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 29.4037, 60.7558)" x="29.4037" y="60.7558">2.0</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:21; text-anchor:middle;" transform="rotate(0, 313.333, 18)" x="313.333" y="18">All Plots.jl Attributes are Available</text>
</g>
<polyline clip-path="url(#clip02)" style="stroke:#0099ff; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  30.6037,238.421 36.3154,255.439 42.0271,269.785 47.7388,281.67 53.4505,291.357 59.1622,299.114 64.8739,305.19 70.5856,309.805 76.2974,313.145 82.0091,315.363
  87.7208,316.583 93.4325,316.901 99.1442,316.391 104.856,315.11 110.568,313.096 116.279,310.374 121.991,306.961 127.703,302.861 133.414,298.073 139.126,292.587
  144.838,286.388 150.55,279.458 156.261,271.778 161.973,263.325 167.685,254.079 173.396,244.015 179.108,233.109 184.82,221.349 190.532,208.739 196.243,195.297
  201.955,181.052 207.667,166.054 213.378,150.406 219.09,134.212 224.802,117.655 230.514,101.003 236.225,84.6096 241.937,68.9143 247.649,54.4359 253.36,41.7277
  259.072,31.7397 264.784,25.3815 270.496,23.3815 276.207,26.2869 281.919,34.4642 287.631,48.0945 293.342,66.8182 299.054,89.4892 304.766,114.812 310.477,141.416
  316.189,167.854 321.901,192.862 327.613,215.734 333.324,235.981 339.036,253.372 344.748,267.934 350.459,279.955 356.171,289.887 361.883,297.882 367.595,304.172
  373.306,308.98 379.018,312.492 384.73,314.859 390.441,316.216 396.153,316.656 401.865,316.257 407.577,315.079 413.288,313.164 419,310.541 424.712,307.224
  430.423,303.22 436.135,298.527 441.847,293.136 447.559,287.035 453.27,280.207 458.982,272.632 464.694,264.298 470.405,255.204 476.117,245.293 481.829,234.527
  487.54,222.885 493.252,210.371 498.964,197.01 504.676,182.85 510.387,167.958 516.099,152.427 521.811,136.369 527.522,119.919 533.234,103.264 538.946,86.7993
  544.658,71.0225 550.369,56.4904 556.081,43.821 561.793,33.6926 567.504,26.8443 573.216,24.0756 578.928,26.207 584.64,33.7032 590.351,46.6392 596.063,64.6238

  "/>
<polyline clip-path="url(#clip02)" style="stroke:#e9746a; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  30.6037,238.421 36.3154,244.036 42.0271,251.326 47.7388,259.782 53.4505,268.95 59.1622,278.455 64.8739,288 70.5856,297.36 76.2974,306.373 82.0091,314.93
  87.7208,322.961 93.4325,330.43 99.1442,337.322 104.856,343.638 110.568,349.394 116.279,354.612 121.991,359.317 127.703,363.537 133.414,367.303 139.126,370.643
  144.838,373.584 150.55,376.15 156.261,378.363 161.973,380.241 167.685,381.796 173.396,383.041 179.108,383.983 184.82,384.62 190.532,384.948 196.243,384.952
  201.955,384.61 207.667,383.895 213.378,382.762 219.09,381.161 224.802,379.026 230.514,376.27 236.225,372.788 241.937,368.455 247.649,363.128 253.36,356.654
  259.072,348.844 264.784,339.593 270.496,328.913 276.207,316.926 281.919,303.872 287.631,290.106 293.342,276.291 299.054,263.351 304.766,252.105 310.477,243.233
  316.189,237.283 321.901,234.508 327.613,234.743 333.324,237.726 339.036,243.08 344.748,250.304 350.459,258.78 356.171,267.86 361.883,277.301 367.595,286.822
  373.306,296.184 379.018,305.218 384.73,313.815 390.441,321.899 396.153,329.431 401.865,336.392 407.577,342.777 413.288,348.595 419,353.87 424.712,358.635
  430.423,362.914 436.135,366.736 441.847,370.13 447.559,373.122 453.27,375.736 458.982,377.993 464.694,379.909 470.405,381.492 476.117,382.761 481.829,383.73
  487.54,384.404 493.252,384.777 498.964,384.835 504.676,384.553 510.387,383.899 516.099,382.829 521.811,381.293 527.522,379.23 533.234,376.583 538.946,373.216
  544.658,368.973 550.369,363.716 556.081,357.323 561.793,349.69 567.504,340.731 573.216,330.375 578.928,318.595 584.64,305.642 590.351,291.963 596.063,278.191

  "/>
<polygon clip-path="url(#clip00)" points="
505.547,89.6215 578.063,89.6215 578.063,44.2615 505.547,44.2615
  " fill="#ffffff" fill-opacity="1"/>
<polyline clip-path="url(#clip00)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  505.547,89.6215 578.063,89.6215 578.063,44.2615 505.547,44.2615 505.547,89.6215
  "/>
<polyline clip-path="url(#clip00)" style="stroke:#0099ff; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  511.547,59.3815 547.547,59.3815
  "/>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:start;" transform="rotate(0, 553.547, 63.8815)" x="553.547" y="63.8815">y1</text>
</g>
<polyline clip-path="url(#clip00)" style="stroke:#e9746a; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  511.547,74.5015 547.547,74.5015
  "/>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:start;" transform="rotate(0, 553.547, 79.0015)" x="553.547" y="79.0015">y2</text>
</g>
</svg>

## Extra Features

The solution object acts both as an array and as an interpolation of the solution

```julia
@show sol.t[3] # Time at the 3rd timestep
@show sol[3] # Value at the third timestep
@show sol(5) # Value at t=5 using the interpolation
```

    sol.t[3] = 0.2927716363580929
    sol[3] = [0.768635,0.887673]
    sol(5) = [1.45932,0.99208]

    2-element Array{Float64,1}:
     1.45932
     0.99208

## Stochastic Differential Equations

Also included are problems for stochastic differential equations

```julia
g = function (t,u,du)
    du[1] = .5*u[1]
    du[2] = .1*u[2]
end
prob = SDEProblem(f,g,u0,tspan)
sol = solve(prob,dt=1/2^4)
plot(sol)
```

<?xml version="1.0" encoding="utf-8"?>
<svg xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink" width="600" height="400" viewBox="0 0 600 400">
<defs>
  <clipPath id="clip00">
    <rect x="0" y="0" width="600" height="400"/>
  </clipPath>
</defs>
<polygon clip-path="url(#clip00)" points="
0,400 600,400 600,0 0,0
  " fill="#ffffff" fill-opacity="1"/>
<defs>
  <clipPath id="clip01">
    <rect x="120" y="0" width="421" height="400"/>
  </clipPath>
</defs>
<polygon clip-path="url(#clip00)" points="
12.8259,384.952 596.063,384.952 596.063,3.93701 12.8259,3.93701
  " fill="#ffffff" fill-opacity="1"/>
<defs>
  <clipPath id="clip02">
    <rect x="12" y="3" width="584" height="382"/>
  </clipPath>
</defs>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  12.8259,379.237 12.8259,9.65223
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  129.473,379.237 129.473,9.65223
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  246.121,379.237 246.121,9.65223
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  362.768,379.237 362.768,9.65223
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  479.416,379.237 479.416,9.65223
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  596.063,379.237 596.063,9.65223
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  21.5745,289.031 587.314,289.031
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  21.5745,190.307 587.314,190.307
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:0.5; fill:none" stroke-dasharray="1, 2" points="
  21.5745,91.5819 587.314,91.5819
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  12.8259,384.952 596.063,384.952
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  12.8259,384.952 12.8259,379.237
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  129.473,384.952 129.473,379.237
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  246.121,384.952 246.121,379.237
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  362.768,384.952 362.768,379.237
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  479.416,384.952 479.416,379.237
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  596.063,384.952 596.063,379.237
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  12.8259,384.952 12.8259,3.93701
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  12.8259,289.031 21.5745,289.031
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  12.8259,190.307 21.5745,190.307
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  12.8259,91.5819 21.5745,91.5819
  "/>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 12.8259, 396.952)" x="12.8259" y="396.952">0</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 129.473, 396.952)" x="129.473" y="396.952">2</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 246.121, 396.952)" x="246.121" y="396.952">4</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 362.768, 396.952)" x="362.768" y="396.952">6</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 479.416, 396.952)" x="479.416" y="396.952">8</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:middle;" transform="rotate(0, 596.063, 396.952)" x="596.063" y="396.952">10</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 11.6259, 293.531)" x="11.6259" y="293.531">1</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 11.6259, 194.807)" x="11.6259" y="194.807">2</text>
</g>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:end;" transform="rotate(0, 11.6259, 96.0819)" x="11.6259" y="96.0819">3</text>
</g>
<polyline clip-path="url(#clip02)" style="stroke:#0099ff; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  12.8259,289.031 15.2547,285.555 17.9871,271.091 20.0577,260.652 21.8599,277.598 23.8873,276.197 26.1682,235.378 28.1848,249.724 30.4533,250.291 32.1303,253.122
  34.0168,261.697 36.1392,269.51 38.044,285.834 40.1868,269.612 42.1195,282.241 44.2937,281.669 46.2451,300.574 48.4403,298.743 50.91,300.12 53.6884,295.263
  56.4183,302.982 59.2495,294.903 61.8767,295.151 64.1527,304.205 66.7132,307.607 69.5937,288.551 72.8344,289.6 76.4801,281.999 79.5703,300.917 83.0469,315.528
  86.4277,321.448 89.1579,327.97 92.2294,337.077 95.6849,333.13 98.5458,343.083 101.764,343.478 105.04,335.322 108.491,334.327 112.374,332.281 116.742,340.609
  120.753,345.988 124.122,347.122 127.912,346.95 132.137,345.824 136.419,342.207 140.955,351.626 145.663,335.921 150.479,325.607 154.187,332.243 158.358,318.791
  163.051,307.873 168.33,292.482 173.472,288.511 177.825,277.087 181.446,233.112 185.487,201.551 187.549,196.83 189.869,184.046 192.478,204.104 195.119,211.782
  198.089,211.729 201.431,199.914 204.455,209.113 207.857,206.233 210.176,164.611 212.424,84.7984 214.076,94.2217 215.935,123.453 217.744,117.977 219.388,80.8382
  220.816,92.4258 222.297,82.6831 223.447,79.174 224.658,55.1382 225.64,49.7975 226.706,49.3293 227.812,86.1547 228.919,68.7543 229.826,60.5119 230.647,40.3618
  231.504,56.8479 232.43,67.7768 233.247,73.2855 234.064,69.5125 234.85,97.7165 235.735,108.025 236.618,92.9323 237.325,109.989 238.121,128.315 239.015,134.217
  239.908,136.441 240.7,146.284 241.538,145.118 242.255,176.759 243.063,167.665 243.872,177.717 244.529,213.147 245.269,214.114 246.101,232.541 247.037,238.364
  248.029,245.016 248.981,261.32 250.026,265.905 250.967,275.407 251.989,283.544 253.067,286.753 254.153,289.141 255.373,295.377 256.746,291.918 258.291,290.997
  259.692,288.392 260.834,293.531 262.12,300.166 263.34,308.53 264.714,306.683 266.259,308.874 267.997,305.739 268.95,316.433 270.023,321.732 271.23,326.576
  272.588,327.372 274.115,330.163 275.834,335.576 277.554,340.18 278.924,347.024 280.466,349.022 282.201,350.295 284.026,353.627 285.985,356.464 288.042,354.732
  289.697,355.991 291.558,358.201 293.653,358.768 295.984,356.945 298.358,358.202 300.662,363.084 303.022,356.296 305.546,358.556 307.956,363.093 310.493,365.449
  312.963,367.108 315.522,366.061 318.072,368.594 320.808,372.54 323.344,372.576 326.197,372.22 329.406,370.562 332.866,368.743 336.759,366.537 340.798,367.327
  345.086,365.262 349.212,362.044 353.715,356.335 358.288,355.707 362.955,356.48 366.83,358.043 371.189,365.136 376.092,368.054 381.317,368.09 387.195,366.99
  393.808,362.608 400.49,356.148 404.518,351.691 407.894,351.007 411.691,361.048 415.963,358.758 420.769,351.505 426.175,336.53 428.337,338.953 430.768,349.346
  433.504,346.417 436.581,340.021 440.043,351.648 443.459,355.916 447.011,346.171 451.006,340.598 455.501,328.401 457.495,325.927 459.739,316.381 462.263,304.037
  464.374,302.433 466.749,297.485 469.42,279.27 470.846,277.57 472.451,290.391 474.255,279.477 476.286,277.69 478.57,252.729 480.204,250.602 482.043,257.684
  484.063,259.928 485.786,274.853 487.724,264.456 489.904,258.149 491.635,260.679 493.582,255.459 495.524,262.065 497.709,254.302 499.667,242.582 501.87,230.321
  504.348,210.033 506.675,174.754 509.134,142.423 510.295,147.299 511.601,136.123 513.07,134.545 514.722,128.818 516.135,136.686 517.725,146.258 519.513,145.475
  521.254,158.531 523.12,129.831 524.444,147.545 525.933,155.552 527.609,119.773 529.009,122.046 530.585,133.074 532.271,151.25 534.164,115.142 536.262,83.037
  538.591,22.2008 539.639,20.056 540.817,43.2506 542.143,36.3693 543.635,82.7513 545.313,86.555 547.201,65.6558 549.139,50.4502 550.865,23.6953 552.549,3.93701
  553.932,46.5469 555.488,42.2913 557.239,107.597 559.208,35.3222 560.519,23.9705 561.994,56.5233 563.654,83.0762 565.503,93.7552 567.374,106.262 569.397,58.9519
  570.784,78.1817 572.345,94.9546 574.101,117.278 575.575,112.53 577.234,72.5824 578.282,56.5011 579.461,62.1492 580.575,69.12 581.704,57.9092 582.974,72.8398
  584.21,73.1061 585.285,82.0893 586.494,105.983 587.8,125.667 588.951,85.4489 589.869,86.3378 590.877,81.9909 591.831,73.0068 592.685,84.1248 593.56,83.4192
  594.395,99.3926 595.335,114.063 596.063,123.35
  "/>
<polyline clip-path="url(#clip02)" style="stroke:#e9746a; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  12.8259,289.031 15.2547,289.306 17.9871,289.809 20.0577,289.06 21.8599,290.918 23.8873,292.236 26.1682,292.322 28.1848,292.45 30.4533,291.401 32.1303,291.967
  34.0168,288.463 36.1392,290.045 38.044,290.429 40.1868,292.412 42.1195,294.572 44.2937,292.211 46.2451,292.652 48.4403,291.627 50.91,294.397 53.6884,298.009
  56.4183,301.128 59.2495,301.356 61.8767,299.898 64.1527,302.378 66.7132,306.726 69.5937,306.95 72.8344,307.991 76.4801,309.305 79.5703,309.651 83.0469,312.696
  86.4277,312.53 89.1579,314.95 92.2294,316.947 95.6849,319.737 98.5458,324.118 101.764,327.467 105.04,333.819 108.491,335.146 112.374,336.993 116.742,339.713
  120.753,341.734 124.122,345.189 127.912,348.952 132.137,351.621 136.419,354.631 140.955,357.011 145.663,360.315 150.479,362.34 154.187,364.263 158.358,364.833
  163.051,366.587 168.33,366.774 173.472,366.545 177.825,366.776 181.446,365.689 185.487,364.985 187.549,363.95 189.869,363.79 192.478,362.237 195.119,361.991
  198.089,361.591 201.431,360.1 204.455,358.613 207.857,356.538 210.176,353.591 212.424,349.943 214.076,346.456 215.935,343.057 217.744,340.571 219.388,336.844
  220.816,334.436 222.297,331.061 223.447,328.242 224.658,325.331 225.64,323.297 226.706,319.702 227.812,317.396 228.919,313.861 229.826,309.648 230.647,306.32
  231.504,302.824 232.43,296.647 233.247,294.031 234.064,290.84 234.85,289.056 235.735,285.047 236.618,280.571 237.325,278.888 238.121,274.834 239.015,270.578
  239.908,263.897 240.7,259.034 241.538,256.19 242.255,252.731 243.063,249.966 243.872,248.396 244.529,248.355 245.269,248.64 246.101,246.563 247.037,243.308
  248.029,242.614 248.981,243.084 250.026,242.753 250.967,242.001 251.989,239.914 253.067,243.332 254.153,246.633 255.373,249.483 256.746,249.828 258.291,253.043
  259.692,252.751 260.834,248.464 262.12,249.591 263.34,249.601 264.714,250.033 266.259,253.38 267.997,258.365 268.95,259.521 270.023,259.286 271.23,259.586
  272.588,258.578 274.115,259.726 275.834,261.112 277.554,265.551 278.924,267.865 280.466,274.978 282.201,278.836 284.026,282.412 285.985,285.174 288.042,288.125
  289.697,293.871 291.558,297.758 293.653,301.848 295.984,306.086 298.358,308.508 300.662,312.321 303.022,313.88 305.546,316.392 307.956,320.755 310.493,323.697
  312.963,327.279 315.522,331.331 318.072,335.92 320.808,339.55 323.344,343.827 326.197,347.244 329.406,350.359 332.866,352.412 336.759,354.964 340.798,356.853
  345.086,358.932 349.212,362.022 353.715,363.816 358.288,366.672 362.955,369.313 366.83,371.006 371.189,372.479 376.092,374.074 381.317,375.042 387.195,376.826
  393.808,378.15 400.49,379.453 404.518,379.98 407.894,380.752 411.691,381.272 415.963,381.939 420.769,382.325 426.175,382.866 428.337,382.991 430.768,383.26
  433.504,383.444 436.581,383.687 440.043,383.684 443.459,383.839 447.011,384.14 451.006,384.188 455.501,384.422 457.495,384.578 459.739,384.541 462.263,384.667
  464.374,384.703 466.749,384.708 469.42,384.843 470.846,384.896 472.451,384.936 474.255,384.952 476.286,384.855 478.57,384.796 480.204,384.745 482.043,384.698
  484.063,384.658 485.786,384.673 487.724,384.764 489.904,384.788 491.635,384.786 493.582,384.684 495.524,384.619 497.709,384.631 499.667,384.615 501.87,384.545
  504.348,384.366 506.675,384.154 509.134,383.911 510.295,383.695 511.601,383.553 513.07,383.343 514.722,383.008 516.135,382.819 517.725,382.671 519.513,382.39
  521.254,382.301 523.12,382.173 524.444,381.963 525.933,381.864 527.609,381.693 529.009,381.372 530.585,381.002 532.271,380.685 534.164,380.227 536.262,379.594
  538.591,378.537 539.639,377.983 540.817,377.068 542.143,376.045 543.635,375.099 545.313,374.083 547.201,373.195 549.139,371.786 550.865,370.07 552.549,367.907
  553.932,366.2 555.488,364.672 557.239,362.82 559.208,361.332 560.519,360.024 561.994,357.834 563.654,355.576 565.503,354.801 567.374,352.548 569.397,349.609
  570.784,347.673 572.345,344.603 574.101,341.508 575.575,339.67 577.234,336.725 578.282,334.288 579.461,332.173 580.575,330.91 581.704,329.649 582.974,326.446
  584.21,321.906 585.285,319.492 586.494,316.816 587.8,312.422 588.951,310.134 589.869,306.361 590.877,303.459 591.831,299.637 592.685,295.448 593.56,292.555
  594.395,290.783 595.335,286.224 596.063,284.1
  "/>
<polygon clip-path="url(#clip00)" points="
505.547,70.177 578.063,70.177 578.063,24.817 505.547,24.817
  " fill="#ffffff" fill-opacity="1"/>
<polyline clip-path="url(#clip00)" style="stroke:#00002d; stroke-width:0.8; stroke-opacity:1; fill:none" points="
  505.547,70.177 578.063,70.177 578.063,24.817 505.547,24.817 505.547,70.177
  "/>
<polyline clip-path="url(#clip00)" style="stroke:#0099ff; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  511.547,39.937 547.547,39.937
  "/>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:start;" transform="rotate(0, 553.547, 44.437)" x="553.547" y="44.437">y1</text>
</g>
<polyline clip-path="url(#clip00)" style="stroke:#e9746a; stroke-width:2.4; stroke-opacity:1; fill:none" points="
  511.547,55.057 547.547,55.057
  "/>
<g clip-path="url(#clip00)">
<text style="fill:#00002d; fill-opacity:1; font-family:Arial,Helvetica Neue,Helvetica,sans-serif; font-size:12; text-anchor:start;" transform="rotate(0, 553.547, 59.557)" x="553.547" y="59.557">y2</text>
</g>
</svg>

## Documentation

For more information, see the documentation: https://github.com/JuliaDiffEq/DifferentialEquations.jl

## Problems

### Problem 1

The DifferentialEquations.jl algorithms choose the number type of their calculation given their input. Use this fact to solve the [Lorenz equation](https://en.wikipedia.org/wiki/Lorenz_system) using BigFloats. You may want to [check out the example notebooks](https://github.com/JuliaDiffEq/DifferentialEquations.jl/tree/master/examples).

### Problem 2

Use the [event handling](http://juliadiffeq.github.io/DifferentialEquations.jl/latest/man/callback_functions.html) the model a bouncing ball with friction, i.e. at every bounce the velocity flips but is decreased to 80%. Does the ball eventually stop bouncing?

### Problem 3

Install the ODE.jl and ODEInterface conditional dependencies (note: ODEInterface may give build issues on Windows!) and run some of the [benchmarks notebooks](https://github.com/JuliaDiffEq/DifferentialEquations.jl/tree/master/benchmarks) on your machine. Do you notice any trends amongst the algorithms? Use the method as shown in the Pleiades benchmarks to benchmark the algorithms against each other on nonlinear problems with no known analytical solution. Try building an example problem with a large number of independent variables to accentuate the differences between the algorithms (Example: the Linear problem in the benchmarks is a 100x100 problem).


# Homework 2

## Imports

```python
import nbtools
nbtools.setup_nb()
```

```python
import itertools
import sympy
from sympy.diffgeom import Manifold, Patch
from pystein import coords, metric, curvature
from pystein.utilities import tensor_pow as tpow, full_simplify
```

## Problem 2

```python

```

```python
M = Manifold('M', dim=3)
P = Patch('origin', M)
r, theta, phi = sympy.symbols('r theta phi', nonnegative=True)
cs = coords.CoordSystem('spherical-polar', P, [r, theta, phi])
dr, dtheta, dphi = cs.base_oneforms()

ds2 = tpow(dr, 2) + r ** 2 * tpow(dtheta, 2) + r ** 2 * sympy.sin(theta) ** 2 * tpow(dphi, 2)
gamma = metric.Metric(twoform=ds2)

gamma
```

$\displaystyle r^{2} \sin^{2}{\left(\theta \right)} \operatorname{d}\phi \otimes \operatorname{d}\phi + r^{2} \operatorname{d}\theta \otimes \operatorname{d}\theta + \operatorname{d}r \otimes \operatorname{d}r$

```python
christoffels = [((i, j, k), full_simplify(curvature.christoffel_symbol_component(i, j, k, gamma)))
                for i, j, k in itertools.product(range(3), range(3), range(3))]
```

```python
curvature.display_components([c for c in christoffels if c[1]])
```

$\begin{split} (\mu=0, \nu=1, \rho=1) &: - r \\ (\mu=0, \nu=2, \rho=2) &: - r \sin^{2}{\left(\theta \right)} \\ (\mu=1, \nu=0, \rho=1) &: \frac{1}{r} \\ (\mu=1, \nu=1, \rho=0) &: \frac{1}{r} \\ (\mu=1, \nu=2, \rho=2) &: - \frac{\sin{\left(2 \theta \right)}}{2} \\ (\mu=2, \nu=0, \rho=2) &: \frac{1}{r} \\ (\mu=2, \nu=1, \rho=2) &: \frac{1}{\tan{\left(\theta \right)}} \\ (\mu=2, \nu=2, \rho=0) &: \frac{1}{r} \\ (\mu=2, \nu=2, \rho=1) &: \frac{1}{\tan{\left(\theta \right)}} \end{split}$

```python
riccis = [((i, j), full_simplify(curvature.ricci_tensor_component(i, j, gamma).doit()))
          for i, j in itertools.product(range(3), range(3))]
```

```python
curvature.display_components(riccis)
```

$\begin{split} (\mu=0, \nu=0) &: 0 \\ (\mu=0, \nu=1) &: 0 \\ (\mu=0, \nu=2) &: 0 \\ (\mu=1, \nu=0) &: 0 \\ (\mu=1, \nu=1) &: 0 \\ (\mu=1, \nu=2) &: 0 \\ (\mu=2, \nu=0) &: 0 \\ (\mu=2, \nu=1) &: 0 \\ (\mu=2, \nu=2) &: 0 \end{split}$

 Computing $\beta_j$

```python
M = sympy.symbols('M')
```

```python
beta_j = sympy.Matrix([sympy.sqrt(2 * M / r), 0, 0])
beta_j
```

$\displaystyle \left[\begin{matrix}\sqrt{2} \sqrt{\frac{M}{r}}\\0\\0\end{matrix}\right]$

```python
(gamma.matrix * beta_j).T
```

$\displaystyle \left[\begin{matrix}\sqrt{2} \sqrt{\frac{M}{r}} & 0 & 0\end{matrix}\right]$

```python
def cov_deriv_one_form(i, j, form, g):
    coord_syms = g.coord_system.base_symbols()
    v = sympy.Derivative(form[j], coord_syms[i])

    for k in range(len(coord_syms)):
        v -= curvature.christoffel_symbol_component(k, i, j, g) * form[k]

    return full_simplify(v.doit())
```

```python
def k_ij(i, j, beta_j, g, alpha = 1):
    d_i_beta_j = cov_deriv_one_form(i, j, beta_j, g)
    d_j_beta_i = cov_deriv_one_form(j, i, beta_j, g)
    return - sympy.Rational(1, 2) / alpha * (d_i_beta_j + d_j_beta_i)
```

```python
K_ij = sympy.Matrix([[k_ij(0, j, beta_j, gamma) for j in range(3)],
                     [k_ij(1, j, beta_j, gamma) for j in range(3)],
                     [k_ij(2, j, beta_j, gamma) for j in range(3)]])
```

```python
K_ij = - (d_i_beta_j + d_j_beta_i) / 2
K_ij #/ beta_j[0]
```

$\displaystyle \left[\begin{matrix}\frac{\sqrt{2} \sqrt{\frac{M}{r}}}{2 r} & 0 & 0\\0 & - \sqrt{2} r \sqrt{\frac{M}{r}} & 0\\0 & 0 & - \sqrt{2} r \sqrt{\frac{M}{r}} \sin^{2}{\left(\theta \right)}\end{matrix}\right]$

$$K_{ij} \approx \frac{1}{r} \beta^r \gamma_{ij}$$

```python
sympy.trace(gamma.matrix.inv() * K_ij)
```

$\displaystyle - \frac{3 \sqrt{2} \sqrt{\frac{M}{r}}}{2 r}$

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```

```python

```


$\nabla ^2  \mathbf{E} + k^2 \mathbf{E} = 0$

$\nabla ^2 \mathbf{H} + k^2 \mathbf{H} = 0$

$ k = \sqrt{\omega ^2 \mu \varepsilon - i \omega \mu \sigma } $

$k_{ground} \simeq (1-i) \sqrt{ \frac{\omega \mu \sigma}{2} }$

$k_{air} \simeq \omega \sqrt{ \mu_0 \varepsilon_0}$

$\begin{split}\left(\begin{matrix} E_{x} \\ E_{y} \end{matrix} \right) = \left(\begin{matrix} \hat{Z}_{xx} & \hat{Z}_{xy} \\ \hat{Z}_{yx} & \hat{Z}_{yy} \end{matrix} \right) \left(\begin{matrix} H_x \\ H_y \end{matrix} \right)\end{split}$

$\begin{split} H_z = \left(\begin{matrix} T_{zx} & T_{zy} \end{matrix} \right) \left(\begin{matrix} H_x \\ H_y \end{matrix} \right)\end{split}$

$\delta = \sqrt{ \frac{2}{\omega \mu \sigma}} \simeq \frac{500}{\sqrt{\sigma f}}$

$ \phi_d = \sum \left(\frac{d^{obs}_i-d^{pred}_i}{\varepsilon} \right) ^2 $

$\phi = \phi_d +\beta \phi_m$

```latex
%%latex
\begin{align}
\curl \vf{E} + i \omega \vf{B} &= 0   \\
\curl \frac{1}{\mu} \vf{B} - \sigma \vf{E} &=   \vf{s}_E
\end{align}
```

\begin{align}
\curl \vf{E} + i \omega \vf{B} &= 0   \\
\curl \frac{1}{\mu} \vf{B} - \sigma \vf{E} &=   \vf{s}_E
\end{align}

```python

```


##### Eletroacústica I - Analogias Elétricas

# Exemplo 1
## Circuito RC com transformador

$R = 1000\ \Omega$

$C = 24\ \mu F$

$\alpha = 3$

## Em termos das impedâncias

## Aplicando a transformação de tensão

## Aplicando a lei de Kirchhoff das tensões (LKT)

$ V - RI - \large\frac{\alpha^{2}}{\text{j}\omega C} I = 0$

$ V = I\left(R + \large\frac{\alpha^{2}}{\text{j}\omega C} \right)$

\begin{equation}
 I = \large\frac{V}{R + \frac{\alpha^{2}}{\text{j}\omega C}}
 \tag{1}
\end{equation}

## A tensão no capacitor é

$ \alpha V_{C} = \large \frac{\alpha^{2}}{\text{j}\omega C} I$

Aplicando a Eq. (1)

$ \alpha V_{C} = \large\frac{\alpha^{2} V}{\text{j}\omega C \left(R + \frac{\alpha^{2}}{\text{j}\omega C}\right)}$

$ V_{C} = \large \frac{\alpha V}{\text{j}\omega RC  + \alpha^{2}}$

## Assim, podemos obter a resposta em frequência

$ H_{c}(\omega) = \large \frac{V_{C}}{V} $

$ H_{c}(\omega) = \large \frac{\alpha V}{\left(\text{j}\omega RC  + \alpha^{2}\right)V} $

\begin{equation}
 H_{c}(\omega) = \large \frac{\alpha }{\text{j}\omega RC  + \alpha^{2}}
 \tag{2}
\end{equation}

Frequência de corte do filtro:

$\omega_{cut} = \large \frac{\alpha^{2}}{RC}$

$f_{cut} = \large \frac{\alpha^{2}}{2\pi RC} $

```python
import numpy as np
from matplotlib import pyplot as plt
plt.rcParams.update({'font.size': 14})
```

```python
# Parametros
R = 1e3  # 1 kOhm
C = 1e-6  # 1 uF
alfa = 3
fcut = alfa**2/(2 * np.pi * R*C)
print('Frequência de corte é {:.2f} Hz'.format(fcut))

# Frequencias
freqs = np.linspace(1, 10000, 1001)
omega = 2 * np.pi * freqs

# FRF
Hc = alfa / (1j * omega * R * C + alfa**2)

# plot
fig, axs = plt.subplots(2,1, figsize = (7,5))
axs[0].semilogx(freqs, 20*np.log10(np.abs(Hc)), linewidth = 2)
axs[0].grid(linestyle = '--', which='both')
axs[0].set_ylim((-40, 5))
axs[0].set_xlim((1, 10000))
axs[0].set_ylabel(r'$|H(\omega)|$ [dB]')
axs[1].semilogx(freqs, np.rad2deg(np.angle(Hc)), linewidth = 2)
axs[1].grid(linestyle = '--', which='both')
axs[1].set_ylabel(r'$\angle H(\omega)$ [$^{\circ}$] ')
axs[1].set_xlabel(r'Frequência [Hz]')
axs[1].set_xlim((1, 10000))
fig.tight_layout()
```


# Announcements
- Problem Set 6 will be on finite difference methods, due Nov 2 (to be posted on D2L).
- __Outlook__: Next week, we'll move on to Monte Carlo Markov Chain methods.

*Note:  The presentation below largely follows part I in "Finite Difference Methods for Ordinary and Partial Differential Equations" by LeVeque (SIAM, 2007).*

# The Heat Equation
Numerical solutions to partial differential equations (PDEs) are a large subfield of applied math, and we won't be able to cover a full survey in this course. Today, we'll solve the heat equation
$$
\partial_{t}u(\vec x,t) = \alpha \vec \nabla^2 u(\vec x,t)\,,
$$
which describes how a quantity $u$ (such as heat) diffuses through a medium, and
with $\alpha$ a positive constant, called _diffusivity_ of the medium. The heat equation is among the most widely studied PDEs, and a prototype for parabolic PDEs.

Today we will numerically solve an example in 1D, applying the finite difference method from Lecture 23 and 24 for the spatial solution, combined with a forward Euler step for the time solution.

## Example Problem
Let consider the 1D heat equation on a wire with extend $x\in[0,1]$,
\begin{equation}
    \partial_{t}u = \partial^2_{x}u , \quad 0 < x < 1, \quad t > 0 \\
\end{equation}
with boundary conditions
\begin{equation}
    u(0,t) = 0, \quad {u}(1,t) = 0\,,\\
\end{equation}
which correspond to endpoints held at constant temperature, and inital temperature distribution
\begin{equation}
    u(x, 0) = 10\sin(\pi x)\,.
\end{equation}

# Discretization
Lecture 23 introduced the _Finite Difference_ method and we used it to solve a boundary value problem with Dirichlet conditions as a linear system.

Let's use the subscript $i$ to indicate the spatial discretization, and subscript $J$ for the time discretization, so that $U_{i,J} = u(x_i,t_J)$.
Today we will incorporate the Dirichlet conditions as ghost cells, $U_{0,J}$ and $U_{N+1,J}$, corresponding to the boundary points.

Then the spatial discretization (at one slice in time) becomes
$$
    A = \frac{1}{\Delta x^2} \begin{bmatrix}
        \Delta x^2 & 0  \\
        1 & -2 & 1 \\
          &  1 & -2 & 1 \\
          &    & \ddots & \ddots & \ddots \\
          &    &        &      1 &     -2 & 1 \\
          &    &        &        &      1 & -2 & 1 \\
          &    &        &        &        &  0 & \Delta x^2
    \end{bmatrix} \quad \quad U_J = \begin{bmatrix}
        U_{0,J} \\ U_{1,J} \\ \vdots \\ U_{N,J} \\ U_{N+1,J}
    \end{bmatrix} \quad \quad b_J = \begin{bmatrix}
        u(0,t_J)=0 \\ f(x_1,t_J) \\ \vdots \\ f(x_{N},t_J) \\ u(1,t_J) =0\,.
    \end{bmatrix}
$$
_Note that $A$ is time independent!_

The next step is to discretize in time
$$
\frac{U_{i,J+1} -U_{i,J}}{\Delta t} = U^{\prime\prime}_{i,J} = \frac{U_{i-1,J}-2U_{i,J}+U_{i+1,j}}{\Delta x^2}\,,
$$

 **Solve today's Example Problem via the construction of a linear system of equations and time stepping.**

 Your expression for $U_{i,J+1}$ in terms of elements of $U_J$:

_The code below is adapted from Lecture 23. The missing parts need to be adapted to today's problem, but you may find looking back at Lecture 23 helpful._

```python
import numpy
import matplotlib.pyplot as plt
# Problem setup
f_t0 = lambda x: 10.*numpy.sin(numpy.pi*x)
#spatial discretization
a = 0.0
b = 1.0
u_a = 0.0
u_b = 0.0

# Spatial Discretization
N_x = 10
#keep boundary points
x = numpy.linspace(a, b, N_x + 2)
delta_x = (b - a) / (N_x + 1)

#time discretization
t_0 = 0
t_final = 10
N_t = 100
delta_t = (t_final-t_0)/(N_t+1)

# Construct matrix A
A = numpy.zeros((N_x + 2, N_x + 2))
diagonal = numpy.ones(N_x + 2) / delta_x**2
A += numpy.diag(diagonal * -2.0, 0)
A += numpy.diag(diagonal[:-1], 1)
A += numpy.diag(diagonal[:-1], -1)

# Now, add boundary conditions for ghost cells to A
#left boundary
#right boundary
# Construct RHS - without boundary conditions
b = f_t0(x)
# add boundary conditions to b
#left boundary
#right boundary

#U[J,:] will contain solution at time J
U = numpy.empty((N_t+2,N_x + 2))
#spatial solution
U[0,:] = numpy.linalg.solve(A, b)
#now do time iteration
for J in range(0,N_t+2):
    #boundary conditions
    for i in range (1,N_x+1):

```

```python

```


# *FEniCS tutorial:* Diffusion of a Gaussian hill in a two-dimensional disc

In this demo, we solve the two-dimensional diffusion equation with Dirichlet
bounadary conditions $u_D$ and source term $f = 0$. The initial condition,
$u_0$, we define as a Gaussian hill.

$$
\begin{align}
  u'   &= \nabla^2 u + f \quad\text{in the unit circle} \\
  u    &= u_D  \hphantom{u+f}\quad\text{on the boundary} \\
  u    &= u_0  \hphantom{u+f}\quad\;\text{at $t = 0$} \\
\end{align}
$$
with
$$
\begin{align}
  u_D  &= 0 \\
  u_0  &= \exp\bigl(-a(x^2 + y^2)\bigr) \\
  f    &= 0
\end{align}
$$

```python
%matplotlib inline
import matplotlib.pyplot as plt
import numpy as np
import time as tm

# import os
# import sys
# import re
# from IPython.display import Image
# from IPython.core.interactiveshell import InteractiveShell
# InteractiveShell.ast_node_interactivity = "all"
```

```python
from fenics import *
from mshr import *
```

```python
## set up the problem and define the main simulation function evolve()

# simulation parameters
# -- time step
T = 0.25          # total simulation time
n_steps = 15      # number of time steps
dt = T / n_steps  # size of time step
nip = 3           # number of intervals between plots
# -- mesh density
md = 24           # mesh density within disc

# create mesh and define function space
domain = Circle(Point(0, 0), 1)
mesh = generate_mesh(domain, md)
V = FunctionSpace(mesh, 'P', 2)

# define boundary condition
def boundary(x, on_boundary):
    return on_boundary

bc = DirichletBC(V, Constant(0), boundary)

# define initial value
u_0 = Expression('exp(-a*pow(x[0] - x0, 2) - a*pow(x[1] - y0, 2))',
                 degree=2, x0=0., y0=0.30, a=5)
u_n = interpolate(u_0, V)

u_min = 0.
u_max = 1.

# define variational problem
u = TrialFunction(V)
v = TestFunction(V)
f = Constant(0)
F = u*v*dx + dt*dot(grad(u), grad(v))*dx - (u_n + dt*f)*v*dx
a, L = lhs(F), rhs(F)

# define time-evolution function
def evolve():

    # report initial state
    yield u_n

    # time-stepping
    u = Function(V)
    t = 0
    for n in range(1, n_steps + 1):

        # update current time
        t += dt

        # compute solution
        solve(a == L, u, bc)

        # report solution at nip-step intervals
        if n % nip == 0:
            yield u

        # update previous solution
        u_n.assign(u)
```

```python
n_rows = 3
n_cols = 5
fig_wd = 15
# default sizing here yields unit aspect ratio
plt.figure(figsize = (fig_wd, fig_wd * n_rows // n_cols))
plt.subplots_adjust(wspace=0.3, hspace=0.3)

idx = 0
for u in evolve():
    idx += 1
    plt.subplot(n_rows, n_cols, idx)
    plot(u, vmin=u_min, vmax=u_max)
```

```python
mesh
```

```python
u
```

```python
plot(u)
```

```python
matplotlib.tri.tricontour.TriContourSet?
```

```python
??plot
```

```python
??_plot_matplotlib
```

```python

```