# Derivation of Likelihood for 21cmMC

Here we will attempt to derive a top-level description of a likelihood function for 21cmMC, using the power spectrum approach. Along with this, we will highlight the various assumptions made, so that we can go off and test them.

## MCMC

The basic idea of MCMC is provide a solution for Bayes' theorem. The question that Bayes theorem answers is that of the probability of a given model parameterisation, given some data. That is, we want to know how probable a given set of parameters of a model is, given we have some data to test it on. Formally:

\begin{equation}
    P(\vec{\theta}|D) \propto P(D|\vec{\theta}) P(\vec{\theta}).
\end{equation}

The last factor is termed the "prior" and specifes our level of belief in a given set of parameter values without the given data, and the $P(D|\vec{\theta})$ is the likelihood. MCMC gives the answer to the LHS by choosing multiple values of $\theta$ in succession in a smart way, such that the final collection of $\theta$ itself forms the distribution that defines $P(\vec{\theta}|D)$.

So, all we need to do is formally specify the likelihood, $P(D|\vec{\theta})$ (hereafter $P_D$).

Keep in mind that $\vec{\theta}$ consists of *anything* that is a priori uncertain in the model. This includes unknown physical parameters, but also the unknown positions of every galaxy, or the unknown uncertainties on parameters. All of these need to be *specified* to determine $P_D$, and the resulting distribution will have as many dimensions as there are parameters (and there may be millions). Luckily, we usually don't care about many of the parameters, and we can easily ignore them properly by *marginalising* over them (i.e. integrating over those dimensions). This happens for free in MCMC, as long as we let those parameters vary freely.

## Likelihood of Lightcones

The first obvious idea for a likelihood of a 21cm lightcone might be to compare the brightness at any image voxel with a model. This is a *bad* idea for a couple of reasons. The first is that the brightnesses between voxels are clearly highly correlated (voxels close together are likely to be similar in brightness, and even more distant ones will be correlated because of cosmology). It's always nicer to deal with independent variables.

The second reason is that we don't *really* care about exactly how the sky looks. What we care about are the physical parameters. For each set of physical parameters, there are a whole family of skies that are possible (dependent on initial conditions). We don't care if we choose the exact sky from this family, we only care about what the family is, and how likely it is that our sky is a part of that family.

**Aside:** if we *did* know the initial conditions, or some information about them, then we could use that information to pare down the size of the families, and that would be useful. We will assume that we have no idea about the initial conditions.

One quantity which at least *claims* to do better on each of these points is the power spectrum. The power spectrum (at least the cosmological one) is expected to be roughly independent in every $k$ mode (whether this is still true after interaction with a radio telescope, I'm not sure, but let's go with it for now). Secondly, the power spectrum does *not* contain extra hidden parameters (i.e. initial conditions), since the phases of the FT are eliminated when going to a power spectrum. Thus we have a deterministic mapping between the physical parameters and the actual power spectrum distribution at a given wavemode $\vec{k}$.

So, should we use the 3D, 2D, or 1D power spectrum? Well, so long as each wavemode is statistically equivalent (i.e. is independent and identically distributed), it does not matter at all. If you were to calculate the likelihood as the sum over every bin, or the likelihood of a single averaged bin, you'd get the same answer. Conversely, you *lose* information if you unwittingly average variables that aren't equivalent.

The cosmological signal should be very close to isotropic, and so we can just as easily use the 1D power spectrum, though there is no true benefit in doing so. Nevertheless, the foregrounds are not isotropic in the parallel direction, and if they cannot be extricated from the signal, it is best to do the analysis at least in 2D. We will assume that both signal and foregrounds are angularly isotropic.

## Likelihood using 2D PS

Lets say we have some data which has a 2D PS we'll call $P_{u\eta}$. This data contains foreground and instrumental noise, and is measured with an instrument (so different scales are pushed around a bit by the frequency-dependent array).

*Though it's probably not true*, let's assume that each of the 2D bins of $P_{u\eta}$ is independent and has a Gaussian distribution. **We should check if this holds**.

Then the likelihood is very simple:

\begin{equation}
    \ln \mathcal{L} = -\frac{1}{2} \sum_{u,\eta} \frac{(P_\theta(u,\eta) - P_{u\eta})^2}{\sigma^2(\theta)}.
\end{equation}

Here $P_\theta$ is the expected power, given the model parameters, and $\sigma^2$ is the expected variance for those parameters.

Now, if we assume that the signal and foregrounds are independent (and I think this is a good assumption at this point), then the power spectrum can be broken up into those pieces. Since we have already assumed that the distribution is Gaussian, then each of those components *must* be Gaussian (again, this might not be true). Each of the pieces is convolved with the instrument, which we take for granted through the rest of the analysis, but we also note that the instrument adds Gaussian noise, so we get:

\begin{equation}
    \ln \mathcal{L} = -\frac{1}{2} \sum_{u,\eta} \frac{(P_{\rm 21} + P_{\rm FG} + P_N - P_{u\eta})^2}{\sigma^2_{\rm 21} + \sigma^2_{\rm FG} + \sigma^2_N}.
\end{equation}

### How to calculate this

This is a pretty simple equation. The only problem is in actually computing it. Note that if the foreground model and instrument are *known*, then they do not change on any given iteration, and they can be pre-computed. This means we can come up with an analytical model for them (remember we need the *expected* power and variance), or just run a bunch of simulations to calculate them.

If we don't know them (there are unknown foreground parameters, for example), then we need to calculate them on every iteration. The best way to do this is by using an analytical model. But what if we don't have one, or it is too difficult to calculate?

Indeed, we *don't* have an analytical model for the 21cm signal. It is modified in real space by all sorts of physical effects that depend on the initial conditions. The only way to get it would be to run a whole bunch of simulations with the same parameters, and get the signal and its variance. But that would be only for a single set of parameters, so on a given iteration -- with different parameters -- it would be incorrect.

I'm not yet quite sure of the best way to deal with this. I'm not sure there is *any* **correct** way, except for actually defining a model power spectrum and its distribution for the 21cm signal.

However, we can do *something*, by estimating the expected power as the average within the given UV annulus, and likewise for the variance. We could also use this method for the foregrounds if we didn't have an analytic model and required one.

Do note that this method is *not correct*: the hope would be that by estimating it like this, but continuing to marginalise over realisations, that the uncertainty would be washed out. However, this is not strictly the case. To do this, one requires to specify the exact probability of the data given the model choice (including the initial conditions). But this is only an *estimate* of that quantity. At best we can hope that the estimate is very close to the truth and has very small Mean Standard Error.

## Does this likelihood do what we want it to?

First of all, it should be pointed out that, as long as we use a power spectrum (and assume Gaussianity etc.), then this is the *only* likelihood we can arrive at. It's not a choice. Doing a 1D or 3D PS is exactly equivalent (so long as we properly account for covariances etc.). A 3D PS is less convenient because we probably anyway need to determine $\sigma_{\rm 21}$ by averaging over annuli in the UV plane, so we might as well just do the average.

Perhaps the question could be restated as "is this the best kind of likelihood, to be able to constrain parameters?". The answer to that question is yes, **if** our above assumptions are true -- i.e. that the statistics of successive frequencies are the same, and that the density perturbations are Gaussian. Neither of these assumptions is true, of course. In that case, while we can *use* this likelihood, it is not the best one to use -- it kills information. But we can worry about that later.

The point is, this likelihood behaves like we would expect. Note that we don't "suppress" the foregrounds in the data. Nevertheless, this suppression is exactly there. In wedge-modes, the power spectrum is dominated by foregrounds, and so it is much easier to get close to the data by small changes in the foregrounds, rather than the signal. However, these modes are significantly down-weighted by the larger values of $\sigma^2_{\rm FG}$ here. This is exactly the quadratic suppression.

## Summary

Finally, a summary. In our case, it seems that our likelihood is

\begin{equation}
    \ln \mathcal{L} = -\frac{1}{2} \sum_{u,\eta} \frac{(P_{\rm 21} + P_{\rm FG} + P_N - P_{u\eta})^2}{\sigma^2_{\rm 21} + \sigma^2_{\rm FG} + \sigma^2_N}.
\end{equation}

and best done in 2D space. The assumptions made here are:

* The power spectrum is independent for each $u, \eta$ co-ordinates.
* The power spectrum has a Gaussian distribution.
* Foregrounds are independent of signal, even after convolution with the instrument.
* The above imply that both the foregrounds and signal have independent, Gaussian distributions.

These assumptions can be checked, and we should do so.

Also, we noted that unless we have an exact physical prediction of $\sigma_{\rm 21}$, this likelihood cannot work in detail. But pending other ideas, we'll press on with estimating it per-iteration, and hope that the inaccuracy is small.

### 1D Case

It would also be nice to *explicitly* derive the likelihood in the case that we average down to 1D. I cannot see how it would be different to the 2D case, but it might be useful to express things in this way, as it would mean better data-compression (i.e. fewer bins to sum over). Could you have a go at this, Bella?

## Likelihood from a simulation

The most tricky thing about our likelihood is that at least one of the models is only computable via simulation, and every time the parameters are visited, its output will change. The uncertainties from such an approximation should be propagated through, but doing so is rather tricky.

To test our ideas about doing this, we can set up a basic test. Let's try to estimate the two parameters $(a,b)$ where our model is

$$ f(x) = ax + bx^2 + \mathcal{N}(0, ab)$$.

We will simulate some data with this model, and then fit the data with the exact model. Then we will fit it, assuming that we don't really know the model, but can simulate it. We'll do this in the simple way, and then also using

from https://arxiv.org/pdf/1511.05969.pdf.

The idea is to test if the resulting statistics are consistent or biased.

```python
import numpy as np
%matplotlib inline
import matplotlib.pyplot as plt
import corner

import emcee
from emcee.utils import sample_ball
from scipy.special import gamma
```

```python
a,b = 4, 3
x = np.arange(1,12)

data = a + b*x**2 + np.random.normal(scale=x**2, size=len(x))
```

```python
plt.plot(x, a+b*x**2)
plt.errorbar(x, data, yerr=a*b)
```

```python
def lnl_deterministic(params, x, data):
    a,b = params

    # Make sure we have positive sigma.
    if a*b <= 0:
        return -np.inf

    model = a + b*x**2

    out = np.sum(-0.5 * (data - model)**2 / (x**2))

    if np.isnan(out):
        print("%s %s returned nan"%(a,b))

    return out
```

```python
sampler = emcee.EnsembleSampler(
    nwalkers = 50,
    dim = 2,
    lnpostfn = lnl_deterministic,
    args = (x, data),
    threads = 6
)
```

```python
pos0_initial = sample_ball([a,b], [1.0, 1.0], size=50)
```

```python
# burnin
pos0, lnprob, rstate = sampler.run_mcmc(pos0_initial, 100)

# actual sampling
sampler.reset()
sampler.run_mcmc(pos0, N=1000, rstate0=rstate, lnprob0=lnprob);
```

```python
corner.corner(sampler.flatchain, labels=['a','b'], truths=[a,b]);
```

```python
def lnl_simulated(params, x, data, n):
    a,b = params

    simulated_data = a + b*x**2 + np.random.normal(scale=x**2, size=(n,len(x)))

    fbar = np.mean(simulated_data, axis=0)
    s = np.std(simulated_data, axis=0)

    return np.sum(-0.5 * (data - fbar)**2 / (s**2))
```

```python
sampler_sim = emcee.EnsembleSampler(
    nwalkers = 50,
    dim = 2,
    lnpostfn = lnl_simulated,
    args = (x, data, 200),
    threads = 6
)
```

```python
# burnin
pos0, lnprob, rstate = sampler_sim.run_mcmc(pos0_initial, 100)

# actual sampling
sampler_sim.reset()
sampler_sim.run_mcmc(pos0, N=1000, rstate0=rstate, lnprob0=lnprob);
```

```python
sampler_sim.acceptance_fraction
```

    array([0.649, 0.646, 0.655, 0.675, 0.646, 0.673, 0.624, 0.662, 0.644,
           0.631, 0.682, 0.648, 0.628, 0.651, 0.651, 0.642, 0.656, 0.639,
           0.613, 0.643, 0.637, 0.672, 0.64 , 0.642, 0.668, 0.664, 0.677,
           0.658, 0.635, 0.629, 0.679, 0.639, 0.684, 0.624, 0.643, 0.629,
           0.638, 0.629, 0.644, 0.626, 0.658, 0.629, 0.642, 0.66 , 0.639,
           0.642, 0.648, 0.675, 0.643, 0.673])

```python
corner.corner(sampler_sim.flatchain, labels=['a','b'], truths=[a,b]);
```

```python
np.mean(sampler_sim.flatchain, axis=0), np.std(sampler_sim.flatchain, axis=0)
```

    (array([4.00881108, 3.10332689]), array([1.07493213, 0.3377872 ]))

```python
np.mean(sampler.flatchain, axis=0), np.std(sampler.flatchain, axis=0)
```

    (array([3.99355287, 2.99978635]), array([0.88315128, 0.04897563]))

```python
def lnl_sellentin(params, x, data, n):
    a,b = params

    simulated_data = a + b*x**2 + np.random.normal(scale=5, size=(n,len(x)))

    fbar = np.mean(simulated_data, axis=0)
    s = np.var(simulated_data, axis=0)
    dets = np.prod(s)

    cp = gamma(n/2)/(np.pi*(n-1))/ gamma((n-2)/2)
    return cp/np.sqrt(dets)/(1 + np.sum((data-fbar)**2/s/(n-1)))
```

```python
sampler_sell = emcee.EnsembleSampler(
    nwalkers = 50,
    dim = 2,
    lnpostfn = lnl_sellentin,
    args = (x, data, 200),
    threads = 6
)
```

    /home/steven/miniconda3/envs/21CMMC/lib/python3.6/site-packages/ipykernel/__main__.py:11: RuntimeWarning: divide by zero encountered in double_scalars
    /home/steven/miniconda3/envs/21CMMC/lib/python3.6/site-packages/ipykernel/__main__.py:11: RuntimeWarning: divide by zero encountered in true_divide
    /home/steven/miniconda3/envs/21CMMC/lib/python3.6/site-packages/ipykernel/__main__.py:11: RuntimeWarning: invalid value encountered in double_scalars

```python
lnl_sellentin([a-100.0,b], x, data, 100)
```

    1.9113208290957297e-11

```python
# burnin
pos0, lnprob, rstate = sampler_sell.run_mcmc(pos0_initial, 100)

# actual sampling
sampler_sell.reset()
sampler_sell.run_mcmc(pos0, N=1000, rstate0=rstate, lnprob0=lnprob);
```

```python
sampler_sell.acceptance_fraction
```

    array([0.5       , 0.83333333, 1.        , 0.83333333, 0.5       ,
           1.        , 1.        , 1.        , 1.        , 1.        ,
           0.66666667, 0.83333333, 1.        , 1.        , 1.        ,
           1.        , 0.66666667, 1.        , 0.83333333, 1.        ,
           1.        , 1.        , 1.        , 0.83333333, 0.66666667,
           0.66666667, 0.83333333, 0.83333333, 0.83333333, 0.66666667,
           0.83333333, 0.83333333, 0.5       , 0.66666667, 0.83333333,
           0.83333333, 0.83333333, 0.83333333, 0.83333333, 0.83333333,
           0.83333333, 0.83333333, 0.66666667, 0.83333333, 0.83333333,
           0.83333333, 0.66666667, 0.83333333, 0.66666667, 0.83333333])

```python
corner.corner(sampler_sell.flatchain, labels=['a','b'], truths=[a,b]);
```

```python
def lnl_hartlap(params, x, data, n):
    a,b = params

    simulated_data = a + b*x**2 + np.random.normal(scale=x**2, size=(n,len(x)))

    fbar = np.mean(simulated_data, axis=0)
    s = np.var(simulated_data, axis=0)

    s *= (n-len(params)-2)/(n-1)

    return np.sum(-0.5 * (data - fbar)**2 / s)
```

```python
sampler_hart = emcee.EnsembleSampler(
    nwalkers = 50,
    dim = 2,
    lnpostfn = lnl_hartlap,
    args = (x, data, 10),
    threads = 6
)
```

```python
# burnin
pos0, lnprob, rstate = sampler_hart.run_mcmc(pos0_initial, 100)

# actual sampling
sampler_hart.reset()
sampler_hart.run_mcmc(pos0, N=3000, rstate0=rstate, lnprob0=lnprob);
```

```python
sampler_hart.acceptance_fraction
```

    array([0.12733333, 0.14466667, 0.12333333, 0.12033333, 0.14033333,
           0.13066667, 0.119     , 0.13866667, 0.13333333, 0.12533333,
           0.11533333, 0.10333333, 0.136     , 0.141     , 0.12933333,
           0.12033333, 0.10333333, 0.111     , 0.13533333, 0.11333333,
           0.14033333, 0.12666667, 0.13233333, 0.102     , 0.12133333,
           0.107     , 0.088     , 0.141     , 0.168     , 0.134     ,
           0.12433333, 0.14333333, 0.09266667, 0.146     , 0.12133333,
           0.107     , 0.136     , 0.115     , 0.129     , 0.125     ,
           0.12466667, 0.08066667, 0.12533333, 0.11533333, 0.132     ,
           0.08966667, 0.11033333, 0.14333333, 0.07733333, 0.11166667])

```python
corner.corner(sampler_hart.flatchain, labels=['a','b'], truths=[a,b]);
```

```python
np.mean(sampler_hart.flatchain, axis=0), np.std(sampler_hart.flatchain, axis=0)
```

    (array([3.88458755, 3.12451383]), array([0.99783554, 0.30304328]))

```python

```


# Linear transformations with images
In this module we will work with linearly aligning images.  We'll apply transformations to images, and estimate optimal transformations.  Here problems will be nonlinear, and we'll need to use iterative methods.

```python
import numpy as np
%matplotlib notebook
import matplotlib as mpl
import matplotlib.pyplot as plt
import os # for file paths
import summerschool as ss
import nibabel as nib # for loading many common neuroimage formats
import scipy.interpolate as spi # use interpolation to deform images
```

## Load some data

```python
fname = os.path.join('mouse_images','PMD2052_orig_target_STS_clean.img')
img = nib.load(fname)
# load the image
I = img.get_data()[:,:,:,0] # note last axes is time, we'd like to remove it
# standardize
I = (I - np.mean(I))/np.std(I)
# downsample so it can run on people's laptops
down = 3
I = ss.downsample_image(I,down)

# set up a domain, a grid of points describing the locations of each voxel
nx = I.shape # number of voxels
dx = img.header['pixdim'][1:4]*down # size of each voxel
x0 = np.arange(nx[0])*dx[0]
x1 = np.arange(nx[1])*dx[1]
x2 = np.arange(nx[2])*dx[2]

# lets define the origin to be in the middle of our image
# linear transformations will be more reasonable this way
# i.e. we'll rotate around the middle, instead of rotating around a corner
x0 = x0 - np.mean(x0)
x1 = x1 - np.mean(x1)
x2 = x2 - np.mean(x2)
X0,X1,X2 = np.meshgrid(x0,x1,x2,indexing='ij')
```

```python
# this function will show 3 orthogonal slices through the mouse image
ss.imshow_slices(x0,x1,x2,I)
```

    <IPython.core.display.Javascript object>

    [<matplotlib.image.AxesImage at 0x7fcb99d38748>,
     <matplotlib.image.AxesImage at 0x7fcb99d38a90>,
     <matplotlib.image.AxesImage at 0x7fcb99d38d68>]

## Transformations act on images through their inverse

A simple example is to consider shifting a function in 1D.  If $f(x)$ is a function that we want to translate to the right by $T$, we translate it by $f(x - T)$ (NOT $f(x+T)$).  Here $-T$ is the inverse of $+T$.

In general, if we want to transform a function with a transformation $\varphi$, we use $f(x) \mapsto f(\varphi^{-1}(x))$.

### Explicit example in 2D

We will consider a simple 2x2 image and the effect of rotation.  If you want to rotate your image clockwise, you must rotate your coordinates counterclockwise.  This will be derived and illustrated in the lab.

## Images are deformed by sampling them at new points: interpolation

How do we evaluate $I(\varphi^{-1}(x))$ on a computer?  We need to define the grid that $I$ is sampled on.  Transform the grid using $\varphi^{-1}$.  Then resample $I$ at the new points defined by this transformed grid.

This resampling is known as interpolation.  Most numerical libaries have built in functions for interpolation, including scipy.

```python
# set some arguments that will be used for interpolation with a dictionary
interp_args = {
    'method':'linear', # this says how we'd like to do interpolation, linear is fast and smooth
    'bounds_error':False, # if we try to sample outside the image, do not raise an error
    'fill_value':None # if we try to sample outside the image, we'll return the nearest pixel value
}
```

```python
# A 2D example image and a grid defining its sample points
I2d = np.array(I[:,:,I.shape[-1]//2])
X02d = X0[:,:,0]
X12d = X1[:,:,0]
```

```python
# make a figure
nplots = 5
f,ax = plt.subplots(2, nplots, sharex=True, sharey=True)
for i in range(nplots):
    ax_ = ax[:,i]
    # choose a transformation
    if i == 0:
        A = np.eye(3)
        titlestring = 'identity'
    elif i == 1:
        T = np.random.randn(2)*2.0
        A = np.array([[1,0,T[0]],[0,1,T[1]],[0,0,1]])
        titlestring = 'translation'
    elif i == 2:
        theta = np.random.rand()*2.0*np.pi/6.0
        A = np.eye(3)
        A[:2,:2]= [[np.cos(theta),-np.sin(theta)],[np.sin(theta),np.cos(theta)]]
        titlestring = 'rotation'
    elif i==3:
        s = np.random.randn()*0.1 + 1.0
        A = np.eye(3)
        A[:2,:2] = np.eye(2)*s
        titlestring = 'scale'
    elif i == 4:
        A = np.eye(3)
        A[:2,:2] += np.random.randn(2,2)*0.2
        titlestring = 'linear'

    # make a transformed grid
    # to move the points in the grid we use the forward transformation A
    AX0 = A[0,0]*X02d + A[0,1]*X12d + A[0,2]
    AX1 = A[1,0]*X02d + A[1,1]*X12d + A[1,2]
    stride = 5
    ss.plot_grid(AX1, AX0, ax=ax_[0],
                 rstride=stride, cstride=stride,
                 color='r', linewidth=0.5, alpha=0.5)

    # make a transformed image
    # to transform the image we use the inverse transformatoin A^{-1}
    B = np.linalg.inv(A)
    X0s = B[0,0]*X02d + B[0,1]*X12d + B[0,2]
    X1s = B[1,0]*X02d + B[1,1]*X12d + B[1,2]

    # we use the built in interpolation function.
    # 1. we input our original grid points as a list of 1D vectors
    # 2. then input our original image
    # 3. then input our transformed grid points to resample the image at
    # Note that we need to "stack" this input into a 3D array,
    # where the first slice is the deformed X grid, and the second slice
    # is the deformed Y grid.
    # 4. Last we input our interpolation options dictionary using "**" for "unpacking"
    AI = spi.interpn([x0,x1],I2d,np.stack([X0s,X1s],axis=-1),**interp_args)
    ax_[0].imshow(AI, cmap='gray',
                  extent=[x1[1],x1[-1],x0[0],x0[-1]], origin='lower') # make sure you set extent and origin!

    ax_[0].set_aspect('equal')
    ax_[0].set_title(titlestring)

    # let's show the vector field
    ax_[1].quiver(X12d[::stride,::stride], X02d[::stride,::stride],
                  AX1[::stride,::stride]-X12d[::stride,::stride],
                  AX0[::stride,::stride]-X02d[::stride,::stride])
    ax_[1].set_aspect('equal')
    ax_[1].set_title('vector field')
f.suptitle('Illustration of several matrix transformations in 2D')
```

    <IPython.core.display.Javascript object>

    Text(0.5,0.98,'Illustration of several matrix transformations in 2D')

## Now we'll show an example transformation in 3D

```python
# construct a transformation
A = np.eye(4)
A[:3,:3] += np.random.randn(3,3)*0.05 # a random deformation
A[:3,-1] += np.random.randn(3)*1.0 # a random shift
# generate the grid points that are transformed by inputs
def sample_points_from_affine(X0,X1,X2,A):
    # find its inverse, using homogeneous coordinates
    B = np.linalg.inv(A)
    # get the sample points by matrix multiplication
    X0s = B[0,0]*X0 + B[0,1]*X1 + B[0,2]*X2 + B[0,3]
    X1s = B[1,0]*X0 + B[1,1]*X1 + B[1,2]*X2 + B[1,3]
    X2s = B[2,0]*X0 + B[2,1]*X1 + B[2,2]*X2 + B[2,3]
    return X0s,X1s,X2s
# this is also defined in the summer school module
X0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)
AI = spi.interpn(
    [x0,x1,x2], # this labels where in space each voxel is
    I, # this says the intensity of our image at each voxel
    np.stack([X0s,X1s,X2s],axis=-1), # this says where we'd like to evaluate the image
    **interp_args # some more arguments that we defined above
)

# now draw it
f,ax=plt.subplots(2,3)
plt.subplots_adjust(wspace=0.45,hspace=0.35)
ss.imshow_slices(x0,x1,x2,I,ax[0])
for a in ax[0]:
    a.set_title('before')
    a.set_xlabel('')
ss.imshow_slices(x0,x1,x2,AI,ax[1])
for a in ax[1]: a.set_title('random transform')
Irandom = AI
```

    <IPython.core.display.Javascript object>

## Computing optimal transformations
As with points, we will write down a sum of square error cost
\begin{align*}
E(A) = \int_X \frac{1}{2}(I(A^{-1}x) - J(x))^2 dx
\end{align*}

As in the point set case we'll take it's gradient with respect to $A$.  However, since this problem is nonlinear, we will not be able to explicitly solve for $A$ by setting the result to 0.  Instead, we'll iteratively make small improvements to our estimate of $A$ by taking steps in the negative of the gradient direction.  This approach to optimization is widely used and is called gradient descent.

#### Optimizing over translation
Let's first consider a simpler problem involving translation $T$ only.  Our square error cost will be

\begin{align*}
E(T) = \int_X \frac{1}{2}(I(x - T) - J(x))^2 dx
\end{align*}

We'll compute directional derivatives in the direction of $\delta T$, and use this to derive the gradient vector.  We use the fact that
\begin{align*}
\frac{d}{d\epsilon}E(T + \epsilon \delta T)\bigg|_{\epsilon} &= (\nabla_T E) ^T \delta A
\end{align*}
This is just the definition of the directional derivative.  It is equal to the gradient dot the direction.

We are interested in the left hand side, which is just a derivative of a function with respect to a single scalar variable.  We will use the chain rule
\begin{align*}.
\frac{d}{d\epsilon} E(T + \epsilon \delta T)\bigg|_{\epsilon = 0} &= \frac{d}{d\epsilon} \int_X \frac{1}{2}(I(x - T  \epsilon \delta T) - J(x))^2 dx \bigg|_{\epsilon = 0}\\
&= \int -[I(x - T) - J(x)] D I(x-T) dx \delta T
\end{align*}
Note that $DI$ is the image gradient as a 1x3 row vector, and $\nabla I$ the image gradient as a 3x1 column vector.

We are free to choose $\delta T$ in whatever way we want, and taking it as each element of a basis gives us the gradient
\begin{align*}
\nabla E(T) &= \int -[I(x-T) - J(x)]\nabla I(x-T)dx
\end{align*}

A gradient descent algorithm would then be to choose an initial guess for $T$, choose a small step size $\epsilon$, and repeatedly update by
\begin{align*}
T \mapsto T - \epsilon \nabla_T E(T) = T - \epsilon \int -[I(x-T) - J(x)]\nabla I(x-T) dx
\end{align*}

#### Optimizing over affine
Now we're ready to return to the affine transformation $A$.  This derivation is more challenging, you may want to skip it and just use the final result.

Since $A$ is a matrix, we'll consider an arbitrary perturbation to $A$, $A\mapsto A + \epsilon \delta A$, and take the gradient with respect to $\epsilon$ for any perturbation.
\begin{align*}
E(A + \epsilon \delta A) = \int_X \frac{1}{2}(I( (A + \epsilon \delta A)^{-1}x) - J(x))^2 dx
\end{align*}
We will use the fact that
\begin{align}
\frac{d}{d\epsilon}E(A + \epsilon \delta A) \bigg|_{\epsilon = 0} = \text{trace} \nabla E^T \delta A
\end{align}
This is just the definition of the directional derivative as the gradient dot the direction.

We are interested in the left hand side, which is just a derivative of a function with respect to a single scalar variable.  Let's consider this expression using the chain rule, first consider the square term
\begin{align*}
\frac{d}{d\epsilon}E(A + \epsilon \delta A) \bigg|_{\epsilon = 0} &= \int_X (I((A + \epsilon \delta A)^{-1}x) - J(x)) \frac{d}{d\epsilon} I((A + \epsilon \delta A)^{-1}x)dx \bigg|_{\epsilon = 0}\\
&=\int_X (I(A^{-1}x) - J(x)) \frac{d}{d\epsilon} I((A + \epsilon \delta A)^{-1}x)dx \bigg|_{\epsilon = 0}\\
\end{align*}
Note that $DI$ is the image gradient as a 1x3 row vector, and $\nabla I$ the image gradient as a 3x1 column vector.

Now consider the image
\begin{align*}
&= \int_X (I(A^{-1}x) - J(x))  DI((A + \epsilon \delta A)^{-1}x) \frac{d}{d\epsilon} (A + \epsilon \delta A)^{-1}x dx \bigg|_{\epsilon = 0}\\
&= \int_X (I(A^{-1}x) - J(x))  DI(A^{-1}x) \frac{d}{d\epsilon} (A + \epsilon \delta A)^{-1}\bigg|_{\epsilon = 0} x dx \\
\end{align*}
Now the final term depends on taking the derivative of the inverse of a matrix. We can use the matrix by scalar identity, $\frac{d}{dt}M^{-1}(t) = - M^{-1}(t) \frac{d}{dt}M(t) M^{-1}(t)$ https://en.wikipedia.org/wiki/Matrix_calculus#Matrix-by-scalar_identities .

Plugging this in with $\epsilon = 0$ gives the result
\begin{align*}
\frac{d}{d\epsilon}E(\epsilon) \bigg|_{\epsilon = 0} = -\int_X (I(A^{-1}x) - J(x))  DI(A^{-1}x) A^{-1}\delta A A^{-1} x dx
\end{align*}
We can simplify this by recalling that $D[I(A^{-1}x)] = DI(A^{-1}x)A^{-1}$, giving
\begin{align*}
= -\int_X (I(A^{-1}x) - J(x))  D[I(A^{-1}x)]\delta A A^{-1} x dx
\end{align*}
Finally to work out the gradient with respect to $A$, we have to write this expression as a gradient, dot $\delta A$.  Taking the dot product on matrices using the trace gives
\begin{align*}
&= -\int_X \text{trace}(I(A^{-1}x) - J(x))  D[I(A^{-1}x)]\delta A A^{-1} x dx \\
&= -\int_X \text{trace}A^{-1} x (I(A^{-1}x) - J(x))  D[I(A^{-1}x)] dx \delta A
\end{align*}
which gives a gradient of
\begin{align*}
\nabla_A E &= -\int_X (I(A^{-1}x)-J) \nabla[I(A^{-1}x)](A^{-1}x)^Tdx
\end{align*}

#### Results of affine gradient calculation

The above derivation gives a gradient of
\begin{align*}
\nabla_A E &= -\int_X (I(A^{-1}x)-J) \nabla[I(A^{-1}x)](A^{-1}x)^Tdx
\end{align*}

A gradient descent algorithm would then be to choose an initial guess for $T$, choose a small step size $\epsilon$, and repeatedly update by
\begin{align*}
A \mapsto A - \epsilon \nabla_A E(A) = T - \epsilon \int_X -(I(A^{-1}x)-J) \nabla[I(A^{-1}x)](A^{-1}x)^T dx
\end{align*}

We implement this below.

```python
# load a second image to map to
fname = os.path.join('mouse_images','PMD3097_orig_target_STS_clean.img')
img = nib.load(fname)
J = img.get_data()[:,:,:,0] # note last axes is time, we'd like to remove it
J = (J - np.mean(J))/np.std(J)
J = ss.downsample_image(J,down)
f,ax = plt.subplots(2,3)
ss.imshow_slices(x0,x1,x2,J,ax[0])
'''
# note that the human MRI images in this repository have already been affine aligned into MNI space
# let's add a small affine transformation and then try to recover it
A = np.eye(4)
A[:3,:3] += np.random.randn(3,3)*0.05 # a random deformation
A[:3,-1] += np.random.randn(3)*2.0 # a random shift
X0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)
J = spi.interpn([x0,x1,x2],J,np.stack([X0s,X1s,X2s],axis=-1),**interp_args)
ss.imshow_slices(x0,x1,x2,J,ax[1])
'''
' '
```

    <IPython.core.display.Javascript object>

    ' '

```python
# now optimize over affine transformations

# initialize a transformation
A = np.eye(4)
# choose a gradient descent step size
# note that the linear part (L) and the translation part (T) often need to be on very different scales
# for human
epsilonL = 1.0e-12
epsilonT = 1.0e-9
# for mouse, standardized data
epsilonL = 3.0e-5
epsilonT = 3.0e-4
# number of iterations of optimization
niter = 25

# initialize some variables we'll display at every iteration
EAll = [] # the energy, for plotting
f,ax = plt.subplots(3,3) # a figure for plotting
plt.subplots_adjust(wspace=0.45,hspace=0.45) # make the figure have a nicer layout
for it in range(niter):
    # find the deformed image, by deforming the grid with the inverse, and using interpolation
    X0s,X1s,X2s = ss.sample_points_from_affine(X0,X1,X2,A)
    AI = spi.interpn([x0,x1,x2],I,np.stack([X0s,X1s,X2s],axis=-1),**interp_args)

    # visualize it, we show the deforming atlas on the first row
    ss.imshow_slices(x0,x1,x2,AI,ax[0])

    # get the error
    err = AI-J

    # we show the error on the second row
    ss.imshow_slices(x0,x1,x2,err,ax[1])
    for a in ax[1]: a.set_xlabel('')

    # find the image gradient.  This is the discrete centered difference, with given voxel size
    AI_0,AI_1,AI_2 = np.gradient(AI,dx[0],dx[1],dx[2])

    # calculate energy and plot it
    E = np.sum(err**2*0.5)*np.prod(dx)
    EAll.append(E)
    ax[2,0].cla()
    ax[2,0].plot(EAll)
    ax[2,0].set_title('cost')
    ax[2,0].set_xlabel('iteration')

    # find the Energy gradient of the linear part using our derivation above
    gradL = np.empty((3,3))
    for i,AI_i in enumerate([AI_0,AI_1,AI_2]):
        for j,AX_j in enumerate([X0s,X1s,X2s]):
            gradL[i,j] = -np.sum(err*AI_i*AX_j)*np.prod(dx)

    # find the Energy gradient of the translation part using our derivation above
    gradT = np.empty(3)
    for i,AI_i in enumerate([AI_0,AI_1,AI_2]):
        gradT[i] = -np.sum(err*AI_i)*np.prod(dx)

    # update using gradient descent
    A[:3,:3] -= epsilonL*gradL
    A[:3,-1] -= epsilonT*gradT

    # draw it in real time
    f.canvas.draw()

# let's write out the transformation to a text file for the next tutorial
with open('affine.txt','wt') as f:
    for i in range(4):
        for j in range(4):
            f.write('{} '.format(A[i,j]))
        f.write('\n')

```

    <IPython.core.display.Javascript object>


```python
import matplotlib.pyplot as plt
import scipy.stats as st
import seaborn as sns
import pandas as pd
from scipy.stats import norm, uniform, expon
from scipy.integrate import quad
from sympy.solvers import solve
from sympy import Symbol
import numpy as np
import scipy.stats as st
from pandas import Series, DataFrame
```

```python
import matplotlib.pyplot as plt
import numpy as np
from pandas import Series
from scipy.stats import norm, probplot
x = Series([24.4, 27.6, 27.8, 27.9, 28.5,
30.1, 30.3, 31.7, 32.2, 32.8,
33.3, 33.5, 34.1, 34.6, 35.8,
35.9, 36.8, 37.1, 39.2, 39.7])
alphak = (np.arange(1, x.size + 1) - 0.5) / x.size
quantile_theor = norm.ppf(q=alphak, loc=x.mean(), scale=x.std())
quantile_empir = np.sort(x)
plt.xlabel("Theor. Quantile")
plt.ylabel("Empirische Quantile")
plt.plot(quantile_theor, quantile_empir, "o")
plt.show()
```

```python
x = Series([24.4, 27.6, 27.8, 27.9, 28.5,
30.1, 30.3, 31.7, 32.2, 32.8,
33.3, 33.5, 34.1, 34.6, 35.8,
35.9, 36.8, 37.1, 39.2, 39.7])
probplot(x, plot=plt)
```

```python
Series([11.96, 5.03, 67.40, 16.07,31.50,7.73, 11.10,22.38]).mean()

```

    21.64625

```python
for n in [10,20,50,100]:
    random_numbers = norm.rvs(size=n)
    x = st.norm.rvs(size=n)
    st.probplot(x,plot=plt)
    plt.show()

```

```python
for df in [20,7,3]:
    for n in [20,100]:
        random_numbers = norm.rvs(size=n)
        x = st.t.rvs(size=n, df=df)
        st.probplot(x,plot=plt)
        plt.title(f"freiheitsgrade: {df} - numbers {n}")
        plt.show()
```

```python
for df in [20,1]:
    for n in [20,100]:
        random_numbers = norm.rvs(size=n)
        x = st.t.rvs(size=n, df=df)
        st.probplot(x,plot=plt)
        plt.title(f"freiheitsgrade: {df} - numbers {n}")
        plt.show()
```

```python
werte =[0,10,11]
simulation = Series(np.random.choice(werte, replace=True, size=1000))
```

```python
plt.subplot(4,2,1)
simulation.hist(bins=[0,1,10,11,12], edgecolor="black")
plt.title("Original")
plt.subplot(4,2,2)
# Normalplot erstellen
st.probplot(simulation,plot=plt)
plt.title("Normal Q-Q Plot")
```

```python
n = 5
# X_1,...,X_n simulieren und in einer n-spaltigen
# Matrix (mit 1000 Zeilen) anordnen
sim = np.random.choice(werte, size=n*1000, replace=True)
sim = DataFrame(np.reshape(sim,(n,1000)))
#In jeder Matrixzeile Mittelwert berechnen
sim_mean = sim.mean()
plt.subplot(4,2,3)
sim_mean.hist(edgecolor="black")
plt.title("Mittelwerte von 5 Beobachtungen")
plt.subplot(4,2,4)
st.probplot(sim_mean,plot=plt)
plt.title("Normal Q-Q Plot")

```

```python
    n = 10
    # X_1,...,X_n simulieren und in einer n-spaltigen
    # Matrix (mit 1000 Zeilen) anordnen
    sim = np.random.choice(werte, size=n*1000, replace=True)
    sim = DataFrame(np.reshape(sim,(n,1000)))
    #In jeder Matrixzeile Mittelwert berechnen
    sim_mean = sim.mean()
    plt.subplot(4,2,5)
    sim_mean.hist(edgecolor="black")
    plt.title("Mittelwerte von 5 Beobachtungen")
```

```python
    n = 200
    # X_1,...,X_n simulieren und in einer n-spaltigen
    # Matrix (mit 1000 Zeilen) anordnen
    sim = np.random.choice(werte, size=n*1000, replace=True)
    sim = DataFrame(np.reshape(sim,(n,1000)))
    #In jeder Matrixzeile Mittelwert berechnen
    sim_mean = sim.mean()
    plt.subplot(4,2,6)
    sim_mean.hist(edgecolor="black")
    plt.title("Mittelwerte von 5 Beobachtungen")
```

```python
    n = 200
    # X_1,...,X_n simulieren und in einer n-spaltigen
    # Matrix (mit 1000 Zeilen) anordnen
    sim = np.random.choice(werte, size=n*1000, replace=True)

    sim = DataFrame(np.reshape(sim,(n,1000)))
    #In jeder Matrixzeile Mittelwert berechnen
    sim_mean = sim.mean()
    print(sim_mean.mean())
    print(sim_mean.std())
    print(sim_mean.var())
    sim_mean.hist(edgecolor="black")
    plt.title("Mittelwerte von 5 Beobachtungen")
```

```python
np.arange(1,13)
```

    array([ 1,  2,  3,  4,  5,  6,  7,  8,  9, 10, 11, 12])

```python
norm.interval(alpha=.95, loc=32, scale=6)
```

    (20.240216092759674, 43.759783907240326)

```python
1-norm.cdf(x=0.9, loc=0, scale=0.45)
```

    0.02275013194817921

```python
1-norm.cdf(x=0.9/0.45)
```

    0.02275013194817921

```python
norm.ppf(q=0.005)
```

    -2.575829303548901

```python
norm.cdf(-2.575829303548901)
```

    0.004999999999999998

```python
2.58*0.45
```

    1.161

```python

```


## Machine Learning Introduction

### Making a Variety of Predictive Machine Learning Models in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### Multiple Machine Learning Models for Subsurface Modeling in Python

Here's a simple workflow, demonstration of training multiple machine learning models with a single dataset.

* This is provided as a fast comparison of general behavoir of these models

* I demonstration model training and tuning (automatically and by-hand)

* Given the 2 predictor features and 1 response feature it is easy to visualize the data and the model

This provides an opportunity to run and visualize a variety of machine learning models for experiential learning.

#### Machine Learning-based Prediction

Predictive machine learning methods with supervised learning:

**Prediction**

* non-parametric method for regression and classification

* a function $\hat{f}$ of the nearest $k$ training data in predictor feature space such that we predict a response feature $Y$ from a set of predictor features $X_1,\ldots,X_m$.

* the prediction is of the form $\hat{Y} = \hat{f}(X_1,\ldots,X_m)$

**Suppervised Learning**

* the response feature label, $Y$, is available over the training and testing data

#### The Machine Learning Models

This workflow includes the following models:

* linear regression
* k-nearest neighbor regression
* decision tree regression
* random forest regression
* tree-based gradient boosting regression

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

If you want to work locally you will need to first copy the data file to your working directory. It is available here:

* Tabular data - unconv_MV.csv found [here](https://github.com/GeostatsGuy/GeoDataSets/blob/master/unconv_MV.csv).

You don't have to download the data if you want to load directly from my github account with the code below.

#### Import Required Packages

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import os                                                   # to set current working directory
import math                                                 # basic calculations like square root
import pandas as pd                                         # DataFrames and plotting
import numpy as np                                          # arrays and matrix math
import matplotlib.pyplot as plt                             # plotting
import matplotlib.ticker as ticker                          # modify the plot axis labels
from sklearn.model_selection import train_test_split        # train and test split
from sklearn.pipeline import Pipeline                       # machine learning modeling pipeline
from sklearn.model_selection import GridSearchCV            # model hyperparameter grid search
from sklearn.model_selection import KFold                   # k-fold cross validation
from sklearn.preprocessing import StandardScaler            # feature standardization
np.random.seed(seed=73038)
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Declare functions

Let's define a couple of convenience functions for model visualization and hyperparameter tuning.

```python
def visualize_model(model,xtrain,xtest,x_min,x_max,ytrain,ytest,y_min,y_max,rtrain,rtest,z_min,z_max,title,colorbar=True):# plots the data points and the decision tree prediction
    n_classes = 10
    #cmap = plt.cm.RdYlBu
    cmap = plt.cm.plasma
    plot_step = max((x_max-x_min),(y_max-y_min))/200
    xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                     np.arange(y_min, y_max, plot_step))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = np.clip(Z,z_min,z_max)
    Z = Z.reshape(xx.shape)
    #cs = plt.contourf(xx, yy, Z, cmap=cmap,vmin=z_min, vmax=z_max, levels=np.linspace(z_min, z_max, 100))
    cs = plt.imshow(Z,interpolation = None,extent = [x_min,x_max,y_min,y_max], vmin = z_min, vmax = z_max,cmap = cmap)

    im = plt.scatter(xtrain,ytrain,s=None, c=rtrain, marker='o', cmap=cmap, norm=None, vmin=z_min, vmax=z_max, alpha=0.8, linewidths=0.5, verts=None, edgecolors="white",label='train')
    im2 = plt.scatter(xtest,ytest,s=None, c=rtest, marker='s', cmap=cmap, norm=None, vmin=z_min, vmax=z_max, alpha=0.8, linewidths=0.5, verts=None, edgecolors="black",label='test')
    plt.xlim([x_min,x_max]); plt.ylim([y_min,y_max])
    plt.title(title)
    plt.xlabel(xtrain.name)
    plt.ylabel(ytrain.name)
    plt.legend(loc = 'upper right')
    if colorbar == True:
      cbar = plt.colorbar(im, orientation = 'vertical')
      cbar.set_label(rtrain.name, rotation=270, labelpad=20)

def visualize_model_sub(ax,model,xtrain,xtest,x_min,x_max,ytrain,ytest,y_min,y_max,rtrain,rtest,z_min,z_max,title,legend=True):# plots the data points and the decision tree prediction
    n_classes = 10
    cmap = plt.cm.RdYlBu
    cmap = plt.cm.plasma
    plot_step = 0.02
    xx, yy = np.meshgrid(np.arange(x_min, x_max, plot_step),
                     np.arange(y_min, y_max, plot_step))
    Z = model.predict(np.c_[xx.ravel(), yy.ravel()])
    Z = np.clip(Z,z_min,z_max)
    Z = Z.reshape(xx.shape)
    cs = ax.contourf(xx, yy, Z, cmap=cmap,vmin=z_min, vmax=z_max, levels=np.linspace(z_min, z_max, 100))
    im = ax.scatter(xtrain,ytrain,s=None, c=rtrain, marker='o', cmap=cmap, norm=None, vmin=z_min, vmax=z_max, alpha=0.8, linewidths=0.5, verts=None, edgecolors="white",label='train')
    im2 = ax.scatter(xtest,ytest,s=None, c=rtest, marker='s', cmap=cmap, norm=None, vmin=z_min, vmax=z_max, alpha=0.8, linewidths=0.5, verts=None, edgecolors="black",label='test')
    ax.set_title(title)
    ax.set_xlabel(xtrain.name)
    ax.set_ylabel(ytrain.name)
    ax.set_xlim(x_min,x_max)
    ax.set_ylim(y_min,y_max)
    if legend == True:
      cbar = plt.colorbar(im, orientation = 'vertical')
      cbar.set_label(rtrain.name, rotation=270, labelpad=20)

def tune_model(model,X,y,par_name,par_min,par_max,npar,spar,log_scale,int_list,title,xlabel):
    scorelist = []
    if log_scale:
        parlist = np.logspace(math.log10(par_min),math.log10(par_max),npar)
    else:
        if int_list == True:
            step = int((par_max-par_min)/npar)
            parlist = np.arange(par_min,par_max+1,step,dtype = int)
        else:
            parlist = np.linspace(par_min,par_max,npar)

    pipe = Pipeline([                                           # the machine learning workflow as a pipeline object
        ('model', model)
    ])

    params = {                                                  # the machine learning workflow method's parameters
        'model__' + str(par_name): parlist,
    }
    folds = 4
    grid_cv_tuned = GridSearchCV(pipe, params, scoring = 'neg_mean_squared_error', # grid search cross validation
                             cv=KFold(n_splits=folds,shuffle=False),
                             refit = True)

    grid_cv_tuned.fit(X,y)                                      # fit model with tuned hyperparameters

    scorelist = abs(grid_cv_tuned.cv_results_['mean_test_score'])

    if log_scale:
        plt.arrow(parlist[np.argmin(scorelist)],0,0,np.max(scorelist)*1.2,width=parlist[np.argmin(scorelist)]/10.0,color='red',head_length=0.0,head_width=0.0)
        plt.arrow(spar,0,0,np.max(scorelist)*1.2,width=spar/10.0,color='black',head_length=0.0,head_width=0.0)
    else:
        plt.arrow(parlist[np.argmin(scorelist)],0,0,np.max(scorelist)*1.2,width=1,color='red',head_length=0.0,head_width=0.0)
        plt.arrow(spar,0,0,np.max(scorelist)*1.2,width=0.5,color='black',head_length=0.0,head_width=0.0)

    plt.annotate('Your Model',[spar,np.max(scorelist)*0.95],rotation=270.0)
    if int_list == True:
        plt.annotate('Tuned Model, ' + xlabel + ' = ' + str(format(parlist[np.argmin(scorelist)],"d")),[parlist[np.argmin(scorelist)],np.max(scorelist)*0.4],rotation=270.0)
    else:
        plt.annotate('Tuned Model, ' + xlabel + ' = ' + str(format(parlist[np.argmin(scorelist)],"4.3f")),[parlist[np.argmin(scorelist)],np.max(scorelist)*0.4],rotation=270.0)

    plt.scatter(parlist,scorelist,s=5, c="red", marker=None, cmap=None, norm=None, vmin=None, vmax=None, alpha=0.8, linewidths=0.5, verts=None, edgecolors="black")
    y_labels = plt.gca().get_yticks()
    plt.gca().yaxis.set_major_formatter(ticker.FormatStrFormatter('%0.0e'))
    plt.title(title); plt.xlabel(xlabel); plt.ylabel('Mean Square Error')
    if log_scale:
        plt.xscale('log');
    plt.xlim(par_min,par_max); plt.ylim(1.0,np.max(scorelist)*1.2)

def check_model(model,xtrain,xtest,ytrain,ytest,rtrain,rtest,rmin,rmax,title):    # plots the estimated vs. the actual
    predict_train = model.predict(np.c_[xtrain,ytrain])
    predict_test = model.predict(np.c_[xtest,ytest])
    plt.scatter(rtrain,predict_train,s=None, c='red',marker='o', cmap=None, norm=None, vmin=None, vmax=None, alpha=0.2, linewidths=0.3, verts=None, edgecolors="black", label = 'train')
    plt.scatter(rtest,predict_test,s=None, c='blue',marker='s', cmap=None, norm=None, vmin=None, vmax=None, alpha=0.2, linewidths=0.3, verts=None, edgecolors="black", label = 'test')
    plt.title(title); plt.xlabel('Actual Production (MCFPD)'); plt.ylabel('Estimated Production (MCFPD)')
    plt.xlim(rmin,rmax); plt.ylim(rmin,rmax)
    plt.legend(loc = 'upper right')
    plt.arrow(rmin,rmin,rmax,rmax,width=0.02,color='black',head_length=0.0,head_width=0.0)
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

* I include this for convience if you want to write out results or load the data file locally

```python
#os.chdir("c:/PGE383")                                       # set the working directory
```

You will have to update the part in quotes with your own working directory and the format is different on a Mac (e.g. "~/PGE").

#### Read the data table

We use a data "unconv_MV.csv" comma delimited file from my GitHub [respository](https://github.com/GeostatsGuy/GeoDataSets).

* this code loads the data directly from my GitHub repository.
* if you have the data file available locally (i.e. you are not connected to the internet) set the working directory to the location with the data file and use the second line below

```python
my_data = pd.read_csv("https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/unconv_MV.csv")   # load the comma delimited data file
#my_data = pd.read_csv("unconv_MV.csv")                     # load the comma delimited data file locally
my_data = my_data.sample(frac = 0.2)                        # reduce the number of data for testing and visualization
num = my_data._get_numeric_data()                           # get the numerical values
production_error = 1500                                     # adjust the error in the response feature
my_data['Production'] = my_data['Production'] + np.random.normal(loc=0.0,scale=production_error,size=len(my_data))
num[num < 0] = 0                                            # truncate negative values to 0.0
```

Let's visualize the first several rows of our data stored in a DataFrame so we can make sure we successfully loaded the data file.

```python
my_data.head(n=3)                                           # preview the first 3 rows of the dataframe
```

      WellIndex
      Por
      LogPerm
      AI
      Brittle
      TOC
      VR
      Production

      262
      263
      18.59
      1.98
      1.91
      56.49
      0.92
      1.57
      4196.571056

      785
      786
      23.31
      2.27
      2.72
      17.93
      2.04
      2.19
      1117.707449

      855
      856
      13.67
      1.14
      2.89
      28.69
      0.62
      1.73
      1306.847675

Let's remove the well index and check the summary summary statistics.

```python
my_data = my_data.iloc[:,1:8]                               # copy all rows and columns 1 through 8, note 0 column is removed
my_data.describe().transpose()                              # calculate summary statistics for the data
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      Por
      200.0
      14.99170
      3.111665
      5.79
      12.955000
      15.035000
      17.092500
      24.650000

      LogPerm
      200.0
      1.41340
      0.404338
      0.24
      1.177500
      1.415000
      1.660000
      2.580000

      AI
      200.0
      3.01940
      0.552788
      1.33
      2.685000
      3.025000
      3.352500
      4.520000

      Brittle
      200.0
      49.88965
      14.801094
      17.93
      39.647500
      49.885000
      59.240000
      89.450000

      TOC
      200.0
      1.02490
      0.499605
      0.00
      0.697500
      1.010000
      1.372500
      2.230000

      VR
      200.0
      2.02435
      0.298912
      0.90
      1.837500
      2.030000
      2.190000
      2.900000

      Production
      200.0
      2424.08672
      1916.796002
      0.00
      935.685819
      2194.198856
      3812.526316
      11353.810415

#### Standardizing Predictor Features

The k-nearest neighbour etc. methods uses a nearest training sample search in feature space (like k-means clustering). To remove the impact feature range from the approach we standardize the features.

* we will standardize our predictor features to have a mean of zero and a variance of one.

* we use the scikikt learn preprocessing to simplify this step and provid a reverse transform.

We also extract the selected, transformed features as convenient DataFrames for the next workflow steps.

```python
transform = StandardScaler();
features = ['Por','Brittle']

my_data['Standardized Porosity'] = transform.fit_transform(my_data.loc[:,features].values)[:,0] # standardize the data features to mean = 0, var = 1.0
my_data['Standardized Brittleness'] = transform.fit_transform(my_data.loc[:,features].values)[:,1] # standardize the data features to mean = 0, var = 1.0

X1 = my_data['Standardized Porosity']
X2 = my_data['Standardized Brittleness']
X = my_data[['Standardized Porosity','Standardized Brittleness']]
y = my_data['Production']
my_data.head(n=3)
```

      Por
      LogPerm
      AI
      Brittle
      TOC
      VR
      Production
      Standardized Porosity
      Standardized Brittleness

      262
      18.59
      1.98
      1.91
      56.49
      0.92
      1.57
      4196.571056
      1.159293
      0.447056

      785
      23.31
      2.27
      2.72
      17.93
      2.04
      2.19
      1117.707449
      2.679972
      -2.164695

      855
      13.67
      1.14
      2.89
      28.69
      0.62
      1.73
      1306.847675
      -0.425822
      -1.435897

We can compare the output above with the original porosity and brittleness. The reverse transform works! We will use it to return to the original feature units.

#### Feature Ranges

Let's set some ranges for plotting. Note for the standardized predictor features we will use -3.5 to 3.5 as the limits.

```python
pormin = 5.0; pormax = 25.0                                 # set minumums and maximums for visualization
brittlemin = 0.0; brittlemax = 100.0
prodmin = 0.0; prodmax = 9000.0
```

This dataset has variables from 1,000 unconventional wells including well average porosity, log transform of permeability (to linearize the relationships with other variables), accoustic impedance (kg/m2s*10^6), brittness ratio (%), total organic carbon (%), vitrinite reflectance (%), and initial production 90 day average (MCFPD).  Note, the dataset is synthetic.

#### Working with Only Two Predictor Features

Let's simplify the problem to 2 predictor features, Porosity and Brittleness to predict Production rate.  We will also reduce the number of wells from 1,000 to 500.  By working with only 2 features, it is very easy to visualize the segmentation of the feature space (it is only 2D and can be shown compleltely on a single plot).

```python
my_data_subset = my_data.iloc[:,[7,8,6]]                    # extract just por, brittle and prod with 100 samples
X_train, X_test, y_train, y_test = train_test_split(X,y, test_size=0.25, random_state=73073)
X1_train = X_train.iloc[:,0]; X2_train = X_train.iloc[:,1]; X1_test = X_test.iloc[:,0]; X2_test = X_test.iloc[:,1];
y_train = pd.DataFrame({'Production':y_train.values})       # force the response feature to be a DataFrame instead of a 1D ndarray
y_test = pd.DataFrame({'Production':y_test.values})
```

Let's first check the univariate statistics of Porosity, Brittleness and Producton.

```python
plt.subplot(231)
plt.hist(X1_train, alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Porosity Training Data (%)'); plt.xlim(-3,3)

plt.subplot(232)
plt.hist(X_train.iloc[:,1], alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Britteness Training Data (%)'); plt.xlim(-3,3)

plt.subplot(233)
plt.hist(y_train.iloc[:,0], alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Production Training Data (MCFPD)'); plt.xlim(prodmin,prodmax)

plt.subplot(234)
plt.hist(X_test.iloc[:,0], alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Porosity Testing Data (%)'); plt.xlim(-3,3)

plt.subplot(235)
plt.hist(X_test.iloc[:,1], alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Britteness Testing Data (%)'); plt.xlim(-3,3)

plt.subplot(236)
plt.hist(y_test.iloc[:,0], alpha = 0.2, color = 'red', edgecolor = 'black', bins=20)
plt.title('Production Testing Data (MCFPD)'); plt.xlim(prodmin,prodmax)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.2, wspace=0.2, hspace=0.2)
plt.show()
```

The distributions are well behaved, we cannot observe obvious gaps nor truncations.  Let's look at a scatter plot of Porosity vs. Brittleness with points colored by Production.

```python
plt.subplot(121)
im = plt.scatter(X_train.iloc[:,0],X_train.iloc[:,1],s=None, c=y_train.iloc[:,0], marker=None, cmap=plt.cm.plasma, norm=None, vmin=prodmin, vmax=prodmax, alpha=0.8, linewidths=0.3, verts=None, edgecolors="black")
plt.title('Training Production vs. Brittleness and Porosity'); plt.xlabel('Porosity (%)'); plt.ylabel('Brittleness (%)')
plt.xlim(-3,3); plt.ylim(-3,3)
cbar = plt.colorbar(im, orientation = 'vertical')
cbar.set_label("Production", rotation=270, labelpad=20)

plt.subplot(122)
im = plt.scatter(X_test.iloc[:,0],X_test.iloc[:,1],s=None, c=y_test.iloc[:,0], marker=None, cmap=plt.cm.plasma, norm=None, vmin=prodmin, vmax=prodmax, alpha=0.8, linewidths=0.3, verts=None, edgecolors="black")
plt.title('Testing Production vs. Brittleness and Porosity'); plt.xlabel('Porosity (%)'); plt.ylabel('Brittleness (%)')
plt.xlim(-3,3); plt.ylim(-3,3)
cbar = plt.colorbar(im, orientation = 'vertical')
cbar.set_label("Production", rotation=270, labelpad=20)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

This problem looks complicated and could not be modeled with simple linear regression.  It appears there is a sweet spot for Brittleness and increasing Porosity is always beneficial for Production.

#### Play with a Variety of Machine Learning Models

Let's try out several machine learning prediction models to predict production with the porosity and brittleness data.

Here's our machine learning prediction model:

\begin{equation}
production = f(porosity,brittleness) + \epsilon
\end{equation}

We will show use the following models:

* multilinear regression [Lecture](https://www.youtube.com/watch?v=0fzbyhWiP84&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=21)
* ridge regression [Lecture](https://www.youtube.com/watch?v=pMGO40yXZ5Y&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=22)
* k-nearest nearbours [Lecture](https://www.youtube.com/watch?v=lzmeChSYvv8&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=30)
* decision tree [Lecture](https://www.youtube.com/watch?v=JUGo1Pu3QT4&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=38)
* random forest [Lecture](https://www.youtube.com/watch?v=m5_wk310fho&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=39)
* gradient boosting [Lecture](https://www.youtube.com/watch?v=___T8_ixIwc&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=40)

check out the links for lectures on my YouTube channel for each.

* I have a lot of lectures available on machine learning in the Subsurface Machine Learning [Playlist](https://www.youtube.com/watch?v=5kBS5ThMHcU&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf)

___

#### Instantiate, Fit and Predict with Multilinear Regression

Let's instantiate, fit and predict with a linear regression model.

Trying Out Hyperparameters:

* there are no hyperparameters to try out with linear regression, standard linear regression does not have an option to tune model complexity / flexibility
* changing the norm from L2 (not commonly done, but demonstrated [here](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/Interactive_Norms.ipynb)) changes sensitivity to the training data

```python
from sklearn.linear_model import LinearRegression

linear_model = LinearRegression()                            # instantiate the prediction model
linear_model.fit(X_train,y_train)                    # train the model with the training data

plt.subplot(121)
visualize_model(linear_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Multilinear Regression Model')

plt.subplot(122)
check_model(linear_model,X_train.iloc[:,0],X_test.iloc[:,0],X_train.iloc[:,1],X_test.iloc[:,1],y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Multilinear Regression Cross Validation Plot')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.3, hspace=0.2)
plt.show()

from sklearn import metrics                                 # measures to check our models
predict_test = linear_model.predict(np.c_[X_test.iloc[:,0],X_test.iloc[:,1]])
print('Model parameters are = ')
print('  Slope X1 = ' + str(round(linear_model.coef_[0][0],2)) + ' and slope X2 = ' + str(round(linear_model.coef_[0][1],2)))
print('  Intercept = ' + str(round(linear_model.intercept_[0],2)))
print('Testing mean square error = ' + str(round(metrics.mean_squared_error(predict_test,y_test.iloc[:,0]),2)))
```

#### Instantiate, Fit and Predict with Ridge Regression

Let's instantiate, fit and predict with a linear regression model.

Trying Out Hyperparameters:

* $\lambda$ - the shrinkage factor applied to the shrinkage penalty, penalizes large absolute slope terms

* linear regression alternative with hyperparamter tuning through regularization and feature selection:

    * LASSO [Lecture](https://www.youtube.com/watch?v=cVFYhlCCI_8&list=PLG19vXLQHvSC2ZKFIkgVpI9fCjkN38kwf&index=23)

```python
from sklearn.linear_model import Ridge

lam = 0.0001                                                  # hyperparameter
lam_min = 1.0e-10; lam_max = 1.0e10; nlam = 20

ridge_model = Ridge(alpha = lam)                             # instantiate the prediction model
ridge_model.fit(X_train,y_train)                             # train the model with the training data

plt.subplot(131)
visualize_model(ridge_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Ridge Regression, Lambda = ' + str(lam))

plt.subplot(132)
check_model(ridge_model,X_train.iloc[:,0],X_test.iloc[:,0],X_train.iloc[:,1],X_test.iloc[:,1],y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Ridge Regression Cross Validation Plot, Lambda = ' + str(lam))

plt.subplot(133)
tune_model(ridge_model,X,y,'alpha',lam_min,lam_max,nlam,lam,True,False,'Ridge Regression Hyperparameter Tuning','Lambda')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.5)
plt.show()
```

#### Instantiate, Fit and Predict with $k$ Nearest Neighbour

Let's instantiate, fit and predict with a $k$ nearest neighbour model.

* instantiate it with the hyperparameters, $k$ nearest neighbours

* train with the training data, we use the standard fit function from scikit learn

Trying Out Hyperparameters:

* **k** - number of nearest training data to use for the estimate
* **low k** - high model complexity / flexibility
* **high k** - low model complexity / flexibility

Just change this line of code below:

```python
k = 5                                                       # hyperparameter
```

```python
from sklearn.neighbors import KNeighborsRegressor

k = 10                                                      # hyperparameter
k_min = 1; k_max=150; nk = 80

neigh_model = KNeighborsRegressor(weights = 'uniform', n_neighbors=k, p = 1) # instantiate the prediction model
neigh_model.fit(X_train,y_train)                            # train the model with the training data

plt.subplot(131)
visualize_model(neigh_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'k-Nearest Neighbours, k = ' + str(k))

plt.subplot(132)
check_model(neigh_model,X_train.iloc[:,0],X_test.iloc[:,0],X_train.iloc[:,1],X_test.iloc[:,1],y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'k-Nearest Neighbours Cross Validation Plot, k = ' + str(k))

plt.subplot(133)
tune_model(neigh_model,X,y,'n_neighbors',k_min,k_max,nk,k,False,True,'k-nearest Neighbours Hyperparameter Tuning','k')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.5)
plt.show()
```

#### Instantiate, Fit and Predict with Decision Tree Regression

Let's instantiate, fit and predict with a decision tree regression model.

Trying Out Hyperparameters:

* **max_leaf_nodes** - maximum number of regions, terminal nodes, in the decision tree, low **max_lead_nodes** - low model complexity / flexibility
* **min_samples_leaf** - minimum number of training data in a regions, terminal nodes, in the decision tree, low **min_samples_leaf** - high model complexity / flexibility
* **max_depth** - maximum number of decision tree layers, low **max_depth** - low modelcomplexity / flexibility

Just change these lines of code below:

```python
max_leaf_nodes = 20                                       # hyperparameters
min_samples_leaf = 2
max_depth = 10
```

```python
from sklearn import tree                                   # tree program from scikit learn (package for machine learning)

nodes = 8                                                 # hyperparameters
nodes_min = 2; nodes_max=100; nnodes = 80

tree_model = tree.DecisionTreeRegressor(max_leaf_nodes = nodes) # instantiate the prediction model
tree_model.fit(X_train,y_train)                            # train the model with the training data

plt.subplot(131)
visualize_model(tree_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Decision Tree, Leaf Nodes = ' + str(nodes))

plt.subplot(132)
check_model(tree_model,X_train.iloc[:,0],X_test.iloc[:,0],X_train.iloc[:,1],X_test.iloc[:,1],y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Decision Tree Cross Validation Plot, Leaf Nodes = ' + str(nodes))

plt.subplot(133)
tune_model(tree_model,X,y,'max_leaf_nodes',nodes_min,nodes_max,nnodes,nodes,False,True,'Decision Tree Hyperparameter Tuning','Leaf Nodes')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.5)
plt.show()
```

#### Instantiate, Fit and Predict with Tree-based Random Forest Regression

Let's instantiate, fit and predict with a random forest regression model.

Trying Out Hyperparameters:

* **max_leaf_nodes** - maximum number of regions, terminal nodes, in the individual decision trees

Other Model Hyperparameters:

* **max_depth** - maximum number of decision tree layers, low **max_depth** - low model complexity / flexibility
* **n_estimators** - the number of decision trees to average to make the ensemble estimate
* **max_features** - number of features to randomly select for each decision, if 2 you will get tree bagging regression

Just change these lines of code below:

```python
fnodes = 80                                               # hyperparameters
max_depth = 10
n_estimators = 1000
max_features = 2
```

```python
from sklearn.ensemble import RandomForestRegressor

fnodes = 20                                              # hyperparameters
fnodes_min = 2; fnodes_max=100; fnnodes = 40

max_depth = 10
n_estimators = 10
max_features = 2

forest_model = RandomForestRegressor(max_leaf_nodes=fnodes,random_state=73073,n_estimators=n_estimators, max_features=max_features)
forest_model.fit(X_train,y_train.iloc[:,0])                         # train the model with training data

plt.subplot(131)
visualize_model(forest_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Random Forest, Leaf Nodes = ' + str(fnodes))

plt.subplot(132)
check_model(forest_model,X_train.iloc[:,0],X_test.iloc[:,0],X_train.iloc[:,1],X_test.iloc[:,1],y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Random Forest Cross Validation Plot, Leaf Nodes = ' + str(fnodes))

plt.subplot(133)
tune_model(forest_model,X,y,'max_leaf_nodes',fnodes_min,fnodes_max,fnnodes,fnodes,False,True,'Random Forest Hyperparameter Tuning','Leaf Nodes')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.5)
plt.show()
```

#### Instantiate, Fit and Predict with Tree-based Gradient Boosting Regression

Let's instantiate, fit and predict with a tree-based gradient boosting regression model.

Trying Out Hyperparameters:

* **loss** - the loss function, 'ls' for least squares, L2 norm solution
* **max_depth** - maximum number of decision tree layers, low **max_depth** - low model complexity / flexibility for each additive model for less ability to capture feature interactions and slower learning
* **learning_rate** - too fast and the solution may be unstable, diverge, and too slow and the solution make take too long

Just change these lines of code below:

```python
params = {
    'loss': 'ls',                                           # L2 Norm - least squares
    'max_depth': 2,                                         # maximum depth of each tree
    'learning_rate': 0.003,                                 # learning learning
    'criterion': 'mse'                                      # individual tree loss
}
```

Note, the above is a dictionary, a convenient Python class to flexibly store information, like the model hyperparameters.

```python
from sklearn.ensemble import GradientBoostingRegressor      # tree-based gradient boosting

learning_rate = 0.003                                      # hyperparamters
learning_rate_min = 0.000001; learning_rate_max = 1.0; learning_rate_n = 20

gnodes = 100
# gnodes_min = 2; gnodes_max=150; gnnodes = 20

params = {
    'loss': 'ls',                                           # L2 Norm - least squares
    'max_leaf_nodes': gnodes,                               # maximum depth of each tree
    'learning_rate': learning_rate,                         # constraint on magnitude of update with each estimator / for slow, robust learning
    'criterion': 'mse'                                      # tree construction criteria is mean square error over training
}

gb_model = GradientBoostingRegressor(n_estimators=1000,**params) # instantiate the prediction model
gb_model.fit(X_train,y_train['Production'])                    # train the model with the training data

plt.subplot(131)
visualize_model(gb_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Gradient Boosting, Learning Rate = ' + str(learning_rate))

plt.subplot(132)
check_model(gb_model,X_train.iloc[:,0],X_test.iloc[:,0],X_train.iloc[:,1],X_test.iloc[:,1],y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Gradient Boosting Cross Validation Plot, Learning Rate = ' + str(learning_rate))

plt.subplot(133)
tune_model(gb_model,X,y,'learning_rate',learning_rate_min,learning_rate_max,learning_rate_n,learning_rate,True,False,'Gradient Boosting Hyperparameter Tuning','Learning Rate')
#tune_model(gb_model,X,y,'max_leaf_nodes',gnodes_min,gnodes_max,gnnodes,gnodes,False,True,'Decision Tree Hyperparameter Tuning','Leaf Nodes')

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Summary

Let's look at all our models together for a visual comparison.

```python
from matplotlib import gridspec
fig = plt.figure(constrained_layout=False)
gs = gridspec.GridSpec(1,6, figure=fig,width_ratios = [1,1,1,1,1,1.2] )

visualize_model_sub(plt.subplot(gs[0]),linear_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Linear Model',legend=False)

visualize_model_sub(plt.subplot(gs[1]),ridge_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Ridge Model, Lambda = ' + str(lam),legend=False)

visualize_model_sub(plt.subplot(gs[2]),neigh_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'k-nearest Neighbour, k = ' + str(k),legend=False)

visualize_model_sub(plt.subplot(gs[3]),tree_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Decision Tree, Leaf Nodes = ' + str(nodes),legend=False)

visualize_model_sub(plt.subplot(gs[4]),forest_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Random Forest, Leaf Nodes = ' + str(fnodes),legend=False)

visualize_model_sub(plt.subplot(gs[5]),gb_model,X_train.iloc[:,0],X_test.iloc[:,0],-3.5,3.5,X_train.iloc[:,1],X_test.iloc[:,1],-3.5,3.5,y_train.iloc[:,0],y_test.iloc[:,0],prodmin,prodmax,'Gradient Boosting, Leaf Nodes = ' + str(gnodes),legend=True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=6, top=1.2, wspace=0.15, hspace=0.2)

fig.savefig('Multiple_ML_Models.jpg',dpi = 400, bbox_inches = 'tight')

plt.show()
```

#### Comments

We just ran fast to build a variety of machine learning models with model parameter training and hyperparameter tuning.

* I hope this was a help demonstration.

* See the linked lectures at the beginning of this workflow for much more information about machine learning.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin
On twitter I'm the @GeostatsGuy.

***

#### More on Michael Pyrcz and the Texas Center for Geostatistics:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)


# Minimizing memory usage: a matrix-free iterative solver
## How to deal with dense BEM matrices?

[In the previous section, I explained how to directly discretize a free surface using TDEs](sa_tdes). A downside of this approach is that the surface matrix can get very large very quickly. If I make the width of an element half as large, then there will be 2x many elements per dimension and 4x as many elements overall. And because the interaction matrix is dense, 4x as many elements leads to 16x as many matrix entries. In other words, $n$, the number of elements, scales like $O(h^2)$ in terms of the element width $h$. And the number of matrix rows or columns is exactly $3n$ (the 3 comes from the vector nature of the problem). That requires storing $9n^2$ entries. And, even worse, using a direct solver (LU decomposition, Gaussian elimination, etc) with such a matrix requires time like $O(n^3)$. Even for quite small problems with 10,000 elements, the cost of storage and solution get very large. And without an absolutely enormous machine or a distributed parallel implementation, solving a problem with 200,000 elements will just not be possible. On the other hand, in an ideal world, it would be nice to be able to solve problems with millions or even tens or hundreds of millions of elements.

Fundamentally, the problem is that the interaction matrix is dense. There are two approaches for resolving this problem:
1. Don't store the matrix!
2. Compress the matrix by taking advantage of low rank sub-blocks.

Eventually approach #2 will be critical since it is scalable up to very large problems. And that's exactly what I'll do in the next sections where I'll investigate low-rank methods and hierarchical matrices (H-matrices). However, here, I'll demonstrate approach #1 by using a matrix-free iterative solver. Ultimately, this is just a small patch on a big problem and it won't be a sustainable solution. But, it's immediately useful when you don't have a working implementation, are running into RAM constraints and are okay with a fairly slow solution. It's also useful to introduce iterative linear solvers since they are central to solving BEM linear systems.

When we solve a linear system without storing the matrix, [the method is called "matrix-free"](https://en.wikipedia.org/wiki/Matrix-free_methods). Generally, we'll just recompute any matrix entry whenever we need. How does this do algorithmically? The storage requirements drop to just the $O(n)$ source and observation info instead of the $O(n^2)$ dense matrix. And, as I'll demonstrate, for some problems, the runtime will drop to $O(n^2)$ instead of $O(n^3)$ because solving linear systems will be possible with a fixed and fairly small number of matrix-vector products.

## A demonstration on a large mesh.

To get started, I'll just copy the code to set up the linear system for the South America problem from the previous section. But, as a twist, I'll going to use a mesh with several times more elements. This surface mesh has 28,388 elements. As a result, the matrix would have 3x that many rows and columns and would require 58 GB of memory to store. That's still small enough that it could be stored on a medium sized workstation. But, it's too big for my personal computer!

```python
import cutde
import numpy as np
import matplotlib.pyplot as plt
from pyproj import Transformer

plt.rcParams["text.usetex"] = True
%config InlineBackend.figure_format='retina'
(surf_pts_lonlat, surf_tris), (fault_pts_lonlat, fault_tris) = np.load(
    "sa_mesh16_7216.npy", allow_pickle=True
)
```

```python
print("Memory required to store this matrix: ", (surf_tris.shape[0] * 3) ** 2 * 8 / 1e9)
```

    Memory required to store this matrix:  58.023255168

```python
transformer = Transformer.from_crs(
    "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs",
    "+proj=geocent +datum=WGS84 +units=m +no_defs",
)
surf_pts_xyz = np.array(
    transformer.transform(
        surf_pts_lonlat[:, 0], surf_pts_lonlat[:, 1], surf_pts_lonlat[:, 2]
    )
).T.copy()
fault_pts_xyz = np.array(
    transformer.transform(
        fault_pts_lonlat[:, 0], fault_pts_lonlat[:, 1], fault_pts_lonlat[:, 2]
    )
).T.copy()

surf_tri_pts_xyz = surf_pts_xyz[surf_tris]
surf_xyz_to_tdcs_R = cutde.compute_efcs_to_tdcs_rotations(surf_tri_pts_xyz)

fault_centers_lonlat = np.mean(fault_pts_lonlat[fault_tris], axis=1)
fault_lonlat_to_xyz_T = cutde.compute_projection_transforms(
    fault_centers_lonlat, transformer
)

fault_tri_pts_xyz = fault_pts_xyz[fault_tris]
fault_xyz_to_tdcs_R = cutde.compute_efcs_to_tdcs_rotations(fault_tri_pts_xyz)

fault_tri_pts_lonlat = fault_pts_lonlat[fault_tris]
fault_tdcs2_to_lonlat_R = cutde.compute_efcs_to_tdcs_rotations(fault_tri_pts_lonlat)

strike_lonlat = fault_tdcs2_to_lonlat_R[:, 0, :]
dip_lonlat = fault_tdcs2_to_lonlat_R[:, 1, :]

strike_xyz = np.sum(fault_lonlat_to_xyz_T * strike_lonlat[:, None, :], axis=2)
strike_xyz /= np.linalg.norm(strike_xyz, axis=1)[:, None]
dip_xyz = np.sum(fault_lonlat_to_xyz_T * dip_lonlat[:, None, :], axis=2)
dip_xyz /= np.linalg.norm(dip_xyz, axis=1)[:, None]

ft = np.float32
# The normal vectors for each triangle are the third rows of the XYZ->TDCS rotation matrices.
Vnormal = surf_xyz_to_tdcs_R[:, 2, :]
surf_centers_xyz = np.mean(surf_tri_pts_xyz, axis=1)
surf_tri_pts_xyz_conv = surf_tri_pts_xyz.astype(ft)

# The rotation matrix from TDCS to XYZ is the transpose of XYZ to TDCS.
# The inverse of a rotation matrix is its transpose.
surf_tdcs_to_xyz_R = np.transpose(surf_xyz_to_tdcs_R, (0, 2, 1)).astype(ft)
```

Proceeding like the previous section, the next step would be to construct our surface to surface left hand side matrix. But, instead, I'm just going to compute the action of that matrix without ever storing the entire matrix. Essentially, each matrix entry will be recomputed whenever it is needed. The `cutde.disp_free` and `cutde.strain_free` were written for this purpose.

First, let's check that the `cutde.disp_free` matrix free TDE computation is doing what I said it does. That is, it should be computing a matrix vector product. Since our problem is too big to generate the full matrix in memory, I'll just use the first 100 elements for this test.

First, I'll compute the matrix form. This should look familiar! I multiply the matrix by a random slip vector.

```python
test_centers = (surf_centers_xyz - 1.0 * Vnormal)[:100].astype(ft)
mat = cutde.disp_matrix(test_centers, surf_tri_pts_xyz_conv[:100], 0.25).reshape(
    (300, 300)
)
slip = np.random.rand(mat.shape[1]).astype(ft)
correct_disp = mat.dot(slip)
```

And now the matrix free version. Note that the slip is passed to the `disp_free` function. This makes sense since it is required for a matrix-vector product even though it is not required to construct the matrix with `cutde.disp_matrix`.

```python
test_disp = cutde.disp_free(
    test_centers, surf_tri_pts_xyz_conv[:100], slip.reshape((-1, 3)), 0.25
)
```

And let's calculate the error... It looks good for the first element. For 32-bit floats, this is machine precision.

```python
err = correct_disp.reshape((-1, 3)) - test_disp
err[0]
```

    array([ 4.4703484e-08, -1.7881393e-07, -5.4389238e-07], dtype=float32)

```python
np.mean(np.abs(err)), np.max(np.abs(err))
```

    (2.9506162e-07, 1.2218952e-06)

Okay, now that I've shown that `cutde.disp_free` is trustworthy, let's construct the full action of the left-hand side matrix. We need to transform all the rotation and extrapolation steps into a form that makes sense in an "on the fly" setting where we're not storing a matrix.

```python
offsets = [2.0, 1.0]
offset_centers = [(surf_centers_xyz - off * Vnormal).astype(ft) for off in offsets]
surf_xyz_to_tdcs_R = surf_xyz_to_tdcs_R.astype(ft)

# The extrapolate to the boundary step looked like:
# lhs = 2 * eps_mats[1] - eps_mats[0]
# This array stores the coefficients so that we can apply that formula
# on the fly.
extrapolation_mult = [-1, 2]

def matvec(x):
    # Step 1) Rotate slip into the TDE-centric coordinate system.
    slip_xyz = x.reshape((-1, 3)).astype(ft)
    slip_tdcs = np.ascontiguousarray(
        np.sum(surf_xyz_to_tdcs_R * slip_xyz[:, None, :], axis=2)
    )
    # Step 2) Compute the two point extrapolation to the boundary.
    out = np.zeros_like(offset_centers[0])
    for i, off in enumerate(offsets):
        out += extrapolation_mult[i] * cutde.disp_free(
            offset_centers[i], surf_tri_pts_xyz_conv, slip_tdcs, 0.25
        )
    out = out.flatten()

    # Step 3) Don't forget the diagonal Identity matrix term!
    out += x
    return out
```

```python
%%time
matvec(np.random.rand(surf_tris.shape[0] * 3))
```

    CPU times: user 8.16 s, sys: 0 ns, total: 8.16 s
    Wall time: 8.16 s

    array([-0.07595471,  0.14888829,  0.00571918, ...,  0.11337244,  0.06799013,  0.37440816], dtype=float32)

Great! We computed a matrix-free matrix-vector product! This little snippet below will demonstrate that the memory usage is still well under 1 GB proving that we're not storing a matrix anywhere.

```python
import os, psutil

process = psutil.Process(os.getpid())
print(process.memory_info().rss / 1e9)
```

    0.178192384

## Iterative linear solution

Okay, so how do we use this matrix-vector product to solve the linear system? Because the entire matrix is never in memory, direct solvers like LU decomposition or Cholesky decomposition are no longer an option. But, iterative linear solvers are still an option. The [conjugate gradient (CG) method](https://en.wikipedia.org/wiki/Conjugate_gradient_method) is a well-known example of an iterative solver. However, CG requires a symmetric positive definite matrix. Because our columns come from integrals over elements but our rows come from observation points, there is an inherent asymmetry to the boundary element matrices we are producing here. [GMRES](https://en.wikipedia.org/wiki/Generalized_minimal_residual_method) is an iterative linear solver that tolerates asymmetry. It's specifically a type of ["Krylov subspace"](https://en.wikipedia.org/wiki/Krylov_subspace) iterative linear solver and as such requires only the set of vectors:
\begin{equation}
\{b, Ab, A^2b, ..., A^nb\}
\end{equation}
As such, only having an implementation of the matrix vector product $Ab$ is required since the later iterates can be computed with multiple matrix vector product. For example, $A^2b = A(Ab)$.

To start, we compute the right-hand side which is nothing new or fancy.

```python
slip = np.sum(fault_xyz_to_tdcs_R * dip_xyz[:, None, :], axis=2)
rhs = cutde.disp_free(
    surf_centers_xyz.astype(ft), fault_pts_xyz[fault_tris].astype(ft), slip, 0.25
).flatten()
```

    /home/tbent/Dropbox/active/eq/cutde/cutde/fullspace.py:69: UserWarning: The slips input array has type float64 but needs to be converted to dtype float32. Converting slips to float32 may be expensive.
      warnings.warn(
    /home/tbent/Dropbox/active/eq/cutde/cutde/fullspace.py:78: UserWarning: The slips input array has Fortran ordering. Converting to C ordering. This may be expensive.
      warnings.warn(

Now, the fun stuff: Here, I'll use the [`scipy` implementation of GMRES](https://docs.scipy.org/doc/scipy/reference/generated/scipy.sparse.linalg.gmres.html). First, we need to do use the `scipy.sparse.linalg.LinearOperator` interface to wrap our `matvec` function in a form that the `gmres` function will recognize as a something that represents a linear system that can be solved.

```python
import time
import scipy.sparse.linalg as spla

# The number of rows and columns
n = surf_tris.shape[0] * 3

# The matrix vector product function that serves as the "backend" for the LinearOperator.
# This is just a handy wrapper around matvec to track the number of matrix-vector products
# used during the linear solve process.
def M(x):
    M.n_iter += 1
    start = time.time()
    out = matvec(x)
    print("n_matvec", M.n_iter, "took", time.time() - start)
    return out

M.n_iter = 0
lhs = spla.LinearOperator((n, n), M, dtype=rhs.dtype)
lhs.shape
```

    (85164, 85164)

And then we can pass that `LinearOperator` as the left hand side of a system of equations to `gmres`. I'm also going to pass a simple callback that will print the current residual norm at each step of the iterative solver and require a solution tolerance of `1e-4`.

```python
np.linalg.norm(rhs)
```

    26.985018

```python
soln = spla.gmres(
    lhs,
    rhs,
    tol=1e-4,
    atol=1e-4,
    restart=100,
    maxiter=1,
    callback_type="pr_norm",
    callback=lambda x: print(x),
)
soln = soln[0].reshape((-1, 3))
```

    n_matvec 1 took 8.181740522384644
    n_matvec 2 took 8.217356204986572
    0.21284150158882
    n_matvec 3 took 8.110730171203613
    0.026337904611568017
    n_matvec 4 took 8.143013715744019
    0.0058701005433943266
    n_matvec 5 took 8.101205348968506
    0.0016042871687649372
    n_matvec 6 took 8.221377849578857
    0.0004220627227557851
    n_matvec 7 took 8.234569311141968
    9.5677343550821e-05

As the figures below demonstrate, only eight matrix-vector products got us a great solution!

```python
inverse_transformer = Transformer.from_crs(
    "+proj=geocent +datum=WGS84 +units=m +no_defs",
    "+proj=longlat +ellps=WGS84 +datum=WGS84 +no_defs",
)
surf_xyz_to_lonlat_T = cutde.compute_projection_transforms(
    surf_centers_xyz, inverse_transformer
)
surf_xyz_to_lonlat_T /= np.linalg.norm(surf_xyz_to_lonlat_T, axis=2)[:, :, None]
soln_lonlat = np.sum(surf_xyz_to_lonlat_T * soln[:, None, :], axis=2)

plt.figure(figsize=(13, 6))
for d in range(3):
    plt.subplot(1, 3, 1 + d)
    cntf = plt.tripcolor(
        surf_pts_lonlat[:, 0], surf_pts_lonlat[:, 1], surf_tris, soln_lonlat[:, d]
    )
    plt.colorbar(cntf)
    plt.axis("equal")
    plt.xlim([-85, -70])
    plt.ylim([-50, 10])
    plt.title(
        ["$u_{\\textrm{east}}$", "$u_{\\textrm{north}}$", "$u_{\\textrm{up}}$"][d]
    )
plt.show()
```

## Performance and convergence

An important thing to note about the solution above is that only a few matrix-vector products are required to get to a high-level of accuracy. GMRES (and many other iterative linear and nonlinear optimization algorithms) converges at a rate proportional to the condition number of the matrix {cite:p}`Saad1986`. So in order to productively use an iterative linear solver, we need to have a matrix with a small condition number. It turns out that these free surface self-interaction matrices have condition numbers that are very close to 1.0, meaning that all the eigenvalues are very similar in magnitude. As a result, a highly accurate solution with GMRES requires less than ten matrix-vector products even for very large matrices.

Because of this dependence on the condition number, in the worst case, iterative solvers are not faster than a direct solver. However, suppose that we need only 10 matrix-vector products. Then, the runtime is approximately $10(2n^2)$ because each matrix-vector product requires $2n^2$ operations (one multiplication and one addition per matrix entry). As a result, GMRES is solving the problem in $O(n^2)$ instead of the $O(n^3)$ asymptotic runtime of direct methods like LU decomposition. So, in addition to requiring less memory, the matrix free method here forced us into actually using a faster linear solver. Of course, LU decomposition comes out ahead again if we need to solve many linear systems with the same left hand side and different right hand sides. That is not the case here but would be relevant for many other problems (e.g. problems involving time stepping).

The mess of code below builds a few figures that demonstrate these points regarding performance and accuracy as a function of the number of elements.

```python
import time

fault_L = 1000.0
fault_H = 1000.0
fault_D = 0.0
fault_pts = np.array(
    [
        [-fault_L, 0, -fault_D],
        [fault_L, 0, -fault_D],
        [fault_L, 0, -fault_D - fault_H],
        [-fault_L, 0, -fault_D - fault_H],
    ]
)
fault_tris = np.array([[0, 1, 2], [0, 2, 3]], dtype=np.int64)

results = []
for n_els_per_dim in [2, 4, 8, 16, 32, 48]:
    surf_L = 4000
    mesh_xs = np.linspace(-surf_L, surf_L, n_els_per_dim + 1)
    mesh_ys = np.linspace(-surf_L, surf_L, n_els_per_dim + 1)
    mesh_xg, mesh_yg = np.meshgrid(mesh_xs, mesh_ys)
    surf_pts = np.array([mesh_xg, mesh_yg, 0 * mesh_yg]).reshape((3, -1)).T.copy()
    surf_tris = []
    nx = ny = n_els_per_dim + 1
    idx = lambda i, j: i * ny + j
    for i in range(n_els_per_dim):
        for j in range(n_els_per_dim):
            x1, x2 = mesh_xs[i : i + 2]
            y1, y2 = mesh_ys[j : j + 2]
            surf_tris.append([idx(i, j), idx(i + 1, j), idx(i + 1, j + 1)])
            surf_tris.append([idx(i, j), idx(i + 1, j + 1), idx(i, j + 1)])
    surf_tris = np.array(surf_tris, dtype=np.int64)
    surf_tri_pts = surf_pts[surf_tris]
    surf_centroids = np.mean(surf_tri_pts, axis=1)

    fault_surf_mat = cutde.disp_matrix(surf_centroids, fault_pts[fault_tris], 0.25)
    rhs = np.sum(fault_surf_mat[:, :, :, 0], axis=2).flatten()

    start = time.time()
    eps_mats = []
    offsets = [0.002, 0.001]
    offset_centers = [
        np.mean(surf_tri_pts, axis=1) - off * np.array([0, 0, 1]) for off in offsets
    ]
    for i, off in enumerate(offsets):
        eps_mats.append(cutde.disp_matrix(offset_centers[i], surf_pts[surf_tris], 0.25))
    lhs = 2 * eps_mats[1] - eps_mats[0]
    lhs_reordered = np.empty_like(lhs)
    lhs_reordered[:, :, :, 0] = lhs[:, :, :, 1]
    lhs_reordered[:, :, :, 1] = lhs[:, :, :, 0]
    lhs_reordered[:, :, :, 2] = lhs[:, :, :, 2]
    lhs_reordered = lhs_reordered.reshape(
        (surf_tris.shape[0] * 3, surf_tris.shape[0] * 3)
    )
    lhs_reordered += np.eye(lhs_reordered.shape[0])
    direct_build_time = time.time() - start
    start = time.time()
    soln = np.linalg.solve(lhs_reordered, rhs).reshape((-1, 3))
    direct_solve_time = time.time() - start

    def matvec(x):
        extrapolation_mult = [-1, 2]
        slip = np.empty((surf_centroids.shape[0], 3))
        xrshp = x.reshape((-1, 3))
        slip[:, 0] = xrshp[:, 1]
        slip[:, 1] = xrshp[:, 0]
        slip[:, 2] = xrshp[:, 2]
        out = np.zeros_like(offset_centers[0])
        for i, off in enumerate(offsets):
            out += extrapolation_mult[i] * cutde.disp_free(
                offset_centers[i], surf_tri_pts, slip, 0.25
            )
        return out.flatten() + x

    n = surf_tris.shape[0] * 3

    def M(x):
        M.n_iter += 1
        return matvec(x)

    M.n_iter = 0
    lhs = spla.LinearOperator((n, n), M, dtype=rhs.dtype)
    start = time.time()
    soln_iter = spla.gmres(lhs, rhs, tol=1e-4)[0].reshape((-1, 3))
    iterative_runtime = time.time() - start

    l1_err = np.mean(np.abs((soln_iter - soln) / soln))
    results.append(
        dict(
            l1_err=l1_err,
            n_elements=surf_tris.shape[0],
            iterations=M.n_iter,
            direct_build_time=direct_build_time,
            direct_solve_time=direct_solve_time,
            iterative_runtime=iterative_runtime,
            direct_memory=rhs.nbytes + lhs_reordered.nbytes,
            iterative_memory=rhs.nbytes,
        )
    )
```

```python
import pandas as pd

results_df = pd.DataFrame({k: [r[k] for r in results] for k in results[0].keys()})
results_df["direct_runtime"] = (
    results_df["direct_build_time"] + results_df["direct_solve_time"]
)
results_df
```

      l1_err
      n_elements
      iterations
      direct_build_time
      direct_solve_time
      iterative_runtime
      direct_memory
      iterative_memory
      direct_runtime

      0
      0.000390
      8
      6
      0.001730
      0.000297
      0.196446
      4800
      192
      0.002027

      1
      0.000052
      32
      7
      0.001941
      0.000353
      0.045686
      74496
      768
      0.002294

      2
      0.000371
      128
      7
      0.007796
      0.001663
      0.152800
      1182720
      3072
      0.009458

      3
      0.000520
      512
      7
      0.085798
      0.037337
      0.748936
      18886656
      12288
      0.123135

      4
      0.000599
      2048
      7
      1.271554
      1.452238
      2.974715
      302039040
      49152
      2.723793

      5
      0.000887
      4608
      7
      6.120567
      13.449475
      11.854967
      1528934400
      110592
      19.570042

```python
plt.rcParams["text.usetex"] = False
```

```python
plt.figure(figsize=(8, 4))
plt.subplot(1, 2, 1)
plt.plot(results_df["n_elements"], results_df["direct_runtime"], label="direct")
plt.plot(results_df["n_elements"], results_df["iterative_runtime"], label="iterative")
plt.legend()
plt.title("Run time (secs)")
plt.subplot(1, 2, 2)
plt.plot(results_df["n_elements"], results_df["direct_memory"] / 1e6, label="direct")
plt.plot(
    results_df["n_elements"], results_df["iterative_memory"] / 1e6, label="iterative"
)
plt.legend()
plt.title("Memory usage (MB)")
plt.show()
```