## Data Analytics

### Bootstrap Hypothesis Testing in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

##  Bootstrap Hypothesis Testing Demonstration

We start with difference in means and then move to difference in variances. We answer the question:

* is the observed difference in the statistic between 2 sample sets significant?

* should we infer that the 2 sample sets are from 2 populations with different population parameters?

* or could the observed difference be due to random effect.

We will use bootstrap methods to sample realizations given the specified null hypothesis, $H_0$ is true to build the sampling distribution. Then we look at the observed effect relative to this smapling distribution to assess significance.

* the tests we will demonstrated have analytical solutions and we will show that our bootstrap methods reproduce these analytically-known distributions while providing a more general workflow for any significance problem.

### Boostrap and Analytical Methods for Hypothesis Testing, Difference in Means

* we calculate the hypothesis test for different in means with boostrap and compare to the analytical expression

* **Welch's t-test**: we assume the features are Gaussian distributed and the variance are unequal

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

#### Hypothesis Testing

Powerful methodology for spatial data analytics:

1. extracted sample set 1 and 2, the means look different, but are they?
2. should we suspect that the samples are in fact from 2 different populations?

Now, let's try the t-test, hypothesis test for difference in means.  This test assumes that the variances are similar along with the data being Gaussian distributed (see the course notes for more on this).  This is our test:

\begin{equation}
H_0: \mu_{X1} = \mu_{X2}
\end{equation}

\begin{equation}
H_1: \mu_{X1} \ne \mu_{X2}
\end{equation}

To test this we will calculate the t statistic with the bootstrap and analytical approaches.

#### The Welch's t-test for Difference in Means by Analytical Methods

We work with the following test statistic, *t statistic*, from the two sample sets.

\begin{equation}
\hat{t} = \frac{\overline{x}_1 - \overline{x}_2}{\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}}
\end{equation}

where $\overline{x}_1$ and $\overline{x}_2$ are the sample means, $s^2_1$ and $s^2_2$ are the sample variances and $n_1$ and $n_2$ are the numer of samples from the two datasets.

The critical value, $t_{critical}$ is calculated by the analytical expression by:

\begin{equation}
t_{critical} = \left|t(\frac{\alpha}{2},\nu)\right|
\end{equation}

The degrees of freedom, $\nu$, is calculated as follows:

\begin{equation}
\nu = \frac{\left(\frac{1}{n_1} + \frac{\mu}{n_2}\right)^2}{\frac{1}{n_1^2(n_1-1)} + \frac{\mu^2}{n_2^2(n_2-1)}}
\end{equation}

Alternatively, the sampling distribution of the t_{statistic} and t_{critical} may be calculated empirically with bootstrap.

The workflow proceeds as:

* shift both sample sets to have the mean of the combined data, $x_1$ → $x^*_1$, $x_2$ → $x^*_2$

* for each bootstrap realization, $\ell=1\ldots,L$

    * perform $n_1$ Monte Carlo simulations, draws with replacement, from sample set $x^*_1$

    * perform $n_2$ Monte Carlo simulations, draws with replacement, from sample set $x^*_2$

    * calculate the t_{statistic} realization, $\hat{t}^{\ell}$ given the resulting sample means $\overline{x}^{*,\ell}_1$ and $\overline{x}^{*,\ell}_2$ and the sample variances $s^{*,2,\ell}_1$ and $s^{*,2,\ell}_2$

* pool the results to assemble the $t_{statistic}$ sampling distribution

* calculate the cumulative probability of the observed t_{statistic}m, $\hat{t}$, from the boostrap distribution based on $\hat{t}^{\ell}$, $\ell = 1,\ldots,L$.

Here's some prerequisite information on the boostrap.

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The Bootstrap Approach (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out and compare the result to the analytical form of the confidence interval for the sample mean.

#### Objective

Provide an example and demonstration for:

1. interactive plotting in Jupyter Notebooks with Python packages matplotlib and ipywidgets
2. provide an intuitive hands-on example of confidence intervals and compare to statistical boostrap

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

#### Load the Required Libraries

The following code loads the required libraries.

```python
import os                                                 # set working directory, run executables
import matplotlib                                         # plotting
import matplotlib.pyplot as plt                           # plotting
import seaborn as sns                                     # plotting
import numpy as np                                        # working with arrays
import pandas as pd                                       # working with DataFrames
import statistics as stats                                # statistics like the mode
import random                                             # random drawing / bootstrap realizations of the data
from scipy.stats import t                                 # Student's t distribution for analytical solution
from scipy.stats import ttest_ind
from scipy.stats import f as F                            # Student's F distribution for analytical solution
from scipy.stats import percentileofscore                 # forward of CDF
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv')                # load our data table
```

Let's drop some samples so that we increase the variations in bootstrap samples for our demonstration below.

```python
np.random.seed(seed = 73073)
df = df.sample(frac = 0.2)                                # extract 50 random samples to reduce the size of the dataset
print('Using ' + str(len(df)) + ' number of samples')
```

    Using 58 number of samples

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
df.head()                                                 # display first 4 samples in the table as a preview
```

      X
      Y
      Facies
      Porosity
      Perm

      150
      20
      69
      0
      0.104484
      3.003225

      17
      400
      600
      1
      0.163054
      317.883581

      68
      640
      529
      1
      0.120547
      10.801778

      234
      490
      289
      1
      0.144525
      16.251107

      214
      470
      899
      1
      0.211944
      1302.937858

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 0 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()                                 # data summary statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      58.0
      418.275862
      231.457153
      10.000000
      272.500000
      400.000000
      560.000000
      990.000000

      Y
      58.0
      493.224138
      287.393614
      39.000000
      279.000000
      499.500000
      676.500000
      989.000000

      Facies
      58.0
      0.810345
      0.395452
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      58.0
      0.136501
      0.040473
      0.074349
      0.105662
      0.126192
      0.147067
      0.223661

      Perm
      58.0
      270.961923
      590.805185
      0.093252
      4.112400
      15.105651
      79.314434
      2372.383732

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display. I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly). We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0                                 # range of x values
ymin = 0.0; ymax = 1000.0                                 # range of y values
pormin = 0.05; pormax = 0.25;                             # range of porosity values
permmin = 0.01; permmax = 10000                           # range of permeability values
cmap = plt.cm.plasma                                      # color map
cumul_prob = np.linspace(0.0,1.0,100)                     # list of cumulative probabilities
```

Let's try out location maps, histograms and scatter plots.

```python
plt.subplot(231)                                          # location maps, histograms and scatter plots
plt.scatter(df['X'],df['Y'],s = 20,c = df['Porosity'],cmap = plt.cm.inferno,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = pormin,vmax = pormax)
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Porosity Location Map')

plt.subplot(232)
plt.scatter(df['X'],df['Y'],s = 20,c = df['Perm'],cmap = plt.cm.inferno,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = permmin,vmax = permmax,norm=matplotlib.colors.LogNorm())
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Permeability Location Map')

plt.subplot(233)
plt.scatter(df['X'],df['Y'],s = 20,c = df['Facies'],cmap = plt.cm.gray,linewidths = 0.3,edgecolor = 'black',alpha = 0.8,vmin = 0.0,vmax = 1.0)
plt.scatter(x=[800,800],y=[900,830],c=[1,0],cmap = plt.cm.gray,s=20,edgecolor = 'black')
plt.text(850, 890, r'Sand', fontsize=12); plt.text(850, 820, r'Shale', fontsize=12)
plt.colorbar(); plt.xlabel('X (m)'); plt.ylabel('Y (m)'); plt.title('Facies Location Map')

plt.subplot(234)
plt.hist(df['Porosity'],color = 'red',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)))
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')

plt.subplot(235)
plt.hist(df['Perm'],color = 'red',alpha = 0.3,edgecolor='black',bins=np.logspace(np.log10(permmin),np.log10(permmax),int(len(df)/3)))
plt.xlabel('Permeability (mD)'); plt.ylabel('Frequency'); plt.title('Permeability Histogram'); plt.xscale('log')

plt.subplot(236)
plt.scatter(df['Porosity'],df['Perm'],s = 20,color = 'red',alpha = 0.3,edgecolor='black')
plt.ylabel('Permeability (mD)'); plt.xlabel('Porosity (fraction)'); plt.title('Permeability-Porosity Scatter Plot')
plt.yscale('log')
sns.kdeplot(x=df['Porosity'],y=df['Perm'],color = 'black',alpha = 0.2,levels = 4)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=2.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### By-Facies Data Visualization

Let's check the statistics by-facies. First, let's break up the DataFrame, df, into separate DataFrames for each facies:

1. Sand (facies = 0)
2. Shale (facies = 1)

```python
df_sand = df[df['Facies']==1]                             # extract sand and shale facies samples to a new DataFrame
df_shale = df[df['Facies']==0]
```

Now we can plot the distributions by facies.

```python
plt.subplot(131)
plt.hist(df_shale['Porosity'],color = 'gray',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='shale')
plt.hist(df_sand['Porosity'],color = 'yellow',alpha = 0.2,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='sand')
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')
plt.legend()

plt.subplot(132)
plt.hist(df_shale['Perm'],color = 'gray',alpha = 0.3,edgecolor='black',bins=np.logspace(np.log10(permmin),np.log10(permmax),int(len(df)/3)),label='shale')
plt.hist(df_sand['Perm'],color = 'yellow',alpha = 0.2,edgecolor='black',bins=np.logspace(np.log10(permmin),np.log10(permmax),int(len(df)/3)),label='sand')
plt.legend()

plt.xlabel('Permeability (mD)'); plt.ylabel('Frequency'); plt.title('Permeability Histogram'); plt.xscale('log')

plt.subplot(133)
plt.scatter(df_shale['Porosity'],df_shale['Perm'],s = 20,color = 'gray',alpha = 0.3,edgecolor='black',label='shale')
plt.scatter(df_sand['Porosity'],df_sand['Perm'],s = 20,color = 'yellow',alpha = 0.3,edgecolor='black',label='sand')
plt.legend()

plt.ylabel('Permeability (mD)'); plt.xlabel('Porosity (fraction)'); plt.title('Permeability-Porosity Scatter Plot')
plt.yscale('log')
sns.kdeplot(x=df['Porosity'],y=df['Perm'],color = 'black',alpha = 0.2,levels = 4)

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

### Bootstrap Hypothesis Test

#### Boostrap Student's t-test, Welch's Test Difference in Means

First let's calculate the t-statistic observed in our dataset, assuming unequal variances with the Welch's t-test.

* the observed effect, difference in means, $\overline{x}_1 - \overline{x}_2$, standardized by standard error, $\sqrt{\frac{s^2_1}{n_1} + \frac{s^2_2}{n_2}}$

```python
t_stat_obs, p_value_obs = ttest_ind(df_shale['Porosity'],df_sand['Porosity'],equal_var=False)
print("The observed Welch's t-statistic is " + str(round(t_stat_obs,2)))
```

    The observed Welch's t-statistic is -8.17

Given our knowledge about the theory, the sampling distribution will be standard t-distributed,we already know that this effect will be significant!

To build the sampling distribution given the null hypothesis is correct, we need to modifed the data so that the null hypothesis is true.

* we shift both datasets to have the same mean.

The procedure to accomplish this is:

1. make a deep copy of the porosity values as a 1D ndarray from each sand and shale DataFrames. We do this to ensure we don't change the original DataFrame, i.e. a slice would be a shallow copy and the mean would be shifted for our original data set!

2. calculate the common mean by pooling all of the data together

3. apply the affine correction to shift the mean, $X^{*} = X - \overline{X}
+ \overline{X}_{pooled}$

Let's make a plot to check the result.

```python
por_sand = df_sand.copy(deep = True)['Porosity'].values   # make a deepcopy of porosity by-facies
por_shale = df_shale.copy(deep = True)['Porosity'].values

global_average = np.average(np.concatenate([por_sand,por_shale])) # shift the means to be equal to the globla mean
por_sand_s = por_sand - np.average(por_sand) + global_average
por_shale_s = por_shale - np.average(por_shale) + global_average

plt.subplot(111)                                          # plot the new shifted histograms
plt.hist(por_shale_s,color = 'gray',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='shale')
plt.hist(por_sand_s,color = 'yellow',alpha = 0.2,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='sand')
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

We now have two datasets for which the null hypothesis is true.

\begin{equation}
H_0: \mu_{\phi_{sand}} = \mu_{\phi_{shale}}
\end{equation}

We are saying, these two sample sets came from populations with the same means.

* hypothesis tests are inferential, they are related to the population.

Let's demonstrate the Welch's t-test, no assumption of equal population variances

* bootstrap provides a new realization of each sample set

* we calculate the realization of the t-statistic given the mean and standard deviation from the sample set realization

\begin{equation}
\hat{t}^{\ell} = \frac{\overline{x^{\ell}}_1 - \overline{x^{\ell}}_2}{\sqrt{\frac{{s^2}^{\ell}_1}{n_1} + \frac{{s^2}^{\ell}_2}{n_2}}}
\end{equation}

```python
L = 10000                                                 # number of bootstrap realizations, $L$, to sample the t-statistic
alpha = 0.05                                              # alpha level

t_stat = np.zeros(L); p_value = np.zeros(L)               # make arrays to store the t-statistic, and p-value realizations

u = stats.variance(por_sand)/stats.variance(por_shale)    # calculate the DOF for the Welch's t-test, a bit complicated
dof = ((1/len(por_shale)) + u/len(por_sand))**2/(1/(len(por_shale)**2*(len(por_shale)-1)) + u**2/(len(por_sand)**2*(len(por_sand)-1)) )

for l in range(0, L):                                     # loop over bootstrap realizations
    por_shale_real = random.choices(por_shale_s, weights=None, cum_weights=None, k=len(por_shale_s)) # bootstrap realization of shale porosity
    por_sand_real = random.choices(por_sand_s, weights=None, cum_weights=None, k=len(por_sand_s)) # bootstrap realization of sand porosity
    t_stat[l], p_value[l] = ttest_ind(por_shale_real,por_sand_real,equal_var=False)

lower_bound = np.percentile(t_stat,alpha*0.5*100)
upper_bound = np.percentile(t_stat,(1-alpha*0.5)*100)

plt.subplot(121)                                          # original porosity histograms
plt.hist(df_shale['Porosity'],color = 'gray',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='shale')
plt.hist(df_sand['Porosity'],color = 'yellow',alpha = 0.2,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='sand')
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')
plt.legend()

plt.subplot(122)                                          # bootstrap t-statistic distribution by bootstrap, analytical and observed t-statistic
plt.hist(t_stat,color = 'red',alpha = 0.3,edgecolor='black',bins=np.linspace(-10,10,100),density=True,label='bootstrap')
plt.axvline(x=t_stat_obs,c='black',label='observed')
plt.xlabel('t-statistic Bootstrap Realizations'); plt.ylabel('Frequency'); plt.title("Bootstrap and Analytical Welch's t-test")

analytical = t.pdf(np.linspace(-10,10,100), dof,loc=0,scale=1.0) # plot the analytically-derived sampling distribution

plt.plot(np.linspace(-10,10,100),analytical,color = 'black',alpha=0.4,label="analytical")
plt.fill_between(np.linspace(-10,10,100), 0, analytical, where = np.linspace(-10,10,100) <= lower_bound, facecolor='red', interpolate=True, alpha = 0.5,label = "Reject")
plt.fill_between(np.linspace(-10,10,100), 0, analytical, where = np.linspace(-10,10,100) >= upper_bound, facecolor='red', interpolate=True, alpha = 0.5)
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('The p-value of the observed effect is ' + str(round((percentileofscore(t_stat,t_stat_obs))/100,10)))
```

#### Bootstrap Student's t-test, Equal, Pooled Variance Difference in Means

Let's repeat this for the Student's t-test, equal variances, pooled variance method.

The t-statistic bootstrap realizations are calculated as:

\begin{equation}
\hat{t}^{\ell} = \frac{ \overline{x^{\ell}}_1 - \overline{x^{\ell}}_2 }{ \sqrt{\left( \frac{1}{n_1}+\frac{1}{n_2} \right) \left( \frac{(n_1-1){s^2}^{\ell}_1 + (n_2-1){s^2}^{\ell}_2}{n_1 + n_2 - 2} \right)  }}
\end{equation}

For the SciPy statististics module, we just need to change the 'equal_var' parameter to True.

But we need to go an extra step, since the test assumes the variances are equal, we must correct the data to have equal variance and not just equal means, to honor the null hypothesis to calculate the correct t-statistic sampling distribution

* if you would like to see for yourself, try changing the code below:

```python
correct_samples_equal_variance = True
```

to:

```python
correct_samples_equal_variance = False
```

 and observe the mismatch between the analytical and bootstrap t-statistic distributions.

```python
L = 10000                                                 # number of bootstrap realizations, $L$, to sample the t-statistic
alpha = 0.05                                              # alpha level
correct_samples_equal_variance = True                     # True - correct sample variances, False - only correct the means

t_stat = np.zeros(L); p_value = np.zeros(L)               # make arrays to store the t-statistic, and p-value realizations

global_average = np.average(np.concatenate([por_sand,por_shale])) # shift the means to be equal to the globla mean
global_stdev = np.std(np.concatenate([por_sand,por_shale])) # shift the standard deviation to be equal to the globla standard deviation

por_sand_affine = (por_sand - np.average(por_sand))*global_stdev/np.std(por_sand) + global_average
por_shale_affine = (por_shale - np.average(por_shale))*global_stdev/np.std(por_shale) + global_average

for l in range(0, L):                                     # loop over bootstrap realizations
    if correct_samples_equal_variance:
        por_shale_real = random.choices(por_shale_affine, weights=None, cum_weights=None, k=len(por_shale_s)) # bootstrap realization of shale porosity
        por_sand_real = random.choices(por_sand_affine, weights=None, cum_weights=None, k=len(por_sand_s)) # bootstrap realization of sand porosity
    else:
        por_shale_real = random.choices(por_shale_s, weights=None, cum_weights=None, k=len(por_shale_s)) # bootstrap realization of shale porosity
        por_sand_real = random.choices(por_sand_s, weights=None, cum_weights=None, k=len(por_sand_s)) # bootstrap realization of sand porosity

    t_stat[l], p_value[l] = ttest_ind(por_shale_real,por_sand_real,equal_var=True)

lower_bound = np.percentile(t_stat,alpha*0.5*100)
upper_bound = np.percentile(t_stat,(1-alpha*0.5)*100)

plt.subplot(121)                                          # original porosity histograms
plt.hist(df_shale['Porosity'],color = 'gray',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='shale')
plt.hist(df_sand['Porosity'],color = 'yellow',alpha = 0.2,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='sand')
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')
plt.legend()

plt.subplot(122)                                          # bootstrap t-statistic distribution by bootstrap, analytical and observed t-statistic
plt.hist(t_stat,color = 'red',alpha = 0.3,edgecolor='black',bins=np.linspace(-10,10,100),density=True,label='bootstrap')
plt.axvline(x=t_stat_obs,c='black',label='observed')
plt.xlabel('t-statistic Bootstrap Realizations'); plt.ylabel('Frequency'); plt.title("Bootstrap and Analytical Pooled Variance t-test")

analytical = t.pdf(np.linspace(-10,10,100), len(df_sand)+len(df_shale)-2,loc=0,scale=1.0) # plot the analytically-derived sampling distribution
plt.plot(np.linspace(-10,10,100),analytical,color = 'black',alpha=0.4,label="analytical")
plt.fill_between(np.linspace(-10,10,100), 0, analytical, where = np.linspace(-10,10,100) <= lower_bound, facecolor='red', interpolate=True, alpha = 0.5,label = "Reject")
plt.fill_between(np.linspace(-10,10,100), 0, analytical, where = np.linspace(-10,10,100) >= upper_bound, facecolor='red', interpolate=True, alpha = 0.5)
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('The p-value of the observed effect is ' + str(round((percentileofscore(t_stat,t_stat_obs))/100,10)))
```

#### Bootstrap F-test for Difference in Variances

Now let's try out the diffence in Variances,

\begin{equation}
H_0: \frac{\sigma^2_2}{\sigma^2_1} = 1.0
\end{equation}

where, $\sigma^2_2 > \sigma^2_1$.

The F-statistic bootstrap realizations are calculated as:

\begin{equation}
\hat{F}^{\ell} = \frac{{s^2}^{\ell}_2}{{s^2}^{\ell}_1}
\end{equation}

Once again we need to go an extra step, since the test assumes the variances are equal, to do this we pool all the data together and bootstrap resample for each new sample set the the approapriate number $n_{1}$ and $n_{2}$.

* if you would like to see for yourself, try changing the code below:

```python
correct_samples_equal_variance = True
```

to:

```python
correct_samples_equal_variance = False
```

 and observe the mismatch between the analytical and bootstrap t-statistic distributions.

```python
L = 10000                                                 # number of bootstrap realizations, $L$, to sample the t-statistic
alpha = 0.05                                              # alpha level
correct_samples_equal_variance = True                     # True - correct sample variances, False - only correct the means

if stats.pvariance(por_shale) > stats.pvariance(por_sand):
    F_stat_obs = stats.pvariance(por_shale)/stats.pvariance(por_sand)
    na = len(por_shale); nb = len(por_sand)
else:
    F_stat_obs = stats.pvariance(por_sand)/stats.pvariance(por_shale)
    na = len(por_sand); nb = len(por_shale)

F_stat = np.zeros(L)                                      # make arrays to store the t-statistic, and p-value realizations

for l in range(0, L):                                     # loop over bootstrap realizations
    por_shale_real = random.choices(np.concatenate([por_sand,por_shale]), weights=None, cum_weights=None, k=len(por_shale)) # bootstrap realization of shale porosity
    por_sand_real = random.choices(np.concatenate([por_sand,por_shale]), weights=None, cum_weights=None, k=len(por_sand)) # bootstrap realization of sand porosity

    if stats.variance(por_shale) > stats.variance(por_sand): # consistent with numerator and denominator selection
        F_stat[l] = stats.variance(por_shale_real)/stats.variance(por_sand_real)
    else:
        F_stat[l] = stats.variance(por_sand_real)/stats.variance(por_shale_real)

lower_bound = np.percentile(F_stat,alpha*0.5*100)
upper_bound = np.percentile(F_stat,(1-alpha*0.5)*100)

plt.subplot(121)                                          # original porosity histograms
plt.hist(df_shale['Porosity'],color = 'gray',alpha = 0.3,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='shale')
plt.hist(df_sand['Porosity'],color = 'yellow',alpha = 0.2,edgecolor='black',bins = np.linspace(pormin,pormax,int(len(df)/3)),label='sand')
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity Histogram')
plt.legend()

plt.subplot(122)                                          # bootstrap t-statistic distribution by bootstrap, analytical and observed t-statistic
plt.hist(F_stat,color = 'red',alpha = 0.3,edgecolor='black',bins=np.linspace(0,15,100),density=True,label='bootstrap')
plt.axvline(x=F_stat_obs,c='black',label='observed')
plt.xlabel('F-statistic Bootstrap Realizations'); plt.ylabel('Frequency'); plt.title("Bootstrap and Analytical F-test Difference in Variances")
plt.xlim(0,15)

analytical = F.pdf(np.linspace(0,15,100),(na-1),(nb-1)) # plot the analytically-derived sampling distribution
plt.plot(np.linspace(0,15,100),analytical,color = 'black',alpha=0.4,label="analytical")
upper_bound = F.ppf(1-alpha, na-1, nb-1)

plt.fill_between(np.linspace(0,15,100), 0, analytical, where = np.linspace(0,15,100) >= upper_bound, facecolor='red', interpolate=True, alpha = 0.5)
plt.legend()

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('The p-value of the observed effect is ' + str(round((100 - percentileofscore(F_stat,F_stat_obs))/100,5)))
```

#### Observations

Some observations:

* bootstrap can be applied to calculate the sampling distribution for a hypothesis test

* we can then calculate the p-value of the observed effect as the percentile of this sampling distribution

* we must set the sample data to honor the null hypothesis and any associated assumptions, e.g. equal variances

* we bootstrap the sampling statistic, t-statistic, as it is formulated given all hypothesis assumptions, this is the reason we don't just bootstrap the effect, the difference in means

#### Comments

This was a demonstration of bootstrap hypothesis testing for the significance in difference in means between 2 sample sets in Jupyter Notebook Python.

I have many other demonstrations on data analytics and machine learning, e.g. on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
from scipy.optimize import curve_fit

data = np.fromfile("fit_result_sedimenting_n_1p595r_1p45fix_0p0535_wav_532.dat", dtype=np.float64)
data = data.reshape(len(data)//10,10)

x = data[:,0]*0.0535
y = data[:,1]*0.0535
z = data[:,2]*0.0535
```

```python
import numpy as np
import matplotlib.pyplot as plt

### Just some matplotlib tweaks
import matplotlib as mpl

mpl.rcParams["xtick.direction"] = "in"
mpl.rcParams["ytick.direction"] = "in"
mpl.rcParams["lines.markeredgecolor"] = "k"
mpl.rcParams["lines.markeredgewidth"] = 1
mpl.rcParams["figure.dpi"] = 200
from matplotlib import rc
rc('font', family='serif')
rc('text', usetex=True)
rc('xtick', labelsize='medium')
rc('ytick', labelsize='medium')
rc("axes", labelsize = "large")
def cm2inch(value):
    return value/2.54
```

```python
def traj_sedim(x, a, b):
    return a * x + b

```

```python
# We take the data where we are sure that the particle is far enough from the wall - and from the
z = z[0:6000]
time = np.arange(0,len(z)) / 100

```

```python
plt.figure(figsize=(cm2inch(6*1.68),cm2inch(6)))

plt.plot(time, z)
plt.xlabel("$t$ (s)")
plt.ylabel("$z$ ($\\mu$m)")
plt.tight_layout(pad=0.1)
plt.savefig("traj_sedimenting.svg")
```

```python
popt, pcov = curve_fit(traj_sedim, time, z)
```

```python
plt.plot(time, z)
plt.plot(time, traj_sedim(time, *popt))
plt.xlabel("$t$ (s)")
plt.ylabel("$z$ ($\\mu$m)")
```

```python
popt
```

    array([-0.26669228, 30.0528722 ])

The sedimentation drift velocity for a spherical particle in bulk writes:

\begin{equation}
v_\mathrm{sed} = \frac{2}{9} \frac{\Delta \rho a^2 g}{\eta}
\end{equation}

```python
a = 1.45e-6
v_th = 2/9 * 50 * a ** 2 * 9.81 / 0.001 * 1e6
```

```python
v_th
```

    0.22917250000000006

```python
def msd(x, Dt):
	"""Function that returns the MSD for a list of time indices Dt for a trajectory x"""
	_msd = lambda x, Dt: np.mean((x[:-Dt] - x[Dt:]) ** 2)
	return [_msd(x, i) for i in Dt]
```

```python
Dt = np.arange(1,1000,5)
```

```python
MSD = msd(z, Dt)

```

```python
def msd_th(x, d, v, n):
    return 2*d*x + v**2 * x**2 + n**2

def msd_log(x, d, v, n):
    return np.log( 2*d*x + v**2 * x**2 + n**2)
```

```python
plt.plot(np.log(Dt/100),np.log(MSD), "o")
```

```python
popt, pcov = curve_fit(msd_log, Dt/100, np.log(MSD), p0 = [0.13, 0.22, 4e-2], maxfev=2000, ftol=1e-2)
```

```python
popt
```

    array([0.14092578, 0.22348519, 0.04612557])

```python
perr = np.sqrt(np.diag(pcov))
```

```python
perr
```

    array([0.0007985 , 0.0008066 , 0.00149602])

```python
Dt = np.array([*np.arange(1,10,1), *np.arange(10,100,10), *np.arange(100,1000,100)])
MSD = msd(z, Dt)

```

```python
plt.figure(figsize=( cm2inch(6*1.68),cm2inch(6)))
plt.loglog(Dt/100,MSD, "o")
t_th=np.linspace(0.7e-2, 20,10000)
plt.plot(t_th, msd_th(t_th, *popt), color = "k", linewidth=2)
plt.xlim((0.8e-2, 15))
plt.ylim((None, 10))
plt.xlabel("$\Delta t$ (s)")
plt.ylabel("MSD ($\mathrm{\mu m^2}$)")
plt.tight_layout(pad=0.1)
plt.savefig("msd_sedimentation.svg")
```

```python
relat_err_D = abs(0.144 - 0.146)/0.146
relat_err_D*100
```

    1.3698630136986314

```python
relat_err_v = abs(222 - 229)/229
relat_err_v*100
```

    3.056768558951965

```python
popt[1]
```

    0.2217820942226398

```python
popt[0]
```

    0.14377799719977521

```python
9/2 * 222e-9 * 0.001 / (1.45e-6 ** 2) / 9.81
```

    48.43513074212656

```python
(50 - 48.5) / 50 * 100
```

    3.0

```python

```


```python
%matplotlib inline
%load_ext Cython
#http://people.duke.edu/~ccc14/sta-663-2016/18D_Cython.html

import matplotlib.pylab as plt
import random
import numpy as np
```

#### Matrix Multiplication

Let's write a function to multiply 2 square (NxN) matrices together.

First is out niave wave following the way we would do it by hand.

\begin{equation}
A =\begin{pmatrix}
 a_{11} & a_{12} & \cdots & a_{1m} \\
 a_{21} & a_{22} & \cdots & a_{2m} \\
\vdots & \vdots & \ddots & \vdots \\
 a_{n1} & a_{n2} & \cdots & a_{nm} \\
\end{pmatrix}
\end{equation}

\begin{equation}
B=\begin{pmatrix}
 b_{11} & b_{12} & \cdots & b_{1p} \\
 b_{21} & b_{22} & \cdots & b_{2p} \\
\vdots & \vdots & \ddots & \vdots \\
 b_{m1} & b_{m2} & \cdots & b_{mp} \\
\end{pmatrix}
\end{equation}

\begin{equation}
C = AB
\end{equation}

\begin{equation}
C=\begin{pmatrix}
 c_{11} & c_{12} & \cdots & c_{1p} \\
 c_{21} & c_{22} & \cdots & c_{2p} \\
\vdots & \vdots & \ddots & \vdots \\
 c_{n1} & c_{n2} & \cdots & c_{np} \\
\end{pmatrix}
\end{equation}

\begin{equation}
c_{ij} = a_{i1}b_{1j} + ... + a_{im}b_{mj} = \sum_{k=1}^m a_{ik}b_{kj}
\end{equation}

```python
def mat_mul_py(A, B):
    # Set all the elemets of C to 0
    C = [[0 for i in range(len(A))] for j in range(len(A))]
    # For each col
    for i in range(len(A)):
        # For each row
        for j in range(len(A)):
            # Do the sum for the elements
            for k in range(len(A)):
                # Do the sum of the elements
                C[i][j] += A[i][k] * B[k][j]
    return C
```

```python
size = 32

A = np.random.random((size,size)).astype(np.float32)
B = np.random.random((size,size)).astype(np.float32)

%time x = mat_mul_py(A,B)
```

Using what you've learned so far modify the code below to improve the speed of your matrix multiplication function by trying to remove the yellow portions of the anotated code.

Look back at chapter 2 Cython Fast for hints.

```cython
%%cython -a

def mat_mul_cy(A, B):
    # Set all the elemets of C to 0
    C = [[0 for i in range(len(A))] for j in range(len(A))]
    # For each col
    for i in range(len(A)):
        # For each row
        for j in range(len(A)):
            # Do the sum for the elements
            for k in range(len(A)):
                # Do the sum of the elements
                C[i][j] += A[i][k] * B[k][j]
    return C
```

```python
%time x = mat_mul_cy(A,B)
```

```python
#%load mat_mul_cy_fast.py
# NOTE: %%cython has to be the first line so remove the load line as well

def mat_mul_cy_fast(A,B,C):
    print("\033[1m\033[91m\n\nLoad the py file for the answer\n\n\033[0m")
```

```python
size = 512

A = np.random.random((size,size)).astype(np.float32)
B = np.random.random((size,size)).astype(np.float32)
C = np.zeros((A.shape[0], B.shape[1])).astype(np.float32)

%time mat_mul_cy_fast(A,B,C)
```

## Sometimes it's best to just use the predefined functions though....

```python
#A = np.random.random((size,size)).astype(np.float32)
#B = np.random.random((size,size)).astype(np.float32)
#C = np.zeros((A.shape[0], A.shape[1])).astype(np.float32)
#%time C = A * B
```

Again look though the code below and try to remove as much of the yellow portions as possible to increase the speed of the code.

```cython
%%cython -a

cimport cython

### import and use C functions
#cdef extern from "complex.h":
#    double cabs(double complex)

def mandel(x, y, max_iters):
    c = complex(x, y)
    z = 0.0j
    for i in range(max_iters):
        z = z*z + c
        if z.real*z.real + z.imag*z.imag >= 4:
        # if cabs(z) >= 2:
            return i
    return max_iters

def create_fractal(xmin, xmax, ymin, ymax, image, iters):
    height, width = image.shape

    pixel_size_x = (xmax - xmin)/width
    pixel_size_y = (ymax - ymin)/height

    for x in range(width):
        real = xmin + x*pixel_size_x
        for y in range(height):
            imag = ymin + y*pixel_size_y
            color = mandel(real, imag, iters)
            image[y, x]  = color

```

```python
gimage = np.zeros((1080, 1920), dtype=np.uint32)
xmin, xmax, ymin, ymax = [-2.0, 1.0, -1.0, 1.0]
iters = 100

%time create_fractal(xmin, xmax, ymin, ymax, gimage, iters)

plt.figure(figsize=(15,15))
plt.grid(False)
plt.imshow(gimage, cmap='viridis')
plt.show()
```

```python
#%load fractal.py
# NOTE: %%cython has to be the first line so remove the load line as well

def create_fractal_cython(xmin, xmax, ymin, ymax, gimage, iters):
    print("\033[1m\033[91m\n\nLoad the py file for the answer\n\n\033[0m")
```

```python
gimage = np.zeros((1080, 1920), dtype=np.uint32)
xmin, xmax, ymin, ymax = [-2.0, 1.0, -1.0, 1.0]
iters = 50

%time create_fractal_cython(xmin, xmax, ymin, ymax, gimage, iters)

plt.figure(figsize=(16,9))
plt.grid(False)
plt.imshow(gimage, cmap='viridis')
plt.show()
```

```python

```


# What is this about?

Hi! Within this notebook I started to teach myself the theory and practice of Bayesian inference and MCMC sampling by means of a toy problem. In case you stumble across an error or other ways to improve this notebook (not unlikely!), please let know. Thanks! :-)

A big thanks goes out to Josh Fass for excellent discussions and offering expertise on this topic!

#### Before we dive in...
... please execute the two cells below. It will install some packages from conda and load modules and classes relevant to this notebook. For implementation details, have a look at fourier_series and MH_sampler in mcmc_sampler.py

```python
!conda install --yes -c conda-forge matplotlib numpy scipy dask tqdm
```

    Collecting package metadata (current_repodata.json): done
    Solving environment: done

    # All requested packages already installed.

```python
import matplotlib.pyplot as plt
import numpy as np
import dask
from dask.distributed import Client
from mcmc_sampler import fourier_series, MH_sampler

### You may change n_workers to whatever number of
### processors you'd like to use here
client = Client(threads_per_worker=1, n_workers=5)
```

# Sampling parameters using Bayesian inference

## 1.) A toy problem: A Fourier series of unknown length
Fourier series (a cosine series) are used by most molecular machanics force fields in order to model torsion potentials. The general form of a torsion potential includes force constants $V_n$ (i.e. an amplitude in this context) and the phase angles $\delta_n$ as paramters:

\begin{equation}\label{eq:TorsionSeries}
V_{tors}(\phi) = \sum_{n} {1 \over 2 } V_n (1 + cos(n\phi + \delta_n))
\end{equation}

For a given torsion around a bond B-C, the number of terms in this series depends on the symmetry of the molecular fragment A-B-C-D. Also, it is possible that some terms have a $V_n$ of zero. In the context of this notebook, we will use a simplified version of the equation above and just assume a single set of parameters $ \theta_N: \{a_1, a_2, ..., a_N\}$:

\begin{equation}\label{eq:CosineSeries}
f(x)_{N} = \sum_{n}^{N} a_n cos(nx)
\end{equation}

We will assume that there is some actual (true) process that generates data, which we can observe, following eq. \ref{eq:CosineSeries}. However, we do not know how exactly this process looks like (i.e. we do not know the precise parameters $a_n$ in eq. \ref{eq:CosineSeries}), we just know that it must be a process following eq. \ref{eq:CosineSeries}. The objective of this notebook is the exploration of ways to use Bayesian statistics to find out about the nature of the true process. That means, we want to know the parameters $\theta_N$ and $N$ in eq. \ref{eq:CosineSeries} of the true process. However, we will only be able to obtain a limited number of samples (observations) from that true process. To make things worse, we cannot observe these samples without noise and we do not know how large this noise is. Since the data is sparse and also noisy, it is much more useful to know the distribution of parameters $\theta_N$ and $N$ given the data generated from the true process, rather than estimating the precise parameter values. This setting is ideal for Bayesian statistics.

## 2.) Using Bayesian Statistics for Model Inference
In Bayesian inference, we are searching for the posterior distribution of the parameters, $p(\theta_N|Y)$, i.e. we are interested in the parameter distribution that generated the observed data $Y$. It is important to realize that we are not interested in the true parameter values $\hat{\theta}_N$, but really the distribution of these parameter values under the observed data. Since we are also interested in quantifying the noise contained in the true process, we want to include an unknown noise parameter into our considerations. We will assume that the true process contains some noise following a Gaussian distribution, so that $y_i = f(x_i) + \epsilon$, where $\epsilon \sim \mathcal{N}(0,\sigma^2)$, thus we will include $\sigma^2$ as an unknown parameter.

The posterior distribution for a given model $\mathcal{M}_k: f(x)_k$, with $k>0$ is defined as follows:

\begin{equation}\label{eq:BayesEquation}
p(\theta_k, \sigma^2|Y) = {{p(Y|\theta_k,\sigma^2) p(\theta_k) p(\sigma^2)} \over p(Y)} \propto p(Y|\theta_k,\sigma^2) p(\theta_k) p(\sigma^2)
\end{equation}

- $p(Y)$: Evidence
- $p(Y|\theta_k)$: Likelihood
- $p(\theta_k)$: Prior on $\theta_k$
- $p(\sigma^2)$: Prior on $\sigma^2$
- $p(\theta_k, \sigma^2|Y)$: Posterior

### Evidence
The evidence $p(Y)$ can be treated as a normalizing constant as long as one is only interested in the posterior distribution of a particular model $\theta_k$ (i.e. with fixed $N$). In that case, the posterior distribution is only known up to this normalizing constant, which is fine. However, if one wants to compare different models ${\mathcal{M}_k}$ (and we want to do that at a later point!), it will become important to not ignore that normalizing constant.

### Likelihood
The likelihood is a measure for the probability that the given data can be observed under the model parameters $\theta_k$ (it is not a likelihood though!). We will assume that each observed realization $Y_i$ is drawn from a Gaussian distribution (see above). However, we do not know the exact parameters of that distribution. Our task is to find these parameters, $(\mu_i, \sigma^2)$ during the Bayesian inference. The parameters $(\mu_i, \sigma^2)$ will be informed by our choice of $\theta_k$, i.e.

\begin{equation}\label{eq:LikelihoodMu}
\mu_i(\theta_k) = f(x_i)_{\theta_k}
\end{equation}

\begin{equation}\label{eq:LikelihoodSigma}
Y_i \sim \mathcal{N}(\mu_i,\sigma^2)
\end{equation}

\begin{equation}\label{eq:Likelihood}
p(Y|\theta,\sigma) = \prod_i {1 \over \sqrt{2 \pi \sigma^2}} exp\big( { -(Y_i-\mu_i)^2 \over 2 \sigma^2}\big)
\end{equation}

### The prior

The prior probabilities $p(\theta_k)$ and $p(\sigma^2)$ reflect our prior knowledge of the parameter vector $\theta_k$ and the noise parameter $\sigma^2$. In case we already know that the parameter is likely to be in specific range or if certain parameter values are not resonable (e.g. in a physics-based model) then the prior probability can reflect that knowledge. Other than that, one can also be rather objective and assign equal prior probility to all realizations of $\theta_k$. The latter is exactly what we will do in the following. We assign non-zero probability to all parameters on the interval $[b,c]$ and zero probability otherwise:

\[
    p(a_i) =
\begin{cases}
    (c-b)^{-1},& \text{if } b < a_i < c \\
    0, & \text{otherwise}
\end{cases}
\]

\begin{equation}\label{eq:PriorCombined}
p(\theta_k) = \prod_i^k p(a_i)
\end{equation}

The prior over the noise parameter will be an inverse gamma distribution, i.e. $\sigma \sim \mathcal{IG}(\alpha,\beta)$, where $\beta=1$ and $\alpha$ is a hyperparameter that we have to specify later:

\begin{equation}\label{eq:InverseGamma}
p_\alpha(\sigma) = {1 \over \Gamma(\alpha)} \sigma ^{-\alpha-1} exp\big(-{1 \over \sigma}\big)
\end{equation}

### The posterior

The posterior distribution, see eq. \ref{eq:BayesEquation}, in most cases cannot be solved analytically and must be sampled using special algorithms. That means, instead of computing the posterior directly, we will draw samples from the posterior distribution, which should converge to the true posterior distribution if we use an infinite amount of sampling. The sampling algorithm that we are going to use is called Metropolis-Hastings and is probably one of the most common algorithms from the Markov Chain Monte Carlo family. It will be explained in the following section.

## 3.) Metropolis-Hastings Algorithm

The original aim of this algorithm is to construct a Markov Chain that has the same distribution as the desired target distribution (in our case the posterior probability distribution). A Markov Chain describes a "chain" (i.e. a sequence) of states (i.e. realizations of the random variable, here $\theta$), in which the probability to realize the current state only depends on the previous state. This is called the Markov property. Assuming that the states are living in a finite state space, the Markov property is (we will stick to $\theta$ being our random variable):

\begin{equation}\label{eq:MarkovProperty}
p(\theta^{(t+1)} = \theta' | \theta^{(t)} = \theta, ... , \theta^{(0)} = z ) =
p(\theta^{(t+1)} = \theta' | \theta^{(t)} = \theta )
\end{equation}

Here, $\theta^{(t+1)}$ is the current state of the Markov Chain and $\theta^{(0)}, \theta^{(1)}, ... , \theta^{(t)}$ are all previous states. Another way of looking at the Markov property is to realize that these are transition functions, that is

\begin{equation}\label{eq:TransitionFunction}
A(\theta, \theta') =  p(\theta^{(t+1)} = \theta' | \theta^{(t)} = \theta )
\end{equation}

If we now assume that the Markov Chain is reversible (i.e. if we get the same result irrespective of running it backwards or forwards), we can apply the detailed balance condition:

\begin{equation}\label{eq:DetailedBalance}
p(\theta|Y) A(\theta, \theta') = p(\theta'|Y) A(\theta', \theta)
\end{equation}

Integrating out $\theta$ results in

\begin{equation}\label{eq:DetailedBalanceIntegrated}
\int d\theta p(\theta|Y) A(\theta, \theta') = p(\theta'|Y)
\end{equation}

Eq. \ref{eq:DetailedBalanceIntegrated} is an important result, since it shows that sampling from the target distribution $p(\theta'|Y)$ is invariant with respect to the transition function $A(\cdot ,y)$ as long as we ensure that we are markovian and reversible! This is where the Metropolis-Hasting Algorithm comes into the game. Metropolis suggested to use the following algorithm (again we use $\theta$ as our random variable):

The Markov Chain is initialized at state $\theta^{(t)} = \theta^{(0)}$
\begin{enumerate}
\item Propose a new state $\theta'$ from a proposal distribution $q(\theta^{(t)}, \theta')$
\item Evaluate the posterior distribution (the target distribution), $p(\theta',Y)$ at the proposed state.
Then, calculate the acceptance probability $\alpha(\theta^{(t)}, \theta')$ for this transition:

\begin{equation}\label{eq:AcceptanceProbability}
\alpha(\theta^{(t)}, \theta') = min\Big(1, {p(\theta'|Y) q(\theta',\theta^{(t)})
\over p(\theta^{(t)}|Y) q(\theta^{(t)},\theta')} \Big)
\end{equation}

Draw a random number from a uniform distribution $u \propto unif(0,1)$. Update the state $t+1$ of the chain according to

\[
    \theta^{(t+1)} =
\begin{cases}
    \theta',& \text{if } u \leq \alpha(\theta^{(t)}, \theta')\\
    \theta^{(t)}, & \text{otherwise}
\end{cases}
\]

\end{enumerate}

The acceptance probability $\alpha$ (see eq. \ref{eq:AcceptanceProbability}) in the MH algorithm is related to the Markov property (see eq. \ref{eq:DetailedBalance}) through the relation $\alpha(\theta^{(t)}, \theta') = { A(\theta, \theta') \over q(\theta, \theta')}$. Thus, one can rewrite the acceptance probability as

\begin{equation}\label{eq:AcceptanceProbability_Markov}
p(\theta|Y) A(\theta, \theta') = min \Big( p(\theta|Y) q(\theta, \theta'), p(\theta'|Y) q(\theta', \theta) \Big)
\end{equation}

This algorithm allows Markovian sampling using a Markov Chain Monte Carlo strategy (MCMC). This particular algorithm is maybe not the most efficient one, but it is (A) easy to implement and (B) sufficiently efficient for low dimensional problems like the ones we are dealing with in this example.

### Reading

* "Monte Carlo Strategies in Scientific Computing", Jun S. Liu

## 4.) Let's Generate Some Data!

Next, we will generate data $Y$ from a process that follows eq. \ref{eq:CosineSeries} and has three non-zero terms, $a_1, a_2, a_3$. Also this process contains noise from a zero-mean Gaussian distribution. The parameters are

$a_1  = 5.10, a_2 = 0.13, a_3 = 3.24, \sigma^2 = 0.3$

```python
### Parameters
p = np.array([5.10, 0.13, 3.24])

### You may either reuse these x,y values, which should get you close
### to the outcome of my calculations or just generate your own data.
x = np.array([-3.14159265, -2.74159265, -2.34159265, -1.94159265, -1.54159265, -1.14159265,
     -0.74159265, -0.34159265,  0.05840735,  0.45840735,  0.85840735,  1.25840735,
      1.65840735,  2.05840735,  2.45840735,  2.85840735])

y = np.array([-8.92247276, -6.19605699, -0.7293233,   1.37097261, -0.65165646, -0.68584825,
      1.95369845,  6.46307948,  8.36168418,  5.47697352, -0.07383572, -1.2556803,
      0.19878288,  1.5263483,  -2.68581297, -6.69607447])

### If you want to generate your own data,
### uncomment the following lines. The generated
### data will contain a zero-mean noise term.
#x = np.arange(-np.pi,np.pi,0.4)
#y = fourier_series(x,p) + np.random.normal(0,np.sqrt(0.3),size=x.size)

plt.plot(x,y)
plt.xlabel("x")
plt.ylabel("y")
plt.show()
```

## 5.) Model Inference Using within-model MH Sampling

In the first step of this notebook tutorial, we want to sample the parameter posterior distribution from each model $\mathcal{M}_k$ individually using the MH algorithm . Since we will sample $\theta_k$ for each model seperately and the individual models do not know about each other, we call this approach within-model sampling, i.e. we sample each set of parameters from its own distribution $\theta_k \sim p(\theta_k | k, Y)$.

### MC Propagation and Hyperparameters

Some parameters must be set by us in advance. These hyperparameters are the step width $w$ and the shape parameter $\alpha$ of the inverse gamma distribution (see eq. \ref{eq:InverseGamma}). A new state $(\theta_k, \sigma^2)'$ is proposed by adding a random perturbation to the current state of the Markov Chain. This perturbation follows a zero-mean normal distribution with standard deviation $s$ (this is our "step width"). By using a normal distribution we also guarantee symmetric moves. Specificually for the vector components $a_i$

\begin{equation}\label{eq:Propagation}
a_i' = a_i^{(t)} + w_i
\end{equation}

\begin{equation*}
w_i \sim \mathcal{N}(0, s)
\end{equation*}

And we will do the same thing for the proposal of the variance ${\sigma^2}'$, however we will propose on ${\sigma}'$ in order to always have ${\sigma^2}' > 0$. It is generally encouraged to play around with the these hyperparameters. A nice lesson is to vary the step width $s$ and see how the number of accepted moves is going to vary!

### Start the Sampling!

In order to start the sampling we will need to guess some initial parameters for each model. Of course we don't set it to the correct parameters, instead we use the correct parameters and add considerable amount of noise. Ideally, one would repeat these runs with different initial parameter settings in order to confirm that they actually all converge to the same stationary distribution (remember the Markov Property eq. \ref{eq:MarkovProperty} ?). However, in our case we only carry out a single (but long) run for each model.

Now execute the below code block in order start the sampling. This should take about 15 minutes on a desktop computer, if you initialized at least 5 dask workers (see code block in the very beginning of this notebook!). We need several sampling steps in order to get converged results for the calculation of the evidence at a later stage. A total of 100000 steps per run seems to be reliable here.

```python
### This is our step width 's'
step_width  = 0.05
### This is our prior width for the inverse gamma distribution
sigma_width = 2.0
### This is our reference that defines the boundaries on
### parameter prior.
p_ref       = np.array([0., 6])
### This is the initial value of the standard deviation.
### We will use the same value for each MH run
sigma0      = 0.1

### This is a list of initial parameter vectors. They are of
### varying length, going from 2 up to 6 parameters.
p0_list = [p[:2] + 1.0 * np.random.random(2),
           p     + 1.0 * np.random.random(p.shape),
           np.concatenate((p + 1.0 * np.random.random(p.shape),[1.0])),
           np.concatenate((p + 1.0 * np.random.random(p.shape),[1.0, 1.0])),
           np.concatenate((p + 1.0 * np.random.random(p.shape),[1.0, 1.0, 1.0]))
           ]

### Let's loop over some models and see how that
### changes the Bayesian quantities and the parameters
lazy_results = list()
def worker(sampler, *args):
    sampler.run(*args)
    return sampler
for p0 in p0_list:
    sampler = MH_sampler(step_width,
                         sigma_width,
                         p_ref,
                         x,
                         y,
                         fourier_series)

    lazy_result = dask.delayed(worker)(sampler, p0, sigma0, 100000)
    lazy_results.append(lazy_result)
sampler_list = list(dask.compute(*lazy_results))
```

### Understanding the Results of the MH Sampling

We started with parameters that were somewhat different from the ones of the true model (c.f. compare variables p0_list and p). Therefore we get low posterior probabilities in the beginning of the sampling. It is customary to discard these samples ('burn in') and only keep those that a drawn from the stationary posterior. In our case, we don't really have to do it, since we have lots of sampling, however we do it anyway since it is common practice. We will identify the first 2000 steps as the 'burn in' period, after this period the $-ln(posterior)$ over all MCMC steps (see below) seems to be stable (note the different scale of the individual y-axis).

```python
fig_traj, ax_traj = plt.subplots(1,5, figsize=(20,5))
for i in range(5):
    sampler          = sampler_list[i]
    accept_posterior = np.array(sampler.accept_posterior)[10:]
    ax_traj[i].plot(-np.log(accept_posterior))
    ax_traj[i].set_title(f"{i+2} parameters")
    ax_traj[i].set_ylabel("-ln(posterior)")
    ax_traj[i].set_xlabel("step")
fig_traj.tight_layout()
fig_traj.show()
```

### Parameter distribution

Next, let's look at the distribution of the parameters (see below, the dashed lines indicate the true values). We find that the MH sampler recovers the true parameters. For $\mathcal{M}_2$ we find a broad distribution of both parameters $a_1$ and $a_2$, this is not surprising since it is too simple a model to be able to capture the true data. The somewhat broadend parameter distribution indicates that a large number of parameter realizations is possible given data, since all of these realizations result in poor posterior probability. The correct model,  $\mathcal{M}_3$ on the other hand shows nicely distributed parameters, matching the true ones quite nicely. The other 3 models, $\mathcal{M}_4$, $\mathcal{M}_5$ and $\mathcal{M}_6$, show similar distributions but parameters $a_4$, $a_5$ and $a_6$ are sampled close to zero in order cope with the fact that the true model does not need those parameters.

Also see the overview of all the models in the graphs of the second row below (the dashed line indicates the true model). One sees that no solution of $\mathcal{M}_2$ is actually coming close to the true model.

The below codeblock should take about 10 minute to finish.

```python
burn_in = 2000

fig_parms, ax_parms = plt.subplots(2,5, figsize=(20,10))
for i in range(5):
    sampler      = sampler_list[i]
    accept_theta = sampler.accept_theta[burn_in::10]
    maxbins      = 0.
    for d in range(accept_theta.shape[1]):
        n, bins, patches = ax_parms[0][i].hist(accept_theta[:,d], label=f"a{d+1}")
        if np.max(n)>maxbins:
            maxbins = np.max(n)
    ax_parms[0][i].vlines(x=p[0], ymin=0, ymax=maxbins, linestyles="--")
    ax_parms[0][i].vlines(x=p[1], ymin=0, ymax=maxbins, linestyles="--")
    ax_parms[0][i].vlines(x=p[2], ymin=0, ymax=maxbins, linestyles="--")
    ax_parms[0][i].set_title(f"{i+2} parameters")
    ax_parms[0][i].set_ylabel("Counts")
    ax_parms[0][i].set_xlabel("Parameter value")
    ax_parms[0][i].legend()

    for theta in accept_theta:
        ax_parms[1][i].plot(x, fourier_series(x, theta), linewidth=0.5)
    ax_parms[1][i].set_ylabel("y")
    ax_parms[1][i].set_xlabel("x")

for i in range(5):
    ax_parms[1][i].plot(x, fourier_series(x, p),
                        color="black", linestyle="--",
                        linewidth=4)

fig_parms.tight_layout()
fig_parms.show()
```

### Model evidence
The logical conclusion up until now would be (assuming we do not know the true model), that all but $\mathcal(M)_2$ are possible. If we apply the rule of Occam's razor, we would just assume that $\mathcal(M)_3$ is correct, since it is the most simplistic one among the remaining 4 models. However, the definition of simplistic is not quite objective and generally is not merely based on the number of parameters in a given model. In the next steps, we want to find out the true model and hopefully fulfilling Occam's at the same time using a less subjective approach. In order to do so, we want to recall the normalizing constant $p(Y)$ introduced in the beginning. This normalazing constant, also called evidence, is important in the context of model selection for a given model $\mathcal{M}_k$. It can actually be computed from quantities we already know, see

\begin{equation}\label{eq:Evidence}
p(Y) = p(Y|\mathcal{M}_k) = \int p(\theta_k|\mathcal{M}_k) p(Y|\theta_k, \mathcal{M}_k) d\theta_k
\end{equation}

We can use this quantity for estimating the model posterior, i.e. a quantification of the probability that a given model $\mathcal{M}_k$ is true given the data $Y$.

\begin{equation}\label{eq:ModelPosterior}
p(\mathcal{M}_k|Y) \propto p(Y|\mathcal{M}_k) p(\mathcal{M}_k)
\end{equation}

This is exactly what we need to make an argument about which model is more likely than another. However, in practice the above integral is hard to compute. One way to approximate it, is using a harmonic mean estimator over the $N$ likelihood samples from the within-model parameter posterior (which comes with problems of its own, but it is sufficient for us) and assuming a uniform model prior $p(\mathcal{M}_k)$.  Then,

\begin{equation}\label{eq:ModelPosteriorEstimate}
\widetilde{p}(\mathcal{M}_k|Y) = \Bigg[ {1 \over N} \sum_i^N {1 \over P(Y|\theta_i,\mathcal{M}_k)} \Bigg]_{p(\theta | Y,\mathcal{M}_k)}^{-1}
\end{equation}

By setting the individual model evidence estimates $\widetilde{p}(\mathcal{M}_k|Y)$ for all $\mathcal{M}_k$ into relation of each other, one can quantify the support for a given model over another one given the data $Y$. These relations are called Bayes factors. The higher the ratio $\widetilde{p}(\mathcal{M}_i|Y) / \widetilde{p}(\mathcal{M}_j|Y)$, the more likely model $\mathcal{M}_i$ over $\mathcal{M}_j$. These Bayes factors are computed below as well as the individual model evidence estimates for each model (left graph). It is apparent that model $\mathcal{M}_3$ is the most likely one given the data. However, we would not get the same result if we would try to pick the model with the highest likelihood (see right graph). The maximum liklihood for each model would indicate that $\mathcal{M}_6$ is actually the best model!

### Reading
* Weinberg, M. D. (2012). Computing the bayes factor from a markov chain monte carlo simulation of the posterior distribution. Bayesian Analysis, 7(3), 737–770.
* http://alumni.media.mit.edu/~tpminka/statlearn/demo/
* https://radfordneal.wordpress.com/2008/08/17/the-harmonic-mean-of-the-likelihood-worst-monte-carlo-method-ever/

```python
burn_in = 2000

likelihood_list = [np.array(sampler_list[0].accept_likelihood)[burn_in:],
                   np.array(sampler_list[1].accept_likelihood)[burn_in:],
                   np.array(sampler_list[2].accept_likelihood)[burn_in:],
                   np.array(sampler_list[3].accept_likelihood)[burn_in:],
                   np.array(sampler_list[4].accept_likelihood)[burn_in:]
                  ]

evidence_list = [1./np.mean(1./likelihood_list[0]),
                 1./np.mean(1./likelihood_list[1]),
                 1./np.mean(1./likelihood_list[2]),
                 1./np.mean(1./likelihood_list[3]),
                 1./np.mean(1./likelihood_list[4])
                ]

fig_li_ev, ax_li_ev = plt.subplots(1,2)
for i in range(5):
    sampler           = sampler_list[i]
    evidence          = evidence_list[i]
    likelihood        = np.max(likelihood_list[i])
    ax_li_ev[0].bar(i+2, evidence, color="black")
    ax_li_ev[1].bar(i+2, likelihood, color="black")

ax_li_ev[0].set_xlabel("k")
ax_li_ev[0].set_ylabel("Evidence")
ax_li_ev[1].set_xlabel("k")
ax_li_ev[1].set_ylabel("max(Likelihood)")

fig_li_ev.tight_layout()
fig_li_ev.show()
print("Bayes Factor M3/M2:", evidence_list[1]/evidence_list[0])
print("Bayes Factor M3/M4:", evidence_list[1]/evidence_list[2])
print("Bayes Factor M3/M5:", evidence_list[1]/evidence_list[3])
print("Bayes Factor M3/M6:", evidence_list[1]/evidence_list[4])
```

## 6.) Model Inference Using Across-model sampling with RJ-MCMC

Ok, so we have seen that we can recover the correct model by computing the evidence for each single model. But we need lots of sampling (100000 steps) in order to get converged results. Also, we need to carry out sampling for each model seperately. To be fair, there are probably more efficent sampling algorithms than MH for this kind of inference problem. Also, we did not optimize the hyperparameters. Nonetheless, there is an elegant way to get the correct answer with much less sampling effort, called Reversible Jump MCMC (RJ-MCMC). Instead of explicitly sampling the posterior of each individual model, one can also sample over a posterior that lives on the parameter space of different models simultaniously. This approach is called across-model sampling and effectively samples from the distribution $(k,\theta_k) \sim p(k, \theta_k | Y)$. The Markov Chain will then have spent time in the different models in proportion to their model evidence. This is especially efficient for models that are nested, i.e. when parameters from one model can be carried over to another model.

### Just another way of Metropolis-Hasting'ing

How does RJ-MCMC work? It is essentially a variation of standard MH sampling. Because all we will do in RJ-MCMC is just normal MCMC but with proposal and target distributions (i.e. the posterior distribution) living on different (sub)spaces with different dimensionality (this point will become clear later)! The general acceptance probability for RJ-MCMC is (note that we omit $\sigma$ here for the sake of clarity):

\begin{equation}\label{eq:RjmcmcAcceptance}
\alpha(k, \theta_{k}, k', \theta'_{k'}) = min\Bigg(1, { p(k', \theta'_{k'}|Y) q(k, \theta_{k}, k', \theta'_{k'})
                                                  \over p(k, \theta_{k}|Y) q(k', \theta'_{k'},k, \theta_{k}) } \Bigg)
\end{equation}

This is really not so much different than the acceptance probability for regular MH (see eq. \ref{eq:AcceptanceProbability})! The only difference is that now we sample from the joint posterior distribution $p(k, \theta_k | Y)$ and propose from the joint distribution $q(k, \theta_k, \cdot)$. The posterior distribution $p(k, \theta|Y)$ is computed similarly as the one for within-model sampling (see eq. \ref{eq:BayesEquation}). However, the proposal distribution  will be computed in a different manner in order to reflect the difference in dimensionalty (if any) between proposal and target distribution. Furthermore, the general MH algorithm will be slightly adapted as follows:

\begin{enumerate}
\item Generate a new state *within* $\theta'_k$ from a proposal distribution $q_2(\theta^{(t)_k}, \theta'_k)$ (yet to be defined).
\item Propose an across model move according to $q_1(k^{(t)}, k')$ (yet to be defined).
\item Generate an updated parameter $\theta'_{k'}$ according to the within and across model move.
\item Evaluate the posterior distribution according to eq. \ref{eq:RjmcmcAcceptance} and standard MH rejection/acceptance scheme.
\end{enumerate}

The following two paragraphs will outline two general ways to implement this algorithm.

#### The general across-model move

For a proposal to jump from model $k$ to $k+1$ (let's call this an expansion move, since we "expand" our model), the proposal probability can be stated as

\begin{equation*}
q(k+1, \theta_{k+1},k, \theta_{k}) = q_1(k+1, k) q_2(\theta_{k+1}, \theta_{k})
\end{equation*}

The parameter proposal can be further marginalized as

\begin{equation*}
q_2(\theta_{k+1}, \theta_{k}) = q^{(k+1)}_2(\theta_{k+1}| \theta_{k}) q^{(1:k)}_2(\theta_{k})
\end{equation*}

Here, $q^{(k+1)}_2)$ is the proposal conditional probability for the $k+1$ element in the parameter vector. The $q^{(1:k)}_2$ probability accounts for the proposal of the first $k$ elements in the parameter vector, this is the regular MH proposal probability.

Conversly, the marginalized parameter proposal probability for an across model move from $k+1$ to $k$ (i.e. a contraction move) is (note that we just keep the $(1:k)$ parameters):

\begin{equation*}
q_2(\theta_{k}, \theta_{k+1}) = q^{(1:k)}_2(\theta_{k+1})
\end{equation*}

If we combine the above considerations with eq. \ref{eq:RjmcmcAcceptance}, we get an acceptance probability for an expansion move (assuming symmetric proposal $q^{(1:k)}_2(\cdot)$ ):

\begin{equation}\label{eq:RjmcmcAcceptance_GeneralCaseExpansion}
\alpha(k, \theta_{k}, k+1, \theta_{k+1}) = min\Bigg(1, { p(k+1, \theta_{k+1}|Y) q_1(k, k+1)
                                                   \over p(k, \theta_{k}|Y) q_1(k+1, k) q^{(k+1)}_2(\theta_{k+1}|                                                              \theta_{k})
                                                   }
                                              \Bigg)
\end{equation}

Note that the acceptance probability for the contraction move is just the inverse of eq. \ref{eq:RjmcmcAcceptance_GeneralCaseExpansion}.

The across model moves introduced so far sample the additional element in the parameter vector during an expansion move $u = \theta^{(k+1)}_{k+1}$ from $u \sim q^{(k+1)}_2(\theta_{k+1}| \cdot)$. While doing this, we ensured to have no overall change in dimensionality for an expansion move and its corresponding contraction move. That is, we imposed the following important boundary condition to any across model proposal ($dim(u)$ is the number of elements of vector $u$, including zero):

\begin{equation}\label{eq:BoundaryConditionMove}
k + dim(u) = k' + dim(u')
\end{equation}

#### Using deterministic functions to inform across model moves

In many cases, it is beneficial to use an informative deterministic function after the parameter proposal step in order to transform the proposed parameter to something reasonable. This concept can be incorporated into RJ-MCMC, as long as this function has a proper inverse function, so that we can propose the reverse move. Then, an expansion move (after drawing $u$) is set to

\begin{equation*}
\theta_{k+1} = h(\theta_k, u) = (\theta_k, g(u))
\end{equation*}

whereas the corresponding contraction move is attempted according to ($h'$ and $g'$ are the reversed functions)

\begin{equation*}
(\theta_{k}, u) = h'(\theta_{k+1}) = (\theta^{(1:k)}_{k+1}, g'(\theta^{(k)}_{k+1}))
\end{equation*}

If we use a deterministic function to manipulate the integration variable (remember that we still try to solve eq. \ref{eq:DetailedBalanceIntegrated}), we need a Jacobian factor $|J|$ to account for the change of variables in the integration variable. The expansion move then is

\begin{equation}\label{eq:RjmcmcAcceptance_DeterministicCaseExpansion}
\alpha(k, \theta_{k}, k+1, \theta_{k+1}) = min\Bigg(1, { p(k+1, \theta_{k+1}|Y) q_1(k, k+1) q_2(u_k)
                                                   \over p(k, \theta_{k}|Y) q_1(k+1, k) q_2(u_{k+1})}
                                                   \Big| {\partial(h'(\theta_{k+1})) \over \partial(\theta_k, u_k)}                                                      \Big|
                                              \Bigg)
\end{equation}

Similarly to the general case, the acceptance probability for the contraction move is obtained from the reverse of eq. \ref{eq:RjmcmcAcceptance_DeterministicCaseExpansion}.

Note how the Jacobian will be unity after setting $h(\theta_k, u) = (\theta_k, u)$ and $h'(\theta_{k+1}) = (\theta^{(1:k)}_{k+1})$ and we obtain the result of the general case (eq. \ref{eq:RjmcmcAcceptance_GeneralCaseExpansion}).

### Jump into the Sampling

The code block in the next cell will execute RJ-MCMC sampling with the "general" case (see eq. \ref{eq:RjmcmcAcceptance_GeneralCaseExpansion}). Note that parameters for an expansion move are proposed according to

\begin{equation}\label{eq:PropagationExpansion}
a_{k+1}' = a_k^{(t)} + w_k
\end{equation}

\begin{equation*}
w_k \sim \mathcal{N}(0, s_e)
\end{equation*}

The parameter $s_e$ controls the width of the proposal distribution $q_2(\theta^{(t)_k}, \theta'_{k+1})$. The implementation below can also use deterministic type moves (set the variable 'dete' to 'True') by using $g(u) = s_e exp(u)$ and $g'(a_{k+1}) = log(a_{k+1})-log(s_e)$.

## Reading

* Waagepetersen, R., & Sorensen, D. (2001). A tutorial on reversible jump MCMC with a view toward applications in QTL-mapping. International Statistical Review, 69(1), 49–61.
* Green, P. J. (1995). Reversible jump Markov chain monte carlo computation and Bayesian model determination. Biometrika, 82(4), 711–732.
* Hastie, D. I., & Green, P. J. (2012). Model choice using reversible jump Markov chain Monte Carlo. Statistica Neerlandica, 66(3), 309–338.
* Godsill, S. J. (2001). On the relationship between markov chain monte carlo methods for model uncertainty. Journal of Computational and Graphical Statistics, 10(2), 230–248.

```python
### This is our step width 's'
step_width  = 0.05
### This is our prior width for the inverse gamma distribution
sigma_width = 2.0
### This is our reference that defines the boundaries on
### parameter prior.
p_ref       = np.array([0., 6])
### This is the initial value of the standard deviation.
### We will use the same value for each MH run
sigma0      = 0.1
### This is a parameter that controls the mapping for the
### dimension jumping
se = 1.0
### Use deterministic sampling
dete = False

### This is a list of initial parameter vectors. They are of
### varying length, going from 2 up to 6 parameters.
p0_list = [p[:2] + 1.0 * np.random.random(2),
           p     + 1.0 * np.random.random(p.shape),
           np.concatenate((p + 1.0 * np.random.random(p.shape),[1.0])),
           np.concatenate((p + 1.0 * np.random.random(p.shape),[1.0, 1.0])),
           np.concatenate((p + 1.0 * np.random.random(p.shape),[1.0, 1.0, 1.0]))
           ]

### Let's loop over some models and see how that
### changes the Bayesian quantities and the parameters
lazy_results = list()
def worker(sampler, *args):
    sampler.run(*args)
    return sampler
for p0 in p0_list:
    sampler = MH_sampler(step_width,
                         sigma_width,
                         p_ref,
                         x,
                         y,
                         fourier_series)

    ### This is samplig by using the general RJ-MCMC
    lazy_result = dask.delayed(worker)(sampler, p0, sigma0, 10000, se, True, dete)
    lazy_results.append(lazy_result)
sampler_list_rj = list(dask.compute(*lazy_results))
```

### Analysis of the RJ-MCMC Sampling

Note how we only need 1/10 of the sampling from the previous example where we had to sample each modell individually. If we closely look at the trajectories below, we see that even a much smaller number of sampling steps would have been sufficient to arrive at the correct model. However, we also see that although the correct model is found early for most intitial conditions, the $-ln(posterior)$ still decreases and in real live we likely want to have more sampling in order to ensure we have enough within-model samples. Also, it is important to realize that the RJ-MCMC sampler arrives at the correct model independent of the starting conditions. This is great!

```python
fig_traj, ax_traj = plt.subplots(3, 5, figsize=(15,10))
for i in range(5):
    sampler          = sampler_list_rj[i]
    accept_posterior = sampler.accept_posterior
    accept_dim       = sampler.accept_dim

    ax_traj[0][i].plot(-np.log(accept_posterior))
    ax_traj[0][i].set_title(f"N={i+2} initial \n parameters")
    ax_traj[0][i].set_ylabel("-ln(posterior)")
    ax_traj[0][i].set_xlabel("step")

    ax_traj[1][i].plot(accept_dim)
    ax_traj[1][i].set_ylabel("N parameters")
    ax_traj[1][i].set_xlabel("step")
    ax_traj[1][i].set_yticks(list(range(1,11)))
    ax_traj[1][i].set_ylim(1,11)

    ax_traj[2][i].hist(accept_dim)
    ax_traj[2][i].set_ylabel("Counts(N Parameters)")
    ax_traj[2][i].set_xlabel("N parameters")
    ax_traj[2][i].set_xticks(list(range(1,11)))
    ax_traj[2][i].set_xlim(1,11)

fig_traj.tight_layout()
fig_traj.show()
```


### Gradient Descent and FISTA

$\newcommand\measurementvec{\mathbf{b}}
\newcommand\measurementmtx{\mathbf{A}}
\newcommand\imagevec{\mathbf{v}}
\newcommand\psf{\mathbf{h}}
\newcommand{\crop}{\mathbf{C}}
\newcommand\full{\mathbf{A}}
\newcommand{\ftpsf}{\mathbf{H}}$
Gradient descent is an iterative algorithm that finds the minimum of a convex function by following the slope "downhill" until it reaches a minimum. To solve the minimization problem
\begin{equation*}
    \operatorname{minimize} g(\mathbf{x}),
\end{equation*}
we find the gradient of $g$ wrt $\mathbf{x}$, $\nabla_\mathbf{x} g$, and use the property that the gradient always points in the direction of steepest _ascent_. In order to minimize $g$, we go the other direction:
$$\begin{align*}
    \mathbf{x}_0 &= \text{ initial guess} \\
    \mathbf{x}_{k+1} &\leftarrow \mathbf{x}_k - \alpha_k \nabla g(\mathbf{x}_k),
\end{align*}$$
where $\alpha$ is a step size that determines how far in the descent direction we go at each iteration.

Applied to our problem:
$$\begin{align*}
    g(\imagevec) &= \frac{1}{2} \|\full\imagevec- \measurementvec \|_2^2 \\
    \nabla_\imagevec g(\imagevec) &= \full^H (\full\imagevec-\measurementvec),
\end{align*}$$
where $\full^H$ is the adjoint of $\full$, $\measurementvec$ is the sensor measurement and $\imagevec$ is the image of the scene.

We use more efficient variants of this algorithm, like Nesterov Momentum and FISTA, both of which are shown below.

#### Loading and preparing our images

```python
import numpy as np
import numpy.fft as fft
import matplotlib.pyplot as plt
from IPython import display
from PIL import Image

%matplotlib inline
```

The code takes in two grayscale images: a point spread function (PSF) $\texttt{psfname}$ and a sensor measurement $\texttt{imgname}$. The images can be downsampled by a factor $f$, which must be a of the form $1/{2^k}$, for some non negative integer $k$ (typically between 1/2 and 1/8).

```python
psfname = "./psf_sample.tif"
imgname = "./rawdata_hand_sample.tif"

# Downsampling factor (used to shrink images)
f = 1/8

# Number of iterations
iters = 100
```

```python
def loaddata(show_im=True):
    psf = Image.open(psfname)
    psf = np.array(psf, dtype='float32')
    data = Image.open(imgname)
    data = np.array(data, dtype='float32')

    """In the picamera, there is a non-trivial background
    (even in the dark) that must be subtracted"""
    bg = np.mean(psf[5:15,5:15])
    psf -= bg
    data -= bg

    """Resize to a more manageable size to do reconstruction on.
    Because resizing is downsampling, it is subject to aliasing
    (artifacts produced by the periodic nature of sampling). Demosaicing is an attempt
    to account for/reduce the aliasing caused. In this application, we do the simplest
    possible demosaicing algorithm: smoothing/blurring the image with a box filter"""

    def resize(img, factor):
        num = int(-np.log2(factor))
        for i in range(num):
            img = 0.25*(img[::2,::2,...]+img[1::2,::2,...]+img[::2,1::2,...]+img[1::2,1::2,...])
        return img

    psf = resize(psf, f)
    data = resize(data, f)

    """Now we normalize the images so they have the same total power. Technically not a
    necessary step, but the optimal hyperparameters are a function of the total power in
    the PSF (among other things), so it makes sense to standardize it"""

    psf /= np.linalg.norm(psf.ravel())
    data /= np.linalg.norm(data.ravel())

    if show_im:
        fig1 = plt.figure()
        plt.imshow(psf, cmap='gray')
        plt.title('PSF')
        display.display(fig1)
        fig2 = plt.figure()
        plt.imshow(data, cmap='gray')
        plt.title('Raw data')
        display.display(fig2)
    return psf, data

```

### Calculating convolutions using $\texttt{fft}$
We want to calculate convolutions efficiently. To do this, we use the "fast fourier transform" $\texttt{fft2}$ which computes the Discrete Fourier Transform (DFT). The convolution theorem for DFTs only holds for circular convolutions. We can still recover a linear convolution by first padding the input images then cropping the output of the inverse DFT:
\begin{equation}
h*x=\mathcal{F}^{-1}[\mathcal{F}[h]\cdot\mathcal{F}[x]] = \texttt{crop}\left[\ \texttt{DFT}^{-1}\left\{\ \texttt{DFT} [\ \texttt{pad}[h]\ ]\cdot\texttt{DFT}[\ \texttt{pad}[x]\ ]\ \right\} \ \right]
\end{equation}

Recovering the linear convolution correctly requires that we double the dimensions of our images. To take full advantage of the speed of the $\texttt{fft2}$ algorithm, we actually pad $\texttt{full_size}$, which is the nearest power of two that is larger than that size.

We have chosen $\texttt{full_size}$ in such a way that it provides enough padding to make circular and linear convolutions look the same after being cropped back down to $\texttt{sensor_size}$. That way, the "sensor crop" due to the sensor's finite size and the "fft crop" above are the same, and we just need one crop function.

Along with initialization, we compute $\texttt{H} = \texttt{fft2}(\texttt{hpad})$ and $\texttt{Hadj} = \texttt{H}^*$, which are constant matrices that will be needed to calculate the action of $\measurementmtx$ and $\measurementmtx^H$ at every iteration.

Lastly, we must take into account one more practical difference. In imaging, we often treat the center of the image as the origin of the coordinate system. This is theoretically convenient, but fft algorithms assume the origin of the image is the top left pixel. The magnitude of the fft doesn't change because of this distinction, but the phase does, since it is sensitive to shifts in real space. An example with the simplest function, a delta function, is displayed below. In order to correct this problem, we use $\texttt{ifftshift}$ to move the origin of an image to the top left corner and $\texttt{fftshift}$ to move the origin from the top left corner to the center.

```python
def no_shift():
    delta = np.zeros((5,5))
    delta[2][2] = 1
    fft_mag = np.abs(fft.fft2(delta))
    fft_arg = np.angle(fft.fft2(delta))

    fig, ax = plt.subplots(nrows=1, ncols=3)
    fig.tight_layout()
    ax[0].imshow(delta, cmap='gray')
    ax[0].set_title('Delta function in \n real space')

    ax[1].imshow(fft_mag,vmin=-3,vmax=3,cmap='gray')
    ax[1].set_title('Magnitude of FT of \n a delta function')

    ax[2].imshow(fft_arg,vmin=-3,vmax=3,cmap='gray')
    ax[2].set_title('Phase of FT of \n delta function')

no_shift()

def shift():
    delta = np.zeros((5,5))
    delta[2][2] = 1
    delta_shifted = fft.ifftshift(delta)
    fft_mag = np.abs(fft.fft2(delta_shifted))
    fft_arg = np.angle(fft.fft2(delta_shifted))

    fig2, ax2 = plt.subplots(nrows=1, ncols=3)
    fig2.tight_layout()
    ax2[0].imshow(delta_shifted, cmap='gray')
    ax2[0].set_title('Delta function shifted in \n real space')

    ax2[1].imshow(fft_mag,vmin=-3,vmax=3,cmap='gray')
    ax2[1].set_title('Magnitude of FT of a \n shifted delta function')

    ax2[2].imshow(fft_arg,vmin=-3,vmax=3,cmap='gray')
    ax2[2].set_title('Phase of FT of a \n shifted delta function')

shift()
```

For this notebook and the ADMM notebook, we follow the following convention so we don't have to worry about this issue again:
1. All images in _real_ space are stored with the origin in the center (so they can be displayed correctly)
2. All images in _Fourier_ space are stored with the origin in the top left corner (so they can be used for processing correctly)
3. The above rules mean that, to perform a convolution between two real space images $h$ and $x$, we do $$\texttt{fftshift}( \texttt{ifft} [\texttt{fft}[ \texttt{ifftshift}(h) \cdot \texttt{ifftshift}(x) ] ] )$$ instead of $$\texttt{ifft}[\texttt{fft}[h \cdot x]]$$
The rules imply that if we store the fourier transform of $h$ for future use, instead of storing $\texttt{fft}[h]$, we store $\texttt{fft}[\texttt{ifftshift}(h)]$.

```python
def initMatrices(h):
    pixel_start = (np.max(h) + np.min(h))/2
    x = np.ones(h.shape)*pixel_start

    init_shape = h.shape
    padded_shape = [nextPow2(2*n - 1) for n in init_shape]
    starti = (padded_shape[0]- init_shape[0])//2
    endi = starti + init_shape[0]
    startj = (padded_shape[1]//2) - (init_shape[1]//2)
    endj = startj + init_shape[1]
    hpad = np.zeros(padded_shape)
    hpad[starti:endi, startj:endj] = h

    H = fft.fft2(fft.ifftshift(hpad), norm="ortho")
    Hadj = np.conj(H)

    def crop(X):
        return X[starti:endi, startj:endj]

    def pad(v):
        vpad = np.zeros(padded_shape).astype(np.complex64)
        vpad[starti:endi, startj:endj] = v
        return vpad

    utils = [crop, pad]
    v = np.real(pad(x))

    return H, Hadj, v, utils

def nextPow2(n):
    return int(2**np.ceil(np.log2(n)))
```

#### Computing the gradient

The most important step in Gradient Descent is calculating the gradient
$$ \nabla_\imagevec \  g(\imagevec) = \full^H (\full\imagevec-\measurementvec)$$
We do this in 2 steps:
1. We compute the action of $\full$ on $\imagevec$, using $\texttt{calcA}$
2. We compute the action of $\full^H$ on $\texttt{diff} = \texttt{Av-b}$ using $\texttt{calcAHerm}$

Here, $\texttt{vk}$ is the current padded estimate of the scene and $\texttt{b}$ is the sensor measurement.

```python
def grad(Hadj, H, vk, b, crop, pad):
    Av = calcA(H, vk, crop)
    diff = Av - b
    return np.real(calcAHerm(Hadj, diff, pad))
```

We write $\full$ as:
$$ \full\imagevec \iff \mathrm{crop} \left[ \mathcal{F}^{-1} \left\{\mathcal{F}(h) \cdot \mathcal{F}(v)\right\} \right]$$
In code, this becomes
\begin{align*}
\texttt{calcA}(\texttt{vk}) & = \texttt{crop}\ (\texttt{ifft}\ (\texttt{fft}(\texttt{hpad}) \cdot \texttt{fft}(\texttt{vk})\ )\  )\\
& = \texttt{crop}\ (\texttt{ifft}\ (\texttt{H} \cdot \texttt{Vk}))
\end{align*}
where $\cdot$ represents point-wise multiplication

```python
def calcA(H, vk, crop):
    Vk = fft.fft2(fft.ifftshift(vk))
    return crop(fft.fftshift(fft.ifft2(H*Vk)))
```

We first pad $\texttt{diff}$ , giving us $\texttt{xpad}$, then we take the 2D fourier transform, $\texttt{X} = \mathcal{F}(\texttt{xpad})$. The action of the adjoint of $A$ is

$$ A^H \mathbf{x} \iff \mathcal{F}^{-1} \left\{ \mathcal{F}(\psf)^* \cdot \mathcal{F}( \operatorname{pad}\left[x\right]) \right\}$$
This becomes
\begin{align*}
\texttt{calcAHerm}(\texttt{xk}) &= \texttt{ifft}\ (\ (\texttt{fft}(\texttt{h}))^H \cdot \texttt{fft}\ (\texttt{pad}(\texttt{diff}))\ ) \\
& = \texttt{ifft}\ (\texttt{Hadj} \cdot \texttt{X})
\end{align*}

```python
def calcAHerm(Hadj, diff, pad):
    xpad = pad(diff)
    X = fft.fft2(fft.ifftshift(xpad))
    return fft.fftshift(fft.ifft2(Hadj*X))
```

#### Putting it all together

This is the main function, which calculates the gradients and updates our estimation of the scene:
\begin{align*}
    &\imagevec_0 = \text{ anything} \\
    &\text{for } k = 0 \text{ to num_iters:}\\
    &\quad \quad \imagevec_{k+1} \leftarrow \texttt{gradient_update}(\imagevec_k) \\
\end{align*}

There are different ways of doing the gradient update. The three we will show are regular GD, Nesterov momentum update, and FISTA.

To guarantee convergence, we set the step size to be
\begin{align*}
\alpha_k &< \frac{2}{\|\measurementmtx^H \measurementmtx\|}_2 \approx \frac{2}{\lambda_{max}(\mathbf{M}^H\mathbf{M})}
\end{align*}

To calculate this, we use the property that $\mathbf{M}$ is diagonalizable by a Fourier Transform:
$$\begin{align*}
\mathbf{M}^H\mathbf{M} &= \left(\mathbf{F}^{-1} \mathrm{diag}(\mathbf{Fh}) \ \mathbf{F}\right)^H \ \mathbf{F}^{-1} \mathrm{diag}(\mathbf{Fh}) \ \mathbf{F} \\
&= \mathbf{F}^{-1} \mathrm{diag}(\mathbf{Fh})^* \ \mathrm{diag}(\mathbf{Fh}) \ \mathbf{F} \\
\lambda_{max}(\mathbf{M}^H\mathbf{M}) &= \max \left(\mathrm{diag}(\mathbf{Fh})^* \ \mathrm{diag}(\mathbf{Fh}) \right)
\end{align*}$$
In code, we have
\begin{align*}
\alpha = \frac{1.8}{\texttt{max} \left(\texttt{Hadj} \cdot \texttt{H}\right)}
\end{align*}

Since we are dealing with images, one constraint on the reconstructed image $\imagevec_k$ is that all the entries have to be non-negative. We do this by doing projected gradient descent. The projection function $\texttt{proj}$ we use is non-negativity, which  projects $\texttt{vk}$ onto the non-negative halfspace.

```python
def grad_descent(h, b):
    H, Hadj, v, utils = initMatrices(h)
    crop = utils[0]
    pad = utils[1]

    alpha = np.real(1.8/(np.max(Hadj * H)))
    iterations = 0

    def non_neg(xi):
        xi = np.maximum(xi,0)
        return xi

    #proj = lambda x:x #Do no projection
    proj = non_neg #Enforce nonnegativity at every gradient step. Comment out as needed.

    parent_var = [H, Hadj, b, crop, pad, alpha, proj]

    vk = v

    #### uncomment for Nesterov momentum update ####
    #p = 0
    #mu = 0.9
    ################################################

    #### uncomment for FISTA update ################
    tk = 1
    xk = v
    ################################################

    for iterations in range(iters):

        # uncomment for regular GD update
        #vk = gd_update(vk, parent_var)

        # uncomment for Nesterov momentum update
        #vk, p = nesterov_update(vk, p, mu, parent_var)

        # uncomment for FISTA update
        vk, tk, xk = fista_update(vk, tk, xk, parent_var)

        if iterations % 10 == 0:
            image = proj(crop(vk))
            f = plt.figure(1)
            plt.imshow(image, cmap='gray')
            plt.title('Reconstruction after iteration {}'.format(iterations))
            display.display(f)
            display.clear_output(wait=True)

    return proj(crop(vk))

```

#### Gradient descent algorithms

##### Regular Gradient Descent
Regular gradient descent is simply following the negative of the gradient until we reach the minimum:

\begin{align*}
    & \texttt{gradient_update}(\imagevec_k): \\
    &\quad \quad\imagevec'_{k+1} \leftarrow \imagevec_k - \alpha_k \full^H(\full\imagevec_k - \measurementvec) \\
    &\quad \quad \imagevec_{k+1} \leftarrow \operatorname{proj}_{\imagevec \geq 0} (\imagevec_{k+1}')
\end{align*}

```python
def gd_update(vk, parent_var):
    H, Hadj, b, crop, pad, alpha, proj = parent_var

    gradient = grad(Hadj, H, vk, b, crop, pad)
    vk -= alpha*gradient
    vk = proj(vk)

    return xk
```

##### Nesterov Momentum
GD works but it's slow. One way to speed it up is to consider a velocity term, $\mathbf{p}$. Each update becomes:

\begin{align*}
    &\texttt{gradient_update}(\imagevec_k): \\
    &\qquad \mathbf{p}_{k+1} \leftarrow \mu \mathbf{p}_k - \alpha_k \texttt{grad}(\imagevec_k) \\
    &\qquad \imagevec_{k+1}' \leftarrow \imagevec_k - \mu \mathbf{p}_{k} + (1+\mu)\mathbf{p}_{k+1} \\
    &\qquad \imagevec_{k+1} \leftarrow \operatorname{proj}_{\imagevec \geq 0} (\imagevec_{k+1}')
\end{align*}

The parameter $\mu$ is called the momentum and is strictly between 0 and 1.

```python
def nesterov_update(vk, p, mu, parent_var):
    H, Hadj, b, crop, pad, alpha, proj = parent_var

    p_prev = p
    gradient = grad(Hadj, H, vk, b, crop, pad)
    p = mu*p - alpha*gradient
    vk += -mu*p_prev + (1+mu)*p
    vk = proj(vk)

    return vk, p
```

##### FISTA
Instead of Nesterov momentum, we can use FISTA, which speeds up the iterative process. Each iteration of the algorithm is as follows:

\begin{align*}
    &\texttt{gradient_update}(\imagevec_k):\\
    &\qquad \imagevec_k \leftarrow \imagevec_k - \alpha_k \texttt{grad}(\imagevec_k) \\
    &\qquad x_{k} \leftarrow \texttt{proj}(\imagevec_k) \\
    &\qquad t_{k+1} \leftarrow \frac{1+\sqrt{1+4t_k^2}}{2} \\
    &\qquad \imagevec_{k+1} \leftarrow x_{k} + \frac{t_k-1}{t_{k+1}} (x_{k}-x_{k-1}) \\
\end{align*}

```python
def fista_update(vk, tk, xk, parent_var):
    H, Hadj, b, crop, pad, alpha, proj = parent_var

    x_k1 = xk
    gradient = grad(Hadj, H, vk, b, crop, pad)
    vk -= alpha*gradient
    xk = proj(vk)
    t_k1 = (1+np.sqrt(1+4*tk**2))/2
    vk = xk+(tk-1)/t_k1*(xk - x_k1)
    tk = t_k1

    return vk, tk, xk
```

#### Running the algorithm

```python
psf, data = loaddata()
final_im = grad_descent(psf, data)
plt.imshow(final_im, cmap='gray')
plt.title('Final reconstruction after {} iterations'.format(iters))
display.display()

```