## Subsurface Data Analytics

### Bootstrap for Subsurface Data Analytics in Python

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### Exercise: Bootstrap for Subsurface Data Analytics in Python

Here's a simple workflow, demonstration of bootstrap for subsurface modeling workflows. This should help you get started with building subsurface models that integrate uncertainty in the sample statistics.

#### Bootstrap

Uncertainty in the sample statistics
* one source of uncertainty is the paucity of data.
* do 200 or even less wells provide a precise (and accurate estimate) of the mean? standard deviation? skew? P13?

Would it be useful to know the uncertainty in these statistics due to limited sampling?
* what is the impact of uncertainty in the mean porosity e.g. 20%+/-2%?

**Bootstrap** is a method to assess the uncertainty in a sample statistic by repeated random sampling with replacement.

Assumptions
* sufficient, representative sampling, identical, idependent samples

Limitations
1. assumes the samples are representative
2. assumes stationarity
3. only accounts for uncertainty due to too few samples, e.g. no uncertainty due to changes away from data
4. does not account for boundary of area of interest
5. assumes the samples are independent
6. does not account for other local information sources

The **Bootstrap Approach** (Efron, 1982)

Statistical resampling procedure to calculate uncertainty in a calculated statistic from the data itself.
* Does this work?  Prove it to yourself, for uncertainty in the mean solution is standard error:

\begin{equation}
\sigma^2_\overline{x} = \frac{\sigma^2_s}{n}
\end{equation}

Extremely powerful - could calculate uncertainty in any statistic!  e.g. P13, skew etc.
* Would not be possible access general uncertainty in any statistic without bootstrap.
* Advanced forms account for spatial information and sampling strategy (game theory and Journel’s spatial bootstrap (1993).

Steps:

1. assemble a sample set, must be representative, reasonable to assume independence between samples

2. optional: build a cumulative distribution function (CDF)
    * may account for declustering weights, tail extrapolation
    * could use analogous data to support

3. For $\ell = 1, \ldots, L$ realizations, do the following:

    * For $i = \alpha, \ldots, n$ data, do the following:

        * Draw a random sample with replacement from the sample set or Monte Carlo simulate from the CDF (if available).

6. Calculate a realization of the sammary statistic of interest from the $n$ samples, e.g. $m^\ell$, $\sigma^2_{\ell}$. Return to 3 for another realization.

7. Compile and summarize the $L$ realizations of the statistic of interest.

This is a very powerful method.  Let's try it out.

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data_biased.csv at https://git.io/fh0CW

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

```python
import geostatspy.GSLIB as GSLIB          # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats    # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import numpy as np                        # ndarrys for gridded data
import pandas as pd                       # DataFrames for tabular data
import os                                 # set working directory, run executables
import matplotlib.pyplot as plt           # for plotting
import seaborn as sns                     # advanced statistical methods and plots
from scipy import stats                   # summary statistics
import math                               # trig etc.
import scipy.signal as signal             # kernel for moving window calculation
import random
```

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).

```python
os.chdir("c:/PGE383")                     # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv('sample_data_biased.csv') # load our data table (wrong name!)
```

Let's drop some samples so that we increase the variations in bootstrap samples for our demonstration below.

* We will resample 25% of the original data file.

* I use a random number seed so that we all get the same sample to work with.

```python
df.sample(frac = 0.25, random_state = 73073)
```

where *frac* is the fraction to select and *random_state* is the random number seed.

To get different random 25% subset of the data, just drop the 'rand_state' parameter or change the value from '73073' to any other integer.

```python
df = df.sample(frac = 0.25, random_state = 73073) # extract 25% random samples to reduce the size of the dataset
print('Using ' + str(len(df)) + ' number of samples')
```

    Using 72 number of samples

Visualizing the DataFrame would be useful and we already learned about these methods in this demo (https://git.io/fNgRW).

We can preview the DataFrame by printing a slice or by utilizing the 'head' DataFrame member function (with a nice and clean format, see below). With the slice we could look at any subset of the data table and with the head command, add parameter 'n=13' to see the first 13 rows of the dataset.

```python
print(df.iloc[0:5,:])                   # display first 4 samples in the table as a preview
df.head(n=13)                           # we could also use this command for a table preview
```

           X    Y  Facies  Porosity         Perm
    150   20   69       0  0.104484     3.003225
    17   400  600       1  0.163054   317.883581
    68   640  529       1  0.120547    10.801778
    234  490  289       1  0.144525    16.251107
    214  470  899       1  0.211944  1302.937858

      X
      Y
      Facies
      Porosity
      Perm

      150
      20
      69
      0
      0.104484
      3.003225

      17
      400
      600
      1
      0.163054
      317.883581

      68
      640
      529
      1
      0.120547
      10.801778

      234
      490
      289
      1
      0.144525
      16.251107

      214
      470
      899
      1
      0.211944
      1302.937858

      153
      360
      279
      1
      0.134607
      9.776276

      5
      200
      800
      1
      0.154648
      106.491795

      170
      670
      889
      1
      0.119601
      17.052309

      252
      200
      359
      0
      0.077625
      0.540486

      131
      320
      99
      1
      0.141749
      13.154686

      133
      250
      569
      1
      0.120868
      22.277121

      245
      890
      489
      0
      0.103176
      2.846036

      159
      590
      69
      1
      0.114117
      13.251200

#### Summary Statistics for Tabular Data

The table includes X and Y coordinates (meters), Facies 1 and 0 (1 is sandstone and 0 interbedded sand and mudstone), Porosity (fraction), and permeability as Perm (mDarcy).

There are a lot of efficient methods to calculate summary statistics from tabular data in DataFrames. The describe command provides count, mean, minimum, maximum, and quartiles all in a nice data table. We use transpose just to flip the table so that features are on the rows and the statistics are on the columns.

```python
df.describe().transpose()
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      72.0
      448.611111
      239.825674
      0.000000
      295.000000
      415.000000
      640.000000
      990.000000

      Y
      72.0
      495.111111
      293.106026
      39.000000
      279.000000
      494.000000
      716.750000
      999.000000

      Facies
      72.0
      0.833333
      0.375293
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      72.0
      0.135360
      0.039304
      0.074349
      0.106374
      0.124065
      0.147035
      0.223661

      Perm
      72.0
      242.573614
      553.551806
      0.093252
      2.979039
      13.509371
      51.052217
      2372.383732

#### Visualizing Tabular Data with Location Maps

It is natural to set the x and y coordinate and feature ranges manually. e.g. do you want your color bar to go from 0.05887 to 0.24230 exactly? Also, let's pick a color map for display.

* I heard that plasma is known to be friendly to the color blind as the color and intensity vary together (hope I got that right, it was an interesting Twitter conversation started by Matt Hall from Agile if I recall correctly).

We will assume a study area of 0 to 1,000m in x and y and omit any data outside this area.

```python
xmin = 0.0; xmax = 1000.0               # range of x values
ymin = 0.0; ymax = 1000.0               # range of y values
pormin = 0.05; pormax = 0.25;           # range of porosity values
nx = 100; ny = 100; csize = 10.0
cmap = plt.cm.plasma                    # color map
```

Let's try out locmap. This is a reimplementation of GSLIB's locmap program that uses matplotlib. I hope you find it simpler than matplotlib.

* If you want to get more advanced and build custom plots lock at the source.

If you improve it, send me the new code. Any help is appreciated.

* To see the parameters, just type the command name:

```python
GSLIB.locmap
```

    <function geostatspy.GSLIB.locmap(df, xcol, ycol, vcol, xmin, xmax, ymin, ymax, vmin, vmax, title, xlabel, ylabel, vlabel, cmap, fig_name)>

Now we can populate the plotting parameters and visualize the porosity data.

```python
plt.subplot(111)
GSLIB.locmap_st(df,'X','Y','Porosity',xmin,xmax,ymin,ymax,pormin,pormax,'Well Data - Porosity','X(m)','Y(m)','Porosity (fraction)',cmap)
plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Debiasing the Spatial Dataset

We must mitigate bias in the spatial sampling.

* If the samples are biased, then any resulting uncertainty model in a statistic may be biased.

* Cell-based declustering is a common method for debiasing spatial data.

Let's calculate some declustering weights. There is a demonstration on declustering here https://git.io/fhgJl if you need more information.

* I have coded cell based declustering from GSLIB in the GeostatsPy package.

```python
wts, cell_sizes, dmeans = geostats.declus(df,'X','Y','Porosity',iminmax = 1, noff= 10, ncell=100,cmin=10,cmax=2000)
df['Wts'] = wts                           # add weights to the sample data DataFrame
df.head()                                 # preview to check the sample data DataFrame

def weighted_avg_and_std(values, weights): # function to calculate weighted mean and st. dev., from Eric O Lebigot, stack overflow,
    average = np.average(values, weights=weights)
    variance = np.average((values-average)**2, weights=weights)
    return (average, math.sqrt(variance))

sample_avg, sample_stdev = weighted_avg_and_std(df['Porosity'],df['Wts'])
print('Declustered mean = ' + str(round(sample_avg,3)) + ' and declustered standard deviation = ' + str(round(sample_stdev,3)))
```

    There are 72 data with:
       mean of      0.13535995545833332
       min and max  0.074348923 and 0.22366070899999999
       standard dev 0.03902973923480811
    Declustered mean = 0.126 and declustered standard deviation = 0.032

##### A Couple of Bootstrap Realizations

We will attempt boostrap by-hand and manually loop over $L$ realizations and draw $n$ samples to calculate the summary statistics of interest, mean and variance. The choice function from the random package simplifies sampling with replacement from a set of samples with weights.

This command returns a ndarray with k samples with replacment from the 'Porosity' column of our DataFrame (df) accounting for the data weights in column 'Wts'.
```p
samples1 = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))
```

It is instructive to look at a few of these realizations from the original declustered data set.

**CodingTips** - To understand this code below there are just a couple of Python concepts that you need to add to your Python arsenal.

1. declaring arrays - NumPy has a lot of great array (ndarray) functionality. There are build in functions to make a ndarray of any length (and dimension). This includes 'zeros', 'ones' and 'rand', so when we use this code:

 `samples = np.zeros((nreal,len(df))`

 we're making a 2D array of size number of realizations, nreal, x number of data in the DataFrame, df, pre-populated with zeros.  Note the len(df) command returns the number of samples in the DataFrame.

2. For Loops - when we are using the command below, we are instructing the computer to loop over all the indented code below the command for $ireal = 0,1,2,\ldots,nreal-1$ times. For each loop the $ireal$ variable increments, so we can use this to save each result to a different index in the arrays mean and stdev. Note, Python arrays index starting at 0 and stop at the length - 1.

 `for ireal in range(0,nreal):`

 we are running each bootstrap resampled realization, and storing each realization in a row of the 2D array (for a 2D array the first index is the row index and the second index is the column index).

3. Subplots - we can put multple plots together with subplots.  To do this we specify

 `plt.subplot(3,3,1)`

 before each plot.  Where the first value is the number of row, the second number is the number of columns and the third number is the image index from $0,...,nrow x ncol$.  The images sort from left to right and then from top to bottom.

```python
nreal = 1500                               # number of bootstrap realizations
nhist = 8                                 # number of bootstrap histograms to view (max 8)
samples = np.zeros((nreal,len(df))); real_label = []

for ireal in range(0,nreal):
    samples[ireal] = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))

plt.subplot(3,3,1)
GSLIB.hist_st(df['Porosity'],pormin,pormax,False,False,20,df['Wts'],'Porosity (fraction)','Declustered Porosity')

for ireal in range(0,nhist):
    plt.subplot(3,3,ireal+2)
    GSLIB.hist_st(samples[ireal],pormin,pormax,False,False,20,None,'Porosity (fraction)','Bootstrap Porosity Realization ' + str(ireal+1))

plt.subplots_adjust(left=0.0, bottom=0.0, right=3.0, top=3.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Distribution Shape

The distribution seems to be bimodel (two peaks).

* Instead of looking at a few histograms of bootstrap realizations, let's look at many realizations and see if there is always, mostly or just sometimes two modes.

We will visualize many bootstrap realizations of the entire distribution.

This code loops over the realizations and adds each to a probability density plot:

```python
for ireal in range(0,nreal):
    sns.distplot(samples[ireal])
```

where each bootstrap realization is a row in the 2D ndarray samples.

The additional prameters control the visualization:

```python
color = 'black', hist = False, kde = True,kde_kws = {"alpha": 0.02, 'linewidth': 5}
```
for example:

* **kde** is a kernel density estimator to conver the data to a smooth density estimate

* **alpha** is the level of transparency

See what you think.  From the image below can you see the uncertainty in the distribution?

```python
for ireal in range(0,nreal):              # loop over the realizations and plot with seaborn package
    # Draw the density plot
    sns.distplot(samples[ireal], color = 'black', hist = False, kde = True,kde_kws = {"alpha": 0.002, 'linewidth': 5})
plt.title('Bootstrap Porosity Distribution - Uncertainty in the Distribution Shape')
plt.xlabel('Porosity (fraction)')
plt.ylabel('Density')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

Note that the bootstrap distributions vary quite a bit from the original, but the bimodal shape is consistent; therefore the bimodal distribution form is **not due to random effect / sparse sampling**! We are likely dealing with two distinct processes and / or spatial trends.

#### Summarizations Over Bootstrap Realizations

Let's make a loop to conduct $L$ resamples and calculate the average and standard deviation for each  ($m^\ell$, $\sigma^2_{\ell}$,  for $\ell = 0,\dots,L-1$). We then summarization over these $L$ realizations.

I did not find any built-in, concise functions to accomplish this, i.e. with a single line of code, so we are going to do it by hand.

To understand this code there are just a couple of Python concepts that you need to add to your Python arsenal.

1. declaring arrays - NumPy has a lot of great array (ndarray) functionality. There are build in functions to make a ndarray of any length (and dimension). This includes 'zeros', 'ones' and 'rand', so when we use this code:

 `mean = np.zeros(L); stdev = np.zeros(L)`

 we're making arrays of length $L$ pre-populated with zeros.

2. For Loops - when we are using the command below, we are instructing the computer to loop over all the indented code below the command for $l = 0,1,2,\ldots,L-1$ times. For each loop the $l$ variable increments, so we can use this to save each result to a different index in the arrays mean and stdev. Note, Python arrays index starting at 0 and stop at the length - 1.

 `for l in range(0, L):`

 we are running each bootstrap resampled realization, calculating the average and standard deviation and storing them in the arrays that we already declared.

```python
L = 20000                                  # set the number of realizations for uncertainty calculation
mean = np.zeros(L); stdev = np.zeros(L)    # declare arrays to hold the realizations of the statistics
P10 = np.zeros(L)
P50 = np.zeros(L); P90 = np.zeros(L)
for l in range(0, L):                      # loop over realizations
    samples = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))
    mean[l] = np.average(samples)
    stdev[l] = np.std(samples)

plt.subplot(121)
GSLIB.hist_st(mean,0.11,0.15,False,False,50,None,'Average Porosity (fraction)','Bootstrap Uncertainty in Porosity Average')

plt.subplot(122)
GSLIB.hist_st(stdev,0.015,0.045,False,False,50,None,'Standard Deviation Porosity (fraction)','Bootstrap Uncertainty in Porosity Standard Deviation')

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('Summary Statistics for Bootstrap Porosity Average:')
print(stats.describe(mean))
print('\nUncertainty in Porosity Average: P10 ' + str(round(np.percentile(mean,10)*100,1)) + '%, P50 ' + str(round(np.percentile(mean,50)*100,1)) + '%, P90 ' + str(round(np.percentile(mean,90)*100,1)) +"%")

print('\nSummary Statistics for Bootstrap Porosity Standard Deviation:')
print(stats.describe(stdev));
print('\nUncertainty in Porosity Standard Deviation: P10 ' + str(round(np.percentile(stdev,10),3)*100) + '%, P50 ' + str(round(np.percentile(stdev,50),3)*100) + '%, P90 ' + str(round(np.percentile(stdev,90)*100,1)) + "%")

```

#### Uncertainty in the Percentiles

We could even calculate the uncertainty in the percentiles, well-based P10, P50 and P90.

* this demonstrates the flexiblity of the bootstrap method

```python
L = 20000                                  # set the number of realizations for uncertainty calculation
mean = np.zeros(L); stdev = np.zeros(L)    # declare arrays to hold the realizations of the statistics
P10 = np.zeros(L)
P50 = np.zeros(L); P90 = np.zeros(L)
for l in range(0, L):                      # loop over realizations
    samples = random.choices(df['Porosity'].values, weights=df['Wts'].values, cum_weights=None, k=len(df))
    P10[l] = np.percentile(q = 10, a = samples)
    P50[l] = np.percentile(q = 50, a = samples)
    P90[l] = np.percentile(q = 90, a = samples)

plt.subplot(111)
plt.hist(P10, facecolor='red',bins=np.linspace(0.05,0.25,50),alpha=0.2,density=False,edgecolor='black',label='P10')
plt.hist(P50, facecolor='blue',bins=np.linspace(0.05,0.25,50),alpha=0.2,density=False,edgecolor='black',label = 'P50')
plt.hist(P90, facecolor='green',bins=np.linspace(0.05,0.25,50),alpha=0.2,density=False,edgecolor='black',label = 'P90')
plt.xlim([0.05,0.25]); plt.ylim([0,10000.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Bootstrap Uncertainty in Porosity Well-based P10, P50 and P90 ')
plt.legend(loc='upper left')
plt.grid(True)
plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()
```

#### Comments

This was a basic demonstration of bootstrap. Much more could be done, you could replace the statistics, average and standard deviation with any other statistics, for example P90, kurtosis, P13 etc. I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations, trend modeling and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

```python

```


```python
import numpy as np
import matplotlib.pyplot as plt
import numba
import cmath
# plt.style.use('seaborn-pastel')
plt.rcParams['font.family'] = 'sans-serif'
COLOR = 'w'
plt.rcParams['text.color'] = COLOR
plt.rcParams['axes.labelcolor'] = COLOR
plt.rcParams['xtick.color'] = COLOR
plt.rcParams['ytick.color'] = COLOR
plt.rcParams['axes.edgecolor'] = 'k'
SIZE = 8
plt.rcParams['font.size'] = SIZE
plt.rcParams['legend.fontsize'] = SIZE
plt.rcParams['figure.titlesize'] = SIZE
plt.rcParams['figure.dpi'] = 300
plt.rcParams['savefig.dpi'] = 800
# plt.rcParams['image.cmap'] = 'coolwarm'
plt.rcParams['image.cmap'] = 'jet'

res = 200
MC_N = int(1e3)
```

# parameters

```python
Delta_ZPL_0 = 406.83 * 0.  # 406 THz  Meesala
lambda_SO_gs = 46e-3  # 46 GHz  Meesala
lambda_SO_es = 255e-3  # 255 GHz  Meesala

t_para = -1.7e3  # -1.7 PHz/strain  Meesala
t_perp = 0.078e3  # 0.078 PHz/strain   Meesala
d_gs = 1.3e3  # 1.3 PHz/strain   Meesala
d_es = 1.8e3  # 1.8 PHz/strain   Meesala
f_gs = -1.7e3  # -1.7 PHz/strain   Meesala
f_es = -3.4e3 # -3.4 PHz/strain   Meesala

sigma = 2.5e-4 # Kelsey
gamma_10 = 28e-3  # 28 GHz p.3 or 1.33 Ghz p.5 Smallwood PRL
# gamma_10 = 1e-4 # fwhm = 10e-3 Kelsey

poi = -0.2  # Poisson ratio of diamond, Kelsey

# omega_t = np.linspace(405.5, 408.75, res)
# omega_tau = np.linspace(-408.75, -405.5, res)

omega_t = np.linspace(-0.5, 0.5, res)
omega_tau = np.linspace(-0.5, 0.5, res)
```

# colorbar function

```python
# @numba.jit
def colorbar(mappable, label='arb. units'):
    from mpl_toolkits.axes_grid1 import make_axes_locatable
    last_axes = plt.gca()
    ax = mappable.axes
    fig = ax.figure
    divider = make_axes_locatable(ax)
    cax = divider.append_axes("right", size="5%", pad=0.05)
    cbar = fig.colorbar(mappable, cax=cax)
    if label is not None:
        plt.ylabel(label)
    plt.sca(last_axes)
    return cbar
```

# creating mesh

```python
omega_t_mesh, omega_tau_mesh = np.meshgrid(omega_t, omega_tau, indexing='xy')
```

# multiple plot function

```python
# @numba.jit
def slices_plots(
    databox,
    titles=None,
    figure_shape=[4, 4],
    figure_size=(11, 10),
    vertical_gap=0.15,
    horizontal_gap=0.15,
    shared_x_label=r'$\omega_t \quad$',
    shared_x_label_x_position=0.515,
    shared_x_label_y_position=0.09,
    shared_y_label=r'$\omega_\tau \quad$',
    shared_y_label_x_position=0.08,
    shared_y_label_y_position=0.5,
    shared_title=r'$\mathfrak{R} \left\{ \rho^{(3)}_{10,sig} (\omega_t, \omega_\tau) \right\}$',
    shared_title_x_position=0.515,
    shared_title_y_position=0.93,
    share_x_axis=True,
    share_y_axis=True,
    colorbar_labels='arb. units',
    EXTENT=(min(omega_t), max(omega_t), min(omega_tau), max(omega_tau))):

    fig, ax = plt.subplots(
        figure_shape[0],  # row
        figure_shape[1],  # column
        figsize=figure_size,
        sharex=share_x_axis,
        sharey=share_y_axis,
        gridspec_kw={
            "hspace": vertical_gap,
            'wspace': horizontal_gap
        })

    for i in range(np.shape(databox)[0]):
        im = ax.flatten()[i].imshow(databox[i],
                                    origin='lower',
                                    extent=EXTENT)
        ax.flatten()[i].set_title(titles[i])

        if np.size(colorbar_labels) == 1:
            if (i + 1) % figure_shape[1]:
                colorbar(im, None)
            else:
                colorbar(im, colorbar_labels)
        else:
            colorbar(im, colorbar_labels[i])

    fig.suptitle(x=shared_title_x_position,
                 y=shared_title_y_position,
                 t=shared_title,
                 ha='center',
                 size=1.5 * SIZE)

    if shared_x_label != None:  # shared x label
        fig.text(x=shared_x_label_x_position,
                 y=shared_x_label_y_position,
                 s=shared_x_label,
                 ha='center')

    if shared_y_label != None:  # shared y label
        fig.text(x=shared_y_label_x_position,
                 y=shared_y_label_y_position,
                 s=shared_y_label,
                 va='center',
                 rotation='vertical')

    plt.show()
    return fig
```

# complex square function

```python
@numba.vectorize(
    [numba.float64(numba.complex128),
     numba.float32(numba.complex64)])
def abs2(x):

    return x.real**2. + x.imag**2.
```

# complex phase

```python
@numba.vectorize(
    [numba.float64(numba.complex128),
     numba.float32(numba.complex64)])
def phase(x):

    return cmath.phase(x)
```

\begin{equation}\label{eq:1}
    \Delta_{ZPL}(\epsilon_{xx}, \epsilon_{yy}, \epsilon_{zz}) = \Delta_{ZPL,0} + (t_{\parallel,es} - t_{\parallel,gs})\epsilon_{zz} + (t_{\perp,es} - t_{\perp,gs})(\epsilon_{xx} + \epsilon_{yy})
\end{equation}

```python
@numba.jit
def Delta_ZPL(eps_xx=0.,
              eps_yy=0.,
              eps_zz=0.):

    return Delta_ZPL_0 + t_para*eps_zz + t_perp*(eps_xx + eps_yy)
```

\begin{equation}\label{eq:2}
    \Delta_{gs}(\epsilon_{xx}, \epsilon_{yy}, \epsilon_{yz}, \epsilon_{xy}, \epsilon_{zx}) = \sqrt{\lambda_{SO,gs}^2 + 4[d_{gs}(\epsilon_{xx} - \epsilon_{yy}) + f_{gs} \epsilon_{yz}]^2 + 4[-2d_{gs} \epsilon_{xy} + f_{gs} \epsilon_{zx}]^2}
\end{equation}

```python
@numba.jit
def Delta_gs(eps_xx=0.,
             eps_yy=0.,
             eps_yz=0.,
             eps_xy=0.,
             eps_zx=0.):

    return np.sqrt(
        np.sum([
            lambda_SO_gs**2.,
            4. * (d_gs * (eps_xx - eps_yy) + f_gs * eps_yz)**2.,
            4. * (-2. * d_gs * eps_xy + f_gs * eps_zx)**2.
        ]))
```

\begin{equation}\label{eq:3}
    \Delta_{es}(\epsilon_{xx}, \epsilon_{yy}, \epsilon_{yz}, \epsilon_{xy}, \epsilon_{zx}) = \sqrt{\lambda_{SO,es}^2 + 4[d_{es}(\epsilon_{xx} - \epsilon_{yy}) + f_{es} \epsilon_{yz}]^2 + 4[-2d_{es} \epsilon_{xy} + f_{es} \epsilon_{zx}]^2}
\end{equation}

```python
@numba.jit
def Delta_es(eps_xx=0.,
             eps_yy=0.,
             eps_yz=0.,
             eps_xy=0.,
             eps_zx=0.):

    return np.sqrt(
        np.sum([
            lambda_SO_es**2.,
            4. * (d_es * (eps_xx - eps_yy) + f_es * eps_yz)**2.,
            4. * (-2. * d_es * eps_xy + f_es * eps_zx)**2.
        ]))
```

\begin{equation}\label{eq:4}
    \rho (\omega_t, \omega_{\tau}, \epsilon_{xx}, \epsilon_{yy}, \epsilon_{zz}, \epsilon_{yz}, \epsilon_{xy}, \epsilon_{zx}) = \left( \frac{i \sqrt{A} \, \gamma_{10}}{\omega_t-[\Delta_{ZPL} \pm \Delta_{gs} \pm \Delta_{es} - i \gamma_{10}]} \right ) \left (\frac{i \sqrt{A} \, \gamma_{10}}{\omega_{\tau}+[\Delta_{ZPL} \pm \Delta_{gs} \pm \Delta_{es} + i \gamma_{10}]} \right)
\end{equation}

```python
@numba.jit
def rho(omega_t,
        omega_tau,
        eps_xx=0.,
        eps_yy=0.,
        eps_zz=0.,
        eps_yz=0.,
        eps_xy=0.,
        eps_zx=0.):

    Del_ZPL = Delta_ZPL(eps_xx, eps_yy, eps_zz)

    Del_gs = Delta_gs(eps_xx, eps_yy, eps_yz, eps_xy, eps_zx)

    Del_es = Delta_es(eps_xx, eps_yy, eps_yz, eps_xy, eps_zx)

    amp = np.array(
        [0.0938, 0.0833, 0.0579, 0.0576,
         0.0709, 1.0000, 0.0799, 0.1521,
         0.0631, 0.0837, 0.3036, 0.0737,
         0.0522, 0.1212, 0.0581, 0.1228])
#     amp = np.array(
#         [0., 1., 1., 1.,
#          1., 1., 1., 1.,
#          1., 1., 1., 1.,
#          1., 1., 1., 1.])

    sign = np.array([[ 1.,  1.,  1.,  1.],
                     [ 1.,  1.,  1., -1.],
                     [ 1.,  1., -1.,  1.],
                     [ 1.,  1., -1., -1.],
                     [ 1., -1.,  1.,  1.],
                     [ 1., -1.,  1., -1.],
                     [ 1., -1., -1.,  1.],
                     [ 1., -1., -1., -1.],
                     [-1.,  1.,  1.,  1.],
                     [-1.,  1.,  1., -1.],
                     [-1.,  1., -1.,  1.],
                     [-1.,  1., -1., -1.],
                     [-1., -1.,  1.,  1.],
                     [-1., -1.,  1., -1.],
                     [-1., -1., -1.,  1.],
                     [-1., -1., -1., -1.]])

    RHO = 1.j * np.zeros(np.shape(omega_t))

    for i in range(np.shape(sign)[0]):
        t_Lorentz = 1.j * np.sqrt(amp[i]) * gamma_10 / (
            omega_t - (Del_ZPL + sign[i, 0] * Del_gs + sign[i, 1] * Del_es -
                       1.j * gamma_10))
        tau_Lorentz = 1.j * np.sqrt(amp[i]) * gamma_10 / (
            omega_tau + (Del_ZPL + sign[i, 2] * Del_gs + sign[i, 3] * Del_es +
                         1.j * gamma_10))

        RHO += t_Lorentz * tau_Lorentz

    return RHO
```

```python
RHO = abs2(rho(omega_t_mesh, omega_tau_mesh))

plt.figure()
im = plt.imshow(RHO,
                origin='lower',
                extent=(min(omega_t),
                        max(omega_t),
                        min(omega_tau),
                        max(omega_tau)))
plt.xlabel(r'$\omega_t$')
plt.ylabel(r'$\omega_\tau$')
plt.title(r'$\omega_\tau$ mesh')
colorbar(im)
plt.show()

```


# Dynamic Models

### Preliminaries

- Goal
  - Introduction to dynamic (=temporal) Latent Variable Models, including the Hidden Markov Model and Kalman filter.
- Materials
  - Mandatory
    - These lecture notes
  - Optional
    - Bishop pp.605-615 on Hidden Markov Models
    - Bishop pp.635-641 on Kalman filters
    - Faragher (2012), [Understanding the Basis of the Kalman Filter](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Faragher-2012-Understanding-the-Basis-of-the-Kalman-Filter.pdf)
    - Minka (1999), [From Hidden Markov Models to Linear Dynamical Systems](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Minka-1999-from-HMM-to-LDS.pdf)

### Example Problem

- We consider a one-dimensional cart position tracking problem, see [Faragher 2012](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Faragher-2012-Understanding-the-Basis-of-the-Kalman-Filter.pdf).

- The hidden states are the position $z_t$ and velocity $\dot z_t$. We can apply an external acceleration/breaking force $u_t$. (Noisy) observations are represented by $x_t$.

- The equations of motions are given by

$$\begin{align*}
\begin{bmatrix} z_t \\ \dot{z_t}\end{bmatrix} &=  \begin{bmatrix} 1 & \Delta t \\ 0 & 1\end{bmatrix} \begin{bmatrix} z_{t-1} \\ \dot z_{t-1}\end{bmatrix} + \begin{bmatrix} (\Delta t)^2/2 \\ \Delta t\end{bmatrix} u_t + \mathcal{N}(0,\Sigma_z) \\
x_t &= \begin{bmatrix} z_t \\ \dot{z_t}\end{bmatrix} + \mathcal{N}(0,\Sigma_x)
\end{align*}$$

- Infer the position $z_t$ after 10 time steps. (Solution later in this lesson).

### Dynamical Models

- Consider the _ordered_ observation sequence $x^T \triangleq \left(x_1,x_2,\ldots,x_T\right)$.
  - (For brevity, we use the notation $x_t^T$ to denote $(x_t,x_{t+1},\ldots,x_T)$ and drop the subscript if $t=1$).

- We wish to develop a generative model
    $$ p( x^T \,|\, \theta)$$
that 'explains' the time series $x^T$.

- We cannot use the IID assumption $p( x^T  | \theta) = \prod_t p(x_t \,|\, \theta)$. In general, we _can_ use the [**chain rule**](https://en.wikipedia.org/wiki/Chain_rule_(probability)) (a.k.a. the general product rule)

$$\begin{align*}
p(x^T) &= p(x_T|x^{T-1}) \,p(x^{T-1}) \\
  &=  p(x_T|x^{T-1}) \,p(x_{T-1}|x^{T-2}) \cdots p(x_2|x_1)\,p(x_1) \\
  &= p(x_1)\prod_{t=2}^T p(x_t\,|\,x^{t-1})
\end{align*}$$

- Generally, we will want to limit the depth of dependencies on previous observations. For example, the $M$th-order linear **Auto-Regressive** (AR) model
    $$\begin{align*}
  p(x_t\,|\,x^{t-1}) = \mathcal{N}\left(x_t \,\middle|\,  \sum_{m=1}^M a_m x_{t-m}\,,\sigma^2\,\right)
    \end{align*}$$
    limits the dependencies to the past $M$ samples.

### State-space Models

- A limitation of AR models is that they need a lot of parameters in order to create a flexible model. E.g., if $x_t$ is an $K$-dimensional discrete variable, then an $M$th-order AR model will have about $K^{M}$ parameters.

- Similar to our work on Gaussian Mixture models, we can create a flexible dynamic system by introducing _latent_ (unobserved) variables  $z^T \triangleq \left(z_1,z_2,\dots,z_T\right)$ (one $z_t$ for each observation $x_t$). In dynamic systems, $z_t$ are called _state variables_.

- A general **state space model** is defined by
$$\begin{align*}
 p(x^T,z^T) &= \underbrace{p(z_1)}_{\text{initial state}} \prod_{t=2}^T \underbrace{p(z_t\,|\,z^{t-1})}_{\text{state transitions}}\,\prod_{t=1}^T \underbrace{p(x_t\,|\,z_t)}_{\text{observations}}
\end{align*}$$

- A common assumption is to let state transitions be ruled by a _first-order Markov chain_ as

$$
 p(z_t\,|\,z^{t-1}) = p(z_t\,|\,z_{t-1})
$$

- In a Markovian state-space model, the observation sequence $x^T$ is not a first-order Markov chain, i.e., for the state-space model
    $$\begin{align*}
 p(x^T,z^T) &= p(z_1) \prod_{t=2}^T p(z_t\,|\,z_{t-1})\,\prod_{t=1}^T p(x_t\,|\,z_t)
\end{align*}$$
the following statement holds:
    $$p(x_t\,|\,x_{t-1},x_{t-2}) \neq p(x_t\,|\,x_{t-1})\,.$$
In other words, the latent variables $z_t$ represent a memory bank for past observations beyond $t-1$. (Proof as exercise).

- The Forney-style factor graph for a state-space model:

### Hidden Markov Models and Linear Dynamical Systems

- A **Hidden Markov Model** (HMM) is a specific state-space model with discrete-valued state variables $Z_t$.

- E.g., $Z_t$ is a $K$-dimensional hidden binary 'class indicator' with transition probabilities $A_{jk} \triangleq p(z_{tk}=1\,|\,z_{t-1,j}=1)$, or equivalently
  $$p(z_t|z_{t-1}) = \prod_{k=1}^K \prod_{j=1}^K A_{jk}^{z_{t-1,j}z_{tk}}$$
which is usually accompanied by an initial state distribution $\pi_k \triangleq p(z_{1k}=1)$.

- The classical HMM has also discrete-valued observations but in pratice any (probabilistic) observation model $p(x_t|z_t)$ may be coupled to the hidden Markov chain.

- Another well-known state-space model with continuous-valued state variables $Z_t$ is the **(Linear) Gaussian Dynamical System** (LGDS), which is defined as

$$\begin{align*}
p(z_t\,|\,z_{t-1}) &= \mathcal{N}\left(\, A z_{t-1}\,,\,\Sigma_z\,\right) \\
p(x_t\,|\,z_t) &= \mathcal{N}\left(\, C z_t\,,\,\Sigma_x\,\right) \\
p(z_1) &= \mathcal{N}\left(\, \mu_1\,,\,\Sigma_1\,\right)
\end{align*}$$

- Note that the joint distribution over $\{(x_1,z_1),\ldots,(x_t,z_t)\}$ is a (large-dimensional) Gaussian distribution. This means that, in principle, every inference problem on the LGDS model also leads to a Gaussian distribution.

- HMM's and LGDS's (and variants thereof) are at the basis of a wide range of complex information processing systems, such as speech and language recognition, robotics and automatic car navigation, and even processing of DNA sequences.

### Kalman Filtering

- Technically, a [**Kalman filter**](https://en.wikipedia.org/wiki/Kalman_filter) is the solution to the recursive estimation (inference) of the hidden state $z_t$ based on past observations in an LGDS, i.e., Kalman filtering solves the problem $p(z_t\,|\,x^t)$ based on the previous estimate $p(z_{t-1}\,|\,x^{t-1})$ and a new observation $x_t$ (in the context of the given model specification of course).

- Let's infer the Kalman filter for a scalar linear Gaussian dynamical system:
$$\begin{align*}
    p(z_t\,|\,z_{t-1}) &= \mathcal{N}(z_t\,|\,a z_{t-1},\sigma_z^2) \tag{state transition}   \\
    p(x_t\,|\,z_t) &= \mathcal{N}(x_t\,|\,c z_t,\sigma_x^2) \tag{observation}
\end{align*}$$

- Kalman filtering comprises inferring $p(z_t\,|\,x^t)$ from a given prior estimate $p(z_{t-1}\,|\,x^{t-1})$ and a new observation $x_t$. Let us assume that
$$\begin{align}
p(z_{t-1}\,|\,x^{t-1}) = \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2) \tag{prior}
\end{align}$$

- Note that everything is Gaussian, so this is _in principle_ possible to execute inference problems analytically and the result will be a Gaussian posterior:

$$\begin{align*}
\underbrace{p(z_t\,|\,x^t)}_{\text{posterior}} &= p(z_t\,|\,x_t,x^{t-1}) \propto p(x_t,z_t\,|\,x^{t-1}) \\
  &\propto p(x_t\,|\,z_t) \,p(z_t\,|\,x^{t-1}) \\
  &= p(x_t\,|\,z_t) \, \sum_{z_{t-1}} p(z_t,z_{t-1}\,|\,x^{t-1}) \\
  &= \underbrace{p(x_t\,|\,z_t)}_{\text{observation}} \, \sum_{z_{t-1}} \underbrace{p(z_t\,|\,z_{t-1})}_{\text{state transition}} \, \underbrace{p(z_{t-1}\,|\,x^{t-1})}_{\text{prior}} \\
    &= \mathcal{N}(x_t\,|\,c z_t,\sigma_x^2) \sum_{z_{t-1}} \mathcal{N}(z_t\,|\,a z_{t-1},\sigma_z^2) \, \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2) \\
  &= c\,\mathcal{N}\left(z_t\bigm| \frac{x_t}{c} ,\left(\frac{\sigma_x}{c}\right)^2\right)  \sum_{z_{t-1}} a\,\underbrace{\mathcal{N}\left(z_{t-1}\bigm|  \frac{z_t}{a},\left(\frac{\sigma_z}{a}\right)^2 \right) \, \mathcal{N}(z_{t-1} \,|\, \mu_{t-1}, \sigma_{t-1}^2)}_{\text{use Gaussian multiplication formula SRG-6}} \\
  &\propto \underbrace{\mathcal{N}\left(z_t\,\bigm| \,\frac{x_t}{c} ,\left(\frac{\sigma_x}{c}\right)^2\right) \cdot \mathcal{N}\left(z_t\, \bigm|\,a \mu_{t-1},\sigma_z^2 + \left(a \sigma_{t-1}\right)^2 \right)}_{\text{use SRG-6 again}} \\
  &\propto \mathcal{N}\left( z_t \,|\, \mu_t, \sigma_t^2\right)
\end{align*}$$
with
$$\begin{align*}
  \rho_t^2 &= a^2 \sigma_{t-1}^2 + \sigma_z^2 \tag{predicted variance}\\
  K_t &= \frac{c \rho_t^2}{c^2 \rho_t^2 + \sigma_x^2} \tag{Kalman gain} \\
  \mu_t &= \underbrace{a \mu_{t-1}}_{\text{prior prediction}} + K_t \cdot \underbrace{\left( x_t - c a \mu_{t-1}\right)}_{\text{prediction error}} \tag{posterior mean}\\
  \sigma_t^2 &= \left( 1 - c\cdot K_t \right) \rho_t^2 \tag{posterior variance}
\end{align*}$$

- Kalman filtering consists of computing/updating these last four equations for each new observation ($x_t$).

### Multi-dimensional Kalman Filtering

- The Kalman filter equations can also be derived for multidimensional state-space models. In particular, for the model
$$\begin{align*}
z_t &= A z_{t-1} + \mathcal{N}(0,\Gamma) \\
x_t &= C z_t + \mathcal{N}(0,\Sigma)
\end{align*}$$
the Kalman filter update equations for the posterior $p(z_t |x^t) = \mathcal{N}\left(z_t \bigm| \mu_t, V_t \right)$ are given by (see Bishop, pg.639)
$$\begin{align*}
P_t &= A V_{t-1} A^T + \Gamma \tag{predicted variance}\\
K_t &= P_t C^T \cdot \left(C P_t C^T  + \Sigma \right)^{-1} \tag{Kalman gain} \\
\mu_t &= A \mu_{t-1} + K_t\cdot\left(x_t - C A \mu_{t-1} \right) \tag{posterior state mean}\\
V_t &= \left(I-K_t C \right) P_{t}  \tag{posterior state variance}
\end{align*}$$

### Kalman Filtering and the Cart Position Tracking Example Revisited

#### CODE EXAMPLE

- We can now solve the cart tracking problem of the introductory example by implementing the Kalman filter.

```julia
using Pkg;Pkg.activate("probprog/workspace/");Pkg.instantiate()
IJulia.clear_output();
```

```julia
using LinearAlgebra, PyPlot
include("scripts/cart_tracking_helpers.jl")

# Specify the model parameters
Δt = 1.0                     # assume the time steps to be equal in size
A = [1.0 Δt;
     0.0 1.0]
b = [0.5*Δt^2; Δt]
Σz = convert(Matrix,Diagonal([0.2*Δt; 0.1*Δt])) # process noise covariance
Σx = convert(Matrix,Diagonal([1.0; 2.0]))     # observation noise covariance;

# Generate noisy observations
n = 10                # perform 10 timesteps
z_start = [10.0; 2.0] # initial state
u = 0.2 * ones(n)     # constant input u
noisy_x = generateNoisyMeasurements(z_start, u, A, b, Σz, Σx);

m_z = noisy_x[1]                                    # initial predictive mean
V_z = A * (1e8*Diagonal(I,2) * A') + Σz             # initial predictive covariance

for t = 2:n
    global m_z, V_z, m_pred_z, V_pred_z
    #predict
    m_pred_z = A * m_z + b * u[t]                   # predictive mean
    V_pred_z = A * V_z * A' + Σz                    # predictive covariance
    #update
    gain = V_pred_z * inv(V_pred_z + Σx)            # Kalman gain
    m_z = m_pred_z + gain * (noisy_x[t] - m_pred_z) # posterior mean update
    V_z = (Diagonal(I,2)-gain)*V_pred_z             # posterior covariance update
end
println("Prediction: ",ProbabilityDistribution(Multivariate,GaussianMeanVariance,m=m_pred_z,v=V_pred_z))
println("Measurement: ", ProbabilityDistribution(Multivariate,GaussianMeanVariance,m=noisy_x[n],v=Σx))
println("Posterior: ", ProbabilityDistribution(Multivariate,GaussianMeanVariance,m=m_z,v=V_z))
plotCartPrediction2(m_pred_z[1], V_pred_z[1], m_z[1], V_z[1], noisy_x[n][1], Σx[1][1]);
```

### Recap Dynamical Models

- Dynamical systems do not obey the sample-by-sample independence assumption, but still can be specified, and state and parameter estimation equations can be solved by similar tools as for static models.

- Two of the more famous and powerful models with latent states include the hidden Markov model (with discrete states) and the Linear Gaussian dynamical system (with continuous states).

- For the LGDS, the Kalman filter is a well-known recursive state estimation procedure. The Kalman filter can be derived through Bayesian update rules on Gaussian distributions.

- If anything changes in the model, e.g., the state noise is not Gaussian, then you have to re-derive the inference equations again from scratch and it may not lead to an analytically pleasing answer.

- $\Rightarrow$ Generally, we will want to automate the inference processes. This topic is the domain of Probabilistic Programming, whih we will discuss later in this course.

##  OPTIONAL SLIDES

### Extensions of Generative Gaussian Models

- Using the methods of the previous lessons, it is possible to create your own new models based on stacking Gaussian and categorical distributions in new ways:

### Message Passing in State-space Models

- As we have seen, inference tasks in linear Gaussian state space models can be analytically solved, see e.g. [Faragher, 2012](https://github.com/bertdv/BMLIP/blob/master/lessons/notebooks/files/Faragher-2012-Understanding-the-Basis-of-the-Kalman-Filter.pdf) and the Kalman filter derivation above.

- However, these derivations quickly become cumbersome and prone to errors.

- Alternatively, we could specify the generative model in a (Forney-style) factor graph and use automated message passing to infer the posterior over the hidden variables. E.g., the message passing schedule for Kalman filtering looks like this:

### The Cart Tracking Problem Revisited: Inference by Message Passing

- Let's solve the cart tracking problem by sum-product message passing in a factor graph like the one depicted above. All we have to do is create factor nodes for the state-transition model $p(z_t|z_{t-1})$ and the observation model $p(x_t|z_t)$. Then we just build the factor graph and let [ForneyLab](http://forneylab.org) execute the message passing schedule.

#### CODE EXAMPLE

- We'll implement the following model using ForneyLab:

$$\begin{align*}
\begin{bmatrix} z_t \\ \dot{z_t}\end{bmatrix} &=  \begin{bmatrix} 1 & \Delta t \\ 0 & 1\end{bmatrix} \begin{bmatrix} z_{t-1} \\ \dot z_{t-1}\end{bmatrix} + \begin{bmatrix} (\Delta t)^2/2 \\ \Delta t\end{bmatrix} u_t + \mathcal{N}(0,\Sigma_z) \\
\mathbf{x}_t &= \begin{bmatrix} z_t \\ \dot{z_t}\end{bmatrix} + \mathcal{N}(0,\Sigma_x)
\end{align*}$$

Since the factor graph is just a concatination of $n$ identical "sections" (one for each time step), we only have to specify a single section. When running the message passing algorithm we will explictly use the posterior of the previous timestep as prior in the next one. Let's build a section of the factor graph:

```julia
fg = FactorGraph()
z_prev_m = Variable(id=:z_prev_m); placeholder(z_prev_m, :z_prev_m, dims=(2,))
z_prev_v = Variable(id=:z_prev_v); placeholder(z_prev_v, :z_prev_v, dims=(2,2))
bu = Variable(id=:bu); placeholder(bu, :bu, dims=(2,))

@RV z_prev ~ GaussianMeanVariance(z_prev_m, z_prev_v, id=:z_prev) # p(z_prev)
@RV noise_z ~ GaussianMeanVariance(constant(zeros(2), id=:noise_z_m), constant(Σz, id=:noise_z_v)) # process noise
@RV z = constant(A, id=:A) * z_prev + bu + noise_z; z.id = :z # p(z|z_prev) (state transition model)
@RV x ~ GaussianMeanVariance(z, constant(Σx, id=:Σx)) # p(x|z) (observation model)
placeholder(x, :x, dims=(2,));
ForneyLab.draw(fg)
```

<?xml version="1.0" encoding="UTF-8" standalone="no"?>
<!DOCTYPE svg PUBLIC "-//W3C//DTD SVG 1.1//EN"
 "http://www.w3.org/Graphics/SVG/1.1/DTD/svg11.dtd">

<svg width="676pt" height="746pt"
 viewBox="0.00 0.00 675.50 746.00" xmlns="http://www.w3.org/2000/svg" xmlns:xlink="http://www.w3.org/1999/xlink">
<g id="graph0" class="graph" transform="scale(1 1) rotate(0) translate(4 742)">
<title>G</title>
<polygon fill="white" stroke="none" points="-4,4 -4,-742 671.5,-742 671.5,4 -4,4"/>

<g id="node1" class="node"><title>1812756999250375734</title>
<polygon fill="none" stroke="black" points="242.5,-405 138.5,-405 138.5,-333 242.5,-333 242.5,-405"/>
<text text-anchor="middle" x="190.5" y="-371.8" font-family="Times,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="190.5" y="-361.8" font-family="Times,serif" font-size="9.00">gaussianmeanvariance_1</text>
</g>

<g id="node8" class="node"><title>14895999851791313906</title>
<polygon fill="lightgrey" stroke="black" points="55,-279 0,-279 0,-225 55,-225 55,-279"/>
<text text-anchor="middle" x="27.5" y="-249.8" font-family="Times,serif" font-size="9.00">noise_z_m</text>
</g>

<g id="edge6" class="edge"><title>1812756999250375734--14895999851791313906</title>
<path fill="none" stroke="black" d="M140.709,-332.871C112.622,-313.055 78.5821,-289.04 55.2084,-272.549"/>
<text text-anchor="start" x="114.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">noise_z_m</text>
<text text-anchor="start" x="55.2084" y="-275.149" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="125.709" y="-326.471" font-family="Times,serif" font-size="8.00">2 m </text>
</g>

<g id="node10" class="node"><title>547507601661408333</title>
<polygon fill="lightgrey" stroke="black" points="224.5,-279 170.5,-279 170.5,-225 224.5,-225 224.5,-279"/>
<text text-anchor="middle" x="197.5" y="-249.8" font-family="Times,serif" font-size="9.00">noise_z_v</text>
</g>

<g id="edge2" class="edge"><title>1812756999250375734--547507601661408333</title>
<path fill="none" stroke="black" d="M192.638,-332.871C193.692,-315.565 194.94,-295.056 195.908,-279.154"/>
<text text-anchor="start" x="194.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">noise_z_v</text>
<text text-anchor="start" x="176.908" y="-281.754" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="179.638" y="-326.471" font-family="Times,serif" font-size="8.00">3 v </text>
</g>

<g id="node2" class="node"><title>9394537050200738147</title>
<polygon fill="none" stroke="black" points="312.5,-522 240.5,-522 240.5,-450 312.5,-450 312.5,-522"/>
<text text-anchor="middle" x="276.5" y="-488.8" font-family="Times,serif" font-size="9.00">+</text>
<text text-anchor="middle" x="276.5" y="-478.8" font-family="Times,serif" font-size="9.00">addition_2</text>
</g>

<g id="edge1" class="edge"><title>9394537050200738147--1812756999250375734</title>
<path fill="none" stroke="black" d="M240.417,-462.122C229.549,-453.761 218.419,-443.498 210.5,-432 204.983,-423.989 200.915,-414.364 197.946,-405.113"/>
<text text-anchor="start" x="210.5" y="-425.6" font-family="Times,serif" font-size="8.00" fill="red">noise_z</text>
<text text-anchor="start" x="178.946" y="-407.713" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="221.417" y="-464.722" font-family="Times,serif" font-size="8.00">3 in2 </text>
</g>

<g id="node11" class="node"><title>12903692398295715415</title>
<polygon fill="none" stroke="black" points="429.5,-405 357.5,-405 357.5,-333 429.5,-333 429.5,-405"/>
<text text-anchor="middle" x="393.5" y="-371.8" font-family="Times,serif" font-size="9.00">+</text>
<text text-anchor="middle" x="393.5" y="-361.8" font-family="Times,serif" font-size="9.00">addition_1</text>
</g>

<g id="edge13" class="edge"><title>9394537050200738147--12903692398295715415</title>
<path fill="none" stroke="black" d="M312.24,-449.871C326.621,-435.736 343.176,-419.463 357.576,-405.31"/>
<text text-anchor="start" x="339.5" y="-425.6" font-family="Times,serif" font-size="8.00" fill="red">variable_2</text>
<text text-anchor="start" x="338.576" y="-407.91" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="293.24" y="-443.471" font-family="Times,serif" font-size="8.00">2 in1 </text>
</g>

<g id="node3" class="node"><title>17878356443197158701</title>
<polygon fill="lightgrey" stroke="black" points="425.5,-738 359.5,-738 359.5,-684 425.5,-684 425.5,-738"/>
<text text-anchor="middle" x="392.5" y="-708.8" font-family="Times,serif" font-size="9.00">placeholder_x</text>
</g>

<g id="node5" class="node"><title>12992180138497340926</title>
<polygon fill="none" stroke="black" points="444.5,-639 340.5,-639 340.5,-567 444.5,-567 444.5,-639"/>
<text text-anchor="middle" x="392.5" y="-605.8" font-family="Times,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="392.5" y="-595.8" font-family="Times,serif" font-size="9.00">gaussianmeanvariance_2</text>
</g>

<g id="edge5" class="edge"><title>17878356443197158701--12992180138497340926</title>
<path fill="none" stroke="black" d="M392.5,-683.798C392.5,-670.31 392.5,-653.68 392.5,-639.099"/>
<text text-anchor="start" x="392.5" y="-659.6" font-family="Times,serif" font-size="8.00" fill="red">x</text>
<text text-anchor="start" x="373.5" y="-641.699" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="373.5" y="-677.398" font-family="Times,serif" font-size="8.00">1 out </text>
</g>

<g id="node4" class="node"><title>11780604053988617847</title>
<polygon fill="none" stroke="black" points="498.5,-171 426.5,-171 426.5,-99 498.5,-99 498.5,-171"/>
<text text-anchor="middle" x="462.5" y="-137.8" font-family="Times,serif" font-size="9.00">𝒩</text>
<text text-anchor="middle" x="462.5" y="-127.8" font-family="Times,serif" font-size="9.00">z_prev</text>
</g>

<g id="node6" class="node"><title>141279589740639774</title>
<polygon fill="lightgrey" stroke="black" points="406,-54 309,-54 309,-0 406,-0 406,-54"/>
<text text-anchor="middle" x="357.5" y="-24.8" font-family="Times,serif" font-size="9.00">placeholder_z_prev_m</text>
</g>

<g id="edge9" class="edge"><title>11780604053988617847--141279589740639774</title>
<path fill="none" stroke="black" d="M426.282,-109.598C415.134,-101.218 403.243,-91.3286 393.5,-81 385.867,-72.9086 378.705,-63.0514 372.809,-54.0427"/>
<text text-anchor="start" x="393.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="red">z_prev_m</text>
<text text-anchor="start" x="353.809" y="-56.6427" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="411.282" y="-112.198" font-family="Times,serif" font-size="8.00">2 m </text>
</g>

<g id="node9" class="node"><title>18367179973254092605</title>
<polygon fill="lightgrey" stroke="black" points="616,-54 521,-54 521,-0 616,-0 616,-54"/>
<text text-anchor="middle" x="568.5" y="-24.8" font-family="Times,serif" font-size="9.00">placeholder_z_prev_v</text>
</g>

<g id="edge12" class="edge"><title>11780604053988617847--18367179973254092605</title>
<path fill="none" stroke="black" d="M497.502,-98.9979C512.096,-84.4044 528.755,-67.7451 542.267,-54.2331"/>
<text text-anchor="start" x="522.5" y="-74.6" font-family="Times,serif" font-size="8.00" fill="red">z_prev_v</text>
<text text-anchor="start" x="523.267" y="-56.8331" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="484.502" y="-92.5979" font-family="Times,serif" font-size="8.00">3 v </text>
</g>

<g id="edge10" class="edge"><title>12992180138497340926--9394537050200738147</title>
<path fill="none" stroke="black" d="M357.066,-566.871C342.808,-552.736 326.394,-536.463 312.117,-522.31"/>
<text text-anchor="start" x="338.5" y="-542.6" font-family="Times,serif" font-size="8.00" fill="red">z</text>
<text text-anchor="start" x="293.117" y="-524.91" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="342.066" y="-560.471" font-family="Times,serif" font-size="8.00">2 m </text>
</g>

<g id="node12" class="node"><title>1600820188567460050</title>
<polygon fill="lightgrey" stroke="black" points="481.5,-513 427.5,-513 427.5,-459 481.5,-459 481.5,-513"/>
<text text-anchor="middle" x="454.5" y="-483.8" font-family="Times,serif" font-size="9.00">Σx</text>
</g>

<g id="edge7" class="edge"><title>12992180138497340926--1600820188567460050</title>
<path fill="none" stroke="black" d="M411.439,-566.871C420.769,-549.565 431.826,-529.056 440.4,-513.154"/>
<text text-anchor="start" x="425.5" y="-542.6" font-family="Times,serif" font-size="8.00" fill="red">Σx</text>
<text text-anchor="start" x="421.4" y="-515.754" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="398.439" y="-560.471" font-family="Times,serif" font-size="8.00">3 v </text>
</g>

<g id="node7" class="node"><title>15610035397149884921</title>
<polygon fill="lightgrey" stroke="black" points="417,-279 346,-279 346,-225 417,-225 417,-279"/>
<text text-anchor="middle" x="381.5" y="-249.8" font-family="Times,serif" font-size="9.00">placeholder_bu</text>
</g>

<g id="edge3" class="edge"><title>12903692398295715415--15610035397149884921</title>
<path fill="none" stroke="black" d="M389.834,-332.871C388.029,-315.565 385.888,-295.056 384.229,-279.154"/>
<text text-anchor="start" x="388.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">bu</text>
<text text-anchor="start" x="365.229" y="-281.754" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="370.834" y="-326.471" font-family="Times,serif" font-size="8.00">3 in2 </text>
</g>

<g id="node13" class="node"><title>7697426523944366023</title>
<polygon fill="none" stroke="black" points="606.5,-288 532.5,-288 532.5,-216 606.5,-216 606.5,-288"/>
<text text-anchor="middle" x="569.5" y="-254.8" font-family="Times,serif" font-size="9.00">×</text>
<text text-anchor="middle" x="569.5" y="-244.8" font-family="Times,serif" font-size="9.00">multiplication_1</text>
</g>

<g id="edge4" class="edge"><title>12903692398295715415--7697426523944366023</title>
<path fill="none" stroke="black" d="M429.541,-344.451C459.51,-324.868 502.006,-297.101 532.294,-277.311"/>
<text text-anchor="start" x="487.5" y="-308.6" font-family="Times,serif" font-size="8.00" fill="red">variable_1</text>
<text text-anchor="start" x="513.294" y="-279.911" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="429.541" y="-347.051" font-family="Times,serif" font-size="8.00">2 in1 </text>
</g>

<g id="edge8" class="edge"><title>7697426523944366023--11780604053988617847</title>
<path fill="none" stroke="black" d="M536.815,-215.871C523.663,-201.736 508.523,-185.463 495.353,-171.31"/>
<text text-anchor="start" x="519.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="red">z_prev</text>
<text text-anchor="start" x="476.353" y="-173.91" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="517.815" y="-209.471" font-family="Times,serif" font-size="8.00">2 in1 </text>
</g>

<g id="node14" class="node"><title>4936880759005548229</title>
<polygon fill="lightgrey" stroke="black" points="667.5,-162 613.5,-162 613.5,-108 667.5,-108 667.5,-162"/>
<text text-anchor="middle" x="640.5" y="-132.8" font-family="Times,serif" font-size="9.00">A</text>
</g>

<g id="edge11" class="edge"><title>7697426523944366023--4936880759005548229</title>
<path fill="none" stroke="black" d="M591.188,-215.871C601.873,-198.565 614.535,-178.056 624.353,-162.154"/>
<text text-anchor="start" x="607.5" y="-191.6" font-family="Times,serif" font-size="8.00" fill="red">A</text>
<text text-anchor="start" x="605.353" y="-164.754" font-family="Times,serif" font-size="8.00">1 out </text>
<text text-anchor="start" x="578.188" y="-209.471" font-family="Times,serif" font-size="8.00">3 a </text>
</g>
</g>
</svg>

Now that we've built the factor graph, we can perform Kalman filtering by inserting measurement data into the factor graph and performing message passing.

```julia
include("scripts/cart_tracking_helpers.jl")
eval(Meta.parse(sumProductAlgorithm(z))) # build message passing algorithm
marginals = Dict()
messages = Array{Message}(undef,6)
z_prev_m_0 = noisy_x[1]
z_prev_v_0 = A * (1e8*Diagonal(I,2) * A') + Σz
for t=2:n
    data = Dict(:x => noisy_x[t], :bu => b*u[t],:z_prev_m => z_prev_m_0, :z_prev_v => z_prev_v_0)
    step!(data, marginals, messages) # perform msg passing (single timestep)
    # Posterior of z becomes prior of z in the next timestep:
    z_prev_m_0 = ForneyLab.unsafeMean(marginals[:z])
    z_prev_v_0 = ForneyLab.unsafeCov(marginals[:z])
end
# Collect prediction p(z[n]|z[n-1]), measurement p(z[n]|x[n]), corrected prediction p(z[n]|z[n-1],x[n])
prediction      = messages[5].dist # the message index is found by manual inspection of the schedule
measurement     = messages[6].dist
corr_prediction = convert(ProbabilityDistribution{Multivariate, GaussianMeanVariance}, marginals[:z])
println("Prediction: ",prediction)
println("Measurement: ",measurement)
println("Posterior: ", corr_prediction)

# Make a fancy plot of the prediction, noisy measurement, and corrected prediction after n timesteps
plotCartPrediction(prediction, measurement, corr_prediction);

```

- Note that both the analytical Kalman filtering solution and the message passing solution lead to the same results. The advantage of message passing-based inference with ForneyLab is that we did not need to derive any inference equations. ForneyLab took care of all that.

- ForneyLab is an example of a **probabilistic programming language** (PPL). More about PPLs later in the PP minicourse.

```julia
open("../../styles/aipstyle.html") do f display("text/html", read(f, String)) end
```

<link href='http://fonts.googleapis.com/css?family=Alegreya+Sans:100,300,400,500,700,800,900,100italic,300italic,400italic,500italic,700italic,800italic,900italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Arvo:400,700,400italic' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=PT+Mono' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Shadows+Into+Light' rel='stylesheet' type='text/css'>
<link href='http://fonts.googleapis.com/css?family=Nixie+One' rel='stylesheet' type='text/css'>

```julia

```


## Introduction to Data Analytics and Geostatistics and Spatial Data Analytics Courses

## GeostatsPy: Spatial Continuity Directions for Subsurface Data Analytics in Python

### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### PGE 383 Exercise: Methods to Detect Directions of Continuity with GeostatsPy

Here's a simple workflow on detecting the major spatial continuity directions in a spatial dataset with variogram analysis. This information is essential to optimum well placement and prectiction away from wells.  First let's explain the concept of spatial continuity and the variogram.

#### Spatial Continuity

**Spatial Continuity** is the correlation between values over distance.

* No spatial continuity – no correlation between values over distance, random values at each location in space regardless of separation distance.

* Homogenous phenomenon have perfect spatial continuity, since all values as the same (or very similar) they are correlated.

We need a statistic to quantify spatial continuity! A convenient method is the Semivariogram.

#### The Semivariogram

Function of difference over distance.

* The expected (average) squared difference between values separated by a lag distance vector (distance and direction), $h$:

\begin{equation}
\gamma(\bf{h}) = \frac{1}{2 N(\bf{h})} \sum^{N(\bf{h})}_{\alpha=1} (z(\bf{u}_\alpha) - z(\bf{u}_\alpha + \bf{h}))^2
\end{equation}

where $z(\bf{u}_\alpha)$ and $z(\bf{u}_\alpha + \bf{h})$ are the spatial sample values at tail and head locations of the lag vector respectively.

* Calculated over a suite of lag distances to obtain a continuous function.

* the $\frac{1}{2}$ term converts a variogram into a semivariogram, but in practice the term variogram is used instead of semivariogram.
* We prefer the semivariogram because it relates directly to the covariance function, $C_x(\bf{h})$ and univariate variance, $\sigma^2_x$:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

Note the correlogram is related to the covariance function as:

\begin{equation}
\rho_x(\bf{h}) = \frac{C_x(\bf{h})}{\sigma^2_x}
\end{equation}

The correlogram provides of function of the $\bf{h}-\bf{h}$ scatter plot correlation vs. lag offset $\bf{h}$.

\begin{equation}
-1.0 \le \rho_x(\bf{h}) \le 1.0
\end{equation}

#### Variogram Observations

The following are common observations for variograms that should assist with their practical use.

##### Observation \#1 - As distance increases, variability increase (in general).

This is common since in general, over greater distance offsets, there is often more difference between the head and tail samples.

In some cases, such as with spatial cyclicity of the hole effect variogram model the variogram may have negative slope over somelag distance intervals

Negative slopes at lag distances greater than half the data extent are often caused by too few pairs for a reliable variogram calculation

##### Observation \#2 - Calculated with over all possible pairs separated by lag vector, $\bf{𝐡}$.

We scan through the entire data set, searching for all possible pair combinations with all other data.  We then calculate the variogram as one half the expectation of squared difference between all pairs.

More pairs results in a more reliable measure.

##### Observation \#3 - Need to plot the sill to know the degree of correlation.

**Sill** is the variance, $\sigma^2_x$

Given stationarity of the variance, $\sigma^2_x$, and variogram $\gamma(\bf{h})$:

we can define the covariance function:

\begin{equation}
C_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

The covariance measure is a measure of similarity over distance (the mirror image of the variogram as shown by the equation above).

Given a standardized distribution $\sigma^2_x = 1.0$, the covariance, $C_x(\bf{h})$, is equal to the correlogram, $\rho_x(\bf{h})$:

\begin{equation}
\rho_x(\bf{h}) = \sigma^2_x - \gamma(\bf{h})
\end{equation}

##### Observation \#4 - The lag distance at which the variogram reaches the sill is know as the range.

At the range, knowing the data value at the tail location provides no information about a value at the head location of the lag distance vector.

##### Observation \#5 - The nugget effect, a discontinuity at the origin

Sometimes there is a discontinuity in the variogram at distances less than the minimum data spacing.  This is known as **nugget effect**.

The ratio of nugget / sill, is known as relative nugget effect (%). Modeled as a discontinuity with no correlation structure that at lags, $h \gt \epsilon$, an infinitesimal lag distance, and perfect correlation at $\bf{h} = 0$.
Caution when including nuggect effect in the variogram model as measurement error, mixing populations cause apparent nugget effect

This exercise demonstrates the semivariogram calculation with GeostatsPy. The steps include:

1. generate a 2D model with sequential Gaussian simulation
2. sample from the simulation
3. calculate and visualize experimental semivariograms

#### Detecting Directions of Continuity

Spatial continuity can be described with nested spatial continuity models:

\begin{equation}
\Gamma_x(\bf{h}) = \sum_{i=1}^{nst} \gamma_i(\bf{h})
\end{equation}

where $\Gamma_x(\bf{h})$ is the nested variogram model resulting from the summation of $nst$ nested variograms  $\gamma_i(\bf{h})$.

Each one of these variogram structures, $\gamma_i(\bf{h})$, is based on a geometric anisotropy model parameterized by the orientation and range in the major and minor directions.  In 2D this is simply an azimuth and ranges, $azi$, $a_{maj}$ and $a_{min}$. Note, the range in the minor direction (orthogonal to the major direction.

The geometric anisotropy model assumes that the range in all off-diagonal directions is based on an ellipse with the major and minor axes alligned with and set to the major and minor for the variogram.

\begin{equation}
\bf{h}_i = \sqrt{\left(\frac{r_{maj}}{a_{maj_i}}\right)^2 + \left(\frac{r_{maj}}{a_{maj_i}}\right)^2}
\end{equation}

Therefore, if we know the major direction, range in major and minor directions, we may completely describe each nested componnent of the complete spatial continuity of the variable of interest, $i = 1,\dots,nst$.

In this workflow we will explore methods to detect directionality from a spatial dataset.

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - [sample_data.csv](https://git.io/fh0CW) at https://git.io/fh0CW.

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Load the required libraries

The following code loads the required libraries.

```python
import geostatspy.GSLIB as GSLIB                       # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats                 # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
import os                                               # to set current working directory
import numpy as np                                      # arrays and matrix math
import pandas as pd                                     # DataFrames
import matplotlib.pyplot as plt                         # plotting
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Set the working directory

I always like to do this so I don't lose files and to simplify subsequent read and writes (avoid including the full address each time).  Also, in this case make sure to place the required (see above) GSLIB executables in this directory or a location identified in the environmental variable *Path*.

```python
os.chdir("d:/PGE383")                                   # set the working directory
```

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

```python
df = pd.read_csv("sample_data_biased.csv")                     # read a .csv file in as a DataFrame
#print(df.iloc[0:5,:])                                  # display first 4 samples in the table as a preview
df.head()                                               # we could also use this command for a table preview
```

      X
      Y
      Facies
      Porosity
      Perm

      0
      100
      900
      1
      0.115359
      5.736104

      1
      100
      800
      1
      0.136425
      17.211462

      2
      100
      600
      1
      0.135810
      43.724752

      3
      100
      500
      0
      0.094414
      1.609942

      4
      100
      100
      0
      0.113049
      10.886001

We will work with all facies pooled together. I wanted to simplify this workflow and focus more on spatial continuity direction detection. Finally, by not using facies we do have more samples to support our statistical inference. Most often facies are essential in the subsurface model. Don't worry we will check if this is reasonable in a bit.

You are welcome to repeat this workflow on a by-facies basis.  The following code could be used to build DataFrames ('df_sand' and 'df_shale') for each facies.

```p
df_sand = pd.DataFrame.copy(df[df['Facies'] == 1]).reset_index()  # copy only 'Facies' = sand records
df_shale = pd.DataFrame.copy(df[df['Facies'] == 0]).reset_index() # copy only 'Facies' = shale records
```

Let's look at summary statistics for all facies combined:

```python
df.describe().transpose()                          # summary table of sand only DataFrame statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      289.0
      475.813149
      254.277530
      0.000000
      300.000000
      430.000000
      670.000000
      990.000000

      Y
      289.0
      529.692042
      300.895374
      9.000000
      269.000000
      549.000000
      819.000000
      999.000000

      Facies
      289.0
      0.813149
      0.390468
      0.000000
      1.000000
      1.000000
      1.000000
      1.000000

      Porosity
      289.0
      0.134744
      0.037745
      0.058548
      0.106318
      0.126167
      0.154220
      0.228790

      Perm
      289.0
      207.832368
      559.359350
      0.075819
      3.634086
      14.908970
      71.454424
      5308.842566

Let's transform the porosity and permeaiblity data to standard normal (mean = 0.0, standard deviation = 1.0, Gaussian shape). This is required for sequential Gaussian simulation (common target for our variogram models) and the Gaussian transform assists with outliers and provides more interpretable variograms.

Let's look at the inputs for the GeostatsPy nscore program.  Note the output include an ndarray with the transformed values (in the same order as the input data in Dataframe 'df' and column 'vcol'), and the transformation table in original values and also in normal score values.

```python
geostats.nscore                                         # see the input parameters required by the nscore function
```

    <function geostatspy.geostats.nscore(df, vcol, wcol=None, ismooth=False, dfsmooth=None, smcol=0, smwcol=0)>

The following command will transform the Porosity and Permeabilty to standard normal.

```python
#Transform to Gaussian by Facies
df['NPor'], tvPor, tnsPor = geostats.nscore(df, 'Porosity') # nscore transform for all facies porosity
df['NPerm'], tvPermSand, tnsPermSand = geostats.nscore(df, 'Perm')  # nscore transform for all facies permeability
```

Let's look at the updated DataFrame to make sure that we now have the normal score porosity and permeability.

```python
df.head()                                               # preview sand DataFrame with nscore transforms
```

      X
      Y
      Facies
      Porosity
      Perm
      NPor
      NPerm

      0
      100
      900
      1
      0.115359
      5.736104
      -0.391400
      -0.429142

      1
      100
      800
      1
      0.136425
      17.211462
      0.299307
      0.112995

      2
      100
      600
      1
      0.135810
      43.724752
      0.272201
      0.556521

      3
      100
      500
      0
      0.094414
      1.609942
      -1.269810
      -1.178106

      4
      100
      100
      0
      0.113049
      10.886001
      -0.496733
      -0.156762

That looks good! One way to check is to see if the relative magnitudes of the normal score transformed values match the original values.  e.g. that the normal score transform of 0.10 porosity normal score is less than the normal score transform of 0.14 porsity.  Also, the normal score transform of values close to the mean value should be close to 0.0

Let's also check the original and transformed sand and shale porosity distributions.

```python
plt.subplot(221)                                        # plot original sand and shale porosity histograms
plt.hist(df['Porosity'], facecolor='red',bins=np.linspace(0.0,0.25,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label='Original')
plt.xlim([0.05,0.25]); plt.ylim([0,1.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Porosity')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(222)
plt.hist(df['NPor'], facecolor='blue',bins=np.linspace(-3.0,3.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label = 'Trans')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Nscore Porosity')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(223)                                        # plot nscore transformed sand and shale histograms
plt.hist(df['Perm'], facecolor='red',bins=np.linspace(0.0,1000.0,100000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label='Original')
plt.xlim([0.0,1000.0]); plt.ylim([0,1.0])
plt.xlabel('Porosity (fraction)'); plt.ylabel('Frequency'); plt.title('Permeability')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplot(224)                                        # plot nscore transformed sand and shale histograms
plt.hist(df['NPerm'], facecolor='blue',bins=np.linspace(-3.0,3.0,100000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black',label = 'Trans')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Permeability (mD)'); plt.ylabel('Frequency'); plt.title('Nscore Permeability')
plt.legend(loc='upper left')
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=2.2, wspace=0.2, hspace=0.3)
plt.show()
```

The normal score transform has correctly transformed the porosity and permeability to standard normal.

#### Method \#1:Ocular Inspection of Posted Data

Data visualization is very useful to detect patterns. Our brains are very good at pattern detection. I promote quantitative methods and recognize issues with cognitive bias, but it is important to recognize the value is expert intepretation based on data visualization.

* This data visualization will also be important to assist with parameter selection for the quantitative methods later.

Let's plot the location maps of normal score transforms of porosity and permeability for all facies. We will also include a cross plot of the nscore permeability vs. porosity colored by facies to aid with comparison in spatial features between the porosity and permeability data.

```python
cmap = plt.cm.plasma                    # color map
plt.subplot(131)
GSLIB.locmap_st(df,'X','Y','NPor',0,1000,0,1000,-3,3,'Nscore Porosity - All Facies','X (m)','Y (m)','Nscore Porosity',cmap)

plt.subplot(132)
GSLIB.locmap_st(df,'X','Y','NPerm',0,1000,0,1000,-3,3,'Nscore Permeability - All Facies','X (m)','Y (m)','Nscore Permeability',cmap)

plt.subplot(133)
facies = df['Facies'].values +0.01
plt.scatter(df['NPor'],df['NPerm'],c = facies,edgecolor = 'black',cmap = plt.cm.inferno)
#plt.plot([-3,3],[-3,3],color = 'black')
plt.xlabel(r'Nscore Porosity')
plt.ylabel(r'Nscore Permeability')
plt.title('Nscore Permeability vs. Porosity')
plt.xlim([-3,3])
plt.ylim([-3,3])
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=0.8, wspace=0.5, hspace=0.3)
plt.show()
```

Let's add the variogram map for irreuglar data.  I will add this to GeostatsPy soon.

What do you see?  Here's my observations:

* there is a high degree of spatial agreement between porosity and permeability, this is supported by the high correlation evident in the cross plot.
* there are no discontinuities that could suggest that facies represent a distinct change, rather the porosity and permeability seem continuous and the assigned facies are a truncation of their continous behavoir, we doing 'ok' with no facies
* suspect a 045 azimuth major direction of continuity (up - right)
* there may be cycles in the 135 azimuth
* there will not likely be a nugget effect, but there is an hint of some short scale discontinuity?

**Do you agree?** If you have a different observations, drop me a line at mpyrcz@austin.utexas.edu and I'll add to this lesson with credit.

#### Quantitative Methods

Let's try out variogram maps.

* I realized that I had not coded variogram maps in Python, so I just did it and added it to GeostatsPy.

```python
# GSLIB's VARMAP program (Deutsch and Journel, 1998) converted from the original Fortran to Python
# by Michael Pyrcz, the University of Texas at Austin (Jan, 2019)
# Note simplified for 2D, irrelgular data only

def varmapv(df,xcol,ycol,vcol,tmin,tmax,nxlag,nylag,dxlag,dylag,minnp,isill):

# Parameters - consistent with original GSLIB
# df - DataFrame with the spatial data, xcol, ycol, vcol coordinates and property columns
# tmin, tmax - property trimming limits
# xlag, xltol - lag distance and lag distance tolerance
# nlag - number of lags to calculate
# azm, atol - azimuth and azimuth tolerance
# bandwh - horizontal bandwidth / maximum distance offset orthogonal to azimuth
# isill - 1 for standardize sill

# Load the data

    df_extract = df.loc[(df[vcol] >= tmin) & (df[vcol] <= tmax)]    # trim values outside tmin and tmax
    nd = len(df_extract)
    x = df_extract[xcol].values
    y = df_extract[ycol].values
    vr = df_extract[vcol].values

# Summary statistics for the data after trimming
    avg = vr.mean()
    stdev = vr.std()
    sills = stdev**2.0
    ssq = sills
    vrmin = vr.min()
    vrmax = vr.max()

# Initialize the summation arrays
    npp = np.zeros((nylag*2+1,nxlag*2+1))
    gam = np.zeros((nylag*2+1,nxlag*2+1))
    nppf = np.zeros((nylag*2+1,nxlag*2+1))
    gamf = np.zeros((nylag*2+1,nxlag*2+1))
    hm = np.zeros((nylag*2+1,nxlag*2+1))
    tm = np.zeros((nylag*2+1,nxlag*2+1))
    hv = np.zeros((nylag*2+1,nxlag*2+1))
    tv = np.zeros((nylag*2+1,nxlag*2+1))

# First fix the location of a seed point:
    for i in range(0,nd):
# Second loop over the data:
        for j in range(0,nd):
# The lag:
            ydis = y[j] - y[i]
            iyl = nylag + int(ydis/dylag)
            if iyl < 0 or iyl > nylag*2: # acocunting for 0,...,n-1 array indexing
                continue
            xdis = x[j] - x[i]
            ixl = nxlag + int(xdis/dxlag)
            if ixl < 0 or ixl > nxlag*2: # acocunting for 0,...,n-1 array indexing
                continue

# We have an acceptable pair, therefore accumulate all the statistics
# that are required for the variogram:
            npp[iyl,ixl] = npp[iyl,ixl] + 1 # our ndarrays read from the base to top, so we flip
            tm[iyl,ixl] = tm[iyl,ixl] + vr[i]
            hm[iyl,ixl] = hm[iyl,ixl] + vr[j]
            tv[iyl,ixl] = tm[iyl,ixl] + vr[i]*vr[i]
            hv[iyl,ixl] = hm[iyl,ixl] + vr[j]*vr[j]
            gam[iyl,ixl] = gam[iyl,ixl] + ((vr[i]-vr[j])*(vr[i]-vr[j]))

# Get average values for gam, hm, tm, hv, and tv, then compute
# the correct "variogram" measure:
    for iy in range(0,nylag*2+1):
        for ix in range(0,nxlag*2+1):
            if npp[iy,ix] <= minnp:
                gam[iy,ix] = -999.
                hm[iy,ix]  = -999.
                tm[iy,ix]  = -999.
                hv[iy,ix]  = -999.
                tv[iy,ix]  = -999.
            else:
                rnum = npp[iy,ix]
                gam[iy,ix] = gam[iy,ix] / (2*rnum) # semivariogram
                hm[iy,ix] = hm[iy,ix] / rnum
                tm[iy,ix] = tm[iy,ix] / rnum
                hv[iy,ix] = hv[iy,ix] / rnum - hm[iy,ix]*hm[iy,ix]
                tv[iy,ix] = tv[iy,ix] / rnum - tm[iy,ix]*tm[iy,ix]

# Attempt to standardize:
            if isill > 0:
                gamf[iy,ix] = gamf[iy,ix]/sills

    for iy in range(0,nylag*2+1):
        for ix in range(0,nxlag*2+1):
            gamf[iy,ix] = gam[nylag*2-iy,ix]
            nppf[iy,ix] = npp[nylag*2-iy,ix]

    return gamf, nppf

```

The inputs include:

* input data - DataFrame, 'df', and columns for x, y and property of interest, 'x', 'y' and 'vcol',
* variogram map parameters - number of cells in each direction to search, 'nxlag', nylag', the cell size / lag distance, 'dxlag' and 'dylag'''
* search - the minimum number of pairs reuqired to assign a result, 'mnpairs'
* normalization - 1 for standardize variance to 1.0 and 0 for not

The output is a 2D ndarray with the variogram map and the number of pairs.

```python
vmap, npmap = varmapv(df,'X','Y','NPor',tmin=-999,tmax=999,nxlag=11,nylag=11,dxlag=50,dylag=50,minnp=1,isill=1)

xmin = -575;ymin = -575; xmax = 575; ymax = 575; step = 50; vmin = 0.0; vmax = 1.6
plt.subplot(111)
xx, yy = np.meshgrid(np.arange(xmin, xmax, step), np.arange(ymax, ymin, -1 * step))
#im = plt.contourf(xx,yy,vmap,cmap=cmap,vmin=vmin,vmax=vmax,levels=np.linspace(vmin, vmax, 100),)
plt.imshow(vmap,interpolation = None,extent = [xmin,xmax,ymin,ymax], vmin = vmin, vmax = vmax,cmap = cmap)
plt.title('Variogram Map'); plt.xlabel('X Offset (m)'); plt.ylabel('Y Offset (m)')
cbar = plt.colorbar(im, orientation="vertical", ticks=np.linspace(vmin, vmax, 10))
cbar.set_label('Variogram Value', rotation=270, labelpad=20)

plt.subplots_adjust(left=0.0, bottom=0.0, right=1.0, top=1.2, wspace=0.2, hspace=0.2)
plt.show()

print('The shape of the output is ' + str(vmap.shape))
```

Notice that the output ndarray is 23, 23 cells? We asked for the number of cells to extend in each direction, 11 and 11 in x and y.  The map has an origin (zero distance) cell in the middle and extends 11 in both positive and negative directions.  So we have 2*$nx$ + 1, 2*$ny$ + 1 cells in the resulting variogram map and the $xmin = -1 * (nx * x_{cellsize} + \frac{1}{2} x_{cellsize})$ and the $xmax = nx * x_{cellsize} + \frac{1}{2} x_{cellsize}$

What do you think of this variogram map? These are my observations:

* major continuity direction is at azimuth 045
* there is a high degree of geometric anisotropy
* there is cyclicity in the 135 direction
* there may be some cyclicity in the 045 direction

From this variogram map we can immediately see directionality in our spatial data.

#### Experimental Variograms

Another method for exploring spatial data directionality is the calculation of multiple experimental variograms for a variety of directions.

We can use the location maps to help determine good variogram calculation parameters.

```p
tmin = -9999.; tmax = 9999.;
lag_dist = 100.0; lag_tol = 50.0; nlag = 7; bandh = 9999.9; azi = azi; atol = 22.5; isill = 1
```
* tmin, tmax are trimming limits - set to have no impact, no need to filter the data
* lag_dist, lag_tol are the lag distance, lag tolerance - set based on the common data spacing (100m) and tolerance as 100% of lag distance for additonal smoothing
* nlag is number of lags - set to extend just past 50 of the data extent
* bandh is the horizontal band width - set to have no effect
* azi is the azimuth -  it has not effect since we set atol, the azimuth tolerance, to 90.0
* isill is a boolean to standardize the distribution to a variance of 1 - it has no effect since the nscore transform sets the variance to 1.0

```python
tmin = -9999.; tmax = 9999.                             # no trimming
lag_dist = 100.0; lag_tol = 50.0; nlag = 7;            # maximum lag is 700m and tolerance > 1/2 lag distance for smoothing
bandh = 9999.9; atol = 22.5                             # no bandwidth, directional variograms
isill = 1                                               # standardize sill
azi_mat = [0,22.5,45,67.5,90,112.5,135,157.5]           # directions in azimuth to consider
```

Let's try running these variograms and visualizing them on separate plots.  I'll demonstrate a method to promgramtically loop over each direction for efficiency (and code brevity).

* we have the direction in the list called 'azi_mat'
* we use the command:
```p
for iazi in range(0,len(azi_mat)):
```
to loop over all the elements in the list with index 'iazi'

* we run the variogram calculation and store the reuslts in 2D arrays, iy is direction, ix is the lag
* we use subplots with the 'iazi' index to add each plot

```p
    plt.subplot(4,2,iazi+1)
```
we add one because the plot index must be $1,\ldots,n$, but arrays / list index as $0,\ldots,n-1$ in Python.

```python
# Arrays to store the results
lag = np.zeros((len(azi_mat),nlag+2)); gamma = np.zeros((len(azi_mat),nlag+2)); npp = np.zeros((len(azi_mat),nlag+2));

for iazi in range(0,len(azi_mat)):                      # Loop over all directions
    lag[iazi,:], gamma[iazi,:], npp[iazi,:] = geostats.gamv(df,"X","Y","NPor",tmin,tmax,lag_dist,lag_tol,nlag,azi_mat[iazi],atol,bandh,isill)
    plt.subplot(4,2,iazi+1)
    plt.plot(lag[iazi,:],gamma[iazi,:],'x',color = 'black',label = 'Azimuth ' +str(azi_mat[iazi]))
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    plt.title('Directional NSCORE Porosity Variogram')
    plt.xlim([0,700])
    plt.ylim([0,1.8])
    plt.legend(loc='upper left')
    plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=4.2, wspace=0.2, hspace=0.3)
plt.show()
```

The directional variograms provide a very clear image of directionality. The ranges vary from 300m, 500m to zonal anisotropy, to 500m, 350m, 280m, 250m and finally 280m.  We are observing the actually spatial continuity ellipse by exploring a variety of directions!

* We can observe that Azimuth 045 is the major direction and Azimuth 135 is the minor direction.

This is a very powerful tool for exploring directionality in spatial datasets.

#### Comments

This was a basic demonstration of spatial data directional continuity analysis. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

I hope this was helpful,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)