## Interactive Variogram Calculation and Modeling Demonstration

### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

### The Interactive Workflow

Here's an interactive workflow for calculating directional experimental variograms in 2D.

* setting the variogram calculation parameters for identifying spatial data pairs

This approach is essential for quantifying spatial continuity with sparsely sampled, irregular spatial data.

I have more comprehensive workflows for variogram calculation:

* [Experimental Variogram Calculation in Python with GeostatsPy](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/GeostatsPy_variogram_calculation.ipynb)

* [Determination of Major and Minor Spatial Continuity Directions in Python with GeostatsPy](https://github.com/GeostatsGuy/PythonNumericalDemos/blob/master/GeostatsPy_spatial_continuity_directions.ipynb)

#### Spatial Continuity

**Spatial Continuity** is the correlation between values over distance.

* No spatial continuity – no correlation between values over distance, random values at each location in space regardless of separation distance.

* Homogenous phenomenon have perfect spatial continuity, since all values as the same (or very similar) they are correlated.

We need a statistic to quantify spatial continuity! A convenient method is the Semivariogram.

#### The Semivariogram

Function of difference over distance.

* The expected (average) squared difference between values separated by a lag distance vector (distance and direction), $h$:

\begin{equation}
\gamma(\textbf{h}) = \frac{1}{2 N(\textbf{h})} \sum^{N(\textbf{h})}_{\alpha=1} (z(\textbf{u}_\alpha) - z(\bf{u}_\alpha + \textbf{h}))^2
\end{equation}

where $z(\textbf{u}_\alpha)$ and $z(\textbf{u}_\alpha + \textbf{h})$ are the spatial sample values at tail and head locations of the lag vector respectively.

* Calculated over a suite of lag distances to obtain a continuous function.

* the $\frac{1}{2}$ term converts a variogram into a semivariogram, but in practice the term variogram is used instead of semivariogram.
* We prefer the semivariogram because it relates directly to the covariance function, $C_x(\textbf{h})$ and univariate variance, $\sigma^2_x$:

\begin{equation}
C_x(\textbf{h}) = \sigma^2_x - \gamma(\textbf{h})
\end{equation}

Note the correlogram is related to the covariance function as:

\begin{equation}
\rho_x(\textbf{h}) = \frac{C_x(\textbf{h})}{\sigma^2_x}
\end{equation}

The correlogram provides of function of the $\textbf{h}-\textbf{h}$ scatter plot correlation vs. lag offset $\textbf{h}$.

\begin{equation}
-1.0 \le \rho_x(\textbf{h}) \le 1.0
\end{equation}

#### Variogram Observations

The following are common observations for variograms that should assist with their practical use.

##### Observation \#1 - As distance increases, variability increase (in general).

This is common since in general, over greater distance offsets, there is often more difference between the head and tail samples.

In some cases, such as with spatial cyclicity of the hole effect variogram model the variogram may have negative slope over somelag distance intervals

Negative slopes at lag distances greater than half the data extent are often caused by too few pairs for a reliable variogram calculation

##### Observation \#2 - Calculated with over all possible pairs separated by lag vector, $\textbf{h}$.

We scan through the entire data set, searching for all possible pair combinations with all other data.  We then calculate the variogram as one half the expectation of squared difference between all pairs.

More pairs results in a more reliable measure.

##### Observation \#3 - Need to plot the sill to know the degree of correlation.

**Sill** is the variance, $\sigma^2_x$

Given stationarity of the variance, $\sigma^2_x$, and variogram $\gamma(\textbf{h})$:

we can define the covariance function:

\begin{equation}
C_x(\textbf{h}) = \sigma^2_x - \gamma(\textbf{h})
\end{equation}

The covariance measure is a measure of similarity over distance (the mirror image of the variogram as shown by the equation above).

Given a standardized distribution $\sigma^2_x = 1.0$, the covariance, $C_x(\textbf{h})$, is equal to the correlogram, $\rho_x(\textbf{h})$:

\begin{equation}
\rho_x(\textbf{h}) = \sigma^2_x - \gamma(\textbf{h})
\end{equation}

##### Observation \#4 - The lag distance at which the variogram reaches the sill is know as the range.

At the range, knowing the data value at the tail location provides no information about a value at the head location of the lag distance vector.

##### Observation \#5 - The nugget effect, a discontinuity at the origin

Sometimes there is a discontinuity in the variogram at distances less than the minimum data spacing.  This is known as **nugget effect**.

The ratio of nugget / sill, is known as relative nugget effect (%). Modeled as a discontinuity with no correlation structure that at lags, $h \gt \epsilon$, an infinitesimal lag distance, and perfect correlation at $\textbf{h} = 0$.
Caution when including nuggect effect in the variogram model as measurement error, mixing populations cause apparent nugget effect

This exercise demonstrates the semivariogram calculation with GeostatsPy. The steps include:

1. generate a 2D model with sequential Gaussian simulation
2. sample from the simulation
3. calculate and visualize experimental semivariograms

#### Variogram Calculation Parameters

The variogram calculation parameters include:

* **azimuth** is the azimuth of the lag vector

* **azimuth tolerance** is the maximum allowable departure from the azimuth (isotropic variograms are calculated with an azimuth tolerance of to 90.0)

* **unit lag distance** the size of the bins in lag distance, usually set to the minimum data spacing

* **lag distance tolerance** - the allowable tolerance in lage distance, commonly set to 50% of unit lag distanceonal smoothing

* **number of lags** - set based on the spatial extent of the dataset, we can typically calculate reliable variograms up to 1/2 the extent of the dataset

* **bandwidth** is the maximum offset allowable from the lag vector

#### Variogram Modeling

Spatial continuity can be modeled with nested, positive definate variogram structures:

\begin{equation}
\Gamma_x(\textbf{h}) = \sum_{i=1}^{nst} \gamma_i(\textbf{h})
\end{equation}

where $\Gamma_x(\textbf{h})$ is the nested variogram model resulting from the summation of $nst$ nested variograms  $\gamma_i(\textbf{h})$.

The types of structure commonly applied include:

* spherical

* exponential

* gaussian

* nugget

Other less common models include:

* hole effect

* dampenned hole effect

* power law

these will not be covered here.

Each one of these variogram structures, $\gamma_i(\textbf{h})$, is based on a geometric anisotropy model parameterized by the orientation and range in the major and minor directions.  In 2D this is simply an azimuth and ranges, $azi$, $a_{maj}$ and $a_{min}$. Note, the range in the minor direction (orthogonal to the major direction).

The geometric anisotropy model assumes that the range in all off-diagonal directions is based on an ellipse with the major and minor axes alligned with and set to the major and minor for the variogram.

\begin{equation}
\textbf{h}_i = \sqrt{\left(\frac{r_{maj}}{a_{maj_i}}\right)^2 + \left(\frac{r_{maj}}{a_{maj_i}}\right)^2}
\end{equation}

Therefore, if we know the major direction, range in major and minor directions, we may completely describe each nested componnent of the complete spatial continuity of the variable of interest, $i = 1,\dots,nst$.

Some comments on modeling nested variograms:

* we can capture nugget, short and long range continuity structures

* we rely on the geometric anisotropy model, so all structures must inform the same level of contribution (porportion of the sill) in all directions.

* the geometric anisotropy model is based on azimuth of the major direction of continuity, range in the major direction and range in the minor direction (orthogonal to the major direction).  The range is interpolated between the major and minor azimuths with a ellipse model

* we can vary the type of variogram, direction or azimuth of the major direction, and major and minor ranges by structure

In this workflow we will explore methods to:

1. detect directionality from a spatial dataset
2. calculate the directional variograms in the major and minor directions
3. build a consistent 2D model fit to the major and minor directions

#### Objective

In the PGE 383: Stochastic Subsurface Modeling class I want to provide hands-on experience with building subsurface modeling workflows. Python provides an excellent vehicle to accomplish this. I have coded a package called GeostatsPy with GSLIB: Geostatistical Library (Deutsch and Journel, 1998) functionality that provides basic building blocks for building subsurface modeling workflows.

The objective is to remove the hurdles of subsurface modeling workflow construction by providing building blocks and sufficient examples. This is not a coding class per se, but we need the ability to 'script' workflows working with numerical methods.

#### Getting Started

Here's the steps to get setup in Python with the GeostatsPy package:

1. Install Anaconda 3 on your machine (https://www.anaconda.com/download/).
2. From Anaconda Navigator (within Anaconda3 group), go to the environment tab, click on base (root) green arrow and open a terminal.
3. In the terminal type: pip install geostatspy.
4. Open Jupyter and in the top block get started by copy and pasting the code block below from this Jupyter Notebook to start using the geostatspy functionality.

You will need to copy the data file to your working directory.  They are available here:

* Tabular data - sample_data.csv at https://git.io/fh4gm.

There are exampled below with these functions. You can go here to see a list of the available functions, https://git.io/fh4eX, other example workflows and source code.

#### Load the required libraries

The following code loads the required libraries.

```python
import geostatspy.GSLIB as GSLIB                       # GSLIB utilies, visualization and wrapper
import geostatspy.geostats as geostats                 # GSLIB methods convert to Python
```

We will also need some standard packages. These should have been installed with Anaconda 3.

```python
%matplotlib inline
import sys                                              # supress output to screen for interactive variogram modeling
import io
import numpy as np                                      # arrays and matrix math
import pandas as pd                                     # DataFrames
import matplotlib.pyplot as plt                         # plotting
from matplotlib.pyplot import cm                        # color maps
from ipywidgets import interactive                      # widgets and interactivity
from ipywidgets import widgets
from ipywidgets import Layout
from ipywidgets import Label
from ipywidgets import VBox, HBox
```

If you get a package import error, you may have to first install some of these packages. This can usually be accomplished by opening up a command window on Windows and then typing 'python -m pip install [package-name]'. More assistance is available with the respective package docs.

#### Loading Tabular Data

Here's the command to load our comma delimited data file in to a Pandas' DataFrame object.

* the data file is [sample_data_MV_biased.csv](https://github.com/GeostatsGuy/GeoDataSets/blob/master/sample_data_MV_biased.csv)

```python
data_url = 'https://raw.githubusercontent.com/GeostatsGuy/GeoDataSets/master/sample_data_MV_biased.csv'
df = pd.read_csv(data_url)
df = df.iloc[:,1:]
df.head()                                               # we could also use this command for a table preview
```

      X
      Y
      Facies
      Porosity
      Perm
      AI

      0
      100.0
      900.0
      0.0
      0.101319
      1.996868
      5590.417154

      1
      100.0
      800.0
      1.0
      0.147676
      10.711789
      3470.845666

      2
      100.0
      700.0
      1.0
      0.145912
      17.818143
      3586.988513

      3
      100.0
      600.0
      1.0
      0.186167
      217.109365
      3732.114787

      4
      100.0
      500.0
      1.0
      0.146088
      16.717367
      2534.551236

The features:

* **X** - x coordinate in meters
* **Y** - y coordinate in meters
* **Porosity** - rock porosity averaged over a specific rock unit from a vertical well
* **Perm** - rock permeability averaged (scaled up) over a specific rock unit from a vertical well
* **AI** - acoustic impedance from a seismic cube assigned at a specific rock unit and at the location of a vertical well
* **facies** - facies, 0 - shale, 1 - sandstone

Concerning facies:

We will work with all facies pooled together. I wanted to simplify this workflow and focus more on spatial continuity direction detection. Finally, by not using facies we do have more samples to support our statistical inference. Most often facies are essential in the subsurface model. Don't worry we will check if this is reasonable in a bit.

You are welcome to repeat this workflow on a by-facies basis.  The following code could be used to build DataFrames ('df_sand' and 'df_shale') for each facies.

```p
df_sand = pd.DataFrame.copy(df[df['Facies'] == 1]).reset_index()  # copy only 'Facies' = sand records
df_shale = pd.DataFrame.copy(df[df['Facies'] == 0]).reset_index() # copy only 'Facies' = shale records
```

Let's look at summary statistics for all facies combined:

```python
df.describe().transpose()                               # summary table of sand only DataFrame statistics
```

      count
      mean
      std
      min
      25%
      50%
      75%
      max

      X
      368.0
      499.565217
      289.770794
      0.000000
      240.000000
      500.000000
      762.500000
      990.000000

      Y
      368.0
      520.644022
      277.412187
      9.000000
      269.000000
      539.000000
      769.000000
      999.000000

      Facies
      368.0
      0.597826
      0.491004
      0.000000
      0.000000
      1.000000
      1.000000
      1.000000

      Porosity
      368.0
      0.127026
      0.030642
      0.041122
      0.103412
      0.125842
      0.148623
      0.210258

      Perm
      368.0
      85.617362
      228.362654
      0.094627
      2.297348
      10.377292
      50.581288
      1991.097723

      AI
      368.0
      4791.736646
      974.560569
      1981.177309
      4110.728374
      4713.325533
      5464.043562
      7561.250336

#### Set the Model Parameters

See the the following model parameters:

* **xmin**, **xmax**, **ymin** and **ymax** - extents of the dataset for plotting
* **feature** and **feature_units** - feature of interest and associated units
* **vmin** and **vmax** - minimum and maximum of the feature of interest

```python
xmin = 0.0; xmax = 1000.0                                # spatial extents in x and y
ymin = 0.0; ymax = 1000.0
feature = 'Porosity'; feature_units = 'percentage'         # name and units of the feature of interest
vmin = 0.0; vmax = 0.25                                  # min and max of the feature of interest
cmap = plt.cm.inferno                                    # set the color map
```

Let's transform the feature to standard normal (mean = 0.0, standard deviation = 1.0, Gaussian shape). This is required for sequential Gaussian simulation (common target for our variogram models) and the Gaussian transform assists with outliers and provides more interpretable variograms.

Let's look at the inputs for the GeostatsPy nscore program.  Note the output include an ndarray with the transformed values (in the same order as the input data in Dataframe 'df' and column 'vcol'), and the transformation table in original values and also in normal score values.

```python
geostats.nscore                                         # see the input parameters required by the nscore function
```

    <function geostatspy.geostats.nscore(df, vcol, wcol=None, ismooth=False, dfsmooth=None, smcol=0, smwcol=0)>

The following command will transform the Porosity and Permeabilty to standard normal.

```python
#Transform to Gaussian by Facies
df['N' + feature], tvPor, tnsPor = geostats.nscore(df, feature) # nscore transform for all facies porosity
```

Let's look at the updated DataFrame to make sure that we now have the normal score porosity and permeability.

```python
df.head()                                               # preview sand DataFrame with nscore transforms
```

      X
      Y
      Facies
      Porosity
      Perm
      AI
      NPorosity

      0
      100.0
      900.0
      0.0
      0.101319
      1.996868
      5590.417154
      -0.749088

      1
      100.0
      800.0
      1.0
      0.147676
      10.711789
      3470.845666
      0.653263

      2
      100.0
      700.0
      1.0
      0.145912
      17.818143
      3586.988513
      0.611663

      3
      100.0
      600.0
      1.0
      0.186167
      217.109365
      3732.114787
      1.993601

      4
      100.0
      500.0
      1.0
      0.146088
      16.717367
      2534.551236
      0.628172

That looks good! One way to check is to see if the relative magnitudes of the normal score transformed values match the original values.  e.g. that the normal score transform of 0.10 porosity normal score is less than the normal score transform of 0.14 porsity.  Also, the normal score transform of values close to the mean value should be close to 0.0

Let's also check the original and transformed sand and shale porosity distributions.

```python
plt.subplot(121)                                        # plot original sand and shale porosity histograms
plt.hist(df[feature], facecolor='red',bins=np.linspace(vmin,vmax,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black')
plt.xlim([vmin,vmax]); plt.ylim([0,1.0])
plt.xlabel(feature + '(' + feature_units + ')'); plt.ylabel('Frequency'); plt.title('Porosity')
plt.grid(True)

plt.subplot(122)
plt.hist(df['N'+feature], facecolor='blue',bins=np.linspace(-3.0,3.0,1000),histtype="stepfilled",alpha=0.2,density=True,cumulative=True,edgecolor='black')
plt.xlim([-3.0,3.0]); plt.ylim([0,1.0])
plt.xlabel('Gaussian Transformed ' + feature); plt.ylabel('Frequency'); plt.title('Guassian Transformed ' + feature)
plt.grid(True)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.2, wspace=0.2, hspace=0.3)
plt.show()
```

We can see that the normal score transform has correctly transformed the feature to standard normal.

#### Inspection of Posted Data

Data visualization is very useful to detect patterns. Our brains are very good at pattern detection. I promote quantitative methods and recognize issues with cognitive bias, but it is important to recognize the value is expert intepretation based on data visualization.

* This data visualization will also be important to assist with parameter selection for the variogram calculation search template.

Let's plot the location maps of the original feature and the normal score transforms of the feature.

```python
plt.subplot(121)                                        # location map of normal score transform of porosity
GSLIB.locmap_st(df,'X','Y',feature,0,1000,0,1000,vmin,vmax,feature,'X (m)','Y (m)',feature,cmap)

plt.subplot(122)                                        # location map of normal score transform of porosity
GSLIB.locmap_st(df,'X','Y','N'+feature,0,1000,0,1000,-3,3,'Gaussian Transformed ' + feature,'X (m)','Y (m)','Gaussian Transformed ' + feature,cmap)

plt.subplots_adjust(left=0.0, bottom=0.0, right=2.0, top=1.0, wspace=0.5, hspace=0.3)
plt.show()
```

What do you see?  Here's my observations:

* there is a high degree of spatial agreement between porosity and permeability, this is supported by the high correlation evident in the cross plot.
* there are no discontinuities that could suggest that facies represent a distinct change, rather the porosity and permeability seem continuous and the assigned facies are a truncation of their continous behavoir, we doing 'ok' with no facies
* suspect a 045 azimuth major direction of continuity (up - right)
* there may be cycles in the 135 azimuth
* there will not likely be a nugget effect, but there is an hint of some short scale discontinuity?

**Do you agree?** If you have a different observations, drop me a line at mpyrcz@austin.utexas.edu and I'll add to this lesson with credit!

#### Experimental Variograms

We can use the location maps to help determine good variogram calculation parameters. For example:

```p
tmin = -9999.; tmax = 9999.;
lag_dist = 100.0; lag_tol = 50.0; nlag = 7; bandh = 9999.9; azi = azi; atol = 22.5; isill = 1
```
* **tmin**, **tmax** are trimming limits - set to have no impact, no need to filter the data
* **lag_dist**, **lag_tol** are the lag distance, lag tolerance - set based on the common data spacing (100m) and tolerance as 100% of lag distance for additonal smoothing
* **nlag** is number of lags - set to extend just past 50 of the data extent
* **bandh** is the horizontal band width - set to have no effect
* **azi** is the azimuth -  it has not effect since we set atol, the azimuth tolerance, to 90.0
* **isill** is a boolean to standardize the distribution to a variance of 1 - it has no effect since the previous nscore transform sets the variance to 1.0

#### Dashboard for Interactive Variogram Calculation and Modeling

Below we make a dashboard with the ipywidgets and matplotlib Python packages for calculating and modeling experimental variograms.

* allowing you to calculate and model the variogram of the normal score transformed variogram interactively while changing (and exploring) the search template parameters.

* first calculate the isotropic or directional variogram(s)

* then fit the same isotropic or directional variogram(s)

```python
# interactive calculation of the experimental variogram
l = widgets.Text(value='                              Variogram Calculation Interactive Demonstration, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
lag = widgets.FloatSlider(min = 10, max = 500, value = 10, step = 10, description = 'lag',orientation='vertical',layout=Layout(width='90px', height='200px'))
lag.style.handle_color = 'gray'

lag_tol = widgets.FloatSlider(min = 5, max = 500, value = 5, step = 10, description = 'lag tolerance',orientation='vertical',layout=Layout(width='90px', height='200px'))
lag_tol.style.handle_color = 'gray'

nlag = widgets.IntSlider(min = 1, max = 100, value = 100, step = 1, description = 'number of lags',orientation='vertical',layout=Layout(width='90px', height='200px'))
nlag.style.handle_color = 'gray'

azi = widgets.FloatSlider(min = 0, max = 360, value = 0, step = 5, description = 'azimuth',orientation='vertical',layout=Layout(width='90px', height='200px'))
azi.style.handle_color = 'gray'

azi_tol = widgets.FloatSlider(min = 10, max = 90, value = 10, step = 5, description = 'azimuth tolerance',orientation='vertical',layout=Layout(width='120px', height='200px'))
azi_tol.style.handle_color = 'gray'

bandwidth = widgets.FloatSlider(min = 100, max = 2000, value = 2000, step = 100, description = 'bandwidth',orientation='vertical',layout=Layout(width='90px', height='200px'))
azi_tol.style.handle_color = 'gray'

ui1 = widgets.HBox([lag,lag_tol,nlag,azi,azi_tol,bandwidth],) # basic widget formatting
ui = widgets.VBox([l,ui1],)

def f_make(lag,lag_tol,nlag,azi,azi_tol,bandwidth):     # function to take parameters, calculate variogram and plot
#    text_trap = io.StringIO()
#    sys.stdout = text_trap
    global lags,gammas,npps,lags2,gammas2,npps2
    tmin = -9999.9; tmax = 9999.9
    lags, gammas, npps = geostats.gamv(df,"X","Y","N"+feature,tmin,tmax,lag,lag_tol,nlag,azi,azi_tol,bandwidth,isill=1.0)
    lags2, gammas2, npps2 = geostats.gamv(df,"X","Y","N"+feature,tmin,tmax,lag,lag_tol,nlag,azi+90.0,azi_tol,bandwidth,isill=1.0)

    plt.subplot(111)                                    # plot experimental variogram
    plt.scatter(lags,gammas,color = 'black',s = npps*0.03,label = 'Major Azimuth ' +str(azi), alpha = 0.8)
    plt.scatter(lags2,gammas2,color = 'red',s = npps*0.03,label = 'Minor Azimuth ' +str(azi+90.0), alpha = 0.8)
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    if azi_tol < 90.0:
        plt.title('Directional NSCORE ' + feature + ' Variogram - Azi. ' + str(azi) + ', Azi. Tol.' + str(azi_tol))
    else:
        plt.title('Omni Directional NSCORE ' + feature + ' Variogram ')
    plt.xlim([0,1000]); plt.ylim([0,1.8])
    plt.legend(loc="lower right")
    plt.grid(True)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=1.5, top=1.0, wspace=0.3, hspace=0.3)
    plt.show()

    return

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'lag':lag,'lag_tol':lag_tol,'nlag':nlag,'azi':azi,'azi_tol':azi_tol,'bandwidth':bandwidth})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Variogram Calculation Demonstration

* calculate omnidirectional and direction experimental variograms

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Calculate interpretable experimental variograms for sparse, irregularly-space spatial data. Note, size of the experimental point is scaled by the number of pairs.

* **azimuth** is the azimuth of the lag vector

* **azimuth tolerance** is the maximum allowable departure from the azimuth

* **unit lag distance** the size of the bins in lag distance

* **lag distance tolerance** - the allowable tolerance in lage distance

* **number of lags** - number of lags in the experimental variogram

* **bandwidth** - maximum departure from the lag vector

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                              Variogram Calculation Interactive Demonstration, Mich…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

```python
# interactive calculation of the sample set (control of source parametric distribution and number of samples)
l = widgets.Text(value='                              Variogram Modeling, Michael Pyrcz, Associate Professor, The University of Texas at Austin',layout=Layout(width='950px', height='30px'))
nug = widgets.FloatSlider(min = 0, max = 1.0, value = 0.0, step = 0.1, description = 'nug',orientation='vertical',layout=Layout(width='60px', height='200px'))
nug.style.handle_color = 'gray'
it1 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type1:',disabled=False,layout=Layout(width='200px', height='30px'))
c1 = widgets.FloatSlider(min=0.0, max = 1.0, value = 0.1, description = 'c1',orientation='vertical',layout=Layout(width='60px', height='200px'))
c1.style.handle_color = 'gray'
hmaj1 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 0.01, step = 25.0, description = 'hmaj1',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmaj1.style.handle_color = 'black'
hmin1 = widgets.FloatSlider(min = 0.01, max = 10000.0, value = 0.01, step = 25.0, description = 'hmin1',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmin1.style.handle_color = 'red'

it2 = widgets.Dropdown(options=['Spherical', 'Exponential', 'Gaussian'],value='Spherical',
    description='Type2:',disabled=False,layout=Layout(width='200px', height='30px'))
c2 = widgets.FloatSlider(min=0.0, max = 1.0, value = 0.0, description = 'c2',orientation='vertical',layout=Layout(width='60px', height='200px'))
c2.style.handle_color = 'gray'
hmaj2 = widgets.FloatSlider(min=0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmaj2',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmaj2.style.handle_color = 'black'
hmin2 = widgets.FloatSlider(min = 0.01, max = 10000.0, value = 0.01, step = 100.0, description = 'hmin2',orientation='vertical',layout=Layout(width='60px', height='200px'))
hmin2.style.handle_color = 'red'

ui1 = widgets.HBox([nug,it1,c1,hmaj1,hmin1,it2,c2,hmaj2,hmin2],)                   # basic widget formatting
#ui2 = widgets.HBox([it2,c2,hmaj2,hmin2],)                   # basic widget formatting
ui = widgets.VBox([l,ui1],)

def convert_type(it):
    if it == 'Spherical':
        return 1
    elif it == 'Exponential':
        return 2
    else:
        return 3

def f_make(nug,it1,c1, hmaj1, hmin1, it2, c2, hmaj2, hmin2):                       # function to take parameters, make sample and plot
    text_trap = io.StringIO()
    sys.stdout = text_trap

    it1 = convert_type(it1); it2 = convert_type(it2)
    if c2 > 0.0:
        nst = 2
    else:
        nst = 1

    vario = GSLIB.make_variogram(nug,nst,it1,c1,0.0,hmaj1,hmin1,it2,c2,0.0,hmaj2,hmin2) # make model object
    nlag = 100; xlag = 10;
    index_maj,h_maj,gam_maj,cov_maj,ro_maj = geostats.vmodel(nlag,xlag,0.0,vario)   # project the model in the major azimuth                                                  # project the model in the 135 azimuth
    index_min,h_min,gam_min,cov_min,ro_min = geostats.vmodel(nlag,xlag,90.0,vario)

    plt.subplot(111)                                    # plot experimental variogram
    plt.scatter(lags,gammas,color = 'black',s = npps*0.03,label = 'Major Azimuth ' +str(azi.value), alpha = 0.8)
    plt.plot(h_maj,gam_maj,color = 'black')
    plt.scatter(lags2,gammas2,color = 'red',s = npps*0.03,label = 'Minor Azimuth ' +str(azi.value+90.0), alpha = 0.8)
    plt.plot(h_min,gam_min,color = 'red')
    plt.plot([0,2000],[1.0,1.0],color = 'black')
    plt.xlabel(r'Lag Distance $\bf(h)$, (m)')
    plt.ylabel(r'$\gamma \bf(h)$')
    if azi_tol.value < 90.0:
        plt.title('Directional NSCORE ' + feature + ' Variogram - Azi. ' + str(azi.value) + ', Azi. Tol.' + str(azi_tol.value))
    else:
        plt.title('Omni Directional NSCORE ' + feature + ' Variogram ')
    plt.xlim([0,1000]); plt.ylim([0,1.8])
    plt.legend(loc="lower right")
    plt.grid(True)

    plt.subplots_adjust(left=0.0, bottom=0.0, right=2.2, top=1.5, wspace=0.3, hspace=0.3)
    plt.show()

# connect the function to make the samples and plot to the widgets
interactive_plot = widgets.interactive_output(f_make, {'nug':nug, 'it1':it1,'c1':c1, 'hmaj1':hmaj1, 'hmin1':hmin1, 'it2':it2, 'c2':c2, 'hmaj2':hmaj2, 'hmin2':hmin2})
interactive_plot.clear_output(wait = True)               # reduce flickering by delaying plot updating
```

### Interactive Nested Variogram Modeling Demostration

* select the nested structures and their types, contributions and major and minor ranges

#### Michael Pyrcz, Associate Professor, University of Texas at Austin

##### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1) | [GeostatsPy](https://github.com/GeostatsGuy/GeostatsPy)

### The Problem

Fit a positive definite variogram model based on the addition of multiple structures each describing spatial components of the feature variance

* **nug**: nugget effect

* **c1 / c2**: contributions of the sill

* **hmaj1 / hmaj2**: range in the major direction

* **hmin1 / hmin2**: range in the minor direction

```python
display(ui, interactive_plot)                           # display the interactive plot
```

    VBox(children=(Text(value='                              Variogram Modeling, Michael Pyrcz, Associate Professo…

    Output(outputs=({'output_type': 'display_data', 'data': {'text/plain': '
', 'i…

#### Comments

This was a basic demonstration / exercise of variogram calculation and modeling for spatial continuity analysis. Much more could be done, I have other demonstrations on the basics of working with DataFrames, ndarrays, univariate statistics, plotting data, declustering, data transformations and many other workflows available at https://github.com/GeostatsGuy/PythonNumericalDemos and https://github.com/GeostatsGuy/GeostatsPy.

#### The Author:

### Michael Pyrcz, Associate Professor, University of Texas at Austin
*Novel Data Analytics, Geostatistics and Machine Learning Subsurface Solutions*

With over 17 years of experience in subsurface consulting, research and development, Michael has returned to academia driven by his passion for teaching and enthusiasm for enhancing engineers' and geoscientists' impact in subsurface resource development.

For more about Michael check out these links:

#### [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

#### Want to Work Together?

I hope this content is helpful to those that want to learn more about subsurface modeling, data analytics and machine learning. Students and working professionals are welcome to participate.

* Want to invite me to visit your company for training, mentoring, project review, workflow design and / or consulting? I'd be happy to drop by and work with you!

* Interested in partnering, supporting my graduate student research or my Subsurface Data Analytics and Machine Learning consortium (co-PIs including Profs. Foster, Torres-Verdin and van Oort)? My research combines data analytics, stochastic modeling and machine learning theory with practice to develop novel methods and workflows to add value. We are solving challenging subsurface problems!

* I can be reached at mpyrcz@austin.utexas.edu.

I'm always happy to discuss,

*Michael*

Michael Pyrcz, Ph.D., P.Eng. Associate Professor The Hildebrand Department of Petroleum and Geosystems Engineering, Bureau of Economic Geology, The Jackson School of Geosciences, The University of Texas at Austin

#### More Resources Available at: [Twitter](https://twitter.com/geostatsguy) | [GitHub](https://github.com/GeostatsGuy) | [Website](http://michaelpyrcz.com) | [GoogleScholar](https://scholar.google.com/citations?user=QVZ20eQAAAAJ&hl=en&oi=ao) | [Book](https://www.amazon.com/Geostatistical-Reservoir-Modeling-Michael-Pyrcz/dp/0199731446) | [YouTube](https://www.youtube.com/channel/UCLqEr-xV-ceHdXXXrTId5ig)  | [LinkedIn](https://www.linkedin.com/in/michael-pyrcz-61a648a1)

```python

```


Text provided under a Creative Commons Attribution license, CC-BY. All code is made available under the FSF-approved MIT license. (c) Yves Dubief, 2016. NSF for support via NSF-CBET award #1258697.

Only submissions in html format are accepted. Replace NetID by your NetID in the cell below and run the cell. DO NOT CHANGE THE NAME OF THIS NOTEBOOK. If characters have been added during download, rename the file to its original name. Do not modify the last cell, and don't forget to run it.

Thanks!

The following cell should always be the first coding cell of your python notebooks

```python
"""
importing the necessary libraries, do not modify
"""
%matplotlib inline
# plots graphs within the notebook
%config InlineBackend.figure_format='svg' # not sure what this does, may be default images to svg format

from IPython.display import display,Image, Latex, Math
from __future__ import division
from sympy.interactive import printing
printing.init_printing(use_latex='mathjax')

import time
from IPython.display import clear_output

import SchemDraw as schem
import SchemDraw.elements as e

import matplotlib.pyplot as plt
import numpy as np
import math
import scipy.constants as sc

import sympy as sym

from IPython.core.display import HTML
def header(text):
    raw_html = '<h4>' + str(text) + '</h4>'
    return raw_html

def box(text):
    raw_html = '<div style="border:1px dotted black;padding:2em;">'+str(text)+'</div>'
    return HTML(raw_html)

def nobox(text):
    raw_html = '<p>'+str(text)+'</p>'
    return HTML(raw_html)

def addContent(raw_html):
    global htmlContent
    htmlContent += raw_html

class PDF(object):
  def __init__(self, pdf, size=(200,200)):
    self.pdf = pdf
    self.size = size

  def _repr_html_(self):
    return ''.format(self.pdf, self.size)

  def _repr_latex_(self):
    return r'\includegraphics[width=1.0\textwidth]{{{0}}}'.format(self.pdf)

class ListTable(list):
    """ Overridden list class which takes a 2-dimensional list of
        the form [[1,2,3],[4,5,6]], and renders an HTML Table in
        IPython Notebook. """

    def _repr_html_(self):
        html = ["<table>"]
        for row in self:
            html.append("<tr>")

            for col in row:
                html.append("<td>{0}</td>".format(col))

            html.append("</tr>")
        html.append("</table>")
        return ''.join(html)

font = {'family' : 'serif',
        'color'  : 'black',
        'weight' : 'normal',
        'size'   : 18,
        }

from scipy.constants import convert_temperature
def C2K(T):
    return convert_temperature(T,'Celsius','Kelvin')
def C2F(T):
    return convert_temperature(T,'Celsius','Fahrenheit')
def F2K(T):
    return convert_temperature(T,'Fahrenheit','Kelvin')
def F2C(T):
    return convert_temperature(T,'Fahrenheit','Celsius')
def K2F(T):
    return convert_temperature(T,'Kelvin','Fahrenheit')
def K2C(T):
    return convert_temperature(T,'Kelvin','Celsius')

time_stamp = np.zeros(10)
time_stamp[0] = time.time()
```

Problem 7.63

Consider two very long, straight fins of uniform cross section, as shown in Figure 3.17 of your book. The rectangular fin has dimensions $t=1\text{mm}$ and $w=20\text{mm}$. The circular pin fin has the same cross-sectional area as the rectangular fin. Both fins are constructed of aluminum with $k=237\text{W/m.K}$. In both cases, the base temperature is $T_b=85^\circ\text{􏰠C}$. Airflow is directed as shown in the figure, with $T_\infty=20^\circ\text{􏰠C}$ and $u_\infty=5\text{m/s}$.

(a) Calculate the heat loss from each fin. Assume that the heat transfer coefficient on the edges of the rectangular fin is equal to the average value on the upper and lower surfaces.

(b) What diameter cylindrical fin would be needed to provide the same fin heat transfer rate as the rectangular cross-section fin?

```python
from NewLibraries import HT_conduction_extended_surfaces as condext
from NewLibraries import HT_external_convection as extconv

```

Rectangular fin

Heat transfer is in steady state. The surface temperature is assumed to be close to the base temperature $T_b$.
We assume uniform heat transfer rate over the surface the fin.

```python
# Parameters:

U_infty = 5 #m/s
T_infty = 20. #C
T_b = 85. #C
w = 20e-3 #m
t = 1e-3 #m

# film temperature
T_f = (T_b+T_infty)/2.

from Libraries import thermodynamics as thermo

air = thermo.Fluid('air')
air.get_properties(C2K(T_f))

Re = U_infty*w/air.nu

print("End plate Reynolds number: %5.0f" %Re)
#?extconv.FlatPlate
rect_fin = extconv.FlatPlate("laminar","isothermal",U_infty,air.nu,air.alpha,w,0.,5.0e5)
rect_fin.average(w)
print("Average Nu: %4.1f" %rect_fin.Nu_ave)

h_rect_fin = rect_fin.Nu_ave*air.k/w
print("Average convection heat transfer coefficient: %2.1f W/m^2.K" %h_rect_fin)

# Solving for heat transfer over extended surfaces
P = 2*(w+t) #Perimeter
A_c = w*t # Cross sectional area
theta_b = T_b - T_infty
k_alum = 237. #W/mk

M = np.sqrt(h_rect_fin*P*A_c*k_alum)*theta_b
q_f = M

print("Heat transfer for an infinite fin: %2.2f W" %q_f)
```

    End plate Reynolds number:  5502
    Average Nu: 44.0
    Average convection heat transfer coefficient: 61.5 W/m^2.K
    Heat transfer for an infinite fin: 7.19 W

Problem 7.41

Consider atmospheric air at $u_\infty=2\text{m/s}$ and $T_\infty=300\text{K}$ in parallel flow over an isothermal flat plate of length $L=\text{m}$ and temperature $T_s=350\text{K}$.

 Compute the local convection coefficient at the leading and trailing edges of the heated plate with or without an unheated starting length of $\xi=1\text{m}$.

Compute the average convection coefficient for the plate for the same condition as part (1)

Plot the variation of the loca convection coefficient over the plate with and without unheated starting length.

 Assumptions

The flow and heat transfer are steady. Radiation is negligible.

 (1)

First you need to calculate the Reynolds number to figure out if the regime is laminar or turbulent. There are two systems, one $1\text{m}$ long, the other $1\text{m}$ long. You can either calculate the Reynolds number or make an assumption on the regime ("laminar" or "mixed").

```python
from NewLibraries import HT_external_convection as extconv
from NewLibraries import thermodynamics as thermo
u_infty = 2. #m/S
T_infty = 300. #K
T_s = 350. #K
T_f = (T_s+T_infty)/2.
L_2m = 2.
xi_2m = 1. #m
L_1m = 1. #m
flow = thermo.Fluid("air")
flow.get_properties(T_f)
?extconv.FlatPlate
# 2m plate
plate_2m=extconv.FlatPlate("laminar",'unheated starting length',u_infty,flow.nu,flow.alpha,L_2m,xi = xi_2m)
print("Trailing edge Reynolds number for 2m plate: %1.3e " %plate_2m.Re_L)
# 1m plate
plate_1m=extconv.FlatPlate("laminar",'isothermal',u_infty,flow.nu,flow.alpha,L_1m)
print("Trailing edge Reynolds number for 1m plate: %1.3e " %plate_1m.Re_L)
plate_2m.local(L_2m)

```

    Trailing edge Reynolds number for 2m plate: 2.208e+05
    Trailing edge Reynolds number for 1m plate: 1.104e+05

The regime is indeed laminar. The local convection coefficient may now be determined using, for the $1\text{m}$ plate,
$$
Nu_x=\frac{h_x x}{k}=0.332Re_x^{1/2}Pr^{1/3}
$$
and for the $2\text{m}$ plate:
$$
Nu_x=\frac{\left.Nu_x\right\vert_{\xi=0}}{\left[1-(\xi/x)^{3/4}\right]^{1/3}}
$$
At the leading edge of the heated plate, the temperature profile jumps from $T_\infty$ in the flow to $T_s$ at the wall. Since there is no flow at the wall, Fourier's law applies:
$$
q_x'' = \left.-k\frac{\partial T}{\partial y}\right\vert_{x,y=0}=h_x(T_s-T_\infty)
$$
The jump results in $\left.\partial T/\partial y\right\vert_{x,y=0}\rightarrow\infty$, therefore at the leading edge of either heated plate:
$$
h_{x=\text{leading edge}}=\infty
$$

```python
# 2m plate
plate_2m.local(L_2m)
hx_2m = plate_2m.Nu_x*flow.k/L_2m
print("Local convection coefficient at the trailing edge for 2m plate: %1.2f W/m^.K" %hx_2m)

# 1m plate

plate_1m.local(L_1m)
hx_1m = plate_1m.Nu_x*flow.k/L_1m
print("Local convection coefficient at the trailing edge for 1m plate: %1.2f W/m^.K" %hx_1m)

```

    Local convection coefficient at the trailing edge for 2m plate: 2.63 W/m^.K
    Local convection coefficient at the trailing edge for 1m plate: 2.75 W/m^.K

(2)

Similarly, the average coefficients, calculated from the correlations for the average Nusselt number $\overline{Nu}$, are:

```python
# 2m plate
plate_2m.average(L_2m)
have_2m = plate_2m.Nu_ave*flow.k/L_2m
print("Average convection coefficient at the trailing edge for 2m plate: %1.2f W/m^.K" %have_2m)

# 1m plate

plate_1m.average(L_1m)
have_1m = plate_1m.Nu_ave*flow.k/L_1m
print("Average convection coefficient at the trailing edge for 1m plate: %1.2f W/m^.K" %have_1m)

```

    Average convection coefficient at the trailing edge for 2m plate: 4.26 W/m^.K
    Average convection coefficient at the trailing edge for 1m plate: 5.50 W/m^.K

(3)

To plot the variation of the local convection coefficient, I used a logarithmic distribution of points on the plate rather than a linear one in order to better capture the power-law variation of $h_x$. It also avoids the leading edge for which $h_x=\infty$. The two distributions are plotted with the $x$ origin at the leading edge of the heated plate.

```python
N = 50
x_1m = np.logspace(-2,0,N)
x_2m = xi_2m+x_1m
h_x_1m = np.zeros(N)
h_x_2m = np.zeros(N)
for i in range(N):
    plate_1m.local(x_1m[i])
    h_x_1m[i] = plate_1m.Nu_x*flow.k/x_1m[i]
    plate_2m.local(x_2m[i])
    h_x_2m[i] = plate_2m.Nu_x*flow.k/x_2m[i]
plt.plot(x_1m, h_x_1m, lw=2, label="Without unheated starting length")
plt.plot(x_1m, h_x_2m, 'r--', lw=2, label="With unheated starting length")
plt.legend(loc=3, bbox_to_anchor=[0, 1],
           ncol=1, shadow=True, fancybox=True)
plt.ylabel(r'$h_x}$ ($W/m^2K$)', fontdict = font)
plt.xlabel(r'$x$ (m)', fontdict = font)
plt.grid(True)
plt.show()
```

Problem 7.48

An $L=􏰡1\text{-m}$-long vertical copper tube of inner diameter $D_i=20\text{mm}$ and wall thickness $t=2\text{mm}$ contains liquid water at $T_w\approx 0^\circ\text{􏰠C}$. On a winter day, air at $V_\infty=􏰡3\text{m/s}$, $T_\infty=􏰝􏰡-20^\circ\text{􏰠C}$ is in crossflow over the tube.

 Determine the heat loss per unit mass from the water (W/kg) when the tube is full of water.

Determine the heat loss from the water (W/kg) when the tube is half full.

(1)

Assumptions

Steady-state conditions, inner wall is at $T_w\approx 0^\circ\text{􏰠C}$ and negligible radiation.

Convection coefficient

Regardless of the configuration, we need the average convection coefficient from the tube to the outside air. Although not entirely accurate, the air film temperature is estimated at:
$$
T_f=\frac{T_w+T_\infty}{2}=\frac{0^\circ\text{C}-20^\circ\text{C}}{2}
$$

```python
from NewLibraries import HT_external_convection as extconv
from NewLibraries import thermodynamics as thermo
T_w = 0.
T_infty = -20.
T_f = -10.
V_infty = 3. #m/s
D_o = 24e-3 #m
D_i = 20e-3 #m
L = 1. #m
air = thermo.Fluid("air")
air.get_properties(C2K(T_f))
Re_o = V_infty*D_o/air.nu

print("Re_o=%3.0f" %Re_o)
?extconv.CircularCylinder

tubeHT = extconv.CircularCylinder('Churchill-Bernstein',Re_o,air.Pr)

h_o = tubeHT.Nu*air.k/D_o
print("Average convection coefficient h_o=%2.2f W/m^2.K (using Churchill-Bernstein)" %h_o)
```

    Re_o=5737
    Average convection coefficient h_o=38.92 W/m^2.K (using Churchill-Bernstein)

The heat rate is readily available from the thermal resistance analysis of the problem

```python
from NewLibraries import HT_thermal_resistance as res
R = []
R.append(res.Resistance(r"$\frac{\ln(D_o/D_i)}{2\pi k L}$","W"))
R.append(res.Resistance(r"$\frac{1}{h_oA_o}$","W"))

#?res.Resistance
k_copper = 401. #W/m.K
R[0].conduction("cylindrical", k_copper, L_pipe = L, r_a = D_i/2., r_b = D_o/2.)
R[1].convection(h_o,np.pi*D_o*L)
d = schem.Drawing()
d.add( e.DOT, label = r"$T_m$")
d.add( e.RES, d = "right", botlabel = R[0].name)
d.add( e.DOT)
d.add( e.RES, d = "right", botlabel = R[1].name)
d.add( e.DOT, label = r"$T_\infty$")
Flux0 = d.add( e.LINE, lftlabel = r"$q_1$", endpts = [[-2.25, 0], [-0.25, 0]])
d.labelI(Flux0, arrowofst = 0)
d.draw()
R_tot = res.serial_sum(R,0,1)
```

The heat rate is
$$
q_1 = \frac{1}{R_{tot}}\left(T_w-T_\infty\right),
$$
and the mass of water contained in the copper tube is
$$
M_1 = \rho_w\frac{\pi}{4}D_i^2L,
$$
where $\rho_w\approx 1000\text{kg/m$^3$}$. Hence the heat loss per unit mass is
$$q_1/M_1.$$

```python
q_1 = (T_w - T_infty)/R_tot
print("Heat rate for pipe full of water: %2.2f W" %q_1)
rho_w = 1000. #kg/m^3
M_1 = (np.pi/4.)*D_i**2*L*rho_w
q_M1 = q_1/M_1
print("Heat loss per unit mass: %2.2f W/kg" %q_M1)
```

    Heat rate for pipe full of water: 58.68 W
    Heat loss per unit mass: 186.79 W/kg

(2)

When the pipe is half full, the upper part acts as a fin and the heat rate is the combination of the convection heat rate and the fin heat rate. The heat loss of the lower section is half that of the tube full of water but so is the mass of water. Therefore the heat loss per unit mass in the lower half is the same as before $q_1/M_1$

 The cross sectional area of the fin is
$$
A_c = \frac{\pi}{4}\left(D_o^2-D_i^2\right)
$$
and the perimeter is
$$
P=\pi D_o.
$$
The convection coefficient is $h_o$ calculated before.

```python
from NewLibraries import HT_conduction_extended_surfaces as condext
A_c = (np.pi/4.)*(D_o**2-D_i**2)
P_c = np.pi*D_o
T_b = T_w
L_fin = 0.5 #m

fin = condext.ExtSurfaces(T_b,T_infty,T_infty,k_copper,h_o,P_c,A_c,L_fin)
fin.heat_rate('infinite')
print("Heat loss through the fin: %2.2f W" %fin.q_f)
q_Mfin = fin.q_f/(M_1/2.)
print("Total heat loss per unit mass of water: %3.2f + %2.2f = %3.2f W" %(q_M1,q_Mfin,q_M1+q_Mfin))
```

    Heat loss through the fin: 8.07 W
    Total heat loss per unit mass of water: 186.79 + 51.35 = 238.15 W

Problem 7.56

Hot water at $50^\circ\text{C}$ is routed from one building in which it is generated to an adjoining building in which it is used for space heating. Transfer between the buildings occurs in a steel pipe ($k=􏰡60\text{W/m􏰘K}$) of $100\text{-mm}$ outside diameter and $8\text{-mm}$ wall thickness. During the winter, representative environmental conditions involve air at $T_\infty=􏰝-􏰕5^\circ\text{􏰠C}$ and $V_\infty=􏰡3\text{m/s}$ in cross flow over the pipe.

If the cost of producing the hot water is $\$0.10$ per $\text{kW.h}$, what is the representative daily cost of heat loss from an uninsulated pipe to the air per meter of pipe length? The convection resistance associated with water flow in the pipe may be neglected.

Determine the savings associated with application of a $10\text{-mm}$-thick coating of urethane insulation ($k=0.026\text{W/m.K}$) to the outer surface of the pipe.

```python
T_infty = -5. #C
V_infty = 3. #m/s
T_water = 50. #C
D_i_pipe = 84e-3 #m
D_o_pipe = 100e-3 #m
k_pipe = 60 #W/m.K

R = []
R.append(res.Resistance(r"$\frac{\ln(D_o/D_i)}{2\pi k_{pipe}}$","W/m"))
R.append(res.Resistance(r"$\frac{1}{h_o\pi D_o}$","W/m"))
d = schem.Drawing()
d.add( e.DOT, label=r"$T_{water}$")
d.add( e.RES, d = 'right', botlabel=R[0].name)
d.add( e.DOT)
d.add( e.RES, d = 'right', botlabel=R[1].name)
d.add( e.DOT, label = r"$T_\infty$")
Flux0 = d.add( e.LINE, lftlabel = r"$q_1$", endpts = [[-2.25, 0], [-0.25, 0]])
d.labelI(Flux0, arrowofst = 0)
d.draw()
R[0].conduction("cylindrical", k = k_pipe, r_a = D_i_pipe/2., r_b = D_o_pipe/2.)

```

```python

#first guess: T_s = T_water
T_f = (T_water+T_infty)/2.
air = thermo.Fluid("air")
air.get_properties(C2K(T_f))
print("First film temperature guess: %3.2f K" %C2K(T_f))
Re = V_infty*D_o_pipe/air.nu
print("Reynolds number: %5.0f"%Re)
#?extconv.CircularCylinder
air_conv = extconv.CircularCylinder('Churchill-Bernstein',Re,air.Pr)
print("Nu = %2.2f" %air_conv.Nu)
h_o = air_conv.Nu*air.k/D_o_pipe

print("Convection coefficient %2.2f W/m^2.K" %h_o)
R[1].convection(h_o,np.pi*D_o_pipe)
q_1 = (T_water-T_infty)/res.serial_sum(R,0,1)
print("Heat loss per unit pipe length: q'=%3.0f W" %q_1)
T_s = T_water - q_1*R[0].R
print("Surface temperature: T_s = %2.1f C"%T_s)
```

    First film temperature guess: 295.65 K
    Reynolds number: 19554
    Nu = 78.36
    Convection coefficient 20.27 W/m^2.K
    Heat loss per unit pipe length: q'=349 W
    Surface temperature: T_s = 49.8 C

```python
Cost = q_1 * 0.1e-3 * 24.
print("Daily cost: $%1.3f /m.d" %Cost)
```

    Daily cost: $0.838 /m.d

```python
D_o_ins = 120e-3 #m
k_ins = 0.026 #W/m.K
R2 = []
R2.append(res.Resistance(r"$\frac{\ln(D_{o,p}/D_{i,p})}{2\pi k_{pipe}}$","W/m"))
R2.append(res.Resistance(r"$\frac{\ln(D_{o,ins}/D_{o,p})}{2\pi k_{ins}}$","W/m"))
R2.append(res.Resistance(r"$\frac{1}{h_o\pi D_{o.ins}}$","W/m"))

d = schem.Drawing()
d.add( e.DOT, label=r"$T_{water}$")
d.add( e.RES, d = 'right', botlabel=R2[0].name)
d.add( e.DOT)
d.add( e.RES, d = 'right', botlabel=R2[1].name)
d.add( e.DOT)
d.add( e.RES, d = 'right', botlabel=R2[2].name)
d.add( e.DOT, label = r"$T_\infty$")
Flux0 = d.add( e.LINE, lftlabel = r"$q_2$", endpts = [[-2.25, 0], [-0.25, 0]])
d.labelI(Flux0, arrowofst = 0)
d.draw()
R2[0].conduction("cylindrical", k = k_pipe, r_a = D_i_pipe/2., r_b = D_o_pipe/2.)
R2[0].conduction("cylindrical", k = k_ins, r_a = D_o_pipe/2., r_b = D_o_ins/2.)
```

```python
#first guess: T_s = T_water
T_f = (T_water+T_infty)/2.
air = thermo.Fluid("air")
air.get_properties(C2K(T_f))
print("First film temperature guess: %3.2f K" %C2K(T_f))
Re = V_infty*D_o_pipe/air.nu
print("Reynolds number: %5.0f"%Re)
#?extconv.CircularCylinder
air_conv = extconv.CircularCylinder('Churchill-Bernstein',Re,air.Pr)
print("Nu = %2.2f" %air_conv.Nu)
h_o = air_conv.Nu*air.k/D_o_pipe

print("Convection coefficient %2.2f W/m^2.K" %h_o)
R2[2].convection(h_o,np.pi*D_o_ins)
q_2 = (T_water-T_infty)/res.serial_sum(R,0,1)
print("Heat loss per unit pipe length: q'=%3.1f W" %q_2)
T_s = T_infty + q_2*R2[2].R
print("Surface temperature: T_s = %2.1f C"%T_s)
```

    First film temperature guess: 295.65 K
    Reynolds number: 19554
    Nu = 78.36
    Convection coefficient 20.27 W/m^2.K
    Heat loss per unit pipe length: q'=349.3 W
    Surface temperature: T_s = 40.7 C

```python
#second guess: T_s = 0.7
T_f = (0.7+T_infty)/2.
air = thermo.Fluid("air")
air.get_properties(C2K(T_f))
print("second film temperature guess: %3.2f K" %C2K(T_f))
Re = V_infty*D_o_pipe/air.nu
print("Reynolds number: %5.0f"%Re)
#?extconv.CircularCylinder
air_conv = extconv.CircularCylinder('Churchill-Bernstein',Re,air.Pr)
print("Nu = %2.2f" %air_conv.Nu)
h_o = air_conv.Nu*air.k/D_o_pipe

print("Convection coefficient %2.2f W/m^2.K" %h_o)
R2[2].convection(h_o,np.pi*D_o_ins)
q_2 = (T_water-T_infty)/res.serial_sum(R,0,1)
print("Heat loss per unit pipe length: q'=%3.1f W" %q_2)
T_s = T_infty + q_2*R2[2].R
print("Surface temperature: T_s = %2.1f C"%T_s)
Cost = q_2 * 0.1e-3 * 24.
print("Daily cost: $%1.3f /m.d" %Cost)
```

    second film temperature guess: 271.00 K
    Reynolds number: 22833
    Nu = 85.74
    Convection coefficient 20.69 W/m^2.K
    Heat loss per unit pipe length: q'=349.3 W
    Surface temperature: T_s = 39.8 C
    Daily cost: $0.838 /m.d

 PB 7.67

To augment heat transfer between two flowing fluids, it is proposed to insert a $100\text{-mm}$-long, $5\text{-mm}$-diameter 2024 aluminum pin fin through the wall separating the two fluids. The pin is inserted to a depth of $d$ into fluid 1. Fluid 1 is air with a mean temperature of $10^\circ\text{C}$ and velocity of $10\text{m/s}$. Fluid 2 is air with a mean temperature of $40^\circ\text{C}$ and velocity of $3\text{m/s}$.

Determine the rate of heat transfer from the warm air to the cool air through the pin fin for $d=50\text{mm}$.

Plot the variation of the heat transfer rate with the insertion distance, $d$. Does an optimal insertion distance exist?

Assumptions

 Steady state, negligible radiation, boundary layers are neglected (velocity is uniform) and all thermodynamics properties are estimated at the mean of the two stream velocities ($T_f=25^\circ\text{C}$)

Method

Since the distance $d$ varies in question 2, we develop our analysis as a function of the parameter d. The analysis is to be conducted using the assumption of adiabatic length with the modified lengths:
$$
L_{c1}=d+\frac{D}{4},
$$
$$
L_{c2}=L-d+\frac{D}{4},
$$
such that the heat rates are
$$
q_1 = M_1\tanh(m_1L_{c1}),
$$
$$
q_2 = M_2\tanh(m_2L_{c2}),
$$
with
$$
M_1=\sqrt{h_1PkA_c}\theta_{b1}=\sqrt{h_1PkA_c}(T_b-T_{\infty 1}),\; m_1=\sqrt{\frac{h_1P}{kA_c}},
$$
$$
M_2=\sqrt{h_2PkA_c}\theta_{b2}=\sqrt{h_2PkA_c}(T_b-T_{\infty 2}),\; m_2=\sqrt{\frac{h_2P}{kA_c}}.
$$
Here the base temperature $T_b$ is the temperature at the wall.
At thermal equilibrium, heat enters the bottom part of the fin, transfers to the upper part and escpaes into the fluid 1. At the wall, the heat rate leaving the bottom part must be equal to the heat rate entering the upper part. Since $\theta_b = T_b-T_infty$, $q_2$ is negative and $q_1$ is positive. Therefore conservation of energy at the interface between the two fluid writes:
$$
q_1=-q_2.
$$
For any given $d$, the only unknown in this equation is $T_b$ in variables $M_1$ and $M_2$:
$$
\sqrt{h_1PkA_c}(T_b-T_{\infty 1})\tanh(m_1L_{c1})=-\sqrt{h_2PkA_c}(T_b-T_{\infty 2})\tanh(m_2L_{c2})
$$
or
$$
\left(\frac{\sqrt{h_1PkA_c}\tanh(m_1L_{c1})}{\sqrt{h_2PkA_c}\tanh(m_2L_{c2})}+1\right)T_b=
\frac{\sqrt{h_1PkA_c}\tanh(m_1L_{c1})}{\sqrt{h_2PkA_c}\tanh(m_2L_{c2})}T_{\infty 1}+T_{\infty 2}
$$
which can be reduced to:
$$
T_b=\frac{CT_{\infty 1}+T_{\infty 2}}{1+C}
$$
with
$$
C=\frac{\sqrt{h_1PkA_c}\tanh(m_1L_{c1})}{\sqrt{h_2PkA_c}\tanh(m_2L_{c2})}
$$
and
$$
P=\pi D\text{, and } A_c=\pi\frac{D^2}{4}
$$
The convection coefficients are determined using the Churchill Bernstein correlation. From the knowledge of $T_b$, the heat rate can be computed from either $q_1$ or $q_2$. The following defines a function to compute $q_1$.

```python
D = 5.e-3 #mm

T_infty1 = 10. #C
V_infty1 = 10. #m/s
T_infty2 = 40. #C
V_infty2 = 3. #m/s
T_f = 25. #C
k_fin = 177. #W/m.K
air = thermo.Fluid("air")
air.get_properties(C2K(T_f))
Re_1 = V_infty1*D/air.nu
Re_2 = V_infty2*D/air.nu
L = 100.e-3 #mm

airflow_1 = extconv.CircularCylinder('Churchill-Bernstein',Re_1,air.Pr)
airflow_2 = extconv.CircularCylinder('Churchill-Bernstein',Re_2,air.Pr)

h_1 = airflow_1.Nu*air.k/D
h_2 = airflow_2.Nu*air.k/D
print("Convection coefficient for Fluid 1: %3.1f W/m^2.K" %h_1)
print("Convection coefficient for Fluid 2: %3.1f W/m^2.K" %h_2)
P = np.pi*D
A_c = 0.25*np.pi*D**2
m_1 = np.sqrt(h_1*P/(k_fin*A_c))
m_2 = np.sqrt(h_2*P/(k_fin*A_c))
M_1_wotheta = np.sqrt(h_1*P*k_fin*A_c)
M_2_wotheta = np.sqrt(h_2*P*k_fin*A_c)

def heat_rate(d):
    L_c1 = d + D/4.
    L_c2 = L - d + D/4.
    C = (M_1_wotheta*np.tanh(m_1*L_c1))/(M_2_wotheta*np.tanh(m_2*L_c2))
    T_b = (C*T_infty1 + T_infty2)/(1.+ C)
    M_1 = M_1_wotheta*(T_b - T_infty1)
    return M_1*np.tanh(m_1*L_c1)

print("Heat rate for d=50 mm: %1.3f W" %heat_rate(50e-3))

```

    Convection coefficient for Fluid 1: 152.0 W/m^2.K
    Convection coefficient for Fluid 2: 81.9 W/m^2.K
    Heat rate for d=50 mm: 0.927 W

```python
N = 100
d = np.linspace(0.,0.1,N)
q = np.zeros(N)
for i in range (N):
    q[i] = heat_rate(d[i])
plt.plot(d, q, lw=2)
plt.xlabel(r'$d$ ($m$)', fontdict = font)
plt.ylabel(r'$q$ ($W$)', fontdict = font)
plt.grid(True)
plt.show()
```

```python
dd = d[1]-d[0]
dq = np.gradient(q,dd)
plt.plot(d,dq,lw = 2)
plt.xlim(0.042,0.0425)
plt.ylim(-0.1,0.1)
plt.xlabel(r'$d$ ($m$)', fontdict = font)
plt.ylabel(r'$\partial q/\partial d$ ($W/m$)', fontdict = font)
plt.grid(True)
plt.show()
```

By taking the derivative of $q$, the maximum heat rate is $d\approx 42.15\text{mm}$


# Training a neural network on QM9

This tutorial will explain how to use SchNetPack for training a model
on the QM9 dataset and how the trained model can be used for further.

First, we import the necessary modules and create a new directory for the data and our model.

```python
import os
import schnetpack as spk

qm9tut = './qm9tut'
if not os.path.exists('qm9tut'):
    os.makedirs(qm9tut)
```

## Loading the data

As explained in the [previous tutorial](tutorial_01_preparing_data.ipynb), datasets in SchNetPack are loaded with the `AtomsLoader` class or one of the sub-classes that are specialized for common benchmark datasets.
The `QM9` dataset class will download and convert the data. We will only use the inner energy at 0K `U0`, so all other properties do not need to be loaded:

```python
from schnetpack.datasets import QM9

qm9data = QM9('./qm9.db', download=True, load_only=[QM9.U0], remove_uncharacterized=True)
```

### Splitting the data

Next, we split the data into training, validation and test set. Here, we choose to use 1000 training examples, 1000 examples for validation and the remaining data as test set.
The corresponding indices are stored in the `split.npz` file.

```python
train, val, test = spk.train_test_split(
        data=qm9data,
        num_train=1000,
        num_val=500,
        split_file=os.path.join(qm9tut, "split.npz"),
    )
```

Next, we will create an `AtomsLoader` for each split.
This will take care of shuffling, batching and asynchronously loading the data during training and evaluation.

```python
train_loader = spk.AtomsLoader(train, batch_size=100, shuffle=True)
val_loader = spk.AtomsLoader(val, batch_size=100)
```

### Dataset statistics

Before building the model, we need some statistic about our target property for good initial conditions. We will get this from the training examples.
For QM9, we also have single-atom reference values stored in the metadata:

```python
atomrefs = qm9data.get_atomref(QM9.U0)
print('U0 of hyrogen:', '{:.2f}'.format(atomrefs[QM9.U0][1][0]), 'eV')
print('U0 of carbon:', '{:.2f}'.format(atomrefs[QM9.U0][6][0]), 'eV')
print('U0 of oxygen:', '{:.2f}'.format(atomrefs[QM9.U0][8][0]), 'eV')
```

    U0 of hyrogen: -13.61 eV
    U0 of carbon: -1029.86 eV
    U0 of oxygen: -2042.61 eV

These can be used together with the mean and standard deviation of the energy per atom to initialize the model with a good guess of the energy of a molecule. When calculating these statistics, we pass the atomref to take into account, that the model will add these atomrefs to the predicted energy later, so that this part of the energy does not have to be considered in the statistics, i.e.
\begin{equation}
\mu_{U_0} = \frac{1}{n_\text{train}} \sum_{n=1}^{n_\text{train}} \left( U_{0,n} - \sum_{i=1}^{n_{\text{atoms},n}} U_{0,Z_{n,i}} \right)
\end{equation}
for the mean and analog for the standard deviation. In this case, this corresponds to the mean and std. dev of the *atomization energy* per atom.

```python
means, stddevs = train_loader.get_statistics(
    QM9.U0, get_atomwise_statistics=True, single_atom_ref=atomrefs
)
print('Mean atomization energy / atom:', means[QM9.U0])
print('Std. dev. atomization energy / atom:', stddevs[QM9.U0])
```

    Mean atomization energy / atom: tensor([-4.2497])
    Std. dev. atomization energy / atom: tensor([0.1967])

## Building the model

The next step is to build the neural network model.
This consists of 2 parts:

1. The representation part which either constructs atom-wise features, e.g. with SchNet, or build a fixed descriptor such as atom-centered symmetry functions.
2. One or more output modules for property prediction.

We will use a `SchNet` module with 3 interaction layers, a 5 Angstrom cosine cutoff with pairwise distances expanded on 20 Gaussians and 50 atomwise features and convolution filters here, since we only have a few training examples (click [here](../modules/representation.rst#module-schnetpack.representation.schnet) for details on SchNet). Then, we will use one `Atomwise` modules to predict the energy, which takes mean and standard deviation per atom of the property for initialization. Both modules are then combined to an `AtomisticModel`.

```python
schnet = spk.representation.SchNet(
    n_atom_basis=30, n_filters=30, n_gaussians=20, n_interactions=5,
    cutoff=4., cutoff_network=spk.nn.cutoff.CosineCutoff
)

output_U0 = spk.atomistic.Atomwise(n_in=30, atomref=atomrefs[QM9.U0], property=QM9.U0,
                                   mean=means[QM9.U0], stddev=stddevs[QM9.U0])
model = spk.AtomisticModel(representation=schnet, output_modules=output_U0)
```

## Training the model

To train the model, we will use the `Trainer` class that comes with SchNetPack.
For this, we need to first define a loss function and select an optimizer.
As the loss function, we will use the mean squared error of the energy
$\ell(E_\text{ref}, E_\text{pred}) = \frac{1}{n_\text{train}} \sum_{n=1}^{n_\text{train}} (E_\text{ref} - E_\text{pred})^2$

This will be minimized using the Adam optimizer from PyTorch.

```python
from torch.optim import Adam

# loss function
def mse_loss(batch, result):
    diff = batch[QM9.U0]-result[QM9.U0]
    err_sq = torch.mean(diff ** 2)
    return err_sq

# build optimizer
optimizer = Adam(model.parameters(), lr=1e-2)
```

We can give the `Trainer` hooks, that are called at certain points during the training loop.
This is useful to customize the training process, e.g. with logging, learning rate schedules or stopping criteria.
Here, we set up a basis logging as well as a learning rate schedule that reduces the learning rate by factor 0.8 after 5 epochs without improvement of the validation loss.

The logger takes a list of validation metrics that specify what is going to be stored.
In this example, we want to log the mean absolute and root mean squared error of the $U_0$ energy prediction.

```python
# before setting up the trainer, remove previous training checkpoints and logs
%rm -r ./qm9tut/checkpoints
%rm -r ./qm9tut/log.csv

import schnetpack.train as trn

loss = trn.build_mse_loss([QM9.U0])

metrics = [spk.metrics.MeanAbsoluteError(QM9.U0)]
hooks = [
    trn.CSVHook(log_path=qm9tut, metrics=metrics),
    trn.ReduceLROnPlateauHook(
        optimizer,
        patience=5, factor=0.8, min_lr=1e-6,
        stop_after_min=True
    )
]

trainer = trn.Trainer(
    model_path=qm9tut,
    model=model,
    hooks=hooks,
    loss_fn=loss,
    optimizer=optimizer,
    train_loader=train_loader,
    validation_loader=val_loader,
)
```

We can run the training for a given number of epochs. If we don't give a number, the trainer runs until a stopping criterion is met. For the purpose of this tutorial, we let it train for 200 epochs (on GPU this should take about x minutes).

```python
device = "cuda" # change to 'cpu' if gpu is not available
n_epochs = 300 # takes about 10 min on a notebook GPU. reduces for playing around
trainer.train(device=device, n_epochs=n_epochs)
```

You can also call this method several times to continue training. For the training to run until convergence, remove the n_epochs argument or set it to a very large number.

Let us finally have a look at the CSV log:

```python
import numpy as np
import matplotlib.pyplot as plt
from ase.units import kcal, mol

results = np.loadtxt(os.path.join(qm9tut, 'log.csv'), skiprows=1, delimiter=',')

time = results[:,0]-results[0,0]
learning_rate = results[:,1]
train_loss = results[:,2]
val_loss = results[:,3]
val_mae = results[:,4]

print('Final validation MAE:', np.round(val_mae[-1], 2), 'eV =',
      np.round(val_mae[-1] / (kcal/mol), 2), 'kcal/mol')

plt.figure(figsize=(14,5))
plt.subplot(1,2,1)
plt.plot(time, val_loss, label='Validation')
plt.plot(time, train_loss, label='Train')
plt.yscale('log')
plt.ylabel('Loss [eV]')
plt.xlabel('Time [s]')
plt.legend()
plt.subplot(1,2,2)
plt.plot(time, val_mae)
plt.ylabel('mean abs. error [eV]')
plt.xlabel('Time [s]')
plt.show()
```

    Final validation MAE: 0.14 eV = 3.2 kcal/mol

    <Figure size 1400x500 with 2 Axes>

Of course, the prediction can be improved by letting the training run longer, increasing the patience, the number of neurons and interactions or using regularization techniques.

## Using the model

Having trained a model for QM9, we are going to use it to obtain some predictions.
First, we need to load the model. The `Trainer` stores the best model in the model directory which can be loaded using PyTorch:

```python
import torch
best_model = torch.load(os.path.join(qm9tut, 'best_model'))
```

To evaluate it on the test data, we create a data loader for the test split:

```python
test_loader = spk.AtomsLoader(test, batch_size=100)

err = 0
print(len(test_loader))
for count, batch in enumerate(test_loader):
    # move batch to GPU, if necessary
    batch = {k: v.to(device) for k, v in batch.items()}

    # apply model
    pred = best_model(batch)

    # calculate absolute error
    tmp = torch.sum(torch.abs(pred[QM9.U0]-batch[QM9.U0]))
    tmp = tmp.detach().cpu().numpy() # detach from graph & convert to numpy
    err += tmp

    # log progress
    percent = '{:3.2f}'.format(count/len(test_loader)*100)
    print('Progress:', percent+'%'+' '*(5-len(percent)), end="\r")

err /= len(test)
print('Test MAE', np.round(err, 2), 'eV =',
      np.round(err / (kcal/mol), 2), 'kcal/mol')
```

    1328
    Progress: 87.73%

If your data is not already in SchNetPack format, a more convenient way is to use ASE atoms objects with the provided `AtomsConverter`.

```python
converter = spk.data.AtomsConverter(device=device)
at, props = test.get_properties(idx=153)

inputs = converter(at)
print('Keys:', list(inputs.keys()))
print('Truth:', props[QM9.U0].cpu().numpy()[0])

pred = model(inputs)

print('Prediction:', pred[QM9.U0].detach().cpu().numpy()[0,0])
```

Alternatively, one can use the `SpkCalculator` as an interface to ASE:

```python
calculator = spk.interfaces.SpkCalculator(model=model, device=device, energy=QM9.U0)
at.set_calculator(calculator)
print('Prediction:', at.get_total_energy())
```

## Summary
In this tutorial, we have trained and evaluated a SchNet model on a small subset of QM9.
A full training script with is available [here](../../src/examples/qm9_tutorial.py).

In the next tutorial, we will show how to learn potential energy surfaces and forces field as well as performing molecular dynamics simulations with SchNetPack.


# Problema de selección de portafolio con preferencias media-varianza

En la clase pasada hablamos acerca de:
- preferencias,
- funciones de utilidad,
- la actitud de los inversionistas de cara al riesgo,
- la aversión al riesgo, entre otros.

Todas ellas son piezas que necesitamos para responder la pregunta de ¿cómo un inversionista toma la decisión óptima de selección de portafolio?

En esta clase al fin estamos listos para ensamblar estos conceptos y escribir el problema de selección de portafolios.

En el camino aprenderemos acerca del concepto de **utilidad esperada**, que nos permite trabajar con incertidumbre en el modelado económico (una de las ideas más importantes en economía). Esta idea tiene más de 60 años, y básicamente dice que los individuos, cuando están de cara a incertidumbre, maximizan el valor esperado de su utilidad (solo cierto si somos homo economicus).

Además del concepto de utilidad esperada, aprenderemos acerca de **preferencias media-varianza**. Es decir, supondremos que los inversionistas toman decisiones basados en un tipo particular de preferencias.

Con lo anterior, estableceremos el problema de selección de portafolios.

**Objetivos:**

- ¿Qué es utilidad esperada?
- ¿Qué son las preferencias media-varianza?
- Funciones de utilidad media-varianza.
- Enunciado y solución del problema básico de selección de portafolio.

*Referencia:*
- Notas del curso "Portfolio Selection and Risk Management", Rice University, disponible en Coursera.
___

## 1. Utilidad esperada
- Básicamente combina las probabilidades de los resultados con cómo los inversionistas se sienten con dichos resultados.
- En otras palabras, la utilidad esperada multiplica la probabilidad de suceso de un evento con la utilidad que genera dicho evento.

Recordemos que las *funciones de utilidad* permiten a los inversionistas expresar cómo se sienten con los resultados, especialmente en los malos ratos.

Entonces la *utilidad esperada* es una herramienta que nos permite cuantificar cómo nos sentimos en nuestros malos momentos económicos, capturando el riesgo con la probabilidad de ocurrencia de dichos malos momentos.

Dado este marco de trabajo, cualquier decisión se puede escribir como la maximización de la utilidad esperada:
\begin{align}
\max_{\theta} & \quad E[U(W)], \\
\end{align}
mediante la escogencia de cierta variable $\theta$ (gastos, planes de ahorro, compra de activos, planes de producción, etc.).

Para nuestros propósitos, la variable de decisión serán los pesos o ponderaciones del portafolio.

Adicionalmente, en el contexto de la decisión de distribución de la riqueza entre activos, el problema de maximización tendrá comúnmente las siguientes restricciones:
- universo de inversión,
- posición en los activos dados.

**Ejemplo.**

Supongamos que un inversionista debe determinar la composición óptima de su portafolio, que contiene activos y bonos. Supongamos que son los únicos instrumentos disponibles.

Sean:
- $w_s$: peso o ponderación de activos en el portafolio,
- $w_b$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

De manera que podemos escribir el problema de selección de portafolios como la maximización de la utilidad esperade de nuestra riqueza futura, la cual dependerá de nuestros rendimientos:

\begin{align}
\max_{w_s,w_b} &\quad E[U(W)]\\
\text{s. a.}   &\quad W=W_0(1+w_sr_s+w_br_b)\\
               &\quad w_s+w_b=1\\
               &\quad w_s,w_b\geq0
\end{align}

Preguntas:
- ¿Qué significan las restricciones?
- Ya que tenemos planteado este problema básico, ¿qué haría falta para empezar a resolverlo?
___

## 2. Preferencias media-varianza

### 2.1. Utilidad media-varianza

Entonces, ¿qué funciones de utilidad deberíamos de usar en este problema de selección de portafolios?

- La respuesta es: **preferencias media-varianza**.
- Éstas serán representadas en términos de funciones de utilidad como: **utilidad media-varianza**.

Usamos la *utilidad media-varianza* en el problema de selección de portafolios dado que ésta decribe el "trade-off" entre riesgo y rendimiento que enfrentan los inversionistas. La *utilidad media-varianza* está dada por la siguiente expresión:

$$U=E[r_p]-\frac{1}{2}\gamma\sigma_p^2,$$

donde
- $E[r_p]$ es el rendimiento esperado del portafolio,
- $\sigma_p^2$ es la varianza del portafolio, y
- $\gamma$ es el coeficiente de aversión al riesgo.

#### Intuición acerca de la función de utilidad media-varianza:
- Sólo se preocupa por medias :) y varianzas :(.
- Incrementa con: rendimiento esperado del portafolio.
- Decrece con: varianza del portafolio.
- Malos tiempos: rendimientos son bajos y las volatilidades son altas.
- Conecta bastante bien con la teoría moderna de portafolios, la cual caracteriza los rendimientos con medias y varianzas únicamente.
- Criticada por su limitación: supone que los inversionistas sólo se preocupan por medias y varianzas.

### 2.2. Curvas de indiferencia

*¿Recuerdan las curvas de nivel que se ven en cálculo de varias variables?*
- Bien, acá nos servirán para representar la utilidad media-varianza gráficamente.
- En el contexto de utilidad media-varianza, las curvas de nivel se llaman **curvas de indiferencia**.

Dados ciertos niveles de utilidad $U_1>U_2>U_3$, las curvas de indiferencia relativas a estos niveles de utilidad, son los lugares geométricos en el espacio de rendimiento esperado vs. volatilidad representados por las siguientes expresiones

$$U_1=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_1,$$

$$U_2=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_2,$$

$$U_3=E[r_p]-\frac{1}{2}\gamma\sigma_p^2\Rightarrow E[r_p]=\frac{1}{2}\gamma\sigma_p^2+U_3.$$

**Gráficamente**

```python
# Importar numpy y pyplot
import numpy as np
import matplotlib.pyplot as plt
%matplotlib inline
```

```python
# Coeficiente de aversión al riesgo (entre 1 y 10 comúnmente)
gamma = 7
# Niveles de utilidad
U1, U2, U3 = 0.7, 0.5, 0.3
# Vector de volatilidades (sugerido 1%-60%)
sp = np.linspace(0.01, 0.6)
# Curvas de indiferencia
Erp1 = 0.5 * gamma * sp**2 + U1
Erp2 = 0.5 * gamma * sp**2 + U2
Erp3 = 0.5 * gamma * sp**2 + U3
```

```python
# Gráfica
plt.plot(sp, Erp1, label=f'Nivel de utilidad $U_1$={U1}')
plt.plot(sp, Erp2, label=f'Nivel de utilidad $U_2$={U2}')
plt.plot(sp, Erp3, label=f'Nivel de utilidad $U_3$={U3}')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
plt.grid()
plt.legend(loc='best')
```

Bueno, ¿y porqué se llaman curvas de indiferencia?, ¿qué representa una cuva de indiferencia?

- Porque sobre una misma curva el nivel de utilidad es el mismo (es indiferente).
- Son todas las combinaciones de riesgo y rendimiento que producen un mismo nivel de utilidad.

Volviendo al problema de selección de portafolios, queremos la utilidad más alta.
- ¿Cuál de las anteriores curvas de indiferencia corresponde a la utilidad más alta?
- Intuitivamente, ¿porqué?
- Curvas de indiferencia para niveles de utilidad más altos, estarán...

Notamos además que las anteriores curvas de indiferencia son *paralelas* una con otra. Claro, las dibujamos con el mismo coeficiente de aversión al riesgo.

$$\frac{d E[r_p]}{d\sigma_p}=\frac{d }{d\sigma_p}\left[\frac{1}{2}\gamma\sigma_p^2+U\right]=\gamma\sigma_p.$$

¿Cómo cambian estas curvas para coeficientes de aversión al riesgo más altos?

```python
# Coeficientes de aversión al riesgo (entre 1 y 10 comúnmente)
g1, g2, g3 = 3, 5, 7
# Nivel de utilidad
U = 0.3
# Vector de volatilidades (sugerido 1%-60%)
sp = np.linspace(0.01, 0.6)
# Curvas de indiferencia
Erp1 = 0.5 * g1 * sp**2 + U
Erp2 = 0.5 * g2 * sp**2 + U
Erp3 = 0.5 * g3 * sp**2 + U
```

```python
# Gráfica
plt.plot(sp, Erp1, label=f'Coeficiente de aversión al riesgo $\gamma_1$={g1}')
plt.plot(sp, Erp2, label=f'Coeficiente de aversión al riesgo $\gamma_2$={g2}')
plt.plot(sp, Erp3, label=f'Coeficiente de aversión al riesgo $\gamma_3$={g3}')
plt.xlabel('Volatilidad $\sigma$')
plt.ylabel('Rendimiento esperado $E[r]$')
plt.grid()
plt.legend(loc='best')
```

¿Cómo interpretamos las anteriores gráficas?, ¿qué pasa con las personas más aversas al riesgo?
- Se puede ver de dos maneras: para un mismo nivel de rendimiento esperado, una persona más aversa al riesgo soporta un nivel menor de riesgo; equivalentemente, para un mismo nivel de riesgo, una persona más aversa al riesgo requerirá un nivel de rendimiento esperado más alto.

Con todo lo anterior, el problema de selección de portafolios se puede plantear como *encontrar la curva de indeferencia más alta dado el conjunto de oportunidades de inversión y restricciones*.

## 3. Problema de selección de portafolios: una ilustración

Ahora ilustraremos el problema de selección de portafolios con algunos datos.
- Por ahora solo queremos ilustrar gráficamente cómo se resuelve este problema. Trabajar en la intuición.
- En las siguientes dos clases nos enfocaremos en cómo resolverlo analíticamente.

Acá tenemos el rendimiento medio anual y la volatilidad para dos instrumentos usando datos de EU: instrumentos de deuda (bonos) y acciones. Supondremos que el inversionista solo puede invertir en estas dos clases de instrumentos.

```python
# Importamos pandas
import pandas as pd
```

```python
# Datos
data = pd.DataFrame(index=['Stocks','Bonds', 'CorrSB'], columns=['Mean', 'Std'])
data['Mean'] = [0.119, 0.0591, 0.113]
data['Std'] = [0.1915, 0.0833, None]
data
```

      Mean
      Std

      Stocks
      0.1190
      0.1915

      Bonds
      0.0591
      0.0833

      CorrSB
      0.1130
      NaN

Entonces, ¿cuál es la distribución de riqueza óptima?, o más bien, ¿cuál es la composición óptima del portafolio para un inversionista dado su nivel de aversión al riesgo?

**Primero.** Recordamos que, para dos activos, podemos trazar la frontera de mínima varianza tomando todas las posibles combinaciones de los dos activos.

De nuevo, sean:
- $w_s=w$: peso o ponderación de activos en el portafolio,
- $w_b=1-w$: peso o ponderación de bonos en el portafolio,
- $r_s$: rendimiento de los activos, y
- $r_b$: rendimiento de los bonos.

Entonces

$$E[r_p]=wE[r_{s}]+(1-w)E[r_b]$$

$$\sigma_p^2=w^2\sigma_{s}^2+(1-w)^2\sigma_b^2+2w(1-w)\rho_{s,b}\sigma_s\sigma_b$$

```python
# Vector de w variando entre 0 y 1 con n pasos
w = np.linspace(0, 1, 101)
# Rendimientos esperados individuales
Es = data.loc['Stocks', 'Mean']
Eb = data.loc['Bonds', 'Mean']
# Volatilidades individuales
ss = data.loc['Stocks', 'Std']
sb = data.loc['Bonds', 'Std']
# Correlacion
rsb = data.loc['CorrSB', 'Mean']
```

```python
# Crear un DataFrame cuyas columnas sean rendimiento
# y volatilidad del portafolio para cada una de las w
# generadas
portafolios = pd.DataFrame({'w': w,
                            'Mean': w * Es + (1 - w) * Eb,
                            'Std': ((w * ss)**2 + ((1 - w) * sb)**2 + 2 * w * (1 - w) * ss * sb * rsb)**0.5
                           })
portafolios
```

      w
      Mean
      Std

      0
      0.00
      0.059100
      0.083300

      1
      0.01
      0.059699
      0.082705

      2
      0.02
      0.060298
      0.082155

      3
      0.03
      0.060897
      0.081650

      4
      0.04
      0.061496
      0.081191

      ...
      ...
      ...
      ...

      96
      0.96
      0.116604
      0.184246

      97
      0.97
      0.117203
      0.186054

      98
      0.98
      0.117802
      0.187866

      99
      0.99
      0.118401
      0.189681

      100
      1.00
      0.119000
      0.191500

101 rows × 3 columns

```python
# Gráfica
plt.plot(portafolios['Std'], portafolios['Mean'], label='Portafolios')
plt.plot(sb, Eb, 'og', ms=5, label='Bonds')
plt.plot(ss, Es, 'or', ms=5, label='Stocks')
plt.grid()
plt.legend(loc='best')
plt.xlabel('Volatility $\sigma$')
plt.ylabel('Expected return $E[r]$')
```

**Segundo.** Graficamos en la misma ventana, curvas de indiferencia.

```python
# Niveles de utilidad
U1, U2, U3 = 0.07, 0.06, 0.05
# Coeficiente de aversión al riesgo
g = 5
# Curvas de indiferencia
sp = np.linspace(0.08, 0.18)
Erp1 = 0.5 * g * sp**2 + U1
Erp2 = 0.5 * g * sp**2 + U2
Erp3 = 0.5 * g * sp**2 + U3
```

```python
# Gráfica
plt.plot(portafolios['Std'], portafolios['Mean'], label='Portafolios')
plt.plot(sb, Eb, 'og', ms=5, label='Bonds')
plt.plot(ss, Es, 'or', ms=5, label='Stocks')
plt.plot(sp, Erp1, label=f'Nivel de utilidad $U_1$={U1}')
plt.plot(sp, Erp2, label=f'Nivel de utilidad $U_2$={U2}')
plt.plot(sp, Erp3, label=f'Nivel de utilidad $U_3$={U3}')
plt.grid()
plt.legend(loc='best')
plt.xlabel('Volatility $\sigma$')
plt.ylabel('Expected return $E[r]$')
```

**Tercero.** La elección óptima está dada por la curva de indiferencia para el nivel de utilidad más alto que es tangente a la frontera media-varianza.
- Claramente, esta selección depende del coeficiente de aversión al riesgo.

```python
# Gráfica con zoom
# Gráfica
plt.plot(portafolios['Std'], portafolios['Mean'], label='Portafolios')
plt.plot(sb, Eb, 'og', ms=5, label='Bonds')
plt.plot(ss, Es, 'or', ms=5, label='Stocks')
plt.plot(sp, Erp1, label=f'Nivel de utilidad $U_1$={U1}')
plt.plot(sp, Erp2, label=f'Nivel de utilidad $U_2$={U2}')
plt.plot(sp, Erp3, label=f'Nivel de utilidad $U_3$={U3}')
plt.grid()
plt.legend(loc='best')
plt.xlabel('Volatility $\sigma$')
plt.ylabel('Expected return $E[r]$')
plt.axis([0.09, 0.11, 0.08, 0.09])
```

# Anuncios parroquiales

## 1. Quiz la siguiente clase.
## 2. Un par de artículos del WSJ y el NYT que discuten herramientas disponibles para la medición de su propia tolerancia al riesgo:
- [Artículo 1](https://www.nytimes.com/2016/02/13/your-money/as-stocks-fall-its-time-to-measure-your-risk-tolerance.html)
- [Artículo 2](https://www.wsj.com/articles/check-your-tolerance-for-investment-risk-now-before-markets-sag-1405619939)

Created with Jupyter by Esteban Jiménez Rodríguez.


# How does the SFF method work?

Vanderburg and Johnson (2014) introduced a method for "Self Flat Fielding" by tracking how the lightcurve changes with motion of the spacecraft:

[A Technique for Extracting Highly Precise Photometry for the Two-Wheeled Kepler Mission](http://adsabs.harvard.edu/abs/2014PASP..126..948V)

In this notebook we replicate the K2SFF method following the same example source, #60021426, as that in the publication.  We aim to demystify the technique, which is extremely popular within the K2 community.  We have focused on reproducibility, so that we achieve the same result at the publication.

The Vanderburg & Johnson 2014 paper uses data from the Kepler two-wheel "Concept Engineering Test", predating campaign 0, and sometimes called campaign *"eng"* or abbreviated CET.  This vestigal "campaign" lacks some of the standardization of later K2 campaigns--- it was much shorter, only about 9 days long, it lacks some of the standard quality flags, targets have non-traditional EPIC IDs, and other quirks.

```python
%matplotlib inline
import numpy as np
import matplotlib.pyplot as plt
from astropy.io import fits
import pandas as pd
```

## Retrieve the K2SFF data for ENG test source `60021426`

First we will retrieve data and inspect the mask used in the paper.

```python
path = 'http://archive.stsci.edu/hlsps/k2sff/cet/060000000/21426/hlsp_k2sff_k2_lightcurve_060021426-cet_kepler_v1_llc.fits'
vdb_fits = fits.open(path)
```

The `BESTAPER` keyword explains which aperture was chosen as the "best" by Vanderburg & Johnson 2014.  The FITS header for that slice contains the metadata needed to reproduce the mask.

```python
keys = ['MASKTYPE', 'MASKINDE', 'NPIXSAP']
_ = [print(key, ' : ', vdb_fits['BESTAPER'].header[key]) for key in keys]
```

    MASKTYPE  :  PRF FIT
    MASKINDE  :  4
    NPIXSAP  :  29.0

We want the *exact same* mask as Vanderburg & Johnson 2014, but the publication version and MAST version differ!

Publication version:

MAST Version:

Aperture 7 should yield a bigger mask, more similar to what was used in the paper.

```python
VDB_J_mask = vdb_fits['PRF_APER_TBL'].data[7,:, :] == True
VDB_J_mask.sum()
```

    53

Save the mask for easy use in our next notebook.

```python
np.save('VDB_J_2014_mask.npy', VDB_J_mask)
```

## Manually reproduce with the Vanderburg-provided diagnostic data
Retrieve the Vanderburg-provided diagnostic data for the Kepler ENG testing.
Uncomment the line below to retrieve the data programmatically, or manually get the linked file in a browser and save it to this directory.

```python
#! wget https://www.cfa.harvard.edu/~avanderb/k2/ep60021426alldiagnostics.csv
```

```python
df = pd.read_csv('ep60021426alldiagnostics.csv',index_col=False)
df.head()
```

      BJD - 2454833
      Raw Flux
      Corrected Flux
      X-centroid
      Y-centroid
      arclength
      Correction
      Thrusters On

      0
      1862.502368
      0.995119
      0.995985
      25.135097
      24.661074
      2.327480
      0.999130
      1.0

      1
      1862.522801
      0.997313
      0.996767
      25.289752
      24.418689
      1.175322
      1.000548
      1.0

      2
      1862.543235
      0.996713
      0.996136
      25.288052
      24.429406
      1.214627
      1.000580
      0.0

      3
      1862.563668
      0.996930
      0.996277
      25.275216
      24.448405
      1.306617
      1.000656
      0.0

      4
      1862.584102
      0.996862
      0.996228
      25.253864
      24.480184
      1.460259
      1.000636
      0.0

We can mean-subtract the provided $x-y$ centroids, assigning them column and row identifiers, then rotate the coordinates into their major and minor axes.

```python
col = df[' X-centroid'].values
col = col - np.mean(col)
row = df[' Y-centroid'].values
row = row - np.mean(row)
```

```python
def _get_eigen_vectors(centroid_col, centroid_row):
    '''get the eigenvalues and eigenvectors given centroid x, y positions'''
    centroids = np.array([centroid_col, centroid_row])
    eig_val, eig_vec = np.linalg.eigh(np.cov(centroids))
    return eig_val, eig_vec
```

```python
def _rotate(eig_vec, centroid_col, centroid_row):
    '''rotate the centroids into their predominant linear axis'''
    centroids = np.array([centroid_col, centroid_row])
    return np.dot(eig_vec, centroids)
```

```python
eig_val, eig_vec = _get_eigen_vectors(col, row)
v1, v2 = eig_vec
```

The major axis is the latter.

```python
platescale = 4.0 # The Kepler plate scale; has units of arcseconds / pixel
```

```python
plt.figure(figsize=(5, 6))
plt.plot(col * platescale, row * platescale, 'ko', ms=4)
plt.plot(col * platescale, row * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel('X position [arcseconds]')
plt.ylabel('Y position [arcseconds]')
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, v1[0]], [0, v1[1]], color='blue', lw=3)
plt.plot([0, v2[0]], [0, v2[1]], color='blue', lw=3);
```

Following the form of **Figure 2** of Vanderburg & Johsnon 2014.

```python
rot_colp, rot_rowp = _rotate(eig_vec, col, row) #units in pixels
```

You can rotate into the new reference frame.

```python
plt.figure(figsize=(5, 6))
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ko', ms=4)
plt.plot(rot_rowp * platescale, rot_colp * platescale, 'ro', ms=1)
plt.xticks([-2, -1,0, 1, 2])
plt.yticks([-2, -1,0, 1, 2])
plt.xlabel("X' position [arcseconds]")
plt.ylabel("Y' position [arcseconds]")
plt.xlim(-2, 2)
plt.ylim(-2, 2)
plt.plot([0, 1], [0, 0], color='blue')
plt.plot([0, 0], [0, 1], color='blue');
```

We need to calculate the arclength using:
\begin{equation}s= \int_{x'_0}^{x'_1}\sqrt{1+\left( \frac{dy'_p}{dx'}\right)^2} dx'\end{equation}

where $x^\prime_0$ is the transformed $x$ coordinate of the point with the smallest $x^\prime$ position, and $y^\prime_p$ is the best--fit polynomial function.

Fit a $5^{th}$ order polynomial to the rotated coordinates.

```python
z = np.polyfit(rot_rowp, rot_colp, 5)
p5 = np.poly1d(z)
p5_deriv = p5.deriv()
```

```python
x0_prime = np.min(rot_rowp)
xmax_prime = np.max(rot_rowp)
x_dense = np.linspace(x0_prime, xmax_prime, 2000)
```

```python
plt.plot(rot_rowp, rot_colp, '.')
plt.plot(x_dense, p5(x_dense))
plt.ylabel('Position along minor axis (pixels)')
plt.xlabel('Position along major axis (pixels)')
plt.title('Performance of polynomial regression')
plt.ylim(-0.1, 0.1);
```

We see evidence for a [bias-variance tradeoff](https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff), suggesting some modest opportunity for improvement.

```python
@np.vectorize
def arclength(x):
    '''Input x1_prime, get out arclength'''
    gi = x_dense <x
    s_integrand = np.sqrt(1 + p5_deriv(x_dense[gi]) ** 2)
    s = np.trapz(s_integrand, x=x_dense[gi])
    return s
```

Let's double check that we compute the same arclength as the published paper.

```python
aspect_ratio = plt.figaspect(1)
plt.figure(figsize=aspect_ratio)
plt.plot(df[' arclength'], arclength(rot_rowp)*4.0, '.')
plt.xlabel('$s$  (Vanderburg & Johnson 2014)')
plt.ylabel('$s$  (This work)')
plt.plot([0, 4], [0, 4], 'k--');
```

Yes, we compute arclength correctly.

Now we apply a **high-pass filter** to the raw lightcurve data.  We follow the original paper by using *BSplines* with 1.5 day breakpoints.  You can also apply data exclusion at this stage.

```python
from scipy.interpolate import BSpline
from scipy import interpolate
```

```python
times, raw_fluxes = df['BJD - 2454833'].values, df[' Raw Flux'].values
```

We find the weighted least square spline for a given set of knots, $t$.  We supply interior knots as knots on the ends are added automatically, as stated in the `interpolate.splrep()` docstring.

```python
interior_knots = np.arange(times[0]+1.5, times[0]+6, 1.5)
t,c,k = interpolate.splrep(times, raw_fluxes, s=0, task=-1, t=interior_knots)
bspl = BSpline(t,c,k)
```

```python
plt.plot(times, raw_fluxes, '.')
plt.plot(times, bspl(times))
plt.xlabel('$t$ (days)')
plt.ylabel('Raw Flux');
```

The Spline fit looks good, so we can normalize the flux by the long-term trend.
Plot the normalized flux versus arclength to see the position-dependent flux.

```python
fluxes = raw_fluxes/bspl(times)
```

Mask the data by keeping only the good samples.

```python
bi = df[' Thrusters On'].values == 1.0
gi = df[' Thrusters On'].values == 0.0
clean_fluxes = fluxes[gi]
al = arclength(rot_rowp[gi]) * platescale
```

```python
sorted_inds = np.argsort(al)
```

We will follow the paper by interpolating **flux versus arclength position** in 15 bins of means, which is a *piecewise linear fit*.

```python
knots = np.array([np.min(al)]+
                 [np.median(splt) for splt in np.array_split(al[sorted_inds], 15)]+
                 [np.max(al)])
```

```python
bin_means = np.array([clean_fluxes[sorted_inds][0]]+
                     [np.mean(splt) for splt in np.array_split(clean_fluxes[sorted_inds], 15)]+
                     [clean_fluxes[sorted_inds][-1]])
```

```python
zz = np.polyfit(al, clean_fluxes,6)
sff = np.poly1d(zz)
al_dense = np.linspace(0, 4, 1000)
interp_func = interpolate.interp1d(knots, bin_means)
```

```python
plt.figure(figsize=(5, 6))
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'ko', ms=4)
plt.plot(arclength(rot_rowp)*4.0, fluxes, 'o', color='#3498db', ms=3)
plt.plot(arclength(rot_rowp[bi])*4.0, fluxes[bi], 'o', color='r', ms=3)
plt.plot(np.sort(al), interp_func(np.sort(al)), '-', color='#e67e22')

plt.xticks([0, 1,2, 3, 4])
plt.minorticks_on()
plt.xlabel('Arclength [arcseconds]')
plt.ylabel('Relative Brightness')
plt.title('EPIC 60021426, Kp =10.3')
plt.xlim(0,4)
plt.ylim(0.997, 1.002);
```

Following **Figure 4** of Vanderburg & Johnson 2014.

Apply the Self Flat Field (SFF) correction:

```python
corr_fluxes = clean_fluxes / interp_func(al)
```

```python
plt.figure(figsize=(10,6))

dy = 0.004
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'ko', ms=4)
plt.plot(df['BJD - 2454833'], df[' Raw Flux']+dy, 'o', color='#3498db', ms=3)
plt.plot(df['BJD - 2454833'][bi], df[' Raw Flux'][bi]+dy, 'o', color='r', ms=3)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='k', ms = 4)
plt.plot(df['BJD - 2454833'][gi], corr_fluxes*bspl(times[gi]), 'o', color='#e67e22', ms = 3)

plt.xlabel('BJD - 2454833')
plt.ylabel('Relative Brightness')

plt.xlim(1862, 1870)
plt.ylim(0.994, 1.008);
```

Following **Figure 5** of Vanderburg & Johnson 2015.

*The end.*